2023-11-02T17:35:29.077 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-02T17:35:29.078 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849 --verbose --owner scheduled_yuriw@teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi} --name yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849/orig.config.yaml 2023-11-02T17:35:29.115 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-02T17:35:29.197 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849 branch: wip-yuri3-testing-2023-10-31-1116-reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7442849' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri3-testing-2023-10-31-1116-reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: b8ceb241941938aa7529adcc4b99d42d196489ec wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b8ceb241941938aa7529adcc4b99d42d196489ec rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py workunit: branch: wip-yuri3-testing-2023-10-31-1116-reef sha1: b8ceb241941938aa7529adcc4b99d42d196489ec owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 7837 sha1: b8ceb241941938aa7529adcc4b99d42d196489ec sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri3-testing-2023-10-31-1116-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b8ceb241941938aa7529adcc4b99d42d196489ec targets: smithi017.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-01_14:50:55 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-11-02T17:35:29.197 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa; will attempt to use it 2023-11-02T17:35:29.198 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/tasks 2023-11-02T17:35:29.198 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-02T17:35:29.200 INFO:teuthology.task.internal:Checking packages... 2023-11-02T17:35:29.224 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b8ceb241941938aa7529adcc4b99d42d196489ec' 2023-11-02T17:35:29.224 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-02T17:35:29.225 INFO:teuthology.packaging:ref: None 2023-11-02T17:35:29.225 INFO:teuthology.packaging:tag: None 2023-11-02T17:35:29.225 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-10-31-1116-reef 2023-11-02T17:35:29.226 INFO:teuthology.packaging:sha1: b8ceb241941938aa7529adcc4b99d42d196489ec 2023-11-02T17:35:29.226 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri3-testing-2023-10-31-1116-reef 2023-11-02T17:35:29.367 INFO:teuthology.task.internal:Found packages for ceph version 18.2.0-1164-gb8ceb241-1focal 2023-11-02T17:35:29.367 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-02T17:35:29.374 INFO:teuthology.task.internal:no buildpackages task found 2023-11-02T17:35:29.374 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-02T17:35:29.386 INFO:teuthology.task.internal:Saving configuration 2023-11-02T17:35:29.400 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-02T17:35:29.468 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-02T17:35:29.492 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-02 17:29:43.177004', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-11-02T17:35:29.517 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-02 17:29:43.175265', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-11-02T17:35:29.518 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-02T17:35:29.541 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-11-02T17:35:29.541 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-11-02T17:35:29.541 INFO:teuthology.run_tasks:Running task console_log... 2023-11-02T17:35:29.672 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdb20234a60>, signals=[15]) 2023-11-02T17:35:29.673 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-02T17:35:29.723 INFO:teuthology.task.internal:Opening connections... 2023-11-02T17:35:29.723 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2023-11-02T17:35:29.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T17:35:29.772 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2023-11-02T17:35:29.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T17:35:29.822 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-02T17:35:29.859 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2023-11-02T17:35:29.866 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2023-11-02T17:35:29.866 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2023-11-02T17:35:29.914 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="Ubuntu" 2023-11-02T17:35:29.914 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-11-02T17:35:29.915 INFO:teuthology.orchestra.run.smithi017.stdout:ID=ubuntu 2023-11-02T17:35:29.915 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE=debian 2023-11-02T17:35:29.915 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-11-02T17:35:29.915 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="20.04" 2023-11-02T17:35:29.916 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-02T17:35:29.916 INFO:teuthology.orchestra.run.smithi017.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-02T17:35:29.916 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-02T17:35:29.916 INFO:teuthology.orchestra.run.smithi017.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-02T17:35:29.916 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_CODENAME=focal 2023-11-02T17:35:29.917 INFO:teuthology.orchestra.run.smithi017.stdout:UBUNTU_CODENAME=focal 2023-11-02T17:35:29.917 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2023-11-02T17:35:29.942 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2023-11-02T17:35:29.949 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2023-11-02T17:35:29.949 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2023-11-02T17:35:29.998 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="Ubuntu" 2023-11-02T17:35:29.999 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-11-02T17:35:29.999 INFO:teuthology.orchestra.run.smithi069.stdout:ID=ubuntu 2023-11-02T17:35:29.999 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE=debian 2023-11-02T17:35:29.999 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-11-02T17:35:30.000 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="20.04" 2023-11-02T17:35:30.000 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-02T17:35:30.000 INFO:teuthology.orchestra.run.smithi069.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-02T17:35:30.000 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-02T17:35:30.000 INFO:teuthology.orchestra.run.smithi069.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-02T17:35:30.001 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_CODENAME=focal 2023-11-02T17:35:30.001 INFO:teuthology.orchestra.run.smithi069.stdout:UBUNTU_CODENAME=focal 2023-11-02T17:35:30.001 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2023-11-02T17:35:30.024 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-02T17:35:30.033 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-02T17:35:30.178 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-02T17:35:30.179 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2023-11-02T17:35:30.182 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2023-11-02T17:35:30.187 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-02T17:35:30.254 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-02T17:35:30.254 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2023-11-02T17:35:30.257 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2023-11-02T17:35:30.278 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-02T17:35:30.377 INFO:teuthology.run_tasks:Running task kernel... 2023-11-02T17:35:30.500 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-02T17:35:30.502 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-02T17:35:30.502 DEBUG:teuthology.orchestra.run.smithi017:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-02T17:35:30.504 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-02T17:35:30.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T17:35:30.511 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2023-11-02T17:35:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T17:35:30.513 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-11-02T17:35:30.520 INFO:teuthology.orchestra.run.smithi017.stdout:5.4.0-124-generic 2023-11-02T17:35:30.520 INFO:teuthology.task.kernel:Running kernel on smithi017: 5.4.0-124-generic 2023-11-02T17:35:30.521 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get clean 2023-11-02T17:35:30.522 INFO:teuthology.orchestra.run.smithi069.stdout:5.4.0-124-generic 2023-11-02T17:35:30.522 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.4.0-124-generic 2023-11-02T17:35:30.522 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get clean 2023-11-02T17:35:30.673 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get update 2023-11-02T17:35:30.689 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get update 2023-11-02T17:35:30.825 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-02T17:35:30.838 INFO:teuthology.orchestra.run.smithi017.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-11-02T17:35:30.845 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-02T17:35:30.854 INFO:teuthology.orchestra.run.smithi069.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-11-02T17:35:30.897 INFO:teuthology.orchestra.run.smithi017.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-11-02T17:35:30.927 INFO:teuthology.orchestra.run.smithi069.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-11-02T17:35:30.938 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-11-02T17:35:31.192 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [906 kB] 2023-11-02T17:35:31.230 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [906 kB] 2023-11-02T17:35:31.275 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,955 kB] 2023-11-02T17:35:31.313 INFO:teuthology.orchestra.run.smithi017.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [481 kB] 2023-11-02T17:35:31.316 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2023-11-02T17:35:31.317 INFO:teuthology.orchestra.run.smithi017.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,480 kB] 2023-11-02T17:35:31.322 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,955 kB] 2023-11-02T17:35:31.336 INFO:teuthology.orchestra.run.smithi017.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.2 kB] 2023-11-02T17:35:31.336 INFO:teuthology.orchestra.run.smithi017.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [346 kB] 2023-11-02T17:35:31.339 INFO:teuthology.orchestra.run.smithi017.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [568 B] 2023-11-02T17:35:31.339 INFO:teuthology.orchestra.run.smithi017.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,130 kB] 2023-11-02T17:35:31.352 INFO:teuthology.orchestra.run.smithi017.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [753 kB] 2023-11-02T17:35:31.357 INFO:teuthology.orchestra.run.smithi017.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [269 kB] 2023-11-02T17:35:31.359 INFO:teuthology.orchestra.run.smithi017.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2023-11-02T17:35:31.360 INFO:teuthology.orchestra.run.smithi017.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,472 B] 2023-11-02T17:35:31.360 INFO:teuthology.orchestra.run.smithi017.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.9 kB] 2023-11-02T17:35:31.363 INFO:teuthology.orchestra.run.smithi017.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-11-02T17:35:31.363 INFO:teuthology.orchestra.run.smithi017.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-11-02T17:35:31.364 INFO:teuthology.orchestra.run.smithi017.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-11-02T17:35:31.364 INFO:teuthology.orchestra.run.smithi017.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-11-02T17:35:31.367 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [481 kB] 2023-11-02T17:35:31.371 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2023-11-02T17:35:31.372 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.2 kB] 2023-11-02T17:35:31.372 INFO:teuthology.orchestra.run.smithi069.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,480 kB] 2023-11-02T17:35:31.380 INFO:teuthology.orchestra.run.smithi017.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-11-02T17:35:31.380 INFO:teuthology.orchestra.run.smithi017.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-11-02T17:35:31.381 INFO:teuthology.orchestra.run.smithi017.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-11-02T17:35:31.381 INFO:teuthology.orchestra.run.smithi017.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-11-02T17:35:31.393 INFO:teuthology.orchestra.run.smithi069.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [346 kB] 2023-11-02T17:35:31.396 INFO:teuthology.orchestra.run.smithi069.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [568 B] 2023-11-02T17:35:31.396 INFO:teuthology.orchestra.run.smithi069.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [753 kB] 2023-11-02T17:35:31.402 INFO:teuthology.orchestra.run.smithi069.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,130 kB] 2023-11-02T17:35:31.413 INFO:teuthology.orchestra.run.smithi069.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [269 kB] 2023-11-02T17:35:31.415 INFO:teuthology.orchestra.run.smithi069.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2023-11-02T17:35:31.416 INFO:teuthology.orchestra.run.smithi069.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,472 B] 2023-11-02T17:35:31.417 INFO:teuthology.orchestra.run.smithi069.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.9 kB] 2023-11-02T17:35:31.417 INFO:teuthology.orchestra.run.smithi069.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-11-02T17:35:31.417 INFO:teuthology.orchestra.run.smithi069.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-11-02T17:35:31.435 INFO:teuthology.orchestra.run.smithi069.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-11-02T17:35:31.436 INFO:teuthology.orchestra.run.smithi069.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-11-02T17:35:31.436 INFO:teuthology.orchestra.run.smithi069.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-11-02T17:35:31.437 INFO:teuthology.orchestra.run.smithi069.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-11-02T17:35:31.455 INFO:teuthology.orchestra.run.smithi069.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-11-02T17:35:31.456 INFO:teuthology.orchestra.run.smithi069.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-11-02T17:35:31.498 INFO:teuthology.orchestra.run.smithi017.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [672 kB] 2023-11-02T17:35:31.826 INFO:teuthology.orchestra.run.smithi017.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,564 kB] 2023-11-02T17:35:32.014 INFO:teuthology.orchestra.run.smithi017.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [398 kB] 2023-11-02T17:35:32.029 INFO:teuthology.orchestra.run.smithi017.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-11-02T17:35:32.029 INFO:teuthology.orchestra.run.smithi017.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-11-02T17:35:32.031 INFO:teuthology.orchestra.run.smithi017.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,360 kB] 2023-11-02T17:35:32.098 INFO:teuthology.orchestra.run.smithi017.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [329 kB] 2023-11-02T17:35:32.104 INFO:teuthology.orchestra.run.smithi017.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-11-02T17:35:32.105 INFO:teuthology.orchestra.run.smithi017.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [619 kB] 2023-11-02T17:35:32.118 INFO:teuthology.orchestra.run.smithi017.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [899 kB] 2023-11-02T17:35:32.318 INFO:teuthology.orchestra.run.smithi017.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [188 kB] 2023-11-02T17:35:32.738 INFO:teuthology.orchestra.run.smithi017.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2023-11-02T17:35:32.742 INFO:teuthology.orchestra.run.smithi017.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-11-02T17:35:32.744 INFO:teuthology.orchestra.run.smithi017.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-11-02T17:35:32.752 INFO:teuthology.orchestra.run.smithi017.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-11-02T17:35:32.754 INFO:teuthology.orchestra.run.smithi017.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-11-02T17:35:34.429 INFO:teuthology.orchestra.run.smithi069.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-11-02T17:35:34.658 INFO:teuthology.orchestra.run.smithi069.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,564 kB] 2023-11-02T17:35:34.787 INFO:teuthology.orchestra.run.smithi069.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [672 kB] 2023-11-02T17:35:34.796 INFO:teuthology.orchestra.run.smithi069.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [398 kB] 2023-11-02T17:35:34.801 INFO:teuthology.orchestra.run.smithi069.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-11-02T17:35:34.802 INFO:teuthology.orchestra.run.smithi069.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-11-02T17:35:34.802 INFO:teuthology.orchestra.run.smithi069.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,360 kB] 2023-11-02T17:35:34.837 INFO:teuthology.orchestra.run.smithi069.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [329 kB] 2023-11-02T17:35:34.842 INFO:teuthology.orchestra.run.smithi069.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-11-02T17:35:34.842 INFO:teuthology.orchestra.run.smithi069.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [619 kB] 2023-11-02T17:35:34.850 INFO:teuthology.orchestra.run.smithi069.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [899 kB] 2023-11-02T17:35:34.865 INFO:teuthology.orchestra.run.smithi069.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [188 kB] 2023-11-02T17:35:34.867 INFO:teuthology.orchestra.run.smithi069.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2023-11-02T17:35:34.868 INFO:teuthology.orchestra.run.smithi069.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-11-02T17:35:34.868 INFO:teuthology.orchestra.run.smithi069.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-11-02T17:35:34.868 INFO:teuthology.orchestra.run.smithi069.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-11-02T17:35:34.888 INFO:teuthology.orchestra.run.smithi069.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-11-02T17:35:35.428 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 18.0 MB in 3s (6,863 kB/s) 2023-11-02T17:35:36.924 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2023-11-02T17:35:36.951 DEBUG:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-02T17:35:37.027 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2023-11-02T17:35:37.237 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2023-11-02T17:35:37.238 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2023-11-02T17:35:37.436 INFO:teuthology.orchestra.run.smithi017.stdout:The following additional packages will be installed: 2023-11-02T17:35:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: linux-generic linux-headers-5.4.0-166 linux-headers-5.4.0-166-generic 2023-11-02T17:35:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: linux-headers-generic linux-image-5.4.0-166-generic 2023-11-02T17:35:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: linux-modules-5.4.0-166-generic linux-modules-extra-5.4.0-166-generic 2023-11-02T17:35:37.437 INFO:teuthology.orchestra.run.smithi017.stdout:Suggested packages: 2023-11-02T17:35:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-11-02T17:35:37.438 INFO:teuthology.orchestra.run.smithi017.stdout:Recommended packages: 2023-11-02T17:35:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: thermald 2023-11-02T17:35:37.496 INFO:teuthology.orchestra.run.smithi017.stdout:The following NEW packages will be installed: 2023-11-02T17:35:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: linux-headers-5.4.0-166 linux-headers-5.4.0-166-generic 2023-11-02T17:35:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: linux-image-5.4.0-166-generic linux-modules-5.4.0-166-generic 2023-11-02T17:35:37.497 INFO:teuthology.orchestra.run.smithi017.stdout: linux-modules-extra-5.4.0-166-generic 2023-11-02T17:35:37.497 INFO:teuthology.orchestra.run.smithi017.stdout:The following packages will be upgraded: 2023-11-02T17:35:37.498 INFO:teuthology.orchestra.run.smithi017.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-02T17:35:37.634 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 18.0 MB in 5s (3,662 kB/s) 2023-11-02T17:35:37.682 INFO:teuthology.orchestra.run.smithi017.stdout:3 upgraded, 5 newly installed, 0 to remove and 272 not upgraded. 2023-11-02T17:35:37.682 INFO:teuthology.orchestra.run.smithi017.stdout:Need to get 77.1 MB of archives. 2023-11-02T17:35:37.682 INFO:teuthology.orchestra.run.smithi017.stdout:After this operation, 380 MB of additional disk space will be used. 2023-11-02T17:35:37.682 INFO:teuthology.orchestra.run.smithi017.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-166-generic amd64 5.4.0-166.183 [15.0 MB] 2023-11-02T17:35:38.832 INFO:teuthology.orchestra.run.smithi017.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-166-generic amd64 5.4.0-166.183 [10.5 MB] 2023-11-02T17:35:39.105 INFO:teuthology.orchestra.run.smithi017.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-166-generic amd64 5.4.0-166.183 [39.2 MB] 2023-11-02T17:35:39.175 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-11-02T17:35:39.201 DEBUG:teuthology.orchestra.run.smithi069:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-02T17:35:39.274 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-11-02T17:35:39.484 INFO:teuthology.orchestra.run.smithi069.stdout:Building dependency tree... 2023-11-02T17:35:39.485 INFO:teuthology.orchestra.run.smithi069.stdout:Reading state information... 2023-11-02T17:35:39.664 INFO:teuthology.orchestra.run.smithi069.stdout:The following additional packages will be installed: 2023-11-02T17:35:39.664 INFO:teuthology.orchestra.run.smithi069.stdout: linux-generic linux-headers-5.4.0-166 linux-headers-5.4.0-166-generic 2023-11-02T17:35:39.664 INFO:teuthology.orchestra.run.smithi069.stdout: linux-headers-generic linux-image-5.4.0-166-generic 2023-11-02T17:35:39.665 INFO:teuthology.orchestra.run.smithi069.stdout: linux-modules-5.4.0-166-generic linux-modules-extra-5.4.0-166-generic 2023-11-02T17:35:39.665 INFO:teuthology.orchestra.run.smithi069.stdout:Suggested packages: 2023-11-02T17:35:39.666 INFO:teuthology.orchestra.run.smithi069.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-11-02T17:35:39.666 INFO:teuthology.orchestra.run.smithi069.stdout:Recommended packages: 2023-11-02T17:35:39.666 INFO:teuthology.orchestra.run.smithi069.stdout: thermald 2023-11-02T17:35:39.718 INFO:teuthology.orchestra.run.smithi069.stdout:The following NEW packages will be installed: 2023-11-02T17:35:39.718 INFO:teuthology.orchestra.run.smithi069.stdout: linux-headers-5.4.0-166 linux-headers-5.4.0-166-generic 2023-11-02T17:35:39.718 INFO:teuthology.orchestra.run.smithi069.stdout: linux-image-5.4.0-166-generic linux-modules-5.4.0-166-generic 2023-11-02T17:35:39.719 INFO:teuthology.orchestra.run.smithi069.stdout: linux-modules-extra-5.4.0-166-generic 2023-11-02T17:35:39.720 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages will be upgraded: 2023-11-02T17:35:39.720 INFO:teuthology.orchestra.run.smithi069.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-02T17:35:39.778 INFO:teuthology.orchestra.run.smithi069.stdout:3 upgraded, 5 newly installed, 0 to remove and 272 not upgraded. 2023-11-02T17:35:39.778 INFO:teuthology.orchestra.run.smithi069.stdout:Need to get 77.1 MB of archives. 2023-11-02T17:35:39.778 INFO:teuthology.orchestra.run.smithi069.stdout:After this operation, 380 MB of additional disk space will be used. 2023-11-02T17:35:39.778 INFO:teuthology.orchestra.run.smithi069.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-166-generic amd64 5.4.0-166.183 [15.0 MB] 2023-11-02T17:35:40.156 INFO:teuthology.orchestra.run.smithi069.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-166-generic amd64 5.4.0-166.183 [10.5 MB] 2023-11-02T17:35:40.255 INFO:teuthology.orchestra.run.smithi069.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-166-generic amd64 5.4.0-166.183 [39.2 MB] 2023-11-02T17:35:40.469 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.166.163 [1,900 B] 2023-11-02T17:35:40.470 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.166.163 [2,592 B] 2023-11-02T17:35:40.470 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-166 all 5.4.0-166.183 [11.0 MB] 2023-11-02T17:35:40.791 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.166.163 [1,900 B] 2023-11-02T17:35:40.792 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.166.163 [2,592 B] 2023-11-02T17:35:40.792 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-166 all 5.4.0-166.183 [11.0 MB] 2023-11-02T17:35:40.881 INFO:teuthology.orchestra.run.smithi017.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-166-generic amd64 5.4.0-166.183 [1,370 kB] 2023-11-02T17:35:40.887 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-166-generic amd64 5.4.0-166.183 [1,370 kB] 2023-11-02T17:35:40.898 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.166.163 [2,464 B] 2023-11-02T17:35:40.901 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.166.163 [2,464 B] 2023-11-02T17:35:41.563 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 77.1 MB in 1s (65.8 MB/s) 2023-11-02T17:35:41.591 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 77.1 MB in 3s (22.7 MB/s) 2023-11-02T17:35:41.765 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-modules-5.4.0-166-generic. 2023-11-02T17:35:41.781 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-modules-5.4.0-166-generic. 2023-11-02T17:35:43.541 INFO:teuthology.orchestra.run.smithi069.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-11-02T17:35:43.546 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../0-linux-modules-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-02T17:35:43.549 INFO:teuthology.orchestra.run.smithi017.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-11-02T17:35:43.554 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../0-linux-modules-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-02T17:35:43.678 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-modules-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:35:43.703 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-modules-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:35:45.647 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-image-5.4.0-166-generic. 2023-11-02T17:35:45.662 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../1-linux-image-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-02T17:35:45.761 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-image-5.4.0-166-generic. 2023-11-02T17:35:45.770 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-image-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:35:45.774 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../1-linux-image-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-02T17:35:45.873 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-image-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:35:46.900 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-166-generic. 2023-11-02T17:35:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-02T17:35:46.949 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-modules-extra-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:35:47.006 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-166-generic. 2023-11-02T17:35:47.019 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-02T17:35:47.055 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-modules-extra-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:35:51.770 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../3-linux-generic_5.4.0.166.163_amd64.deb ... 2023-11-02T17:35:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-generic (5.4.0.166.163) over (5.4.0.124.125) ... 2023-11-02T17:35:52.193 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../3-linux-generic_5.4.0.166.163_amd64.deb ... 2023-11-02T17:35:52.254 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.166.163_amd64.deb ... 2023-11-02T17:35:52.316 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-generic (5.4.0.166.163) over (5.4.0.124.125) ... 2023-11-02T17:35:52.406 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-image-generic (5.4.0.166.163) over (5.4.0.124.125) ... 2023-11-02T17:35:52.717 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.166.163_amd64.deb ... 2023-11-02T17:35:52.751 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-headers-5.4.0-166. 2023-11-02T17:35:52.766 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../5-linux-headers-5.4.0-166_5.4.0-166.183_all.deb ... 2023-11-02T17:35:52.809 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-headers-5.4.0-166 (5.4.0-166.183) ... 2023-11-02T17:35:52.861 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-image-generic (5.4.0.166.163) over (5.4.0.124.125) ... 2023-11-02T17:35:53.214 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-headers-5.4.0-166. 2023-11-02T17:35:53.231 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../5-linux-headers-5.4.0-166_5.4.0-166.183_all.deb ... 2023-11-02T17:35:53.297 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-headers-5.4.0-166 (5.4.0-166.183) ... 2023-11-02T17:35:58.901 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-headers-5.4.0-166-generic. 2023-11-02T17:35:58.923 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../6-linux-headers-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-02T17:35:58.976 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-headers-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:35:59.094 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-headers-5.4.0-166-generic. 2023-11-02T17:35:59.118 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../6-linux-headers-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-02T17:35:59.160 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-headers-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:36:02.130 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.166.163_amd64.deb ... 2023-11-02T17:36:02.277 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-headers-generic (5.4.0.166.163) over (5.4.0.124.125) ... 2023-11-02T17:36:02.367 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.166.163_amd64.deb ... 2023-11-02T17:36:02.488 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-headers-generic (5.4.0.166.163) over (5.4.0.124.125) ... 2023-11-02T17:36:02.635 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-modules-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:36:02.839 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-modules-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:36:03.898 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-headers-5.4.0-166 (5.4.0-166.183) ... 2023-11-02T17:36:04.049 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-image-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:36:04.111 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-headers-5.4.0-166 (5.4.0-166.183) ... 2023-11-02T17:36:04.262 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-image-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:36:05.414 INFO:teuthology.orchestra.run.smithi069.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-166-generic 2023-11-02T17:36:05.414 INFO:teuthology.orchestra.run.smithi069.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-166-generic 2023-11-02T17:36:05.572 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-headers-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:36:05.579 INFO:teuthology.orchestra.run.smithi017.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-166-generic 2023-11-02T17:36:05.579 INFO:teuthology.orchestra.run.smithi017.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-166-generic 2023-11-02T17:36:05.698 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-modules-extra-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:36:05.735 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-headers-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:36:05.878 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-modules-extra-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:36:06.908 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-headers-generic (5.4.0.166.163) ... 2023-11-02T17:36:07.034 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-image-generic (5.4.0.166.163) ... 2023-11-02T17:36:07.104 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-headers-generic (5.4.0.166.163) ... 2023-11-02T17:36:07.177 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-generic (5.4.0.166.163) ... 2023-11-02T17:36:07.238 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-image-generic (5.4.0.166.163) ... 2023-11-02T17:36:07.320 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for linux-image-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:36:07.383 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-generic (5.4.0.166.163) ... 2023-11-02T17:36:07.516 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for linux-image-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-02T17:36:07.591 INFO:teuthology.orchestra.run.smithi017.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-02T17:36:07.592 INFO:teuthology.orchestra.run.smithi017.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-166-generic 2023-11-02T17:36:07.596 INFO:teuthology.orchestra.run.smithi069.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-02T17:36:07.596 INFO:teuthology.orchestra.run.smithi069.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-166-generic 2023-11-02T17:36:30.001 INFO:teuthology.orchestra.run.smithi017.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-02T17:36:30.002 INFO:teuthology.orchestra.run.smithi017.stdout:Sourcing file `/etc/default/grub' 2023-11-02T17:36:30.010 INFO:teuthology.orchestra.run.smithi017.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-02T17:36:30.036 INFO:teuthology.orchestra.run.smithi017.stdout:Generating grub configuration file ... 2023-11-02T17:36:30.468 INFO:teuthology.orchestra.run.smithi017.stdout:Found linux image: /boot/vmlinuz-5.4.0-166-generic 2023-11-02T17:36:30.492 INFO:teuthology.orchestra.run.smithi017.stdout:Found initrd image: /boot/initrd.img-5.4.0-166-generic 2023-11-02T17:36:30.718 INFO:teuthology.orchestra.run.smithi069.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-02T17:36:30.718 INFO:teuthology.orchestra.run.smithi069.stdout:Sourcing file `/etc/default/grub' 2023-11-02T17:36:30.726 INFO:teuthology.orchestra.run.smithi069.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-02T17:36:30.735 INFO:teuthology.orchestra.run.smithi069.stdout:Generating grub configuration file ... 2023-11-02T17:36:30.951 INFO:teuthology.orchestra.run.smithi017.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-11-02T17:36:30.962 INFO:teuthology.orchestra.run.smithi017.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-11-02T17:36:31.089 INFO:teuthology.orchestra.run.smithi069.stdout:Found linux image: /boot/vmlinuz-5.4.0-166-generic 2023-11-02T17:36:31.121 INFO:teuthology.orchestra.run.smithi069.stdout:Found initrd image: /boot/initrd.img-5.4.0-166-generic 2023-11-02T17:36:31.517 INFO:teuthology.orchestra.run.smithi069.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-11-02T17:36:31.528 INFO:teuthology.orchestra.run.smithi069.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-11-02T17:36:33.173 INFO:teuthology.orchestra.run.smithi017.stdout:done 2023-11-02T17:36:33.599 DEBUG:teuthology.orchestra.run.smithi017:> dpkg -s linux-image-generic 2023-11-02T17:36:33.656 INFO:teuthology.orchestra.run.smithi017.stdout:Package: linux-image-generic 2023-11-02T17:36:33.656 INFO:teuthology.orchestra.run.smithi017.stdout:Status: install ok installed 2023-11-02T17:36:33.657 INFO:teuthology.orchestra.run.smithi017.stdout:Priority: optional 2023-11-02T17:36:33.657 INFO:teuthology.orchestra.run.smithi017.stdout:Section: kernel 2023-11-02T17:36:33.657 INFO:teuthology.orchestra.run.smithi017.stdout:Installed-Size: 20 2023-11-02T17:36:33.657 INFO:teuthology.orchestra.run.smithi017.stdout:Maintainer: Ubuntu Kernel Team 2023-11-02T17:36:33.657 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: amd64 2023-11-02T17:36:33.658 INFO:teuthology.orchestra.run.smithi017.stdout:Source: linux-meta 2023-11-02T17:36:33.658 INFO:teuthology.orchestra.run.smithi017.stdout:Version: 5.4.0.166.163 2023-11-02T17:36:33.658 INFO:teuthology.orchestra.run.smithi017.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-11-02T17:36:33.658 INFO:teuthology.orchestra.run.smithi017.stdout:Depends: linux-image-5.4.0-166-generic, linux-modules-extra-5.4.0-166-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-02T17:36:33.659 INFO:teuthology.orchestra.run.smithi017.stdout:Recommends: thermald 2023-11-02T17:36:33.659 INFO:teuthology.orchestra.run.smithi017.stdout:Description: Generic Linux kernel image 2023-11-02T17:36:33.659 INFO:teuthology.orchestra.run.smithi017.stdout: This package will always depend on the latest generic kernel image 2023-11-02T17:36:33.659 INFO:teuthology.orchestra.run.smithi017.stdout: available. 2023-11-02T17:36:33.660 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-166-generic 2023-11-02T17:36:33.660 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-02T17:36:33.660 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-02T17:36:33.661 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-11-02T17:36:33.661 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi017.front.sepia.ceph.com, path=None, version=distro) 2023-11-02T17:36:33.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get clean 2023-11-02T17:36:33.748 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get update 2023-11-02T17:36:33.781 INFO:teuthology.orchestra.run.smithi069.stdout:done 2023-11-02T17:36:33.878 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-11-02T17:36:33.998 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-02T17:36:34.080 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-11-02T17:36:34.162 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-11-02T17:36:34.220 DEBUG:teuthology.orchestra.run.smithi069:> dpkg -s linux-image-generic 2023-11-02T17:36:34.279 INFO:teuthology.orchestra.run.smithi069.stdout:Package: linux-image-generic 2023-11-02T17:36:34.279 INFO:teuthology.orchestra.run.smithi069.stdout:Status: install ok installed 2023-11-02T17:36:34.279 INFO:teuthology.orchestra.run.smithi069.stdout:Priority: optional 2023-11-02T17:36:34.279 INFO:teuthology.orchestra.run.smithi069.stdout:Section: kernel 2023-11-02T17:36:34.280 INFO:teuthology.orchestra.run.smithi069.stdout:Installed-Size: 20 2023-11-02T17:36:34.280 INFO:teuthology.orchestra.run.smithi069.stdout:Maintainer: Ubuntu Kernel Team 2023-11-02T17:36:34.280 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: amd64 2023-11-02T17:36:34.280 INFO:teuthology.orchestra.run.smithi069.stdout:Source: linux-meta 2023-11-02T17:36:34.281 INFO:teuthology.orchestra.run.smithi069.stdout:Version: 5.4.0.166.163 2023-11-02T17:36:34.281 INFO:teuthology.orchestra.run.smithi069.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-11-02T17:36:34.281 INFO:teuthology.orchestra.run.smithi069.stdout:Depends: linux-image-5.4.0-166-generic, linux-modules-extra-5.4.0-166-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-02T17:36:34.281 INFO:teuthology.orchestra.run.smithi069.stdout:Recommends: thermald 2023-11-02T17:36:34.282 INFO:teuthology.orchestra.run.smithi069.stdout:Description: Generic Linux kernel image 2023-11-02T17:36:34.282 INFO:teuthology.orchestra.run.smithi069.stdout: This package will always depend on the latest generic kernel image 2023-11-02T17:36:34.282 INFO:teuthology.orchestra.run.smithi069.stdout: available. 2023-11-02T17:36:34.283 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-166-generic 2023-11-02T17:36:34.283 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-02T17:36:34.283 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-02T17:36:34.284 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-11-02T17:36:34.284 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2023-11-02T17:36:34.284 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get clean 2023-11-02T17:36:34.362 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get update 2023-11-02T17:36:34.494 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-11-02T17:36:34.609 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-02T17:36:34.691 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-11-02T17:36:34.773 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-11-02T17:36:35.707 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2023-11-02T17:36:35.733 DEBUG:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-02T17:36:35.806 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2023-11-02T17:36:35.993 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2023-11-02T17:36:35.994 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2023-11-02T17:36:36.215 INFO:teuthology.orchestra.run.smithi017.stdout:linux-image-generic is already the newest version (5.4.0.166.163). 2023-11-02T17:36:36.216 INFO:teuthology.orchestra.run.smithi017.stdout:0 upgraded, 0 newly installed, 0 to remove and 272 not upgraded. 2023-11-02T17:36:36.217 DEBUG:teuthology.orchestra.run.smithi017:> dpkg -s linux-image-generic 2023-11-02T17:36:36.238 INFO:teuthology.orchestra.run.smithi017.stdout:Package: linux-image-generic 2023-11-02T17:36:36.238 INFO:teuthology.orchestra.run.smithi017.stdout:Status: install ok installed 2023-11-02T17:36:36.238 INFO:teuthology.orchestra.run.smithi017.stdout:Priority: optional 2023-11-02T17:36:36.238 INFO:teuthology.orchestra.run.smithi017.stdout:Section: kernel 2023-11-02T17:36:36.239 INFO:teuthology.orchestra.run.smithi017.stdout:Installed-Size: 20 2023-11-02T17:36:36.239 INFO:teuthology.orchestra.run.smithi017.stdout:Maintainer: Ubuntu Kernel Team 2023-11-02T17:36:36.239 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: amd64 2023-11-02T17:36:36.239 INFO:teuthology.orchestra.run.smithi017.stdout:Source: linux-meta 2023-11-02T17:36:36.239 INFO:teuthology.orchestra.run.smithi017.stdout:Version: 5.4.0.166.163 2023-11-02T17:36:36.240 INFO:teuthology.orchestra.run.smithi017.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-11-02T17:36:36.240 INFO:teuthology.orchestra.run.smithi017.stdout:Depends: linux-image-5.4.0-166-generic, linux-modules-extra-5.4.0-166-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-02T17:36:36.240 INFO:teuthology.orchestra.run.smithi017.stdout:Recommends: thermald 2023-11-02T17:36:36.240 INFO:teuthology.orchestra.run.smithi017.stdout:Description: Generic Linux kernel image 2023-11-02T17:36:36.240 INFO:teuthology.orchestra.run.smithi017.stdout: This package will always depend on the latest generic kernel image 2023-11-02T17:36:36.241 INFO:teuthology.orchestra.run.smithi017.stdout: available. 2023-11-02T17:36:36.241 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2023-11-02T17:36:36.246 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.hq4aDrSiWf 2023-11-02T17:36:36.247 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /boot/grub/grub.cfg /tmp/tmp.hq4aDrSiWf 2023-11-02T17:36:36.250 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-11-02T17:36:36.261 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.hq4aDrSiWf 2023-11-02T17:36:36.276 DEBUG:teuthology.orchestra.run.smithi069:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-02T17:36:36.339 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.hq4aDrSiWf is 9KB 2023-11-02T17:36:36.345 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-11-02T17:36:36.353 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.hq4aDrSiWf 2023-11-02T17:36:36.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-02T17:36:36.417 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:36:36.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-02T17:36:36.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-02T17:36:36.452 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-166-generic 2023-11-02T17:36:36.452 DEBUG:teuthology.orchestra.run.smithi017:> sudo update-grub 2023-11-02T17:36:36.532 INFO:teuthology.orchestra.run.smithi069.stdout:Building dependency tree... 2023-11-02T17:36:36.533 INFO:teuthology.orchestra.run.smithi069.stdout:Reading state information... 2023-11-02T17:36:36.746 INFO:teuthology.orchestra.run.smithi069.stdout:linux-image-generic is already the newest version (5.4.0.166.163). 2023-11-02T17:36:36.746 INFO:teuthology.orchestra.run.smithi069.stdout:0 upgraded, 0 newly installed, 0 to remove and 272 not upgraded. 2023-11-02T17:36:36.748 DEBUG:teuthology.orchestra.run.smithi069:> dpkg -s linux-image-generic 2023-11-02T17:36:36.769 INFO:teuthology.orchestra.run.smithi069.stdout:Package: linux-image-generic 2023-11-02T17:36:36.769 INFO:teuthology.orchestra.run.smithi069.stdout:Status: install ok installed 2023-11-02T17:36:36.769 INFO:teuthology.orchestra.run.smithi069.stdout:Priority: optional 2023-11-02T17:36:36.769 INFO:teuthology.orchestra.run.smithi069.stdout:Section: kernel 2023-11-02T17:36:36.770 INFO:teuthology.orchestra.run.smithi069.stdout:Installed-Size: 20 2023-11-02T17:36:36.770 INFO:teuthology.orchestra.run.smithi069.stdout:Maintainer: Ubuntu Kernel Team 2023-11-02T17:36:36.770 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: amd64 2023-11-02T17:36:36.770 INFO:teuthology.orchestra.run.smithi069.stdout:Source: linux-meta 2023-11-02T17:36:36.770 INFO:teuthology.orchestra.run.smithi069.stdout:Version: 5.4.0.166.163 2023-11-02T17:36:36.771 INFO:teuthology.orchestra.run.smithi069.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-11-02T17:36:36.771 INFO:teuthology.orchestra.run.smithi069.stdout:Depends: linux-image-5.4.0-166-generic, linux-modules-extra-5.4.0-166-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-02T17:36:36.771 INFO:teuthology.orchestra.run.smithi069.stdout:Recommends: thermald 2023-11-02T17:36:36.771 INFO:teuthology.orchestra.run.smithi069.stdout:Description: Generic Linux kernel image 2023-11-02T17:36:36.771 INFO:teuthology.orchestra.run.smithi069.stdout: This package will always depend on the latest generic kernel image 2023-11-02T17:36:36.772 INFO:teuthology.orchestra.run.smithi069.stdout: available. 2023-11-02T17:36:36.772 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2023-11-02T17:36:36.818 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.Lbb88lJjQ1 2023-11-02T17:36:36.819 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Lbb88lJjQ1 2023-11-02T17:36:36.833 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.Lbb88lJjQ1 2023-11-02T17:36:36.883 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.Lbb88lJjQ1 is 9KB 2023-11-02T17:36:36.896 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.Lbb88lJjQ1 2023-11-02T17:36:36.903 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-02T17:36:36.959 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:36:36.960 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-02T17:36:36.960 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-02T17:36:36.996 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-166-generic 2023-11-02T17:36:36.996 DEBUG:teuthology.orchestra.run.smithi069:> sudo update-grub 2023-11-02T17:36:38.037 INFO:teuthology.orchestra.run.smithi017.stderr:Sourcing file `/etc/default/grub' 2023-11-02T17:36:38.068 INFO:teuthology.orchestra.run.smithi017.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-02T17:36:38.076 INFO:teuthology.orchestra.run.smithi017.stderr:Generating grub configuration file ... 2023-11-02T17:36:38.401 INFO:teuthology.orchestra.run.smithi069.stderr:Sourcing file `/etc/default/grub' 2023-11-02T17:36:38.402 INFO:teuthology.orchestra.run.smithi017.stderr:Found linux image: /boot/vmlinuz-5.4.0-166-generic 2023-11-02T17:36:38.408 INFO:teuthology.orchestra.run.smithi069.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-02T17:36:38.413 INFO:teuthology.orchestra.run.smithi017.stderr:Found initrd image: /boot/initrd.img-5.4.0-166-generic 2023-11-02T17:36:38.419 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2023-11-02T17:36:38.706 INFO:teuthology.orchestra.run.smithi069.stderr:Found linux image: /boot/vmlinuz-5.4.0-166-generic 2023-11-02T17:36:38.718 INFO:teuthology.orchestra.run.smithi069.stderr:Found initrd image: /boot/initrd.img-5.4.0-166-generic 2023-11-02T17:36:38.772 INFO:teuthology.orchestra.run.smithi017.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-11-02T17:36:38.784 INFO:teuthology.orchestra.run.smithi017.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-11-02T17:36:39.111 INFO:teuthology.orchestra.run.smithi069.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-11-02T17:36:39.123 INFO:teuthology.orchestra.run.smithi069.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-11-02T17:36:40.012 INFO:teuthology.orchestra.run.smithi017.stderr:done 2023-11-02T17:36:40.014 DEBUG:teuthology.orchestra.run.smithi017:> sudo shutdown -r now 2023-11-02T17:36:40.155 INFO:teuthology.orchestra.run.smithi069.stderr:done 2023-11-02T17:36:40.156 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2023-11-02T17:37:10.030 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-11-02T17:37:10.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-11-02T17:37:10.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T17:37:10.159 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-11-02T17:37:10.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-11-02T17:37:10.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T17:37:28.442 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2023-11-02T17:37:28.698 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2023-11-02T17:37:37.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-11-02T17:37:37.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T17:37:37.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-11-02T17:37:37.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T17:37:40.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2023-11-02T17:37:52.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-11-02T17:37:52.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T17:38:24.460 DEBUG:teuthology.orchestra.run.smithi069:> true 2023-11-02T17:38:25.016 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-11-02T17:38:25.016 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "5.4.0-166-generic"... 2023-11-02T17:38:25.016 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-11-02T17:38:25.023 INFO:teuthology.orchestra.run.smithi069.stdout:5.4.0-166-generic 2023-11-02T17:38:25.024 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-166-generic vs 5.4.0-166-generic 2023-11-02T17:38:25.024 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-02T17:38:25.024 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-02T17:38:26.026 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-11-02T17:38:26.026 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-02T17:38:26.106 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2023-11-02T17:38:26.134 DEBUG:teuthology.parallel:result is None 2023-11-02T17:38:37.456 DEBUG:teuthology.orchestra.remote:timed out 2023-11-02T17:38:49.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-11-02T17:38:49.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T17:38:49.663 DEBUG:teuthology.orchestra.run.smithi017:> true 2023-11-02T17:38:50.216 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-11-02T17:38:50.217 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "5.4.0-166-generic"... 2023-11-02T17:38:50.217 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2023-11-02T17:38:50.224 INFO:teuthology.orchestra.run.smithi017.stdout:5.4.0-166-generic 2023-11-02T17:38:50.225 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-166-generic vs 5.4.0-166-generic 2023-11-02T17:38:50.225 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-02T17:38:50.225 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-02T17:38:51.226 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-11-02T17:38:51.227 DEBUG:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-02T17:38:51.325 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2023-11-02T17:38:51.349 DEBUG:teuthology.parallel:result is None 2023-11-02T17:38:51.349 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-02T17:38:51.358 INFO:teuthology.task.internal:Creating test directory... 2023-11-02T17:38:51.359 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-02T17:38:51.361 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-02T17:38:51.368 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-02T17:38:51.443 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-02T17:38:51.457 INFO:teuthology.task.internal:Creating archive directory... 2023-11-02T17:38:51.457 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-02T17:38:51.460 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-02T17:38:51.482 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-02T17:38:51.688 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-02T17:38:51.689 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-02T17:38:51.691 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-02T17:38:51.707 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T17:38:51.718 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T17:38:51.720 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T17:38:51.731 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T17:38:51.733 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-02T17:38:51.811 INFO:teuthology.task.internal:Configuring sudo... 2023-11-02T17:38:51.811 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-02T17:38:51.814 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-02T17:38:51.831 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-02T17:38:51.867 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-02T17:38:51.868 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-02T17:38:51.872 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-02T17:38:51.879 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-02T17:38:51.887 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-02T17:38:51.892 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:38:51.892 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-02T17:38:51.961 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-02T17:38:51.968 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-02T17:38:52.019 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:38:52.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-02T17:38:52.044 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2023-11-02T17:38:52.047 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2023-11-02T17:38:52.108 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-02T17:38:52.119 INFO:teuthology.task.internal:Starting timer... 2023-11-02T17:38:52.119 INFO:teuthology.run_tasks:Running task pcp... 2023-11-02T17:38:52.222 INFO:teuthology.run_tasks:Running task selinux... 2023-11-02T17:38:52.269 DEBUG:teuthology.task.selinux:Excluding smithi017: OS 'ubuntu' does not support SELinux 2023-11-02T17:38:52.270 DEBUG:teuthology.task.selinux:Excluding smithi069: OS 'ubuntu' does not support SELinux 2023-11-02T17:38:52.270 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-02T17:38:52.270 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-11-02T17:38:52.271 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-02T17:38:52.271 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-02T17:38:52.281 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-02T17:38:52.287 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-02T17:38:52.403 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-02T17:38:52.421 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-02T17:38:52.422 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi017.front.sepia.ceph.com,smithi069.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-02T17:44:46.344 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi069.front.sepia.ceph.com')] 2023-11-02T17:44:46.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-11-02T17:44:46.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T17:44:46.392 DEBUG:teuthology.orchestra.run.smithi017:> true 2023-11-02T17:44:46.516 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-11-02T17:44:46.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-11-02T17:44:46.517 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T17:44:46.564 DEBUG:teuthology.orchestra.run.smithi069:> true 2023-11-02T17:44:46.683 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-11-02T17:44:46.684 INFO:teuthology.run_tasks:Running task clock... 2023-11-02T17:44:46.696 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-02T17:44:46.697 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-02T17:44:46.697 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-02T17:44:46.699 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-02T17:44:46.700 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-02T17:44:46.728 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:46 ntpd[22789]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-11-02T17:44:46.729 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:46 ntpd[22789]: Command line: ntpd -gq 2023-11-02T17:44:46.730 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:46 ntpd[22789]: proto: precision = 0.120 usec (-23) 2023-11-02T17:44:46.730 INFO:teuthology.orchestra.run.smithi017.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-02T17:44:46.731 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:46 ntpd[22789]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-02T17:44:46.731 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:46 ntpd[22789]: restrict ::: KOD does nothing without LIMITED. 2023-11-02T17:44:46.732 INFO:teuthology.orchestra.run.smithi017.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-02T17:44:46.733 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:46 ntpd[22789]: Listen and drop on 0 v6wildcard [::]:123 2023-11-02T17:44:46.733 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:46 ntpd[22789]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-02T17:44:46.733 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:46 ntpd[22789]: Listen normally on 2 lo 127.0.0.1:123 2023-11-02T17:44:46.733 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:46 ntpd[22789]: Listen normally on 3 ens1f0 172.21.15.17:123 2023-11-02T17:44:46.733 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:46 ntpd[22789]: Listen normally on 4 lo [::1]:123 2023-11-02T17:44:46.734 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:46 ntpd[22789]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe59:d992%4]:123 2023-11-02T17:44:46.734 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:46 ntpd[22789]: Listening on routing socket on fd #22 for interface updates 2023-11-02T17:44:46.758 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:46 ntpd[22896]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-11-02T17:44:46.759 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:46 ntpd[22896]: Command line: ntpd -gq 2023-11-02T17:44:46.759 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:46 ntpd[22896]: proto: precision = 0.074 usec (-24) 2023-11-02T17:44:46.760 INFO:teuthology.orchestra.run.smithi069.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-02T17:44:46.760 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:46 ntpd[22896]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-02T17:44:46.761 INFO:teuthology.orchestra.run.smithi069.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-02T17:44:46.762 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:46 ntpd[22896]: restrict ::: KOD does nothing without LIMITED. 2023-11-02T17:44:46.762 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:46 ntpd[22896]: Listen and drop on 0 v6wildcard [::]:123 2023-11-02T17:44:46.762 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:46 ntpd[22896]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-02T17:44:46.762 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:46 ntpd[22896]: Listen normally on 2 lo 127.0.0.1:123 2023-11-02T17:44:46.763 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:46 ntpd[22896]: Listen normally on 3 enp3s0f1 172.21.15.69:123 2023-11-02T17:44:46.763 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:46 ntpd[22896]: Listen normally on 4 lo [::1]:123 2023-11-02T17:44:46.763 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:46 ntpd[22896]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d737%5]:123 2023-11-02T17:44:46.763 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:46 ntpd[22896]: Listening on routing socket on fd #22 for interface updates 2023-11-02T17:44:47.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:47 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:47.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:47 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:47.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:47 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:47.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:47 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:47.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:47 ntpd[22789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:47.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:47 ntpd[22789]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:47.759 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:47 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:47.759 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:47 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:47.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:47 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:47.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:47 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:47.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:47 ntpd[22896]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:47.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:47 ntpd[22896]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:49.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:49 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:49.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:49 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:49.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:49 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:49.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:49 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:49.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:49 ntpd[22789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:49.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:49 ntpd[22789]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:49.759 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:49 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:49.759 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:49 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:49.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:49 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:49.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:49 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:49.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:49 ntpd[22896]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:49.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:49 ntpd[22896]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:50.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:50.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:50.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:50.732 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:50.732 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:50.732 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:50.732 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:50.733 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:50.733 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:50.733 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:50.733 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:50.734 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:50.734 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:50.734 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:50 ntpd[22789]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:50.759 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:50 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:50.759 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:50 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:50.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:50 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:50.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:50 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:50.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:50 ntpd[22896]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:50.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:50 ntpd[22896]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:51.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:51 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:51.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:51 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:51.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:51 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:51.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:51 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:51.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:51 ntpd[22789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:51.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:51 ntpd[22789]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:51.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:51.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:51.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:51.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:51.761 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:51.761 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:51.761 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:51.761 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:51.761 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:51.762 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:51.762 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:51.762 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:51.762 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:51.762 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:51 ntpd[22896]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:52.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:52 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:52.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:52 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:52.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:52 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:52.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:52 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:52.732 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:52 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:52.732 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:52 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:52.732 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:52 ntpd[22789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:52.732 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:52 ntpd[22789]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:52.732 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:52 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:52.733 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:52 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:52.733 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:52 ntpd[22789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:52.733 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:52 ntpd[22789]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:52.759 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:52 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:52.759 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:52 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:52.759 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:52 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:52.759 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:52 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:52.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:52 ntpd[22896]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:52.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:52 ntpd[22896]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:53.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:53 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:53.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:53 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:53.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:53 ntpd[22789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:53.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:53 ntpd[22789]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:53.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:53 ntpd[22789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:53.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:53 ntpd[22789]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:53.732 INFO:teuthology.orchestra.run.smithi017.stdout: 2 Nov 17:44:53 ntpd[22789]: ntpd: time slew +0.004378 s 2023-11-02T17:44:53.732 INFO:teuthology.orchestra.run.smithi017.stdout:ntpd: time slew +0.004378s 2023-11-02T17:44:53.733 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:53 ntpd[22789]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-02T17:44:53.733 INFO:teuthology.orchestra.run.smithi017.stderr: 2 Nov 17:44:53 ntpd[22789]: can't open /var/log/ntpstats/loopstats.20231102: Permission denied 2023-11-02T17:44:53.759 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:53.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:53.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:53.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:53.760 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:53.761 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:53.761 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:53.761 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:53.761 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-02T17:44:53.761 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: can't open /var/log/ntpstats/rawstats.20231102: Permission denied 2023-11-02T17:44:53.762 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-02T17:44:53.762 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: can't open /var/log/ntpstats/peerstats.20231102: Permission denied 2023-11-02T17:44:53.762 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Nov 17:44:53 ntpd[22896]: ntpd: time slew +0.000023 s 2023-11-02T17:44:53.763 INFO:teuthology.orchestra.run.smithi069.stdout:ntpd: time slew +0.000023s 2023-11-02T17:44:53.763 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-02T17:44:53.763 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Nov 17:44:53 ntpd[22896]: can't open /var/log/ntpstats/loopstats.20231102: Permission denied 2023-11-02T17:44:53.799 INFO:teuthology.orchestra.run.smithi017.stdout: remote refid st t when poll reach delay offset jitter 2023-11-02T17:44:53.799 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================== 2023-11-02T17:44:53.799 INFO:teuthology.orchestra.run.smithi017.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-02T17:44:53.800 INFO:teuthology.orchestra.run.smithi017.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-02T17:44:53.800 INFO:teuthology.orchestra.run.smithi017.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-02T17:44:53.800 INFO:teuthology.orchestra.run.smithi017.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-02T17:44:53.829 INFO:teuthology.orchestra.run.smithi069.stdout: remote refid st t when poll reach delay offset jitter 2023-11-02T17:44:53.830 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================== 2023-11-02T17:44:53.830 INFO:teuthology.orchestra.run.smithi069.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-02T17:44:53.830 INFO:teuthology.orchestra.run.smithi069.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-02T17:44:53.830 INFO:teuthology.orchestra.run.smithi069.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-02T17:44:53.831 INFO:teuthology.orchestra.run.smithi069.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-02T17:44:53.832 INFO:teuthology.run_tasks:Running task install... 2023-11-02T17:44:53.840 DEBUG:teuthology.task.install:project ceph 2023-11-02T17:44:53.841 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec'}} 2023-11-02T17:44:53.841 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec'} 2023-11-02T17:44:53.842 INFO:teuthology.task.install:Using flavor: default 2023-11-02T17:44:53.848 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-02T17:44:53.848 INFO:teuthology.task.install:extra packages: [] 2023-11-02T17:44:53.849 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-key list | grep Ceph 2023-11-02T17:44:53.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-key list | grep Ceph 2023-11-02T17:44:53.946 INFO:teuthology.orchestra.run.smithi069.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-11-02T17:44:53.957 INFO:teuthology.orchestra.run.smithi017.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-11-02T17:44:54.004 INFO:teuthology.orchestra.run.smithi069.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-02T17:44:54.005 INFO:teuthology.orchestra.run.smithi069.stdout:uid [ unknown] Ceph.com (release key) 2023-11-02T17:44:54.006 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-11-02T17:44:54.006 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=b8ceb241941938aa7529adcc4b99d42d196489ec 2023-11-02T17:44:54.018 INFO:teuthology.orchestra.run.smithi017.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-02T17:44:54.018 INFO:teuthology.orchestra.run.smithi017.stdout:uid [ unknown] Ceph.com (release key) 2023-11-02T17:44:54.019 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-11-02T17:44:54.020 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=b8ceb241941938aa7529adcc4b99d42d196489ec 2023-11-02T17:44:54.224 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default/ 2023-11-02T17:44:54.224 INFO:teuthology.task.install.deb:Package version is 18.2.0-1164-gb8ceb241-1focal 2023-11-02T17:44:54.234 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default/ 2023-11-02T17:44:54.235 INFO:teuthology.task.install.deb:Package version is 18.2.0-1164-gb8ceb241-1focal 2023-11-02T17:44:54.401 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:44:54.401 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-02T17:44:54.406 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:44:54.407 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-02T17:44:54.418 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get update 2023-11-02T17:44:54.423 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get update 2023-11-02T17:44:54.561 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-11-02T17:44:54.562 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-02T17:44:54.570 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-11-02T17:44:54.587 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-11-02T17:44:54.610 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-11-02T17:44:54.681 INFO:teuthology.orchestra.run.smithi017.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal InRelease 2023-11-02T17:44:54.720 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-11-02T17:44:54.724 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-02T17:44:54.725 INFO:teuthology.orchestra.run.smithi069.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal InRelease 2023-11-02T17:44:54.758 INFO:teuthology.orchestra.run.smithi017.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal Release.gpg 2023-11-02T17:44:54.761 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-11-02T17:44:54.806 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-11-02T17:44:54.812 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 Packages [17.3 kB] 2023-11-02T17:44:54.813 INFO:teuthology.orchestra.run.smithi069.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal Release.gpg 2023-11-02T17:44:54.856 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main i386 Packages [3,406 B] 2023-11-02T17:44:54.882 INFO:teuthology.orchestra.run.smithi017.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main i386 Packages [3,406 B] 2023-11-02T17:44:54.896 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-11-02T17:44:54.913 INFO:teuthology.orchestra.run.smithi069.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 Packages [17.3 kB] 2023-11-02T17:44:55.234 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 28.4 kB in 1s (43.5 kB/s) 2023-11-02T17:44:55.259 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 28.4 kB in 1s (44.9 kB/s) 2023-11-02T17:44:56.313 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-11-02T17:44:56.327 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2023-11-02T17:44:56.339 DEBUG:teuthology.orchestra.run.smithi069:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.0-1164-gb8ceb241-1focal cephadm=18.2.0-1164-gb8ceb241-1focal ceph-mds=18.2.0-1164-gb8ceb241-1focal ceph-mgr=18.2.0-1164-gb8ceb241-1focal ceph-common=18.2.0-1164-gb8ceb241-1focal ceph-fuse=18.2.0-1164-gb8ceb241-1focal ceph-test=18.2.0-1164-gb8ceb241-1focal ceph-volume=18.2.0-1164-gb8ceb241-1focal radosgw=18.2.0-1164-gb8ceb241-1focal python3-rados=18.2.0-1164-gb8ceb241-1focal python3-rgw=18.2.0-1164-gb8ceb241-1focal python3-cephfs=18.2.0-1164-gb8ceb241-1focal python3-rbd=18.2.0-1164-gb8ceb241-1focal libcephfs2=18.2.0-1164-gb8ceb241-1focal libcephfs-dev=18.2.0-1164-gb8ceb241-1focal librados2=18.2.0-1164-gb8ceb241-1focal librbd1=18.2.0-1164-gb8ceb241-1focal rbd-fuse=18.2.0-1164-gb8ceb241-1focal 2023-11-02T17:44:56.353 DEBUG:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.0-1164-gb8ceb241-1focal cephadm=18.2.0-1164-gb8ceb241-1focal ceph-mds=18.2.0-1164-gb8ceb241-1focal ceph-mgr=18.2.0-1164-gb8ceb241-1focal ceph-common=18.2.0-1164-gb8ceb241-1focal ceph-fuse=18.2.0-1164-gb8ceb241-1focal ceph-test=18.2.0-1164-gb8ceb241-1focal ceph-volume=18.2.0-1164-gb8ceb241-1focal radosgw=18.2.0-1164-gb8ceb241-1focal python3-rados=18.2.0-1164-gb8ceb241-1focal python3-rgw=18.2.0-1164-gb8ceb241-1focal python3-cephfs=18.2.0-1164-gb8ceb241-1focal python3-rbd=18.2.0-1164-gb8ceb241-1focal libcephfs2=18.2.0-1164-gb8ceb241-1focal libcephfs-dev=18.2.0-1164-gb8ceb241-1focal librados2=18.2.0-1164-gb8ceb241-1focal librbd1=18.2.0-1164-gb8ceb241-1focal rbd-fuse=18.2.0-1164-gb8ceb241-1focal 2023-11-02T17:44:56.405 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-11-02T17:44:56.430 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2023-11-02T17:44:56.587 INFO:teuthology.orchestra.run.smithi069.stdout:Building dependency tree... 2023-11-02T17:44:56.588 INFO:teuthology.orchestra.run.smithi069.stdout:Reading state information... 2023-11-02T17:44:56.640 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2023-11-02T17:44:56.641 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2023-11-02T17:44:56.760 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages were automatically installed and are no longer required: 2023-11-02T17:44:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-11-02T17:44:56.762 INFO:teuthology.orchestra.run.smithi069.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-02T17:44:56.762 INFO:teuthology.orchestra.run.smithi069.stdout:The following additional packages will be installed: 2023-11-02T17:44:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-02T17:44:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-11-02T17:44:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-11-02T17:44:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-11-02T17:44:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-11-02T17:44:56.764 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-11-02T17:44:56.764 INFO:teuthology.orchestra.run.smithi069.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2023-11-02T17:44:56.764 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2023-11-02T17:44:56.764 INFO:teuthology.orchestra.run.smithi069.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-11-02T17:44:56.765 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2023-11-02T17:44:56.765 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-11-02T17:44:56.765 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-11-02T17:44:56.765 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-11-02T17:44:56.766 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-11-02T17:44:56.766 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-11-02T17:44:56.766 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-11-02T17:44:56.766 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2023-11-02T17:44:56.766 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-11-02T17:44:56.767 INFO:teuthology.orchestra.run.smithi069.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-11-02T17:44:56.767 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-11-02T17:44:56.767 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-11-02T17:44:56.767 INFO:teuthology.orchestra.run.smithi069.stdout: qttranslations5-l10n socat xmlstarlet 2023-11-02T17:44:56.767 INFO:teuthology.orchestra.run.smithi069.stdout:Suggested packages: 2023-11-02T17:44:56.768 INFO:teuthology.orchestra.run.smithi069.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-11-02T17:44:56.768 INFO:teuthology.orchestra.run.smithi069.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-11-02T17:44:56.768 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-11-02T17:44:56.768 INFO:teuthology.orchestra.run.smithi069.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-11-02T17:44:56.768 INFO:teuthology.orchestra.run.smithi069.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-11-02T17:44:56.769 INFO:teuthology.orchestra.run.smithi069.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-11-02T17:44:56.769 INFO:teuthology.orchestra.run.smithi069.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-11-02T17:44:56.769 INFO:teuthology.orchestra.run.smithi069.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-11-02T17:44:56.769 INFO:teuthology.orchestra.run.smithi069.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-11-02T17:44:56.769 INFO:teuthology.orchestra.run.smithi069.stdout:Recommended packages: 2023-11-02T17:44:56.770 INFO:teuthology.orchestra.run.smithi069.stdout: btrfs-tools 2023-11-02T17:44:56.833 INFO:teuthology.orchestra.run.smithi017.stdout:The following packages were automatically installed and are no longer required: 2023-11-02T17:44:56.834 INFO:teuthology.orchestra.run.smithi017.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-11-02T17:44:56.834 INFO:teuthology.orchestra.run.smithi017.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-02T17:44:56.835 INFO:teuthology.orchestra.run.smithi017.stdout:The following additional packages will be installed: 2023-11-02T17:44:56.835 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-02T17:44:56.835 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-11-02T17:44:56.835 INFO:teuthology.orchestra.run.smithi017.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-11-02T17:44:56.835 INFO:teuthology.orchestra.run.smithi017.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-11-02T17:44:56.836 INFO:teuthology.orchestra.run.smithi017.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-11-02T17:44:56.837 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-11-02T17:44:56.837 INFO:teuthology.orchestra.run.smithi017.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2023-11-02T17:44:56.837 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2023-11-02T17:44:56.837 INFO:teuthology.orchestra.run.smithi017.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-11-02T17:44:56.838 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2023-11-02T17:44:56.838 INFO:teuthology.orchestra.run.smithi017.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-11-02T17:44:56.838 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-11-02T17:44:56.838 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-11-02T17:44:56.838 INFO:teuthology.orchestra.run.smithi017.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-11-02T17:44:56.839 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-11-02T17:44:56.839 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-11-02T17:44:56.839 INFO:teuthology.orchestra.run.smithi017.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2023-11-02T17:44:56.839 INFO:teuthology.orchestra.run.smithi017.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-11-02T17:44:56.839 INFO:teuthology.orchestra.run.smithi017.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-11-02T17:44:56.840 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-11-02T17:44:56.840 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-11-02T17:44:56.840 INFO:teuthology.orchestra.run.smithi017.stdout: qttranslations5-l10n socat xmlstarlet 2023-11-02T17:44:56.840 INFO:teuthology.orchestra.run.smithi017.stdout:Suggested packages: 2023-11-02T17:44:56.840 INFO:teuthology.orchestra.run.smithi017.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-11-02T17:44:56.841 INFO:teuthology.orchestra.run.smithi017.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-11-02T17:44:56.841 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-11-02T17:44:56.841 INFO:teuthology.orchestra.run.smithi017.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-11-02T17:44:56.841 INFO:teuthology.orchestra.run.smithi017.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-11-02T17:44:56.841 INFO:teuthology.orchestra.run.smithi017.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-11-02T17:44:56.841 INFO:teuthology.orchestra.run.smithi017.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-11-02T17:44:56.842 INFO:teuthology.orchestra.run.smithi017.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-11-02T17:44:56.842 INFO:teuthology.orchestra.run.smithi017.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-11-02T17:44:56.842 INFO:teuthology.orchestra.run.smithi017.stdout:Recommended packages: 2023-11-02T17:44:56.842 INFO:teuthology.orchestra.run.smithi017.stdout: btrfs-tools 2023-11-02T17:44:56.881 INFO:teuthology.orchestra.run.smithi069.stdout:The following NEW packages will be installed: 2023-11-02T17:44:56.881 INFO:teuthology.orchestra.run.smithi069.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-02T17:44:56.882 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-02T17:44:56.882 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-11-02T17:44:56.882 INFO:teuthology.orchestra.run.smithi069.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-11-02T17:44:56.882 INFO:teuthology.orchestra.run.smithi069.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-11-02T17:44:56.882 INFO:teuthology.orchestra.run.smithi069.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-11-02T17:44:56.882 INFO:teuthology.orchestra.run.smithi069.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-11-02T17:44:56.884 INFO:teuthology.orchestra.run.smithi069.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2023-11-02T17:44:56.884 INFO:teuthology.orchestra.run.smithi069.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2023-11-02T17:44:56.884 INFO:teuthology.orchestra.run.smithi069.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-11-02T17:44:56.884 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-02T17:44:56.884 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-11-02T17:44:56.885 INFO:teuthology.orchestra.run.smithi069.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-11-02T17:44:56.885 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-11-02T17:44:56.885 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-11-02T17:44:56.885 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-11-02T17:44:56.885 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-11-02T17:44:56.886 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-11-02T17:44:56.886 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-11-02T17:44:56.886 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-11-02T17:44:56.886 INFO:teuthology.orchestra.run.smithi069.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-11-02T17:44:56.886 INFO:teuthology.orchestra.run.smithi069.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-11-02T17:44:56.887 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-11-02T17:44:56.887 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2023-11-02T17:44:56.887 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet 2023-11-02T17:44:56.887 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages will be upgraded: 2023-11-02T17:44:56.887 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 librbd1 2023-11-02T17:44:56.952 INFO:teuthology.orchestra.run.smithi017.stdout:The following NEW packages will be installed: 2023-11-02T17:44:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-02T17:44:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-02T17:44:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-11-02T17:44:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-11-02T17:44:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-11-02T17:44:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-11-02T17:44:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-11-02T17:44:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2023-11-02T17:44:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2023-11-02T17:44:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-11-02T17:44:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-02T17:44:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-11-02T17:44:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-11-02T17:44:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-11-02T17:44:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-11-02T17:44:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-11-02T17:44:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-11-02T17:44:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-11-02T17:44:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-11-02T17:44:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-11-02T17:44:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-11-02T17:44:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-11-02T17:44:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-11-02T17:44:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2023-11-02T17:44:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: xmlstarlet 2023-11-02T17:44:56.959 INFO:teuthology.orchestra.run.smithi017.stdout:The following packages will be upgraded: 2023-11-02T17:44:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: librados2 librbd1 2023-11-02T17:44:57.018 INFO:teuthology.orchestra.run.smithi017.stdout:2 upgraded, 110 newly installed, 0 to remove and 254 not upgraded. 2023-11-02T17:44:57.018 INFO:teuthology.orchestra.run.smithi017.stdout:Need to get 208 MB of archives. 2023-11-02T17:44:57.018 INFO:teuthology.orchestra.run.smithi017.stdout:After this operation, 1,029 MB of additional disk space will be used. 2023-11-02T17:44:57.019 INFO:teuthology.orchestra.run.smithi017.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-11-02T17:44:57.047 INFO:teuthology.orchestra.run.smithi069.stdout:2 upgraded, 110 newly installed, 0 to remove and 254 not upgraded. 2023-11-02T17:44:57.047 INFO:teuthology.orchestra.run.smithi069.stdout:Need to get 208 MB of archives. 2023-11-02T17:44:57.047 INFO:teuthology.orchestra.run.smithi069.stdout:After this operation, 1,029 MB of additional disk space will be used. 2023-11-02T17:44:57.048 INFO:teuthology.orchestra.run.smithi069.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.0-1164-gb8ceb241-1focal [3,438 kB] 2023-11-02T17:44:57.075 INFO:teuthology.orchestra.run.smithi069.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-11-02T17:44:57.099 INFO:teuthology.orchestra.run.smithi017.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-11-02T17:44:57.102 INFO:teuthology.orchestra.run.smithi017.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-11-02T17:44:57.120 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-11-02T17:44:57.171 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-11-02T17:44:57.173 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-11-02T17:44:57.179 INFO:teuthology.orchestra.run.smithi017.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-11-02T17:44:57.181 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-11-02T17:44:57.181 INFO:teuthology.orchestra.run.smithi017.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-11-02T17:44:57.182 INFO:teuthology.orchestra.run.smithi017.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-11-02T17:44:57.183 INFO:teuthology.orchestra.run.smithi017.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-11-02T17:44:57.183 INFO:teuthology.orchestra.run.smithi017.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.0-1164-gb8ceb241-1focal [3,438 kB] 2023-11-02T17:44:57.186 INFO:teuthology.orchestra.run.smithi017.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-11-02T17:44:57.186 INFO:teuthology.orchestra.run.smithi017.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-11-02T17:44:57.188 INFO:teuthology.orchestra.run.smithi017.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-11-02T17:44:57.205 INFO:teuthology.orchestra.run.smithi017.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-11-02T17:44:57.205 INFO:teuthology.orchestra.run.smithi017.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-11-02T17:44:57.206 INFO:teuthology.orchestra.run.smithi017.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-11-02T17:44:57.207 INFO:teuthology.orchestra.run.smithi017.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-02T17:44:57.208 INFO:teuthology.orchestra.run.smithi017.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-11-02T17:44:57.208 INFO:teuthology.orchestra.run.smithi017.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-11-02T17:44:57.209 INFO:teuthology.orchestra.run.smithi017.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-11-02T17:44:57.209 INFO:teuthology.orchestra.run.smithi017.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-11-02T17:44:57.210 INFO:teuthology.orchestra.run.smithi017.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-11-02T17:44:57.222 INFO:teuthology.orchestra.run.smithi017.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-11-02T17:44:57.229 INFO:teuthology.orchestra.run.smithi017.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-11-02T17:44:57.235 INFO:teuthology.orchestra.run.smithi017.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-11-02T17:44:57.235 INFO:teuthology.orchestra.run.smithi017.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-11-02T17:44:57.236 INFO:teuthology.orchestra.run.smithi017.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-11-02T17:44:57.237 INFO:teuthology.orchestra.run.smithi017.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-11-02T17:44:57.238 INFO:teuthology.orchestra.run.smithi017.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-11-02T17:44:57.241 INFO:teuthology.orchestra.run.smithi017.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-11-02T17:44:57.244 INFO:teuthology.orchestra.run.smithi017.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-11-02T17:44:57.244 INFO:teuthology.orchestra.run.smithi017.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2023-11-02T17:44:57.256 INFO:teuthology.orchestra.run.smithi017.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-11-02T17:44:57.257 INFO:teuthology.orchestra.run.smithi017.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-11-02T17:44:57.257 INFO:teuthology.orchestra.run.smithi017.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-11-02T17:44:57.258 INFO:teuthology.orchestra.run.smithi017.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-11-02T17:44:57.258 INFO:teuthology.orchestra.run.smithi017.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-11-02T17:44:57.274 INFO:teuthology.orchestra.run.smithi017.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-11-02T17:44:57.275 INFO:teuthology.orchestra.run.smithi017.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-11-02T17:44:57.276 INFO:teuthology.orchestra.run.smithi017.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-11-02T17:44:57.276 INFO:teuthology.orchestra.run.smithi017.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-11-02T17:44:57.277 INFO:teuthology.orchestra.run.smithi017.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-11-02T17:44:57.291 INFO:teuthology.orchestra.run.smithi017.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-02T17:44:57.291 INFO:teuthology.orchestra.run.smithi017.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-11-02T17:44:57.292 INFO:teuthology.orchestra.run.smithi017.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-11-02T17:44:57.293 INFO:teuthology.orchestra.run.smithi017.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-11-02T17:44:57.294 INFO:teuthology.orchestra.run.smithi017.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-11-02T17:44:57.307 INFO:teuthology.orchestra.run.smithi017.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-11-02T17:44:57.308 INFO:teuthology.orchestra.run.smithi017.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-11-02T17:44:57.326 INFO:teuthology.orchestra.run.smithi069.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.0-1164-gb8ceb241-1focal [4,001 kB] 2023-11-02T17:44:57.388 INFO:teuthology.orchestra.run.smithi017.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-11-02T17:44:57.402 INFO:teuthology.orchestra.run.smithi017.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-11-02T17:44:57.404 INFO:teuthology.orchestra.run.smithi017.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-11-02T17:44:57.415 INFO:teuthology.orchestra.run.smithi017.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-11-02T17:44:57.416 INFO:teuthology.orchestra.run.smithi017.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-11-02T17:44:57.472 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-11-02T17:44:57.475 INFO:teuthology.orchestra.run.smithi017.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-11-02T17:44:57.476 INFO:teuthology.orchestra.run.smithi017.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-11-02T17:44:57.477 INFO:teuthology.orchestra.run.smithi017.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-11-02T17:44:57.477 INFO:teuthology.orchestra.run.smithi017.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-11-02T17:44:57.478 INFO:teuthology.orchestra.run.smithi017.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-11-02T17:44:57.479 INFO:teuthology.orchestra.run.smithi017.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-11-02T17:44:57.480 INFO:teuthology.orchestra.run.smithi017.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-11-02T17:44:57.486 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-11-02T17:44:57.493 INFO:teuthology.orchestra.run.smithi017.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-11-02T17:44:57.498 INFO:teuthology.orchestra.run.smithi017.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-11-02T17:44:57.498 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.0-1164-gb8ceb241-1focal [738 kB] 2023-11-02T17:44:57.510 INFO:teuthology.orchestra.run.smithi017.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-11-02T17:44:57.511 INFO:teuthology.orchestra.run.smithi017.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-11-02T17:44:57.512 INFO:teuthology.orchestra.run.smithi017.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-11-02T17:44:57.515 INFO:teuthology.orchestra.run.smithi017.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-11-02T17:44:57.518 INFO:teuthology.orchestra.run.smithi017.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-11-02T17:44:57.522 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.0-1164-gb8ceb241-1focal [388 kB] 2023-11-02T17:44:57.536 INFO:teuthology.orchestra.run.smithi017.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-11-02T17:44:57.538 INFO:teuthology.orchestra.run.smithi017.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2023-11-02T17:44:57.540 INFO:teuthology.orchestra.run.smithi017.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-11-02T17:44:57.541 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.0-1164-gb8ceb241-1focal [31.9 kB] 2023-11-02T17:44:57.545 INFO:teuthology.orchestra.run.smithi069.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.0-1164-gb8ceb241-1focal [201 kB] 2023-11-02T17:44:57.563 INFO:teuthology.orchestra.run.smithi069.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.0-1164-gb8ceb241-1focal [60.3 kB] 2023-11-02T17:44:57.570 INFO:teuthology.orchestra.run.smithi069.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-11-02T17:44:57.576 INFO:teuthology.orchestra.run.smithi069.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.0-1164-gb8ceb241-1focal [365 kB] 2023-11-02T17:44:57.582 INFO:teuthology.orchestra.run.smithi017.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-11-02T17:44:57.583 INFO:teuthology.orchestra.run.smithi017.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-11-02T17:44:57.583 INFO:teuthology.orchestra.run.smithi017.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-11-02T17:44:57.584 INFO:teuthology.orchestra.run.smithi017.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-11-02T17:44:57.595 INFO:teuthology.orchestra.run.smithi017.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-02T17:44:57.596 INFO:teuthology.orchestra.run.smithi017.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-11-02T17:44:57.597 INFO:teuthology.orchestra.run.smithi017.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-11-02T17:44:57.597 INFO:teuthology.orchestra.run.smithi017.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-11-02T17:44:57.598 INFO:teuthology.orchestra.run.smithi017.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-11-02T17:44:57.624 INFO:teuthology.orchestra.run.smithi069.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.0-1164-gb8ceb241-1focal [6,582 kB] 2023-11-02T17:44:57.814 INFO:teuthology.orchestra.run.smithi069.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-11-02T17:44:57.824 INFO:teuthology.orchestra.run.smithi069.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-11-02T17:44:57.852 INFO:teuthology.orchestra.run.smithi069.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-11-02T17:44:57.861 INFO:teuthology.orchestra.run.smithi069.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-11-02T17:44:57.862 INFO:teuthology.orchestra.run.smithi069.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-11-02T17:44:57.866 INFO:teuthology.orchestra.run.smithi069.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-11-02T17:44:57.866 INFO:teuthology.orchestra.run.smithi069.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-11-02T17:44:57.950 INFO:teuthology.orchestra.run.smithi069.stdout:Get:21 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.0-1164-gb8ceb241-1focal [113 kB] 2023-11-02T17:44:57.951 INFO:teuthology.orchestra.run.smithi069.stdout:Get:22 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.0-1164-gb8ceb241-1focal [476 kB] 2023-11-02T17:44:57.962 INFO:teuthology.orchestra.run.smithi069.stdout:Get:23 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.0-1164-gb8ceb241-1focal [25.1 MB] 2023-11-02T17:44:58.043 INFO:teuthology.orchestra.run.smithi069.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-11-02T17:44:58.292 INFO:teuthology.orchestra.run.smithi069.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-11-02T17:44:58.463 INFO:teuthology.orchestra.run.smithi069.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-11-02T17:44:58.476 INFO:teuthology.orchestra.run.smithi069.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-11-02T17:44:58.484 INFO:teuthology.orchestra.run.smithi069.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-11-02T17:44:58.490 INFO:teuthology.orchestra.run.smithi069.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-11-02T17:44:58.520 INFO:teuthology.orchestra.run.smithi069.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-02T17:44:58.526 INFO:teuthology.orchestra.run.smithi069.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-11-02T17:44:58.527 INFO:teuthology.orchestra.run.smithi069.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-11-02T17:44:58.557 INFO:teuthology.orchestra.run.smithi069.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-11-02T17:44:58.565 INFO:teuthology.orchestra.run.smithi069.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-11-02T17:44:58.748 INFO:teuthology.orchestra.run.smithi069.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-11-02T17:44:59.061 INFO:teuthology.orchestra.run.smithi069.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-11-02T17:44:59.073 INFO:teuthology.orchestra.run.smithi069.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-11-02T17:44:59.180 INFO:teuthology.orchestra.run.smithi017.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.0-1164-gb8ceb241-1focal [4,001 kB] 2023-11-02T17:44:59.230 INFO:teuthology.orchestra.run.smithi069.stdout:Get:38 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.0-1164-gb8ceb241-1focal [11.0 MB] 2023-11-02T17:44:59.309 INFO:teuthology.orchestra.run.smithi069.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-11-02T17:44:59.310 INFO:teuthology.orchestra.run.smithi069.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-11-02T17:44:59.316 INFO:teuthology.orchestra.run.smithi069.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-11-02T17:44:59.323 INFO:teuthology.orchestra.run.smithi069.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-11-02T17:44:59.339 INFO:teuthology.orchestra.run.smithi069.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-11-02T17:44:59.366 INFO:teuthology.orchestra.run.smithi069.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-11-02T17:44:59.393 INFO:teuthology.orchestra.run.smithi069.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-11-02T17:44:59.398 INFO:teuthology.orchestra.run.smithi069.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2023-11-02T17:44:59.578 INFO:teuthology.orchestra.run.smithi069.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-11-02T17:44:59.866 INFO:teuthology.orchestra.run.smithi069.stdout:Get:48 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.0-1164-gb8ceb241-1focal [234 kB] 2023-11-02T17:44:59.898 INFO:teuthology.orchestra.run.smithi069.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.0-1164-gb8ceb241-1focal [147 kB] 2023-11-02T17:44:59.904 INFO:teuthology.orchestra.run.smithi069.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.0-1164-gb8ceb241-1focal [1,565 kB] 2023-11-02T17:44:59.912 INFO:teuthology.orchestra.run.smithi069.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-11-02T17:44:59.935 INFO:teuthology.orchestra.run.smithi069.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-11-02T17:44:59.948 INFO:teuthology.orchestra.run.smithi069.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-11-02T17:44:59.974 INFO:teuthology.orchestra.run.smithi069.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-11-02T17:44:59.989 INFO:teuthology.orchestra.run.smithi069.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.0-1164-gb8ceb241-1focal [9,404 kB] 2023-11-02T17:44:59.989 INFO:teuthology.orchestra.run.smithi069.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-11-02T17:45:00.038 INFO:teuthology.orchestra.run.smithi069.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-11-02T17:45:00.042 INFO:teuthology.orchestra.run.smithi069.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-11-02T17:45:00.060 INFO:teuthology.orchestra.run.smithi069.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-11-02T17:45:00.070 INFO:teuthology.orchestra.run.smithi069.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-11-02T17:45:00.122 INFO:teuthology.orchestra.run.smithi069.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-02T17:45:00.175 INFO:teuthology.orchestra.run.smithi017.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.0-1164-gb8ceb241-1focal [738 kB] 2023-11-02T17:45:00.289 INFO:teuthology.orchestra.run.smithi069.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-11-02T17:45:00.337 INFO:teuthology.orchestra.run.smithi017.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.0-1164-gb8ceb241-1focal [388 kB] 2023-11-02T17:45:00.372 INFO:teuthology.orchestra.run.smithi017.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.0-1164-gb8ceb241-1focal [31.9 kB] 2023-11-02T17:45:00.376 INFO:teuthology.orchestra.run.smithi017.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.0-1164-gb8ceb241-1focal [201 kB] 2023-11-02T17:45:00.431 INFO:teuthology.orchestra.run.smithi017.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.0-1164-gb8ceb241-1focal [60.3 kB] 2023-11-02T17:45:00.435 INFO:teuthology.orchestra.run.smithi017.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.0-1164-gb8ceb241-1focal [365 kB] 2023-11-02T17:45:00.507 INFO:teuthology.orchestra.run.smithi017.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.0-1164-gb8ceb241-1focal [6,582 kB] 2023-11-02T17:45:00.628 INFO:teuthology.orchestra.run.smithi069.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-11-02T17:45:00.685 INFO:teuthology.orchestra.run.smithi069.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-11-02T17:45:00.750 INFO:teuthology.orchestra.run.smithi069.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-11-02T17:45:00.755 INFO:teuthology.orchestra.run.smithi069.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-11-02T17:45:00.771 INFO:teuthology.orchestra.run.smithi069.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-11-02T17:45:00.772 INFO:teuthology.orchestra.run.smithi069.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.0-1164-gb8ceb241-1focal [31.3 MB] 2023-11-02T17:45:01.278 INFO:teuthology.orchestra.run.smithi069.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-11-02T17:45:01.353 INFO:teuthology.orchestra.run.smithi069.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-11-02T17:45:01.355 INFO:teuthology.orchestra.run.smithi069.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-11-02T17:45:01.391 INFO:teuthology.orchestra.run.smithi069.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-11-02T17:45:01.391 INFO:teuthology.orchestra.run.smithi069.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-11-02T17:45:01.466 INFO:teuthology.orchestra.run.smithi017.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.0-1164-gb8ceb241-1focal [113 kB] 2023-11-02T17:45:01.476 INFO:teuthology.orchestra.run.smithi017.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.0-1164-gb8ceb241-1focal [476 kB] 2023-11-02T17:45:01.550 INFO:teuthology.orchestra.run.smithi017.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.0-1164-gb8ceb241-1focal [25.1 MB] 2023-11-02T17:45:01.555 INFO:teuthology.orchestra.run.smithi069.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-11-02T17:45:01.751 INFO:teuthology.orchestra.run.smithi069.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-11-02T17:45:01.795 INFO:teuthology.orchestra.run.smithi069.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-11-02T17:45:01.858 INFO:teuthology.orchestra.run.smithi069.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-11-02T17:45:01.878 INFO:teuthology.orchestra.run.smithi069.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-11-02T17:45:01.957 INFO:teuthology.orchestra.run.smithi069.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-11-02T17:45:01.967 INFO:teuthology.orchestra.run.smithi069.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-11-02T17:45:01.987 INFO:teuthology.orchestra.run.smithi069.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-11-02T17:45:02.083 INFO:teuthology.orchestra.run.smithi069.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-11-02T17:45:02.113 INFO:teuthology.orchestra.run.smithi069.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-11-02T17:45:02.134 INFO:teuthology.orchestra.run.smithi069.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-11-02T17:45:02.306 INFO:teuthology.orchestra.run.smithi069.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-11-02T17:45:02.791 INFO:teuthology.orchestra.run.smithi069.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-11-02T17:45:02.841 INFO:teuthology.orchestra.run.smithi069.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-11-02T17:45:03.070 INFO:teuthology.orchestra.run.smithi069.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-11-02T17:45:03.081 INFO:teuthology.orchestra.run.smithi069.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2023-11-02T17:45:03.090 INFO:teuthology.orchestra.run.smithi069.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-11-02T17:45:03.244 INFO:teuthology.orchestra.run.smithi069.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-11-02T17:45:03.245 INFO:teuthology.orchestra.run.smithi069.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-11-02T17:45:03.245 INFO:teuthology.orchestra.run.smithi069.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-11-02T17:45:03.246 INFO:teuthology.orchestra.run.smithi069.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-11-02T17:45:03.288 INFO:teuthology.orchestra.run.smithi069.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-02T17:45:03.453 INFO:teuthology.orchestra.run.smithi069.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-11-02T17:45:03.832 INFO:teuthology.orchestra.run.smithi069.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-11-02T17:45:03.842 INFO:teuthology.orchestra.run.smithi069.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-11-02T17:45:03.864 INFO:teuthology.orchestra.run.smithi069.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-11-02T17:45:04.303 INFO:teuthology.orchestra.run.smithi069.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.0-1164-gb8ceb241-1focal [13.7 kB] 2023-11-02T17:45:04.304 INFO:teuthology.orchestra.run.smithi069.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.0-1164-gb8ceb241-1focal [847 kB] 2023-11-02T17:45:04.318 INFO:teuthology.orchestra.run.smithi069.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.0-1164-gb8ceb241-1focal [2,309 kB] 2023-11-02T17:45:04.460 INFO:teuthology.orchestra.run.smithi069.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.0-1164-gb8ceb241-1focal [234 kB] 2023-11-02T17:45:04.506 INFO:teuthology.orchestra.run.smithi069.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.0-1164-gb8ceb241-1focal [136 kB] 2023-11-02T17:45:04.512 INFO:teuthology.orchestra.run.smithi069.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.0-1164-gb8ceb241-1focal [1,832 kB] 2023-11-02T17:45:04.724 INFO:teuthology.orchestra.run.smithi069.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.0-1164-gb8ceb241-1focal [7,897 kB] 2023-11-02T17:45:05.238 INFO:teuthology.orchestra.run.smithi017.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.0-1164-gb8ceb241-1focal [11.0 MB] 2023-11-02T17:45:05.512 INFO:teuthology.orchestra.run.smithi069.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.0-1164-gb8ceb241-1focal [13.8 kB] 2023-11-02T17:45:05.512 INFO:teuthology.orchestra.run.smithi069.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.0-1164-gb8ceb241-1focal [50.1 MB] 2023-11-02T17:45:06.990 INFO:teuthology.orchestra.run.smithi017.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.0-1164-gb8ceb241-1focal [234 kB] 2023-11-02T17:45:07.014 INFO:teuthology.orchestra.run.smithi017.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.0-1164-gb8ceb241-1focal [147 kB] 2023-11-02T17:45:07.035 INFO:teuthology.orchestra.run.smithi017.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.0-1164-gb8ceb241-1focal [1,565 kB] 2023-11-02T17:45:07.304 INFO:teuthology.orchestra.run.smithi017.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.0-1164-gb8ceb241-1focal [9,404 kB] 2023-11-02T17:45:08.477 INFO:teuthology.orchestra.run.smithi017.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.0-1164-gb8ceb241-1focal [31.3 MB] 2023-11-02T17:45:10.624 INFO:teuthology.orchestra.run.smithi069.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.0-1164-gb8ceb241-1focal [118 kB] 2023-11-02T17:45:10.626 INFO:teuthology.orchestra.run.smithi069.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.0-1164-gb8ceb241-1focal [37.3 kB] 2023-11-02T17:45:10.627 INFO:teuthology.orchestra.run.smithi069.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.0-1164-gb8ceb241-1focal [13.4 MB] 2023-11-02T17:45:11.794 INFO:teuthology.orchestra.run.smithi069.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.0-1164-gb8ceb241-1focal [107 kB] 2023-11-02T17:45:12.459 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 208 MB in 15s (13.9 MB/s) 2023-11-02T17:45:12.616 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-11-02T17:45:12.675 INFO:teuthology.orchestra.run.smithi069.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141730 files and directories currently installed.) 2023-11-02T17:45:12.679 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-11-02T17:45:12.765 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-11-02T17:45:12.930 INFO:teuthology.orchestra.run.smithi017.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.0-1164-gb8ceb241-1focal [13.7 kB] 2023-11-02T17:45:12.932 INFO:teuthology.orchestra.run.smithi017.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.0-1164-gb8ceb241-1focal [847 kB] 2023-11-02T17:45:12.945 INFO:teuthology.orchestra.run.smithi017.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.0-1164-gb8ceb241-1focal [2,309 kB] 2023-11-02T17:45:13.118 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-02T17:45:13.133 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-11-02T17:45:13.176 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-11-02T17:45:13.192 INFO:teuthology.orchestra.run.smithi017.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.0-1164-gb8ceb241-1focal [234 kB] 2023-11-02T17:45:13.218 INFO:teuthology.orchestra.run.smithi017.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.0-1164-gb8ceb241-1focal [136 kB] 2023-11-02T17:45:13.238 INFO:teuthology.orchestra.run.smithi017.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.0-1164-gb8ceb241-1focal [1,832 kB] 2023-11-02T17:45:13.460 INFO:teuthology.orchestra.run.smithi017.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.0-1164-gb8ceb241-1focal [7,897 kB] 2023-11-02T17:45:13.521 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-02T17:45:13.539 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-11-02T17:45:13.579 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-11-02T17:45:13.949 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-02T17:45:13.967 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-11-02T17:45:14.008 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-11-02T17:45:14.444 INFO:teuthology.orchestra.run.smithi017.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.0-1164-gb8ceb241-1focal [13.8 kB] 2023-11-02T17:45:14.445 INFO:teuthology.orchestra.run.smithi017.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.0-1164-gb8ceb241-1focal [50.1 MB] 2023-11-02T17:45:14.502 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-02T17:45:14.519 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-11-02T17:45:14.560 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-11-02T17:45:14.896 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-02T17:45:14.912 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-11-02T17:45:14.980 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-11-02T17:45:15.366 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-11-02T17:45:15.381 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-11-02T17:45:15.424 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-11-02T17:45:15.818 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../007-librbd1_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:15.919 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librbd1 (18.2.0-1164-gb8ceb241-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-11-02T17:45:16.599 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../008-librados2_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:16.723 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librados2 (18.2.0-1164-gb8ceb241-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-11-02T17:45:17.445 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libcephfs2. 2023-11-02T17:45:17.463 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../009-libcephfs2_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:17.503 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libcephfs2 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:17.789 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rados. 2023-11-02T17:45:17.807 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../010-python3-rados_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:17.839 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rados (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:18.180 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-02T17:45:18.196 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:45:18.230 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-ceph-argparse (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:18.474 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cephfs. 2023-11-02T17:45:18.490 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../012-python3-cephfs_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:18.532 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cephfs (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:18.885 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-02T17:45:18.901 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:45:18.943 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-ceph-common (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:19.264 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-prettytable. 2023-11-02T17:45:19.280 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2023-11-02T17:45:19.322 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-11-02T17:45:19.608 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rbd. 2023-11-02T17:45:19.691 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../015-python3-rbd_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:19.692 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rbd (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:20.010 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-11-02T17:45:20.024 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-11-02T17:45:20.068 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-11-02T17:45:20.447 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-11-02T17:45:20.463 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2023-11-02T17:45:20.505 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-11-02T17:45:20.833 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-02T17:45:20.849 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-11-02T17:45:20.891 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-11-02T17:45:20.928 INFO:teuthology.orchestra.run.smithi017.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.0-1164-gb8ceb241-1focal [118 kB] 2023-11-02T17:45:20.931 INFO:teuthology.orchestra.run.smithi017.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.0-1164-gb8ceb241-1focal [37.3 kB] 2023-11-02T17:45:20.932 INFO:teuthology.orchestra.run.smithi017.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.0-1164-gb8ceb241-1focal [13.4 MB] 2023-11-02T17:45:21.261 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librgw2. 2023-11-02T17:45:21.278 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../019-librgw2_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:21.318 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librgw2 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:21.566 INFO:teuthology.orchestra.run.smithi017.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-31-1116-reef/b8ceb241941938aa7529adcc4b99d42d196489ec/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.0-1164-gb8ceb241-1focal [107 kB] 2023-11-02T17:45:22.205 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rgw. 2023-11-02T17:45:22.207 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 208 MB in 25s (8,437 kB/s) 2023-11-02T17:45:22.223 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../020-python3-rgw_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:22.354 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rgw (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:22.379 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-11-02T17:45:22.438 INFO:teuthology.orchestra.run.smithi017.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141730 files and directories currently installed.) 2023-11-02T17:45:22.442 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-11-02T17:45:22.569 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-11-02T17:45:22.824 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liboath0. 2023-11-02T17:45:22.840 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-11-02T17:45:22.882 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-11-02T17:45:22.922 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-02T17:45:22.941 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-11-02T17:45:22.972 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-11-02T17:45:23.210 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libradosstriper1. 2023-11-02T17:45:23.225 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../022-libradosstriper1_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:23.268 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libradosstriper1 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:23.308 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-02T17:45:23.328 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-11-02T17:45:23.366 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-11-02T17:45:23.763 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-common. 2023-11-02T17:45:23.779 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../023-ceph-common_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:23.821 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-common (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:23.878 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-02T17:45:23.895 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-11-02T17:45:23.947 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-11-02T17:45:24.432 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-02T17:45:24.450 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-11-02T17:45:24.482 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-11-02T17:45:24.919 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-02T17:45:24.935 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-11-02T17:45:24.977 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-11-02T17:45:25.481 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-11-02T17:45:25.498 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-11-02T17:45:25.540 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-11-02T17:45:25.931 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../007-librbd1_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:26.089 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librbd1 (18.2.0-1164-gb8ceb241-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-11-02T17:45:26.336 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-base. 2023-11-02T17:45:26.352 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../024-ceph-base_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:26.404 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-base (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:26.767 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../008-librados2_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:26.926 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librados2 (18.2.0-1164-gb8ceb241-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-11-02T17:45:27.724 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libcephfs2. 2023-11-02T17:45:27.744 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../009-libcephfs2_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:27.788 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libcephfs2 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:28.183 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-rados. 2023-11-02T17:45:28.200 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../010-python3-rados_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:28.241 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-rados (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:28.432 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-02T17:45:28.450 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2023-11-02T17:45:28.490 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-11-02T17:45:28.552 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-02T17:45:28.570 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:45:28.626 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-ceph-argparse (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:28.818 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-natsort. 2023-11-02T17:45:28.835 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2023-11-02T17:45:28.876 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-11-02T17:45:28.904 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cephfs. 2023-11-02T17:45:28.922 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../012-python3-cephfs_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:28.962 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cephfs (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:29.238 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-logutils. 2023-11-02T17:45:29.255 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2023-11-02T17:45:29.296 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-11-02T17:45:29.315 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-02T17:45:29.331 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:45:29.373 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-ceph-common (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:29.582 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-02T17:45:29.601 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-11-02T17:45:29.632 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-11-02T17:45:29.760 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-prettytable. 2023-11-02T17:45:29.778 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2023-11-02T17:45:29.818 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-11-02T17:45:29.935 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-mako. 2023-11-02T17:45:29.951 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-11-02T17:45:29.993 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-11-02T17:45:30.079 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-rbd. 2023-11-02T17:45:30.096 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../015-python3-rbd_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:30.129 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-rbd (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:30.330 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-02T17:45:30.349 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-02T17:45:30.388 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-02T17:45:30.716 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-02T17:45:30.731 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-11-02T17:45:30.774 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-11-02T17:45:31.077 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-webob. 2023-11-02T17:45:31.092 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2023-11-02T17:45:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-11-02T17:45:31.248 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-11-02T17:45:31.264 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-11-02T17:45:31.306 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-11-02T17:45:31.429 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-soupsieve. 2023-11-02T17:45:31.446 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-11-02T17:45:31.479 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-11-02T17:45:31.693 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-11-02T17:45:31.708 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2023-11-02T17:45:31.751 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-11-02T17:45:31.773 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-bs4. 2023-11-02T17:45:31.793 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2023-11-02T17:45:31.832 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-11-02T17:45:32.079 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-02T17:45:32.095 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-11-02T17:45:32.129 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-11-02T17:45:32.210 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-waitress. 2023-11-02T17:45:32.229 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-11-02T17:45:32.271 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-11-02T17:45:32.524 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package librgw2. 2023-11-02T17:45:32.540 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../019-librgw2_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:32.582 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librgw2 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:32.613 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-tempita. 2023-11-02T17:45:32.629 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2023-11-02T17:45:32.671 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-11-02T17:45:32.999 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-paste. 2023-11-02T17:45:33.015 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-11-02T17:45:33.057 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-11-02T17:45:33.352 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-02T17:45:33.368 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-11-02T17:45:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-11-02T17:45:33.451 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-rgw. 2023-11-02T17:45:33.467 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../020-python3-rgw_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:33.509 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-rgw (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:33.688 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-02T17:45:33.706 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2023-11-02T17:45:33.746 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-11-02T17:45:33.871 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package liboath0. 2023-11-02T17:45:33.890 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-11-02T17:45:33.962 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-11-02T17:45:34.049 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-webtest. 2023-11-02T17:45:34.064 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2023-11-02T17:45:34.107 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-11-02T17:45:34.298 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libradosstriper1. 2023-11-02T17:45:34.315 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../022-libradosstriper1_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:34.356 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libradosstriper1 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:34.443 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pecan. 2023-11-02T17:45:34.460 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-11-02T17:45:34.501 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-11-02T17:45:34.856 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-02T17:45:34.872 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-11-02T17:45:34.876 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-common. 2023-11-02T17:45:34.896 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../023-ceph-common_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:34.914 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-11-02T17:45:34.934 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-common (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:35.234 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-02T17:45:35.251 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:45:35.291 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-modules-core (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:35.646 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-02T17:45:35.666 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:35.704 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libsqlite3-mod-ceph (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:36.049 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cryptography. 2023-11-02T17:45:36.067 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../045-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-11-02T17:45:36.107 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-11-02T17:45:36.477 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-openssl. 2023-11-02T17:45:36.497 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../046-python3-openssl_19.0.0-1build1_all.deb ... 2023-11-02T17:45:36.536 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-11-02T17:45:36.897 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr. 2023-11-02T17:45:36.913 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../047-ceph-mgr_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:36.955 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:37.440 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-base. 2023-11-02T17:45:37.458 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../024-ceph-base_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:37.467 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mon. 2023-11-02T17:45:37.483 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../048-ceph-mon_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:37.516 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mon (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:37.542 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-base (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:38.770 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-osd. 2023-11-02T17:45:38.787 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../049-ceph-osd_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:38.828 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-osd (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:38.986 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-02T17:45:39.003 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2023-11-02T17:45:39.036 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-11-02T17:45:39.364 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-natsort. 2023-11-02T17:45:39.383 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2023-11-02T17:45:39.422 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-11-02T17:45:39.692 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-logutils. 2023-11-02T17:45:39.711 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2023-11-02T17:45:39.742 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-11-02T17:45:40.061 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-02T17:45:40.081 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-11-02T17:45:40.144 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-11-02T17:45:40.466 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-mako. 2023-11-02T17:45:40.483 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-11-02T17:45:40.516 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-11-02T17:45:40.877 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-02T17:45:40.897 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-02T17:45:40.935 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-02T17:45:41.289 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-02T17:45:41.304 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-11-02T17:45:41.346 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-11-02T17:45:41.683 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-webob. 2023-11-02T17:45:41.702 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2023-11-02T17:45:41.741 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-11-02T17:45:41.764 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph. 2023-11-02T17:45:41.780 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../050-ceph_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:41.822 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:42.086 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-soupsieve. 2023-11-02T17:45:42.105 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-11-02T17:45:42.161 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-11-02T17:45:42.191 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-fuse. 2023-11-02T17:45:42.208 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../051-ceph-fuse_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:42.250 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-fuse (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:42.497 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-bs4. 2023-11-02T17:45:42.513 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2023-11-02T17:45:42.555 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-11-02T17:45:42.678 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mds. 2023-11-02T17:45:42.693 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../052-ceph-mds_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:42.727 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mds (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:42.942 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-waitress. 2023-11-02T17:45:42.959 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-11-02T17:45:43.028 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-11-02T17:45:43.222 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package cephadm. 2023-11-02T17:45:43.242 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../053-cephadm_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:43.280 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking cephadm (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:43.395 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-tempita. 2023-11-02T17:45:43.412 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2023-11-02T17:45:43.453 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-11-02T17:45:43.599 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-02T17:45:43.618 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../054-python3-asyncssh_1.12.2-1_all.deb ... 2023-11-02T17:45:43.657 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2023-11-02T17:45:43.798 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-paste. 2023-11-02T17:45:43.817 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-11-02T17:45:43.889 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-11-02T17:45:44.061 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jinja2. 2023-11-02T17:45:44.076 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../055-python3-jinja2_2.10.1-2_all.deb ... 2023-11-02T17:45:44.119 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-11-02T17:45:44.234 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-02T17:45:44.251 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-11-02T17:45:44.292 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-11-02T17:45:44.463 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-02T17:45:44.480 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../056-ceph-mgr-cephadm_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:45:44.521 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-cephadm (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:44.628 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-02T17:45:44.649 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2023-11-02T17:45:44.687 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-11-02T17:45:44.858 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-02T17:45:44.873 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../057-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-11-02T17:45:44.916 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-11-02T17:45:45.049 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-webtest. 2023-11-02T17:45:45.065 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2023-11-02T17:45:45.107 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-11-02T17:45:45.244 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jwt. 2023-11-02T17:45:45.260 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-11-02T17:45:45.301 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-11-02T17:45:45.443 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pecan. 2023-11-02T17:45:45.460 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-11-02T17:45:45.501 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-11-02T17:45:45.613 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-attr. 2023-11-02T17:45:45.631 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2023-11-02T17:45:45.671 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-11-02T17:45:45.906 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-02T17:45:45.923 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-11-02T17:45:46.006 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-11-02T17:45:46.057 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pluggy. 2023-11-02T17:45:46.073 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2023-11-02T17:45:46.115 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-11-02T17:45:46.427 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-py. 2023-11-02T17:45:46.442 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-02T17:45:46.444 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-11-02T17:45:46.459 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:45:46.484 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-11-02T17:45:46.500 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-modules-core (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:46.830 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-atomicwrites. 2023-11-02T17:45:46.845 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-11-02T17:45:46.889 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-11-02T17:45:46.897 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-02T17:45:46.915 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:46.955 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libsqlite3-mod-ceph (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:47.190 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-packaging. 2023-11-02T17:45:47.207 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2023-11-02T17:45:47.248 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-packaging (20.3-1) ... 2023-11-02T17:45:47.325 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cryptography. 2023-11-02T17:45:47.343 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../045-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-11-02T17:45:47.408 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-11-02T17:45:47.576 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-02T17:45:47.592 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-11-02T17:45:47.634 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-11-02T17:45:47.754 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-openssl. 2023-11-02T17:45:47.774 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../046-python3-openssl_19.0.0-1build1_all.deb ... 2023-11-02T17:45:47.803 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-11-02T17:45:47.937 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pytest. 2023-11-02T17:45:47.955 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2023-11-02T17:45:47.995 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-11-02T17:45:48.115 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr. 2023-11-02T17:45:48.134 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../047-ceph-mgr_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:48.165 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:48.357 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-02T17:45:48.372 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2023-11-02T17:45:48.431 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-02T17:45:48.660 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mon. 2023-11-02T17:45:48.678 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../048-ceph-mon_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:48.684 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libjs-underscore. 2023-11-02T17:45:48.702 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-11-02T17:45:48.718 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mon (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:48.742 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-11-02T17:45:49.044 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-11-02T17:45:49.060 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-11-02T17:45:49.094 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-11-02T17:45:49.397 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-routes. 2023-11-02T17:45:49.413 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2023-11-02T17:45:49.455 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-11-02T17:45:49.783 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-02T17:45:49.802 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:45:49.841 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-dashboard (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:50.113 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-osd. 2023-11-02T17:45:50.131 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../049-ceph-osd_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:50.171 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-osd (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:50.336 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-decorator. 2023-11-02T17:45:50.352 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-11-02T17:45:50.394 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-11-02T17:45:50.730 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-11-02T17:45:50.746 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-11-02T17:45:50.788 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-11-02T17:45:51.091 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-scipy. 2023-11-02T17:45:51.108 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-11-02T17:45:51.149 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-11-02T17:45:52.424 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-11-02T17:45:52.443 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-11-02T17:45:52.482 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-11-02T17:45:53.044 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-joblib. 2023-11-02T17:45:53.060 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2023-11-02T17:45:53.102 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-11-02T17:45:53.466 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-sklearn. 2023-11-02T17:45:53.467 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph. 2023-11-02T17:45:53.482 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-11-02T17:45:53.483 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../050-ceph_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:53.524 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-11-02T17:45:53.524 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:53.900 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-fuse. 2023-11-02T17:45:53.917 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../051-ceph-fuse_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:53.960 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-fuse (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:54.405 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mds. 2023-11-02T17:45:54.423 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../052-ceph-mds_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:54.488 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mds (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:54.499 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-02T17:45:54.516 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:45:54.557 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:55.092 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package cephadm. 2023-11-02T17:45:55.108 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../053-cephadm_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:45:55.150 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking cephadm (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:55.478 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-02T17:45:55.497 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../054-python3-asyncssh_1.12.2-1_all.deb ... 2023-11-02T17:45:55.536 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2023-11-02T17:45:55.668 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cachetools. 2023-11-02T17:45:55.684 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2023-11-02T17:45:55.726 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-11-02T17:45:55.956 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-jinja2. 2023-11-02T17:45:55.974 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../055-python3-jinja2_2.10.1-2_all.deb ... 2023-11-02T17:45:56.013 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-11-02T17:45:56.037 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-02T17:45:56.058 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-11-02T17:45:56.096 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-11-02T17:45:56.367 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-02T17:45:56.383 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../056-ceph-mgr-cephadm_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:45:56.424 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-02T17:45:56.425 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-cephadm (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:45:56.440 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-11-02T17:45:56.482 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-11-02T17:45:56.904 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rsa. 2023-11-02T17:45:56.921 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2023-11-02T17:45:56.962 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rsa (4.0-3) ... 2023-11-02T17:45:57.142 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-02T17:45:57.158 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../057-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-11-02T17:45:57.253 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-11-02T17:45:57.275 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-google-auth. 2023-11-02T17:45:57.295 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2023-11-02T17:45:57.333 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-11-02T17:45:57.573 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-jwt. 2023-11-02T17:45:57.593 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-11-02T17:45:57.631 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-11-02T17:45:57.678 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-blinker. 2023-11-02T17:45:57.696 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-11-02T17:45:57.736 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-11-02T17:45:57.984 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-attr. 2023-11-02T17:45:58.000 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2023-11-02T17:45:58.059 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-11-02T17:45:58.065 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-oauthlib. 2023-11-02T17:45:58.081 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-11-02T17:45:58.123 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-11-02T17:45:58.387 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pluggy. 2023-11-02T17:45:58.406 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2023-11-02T17:45:58.444 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-11-02T17:45:58.476 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-02T17:45:58.493 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-11-02T17:45:58.534 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-11-02T17:45:58.789 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-py. 2023-11-02T17:45:58.806 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-11-02T17:45:58.846 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-websocket. 2023-11-02T17:45:58.847 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-11-02T17:45:58.862 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-11-02T17:45:58.904 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-11-02T17:45:59.224 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-02T17:45:59.241 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-11-02T17:45:59.251 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-atomicwrites. 2023-11-02T17:45:59.267 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-11-02T17:45:59.279 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-11-02T17:45:59.309 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-11-02T17:45:59.637 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-packaging. 2023-11-02T17:45:59.657 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2023-11-02T17:45:59.694 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-packaging (20.3-1) ... 2023-11-02T17:45:59.835 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-02T17:45:59.852 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:45:59.893 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-k8sevents (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:00.047 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-02T17:46:00.066 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-11-02T17:46:00.114 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-11-02T17:46:00.221 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-02T17:46:00.237 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2023-11-02T17:46:00.279 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-11-02T17:46:00.467 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pytest. 2023-11-02T17:46:00.486 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2023-11-02T17:46:00.525 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-11-02T17:46:00.632 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-02T17:46:00.651 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-11-02T17:46:00.690 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-11-02T17:46:00.908 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-02T17:46:00.926 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2023-11-02T17:46:00.968 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package jq. 2023-11-02T17:46:00.970 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-02T17:46:00.984 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-11-02T17:46:01.026 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-11-02T17:46:01.256 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libjs-underscore. 2023-11-02T17:46:01.272 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-11-02T17:46:01.279 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package socat. 2023-11-02T17:46:01.295 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2023-11-02T17:46:01.314 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-11-02T17:46:01.370 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking socat (1.7.3.3-2) ... 2023-11-02T17:46:01.600 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-11-02T17:46:01.616 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-11-02T17:46:01.658 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-11-02T17:46:01.682 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package xmlstarlet. 2023-11-02T17:46:01.700 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-11-02T17:46:01.740 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-11-02T17:46:02.011 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-routes. 2023-11-02T17:46:02.028 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2023-11-02T17:46:02.086 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-11-02T17:46:02.128 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-test. 2023-11-02T17:46:02.146 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../094-ceph-test_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:46:02.188 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-test (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:02.431 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-02T17:46:02.450 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:46:02.488 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-dashboard (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:03.051 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-decorator. 2023-11-02T17:46:03.068 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-11-02T17:46:03.109 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-11-02T17:46:03.437 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-11-02T17:46:03.453 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-11-02T17:46:03.520 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-11-02T17:46:03.856 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-scipy. 2023-11-02T17:46:03.872 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-11-02T17:46:03.914 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-11-02T17:46:05.256 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-11-02T17:46:05.275 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-11-02T17:46:05.314 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-11-02T17:46:05.893 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-joblib. 2023-11-02T17:46:05.909 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2023-11-02T17:46:05.951 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-11-02T17:46:06.421 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-sklearn. 2023-11-02T17:46:06.438 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-11-02T17:46:06.479 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-11-02T17:46:06.736 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-volume. 2023-11-02T17:46:06.755 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../095-ceph-volume_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:46:06.794 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-volume (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:07.080 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package fonts-mathjax. 2023-11-02T17:46:07.096 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-11-02T17:46:07.138 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-11-02T17:46:07.196 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-02T17:46:07.213 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:46:07.279 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:07.692 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-02T17:46:07.708 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:46:07.750 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libcephfs-dev (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:08.136 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package nvme-cli. 2023-11-02T17:46:08.153 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-11-02T17:46:08.212 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-11-02T17:46:08.542 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-02T17:46:08.558 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2023-11-02T17:46:08.566 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cachetools. 2023-11-02T17:46:08.583 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2023-11-02T17:46:08.616 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2023-11-02T17:46:08.624 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-11-02T17:46:08.944 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-02T17:46:08.960 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-11-02T17:46:08.969 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libjs-mathjax. 2023-11-02T17:46:08.985 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-11-02T17:46:09.010 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-11-02T17:46:09.044 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-11-02T17:46:09.363 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-02T17:46:09.380 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-11-02T17:46:09.421 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-11-02T17:46:09.877 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-rsa. 2023-11-02T17:46:09.895 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2023-11-02T17:46:09.943 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-rsa (4.0-3) ... 2023-11-02T17:46:10.281 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-google-auth. 2023-11-02T17:46:10.299 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2023-11-02T17:46:10.336 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-11-02T17:46:10.648 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python-natsort-doc. 2023-11-02T17:46:10.666 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2023-11-02T17:46:10.705 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-11-02T17:46:10.725 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-blinker. 2023-11-02T17:46:10.743 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-11-02T17:46:10.783 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-11-02T17:46:11.042 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-webencodings. 2023-11-02T17:46:11.062 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-11-02T17:46:11.100 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-11-02T17:46:11.136 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-oauthlib. 2023-11-02T17:46:11.152 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-11-02T17:46:11.194 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-11-02T17:46:11.429 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-html5lib. 2023-11-02T17:46:11.445 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-11-02T17:46:11.486 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-11-02T17:46:11.598 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-02T17:46:11.618 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-11-02T17:46:11.656 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-11-02T17:46:11.840 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-11-02T17:46:11.857 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-11-02T17:46:11.898 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-11-02T17:46:12.018 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-websocket. 2023-11-02T17:46:12.035 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-11-02T17:46:12.076 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-11-02T17:46:12.359 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pastescript. 2023-11-02T17:46:12.379 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2023-11-02T17:46:12.417 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-02T17:46:12.446 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-02T17:46:12.466 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-11-02T17:46:12.504 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-11-02T17:46:12.738 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-psutil. 2023-11-02T17:46:12.756 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-11-02T17:46:12.796 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-11-02T17:46:13.093 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-02T17:46:13.114 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:46:13.151 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-k8sevents (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:13.166 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-02T17:46:13.182 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-11-02T17:46:13.224 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-11-02T17:46:13.518 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-simplejson. 2023-11-02T17:46:13.524 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-02T17:46:13.539 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-11-02T17:46:13.541 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2023-11-02T17:46:13.582 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-11-02T17:46:13.585 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-11-02T17:46:13.879 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-02T17:46:13.896 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-11-02T17:46:13.937 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-11-02T17:46:13.961 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-02T17:46:13.979 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-11-02T17:46:14.019 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-11-02T17:46:14.322 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package jq. 2023-11-02T17:46:14.338 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-11-02T17:46:14.380 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-11-02T17:46:14.628 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package radosgw. 2023-11-02T17:46:14.647 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../110-radosgw_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:46:14.673 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package socat. 2023-11-02T17:46:14.685 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking radosgw (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:14.694 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2023-11-02T17:46:14.731 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking socat (1.7.3.3-2) ... 2023-11-02T17:46:15.085 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package xmlstarlet. 2023-11-02T17:46:15.103 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-11-02T17:46:15.134 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-11-02T17:46:15.497 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-test. 2023-11-02T17:46:15.517 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../094-ceph-test_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:46:15.555 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-test (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:16.114 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package rbd-fuse. 2023-11-02T17:46:16.131 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../111-rbd-fuse_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:46:16.172 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking rbd-fuse (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:16.468 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-attr (19.3.0-2) ... 2023-11-02T17:46:16.769 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-11-02T17:46:16.895 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-11-02T17:46:17.021 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-11-02T17:46:17.215 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up cephadm (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:17.809 INFO:teuthology.orchestra.run.smithi069.stdout:Adding system user cephadm....done 2023-11-02T17:46:17.966 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-11-02T17:46:18.092 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-11-02T17:46:18.438 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-02T17:46:18.731 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-11-02T17:46:18.848 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-11-02T17:46:18.976 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-11-02T17:46:19.326 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-11-02T17:46:19.730 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-11-02T17:46:20.040 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-11-02T17:46:20.316 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-11-02T17:46:20.496 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-volume. 2023-11-02T17:46:20.512 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../095-ceph-volume_18.2.0-1164-gb8ceb241-1focal_all.deb ... 2023-11-02T17:46:20.553 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-volume (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:20.600 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-11-02T17:46:20.891 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package fonts-mathjax. 2023-11-02T17:46:20.901 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-ceph-argparse (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:20.908 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-11-02T17:46:20.974 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-11-02T17:46:21.202 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-11-02T17:46:21.353 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-11-02T17:46:21.536 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-02T17:46:21.555 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:46:21.594 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libcephfs-dev (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:21.638 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-11-02T17:46:21.931 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-11-02T17:46:22.023 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package nvme-cli. 2023-11-02T17:46:22.039 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-11-02T17:46:22.056 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-11-02T17:46:22.123 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-11-02T17:46:22.441 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-11-02T17:46:22.470 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-02T17:46:22.485 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2023-11-02T17:46:22.527 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2023-11-02T17:46:22.725 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-11-02T17:46:22.922 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libjs-mathjax. 2023-11-02T17:46:22.942 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-11-02T17:46:22.985 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-11-02T17:46:22.996 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-11-02T17:46:23.328 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-11-02T17:46:23.612 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-11-02T17:46:23.955 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-11-02T17:46:24.248 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-packaging (20.3-1) ... 2023-11-02T17:46:24.532 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-11-02T17:46:24.641 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-11-02T17:46:24.687 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-natsort-doc. 2023-11-02T17:46:24.705 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2023-11-02T17:46:24.744 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-11-02T17:46:24.767 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-11-02T17:46:25.139 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-webencodings. 2023-11-02T17:46:25.159 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-11-02T17:46:25.197 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-11-02T17:46:25.327 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-11-02T17:46:25.534 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-html5lib. 2023-11-02T17:46:25.550 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-11-02T17:46:25.591 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-11-02T17:46:25.729 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-11-02T17:46:25.999 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-11-02T17:46:26.019 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-11-02T17:46:26.056 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-11-02T17:46:26.114 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-11-02T17:46:26.484 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up socat (1.7.3.3-2) ... 2023-11-02T17:46:26.560 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pastescript. 2023-11-02T17:46:26.579 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2023-11-02T17:46:26.610 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-ceph-common (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:26.618 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-02T17:46:26.929 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-11-02T17:46:26.980 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-psutil. 2023-11-02T17:46:26.997 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-11-02T17:46:27.038 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-11-02T17:46:27.063 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-11-02T17:46:27.389 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-11-02T17:46:27.406 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-02T17:46:27.426 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-11-02T17:46:27.466 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-11-02T17:46:27.515 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-11-02T17:46:27.658 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-11-02T17:46:27.802 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-simplejson. 2023-11-02T17:46:27.820 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-11-02T17:46:27.860 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-11-02T17:46:27.936 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-11-02T17:46:28.247 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-11-02T17:46:28.398 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-11-02T17:46:28.524 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-11-02T17:46:28.556 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-02T17:46:28.574 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-11-02T17:46:28.614 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-11-02T17:46:28.834 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-11-02T17:46:29.101 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-11-02T17:46:29.397 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package radosgw. 2023-11-02T17:46:29.414 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../110-radosgw_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:46:29.419 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-11-02T17:46:29.454 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking radosgw (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:29.545 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-02T17:46:29.815 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-11-02T17:46:30.133 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-11-02T17:46:30.425 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-11-02T17:46:30.793 INFO:teuthology.orchestra.run.smithi069.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-11-02T17:46:31.045 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-11-02T17:46:31.083 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package rbd-fuse. 2023-11-02T17:46:31.100 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../111-rbd-fuse_18.2.0-1164-gb8ceb241-1focal_amd64.deb ... 2023-11-02T17:46:31.166 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking rbd-fuse (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:31.179 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-11-02T17:46:31.488 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-11-02T17:46:31.502 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-attr (19.3.0-2) ... 2023-11-02T17:46:31.848 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-11-02T17:46:31.880 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-11-02T17:46:31.974 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-11-02T17:46:32.033 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-11-02T17:46:32.100 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-11-02T17:46:32.142 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-11-02T17:46:32.377 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up cephadm (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:32.434 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-11-02T17:46:32.736 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-11-02T17:46:33.039 INFO:teuthology.orchestra.run.smithi017.stdout:Adding system user cephadm....done 2023-11-02T17:46:33.217 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-11-02T17:46:33.335 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-11-02T17:46:33.778 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-02T17:46:34.088 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-11-02T17:46:34.214 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-11-02T17:46:34.216 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-11-02T17:46:34.359 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-11-02T17:46:34.373 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-11-02T17:46:34.719 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-11-02T17:46:34.725 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-11-02T17:46:35.020 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-11-02T17:46:35.127 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-11-02T17:46:35.330 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-11-02T17:46:35.439 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-11-02T17:46:35.512 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-11-02T17:46:35.782 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-11-02T17:46:35.813 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-11-02T17:46:36.109 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2023-11-02T17:46:36.114 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-11-02T17:46:36.424 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-ceph-argparse (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:36.485 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-11-02T17:46:36.766 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-11-02T17:46:36.837 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-11-02T17:46:36.918 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-11-02T17:46:37.236 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-11-02T17:46:37.537 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-11-02T17:46:37.665 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-11-02T17:46:37.892 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-11-02T17:46:38.018 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-11-02T17:46:38.100 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-11-02T17:46:38.144 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-11-02T17:46:38.270 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rsa (4.0-3) ... 2023-11-02T17:46:38.393 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-11-02T17:46:38.596 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-11-02T17:46:38.694 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-11-02T17:46:38.706 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-11-02T17:46:39.007 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-11-02T17:46:39.045 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-11-02T17:46:39.291 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-02T17:46:39.329 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-11-02T17:46:39.601 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-11-02T17:46:39.673 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-11-02T17:46:39.953 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librados2 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:39.977 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-packaging (20.3-1) ... 2023-11-02T17:46:40.079 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2023-11-02T17:46:40.197 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librgw2 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:40.315 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libsqlite3-mod-ceph (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:40.320 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-11-02T17:46:40.479 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-11-02T17:46:40.518 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libcephfs2 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:40.605 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-11-02T17:46:40.666 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-routes (2.4.1-2) ... 2023-11-02T17:46:40.948 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-11-02T17:46:40.962 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libradosstriper1 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:41.088 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-11-02T17:46:41.350 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-11-02T17:46:41.381 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librbd1 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:41.541 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-modules-core (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:41.667 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-fuse (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:41.702 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-11-02T17:46:41.883 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-02T17:46:41.883 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-02T17:46:42.130 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up socat (1.7.3.3-2) ... 2023-11-02T17:46:42.231 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-ceph-common (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:42.352 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libcephfs-dev (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:42.536 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rados (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:42.541 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-11-02T17:46:42.671 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-11-02T17:46:42.709 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-11-02T17:46:43.069 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-11-02T17:46:43.195 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-11-02T17:46:43.346 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-11-02T17:46:43.625 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rbd (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:43.655 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-11-02T17:46:43.777 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up rbd-fuse (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:43.902 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rgw (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:43.974 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-11-02T17:46:44.028 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cephfs (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:44.075 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-11-02T17:46:44.156 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-common (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:44.226 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-11-02T17:46:44.552 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-11-02T17:46:44.735 INFO:teuthology.orchestra.run.smithi069.stdout:Adding group ceph....done 2023-11-02T17:46:44.853 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-11-02T17:46:45.112 INFO:teuthology.orchestra.run.smithi069.stdout:Adding system user ceph....done 2023-11-02T17:46:45.163 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-11-02T17:46:45.313 INFO:teuthology.orchestra.run.smithi069.stdout:Setting system user ceph properties....done 2023-11-02T17:46:45.331 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-02T17:46:45.337 INFO:teuthology.orchestra.run.smithi069.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-02T17:46:45.481 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-02T17:46:45.607 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-11-02T17:46:45.809 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-02T17:46:45.900 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-11-02T17:46:46.201 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-11-02T17:46:46.276 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-test (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:46.402 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up radosgw (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:46.570 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-11-02T17:46:46.845 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-11-02T17:46:46.879 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-02T17:46:46.879 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-02T17:46:46.996 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-11-02T17:46:47.339 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-11-02T17:46:47.345 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-base (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:47.683 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-11-02T17:46:47.737 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-02T17:46:47.783 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-11-02T17:46:47.943 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-11-02T17:46:48.181 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mds (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:48.286 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-11-02T17:46:48.419 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-02T17:46:48.419 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-02T17:46:48.587 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-11-02T17:46:48.851 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:49.082 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-02T17:46:49.082 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-02T17:46:49.519 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-osd (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:50.011 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-02T17:46:50.011 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-02T17:46:50.110 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-11-02T17:46:50.269 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-11-02T17:46:50.462 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-k8sevents (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:50.646 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-11-02T17:46:50.738 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-diskprediction-local (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:50.980 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-11-02T17:46:51.082 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mon (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:51.309 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-02T17:46:51.309 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-02T17:46:51.348 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-11-02T17:46:51.449 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-11-02T17:46:51.758 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-cephadm (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:51.784 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-11-02T17:46:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:52.027 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-dashboard (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:52.171 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-volume (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:52.194 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2023-11-02T17:46:52.549 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-11-02T17:46:52.595 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-11-02T17:46:52.700 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-11-02T17:46:52.955 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-11-02T17:46:53.010 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-11-02T17:46:54.087 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-11-02T17:46:54.246 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-11-02T17:46:54.389 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-11-02T17:46:54.523 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-rsa (4.0-3) ... 2023-11-02T17:46:54.817 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-11-02T17:46:54.926 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-11-02T17:46:55.227 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-11-02T17:46:55.536 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-02T17:46:55.813 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-11-02T17:46:55.888 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-11-02T17:46:56.281 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librados2 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:56.382 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2023-11-02T17:46:56.525 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librgw2 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:56.634 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libsqlite3-mod-ceph (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:56.811 INFO:teuthology.orchestra.run.smithi069.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-02T17:46:56.814 DEBUG:teuthology.parallel:result is None 2023-11-02T17:46:56.879 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libcephfs2 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:57.008 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-routes (2.4.1-2) ... 2023-11-02T17:46:57.326 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libradosstriper1 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:57.477 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-11-02T17:46:57.795 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librbd1 (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:57.954 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-modules-core (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:58.080 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-fuse (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:58.312 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-02T17:46:58.312 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-02T17:46:58.823 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libcephfs-dev (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:58.949 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-rados (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:46:59.439 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-11-02T17:47:00.396 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-rbd (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:00.506 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up rbd-fuse (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:00.667 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-rgw (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:00.793 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cephfs (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:00.944 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-common (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:01.480 INFO:teuthology.orchestra.run.smithi017.stdout:Adding group ceph....done 2023-11-02T17:47:01.873 INFO:teuthology.orchestra.run.smithi017.stdout:Adding system user ceph....done 2023-11-02T17:47:02.084 INFO:teuthology.orchestra.run.smithi017.stdout:Setting system user ceph properties....done 2023-11-02T17:47:02.107 INFO:teuthology.orchestra.run.smithi017.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-02T17:47:02.262 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-02T17:47:02.585 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-02T17:47:03.046 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-test (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:03.147 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up radosgw (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:03.661 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-02T17:47:03.662 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-02T17:47:04.107 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-base (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:04.502 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-02T17:47:04.985 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mds (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:05.222 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-02T17:47:05.222 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-02T17:47:05.686 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:05.950 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-02T17:47:05.950 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-02T17:47:06.413 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-osd (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:06.906 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-02T17:47:06.907 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-02T17:47:07.381 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-k8sevents (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:07.524 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-diskprediction-local (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:07.675 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mon (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:07.904 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-02T17:47:07.905 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-02T17:47:08.385 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-cephadm (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:08.527 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:08.686 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-dashboard (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:08.829 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-volume (18.2.0-1164-gb8ceb241-1focal) ... 2023-11-02T17:47:09.274 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-11-02T17:47:09.400 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-11-02T17:47:09.694 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-11-02T17:47:12.630 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-11-02T17:47:13.262 INFO:teuthology.orchestra.run.smithi017.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-02T17:47:13.267 DEBUG:teuthology.parallel:result is None 2023-11-02T17:47:13.267 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=b8ceb241941938aa7529adcc4b99d42d196489ec 2023-11-02T17:47:13.482 DEBUG:teuthology.orchestra.run.smithi017:> dpkg-query -W -f '${Version}' ceph 2023-11-02T17:47:13.506 INFO:teuthology.orchestra.run.smithi017.stdout:18.2.0-1164-gb8ceb241-1focal 2023-11-02T17:47:13.507 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-1164-gb8ceb241-1focal 2023-11-02T17:47:13.507 INFO:teuthology.task.install:The correct ceph version 18.2.0-1164-gb8ceb241-1focal is installed. 2023-11-02T17:47:13.508 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=b8ceb241941938aa7529adcc4b99d42d196489ec 2023-11-02T17:47:13.649 DEBUG:teuthology.orchestra.run.smithi069:> dpkg-query -W -f '${Version}' ceph 2023-11-02T17:47:13.674 INFO:teuthology.orchestra.run.smithi069.stdout:18.2.0-1164-gb8ceb241-1focal 2023-11-02T17:47:13.674 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-1164-gb8ceb241-1focal 2023-11-02T17:47:13.674 INFO:teuthology.task.install:The correct ceph version 18.2.0-1164-gb8ceb241-1focal is installed. 2023-11-02T17:47:13.675 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-02T17:47:13.676 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:13.676 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-02T17:47:13.692 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:13.693 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-02T17:47:13.731 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-02T17:47:13.731 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:13.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/daemon-helper 2023-11-02T17:47:13.749 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-02T17:47:13.808 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:13.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2023-11-02T17:47:13.825 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-02T17:47:13.834 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-02T17:47:13.835 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:13.835 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-02T17:47:13.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-02T17:47:13.906 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:13.907 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-02T17:47:13.921 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-02T17:47:13.937 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-02T17:47:13.937 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:13.937 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/stdin-killer 2023-11-02T17:47:13.953 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-02T17:47:14.012 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:14.012 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2023-11-02T17:47:14.027 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-02T17:47:14.085 INFO:teuthology.run_tasks:Running task ceph... 2023-11-02T17:47:14.144 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-02T17:47:14.145 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 777 /var/log/ceph 2023-11-02T17:47:14.148 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /var/log/ceph 2023-11-02T17:47:14.163 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-02T17:47:14.163 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-02T17:47:14.206 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-02T17:47:14.221 INFO:tasks.ceph:Creating extra log directories... 2023-11-02T17:47:14.222 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-02T17:47:14.224 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-02T17:47:14.289 INFO:tasks.ceph:Creating ceph cluster c1... 2023-11-02T17:47:14.289 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-02T17:47:14.289 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849', 'branch': 'wip-yuri3-testing-2023-10-31-1116-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7442849', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2023-10-31-1116-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-yuri3-testing-2023-10-31-1116-reef', 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 7837, 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri3-testing-2023-10-31-1116-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec', 'targets': {'smithi017.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-11-01_14:50:55', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-11-02T17:47:14.290 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-11-02T17:47:14.293 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-11-02T17:47:14.299 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/run/ceph 2023-11-02T17:47:14.301 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2023-11-02T17:47:14.316 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:14.316 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2023-11-02T17:47:14.359 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-02T17:47:14.360 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 2023-11-02T17:47:14.367 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-02T17:47:14.367 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T17:47:14.367 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 359 Links: 1 2023-11-02T17:47:14.367 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T17:47:14.368 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2023-11-02 17:43:10.777354267 +0000 2023-11-02T17:47:14.368 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2023-11-02 17:43:10.773354359 +0000 2023-11-02T17:47:14.368 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2023-11-02 17:43:10.773354359 +0000 2023-11-02T17:47:14.368 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2023-11-02T17:47:14.369 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-02T17:47:14.380 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2023-11-02T17:47:14.380 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2023-11-02T17:47:14.381 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000316393 s, 1.6 MB/s 2023-11-02T17:47:14.382 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-02T17:47:14.433 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 2023-11-02T17:47:14.440 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-02T17:47:14.441 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T17:47:14.441 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 367 Links: 1 2023-11-02T17:47:14.441 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T17:47:14.441 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2023-11-02 17:43:11.589335537 +0000 2023-11-02T17:47:14.442 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2023-11-02 17:43:11.293342365 +0000 2023-11-02T17:47:14.442 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2023-11-02 17:43:11.293342365 +0000 2023-11-02T17:47:14.442 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2023-11-02T17:47:14.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-02T17:47:14.456 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2023-11-02T17:47:14.456 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2023-11-02T17:47:14.456 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000293374 s, 1.7 MB/s 2023-11-02T17:47:14.457 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-02T17:47:14.509 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 2023-11-02T17:47:14.517 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-02T17:47:14.517 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T17:47:14.517 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 372 Links: 1 2023-11-02T17:47:14.518 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T17:47:14.518 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2023-11-02 17:43:11.793330832 +0000 2023-11-02T17:47:14.518 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2023-11-02 17:43:11.789330923 +0000 2023-11-02T17:47:14.518 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2023-11-02 17:43:11.789330923 +0000 2023-11-02T17:47:14.518 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2023-11-02T17:47:14.519 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-02T17:47:14.533 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2023-11-02T17:47:14.534 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2023-11-02T17:47:14.534 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.00026872 s, 1.9 MB/s 2023-11-02T17:47:14.535 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-02T17:47:14.543 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 2023-11-02T17:47:14.550 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-02T17:47:14.550 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T17:47:14.550 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 377 Links: 1 2023-11-02T17:47:14.550 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T17:47:14.551 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2023-11-02 17:43:12.573312840 +0000 2023-11-02T17:47:14.551 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2023-11-02 17:43:12.293319298 +0000 2023-11-02T17:47:14.551 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2023-11-02 17:43:12.293319298 +0000 2023-11-02T17:47:14.551 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2023-11-02T17:47:14.552 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-02T17:47:14.566 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2023-11-02T17:47:14.566 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2023-11-02T17:47:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000237277 s, 2.2 MB/s 2023-11-02T17:47:14.568 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-02T17:47:14.621 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-02T17:47:14.622 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-02T17:47:14.622 INFO:tasks.ceph:Generating config... 2023-11-02T17:47:14.623 INFO:tasks.ceph:[client] debug rgw = 20 2023-11-02T17:47:14.623 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-11-02T17:47:14.624 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-11-02T17:47:14.624 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-11-02T17:47:14.624 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-11-02T17:47:14.624 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-11-02T17:47:14.625 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-11-02T17:47:14.625 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-11-02T17:47:14.625 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-11-02T17:47:14.626 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-11-02T17:47:14.626 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-11-02T17:47:14.626 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-11-02T17:47:14.626 INFO:tasks.ceph:[client] setgroup = ceph 2023-11-02T17:47:14.627 INFO:tasks.ceph:[client] setuser = ceph 2023-11-02T17:47:14.627 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-11-02T17:47:14.627 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-02T17:47:14.627 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-02T17:47:14.628 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-02T17:47:14.628 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-02T17:47:14.628 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-02T17:47:14.628 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-02T17:47:14.629 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-02T17:47:14.629 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-11-02T17:47:14.629 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-11-02T17:47:14.629 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-11-02T17:47:14.630 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-11-02T17:47:14.630 INFO:tasks.ceph:Setting up c1.mon.a... 2023-11-02T17:47:14.630 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2023-11-02T17:47:14.670 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/c1.keyring 2023-11-02T17:47:14.686 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2023-11-02T17:47:14.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-11-02T17:47:14.739 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.17')] 2023-11-02T17:47:14.740 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.17'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-11-02T17:47:14.741 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:14.741 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-02T17:47:14.787 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.17 --print /home/ubuntu/cephtest/c1.monmap 2023-11-02T17:47:14.825 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:47:14.825 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:47:14.826 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2023-11-02T17:47:14.826 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: generated fsid 50f513e4-d9c7-4c20-8e66-40810a4a5dc7 2023-11-02T17:47:14.827 INFO:teuthology.orchestra.run.smithi017.stdout:setting min_mon_release = pacific 2023-11-02T17:47:14.827 INFO:teuthology.orchestra.run.smithi017.stdout:epoch 0 2023-11-02T17:47:14.827 INFO:teuthology.orchestra.run.smithi017.stdout:fsid 50f513e4-d9c7-4c20-8e66-40810a4a5dc7 2023-11-02T17:47:14.827 INFO:teuthology.orchestra.run.smithi017.stdout:last_changed 2023-11-02T17:47:14.824486+0000 2023-11-02T17:47:14.827 INFO:teuthology.orchestra.run.smithi017.stdout:created 2023-11-02T17:47:14.824486+0000 2023-11-02T17:47:14.828 INFO:teuthology.orchestra.run.smithi017.stdout:min_mon_release 16 (pacific) 2023-11-02T17:47:14.828 INFO:teuthology.orchestra.run.smithi017.stdout:election_strategy: 1 2023-11-02T17:47:14.828 INFO:teuthology.orchestra.run.smithi017.stdout:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2023-11-02T17:47:14.828 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2023-11-02T17:47:14.829 DEBUG:teuthology.orchestra.run.smithi017:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-02T17:47:14.834 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 50f513e4-d9c7-4c20-8e66-40810a4a5dc7... 2023-11-02T17:47:14.835 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-11-02T17:47:14.837 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-11-02T17:47:14.866 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2023-11-02T17:47:14.867 INFO:teuthology.orchestra.run.smithi017.stdout: chdir = "" 2023-11-02T17:47:14.867 INFO:teuthology.orchestra.run.smithi017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-02T17:47:14.867 INFO:teuthology.orchestra.run.smithi017.stdout: auth supported = cephx 2023-11-02T17:47:14.867 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.867 INFO:teuthology.orchestra.run.smithi017.stdout: filestore xattr use omap = true 2023-11-02T17:47:14.868 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.868 INFO:teuthology.orchestra.run.smithi017.stdout: mon clock drift allowed = 1.000 2023-11-02T17:47:14.868 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.868 INFO:teuthology.orchestra.run.smithi017.stdout: osd crush chooseleaf type = 0 2023-11-02T17:47:14.868 INFO:teuthology.orchestra.run.smithi017.stdout: auth debug = true 2023-11-02T17:47:14.869 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.869 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on old message = true 2023-11-02T17:47:14.869 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on bug = true 2023-11-02T17:47:14.869 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.869 INFO:teuthology.orchestra.run.smithi017.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-02T17:47:14.869 INFO:teuthology.orchestra.run.smithi017.stdout: mon pg warn max object skew = 0 2023-11-02T17:47:14.870 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.870 INFO:teuthology.orchestra.run.smithi017.stdout: # disable pg_autoscaler by default for new pools 2023-11-02T17:47:14.870 INFO:teuthology.orchestra.run.smithi017.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-02T17:47:14.870 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.870 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default size = 2 2023-11-02T17:47:14.871 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.871 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow primary affinity = true 2023-11-02T17:47:14.871 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow pg remap = true 2023-11-02T17:47:14.871 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on legacy crush tunables = false 2023-11-02T17:47:14.871 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on crush straw calc version zero = false 2023-11-02T17:47:14.872 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on no sortbitwise = false 2023-11-02T17:47:14.872 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on osd down out interval zero = false 2023-11-02T17:47:14.872 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on too few osds = false 2023-11-02T17:47:14.872 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-02T17:47:14.872 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-02T17:47:14.872 INFO:teuthology.orchestra.run.smithi017.stdout: mon_allow_pool_size_one = true 2023-11-02T17:47:14.873 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.873 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-02T17:47:14.873 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.873 INFO:teuthology.orchestra.run.smithi017.stdout: osd default data pool replay window = 5 2023-11-02T17:47:14.873 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.874 INFO:teuthology.orchestra.run.smithi017.stdout: mon allow pool delete = true 2023-11-02T17:47:14.874 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.874 INFO:teuthology.orchestra.run.smithi017.stdout: mon cluster log file level = debug 2023-11-02T17:47:14.874 INFO:teuthology.orchestra.run.smithi017.stdout: debug asserts on shutdown = true 2023-11-02T17:47:14.874 INFO:teuthology.orchestra.run.smithi017.stdout: mon health detail to clog = false 2023-11-02T17:47:14.874 INFO:teuthology.orchestra.run.smithi017.stdout: mon host = 172.21.15.17 2023-11-02T17:47:14.875 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = 50f513e4-d9c7-4c20-8e66-40810a4a5dc7 2023-11-02T17:47:14.875 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.875 INFO:teuthology.orchestra.run.smithi017.stdout:[osd] 2023-11-02T17:47:14.875 INFO:teuthology.orchestra.run.smithi017.stdout: osd journal size = 100 2023-11-02T17:47:14.875 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.876 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub load threshold = 5.0 2023-11-02T17:47:14.876 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub max interval = 600 2023-11-02T17:47:14.876 INFO:teuthology.orchestra.run.smithi017.stdout: osd mclock profile = high_recovery_ops 2023-11-02T17:47:14.876 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.876 INFO:teuthology.orchestra.run.smithi017.stdout: osd recover clone overlap = true 2023-11-02T17:47:14.877 INFO:teuthology.orchestra.run.smithi017.stdout: osd recovery max chunk = 1048576 2023-11-02T17:47:14.877 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.877 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug shutdown = true 2023-11-02T17:47:14.877 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug op order = true 2023-11-02T17:47:14.877 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify stray on activate = true 2023-11-02T17:47:14.877 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.878 INFO:teuthology.orchestra.run.smithi017.stdout: osd open classes on start = true 2023-11-02T17:47:14.878 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug pg log writeout = true 2023-11-02T17:47:14.878 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.878 INFO:teuthology.orchestra.run.smithi017.stdout: osd deep scrub update digest min age = 30 2023-11-02T17:47:14.878 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.879 INFO:teuthology.orchestra.run.smithi017.stdout: osd map max advance = 10 2023-11-02T17:47:14.879 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.879 INFO:teuthology.orchestra.run.smithi017.stdout: journal zero on create = true 2023-11-02T17:47:14.879 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.879 INFO:teuthology.orchestra.run.smithi017.stdout: filestore ondisk finisher threads = 3 2023-11-02T17:47:14.880 INFO:teuthology.orchestra.run.smithi017.stdout: filestore apply finisher threads = 3 2023-11-02T17:47:14.880 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.880 INFO:teuthology.orchestra.run.smithi017.stdout: bdev debug aio = true 2023-11-02T17:47:14.880 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug misdirected ops = true 2023-11-02T17:47:14.880 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2023-11-02T17:47:14.880 INFO:teuthology.orchestra.run.smithi017.stdout: debug osd = 20 2023-11-02T17:47:14.881 INFO:teuthology.orchestra.run.smithi017.stdout: rocksdb delete range threshold = 0 2023-11-02T17:47:14.881 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.881 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr] 2023-11-02T17:47:14.881 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2023-11-02T17:47:14.881 INFO:teuthology.orchestra.run.smithi017.stdout: debug mgr = 20 2023-11-02T17:47:14.882 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2023-11-02T17:47:14.882 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2023-11-02T17:47:14.882 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2023-11-02T17:47:14.882 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2023-11-02T17:47:14.882 INFO:teuthology.orchestra.run.smithi017.stdout: mgr/telemetry/nag = false 2023-11-02T17:47:14.883 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.883 INFO:teuthology.orchestra.run.smithi017.stdout:[mon] 2023-11-02T17:47:14.883 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2023-11-02T17:47:14.883 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2023-11-02T17:47:14.883 INFO:teuthology.orchestra.run.smithi017.stdout: debug paxos = 20 2023-11-02T17:47:14.883 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2023-11-02T17:47:14.884 INFO:teuthology.orchestra.run.smithi017.stdout: mon data avail warn = 5 2023-11-02T17:47:14.884 INFO:teuthology.orchestra.run.smithi017.stdout: mon mgr mkfs grace = 240 2023-11-02T17:47:14.884 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2023-11-02T17:47:14.884 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd reporter subtree level = osd 2023-11-02T17:47:14.884 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd prime pg temp = true 2023-11-02T17:47:14.885 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2023-11-02T17:47:14.885 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.885 INFO:teuthology.orchestra.run.smithi017.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-02T17:47:14.885 INFO:teuthology.orchestra.run.smithi017.stdout: auth mon ticket ttl = 660 # 11m 2023-11-02T17:47:14.885 INFO:teuthology.orchestra.run.smithi017.stdout: auth service ticket ttl = 240 # 4m 2023-11-02T17:47:14.885 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.886 INFO:teuthology.orchestra.run.smithi017.stdout: # don't complain about insecure global_id in the test suite 2023-11-02T17:47:14.886 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-02T17:47:14.886 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-02T17:47:14.886 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.886 INFO:teuthology.orchestra.run.smithi017.stdout: # 1m isn't quite enough 2023-11-02T17:47:14.887 INFO:teuthology.orchestra.run.smithi017.stdout: mon_down_mkfs_grace = 2m 2023-11-02T17:47:14.887 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.887 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_filestore_osds = false 2023-11-02T17:47:14.887 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:14.887 INFO:teuthology.orchestra.run.smithi017.stdout:[client] 2023-11-02T17:47:14.888 INFO:teuthology.orchestra.run.smithi017.stdout: rgw cache enabled = true 2023-11-02T17:47:14.888 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable ops log = true 2023-11-02T17:47:14.888 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable usage log = true 2023-11-02T17:47:14.888 INFO:teuthology.orchestra.run.smithi017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-02T17:47:14.888 INFO:teuthology.orchestra.run.smithi017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-02T17:47:14.889 INFO:teuthology.orchestra.run.smithi017.stdout: debug rgw = 20 2023-11-02T17:47:14.889 INFO:teuthology.orchestra.run.smithi017.stdout: debug rgw sync = 20 2023-11-02T17:47:14.889 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt require ssl = False 2023-11-02T17:47:14.889 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt s3 kms backend = testing 2023-11-02T17:47:14.889 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-11-02T17:47:14.889 INFO:teuthology.orchestra.run.smithi017.stdout: rgw curl low speed time = 300 2023-11-02T17:47:14.890 INFO:teuthology.orchestra.run.smithi017.stdout: rgw data log num shards = 4 2023-11-02T17:47:14.890 INFO:teuthology.orchestra.run.smithi017.stdout: rgw md log max shards = 4 2023-11-02T17:47:14.890 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync data inject err probability = 0.1 2023-11-02T17:47:14.890 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync log trim interval = 0 2023-11-02T17:47:14.890 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync meta inject err probability = 0.1 2023-11-02T17:47:14.891 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync obj etag verify = True 2023-11-02T17:47:14.891 INFO:teuthology.orchestra.run.smithi017.stdout: setgroup = ceph 2023-11-02T17:47:14.891 INFO:teuthology.orchestra.run.smithi017.stdout: setuser = ceph 2023-11-02T17:47:14.891 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.a] 2023-11-02T17:47:14.891 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2023-11-02T17:47:14.892 INFO:teuthology.orchestra.run.smithi017.stdout: rgw data notify interval msec = 0 2023-11-02T17:47:14.892 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2023-11-02T17:47:14.892 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 10 2023-11-02T17:47:14.892 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2023-11-02T17:47:14.892 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 1000 2023-11-02T17:47:14.893 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2023-11-02T17:47:14.893 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 10000 2023-11-02T17:47:14.894 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2023-11-02T17:47:14.894 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2023-11-02T17:47:14.895 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-02T17:47:14.895 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2023-11-02T17:47:14.895 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.895 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2023-11-02T17:47:14.895 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.896 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2023-11-02T17:47:14.896 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.896 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2023-11-02T17:47:14.896 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2023-11-02T17:47:14.896 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.897 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2023-11-02T17:47:14.897 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2023-11-02T17:47:14.897 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.897 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-02T17:47:14.897 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2023-11-02T17:47:14.897 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.898 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2023-11-02T17:47:14.898 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-02T17:47:14.898 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.898 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2023-11-02T17:47:14.899 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.899 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2023-11-02T17:47:14.899 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2023-11-02T17:47:14.899 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2023-11-02T17:47:14.899 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2023-11-02T17:47:14.900 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2023-11-02T17:47:14.900 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2023-11-02T17:47:14.900 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2023-11-02T17:47:14.900 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-02T17:47:14.900 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-02T17:47:14.900 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2023-11-02T17:47:14.901 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.901 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-02T17:47:14.901 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.901 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2023-11-02T17:47:14.902 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.902 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2023-11-02T17:47:14.902 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.902 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2023-11-02T17:47:14.902 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2023-11-02T17:47:14.903 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2023-11-02T17:47:14.903 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = 172.21.15.17 2023-11-02T17:47:14.903 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 50f513e4-d9c7-4c20-8e66-40810a4a5dc7 2023-11-02T17:47:14.903 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.903 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2023-11-02T17:47:14.904 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2023-11-02T17:47:14.904 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.904 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2023-11-02T17:47:14.904 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2023-11-02T17:47:14.904 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2023-11-02T17:47:14.905 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.905 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2023-11-02T17:47:14.905 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2023-11-02T17:47:14.905 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.906 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2023-11-02T17:47:14.906 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2023-11-02T17:47:14.906 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2023-11-02T17:47:14.906 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.906 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2023-11-02T17:47:14.907 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2023-11-02T17:47:14.907 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.907 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2023-11-02T17:47:14.907 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.907 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2023-11-02T17:47:14.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.908 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2023-11-02T17:47:14.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.908 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2023-11-02T17:47:14.908 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2023-11-02T17:47:14.909 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.909 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2023-11-02T17:47:14.909 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2023-11-02T17:47:14.909 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-11-02T17:47:14.909 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2023-11-02T17:47:14.909 INFO:teuthology.orchestra.run.smithi069.stdout: rocksdb delete range threshold = 0 2023-11-02T17:47:14.910 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.910 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2023-11-02T17:47:14.910 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-11-02T17:47:14.910 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2023-11-02T17:47:14.910 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-11-02T17:47:14.911 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-11-02T17:47:14.911 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-11-02T17:47:14.911 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-11-02T17:47:14.911 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2023-11-02T17:47:14.913 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.913 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2023-11-02T17:47:14.914 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-11-02T17:47:14.914 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-11-02T17:47:14.914 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2023-11-02T17:47:14.914 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-11-02T17:47:14.915 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2023-11-02T17:47:14.915 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2023-11-02T17:47:14.915 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-11-02T17:47:14.915 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2023-11-02T17:47:14.915 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2023-11-02T17:47:14.916 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-11-02T17:47:14.916 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.916 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-02T17:47:14.916 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2023-11-02T17:47:14.916 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2023-11-02T17:47:14.917 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.917 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2023-11-02T17:47:14.917 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-02T17:47:14.917 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-02T17:47:14.917 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.918 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2023-11-02T17:47:14.918 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2023-11-02T17:47:14.918 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.918 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2023-11-02T17:47:14.918 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:14.919 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2023-11-02T17:47:14.919 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2023-11-02T17:47:14.919 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2023-11-02T17:47:14.919 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2023-11-02T17:47:14.919 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-02T17:47:14.920 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-02T17:47:14.920 INFO:teuthology.orchestra.run.smithi069.stdout: debug rgw = 20 2023-11-02T17:47:14.920 INFO:teuthology.orchestra.run.smithi069.stdout: debug rgw sync = 20 2023-11-02T17:47:14.920 INFO:teuthology.orchestra.run.smithi069.stdout: rgw crypt require ssl = False 2023-11-02T17:47:14.920 INFO:teuthology.orchestra.run.smithi069.stdout: rgw crypt s3 kms backend = testing 2023-11-02T17:47:14.920 INFO:teuthology.orchestra.run.smithi069.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-11-02T17:47:14.921 INFO:teuthology.orchestra.run.smithi069.stdout: rgw curl low speed time = 300 2023-11-02T17:47:14.921 INFO:teuthology.orchestra.run.smithi069.stdout: rgw data log num shards = 4 2023-11-02T17:47:14.921 INFO:teuthology.orchestra.run.smithi069.stdout: rgw md log max shards = 4 2023-11-02T17:47:14.921 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync data inject err probability = 0.1 2023-11-02T17:47:14.921 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync log trim interval = 0 2023-11-02T17:47:14.922 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync meta inject err probability = 0.1 2023-11-02T17:47:14.922 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync obj etag verify = True 2023-11-02T17:47:14.922 INFO:teuthology.orchestra.run.smithi069.stdout: setgroup = ceph 2023-11-02T17:47:14.922 INFO:teuthology.orchestra.run.smithi069.stdout: setuser = ceph 2023-11-02T17:47:14.922 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2023-11-02T17:47:14.923 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2023-11-02T17:47:14.923 INFO:teuthology.orchestra.run.smithi069.stdout: rgw data notify interval msec = 0 2023-11-02T17:47:14.923 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2023-11-02T17:47:14.923 INFO:teuthology.orchestra.run.smithi069.stdout: osd_max_omap_entries_per_request = 10 2023-11-02T17:47:14.924 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2023-11-02T17:47:14.924 INFO:teuthology.orchestra.run.smithi069.stdout: osd_max_omap_entries_per_request = 1000 2023-11-02T17:47:14.924 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2023-11-02T17:47:14.924 INFO:teuthology.orchestra.run.smithi069.stdout: osd_max_omap_entries_per_request = 10000 2023-11-02T17:47:14.924 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2023-11-02T17:47:14.925 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2023-11-02T17:47:14.990 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-02T17:47:14.991 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:14.991 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2023-11-02T17:47:14.998 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:14.998 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2023-11-02T17:47:15.004 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi017.front.sepia.ceph.com 2023-11-02T17:47:15.005 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:15.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/c1.keyring 2023-11-02T17:47:15.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-11-02T17:47:15.074 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:15.074 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-11-02T17:47:15.082 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2023-11-02T17:47:15.082 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:15.082 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/c1.keyring 2023-11-02T17:47:15.083 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-11-02T17:47:15.109 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:15.109 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-11-02T17:47:15.159 INFO:tasks.ceph:Setting up mon nodes... 2023-11-02T17:47:15.160 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-02T17:47:15.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2023-11-02T17:47:15.212 INFO:teuthology.orchestra.run.smithi017.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2023-11-02T17:47:15.215 INFO:tasks.ceph:Setting up mds nodes... 2023-11-02T17:47:15.215 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-02T17:47:15.216 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2023-11-02T17:47:15.255 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/c1.client.0.keyring 2023-11-02T17:47:15.269 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2023-11-02T17:47:15.354 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/c1.client.1.keyring 2023-11-02T17:47:15.368 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-02T17:47:15.368 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-02T17:47:15.368 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2023-11-02T17:47:15.379 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-02T17:47:15.379 INFO:tasks.ceph:role: c1.osd.0 2023-11-02T17:47:15.379 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com 2023-11-02T17:47:15.379 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-02T17:47:15.422 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T17:47:15.422 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T17:47:15.422 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T17:47:15.422 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 2023-11-02T17:47:15.423 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T17:47:15.423 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2023-11-02T17:47:15.423 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T17:47:15.423 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-02T17:47:15.423 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T17:47:15.424 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T17:47:15.425 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2023-11-02T17:47:15.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2023-11-02T17:47:15.493 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2023-11-02T17:47:15.503 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: /sbin/restorecon: command not found 2023-11-02T17:47:15.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T17:47:15.504 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2023-11-02T17:47:15.516 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-02T17:47:15.516 INFO:tasks.ceph:role: c1.osd.1 2023-11-02T17:47:15.516 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com 2023-11-02T17:47:15.516 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-02T17:47:15.558 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T17:47:15.559 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T17:47:15.559 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T17:47:15.559 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 2023-11-02T17:47:15.559 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T17:47:15.560 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2023-11-02T17:47:15.560 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T17:47:15.560 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-02T17:47:15.560 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T17:47:15.560 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T17:47:15.561 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2023-11-02T17:47:15.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2023-11-02T17:47:15.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2023-11-02T17:47:15.634 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: /sbin/restorecon: command not found 2023-11-02T17:47:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T17:47:15.636 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2023-11-02T17:47:15.652 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-02T17:47:15.652 INFO:tasks.ceph:role: c1.osd.2 2023-11-02T17:47:15.653 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com 2023-11-02T17:47:15.653 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-02T17:47:15.694 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T17:47:15.695 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T17:47:15.695 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T17:47:15.695 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 2023-11-02T17:47:15.695 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T17:47:15.696 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2023-11-02T17:47:15.696 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T17:47:15.696 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-02T17:47:15.696 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T17:47:15.696 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T17:47:15.697 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2023-11-02T17:47:15.697 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2023-11-02T17:47:15.763 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2023-11-02T17:47:15.771 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: /sbin/restorecon: command not found 2023-11-02T17:47:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T17:47:15.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2023-11-02T17:47:15.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:15.844+0000 7fabaa909700 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2023-11-02T17:47:15.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:15.844+0000 7fabaa909700 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2023-11-02T17:47:15.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:15.844+0000 7fabaa909700 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-11-02T17:47:15.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:15.844+0000 7fabaa909700 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-11-02T17:47:15.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:15.844+0000 7fabaa909700 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-11-02T17:47:15.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:15.844+0000 7fabaa909700 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2023-11-02T17:47:18.064 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2023-11-02T17:47:18.081 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2023-11-02T17:47:18.175 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:18.168+0000 7ff7d1fb2700 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2023-11-02T17:47:18.176 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:18.168+0000 7ff7d1fb2700 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2023-11-02T17:47:18.176 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:18.168+0000 7ff7d1fb2700 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-11-02T17:47:18.176 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:18.168+0000 7ff7d1fb2700 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-11-02T17:47:18.176 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:18.168+0000 7ff7d1fb2700 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-11-02T17:47:18.177 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:18.168+0000 7ff7d1fb2700 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2023-11-02T17:47:20.377 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2023-11-02T17:47:20.394 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2023-11-02T17:47:20.437 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:20.427+0000 7f44ad83a700 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2023-11-02T17:47:20.437 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:20.427+0000 7f44ad83a700 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2023-11-02T17:47:20.437 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:20.427+0000 7f44ad83a700 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-11-02T17:47:20.438 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:20.427+0000 7f44ad83a700 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-11-02T17:47:20.438 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:20.427+0000 7f44ad83a700 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-11-02T17:47:20.438 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:47:20.431+0000 7f44ad83a700 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2023-11-02T17:47:22.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2023-11-02T17:47:22.668 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-02T17:47:22.669 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:22.669 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2023-11-02T17:47:22.724 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:22.725 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2023-11-02T17:47:22.735 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:22.736 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2023-11-02T17:47:22.748 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:22.748 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2023-11-02T17:47:22.759 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:22.759 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2023-11-02T17:47:22.807 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:22.808 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2023-11-02T17:47:22.815 INFO:tasks.ceph:Adding keys to all mons... 2023-11-02T17:47:22.815 DEBUG:teuthology.orchestra.run.smithi017:> sudo tee -a /etc/ceph/c1.keyring 2023-11-02T17:47:22.830 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr.x] 2023-11-02T17:47:22.830 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCj4ENlpVCSDBAAsvZUBxJXlA8dDXgszH09Qw== 2023-11-02T17:47:22.831 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2023-11-02T17:47:22.831 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCj4ENlr5WMMhAAsRVt3Glo+t3lB0g3xOe3pw== 2023-11-02T17:47:22.831 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2023-11-02T17:47:22.831 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCm4ENlDBBZChAA8ZgIn4/OlRNU7ehMNzSviw== 2023-11-02T17:47:22.831 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2023-11-02T17:47:22.832 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCo4ENlESHxGRAAjaGONn/Vw6iC9mI7dPtfyQ== 2023-11-02T17:47:22.832 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2023-11-02T17:47:22.832 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCj4ENlGdQjDxAA4xOr4IS6B9RIDN2LKfk5/Q== 2023-11-02T17:47:22.832 INFO:teuthology.orchestra.run.smithi017.stdout:[client.1] 2023-11-02T17:47:22.832 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCj4ENltg0JFRAA0UwWuiGPhfuiZhvPlohANQ== 2023-11-02T17:47:22.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-02T17:47:22.879 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T17:47:22.927 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T17:47:22.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T17:47:23.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-02T17:47:23.077 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-02T17:47:23.127 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-02T17:47:23.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2023-11-02T17:47:23.142 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2023-11-02T17:47:23.214 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2023-11-02T17:47:23.230 DEBUG:teuthology.orchestra.run.smithi017:> rm -- /home/ubuntu/cephtest/c1.monmap 2023-11-02T17:47:23.279 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2023-11-02T17:47:23.279 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2023-11-02T17:47:23.280 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2023-11-02T17:47:23.283 INFO:tasks.ceph.c1.mon.a:Started 2023-11-02T17:47:23.283 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2023-11-02T17:47:23.283 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2023-11-02T17:47:23.284 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2023-11-02T17:47:23.286 INFO:tasks.ceph.c1.mgr.x:Started 2023-11-02T17:47:23.287 INFO:tasks.ceph:Setting crush tunables to default 2023-11-02T17:47:23.287 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd crush tunables default 2023-11-02T17:47:23.514 INFO:teuthology.orchestra.run.smithi017.stderr:adjusted tunables profile to default 2023-11-02T17:47:23.526 INFO:tasks.ceph:check_enable_crimson: False 2023-11-02T17:47:23.526 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2023-11-02T17:47:23.527 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:23.527 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2023-11-02T17:47:23.537 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:23.537 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2023-11-02T17:47:23.548 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:23.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2023-11-02T17:47:23.606 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd new fde9c108-768b-4403-b5ab-a4bff75f7ab2 0 2023-11-02T17:47:23.627 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:23.619+0000 7fd8d11f42c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-02T17:47:23.809 INFO:teuthology.orchestra.run.smithi017.stdout:0 2023-11-02T17:47:23.822 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd new db44b8ec-897b-447f-bb62-9ab776d0ecab 1 2023-11-02T17:47:24.023 INFO:teuthology.orchestra.run.smithi017.stdout:1 2023-11-02T17:47:24.041 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd new 88e8afb5-9fa7-461d-98ff-028b3b7123d3 2 2023-11-02T17:47:24.250 INFO:teuthology.orchestra.run.smithi017.stdout:2 2023-11-02T17:47:24.263 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2023-11-02T17:47:24.264 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2023-11-02T17:47:24.266 INFO:tasks.ceph.c1.osd.0:Started 2023-11-02T17:47:24.266 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2023-11-02T17:47:24.267 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2023-11-02T17:47:24.269 INFO:tasks.ceph.c1.osd.1:Started 2023-11-02T17:47:24.269 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2023-11-02T17:47:24.270 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2023-11-02T17:47:24.272 INFO:tasks.ceph.c1.osd.2:Started 2023-11-02T17:47:24.273 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-11-02T17:47:24.293 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:24.283+0000 7fd8d11f42c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-02T17:47:24.374 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:24.367+0000 7fd8d11f42c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-02T17:47:24.473 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:24.473 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":5,"fsid":"50f513e4-d9c7-4c20-8e66-40810a4a5dc7","created":"2023-11-02T17:47:23.404808+0000","modified":"2023-11-02T17:47:24.245525+0000","last_up_change":"0.000000","last_in_change":"2023-11-02T17:47:24.245525+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fde9c108-768b-4403-b5ab-a4bff75f7ab2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db44b8ec-897b-447f-bb62-9ab776d0ecab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"88e8afb5-9fa7-461d-98ff-028b3b7123d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T17:47:24.485 INFO:tasks.ceph.ceph_manager.c1:[] 2023-11-02T17:47:24.486 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-02T17:47:24.539 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:24.531+0000 7fd8d11f42c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-02T17:47:24.623 INFO:tasks.ceph.c1.osd.1.smithi017.stderr:2023-11-02T17:47:24.615+0000 7ff013463700 -1 Falling back to public interface 2023-11-02T17:47:24.639 INFO:tasks.ceph.c1.osd.0.smithi017.stderr:2023-11-02T17:47:24.631+0000 7f1cb07af700 -1 Falling back to public interface 2023-11-02T17:47:24.756 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:24.747+0000 7fd8d11f42c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-02T17:47:24.788 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-11-02T17:47:24.813 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:24.803+0000 7fd8d11f42c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-02T17:47:24.910 INFO:tasks.ceph.c1.osd.2.smithi017.stderr:2023-11-02T17:47:24.903+0000 7f58a339a700 -1 Falling back to public interface 2023-11-02T17:47:24.969 INFO:teuthology.misc.health.smithi017.stdout: 2023-11-02T17:47:24.969 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":5,"fsid":"50f513e4-d9c7-4c20-8e66-40810a4a5dc7","created":"2023-11-02T17:47:23.404808+0000","modified":"2023-11-02T17:47:24.245525+0000","last_up_change":"0.000000","last_in_change":"2023-11-02T17:47:24.245525+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fde9c108-768b-4403-b5ab-a4bff75f7ab2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db44b8ec-897b-447f-bb62-9ab776d0ecab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"88e8afb5-9fa7-461d-98ff-028b3b7123d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T17:47:24.981 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-02T17:47:25.012 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:25.003+0000 7fd8d11f42c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-02T17:47:25.082 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:25.075+0000 7fd8d11f42c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-02T17:47:25.153 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:25.143+0000 7fd8d11f42c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-02T17:47:25.588 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:25.579+0000 7fd8d11f42c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-02T17:47:25.718 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:25.711+0000 7fd8d11f42c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-02T17:47:26.090 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:26.083+0000 7fd8d11f42c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-02T17:47:26.280 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:26.271+0000 7fd8d11f42c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-02T17:47:26.370 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:26.363+0000 7fd8d11f42c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-02T17:47:26.433 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:26.427+0000 7fd8d11f42c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-02T17:47:27.134 INFO:tasks.ceph.c1.osd.0.smithi017.stderr:2023-11-02T17:47:27.127+0000 7f1cb07af700 -1 osd.0 0 log_to_monitors true 2023-11-02T17:47:27.295 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:27.287+0000 7fd8d11f42c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-02T17:47:27.368 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:27.359+0000 7fd8d11f42c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-02T17:47:27.383 INFO:tasks.ceph.c1.osd.1.smithi017.stderr:2023-11-02T17:47:27.375+0000 7ff013463700 -1 osd.1 0 log_to_monitors true 2023-11-02T17:47:27.532 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:27.523+0000 7fd8d11f42c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-02T17:47:27.598 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:27.591+0000 7fd8d11f42c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-02T17:47:27.672 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:27.663+0000 7fd8d11f42c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-02T17:47:27.851 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:27.843+0000 7fd8d11f42c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-02T17:47:27.884 INFO:tasks.ceph.c1.osd.2.smithi017.stderr:2023-11-02T17:47:27.875+0000 7f58a339a700 -1 osd.2 0 log_to_monitors true 2023-11-02T17:47:28.082 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:28.075+0000 7fd8d11f42c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-02T17:47:28.746 INFO:tasks.ceph.c1.osd.0.smithi017.stderr:2023-11-02T17:47:28.739+0000 7f1cac5ee700 -1 osd.0 0 waiting for initial osdmap 2023-11-02T17:47:28.750 INFO:tasks.ceph.c1.osd.0.smithi017.stderr:2023-11-02T17:47:28.743+0000 7f1ca409d700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T17:47:28.799 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:28.791+0000 7fd8d11f42c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-02T17:47:29.827 INFO:tasks.ceph.c1.mgr.x.smithi017.stderr:2023-11-02T17:47:29.819+0000 7fd8b8276700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-02T17:47:29.938 INFO:tasks.ceph.c1.osd.1.smithi017.stderr:2023-11-02T17:47:29.931+0000 7ff00f2a2700 -1 osd.1 0 waiting for initial osdmap 2023-11-02T17:47:29.942 INFO:tasks.ceph.c1.osd.1.smithi017.stderr:2023-11-02T17:47:29.935+0000 7ff006d51700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T17:47:29.949 INFO:tasks.ceph.c1.osd.2.smithi017.stderr:2023-11-02T17:47:29.943+0000 7f589fb60700 -1 osd.2 0 waiting for initial osdmap 2023-11-02T17:47:29.953 INFO:tasks.ceph.c1.osd.2.smithi017.stderr:2023-11-02T17:47:29.947+0000 7f5896c88700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T17:47:31.286 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-11-02T17:47:31.587 INFO:teuthology.misc.health.smithi017.stdout: 2023-11-02T17:47:31.587 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":9,"fsid":"50f513e4-d9c7-4c20-8e66-40810a4a5dc7","created":"2023-11-02T17:47:23.404808+0000","modified":"2023-11-02T17:47:30.948623+0000","last_up_change":"2023-11-02T17:47:30.948623+0000","last_in_change":"2023-11-02T17:47:24.245525+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fde9c108-768b-4403-b5ab-a4bff75f7ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":29008},{"type":"v1","addr":"172.21.15.17:6809","nonce":29008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":29008},{"type":"v1","addr":"172.21.15.17:6811","nonce":29008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":29008},{"type":"v1","addr":"172.21.15.17:6815","nonce":29008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":29008},{"type":"v1","addr":"172.21.15.17:6813","nonce":29008}]},"public_addr":"172.21.15.17:6809/29008","cluster_addr":"172.21.15.17:6811/29008","heartbeat_back_addr":"172.21.15.17:6815/29008","heartbeat_front_addr":"172.21.15.17:6813/29008","state":["exists","up"]},{"osd":1,"uuid":"db44b8ec-897b-447f-bb62-9ab776d0ecab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":29007},{"type":"v1","addr":"172.21.15.17:6801","nonce":29007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":29007},{"type":"v1","addr":"172.21.15.17:6803","nonce":29007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":29007},{"type":"v1","addr":"172.21.15.17:6807","nonce":29007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":29007},{"type":"v1","addr":"172.21.15.17:6805","nonce":29007}]},"public_addr":"172.21.15.17:6801/29007","cluster_addr":"172.21.15.17:6803/29007","heartbeat_back_addr":"172.21.15.17:6807/29007","heartbeat_front_addr":"172.21.15.17:6805/29007","state":["exists","up"]},{"osd":2,"uuid":"88e8afb5-9fa7-461d-98ff-028b3b7123d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":29009},{"type":"v1","addr":"172.21.15.17:6817","nonce":29009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":29009},{"type":"v1","addr":"172.21.15.17:6819","nonce":29009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":29009},{"type":"v1","addr":"172.21.15.17:6823","nonce":29009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":29009},{"type":"v1","addr":"172.21.15.17:6821","nonce":29009}]},"public_addr":"172.21.15.17:6817/29009","cluster_addr":"172.21.15.17:6819/29009","heartbeat_back_addr":"172.21.15.17:6823/29009","heartbeat_front_addr":"172.21.15.17:6821/29009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:28.120393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T17:47:31.599 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-02T17:47:31.600 INFO:tasks.ceph:Creating RBD pool 2023-11-02T17:47:31.600 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd pool create rbd 8 2023-11-02T17:47:32.229 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'rbd' created 2023-11-02T17:47:32.242 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-02T17:47:33.244 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-02T17:47:33.257 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2023-11-02T17:47:33.257 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2023-11-02T17:47:33.257 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2023-11-02T17:47:33.258 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2023-11-02T17:47:33.258 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-02T17:47:33.600 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:33.615 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":4,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":28824},{"type":"v1","addr":"172.21.15.17:6825","nonce":28824}]},"active_addr":"172.21.15.17:6825/28824","active_change":"2023-11-02T17:47:28.799501+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":2685300937}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":47216167}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":3034513158}]}]} 2023-11-02T17:47:33.617 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2023-11-02T17:47:33.618 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2023-11-02T17:47:33.618 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-11-02T17:47:33.919 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:33.920 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":11,"fsid":"50f513e4-d9c7-4c20-8e66-40810a4a5dc7","created":"2023-11-02T17:47:23.404808+0000","modified":"2023-11-02T17:47:33.222683+0000","last_up_change":"2023-11-02T17:47:30.948623+0000","last_in_change":"2023-11-02T17:47:24.245525+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-02T17:47:31.934180+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-02T17:47:32.875264+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fde9c108-768b-4403-b5ab-a4bff75f7ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":29008},{"type":"v1","addr":"172.21.15.17:6809","nonce":29008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":29008},{"type":"v1","addr":"172.21.15.17:6811","nonce":29008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":29008},{"type":"v1","addr":"172.21.15.17:6815","nonce":29008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":29008},{"type":"v1","addr":"172.21.15.17:6813","nonce":29008}]},"public_addr":"172.21.15.17:6809/29008","cluster_addr":"172.21.15.17:6811/29008","heartbeat_back_addr":"172.21.15.17:6815/29008","heartbeat_front_addr":"172.21.15.17:6813/29008","state":["exists","up"]},{"osd":1,"uuid":"db44b8ec-897b-447f-bb62-9ab776d0ecab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":29007},{"type":"v1","addr":"172.21.15.17:6801","nonce":29007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":29007},{"type":"v1","addr":"172.21.15.17:6803","nonce":29007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":29007},{"type":"v1","addr":"172.21.15.17:6807","nonce":29007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":29007},{"type":"v1","addr":"172.21.15.17:6805","nonce":29007}]},"public_addr":"172.21.15.17:6801/29007","cluster_addr":"172.21.15.17:6803/29007","heartbeat_back_addr":"172.21.15.17:6807/29007","heartbeat_front_addr":"172.21.15.17:6805/29007","state":["exists","up"]},{"osd":2,"uuid":"88e8afb5-9fa7-461d-98ff-028b3b7123d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":29009},{"type":"v1","addr":"172.21.15.17:6817","nonce":29009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":29009},{"type":"v1","addr":"172.21.15.17:6819","nonce":29009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":29009},{"type":"v1","addr":"172.21.15.17:6823","nonce":29009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":29009},{"type":"v1","addr":"172.21.15.17:6821","nonce":29009}]},"public_addr":"172.21.15.17:6817/29009","cluster_addr":"172.21.15.17:6819/29009","heartbeat_back_addr":"172.21.15.17:6823/29009","heartbeat_front_addr":"172.21.15.17:6821/29009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:28.120393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:28.376986+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:28.915527+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T17:47:33.938 INFO:tasks.ceph.ceph_manager.c1:all up! 2023-11-02T17:47:33.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-11-02T17:47:34.244 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:34.245 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":12,"fsid":"50f513e4-d9c7-4c20-8e66-40810a4a5dc7","created":"2023-11-02T17:47:23.404808+0000","modified":"2023-11-02T17:47:34.233495+0000","last_up_change":"2023-11-02T17:47:30.948623+0000","last_in_change":"2023-11-02T17:47:24.245525+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-02T17:47:31.934180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-02T17:47:32.875264+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fde9c108-768b-4403-b5ab-a4bff75f7ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":29008},{"type":"v1","addr":"172.21.15.17:6809","nonce":29008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":29008},{"type":"v1","addr":"172.21.15.17:6811","nonce":29008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":29008},{"type":"v1","addr":"172.21.15.17:6815","nonce":29008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":29008},{"type":"v1","addr":"172.21.15.17:6813","nonce":29008}]},"public_addr":"172.21.15.17:6809/29008","cluster_addr":"172.21.15.17:6811/29008","heartbeat_back_addr":"172.21.15.17:6815/29008","heartbeat_front_addr":"172.21.15.17:6813/29008","state":["exists","up"]},{"osd":1,"uuid":"db44b8ec-897b-447f-bb62-9ab776d0ecab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":29007},{"type":"v1","addr":"172.21.15.17:6801","nonce":29007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":29007},{"type":"v1","addr":"172.21.15.17:6803","nonce":29007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":29007},{"type":"v1","addr":"172.21.15.17:6807","nonce":29007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":29007},{"type":"v1","addr":"172.21.15.17:6805","nonce":29007}]},"public_addr":"172.21.15.17:6801/29007","cluster_addr":"172.21.15.17:6803/29007","heartbeat_back_addr":"172.21.15.17:6807/29007","heartbeat_front_addr":"172.21.15.17:6805/29007","state":["exists","up"]},{"osd":2,"uuid":"88e8afb5-9fa7-461d-98ff-028b3b7123d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":29009},{"type":"v1","addr":"172.21.15.17:6817","nonce":29009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":29009},{"type":"v1","addr":"172.21.15.17:6819","nonce":29009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":29009},{"type":"v1","addr":"172.21.15.17:6823","nonce":29009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":29009},{"type":"v1","addr":"172.21.15.17:6821","nonce":29009}]},"public_addr":"172.21.15.17:6817/29009","cluster_addr":"172.21.15.17:6819/29009","heartbeat_back_addr":"172.21.15.17:6823/29009","heartbeat_front_addr":"172.21.15.17:6821/29009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:28.120393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:28.376986+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:28.915527+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T17:47:34.262 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2023-11-02T17:47:34.262 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2023-11-02T17:47:34.263 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2023-11-02T17:47:34.435 INFO:teuthology.orchestra.run.smithi017.stdout:38654705667 2023-11-02T17:47:34.435 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-11-02T17:47:34.439 INFO:teuthology.orchestra.run.smithi017.stdout:38654705667 2023-11-02T17:47:34.439 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-11-02T17:47:34.458 INFO:teuthology.orchestra.run.smithi017.stdout:34359738370 2023-11-02T17:47:34.458 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-11-02T17:47:34.748 INFO:teuthology.orchestra.run.smithi017.stdout:34359738369 2023-11-02T17:47:34.761 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2023-11-02T17:47:34.785 INFO:teuthology.orchestra.run.smithi017.stdout:38654705666 2023-11-02T17:47:34.799 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.2 2023-11-02T17:47:34.800 INFO:teuthology.orchestra.run.smithi017.stdout:38654705666 2023-11-02T17:47:34.814 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.1 2023-11-02T17:47:35.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-11-02T17:47:35.801 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-11-02T17:47:35.815 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-11-02T17:47:36.074 INFO:teuthology.orchestra.run.smithi017.stdout:34359738370 2023-11-02T17:47:36.088 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.0 2023-11-02T17:47:36.088 DEBUG:teuthology.parallel:result is None 2023-11-02T17:47:36.090 INFO:teuthology.orchestra.run.smithi017.stdout:38654705667 2023-11-02T17:47:36.103 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705667 for osd.2 2023-11-02T17:47:36.104 DEBUG:teuthology.parallel:result is None 2023-11-02T17:47:36.105 INFO:teuthology.orchestra.run.smithi017.stdout:38654705667 2023-11-02T17:47:36.118 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705667 for osd.1 2023-11-02T17:47:36.119 DEBUG:teuthology.parallel:result is None 2023-11-02T17:47:36.119 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-11-02T17:47:36.119 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-11-02T17:47:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2023-11-02T17:47:36.427 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-11-02T17:47:35.245535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81144,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314491656,"statfs":{"total":322122547200,"available":322039455744,"internally_reserved":0,"allocated":294912,"data_stored":72480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.435922"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.319222+0000","last_change":"2023-11-02T17:47:33.238355+0000","last_active":"2023-11-02T17:47:34.319222+0000","last_peered":"2023-11-02T17:47:34.319222+0000","last_clean":"2023-11-02T17:47:34.319222+0000","last_became_active":"2023-11-02T17:47:33.238020+0000","last_became_peered":"2023-11-02T17:47:33.238020+0000","last_unstale":"2023-11-02T17:47:34.319222+0000","last_undegraded":"2023-11-02T17:47:34.319222+0000","last_fullsized":"2023-11-02T17:47:34.319222+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T17:51:07.842360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.319470+0000","last_change":"2023-11-02T17:47:33.239294+0000","last_active":"2023-11-02T17:47:34.319470+0000","last_peered":"2023-11-02T17:47:34.319470+0000","last_clean":"2023-11-02T17:47:34.319470+0000","last_became_active":"2023-11-02T17:47:33.239017+0000","last_became_peered":"2023-11-02T17:47:33.239017+0000","last_unstale":"2023-11-02T17:47:34.319470+0000","last_undegraded":"2023-11-02T17:47:34.319470+0000","last_fullsized":"2023-11-02T17:47:34.319470+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T03:49:32.847365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.238117+0000","last_change":"2023-11-02T17:47:33.240804+0000","last_active":"2023-11-02T17:47:34.238117+0000","last_peered":"2023-11-02T17:47:34.238117+0000","last_clean":"2023-11-02T17:47:34.238117+0000","last_became_active":"2023-11-02T17:47:33.240445+0000","last_became_peered":"2023-11-02T17:47:33.240445+0000","last_unstale":"2023-11-02T17:47:34.238117+0000","last_undegraded":"2023-11-02T17:47:34.238117+0000","last_fullsized":"2023-11-02T17:47:34.238117+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T22:18:31.855384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.319841+0000","last_change":"2023-11-02T17:47:33.238717+0000","last_active":"2023-11-02T17:47:34.319841+0000","last_peered":"2023-11-02T17:47:34.319841+0000","last_clean":"2023-11-02T17:47:34.319841+0000","last_became_active":"2023-11-02T17:47:33.238446+0000","last_became_peered":"2023-11-02T17:47:33.238446+0000","last_unstale":"2023-11-02T17:47:34.319841+0000","last_undegraded":"2023-11-02T17:47:34.319841+0000","last_fullsized":"2023-11-02T17:47:34.319841+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T05:09:01.979938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'58","reported_seq":87,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.416890+0000","last_change":"2023-11-02T17:47:34.321292+0000","last_active":"2023-11-02T17:47:34.416890+0000","last_peered":"2023-11-02T17:47:34.416890+0000","last_clean":"2023-11-02T17:47:34.416890+0000","last_became_active":"2023-11-02T17:47:34.320949+0000","last_became_peered":"2023-11-02T17:47:34.320949+0000","last_unstale":"2023-11-02T17:47:34.416890+0000","last_undegraded":"2023-11-02T17:47:34.416890+0000","last_fullsized":"2023-11-02T17:47:34.416890+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:33.222683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:33.222683+0000","last_clean_scrub_stamp":"2023-11-02T17:47:33.222683+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T22:07:40.317679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.319486+0000","last_change":"2023-11-02T17:47:33.238428+0000","last_active":"2023-11-02T17:47:34.319486+0000","last_peered":"2023-11-02T17:47:34.319486+0000","last_clean":"2023-11-02T17:47:34.319486+0000","last_became_active":"2023-11-02T17:47:33.238078+0000","last_became_peered":"2023-11-02T17:47:33.238078+0000","last_unstale":"2023-11-02T17:47:34.319486+0000","last_undegraded":"2023-11-02T17:47:34.319486+0000","last_fullsized":"2023-11-02T17:47:34.319486+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T00:29:57.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.238921+0000","last_change":"2023-11-02T17:47:33.242060+0000","last_active":"2023-11-02T17:47:34.238921+0000","last_peered":"2023-11-02T17:47:34.238921+0000","last_clean":"2023-11-02T17:47:34.238921+0000","last_became_active":"2023-11-02T17:47:33.241799+0000","last_became_peered":"2023-11-02T17:47:33.241799+0000","last_unstale":"2023-11-02T17:47:34.238921+0000","last_undegraded":"2023-11-02T17:47:34.238921+0000","last_fullsized":"2023-11-02T17:47:34.238921+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T20:53:26.670399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.238193+0000","last_change":"2023-11-02T17:47:33.238344+0000","last_active":"2023-11-02T17:47:34.238193+0000","last_peered":"2023-11-02T17:47:34.238193+0000","last_clean":"2023-11-02T17:47:34.238193+0000","last_became_active":"2023-11-02T17:47:33.238060+0000","last_became_peered":"2023-11-02T17:47:33.238060+0000","last_unstale":"2023-11-02T17:47:34.238193+0000","last_undegraded":"2023-11-02T17:47:34.238193+0000","last_fullsized":"2023-11-02T17:47:34.238193+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T18:50:04.902879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.319576+0000","last_change":"2023-11-02T17:47:33.237696+0000","last_active":"2023-11-02T17:47:34.319576+0000","last_peered":"2023-11-02T17:47:34.319576+0000","last_clean":"2023-11-02T17:47:34.319576+0000","last_became_active":"2023-11-02T17:47:33.237262+0000","last_became_peered":"2023-11-02T17:47:33.237262+0000","last_unstale":"2023-11-02T17:47:34.319576+0000","last_undegraded":"2023-11-02T17:47:34.319576+0000","last_fullsized":"2023-11-02T17:47:34.319576+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T22:07:39.308069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-02T17:47:36.428 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-11-02T17:47:36.730 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:36.733 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2023-11-02T17:47:36.745 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-11-02T17:47:35.245535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81144,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314491656,"statfs":{"total":322122547200,"available":322039455744,"internally_reserved":0,"allocated":294912,"data_stored":72480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.435922"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.319222+0000","last_change":"2023-11-02T17:47:33.238355+0000","last_active":"2023-11-02T17:47:34.319222+0000","last_peered":"2023-11-02T17:47:34.319222+0000","last_clean":"2023-11-02T17:47:34.319222+0000","last_became_active":"2023-11-02T17:47:33.238020+0000","last_became_peered":"2023-11-02T17:47:33.238020+0000","last_unstale":"2023-11-02T17:47:34.319222+0000","last_undegraded":"2023-11-02T17:47:34.319222+0000","last_fullsized":"2023-11-02T17:47:34.319222+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T17:51:07.842360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.319470+0000","last_change":"2023-11-02T17:47:33.239294+0000","last_active":"2023-11-02T17:47:34.319470+0000","last_peered":"2023-11-02T17:47:34.319470+0000","last_clean":"2023-11-02T17:47:34.319470+0000","last_became_active":"2023-11-02T17:47:33.239017+0000","last_became_peered":"2023-11-02T17:47:33.239017+0000","last_unstale":"2023-11-02T17:47:34.319470+0000","last_undegraded":"2023-11-02T17:47:34.319470+0000","last_fullsized":"2023-11-02T17:47:34.319470+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T03:49:32.847365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.238117+0000","last_change":"2023-11-02T17:47:33.240804+0000","last_active":"2023-11-02T17:47:34.238117+0000","last_peered":"2023-11-02T17:47:34.238117+0000","last_clean":"2023-11-02T17:47:34.238117+0000","last_became_active":"2023-11-02T17:47:33.240445+0000","last_became_peered":"2023-11-02T17:47:33.240445+0000","last_unstale":"2023-11-02T17:47:34.238117+0000","last_undegraded":"2023-11-02T17:47:34.238117+0000","last_fullsized":"2023-11-02T17:47:34.238117+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T22:18:31.855384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.319841+0000","last_change":"2023-11-02T17:47:33.238717+0000","last_active":"2023-11-02T17:47:34.319841+0000","last_peered":"2023-11-02T17:47:34.319841+0000","last_clean":"2023-11-02T17:47:34.319841+0000","last_became_active":"2023-11-02T17:47:33.238446+0000","last_became_peered":"2023-11-02T17:47:33.238446+0000","last_unstale":"2023-11-02T17:47:34.319841+0000","last_undegraded":"2023-11-02T17:47:34.319841+0000","last_fullsized":"2023-11-02T17:47:34.319841+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T05:09:01.979938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'58","reported_seq":87,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.416890+0000","last_change":"2023-11-02T17:47:34.321292+0000","last_active":"2023-11-02T17:47:34.416890+0000","last_peered":"2023-11-02T17:47:34.416890+0000","last_clean":"2023-11-02T17:47:34.416890+0000","last_became_active":"2023-11-02T17:47:34.320949+0000","last_became_peered":"2023-11-02T17:47:34.320949+0000","last_unstale":"2023-11-02T17:47:34.416890+0000","last_undegraded":"2023-11-02T17:47:34.416890+0000","last_fullsized":"2023-11-02T17:47:34.416890+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:33.222683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:33.222683+0000","last_clean_scrub_stamp":"2023-11-02T17:47:33.222683+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T22:07:40.317679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.319486+0000","last_change":"2023-11-02T17:47:33.238428+0000","last_active":"2023-11-02T17:47:34.319486+0000","last_peered":"2023-11-02T17:47:34.319486+0000","last_clean":"2023-11-02T17:47:34.319486+0000","last_became_active":"2023-11-02T17:47:33.238078+0000","last_became_peered":"2023-11-02T17:47:33.238078+0000","last_unstale":"2023-11-02T17:47:34.319486+0000","last_undegraded":"2023-11-02T17:47:34.319486+0000","last_fullsized":"2023-11-02T17:47:34.319486+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T00:29:57.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.238921+0000","last_change":"2023-11-02T17:47:33.242060+0000","last_active":"2023-11-02T17:47:34.238921+0000","last_peered":"2023-11-02T17:47:34.238921+0000","last_clean":"2023-11-02T17:47:34.238921+0000","last_became_active":"2023-11-02T17:47:33.241799+0000","last_became_peered":"2023-11-02T17:47:33.241799+0000","last_unstale":"2023-11-02T17:47:34.238921+0000","last_undegraded":"2023-11-02T17:47:34.238921+0000","last_fullsized":"2023-11-02T17:47:34.238921+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T20:53:26.670399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.238193+0000","last_change":"2023-11-02T17:47:33.238344+0000","last_active":"2023-11-02T17:47:34.238193+0000","last_peered":"2023-11-02T17:47:34.238193+0000","last_clean":"2023-11-02T17:47:34.238193+0000","last_became_active":"2023-11-02T17:47:33.238060+0000","last_became_peered":"2023-11-02T17:47:33.238060+0000","last_unstale":"2023-11-02T17:47:34.238193+0000","last_undegraded":"2023-11-02T17:47:34.238193+0000","last_fullsized":"2023-11-02T17:47:34.238193+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T18:50:04.902879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-02T17:47:34.319576+0000","last_change":"2023-11-02T17:47:33.237696+0000","last_active":"2023-11-02T17:47:34.319576+0000","last_peered":"2023-11-02T17:47:34.319576+0000","last_clean":"2023-11-02T17:47:34.319576+0000","last_became_active":"2023-11-02T17:47:33.237262+0000","last_became_peered":"2023-11-02T17:47:33.237262+0000","last_unstale":"2023-11-02T17:47:34.319576+0000","last_undegraded":"2023-11-02T17:47:34.319576+0000","last_fullsized":"2023-11-02T17:47:34.319576+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:32.213073+0000","last_clean_scrub_stamp":"2023-11-02T17:47:32.213073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T22:07:39.308069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-02T17:47:36.746 INFO:tasks.ceph.ceph_manager.c1:clean! 2023-11-02T17:47:36.746 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2023-11-02T17:47:36.747 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2023-11-02T17:47:36.747 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2023-11-02T17:47:37.080 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.081 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-02T17:47:37.094 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2023-11-02T17:47:37.095 INFO:teuthology.run_tasks:Running task ceph... 2023-11-02T17:47:37.104 INFO:tasks.ceph:Creating ceph cluster c2... 2023-11-02T17:47:37.104 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-02T17:47:37.104 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849', 'branch': 'wip-yuri3-testing-2023-10-31-1116-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7442849', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2023-10-31-1116-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-yuri3-testing-2023-10-31-1116-reef', 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 7837, 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri3-testing-2023-10-31-1116-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec', 'targets': {'smithi017.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b8ceb241941938aa7529adcc4b99d42d196489ec', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-11-01_14:50:55', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-11-02T17:47:37.105 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-11-02T17:47:37.108 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-11-02T17:47:37.116 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/run/ceph 2023-11-02T17:47:37.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2023-11-02T17:47:37.174 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:37.175 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2023-11-02T17:47:37.223 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-02T17:47:37.223 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2023-11-02T17:47:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-02T17:47:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T17:47:37.230 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 360 Links: 1 2023-11-02T17:47:37.231 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T17:47:37.231 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-11-02 17:43:08.457207785 +0000 2023-11-02T17:47:37.231 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-11-02 17:43:08.165214519 +0000 2023-11-02T17:47:37.231 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-11-02 17:43:08.165214519 +0000 2023-11-02T17:47:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-11-02T17:47:37.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-02T17:47:37.243 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-11-02T17:47:37.243 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-11-02T17:47:37.244 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000398378 s, 1.3 MB/s 2023-11-02T17:47:37.245 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-02T17:47:37.297 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2023-11-02T17:47:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-02T17:47:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T17:47:37.305 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 364 Links: 1 2023-11-02T17:47:37.305 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T17:47:37.305 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-11-02 17:43:09.001195239 +0000 2023-11-02T17:47:37.305 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-11-02 17:43:08.701202157 +0000 2023-11-02T17:47:37.306 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-11-02 17:43:08.701202157 +0000 2023-11-02T17:47:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-11-02T17:47:37.306 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-02T17:47:37.363 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-11-02T17:47:37.363 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-11-02T17:47:37.363 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000402475 s, 1.3 MB/s 2023-11-02T17:47:37.364 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-02T17:47:37.373 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2023-11-02T17:47:37.378 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-02T17:47:37.379 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T17:47:37.379 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 371 Links: 1 2023-11-02T17:47:37.379 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T17:47:37.379 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-11-02 17:43:09.241189704 +0000 2023-11-02T17:47:37.379 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-11-02 17:43:09.237189796 +0000 2023-11-02T17:47:37.380 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-11-02 17:43:09.237189796 +0000 2023-11-02T17:47:37.380 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-11-02T17:47:37.380 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-02T17:47:37.434 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-11-02T17:47:37.434 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-11-02T17:47:37.434 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000392591 s, 1.3 MB/s 2023-11-02T17:47:37.435 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-02T17:47:37.443 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2023-11-02T17:47:37.449 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-02T17:47:37.449 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T17:47:37.449 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 379 Links: 1 2023-11-02T17:47:37.450 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T17:47:37.450 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-11-02 17:43:10.057170887 +0000 2023-11-02T17:47:37.450 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-11-02 17:43:09.793176975 +0000 2023-11-02T17:47:37.450 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-11-02 17:43:09.793176975 +0000 2023-11-02T17:47:37.450 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-11-02T17:47:37.451 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-02T17:47:37.505 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-11-02T17:47:37.505 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-11-02T17:47:37.505 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000410105 s, 1.2 MB/s 2023-11-02T17:47:37.506 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-02T17:47:37.515 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-02T17:47:37.516 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-02T17:47:37.516 INFO:tasks.ceph:Generating config... 2023-11-02T17:47:37.517 INFO:tasks.ceph:[client] debug rgw = 20 2023-11-02T17:47:37.518 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-11-02T17:47:37.518 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-11-02T17:47:37.518 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-11-02T17:47:37.518 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-11-02T17:47:37.519 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-11-02T17:47:37.519 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-11-02T17:47:37.519 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-11-02T17:47:37.519 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-11-02T17:47:37.520 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-11-02T17:47:37.520 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-11-02T17:47:37.520 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-11-02T17:47:37.521 INFO:tasks.ceph:[client] setgroup = ceph 2023-11-02T17:47:37.521 INFO:tasks.ceph:[client] setuser = ceph 2023-11-02T17:47:37.521 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-11-02T17:47:37.521 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-02T17:47:37.522 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-02T17:47:37.522 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-02T17:47:37.522 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-02T17:47:37.522 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-02T17:47:37.523 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-02T17:47:37.523 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-02T17:47:37.523 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-11-02T17:47:37.523 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-11-02T17:47:37.524 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-11-02T17:47:37.524 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-11-02T17:47:37.524 INFO:tasks.ceph:Setting up c2.mon.a... 2023-11-02T17:47:37.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2023-11-02T17:47:37.565 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/c2.keyring 2023-11-02T17:47:37.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2023-11-02T17:47:37.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-11-02T17:47:37.671 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.69')] 2023-11-02T17:47:37.672 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.69'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-11-02T17:47:37.673 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:37.673 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-02T17:47:37.679 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.69 --print /home/ubuntu/cephtest/c2.monmap 2023-11-02T17:47:37.758 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:47:37.758 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:47:37.759 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2023-11-02T17:47:37.760 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: generated fsid ade0ccbe-e106-4268-a9a3-f0934f1f75f2 2023-11-02T17:47:37.760 INFO:teuthology.orchestra.run.smithi069.stdout:setting min_mon_release = pacific 2023-11-02T17:47:37.760 INFO:teuthology.orchestra.run.smithi069.stdout:epoch 0 2023-11-02T17:47:37.760 INFO:teuthology.orchestra.run.smithi069.stdout:fsid ade0ccbe-e106-4268-a9a3-f0934f1f75f2 2023-11-02T17:47:37.760 INFO:teuthology.orchestra.run.smithi069.stdout:last_changed 2023-11-02T17:47:37.759296+0000 2023-11-02T17:47:37.761 INFO:teuthology.orchestra.run.smithi069.stdout:created 2023-11-02T17:47:37.759296+0000 2023-11-02T17:47:37.761 INFO:teuthology.orchestra.run.smithi069.stdout:min_mon_release 16 (pacific) 2023-11-02T17:47:37.761 INFO:teuthology.orchestra.run.smithi069.stdout:election_strategy: 1 2023-11-02T17:47:37.761 INFO:teuthology.orchestra.run.smithi069.stdout:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-11-02T17:47:37.761 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2023-11-02T17:47:37.763 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-02T17:47:37.770 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID ade0ccbe-e106-4268-a9a3-f0934f1f75f2... 2023-11-02T17:47:37.771 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-11-02T17:47:37.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-11-02T17:47:37.807 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2023-11-02T17:47:37.807 INFO:teuthology.orchestra.run.smithi017.stdout: chdir = "" 2023-11-02T17:47:37.807 INFO:teuthology.orchestra.run.smithi017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-02T17:47:37.808 INFO:teuthology.orchestra.run.smithi017.stdout: auth supported = cephx 2023-11-02T17:47:37.808 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.808 INFO:teuthology.orchestra.run.smithi017.stdout: filestore xattr use omap = true 2023-11-02T17:47:37.808 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.808 INFO:teuthology.orchestra.run.smithi017.stdout: mon clock drift allowed = 1.000 2023-11-02T17:47:37.809 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.809 INFO:teuthology.orchestra.run.smithi017.stdout: osd crush chooseleaf type = 0 2023-11-02T17:47:37.809 INFO:teuthology.orchestra.run.smithi017.stdout: auth debug = true 2023-11-02T17:47:37.809 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.809 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on old message = true 2023-11-02T17:47:37.810 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on bug = true 2023-11-02T17:47:37.810 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.810 INFO:teuthology.orchestra.run.smithi017.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-02T17:47:37.810 INFO:teuthology.orchestra.run.smithi017.stdout: mon pg warn max object skew = 0 2023-11-02T17:47:37.811 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.811 INFO:teuthology.orchestra.run.smithi017.stdout: # disable pg_autoscaler by default for new pools 2023-11-02T17:47:37.811 INFO:teuthology.orchestra.run.smithi017.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-02T17:47:37.811 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.811 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default size = 2 2023-11-02T17:47:37.812 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.812 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow primary affinity = true 2023-11-02T17:47:37.812 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow pg remap = true 2023-11-02T17:47:37.812 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on legacy crush tunables = false 2023-11-02T17:47:37.812 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on crush straw calc version zero = false 2023-11-02T17:47:37.813 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on no sortbitwise = false 2023-11-02T17:47:37.813 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on osd down out interval zero = false 2023-11-02T17:47:37.813 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on too few osds = false 2023-11-02T17:47:37.813 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-02T17:47:37.813 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-02T17:47:37.814 INFO:teuthology.orchestra.run.smithi017.stdout: mon_allow_pool_size_one = true 2023-11-02T17:47:37.814 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.814 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-02T17:47:37.814 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.814 INFO:teuthology.orchestra.run.smithi017.stdout: osd default data pool replay window = 5 2023-11-02T17:47:37.815 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.815 INFO:teuthology.orchestra.run.smithi017.stdout: mon allow pool delete = true 2023-11-02T17:47:37.815 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.815 INFO:teuthology.orchestra.run.smithi017.stdout: mon cluster log file level = debug 2023-11-02T17:47:37.815 INFO:teuthology.orchestra.run.smithi017.stdout: debug asserts on shutdown = true 2023-11-02T17:47:37.816 INFO:teuthology.orchestra.run.smithi017.stdout: mon health detail to clog = false 2023-11-02T17:47:37.816 INFO:teuthology.orchestra.run.smithi017.stdout: mon host = 172.21.15.69 2023-11-02T17:47:37.816 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = ade0ccbe-e106-4268-a9a3-f0934f1f75f2 2023-11-02T17:47:37.816 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.816 INFO:teuthology.orchestra.run.smithi017.stdout:[osd] 2023-11-02T17:47:37.817 INFO:teuthology.orchestra.run.smithi017.stdout: osd journal size = 100 2023-11-02T17:47:37.817 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.817 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub load threshold = 5.0 2023-11-02T17:47:37.817 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub max interval = 600 2023-11-02T17:47:37.817 INFO:teuthology.orchestra.run.smithi017.stdout: osd mclock profile = high_recovery_ops 2023-11-02T17:47:37.818 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.818 INFO:teuthology.orchestra.run.smithi017.stdout: osd recover clone overlap = true 2023-11-02T17:47:37.818 INFO:teuthology.orchestra.run.smithi017.stdout: osd recovery max chunk = 1048576 2023-11-02T17:47:37.818 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.818 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug shutdown = true 2023-11-02T17:47:37.819 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug op order = true 2023-11-02T17:47:37.819 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify stray on activate = true 2023-11-02T17:47:37.821 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.821 INFO:teuthology.orchestra.run.smithi017.stdout: osd open classes on start = true 2023-11-02T17:47:37.821 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug pg log writeout = true 2023-11-02T17:47:37.821 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.821 INFO:teuthology.orchestra.run.smithi017.stdout: osd deep scrub update digest min age = 30 2023-11-02T17:47:37.822 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.822 INFO:teuthology.orchestra.run.smithi017.stdout: osd map max advance = 10 2023-11-02T17:47:37.822 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.822 INFO:teuthology.orchestra.run.smithi017.stdout: journal zero on create = true 2023-11-02T17:47:37.823 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.823 INFO:teuthology.orchestra.run.smithi017.stdout: filestore ondisk finisher threads = 3 2023-11-02T17:47:37.823 INFO:teuthology.orchestra.run.smithi017.stdout: filestore apply finisher threads = 3 2023-11-02T17:47:37.823 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.823 INFO:teuthology.orchestra.run.smithi017.stdout: bdev debug aio = true 2023-11-02T17:47:37.824 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug misdirected ops = true 2023-11-02T17:47:37.824 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2023-11-02T17:47:37.824 INFO:teuthology.orchestra.run.smithi017.stdout: debug osd = 20 2023-11-02T17:47:37.824 INFO:teuthology.orchestra.run.smithi017.stdout: rocksdb delete range threshold = 0 2023-11-02T17:47:37.825 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.825 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr] 2023-11-02T17:47:37.825 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2023-11-02T17:47:37.825 INFO:teuthology.orchestra.run.smithi017.stdout: debug mgr = 20 2023-11-02T17:47:37.826 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2023-11-02T17:47:37.826 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2023-11-02T17:47:37.826 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2023-11-02T17:47:37.826 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2023-11-02T17:47:37.826 INFO:teuthology.orchestra.run.smithi017.stdout: mgr/telemetry/nag = false 2023-11-02T17:47:37.827 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.827 INFO:teuthology.orchestra.run.smithi017.stdout:[mon] 2023-11-02T17:47:37.827 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2023-11-02T17:47:37.827 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2023-11-02T17:47:37.828 INFO:teuthology.orchestra.run.smithi017.stdout: debug paxos = 20 2023-11-02T17:47:37.828 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2023-11-02T17:47:37.828 INFO:teuthology.orchestra.run.smithi017.stdout: mon data avail warn = 5 2023-11-02T17:47:37.828 INFO:teuthology.orchestra.run.smithi017.stdout: mon mgr mkfs grace = 240 2023-11-02T17:47:37.828 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2023-11-02T17:47:37.829 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd reporter subtree level = osd 2023-11-02T17:47:37.829 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd prime pg temp = true 2023-11-02T17:47:37.829 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2023-11-02T17:47:37.829 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.830 INFO:teuthology.orchestra.run.smithi017.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-02T17:47:37.830 INFO:teuthology.orchestra.run.smithi017.stdout: auth mon ticket ttl = 660 # 11m 2023-11-02T17:47:37.830 INFO:teuthology.orchestra.run.smithi017.stdout: auth service ticket ttl = 240 # 4m 2023-11-02T17:47:37.830 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.831 INFO:teuthology.orchestra.run.smithi017.stdout: # don't complain about insecure global_id in the test suite 2023-11-02T17:47:37.831 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-02T17:47:37.831 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-02T17:47:37.831 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.831 INFO:teuthology.orchestra.run.smithi017.stdout: # 1m isn't quite enough 2023-11-02T17:47:37.832 INFO:teuthology.orchestra.run.smithi017.stdout: mon_down_mkfs_grace = 2m 2023-11-02T17:47:37.832 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.832 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_filestore_osds = false 2023-11-02T17:47:37.832 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:47:37.832 INFO:teuthology.orchestra.run.smithi017.stdout:[client] 2023-11-02T17:47:37.833 INFO:teuthology.orchestra.run.smithi017.stdout: rgw cache enabled = true 2023-11-02T17:47:37.833 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable ops log = true 2023-11-02T17:47:37.833 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable usage log = true 2023-11-02T17:47:37.833 INFO:teuthology.orchestra.run.smithi017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-02T17:47:37.834 INFO:teuthology.orchestra.run.smithi017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-02T17:47:37.834 INFO:teuthology.orchestra.run.smithi017.stdout: debug rgw = 20 2023-11-02T17:47:37.834 INFO:teuthology.orchestra.run.smithi017.stdout: debug rgw sync = 20 2023-11-02T17:47:37.834 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt require ssl = False 2023-11-02T17:47:37.835 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt s3 kms backend = testing 2023-11-02T17:47:37.835 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-11-02T17:47:37.835 INFO:teuthology.orchestra.run.smithi017.stdout: rgw curl low speed time = 300 2023-11-02T17:47:37.835 INFO:teuthology.orchestra.run.smithi017.stdout: rgw data log num shards = 4 2023-11-02T17:47:37.835 INFO:teuthology.orchestra.run.smithi017.stdout: rgw md log max shards = 4 2023-11-02T17:47:37.836 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync data inject err probability = 0.1 2023-11-02T17:47:37.836 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync log trim interval = 0 2023-11-02T17:47:37.836 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync meta inject err probability = 0.1 2023-11-02T17:47:37.836 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync obj etag verify = True 2023-11-02T17:47:37.837 INFO:teuthology.orchestra.run.smithi017.stdout: setgroup = ceph 2023-11-02T17:47:37.837 INFO:teuthology.orchestra.run.smithi017.stdout: setuser = ceph 2023-11-02T17:47:37.837 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.a] 2023-11-02T17:47:37.837 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2023-11-02T17:47:37.837 INFO:teuthology.orchestra.run.smithi017.stdout: rgw data notify interval msec = 0 2023-11-02T17:47:37.838 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2023-11-02T17:47:37.838 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 10 2023-11-02T17:47:37.838 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2023-11-02T17:47:37.838 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 1000 2023-11-02T17:47:37.839 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2023-11-02T17:47:37.839 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 10000 2023-11-02T17:47:37.841 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2023-11-02T17:47:37.841 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2023-11-02T17:47:37.841 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-02T17:47:37.841 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2023-11-02T17:47:37.841 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.842 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2023-11-02T17:47:37.842 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.842 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2023-11-02T17:47:37.842 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.843 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2023-11-02T17:47:37.843 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2023-11-02T17:47:37.843 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.843 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2023-11-02T17:47:37.844 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2023-11-02T17:47:37.844 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.844 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-02T17:47:37.844 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2023-11-02T17:47:37.844 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.845 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2023-11-02T17:47:37.845 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-02T17:47:37.845 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.845 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2023-11-02T17:47:37.846 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.846 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2023-11-02T17:47:37.846 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2023-11-02T17:47:37.846 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2023-11-02T17:47:37.846 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2023-11-02T17:47:37.847 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2023-11-02T17:47:37.847 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2023-11-02T17:47:37.847 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2023-11-02T17:47:37.847 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-02T17:47:37.848 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-02T17:47:37.848 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2023-11-02T17:47:37.848 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.848 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-02T17:47:37.848 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.849 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2023-11-02T17:47:37.849 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.849 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2023-11-02T17:47:37.849 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.849 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2023-11-02T17:47:37.850 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2023-11-02T17:47:37.850 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2023-11-02T17:47:37.850 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = 172.21.15.69 2023-11-02T17:47:37.850 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = ade0ccbe-e106-4268-a9a3-f0934f1f75f2 2023-11-02T17:47:37.851 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.851 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2023-11-02T17:47:37.851 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2023-11-02T17:47:37.851 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.851 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2023-11-02T17:47:37.852 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2023-11-02T17:47:37.852 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2023-11-02T17:47:37.852 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.852 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2023-11-02T17:47:37.853 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2023-11-02T17:47:37.853 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.853 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2023-11-02T17:47:37.853 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2023-11-02T17:47:37.854 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2023-11-02T17:47:37.854 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.854 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2023-11-02T17:47:37.854 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2023-11-02T17:47:37.855 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.855 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2023-11-02T17:47:37.855 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.855 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2023-11-02T17:47:37.855 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.856 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2023-11-02T17:47:37.856 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.856 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2023-11-02T17:47:37.856 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2023-11-02T17:47:37.856 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.857 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2023-11-02T17:47:37.857 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2023-11-02T17:47:37.857 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-11-02T17:47:37.857 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2023-11-02T17:47:37.858 INFO:teuthology.orchestra.run.smithi069.stdout: rocksdb delete range threshold = 0 2023-11-02T17:47:37.858 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.858 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2023-11-02T17:47:37.858 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-11-02T17:47:37.858 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2023-11-02T17:47:37.859 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-11-02T17:47:37.859 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-11-02T17:47:37.859 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-11-02T17:47:37.859 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-11-02T17:47:37.860 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2023-11-02T17:47:37.860 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.860 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2023-11-02T17:47:37.860 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-11-02T17:47:37.860 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-11-02T17:47:37.861 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2023-11-02T17:47:37.861 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-11-02T17:47:37.861 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2023-11-02T17:47:37.861 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2023-11-02T17:47:37.862 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-11-02T17:47:37.862 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2023-11-02T17:47:37.862 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2023-11-02T17:47:37.862 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-11-02T17:47:37.862 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.863 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-02T17:47:37.863 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2023-11-02T17:47:37.863 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2023-11-02T17:47:37.863 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.863 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2023-11-02T17:47:37.864 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-02T17:47:37.864 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-02T17:47:37.864 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.864 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2023-11-02T17:47:37.864 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2023-11-02T17:47:37.865 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.865 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2023-11-02T17:47:37.865 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:37.865 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2023-11-02T17:47:37.866 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2023-11-02T17:47:37.866 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2023-11-02T17:47:37.866 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2023-11-02T17:47:37.866 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-02T17:47:37.866 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-02T17:47:37.867 INFO:teuthology.orchestra.run.smithi069.stdout: debug rgw = 20 2023-11-02T17:47:37.867 INFO:teuthology.orchestra.run.smithi069.stdout: debug rgw sync = 20 2023-11-02T17:47:37.867 INFO:teuthology.orchestra.run.smithi069.stdout: rgw crypt require ssl = False 2023-11-02T17:47:37.867 INFO:teuthology.orchestra.run.smithi069.stdout: rgw crypt s3 kms backend = testing 2023-11-02T17:47:37.868 INFO:teuthology.orchestra.run.smithi069.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-11-02T17:47:37.868 INFO:teuthology.orchestra.run.smithi069.stdout: rgw curl low speed time = 300 2023-11-02T17:47:37.868 INFO:teuthology.orchestra.run.smithi069.stdout: rgw data log num shards = 4 2023-11-02T17:47:37.868 INFO:teuthology.orchestra.run.smithi069.stdout: rgw md log max shards = 4 2023-11-02T17:47:37.868 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync data inject err probability = 0.1 2023-11-02T17:47:37.869 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync log trim interval = 0 2023-11-02T17:47:37.869 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync meta inject err probability = 0.1 2023-11-02T17:47:37.869 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync obj etag verify = True 2023-11-02T17:47:37.869 INFO:teuthology.orchestra.run.smithi069.stdout: setgroup = ceph 2023-11-02T17:47:37.869 INFO:teuthology.orchestra.run.smithi069.stdout: setuser = ceph 2023-11-02T17:47:37.870 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2023-11-02T17:47:37.870 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2023-11-02T17:47:37.870 INFO:teuthology.orchestra.run.smithi069.stdout: rgw data notify interval msec = 0 2023-11-02T17:47:37.870 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2023-11-02T17:47:37.871 INFO:teuthology.orchestra.run.smithi069.stdout: osd_max_omap_entries_per_request = 10 2023-11-02T17:47:37.871 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2023-11-02T17:47:37.871 INFO:teuthology.orchestra.run.smithi069.stdout: osd_max_omap_entries_per_request = 1000 2023-11-02T17:47:37.871 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2023-11-02T17:47:37.871 INFO:teuthology.orchestra.run.smithi069.stdout: osd_max_omap_entries_per_request = 10000 2023-11-02T17:47:37.872 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2023-11-02T17:47:37.872 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2023-11-02T17:47:37.940 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-02T17:47:37.941 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:37.941 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2023-11-02T17:47:37.949 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:37.949 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2023-11-02T17:47:37.956 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi017.front.sepia.ceph.com 2023-11-02T17:47:37.956 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:37.957 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/c2.keyring 2023-11-02T17:47:37.957 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-11-02T17:47:37.985 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-11-02T17:47:37.985 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-11-02T17:47:38.035 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2023-11-02T17:47:38.036 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:38.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/c2.keyring 2023-11-02T17:47:38.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-11-02T17:47:38.060 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:38.060 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-11-02T17:47:38.065 INFO:tasks.ceph:Setting up mon nodes... 2023-11-02T17:47:38.066 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-02T17:47:38.066 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2023-11-02T17:47:38.116 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2023-11-02T17:47:38.119 INFO:tasks.ceph:Setting up mds nodes... 2023-11-02T17:47:38.119 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-02T17:47:38.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2023-11-02T17:47:38.159 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/c2.client.0.keyring 2023-11-02T17:47:38.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2023-11-02T17:47:38.256 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/c2.client.1.keyring 2023-11-02T17:47:38.270 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-02T17:47:38.271 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-02T17:47:38.271 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2023-11-02T17:47:38.282 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-02T17:47:38.282 INFO:tasks.ceph:role: c2.osd.0 2023-11-02T17:47:38.282 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-02T17:47:38.283 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-02T17:47:38.812 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T17:47:38.813 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T17:47:38.813 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T17:47:38.813 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 2023-11-02T17:47:38.813 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T17:47:38.813 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-11-02T17:47:38.814 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T17:47:38.814 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-02T17:47:38.814 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T17:47:38.814 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T17:47:38.815 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-11-02T17:47:38.816 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2023-11-02T17:47:38.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2023-11-02T17:47:38.929 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: /sbin/restorecon: command not found 2023-11-02T17:47:38.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T17:47:38.931 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2023-11-02T17:47:38.988 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-02T17:47:38.988 INFO:tasks.ceph:role: c2.osd.1 2023-11-02T17:47:38.988 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-02T17:47:38.989 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-02T17:47:39.486 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T17:47:39.487 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T17:47:39.487 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T17:47:39.487 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 2023-11-02T17:47:39.487 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T17:47:39.487 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-11-02T17:47:39.488 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T17:47:39.488 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-02T17:47:39.488 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T17:47:39.488 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T17:47:39.489 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-11-02T17:47:39.490 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2023-11-02T17:47:39.524 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2023-11-02T17:47:39.580 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: /sbin/restorecon: command not found 2023-11-02T17:47:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T17:47:39.582 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2023-11-02T17:47:39.596 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-02T17:47:39.596 INFO:tasks.ceph:role: c2.osd.2 2023-11-02T17:47:39.596 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-02T17:47:39.597 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-02T17:47:40.061 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T17:47:40.062 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T17:47:40.062 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T17:47:40.062 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 2023-11-02T17:47:40.063 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T17:47:40.063 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-11-02T17:47:40.063 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T17:47:40.063 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-02T17:47:40.063 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T17:47:40.064 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T17:47:40.065 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-11-02T17:47:40.065 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2023-11-02T17:47:40.117 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2023-11-02T17:47:40.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T17:47:40.127 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: /sbin/restorecon: command not found 2023-11-02T17:47:40.127 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2023-11-02T17:47:40.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:40.194+0000 7f527f823700 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2023-11-02T17:47:40.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:40.194+0000 7f527f823700 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2023-11-02T17:47:40.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:40.194+0000 7f527f823700 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-11-02T17:47:40.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:40.194+0000 7f527f823700 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-11-02T17:47:40.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:40.194+0000 7f527f823700 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-11-02T17:47:40.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:40.202+0000 7f527f823700 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2023-11-02T17:47:42.668 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2023-11-02T17:47:42.683 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2023-11-02T17:47:42.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:42.722+0000 7f5ea4f2f700 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2023-11-02T17:47:42.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:42.722+0000 7f5ea4f2f700 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2023-11-02T17:47:42.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:42.722+0000 7f5ea4f2f700 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-11-02T17:47:42.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:42.726+0000 7f5ea4f2f700 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-11-02T17:47:42.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:42.726+0000 7f5ea4f2f700 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-11-02T17:47:42.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:42.726+0000 7f5ea4f2f700 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2023-11-02T17:47:44.940 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2023-11-02T17:47:44.953 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2023-11-02T17:47:45.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:45.038+0000 7f83a9139700 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2023-11-02T17:47:45.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:45.038+0000 7f83a9139700 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2023-11-02T17:47:45.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:45.038+0000 7f83a9139700 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-11-02T17:47:45.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:45.038+0000 7f83a9139700 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-11-02T17:47:45.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:45.038+0000 7f83a9139700 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-11-02T17:47:45.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:47:45.042+0000 7f83a9139700 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2023-11-02T17:47:47.251 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2023-11-02T17:47:47.267 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-02T17:47:47.267 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:47.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2023-11-02T17:47:47.279 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:47.279 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2023-11-02T17:47:47.336 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:47.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2023-11-02T17:47:47.347 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:47.347 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2023-11-02T17:47:47.361 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:47.361 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2023-11-02T17:47:47.411 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:47.411 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2023-11-02T17:47:47.418 INFO:tasks.ceph:Adding keys to all mons... 2023-11-02T17:47:47.418 DEBUG:teuthology.orchestra.run.smithi069:> sudo tee -a /etc/ceph/c2.keyring 2023-11-02T17:47:47.431 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.x] 2023-11-02T17:47:47.431 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC64ENlk8f5BhAAWBFjUR+2fy6KQEEvvCd6+Q== 2023-11-02T17:47:47.431 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2023-11-02T17:47:47.432 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC84ENlLg/qCxAAt7Ub5wiO5WCrlaQi5vHJlg== 2023-11-02T17:47:47.432 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2023-11-02T17:47:47.432 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC+4ENlrWd2KxAAuZR3AJxKAiXzY1NNnax6Lg== 2023-11-02T17:47:47.432 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2023-11-02T17:47:47.432 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDB4ENl8pynAhAAp52d17a7pWyZ/siNe1zHBg== 2023-11-02T17:47:47.433 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2023-11-02T17:47:47.433 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC64ENli2iCCRAAI/GpYImzvJwcII9Z0Pmgfw== 2023-11-02T17:47:47.433 INFO:teuthology.orchestra.run.smithi069.stdout:[client.1] 2023-11-02T17:47:47.433 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC64ENlC4VWDxAAuFzNHIaGYtfperj+z7Az9A== 2023-11-02T17:47:47.434 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-02T17:47:47.530 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T17:47:47.574 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T17:47:47.619 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T17:47:47.665 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-02T17:47:47.711 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-02T17:47:47.759 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-02T17:47:47.760 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2023-11-02T17:47:47.815 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2023-11-02T17:47:47.881 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2023-11-02T17:47:47.893 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/c2.monmap 2023-11-02T17:47:47.942 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2023-11-02T17:47:47.943 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2023-11-02T17:47:47.943 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2023-11-02T17:47:47.946 INFO:tasks.ceph.c2.mon.a:Started 2023-11-02T17:47:47.946 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2023-11-02T17:47:47.946 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2023-11-02T17:47:47.948 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2023-11-02T17:47:47.950 INFO:tasks.ceph.c2.mgr.x:Started 2023-11-02T17:47:47.950 INFO:tasks.ceph:Setting crush tunables to default 2023-11-02T17:47:47.951 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c2 osd crush tunables default 2023-11-02T17:47:48.220 INFO:teuthology.orchestra.run.smithi069.stderr:adjusted tunables profile to default 2023-11-02T17:47:48.231 INFO:tasks.ceph:check_enable_crimson: False 2023-11-02T17:47:48.231 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2023-11-02T17:47:48.232 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:48.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2023-11-02T17:47:48.243 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:48.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2023-11-02T17:47:48.254 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-02T17:47:48.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2023-11-02T17:47:48.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c2 osd new 6622f393-d74e-47ac-b95b-372a0d369b90 0 2023-11-02T17:47:48.342 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:48.338+0000 7f4aef3f22c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-02T17:47:48.515 INFO:teuthology.orchestra.run.smithi069.stdout:0 2023-11-02T17:47:48.527 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c2 osd new 8c8b024d-4975-4417-a302-98b51ccc2050 1 2023-11-02T17:47:48.725 INFO:teuthology.orchestra.run.smithi069.stdout:1 2023-11-02T17:47:48.738 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c2 osd new 73307c71-5662-4649-91b5-eaf57e478f9d 2 2023-11-02T17:47:48.940 INFO:teuthology.orchestra.run.smithi069.stdout:2 2023-11-02T17:47:48.957 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2023-11-02T17:47:48.958 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2023-11-02T17:47:48.960 INFO:tasks.ceph.c2.osd.0:Started 2023-11-02T17:47:48.960 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2023-11-02T17:47:48.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2023-11-02T17:47:48.963 INFO:tasks.ceph.c2.osd.1:Started 2023-11-02T17:47:48.963 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2023-11-02T17:47:48.963 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2023-11-02T17:47:48.965 INFO:tasks.ceph.c2.osd.2:Started 2023-11-02T17:47:48.966 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-11-02T17:47:48.991 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:48.986+0000 7f4aef3f22c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-02T17:47:49.064 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:49.058+0000 7f4aef3f22c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-02T17:47:49.162 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:47:49.162 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":5,"fsid":"ade0ccbe-e106-4268-a9a3-f0934f1f75f2","created":"2023-11-02T17:47:48.098630+0000","modified":"2023-11-02T17:47:48.937072+0000","last_up_change":"0.000000","last_in_change":"2023-11-02T17:47:48.937072+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6622f393-d74e-47ac-b95b-372a0d369b90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8c8b024d-4975-4417-a302-98b51ccc2050","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"73307c71-5662-4649-91b5-eaf57e478f9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T17:47:49.175 INFO:tasks.ceph.ceph_manager.c2:[] 2023-11-02T17:47:49.175 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-02T17:47:49.221 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:49.218+0000 7f4aef3f22c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-02T17:47:49.322 INFO:tasks.ceph.c2.osd.0.smithi069.stderr:2023-11-02T17:47:49.318+0000 7f5163cbd700 -1 Falling back to public interface 2023-11-02T17:47:49.430 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:49.426+0000 7f4aef3f22c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-02T17:47:49.478 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-11-02T17:47:49.479 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:49.474+0000 7f4aef3f22c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-02T17:47:49.598 INFO:tasks.ceph.c2.osd.2.smithi069.stderr:2023-11-02T17:47:49.594+0000 7fb616030700 -1 Falling back to public interface 2023-11-02T17:47:49.606 INFO:tasks.ceph.c2.osd.1.smithi069.stderr:2023-11-02T17:47:49.602+0000 7f847b277700 -1 Falling back to public interface 2023-11-02T17:47:49.655 INFO:teuthology.misc.health.smithi069.stdout: 2023-11-02T17:47:49.655 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":5,"fsid":"ade0ccbe-e106-4268-a9a3-f0934f1f75f2","created":"2023-11-02T17:47:48.098630+0000","modified":"2023-11-02T17:47:48.937072+0000","last_up_change":"0.000000","last_in_change":"2023-11-02T17:47:48.937072+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6622f393-d74e-47ac-b95b-372a0d369b90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8c8b024d-4975-4417-a302-98b51ccc2050","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"73307c71-5662-4649-91b5-eaf57e478f9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T17:47:49.656 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:49.650+0000 7f4aef3f22c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-02T17:47:49.665 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-02T17:47:49.718 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:49.714+0000 7f4aef3f22c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-02T17:47:49.780 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:49.774+0000 7f4aef3f22c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-02T17:47:50.177 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:50.174+0000 7f4aef3f22c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-02T17:47:50.294 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:50.290+0000 7f4aef3f22c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-02T17:47:50.618 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:50.614+0000 7f4aef3f22c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-02T17:47:50.788 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:50.782+0000 7f4aef3f22c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-02T17:47:50.855 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:50.850+0000 7f4aef3f22c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-02T17:47:50.909 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:50.906+0000 7f4aef3f22c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-02T17:47:51.733 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:51.730+0000 7f4aef3f22c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-02T17:47:51.801 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:51.798+0000 7f4aef3f22c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-02T17:47:51.951 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:51.946+0000 7f4aef3f22c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-02T17:47:52.023 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:52.018+0000 7f4aef3f22c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-02T17:47:52.093 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:52.090+0000 7f4aef3f22c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-02T17:47:52.120 INFO:tasks.ceph.c2.osd.2.smithi069.stderr:2023-11-02T17:47:52.114+0000 7fb616030700 -1 osd.2 0 log_to_monitors true 2023-11-02T17:47:52.269 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:52.262+0000 7f4aef3f22c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-02T17:47:52.335 INFO:tasks.ceph.c2.osd.0.smithi069.stderr:2023-11-02T17:47:52.330+0000 7f5163cbd700 -1 osd.0 0 log_to_monitors true 2023-11-02T17:47:52.408 INFO:tasks.ceph.c2.osd.1.smithi069.stderr:2023-11-02T17:47:52.402+0000 7f847b277700 -1 osd.1 0 log_to_monitors true 2023-11-02T17:47:52.493 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:52.490+0000 7f4aef3f22c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-02T17:47:53.161 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:53.158+0000 7f4aef3f22c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-02T17:47:53.963 INFO:tasks.ceph.c2.osd.2.smithi069.stderr:2023-11-02T17:47:53.958+0000 7fb611e6f700 -1 osd.2 0 waiting for initial osdmap 2023-11-02T17:47:53.966 INFO:tasks.ceph.c2.osd.2.smithi069.stderr:2023-11-02T17:47:53.962+0000 7fb60991e700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T17:47:54.223 INFO:tasks.ceph.c2.mgr.x.smithi069.stderr:2023-11-02T17:47:54.218+0000 7f4ad6474700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-02T17:47:55.269 INFO:tasks.ceph.c2.osd.0.smithi069.stderr:2023-11-02T17:47:55.262+0000 7f5160483700 -1 osd.0 0 waiting for initial osdmap 2023-11-02T17:47:55.275 INFO:tasks.ceph.c2.osd.0.smithi069.stderr:2023-11-02T17:47:55.270+0000 7f51575ab700 -1 osd.0 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T17:47:55.306 INFO:tasks.ceph.c2.osd.1.smithi069.stderr:2023-11-02T17:47:55.302+0000 7f84770b6700 -1 osd.1 0 waiting for initial osdmap 2023-11-02T17:47:55.310 INFO:tasks.ceph.c2.osd.1.smithi069.stderr:2023-11-02T17:47:55.306+0000 7f846eb65700 -1 osd.1 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T17:47:55.968 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-11-02T17:47:56.264 INFO:teuthology.misc.health.smithi069.stdout: 2023-11-02T17:47:56.264 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":9,"fsid":"ade0ccbe-e106-4268-a9a3-f0934f1f75f2","created":"2023-11-02T17:47:48.098630+0000","modified":"2023-11-02T17:47:55.235823+0000","last_up_change":"2023-11-02T17:47:54.183318+0000","last_in_change":"2023-11-02T17:47:48.937072+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6622f393-d74e-47ac-b95b-372a0d369b90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8c8b024d-4975-4417-a302-98b51ccc2050","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"73307c71-5662-4649-91b5-eaf57e478f9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":29154},{"type":"v1","addr":"172.21.15.69:6809","nonce":29154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":29154},{"type":"v1","addr":"172.21.15.69:6811","nonce":29154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":29154},{"type":"v1","addr":"172.21.15.69:6815","nonce":29154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":29154},{"type":"v1","addr":"172.21.15.69:6813","nonce":29154}]},"public_addr":"172.21.15.69:6809/29154","cluster_addr":"172.21.15.69:6811/29154","heartbeat_back_addr":"172.21.15.69:6815/29154","heartbeat_front_addr":"172.21.15.69:6813/29154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:53.135153+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T17:47:56.283 DEBUG:teuthology.misc:1 of 3 OSDs are up 2023-11-02T17:48:02.586 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-11-02T17:48:02.879 INFO:teuthology.misc.health.smithi069.stdout: 2023-11-02T17:48:02.880 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":13,"fsid":"ade0ccbe-e106-4268-a9a3-f0934f1f75f2","created":"2023-11-02T17:47:48.098630+0000","modified":"2023-11-02T17:47:59.277418+0000","last_up_change":"2023-11-02T17:47:56.270462+0000","last_in_change":"2023-11-02T17:47:48.937072+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-02T17:47:57.272825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6622f393-d74e-47ac-b95b-372a0d369b90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":29152},{"type":"v1","addr":"172.21.15.69:6801","nonce":29152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":29152},{"type":"v1","addr":"172.21.15.69:6803","nonce":29152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":29152},{"type":"v1","addr":"172.21.15.69:6807","nonce":29152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":29152},{"type":"v1","addr":"172.21.15.69:6805","nonce":29152}]},"public_addr":"172.21.15.69:6801/29152","cluster_addr":"172.21.15.69:6803/29152","heartbeat_back_addr":"172.21.15.69:6807/29152","heartbeat_front_addr":"172.21.15.69:6805/29152","state":["exists","up"]},{"osd":1,"uuid":"8c8b024d-4975-4417-a302-98b51ccc2050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":29153},{"type":"v1","addr":"172.21.15.69:6817","nonce":29153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":29153},{"type":"v1","addr":"172.21.15.69:6819","nonce":29153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":29153},{"type":"v1","addr":"172.21.15.69:6823","nonce":29153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":29153},{"type":"v1","addr":"172.21.15.69:6821","nonce":29153}]},"public_addr":"172.21.15.69:6817/29153","cluster_addr":"172.21.15.69:6819/29153","heartbeat_back_addr":"172.21.15.69:6823/29153","heartbeat_front_addr":"172.21.15.69:6821/29153","state":["exists","up"]},{"osd":2,"uuid":"73307c71-5662-4649-91b5-eaf57e478f9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":29154},{"type":"v1","addr":"172.21.15.69:6809","nonce":29154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":29154},{"type":"v1","addr":"172.21.15.69:6811","nonce":29154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":29154},{"type":"v1","addr":"172.21.15.69:6815","nonce":29154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":29154},{"type":"v1","addr":"172.21.15.69:6813","nonce":29154}]},"public_addr":"172.21.15.69:6809/29154","cluster_addr":"172.21.15.69:6811/29154","heartbeat_back_addr":"172.21.15.69:6815/29154","heartbeat_front_addr":"172.21.15.69:6813/29154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:53.359371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:53.377615+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:53.135153+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T17:48:02.892 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-02T17:48:02.893 INFO:tasks.ceph:Creating RBD pool 2023-11-02T17:48:02.893 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c2 osd pool create rbd 8 2023-11-02T17:48:04.080 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'rbd' created 2023-11-02T17:48:04.092 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-02T17:48:05.088 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-02T17:48:05.101 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2023-11-02T17:48:05.102 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2023-11-02T17:48:05.102 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2023-11-02T17:48:05.103 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2023-11-02T17:48:05.103 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-02T17:48:05.435 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:48:05.452 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":28968},{"type":"v1","addr":"172.21.15.69:6825","nonce":28968}]},"active_addr":"172.21.15.69:6825/28968","active_change":"2023-11-02T17:47:53.187871+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":992579901}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":1869354412}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":957674851}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":3925772015}]}]} 2023-11-02T17:48:05.456 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2023-11-02T17:48:05.456 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2023-11-02T17:48:05.456 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-11-02T17:48:05.753 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:48:05.754 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":15,"fsid":"ade0ccbe-e106-4268-a9a3-f0934f1f75f2","created":"2023-11-02T17:47:48.098630+0000","modified":"2023-11-02T17:48:05.068313+0000","last_up_change":"2023-11-02T17:47:56.270462+0000","last_in_change":"2023-11-02T17:47:48.937072+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-02T17:47:57.272825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-02T17:48:03.230265+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6622f393-d74e-47ac-b95b-372a0d369b90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":29152},{"type":"v1","addr":"172.21.15.69:6801","nonce":29152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":29152},{"type":"v1","addr":"172.21.15.69:6803","nonce":29152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":29152},{"type":"v1","addr":"172.21.15.69:6807","nonce":29152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":29152},{"type":"v1","addr":"172.21.15.69:6805","nonce":29152}]},"public_addr":"172.21.15.69:6801/29152","cluster_addr":"172.21.15.69:6803/29152","heartbeat_back_addr":"172.21.15.69:6807/29152","heartbeat_front_addr":"172.21.15.69:6805/29152","state":["exists","up"]},{"osd":1,"uuid":"8c8b024d-4975-4417-a302-98b51ccc2050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":29153},{"type":"v1","addr":"172.21.15.69:6817","nonce":29153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":29153},{"type":"v1","addr":"172.21.15.69:6819","nonce":29153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":29153},{"type":"v1","addr":"172.21.15.69:6823","nonce":29153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":29153},{"type":"v1","addr":"172.21.15.69:6821","nonce":29153}]},"public_addr":"172.21.15.69:6817/29153","cluster_addr":"172.21.15.69:6819/29153","heartbeat_back_addr":"172.21.15.69:6823/29153","heartbeat_front_addr":"172.21.15.69:6821/29153","state":["exists","up"]},{"osd":2,"uuid":"73307c71-5662-4649-91b5-eaf57e478f9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":29154},{"type":"v1","addr":"172.21.15.69:6809","nonce":29154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":29154},{"type":"v1","addr":"172.21.15.69:6811","nonce":29154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":29154},{"type":"v1","addr":"172.21.15.69:6815","nonce":29154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":29154},{"type":"v1","addr":"172.21.15.69:6813","nonce":29154}]},"public_addr":"172.21.15.69:6809/29154","cluster_addr":"172.21.15.69:6811/29154","heartbeat_back_addr":"172.21.15.69:6815/29154","heartbeat_front_addr":"172.21.15.69:6813/29154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:53.359371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:53.377615+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:53.135153+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T17:48:05.767 INFO:tasks.ceph.ceph_manager.c2:all up! 2023-11-02T17:48:05.767 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-11-02T17:48:06.063 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:48:06.063 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":15,"fsid":"ade0ccbe-e106-4268-a9a3-f0934f1f75f2","created":"2023-11-02T17:47:48.098630+0000","modified":"2023-11-02T17:48:05.068313+0000","last_up_change":"2023-11-02T17:47:56.270462+0000","last_in_change":"2023-11-02T17:47:48.937072+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-02T17:47:57.272825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-02T17:48:03.230265+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6622f393-d74e-47ac-b95b-372a0d369b90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":29152},{"type":"v1","addr":"172.21.15.69:6801","nonce":29152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":29152},{"type":"v1","addr":"172.21.15.69:6803","nonce":29152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":29152},{"type":"v1","addr":"172.21.15.69:6807","nonce":29152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":29152},{"type":"v1","addr":"172.21.15.69:6805","nonce":29152}]},"public_addr":"172.21.15.69:6801/29152","cluster_addr":"172.21.15.69:6803/29152","heartbeat_back_addr":"172.21.15.69:6807/29152","heartbeat_front_addr":"172.21.15.69:6805/29152","state":["exists","up"]},{"osd":1,"uuid":"8c8b024d-4975-4417-a302-98b51ccc2050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":29153},{"type":"v1","addr":"172.21.15.69:6817","nonce":29153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":29153},{"type":"v1","addr":"172.21.15.69:6819","nonce":29153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":29153},{"type":"v1","addr":"172.21.15.69:6823","nonce":29153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":29153},{"type":"v1","addr":"172.21.15.69:6821","nonce":29153}]},"public_addr":"172.21.15.69:6817/29153","cluster_addr":"172.21.15.69:6819/29153","heartbeat_back_addr":"172.21.15.69:6823/29153","heartbeat_front_addr":"172.21.15.69:6821/29153","state":["exists","up"]},{"osd":2,"uuid":"73307c71-5662-4649-91b5-eaf57e478f9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":29154},{"type":"v1","addr":"172.21.15.69:6809","nonce":29154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":29154},{"type":"v1","addr":"172.21.15.69:6811","nonce":29154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":29154},{"type":"v1","addr":"172.21.15.69:6815","nonce":29154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":29154},{"type":"v1","addr":"172.21.15.69:6813","nonce":29154}]},"public_addr":"172.21.15.69:6809/29154","cluster_addr":"172.21.15.69:6811/29154","heartbeat_back_addr":"172.21.15.69:6815/29154","heartbeat_front_addr":"172.21.15.69:6813/29154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:53.359371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:53.377615+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T17:47:53.135153+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T17:48:06.078 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2023-11-02T17:48:06.079 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2023-11-02T17:48:06.080 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2023-11-02T17:48:06.220 INFO:teuthology.orchestra.run.smithi069.stdout:42949672965 2023-11-02T17:48:06.221 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-11-02T17:48:06.228 INFO:teuthology.orchestra.run.smithi069.stdout:34359738372 2023-11-02T17:48:06.229 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-11-02T17:48:06.230 INFO:teuthology.orchestra.run.smithi069.stdout:42949672964 2023-11-02T17:48:06.230 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-11-02T17:48:06.507 INFO:teuthology.orchestra.run.smithi069.stdout:42949672964 2023-11-02T17:48:06.520 INFO:tasks.ceph.ceph_manager.c2:need seq 42949672965 got 42949672964 for osd.1 2023-11-02T17:48:06.553 INFO:teuthology.orchestra.run.smithi069.stdout:34359738371 2023-11-02T17:48:06.565 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.2 2023-11-02T17:48:06.567 INFO:teuthology.orchestra.run.smithi069.stdout:42949672963 2023-11-02T17:48:06.581 INFO:tasks.ceph.ceph_manager.c2:need seq 42949672964 got 42949672963 for osd.0 2023-11-02T17:48:07.522 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-11-02T17:48:07.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-11-02T17:48:07.582 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-11-02T17:48:07.842 INFO:teuthology.orchestra.run.smithi069.stdout:42949672964 2023-11-02T17:48:07.860 INFO:teuthology.orchestra.run.smithi069.stdout:34359738371 2023-11-02T17:48:07.861 INFO:tasks.ceph.ceph_manager.c2:need seq 42949672965 got 42949672964 for osd.1 2023-11-02T17:48:07.868 INFO:teuthology.orchestra.run.smithi069.stdout:42949672963 2023-11-02T17:48:07.873 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.2 2023-11-02T17:48:07.882 INFO:tasks.ceph.ceph_manager.c2:need seq 42949672964 got 42949672963 for osd.0 2023-11-02T17:48:08.862 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-11-02T17:48:08.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-11-02T17:48:08.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-11-02T17:48:09.166 INFO:teuthology.orchestra.run.smithi069.stdout:42949672965 2023-11-02T17:48:09.179 INFO:tasks.ceph.ceph_manager.c2:need seq 42949672965 got 42949672965 for osd.1 2023-11-02T17:48:09.179 DEBUG:teuthology.parallel:result is None 2023-11-02T17:48:09.202 INFO:teuthology.orchestra.run.smithi069.stdout:42949672964 2023-11-02T17:48:09.213 INFO:teuthology.orchestra.run.smithi069.stdout:34359738372 2023-11-02T17:48:09.216 INFO:tasks.ceph.ceph_manager.c2:need seq 42949672964 got 42949672964 for osd.0 2023-11-02T17:48:09.216 DEBUG:teuthology.parallel:result is None 2023-11-02T17:48:09.228 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2023-11-02T17:48:09.228 DEBUG:teuthology.parallel:result is None 2023-11-02T17:48:09.229 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-11-02T17:48:09.229 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-11-02T17:48:09.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:48:09.518 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-11-02T17:48:09.528 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2023-11-02T17:48:09.217086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82404,"kb_used_data":1548,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490396,"statfs":{"total":322122547200,"available":322038165504,"internally_reserved":0,"allocated":1585152,"data_stored":1289983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001500"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.099421+0000","last_change":"2023-11-02T17:48:05.094862+0000","last_active":"2023-11-02T17:48:05.099421+0000","last_peered":"2023-11-02T17:48:05.099421+0000","last_clean":"2023-11-02T17:48:05.099421+0000","last_became_active":"2023-11-02T17:48:05.094619+0000","last_became_peered":"2023-11-02T17:48:05.094619+0000","last_unstale":"2023-11-02T17:48:05.099421+0000","last_undegraded":"2023-11-02T17:48:05.099421+0000","last_fullsized":"2023-11-02T17:48:05.099421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T19:31:05.400708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.097030+0000","last_change":"2023-11-02T17:48:05.092199+0000","last_active":"2023-11-02T17:48:05.097030+0000","last_peered":"2023-11-02T17:48:05.097030+0000","last_clean":"2023-11-02T17:48:05.097030+0000","last_became_active":"2023-11-02T17:48:05.091877+0000","last_became_peered":"2023-11-02T17:48:05.091877+0000","last_unstale":"2023-11-02T17:48:05.097030+0000","last_undegraded":"2023-11-02T17:48:05.097030+0000","last_fullsized":"2023-11-02T17:48:05.097030+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T17:49:10.769417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.097052+0000","last_change":"2023-11-02T17:48:05.092367+0000","last_active":"2023-11-02T17:48:05.097052+0000","last_peered":"2023-11-02T17:48:05.097052+0000","last_clean":"2023-11-02T17:48:05.097052+0000","last_became_active":"2023-11-02T17:48:05.092061+0000","last_became_peered":"2023-11-02T17:48:05.092061+0000","last_unstale":"2023-11-02T17:48:05.097052+0000","last_undegraded":"2023-11-02T17:48:05.097052+0000","last_fullsized":"2023-11-02T17:48:05.097052+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T20:20:27.670178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.097060+0000","last_change":"2023-11-02T17:48:05.092610+0000","last_active":"2023-11-02T17:48:05.097060+0000","last_peered":"2023-11-02T17:48:05.097060+0000","last_clean":"2023-11-02T17:48:05.097060+0000","last_became_active":"2023-11-02T17:48:05.092348+0000","last_became_peered":"2023-11-02T17:48:05.092348+0000","last_unstale":"2023-11-02T17:48:05.097060+0000","last_undegraded":"2023-11-02T17:48:05.097060+0000","last_fullsized":"2023-11-02T17:48:05.097060+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T04:29:56.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-11-02T17:48:06.089700+0000","last_change":"2023-11-02T17:48:05.088324+0000","last_active":"2023-11-02T17:48:06.089700+0000","last_peered":"2023-11-02T17:48:06.089700+0000","last_clean":"2023-11-02T17:48:06.089700+0000","last_became_active":"2023-11-02T17:48:05.087881+0000","last_became_peered":"2023-11-02T17:48:05.087881+0000","last_unstale":"2023-11-02T17:48:06.089700+0000","last_undegraded":"2023-11-02T17:48:06.089700+0000","last_fullsized":"2023-11-02T17:48:06.089700+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T04:22:35.873814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.092824+0000","last_change":"2023-11-02T17:48:05.092824+0000","last_active":"2023-11-02T17:48:05.092824+0000","last_peered":"2023-11-02T17:48:05.092824+0000","last_clean":"2023-11-02T17:48:05.092824+0000","last_became_active":"2023-11-02T17:48:05.092521+0000","last_became_peered":"2023-11-02T17:48:05.092521+0000","last_unstale":"2023-11-02T17:48:05.092824+0000","last_undegraded":"2023-11-02T17:48:05.092824+0000","last_fullsized":"2023-11-02T17:48:05.092824+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T01:03:28.677798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.092981+0000","last_change":"2023-11-02T17:48:05.092981+0000","last_active":"2023-11-02T17:48:05.092981+0000","last_peered":"2023-11-02T17:48:05.092981+0000","last_clean":"2023-11-02T17:48:05.092981+0000","last_became_active":"2023-11-02T17:48:05.092486+0000","last_became_peered":"2023-11-02T17:48:05.092486+0000","last_unstale":"2023-11-02T17:48:05.092981+0000","last_undegraded":"2023-11-02T17:48:05.092981+0000","last_fullsized":"2023-11-02T17:48:05.092981+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T02:42:18.203776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.094195+0000","last_change":"2023-11-02T17:48:05.089864+0000","last_active":"2023-11-02T17:48:05.094195+0000","last_peered":"2023-11-02T17:48:05.094195+0000","last_clean":"2023-11-02T17:48:05.094195+0000","last_became_active":"2023-11-02T17:48:05.089553+0000","last_became_peered":"2023-11-02T17:48:05.089553+0000","last_unstale":"2023-11-02T17:48:05.094195+0000","last_undegraded":"2023-11-02T17:48:05.094195+0000","last_fullsized":"2023-11-02T17:48:05.094195+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T21:27:12.507821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":139,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-02T17:48:04.071120+0000","last_change":"2023-11-02T17:47:58.417557+0000","last_active":"2023-11-02T17:48:04.071120+0000","last_peered":"2023-11-02T17:48:04.071120+0000","last_clean":"2023-11-02T17:48:04.071120+0000","last_became_active":"2023-11-02T17:47:58.415500+0000","last_became_peered":"2023-11-02T17:47:58.415500+0000","last_unstale":"2023-11-02T17:48:04.071120+0000","last_undegraded":"2023-11-02T17:48:04.071120+0000","last_fullsized":"2023-11-02T17:48:04.071120+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:57.274424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:57.274424+0000","last_clean_scrub_stamp":"2023-11-02T17:47:57.274424+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T02:12:01.107698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672966,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27660,"kb_used_data":708,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829940,"statfs":{"total":107374182400,"available":107345858560,"internally_reserved":0,"allocated":724992,"data_stored":625864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]}]},{"osd":0,"up_from":10,"seq":42949672965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27664,"kb_used_data":712,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829936,"statfs":{"total":107374182400,"available":107345854464,"internally_reserved":0,"allocated":729088,"data_stored":628623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":131072,"data_stored":35496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-02T17:48:09.529 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-11-02T17:48:09.824 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:48:09.826 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-11-02T17:48:09.840 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2023-11-02T17:48:09.217086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82404,"kb_used_data":1548,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490396,"statfs":{"total":322122547200,"available":322038165504,"internally_reserved":0,"allocated":1585152,"data_stored":1289983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001500"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.099421+0000","last_change":"2023-11-02T17:48:05.094862+0000","last_active":"2023-11-02T17:48:05.099421+0000","last_peered":"2023-11-02T17:48:05.099421+0000","last_clean":"2023-11-02T17:48:05.099421+0000","last_became_active":"2023-11-02T17:48:05.094619+0000","last_became_peered":"2023-11-02T17:48:05.094619+0000","last_unstale":"2023-11-02T17:48:05.099421+0000","last_undegraded":"2023-11-02T17:48:05.099421+0000","last_fullsized":"2023-11-02T17:48:05.099421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T19:31:05.400708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.097030+0000","last_change":"2023-11-02T17:48:05.092199+0000","last_active":"2023-11-02T17:48:05.097030+0000","last_peered":"2023-11-02T17:48:05.097030+0000","last_clean":"2023-11-02T17:48:05.097030+0000","last_became_active":"2023-11-02T17:48:05.091877+0000","last_became_peered":"2023-11-02T17:48:05.091877+0000","last_unstale":"2023-11-02T17:48:05.097030+0000","last_undegraded":"2023-11-02T17:48:05.097030+0000","last_fullsized":"2023-11-02T17:48:05.097030+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T17:49:10.769417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.097052+0000","last_change":"2023-11-02T17:48:05.092367+0000","last_active":"2023-11-02T17:48:05.097052+0000","last_peered":"2023-11-02T17:48:05.097052+0000","last_clean":"2023-11-02T17:48:05.097052+0000","last_became_active":"2023-11-02T17:48:05.092061+0000","last_became_peered":"2023-11-02T17:48:05.092061+0000","last_unstale":"2023-11-02T17:48:05.097052+0000","last_undegraded":"2023-11-02T17:48:05.097052+0000","last_fullsized":"2023-11-02T17:48:05.097052+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T20:20:27.670178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.097060+0000","last_change":"2023-11-02T17:48:05.092610+0000","last_active":"2023-11-02T17:48:05.097060+0000","last_peered":"2023-11-02T17:48:05.097060+0000","last_clean":"2023-11-02T17:48:05.097060+0000","last_became_active":"2023-11-02T17:48:05.092348+0000","last_became_peered":"2023-11-02T17:48:05.092348+0000","last_unstale":"2023-11-02T17:48:05.097060+0000","last_undegraded":"2023-11-02T17:48:05.097060+0000","last_fullsized":"2023-11-02T17:48:05.097060+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T04:29:56.319042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-11-02T17:48:06.089700+0000","last_change":"2023-11-02T17:48:05.088324+0000","last_active":"2023-11-02T17:48:06.089700+0000","last_peered":"2023-11-02T17:48:06.089700+0000","last_clean":"2023-11-02T17:48:06.089700+0000","last_became_active":"2023-11-02T17:48:05.087881+0000","last_became_peered":"2023-11-02T17:48:05.087881+0000","last_unstale":"2023-11-02T17:48:06.089700+0000","last_undegraded":"2023-11-02T17:48:06.089700+0000","last_fullsized":"2023-11-02T17:48:06.089700+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T04:22:35.873814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.092824+0000","last_change":"2023-11-02T17:48:05.092824+0000","last_active":"2023-11-02T17:48:05.092824+0000","last_peered":"2023-11-02T17:48:05.092824+0000","last_clean":"2023-11-02T17:48:05.092824+0000","last_became_active":"2023-11-02T17:48:05.092521+0000","last_became_peered":"2023-11-02T17:48:05.092521+0000","last_unstale":"2023-11-02T17:48:05.092824+0000","last_undegraded":"2023-11-02T17:48:05.092824+0000","last_fullsized":"2023-11-02T17:48:05.092824+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T01:03:28.677798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.092981+0000","last_change":"2023-11-02T17:48:05.092981+0000","last_active":"2023-11-02T17:48:05.092981+0000","last_peered":"2023-11-02T17:48:05.092981+0000","last_clean":"2023-11-02T17:48:05.092981+0000","last_became_active":"2023-11-02T17:48:05.092486+0000","last_became_peered":"2023-11-02T17:48:05.092486+0000","last_unstale":"2023-11-02T17:48:05.092981+0000","last_undegraded":"2023-11-02T17:48:05.092981+0000","last_fullsized":"2023-11-02T17:48:05.092981+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T02:42:18.203776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-02T17:48:05.094195+0000","last_change":"2023-11-02T17:48:05.089864+0000","last_active":"2023-11-02T17:48:05.094195+0000","last_peered":"2023-11-02T17:48:05.094195+0000","last_clean":"2023-11-02T17:48:05.094195+0000","last_became_active":"2023-11-02T17:48:05.089553+0000","last_became_peered":"2023-11-02T17:48:05.089553+0000","last_unstale":"2023-11-02T17:48:05.094195+0000","last_undegraded":"2023-11-02T17:48:05.094195+0000","last_fullsized":"2023-11-02T17:48:05.094195+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:48:04.062255+0000","last_clean_scrub_stamp":"2023-11-02T17:48:04.062255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T21:27:12.507821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":139,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-02T17:48:04.071120+0000","last_change":"2023-11-02T17:47:58.417557+0000","last_active":"2023-11-02T17:48:04.071120+0000","last_peered":"2023-11-02T17:48:04.071120+0000","last_clean":"2023-11-02T17:48:04.071120+0000","last_became_active":"2023-11-02T17:47:58.415500+0000","last_became_peered":"2023-11-02T17:47:58.415500+0000","last_unstale":"2023-11-02T17:48:04.071120+0000","last_undegraded":"2023-11-02T17:48:04.071120+0000","last_fullsized":"2023-11-02T17:48:04.071120+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T17:47:57.274424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T17:47:57.274424+0000","last_clean_scrub_stamp":"2023-11-02T17:47:57.274424+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T02:12:01.107698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672966,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27660,"kb_used_data":708,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829940,"statfs":{"total":107374182400,"available":107345858560,"internally_reserved":0,"allocated":724992,"data_stored":625864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]}]},{"osd":0,"up_from":10,"seq":42949672965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27664,"kb_used_data":712,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829936,"statfs":{"total":107374182400,"available":107345854464,"internally_reserved":0,"allocated":729088,"data_stored":628623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":131072,"data_stored":35496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-02T17:48:09.841 INFO:tasks.ceph.ceph_manager.c2:clean! 2023-11-02T17:48:09.842 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2023-11-02T17:48:09.842 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2023-11-02T17:48:09.842 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2023-11-02T17:48:10.173 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-02T17:48:10.174 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-02T17:48:10.188 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2023-11-02T17:48:10.188 INFO:teuthology.run_tasks:Running task rgw... 2023-11-02T17:48:10.206 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2023-11-02T17:48:10.207 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2023-11-02T17:48:10.207 INFO:tasks.rgw:Creating data pools 2023-11-02T17:48:10.207 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2023-11-02T17:48:10.208 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-11-02T17:48:11.467 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'default.rgw.buckets.data' created 2023-11-02T17:48:11.479 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-11-02T17:48:12.471 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-11-02T17:48:12.494 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-11-02T17:48:13.479 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'default.rgw.buckets.index' created 2023-11-02T17:48:13.490 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-11-02T17:48:14.493 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-11-02T17:48:14.507 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2023-11-02T17:48:14.508 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-11-02T17:48:14.801 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'default.rgw.buckets.data' already exists 2023-11-02T17:48:14.811 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-11-02T17:48:15.484 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-11-02T17:48:15.497 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-11-02T17:48:15.810 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'default.rgw.buckets.index' already exists 2023-11-02T17:48:15.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-11-02T17:48:16.491 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-11-02T17:48:16.505 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2023-11-02T17:48:16.505 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-11-02T17:48:17.149 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'default.rgw.buckets.data' created 2023-11-02T17:48:17.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-11-02T17:48:18.176 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-11-02T17:48:18.186 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-11-02T17:48:19.163 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'default.rgw.buckets.index' created 2023-11-02T17:48:19.175 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-11-02T17:48:20.168 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-11-02T17:48:20.187 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2023-11-02T17:48:20.188 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-11-02T17:48:20.481 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'default.rgw.buckets.data' already exists 2023-11-02T17:48:20.498 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-11-02T17:48:21.171 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-11-02T17:48:21.184 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-11-02T17:48:21.485 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'default.rgw.buckets.index' already exists 2023-11-02T17:48:21.495 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-11-02T17:48:22.179 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-11-02T17:48:22.192 DEBUG:tasks.rgw:Pools created 2023-11-02T17:48:22.192 INFO:tasks.rgw:Configuring compression type = random 2023-11-02T17:48:22.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2023-11-02T17:48:22.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2023-11-02T17:48:22.194 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2023-11-02T17:48:22.235 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:22.235 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:22.256 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:22.250+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.443 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.438+0000 7fd847d49f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:23.443 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.438+0000 7fd847d49f40 20 realm 2023-11-02T17:48:23.443 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.438+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.438+0000 7fd847d49f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:23.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.438+0000 7fd847d49f40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-11-02T17:48:23.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.438+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.438+0000 7fd847d49f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:23.445 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.438+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.455 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.450+0000 7fd847d49f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:23.455 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.450+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.450+0000 7fd847d49f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:23.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.450+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.450+0000 7fd847d49f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:23.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.450+0000 7fd847d49f40 10 Creating default zonegroup 2023-11-02T17:48:23.457 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.450+0000 7fd847d49f40 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-11-02T17:48:23.472 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.466+0000 7fd847d49f40 10 WARNING: driver->list_zones() returned r=-2 2023-11-02T17:48:23.472 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.466+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.473 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.466+0000 7fd847d49f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:23.476 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.470+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.477 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.470+0000 7fd847d49f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:23.477 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.470+0000 7fd847d49f40 10 could not read realm id: (2) No such file or directory 2023-11-02T17:48:23.477 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.470+0000 7fd847d49f40 10 WARNING: failed to set zone as default, r=-22 2023-11-02T17:48:23.477 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.470+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.478 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.470+0000 7fd847d49f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:23.480 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.474+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.481 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.474+0000 7fd847d49f40 20 rados_obj.operate() r=0 bl.length=874 2023-11-02T17:48:23.481 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.474+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.474+0000 7fd847d49f40 20 rados_obj.operate() r=0 bl.length=420 2023-11-02T17:48:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.474+0000 7fd847d49f40 10 Using default name default 2023-11-02T17:48:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.474+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.483 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.478+0000 7fd847d49f40 20 rados_obj.operate() r=0 bl.length=46 2023-11-02T17:48:23.483 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.478+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.484 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.478+0000 7fd847d49f40 20 rados_obj.operate() r=0 bl.length=874 2023-11-02T17:48:23.484 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.478+0000 7fd847d49f40 20 zone default found 2023-11-02T17:48:23.484 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.478+0000 7fd847d49f40 4 Realm: () 2023-11-02T17:48:23.485 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.478+0000 7fd847d49f40 4 ZoneGroup: default (46569e14-35b6-49cd-8c5e-c04a5b24daf2) 2023-11-02T17:48:23.485 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.478+0000 7fd847d49f40 4 Zone: default (a4c476aa-18da-47ad-b650-3b3011ad64a5) 2023-11-02T17:48:23.485 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.478+0000 7fd847d49f40 10 cannot find current period zonegroup using local zonegroup configuration 2023-11-02T17:48:23.485 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.478+0000 7fd847d49f40 20 zonegroup default 2023-11-02T17:48:23.485 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.478+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:23.486 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.478+0000 7fd847d49f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:23.486 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:23.478+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:25.463 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:25.458+0000 7fd847d49f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:25.463 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:25.458+0000 7fd847d49f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:25.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:25.458+0000 7fd847d49f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:25.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:25.458+0000 7fd847d49f40 20 started sync module instance, tier type = 2023-11-02T17:48:25.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:25.458+0000 7fd847d49f40 20 started zone id=a4c476aa-18da-47ad-b650-3b3011ad64a5 (name=default) with tier type = 2023-11-02T17:48:27.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 add_watcher() i=0 2023-11-02T17:48:27.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 add_watcher() i=1 2023-11-02T17:48:27.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 add_watcher() i=2 2023-11-02T17:48:27.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 add_watcher() i=3 2023-11-02T17:48:27.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 add_watcher() i=4 2023-11-02T17:48:27.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 add_watcher() i=5 2023-11-02T17:48:27.505 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 add_watcher() i=6 2023-11-02T17:48:27.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 add_watcher() i=7 2023-11-02T17:48:27.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 2 all 8 watchers are set, enabling cache 2023-11-02T17:48:27.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-11-02T17:48:27.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-11-02T17:48:27.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-11-02T17:48:27.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-11-02T17:48:27.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-11-02T17:48:27.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-11-02T17:48:27.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-11-02T17:48:27.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-11-02T17:48:27.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-11-02T17:48:27.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-11-02T17:48:27.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-11-02T17:48:27.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.498+0000 7fd847d49f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-11-02T17:48:27.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.502+0000 7fd847d49f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-11-02T17:48:27.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.502+0000 7fd847d49f40 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-11-02T17:48:27.510 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.502+0000 7fd847d49f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-11-02T17:48:27.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.506+0000 7fd82d4d0700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-11-02T17:48:27.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.506+0000 7fd847d49f40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-11-02T17:48:27.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:27.506+0000 7fd847d49f40 20 check_secure_mon_conn(): mode 1 is insecure 2023-11-02T17:48:30.498 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.494+0000 7fd847d49f40 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-11-02T17:48:30.498 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.494+0000 7fd847d49f40 5 note: GC not initialized 2023-11-02T17:48:30.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.502+0000 7fd828cc7700 20 reqs_thread_entry: start 2023-11-02T17:48:30.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.502+0000 7fd847d49f40 20 init_complete bucket index max shards: 11 2023-11-02T17:48:30.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.502+0000 7fd847d49f40 10 rgw notify: Started notification manager with: 1 workers 2023-11-02T17:48:30.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.502+0000 7fd826cc3700 20 rgw notify: INFO: next queues processing will happen at: Thu Nov 2 17:49:00 2023 2023-11-02T17:48:30.509 INFO:teuthology.orchestra.run.smithi017.stderr: 2023-11-02T17:48:30.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.502+0000 7fd8264a1700 20 reqs_thread_entry: start 2023-11-02T17:48:30.537 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.530+0000 7fd82d4d0700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-11-02T17:48:30.541 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.534+0000 7fd847d49f40 20 remove_watcher() i=0 2023-11-02T17:48:30.541 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.534+0000 7fd847d49f40 2 removed watcher, disabling cache 2023-11-02T17:48:30.542 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.538+0000 7fd847d49f40 20 remove_watcher() i=1 2023-11-02T17:48:30.544 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.538+0000 7fd847d49f40 20 remove_watcher() i=2 2023-11-02T17:48:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.542+0000 7fd847d49f40 20 remove_watcher() i=3 2023-11-02T17:48:30.548 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.542+0000 7fd847d49f40 20 remove_watcher() i=4 2023-11-02T17:48:30.550 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.546+0000 7fd847d49f40 20 remove_watcher() i=5 2023-11-02T17:48:30.552 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.546+0000 7fd847d49f40 20 remove_watcher() i=6 2023-11-02T17:48:30.554 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.550+0000 7fd847d49f40 20 remove_watcher() i=7 2023-11-02T17:48:30.558 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T17:48:30.562 DEBUG:tasks.util.rgw: json result: [] 2023-11-02T17:48:30.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-11-02T17:48:30.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-11-02T17:48:30.563 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-11-02T17:48:30.609 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:30.609 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:30.627 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.622+0000 7f662b5daf40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-11-02T17:48:30.627 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.622+0000 7f662b5daf40 20 check_secure_mon_conn(): mode 1 is insecure 2023-11-02T17:48:30.633 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.626+0000 7f660d55a700 20 reqs_thread_entry: start 2023-11-02T17:48:30.644 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:48:30.645 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a4c476aa-18da-47ad-b650-3b3011ad64a5", 2023-11-02T17:48:30.645 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default", 2023-11-02T17:48:30.645 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "default.rgw.meta:root", 2023-11-02T17:48:30.645 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "default.rgw.control", 2023-11-02T17:48:30.645 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "default.rgw.log:gc", 2023-11-02T17:48:30.646 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "default.rgw.log:lc", 2023-11-02T17:48:30.646 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "default.rgw.log", 2023-11-02T17:48:30.646 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-11-02T17:48:30.646 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-11-02T17:48:30.646 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "default.rgw.meta:roles", 2023-11-02T17:48:30.647 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-11-02T17:48:30.647 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-11-02T17:48:30.647 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-11-02T17:48:30.647 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-11-02T17:48:30.647 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-11-02T17:48:30.648 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "default.rgw.otp", 2023-11-02T17:48:30.648 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2023-11-02T17:48:30.648 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "", 2023-11-02T17:48:30.648 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "" 2023-11-02T17:48:30.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:30.648 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2023-11-02T17:48:30.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:48:30.649 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2023-11-02T17:48:30.649 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:48:30.649 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "default.rgw.buckets.index", 2023-11-02T17:48:30.650 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2023-11-02T17:48:30.650 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2023-11-02T17:48:30.650 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "default.rgw.buckets.data", 2023-11-02T17:48:30.650 INFO:teuthology.orchestra.run.smithi017.stdout: "compression_type": "random" 2023-11-02T17:48:30.650 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:30.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:30.651 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-11-02T17:48:30.651 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0, 2023-11-02T17:48:30.651 INFO:teuthology.orchestra.run.smithi017.stdout: "inline_data": true 2023-11-02T17:48:30.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:30.652 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:30.652 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:30.652 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "", 2023-11-02T17:48:30.652 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "default.rgw.log:notif" 2023-11-02T17:48:30.652 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:48:30.653 DEBUG:tasks.util.rgw: json result: {'id': 'a4c476aa-18da-47ad-b650-3b3011ad64a5', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-11-02T17:48:30.654 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2023-11-02T17:48:30.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2023-11-02T17:48:30.654 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2023-11-02T17:48:30.693 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:30.694 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:30.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:30.717 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:30.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 20 realm 2023-11-02T17:48:30.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:30.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:30.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-11-02T17:48:30.719 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:30.719 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:30.719 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:30.719 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 20 rados_obj.operate() r=0 bl.length=46 2023-11-02T17:48:30.720 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:30.720 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 20 rados_obj.operate() r=0 bl.length=890 2023-11-02T17:48:30.720 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.710+0000 7f8f7a988f40 20 searching for the correct realm 2023-11-02T17:48:30.740 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 RGWRados::pool_iterate: got zonegroup_info.46569e14-35b6-49cd-8c5e-c04a5b24daf2 2023-11-02T17:48:30.740 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 RGWRados::pool_iterate: got zone_info.a4c476aa-18da-47ad-b650-3b3011ad64a5 2023-11-02T17:48:30.740 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 RGWRados::pool_iterate: got zone_names.default 2023-11-02T17:48:30.740 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-11-02T17:48:30.741 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:30.741 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:30.742 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:30.742 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 rados_obj.operate() r=0 bl.length=46 2023-11-02T17:48:30.742 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:30.742 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 rados_obj.operate() r=0 bl.length=420 2023-11-02T17:48:30.743 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 zone default found 2023-11-02T17:48:30.743 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 4 Realm: () 2023-11-02T17:48:30.743 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 4 ZoneGroup: default (46569e14-35b6-49cd-8c5e-c04a5b24daf2) 2023-11-02T17:48:30.743 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 4 Zone: default (a4c476aa-18da-47ad-b650-3b3011ad64a5) 2023-11-02T17:48:30.743 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 10 cannot find current period zonegroup using local zonegroup configuration 2023-11-02T17:48:30.744 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 zonegroup default 2023-11-02T17:48:30.744 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.734+0000 7f8f7a988f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:30.745 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.738+0000 7f8f7a988f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:30.745 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.738+0000 7f8f7a988f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:30.745 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.738+0000 7f8f7a988f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:30.745 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.738+0000 7f8f7a988f40 20 rados->read ofs=0 len=0 2023-11-02T17:48:30.746 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.738+0000 7f8f7a988f40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:30.746 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.738+0000 7f8f7a988f40 20 started sync module instance, tier type = 2023-11-02T17:48:30.746 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.738+0000 7f8f7a988f40 20 started zone id=a4c476aa-18da-47ad-b650-3b3011ad64a5 (name=default) with tier type = 2023-11-02T17:48:30.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.754+0000 7f8f7a988f40 20 add_watcher() i=0 2023-11-02T17:48:30.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.754+0000 7f8f7a988f40 20 add_watcher() i=1 2023-11-02T17:48:30.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.754+0000 7f8f7a988f40 20 add_watcher() i=2 2023-11-02T17:48:30.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.754+0000 7f8f7a988f40 20 add_watcher() i=3 2023-11-02T17:48:30.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.754+0000 7f8f7a988f40 20 add_watcher() i=4 2023-11-02T17:48:30.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.754+0000 7f8f7a988f40 20 add_watcher() i=5 2023-11-02T17:48:30.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.754+0000 7f8f7a988f40 20 add_watcher() i=6 2023-11-02T17:48:30.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.758+0000 7f8f7a988f40 20 add_watcher() i=7 2023-11-02T17:48:30.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.758+0000 7f8f7a988f40 2 all 8 watchers are set, enabling cache 2023-11-02T17:48:30.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.758+0000 7f8f6010f700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-11-02T17:48:30.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.758+0000 7f8f7a988f40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-11-02T17:48:30.766 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.758+0000 7f8f7a988f40 20 check_secure_mon_conn(): mode 1 is insecure 2023-11-02T17:48:30.772 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.766+0000 7f8f7a988f40 5 note: GC not initialized 2023-11-02T17:48:30.774 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.766+0000 7f8f5b906700 20 reqs_thread_entry: start 2023-11-02T17:48:30.774 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.770+0000 7f8f7a988f40 20 init_complete bucket index max shards: 11 2023-11-02T17:48:30.774 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.770+0000 7f8f7a988f40 10 rgw notify: Started notification manager with: 1 workers 2023-11-02T17:48:30.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.770+0000 7f8f59101700 20 reqs_thread_entry: start 2023-11-02T17:48:30.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.770+0000 7f8f59902700 20 rgw notify: INFO: next queues processing will happen at: Thu Nov 2 17:49:01 2023 2023-11-02T17:48:30.775 INFO:teuthology.orchestra.run.smithi017.stderr: 2023-11-02T17:48:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.790+0000 7f8f6010f700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-11-02T17:48:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.798+0000 7f8f7a988f40 20 remove_watcher() i=0 2023-11-02T17:48:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.798+0000 7f8f7a988f40 2 removed watcher, disabling cache 2023-11-02T17:48:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.798+0000 7f8f7a988f40 20 remove_watcher() i=1 2023-11-02T17:48:30.806 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.802+0000 7f8f7a988f40 20 remove_watcher() i=2 2023-11-02T17:48:30.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.802+0000 7f8f7a988f40 20 remove_watcher() i=3 2023-11-02T17:48:30.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.806+0000 7f8f7a988f40 20 remove_watcher() i=4 2023-11-02T17:48:30.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.806+0000 7f8f7a988f40 20 remove_watcher() i=5 2023-11-02T17:48:30.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.810+0000 7f8f7a988f40 20 remove_watcher() i=6 2023-11-02T17:48:30.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.810+0000 7f8f7a988f40 20 remove_watcher() i=7 2023-11-02T17:48:30.822 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T17:48:30.826 DEBUG:tasks.util.rgw: json result: [] 2023-11-02T17:48:30.826 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-11-02T17:48:30.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-11-02T17:48:30.827 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-11-02T17:48:30.915 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:30.915 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:30.934 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.926+0000 7ff23dabcf40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-11-02T17:48:30.934 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.926+0000 7ff23dabcf40 20 check_secure_mon_conn(): mode 1 is insecure 2023-11-02T17:48:30.940 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:30.934+0000 7ff21fa3c700 20 reqs_thread_entry: start 2023-11-02T17:48:30.952 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:48:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a4c476aa-18da-47ad-b650-3b3011ad64a5", 2023-11-02T17:48:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default", 2023-11-02T17:48:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "default.rgw.meta:root", 2023-11-02T17:48:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "default.rgw.control", 2023-11-02T17:48:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "default.rgw.log:gc", 2023-11-02T17:48:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "default.rgw.log:lc", 2023-11-02T17:48:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "default.rgw.log", 2023-11-02T17:48:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-11-02T17:48:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-11-02T17:48:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "default.rgw.meta:roles", 2023-11-02T17:48:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-11-02T17:48:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-11-02T17:48:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-11-02T17:48:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-11-02T17:48:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-11-02T17:48:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "default.rgw.otp", 2023-11-02T17:48:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2023-11-02T17:48:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "", 2023-11-02T17:48:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "" 2023-11-02T17:48:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2023-11-02T17:48:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:48:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2023-11-02T17:48:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:48:30.957 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "default.rgw.buckets.index", 2023-11-02T17:48:30.957 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2023-11-02T17:48:30.957 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2023-11-02T17:48:30.957 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "default.rgw.buckets.data", 2023-11-02T17:48:30.957 INFO:teuthology.orchestra.run.smithi017.stdout: "compression_type": "random" 2023-11-02T17:48:30.958 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:30.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:30.958 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-11-02T17:48:30.958 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0, 2023-11-02T17:48:30.958 INFO:teuthology.orchestra.run.smithi017.stdout: "inline_data": true 2023-11-02T17:48:30.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:30.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:30.959 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:30.959 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "", 2023-11-02T17:48:30.959 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "default.rgw.log:notif" 2023-11-02T17:48:30.959 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:48:30.960 DEBUG:tasks.util.rgw: json result: {'id': 'a4c476aa-18da-47ad-b650-3b3011ad64a5', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-11-02T17:48:30.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2023-11-02T17:48:30.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2023-11-02T17:48:30.962 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2023-11-02T17:48:31.002 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:31.003 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:31.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:31.016+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.116+0000 7f3724deaf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.116+0000 7f3724deaf40 20 realm 2023-11-02T17:48:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.116+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.116+0000 7f3724deaf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.116+0000 7f3724deaf40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-11-02T17:48:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.116+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.116+0000 7f3724deaf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.120+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.123 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.120+0000 7f3724deaf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:32.123 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.120+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.120+0000 7f3724deaf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:32.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.120+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.120+0000 7f3724deaf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:32.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.120+0000 7f3724deaf40 10 Creating default zonegroup 2023-11-02T17:48:32.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.120+0000 7f3724deaf40 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-11-02T17:48:32.138 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.136+0000 7f3724deaf40 10 WARNING: driver->list_zones() returned r=-2 2023-11-02T17:48:32.138 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.136+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.139 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.136+0000 7f3724deaf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:32.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.136+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.140+0000 7f3724deaf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:32.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.140+0000 7f3724deaf40 10 could not read realm id: (2) No such file or directory 2023-11-02T17:48:32.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.140+0000 7f3724deaf40 10 WARNING: failed to set zone as default, r=-22 2023-11-02T17:48:32.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.140+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.140+0000 7f3724deaf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:32.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.144+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.144+0000 7f3724deaf40 20 rados_obj.operate() r=0 bl.length=874 2023-11-02T17:48:32.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.144+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.144+0000 7f3724deaf40 20 rados_obj.operate() r=0 bl.length=420 2023-11-02T17:48:32.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.144+0000 7f3724deaf40 10 Using default name default 2023-11-02T17:48:32.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.144+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.150 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.144+0000 7f3724deaf40 20 rados_obj.operate() r=0 bl.length=46 2023-11-02T17:48:32.150 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.144+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.148+0000 7f3724deaf40 20 rados_obj.operate() r=0 bl.length=874 2023-11-02T17:48:32.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.148+0000 7f3724deaf40 20 zone default found 2023-11-02T17:48:32.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.148+0000 7f3724deaf40 4 Realm: () 2023-11-02T17:48:32.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.148+0000 7f3724deaf40 4 ZoneGroup: default (6a403571-3777-4934-a050-9fac42e717bf) 2023-11-02T17:48:32.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.148+0000 7f3724deaf40 4 Zone: default (d7b508be-b020-47b1-a6be-cde3a5fe7809) 2023-11-02T17:48:32.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.148+0000 7f3724deaf40 10 cannot find current period zonegroup using local zonegroup configuration 2023-11-02T17:48:32.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.148+0000 7f3724deaf40 20 zonegroup default 2023-11-02T17:48:32.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.148+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:32.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.148+0000 7f3724deaf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:32.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:32.148+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:34.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:34.120+0000 7f3724deaf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:34.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:34.120+0000 7f3724deaf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:34.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:34.120+0000 7f3724deaf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:34.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:34.120+0000 7f3724deaf40 20 started sync module instance, tier type = 2023-11-02T17:48:34.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:34.120+0000 7f3724deaf40 20 started zone id=d7b508be-b020-47b1-a6be-cde3a5fe7809 (name=default) with tier type = 2023-11-02T17:48:36.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.156+0000 7f3724deaf40 20 add_watcher() i=0 2023-11-02T17:48:36.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.156+0000 7f3724deaf40 20 add_watcher() i=1 2023-11-02T17:48:36.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.156+0000 7f3724deaf40 20 add_watcher() i=2 2023-11-02T17:48:36.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.156+0000 7f3724deaf40 20 add_watcher() i=3 2023-11-02T17:48:36.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.156+0000 7f3724deaf40 20 add_watcher() i=4 2023-11-02T17:48:36.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.156+0000 7f3724deaf40 20 add_watcher() i=5 2023-11-02T17:48:36.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.156+0000 7f3724deaf40 20 add_watcher() i=6 2023-11-02T17:48:36.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 add_watcher() i=7 2023-11-02T17:48:36.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 2 all 8 watchers are set, enabling cache 2023-11-02T17:48:36.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-11-02T17:48:36.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-11-02T17:48:36.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-11-02T17:48:36.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-11-02T17:48:36.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-11-02T17:48:36.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-11-02T17:48:36.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-11-02T17:48:36.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-11-02T17:48:36.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-11-02T17:48:36.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-11-02T17:48:36.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-11-02T17:48:36.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-11-02T17:48:36.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-11-02T17:48:36.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-11-02T17:48:36.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.160+0000 7f3724deaf40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-11-02T17:48:36.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.164+0000 7f370a571700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-11-02T17:48:36.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.164+0000 7f3724deaf40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-11-02T17:48:36.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:36.164+0000 7f3724deaf40 20 check_secure_mon_conn(): mode 1 is insecure 2023-11-02T17:48:39.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.160+0000 7f3724deaf40 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-11-02T17:48:39.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.160+0000 7f3724deaf40 5 note: GC not initialized 2023-11-02T17:48:39.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.176+0000 7f3705d68700 20 reqs_thread_entry: start 2023-11-02T17:48:39.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.176+0000 7f3724deaf40 20 init_complete bucket index max shards: 11 2023-11-02T17:48:39.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.176+0000 7f3724deaf40 10 rgw notify: Started notification manager with: 1 workers 2023-11-02T17:48:39.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.176+0000 7f3703d64700 20 rgw notify: INFO: next queues processing will happen at: Thu Nov 2 17:49:09 2023 2023-11-02T17:48:39.181 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-11-02T17:48:39.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.176+0000 7f3703542700 20 reqs_thread_entry: start 2023-11-02T17:48:39.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.204+0000 7f370a571700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-11-02T17:48:39.212 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.208+0000 7f3724deaf40 20 remove_watcher() i=0 2023-11-02T17:48:39.213 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.208+0000 7f3724deaf40 2 removed watcher, disabling cache 2023-11-02T17:48:39.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.212+0000 7f3724deaf40 20 remove_watcher() i=1 2023-11-02T17:48:39.216 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.212+0000 7f3724deaf40 20 remove_watcher() i=2 2023-11-02T17:48:39.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.216+0000 7f3724deaf40 20 remove_watcher() i=3 2023-11-02T17:48:39.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.216+0000 7f3724deaf40 20 remove_watcher() i=4 2023-11-02T17:48:39.222 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.220+0000 7f3724deaf40 20 remove_watcher() i=5 2023-11-02T17:48:39.224 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.220+0000 7f3724deaf40 20 remove_watcher() i=6 2023-11-02T17:48:39.226 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.224+0000 7f3724deaf40 20 remove_watcher() i=7 2023-11-02T17:48:39.231 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2023-11-02T17:48:39.233 DEBUG:tasks.util.rgw: json result: [] 2023-11-02T17:48:39.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-11-02T17:48:39.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-11-02T17:48:39.235 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-11-02T17:48:39.321 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:39.321 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:39.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.332+0000 7fc3b2c03f40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-11-02T17:48:39.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.332+0000 7fc3b2c03f40 20 check_secure_mon_conn(): mode 1 is insecure 2023-11-02T17:48:39.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.340+0000 7fc394b83700 20 reqs_thread_entry: start 2023-11-02T17:48:39.356 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:48:39.356 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "d7b508be-b020-47b1-a6be-cde3a5fe7809", 2023-11-02T17:48:39.356 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default", 2023-11-02T17:48:39.356 INFO:teuthology.orchestra.run.smithi069.stdout: "domain_root": "default.rgw.meta:root", 2023-11-02T17:48:39.357 INFO:teuthology.orchestra.run.smithi069.stdout: "control_pool": "default.rgw.control", 2023-11-02T17:48:39.357 INFO:teuthology.orchestra.run.smithi069.stdout: "gc_pool": "default.rgw.log:gc", 2023-11-02T17:48:39.357 INFO:teuthology.orchestra.run.smithi069.stdout: "lc_pool": "default.rgw.log:lc", 2023-11-02T17:48:39.357 INFO:teuthology.orchestra.run.smithi069.stdout: "log_pool": "default.rgw.log", 2023-11-02T17:48:39.357 INFO:teuthology.orchestra.run.smithi069.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-11-02T17:48:39.358 INFO:teuthology.orchestra.run.smithi069.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-11-02T17:48:39.358 INFO:teuthology.orchestra.run.smithi069.stdout: "roles_pool": "default.rgw.meta:roles", 2023-11-02T17:48:39.358 INFO:teuthology.orchestra.run.smithi069.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-11-02T17:48:39.358 INFO:teuthology.orchestra.run.smithi069.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-11-02T17:48:39.359 INFO:teuthology.orchestra.run.smithi069.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-11-02T17:48:39.359 INFO:teuthology.orchestra.run.smithi069.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-11-02T17:48:39.359 INFO:teuthology.orchestra.run.smithi069.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-11-02T17:48:39.359 INFO:teuthology.orchestra.run.smithi069.stdout: "otp_pool": "default.rgw.otp", 2023-11-02T17:48:39.359 INFO:teuthology.orchestra.run.smithi069.stdout: "system_key": { 2023-11-02T17:48:39.360 INFO:teuthology.orchestra.run.smithi069.stdout: "access_key": "", 2023-11-02T17:48:39.360 INFO:teuthology.orchestra.run.smithi069.stdout: "secret_key": "" 2023-11-02T17:48:39.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:48:39.360 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_pools": [ 2023-11-02T17:48:39.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:48:39.361 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "default-placement", 2023-11-02T17:48:39.361 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:48:39.361 INFO:teuthology.orchestra.run.smithi069.stdout: "index_pool": "default.rgw.buckets.index", 2023-11-02T17:48:39.361 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": { 2023-11-02T17:48:39.361 INFO:teuthology.orchestra.run.smithi069.stdout: "STANDARD": { 2023-11-02T17:48:39.362 INFO:teuthology.orchestra.run.smithi069.stdout: "data_pool": "default.rgw.buckets.data", 2023-11-02T17:48:39.362 INFO:teuthology.orchestra.run.smithi069.stdout: "compression_type": "random" 2023-11-02T17:48:39.362 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:48:39.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:48:39.362 INFO:teuthology.orchestra.run.smithi069.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-11-02T17:48:39.363 INFO:teuthology.orchestra.run.smithi069.stdout: "index_type": 0, 2023-11-02T17:48:39.363 INFO:teuthology.orchestra.run.smithi069.stdout: "inline_data": true 2023-11-02T17:48:39.363 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:48:39.363 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:48:39.363 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:48:39.364 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "", 2023-11-02T17:48:39.364 INFO:teuthology.orchestra.run.smithi069.stdout: "notif_pool": "default.rgw.log:notif" 2023-11-02T17:48:39.364 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:48:39.365 DEBUG:tasks.util.rgw: json result: {'id': 'd7b508be-b020-47b1-a6be-cde3a5fe7809', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-11-02T17:48:39.365 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2023-11-02T17:48:39.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2023-11-02T17:48:39.366 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2023-11-02T17:48:39.406 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:39.406 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.420+0000 7f26980caf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:39.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.424+0000 7f26980caf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.424+0000 7f26980caf40 20 realm 2023-11-02T17:48:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.424+0000 7f26980caf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:39.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.424+0000 7f26980caf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:39.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.424+0000 7f26980caf40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-11-02T17:48:39.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.424+0000 7f26980caf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:39.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.424+0000 7f26980caf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:39.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.424+0000 7f26980caf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:39.430 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.424+0000 7f26980caf40 20 rados_obj.operate() r=0 bl.length=46 2023-11-02T17:48:39.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.424+0000 7f26980caf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:39.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.428+0000 7f26980caf40 20 rados_obj.operate() r=0 bl.length=890 2023-11-02T17:48:39.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.428+0000 7f26980caf40 20 searching for the correct realm 2023-11-02T17:48:39.454 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.448+0000 7f26980caf40 20 RGWRados::pool_iterate: got zonegroup_info.6a403571-3777-4934-a050-9fac42e717bf 2023-11-02T17:48:39.454 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.448+0000 7f26980caf40 20 RGWRados::pool_iterate: got zone_info.d7b508be-b020-47b1-a6be-cde3a5fe7809 2023-11-02T17:48:39.454 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.448+0000 7f26980caf40 20 RGWRados::pool_iterate: got zone_names.default 2023-11-02T17:48:39.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.448+0000 7f26980caf40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-11-02T17:48:39.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.448+0000 7f26980caf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:39.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:39.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:39.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 rados_obj.operate() r=0 bl.length=46 2023-11-02T17:48:39.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:39.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 rados_obj.operate() r=0 bl.length=420 2023-11-02T17:48:39.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 zone default found 2023-11-02T17:48:39.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 4 Realm: () 2023-11-02T17:48:39.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 4 ZoneGroup: default (6a403571-3777-4934-a050-9fac42e717bf) 2023-11-02T17:48:39.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 4 Zone: default (d7b508be-b020-47b1-a6be-cde3a5fe7809) 2023-11-02T17:48:39.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 10 cannot find current period zonegroup using local zonegroup configuration 2023-11-02T17:48:39.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 zonegroup default 2023-11-02T17:48:39.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:39.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:39.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:39.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:39.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 rados->read ofs=0 len=0 2023-11-02T17:48:39.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 rados_obj.operate() r=-2 bl.length=0 2023-11-02T17:48:39.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 started sync module instance, tier type = 2023-11-02T17:48:39.461 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.452+0000 7f26980caf40 20 started zone id=d7b508be-b020-47b1-a6be-cde3a5fe7809 (name=default) with tier type = 2023-11-02T17:48:39.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.480+0000 7f26980caf40 20 add_watcher() i=0 2023-11-02T17:48:39.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.480+0000 7f26980caf40 20 add_watcher() i=1 2023-11-02T17:48:39.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.480+0000 7f26980caf40 20 add_watcher() i=2 2023-11-02T17:48:39.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.480+0000 7f26980caf40 20 add_watcher() i=3 2023-11-02T17:48:39.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.480+0000 7f26980caf40 20 add_watcher() i=4 2023-11-02T17:48:39.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.480+0000 7f26980caf40 20 add_watcher() i=5 2023-11-02T17:48:39.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.480+0000 7f26980caf40 20 add_watcher() i=6 2023-11-02T17:48:39.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.484+0000 7f26980caf40 20 add_watcher() i=7 2023-11-02T17:48:39.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.484+0000 7f26980caf40 2 all 8 watchers are set, enabling cache 2023-11-02T17:48:39.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.488+0000 7f267d851700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-11-02T17:48:39.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.488+0000 7f26980caf40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-11-02T17:48:39.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.488+0000 7f26980caf40 20 check_secure_mon_conn(): mode 1 is insecure 2023-11-02T17:48:39.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.496+0000 7f26980caf40 5 note: GC not initialized 2023-11-02T17:48:39.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.496+0000 7f2679048700 20 reqs_thread_entry: start 2023-11-02T17:48:39.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.496+0000 7f26980caf40 20 init_complete bucket index max shards: 11 2023-11-02T17:48:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.496+0000 7f26980caf40 10 rgw notify: Started notification manager with: 1 workers 2023-11-02T17:48:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.496+0000 7f2677044700 20 rgw notify: INFO: next queues processing will happen at: Thu Nov 2 17:49:09 2023 2023-11-02T17:48:39.502 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-11-02T17:48:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.496+0000 7f2676822700 20 reqs_thread_entry: start 2023-11-02T17:48:39.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.520+0000 7f267d851700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-11-02T17:48:39.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.524+0000 7f26980caf40 20 remove_watcher() i=0 2023-11-02T17:48:39.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.524+0000 7f26980caf40 2 removed watcher, disabling cache 2023-11-02T17:48:39.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.528+0000 7f26980caf40 20 remove_watcher() i=1 2023-11-02T17:48:39.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.532+0000 7f26980caf40 20 remove_watcher() i=2 2023-11-02T17:48:39.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.532+0000 7f26980caf40 20 remove_watcher() i=3 2023-11-02T17:48:39.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.536+0000 7f26980caf40 20 remove_watcher() i=4 2023-11-02T17:48:39.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.536+0000 7f26980caf40 20 remove_watcher() i=5 2023-11-02T17:48:39.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.540+0000 7f26980caf40 20 remove_watcher() i=6 2023-11-02T17:48:39.545 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.540+0000 7f26980caf40 20 remove_watcher() i=7 2023-11-02T17:48:39.549 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2023-11-02T17:48:39.553 DEBUG:tasks.util.rgw: json result: [] 2023-11-02T17:48:39.553 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-11-02T17:48:39.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-11-02T17:48:39.554 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-11-02T17:48:39.600 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:39.601 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:39.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.616+0000 7f2b3e8cdf40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-11-02T17:48:39.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.616+0000 7f2b3e8cdf40 20 check_secure_mon_conn(): mode 1 is insecure 2023-11-02T17:48:39.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:48:39.624+0000 7f2b2084d700 20 reqs_thread_entry: start 2023-11-02T17:48:39.639 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:48:39.639 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "d7b508be-b020-47b1-a6be-cde3a5fe7809", 2023-11-02T17:48:39.639 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default", 2023-11-02T17:48:39.640 INFO:teuthology.orchestra.run.smithi069.stdout: "domain_root": "default.rgw.meta:root", 2023-11-02T17:48:39.640 INFO:teuthology.orchestra.run.smithi069.stdout: "control_pool": "default.rgw.control", 2023-11-02T17:48:39.640 INFO:teuthology.orchestra.run.smithi069.stdout: "gc_pool": "default.rgw.log:gc", 2023-11-02T17:48:39.640 INFO:teuthology.orchestra.run.smithi069.stdout: "lc_pool": "default.rgw.log:lc", 2023-11-02T17:48:39.640 INFO:teuthology.orchestra.run.smithi069.stdout: "log_pool": "default.rgw.log", 2023-11-02T17:48:39.641 INFO:teuthology.orchestra.run.smithi069.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-11-02T17:48:39.641 INFO:teuthology.orchestra.run.smithi069.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-11-02T17:48:39.641 INFO:teuthology.orchestra.run.smithi069.stdout: "roles_pool": "default.rgw.meta:roles", 2023-11-02T17:48:39.641 INFO:teuthology.orchestra.run.smithi069.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-11-02T17:48:39.642 INFO:teuthology.orchestra.run.smithi069.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-11-02T17:48:39.642 INFO:teuthology.orchestra.run.smithi069.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-11-02T17:48:39.642 INFO:teuthology.orchestra.run.smithi069.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-11-02T17:48:39.642 INFO:teuthology.orchestra.run.smithi069.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-11-02T17:48:39.643 INFO:teuthology.orchestra.run.smithi069.stdout: "otp_pool": "default.rgw.otp", 2023-11-02T17:48:39.643 INFO:teuthology.orchestra.run.smithi069.stdout: "system_key": { 2023-11-02T17:48:39.643 INFO:teuthology.orchestra.run.smithi069.stdout: "access_key": "", 2023-11-02T17:48:39.643 INFO:teuthology.orchestra.run.smithi069.stdout: "secret_key": "" 2023-11-02T17:48:39.643 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:48:39.644 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_pools": [ 2023-11-02T17:48:39.644 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:48:39.644 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "default-placement", 2023-11-02T17:48:39.644 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:48:39.644 INFO:teuthology.orchestra.run.smithi069.stdout: "index_pool": "default.rgw.buckets.index", 2023-11-02T17:48:39.645 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": { 2023-11-02T17:48:39.645 INFO:teuthology.orchestra.run.smithi069.stdout: "STANDARD": { 2023-11-02T17:48:39.645 INFO:teuthology.orchestra.run.smithi069.stdout: "data_pool": "default.rgw.buckets.data", 2023-11-02T17:48:39.645 INFO:teuthology.orchestra.run.smithi069.stdout: "compression_type": "random" 2023-11-02T17:48:39.645 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:48:39.646 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:48:39.646 INFO:teuthology.orchestra.run.smithi069.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-11-02T17:48:39.646 INFO:teuthology.orchestra.run.smithi069.stdout: "index_type": 0, 2023-11-02T17:48:39.646 INFO:teuthology.orchestra.run.smithi069.stdout: "inline_data": true 2023-11-02T17:48:39.647 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:48:39.647 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:48:39.647 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:48:39.647 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "", 2023-11-02T17:48:39.648 INFO:teuthology.orchestra.run.smithi069.stdout: "notif_pool": "default.rgw.log:notif" 2023-11-02T17:48:39.648 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:48:39.649 DEBUG:tasks.util.rgw: json result: {'id': 'd7b508be-b020-47b1-a6be-cde3a5fe7809', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-11-02T17:48:39.649 INFO:tasks.rgw:Starting rgw... 2023-11-02T17:48:39.650 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2023-11-02T17:48:39.650 INFO:tasks.rgw:Using beast as radosgw frontend 2023-11-02T17:48:39.650 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-11-02T17:48:39.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-11-02T17:48:39.653 INFO:tasks.rgw.c1.client.0:Started 2023-11-02T17:48:39.654 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2023-11-02T17:48:39.654 INFO:tasks.rgw:Using beast as radosgw frontend 2023-11-02T17:48:39.655 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-11-02T17:48:39.655 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-11-02T17:48:39.657 INFO:tasks.rgw.c1.client.1:Started 2023-11-02T17:48:39.658 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2023-11-02T17:48:39.658 INFO:tasks.rgw:Using beast as radosgw frontend 2023-11-02T17:48:39.658 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-11-02T17:48:39.659 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-11-02T17:48:39.661 INFO:tasks.rgw.c2.client.0:Started 2023-11-02T17:48:39.661 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2023-11-02T17:48:39.662 INFO:tasks.rgw:Using beast as radosgw frontend 2023-11-02T17:48:39.662 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-11-02T17:48:39.662 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-11-02T17:48:39.665 INFO:tasks.rgw.c2.client.1:Started 2023-11-02T17:48:39.665 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi017.front.sepia.ceph.com:8000/ 2023-11-02T17:48:39.665 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8000/ 2023-11-02T17:48:39.711 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:48:39.711 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:48:39.713 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-11-02T17:48:39.714 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-11-02T17:48:39.714 INFO:teuthology.orchestra.run.smithi017.stderr:curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8000: Connection refused 2023-11-02T17:48:40.714 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8000/ 2023-11-02T17:48:40.731 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:48:40.731 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:48:40.735 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-11-02T17:48:40.736 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2023-11-02T17:48:40.736 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi017.front.sepia.ceph.com:8001/ 2023-11-02T17:48:40.736 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001/ 2023-11-02T17:48:40.755 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:48:40.755 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:48:40.758 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-11-02T17:48:40.759 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2023-11-02T17:48:40.759 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi069.front.sepia.ceph.com:8000/ 2023-11-02T17:48:40.760 DEBUG:teuthology.orchestra.run.smithi069:> curl http://smithi069.front.sepia.ceph.com:8000/ 2023-11-02T17:48:40.790 INFO:teuthology.orchestra.run.smithi069.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:48:40.791 INFO:teuthology.orchestra.run.smithi069.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:48:40.793 INFO:teuthology.orchestra.run.smithi069.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2023-11-02T17:48:40.794 INFO:teuthology.orchestra.run.smithi069.stdout:anonymous 2023-11-02T17:48:40.794 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi069.front.sepia.ceph.com:8001/ 2023-11-02T17:48:40.794 DEBUG:teuthology.orchestra.run.smithi069:> curl http://smithi069.front.sepia.ceph.com:8001/ 2023-11-02T17:48:40.855 INFO:teuthology.orchestra.run.smithi069.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:48:40.855 INFO:teuthology.orchestra.run.smithi069.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:48:40.858 INFO:teuthology.orchestra.run.smithi069.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 104k 2023-11-02T17:48:40.859 INFO:teuthology.orchestra.run.smithi069.stdout:anonymous 2023-11-02T17:48:40.860 INFO:teuthology.run_tasks:Running task rgw-multisite... 2023-11-02T17:48:40.920 INFO:tasks.rgw_multisite:creating realm.. 2023-11-02T17:48:40.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:40.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:40.921 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T17:48:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:41.019 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:48:41.019 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:48:41.019 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "test-realm", 2023-11-02T17:48:41.019 INFO:teuthology.orchestra.run.smithi017.stdout: "current_period": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T17:48:41.020 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1 2023-11-02T17:48:41.020 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:48:41.022 INFO:tasks.rgw_multisite:creating master zonegroup.. 2023-11-02T17:48:41.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:41.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:41.023 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T17:48:41.066 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:41.067 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:41.090+0000 7f3b772b7f40 -1 old format 2023-11-02T17:48:41.109 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:48:41.109 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T17:48:41.109 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T17:48:41.109 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T17:48:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T17:48:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T17:48:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T17:48:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T17:48:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T17:48:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "", 2023-11-02T17:48:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [], 2023-11-02T17:48:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T17:48:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:48:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T17:48:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T17:48:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [] 2023-11-02T17:48:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T17:48:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:48:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T17:48:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T17:48:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T17:48:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T17:48:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:48:41.114 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:48:41.115 INFO:tasks.rgw_multisite:creating master zone.. 2023-11-02T17:48:41.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--endpoints', 'http://smithi017.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:41.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--endpoints', 'http://smithi017.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:41.116 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key J46L5YZ9VLF35XGJ --secret u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9 --endpoints http://smithi017.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T17:48:41.170 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:41.198 INFO:teuthology.orchestra.run.smithi017.stderr:NOTICE: set zone's realm_id=a2bfe17b-2975-4441-84f6-480a071e229b 2023-11-02T17:48:41.241 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:48:41.242 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:48:41.242 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T17:48:41.242 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "a1.rgw.meta:root", 2023-11-02T17:48:41.242 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "a1.rgw.control", 2023-11-02T17:48:41.242 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "a1.rgw.log:gc", 2023-11-02T17:48:41.243 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "a1.rgw.log:lc", 2023-11-02T17:48:41.243 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "a1.rgw.log", 2023-11-02T17:48:41.243 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-11-02T17:48:41.243 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-11-02T17:48:41.243 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-11-02T17:48:41.244 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-11-02T17:48:41.244 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-11-02T17:48:41.244 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-11-02T17:48:41.244 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-11-02T17:48:41.244 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-11-02T17:48:41.245 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "a1.rgw.otp", 2023-11-02T17:48:41.245 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2023-11-02T17:48:41.245 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T17:48:41.245 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T17:48:41.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:41.246 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2023-11-02T17:48:41.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:48:41.246 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2023-11-02T17:48:41.246 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:48:41.246 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "a1.rgw.buckets.index", 2023-11-02T17:48:41.247 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2023-11-02T17:48:41.247 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2023-11-02T17:48:41.247 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "a1.rgw.buckets.data" 2023-11-02T17:48:41.247 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:41.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:41.248 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-11-02T17:48:41.248 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0, 2023-11-02T17:48:41.248 INFO:teuthology.orchestra.run.smithi017.stdout: "inline_data": true 2023-11-02T17:48:41.248 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:41.248 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:41.249 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:41.249 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:48:41.249 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "a1.rgw.log:notif" 2023-11-02T17:48:41.249 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:48:41.250 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2023-11-02T17:48:42.450 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'a1.rgw.buckets.data' created 2023-11-02T17:48:42.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2023-11-02T17:48:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2023-11-02T17:48:43.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:43.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:43.487 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T17:48:43.528 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:43.528 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:43.558 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:48:43.559 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:48:43.559 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T17:48:43.559 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "a1.rgw.meta:root", 2023-11-02T17:48:43.559 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "a1.rgw.control", 2023-11-02T17:48:43.560 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "a1.rgw.log:gc", 2023-11-02T17:48:43.560 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "a1.rgw.log:lc", 2023-11-02T17:48:43.560 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "a1.rgw.log", 2023-11-02T17:48:43.560 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-11-02T17:48:43.560 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-11-02T17:48:43.561 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-11-02T17:48:43.561 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-11-02T17:48:43.561 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-11-02T17:48:43.561 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-11-02T17:48:43.561 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-11-02T17:48:43.562 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-11-02T17:48:43.562 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "a1.rgw.otp", 2023-11-02T17:48:43.562 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2023-11-02T17:48:43.562 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T17:48:43.563 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T17:48:43.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:43.563 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2023-11-02T17:48:43.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:48:43.563 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2023-11-02T17:48:43.563 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:48:43.564 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "a1.rgw.buckets.index", 2023-11-02T17:48:43.564 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2023-11-02T17:48:43.564 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2023-11-02T17:48:43.564 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "a1.rgw.buckets.data", 2023-11-02T17:48:43.564 INFO:teuthology.orchestra.run.smithi017.stdout: "compression_type": "random" 2023-11-02T17:48:43.565 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:43.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:43.565 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-11-02T17:48:43.565 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0, 2023-11-02T17:48:43.565 INFO:teuthology.orchestra.run.smithi017.stdout: "inline_data": true 2023-11-02T17:48:43.566 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:43.566 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:43.566 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:43.566 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:48:43.566 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "a1.rgw.log:notif" 2023-11-02T17:48:43.567 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:48:43.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:43.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:43.568 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T17:48:43.612 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:43.612 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:43.637 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:43.630+0000 7f43b9b6df40 0 period (5f87da71-3b30-4008-9009-a0341777f9d7 does not have zone d61b2bea-59a8-4454-8491-ac3afbd04937 configured 2023-11-02T17:48:50.596 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:48:50.597 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:48:50.597 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2023-11-02T17:48:50.598 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T17:48:50.598 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T17:48:50.598 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T17:48:50.599 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:48:50.599 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T17:48:50.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:48:50.600 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T17:48:50.600 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T17:48:50.600 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T17:48:50.600 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T17:48:50.600 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T17:48:50.601 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T17:48:50.601 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:50.601 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T17:48:50.601 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T17:48:50.602 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:48:50.602 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T17:48:50.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:48:50.602 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:48:50.602 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T17:48:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T17:48:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T17:48:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T17:48:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": false, 2023-11-02T17:48:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T17:48:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T17:48:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T17:48:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T17:48:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T17:48:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T17:48:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T17:48:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T17:48:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T17:48:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:48:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T17:48:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:48:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T17:48:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T17:48:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T17:48:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T17:48:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:48:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T17:48:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:48:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T17:48:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T17:48:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T17:48:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T17:48:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:48:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T17:48:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:48:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:48:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T17:48:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:48:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T17:48:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:48:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T17:48:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T17:48:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T17:48:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T17:48:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T17:48:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T17:48:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T17:48:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T17:48:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T17:48:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T17:48:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T17:48:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T17:48:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T17:48:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T17:48:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T17:48:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T17:48:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T17:48:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T17:48:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T17:48:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T17:48:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T17:48:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T17:48:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T17:48:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T17:48:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T17:48:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T17:48:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T17:48:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T17:48:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T17:48:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T17:48:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T17:48:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:48:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T17:48:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:48:50.621 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:48:50.623 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-11-02T17:48:50.623 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T17:48:50.670 INFO:tasks.rgw.c1.client.0.smithi017.stdout:2023-11-02T17:48:50.666+0000 7fe6ad458700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 32290) UID: 0 2023-11-02T17:48:50.671 INFO:tasks.rgw.c1.client.0.smithi017.stdout:2023-11-02T17:48:50.666+0000 7fe6b0cb7f80 -1 shutting down 2023-11-02T17:48:56.726 INFO:tasks.rgw.c1.client.0:Stopped 2023-11-02T17:48:56.726 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-11-02T17:48:56.727 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-11-02T17:48:56.729 INFO:tasks.rgw.c1.client.0:Started 2023-11-02T17:48:56.730 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8000 2023-11-02T17:48:56.748 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:48:56.749 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:48:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-11-02T17:48:56.751 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-11-02T17:48:56.752 INFO:teuthology.orchestra.run.smithi017.stderr:curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8000: Connection refused 2023-11-02T17:48:56.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-11-02T17:48:57.752 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8000 2023-11-02T17:48:57.769 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:48:57.769 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:48:57.785 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 13375 0 --:--:-- --:--:-- --:--:-- 13375 2023-11-02T17:48:57.786 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2023-11-02T17:48:57.786 INFO:tasks.rgw_multisite:creating admin user.. 2023-11-02T17:48:57.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:57.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:57.787 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key J46L5YZ9VLF35XGJ --secret u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T17:48:57.839 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:57.840 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:57.903 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:48:57.904 INFO:teuthology.orchestra.run.smithi017.stdout: "user_id": "realm-admin", 2023-11-02T17:48:57.904 INFO:teuthology.orchestra.run.smithi017.stdout: "display_name": "Realm Admin", 2023-11-02T17:48:57.904 INFO:teuthology.orchestra.run.smithi017.stdout: "email": "", 2023-11-02T17:48:57.904 INFO:teuthology.orchestra.run.smithi017.stdout: "suspended": 0, 2023-11-02T17:48:57.905 INFO:teuthology.orchestra.run.smithi017.stdout: "max_buckets": 1000, 2023-11-02T17:48:57.905 INFO:teuthology.orchestra.run.smithi017.stdout: "subusers": [], 2023-11-02T17:48:57.905 INFO:teuthology.orchestra.run.smithi017.stdout: "keys": [ 2023-11-02T17:48:57.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:48:57.905 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "realm-admin", 2023-11-02T17:48:57.905 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T17:48:57.906 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T17:48:57.906 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:57.906 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:57.906 INFO:teuthology.orchestra.run.smithi017.stdout: "swift_keys": [], 2023-11-02T17:48:57.906 INFO:teuthology.orchestra.run.smithi017.stdout: "caps": [], 2023-11-02T17:48:57.907 INFO:teuthology.orchestra.run.smithi017.stdout: "op_mask": "read, write, delete", 2023-11-02T17:48:57.907 INFO:teuthology.orchestra.run.smithi017.stdout: "system": true, 2023-11-02T17:48:57.907 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "", 2023-11-02T17:48:57.907 INFO:teuthology.orchestra.run.smithi017.stdout: "default_storage_class": "", 2023-11-02T17:48:57.907 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_tags": [], 2023-11-02T17:48:57.908 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T17:48:57.908 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T17:48:57.908 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T17:48:57.908 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T17:48:57.908 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T17:48:57.908 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T17:48:57.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:57.909 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T17:48:57.909 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T17:48:57.909 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T17:48:57.909 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T17:48:57.910 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T17:48:57.910 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T17:48:57.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:57.910 INFO:teuthology.orchestra.run.smithi017.stdout: "temp_url_keys": [], 2023-11-02T17:48:57.910 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "rgw", 2023-11-02T17:48:57.911 INFO:teuthology.orchestra.run.smithi017.stdout: "mfa_ids": [] 2023-11-02T17:48:57.911 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:48:57.911 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:48:57.931 INFO:tasks.rgw_multisite:creating zone.. 2023-11-02T17:48:57.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--endpoints', 'http://smithi017.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:57.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--endpoints', 'http://smithi017.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:57.932 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key J46L5YZ9VLF35XGJ --secret u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9 --endpoints http://smithi017.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T17:48:57.980 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:57.980 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:58.013 INFO:teuthology.orchestra.run.smithi017.stderr:NOTICE: set zone's realm_id=a2bfe17b-2975-4441-84f6-480a071e229b 2023-11-02T17:48:58.056 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:48:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T17:48:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T17:48:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "a2.rgw.meta:root", 2023-11-02T17:48:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "a2.rgw.control", 2023-11-02T17:48:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "a2.rgw.log:gc", 2023-11-02T17:48:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "a2.rgw.log:lc", 2023-11-02T17:48:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "a2.rgw.log", 2023-11-02T17:48:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-11-02T17:48:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-11-02T17:48:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-11-02T17:48:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-11-02T17:48:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-11-02T17:48:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-11-02T17:48:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-11-02T17:48:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-11-02T17:48:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "a2.rgw.otp", 2023-11-02T17:48:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2023-11-02T17:48:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T17:48:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T17:48:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2023-11-02T17:48:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:48:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2023-11-02T17:48:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:48:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "a2.rgw.buckets.index", 2023-11-02T17:48:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2023-11-02T17:48:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2023-11-02T17:48:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "a2.rgw.buckets.data" 2023-11-02T17:48:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-11-02T17:48:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0, 2023-11-02T17:48:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: "inline_data": true 2023-11-02T17:48:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:48:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "a2.rgw.log:notif" 2023-11-02T17:48:58.065 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:48:58.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2023-11-02T17:48:58.442 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'a2.rgw.buckets.data' created 2023-11-02T17:48:58.452 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2023-11-02T17:48:59.474 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2023-11-02T17:48:59.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:59.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:59.496 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T17:48:59.537 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:59.537 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:59.568 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:48:59.568 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T17:48:59.569 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T17:48:59.569 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "a2.rgw.meta:root", 2023-11-02T17:48:59.569 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "a2.rgw.control", 2023-11-02T17:48:59.569 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "a2.rgw.log:gc", 2023-11-02T17:48:59.569 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "a2.rgw.log:lc", 2023-11-02T17:48:59.569 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "a2.rgw.log", 2023-11-02T17:48:59.570 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-11-02T17:48:59.570 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-11-02T17:48:59.570 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-11-02T17:48:59.570 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-11-02T17:48:59.571 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-11-02T17:48:59.571 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-11-02T17:48:59.571 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-11-02T17:48:59.571 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-11-02T17:48:59.571 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "a2.rgw.otp", 2023-11-02T17:48:59.571 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2023-11-02T17:48:59.572 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T17:48:59.572 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T17:48:59.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:59.572 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2023-11-02T17:48:59.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:48:59.572 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2023-11-02T17:48:59.573 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:48:59.573 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "a2.rgw.buckets.index", 2023-11-02T17:48:59.573 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2023-11-02T17:48:59.573 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2023-11-02T17:48:59.573 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "a2.rgw.buckets.data", 2023-11-02T17:48:59.574 INFO:teuthology.orchestra.run.smithi017.stdout: "compression_type": "random" 2023-11-02T17:48:59.574 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:59.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:48:59.574 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-11-02T17:48:59.574 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0, 2023-11-02T17:48:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: "inline_data": true 2023-11-02T17:48:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:48:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:48:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:48:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "a2.rgw.log:notif" 2023-11-02T17:48:59.576 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:48:59.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:59.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:48:59.577 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T17:48:59.621 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:48:59.621 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:48:59.644 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:48:59.638+0000 7fdab086bf40 0 period (b3a94447-46df-4768-8340-cf63e4392a5c does not have zone 39a01552-6d59-4ce7-b7e5-915ca69b979a configured 2023-11-02T17:49:02.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-11-02T17:49:06.552 INFO:teuthology.orchestra.run.smithi017.stderr:Sending period to new master zone d61b2bea-59a8-4454-8491-ac3afbd04937 2023-11-02T17:49:06.587 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:49:06.587 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:49:06.587 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2, 2023-11-02T17:49:06.587 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T17:49:06.588 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T17:49:06.588 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T17:49:06.588 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:49:06.588 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T17:49:06.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:06.589 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T17:49:06.589 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T17:49:06.589 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T17:49:06.589 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T17:49:06.589 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T17:49:06.590 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T17:49:06.590 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:49:06.590 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T17:49:06.590 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T17:49:06.590 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:49:06.591 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T17:49:06.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:06.591 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T17:49:06.591 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T17:49:06.591 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T17:49:06.592 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T17:49:06.592 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:49:06.592 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T17:49:06.592 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T17:49:06.592 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T17:49:06.593 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T17:49:06.593 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T17:49:06.593 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T17:49:06.593 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T17:49:06.593 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T17:49:06.594 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T17:49:06.594 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T17:49:06.594 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T17:49:06.594 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:49:06.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:06.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:06.595 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:49:06.595 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T17:49:06.595 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T17:49:06.595 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T17:49:06.596 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:49:06.596 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T17:49:06.596 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T17:49:06.596 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T17:49:06.596 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T17:49:06.597 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T17:49:06.597 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T17:49:06.597 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T17:49:06.597 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T17:49:06.597 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T17:49:06.598 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T17:49:06.598 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T17:49:06.598 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:49:06.598 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:06.598 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:49:06.599 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T17:49:06.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:06.599 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T17:49:06.599 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T17:49:06.599 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T17:49:06.600 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T17:49:06.600 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:49:06.600 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:06.600 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:49:06.600 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T17:49:06.601 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:06.601 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T17:49:06.601 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T17:49:06.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:06.601 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T17:49:06.602 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T17:49:06.602 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:49:06.602 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:06.602 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:49:06.602 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T17:49:06.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:06.603 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T17:49:06.603 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T17:49:06.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:06.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:06.604 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:49:06.604 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T17:49:06.604 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:06.604 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:49:06.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:06.604 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T17:49:06.605 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:49:06.605 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T17:49:06.605 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T17:49:06.605 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T17:49:06.606 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T17:49:06.606 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T17:49:06.606 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T17:49:06.606 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T17:49:06.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:06.607 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T17:49:06.607 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T17:49:06.607 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T17:49:06.607 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T17:49:06.607 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T17:49:06.608 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T17:49:06.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:06.608 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T17:49:06.608 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T17:49:06.608 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T17:49:06.609 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T17:49:06.609 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T17:49:06.609 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T17:49:06.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:06.609 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T17:49:06.610 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T17:49:06.610 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T17:49:06.610 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T17:49:06.610 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T17:49:06.612 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T17:49:06.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:06.612 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T17:49:06.612 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T17:49:06.613 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T17:49:06.613 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T17:49:06.613 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T17:49:06.613 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T17:49:06.613 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:06.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:06.614 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:06.614 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T17:49:06.614 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:06.615 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:49:06.615 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-11-02T17:49:06.616 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T17:49:06.616 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T17:49:06.609+0000 7fed07dbe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 32294) UID: 0 2023-11-02T17:49:06.617 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T17:49:06.609+0000 7fed0b61df80 -1 shutting down 2023-11-02T17:49:08.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-11-02T17:49:12.718 INFO:tasks.rgw.c1.client.1:Stopped 2023-11-02T17:49:12.718 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-11-02T17:49:12.719 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-11-02T17:49:12.722 INFO:tasks.rgw.c1.client.1:Started 2023-11-02T17:49:12.722 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T17:49:12.741 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:49:12.741 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:49:12.741 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-11-02T17:49:12.741 INFO:teuthology.orchestra.run.smithi017.stderr:curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8001: Connection refused 2023-11-02T17:49:12.742 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-11-02T17:49:13.744 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T17:49:13.761 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:49:13.761 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:49:13.765 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-11-02T17:49:13.767 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2023-11-02T17:49:13.768 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-11-02T17:49:13.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:13.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:13.770 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi017.front.sepia.ceph.com:8000 --access-key J46L5YZ9VLF35XGJ --secret u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-11-02T17:49:13.819 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:13.819 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:13.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-11-02T17:49:13.892 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:49:13.892 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:13.892 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "test-realm", 2023-11-02T17:49:13.892 INFO:teuthology.orchestra.run.smithi069.stdout: "current_period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:49:13.893 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 2 2023-11-02T17:49:13.893 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:49:13.895 INFO:tasks.rgw_multisite:creating zonegroup.. 2023-11-02T17:49:13.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:13.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:13.896 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2023-11-02T17:49:13.938 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:13.938 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:13.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:49:13.962+0000 7f2d0d7dbf40 -1 old format 2023-11-02T17:49:13.982 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:49:13.982 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "b", 2023-11-02T17:49:13.982 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "b", 2023-11-02T17:49:13.982 INFO:teuthology.orchestra.run.smithi069.stdout: "api_name": "", 2023-11-02T17:49:13.983 INFO:teuthology.orchestra.run.smithi069.stdout: "is_master": false, 2023-11-02T17:49:13.983 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [ 2023-11-02T17:49:13.983 INFO:teuthology.orchestra.run.smithi069.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T17:49:13.983 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:13.984 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames": [], 2023-11-02T17:49:13.984 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames_s3website": [], 2023-11-02T17:49:13.984 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zone": "", 2023-11-02T17:49:13.984 INFO:teuthology.orchestra.run.smithi069.stdout: "zones": [], 2023-11-02T17:49:13.984 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_targets": [ 2023-11-02T17:49:13.985 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:13.985 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default-placement", 2023-11-02T17:49:13.985 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2023-11-02T17:49:13.985 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": [] 2023-11-02T17:49:13.986 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:13.986 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:13.986 INFO:teuthology.orchestra.run.smithi069.stdout: "default_placement": "default-placement", 2023-11-02T17:49:13.986 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:13.987 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_policy": { 2023-11-02T17:49:13.987 INFO:teuthology.orchestra.run.smithi069.stdout: "groups": [] 2023-11-02T17:49:13.987 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:13.987 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled_features": [ 2023-11-02T17:49:13.987 INFO:teuthology.orchestra.run.smithi069.stdout: "resharding" 2023-11-02T17:49:13.988 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:13.988 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:49:13.989 INFO:tasks.rgw_multisite:creating zone.. 2023-11-02T17:49:13.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--endpoints', 'http://smithi069.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:13.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--endpoints', 'http://smithi069.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:13.990 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key J46L5YZ9VLF35XGJ --secret u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9 --endpoints http://smithi069.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-11-02T17:49:14.041 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:14.042 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:14.069 INFO:teuthology.orchestra.run.smithi069.stderr:NOTICE: set zone's realm_id=a2bfe17b-2975-4441-84f6-480a071e229b 2023-11-02T17:49:14.108 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:49:14.108 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T17:49:14.109 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "b1", 2023-11-02T17:49:14.109 INFO:teuthology.orchestra.run.smithi069.stdout: "domain_root": "b1.rgw.meta:root", 2023-11-02T17:49:14.109 INFO:teuthology.orchestra.run.smithi069.stdout: "control_pool": "b1.rgw.control", 2023-11-02T17:49:14.109 INFO:teuthology.orchestra.run.smithi069.stdout: "gc_pool": "b1.rgw.log:gc", 2023-11-02T17:49:14.110 INFO:teuthology.orchestra.run.smithi069.stdout: "lc_pool": "b1.rgw.log:lc", 2023-11-02T17:49:14.110 INFO:teuthology.orchestra.run.smithi069.stdout: "log_pool": "b1.rgw.log", 2023-11-02T17:49:14.110 INFO:teuthology.orchestra.run.smithi069.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-11-02T17:49:14.110 INFO:teuthology.orchestra.run.smithi069.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-11-02T17:49:14.111 INFO:teuthology.orchestra.run.smithi069.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-11-02T17:49:14.111 INFO:teuthology.orchestra.run.smithi069.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-11-02T17:49:14.111 INFO:teuthology.orchestra.run.smithi069.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-11-02T17:49:14.111 INFO:teuthology.orchestra.run.smithi069.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-11-02T17:49:14.112 INFO:teuthology.orchestra.run.smithi069.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-11-02T17:49:14.112 INFO:teuthology.orchestra.run.smithi069.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-11-02T17:49:14.112 INFO:teuthology.orchestra.run.smithi069.stdout: "otp_pool": "b1.rgw.otp", 2023-11-02T17:49:14.112 INFO:teuthology.orchestra.run.smithi069.stdout: "system_key": { 2023-11-02T17:49:14.112 INFO:teuthology.orchestra.run.smithi069.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T17:49:14.113 INFO:teuthology.orchestra.run.smithi069.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T17:49:14.113 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:14.113 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_pools": [ 2023-11-02T17:49:14.113 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:14.114 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "default-placement", 2023-11-02T17:49:14.114 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:49:14.114 INFO:teuthology.orchestra.run.smithi069.stdout: "index_pool": "b1.rgw.buckets.index", 2023-11-02T17:49:14.114 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": { 2023-11-02T17:49:14.115 INFO:teuthology.orchestra.run.smithi069.stdout: "STANDARD": { 2023-11-02T17:49:14.115 INFO:teuthology.orchestra.run.smithi069.stdout: "data_pool": "b1.rgw.buckets.data" 2023-11-02T17:49:14.115 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:14.115 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:14.115 INFO:teuthology.orchestra.run.smithi069.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-11-02T17:49:14.116 INFO:teuthology.orchestra.run.smithi069.stdout: "index_type": 0, 2023-11-02T17:49:14.116 INFO:teuthology.orchestra.run.smithi069.stdout: "inline_data": true 2023-11-02T17:49:14.116 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:14.116 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:14.117 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:14.117 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:14.117 INFO:teuthology.orchestra.run.smithi069.stdout: "notif_pool": "b1.rgw.log:notif" 2023-11-02T17:49:14.117 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:49:14.118 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2023-11-02T17:49:15.172 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'b1.rgw.buckets.data' created 2023-11-02T17:49:15.194 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2023-11-02T17:49:16.190 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2023-11-02T17:49:16.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:16.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:16.214 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-11-02T17:49:16.252 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:16.253 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:16.285 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:49:16.285 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T17:49:16.285 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "b1", 2023-11-02T17:49:16.286 INFO:teuthology.orchestra.run.smithi069.stdout: "domain_root": "b1.rgw.meta:root", 2023-11-02T17:49:16.286 INFO:teuthology.orchestra.run.smithi069.stdout: "control_pool": "b1.rgw.control", 2023-11-02T17:49:16.286 INFO:teuthology.orchestra.run.smithi069.stdout: "gc_pool": "b1.rgw.log:gc", 2023-11-02T17:49:16.286 INFO:teuthology.orchestra.run.smithi069.stdout: "lc_pool": "b1.rgw.log:lc", 2023-11-02T17:49:16.286 INFO:teuthology.orchestra.run.smithi069.stdout: "log_pool": "b1.rgw.log", 2023-11-02T17:49:16.287 INFO:teuthology.orchestra.run.smithi069.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-11-02T17:49:16.287 INFO:teuthology.orchestra.run.smithi069.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-11-02T17:49:16.287 INFO:teuthology.orchestra.run.smithi069.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-11-02T17:49:16.287 INFO:teuthology.orchestra.run.smithi069.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-11-02T17:49:16.287 INFO:teuthology.orchestra.run.smithi069.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-11-02T17:49:16.288 INFO:teuthology.orchestra.run.smithi069.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-11-02T17:49:16.288 INFO:teuthology.orchestra.run.smithi069.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-11-02T17:49:16.288 INFO:teuthology.orchestra.run.smithi069.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-11-02T17:49:16.288 INFO:teuthology.orchestra.run.smithi069.stdout: "otp_pool": "b1.rgw.otp", 2023-11-02T17:49:16.288 INFO:teuthology.orchestra.run.smithi069.stdout: "system_key": { 2023-11-02T17:49:16.289 INFO:teuthology.orchestra.run.smithi069.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T17:49:16.289 INFO:teuthology.orchestra.run.smithi069.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T17:49:16.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:16.289 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_pools": [ 2023-11-02T17:49:16.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:16.289 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "default-placement", 2023-11-02T17:49:16.290 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:49:16.290 INFO:teuthology.orchestra.run.smithi069.stdout: "index_pool": "b1.rgw.buckets.index", 2023-11-02T17:49:16.290 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": { 2023-11-02T17:49:16.290 INFO:teuthology.orchestra.run.smithi069.stdout: "STANDARD": { 2023-11-02T17:49:16.290 INFO:teuthology.orchestra.run.smithi069.stdout: "data_pool": "b1.rgw.buckets.data", 2023-11-02T17:49:16.291 INFO:teuthology.orchestra.run.smithi069.stdout: "compression_type": "random" 2023-11-02T17:49:16.291 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:16.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:16.291 INFO:teuthology.orchestra.run.smithi069.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-11-02T17:49:16.292 INFO:teuthology.orchestra.run.smithi069.stdout: "index_type": 0, 2023-11-02T17:49:16.292 INFO:teuthology.orchestra.run.smithi069.stdout: "inline_data": true 2023-11-02T17:49:16.292 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:16.292 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:16.292 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:16.293 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:16.293 INFO:teuthology.orchestra.run.smithi069.stdout: "notif_pool": "b1.rgw.log:notif" 2023-11-02T17:49:16.293 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:49:16.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:16.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:16.295 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-11-02T17:49:16.375 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:16.377 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:16.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:49:16.390+0000 7f2d9fc0df40 0 period (b3a94447-46df-4768-8340-cf63e4392a5c does not have zone bf0a8e78-6008-4d42-8f71-c8dec4a40cb0 configured 2023-11-02T17:49:23.270 INFO:teuthology.orchestra.run.smithi069.stderr:Sending period to new master zone d61b2bea-59a8-4454-8491-ac3afbd04937 2023-11-02T17:49:23.332 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:49:23.332 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:49:23.333 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 3, 2023-11-02T17:49:23.333 INFO:teuthology.orchestra.run.smithi069.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T17:49:23.333 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_status": [], 2023-11-02T17:49:23.333 INFO:teuthology.orchestra.run.smithi069.stdout: "period_map": { 2023-11-02T17:49:23.333 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:49:23.334 INFO:teuthology.orchestra.run.smithi069.stdout: "zonegroups": [ 2023-11-02T17:49:23.334 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:23.334 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "a", 2023-11-02T17:49:23.334 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "a", 2023-11-02T17:49:23.335 INFO:teuthology.orchestra.run.smithi069.stdout: "api_name": "", 2023-11-02T17:49:23.335 INFO:teuthology.orchestra.run.smithi069.stdout: "is_master": true, 2023-11-02T17:49:23.335 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [ 2023-11-02T17:49:23.335 INFO:teuthology.orchestra.run.smithi069.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T17:49:23.336 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:23.336 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames": [], 2023-11-02T17:49:23.336 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames_s3website": [], 2023-11-02T17:49:23.336 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:49:23.337 INFO:teuthology.orchestra.run.smithi069.stdout: "zones": [ 2023-11-02T17:49:23.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:23.337 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T17:49:23.337 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "a2", 2023-11-02T17:49:23.337 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [ 2023-11-02T17:49:23.338 INFO:teuthology.orchestra.run.smithi069.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T17:49:23.338 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:23.338 INFO:teuthology.orchestra.run.smithi069.stdout: "log_meta": false, 2023-11-02T17:49:23.338 INFO:teuthology.orchestra.run.smithi069.stdout: "log_data": true, 2023-11-02T17:49:23.339 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_index_max_shards": 11, 2023-11-02T17:49:23.339 INFO:teuthology.orchestra.run.smithi069.stdout: "read_only": false, 2023-11-02T17:49:23.339 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_type": "", 2023-11-02T17:49:23.339 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from_all": true, 2023-11-02T17:49:23.339 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from": [], 2023-11-02T17:49:23.340 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect_zone": "", 2023-11-02T17:49:23.340 INFO:teuthology.orchestra.run.smithi069.stdout: "supported_features": [ 2023-11-02T17:49:23.340 INFO:teuthology.orchestra.run.smithi069.stdout: "compress-encrypted", 2023-11-02T17:49:23.340 INFO:teuthology.orchestra.run.smithi069.stdout: "resharding" 2023-11-02T17:49:23.340 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:23.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:23.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:23.341 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:49:23.341 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "a1", 2023-11-02T17:49:23.341 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [ 2023-11-02T17:49:23.342 INFO:teuthology.orchestra.run.smithi069.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T17:49:23.342 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:23.342 INFO:teuthology.orchestra.run.smithi069.stdout: "log_meta": false, 2023-11-02T17:49:23.342 INFO:teuthology.orchestra.run.smithi069.stdout: "log_data": true, 2023-11-02T17:49:23.342 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_index_max_shards": 11, 2023-11-02T17:49:23.343 INFO:teuthology.orchestra.run.smithi069.stdout: "read_only": false, 2023-11-02T17:49:23.343 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_type": "", 2023-11-02T17:49:23.343 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from_all": true, 2023-11-02T17:49:23.343 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from": [], 2023-11-02T17:49:23.343 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect_zone": "", 2023-11-02T17:49:23.344 INFO:teuthology.orchestra.run.smithi069.stdout: "supported_features": [ 2023-11-02T17:49:23.344 INFO:teuthology.orchestra.run.smithi069.stdout: "compress-encrypted", 2023-11-02T17:49:23.344 INFO:teuthology.orchestra.run.smithi069.stdout: "resharding" 2023-11-02T17:49:23.344 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:23.344 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:23.345 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:23.345 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_targets": [ 2023-11-02T17:49:23.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:23.345 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default-placement", 2023-11-02T17:49:23.345 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2023-11-02T17:49:23.346 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": [ 2023-11-02T17:49:23.346 INFO:teuthology.orchestra.run.smithi069.stdout: "STANDARD" 2023-11-02T17:49:23.346 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:23.346 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:23.346 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:23.347 INFO:teuthology.orchestra.run.smithi069.stdout: "default_placement": "default-placement", 2023-11-02T17:49:23.347 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:23.347 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_policy": { 2023-11-02T17:49:23.347 INFO:teuthology.orchestra.run.smithi069.stdout: "groups": [] 2023-11-02T17:49:23.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:23.348 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled_features": [ 2023-11-02T17:49:23.348 INFO:teuthology.orchestra.run.smithi069.stdout: "resharding" 2023-11-02T17:49:23.348 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:23.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:23.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:23.349 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "b", 2023-11-02T17:49:23.349 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "b", 2023-11-02T17:49:23.349 INFO:teuthology.orchestra.run.smithi069.stdout: "api_name": "", 2023-11-02T17:49:23.349 INFO:teuthology.orchestra.run.smithi069.stdout: "is_master": false, 2023-11-02T17:49:23.349 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [ 2023-11-02T17:49:23.350 INFO:teuthology.orchestra.run.smithi069.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T17:49:23.350 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:23.350 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames": [], 2023-11-02T17:49:23.350 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames_s3website": [], 2023-11-02T17:49:23.350 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T17:49:23.351 INFO:teuthology.orchestra.run.smithi069.stdout: "zones": [ 2023-11-02T17:49:23.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:23.351 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T17:49:23.351 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "b1", 2023-11-02T17:49:23.351 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [ 2023-11-02T17:49:23.352 INFO:teuthology.orchestra.run.smithi069.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T17:49:23.352 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:23.352 INFO:teuthology.orchestra.run.smithi069.stdout: "log_meta": false, 2023-11-02T17:49:23.352 INFO:teuthology.orchestra.run.smithi069.stdout: "log_data": false, 2023-11-02T17:49:23.352 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_index_max_shards": 11, 2023-11-02T17:49:23.353 INFO:teuthology.orchestra.run.smithi069.stdout: "read_only": false, 2023-11-02T17:49:23.353 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_type": "", 2023-11-02T17:49:23.353 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from_all": true, 2023-11-02T17:49:23.353 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from": [], 2023-11-02T17:49:23.353 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect_zone": "", 2023-11-02T17:49:23.354 INFO:teuthology.orchestra.run.smithi069.stdout: "supported_features": [ 2023-11-02T17:49:23.354 INFO:teuthology.orchestra.run.smithi069.stdout: "compress-encrypted", 2023-11-02T17:49:23.356 INFO:teuthology.orchestra.run.smithi069.stdout: "resharding" 2023-11-02T17:49:23.356 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:23.356 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:23.356 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:23.356 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_targets": [ 2023-11-02T17:49:23.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:23.357 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default-placement", 2023-11-02T17:49:23.357 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2023-11-02T17:49:23.357 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": [ 2023-11-02T17:49:23.358 INFO:teuthology.orchestra.run.smithi069.stdout: "STANDARD" 2023-11-02T17:49:23.358 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:23.358 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:23.358 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:23.358 INFO:teuthology.orchestra.run.smithi069.stdout: "default_placement": "default-placement", 2023-11-02T17:49:23.359 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:23.359 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_policy": { 2023-11-02T17:49:23.359 INFO:teuthology.orchestra.run.smithi069.stdout: "groups": [] 2023-11-02T17:49:23.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:23.359 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled_features": [ 2023-11-02T17:49:23.360 INFO:teuthology.orchestra.run.smithi069.stdout: "resharding" 2023-11-02T17:49:23.360 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:23.360 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:23.360 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:23.360 INFO:teuthology.orchestra.run.smithi069.stdout: "short_zone_ids": [ 2023-11-02T17:49:23.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:23.361 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T17:49:23.361 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 3699239738 2023-11-02T17:49:23.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:23.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:23.361 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T17:49:23.362 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 2602129372 2023-11-02T17:49:23.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:23.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:23.362 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:49:23.362 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 2073569678 2023-11-02T17:49:23.363 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:23.363 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:23.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:23.363 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zonegroup": "a", 2023-11-02T17:49:23.364 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:49:23.364 INFO:teuthology.orchestra.run.smithi069.stdout: "period_config": { 2023-11-02T17:49:23.364 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_quota": { 2023-11-02T17:49:23.364 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false, 2023-11-02T17:49:23.364 INFO:teuthology.orchestra.run.smithi069.stdout: "check_on_raw": false, 2023-11-02T17:49:23.365 INFO:teuthology.orchestra.run.smithi069.stdout: "max_size": -1, 2023-11-02T17:49:23.365 INFO:teuthology.orchestra.run.smithi069.stdout: "max_size_kb": 0, 2023-11-02T17:49:23.365 INFO:teuthology.orchestra.run.smithi069.stdout: "max_objects": -1 2023-11-02T17:49:23.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:23.365 INFO:teuthology.orchestra.run.smithi069.stdout: "user_quota": { 2023-11-02T17:49:23.366 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false, 2023-11-02T17:49:23.366 INFO:teuthology.orchestra.run.smithi069.stdout: "check_on_raw": false, 2023-11-02T17:49:23.366 INFO:teuthology.orchestra.run.smithi069.stdout: "max_size": -1, 2023-11-02T17:49:23.366 INFO:teuthology.orchestra.run.smithi069.stdout: "max_size_kb": 0, 2023-11-02T17:49:23.366 INFO:teuthology.orchestra.run.smithi069.stdout: "max_objects": -1 2023-11-02T17:49:23.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:23.367 INFO:teuthology.orchestra.run.smithi069.stdout: "user_ratelimit": { 2023-11-02T17:49:23.367 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_ops": 0, 2023-11-02T17:49:23.367 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_ops": 0, 2023-11-02T17:49:23.367 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_bytes": 0, 2023-11-02T17:49:23.368 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_bytes": 0, 2023-11-02T17:49:23.368 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false 2023-11-02T17:49:23.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:23.368 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_ratelimit": { 2023-11-02T17:49:23.368 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_ops": 0, 2023-11-02T17:49:23.369 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_ops": 0, 2023-11-02T17:49:23.369 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_bytes": 0, 2023-11-02T17:49:23.369 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_bytes": 0, 2023-11-02T17:49:23.369 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false 2023-11-02T17:49:23.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:23.370 INFO:teuthology.orchestra.run.smithi069.stdout: "anonymous_ratelimit": { 2023-11-02T17:49:23.370 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_ops": 0, 2023-11-02T17:49:23.370 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_ops": 0, 2023-11-02T17:49:23.370 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_bytes": 0, 2023-11-02T17:49:23.370 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_bytes": 0, 2023-11-02T17:49:23.371 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false 2023-11-02T17:49:23.371 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:23.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:23.371 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:23.371 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_name": "test-realm", 2023-11-02T17:49:23.372 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:49:23.372 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:49:23.373 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-11-02T17:49:23.373 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T17:49:23.374 INFO:tasks.rgw.c2.client.0.smithi069.stdout:2023-11-02T17:49:23.366+0000 7f0251e2b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 33246) UID: 0 2023-11-02T17:49:23.374 INFO:tasks.rgw.c2.client.0.smithi069.stdout:2023-11-02T17:49:23.366+0000 7f025568af80 -1 shutting down 2023-11-02T17:49:24.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-11-02T17:49:29.475 INFO:tasks.rgw.c2.client.0:Stopped 2023-11-02T17:49:29.475 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-11-02T17:49:29.476 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-11-02T17:49:29.479 INFO:tasks.rgw.c2.client.0:Started 2023-11-02T17:49:29.480 DEBUG:teuthology.orchestra.run.smithi069:> curl http://smithi069.front.sepia.ceph.com:8000 2023-11-02T17:49:29.497 INFO:teuthology.orchestra.run.smithi069.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:49:29.497 INFO:teuthology.orchestra.run.smithi069.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:49:29.498 INFO:teuthology.orchestra.run.smithi069.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-11-02T17:49:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-11-02T17:49:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:curl: (7) Failed to connect to smithi069.front.sepia.ceph.com port 8000: Connection refused 2023-11-02T17:49:29.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-11-02T17:49:30.500 DEBUG:teuthology.orchestra.run.smithi069:> curl http://smithi069.front.sepia.ceph.com:8000 2023-11-02T17:49:30.518 INFO:teuthology.orchestra.run.smithi069.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:49:30.518 INFO:teuthology.orchestra.run.smithi069.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:49:30.521 INFO:teuthology.orchestra.run.smithi069.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-11-02T17:49:30.522 INFO:teuthology.orchestra.run.smithi069.stdout:anonymous 2023-11-02T17:49:30.522 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-11-02T17:49:30.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:30.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:30.523 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi017.front.sepia.ceph.com:8000 --access-key J46L5YZ9VLF35XGJ --secret u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9 --url http://smithi017.front.sepia.ceph.com:8000 --access-key J46L5YZ9VLF35XGJ --secret u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-11-02T17:49:30.573 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:30.573 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:30.645 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:49:30.646 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:30.646 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "test-realm", 2023-11-02T17:49:30.646 INFO:teuthology.orchestra.run.smithi069.stdout: "current_period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:49:30.646 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 2 2023-11-02T17:49:30.647 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:49:30.648 INFO:tasks.rgw_multisite:creating zone.. 2023-11-02T17:49:30.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--endpoints', 'http://smithi069.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:30.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--endpoints', 'http://smithi069.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:30.650 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key J46L5YZ9VLF35XGJ --secret u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9 --endpoints http://smithi069.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-11-02T17:49:30.701 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:30.701 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:30.730 INFO:teuthology.orchestra.run.smithi069.stderr:NOTICE: set zone's realm_id=a2bfe17b-2975-4441-84f6-480a071e229b 2023-11-02T17:49:30.769 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:49:30.769 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T17:49:30.770 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "b2", 2023-11-02T17:49:30.770 INFO:teuthology.orchestra.run.smithi069.stdout: "domain_root": "b2.rgw.meta:root", 2023-11-02T17:49:30.770 INFO:teuthology.orchestra.run.smithi069.stdout: "control_pool": "b2.rgw.control", 2023-11-02T17:49:30.770 INFO:teuthology.orchestra.run.smithi069.stdout: "gc_pool": "b2.rgw.log:gc", 2023-11-02T17:49:30.770 INFO:teuthology.orchestra.run.smithi069.stdout: "lc_pool": "b2.rgw.log:lc", 2023-11-02T17:49:30.771 INFO:teuthology.orchestra.run.smithi069.stdout: "log_pool": "b2.rgw.log", 2023-11-02T17:49:30.771 INFO:teuthology.orchestra.run.smithi069.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-11-02T17:49:30.771 INFO:teuthology.orchestra.run.smithi069.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-11-02T17:49:30.771 INFO:teuthology.orchestra.run.smithi069.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-11-02T17:49:30.771 INFO:teuthology.orchestra.run.smithi069.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-11-02T17:49:30.772 INFO:teuthology.orchestra.run.smithi069.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-11-02T17:49:30.772 INFO:teuthology.orchestra.run.smithi069.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-11-02T17:49:30.772 INFO:teuthology.orchestra.run.smithi069.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-11-02T17:49:30.772 INFO:teuthology.orchestra.run.smithi069.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-11-02T17:49:30.772 INFO:teuthology.orchestra.run.smithi069.stdout: "otp_pool": "b2.rgw.otp", 2023-11-02T17:49:30.773 INFO:teuthology.orchestra.run.smithi069.stdout: "system_key": { 2023-11-02T17:49:30.773 INFO:teuthology.orchestra.run.smithi069.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T17:49:30.773 INFO:teuthology.orchestra.run.smithi069.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T17:49:30.773 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:30.773 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_pools": [ 2023-11-02T17:49:30.774 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:30.774 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "default-placement", 2023-11-02T17:49:30.774 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:49:30.774 INFO:teuthology.orchestra.run.smithi069.stdout: "index_pool": "b2.rgw.buckets.index", 2023-11-02T17:49:30.775 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": { 2023-11-02T17:49:30.775 INFO:teuthology.orchestra.run.smithi069.stdout: "STANDARD": { 2023-11-02T17:49:30.775 INFO:teuthology.orchestra.run.smithi069.stdout: "data_pool": "b2.rgw.buckets.data" 2023-11-02T17:49:30.775 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:30.775 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:30.776 INFO:teuthology.orchestra.run.smithi069.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-11-02T17:49:30.776 INFO:teuthology.orchestra.run.smithi069.stdout: "index_type": 0, 2023-11-02T17:49:30.776 INFO:teuthology.orchestra.run.smithi069.stdout: "inline_data": true 2023-11-02T17:49:30.776 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:30.777 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:30.777 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:30.777 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:30.777 INFO:teuthology.orchestra.run.smithi069.stdout: "notif_pool": "b2.rgw.log:notif" 2023-11-02T17:49:30.777 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:49:30.779 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2023-11-02T17:49:31.197 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'b2.rgw.buckets.data' created 2023-11-02T17:49:31.216 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2023-11-02T17:49:32.222 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2023-11-02T17:49:32.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:32.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:32.251 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-11-02T17:49:32.290 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:32.290 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:32.321 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:49:32.321 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T17:49:32.321 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "b2", 2023-11-02T17:49:32.321 INFO:teuthology.orchestra.run.smithi069.stdout: "domain_root": "b2.rgw.meta:root", 2023-11-02T17:49:32.322 INFO:teuthology.orchestra.run.smithi069.stdout: "control_pool": "b2.rgw.control", 2023-11-02T17:49:32.322 INFO:teuthology.orchestra.run.smithi069.stdout: "gc_pool": "b2.rgw.log:gc", 2023-11-02T17:49:32.322 INFO:teuthology.orchestra.run.smithi069.stdout: "lc_pool": "b2.rgw.log:lc", 2023-11-02T17:49:32.322 INFO:teuthology.orchestra.run.smithi069.stdout: "log_pool": "b2.rgw.log", 2023-11-02T17:49:32.323 INFO:teuthology.orchestra.run.smithi069.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-11-02T17:49:32.323 INFO:teuthology.orchestra.run.smithi069.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-11-02T17:49:32.323 INFO:teuthology.orchestra.run.smithi069.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-11-02T17:49:32.323 INFO:teuthology.orchestra.run.smithi069.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-11-02T17:49:32.324 INFO:teuthology.orchestra.run.smithi069.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-11-02T17:49:32.324 INFO:teuthology.orchestra.run.smithi069.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-11-02T17:49:32.324 INFO:teuthology.orchestra.run.smithi069.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-11-02T17:49:32.324 INFO:teuthology.orchestra.run.smithi069.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-11-02T17:49:32.324 INFO:teuthology.orchestra.run.smithi069.stdout: "otp_pool": "b2.rgw.otp", 2023-11-02T17:49:32.325 INFO:teuthology.orchestra.run.smithi069.stdout: "system_key": { 2023-11-02T17:49:32.325 INFO:teuthology.orchestra.run.smithi069.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T17:49:32.325 INFO:teuthology.orchestra.run.smithi069.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T17:49:32.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:32.326 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_pools": [ 2023-11-02T17:49:32.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:32.326 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "default-placement", 2023-11-02T17:49:32.326 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:49:32.326 INFO:teuthology.orchestra.run.smithi069.stdout: "index_pool": "b2.rgw.buckets.index", 2023-11-02T17:49:32.327 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": { 2023-11-02T17:49:32.327 INFO:teuthology.orchestra.run.smithi069.stdout: "STANDARD": { 2023-11-02T17:49:32.327 INFO:teuthology.orchestra.run.smithi069.stdout: "data_pool": "b2.rgw.buckets.data", 2023-11-02T17:49:32.327 INFO:teuthology.orchestra.run.smithi069.stdout: "compression_type": "random" 2023-11-02T17:49:32.327 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:32.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:32.328 INFO:teuthology.orchestra.run.smithi069.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-11-02T17:49:32.328 INFO:teuthology.orchestra.run.smithi069.stdout: "index_type": 0, 2023-11-02T17:49:32.328 INFO:teuthology.orchestra.run.smithi069.stdout: "inline_data": true 2023-11-02T17:49:32.328 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:32.329 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:32.329 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:32.329 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:32.329 INFO:teuthology.orchestra.run.smithi069.stdout: "notif_pool": "b2.rgw.log:notif" 2023-11-02T17:49:32.329 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:49:32.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:32.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:32.331 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-11-02T17:49:32.374 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:32.374 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:32.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:49:32.390+0000 7f32354c6f40 0 period (b3a94447-46df-4768-8340-cf63e4392a5c does not have zone da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c configured 2023-11-02T17:49:39.307 INFO:teuthology.orchestra.run.smithi069.stderr:Sending period to new master zone d61b2bea-59a8-4454-8491-ac3afbd04937 2023-11-02T17:49:39.368 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:49:39.368 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:49:39.369 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 4, 2023-11-02T17:49:39.369 INFO:teuthology.orchestra.run.smithi069.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T17:49:39.369 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_status": [], 2023-11-02T17:49:39.369 INFO:teuthology.orchestra.run.smithi069.stdout: "period_map": { 2023-11-02T17:49:39.369 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:49:39.370 INFO:teuthology.orchestra.run.smithi069.stdout: "zonegroups": [ 2023-11-02T17:49:39.370 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:39.370 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "a", 2023-11-02T17:49:39.370 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "a", 2023-11-02T17:49:39.371 INFO:teuthology.orchestra.run.smithi069.stdout: "api_name": "", 2023-11-02T17:49:39.371 INFO:teuthology.orchestra.run.smithi069.stdout: "is_master": true, 2023-11-02T17:49:39.371 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [ 2023-11-02T17:49:39.371 INFO:teuthology.orchestra.run.smithi069.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T17:49:39.371 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:39.372 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames": [], 2023-11-02T17:49:39.372 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames_s3website": [], 2023-11-02T17:49:39.372 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:49:39.372 INFO:teuthology.orchestra.run.smithi069.stdout: "zones": [ 2023-11-02T17:49:39.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:39.373 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T17:49:39.373 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "a2", 2023-11-02T17:49:39.373 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [ 2023-11-02T17:49:39.373 INFO:teuthology.orchestra.run.smithi069.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T17:49:39.374 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:39.374 INFO:teuthology.orchestra.run.smithi069.stdout: "log_meta": false, 2023-11-02T17:49:39.374 INFO:teuthology.orchestra.run.smithi069.stdout: "log_data": true, 2023-11-02T17:49:39.374 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_index_max_shards": 11, 2023-11-02T17:49:39.375 INFO:teuthology.orchestra.run.smithi069.stdout: "read_only": false, 2023-11-02T17:49:39.375 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_type": "", 2023-11-02T17:49:39.375 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from_all": true, 2023-11-02T17:49:39.375 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from": [], 2023-11-02T17:49:39.376 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect_zone": "", 2023-11-02T17:49:39.376 INFO:teuthology.orchestra.run.smithi069.stdout: "supported_features": [ 2023-11-02T17:49:39.376 INFO:teuthology.orchestra.run.smithi069.stdout: "compress-encrypted", 2023-11-02T17:49:39.376 INFO:teuthology.orchestra.run.smithi069.stdout: "resharding" 2023-11-02T17:49:39.376 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:39.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:39.377 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:49:39.377 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "a1", 2023-11-02T17:49:39.378 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [ 2023-11-02T17:49:39.378 INFO:teuthology.orchestra.run.smithi069.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T17:49:39.378 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:39.378 INFO:teuthology.orchestra.run.smithi069.stdout: "log_meta": false, 2023-11-02T17:49:39.379 INFO:teuthology.orchestra.run.smithi069.stdout: "log_data": true, 2023-11-02T17:49:39.379 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_index_max_shards": 11, 2023-11-02T17:49:39.379 INFO:teuthology.orchestra.run.smithi069.stdout: "read_only": false, 2023-11-02T17:49:39.379 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_type": "", 2023-11-02T17:49:39.379 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from_all": true, 2023-11-02T17:49:39.380 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from": [], 2023-11-02T17:49:39.380 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect_zone": "", 2023-11-02T17:49:39.380 INFO:teuthology.orchestra.run.smithi069.stdout: "supported_features": [ 2023-11-02T17:49:39.380 INFO:teuthology.orchestra.run.smithi069.stdout: "compress-encrypted", 2023-11-02T17:49:39.381 INFO:teuthology.orchestra.run.smithi069.stdout: "resharding" 2023-11-02T17:49:39.381 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:39.381 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:39.381 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:39.381 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_targets": [ 2023-11-02T17:49:39.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:39.382 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default-placement", 2023-11-02T17:49:39.382 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2023-11-02T17:49:39.382 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": [ 2023-11-02T17:49:39.382 INFO:teuthology.orchestra.run.smithi069.stdout: "STANDARD" 2023-11-02T17:49:39.383 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:39.383 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:39.383 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:39.383 INFO:teuthology.orchestra.run.smithi069.stdout: "default_placement": "default-placement", 2023-11-02T17:49:39.383 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:39.384 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_policy": { 2023-11-02T17:49:39.384 INFO:teuthology.orchestra.run.smithi069.stdout: "groups": [] 2023-11-02T17:49:39.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.384 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled_features": [ 2023-11-02T17:49:39.385 INFO:teuthology.orchestra.run.smithi069.stdout: "resharding" 2023-11-02T17:49:39.385 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:39.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:39.385 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "b", 2023-11-02T17:49:39.386 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "b", 2023-11-02T17:49:39.386 INFO:teuthology.orchestra.run.smithi069.stdout: "api_name": "", 2023-11-02T17:49:39.386 INFO:teuthology.orchestra.run.smithi069.stdout: "is_master": false, 2023-11-02T17:49:39.386 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [ 2023-11-02T17:49:39.386 INFO:teuthology.orchestra.run.smithi069.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T17:49:39.387 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:39.387 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames": [], 2023-11-02T17:49:39.387 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames_s3website": [], 2023-11-02T17:49:39.387 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T17:49:39.388 INFO:teuthology.orchestra.run.smithi069.stdout: "zones": [ 2023-11-02T17:49:39.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:39.388 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T17:49:39.388 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "b1", 2023-11-02T17:49:39.388 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [ 2023-11-02T17:49:39.389 INFO:teuthology.orchestra.run.smithi069.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T17:49:39.389 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:39.389 INFO:teuthology.orchestra.run.smithi069.stdout: "log_meta": false, 2023-11-02T17:49:39.389 INFO:teuthology.orchestra.run.smithi069.stdout: "log_data": true, 2023-11-02T17:49:39.390 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_index_max_shards": 11, 2023-11-02T17:49:39.390 INFO:teuthology.orchestra.run.smithi069.stdout: "read_only": false, 2023-11-02T17:49:39.390 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_type": "", 2023-11-02T17:49:39.390 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from_all": true, 2023-11-02T17:49:39.390 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from": [], 2023-11-02T17:49:39.391 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect_zone": "", 2023-11-02T17:49:39.391 INFO:teuthology.orchestra.run.smithi069.stdout: "supported_features": [ 2023-11-02T17:49:39.391 INFO:teuthology.orchestra.run.smithi069.stdout: "compress-encrypted", 2023-11-02T17:49:39.392 INFO:teuthology.orchestra.run.smithi069.stdout: "resharding" 2023-11-02T17:49:39.392 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:39.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.393 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:39.393 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T17:49:39.393 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "b2", 2023-11-02T17:49:39.394 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [ 2023-11-02T17:49:39.394 INFO:teuthology.orchestra.run.smithi069.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T17:49:39.394 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:39.394 INFO:teuthology.orchestra.run.smithi069.stdout: "log_meta": false, 2023-11-02T17:49:39.395 INFO:teuthology.orchestra.run.smithi069.stdout: "log_data": true, 2023-11-02T17:49:39.395 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_index_max_shards": 11, 2023-11-02T17:49:39.395 INFO:teuthology.orchestra.run.smithi069.stdout: "read_only": false, 2023-11-02T17:49:39.395 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_type": "", 2023-11-02T17:49:39.395 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from_all": true, 2023-11-02T17:49:39.396 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from": [], 2023-11-02T17:49:39.396 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect_zone": "", 2023-11-02T17:49:39.396 INFO:teuthology.orchestra.run.smithi069.stdout: "supported_features": [ 2023-11-02T17:49:39.396 INFO:teuthology.orchestra.run.smithi069.stdout: "compress-encrypted", 2023-11-02T17:49:39.397 INFO:teuthology.orchestra.run.smithi069.stdout: "resharding" 2023-11-02T17:49:39.397 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:39.397 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:39.397 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:39.397 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_targets": [ 2023-11-02T17:49:39.398 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:39.399 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default-placement", 2023-11-02T17:49:39.399 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2023-11-02T17:49:39.399 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": [ 2023-11-02T17:49:39.400 INFO:teuthology.orchestra.run.smithi069.stdout: "STANDARD" 2023-11-02T17:49:39.400 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:39.400 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:39.400 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:39.401 INFO:teuthology.orchestra.run.smithi069.stdout: "default_placement": "default-placement", 2023-11-02T17:49:39.401 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:39.401 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_policy": { 2023-11-02T17:49:39.401 INFO:teuthology.orchestra.run.smithi069.stdout: "groups": [] 2023-11-02T17:49:39.401 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.402 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled_features": [ 2023-11-02T17:49:39.402 INFO:teuthology.orchestra.run.smithi069.stdout: "resharding" 2023-11-02T17:49:39.402 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:39.402 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:39.403 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2023-11-02T17:49:39.403 INFO:teuthology.orchestra.run.smithi069.stdout: "short_zone_ids": [ 2023-11-02T17:49:39.403 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:39.403 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T17:49:39.403 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 3699239738 2023-11-02T17:49:39.404 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.404 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:39.404 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T17:49:39.404 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 2602129372 2023-11-02T17:49:39.404 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.405 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:39.405 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:49:39.405 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 2073569678 2023-11-02T17:49:39.405 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.406 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:49:39.406 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T17:49:39.406 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 3449465395 2023-11-02T17:49:39.406 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:39.406 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:49:39.407 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.407 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zonegroup": "a", 2023-11-02T17:49:39.407 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T17:49:39.407 INFO:teuthology.orchestra.run.smithi069.stdout: "period_config": { 2023-11-02T17:49:39.408 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_quota": { 2023-11-02T17:49:39.408 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false, 2023-11-02T17:49:39.408 INFO:teuthology.orchestra.run.smithi069.stdout: "check_on_raw": false, 2023-11-02T17:49:39.408 INFO:teuthology.orchestra.run.smithi069.stdout: "max_size": -1, 2023-11-02T17:49:39.408 INFO:teuthology.orchestra.run.smithi069.stdout: "max_size_kb": 0, 2023-11-02T17:49:39.409 INFO:teuthology.orchestra.run.smithi069.stdout: "max_objects": -1 2023-11-02T17:49:39.409 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.409 INFO:teuthology.orchestra.run.smithi069.stdout: "user_quota": { 2023-11-02T17:49:39.409 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false, 2023-11-02T17:49:39.409 INFO:teuthology.orchestra.run.smithi069.stdout: "check_on_raw": false, 2023-11-02T17:49:39.410 INFO:teuthology.orchestra.run.smithi069.stdout: "max_size": -1, 2023-11-02T17:49:39.410 INFO:teuthology.orchestra.run.smithi069.stdout: "max_size_kb": 0, 2023-11-02T17:49:39.410 INFO:teuthology.orchestra.run.smithi069.stdout: "max_objects": -1 2023-11-02T17:49:39.410 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.411 INFO:teuthology.orchestra.run.smithi069.stdout: "user_ratelimit": { 2023-11-02T17:49:39.411 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_ops": 0, 2023-11-02T17:49:39.411 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_ops": 0, 2023-11-02T17:49:39.411 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_bytes": 0, 2023-11-02T17:49:39.411 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_bytes": 0, 2023-11-02T17:49:39.412 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false 2023-11-02T17:49:39.412 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.412 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_ratelimit": { 2023-11-02T17:49:39.412 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_ops": 0, 2023-11-02T17:49:39.413 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_ops": 0, 2023-11-02T17:49:39.413 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_bytes": 0, 2023-11-02T17:49:39.413 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_bytes": 0, 2023-11-02T17:49:39.413 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false 2023-11-02T17:49:39.413 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.414 INFO:teuthology.orchestra.run.smithi069.stdout: "anonymous_ratelimit": { 2023-11-02T17:49:39.414 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_ops": 0, 2023-11-02T17:49:39.414 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_ops": 0, 2023-11-02T17:49:39.414 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_bytes": 0, 2023-11-02T17:49:39.415 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_bytes": 0, 2023-11-02T17:49:39.415 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false 2023-11-02T17:49:39.415 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:49:39.415 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:49:39.415 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T17:49:39.416 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_name": "test-realm", 2023-11-02T17:49:39.416 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:49:39.416 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:49:39.417 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-11-02T17:49:39.417 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T17:49:39.435 INFO:tasks.rgw.c2.client.1.smithi069.stdout:2023-11-02T17:49:39.430+0000 7fcbb3880700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 33248) UID: 0 2023-11-02T17:49:39.436 INFO:tasks.rgw.c2.client.1.smithi069.stdout:2023-11-02T17:49:39.430+0000 7fcbb70dff80 -1 shutting down 2023-11-02T17:49:41.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-11-02T17:49:45.520 INFO:tasks.rgw.c2.client.1:Stopped 2023-11-02T17:49:45.520 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-11-02T17:49:45.521 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-11-02T17:49:45.523 INFO:tasks.rgw.c2.client.1:Started 2023-11-02T17:49:45.524 DEBUG:teuthology.orchestra.run.smithi069:> curl http://smithi069.front.sepia.ceph.com:8001 2023-11-02T17:49:45.539 INFO:teuthology.orchestra.run.smithi069.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:49:45.540 INFO:teuthology.orchestra.run.smithi069.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:49:45.540 INFO:teuthology.orchestra.run.smithi069.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-11-02T17:49:45.540 INFO:teuthology.orchestra.run.smithi069.stderr:curl: (7) Failed to connect to smithi069.front.sepia.ceph.com port 8001: Connection refused 2023-11-02T17:49:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-11-02T17:49:46.542 DEBUG:teuthology.orchestra.run.smithi069:> curl http://smithi069.front.sepia.ceph.com:8001 2023-11-02T17:49:46.560 INFO:teuthology.orchestra.run.smithi069.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T17:49:46.561 INFO:teuthology.orchestra.run.smithi069.stderr: Dload Upload Total Spent Left Speed 2023-11-02T17:49:46.564 INFO:teuthology.orchestra.run.smithi069.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-11-02T17:49:46.565 INFO:teuthology.orchestra.run.smithi069.stdout:anonymous 2023-11-02T17:49:46.566 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2023-11-02T17:49:46.566 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2023-11-02T17:49:46.578 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/../src/test/rgw/rgw_multi 2023-11-02T17:49:46.606 INFO:tasks.rgw_multisite_tests:creating test user.. 2023-11-02T17:49:46.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:46.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T17:49:46.607 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T17:49:46.662 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:46.663 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:46.726 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:49:46.727 INFO:teuthology.orchestra.run.smithi017.stdout: "user_id": "rgw-multisite-test-user", 2023-11-02T17:49:46.727 INFO:teuthology.orchestra.run.smithi017.stdout: "display_name": "Multisite Test User", 2023-11-02T17:49:46.727 INFO:teuthology.orchestra.run.smithi017.stdout: "email": "", 2023-11-02T17:49:46.727 INFO:teuthology.orchestra.run.smithi017.stdout: "suspended": 0, 2023-11-02T17:49:46.727 INFO:teuthology.orchestra.run.smithi017.stdout: "max_buckets": 1000, 2023-11-02T17:49:46.728 INFO:teuthology.orchestra.run.smithi017.stdout: "subusers": [], 2023-11-02T17:49:46.728 INFO:teuthology.orchestra.run.smithi017.stdout: "keys": [ 2023-11-02T17:49:46.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:46.728 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "rgw-multisite-test-user", 2023-11-02T17:49:46.728 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "7LD6AK3P9XRGER738BKH", 2023-11-02T17:49:46.729 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "rfQvq9EDj0g0HBMqXGPxxB3CQy7L48r7ch2Byr0V" 2023-11-02T17:49:46.729 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:46.729 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:49:46.729 INFO:teuthology.orchestra.run.smithi017.stdout: "swift_keys": [], 2023-11-02T17:49:46.730 INFO:teuthology.orchestra.run.smithi017.stdout: "caps": [ 2023-11-02T17:49:46.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:46.730 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "roles", 2023-11-02T17:49:46.730 INFO:teuthology.orchestra.run.smithi017.stdout: "perm": "*" 2023-11-02T17:49:46.730 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:46.731 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T17:49:46.731 INFO:teuthology.orchestra.run.smithi017.stdout: "op_mask": "read, write, delete", 2023-11-02T17:49:46.731 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "", 2023-11-02T17:49:46.731 INFO:teuthology.orchestra.run.smithi017.stdout: "default_storage_class": "", 2023-11-02T17:49:46.732 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_tags": [], 2023-11-02T17:49:46.732 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T17:49:46.732 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T17:49:46.732 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T17:49:46.732 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T17:49:46.733 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T17:49:46.733 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T17:49:46.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:46.733 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T17:49:46.733 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T17:49:46.734 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T17:49:46.734 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T17:49:46.734 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T17:49:46.734 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T17:49:46.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:46.735 INFO:teuthology.orchestra.run.smithi017.stdout: "temp_url_keys": [], 2023-11-02T17:49:46.735 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "rgw", 2023-11-02T17:49:46.735 INFO:teuthology.orchestra.run.smithi017.stdout: "mfa_ids": [] 2023-11-02T17:49:46.735 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:49:46.736 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T17:49:46.754 INFO:rgw_multi.tests:meta checkpoint 2023-11-02T17:49:46.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:49:46.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:49:46.756 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:49:46.800 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:46.800 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:46.855 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T17:49:46.881 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T17:49:46.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:46.881 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:49:46.882 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T17:49:46.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:46.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:46.883 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:46.883 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:49:46.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:46.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:46.883 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:46.884 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:49:46.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:46.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:46.884 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:46.884 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:49:46.885 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:46.885 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T17:49:46.886 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T17:49:46.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:49:46.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:49:46.887 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:49:46.910 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-11-02T17:49:46.930 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:46.931 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:49:46.978+0000 7fe50a963f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:49:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:49:46.978+0000 7fe50a963f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:49:47.006 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:49:47.006 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:49:47.007 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:49:47.007 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:49:47.007 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:49:47.007 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:49:47.008 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:47.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:47.008 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:49:47.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:47.008 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:49:47.009 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:49:47.009 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:49:47.009 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:47.009 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:49:47.009 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:49:47.010 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:49:47.010 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:49:47.010 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:47.010 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:47.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:47.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:47.011 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:49:47.011 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:49:47.011 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:49:47.012 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:47.012 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:49:47.012 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:49:47.012 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:49:47.012 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:49:47.013 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:47.013 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:47.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:47.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:47.014 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:49:47.014 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:49:47.014 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:49:47.014 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:47.014 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:49:47.015 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:49:47.015 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:49:47.015 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:49:47.015 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:47.016 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:47.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:47.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:47.016 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:49:47.016 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:49:47.017 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:49:47.017 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:47.017 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:49:47.017 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:49:47.017 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:49:47.018 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:49:47.018 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:47.018 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:47.018 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:47.018 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:49:47.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:47.019 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:49:47.019 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:49:47.019 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:49:47.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:47.020 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:49:46Z" 2023-11-02T17:49:47.020 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:49:47.021 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:49:52.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:49:52.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:49:52.023 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:49:52.069 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:52.070 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:52.126 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:49:52.122+0000 7ffa8c6eef40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:49:52.127 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:49:52.122+0000 7ffa8c6eef40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:49:52.154 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:49:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:49:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:49:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:49:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:49:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:49:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:49:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:49:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:49:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:49:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:49:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:49:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:49:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:49:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:49:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:49:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:49:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:49:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:49:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:49:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:49:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:49:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:49:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:49:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:49:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:49:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:49:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:49:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:49:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:49:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:49:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:49:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:49:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:49:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:49:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:49:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:49:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:49:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:49:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:49:52Z" 2023-11-02T17:49:52.168 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:49:52.169 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:49:57.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:49:57.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:49:57.171 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:49:57.217 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:49:57.217 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:49:57.271 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:49:57.266+0000 7fe13f8e1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:49:57.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:49:57.266+0000 7fe13f8e1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:49:57.296 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:49:57.296 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:49:57.296 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:49:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:49:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:49:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:49:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:49:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:49:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:49:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:49:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:49:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:49:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:49:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:49:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:49:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:49:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:49:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:49:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:49:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:49:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:49:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:49:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:49:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:49:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:49:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:49:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:49:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:49:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:49:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:49:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:49:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:49:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:49:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:49:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:49:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:49:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:49:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:49:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:49:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:49:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:49:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:49:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:49:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:49:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:49:57Z" 2023-11-02T17:49:57.309 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:49:57.310 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:50:02.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:02.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:02.312 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:50:02.359 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:50:02.359 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:50:02.414 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:02.410+0000 7f108fc1bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:02.410+0000 7f108fc1bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:02.440 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:50:02.440 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:50:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:50:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:50:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:50:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:50:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:50:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:02.443 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:50:02.443 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:02.443 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:02.443 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:02.443 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:02.444 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:50:02.444 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:02.444 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:02.444 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:02.445 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:02.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:02.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:02.445 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:50:02.445 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:02.446 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:02.446 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:02.446 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:02.446 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:02.447 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:02.447 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:02.447 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:02.447 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:02.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:50:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:02.449 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:02.449 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:02.449 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:02.449 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:02.450 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:02.450 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:02.450 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:02.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:02.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:02.451 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:50:02.451 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:02.451 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:02.451 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:02.452 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:02.452 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:02.452 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:02.452 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:02.452 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:02.453 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:02.453 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:02.453 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:50:02.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:02.454 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:50:02.454 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:50:02.454 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:50:02.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:02.454 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:50:02Z" 2023-11-02T17:50:02.455 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:50:02.456 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:50:07.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:07.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:07.458 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:50:07.503 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:50:07.503 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:50:07.552 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:07.545+0000 7f4a18e25f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:07.554 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:07.549+0000 7f4a18e25f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:07.579 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:50:07.579 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:50:07.580 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:50:07.580 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:50:07.580 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:50:07.580 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:50:07.580 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:07.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:07.581 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:50:07.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:07.581 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:50:07.582 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:07.582 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:07.582 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:07.582 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:07.582 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:50:07.583 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:07.583 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:07.583 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:07.583 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:07.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:07.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:07.584 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:50:07.584 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:07.584 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:07.585 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:07.585 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:07.585 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:07.585 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:07.585 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:07.585 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:07.586 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:07.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:07.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:07.586 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:50:07.586 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:07.587 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:07.587 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:07.587 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:07.587 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:07.587 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:07.588 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:07.588 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:07.588 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:07.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:07.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:07.589 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:50:07.589 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:07.589 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:07.589 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:07.589 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:07.590 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:07.590 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:07.590 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:07.590 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:07.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:07.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:07.591 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:50:07.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:07.591 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:50:07.591 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:50:07.591 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:50:07.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:07.592 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:50:07Z" 2023-11-02T17:50:07.592 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:50:07.593 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:50:12.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:12.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:12.596 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:50:12.642 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:50:12.642 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:50:12.700 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:12.693+0000 7fac10a62f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:12.702 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:12.697+0000 7fac10a62f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:12.727 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:50:12.727 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:50:12.727 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:50:12.728 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:50:12.728 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:50:12.728 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:50:12.728 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:12.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:12.729 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:50:12.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:12.729 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:50:12.729 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:12.730 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:12.730 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:12.730 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:12.730 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:50:12.730 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:12.731 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:12.731 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:12.731 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:12.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:12.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:12.732 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:50:12.732 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:12.732 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:12.732 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:12.732 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:12.733 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:12.733 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:12.733 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:12.733 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:12.733 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:12.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:12.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:12.734 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:50:12.734 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:12.735 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:12.735 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:12.735 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:12.735 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:12.735 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:12.735 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:12.736 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:12.736 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:12.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:12.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:12.737 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:50:12.737 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:12.737 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:12.737 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:12.737 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:12.738 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:12.738 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:12.738 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:12.738 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:12.738 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:12.739 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:12.739 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:50:12.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:12.739 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:50:12.739 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:50:12.740 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:50:12.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:12.740 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:50:12Z" 2023-11-02T17:50:12.740 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:50:12.741 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:50:17.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:17.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:17.744 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:50:17.791 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:50:17.791 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:50:17.844 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:17.837+0000 7fd979185f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:17.845 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:17.841+0000 7fd979185f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:17.871 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:50:17.871 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:50:17.871 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:50:17.872 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:50:17.872 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:50:17.872 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:50:17.872 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:17.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:17.873 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:50:17.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:17.873 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:50:17.873 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:17.873 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:17.873 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:17.874 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:17.874 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:50:17.874 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:17.874 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:17.874 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:17.875 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:17.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:17.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:17.875 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:50:17.875 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:17.876 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:17.876 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:17.876 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:17.876 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:17.876 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:17.877 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:17.877 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:17.877 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:17.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:17.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:17.878 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:50:17.878 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:17.878 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:17.878 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:17.879 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:17.879 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:17.879 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:17.879 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:17.879 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:17.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:17.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:17.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:17.880 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:50:17.880 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:17.880 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:17.881 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:17.881 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:17.881 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:17.881 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:17.881 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:17.882 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:17.882 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:17.882 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:17.882 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:50:17.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:17.883 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:50:17.883 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:50:17.883 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:50:17.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:17.883 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:50:17Z" 2023-11-02T17:50:17.884 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:50:17.885 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:50:22.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:22.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:22.887 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:50:22.933 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:50:22.933 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:50:22.990 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:22.985+0000 7f050b3d3f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:22.992 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:22.985+0000 7f050b3d3f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:23.017 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:50:23.017 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:50:23.017 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:50:23.017 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:50:23.018 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:50:23.018 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:50:23.018 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:23.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:23.018 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:50:23.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:23.019 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:50:23.019 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:23.019 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:23.019 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:23.020 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:23.020 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:50:23.020 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:23.020 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:23.021 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:23.021 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:23.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:23.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:23.022 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:50:23.022 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:23.022 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:23.022 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:23.023 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:23.023 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:23.023 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:23.023 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:23.023 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:23.024 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:23.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:23.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:23.024 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:50:23.024 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:23.025 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:23.025 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:23.025 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:23.025 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:23.025 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:23.025 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:23.026 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:23.026 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:23.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:23.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:23.027 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:50:23.027 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:23.027 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:23.027 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:23.027 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:23.028 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:23.028 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:23.028 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:23.028 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:23.028 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:23.030 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:23.030 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:50:23.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:23.030 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:50:23.030 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:50:23.031 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:50:23.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:23.031 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:50:22Z" 2023-11-02T17:50:23.031 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:50:23.032 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:50:28.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:28.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:28.035 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:50:28.082 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:50:28.082 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:50:28.139 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:28.133+0000 7fdf0ed4af40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:28.141 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:28.133+0000 7fdf0ed4af40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:28.166 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:50:28.166 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:50:28.167 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:50:28.167 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:50:28.167 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:50:28.167 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:50:28.168 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:28.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:28.168 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:50:28.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:28.168 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:50:28.169 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:28.169 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:28.169 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:28.169 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:28.169 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:50:28.170 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:28.170 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:28.170 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:28.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:28.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:50:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:28.172 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:28.172 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:28.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:28.172 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:28.172 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:28.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:28.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:28.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:28.173 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:50:28.174 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:28.174 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:28.174 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:28.174 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:28.174 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:28.174 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:28.175 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:28.175 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:28.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:28.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:28.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:28.176 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:50:28.176 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:28.176 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:28.176 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:28.176 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:28.177 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:28.177 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:28.177 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:28.177 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:28.177 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:28.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:28.178 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:50:28.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:28.178 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:50:28.178 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:50:28.179 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:50:28.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:28.179 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:50:28Z" 2023-11-02T17:50:28.179 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:50:28.181 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:50:33.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:33.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:33.182 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:50:33.228 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:50:33.229 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:50:33.287 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:33.281+0000 7fa02ebc2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:33.288 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:33.281+0000 7fa02ebc2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:33.313 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:50:33.314 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:50:33.314 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:50:33.314 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:50:33.314 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:50:33.315 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:50:33.315 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:33.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:33.315 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:50:33.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:33.316 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:50:33.316 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:33.316 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:33.316 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:33.316 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:33.317 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:50:33.317 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:33.317 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:33.317 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:33.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:33.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:33.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:33.318 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:50:33.318 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:33.318 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:33.319 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:33.319 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:33.319 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:33.319 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:33.320 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:33.320 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:33.320 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:33.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:33.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:33.320 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:50:33.321 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:33.321 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:33.321 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:33.321 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:33.321 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:33.322 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:33.322 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:33.322 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:33.322 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:33.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:33.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:33.323 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:50:33.323 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:33.323 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:33.323 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:33.324 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:33.324 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:33.324 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:33.324 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:33.324 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:33.325 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:33.325 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:33.325 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:50:33.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:33.325 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:50:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:50:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:50:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:50:33Z" 2023-11-02T17:50:33.326 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:50:33.328 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:50:38.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:38.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:38.330 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:50:38.376 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:50:38.376 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:50:38.434 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:38.429+0000 7f258ba87f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:38.436 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:38.429+0000 7f258ba87f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:38.461 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:50:38.461 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:50:38.461 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:50:38.461 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:50:38.462 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:50:38.462 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:50:38.462 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:38.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:38.462 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:50:38.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:38.463 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:50:38.463 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:38.463 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:38.464 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:38.464 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:38.464 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:50:38.464 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:38.464 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:38.465 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:38.465 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:38.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:38.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:38.466 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:50:38.466 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:38.466 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:38.466 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:38.467 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:38.467 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:38.467 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:38.467 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:38.467 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:38.468 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:38.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:38.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:38.468 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:50:38.469 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:38.469 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:38.469 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:38.469 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:38.469 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:38.470 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:38.470 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:38.470 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:38.470 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:38.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:38.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:38.471 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:50:38.471 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:38.471 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:38.472 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:38.472 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:38.472 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:38.472 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:38.472 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:38.473 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:38.473 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:38.473 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:38.473 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:50:38.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:38.474 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:50:38.474 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:50:38.474 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:50:38.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:38.475 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:50:38Z" 2023-11-02T17:50:38.475 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:50:38.476 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:50:43.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:43.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:43.478 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:50:43.523 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:50:43.523 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:50:43.574 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:43.569+0000 7f160c5b4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:43.569+0000 7f160c5b4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:43.600 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:50:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:50:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:50:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:50:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:50:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:50:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:50:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:50:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:50:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:50:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:50:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:50:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:50:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:50:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:50:43.613 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:50:43.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:43.613 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:50:43Z" 2023-11-02T17:50:43.613 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:50:43.614 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:50:48.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:48.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:48.616 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:50:48.663 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:50:48.663 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:50:48.721 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:48.717+0000 7f3a74264f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:48.723 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:48.717+0000 7f3a74264f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:48.747 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:50:48.748 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:50:48.748 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:50:48.748 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:50:48.748 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:50:48.749 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:50:48.749 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:48.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:48.749 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:50:48.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:48.750 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:50:48.750 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:48.750 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:48.750 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:48.751 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:48.751 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:50:48.751 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:48.751 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:48.751 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:48.752 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:48.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:48.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:48.752 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:50:48.752 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:48.753 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:48.753 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:48.753 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:48.753 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:48.753 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:48.754 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:48.754 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:48.754 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:48.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:48.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:48.755 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:50:48.755 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:48.755 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:48.755 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:48.755 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:48.755 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:48.756 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:48.756 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:48.756 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:48.756 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:48.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:48.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:48.757 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:50:48.757 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:48.757 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:48.757 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:48.757 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:48.758 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:48.758 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:48.758 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:48.758 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:48.758 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:48.758 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:48.759 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:50:48.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:48.759 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:50:48.759 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:50:48.759 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:50:48.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:48.760 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:50:48Z" 2023-11-02T17:50:48.760 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:50:48.761 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:50:53.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:53.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:53.763 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:50:53.810 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:50:53.811 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:50:53.866 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:53.861+0000 7fa627d19f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:53.868 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:53.861+0000 7fa627d19f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:53.893 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:50:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:50:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:50:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:50:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:50:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:50:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:50:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:50:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:50:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:50:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:50:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:50:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:53.904 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:53.904 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:53.904 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:53.904 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:53.904 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:53.905 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:50:53.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:53.905 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:50:53.905 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:50:53.905 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:50:53.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:53.906 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:50:53Z" 2023-11-02T17:50:53.906 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:50:53.907 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:50:58.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:58.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:50:58.909 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:50:58.957 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:50:58.958 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:50:59.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:59.005+0000 7f4e4822df40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:59.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:50:59.009+0000 7f4e4822df40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:50:59.038 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:50:59.038 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:50:59.038 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:50:59.039 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:50:59.039 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:50:59.039 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:50:59.039 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:59.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:59.040 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:50:59.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:59.040 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:50:59.040 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:59.041 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:59.041 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:59.041 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:59.041 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:50:59.041 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:59.042 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:59.042 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:59.042 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:59.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:59.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:59.043 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:50:59.043 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:59.043 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:59.043 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:59.043 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:59.044 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:59.044 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:59.044 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:59.044 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:59.045 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:59.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:59.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:59.045 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:50:59.045 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:59.046 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:59.046 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:59.046 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:59.046 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:59.046 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:59.047 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:59.047 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:59.047 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:59.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:59.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:50:59.048 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:50:59.048 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:50:59.048 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:50:59.048 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:50:59.049 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:50:59.049 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:50:59.049 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:50:59.049 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:50:59.049 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:50:59.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:59.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:50:59.050 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:50:59.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:59.050 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:50:59.051 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:50:59.051 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:50:59.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:50:59.051 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:50:59Z" 2023-11-02T17:50:59.051 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:50:59.052 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:51:04.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:04.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:04.055 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:04.102 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:04.103 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:04.161 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:04.153+0000 7f7410160f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:04.162 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:04.157+0000 7f7410160f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:04.186 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:04Z" 2023-11-02T17:51:04.200 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:04.201 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:51:09.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:09.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:09.203 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:09.253 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:09.253 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:09.301+0000 7f047b30ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:09.305+0000 7f047b30ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:09.336 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:09.337 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:09.337 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:09.337 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:09.337 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:09.337 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:09.338 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:09.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:09.338 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:09.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:09.338 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:09.339 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:09.339 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:09.339 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:09.339 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:09.339 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:09.340 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:09.340 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:09.340 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:09.340 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:09.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:09.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:09.341 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:09.341 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:09.341 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:09.341 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:09.342 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:09.342 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:09.342 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:09.342 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:09.342 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:09.343 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:09.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:09.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:09.343 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:09.343 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:09.344 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:09.344 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:09.344 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:09.344 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:09.344 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:09.345 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:09.345 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:09.345 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:09.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:09.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:09.346 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:09.346 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:09.346 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:09.346 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:09.346 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:09.347 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:09.347 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:09.347 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:09.347 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:09.347 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:09.348 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:09.348 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:09.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:09.348 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:09.348 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:09.349 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:09.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:09.349 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:09Z" 2023-11-02T17:51:09.349 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:09.350 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:51:14.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:14.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:14.352 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:14.399 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:14.400 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:14.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:14.449+0000 7ffa2b782f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:14.457 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:14.453+0000 7ffa2b782f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:14.483 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:14.483 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:14.483 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:14.483 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:14.483 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:14.484 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:14.484 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:14.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:14.484 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:14.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:14.485 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:14.485 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:14.485 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:14.485 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:14.485 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:14.486 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:14.486 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:14.486 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:14.486 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:14.486 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:14.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:14.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:14.487 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:14.487 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:14.487 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:14.487 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:14.488 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:14.488 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:14.488 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:14.488 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:14.488 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:14.488 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:14.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:14.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:14.489 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:14.489 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:14.489 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:14.490 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:14.490 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:14.490 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:14.490 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:14.490 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:14.490 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:14.491 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:14.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:14.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:14.491 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:14.491 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:14.492 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:14.492 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:14.492 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:14.492 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:14.492 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:14.492 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:14.493 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:14.493 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:14.493 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:14.493 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:14.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:14.494 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:14.494 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:14.494 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:14.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:14.494 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:14Z" 2023-11-02T17:51:14.495 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:14.496 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:51:19.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:19.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:19.498 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:19.545 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:19.545 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:19.598 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:19.593+0000 7f822f235f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:19.599 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:19.593+0000 7f822f235f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:19.623 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:19.623 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:19.624 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:19.624 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:19.624 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:19.624 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:19.624 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:19.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:19.625 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:19.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:19.625 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:19.625 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:19.626 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:19.626 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:19.626 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:19.626 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:19.626 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:19.627 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:19.627 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:19.627 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:19.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:19.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:19.628 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:19.628 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:19.628 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:19.628 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:19.628 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:19.629 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:19.629 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:19.629 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:19.629 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:19.629 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:19.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:19.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:19.630 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:19.630 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:19.631 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:19.631 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:19.631 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:19.631 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:19.631 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:19.632 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:19.632 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:19.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:19.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:19.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:19.633 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:19.633 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:19.633 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:19.633 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:19.633 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:19.634 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:19.634 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:19.634 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:19.634 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:19.634 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:19.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:19.635 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:19.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:19.635 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:19.636 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:19.636 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:19.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:19.636 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:19Z" 2023-11-02T17:51:19.636 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:19.637 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:51:24.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:24.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:24.639 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:24.684 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:24.685 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:24.742 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:24.737+0000 7f07c8d81f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:24.744 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:24.737+0000 7f07c8d81f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:24.769 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:24.770 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:24.770 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:24.770 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:24.770 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:24.771 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:24.771 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:24.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:24.771 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:24.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:24.772 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:24.772 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:24.772 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:24.772 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:24.773 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:24.773 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:24.773 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:24.773 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:24.773 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:24.774 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:24.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:24.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:24.774 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:24.775 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:24.775 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:24.775 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:24.775 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:24.775 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:24.776 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:24.776 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:24.776 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:24.776 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:24.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:24.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:24.777 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:24.777 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:24.777 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:24.778 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:24.778 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:24.778 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:24.778 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:24.779 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:24.779 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:24.779 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:24.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:24.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:24.780 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:24.780 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:24.780 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:24.780 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:24.780 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:24.781 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:24.781 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:24.781 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:24.781 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:24.782 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:24.782 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:24.782 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:24.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:24.782 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:24.783 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:24.783 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:24.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:24.783 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:24Z" 2023-11-02T17:51:24.783 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:24.785 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:51:29.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:29.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:29.787 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:29.832 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:29.833 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:29.891 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:29.885+0000 7f177de41f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:29.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:29.885+0000 7f177de41f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:29.918 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:29.918 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:29.919 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:29.919 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:29.919 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:29.919 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:29.919 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:29.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:29.920 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:29.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:29.920 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:29.920 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:29.921 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:29.921 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:29.921 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:29.921 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:29.921 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:29.922 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:29.922 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:29.922 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:29.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:29.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:29.923 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:29.923 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:29.923 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:29.923 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:29.923 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:29.924 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:29.924 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:29.924 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:29.924 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:29.924 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:29.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:29.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:29.925 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:29.925 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:29.925 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:29.926 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:29.926 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:29.926 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:29.926 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:29.926 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:29.927 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:29.927 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:29.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:29.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:29.928 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:29.928 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:29.928 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:29.928 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:29.928 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:29.929 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:29.929 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:29.929 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:29.929 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:29.929 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:29.930 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:29.930 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:29.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:29.930 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:29.930 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:29.931 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:29.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:29.931 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:29Z" 2023-11-02T17:51:29.931 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:29.932 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:51:34.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:34.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:34.934 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:34.982 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:34.983 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:35.042 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:35.037+0000 7f4a25b18f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:35.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:35.037+0000 7f4a25b18f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:35.067 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:35.068 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:35.068 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:35.068 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:35.068 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:35.069 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:35.069 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:35.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:35.069 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:35.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:35.070 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:35.070 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:35.070 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:35.070 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:35.070 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:35.071 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:35.071 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:35.071 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:35.071 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:35.071 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:35.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:35.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:35.072 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:35.072 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:35.072 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:35.073 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:35.073 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:35.073 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:35.073 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:35.073 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:35.074 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:35.074 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:35.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:35.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:35.074 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:35.075 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:35.075 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:35.075 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:35.075 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:35.075 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:35.076 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:35.076 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:35.076 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:35.076 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:35.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:35.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:35.077 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:35.077 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:35.077 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:35.077 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:35.078 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:35.078 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:35.078 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:35.078 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:35.078 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:35.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:35.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:35.079 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:35.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:35.079 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:35.080 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:35.080 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:35.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:35.080 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:35Z" 2023-11-02T17:51:35.080 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:35.081 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:51:40.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:40.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:40.084 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:40.133 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:40.133 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:40.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:40.185+0000 7fcd12f28f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:40.193 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:40.189+0000 7fcd12f28f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:40.219 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:40.219 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:40.220 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:40.220 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:40.220 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:40.220 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:40.220 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:40.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:40.221 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:40.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:40.221 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:40.222 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:40.222 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:40.222 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:40.222 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:40.222 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:40.223 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:40.223 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:40.223 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:40.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:40.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:40.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:40.224 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:40.224 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:40.224 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:40.224 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:40.225 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:40.225 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:40.225 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:40.225 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:40.225 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:40.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:40.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:40.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:40.226 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:40.226 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:40.226 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:40.227 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:40.227 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:40.227 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:40.227 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:40.227 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:40.228 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:40.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:40.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:40.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:40.228 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:40.229 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:40.229 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:40.229 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:40.229 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:40.229 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:40.230 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:40.230 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:40.230 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:40.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:40.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:40.231 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:40.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:40.231 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:40.231 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:40.231 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:40.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:40.232 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:40Z" 2023-11-02T17:51:40.232 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:40.233 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1698947386.719075_41.1 target= 2023-11-02T17:51:45.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:45.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:45.236 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:45.282 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:45.283 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:45.340 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:45.333+0000 7f7229446f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:45.342 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:45.337+0000 7f7229446f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:45.367 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:45.367 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:45.367 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:45.368 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:45.368 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:45.368 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:45.368 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:45.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.369 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:45.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:45.369 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:45.369 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:45.370 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:45.370 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:45.370 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:45.370 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:45.371 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:45.371 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:45.371 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:45.371 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:45.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:45.372 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:45.372 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:45.372 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:45.373 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:45.373 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:45.373 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:45.373 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:45.377 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:45.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:45.377 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:45.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.381 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:45Z" 2023-11-02T17:51:45.381 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:45.382 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T17:51:45.383 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-11-02T17:51:45.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:45.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:45.384 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:45.430 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:45.430 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:45.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:51:45.479+0000 7f0b3a791f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:45.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:51:45.479+0000 7f0b3a791f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:45.509 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:51:45.509 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_status": { 2023-11-02T17:51:45.510 INFO:teuthology.orchestra.run.smithi069.stdout: "info": { 2023-11-02T17:51:45.510 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "sync", 2023-11-02T17:51:45.510 INFO:teuthology.orchestra.run.smithi069.stdout: "num_shards": 4, 2023-11-02T17:51:45.510 INFO:teuthology.orchestra.run.smithi069.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:45.511 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:45.511 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:45.511 INFO:teuthology.orchestra.run.smithi069.stdout: "markers": [ 2023-11-02T17:51:45.511 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:45.511 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 0, 2023-11-02T17:51:45.512 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:45.512 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:45.512 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:45.512 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:45.513 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 1, 2023-11-02T17:51:45.513 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:45.513 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:45.513 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:45.513 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:45.514 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:45.514 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:45.514 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 1, 2023-11-02T17:51:45.514 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:45.514 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:45.515 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "", 2023-11-02T17:51:45.515 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:45.518 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 0, 2023-11-02T17:51:45.518 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:45.518 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "0.000000", 2023-11-02T17:51:45.518 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:45.519 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:45.519 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:45.519 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:45.519 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 2, 2023-11-02T17:51:45.519 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:45.520 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:45.520 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "", 2023-11-02T17:51:45.520 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:45.520 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 0, 2023-11-02T17:51:45.521 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:45.521 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "0.000000", 2023-11-02T17:51:45.521 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:45.521 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:45.521 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:45.522 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:45.522 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 3, 2023-11-02T17:51:45.522 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:45.522 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:45.522 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "", 2023-11-02T17:51:45.523 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:45.523 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 0, 2023-11-02T17:51:45.523 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:45.523 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "0.000000", 2023-11-02T17:51:45.523 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:45.524 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:45.524 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:45.524 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:51:45.524 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:45.524 INFO:teuthology.orchestra.run.smithi069.stdout: "full_sync": { 2023-11-02T17:51:45.525 INFO:teuthology.orchestra.run.smithi069.stdout: "total": 1, 2023-11-02T17:51:45.525 INFO:teuthology.orchestra.run.smithi069.stdout: "complete": 1 2023-11-02T17:51:45.525 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:45.525 INFO:teuthology.orchestra.run.smithi069.stdout: "current_time": "2023-11-02T17:51:45Z" 2023-11-02T17:51:45.525 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:51:45.526 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-11-02T17:51:45.527 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-11-02T17:51:45.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:45.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:45.528 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:45.574 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:45.574 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:45.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:51:45.623+0000 7f9101e6ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:45.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:51:45.623+0000 7f9101e6ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:45.654 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:51:45.655 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_status": { 2023-11-02T17:51:45.655 INFO:teuthology.orchestra.run.smithi069.stdout: "info": { 2023-11-02T17:51:45.655 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "sync", 2023-11-02T17:51:45.655 INFO:teuthology.orchestra.run.smithi069.stdout: "num_shards": 4, 2023-11-02T17:51:45.655 INFO:teuthology.orchestra.run.smithi069.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:45.656 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:45.656 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:45.656 INFO:teuthology.orchestra.run.smithi069.stdout: "markers": [ 2023-11-02T17:51:45.656 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:45.656 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 0, 2023-11-02T17:51:45.657 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:45.657 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:45.657 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:45.657 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:45.658 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 1, 2023-11-02T17:51:45.658 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:45.658 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:45.658 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:45.658 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:45.659 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:45.659 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:45.659 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 1, 2023-11-02T17:51:45.659 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:45.659 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:45.660 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "", 2023-11-02T17:51:45.660 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:45.660 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 0, 2023-11-02T17:51:45.660 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:45.661 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "0.000000", 2023-11-02T17:51:45.661 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:45.661 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:45.661 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:45.661 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:45.661 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 2, 2023-11-02T17:51:45.662 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:45.662 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:45.662 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "", 2023-11-02T17:51:45.662 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:45.662 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 0, 2023-11-02T17:51:45.663 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:45.663 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "0.000000", 2023-11-02T17:51:45.663 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:45.663 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:45.663 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:45.664 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:45.664 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 3, 2023-11-02T17:51:45.664 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:45.664 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:45.664 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "", 2023-11-02T17:51:45.665 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:45.665 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 0, 2023-11-02T17:51:45.665 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:45.665 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "0.000000", 2023-11-02T17:51:45.666 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:45.666 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:45.666 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:45.666 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:51:45.666 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:45.667 INFO:teuthology.orchestra.run.smithi069.stdout: "full_sync": { 2023-11-02T17:51:45.667 INFO:teuthology.orchestra.run.smithi069.stdout: "total": 1, 2023-11-02T17:51:45.667 INFO:teuthology.orchestra.run.smithi069.stdout: "complete": 1 2023-11-02T17:51:45.667 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:45.667 INFO:teuthology.orchestra.run.smithi069.stdout: "current_time": "2023-11-02T17:51:45Z" 2023-11-02T17:51:45.668 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:51:45.669 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-11-02T17:51:45.669 INFO:rgw_multi.tests:meta checkpoint 2023-11-02T17:51:45.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:45.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:45.670 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:45.719 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:45.720 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:45.780 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T17:51:45.804 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T17:51:45.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:45.805 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:45.805 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T17:51:45.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:45.806 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:45.806 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:45.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:45.807 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:45.807 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:45.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:45.807 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:45.808 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:45.808 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:45.808 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T17:51:45.809 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T17:51:45.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:45.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:45.810 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:45.854 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:45.854 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:45.905+0000 7f8ae9560f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:45.915 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:45.909+0000 7f8ae9560f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:45.939 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:45.939 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:45.939 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:45.940 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:45.940 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:45.940 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:45.940 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:45.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.941 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:45.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:45.941 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:45.941 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:45.941 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:45.942 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:45.942 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:45.942 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:45.942 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:45.942 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:45.943 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:45.943 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:45.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:45.943 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:45.943 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:45.944 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:45.944 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:45.944 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:45.944 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:45.944 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:45.945 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:45.945 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:45.945 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:45.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:45.945 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:45.946 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:45.946 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:45.946 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:45.946 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:45.946 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:45.947 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:45.947 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:45.947 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:45.947 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:45.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:45.948 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:45.948 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:45.948 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:45.948 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:45.949 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:45.949 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:45.949 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:45.949 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:45.949 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:45.950 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:45.950 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:45.950 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:45.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.950 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:45.951 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:45.951 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:45.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:45.951 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:45Z" 2023-11-02T17:51:45.951 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:45.952 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T17:51:45.953 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-11-02T17:51:45.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:45.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:45.954 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:45.999 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:45.999 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:46.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:51:46.047+0000 7fc2854fbf40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:46.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:51:46.047+0000 7fc2854fbf40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:46.077 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:51:46.078 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_status": { 2023-11-02T17:51:46.078 INFO:teuthology.orchestra.run.smithi069.stdout: "info": { 2023-11-02T17:51:46.078 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "sync", 2023-11-02T17:51:46.078 INFO:teuthology.orchestra.run.smithi069.stdout: "num_shards": 4, 2023-11-02T17:51:46.079 INFO:teuthology.orchestra.run.smithi069.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:46.079 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:46.079 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:46.079 INFO:teuthology.orchestra.run.smithi069.stdout: "markers": [ 2023-11-02T17:51:46.079 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:46.080 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 0, 2023-11-02T17:51:46.080 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:46.080 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:46.080 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:46.080 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:46.081 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 1, 2023-11-02T17:51:46.081 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:46.081 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:46.081 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:46.081 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:46.081 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:46.082 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:46.082 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 1, 2023-11-02T17:51:46.082 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:46.082 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:46.082 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "", 2023-11-02T17:51:46.083 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:46.083 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 0, 2023-11-02T17:51:46.083 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:46.083 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "0.000000", 2023-11-02T17:51:46.083 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:46.084 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:46.084 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:46.084 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:46.084 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 2, 2023-11-02T17:51:46.084 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:46.085 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:46.085 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "", 2023-11-02T17:51:46.085 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:46.085 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 0, 2023-11-02T17:51:46.085 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:46.086 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "0.000000", 2023-11-02T17:51:46.086 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:46.086 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:46.086 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:46.086 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:46.087 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 3, 2023-11-02T17:51:46.087 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:46.087 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:46.087 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "", 2023-11-02T17:51:46.087 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:46.088 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 0, 2023-11-02T17:51:46.088 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:46.088 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "0.000000", 2023-11-02T17:51:46.088 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:46.089 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:46.089 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:46.089 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:51:46.089 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:46.089 INFO:teuthology.orchestra.run.smithi069.stdout: "full_sync": { 2023-11-02T17:51:46.090 INFO:teuthology.orchestra.run.smithi069.stdout: "total": 1, 2023-11-02T17:51:46.090 INFO:teuthology.orchestra.run.smithi069.stdout: "complete": 1 2023-11-02T17:51:46.090 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:46.090 INFO:teuthology.orchestra.run.smithi069.stdout: "current_time": "2023-11-02T17:51:46Z" 2023-11-02T17:51:46.090 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:51:46.091 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-11-02T17:51:46.092 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-11-02T17:51:46.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:46.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:46.093 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:46.140 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:46.140 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:46.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:51:46.183+0000 7f2355834f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:46.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-02T17:51:46.183+0000 7f2355834f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:46.209 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-02T17:51:46.209 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_status": { 2023-11-02T17:51:46.209 INFO:teuthology.orchestra.run.smithi069.stdout: "info": { 2023-11-02T17:51:46.209 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "sync", 2023-11-02T17:51:46.210 INFO:teuthology.orchestra.run.smithi069.stdout: "num_shards": 4, 2023-11-02T17:51:46.210 INFO:teuthology.orchestra.run.smithi069.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:46.210 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:46.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:46.210 INFO:teuthology.orchestra.run.smithi069.stdout: "markers": [ 2023-11-02T17:51:46.211 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:46.211 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 0, 2023-11-02T17:51:46.211 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:46.211 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:46.211 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:46.212 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:46.212 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 1, 2023-11-02T17:51:46.212 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:46.212 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:46.212 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:46.213 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:46.213 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:46.213 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:46.213 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 1, 2023-11-02T17:51:46.213 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:46.214 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:46.214 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "", 2023-11-02T17:51:46.214 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:46.214 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 0, 2023-11-02T17:51:46.214 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:46.215 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "0.000000", 2023-11-02T17:51:46.215 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:46.215 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:46.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:46.215 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:46.216 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 2, 2023-11-02T17:51:46.216 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:46.216 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:46.216 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "", 2023-11-02T17:51:46.216 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:46.217 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 0, 2023-11-02T17:51:46.217 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:46.217 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "0.000000", 2023-11-02T17:51:46.217 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:46.218 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:46.218 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:46.218 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-02T17:51:46.218 INFO:teuthology.orchestra.run.smithi069.stdout: "key": 3, 2023-11-02T17:51:46.218 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2023-11-02T17:51:46.219 INFO:teuthology.orchestra.run.smithi069.stdout: "state": 1, 2023-11-02T17:51:46.219 INFO:teuthology.orchestra.run.smithi069.stdout: "marker": "", 2023-11-02T17:51:46.219 INFO:teuthology.orchestra.run.smithi069.stdout: "next_step_marker": "", 2023-11-02T17:51:46.219 INFO:teuthology.orchestra.run.smithi069.stdout: "total_entries": 0, 2023-11-02T17:51:46.219 INFO:teuthology.orchestra.run.smithi069.stdout: "pos": 0, 2023-11-02T17:51:46.220 INFO:teuthology.orchestra.run.smithi069.stdout: "timestamp": "0.000000", 2023-11-02T17:51:46.220 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2023-11-02T17:51:46.220 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:46.220 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-02T17:51:46.221 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-02T17:51:46.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:46.221 INFO:teuthology.orchestra.run.smithi069.stdout: "full_sync": { 2023-11-02T17:51:46.221 INFO:teuthology.orchestra.run.smithi069.stdout: "total": 1, 2023-11-02T17:51:46.221 INFO:teuthology.orchestra.run.smithi069.stdout: "complete": 1 2023-11-02T17:51:46.222 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-02T17:51:46.222 INFO:teuthology.orchestra.run.smithi069.stdout: "current_time": "2023-11-02T17:51:46Z" 2023-11-02T17:51:46.222 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-02T17:51:46.223 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-11-02T17:51:46.224 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2023-11-02T17:51:46.243 DEBUG:boto:Using access key provided by client. 2023-11-02T17:51:46.245 DEBUG:boto:Using secret key provided by client. 2023-11-02T17:51:46.246 DEBUG:boto:Using access key provided by client. 2023-11-02T17:51:46.247 DEBUG:boto:Using secret key provided by client. 2023-11-02T17:51:46.248 DEBUG:boto:Using access key provided by client. 2023-11-02T17:51:46.249 DEBUG:boto:Using secret key provided by client. 2023-11-02T17:51:46.250 DEBUG:boto:Using access key provided by client. 2023-11-02T17:51:46.251 DEBUG:boto:Using secret key provided by client. 2023-11-02T17:51:46.252 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-1 2023-11-02T17:51:46.252 DEBUG:boto:path=/ibtljn-1/ 2023-11-02T17:51:46.253 DEBUG:boto:auth_path=/ibtljn-1/ 2023-11-02T17:51:46.253 DEBUG:boto:Method: PUT 2023-11-02T17:51:46.254 DEBUG:boto:Path: /ibtljn-1/ 2023-11-02T17:51:46.254 DEBUG:boto:Data: 2023-11-02T17:51:46.255 DEBUG:boto:Headers: {} 2023-11-02T17:51:46.255 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:46.255 DEBUG:boto:Port: 8000 2023-11-02T17:51:46.256 DEBUG:boto:Params: {} 2023-11-02T17:51:46.256 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T17:51:46.257 DEBUG:boto:Token: None 2023-11-02T17:51:46.258 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:51:46 GMT /ibtljn-1/ 2023-11-02T17:51:46.258 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Dn0K34g23gCKbsJ52m44WFLV9u8= 2023-11-02T17:51:46.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Dn0K34g23gCKbsJ52m44WFLV9u8=', 'Content-Length': '0'} 2023-11-02T17:51:48.528 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d2f18c6594adeeb3-006543e1b2-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:48.529 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-2 2023-11-02T17:51:48.529 DEBUG:boto:path=/ibtljn-2/ 2023-11-02T17:51:48.530 DEBUG:boto:auth_path=/ibtljn-2/ 2023-11-02T17:51:48.530 DEBUG:boto:Method: PUT 2023-11-02T17:51:48.531 DEBUG:boto:Path: /ibtljn-2/ 2023-11-02T17:51:48.531 DEBUG:boto:Data: 2023-11-02T17:51:48.531 DEBUG:boto:Headers: {} 2023-11-02T17:51:48.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:48.532 DEBUG:boto:Port: 8001 2023-11-02T17:51:48.533 DEBUG:boto:Params: {} 2023-11-02T17:51:48.533 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T17:51:48.534 DEBUG:boto:Token: None 2023-11-02T17:51:48.534 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:51:48 GMT /ibtljn-2/ 2023-11-02T17:51:48.535 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Eb27lOyQqZfmIwfO/OIXNcjtV44= 2023-11-02T17:51:48.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Eb27lOyQqZfmIwfO/OIXNcjtV44=', 'Content-Length': '0'} 2023-11-02T17:51:51.558 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001541130bf18efceb-006543e1b4-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:51 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:51.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:51.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:51.560 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:51.596 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:51.596 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:51.650 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T17:51:51.674 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T17:51:51.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:51.674 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:51.675 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T17:51:51.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:51.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:51.675 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:51.676 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:51.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:51.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:51.676 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:51.676 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947508.555290_59.1", 2023-11-02T17:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:51:48.555290Z" 2023-11-02T17:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T17:51:51.679 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947508.555290_59.1'} 2023-11-02T17:51:51.679 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T17:51:51.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:51.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:51.681 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:51.727 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:51.727 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:51.788 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:51.781+0000 7f00225b4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:51.790 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:51.785+0000 7f00225b4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:51.816 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:51.816 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:51.816 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:51.816 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:51.817 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:51.817 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:51.817 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:51.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:51.818 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:51.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:51.818 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:51.818 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:51.819 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:51.819 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:51.819 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:51.819 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:51.819 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:51.820 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:51.820 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:51.820 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:51.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:51.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:51.821 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:51.821 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:51.821 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:51.821 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:51.822 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:51.822 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:51.822 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:51.822 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:51.823 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:51.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:51.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:51.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:51.823 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:51.824 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:51.824 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:51.824 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:51.824 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:51.824 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:51.825 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:51.825 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:51.825 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:51.825 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:51.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:51.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:51.826 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:51.826 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:51.826 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:51.827 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947508.500891_52.1", 2023-11-02T17:51:51.827 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:51.827 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:51.827 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:51.828 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:51:48.500891Z", 2023-11-02T17:51:51.828 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:51.828 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:51.828 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:51.828 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:51.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:51.829 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:51.829 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:51.829 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:51.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:51.830 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:51Z" 2023-11-02T17:51:51.830 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:51.831 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947508.500891_52.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:51:48.500891Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T17:51:51Z" } 2023-11-02T17:51:51.832 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947508.500891_52.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:51:48.500891Z', 'realm_epoch': 2}}] 2023-11-02T17:51:51.832 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947508.555290_59.1'} 2023-11-02T17:51:51.833 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947508.500891_52.1'} 2023-11-02T17:51:51.833 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1698947508.555290_59.1 target=1_1698947508.500891_52.1 2023-11-02T17:51:56.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:56.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:56.835 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:56.881 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:56.881 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:56.929+0000 7f123f11ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:56.929+0000 7f123f11ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:56.962 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947508.555290_59.1", 2023-11-02T17:51:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:56.975 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:56.975 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:56.975 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:51:48.555290Z", 2023-11-02T17:51:56.975 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:56.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:56.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:56.976 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:56.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:56.976 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:56Z" 2023-11-02T17:51:56.978 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:56.979 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947508.555290_59.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:51:48.555290Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T17:51:56Z" } 2023-11-02T17:51:56.979 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947508.555290_59.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:51:48.555290Z', 'realm_epoch': 2}}] 2023-11-02T17:51:56.980 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947508.555290_59.1'} 2023-11-02T17:51:56.980 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947508.555290_59.1'} 2023-11-02T17:51:56.981 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T17:51:56.981 DEBUG:boto:path=/ibtljn-1/ 2023-11-02T17:51:56.982 DEBUG:boto:auth_path=/ibtljn-1/ 2023-11-02T17:51:56.982 DEBUG:boto:Method: HEAD 2023-11-02T17:51:56.983 DEBUG:boto:Path: /ibtljn-1/ 2023-11-02T17:51:56.983 DEBUG:boto:Data: 2023-11-02T17:51:56.984 DEBUG:boto:Headers: {} 2023-11-02T17:51:56.984 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:56.984 DEBUG:boto:Port: 8000 2023-11-02T17:51:56.985 DEBUG:boto:Params: {} 2023-11-02T17:51:56.985 DEBUG:boto:Token: None 2023-11-02T17:51:56.986 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:56 GMT /ibtljn-1/ 2023-11-02T17:51:56.989 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E8HD+TWX/kSmL749BHvF9A3gtP4= 2023-11-02T17:51:56.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E8HD+TWX/kSmL749BHvF9A3gtP4=', 'Content-Length': '0'} 2023-11-02T17:51:56.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001cb1f67f5c94de7-006543e1bc-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:56.995 DEBUG:boto:path=/ibtljn-2/ 2023-11-02T17:51:56.995 DEBUG:boto:auth_path=/ibtljn-2/ 2023-11-02T17:51:56.996 DEBUG:boto:Method: HEAD 2023-11-02T17:51:56.996 DEBUG:boto:Path: /ibtljn-2/ 2023-11-02T17:51:56.997 DEBUG:boto:Data: 2023-11-02T17:51:56.997 DEBUG:boto:Headers: {} 2023-11-02T17:51:56.998 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:56.998 DEBUG:boto:Port: 8000 2023-11-02T17:51:56.998 DEBUG:boto:Params: {} 2023-11-02T17:51:56.999 DEBUG:boto:Token: None 2023-11-02T17:51:56.999 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:56 GMT /ibtljn-2/ 2023-11-02T17:51:57.000 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UjeJ+m9xgXW3ZhxSDmq5e8oAy94= 2023-11-02T17:51:57.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UjeJ+m9xgXW3ZhxSDmq5e8oAy94=', 'Content-Length': '0'} 2023-11-02T17:51:57.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d105fecbfd2171dc-006543e1bc-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.006 DEBUG:boto:path=/ibtljn-1/ 2023-11-02T17:51:57.006 DEBUG:boto:auth_path=/ibtljn-1/ 2023-11-02T17:51:57.006 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.007 DEBUG:boto:Path: /ibtljn-1/ 2023-11-02T17:51:57.007 DEBUG:boto:Data: 2023-11-02T17:51:57.008 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.008 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:57.009 DEBUG:boto:Port: 8001 2023-11-02T17:51:57.009 DEBUG:boto:Params: {} 2023-11-02T17:51:57.010 DEBUG:boto:Token: None 2023-11-02T17:51:57.010 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-1/ 2023-11-02T17:51:57.010 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xwzvWwXU4y26obUW3o5tTfORVWM= 2023-11-02T17:51:57.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xwzvWwXU4y26obUW3o5tTfORVWM=', 'Content-Length': '0'} 2023-11-02T17:51:57.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4aeeef2beda31c4-006543e1bd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.015 DEBUG:boto:path=/ibtljn-2/ 2023-11-02T17:51:57.016 DEBUG:boto:auth_path=/ibtljn-2/ 2023-11-02T17:51:57.016 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.016 DEBUG:boto:Path: /ibtljn-2/ 2023-11-02T17:51:57.017 DEBUG:boto:Data: 2023-11-02T17:51:57.017 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.018 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:57.018 DEBUG:boto:Port: 8001 2023-11-02T17:51:57.019 DEBUG:boto:Params: {} 2023-11-02T17:51:57.019 DEBUG:boto:Token: None 2023-11-02T17:51:57.019 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-2/ 2023-11-02T17:51:57.020 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OdwTbG+LnslZoBFLIIdiCWGijmw= 2023-11-02T17:51:57.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OdwTbG+LnslZoBFLIIdiCWGijmw=', 'Content-Length': '0'} 2023-11-02T17:51:57.024 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091fbcc298c44a1c7-006543e1bd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2023-11-02T17:51:57.026 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-3 2023-11-02T17:51:57.026 DEBUG:boto:path=/ibtljn-3/ 2023-11-02T17:51:57.026 DEBUG:boto:auth_path=/ibtljn-3/ 2023-11-02T17:51:57.027 DEBUG:boto:Method: PUT 2023-11-02T17:51:57.027 DEBUG:boto:Path: /ibtljn-3/ 2023-11-02T17:51:57.028 DEBUG:boto:Data: 2023-11-02T17:51:57.028 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.029 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:57.029 DEBUG:boto:Port: 8000 2023-11-02T17:51:57.029 DEBUG:boto:Params: {} 2023-11-02T17:51:57.030 DEBUG:boto:Token: None 2023-11-02T17:51:57.030 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-3/ 2023-11-02T17:51:57.031 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HBWksgdnHMty6KIbirUM76CG3g8= 2023-11-02T17:51:57.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HBWksgdnHMty6KIbirUM76CG3g8=', 'Content-Length': '0'} 2023-11-02T17:51:57.056 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003171060da0c55e69-006543e1bd-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.056 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-4 2023-11-02T17:51:57.057 DEBUG:boto:path=/ibtljn-4/ 2023-11-02T17:51:57.057 DEBUG:boto:auth_path=/ibtljn-4/ 2023-11-02T17:51:57.058 DEBUG:boto:Method: PUT 2023-11-02T17:51:57.058 DEBUG:boto:Path: /ibtljn-4/ 2023-11-02T17:51:57.058 DEBUG:boto:Data: 2023-11-02T17:51:57.059 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.059 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:57.060 DEBUG:boto:Port: 8001 2023-11-02T17:51:57.060 DEBUG:boto:Params: {} 2023-11-02T17:51:57.060 DEBUG:boto:Token: None 2023-11-02T17:51:57.061 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-4/ 2023-11-02T17:51:57.061 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/XhGeREviaO/y50RRgW+4lifoyA= 2023-11-02T17:51:57.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/XhGeREviaO/y50RRgW+4lifoyA=', 'Content-Length': '0'} 2023-11-02T17:51:57.101 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004c491e0677815639-006543e1bd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:57.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:57.102 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:57.144 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:57.145 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:57.204 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T17:51:57.232 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T17:51:57.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T17:51:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947517.082989_67.1", 2023-11-02T17:51:57.236 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:51:57.082989Z" 2023-11-02T17:51:57.236 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:57.236 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T17:51:57.237 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947517.082989_67.1'} 2023-11-02T17:51:57.237 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T17:51:57.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:57.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:57.239 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:57.285 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:57.285 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:57.339 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:57.333+0000 7f6a6bff5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:57.341 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:57.337+0000 7f6a6bff5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:57.365 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:57.366 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:57.366 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:57.366 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:57.367 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:57.367 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:57.367 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:57.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.367 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:57.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.368 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:57.368 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:57.368 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:57.368 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:57.369 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:57.369 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:57.369 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:57.369 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:57.370 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:57.370 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:57.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.371 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:57.371 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:57.371 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:57.371 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:57.371 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:57.372 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:57.372 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:57.372 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:57.372 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:57.373 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:57.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.373 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:57.373 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:57.374 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:57.374 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:57.374 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:57.374 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:57.375 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:57.375 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:57.375 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:57.375 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:57.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.376 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:57.376 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:57.376 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:57.376 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947517.082989_67.1", 2023-11-02T17:51:57.377 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:57.377 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:57.377 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:57.377 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:51:57.082989Z", 2023-11-02T17:51:57.378 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:57.378 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:57.378 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:57.378 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:57.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.379 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:57.379 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:57.379 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:57.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.379 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:57Z" 2023-11-02T17:51:57.380 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:57.381 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947517.082989_67.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:51:57.082989Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T17:51:57Z" } 2023-11-02T17:51:57.381 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947517.082989_67.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:51:57.082989Z', 'realm_epoch': 2}}] 2023-11-02T17:51:57.382 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947517.082989_67.1'} 2023-11-02T17:51:57.382 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947517.082989_67.1'} 2023-11-02T17:51:57.382 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T17:51:57.383 DEBUG:boto:path=/ibtljn-3/ 2023-11-02T17:51:57.383 DEBUG:boto:auth_path=/ibtljn-3/ 2023-11-02T17:51:57.384 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.384 DEBUG:boto:Path: /ibtljn-3/ 2023-11-02T17:51:57.385 DEBUG:boto:Data: 2023-11-02T17:51:57.385 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.385 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:57.386 DEBUG:boto:Port: 8000 2023-11-02T17:51:57.386 DEBUG:boto:Params: {} 2023-11-02T17:51:57.387 DEBUG:boto:Token: None 2023-11-02T17:51:57.387 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-3/ 2023-11-02T17:51:57.388 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2LasaeNejvYyy2anK18gaHUEC8I= 2023-11-02T17:51:57.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2LasaeNejvYyy2anK18gaHUEC8I=', 'Content-Length': '0'} 2023-11-02T17:51:57.393 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000066a6df9d10abed9-006543e1bd-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.393 DEBUG:boto:path=/ibtljn-4/ 2023-11-02T17:51:57.394 DEBUG:boto:auth_path=/ibtljn-4/ 2023-11-02T17:51:57.394 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.395 DEBUG:boto:Path: /ibtljn-4/ 2023-11-02T17:51:57.395 DEBUG:boto:Data: 2023-11-02T17:51:57.395 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.396 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:57.396 DEBUG:boto:Port: 8000 2023-11-02T17:51:57.397 DEBUG:boto:Params: {} 2023-11-02T17:51:57.397 DEBUG:boto:Token: None 2023-11-02T17:51:57.398 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-4/ 2023-11-02T17:51:57.398 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pwzyuJMrb1zSvgwvbHo2Da/GzxI= 2023-11-02T17:51:57.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pwzyuJMrb1zSvgwvbHo2Da/GzxI=', 'Content-Length': '0'} 2023-11-02T17:51:57.402 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cabde49ec6e491c3-006543e1bd-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.403 DEBUG:boto:path=/ibtljn-3/ 2023-11-02T17:51:57.403 DEBUG:boto:auth_path=/ibtljn-3/ 2023-11-02T17:51:57.404 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.404 DEBUG:boto:Path: /ibtljn-3/ 2023-11-02T17:51:57.405 DEBUG:boto:Data: 2023-11-02T17:51:57.405 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.405 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:57.406 DEBUG:boto:Port: 8001 2023-11-02T17:51:57.406 DEBUG:boto:Params: {} 2023-11-02T17:51:57.407 DEBUG:boto:Token: None 2023-11-02T17:51:57.407 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-3/ 2023-11-02T17:51:57.408 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2LasaeNejvYyy2anK18gaHUEC8I= 2023-11-02T17:51:57.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2LasaeNejvYyy2anK18gaHUEC8I=', 'Content-Length': '0'} 2023-11-02T17:51:57.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000026d176806834033-006543e1bd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.413 DEBUG:boto:path=/ibtljn-4/ 2023-11-02T17:51:57.413 DEBUG:boto:auth_path=/ibtljn-4/ 2023-11-02T17:51:57.414 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.414 DEBUG:boto:Path: /ibtljn-4/ 2023-11-02T17:51:57.414 DEBUG:boto:Data: 2023-11-02T17:51:57.415 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.415 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:57.416 DEBUG:boto:Port: 8001 2023-11-02T17:51:57.416 DEBUG:boto:Params: {} 2023-11-02T17:51:57.417 DEBUG:boto:Token: None 2023-11-02T17:51:57.417 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-4/ 2023-11-02T17:51:57.418 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pwzyuJMrb1zSvgwvbHo2Da/GzxI= 2023-11-02T17:51:57.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pwzyuJMrb1zSvgwvbHo2Da/GzxI=', 'Content-Length': '0'} 2023-11-02T17:51:57.423 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c6df7885d74b55b-006543e1bd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.423 DEBUG:boto:path=/ibtljn-3/ 2023-11-02T17:51:57.423 DEBUG:boto:auth_path=/ibtljn-3/ 2023-11-02T17:51:57.424 DEBUG:boto:Method: PUT 2023-11-02T17:51:57.424 DEBUG:boto:Path: /ibtljn-3/ 2023-11-02T17:51:57.425 DEBUG:boto:Data: 2023-11-02T17:51:57.425 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:57.426 DEBUG:boto:Port: 8000 2023-11-02T17:51:57.426 DEBUG:boto:Params: {} 2023-11-02T17:51:57.427 DEBUG:boto:Token: None 2023-11-02T17:51:57.427 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-3/ 2023-11-02T17:51:57.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HBWksgdnHMty6KIbirUM76CG3g8= 2023-11-02T17:51:57.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HBWksgdnHMty6KIbirUM76CG3g8=', 'Content-Length': '0'} 2023-11-02T17:51:57.434 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f6fcbde9ba41731f-006543e1bd-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.435 DEBUG:boto:path=/ibtljn-4/ 2023-11-02T17:51:57.435 DEBUG:boto:auth_path=/ibtljn-4/ 2023-11-02T17:51:57.435 DEBUG:boto:Method: PUT 2023-11-02T17:51:57.436 DEBUG:boto:Path: /ibtljn-4/ 2023-11-02T17:51:57.436 DEBUG:boto:Data: 2023-11-02T17:51:57.437 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:57.437 DEBUG:boto:Port: 8000 2023-11-02T17:51:57.438 DEBUG:boto:Params: {} 2023-11-02T17:51:57.438 DEBUG:boto:Token: None 2023-11-02T17:51:57.439 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-4/ 2023-11-02T17:51:57.439 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/XhGeREviaO/y50RRgW+4lifoyA= 2023-11-02T17:51:57.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/XhGeREviaO/y50RRgW+4lifoyA=', 'Content-Length': '0'} 2023-11-02T17:51:57.445 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc4f2413ee5a7eb8-006543e1bd-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.446 DEBUG:boto:path=/ibtljn-3/ 2023-11-02T17:51:57.446 DEBUG:boto:auth_path=/ibtljn-3/ 2023-11-02T17:51:57.446 DEBUG:boto:Method: PUT 2023-11-02T17:51:57.447 DEBUG:boto:Path: /ibtljn-3/ 2023-11-02T17:51:57.447 DEBUG:boto:Data: 2023-11-02T17:51:57.448 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.448 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:57.448 DEBUG:boto:Port: 8001 2023-11-02T17:51:57.449 DEBUG:boto:Params: {} 2023-11-02T17:51:57.449 DEBUG:boto:Token: None 2023-11-02T17:51:57.450 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-3/ 2023-11-02T17:51:57.450 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HBWksgdnHMty6KIbirUM76CG3g8= 2023-11-02T17:51:57.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HBWksgdnHMty6KIbirUM76CG3g8=', 'Content-Length': '0'} 2023-11-02T17:51:57.463 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000033cca1df2ffc9fe3-006543e1bd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.463 DEBUG:boto:path=/ibtljn-4/ 2023-11-02T17:51:57.464 DEBUG:boto:auth_path=/ibtljn-4/ 2023-11-02T17:51:57.464 DEBUG:boto:Method: PUT 2023-11-02T17:51:57.465 DEBUG:boto:Path: /ibtljn-4/ 2023-11-02T17:51:57.465 DEBUG:boto:Data: 2023-11-02T17:51:57.465 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.466 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:57.466 DEBUG:boto:Port: 8001 2023-11-02T17:51:57.466 DEBUG:boto:Params: {} 2023-11-02T17:51:57.467 DEBUG:boto:Token: None 2023-11-02T17:51:57.467 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-4/ 2023-11-02T17:51:57.468 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/XhGeREviaO/y50RRgW+4lifoyA= 2023-11-02T17:51:57.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/XhGeREviaO/y50RRgW+4lifoyA=', 'Content-Length': '0'} 2023-11-02T17:51:57.481 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a35c1eac5c559678-006543e1bd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.482 DEBUG:boto:path=/ibtljn-3/ 2023-11-02T17:51:57.482 DEBUG:boto:auth_path=/ibtljn-3/ 2023-11-02T17:51:57.483 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.483 DEBUG:boto:Path: /ibtljn-3/ 2023-11-02T17:51:57.483 DEBUG:boto:Data: 2023-11-02T17:51:57.484 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.484 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:57.485 DEBUG:boto:Port: 8000 2023-11-02T17:51:57.485 DEBUG:boto:Params: {} 2023-11-02T17:51:57.485 DEBUG:boto:Token: None 2023-11-02T17:51:57.486 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-3/ 2023-11-02T17:51:57.486 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2LasaeNejvYyy2anK18gaHUEC8I= 2023-11-02T17:51:57.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2LasaeNejvYyy2anK18gaHUEC8I=', 'Content-Length': '0'} 2023-11-02T17:51:57.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ea56f1963ac7a8b-006543e1bd-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.491 DEBUG:boto:path=/ibtljn-4/ 2023-11-02T17:51:57.492 DEBUG:boto:auth_path=/ibtljn-4/ 2023-11-02T17:51:57.492 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.493 DEBUG:boto:Path: /ibtljn-4/ 2023-11-02T17:51:57.493 DEBUG:boto:Data: 2023-11-02T17:51:57.493 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.494 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:57.494 DEBUG:boto:Port: 8000 2023-11-02T17:51:57.495 DEBUG:boto:Params: {} 2023-11-02T17:51:57.495 DEBUG:boto:Token: None 2023-11-02T17:51:57.495 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-4/ 2023-11-02T17:51:57.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pwzyuJMrb1zSvgwvbHo2Da/GzxI= 2023-11-02T17:51:57.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pwzyuJMrb1zSvgwvbHo2Da/GzxI=', 'Content-Length': '0'} 2023-11-02T17:51:57.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005c4d90065f20190-006543e1bd-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.499 DEBUG:boto:path=/ibtljn-3/ 2023-11-02T17:51:57.500 DEBUG:boto:auth_path=/ibtljn-3/ 2023-11-02T17:51:57.500 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.500 DEBUG:boto:Path: /ibtljn-3/ 2023-11-02T17:51:57.501 DEBUG:boto:Data: 2023-11-02T17:51:57.501 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.502 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:57.502 DEBUG:boto:Port: 8001 2023-11-02T17:51:57.502 DEBUG:boto:Params: {} 2023-11-02T17:51:57.503 DEBUG:boto:Token: None 2023-11-02T17:51:57.503 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-3/ 2023-11-02T17:51:57.504 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2LasaeNejvYyy2anK18gaHUEC8I= 2023-11-02T17:51:57.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2LasaeNejvYyy2anK18gaHUEC8I=', 'Content-Length': '0'} 2023-11-02T17:51:57.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000957adefc95a20f5b-006543e1bd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.508 DEBUG:boto:path=/ibtljn-4/ 2023-11-02T17:51:57.508 DEBUG:boto:auth_path=/ibtljn-4/ 2023-11-02T17:51:57.509 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.509 DEBUG:boto:Path: /ibtljn-4/ 2023-11-02T17:51:57.510 DEBUG:boto:Data: 2023-11-02T17:51:57.510 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:57.511 DEBUG:boto:Port: 8001 2023-11-02T17:51:57.511 DEBUG:boto:Params: {} 2023-11-02T17:51:57.512 DEBUG:boto:Token: None 2023-11-02T17:51:57.512 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-4/ 2023-11-02T17:51:57.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pwzyuJMrb1zSvgwvbHo2Da/GzxI= 2023-11-02T17:51:57.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pwzyuJMrb1zSvgwvbHo2Da/GzxI=', 'Content-Length': '0'} 2023-11-02T17:51:57.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061ce05f13299b73a-006543e1bd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:57.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:57.519 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:57.559 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:57.560 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T17:51:57.644 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T17:51:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T17:51:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.646 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:57.646 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:57.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947517.082989_67.1", 2023-11-02T17:51:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:51:57.082989Z" 2023-11-02T17:51:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:57.648 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T17:51:57.649 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947517.082989_67.1'} 2023-11-02T17:51:57.650 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T17:51:57.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:57.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:57.651 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:57.695 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:57.696 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:57.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:57.745+0000 7fe16a4a1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:57.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:57.745+0000 7fe16a4a1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:57.778 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:57.779 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:57.779 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:57.779 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:57.779 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:57.780 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:57.780 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:57.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.780 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:57.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.781 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:57.781 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:57.781 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:57.781 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:57.782 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:57.782 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:57.782 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:57.782 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:57.783 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:57.783 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:57.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.784 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:57.784 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:57.784 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:57.784 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:57.784 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:57.785 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:57.785 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:57.785 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:57.785 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:57.786 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:57.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.786 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:57.786 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:57.787 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:57.787 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:57.787 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:57.787 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:57.788 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:57.788 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:57.788 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:57.788 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:57.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:57.789 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:57.789 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:57.790 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:57.790 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947517.082989_67.1", 2023-11-02T17:51:57.790 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:57.790 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:57.790 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:57.791 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:51:57.082989Z", 2023-11-02T17:51:57.791 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:57.791 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:57.791 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:57.791 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:57.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.792 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:57.792 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:57.792 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:57.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:57.793 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:57Z" 2023-11-02T17:51:57.793 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:57.794 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947517.082989_67.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:51:57.082989Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T17:51:57Z" } 2023-11-02T17:51:57.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947517.082989_67.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:51:57.082989Z', 'realm_epoch': 2}}] 2023-11-02T17:51:57.795 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947517.082989_67.1'} 2023-11-02T17:51:57.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947517.082989_67.1'} 2023-11-02T17:51:57.796 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T17:51:57.796 DEBUG:boto:path=/ibtljn-3/ 2023-11-02T17:51:57.797 DEBUG:boto:auth_path=/ibtljn-3/ 2023-11-02T17:51:57.797 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.798 DEBUG:boto:Path: /ibtljn-3/ 2023-11-02T17:51:57.798 DEBUG:boto:Data: 2023-11-02T17:51:57.798 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:57.799 DEBUG:boto:Port: 8000 2023-11-02T17:51:57.800 DEBUG:boto:Params: {} 2023-11-02T17:51:57.800 DEBUG:boto:Token: None 2023-11-02T17:51:57.801 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-3/ 2023-11-02T17:51:57.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2LasaeNejvYyy2anK18gaHUEC8I= 2023-11-02T17:51:57.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2LasaeNejvYyy2anK18gaHUEC8I=', 'Content-Length': '0'} 2023-11-02T17:51:57.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e5c242f4d2921cc-006543e1bd-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.807 DEBUG:boto:path=/ibtljn-4/ 2023-11-02T17:51:57.807 DEBUG:boto:auth_path=/ibtljn-4/ 2023-11-02T17:51:57.807 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.808 DEBUG:boto:Path: /ibtljn-4/ 2023-11-02T17:51:57.808 DEBUG:boto:Data: 2023-11-02T17:51:57.809 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.809 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:57.810 DEBUG:boto:Port: 8000 2023-11-02T17:51:57.810 DEBUG:boto:Params: {} 2023-11-02T17:51:57.813 DEBUG:boto:Token: None 2023-11-02T17:51:57.813 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-4/ 2023-11-02T17:51:57.814 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pwzyuJMrb1zSvgwvbHo2Da/GzxI= 2023-11-02T17:51:57.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pwzyuJMrb1zSvgwvbHo2Da/GzxI=', 'Content-Length': '0'} 2023-11-02T17:51:57.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000527df3d44c2f9423-006543e1bd-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.821 DEBUG:boto:path=/ibtljn-3/ 2023-11-02T17:51:57.822 DEBUG:boto:auth_path=/ibtljn-3/ 2023-11-02T17:51:57.822 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.823 DEBUG:boto:Path: /ibtljn-3/ 2023-11-02T17:51:57.823 DEBUG:boto:Data: 2023-11-02T17:51:57.824 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.824 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:57.824 DEBUG:boto:Port: 8001 2023-11-02T17:51:57.825 DEBUG:boto:Params: {} 2023-11-02T17:51:57.825 DEBUG:boto:Token: None 2023-11-02T17:51:57.826 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-3/ 2023-11-02T17:51:57.826 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2LasaeNejvYyy2anK18gaHUEC8I= 2023-11-02T17:51:57.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2LasaeNejvYyy2anK18gaHUEC8I=', 'Content-Length': '0'} 2023-11-02T17:51:57.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2777b37ba9b1108-006543e1bd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.832 DEBUG:boto:path=/ibtljn-4/ 2023-11-02T17:51:57.832 DEBUG:boto:auth_path=/ibtljn-4/ 2023-11-02T17:51:57.832 DEBUG:boto:Method: HEAD 2023-11-02T17:51:57.833 DEBUG:boto:Path: /ibtljn-4/ 2023-11-02T17:51:57.833 DEBUG:boto:Data: 2023-11-02T17:51:57.834 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.834 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:57.835 DEBUG:boto:Port: 8001 2023-11-02T17:51:57.835 DEBUG:boto:Params: {} 2023-11-02T17:51:57.836 DEBUG:boto:Token: None 2023-11-02T17:51:57.836 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-4/ 2023-11-02T17:51:57.837 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pwzyuJMrb1zSvgwvbHo2Da/GzxI= 2023-11-02T17:51:57.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pwzyuJMrb1zSvgwvbHo2Da/GzxI=', 'Content-Length': '0'} 2023-11-02T17:51:57.841 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd2399c563fd52c9-006543e1bd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2023-11-02T17:51:57.843 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-5 2023-11-02T17:51:57.843 DEBUG:boto:path=/ibtljn-5/ 2023-11-02T17:51:57.844 DEBUG:boto:auth_path=/ibtljn-5/ 2023-11-02T17:51:57.844 DEBUG:boto:Method: PUT 2023-11-02T17:51:57.844 DEBUG:boto:Path: /ibtljn-5/ 2023-11-02T17:51:57.845 DEBUG:boto:Data: 2023-11-02T17:51:57.845 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.846 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:57.846 DEBUG:boto:Port: 8000 2023-11-02T17:51:57.847 DEBUG:boto:Params: {} 2023-11-02T17:51:57.847 DEBUG:boto:Token: None 2023-11-02T17:51:57.848 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-5/ 2023-11-02T17:51:57.848 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ybapYfsrAoC03iM84fvZuxt5DRw= 2023-11-02T17:51:57.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ybapYfsrAoC03iM84fvZuxt5DRw=', 'Content-Length': '0'} 2023-11-02T17:51:57.873 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b398287546b297df-006543e1bd-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.874 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-6 2023-11-02T17:51:57.874 DEBUG:boto:path=/ibtljn-6/ 2023-11-02T17:51:57.875 DEBUG:boto:auth_path=/ibtljn-6/ 2023-11-02T17:51:57.875 DEBUG:boto:Method: PUT 2023-11-02T17:51:57.876 DEBUG:boto:Path: /ibtljn-6/ 2023-11-02T17:51:57.876 DEBUG:boto:Data: 2023-11-02T17:51:57.876 DEBUG:boto:Headers: {} 2023-11-02T17:51:57.877 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:57.877 DEBUG:boto:Port: 8001 2023-11-02T17:51:57.878 DEBUG:boto:Params: {} 2023-11-02T17:51:57.878 DEBUG:boto:Token: None 2023-11-02T17:51:57.879 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:51:57 GMT /ibtljn-6/ 2023-11-02T17:51:57.879 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UAAex0NRjStdeLgaJU4XGP9G3MM= 2023-11-02T17:51:57.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UAAex0NRjStdeLgaJU4XGP9G3MM=', 'Content-Length': '0'} 2023-11-02T17:51:57.920 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000380fa9693838b1cc-006543e1bd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:57.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:57.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:57.922 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:57.963 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:57.964 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:58.022 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T17:51:58.055 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T17:51:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T17:51:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947517.901730_75.1", 2023-11-02T17:51:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:51:57.901730Z" 2023-11-02T17:51:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:58.059 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T17:51:58.060 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947517.901730_75.1'} 2023-11-02T17:51:58.061 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T17:51:58.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:58.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:58.062 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:58.103 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:58.103 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:58.158 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:58.153+0000 7fd69b252f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:58.159 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:58.153+0000 7fd69b252f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:58.185 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947517.901730_75.1", 2023-11-02T17:51:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:51:57.901730Z", 2023-11-02T17:51:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:58Z" 2023-11-02T17:51:58.201 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:58.202 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947517.901730_75.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:51:57.901730Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T17:51:58Z" } 2023-11-02T17:51:58.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947517.901730_75.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:51:57.901730Z', 'realm_epoch': 2}}] 2023-11-02T17:51:58.203 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947517.901730_75.1'} 2023-11-02T17:51:58.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947517.901730_75.1'} 2023-11-02T17:51:58.204 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T17:51:58.204 DEBUG:boto:path=/ibtljn-5/ 2023-11-02T17:51:58.205 DEBUG:boto:auth_path=/ibtljn-5/ 2023-11-02T17:51:58.205 DEBUG:boto:Method: HEAD 2023-11-02T17:51:58.206 DEBUG:boto:Path: /ibtljn-5/ 2023-11-02T17:51:58.206 DEBUG:boto:Data: 2023-11-02T17:51:58.207 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.207 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.207 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.208 DEBUG:boto:Params: {} 2023-11-02T17:51:58.208 DEBUG:boto:Token: None 2023-11-02T17:51:58.209 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-5/ 2023-11-02T17:51:58.209 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qh8fiwkNIjI+cTjdbeynl56n24A= 2023-11-02T17:51:58.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qh8fiwkNIjI+cTjdbeynl56n24A=', 'Content-Length': '0'} 2023-11-02T17:51:58.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c3df557de7658a6-006543e1be-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.215 DEBUG:boto:path=/ibtljn-6/ 2023-11-02T17:51:58.215 DEBUG:boto:auth_path=/ibtljn-6/ 2023-11-02T17:51:58.216 DEBUG:boto:Method: HEAD 2023-11-02T17:51:58.216 DEBUG:boto:Path: /ibtljn-6/ 2023-11-02T17:51:58.217 DEBUG:boto:Data: 2023-11-02T17:51:58.217 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.217 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.218 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.218 DEBUG:boto:Params: {} 2023-11-02T17:51:58.219 DEBUG:boto:Token: None 2023-11-02T17:51:58.219 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-6/ 2023-11-02T17:51:58.220 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yzexDVYEMtl5KhKhF++YFiGT854= 2023-11-02T17:51:58.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yzexDVYEMtl5KhKhF++YFiGT854=', 'Content-Length': '0'} 2023-11-02T17:51:58.224 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f007d8f1666d68f-006543e1be-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.225 DEBUG:boto:path=/ibtljn-5/ 2023-11-02T17:51:58.225 DEBUG:boto:auth_path=/ibtljn-5/ 2023-11-02T17:51:58.225 DEBUG:boto:Method: HEAD 2023-11-02T17:51:58.226 DEBUG:boto:Path: /ibtljn-5/ 2023-11-02T17:51:58.226 DEBUG:boto:Data: 2023-11-02T17:51:58.227 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.227 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:58.228 DEBUG:boto:Port: 8001 2023-11-02T17:51:58.228 DEBUG:boto:Params: {} 2023-11-02T17:51:58.228 DEBUG:boto:Token: None 2023-11-02T17:51:58.229 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-5/ 2023-11-02T17:51:58.229 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qh8fiwkNIjI+cTjdbeynl56n24A= 2023-11-02T17:51:58.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qh8fiwkNIjI+cTjdbeynl56n24A=', 'Content-Length': '0'} 2023-11-02T17:51:58.234 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f003a4123e066b87-006543e1be-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.235 DEBUG:boto:path=/ibtljn-6/ 2023-11-02T17:51:58.235 DEBUG:boto:auth_path=/ibtljn-6/ 2023-11-02T17:51:58.236 DEBUG:boto:Method: HEAD 2023-11-02T17:51:58.236 DEBUG:boto:Path: /ibtljn-6/ 2023-11-02T17:51:58.237 DEBUG:boto:Data: 2023-11-02T17:51:58.237 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.238 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:58.238 DEBUG:boto:Port: 8001 2023-11-02T17:51:58.239 DEBUG:boto:Params: {} 2023-11-02T17:51:58.239 DEBUG:boto:Token: None 2023-11-02T17:51:58.240 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-6/ 2023-11-02T17:51:58.240 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yzexDVYEMtl5KhKhF++YFiGT854= 2023-11-02T17:51:58.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yzexDVYEMtl5KhKhF++YFiGT854=', 'Content-Length': '0'} 2023-11-02T17:51:58.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f9ab6632e0eb9dd-006543e1be-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.246 DEBUG:boto:path=/ibtljn-5/ 2023-11-02T17:51:58.246 DEBUG:boto:auth_path=/ibtljn-5/ 2023-11-02T17:51:58.246 DEBUG:boto:Method: DELETE 2023-11-02T17:51:58.247 DEBUG:boto:Path: /ibtljn-5/ 2023-11-02T17:51:58.247 DEBUG:boto:Data: 2023-11-02T17:51:58.248 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.248 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.248 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.249 DEBUG:boto:Params: {} 2023-11-02T17:51:58.249 DEBUG:boto:Token: None 2023-11-02T17:51:58.250 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-5/ 2023-11-02T17:51:58.250 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDpIFcrI8TIxD6nnsrqV0Yhef3I= 2023-11-02T17:51:58.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDpIFcrI8TIxD6nnsrqV0Yhef3I=', 'Content-Length': '0'} 2023-11-02T17:51:58.315 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000a866c8cd467928e-006543e1be-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.316 DEBUG:boto:path=/ibtljn-6/ 2023-11-02T17:51:58.316 DEBUG:boto:auth_path=/ibtljn-6/ 2023-11-02T17:51:58.317 DEBUG:boto:Method: DELETE 2023-11-02T17:51:58.317 DEBUG:boto:Path: /ibtljn-6/ 2023-11-02T17:51:58.318 DEBUG:boto:Data: 2023-11-02T17:51:58.318 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.318 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:58.319 DEBUG:boto:Port: 8001 2023-11-02T17:51:58.319 DEBUG:boto:Params: {} 2023-11-02T17:51:58.320 DEBUG:boto:Token: None 2023-11-02T17:51:58.320 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-6/ 2023-11-02T17:51:58.321 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NG6UHLlGGJd+8EeoqPiD2LnOr0Y= 2023-11-02T17:51:58.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NG6UHLlGGJd+8EeoqPiD2LnOr0Y=', 'Content-Length': '0'} 2023-11-02T17:51:58.458 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ae6bb50888c86e0e-006543e1be-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:58.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:58.460 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:58.503 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:58.503 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:58.551 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T17:51:58.573 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T17:51:58.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.573 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:58.574 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T17:51:58.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.574 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:58.575 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:58.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.575 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:58.576 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:58.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.576 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947518.392549_83.1", 2023-11-02T17:51:58.576 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:51:58.392549Z" 2023-11-02T17:51:58.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:58.577 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T17:51:58.578 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947518.392549_83.1'} 2023-11-02T17:51:58.578 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T17:51:58.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:58.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:58.580 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:58.623 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:58.625 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:58.682 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:58.677+0000 7f747fdc3f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:58.683 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:58.677+0000 7f747fdc3f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:58.710 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:58.710 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:58.710 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:58.711 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:58.711 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:58.711 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:58.711 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:58.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.712 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:58.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.712 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:58.712 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:58.713 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:58.713 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:58.713 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:58.713 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:58.714 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:58.714 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:58.714 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:58.714 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:58.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.715 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:58.715 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:58.716 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:58.716 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:58.716 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:58.716 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:58.716 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:58.717 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:58.717 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:58.717 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:58.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.718 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:58.718 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:58.719 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:58.719 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:58.719 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:58.719 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:58.720 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:58.720 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:58.720 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:58.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:58.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:58.721 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:58.721 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:58.721 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:58.722 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947518.392549_83.1", 2023-11-02T17:51:58.722 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:58.722 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:58.722 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:58.722 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:51:58.392549Z", 2023-11-02T17:51:58.723 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:58.723 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:58.723 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:58.723 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:58.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.724 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:58.724 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:58.724 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:58.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:58.724 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:58Z" 2023-11-02T17:51:58.725 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:58.726 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947518.392549_83.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:51:58.392549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T17:51:58Z" } 2023-11-02T17:51:58.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947518.392549_83.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:51:58.392549Z', 'realm_epoch': 2}}] 2023-11-02T17:51:58.727 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947518.392549_83.1'} 2023-11-02T17:51:58.727 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947518.392549_83.1'} 2023-11-02T17:51:58.728 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T17:51:58.728 DEBUG:boto:path=/ibtljn-5/ 2023-11-02T17:51:58.729 DEBUG:boto:auth_path=/ibtljn-5/ 2023-11-02T17:51:58.729 DEBUG:boto:Method: HEAD 2023-11-02T17:51:58.730 DEBUG:boto:Path: /ibtljn-5/ 2023-11-02T17:51:58.730 DEBUG:boto:Data: 2023-11-02T17:51:58.730 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.731 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.731 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.732 DEBUG:boto:Params: {} 2023-11-02T17:51:58.732 DEBUG:boto:Token: None 2023-11-02T17:51:58.733 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-5/ 2023-11-02T17:51:58.733 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qh8fiwkNIjI+cTjdbeynl56n24A= 2023-11-02T17:51:58.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qh8fiwkNIjI+cTjdbeynl56n24A=', 'Content-Length': '0'} 2023-11-02T17:51:58.736 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000d163277243dc65dd-006543e1be-1111-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.736 DEBUG:boto:path=/ibtljn-6/ 2023-11-02T17:51:58.737 DEBUG:boto:auth_path=/ibtljn-6/ 2023-11-02T17:51:58.737 DEBUG:boto:Method: HEAD 2023-11-02T17:51:58.738 DEBUG:boto:Path: /ibtljn-6/ 2023-11-02T17:51:58.738 DEBUG:boto:Data: 2023-11-02T17:51:58.738 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.739 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.739 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.740 DEBUG:boto:Params: {} 2023-11-02T17:51:58.740 DEBUG:boto:Token: None 2023-11-02T17:51:58.741 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-6/ 2023-11-02T17:51:58.741 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yzexDVYEMtl5KhKhF++YFiGT854= 2023-11-02T17:51:58.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yzexDVYEMtl5KhKhF++YFiGT854=', 'Content-Length': '0'} 2023-11-02T17:51:58.743 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000e88310f0c791b7a6-006543e1be-1111-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.743 DEBUG:boto:path=/ibtljn-5/ 2023-11-02T17:51:58.744 DEBUG:boto:auth_path=/ibtljn-5/ 2023-11-02T17:51:58.744 DEBUG:boto:Method: HEAD 2023-11-02T17:51:58.745 DEBUG:boto:Path: /ibtljn-5/ 2023-11-02T17:51:58.745 DEBUG:boto:Data: 2023-11-02T17:51:58.746 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.746 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:58.746 DEBUG:boto:Port: 8001 2023-11-02T17:51:58.747 DEBUG:boto:Params: {} 2023-11-02T17:51:58.747 DEBUG:boto:Token: None 2023-11-02T17:51:58.748 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-5/ 2023-11-02T17:51:58.748 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qh8fiwkNIjI+cTjdbeynl56n24A= 2023-11-02T17:51:58.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qh8fiwkNIjI+cTjdbeynl56n24A=', 'Content-Length': '0'} 2023-11-02T17:51:58.752 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000f4cbafb97dd0855a-006543e1be-1115-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.752 DEBUG:boto:path=/ibtljn-6/ 2023-11-02T17:51:58.753 DEBUG:boto:auth_path=/ibtljn-6/ 2023-11-02T17:51:58.753 DEBUG:boto:Method: HEAD 2023-11-02T17:51:58.753 DEBUG:boto:Path: /ibtljn-6/ 2023-11-02T17:51:58.754 DEBUG:boto:Data: 2023-11-02T17:51:58.754 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.755 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:58.755 DEBUG:boto:Port: 8001 2023-11-02T17:51:58.756 DEBUG:boto:Params: {} 2023-11-02T17:51:58.756 DEBUG:boto:Token: None 2023-11-02T17:51:58.757 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-6/ 2023-11-02T17:51:58.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yzexDVYEMtl5KhKhF++YFiGT854= 2023-11-02T17:51:58.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yzexDVYEMtl5KhKhF++YFiGT854=', 'Content-Length': '0'} 2023-11-02T17:51:58.759 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000c27ac5f18acb2448-006543e1be-1115-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2023-11-02T17:51:58.761 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-7 2023-11-02T17:51:58.761 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:51:58.762 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:51:58.762 DEBUG:boto:Method: PUT 2023-11-02T17:51:58.762 DEBUG:boto:Path: /ibtljn-7/ 2023-11-02T17:51:58.763 DEBUG:boto:Data: 2023-11-02T17:51:58.763 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.764 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.765 DEBUG:boto:Params: {} 2023-11-02T17:51:58.765 DEBUG:boto:Token: None 2023-11-02T17:51:58.765 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-7/ 2023-11-02T17:51:58.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MndPNd8ICUVmq7DjxQJYlWyYMRM= 2023-11-02T17:51:58.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MndPNd8ICUVmq7DjxQJYlWyYMRM=', 'Content-Length': '0'} 2023-11-02T17:51:58.791 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000151ebed6d68b8676-006543e1be-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.792 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-8 2023-11-02T17:51:58.792 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:51:58.793 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:51:58.793 DEBUG:boto:Method: PUT 2023-11-02T17:51:58.794 DEBUG:boto:Path: /ibtljn-8/ 2023-11-02T17:51:58.794 DEBUG:boto:Data: 2023-11-02T17:51:58.794 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.795 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:58.795 DEBUG:boto:Port: 8001 2023-11-02T17:51:58.796 DEBUG:boto:Params: {} 2023-11-02T17:51:58.796 DEBUG:boto:Token: None 2023-11-02T17:51:58.797 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-8/ 2023-11-02T17:51:58.797 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hRKT79+CyMD/pZvSmqMD4fiU7CY= 2023-11-02T17:51:58.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hRKT79+CyMD/pZvSmqMD4fiU7CY=', 'Content-Length': '0'} 2023-11-02T17:51:58.838 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000064648c9c7906ed4-006543e1be-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.839 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:51:58.839 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:51:58.840 DEBUG:boto:Method: HEAD 2023-11-02T17:51:58.840 DEBUG:boto:Path: /ibtljn-7/ 2023-11-02T17:51:58.841 DEBUG:boto:Data: 2023-11-02T17:51:58.841 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.842 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.842 DEBUG:boto:Params: {} 2023-11-02T17:51:58.843 DEBUG:boto:Token: None 2023-11-02T17:51:58.843 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-7/ 2023-11-02T17:51:58.844 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2bI3eXEqHNcOWQoHfb43RGxpTb8= 2023-11-02T17:51:58.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2bI3eXEqHNcOWQoHfb43RGxpTb8=', 'Content-Length': '0'} 2023-11-02T17:51:58.848 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006852ca11855710cf-006543e1be-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.849 DEBUG:boto:path=/ibtljn-7/myobj 2023-11-02T17:51:58.850 DEBUG:boto:auth_path=/ibtljn-7/myobj 2023-11-02T17:51:58.851 DEBUG:boto:Method: PUT 2023-11-02T17:51:58.851 DEBUG:boto:Path: /ibtljn-7/myobj 2023-11-02T17:51:58.851 DEBUG:boto:Data: 2023-11-02T17:51:58.852 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T17:51:58.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.853 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.853 DEBUG:boto:Params: {} 2023-11-02T17:51:58.854 DEBUG:boto:Token: None 2023-11-02T17:51:58.854 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-7/myobj 2023-11-02T17:51:58.855 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w+BsoXKS6gbx/srzjI2gwoRyfZQ= 2023-11-02T17:51:58.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w+BsoXKS6gbx/srzjI2gwoRyfZQ='} 2023-11-02T17:51:58.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d510ee16c0be02b0-006543e1be-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.871 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:51:58.872 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:51:58.872 DEBUG:boto:Method: HEAD 2023-11-02T17:51:58.873 DEBUG:boto:Path: /ibtljn-7/ 2023-11-02T17:51:58.873 DEBUG:boto:Data: 2023-11-02T17:51:58.873 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.874 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.874 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.875 DEBUG:boto:Params: {} 2023-11-02T17:51:58.875 DEBUG:boto:Token: None 2023-11-02T17:51:58.876 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-7/ 2023-11-02T17:51:58.876 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2bI3eXEqHNcOWQoHfb43RGxpTb8= 2023-11-02T17:51:58.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2bI3eXEqHNcOWQoHfb43RGxpTb8=', 'Content-Length': '0'} 2023-11-02T17:51:58.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b32ca869a52600b-006543e1be-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.880 DEBUG:boto:path=/ibtljn-7/_myobj 2023-11-02T17:51:58.881 DEBUG:boto:auth_path=/ibtljn-7/_myobj 2023-11-02T17:51:58.881 DEBUG:boto:Method: PUT 2023-11-02T17:51:58.882 DEBUG:boto:Path: /ibtljn-7/_myobj 2023-11-02T17:51:58.882 DEBUG:boto:Data: 2023-11-02T17:51:58.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T17:51:58.883 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.884 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.884 DEBUG:boto:Params: {} 2023-11-02T17:51:58.884 DEBUG:boto:Token: None 2023-11-02T17:51:58.885 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-7/_myobj 2023-11-02T17:51:58.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R/DZEgFhb/lRroJlPxn5MuZPQQA= 2023-11-02T17:51:58.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R/DZEgFhb/lRroJlPxn5MuZPQQA='} 2023-11-02T17:51:58.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041a40b96eb5aa6ed-006543e1be-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.931 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:51:58.931 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:51:58.932 DEBUG:boto:Method: HEAD 2023-11-02T17:51:58.932 DEBUG:boto:Path: /ibtljn-7/ 2023-11-02T17:51:58.933 DEBUG:boto:Data: 2023-11-02T17:51:58.933 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.933 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.934 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.934 DEBUG:boto:Params: {} 2023-11-02T17:51:58.935 DEBUG:boto:Token: None 2023-11-02T17:51:58.935 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-7/ 2023-11-02T17:51:58.936 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2bI3eXEqHNcOWQoHfb43RGxpTb8= 2023-11-02T17:51:58.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2bI3eXEqHNcOWQoHfb43RGxpTb8=', 'Content-Length': '0'} 2023-11-02T17:51:58.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee036f508ef8250b-006543e1be-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.941 DEBUG:boto:path=/ibtljn-7/%3A 2023-11-02T17:51:58.941 DEBUG:boto:auth_path=/ibtljn-7/%3A 2023-11-02T17:51:58.942 DEBUG:boto:Method: PUT 2023-11-02T17:51:58.942 DEBUG:boto:Path: /ibtljn-7/%3A 2023-11-02T17:51:58.943 DEBUG:boto:Data: 2023-11-02T17:51:58.943 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T17:51:58.944 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.944 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.945 DEBUG:boto:Params: {} 2023-11-02T17:51:58.945 DEBUG:boto:Token: None 2023-11-02T17:51:58.946 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-7/%3A 2023-11-02T17:51:58.946 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UengCsLz3MZRBBaY8mkQXsmqGI8= 2023-11-02T17:51:58.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UengCsLz3MZRBBaY8mkQXsmqGI8='} 2023-11-02T17:51:58.969 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b4510901970c6d5-006543e1be-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.970 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:51:58.970 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:51:58.971 DEBUG:boto:Method: HEAD 2023-11-02T17:51:58.971 DEBUG:boto:Path: /ibtljn-7/ 2023-11-02T17:51:58.972 DEBUG:boto:Data: 2023-11-02T17:51:58.972 DEBUG:boto:Headers: {} 2023-11-02T17:51:58.973 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.973 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.973 DEBUG:boto:Params: {} 2023-11-02T17:51:58.974 DEBUG:boto:Token: None 2023-11-02T17:51:58.974 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-7/ 2023-11-02T17:51:58.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2bI3eXEqHNcOWQoHfb43RGxpTb8= 2023-11-02T17:51:58.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2bI3eXEqHNcOWQoHfb43RGxpTb8=', 'Content-Length': '0'} 2023-11-02T17:51:58.979 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7475233f9b12837-006543e1be-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:58.980 DEBUG:boto:path=/ibtljn-7/%26 2023-11-02T17:51:58.980 DEBUG:boto:auth_path=/ibtljn-7/%26 2023-11-02T17:51:58.981 DEBUG:boto:Method: PUT 2023-11-02T17:51:58.981 DEBUG:boto:Path: /ibtljn-7/%26 2023-11-02T17:51:58.982 DEBUG:boto:Data: 2023-11-02T17:51:58.982 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T17:51:58.983 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:51:58.983 DEBUG:boto:Port: 8000 2023-11-02T17:51:58.983 DEBUG:boto:Params: {} 2023-11-02T17:51:58.984 DEBUG:boto:Token: None 2023-11-02T17:51:58.984 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 17:51:58 GMT /ibtljn-7/%26 2023-11-02T17:51:58.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:df6RT7uxLmYYVBVUiXdLolip1C4= 2023-11-02T17:51:58.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:51:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:df6RT7uxLmYYVBVUiXdLolip1C4='} 2023-11-02T17:51:59.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef67153df3aea59f-006543e1be-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:59.031 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:51:59.031 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:51:59.032 DEBUG:boto:Method: HEAD 2023-11-02T17:51:59.032 DEBUG:boto:Path: /ibtljn-8/ 2023-11-02T17:51:59.033 DEBUG:boto:Data: 2023-11-02T17:51:59.033 DEBUG:boto:Headers: {} 2023-11-02T17:51:59.033 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:59.034 DEBUG:boto:Port: 8001 2023-11-02T17:51:59.034 DEBUG:boto:Params: {} 2023-11-02T17:51:59.035 DEBUG:boto:Token: None 2023-11-02T17:51:59.035 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:59 GMT /ibtljn-8/ 2023-11-02T17:51:59.037 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fT6CHmsmZ1ZdoUscCkQ+SbQAV+Q= 2023-11-02T17:51:59.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fT6CHmsmZ1ZdoUscCkQ+SbQAV+Q=', 'Content-Length': '0'} 2023-11-02T17:51:59.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3b62e6e2c6f81ea-006543e1bf-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:59.042 DEBUG:boto:path=/ibtljn-8/myobj 2023-11-02T17:51:59.043 DEBUG:boto:auth_path=/ibtljn-8/myobj 2023-11-02T17:51:59.043 DEBUG:boto:Method: PUT 2023-11-02T17:51:59.043 DEBUG:boto:Path: /ibtljn-8/myobj 2023-11-02T17:51:59.044 DEBUG:boto:Data: 2023-11-02T17:51:59.044 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T17:51:59.045 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:59.045 DEBUG:boto:Port: 8001 2023-11-02T17:51:59.045 DEBUG:boto:Params: {} 2023-11-02T17:51:59.046 DEBUG:boto:Token: None 2023-11-02T17:51:59.046 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 17:51:59 GMT /ibtljn-8/myobj 2023-11-02T17:51:59.047 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i+ANRX704TlkmDStjL5z8PeOlbA= 2023-11-02T17:51:59.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:51:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i+ANRX704TlkmDStjL5z8PeOlbA='} 2023-11-02T17:51:59.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3afdc934eb8b72c-006543e1bf-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:59.074 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:51:59.074 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:51:59.075 DEBUG:boto:Method: HEAD 2023-11-02T17:51:59.075 DEBUG:boto:Path: /ibtljn-8/ 2023-11-02T17:51:59.076 DEBUG:boto:Data: 2023-11-02T17:51:59.076 DEBUG:boto:Headers: {} 2023-11-02T17:51:59.077 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:59.077 DEBUG:boto:Port: 8001 2023-11-02T17:51:59.078 DEBUG:boto:Params: {} 2023-11-02T17:51:59.078 DEBUG:boto:Token: None 2023-11-02T17:51:59.079 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:59 GMT /ibtljn-8/ 2023-11-02T17:51:59.079 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fT6CHmsmZ1ZdoUscCkQ+SbQAV+Q= 2023-11-02T17:51:59.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fT6CHmsmZ1ZdoUscCkQ+SbQAV+Q=', 'Content-Length': '0'} 2023-11-02T17:51:59.083 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009be77037ff5583a5-006543e1bf-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:59.084 DEBUG:boto:path=/ibtljn-8/_myobj 2023-11-02T17:51:59.084 DEBUG:boto:auth_path=/ibtljn-8/_myobj 2023-11-02T17:51:59.085 DEBUG:boto:Method: PUT 2023-11-02T17:51:59.085 DEBUG:boto:Path: /ibtljn-8/_myobj 2023-11-02T17:51:59.086 DEBUG:boto:Data: 2023-11-02T17:51:59.086 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T17:51:59.087 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:59.087 DEBUG:boto:Port: 8001 2023-11-02T17:51:59.087 DEBUG:boto:Params: {} 2023-11-02T17:51:59.088 DEBUG:boto:Token: None 2023-11-02T17:51:59.088 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 17:51:59 GMT /ibtljn-8/_myobj 2023-11-02T17:51:59.089 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VRs77HUo9kVSCLWNLu3oT0n5AmE= 2023-11-02T17:51:59.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:51:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VRs77HUo9kVSCLWNLu3oT0n5AmE='} 2023-11-02T17:51:59.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d335f5f2aa7798bb-006543e1bf-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:59.110 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:51:59.110 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:51:59.111 DEBUG:boto:Method: HEAD 2023-11-02T17:51:59.111 DEBUG:boto:Path: /ibtljn-8/ 2023-11-02T17:51:59.111 DEBUG:boto:Data: 2023-11-02T17:51:59.112 DEBUG:boto:Headers: {} 2023-11-02T17:51:59.112 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:59.113 DEBUG:boto:Port: 8001 2023-11-02T17:51:59.113 DEBUG:boto:Params: {} 2023-11-02T17:51:59.114 DEBUG:boto:Token: None 2023-11-02T17:51:59.114 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:59 GMT /ibtljn-8/ 2023-11-02T17:51:59.115 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fT6CHmsmZ1ZdoUscCkQ+SbQAV+Q= 2023-11-02T17:51:59.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fT6CHmsmZ1ZdoUscCkQ+SbQAV+Q=', 'Content-Length': '0'} 2023-11-02T17:51:59.119 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f910ac6587112e8-006543e1bf-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:59.120 DEBUG:boto:path=/ibtljn-8/%3A 2023-11-02T17:51:59.120 DEBUG:boto:auth_path=/ibtljn-8/%3A 2023-11-02T17:51:59.121 DEBUG:boto:Method: PUT 2023-11-02T17:51:59.121 DEBUG:boto:Path: /ibtljn-8/%3A 2023-11-02T17:51:59.121 DEBUG:boto:Data: 2023-11-02T17:51:59.122 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T17:51:59.122 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:59.123 DEBUG:boto:Port: 8001 2023-11-02T17:51:59.123 DEBUG:boto:Params: {} 2023-11-02T17:51:59.124 DEBUG:boto:Token: None 2023-11-02T17:51:59.124 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 17:51:59 GMT /ibtljn-8/%3A 2023-11-02T17:51:59.125 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Cie0cMPx0yaH8K/WYEiUUBLYVtg= 2023-11-02T17:51:59.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:51:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Cie0cMPx0yaH8K/WYEiUUBLYVtg='} 2023-11-02T17:51:59.171 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b528ed4c4f5bd40-006543e1bf-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:59.171 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:51:59.172 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:51:59.172 DEBUG:boto:Method: HEAD 2023-11-02T17:51:59.172 DEBUG:boto:Path: /ibtljn-8/ 2023-11-02T17:51:59.173 DEBUG:boto:Data: 2023-11-02T17:51:59.173 DEBUG:boto:Headers: {} 2023-11-02T17:51:59.174 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:59.174 DEBUG:boto:Port: 8001 2023-11-02T17:51:59.175 DEBUG:boto:Params: {} 2023-11-02T17:51:59.175 DEBUG:boto:Token: None 2023-11-02T17:51:59.176 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:51:59 GMT /ibtljn-8/ 2023-11-02T17:51:59.176 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fT6CHmsmZ1ZdoUscCkQ+SbQAV+Q= 2023-11-02T17:51:59.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:51:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fT6CHmsmZ1ZdoUscCkQ+SbQAV+Q=', 'Content-Length': '0'} 2023-11-02T17:51:59.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e45c30792f8f005-006543e1bf-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:59.181 DEBUG:boto:path=/ibtljn-8/%26 2023-11-02T17:51:59.181 DEBUG:boto:auth_path=/ibtljn-8/%26 2023-11-02T17:51:59.182 DEBUG:boto:Method: PUT 2023-11-02T17:51:59.182 DEBUG:boto:Path: /ibtljn-8/%26 2023-11-02T17:51:59.183 DEBUG:boto:Data: 2023-11-02T17:51:59.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T17:51:59.184 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:51:59.184 DEBUG:boto:Port: 8001 2023-11-02T17:51:59.185 DEBUG:boto:Params: {} 2023-11-02T17:51:59.185 DEBUG:boto:Token: None 2023-11-02T17:51:59.186 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 17:51:59 GMT /ibtljn-8/%26 2023-11-02T17:51:59.186 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qzy3fF9Z7jOxeCDUVI+N7/gKJsk= 2023-11-02T17:51:59.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:51:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qzy3fF9Z7jOxeCDUVI+N7/gKJsk='} 2023-11-02T17:51:59.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000317ee607be654c61-006543e1bf-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:51:59.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:59.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:59.200 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:59.243 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:59.244 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:59.302 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T17:51:59.329 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T17:51:59.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:59.329 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:59.330 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T17:51:59.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:59.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:59.330 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:59.331 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:59.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:59.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:59.331 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:59.332 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:51:59.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:59.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:59.332 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947518.818738_91.1", 2023-11-02T17:51:59.332 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:51:58.818738Z" 2023-11-02T17:51:59.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:59.333 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T17:51:59.334 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947518.818738_91.1'} 2023-11-02T17:51:59.334 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T17:51:59.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:59.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:51:59.336 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:51:59.420 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:59.421 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:59.475 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:59.469+0000 7f45b4986f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:59.476 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:59.469+0000 7f45b4986f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:51:59.499 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:51:59.499 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:51:59.499 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:51:59.500 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:51:59.500 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:51:59.500 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:51:59.500 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:59.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:59.501 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:51:59.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:59.501 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:51:59.501 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:59.502 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:59.502 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:51:59.502 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:59.502 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:51:59.503 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:59.503 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:51:59.503 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:59.503 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:59.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:59.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:59.504 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:51:59.504 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:59.505 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:59.505 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:59.505 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:59.505 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:59.506 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:59.506 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:59.506 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:59.506 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:59.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:59.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:59.507 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:51:59.507 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:59.508 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:59.508 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:51:59.508 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:59.508 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:59.508 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:59.509 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:51:59.509 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:59.509 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:59.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:59.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:51:59.510 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:51:59.510 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:51:59.510 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:51:59.511 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947518.818738_91.1", 2023-11-02T17:51:59.511 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:51:59.511 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:51:59.511 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:51:59.512 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:51:58.818738Z", 2023-11-02T17:51:59.512 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:51:59.512 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:59.512 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:51:59.513 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:51:59.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:59.513 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:51:59.513 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:51:59.514 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:51:59.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:51:59.514 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:51:59Z" 2023-11-02T17:51:59.514 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:51:59.515 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947518.818738_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:51:58.818738Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T17:51:59Z" } 2023-11-02T17:51:59.516 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947518.818738_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:51:58.818738Z', 'realm_epoch': 2}}] 2023-11-02T17:51:59.516 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947518.818738_91.1'} 2023-11-02T17:51:59.517 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '', 3: '1_1698947518.818738_91.1'} 2023-11-02T17:51:59.517 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T17:51:59.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:51:59.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:51:59.519 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:51:59.568 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:51:59.568 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:51:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:51:59.625+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:52:04.634 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:52:04.629+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:52:09.635 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:52:09.629+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:52:14.636 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:52:14.629+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:52:19.637 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:52:19.629+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:52:24.638 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:52:24.633+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:52:29.639 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:52:29.632+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:52:34.640 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:52:34.632+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:52:39.641 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:52:39.632+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:52:44.643 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:52:44.636+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:52:49.644 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:52:49.636+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:52:54.645 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:52:54.636+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:52:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:52:59.640+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:53:04.647 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:04.640+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:53:09.648 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:09.640+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:53:14.649 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:14.640+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:53:19.650 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:19.644+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:53:24.651 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:24.644+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:53:29.652 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:29.644+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:53:34.654 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:34.647+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:53:39.655 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:39.647+0000 7f2aaa61cf40 1 waiting to reach incremental sync.. 2023-11-02T17:53:44.659 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:44.651+0000 7f2aaa61cf40 1 bucket sync caught up with source: 2023-11-02T17:53:44.659 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.3.6, , 00000000001.7.6, , , , 00000000001.5.6, 00000000001.9.6, , ] 2023-11-02T17:53:44.659 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.3.6, , 00000000001.7.6, , , , 00000000001.5.6, 00000000001.9.6, , ] 2023-11-02T17:53:44.659 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:44.651+0000 7f2aaa61cf40 0 bucket checkpoint complete 2023-11-02T17:53:44.687 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-7 zones={a2, a1} 2023-11-02T17:53:44.688 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:44.688 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:44.689 DEBUG:boto:Method: HEAD 2023-11-02T17:53:44.689 DEBUG:boto:Path: /ibtljn-7/ 2023-11-02T17:53:44.690 DEBUG:boto:Data: 2023-11-02T17:53:44.690 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:44.691 DEBUG:boto:Port: 8001 2023-11-02T17:53:44.691 DEBUG:boto:Params: {} 2023-11-02T17:53:44.692 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T17:53:44.693 DEBUG:boto:Token: None 2023-11-02T17:53:44.693 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/ 2023-11-02T17:53:44.694 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A/MjPOP7YuYQD0PG4PeLRN6sX84= 2023-11-02T17:53:44.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A/MjPOP7YuYQD0PG4PeLRN6sX84=', 'Content-Length': '0'} 2023-11-02T17:53:44.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b03fd2959b3c7db-006543e228-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.701 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:44.702 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:44.702 DEBUG:boto:Method: HEAD 2023-11-02T17:53:44.703 DEBUG:boto:Path: /ibtljn-7/ 2023-11-02T17:53:44.703 DEBUG:boto:Data: 2023-11-02T17:53:44.703 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.704 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:44.704 DEBUG:boto:Port: 8000 2023-11-02T17:53:44.705 DEBUG:boto:Params: {} 2023-11-02T17:53:44.705 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T17:53:44.706 DEBUG:boto:Token: None 2023-11-02T17:53:44.706 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/ 2023-11-02T17:53:44.706 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A/MjPOP7YuYQD0PG4PeLRN6sX84= 2023-11-02T17:53:44.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A/MjPOP7YuYQD0PG4PeLRN6sX84=', 'Content-Length': '0'} 2023-11-02T17:53:44.713 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f651f46123c215d5-006543e228-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.713 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T17:53:44.714 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:44.715 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:44.715 DEBUG:boto:path=/ibtljn-7/?versions&max-keys=999 2023-11-02T17:53:44.716 DEBUG:boto:auth_path=/ibtljn-7/?versions&max-keys=999 2023-11-02T17:53:44.716 DEBUG:boto:Method: GET 2023-11-02T17:53:44.717 DEBUG:boto:Path: /ibtljn-7/?versions&max-keys=999 2023-11-02T17:53:44.717 DEBUG:boto:Data: 2023-11-02T17:53:44.717 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.718 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:44.718 DEBUG:boto:Port: 8001 2023-11-02T17:53:44.719 DEBUG:boto:Params: {} 2023-11-02T17:53:44.719 DEBUG:boto:Token: None 2023-11-02T17:53:44.720 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/?versions 2023-11-02T17:53:44.720 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uqgQo+5TgA4Xn1Pr2YcPUqoc09A= 2023-11-02T17:53:44.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uqgQo+5TgA4Xn1Pr2YcPUqoc09A=', 'Content-Length': '0'} 2023-11-02T17:53:44.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066ebe80377ec8d53-006543e228-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.726 DEBUG:boto:b'ibtljn-7999false&nulltrue2023-11-02T17:51:58.989Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-11-02T17:51:58.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-11-02T17:51:58.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-11-02T17:51:58.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:44.728 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-11-02T17:53:44.729 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-11-02T17:53:44.729 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-11-02T17:53:44.729 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-11-02T17:53:44.730 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T17:53:44.730 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:44.731 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:44.731 DEBUG:boto:path=/ibtljn-7/?versions&max-keys=999 2023-11-02T17:53:44.732 DEBUG:boto:auth_path=/ibtljn-7/?versions&max-keys=999 2023-11-02T17:53:44.732 DEBUG:boto:Method: GET 2023-11-02T17:53:44.732 DEBUG:boto:Path: /ibtljn-7/?versions&max-keys=999 2023-11-02T17:53:44.733 DEBUG:boto:Data: 2023-11-02T17:53:44.733 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.734 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:44.734 DEBUG:boto:Port: 8000 2023-11-02T17:53:44.734 DEBUG:boto:Params: {} 2023-11-02T17:53:44.735 DEBUG:boto:Token: None 2023-11-02T17:53:44.735 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/?versions 2023-11-02T17:53:44.736 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uqgQo+5TgA4Xn1Pr2YcPUqoc09A= 2023-11-02T17:53:44.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uqgQo+5TgA4Xn1Pr2YcPUqoc09A=', 'Content-Length': '0'} 2023-11-02T17:53:44.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdaef47edbb27977-006543e228-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.741 DEBUG:boto:b'ibtljn-7999false&nulltrue2023-11-02T17:51:58.989Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-11-02T17:51:58.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-11-02T17:51:58.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-11-02T17:51:58.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:44.742 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-11-02T17:53:44.742 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-11-02T17:53:44.743 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-11-02T17:53:44.743 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-11-02T17:53:44.744 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:44.744 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:44.744 DEBUG:boto:path=/ibtljn-7/?versions&max-keys=999 2023-11-02T17:53:44.745 DEBUG:boto:auth_path=/ibtljn-7/?versions&max-keys=999 2023-11-02T17:53:44.745 DEBUG:boto:Method: GET 2023-11-02T17:53:44.746 DEBUG:boto:Path: /ibtljn-7/?versions&max-keys=999 2023-11-02T17:53:44.746 DEBUG:boto:Data: 2023-11-02T17:53:44.747 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.747 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:44.747 DEBUG:boto:Port: 8001 2023-11-02T17:53:44.748 DEBUG:boto:Params: {} 2023-11-02T17:53:44.748 DEBUG:boto:Token: None 2023-11-02T17:53:44.749 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/?versions 2023-11-02T17:53:44.749 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uqgQo+5TgA4Xn1Pr2YcPUqoc09A= 2023-11-02T17:53:44.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uqgQo+5TgA4Xn1Pr2YcPUqoc09A=', 'Content-Length': '0'} 2023-11-02T17:53:44.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddafd7f01057f0e7-006543e228-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.755 DEBUG:boto:b'ibtljn-7999false&nulltrue2023-11-02T17:51:58.989Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-11-02T17:51:58.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-11-02T17:51:58.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-11-02T17:51:58.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:44.756 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:44.756 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:44.757 DEBUG:boto:path=/ibtljn-7/?versions&max-keys=999 2023-11-02T17:53:44.757 DEBUG:boto:auth_path=/ibtljn-7/?versions&max-keys=999 2023-11-02T17:53:44.757 DEBUG:boto:Method: GET 2023-11-02T17:53:44.758 DEBUG:boto:Path: /ibtljn-7/?versions&max-keys=999 2023-11-02T17:53:44.758 DEBUG:boto:Data: 2023-11-02T17:53:44.759 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.759 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:44.760 DEBUG:boto:Port: 8000 2023-11-02T17:53:44.760 DEBUG:boto:Params: {} 2023-11-02T17:53:44.761 DEBUG:boto:Token: None 2023-11-02T17:53:44.761 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/?versions 2023-11-02T17:53:44.761 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uqgQo+5TgA4Xn1Pr2YcPUqoc09A= 2023-11-02T17:53:44.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uqgQo+5TgA4Xn1Pr2YcPUqoc09A=', 'Content-Length': '0'} 2023-11-02T17:53:44.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000650f6ff21cc0e1db-006543e228-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.769 DEBUG:boto:b'ibtljn-7999false&nulltrue2023-11-02T17:51:58.989Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-11-02T17:51:58.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-11-02T17:51:58.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-11-02T17:51:58.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:44.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-11-02T17:53:44.770 DEBUG:boto:path=/ibtljn-7/%26 2023-11-02T17:53:44.771 DEBUG:boto:auth_path=/ibtljn-7/%26 2023-11-02T17:53:44.771 DEBUG:boto:path=/ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.772 DEBUG:boto:auth_path=/ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.772 DEBUG:boto:Method: GET 2023-11-02T17:53:44.773 DEBUG:boto:Path: /ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.773 DEBUG:boto:Data: 2023-11-02T17:53:44.774 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.774 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:44.775 DEBUG:boto:Port: 8001 2023-11-02T17:53:44.775 DEBUG:boto:Params: {} 2023-11-02T17:53:44.776 DEBUG:boto:Token: None 2023-11-02T17:53:44.777 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.777 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rTcNdYpaPa5XtdmY9lHSvF1C7sE= 2023-11-02T17:53:44.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rTcNdYpaPa5XtdmY9lHSvF1C7sE=', 'Content-Length': '0'} 2023-11-02T17:53:44.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a680d53d0e5e929c-006543e228-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.822 DEBUG:boto:path=/ibtljn-7/%26 2023-11-02T17:53:44.823 DEBUG:boto:auth_path=/ibtljn-7/%26 2023-11-02T17:53:44.823 DEBUG:boto:path=/ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.824 DEBUG:boto:auth_path=/ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.824 DEBUG:boto:Method: GET 2023-11-02T17:53:44.825 DEBUG:boto:Path: /ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.825 DEBUG:boto:Data: 2023-11-02T17:53:44.826 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.826 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:44.827 DEBUG:boto:Port: 8000 2023-11-02T17:53:44.827 DEBUG:boto:Params: {} 2023-11-02T17:53:44.828 DEBUG:boto:Token: None 2023-11-02T17:53:44.828 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.829 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rTcNdYpaPa5XtdmY9lHSvF1C7sE= 2023-11-02T17:53:44.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rTcNdYpaPa5XtdmY9lHSvF1C7sE=', 'Content-Length': '0'} 2023-11-02T17:53:44.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de199be3ce303036-006543e228-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.833 DEBUG:boto:path=/ibtljn-7/%26 2023-11-02T17:53:44.834 DEBUG:boto:auth_path=/ibtljn-7/%26 2023-11-02T17:53:44.834 DEBUG:boto:path=/ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.835 DEBUG:boto:auth_path=/ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.835 DEBUG:boto:Method: HEAD 2023-11-02T17:53:44.836 DEBUG:boto:Path: /ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.836 DEBUG:boto:Data: 2023-11-02T17:53:44.837 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.837 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:44.837 DEBUG:boto:Port: 8001 2023-11-02T17:53:44.838 DEBUG:boto:Params: {} 2023-11-02T17:53:44.838 DEBUG:boto:Token: None 2023-11-02T17:53:44.839 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.839 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vp/IoBClzvmubVS8cMrm0IVkRZY= 2023-11-02T17:53:44.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vp/IoBClzvmubVS8cMrm0IVkRZY=', 'Content-Length': '0'} 2023-11-02T17:53:44.843 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df3d7f05ffca3f35-006543e228-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.844 DEBUG:boto:path=/ibtljn-7/%26 2023-11-02T17:53:44.844 DEBUG:boto:auth_path=/ibtljn-7/%26 2023-11-02T17:53:44.845 DEBUG:boto:path=/ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.845 DEBUG:boto:auth_path=/ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.846 DEBUG:boto:Method: HEAD 2023-11-02T17:53:44.846 DEBUG:boto:Path: /ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.847 DEBUG:boto:Data: 2023-11-02T17:53:44.847 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.848 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:44.848 DEBUG:boto:Port: 8000 2023-11-02T17:53:44.849 DEBUG:boto:Params: {} 2023-11-02T17:53:44.849 DEBUG:boto:Token: None 2023-11-02T17:53:44.850 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.851 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vp/IoBClzvmubVS8cMrm0IVkRZY= 2023-11-02T17:53:44.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vp/IoBClzvmubVS8cMrm0IVkRZY=', 'Content-Length': '0'} 2023-11-02T17:53:44.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000981f769412c8ac0-006543e228-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-11-02T17:53:44.855 DEBUG:boto:path=/ibtljn-7/%26 2023-11-02T17:53:44.856 DEBUG:boto:auth_path=/ibtljn-7/%26 2023-11-02T17:53:44.856 DEBUG:boto:path=/ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.857 DEBUG:boto:auth_path=/ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.857 DEBUG:boto:Method: GET 2023-11-02T17:53:44.858 DEBUG:boto:Path: /ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.858 DEBUG:boto:Data: 2023-11-02T17:53:44.859 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.859 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:44.860 DEBUG:boto:Port: 8001 2023-11-02T17:53:44.860 DEBUG:boto:Params: {} 2023-11-02T17:53:44.861 DEBUG:boto:Token: None 2023-11-02T17:53:44.861 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.862 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rTcNdYpaPa5XtdmY9lHSvF1C7sE= 2023-11-02T17:53:44.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rTcNdYpaPa5XtdmY9lHSvF1C7sE=', 'Content-Length': '0'} 2023-11-02T17:53:44.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a469e50bb6e999c-006543e228-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.866 DEBUG:boto:path=/ibtljn-7/%26 2023-11-02T17:53:44.867 DEBUG:boto:auth_path=/ibtljn-7/%26 2023-11-02T17:53:44.867 DEBUG:boto:path=/ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.868 DEBUG:boto:auth_path=/ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.868 DEBUG:boto:Method: GET 2023-11-02T17:53:44.869 DEBUG:boto:Path: /ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.869 DEBUG:boto:Data: 2023-11-02T17:53:44.870 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:44.871 DEBUG:boto:Port: 8000 2023-11-02T17:53:44.871 DEBUG:boto:Params: {} 2023-11-02T17:53:44.872 DEBUG:boto:Token: None 2023-11-02T17:53:44.872 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/%26?versionId=null 2023-11-02T17:53:44.873 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rTcNdYpaPa5XtdmY9lHSvF1C7sE= 2023-11-02T17:53:44.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rTcNdYpaPa5XtdmY9lHSvF1C7sE=', 'Content-Length': '0'} 2023-11-02T17:53:44.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000460c572a102cfe96-006543e228-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.877 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:44.878 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:44.878 DEBUG:boto:path=/ibtljn-7/?prefix=%26 2023-11-02T17:53:44.879 DEBUG:boto:auth_path=/ibtljn-7/?prefix=%26 2023-11-02T17:53:44.879 DEBUG:boto:Method: GET 2023-11-02T17:53:44.880 DEBUG:boto:Path: /ibtljn-7/?prefix=%26 2023-11-02T17:53:44.880 DEBUG:boto:Data: 2023-11-02T17:53:44.881 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.881 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:44.882 DEBUG:boto:Port: 8001 2023-11-02T17:53:44.882 DEBUG:boto:Params: {} 2023-11-02T17:53:44.883 DEBUG:boto:Token: None 2023-11-02T17:53:44.883 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/ 2023-11-02T17:53:44.884 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3gECPOYCanPj+uA91Wq4MaeLees= 2023-11-02T17:53:44.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3gECPOYCanPj+uA91Wq4MaeLees=', 'Content-Length': '0'} 2023-11-02T17:53:44.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d3ae08b93edf2b2-006543e228-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.890 DEBUG:boto:b'ibtljn-7&1000false&2023-11-02T17:51:58.989Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:44.891 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:44.892 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:44.892 DEBUG:boto:path=/ibtljn-7/?prefix=%26 2023-11-02T17:53:44.893 DEBUG:boto:auth_path=/ibtljn-7/?prefix=%26 2023-11-02T17:53:44.893 DEBUG:boto:Method: GET 2023-11-02T17:53:44.893 DEBUG:boto:Path: /ibtljn-7/?prefix=%26 2023-11-02T17:53:44.894 DEBUG:boto:Data: 2023-11-02T17:53:44.894 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.895 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:44.896 DEBUG:boto:Port: 8000 2023-11-02T17:53:44.896 DEBUG:boto:Params: {} 2023-11-02T17:53:44.897 DEBUG:boto:Token: None 2023-11-02T17:53:44.897 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/ 2023-11-02T17:53:44.898 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3gECPOYCanPj+uA91Wq4MaeLees= 2023-11-02T17:53:44.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3gECPOYCanPj+uA91Wq4MaeLees=', 'Content-Length': '0'} 2023-11-02T17:53:44.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000903b1a726fa09c15-006543e228-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.946 DEBUG:boto:b'ibtljn-7&1000false&2023-11-02T17:51:58.989Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:44.947 DEBUG:boto:path=/ibtljn-7/%26 2023-11-02T17:53:44.947 DEBUG:boto:auth_path=/ibtljn-7/%26 2023-11-02T17:53:44.948 DEBUG:boto:Method: HEAD 2023-11-02T17:53:44.948 DEBUG:boto:Path: /ibtljn-7/%26 2023-11-02T17:53:44.949 DEBUG:boto:Data: 2023-11-02T17:53:44.949 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:44.950 DEBUG:boto:Port: 8001 2023-11-02T17:53:44.951 DEBUG:boto:Params: {} 2023-11-02T17:53:44.951 DEBUG:boto:Token: None 2023-11-02T17:53:44.952 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/%26 2023-11-02T17:53:44.952 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/uh1RAiFOsCxZKm990lby2hC/f0= 2023-11-02T17:53:44.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/uh1RAiFOsCxZKm990lby2hC/f0=', 'Content-Length': '0'} 2023-11-02T17:53:44.956 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f189d8badbc420f2-006543e228-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.957 DEBUG:boto:path=/ibtljn-7/%26 2023-11-02T17:53:44.957 DEBUG:boto:auth_path=/ibtljn-7/%26 2023-11-02T17:53:44.958 DEBUG:boto:Method: HEAD 2023-11-02T17:53:44.958 DEBUG:boto:Path: /ibtljn-7/%26 2023-11-02T17:53:44.959 DEBUG:boto:Data: 2023-11-02T17:53:44.959 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.960 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:44.960 DEBUG:boto:Port: 8000 2023-11-02T17:53:44.960 DEBUG:boto:Params: {} 2023-11-02T17:53:44.961 DEBUG:boto:Token: None 2023-11-02T17:53:44.961 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/%26 2023-11-02T17:53:44.962 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/uh1RAiFOsCxZKm990lby2hC/f0= 2023-11-02T17:53:44.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/uh1RAiFOsCxZKm990lby2hC/f0=', 'Content-Length': '0'} 2023-11-02T17:53:44.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b76dff3c4f9038a-006543e228-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:44.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-11-02T17:53:44.967 DEBUG:boto:path=/ibtljn-7/%26 2023-11-02T17:53:44.967 DEBUG:boto:auth_path=/ibtljn-7/%26 2023-11-02T17:53:44.968 DEBUG:boto:Method: GET 2023-11-02T17:53:44.968 DEBUG:boto:Path: /ibtljn-7/%26 2023-11-02T17:53:44.969 DEBUG:boto:Data: 2023-11-02T17:53:44.969 DEBUG:boto:Headers: {} 2023-11-02T17:53:44.970 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:44.970 DEBUG:boto:Port: 8001 2023-11-02T17:53:44.971 DEBUG:boto:Params: {} 2023-11-02T17:53:44.971 DEBUG:boto:Token: None 2023-11-02T17:53:44.972 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:44 GMT /ibtljn-7/%26 2023-11-02T17:53:44.973 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qrcDKPsIFDGeg5MUg4uJFowetqg= 2023-11-02T17:53:44.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qrcDKPsIFDGeg5MUg4uJFowetqg=', 'Content-Length': '0'} 2023-11-02T17:53:44.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f969c57e38a6a409-006543e228-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.018 DEBUG:boto:path=/ibtljn-7/%26 2023-11-02T17:53:45.019 DEBUG:boto:auth_path=/ibtljn-7/%26 2023-11-02T17:53:45.020 DEBUG:boto:Method: GET 2023-11-02T17:53:45.020 DEBUG:boto:Path: /ibtljn-7/%26 2023-11-02T17:53:45.021 DEBUG:boto:Data: 2023-11-02T17:53:45.021 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.022 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.022 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.023 DEBUG:boto:Params: {} 2023-11-02T17:53:45.023 DEBUG:boto:Token: None 2023-11-02T17:53:45.023 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/%26 2023-11-02T17:53:45.024 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KeNq5oE69oN90ZLAXkNLBwgvxj4= 2023-11-02T17:53:45.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KeNq5oE69oN90ZLAXkNLBwgvxj4=', 'Content-Length': '0'} 2023-11-02T17:53:45.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d3fdc06620daf80-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.029 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.029 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.030 DEBUG:boto:path=/ibtljn-7/?prefix=%26 2023-11-02T17:53:45.030 DEBUG:boto:auth_path=/ibtljn-7/?prefix=%26 2023-11-02T17:53:45.030 DEBUG:boto:Method: GET 2023-11-02T17:53:45.031 DEBUG:boto:Path: /ibtljn-7/?prefix=%26 2023-11-02T17:53:45.031 DEBUG:boto:Data: 2023-11-02T17:53:45.032 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.033 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.033 DEBUG:boto:Params: {} 2023-11-02T17:53:45.034 DEBUG:boto:Token: None 2023-11-02T17:53:45.035 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009404309c6e18861b-006543e229-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.041 DEBUG:boto:b'ibtljn-7&1000false&2023-11-02T17:51:58.989Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.042 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.043 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.043 DEBUG:boto:path=/ibtljn-7/?prefix=%26 2023-11-02T17:53:45.044 DEBUG:boto:auth_path=/ibtljn-7/?prefix=%26 2023-11-02T17:53:45.044 DEBUG:boto:Method: GET 2023-11-02T17:53:45.044 DEBUG:boto:Path: /ibtljn-7/?prefix=%26 2023-11-02T17:53:45.045 DEBUG:boto:Data: 2023-11-02T17:53:45.045 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.046 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.047 DEBUG:boto:Params: {} 2023-11-02T17:53:45.047 DEBUG:boto:Token: None 2023-11-02T17:53:45.048 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.048 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d1652b154c51b4b-006543e229-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.054 DEBUG:boto:b'ibtljn-7&1000false&2023-11-02T17:51:58.989Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-11-02T17:53:45.056 DEBUG:boto:path=/ibtljn-7/%3A 2023-11-02T17:53:45.056 DEBUG:boto:auth_path=/ibtljn-7/%3A 2023-11-02T17:53:45.057 DEBUG:boto:path=/ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.057 DEBUG:boto:auth_path=/ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.058 DEBUG:boto:Method: GET 2023-11-02T17:53:45.058 DEBUG:boto:Path: /ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.059 DEBUG:boto:Data: 2023-11-02T17:53:45.059 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.060 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.060 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.061 DEBUG:boto:Params: {} 2023-11-02T17:53:45.061 DEBUG:boto:Token: None 2023-11-02T17:53:45.062 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.062 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NhlzEIPJOmbklN69GAwbVzAS1ss= 2023-11-02T17:53:45.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NhlzEIPJOmbklN69GAwbVzAS1ss=', 'Content-Length': '0'} 2023-11-02T17:53:45.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d36c4dc3891cee5d-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.067 DEBUG:boto:path=/ibtljn-7/%3A 2023-11-02T17:53:45.067 DEBUG:boto:auth_path=/ibtljn-7/%3A 2023-11-02T17:53:45.068 DEBUG:boto:path=/ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.068 DEBUG:boto:auth_path=/ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.069 DEBUG:boto:Method: GET 2023-11-02T17:53:45.069 DEBUG:boto:Path: /ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.070 DEBUG:boto:Data: 2023-11-02T17:53:45.070 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.071 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.071 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.071 DEBUG:boto:Params: {} 2023-11-02T17:53:45.072 DEBUG:boto:Token: None 2023-11-02T17:53:45.072 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.073 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NhlzEIPJOmbklN69GAwbVzAS1ss= 2023-11-02T17:53:45.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NhlzEIPJOmbklN69GAwbVzAS1ss=', 'Content-Length': '0'} 2023-11-02T17:53:45.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c144db161a097d9-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.118 DEBUG:boto:path=/ibtljn-7/%3A 2023-11-02T17:53:45.119 DEBUG:boto:auth_path=/ibtljn-7/%3A 2023-11-02T17:53:45.119 DEBUG:boto:path=/ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.120 DEBUG:boto:auth_path=/ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.120 DEBUG:boto:Method: HEAD 2023-11-02T17:53:45.121 DEBUG:boto:Path: /ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.121 DEBUG:boto:Data: 2023-11-02T17:53:45.122 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.122 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.123 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.123 DEBUG:boto:Params: {} 2023-11-02T17:53:45.124 DEBUG:boto:Token: None 2023-11-02T17:53:45.124 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.125 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2w1ER+M8bCOlJR6RgoIMSBubMp8= 2023-11-02T17:53:45.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2w1ER+M8bCOlJR6RgoIMSBubMp8=', 'Content-Length': '0'} 2023-11-02T17:53:45.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da9023c86ffd6222-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.129 DEBUG:boto:path=/ibtljn-7/%3A 2023-11-02T17:53:45.130 DEBUG:boto:auth_path=/ibtljn-7/%3A 2023-11-02T17:53:45.130 DEBUG:boto:path=/ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.131 DEBUG:boto:auth_path=/ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.131 DEBUG:boto:Method: HEAD 2023-11-02T17:53:45.132 DEBUG:boto:Path: /ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.132 DEBUG:boto:Data: 2023-11-02T17:53:45.133 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.133 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.134 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.134 DEBUG:boto:Params: {} 2023-11-02T17:53:45.135 DEBUG:boto:Token: None 2023-11-02T17:53:45.135 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.136 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2w1ER+M8bCOlJR6RgoIMSBubMp8= 2023-11-02T17:53:45.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2w1ER+M8bCOlJR6RgoIMSBubMp8=', 'Content-Length': '0'} 2023-11-02T17:53:45.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae807a4bafbabc81-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-11-02T17:53:45.141 DEBUG:boto:path=/ibtljn-7/%3A 2023-11-02T17:53:45.141 DEBUG:boto:auth_path=/ibtljn-7/%3A 2023-11-02T17:53:45.141 DEBUG:boto:path=/ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.142 DEBUG:boto:auth_path=/ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.142 DEBUG:boto:Method: GET 2023-11-02T17:53:45.143 DEBUG:boto:Path: /ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.143 DEBUG:boto:Data: 2023-11-02T17:53:45.144 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.144 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.144 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.145 DEBUG:boto:Params: {} 2023-11-02T17:53:45.145 DEBUG:boto:Token: None 2023-11-02T17:53:45.146 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.147 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NhlzEIPJOmbklN69GAwbVzAS1ss= 2023-11-02T17:53:45.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NhlzEIPJOmbklN69GAwbVzAS1ss=', 'Content-Length': '0'} 2023-11-02T17:53:45.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069f40f6d0472f16e-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.151 DEBUG:boto:path=/ibtljn-7/%3A 2023-11-02T17:53:45.151 DEBUG:boto:auth_path=/ibtljn-7/%3A 2023-11-02T17:53:45.152 DEBUG:boto:path=/ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.152 DEBUG:boto:auth_path=/ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.153 DEBUG:boto:Method: GET 2023-11-02T17:53:45.153 DEBUG:boto:Path: /ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.154 DEBUG:boto:Data: 2023-11-02T17:53:45.154 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.155 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.155 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.156 DEBUG:boto:Params: {} 2023-11-02T17:53:45.156 DEBUG:boto:Token: None 2023-11-02T17:53:45.157 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/%3A?versionId=null 2023-11-02T17:53:45.157 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NhlzEIPJOmbklN69GAwbVzAS1ss= 2023-11-02T17:53:45.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NhlzEIPJOmbklN69GAwbVzAS1ss=', 'Content-Length': '0'} 2023-11-02T17:53:45.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b572f0b1c46770c3-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.162 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.162 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.163 DEBUG:boto:path=/ibtljn-7/?prefix=%3A 2023-11-02T17:53:45.163 DEBUG:boto:auth_path=/ibtljn-7/?prefix=%3A 2023-11-02T17:53:45.164 DEBUG:boto:Method: GET 2023-11-02T17:53:45.164 DEBUG:boto:Path: /ibtljn-7/?prefix=%3A 2023-11-02T17:53:45.165 DEBUG:boto:Data: 2023-11-02T17:53:45.165 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.166 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.166 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.167 DEBUG:boto:Params: {} 2023-11-02T17:53:45.167 DEBUG:boto:Token: None 2023-11-02T17:53:45.168 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.169 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7f43d0007d4620f-006543e229-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.214 DEBUG:boto:b'ibtljn-7:1000false:2023-11-02T17:51:58.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.215 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.216 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.216 DEBUG:boto:path=/ibtljn-7/?prefix=%3A 2023-11-02T17:53:45.217 DEBUG:boto:auth_path=/ibtljn-7/?prefix=%3A 2023-11-02T17:53:45.217 DEBUG:boto:Method: GET 2023-11-02T17:53:45.218 DEBUG:boto:Path: /ibtljn-7/?prefix=%3A 2023-11-02T17:53:45.218 DEBUG:boto:Data: 2023-11-02T17:53:45.219 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.219 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.220 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.220 DEBUG:boto:Params: {} 2023-11-02T17:53:45.221 DEBUG:boto:Token: None 2023-11-02T17:53:45.222 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.222 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfab44502b7a14a0-006543e229-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.229 DEBUG:boto:b'ibtljn-7:1000false:2023-11-02T17:51:58.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.229 DEBUG:boto:path=/ibtljn-7/%3A 2023-11-02T17:53:45.230 DEBUG:boto:auth_path=/ibtljn-7/%3A 2023-11-02T17:53:45.230 DEBUG:boto:Method: HEAD 2023-11-02T17:53:45.231 DEBUG:boto:Path: /ibtljn-7/%3A 2023-11-02T17:53:45.231 DEBUG:boto:Data: 2023-11-02T17:53:45.232 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.232 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.233 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.233 DEBUG:boto:Params: {} 2023-11-02T17:53:45.234 DEBUG:boto:Token: None 2023-11-02T17:53:45.234 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/%3A 2023-11-02T17:53:45.235 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BdZlCGH6QPmY0ocOL8eRHSkqPMI= 2023-11-02T17:53:45.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BdZlCGH6QPmY0ocOL8eRHSkqPMI=', 'Content-Length': '0'} 2023-11-02T17:53:45.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8127e9a2c247229-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.239 DEBUG:boto:path=/ibtljn-7/%3A 2023-11-02T17:53:45.239 DEBUG:boto:auth_path=/ibtljn-7/%3A 2023-11-02T17:53:45.240 DEBUG:boto:Method: HEAD 2023-11-02T17:53:45.240 DEBUG:boto:Path: /ibtljn-7/%3A 2023-11-02T17:53:45.240 DEBUG:boto:Data: 2023-11-02T17:53:45.241 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.241 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.242 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.242 DEBUG:boto:Params: {} 2023-11-02T17:53:45.243 DEBUG:boto:Token: None 2023-11-02T17:53:45.243 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/%3A 2023-11-02T17:53:45.244 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BdZlCGH6QPmY0ocOL8eRHSkqPMI= 2023-11-02T17:53:45.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BdZlCGH6QPmY0ocOL8eRHSkqPMI=', 'Content-Length': '0'} 2023-11-02T17:53:45.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b15b4912faca9b9a-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-11-02T17:53:45.248 DEBUG:boto:path=/ibtljn-7/%3A 2023-11-02T17:53:45.249 DEBUG:boto:auth_path=/ibtljn-7/%3A 2023-11-02T17:53:45.249 DEBUG:boto:Method: GET 2023-11-02T17:53:45.250 DEBUG:boto:Path: /ibtljn-7/%3A 2023-11-02T17:53:45.250 DEBUG:boto:Data: 2023-11-02T17:53:45.250 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.251 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.251 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.252 DEBUG:boto:Params: {} 2023-11-02T17:53:45.252 DEBUG:boto:Token: None 2023-11-02T17:53:45.253 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/%3A 2023-11-02T17:53:45.253 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Yqs6CjFA74Cms8ou8pvk7qVQsd0= 2023-11-02T17:53:45.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Yqs6CjFA74Cms8ou8pvk7qVQsd0=', 'Content-Length': '0'} 2023-11-02T17:53:45.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000564414fbc66c63ed-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.258 DEBUG:boto:path=/ibtljn-7/%3A 2023-11-02T17:53:45.258 DEBUG:boto:auth_path=/ibtljn-7/%3A 2023-11-02T17:53:45.259 DEBUG:boto:Method: GET 2023-11-02T17:53:45.259 DEBUG:boto:Path: /ibtljn-7/%3A 2023-11-02T17:53:45.260 DEBUG:boto:Data: 2023-11-02T17:53:45.260 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.261 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.261 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.262 DEBUG:boto:Params: {} 2023-11-02T17:53:45.262 DEBUG:boto:Token: None 2023-11-02T17:53:45.263 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/%3A 2023-11-02T17:53:45.263 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Yqs6CjFA74Cms8ou8pvk7qVQsd0= 2023-11-02T17:53:45.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Yqs6CjFA74Cms8ou8pvk7qVQsd0=', 'Content-Length': '0'} 2023-11-02T17:53:45.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000baba9b39776217d8-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.310 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.311 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.311 DEBUG:boto:path=/ibtljn-7/?prefix=%3A 2023-11-02T17:53:45.312 DEBUG:boto:auth_path=/ibtljn-7/?prefix=%3A 2023-11-02T17:53:45.312 DEBUG:boto:Method: GET 2023-11-02T17:53:45.313 DEBUG:boto:Path: /ibtljn-7/?prefix=%3A 2023-11-02T17:53:45.313 DEBUG:boto:Data: 2023-11-02T17:53:45.314 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.314 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.314 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.315 DEBUG:boto:Params: {} 2023-11-02T17:53:45.316 DEBUG:boto:Token: None 2023-11-02T17:53:45.316 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.317 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7eba94ac1c4211e-006543e229-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.323 DEBUG:boto:b'ibtljn-7:1000false:2023-11-02T17:51:58.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.323 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.324 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.324 DEBUG:boto:path=/ibtljn-7/?prefix=%3A 2023-11-02T17:53:45.325 DEBUG:boto:auth_path=/ibtljn-7/?prefix=%3A 2023-11-02T17:53:45.325 DEBUG:boto:Method: GET 2023-11-02T17:53:45.326 DEBUG:boto:Path: /ibtljn-7/?prefix=%3A 2023-11-02T17:53:45.326 DEBUG:boto:Data: 2023-11-02T17:53:45.327 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.327 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.328 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.328 DEBUG:boto:Params: {} 2023-11-02T17:53:45.329 DEBUG:boto:Token: None 2023-11-02T17:53:45.329 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.330 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eee1fee8b2d4bef-006543e229-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.336 DEBUG:boto:b'ibtljn-7:1000false:2023-11-02T17:51:58.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-11-02T17:53:45.337 DEBUG:boto:path=/ibtljn-7/_myobj 2023-11-02T17:53:45.338 DEBUG:boto:auth_path=/ibtljn-7/_myobj 2023-11-02T17:53:45.338 DEBUG:boto:path=/ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.339 DEBUG:boto:auth_path=/ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.339 DEBUG:boto:Method: GET 2023-11-02T17:53:45.339 DEBUG:boto:Path: /ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.340 DEBUG:boto:Data: 2023-11-02T17:53:45.340 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.341 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.341 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.342 DEBUG:boto:Params: {} 2023-11-02T17:53:45.342 DEBUG:boto:Token: None 2023-11-02T17:53:45.343 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.343 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iBa8DaWjPJrbTuUlc6Cs7+MBFwo= 2023-11-02T17:53:45.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iBa8DaWjPJrbTuUlc6Cs7+MBFwo=', 'Content-Length': '0'} 2023-11-02T17:53:45.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000981fc5c5bb65e4d1-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.390 DEBUG:boto:path=/ibtljn-7/_myobj 2023-11-02T17:53:45.391 DEBUG:boto:auth_path=/ibtljn-7/_myobj 2023-11-02T17:53:45.391 DEBUG:boto:path=/ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.392 DEBUG:boto:auth_path=/ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.392 DEBUG:boto:Method: GET 2023-11-02T17:53:45.393 DEBUG:boto:Path: /ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.393 DEBUG:boto:Data: 2023-11-02T17:53:45.393 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.394 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.394 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.395 DEBUG:boto:Params: {} 2023-11-02T17:53:45.395 DEBUG:boto:Token: None 2023-11-02T17:53:45.396 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.397 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iBa8DaWjPJrbTuUlc6Cs7+MBFwo= 2023-11-02T17:53:45.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iBa8DaWjPJrbTuUlc6Cs7+MBFwo=', 'Content-Length': '0'} 2023-11-02T17:53:45.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007dbc3125258e2e8e-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.401 DEBUG:boto:path=/ibtljn-7/_myobj 2023-11-02T17:53:45.401 DEBUG:boto:auth_path=/ibtljn-7/_myobj 2023-11-02T17:53:45.402 DEBUG:boto:path=/ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.402 DEBUG:boto:auth_path=/ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.403 DEBUG:boto:Method: HEAD 2023-11-02T17:53:45.403 DEBUG:boto:Path: /ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.404 DEBUG:boto:Data: 2023-11-02T17:53:45.404 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.404 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.405 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.405 DEBUG:boto:Params: {} 2023-11-02T17:53:45.406 DEBUG:boto:Token: None 2023-11-02T17:53:45.406 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.407 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z2UbAShz9qoL5aJL7QOam8zZfjg= 2023-11-02T17:53:45.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z2UbAShz9qoL5aJL7QOam8zZfjg=', 'Content-Length': '0'} 2023-11-02T17:53:45.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f670b384e889049-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.411 DEBUG:boto:path=/ibtljn-7/_myobj 2023-11-02T17:53:45.411 DEBUG:boto:auth_path=/ibtljn-7/_myobj 2023-11-02T17:53:45.412 DEBUG:boto:path=/ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.412 DEBUG:boto:auth_path=/ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.412 DEBUG:boto:Method: HEAD 2023-11-02T17:53:45.413 DEBUG:boto:Path: /ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.413 DEBUG:boto:Data: 2023-11-02T17:53:45.414 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.414 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.414 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.415 DEBUG:boto:Params: {} 2023-11-02T17:53:45.415 DEBUG:boto:Token: None 2023-11-02T17:53:45.416 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.416 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z2UbAShz9qoL5aJL7QOam8zZfjg= 2023-11-02T17:53:45.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z2UbAShz9qoL5aJL7QOam8zZfjg=', 'Content-Length': '0'} 2023-11-02T17:53:45.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da0945b2455bed3c-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-11-02T17:53:45.421 DEBUG:boto:path=/ibtljn-7/_myobj 2023-11-02T17:53:45.421 DEBUG:boto:auth_path=/ibtljn-7/_myobj 2023-11-02T17:53:45.421 DEBUG:boto:path=/ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.422 DEBUG:boto:auth_path=/ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.422 DEBUG:boto:Method: GET 2023-11-02T17:53:45.423 DEBUG:boto:Path: /ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.423 DEBUG:boto:Data: 2023-11-02T17:53:45.424 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.424 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.425 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.425 DEBUG:boto:Params: {} 2023-11-02T17:53:45.426 DEBUG:boto:Token: None 2023-11-02T17:53:45.426 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.427 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iBa8DaWjPJrbTuUlc6Cs7+MBFwo= 2023-11-02T17:53:45.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iBa8DaWjPJrbTuUlc6Cs7+MBFwo=', 'Content-Length': '0'} 2023-11-02T17:53:45.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b325cfd0a24888b7-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.431 DEBUG:boto:path=/ibtljn-7/_myobj 2023-11-02T17:53:45.431 DEBUG:boto:auth_path=/ibtljn-7/_myobj 2023-11-02T17:53:45.432 DEBUG:boto:path=/ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.432 DEBUG:boto:auth_path=/ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.433 DEBUG:boto:Method: GET 2023-11-02T17:53:45.433 DEBUG:boto:Path: /ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.434 DEBUG:boto:Data: 2023-11-02T17:53:45.434 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.435 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.436 DEBUG:boto:Params: {} 2023-11-02T17:53:45.436 DEBUG:boto:Token: None 2023-11-02T17:53:45.437 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/_myobj?versionId=null 2023-11-02T17:53:45.437 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iBa8DaWjPJrbTuUlc6Cs7+MBFwo= 2023-11-02T17:53:45.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iBa8DaWjPJrbTuUlc6Cs7+MBFwo=', 'Content-Length': '0'} 2023-11-02T17:53:45.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c0c5aa2edf3fa89-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.482 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.483 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.483 DEBUG:boto:path=/ibtljn-7/?prefix=_myobj 2023-11-02T17:53:45.484 DEBUG:boto:auth_path=/ibtljn-7/?prefix=_myobj 2023-11-02T17:53:45.484 DEBUG:boto:Method: GET 2023-11-02T17:53:45.485 DEBUG:boto:Path: /ibtljn-7/?prefix=_myobj 2023-11-02T17:53:45.485 DEBUG:boto:Data: 2023-11-02T17:53:45.485 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.486 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.486 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.487 DEBUG:boto:Params: {} 2023-11-02T17:53:45.487 DEBUG:boto:Token: None 2023-11-02T17:53:45.488 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.488 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e6ee99ded9cb8b8-006543e229-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.494 DEBUG:boto:b'ibtljn-7_myobj1000false_myobj2023-11-02T17:51:58.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.495 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.495 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.496 DEBUG:boto:path=/ibtljn-7/?prefix=_myobj 2023-11-02T17:53:45.496 DEBUG:boto:auth_path=/ibtljn-7/?prefix=_myobj 2023-11-02T17:53:45.497 DEBUG:boto:Method: GET 2023-11-02T17:53:45.497 DEBUG:boto:Path: /ibtljn-7/?prefix=_myobj 2023-11-02T17:53:45.498 DEBUG:boto:Data: 2023-11-02T17:53:45.498 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.499 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.499 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.500 DEBUG:boto:Params: {} 2023-11-02T17:53:45.500 DEBUG:boto:Token: None 2023-11-02T17:53:45.501 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.501 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053ed10bc12186706-006543e229-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.507 DEBUG:boto:b'ibtljn-7_myobj1000false_myobj2023-11-02T17:51:58.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.508 DEBUG:boto:path=/ibtljn-7/_myobj 2023-11-02T17:53:45.508 DEBUG:boto:auth_path=/ibtljn-7/_myobj 2023-11-02T17:53:45.509 DEBUG:boto:Method: HEAD 2023-11-02T17:53:45.509 DEBUG:boto:Path: /ibtljn-7/_myobj 2023-11-02T17:53:45.510 DEBUG:boto:Data: 2023-11-02T17:53:45.510 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.511 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.511 DEBUG:boto:Params: {} 2023-11-02T17:53:45.512 DEBUG:boto:Token: None 2023-11-02T17:53:45.512 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/_myobj 2023-11-02T17:53:45.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GGve3VJSw5fUwjWSVfYyK5ZJe70= 2023-11-02T17:53:45.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GGve3VJSw5fUwjWSVfYyK5ZJe70=', 'Content-Length': '0'} 2023-11-02T17:53:45.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b76b586cf3602d9-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.517 DEBUG:boto:path=/ibtljn-7/_myobj 2023-11-02T17:53:45.517 DEBUG:boto:auth_path=/ibtljn-7/_myobj 2023-11-02T17:53:45.518 DEBUG:boto:Method: HEAD 2023-11-02T17:53:45.518 DEBUG:boto:Path: /ibtljn-7/_myobj 2023-11-02T17:53:45.519 DEBUG:boto:Data: 2023-11-02T17:53:45.519 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.520 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.520 DEBUG:boto:Params: {} 2023-11-02T17:53:45.521 DEBUG:boto:Token: None 2023-11-02T17:53:45.521 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/_myobj 2023-11-02T17:53:45.522 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GGve3VJSw5fUwjWSVfYyK5ZJe70= 2023-11-02T17:53:45.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GGve3VJSw5fUwjWSVfYyK5ZJe70=', 'Content-Length': '0'} 2023-11-02T17:53:45.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006178fae4955d1fc6-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-11-02T17:53:45.526 DEBUG:boto:path=/ibtljn-7/_myobj 2023-11-02T17:53:45.527 DEBUG:boto:auth_path=/ibtljn-7/_myobj 2023-11-02T17:53:45.527 DEBUG:boto:Method: GET 2023-11-02T17:53:45.527 DEBUG:boto:Path: /ibtljn-7/_myobj 2023-11-02T17:53:45.528 DEBUG:boto:Data: 2023-11-02T17:53:45.528 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.529 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.529 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.530 DEBUG:boto:Params: {} 2023-11-02T17:53:45.530 DEBUG:boto:Token: None 2023-11-02T17:53:45.531 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/_myobj 2023-11-02T17:53:45.531 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uYpyG2XUhBBfWvpTtME8dWcAA4I= 2023-11-02T17:53:45.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uYpyG2XUhBBfWvpTtME8dWcAA4I=', 'Content-Length': '0'} 2023-11-02T17:53:45.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000012496a48da9fbdf-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.578 DEBUG:boto:path=/ibtljn-7/_myobj 2023-11-02T17:53:45.579 DEBUG:boto:auth_path=/ibtljn-7/_myobj 2023-11-02T17:53:45.579 DEBUG:boto:Method: GET 2023-11-02T17:53:45.580 DEBUG:boto:Path: /ibtljn-7/_myobj 2023-11-02T17:53:45.580 DEBUG:boto:Data: 2023-11-02T17:53:45.581 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.581 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.581 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.582 DEBUG:boto:Params: {} 2023-11-02T17:53:45.583 DEBUG:boto:Token: None 2023-11-02T17:53:45.583 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/_myobj 2023-11-02T17:53:45.583 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uYpyG2XUhBBfWvpTtME8dWcAA4I= 2023-11-02T17:53:45.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uYpyG2XUhBBfWvpTtME8dWcAA4I=', 'Content-Length': '0'} 2023-11-02T17:53:45.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7016b908077c85b-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.588 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.588 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.588 DEBUG:boto:path=/ibtljn-7/?prefix=_myobj 2023-11-02T17:53:45.589 DEBUG:boto:auth_path=/ibtljn-7/?prefix=_myobj 2023-11-02T17:53:45.589 DEBUG:boto:Method: GET 2023-11-02T17:53:45.590 DEBUG:boto:Path: /ibtljn-7/?prefix=_myobj 2023-11-02T17:53:45.590 DEBUG:boto:Data: 2023-11-02T17:53:45.591 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.591 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.592 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.592 DEBUG:boto:Params: {} 2023-11-02T17:53:45.593 DEBUG:boto:Token: None 2023-11-02T17:53:45.593 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.594 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000113af4bca647874f-006543e229-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.600 DEBUG:boto:b'ibtljn-7_myobj1000false_myobj2023-11-02T17:51:58.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.601 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.601 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.602 DEBUG:boto:path=/ibtljn-7/?prefix=_myobj 2023-11-02T17:53:45.602 DEBUG:boto:auth_path=/ibtljn-7/?prefix=_myobj 2023-11-02T17:53:45.603 DEBUG:boto:Method: GET 2023-11-02T17:53:45.603 DEBUG:boto:Path: /ibtljn-7/?prefix=_myobj 2023-11-02T17:53:45.604 DEBUG:boto:Data: 2023-11-02T17:53:45.604 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.605 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.605 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.606 DEBUG:boto:Params: {} 2023-11-02T17:53:45.606 DEBUG:boto:Token: None 2023-11-02T17:53:45.607 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.607 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000219658b6a4dbae00-006543e229-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.654 DEBUG:boto:b'ibtljn-7_myobj1000false_myobj2023-11-02T17:51:58.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-11-02T17:53:45.656 DEBUG:boto:path=/ibtljn-7/myobj 2023-11-02T17:53:45.656 DEBUG:boto:auth_path=/ibtljn-7/myobj 2023-11-02T17:53:45.656 DEBUG:boto:path=/ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.657 DEBUG:boto:auth_path=/ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.657 DEBUG:boto:Method: GET 2023-11-02T17:53:45.658 DEBUG:boto:Path: /ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.658 DEBUG:boto:Data: 2023-11-02T17:53:45.659 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.659 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.660 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.660 DEBUG:boto:Params: {} 2023-11-02T17:53:45.661 DEBUG:boto:Token: None 2023-11-02T17:53:45.661 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.662 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a8oDGULEfYuclETU8hvdOg465Mw= 2023-11-02T17:53:45.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a8oDGULEfYuclETU8hvdOg465Mw=', 'Content-Length': '0'} 2023-11-02T17:53:45.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059e7f2edae5d659d-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.666 DEBUG:boto:path=/ibtljn-7/myobj 2023-11-02T17:53:45.667 DEBUG:boto:auth_path=/ibtljn-7/myobj 2023-11-02T17:53:45.667 DEBUG:boto:path=/ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.668 DEBUG:boto:auth_path=/ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.668 DEBUG:boto:Method: GET 2023-11-02T17:53:45.669 DEBUG:boto:Path: /ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.669 DEBUG:boto:Data: 2023-11-02T17:53:45.669 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.670 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.670 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.671 DEBUG:boto:Params: {} 2023-11-02T17:53:45.671 DEBUG:boto:Token: None 2023-11-02T17:53:45.672 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.672 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a8oDGULEfYuclETU8hvdOg465Mw= 2023-11-02T17:53:45.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a8oDGULEfYuclETU8hvdOg465Mw=', 'Content-Length': '0'} 2023-11-02T17:53:45.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8cadd4339e04ac0-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.676 DEBUG:boto:path=/ibtljn-7/myobj 2023-11-02T17:53:45.677 DEBUG:boto:auth_path=/ibtljn-7/myobj 2023-11-02T17:53:45.677 DEBUG:boto:path=/ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.678 DEBUG:boto:auth_path=/ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.678 DEBUG:boto:Method: HEAD 2023-11-02T17:53:45.679 DEBUG:boto:Path: /ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.679 DEBUG:boto:Data: 2023-11-02T17:53:45.679 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.680 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.681 DEBUG:boto:Params: {} 2023-11-02T17:53:45.681 DEBUG:boto:Token: None 2023-11-02T17:53:45.682 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.682 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sEjk1PKLo0ms4C7wNyZ6qrLQObs= 2023-11-02T17:53:45.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sEjk1PKLo0ms4C7wNyZ6qrLQObs=', 'Content-Length': '0'} 2023-11-02T17:53:45.686 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d5c1931fd853466-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.686 DEBUG:boto:path=/ibtljn-7/myobj 2023-11-02T17:53:45.687 DEBUG:boto:auth_path=/ibtljn-7/myobj 2023-11-02T17:53:45.687 DEBUG:boto:path=/ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.687 DEBUG:boto:auth_path=/ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.688 DEBUG:boto:Method: HEAD 2023-11-02T17:53:45.688 DEBUG:boto:Path: /ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.689 DEBUG:boto:Data: 2023-11-02T17:53:45.689 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.690 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.690 DEBUG:boto:Params: {} 2023-11-02T17:53:45.691 DEBUG:boto:Token: None 2023-11-02T17:53:45.691 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.692 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sEjk1PKLo0ms4C7wNyZ6qrLQObs= 2023-11-02T17:53:45.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sEjk1PKLo0ms4C7wNyZ6qrLQObs=', 'Content-Length': '0'} 2023-11-02T17:53:45.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022684b5b50b5ffe4-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-11-02T17:53:45.696 DEBUG:boto:path=/ibtljn-7/myobj 2023-11-02T17:53:45.697 DEBUG:boto:auth_path=/ibtljn-7/myobj 2023-11-02T17:53:45.697 DEBUG:boto:path=/ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.697 DEBUG:boto:auth_path=/ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.698 DEBUG:boto:Method: GET 2023-11-02T17:53:45.698 DEBUG:boto:Path: /ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.699 DEBUG:boto:Data: 2023-11-02T17:53:45.699 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.700 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.701 DEBUG:boto:Params: {} 2023-11-02T17:53:45.701 DEBUG:boto:Token: None 2023-11-02T17:53:45.702 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.702 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a8oDGULEfYuclETU8hvdOg465Mw= 2023-11-02T17:53:45.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a8oDGULEfYuclETU8hvdOg465Mw=', 'Content-Length': '0'} 2023-11-02T17:53:45.706 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005231c2aa3900dfb1-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.706 DEBUG:boto:path=/ibtljn-7/myobj 2023-11-02T17:53:45.707 DEBUG:boto:auth_path=/ibtljn-7/myobj 2023-11-02T17:53:45.707 DEBUG:boto:path=/ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.708 DEBUG:boto:auth_path=/ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.708 DEBUG:boto:Method: GET 2023-11-02T17:53:45.709 DEBUG:boto:Path: /ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.709 DEBUG:boto:Data: 2023-11-02T17:53:45.709 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.710 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.710 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.711 DEBUG:boto:Params: {} 2023-11-02T17:53:45.711 DEBUG:boto:Token: None 2023-11-02T17:53:45.712 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/myobj?versionId=null 2023-11-02T17:53:45.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a8oDGULEfYuclETU8hvdOg465Mw= 2023-11-02T17:53:45.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a8oDGULEfYuclETU8hvdOg465Mw=', 'Content-Length': '0'} 2023-11-02T17:53:45.715 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b12bfb64b85ddb78-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.758 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.759 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.759 DEBUG:boto:path=/ibtljn-7/?prefix=myobj 2023-11-02T17:53:45.760 DEBUG:boto:auth_path=/ibtljn-7/?prefix=myobj 2023-11-02T17:53:45.760 DEBUG:boto:Method: GET 2023-11-02T17:53:45.761 DEBUG:boto:Path: /ibtljn-7/?prefix=myobj 2023-11-02T17:53:45.761 DEBUG:boto:Data: 2023-11-02T17:53:45.761 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.762 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.762 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.763 DEBUG:boto:Params: {} 2023-11-02T17:53:45.763 DEBUG:boto:Token: None 2023-11-02T17:53:45.764 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.764 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d91da34ced2e45c-006543e229-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.771 DEBUG:boto:b'ibtljn-7myobj1000falsemyobj2023-11-02T17:51:58.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.771 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.772 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.772 DEBUG:boto:path=/ibtljn-7/?prefix=myobj 2023-11-02T17:53:45.773 DEBUG:boto:auth_path=/ibtljn-7/?prefix=myobj 2023-11-02T17:53:45.773 DEBUG:boto:Method: GET 2023-11-02T17:53:45.774 DEBUG:boto:Path: /ibtljn-7/?prefix=myobj 2023-11-02T17:53:45.774 DEBUG:boto:Data: 2023-11-02T17:53:45.774 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.775 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.775 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.776 DEBUG:boto:Params: {} 2023-11-02T17:53:45.776 DEBUG:boto:Token: None 2023-11-02T17:53:45.777 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.777 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2fbe1d2c16156d6-006543e229-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.784 DEBUG:boto:b'ibtljn-7myobj1000falsemyobj2023-11-02T17:51:58.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.784 DEBUG:boto:path=/ibtljn-7/myobj 2023-11-02T17:53:45.785 DEBUG:boto:auth_path=/ibtljn-7/myobj 2023-11-02T17:53:45.785 DEBUG:boto:Method: HEAD 2023-11-02T17:53:45.785 DEBUG:boto:Path: /ibtljn-7/myobj 2023-11-02T17:53:45.786 DEBUG:boto:Data: 2023-11-02T17:53:45.786 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.787 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.787 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.788 DEBUG:boto:Params: {} 2023-11-02T17:53:45.788 DEBUG:boto:Token: None 2023-11-02T17:53:45.789 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/myobj 2023-11-02T17:53:45.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mhn13w7b15GqZps3hRtRhvzGgT4= 2023-11-02T17:53:45.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mhn13w7b15GqZps3hRtRhvzGgT4=', 'Content-Length': '0'} 2023-11-02T17:53:45.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009172887ea6f690cc-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.793 DEBUG:boto:path=/ibtljn-7/myobj 2023-11-02T17:53:45.794 DEBUG:boto:auth_path=/ibtljn-7/myobj 2023-11-02T17:53:45.794 DEBUG:boto:Method: HEAD 2023-11-02T17:53:45.794 DEBUG:boto:Path: /ibtljn-7/myobj 2023-11-02T17:53:45.795 DEBUG:boto:Data: 2023-11-02T17:53:45.795 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.796 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.796 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.797 DEBUG:boto:Params: {} 2023-11-02T17:53:45.797 DEBUG:boto:Token: None 2023-11-02T17:53:45.798 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/myobj 2023-11-02T17:53:45.798 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mhn13w7b15GqZps3hRtRhvzGgT4= 2023-11-02T17:53:45.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mhn13w7b15GqZps3hRtRhvzGgT4=', 'Content-Length': '0'} 2023-11-02T17:53:45.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d89ac227ce1a063b-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-11-02T17:53:45.803 DEBUG:boto:path=/ibtljn-7/myobj 2023-11-02T17:53:45.803 DEBUG:boto:auth_path=/ibtljn-7/myobj 2023-11-02T17:53:45.804 DEBUG:boto:Method: GET 2023-11-02T17:53:45.804 DEBUG:boto:Path: /ibtljn-7/myobj 2023-11-02T17:53:45.804 DEBUG:boto:Data: 2023-11-02T17:53:45.805 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.805 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.806 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.806 DEBUG:boto:Params: {} 2023-11-02T17:53:45.807 DEBUG:boto:Token: None 2023-11-02T17:53:45.807 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/myobj 2023-11-02T17:53:45.808 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qmLhOn7GlpFJa6rJ/bXrhdvRoqw= 2023-11-02T17:53:45.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qmLhOn7GlpFJa6rJ/bXrhdvRoqw=', 'Content-Length': '0'} 2023-11-02T17:53:45.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000659a27af40dfb60b-006543e229-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.812 DEBUG:boto:path=/ibtljn-7/myobj 2023-11-02T17:53:45.813 DEBUG:boto:auth_path=/ibtljn-7/myobj 2023-11-02T17:53:45.813 DEBUG:boto:Method: GET 2023-11-02T17:53:45.813 DEBUG:boto:Path: /ibtljn-7/myobj 2023-11-02T17:53:45.814 DEBUG:boto:Data: 2023-11-02T17:53:45.814 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.815 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.815 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.816 DEBUG:boto:Params: {} 2023-11-02T17:53:45.816 DEBUG:boto:Token: None 2023-11-02T17:53:45.817 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/myobj 2023-11-02T17:53:45.817 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qmLhOn7GlpFJa6rJ/bXrhdvRoqw= 2023-11-02T17:53:45.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qmLhOn7GlpFJa6rJ/bXrhdvRoqw=', 'Content-Length': '0'} 2023-11-02T17:53:45.820 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000413bb8646aa0c90-006543e229-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.821 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.821 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.822 DEBUG:boto:path=/ibtljn-7/?prefix=myobj 2023-11-02T17:53:45.823 DEBUG:boto:auth_path=/ibtljn-7/?prefix=myobj 2023-11-02T17:53:45.823 DEBUG:boto:Method: GET 2023-11-02T17:53:45.826 DEBUG:boto:Path: /ibtljn-7/?prefix=myobj 2023-11-02T17:53:45.826 DEBUG:boto:Data: 2023-11-02T17:53:45.827 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.827 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:53:45.828 DEBUG:boto:Port: 8001 2023-11-02T17:53:45.828 DEBUG:boto:Params: {} 2023-11-02T17:53:45.829 DEBUG:boto:Token: None 2023-11-02T17:53:45.829 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.830 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae05dbcccd2e2ff3-006543e229-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.878 DEBUG:boto:b'ibtljn-7myobj1000falsemyobj2023-11-02T17:51:58.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.879 DEBUG:boto:path=/ibtljn-7/ 2023-11-02T17:53:45.880 DEBUG:boto:auth_path=/ibtljn-7/ 2023-11-02T17:53:45.880 DEBUG:boto:path=/ibtljn-7/?prefix=myobj 2023-11-02T17:53:45.880 DEBUG:boto:auth_path=/ibtljn-7/?prefix=myobj 2023-11-02T17:53:45.881 DEBUG:boto:Method: GET 2023-11-02T17:53:45.881 DEBUG:boto:Path: /ibtljn-7/?prefix=myobj 2023-11-02T17:53:45.882 DEBUG:boto:Data: 2023-11-02T17:53:45.882 DEBUG:boto:Headers: {} 2023-11-02T17:53:45.882 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:53:45.883 DEBUG:boto:Port: 8000 2023-11-02T17:53:45.883 DEBUG:boto:Params: {} 2023-11-02T17:53:45.884 DEBUG:boto:Token: None 2023-11-02T17:53:45.884 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:53:45 GMT /ibtljn-7/ 2023-11-02T17:53:45.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU= 2023-11-02T17:53:45.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:53:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nefWRMNgJ9xpHnh1okOusj4VBLU=', 'Content-Length': '0'} 2023-11-02T17:53:45.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5efe0ebb798739-006543e229-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:53:45.891 DEBUG:boto:b'ibtljn-7myobj1000falsemyobj2023-11-02T17:51:58.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:53:45.891 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-7 zones={a2, a1} 2023-11-02T17:53:45.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:53:45.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:53:45.893 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:53:45.948 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:53:45.948 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:53:46.015 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:46.007+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:53:51.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:51.011+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:53:56.018 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:53:56.015+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:54:01.019 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:54:01.015+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:54:06.020 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:54:06.015+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:54:11.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:54:11.015+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:54:16.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:54:16.019+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:54:21.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:54:21.019+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:54:26.024 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:54:26.018+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:54:31.025 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:54:31.018+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:54:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:54:36.022+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:54:41.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:54:41.022+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:54:46.029 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:54:46.022+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:54:51.030 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:54:51.022+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:54:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:54:56.026+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:55:01.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:01.026+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:55:06.033 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:06.026+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:55:11.035 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:11.030+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:55:16.036 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:16.030+0000 7f4f11fe6f40 1 waiting to reach incremental sync.. 2023-11-02T17:55:21.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:21.033+0000 7f4f11fe6f40 1 bucket sync caught up with source: 2023-11-02T17:55:21.041 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.9.6, , 00000000001.8.6, , , , 00000000001.5.6, 00000000001.7.6, , ] 2023-11-02T17:55:21.042 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.9.6, , 00000000001.8.6, , , , 00000000001.5.6, 00000000001.7.6, , ] 2023-11-02T17:55:21.042 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:21.033+0000 7f4f11fe6f40 0 bucket checkpoint complete 2023-11-02T17:55:21.066 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-8 zones={a1, a2} 2023-11-02T17:55:21.067 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.067 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.068 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.068 DEBUG:boto:Path: /ibtljn-8/ 2023-11-02T17:55:21.069 DEBUG:boto:Data: 2023-11-02T17:55:21.069 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.070 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.071 DEBUG:boto:Params: {} 2023-11-02T17:55:21.071 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T17:55:21.072 DEBUG:boto:Token: None 2023-11-02T17:55:21.072 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.073 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RWlVU6OAKr6xc7ccjaiO/1zmlOU= 2023-11-02T17:55:21.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RWlVU6OAKr6xc7ccjaiO/1zmlOU=', 'Content-Length': '0'} 2023-11-02T17:55:21.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a288a908ea0be38-006543e289-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.080 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.081 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.081 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.082 DEBUG:boto:Path: /ibtljn-8/ 2023-11-02T17:55:21.082 DEBUG:boto:Data: 2023-11-02T17:55:21.082 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.083 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.083 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.084 DEBUG:boto:Params: {} 2023-11-02T17:55:21.084 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T17:55:21.085 DEBUG:boto:Token: None 2023-11-02T17:55:21.085 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.086 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RWlVU6OAKr6xc7ccjaiO/1zmlOU= 2023-11-02T17:55:21.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RWlVU6OAKr6xc7ccjaiO/1zmlOU=', 'Content-Length': '0'} 2023-11-02T17:55:21.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a41d3ab265a09c9-006543e289-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.092 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T17:55:21.093 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.093 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.094 DEBUG:boto:path=/ibtljn-8/?versions&max-keys=999 2023-11-02T17:55:21.094 DEBUG:boto:auth_path=/ibtljn-8/?versions&max-keys=999 2023-11-02T17:55:21.095 DEBUG:boto:Method: GET 2023-11-02T17:55:21.095 DEBUG:boto:Path: /ibtljn-8/?versions&max-keys=999 2023-11-02T17:55:21.095 DEBUG:boto:Data: 2023-11-02T17:55:21.096 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.096 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.097 DEBUG:boto:Params: {} 2023-11-02T17:55:21.097 DEBUG:boto:Token: None 2023-11-02T17:55:21.098 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/?versions 2023-11-02T17:55:21.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VtuYNtgkNcHWVr8GzZgGKnIg64g= 2023-11-02T17:55:21.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VtuYNtgkNcHWVr8GzZgGKnIg64g=', 'Content-Length': '0'} 2023-11-02T17:55:21.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcae6350c73df724-006543e289-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.104 DEBUG:boto:b'ibtljn-8999false&nulltrue2023-11-02T17:51:59.188Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-11-02T17:51:59.129Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-11-02T17:51:59.090Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-11-02T17:51:59.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.105 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-11-02T17:55:21.105 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-11-02T17:55:21.106 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-11-02T17:55:21.106 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-11-02T17:55:21.107 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T17:55:21.107 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.107 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.108 DEBUG:boto:path=/ibtljn-8/?versions&max-keys=999 2023-11-02T17:55:21.108 DEBUG:boto:auth_path=/ibtljn-8/?versions&max-keys=999 2023-11-02T17:55:21.109 DEBUG:boto:Method: GET 2023-11-02T17:55:21.109 DEBUG:boto:Path: /ibtljn-8/?versions&max-keys=999 2023-11-02T17:55:21.110 DEBUG:boto:Data: 2023-11-02T17:55:21.110 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.110 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.111 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.111 DEBUG:boto:Params: {} 2023-11-02T17:55:21.112 DEBUG:boto:Token: None 2023-11-02T17:55:21.112 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/?versions 2023-11-02T17:55:21.113 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VtuYNtgkNcHWVr8GzZgGKnIg64g= 2023-11-02T17:55:21.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VtuYNtgkNcHWVr8GzZgGKnIg64g=', 'Content-Length': '0'} 2023-11-02T17:55:21.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3eb8318262c8663-006543e289-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.118 DEBUG:boto:b'ibtljn-8999false&nulltrue2023-11-02T17:51:59.188Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-11-02T17:51:59.129Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-11-02T17:51:59.090Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-11-02T17:51:59.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.119 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-11-02T17:55:21.120 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-11-02T17:55:21.120 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-11-02T17:55:21.120 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-11-02T17:55:21.121 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.121 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.122 DEBUG:boto:path=/ibtljn-8/?versions&max-keys=999 2023-11-02T17:55:21.122 DEBUG:boto:auth_path=/ibtljn-8/?versions&max-keys=999 2023-11-02T17:55:21.123 DEBUG:boto:Method: GET 2023-11-02T17:55:21.123 DEBUG:boto:Path: /ibtljn-8/?versions&max-keys=999 2023-11-02T17:55:21.124 DEBUG:boto:Data: 2023-11-02T17:55:21.124 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.124 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.125 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.125 DEBUG:boto:Params: {} 2023-11-02T17:55:21.126 DEBUG:boto:Token: None 2023-11-02T17:55:21.126 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/?versions 2023-11-02T17:55:21.127 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VtuYNtgkNcHWVr8GzZgGKnIg64g= 2023-11-02T17:55:21.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VtuYNtgkNcHWVr8GzZgGKnIg64g=', 'Content-Length': '0'} 2023-11-02T17:55:21.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae71e82036b8395f-006543e289-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.132 DEBUG:boto:b'ibtljn-8999false&nulltrue2023-11-02T17:51:59.188Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-11-02T17:51:59.129Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-11-02T17:51:59.090Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-11-02T17:51:59.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.133 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.134 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.134 DEBUG:boto:path=/ibtljn-8/?versions&max-keys=999 2023-11-02T17:55:21.134 DEBUG:boto:auth_path=/ibtljn-8/?versions&max-keys=999 2023-11-02T17:55:21.135 DEBUG:boto:Method: GET 2023-11-02T17:55:21.135 DEBUG:boto:Path: /ibtljn-8/?versions&max-keys=999 2023-11-02T17:55:21.136 DEBUG:boto:Data: 2023-11-02T17:55:21.136 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.136 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.137 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.137 DEBUG:boto:Params: {} 2023-11-02T17:55:21.138 DEBUG:boto:Token: None 2023-11-02T17:55:21.138 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/?versions 2023-11-02T17:55:21.139 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VtuYNtgkNcHWVr8GzZgGKnIg64g= 2023-11-02T17:55:21.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VtuYNtgkNcHWVr8GzZgGKnIg64g=', 'Content-Length': '0'} 2023-11-02T17:55:21.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046fd7b2c3b4e6e21-006543e289-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.145 DEBUG:boto:b'ibtljn-8999false&nulltrue2023-11-02T17:51:59.188Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-11-02T17:51:59.129Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-11-02T17:51:59.090Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-11-02T17:51:59.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-11-02T17:55:21.146 DEBUG:boto:path=/ibtljn-8/%26 2023-11-02T17:55:21.146 DEBUG:boto:auth_path=/ibtljn-8/%26 2023-11-02T17:55:21.147 DEBUG:boto:path=/ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.147 DEBUG:boto:auth_path=/ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.148 DEBUG:boto:Method: GET 2023-11-02T17:55:21.148 DEBUG:boto:Path: /ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.149 DEBUG:boto:Data: 2023-11-02T17:55:21.149 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.150 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.150 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.150 DEBUG:boto:Params: {} 2023-11-02T17:55:21.151 DEBUG:boto:Token: None 2023-11-02T17:55:21.151 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.152 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m1z3qHBAqo6S2gmTIyjSSIeOtTA= 2023-11-02T17:55:21.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m1z3qHBAqo6S2gmTIyjSSIeOtTA=', 'Content-Length': '0'} 2023-11-02T17:55:21.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066ab069cb1d3b04a-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.198 DEBUG:boto:path=/ibtljn-8/%26 2023-11-02T17:55:21.199 DEBUG:boto:auth_path=/ibtljn-8/%26 2023-11-02T17:55:21.200 DEBUG:boto:path=/ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.200 DEBUG:boto:auth_path=/ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.201 DEBUG:boto:Method: GET 2023-11-02T17:55:21.201 DEBUG:boto:Path: /ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.201 DEBUG:boto:Data: 2023-11-02T17:55:21.202 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.203 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.203 DEBUG:boto:Params: {} 2023-11-02T17:55:21.204 DEBUG:boto:Token: None 2023-11-02T17:55:21.204 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.205 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m1z3qHBAqo6S2gmTIyjSSIeOtTA= 2023-11-02T17:55:21.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m1z3qHBAqo6S2gmTIyjSSIeOtTA=', 'Content-Length': '0'} 2023-11-02T17:55:21.209 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0d9fb0b8a7d2c99-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.209 DEBUG:boto:path=/ibtljn-8/%26 2023-11-02T17:55:21.210 DEBUG:boto:auth_path=/ibtljn-8/%26 2023-11-02T17:55:21.210 DEBUG:boto:path=/ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.211 DEBUG:boto:auth_path=/ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.211 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.212 DEBUG:boto:Path: /ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.212 DEBUG:boto:Data: 2023-11-02T17:55:21.213 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.213 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.213 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.214 DEBUG:boto:Params: {} 2023-11-02T17:55:21.214 DEBUG:boto:Token: None 2023-11-02T17:55:21.215 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.215 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pZQmbetVFtA1nWkB8T0NVPb3OU0= 2023-11-02T17:55:21.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pZQmbetVFtA1nWkB8T0NVPb3OU0=', 'Content-Length': '0'} 2023-11-02T17:55:21.219 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054cfad49c9b652fb-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.220 DEBUG:boto:path=/ibtljn-8/%26 2023-11-02T17:55:21.220 DEBUG:boto:auth_path=/ibtljn-8/%26 2023-11-02T17:55:21.221 DEBUG:boto:path=/ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.221 DEBUG:boto:auth_path=/ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.221 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.222 DEBUG:boto:Path: /ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.222 DEBUG:boto:Data: 2023-11-02T17:55:21.223 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.224 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.224 DEBUG:boto:Params: {} 2023-11-02T17:55:21.225 DEBUG:boto:Token: None 2023-11-02T17:55:21.225 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pZQmbetVFtA1nWkB8T0NVPb3OU0= 2023-11-02T17:55:21.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pZQmbetVFtA1nWkB8T0NVPb3OU0=', 'Content-Length': '0'} 2023-11-02T17:55:21.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a8528672e2aaf36-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-11-02T17:55:21.232 DEBUG:boto:path=/ibtljn-8/%26 2023-11-02T17:55:21.233 DEBUG:boto:auth_path=/ibtljn-8/%26 2023-11-02T17:55:21.234 DEBUG:boto:path=/ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.234 DEBUG:boto:auth_path=/ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.234 DEBUG:boto:Method: GET 2023-11-02T17:55:21.235 DEBUG:boto:Path: /ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.235 DEBUG:boto:Data: 2023-11-02T17:55:21.236 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.236 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.237 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.237 DEBUG:boto:Params: {} 2023-11-02T17:55:21.238 DEBUG:boto:Token: None 2023-11-02T17:55:21.239 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.239 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m1z3qHBAqo6S2gmTIyjSSIeOtTA= 2023-11-02T17:55:21.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m1z3qHBAqo6S2gmTIyjSSIeOtTA=', 'Content-Length': '0'} 2023-11-02T17:55:21.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a781daf2de961a9a-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.243 DEBUG:boto:path=/ibtljn-8/%26 2023-11-02T17:55:21.244 DEBUG:boto:auth_path=/ibtljn-8/%26 2023-11-02T17:55:21.245 DEBUG:boto:path=/ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.245 DEBUG:boto:auth_path=/ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.246 DEBUG:boto:Method: GET 2023-11-02T17:55:21.246 DEBUG:boto:Path: /ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.247 DEBUG:boto:Data: 2023-11-02T17:55:21.247 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.247 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.248 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.248 DEBUG:boto:Params: {} 2023-11-02T17:55:21.249 DEBUG:boto:Token: None 2023-11-02T17:55:21.249 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%26?versionId=null 2023-11-02T17:55:21.250 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m1z3qHBAqo6S2gmTIyjSSIeOtTA= 2023-11-02T17:55:21.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m1z3qHBAqo6S2gmTIyjSSIeOtTA=', 'Content-Length': '0'} 2023-11-02T17:55:21.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009967a2195fea50c4-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.294 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.295 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.295 DEBUG:boto:path=/ibtljn-8/?prefix=%26 2023-11-02T17:55:21.296 DEBUG:boto:auth_path=/ibtljn-8/?prefix=%26 2023-11-02T17:55:21.296 DEBUG:boto:Method: GET 2023-11-02T17:55:21.297 DEBUG:boto:Path: /ibtljn-8/?prefix=%26 2023-11-02T17:55:21.297 DEBUG:boto:Data: 2023-11-02T17:55:21.298 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.298 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.299 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.299 DEBUG:boto:Params: {} 2023-11-02T17:55:21.300 DEBUG:boto:Token: None 2023-11-02T17:55:21.300 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.301 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8= 2023-11-02T17:55:21.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8=', 'Content-Length': '0'} 2023-11-02T17:55:21.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d65c9391567041d-006543e289-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.307 DEBUG:boto:b'ibtljn-8&1000false&2023-11-02T17:51:59.188Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.308 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.308 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.309 DEBUG:boto:path=/ibtljn-8/?prefix=%26 2023-11-02T17:55:21.309 DEBUG:boto:auth_path=/ibtljn-8/?prefix=%26 2023-11-02T17:55:21.310 DEBUG:boto:Method: GET 2023-11-02T17:55:21.310 DEBUG:boto:Path: /ibtljn-8/?prefix=%26 2023-11-02T17:55:21.311 DEBUG:boto:Data: 2023-11-02T17:55:21.311 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.311 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.312 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.312 DEBUG:boto:Params: {} 2023-11-02T17:55:21.313 DEBUG:boto:Token: None 2023-11-02T17:55:21.314 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.314 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8= 2023-11-02T17:55:21.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8=', 'Content-Length': '0'} 2023-11-02T17:55:21.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072f4c1c0f9e782f2-006543e289-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.320 DEBUG:boto:b'ibtljn-8&1000false&2023-11-02T17:51:59.188Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.321 DEBUG:boto:path=/ibtljn-8/%26 2023-11-02T17:55:21.321 DEBUG:boto:auth_path=/ibtljn-8/%26 2023-11-02T17:55:21.322 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.322 DEBUG:boto:Path: /ibtljn-8/%26 2023-11-02T17:55:21.323 DEBUG:boto:Data: 2023-11-02T17:55:21.323 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.324 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.324 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.324 DEBUG:boto:Params: {} 2023-11-02T17:55:21.325 DEBUG:boto:Token: None 2023-11-02T17:55:21.325 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%26 2023-11-02T17:55:21.326 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8mF+VtXAAYqt/ke2J3KYvnBN2SM= 2023-11-02T17:55:21.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8mF+VtXAAYqt/ke2J3KYvnBN2SM=', 'Content-Length': '0'} 2023-11-02T17:55:21.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebfd3205c7cd0a49-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.330 DEBUG:boto:path=/ibtljn-8/%26 2023-11-02T17:55:21.330 DEBUG:boto:auth_path=/ibtljn-8/%26 2023-11-02T17:55:21.331 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.331 DEBUG:boto:Path: /ibtljn-8/%26 2023-11-02T17:55:21.332 DEBUG:boto:Data: 2023-11-02T17:55:21.332 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.332 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.333 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.333 DEBUG:boto:Params: {} 2023-11-02T17:55:21.334 DEBUG:boto:Token: None 2023-11-02T17:55:21.334 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%26 2023-11-02T17:55:21.335 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8mF+VtXAAYqt/ke2J3KYvnBN2SM= 2023-11-02T17:55:21.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8mF+VtXAAYqt/ke2J3KYvnBN2SM=', 'Content-Length': '0'} 2023-11-02T17:55:21.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d03d07616c6e6cc5-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-11-02T17:55:21.339 DEBUG:boto:path=/ibtljn-8/%26 2023-11-02T17:55:21.340 DEBUG:boto:auth_path=/ibtljn-8/%26 2023-11-02T17:55:21.340 DEBUG:boto:Method: GET 2023-11-02T17:55:21.341 DEBUG:boto:Path: /ibtljn-8/%26 2023-11-02T17:55:21.341 DEBUG:boto:Data: 2023-11-02T17:55:21.342 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.342 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.343 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.343 DEBUG:boto:Params: {} 2023-11-02T17:55:21.344 DEBUG:boto:Token: None 2023-11-02T17:55:21.344 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%26 2023-11-02T17:55:21.345 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vDhetUqjQK/D9Vb+Ae7nX7qjRCM= 2023-11-02T17:55:21.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vDhetUqjQK/D9Vb+Ae7nX7qjRCM=', 'Content-Length': '0'} 2023-11-02T17:55:21.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093b1bf79a65ee8a8-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.390 DEBUG:boto:path=/ibtljn-8/%26 2023-11-02T17:55:21.391 DEBUG:boto:auth_path=/ibtljn-8/%26 2023-11-02T17:55:21.391 DEBUG:boto:Method: GET 2023-11-02T17:55:21.392 DEBUG:boto:Path: /ibtljn-8/%26 2023-11-02T17:55:21.392 DEBUG:boto:Data: 2023-11-02T17:55:21.393 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.393 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.394 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.394 DEBUG:boto:Params: {} 2023-11-02T17:55:21.395 DEBUG:boto:Token: None 2023-11-02T17:55:21.395 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%26 2023-11-02T17:55:21.396 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vDhetUqjQK/D9Vb+Ae7nX7qjRCM= 2023-11-02T17:55:21.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vDhetUqjQK/D9Vb+Ae7nX7qjRCM=', 'Content-Length': '0'} 2023-11-02T17:55:21.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000678e3d24b8690c83-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.400 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.401 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.401 DEBUG:boto:path=/ibtljn-8/?prefix=%26 2023-11-02T17:55:21.402 DEBUG:boto:auth_path=/ibtljn-8/?prefix=%26 2023-11-02T17:55:21.402 DEBUG:boto:Method: GET 2023-11-02T17:55:21.403 DEBUG:boto:Path: /ibtljn-8/?prefix=%26 2023-11-02T17:55:21.403 DEBUG:boto:Data: 2023-11-02T17:55:21.404 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.404 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.405 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.405 DEBUG:boto:Params: {} 2023-11-02T17:55:21.406 DEBUG:boto:Token: None 2023-11-02T17:55:21.406 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.407 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8= 2023-11-02T17:55:21.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8=', 'Content-Length': '0'} 2023-11-02T17:55:21.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf80942326fbce62-006543e289-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.413 DEBUG:boto:b'ibtljn-8&1000false&2023-11-02T17:51:59.188Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.413 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.414 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.415 DEBUG:boto:path=/ibtljn-8/?prefix=%26 2023-11-02T17:55:21.415 DEBUG:boto:auth_path=/ibtljn-8/?prefix=%26 2023-11-02T17:55:21.415 DEBUG:boto:Method: GET 2023-11-02T17:55:21.416 DEBUG:boto:Path: /ibtljn-8/?prefix=%26 2023-11-02T17:55:21.416 DEBUG:boto:Data: 2023-11-02T17:55:21.417 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.417 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.417 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.418 DEBUG:boto:Params: {} 2023-11-02T17:55:21.418 DEBUG:boto:Token: None 2023-11-02T17:55:21.419 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.419 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8= 2023-11-02T17:55:21.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8=', 'Content-Length': '0'} 2023-11-02T17:55:21.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e17a265eaefd60f-006543e289-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.466 DEBUG:boto:b'ibtljn-8&1000false&2023-11-02T17:51:59.188Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-11-02T17:55:21.467 DEBUG:boto:path=/ibtljn-8/%3A 2023-11-02T17:55:21.468 DEBUG:boto:auth_path=/ibtljn-8/%3A 2023-11-02T17:55:21.468 DEBUG:boto:path=/ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.469 DEBUG:boto:auth_path=/ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.469 DEBUG:boto:Method: GET 2023-11-02T17:55:21.469 DEBUG:boto:Path: /ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.470 DEBUG:boto:Data: 2023-11-02T17:55:21.470 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.471 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.471 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.472 DEBUG:boto:Params: {} 2023-11-02T17:55:21.472 DEBUG:boto:Token: None 2023-11-02T17:55:21.472 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.473 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NwzxAp4UrFFNqk/vF+0xeqb7xTY= 2023-11-02T17:55:21.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NwzxAp4UrFFNqk/vF+0xeqb7xTY=', 'Content-Length': '0'} 2023-11-02T17:55:21.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb030281992a0c98-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.477 DEBUG:boto:path=/ibtljn-8/%3A 2023-11-02T17:55:21.478 DEBUG:boto:auth_path=/ibtljn-8/%3A 2023-11-02T17:55:21.478 DEBUG:boto:path=/ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.479 DEBUG:boto:auth_path=/ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.479 DEBUG:boto:Method: GET 2023-11-02T17:55:21.480 DEBUG:boto:Path: /ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.480 DEBUG:boto:Data: 2023-11-02T17:55:21.481 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.481 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.482 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.482 DEBUG:boto:Params: {} 2023-11-02T17:55:21.483 DEBUG:boto:Token: None 2023-11-02T17:55:21.483 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.484 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NwzxAp4UrFFNqk/vF+0xeqb7xTY= 2023-11-02T17:55:21.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NwzxAp4UrFFNqk/vF+0xeqb7xTY=', 'Content-Length': '0'} 2023-11-02T17:55:21.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000882622895eb46dcb-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.488 DEBUG:boto:path=/ibtljn-8/%3A 2023-11-02T17:55:21.489 DEBUG:boto:auth_path=/ibtljn-8/%3A 2023-11-02T17:55:21.489 DEBUG:boto:path=/ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.490 DEBUG:boto:auth_path=/ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.490 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.491 DEBUG:boto:Path: /ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.491 DEBUG:boto:Data: 2023-11-02T17:55:21.492 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.492 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.492 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.493 DEBUG:boto:Params: {} 2023-11-02T17:55:21.493 DEBUG:boto:Token: None 2023-11-02T17:55:21.494 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.494 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:K/VMcgvKkae9UFwekbks+RYpxQw= 2023-11-02T17:55:21.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:K/VMcgvKkae9UFwekbks+RYpxQw=', 'Content-Length': '0'} 2023-11-02T17:55:21.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8d7ce96bd1c7bf9-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.498 DEBUG:boto:path=/ibtljn-8/%3A 2023-11-02T17:55:21.498 DEBUG:boto:auth_path=/ibtljn-8/%3A 2023-11-02T17:55:21.499 DEBUG:boto:path=/ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.499 DEBUG:boto:auth_path=/ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.499 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.500 DEBUG:boto:Path: /ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.500 DEBUG:boto:Data: 2023-11-02T17:55:21.501 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.501 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.502 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.502 DEBUG:boto:Params: {} 2023-11-02T17:55:21.503 DEBUG:boto:Token: None 2023-11-02T17:55:21.503 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.503 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:K/VMcgvKkae9UFwekbks+RYpxQw= 2023-11-02T17:55:21.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:K/VMcgvKkae9UFwekbks+RYpxQw=', 'Content-Length': '0'} 2023-11-02T17:55:21.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e64a6b2e265f43a6-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-11-02T17:55:21.508 DEBUG:boto:path=/ibtljn-8/%3A 2023-11-02T17:55:21.508 DEBUG:boto:auth_path=/ibtljn-8/%3A 2023-11-02T17:55:21.509 DEBUG:boto:path=/ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.509 DEBUG:boto:auth_path=/ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.509 DEBUG:boto:Method: GET 2023-11-02T17:55:21.510 DEBUG:boto:Path: /ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.510 DEBUG:boto:Data: 2023-11-02T17:55:21.511 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.511 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.512 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.512 DEBUG:boto:Params: {} 2023-11-02T17:55:21.513 DEBUG:boto:Token: None 2023-11-02T17:55:21.513 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.514 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NwzxAp4UrFFNqk/vF+0xeqb7xTY= 2023-11-02T17:55:21.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NwzxAp4UrFFNqk/vF+0xeqb7xTY=', 'Content-Length': '0'} 2023-11-02T17:55:21.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000504e3a04be4e6bc4-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.518 DEBUG:boto:path=/ibtljn-8/%3A 2023-11-02T17:55:21.519 DEBUG:boto:auth_path=/ibtljn-8/%3A 2023-11-02T17:55:21.519 DEBUG:boto:path=/ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.520 DEBUG:boto:auth_path=/ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.520 DEBUG:boto:Method: GET 2023-11-02T17:55:21.521 DEBUG:boto:Path: /ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.521 DEBUG:boto:Data: 2023-11-02T17:55:21.522 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.522 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.523 DEBUG:boto:Params: {} 2023-11-02T17:55:21.523 DEBUG:boto:Token: None 2023-11-02T17:55:21.524 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%3A?versionId=null 2023-11-02T17:55:21.525 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NwzxAp4UrFFNqk/vF+0xeqb7xTY= 2023-11-02T17:55:21.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NwzxAp4UrFFNqk/vF+0xeqb7xTY=', 'Content-Length': '0'} 2023-11-02T17:55:21.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ecc8af2323ab340-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.571 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.571 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.572 DEBUG:boto:path=/ibtljn-8/?prefix=%3A 2023-11-02T17:55:21.572 DEBUG:boto:auth_path=/ibtljn-8/?prefix=%3A 2023-11-02T17:55:21.573 DEBUG:boto:Method: GET 2023-11-02T17:55:21.573 DEBUG:boto:Path: /ibtljn-8/?prefix=%3A 2023-11-02T17:55:21.574 DEBUG:boto:Data: 2023-11-02T17:55:21.574 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.575 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.575 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.575 DEBUG:boto:Params: {} 2023-11-02T17:55:21.576 DEBUG:boto:Token: None 2023-11-02T17:55:21.576 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.577 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8= 2023-11-02T17:55:21.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8=', 'Content-Length': '0'} 2023-11-02T17:55:21.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e13adb60b949848e-006543e289-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.583 DEBUG:boto:b'ibtljn-8:1000false:2023-11-02T17:51:59.129Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.584 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.584 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.585 DEBUG:boto:path=/ibtljn-8/?prefix=%3A 2023-11-02T17:55:21.585 DEBUG:boto:auth_path=/ibtljn-8/?prefix=%3A 2023-11-02T17:55:21.586 DEBUG:boto:Method: GET 2023-11-02T17:55:21.586 DEBUG:boto:Path: /ibtljn-8/?prefix=%3A 2023-11-02T17:55:21.587 DEBUG:boto:Data: 2023-11-02T17:55:21.587 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.588 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.588 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.589 DEBUG:boto:Params: {} 2023-11-02T17:55:21.589 DEBUG:boto:Token: None 2023-11-02T17:55:21.590 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.590 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8= 2023-11-02T17:55:21.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8=', 'Content-Length': '0'} 2023-11-02T17:55:21.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad1a2852ec34dfbd-006543e289-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.597 DEBUG:boto:b'ibtljn-8:1000false:2023-11-02T17:51:59.129Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.597 DEBUG:boto:path=/ibtljn-8/%3A 2023-11-02T17:55:21.598 DEBUG:boto:auth_path=/ibtljn-8/%3A 2023-11-02T17:55:21.598 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.599 DEBUG:boto:Path: /ibtljn-8/%3A 2023-11-02T17:55:21.599 DEBUG:boto:Data: 2023-11-02T17:55:21.600 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.600 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.601 DEBUG:boto:Params: {} 2023-11-02T17:55:21.601 DEBUG:boto:Token: None 2023-11-02T17:55:21.602 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%3A 2023-11-02T17:55:21.602 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b73y7jnkT7vJHmrcwrE5MPtiWvw= 2023-11-02T17:55:21.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b73y7jnkT7vJHmrcwrE5MPtiWvw=', 'Content-Length': '0'} 2023-11-02T17:55:21.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f312fa8679b309d8-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.607 DEBUG:boto:path=/ibtljn-8/%3A 2023-11-02T17:55:21.607 DEBUG:boto:auth_path=/ibtljn-8/%3A 2023-11-02T17:55:21.608 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.608 DEBUG:boto:Path: /ibtljn-8/%3A 2023-11-02T17:55:21.609 DEBUG:boto:Data: 2023-11-02T17:55:21.609 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.609 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.610 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.610 DEBUG:boto:Params: {} 2023-11-02T17:55:21.611 DEBUG:boto:Token: None 2023-11-02T17:55:21.611 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%3A 2023-11-02T17:55:21.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b73y7jnkT7vJHmrcwrE5MPtiWvw= 2023-11-02T17:55:21.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b73y7jnkT7vJHmrcwrE5MPtiWvw=', 'Content-Length': '0'} 2023-11-02T17:55:21.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e3295687d150083-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-11-02T17:55:21.617 DEBUG:boto:path=/ibtljn-8/%3A 2023-11-02T17:55:21.617 DEBUG:boto:auth_path=/ibtljn-8/%3A 2023-11-02T17:55:21.618 DEBUG:boto:Method: GET 2023-11-02T17:55:21.618 DEBUG:boto:Path: /ibtljn-8/%3A 2023-11-02T17:55:21.618 DEBUG:boto:Data: 2023-11-02T17:55:21.619 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.620 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.620 DEBUG:boto:Params: {} 2023-11-02T17:55:21.621 DEBUG:boto:Token: None 2023-11-02T17:55:21.621 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%3A 2023-11-02T17:55:21.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QyYDfnknN96y+pC9t86c0J/RHSk= 2023-11-02T17:55:21.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QyYDfnknN96y+pC9t86c0J/RHSk=', 'Content-Length': '0'} 2023-11-02T17:55:21.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047e5c30d5b3b5919-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.625 DEBUG:boto:path=/ibtljn-8/%3A 2023-11-02T17:55:21.626 DEBUG:boto:auth_path=/ibtljn-8/%3A 2023-11-02T17:55:21.626 DEBUG:boto:Method: GET 2023-11-02T17:55:21.627 DEBUG:boto:Path: /ibtljn-8/%3A 2023-11-02T17:55:21.627 DEBUG:boto:Data: 2023-11-02T17:55:21.628 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.628 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.628 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.629 DEBUG:boto:Params: {} 2023-11-02T17:55:21.629 DEBUG:boto:Token: None 2023-11-02T17:55:21.630 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/%3A 2023-11-02T17:55:21.630 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QyYDfnknN96y+pC9t86c0J/RHSk= 2023-11-02T17:55:21.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QyYDfnknN96y+pC9t86c0J/RHSk=', 'Content-Length': '0'} 2023-11-02T17:55:21.634 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026342af13d0e6725-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.674 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.675 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.675 DEBUG:boto:path=/ibtljn-8/?prefix=%3A 2023-11-02T17:55:21.676 DEBUG:boto:auth_path=/ibtljn-8/?prefix=%3A 2023-11-02T17:55:21.676 DEBUG:boto:Method: GET 2023-11-02T17:55:21.677 DEBUG:boto:Path: /ibtljn-8/?prefix=%3A 2023-11-02T17:55:21.677 DEBUG:boto:Data: 2023-11-02T17:55:21.678 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.678 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.679 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.679 DEBUG:boto:Params: {} 2023-11-02T17:55:21.679 DEBUG:boto:Token: None 2023-11-02T17:55:21.680 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.680 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8= 2023-11-02T17:55:21.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8=', 'Content-Length': '0'} 2023-11-02T17:55:21.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000148dc64e7d75d4ec-006543e289-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.687 DEBUG:boto:b'ibtljn-8:1000false:2023-11-02T17:51:59.129Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.687 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.688 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.688 DEBUG:boto:path=/ibtljn-8/?prefix=%3A 2023-11-02T17:55:21.689 DEBUG:boto:auth_path=/ibtljn-8/?prefix=%3A 2023-11-02T17:55:21.689 DEBUG:boto:Method: GET 2023-11-02T17:55:21.690 DEBUG:boto:Path: /ibtljn-8/?prefix=%3A 2023-11-02T17:55:21.690 DEBUG:boto:Data: 2023-11-02T17:55:21.690 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.691 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.691 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.692 DEBUG:boto:Params: {} 2023-11-02T17:55:21.692 DEBUG:boto:Token: None 2023-11-02T17:55:21.692 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.693 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8= 2023-11-02T17:55:21.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8=', 'Content-Length': '0'} 2023-11-02T17:55:21.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005463f83c0cedd280-006543e289-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.699 DEBUG:boto:b'ibtljn-8:1000false:2023-11-02T17:51:59.129Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-11-02T17:55:21.700 DEBUG:boto:path=/ibtljn-8/_myobj 2023-11-02T17:55:21.701 DEBUG:boto:auth_path=/ibtljn-8/_myobj 2023-11-02T17:55:21.701 DEBUG:boto:path=/ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.701 DEBUG:boto:auth_path=/ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.702 DEBUG:boto:Method: GET 2023-11-02T17:55:21.702 DEBUG:boto:Path: /ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.703 DEBUG:boto:Data: 2023-11-02T17:55:21.703 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.704 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.704 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.705 DEBUG:boto:Params: {} 2023-11-02T17:55:21.705 DEBUG:boto:Token: None 2023-11-02T17:55:21.706 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.706 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:baGvCXL+b1cAROeX8+rpJYeZSM4= 2023-11-02T17:55:21.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:baGvCXL+b1cAROeX8+rpJYeZSM4=', 'Content-Length': '0'} 2023-11-02T17:55:21.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000849e1f8bbfdbd465-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.754 DEBUG:boto:path=/ibtljn-8/_myobj 2023-11-02T17:55:21.755 DEBUG:boto:auth_path=/ibtljn-8/_myobj 2023-11-02T17:55:21.755 DEBUG:boto:path=/ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.756 DEBUG:boto:auth_path=/ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.756 DEBUG:boto:Method: GET 2023-11-02T17:55:21.757 DEBUG:boto:Path: /ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.757 DEBUG:boto:Data: 2023-11-02T17:55:21.758 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.758 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.759 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.759 DEBUG:boto:Params: {} 2023-11-02T17:55:21.760 DEBUG:boto:Token: None 2023-11-02T17:55:21.760 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.761 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:baGvCXL+b1cAROeX8+rpJYeZSM4= 2023-11-02T17:55:21.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:baGvCXL+b1cAROeX8+rpJYeZSM4=', 'Content-Length': '0'} 2023-11-02T17:55:21.764 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087b7ba7e3f8cb92c-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.765 DEBUG:boto:path=/ibtljn-8/_myobj 2023-11-02T17:55:21.765 DEBUG:boto:auth_path=/ibtljn-8/_myobj 2023-11-02T17:55:21.766 DEBUG:boto:path=/ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.766 DEBUG:boto:auth_path=/ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.767 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.767 DEBUG:boto:Path: /ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.768 DEBUG:boto:Data: 2023-11-02T17:55:21.768 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.768 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.769 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.769 DEBUG:boto:Params: {} 2023-11-02T17:55:21.770 DEBUG:boto:Token: None 2023-11-02T17:55:21.770 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.771 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xsffnh+4YYqJfQeR0XltaQVKmLQ= 2023-11-02T17:55:21.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xsffnh+4YYqJfQeR0XltaQVKmLQ=', 'Content-Length': '0'} 2023-11-02T17:55:21.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2df5ee2c222ff68-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.775 DEBUG:boto:path=/ibtljn-8/_myobj 2023-11-02T17:55:21.775 DEBUG:boto:auth_path=/ibtljn-8/_myobj 2023-11-02T17:55:21.775 DEBUG:boto:path=/ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.776 DEBUG:boto:auth_path=/ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.776 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.777 DEBUG:boto:Path: /ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.777 DEBUG:boto:Data: 2023-11-02T17:55:21.777 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.778 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.778 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.779 DEBUG:boto:Params: {} 2023-11-02T17:55:21.779 DEBUG:boto:Token: None 2023-11-02T17:55:21.780 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.780 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xsffnh+4YYqJfQeR0XltaQVKmLQ= 2023-11-02T17:55:21.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xsffnh+4YYqJfQeR0XltaQVKmLQ=', 'Content-Length': '0'} 2023-11-02T17:55:21.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db8c7c5f5a4443cb-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-11-02T17:55:21.785 DEBUG:boto:path=/ibtljn-8/_myobj 2023-11-02T17:55:21.785 DEBUG:boto:auth_path=/ibtljn-8/_myobj 2023-11-02T17:55:21.785 DEBUG:boto:path=/ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.786 DEBUG:boto:auth_path=/ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.786 DEBUG:boto:Method: GET 2023-11-02T17:55:21.787 DEBUG:boto:Path: /ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.787 DEBUG:boto:Data: 2023-11-02T17:55:21.788 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.788 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.788 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.789 DEBUG:boto:Params: {} 2023-11-02T17:55:21.789 DEBUG:boto:Token: None 2023-11-02T17:55:21.790 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.790 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:baGvCXL+b1cAROeX8+rpJYeZSM4= 2023-11-02T17:55:21.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:baGvCXL+b1cAROeX8+rpJYeZSM4=', 'Content-Length': '0'} 2023-11-02T17:55:21.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6e2ae913cdd6148-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.794 DEBUG:boto:path=/ibtljn-8/_myobj 2023-11-02T17:55:21.795 DEBUG:boto:auth_path=/ibtljn-8/_myobj 2023-11-02T17:55:21.795 DEBUG:boto:path=/ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.795 DEBUG:boto:auth_path=/ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.796 DEBUG:boto:Method: GET 2023-11-02T17:55:21.796 DEBUG:boto:Path: /ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.797 DEBUG:boto:Data: 2023-11-02T17:55:21.797 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.797 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.798 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.798 DEBUG:boto:Params: {} 2023-11-02T17:55:21.799 DEBUG:boto:Token: None 2023-11-02T17:55:21.799 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/_myobj?versionId=null 2023-11-02T17:55:21.800 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:baGvCXL+b1cAROeX8+rpJYeZSM4= 2023-11-02T17:55:21.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:baGvCXL+b1cAROeX8+rpJYeZSM4=', 'Content-Length': '0'} 2023-11-02T17:55:21.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000931557e570283b40-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.846 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.847 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.847 DEBUG:boto:path=/ibtljn-8/?prefix=_myobj 2023-11-02T17:55:21.848 DEBUG:boto:auth_path=/ibtljn-8/?prefix=_myobj 2023-11-02T17:55:21.848 DEBUG:boto:Method: GET 2023-11-02T17:55:21.849 DEBUG:boto:Path: /ibtljn-8/?prefix=_myobj 2023-11-02T17:55:21.849 DEBUG:boto:Data: 2023-11-02T17:55:21.850 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.850 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.851 DEBUG:boto:Params: {} 2023-11-02T17:55:21.851 DEBUG:boto:Token: None 2023-11-02T17:55:21.852 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.852 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8= 2023-11-02T17:55:21.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8=', 'Content-Length': '0'} 2023-11-02T17:55:21.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045a2e0a6fa0bb34c-006543e289-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.859 DEBUG:boto:b'ibtljn-8_myobj1000false_myobj2023-11-02T17:51:59.090Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.859 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.860 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.860 DEBUG:boto:path=/ibtljn-8/?prefix=_myobj 2023-11-02T17:55:21.861 DEBUG:boto:auth_path=/ibtljn-8/?prefix=_myobj 2023-11-02T17:55:21.861 DEBUG:boto:Method: GET 2023-11-02T17:55:21.862 DEBUG:boto:Path: /ibtljn-8/?prefix=_myobj 2023-11-02T17:55:21.862 DEBUG:boto:Data: 2023-11-02T17:55:21.862 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.863 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.863 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.864 DEBUG:boto:Params: {} 2023-11-02T17:55:21.864 DEBUG:boto:Token: None 2023-11-02T17:55:21.865 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.865 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8= 2023-11-02T17:55:21.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8=', 'Content-Length': '0'} 2023-11-02T17:55:21.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aea17e209e3bd735-006543e289-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.871 DEBUG:boto:b'ibtljn-8_myobj1000false_myobj2023-11-02T17:51:59.090Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.872 DEBUG:boto:path=/ibtljn-8/_myobj 2023-11-02T17:55:21.872 DEBUG:boto:auth_path=/ibtljn-8/_myobj 2023-11-02T17:55:21.873 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.873 DEBUG:boto:Path: /ibtljn-8/_myobj 2023-11-02T17:55:21.873 DEBUG:boto:Data: 2023-11-02T17:55:21.874 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.874 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.875 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.875 DEBUG:boto:Params: {} 2023-11-02T17:55:21.876 DEBUG:boto:Token: None 2023-11-02T17:55:21.876 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/_myobj 2023-11-02T17:55:21.876 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RzjtKYf5ccwaY7Xu8DJ0zzwjiP0= 2023-11-02T17:55:21.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RzjtKYf5ccwaY7Xu8DJ0zzwjiP0=', 'Content-Length': '0'} 2023-11-02T17:55:21.880 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edc3a7a3595ae5c8-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.880 DEBUG:boto:path=/ibtljn-8/_myobj 2023-11-02T17:55:21.881 DEBUG:boto:auth_path=/ibtljn-8/_myobj 2023-11-02T17:55:21.881 DEBUG:boto:Method: HEAD 2023-11-02T17:55:21.882 DEBUG:boto:Path: /ibtljn-8/_myobj 2023-11-02T17:55:21.882 DEBUG:boto:Data: 2023-11-02T17:55:21.882 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.883 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.883 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.884 DEBUG:boto:Params: {} 2023-11-02T17:55:21.884 DEBUG:boto:Token: None 2023-11-02T17:55:21.885 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/_myobj 2023-11-02T17:55:21.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RzjtKYf5ccwaY7Xu8DJ0zzwjiP0= 2023-11-02T17:55:21.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RzjtKYf5ccwaY7Xu8DJ0zzwjiP0=', 'Content-Length': '0'} 2023-11-02T17:55:21.888 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000368f71192c577922-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-11-02T17:55:21.889 DEBUG:boto:path=/ibtljn-8/_myobj 2023-11-02T17:55:21.890 DEBUG:boto:auth_path=/ibtljn-8/_myobj 2023-11-02T17:55:21.890 DEBUG:boto:Method: GET 2023-11-02T17:55:21.891 DEBUG:boto:Path: /ibtljn-8/_myobj 2023-11-02T17:55:21.891 DEBUG:boto:Data: 2023-11-02T17:55:21.892 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.892 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.893 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.893 DEBUG:boto:Params: {} 2023-11-02T17:55:21.893 DEBUG:boto:Token: None 2023-11-02T17:55:21.894 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/_myobj 2023-11-02T17:55:21.894 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KmGi7YxY5mgmVhWcQF5YpNL35CI= 2023-11-02T17:55:21.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KmGi7YxY5mgmVhWcQF5YpNL35CI=', 'Content-Length': '0'} 2023-11-02T17:55:21.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea6566d024efeab0-006543e289-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.942 DEBUG:boto:path=/ibtljn-8/_myobj 2023-11-02T17:55:21.943 DEBUG:boto:auth_path=/ibtljn-8/_myobj 2023-11-02T17:55:21.943 DEBUG:boto:Method: GET 2023-11-02T17:55:21.944 DEBUG:boto:Path: /ibtljn-8/_myobj 2023-11-02T17:55:21.944 DEBUG:boto:Data: 2023-11-02T17:55:21.945 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.945 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.946 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.946 DEBUG:boto:Params: {} 2023-11-02T17:55:21.947 DEBUG:boto:Token: None 2023-11-02T17:55:21.947 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/_myobj 2023-11-02T17:55:21.948 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KmGi7YxY5mgmVhWcQF5YpNL35CI= 2023-11-02T17:55:21.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KmGi7YxY5mgmVhWcQF5YpNL35CI=', 'Content-Length': '0'} 2023-11-02T17:55:21.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf0178b217335e46-006543e289-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.952 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.952 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.953 DEBUG:boto:path=/ibtljn-8/?prefix=_myobj 2023-11-02T17:55:21.953 DEBUG:boto:auth_path=/ibtljn-8/?prefix=_myobj 2023-11-02T17:55:21.954 DEBUG:boto:Method: GET 2023-11-02T17:55:21.954 DEBUG:boto:Path: /ibtljn-8/?prefix=_myobj 2023-11-02T17:55:21.954 DEBUG:boto:Data: 2023-11-02T17:55:21.955 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.955 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:21.956 DEBUG:boto:Port: 8000 2023-11-02T17:55:21.956 DEBUG:boto:Params: {} 2023-11-02T17:55:21.957 DEBUG:boto:Token: None 2023-11-02T17:55:21.957 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.958 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8= 2023-11-02T17:55:21.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8=', 'Content-Length': '0'} 2023-11-02T17:55:21.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b193e9d3856b94f9-006543e289-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:21.963 DEBUG:boto:b'ibtljn-8_myobj1000false_myobj2023-11-02T17:51:59.090Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:21.964 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:21.964 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:21.965 DEBUG:boto:path=/ibtljn-8/?prefix=_myobj 2023-11-02T17:55:21.965 DEBUG:boto:auth_path=/ibtljn-8/?prefix=_myobj 2023-11-02T17:55:21.966 DEBUG:boto:Method: GET 2023-11-02T17:55:21.966 DEBUG:boto:Path: /ibtljn-8/?prefix=_myobj 2023-11-02T17:55:21.967 DEBUG:boto:Data: 2023-11-02T17:55:21.967 DEBUG:boto:Headers: {} 2023-11-02T17:55:21.967 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:21.968 DEBUG:boto:Port: 8001 2023-11-02T17:55:21.968 DEBUG:boto:Params: {} 2023-11-02T17:55:21.969 DEBUG:boto:Token: None 2023-11-02T17:55:21.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:21 GMT /ibtljn-8/ 2023-11-02T17:55:21.970 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8= 2023-11-02T17:55:21.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N6hLksHpodt1CB+mtRvcs/NIAD8=', 'Content-Length': '0'} 2023-11-02T17:55:21.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015aefc334fa39947-006543e289-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.022 DEBUG:boto:b'ibtljn-8_myobj1000false_myobj2023-11-02T17:51:59.090Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:22.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-11-02T17:55:22.024 DEBUG:boto:path=/ibtljn-8/myobj 2023-11-02T17:55:22.024 DEBUG:boto:auth_path=/ibtljn-8/myobj 2023-11-02T17:55:22.024 DEBUG:boto:path=/ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.025 DEBUG:boto:auth_path=/ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.025 DEBUG:boto:Method: GET 2023-11-02T17:55:22.026 DEBUG:boto:Path: /ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.026 DEBUG:boto:Data: 2023-11-02T17:55:22.027 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.027 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:22.027 DEBUG:boto:Port: 8000 2023-11-02T17:55:22.028 DEBUG:boto:Params: {} 2023-11-02T17:55:22.028 DEBUG:boto:Token: None 2023-11-02T17:55:22.029 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.029 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f/SpLVPVnRo1Q1hD2JNb8/a0jSg= 2023-11-02T17:55:22.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f/SpLVPVnRo1Q1hD2JNb8/a0jSg=', 'Content-Length': '0'} 2023-11-02T17:55:22.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088742a8063566227-006543e28a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.034 DEBUG:boto:path=/ibtljn-8/myobj 2023-11-02T17:55:22.034 DEBUG:boto:auth_path=/ibtljn-8/myobj 2023-11-02T17:55:22.034 DEBUG:boto:path=/ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.035 DEBUG:boto:auth_path=/ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.035 DEBUG:boto:Method: GET 2023-11-02T17:55:22.036 DEBUG:boto:Path: /ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.036 DEBUG:boto:Data: 2023-11-02T17:55:22.036 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.037 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:22.037 DEBUG:boto:Port: 8001 2023-11-02T17:55:22.038 DEBUG:boto:Params: {} 2023-11-02T17:55:22.038 DEBUG:boto:Token: None 2023-11-02T17:55:22.039 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.039 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f/SpLVPVnRo1Q1hD2JNb8/a0jSg= 2023-11-02T17:55:22.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f/SpLVPVnRo1Q1hD2JNb8/a0jSg=', 'Content-Length': '0'} 2023-11-02T17:55:22.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002886f81e47fbbf89-006543e28a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.044 DEBUG:boto:path=/ibtljn-8/myobj 2023-11-02T17:55:22.044 DEBUG:boto:auth_path=/ibtljn-8/myobj 2023-11-02T17:55:22.044 DEBUG:boto:path=/ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.045 DEBUG:boto:auth_path=/ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.045 DEBUG:boto:Method: HEAD 2023-11-02T17:55:22.046 DEBUG:boto:Path: /ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.046 DEBUG:boto:Data: 2023-11-02T17:55:22.047 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.047 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:22.047 DEBUG:boto:Port: 8000 2023-11-02T17:55:22.048 DEBUG:boto:Params: {} 2023-11-02T17:55:22.048 DEBUG:boto:Token: None 2023-11-02T17:55:22.049 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.049 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G52i9+LcQ3SRgBtb5oDXhOB01+0= 2023-11-02T17:55:22.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G52i9+LcQ3SRgBtb5oDXhOB01+0=', 'Content-Length': '0'} 2023-11-02T17:55:22.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052c36bcbeca7e875-006543e28a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.053 DEBUG:boto:path=/ibtljn-8/myobj 2023-11-02T17:55:22.053 DEBUG:boto:auth_path=/ibtljn-8/myobj 2023-11-02T17:55:22.054 DEBUG:boto:path=/ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.054 DEBUG:boto:auth_path=/ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.055 DEBUG:boto:Method: HEAD 2023-11-02T17:55:22.055 DEBUG:boto:Path: /ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.055 DEBUG:boto:Data: 2023-11-02T17:55:22.056 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.056 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:22.057 DEBUG:boto:Port: 8001 2023-11-02T17:55:22.057 DEBUG:boto:Params: {} 2023-11-02T17:55:22.057 DEBUG:boto:Token: None 2023-11-02T17:55:22.058 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.058 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G52i9+LcQ3SRgBtb5oDXhOB01+0= 2023-11-02T17:55:22.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G52i9+LcQ3SRgBtb5oDXhOB01+0=', 'Content-Length': '0'} 2023-11-02T17:55:22.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d52143eb21624e08-006543e28a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-11-02T17:55:22.063 DEBUG:boto:path=/ibtljn-8/myobj 2023-11-02T17:55:22.063 DEBUG:boto:auth_path=/ibtljn-8/myobj 2023-11-02T17:55:22.064 DEBUG:boto:path=/ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.064 DEBUG:boto:auth_path=/ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.064 DEBUG:boto:Method: GET 2023-11-02T17:55:22.065 DEBUG:boto:Path: /ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.065 DEBUG:boto:Data: 2023-11-02T17:55:22.066 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.066 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:22.067 DEBUG:boto:Port: 8000 2023-11-02T17:55:22.067 DEBUG:boto:Params: {} 2023-11-02T17:55:22.068 DEBUG:boto:Token: None 2023-11-02T17:55:22.068 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.069 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f/SpLVPVnRo1Q1hD2JNb8/a0jSg= 2023-11-02T17:55:22.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f/SpLVPVnRo1Q1hD2JNb8/a0jSg=', 'Content-Length': '0'} 2023-11-02T17:55:22.072 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ef812ef65676fbd-006543e28a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.072 DEBUG:boto:path=/ibtljn-8/myobj 2023-11-02T17:55:22.073 DEBUG:boto:auth_path=/ibtljn-8/myobj 2023-11-02T17:55:22.073 DEBUG:boto:path=/ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.074 DEBUG:boto:auth_path=/ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.074 DEBUG:boto:Method: GET 2023-11-02T17:55:22.075 DEBUG:boto:Path: /ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.075 DEBUG:boto:Data: 2023-11-02T17:55:22.076 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.076 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:22.076 DEBUG:boto:Port: 8001 2023-11-02T17:55:22.077 DEBUG:boto:Params: {} 2023-11-02T17:55:22.077 DEBUG:boto:Token: None 2023-11-02T17:55:22.078 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/myobj?versionId=null 2023-11-02T17:55:22.078 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f/SpLVPVnRo1Q1hD2JNb8/a0jSg= 2023-11-02T17:55:22.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f/SpLVPVnRo1Q1hD2JNb8/a0jSg=', 'Content-Length': '0'} 2023-11-02T17:55:22.082 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026d0d02a12e25540-006543e28a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.082 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:22.083 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:22.083 DEBUG:boto:path=/ibtljn-8/?prefix=myobj 2023-11-02T17:55:22.084 DEBUG:boto:auth_path=/ibtljn-8/?prefix=myobj 2023-11-02T17:55:22.084 DEBUG:boto:Method: GET 2023-11-02T17:55:22.085 DEBUG:boto:Path: /ibtljn-8/?prefix=myobj 2023-11-02T17:55:22.085 DEBUG:boto:Data: 2023-11-02T17:55:22.086 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.086 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:22.086 DEBUG:boto:Port: 8000 2023-11-02T17:55:22.087 DEBUG:boto:Params: {} 2023-11-02T17:55:22.087 DEBUG:boto:Token: None 2023-11-02T17:55:22.088 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/ 2023-11-02T17:55:22.088 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YGrNyFZA8mwCojduR+w5DCxxx4M= 2023-11-02T17:55:22.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YGrNyFZA8mwCojduR+w5DCxxx4M=', 'Content-Length': '0'} 2023-11-02T17:55:22.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac4c5218dc154426-006543e28a-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.094 DEBUG:boto:b'ibtljn-8myobj1000falsemyobj2023-11-02T17:51:59.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:22.095 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:22.096 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:22.096 DEBUG:boto:path=/ibtljn-8/?prefix=myobj 2023-11-02T17:55:22.096 DEBUG:boto:auth_path=/ibtljn-8/?prefix=myobj 2023-11-02T17:55:22.097 DEBUG:boto:Method: GET 2023-11-02T17:55:22.097 DEBUG:boto:Path: /ibtljn-8/?prefix=myobj 2023-11-02T17:55:22.098 DEBUG:boto:Data: 2023-11-02T17:55:22.098 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.099 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:22.099 DEBUG:boto:Port: 8001 2023-11-02T17:55:22.099 DEBUG:boto:Params: {} 2023-11-02T17:55:22.100 DEBUG:boto:Token: None 2023-11-02T17:55:22.100 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/ 2023-11-02T17:55:22.101 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YGrNyFZA8mwCojduR+w5DCxxx4M= 2023-11-02T17:55:22.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YGrNyFZA8mwCojduR+w5DCxxx4M=', 'Content-Length': '0'} 2023-11-02T17:55:22.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028bea889575b7fb0-006543e28a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.107 DEBUG:boto:b'ibtljn-8myobj1000falsemyobj2023-11-02T17:51:59.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:22.107 DEBUG:boto:path=/ibtljn-8/myobj 2023-11-02T17:55:22.108 DEBUG:boto:auth_path=/ibtljn-8/myobj 2023-11-02T17:55:22.108 DEBUG:boto:Method: HEAD 2023-11-02T17:55:22.109 DEBUG:boto:Path: /ibtljn-8/myobj 2023-11-02T17:55:22.109 DEBUG:boto:Data: 2023-11-02T17:55:22.110 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.110 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:22.110 DEBUG:boto:Port: 8000 2023-11-02T17:55:22.111 DEBUG:boto:Params: {} 2023-11-02T17:55:22.111 DEBUG:boto:Token: None 2023-11-02T17:55:22.112 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/myobj 2023-11-02T17:55:22.112 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:y+U5LIG6T9zLUWhy6ng1bla5buw= 2023-11-02T17:55:22.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:y+U5LIG6T9zLUWhy6ng1bla5buw=', 'Content-Length': '0'} 2023-11-02T17:55:22.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075f87a9c409347fd-006543e28a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.116 DEBUG:boto:path=/ibtljn-8/myobj 2023-11-02T17:55:22.116 DEBUG:boto:auth_path=/ibtljn-8/myobj 2023-11-02T17:55:22.117 DEBUG:boto:Method: HEAD 2023-11-02T17:55:22.117 DEBUG:boto:Path: /ibtljn-8/myobj 2023-11-02T17:55:22.118 DEBUG:boto:Data: 2023-11-02T17:55:22.118 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.118 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:22.119 DEBUG:boto:Port: 8001 2023-11-02T17:55:22.119 DEBUG:boto:Params: {} 2023-11-02T17:55:22.120 DEBUG:boto:Token: None 2023-11-02T17:55:22.120 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/myobj 2023-11-02T17:55:22.121 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:y+U5LIG6T9zLUWhy6ng1bla5buw= 2023-11-02T17:55:22.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:y+U5LIG6T9zLUWhy6ng1bla5buw=', 'Content-Length': '0'} 2023-11-02T17:55:22.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe736196984e51c9-006543e28a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-11-02T17:55:22.125 DEBUG:boto:path=/ibtljn-8/myobj 2023-11-02T17:55:22.125 DEBUG:boto:auth_path=/ibtljn-8/myobj 2023-11-02T17:55:22.126 DEBUG:boto:Method: GET 2023-11-02T17:55:22.126 DEBUG:boto:Path: /ibtljn-8/myobj 2023-11-02T17:55:22.127 DEBUG:boto:Data: 2023-11-02T17:55:22.127 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.128 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:22.128 DEBUG:boto:Port: 8000 2023-11-02T17:55:22.129 DEBUG:boto:Params: {} 2023-11-02T17:55:22.129 DEBUG:boto:Token: None 2023-11-02T17:55:22.129 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/myobj 2023-11-02T17:55:22.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MY9p3ZbFOpe0PSInU0rd3aQLXpc= 2023-11-02T17:55:22.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MY9p3ZbFOpe0PSInU0rd3aQLXpc=', 'Content-Length': '0'} 2023-11-02T17:55:22.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f38c18aa1721eb17-006543e28a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.174 DEBUG:boto:path=/ibtljn-8/myobj 2023-11-02T17:55:22.175 DEBUG:boto:auth_path=/ibtljn-8/myobj 2023-11-02T17:55:22.175 DEBUG:boto:Method: GET 2023-11-02T17:55:22.176 DEBUG:boto:Path: /ibtljn-8/myobj 2023-11-02T17:55:22.176 DEBUG:boto:Data: 2023-11-02T17:55:22.176 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.177 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:22.177 DEBUG:boto:Port: 8001 2023-11-02T17:55:22.178 DEBUG:boto:Params: {} 2023-11-02T17:55:22.178 DEBUG:boto:Token: None 2023-11-02T17:55:22.179 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/myobj 2023-11-02T17:55:22.179 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MY9p3ZbFOpe0PSInU0rd3aQLXpc= 2023-11-02T17:55:22.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MY9p3ZbFOpe0PSInU0rd3aQLXpc=', 'Content-Length': '0'} 2023-11-02T17:55:22.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:51:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007dca1b7e487f7756-006543e28a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.183 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:22.184 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:22.184 DEBUG:boto:path=/ibtljn-8/?prefix=myobj 2023-11-02T17:55:22.185 DEBUG:boto:auth_path=/ibtljn-8/?prefix=myobj 2023-11-02T17:55:22.185 DEBUG:boto:Method: GET 2023-11-02T17:55:22.186 DEBUG:boto:Path: /ibtljn-8/?prefix=myobj 2023-11-02T17:55:22.186 DEBUG:boto:Data: 2023-11-02T17:55:22.186 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.187 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:22.187 DEBUG:boto:Port: 8000 2023-11-02T17:55:22.188 DEBUG:boto:Params: {} 2023-11-02T17:55:22.188 DEBUG:boto:Token: None 2023-11-02T17:55:22.189 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/ 2023-11-02T17:55:22.189 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YGrNyFZA8mwCojduR+w5DCxxx4M= 2023-11-02T17:55:22.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YGrNyFZA8mwCojduR+w5DCxxx4M=', 'Content-Length': '0'} 2023-11-02T17:55:22.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dec6f348340529b-006543e28a-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.195 DEBUG:boto:b'ibtljn-8myobj1000falsemyobj2023-11-02T17:51:59.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:22.195 DEBUG:boto:path=/ibtljn-8/ 2023-11-02T17:55:22.196 DEBUG:boto:auth_path=/ibtljn-8/ 2023-11-02T17:55:22.196 DEBUG:boto:path=/ibtljn-8/?prefix=myobj 2023-11-02T17:55:22.197 DEBUG:boto:auth_path=/ibtljn-8/?prefix=myobj 2023-11-02T17:55:22.197 DEBUG:boto:Method: GET 2023-11-02T17:55:22.197 DEBUG:boto:Path: /ibtljn-8/?prefix=myobj 2023-11-02T17:55:22.198 DEBUG:boto:Data: 2023-11-02T17:55:22.198 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.199 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:22.199 DEBUG:boto:Port: 8001 2023-11-02T17:55:22.199 DEBUG:boto:Params: {} 2023-11-02T17:55:22.200 DEBUG:boto:Token: None 2023-11-02T17:55:22.200 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-8/ 2023-11-02T17:55:22.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YGrNyFZA8mwCojduR+w5DCxxx4M= 2023-11-02T17:55:22.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YGrNyFZA8mwCojduR+w5DCxxx4M=', 'Content-Length': '0'} 2023-11-02T17:55:22.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0956d0661ff4e3b-006543e28a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.206 DEBUG:boto:b'ibtljn-8myobj1000falsemyobj2023-11-02T17:51:59.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:22.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-8 zones={a1, a2} 2023-11-02T17:55:22.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2023-11-02T17:55:22.208 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-9 2023-11-02T17:55:22.209 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:22.209 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:22.209 DEBUG:boto:Method: PUT 2023-11-02T17:55:22.210 DEBUG:boto:Path: /ibtljn-9/ 2023-11-02T17:55:22.210 DEBUG:boto:Data: 2023-11-02T17:55:22.211 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.211 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:22.211 DEBUG:boto:Port: 8000 2023-11-02T17:55:22.212 DEBUG:boto:Params: {} 2023-11-02T17:55:22.212 DEBUG:boto:Token: None 2023-11-02T17:55:22.213 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-9/ 2023-11-02T17:55:22.213 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ksznhH/7IFEY3n5JZHRvQcxNusY= 2023-11-02T17:55:22.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ksznhH/7IFEY3n5JZHRvQcxNusY=', 'Content-Length': '0'} 2023-11-02T17:55:22.237 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f24448a3f8ecb60b-006543e28a-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.237 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-10 2023-11-02T17:55:22.238 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:55:22.238 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:55:22.238 DEBUG:boto:Method: PUT 2023-11-02T17:55:22.239 DEBUG:boto:Path: /ibtljn-10/ 2023-11-02T17:55:22.239 DEBUG:boto:Data: 2023-11-02T17:55:22.240 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.240 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:22.240 DEBUG:boto:Port: 8001 2023-11-02T17:55:22.241 DEBUG:boto:Params: {} 2023-11-02T17:55:22.241 DEBUG:boto:Token: None 2023-11-02T17:55:22.242 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-10/ 2023-11-02T17:55:22.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W4g4kLSlMXLx/f4thaaQPy/VlT4= 2023-11-02T17:55:22.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W4g4kLSlMXLx/f4thaaQPy/VlT4=', 'Content-Length': '0'} 2023-11-02T17:55:22.280 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cbdb8d1f46b1098d-006543e28a-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.281 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:22.281 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:22.282 DEBUG:boto:Method: HEAD 2023-11-02T17:55:22.282 DEBUG:boto:Path: /ibtljn-9/ 2023-11-02T17:55:22.283 DEBUG:boto:Data: 2023-11-02T17:55:22.283 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.283 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:22.284 DEBUG:boto:Port: 8000 2023-11-02T17:55:22.284 DEBUG:boto:Params: {} 2023-11-02T17:55:22.285 DEBUG:boto:Token: None 2023-11-02T17:55:22.285 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-9/ 2023-11-02T17:55:22.286 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oBIoVQzeQDaNWZJEzLA7Ues7tD8= 2023-11-02T17:55:22.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oBIoVQzeQDaNWZJEzLA7Ues7tD8=', 'Content-Length': '0'} 2023-11-02T17:55:22.289 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd695706a7ff9318-006543e28a-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.290 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:22.290 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:22.291 DEBUG:boto:Method: PUT 2023-11-02T17:55:22.291 DEBUG:boto:Path: /ibtljn-9/myobj 2023-11-02T17:55:22.292 DEBUG:boto:Data: 2023-11-02T17:55:22.292 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T17:55:22.292 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:22.293 DEBUG:boto:Port: 8000 2023-11-02T17:55:22.293 DEBUG:boto:Params: {} 2023-11-02T17:55:22.294 DEBUG:boto:Token: None 2023-11-02T17:55:22.294 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-9/myobj 2023-11-02T17:55:22.295 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6PIzC0UeuPTxz9qn4PkB3t86ZBI= 2023-11-02T17:55:22.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6PIzC0UeuPTxz9qn4PkB3t86ZBI='} 2023-11-02T17:55:22.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4f4c5f570108fdb-006543e28a-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.308 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:55:22.308 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:55:22.308 DEBUG:boto:Method: HEAD 2023-11-02T17:55:22.309 DEBUG:boto:Path: /ibtljn-10/ 2023-11-02T17:55:22.309 DEBUG:boto:Data: 2023-11-02T17:55:22.310 DEBUG:boto:Headers: {} 2023-11-02T17:55:22.310 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:22.310 DEBUG:boto:Port: 8001 2023-11-02T17:55:22.311 DEBUG:boto:Params: {} 2023-11-02T17:55:22.311 DEBUG:boto:Token: None 2023-11-02T17:55:22.312 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-10/ 2023-11-02T17:55:22.312 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yOHPicAnNLnNLC+/HC+OoJjHB44= 2023-11-02T17:55:22.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yOHPicAnNLnNLC+/HC+OoJjHB44=', 'Content-Length': '0'} 2023-11-02T17:55:22.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000917939711571ce6c-006543e28a-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.317 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:55:22.317 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:55:22.318 DEBUG:boto:Method: PUT 2023-11-02T17:55:22.318 DEBUG:boto:Path: /ibtljn-10/myobj 2023-11-02T17:55:22.319 DEBUG:boto:Data: 2023-11-02T17:55:22.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T17:55:22.319 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:22.320 DEBUG:boto:Port: 8001 2023-11-02T17:55:22.320 DEBUG:boto:Params: {} 2023-11-02T17:55:22.321 DEBUG:boto:Token: None 2023-11-02T17:55:22.321 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 17:55:22 GMT /ibtljn-10/myobj 2023-11-02T17:55:22.322 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z0nsOzujDYxZSNgj9djB9WyjCtY= 2023-11-02T17:55:22.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:55:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z0nsOzujDYxZSNgj9djB9WyjCtY='} 2023-11-02T17:55:22.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002970c06bad0f4f60-006543e28a-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:22.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:55:22.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:55:22.369 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:55:22.407 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:55:22.408 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:55:22.462 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T17:55:22.490 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T17:55:22.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:55:22.491 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:55:22.491 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T17:55:22.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:55:22.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:55:22.491 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:55:22.492 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:55:22.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:55:22.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:55:22.492 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.262469_50.1", 2023-11-02T17:55:22.493 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.262469Z" 2023-11-02T17:55:22.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:55:22.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:55:22.493 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T17:55:22.493 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T17:55:22.494 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:55:22.494 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T17:55:22.495 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947722.262469_50.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:55:22.495 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T17:55:22.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:55:22.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:55:22.497 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:55:22.540 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:55:22.540 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:55:22.595 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:22.589+0000 7f477f3b2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:55:22.597 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:22.589+0000 7f477f3b2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:55:22.623 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:55:22.623 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:55:22.624 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:55:22.624 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:55:22.624 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:55:22.624 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:55:22.625 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:55:22.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:55:22.625 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:55:22.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:55:22.626 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:55:22.626 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:55:22.626 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:55:22.626 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:55:22.626 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:55:22.627 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:55:22.627 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:55:22.627 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:55:22.627 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:55:22.628 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:55:22.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:55:22.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:55:22.628 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:55:22.629 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:55:22.629 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:55:22.629 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:55:22.629 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:55:22.630 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:55:22.630 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:55:22.630 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:55:22.630 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:55:22.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:55:22.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:55:22.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:55:22.631 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:55:22.631 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:55:22.632 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:55:22.632 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.262469_50.1", 2023-11-02T17:55:22.632 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:55:22.632 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:55:22.633 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:55:22.633 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.262469Z", 2023-11-02T17:55:22.633 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:55:22.633 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:55:22.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:55:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:55:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:55:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:55:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:55:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T17:55:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:55:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:55:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:55:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T17:55:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:55:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:55:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:55:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:55:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:55:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:55:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:55:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:55:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:55:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:55:22Z" 2023-11-02T17:55:22.638 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:55:22.639 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698947722.262469_50.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.262469Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T17:55:22Z" } 2023-11-02T17:55:22.640 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698947722.262469_50.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.262469Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T17:55:22.640 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947722.262469_50.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:55:22.641 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947722.262469_50.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:55:22.641 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T17:55:22.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:55:22.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:55:22.642 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:55:22.735 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:55:22.736 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:55:22.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:22.793+0000 7f6fff68bf40 1 waiting to reach incremental sync.. 2023-11-02T17:55:27.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:27.793+0000 7f6fff68bf40 1 waiting to reach incremental sync.. 2023-11-02T17:55:32.801 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:32.793+0000 7f6fff68bf40 1 waiting to reach incremental sync.. 2023-11-02T17:55:37.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:37.793+0000 7f6fff68bf40 1 waiting to reach incremental sync.. 2023-11-02T17:55:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:42.797+0000 7f6fff68bf40 1 bucket sync caught up with source: 2023-11-02T17:55:42.806 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.3.6, , , , , , , , , ] 2023-11-02T17:55:42.806 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.3.6, , , , , , , , , ] 2023-11-02T17:55:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:42.797+0000 7f6fff68bf40 0 bucket checkpoint complete 2023-11-02T17:55:42.834 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-9 zones={a2, a1} 2023-11-02T17:55:42.835 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:42.835 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:42.836 DEBUG:boto:Method: HEAD 2023-11-02T17:55:42.836 DEBUG:boto:Path: /ibtljn-9/ 2023-11-02T17:55:42.837 DEBUG:boto:Data: 2023-11-02T17:55:42.838 DEBUG:boto:Headers: {} 2023-11-02T17:55:42.838 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:42.838 DEBUG:boto:Port: 8001 2023-11-02T17:55:42.839 DEBUG:boto:Params: {} 2023-11-02T17:55:42.840 DEBUG:boto:Token: None 2023-11-02T17:55:42.840 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:42 GMT /ibtljn-9/ 2023-11-02T17:55:42.841 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mX4XgmI8s1IZckJC3XZlWy91GSI= 2023-11-02T17:55:42.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mX4XgmI8s1IZckJC3XZlWy91GSI=', 'Content-Length': '0'} 2023-11-02T17:55:42.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b67a6bcf64b5494-006543e29e-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:55:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:42.846 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:42.847 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:42.847 DEBUG:boto:Method: HEAD 2023-11-02T17:55:42.848 DEBUG:boto:Path: /ibtljn-9/ 2023-11-02T17:55:42.848 DEBUG:boto:Data: 2023-11-02T17:55:42.848 DEBUG:boto:Headers: {} 2023-11-02T17:55:42.849 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:42.849 DEBUG:boto:Port: 8000 2023-11-02T17:55:42.850 DEBUG:boto:Params: {} 2023-11-02T17:55:42.850 DEBUG:boto:Token: None 2023-11-02T17:55:42.851 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:42 GMT /ibtljn-9/ 2023-11-02T17:55:42.851 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mX4XgmI8s1IZckJC3XZlWy91GSI= 2023-11-02T17:55:42.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mX4XgmI8s1IZckJC3XZlWy91GSI=', 'Content-Length': '0'} 2023-11-02T17:55:42.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031cf50427bfff11c-006543e29e-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:55:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:42.857 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T17:55:42.857 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:42.857 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:42.858 DEBUG:boto:path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:55:42.858 DEBUG:boto:auth_path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:55:42.859 DEBUG:boto:Method: GET 2023-11-02T17:55:42.859 DEBUG:boto:Path: /ibtljn-9/?versions&max-keys=999 2023-11-02T17:55:42.860 DEBUG:boto:Data: 2023-11-02T17:55:42.860 DEBUG:boto:Headers: {} 2023-11-02T17:55:42.860 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:42.861 DEBUG:boto:Port: 8001 2023-11-02T17:55:42.861 DEBUG:boto:Params: {} 2023-11-02T17:55:42.862 DEBUG:boto:Token: None 2023-11-02T17:55:42.862 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:42 GMT /ibtljn-9/?versions 2023-11-02T17:55:42.863 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iEAi56RX7t9+STa/GMRVCGuqm30= 2023-11-02T17:55:42.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iEAi56RX7t9+STa/GMRVCGuqm30=', 'Content-Length': '0'} 2023-11-02T17:55:42.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016c7d25a566776ad-006543e29e-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:42.868 DEBUG:boto:b'ibtljn-9999falsemyobjnulltrue2023-11-02T17:55:22.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:42.869 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-11-02T17:55:42.870 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T17:55:42.871 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:42.871 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:42.872 DEBUG:boto:path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:55:42.872 DEBUG:boto:auth_path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:55:42.872 DEBUG:boto:Method: GET 2023-11-02T17:55:42.873 DEBUG:boto:Path: /ibtljn-9/?versions&max-keys=999 2023-11-02T17:55:42.873 DEBUG:boto:Data: 2023-11-02T17:55:42.874 DEBUG:boto:Headers: {} 2023-11-02T17:55:42.874 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:42.875 DEBUG:boto:Port: 8000 2023-11-02T17:55:42.875 DEBUG:boto:Params: {} 2023-11-02T17:55:42.876 DEBUG:boto:Token: None 2023-11-02T17:55:42.876 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:42 GMT /ibtljn-9/?versions 2023-11-02T17:55:42.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iEAi56RX7t9+STa/GMRVCGuqm30= 2023-11-02T17:55:42.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iEAi56RX7t9+STa/GMRVCGuqm30=', 'Content-Length': '0'} 2023-11-02T17:55:42.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e11898daa76d422-006543e29e-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:42.922 DEBUG:boto:b'ibtljn-9999falsemyobjnulltrue2023-11-02T17:55:22.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:42.923 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-11-02T17:55:42.924 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:42.924 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:42.925 DEBUG:boto:path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:55:42.925 DEBUG:boto:auth_path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:55:42.926 DEBUG:boto:Method: GET 2023-11-02T17:55:42.926 DEBUG:boto:Path: /ibtljn-9/?versions&max-keys=999 2023-11-02T17:55:42.927 DEBUG:boto:Data: 2023-11-02T17:55:42.927 DEBUG:boto:Headers: {} 2023-11-02T17:55:42.928 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:42.929 DEBUG:boto:Port: 8001 2023-11-02T17:55:42.929 DEBUG:boto:Params: {} 2023-11-02T17:55:42.930 DEBUG:boto:Token: None 2023-11-02T17:55:42.931 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:42 GMT /ibtljn-9/?versions 2023-11-02T17:55:42.932 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iEAi56RX7t9+STa/GMRVCGuqm30= 2023-11-02T17:55:42.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iEAi56RX7t9+STa/GMRVCGuqm30=', 'Content-Length': '0'} 2023-11-02T17:55:42.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000081b530ed366b7a7-006543e29e-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:42.938 DEBUG:boto:b'ibtljn-9999falsemyobjnulltrue2023-11-02T17:55:22.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:42.939 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:42.939 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:42.940 DEBUG:boto:path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:55:42.940 DEBUG:boto:auth_path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:55:42.940 DEBUG:boto:Method: GET 2023-11-02T17:55:42.941 DEBUG:boto:Path: /ibtljn-9/?versions&max-keys=999 2023-11-02T17:55:42.941 DEBUG:boto:Data: 2023-11-02T17:55:42.942 DEBUG:boto:Headers: {} 2023-11-02T17:55:42.943 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:42.943 DEBUG:boto:Port: 8000 2023-11-02T17:55:42.944 DEBUG:boto:Params: {} 2023-11-02T17:55:42.944 DEBUG:boto:Token: None 2023-11-02T17:55:42.945 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:42 GMT /ibtljn-9/?versions 2023-11-02T17:55:42.945 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iEAi56RX7t9+STa/GMRVCGuqm30= 2023-11-02T17:55:42.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iEAi56RX7t9+STa/GMRVCGuqm30=', 'Content-Length': '0'} 2023-11-02T17:55:42.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ae8df20f180ccfa-006543e29e-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:42.951 DEBUG:boto:b'ibtljn-9999falsemyobjnulltrue2023-11-02T17:55:22.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:42.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-11-02T17:55:42.952 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:42.953 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:42.953 DEBUG:boto:path=/ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.954 DEBUG:boto:auth_path=/ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.954 DEBUG:boto:Method: GET 2023-11-02T17:55:42.954 DEBUG:boto:Path: /ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.955 DEBUG:boto:Data: 2023-11-02T17:55:42.955 DEBUG:boto:Headers: {} 2023-11-02T17:55:42.956 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:42.956 DEBUG:boto:Port: 8001 2023-11-02T17:55:42.957 DEBUG:boto:Params: {} 2023-11-02T17:55:42.957 DEBUG:boto:Token: None 2023-11-02T17:55:42.958 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:42 GMT /ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.958 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hVm/ZdyMAOQWXZWi25LL0rfLdq0= 2023-11-02T17:55:42.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hVm/ZdyMAOQWXZWi25LL0rfLdq0=', 'Content-Length': '0'} 2023-11-02T17:55:42.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004af4def7f8e2af2b-006543e29e-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:42.963 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:42.963 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:42.963 DEBUG:boto:path=/ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.964 DEBUG:boto:auth_path=/ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.964 DEBUG:boto:Method: GET 2023-11-02T17:55:42.965 DEBUG:boto:Path: /ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.965 DEBUG:boto:Data: 2023-11-02T17:55:42.966 DEBUG:boto:Headers: {} 2023-11-02T17:55:42.966 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:42.967 DEBUG:boto:Port: 8000 2023-11-02T17:55:42.967 DEBUG:boto:Params: {} 2023-11-02T17:55:42.968 DEBUG:boto:Token: None 2023-11-02T17:55:42.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:42 GMT /ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hVm/ZdyMAOQWXZWi25LL0rfLdq0= 2023-11-02T17:55:42.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hVm/ZdyMAOQWXZWi25LL0rfLdq0=', 'Content-Length': '0'} 2023-11-02T17:55:42.974 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005749c7f0fca57ecb-006543e29e-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:42.976 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:42.977 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:42.977 DEBUG:boto:path=/ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.978 DEBUG:boto:auth_path=/ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.978 DEBUG:boto:Method: HEAD 2023-11-02T17:55:42.979 DEBUG:boto:Path: /ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.979 DEBUG:boto:Data: 2023-11-02T17:55:42.979 DEBUG:boto:Headers: {} 2023-11-02T17:55:42.980 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:42.980 DEBUG:boto:Port: 8001 2023-11-02T17:55:42.981 DEBUG:boto:Params: {} 2023-11-02T17:55:42.981 DEBUG:boto:Token: None 2023-11-02T17:55:42.982 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:42 GMT /ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.982 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RsBUTmr/RS7y4B1A5vDsntyHEps= 2023-11-02T17:55:42.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RsBUTmr/RS7y4B1A5vDsntyHEps=', 'Content-Length': '0'} 2023-11-02T17:55:42.986 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042a10b16ab3b02f1-006543e29e-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:42.987 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:42.987 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:42.987 DEBUG:boto:path=/ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.988 DEBUG:boto:auth_path=/ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.988 DEBUG:boto:Method: HEAD 2023-11-02T17:55:42.989 DEBUG:boto:Path: /ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.989 DEBUG:boto:Data: 2023-11-02T17:55:42.989 DEBUG:boto:Headers: {} 2023-11-02T17:55:42.990 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:42.990 DEBUG:boto:Port: 8000 2023-11-02T17:55:42.991 DEBUG:boto:Params: {} 2023-11-02T17:55:42.991 DEBUG:boto:Token: None 2023-11-02T17:55:42.992 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:42 GMT /ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.992 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RsBUTmr/RS7y4B1A5vDsntyHEps= 2023-11-02T17:55:42.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RsBUTmr/RS7y4B1A5vDsntyHEps=', 'Content-Length': '0'} 2023-11-02T17:55:42.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f884d78150b160fc-006543e29e-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:42.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-11-02T17:55:42.997 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:42.997 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:42.997 DEBUG:boto:path=/ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.998 DEBUG:boto:auth_path=/ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.998 DEBUG:boto:Method: GET 2023-11-02T17:55:42.999 DEBUG:boto:Path: /ibtljn-9/myobj?versionId=null 2023-11-02T17:55:42.999 DEBUG:boto:Data: 2023-11-02T17:55:43.000 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.000 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.001 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.001 DEBUG:boto:Params: {} 2023-11-02T17:55:43.002 DEBUG:boto:Token: None 2023-11-02T17:55:43.002 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/myobj?versionId=null 2023-11-02T17:55:43.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SbhUU1PvuVlNjRZw++3XXARv7w0= 2023-11-02T17:55:43.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SbhUU1PvuVlNjRZw++3XXARv7w0=', 'Content-Length': '0'} 2023-11-02T17:55:43.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfba2a16cf5d5536-006543e29e-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.007 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:43.008 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:43.009 DEBUG:boto:path=/ibtljn-9/myobj?versionId=null 2023-11-02T17:55:43.010 DEBUG:boto:auth_path=/ibtljn-9/myobj?versionId=null 2023-11-02T17:55:43.010 DEBUG:boto:Method: GET 2023-11-02T17:55:43.011 DEBUG:boto:Path: /ibtljn-9/myobj?versionId=null 2023-11-02T17:55:43.011 DEBUG:boto:Data: 2023-11-02T17:55:43.012 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.012 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.013 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.013 DEBUG:boto:Params: {} 2023-11-02T17:55:43.014 DEBUG:boto:Token: None 2023-11-02T17:55:43.014 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/myobj?versionId=null 2023-11-02T17:55:43.015 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SbhUU1PvuVlNjRZw++3XXARv7w0= 2023-11-02T17:55:43.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SbhUU1PvuVlNjRZw++3XXARv7w0=', 'Content-Length': '0'} 2023-11-02T17:55:43.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087b26ed1084b5225-006543e29f-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.066 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:43.067 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:43.067 DEBUG:boto:path=/ibtljn-9/?prefix=myobj 2023-11-02T17:55:43.068 DEBUG:boto:auth_path=/ibtljn-9/?prefix=myobj 2023-11-02T17:55:43.068 DEBUG:boto:Method: GET 2023-11-02T17:55:43.069 DEBUG:boto:Path: /ibtljn-9/?prefix=myobj 2023-11-02T17:55:43.069 DEBUG:boto:Data: 2023-11-02T17:55:43.070 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.071 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.071 DEBUG:boto:Params: {} 2023-11-02T17:55:43.072 DEBUG:boto:Token: None 2023-11-02T17:55:43.072 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/ 2023-11-02T17:55:43.073 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oCPuFyUlRHa1lK7NiCHhRv1onrs= 2023-11-02T17:55:43.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oCPuFyUlRHa1lK7NiCHhRv1onrs=', 'Content-Length': '0'} 2023-11-02T17:55:43.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0082f8992b3dbff-006543e29f-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.079 DEBUG:boto:b'ibtljn-9myobj1000falsemyobj2023-11-02T17:55:22.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:43.080 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:43.081 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:43.081 DEBUG:boto:path=/ibtljn-9/?prefix=myobj 2023-11-02T17:55:43.082 DEBUG:boto:auth_path=/ibtljn-9/?prefix=myobj 2023-11-02T17:55:43.082 DEBUG:boto:Method: GET 2023-11-02T17:55:43.083 DEBUG:boto:Path: /ibtljn-9/?prefix=myobj 2023-11-02T17:55:43.083 DEBUG:boto:Data: 2023-11-02T17:55:43.083 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.084 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.084 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.085 DEBUG:boto:Params: {} 2023-11-02T17:55:43.085 DEBUG:boto:Token: None 2023-11-02T17:55:43.086 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/ 2023-11-02T17:55:43.086 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oCPuFyUlRHa1lK7NiCHhRv1onrs= 2023-11-02T17:55:43.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oCPuFyUlRHa1lK7NiCHhRv1onrs=', 'Content-Length': '0'} 2023-11-02T17:55:43.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de3226396411ed86-006543e29f-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.094 DEBUG:boto:b'ibtljn-9myobj1000falsemyobj2023-11-02T17:55:22.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:43.094 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:43.095 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:43.095 DEBUG:boto:Method: HEAD 2023-11-02T17:55:43.096 DEBUG:boto:Path: /ibtljn-9/myobj 2023-11-02T17:55:43.096 DEBUG:boto:Data: 2023-11-02T17:55:43.097 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.097 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.098 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.098 DEBUG:boto:Params: {} 2023-11-02T17:55:43.098 DEBUG:boto:Token: None 2023-11-02T17:55:43.099 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/myobj 2023-11-02T17:55:43.100 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8ejCC2oYAghl7VTNfTM39UKi1g8= 2023-11-02T17:55:43.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8ejCC2oYAghl7VTNfTM39UKi1g8=', 'Content-Length': '0'} 2023-11-02T17:55:43.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1511b11424e8aa1-006543e29f-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.104 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:43.104 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:43.105 DEBUG:boto:Method: HEAD 2023-11-02T17:55:43.105 DEBUG:boto:Path: /ibtljn-9/myobj 2023-11-02T17:55:43.105 DEBUG:boto:Data: 2023-11-02T17:55:43.106 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.107 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.107 DEBUG:boto:Params: {} 2023-11-02T17:55:43.108 DEBUG:boto:Token: None 2023-11-02T17:55:43.108 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/myobj 2023-11-02T17:55:43.109 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8ejCC2oYAghl7VTNfTM39UKi1g8= 2023-11-02T17:55:43.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8ejCC2oYAghl7VTNfTM39UKi1g8=', 'Content-Length': '0'} 2023-11-02T17:55:43.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009beb245f6597cd94-006543e29f-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-11-02T17:55:43.113 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:43.114 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:43.114 DEBUG:boto:Method: GET 2023-11-02T17:55:43.115 DEBUG:boto:Path: /ibtljn-9/myobj 2023-11-02T17:55:43.115 DEBUG:boto:Data: 2023-11-02T17:55:43.116 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.116 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.117 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.117 DEBUG:boto:Params: {} 2023-11-02T17:55:43.117 DEBUG:boto:Token: None 2023-11-02T17:55:43.118 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/myobj 2023-11-02T17:55:43.119 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:srcLSRkRo5KGKRL1w7DK2fvDIWU= 2023-11-02T17:55:43.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:srcLSRkRo5KGKRL1w7DK2fvDIWU=', 'Content-Length': '0'} 2023-11-02T17:55:43.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001546a5114ce40308-006543e29f-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.166 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:43.167 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:43.167 DEBUG:boto:Method: GET 2023-11-02T17:55:43.168 DEBUG:boto:Path: /ibtljn-9/myobj 2023-11-02T17:55:43.168 DEBUG:boto:Data: 2023-11-02T17:55:43.169 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.169 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.170 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.170 DEBUG:boto:Params: {} 2023-11-02T17:55:43.171 DEBUG:boto:Token: None 2023-11-02T17:55:43.171 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/myobj 2023-11-02T17:55:43.172 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:srcLSRkRo5KGKRL1w7DK2fvDIWU= 2023-11-02T17:55:43.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:srcLSRkRo5KGKRL1w7DK2fvDIWU=', 'Content-Length': '0'} 2023-11-02T17:55:43.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f41f0d8274e77bc3-006543e29f-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.176 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:43.176 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:43.177 DEBUG:boto:path=/ibtljn-9/?prefix=myobj 2023-11-02T17:55:43.177 DEBUG:boto:auth_path=/ibtljn-9/?prefix=myobj 2023-11-02T17:55:43.178 DEBUG:boto:Method: GET 2023-11-02T17:55:43.178 DEBUG:boto:Path: /ibtljn-9/?prefix=myobj 2023-11-02T17:55:43.179 DEBUG:boto:Data: 2023-11-02T17:55:43.179 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.180 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.180 DEBUG:boto:Params: {} 2023-11-02T17:55:43.181 DEBUG:boto:Token: None 2023-11-02T17:55:43.181 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/ 2023-11-02T17:55:43.182 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oCPuFyUlRHa1lK7NiCHhRv1onrs= 2023-11-02T17:55:43.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oCPuFyUlRHa1lK7NiCHhRv1onrs=', 'Content-Length': '0'} 2023-11-02T17:55:43.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ce404ace84f3679-006543e29f-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.187 DEBUG:boto:b'ibtljn-9myobj1000falsemyobj2023-11-02T17:55:22.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:43.188 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:43.188 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:43.189 DEBUG:boto:path=/ibtljn-9/?prefix=myobj 2023-11-02T17:55:43.189 DEBUG:boto:auth_path=/ibtljn-9/?prefix=myobj 2023-11-02T17:55:43.190 DEBUG:boto:Method: GET 2023-11-02T17:55:43.190 DEBUG:boto:Path: /ibtljn-9/?prefix=myobj 2023-11-02T17:55:43.191 DEBUG:boto:Data: 2023-11-02T17:55:43.191 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.192 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.192 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.193 DEBUG:boto:Params: {} 2023-11-02T17:55:43.193 DEBUG:boto:Token: None 2023-11-02T17:55:43.194 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/ 2023-11-02T17:55:43.195 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oCPuFyUlRHa1lK7NiCHhRv1onrs= 2023-11-02T17:55:43.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oCPuFyUlRHa1lK7NiCHhRv1onrs=', 'Content-Length': '0'} 2023-11-02T17:55:43.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cbe16d194a1746e-006543e29f-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.246 DEBUG:boto:b'ibtljn-9myobj1000falsemyobj2023-11-02T17:55:22.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:43.247 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-9 zones={a2, a1} 2023-11-02T17:55:43.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:55:43.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:55:43.249 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:55:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:55:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:55:43.371 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:43.365+0000 7fe79a353f40 1 bucket sync caught up with source: 2023-11-02T17:55:43.372 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-11-02T17:55:43.372 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.11.6, , , , , , , , , ] 2023-11-02T17:55:43.372 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:43.365+0000 7fe79a353f40 0 bucket checkpoint complete 2023-11-02T17:55:43.397 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-10 zones={a1, a2} 2023-11-02T17:55:43.397 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:55:43.398 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:55:43.398 DEBUG:boto:Method: HEAD 2023-11-02T17:55:43.399 DEBUG:boto:Path: /ibtljn-10/ 2023-11-02T17:55:43.399 DEBUG:boto:Data: 2023-11-02T17:55:43.400 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.400 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.401 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.401 DEBUG:boto:Params: {} 2023-11-02T17:55:43.402 DEBUG:boto:Token: None 2023-11-02T17:55:43.402 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/ 2023-11-02T17:55:43.403 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w2N4EhID1fpnE0MNLklpByb0nyA= 2023-11-02T17:55:43.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w2N4EhID1fpnE0MNLklpByb0nyA=', 'Content-Length': '0'} 2023-11-02T17:55:43.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bd124473006885b-006543e29f-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.408 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:55:43.409 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:55:43.410 DEBUG:boto:Method: HEAD 2023-11-02T17:55:43.410 DEBUG:boto:Path: /ibtljn-10/ 2023-11-02T17:55:43.411 DEBUG:boto:Data: 2023-11-02T17:55:43.412 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.412 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.413 DEBUG:boto:Params: {} 2023-11-02T17:55:43.413 DEBUG:boto:Token: None 2023-11-02T17:55:43.414 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/ 2023-11-02T17:55:43.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w2N4EhID1fpnE0MNLklpByb0nyA= 2023-11-02T17:55:43.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w2N4EhID1fpnE0MNLklpByb0nyA=', 'Content-Length': '0'} 2023-11-02T17:55:43.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000daac716cf3e7014c-006543e29f-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.419 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T17:55:43.420 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:55:43.420 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:55:43.421 DEBUG:boto:path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:55:43.421 DEBUG:boto:auth_path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:55:43.422 DEBUG:boto:Method: GET 2023-11-02T17:55:43.422 DEBUG:boto:Path: /ibtljn-10/?versions&max-keys=999 2023-11-02T17:55:43.422 DEBUG:boto:Data: 2023-11-02T17:55:43.423 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.423 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.424 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.424 DEBUG:boto:Params: {} 2023-11-02T17:55:43.425 DEBUG:boto:Token: None 2023-11-02T17:55:43.425 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/?versions 2023-11-02T17:55:43.426 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BBmKnbUkq0kahqj/X0IpuSshfmc= 2023-11-02T17:55:43.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BBmKnbUkq0kahqj/X0IpuSshfmc=', 'Content-Length': '0'} 2023-11-02T17:55:43.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d794dd6bfe59b2d6-006543e29f-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.432 DEBUG:boto:b'ibtljn-10999falsemyobjnulltrue2023-11-02T17:55:22.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:43.433 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-11-02T17:55:43.433 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T17:55:43.434 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:55:43.434 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:55:43.434 DEBUG:boto:path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:55:43.435 DEBUG:boto:auth_path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:55:43.435 DEBUG:boto:Method: GET 2023-11-02T17:55:43.436 DEBUG:boto:Path: /ibtljn-10/?versions&max-keys=999 2023-11-02T17:55:43.436 DEBUG:boto:Data: 2023-11-02T17:55:43.436 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.438 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.438 DEBUG:boto:Params: {} 2023-11-02T17:55:43.439 DEBUG:boto:Token: None 2023-11-02T17:55:43.440 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/?versions 2023-11-02T17:55:43.440 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BBmKnbUkq0kahqj/X0IpuSshfmc= 2023-11-02T17:55:43.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BBmKnbUkq0kahqj/X0IpuSshfmc=', 'Content-Length': '0'} 2023-11-02T17:55:43.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0a1aab5e9198f0a-006543e29f-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.446 DEBUG:boto:b'ibtljn-10999falsemyobjnulltrue2023-11-02T17:55:22.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:43.446 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-11-02T17:55:43.447 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:55:43.447 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:55:43.448 DEBUG:boto:path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:55:43.449 DEBUG:boto:auth_path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:55:43.449 DEBUG:boto:Method: GET 2023-11-02T17:55:43.449 DEBUG:boto:Path: /ibtljn-10/?versions&max-keys=999 2023-11-02T17:55:43.450 DEBUG:boto:Data: 2023-11-02T17:55:43.451 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.452 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.453 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.453 DEBUG:boto:Params: {} 2023-11-02T17:55:43.454 DEBUG:boto:Token: None 2023-11-02T17:55:43.455 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/?versions 2023-11-02T17:55:43.455 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BBmKnbUkq0kahqj/X0IpuSshfmc= 2023-11-02T17:55:43.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BBmKnbUkq0kahqj/X0IpuSshfmc=', 'Content-Length': '0'} 2023-11-02T17:55:43.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039cdf30f18e9ef92-006543e29f-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.462 DEBUG:boto:b'ibtljn-10999falsemyobjnulltrue2023-11-02T17:55:22.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:43.463 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:55:43.463 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:55:43.464 DEBUG:boto:path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:55:43.464 DEBUG:boto:auth_path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:55:43.464 DEBUG:boto:Method: GET 2023-11-02T17:55:43.465 DEBUG:boto:Path: /ibtljn-10/?versions&max-keys=999 2023-11-02T17:55:43.465 DEBUG:boto:Data: 2023-11-02T17:55:43.466 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.466 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.467 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.467 DEBUG:boto:Params: {} 2023-11-02T17:55:43.468 DEBUG:boto:Token: None 2023-11-02T17:55:43.468 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/?versions 2023-11-02T17:55:43.469 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BBmKnbUkq0kahqj/X0IpuSshfmc= 2023-11-02T17:55:43.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BBmKnbUkq0kahqj/X0IpuSshfmc=', 'Content-Length': '0'} 2023-11-02T17:55:43.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce3cd7ed4018158a-006543e29f-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.514 DEBUG:boto:b'ibtljn-10999falsemyobjnulltrue2023-11-02T17:55:22.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:43.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-11-02T17:55:43.516 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:55:43.516 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:55:43.516 DEBUG:boto:path=/ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.517 DEBUG:boto:auth_path=/ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.517 DEBUG:boto:Method: GET 2023-11-02T17:55:43.518 DEBUG:boto:Path: /ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.518 DEBUG:boto:Data: 2023-11-02T17:55:43.519 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.520 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.520 DEBUG:boto:Params: {} 2023-11-02T17:55:43.521 DEBUG:boto:Token: None 2023-11-02T17:55:43.521 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.522 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oE6qN42V3BWlbrVDmMa9cBete38= 2023-11-02T17:55:43.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oE6qN42V3BWlbrVDmMa9cBete38=', 'Content-Length': '0'} 2023-11-02T17:55:43.526 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc9531d91ba6281f-006543e29f-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.527 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:55:43.528 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:55:43.528 DEBUG:boto:path=/ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.529 DEBUG:boto:auth_path=/ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.530 DEBUG:boto:Method: GET 2023-11-02T17:55:43.530 DEBUG:boto:Path: /ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.531 DEBUG:boto:Data: 2023-11-02T17:55:43.531 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.532 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.532 DEBUG:boto:Params: {} 2023-11-02T17:55:43.533 DEBUG:boto:Token: None 2023-11-02T17:55:43.534 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.534 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oE6qN42V3BWlbrVDmMa9cBete38= 2023-11-02T17:55:43.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oE6qN42V3BWlbrVDmMa9cBete38=', 'Content-Length': '0'} 2023-11-02T17:55:43.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfe8831e3451f87a-006543e29f-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.538 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:55:43.538 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:55:43.539 DEBUG:boto:path=/ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.540 DEBUG:boto:auth_path=/ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.541 DEBUG:boto:Method: HEAD 2023-11-02T17:55:43.541 DEBUG:boto:Path: /ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.542 DEBUG:boto:Data: 2023-11-02T17:55:43.542 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.542 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.543 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.543 DEBUG:boto:Params: {} 2023-11-02T17:55:43.544 DEBUG:boto:Token: None 2023-11-02T17:55:43.545 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.545 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a/ZNsYGviXnszmExLqYt9QMB/2U= 2023-11-02T17:55:43.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a/ZNsYGviXnszmExLqYt9QMB/2U=', 'Content-Length': '0'} 2023-11-02T17:55:43.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7f623aaa16aa0e9-006543e29f-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.549 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:55:43.550 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:55:43.551 DEBUG:boto:path=/ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.551 DEBUG:boto:auth_path=/ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.552 DEBUG:boto:Method: HEAD 2023-11-02T17:55:43.552 DEBUG:boto:Path: /ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.553 DEBUG:boto:Data: 2023-11-02T17:55:43.553 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.555 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.556 DEBUG:boto:Params: {} 2023-11-02T17:55:43.556 DEBUG:boto:Token: None 2023-11-02T17:55:43.557 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.557 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a/ZNsYGviXnszmExLqYt9QMB/2U= 2023-11-02T17:55:43.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a/ZNsYGviXnszmExLqYt9QMB/2U=', 'Content-Length': '0'} 2023-11-02T17:55:43.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000635867edf4be82f1-006543e29f-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-11-02T17:55:43.562 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:55:43.562 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:55:43.563 DEBUG:boto:path=/ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.563 DEBUG:boto:auth_path=/ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.564 DEBUG:boto:Method: GET 2023-11-02T17:55:43.564 DEBUG:boto:Path: /ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.564 DEBUG:boto:Data: 2023-11-02T17:55:43.565 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.565 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.566 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.566 DEBUG:boto:Params: {} 2023-11-02T17:55:43.567 DEBUG:boto:Token: None 2023-11-02T17:55:43.567 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.568 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oE6qN42V3BWlbrVDmMa9cBete38= 2023-11-02T17:55:43.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oE6qN42V3BWlbrVDmMa9cBete38=', 'Content-Length': '0'} 2023-11-02T17:55:43.572 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae2ee7a95a1d735a-006543e29f-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.572 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:55:43.573 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:55:43.573 DEBUG:boto:path=/ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.574 DEBUG:boto:auth_path=/ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.574 DEBUG:boto:Method: GET 2023-11-02T17:55:43.575 DEBUG:boto:Path: /ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.575 DEBUG:boto:Data: 2023-11-02T17:55:43.575 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.576 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.576 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.577 DEBUG:boto:Params: {} 2023-11-02T17:55:43.577 DEBUG:boto:Token: None 2023-11-02T17:55:43.578 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/myobj?versionId=null 2023-11-02T17:55:43.578 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oE6qN42V3BWlbrVDmMa9cBete38= 2023-11-02T17:55:43.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oE6qN42V3BWlbrVDmMa9cBete38=', 'Content-Length': '0'} 2023-11-02T17:55:43.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b681b1d69f80f3e5-006543e29f-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.582 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:55:43.583 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:55:43.583 DEBUG:boto:path=/ibtljn-10/?prefix=myobj 2023-11-02T17:55:43.584 DEBUG:boto:auth_path=/ibtljn-10/?prefix=myobj 2023-11-02T17:55:43.584 DEBUG:boto:Method: GET 2023-11-02T17:55:43.584 DEBUG:boto:Path: /ibtljn-10/?prefix=myobj 2023-11-02T17:55:43.585 DEBUG:boto:Data: 2023-11-02T17:55:43.585 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.586 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.586 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.587 DEBUG:boto:Params: {} 2023-11-02T17:55:43.589 DEBUG:boto:Token: None 2023-11-02T17:55:43.590 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/ 2023-11-02T17:55:43.590 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I4EXaj62060Gme9zVp/tBMBEjmM= 2023-11-02T17:55:43.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I4EXaj62060Gme9zVp/tBMBEjmM=', 'Content-Length': '0'} 2023-11-02T17:55:43.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008be03624aa86af11-006543e29f-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.642 DEBUG:boto:b'ibtljn-10myobj1000falsemyobj2023-11-02T17:55:22.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:43.643 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:55:43.644 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:55:43.644 DEBUG:boto:path=/ibtljn-10/?prefix=myobj 2023-11-02T17:55:43.645 DEBUG:boto:auth_path=/ibtljn-10/?prefix=myobj 2023-11-02T17:55:43.645 DEBUG:boto:Method: GET 2023-11-02T17:55:43.646 DEBUG:boto:Path: /ibtljn-10/?prefix=myobj 2023-11-02T17:55:43.646 DEBUG:boto:Data: 2023-11-02T17:55:43.647 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.647 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.648 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.648 DEBUG:boto:Params: {} 2023-11-02T17:55:43.649 DEBUG:boto:Token: None 2023-11-02T17:55:43.649 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/ 2023-11-02T17:55:43.650 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I4EXaj62060Gme9zVp/tBMBEjmM= 2023-11-02T17:55:43.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I4EXaj62060Gme9zVp/tBMBEjmM=', 'Content-Length': '0'} 2023-11-02T17:55:43.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c75596fa6b24458-006543e29f-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.656 DEBUG:boto:b'ibtljn-10myobj1000falsemyobj2023-11-02T17:55:22.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:43.656 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:55:43.657 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:55:43.657 DEBUG:boto:Method: HEAD 2023-11-02T17:55:43.658 DEBUG:boto:Path: /ibtljn-10/myobj 2023-11-02T17:55:43.659 DEBUG:boto:Data: 2023-11-02T17:55:43.660 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.660 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.661 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.661 DEBUG:boto:Params: {} 2023-11-02T17:55:43.661 DEBUG:boto:Token: None 2023-11-02T17:55:43.662 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/myobj 2023-11-02T17:55:43.663 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ybmqMfOR4YKgWeK3LWxxOGNfofQ= 2023-11-02T17:55:43.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ybmqMfOR4YKgWeK3LWxxOGNfofQ=', 'Content-Length': '0'} 2023-11-02T17:55:43.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000508da10a425a45c2-006543e29f-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.666 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:55:43.667 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:55:43.667 DEBUG:boto:Method: HEAD 2023-11-02T17:55:43.669 DEBUG:boto:Path: /ibtljn-10/myobj 2023-11-02T17:55:43.669 DEBUG:boto:Data: 2023-11-02T17:55:43.669 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.670 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.670 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.671 DEBUG:boto:Params: {} 2023-11-02T17:55:43.671 DEBUG:boto:Token: None 2023-11-02T17:55:43.672 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/myobj 2023-11-02T17:55:43.672 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ybmqMfOR4YKgWeK3LWxxOGNfofQ= 2023-11-02T17:55:43.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ybmqMfOR4YKgWeK3LWxxOGNfofQ=', 'Content-Length': '0'} 2023-11-02T17:55:43.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f304de39bd23f7d-006543e29f-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-11-02T17:55:43.676 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:55:43.677 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:55:43.677 DEBUG:boto:Method: GET 2023-11-02T17:55:43.678 DEBUG:boto:Path: /ibtljn-10/myobj 2023-11-02T17:55:43.678 DEBUG:boto:Data: 2023-11-02T17:55:43.679 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.679 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.680 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.680 DEBUG:boto:Params: {} 2023-11-02T17:55:43.680 DEBUG:boto:Token: None 2023-11-02T17:55:43.681 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/myobj 2023-11-02T17:55:43.681 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hyAobgAUVd10Qv4aQzQr+4Xszyc= 2023-11-02T17:55:43.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hyAobgAUVd10Qv4aQzQr+4Xszyc=', 'Content-Length': '0'} 2023-11-02T17:55:43.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bee885ce2bbc2a7-006543e29f-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.685 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:55:43.686 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:55:43.686 DEBUG:boto:Method: GET 2023-11-02T17:55:43.687 DEBUG:boto:Path: /ibtljn-10/myobj 2023-11-02T17:55:43.687 DEBUG:boto:Data: 2023-11-02T17:55:43.688 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.688 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.689 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.689 DEBUG:boto:Params: {} 2023-11-02T17:55:43.689 DEBUG:boto:Token: None 2023-11-02T17:55:43.690 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/myobj 2023-11-02T17:55:43.690 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hyAobgAUVd10Qv4aQzQr+4Xszyc= 2023-11-02T17:55:43.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hyAobgAUVd10Qv4aQzQr+4Xszyc=', 'Content-Length': '0'} 2023-11-02T17:55:43.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a06622d2a15b3f0-006543e29f-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.738 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:55:43.739 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:55:43.739 DEBUG:boto:path=/ibtljn-10/?prefix=myobj 2023-11-02T17:55:43.740 DEBUG:boto:auth_path=/ibtljn-10/?prefix=myobj 2023-11-02T17:55:43.740 DEBUG:boto:Method: GET 2023-11-02T17:55:43.741 DEBUG:boto:Path: /ibtljn-10/?prefix=myobj 2023-11-02T17:55:43.741 DEBUG:boto:Data: 2023-11-02T17:55:43.742 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.742 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.743 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.744 DEBUG:boto:Params: {} 2023-11-02T17:55:43.745 DEBUG:boto:Token: None 2023-11-02T17:55:43.745 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/ 2023-11-02T17:55:43.746 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I4EXaj62060Gme9zVp/tBMBEjmM= 2023-11-02T17:55:43.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I4EXaj62060Gme9zVp/tBMBEjmM=', 'Content-Length': '0'} 2023-11-02T17:55:43.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cda8fd74ec98c19-006543e29f-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.752 DEBUG:boto:b'ibtljn-10myobj1000falsemyobj2023-11-02T17:55:22.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:43.753 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:55:43.753 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:55:43.754 DEBUG:boto:path=/ibtljn-10/?prefix=myobj 2023-11-02T17:55:43.754 DEBUG:boto:auth_path=/ibtljn-10/?prefix=myobj 2023-11-02T17:55:43.755 DEBUG:boto:Method: GET 2023-11-02T17:55:43.755 DEBUG:boto:Path: /ibtljn-10/?prefix=myobj 2023-11-02T17:55:43.756 DEBUG:boto:Data: 2023-11-02T17:55:43.756 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.756 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:55:43.757 DEBUG:boto:Port: 8001 2023-11-02T17:55:43.757 DEBUG:boto:Params: {} 2023-11-02T17:55:43.758 DEBUG:boto:Token: None 2023-11-02T17:55:43.758 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-10/ 2023-11-02T17:55:43.759 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I4EXaj62060Gme9zVp/tBMBEjmM= 2023-11-02T17:55:43.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I4EXaj62060Gme9zVp/tBMBEjmM=', 'Content-Length': '0'} 2023-11-02T17:55:43.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000475353dec2026992-006543e29f-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.765 DEBUG:boto:b'ibtljn-10myobj1000falsemyobj2023-11-02T17:55:22.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:55:43.766 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-10 zones={a1, a2} 2023-11-02T17:55:43.767 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:55:43.767 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:55:43.768 DEBUG:boto:Method: HEAD 2023-11-02T17:55:43.768 DEBUG:boto:Path: /ibtljn-9/ 2023-11-02T17:55:43.769 DEBUG:boto:Data: 2023-11-02T17:55:43.769 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.770 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.770 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.770 DEBUG:boto:Params: {} 2023-11-02T17:55:43.771 DEBUG:boto:Token: None 2023-11-02T17:55:43.771 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/ 2023-11-02T17:55:43.772 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NjWNNi+9veMguEuN7nXSEYbB2z0= 2023-11-02T17:55:43.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NjWNNi+9veMguEuN7nXSEYbB2z0=', 'Content-Length': '0'} 2023-11-02T17:55:43.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bdc0eaf09201966-006543e29f-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.777 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:43.778 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:43.778 DEBUG:boto:Method: HEAD 2023-11-02T17:55:43.779 DEBUG:boto:Path: /ibtljn-9/myobj 2023-11-02T17:55:43.779 DEBUG:boto:Data: 2023-11-02T17:55:43.780 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.780 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.781 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.781 DEBUG:boto:Params: {} 2023-11-02T17:55:43.782 DEBUG:boto:Token: None 2023-11-02T17:55:43.782 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/myobj 2023-11-02T17:55:43.783 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8ejCC2oYAghl7VTNfTM39UKi1g8= 2023-11-02T17:55:43.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8ejCC2oYAghl7VTNfTM39UKi1g8=', 'Content-Length': '0'} 2023-11-02T17:55:43.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc3c6f87d68da395-006543e29f-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.786 DEBUG:boto:path=/ibtljn-9/myobj 2023-11-02T17:55:43.787 DEBUG:boto:auth_path=/ibtljn-9/myobj 2023-11-02T17:55:43.788 DEBUG:boto:Method: DELETE 2023-11-02T17:55:43.788 DEBUG:boto:Path: /ibtljn-9/myobj 2023-11-02T17:55:43.788 DEBUG:boto:Data: 2023-11-02T17:55:43.789 DEBUG:boto:Headers: {} 2023-11-02T17:55:43.789 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:55:43.790 DEBUG:boto:Port: 8000 2023-11-02T17:55:43.790 DEBUG:boto:Params: {} 2023-11-02T17:55:43.791 DEBUG:boto:Token: None 2023-11-02T17:55:43.791 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 17:55:43 GMT /ibtljn-9/myobj 2023-11-02T17:55:43.794 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C3hMs/Ec5CRinBDQf5/qQPXWnNs= 2023-11-02T17:55:43.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:55:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C3hMs/Ec5CRinBDQf5/qQPXWnNs=', 'Content-Length': '0'} 2023-11-02T17:55:43.804 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004fc0cccf368a793c-006543e29f-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:55:43.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:55:43.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:55:43.806 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:55:43.860 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:55:43.860 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:55:43.927 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:43.921+0000 7f198eed6f40 1 waiting for incremental sync to catch up: 2023-11-02T17:55:43.927 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.3.6, , , , , , , , , ] 2023-11-02T17:55:43.927 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.5.6, , , , , , , , , ] 2023-11-02T17:55:48.929 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:48.921+0000 7f198eed6f40 1 waiting for incremental sync to catch up: 2023-11-02T17:55:48.930 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.3.6, , , , , , , , , ] 2023-11-02T17:55:48.930 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.5.6, , , , , , , , , ] 2023-11-02T17:55:53.933 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:53.925+0000 7f198eed6f40 1 waiting for incremental sync to catch up: 2023-11-02T17:55:53.934 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.3.6, , , , , , , , , ] 2023-11-02T17:55:53.934 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.5.6, , , , , , , , , ] 2023-11-02T17:55:58.935 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:55:58.929+0000 7f198eed6f40 1 waiting for incremental sync to catch up: 2023-11-02T17:55:58.936 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.3.6, , , , , , , , , ] 2023-11-02T17:55:58.936 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.5.6, , , , , , , , , ] 2023-11-02T17:56:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:03.929+0000 7f198eed6f40 1 bucket sync caught up with source: 2023-11-02T17:56:03.938 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.5.6, , , , , , , , , ] 2023-11-02T17:56:03.939 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.5.6, , , , , , , , , ] 2023-11-02T17:56:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:03.929+0000 7f198eed6f40 0 bucket checkpoint complete 2023-11-02T17:56:03.967 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-9 zones={a2, a1} 2023-11-02T17:56:03.967 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:56:03.968 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:56:03.968 DEBUG:boto:Method: HEAD 2023-11-02T17:56:03.969 DEBUG:boto:Path: /ibtljn-9/ 2023-11-02T17:56:03.969 DEBUG:boto:Data: 2023-11-02T17:56:03.970 DEBUG:boto:Headers: {} 2023-11-02T17:56:03.970 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:03.970 DEBUG:boto:Port: 8001 2023-11-02T17:56:03.971 DEBUG:boto:Params: {} 2023-11-02T17:56:03.971 DEBUG:boto:Token: None 2023-11-02T17:56:03.972 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:56:03 GMT /ibtljn-9/ 2023-11-02T17:56:03.972 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lGKLKYV+gwexjmyW143X2e+9Ph0= 2023-11-02T17:56:03.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lGKLKYV+gwexjmyW143X2e+9Ph0=', 'Content-Length': '0'} 2023-11-02T17:56:03.977 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054a1c89a406c6991-006543e2b3-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:03.978 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:56:03.978 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:56:03.979 DEBUG:boto:Method: HEAD 2023-11-02T17:56:03.979 DEBUG:boto:Path: /ibtljn-9/ 2023-11-02T17:56:03.980 DEBUG:boto:Data: 2023-11-02T17:56:03.980 DEBUG:boto:Headers: {} 2023-11-02T17:56:03.980 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:56:03.981 DEBUG:boto:Port: 8000 2023-11-02T17:56:03.981 DEBUG:boto:Params: {} 2023-11-02T17:56:03.982 DEBUG:boto:Token: None 2023-11-02T17:56:03.982 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:56:03 GMT /ibtljn-9/ 2023-11-02T17:56:03.983 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lGKLKYV+gwexjmyW143X2e+9Ph0= 2023-11-02T17:56:03.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lGKLKYV+gwexjmyW143X2e+9Ph0=', 'Content-Length': '0'} 2023-11-02T17:56:03.987 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad58279337be687c-006543e2b3-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:03.988 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T17:56:03.988 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:56:03.989 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:56:03.989 DEBUG:boto:path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:56:03.990 DEBUG:boto:auth_path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:56:03.990 DEBUG:boto:Method: GET 2023-11-02T17:56:03.991 DEBUG:boto:Path: /ibtljn-9/?versions&max-keys=999 2023-11-02T17:56:03.991 DEBUG:boto:Data: 2023-11-02T17:56:03.991 DEBUG:boto:Headers: {} 2023-11-02T17:56:03.992 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:03.992 DEBUG:boto:Port: 8001 2023-11-02T17:56:03.993 DEBUG:boto:Params: {} 2023-11-02T17:56:03.993 DEBUG:boto:Token: None 2023-11-02T17:56:03.994 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:56:03 GMT /ibtljn-9/?versions 2023-11-02T17:56:03.994 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uCo0Nvbrxd4EJ2dOas0V55YBvDw= 2023-11-02T17:56:03.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uCo0Nvbrxd4EJ2dOas0V55YBvDw=', 'Content-Length': '0'} 2023-11-02T17:56:04.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037cfaa0466082f58-006543e2b3-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:04.000 DEBUG:boto:b'ibtljn-9999false' 2023-11-02T17:56:04.001 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T17:56:04.001 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:56:04.002 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:56:04.002 DEBUG:boto:path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:56:04.003 DEBUG:boto:auth_path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:56:04.003 DEBUG:boto:Method: GET 2023-11-02T17:56:04.004 DEBUG:boto:Path: /ibtljn-9/?versions&max-keys=999 2023-11-02T17:56:04.004 DEBUG:boto:Data: 2023-11-02T17:56:04.005 DEBUG:boto:Headers: {} 2023-11-02T17:56:04.005 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:56:04.005 DEBUG:boto:Port: 8000 2023-11-02T17:56:04.006 DEBUG:boto:Params: {} 2023-11-02T17:56:04.006 DEBUG:boto:Token: None 2023-11-02T17:56:04.007 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:56:04 GMT /ibtljn-9/?versions 2023-11-02T17:56:04.007 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5tzR4rJXQukpQXOAEPsYdV2NPhQ= 2023-11-02T17:56:04.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5tzR4rJXQukpQXOAEPsYdV2NPhQ=', 'Content-Length': '0'} 2023-11-02T17:56:04.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035cd491219bc0faf-006543e2b4-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:56:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:04.013 DEBUG:boto:b'ibtljn-9999false' 2023-11-02T17:56:04.013 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:56:04.014 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:56:04.014 DEBUG:boto:path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:56:04.015 DEBUG:boto:auth_path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:56:04.015 DEBUG:boto:Method: GET 2023-11-02T17:56:04.016 DEBUG:boto:Path: /ibtljn-9/?versions&max-keys=999 2023-11-02T17:56:04.016 DEBUG:boto:Data: 2023-11-02T17:56:04.016 DEBUG:boto:Headers: {} 2023-11-02T17:56:04.017 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:04.017 DEBUG:boto:Port: 8001 2023-11-02T17:56:04.018 DEBUG:boto:Params: {} 2023-11-02T17:56:04.018 DEBUG:boto:Token: None 2023-11-02T17:56:04.019 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:56:04 GMT /ibtljn-9/?versions 2023-11-02T17:56:04.019 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5tzR4rJXQukpQXOAEPsYdV2NPhQ= 2023-11-02T17:56:04.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5tzR4rJXQukpQXOAEPsYdV2NPhQ=', 'Content-Length': '0'} 2023-11-02T17:56:04.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027277fd487d55c62-006543e2b4-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:56:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:04.066 DEBUG:boto:b'ibtljn-9999false' 2023-11-02T17:56:04.067 DEBUG:boto:path=/ibtljn-9/ 2023-11-02T17:56:04.067 DEBUG:boto:auth_path=/ibtljn-9/ 2023-11-02T17:56:04.068 DEBUG:boto:path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:56:04.068 DEBUG:boto:auth_path=/ibtljn-9/?versions&max-keys=999 2023-11-02T17:56:04.068 DEBUG:boto:Method: GET 2023-11-02T17:56:04.069 DEBUG:boto:Path: /ibtljn-9/?versions&max-keys=999 2023-11-02T17:56:04.069 DEBUG:boto:Data: 2023-11-02T17:56:04.070 DEBUG:boto:Headers: {} 2023-11-02T17:56:04.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:56:04.070 DEBUG:boto:Port: 8000 2023-11-02T17:56:04.071 DEBUG:boto:Params: {} 2023-11-02T17:56:04.071 DEBUG:boto:Token: None 2023-11-02T17:56:04.072 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:56:04 GMT /ibtljn-9/?versions 2023-11-02T17:56:04.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5tzR4rJXQukpQXOAEPsYdV2NPhQ= 2023-11-02T17:56:04.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5tzR4rJXQukpQXOAEPsYdV2NPhQ=', 'Content-Length': '0'} 2023-11-02T17:56:04.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000941a1e1eacf8b3c5-006543e2b4-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:56:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:04.078 DEBUG:boto:b'ibtljn-9999false' 2023-11-02T17:56:04.078 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-9 zones={a2, a1} 2023-11-02T17:56:04.079 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:56:04.079 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:56:04.080 DEBUG:boto:Method: HEAD 2023-11-02T17:56:04.080 DEBUG:boto:Path: /ibtljn-10/ 2023-11-02T17:56:04.081 DEBUG:boto:Data: 2023-11-02T17:56:04.081 DEBUG:boto:Headers: {} 2023-11-02T17:56:04.081 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:04.082 DEBUG:boto:Port: 8001 2023-11-02T17:56:04.082 DEBUG:boto:Params: {} 2023-11-02T17:56:04.083 DEBUG:boto:Token: None 2023-11-02T17:56:04.083 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:56:04 GMT /ibtljn-10/ 2023-11-02T17:56:04.084 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o4CiP5p2jiCTuKA2Caj+jD4e8cE= 2023-11-02T17:56:04.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o4CiP5p2jiCTuKA2Caj+jD4e8cE=', 'Content-Length': '0'} 2023-11-02T17:56:04.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000067cad00ac918bf2-006543e2b4-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:56:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:04.089 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:56:04.089 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:56:04.089 DEBUG:boto:Method: HEAD 2023-11-02T17:56:04.090 DEBUG:boto:Path: /ibtljn-10/myobj 2023-11-02T17:56:04.090 DEBUG:boto:Data: 2023-11-02T17:56:04.091 DEBUG:boto:Headers: {} 2023-11-02T17:56:04.091 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:04.094 DEBUG:boto:Port: 8001 2023-11-02T17:56:04.094 DEBUG:boto:Params: {} 2023-11-02T17:56:04.095 DEBUG:boto:Token: None 2023-11-02T17:56:04.095 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:56:04 GMT /ibtljn-10/myobj 2023-11-02T17:56:04.096 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XvW851SwHwkfJR/D19Iopwi8Uo4= 2023-11-02T17:56:04.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XvW851SwHwkfJR/D19Iopwi8Uo4=', 'Content-Length': '0'} 2023-11-02T17:56:04.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:55:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c41b875cc651956-006543e2b4-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:56:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:04.100 DEBUG:boto:path=/ibtljn-10/myobj 2023-11-02T17:56:04.100 DEBUG:boto:auth_path=/ibtljn-10/myobj 2023-11-02T17:56:04.101 DEBUG:boto:Method: DELETE 2023-11-02T17:56:04.101 DEBUG:boto:Path: /ibtljn-10/myobj 2023-11-02T17:56:04.101 DEBUG:boto:Data: 2023-11-02T17:56:04.102 DEBUG:boto:Headers: {} 2023-11-02T17:56:04.102 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:04.103 DEBUG:boto:Port: 8001 2023-11-02T17:56:04.103 DEBUG:boto:Params: {} 2023-11-02T17:56:04.103 DEBUG:boto:Token: None 2023-11-02T17:56:04.104 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 17:56:04 GMT /ibtljn-10/myobj 2023-11-02T17:56:04.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q//ENEq5PqfA5xvlrmRPaNc42hk= 2023-11-02T17:56:04.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q//ENEq5PqfA5xvlrmRPaNc42hk=', 'Content-Length': '0'} 2023-11-02T17:56:04.117 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000049c779f5671333bd-006543e2b4-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:56:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:04.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:56:04.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:56:04.118 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:56:04.172 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:56:04.172 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:56:04.236 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:04.229+0000 7f650ae32f40 1 waiting for incremental sync to catch up: 2023-11-02T17:56:04.237 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-11-02T17:56:04.237 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2023-11-02T17:56:09.239 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:09.232+0000 7f650ae32f40 1 waiting for incremental sync to catch up: 2023-11-02T17:56:09.240 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-11-02T17:56:09.240 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2023-11-02T17:56:14.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:14.232+0000 7f650ae32f40 1 waiting for incremental sync to catch up: 2023-11-02T17:56:14.242 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-11-02T17:56:14.242 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2023-11-02T17:56:19.245 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:19.236+0000 7f650ae32f40 1 waiting for incremental sync to catch up: 2023-11-02T17:56:19.245 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-11-02T17:56:19.245 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2023-11-02T17:56:24.247 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:24.236+0000 7f650ae32f40 1 bucket sync caught up with source: 2023-11-02T17:56:24.248 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.15.6, , , , , , , , , ] 2023-11-02T17:56:24.248 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2023-11-02T17:56:24.248 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:24.236+0000 7f650ae32f40 0 bucket checkpoint complete 2023-11-02T17:56:24.278 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-10 zones={a1, a2} 2023-11-02T17:56:24.278 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:56:24.279 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:56:24.279 DEBUG:boto:Method: HEAD 2023-11-02T17:56:24.280 DEBUG:boto:Path: /ibtljn-10/ 2023-11-02T17:56:24.280 DEBUG:boto:Data: 2023-11-02T17:56:24.281 DEBUG:boto:Headers: {} 2023-11-02T17:56:24.281 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:56:24.281 DEBUG:boto:Port: 8000 2023-11-02T17:56:24.282 DEBUG:boto:Params: {} 2023-11-02T17:56:24.285 DEBUG:boto:Token: None 2023-11-02T17:56:24.286 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:56:24 GMT /ibtljn-10/ 2023-11-02T17:56:24.286 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T+zTfWWRMjE6KH87FmjZDjRt6CU= 2023-11-02T17:56:24.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T+zTfWWRMjE6KH87FmjZDjRt6CU=', 'Content-Length': '0'} 2023-11-02T17:56:24.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009521bb005e73b428-006543e2c8-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:24.291 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:56:24.292 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:56:24.292 DEBUG:boto:Method: HEAD 2023-11-02T17:56:24.293 DEBUG:boto:Path: /ibtljn-10/ 2023-11-02T17:56:24.293 DEBUG:boto:Data: 2023-11-02T17:56:24.293 DEBUG:boto:Headers: {} 2023-11-02T17:56:24.294 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:24.294 DEBUG:boto:Port: 8001 2023-11-02T17:56:24.295 DEBUG:boto:Params: {} 2023-11-02T17:56:24.295 DEBUG:boto:Token: None 2023-11-02T17:56:24.296 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:56:24 GMT /ibtljn-10/ 2023-11-02T17:56:24.296 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T+zTfWWRMjE6KH87FmjZDjRt6CU= 2023-11-02T17:56:24.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T+zTfWWRMjE6KH87FmjZDjRt6CU=', 'Content-Length': '0'} 2023-11-02T17:56:24.301 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000002932eda0f882f7-006543e2c8-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:24.301 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T17:56:24.302 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:56:24.302 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:56:24.303 DEBUG:boto:path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:56:24.303 DEBUG:boto:auth_path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:56:24.304 DEBUG:boto:Method: GET 2023-11-02T17:56:24.304 DEBUG:boto:Path: /ibtljn-10/?versions&max-keys=999 2023-11-02T17:56:24.305 DEBUG:boto:Data: 2023-11-02T17:56:24.305 DEBUG:boto:Headers: {} 2023-11-02T17:56:24.305 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:56:24.306 DEBUG:boto:Port: 8000 2023-11-02T17:56:24.306 DEBUG:boto:Params: {} 2023-11-02T17:56:24.307 DEBUG:boto:Token: None 2023-11-02T17:56:24.307 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:56:24 GMT /ibtljn-10/?versions 2023-11-02T17:56:24.308 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cIiJxmLQgJzCDgEOvo0fXXXr2D0= 2023-11-02T17:56:24.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cIiJxmLQgJzCDgEOvo0fXXXr2D0=', 'Content-Length': '0'} 2023-11-02T17:56:24.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aaab905d20953cd-006543e2c8-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:24.314 DEBUG:boto:b'ibtljn-10999false' 2023-11-02T17:56:24.314 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T17:56:24.315 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:56:24.315 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:56:24.316 DEBUG:boto:path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:56:24.316 DEBUG:boto:auth_path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:56:24.316 DEBUG:boto:Method: GET 2023-11-02T17:56:24.317 DEBUG:boto:Path: /ibtljn-10/?versions&max-keys=999 2023-11-02T17:56:24.317 DEBUG:boto:Data: 2023-11-02T17:56:24.318 DEBUG:boto:Headers: {} 2023-11-02T17:56:24.318 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:24.319 DEBUG:boto:Port: 8001 2023-11-02T17:56:24.319 DEBUG:boto:Params: {} 2023-11-02T17:56:24.320 DEBUG:boto:Token: None 2023-11-02T17:56:24.320 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:56:24 GMT /ibtljn-10/?versions 2023-11-02T17:56:24.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cIiJxmLQgJzCDgEOvo0fXXXr2D0= 2023-11-02T17:56:24.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cIiJxmLQgJzCDgEOvo0fXXXr2D0=', 'Content-Length': '0'} 2023-11-02T17:56:24.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000353282e33994fa20-006543e2c8-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:24.327 DEBUG:boto:b'ibtljn-10999false' 2023-11-02T17:56:24.328 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:56:24.328 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:56:24.329 DEBUG:boto:path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:56:24.329 DEBUG:boto:auth_path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:56:24.330 DEBUG:boto:Method: GET 2023-11-02T17:56:24.330 DEBUG:boto:Path: /ibtljn-10/?versions&max-keys=999 2023-11-02T17:56:24.331 DEBUG:boto:Data: 2023-11-02T17:56:24.331 DEBUG:boto:Headers: {} 2023-11-02T17:56:24.331 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:56:24.332 DEBUG:boto:Port: 8000 2023-11-02T17:56:24.332 DEBUG:boto:Params: {} 2023-11-02T17:56:24.333 DEBUG:boto:Token: None 2023-11-02T17:56:24.333 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:56:24 GMT /ibtljn-10/?versions 2023-11-02T17:56:24.334 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cIiJxmLQgJzCDgEOvo0fXXXr2D0= 2023-11-02T17:56:24.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cIiJxmLQgJzCDgEOvo0fXXXr2D0=', 'Content-Length': '0'} 2023-11-02T17:56:24.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c1e21fb8c19e3b0-006543e2c8-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:24.378 DEBUG:boto:b'ibtljn-10999false' 2023-11-02T17:56:24.379 DEBUG:boto:path=/ibtljn-10/ 2023-11-02T17:56:24.379 DEBUG:boto:auth_path=/ibtljn-10/ 2023-11-02T17:56:24.380 DEBUG:boto:path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:56:24.380 DEBUG:boto:auth_path=/ibtljn-10/?versions&max-keys=999 2023-11-02T17:56:24.381 DEBUG:boto:Method: GET 2023-11-02T17:56:24.381 DEBUG:boto:Path: /ibtljn-10/?versions&max-keys=999 2023-11-02T17:56:24.382 DEBUG:boto:Data: 2023-11-02T17:56:24.382 DEBUG:boto:Headers: {} 2023-11-02T17:56:24.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:24.383 DEBUG:boto:Port: 8001 2023-11-02T17:56:24.384 DEBUG:boto:Params: {} 2023-11-02T17:56:24.384 DEBUG:boto:Token: None 2023-11-02T17:56:24.385 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:56:24 GMT /ibtljn-10/?versions 2023-11-02T17:56:24.385 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cIiJxmLQgJzCDgEOvo0fXXXr2D0= 2023-11-02T17:56:24.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cIiJxmLQgJzCDgEOvo0fXXXr2D0=', 'Content-Length': '0'} 2023-11-02T17:56:24.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083d9c5128ccdc1a2-006543e2c8-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:24.391 DEBUG:boto:b'ibtljn-10999false' 2023-11-02T17:56:24.392 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-10 zones={a1, a2} 2023-11-02T17:56:24.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2023-11-02T17:56:24.393 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-11 2023-11-02T17:56:24.394 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:56:24.394 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:56:24.395 DEBUG:boto:Method: PUT 2023-11-02T17:56:24.395 DEBUG:boto:Path: /ibtljn-11/ 2023-11-02T17:56:24.395 DEBUG:boto:Data: 2023-11-02T17:56:24.396 DEBUG:boto:Headers: {} 2023-11-02T17:56:24.396 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:56:24.397 DEBUG:boto:Port: 8000 2023-11-02T17:56:24.397 DEBUG:boto:Params: {} 2023-11-02T17:56:24.398 DEBUG:boto:Token: None 2023-11-02T17:56:24.398 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:56:24 GMT /ibtljn-11/ 2023-11-02T17:56:24.399 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xDxKOJGaJb+3LBiIqNB3ioyQPzo= 2023-11-02T17:56:24.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xDxKOJGaJb+3LBiIqNB3ioyQPzo=', 'Content-Length': '0'} 2023-11-02T17:56:24.423 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002f9d0a7e9fc47833-006543e2c8-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:24.424 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-12 2023-11-02T17:56:24.424 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:56:24.425 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:56:24.425 DEBUG:boto:Method: PUT 2023-11-02T17:56:24.425 DEBUG:boto:Path: /ibtljn-12/ 2023-11-02T17:56:24.426 DEBUG:boto:Data: 2023-11-02T17:56:24.426 DEBUG:boto:Headers: {} 2023-11-02T17:56:24.427 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:24.427 DEBUG:boto:Port: 8001 2023-11-02T17:56:24.428 DEBUG:boto:Params: {} 2023-11-02T17:56:24.428 DEBUG:boto:Token: None 2023-11-02T17:56:24.428 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:56:24 GMT /ibtljn-12/ 2023-11-02T17:56:24.429 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fnmV19wdnaGOlfty440LNJgBi5A= 2023-11-02T17:56:24.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fnmV19wdnaGOlfty440LNJgBi5A=', 'Content-Length': '0'} 2023-11-02T17:56:24.469 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a3430618fd8142e1-006543e2c8-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:24.470 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:56:24.470 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:56:24.470 DEBUG:boto:path=/ibtljn-11/?versioning 2023-11-02T17:56:24.471 DEBUG:boto:auth_path=/ibtljn-11/?versioning 2023-11-02T17:56:24.471 DEBUG:boto:Method: PUT 2023-11-02T17:56:24.472 DEBUG:boto:Path: /ibtljn-11/?versioning 2023-11-02T17:56:24.472 DEBUG:boto:Data: Enabled Disabled 2023-11-02T17:56:24.472 DEBUG:boto:Headers: {} 2023-11-02T17:56:24.473 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:56:24.473 DEBUG:boto:Port: 8000 2023-11-02T17:56:24.473 DEBUG:boto:Params: {} 2023-11-02T17:56:24.474 DEBUG:boto:Token: None 2023-11-02T17:56:24.474 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:56:24 GMT /ibtljn-11/?versioning 2023-11-02T17:56:24.475 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7GdQzcAIWeDptZZLWlPjBhrhMok= 2023-11-02T17:56:24.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7GdQzcAIWeDptZZLWlPjBhrhMok=', 'Content-Length': '228'} 2023-11-02T17:56:24.484 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000099f5e588d58bde11-006543e2c8-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:24.484 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:56:24.485 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:56:24.485 DEBUG:boto:path=/ibtljn-12/?versioning 2023-11-02T17:56:24.486 DEBUG:boto:auth_path=/ibtljn-12/?versioning 2023-11-02T17:56:24.486 DEBUG:boto:Method: PUT 2023-11-02T17:56:24.487 DEBUG:boto:Path: /ibtljn-12/?versioning 2023-11-02T17:56:24.487 DEBUG:boto:Data: Enabled Disabled 2023-11-02T17:56:24.488 DEBUG:boto:Headers: {} 2023-11-02T17:56:24.488 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:24.488 DEBUG:boto:Port: 8001 2023-11-02T17:56:24.489 DEBUG:boto:Params: {} 2023-11-02T17:56:24.489 DEBUG:boto:Token: None 2023-11-02T17:56:24.490 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:56:24 GMT /ibtljn-12/?versioning 2023-11-02T17:56:24.490 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Sen19MPgu7E38zGMd6ofJuq0EaE= 2023-11-02T17:56:24.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Sen19MPgu7E38zGMd6ofJuq0EaE=', 'Content-Length': '228'} 2023-11-02T17:56:24.506 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f1cea8f7a6374835-006543e2c8-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:24.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:56:24.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:56:24.508 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:56:24.552 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:56:24.552 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:56:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T17:56:24.629 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T17:56:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:56:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:56:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T17:56:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:56:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:56:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:56:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:56:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947784.496808_64.1", 2023-11-02T17:56:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:56:24.496808Z" 2023-11-02T17:56:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:56:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T17:56:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T17:56:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:56:24.633 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T17:56:24.634 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947784.496808_64.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:56:24.634 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T17:56:24.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:56:24.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:56:24.636 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:56:24.718 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:56:24.719 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:56:24.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:24.764+0000 7f4f2bdaff40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:56:24.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:24.768+0000 7f4f2bdaff40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:56:24.802 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:56:24.802 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:56:24.802 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:56:24.803 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:56:24.803 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:56:24.803 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:56:24.803 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:56:24.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:24.804 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:56:24.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:56:24.804 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:56:24.804 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:56:24.805 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:56:24.805 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:56:24.805 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:56:24.805 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:56:24.805 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:56:24.806 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:56:24.806 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:56:24.806 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:56:24.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:24.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:56:24.807 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:56:24.807 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:56:24.807 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:56:24.807 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:56:24.808 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:56:24.808 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:56:24.808 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:56:24.808 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:56:24.808 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:56:24.809 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:56:24.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:24.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:56:24.809 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:56:24.810 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:56:24.810 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:56:24.810 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947784.407953_53.1", 2023-11-02T17:56:24.810 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:56:24.811 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:56:24.811 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:56:24.811 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:56:24.407953Z", 2023-11-02T17:56:24.811 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:56:24.812 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:56:24.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:24.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:56:24.812 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:56:24.813 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:56:24.813 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:56:24.813 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T17:56:24.813 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:56:24.814 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:56:24.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:56:24.814 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T17:56:24.814 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:56:24.815 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:56:24.815 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:56:24.815 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:56:24.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:24.815 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:56:24.816 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:56:24.816 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:56:24.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:24.816 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:56:24Z" 2023-11-02T17:56:24.817 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:56:24.818 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698947784.407953_53.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:56:24.407953Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T17:56:24Z" } 2023-11-02T17:56:24.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698947784.407953_53.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:56:24.407953Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T17:56:24.819 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947784.496808_64.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:56:24.819 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947784.407953_53.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:56:24.820 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698947784.496808_64.1 target=1_1698947784.407953_53.1 2023-11-02T17:56:29.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:56:29.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:56:29.822 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:56:29.868 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:56:29.868 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:56:29.927 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:29.916+0000 7f66ac747f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:56:29.928 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:29.920+0000 7f66ac747f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:56:29.953 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:56:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:56:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:56:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:56:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:56:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:56:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:56:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:56:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:56:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:56:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:56:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:56:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:56:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:56:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:56:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:56:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:56:29.958 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:56:29.958 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:56:29.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:29.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:56:29.959 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:56:29.959 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:56:29.959 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:56:29.959 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:56:29.959 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:56:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:56:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:56:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:56:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:56:29.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:56:29.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:29.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:56:29.961 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:56:29.962 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:56:29.962 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:56:29.962 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947784.496808_64.1", 2023-11-02T17:56:29.962 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:56:29.963 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:56:29.963 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:56:29.963 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:56:24.496808Z", 2023-11-02T17:56:29.964 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:56:29.964 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:56:29.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:29.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:56:29.964 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:56:29.965 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:56:29.965 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:56:29.965 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T17:56:29.965 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:56:29.966 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:56:29.966 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:56:29.966 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T17:56:29.966 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:56:29.967 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:56:29.967 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:56:29.967 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:56:29.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:29.967 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:56:29.968 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:56:29.968 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:56:29.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:56:29.968 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:56:29Z" 2023-11-02T17:56:29.969 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:56:29.970 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698947784.496808_64.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:56:24.496808Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T17:56:29Z" } 2023-11-02T17:56:29.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698947784.496808_64.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:56:24.496808Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T17:56:29.971 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947784.496808_64.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:56:29.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947784.496808_64.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:56:29.972 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T17:56:29.972 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:56:29.973 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:56:29.973 DEBUG:boto:Method: HEAD 2023-11-02T17:56:29.974 DEBUG:boto:Path: /ibtljn-11/ 2023-11-02T17:56:29.974 DEBUG:boto:Data: 2023-11-02T17:56:29.974 DEBUG:boto:Headers: {} 2023-11-02T17:56:29.975 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:56:29.975 DEBUG:boto:Port: 8000 2023-11-02T17:56:29.976 DEBUG:boto:Params: {} 2023-11-02T17:56:29.976 DEBUG:boto:Token: None 2023-11-02T17:56:29.977 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:56:29 GMT /ibtljn-11/ 2023-11-02T17:56:29.978 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y/lyNSVNNSCdjVyA7t/N8MNNTr8= 2023-11-02T17:56:29.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y/lyNSVNNSCdjVyA7t/N8MNNTr8=', 'Content-Length': '0'} 2023-11-02T17:56:29.983 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007aeba00d9b053460-006543e2cd-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:29.983 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:56:29.984 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:56:29.984 DEBUG:boto:Method: PUT 2023-11-02T17:56:29.985 DEBUG:boto:Path: /ibtljn-11/dummy 2023-11-02T17:56:29.985 DEBUG:boto:Data: 2023-11-02T17:56:29.986 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-11-02T17:56:29.986 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:56:29.987 DEBUG:boto:Port: 8000 2023-11-02T17:56:29.987 DEBUG:boto:Params: {} 2023-11-02T17:56:29.987 DEBUG:boto:Token: None 2023-11-02T17:56:29.988 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 02 Nov 2023 17:56:29 GMT /ibtljn-11/dummy 2023-11-02T17:56:29.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8AHM54AOKrdUJscvkAccQ276Eq0= 2023-11-02T17:56:29.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:56:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8AHM54AOKrdUJscvkAccQ276Eq0='} 2023-11-02T17:56:30.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-amz-request-id', 'tx00000ee7a443200411334-006543e2cd-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:30.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:56:30.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:56:30.016 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:56:30.067 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:56:30.067 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:56:30.132 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:30.124+0000 7f7cdbfbff40 1 waiting to reach incremental sync.. 2023-11-02T17:56:35.133 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:35.124+0000 7f7cdbfbff40 1 waiting to reach incremental sync.. 2023-11-02T17:56:40.134 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:40.124+0000 7f7cdbfbff40 1 waiting to reach incremental sync.. 2023-11-02T17:56:45.138 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:45.128+0000 7f7cdbfbff40 1 bucket sync caught up with source: 2023-11-02T17:56:45.138 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.9.11, , , , , ] 2023-11-02T17:56:45.139 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.9.11, , , , , ] 2023-11-02T17:56:45.139 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:45.128+0000 7f7cdbfbff40 0 bucket checkpoint complete 2023-11-02T17:56:45.168 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:56:45.169 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:56:45.169 DEBUG:boto:Method: HEAD 2023-11-02T17:56:45.170 DEBUG:boto:Path: /ibtljn-12/ 2023-11-02T17:56:45.171 DEBUG:boto:Data: 2023-11-02T17:56:45.171 DEBUG:boto:Headers: {} 2023-11-02T17:56:45.172 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:45.172 DEBUG:boto:Port: 8001 2023-11-02T17:56:45.172 DEBUG:boto:Params: {} 2023-11-02T17:56:45.173 DEBUG:boto:Token: None 2023-11-02T17:56:45.174 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:56:45 GMT /ibtljn-12/ 2023-11-02T17:56:45.174 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0WYZnSf5cow2zb28mATgMRYdrf8= 2023-11-02T17:56:45.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:56:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0WYZnSf5cow2zb28mATgMRYdrf8=', 'Content-Length': '0'} 2023-11-02T17:56:45.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073ffd87573ba71b1-006543e2dd-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:45.180 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:56:45.180 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:56:45.181 DEBUG:boto:Method: PUT 2023-11-02T17:56:45.181 DEBUG:boto:Path: /ibtljn-12/dummy 2023-11-02T17:56:45.182 DEBUG:boto:Data: 2023-11-02T17:56:45.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-11-02T17:56:45.183 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:56:45.184 DEBUG:boto:Port: 8001 2023-11-02T17:56:45.184 DEBUG:boto:Params: {} 2023-11-02T17:56:45.185 DEBUG:boto:Token: None 2023-11-02T17:56:45.185 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 02 Nov 2023 17:56:45 GMT /ibtljn-12/dummy 2023-11-02T17:56:45.186 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wTZmCK81PtqwPIt/auMRh/leW/U= 2023-11-02T17:56:45.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:56:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wTZmCK81PtqwPIt/auMRh/leW/U='} 2023-11-02T17:56:45.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-amz-request-id', 'tx000000977762cde017b5e-006543e2dd-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:56:45.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:56:45.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:56:45.232 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:56:45.286 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:56:45.286 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:56:45.348 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:45.340+0000 7f27bcd17f40 1 waiting to reach incremental sync.. 2023-11-02T17:56:50.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:50.340+0000 7f27bcd17f40 1 waiting to reach incremental sync.. 2023-11-02T17:56:55.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:56:55.339+0000 7f27bcd17f40 1 waiting to reach incremental sync.. 2023-11-02T17:57:00.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:00.343+0000 7f27bcd17f40 1 waiting to reach incremental sync.. 2023-11-02T17:57:05.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:05.343+0000 7f27bcd17f40 1 bucket sync caught up with source: 2023-11-02T17:57:05.356 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.23.11, , , , , ] 2023-11-02T17:57:05.356 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.23.11, , , , , ] 2023-11-02T17:57:05.356 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:05.343+0000 7f27bcd17f40 0 bucket checkpoint complete 2023-11-02T17:57:05.381 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:57:05.381 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:57:05.382 DEBUG:boto:Method: HEAD 2023-11-02T17:57:05.382 DEBUG:boto:Path: /ibtljn-11/ 2023-11-02T17:57:05.383 DEBUG:boto:Data: 2023-11-02T17:57:05.383 DEBUG:boto:Headers: {} 2023-11-02T17:57:05.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:57:05.384 DEBUG:boto:Port: 8000 2023-11-02T17:57:05.384 DEBUG:boto:Params: {} 2023-11-02T17:57:05.385 DEBUG:boto:Token: None 2023-11-02T17:57:05.385 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-11/ 2023-11-02T17:57:05.386 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y+BQtlezlhID+KOyA95alW9KUwM= 2023-11-02T17:57:05.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y+BQtlezlhID+KOyA95alW9KUwM=', 'Content-Length': '0'} 2023-11-02T17:57:05.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069b4141a3458d22e-006543e2f1-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.391 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:57:05.392 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:57:05.392 DEBUG:boto:Method: PUT 2023-11-02T17:57:05.393 DEBUG:boto:Path: /ibtljn-11/obj-a1 2023-11-02T17:57:05.393 DEBUG:boto:Data: 2023-11-02T17:57:05.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:57:05.394 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:57:05.394 DEBUG:boto:Port: 8000 2023-11-02T17:57:05.395 DEBUG:boto:Params: {} 2023-11-02T17:57:05.395 DEBUG:boto:Token: None 2023-11-02T17:57:05.396 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-11/obj-a1 2023-11-02T17:57:05.396 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwoAxCRT4R6DsqZKT+pP9Ms3TYM= 2023-11-02T17:57:05.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwoAxCRT4R6DsqZKT+pP9Ms3TYM='} 2023-11-02T17:57:05.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-amz-request-id', 'tx000004d7244bfdc377589-006543e2f1-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.419 DEBUG:rgw_multi.tests:version1 id=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:57:05.419 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:57:05.420 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:57:05.420 DEBUG:boto:Method: PUT 2023-11-02T17:57:05.420 DEBUG:boto:Path: /ibtljn-11/obj-a1 2023-11-02T17:57:05.421 DEBUG:boto:Data: 2023-11-02T17:57:05.421 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:57:05.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:57:05.422 DEBUG:boto:Port: 8000 2023-11-02T17:57:05.422 DEBUG:boto:Params: {} 2023-11-02T17:57:05.423 DEBUG:boto:Token: None 2023-11-02T17:57:05.423 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-11/obj-a1 2023-11-02T17:57:05.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mrQpiQvigh7KGRvYtWjVCjzhAY0= 2023-11-02T17:57:05.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mrQpiQvigh7KGRvYtWjVCjzhAY0='} 2023-11-02T17:57:05.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UI.zWF1P13l13BOmph0c7LwmD3WP2xw'), ('x-amz-request-id', 'tx000003113ab3fc051f623-006543e2f1-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.471 DEBUG:rgw_multi.tests:version2 id=UI.zWF1P13l13BOmph0c7LwmD3WP2xw 2023-11-02T17:57:05.471 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:57:05.472 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:57:05.472 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=UI.zWF1P13l13BOmph0c7LwmD3WP2xw 2023-11-02T17:57:05.473 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=UI.zWF1P13l13BOmph0c7LwmD3WP2xw 2023-11-02T17:57:05.473 DEBUG:boto:Method: DELETE 2023-11-02T17:57:05.473 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=UI.zWF1P13l13BOmph0c7LwmD3WP2xw 2023-11-02T17:57:05.474 DEBUG:boto:Data: 2023-11-02T17:57:05.474 DEBUG:boto:Headers: {} 2023-11-02T17:57:05.475 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:57:05.475 DEBUG:boto:Port: 8000 2023-11-02T17:57:05.476 DEBUG:boto:Params: {} 2023-11-02T17:57:05.476 DEBUG:boto:Token: None 2023-11-02T17:57:05.477 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-11/obj-a1?versionId=UI.zWF1P13l13BOmph0c7LwmD3WP2xw 2023-11-02T17:57:05.477 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ju5zEiu1XYO2E/RtYUnEVgV1m58= 2023-11-02T17:57:05.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ju5zEiu1XYO2E/RtYUnEVgV1m58=', 'Content-Length': '0'} 2023-11-02T17:57:05.499 DEBUG:boto:Response headers: [('x-amz-version-id', 'UI.zWF1P13l13BOmph0c7LwmD3WP2xw'), ('x-amz-request-id', 'tx0000014297d450f18db46-006543e2f1-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.500 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:57:05.500 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:57:05.501 DEBUG:boto:Method: PUT 2023-11-02T17:57:05.501 DEBUG:boto:Path: /ibtljn-11/obj-a1 2023-11-02T17:57:05.502 DEBUG:boto:Data: 2023-11-02T17:57:05.502 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:57:05.503 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:57:05.504 DEBUG:boto:Port: 8000 2023-11-02T17:57:05.505 DEBUG:boto:Params: {} 2023-11-02T17:57:05.505 DEBUG:boto:Token: None 2023-11-02T17:57:05.506 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-11/obj-a1 2023-11-02T17:57:05.506 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nMWur6Ckd1fcF277dt6xgYXs8xE= 2023-11-02T17:57:05.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nMWur6Ckd1fcF277dt6xgYXs8xE='} 2023-11-02T17:57:05.525 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9yz-6v4AnfZzI2XDXIEtf-D7JxwKQCO'), ('x-amz-request-id', 'tx000004ad9af1c8789871c-006543e2f1-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.526 DEBUG:rgw_multi.tests:version3 id=9yz-6v4AnfZzI2XDXIEtf-D7JxwKQCO 2023-11-02T17:57:05.526 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:57:05.527 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:57:05.527 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=9yz-6v4AnfZzI2XDXIEtf-D7JxwKQCO 2023-11-02T17:57:05.527 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=9yz-6v4AnfZzI2XDXIEtf-D7JxwKQCO 2023-11-02T17:57:05.528 DEBUG:boto:Method: DELETE 2023-11-02T17:57:05.528 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=9yz-6v4AnfZzI2XDXIEtf-D7JxwKQCO 2023-11-02T17:57:05.528 DEBUG:boto:Data: 2023-11-02T17:57:05.529 DEBUG:boto:Headers: {} 2023-11-02T17:57:05.529 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:57:05.530 DEBUG:boto:Port: 8000 2023-11-02T17:57:05.530 DEBUG:boto:Params: {} 2023-11-02T17:57:05.531 DEBUG:boto:Token: None 2023-11-02T17:57:05.531 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-11/obj-a1?versionId=9yz-6v4AnfZzI2XDXIEtf-D7JxwKQCO 2023-11-02T17:57:05.531 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XvdV/dDAw/wHH/lyH4oj2+TYjCQ= 2023-11-02T17:57:05.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XvdV/dDAw/wHH/lyH4oj2+TYjCQ=', 'Content-Length': '0'} 2023-11-02T17:57:05.550 DEBUG:boto:Response headers: [('x-amz-version-id', '9yz-6v4AnfZzI2XDXIEtf-D7JxwKQCO'), ('x-amz-request-id', 'tx00000626283e159f7ef32-006543e2f1-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.551 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:57:05.551 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:57:05.552 DEBUG:boto:Method: HEAD 2023-11-02T17:57:05.552 DEBUG:boto:Path: /ibtljn-11/ 2023-11-02T17:57:05.552 DEBUG:boto:Data: 2023-11-02T17:57:05.553 DEBUG:boto:Headers: {} 2023-11-02T17:57:05.553 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:57:05.554 DEBUG:boto:Port: 8001 2023-11-02T17:57:05.554 DEBUG:boto:Params: {} 2023-11-02T17:57:05.555 DEBUG:boto:Token: None 2023-11-02T17:57:05.555 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-11/ 2023-11-02T17:57:05.555 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y+BQtlezlhID+KOyA95alW9KUwM= 2023-11-02T17:57:05.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y+BQtlezlhID+KOyA95alW9KUwM=', 'Content-Length': '0'} 2023-11-02T17:57:05.559 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4c21a4b93940984-006543e2f1-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.559 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:57:05.560 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:57:05.560 DEBUG:boto:Method: PUT 2023-11-02T17:57:05.561 DEBUG:boto:Path: /ibtljn-11/obj-a2 2023-11-02T17:57:05.561 DEBUG:boto:Data: 2023-11-02T17:57:05.561 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:57:05.562 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:57:05.562 DEBUG:boto:Port: 8001 2023-11-02T17:57:05.563 DEBUG:boto:Params: {} 2023-11-02T17:57:05.563 DEBUG:boto:Token: None 2023-11-02T17:57:05.564 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-11/obj-a2 2023-11-02T17:57:05.564 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S5CdbC+MOCuODg+5GYHsJ9FjGv0= 2023-11-02T17:57:05.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S5CdbC+MOCuODg+5GYHsJ9FjGv0='} 2023-11-02T17:57:05.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-amz-request-id', 'tx0000099eab239170d18e5-006543e2f1-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.585 DEBUG:rgw_multi.tests:version1 id=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:57:05.585 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:57:05.586 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:57:05.586 DEBUG:boto:Method: PUT 2023-11-02T17:57:05.587 DEBUG:boto:Path: /ibtljn-11/obj-a2 2023-11-02T17:57:05.587 DEBUG:boto:Data: 2023-11-02T17:57:05.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:57:05.588 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:57:05.588 DEBUG:boto:Port: 8001 2023-11-02T17:57:05.589 DEBUG:boto:Params: {} 2023-11-02T17:57:05.589 DEBUG:boto:Token: None 2023-11-02T17:57:05.590 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-11/obj-a2 2023-11-02T17:57:05.590 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7LZWTp66KkiJIoZwbHbvct7od7I= 2023-11-02T17:57:05.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7LZWTp66KkiJIoZwbHbvct7od7I='} 2023-11-02T17:57:05.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1wNn0gva1vunVpIARQqeRPkVr5zoxr5'), ('x-amz-request-id', 'tx00000fecdfd839049aba8-006543e2f1-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.610 DEBUG:rgw_multi.tests:version2 id=1wNn0gva1vunVpIARQqeRPkVr5zoxr5 2023-11-02T17:57:05.611 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:57:05.611 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:57:05.612 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=1wNn0gva1vunVpIARQqeRPkVr5zoxr5 2023-11-02T17:57:05.612 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=1wNn0gva1vunVpIARQqeRPkVr5zoxr5 2023-11-02T17:57:05.612 DEBUG:boto:Method: DELETE 2023-11-02T17:57:05.613 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=1wNn0gva1vunVpIARQqeRPkVr5zoxr5 2023-11-02T17:57:05.613 DEBUG:boto:Data: 2023-11-02T17:57:05.614 DEBUG:boto:Headers: {} 2023-11-02T17:57:05.614 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:57:05.615 DEBUG:boto:Port: 8001 2023-11-02T17:57:05.615 DEBUG:boto:Params: {} 2023-11-02T17:57:05.616 DEBUG:boto:Token: None 2023-11-02T17:57:05.616 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-11/obj-a2?versionId=1wNn0gva1vunVpIARQqeRPkVr5zoxr5 2023-11-02T17:57:05.616 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6FCvVcf0ktfkblE4i2K37he7/qI= 2023-11-02T17:57:05.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6FCvVcf0ktfkblE4i2K37he7/qI=', 'Content-Length': '0'} 2023-11-02T17:57:05.640 DEBUG:boto:Response headers: [('x-amz-version-id', '1wNn0gva1vunVpIARQqeRPkVr5zoxr5'), ('x-amz-request-id', 'tx00000dc35d25ab236d1ac-006543e2f1-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.640 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:57:05.641 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:57:05.641 DEBUG:boto:Method: PUT 2023-11-02T17:57:05.642 DEBUG:boto:Path: /ibtljn-11/obj-a2 2023-11-02T17:57:05.642 DEBUG:boto:Data: 2023-11-02T17:57:05.642 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:57:05.643 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:57:05.643 DEBUG:boto:Port: 8001 2023-11-02T17:57:05.644 DEBUG:boto:Params: {} 2023-11-02T17:57:05.644 DEBUG:boto:Token: None 2023-11-02T17:57:05.644 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-11/obj-a2 2023-11-02T17:57:05.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jbEYvNuVpYdIdMjRDVEcKMRYBAw= 2023-11-02T17:57:05.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jbEYvNuVpYdIdMjRDVEcKMRYBAw='} 2023-11-02T17:57:05.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '4x4mrWpNwbMH9CQtb4yWuBluek8nzXj'), ('x-amz-request-id', 'tx0000000dad70c3b4d8ffe-006543e2f1-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.691 DEBUG:rgw_multi.tests:version3 id=4x4mrWpNwbMH9CQtb4yWuBluek8nzXj 2023-11-02T17:57:05.691 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:57:05.692 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:57:05.692 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=4x4mrWpNwbMH9CQtb4yWuBluek8nzXj 2023-11-02T17:57:05.693 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=4x4mrWpNwbMH9CQtb4yWuBluek8nzXj 2023-11-02T17:57:05.693 DEBUG:boto:Method: DELETE 2023-11-02T17:57:05.693 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=4x4mrWpNwbMH9CQtb4yWuBluek8nzXj 2023-11-02T17:57:05.694 DEBUG:boto:Data: 2023-11-02T17:57:05.694 DEBUG:boto:Headers: {} 2023-11-02T17:57:05.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:57:05.695 DEBUG:boto:Port: 8001 2023-11-02T17:57:05.695 DEBUG:boto:Params: {} 2023-11-02T17:57:05.696 DEBUG:boto:Token: None 2023-11-02T17:57:05.696 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-11/obj-a2?versionId=4x4mrWpNwbMH9CQtb4yWuBluek8nzXj 2023-11-02T17:57:05.697 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LqZRFw2rj9TdbqLmiITxAMcSP7M= 2023-11-02T17:57:05.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LqZRFw2rj9TdbqLmiITxAMcSP7M=', 'Content-Length': '0'} 2023-11-02T17:57:05.721 DEBUG:boto:Response headers: [('x-amz-version-id', '4x4mrWpNwbMH9CQtb4yWuBluek8nzXj'), ('x-amz-request-id', 'tx0000044f7097ed945c766-006543e2f1-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.722 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:57:05.723 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:57:05.723 DEBUG:boto:Method: HEAD 2023-11-02T17:57:05.724 DEBUG:boto:Path: /ibtljn-12/ 2023-11-02T17:57:05.724 DEBUG:boto:Data: 2023-11-02T17:57:05.724 DEBUG:boto:Headers: {} 2023-11-02T17:57:05.725 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:57:05.725 DEBUG:boto:Port: 8000 2023-11-02T17:57:05.726 DEBUG:boto:Params: {} 2023-11-02T17:57:05.726 DEBUG:boto:Token: None 2023-11-02T17:57:05.727 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-12/ 2023-11-02T17:57:05.727 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OtWCNg1nuqJI8RsUGioMGZm67bM= 2023-11-02T17:57:05.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OtWCNg1nuqJI8RsUGioMGZm67bM=', 'Content-Length': '0'} 2023-11-02T17:57:05.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1b87f80c897a85e-006543e2f1-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.734 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:57:05.734 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:57:05.735 DEBUG:boto:Method: PUT 2023-11-02T17:57:05.735 DEBUG:boto:Path: /ibtljn-12/obj-a1 2023-11-02T17:57:05.736 DEBUG:boto:Data: 2023-11-02T17:57:05.736 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:57:05.736 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:57:05.737 DEBUG:boto:Port: 8000 2023-11-02T17:57:05.737 DEBUG:boto:Params: {} 2023-11-02T17:57:05.738 DEBUG:boto:Token: None 2023-11-02T17:57:05.738 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-12/obj-a1 2023-11-02T17:57:05.739 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OEi9mMPrATkrT8IjM9N+mBiC9so= 2023-11-02T17:57:05.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OEi9mMPrATkrT8IjM9N+mBiC9so='} 2023-11-02T17:57:05.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-amz-request-id', 'tx00000b60118522bc6073a-006543e2f1-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.762 DEBUG:rgw_multi.tests:version1 id=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:57:05.763 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:57:05.763 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:57:05.764 DEBUG:boto:Method: PUT 2023-11-02T17:57:05.764 DEBUG:boto:Path: /ibtljn-12/obj-a1 2023-11-02T17:57:05.765 DEBUG:boto:Data: 2023-11-02T17:57:05.765 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:57:05.765 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:57:05.766 DEBUG:boto:Port: 8000 2023-11-02T17:57:05.766 DEBUG:boto:Params: {} 2023-11-02T17:57:05.767 DEBUG:boto:Token: None 2023-11-02T17:57:05.767 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-12/obj-a1 2023-11-02T17:57:05.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xpTGjnlfDL03r+7Nz+MuxPQTzXk= 2023-11-02T17:57:05.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xpTGjnlfDL03r+7Nz+MuxPQTzXk='} 2023-11-02T17:57:05.789 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7wc0Yzx5IZiY7VLVk1gYpz-oODFnuGz'), ('x-amz-request-id', 'tx000001cbfa100dec86c6d-006543e2f1-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.789 DEBUG:rgw_multi.tests:version2 id=7wc0Yzx5IZiY7VLVk1gYpz-oODFnuGz 2023-11-02T17:57:05.790 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:57:05.790 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:57:05.791 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=7wc0Yzx5IZiY7VLVk1gYpz-oODFnuGz 2023-11-02T17:57:05.791 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=7wc0Yzx5IZiY7VLVk1gYpz-oODFnuGz 2023-11-02T17:57:05.791 DEBUG:boto:Method: DELETE 2023-11-02T17:57:05.792 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=7wc0Yzx5IZiY7VLVk1gYpz-oODFnuGz 2023-11-02T17:57:05.792 DEBUG:boto:Data: 2023-11-02T17:57:05.793 DEBUG:boto:Headers: {} 2023-11-02T17:57:05.793 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:57:05.793 DEBUG:boto:Port: 8000 2023-11-02T17:57:05.794 DEBUG:boto:Params: {} 2023-11-02T17:57:05.794 DEBUG:boto:Token: None 2023-11-02T17:57:05.795 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-12/obj-a1?versionId=7wc0Yzx5IZiY7VLVk1gYpz-oODFnuGz 2023-11-02T17:57:05.795 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+ixg4FH+klkyuB7ApdXXX2qiH0E= 2023-11-02T17:57:05.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+ixg4FH+klkyuB7ApdXXX2qiH0E=', 'Content-Length': '0'} 2023-11-02T17:57:05.816 DEBUG:boto:Response headers: [('x-amz-version-id', '7wc0Yzx5IZiY7VLVk1gYpz-oODFnuGz'), ('x-amz-request-id', 'tx00000025a0aa8417b509f-006543e2f1-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.817 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:57:05.817 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:57:05.818 DEBUG:boto:Method: PUT 2023-11-02T17:57:05.818 DEBUG:boto:Path: /ibtljn-12/obj-a1 2023-11-02T17:57:05.818 DEBUG:boto:Data: 2023-11-02T17:57:05.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:57:05.819 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:57:05.820 DEBUG:boto:Port: 8000 2023-11-02T17:57:05.820 DEBUG:boto:Params: {} 2023-11-02T17:57:05.821 DEBUG:boto:Token: None 2023-11-02T17:57:05.821 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-12/obj-a1 2023-11-02T17:57:05.822 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qbwYZxBywGB/HMOP5qgtW0ZsuGs= 2023-11-02T17:57:05.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qbwYZxBywGB/HMOP5qgtW0ZsuGs='} 2023-11-02T17:57:05.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OkICDozkFN4N1su.SDGTSh5jI.GT-ZU'), ('x-amz-request-id', 'tx000004f8b1ea6aee767d0-006543e2f1-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.867 DEBUG:rgw_multi.tests:version3 id=OkICDozkFN4N1su.SDGTSh5jI.GT-ZU 2023-11-02T17:57:05.868 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:57:05.868 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:57:05.869 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=OkICDozkFN4N1su.SDGTSh5jI.GT-ZU 2023-11-02T17:57:05.869 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=OkICDozkFN4N1su.SDGTSh5jI.GT-ZU 2023-11-02T17:57:05.869 DEBUG:boto:Method: DELETE 2023-11-02T17:57:05.870 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=OkICDozkFN4N1su.SDGTSh5jI.GT-ZU 2023-11-02T17:57:05.870 DEBUG:boto:Data: 2023-11-02T17:57:05.871 DEBUG:boto:Headers: {} 2023-11-02T17:57:05.871 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:57:05.871 DEBUG:boto:Port: 8000 2023-11-02T17:57:05.872 DEBUG:boto:Params: {} 2023-11-02T17:57:05.872 DEBUG:boto:Token: None 2023-11-02T17:57:05.873 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-12/obj-a1?versionId=OkICDozkFN4N1su.SDGTSh5jI.GT-ZU 2023-11-02T17:57:05.873 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qt4yMt/xLnlDDSV7ebJsgCaruY8= 2023-11-02T17:57:05.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qt4yMt/xLnlDDSV7ebJsgCaruY8=', 'Content-Length': '0'} 2023-11-02T17:57:05.896 DEBUG:boto:Response headers: [('x-amz-version-id', 'OkICDozkFN4N1su.SDGTSh5jI.GT-ZU'), ('x-amz-request-id', 'tx000007fedf5403f4ab19c-006543e2f1-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.897 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:57:05.897 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:57:05.898 DEBUG:boto:Method: HEAD 2023-11-02T17:57:05.898 DEBUG:boto:Path: /ibtljn-12/ 2023-11-02T17:57:05.899 DEBUG:boto:Data: 2023-11-02T17:57:05.899 DEBUG:boto:Headers: {} 2023-11-02T17:57:05.899 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:57:05.900 DEBUG:boto:Port: 8001 2023-11-02T17:57:05.900 DEBUG:boto:Params: {} 2023-11-02T17:57:05.901 DEBUG:boto:Token: None 2023-11-02T17:57:05.901 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-12/ 2023-11-02T17:57:05.902 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OtWCNg1nuqJI8RsUGioMGZm67bM= 2023-11-02T17:57:05.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OtWCNg1nuqJI8RsUGioMGZm67bM=', 'Content-Length': '0'} 2023-11-02T17:57:05.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b88569499b8587f-006543e2f1-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.906 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:57:05.907 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:57:05.907 DEBUG:boto:Method: PUT 2023-11-02T17:57:05.908 DEBUG:boto:Path: /ibtljn-12/obj-a2 2023-11-02T17:57:05.908 DEBUG:boto:Data: 2023-11-02T17:57:05.909 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:57:05.909 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:57:05.909 DEBUG:boto:Port: 8001 2023-11-02T17:57:05.910 DEBUG:boto:Params: {} 2023-11-02T17:57:05.910 DEBUG:boto:Token: None 2023-11-02T17:57:05.911 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-12/obj-a2 2023-11-02T17:57:05.911 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GXJhWRGcaVrS/Iq9O1mivrBtbCU= 2023-11-02T17:57:05.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXJhWRGcaVrS/Iq9O1mivrBtbCU='} 2023-11-02T17:57:05.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-amz-request-id', 'tx000005bb2e5dc0d576a3f-006543e2f1-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.938 DEBUG:rgw_multi.tests:version1 id=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:57:05.938 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:57:05.939 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:57:05.939 DEBUG:boto:Method: PUT 2023-11-02T17:57:05.939 DEBUG:boto:Path: /ibtljn-12/obj-a2 2023-11-02T17:57:05.940 DEBUG:boto:Data: 2023-11-02T17:57:05.940 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:57:05.941 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:57:05.941 DEBUG:boto:Port: 8001 2023-11-02T17:57:05.941 DEBUG:boto:Params: {} 2023-11-02T17:57:05.942 DEBUG:boto:Token: None 2023-11-02T17:57:05.942 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-12/obj-a2 2023-11-02T17:57:05.943 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HRGP9PgktIlq7aPSZSBSoEHQpRU= 2023-11-02T17:57:05.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HRGP9PgktIlq7aPSZSBSoEHQpRU='} 2023-11-02T17:57:05.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'mWQ1yufPbLeTk5faxC34XwaNVq-DzK2'), ('x-amz-request-id', 'tx00000d36c80ffa36371b5-006543e2f1-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.964 DEBUG:rgw_multi.tests:version2 id=mWQ1yufPbLeTk5faxC34XwaNVq-DzK2 2023-11-02T17:57:05.965 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:57:05.965 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:57:05.966 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=mWQ1yufPbLeTk5faxC34XwaNVq-DzK2 2023-11-02T17:57:05.966 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=mWQ1yufPbLeTk5faxC34XwaNVq-DzK2 2023-11-02T17:57:05.966 DEBUG:boto:Method: DELETE 2023-11-02T17:57:05.967 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=mWQ1yufPbLeTk5faxC34XwaNVq-DzK2 2023-11-02T17:57:05.967 DEBUG:boto:Data: 2023-11-02T17:57:05.968 DEBUG:boto:Headers: {} 2023-11-02T17:57:05.968 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:57:05.969 DEBUG:boto:Port: 8001 2023-11-02T17:57:05.969 DEBUG:boto:Params: {} 2023-11-02T17:57:05.969 DEBUG:boto:Token: None 2023-11-02T17:57:05.970 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-12/obj-a2?versionId=mWQ1yufPbLeTk5faxC34XwaNVq-DzK2 2023-11-02T17:57:05.970 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8yNmE7Nnx907bodQuv/sbibA8zw= 2023-11-02T17:57:05.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8yNmE7Nnx907bodQuv/sbibA8zw=', 'Content-Length': '0'} 2023-11-02T17:57:05.991 DEBUG:boto:Response headers: [('x-amz-version-id', 'mWQ1yufPbLeTk5faxC34XwaNVq-DzK2'), ('x-amz-request-id', 'tx00000bb7187cdc7527901-006543e2f1-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:05.992 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:57:05.993 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:57:05.993 DEBUG:boto:Method: PUT 2023-11-02T17:57:05.994 DEBUG:boto:Path: /ibtljn-12/obj-a2 2023-11-02T17:57:05.994 DEBUG:boto:Data: 2023-11-02T17:57:05.995 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:57:05.995 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:57:05.995 DEBUG:boto:Port: 8001 2023-11-02T17:57:05.996 DEBUG:boto:Params: {} 2023-11-02T17:57:05.996 DEBUG:boto:Token: None 2023-11-02T17:57:05.997 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 02 Nov 2023 17:57:05 GMT /ibtljn-12/obj-a2 2023-11-02T17:57:05.997 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pUYH+NHGUTt8NJ408wIObzSc88k= 2023-11-02T17:57:05.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:57:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pUYH+NHGUTt8NJ408wIObzSc88k='} 2023-11-02T17:57:06.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '4xYBVB-W5Bp4ct1w6ZcNMImH5RUj15I'), ('x-amz-request-id', 'tx000009655f012eeb3843d-006543e2f1-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:57:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:06.047 DEBUG:rgw_multi.tests:version3 id=4xYBVB-W5Bp4ct1w6ZcNMImH5RUj15I 2023-11-02T17:57:06.048 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:57:06.048 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:57:06.049 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=4xYBVB-W5Bp4ct1w6ZcNMImH5RUj15I 2023-11-02T17:57:06.049 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=4xYBVB-W5Bp4ct1w6ZcNMImH5RUj15I 2023-11-02T17:57:06.049 DEBUG:boto:Method: DELETE 2023-11-02T17:57:06.050 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=4xYBVB-W5Bp4ct1w6ZcNMImH5RUj15I 2023-11-02T17:57:06.050 DEBUG:boto:Data: 2023-11-02T17:57:06.051 DEBUG:boto:Headers: {} 2023-11-02T17:57:06.051 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:57:06.051 DEBUG:boto:Port: 8001 2023-11-02T17:57:06.052 DEBUG:boto:Params: {} 2023-11-02T17:57:06.052 DEBUG:boto:Token: None 2023-11-02T17:57:06.053 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 17:57:06 GMT /ibtljn-12/obj-a2?versionId=4xYBVB-W5Bp4ct1w6ZcNMImH5RUj15I 2023-11-02T17:57:06.053 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3boSNdUOsSDIFQ47CCOSHwStIyM= 2023-11-02T17:57:06.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:57:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3boSNdUOsSDIFQ47CCOSHwStIyM=', 'Content-Length': '0'} 2023-11-02T17:57:06.077 DEBUG:boto:Response headers: [('x-amz-version-id', '4xYBVB-W5Bp4ct1w6ZcNMImH5RUj15I'), ('x-amz-request-id', 'tx00000f80a70ac8ab0a7ec-006543e2f2-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:57:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:57:06.078 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-11 2023-11-02T17:57:06.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:57:06.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:57:06.079 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:57:06.133 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:57:06.134 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:57:06.206 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:06.195+0000 7f2c5ac5af40 1 waiting for incremental sync to catch up: 2023-11-02T17:57:06.206 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.9.11, , , , , ] 2023-11-02T17:57:06.207 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.24.6, ] 2023-11-02T17:57:11.209 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:11.203+0000 7f2c5ac5af40 1 waiting for incremental sync to catch up: 2023-11-02T17:57:11.209 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.9.11, , , , , ] 2023-11-02T17:57:11.209 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.24.6, ] 2023-11-02T17:57:16.211 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:16.203+0000 7f2c5ac5af40 1 waiting for incremental sync to catch up: 2023-11-02T17:57:16.211 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.9.11, , , , , ] 2023-11-02T17:57:16.212 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.24.6, ] 2023-11-02T17:57:21.213 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:21.207+0000 7f2c5ac5af40 1 waiting for incremental sync to catch up: 2023-11-02T17:57:21.214 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000001.7.6, ] 2023-11-02T17:57:21.214 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.24.6, ] 2023-11-02T17:57:26.216 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:26.211+0000 7f2c5ac5af40 1 waiting for incremental sync to catch up: 2023-11-02T17:57:26.216 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000001.7.6, ] 2023-11-02T17:57:26.217 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.24.6, ] 2023-11-02T17:57:31.219 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:31.211+0000 7f2c5ac5af40 1 waiting for incremental sync to catch up: 2023-11-02T17:57:31.219 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000001.7.6, ] 2023-11-02T17:57:31.219 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.24.6, ] 2023-11-02T17:57:36.221 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:36.215+0000 7f2c5ac5af40 1 waiting for incremental sync to catch up: 2023-11-02T17:57:36.222 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000001.7.6, ] 2023-11-02T17:57:36.222 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.24.6, ] 2023-11-02T17:57:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:41.219+0000 7f2c5ac5af40 1 bucket sync caught up with source: 2023-11-02T17:57:41.225 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000010.24.6, ] 2023-11-02T17:57:41.225 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.24.6, ] 2023-11-02T17:57:41.225 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:41.219+0000 7f2c5ac5af40 0 bucket checkpoint complete 2023-11-02T17:57:41.255 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-11 2023-11-02T17:57:41.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:57:41.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:57:41.256 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:57:41.308 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:57:41.309 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:57:41.375 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:41.367+0000 7f9376858f40 1 waiting for incremental sync to catch up: 2023-11-02T17:57:41.375 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.20.11, , , 00000000010.31.6, , ] 2023-11-02T17:57:41.376 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.20.11, , , 00000000010.31.6, 00000000002.18.11, ] 2023-11-02T17:57:46.378 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:46.370+0000 7f9376858f40 1 waiting for incremental sync to catch up: 2023-11-02T17:57:46.378 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.20.11, , , 00000000010.31.6, , ] 2023-11-02T17:57:46.378 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.20.11, , , 00000000010.31.6, 00000000002.18.11, ] 2023-11-02T17:57:51.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:51.374+0000 7f9376858f40 1 waiting for incremental sync to catch up: 2023-11-02T17:57:51.381 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.20.11, , , 00000000010.31.6, , ] 2023-11-02T17:57:51.381 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.20.11, , , 00000000010.31.6, 00000000002.18.11, ] 2023-11-02T17:57:56.383 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:57:56.374+0000 7f9376858f40 1 waiting for incremental sync to catch up: 2023-11-02T17:57:56.383 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.20.11, , , 00000000010.31.6, , ] 2023-11-02T17:57:56.384 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.20.11, , , 00000000010.31.6, 00000000002.18.11, ] 2023-11-02T17:58:01.385 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:01.378+0000 7f9376858f40 1 bucket sync caught up with source: 2023-11-02T17:58:01.386 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.20.11, , , 00000000010.31.6, 00000000002.18.11, ] 2023-11-02T17:58:01.386 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.20.11, , , 00000000010.31.6, 00000000002.18.11, ] 2023-11-02T17:58:01.386 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:01.378+0000 7f9376858f40 0 bucket checkpoint complete 2023-11-02T17:58:01.415 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-11 zones={a2, a1} 2023-11-02T17:58:01.416 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.416 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.417 DEBUG:boto:Method: HEAD 2023-11-02T17:58:01.417 DEBUG:boto:Path: /ibtljn-11/ 2023-11-02T17:58:01.418 DEBUG:boto:Data: 2023-11-02T17:58:01.418 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.419 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.419 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.419 DEBUG:boto:Params: {} 2023-11-02T17:58:01.420 DEBUG:boto:Token: None 2023-11-02T17:58:01.420 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/ 2023-11-02T17:58:01.421 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9CQLaGLMRv+OFjaAZtUey4IAPDA= 2023-11-02T17:58:01.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9CQLaGLMRv+OFjaAZtUey4IAPDA=', 'Content-Length': '0'} 2023-11-02T17:58:01.425 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d35b094aae20c27-006543e329-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.426 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.426 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.427 DEBUG:boto:Method: HEAD 2023-11-02T17:58:01.427 DEBUG:boto:Path: /ibtljn-11/ 2023-11-02T17:58:01.427 DEBUG:boto:Data: 2023-11-02T17:58:01.428 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.428 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.429 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.429 DEBUG:boto:Params: {} 2023-11-02T17:58:01.429 DEBUG:boto:Token: None 2023-11-02T17:58:01.430 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/ 2023-11-02T17:58:01.430 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9CQLaGLMRv+OFjaAZtUey4IAPDA= 2023-11-02T17:58:01.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9CQLaGLMRv+OFjaAZtUey4IAPDA=', 'Content-Length': '0'} 2023-11-02T17:58:01.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff32ec20d390fc71-006543e329-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.435 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T17:58:01.435 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.436 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.436 DEBUG:boto:path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:01.437 DEBUG:boto:auth_path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:01.437 DEBUG:boto:Method: GET 2023-11-02T17:58:01.437 DEBUG:boto:Path: /ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:01.438 DEBUG:boto:Data: 2023-11-02T17:58:01.438 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.439 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.439 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.439 DEBUG:boto:Params: {} 2023-11-02T17:58:01.440 DEBUG:boto:Token: None 2023-11-02T17:58:01.440 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/?versions 2023-11-02T17:58:01.441 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3KAmoupLW+iNSp6LBklaSnW5uzk= 2023-11-02T17:58:01.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3KAmoupLW+iNSp6LBklaSnW5uzk=', 'Content-Length': '0'} 2023-11-02T17:58:01.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f871ba820354408d-006543e329-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.447 DEBUG:boto:b'ibtljn-11999falsedummygRMVA8xSztZb81HITCjm-8FBwHTFtyMtrue2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1TsWgaTvYq-GNh2xE46pAYWldI9CUwmYtrue2023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3Mtrue2023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:01.448 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T17:58:01.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-11-02T17:58:01.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-11-02T17:58:01.449 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T17:58:01.449 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.450 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.450 DEBUG:boto:path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:01.451 DEBUG:boto:auth_path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:01.451 DEBUG:boto:Method: GET 2023-11-02T17:58:01.451 DEBUG:boto:Path: /ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:01.452 DEBUG:boto:Data: 2023-11-02T17:58:01.452 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.453 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.453 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.453 DEBUG:boto:Params: {} 2023-11-02T17:58:01.454 DEBUG:boto:Token: None 2023-11-02T17:58:01.454 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/?versions 2023-11-02T17:58:01.455 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3KAmoupLW+iNSp6LBklaSnW5uzk= 2023-11-02T17:58:01.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3KAmoupLW+iNSp6LBklaSnW5uzk=', 'Content-Length': '0'} 2023-11-02T17:58:01.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb894a5ff6c3bc2-006543e329-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.460 DEBUG:boto:b'ibtljn-11999falsedummygRMVA8xSztZb81HITCjm-8FBwHTFtyMtrue2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1TsWgaTvYq-GNh2xE46pAYWldI9CUwmYtrue2023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3Mtrue2023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:01.461 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T17:58:01.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-11-02T17:58:01.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-11-02T17:58:01.462 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.463 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.463 DEBUG:boto:path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:01.463 DEBUG:boto:auth_path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:01.464 DEBUG:boto:Method: GET 2023-11-02T17:58:01.464 DEBUG:boto:Path: /ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:01.465 DEBUG:boto:Data: 2023-11-02T17:58:01.465 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.465 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.466 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.466 DEBUG:boto:Params: {} 2023-11-02T17:58:01.467 DEBUG:boto:Token: None 2023-11-02T17:58:01.467 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/?versions 2023-11-02T17:58:01.468 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3KAmoupLW+iNSp6LBklaSnW5uzk= 2023-11-02T17:58:01.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3KAmoupLW+iNSp6LBklaSnW5uzk=', 'Content-Length': '0'} 2023-11-02T17:58:01.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ae2916075e42daf-006543e329-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.474 DEBUG:boto:b'ibtljn-11999falsedummygRMVA8xSztZb81HITCjm-8FBwHTFtyMtrue2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1TsWgaTvYq-GNh2xE46pAYWldI9CUwmYtrue2023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3Mtrue2023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:01.475 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.475 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.475 DEBUG:boto:path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:01.476 DEBUG:boto:auth_path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:01.476 DEBUG:boto:Method: GET 2023-11-02T17:58:01.477 DEBUG:boto:Path: /ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:01.477 DEBUG:boto:Data: 2023-11-02T17:58:01.478 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.478 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.478 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.479 DEBUG:boto:Params: {} 2023-11-02T17:58:01.479 DEBUG:boto:Token: None 2023-11-02T17:58:01.480 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/?versions 2023-11-02T17:58:01.480 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3KAmoupLW+iNSp6LBklaSnW5uzk= 2023-11-02T17:58:01.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3KAmoupLW+iNSp6LBklaSnW5uzk=', 'Content-Length': '0'} 2023-11-02T17:58:01.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000396b46c3fe397ed9-006543e329-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.486 DEBUG:boto:b'ibtljn-11999falsedummygRMVA8xSztZb81HITCjm-8FBwHTFtyMtrue2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1TsWgaTvYq-GNh2xE46pAYWldI9CUwmYtrue2023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3Mtrue2023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:01.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:58:01.487 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:01.487 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:01.488 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.488 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.489 DEBUG:boto:Method: GET 2023-11-02T17:58:01.489 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.490 DEBUG:boto:Data: 2023-11-02T17:58:01.490 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.490 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.491 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.491 DEBUG:boto:Params: {} 2023-11-02T17:58:01.492 DEBUG:boto:Token: None 2023-11-02T17:58:01.492 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.493 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+dAwtzfhWRv/g+uQ6Te+JBjxSNQ= 2023-11-02T17:58:01.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+dAwtzfhWRv/g+uQ6Te+JBjxSNQ=', 'Content-Length': '0'} 2023-11-02T17:58:01.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2197ffb67e0299b-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.496 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:01.497 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:01.497 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.498 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.498 DEBUG:boto:Method: GET 2023-11-02T17:58:01.498 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.499 DEBUG:boto:Data: 2023-11-02T17:58:01.499 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.500 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.500 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.501 DEBUG:boto:Params: {} 2023-11-02T17:58:01.501 DEBUG:boto:Token: None 2023-11-02T17:58:01.502 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.502 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+dAwtzfhWRv/g+uQ6Te+JBjxSNQ= 2023-11-02T17:58:01.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+dAwtzfhWRv/g+uQ6Te+JBjxSNQ=', 'Content-Length': '0'} 2023-11-02T17:58:01.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006c636fd536fa09fd-006543e329-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.505 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:01.506 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:01.506 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.507 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.507 DEBUG:boto:Method: HEAD 2023-11-02T17:58:01.508 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.508 DEBUG:boto:Data: 2023-11-02T17:58:01.508 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.509 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.509 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.510 DEBUG:boto:Params: {} 2023-11-02T17:58:01.510 DEBUG:boto:Token: None 2023-11-02T17:58:01.511 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.511 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BgQdObKgipRtjxceqfp6Grm7U14= 2023-11-02T17:58:01.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BgQdObKgipRtjxceqfp6Grm7U14=', 'Content-Length': '0'} 2023-11-02T17:58:01.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000009dbfdb7ec6ccaf3-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.515 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:01.515 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:01.515 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.516 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.516 DEBUG:boto:Method: HEAD 2023-11-02T17:58:01.516 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.517 DEBUG:boto:Data: 2023-11-02T17:58:01.517 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.518 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.518 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.518 DEBUG:boto:Params: {} 2023-11-02T17:58:01.519 DEBUG:boto:Token: None 2023-11-02T17:58:01.519 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.520 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BgQdObKgipRtjxceqfp6Grm7U14= 2023-11-02T17:58:01.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BgQdObKgipRtjxceqfp6Grm7U14=', 'Content-Length': '0'} 2023-11-02T17:58:01.523 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000777cb2c8598a0a43-006543e329-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:58:01.524 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:01.524 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:01.524 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.525 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.525 DEBUG:boto:Method: GET 2023-11-02T17:58:01.526 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.526 DEBUG:boto:Data: 2023-11-02T17:58:01.527 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.527 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.527 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.528 DEBUG:boto:Params: {} 2023-11-02T17:58:01.528 DEBUG:boto:Token: None 2023-11-02T17:58:01.529 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.529 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+dAwtzfhWRv/g+uQ6Te+JBjxSNQ= 2023-11-02T17:58:01.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+dAwtzfhWRv/g+uQ6Te+JBjxSNQ=', 'Content-Length': '0'} 2023-11-02T17:58:01.533 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003df7d057b940044c-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.533 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:01.534 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:01.534 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.535 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.535 DEBUG:boto:Method: GET 2023-11-02T17:58:01.535 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.536 DEBUG:boto:Data: 2023-11-02T17:58:01.536 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.537 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.537 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.537 DEBUG:boto:Params: {} 2023-11-02T17:58:01.538 DEBUG:boto:Token: None 2023-11-02T17:58:01.538 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.539 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+dAwtzfhWRv/g+uQ6Te+JBjxSNQ= 2023-11-02T17:58:01.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+dAwtzfhWRv/g+uQ6Te+JBjxSNQ=', 'Content-Length': '0'} 2023-11-02T17:58:01.541 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d0b6d767e531042-006543e329-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.542 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.543 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.543 DEBUG:boto:path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:01.543 DEBUG:boto:auth_path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:01.544 DEBUG:boto:Method: GET 2023-11-02T17:58:01.544 DEBUG:boto:Path: /ibtljn-11/?prefix=dummy 2023-11-02T17:58:01.544 DEBUG:boto:Data: 2023-11-02T17:58:01.545 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.545 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.546 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.546 DEBUG:boto:Params: {} 2023-11-02T17:58:01.546 DEBUG:boto:Token: None 2023-11-02T17:58:01.547 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/ 2023-11-02T17:58:01.547 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s= 2023-11-02T17:58:01.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s=', 'Content-Length': '0'} 2023-11-02T17:58:01.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062e8a700a4b73557-006543e329-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.594 DEBUG:boto:b'ibtljn-11dummy1000falsedummy2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:01.595 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.595 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.596 DEBUG:boto:path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:01.596 DEBUG:boto:auth_path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:01.597 DEBUG:boto:Method: GET 2023-11-02T17:58:01.597 DEBUG:boto:Path: /ibtljn-11/?prefix=dummy 2023-11-02T17:58:01.597 DEBUG:boto:Data: 2023-11-02T17:58:01.598 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.598 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.599 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.599 DEBUG:boto:Params: {} 2023-11-02T17:58:01.599 DEBUG:boto:Token: None 2023-11-02T17:58:01.600 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/ 2023-11-02T17:58:01.600 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s= 2023-11-02T17:58:01.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s=', 'Content-Length': '0'} 2023-11-02T17:58:01.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017fc50df8c52195b-006543e329-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.606 DEBUG:boto:b'ibtljn-11dummy1000falsedummy2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:01.607 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:01.607 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:01.608 DEBUG:boto:Method: HEAD 2023-11-02T17:58:01.608 DEBUG:boto:Path: /ibtljn-11/dummy 2023-11-02T17:58:01.609 DEBUG:boto:Data: 2023-11-02T17:58:01.609 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.609 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.610 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.610 DEBUG:boto:Params: {} 2023-11-02T17:58:01.611 DEBUG:boto:Token: None 2023-11-02T17:58:01.611 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/dummy 2023-11-02T17:58:01.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RJQozc6O2ggw4k6bhMLbBQSu+Bk= 2023-11-02T17:58:01.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RJQozc6O2ggw4k6bhMLbBQSu+Bk=', 'Content-Length': '0'} 2023-11-02T17:58:01.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006771e76a3059d4fd-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.616 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:01.617 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:01.617 DEBUG:boto:Method: HEAD 2023-11-02T17:58:01.617 DEBUG:boto:Path: /ibtljn-11/dummy 2023-11-02T17:58:01.618 DEBUG:boto:Data: 2023-11-02T17:58:01.618 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.619 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.620 DEBUG:boto:Params: {} 2023-11-02T17:58:01.620 DEBUG:boto:Token: None 2023-11-02T17:58:01.620 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/dummy 2023-11-02T17:58:01.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RJQozc6O2ggw4k6bhMLbBQSu+Bk= 2023-11-02T17:58:01.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RJQozc6O2ggw4k6bhMLbBQSu+Bk=', 'Content-Length': '0'} 2023-11-02T17:58:01.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3c9823c15e8258a-006543e329-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:58:01.627 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:01.627 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:01.628 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.628 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.628 DEBUG:boto:Method: GET 2023-11-02T17:58:01.629 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.629 DEBUG:boto:Data: 2023-11-02T17:58:01.630 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.630 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.634 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.634 DEBUG:boto:Params: {} 2023-11-02T17:58:01.635 DEBUG:boto:Token: None 2023-11-02T17:58:01.635 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.636 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+dAwtzfhWRv/g+uQ6Te+JBjxSNQ= 2023-11-02T17:58:01.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+dAwtzfhWRv/g+uQ6Te+JBjxSNQ=', 'Content-Length': '0'} 2023-11-02T17:58:01.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e9eb7d1f9f1d0bae-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.640 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:01.640 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:01.641 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.641 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.642 DEBUG:boto:Method: GET 2023-11-02T17:58:01.642 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.643 DEBUG:boto:Data: 2023-11-02T17:58:01.643 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.644 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.644 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.645 DEBUG:boto:Params: {} 2023-11-02T17:58:01.645 DEBUG:boto:Token: None 2023-11-02T17:58:01.646 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:01.646 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+dAwtzfhWRv/g+uQ6Te+JBjxSNQ= 2023-11-02T17:58:01.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+dAwtzfhWRv/g+uQ6Te+JBjxSNQ=', 'Content-Length': '0'} 2023-11-02T17:58:01.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000043be0322508899e8-006543e329-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.651 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.651 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.651 DEBUG:boto:path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:01.652 DEBUG:boto:auth_path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:01.652 DEBUG:boto:Method: GET 2023-11-02T17:58:01.653 DEBUG:boto:Path: /ibtljn-11/?prefix=dummy 2023-11-02T17:58:01.653 DEBUG:boto:Data: 2023-11-02T17:58:01.654 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.655 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.655 DEBUG:boto:Params: {} 2023-11-02T17:58:01.655 DEBUG:boto:Token: None 2023-11-02T17:58:01.656 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/ 2023-11-02T17:58:01.657 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s= 2023-11-02T17:58:01.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s=', 'Content-Length': '0'} 2023-11-02T17:58:01.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed05aee4ec278274-006543e329-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.706 DEBUG:boto:b'ibtljn-11dummy1000falsedummy2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:01.707 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.708 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.708 DEBUG:boto:path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:01.709 DEBUG:boto:auth_path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:01.709 DEBUG:boto:Method: GET 2023-11-02T17:58:01.710 DEBUG:boto:Path: /ibtljn-11/?prefix=dummy 2023-11-02T17:58:01.710 DEBUG:boto:Data: 2023-11-02T17:58:01.710 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.711 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.711 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.712 DEBUG:boto:Params: {} 2023-11-02T17:58:01.712 DEBUG:boto:Token: None 2023-11-02T17:58:01.713 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/ 2023-11-02T17:58:01.713 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s= 2023-11-02T17:58:01.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s=', 'Content-Length': '0'} 2023-11-02T17:58:01.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee6e180742224db9-006543e329-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.720 DEBUG:boto:b'ibtljn-11dummy1000falsedummy2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:01.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-11-02T17:58:01.721 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.721 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.722 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.722 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.723 DEBUG:boto:Method: GET 2023-11-02T17:58:01.723 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.724 DEBUG:boto:Data: 2023-11-02T17:58:01.724 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.725 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.725 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.726 DEBUG:boto:Params: {} 2023-11-02T17:58:01.726 DEBUG:boto:Token: None 2023-11-02T17:58:01.727 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.727 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RZT8SQaRyeuK0qVJ4FhPxcdDs4s= 2023-11-02T17:58:01.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RZT8SQaRyeuK0qVJ4FhPxcdDs4s=', 'Content-Length': '0'} 2023-11-02T17:58:01.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df3e8c3109d5504a-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.732 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.732 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.733 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.733 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.733 DEBUG:boto:Method: GET 2023-11-02T17:58:01.734 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.734 DEBUG:boto:Data: 2023-11-02T17:58:01.735 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.735 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.736 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.736 DEBUG:boto:Params: {} 2023-11-02T17:58:01.737 DEBUG:boto:Token: None 2023-11-02T17:58:01.737 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.738 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RZT8SQaRyeuK0qVJ4FhPxcdDs4s= 2023-11-02T17:58:01.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RZT8SQaRyeuK0qVJ4FhPxcdDs4s=', 'Content-Length': '0'} 2023-11-02T17:58:01.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a4971f00e8a122b-006543e329-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.786 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.787 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.787 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.788 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.788 DEBUG:boto:Method: HEAD 2023-11-02T17:58:01.789 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.789 DEBUG:boto:Data: 2023-11-02T17:58:01.789 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.790 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.791 DEBUG:boto:Params: {} 2023-11-02T17:58:01.791 DEBUG:boto:Token: None 2023-11-02T17:58:01.792 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.792 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F6OfzXrLiO43w4izcCoDm4cgVK0= 2023-11-02T17:58:01.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F6OfzXrLiO43w4izcCoDm4cgVK0=', 'Content-Length': '0'} 2023-11-02T17:58:01.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000553d567b06fd4e95-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.797 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.797 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.798 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.798 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.798 DEBUG:boto:Method: HEAD 2023-11-02T17:58:01.799 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.799 DEBUG:boto:Data: 2023-11-02T17:58:01.800 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.800 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.801 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.801 DEBUG:boto:Params: {} 2023-11-02T17:58:01.801 DEBUG:boto:Token: None 2023-11-02T17:58:01.802 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.802 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F6OfzXrLiO43w4izcCoDm4cgVK0= 2023-11-02T17:58:01.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F6OfzXrLiO43w4izcCoDm4cgVK0=', 'Content-Length': '0'} 2023-11-02T17:58:01.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7baf1052562fa46-006543e329-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-11-02T17:58:01.807 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.807 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.808 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.808 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.808 DEBUG:boto:Method: GET 2023-11-02T17:58:01.809 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.809 DEBUG:boto:Data: 2023-11-02T17:58:01.810 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.810 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.811 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.811 DEBUG:boto:Params: {} 2023-11-02T17:58:01.812 DEBUG:boto:Token: None 2023-11-02T17:58:01.812 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.813 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RZT8SQaRyeuK0qVJ4FhPxcdDs4s= 2023-11-02T17:58:01.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RZT8SQaRyeuK0qVJ4FhPxcdDs4s=', 'Content-Length': '0'} 2023-11-02T17:58:01.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002719a9b2ef425225-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.818 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.818 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.819 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.819 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.820 DEBUG:boto:Method: GET 2023-11-02T17:58:01.820 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.820 DEBUG:boto:Data: 2023-11-02T17:58:01.821 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.821 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.822 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.822 DEBUG:boto:Params: {} 2023-11-02T17:58:01.823 DEBUG:boto:Token: None 2023-11-02T17:58:01.823 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.824 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RZT8SQaRyeuK0qVJ4FhPxcdDs4s= 2023-11-02T17:58:01.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RZT8SQaRyeuK0qVJ4FhPxcdDs4s=', 'Content-Length': '0'} 2023-11-02T17:58:01.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b594e5f9aec2574-006543e329-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.828 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.828 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.829 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:01.829 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:01.830 DEBUG:boto:Method: GET 2023-11-02T17:58:01.830 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:01.830 DEBUG:boto:Data: 2023-11-02T17:58:01.831 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.831 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.832 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.832 DEBUG:boto:Params: {} 2023-11-02T17:58:01.833 DEBUG:boto:Token: None 2023-11-02T17:58:01.833 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/ 2023-11-02T17:58:01.834 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s= 2023-11-02T17:58:01.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s=', 'Content-Length': '0'} 2023-11-02T17:58:01.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a075efb23bc2113b-006543e329-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.839 DEBUG:boto:b'ibtljn-11obj-a11000falseobj-a12023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:01.840 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.841 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.841 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:01.841 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:01.842 DEBUG:boto:Method: GET 2023-11-02T17:58:01.842 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:01.843 DEBUG:boto:Data: 2023-11-02T17:58:01.843 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.844 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.844 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.845 DEBUG:boto:Params: {} 2023-11-02T17:58:01.845 DEBUG:boto:Token: None 2023-11-02T17:58:01.846 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/ 2023-11-02T17:58:01.846 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s= 2023-11-02T17:58:01.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s=', 'Content-Length': '0'} 2023-11-02T17:58:01.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e7d730f825b310-006543e329-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.852 DEBUG:boto:b'ibtljn-11obj-a11000falseobj-a12023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:01.853 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.854 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.854 DEBUG:boto:Method: HEAD 2023-11-02T17:58:01.854 DEBUG:boto:Path: /ibtljn-11/obj-a1 2023-11-02T17:58:01.855 DEBUG:boto:Data: 2023-11-02T17:58:01.855 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.856 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.856 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.856 DEBUG:boto:Params: {} 2023-11-02T17:58:01.857 DEBUG:boto:Token: None 2023-11-02T17:58:01.857 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a1 2023-11-02T17:58:01.858 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z1aN6Psgp1dVoSMcDp96xbCufoc= 2023-11-02T17:58:01.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z1aN6Psgp1dVoSMcDp96xbCufoc=', 'Content-Length': '0'} 2023-11-02T17:58:01.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e890d50c56e67a27-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.863 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.863 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.863 DEBUG:boto:Method: HEAD 2023-11-02T17:58:01.864 DEBUG:boto:Path: /ibtljn-11/obj-a1 2023-11-02T17:58:01.864 DEBUG:boto:Data: 2023-11-02T17:58:01.865 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.865 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.865 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.866 DEBUG:boto:Params: {} 2023-11-02T17:58:01.866 DEBUG:boto:Token: None 2023-11-02T17:58:01.867 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a1 2023-11-02T17:58:01.867 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z1aN6Psgp1dVoSMcDp96xbCufoc= 2023-11-02T17:58:01.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z1aN6Psgp1dVoSMcDp96xbCufoc=', 'Content-Length': '0'} 2023-11-02T17:58:01.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000296bc706a0008929-006543e329-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-11-02T17:58:01.873 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.874 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.874 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.875 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.875 DEBUG:boto:Method: GET 2023-11-02T17:58:01.876 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.876 DEBUG:boto:Data: 2023-11-02T17:58:01.876 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.877 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.877 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.878 DEBUG:boto:Params: {} 2023-11-02T17:58:01.878 DEBUG:boto:Token: None 2023-11-02T17:58:01.879 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.879 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RZT8SQaRyeuK0qVJ4FhPxcdDs4s= 2023-11-02T17:58:01.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RZT8SQaRyeuK0qVJ4FhPxcdDs4s=', 'Content-Length': '0'} 2023-11-02T17:58:01.883 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b6f56816775b0e9-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.926 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.927 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:01.927 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.928 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.928 DEBUG:boto:Method: GET 2023-11-02T17:58:01.928 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.929 DEBUG:boto:Data: 2023-11-02T17:58:01.929 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.930 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.931 DEBUG:boto:Params: {} 2023-11-02T17:58:01.931 DEBUG:boto:Token: None 2023-11-02T17:58:01.931 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:01.932 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RZT8SQaRyeuK0qVJ4FhPxcdDs4s= 2023-11-02T17:58:01.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RZT8SQaRyeuK0qVJ4FhPxcdDs4s=', 'Content-Length': '0'} 2023-11-02T17:58:01.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007299b423479dfa8e-006543e329-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.936 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.937 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.937 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:01.938 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:01.938 DEBUG:boto:Method: GET 2023-11-02T17:58:01.939 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:01.939 DEBUG:boto:Data: 2023-11-02T17:58:01.939 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.940 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.940 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.941 DEBUG:boto:Params: {} 2023-11-02T17:58:01.941 DEBUG:boto:Token: None 2023-11-02T17:58:01.942 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/ 2023-11-02T17:58:01.942 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s= 2023-11-02T17:58:01.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s=', 'Content-Length': '0'} 2023-11-02T17:58:01.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f66a76bbb5a2215-006543e329-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.948 DEBUG:boto:b'ibtljn-11obj-a11000falseobj-a12023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:01.949 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:01.949 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:01.950 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:01.950 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:01.950 DEBUG:boto:Method: GET 2023-11-02T17:58:01.951 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:01.951 DEBUG:boto:Data: 2023-11-02T17:58:01.952 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.952 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.952 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.953 DEBUG:boto:Params: {} 2023-11-02T17:58:01.953 DEBUG:boto:Token: None 2023-11-02T17:58:01.954 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/ 2023-11-02T17:58:01.954 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s= 2023-11-02T17:58:01.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sbMFEddFmv8Q9K48VLdMYCj23+s=', 'Content-Length': '0'} 2023-11-02T17:58:01.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dd6247a949397f0-006543e329-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.960 DEBUG:boto:b'ibtljn-11obj-a11000falseobj-a12023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:01.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-11-02T17:58:01.961 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:01.961 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:01.962 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.962 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.963 DEBUG:boto:Method: GET 2023-11-02T17:58:01.963 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.964 DEBUG:boto:Data: 2023-11-02T17:58:01.964 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.964 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.965 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.965 DEBUG:boto:Params: {} 2023-11-02T17:58:01.966 DEBUG:boto:Token: None 2023-11-02T17:58:01.966 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.967 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QkzxURAcs016drwUyB1AE8loRns= 2023-11-02T17:58:01.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QkzxURAcs016drwUyB1AE8loRns=', 'Content-Length': '0'} 2023-11-02T17:58:01.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000349c28639dcf5085-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.971 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:01.971 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:01.972 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.972 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.973 DEBUG:boto:Method: GET 2023-11-02T17:58:01.973 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.973 DEBUG:boto:Data: 2023-11-02T17:58:01.974 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.974 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.975 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.975 DEBUG:boto:Params: {} 2023-11-02T17:58:01.975 DEBUG:boto:Token: None 2023-11-02T17:58:01.976 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.976 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QkzxURAcs016drwUyB1AE8loRns= 2023-11-02T17:58:01.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QkzxURAcs016drwUyB1AE8loRns=', 'Content-Length': '0'} 2023-11-02T17:58:01.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002264cdc3866f2b96-006543e329-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.981 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:01.981 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:01.982 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.982 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.982 DEBUG:boto:Method: HEAD 2023-11-02T17:58:01.983 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.983 DEBUG:boto:Data: 2023-11-02T17:58:01.984 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.984 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:01.984 DEBUG:boto:Port: 8001 2023-11-02T17:58:01.985 DEBUG:boto:Params: {} 2023-11-02T17:58:01.985 DEBUG:boto:Token: None 2023-11-02T17:58:01.986 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.986 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ktCH2DJOH5SDR6jK3LbonuisWzU= 2023-11-02T17:58:01.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ktCH2DJOH5SDR6jK3LbonuisWzU=', 'Content-Length': '0'} 2023-11-02T17:58:01.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000994cf92cb3c476cc-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.990 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:01.990 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:01.991 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.991 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.991 DEBUG:boto:Method: HEAD 2023-11-02T17:58:01.992 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.992 DEBUG:boto:Data: 2023-11-02T17:58:01.993 DEBUG:boto:Headers: {} 2023-11-02T17:58:01.993 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:01.993 DEBUG:boto:Port: 8000 2023-11-02T17:58:01.994 DEBUG:boto:Params: {} 2023-11-02T17:58:01.994 DEBUG:boto:Token: None 2023-11-02T17:58:01.995 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:01 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:01.995 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ktCH2DJOH5SDR6jK3LbonuisWzU= 2023-11-02T17:58:01.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ktCH2DJOH5SDR6jK3LbonuisWzU=', 'Content-Length': '0'} 2023-11-02T17:58:01.998 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021a4566f51bc6ee3-006543e329-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:01.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-11-02T17:58:01.999 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:02.000 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:02.000 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.000 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.001 DEBUG:boto:Method: GET 2023-11-02T17:58:02.001 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.002 DEBUG:boto:Data: 2023-11-02T17:58:02.002 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.002 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.003 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.003 DEBUG:boto:Params: {} 2023-11-02T17:58:02.004 DEBUG:boto:Token: None 2023-11-02T17:58:02.004 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.005 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j19c2PCOqgZLbd2zTTaVF76F2Fo= 2023-11-02T17:58:02.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j19c2PCOqgZLbd2zTTaVF76F2Fo=', 'Content-Length': '0'} 2023-11-02T17:58:02.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f3da4ba731f6be1-006543e329-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.050 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:02.051 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:02.051 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.052 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.052 DEBUG:boto:Method: GET 2023-11-02T17:58:02.053 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.053 DEBUG:boto:Data: 2023-11-02T17:58:02.053 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.054 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.054 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.055 DEBUG:boto:Params: {} 2023-11-02T17:58:02.055 DEBUG:boto:Token: None 2023-11-02T17:58:02.056 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.056 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j19c2PCOqgZLbd2zTTaVF76F2Fo= 2023-11-02T17:58:02.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j19c2PCOqgZLbd2zTTaVF76F2Fo=', 'Content-Length': '0'} 2023-11-02T17:58:02.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000538c190e575305fc-006543e32a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.102 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:02.103 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:02.103 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:02.104 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:02.104 DEBUG:boto:Method: GET 2023-11-02T17:58:02.105 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:02.105 DEBUG:boto:Data: 2023-11-02T17:58:02.105 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.106 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.107 DEBUG:boto:Params: {} 2023-11-02T17:58:02.107 DEBUG:boto:Token: None 2023-11-02T17:58:02.107 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-11/ 2023-11-02T17:58:02.108 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jQjRtl6/SkkFaP/8xjl7kbJtnWk= 2023-11-02T17:58:02.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jQjRtl6/SkkFaP/8xjl7kbJtnWk=', 'Content-Length': '0'} 2023-11-02T17:58:02.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000595d266dff98c80e-006543e32a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.114 DEBUG:boto:b'ibtljn-11obj-a21000falseobj-a22023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:02.115 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:02.115 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:02.116 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:02.116 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:02.116 DEBUG:boto:Method: GET 2023-11-02T17:58:02.117 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:02.117 DEBUG:boto:Data: 2023-11-02T17:58:02.118 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.118 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.119 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.119 DEBUG:boto:Params: {} 2023-11-02T17:58:02.119 DEBUG:boto:Token: None 2023-11-02T17:58:02.120 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-11/ 2023-11-02T17:58:02.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jQjRtl6/SkkFaP/8xjl7kbJtnWk= 2023-11-02T17:58:02.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jQjRtl6/SkkFaP/8xjl7kbJtnWk=', 'Content-Length': '0'} 2023-11-02T17:58:02.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df5a5db54e166c8e-006543e32a-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.170 DEBUG:boto:b'ibtljn-11obj-a21000falseobj-a22023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:02.171 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:02.172 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:02.172 DEBUG:boto:Method: HEAD 2023-11-02T17:58:02.172 DEBUG:boto:Path: /ibtljn-11/obj-a2 2023-11-02T17:58:02.173 DEBUG:boto:Data: 2023-11-02T17:58:02.173 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.174 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.174 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.175 DEBUG:boto:Params: {} 2023-11-02T17:58:02.175 DEBUG:boto:Token: None 2023-11-02T17:58:02.176 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-11/obj-a2 2023-11-02T17:58:02.176 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DmFZW2VkM1MQUb7Z9JNUrPU6jww= 2023-11-02T17:58:02.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DmFZW2VkM1MQUb7Z9JNUrPU6jww=', 'Content-Length': '0'} 2023-11-02T17:58:02.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004bafc1e85109194a-006543e32a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.181 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:02.181 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:02.182 DEBUG:boto:Method: HEAD 2023-11-02T17:58:02.182 DEBUG:boto:Path: /ibtljn-11/obj-a2 2023-11-02T17:58:02.183 DEBUG:boto:Data: 2023-11-02T17:58:02.183 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.184 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.184 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.184 DEBUG:boto:Params: {} 2023-11-02T17:58:02.185 DEBUG:boto:Token: None 2023-11-02T17:58:02.185 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-11/obj-a2 2023-11-02T17:58:02.186 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DmFZW2VkM1MQUb7Z9JNUrPU6jww= 2023-11-02T17:58:02.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DmFZW2VkM1MQUb7Z9JNUrPU6jww=', 'Content-Length': '0'} 2023-11-02T17:58:02.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005124d2d0cc873d3e-006543e32a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-11-02T17:58:02.191 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:02.192 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:02.192 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.193 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.193 DEBUG:boto:Method: GET 2023-11-02T17:58:02.193 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.194 DEBUG:boto:Data: 2023-11-02T17:58:02.194 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.195 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.195 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.196 DEBUG:boto:Params: {} 2023-11-02T17:58:02.196 DEBUG:boto:Token: None 2023-11-02T17:58:02.197 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.197 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j19c2PCOqgZLbd2zTTaVF76F2Fo= 2023-11-02T17:58:02.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j19c2PCOqgZLbd2zTTaVF76F2Fo=', 'Content-Length': '0'} 2023-11-02T17:58:02.200 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001489229b1c59a884-006543e32a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.201 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:02.201 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:02.202 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.202 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.203 DEBUG:boto:Method: GET 2023-11-02T17:58:02.203 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.204 DEBUG:boto:Data: 2023-11-02T17:58:02.204 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.204 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.205 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.205 DEBUG:boto:Params: {} 2023-11-02T17:58:02.206 DEBUG:boto:Token: None 2023-11-02T17:58:02.206 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:02.206 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j19c2PCOqgZLbd2zTTaVF76F2Fo= 2023-11-02T17:58:02.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j19c2PCOqgZLbd2zTTaVF76F2Fo=', 'Content-Length': '0'} 2023-11-02T17:58:02.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003404bd66f64b2d6d-006543e32a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.254 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:02.255 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:02.255 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:02.256 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:02.256 DEBUG:boto:Method: GET 2023-11-02T17:58:02.256 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:02.257 DEBUG:boto:Data: 2023-11-02T17:58:02.257 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.258 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.258 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.258 DEBUG:boto:Params: {} 2023-11-02T17:58:02.259 DEBUG:boto:Token: None 2023-11-02T17:58:02.259 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-11/ 2023-11-02T17:58:02.260 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jQjRtl6/SkkFaP/8xjl7kbJtnWk= 2023-11-02T17:58:02.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jQjRtl6/SkkFaP/8xjl7kbJtnWk=', 'Content-Length': '0'} 2023-11-02T17:58:02.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5798997068dc82a-006543e32a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.266 DEBUG:boto:b'ibtljn-11obj-a21000falseobj-a22023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:02.267 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:02.267 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:02.267 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:02.268 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:02.268 DEBUG:boto:Method: GET 2023-11-02T17:58:02.269 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:02.269 DEBUG:boto:Data: 2023-11-02T17:58:02.270 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.270 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.271 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.271 DEBUG:boto:Params: {} 2023-11-02T17:58:02.271 DEBUG:boto:Token: None 2023-11-02T17:58:02.272 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-11/ 2023-11-02T17:58:02.272 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jQjRtl6/SkkFaP/8xjl7kbJtnWk= 2023-11-02T17:58:02.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jQjRtl6/SkkFaP/8xjl7kbJtnWk=', 'Content-Length': '0'} 2023-11-02T17:58:02.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be3153ab3815c25a-006543e32a-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.278 DEBUG:boto:b'ibtljn-11obj-a21000falseobj-a22023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:02.279 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-11 zones={a2, a1} 2023-11-02T17:58:02.279 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-12 2023-11-02T17:58:02.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:02.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:02.280 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:58:02.332 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:58:02.332 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:58:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:02.386+0000 7f44c7c93f40 1 bucket sync caught up with source: 2023-11-02T17:58:02.396 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.18.11, 00000000010.24.6, ] 2023-11-02T17:58:02.396 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.18.11, 00000000010.24.6, ] 2023-11-02T17:58:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:02.390+0000 7f44c7c93f40 0 bucket checkpoint complete 2023-11-02T17:58:02.422 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-12 2023-11-02T17:58:02.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:02.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:02.423 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:58:02.473 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:58:02.473 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:58:02.535 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:02.526+0000 7f2cc6c70f40 1 bucket sync caught up with source: 2023-11-02T17:58:02.536 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.45.10, , , 00000000010.43.6, 00000000003.18.10, ] 2023-11-02T17:58:02.536 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.45.10, , , 00000000010.43.6, 00000000003.18.10, ] 2023-11-02T17:58:02.536 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:02.526+0000 7f2cc6c70f40 0 bucket checkpoint complete 2023-11-02T17:58:02.561 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-12 zones={a2, a1} 2023-11-02T17:58:02.562 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:02.562 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:02.563 DEBUG:boto:Method: HEAD 2023-11-02T17:58:02.563 DEBUG:boto:Path: /ibtljn-12/ 2023-11-02T17:58:02.563 DEBUG:boto:Data: 2023-11-02T17:58:02.564 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.565 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.565 DEBUG:boto:Params: {} 2023-11-02T17:58:02.566 DEBUG:boto:Token: None 2023-11-02T17:58:02.566 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/ 2023-11-02T17:58:02.567 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dcidTZMI7M89jn8RIyXwx6Bj9Qc= 2023-11-02T17:58:02.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dcidTZMI7M89jn8RIyXwx6Bj9Qc=', 'Content-Length': '0'} 2023-11-02T17:58:02.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c47407242934d667-006543e32a-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.571 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:02.572 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:02.572 DEBUG:boto:Method: HEAD 2023-11-02T17:58:02.573 DEBUG:boto:Path: /ibtljn-12/ 2023-11-02T17:58:02.573 DEBUG:boto:Data: 2023-11-02T17:58:02.573 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.574 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.575 DEBUG:boto:Params: {} 2023-11-02T17:58:02.575 DEBUG:boto:Token: None 2023-11-02T17:58:02.576 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/ 2023-11-02T17:58:02.576 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dcidTZMI7M89jn8RIyXwx6Bj9Qc= 2023-11-02T17:58:02.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dcidTZMI7M89jn8RIyXwx6Bj9Qc=', 'Content-Length': '0'} 2023-11-02T17:58:02.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003caec23977e852a9-006543e32a-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.581 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T17:58:02.581 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:02.581 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:02.582 DEBUG:boto:path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:02.582 DEBUG:boto:auth_path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:02.583 DEBUG:boto:Method: GET 2023-11-02T17:58:02.583 DEBUG:boto:Path: /ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:02.583 DEBUG:boto:Data: 2023-11-02T17:58:02.584 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.584 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.584 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.585 DEBUG:boto:Params: {} 2023-11-02T17:58:02.585 DEBUG:boto:Token: None 2023-11-02T17:58:02.586 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/?versions 2023-11-02T17:58:02.586 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tCNGaZslYc16VSORRbDIwO473Zk= 2023-11-02T17:58:02.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tCNGaZslYc16VSORRbDIwO473Zk=', 'Content-Length': '0'} 2023-11-02T17:58:02.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027610eb00c92fa90-006543e32a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.592 DEBUG:boto:b'ibtljn-12999falsedummyNCgTgcQgi48JNmlzIJZc2XV1YwJaH71true2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaEtrue2023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2iLD9aIguC0onhAlOj.xw657ssc7CVJGtrue2023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:02.593 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T17:58:02.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-11-02T17:58:02.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-11-02T17:58:02.594 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T17:58:02.595 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:02.595 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:02.595 DEBUG:boto:path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:02.596 DEBUG:boto:auth_path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:02.596 DEBUG:boto:Method: GET 2023-11-02T17:58:02.597 DEBUG:boto:Path: /ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:02.597 DEBUG:boto:Data: 2023-11-02T17:58:02.597 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.598 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.598 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.599 DEBUG:boto:Params: {} 2023-11-02T17:58:02.599 DEBUG:boto:Token: None 2023-11-02T17:58:02.600 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/?versions 2023-11-02T17:58:02.600 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tCNGaZslYc16VSORRbDIwO473Zk= 2023-11-02T17:58:02.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tCNGaZslYc16VSORRbDIwO473Zk=', 'Content-Length': '0'} 2023-11-02T17:58:02.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4c2b52242fc7e9c-006543e32a-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.606 DEBUG:boto:b'ibtljn-12999falsedummyNCgTgcQgi48JNmlzIJZc2XV1YwJaH71true2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaEtrue2023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2iLD9aIguC0onhAlOj.xw657ssc7CVJGtrue2023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:02.607 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T17:58:02.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-11-02T17:58:02.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-11-02T17:58:02.608 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:02.608 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:02.609 DEBUG:boto:path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:02.609 DEBUG:boto:auth_path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:02.610 DEBUG:boto:Method: GET 2023-11-02T17:58:02.610 DEBUG:boto:Path: /ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:02.611 DEBUG:boto:Data: 2023-11-02T17:58:02.611 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.611 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.612 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.612 DEBUG:boto:Params: {} 2023-11-02T17:58:02.613 DEBUG:boto:Token: None 2023-11-02T17:58:02.613 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/?versions 2023-11-02T17:58:02.613 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tCNGaZslYc16VSORRbDIwO473Zk= 2023-11-02T17:58:02.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tCNGaZslYc16VSORRbDIwO473Zk=', 'Content-Length': '0'} 2023-11-02T17:58:02.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f22c5cf4b467c6b7-006543e32a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.620 DEBUG:boto:b'ibtljn-12999falsedummyNCgTgcQgi48JNmlzIJZc2XV1YwJaH71true2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaEtrue2023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2iLD9aIguC0onhAlOj.xw657ssc7CVJGtrue2023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:02.620 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:02.621 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:02.621 DEBUG:boto:path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:02.622 DEBUG:boto:auth_path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:02.622 DEBUG:boto:Method: GET 2023-11-02T17:58:02.623 DEBUG:boto:Path: /ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:02.623 DEBUG:boto:Data: 2023-11-02T17:58:02.623 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.624 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.624 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.625 DEBUG:boto:Params: {} 2023-11-02T17:58:02.625 DEBUG:boto:Token: None 2023-11-02T17:58:02.626 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/?versions 2023-11-02T17:58:02.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tCNGaZslYc16VSORRbDIwO473Zk= 2023-11-02T17:58:02.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tCNGaZslYc16VSORRbDIwO473Zk=', 'Content-Length': '0'} 2023-11-02T17:58:02.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072ba3574732fcf19-006543e32a-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.632 DEBUG:boto:b'ibtljn-12999falsedummyNCgTgcQgi48JNmlzIJZc2XV1YwJaH71true2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaEtrue2023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2iLD9aIguC0onhAlOj.xw657ssc7CVJGtrue2023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:02.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:58:02.633 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:02.633 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:02.634 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.634 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.635 DEBUG:boto:Method: GET 2023-11-02T17:58:02.635 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.635 DEBUG:boto:Data: 2023-11-02T17:58:02.636 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.636 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.637 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.637 DEBUG:boto:Params: {} 2023-11-02T17:58:02.638 DEBUG:boto:Token: None 2023-11-02T17:58:02.638 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.639 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OmkqCcWssoOuW8SCH57G6rplojo= 2023-11-02T17:58:02.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OmkqCcWssoOuW8SCH57G6rplojo=', 'Content-Length': '0'} 2023-11-02T17:58:02.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000faefbd91b75255bc-006543e32a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.643 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:02.643 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:02.643 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.644 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.644 DEBUG:boto:Method: GET 2023-11-02T17:58:02.644 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.645 DEBUG:boto:Data: 2023-11-02T17:58:02.645 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.646 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.646 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.647 DEBUG:boto:Params: {} 2023-11-02T17:58:02.647 DEBUG:boto:Token: None 2023-11-02T17:58:02.648 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.648 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OmkqCcWssoOuW8SCH57G6rplojo= 2023-11-02T17:58:02.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OmkqCcWssoOuW8SCH57G6rplojo=', 'Content-Length': '0'} 2023-11-02T17:58:02.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cdfcf0cbfa5962bf-006543e32a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.652 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:02.652 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:02.653 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.653 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.653 DEBUG:boto:Method: HEAD 2023-11-02T17:58:02.654 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.654 DEBUG:boto:Data: 2023-11-02T17:58:02.655 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.655 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.655 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.656 DEBUG:boto:Params: {} 2023-11-02T17:58:02.656 DEBUG:boto:Token: None 2023-11-02T17:58:02.656 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.657 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AKVpdTSE8m/brTpMw/rMvg37ozE= 2023-11-02T17:58:02.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AKVpdTSE8m/brTpMw/rMvg37ozE=', 'Content-Length': '0'} 2023-11-02T17:58:02.660 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cfc2daa04436b97e-006543e32a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.661 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:02.661 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:02.661 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.662 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.662 DEBUG:boto:Method: HEAD 2023-11-02T17:58:02.663 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.663 DEBUG:boto:Data: 2023-11-02T17:58:02.664 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.664 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.664 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.665 DEBUG:boto:Params: {} 2023-11-02T17:58:02.665 DEBUG:boto:Token: None 2023-11-02T17:58:02.666 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.666 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AKVpdTSE8m/brTpMw/rMvg37ozE= 2023-11-02T17:58:02.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AKVpdTSE8m/brTpMw/rMvg37ozE=', 'Content-Length': '0'} 2023-11-02T17:58:02.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d1200ab758c2668a-006543e32a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:58:02.671 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:02.671 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:02.672 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.672 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.672 DEBUG:boto:Method: GET 2023-11-02T17:58:02.673 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.673 DEBUG:boto:Data: 2023-11-02T17:58:02.673 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.674 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.675 DEBUG:boto:Params: {} 2023-11-02T17:58:02.675 DEBUG:boto:Token: None 2023-11-02T17:58:02.676 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OmkqCcWssoOuW8SCH57G6rplojo= 2023-11-02T17:58:02.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OmkqCcWssoOuW8SCH57G6rplojo=', 'Content-Length': '0'} 2023-11-02T17:58:02.680 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fd2446613c84127a-006543e32a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.680 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:02.681 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:02.681 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.681 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.682 DEBUG:boto:Method: GET 2023-11-02T17:58:02.682 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.683 DEBUG:boto:Data: 2023-11-02T17:58:02.683 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.683 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.684 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.684 DEBUG:boto:Params: {} 2023-11-02T17:58:02.685 DEBUG:boto:Token: None 2023-11-02T17:58:02.685 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.685 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OmkqCcWssoOuW8SCH57G6rplojo= 2023-11-02T17:58:02.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OmkqCcWssoOuW8SCH57G6rplojo=', 'Content-Length': '0'} 2023-11-02T17:58:02.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b9718060af9cabf-006543e32a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.689 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:02.690 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:02.690 DEBUG:boto:path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:02.691 DEBUG:boto:auth_path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:02.691 DEBUG:boto:Method: GET 2023-11-02T17:58:02.692 DEBUG:boto:Path: /ibtljn-12/?prefix=dummy 2023-11-02T17:58:02.692 DEBUG:boto:Data: 2023-11-02T17:58:02.692 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.693 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.693 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.693 DEBUG:boto:Params: {} 2023-11-02T17:58:02.694 DEBUG:boto:Token: None 2023-11-02T17:58:02.694 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/ 2023-11-02T17:58:02.695 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AVXNGOwNdZWKBfzEZU9pSsCsgHY= 2023-11-02T17:58:02.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AVXNGOwNdZWKBfzEZU9pSsCsgHY=', 'Content-Length': '0'} 2023-11-02T17:58:02.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce9b73e345400d07-006543e32a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.742 DEBUG:boto:b'ibtljn-12dummy1000falsedummy2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:02.743 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:02.744 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:02.744 DEBUG:boto:path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:02.744 DEBUG:boto:auth_path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:02.745 DEBUG:boto:Method: GET 2023-11-02T17:58:02.745 DEBUG:boto:Path: /ibtljn-12/?prefix=dummy 2023-11-02T17:58:02.746 DEBUG:boto:Data: 2023-11-02T17:58:02.746 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.746 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.747 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.747 DEBUG:boto:Params: {} 2023-11-02T17:58:02.748 DEBUG:boto:Token: None 2023-11-02T17:58:02.748 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/ 2023-11-02T17:58:02.749 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AVXNGOwNdZWKBfzEZU9pSsCsgHY= 2023-11-02T17:58:02.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AVXNGOwNdZWKBfzEZU9pSsCsgHY=', 'Content-Length': '0'} 2023-11-02T17:58:02.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce556c1e5b084073-006543e32a-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.755 DEBUG:boto:b'ibtljn-12dummy1000falsedummy2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:02.755 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:02.756 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:02.756 DEBUG:boto:Method: HEAD 2023-11-02T17:58:02.757 DEBUG:boto:Path: /ibtljn-12/dummy 2023-11-02T17:58:02.757 DEBUG:boto:Data: 2023-11-02T17:58:02.757 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.758 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.758 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.759 DEBUG:boto:Params: {} 2023-11-02T17:58:02.759 DEBUG:boto:Token: None 2023-11-02T17:58:02.760 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/dummy 2023-11-02T17:58:02.760 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l2hFIVZJCDP4WcX1oN3ClW3zlUU= 2023-11-02T17:58:02.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l2hFIVZJCDP4WcX1oN3ClW3zlUU=', 'Content-Length': '0'} 2023-11-02T17:58:02.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c37076550b1726e-006543e32a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.765 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:02.765 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:02.765 DEBUG:boto:Method: HEAD 2023-11-02T17:58:02.766 DEBUG:boto:Path: /ibtljn-12/dummy 2023-11-02T17:58:02.766 DEBUG:boto:Data: 2023-11-02T17:58:02.767 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.767 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.767 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.768 DEBUG:boto:Params: {} 2023-11-02T17:58:02.768 DEBUG:boto:Token: None 2023-11-02T17:58:02.768 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/dummy 2023-11-02T17:58:02.769 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l2hFIVZJCDP4WcX1oN3ClW3zlUU= 2023-11-02T17:58:02.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l2hFIVZJCDP4WcX1oN3ClW3zlUU=', 'Content-Length': '0'} 2023-11-02T17:58:02.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057e8bfa40359b22c-006543e32a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:58:02.774 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:02.775 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:02.775 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.775 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.776 DEBUG:boto:Method: GET 2023-11-02T17:58:02.776 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.776 DEBUG:boto:Data: 2023-11-02T17:58:02.777 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.777 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.778 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.778 DEBUG:boto:Params: {} 2023-11-02T17:58:02.778 DEBUG:boto:Token: None 2023-11-02T17:58:02.779 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.779 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OmkqCcWssoOuW8SCH57G6rplojo= 2023-11-02T17:58:02.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OmkqCcWssoOuW8SCH57G6rplojo=', 'Content-Length': '0'} 2023-11-02T17:58:02.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f48ae3e642c8a11a-006543e32a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.783 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:02.784 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:02.784 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.785 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.785 DEBUG:boto:Method: GET 2023-11-02T17:58:02.785 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.786 DEBUG:boto:Data: 2023-11-02T17:58:02.786 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.787 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.787 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.787 DEBUG:boto:Params: {} 2023-11-02T17:58:02.788 DEBUG:boto:Token: None 2023-11-02T17:58:02.788 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:02.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OmkqCcWssoOuW8SCH57G6rplojo= 2023-11-02T17:58:02.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OmkqCcWssoOuW8SCH57G6rplojo=', 'Content-Length': '0'} 2023-11-02T17:58:02.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002eb256fa420fc575-006543e32a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.793 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:02.793 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:02.794 DEBUG:boto:path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:02.794 DEBUG:boto:auth_path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:02.794 DEBUG:boto:Method: GET 2023-11-02T17:58:02.795 DEBUG:boto:Path: /ibtljn-12/?prefix=dummy 2023-11-02T17:58:02.795 DEBUG:boto:Data: 2023-11-02T17:58:02.796 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.796 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.797 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.797 DEBUG:boto:Params: {} 2023-11-02T17:58:02.797 DEBUG:boto:Token: None 2023-11-02T17:58:02.798 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/ 2023-11-02T17:58:02.798 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AVXNGOwNdZWKBfzEZU9pSsCsgHY= 2023-11-02T17:58:02.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AVXNGOwNdZWKBfzEZU9pSsCsgHY=', 'Content-Length': '0'} 2023-11-02T17:58:02.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008509081013bed8a0-006543e32a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.846 DEBUG:boto:b'ibtljn-12dummy1000falsedummy2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:02.847 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:02.848 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:02.848 DEBUG:boto:path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:02.848 DEBUG:boto:auth_path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:02.849 DEBUG:boto:Method: GET 2023-11-02T17:58:02.849 DEBUG:boto:Path: /ibtljn-12/?prefix=dummy 2023-11-02T17:58:02.850 DEBUG:boto:Data: 2023-11-02T17:58:02.850 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.851 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.851 DEBUG:boto:Params: {} 2023-11-02T17:58:02.852 DEBUG:boto:Token: None 2023-11-02T17:58:02.852 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/ 2023-11-02T17:58:02.853 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AVXNGOwNdZWKBfzEZU9pSsCsgHY= 2023-11-02T17:58:02.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AVXNGOwNdZWKBfzEZU9pSsCsgHY=', 'Content-Length': '0'} 2023-11-02T17:58:02.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa57e505579dd703-006543e32a-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.859 DEBUG:boto:b'ibtljn-12dummy1000falsedummy2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:02.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-11-02T17:58:02.860 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:02.860 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:02.861 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.861 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.862 DEBUG:boto:Method: GET 2023-11-02T17:58:02.862 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.862 DEBUG:boto:Data: 2023-11-02T17:58:02.863 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.863 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.864 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.864 DEBUG:boto:Params: {} 2023-11-02T17:58:02.864 DEBUG:boto:Token: None 2023-11-02T17:58:02.865 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.865 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:br39/uPJl5Qm52BgJh5ku3fdKug= 2023-11-02T17:58:02.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:br39/uPJl5Qm52BgJh5ku3fdKug=', 'Content-Length': '0'} 2023-11-02T17:58:02.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020fa015876f5108e-006543e32a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.870 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:02.870 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:02.871 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.871 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.872 DEBUG:boto:Method: GET 2023-11-02T17:58:02.872 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.872 DEBUG:boto:Data: 2023-11-02T17:58:02.873 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.873 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.874 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.874 DEBUG:boto:Params: {} 2023-11-02T17:58:02.875 DEBUG:boto:Token: None 2023-11-02T17:58:02.875 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.876 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:br39/uPJl5Qm52BgJh5ku3fdKug= 2023-11-02T17:58:02.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:br39/uPJl5Qm52BgJh5ku3fdKug=', 'Content-Length': '0'} 2023-11-02T17:58:02.879 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b5b07c5bd4c5802-006543e32a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.922 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:02.923 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:02.923 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.924 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.924 DEBUG:boto:Method: HEAD 2023-11-02T17:58:02.925 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.925 DEBUG:boto:Data: 2023-11-02T17:58:02.925 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.926 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.926 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.927 DEBUG:boto:Params: {} 2023-11-02T17:58:02.927 DEBUG:boto:Token: None 2023-11-02T17:58:02.928 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.928 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Pkva6QAzgVEHnvidZYQj01eqdE= 2023-11-02T17:58:02.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Pkva6QAzgVEHnvidZYQj01eqdE=', 'Content-Length': '0'} 2023-11-02T17:58:02.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000339279d45cafad1f-006543e32a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.933 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:02.933 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:02.934 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.934 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.934 DEBUG:boto:Method: HEAD 2023-11-02T17:58:02.935 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.935 DEBUG:boto:Data: 2023-11-02T17:58:02.936 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.936 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.937 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.937 DEBUG:boto:Params: {} 2023-11-02T17:58:02.937 DEBUG:boto:Token: None 2023-11-02T17:58:02.938 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.938 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Pkva6QAzgVEHnvidZYQj01eqdE= 2023-11-02T17:58:02.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Pkva6QAzgVEHnvidZYQj01eqdE=', 'Content-Length': '0'} 2023-11-02T17:58:02.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc9ff401001c6dab-006543e32a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-11-02T17:58:02.943 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:02.943 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:02.943 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.944 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.944 DEBUG:boto:Method: GET 2023-11-02T17:58:02.945 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.945 DEBUG:boto:Data: 2023-11-02T17:58:02.946 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.946 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.946 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.947 DEBUG:boto:Params: {} 2023-11-02T17:58:02.947 DEBUG:boto:Token: None 2023-11-02T17:58:02.948 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.948 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:br39/uPJl5Qm52BgJh5ku3fdKug= 2023-11-02T17:58:02.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:br39/uPJl5Qm52BgJh5ku3fdKug=', 'Content-Length': '0'} 2023-11-02T17:58:02.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f9c3b9b95ac7e5b-006543e32a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.952 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:02.953 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:02.953 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.954 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.954 DEBUG:boto:Method: GET 2023-11-02T17:58:02.955 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.955 DEBUG:boto:Data: 2023-11-02T17:58:02.955 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.956 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:02.956 DEBUG:boto:Port: 8000 2023-11-02T17:58:02.957 DEBUG:boto:Params: {} 2023-11-02T17:58:02.957 DEBUG:boto:Token: None 2023-11-02T17:58:02.958 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:02.958 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:br39/uPJl5Qm52BgJh5ku3fdKug= 2023-11-02T17:58:02.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:br39/uPJl5Qm52BgJh5ku3fdKug=', 'Content-Length': '0'} 2023-11-02T17:58:02.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4937d098ba0b73f-006543e32a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:02.962 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:02.963 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:02.963 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:02.964 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:02.964 DEBUG:boto:Method: GET 2023-11-02T17:58:02.965 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:02.965 DEBUG:boto:Data: 2023-11-02T17:58:02.965 DEBUG:boto:Headers: {} 2023-11-02T17:58:02.966 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:02.966 DEBUG:boto:Port: 8001 2023-11-02T17:58:02.967 DEBUG:boto:Params: {} 2023-11-02T17:58:02.967 DEBUG:boto:Token: None 2023-11-02T17:58:02.968 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:02 GMT /ibtljn-12/ 2023-11-02T17:58:02.968 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AVXNGOwNdZWKBfzEZU9pSsCsgHY= 2023-11-02T17:58:02.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AVXNGOwNdZWKBfzEZU9pSsCsgHY=', 'Content-Length': '0'} 2023-11-02T17:58:02.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000333f72d84820096-006543e32a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.018 DEBUG:boto:b'ibtljn-12obj-a11000falseobj-a12023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:03.019 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:03.019 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:03.020 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:03.020 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:03.021 DEBUG:boto:Method: GET 2023-11-02T17:58:03.021 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:03.021 DEBUG:boto:Data: 2023-11-02T17:58:03.022 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.022 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.023 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.023 DEBUG:boto:Params: {} 2023-11-02T17:58:03.024 DEBUG:boto:Token: None 2023-11-02T17:58:03.024 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/ 2023-11-02T17:58:03.025 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8= 2023-11-02T17:58:03.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8=', 'Content-Length': '0'} 2023-11-02T17:58:03.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba66466cb79ceee5-006543e32b-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.031 DEBUG:boto:b'ibtljn-12obj-a11000falseobj-a12023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:03.032 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:03.032 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:03.032 DEBUG:boto:Method: HEAD 2023-11-02T17:58:03.033 DEBUG:boto:Path: /ibtljn-12/obj-a1 2023-11-02T17:58:03.033 DEBUG:boto:Data: 2023-11-02T17:58:03.033 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.034 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.034 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.035 DEBUG:boto:Params: {} 2023-11-02T17:58:03.035 DEBUG:boto:Token: None 2023-11-02T17:58:03.036 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a1 2023-11-02T17:58:03.036 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ScrG7t35E3b69ATgKNsFABww+MI= 2023-11-02T17:58:03.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ScrG7t35E3b69ATgKNsFABww+MI=', 'Content-Length': '0'} 2023-11-02T17:58:03.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084bed6cb87168d07-006543e32b-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.041 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:03.041 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:03.041 DEBUG:boto:Method: HEAD 2023-11-02T17:58:03.042 DEBUG:boto:Path: /ibtljn-12/obj-a1 2023-11-02T17:58:03.042 DEBUG:boto:Data: 2023-11-02T17:58:03.043 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.043 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.044 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.044 DEBUG:boto:Params: {} 2023-11-02T17:58:03.044 DEBUG:boto:Token: None 2023-11-02T17:58:03.045 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a1 2023-11-02T17:58:03.045 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ScrG7t35E3b69ATgKNsFABww+MI= 2023-11-02T17:58:03.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ScrG7t35E3b69ATgKNsFABww+MI=', 'Content-Length': '0'} 2023-11-02T17:58:03.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd7b570703332048-006543e32b-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-11-02T17:58:03.050 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:03.050 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:03.051 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:03.051 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:03.052 DEBUG:boto:Method: GET 2023-11-02T17:58:03.052 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:03.052 DEBUG:boto:Data: 2023-11-02T17:58:03.053 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.053 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.054 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.054 DEBUG:boto:Params: {} 2023-11-02T17:58:03.055 DEBUG:boto:Token: None 2023-11-02T17:58:03.055 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:03.056 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HtWl2hrL9IBJ5eBGrm/SMwBXYu0= 2023-11-02T17:58:03.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HtWl2hrL9IBJ5eBGrm/SMwBXYu0=', 'Content-Length': '0'} 2023-11-02T17:58:03.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028308b73f565580d-006543e32b-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.060 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:03.060 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:03.061 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:03.061 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:03.061 DEBUG:boto:Method: GET 2023-11-02T17:58:03.062 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:03.062 DEBUG:boto:Data: 2023-11-02T17:58:03.065 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.065 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.066 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.066 DEBUG:boto:Params: {} 2023-11-02T17:58:03.067 DEBUG:boto:Token: None 2023-11-02T17:58:03.067 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:03.068 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HtWl2hrL9IBJ5eBGrm/SMwBXYu0= 2023-11-02T17:58:03.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HtWl2hrL9IBJ5eBGrm/SMwBXYu0=', 'Content-Length': '0'} 2023-11-02T17:58:03.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009233954ca851af1e-006543e32b-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.072 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:03.073 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:03.073 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:03.073 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:03.074 DEBUG:boto:Method: GET 2023-11-02T17:58:03.074 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:03.075 DEBUG:boto:Data: 2023-11-02T17:58:03.075 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.076 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.076 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.076 DEBUG:boto:Params: {} 2023-11-02T17:58:03.077 DEBUG:boto:Token: None 2023-11-02T17:58:03.077 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/ 2023-11-02T17:58:03.078 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8= 2023-11-02T17:58:03.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8=', 'Content-Length': '0'} 2023-11-02T17:58:03.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8b37c1dbc680551-006543e32b-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.126 DEBUG:boto:b'ibtljn-12obj-a11000falseobj-a12023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:03.127 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:03.128 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:03.128 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:03.129 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:03.129 DEBUG:boto:Method: GET 2023-11-02T17:58:03.129 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:03.130 DEBUG:boto:Data: 2023-11-02T17:58:03.130 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.131 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.131 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.132 DEBUG:boto:Params: {} 2023-11-02T17:58:03.132 DEBUG:boto:Token: None 2023-11-02T17:58:03.133 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/ 2023-11-02T17:58:03.133 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8= 2023-11-02T17:58:03.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8=', 'Content-Length': '0'} 2023-11-02T17:58:03.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015545b635526acca-006543e32b-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.139 DEBUG:boto:b'ibtljn-12obj-a11000falseobj-a12023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:03.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-11-02T17:58:03.140 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.141 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.141 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.141 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.142 DEBUG:boto:Method: GET 2023-11-02T17:58:03.142 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.143 DEBUG:boto:Data: 2023-11-02T17:58:03.143 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.144 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.144 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.144 DEBUG:boto:Params: {} 2023-11-02T17:58:03.145 DEBUG:boto:Token: None 2023-11-02T17:58:03.145 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.146 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7FXwi+t1TL10vPa/fGN4FBWZMp8= 2023-11-02T17:58:03.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7FXwi+t1TL10vPa/fGN4FBWZMp8=', 'Content-Length': '0'} 2023-11-02T17:58:03.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c136d42d686fb2ae-006543e32b-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.150 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.151 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.151 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.152 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.152 DEBUG:boto:Method: GET 2023-11-02T17:58:03.153 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.153 DEBUG:boto:Data: 2023-11-02T17:58:03.153 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.154 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.154 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.155 DEBUG:boto:Params: {} 2023-11-02T17:58:03.155 DEBUG:boto:Token: None 2023-11-02T17:58:03.156 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.156 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7FXwi+t1TL10vPa/fGN4FBWZMp8= 2023-11-02T17:58:03.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7FXwi+t1TL10vPa/fGN4FBWZMp8=', 'Content-Length': '0'} 2023-11-02T17:58:03.159 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000163c5dbdd16bab61-006543e32b-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.202 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.203 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.203 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.204 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.204 DEBUG:boto:Method: HEAD 2023-11-02T17:58:03.205 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.205 DEBUG:boto:Data: 2023-11-02T17:58:03.205 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.206 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.206 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.207 DEBUG:boto:Params: {} 2023-11-02T17:58:03.207 DEBUG:boto:Token: None 2023-11-02T17:58:03.208 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aIIzeEk0gQmtlT15Zzkc92oseFc= 2023-11-02T17:58:03.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aIIzeEk0gQmtlT15Zzkc92oseFc=', 'Content-Length': '0'} 2023-11-02T17:58:03.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f40f47b85e2e406-006543e32b-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.212 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.213 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.213 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.214 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.214 DEBUG:boto:Method: HEAD 2023-11-02T17:58:03.214 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.215 DEBUG:boto:Data: 2023-11-02T17:58:03.215 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.216 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.216 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.216 DEBUG:boto:Params: {} 2023-11-02T17:58:03.217 DEBUG:boto:Token: None 2023-11-02T17:58:03.217 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.218 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aIIzeEk0gQmtlT15Zzkc92oseFc= 2023-11-02T17:58:03.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aIIzeEk0gQmtlT15Zzkc92oseFc=', 'Content-Length': '0'} 2023-11-02T17:58:03.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012939bb3e75872bc-006543e32b-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-11-02T17:58:03.222 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.223 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.223 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.223 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.224 DEBUG:boto:Method: GET 2023-11-02T17:58:03.224 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.225 DEBUG:boto:Data: 2023-11-02T17:58:03.225 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.226 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.226 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.227 DEBUG:boto:Params: {} 2023-11-02T17:58:03.227 DEBUG:boto:Token: None 2023-11-02T17:58:03.228 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.228 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7FXwi+t1TL10vPa/fGN4FBWZMp8= 2023-11-02T17:58:03.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7FXwi+t1TL10vPa/fGN4FBWZMp8=', 'Content-Length': '0'} 2023-11-02T17:58:03.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f2304820b6e8739-006543e32b-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.233 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.233 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.234 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.234 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.235 DEBUG:boto:Method: GET 2023-11-02T17:58:03.235 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.235 DEBUG:boto:Data: 2023-11-02T17:58:03.236 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.236 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.237 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.237 DEBUG:boto:Params: {} 2023-11-02T17:58:03.238 DEBUG:boto:Token: None 2023-11-02T17:58:03.238 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.239 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7FXwi+t1TL10vPa/fGN4FBWZMp8= 2023-11-02T17:58:03.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7FXwi+t1TL10vPa/fGN4FBWZMp8=', 'Content-Length': '0'} 2023-11-02T17:58:03.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f3ff9baa548f0b3-006543e32b-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.243 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:03.243 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:03.244 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:03.244 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:03.244 DEBUG:boto:Method: GET 2023-11-02T17:58:03.245 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:03.245 DEBUG:boto:Data: 2023-11-02T17:58:03.246 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.246 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.247 DEBUG:boto:Params: {} 2023-11-02T17:58:03.247 DEBUG:boto:Token: None 2023-11-02T17:58:03.248 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/ 2023-11-02T17:58:03.248 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8= 2023-11-02T17:58:03.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8=', 'Content-Length': '0'} 2023-11-02T17:58:03.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81caf1705e6bf66-006543e32b-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.298 DEBUG:boto:b'ibtljn-12obj-a21000falseobj-a22023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:03.299 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:03.300 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:03.300 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:03.300 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:03.301 DEBUG:boto:Method: GET 2023-11-02T17:58:03.301 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:03.302 DEBUG:boto:Data: 2023-11-02T17:58:03.302 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.303 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.304 DEBUG:boto:Params: {} 2023-11-02T17:58:03.304 DEBUG:boto:Token: None 2023-11-02T17:58:03.305 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/ 2023-11-02T17:58:03.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8= 2023-11-02T17:58:03.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8=', 'Content-Length': '0'} 2023-11-02T17:58:03.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053077441d0a6bcd7-006543e32b-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.312 DEBUG:boto:b'ibtljn-12obj-a21000falseobj-a22023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:03.313 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.313 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.314 DEBUG:boto:Method: HEAD 2023-11-02T17:58:03.314 DEBUG:boto:Path: /ibtljn-12/obj-a2 2023-11-02T17:58:03.315 DEBUG:boto:Data: 2023-11-02T17:58:03.315 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.315 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.316 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.316 DEBUG:boto:Params: {} 2023-11-02T17:58:03.316 DEBUG:boto:Token: None 2023-11-02T17:58:03.317 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a2 2023-11-02T17:58:03.317 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gCkLWOpiacIGLdKX3tuw+GagF+Y= 2023-11-02T17:58:03.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gCkLWOpiacIGLdKX3tuw+GagF+Y=', 'Content-Length': '0'} 2023-11-02T17:58:03.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065bb7b2f583c592d-006543e32b-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.324 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.324 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.325 DEBUG:boto:Method: HEAD 2023-11-02T17:58:03.325 DEBUG:boto:Path: /ibtljn-12/obj-a2 2023-11-02T17:58:03.326 DEBUG:boto:Data: 2023-11-02T17:58:03.326 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.327 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.327 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.327 DEBUG:boto:Params: {} 2023-11-02T17:58:03.328 DEBUG:boto:Token: None 2023-11-02T17:58:03.328 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a2 2023-11-02T17:58:03.329 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gCkLWOpiacIGLdKX3tuw+GagF+Y= 2023-11-02T17:58:03.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gCkLWOpiacIGLdKX3tuw+GagF+Y=', 'Content-Length': '0'} 2023-11-02T17:58:03.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e8daeb0355aeca3-006543e32b-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-11-02T17:58:03.334 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.335 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.335 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.335 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.336 DEBUG:boto:Method: GET 2023-11-02T17:58:03.336 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.337 DEBUG:boto:Data: 2023-11-02T17:58:03.337 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.337 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.338 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.338 DEBUG:boto:Params: {} 2023-11-02T17:58:03.339 DEBUG:boto:Token: None 2023-11-02T17:58:03.339 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.340 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7FXwi+t1TL10vPa/fGN4FBWZMp8= 2023-11-02T17:58:03.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7FXwi+t1TL10vPa/fGN4FBWZMp8=', 'Content-Length': '0'} 2023-11-02T17:58:03.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e21e0a4cbc021b9-006543e32b-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.344 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.344 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.345 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.345 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.346 DEBUG:boto:Method: GET 2023-11-02T17:58:03.346 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.347 DEBUG:boto:Data: 2023-11-02T17:58:03.347 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.347 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.348 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.348 DEBUG:boto:Params: {} 2023-11-02T17:58:03.349 DEBUG:boto:Token: None 2023-11-02T17:58:03.349 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:03.350 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7FXwi+t1TL10vPa/fGN4FBWZMp8= 2023-11-02T17:58:03.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7FXwi+t1TL10vPa/fGN4FBWZMp8=', 'Content-Length': '0'} 2023-11-02T17:58:03.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007799bd253e3ca043-006543e32b-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.394 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:03.395 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:03.395 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:03.396 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:03.396 DEBUG:boto:Method: GET 2023-11-02T17:58:03.397 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:03.397 DEBUG:boto:Data: 2023-11-02T17:58:03.397 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.398 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.399 DEBUG:boto:Params: {} 2023-11-02T17:58:03.399 DEBUG:boto:Token: None 2023-11-02T17:58:03.400 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/ 2023-11-02T17:58:03.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8= 2023-11-02T17:58:03.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8=', 'Content-Length': '0'} 2023-11-02T17:58:03.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049d98e84acd25225-006543e32b-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.406 DEBUG:boto:b'ibtljn-12obj-a21000falseobj-a22023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:03.407 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:03.407 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:03.408 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:03.408 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:03.408 DEBUG:boto:Method: GET 2023-11-02T17:58:03.409 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:03.409 DEBUG:boto:Data: 2023-11-02T17:58:03.410 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.410 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.411 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.411 DEBUG:boto:Params: {} 2023-11-02T17:58:03.411 DEBUG:boto:Token: None 2023-11-02T17:58:03.412 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/ 2023-11-02T17:58:03.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8= 2023-11-02T17:58:03.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bhe7erCygqjcWJOObJNFitC/P+8=', 'Content-Length': '0'} 2023-11-02T17:58:03.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061b56d20954ad95c-006543e32b-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.418 DEBUG:boto:b'ibtljn-12obj-a21000falseobj-a22023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:03.419 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-12 zones={a2, a1} 2023-11-02T17:58:03.420 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:03.420 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:03.420 DEBUG:boto:Method: HEAD 2023-11-02T17:58:03.421 DEBUG:boto:Path: /ibtljn-11/ 2023-11-02T17:58:03.421 DEBUG:boto:Data: 2023-11-02T17:58:03.421 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.422 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.423 DEBUG:boto:Params: {} 2023-11-02T17:58:03.423 DEBUG:boto:Token: None 2023-11-02T17:58:03.423 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-11/ 2023-11-02T17:58:03.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Xf2QZlWliK7fmvgEMrhGxEO33/w= 2023-11-02T17:58:03.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Xf2QZlWliK7fmvgEMrhGxEO33/w=', 'Content-Length': '0'} 2023-11-02T17:58:03.429 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000509a68de504d8517-006543e32b-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.429 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:03.430 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:03.430 DEBUG:boto:path=/ibtljn-11/?versions&max-keys=999&prefix=obj-a1 2023-11-02T17:58:03.430 DEBUG:boto:auth_path=/ibtljn-11/?versions&max-keys=999&prefix=obj-a1 2023-11-02T17:58:03.431 DEBUG:boto:Method: GET 2023-11-02T17:58:03.431 DEBUG:boto:Path: /ibtljn-11/?versions&max-keys=999&prefix=obj-a1 2023-11-02T17:58:03.432 DEBUG:boto:Data: 2023-11-02T17:58:03.432 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.432 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.433 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.433 DEBUG:boto:Params: {} 2023-11-02T17:58:03.434 DEBUG:boto:Token: None 2023-11-02T17:58:03.434 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-11/?versions 2023-11-02T17:58:03.435 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uZuA7bYaj1IuHE3pA3NMZKBPTiM= 2023-11-02T17:58:03.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uZuA7bYaj1IuHE3pA3NMZKBPTiM=', 'Content-Length': '0'} 2023-11-02T17:58:03.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003770c6e4bc61695c-006543e32b-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.482 DEBUG:boto:b'ibtljn-11obj-a1999falseobj-a1TsWgaTvYq-GNh2xE46pAYWldI9CUwmYtrue2023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:03.483 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:03.484 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:03.484 DEBUG:boto:path=/ibtljn-11/obj-a1?acl 2023-11-02T17:58:03.484 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?acl 2023-11-02T17:58:03.485 DEBUG:boto:Method: PUT 2023-11-02T17:58:03.485 DEBUG:boto:Path: /ibtljn-11/obj-a1?acl 2023-11-02T17:58:03.485 DEBUG:boto:Data: 2023-11-02T17:58:03.486 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-11-02T17:58:03.486 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.486 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.487 DEBUG:boto:Params: {} 2023-11-02T17:58:03.487 DEBUG:boto:Token: None 2023-11-02T17:58:03.488 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:58:03 GMT x-amz-acl:public-read /ibtljn-11/obj-a1?acl 2023-11-02T17:58:03.488 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LjWRhSYynnaj21Mu1//BaqEp5IU= 2023-11-02T17:58:03.489 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LjWRhSYynnaj21Mu1//BaqEp5IU=', 'Content-Length': '0'} 2023-11-02T17:58:03.504 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086604eb5201fb4b2-006543e32b-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.504 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:03.505 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:03.505 DEBUG:boto:Method: HEAD 2023-11-02T17:58:03.506 DEBUG:boto:Path: /ibtljn-11/ 2023-11-02T17:58:03.506 DEBUG:boto:Data: 2023-11-02T17:58:03.506 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.507 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.507 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.508 DEBUG:boto:Params: {} 2023-11-02T17:58:03.508 DEBUG:boto:Token: None 2023-11-02T17:58:03.508 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-11/ 2023-11-02T17:58:03.509 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Xf2QZlWliK7fmvgEMrhGxEO33/w= 2023-11-02T17:58:03.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Xf2QZlWliK7fmvgEMrhGxEO33/w=', 'Content-Length': '0'} 2023-11-02T17:58:03.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eefbfb5986793ba4-006543e32b-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.513 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:03.514 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:03.514 DEBUG:boto:path=/ibtljn-11/?versions&max-keys=999&prefix=obj-a2 2023-11-02T17:58:03.514 DEBUG:boto:auth_path=/ibtljn-11/?versions&max-keys=999&prefix=obj-a2 2023-11-02T17:58:03.515 DEBUG:boto:Method: GET 2023-11-02T17:58:03.515 DEBUG:boto:Path: /ibtljn-11/?versions&max-keys=999&prefix=obj-a2 2023-11-02T17:58:03.516 DEBUG:boto:Data: 2023-11-02T17:58:03.516 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.516 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.517 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.517 DEBUG:boto:Params: {} 2023-11-02T17:58:03.518 DEBUG:boto:Token: None 2023-11-02T17:58:03.518 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-11/?versions 2023-11-02T17:58:03.519 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uZuA7bYaj1IuHE3pA3NMZKBPTiM= 2023-11-02T17:58:03.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uZuA7bYaj1IuHE3pA3NMZKBPTiM=', 'Content-Length': '0'} 2023-11-02T17:58:03.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024d0e8338b362272-006543e32b-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.522 DEBUG:boto:b'ibtljn-11obj-a2999falseobj-a2b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3Mtrue2023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:03.523 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:03.523 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:03.524 DEBUG:boto:path=/ibtljn-11/obj-a2?acl 2023-11-02T17:58:03.524 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?acl 2023-11-02T17:58:03.525 DEBUG:boto:Method: PUT 2023-11-02T17:58:03.525 DEBUG:boto:Path: /ibtljn-11/obj-a2?acl 2023-11-02T17:58:03.525 DEBUG:boto:Data: 2023-11-02T17:58:03.526 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-11-02T17:58:03.526 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.527 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.527 DEBUG:boto:Params: {} 2023-11-02T17:58:03.527 DEBUG:boto:Token: None 2023-11-02T17:58:03.528 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:58:03 GMT x-amz-acl:public-read /ibtljn-11/obj-a2?acl 2023-11-02T17:58:03.528 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:43+0VixH/WVUeT1+mm9bQ40dB+E= 2023-11-02T17:58:03.529 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:43+0VixH/WVUeT1+mm9bQ40dB+E=', 'Content-Length': '0'} 2023-11-02T17:58:03.540 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002fce9bd10b75b1bf-006543e32b-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.540 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:03.541 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:03.541 DEBUG:boto:Method: HEAD 2023-11-02T17:58:03.541 DEBUG:boto:Path: /ibtljn-12/ 2023-11-02T17:58:03.542 DEBUG:boto:Data: 2023-11-02T17:58:03.542 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.543 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.543 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.543 DEBUG:boto:Params: {} 2023-11-02T17:58:03.544 DEBUG:boto:Token: None 2023-11-02T17:58:03.544 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/ 2023-11-02T17:58:03.545 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7fixFDV/jy3hTOoEU+vSmauELiA= 2023-11-02T17:58:03.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7fixFDV/jy3hTOoEU+vSmauELiA=', 'Content-Length': '0'} 2023-11-02T17:58:03.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbd89e51a5a58e2b-006543e32b-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.549 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:03.550 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:03.550 DEBUG:boto:path=/ibtljn-12/?versions&max-keys=999&prefix=obj-a1 2023-11-02T17:58:03.550 DEBUG:boto:auth_path=/ibtljn-12/?versions&max-keys=999&prefix=obj-a1 2023-11-02T17:58:03.551 DEBUG:boto:Method: GET 2023-11-02T17:58:03.551 DEBUG:boto:Path: /ibtljn-12/?versions&max-keys=999&prefix=obj-a1 2023-11-02T17:58:03.552 DEBUG:boto:Data: 2023-11-02T17:58:03.552 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.552 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.553 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.553 DEBUG:boto:Params: {} 2023-11-02T17:58:03.554 DEBUG:boto:Token: None 2023-11-02T17:58:03.554 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/?versions 2023-11-02T17:58:03.555 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JPM9NbKk54nj0n9KYstGUiBcXfY= 2023-11-02T17:58:03.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JPM9NbKk54nj0n9KYstGUiBcXfY=', 'Content-Length': '0'} 2023-11-02T17:58:03.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a7ba21febb4a0a8-006543e32b-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.560 DEBUG:boto:b'ibtljn-12obj-a1999falseobj-a1IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaEtrue2023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:03.561 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:03.561 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:03.561 DEBUG:boto:path=/ibtljn-12/obj-a1?acl 2023-11-02T17:58:03.562 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?acl 2023-11-02T17:58:03.562 DEBUG:boto:Method: PUT 2023-11-02T17:58:03.563 DEBUG:boto:Path: /ibtljn-12/obj-a1?acl 2023-11-02T17:58:03.563 DEBUG:boto:Data: 2023-11-02T17:58:03.564 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-11-02T17:58:03.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:03.564 DEBUG:boto:Port: 8000 2023-11-02T17:58:03.565 DEBUG:boto:Params: {} 2023-11-02T17:58:03.565 DEBUG:boto:Token: None 2023-11-02T17:58:03.566 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:58:03 GMT x-amz-acl:public-read /ibtljn-12/obj-a1?acl 2023-11-02T17:58:03.566 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTc0I7AalUt0OKSJE9QnQDQ1IdE= 2023-11-02T17:58:03.567 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTc0I7AalUt0OKSJE9QnQDQ1IdE=', 'Content-Length': '0'} 2023-11-02T17:58:03.579 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000f24a72722219774-006543e32b-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.579 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:03.580 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:03.580 DEBUG:boto:Method: HEAD 2023-11-02T17:58:03.581 DEBUG:boto:Path: /ibtljn-12/ 2023-11-02T17:58:03.581 DEBUG:boto:Data: 2023-11-02T17:58:03.582 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.582 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.582 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.583 DEBUG:boto:Params: {} 2023-11-02T17:58:03.583 DEBUG:boto:Token: None 2023-11-02T17:58:03.584 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/ 2023-11-02T17:58:03.584 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7fixFDV/jy3hTOoEU+vSmauELiA= 2023-11-02T17:58:03.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7fixFDV/jy3hTOoEU+vSmauELiA=', 'Content-Length': '0'} 2023-11-02T17:58:03.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e2d2602b3819b01-006543e32b-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.589 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:03.589 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:03.590 DEBUG:boto:path=/ibtljn-12/?versions&max-keys=999&prefix=obj-a2 2023-11-02T17:58:03.590 DEBUG:boto:auth_path=/ibtljn-12/?versions&max-keys=999&prefix=obj-a2 2023-11-02T17:58:03.590 DEBUG:boto:Method: GET 2023-11-02T17:58:03.591 DEBUG:boto:Path: /ibtljn-12/?versions&max-keys=999&prefix=obj-a2 2023-11-02T17:58:03.591 DEBUG:boto:Data: 2023-11-02T17:58:03.591 DEBUG:boto:Headers: {} 2023-11-02T17:58:03.592 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.592 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.593 DEBUG:boto:Params: {} 2023-11-02T17:58:03.593 DEBUG:boto:Token: None 2023-11-02T17:58:03.593 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:03 GMT /ibtljn-12/?versions 2023-11-02T17:58:03.594 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JPM9NbKk54nj0n9KYstGUiBcXfY= 2023-11-02T17:58:03.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JPM9NbKk54nj0n9KYstGUiBcXfY=', 'Content-Length': '0'} 2023-11-02T17:58:03.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3526ea08473e79d-006543e32b-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.642 DEBUG:boto:b'ibtljn-12obj-a2999falseobj-a2iLD9aIguC0onhAlOj.xw657ssc7CVJGtrue2023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:03.643 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.644 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:03.644 DEBUG:boto:path=/ibtljn-12/obj-a2?acl 2023-11-02T17:58:03.645 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?acl 2023-11-02T17:58:03.645 DEBUG:boto:Method: PUT 2023-11-02T17:58:03.646 DEBUG:boto:Path: /ibtljn-12/obj-a2?acl 2023-11-02T17:58:03.646 DEBUG:boto:Data: 2023-11-02T17:58:03.646 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-11-02T17:58:03.647 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:03.647 DEBUG:boto:Port: 8001 2023-11-02T17:58:03.648 DEBUG:boto:Params: {} 2023-11-02T17:58:03.648 DEBUG:boto:Token: None 2023-11-02T17:58:03.649 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:58:03 GMT x-amz-acl:public-read /ibtljn-12/obj-a2?acl 2023-11-02T17:58:03.649 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:19rN56NsoDSns5W76uL4QGxmqns= 2023-11-02T17:58:03.649 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:19rN56NsoDSns5W76uL4QGxmqns=', 'Content-Length': '0'} 2023-11-02T17:58:03.666 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004e4d8de15206be99-006543e32b-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:03.667 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-11 2023-11-02T17:58:03.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:03.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:03.669 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:58:03.719 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:58:03.720 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:58:03.791 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:03.782+0000 7ffbc715ff40 1 waiting for incremental sync to catch up: 2023-11-02T17:58:03.792 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000010.24.6, ] 2023-11-02T17:58:03.792 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000011.27.6, ] 2023-11-02T17:58:08.794 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:08.786+0000 7ffbc715ff40 1 waiting for incremental sync to catch up: 2023-11-02T17:58:08.795 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000010.24.6, ] 2023-11-02T17:58:08.795 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000011.27.6, ] 2023-11-02T17:58:13.797 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:13.790+0000 7ffbc715ff40 1 waiting for incremental sync to catch up: 2023-11-02T17:58:13.797 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000010.24.6, ] 2023-11-02T17:58:13.798 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000011.27.6, ] 2023-11-02T17:58:18.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:18.790+0000 7ffbc715ff40 1 waiting for incremental sync to catch up: 2023-11-02T17:58:18.800 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000010.24.6, ] 2023-11-02T17:58:18.800 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000011.27.6, ] 2023-11-02T17:58:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:23.794+0000 7ffbc715ff40 1 bucket sync caught up with source: 2023-11-02T17:58:23.803 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000011.27.6, ] 2023-11-02T17:58:23.803 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.9.11, 00000000011.27.6, ] 2023-11-02T17:58:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:23.794+0000 7ffbc715ff40 0 bucket checkpoint complete 2023-11-02T17:58:23.831 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-11 2023-11-02T17:58:23.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:23.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:23.833 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:58:23.886 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:58:23.887 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:58:23.955 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:23.946+0000 7fc12e7a8f40 1 bucket sync caught up with source: 2023-11-02T17:58:23.955 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.20.11, , , 00000000011.34.6, 00000000002.18.11, ] 2023-11-02T17:58:23.956 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.20.11, , , 00000000011.34.6, 00000000002.18.11, ] 2023-11-02T17:58:23.956 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:23.946+0000 7fc12e7a8f40 0 bucket checkpoint complete 2023-11-02T17:58:23.982 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-11 zones={a2, a1} 2023-11-02T17:58:23.983 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:23.983 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:23.984 DEBUG:boto:Method: HEAD 2023-11-02T17:58:23.984 DEBUG:boto:Path: /ibtljn-11/ 2023-11-02T17:58:23.985 DEBUG:boto:Data: 2023-11-02T17:58:23.985 DEBUG:boto:Headers: {} 2023-11-02T17:58:23.985 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:23.986 DEBUG:boto:Port: 8001 2023-11-02T17:58:23.986 DEBUG:boto:Params: {} 2023-11-02T17:58:23.987 DEBUG:boto:Token: None 2023-11-02T17:58:23.987 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:23 GMT /ibtljn-11/ 2023-11-02T17:58:23.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tXssZmd2WRIZwxyiLP7WEfdQsLA= 2023-11-02T17:58:23.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tXssZmd2WRIZwxyiLP7WEfdQsLA=', 'Content-Length': '0'} 2023-11-02T17:58:23.992 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000027a1a69b2cfe32f-006543e33f-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:23.993 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:23.993 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:23.994 DEBUG:boto:Method: HEAD 2023-11-02T17:58:23.994 DEBUG:boto:Path: /ibtljn-11/ 2023-11-02T17:58:23.994 DEBUG:boto:Data: 2023-11-02T17:58:23.995 DEBUG:boto:Headers: {} 2023-11-02T17:58:23.995 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:23.996 DEBUG:boto:Port: 8000 2023-11-02T17:58:23.996 DEBUG:boto:Params: {} 2023-11-02T17:58:23.996 DEBUG:boto:Token: None 2023-11-02T17:58:23.997 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:23 GMT /ibtljn-11/ 2023-11-02T17:58:23.997 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tXssZmd2WRIZwxyiLP7WEfdQsLA= 2023-11-02T17:58:23.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tXssZmd2WRIZwxyiLP7WEfdQsLA=', 'Content-Length': '0'} 2023-11-02T17:58:24.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001abe5dfd252040dc-006543e33f-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.003 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T17:58:24.003 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.004 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.004 DEBUG:boto:path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:24.004 DEBUG:boto:auth_path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:24.005 DEBUG:boto:Method: GET 2023-11-02T17:58:24.005 DEBUG:boto:Path: /ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:24.006 DEBUG:boto:Data: 2023-11-02T17:58:24.006 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.007 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.007 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.007 DEBUG:boto:Params: {} 2023-11-02T17:58:24.008 DEBUG:boto:Token: None 2023-11-02T17:58:24.008 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/?versions 2023-11-02T17:58:24.009 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KR7oyp26Pc1CIiHFEO+X9CUINf4= 2023-11-02T17:58:24.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KR7oyp26Pc1CIiHFEO+X9CUINf4=', 'Content-Length': '0'} 2023-11-02T17:58:24.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000214aeac175c9443b-006543e340-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.015 DEBUG:boto:b'ibtljn-11999falsedummygRMVA8xSztZb81HITCjm-8FBwHTFtyMtrue2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1TsWgaTvYq-GNh2xE46pAYWldI9CUwmYtrue2023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3Mtrue2023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.015 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T17:58:24.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-11-02T17:58:24.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-11-02T17:58:24.017 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T17:58:24.017 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.018 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.018 DEBUG:boto:path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:24.019 DEBUG:boto:auth_path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:24.019 DEBUG:boto:Method: GET 2023-11-02T17:58:24.019 DEBUG:boto:Path: /ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:24.020 DEBUG:boto:Data: 2023-11-02T17:58:24.020 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.021 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.021 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.021 DEBUG:boto:Params: {} 2023-11-02T17:58:24.022 DEBUG:boto:Token: None 2023-11-02T17:58:24.022 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/?versions 2023-11-02T17:58:24.023 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KR7oyp26Pc1CIiHFEO+X9CUINf4= 2023-11-02T17:58:24.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KR7oyp26Pc1CIiHFEO+X9CUINf4=', 'Content-Length': '0'} 2023-11-02T17:58:24.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039df40d8f8b88862-006543e340-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.030 DEBUG:boto:b'ibtljn-11999falsedummygRMVA8xSztZb81HITCjm-8FBwHTFtyMtrue2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1TsWgaTvYq-GNh2xE46pAYWldI9CUwmYtrue2023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3Mtrue2023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.031 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T17:58:24.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-11-02T17:58:24.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-11-02T17:58:24.032 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.032 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.033 DEBUG:boto:path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:24.033 DEBUG:boto:auth_path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:24.034 DEBUG:boto:Method: GET 2023-11-02T17:58:24.034 DEBUG:boto:Path: /ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:24.034 DEBUG:boto:Data: 2023-11-02T17:58:24.035 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.035 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.036 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.036 DEBUG:boto:Params: {} 2023-11-02T17:58:24.037 DEBUG:boto:Token: None 2023-11-02T17:58:24.037 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/?versions 2023-11-02T17:58:24.038 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KR7oyp26Pc1CIiHFEO+X9CUINf4= 2023-11-02T17:58:24.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KR7oyp26Pc1CIiHFEO+X9CUINf4=', 'Content-Length': '0'} 2023-11-02T17:58:24.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015482fec97643757-006543e340-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.044 DEBUG:boto:b'ibtljn-11999falsedummygRMVA8xSztZb81HITCjm-8FBwHTFtyMtrue2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1TsWgaTvYq-GNh2xE46pAYWldI9CUwmYtrue2023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3Mtrue2023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.045 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.046 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.046 DEBUG:boto:path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:24.047 DEBUG:boto:auth_path=/ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:24.047 DEBUG:boto:Method: GET 2023-11-02T17:58:24.047 DEBUG:boto:Path: /ibtljn-11/?versions&max-keys=999 2023-11-02T17:58:24.048 DEBUG:boto:Data: 2023-11-02T17:58:24.048 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.049 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.049 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.050 DEBUG:boto:Params: {} 2023-11-02T17:58:24.050 DEBUG:boto:Token: None 2023-11-02T17:58:24.051 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/?versions 2023-11-02T17:58:24.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KR7oyp26Pc1CIiHFEO+X9CUINf4= 2023-11-02T17:58:24.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KR7oyp26Pc1CIiHFEO+X9CUINf4=', 'Content-Length': '0'} 2023-11-02T17:58:24.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073e1932b6d858c2e-006543e340-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.057 DEBUG:boto:b'ibtljn-11999falsedummygRMVA8xSztZb81HITCjm-8FBwHTFtyMtrue2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1TsWgaTvYq-GNh2xE46pAYWldI9CUwmYtrue2023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3Mtrue2023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:58:24.059 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:24.059 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:24.059 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.060 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.060 DEBUG:boto:Method: GET 2023-11-02T17:58:24.061 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.061 DEBUG:boto:Data: 2023-11-02T17:58:24.062 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.062 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.063 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.063 DEBUG:boto:Params: {} 2023-11-02T17:58:24.064 DEBUG:boto:Token: None 2023-11-02T17:58:24.064 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.065 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+1pjnBXqTjJYeRIvpQ6+u001Gb8= 2023-11-02T17:58:24.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+1pjnBXqTjJYeRIvpQ6+u001Gb8=', 'Content-Length': '0'} 2023-11-02T17:58:24.068 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081c2fd0ec9dde255-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.069 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:24.070 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:24.070 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.071 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.071 DEBUG:boto:Method: GET 2023-11-02T17:58:24.072 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.072 DEBUG:boto:Data: 2023-11-02T17:58:24.073 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.073 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.074 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.074 DEBUG:boto:Params: {} 2023-11-02T17:58:24.075 DEBUG:boto:Token: None 2023-11-02T17:58:24.075 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.076 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+1pjnBXqTjJYeRIvpQ6+u001Gb8= 2023-11-02T17:58:24.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+1pjnBXqTjJYeRIvpQ6+u001Gb8=', 'Content-Length': '0'} 2023-11-02T17:58:24.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012e744104eae734f-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.080 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:24.080 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:24.081 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.081 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.081 DEBUG:boto:Method: HEAD 2023-11-02T17:58:24.082 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.082 DEBUG:boto:Data: 2023-11-02T17:58:24.083 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.083 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.084 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.084 DEBUG:boto:Params: {} 2023-11-02T17:58:24.085 DEBUG:boto:Token: None 2023-11-02T17:58:24.085 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.086 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M6aqVhwr1zK0RLhhXxe7hH0SE/g= 2023-11-02T17:58:24.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M6aqVhwr1zK0RLhhXxe7hH0SE/g=', 'Content-Length': '0'} 2023-11-02T17:58:24.089 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000682ba18b7048c7eb-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.091 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:24.091 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:24.092 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.092 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.093 DEBUG:boto:Method: HEAD 2023-11-02T17:58:24.093 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.093 DEBUG:boto:Data: 2023-11-02T17:58:24.094 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.094 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.095 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.095 DEBUG:boto:Params: {} 2023-11-02T17:58:24.096 DEBUG:boto:Token: None 2023-11-02T17:58:24.097 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M6aqVhwr1zK0RLhhXxe7hH0SE/g= 2023-11-02T17:58:24.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M6aqVhwr1zK0RLhhXxe7hH0SE/g=', 'Content-Length': '0'} 2023-11-02T17:58:24.101 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006766dbe106416458-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:58:24.103 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:24.103 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:24.104 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.104 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.105 DEBUG:boto:Method: GET 2023-11-02T17:58:24.105 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.105 DEBUG:boto:Data: 2023-11-02T17:58:24.106 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.107 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.107 DEBUG:boto:Params: {} 2023-11-02T17:58:24.108 DEBUG:boto:Token: None 2023-11-02T17:58:24.108 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.109 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+1pjnBXqTjJYeRIvpQ6+u001Gb8= 2023-11-02T17:58:24.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+1pjnBXqTjJYeRIvpQ6+u001Gb8=', 'Content-Length': '0'} 2023-11-02T17:58:24.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d04a0bab19634691-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.113 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:24.114 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:24.114 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.115 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.115 DEBUG:boto:Method: GET 2023-11-02T17:58:24.115 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.116 DEBUG:boto:Data: 2023-11-02T17:58:24.116 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.117 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.117 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.118 DEBUG:boto:Params: {} 2023-11-02T17:58:24.118 DEBUG:boto:Token: None 2023-11-02T17:58:24.119 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.119 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+1pjnBXqTjJYeRIvpQ6+u001Gb8= 2023-11-02T17:58:24.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+1pjnBXqTjJYeRIvpQ6+u001Gb8=', 'Content-Length': '0'} 2023-11-02T17:58:24.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000020e03645bc95e59b-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.123 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.124 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.124 DEBUG:boto:path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:24.125 DEBUG:boto:auth_path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:24.125 DEBUG:boto:Method: GET 2023-11-02T17:58:24.125 DEBUG:boto:Path: /ibtljn-11/?prefix=dummy 2023-11-02T17:58:24.126 DEBUG:boto:Data: 2023-11-02T17:58:24.126 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.127 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.127 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.128 DEBUG:boto:Params: {} 2023-11-02T17:58:24.128 DEBUG:boto:Token: None 2023-11-02T17:58:24.129 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/ 2023-11-02T17:58:24.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U= 2023-11-02T17:58:24.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U=', 'Content-Length': '0'} 2023-11-02T17:58:24.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026e0288271872865-006543e340-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.174 DEBUG:boto:b'ibtljn-11dummy1000falsedummy2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.175 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.176 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.176 DEBUG:boto:path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:24.177 DEBUG:boto:auth_path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:24.177 DEBUG:boto:Method: GET 2023-11-02T17:58:24.178 DEBUG:boto:Path: /ibtljn-11/?prefix=dummy 2023-11-02T17:58:24.178 DEBUG:boto:Data: 2023-11-02T17:58:24.178 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.179 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.180 DEBUG:boto:Params: {} 2023-11-02T17:58:24.180 DEBUG:boto:Token: None 2023-11-02T17:58:24.181 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/ 2023-11-02T17:58:24.181 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U= 2023-11-02T17:58:24.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U=', 'Content-Length': '0'} 2023-11-02T17:58:24.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba1f663e5e6855ab-006543e340-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.187 DEBUG:boto:b'ibtljn-11dummy1000falsedummy2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.188 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:24.189 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:24.190 DEBUG:boto:Method: HEAD 2023-11-02T17:58:24.190 DEBUG:boto:Path: /ibtljn-11/dummy 2023-11-02T17:58:24.191 DEBUG:boto:Data: 2023-11-02T17:58:24.192 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.192 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.193 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.193 DEBUG:boto:Params: {} 2023-11-02T17:58:24.194 DEBUG:boto:Token: None 2023-11-02T17:58:24.194 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/dummy 2023-11-02T17:58:24.195 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqpweF6U0GdQXzqLHzOJzFpUpNU= 2023-11-02T17:58:24.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqpweF6U0GdQXzqLHzOJzFpUpNU=', 'Content-Length': '0'} 2023-11-02T17:58:24.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cffd88f5dc62def1-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.200 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:24.200 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:24.201 DEBUG:boto:Method: HEAD 2023-11-02T17:58:24.201 DEBUG:boto:Path: /ibtljn-11/dummy 2023-11-02T17:58:24.202 DEBUG:boto:Data: 2023-11-02T17:58:24.202 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.203 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.203 DEBUG:boto:Params: {} 2023-11-02T17:58:24.204 DEBUG:boto:Token: None 2023-11-02T17:58:24.204 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/dummy 2023-11-02T17:58:24.205 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqpweF6U0GdQXzqLHzOJzFpUpNU= 2023-11-02T17:58:24.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqpweF6U0GdQXzqLHzOJzFpUpNU=', 'Content-Length': '0'} 2023-11-02T17:58:24.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ec145786a6c782e-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:58:24.210 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:24.211 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:24.211 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.211 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.212 DEBUG:boto:Method: GET 2023-11-02T17:58:24.212 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.213 DEBUG:boto:Data: 2023-11-02T17:58:24.213 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.214 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.214 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.215 DEBUG:boto:Params: {} 2023-11-02T17:58:24.215 DEBUG:boto:Token: None 2023-11-02T17:58:24.216 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.216 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+1pjnBXqTjJYeRIvpQ6+u001Gb8= 2023-11-02T17:58:24.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+1pjnBXqTjJYeRIvpQ6+u001Gb8=', 'Content-Length': '0'} 2023-11-02T17:58:24.220 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000030de7619f6be26f0-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.220 DEBUG:boto:path=/ibtljn-11/dummy 2023-11-02T17:58:24.221 DEBUG:boto:auth_path=/ibtljn-11/dummy 2023-11-02T17:58:24.221 DEBUG:boto:path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.222 DEBUG:boto:auth_path=/ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.222 DEBUG:boto:Method: GET 2023-11-02T17:58:24.222 DEBUG:boto:Path: /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.223 DEBUG:boto:Data: 2023-11-02T17:58:24.223 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.224 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.224 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.225 DEBUG:boto:Params: {} 2023-11-02T17:58:24.225 DEBUG:boto:Token: None 2023-11-02T17:58:24.226 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/dummy?versionId=gRMVA8xSztZb81HITCjm-8FBwHTFtyM 2023-11-02T17:58:24.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+1pjnBXqTjJYeRIvpQ6+u001Gb8= 2023-11-02T17:58:24.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+1pjnBXqTjJYeRIvpQ6+u001Gb8=', 'Content-Length': '0'} 2023-11-02T17:58:24.229 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:29 GMT'), ('x-amz-version-id', 'gRMVA8xSztZb81HITCjm-8FBwHTFtyM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c3d3b5937120507e-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.230 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.231 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.231 DEBUG:boto:path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:24.231 DEBUG:boto:auth_path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:24.232 DEBUG:boto:Method: GET 2023-11-02T17:58:24.232 DEBUG:boto:Path: /ibtljn-11/?prefix=dummy 2023-11-02T17:58:24.233 DEBUG:boto:Data: 2023-11-02T17:58:24.233 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.234 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.234 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.234 DEBUG:boto:Params: {} 2023-11-02T17:58:24.235 DEBUG:boto:Token: None 2023-11-02T17:58:24.235 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/ 2023-11-02T17:58:24.236 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U= 2023-11-02T17:58:24.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U=', 'Content-Length': '0'} 2023-11-02T17:58:24.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ce5cfc293c04daa-006543e340-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.282 DEBUG:boto:b'ibtljn-11dummy1000falsedummy2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.283 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.284 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.284 DEBUG:boto:path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:24.285 DEBUG:boto:auth_path=/ibtljn-11/?prefix=dummy 2023-11-02T17:58:24.285 DEBUG:boto:Method: GET 2023-11-02T17:58:24.286 DEBUG:boto:Path: /ibtljn-11/?prefix=dummy 2023-11-02T17:58:24.286 DEBUG:boto:Data: 2023-11-02T17:58:24.287 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.287 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.287 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.288 DEBUG:boto:Params: {} 2023-11-02T17:58:24.288 DEBUG:boto:Token: None 2023-11-02T17:58:24.289 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/ 2023-11-02T17:58:24.289 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U= 2023-11-02T17:58:24.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U=', 'Content-Length': '0'} 2023-11-02T17:58:24.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d99ff91644b7f3b8-006543e340-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.295 DEBUG:boto:b'ibtljn-11dummy1000falsedummy2023-11-02T17:56:29.988Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-11-02T17:58:24.296 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.297 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.297 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.298 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.298 DEBUG:boto:Method: GET 2023-11-02T17:58:24.299 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.299 DEBUG:boto:Data: 2023-11-02T17:58:24.300 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.300 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.300 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.301 DEBUG:boto:Params: {} 2023-11-02T17:58:24.301 DEBUG:boto:Token: None 2023-11-02T17:58:24.302 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.302 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WJIiHm5G0l4CWmcdWlAKYEglenE= 2023-11-02T17:58:24.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WJIiHm5G0l4CWmcdWlAKYEglenE=', 'Content-Length': '0'} 2023-11-02T17:58:24.306 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8b655ffb2baad40-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.306 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.307 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.307 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.308 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.308 DEBUG:boto:Method: GET 2023-11-02T17:58:24.309 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.309 DEBUG:boto:Data: 2023-11-02T17:58:24.310 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.310 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.311 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.311 DEBUG:boto:Params: {} 2023-11-02T17:58:24.311 DEBUG:boto:Token: None 2023-11-02T17:58:24.312 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.312 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WJIiHm5G0l4CWmcdWlAKYEglenE= 2023-11-02T17:58:24.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WJIiHm5G0l4CWmcdWlAKYEglenE=', 'Content-Length': '0'} 2023-11-02T17:58:24.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000857c8aa250f34d4a-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.358 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.359 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.359 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.360 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.360 DEBUG:boto:Method: HEAD 2023-11-02T17:58:24.361 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.361 DEBUG:boto:Data: 2023-11-02T17:58:24.362 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.362 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.362 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.363 DEBUG:boto:Params: {} 2023-11-02T17:58:24.363 DEBUG:boto:Token: None 2023-11-02T17:58:24.364 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.364 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fAYipA24QUATtHKYsrjqc3fo9+o= 2023-11-02T17:58:24.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fAYipA24QUATtHKYsrjqc3fo9+o=', 'Content-Length': '0'} 2023-11-02T17:58:24.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dffb4fe258a047e-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.369 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.369 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.369 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.370 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.370 DEBUG:boto:Method: HEAD 2023-11-02T17:58:24.371 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.371 DEBUG:boto:Data: 2023-11-02T17:58:24.372 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.372 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.372 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.373 DEBUG:boto:Params: {} 2023-11-02T17:58:24.373 DEBUG:boto:Token: None 2023-11-02T17:58:24.374 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.374 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fAYipA24QUATtHKYsrjqc3fo9+o= 2023-11-02T17:58:24.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fAYipA24QUATtHKYsrjqc3fo9+o=', 'Content-Length': '0'} 2023-11-02T17:58:24.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b7896a9c5f4f961-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-11-02T17:58:24.379 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.379 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.379 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.380 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.380 DEBUG:boto:Method: GET 2023-11-02T17:58:24.381 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.381 DEBUG:boto:Data: 2023-11-02T17:58:24.382 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.382 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.383 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.383 DEBUG:boto:Params: {} 2023-11-02T17:58:24.383 DEBUG:boto:Token: None 2023-11-02T17:58:24.384 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.384 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WJIiHm5G0l4CWmcdWlAKYEglenE= 2023-11-02T17:58:24.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WJIiHm5G0l4CWmcdWlAKYEglenE=', 'Content-Length': '0'} 2023-11-02T17:58:24.388 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d452d640c3989c11-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.389 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.389 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.389 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.390 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.390 DEBUG:boto:Method: GET 2023-11-02T17:58:24.391 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.391 DEBUG:boto:Data: 2023-11-02T17:58:24.392 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.392 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.392 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.393 DEBUG:boto:Params: {} 2023-11-02T17:58:24.393 DEBUG:boto:Token: None 2023-11-02T17:58:24.394 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WJIiHm5G0l4CWmcdWlAKYEglenE= 2023-11-02T17:58:24.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WJIiHm5G0l4CWmcdWlAKYEglenE=', 'Content-Length': '0'} 2023-11-02T17:58:24.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f5bccd08a58893b-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.399 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.399 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.400 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:24.400 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:24.400 DEBUG:boto:Method: GET 2023-11-02T17:58:24.401 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:24.401 DEBUG:boto:Data: 2023-11-02T17:58:24.402 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.402 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.403 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.403 DEBUG:boto:Params: {} 2023-11-02T17:58:24.404 DEBUG:boto:Token: None 2023-11-02T17:58:24.404 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/ 2023-11-02T17:58:24.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U= 2023-11-02T17:58:24.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U=', 'Content-Length': '0'} 2023-11-02T17:58:24.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d31a0126541e71a7-006543e340-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.450 DEBUG:boto:b'ibtljn-11obj-a11000falseobj-a12023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.451 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.452 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.452 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:24.453 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:24.453 DEBUG:boto:Method: GET 2023-11-02T17:58:24.454 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:24.454 DEBUG:boto:Data: 2023-11-02T17:58:24.455 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.455 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.456 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.456 DEBUG:boto:Params: {} 2023-11-02T17:58:24.457 DEBUG:boto:Token: None 2023-11-02T17:58:24.457 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/ 2023-11-02T17:58:24.457 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U= 2023-11-02T17:58:24.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U=', 'Content-Length': '0'} 2023-11-02T17:58:24.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009acbe76c165fd530-006543e340-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.463 DEBUG:boto:b'ibtljn-11obj-a11000falseobj-a12023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.464 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.464 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.465 DEBUG:boto:Method: HEAD 2023-11-02T17:58:24.465 DEBUG:boto:Path: /ibtljn-11/obj-a1 2023-11-02T17:58:24.466 DEBUG:boto:Data: 2023-11-02T17:58:24.466 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.467 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.467 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.467 DEBUG:boto:Params: {} 2023-11-02T17:58:24.468 DEBUG:boto:Token: None 2023-11-02T17:58:24.468 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a1 2023-11-02T17:58:24.469 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tzoh7HbTPtE+SonBvSFmqli9Qw4= 2023-11-02T17:58:24.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tzoh7HbTPtE+SonBvSFmqli9Qw4=', 'Content-Length': '0'} 2023-11-02T17:58:24.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002db1348d631e4b91-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.474 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.474 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.475 DEBUG:boto:Method: HEAD 2023-11-02T17:58:24.475 DEBUG:boto:Path: /ibtljn-11/obj-a1 2023-11-02T17:58:24.476 DEBUG:boto:Data: 2023-11-02T17:58:24.477 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.478 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.478 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.478 DEBUG:boto:Params: {} 2023-11-02T17:58:24.479 DEBUG:boto:Token: None 2023-11-02T17:58:24.479 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a1 2023-11-02T17:58:24.480 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tzoh7HbTPtE+SonBvSFmqli9Qw4= 2023-11-02T17:58:24.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tzoh7HbTPtE+SonBvSFmqli9Qw4=', 'Content-Length': '0'} 2023-11-02T17:58:24.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bbd3e343b88e4b8-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-11-02T17:58:24.485 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.485 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.486 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.486 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.487 DEBUG:boto:Method: GET 2023-11-02T17:58:24.487 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.488 DEBUG:boto:Data: 2023-11-02T17:58:24.488 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.488 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.489 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.489 DEBUG:boto:Params: {} 2023-11-02T17:58:24.490 DEBUG:boto:Token: None 2023-11-02T17:58:24.490 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.491 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WJIiHm5G0l4CWmcdWlAKYEglenE= 2023-11-02T17:58:24.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WJIiHm5G0l4CWmcdWlAKYEglenE=', 'Content-Length': '0'} 2023-11-02T17:58:24.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005eb3a5c44dc2e009-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.495 DEBUG:boto:path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.495 DEBUG:boto:auth_path=/ibtljn-11/obj-a1 2023-11-02T17:58:24.496 DEBUG:boto:path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.496 DEBUG:boto:auth_path=/ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.497 DEBUG:boto:Method: GET 2023-11-02T17:58:24.497 DEBUG:boto:Path: /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.498 DEBUG:boto:Data: 2023-11-02T17:58:24.498 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.499 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.499 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.499 DEBUG:boto:Params: {} 2023-11-02T17:58:24.500 DEBUG:boto:Token: None 2023-11-02T17:58:24.500 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a1?versionId=TsWgaTvYq-GNh2xE46pAYWldI9CUwmY 2023-11-02T17:58:24.501 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WJIiHm5G0l4CWmcdWlAKYEglenE= 2023-11-02T17:58:24.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WJIiHm5G0l4CWmcdWlAKYEglenE=', 'Content-Length': '0'} 2023-11-02T17:58:24.504 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'TsWgaTvYq-GNh2xE46pAYWldI9CUwmY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d93be1f07002821-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.546 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.547 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.547 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:24.548 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:24.548 DEBUG:boto:Method: GET 2023-11-02T17:58:24.549 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:24.549 DEBUG:boto:Data: 2023-11-02T17:58:24.550 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.550 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.551 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.551 DEBUG:boto:Params: {} 2023-11-02T17:58:24.551 DEBUG:boto:Token: None 2023-11-02T17:58:24.552 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/ 2023-11-02T17:58:24.552 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U= 2023-11-02T17:58:24.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U=', 'Content-Length': '0'} 2023-11-02T17:58:24.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000578c7c5d32412bed-006543e340-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.558 DEBUG:boto:b'ibtljn-11obj-a11000falseobj-a12023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.559 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.560 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.560 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:24.561 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:24.561 DEBUG:boto:Method: GET 2023-11-02T17:58:24.561 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a1 2023-11-02T17:58:24.562 DEBUG:boto:Data: 2023-11-02T17:58:24.562 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.563 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.563 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.564 DEBUG:boto:Params: {} 2023-11-02T17:58:24.564 DEBUG:boto:Token: None 2023-11-02T17:58:24.564 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/ 2023-11-02T17:58:24.565 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U= 2023-11-02T17:58:24.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U=', 'Content-Length': '0'} 2023-11-02T17:58:24.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036089a9b059abc86-006543e340-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.571 DEBUG:boto:b'ibtljn-11obj-a11000falseobj-a12023-11-02T17:57:05.395Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-11-02T17:58:24.572 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.573 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.573 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.574 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.574 DEBUG:boto:Method: GET 2023-11-02T17:58:24.575 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.575 DEBUG:boto:Data: 2023-11-02T17:58:24.575 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.576 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.576 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.577 DEBUG:boto:Params: {} 2023-11-02T17:58:24.577 DEBUG:boto:Token: None 2023-11-02T17:58:24.578 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.578 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C+3m/eIT4oJDIVrj5sh8oWC13wE= 2023-11-02T17:58:24.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C+3m/eIT4oJDIVrj5sh8oWC13wE=', 'Content-Length': '0'} 2023-11-02T17:58:24.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5483529da17a9df-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.626 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.627 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.627 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.628 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.628 DEBUG:boto:Method: GET 2023-11-02T17:58:24.629 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.629 DEBUG:boto:Data: 2023-11-02T17:58:24.630 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.630 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.630 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.631 DEBUG:boto:Params: {} 2023-11-02T17:58:24.631 DEBUG:boto:Token: None 2023-11-02T17:58:24.632 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.632 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C+3m/eIT4oJDIVrj5sh8oWC13wE= 2023-11-02T17:58:24.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C+3m/eIT4oJDIVrj5sh8oWC13wE=', 'Content-Length': '0'} 2023-11-02T17:58:24.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e63e05b94adf10c7-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.637 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.637 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.638 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.638 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.639 DEBUG:boto:Method: HEAD 2023-11-02T17:58:24.639 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.640 DEBUG:boto:Data: 2023-11-02T17:58:24.640 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.641 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.641 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.642 DEBUG:boto:Params: {} 2023-11-02T17:58:24.642 DEBUG:boto:Token: None 2023-11-02T17:58:24.643 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.644 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wjXSlgkDFtQpMXQupXGAxQNpIz4= 2023-11-02T17:58:24.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wjXSlgkDFtQpMXQupXGAxQNpIz4=', 'Content-Length': '0'} 2023-11-02T17:58:24.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed25adf9d4eefda7-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.648 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.649 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.649 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.650 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.650 DEBUG:boto:Method: HEAD 2023-11-02T17:58:24.651 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.651 DEBUG:boto:Data: 2023-11-02T17:58:24.652 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.652 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.653 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.653 DEBUG:boto:Params: {} 2023-11-02T17:58:24.654 DEBUG:boto:Token: None 2023-11-02T17:58:24.654 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.655 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wjXSlgkDFtQpMXQupXGAxQNpIz4= 2023-11-02T17:58:24.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wjXSlgkDFtQpMXQupXGAxQNpIz4=', 'Content-Length': '0'} 2023-11-02T17:58:24.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035039adf2b4b9b22-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-11-02T17:58:24.659 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.660 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.660 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.661 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.661 DEBUG:boto:Method: GET 2023-11-02T17:58:24.662 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.662 DEBUG:boto:Data: 2023-11-02T17:58:24.663 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.664 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.664 DEBUG:boto:Params: {} 2023-11-02T17:58:24.665 DEBUG:boto:Token: None 2023-11-02T17:58:24.665 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.666 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C+3m/eIT4oJDIVrj5sh8oWC13wE= 2023-11-02T17:58:24.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C+3m/eIT4oJDIVrj5sh8oWC13wE=', 'Content-Length': '0'} 2023-11-02T17:58:24.669 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000254128459c71de19-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.670 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.670 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.671 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.671 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.672 DEBUG:boto:Method: GET 2023-11-02T17:58:24.672 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.673 DEBUG:boto:Data: 2023-11-02T17:58:24.673 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.674 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.674 DEBUG:boto:Params: {} 2023-11-02T17:58:24.675 DEBUG:boto:Token: None 2023-11-02T17:58:24.676 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C+3m/eIT4oJDIVrj5sh8oWC13wE= 2023-11-02T17:58:24.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C+3m/eIT4oJDIVrj5sh8oWC13wE=', 'Content-Length': '0'} 2023-11-02T17:58:24.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1e2a330e50802d1-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.681 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.681 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.682 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:24.683 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:24.683 DEBUG:boto:Method: GET 2023-11-02T17:58:24.684 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:24.684 DEBUG:boto:Data: 2023-11-02T17:58:24.685 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.685 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.686 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.687 DEBUG:boto:Params: {} 2023-11-02T17:58:24.687 DEBUG:boto:Token: None 2023-11-02T17:58:24.688 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/ 2023-11-02T17:58:24.688 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U= 2023-11-02T17:58:24.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U=', 'Content-Length': '0'} 2023-11-02T17:58:24.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000562880e7a1b06dd-006543e340-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.696 DEBUG:boto:b'ibtljn-11obj-a21000falseobj-a22023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.696 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.697 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.697 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:24.698 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:24.698 DEBUG:boto:Method: GET 2023-11-02T17:58:24.698 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:24.699 DEBUG:boto:Data: 2023-11-02T17:58:24.699 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.700 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.701 DEBUG:boto:Params: {} 2023-11-02T17:58:24.701 DEBUG:boto:Token: None 2023-11-02T17:58:24.702 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/ 2023-11-02T17:58:24.702 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U= 2023-11-02T17:58:24.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U=', 'Content-Length': '0'} 2023-11-02T17:58:24.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4a5ef540171e05b-006543e340-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.754 DEBUG:boto:b'ibtljn-11obj-a21000falseobj-a22023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.755 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.756 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.756 DEBUG:boto:Method: HEAD 2023-11-02T17:58:24.757 DEBUG:boto:Path: /ibtljn-11/obj-a2 2023-11-02T17:58:24.757 DEBUG:boto:Data: 2023-11-02T17:58:24.757 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.758 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.758 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.759 DEBUG:boto:Params: {} 2023-11-02T17:58:24.759 DEBUG:boto:Token: None 2023-11-02T17:58:24.760 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a2 2023-11-02T17:58:24.760 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cqAlwDyJQsVU+0YbOoot+H+xBqs= 2023-11-02T17:58:24.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cqAlwDyJQsVU+0YbOoot+H+xBqs=', 'Content-Length': '0'} 2023-11-02T17:58:24.765 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bf98e964a1b9ff1-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.765 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.766 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.766 DEBUG:boto:Method: HEAD 2023-11-02T17:58:24.767 DEBUG:boto:Path: /ibtljn-11/obj-a2 2023-11-02T17:58:24.767 DEBUG:boto:Data: 2023-11-02T17:58:24.768 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.768 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.769 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.769 DEBUG:boto:Params: {} 2023-11-02T17:58:24.769 DEBUG:boto:Token: None 2023-11-02T17:58:24.770 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a2 2023-11-02T17:58:24.770 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cqAlwDyJQsVU+0YbOoot+H+xBqs= 2023-11-02T17:58:24.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cqAlwDyJQsVU+0YbOoot+H+xBqs=', 'Content-Length': '0'} 2023-11-02T17:58:24.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000456293cb80212917-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-11-02T17:58:24.776 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.776 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.777 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.777 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.777 DEBUG:boto:Method: GET 2023-11-02T17:58:24.778 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.778 DEBUG:boto:Data: 2023-11-02T17:58:24.779 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.779 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.780 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.780 DEBUG:boto:Params: {} 2023-11-02T17:58:24.781 DEBUG:boto:Token: None 2023-11-02T17:58:24.781 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.781 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C+3m/eIT4oJDIVrj5sh8oWC13wE= 2023-11-02T17:58:24.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C+3m/eIT4oJDIVrj5sh8oWC13wE=', 'Content-Length': '0'} 2023-11-02T17:58:24.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e60c944a6f88bb24-006543e340-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.826 DEBUG:boto:path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.827 DEBUG:boto:auth_path=/ibtljn-11/obj-a2 2023-11-02T17:58:24.828 DEBUG:boto:path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.828 DEBUG:boto:auth_path=/ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.828 DEBUG:boto:Method: GET 2023-11-02T17:58:24.829 DEBUG:boto:Path: /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.829 DEBUG:boto:Data: 2023-11-02T17:58:24.830 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.830 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.831 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.831 DEBUG:boto:Params: {} 2023-11-02T17:58:24.832 DEBUG:boto:Token: None 2023-11-02T17:58:24.832 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/obj-a2?versionId=b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M 2023-11-02T17:58:24.833 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C+3m/eIT4oJDIVrj5sh8oWC13wE= 2023-11-02T17:58:24.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C+3m/eIT4oJDIVrj5sh8oWC13wE=', 'Content-Length': '0'} 2023-11-02T17:58:24.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'b2DXXw6eZzyZM8WM2VEcRAxkuJLhW3M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000445791cd10ed58f4-006543e340-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.837 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.838 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.838 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:24.839 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:24.839 DEBUG:boto:Method: GET 2023-11-02T17:58:24.839 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:24.840 DEBUG:boto:Data: 2023-11-02T17:58:24.840 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:24.841 DEBUG:boto:Port: 8001 2023-11-02T17:58:24.842 DEBUG:boto:Params: {} 2023-11-02T17:58:24.842 DEBUG:boto:Token: None 2023-11-02T17:58:24.843 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/ 2023-11-02T17:58:24.843 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U= 2023-11-02T17:58:24.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U=', 'Content-Length': '0'} 2023-11-02T17:58:24.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8e8d250d6efad65-006543e340-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.849 DEBUG:boto:b'ibtljn-11obj-a21000falseobj-a22023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.850 DEBUG:boto:path=/ibtljn-11/ 2023-11-02T17:58:24.851 DEBUG:boto:auth_path=/ibtljn-11/ 2023-11-02T17:58:24.851 DEBUG:boto:path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:24.852 DEBUG:boto:auth_path=/ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:24.852 DEBUG:boto:Method: GET 2023-11-02T17:58:24.852 DEBUG:boto:Path: /ibtljn-11/?prefix=obj-a2 2023-11-02T17:58:24.853 DEBUG:boto:Data: 2023-11-02T17:58:24.853 DEBUG:boto:Headers: {} 2023-11-02T17:58:24.854 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:24.854 DEBUG:boto:Port: 8000 2023-11-02T17:58:24.855 DEBUG:boto:Params: {} 2023-11-02T17:58:24.855 DEBUG:boto:Token: None 2023-11-02T17:58:24.856 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:24 GMT /ibtljn-11/ 2023-11-02T17:58:24.856 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U= 2023-11-02T17:58:24.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k0+TfQUkwnTPoca4WA0Bn0Fp96U=', 'Content-Length': '0'} 2023-11-02T17:58:24.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3bdc3200fb79366-006543e340-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:24.862 DEBUG:boto:b'ibtljn-11obj-a21000falseobj-a22023-11-02T17:57:05.562Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:24.863 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-11 zones={a2, a1} 2023-11-02T17:58:24.863 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-12 2023-11-02T17:58:24.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:24.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:24.865 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:58:24.918 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:58:24.919 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:58:24.991 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:24.982+0000 7fee3e71cf40 1 bucket sync caught up with source: 2023-11-02T17:58:24.991 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.18.11, 00000000011.27.6, ] 2023-11-02T17:58:24.992 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.18.11, 00000000011.27.6, ] 2023-11-02T17:58:24.992 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:24.982+0000 7fee3e71cf40 0 bucket checkpoint complete 2023-11-02T17:58:25.016 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-12 2023-11-02T17:58:25.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:25.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:25.017 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:58:25.069 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:58:25.069 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:58:25.138 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:25.130+0000 7fccc3d89f40 1 bucket sync caught up with source: 2023-11-02T17:58:25.138 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.45.10, , , 00000000011.49.6, 00000000003.18.10, ] 2023-11-02T17:58:25.139 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.45.10, , , 00000000011.49.6, 00000000003.18.10, ] 2023-11-02T17:58:25.139 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:25.130+0000 7fccc3d89f40 0 bucket checkpoint complete 2023-11-02T17:58:25.164 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-12 zones={a2, a1} 2023-11-02T17:58:25.164 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.165 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.165 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.166 DEBUG:boto:Path: /ibtljn-12/ 2023-11-02T17:58:25.167 DEBUG:boto:Data: 2023-11-02T17:58:25.167 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.168 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.168 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.169 DEBUG:boto:Params: {} 2023-11-02T17:58:25.169 DEBUG:boto:Token: None 2023-11-02T17:58:25.170 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.170 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9sZrIKsvVNbZu30PyrzCNGGzSIc= 2023-11-02T17:58:25.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9sZrIKsvVNbZu30PyrzCNGGzSIc=', 'Content-Length': '0'} 2023-11-02T17:58:25.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001fd1db9d960bb7f6-006543e341-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.175 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.175 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.176 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.176 DEBUG:boto:Path: /ibtljn-12/ 2023-11-02T17:58:25.177 DEBUG:boto:Data: 2023-11-02T17:58:25.177 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.178 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.178 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.179 DEBUG:boto:Params: {} 2023-11-02T17:58:25.179 DEBUG:boto:Token: None 2023-11-02T17:58:25.180 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.180 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9sZrIKsvVNbZu30PyrzCNGGzSIc= 2023-11-02T17:58:25.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9sZrIKsvVNbZu30PyrzCNGGzSIc=', 'Content-Length': '0'} 2023-11-02T17:58:25.184 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ec6ac3a8f9161de-006543e341-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.185 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T17:58:25.185 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.186 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.186 DEBUG:boto:path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:25.187 DEBUG:boto:auth_path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:25.187 DEBUG:boto:Method: GET 2023-11-02T17:58:25.187 DEBUG:boto:Path: /ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:25.188 DEBUG:boto:Data: 2023-11-02T17:58:25.188 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.189 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.189 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.190 DEBUG:boto:Params: {} 2023-11-02T17:58:25.190 DEBUG:boto:Token: None 2023-11-02T17:58:25.191 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/?versions 2023-11-02T17:58:25.191 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+DgBgmHb/2TqHM92bXLdcbfHKLw= 2023-11-02T17:58:25.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+DgBgmHb/2TqHM92bXLdcbfHKLw=', 'Content-Length': '0'} 2023-11-02T17:58:25.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f59242a6d466874d-006543e341-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.197 DEBUG:boto:b'ibtljn-12999falsedummyNCgTgcQgi48JNmlzIJZc2XV1YwJaH71true2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaEtrue2023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2iLD9aIguC0onhAlOj.xw657ssc7CVJGtrue2023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.198 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T17:58:25.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-11-02T17:58:25.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-11-02T17:58:25.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T17:58:25.200 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.200 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.200 DEBUG:boto:path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:25.201 DEBUG:boto:auth_path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:25.201 DEBUG:boto:Method: GET 2023-11-02T17:58:25.202 DEBUG:boto:Path: /ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:25.202 DEBUG:boto:Data: 2023-11-02T17:58:25.203 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.203 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.203 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.204 DEBUG:boto:Params: {} 2023-11-02T17:58:25.204 DEBUG:boto:Token: None 2023-11-02T17:58:25.205 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/?versions 2023-11-02T17:58:25.205 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+DgBgmHb/2TqHM92bXLdcbfHKLw= 2023-11-02T17:58:25.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+DgBgmHb/2TqHM92bXLdcbfHKLw=', 'Content-Length': '0'} 2023-11-02T17:58:25.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000326e673d12bfcc4-006543e341-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.211 DEBUG:boto:b'ibtljn-12999falsedummyNCgTgcQgi48JNmlzIJZc2XV1YwJaH71true2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaEtrue2023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2iLD9aIguC0onhAlOj.xw657ssc7CVJGtrue2023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.212 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T17:58:25.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-11-02T17:58:25.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-11-02T17:58:25.213 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.214 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.214 DEBUG:boto:path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:25.215 DEBUG:boto:auth_path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:25.215 DEBUG:boto:Method: GET 2023-11-02T17:58:25.216 DEBUG:boto:Path: /ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:25.216 DEBUG:boto:Data: 2023-11-02T17:58:25.217 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.217 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.218 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.218 DEBUG:boto:Params: {} 2023-11-02T17:58:25.219 DEBUG:boto:Token: None 2023-11-02T17:58:25.219 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/?versions 2023-11-02T17:58:25.220 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+DgBgmHb/2TqHM92bXLdcbfHKLw= 2023-11-02T17:58:25.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+DgBgmHb/2TqHM92bXLdcbfHKLw=', 'Content-Length': '0'} 2023-11-02T17:58:25.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000361de08bee8b4746-006543e341-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.226 DEBUG:boto:b'ibtljn-12999falsedummyNCgTgcQgi48JNmlzIJZc2XV1YwJaH71true2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaEtrue2023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2iLD9aIguC0onhAlOj.xw657ssc7CVJGtrue2023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.227 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.228 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.228 DEBUG:boto:path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:25.229 DEBUG:boto:auth_path=/ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:25.229 DEBUG:boto:Method: GET 2023-11-02T17:58:25.230 DEBUG:boto:Path: /ibtljn-12/?versions&max-keys=999 2023-11-02T17:58:25.230 DEBUG:boto:Data: 2023-11-02T17:58:25.231 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.232 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.232 DEBUG:boto:Params: {} 2023-11-02T17:58:25.233 DEBUG:boto:Token: None 2023-11-02T17:58:25.233 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/?versions 2023-11-02T17:58:25.234 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+DgBgmHb/2TqHM92bXLdcbfHKLw= 2023-11-02T17:58:25.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+DgBgmHb/2TqHM92bXLdcbfHKLw=', 'Content-Length': '0'} 2023-11-02T17:58:25.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087172349e3c16780-006543e341-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.239 DEBUG:boto:b'ibtljn-12999falsedummyNCgTgcQgi48JNmlzIJZc2XV1YwJaH71true2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaEtrue2023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2iLD9aIguC0onhAlOj.xw657ssc7CVJGtrue2023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:58:25.241 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:25.241 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:25.242 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.242 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.243 DEBUG:boto:Method: GET 2023-11-02T17:58:25.243 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.245 DEBUG:boto:Data: 2023-11-02T17:58:25.246 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.247 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.247 DEBUG:boto:Params: {} 2023-11-02T17:58:25.248 DEBUG:boto:Token: None 2023-11-02T17:58:25.248 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.249 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Iqb9VYJaMVgq6H3Yip9O4yZS3b4= 2023-11-02T17:58:25.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Iqb9VYJaMVgq6H3Yip9O4yZS3b4=', 'Content-Length': '0'} 2023-11-02T17:58:25.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000788cdc626d7fdfde-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.253 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:25.253 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:25.254 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.254 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.255 DEBUG:boto:Method: GET 2023-11-02T17:58:25.255 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.255 DEBUG:boto:Data: 2023-11-02T17:58:25.256 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.256 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.257 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.257 DEBUG:boto:Params: {} 2023-11-02T17:58:25.258 DEBUG:boto:Token: None 2023-11-02T17:58:25.258 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.258 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Iqb9VYJaMVgq6H3Yip9O4yZS3b4= 2023-11-02T17:58:25.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Iqb9VYJaMVgq6H3Yip9O4yZS3b4=', 'Content-Length': '0'} 2023-11-02T17:58:25.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e5aee52ae3d6bb8-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.262 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:25.263 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:25.263 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.264 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.264 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.265 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.265 DEBUG:boto:Data: 2023-11-02T17:58:25.266 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.266 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.267 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.267 DEBUG:boto:Params: {} 2023-11-02T17:58:25.268 DEBUG:boto:Token: None 2023-11-02T17:58:25.268 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.268 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EWoVo+/ZVi9GNZR+X9xLm0pRKE0= 2023-11-02T17:58:25.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EWoVo+/ZVi9GNZR+X9xLm0pRKE0=', 'Content-Length': '0'} 2023-11-02T17:58:25.272 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fbc332332920b1c4-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.272 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:25.273 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:25.273 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.273 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.274 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.274 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.275 DEBUG:boto:Data: 2023-11-02T17:58:25.275 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.276 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.276 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.276 DEBUG:boto:Params: {} 2023-11-02T17:58:25.277 DEBUG:boto:Token: None 2023-11-02T17:58:25.277 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.278 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EWoVo+/ZVi9GNZR+X9xLm0pRKE0= 2023-11-02T17:58:25.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EWoVo+/ZVi9GNZR+X9xLm0pRKE0=', 'Content-Length': '0'} 2023-11-02T17:58:25.281 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f110cadcc6ad844a-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:58:25.282 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:25.283 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:25.283 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.284 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.284 DEBUG:boto:Method: GET 2023-11-02T17:58:25.285 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.285 DEBUG:boto:Data: 2023-11-02T17:58:25.285 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.286 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.286 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.287 DEBUG:boto:Params: {} 2023-11-02T17:58:25.287 DEBUG:boto:Token: None 2023-11-02T17:58:25.288 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.288 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Iqb9VYJaMVgq6H3Yip9O4yZS3b4= 2023-11-02T17:58:25.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Iqb9VYJaMVgq6H3Yip9O4yZS3b4=', 'Content-Length': '0'} 2023-11-02T17:58:25.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000088f3c47d54051b26-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.292 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:25.293 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:25.293 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.293 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.294 DEBUG:boto:Method: GET 2023-11-02T17:58:25.294 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.295 DEBUG:boto:Data: 2023-11-02T17:58:25.295 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.296 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.296 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.296 DEBUG:boto:Params: {} 2023-11-02T17:58:25.297 DEBUG:boto:Token: None 2023-11-02T17:58:25.297 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.298 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Iqb9VYJaMVgq6H3Yip9O4yZS3b4= 2023-11-02T17:58:25.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Iqb9VYJaMVgq6H3Yip9O4yZS3b4=', 'Content-Length': '0'} 2023-11-02T17:58:25.301 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000203db18263ab79f1-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.302 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.302 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.303 DEBUG:boto:path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:25.303 DEBUG:boto:auth_path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:25.304 DEBUG:boto:Method: GET 2023-11-02T17:58:25.304 DEBUG:boto:Path: /ibtljn-12/?prefix=dummy 2023-11-02T17:58:25.305 DEBUG:boto:Data: 2023-11-02T17:58:25.305 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.306 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.306 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.307 DEBUG:boto:Params: {} 2023-11-02T17:58:25.307 DEBUG:boto:Token: None 2023-11-02T17:58:25.308 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.308 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8= 2023-11-02T17:58:25.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8=', 'Content-Length': '0'} 2023-11-02T17:58:25.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc50c33ac3437500-006543e341-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.354 DEBUG:boto:b'ibtljn-12dummy1000falsedummy2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.355 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.355 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.356 DEBUG:boto:path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:25.357 DEBUG:boto:auth_path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:25.357 DEBUG:boto:Method: GET 2023-11-02T17:58:25.357 DEBUG:boto:Path: /ibtljn-12/?prefix=dummy 2023-11-02T17:58:25.358 DEBUG:boto:Data: 2023-11-02T17:58:25.358 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.359 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.359 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.360 DEBUG:boto:Params: {} 2023-11-02T17:58:25.360 DEBUG:boto:Token: None 2023-11-02T17:58:25.361 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.362 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8= 2023-11-02T17:58:25.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8=', 'Content-Length': '0'} 2023-11-02T17:58:25.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ae0b981c3fd694-006543e341-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.368 DEBUG:boto:b'ibtljn-12dummy1000falsedummy2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.369 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:25.369 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:25.369 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.370 DEBUG:boto:Path: /ibtljn-12/dummy 2023-11-02T17:58:25.370 DEBUG:boto:Data: 2023-11-02T17:58:25.371 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.371 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.371 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.372 DEBUG:boto:Params: {} 2023-11-02T17:58:25.372 DEBUG:boto:Token: None 2023-11-02T17:58:25.373 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/dummy 2023-11-02T17:58:25.373 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uBIrGCp1qQLxzDyfmk7saQnTpEs= 2023-11-02T17:58:25.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uBIrGCp1qQLxzDyfmk7saQnTpEs=', 'Content-Length': '0'} 2023-11-02T17:58:25.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f2f468754c1e1dcf-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.378 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:25.378 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:25.379 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.379 DEBUG:boto:Path: /ibtljn-12/dummy 2023-11-02T17:58:25.380 DEBUG:boto:Data: 2023-11-02T17:58:25.380 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.380 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.381 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.381 DEBUG:boto:Params: {} 2023-11-02T17:58:25.382 DEBUG:boto:Token: None 2023-11-02T17:58:25.382 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/dummy 2023-11-02T17:58:25.382 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uBIrGCp1qQLxzDyfmk7saQnTpEs= 2023-11-02T17:58:25.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uBIrGCp1qQLxzDyfmk7saQnTpEs=', 'Content-Length': '0'} 2023-11-02T17:58:25.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000066ee699dc0f7d1d9-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:58:25.388 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:25.388 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:25.389 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.389 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.390 DEBUG:boto:Method: GET 2023-11-02T17:58:25.390 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.390 DEBUG:boto:Data: 2023-11-02T17:58:25.391 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.391 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.392 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.392 DEBUG:boto:Params: {} 2023-11-02T17:58:25.393 DEBUG:boto:Token: None 2023-11-02T17:58:25.393 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Iqb9VYJaMVgq6H3Yip9O4yZS3b4= 2023-11-02T17:58:25.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Iqb9VYJaMVgq6H3Yip9O4yZS3b4=', 'Content-Length': '0'} 2023-11-02T17:58:25.397 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb75c742950601e8-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.398 DEBUG:boto:path=/ibtljn-12/dummy 2023-11-02T17:58:25.398 DEBUG:boto:auth_path=/ibtljn-12/dummy 2023-11-02T17:58:25.399 DEBUG:boto:path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.399 DEBUG:boto:auth_path=/ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.400 DEBUG:boto:Method: GET 2023-11-02T17:58:25.400 DEBUG:boto:Path: /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.401 DEBUG:boto:Data: 2023-11-02T17:58:25.401 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.402 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.402 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.402 DEBUG:boto:Params: {} 2023-11-02T17:58:25.403 DEBUG:boto:Token: None 2023-11-02T17:58:25.404 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/dummy?versionId=NCgTgcQgi48JNmlzIJZc2XV1YwJaH71 2023-11-02T17:58:25.404 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Iqb9VYJaMVgq6H3Yip9O4yZS3b4= 2023-11-02T17:58:25.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Iqb9VYJaMVgq6H3Yip9O4yZS3b4=', 'Content-Length': '0'} 2023-11-02T17:58:25.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:56:45 GMT'), ('x-amz-version-id', 'NCgTgcQgi48JNmlzIJZc2XV1YwJaH71'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001493ddff9d58411c-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.408 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.409 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.409 DEBUG:boto:path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:25.410 DEBUG:boto:auth_path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:25.410 DEBUG:boto:Method: GET 2023-11-02T17:58:25.411 DEBUG:boto:Path: /ibtljn-12/?prefix=dummy 2023-11-02T17:58:25.411 DEBUG:boto:Data: 2023-11-02T17:58:25.411 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.412 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.413 DEBUG:boto:Params: {} 2023-11-02T17:58:25.413 DEBUG:boto:Token: None 2023-11-02T17:58:25.414 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8= 2023-11-02T17:58:25.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8=', 'Content-Length': '0'} 2023-11-02T17:58:25.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eeef18076a85b16-006543e341-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.420 DEBUG:boto:b'ibtljn-12dummy1000falsedummy2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.421 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.422 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.422 DEBUG:boto:path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:25.423 DEBUG:boto:auth_path=/ibtljn-12/?prefix=dummy 2023-11-02T17:58:25.423 DEBUG:boto:Method: GET 2023-11-02T17:58:25.424 DEBUG:boto:Path: /ibtljn-12/?prefix=dummy 2023-11-02T17:58:25.424 DEBUG:boto:Data: 2023-11-02T17:58:25.425 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.425 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.426 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.426 DEBUG:boto:Params: {} 2023-11-02T17:58:25.427 DEBUG:boto:Token: None 2023-11-02T17:58:25.427 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8= 2023-11-02T17:58:25.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8=', 'Content-Length': '0'} 2023-11-02T17:58:25.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000817fa3d53fbdaaf1-006543e341-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.478 DEBUG:boto:b'ibtljn-12dummy1000falsedummy2023-11-02T17:56:45.185Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-11-02T17:58:25.480 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.480 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.480 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.481 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.481 DEBUG:boto:Method: GET 2023-11-02T17:58:25.481 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.482 DEBUG:boto:Data: 2023-11-02T17:58:25.482 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.483 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.483 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.484 DEBUG:boto:Params: {} 2023-11-02T17:58:25.484 DEBUG:boto:Token: None 2023-11-02T17:58:25.485 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.485 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HzIaicTebnYlety34cMVOFuJjfo= 2023-11-02T17:58:25.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HzIaicTebnYlety34cMVOFuJjfo=', 'Content-Length': '0'} 2023-11-02T17:58:25.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073bbf0e320998992-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.490 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.491 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.491 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.491 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.492 DEBUG:boto:Method: GET 2023-11-02T17:58:25.492 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.493 DEBUG:boto:Data: 2023-11-02T17:58:25.493 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.494 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.494 DEBUG:boto:Params: {} 2023-11-02T17:58:25.495 DEBUG:boto:Token: None 2023-11-02T17:58:25.495 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HzIaicTebnYlety34cMVOFuJjfo= 2023-11-02T17:58:25.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HzIaicTebnYlety34cMVOFuJjfo=', 'Content-Length': '0'} 2023-11-02T17:58:25.499 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b45e42985f78ae5-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.500 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.500 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.501 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.501 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.502 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.502 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.503 DEBUG:boto:Data: 2023-11-02T17:58:25.503 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.504 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.504 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.505 DEBUG:boto:Params: {} 2023-11-02T17:58:25.505 DEBUG:boto:Token: None 2023-11-02T17:58:25.506 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.506 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aGkSRX9tNBktfvpe+jjzjSVrR6s= 2023-11-02T17:58:25.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aGkSRX9tNBktfvpe+jjzjSVrR6s=', 'Content-Length': '0'} 2023-11-02T17:58:25.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6a1511f57275a20-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.510 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.510 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.511 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.511 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.512 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.512 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.513 DEBUG:boto:Data: 2023-11-02T17:58:25.513 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.513 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.514 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.514 DEBUG:boto:Params: {} 2023-11-02T17:58:25.515 DEBUG:boto:Token: None 2023-11-02T17:58:25.516 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.516 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aGkSRX9tNBktfvpe+jjzjSVrR6s= 2023-11-02T17:58:25.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aGkSRX9tNBktfvpe+jjzjSVrR6s=', 'Content-Length': '0'} 2023-11-02T17:58:25.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ecf0549178dee10-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-11-02T17:58:25.520 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.521 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.521 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.522 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.522 DEBUG:boto:Method: GET 2023-11-02T17:58:25.523 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.523 DEBUG:boto:Data: 2023-11-02T17:58:25.524 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.524 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.525 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.525 DEBUG:boto:Params: {} 2023-11-02T17:58:25.526 DEBUG:boto:Token: None 2023-11-02T17:58:25.526 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.527 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HzIaicTebnYlety34cMVOFuJjfo= 2023-11-02T17:58:25.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HzIaicTebnYlety34cMVOFuJjfo=', 'Content-Length': '0'} 2023-11-02T17:58:25.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055de3a3d0a0b6b4e-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.531 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.531 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.532 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.532 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.533 DEBUG:boto:Method: GET 2023-11-02T17:58:25.533 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.533 DEBUG:boto:Data: 2023-11-02T17:58:25.534 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.534 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.535 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.535 DEBUG:boto:Params: {} 2023-11-02T17:58:25.536 DEBUG:boto:Token: None 2023-11-02T17:58:25.536 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.537 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HzIaicTebnYlety34cMVOFuJjfo= 2023-11-02T17:58:25.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HzIaicTebnYlety34cMVOFuJjfo=', 'Content-Length': '0'} 2023-11-02T17:58:25.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0eef9190e8f229d-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.582 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.583 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.583 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:25.584 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:25.584 DEBUG:boto:Method: GET 2023-11-02T17:58:25.585 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:25.585 DEBUG:boto:Data: 2023-11-02T17:58:25.586 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.586 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.587 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.587 DEBUG:boto:Params: {} 2023-11-02T17:58:25.587 DEBUG:boto:Token: None 2023-11-02T17:58:25.588 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.588 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8= 2023-11-02T17:58:25.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8=', 'Content-Length': '0'} 2023-11-02T17:58:25.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064539cf9c980bbc5-006543e341-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.595 DEBUG:boto:b'ibtljn-12obj-a11000falseobj-a12023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.595 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.596 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.596 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:25.597 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:25.597 DEBUG:boto:Method: GET 2023-11-02T17:58:25.598 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:25.598 DEBUG:boto:Data: 2023-11-02T17:58:25.598 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.599 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.599 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.600 DEBUG:boto:Params: {} 2023-11-02T17:58:25.600 DEBUG:boto:Token: None 2023-11-02T17:58:25.601 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8= 2023-11-02T17:58:25.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8=', 'Content-Length': '0'} 2023-11-02T17:58:25.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de61c2e1ceff6fec-006543e341-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.608 DEBUG:boto:b'ibtljn-12obj-a11000falseobj-a12023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.609 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.609 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.610 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.610 DEBUG:boto:Path: /ibtljn-12/obj-a1 2023-11-02T17:58:25.611 DEBUG:boto:Data: 2023-11-02T17:58:25.611 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.611 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.612 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.612 DEBUG:boto:Params: {} 2023-11-02T17:58:25.613 DEBUG:boto:Token: None 2023-11-02T17:58:25.613 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a1 2023-11-02T17:58:25.614 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9rIJ9mnLzb2xlFVCc8hXPfmcSFg= 2023-11-02T17:58:25.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9rIJ9mnLzb2xlFVCc8hXPfmcSFg=', 'Content-Length': '0'} 2023-11-02T17:58:25.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006deae908fb28b097-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.619 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.619 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.620 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.620 DEBUG:boto:Path: /ibtljn-12/obj-a1 2023-11-02T17:58:25.621 DEBUG:boto:Data: 2023-11-02T17:58:25.621 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.621 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.622 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.622 DEBUG:boto:Params: {} 2023-11-02T17:58:25.623 DEBUG:boto:Token: None 2023-11-02T17:58:25.623 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a1 2023-11-02T17:58:25.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9rIJ9mnLzb2xlFVCc8hXPfmcSFg= 2023-11-02T17:58:25.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9rIJ9mnLzb2xlFVCc8hXPfmcSFg=', 'Content-Length': '0'} 2023-11-02T17:58:25.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b1a19f4b0d64a75-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-11-02T17:58:25.629 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.630 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.630 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.631 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.631 DEBUG:boto:Method: GET 2023-11-02T17:58:25.631 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.632 DEBUG:boto:Data: 2023-11-02T17:58:25.632 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.633 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.633 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.634 DEBUG:boto:Params: {} 2023-11-02T17:58:25.634 DEBUG:boto:Token: None 2023-11-02T17:58:25.635 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.635 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HzIaicTebnYlety34cMVOFuJjfo= 2023-11-02T17:58:25.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HzIaicTebnYlety34cMVOFuJjfo=', 'Content-Length': '0'} 2023-11-02T17:58:25.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010dbdb87e0e46041-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.639 DEBUG:boto:path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.640 DEBUG:boto:auth_path=/ibtljn-12/obj-a1 2023-11-02T17:58:25.640 DEBUG:boto:path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.641 DEBUG:boto:auth_path=/ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.641 DEBUG:boto:Method: GET 2023-11-02T17:58:25.641 DEBUG:boto:Path: /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.642 DEBUG:boto:Data: 2023-11-02T17:58:25.642 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.643 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.643 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.644 DEBUG:boto:Params: {} 2023-11-02T17:58:25.644 DEBUG:boto:Token: None 2023-11-02T17:58:25.645 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a1?versionId=IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE 2023-11-02T17:58:25.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HzIaicTebnYlety34cMVOFuJjfo= 2023-11-02T17:58:25.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HzIaicTebnYlety34cMVOFuJjfo=', 'Content-Length': '0'} 2023-11-02T17:58:25.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'IJ1MPe5QqJJ29GJGMOpjyuPWYtNodaE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009aa74a694f71e92c-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.690 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.692 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.692 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:25.693 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:25.693 DEBUG:boto:Method: GET 2023-11-02T17:58:25.694 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:25.694 DEBUG:boto:Data: 2023-11-02T17:58:25.695 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.696 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.696 DEBUG:boto:Params: {} 2023-11-02T17:58:25.697 DEBUG:boto:Token: None 2023-11-02T17:58:25.697 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.698 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8= 2023-11-02T17:58:25.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8=', 'Content-Length': '0'} 2023-11-02T17:58:25.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007af71f368069afa0-006543e341-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.703 DEBUG:boto:b'ibtljn-12obj-a11000falseobj-a12023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.704 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.704 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.705 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:25.705 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:25.706 DEBUG:boto:Method: GET 2023-11-02T17:58:25.706 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a1 2023-11-02T17:58:25.707 DEBUG:boto:Data: 2023-11-02T17:58:25.707 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.708 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.708 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.708 DEBUG:boto:Params: {} 2023-11-02T17:58:25.709 DEBUG:boto:Token: None 2023-11-02T17:58:25.709 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.710 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8= 2023-11-02T17:58:25.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8=', 'Content-Length': '0'} 2023-11-02T17:58:25.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065960604a273bb3e-006543e341-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.716 DEBUG:boto:b'ibtljn-12obj-a11000falseobj-a12023-11-02T17:57:05.737Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-11-02T17:58:25.717 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.718 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.718 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.719 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.719 DEBUG:boto:Method: GET 2023-11-02T17:58:25.720 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.720 DEBUG:boto:Data: 2023-11-02T17:58:25.721 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.721 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.721 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.722 DEBUG:boto:Params: {} 2023-11-02T17:58:25.722 DEBUG:boto:Token: None 2023-11-02T17:58:25.723 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.723 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPKyW3K11wmycIfuQ9jwyot4xns= 2023-11-02T17:58:25.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPKyW3K11wmycIfuQ9jwyot4xns=', 'Content-Length': '0'} 2023-11-02T17:58:25.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000499e35643d0eed03-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.770 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.771 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.771 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.772 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.772 DEBUG:boto:Method: GET 2023-11-02T17:58:25.773 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.773 DEBUG:boto:Data: 2023-11-02T17:58:25.774 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.774 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.774 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.775 DEBUG:boto:Params: {} 2023-11-02T17:58:25.775 DEBUG:boto:Token: None 2023-11-02T17:58:25.776 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.777 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPKyW3K11wmycIfuQ9jwyot4xns= 2023-11-02T17:58:25.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPKyW3K11wmycIfuQ9jwyot4xns=', 'Content-Length': '0'} 2023-11-02T17:58:25.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000284cd5f777b42cad-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.781 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.781 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.782 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.782 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.783 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.783 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.784 DEBUG:boto:Data: 2023-11-02T17:58:25.784 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.785 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.785 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.785 DEBUG:boto:Params: {} 2023-11-02T17:58:25.786 DEBUG:boto:Token: None 2023-11-02T17:58:25.786 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.787 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VYPZSO7No9w7TtUVYj7giF9FF1o= 2023-11-02T17:58:25.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VYPZSO7No9w7TtUVYj7giF9FF1o=', 'Content-Length': '0'} 2023-11-02T17:58:25.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042321c5b47000dcc-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.791 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.791 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.791 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.792 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.792 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.793 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.793 DEBUG:boto:Data: 2023-11-02T17:58:25.793 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.794 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.794 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.795 DEBUG:boto:Params: {} 2023-11-02T17:58:25.795 DEBUG:boto:Token: None 2023-11-02T17:58:25.796 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.796 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VYPZSO7No9w7TtUVYj7giF9FF1o= 2023-11-02T17:58:25.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VYPZSO7No9w7TtUVYj7giF9FF1o=', 'Content-Length': '0'} 2023-11-02T17:58:25.800 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1d7dac83d5c4902-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-11-02T17:58:25.801 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.801 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.802 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.802 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.803 DEBUG:boto:Method: GET 2023-11-02T17:58:25.803 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.804 DEBUG:boto:Data: 2023-11-02T17:58:25.804 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.805 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.805 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.806 DEBUG:boto:Params: {} 2023-11-02T17:58:25.806 DEBUG:boto:Token: None 2023-11-02T17:58:25.807 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.807 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPKyW3K11wmycIfuQ9jwyot4xns= 2023-11-02T17:58:25.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPKyW3K11wmycIfuQ9jwyot4xns=', 'Content-Length': '0'} 2023-11-02T17:58:25.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000980688b6c9db9161-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.811 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.812 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.812 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.813 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.813 DEBUG:boto:Method: GET 2023-11-02T17:58:25.813 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.814 DEBUG:boto:Data: 2023-11-02T17:58:25.814 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.815 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.815 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.816 DEBUG:boto:Params: {} 2023-11-02T17:58:25.816 DEBUG:boto:Token: None 2023-11-02T17:58:25.817 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.817 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPKyW3K11wmycIfuQ9jwyot4xns= 2023-11-02T17:58:25.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPKyW3K11wmycIfuQ9jwyot4xns=', 'Content-Length': '0'} 2023-11-02T17:58:25.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df226227182e0b49-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.866 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.867 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.867 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:25.868 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:25.868 DEBUG:boto:Method: GET 2023-11-02T17:58:25.869 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:25.869 DEBUG:boto:Data: 2023-11-02T17:58:25.870 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.870 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.871 DEBUG:boto:Params: {} 2023-11-02T17:58:25.871 DEBUG:boto:Token: None 2023-11-02T17:58:25.872 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.872 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8= 2023-11-02T17:58:25.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8=', 'Content-Length': '0'} 2023-11-02T17:58:25.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009931d318c771047-006543e341-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.878 DEBUG:boto:b'ibtljn-12obj-a21000falseobj-a22023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.879 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.880 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.880 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:25.880 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:25.881 DEBUG:boto:Method: GET 2023-11-02T17:58:25.881 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:25.882 DEBUG:boto:Data: 2023-11-02T17:58:25.882 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.883 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.883 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.884 DEBUG:boto:Params: {} 2023-11-02T17:58:25.884 DEBUG:boto:Token: None 2023-11-02T17:58:25.885 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8= 2023-11-02T17:58:25.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8=', 'Content-Length': '0'} 2023-11-02T17:58:25.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000127ebaa232232d72-006543e341-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.891 DEBUG:boto:b'ibtljn-12obj-a21000falseobj-a22023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.892 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.892 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.893 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.893 DEBUG:boto:Path: /ibtljn-12/obj-a2 2023-11-02T17:58:25.894 DEBUG:boto:Data: 2023-11-02T17:58:25.894 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.895 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.895 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.895 DEBUG:boto:Params: {} 2023-11-02T17:58:25.896 DEBUG:boto:Token: None 2023-11-02T17:58:25.896 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a2 2023-11-02T17:58:25.897 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X643u0LjiHA/2r3w3ZuBqAe5RVE= 2023-11-02T17:58:25.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X643u0LjiHA/2r3w3ZuBqAe5RVE=', 'Content-Length': '0'} 2023-11-02T17:58:25.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f3ae146084d4819-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.904 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.905 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.905 DEBUG:boto:Method: HEAD 2023-11-02T17:58:25.906 DEBUG:boto:Path: /ibtljn-12/obj-a2 2023-11-02T17:58:25.906 DEBUG:boto:Data: 2023-11-02T17:58:25.906 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.907 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.907 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.908 DEBUG:boto:Params: {} 2023-11-02T17:58:25.908 DEBUG:boto:Token: None 2023-11-02T17:58:25.909 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a2 2023-11-02T17:58:25.909 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X643u0LjiHA/2r3w3ZuBqAe5RVE= 2023-11-02T17:58:25.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X643u0LjiHA/2r3w3ZuBqAe5RVE=', 'Content-Length': '0'} 2023-11-02T17:58:25.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2915feda21f9b37-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-11-02T17:58:25.915 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.915 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.915 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.916 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.916 DEBUG:boto:Method: GET 2023-11-02T17:58:25.917 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.917 DEBUG:boto:Data: 2023-11-02T17:58:25.918 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.919 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.919 DEBUG:boto:Params: {} 2023-11-02T17:58:25.920 DEBUG:boto:Token: None 2023-11-02T17:58:25.920 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.921 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPKyW3K11wmycIfuQ9jwyot4xns= 2023-11-02T17:58:25.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPKyW3K11wmycIfuQ9jwyot4xns=', 'Content-Length': '0'} 2023-11-02T17:58:25.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055bb0442fb3ee2e6-006543e341-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.966 DEBUG:boto:path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.967 DEBUG:boto:auth_path=/ibtljn-12/obj-a2 2023-11-02T17:58:25.967 DEBUG:boto:path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.968 DEBUG:boto:auth_path=/ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.968 DEBUG:boto:Method: GET 2023-11-02T17:58:25.969 DEBUG:boto:Path: /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.969 DEBUG:boto:Data: 2023-11-02T17:58:25.970 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.970 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.970 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.971 DEBUG:boto:Params: {} 2023-11-02T17:58:25.971 DEBUG:boto:Token: None 2023-11-02T17:58:25.972 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/obj-a2?versionId=iLD9aIguC0onhAlOj.xw657ssc7CVJG 2023-11-02T17:58:25.972 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPKyW3K11wmycIfuQ9jwyot4xns= 2023-11-02T17:58:25.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPKyW3K11wmycIfuQ9jwyot4xns=', 'Content-Length': '0'} 2023-11-02T17:58:25.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:57:05 GMT'), ('x-amz-version-id', 'iLD9aIguC0onhAlOj.xw657ssc7CVJG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005179f250744096f0-006543e341-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.977 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.977 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.978 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:25.978 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:25.979 DEBUG:boto:Method: GET 2023-11-02T17:58:25.979 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:25.979 DEBUG:boto:Data: 2023-11-02T17:58:25.980 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.980 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:58:25.981 DEBUG:boto:Port: 8001 2023-11-02T17:58:25.981 DEBUG:boto:Params: {} 2023-11-02T17:58:25.982 DEBUG:boto:Token: None 2023-11-02T17:58:25.982 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.983 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8= 2023-11-02T17:58:25.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8=', 'Content-Length': '0'} 2023-11-02T17:58:25.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4a2c1d800415d81-006543e341-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:25.989 DEBUG:boto:b'ibtljn-12obj-a21000falseobj-a22023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:25.990 DEBUG:boto:path=/ibtljn-12/ 2023-11-02T17:58:25.990 DEBUG:boto:auth_path=/ibtljn-12/ 2023-11-02T17:58:25.991 DEBUG:boto:path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:25.991 DEBUG:boto:auth_path=/ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:25.992 DEBUG:boto:Method: GET 2023-11-02T17:58:25.992 DEBUG:boto:Path: /ibtljn-12/?prefix=obj-a2 2023-11-02T17:58:25.993 DEBUG:boto:Data: 2023-11-02T17:58:25.993 DEBUG:boto:Headers: {} 2023-11-02T17:58:25.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:25.994 DEBUG:boto:Port: 8000 2023-11-02T17:58:25.994 DEBUG:boto:Params: {} 2023-11-02T17:58:25.995 DEBUG:boto:Token: None 2023-11-02T17:58:25.995 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:58:25 GMT /ibtljn-12/ 2023-11-02T17:58:25.996 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8= 2023-11-02T17:58:25.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DwLN+VYxekyrA+TinFz78W0UGY8=', 'Content-Length': '0'} 2023-11-02T17:58:26.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002694d6aad0dff7a7-006543e341-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:58:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:26.042 DEBUG:boto:b'ibtljn-12obj-a21000falseobj-a22023-11-02T17:57:05.910Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:58:26.043 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-12 zones={a2, a1} 2023-11-02T17:58:26.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2023-11-02T17:58:26.045 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:58:26.046 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:58:26.046 DEBUG:boto:Method: PUT 2023-11-02T17:58:26.047 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:58:26.047 DEBUG:boto:Data: 2023-11-02T17:58:26.048 DEBUG:boto:Headers: {} 2023-11-02T17:58:26.048 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:26.049 DEBUG:boto:Port: 8000 2023-11-02T17:58:26.049 DEBUG:boto:Params: {} 2023-11-02T17:58:26.050 DEBUG:boto:Token: None 2023-11-02T17:58:26.050 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:58:26 GMT /ibtljn-13/ 2023-11-02T17:58:26.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PRqkDhuyebCIKVNTqrc2XptIq6E= 2023-11-02T17:58:26.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PRqkDhuyebCIKVNTqrc2XptIq6E=', 'Content-Length': '0'} 2023-11-02T17:58:26.075 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000601e8e336b9ade93-006543e342-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:26.076 DEBUG:rgw_multi.tests:created bucket=ibtljn-13 2023-11-02T17:58:26.077 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:58:26.077 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:58:26.078 DEBUG:boto:path=/ibtljn-13/?versioning 2023-11-02T17:58:26.078 DEBUG:boto:auth_path=/ibtljn-13/?versioning 2023-11-02T17:58:26.079 DEBUG:boto:Method: PUT 2023-11-02T17:58:26.079 DEBUG:boto:Path: /ibtljn-13/?versioning 2023-11-02T17:58:26.080 DEBUG:boto:Data: Enabled Disabled 2023-11-02T17:58:26.080 DEBUG:boto:Headers: {} 2023-11-02T17:58:26.081 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:26.081 DEBUG:boto:Port: 8000 2023-11-02T17:58:26.082 DEBUG:boto:Params: {} 2023-11-02T17:58:26.082 DEBUG:boto:Token: None 2023-11-02T17:58:26.083 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 17:58:26 GMT /ibtljn-13/?versioning 2023-11-02T17:58:26.083 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f2IfqeC/wlw6ItZxhAYohtS+0GE= 2023-11-02T17:58:26.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f2IfqeC/wlw6ItZxhAYohtS+0GE=', 'Content-Length': '228'} 2023-11-02T17:58:26.093 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f5f78ebe479e8d00-006543e342-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:26.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:58:26.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:58:26.094 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:58:26.137 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:58:26.138 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:58:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T17:58:26.226 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T17:58:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:58:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:58:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T17:58:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:58:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:58:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T17:58:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:58:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947906.086407_70.1", 2023-11-02T17:58:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:58:26.086407Z" 2023-11-02T17:58:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:58:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T17:58:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T17:58:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:58:26.230 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T17:58:26.231 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947906.086407_70.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:58:26.232 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T17:58:26.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:58:26.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:58:26.233 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:58:26.285 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:58:26.285 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:58:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:26.326+0000 7f8e2a05ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:58:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:26.326+0000 7f8e2a05ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:58:26.360 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:58:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:58:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:58:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:58:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:58:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:58:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:58:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:58:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:58:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:58:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:58:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:58:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:58:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:58:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:58:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:58:26.365 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:58:26.365 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:58:26.365 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:58:26.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:26.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:58:26.366 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:58:26.366 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:58:26.366 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:58:26.367 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:58:26.367 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:58:26.367 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:58:26.367 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:58:26.367 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:58:26.368 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:58:26.368 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:58:26.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:26.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:58:26.369 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:58:26.369 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:58:26.369 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:58:26.369 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947906.059364_65.1", 2023-11-02T17:58:26.370 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:58:26.370 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:58:26.370 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:58:26.370 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:58:26.059364Z", 2023-11-02T17:58:26.370 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:58:26.371 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:58:26.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:26.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:58:26.371 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:58:26.372 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:58:26.372 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:58:26.372 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T17:58:26.372 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:58:26.372 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:58:26.373 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:58:26.373 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T17:58:26.373 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:58:26.373 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:58:26.373 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:58:26.374 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:58:26.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:26.374 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:58:26.374 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:58:26.374 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:58:26.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:26.375 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:58:26Z" 2023-11-02T17:58:26.375 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:58:26.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698947906.059364_65.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:58:26.059364Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T17:58:26Z" } 2023-11-02T17:58:26.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698947906.059364_65.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:58:26.059364Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T17:58:26.377 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947906.086407_70.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:58:26.378 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947906.059364_65.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:58:26.378 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698947906.086407_70.1 target=1_1698947906.059364_65.1 2023-11-02T17:58:31.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:58:31.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T17:58:31.382 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T17:58:31.428 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:58:31.429 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:58:31.485 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:31.477+0000 7f8d61ef8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:58:31.486 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:31.477+0000 7f8d61ef8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T17:58:31.509 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T17:58:31.510 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T17:58:31.510 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T17:58:31.510 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T17:58:31.511 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T17:58:31.511 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T17:58:31.511 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:58:31.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:31.511 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T17:58:31.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:58:31.512 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T17:58:31.512 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:58:31.512 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:58:31.513 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T17:58:31.513 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:58:31.513 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T17:58:31.513 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:58:31.513 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T17:58:31.514 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:58:31.514 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:58:31.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:31.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:58:31.515 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T17:58:31.515 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:58:31.515 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:58:31.515 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T17:58:31.515 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:58:31.516 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:58:31.516 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:58:31.516 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T17:58:31.516 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:58:31.517 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:58:31.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:31.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:58:31.517 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T17:58:31.517 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:58:31.518 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:58:31.518 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947906.086407_70.1", 2023-11-02T17:58:31.518 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:58:31.518 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:58:31.519 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:58:31.519 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:58:26.086407Z", 2023-11-02T17:58:31.519 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:58:31.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:58:31.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:31.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T17:58:31.520 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T17:58:31.520 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T17:58:31.520 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T17:58:31.521 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T17:58:31.521 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T17:58:31.521 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T17:58:31.521 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T17:58:31.521 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T17:58:31.522 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T17:58:31.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:58:31.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T17:58:31.522 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T17:58:31.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:31.523 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T17:58:31.523 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T17:58:31.523 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T17:58:31.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T17:58:31.523 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T17:58:31Z" 2023-11-02T17:58:31.524 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T17:58:31.525 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698947906.086407_70.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:58:26.086407Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T17:58:31Z" } 2023-11-02T17:58:31.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698947906.086407_70.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:58:26.086407Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T17:58:31.526 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947906.086407_70.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:58:31.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698947906.086407_70.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T17:58:31.527 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T17:58:31.527 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:58:31.527 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:58:31.528 DEBUG:boto:Method: HEAD 2023-11-02T17:58:31.528 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:58:31.529 DEBUG:boto:Data: 2023-11-02T17:58:31.529 DEBUG:boto:Headers: {} 2023-11-02T17:58:31.530 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:31.530 DEBUG:boto:Port: 8000 2023-11-02T17:58:31.530 DEBUG:boto:Params: {} 2023-11-02T17:58:31.531 DEBUG:boto:Token: None 2023-11-02T17:58:31.531 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:58:31 GMT /ibtljn-13/ 2023-11-02T17:58:31.532 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fOhXcXOiKiTeOBNqIT2WUkh7Jc4= 2023-11-02T17:58:31.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:58:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fOhXcXOiKiTeOBNqIT2WUkh7Jc4=', 'Content-Length': '0'} 2023-11-02T17:58:31.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6234413520ac237-006543e347-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:31.537 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T17:58:31.538 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T17:58:31.538 DEBUG:boto:Method: PUT 2023-11-02T17:58:31.538 DEBUG:boto:Path: /ibtljn-13/dummy 2023-11-02T17:58:31.539 DEBUG:boto:Data: 2023-11-02T17:58:31.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-11-02T17:58:31.540 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:58:31.540 DEBUG:boto:Port: 8000 2023-11-02T17:58:31.541 DEBUG:boto:Params: {} 2023-11-02T17:58:31.541 DEBUG:boto:Token: None 2023-11-02T17:58:31.541 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 02 Nov 2023 17:58:31 GMT /ibtljn-13/dummy 2023-11-02T17:58:31.542 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QqXiXkhs0h3Pud9M2h/6txMWPRk= 2023-11-02T17:58:31.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:58:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QqXiXkhs0h3Pud9M2h/6txMWPRk='} 2023-11-02T17:58:31.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-amz-request-id', 'tx0000078315aa9ffaaf6ab-006543e347-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:58:31.587 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-13 2023-11-02T17:58:31.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:31.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:31.589 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:58:31.642 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:58:31.643 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:58:31.709 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:31.701+0000 7fd7d96cef40 1 waiting to reach incremental sync.. 2023-11-02T17:58:36.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:36.701+0000 7fd7d96cef40 1 waiting to reach incremental sync.. 2023-11-02T17:58:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:41.705+0000 7fd7d96cef40 1 bucket sync caught up with source: 2023-11-02T17:58:41.714 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.7.11, , , , , ] 2023-11-02T17:58:41.714 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.7.11, , , , , ] 2023-11-02T17:58:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:41.705+0000 7fd7d96cef40 0 bucket checkpoint complete 2023-11-02T17:58:41.741 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-13 2023-11-02T17:58:41.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:41.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:58:41.743 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:58:41.795 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:58:41.795 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:58:41.894 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:41.885+0000 7f29ddc87f40 1 waiting to reach incremental sync.. 2023-11-02T17:58:46.894 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:46.885+0000 7f29ddc87f40 1 waiting to reach incremental sync.. 2023-11-02T17:58:51.895 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:51.885+0000 7f29ddc87f40 1 waiting to reach incremental sync.. 2023-11-02T17:58:56.897 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:58:56.885+0000 7f29ddc87f40 1 waiting to reach incremental sync.. 2023-11-02T17:59:01.896 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:01.885+0000 7f29ddc87f40 1 bucket sync caught up with source: 2023-11-02T17:59:01.897 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.12.11, , , , , ] 2023-11-02T17:59:01.897 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.12.11, , , , , ] 2023-11-02T17:59:01.897 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:01.885+0000 7f29ddc87f40 0 bucket checkpoint complete 2023-11-02T17:59:01.924 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-13 zones={a2, a1} 2023-11-02T17:59:01.925 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:01.926 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:01.926 DEBUG:boto:Method: HEAD 2023-11-02T17:59:01.926 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:01.927 DEBUG:boto:Data: 2023-11-02T17:59:01.927 DEBUG:boto:Headers: {} 2023-11-02T17:59:01.928 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:01.928 DEBUG:boto:Port: 8001 2023-11-02T17:59:01.929 DEBUG:boto:Params: {} 2023-11-02T17:59:01.929 DEBUG:boto:Token: None 2023-11-02T17:59:01.930 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:01 GMT /ibtljn-13/ 2023-11-02T17:59:01.930 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KNnCVavUCpfdCMkD9gLkjzH36UI= 2023-11-02T17:59:01.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KNnCVavUCpfdCMkD9gLkjzH36UI=', 'Content-Length': '0'} 2023-11-02T17:59:01.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee9ab618a3eaf376-006543e365-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:01.935 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:01.935 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:01.936 DEBUG:boto:Method: HEAD 2023-11-02T17:59:01.936 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:01.937 DEBUG:boto:Data: 2023-11-02T17:59:01.937 DEBUG:boto:Headers: {} 2023-11-02T17:59:01.938 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:01.938 DEBUG:boto:Port: 8000 2023-11-02T17:59:01.939 DEBUG:boto:Params: {} 2023-11-02T17:59:01.939 DEBUG:boto:Token: None 2023-11-02T17:59:01.939 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:01 GMT /ibtljn-13/ 2023-11-02T17:59:01.940 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KNnCVavUCpfdCMkD9gLkjzH36UI= 2023-11-02T17:59:01.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KNnCVavUCpfdCMkD9gLkjzH36UI=', 'Content-Length': '0'} 2023-11-02T17:59:01.945 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043588347c2c4d4d7-006543e365-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:01.945 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T17:59:01.946 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:01.946 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:01.947 DEBUG:boto:path=/ibtljn-13/?versions&max-keys=999 2023-11-02T17:59:01.948 DEBUG:boto:auth_path=/ibtljn-13/?versions&max-keys=999 2023-11-02T17:59:01.948 DEBUG:boto:Method: GET 2023-11-02T17:59:01.949 DEBUG:boto:Path: /ibtljn-13/?versions&max-keys=999 2023-11-02T17:59:01.949 DEBUG:boto:Data: 2023-11-02T17:59:01.949 DEBUG:boto:Headers: {} 2023-11-02T17:59:01.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:01.950 DEBUG:boto:Port: 8001 2023-11-02T17:59:01.951 DEBUG:boto:Params: {} 2023-11-02T17:59:01.951 DEBUG:boto:Token: None 2023-11-02T17:59:01.952 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:01 GMT /ibtljn-13/?versions 2023-11-02T17:59:01.952 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5LyGrrN1ApZqtchE/wtPnQrxsyI= 2023-11-02T17:59:01.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5LyGrrN1ApZqtchE/wtPnQrxsyI=', 'Content-Length': '0'} 2023-11-02T17:59:01.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d25357a57207aa3-006543e365-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:01.958 DEBUG:boto:b'ibtljn-13999falsedummyYfZfHEB2viPzAqRNKaaAKV7gnCKC.W1true2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:59:01.959 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T17:59:01.960 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T17:59:01.960 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:01.961 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:01.961 DEBUG:boto:path=/ibtljn-13/?versions&max-keys=999 2023-11-02T17:59:01.962 DEBUG:boto:auth_path=/ibtljn-13/?versions&max-keys=999 2023-11-02T17:59:01.962 DEBUG:boto:Method: GET 2023-11-02T17:59:01.963 DEBUG:boto:Path: /ibtljn-13/?versions&max-keys=999 2023-11-02T17:59:01.963 DEBUG:boto:Data: 2023-11-02T17:59:01.963 DEBUG:boto:Headers: {} 2023-11-02T17:59:01.964 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:01.964 DEBUG:boto:Port: 8000 2023-11-02T17:59:01.965 DEBUG:boto:Params: {} 2023-11-02T17:59:01.965 DEBUG:boto:Token: None 2023-11-02T17:59:01.966 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:01 GMT /ibtljn-13/?versions 2023-11-02T17:59:01.967 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5LyGrrN1ApZqtchE/wtPnQrxsyI= 2023-11-02T17:59:01.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5LyGrrN1ApZqtchE/wtPnQrxsyI=', 'Content-Length': '0'} 2023-11-02T17:59:01.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000413215dd79abb259-006543e365-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:01.973 DEBUG:boto:b'ibtljn-13999falsedummyYfZfHEB2viPzAqRNKaaAKV7gnCKC.W1true2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:59:01.974 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T17:59:01.974 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:01.975 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:01.975 DEBUG:boto:path=/ibtljn-13/?versions&max-keys=999 2023-11-02T17:59:01.976 DEBUG:boto:auth_path=/ibtljn-13/?versions&max-keys=999 2023-11-02T17:59:01.976 DEBUG:boto:Method: GET 2023-11-02T17:59:01.977 DEBUG:boto:Path: /ibtljn-13/?versions&max-keys=999 2023-11-02T17:59:01.977 DEBUG:boto:Data: 2023-11-02T17:59:01.977 DEBUG:boto:Headers: {} 2023-11-02T17:59:01.978 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:01.978 DEBUG:boto:Port: 8001 2023-11-02T17:59:01.979 DEBUG:boto:Params: {} 2023-11-02T17:59:01.979 DEBUG:boto:Token: None 2023-11-02T17:59:01.980 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:01 GMT /ibtljn-13/?versions 2023-11-02T17:59:01.980 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5LyGrrN1ApZqtchE/wtPnQrxsyI= 2023-11-02T17:59:01.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5LyGrrN1ApZqtchE/wtPnQrxsyI=', 'Content-Length': '0'} 2023-11-02T17:59:01.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026619be88affb55f-006543e365-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.026 DEBUG:boto:b'ibtljn-13999falsedummyYfZfHEB2viPzAqRNKaaAKV7gnCKC.W1true2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:59:02.027 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.028 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.028 DEBUG:boto:path=/ibtljn-13/?versions&max-keys=999 2023-11-02T17:59:02.029 DEBUG:boto:auth_path=/ibtljn-13/?versions&max-keys=999 2023-11-02T17:59:02.029 DEBUG:boto:Method: GET 2023-11-02T17:59:02.030 DEBUG:boto:Path: /ibtljn-13/?versions&max-keys=999 2023-11-02T17:59:02.030 DEBUG:boto:Data: 2023-11-02T17:59:02.031 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.031 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.031 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.032 DEBUG:boto:Params: {} 2023-11-02T17:59:02.032 DEBUG:boto:Token: None 2023-11-02T17:59:02.033 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/?versions 2023-11-02T17:59:02.033 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1HrlfbauzLOAOEV+3gQXExy/6+8= 2023-11-02T17:59:02.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1HrlfbauzLOAOEV+3gQXExy/6+8=', 'Content-Length': '0'} 2023-11-02T17:59:02.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005958fce65d24705f-006543e366-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.039 DEBUG:boto:b'ibtljn-13999falsedummyYfZfHEB2viPzAqRNKaaAKV7gnCKC.W1true2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:59:02.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:59:02.041 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T17:59:02.041 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T17:59:02.042 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.042 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.043 DEBUG:boto:Method: GET 2023-11-02T17:59:02.043 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.044 DEBUG:boto:Data: 2023-11-02T17:59:02.044 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.045 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.045 DEBUG:boto:Params: {} 2023-11-02T17:59:02.046 DEBUG:boto:Token: None 2023-11-02T17:59:02.046 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.047 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:58e+cUSGdnf6+Voz4pvmr9xZM10= 2023-11-02T17:59:02.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:58e+cUSGdnf6+Voz4pvmr9xZM10=', 'Content-Length': '0'} 2023-11-02T17:59:02.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b16260371daeb809-006543e366-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.051 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T17:59:02.051 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T17:59:02.052 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.052 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.053 DEBUG:boto:Method: GET 2023-11-02T17:59:02.053 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.053 DEBUG:boto:Data: 2023-11-02T17:59:02.054 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.054 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.055 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.055 DEBUG:boto:Params: {} 2023-11-02T17:59:02.056 DEBUG:boto:Token: None 2023-11-02T17:59:02.056 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.057 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:58e+cUSGdnf6+Voz4pvmr9xZM10= 2023-11-02T17:59:02.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:58e+cUSGdnf6+Voz4pvmr9xZM10=', 'Content-Length': '0'} 2023-11-02T17:59:02.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012dd64bfbe8db480-006543e366-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.061 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T17:59:02.061 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T17:59:02.062 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.062 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.063 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.063 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.064 DEBUG:boto:Data: 2023-11-02T17:59:02.064 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.064 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.065 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.065 DEBUG:boto:Params: {} 2023-11-02T17:59:02.066 DEBUG:boto:Token: None 2023-11-02T17:59:02.066 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.067 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jdp77K0qGpDjv5VSLqVRu1P4uIg= 2023-11-02T17:59:02.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdp77K0qGpDjv5VSLqVRu1P4uIg=', 'Content-Length': '0'} 2023-11-02T17:59:02.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf264c9e5a4a1293-006543e366-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.071 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T17:59:02.071 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T17:59:02.072 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.072 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.073 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.073 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.073 DEBUG:boto:Data: 2023-11-02T17:59:02.074 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.074 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.075 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.075 DEBUG:boto:Params: {} 2023-11-02T17:59:02.076 DEBUG:boto:Token: None 2023-11-02T17:59:02.076 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.077 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jdp77K0qGpDjv5VSLqVRu1P4uIg= 2023-11-02T17:59:02.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdp77K0qGpDjv5VSLqVRu1P4uIg=', 'Content-Length': '0'} 2023-11-02T17:59:02.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c7fca79698cdd2a0-006543e366-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:59:02.081 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T17:59:02.082 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T17:59:02.082 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.083 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.083 DEBUG:boto:Method: GET 2023-11-02T17:59:02.084 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.084 DEBUG:boto:Data: 2023-11-02T17:59:02.084 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.085 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.085 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.086 DEBUG:boto:Params: {} 2023-11-02T17:59:02.086 DEBUG:boto:Token: None 2023-11-02T17:59:02.087 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.087 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:58e+cUSGdnf6+Voz4pvmr9xZM10= 2023-11-02T17:59:02.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:58e+cUSGdnf6+Voz4pvmr9xZM10=', 'Content-Length': '0'} 2023-11-02T17:59:02.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a3e64281b49a333d-006543e366-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.091 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T17:59:02.092 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T17:59:02.092 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.093 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.093 DEBUG:boto:Method: GET 2023-11-02T17:59:02.093 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.094 DEBUG:boto:Data: 2023-11-02T17:59:02.094 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.095 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.095 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.096 DEBUG:boto:Params: {} 2023-11-02T17:59:02.096 DEBUG:boto:Token: None 2023-11-02T17:59:02.097 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.097 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:58e+cUSGdnf6+Voz4pvmr9xZM10= 2023-11-02T17:59:02.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:58e+cUSGdnf6+Voz4pvmr9xZM10=', 'Content-Length': '0'} 2023-11-02T17:59:02.101 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000878ae8e3c35612d5-006543e366-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.101 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.102 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.102 DEBUG:boto:path=/ibtljn-13/?prefix=dummy 2023-11-02T17:59:02.103 DEBUG:boto:auth_path=/ibtljn-13/?prefix=dummy 2023-11-02T17:59:02.103 DEBUG:boto:Method: GET 2023-11-02T17:59:02.103 DEBUG:boto:Path: /ibtljn-13/?prefix=dummy 2023-11-02T17:59:02.104 DEBUG:boto:Data: 2023-11-02T17:59:02.104 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.105 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.105 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.106 DEBUG:boto:Params: {} 2023-11-02T17:59:02.106 DEBUG:boto:Token: None 2023-11-02T17:59:02.107 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.107 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wW2o8OpMCuP481TRDGuRpayYKQ0= 2023-11-02T17:59:02.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wW2o8OpMCuP481TRDGuRpayYKQ0=', 'Content-Length': '0'} 2023-11-02T17:59:02.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053ca620ac8ad54b9-006543e366-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.154 DEBUG:boto:b'ibtljn-13dummy1000falsedummy2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:59:02.155 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.156 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.156 DEBUG:boto:path=/ibtljn-13/?prefix=dummy 2023-11-02T17:59:02.156 DEBUG:boto:auth_path=/ibtljn-13/?prefix=dummy 2023-11-02T17:59:02.157 DEBUG:boto:Method: GET 2023-11-02T17:59:02.157 DEBUG:boto:Path: /ibtljn-13/?prefix=dummy 2023-11-02T17:59:02.158 DEBUG:boto:Data: 2023-11-02T17:59:02.158 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.159 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.159 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.160 DEBUG:boto:Params: {} 2023-11-02T17:59:02.161 DEBUG:boto:Token: None 2023-11-02T17:59:02.161 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.162 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wW2o8OpMCuP481TRDGuRpayYKQ0= 2023-11-02T17:59:02.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wW2o8OpMCuP481TRDGuRpayYKQ0=', 'Content-Length': '0'} 2023-11-02T17:59:02.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f98a1d71cf22bfb-006543e366-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.168 DEBUG:boto:b'ibtljn-13dummy1000falsedummy2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:59:02.168 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T17:59:02.169 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T17:59:02.169 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.170 DEBUG:boto:Path: /ibtljn-13/dummy 2023-11-02T17:59:02.170 DEBUG:boto:Data: 2023-11-02T17:59:02.171 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.171 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.172 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.172 DEBUG:boto:Params: {} 2023-11-02T17:59:02.173 DEBUG:boto:Token: None 2023-11-02T17:59:02.173 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/dummy 2023-11-02T17:59:02.173 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cJT1Pxzw9+z/eecuvQur4hLeQyw= 2023-11-02T17:59:02.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cJT1Pxzw9+z/eecuvQur4hLeQyw=', 'Content-Length': '0'} 2023-11-02T17:59:02.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb969ead6cc007f8-006543e366-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.179 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T17:59:02.179 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T17:59:02.179 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.180 DEBUG:boto:Path: /ibtljn-13/dummy 2023-11-02T17:59:02.180 DEBUG:boto:Data: 2023-11-02T17:59:02.181 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.181 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.182 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.182 DEBUG:boto:Params: {} 2023-11-02T17:59:02.182 DEBUG:boto:Token: None 2023-11-02T17:59:02.183 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/dummy 2023-11-02T17:59:02.183 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cJT1Pxzw9+z/eecuvQur4hLeQyw= 2023-11-02T17:59:02.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cJT1Pxzw9+z/eecuvQur4hLeQyw=', 'Content-Length': '0'} 2023-11-02T17:59:02.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d3eca034d864d830-006543e366-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T17:59:02.189 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T17:59:02.189 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T17:59:02.190 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.190 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.191 DEBUG:boto:Method: GET 2023-11-02T17:59:02.191 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.192 DEBUG:boto:Data: 2023-11-02T17:59:02.192 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.193 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.193 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.194 DEBUG:boto:Params: {} 2023-11-02T17:59:02.194 DEBUG:boto:Token: None 2023-11-02T17:59:02.195 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.195 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:58e+cUSGdnf6+Voz4pvmr9xZM10= 2023-11-02T17:59:02.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:58e+cUSGdnf6+Voz4pvmr9xZM10=', 'Content-Length': '0'} 2023-11-02T17:59:02.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000013621380ca54fbd2-006543e366-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.200 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T17:59:02.200 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T17:59:02.201 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.201 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.202 DEBUG:boto:Method: GET 2023-11-02T17:59:02.202 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.203 DEBUG:boto:Data: 2023-11-02T17:59:02.203 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.204 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.204 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.205 DEBUG:boto:Params: {} 2023-11-02T17:59:02.205 DEBUG:boto:Token: None 2023-11-02T17:59:02.206 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T17:59:02.206 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:58e+cUSGdnf6+Voz4pvmr9xZM10= 2023-11-02T17:59:02.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:58e+cUSGdnf6+Voz4pvmr9xZM10=', 'Content-Length': '0'} 2023-11-02T17:59:02.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef1e5afb275884f6-006543e366-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.211 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.211 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.212 DEBUG:boto:path=/ibtljn-13/?prefix=dummy 2023-11-02T17:59:02.212 DEBUG:boto:auth_path=/ibtljn-13/?prefix=dummy 2023-11-02T17:59:02.213 DEBUG:boto:Method: GET 2023-11-02T17:59:02.213 DEBUG:boto:Path: /ibtljn-13/?prefix=dummy 2023-11-02T17:59:02.214 DEBUG:boto:Data: 2023-11-02T17:59:02.214 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.215 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.215 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.216 DEBUG:boto:Params: {} 2023-11-02T17:59:02.216 DEBUG:boto:Token: None 2023-11-02T17:59:02.217 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.218 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wW2o8OpMCuP481TRDGuRpayYKQ0= 2023-11-02T17:59:02.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wW2o8OpMCuP481TRDGuRpayYKQ0=', 'Content-Length': '0'} 2023-11-02T17:59:02.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016153587bc86699d-006543e366-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.224 DEBUG:boto:b'ibtljn-13dummy1000falsedummy2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:59:02.224 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.225 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.225 DEBUG:boto:path=/ibtljn-13/?prefix=dummy 2023-11-02T17:59:02.225 DEBUG:boto:auth_path=/ibtljn-13/?prefix=dummy 2023-11-02T17:59:02.226 DEBUG:boto:Method: GET 2023-11-02T17:59:02.226 DEBUG:boto:Path: /ibtljn-13/?prefix=dummy 2023-11-02T17:59:02.227 DEBUG:boto:Data: 2023-11-02T17:59:02.227 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.228 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.228 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.229 DEBUG:boto:Params: {} 2023-11-02T17:59:02.229 DEBUG:boto:Token: None 2023-11-02T17:59:02.230 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.230 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wW2o8OpMCuP481TRDGuRpayYKQ0= 2023-11-02T17:59:02.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wW2o8OpMCuP481TRDGuRpayYKQ0=', 'Content-Length': '0'} 2023-11-02T17:59:02.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000382ec3c9abb5b351-006543e366-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.236 DEBUG:boto:b'ibtljn-13dummy1000falsedummy2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T17:59:02.237 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-13 zones={a2, a1} 2023-11-02T17:59:02.237 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.238 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.238 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.239 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.239 DEBUG:boto:Data: 2023-11-02T17:59:02.240 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.240 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.241 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.241 DEBUG:boto:Params: {} 2023-11-02T17:59:02.242 DEBUG:boto:Token: None 2023-11-02T17:59:02.242 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5ff5f4a6998069e-006543e366-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.248 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.248 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.249 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.249 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.249 DEBUG:boto:Data: 2023-11-02T17:59:02.250 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.250 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.251 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.251 DEBUG:boto:Params: {} 2023-11-02T17:59:02.252 DEBUG:boto:Token: None 2023-11-02T17:59:02.252 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.253 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'kcaDEYIG28LX9ix1nN7FlHWFX8nqevl'), ('x-amz-request-id', 'tx000006465cff49ed9e966-006543e366-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.303 DEBUG:rgw_multi.tests:zone=a1 version=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T17:59:02.304 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.304 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.304 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.305 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.305 DEBUG:boto:Data: 2023-11-02T17:59:02.306 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.306 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.307 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.307 DEBUG:boto:Params: {} 2023-11-02T17:59:02.308 DEBUG:boto:Token: None 2023-11-02T17:59:02.308 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.309 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087bbda5cdd0ccb01-006543e366-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.314 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.314 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.315 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.315 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.316 DEBUG:boto:Data: 2023-11-02T17:59:02.316 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.317 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.317 DEBUG:boto:Params: {} 2023-11-02T17:59:02.318 DEBUG:boto:Token: None 2023-11-02T17:59:02.318 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks'), ('x-amz-request-id', 'tx00000aa2fe949363c85db-006543e366-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.363 DEBUG:rgw_multi.tests:zone=a2 version=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T17:59:02.363 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.364 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.364 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.365 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.365 DEBUG:boto:Data: 2023-11-02T17:59:02.365 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.366 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.367 DEBUG:boto:Params: {} 2023-11-02T17:59:02.367 DEBUG:boto:Token: None 2023-11-02T17:59:02.367 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.368 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000208b584bb676cd00-006543e366-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.373 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.374 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.374 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.375 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.375 DEBUG:boto:Data: 2023-11-02T17:59:02.375 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.376 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.376 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.377 DEBUG:boto:Params: {} 2023-11-02T17:59:02.377 DEBUG:boto:Token: None 2023-11-02T17:59:02.378 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.378 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UiuCrzXqZHzsdVsjA0GimR8Ij68xCu-'), ('x-amz-request-id', 'tx0000069a76e8037f3259d-006543e366-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.400 DEBUG:rgw_multi.tests:zone=a1 version=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T17:59:02.401 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.401 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.401 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.402 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.402 DEBUG:boto:Data: 2023-11-02T17:59:02.403 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.403 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.404 DEBUG:boto:Params: {} 2023-11-02T17:59:02.404 DEBUG:boto:Token: None 2023-11-02T17:59:02.405 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec7d378d6fa33c1d-006543e366-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.410 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.411 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.411 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.411 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.412 DEBUG:boto:Data: 2023-11-02T17:59:02.412 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.413 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.413 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.413 DEBUG:boto:Params: {} 2023-11-02T17:59:02.414 DEBUG:boto:Token: None 2023-11-02T17:59:02.414 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.415 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Zv-i3fTD6jXNRrAyl830BGdra2ND4x-'), ('x-amz-request-id', 'tx000005fd3896a7278a12c-006543e366-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.435 DEBUG:rgw_multi.tests:zone=a2 version=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T17:59:02.435 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.435 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.436 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.436 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.437 DEBUG:boto:Data: 2023-11-02T17:59:02.437 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.438 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.438 DEBUG:boto:Params: {} 2023-11-02T17:59:02.439 DEBUG:boto:Token: None 2023-11-02T17:59:02.439 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.440 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0593812217f33df-006543e366-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.444 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.445 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.445 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.445 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.446 DEBUG:boto:Data: 2023-11-02T17:59:02.446 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.447 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.447 DEBUG:boto:Params: {} 2023-11-02T17:59:02.448 DEBUG:boto:Token: None 2023-11-02T17:59:02.448 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.449 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.468 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'AorrUYLUrfeGImHVayGt7jrmhCqlMLI'), ('x-amz-request-id', 'tx00000f414fa874a478e2b-006543e366-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.469 DEBUG:rgw_multi.tests:zone=a1 version=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T17:59:02.469 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.470 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.470 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.471 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.471 DEBUG:boto:Data: 2023-11-02T17:59:02.471 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.472 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.472 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.472 DEBUG:boto:Params: {} 2023-11-02T17:59:02.473 DEBUG:boto:Token: None 2023-11-02T17:59:02.473 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.474 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a33c959b3014446e-006543e366-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.479 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.479 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.480 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.480 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.481 DEBUG:boto:Data: 2023-11-02T17:59:02.481 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.481 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.482 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.482 DEBUG:boto:Params: {} 2023-11-02T17:59:02.483 DEBUG:boto:Token: None 2023-11-02T17:59:02.483 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.484 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK'), ('x-amz-request-id', 'tx000006465b5ba9166a6a2-006543e366-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.504 DEBUG:rgw_multi.tests:zone=a2 version=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T17:59:02.505 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.505 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.506 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.506 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.506 DEBUG:boto:Data: 2023-11-02T17:59:02.507 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.507 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.507 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.508 DEBUG:boto:Params: {} 2023-11-02T17:59:02.508 DEBUG:boto:Token: None 2023-11-02T17:59:02.509 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.509 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074bddacee5daecd3-006543e366-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.516 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.517 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.517 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.517 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.518 DEBUG:boto:Data: 2023-11-02T17:59:02.518 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.518 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.519 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.519 DEBUG:boto:Params: {} 2023-11-02T17:59:02.520 DEBUG:boto:Token: None 2023-11-02T17:59:02.520 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.566 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE'), ('x-amz-request-id', 'tx00000a77a35f372f1ebdd-006543e366-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.567 DEBUG:rgw_multi.tests:zone=a1 version=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T17:59:02.567 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.568 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.568 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.568 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.569 DEBUG:boto:Data: 2023-11-02T17:59:02.569 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.570 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.570 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.571 DEBUG:boto:Params: {} 2023-11-02T17:59:02.571 DEBUG:boto:Token: None 2023-11-02T17:59:02.571 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.572 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035f2877c33709a1f-006543e366-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.577 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.577 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.578 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.578 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.579 DEBUG:boto:Data: 2023-11-02T17:59:02.579 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.580 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.580 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.580 DEBUG:boto:Params: {} 2023-11-02T17:59:02.581 DEBUG:boto:Token: None 2023-11-02T17:59:02.581 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.582 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr'), ('x-amz-request-id', 'tx0000060c11d7f49ec2f32-006543e366-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.604 DEBUG:rgw_multi.tests:zone=a2 version=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T17:59:02.604 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.605 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.605 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.605 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.606 DEBUG:boto:Data: 2023-11-02T17:59:02.606 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.607 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.607 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.607 DEBUG:boto:Params: {} 2023-11-02T17:59:02.608 DEBUG:boto:Token: None 2023-11-02T17:59:02.608 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.609 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ffd90872041908b-006543e366-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.613 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.614 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.614 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.615 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.615 DEBUG:boto:Data: 2023-11-02T17:59:02.615 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.616 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.616 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.617 DEBUG:boto:Params: {} 2023-11-02T17:59:02.617 DEBUG:boto:Token: None 2023-11-02T17:59:02.617 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.618 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4'), ('x-amz-request-id', 'tx00000c07762e3ba20562b-006543e366-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.638 DEBUG:rgw_multi.tests:zone=a1 version=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T17:59:02.639 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.639 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.639 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.640 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.640 DEBUG:boto:Data: 2023-11-02T17:59:02.641 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.641 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.641 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.642 DEBUG:boto:Params: {} 2023-11-02T17:59:02.642 DEBUG:boto:Token: None 2023-11-02T17:59:02.643 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.643 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e4b1185ebac6c85-006543e366-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.649 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.649 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.649 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.650 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.650 DEBUG:boto:Data: 2023-11-02T17:59:02.651 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.652 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.652 DEBUG:boto:Params: {} 2023-11-02T17:59:02.652 DEBUG:boto:Token: None 2023-11-02T17:59:02.653 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK'), ('x-amz-request-id', 'tx0000029ba52aa6829d2e2-006543e366-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.699 DEBUG:rgw_multi.tests:zone=a2 version=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T17:59:02.699 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.700 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.700 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.701 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.701 DEBUG:boto:Data: 2023-11-02T17:59:02.702 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.702 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.702 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.703 DEBUG:boto:Params: {} 2023-11-02T17:59:02.703 DEBUG:boto:Token: None 2023-11-02T17:59:02.704 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.704 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090f028d1ca51a2ea-006543e366-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.709 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.710 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.710 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.711 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.711 DEBUG:boto:Data: 2023-11-02T17:59:02.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.712 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.712 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.713 DEBUG:boto:Params: {} 2023-11-02T17:59:02.713 DEBUG:boto:Token: None 2023-11-02T17:59:02.714 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.714 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx'), ('x-amz-request-id', 'tx000002354b9fc91aaa006-006543e366-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.736 DEBUG:rgw_multi.tests:zone=a1 version=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T17:59:02.736 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.736 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.737 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.737 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.738 DEBUG:boto:Data: 2023-11-02T17:59:02.738 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.739 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.739 DEBUG:boto:Params: {} 2023-11-02T17:59:02.740 DEBUG:boto:Token: None 2023-11-02T17:59:02.740 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.741 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fd416f49b101f3f-006543e366-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.745 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.745 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.746 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.746 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.747 DEBUG:boto:Data: 2023-11-02T17:59:02.747 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.747 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.748 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.748 DEBUG:boto:Params: {} 2023-11-02T17:59:02.749 DEBUG:boto:Token: None 2023-11-02T17:59:02.749 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.750 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT'), ('x-amz-request-id', 'tx000007baaec2db6459664-006543e366-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.769 DEBUG:rgw_multi.tests:zone=a2 version=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T17:59:02.770 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.770 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.771 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.771 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.771 DEBUG:boto:Data: 2023-11-02T17:59:02.772 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.772 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.773 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.773 DEBUG:boto:Params: {} 2023-11-02T17:59:02.773 DEBUG:boto:Token: None 2023-11-02T17:59:02.774 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.774 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096ed177f73916fb5-006543e366-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.779 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.779 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.780 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.780 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.780 DEBUG:boto:Data: 2023-11-02T17:59:02.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.781 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.782 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.782 DEBUG:boto:Params: {} 2023-11-02T17:59:02.782 DEBUG:boto:Token: None 2023-11-02T17:59:02.783 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.783 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qyKs0COjEwP9FUldxy3dhJMgbtCIdfw'), ('x-amz-request-id', 'tx000004a0b595435ff20d3-006543e366-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.827 DEBUG:rgw_multi.tests:zone=a1 version=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T17:59:02.827 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.828 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.828 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.829 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.829 DEBUG:boto:Data: 2023-11-02T17:59:02.829 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.830 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.830 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.831 DEBUG:boto:Params: {} 2023-11-02T17:59:02.831 DEBUG:boto:Token: None 2023-11-02T17:59:02.832 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.832 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b18b1a7700c34eb3-006543e366-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.837 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.838 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.838 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.839 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.839 DEBUG:boto:Data: 2023-11-02T17:59:02.840 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.840 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.840 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.841 DEBUG:boto:Params: {} 2023-11-02T17:59:02.841 DEBUG:boto:Token: None 2023-11-02T17:59:02.841 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.842 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bHdUIMsrwekD7ydoBEbIfYoJgPScbWU'), ('x-amz-request-id', 'tx00000c4b1690324083e1c-006543e366-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.863 DEBUG:rgw_multi.tests:zone=a2 version=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T17:59:02.864 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.864 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.864 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.865 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.865 DEBUG:boto:Data: 2023-11-02T17:59:02.865 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.866 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.866 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.866 DEBUG:boto:Params: {} 2023-11-02T17:59:02.867 DEBUG:boto:Token: None 2023-11-02T17:59:02.867 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.868 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.871 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003112eb425ddbbe5b-006543e366-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.872 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.873 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.873 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.873 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.874 DEBUG:boto:Data: 2023-11-02T17:59:02.874 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.875 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.875 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.875 DEBUG:boto:Params: {} 2023-11-02T17:59:02.876 DEBUG:boto:Token: None 2023-11-02T17:59:02.876 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'vH09Eyl.K.hzks5sEqA60vXpTkAW46q'), ('x-amz-request-id', 'tx00000e8f95a987173fab0-006543e366-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.898 DEBUG:rgw_multi.tests:zone=a1 version=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T17:59:02.899 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.899 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.899 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.900 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.900 DEBUG:boto:Data: 2023-11-02T17:59:02.901 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.901 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.901 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.902 DEBUG:boto:Params: {} 2023-11-02T17:59:02.902 DEBUG:boto:Token: None 2023-11-02T17:59:02.903 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.903 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.907 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4f77573d0a5cae1-006543e366-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.908 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.908 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.908 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.909 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.909 DEBUG:boto:Data: 2023-11-02T17:59:02.910 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:02.911 DEBUG:boto:Port: 8001 2023-11-02T17:59:02.911 DEBUG:boto:Params: {} 2023-11-02T17:59:02.911 DEBUG:boto:Token: None 2023-11-02T17:59:02.912 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.912 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q-'), ('x-amz-request-id', 'tx000003fde216ea1523be5-006543e366-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.955 DEBUG:rgw_multi.tests:zone=a2 version=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T17:59:02.955 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.956 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.956 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.956 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.957 DEBUG:boto:Data: 2023-11-02T17:59:02.957 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.957 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.958 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.958 DEBUG:boto:Params: {} 2023-11-02T17:59:02.959 DEBUG:boto:Token: None 2023-11-02T17:59:02.959 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/ 2023-11-02T17:59:02.960 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk= 2023-11-02T17:59:02.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zcQMDmr4efUDF0iONCJ3JZ/I/Dk=', 'Content-Length': '0'} 2023-11-02T17:59:02.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004077db0577ba272a-006543e366-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.965 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:02.965 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:02.966 DEBUG:boto:Method: PUT 2023-11-02T17:59:02.966 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:02.970 DEBUG:boto:Data: 2023-11-02T17:59:02.970 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:02.971 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:02.971 DEBUG:boto:Port: 8000 2023-11-02T17:59:02.971 DEBUG:boto:Params: {} 2023-11-02T17:59:02.972 DEBUG:boto:Token: None 2023-11-02T17:59:02.972 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:02 GMT /ibtljn-13/obj 2023-11-02T17:59:02.973 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw= 2023-11-02T17:59:02.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cWiUB7SmnXO6zF0TipfeHHyD/nw='} 2023-11-02T17:59:02.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW'), ('x-amz-request-id', 'tx00000d8200de56063e304-006543e366-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:02.996 DEBUG:rgw_multi.tests:zone=a1 version=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T17:59:02.997 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:02.997 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:02.998 DEBUG:boto:Method: HEAD 2023-11-02T17:59:02.998 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:02.998 DEBUG:boto:Data: 2023-11-02T17:59:02.999 DEBUG:boto:Headers: {} 2023-11-02T17:59:02.999 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:03.000 DEBUG:boto:Port: 8001 2023-11-02T17:59:03.000 DEBUG:boto:Params: {} 2023-11-02T17:59:03.001 DEBUG:boto:Token: None 2023-11-02T17:59:03.001 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:03 GMT /ibtljn-13/ 2023-11-02T17:59:03.001 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:582KYSCbclnWIT9CpSsL12zRSJU= 2023-11-02T17:59:03.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:582KYSCbclnWIT9CpSsL12zRSJU=', 'Content-Length': '0'} 2023-11-02T17:59:03.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aed69680d90e44d1-006543e366-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:03.007 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:03.008 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:03.008 DEBUG:boto:Method: PUT 2023-11-02T17:59:03.009 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:03.009 DEBUG:boto:Data: 2023-11-02T17:59:03.009 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:03.010 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:03.010 DEBUG:boto:Port: 8001 2023-11-02T17:59:03.011 DEBUG:boto:Params: {} 2023-11-02T17:59:03.011 DEBUG:boto:Token: None 2023-11-02T17:59:03.012 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:03 GMT /ibtljn-13/obj 2023-11-02T17:59:03.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4tk1Fne/L9NxKIDTNl9WahlPfxU= 2023-11-02T17:59:03.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4tk1Fne/L9NxKIDTNl9WahlPfxU='} 2023-11-02T17:59:03.033 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj'), ('x-amz-request-id', 'tx000000661c0675db69af7-006543e367-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:03.034 DEBUG:rgw_multi.tests:zone=a2 version=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T17:59:03.034 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:03.034 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:03.035 DEBUG:boto:Method: HEAD 2023-11-02T17:59:03.035 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:03.036 DEBUG:boto:Data: 2023-11-02T17:59:03.036 DEBUG:boto:Headers: {} 2023-11-02T17:59:03.036 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:03.037 DEBUG:boto:Port: 8000 2023-11-02T17:59:03.037 DEBUG:boto:Params: {} 2023-11-02T17:59:03.038 DEBUG:boto:Token: None 2023-11-02T17:59:03.038 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:03 GMT /ibtljn-13/ 2023-11-02T17:59:03.039 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:582KYSCbclnWIT9CpSsL12zRSJU= 2023-11-02T17:59:03.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:582KYSCbclnWIT9CpSsL12zRSJU=', 'Content-Length': '0'} 2023-11-02T17:59:03.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d2112a03817915d-006543e367-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:03.043 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:03.044 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:03.044 DEBUG:boto:Method: PUT 2023-11-02T17:59:03.045 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:03.045 DEBUG:boto:Data: 2023-11-02T17:59:03.045 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:03.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T17:59:03.046 DEBUG:boto:Port: 8000 2023-11-02T17:59:03.047 DEBUG:boto:Params: {} 2023-11-02T17:59:03.047 DEBUG:boto:Token: None 2023-11-02T17:59:03.048 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:03 GMT /ibtljn-13/obj 2023-11-02T17:59:03.048 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4tk1Fne/L9NxKIDTNl9WahlPfxU= 2023-11-02T17:59:03.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4tk1Fne/L9NxKIDTNl9WahlPfxU='} 2023-11-02T17:59:03.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'LH98R3N3MJCMJx3TJUp072XhY3JxYyI'), ('x-amz-request-id', 'tx00000631fa8c1da837a59-006543e367-1111-a1'), ('Date', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:03.095 DEBUG:rgw_multi.tests:zone=a1 version=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T17:59:03.096 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T17:59:03.096 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T17:59:03.097 DEBUG:boto:Method: HEAD 2023-11-02T17:59:03.097 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T17:59:03.098 DEBUG:boto:Data: 2023-11-02T17:59:03.098 DEBUG:boto:Headers: {} 2023-11-02T17:59:03.098 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:03.099 DEBUG:boto:Port: 8001 2023-11-02T17:59:03.099 DEBUG:boto:Params: {} 2023-11-02T17:59:03.100 DEBUG:boto:Token: None 2023-11-02T17:59:03.100 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 17:59:03 GMT /ibtljn-13/ 2023-11-02T17:59:03.101 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:582KYSCbclnWIT9CpSsL12zRSJU= 2023-11-02T17:59:03.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 17:59:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:582KYSCbclnWIT9CpSsL12zRSJU=', 'Content-Length': '0'} 2023-11-02T17:59:03.106 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000510317a800012841-006543e367-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:03.106 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T17:59:03.107 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T17:59:03.107 DEBUG:boto:Method: PUT 2023-11-02T17:59:03.108 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T17:59:03.108 DEBUG:boto:Data: 2023-11-02T17:59:03.108 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-11-02T17:59:03.109 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T17:59:03.109 DEBUG:boto:Port: 8001 2023-11-02T17:59:03.110 DEBUG:boto:Params: {} 2023-11-02T17:59:03.110 DEBUG:boto:Token: None 2023-11-02T17:59:03.111 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 02 Nov 2023 17:59:03 GMT /ibtljn-13/obj 2023-11-02T17:59:03.112 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4tk1Fne/L9NxKIDTNl9WahlPfxU= 2023-11-02T17:59:03.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 17:59:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4tk1Fne/L9NxKIDTNl9WahlPfxU='} 2023-11-02T17:59:03.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-amz-request-id', 'tx000001ed6faf4ec904b1a-006543e367-1115-a2'), ('Date', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T17:59:03.135 DEBUG:rgw_multi.tests:zone=a2 version=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T17:59:03.135 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-13 2023-11-02T17:59:03.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:59:03.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T17:59:03.137 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T17:59:03.192 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T17:59:03.192 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T17:59:03.262 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:03.253+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T17:59:03.263 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.7.11, , , , , ] 2023-11-02T17:59:03.263 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T17:59:08.265 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:08.257+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T17:59:08.265 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.7.11, , , , , ] 2023-11-02T17:59:08.266 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T17:59:13.268 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:13.257+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T17:59:13.268 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.7.11, , , , , ] 2023-11-02T17:59:13.269 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T17:59:18.271 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:18.260+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T17:59:18.271 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.7.11, , , , , ] 2023-11-02T17:59:18.271 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T17:59:23.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:23.264+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T17:59:23.274 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000001.4.6, , ] 2023-11-02T17:59:23.274 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T17:59:28.276 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:28.264+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T17:59:28.276 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000001.4.6, , ] 2023-11-02T17:59:28.277 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T17:59:33.281 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:33.268+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T17:59:33.282 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000001.4.6, , ] 2023-11-02T17:59:33.282 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T17:59:38.282 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:38.272+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T17:59:38.283 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000001.4.6, , ] 2023-11-02T17:59:38.283 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T17:59:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:43.272+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T17:59:43.285 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000015.32.6, , ] 2023-11-02T17:59:43.285 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T17:59:48.287 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:48.276+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T17:59:48.287 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000015.32.6, , ] 2023-11-02T17:59:48.288 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T17:59:53.290 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:53.280+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T17:59:53.290 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000015.32.6, , ] 2023-11-02T17:59:53.290 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T17:59:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T17:59:58.280+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T17:59:58.293 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000015.32.6, , ] 2023-11-02T17:59:58.293 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T18:00:03.295 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:03.284+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T18:00:03.296 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000015.32.6, , ] 2023-11-02T18:00:03.296 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T18:00:08.298 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:08.287+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T18:00:08.299 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000015.32.6, , ] 2023-11-02T18:00:08.299 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T18:00:13.301 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:13.291+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T18:00:13.301 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000015.32.6, , ] 2023-11-02T18:00:13.302 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T18:00:18.304 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:18.295+0000 7fb561117f40 1 waiting for incremental sync to catch up: 2023-11-02T18:00:18.304 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000015.32.6, , ] 2023-11-02T18:00:18.304 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T18:00:23.306 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:23.299+0000 7fb561117f40 1 bucket sync caught up with source: 2023-11-02T18:00:23.307 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000031.72.12, , ] 2023-11-02T18:00:23.307 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.41.12, , ] 2023-11-02T18:00:23.307 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:23.299+0000 7fb561117f40 0 bucket checkpoint complete 2023-11-02T18:00:23.336 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-13 2023-11-02T18:00:23.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:00:23.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:00:23.338 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:00:23.389 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:00:23.389 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:00:23.458 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:23.451+0000 7f9a6aebef40 1 waiting for incremental sync to catch up: 2023-11-02T18:00:23.459 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000027.65.6, , ] 2023-11-02T18:00:23.459 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000031.75.9, , ] 2023-11-02T18:00:28.461 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:28.451+0000 7f9a6aebef40 1 waiting for incremental sync to catch up: 2023-11-02T18:00:28.461 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000027.65.6, , ] 2023-11-02T18:00:28.462 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000031.75.9, , ] 2023-11-02T18:00:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:33.455+0000 7f9a6aebef40 1 waiting for incremental sync to catch up: 2023-11-02T18:00:33.464 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000027.65.6, , ] 2023-11-02T18:00:33.464 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000031.75.9, , ] 2023-11-02T18:00:38.466 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:38.459+0000 7f9a6aebef40 1 waiting for incremental sync to catch up: 2023-11-02T18:00:38.467 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000027.65.6, , ] 2023-11-02T18:00:38.467 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000031.75.9, , ] 2023-11-02T18:00:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:43.459+0000 7f9a6aebef40 1 bucket sync caught up with source: 2023-11-02T18:00:43.470 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000031.75.9, , ] 2023-11-02T18:00:43.470 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000031.75.9, , ] 2023-11-02T18:00:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:43.459+0000 7f9a6aebef40 0 bucket checkpoint complete 2023-11-02T18:00:43.500 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-13 zones={a2, a1} 2023-11-02T18:00:43.501 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:43.501 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:43.502 DEBUG:boto:Method: HEAD 2023-11-02T18:00:43.502 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T18:00:43.503 DEBUG:boto:Data: 2023-11-02T18:00:43.503 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.503 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.504 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.504 DEBUG:boto:Params: {} 2023-11-02T18:00:43.505 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:00:43.505 DEBUG:boto:Token: None 2023-11-02T18:00:43.506 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/ 2023-11-02T18:00:43.507 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kWIB0bnc7i60h1Mc4ozDAZVGuHs= 2023-11-02T18:00:43.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kWIB0bnc7i60h1Mc4ozDAZVGuHs=', 'Content-Length': '0'} 2023-11-02T18:00:43.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1adc5ad88665437-006543e3cb-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.513 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:43.513 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:43.514 DEBUG:boto:Method: HEAD 2023-11-02T18:00:43.514 DEBUG:boto:Path: /ibtljn-13/ 2023-11-02T18:00:43.515 DEBUG:boto:Data: 2023-11-02T18:00:43.515 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.516 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.516 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.516 DEBUG:boto:Params: {} 2023-11-02T18:00:43.517 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:00:43.518 DEBUG:boto:Token: None 2023-11-02T18:00:43.518 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/ 2023-11-02T18:00:43.519 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kWIB0bnc7i60h1Mc4ozDAZVGuHs= 2023-11-02T18:00:43.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kWIB0bnc7i60h1Mc4ozDAZVGuHs=', 'Content-Length': '0'} 2023-11-02T18:00:43.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006618510c74828414-006543e3cb-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.524 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:00:43.524 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:43.525 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:43.525 DEBUG:boto:path=/ibtljn-13/?versions&max-keys=999 2023-11-02T18:00:43.526 DEBUG:boto:auth_path=/ibtljn-13/?versions&max-keys=999 2023-11-02T18:00:43.526 DEBUG:boto:Method: GET 2023-11-02T18:00:43.526 DEBUG:boto:Path: /ibtljn-13/?versions&max-keys=999 2023-11-02T18:00:43.527 DEBUG:boto:Data: 2023-11-02T18:00:43.527 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.528 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.528 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.529 DEBUG:boto:Params: {} 2023-11-02T18:00:43.529 DEBUG:boto:Token: None 2023-11-02T18:00:43.530 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/?versions 2023-11-02T18:00:43.530 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SFEb7121DgxOLieJNNOv8wwscIU= 2023-11-02T18:00:43.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SFEb7121DgxOLieJNNOv8wwscIU=', 'Content-Length': '0'} 2023-11-02T18:00:43.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f594bfa09b1253ee-006543e3cb-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.535 DEBUG:boto:b'ibtljn-13999falsedummyYfZfHEB2viPzAqRNKaaAKV7gnCKC.W1true2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4true2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLH98R3N3MJCMJx3TJUp072XhY3JxYyIfalse2023-11-02T17:59:03.046Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLuAX4QvkX.Ol7bCELhrZp6YF1yw.dQjfalse2023-11-02T17:59:03.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYOZNVAi3SCKH-38AqA3kgfKr.a7X3TWfalse2023-11-02T17:59:02.971Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q-false2023-11-02T17:59:02.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvH09Eyl.K.hzks5sEqA60vXpTkAW46qfalse2023-11-02T17:59:02.874Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbHdUIMsrwekD7ydoBEbIfYoJgPScbWUfalse2023-11-02T17:59:02.839Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqyKs0COjEwP9FUldxy3dhJMgbtCIdfwfalse2023-11-02T17:59:02.781Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUwj.T0NKxKge-5JRpZgrYHyZpK8hFyTfalse2023-11-02T17:59:02.747Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjx6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qxfalse2023-11-02T17:59:02.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5FPt.NTakDYr3cN5mfK-9z0Ba.dVchKfalse2023-11-02T17:59:02.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4false2023-11-02T17:59:02.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjuGmB9Tk0n1pDhg1QtaLQuoFjN.7mdEfalse2023-11-02T17:59:02.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlHwkVsQ5SjaueCyARyAbGWVEKo0Y8srfalse2023-11-02T17:59:02.579Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAorrUYLUrfeGImHVayGt7jrmhCqlMLIfalse2023-11-02T17:59:02.446Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBqp.o7pDoCMqVNZ9auXuTjozZC9BlCKfalse2023-11-02T17:59:02.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUiuCrzXqZHzsdVsjA0GimR8Ij68xCu-false2023-11-02T17:59:02.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZv-i3fTD6jXNRrAyl830BGdra2ND4x-false2023-11-02T17:59:02.412Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ksfalse2023-11-02T17:59:02.317Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkcaDEYIG28LX9ix1nN7FlHWFX8nqevlfalse2023-11-02T17:59:02.251Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:43.537 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T18:00:43.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.545 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:00:43.546 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:43.546 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:43.546 DEBUG:boto:path=/ibtljn-13/?versions&max-keys=999 2023-11-02T18:00:43.547 DEBUG:boto:auth_path=/ibtljn-13/?versions&max-keys=999 2023-11-02T18:00:43.547 DEBUG:boto:Method: GET 2023-11-02T18:00:43.548 DEBUG:boto:Path: /ibtljn-13/?versions&max-keys=999 2023-11-02T18:00:43.548 DEBUG:boto:Data: 2023-11-02T18:00:43.548 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.549 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.549 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.550 DEBUG:boto:Params: {} 2023-11-02T18:00:43.550 DEBUG:boto:Token: None 2023-11-02T18:00:43.551 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/?versions 2023-11-02T18:00:43.551 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SFEb7121DgxOLieJNNOv8wwscIU= 2023-11-02T18:00:43.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SFEb7121DgxOLieJNNOv8wwscIU=', 'Content-Length': '0'} 2023-11-02T18:00:43.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d70450f55cd419a-006543e3cb-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.557 DEBUG:boto:b'ibtljn-13999falsedummyYfZfHEB2viPzAqRNKaaAKV7gnCKC.W1true2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4true2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLH98R3N3MJCMJx3TJUp072XhY3JxYyIfalse2023-11-02T17:59:03.046Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLuAX4QvkX.Ol7bCELhrZp6YF1yw.dQjfalse2023-11-02T17:59:03.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYOZNVAi3SCKH-38AqA3kgfKr.a7X3TWfalse2023-11-02T17:59:02.971Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q-false2023-11-02T17:59:02.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvH09Eyl.K.hzks5sEqA60vXpTkAW46qfalse2023-11-02T17:59:02.874Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbHdUIMsrwekD7ydoBEbIfYoJgPScbWUfalse2023-11-02T17:59:02.839Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqyKs0COjEwP9FUldxy3dhJMgbtCIdfwfalse2023-11-02T17:59:02.781Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUwj.T0NKxKge-5JRpZgrYHyZpK8hFyTfalse2023-11-02T17:59:02.747Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjx6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qxfalse2023-11-02T17:59:02.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5FPt.NTakDYr3cN5mfK-9z0Ba.dVchKfalse2023-11-02T17:59:02.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4false2023-11-02T17:59:02.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjuGmB9Tk0n1pDhg1QtaLQuoFjN.7mdEfalse2023-11-02T17:59:02.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlHwkVsQ5SjaueCyARyAbGWVEKo0Y8srfalse2023-11-02T17:59:02.579Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAorrUYLUrfeGImHVayGt7jrmhCqlMLIfalse2023-11-02T17:59:02.446Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBqp.o7pDoCMqVNZ9auXuTjozZC9BlCKfalse2023-11-02T17:59:02.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUiuCrzXqZHzsdVsjA0GimR8Ij68xCu-false2023-11-02T17:59:02.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZv-i3fTD6jXNRrAyl830BGdra2ND4x-false2023-11-02T17:59:02.412Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ksfalse2023-11-02T17:59:02.317Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkcaDEYIG28LX9ix1nN7FlHWFX8nqevlfalse2023-11-02T17:59:02.251Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:43.559 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T18:00:43.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:00:43.567 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:43.567 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:43.568 DEBUG:boto:path=/ibtljn-13/?versions&max-keys=999 2023-11-02T18:00:43.568 DEBUG:boto:auth_path=/ibtljn-13/?versions&max-keys=999 2023-11-02T18:00:43.569 DEBUG:boto:Method: GET 2023-11-02T18:00:43.569 DEBUG:boto:Path: /ibtljn-13/?versions&max-keys=999 2023-11-02T18:00:43.571 DEBUG:boto:Data: 2023-11-02T18:00:43.571 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.571 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.572 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.572 DEBUG:boto:Params: {} 2023-11-02T18:00:43.573 DEBUG:boto:Token: None 2023-11-02T18:00:43.573 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/?versions 2023-11-02T18:00:43.574 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SFEb7121DgxOLieJNNOv8wwscIU= 2023-11-02T18:00:43.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SFEb7121DgxOLieJNNOv8wwscIU=', 'Content-Length': '0'} 2023-11-02T18:00:43.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030198c2afa1d3a10-006543e3cb-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.581 DEBUG:boto:b'ibtljn-13999falsedummyYfZfHEB2viPzAqRNKaaAKV7gnCKC.W1true2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4true2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLH98R3N3MJCMJx3TJUp072XhY3JxYyIfalse2023-11-02T17:59:03.046Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLuAX4QvkX.Ol7bCELhrZp6YF1yw.dQjfalse2023-11-02T17:59:03.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYOZNVAi3SCKH-38AqA3kgfKr.a7X3TWfalse2023-11-02T17:59:02.971Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q-false2023-11-02T17:59:02.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvH09Eyl.K.hzks5sEqA60vXpTkAW46qfalse2023-11-02T17:59:02.874Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbHdUIMsrwekD7ydoBEbIfYoJgPScbWUfalse2023-11-02T17:59:02.839Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqyKs0COjEwP9FUldxy3dhJMgbtCIdfwfalse2023-11-02T17:59:02.781Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUwj.T0NKxKge-5JRpZgrYHyZpK8hFyTfalse2023-11-02T17:59:02.747Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjx6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qxfalse2023-11-02T17:59:02.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5FPt.NTakDYr3cN5mfK-9z0Ba.dVchKfalse2023-11-02T17:59:02.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4false2023-11-02T17:59:02.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjuGmB9Tk0n1pDhg1QtaLQuoFjN.7mdEfalse2023-11-02T17:59:02.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlHwkVsQ5SjaueCyARyAbGWVEKo0Y8srfalse2023-11-02T17:59:02.579Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAorrUYLUrfeGImHVayGt7jrmhCqlMLIfalse2023-11-02T17:59:02.446Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBqp.o7pDoCMqVNZ9auXuTjozZC9BlCKfalse2023-11-02T17:59:02.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUiuCrzXqZHzsdVsjA0GimR8Ij68xCu-false2023-11-02T17:59:02.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZv-i3fTD6jXNRrAyl830BGdra2ND4x-false2023-11-02T17:59:02.412Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ksfalse2023-11-02T17:59:02.317Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkcaDEYIG28LX9ix1nN7FlHWFX8nqevlfalse2023-11-02T17:59:02.251Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:43.583 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:43.583 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:43.584 DEBUG:boto:path=/ibtljn-13/?versions&max-keys=999 2023-11-02T18:00:43.584 DEBUG:boto:auth_path=/ibtljn-13/?versions&max-keys=999 2023-11-02T18:00:43.585 DEBUG:boto:Method: GET 2023-11-02T18:00:43.585 DEBUG:boto:Path: /ibtljn-13/?versions&max-keys=999 2023-11-02T18:00:43.585 DEBUG:boto:Data: 2023-11-02T18:00:43.586 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.586 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.587 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.587 DEBUG:boto:Params: {} 2023-11-02T18:00:43.587 DEBUG:boto:Token: None 2023-11-02T18:00:43.588 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/?versions 2023-11-02T18:00:43.588 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SFEb7121DgxOLieJNNOv8wwscIU= 2023-11-02T18:00:43.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SFEb7121DgxOLieJNNOv8wwscIU=', 'Content-Length': '0'} 2023-11-02T18:00:43.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d2ebac159136004-006543e3cb-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.595 DEBUG:boto:b'ibtljn-13999falsedummyYfZfHEB2viPzAqRNKaaAKV7gnCKC.W1true2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4true2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLH98R3N3MJCMJx3TJUp072XhY3JxYyIfalse2023-11-02T17:59:03.046Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLuAX4QvkX.Ol7bCELhrZp6YF1yw.dQjfalse2023-11-02T17:59:03.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYOZNVAi3SCKH-38AqA3kgfKr.a7X3TWfalse2023-11-02T17:59:02.971Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q-false2023-11-02T17:59:02.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvH09Eyl.K.hzks5sEqA60vXpTkAW46qfalse2023-11-02T17:59:02.874Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbHdUIMsrwekD7ydoBEbIfYoJgPScbWUfalse2023-11-02T17:59:02.839Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqyKs0COjEwP9FUldxy3dhJMgbtCIdfwfalse2023-11-02T17:59:02.781Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUwj.T0NKxKge-5JRpZgrYHyZpK8hFyTfalse2023-11-02T17:59:02.747Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjx6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qxfalse2023-11-02T17:59:02.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5FPt.NTakDYr3cN5mfK-9z0Ba.dVchKfalse2023-11-02T17:59:02.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4false2023-11-02T17:59:02.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjuGmB9Tk0n1pDhg1QtaLQuoFjN.7mdEfalse2023-11-02T17:59:02.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlHwkVsQ5SjaueCyARyAbGWVEKo0Y8srfalse2023-11-02T17:59:02.579Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAorrUYLUrfeGImHVayGt7jrmhCqlMLIfalse2023-11-02T17:59:02.446Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBqp.o7pDoCMqVNZ9auXuTjozZC9BlCKfalse2023-11-02T17:59:02.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUiuCrzXqZHzsdVsjA0GimR8Ij68xCu-false2023-11-02T17:59:02.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZv-i3fTD6jXNRrAyl830BGdra2ND4x-false2023-11-02T17:59:02.412Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ksfalse2023-11-02T17:59:02.317Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkcaDEYIG28LX9ix1nN7FlHWFX8nqevlfalse2023-11-02T17:59:02.251Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:43.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T18:00:43.597 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T18:00:43.597 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T18:00:43.598 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.598 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.599 DEBUG:boto:Method: GET 2023-11-02T18:00:43.599 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.600 DEBUG:boto:Data: 2023-11-02T18:00:43.600 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.601 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.601 DEBUG:boto:Params: {} 2023-11-02T18:00:43.602 DEBUG:boto:Token: None 2023-11-02T18:00:43.602 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.603 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P2MC4wFYVZGduxRxt0JfYfZjnCI= 2023-11-02T18:00:43.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P2MC4wFYVZGduxRxt0JfYfZjnCI=', 'Content-Length': '0'} 2023-11-02T18:00:43.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f91c69458c18d03f-006543e3cb-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.607 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T18:00:43.607 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T18:00:43.608 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.608 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.609 DEBUG:boto:Method: GET 2023-11-02T18:00:43.609 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.610 DEBUG:boto:Data: 2023-11-02T18:00:43.610 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.611 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.611 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.612 DEBUG:boto:Params: {} 2023-11-02T18:00:43.612 DEBUG:boto:Token: None 2023-11-02T18:00:43.613 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.613 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P2MC4wFYVZGduxRxt0JfYfZjnCI= 2023-11-02T18:00:43.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P2MC4wFYVZGduxRxt0JfYfZjnCI=', 'Content-Length': '0'} 2023-11-02T18:00:43.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000018f6dcdf0a6250b7-006543e3cb-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.617 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T18:00:43.617 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T18:00:43.618 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.618 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.619 DEBUG:boto:Method: HEAD 2023-11-02T18:00:43.619 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.620 DEBUG:boto:Data: 2023-11-02T18:00:43.620 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.620 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.621 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.621 DEBUG:boto:Params: {} 2023-11-02T18:00:43.622 DEBUG:boto:Token: None 2023-11-02T18:00:43.622 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.623 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L5M1XWHA0OSdQ6sJlRCKyGAnpf4= 2023-11-02T18:00:43.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L5M1XWHA0OSdQ6sJlRCKyGAnpf4=', 'Content-Length': '0'} 2023-11-02T18:00:43.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000539f69cb3f3913b2-006543e3cb-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.627 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T18:00:43.627 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T18:00:43.627 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.628 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.628 DEBUG:boto:Method: HEAD 2023-11-02T18:00:43.629 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.629 DEBUG:boto:Data: 2023-11-02T18:00:43.629 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.630 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.631 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.631 DEBUG:boto:Params: {} 2023-11-02T18:00:43.631 DEBUG:boto:Token: None 2023-11-02T18:00:43.632 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.632 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L5M1XWHA0OSdQ6sJlRCKyGAnpf4= 2023-11-02T18:00:43.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L5M1XWHA0OSdQ6sJlRCKyGAnpf4=', 'Content-Length': '0'} 2023-11-02T18:00:43.636 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c176fd3bd003e28-006543e3cb-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T18:00:43.637 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T18:00:43.637 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T18:00:43.637 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.638 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.638 DEBUG:boto:Method: GET 2023-11-02T18:00:43.639 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.639 DEBUG:boto:Data: 2023-11-02T18:00:43.640 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.640 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.640 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.641 DEBUG:boto:Params: {} 2023-11-02T18:00:43.641 DEBUG:boto:Token: None 2023-11-02T18:00:43.642 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.642 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P2MC4wFYVZGduxRxt0JfYfZjnCI= 2023-11-02T18:00:43.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P2MC4wFYVZGduxRxt0JfYfZjnCI=', 'Content-Length': '0'} 2023-11-02T18:00:43.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016ea0c30e3e4f251-006543e3cb-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.646 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T18:00:43.647 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T18:00:43.647 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.648 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.648 DEBUG:boto:Method: GET 2023-11-02T18:00:43.648 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.649 DEBUG:boto:Data: 2023-11-02T18:00:43.650 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.650 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.650 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.651 DEBUG:boto:Params: {} 2023-11-02T18:00:43.651 DEBUG:boto:Token: None 2023-11-02T18:00:43.652 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.652 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P2MC4wFYVZGduxRxt0JfYfZjnCI= 2023-11-02T18:00:43.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P2MC4wFYVZGduxRxt0JfYfZjnCI=', 'Content-Length': '0'} 2023-11-02T18:00:43.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004c5c6beaddf1c7da-006543e3cb-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.656 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:43.657 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:43.657 DEBUG:boto:path=/ibtljn-13/?prefix=dummy 2023-11-02T18:00:43.658 DEBUG:boto:auth_path=/ibtljn-13/?prefix=dummy 2023-11-02T18:00:43.658 DEBUG:boto:Method: GET 2023-11-02T18:00:43.659 DEBUG:boto:Path: /ibtljn-13/?prefix=dummy 2023-11-02T18:00:43.659 DEBUG:boto:Data: 2023-11-02T18:00:43.659 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.660 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.660 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.661 DEBUG:boto:Params: {} 2023-11-02T18:00:43.661 DEBUG:boto:Token: None 2023-11-02T18:00:43.662 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/ 2023-11-02T18:00:43.662 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZUms1EIZswYH+lG8MUeOtrVdE40= 2023-11-02T18:00:43.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZUms1EIZswYH+lG8MUeOtrVdE40=', 'Content-Length': '0'} 2023-11-02T18:00:43.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001adc1580e29f977d-006543e3cb-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.710 DEBUG:boto:b'ibtljn-13dummy1000falsedummy2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:43.711 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:43.712 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:43.712 DEBUG:boto:path=/ibtljn-13/?prefix=dummy 2023-11-02T18:00:43.713 DEBUG:boto:auth_path=/ibtljn-13/?prefix=dummy 2023-11-02T18:00:43.713 DEBUG:boto:Method: GET 2023-11-02T18:00:43.714 DEBUG:boto:Path: /ibtljn-13/?prefix=dummy 2023-11-02T18:00:43.714 DEBUG:boto:Data: 2023-11-02T18:00:43.715 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.715 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.716 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.716 DEBUG:boto:Params: {} 2023-11-02T18:00:43.717 DEBUG:boto:Token: None 2023-11-02T18:00:43.717 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/ 2023-11-02T18:00:43.718 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZUms1EIZswYH+lG8MUeOtrVdE40= 2023-11-02T18:00:43.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZUms1EIZswYH+lG8MUeOtrVdE40=', 'Content-Length': '0'} 2023-11-02T18:00:43.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044adf5b4df05b0b7-006543e3cb-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.723 DEBUG:boto:b'ibtljn-13dummy1000falsedummy2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:43.724 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T18:00:43.724 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T18:00:43.725 DEBUG:boto:Method: HEAD 2023-11-02T18:00:43.725 DEBUG:boto:Path: /ibtljn-13/dummy 2023-11-02T18:00:43.726 DEBUG:boto:Data: 2023-11-02T18:00:43.726 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.727 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.727 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.728 DEBUG:boto:Params: {} 2023-11-02T18:00:43.728 DEBUG:boto:Token: None 2023-11-02T18:00:43.729 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/dummy 2023-11-02T18:00:43.729 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CmB0rRREIaKOD87mJQwwfOJOt+M= 2023-11-02T18:00:43.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CmB0rRREIaKOD87mJQwwfOJOt+M=', 'Content-Length': '0'} 2023-11-02T18:00:43.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000004c88bd801883afe-006543e3cb-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.734 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T18:00:43.734 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T18:00:43.735 DEBUG:boto:Method: HEAD 2023-11-02T18:00:43.735 DEBUG:boto:Path: /ibtljn-13/dummy 2023-11-02T18:00:43.736 DEBUG:boto:Data: 2023-11-02T18:00:43.736 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.736 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.737 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.737 DEBUG:boto:Params: {} 2023-11-02T18:00:43.738 DEBUG:boto:Token: None 2023-11-02T18:00:43.738 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/dummy 2023-11-02T18:00:43.739 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CmB0rRREIaKOD87mJQwwfOJOt+M= 2023-11-02T18:00:43.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CmB0rRREIaKOD87mJQwwfOJOt+M=', 'Content-Length': '0'} 2023-11-02T18:00:43.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a6b3de76bad7a9ef-006543e3cb-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T18:00:43.744 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T18:00:43.744 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T18:00:43.745 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.745 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.746 DEBUG:boto:Method: GET 2023-11-02T18:00:43.746 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.747 DEBUG:boto:Data: 2023-11-02T18:00:43.747 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.748 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.748 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.748 DEBUG:boto:Params: {} 2023-11-02T18:00:43.749 DEBUG:boto:Token: None 2023-11-02T18:00:43.749 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.750 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P2MC4wFYVZGduxRxt0JfYfZjnCI= 2023-11-02T18:00:43.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P2MC4wFYVZGduxRxt0JfYfZjnCI=', 'Content-Length': '0'} 2023-11-02T18:00:43.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d686171620965ae4-006543e3cb-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.754 DEBUG:boto:path=/ibtljn-13/dummy 2023-11-02T18:00:43.754 DEBUG:boto:auth_path=/ibtljn-13/dummy 2023-11-02T18:00:43.755 DEBUG:boto:path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.755 DEBUG:boto:auth_path=/ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.756 DEBUG:boto:Method: GET 2023-11-02T18:00:43.756 DEBUG:boto:Path: /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.757 DEBUG:boto:Data: 2023-11-02T18:00:43.757 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.758 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.758 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.758 DEBUG:boto:Params: {} 2023-11-02T18:00:43.759 DEBUG:boto:Token: None 2023-11-02T18:00:43.759 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/dummy?versionId=YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1 2023-11-02T18:00:43.760 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P2MC4wFYVZGduxRxt0JfYfZjnCI= 2023-11-02T18:00:43.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P2MC4wFYVZGduxRxt0JfYfZjnCI=', 'Content-Length': '0'} 2023-11-02T18:00:43.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:58:31 GMT'), ('x-amz-version-id', 'YfZfHEB2viPzAqRNKaaAKV7gnCKC.W1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092a781ad9bc9a69f-006543e3cb-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.766 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:43.766 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:43.767 DEBUG:boto:path=/ibtljn-13/?prefix=dummy 2023-11-02T18:00:43.767 DEBUG:boto:auth_path=/ibtljn-13/?prefix=dummy 2023-11-02T18:00:43.767 DEBUG:boto:Method: GET 2023-11-02T18:00:43.768 DEBUG:boto:Path: /ibtljn-13/?prefix=dummy 2023-11-02T18:00:43.768 DEBUG:boto:Data: 2023-11-02T18:00:43.769 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.769 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.770 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.770 DEBUG:boto:Params: {} 2023-11-02T18:00:43.771 DEBUG:boto:Token: None 2023-11-02T18:00:43.771 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/ 2023-11-02T18:00:43.772 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZUms1EIZswYH+lG8MUeOtrVdE40= 2023-11-02T18:00:43.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZUms1EIZswYH+lG8MUeOtrVdE40=', 'Content-Length': '0'} 2023-11-02T18:00:43.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024f1bd9fe1ea17cb-006543e3cb-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.778 DEBUG:boto:b'ibtljn-13dummy1000falsedummy2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:43.778 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:43.779 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:43.779 DEBUG:boto:path=/ibtljn-13/?prefix=dummy 2023-11-02T18:00:43.780 DEBUG:boto:auth_path=/ibtljn-13/?prefix=dummy 2023-11-02T18:00:43.780 DEBUG:boto:Method: GET 2023-11-02T18:00:43.781 DEBUG:boto:Path: /ibtljn-13/?prefix=dummy 2023-11-02T18:00:43.781 DEBUG:boto:Data: 2023-11-02T18:00:43.782 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.782 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.782 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.783 DEBUG:boto:Params: {} 2023-11-02T18:00:43.783 DEBUG:boto:Token: None 2023-11-02T18:00:43.784 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/ 2023-11-02T18:00:43.784 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZUms1EIZswYH+lG8MUeOtrVdE40= 2023-11-02T18:00:43.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZUms1EIZswYH+lG8MUeOtrVdE40=', 'Content-Length': '0'} 2023-11-02T18:00:43.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d336431e520b6b74-006543e3cb-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.830 DEBUG:boto:b'ibtljn-13dummy1000falsedummy2023-11-02T17:58:31.540Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:43.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:43.832 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:43.832 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:43.833 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.833 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.833 DEBUG:boto:Method: GET 2023-11-02T18:00:43.834 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.834 DEBUG:boto:Data: 2023-11-02T18:00:43.835 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.835 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.836 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.836 DEBUG:boto:Params: {} 2023-11-02T18:00:43.837 DEBUG:boto:Token: None 2023-11-02T18:00:43.837 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.838 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H7173mG6QYOqbg42uVsPZPRlTVc= 2023-11-02T18:00:43.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H7173mG6QYOqbg42uVsPZPRlTVc=', 'Content-Length': '0'} 2023-11-02T18:00:43.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a197c1405dc44f70-006543e3cb-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.842 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:43.842 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:43.843 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.843 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.844 DEBUG:boto:Method: GET 2023-11-02T18:00:43.844 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.845 DEBUG:boto:Data: 2023-11-02T18:00:43.845 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.845 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.846 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.846 DEBUG:boto:Params: {} 2023-11-02T18:00:43.847 DEBUG:boto:Token: None 2023-11-02T18:00:43.847 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.848 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H7173mG6QYOqbg42uVsPZPRlTVc= 2023-11-02T18:00:43.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H7173mG6QYOqbg42uVsPZPRlTVc=', 'Content-Length': '0'} 2023-11-02T18:00:43.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb6e179383e61894-006543e3cb-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.852 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:43.852 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:43.853 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.853 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.854 DEBUG:boto:Method: HEAD 2023-11-02T18:00:43.854 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.854 DEBUG:boto:Data: 2023-11-02T18:00:43.855 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.855 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.856 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.856 DEBUG:boto:Params: {} 2023-11-02T18:00:43.857 DEBUG:boto:Token: None 2023-11-02T18:00:43.857 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.858 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fAP5DU9SYMqaDxRIrrJfkpptRa8= 2023-11-02T18:00:43.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fAP5DU9SYMqaDxRIrrJfkpptRa8=', 'Content-Length': '0'} 2023-11-02T18:00:43.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000427f042537d76c7f-006543e3cb-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.861 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:43.862 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:43.862 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.863 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.863 DEBUG:boto:Method: HEAD 2023-11-02T18:00:43.864 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.864 DEBUG:boto:Data: 2023-11-02T18:00:43.865 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.865 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.866 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.866 DEBUG:boto:Params: {} 2023-11-02T18:00:43.867 DEBUG:boto:Token: None 2023-11-02T18:00:43.867 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.868 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fAP5DU9SYMqaDxRIrrJfkpptRa8= 2023-11-02T18:00:43.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fAP5DU9SYMqaDxRIrrJfkpptRa8=', 'Content-Length': '0'} 2023-11-02T18:00:43.871 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006657c84f00e2050e-006543e3cb-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:43.872 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:43.872 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:43.873 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.873 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.874 DEBUG:boto:Method: GET 2023-11-02T18:00:43.874 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.875 DEBUG:boto:Data: 2023-11-02T18:00:43.875 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.876 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.876 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.877 DEBUG:boto:Params: {} 2023-11-02T18:00:43.877 DEBUG:boto:Token: None 2023-11-02T18:00:43.878 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.878 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H7173mG6QYOqbg42uVsPZPRlTVc= 2023-11-02T18:00:43.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H7173mG6QYOqbg42uVsPZPRlTVc=', 'Content-Length': '0'} 2023-11-02T18:00:43.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030d7ff6fa6269a20-006543e3cb-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.882 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:43.883 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:43.883 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.883 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.884 DEBUG:boto:Method: GET 2023-11-02T18:00:43.884 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.885 DEBUG:boto:Data: 2023-11-02T18:00:43.885 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.886 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.886 DEBUG:boto:Params: {} 2023-11-02T18:00:43.887 DEBUG:boto:Token: None 2023-11-02T18:00:43.887 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:43.888 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H7173mG6QYOqbg42uVsPZPRlTVc= 2023-11-02T18:00:43.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H7173mG6QYOqbg42uVsPZPRlTVc=', 'Content-Length': '0'} 2023-11-02T18:00:43.891 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a402e910e635e24-006543e3cb-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.938 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:43.939 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:43.939 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:43.940 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:43.940 DEBUG:boto:Method: GET 2023-11-02T18:00:43.941 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:43.941 DEBUG:boto:Data: 2023-11-02T18:00:43.942 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.942 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:43.942 DEBUG:boto:Port: 8001 2023-11-02T18:00:43.943 DEBUG:boto:Params: {} 2023-11-02T18:00:43.943 DEBUG:boto:Token: None 2023-11-02T18:00:43.944 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/ 2023-11-02T18:00:43.944 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZUms1EIZswYH+lG8MUeOtrVdE40= 2023-11-02T18:00:43.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZUms1EIZswYH+lG8MUeOtrVdE40=', 'Content-Length': '0'} 2023-11-02T18:00:43.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a754b67eb7fc9ea-006543e3cb-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:43.951 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:43.952 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:43.953 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:43.953 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:43.953 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:43.954 DEBUG:boto:Method: GET 2023-11-02T18:00:43.954 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:43.955 DEBUG:boto:Data: 2023-11-02T18:00:43.955 DEBUG:boto:Headers: {} 2023-11-02T18:00:43.956 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:43.956 DEBUG:boto:Port: 8000 2023-11-02T18:00:43.957 DEBUG:boto:Params: {} 2023-11-02T18:00:43.957 DEBUG:boto:Token: None 2023-11-02T18:00:43.958 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:43 GMT /ibtljn-13/ 2023-11-02T18:00:43.958 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZUms1EIZswYH+lG8MUeOtrVdE40= 2023-11-02T18:00:43.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZUms1EIZswYH+lG8MUeOtrVdE40=', 'Content-Length': '0'} 2023-11-02T18:00:43.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000730efd166dd96a44-006543e3cb-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.006 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.007 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.008 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.008 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.008 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:44.009 DEBUG:boto:Data: 2023-11-02T18:00:44.009 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.010 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.010 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.011 DEBUG:boto:Params: {} 2023-11-02T18:00:44.011 DEBUG:boto:Token: None 2023-11-02T18:00:44.012 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj 2023-11-02T18:00:44.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0= 2023-11-02T18:00:44.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0=', 'Content-Length': '0'} 2023-11-02T18:00:44.019 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df9cfc85789cc4f2-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.020 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.020 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.020 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.021 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:44.021 DEBUG:boto:Data: 2023-11-02T18:00:44.022 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.022 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.022 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.023 DEBUG:boto:Params: {} 2023-11-02T18:00:44.023 DEBUG:boto:Token: None 2023-11-02T18:00:44.024 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj 2023-11-02T18:00:44.024 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0= 2023-11-02T18:00:44.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0=', 'Content-Length': '0'} 2023-11-02T18:00:44.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065b7b607e94266e4-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:44.032 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.032 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.033 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.033 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.034 DEBUG:boto:Method: GET 2023-11-02T18:00:44.034 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.034 DEBUG:boto:Data: 2023-11-02T18:00:44.035 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.035 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.036 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.036 DEBUG:boto:Params: {} 2023-11-02T18:00:44.037 DEBUG:boto:Token: None 2023-11-02T18:00:44.037 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.038 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0= 2023-11-02T18:00:44.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0=', 'Content-Length': '0'} 2023-11-02T18:00:44.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000996e789e61aa623d-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.042 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.042 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.043 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.043 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.044 DEBUG:boto:Method: GET 2023-11-02T18:00:44.044 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.045 DEBUG:boto:Data: 2023-11-02T18:00:44.045 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.046 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.047 DEBUG:boto:Params: {} 2023-11-02T18:00:44.047 DEBUG:boto:Token: None 2023-11-02T18:00:44.048 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.048 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0= 2023-11-02T18:00:44.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0=', 'Content-Length': '0'} 2023-11-02T18:00:44.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064916b407becb18b-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.094 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.095 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.095 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.096 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.096 DEBUG:boto:Method: GET 2023-11-02T18:00:44.097 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.097 DEBUG:boto:Data: 2023-11-02T18:00:44.098 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.098 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.099 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.099 DEBUG:boto:Params: {} 2023-11-02T18:00:44.099 DEBUG:boto:Token: None 2023-11-02T18:00:44.100 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.100 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be03161f0139120d-006543e3cc-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.108 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.109 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.110 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.110 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.111 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.111 DEBUG:boto:Method: GET 2023-11-02T18:00:44.112 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.112 DEBUG:boto:Data: 2023-11-02T18:00:44.112 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.113 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.113 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.114 DEBUG:boto:Params: {} 2023-11-02T18:00:44.114 DEBUG:boto:Token: None 2023-11-02T18:00:44.115 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.115 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1b132f4ea4e8b37-006543e3cc-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.121 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:44.123 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.123 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.123 DEBUG:boto:path=/ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.124 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.124 DEBUG:boto:Method: GET 2023-11-02T18:00:44.125 DEBUG:boto:Path: /ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.125 DEBUG:boto:Data: 2023-11-02T18:00:44.126 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.126 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.127 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.127 DEBUG:boto:Params: {} 2023-11-02T18:00:44.127 DEBUG:boto:Token: None 2023-11-02T18:00:44.128 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.128 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:81XV7wKTqJXfyHj7VoSzeHqj0mg= 2023-11-02T18:00:44.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:81XV7wKTqJXfyHj7VoSzeHqj0mg=', 'Content-Length': '0'} 2023-11-02T18:00:44.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', 'LH98R3N3MJCMJx3TJUp072XhY3JxYyI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000931171a57034ecbc-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.174 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.175 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.175 DEBUG:boto:path=/ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.176 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.176 DEBUG:boto:Method: GET 2023-11-02T18:00:44.177 DEBUG:boto:Path: /ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.177 DEBUG:boto:Data: 2023-11-02T18:00:44.178 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.178 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.178 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.179 DEBUG:boto:Params: {} 2023-11-02T18:00:44.179 DEBUG:boto:Token: None 2023-11-02T18:00:44.180 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.180 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:81XV7wKTqJXfyHj7VoSzeHqj0mg= 2023-11-02T18:00:44.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:81XV7wKTqJXfyHj7VoSzeHqj0mg=', 'Content-Length': '0'} 2023-11-02T18:00:44.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', 'LH98R3N3MJCMJx3TJUp072XhY3JxYyI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002b1b621b7920e98-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.185 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.185 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.186 DEBUG:boto:path=/ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.186 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.187 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.187 DEBUG:boto:Path: /ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.187 DEBUG:boto:Data: 2023-11-02T18:00:44.188 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.188 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.189 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.189 DEBUG:boto:Params: {} 2023-11-02T18:00:44.190 DEBUG:boto:Token: None 2023-11-02T18:00:44.190 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.191 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q0Dv9b9f+o5gqanM9lHW4P0MkdA= 2023-11-02T18:00:44.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q0Dv9b9f+o5gqanM9lHW4P0MkdA=', 'Content-Length': '0'} 2023-11-02T18:00:44.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', 'LH98R3N3MJCMJx3TJUp072XhY3JxYyI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000196fba01067aa307-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.195 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.195 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.196 DEBUG:boto:path=/ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.196 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.197 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.197 DEBUG:boto:Path: /ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.198 DEBUG:boto:Data: 2023-11-02T18:00:44.198 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.198 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.199 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.199 DEBUG:boto:Params: {} 2023-11-02T18:00:44.200 DEBUG:boto:Token: None 2023-11-02T18:00:44.200 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q0Dv9b9f+o5gqanM9lHW4P0MkdA= 2023-11-02T18:00:44.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q0Dv9b9f+o5gqanM9lHW4P0MkdA=', 'Content-Length': '0'} 2023-11-02T18:00:44.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', 'LH98R3N3MJCMJx3TJUp072XhY3JxYyI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4b020835b907da2-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:44.205 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.205 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.206 DEBUG:boto:path=/ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.206 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.207 DEBUG:boto:Method: GET 2023-11-02T18:00:44.207 DEBUG:boto:Path: /ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.208 DEBUG:boto:Data: 2023-11-02T18:00:44.208 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.208 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.209 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.209 DEBUG:boto:Params: {} 2023-11-02T18:00:44.210 DEBUG:boto:Token: None 2023-11-02T18:00:44.210 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.211 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:81XV7wKTqJXfyHj7VoSzeHqj0mg= 2023-11-02T18:00:44.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:81XV7wKTqJXfyHj7VoSzeHqj0mg=', 'Content-Length': '0'} 2023-11-02T18:00:44.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', 'LH98R3N3MJCMJx3TJUp072XhY3JxYyI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007cb419a43530e28-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.215 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.216 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.216 DEBUG:boto:path=/ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.216 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.217 DEBUG:boto:Method: GET 2023-11-02T18:00:44.217 DEBUG:boto:Path: /ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.218 DEBUG:boto:Data: 2023-11-02T18:00:44.218 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.219 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.219 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.220 DEBUG:boto:Params: {} 2023-11-02T18:00:44.220 DEBUG:boto:Token: None 2023-11-02T18:00:44.221 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=LH98R3N3MJCMJx3TJUp072XhY3JxYyI 2023-11-02T18:00:44.221 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:81XV7wKTqJXfyHj7VoSzeHqj0mg= 2023-11-02T18:00:44.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:81XV7wKTqJXfyHj7VoSzeHqj0mg=', 'Content-Length': '0'} 2023-11-02T18:00:44.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', 'LH98R3N3MJCMJx3TJUp072XhY3JxYyI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edfe3077de4b30f1-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.266 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.267 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.267 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.268 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.268 DEBUG:boto:Method: GET 2023-11-02T18:00:44.269 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.269 DEBUG:boto:Data: 2023-11-02T18:00:44.270 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.270 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.270 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.271 DEBUG:boto:Params: {} 2023-11-02T18:00:44.271 DEBUG:boto:Token: None 2023-11-02T18:00:44.272 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.272 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d81e3367ed2fe5b-006543e3cc-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.280 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.281 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.281 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.282 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.282 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.283 DEBUG:boto:Method: GET 2023-11-02T18:00:44.283 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.284 DEBUG:boto:Data: 2023-11-02T18:00:44.284 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.285 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.285 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.286 DEBUG:boto:Params: {} 2023-11-02T18:00:44.286 DEBUG:boto:Token: None 2023-11-02T18:00:44.287 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.288 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000426e3c999ce3aa7-006543e3cc-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.294 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.294 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.295 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.295 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.296 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:44.296 DEBUG:boto:Data: 2023-11-02T18:00:44.297 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.297 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.297 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.298 DEBUG:boto:Params: {} 2023-11-02T18:00:44.298 DEBUG:boto:Token: None 2023-11-02T18:00:44.299 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj 2023-11-02T18:00:44.299 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0= 2023-11-02T18:00:44.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0=', 'Content-Length': '0'} 2023-11-02T18:00:44.306 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c80b33cf8ec38629-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.306 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.307 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.308 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.308 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:44.309 DEBUG:boto:Data: 2023-11-02T18:00:44.309 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.309 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.310 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.310 DEBUG:boto:Params: {} 2023-11-02T18:00:44.311 DEBUG:boto:Token: None 2023-11-02T18:00:44.311 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj 2023-11-02T18:00:44.311 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0= 2023-11-02T18:00:44.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0=', 'Content-Length': '0'} 2023-11-02T18:00:44.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036a4df10a9cee943-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:44.319 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.319 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.320 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.321 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.322 DEBUG:boto:Method: GET 2023-11-02T18:00:44.322 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.323 DEBUG:boto:Data: 2023-11-02T18:00:44.324 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.324 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.325 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.325 DEBUG:boto:Params: {} 2023-11-02T18:00:44.326 DEBUG:boto:Token: None 2023-11-02T18:00:44.326 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.327 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0= 2023-11-02T18:00:44.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0=', 'Content-Length': '0'} 2023-11-02T18:00:44.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e7940235ae01c43-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.374 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.375 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.375 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.376 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.376 DEBUG:boto:Method: GET 2023-11-02T18:00:44.377 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.377 DEBUG:boto:Data: 2023-11-02T18:00:44.378 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.378 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.378 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.379 DEBUG:boto:Params: {} 2023-11-02T18:00:44.379 DEBUG:boto:Token: None 2023-11-02T18:00:44.380 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.381 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0= 2023-11-02T18:00:44.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0=', 'Content-Length': '0'} 2023-11-02T18:00:44.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ee0f7bc8bcebb8d-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.385 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.385 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.386 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.386 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.387 DEBUG:boto:Method: GET 2023-11-02T18:00:44.387 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.388 DEBUG:boto:Data: 2023-11-02T18:00:44.388 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.388 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.389 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.390 DEBUG:boto:Params: {} 2023-11-02T18:00:44.391 DEBUG:boto:Token: None 2023-11-02T18:00:44.392 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.392 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d400a8f59ea9e169-006543e3cc-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.399 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.400 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.401 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.401 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.402 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.402 DEBUG:boto:Method: GET 2023-11-02T18:00:44.403 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.403 DEBUG:boto:Data: 2023-11-02T18:00:44.403 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.404 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.404 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.405 DEBUG:boto:Params: {} 2023-11-02T18:00:44.405 DEBUG:boto:Token: None 2023-11-02T18:00:44.406 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.406 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6ba3227797f6e20-006543e3cc-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.454 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:44.456 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.456 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.457 DEBUG:boto:path=/ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.457 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.458 DEBUG:boto:Method: GET 2023-11-02T18:00:44.458 DEBUG:boto:Path: /ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.459 DEBUG:boto:Data: 2023-11-02T18:00:44.459 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.459 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.460 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.460 DEBUG:boto:Params: {} 2023-11-02T18:00:44.461 DEBUG:boto:Token: None 2023-11-02T18:00:44.462 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.462 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wMoAUv58lfJLe/72aD5pv7PQvjg= 2023-11-02T18:00:44.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wMoAUv58lfJLe/72aD5pv7PQvjg=', 'Content-Length': '0'} 2023-11-02T18:00:44.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', 'LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002061380a9bc8370b-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.466 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.467 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.467 DEBUG:boto:path=/ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.468 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.468 DEBUG:boto:Method: GET 2023-11-02T18:00:44.469 DEBUG:boto:Path: /ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.469 DEBUG:boto:Data: 2023-11-02T18:00:44.470 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.470 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.471 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.471 DEBUG:boto:Params: {} 2023-11-02T18:00:44.472 DEBUG:boto:Token: None 2023-11-02T18:00:44.472 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.473 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wMoAUv58lfJLe/72aD5pv7PQvjg= 2023-11-02T18:00:44.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wMoAUv58lfJLe/72aD5pv7PQvjg=', 'Content-Length': '0'} 2023-11-02T18:00:44.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', 'LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cada3450cbdbb154-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.477 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.478 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.478 DEBUG:boto:path=/ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.479 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.479 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.480 DEBUG:boto:Path: /ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.480 DEBUG:boto:Data: 2023-11-02T18:00:44.480 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.481 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.481 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.482 DEBUG:boto:Params: {} 2023-11-02T18:00:44.482 DEBUG:boto:Token: None 2023-11-02T18:00:44.482 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.483 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4hFjiPRpqCt0fBxMrENJ1WLx5xs= 2023-11-02T18:00:44.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4hFjiPRpqCt0fBxMrENJ1WLx5xs=', 'Content-Length': '0'} 2023-11-02T18:00:44.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', 'LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b21d044b5598a436-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.487 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.488 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.488 DEBUG:boto:path=/ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.488 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.489 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.489 DEBUG:boto:Path: /ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.490 DEBUG:boto:Data: 2023-11-02T18:00:44.490 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.490 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.491 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.492 DEBUG:boto:Params: {} 2023-11-02T18:00:44.492 DEBUG:boto:Token: None 2023-11-02T18:00:44.493 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.494 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4hFjiPRpqCt0fBxMrENJ1WLx5xs= 2023-11-02T18:00:44.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4hFjiPRpqCt0fBxMrENJ1WLx5xs=', 'Content-Length': '0'} 2023-11-02T18:00:44.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', 'LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000680ce2072b649dfe-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:44.499 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.500 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.500 DEBUG:boto:path=/ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.501 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.501 DEBUG:boto:Method: GET 2023-11-02T18:00:44.502 DEBUG:boto:Path: /ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.502 DEBUG:boto:Data: 2023-11-02T18:00:44.503 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.503 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.504 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.504 DEBUG:boto:Params: {} 2023-11-02T18:00:44.505 DEBUG:boto:Token: None 2023-11-02T18:00:44.506 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.506 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wMoAUv58lfJLe/72aD5pv7PQvjg= 2023-11-02T18:00:44.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wMoAUv58lfJLe/72aD5pv7PQvjg=', 'Content-Length': '0'} 2023-11-02T18:00:44.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', 'LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c14dc06aa9f3b6b-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.511 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.512 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.512 DEBUG:boto:path=/ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.513 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.513 DEBUG:boto:Method: GET 2023-11-02T18:00:44.514 DEBUG:boto:Path: /ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.514 DEBUG:boto:Data: 2023-11-02T18:00:44.515 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.515 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.515 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.516 DEBUG:boto:Params: {} 2023-11-02T18:00:44.516 DEBUG:boto:Token: None 2023-11-02T18:00:44.517 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj 2023-11-02T18:00:44.517 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wMoAUv58lfJLe/72aD5pv7PQvjg= 2023-11-02T18:00:44.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wMoAUv58lfJLe/72aD5pv7PQvjg=', 'Content-Length': '0'} 2023-11-02T18:00:44.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', 'LuAX4QvkX.Ol7bCELhrZp6YF1yw.dQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008beb8804f8b23b82-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.521 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.521 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.522 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.522 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.523 DEBUG:boto:Method: GET 2023-11-02T18:00:44.523 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.524 DEBUG:boto:Data: 2023-11-02T18:00:44.524 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.525 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.525 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.526 DEBUG:boto:Params: {} 2023-11-02T18:00:44.526 DEBUG:boto:Token: None 2023-11-02T18:00:44.527 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.527 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c323075754e4fccb-006543e3cc-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.574 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.575 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.576 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.576 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.577 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.577 DEBUG:boto:Method: GET 2023-11-02T18:00:44.578 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.578 DEBUG:boto:Data: 2023-11-02T18:00:44.579 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.579 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.580 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.582 DEBUG:boto:Params: {} 2023-11-02T18:00:44.583 DEBUG:boto:Token: None 2023-11-02T18:00:44.583 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.584 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dd8d94217de422d-006543e3cc-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.590 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.590 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.591 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.591 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.592 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:44.592 DEBUG:boto:Data: 2023-11-02T18:00:44.592 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.593 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.594 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.594 DEBUG:boto:Params: {} 2023-11-02T18:00:44.595 DEBUG:boto:Token: None 2023-11-02T18:00:44.595 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj 2023-11-02T18:00:44.596 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0= 2023-11-02T18:00:44.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0=', 'Content-Length': '0'} 2023-11-02T18:00:44.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a98cb32802749ae-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.603 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.604 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.605 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.605 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:44.606 DEBUG:boto:Data: 2023-11-02T18:00:44.607 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.607 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.608 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.608 DEBUG:boto:Params: {} 2023-11-02T18:00:44.609 DEBUG:boto:Token: None 2023-11-02T18:00:44.609 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj 2023-11-02T18:00:44.610 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0= 2023-11-02T18:00:44.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0=', 'Content-Length': '0'} 2023-11-02T18:00:44.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6404260708e4fb5-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:44.617 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.618 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.618 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.619 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.619 DEBUG:boto:Method: GET 2023-11-02T18:00:44.620 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.620 DEBUG:boto:Data: 2023-11-02T18:00:44.621 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.621 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.621 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.622 DEBUG:boto:Params: {} 2023-11-02T18:00:44.623 DEBUG:boto:Token: None 2023-11-02T18:00:44.623 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0= 2023-11-02T18:00:44.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0=', 'Content-Length': '0'} 2023-11-02T18:00:44.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008880b2ab50a37dfc-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.628 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.628 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.629 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.629 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.630 DEBUG:boto:Method: GET 2023-11-02T18:00:44.630 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.631 DEBUG:boto:Data: 2023-11-02T18:00:44.631 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.632 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.632 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.633 DEBUG:boto:Params: {} 2023-11-02T18:00:44.633 DEBUG:boto:Token: None 2023-11-02T18:00:44.634 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.634 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0= 2023-11-02T18:00:44.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0=', 'Content-Length': '0'} 2023-11-02T18:00:44.638 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008e4b11b98bc8f38-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.678 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.679 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.679 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.680 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.680 DEBUG:boto:Method: GET 2023-11-02T18:00:44.681 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.681 DEBUG:boto:Data: 2023-11-02T18:00:44.682 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.682 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.683 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.683 DEBUG:boto:Params: {} 2023-11-02T18:00:44.684 DEBUG:boto:Token: None 2023-11-02T18:00:44.684 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.685 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4ffe5c1048d97c2-006543e3cc-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.692 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.693 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.693 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.693 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.694 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.694 DEBUG:boto:Method: GET 2023-11-02T18:00:44.695 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.695 DEBUG:boto:Data: 2023-11-02T18:00:44.696 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.696 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.697 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.697 DEBUG:boto:Params: {} 2023-11-02T18:00:44.698 DEBUG:boto:Token: None 2023-11-02T18:00:44.698 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.699 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5d6d95641558256-006543e3cc-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.705 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:44.706 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.706 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.707 DEBUG:boto:path=/ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.707 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.708 DEBUG:boto:Method: GET 2023-11-02T18:00:44.708 DEBUG:boto:Path: /ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.709 DEBUG:boto:Data: 2023-11-02T18:00:44.709 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.709 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.710 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.710 DEBUG:boto:Params: {} 2023-11-02T18:00:44.711 DEBUG:boto:Token: None 2023-11-02T18:00:44.711 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:asa427YdknFmvMwpbDWX4LBK2YI= 2023-11-02T18:00:44.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:asa427YdknFmvMwpbDWX4LBK2YI=', 'Content-Length': '0'} 2023-11-02T18:00:44.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006068c179f94ac4c0-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.758 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.759 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.759 DEBUG:boto:path=/ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.760 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.760 DEBUG:boto:Method: GET 2023-11-02T18:00:44.761 DEBUG:boto:Path: /ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.761 DEBUG:boto:Data: 2023-11-02T18:00:44.762 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.762 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.763 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.763 DEBUG:boto:Params: {} 2023-11-02T18:00:44.764 DEBUG:boto:Token: None 2023-11-02T18:00:44.764 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.765 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:asa427YdknFmvMwpbDWX4LBK2YI= 2023-11-02T18:00:44.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:asa427YdknFmvMwpbDWX4LBK2YI=', 'Content-Length': '0'} 2023-11-02T18:00:44.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024a42190149f0c81-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.769 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.770 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.770 DEBUG:boto:path=/ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.770 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.771 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.771 DEBUG:boto:Path: /ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.772 DEBUG:boto:Data: 2023-11-02T18:00:44.772 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.772 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.773 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.773 DEBUG:boto:Params: {} 2023-11-02T18:00:44.774 DEBUG:boto:Token: None 2023-11-02T18:00:44.774 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.775 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gsd4bwonR+F/Jh+FPDRf0GoEF5Y= 2023-11-02T18:00:44.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gsd4bwonR+F/Jh+FPDRf0GoEF5Y=', 'Content-Length': '0'} 2023-11-02T18:00:44.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091e80a9fea855374-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.779 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.779 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.780 DEBUG:boto:path=/ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.780 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.781 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.781 DEBUG:boto:Path: /ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.782 DEBUG:boto:Data: 2023-11-02T18:00:44.782 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.782 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.783 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.783 DEBUG:boto:Params: {} 2023-11-02T18:00:44.784 DEBUG:boto:Token: None 2023-11-02T18:00:44.784 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gsd4bwonR+F/Jh+FPDRf0GoEF5Y= 2023-11-02T18:00:44.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gsd4bwonR+F/Jh+FPDRf0GoEF5Y=', 'Content-Length': '0'} 2023-11-02T18:00:44.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085b4f55f3fe0df26-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:44.791 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.792 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.792 DEBUG:boto:path=/ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.793 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.793 DEBUG:boto:Method: GET 2023-11-02T18:00:44.794 DEBUG:boto:Path: /ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.794 DEBUG:boto:Data: 2023-11-02T18:00:44.795 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.795 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.795 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.796 DEBUG:boto:Params: {} 2023-11-02T18:00:44.796 DEBUG:boto:Token: None 2023-11-02T18:00:44.797 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.798 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:asa427YdknFmvMwpbDWX4LBK2YI= 2023-11-02T18:00:44.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:asa427YdknFmvMwpbDWX4LBK2YI=', 'Content-Length': '0'} 2023-11-02T18:00:44.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d39fe5a77ec0827a-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.803 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.803 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.804 DEBUG:boto:path=/ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.804 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.805 DEBUG:boto:Method: GET 2023-11-02T18:00:44.805 DEBUG:boto:Path: /ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.806 DEBUG:boto:Data: 2023-11-02T18:00:44.806 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.807 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.807 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.808 DEBUG:boto:Params: {} 2023-11-02T18:00:44.808 DEBUG:boto:Token: None 2023-11-02T18:00:44.809 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW 2023-11-02T18:00:44.809 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:asa427YdknFmvMwpbDWX4LBK2YI= 2023-11-02T18:00:44.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:asa427YdknFmvMwpbDWX4LBK2YI=', 'Content-Length': '0'} 2023-11-02T18:00:44.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'YOZNVAi3SCKH-38AqA3kgfKr.a7X3TW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006331b1759a2e1a74-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.854 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.855 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.856 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.856 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.857 DEBUG:boto:Method: GET 2023-11-02T18:00:44.857 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.857 DEBUG:boto:Data: 2023-11-02T18:00:44.858 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.858 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.859 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.859 DEBUG:boto:Params: {} 2023-11-02T18:00:44.860 DEBUG:boto:Token: None 2023-11-02T18:00:44.860 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.861 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c4f778439226d06-006543e3cc-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.867 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.868 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.868 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.869 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.869 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.870 DEBUG:boto:Method: GET 2023-11-02T18:00:44.870 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.871 DEBUG:boto:Data: 2023-11-02T18:00:44.871 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.871 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.872 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.872 DEBUG:boto:Params: {} 2023-11-02T18:00:44.873 DEBUG:boto:Token: None 2023-11-02T18:00:44.873 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.874 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b77c727b38911399-006543e3cc-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.879 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.880 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.881 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.881 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.881 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:44.882 DEBUG:boto:Data: 2023-11-02T18:00:44.882 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.883 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.883 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.884 DEBUG:boto:Params: {} 2023-11-02T18:00:44.884 DEBUG:boto:Token: None 2023-11-02T18:00:44.885 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj 2023-11-02T18:00:44.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0= 2023-11-02T18:00:44.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0=', 'Content-Length': '0'} 2023-11-02T18:00:44.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfe141f2bbdf3b40-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.892 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.893 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.893 DEBUG:boto:Method: HEAD 2023-11-02T18:00:44.894 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:44.894 DEBUG:boto:Data: 2023-11-02T18:00:44.895 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.895 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.896 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.896 DEBUG:boto:Params: {} 2023-11-02T18:00:44.897 DEBUG:boto:Token: None 2023-11-02T18:00:44.897 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj 2023-11-02T18:00:44.898 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0= 2023-11-02T18:00:44.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a8GAHovSujOcsUwlZm7tO+Vm2B0=', 'Content-Length': '0'} 2023-11-02T18:00:44.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e615e3ecb5f9a601-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:44.905 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.906 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.906 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.907 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.907 DEBUG:boto:Method: GET 2023-11-02T18:00:44.907 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.908 DEBUG:boto:Data: 2023-11-02T18:00:44.908 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.909 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.909 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.910 DEBUG:boto:Params: {} 2023-11-02T18:00:44.911 DEBUG:boto:Token: None 2023-11-02T18:00:44.911 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.912 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0= 2023-11-02T18:00:44.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0=', 'Content-Length': '0'} 2023-11-02T18:00:44.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd352ef9e86c5188-006543e3cc-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.958 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:44.959 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:44.959 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.960 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.960 DEBUG:boto:Method: GET 2023-11-02T18:00:44.961 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.961 DEBUG:boto:Data: 2023-11-02T18:00:44.962 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.962 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.963 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.963 DEBUG:boto:Params: {} 2023-11-02T18:00:44.964 DEBUG:boto:Token: None 2023-11-02T18:00:44.964 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:44.965 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0= 2023-11-02T18:00:44.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j7WtJRavzOyf41w0Zfvw88SASt0=', 'Content-Length': '0'} 2023-11-02T18:00:44.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a26f13dacb474076-006543e3cc-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.969 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.969 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.970 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.970 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.971 DEBUG:boto:Method: GET 2023-11-02T18:00:44.971 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.972 DEBUG:boto:Data: 2023-11-02T18:00:44.972 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.973 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:44.973 DEBUG:boto:Port: 8001 2023-11-02T18:00:44.974 DEBUG:boto:Params: {} 2023-11-02T18:00:44.974 DEBUG:boto:Token: None 2023-11-02T18:00:44.975 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000105831b25aed0783-006543e3cc-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:44.982 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:44.983 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:44.983 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:44.984 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.984 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:44.985 DEBUG:boto:Method: GET 2023-11-02T18:00:44.985 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:44.986 DEBUG:boto:Data: 2023-11-02T18:00:44.986 DEBUG:boto:Headers: {} 2023-11-02T18:00:44.987 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:44.987 DEBUG:boto:Port: 8000 2023-11-02T18:00:44.988 DEBUG:boto:Params: {} 2023-11-02T18:00:44.988 DEBUG:boto:Token: None 2023-11-02T18:00:44.989 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:44 GMT /ibtljn-13/ 2023-11-02T18:00:44.989 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE= 2023-11-02T18:00:44.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFSXI9y+rnK2GD1Dn8m63UOYWZE=', 'Content-Length': '0'} 2023-11-02T18:00:44.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d9c2bae598148bd-006543e3cc-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.038 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:45.040 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.040 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.041 DEBUG:boto:path=/ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.041 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.042 DEBUG:boto:Method: GET 2023-11-02T18:00:45.042 DEBUG:boto:Path: /ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.043 DEBUG:boto:Data: 2023-11-02T18:00:45.043 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.044 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.044 DEBUG:boto:Params: {} 2023-11-02T18:00:45.045 DEBUG:boto:Token: None 2023-11-02T18:00:45.046 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.046 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bHq78nj/N7rPg/418VY4pPSxuQY= 2023-11-02T18:00:45.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bHq78nj/N7rPg/418VY4pPSxuQY=', 'Content-Length': '0'} 2023-11-02T18:00:45.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e45074d22b0cf707-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.051 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.051 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.052 DEBUG:boto:path=/ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.052 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.053 DEBUG:boto:Method: GET 2023-11-02T18:00:45.053 DEBUG:boto:Path: /ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.053 DEBUG:boto:Data: 2023-11-02T18:00:45.054 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.054 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.055 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.055 DEBUG:boto:Params: {} 2023-11-02T18:00:45.056 DEBUG:boto:Token: None 2023-11-02T18:00:45.056 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.057 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bHq78nj/N7rPg/418VY4pPSxuQY= 2023-11-02T18:00:45.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bHq78nj/N7rPg/418VY4pPSxuQY=', 'Content-Length': '0'} 2023-11-02T18:00:45.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000003771026edaf272-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.061 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.062 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.062 DEBUG:boto:path=/ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.063 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.063 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.064 DEBUG:boto:Path: /ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.064 DEBUG:boto:Data: 2023-11-02T18:00:45.064 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.065 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.065 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.066 DEBUG:boto:Params: {} 2023-11-02T18:00:45.066 DEBUG:boto:Token: None 2023-11-02T18:00:45.067 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.067 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7E2e8ySjeatcjOsG9SaTO6dv/zk= 2023-11-02T18:00:45.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7E2e8ySjeatcjOsG9SaTO6dv/zk=', 'Content-Length': '0'} 2023-11-02T18:00:45.071 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a711d06868f2156-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.071 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.072 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.072 DEBUG:boto:path=/ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.073 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.073 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.074 DEBUG:boto:Path: /ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.074 DEBUG:boto:Data: 2023-11-02T18:00:45.074 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.075 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.075 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.076 DEBUG:boto:Params: {} 2023-11-02T18:00:45.076 DEBUG:boto:Token: None 2023-11-02T18:00:45.077 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.077 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7E2e8ySjeatcjOsG9SaTO6dv/zk= 2023-11-02T18:00:45.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7E2e8ySjeatcjOsG9SaTO6dv/zk=', 'Content-Length': '0'} 2023-11-02T18:00:45.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb4b1b7fb2cb925e-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:45.082 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.082 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.083 DEBUG:boto:path=/ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.083 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.084 DEBUG:boto:Method: GET 2023-11-02T18:00:45.084 DEBUG:boto:Path: /ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.084 DEBUG:boto:Data: 2023-11-02T18:00:45.085 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.085 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.086 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.086 DEBUG:boto:Params: {} 2023-11-02T18:00:45.087 DEBUG:boto:Token: None 2023-11-02T18:00:45.087 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.088 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bHq78nj/N7rPg/418VY4pPSxuQY= 2023-11-02T18:00:45.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bHq78nj/N7rPg/418VY4pPSxuQY=', 'Content-Length': '0'} 2023-11-02T18:00:45.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000004b215407c5e03b-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.134 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.135 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.135 DEBUG:boto:path=/ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.136 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.136 DEBUG:boto:Method: GET 2023-11-02T18:00:45.137 DEBUG:boto:Path: /ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.137 DEBUG:boto:Data: 2023-11-02T18:00:45.138 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.139 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.139 DEBUG:boto:Params: {} 2023-11-02T18:00:45.140 DEBUG:boto:Token: None 2023-11-02T18:00:45.140 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q- 2023-11-02T18:00:45.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bHq78nj/N7rPg/418VY4pPSxuQY= 2023-11-02T18:00:45.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bHq78nj/N7rPg/418VY4pPSxuQY=', 'Content-Length': '0'} 2023-11-02T18:00:45.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '3ZBJYSiFwypmzTZQ-DUYT7.wn4UD.q-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032d7c6af9d0bf4ad-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.145 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.146 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.146 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.147 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.147 DEBUG:boto:Method: GET 2023-11-02T18:00:45.148 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.148 DEBUG:boto:Data: 2023-11-02T18:00:45.149 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.149 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.149 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.150 DEBUG:boto:Params: {} 2023-11-02T18:00:45.150 DEBUG:boto:Token: None 2023-11-02T18:00:45.151 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090b0c045c57dc631-006543e3cd-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.158 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.159 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.159 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.160 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.160 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.161 DEBUG:boto:Method: GET 2023-11-02T18:00:45.161 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.161 DEBUG:boto:Data: 2023-11-02T18:00:45.162 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.162 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.163 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.163 DEBUG:boto:Params: {} 2023-11-02T18:00:45.163 DEBUG:boto:Token: None 2023-11-02T18:00:45.164 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.164 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a156d958178508-006543e3cd-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.171 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.172 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.172 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.173 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.173 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:45.174 DEBUG:boto:Data: 2023-11-02T18:00:45.174 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.175 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.175 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.175 DEBUG:boto:Params: {} 2023-11-02T18:00:45.176 DEBUG:boto:Token: None 2023-11-02T18:00:45.176 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj 2023-11-02T18:00:45.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P7lnkL+x8WSq0/YmIQVoXl0gU08= 2023-11-02T18:00:45.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P7lnkL+x8WSq0/YmIQVoXl0gU08=', 'Content-Length': '0'} 2023-11-02T18:00:45.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000313b017aecd11459-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.184 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.184 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.185 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.185 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:45.186 DEBUG:boto:Data: 2023-11-02T18:00:45.186 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.187 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.187 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.187 DEBUG:boto:Params: {} 2023-11-02T18:00:45.188 DEBUG:boto:Token: None 2023-11-02T18:00:45.188 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj 2023-11-02T18:00:45.189 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P7lnkL+x8WSq0/YmIQVoXl0gU08= 2023-11-02T18:00:45.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P7lnkL+x8WSq0/YmIQVoXl0gU08=', 'Content-Length': '0'} 2023-11-02T18:00:45.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e84bca3bee68e9c1-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:45.197 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.197 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.197 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.198 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.198 DEBUG:boto:Method: GET 2023-11-02T18:00:45.199 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.199 DEBUG:boto:Data: 2023-11-02T18:00:45.200 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.200 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.200 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.201 DEBUG:boto:Params: {} 2023-11-02T18:00:45.201 DEBUG:boto:Token: None 2023-11-02T18:00:45.202 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.202 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VjNszdCv1I0yu/HHlW+YQgi2+vg= 2023-11-02T18:00:45.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VjNszdCv1I0yu/HHlW+YQgi2+vg=', 'Content-Length': '0'} 2023-11-02T18:00:45.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb9b9188f6748df0-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.246 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.247 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.247 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.248 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.248 DEBUG:boto:Method: GET 2023-11-02T18:00:45.249 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.249 DEBUG:boto:Data: 2023-11-02T18:00:45.250 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.250 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.251 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.251 DEBUG:boto:Params: {} 2023-11-02T18:00:45.252 DEBUG:boto:Token: None 2023-11-02T18:00:45.252 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.253 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VjNszdCv1I0yu/HHlW+YQgi2+vg= 2023-11-02T18:00:45.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VjNszdCv1I0yu/HHlW+YQgi2+vg=', 'Content-Length': '0'} 2023-11-02T18:00:45.256 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000392e6774c65b84c5-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.257 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.257 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.258 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.258 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.259 DEBUG:boto:Method: GET 2023-11-02T18:00:45.259 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.260 DEBUG:boto:Data: 2023-11-02T18:00:45.260 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.260 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.261 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.261 DEBUG:boto:Params: {} 2023-11-02T18:00:45.262 DEBUG:boto:Token: None 2023-11-02T18:00:45.262 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.263 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000157eb27b8bf521e5-006543e3cd-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.269 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.270 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.271 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.271 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.272 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.272 DEBUG:boto:Method: GET 2023-11-02T18:00:45.272 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.273 DEBUG:boto:Data: 2023-11-02T18:00:45.273 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.274 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.274 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.274 DEBUG:boto:Params: {} 2023-11-02T18:00:45.275 DEBUG:boto:Token: None 2023-11-02T18:00:45.276 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.276 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009af6d7a49bb62f31-006543e3cd-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.283 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:45.284 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.285 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.285 DEBUG:boto:path=/ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.286 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.286 DEBUG:boto:Method: GET 2023-11-02T18:00:45.287 DEBUG:boto:Path: /ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.287 DEBUG:boto:Data: 2023-11-02T18:00:45.288 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.288 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.288 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.289 DEBUG:boto:Params: {} 2023-11-02T18:00:45.290 DEBUG:boto:Token: None 2023-11-02T18:00:45.290 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.291 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qz8wSIJxGkazN1SoPFuCa5pGM6c= 2023-11-02T18:00:45.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qz8wSIJxGkazN1SoPFuCa5pGM6c=', 'Content-Length': '0'} 2023-11-02T18:00:45.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'vH09Eyl.K.hzks5sEqA60vXpTkAW46q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdde8904e2748e1a-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.295 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.295 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.296 DEBUG:boto:path=/ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.296 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.297 DEBUG:boto:Method: GET 2023-11-02T18:00:45.297 DEBUG:boto:Path: /ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.297 DEBUG:boto:Data: 2023-11-02T18:00:45.298 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.298 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.299 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.299 DEBUG:boto:Params: {} 2023-11-02T18:00:45.300 DEBUG:boto:Token: None 2023-11-02T18:00:45.300 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.301 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qz8wSIJxGkazN1SoPFuCa5pGM6c= 2023-11-02T18:00:45.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qz8wSIJxGkazN1SoPFuCa5pGM6c=', 'Content-Length': '0'} 2023-11-02T18:00:45.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'vH09Eyl.K.hzks5sEqA60vXpTkAW46q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bff0871e90aebe08-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.346 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.347 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.347 DEBUG:boto:path=/ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.348 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.348 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.349 DEBUG:boto:Path: /ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.349 DEBUG:boto:Data: 2023-11-02T18:00:45.350 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.350 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.350 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.351 DEBUG:boto:Params: {} 2023-11-02T18:00:45.351 DEBUG:boto:Token: None 2023-11-02T18:00:45.352 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.352 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n2GWPbxENhot8QlALlgBMxhoooA= 2023-11-02T18:00:45.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n2GWPbxENhot8QlALlgBMxhoooA=', 'Content-Length': '0'} 2023-11-02T18:00:45.355 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'vH09Eyl.K.hzks5sEqA60vXpTkAW46q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f91048c35404cad-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.356 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.357 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.357 DEBUG:boto:path=/ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.357 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.358 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.358 DEBUG:boto:Path: /ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.359 DEBUG:boto:Data: 2023-11-02T18:00:45.359 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.360 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.360 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.360 DEBUG:boto:Params: {} 2023-11-02T18:00:45.361 DEBUG:boto:Token: None 2023-11-02T18:00:45.361 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.362 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n2GWPbxENhot8QlALlgBMxhoooA= 2023-11-02T18:00:45.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n2GWPbxENhot8QlALlgBMxhoooA=', 'Content-Length': '0'} 2023-11-02T18:00:45.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'vH09Eyl.K.hzks5sEqA60vXpTkAW46q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002db49e01a311231c-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:45.366 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.367 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.367 DEBUG:boto:path=/ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.368 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.368 DEBUG:boto:Method: GET 2023-11-02T18:00:45.369 DEBUG:boto:Path: /ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.369 DEBUG:boto:Data: 2023-11-02T18:00:45.370 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.371 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.371 DEBUG:boto:Params: {} 2023-11-02T18:00:45.372 DEBUG:boto:Token: None 2023-11-02T18:00:45.372 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.373 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qz8wSIJxGkazN1SoPFuCa5pGM6c= 2023-11-02T18:00:45.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qz8wSIJxGkazN1SoPFuCa5pGM6c=', 'Content-Length': '0'} 2023-11-02T18:00:45.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'vH09Eyl.K.hzks5sEqA60vXpTkAW46q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ffd245828c5db7c-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.418 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.419 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.420 DEBUG:boto:path=/ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.420 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.421 DEBUG:boto:Method: GET 2023-11-02T18:00:45.421 DEBUG:boto:Path: /ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.421 DEBUG:boto:Data: 2023-11-02T18:00:45.422 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.423 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.423 DEBUG:boto:Params: {} 2023-11-02T18:00:45.424 DEBUG:boto:Token: None 2023-11-02T18:00:45.424 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=vH09Eyl.K.hzks5sEqA60vXpTkAW46q 2023-11-02T18:00:45.425 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qz8wSIJxGkazN1SoPFuCa5pGM6c= 2023-11-02T18:00:45.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qz8wSIJxGkazN1SoPFuCa5pGM6c=', 'Content-Length': '0'} 2023-11-02T18:00:45.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'vH09Eyl.K.hzks5sEqA60vXpTkAW46q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c24d453334c27188-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.429 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.430 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.430 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.431 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.431 DEBUG:boto:Method: GET 2023-11-02T18:00:45.431 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.432 DEBUG:boto:Data: 2023-11-02T18:00:45.432 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.433 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.433 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.433 DEBUG:boto:Params: {} 2023-11-02T18:00:45.434 DEBUG:boto:Token: None 2023-11-02T18:00:45.435 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.435 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020ab6a8ea9145d7f-006543e3cd-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.442 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.443 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.443 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.444 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.444 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.445 DEBUG:boto:Method: GET 2023-11-02T18:00:45.445 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.446 DEBUG:boto:Data: 2023-11-02T18:00:45.446 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.447 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.448 DEBUG:boto:Params: {} 2023-11-02T18:00:45.448 DEBUG:boto:Token: None 2023-11-02T18:00:45.449 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.449 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a138ea073a37ed95-006543e3cd-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.456 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.456 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.457 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.457 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.458 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:45.458 DEBUG:boto:Data: 2023-11-02T18:00:45.459 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.459 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.459 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.460 DEBUG:boto:Params: {} 2023-11-02T18:00:45.460 DEBUG:boto:Token: None 2023-11-02T18:00:45.461 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj 2023-11-02T18:00:45.461 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P7lnkL+x8WSq0/YmIQVoXl0gU08= 2023-11-02T18:00:45.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P7lnkL+x8WSq0/YmIQVoXl0gU08=', 'Content-Length': '0'} 2023-11-02T18:00:45.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4ffe53b844920ab-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.468 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.469 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.469 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.470 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:45.470 DEBUG:boto:Data: 2023-11-02T18:00:45.471 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.471 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.472 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.472 DEBUG:boto:Params: {} 2023-11-02T18:00:45.473 DEBUG:boto:Token: None 2023-11-02T18:00:45.473 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj 2023-11-02T18:00:45.474 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P7lnkL+x8WSq0/YmIQVoXl0gU08= 2023-11-02T18:00:45.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P7lnkL+x8WSq0/YmIQVoXl0gU08=', 'Content-Length': '0'} 2023-11-02T18:00:45.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ef388f6f17c4d62-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:45.481 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.482 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.482 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.483 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.483 DEBUG:boto:Method: GET 2023-11-02T18:00:45.484 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.484 DEBUG:boto:Data: 2023-11-02T18:00:45.485 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.485 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.486 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.487 DEBUG:boto:Params: {} 2023-11-02T18:00:45.487 DEBUG:boto:Token: None 2023-11-02T18:00:45.488 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.488 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VjNszdCv1I0yu/HHlW+YQgi2+vg= 2023-11-02T18:00:45.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VjNszdCv1I0yu/HHlW+YQgi2+vg=', 'Content-Length': '0'} 2023-11-02T18:00:45.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e642fd62d720f55-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.534 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.535 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.536 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.536 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.537 DEBUG:boto:Method: GET 2023-11-02T18:00:45.537 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.538 DEBUG:boto:Data: 2023-11-02T18:00:45.538 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.539 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.540 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.540 DEBUG:boto:Params: {} 2023-11-02T18:00:45.541 DEBUG:boto:Token: None 2023-11-02T18:00:45.541 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.542 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VjNszdCv1I0yu/HHlW+YQgi2+vg= 2023-11-02T18:00:45.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VjNszdCv1I0yu/HHlW+YQgi2+vg=', 'Content-Length': '0'} 2023-11-02T18:00:45.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bfb4a640db4fad8-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.546 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.547 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.547 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.548 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.548 DEBUG:boto:Method: GET 2023-11-02T18:00:45.549 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.549 DEBUG:boto:Data: 2023-11-02T18:00:45.550 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.550 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.551 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.551 DEBUG:boto:Params: {} 2023-11-02T18:00:45.552 DEBUG:boto:Token: None 2023-11-02T18:00:45.553 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.553 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c5512ab192741ea-006543e3cd-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.559 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.560 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.560 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.561 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.561 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.562 DEBUG:boto:Method: GET 2023-11-02T18:00:45.562 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.563 DEBUG:boto:Data: 2023-11-02T18:00:45.563 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.564 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.565 DEBUG:boto:Params: {} 2023-11-02T18:00:45.565 DEBUG:boto:Token: None 2023-11-02T18:00:45.566 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.566 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1e66b683594fde9-006543e3cd-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.572 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:45.573 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.573 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.574 DEBUG:boto:path=/ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.574 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.575 DEBUG:boto:Method: GET 2023-11-02T18:00:45.575 DEBUG:boto:Path: /ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.575 DEBUG:boto:Data: 2023-11-02T18:00:45.576 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.576 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.577 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.577 DEBUG:boto:Params: {} 2023-11-02T18:00:45.578 DEBUG:boto:Token: None 2023-11-02T18:00:45.578 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.579 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IqU+X3SQYC/wGZkz1F4UsUF/N04= 2023-11-02T18:00:45.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IqU+X3SQYC/wGZkz1F4UsUF/N04=', 'Content-Length': '0'} 2023-11-02T18:00:45.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'bHdUIMsrwekD7ydoBEbIfYoJgPScbWU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b754c68ee01f5b93-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.583 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.584 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.584 DEBUG:boto:path=/ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.584 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.585 DEBUG:boto:Method: GET 2023-11-02T18:00:45.585 DEBUG:boto:Path: /ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.586 DEBUG:boto:Data: 2023-11-02T18:00:45.586 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.587 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.587 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.588 DEBUG:boto:Params: {} 2023-11-02T18:00:45.588 DEBUG:boto:Token: None 2023-11-02T18:00:45.589 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.589 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IqU+X3SQYC/wGZkz1F4UsUF/N04= 2023-11-02T18:00:45.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IqU+X3SQYC/wGZkz1F4UsUF/N04=', 'Content-Length': '0'} 2023-11-02T18:00:45.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'bHdUIMsrwekD7ydoBEbIfYoJgPScbWU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f9acb8b38e748d5-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.594 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.594 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.595 DEBUG:boto:path=/ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.595 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.595 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.596 DEBUG:boto:Path: /ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.596 DEBUG:boto:Data: 2023-11-02T18:00:45.597 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.597 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.598 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.598 DEBUG:boto:Params: {} 2023-11-02T18:00:45.599 DEBUG:boto:Token: None 2023-11-02T18:00:45.599 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.600 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b2rON33p2Pzpi5KiEaD/aY9Iclc= 2023-11-02T18:00:45.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b2rON33p2Pzpi5KiEaD/aY9Iclc=', 'Content-Length': '0'} 2023-11-02T18:00:45.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'bHdUIMsrwekD7ydoBEbIfYoJgPScbWU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6c4aefc3a693746-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.606 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.606 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.607 DEBUG:boto:path=/ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.607 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.608 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.608 DEBUG:boto:Path: /ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.609 DEBUG:boto:Data: 2023-11-02T18:00:45.609 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.609 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.610 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.610 DEBUG:boto:Params: {} 2023-11-02T18:00:45.611 DEBUG:boto:Token: None 2023-11-02T18:00:45.611 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b2rON33p2Pzpi5KiEaD/aY9Iclc= 2023-11-02T18:00:45.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b2rON33p2Pzpi5KiEaD/aY9Iclc=', 'Content-Length': '0'} 2023-11-02T18:00:45.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'bHdUIMsrwekD7ydoBEbIfYoJgPScbWU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d0a0282e4301f14-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:45.616 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.617 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.617 DEBUG:boto:path=/ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.618 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.618 DEBUG:boto:Method: GET 2023-11-02T18:00:45.619 DEBUG:boto:Path: /ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.619 DEBUG:boto:Data: 2023-11-02T18:00:45.619 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.620 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.620 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.621 DEBUG:boto:Params: {} 2023-11-02T18:00:45.621 DEBUG:boto:Token: None 2023-11-02T18:00:45.622 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.622 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IqU+X3SQYC/wGZkz1F4UsUF/N04= 2023-11-02T18:00:45.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IqU+X3SQYC/wGZkz1F4UsUF/N04=', 'Content-Length': '0'} 2023-11-02T18:00:45.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'bHdUIMsrwekD7ydoBEbIfYoJgPScbWU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000449f6731680ab2-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.670 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.671 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.671 DEBUG:boto:path=/ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.672 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.672 DEBUG:boto:Method: GET 2023-11-02T18:00:45.673 DEBUG:boto:Path: /ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.673 DEBUG:boto:Data: 2023-11-02T18:00:45.674 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.674 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.675 DEBUG:boto:Params: {} 2023-11-02T18:00:45.675 DEBUG:boto:Token: None 2023-11-02T18:00:45.676 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=bHdUIMsrwekD7ydoBEbIfYoJgPScbWU 2023-11-02T18:00:45.677 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IqU+X3SQYC/wGZkz1F4UsUF/N04= 2023-11-02T18:00:45.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IqU+X3SQYC/wGZkz1F4UsUF/N04=', 'Content-Length': '0'} 2023-11-02T18:00:45.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'bHdUIMsrwekD7ydoBEbIfYoJgPScbWU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027c293b08fd2ae99-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.681 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.681 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.682 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.682 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.683 DEBUG:boto:Method: GET 2023-11-02T18:00:45.683 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.684 DEBUG:boto:Data: 2023-11-02T18:00:45.684 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.684 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.685 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.685 DEBUG:boto:Params: {} 2023-11-02T18:00:45.686 DEBUG:boto:Token: None 2023-11-02T18:00:45.686 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.687 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f419efb8d52ec2ff-006543e3cd-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.693 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.694 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.695 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.695 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.695 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.696 DEBUG:boto:Method: GET 2023-11-02T18:00:45.696 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.697 DEBUG:boto:Data: 2023-11-02T18:00:45.697 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.698 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.698 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.699 DEBUG:boto:Params: {} 2023-11-02T18:00:45.699 DEBUG:boto:Token: None 2023-11-02T18:00:45.700 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.701 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d317cf96b41fafaa-006543e3cd-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.707 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.707 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.708 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.708 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.709 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:45.709 DEBUG:boto:Data: 2023-11-02T18:00:45.709 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.710 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.710 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.711 DEBUG:boto:Params: {} 2023-11-02T18:00:45.711 DEBUG:boto:Token: None 2023-11-02T18:00:45.712 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj 2023-11-02T18:00:45.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P7lnkL+x8WSq0/YmIQVoXl0gU08= 2023-11-02T18:00:45.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P7lnkL+x8WSq0/YmIQVoXl0gU08=', 'Content-Length': '0'} 2023-11-02T18:00:45.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e94678c07e49e32-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.719 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.719 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.720 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.720 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:45.721 DEBUG:boto:Data: 2023-11-02T18:00:45.721 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.721 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.722 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.722 DEBUG:boto:Params: {} 2023-11-02T18:00:45.723 DEBUG:boto:Token: None 2023-11-02T18:00:45.723 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj 2023-11-02T18:00:45.724 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P7lnkL+x8WSq0/YmIQVoXl0gU08= 2023-11-02T18:00:45.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P7lnkL+x8WSq0/YmIQVoXl0gU08=', 'Content-Length': '0'} 2023-11-02T18:00:45.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a96c104b45a6be7-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:45.731 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.732 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.732 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.733 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.733 DEBUG:boto:Method: GET 2023-11-02T18:00:45.733 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.734 DEBUG:boto:Data: 2023-11-02T18:00:45.734 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.735 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.735 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.736 DEBUG:boto:Params: {} 2023-11-02T18:00:45.736 DEBUG:boto:Token: None 2023-11-02T18:00:45.737 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.737 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VjNszdCv1I0yu/HHlW+YQgi2+vg= 2023-11-02T18:00:45.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VjNszdCv1I0yu/HHlW+YQgi2+vg=', 'Content-Length': '0'} 2023-11-02T18:00:45.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087b4a712abb91996-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.782 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.783 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.783 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.784 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.784 DEBUG:boto:Method: GET 2023-11-02T18:00:45.785 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.785 DEBUG:boto:Data: 2023-11-02T18:00:45.785 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.786 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.786 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.787 DEBUG:boto:Params: {} 2023-11-02T18:00:45.787 DEBUG:boto:Token: None 2023-11-02T18:00:45.788 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:45.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VjNszdCv1I0yu/HHlW+YQgi2+vg= 2023-11-02T18:00:45.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VjNszdCv1I0yu/HHlW+YQgi2+vg=', 'Content-Length': '0'} 2023-11-02T18:00:45.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000addf856c23e594c1-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.793 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.793 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.794 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.794 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.795 DEBUG:boto:Method: GET 2023-11-02T18:00:45.795 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.796 DEBUG:boto:Data: 2023-11-02T18:00:45.796 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.797 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.797 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.798 DEBUG:boto:Params: {} 2023-11-02T18:00:45.798 DEBUG:boto:Token: None 2023-11-02T18:00:45.799 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.799 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6b2304ffe3eb64e-006543e3cd-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.807 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.807 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.809 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.810 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.810 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.811 DEBUG:boto:Method: GET 2023-11-02T18:00:45.811 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.812 DEBUG:boto:Data: 2023-11-02T18:00:45.812 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.812 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.813 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.813 DEBUG:boto:Params: {} 2023-11-02T18:00:45.814 DEBUG:boto:Token: None 2023-11-02T18:00:45.815 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.815 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc32c2d8a064d015-006543e3cd-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.821 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:45.823 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.823 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.824 DEBUG:boto:path=/ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.824 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.825 DEBUG:boto:Method: GET 2023-11-02T18:00:45.825 DEBUG:boto:Path: /ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.826 DEBUG:boto:Data: 2023-11-02T18:00:45.826 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.827 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.827 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.828 DEBUG:boto:Params: {} 2023-11-02T18:00:45.828 DEBUG:boto:Token: None 2023-11-02T18:00:45.829 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.829 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yWyDs7fUBZpJCwHBJoSfJNzUOBg= 2023-11-02T18:00:45.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yWyDs7fUBZpJCwHBJoSfJNzUOBg=', 'Content-Length': '0'} 2023-11-02T18:00:45.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'qyKs0COjEwP9FUldxy3dhJMgbtCIdfw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3168ad0ed902a5d-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.834 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.834 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.835 DEBUG:boto:path=/ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.835 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.836 DEBUG:boto:Method: GET 2023-11-02T18:00:45.836 DEBUG:boto:Path: /ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.837 DEBUG:boto:Data: 2023-11-02T18:00:45.838 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.838 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.839 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.839 DEBUG:boto:Params: {} 2023-11-02T18:00:45.840 DEBUG:boto:Token: None 2023-11-02T18:00:45.841 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.841 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yWyDs7fUBZpJCwHBJoSfJNzUOBg= 2023-11-02T18:00:45.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yWyDs7fUBZpJCwHBJoSfJNzUOBg=', 'Content-Length': '0'} 2023-11-02T18:00:45.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'qyKs0COjEwP9FUldxy3dhJMgbtCIdfw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c251d5770be6f135-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.845 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.846 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.846 DEBUG:boto:path=/ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.847 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.847 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.847 DEBUG:boto:Path: /ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.848 DEBUG:boto:Data: 2023-11-02T18:00:45.848 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.849 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.849 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.850 DEBUG:boto:Params: {} 2023-11-02T18:00:45.850 DEBUG:boto:Token: None 2023-11-02T18:00:45.851 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.851 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7P6ix5m/eK8aqAuPfXz6lQ+2T0g= 2023-11-02T18:00:45.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7P6ix5m/eK8aqAuPfXz6lQ+2T0g=', 'Content-Length': '0'} 2023-11-02T18:00:45.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'qyKs0COjEwP9FUldxy3dhJMgbtCIdfw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec732fd6caca5c2c-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.856 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.856 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.856 DEBUG:boto:path=/ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.857 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.857 DEBUG:boto:Method: HEAD 2023-11-02T18:00:45.858 DEBUG:boto:Path: /ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.858 DEBUG:boto:Data: 2023-11-02T18:00:45.859 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.859 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.860 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.860 DEBUG:boto:Params: {} 2023-11-02T18:00:45.860 DEBUG:boto:Token: None 2023-11-02T18:00:45.861 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.861 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7P6ix5m/eK8aqAuPfXz6lQ+2T0g= 2023-11-02T18:00:45.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7P6ix5m/eK8aqAuPfXz6lQ+2T0g=', 'Content-Length': '0'} 2023-11-02T18:00:45.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'qyKs0COjEwP9FUldxy3dhJMgbtCIdfw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b1175fa826c09cb-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:45.866 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.866 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.867 DEBUG:boto:path=/ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.867 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.868 DEBUG:boto:Method: GET 2023-11-02T18:00:45.868 DEBUG:boto:Path: /ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.868 DEBUG:boto:Data: 2023-11-02T18:00:45.869 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.869 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.870 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.870 DEBUG:boto:Params: {} 2023-11-02T18:00:45.871 DEBUG:boto:Token: None 2023-11-02T18:00:45.871 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.872 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yWyDs7fUBZpJCwHBJoSfJNzUOBg= 2023-11-02T18:00:45.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yWyDs7fUBZpJCwHBJoSfJNzUOBg=', 'Content-Length': '0'} 2023-11-02T18:00:45.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'qyKs0COjEwP9FUldxy3dhJMgbtCIdfw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005626397f767a3c51-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.918 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:45.919 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:45.920 DEBUG:boto:path=/ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.920 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.920 DEBUG:boto:Method: GET 2023-11-02T18:00:45.921 DEBUG:boto:Path: /ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.921 DEBUG:boto:Data: 2023-11-02T18:00:45.922 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.922 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.923 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.923 DEBUG:boto:Params: {} 2023-11-02T18:00:45.924 DEBUG:boto:Token: None 2023-11-02T18:00:45.925 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/obj?versionId=qyKs0COjEwP9FUldxy3dhJMgbtCIdfw 2023-11-02T18:00:45.926 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yWyDs7fUBZpJCwHBJoSfJNzUOBg= 2023-11-02T18:00:45.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yWyDs7fUBZpJCwHBJoSfJNzUOBg=', 'Content-Length': '0'} 2023-11-02T18:00:45.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'qyKs0COjEwP9FUldxy3dhJMgbtCIdfw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df8099fae9fb694c-006543e3cd-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.931 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.931 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.932 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.932 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.932 DEBUG:boto:Method: GET 2023-11-02T18:00:45.933 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.933 DEBUG:boto:Data: 2023-11-02T18:00:45.934 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:45.935 DEBUG:boto:Port: 8001 2023-11-02T18:00:45.935 DEBUG:boto:Params: {} 2023-11-02T18:00:45.936 DEBUG:boto:Token: None 2023-11-02T18:00:45.936 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.937 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085c3adaccc71ce1c-006543e3cd-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:45.986 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:45.987 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:45.988 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:45.988 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.989 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:45.989 DEBUG:boto:Method: GET 2023-11-02T18:00:45.989 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:45.990 DEBUG:boto:Data: 2023-11-02T18:00:45.990 DEBUG:boto:Headers: {} 2023-11-02T18:00:45.991 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:45.991 DEBUG:boto:Port: 8000 2023-11-02T18:00:45.992 DEBUG:boto:Params: {} 2023-11-02T18:00:45.992 DEBUG:boto:Token: None 2023-11-02T18:00:45.993 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:45 GMT /ibtljn-13/ 2023-11-02T18:00:45.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs= 2023-11-02T18:00:45.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bb8Km8mqjFilaXRyNcvvLcJESWs=', 'Content-Length': '0'} 2023-11-02T18:00:45.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000344cd5ae6f8ec8e2-006543e3cd-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.000 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.001 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.001 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.001 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.002 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:46.002 DEBUG:boto:Data: 2023-11-02T18:00:46.003 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.003 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.003 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.004 DEBUG:boto:Params: {} 2023-11-02T18:00:46.004 DEBUG:boto:Token: None 2023-11-02T18:00:46.005 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj 2023-11-02T18:00:46.005 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI= 2023-11-02T18:00:46.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI=', 'Content-Length': '0'} 2023-11-02T18:00:46.012 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdd34ebb80618499-006543e3cd-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.013 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.013 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.014 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.014 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:46.017 DEBUG:boto:Data: 2023-11-02T18:00:46.018 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.018 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.018 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.019 DEBUG:boto:Params: {} 2023-11-02T18:00:46.019 DEBUG:boto:Token: None 2023-11-02T18:00:46.020 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj 2023-11-02T18:00:46.020 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI= 2023-11-02T18:00:46.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI=', 'Content-Length': '0'} 2023-11-02T18:00:46.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0cc1eaf2eb7c1d5-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:46.028 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.028 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.029 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.029 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.030 DEBUG:boto:Method: GET 2023-11-02T18:00:46.030 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.031 DEBUG:boto:Data: 2023-11-02T18:00:46.031 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.032 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.032 DEBUG:boto:Params: {} 2023-11-02T18:00:46.033 DEBUG:boto:Token: None 2023-11-02T18:00:46.033 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.034 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI= 2023-11-02T18:00:46.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI=', 'Content-Length': '0'} 2023-11-02T18:00:46.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e36c1b5c0f587d8-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.078 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.079 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.079 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.080 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.080 DEBUG:boto:Method: GET 2023-11-02T18:00:46.081 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.081 DEBUG:boto:Data: 2023-11-02T18:00:46.081 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.082 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.082 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.083 DEBUG:boto:Params: {} 2023-11-02T18:00:46.083 DEBUG:boto:Token: None 2023-11-02T18:00:46.084 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.084 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI= 2023-11-02T18:00:46.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI=', 'Content-Length': '0'} 2023-11-02T18:00:46.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f6cc5d5beaf1a56-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.089 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:46.089 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:46.090 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.090 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.091 DEBUG:boto:Method: GET 2023-11-02T18:00:46.091 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:46.092 DEBUG:boto:Data: 2023-11-02T18:00:46.092 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.093 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.093 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.094 DEBUG:boto:Params: {} 2023-11-02T18:00:46.094 DEBUG:boto:Token: None 2023-11-02T18:00:46.095 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/ 2023-11-02T18:00:46.095 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc= 2023-11-02T18:00:46.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc=', 'Content-Length': '0'} 2023-11-02T18:00:46.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030fe99cef24e1863-006543e3ce-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.102 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.103 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:46.104 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:46.104 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.105 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.105 DEBUG:boto:Method: GET 2023-11-02T18:00:46.106 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:46.106 DEBUG:boto:Data: 2023-11-02T18:00:46.106 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.107 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.107 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.108 DEBUG:boto:Params: {} 2023-11-02T18:00:46.108 DEBUG:boto:Token: None 2023-11-02T18:00:46.109 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/ 2023-11-02T18:00:46.110 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc= 2023-11-02T18:00:46.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc=', 'Content-Length': '0'} 2023-11-02T18:00:46.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dddeb67419c1cf3f-006543e3ce-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.115 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:46.116 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.117 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.117 DEBUG:boto:path=/ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.118 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.118 DEBUG:boto:Method: GET 2023-11-02T18:00:46.119 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.119 DEBUG:boto:Data: 2023-11-02T18:00:46.120 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.120 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.121 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.121 DEBUG:boto:Params: {} 2023-11-02T18:00:46.122 DEBUG:boto:Token: None 2023-11-02T18:00:46.122 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.123 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FLbcLRTGSlpqtOS28pMg6obnR1o= 2023-11-02T18:00:46.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FLbcLRTGSlpqtOS28pMg6obnR1o=', 'Content-Length': '0'} 2023-11-02T18:00:46.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f1655fcca75a342-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.127 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.128 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.128 DEBUG:boto:path=/ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.129 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.129 DEBUG:boto:Method: GET 2023-11-02T18:00:46.130 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.130 DEBUG:boto:Data: 2023-11-02T18:00:46.131 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.131 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.132 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.132 DEBUG:boto:Params: {} 2023-11-02T18:00:46.132 DEBUG:boto:Token: None 2023-11-02T18:00:46.133 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.133 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FLbcLRTGSlpqtOS28pMg6obnR1o= 2023-11-02T18:00:46.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FLbcLRTGSlpqtOS28pMg6obnR1o=', 'Content-Length': '0'} 2023-11-02T18:00:46.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5f92f8e9d617921-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.138 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.138 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.139 DEBUG:boto:path=/ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.139 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.139 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.140 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.140 DEBUG:boto:Data: 2023-11-02T18:00:46.142 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.142 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.143 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.143 DEBUG:boto:Params: {} 2023-11-02T18:00:46.143 DEBUG:boto:Token: None 2023-11-02T18:00:46.144 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.144 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iUlDY1WaSlLESfaT3Rrje7U239Q= 2023-11-02T18:00:46.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iUlDY1WaSlLESfaT3Rrje7U239Q=', 'Content-Length': '0'} 2023-11-02T18:00:46.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005eacde11539c8d44-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.149 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.149 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.149 DEBUG:boto:path=/ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.150 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.150 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.151 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.151 DEBUG:boto:Data: 2023-11-02T18:00:46.152 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.152 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.152 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.153 DEBUG:boto:Params: {} 2023-11-02T18:00:46.153 DEBUG:boto:Token: None 2023-11-02T18:00:46.154 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.154 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iUlDY1WaSlLESfaT3Rrje7U239Q= 2023-11-02T18:00:46.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iUlDY1WaSlLESfaT3Rrje7U239Q=', 'Content-Length': '0'} 2023-11-02T18:00:46.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc71c0492e4b06d3-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:46.159 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.159 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.160 DEBUG:boto:path=/ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.160 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.161 DEBUG:boto:Method: GET 2023-11-02T18:00:46.161 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.161 DEBUG:boto:Data: 2023-11-02T18:00:46.162 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.162 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.163 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.163 DEBUG:boto:Params: {} 2023-11-02T18:00:46.163 DEBUG:boto:Token: None 2023-11-02T18:00:46.164 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.164 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FLbcLRTGSlpqtOS28pMg6obnR1o= 2023-11-02T18:00:46.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FLbcLRTGSlpqtOS28pMg6obnR1o=', 'Content-Length': '0'} 2023-11-02T18:00:46.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ee94340ecd86734-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.210 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.211 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.211 DEBUG:boto:path=/ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.212 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.212 DEBUG:boto:Method: GET 2023-11-02T18:00:46.213 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.213 DEBUG:boto:Data: 2023-11-02T18:00:46.214 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.214 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.217 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.218 DEBUG:boto:Params: {} 2023-11-02T18:00:46.218 DEBUG:boto:Token: None 2023-11-02T18:00:46.219 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT 2023-11-02T18:00:46.220 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FLbcLRTGSlpqtOS28pMg6obnR1o= 2023-11-02T18:00:46.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FLbcLRTGSlpqtOS28pMg6obnR1o=', 'Content-Length': '0'} 2023-11-02T18:00:46.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Uwj.T0NKxKge-5JRpZgrYHyZpK8hFyT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000820d95f28babc72-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.224 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:46.225 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:46.225 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.226 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.226 DEBUG:boto:Method: GET 2023-11-02T18:00:46.226 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:46.227 DEBUG:boto:Data: 2023-11-02T18:00:46.227 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.228 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.228 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.229 DEBUG:boto:Params: {} 2023-11-02T18:00:46.229 DEBUG:boto:Token: None 2023-11-02T18:00:46.229 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/ 2023-11-02T18:00:46.230 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc= 2023-11-02T18:00:46.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc=', 'Content-Length': '0'} 2023-11-02T18:00:46.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062f9c7bf6d6c27f9-006543e3ce-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.237 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.238 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:46.238 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:46.239 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.239 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.240 DEBUG:boto:Method: GET 2023-11-02T18:00:46.240 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:46.241 DEBUG:boto:Data: 2023-11-02T18:00:46.241 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.242 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.242 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.243 DEBUG:boto:Params: {} 2023-11-02T18:00:46.244 DEBUG:boto:Token: None 2023-11-02T18:00:46.245 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/ 2023-11-02T18:00:46.245 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc= 2023-11-02T18:00:46.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc=', 'Content-Length': '0'} 2023-11-02T18:00:46.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f0df2b27d39d030-006543e3ce-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.294 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.295 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.296 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.296 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.296 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:46.297 DEBUG:boto:Data: 2023-11-02T18:00:46.297 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.298 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.298 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.299 DEBUG:boto:Params: {} 2023-11-02T18:00:46.300 DEBUG:boto:Token: None 2023-11-02T18:00:46.301 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj 2023-11-02T18:00:46.301 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI= 2023-11-02T18:00:46.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI=', 'Content-Length': '0'} 2023-11-02T18:00:46.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007309a608b0c21632-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.308 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.309 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.309 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.310 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:46.310 DEBUG:boto:Data: 2023-11-02T18:00:46.311 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.311 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.311 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.312 DEBUG:boto:Params: {} 2023-11-02T18:00:46.313 DEBUG:boto:Token: None 2023-11-02T18:00:46.313 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj 2023-11-02T18:00:46.314 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI= 2023-11-02T18:00:46.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI=', 'Content-Length': '0'} 2023-11-02T18:00:46.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000471a587688e38548-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:46.322 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.322 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.323 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.323 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.324 DEBUG:boto:Method: GET 2023-11-02T18:00:46.324 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.325 DEBUG:boto:Data: 2023-11-02T18:00:46.325 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.326 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.326 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.327 DEBUG:boto:Params: {} 2023-11-02T18:00:46.327 DEBUG:boto:Token: None 2023-11-02T18:00:46.328 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.328 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI= 2023-11-02T18:00:46.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI=', 'Content-Length': '0'} 2023-11-02T18:00:46.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000aaa6a22c992a20f-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.333 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.333 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.334 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.334 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.335 DEBUG:boto:Method: GET 2023-11-02T18:00:46.335 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.336 DEBUG:boto:Data: 2023-11-02T18:00:46.336 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.337 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.337 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.338 DEBUG:boto:Params: {} 2023-11-02T18:00:46.338 DEBUG:boto:Token: None 2023-11-02T18:00:46.339 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.339 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI= 2023-11-02T18:00:46.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI=', 'Content-Length': '0'} 2023-11-02T18:00:46.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010f05f526643185f-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.344 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:46.344 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:46.345 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.345 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.346 DEBUG:boto:Method: GET 2023-11-02T18:00:46.346 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:46.347 DEBUG:boto:Data: 2023-11-02T18:00:46.347 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.348 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.348 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.349 DEBUG:boto:Params: {} 2023-11-02T18:00:46.349 DEBUG:boto:Token: None 2023-11-02T18:00:46.350 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/ 2023-11-02T18:00:46.350 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc= 2023-11-02T18:00:46.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc=', 'Content-Length': '0'} 2023-11-02T18:00:46.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0d09bff7c23eaf9-006543e3ce-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.398 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.399 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:46.400 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:46.400 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.401 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.401 DEBUG:boto:Method: GET 2023-11-02T18:00:46.402 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:46.402 DEBUG:boto:Data: 2023-11-02T18:00:46.403 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.404 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.404 DEBUG:boto:Params: {} 2023-11-02T18:00:46.405 DEBUG:boto:Token: None 2023-11-02T18:00:46.405 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/ 2023-11-02T18:00:46.406 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc= 2023-11-02T18:00:46.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc=', 'Content-Length': '0'} 2023-11-02T18:00:46.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019d5e699ecd5d2aa-006543e3ce-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.413 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:46.414 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.414 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.415 DEBUG:boto:path=/ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.415 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.416 DEBUG:boto:Method: GET 2023-11-02T18:00:46.416 DEBUG:boto:Path: /ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.416 DEBUG:boto:Data: 2023-11-02T18:00:46.417 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.417 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.418 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.418 DEBUG:boto:Params: {} 2023-11-02T18:00:46.419 DEBUG:boto:Token: None 2023-11-02T18:00:46.419 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.420 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KrxUlDzLed21nV15Uj4zrQmM+mo= 2023-11-02T18:00:46.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KrxUlDzLed21nV15Uj4zrQmM+mo=', 'Content-Length': '0'} 2023-11-02T18:00:46.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fe33b33befc8474-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.426 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.427 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.427 DEBUG:boto:path=/ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.428 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.428 DEBUG:boto:Method: GET 2023-11-02T18:00:46.429 DEBUG:boto:Path: /ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.430 DEBUG:boto:Data: 2023-11-02T18:00:46.430 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.431 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.431 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.432 DEBUG:boto:Params: {} 2023-11-02T18:00:46.432 DEBUG:boto:Token: None 2023-11-02T18:00:46.433 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.433 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KrxUlDzLed21nV15Uj4zrQmM+mo= 2023-11-02T18:00:46.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KrxUlDzLed21nV15Uj4zrQmM+mo=', 'Content-Length': '0'} 2023-11-02T18:00:46.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006aaa313e6c66ed38-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.478 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.479 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.479 DEBUG:boto:path=/ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.480 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.480 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.480 DEBUG:boto:Path: /ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.481 DEBUG:boto:Data: 2023-11-02T18:00:46.481 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.482 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.482 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.483 DEBUG:boto:Params: {} 2023-11-02T18:00:46.483 DEBUG:boto:Token: None 2023-11-02T18:00:46.484 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.484 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PAjJ7TuBVisJQiyGN14DGsQ06SY= 2023-11-02T18:00:46.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PAjJ7TuBVisJQiyGN14DGsQ06SY=', 'Content-Length': '0'} 2023-11-02T18:00:46.488 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b576cc223ef629b-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.489 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.489 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.490 DEBUG:boto:path=/ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.490 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.490 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.491 DEBUG:boto:Path: /ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.491 DEBUG:boto:Data: 2023-11-02T18:00:46.492 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.492 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.493 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.493 DEBUG:boto:Params: {} 2023-11-02T18:00:46.493 DEBUG:boto:Token: None 2023-11-02T18:00:46.494 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.494 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PAjJ7TuBVisJQiyGN14DGsQ06SY= 2023-11-02T18:00:46.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PAjJ7TuBVisJQiyGN14DGsQ06SY=', 'Content-Length': '0'} 2023-11-02T18:00:46.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019d2bf1aceb66af1-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:46.499 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.499 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.500 DEBUG:boto:path=/ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.500 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.501 DEBUG:boto:Method: GET 2023-11-02T18:00:46.501 DEBUG:boto:Path: /ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.501 DEBUG:boto:Data: 2023-11-02T18:00:46.502 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.502 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.503 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.503 DEBUG:boto:Params: {} 2023-11-02T18:00:46.504 DEBUG:boto:Token: None 2023-11-02T18:00:46.504 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.505 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KrxUlDzLed21nV15Uj4zrQmM+mo= 2023-11-02T18:00:46.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KrxUlDzLed21nV15Uj4zrQmM+mo=', 'Content-Length': '0'} 2023-11-02T18:00:46.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9fab0bb2aea26b8-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.550 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.551 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.551 DEBUG:boto:path=/ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.552 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.552 DEBUG:boto:Method: GET 2023-11-02T18:00:46.553 DEBUG:boto:Path: /ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.553 DEBUG:boto:Data: 2023-11-02T18:00:46.554 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.555 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.555 DEBUG:boto:Params: {} 2023-11-02T18:00:46.556 DEBUG:boto:Token: None 2023-11-02T18:00:46.556 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx 2023-11-02T18:00:46.557 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KrxUlDzLed21nV15Uj4zrQmM+mo= 2023-11-02T18:00:46.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KrxUlDzLed21nV15Uj4zrQmM+mo=', 'Content-Length': '0'} 2023-11-02T18:00:46.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'x6dydJf2TZHT6m0PUvIZ2ZGGENUH4Qx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005163681464ffed81-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.562 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:46.563 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:46.563 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.564 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.564 DEBUG:boto:Method: GET 2023-11-02T18:00:46.564 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:46.565 DEBUG:boto:Data: 2023-11-02T18:00:46.565 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.566 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.567 DEBUG:boto:Params: {} 2023-11-02T18:00:46.567 DEBUG:boto:Token: None 2023-11-02T18:00:46.568 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/ 2023-11-02T18:00:46.568 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc= 2023-11-02T18:00:46.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc=', 'Content-Length': '0'} 2023-11-02T18:00:46.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d5ab2c0e5b0ce8c-006543e3ce-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.618 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.619 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:46.620 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:46.620 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.621 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.621 DEBUG:boto:Method: GET 2023-11-02T18:00:46.621 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:46.622 DEBUG:boto:Data: 2023-11-02T18:00:46.622 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.623 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.623 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.624 DEBUG:boto:Params: {} 2023-11-02T18:00:46.624 DEBUG:boto:Token: None 2023-11-02T18:00:46.625 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/ 2023-11-02T18:00:46.625 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc= 2023-11-02T18:00:46.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc=', 'Content-Length': '0'} 2023-11-02T18:00:46.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e815fc1cd66135ee-006543e3ce-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.631 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.632 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.632 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.633 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.633 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:46.634 DEBUG:boto:Data: 2023-11-02T18:00:46.634 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.635 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.635 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.635 DEBUG:boto:Params: {} 2023-11-02T18:00:46.636 DEBUG:boto:Token: None 2023-11-02T18:00:46.637 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj 2023-11-02T18:00:46.637 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI= 2023-11-02T18:00:46.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI=', 'Content-Length': '0'} 2023-11-02T18:00:46.644 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000015ee907f6ae75b8-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.644 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.645 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.645 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.646 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:46.646 DEBUG:boto:Data: 2023-11-02T18:00:46.646 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.647 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.647 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.648 DEBUG:boto:Params: {} 2023-11-02T18:00:46.648 DEBUG:boto:Token: None 2023-11-02T18:00:46.648 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj 2023-11-02T18:00:46.649 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI= 2023-11-02T18:00:46.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI=', 'Content-Length': '0'} 2023-11-02T18:00:46.655 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004141f5579da150c-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:46.656 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.657 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.657 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.658 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.658 DEBUG:boto:Method: GET 2023-11-02T18:00:46.659 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.659 DEBUG:boto:Data: 2023-11-02T18:00:46.659 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.660 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.660 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.661 DEBUG:boto:Params: {} 2023-11-02T18:00:46.661 DEBUG:boto:Token: None 2023-11-02T18:00:46.662 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.662 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI= 2023-11-02T18:00:46.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI=', 'Content-Length': '0'} 2023-11-02T18:00:46.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007dc8511b5d6a4db6-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.667 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.667 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.668 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.668 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.668 DEBUG:boto:Method: GET 2023-11-02T18:00:46.669 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.669 DEBUG:boto:Data: 2023-11-02T18:00:46.670 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.670 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.670 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.671 DEBUG:boto:Params: {} 2023-11-02T18:00:46.671 DEBUG:boto:Token: None 2023-11-02T18:00:46.672 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.672 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI= 2023-11-02T18:00:46.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI=', 'Content-Length': '0'} 2023-11-02T18:00:46.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8c41576bfadeab4-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.718 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:46.719 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:46.719 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.720 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.720 DEBUG:boto:Method: GET 2023-11-02T18:00:46.721 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:46.721 DEBUG:boto:Data: 2023-11-02T18:00:46.721 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.722 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.722 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.723 DEBUG:boto:Params: {} 2023-11-02T18:00:46.723 DEBUG:boto:Token: None 2023-11-02T18:00:46.724 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/ 2023-11-02T18:00:46.724 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc= 2023-11-02T18:00:46.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc=', 'Content-Length': '0'} 2023-11-02T18:00:46.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000178352e6ff62c73-006543e3ce-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.730 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.731 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:46.732 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:46.732 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.732 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.733 DEBUG:boto:Method: GET 2023-11-02T18:00:46.733 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:46.734 DEBUG:boto:Data: 2023-11-02T18:00:46.734 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.735 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.735 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.735 DEBUG:boto:Params: {} 2023-11-02T18:00:46.736 DEBUG:boto:Token: None 2023-11-02T18:00:46.736 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/ 2023-11-02T18:00:46.737 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc= 2023-11-02T18:00:46.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc=', 'Content-Length': '0'} 2023-11-02T18:00:46.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034f973ac764d6eff-006543e3ce-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.743 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:46.744 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.744 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.745 DEBUG:boto:path=/ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.745 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.746 DEBUG:boto:Method: GET 2023-11-02T18:00:46.746 DEBUG:boto:Path: /ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.747 DEBUG:boto:Data: 2023-11-02T18:00:46.747 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.748 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.748 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.748 DEBUG:boto:Params: {} 2023-11-02T18:00:46.749 DEBUG:boto:Token: None 2023-11-02T18:00:46.749 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.750 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xlDG7+R3vnRyjS6+AZ69lW3wYLw= 2023-11-02T18:00:46.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xlDG7+R3vnRyjS6+AZ69lW3wYLw=', 'Content-Length': '0'} 2023-11-02T18:00:46.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe1d72dc864de007-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.794 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.795 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.795 DEBUG:boto:path=/ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.796 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.796 DEBUG:boto:Method: GET 2023-11-02T18:00:46.797 DEBUG:boto:Path: /ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.797 DEBUG:boto:Data: 2023-11-02T18:00:46.798 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.798 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.799 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.799 DEBUG:boto:Params: {} 2023-11-02T18:00:46.800 DEBUG:boto:Token: None 2023-11-02T18:00:46.800 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xlDG7+R3vnRyjS6+AZ69lW3wYLw= 2023-11-02T18:00:46.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xlDG7+R3vnRyjS6+AZ69lW3wYLw=', 'Content-Length': '0'} 2023-11-02T18:00:46.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000378647b3438250a3-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.805 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.806 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.806 DEBUG:boto:path=/ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.807 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.807 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.807 DEBUG:boto:Path: /ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.808 DEBUG:boto:Data: 2023-11-02T18:00:46.808 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.809 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.809 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.809 DEBUG:boto:Params: {} 2023-11-02T18:00:46.810 DEBUG:boto:Token: None 2023-11-02T18:00:46.810 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.811 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IeJi4R4/gGfBhlYXNzC9SrFtG6Q= 2023-11-02T18:00:46.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IeJi4R4/gGfBhlYXNzC9SrFtG6Q=', 'Content-Length': '0'} 2023-11-02T18:00:46.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004463d1cbd9371411-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.815 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.815 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.816 DEBUG:boto:path=/ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.816 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.816 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.817 DEBUG:boto:Path: /ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.817 DEBUG:boto:Data: 2023-11-02T18:00:46.818 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.818 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.818 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.819 DEBUG:boto:Params: {} 2023-11-02T18:00:46.819 DEBUG:boto:Token: None 2023-11-02T18:00:46.820 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.820 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IeJi4R4/gGfBhlYXNzC9SrFtG6Q= 2023-11-02T18:00:46.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IeJi4R4/gGfBhlYXNzC9SrFtG6Q=', 'Content-Length': '0'} 2023-11-02T18:00:46.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071f77ac6155e8c60-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:46.825 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.826 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.826 DEBUG:boto:path=/ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.827 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.827 DEBUG:boto:Method: GET 2023-11-02T18:00:46.827 DEBUG:boto:Path: /ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.828 DEBUG:boto:Data: 2023-11-02T18:00:46.828 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.829 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.829 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.830 DEBUG:boto:Params: {} 2023-11-02T18:00:46.830 DEBUG:boto:Token: None 2023-11-02T18:00:46.831 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.833 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xlDG7+R3vnRyjS6+AZ69lW3wYLw= 2023-11-02T18:00:46.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xlDG7+R3vnRyjS6+AZ69lW3wYLw=', 'Content-Length': '0'} 2023-11-02T18:00:46.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010d927617dec999d-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.838 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.838 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.839 DEBUG:boto:path=/ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.839 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.840 DEBUG:boto:Method: GET 2023-11-02T18:00:46.840 DEBUG:boto:Path: /ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.841 DEBUG:boto:Data: 2023-11-02T18:00:46.841 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.842 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.842 DEBUG:boto:Params: {} 2023-11-02T18:00:46.843 DEBUG:boto:Token: None 2023-11-02T18:00:46.843 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK 2023-11-02T18:00:46.844 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xlDG7+R3vnRyjS6+AZ69lW3wYLw= 2023-11-02T18:00:46.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xlDG7+R3vnRyjS6+AZ69lW3wYLw=', 'Content-Length': '0'} 2023-11-02T18:00:46.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '5FPt.NTakDYr3cN5mfK-9z0Ba.dVchK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082fd2af06a298b46-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.890 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:46.891 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:46.891 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.892 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.892 DEBUG:boto:Method: GET 2023-11-02T18:00:46.893 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:46.893 DEBUG:boto:Data: 2023-11-02T18:00:46.894 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.894 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.894 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.895 DEBUG:boto:Params: {} 2023-11-02T18:00:46.895 DEBUG:boto:Token: None 2023-11-02T18:00:46.896 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/ 2023-11-02T18:00:46.896 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc= 2023-11-02T18:00:46.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc=', 'Content-Length': '0'} 2023-11-02T18:00:46.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007cfe28da3062f10-006543e3ce-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.904 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.904 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:46.905 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:46.905 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.906 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:46.906 DEBUG:boto:Method: GET 2023-11-02T18:00:46.907 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:46.907 DEBUG:boto:Data: 2023-11-02T18:00:46.907 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.908 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.908 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.909 DEBUG:boto:Params: {} 2023-11-02T18:00:46.909 DEBUG:boto:Token: None 2023-11-02T18:00:46.910 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/ 2023-11-02T18:00:46.910 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc= 2023-11-02T18:00:46.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QI+Ve133lVDwmH8+OTaO65DJVc=', 'Content-Length': '0'} 2023-11-02T18:00:46.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2e39f63305e804c-006543e3ce-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.916 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:46.917 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.917 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.918 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.918 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:46.919 DEBUG:boto:Data: 2023-11-02T18:00:46.919 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.919 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.920 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.920 DEBUG:boto:Params: {} 2023-11-02T18:00:46.921 DEBUG:boto:Token: None 2023-11-02T18:00:46.921 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj 2023-11-02T18:00:46.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI= 2023-11-02T18:00:46.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI=', 'Content-Length': '0'} 2023-11-02T18:00:46.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025f8d307cdea80e9-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.929 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.930 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.930 DEBUG:boto:Method: HEAD 2023-11-02T18:00:46.930 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:46.931 DEBUG:boto:Data: 2023-11-02T18:00:46.931 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.932 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.932 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.933 DEBUG:boto:Params: {} 2023-11-02T18:00:46.933 DEBUG:boto:Token: None 2023-11-02T18:00:46.934 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj 2023-11-02T18:00:46.934 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI= 2023-11-02T18:00:46.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9oEP82zrOLsvtGDaRIpyLL7ipZI=', 'Content-Length': '0'} 2023-11-02T18:00:46.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b85d335cfc3c3a2e-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:46.941 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.942 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.942 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.943 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.943 DEBUG:boto:Method: GET 2023-11-02T18:00:46.943 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.944 DEBUG:boto:Data: 2023-11-02T18:00:46.944 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.945 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:46.945 DEBUG:boto:Port: 8001 2023-11-02T18:00:46.946 DEBUG:boto:Params: {} 2023-11-02T18:00:46.946 DEBUG:boto:Token: None 2023-11-02T18:00:46.947 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:46 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.947 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI= 2023-11-02T18:00:46.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+C58pHaYz2aGT9XOW45RlWLt1fI=', 'Content-Length': '0'} 2023-11-02T18:00:46.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4873975e180c479-006543e3ce-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:46.994 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:46.995 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:46.995 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.996 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.996 DEBUG:boto:Method: GET 2023-11-02T18:00:46.997 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:46.997 DEBUG:boto:Data: 2023-11-02T18:00:46.998 DEBUG:boto:Headers: {} 2023-11-02T18:00:46.998 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:46.999 DEBUG:boto:Port: 8000 2023-11-02T18:00:46.999 DEBUG:boto:Params: {} 2023-11-02T18:00:46.999 DEBUG:boto:Token: None 2023-11-02T18:00:47.000 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.001 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:49SLkhaXw0bqm7fqC+egAi8dPq4= 2023-11-02T18:00:47.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:49SLkhaXw0bqm7fqC+egAi8dPq4=', 'Content-Length': '0'} 2023-11-02T18:00:47.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abde899b0ec0d549-006543e3ce-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.046 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:47.047 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:47.048 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.048 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.048 DEBUG:boto:Method: GET 2023-11-02T18:00:47.049 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:47.049 DEBUG:boto:Data: 2023-11-02T18:00:47.050 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.051 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.052 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.052 DEBUG:boto:Params: {} 2023-11-02T18:00:47.053 DEBUG:boto:Token: None 2023-11-02T18:00:47.054 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/ 2023-11-02T18:00:47.054 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk= 2023-11-02T18:00:47.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk=', 'Content-Length': '0'} 2023-11-02T18:00:47.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dda114fbfe5078aa-006543e3cf-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.062 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:47.063 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:47.063 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:47.063 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.064 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.064 DEBUG:boto:Method: GET 2023-11-02T18:00:47.065 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:47.065 DEBUG:boto:Data: 2023-11-02T18:00:47.066 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.066 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.067 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.067 DEBUG:boto:Params: {} 2023-11-02T18:00:47.068 DEBUG:boto:Token: None 2023-11-02T18:00:47.069 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/ 2023-11-02T18:00:47.070 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk= 2023-11-02T18:00:47.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk=', 'Content-Length': '0'} 2023-11-02T18:00:47.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001899ad6ac02a3cd0-006543e3cf-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.118 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:47.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:47.120 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.120 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.120 DEBUG:boto:path=/ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.121 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.121 DEBUG:boto:Method: GET 2023-11-02T18:00:47.122 DEBUG:boto:Path: /ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.122 DEBUG:boto:Data: 2023-11-02T18:00:47.123 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.124 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.124 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.125 DEBUG:boto:Params: {} 2023-11-02T18:00:47.125 DEBUG:boto:Token: None 2023-11-02T18:00:47.126 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.127 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g53PsKTamWO9e7ruz75lWBs2KjI= 2023-11-02T18:00:47.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g53PsKTamWO9e7ruz75lWBs2KjI=', 'Content-Length': '0'} 2023-11-02T18:00:47.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8e3109463223762-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.132 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.133 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.133 DEBUG:boto:path=/ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.134 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.135 DEBUG:boto:Method: GET 2023-11-02T18:00:47.135 DEBUG:boto:Path: /ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.135 DEBUG:boto:Data: 2023-11-02T18:00:47.136 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.137 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.138 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.138 DEBUG:boto:Params: {} 2023-11-02T18:00:47.139 DEBUG:boto:Token: None 2023-11-02T18:00:47.140 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g53PsKTamWO9e7ruz75lWBs2KjI= 2023-11-02T18:00:47.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g53PsKTamWO9e7ruz75lWBs2KjI=', 'Content-Length': '0'} 2023-11-02T18:00:47.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a574334c49ff62d-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.145 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.145 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.146 DEBUG:boto:path=/ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.146 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.147 DEBUG:boto:Method: HEAD 2023-11-02T18:00:47.147 DEBUG:boto:Path: /ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.148 DEBUG:boto:Data: 2023-11-02T18:00:47.148 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.149 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.149 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.149 DEBUG:boto:Params: {} 2023-11-02T18:00:47.150 DEBUG:boto:Token: None 2023-11-02T18:00:47.150 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yEitf/f03JOQRb3sdHhUulD9HMM= 2023-11-02T18:00:47.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yEitf/f03JOQRb3sdHhUulD9HMM=', 'Content-Length': '0'} 2023-11-02T18:00:47.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb216a43d10ca2db-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.155 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.155 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.156 DEBUG:boto:path=/ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.156 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.157 DEBUG:boto:Method: HEAD 2023-11-02T18:00:47.157 DEBUG:boto:Path: /ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.158 DEBUG:boto:Data: 2023-11-02T18:00:47.158 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.159 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.159 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.159 DEBUG:boto:Params: {} 2023-11-02T18:00:47.160 DEBUG:boto:Token: None 2023-11-02T18:00:47.160 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.161 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yEitf/f03JOQRb3sdHhUulD9HMM= 2023-11-02T18:00:47.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yEitf/f03JOQRb3sdHhUulD9HMM=', 'Content-Length': '0'} 2023-11-02T18:00:47.164 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053803b5ec17422b5-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:47.165 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.166 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.166 DEBUG:boto:path=/ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.167 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.167 DEBUG:boto:Method: GET 2023-11-02T18:00:47.167 DEBUG:boto:Path: /ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.168 DEBUG:boto:Data: 2023-11-02T18:00:47.168 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.169 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.169 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.170 DEBUG:boto:Params: {} 2023-11-02T18:00:47.170 DEBUG:boto:Token: None 2023-11-02T18:00:47.171 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.172 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g53PsKTamWO9e7ruz75lWBs2KjI= 2023-11-02T18:00:47.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g53PsKTamWO9e7ruz75lWBs2KjI=', 'Content-Length': '0'} 2023-11-02T18:00:47.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7f5f84d2b2c8ada-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.218 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.219 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.219 DEBUG:boto:path=/ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.220 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.220 DEBUG:boto:Method: GET 2023-11-02T18:00:47.221 DEBUG:boto:Path: /ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.221 DEBUG:boto:Data: 2023-11-02T18:00:47.222 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.222 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.223 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.223 DEBUG:boto:Params: {} 2023-11-02T18:00:47.223 DEBUG:boto:Token: None 2023-11-02T18:00:47.224 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4 2023-11-02T18:00:47.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g53PsKTamWO9e7ruz75lWBs2KjI= 2023-11-02T18:00:47.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g53PsKTamWO9e7ruz75lWBs2KjI=', 'Content-Length': '0'} 2023-11-02T18:00:47.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'CdMbf6uJmKuIWSXwenQ3IZNQU.FGVh4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9b1202fb0af5e12-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.229 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:47.229 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:47.230 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.230 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.231 DEBUG:boto:Method: GET 2023-11-02T18:00:47.231 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:47.232 DEBUG:boto:Data: 2023-11-02T18:00:47.232 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.233 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.233 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.234 DEBUG:boto:Params: {} 2023-11-02T18:00:47.234 DEBUG:boto:Token: None 2023-11-02T18:00:47.235 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/ 2023-11-02T18:00:47.235 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk= 2023-11-02T18:00:47.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk=', 'Content-Length': '0'} 2023-11-02T18:00:47.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c6c96bd4888f745-006543e3cf-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.242 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:47.243 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:47.244 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:47.245 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.246 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.246 DEBUG:boto:Method: GET 2023-11-02T18:00:47.247 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:47.248 DEBUG:boto:Data: 2023-11-02T18:00:47.249 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.249 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.250 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.250 DEBUG:boto:Params: {} 2023-11-02T18:00:47.251 DEBUG:boto:Token: None 2023-11-02T18:00:47.251 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/ 2023-11-02T18:00:47.252 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk= 2023-11-02T18:00:47.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk=', 'Content-Length': '0'} 2023-11-02T18:00:47.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000528ff9cd0a73778a-006543e3cf-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.298 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:47.299 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.299 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.300 DEBUG:boto:Method: HEAD 2023-11-02T18:00:47.300 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:47.301 DEBUG:boto:Data: 2023-11-02T18:00:47.301 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.302 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.302 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.302 DEBUG:boto:Params: {} 2023-11-02T18:00:47.303 DEBUG:boto:Token: None 2023-11-02T18:00:47.303 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj 2023-11-02T18:00:47.304 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t2nLhEUs4GvOnybSGdJyx9TRPfo= 2023-11-02T18:00:47.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t2nLhEUs4GvOnybSGdJyx9TRPfo=', 'Content-Length': '0'} 2023-11-02T18:00:47.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000595791c522e995d4-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.311 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.312 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.312 DEBUG:boto:Method: HEAD 2023-11-02T18:00:47.313 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:47.313 DEBUG:boto:Data: 2023-11-02T18:00:47.314 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.314 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.314 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.315 DEBUG:boto:Params: {} 2023-11-02T18:00:47.315 DEBUG:boto:Token: None 2023-11-02T18:00:47.316 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj 2023-11-02T18:00:47.316 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t2nLhEUs4GvOnybSGdJyx9TRPfo= 2023-11-02T18:00:47.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t2nLhEUs4GvOnybSGdJyx9TRPfo=', 'Content-Length': '0'} 2023-11-02T18:00:47.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1813546cd9f2cba-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:47.324 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.325 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.325 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.326 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.326 DEBUG:boto:Method: GET 2023-11-02T18:00:47.327 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.327 DEBUG:boto:Data: 2023-11-02T18:00:47.328 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.329 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.329 DEBUG:boto:Params: {} 2023-11-02T18:00:47.329 DEBUG:boto:Token: None 2023-11-02T18:00:47.330 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.331 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:49SLkhaXw0bqm7fqC+egAi8dPq4= 2023-11-02T18:00:47.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:49SLkhaXw0bqm7fqC+egAi8dPq4=', 'Content-Length': '0'} 2023-11-02T18:00:47.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000412936bf6ceed9a4-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.382 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.383 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.383 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.384 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.384 DEBUG:boto:Method: GET 2023-11-02T18:00:47.385 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.385 DEBUG:boto:Data: 2023-11-02T18:00:47.386 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.386 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.387 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.388 DEBUG:boto:Params: {} 2023-11-02T18:00:47.388 DEBUG:boto:Token: None 2023-11-02T18:00:47.389 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.390 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:49SLkhaXw0bqm7fqC+egAi8dPq4= 2023-11-02T18:00:47.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:49SLkhaXw0bqm7fqC+egAi8dPq4=', 'Content-Length': '0'} 2023-11-02T18:00:47.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a50709164db198d2-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.442 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:47.445 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:47.446 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.446 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.447 DEBUG:boto:Method: GET 2023-11-02T18:00:47.448 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:47.448 DEBUG:boto:Data: 2023-11-02T18:00:47.449 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.450 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.450 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.451 DEBUG:boto:Params: {} 2023-11-02T18:00:47.452 DEBUG:boto:Token: None 2023-11-02T18:00:47.453 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/ 2023-11-02T18:00:47.453 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk= 2023-11-02T18:00:47.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk=', 'Content-Length': '0'} 2023-11-02T18:00:47.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab5aa34fe9c7e214-006543e3cf-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.461 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:47.462 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:47.462 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:47.463 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.463 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.463 DEBUG:boto:Method: GET 2023-11-02T18:00:47.464 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:47.464 DEBUG:boto:Data: 2023-11-02T18:00:47.465 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.465 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.466 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.466 DEBUG:boto:Params: {} 2023-11-02T18:00:47.467 DEBUG:boto:Token: None 2023-11-02T18:00:47.467 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/ 2023-11-02T18:00:47.468 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk= 2023-11-02T18:00:47.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk=', 'Content-Length': '0'} 2023-11-02T18:00:47.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000977b4044db8649de-006543e3cf-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.474 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:47.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:47.476 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.476 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.476 DEBUG:boto:path=/ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.477 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.477 DEBUG:boto:Method: GET 2023-11-02T18:00:47.478 DEBUG:boto:Path: /ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.478 DEBUG:boto:Data: 2023-11-02T18:00:47.479 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.479 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.480 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.480 DEBUG:boto:Params: {} 2023-11-02T18:00:47.480 DEBUG:boto:Token: None 2023-11-02T18:00:47.481 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.481 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:POj+TLxiBxrxcZ2XDEcCqfg0IZ8= 2023-11-02T18:00:47.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:POj+TLxiBxrxcZ2XDEcCqfg0IZ8=', 'Content-Length': '0'} 2023-11-02T18:00:47.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d19588f29a511510-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.485 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.486 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.486 DEBUG:boto:path=/ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.487 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.487 DEBUG:boto:Method: GET 2023-11-02T18:00:47.488 DEBUG:boto:Path: /ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.488 DEBUG:boto:Data: 2023-11-02T18:00:47.488 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.489 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.489 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.490 DEBUG:boto:Params: {} 2023-11-02T18:00:47.490 DEBUG:boto:Token: None 2023-11-02T18:00:47.491 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.491 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:POj+TLxiBxrxcZ2XDEcCqfg0IZ8= 2023-11-02T18:00:47.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:POj+TLxiBxrxcZ2XDEcCqfg0IZ8=', 'Content-Length': '0'} 2023-11-02T18:00:47.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa0bffe7cb6e2b07-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.538 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.539 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.539 DEBUG:boto:path=/ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.540 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.540 DEBUG:boto:Method: HEAD 2023-11-02T18:00:47.541 DEBUG:boto:Path: /ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.541 DEBUG:boto:Data: 2023-11-02T18:00:47.542 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.542 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.543 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.543 DEBUG:boto:Params: {} 2023-11-02T18:00:47.543 DEBUG:boto:Token: None 2023-11-02T18:00:47.544 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bLUVPNa2Yfq372eThVySqkHhKss= 2023-11-02T18:00:47.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bLUVPNa2Yfq372eThVySqkHhKss=', 'Content-Length': '0'} 2023-11-02T18:00:47.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009254a9759f327e1a-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.548 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.549 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.549 DEBUG:boto:path=/ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.549 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.550 DEBUG:boto:Method: HEAD 2023-11-02T18:00:47.550 DEBUG:boto:Path: /ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.551 DEBUG:boto:Data: 2023-11-02T18:00:47.551 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.552 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.552 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.552 DEBUG:boto:Params: {} 2023-11-02T18:00:47.553 DEBUG:boto:Token: None 2023-11-02T18:00:47.554 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.554 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bLUVPNa2Yfq372eThVySqkHhKss= 2023-11-02T18:00:47.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bLUVPNa2Yfq372eThVySqkHhKss=', 'Content-Length': '0'} 2023-11-02T18:00:47.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000144ee5f33e7a695e-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:47.560 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.561 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.562 DEBUG:boto:path=/ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.562 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.563 DEBUG:boto:Method: GET 2023-11-02T18:00:47.563 DEBUG:boto:Path: /ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.564 DEBUG:boto:Data: 2023-11-02T18:00:47.564 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.565 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.565 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.566 DEBUG:boto:Params: {} 2023-11-02T18:00:47.566 DEBUG:boto:Token: None 2023-11-02T18:00:47.567 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.567 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:POj+TLxiBxrxcZ2XDEcCqfg0IZ8= 2023-11-02T18:00:47.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:POj+TLxiBxrxcZ2XDEcCqfg0IZ8=', 'Content-Length': '0'} 2023-11-02T18:00:47.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f4be7e9c39f1f93-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.614 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.615 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.616 DEBUG:boto:path=/ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.617 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.618 DEBUG:boto:Method: GET 2023-11-02T18:00:47.618 DEBUG:boto:Path: /ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.619 DEBUG:boto:Data: 2023-11-02T18:00:47.620 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.621 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.622 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.622 DEBUG:boto:Params: {} 2023-11-02T18:00:47.623 DEBUG:boto:Token: None 2023-11-02T18:00:47.624 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE 2023-11-02T18:00:47.625 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:POj+TLxiBxrxcZ2XDEcCqfg0IZ8= 2023-11-02T18:00:47.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:POj+TLxiBxrxcZ2XDEcCqfg0IZ8=', 'Content-Length': '0'} 2023-11-02T18:00:47.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'juGmB9Tk0n1pDhg1QtaLQuoFjN.7mdE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffa4d0853c5c6f73-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.631 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:47.632 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:47.633 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.633 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.634 DEBUG:boto:Method: GET 2023-11-02T18:00:47.635 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:47.636 DEBUG:boto:Data: 2023-11-02T18:00:47.636 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.638 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.639 DEBUG:boto:Params: {} 2023-11-02T18:00:47.640 DEBUG:boto:Token: None 2023-11-02T18:00:47.640 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/ 2023-11-02T18:00:47.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk= 2023-11-02T18:00:47.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk=', 'Content-Length': '0'} 2023-11-02T18:00:47.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eeea783d0b6c299-006543e3cf-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.690 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:47.691 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:47.692 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:47.692 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.692 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.693 DEBUG:boto:Method: GET 2023-11-02T18:00:47.693 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:47.694 DEBUG:boto:Data: 2023-11-02T18:00:47.694 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.695 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.695 DEBUG:boto:Params: {} 2023-11-02T18:00:47.696 DEBUG:boto:Token: None 2023-11-02T18:00:47.696 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/ 2023-11-02T18:00:47.697 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk= 2023-11-02T18:00:47.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk=', 'Content-Length': '0'} 2023-11-02T18:00:47.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fc006075097e679-006543e3cf-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.704 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:47.704 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.705 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.705 DEBUG:boto:Method: HEAD 2023-11-02T18:00:47.705 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:47.706 DEBUG:boto:Data: 2023-11-02T18:00:47.706 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.707 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.707 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.707 DEBUG:boto:Params: {} 2023-11-02T18:00:47.708 DEBUG:boto:Token: None 2023-11-02T18:00:47.708 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj 2023-11-02T18:00:47.709 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t2nLhEUs4GvOnybSGdJyx9TRPfo= 2023-11-02T18:00:47.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t2nLhEUs4GvOnybSGdJyx9TRPfo=', 'Content-Length': '0'} 2023-11-02T18:00:47.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f095b583a72c8b6c-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.716 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.717 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.718 DEBUG:boto:Method: HEAD 2023-11-02T18:00:47.718 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:47.718 DEBUG:boto:Data: 2023-11-02T18:00:47.719 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.719 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.720 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.720 DEBUG:boto:Params: {} 2023-11-02T18:00:47.721 DEBUG:boto:Token: None 2023-11-02T18:00:47.721 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj 2023-11-02T18:00:47.722 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t2nLhEUs4GvOnybSGdJyx9TRPfo= 2023-11-02T18:00:47.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t2nLhEUs4GvOnybSGdJyx9TRPfo=', 'Content-Length': '0'} 2023-11-02T18:00:47.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005659b700de9858f-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:47.729 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.730 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.730 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.730 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.731 DEBUG:boto:Method: GET 2023-11-02T18:00:47.732 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.732 DEBUG:boto:Data: 2023-11-02T18:00:47.732 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.733 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.733 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.734 DEBUG:boto:Params: {} 2023-11-02T18:00:47.734 DEBUG:boto:Token: None 2023-11-02T18:00:47.735 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.735 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:49SLkhaXw0bqm7fqC+egAi8dPq4= 2023-11-02T18:00:47.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:49SLkhaXw0bqm7fqC+egAi8dPq4=', 'Content-Length': '0'} 2023-11-02T18:00:47.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd6dcdfc1228e20c-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.740 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.740 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.741 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.741 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.741 DEBUG:boto:Method: GET 2023-11-02T18:00:47.742 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.743 DEBUG:boto:Data: 2023-11-02T18:00:47.743 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.744 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.744 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.744 DEBUG:boto:Params: {} 2023-11-02T18:00:47.745 DEBUG:boto:Token: None 2023-11-02T18:00:47.745 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:47.746 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:49SLkhaXw0bqm7fqC+egAi8dPq4= 2023-11-02T18:00:47.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:49SLkhaXw0bqm7fqC+egAi8dPq4=', 'Content-Length': '0'} 2023-11-02T18:00:47.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7cd1ee1181ad9c2-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.790 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:47.791 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:47.791 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.792 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.792 DEBUG:boto:Method: GET 2023-11-02T18:00:47.793 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:47.793 DEBUG:boto:Data: 2023-11-02T18:00:47.794 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.794 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.795 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.795 DEBUG:boto:Params: {} 2023-11-02T18:00:47.796 DEBUG:boto:Token: None 2023-11-02T18:00:47.796 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/ 2023-11-02T18:00:47.797 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk= 2023-11-02T18:00:47.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk=', 'Content-Length': '0'} 2023-11-02T18:00:47.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b4b6a847b5102a2-006543e3cf-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.804 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:47.805 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:47.805 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:47.806 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.806 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:47.807 DEBUG:boto:Method: GET 2023-11-02T18:00:47.807 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:47.807 DEBUG:boto:Data: 2023-11-02T18:00:47.808 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.808 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.809 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.809 DEBUG:boto:Params: {} 2023-11-02T18:00:47.810 DEBUG:boto:Token: None 2023-11-02T18:00:47.810 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/ 2023-11-02T18:00:47.811 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk= 2023-11-02T18:00:47.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:irqiEmhh7Dbnh6HpErBHsspNtwk=', 'Content-Length': '0'} 2023-11-02T18:00:47.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da5a0552aec27374-006543e3cf-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.858 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:47.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:47.860 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.860 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.861 DEBUG:boto:path=/ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.861 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.862 DEBUG:boto:Method: GET 2023-11-02T18:00:47.862 DEBUG:boto:Path: /ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.863 DEBUG:boto:Data: 2023-11-02T18:00:47.863 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.864 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.864 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.865 DEBUG:boto:Params: {} 2023-11-02T18:00:47.865 DEBUG:boto:Token: None 2023-11-02T18:00:47.866 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.867 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uVja354ljdStoV2ZDhV82epiP9o= 2023-11-02T18:00:47.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uVja354ljdStoV2ZDhV82epiP9o=', 'Content-Length': '0'} 2023-11-02T18:00:47.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ca16d0ea179fb6b-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.871 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.871 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.872 DEBUG:boto:path=/ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.872 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.873 DEBUG:boto:Method: GET 2023-11-02T18:00:47.873 DEBUG:boto:Path: /ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.874 DEBUG:boto:Data: 2023-11-02T18:00:47.874 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.875 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.875 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.876 DEBUG:boto:Params: {} 2023-11-02T18:00:47.876 DEBUG:boto:Token: None 2023-11-02T18:00:47.877 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uVja354ljdStoV2ZDhV82epiP9o= 2023-11-02T18:00:47.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uVja354ljdStoV2ZDhV82epiP9o=', 'Content-Length': '0'} 2023-11-02T18:00:47.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001452db2ac61ab1ed-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.882 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.882 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.883 DEBUG:boto:path=/ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.883 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.883 DEBUG:boto:Method: HEAD 2023-11-02T18:00:47.884 DEBUG:boto:Path: /ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.884 DEBUG:boto:Data: 2023-11-02T18:00:47.885 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.885 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.886 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.886 DEBUG:boto:Params: {} 2023-11-02T18:00:47.887 DEBUG:boto:Token: None 2023-11-02T18:00:47.887 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.888 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eJrXoX1eLaKBHYffmW0OodrCKP0= 2023-11-02T18:00:47.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eJrXoX1eLaKBHYffmW0OodrCKP0=', 'Content-Length': '0'} 2023-11-02T18:00:47.891 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075ce5832a4c9172d-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.892 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.892 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.892 DEBUG:boto:path=/ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.893 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.893 DEBUG:boto:Method: HEAD 2023-11-02T18:00:47.894 DEBUG:boto:Path: /ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.894 DEBUG:boto:Data: 2023-11-02T18:00:47.895 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.895 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.896 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.896 DEBUG:boto:Params: {} 2023-11-02T18:00:47.897 DEBUG:boto:Token: None 2023-11-02T18:00:47.897 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.897 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eJrXoX1eLaKBHYffmW0OodrCKP0= 2023-11-02T18:00:47.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eJrXoX1eLaKBHYffmW0OodrCKP0=', 'Content-Length': '0'} 2023-11-02T18:00:47.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ca7498ee2ffcf60-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:47.902 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.902 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.903 DEBUG:boto:path=/ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.903 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.904 DEBUG:boto:Method: GET 2023-11-02T18:00:47.904 DEBUG:boto:Path: /ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.905 DEBUG:boto:Data: 2023-11-02T18:00:47.905 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:47.906 DEBUG:boto:Port: 8001 2023-11-02T18:00:47.907 DEBUG:boto:Params: {} 2023-11-02T18:00:47.907 DEBUG:boto:Token: None 2023-11-02T18:00:47.908 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.908 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uVja354ljdStoV2ZDhV82epiP9o= 2023-11-02T18:00:47.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uVja354ljdStoV2ZDhV82epiP9o=', 'Content-Length': '0'} 2023-11-02T18:00:47.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000007f42d5c3fcd45f-006543e3cf-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:47.954 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:47.955 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:47.955 DEBUG:boto:path=/ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.956 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.956 DEBUG:boto:Method: GET 2023-11-02T18:00:47.957 DEBUG:boto:Path: /ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.957 DEBUG:boto:Data: 2023-11-02T18:00:47.958 DEBUG:boto:Headers: {} 2023-11-02T18:00:47.958 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:47.959 DEBUG:boto:Port: 8000 2023-11-02T18:00:47.959 DEBUG:boto:Params: {} 2023-11-02T18:00:47.960 DEBUG:boto:Token: None 2023-11-02T18:00:47.961 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:47 GMT /ibtljn-13/obj?versionId=lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr 2023-11-02T18:00:47.961 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uVja354ljdStoV2ZDhV82epiP9o= 2023-11-02T18:00:47.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uVja354ljdStoV2ZDhV82epiP9o=', 'Content-Length': '0'} 2023-11-02T18:00:47.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'lHwkVsQ5SjaueCyARyAbGWVEKo0Y8sr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000256a3e6b2ea7759a-006543e3cf-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.010 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:48.011 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:48.011 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.012 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.012 DEBUG:boto:Method: GET 2023-11-02T18:00:48.013 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:48.013 DEBUG:boto:Data: 2023-11-02T18:00:48.014 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.014 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.015 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.015 DEBUG:boto:Params: {} 2023-11-02T18:00:48.016 DEBUG:boto:Token: None 2023-11-02T18:00:48.016 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/ 2023-11-02T18:00:48.017 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY= 2023-11-02T18:00:48.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY=', 'Content-Length': '0'} 2023-11-02T18:00:48.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c4cf397eee58034-006543e3d0-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.023 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:48.024 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:48.025 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:48.025 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.025 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.026 DEBUG:boto:Method: GET 2023-11-02T18:00:48.027 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:48.027 DEBUG:boto:Data: 2023-11-02T18:00:48.027 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.028 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.028 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.029 DEBUG:boto:Params: {} 2023-11-02T18:00:48.030 DEBUG:boto:Token: None 2023-11-02T18:00:48.030 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/ 2023-11-02T18:00:48.031 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY= 2023-11-02T18:00:48.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY=', 'Content-Length': '0'} 2023-11-02T18:00:48.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e8bee44af60f266-006543e3d0-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.078 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:48.079 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.080 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.080 DEBUG:boto:Method: HEAD 2023-11-02T18:00:48.081 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:48.081 DEBUG:boto:Data: 2023-11-02T18:00:48.082 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.082 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.083 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.083 DEBUG:boto:Params: {} 2023-11-02T18:00:48.084 DEBUG:boto:Token: None 2023-11-02T18:00:48.084 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj 2023-11-02T18:00:48.085 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1kxAKFnLHvzBkojxdw7I4ZmQqMc= 2023-11-02T18:00:48.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1kxAKFnLHvzBkojxdw7I4ZmQqMc=', 'Content-Length': '0'} 2023-11-02T18:00:48.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7fda2f439b7a637-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.092 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.092 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.093 DEBUG:boto:Method: HEAD 2023-11-02T18:00:48.093 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:48.094 DEBUG:boto:Data: 2023-11-02T18:00:48.094 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.095 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.095 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.096 DEBUG:boto:Params: {} 2023-11-02T18:00:48.096 DEBUG:boto:Token: None 2023-11-02T18:00:48.097 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj 2023-11-02T18:00:48.097 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1kxAKFnLHvzBkojxdw7I4ZmQqMc= 2023-11-02T18:00:48.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1kxAKFnLHvzBkojxdw7I4ZmQqMc=', 'Content-Length': '0'} 2023-11-02T18:00:48.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ce237db1bfaddd2-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:48.104 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.105 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.105 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.106 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.106 DEBUG:boto:Method: GET 2023-11-02T18:00:48.107 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.107 DEBUG:boto:Data: 2023-11-02T18:00:48.107 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.108 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.108 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.109 DEBUG:boto:Params: {} 2023-11-02T18:00:48.109 DEBUG:boto:Token: None 2023-11-02T18:00:48.109 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.110 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vzy/aVO+bbpFrScabXpI9OC/RcQ= 2023-11-02T18:00:48.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vzy/aVO+bbpFrScabXpI9OC/RcQ=', 'Content-Length': '0'} 2023-11-02T18:00:48.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d82bdf18efdbb673-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.114 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.114 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.115 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.115 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.116 DEBUG:boto:Method: GET 2023-11-02T18:00:48.116 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.117 DEBUG:boto:Data: 2023-11-02T18:00:48.117 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.118 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.118 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.118 DEBUG:boto:Params: {} 2023-11-02T18:00:48.119 DEBUG:boto:Token: None 2023-11-02T18:00:48.119 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vzy/aVO+bbpFrScabXpI9OC/RcQ= 2023-11-02T18:00:48.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vzy/aVO+bbpFrScabXpI9OC/RcQ=', 'Content-Length': '0'} 2023-11-02T18:00:48.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003457a538fae2a676-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.170 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:48.171 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:48.172 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.172 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.172 DEBUG:boto:Method: GET 2023-11-02T18:00:48.173 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:48.173 DEBUG:boto:Data: 2023-11-02T18:00:48.174 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.174 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.175 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.175 DEBUG:boto:Params: {} 2023-11-02T18:00:48.176 DEBUG:boto:Token: None 2023-11-02T18:00:48.176 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/ 2023-11-02T18:00:48.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY= 2023-11-02T18:00:48.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY=', 'Content-Length': '0'} 2023-11-02T18:00:48.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037f4d52f75ddaaec-006543e3d0-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.184 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:48.185 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:48.186 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:48.186 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.186 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.187 DEBUG:boto:Method: GET 2023-11-02T18:00:48.187 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:48.188 DEBUG:boto:Data: 2023-11-02T18:00:48.188 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.189 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.189 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.190 DEBUG:boto:Params: {} 2023-11-02T18:00:48.190 DEBUG:boto:Token: None 2023-11-02T18:00:48.191 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/ 2023-11-02T18:00:48.191 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY= 2023-11-02T18:00:48.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY=', 'Content-Length': '0'} 2023-11-02T18:00:48.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000946034739ea8a79f-006543e3d0-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.242 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:48.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:48.244 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.244 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.244 DEBUG:boto:path=/ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.245 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.245 DEBUG:boto:Method: GET 2023-11-02T18:00:48.246 DEBUG:boto:Path: /ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.246 DEBUG:boto:Data: 2023-11-02T18:00:48.247 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.247 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.248 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.248 DEBUG:boto:Params: {} 2023-11-02T18:00:48.249 DEBUG:boto:Token: None 2023-11-02T18:00:48.249 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.250 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z0mu2xuh1lfIzdfifsAY+fMr2ZQ= 2023-11-02T18:00:48.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z0mu2xuh1lfIzdfifsAY+fMr2ZQ=', 'Content-Length': '0'} 2023-11-02T18:00:48.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'AorrUYLUrfeGImHVayGt7jrmhCqlMLI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054c15770efa37732-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.254 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.254 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.255 DEBUG:boto:path=/ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.255 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.256 DEBUG:boto:Method: GET 2023-11-02T18:00:48.256 DEBUG:boto:Path: /ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.257 DEBUG:boto:Data: 2023-11-02T18:00:48.257 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.258 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.258 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.259 DEBUG:boto:Params: {} 2023-11-02T18:00:48.259 DEBUG:boto:Token: None 2023-11-02T18:00:48.260 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.260 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z0mu2xuh1lfIzdfifsAY+fMr2ZQ= 2023-11-02T18:00:48.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z0mu2xuh1lfIzdfifsAY+fMr2ZQ=', 'Content-Length': '0'} 2023-11-02T18:00:48.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'AorrUYLUrfeGImHVayGt7jrmhCqlMLI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d067167a7e50952a-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.266 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.266 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.267 DEBUG:boto:path=/ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.267 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.268 DEBUG:boto:Method: HEAD 2023-11-02T18:00:48.268 DEBUG:boto:Path: /ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.269 DEBUG:boto:Data: 2023-11-02T18:00:48.269 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.270 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.270 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.270 DEBUG:boto:Params: {} 2023-11-02T18:00:48.271 DEBUG:boto:Token: None 2023-11-02T18:00:48.271 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.272 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PjF9iKocssnPYXju+EeCipJQvcw= 2023-11-02T18:00:48.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PjF9iKocssnPYXju+EeCipJQvcw=', 'Content-Length': '0'} 2023-11-02T18:00:48.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'AorrUYLUrfeGImHVayGt7jrmhCqlMLI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad3e02f54f733a5b-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.276 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.277 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.277 DEBUG:boto:path=/ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.278 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.278 DEBUG:boto:Method: HEAD 2023-11-02T18:00:48.279 DEBUG:boto:Path: /ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.279 DEBUG:boto:Data: 2023-11-02T18:00:48.279 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.280 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.280 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.281 DEBUG:boto:Params: {} 2023-11-02T18:00:48.281 DEBUG:boto:Token: None 2023-11-02T18:00:48.282 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.283 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PjF9iKocssnPYXju+EeCipJQvcw= 2023-11-02T18:00:48.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PjF9iKocssnPYXju+EeCipJQvcw=', 'Content-Length': '0'} 2023-11-02T18:00:48.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'AorrUYLUrfeGImHVayGt7jrmhCqlMLI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b01ff2f942a13b3-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:48.287 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.288 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.288 DEBUG:boto:path=/ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.289 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.289 DEBUG:boto:Method: GET 2023-11-02T18:00:48.290 DEBUG:boto:Path: /ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.290 DEBUG:boto:Data: 2023-11-02T18:00:48.291 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.291 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.292 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.292 DEBUG:boto:Params: {} 2023-11-02T18:00:48.293 DEBUG:boto:Token: None 2023-11-02T18:00:48.293 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.294 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z0mu2xuh1lfIzdfifsAY+fMr2ZQ= 2023-11-02T18:00:48.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z0mu2xuh1lfIzdfifsAY+fMr2ZQ=', 'Content-Length': '0'} 2023-11-02T18:00:48.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'AorrUYLUrfeGImHVayGt7jrmhCqlMLI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048468917fc8c39ab-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.338 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.339 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.340 DEBUG:boto:path=/ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.341 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.341 DEBUG:boto:Method: GET 2023-11-02T18:00:48.341 DEBUG:boto:Path: /ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.342 DEBUG:boto:Data: 2023-11-02T18:00:48.342 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.343 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.343 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.344 DEBUG:boto:Params: {} 2023-11-02T18:00:48.344 DEBUG:boto:Token: None 2023-11-02T18:00:48.345 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=AorrUYLUrfeGImHVayGt7jrmhCqlMLI 2023-11-02T18:00:48.345 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z0mu2xuh1lfIzdfifsAY+fMr2ZQ= 2023-11-02T18:00:48.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z0mu2xuh1lfIzdfifsAY+fMr2ZQ=', 'Content-Length': '0'} 2023-11-02T18:00:48.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'AorrUYLUrfeGImHVayGt7jrmhCqlMLI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c98e5dbc9d21ed38-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.349 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:48.350 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:48.350 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.351 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.351 DEBUG:boto:Method: GET 2023-11-02T18:00:48.352 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:48.352 DEBUG:boto:Data: 2023-11-02T18:00:48.353 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.353 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.353 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.354 DEBUG:boto:Params: {} 2023-11-02T18:00:48.354 DEBUG:boto:Token: None 2023-11-02T18:00:48.355 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/ 2023-11-02T18:00:48.355 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY= 2023-11-02T18:00:48.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY=', 'Content-Length': '0'} 2023-11-02T18:00:48.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004719be1eb9bb9e3-006543e3d0-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.362 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:48.362 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:48.363 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:48.363 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.364 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.364 DEBUG:boto:Method: GET 2023-11-02T18:00:48.365 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:48.365 DEBUG:boto:Data: 2023-11-02T18:00:48.366 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.366 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.367 DEBUG:boto:Params: {} 2023-11-02T18:00:48.367 DEBUG:boto:Token: None 2023-11-02T18:00:48.368 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/ 2023-11-02T18:00:48.368 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY= 2023-11-02T18:00:48.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY=', 'Content-Length': '0'} 2023-11-02T18:00:48.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077441699d1411918-006543e3d0-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.375 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:48.376 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.376 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.377 DEBUG:boto:Method: HEAD 2023-11-02T18:00:48.377 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:48.378 DEBUG:boto:Data: 2023-11-02T18:00:48.378 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.381 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.382 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.382 DEBUG:boto:Params: {} 2023-11-02T18:00:48.383 DEBUG:boto:Token: None 2023-11-02T18:00:48.383 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj 2023-11-02T18:00:48.384 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1kxAKFnLHvzBkojxdw7I4ZmQqMc= 2023-11-02T18:00:48.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1kxAKFnLHvzBkojxdw7I4ZmQqMc=', 'Content-Length': '0'} 2023-11-02T18:00:48.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000202cc83ab76874b5-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.392 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.393 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.393 DEBUG:boto:Method: HEAD 2023-11-02T18:00:48.394 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:48.395 DEBUG:boto:Data: 2023-11-02T18:00:48.395 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.396 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.396 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.397 DEBUG:boto:Params: {} 2023-11-02T18:00:48.397 DEBUG:boto:Token: None 2023-11-02T18:00:48.398 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj 2023-11-02T18:00:48.399 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1kxAKFnLHvzBkojxdw7I4ZmQqMc= 2023-11-02T18:00:48.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1kxAKFnLHvzBkojxdw7I4ZmQqMc=', 'Content-Length': '0'} 2023-11-02T18:00:48.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a02580a1fbac84a3-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:48.407 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.407 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.407 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.408 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.408 DEBUG:boto:Method: GET 2023-11-02T18:00:48.409 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.409 DEBUG:boto:Data: 2023-11-02T18:00:48.410 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.410 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.411 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.411 DEBUG:boto:Params: {} 2023-11-02T18:00:48.412 DEBUG:boto:Token: None 2023-11-02T18:00:48.412 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.413 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vzy/aVO+bbpFrScabXpI9OC/RcQ= 2023-11-02T18:00:48.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vzy/aVO+bbpFrScabXpI9OC/RcQ=', 'Content-Length': '0'} 2023-11-02T18:00:48.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f08963a1d5b67bb1-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.458 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.459 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.459 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.460 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.460 DEBUG:boto:Method: GET 2023-11-02T18:00:48.461 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.461 DEBUG:boto:Data: 2023-11-02T18:00:48.462 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.463 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.463 DEBUG:boto:Params: {} 2023-11-02T18:00:48.464 DEBUG:boto:Token: None 2023-11-02T18:00:48.464 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vzy/aVO+bbpFrScabXpI9OC/RcQ= 2023-11-02T18:00:48.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vzy/aVO+bbpFrScabXpI9OC/RcQ=', 'Content-Length': '0'} 2023-11-02T18:00:48.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1eda94d0c581af4-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.514 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:48.515 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:48.515 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.516 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.516 DEBUG:boto:Method: GET 2023-11-02T18:00:48.517 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:48.517 DEBUG:boto:Data: 2023-11-02T18:00:48.518 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.518 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.519 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.519 DEBUG:boto:Params: {} 2023-11-02T18:00:48.520 DEBUG:boto:Token: None 2023-11-02T18:00:48.520 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/ 2023-11-02T18:00:48.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY= 2023-11-02T18:00:48.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY=', 'Content-Length': '0'} 2023-11-02T18:00:48.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c5cc364aacf9620-006543e3d0-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.528 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:48.529 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:48.529 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:48.530 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.530 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.531 DEBUG:boto:Method: GET 2023-11-02T18:00:48.531 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:48.532 DEBUG:boto:Data: 2023-11-02T18:00:48.532 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.533 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.533 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.534 DEBUG:boto:Params: {} 2023-11-02T18:00:48.534 DEBUG:boto:Token: None 2023-11-02T18:00:48.535 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/ 2023-11-02T18:00:48.535 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY= 2023-11-02T18:00:48.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY=', 'Content-Length': '0'} 2023-11-02T18:00:48.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cea1e45892ba614-006543e3d0-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.582 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:48.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:48.584 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.584 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.585 DEBUG:boto:path=/ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.585 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.585 DEBUG:boto:Method: GET 2023-11-02T18:00:48.586 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.586 DEBUG:boto:Data: 2023-11-02T18:00:48.587 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.587 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.588 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.588 DEBUG:boto:Params: {} 2023-11-02T18:00:48.589 DEBUG:boto:Token: None 2023-11-02T18:00:48.589 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.590 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXqF6UORPSQ8dk75yst9ArOy7uI= 2023-11-02T18:00:48.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXqF6UORPSQ8dk75yst9ArOy7uI=', 'Content-Length': '0'} 2023-11-02T18:00:48.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c3b9f99ebe9b830-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.594 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.595 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.595 DEBUG:boto:path=/ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.596 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.596 DEBUG:boto:Method: GET 2023-11-02T18:00:48.597 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.597 DEBUG:boto:Data: 2023-11-02T18:00:48.598 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.598 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.599 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.599 DEBUG:boto:Params: {} 2023-11-02T18:00:48.600 DEBUG:boto:Token: None 2023-11-02T18:00:48.600 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXqF6UORPSQ8dk75yst9ArOy7uI= 2023-11-02T18:00:48.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXqF6UORPSQ8dk75yst9ArOy7uI=', 'Content-Length': '0'} 2023-11-02T18:00:48.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f24358227d9b13a-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.606 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.606 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.607 DEBUG:boto:path=/ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.607 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.608 DEBUG:boto:Method: HEAD 2023-11-02T18:00:48.608 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.608 DEBUG:boto:Data: 2023-11-02T18:00:48.609 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.609 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.610 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.610 DEBUG:boto:Params: {} 2023-11-02T18:00:48.611 DEBUG:boto:Token: None 2023-11-02T18:00:48.611 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FyupEeXNFVyxzgiAYi5VItOFWI0= 2023-11-02T18:00:48.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FyupEeXNFVyxzgiAYi5VItOFWI0=', 'Content-Length': '0'} 2023-11-02T18:00:48.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000043d02dff21bbabf-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.616 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.616 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.617 DEBUG:boto:path=/ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.617 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.617 DEBUG:boto:Method: HEAD 2023-11-02T18:00:48.618 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.618 DEBUG:boto:Data: 2023-11-02T18:00:48.619 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.620 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.620 DEBUG:boto:Params: {} 2023-11-02T18:00:48.621 DEBUG:boto:Token: None 2023-11-02T18:00:48.621 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FyupEeXNFVyxzgiAYi5VItOFWI0= 2023-11-02T18:00:48.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FyupEeXNFVyxzgiAYi5VItOFWI0=', 'Content-Length': '0'} 2023-11-02T18:00:48.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000120fa865bbb333a9-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:48.626 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.627 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.627 DEBUG:boto:path=/ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.627 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.628 DEBUG:boto:Method: GET 2023-11-02T18:00:48.628 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.629 DEBUG:boto:Data: 2023-11-02T18:00:48.629 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.630 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.630 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.631 DEBUG:boto:Params: {} 2023-11-02T18:00:48.631 DEBUG:boto:Token: None 2023-11-02T18:00:48.632 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.632 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXqF6UORPSQ8dk75yst9ArOy7uI= 2023-11-02T18:00:48.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXqF6UORPSQ8dk75yst9ArOy7uI=', 'Content-Length': '0'} 2023-11-02T18:00:48.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccb9db5cc8fd20b8-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.636 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.637 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.637 DEBUG:boto:path=/ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.638 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.638 DEBUG:boto:Method: GET 2023-11-02T18:00:48.639 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.639 DEBUG:boto:Data: 2023-11-02T18:00:48.640 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.640 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.640 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.641 DEBUG:boto:Params: {} 2023-11-02T18:00:48.641 DEBUG:boto:Token: None 2023-11-02T18:00:48.642 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK 2023-11-02T18:00:48.642 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXqF6UORPSQ8dk75yst9ArOy7uI= 2023-11-02T18:00:48.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXqF6UORPSQ8dk75yst9ArOy7uI=', 'Content-Length': '0'} 2023-11-02T18:00:48.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Bqp.o7pDoCMqVNZ9auXuTjozZC9BlCK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003693e80cf6807609-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.686 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:48.687 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:48.687 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.688 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.689 DEBUG:boto:Method: GET 2023-11-02T18:00:48.689 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:48.690 DEBUG:boto:Data: 2023-11-02T18:00:48.690 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.691 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.691 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.691 DEBUG:boto:Params: {} 2023-11-02T18:00:48.692 DEBUG:boto:Token: None 2023-11-02T18:00:48.692 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/ 2023-11-02T18:00:48.693 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY= 2023-11-02T18:00:48.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY=', 'Content-Length': '0'} 2023-11-02T18:00:48.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c82e6daa1f8d0ef6-006543e3d0-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.700 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:48.701 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:48.702 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:48.702 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.703 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.704 DEBUG:boto:Method: GET 2023-11-02T18:00:48.704 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:48.705 DEBUG:boto:Data: 2023-11-02T18:00:48.706 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.706 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.706 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.707 DEBUG:boto:Params: {} 2023-11-02T18:00:48.707 DEBUG:boto:Token: None 2023-11-02T18:00:48.708 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/ 2023-11-02T18:00:48.708 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY= 2023-11-02T18:00:48.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY=', 'Content-Length': '0'} 2023-11-02T18:00:48.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b203235d1008315-006543e3d0-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.758 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:48.760 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.760 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.761 DEBUG:boto:Method: HEAD 2023-11-02T18:00:48.761 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:48.762 DEBUG:boto:Data: 2023-11-02T18:00:48.762 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.763 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.763 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.764 DEBUG:boto:Params: {} 2023-11-02T18:00:48.764 DEBUG:boto:Token: None 2023-11-02T18:00:48.764 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj 2023-11-02T18:00:48.765 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1kxAKFnLHvzBkojxdw7I4ZmQqMc= 2023-11-02T18:00:48.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1kxAKFnLHvzBkojxdw7I4ZmQqMc=', 'Content-Length': '0'} 2023-11-02T18:00:48.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0fc43801f79bbe9-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.772 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.773 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.773 DEBUG:boto:Method: HEAD 2023-11-02T18:00:48.773 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:48.774 DEBUG:boto:Data: 2023-11-02T18:00:48.774 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.775 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.775 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.775 DEBUG:boto:Params: {} 2023-11-02T18:00:48.776 DEBUG:boto:Token: None 2023-11-02T18:00:48.776 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj 2023-11-02T18:00:48.777 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1kxAKFnLHvzBkojxdw7I4ZmQqMc= 2023-11-02T18:00:48.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1kxAKFnLHvzBkojxdw7I4ZmQqMc=', 'Content-Length': '0'} 2023-11-02T18:00:48.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b470d1d9e4136d6-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:48.785 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.786 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.786 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.786 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.787 DEBUG:boto:Method: GET 2023-11-02T18:00:48.787 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.788 DEBUG:boto:Data: 2023-11-02T18:00:48.788 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.789 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.789 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.790 DEBUG:boto:Params: {} 2023-11-02T18:00:48.791 DEBUG:boto:Token: None 2023-11-02T18:00:48.791 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.792 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vzy/aVO+bbpFrScabXpI9OC/RcQ= 2023-11-02T18:00:48.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vzy/aVO+bbpFrScabXpI9OC/RcQ=', 'Content-Length': '0'} 2023-11-02T18:00:48.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033483b2cc929d9e8-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.838 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.839 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.840 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.840 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.841 DEBUG:boto:Method: GET 2023-11-02T18:00:48.841 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.842 DEBUG:boto:Data: 2023-11-02T18:00:48.842 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.843 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.843 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.844 DEBUG:boto:Params: {} 2023-11-02T18:00:48.844 DEBUG:boto:Token: None 2023-11-02T18:00:48.845 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:48.845 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vzy/aVO+bbpFrScabXpI9OC/RcQ= 2023-11-02T18:00:48.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vzy/aVO+bbpFrScabXpI9OC/RcQ=', 'Content-Length': '0'} 2023-11-02T18:00:48.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cc64e1d0b4c2d04-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.890 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:48.891 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:48.892 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.892 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.892 DEBUG:boto:Method: GET 2023-11-02T18:00:48.893 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:48.893 DEBUG:boto:Data: 2023-11-02T18:00:48.894 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.894 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.895 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.895 DEBUG:boto:Params: {} 2023-11-02T18:00:48.896 DEBUG:boto:Token: None 2023-11-02T18:00:48.896 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/ 2023-11-02T18:00:48.897 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY= 2023-11-02T18:00:48.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY=', 'Content-Length': '0'} 2023-11-02T18:00:48.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b70670ecb3a8b46f-006543e3d0-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.904 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:48.904 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:48.905 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:48.905 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.906 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:48.906 DEBUG:boto:Method: GET 2023-11-02T18:00:48.907 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:48.907 DEBUG:boto:Data: 2023-11-02T18:00:48.907 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.908 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.908 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.909 DEBUG:boto:Params: {} 2023-11-02T18:00:48.909 DEBUG:boto:Token: None 2023-11-02T18:00:48.910 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/ 2023-11-02T18:00:48.910 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY= 2023-11-02T18:00:48.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fzH5lY/ZGLVKHHe8tFS4T91oltY=', 'Content-Length': '0'} 2023-11-02T18:00:48.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a80c7131f1880804-006543e3d0-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.916 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:48.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:48.917 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.917 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.918 DEBUG:boto:path=/ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.918 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.919 DEBUG:boto:Method: GET 2023-11-02T18:00:48.919 DEBUG:boto:Path: /ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.920 DEBUG:boto:Data: 2023-11-02T18:00:48.920 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.921 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.921 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.921 DEBUG:boto:Params: {} 2023-11-02T18:00:48.922 DEBUG:boto:Token: None 2023-11-02T18:00:48.922 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.923 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d83OIvq6dXCRIUd6l+6mkVFvJw4= 2023-11-02T18:00:48.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d83OIvq6dXCRIUd6l+6mkVFvJw4=', 'Content-Length': '0'} 2023-11-02T18:00:48.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'UiuCrzXqZHzsdVsjA0GimR8Ij68xCu-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006eb677dce01676b0-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.927 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.928 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.928 DEBUG:boto:path=/ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.929 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.929 DEBUG:boto:Method: GET 2023-11-02T18:00:48.930 DEBUG:boto:Path: /ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.930 DEBUG:boto:Data: 2023-11-02T18:00:48.930 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.931 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.931 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.932 DEBUG:boto:Params: {} 2023-11-02T18:00:48.932 DEBUG:boto:Token: None 2023-11-02T18:00:48.933 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d83OIvq6dXCRIUd6l+6mkVFvJw4= 2023-11-02T18:00:48.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d83OIvq6dXCRIUd6l+6mkVFvJw4=', 'Content-Length': '0'} 2023-11-02T18:00:48.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'UiuCrzXqZHzsdVsjA0GimR8Ij68xCu-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fbf48082f28a52d-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.982 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.983 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.983 DEBUG:boto:path=/ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.984 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.984 DEBUG:boto:Method: HEAD 2023-11-02T18:00:48.985 DEBUG:boto:Path: /ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.985 DEBUG:boto:Data: 2023-11-02T18:00:48.986 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.986 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:48.986 DEBUG:boto:Port: 8001 2023-11-02T18:00:48.987 DEBUG:boto:Params: {} 2023-11-02T18:00:48.987 DEBUG:boto:Token: None 2023-11-02T18:00:48.988 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AqyIIztWz6wZdW3ScohN+Uj1gJA= 2023-11-02T18:00:48.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AqyIIztWz6wZdW3ScohN+Uj1gJA=', 'Content-Length': '0'} 2023-11-02T18:00:48.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'UiuCrzXqZHzsdVsjA0GimR8Ij68xCu-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098426e4a81a668c1-006543e3d0-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:48.992 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:48.993 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:48.993 DEBUG:boto:path=/ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.994 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.994 DEBUG:boto:Method: HEAD 2023-11-02T18:00:48.995 DEBUG:boto:Path: /ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.995 DEBUG:boto:Data: 2023-11-02T18:00:48.995 DEBUG:boto:Headers: {} 2023-11-02T18:00:48.996 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:48.996 DEBUG:boto:Port: 8000 2023-11-02T18:00:48.997 DEBUG:boto:Params: {} 2023-11-02T18:00:48.997 DEBUG:boto:Token: None 2023-11-02T18:00:48.998 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:48 GMT /ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:48.998 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AqyIIztWz6wZdW3ScohN+Uj1gJA= 2023-11-02T18:00:48.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AqyIIztWz6wZdW3ScohN+Uj1gJA=', 'Content-Length': '0'} 2023-11-02T18:00:49.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'UiuCrzXqZHzsdVsjA0GimR8Ij68xCu-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3ea1d4baeadd63d-006543e3d0-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:49.003 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.003 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.004 DEBUG:boto:path=/ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:49.004 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:49.005 DEBUG:boto:Method: GET 2023-11-02T18:00:49.005 DEBUG:boto:Path: /ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:49.005 DEBUG:boto:Data: 2023-11-02T18:00:49.006 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.006 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.007 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.007 DEBUG:boto:Params: {} 2023-11-02T18:00:49.008 DEBUG:boto:Token: None 2023-11-02T18:00:49.008 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:49.009 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RyOYJ/DagLW/se33ZZQcIN+Ordg= 2023-11-02T18:00:49.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RyOYJ/DagLW/se33ZZQcIN+Ordg=', 'Content-Length': '0'} 2023-11-02T18:00:49.013 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'UiuCrzXqZHzsdVsjA0GimR8Ij68xCu-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4b6d6e02cd2797c-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.054 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.055 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.055 DEBUG:boto:path=/ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:49.056 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:49.056 DEBUG:boto:Method: GET 2023-11-02T18:00:49.057 DEBUG:boto:Path: /ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:49.057 DEBUG:boto:Data: 2023-11-02T18:00:49.058 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.058 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.059 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.059 DEBUG:boto:Params: {} 2023-11-02T18:00:49.060 DEBUG:boto:Token: None 2023-11-02T18:00:49.060 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=UiuCrzXqZHzsdVsjA0GimR8Ij68xCu- 2023-11-02T18:00:49.061 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RyOYJ/DagLW/se33ZZQcIN+Ordg= 2023-11-02T18:00:49.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RyOYJ/DagLW/se33ZZQcIN+Ordg=', 'Content-Length': '0'} 2023-11-02T18:00:49.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'UiuCrzXqZHzsdVsjA0GimR8Ij68xCu-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037e586361c072a3d-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.065 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:49.066 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:49.066 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.066 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.067 DEBUG:boto:Method: GET 2023-11-02T18:00:49.067 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:49.068 DEBUG:boto:Data: 2023-11-02T18:00:49.068 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.069 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.069 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.070 DEBUG:boto:Params: {} 2023-11-02T18:00:49.070 DEBUG:boto:Token: None 2023-11-02T18:00:49.071 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/ 2023-11-02T18:00:49.071 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA= 2023-11-02T18:00:49.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA=', 'Content-Length': '0'} 2023-11-02T18:00:49.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094ee3e6a76d0ee97-006543e3d1-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.118 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:49.119 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:49.120 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:49.120 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.121 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.121 DEBUG:boto:Method: GET 2023-11-02T18:00:49.122 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:49.122 DEBUG:boto:Data: 2023-11-02T18:00:49.122 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.123 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.123 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.124 DEBUG:boto:Params: {} 2023-11-02T18:00:49.124 DEBUG:boto:Token: None 2023-11-02T18:00:49.125 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/ 2023-11-02T18:00:49.125 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA= 2023-11-02T18:00:49.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA=', 'Content-Length': '0'} 2023-11-02T18:00:49.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e00359fcf807f066-006543e3d1-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.132 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:49.133 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.133 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.134 DEBUG:boto:Method: HEAD 2023-11-02T18:00:49.135 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:49.135 DEBUG:boto:Data: 2023-11-02T18:00:49.136 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.137 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.137 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.138 DEBUG:boto:Params: {} 2023-11-02T18:00:49.139 DEBUG:boto:Token: None 2023-11-02T18:00:49.139 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj 2023-11-02T18:00:49.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q2ZOUB2eF2wC+erffzu4v5xW78A= 2023-11-02T18:00:49.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q2ZOUB2eF2wC+erffzu4v5xW78A=', 'Content-Length': '0'} 2023-11-02T18:00:49.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f74fe25d5e147ad-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.147 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.147 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.148 DEBUG:boto:Method: HEAD 2023-11-02T18:00:49.148 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:49.149 DEBUG:boto:Data: 2023-11-02T18:00:49.150 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.150 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.151 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.152 DEBUG:boto:Params: {} 2023-11-02T18:00:49.152 DEBUG:boto:Token: None 2023-11-02T18:00:49.153 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj 2023-11-02T18:00:49.154 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q2ZOUB2eF2wC+erffzu4v5xW78A= 2023-11-02T18:00:49.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q2ZOUB2eF2wC+erffzu4v5xW78A=', 'Content-Length': '0'} 2023-11-02T18:00:49.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c3701988db840a4-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:49.161 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.161 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.162 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.162 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.163 DEBUG:boto:Method: GET 2023-11-02T18:00:49.163 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.164 DEBUG:boto:Data: 2023-11-02T18:00:49.164 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.165 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.165 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.166 DEBUG:boto:Params: {} 2023-11-02T18:00:49.166 DEBUG:boto:Token: None 2023-11-02T18:00:49.167 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.167 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2HpGynX9deTX34GI5SXK5EQNM7k= 2023-11-02T18:00:49.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2HpGynX9deTX34GI5SXK5EQNM7k=', 'Content-Length': '0'} 2023-11-02T18:00:49.171 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054e2234f41714125-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.171 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.172 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.172 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.173 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.173 DEBUG:boto:Method: GET 2023-11-02T18:00:49.174 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.174 DEBUG:boto:Data: 2023-11-02T18:00:49.174 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.175 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.175 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.176 DEBUG:boto:Params: {} 2023-11-02T18:00:49.176 DEBUG:boto:Token: None 2023-11-02T18:00:49.177 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2HpGynX9deTX34GI5SXK5EQNM7k= 2023-11-02T18:00:49.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2HpGynX9deTX34GI5SXK5EQNM7k=', 'Content-Length': '0'} 2023-11-02T18:00:49.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a848f195f8dc37f-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.226 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:49.227 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:49.228 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.228 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.229 DEBUG:boto:Method: GET 2023-11-02T18:00:49.229 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:49.230 DEBUG:boto:Data: 2023-11-02T18:00:49.230 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.231 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.232 DEBUG:boto:Params: {} 2023-11-02T18:00:49.232 DEBUG:boto:Token: None 2023-11-02T18:00:49.233 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/ 2023-11-02T18:00:49.233 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA= 2023-11-02T18:00:49.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA=', 'Content-Length': '0'} 2023-11-02T18:00:49.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbb27b6616bfecd9-006543e3d1-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.240 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:49.241 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:49.241 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:49.242 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.242 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.243 DEBUG:boto:Method: GET 2023-11-02T18:00:49.243 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:49.244 DEBUG:boto:Data: 2023-11-02T18:00:49.244 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.244 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.245 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.245 DEBUG:boto:Params: {} 2023-11-02T18:00:49.246 DEBUG:boto:Token: None 2023-11-02T18:00:49.246 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/ 2023-11-02T18:00:49.247 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA= 2023-11-02T18:00:49.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA=', 'Content-Length': '0'} 2023-11-02T18:00:49.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c6e882b87a366f3-006543e3d1-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.253 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:49.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:49.255 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.255 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.256 DEBUG:boto:path=/ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.257 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.257 DEBUG:boto:Method: GET 2023-11-02T18:00:49.258 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.258 DEBUG:boto:Data: 2023-11-02T18:00:49.259 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.260 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.260 DEBUG:boto:Params: {} 2023-11-02T18:00:49.261 DEBUG:boto:Token: None 2023-11-02T18:00:49.262 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.262 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDXG9D4uZch044F92iA9RiGS89M= 2023-11-02T18:00:49.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDXG9D4uZch044F92iA9RiGS89M=', 'Content-Length': '0'} 2023-11-02T18:00:49.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Zv-i3fTD6jXNRrAyl830BGdra2ND4x-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000600a4dfd08e4ace3-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.310 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.311 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.311 DEBUG:boto:path=/ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.312 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.312 DEBUG:boto:Method: GET 2023-11-02T18:00:49.313 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.313 DEBUG:boto:Data: 2023-11-02T18:00:49.314 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.314 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.315 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.315 DEBUG:boto:Params: {} 2023-11-02T18:00:49.316 DEBUG:boto:Token: None 2023-11-02T18:00:49.316 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.317 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDXG9D4uZch044F92iA9RiGS89M= 2023-11-02T18:00:49.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDXG9D4uZch044F92iA9RiGS89M=', 'Content-Length': '0'} 2023-11-02T18:00:49.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Zv-i3fTD6jXNRrAyl830BGdra2ND4x-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003dcbe03e30af9040-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.321 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.322 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.322 DEBUG:boto:path=/ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.322 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.323 DEBUG:boto:Method: HEAD 2023-11-02T18:00:49.323 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.324 DEBUG:boto:Data: 2023-11-02T18:00:49.324 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.324 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.325 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.325 DEBUG:boto:Params: {} 2023-11-02T18:00:49.326 DEBUG:boto:Token: None 2023-11-02T18:00:49.326 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.327 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:boLow9O1A611bWEOc1VzthDc0Fg= 2023-11-02T18:00:49.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:boLow9O1A611bWEOc1VzthDc0Fg=', 'Content-Length': '0'} 2023-11-02T18:00:49.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Zv-i3fTD6jXNRrAyl830BGdra2ND4x-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017e22c82a4284104-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.331 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.331 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.332 DEBUG:boto:path=/ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.332 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.333 DEBUG:boto:Method: HEAD 2023-11-02T18:00:49.333 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.333 DEBUG:boto:Data: 2023-11-02T18:00:49.334 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.334 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.335 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.335 DEBUG:boto:Params: {} 2023-11-02T18:00:49.336 DEBUG:boto:Token: None 2023-11-02T18:00:49.336 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.336 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:boLow9O1A611bWEOc1VzthDc0Fg= 2023-11-02T18:00:49.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:boLow9O1A611bWEOc1VzthDc0Fg=', 'Content-Length': '0'} 2023-11-02T18:00:49.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Zv-i3fTD6jXNRrAyl830BGdra2ND4x-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e4cddc897b657d5-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:49.341 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.341 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.342 DEBUG:boto:path=/ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.342 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.343 DEBUG:boto:Method: GET 2023-11-02T18:00:49.343 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.343 DEBUG:boto:Data: 2023-11-02T18:00:49.344 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.344 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.345 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.345 DEBUG:boto:Params: {} 2023-11-02T18:00:49.346 DEBUG:boto:Token: None 2023-11-02T18:00:49.346 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.347 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDXG9D4uZch044F92iA9RiGS89M= 2023-11-02T18:00:49.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDXG9D4uZch044F92iA9RiGS89M=', 'Content-Length': '0'} 2023-11-02T18:00:49.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Zv-i3fTD6jXNRrAyl830BGdra2ND4x-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d56a99b2bd7a7224-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.351 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.351 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.352 DEBUG:boto:path=/ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.352 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.353 DEBUG:boto:Method: GET 2023-11-02T18:00:49.353 DEBUG:boto:Path: /ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.354 DEBUG:boto:Data: 2023-11-02T18:00:49.354 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.354 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.355 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.355 DEBUG:boto:Params: {} 2023-11-02T18:00:49.356 DEBUG:boto:Token: None 2023-11-02T18:00:49.356 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=Zv-i3fTD6jXNRrAyl830BGdra2ND4x- 2023-11-02T18:00:49.357 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDXG9D4uZch044F92iA9RiGS89M= 2023-11-02T18:00:49.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDXG9D4uZch044F92iA9RiGS89M=', 'Content-Length': '0'} 2023-11-02T18:00:49.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'Zv-i3fTD6jXNRrAyl830BGdra2ND4x-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3fb1a1864f38eeb-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.402 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:49.403 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:49.403 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.404 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.404 DEBUG:boto:Method: GET 2023-11-02T18:00:49.405 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:49.405 DEBUG:boto:Data: 2023-11-02T18:00:49.406 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.406 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.407 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.407 DEBUG:boto:Params: {} 2023-11-02T18:00:49.408 DEBUG:boto:Token: None 2023-11-02T18:00:49.408 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/ 2023-11-02T18:00:49.409 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA= 2023-11-02T18:00:49.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA=', 'Content-Length': '0'} 2023-11-02T18:00:49.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000260bd93e0f98303e-006543e3d1-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.415 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:49.416 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:49.416 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:49.417 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.417 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.418 DEBUG:boto:Method: GET 2023-11-02T18:00:49.418 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:49.419 DEBUG:boto:Data: 2023-11-02T18:00:49.419 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.419 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.420 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.420 DEBUG:boto:Params: {} 2023-11-02T18:00:49.421 DEBUG:boto:Token: None 2023-11-02T18:00:49.421 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/ 2023-11-02T18:00:49.422 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA= 2023-11-02T18:00:49.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA=', 'Content-Length': '0'} 2023-11-02T18:00:49.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f87a25fd54c384c9-006543e3d1-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.470 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:49.471 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.472 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.472 DEBUG:boto:Method: HEAD 2023-11-02T18:00:49.473 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:49.473 DEBUG:boto:Data: 2023-11-02T18:00:49.474 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.475 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.475 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.476 DEBUG:boto:Params: {} 2023-11-02T18:00:49.476 DEBUG:boto:Token: None 2023-11-02T18:00:49.477 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj 2023-11-02T18:00:49.477 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q2ZOUB2eF2wC+erffzu4v5xW78A= 2023-11-02T18:00:49.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q2ZOUB2eF2wC+erffzu4v5xW78A=', 'Content-Length': '0'} 2023-11-02T18:00:49.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b1a871de8d1d34f-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.485 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.485 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.485 DEBUG:boto:Method: HEAD 2023-11-02T18:00:49.486 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:49.486 DEBUG:boto:Data: 2023-11-02T18:00:49.487 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.487 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.488 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.488 DEBUG:boto:Params: {} 2023-11-02T18:00:49.488 DEBUG:boto:Token: None 2023-11-02T18:00:49.489 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj 2023-11-02T18:00:49.490 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q2ZOUB2eF2wC+erffzu4v5xW78A= 2023-11-02T18:00:49.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q2ZOUB2eF2wC+erffzu4v5xW78A=', 'Content-Length': '0'} 2023-11-02T18:00:49.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e722c92f1d04f3ce-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:49.497 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.498 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.498 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.499 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.499 DEBUG:boto:Method: GET 2023-11-02T18:00:49.500 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.501 DEBUG:boto:Data: 2023-11-02T18:00:49.502 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.502 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.503 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.503 DEBUG:boto:Params: {} 2023-11-02T18:00:49.504 DEBUG:boto:Token: None 2023-11-02T18:00:49.505 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.505 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2HpGynX9deTX34GI5SXK5EQNM7k= 2023-11-02T18:00:49.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2HpGynX9deTX34GI5SXK5EQNM7k=', 'Content-Length': '0'} 2023-11-02T18:00:49.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea305ee97290ae59-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.554 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.555 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.555 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.556 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.556 DEBUG:boto:Method: GET 2023-11-02T18:00:49.558 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.558 DEBUG:boto:Data: 2023-11-02T18:00:49.559 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.559 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.559 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.560 DEBUG:boto:Params: {} 2023-11-02T18:00:49.560 DEBUG:boto:Token: None 2023-11-02T18:00:49.561 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.561 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2HpGynX9deTX34GI5SXK5EQNM7k= 2023-11-02T18:00:49.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2HpGynX9deTX34GI5SXK5EQNM7k=', 'Content-Length': '0'} 2023-11-02T18:00:49.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001193509076cf8ef2-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.606 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:49.607 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:49.607 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.608 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.608 DEBUG:boto:Method: GET 2023-11-02T18:00:49.608 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:49.609 DEBUG:boto:Data: 2023-11-02T18:00:49.609 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.610 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.610 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.611 DEBUG:boto:Params: {} 2023-11-02T18:00:49.611 DEBUG:boto:Token: None 2023-11-02T18:00:49.611 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/ 2023-11-02T18:00:49.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA= 2023-11-02T18:00:49.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA=', 'Content-Length': '0'} 2023-11-02T18:00:49.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009377c418e9f9dd08-006543e3d1-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.618 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:49.619 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:49.620 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:49.620 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.620 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.621 DEBUG:boto:Method: GET 2023-11-02T18:00:49.621 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:49.622 DEBUG:boto:Data: 2023-11-02T18:00:49.622 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.623 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.623 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.624 DEBUG:boto:Params: {} 2023-11-02T18:00:49.624 DEBUG:boto:Token: None 2023-11-02T18:00:49.625 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/ 2023-11-02T18:00:49.625 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA= 2023-11-02T18:00:49.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA=', 'Content-Length': '0'} 2023-11-02T18:00:49.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000852503c0eda12ce7-006543e3d1-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.675 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:49.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:49.676 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.676 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.677 DEBUG:boto:path=/ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.677 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.677 DEBUG:boto:Method: GET 2023-11-02T18:00:49.678 DEBUG:boto:Path: /ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.678 DEBUG:boto:Data: 2023-11-02T18:00:49.679 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.679 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.680 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.680 DEBUG:boto:Params: {} 2023-11-02T18:00:49.681 DEBUG:boto:Token: None 2023-11-02T18:00:49.681 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.682 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BUrrQWdZxN4n2reSesF3QD5F2g4= 2023-11-02T18:00:49.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BUrrQWdZxN4n2reSesF3QD5F2g4=', 'Content-Length': '0'} 2023-11-02T18:00:49.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a4afd467949ad17-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.686 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.687 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.687 DEBUG:boto:path=/ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.688 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.688 DEBUG:boto:Method: GET 2023-11-02T18:00:49.689 DEBUG:boto:Path: /ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.689 DEBUG:boto:Data: 2023-11-02T18:00:49.689 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.690 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.691 DEBUG:boto:Params: {} 2023-11-02T18:00:49.691 DEBUG:boto:Token: None 2023-11-02T18:00:49.692 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.692 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BUrrQWdZxN4n2reSesF3QD5F2g4= 2023-11-02T18:00:49.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BUrrQWdZxN4n2reSesF3QD5F2g4=', 'Content-Length': '0'} 2023-11-02T18:00:49.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b89e62eb6a69e522-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.696 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.697 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.697 DEBUG:boto:path=/ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.698 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.698 DEBUG:boto:Method: HEAD 2023-11-02T18:00:49.699 DEBUG:boto:Path: /ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.699 DEBUG:boto:Data: 2023-11-02T18:00:49.699 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.700 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.701 DEBUG:boto:Params: {} 2023-11-02T18:00:49.701 DEBUG:boto:Token: None 2023-11-02T18:00:49.702 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.702 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g6f5/5YtXc6fjo4w0ZkwQkZM5pU= 2023-11-02T18:00:49.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g6f5/5YtXc6fjo4w0ZkwQkZM5pU=', 'Content-Length': '0'} 2023-11-02T18:00:49.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e567d502720dbf3-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.706 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.706 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.707 DEBUG:boto:path=/ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.707 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.708 DEBUG:boto:Method: HEAD 2023-11-02T18:00:49.708 DEBUG:boto:Path: /ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.708 DEBUG:boto:Data: 2023-11-02T18:00:49.709 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.709 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.710 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.710 DEBUG:boto:Params: {} 2023-11-02T18:00:49.710 DEBUG:boto:Token: None 2023-11-02T18:00:49.711 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.711 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g6f5/5YtXc6fjo4w0ZkwQkZM5pU= 2023-11-02T18:00:49.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g6f5/5YtXc6fjo4w0ZkwQkZM5pU=', 'Content-Length': '0'} 2023-11-02T18:00:49.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fb91002beb2472f-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:49.716 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.716 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.716 DEBUG:boto:path=/ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.717 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.717 DEBUG:boto:Method: GET 2023-11-02T18:00:49.718 DEBUG:boto:Path: /ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.718 DEBUG:boto:Data: 2023-11-02T18:00:49.718 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.719 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.719 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.720 DEBUG:boto:Params: {} 2023-11-02T18:00:49.720 DEBUG:boto:Token: None 2023-11-02T18:00:49.721 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BUrrQWdZxN4n2reSesF3QD5F2g4= 2023-11-02T18:00:49.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BUrrQWdZxN4n2reSesF3QD5F2g4=', 'Content-Length': '0'} 2023-11-02T18:00:49.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dd4dd258cb2bf3c-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.725 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.726 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.726 DEBUG:boto:path=/ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.727 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.727 DEBUG:boto:Method: GET 2023-11-02T18:00:49.728 DEBUG:boto:Path: /ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.728 DEBUG:boto:Data: 2023-11-02T18:00:49.729 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.729 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.729 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.730 DEBUG:boto:Params: {} 2023-11-02T18:00:49.730 DEBUG:boto:Token: None 2023-11-02T18:00:49.731 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks 2023-11-02T18:00:49.731 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BUrrQWdZxN4n2reSesF3QD5F2g4= 2023-11-02T18:00:49.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BUrrQWdZxN4n2reSesF3QD5F2g4=', 'Content-Length': '0'} 2023-11-02T18:00:49.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', '1rc-p4tkyGtD.jiHCC3wn3kBQ71w-ks'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b621a8cc5f1e389f-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.778 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:49.779 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:49.779 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.779 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.780 DEBUG:boto:Method: GET 2023-11-02T18:00:49.780 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:49.781 DEBUG:boto:Data: 2023-11-02T18:00:49.781 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.782 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.782 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.782 DEBUG:boto:Params: {} 2023-11-02T18:00:49.783 DEBUG:boto:Token: None 2023-11-02T18:00:49.783 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/ 2023-11-02T18:00:49.784 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA= 2023-11-02T18:00:49.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA=', 'Content-Length': '0'} 2023-11-02T18:00:49.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f1f3d811b828483-006543e3d1-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.790 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:49.791 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:49.791 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:49.792 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.792 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.793 DEBUG:boto:Method: GET 2023-11-02T18:00:49.793 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:49.793 DEBUG:boto:Data: 2023-11-02T18:00:49.794 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.794 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.795 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.795 DEBUG:boto:Params: {} 2023-11-02T18:00:49.796 DEBUG:boto:Token: None 2023-11-02T18:00:49.796 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/ 2023-11-02T18:00:49.797 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA= 2023-11-02T18:00:49.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA=', 'Content-Length': '0'} 2023-11-02T18:00:49.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e7a60366e9e215a-006543e3d1-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.846 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:49.847 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.847 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.848 DEBUG:boto:Method: HEAD 2023-11-02T18:00:49.848 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:49.849 DEBUG:boto:Data: 2023-11-02T18:00:49.849 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.849 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.850 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.850 DEBUG:boto:Params: {} 2023-11-02T18:00:49.851 DEBUG:boto:Token: None 2023-11-02T18:00:49.851 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj 2023-11-02T18:00:49.852 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q2ZOUB2eF2wC+erffzu4v5xW78A= 2023-11-02T18:00:49.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q2ZOUB2eF2wC+erffzu4v5xW78A=', 'Content-Length': '0'} 2023-11-02T18:00:49.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1703f2d9ac93e6f-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.859 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.859 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.860 DEBUG:boto:Method: HEAD 2023-11-02T18:00:49.860 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:49.860 DEBUG:boto:Data: 2023-11-02T18:00:49.861 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.861 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.862 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.862 DEBUG:boto:Params: {} 2023-11-02T18:00:49.863 DEBUG:boto:Token: None 2023-11-02T18:00:49.863 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj 2023-11-02T18:00:49.864 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q2ZOUB2eF2wC+erffzu4v5xW78A= 2023-11-02T18:00:49.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q2ZOUB2eF2wC+erffzu4v5xW78A=', 'Content-Length': '0'} 2023-11-02T18:00:49.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036b0900928bf3aa4-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:49.871 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.871 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.872 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.872 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.873 DEBUG:boto:Method: GET 2023-11-02T18:00:49.873 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.873 DEBUG:boto:Data: 2023-11-02T18:00:49.874 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.874 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.875 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.875 DEBUG:boto:Params: {} 2023-11-02T18:00:49.875 DEBUG:boto:Token: None 2023-11-02T18:00:49.876 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.876 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2HpGynX9deTX34GI5SXK5EQNM7k= 2023-11-02T18:00:49.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2HpGynX9deTX34GI5SXK5EQNM7k=', 'Content-Length': '0'} 2023-11-02T18:00:49.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0b05968f0bda7d7-006543e3d1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.922 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:49.923 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:49.923 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.924 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.924 DEBUG:boto:Method: GET 2023-11-02T18:00:49.925 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.925 DEBUG:boto:Data: 2023-11-02T18:00:49.926 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.926 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.926 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.927 DEBUG:boto:Params: {} 2023-11-02T18:00:49.927 DEBUG:boto:Token: None 2023-11-02T18:00:49.928 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:49.928 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2HpGynX9deTX34GI5SXK5EQNM7k= 2023-11-02T18:00:49.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2HpGynX9deTX34GI5SXK5EQNM7k=', 'Content-Length': '0'} 2023-11-02T18:00:49.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ab14ae33092cfc3-006543e3d1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.974 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:49.975 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:49.975 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.976 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.976 DEBUG:boto:Method: GET 2023-11-02T18:00:49.976 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:49.977 DEBUG:boto:Data: 2023-11-02T18:00:49.977 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.978 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:49.978 DEBUG:boto:Port: 8001 2023-11-02T18:00:49.979 DEBUG:boto:Params: {} 2023-11-02T18:00:49.979 DEBUG:boto:Token: None 2023-11-02T18:00:49.980 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/ 2023-11-02T18:00:49.980 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA= 2023-11-02T18:00:49.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA=', 'Content-Length': '0'} 2023-11-02T18:00:49.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3eee0f2402be35a-006543e3d1-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:49.987 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:49.987 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:49.988 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:49.988 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.989 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:49.989 DEBUG:boto:Method: GET 2023-11-02T18:00:49.990 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:49.990 DEBUG:boto:Data: 2023-11-02T18:00:49.990 DEBUG:boto:Headers: {} 2023-11-02T18:00:49.991 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:49.991 DEBUG:boto:Port: 8000 2023-11-02T18:00:49.992 DEBUG:boto:Params: {} 2023-11-02T18:00:49.992 DEBUG:boto:Token: None 2023-11-02T18:00:49.992 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:49 GMT /ibtljn-13/ 2023-11-02T18:00:49.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA= 2023-11-02T18:00:49.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxATBQ82HIDXR7h1EWkq8YEQvCA=', 'Content-Length': '0'} 2023-11-02T18:00:49.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b17849925a6dff13-006543e3d1-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.042 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:50.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:50.043 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:50.044 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:50.044 DEBUG:boto:path=/ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.045 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.045 DEBUG:boto:Method: GET 2023-11-02T18:00:50.045 DEBUG:boto:Path: /ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.046 DEBUG:boto:Data: 2023-11-02T18:00:50.046 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.047 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:50.047 DEBUG:boto:Port: 8001 2023-11-02T18:00:50.048 DEBUG:boto:Params: {} 2023-11-02T18:00:50.048 DEBUG:boto:Token: None 2023-11-02T18:00:50.049 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.049 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lTTHPBmyUtOROZ8pK2RqVxjGsUk= 2023-11-02T18:00:50.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lTTHPBmyUtOROZ8pK2RqVxjGsUk=', 'Content-Length': '0'} 2023-11-02T18:00:50.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'kcaDEYIG28LX9ix1nN7FlHWFX8nqevl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037e239916adad52d-006543e3d2-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.054 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:50.054 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:50.055 DEBUG:boto:path=/ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.055 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.055 DEBUG:boto:Method: GET 2023-11-02T18:00:50.056 DEBUG:boto:Path: /ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.056 DEBUG:boto:Data: 2023-11-02T18:00:50.057 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.057 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:50.058 DEBUG:boto:Port: 8000 2023-11-02T18:00:50.058 DEBUG:boto:Params: {} 2023-11-02T18:00:50.059 DEBUG:boto:Token: None 2023-11-02T18:00:50.059 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.060 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lTTHPBmyUtOROZ8pK2RqVxjGsUk= 2023-11-02T18:00:50.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lTTHPBmyUtOROZ8pK2RqVxjGsUk=', 'Content-Length': '0'} 2023-11-02T18:00:50.063 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'kcaDEYIG28LX9ix1nN7FlHWFX8nqevl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000690fcb7b73b2b835-006543e3d2-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.064 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:50.064 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:50.065 DEBUG:boto:path=/ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.065 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.066 DEBUG:boto:Method: HEAD 2023-11-02T18:00:50.066 DEBUG:boto:Path: /ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.067 DEBUG:boto:Data: 2023-11-02T18:00:50.067 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.067 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:50.068 DEBUG:boto:Port: 8001 2023-11-02T18:00:50.068 DEBUG:boto:Params: {} 2023-11-02T18:00:50.069 DEBUG:boto:Token: None 2023-11-02T18:00:50.069 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.070 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lA7JNFDx+8589QFqv5/NE0sX8BM= 2023-11-02T18:00:50.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lA7JNFDx+8589QFqv5/NE0sX8BM=', 'Content-Length': '0'} 2023-11-02T18:00:50.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'kcaDEYIG28LX9ix1nN7FlHWFX8nqevl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058a78784dac79cfa-006543e3d2-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.073 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:50.074 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:50.074 DEBUG:boto:path=/ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.075 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.075 DEBUG:boto:Method: HEAD 2023-11-02T18:00:50.075 DEBUG:boto:Path: /ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.076 DEBUG:boto:Data: 2023-11-02T18:00:50.076 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.077 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:50.077 DEBUG:boto:Port: 8000 2023-11-02T18:00:50.077 DEBUG:boto:Params: {} 2023-11-02T18:00:50.078 DEBUG:boto:Token: None 2023-11-02T18:00:50.078 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.079 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lA7JNFDx+8589QFqv5/NE0sX8BM= 2023-11-02T18:00:50.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lA7JNFDx+8589QFqv5/NE0sX8BM=', 'Content-Length': '0'} 2023-11-02T18:00:50.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'kcaDEYIG28LX9ix1nN7FlHWFX8nqevl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec3818d6fcf0f795-006543e3d2-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:50.083 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:50.084 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:50.084 DEBUG:boto:path=/ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.084 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.085 DEBUG:boto:Method: GET 2023-11-02T18:00:50.085 DEBUG:boto:Path: /ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.086 DEBUG:boto:Data: 2023-11-02T18:00:50.086 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.087 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:50.087 DEBUG:boto:Port: 8001 2023-11-02T18:00:50.088 DEBUG:boto:Params: {} 2023-11-02T18:00:50.088 DEBUG:boto:Token: None 2023-11-02T18:00:50.089 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.089 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lTTHPBmyUtOROZ8pK2RqVxjGsUk= 2023-11-02T18:00:50.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lTTHPBmyUtOROZ8pK2RqVxjGsUk=', 'Content-Length': '0'} 2023-11-02T18:00:50.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'kcaDEYIG28LX9ix1nN7FlHWFX8nqevl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3450793e4d855fa-006543e3d2-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.138 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:50.139 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:50.139 DEBUG:boto:path=/ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.140 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.140 DEBUG:boto:Method: GET 2023-11-02T18:00:50.140 DEBUG:boto:Path: /ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.141 DEBUG:boto:Data: 2023-11-02T18:00:50.141 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.142 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:50.142 DEBUG:boto:Port: 8000 2023-11-02T18:00:50.142 DEBUG:boto:Params: {} 2023-11-02T18:00:50.143 DEBUG:boto:Token: None 2023-11-02T18:00:50.143 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/obj?versionId=kcaDEYIG28LX9ix1nN7FlHWFX8nqevl 2023-11-02T18:00:50.144 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lTTHPBmyUtOROZ8pK2RqVxjGsUk= 2023-11-02T18:00:50.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lTTHPBmyUtOROZ8pK2RqVxjGsUk=', 'Content-Length': '0'} 2023-11-02T18:00:50.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:02 GMT'), ('x-amz-version-id', 'kcaDEYIG28LX9ix1nN7FlHWFX8nqevl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001004a90003d42c14-006543e3d2-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.148 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:50.149 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:50.149 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:50.150 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:50.150 DEBUG:boto:Method: GET 2023-11-02T18:00:50.151 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:50.151 DEBUG:boto:Data: 2023-11-02T18:00:50.151 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.152 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:50.152 DEBUG:boto:Port: 8001 2023-11-02T18:00:50.153 DEBUG:boto:Params: {} 2023-11-02T18:00:50.153 DEBUG:boto:Token: None 2023-11-02T18:00:50.154 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/ 2023-11-02T18:00:50.154 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zXFFAC14l+6V2TGhVLA9YntPuzI= 2023-11-02T18:00:50.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zXFFAC14l+6V2TGhVLA9YntPuzI=', 'Content-Length': '0'} 2023-11-02T18:00:50.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdf9259ce8480d07-006543e3d2-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.202 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:50.203 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:50.203 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:50.204 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:50.204 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:50.205 DEBUG:boto:Method: GET 2023-11-02T18:00:50.205 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:50.205 DEBUG:boto:Data: 2023-11-02T18:00:50.206 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.206 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:50.207 DEBUG:boto:Port: 8000 2023-11-02T18:00:50.207 DEBUG:boto:Params: {} 2023-11-02T18:00:50.208 DEBUG:boto:Token: None 2023-11-02T18:00:50.208 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/ 2023-11-02T18:00:50.209 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zXFFAC14l+6V2TGhVLA9YntPuzI= 2023-11-02T18:00:50.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zXFFAC14l+6V2TGhVLA9YntPuzI=', 'Content-Length': '0'} 2023-11-02T18:00:50.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e5ae082e728a1b6-006543e3d2-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.214 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:50.215 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:50.215 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:50.216 DEBUG:boto:Method: HEAD 2023-11-02T18:00:50.216 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:50.217 DEBUG:boto:Data: 2023-11-02T18:00:50.217 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.217 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:50.218 DEBUG:boto:Port: 8001 2023-11-02T18:00:50.218 DEBUG:boto:Params: {} 2023-11-02T18:00:50.219 DEBUG:boto:Token: None 2023-11-02T18:00:50.219 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/obj 2023-11-02T18:00:50.220 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vKdhcLHZuCupTsPVDiU/8VSN4OQ= 2023-11-02T18:00:50.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vKdhcLHZuCupTsPVDiU/8VSN4OQ=', 'Content-Length': '0'} 2023-11-02T18:00:50.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7e7e7ef34bc1818-006543e3d2-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.229 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:50.229 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:50.230 DEBUG:boto:Method: HEAD 2023-11-02T18:00:50.230 DEBUG:boto:Path: /ibtljn-13/obj 2023-11-02T18:00:50.231 DEBUG:boto:Data: 2023-11-02T18:00:50.231 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:50.232 DEBUG:boto:Port: 8000 2023-11-02T18:00:50.232 DEBUG:boto:Params: {} 2023-11-02T18:00:50.233 DEBUG:boto:Token: None 2023-11-02T18:00:50.233 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/obj 2023-11-02T18:00:50.234 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vKdhcLHZuCupTsPVDiU/8VSN4OQ= 2023-11-02T18:00:50.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vKdhcLHZuCupTsPVDiU/8VSN4OQ=', 'Content-Length': '0'} 2023-11-02T18:00:50.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd76b5a20345365b-006543e3d2-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:00:50.241 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:50.241 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:50.242 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:50.242 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:50.243 DEBUG:boto:Method: GET 2023-11-02T18:00:50.243 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:50.244 DEBUG:boto:Data: 2023-11-02T18:00:50.244 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.245 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:50.245 DEBUG:boto:Port: 8001 2023-11-02T18:00:50.246 DEBUG:boto:Params: {} 2023-11-02T18:00:50.246 DEBUG:boto:Token: None 2023-11-02T18:00:50.247 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:50.247 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a5ilvT2NW1Kn6jhrDEOtHdtWtQE= 2023-11-02T18:00:50.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a5ilvT2NW1Kn6jhrDEOtHdtWtQE=', 'Content-Length': '0'} 2023-11-02T18:00:50.251 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d56cacf74df01173-006543e3d2-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.298 DEBUG:boto:path=/ibtljn-13/obj 2023-11-02T18:00:50.299 DEBUG:boto:auth_path=/ibtljn-13/obj 2023-11-02T18:00:50.299 DEBUG:boto:path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:50.300 DEBUG:boto:auth_path=/ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:50.300 DEBUG:boto:Method: GET 2023-11-02T18:00:50.301 DEBUG:boto:Path: /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:50.301 DEBUG:boto:Data: 2023-11-02T18:00:50.302 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.302 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:50.303 DEBUG:boto:Port: 8000 2023-11-02T18:00:50.303 DEBUG:boto:Params: {} 2023-11-02T18:00:50.303 DEBUG:boto:Token: None 2023-11-02T18:00:50.304 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/obj?versionId=-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4 2023-11-02T18:00:50.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a5ilvT2NW1Kn6jhrDEOtHdtWtQE= 2023-11-02T18:00:50.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a5ilvT2NW1Kn6jhrDEOtHdtWtQE=', 'Content-Length': '0'} 2023-11-02T18:00:50.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 17:59:03 GMT'), ('x-amz-version-id', '-tuuW1MYMXi.a7fzOdkNYj4q6lZGYg4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d84d4a3913560dd5-006543e3d2-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.354 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:50.355 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:50.355 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:50.356 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:50.356 DEBUG:boto:Method: GET 2023-11-02T18:00:50.357 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:50.357 DEBUG:boto:Data: 2023-11-02T18:00:50.358 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:00:50.358 DEBUG:boto:Port: 8001 2023-11-02T18:00:50.359 DEBUG:boto:Params: {} 2023-11-02T18:00:50.359 DEBUG:boto:Token: None 2023-11-02T18:00:50.360 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/ 2023-11-02T18:00:50.360 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zXFFAC14l+6V2TGhVLA9YntPuzI= 2023-11-02T18:00:50.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zXFFAC14l+6V2TGhVLA9YntPuzI=', 'Content-Length': '0'} 2023-11-02T18:00:50.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007116030673eaef9e-006543e3d2-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.367 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:50.368 DEBUG:boto:path=/ibtljn-13/ 2023-11-02T18:00:50.368 DEBUG:boto:auth_path=/ibtljn-13/ 2023-11-02T18:00:50.369 DEBUG:boto:path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:50.369 DEBUG:boto:auth_path=/ibtljn-13/?prefix=obj 2023-11-02T18:00:50.370 DEBUG:boto:Method: GET 2023-11-02T18:00:50.370 DEBUG:boto:Path: /ibtljn-13/?prefix=obj 2023-11-02T18:00:50.371 DEBUG:boto:Data: 2023-11-02T18:00:50.371 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.372 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:50.372 DEBUG:boto:Port: 8000 2023-11-02T18:00:50.372 DEBUG:boto:Params: {} 2023-11-02T18:00:50.373 DEBUG:boto:Token: None 2023-11-02T18:00:50.373 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-13/ 2023-11-02T18:00:50.374 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zXFFAC14l+6V2TGhVLA9YntPuzI= 2023-11-02T18:00:50.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zXFFAC14l+6V2TGhVLA9YntPuzI=', 'Content-Length': '0'} 2023-11-02T18:00:50.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6a62350d01c778c-006543e3d2-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:50.381 DEBUG:boto:b'ibtljn-13obj1000falseobj2023-11-02T17:59:03.109Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:00:50.382 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-13 zones={a2, a1} 2023-11-02T18:00:50.382 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-11-02T18:00:50.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:00:50.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:00:50.384 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:00:50.429 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:00:50.430 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:00:50.527 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:00:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:00:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:00.706996Z" 2023-11-02T18:00:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001632", 2023-11-02T18:00:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:59:33.498843Z" 2023-11-02T18:00:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-11-02T18:00:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:59:11.495487Z" 2023-11-02T18:00:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-11-02T18:00:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:52:00.720077Z" 2023-11-02T18:00:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:50.531 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:00:50.532 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000001632', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-11-02T18:00:50.532 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:00:50.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:00:50.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:00:50.534 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:00:50.581 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:00:50.581 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:00:50.662 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:00:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:00:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:00:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:00:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:00:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 14562503763241451158 2023-11-02T18:00:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:00:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:00:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:00:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:00:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:00:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:00:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:00:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:00:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:00.706996Z" 2023-11-02T18:00:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:00:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:00:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:00:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001632", 2023-11-02T18:00:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:00:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:00:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:00:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:59:33.498843Z" 2023-11-02T18:00:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:00:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:00:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:00:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-11-02T18:00:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:00:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:00:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:00:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:59:11.495487Z" 2023-11-02T18:00:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:00:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:00:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:00:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-11-02T18:00:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:00:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:00:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:00:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:52:00.720077Z" 2023-11-02T18:00:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:00:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:00:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:00:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:00:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:00:50Z" 2023-11-02T18:00:50.675 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:00:50.676 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14562503763241451158 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:00.706996Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001632", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:59:33.498843Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:59:11.495487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:52:00.720077Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:00:50Z" } 2023-11-02T18:00:50.676 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:00.706996Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001632', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:59:33.498843Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:59:11.495487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:52:00.720077Z'}}] 2023-11-02T18:00:50.677 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000001632', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-11-02T18:00:50.677 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000001632', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-11-02T18:00:50.678 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:00:50.678 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-11-02T18:00:50.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:00:50.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:00:50.679 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:00:50.724 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:00:50.724 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:00:50.822 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:00:50.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.822 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:00:50.822 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:52:00.715412Z" 2023-11-02T18:00:50.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.823 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002337", 2023-11-02T18:00:50.823 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:00:39.563337Z" 2023-11-02T18:00:50.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.824 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-11-02T18:00:50.824 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:58:40.713454Z" 2023-11-02T18:00:50.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.825 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-11-02T18:00:50.825 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:52:40.743883Z" 2023-11-02T18:00:50.825 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:50.825 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:00:50.826 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000002337', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000000929'} 2023-11-02T18:00:50.827 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:00:50.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:00:50.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:00:50.828 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:00:50.872 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:00:50.872 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:00:50.956 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:00:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:00:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:00:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:00:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:00:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:00:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:00:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:00:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:00:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:00:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:00:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:00:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:00:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:00:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:52:00.715412Z" 2023-11-02T18:00:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:00:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:00:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:00:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002337", 2023-11-02T18:00:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:00:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:00:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:00:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:00:39.563337Z" 2023-11-02T18:00:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:00:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:00:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:00:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-11-02T18:00:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:00:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:00:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:00:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:58:40.713454Z" 2023-11-02T18:00:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:00:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:00:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:00:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-11-02T18:00:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:00:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:00:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:00:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:52:40.743883Z" 2023-11-02T18:00:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:00:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:00:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:00:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:00:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:00:50Z" 2023-11-02T18:00:50.969 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:00:50.970 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:52:00.715412Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002337", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:00:39.563337Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:58:40.713454Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:52:40.743883Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:00:50Z" } 2023-11-02T18:00:50.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:52:00.715412Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002337', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:00:39.563337Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:58:40.713454Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:52:40.743883Z'}}] 2023-11-02T18:00:50.971 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000002337', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000000929'} 2023-11-02T18:00:50.972 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000002337', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000000929'} 2023-11-02T18:00:50.972 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:00:50.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2023-11-02T18:00:50.974 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:00:50.974 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:00:50.975 DEBUG:boto:Method: PUT 2023-11-02T18:00:50.975 DEBUG:boto:Path: /ibtljn-14/ 2023-11-02T18:00:50.976 DEBUG:boto:Data: 2023-11-02T18:00:50.976 DEBUG:boto:Headers: {} 2023-11-02T18:00:50.976 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:50.977 DEBUG:boto:Port: 8000 2023-11-02T18:00:50.977 DEBUG:boto:Params: {} 2023-11-02T18:00:50.978 DEBUG:boto:Token: None 2023-11-02T18:00:50.978 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:00:50 GMT /ibtljn-14/ 2023-11-02T18:00:50.979 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BSzX3DdbyKm8Y/mOq/mzwbeTnFg= 2023-11-02T18:00:50.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BSzX3DdbyKm8Y/mOq/mzwbeTnFg=', 'Content-Length': '0'} 2023-11-02T18:00:51.005 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008821d45d2975510f-006543e3d2-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:51.006 DEBUG:rgw_multi.tests:created bucket=ibtljn-14 2023-11-02T18:00:51.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:00:51.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:00:51.007 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:00:51.054 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:00:51.055 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:00:51.107 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:00:51.133 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:00:51.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:51.133 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:00:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:00:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:00:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:00:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948050.996288_84.1", 2023-11-02T18:00:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:00:50.996288Z" 2023-11-02T18:00:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:00:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:00:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:51.137 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:00:51.138 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948050.996288_84.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:00:51.138 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:00:51.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:00:51.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:00:51.139 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:00:51.180 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:00:51.181 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:00:51.233 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:51.222+0000 7fcdf0020f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:00:51.234 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:51.226+0000 7fcdf0020f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:00:51.260 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:00:51.261 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:00:51.261 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:00:51.261 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:00:51.261 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:00:51.261 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:00:51.262 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:00:51.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:51.262 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:00:51.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:51.263 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:00:51.263 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:00:51.263 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:00:51.263 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:00:51.264 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:00:51.264 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:00:51.264 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:00:51.264 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:00:51.265 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:00:51.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:51.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:51.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:51.265 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:00:51.266 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:00:51.266 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:00:51.266 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:00:51.266 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:00:51.266 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:00:51.267 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:00:51.267 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:00:51.267 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:00:51.267 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:51.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:51.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:51.268 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:00:51.268 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:00:51.269 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:00:51.269 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948050.996288_84.1", 2023-11-02T18:00:51.269 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:00:51.269 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:00:51.269 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:00:51.270 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:00:50.996288Z", 2023-11-02T18:00:51.270 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:00:51.270 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:51.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:51.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:00:51.271 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:00:51.271 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:00:51.271 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:00:51.271 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:00:51.272 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:00:51.272 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:00:51.272 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:00:51.272 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:00:51.272 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:00:51.273 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:51.273 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:00:51.273 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:00:51.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:51.273 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:00:51.274 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:00:51.274 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:00:51.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:00:51.274 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:00:51Z" 2023-11-02T18:00:51.275 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:00:51.276 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948050.996288_84.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:00:50.996288Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:00:51Z" } 2023-11-02T18:00:51.276 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948050.996288_84.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:00:50.996288Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:00:51.277 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948050.996288_84.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:00:51.277 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948050.996288_84.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:00:51.277 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:00:51.278 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:00:51.278 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:00:51.279 DEBUG:boto:Method: HEAD 2023-11-02T18:00:51.279 DEBUG:boto:Path: /ibtljn-14/ 2023-11-02T18:00:51.280 DEBUG:boto:Data: 2023-11-02T18:00:51.280 DEBUG:boto:Headers: {} 2023-11-02T18:00:51.280 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:51.281 DEBUG:boto:Port: 8000 2023-11-02T18:00:51.281 DEBUG:boto:Params: {} 2023-11-02T18:00:51.282 DEBUG:boto:Token: None 2023-11-02T18:00:51.282 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:00:51 GMT /ibtljn-14/ 2023-11-02T18:00:51.283 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BevYMtzpc3IS7awy5+tECJ6QdkA= 2023-11-02T18:00:51.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:00:51 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BevYMtzpc3IS7awy5+tECJ6QdkA=', 'Content-Length': '0'} 2023-11-02T18:00:51.287 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000071f9849ba0feab3-006543e3d3-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:51.288 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:00:51.288 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:00:51.289 DEBUG:boto:Method: PUT 2023-11-02T18:00:51.289 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:00:51.289 DEBUG:boto:Data: 2023-11-02T18:00:51.290 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-11-02T18:00:51.290 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:00:51.291 DEBUG:boto:Port: 8000 2023-11-02T18:00:51.291 DEBUG:boto:Params: {} 2023-11-02T18:00:51.291 DEBUG:boto:Token: None 2023-11-02T18:00:51.292 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 02 Nov 2023 18:00:51 GMT /ibtljn-14/obj 2023-11-02T18:00:51.292 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W/sSs/NFhMzrnNUeNirNUAyfhtg= 2023-11-02T18:00:51.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:00:51 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W/sSs/NFhMzrnNUeNirNUAyfhtg='} 2023-11-02T18:00:51.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d0142d66791647e-006543e3d3-1111-a1'), ('Date', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:00:51.308 DEBUG:rgw_multi.tests:created initial version id=None 2023-11-02T18:00:51.308 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-14 2023-11-02T18:00:51.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:00:51.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:00:51.310 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:00:51.362 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:00:51.362 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:00:51.430 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:51.422+0000 7f6593c80f40 1 waiting to reach incremental sync.. 2023-11-02T18:00:56.433 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:00:56.422+0000 7f6593c80f40 1 waiting to reach incremental sync.. 2023-11-02T18:01:01.435 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:01.426+0000 7f6593c80f40 1 bucket sync caught up with source: 2023-11-02T18:01:01.435 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.7.6, , ] 2023-11-02T18:01:01.435 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000001.7.6, , ] 2023-11-02T18:01:01.436 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:01.426+0000 7f6593c80f40 0 bucket checkpoint complete 2023-11-02T18:01:01.465 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-14 2023-11-02T18:01:01.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:01:01.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:01:01.466 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:01:01.519 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:01:01.520 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:01:01.574 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:01.566+0000 7f841e641f40 1 waiting to reach incremental sync.. 2023-11-02T18:01:06.575 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:06.566+0000 7f841e641f40 1 waiting to reach incremental sync.. 2023-11-02T18:01:11.576 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:11.566+0000 7f841e641f40 1 waiting to reach incremental sync.. 2023-11-02T18:01:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:16.566+0000 7f841e641f40 1 waiting to reach incremental sync.. 2023-11-02T18:01:21.580 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:21.570+0000 7f841e641f40 1 bucket sync caught up with source: 2023-11-02T18:01:21.580 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.15.6, , ] 2023-11-02T18:01:21.580 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000001.15.6, , ] 2023-11-02T18:01:21.581 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:21.570+0000 7f841e641f40 0 bucket checkpoint complete 2023-11-02T18:01:21.608 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-14 zones={a2, a1} 2023-11-02T18:01:21.609 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:01:21.609 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:01:21.610 DEBUG:boto:Method: HEAD 2023-11-02T18:01:21.611 DEBUG:boto:Path: /ibtljn-14/ 2023-11-02T18:01:21.612 DEBUG:boto:Data: 2023-11-02T18:01:21.612 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.613 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:01:21.614 DEBUG:boto:Port: 8001 2023-11-02T18:01:21.614 DEBUG:boto:Params: {} 2023-11-02T18:01:21.615 DEBUG:boto:Token: None 2023-11-02T18:01:21.616 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/ 2023-11-02T18:01:21.617 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DmsBMlXidbxWXZGJTp867hgEkyc= 2023-11-02T18:01:21.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DmsBMlXidbxWXZGJTp867hgEkyc=', 'Content-Length': '0'} 2023-11-02T18:01:21.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c09432070073610c-006543e3f1-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.623 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:01:21.623 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:01:21.624 DEBUG:boto:Method: HEAD 2023-11-02T18:01:21.625 DEBUG:boto:Path: /ibtljn-14/ 2023-11-02T18:01:21.625 DEBUG:boto:Data: 2023-11-02T18:01:21.626 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.627 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:21.627 DEBUG:boto:Port: 8000 2023-11-02T18:01:21.628 DEBUG:boto:Params: {} 2023-11-02T18:01:21.629 DEBUG:boto:Token: None 2023-11-02T18:01:21.629 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/ 2023-11-02T18:01:21.630 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DmsBMlXidbxWXZGJTp867hgEkyc= 2023-11-02T18:01:21.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DmsBMlXidbxWXZGJTp867hgEkyc=', 'Content-Length': '0'} 2023-11-02T18:01:21.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bb60eefcc469674-006543e3f1-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.635 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:01:21.636 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:01:21.636 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:01:21.637 DEBUG:boto:path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:01:21.637 DEBUG:boto:auth_path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:01:21.638 DEBUG:boto:Method: GET 2023-11-02T18:01:21.638 DEBUG:boto:Path: /ibtljn-14/?versions&max-keys=999 2023-11-02T18:01:21.639 DEBUG:boto:Data: 2023-11-02T18:01:21.640 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.640 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:01:21.640 DEBUG:boto:Port: 8001 2023-11-02T18:01:21.641 DEBUG:boto:Params: {} 2023-11-02T18:01:21.642 DEBUG:boto:Token: None 2023-11-02T18:01:21.642 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/?versions 2023-11-02T18:01:21.643 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lTQqtZAbP7Hg61sWDdVFEOeGLnQ= 2023-11-02T18:01:21.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lTQqtZAbP7Hg61sWDdVFEOeGLnQ=', 'Content-Length': '0'} 2023-11-02T18:01:21.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ceb03771eb68cf3-006543e3f1-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.649 DEBUG:boto:b'ibtljn-14999falseobjnulltrue2023-11-02T18:00:51.289Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:01:21.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:01:21.651 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:01:21.652 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:01:21.652 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:01:21.653 DEBUG:boto:path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:01:21.653 DEBUG:boto:auth_path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:01:21.654 DEBUG:boto:Method: GET 2023-11-02T18:01:21.654 DEBUG:boto:Path: /ibtljn-14/?versions&max-keys=999 2023-11-02T18:01:21.655 DEBUG:boto:Data: 2023-11-02T18:01:21.655 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.655 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:21.656 DEBUG:boto:Port: 8000 2023-11-02T18:01:21.657 DEBUG:boto:Params: {} 2023-11-02T18:01:21.657 DEBUG:boto:Token: None 2023-11-02T18:01:21.658 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/?versions 2023-11-02T18:01:21.658 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lTQqtZAbP7Hg61sWDdVFEOeGLnQ= 2023-11-02T18:01:21.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lTQqtZAbP7Hg61sWDdVFEOeGLnQ=', 'Content-Length': '0'} 2023-11-02T18:01:21.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7bbdd217e95f4e-006543e3f1-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.706 DEBUG:boto:b'ibtljn-14999falseobjnulltrue2023-11-02T18:00:51.289Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:01:21.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:01:21.708 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:01:21.709 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:01:21.709 DEBUG:boto:path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:01:21.710 DEBUG:boto:auth_path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:01:21.711 DEBUG:boto:Method: GET 2023-11-02T18:01:21.711 DEBUG:boto:Path: /ibtljn-14/?versions&max-keys=999 2023-11-02T18:01:21.712 DEBUG:boto:Data: 2023-11-02T18:01:21.712 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.713 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:01:21.713 DEBUG:boto:Port: 8001 2023-11-02T18:01:21.714 DEBUG:boto:Params: {} 2023-11-02T18:01:21.715 DEBUG:boto:Token: None 2023-11-02T18:01:21.715 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/?versions 2023-11-02T18:01:21.716 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lTQqtZAbP7Hg61sWDdVFEOeGLnQ= 2023-11-02T18:01:21.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lTQqtZAbP7Hg61sWDdVFEOeGLnQ=', 'Content-Length': '0'} 2023-11-02T18:01:21.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a842e846e04f54d-006543e3f1-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.723 DEBUG:boto:b'ibtljn-14999falseobjnulltrue2023-11-02T18:00:51.289Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:01:21.724 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:01:21.724 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:01:21.725 DEBUG:boto:path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:01:21.726 DEBUG:boto:auth_path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:01:21.726 DEBUG:boto:Method: GET 2023-11-02T18:01:21.726 DEBUG:boto:Path: /ibtljn-14/?versions&max-keys=999 2023-11-02T18:01:21.727 DEBUG:boto:Data: 2023-11-02T18:01:21.728 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.728 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:21.728 DEBUG:boto:Port: 8000 2023-11-02T18:01:21.729 DEBUG:boto:Params: {} 2023-11-02T18:01:21.730 DEBUG:boto:Token: None 2023-11-02T18:01:21.731 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/?versions 2023-11-02T18:01:21.731 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lTQqtZAbP7Hg61sWDdVFEOeGLnQ= 2023-11-02T18:01:21.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lTQqtZAbP7Hg61sWDdVFEOeGLnQ=', 'Content-Length': '0'} 2023-11-02T18:01:21.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fea2c92e6804ca44-006543e3f1-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.737 DEBUG:boto:b'ibtljn-14999falseobjnulltrue2023-11-02T18:00:51.289Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:01:21.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:01:21.739 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:01:21.740 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:01:21.741 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.741 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.742 DEBUG:boto:Method: GET 2023-11-02T18:01:21.742 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.743 DEBUG:boto:Data: 2023-11-02T18:01:21.744 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.745 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:01:21.745 DEBUG:boto:Port: 8001 2023-11-02T18:01:21.746 DEBUG:boto:Params: {} 2023-11-02T18:01:21.746 DEBUG:boto:Token: None 2023-11-02T18:01:21.747 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.747 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CRFxi5O8z/62z/fHZcz5FBYJZ/o= 2023-11-02T18:01:21.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CRFxi5O8z/62z/fHZcz5FBYJZ/o=', 'Content-Length': '0'} 2023-11-02T18:01:21.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b217320c5953a0fa-006543e3f1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.752 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:01:21.753 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:01:21.754 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.755 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.755 DEBUG:boto:Method: GET 2023-11-02T18:01:21.756 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.756 DEBUG:boto:Data: 2023-11-02T18:01:21.757 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.758 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:21.758 DEBUG:boto:Port: 8000 2023-11-02T18:01:21.759 DEBUG:boto:Params: {} 2023-11-02T18:01:21.759 DEBUG:boto:Token: None 2023-11-02T18:01:21.760 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.760 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CRFxi5O8z/62z/fHZcz5FBYJZ/o= 2023-11-02T18:01:21.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CRFxi5O8z/62z/fHZcz5FBYJZ/o=', 'Content-Length': '0'} 2023-11-02T18:01:21.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f75a1c1d1c72c73c-006543e3f1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.764 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:01:21.765 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:01:21.765 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.765 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.766 DEBUG:boto:Method: HEAD 2023-11-02T18:01:21.766 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.767 DEBUG:boto:Data: 2023-11-02T18:01:21.767 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.768 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:01:21.768 DEBUG:boto:Port: 8001 2023-11-02T18:01:21.769 DEBUG:boto:Params: {} 2023-11-02T18:01:21.770 DEBUG:boto:Token: None 2023-11-02T18:01:21.771 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.773 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wDQepdtIa7nbtMROk8trfIfDAq4= 2023-11-02T18:01:21.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wDQepdtIa7nbtMROk8trfIfDAq4=', 'Content-Length': '0'} 2023-11-02T18:01:21.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000005a1febb862ddc20-006543e3f1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.778 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:01:21.778 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:01:21.778 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.779 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.779 DEBUG:boto:Method: HEAD 2023-11-02T18:01:21.780 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.781 DEBUG:boto:Data: 2023-11-02T18:01:21.781 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.782 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:21.783 DEBUG:boto:Port: 8000 2023-11-02T18:01:21.783 DEBUG:boto:Params: {} 2023-11-02T18:01:21.784 DEBUG:boto:Token: None 2023-11-02T18:01:21.785 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wDQepdtIa7nbtMROk8trfIfDAq4= 2023-11-02T18:01:21.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wDQepdtIa7nbtMROk8trfIfDAq4=', 'Content-Length': '0'} 2023-11-02T18:01:21.789 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a24c30a5e6ed7400-006543e3f1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:01:21.790 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:01:21.790 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:01:21.791 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.791 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.791 DEBUG:boto:Method: GET 2023-11-02T18:01:21.792 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.792 DEBUG:boto:Data: 2023-11-02T18:01:21.793 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.793 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:01:21.794 DEBUG:boto:Port: 8001 2023-11-02T18:01:21.794 DEBUG:boto:Params: {} 2023-11-02T18:01:21.795 DEBUG:boto:Token: None 2023-11-02T18:01:21.795 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.796 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CRFxi5O8z/62z/fHZcz5FBYJZ/o= 2023-11-02T18:01:21.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CRFxi5O8z/62z/fHZcz5FBYJZ/o=', 'Content-Length': '0'} 2023-11-02T18:01:21.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000523e2df5177f02c6-006543e3f1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.799 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:01:21.800 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:01:21.800 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.801 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.801 DEBUG:boto:Method: GET 2023-11-02T18:01:21.802 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.802 DEBUG:boto:Data: 2023-11-02T18:01:21.803 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.803 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:21.803 DEBUG:boto:Port: 8000 2023-11-02T18:01:21.804 DEBUG:boto:Params: {} 2023-11-02T18:01:21.804 DEBUG:boto:Token: None 2023-11-02T18:01:21.805 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:01:21.805 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CRFxi5O8z/62z/fHZcz5FBYJZ/o= 2023-11-02T18:01:21.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CRFxi5O8z/62z/fHZcz5FBYJZ/o=', 'Content-Length': '0'} 2023-11-02T18:01:21.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033b49973dcbb6fdd-006543e3f1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.809 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:01:21.810 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:01:21.810 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:01:21.811 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:01:21.811 DEBUG:boto:Method: GET 2023-11-02T18:01:21.812 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:01:21.812 DEBUG:boto:Data: 2023-11-02T18:01:21.813 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.813 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:01:21.813 DEBUG:boto:Port: 8001 2023-11-02T18:01:21.814 DEBUG:boto:Params: {} 2023-11-02T18:01:21.814 DEBUG:boto:Token: None 2023-11-02T18:01:21.815 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/ 2023-11-02T18:01:21.815 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n1mEic90C0f/c5KTRV8RPss/VV4= 2023-11-02T18:01:21.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n1mEic90C0f/c5KTRV8RPss/VV4=', 'Content-Length': '0'} 2023-11-02T18:01:21.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008751062b13961fc5-006543e3f1-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.862 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:00:51.289Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:01:21.863 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:01:21.863 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:01:21.864 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:01:21.864 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:01:21.865 DEBUG:boto:Method: GET 2023-11-02T18:01:21.865 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:01:21.866 DEBUG:boto:Data: 2023-11-02T18:01:21.866 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.867 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:21.867 DEBUG:boto:Port: 8000 2023-11-02T18:01:21.868 DEBUG:boto:Params: {} 2023-11-02T18:01:21.868 DEBUG:boto:Token: None 2023-11-02T18:01:21.869 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/ 2023-11-02T18:01:21.869 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n1mEic90C0f/c5KTRV8RPss/VV4= 2023-11-02T18:01:21.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n1mEic90C0f/c5KTRV8RPss/VV4=', 'Content-Length': '0'} 2023-11-02T18:01:21.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097bdaae4eb48fc62-006543e3f1-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.875 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:00:51.289Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:01:21.876 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:01:21.876 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:01:21.876 DEBUG:boto:Method: HEAD 2023-11-02T18:01:21.877 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:01:21.877 DEBUG:boto:Data: 2023-11-02T18:01:21.878 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.878 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:01:21.879 DEBUG:boto:Port: 8001 2023-11-02T18:01:21.879 DEBUG:boto:Params: {} 2023-11-02T18:01:21.880 DEBUG:boto:Token: None 2023-11-02T18:01:21.880 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/obj 2023-11-02T18:01:21.880 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:y61d7iycMCssdKaqkFn1iCx2zE0= 2023-11-02T18:01:21.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:y61d7iycMCssdKaqkFn1iCx2zE0=', 'Content-Length': '0'} 2023-11-02T18:01:21.884 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e5b03d03943e117-006543e3f1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.884 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:01:21.885 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:01:21.885 DEBUG:boto:Method: HEAD 2023-11-02T18:01:21.886 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:01:21.886 DEBUG:boto:Data: 2023-11-02T18:01:21.887 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.887 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:21.888 DEBUG:boto:Port: 8000 2023-11-02T18:01:21.888 DEBUG:boto:Params: {} 2023-11-02T18:01:21.888 DEBUG:boto:Token: None 2023-11-02T18:01:21.889 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/obj 2023-11-02T18:01:21.889 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:y61d7iycMCssdKaqkFn1iCx2zE0= 2023-11-02T18:01:21.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:y61d7iycMCssdKaqkFn1iCx2zE0=', 'Content-Length': '0'} 2023-11-02T18:01:21.893 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008d25bd0b1ca8677-006543e3f1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:01:21.894 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:01:21.894 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:01:21.895 DEBUG:boto:Method: GET 2023-11-02T18:01:21.895 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:01:21.896 DEBUG:boto:Data: 2023-11-02T18:01:21.896 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.896 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:01:21.897 DEBUG:boto:Port: 8001 2023-11-02T18:01:21.897 DEBUG:boto:Params: {} 2023-11-02T18:01:21.898 DEBUG:boto:Token: None 2023-11-02T18:01:21.898 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/obj 2023-11-02T18:01:21.899 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KqjsoaC97k3DHTcLWTmaPoxgFr8= 2023-11-02T18:01:21.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KqjsoaC97k3DHTcLWTmaPoxgFr8=', 'Content-Length': '0'} 2023-11-02T18:01:21.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008743a923391e2367-006543e3f1-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.903 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:01:21.903 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:01:21.904 DEBUG:boto:Method: GET 2023-11-02T18:01:21.904 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:01:21.904 DEBUG:boto:Data: 2023-11-02T18:01:21.905 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.905 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:21.906 DEBUG:boto:Port: 8000 2023-11-02T18:01:21.906 DEBUG:boto:Params: {} 2023-11-02T18:01:21.907 DEBUG:boto:Token: None 2023-11-02T18:01:21.907 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/obj 2023-11-02T18:01:21.908 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KqjsoaC97k3DHTcLWTmaPoxgFr8= 2023-11-02T18:01:21.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KqjsoaC97k3DHTcLWTmaPoxgFr8=', 'Content-Length': '0'} 2023-11-02T18:01:21.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d5e4521baa077bfb-006543e3f1-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.912 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:01:21.912 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:01:21.912 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:01:21.913 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:01:21.913 DEBUG:boto:Method: GET 2023-11-02T18:01:21.914 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:01:21.914 DEBUG:boto:Data: 2023-11-02T18:01:21.915 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.915 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:01:21.916 DEBUG:boto:Port: 8001 2023-11-02T18:01:21.916 DEBUG:boto:Params: {} 2023-11-02T18:01:21.916 DEBUG:boto:Token: None 2023-11-02T18:01:21.917 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/ 2023-11-02T18:01:21.917 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n1mEic90C0f/c5KTRV8RPss/VV4= 2023-11-02T18:01:21.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n1mEic90C0f/c5KTRV8RPss/VV4=', 'Content-Length': '0'} 2023-11-02T18:01:21.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abdc073a454f823c-006543e3f1-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.924 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:00:51.289Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:01:21.924 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:01:21.925 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:01:21.925 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:01:21.926 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:01:21.926 DEBUG:boto:Method: GET 2023-11-02T18:01:21.926 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:01:21.927 DEBUG:boto:Data: 2023-11-02T18:01:21.927 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.928 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:21.928 DEBUG:boto:Port: 8000 2023-11-02T18:01:21.929 DEBUG:boto:Params: {} 2023-11-02T18:01:21.929 DEBUG:boto:Token: None 2023-11-02T18:01:21.930 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/ 2023-11-02T18:01:21.930 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n1mEic90C0f/c5KTRV8RPss/VV4= 2023-11-02T18:01:21.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n1mEic90C0f/c5KTRV8RPss/VV4=', 'Content-Length': '0'} 2023-11-02T18:01:21.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ad0f82f6f69d273-006543e3f1-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.936 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:00:51.289Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:01:21.937 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-14 zones={a2, a1} 2023-11-02T18:01:21.937 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:01:21.938 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:01:21.938 DEBUG:boto:path=/ibtljn-14/?versioning 2023-11-02T18:01:21.939 DEBUG:boto:auth_path=/ibtljn-14/?versioning 2023-11-02T18:01:21.939 DEBUG:boto:Method: PUT 2023-11-02T18:01:21.940 DEBUG:boto:Path: /ibtljn-14/?versioning 2023-11-02T18:01:21.940 DEBUG:boto:Data: Enabled Disabled 2023-11-02T18:01:21.940 DEBUG:boto:Headers: {} 2023-11-02T18:01:21.941 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:21.941 DEBUG:boto:Port: 8000 2023-11-02T18:01:21.942 DEBUG:boto:Params: {} 2023-11-02T18:01:21.942 DEBUG:boto:Token: None 2023-11-02T18:01:21.942 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:01:21 GMT /ibtljn-14/?versioning 2023-11-02T18:01:21.943 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V5LSG5REMAfhrr5qcw5/FQjmXK0= 2023-11-02T18:01:21.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V5LSG5REMAfhrr5qcw5/FQjmXK0=', 'Content-Length': '228'} 2023-11-02T18:01:21.955 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000beac39493f1bfc84-006543e3f1-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:01:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:21.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:01:21.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:01:21.957 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:01:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:01:22.004 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:01:22.062 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:01:22.089 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:01:22.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:01:22.089 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:01:22.089 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:01:22.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:01:22.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:01:22.090 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:01:22.090 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:01:22.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:01:22.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:01:22.091 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948081.946579_86.1", 2023-11-02T18:01:22.091 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:01:21.946579Z" 2023-11-02T18:01:22.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:01:22.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:01:22.092 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:01:22.092 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:01:22.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:01:22.093 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:01:22.094 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948081.946579_86.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:01:22.094 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:01:22.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:01:22.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:01:22.096 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:01:22.140 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:01:22.140 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:01:22.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:22.182+0000 7f736c8dcf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:01:22.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:22.182+0000 7f736c8dcf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:01:22.219 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:01:22.219 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:01:22.219 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:01:22.219 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:01:22.220 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:01:22.220 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:01:22.220 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:01:22.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:01:22.221 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:01:22.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:01:22.221 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:01:22.221 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:01:22.221 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:01:22.222 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:01:22.222 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:01:22.222 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:01:22.222 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:01:22.223 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:01:22.223 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:01:22.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:01:22.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:01:22.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:01:22.224 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:01:22.224 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:01:22.224 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:01:22.224 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:01:22.225 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:01:22.225 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:01:22.225 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:01:22.225 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:01:22.225 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:01:22.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:01:22.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:01:22.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:01:22.226 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:01:22.227 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:01:22.227 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:01:22.227 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948081.946579_86.1", 2023-11-02T18:01:22.227 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:01:22.227 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:01:22.228 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:01:22.228 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:01:21.946579Z", 2023-11-02T18:01:22.228 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:01:22.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:01:22.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:01:22.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:01:22.229 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:01:22.229 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:01:22.229 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:01:22.230 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:01:22.230 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:01:22.230 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:01:22.230 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:01:22.231 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:01:22.231 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:01:22.231 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:01:22.231 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:01:22.231 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:01:22.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:01:22.232 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:01:22.232 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:01:22.232 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:01:22.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:01:22.233 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:01:22Z" 2023-11-02T18:01:22.233 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:01:22.234 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948081.946579_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:01:21.946579Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:01:22Z" } 2023-11-02T18:01:22.235 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948081.946579_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:01:21.946579Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:01:22.235 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948081.946579_86.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:01:22.235 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948081.946579_86.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:01:22.236 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:01:22.236 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:01:22.237 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:01:22.237 DEBUG:boto:Method: HEAD 2023-11-02T18:01:22.237 DEBUG:boto:Path: /ibtljn-14/ 2023-11-02T18:01:22.238 DEBUG:boto:Data: 2023-11-02T18:01:22.238 DEBUG:boto:Headers: {} 2023-11-02T18:01:22.238 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:22.239 DEBUG:boto:Port: 8000 2023-11-02T18:01:22.239 DEBUG:boto:Params: {} 2023-11-02T18:01:22.240 DEBUG:boto:Token: None 2023-11-02T18:01:22.240 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:01:22 GMT /ibtljn-14/ 2023-11-02T18:01:22.241 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1DDI6XXJTNKro5xztVSKE4aIBw8= 2023-11-02T18:01:22.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:01:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1DDI6XXJTNKro5xztVSKE4aIBw8=', 'Content-Length': '0'} 2023-11-02T18:01:22.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a00890eea4a6a1c8-006543e3f2-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:22.246 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:01:22.247 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:01:22.247 DEBUG:boto:Method: PUT 2023-11-02T18:01:22.247 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:01:22.248 DEBUG:boto:Data: 2023-11-02T18:01:22.248 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-11-02T18:01:22.249 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:01:22.249 DEBUG:boto:Port: 8000 2023-11-02T18:01:22.249 DEBUG:boto:Params: {} 2023-11-02T18:01:22.250 DEBUG:boto:Token: None 2023-11-02T18:01:22.250 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 02 Nov 2023 18:01:22 GMT /ibtljn-14/obj 2023-11-02T18:01:22.251 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tHDKqrJwn/qhLlRBNYz8ukQIlac= 2023-11-02T18:01:22.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:01:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tHDKqrJwn/qhLlRBNYz8ukQIlac='} 2023-11-02T18:01:22.275 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-amz-request-id', 'tx000008dce9c27b7a3b6ea-006543e3f2-1111-a1'), ('Date', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:01:22.276 DEBUG:rgw_multi.tests:created new version id=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:01:22.276 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-14 2023-11-02T18:01:22.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:01:22.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:01:22.277 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:01:22.329 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:01:22.329 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:01:22.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:22.386+0000 7ff1d4835f40 1 waiting for incremental sync to catch up: 2023-11-02T18:01:22.395 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.7.6, , ] 2023-11-02T18:01:22.395 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.10.13, , ] 2023-11-02T18:01:27.397 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:27.386+0000 7ff1d4835f40 1 waiting for incremental sync to catch up: 2023-11-02T18:01:27.397 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.7.6, , ] 2023-11-02T18:01:27.398 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.10.13, , ] 2023-11-02T18:01:32.399 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:32.390+0000 7ff1d4835f40 1 waiting for incremental sync to catch up: 2023-11-02T18:01:32.400 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.7.6, , ] 2023-11-02T18:01:32.400 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.10.13, , ] 2023-11-02T18:01:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:37.390+0000 7ff1d4835f40 1 waiting for incremental sync to catch up: 2023-11-02T18:01:37.402 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.7.6, , ] 2023-11-02T18:01:37.402 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.10.13, , ] 2023-11-02T18:01:42.404 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:42.394+0000 7ff1d4835f40 1 bucket sync caught up with source: 2023-11-02T18:01:42.404 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.10.13, , ] 2023-11-02T18:01:42.405 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.10.13, , ] 2023-11-02T18:01:42.405 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:42.394+0000 7ff1d4835f40 0 bucket checkpoint complete 2023-11-02T18:01:42.433 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-14 2023-11-02T18:01:42.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:01:42.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:01:42.435 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:01:42.487 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:01:42.488 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:01:42.552 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:42.542+0000 7f6cdc23af40 1 waiting for incremental sync to catch up: 2023-11-02T18:01:42.552 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.15.6, , ] 2023-11-02T18:01:42.552 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-11-02T18:01:47.558 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:47.550+0000 7f6cdc23af40 1 waiting for incremental sync to catch up: 2023-11-02T18:01:47.558 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.15.6, , ] 2023-11-02T18:01:47.559 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-11-02T18:01:52.561 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:52.550+0000 7f6cdc23af40 1 waiting for incremental sync to catch up: 2023-11-02T18:01:52.561 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.15.6, , ] 2023-11-02T18:01:52.562 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-11-02T18:01:57.563 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:01:57.554+0000 7f6cdc23af40 1 waiting for incremental sync to catch up: 2023-11-02T18:01:57.564 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.15.6, , ] 2023-11-02T18:01:57.564 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-11-02T18:02:02.566 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:02.558+0000 7f6cdc23af40 1 bucket sync caught up with source: 2023-11-02T18:02:02.566 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.19.13, , ] 2023-11-02T18:02:02.567 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-11-02T18:02:02.567 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:02.558+0000 7f6cdc23af40 0 bucket checkpoint complete 2023-11-02T18:02:02.591 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-14 zones={a2, a1} 2023-11-02T18:02:02.592 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:02.592 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:02.593 DEBUG:boto:Method: HEAD 2023-11-02T18:02:02.593 DEBUG:boto:Path: /ibtljn-14/ 2023-11-02T18:02:02.594 DEBUG:boto:Data: 2023-11-02T18:02:02.594 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.594 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.595 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.595 DEBUG:boto:Params: {} 2023-11-02T18:02:02.596 DEBUG:boto:Token: None 2023-11-02T18:02:02.596 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/ 2023-11-02T18:02:02.597 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OreBxARA+65Bh8WJuGntsU2G/S8= 2023-11-02T18:02:02.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OreBxARA+65Bh8WJuGntsU2G/S8=', 'Content-Length': '0'} 2023-11-02T18:02:02.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab154816d80d5853-006543e41a-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.602 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:02.602 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:02.602 DEBUG:boto:Method: HEAD 2023-11-02T18:02:02.603 DEBUG:boto:Path: /ibtljn-14/ 2023-11-02T18:02:02.603 DEBUG:boto:Data: 2023-11-02T18:02:02.604 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.604 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.605 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.605 DEBUG:boto:Params: {} 2023-11-02T18:02:02.605 DEBUG:boto:Token: None 2023-11-02T18:02:02.606 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/ 2023-11-02T18:02:02.606 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OreBxARA+65Bh8WJuGntsU2G/S8= 2023-11-02T18:02:02.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OreBxARA+65Bh8WJuGntsU2G/S8=', 'Content-Length': '0'} 2023-11-02T18:02:02.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087abe96330217652-006543e41a-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.611 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:02:02.612 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:02.612 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:02.613 DEBUG:boto:path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:02.613 DEBUG:boto:auth_path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:02.613 DEBUG:boto:Method: GET 2023-11-02T18:02:02.614 DEBUG:boto:Path: /ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:02.614 DEBUG:boto:Data: 2023-11-02T18:02:02.615 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.615 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.616 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.616 DEBUG:boto:Params: {} 2023-11-02T18:02:02.616 DEBUG:boto:Token: None 2023-11-02T18:02:02.617 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/?versions 2023-11-02T18:02:02.617 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2lAHmK3+2fIiGXtVDk78FJ/UXss= 2023-11-02T18:02:02.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2lAHmK3+2fIiGXtVDk78FJ/UXss=', 'Content-Length': '0'} 2023-11-02T18:02:02.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070cef14201483f08-006543e41a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.623 DEBUG:boto:b'ibtljn-14999falseobjnXZq0ifaOrZEYes8.J-UF5LgfHuiXUltrue2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-11-02T18:00:51.289Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:02.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:02:02.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:02:02.624 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:02:02.624 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:02.625 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:02.625 DEBUG:boto:path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:02.626 DEBUG:boto:auth_path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:02.626 DEBUG:boto:Method: GET 2023-11-02T18:02:02.627 DEBUG:boto:Path: /ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:02.627 DEBUG:boto:Data: 2023-11-02T18:02:02.627 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.628 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.628 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.628 DEBUG:boto:Params: {} 2023-11-02T18:02:02.629 DEBUG:boto:Token: None 2023-11-02T18:02:02.629 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/?versions 2023-11-02T18:02:02.630 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2lAHmK3+2fIiGXtVDk78FJ/UXss= 2023-11-02T18:02:02.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2lAHmK3+2fIiGXtVDk78FJ/UXss=', 'Content-Length': '0'} 2023-11-02T18:02:02.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dd6ff7f24e4799d-006543e41a-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.635 DEBUG:boto:b'ibtljn-14999falseobjnXZq0ifaOrZEYes8.J-UF5LgfHuiXUltrue2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-11-02T18:00:51.289Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:02.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:02:02.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:02:02.637 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:02.637 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:02.638 DEBUG:boto:path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:02.638 DEBUG:boto:auth_path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:02.639 DEBUG:boto:Method: GET 2023-11-02T18:02:02.639 DEBUG:boto:Path: /ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:02.640 DEBUG:boto:Data: 2023-11-02T18:02:02.640 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.641 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.641 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.641 DEBUG:boto:Params: {} 2023-11-02T18:02:02.642 DEBUG:boto:Token: None 2023-11-02T18:02:02.642 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/?versions 2023-11-02T18:02:02.643 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2lAHmK3+2fIiGXtVDk78FJ/UXss= 2023-11-02T18:02:02.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2lAHmK3+2fIiGXtVDk78FJ/UXss=', 'Content-Length': '0'} 2023-11-02T18:02:02.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e877648e4b04f89-006543e41a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.690 DEBUG:boto:b'ibtljn-14999falseobjnXZq0ifaOrZEYes8.J-UF5LgfHuiXUltrue2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-11-02T18:00:51.289Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:02.691 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:02.692 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:02.692 DEBUG:boto:path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:02.693 DEBUG:boto:auth_path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:02.693 DEBUG:boto:Method: GET 2023-11-02T18:02:02.693 DEBUG:boto:Path: /ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:02.694 DEBUG:boto:Data: 2023-11-02T18:02:02.694 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.695 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.695 DEBUG:boto:Params: {} 2023-11-02T18:02:02.696 DEBUG:boto:Token: None 2023-11-02T18:02:02.696 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/?versions 2023-11-02T18:02:02.697 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2lAHmK3+2fIiGXtVDk78FJ/UXss= 2023-11-02T18:02:02.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2lAHmK3+2fIiGXtVDk78FJ/UXss=', 'Content-Length': '0'} 2023-11-02T18:02:02.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006beb29990845082a-006543e41a-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.703 DEBUG:boto:b'ibtljn-14999falseobjnXZq0ifaOrZEYes8.J-UF5LgfHuiXUltrue2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-11-02T18:00:51.289Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:02.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:02:02.704 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.704 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.705 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.705 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.706 DEBUG:boto:Method: GET 2023-11-02T18:02:02.706 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.707 DEBUG:boto:Data: 2023-11-02T18:02:02.707 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.708 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.708 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.708 DEBUG:boto:Params: {} 2023-11-02T18:02:02.709 DEBUG:boto:Token: None 2023-11-02T18:02:02.709 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.710 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NvWPbuecSSHW8gLPm4LZxxfMMx4= 2023-11-02T18:02:02.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NvWPbuecSSHW8gLPm4LZxxfMMx4=', 'Content-Length': '0'} 2023-11-02T18:02:02.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002101ce6977260a80-006543e41a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.714 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.715 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.715 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.716 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.716 DEBUG:boto:Method: GET 2023-11-02T18:02:02.717 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.717 DEBUG:boto:Data: 2023-11-02T18:02:02.717 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.718 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.718 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.719 DEBUG:boto:Params: {} 2023-11-02T18:02:02.719 DEBUG:boto:Token: None 2023-11-02T18:02:02.720 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.720 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NvWPbuecSSHW8gLPm4LZxxfMMx4= 2023-11-02T18:02:02.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NvWPbuecSSHW8gLPm4LZxxfMMx4=', 'Content-Length': '0'} 2023-11-02T18:02:02.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d71b35041a01cc49-006543e41a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.724 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.725 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.725 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.725 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.726 DEBUG:boto:Method: HEAD 2023-11-02T18:02:02.726 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.727 DEBUG:boto:Data: 2023-11-02T18:02:02.727 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.727 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.728 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.728 DEBUG:boto:Params: {} 2023-11-02T18:02:02.729 DEBUG:boto:Token: None 2023-11-02T18:02:02.729 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.730 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u/DrzQwXPG+kSDhliYZRWjsVScQ= 2023-11-02T18:02:02.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u/DrzQwXPG+kSDhliYZRWjsVScQ=', 'Content-Length': '0'} 2023-11-02T18:02:02.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089d778b91b11e968-006543e41a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.734 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.734 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.735 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.735 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.735 DEBUG:boto:Method: HEAD 2023-11-02T18:02:02.736 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.736 DEBUG:boto:Data: 2023-11-02T18:02:02.737 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.737 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.738 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.738 DEBUG:boto:Params: {} 2023-11-02T18:02:02.739 DEBUG:boto:Token: None 2023-11-02T18:02:02.739 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u/DrzQwXPG+kSDhliYZRWjsVScQ= 2023-11-02T18:02:02.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u/DrzQwXPG+kSDhliYZRWjsVScQ=', 'Content-Length': '0'} 2023-11-02T18:02:02.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d8783a6618ba829-006543e41a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:02:02.744 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.744 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.745 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.745 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.746 DEBUG:boto:Method: GET 2023-11-02T18:02:02.746 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.747 DEBUG:boto:Data: 2023-11-02T18:02:02.747 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.747 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.748 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.748 DEBUG:boto:Params: {} 2023-11-02T18:02:02.749 DEBUG:boto:Token: None 2023-11-02T18:02:02.749 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.750 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NvWPbuecSSHW8gLPm4LZxxfMMx4= 2023-11-02T18:02:02.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NvWPbuecSSHW8gLPm4LZxxfMMx4=', 'Content-Length': '0'} 2023-11-02T18:02:02.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a66c6dba17809da9-006543e41a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.754 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.755 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.755 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.755 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.756 DEBUG:boto:Method: GET 2023-11-02T18:02:02.756 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.757 DEBUG:boto:Data: 2023-11-02T18:02:02.757 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.758 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.758 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.759 DEBUG:boto:Params: {} 2023-11-02T18:02:02.759 DEBUG:boto:Token: None 2023-11-02T18:02:02.760 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.760 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NvWPbuecSSHW8gLPm4LZxxfMMx4= 2023-11-02T18:02:02.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NvWPbuecSSHW8gLPm4LZxxfMMx4=', 'Content-Length': '0'} 2023-11-02T18:02:02.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7d4910cc9af130b-006543e41a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.764 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:02.764 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:02.765 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:02.765 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:02.766 DEBUG:boto:Method: GET 2023-11-02T18:02:02.766 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:02.767 DEBUG:boto:Data: 2023-11-02T18:02:02.767 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.768 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.768 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.768 DEBUG:boto:Params: {} 2023-11-02T18:02:02.769 DEBUG:boto:Token: None 2023-11-02T18:02:02.769 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/ 2023-11-02T18:02:02.770 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hauifP89SLezpAImGD3/Jxwvqx8= 2023-11-02T18:02:02.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hauifP89SLezpAImGD3/Jxwvqx8=', 'Content-Length': '0'} 2023-11-02T18:02:02.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066078afa5a4dcb29-006543e41a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.775 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:02.776 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:02.777 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:02.777 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:02.777 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:02.778 DEBUG:boto:Method: GET 2023-11-02T18:02:02.778 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:02.779 DEBUG:boto:Data: 2023-11-02T18:02:02.779 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.780 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.780 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.781 DEBUG:boto:Params: {} 2023-11-02T18:02:02.781 DEBUG:boto:Token: None 2023-11-02T18:02:02.782 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/ 2023-11-02T18:02:02.782 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hauifP89SLezpAImGD3/Jxwvqx8= 2023-11-02T18:02:02.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hauifP89SLezpAImGD3/Jxwvqx8=', 'Content-Length': '0'} 2023-11-02T18:02:02.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001918cec4f2faef56-006543e41a-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.830 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:02.831 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.831 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.832 DEBUG:boto:Method: HEAD 2023-11-02T18:02:02.832 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:02.833 DEBUG:boto:Data: 2023-11-02T18:02:02.833 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.833 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.834 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.834 DEBUG:boto:Params: {} 2023-11-02T18:02:02.835 DEBUG:boto:Token: None 2023-11-02T18:02:02.835 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj 2023-11-02T18:02:02.836 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bIrW+KyDevjIPpDIOOyyn3hq4AA= 2023-11-02T18:02:02.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bIrW+KyDevjIPpDIOOyyn3hq4AA=', 'Content-Length': '0'} 2023-11-02T18:02:02.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb1014aead3f0930-006543e41a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.841 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.842 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.842 DEBUG:boto:Method: HEAD 2023-11-02T18:02:02.843 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:02.843 DEBUG:boto:Data: 2023-11-02T18:02:02.844 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.844 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.844 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.845 DEBUG:boto:Params: {} 2023-11-02T18:02:02.845 DEBUG:boto:Token: None 2023-11-02T18:02:02.846 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj 2023-11-02T18:02:02.846 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bIrW+KyDevjIPpDIOOyyn3hq4AA= 2023-11-02T18:02:02.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bIrW+KyDevjIPpDIOOyyn3hq4AA=', 'Content-Length': '0'} 2023-11-02T18:02:02.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007117f01770c9298a-006543e41a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:02:02.851 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.852 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.852 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.853 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.853 DEBUG:boto:Method: GET 2023-11-02T18:02:02.853 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.854 DEBUG:boto:Data: 2023-11-02T18:02:02.854 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.855 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.855 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.856 DEBUG:boto:Params: {} 2023-11-02T18:02:02.856 DEBUG:boto:Token: None 2023-11-02T18:02:02.857 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.857 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NvWPbuecSSHW8gLPm4LZxxfMMx4= 2023-11-02T18:02:02.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NvWPbuecSSHW8gLPm4LZxxfMMx4=', 'Content-Length': '0'} 2023-11-02T18:02:02.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060d5a57a5170eed5-006543e41a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.861 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.862 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.862 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.863 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.863 DEBUG:boto:Method: GET 2023-11-02T18:02:02.864 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.864 DEBUG:boto:Data: 2023-11-02T18:02:02.864 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.865 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.865 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.866 DEBUG:boto:Params: {} 2023-11-02T18:02:02.866 DEBUG:boto:Token: None 2023-11-02T18:02:02.867 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:02.867 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NvWPbuecSSHW8gLPm4LZxxfMMx4= 2023-11-02T18:02:02.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NvWPbuecSSHW8gLPm4LZxxfMMx4=', 'Content-Length': '0'} 2023-11-02T18:02:02.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b6baf3307c81b84-006543e41a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.872 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:02.872 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:02.873 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:02.873 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:02.873 DEBUG:boto:Method: GET 2023-11-02T18:02:02.874 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:02.874 DEBUG:boto:Data: 2023-11-02T18:02:02.875 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.875 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.876 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.876 DEBUG:boto:Params: {} 2023-11-02T18:02:02.876 DEBUG:boto:Token: None 2023-11-02T18:02:02.877 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/ 2023-11-02T18:02:02.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hauifP89SLezpAImGD3/Jxwvqx8= 2023-11-02T18:02:02.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hauifP89SLezpAImGD3/Jxwvqx8=', 'Content-Length': '0'} 2023-11-02T18:02:02.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5cfe065adee7b4a-006543e41a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.883 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:02.884 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:02.884 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:02.885 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:02.885 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:02.885 DEBUG:boto:Method: GET 2023-11-02T18:02:02.886 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:02.886 DEBUG:boto:Data: 2023-11-02T18:02:02.887 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.887 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.888 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.888 DEBUG:boto:Params: {} 2023-11-02T18:02:02.888 DEBUG:boto:Token: None 2023-11-02T18:02:02.889 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/ 2023-11-02T18:02:02.889 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hauifP89SLezpAImGD3/Jxwvqx8= 2023-11-02T18:02:02.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hauifP89SLezpAImGD3/Jxwvqx8=', 'Content-Length': '0'} 2023-11-02T18:02:02.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007003b34245d01dc-006543e41a-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.938 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:02.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:02:02.939 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.940 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.940 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.941 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.941 DEBUG:boto:Method: GET 2023-11-02T18:02:02.942 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.942 DEBUG:boto:Data: 2023-11-02T18:02:02.942 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.943 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.943 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.944 DEBUG:boto:Params: {} 2023-11-02T18:02:02.944 DEBUG:boto:Token: None 2023-11-02T18:02:02.944 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.945 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BB/Nx31jhPsMnZac4qP/Th4kIsc= 2023-11-02T18:02:02.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BB/Nx31jhPsMnZac4qP/Th4kIsc=', 'Content-Length': '0'} 2023-11-02T18:02:02.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000256a7ac54d9af59e-006543e41a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.949 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.949 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.950 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.950 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.951 DEBUG:boto:Method: GET 2023-11-02T18:02:02.951 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.951 DEBUG:boto:Data: 2023-11-02T18:02:02.952 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.952 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.953 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.953 DEBUG:boto:Params: {} 2023-11-02T18:02:02.953 DEBUG:boto:Token: None 2023-11-02T18:02:02.954 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.954 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BB/Nx31jhPsMnZac4qP/Th4kIsc= 2023-11-02T18:02:02.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BB/Nx31jhPsMnZac4qP/Th4kIsc=', 'Content-Length': '0'} 2023-11-02T18:02:02.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000626ebf2d9fd938dc-006543e41a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.959 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.959 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.960 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.960 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.960 DEBUG:boto:Method: HEAD 2023-11-02T18:02:02.961 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.961 DEBUG:boto:Data: 2023-11-02T18:02:02.962 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.962 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.963 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.963 DEBUG:boto:Params: {} 2023-11-02T18:02:02.963 DEBUG:boto:Token: None 2023-11-02T18:02:02.964 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.964 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I6VO+T4o1JxDpB3A/bJHm/HnzaU= 2023-11-02T18:02:02.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I6VO+T4o1JxDpB3A/bJHm/HnzaU=', 'Content-Length': '0'} 2023-11-02T18:02:02.968 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a41714ebbe634413-006543e41a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.969 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.969 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.970 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.970 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.971 DEBUG:boto:Method: HEAD 2023-11-02T18:02:02.971 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.971 DEBUG:boto:Data: 2023-11-02T18:02:02.972 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.972 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.972 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.973 DEBUG:boto:Params: {} 2023-11-02T18:02:02.973 DEBUG:boto:Token: None 2023-11-02T18:02:02.974 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.974 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I6VO+T4o1JxDpB3A/bJHm/HnzaU= 2023-11-02T18:02:02.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I6VO+T4o1JxDpB3A/bJHm/HnzaU=', 'Content-Length': '0'} 2023-11-02T18:02:02.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048f5055c789333ab-006543e41a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:02:02.980 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.981 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.981 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.982 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.982 DEBUG:boto:Method: GET 2023-11-02T18:02:02.983 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.983 DEBUG:boto:Data: 2023-11-02T18:02:02.984 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.984 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:02.985 DEBUG:boto:Port: 8001 2023-11-02T18:02:02.985 DEBUG:boto:Params: {} 2023-11-02T18:02:02.985 DEBUG:boto:Token: None 2023-11-02T18:02:02.986 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.987 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BB/Nx31jhPsMnZac4qP/Th4kIsc= 2023-11-02T18:02:02.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BB/Nx31jhPsMnZac4qP/Th4kIsc=', 'Content-Length': '0'} 2023-11-02T18:02:02.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000038d714d27dd479f2-006543e41a-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:02.990 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:02.991 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:02.991 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.992 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.992 DEBUG:boto:Method: GET 2023-11-02T18:02:02.993 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.993 DEBUG:boto:Data: 2023-11-02T18:02:02.994 DEBUG:boto:Headers: {} 2023-11-02T18:02:02.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:02.994 DEBUG:boto:Port: 8000 2023-11-02T18:02:02.995 DEBUG:boto:Params: {} 2023-11-02T18:02:02.995 DEBUG:boto:Token: None 2023-11-02T18:02:02.996 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:02 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:02:02.996 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BB/Nx31jhPsMnZac4qP/Th4kIsc= 2023-11-02T18:02:02.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BB/Nx31jhPsMnZac4qP/Th4kIsc=', 'Content-Length': '0'} 2023-11-02T18:02:03.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:00:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f1462976258327ba-006543e41a-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:03.000 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:03.001 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:03.001 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:03.002 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:03.002 DEBUG:boto:Method: GET 2023-11-02T18:02:03.003 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:03.003 DEBUG:boto:Data: 2023-11-02T18:02:03.004 DEBUG:boto:Headers: {} 2023-11-02T18:02:03.004 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:03.005 DEBUG:boto:Port: 8001 2023-11-02T18:02:03.005 DEBUG:boto:Params: {} 2023-11-02T18:02:03.006 DEBUG:boto:Token: None 2023-11-02T18:02:03.006 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:03 GMT /ibtljn-14/ 2023-11-02T18:02:03.007 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C1UZO6BXolicQ39k4sLhWhRrRho= 2023-11-02T18:02:03.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C1UZO6BXolicQ39k4sLhWhRrRho=', 'Content-Length': '0'} 2023-11-02T18:02:03.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffc533ae5877146e-006543e41a-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:03.054 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:03.055 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:03.055 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:03.056 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:03.056 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:03.057 DEBUG:boto:Method: GET 2023-11-02T18:02:03.057 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:03.057 DEBUG:boto:Data: 2023-11-02T18:02:03.058 DEBUG:boto:Headers: {} 2023-11-02T18:02:03.058 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:03.059 DEBUG:boto:Port: 8000 2023-11-02T18:02:03.059 DEBUG:boto:Params: {} 2023-11-02T18:02:03.060 DEBUG:boto:Token: None 2023-11-02T18:02:03.060 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:03 GMT /ibtljn-14/ 2023-11-02T18:02:03.061 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C1UZO6BXolicQ39k4sLhWhRrRho= 2023-11-02T18:02:03.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C1UZO6BXolicQ39k4sLhWhRrRho=', 'Content-Length': '0'} 2023-11-02T18:02:03.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d51929bef9a63948-006543e41b-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:03.067 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:03.068 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:03.068 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:03.068 DEBUG:boto:Method: HEAD 2023-11-02T18:02:03.069 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:03.069 DEBUG:boto:Data: 2023-11-02T18:02:03.070 DEBUG:boto:Headers: {} 2023-11-02T18:02:03.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:03.071 DEBUG:boto:Port: 8001 2023-11-02T18:02:03.071 DEBUG:boto:Params: {} 2023-11-02T18:02:03.072 DEBUG:boto:Token: None 2023-11-02T18:02:03.072 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:03 GMT /ibtljn-14/obj 2023-11-02T18:02:03.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yurfSVPOq08OFWkZxa32xkbF/m8= 2023-11-02T18:02:03.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yurfSVPOq08OFWkZxa32xkbF/m8=', 'Content-Length': '0'} 2023-11-02T18:02:03.077 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ea777302e66b706-006543e41b-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:03.077 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:03.078 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:03.078 DEBUG:boto:Method: HEAD 2023-11-02T18:02:03.079 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:03.079 DEBUG:boto:Data: 2023-11-02T18:02:03.080 DEBUG:boto:Headers: {} 2023-11-02T18:02:03.080 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:03.080 DEBUG:boto:Port: 8000 2023-11-02T18:02:03.081 DEBUG:boto:Params: {} 2023-11-02T18:02:03.081 DEBUG:boto:Token: None 2023-11-02T18:02:03.082 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:03 GMT /ibtljn-14/obj 2023-11-02T18:02:03.083 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yurfSVPOq08OFWkZxa32xkbF/m8= 2023-11-02T18:02:03.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yurfSVPOq08OFWkZxa32xkbF/m8=', 'Content-Length': '0'} 2023-11-02T18:02:03.087 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000091ee0f6700b48a00-006543e41b-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:03.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:02:03.088 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:03.088 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:03.089 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:03.089 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:03.089 DEBUG:boto:Method: GET 2023-11-02T18:02:03.090 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:03.090 DEBUG:boto:Data: 2023-11-02T18:02:03.091 DEBUG:boto:Headers: {} 2023-11-02T18:02:03.091 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:03.092 DEBUG:boto:Port: 8001 2023-11-02T18:02:03.092 DEBUG:boto:Params: {} 2023-11-02T18:02:03.093 DEBUG:boto:Token: None 2023-11-02T18:02:03.093 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:03 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:03.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aaNbnLjIG37XvLvekKRUPorFfmk= 2023-11-02T18:02:03.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aaNbnLjIG37XvLvekKRUPorFfmk=', 'Content-Length': '0'} 2023-11-02T18:02:03.097 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000015b795f0b8cc0297-006543e41b-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:03.098 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:03.098 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:03.099 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:03.099 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:03.100 DEBUG:boto:Method: GET 2023-11-02T18:02:03.100 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:03.101 DEBUG:boto:Data: 2023-11-02T18:02:03.101 DEBUG:boto:Headers: {} 2023-11-02T18:02:03.101 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:03.102 DEBUG:boto:Port: 8000 2023-11-02T18:02:03.102 DEBUG:boto:Params: {} 2023-11-02T18:02:03.103 DEBUG:boto:Token: None 2023-11-02T18:02:03.103 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:03 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:03.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aaNbnLjIG37XvLvekKRUPorFfmk= 2023-11-02T18:02:03.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aaNbnLjIG37XvLvekKRUPorFfmk=', 'Content-Length': '0'} 2023-11-02T18:02:03.107 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008ca7f29a99b4929-006543e41b-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:03.108 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:03.108 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:03.109 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:03.109 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:03.110 DEBUG:boto:Method: GET 2023-11-02T18:02:03.110 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:03.111 DEBUG:boto:Data: 2023-11-02T18:02:03.111 DEBUG:boto:Headers: {} 2023-11-02T18:02:03.112 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:03.112 DEBUG:boto:Port: 8001 2023-11-02T18:02:03.112 DEBUG:boto:Params: {} 2023-11-02T18:02:03.113 DEBUG:boto:Token: None 2023-11-02T18:02:03.113 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:03 GMT /ibtljn-14/ 2023-11-02T18:02:03.114 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C1UZO6BXolicQ39k4sLhWhRrRho= 2023-11-02T18:02:03.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C1UZO6BXolicQ39k4sLhWhRrRho=', 'Content-Length': '0'} 2023-11-02T18:02:03.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073235450dc5d90c2-006543e41b-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:03.120 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:03.120 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:03.121 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:03.121 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:03.122 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:03.122 DEBUG:boto:Method: GET 2023-11-02T18:02:03.123 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:03.123 DEBUG:boto:Data: 2023-11-02T18:02:03.124 DEBUG:boto:Headers: {} 2023-11-02T18:02:03.124 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:03.124 DEBUG:boto:Port: 8000 2023-11-02T18:02:03.125 DEBUG:boto:Params: {} 2023-11-02T18:02:03.125 DEBUG:boto:Token: None 2023-11-02T18:02:03.126 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:03 GMT /ibtljn-14/ 2023-11-02T18:02:03.126 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C1UZO6BXolicQ39k4sLhWhRrRho= 2023-11-02T18:02:03.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C1UZO6BXolicQ39k4sLhWhRrRho=', 'Content-Length': '0'} 2023-11-02T18:02:03.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036c39aebecd247b0-006543e41b-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:03.174 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:03.175 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-14 zones={a2, a1} 2023-11-02T18:02:03.176 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:03.176 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:03.176 DEBUG:boto:path=/ibtljn-14/?versioning 2023-11-02T18:02:03.177 DEBUG:boto:auth_path=/ibtljn-14/?versioning 2023-11-02T18:02:03.177 DEBUG:boto:Method: PUT 2023-11-02T18:02:03.177 DEBUG:boto:Path: /ibtljn-14/?versioning 2023-11-02T18:02:03.178 DEBUG:boto:Data: Suspended Disabled 2023-11-02T18:02:03.178 DEBUG:boto:Headers: {} 2023-11-02T18:02:03.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:03.179 DEBUG:boto:Port: 8000 2023-11-02T18:02:03.179 DEBUG:boto:Params: {} 2023-11-02T18:02:03.180 DEBUG:boto:Token: None 2023-11-02T18:02:03.180 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:02:03 GMT /ibtljn-14/?versioning 2023-11-02T18:02:03.181 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1lArhJKv13jyX0ro8FF96wK0E/Y= 2023-11-02T18:02:03.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1lArhJKv13jyX0ro8FF96wK0E/Y=', 'Content-Length': '230'} 2023-11-02T18:02:03.194 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009022ae0ae9184f91-006543e41b-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:03.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:02:03.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:02:03.196 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:02:03.240 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:02:03.241 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:02:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:02:03.328 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:02:03.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:03.329 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:02:03.329 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:02:03.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:03.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:03.329 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:02:03.330 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:02:03.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:03.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:03.330 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948123.185438_88.1", 2023-11-02T18:02:03.330 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:02:03.185438Z" 2023-11-02T18:02:03.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:03.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:03.331 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:02:03.331 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:02:03.332 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:03.332 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:02:03.333 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948123.185438_88.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:02:03.333 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:02:03.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:02:03.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:02:03.335 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:02:03.419 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:02:03.420 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:02:03.475 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:03.466+0000 7f60859bbf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:02:03.476 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:03.466+0000 7f60859bbf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:02:03.506 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:02:03.506 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:02:03.506 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:02:03.507 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:02:03.507 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:02:03.507 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:02:03.507 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:03.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:03.508 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:02:03.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:03.508 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:02:03.508 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:02:03.509 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:02:03.509 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:02:03.509 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:02:03.509 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:02:03.509 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:02:03.510 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:02:03.510 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:03.510 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:03.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:03.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:03.511 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:02:03.511 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:02:03.511 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:02:03.511 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:02:03.512 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:02:03.512 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:02:03.512 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:02:03.512 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:02:03.513 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:03.513 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:03.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:03.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:03.513 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:02:03.514 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:02:03.514 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:02:03.514 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948123.185438_88.1", 2023-11-02T18:02:03.514 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:02:03.515 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:02:03.515 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:02:03.515 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:02:03.185438Z", 2023-11-02T18:02:03.515 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:03.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:03.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:03.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:03.516 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:02:03.516 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:02:03.517 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:02:03.517 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:02:03.517 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:02:03.517 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:02:03.517 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:02:03.518 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:02:03.518 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:03.518 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:03.518 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:03.519 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:02:03.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:03.519 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:02:03.519 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:02:03.519 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:02:03.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:03.520 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:02:03Z" 2023-11-02T18:02:03.520 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:02:03.521 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948123.185438_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:02:03.185438Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:02:03Z" } 2023-11-02T18:02:03.522 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948123.185438_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:02:03.185438Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:02:03.522 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948123.185438_88.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:02:03.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948123.185438_88.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:02:03.523 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:02:03.524 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:03.524 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:03.524 DEBUG:boto:Method: HEAD 2023-11-02T18:02:03.525 DEBUG:boto:Path: /ibtljn-14/ 2023-11-02T18:02:03.525 DEBUG:boto:Data: 2023-11-02T18:02:03.526 DEBUG:boto:Headers: {} 2023-11-02T18:02:03.526 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:03.527 DEBUG:boto:Port: 8000 2023-11-02T18:02:03.527 DEBUG:boto:Params: {} 2023-11-02T18:02:03.528 DEBUG:boto:Token: None 2023-11-02T18:02:03.528 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:03 GMT /ibtljn-14/ 2023-11-02T18:02:03.529 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:plik81lkj1UM3GW6xGihgx+3tpg= 2023-11-02T18:02:03.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:plik81lkj1UM3GW6xGihgx+3tpg=', 'Content-Length': '0'} 2023-11-02T18:02:03.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062e60727cc28f59a-006543e41b-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:03.533 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:03.533 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:03.534 DEBUG:boto:Method: PUT 2023-11-02T18:02:03.534 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:03.535 DEBUG:boto:Data: 2023-11-02T18:02:03.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-11-02T18:02:03.536 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:03.536 DEBUG:boto:Port: 8000 2023-11-02T18:02:03.536 DEBUG:boto:Params: {} 2023-11-02T18:02:03.537 DEBUG:boto:Token: None 2023-11-02T18:02:03.537 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 02 Nov 2023 18:02:03 GMT /ibtljn-14/obj 2023-11-02T18:02:03.538 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a6Ra2CNgROLqmj0vJEVyLB1fJrY= 2023-11-02T18:02:03.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:02:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a6Ra2CNgROLqmj0vJEVyLB1fJrY='} 2023-11-02T18:02:03.561 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000918fb8421ed769ed-006543e41b-1111-a1'), ('Date', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:03.562 DEBUG:rgw_multi.tests:created null version id=None 2023-11-02T18:02:03.562 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-14 2023-11-02T18:02:03.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:02:03.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:02:03.564 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:02:03.613 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:02:03.614 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:02:03.684 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:03.674+0000 7f8c2c484f40 1 waiting for incremental sync to catch up: 2023-11-02T18:02:03.684 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.10.13, , ] 2023-11-02T18:02:03.684 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000005.14.18, , ] 2023-11-02T18:02:08.686 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:08.678+0000 7f8c2c484f40 1 waiting for incremental sync to catch up: 2023-11-02T18:02:08.687 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.10.13, , ] 2023-11-02T18:02:08.687 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000005.14.18, , ] 2023-11-02T18:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:13.678+0000 7f8c2c484f40 1 waiting for incremental sync to catch up: 2023-11-02T18:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.10.13, , ] 2023-11-02T18:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000005.14.18, , ] 2023-11-02T18:02:18.691 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:18.682+0000 7f8c2c484f40 1 waiting for incremental sync to catch up: 2023-11-02T18:02:18.691 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.10.13, , ] 2023-11-02T18:02:18.692 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000005.14.18, , ] 2023-11-02T18:02:23.694 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:23.682+0000 7f8c2c484f40 1 bucket sync caught up with source: 2023-11-02T18:02:23.694 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000005.14.18, , ] 2023-11-02T18:02:23.694 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000005.14.18, , ] 2023-11-02T18:02:23.695 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:23.682+0000 7f8c2c484f40 0 bucket checkpoint complete 2023-11-02T18:02:23.722 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-14 2023-11-02T18:02:23.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:02:23.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:02:23.724 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:02:23.775 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:02:23.775 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:02:23.844 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:23.834+0000 7fd170904f40 1 waiting for incremental sync to catch up: 2023-11-02T18:02:23.845 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.19.13, , ] 2023-11-02T18:02:23.845 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000005.24.18, , ] 2023-11-02T18:02:28.847 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:28.837+0000 7fd170904f40 1 waiting for incremental sync to catch up: 2023-11-02T18:02:28.848 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.19.13, , ] 2023-11-02T18:02:28.848 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000005.24.18, , ] 2023-11-02T18:02:33.849 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:33.837+0000 7fd170904f40 1 waiting for incremental sync to catch up: 2023-11-02T18:02:33.850 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.19.13, , ] 2023-11-02T18:02:33.850 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000005.24.18, , ] 2023-11-02T18:02:38.852 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:38.841+0000 7fd170904f40 1 waiting for incremental sync to catch up: 2023-11-02T18:02:38.852 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.19.13, , ] 2023-11-02T18:02:38.853 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000005.24.18, , ] 2023-11-02T18:02:43.855 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:43.845+0000 7fd170904f40 1 bucket sync caught up with source: 2023-11-02T18:02:43.855 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000005.24.18, , ] 2023-11-02T18:02:43.855 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000005.24.18, , ] 2023-11-02T18:02:43.856 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:43.845+0000 7fd170904f40 0 bucket checkpoint complete 2023-11-02T18:02:43.883 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-14 zones={a2, a1} 2023-11-02T18:02:43.884 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:43.884 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:43.885 DEBUG:boto:Method: HEAD 2023-11-02T18:02:43.886 DEBUG:boto:Path: /ibtljn-14/ 2023-11-02T18:02:43.886 DEBUG:boto:Data: 2023-11-02T18:02:43.886 DEBUG:boto:Headers: {} 2023-11-02T18:02:43.887 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:43.887 DEBUG:boto:Port: 8001 2023-11-02T18:02:43.888 DEBUG:boto:Params: {} 2023-11-02T18:02:43.888 DEBUG:boto:Token: None 2023-11-02T18:02:43.889 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:43 GMT /ibtljn-14/ 2023-11-02T18:02:43.889 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxUQgknvg6ne3ewUhy2WIYEEjrc= 2023-11-02T18:02:43.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxUQgknvg6ne3ewUhy2WIYEEjrc=', 'Content-Length': '0'} 2023-11-02T18:02:43.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b998c476d2b47af6-006543e443-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:43.895 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:43.895 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:43.896 DEBUG:boto:Method: HEAD 2023-11-02T18:02:43.896 DEBUG:boto:Path: /ibtljn-14/ 2023-11-02T18:02:43.897 DEBUG:boto:Data: 2023-11-02T18:02:43.897 DEBUG:boto:Headers: {} 2023-11-02T18:02:43.898 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:43.898 DEBUG:boto:Port: 8000 2023-11-02T18:02:43.899 DEBUG:boto:Params: {} 2023-11-02T18:02:43.900 DEBUG:boto:Token: None 2023-11-02T18:02:43.900 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:43 GMT /ibtljn-14/ 2023-11-02T18:02:43.901 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxUQgknvg6ne3ewUhy2WIYEEjrc= 2023-11-02T18:02:43.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxUQgknvg6ne3ewUhy2WIYEEjrc=', 'Content-Length': '0'} 2023-11-02T18:02:43.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f8f82b2af5dd32d-006543e443-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:43.906 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:02:43.906 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:43.907 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:43.907 DEBUG:boto:path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:43.908 DEBUG:boto:auth_path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:43.908 DEBUG:boto:Method: GET 2023-11-02T18:02:43.908 DEBUG:boto:Path: /ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:43.909 DEBUG:boto:Data: 2023-11-02T18:02:43.909 DEBUG:boto:Headers: {} 2023-11-02T18:02:43.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:43.910 DEBUG:boto:Port: 8001 2023-11-02T18:02:43.911 DEBUG:boto:Params: {} 2023-11-02T18:02:43.912 DEBUG:boto:Token: None 2023-11-02T18:02:43.913 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:43 GMT /ibtljn-14/?versions 2023-11-02T18:02:43.913 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdMEbPt34jDdUQOX11ganHxpLTs= 2023-11-02T18:02:43.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdMEbPt34jDdUQOX11ganHxpLTs=', 'Content-Length': '0'} 2023-11-02T18:02:43.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043c60e16f076e21d-006543e443-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:43.962 DEBUG:boto:b'ibtljn-14999falseobjnulltrue2023-11-02T18:02:03.534Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnXZq0ifaOrZEYes8.J-UF5LgfHuiXUlfalse2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:43.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:02:43.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:02:43.964 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:02:43.965 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:43.965 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:43.966 DEBUG:boto:path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:43.966 DEBUG:boto:auth_path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:43.966 DEBUG:boto:Method: GET 2023-11-02T18:02:43.967 DEBUG:boto:Path: /ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:43.967 DEBUG:boto:Data: 2023-11-02T18:02:43.968 DEBUG:boto:Headers: {} 2023-11-02T18:02:43.968 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:43.969 DEBUG:boto:Port: 8000 2023-11-02T18:02:43.969 DEBUG:boto:Params: {} 2023-11-02T18:02:43.970 DEBUG:boto:Token: None 2023-11-02T18:02:43.970 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:43 GMT /ibtljn-14/?versions 2023-11-02T18:02:43.971 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdMEbPt34jDdUQOX11ganHxpLTs= 2023-11-02T18:02:43.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdMEbPt34jDdUQOX11ganHxpLTs=', 'Content-Length': '0'} 2023-11-02T18:02:43.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb9a3d0f27d7629a-006543e443-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:43.976 DEBUG:boto:b'ibtljn-14999falseobjnulltrue2023-11-02T18:02:03.534Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnXZq0ifaOrZEYes8.J-UF5LgfHuiXUlfalse2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:43.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:02:43.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:02:43.978 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:43.978 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:43.979 DEBUG:boto:path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:43.979 DEBUG:boto:auth_path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:43.980 DEBUG:boto:Method: GET 2023-11-02T18:02:43.980 DEBUG:boto:Path: /ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:43.980 DEBUG:boto:Data: 2023-11-02T18:02:43.981 DEBUG:boto:Headers: {} 2023-11-02T18:02:43.981 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:43.982 DEBUG:boto:Port: 8001 2023-11-02T18:02:43.982 DEBUG:boto:Params: {} 2023-11-02T18:02:43.983 DEBUG:boto:Token: None 2023-11-02T18:02:43.983 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:43 GMT /ibtljn-14/?versions 2023-11-02T18:02:43.984 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdMEbPt34jDdUQOX11ganHxpLTs= 2023-11-02T18:02:43.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdMEbPt34jDdUQOX11ganHxpLTs=', 'Content-Length': '0'} 2023-11-02T18:02:43.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041f88e12428d9177-006543e443-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:43.989 DEBUG:boto:b'ibtljn-14999falseobjnulltrue2023-11-02T18:02:03.534Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnXZq0ifaOrZEYes8.J-UF5LgfHuiXUlfalse2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:43.990 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:43.991 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:43.991 DEBUG:boto:path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:43.991 DEBUG:boto:auth_path=/ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:43.992 DEBUG:boto:Method: GET 2023-11-02T18:02:43.992 DEBUG:boto:Path: /ibtljn-14/?versions&max-keys=999 2023-11-02T18:02:43.993 DEBUG:boto:Data: 2023-11-02T18:02:43.993 DEBUG:boto:Headers: {} 2023-11-02T18:02:43.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:43.994 DEBUG:boto:Port: 8000 2023-11-02T18:02:43.994 DEBUG:boto:Params: {} 2023-11-02T18:02:43.995 DEBUG:boto:Token: None 2023-11-02T18:02:43.995 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:43 GMT /ibtljn-14/?versions 2023-11-02T18:02:43.996 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdMEbPt34jDdUQOX11ganHxpLTs= 2023-11-02T18:02:43.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdMEbPt34jDdUQOX11ganHxpLTs=', 'Content-Length': '0'} 2023-11-02T18:02:44.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fc526bc11d2176a-006543e443-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.042 DEBUG:boto:b'ibtljn-14999falseobjnulltrue2023-11-02T18:02:03.534Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnXZq0ifaOrZEYes8.J-UF5LgfHuiXUlfalse2023-11-02T18:01:22.247Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:44.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:02:44.044 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.044 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.045 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.045 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.046 DEBUG:boto:Method: GET 2023-11-02T18:02:44.046 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.046 DEBUG:boto:Data: 2023-11-02T18:02:44.047 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.047 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.048 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.048 DEBUG:boto:Params: {} 2023-11-02T18:02:44.049 DEBUG:boto:Token: None 2023-11-02T18:02:44.049 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.050 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bqq2otrGBXc7uUj+p9UdarcBmlE= 2023-11-02T18:02:44.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bqq2otrGBXc7uUj+p9UdarcBmlE=', 'Content-Length': '0'} 2023-11-02T18:02:44.053 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096d026fd1c21225e-006543e444-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.054 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.054 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.055 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.055 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.056 DEBUG:boto:Method: GET 2023-11-02T18:02:44.056 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.057 DEBUG:boto:Data: 2023-11-02T18:02:44.057 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.057 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.058 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.058 DEBUG:boto:Params: {} 2023-11-02T18:02:44.059 DEBUG:boto:Token: None 2023-11-02T18:02:44.059 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.060 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bqq2otrGBXc7uUj+p9UdarcBmlE= 2023-11-02T18:02:44.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bqq2otrGBXc7uUj+p9UdarcBmlE=', 'Content-Length': '0'} 2023-11-02T18:02:44.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1bdbc2ff827b961-006543e444-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.064 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.064 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.064 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.065 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.065 DEBUG:boto:Method: HEAD 2023-11-02T18:02:44.066 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.066 DEBUG:boto:Data: 2023-11-02T18:02:44.066 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.067 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.067 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.068 DEBUG:boto:Params: {} 2023-11-02T18:02:44.068 DEBUG:boto:Token: None 2023-11-02T18:02:44.069 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.069 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JyTE2dYr2Z7p5exUXWEH1sspUZc= 2023-11-02T18:02:44.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JyTE2dYr2Z7p5exUXWEH1sspUZc=', 'Content-Length': '0'} 2023-11-02T18:02:44.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee95794bf44abad4-006543e444-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.073 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.074 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.074 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.074 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.075 DEBUG:boto:Method: HEAD 2023-11-02T18:02:44.075 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.076 DEBUG:boto:Data: 2023-11-02T18:02:44.076 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.076 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.077 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.077 DEBUG:boto:Params: {} 2023-11-02T18:02:44.078 DEBUG:boto:Token: None 2023-11-02T18:02:44.078 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.078 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JyTE2dYr2Z7p5exUXWEH1sspUZc= 2023-11-02T18:02:44.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JyTE2dYr2Z7p5exUXWEH1sspUZc=', 'Content-Length': '0'} 2023-11-02T18:02:44.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034ecd4f2fd6ea0d3-006543e444-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:02:44.083 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.083 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.084 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.084 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.085 DEBUG:boto:Method: GET 2023-11-02T18:02:44.085 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.085 DEBUG:boto:Data: 2023-11-02T18:02:44.086 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.086 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.087 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.087 DEBUG:boto:Params: {} 2023-11-02T18:02:44.088 DEBUG:boto:Token: None 2023-11-02T18:02:44.088 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.089 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bqq2otrGBXc7uUj+p9UdarcBmlE= 2023-11-02T18:02:44.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bqq2otrGBXc7uUj+p9UdarcBmlE=', 'Content-Length': '0'} 2023-11-02T18:02:44.092 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000398088b1651b1616-006543e444-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.093 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.093 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.093 DEBUG:boto:path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.094 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.094 DEBUG:boto:Method: GET 2023-11-02T18:02:44.095 DEBUG:boto:Path: /ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.095 DEBUG:boto:Data: 2023-11-02T18:02:44.096 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.097 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.097 DEBUG:boto:Params: {} 2023-11-02T18:02:44.098 DEBUG:boto:Token: None 2023-11-02T18:02:44.098 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj?versionId=null 2023-11-02T18:02:44.099 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bqq2otrGBXc7uUj+p9UdarcBmlE= 2023-11-02T18:02:44.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bqq2otrGBXc7uUj+p9UdarcBmlE=', 'Content-Length': '0'} 2023-11-02T18:02:44.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e9436be06db4d1cc-006543e444-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.103 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:44.103 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:44.104 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.104 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.104 DEBUG:boto:Method: GET 2023-11-02T18:02:44.105 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:44.105 DEBUG:boto:Data: 2023-11-02T18:02:44.106 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.106 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.107 DEBUG:boto:Params: {} 2023-11-02T18:02:44.107 DEBUG:boto:Token: None 2023-11-02T18:02:44.108 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/ 2023-11-02T18:02:44.108 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc= 2023-11-02T18:02:44.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc=', 'Content-Length': '0'} 2023-11-02T18:02:44.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a25144cd01b57df1-006543e444-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.114 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:02:03.534Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:44.115 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:44.116 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:44.116 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.116 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.117 DEBUG:boto:Method: GET 2023-11-02T18:02:44.117 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:44.118 DEBUG:boto:Data: 2023-11-02T18:02:44.118 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.119 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.119 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.120 DEBUG:boto:Params: {} 2023-11-02T18:02:44.120 DEBUG:boto:Token: None 2023-11-02T18:02:44.121 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/ 2023-11-02T18:02:44.121 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc= 2023-11-02T18:02:44.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc=', 'Content-Length': '0'} 2023-11-02T18:02:44.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab1d85e2120e1609-006543e444-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.126 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:02:03.534Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:44.127 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.128 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.128 DEBUG:boto:Method: HEAD 2023-11-02T18:02:44.128 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:44.129 DEBUG:boto:Data: 2023-11-02T18:02:44.129 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.130 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.130 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.130 DEBUG:boto:Params: {} 2023-11-02T18:02:44.131 DEBUG:boto:Token: None 2023-11-02T18:02:44.131 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj 2023-11-02T18:02:44.132 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:03P/MKq/Fe0L8MzHQy5wdT4XvHQ= 2023-11-02T18:02:44.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:03P/MKq/Fe0L8MzHQy5wdT4XvHQ=', 'Content-Length': '0'} 2023-11-02T18:02:44.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004579337d1a6326fd-006543e444-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.136 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.136 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.137 DEBUG:boto:Method: HEAD 2023-11-02T18:02:44.137 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:44.137 DEBUG:boto:Data: 2023-11-02T18:02:44.138 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.139 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.139 DEBUG:boto:Params: {} 2023-11-02T18:02:44.140 DEBUG:boto:Token: None 2023-11-02T18:02:44.140 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj 2023-11-02T18:02:44.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:03P/MKq/Fe0L8MzHQy5wdT4XvHQ= 2023-11-02T18:02:44.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:03P/MKq/Fe0L8MzHQy5wdT4XvHQ=', 'Content-Length': '0'} 2023-11-02T18:02:44.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e953bd114bf81b61-006543e444-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:02:44.144 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.145 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.145 DEBUG:boto:Method: GET 2023-11-02T18:02:44.146 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:44.146 DEBUG:boto:Data: 2023-11-02T18:02:44.147 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.147 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.148 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.148 DEBUG:boto:Params: {} 2023-11-02T18:02:44.149 DEBUG:boto:Token: None 2023-11-02T18:02:44.149 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj 2023-11-02T18:02:44.150 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tew/WdoPHYk4ArgwaH3c3p/aflc= 2023-11-02T18:02:44.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tew/WdoPHYk4ArgwaH3c3p/aflc=', 'Content-Length': '0'} 2023-11-02T18:02:44.153 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c140a87cf2ebc033-006543e444-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.154 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.154 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.154 DEBUG:boto:Method: GET 2023-11-02T18:02:44.155 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:44.155 DEBUG:boto:Data: 2023-11-02T18:02:44.156 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.156 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.157 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.157 DEBUG:boto:Params: {} 2023-11-02T18:02:44.158 DEBUG:boto:Token: None 2023-11-02T18:02:44.158 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj 2023-11-02T18:02:44.159 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tew/WdoPHYk4ArgwaH3c3p/aflc= 2023-11-02T18:02:44.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tew/WdoPHYk4ArgwaH3c3p/aflc=', 'Content-Length': '0'} 2023-11-02T18:02:44.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008cb3bea82687706d-006543e444-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.163 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:44.163 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:44.164 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.164 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.164 DEBUG:boto:Method: GET 2023-11-02T18:02:44.165 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:44.165 DEBUG:boto:Data: 2023-11-02T18:02:44.166 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.166 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.167 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.167 DEBUG:boto:Params: {} 2023-11-02T18:02:44.167 DEBUG:boto:Token: None 2023-11-02T18:02:44.168 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/ 2023-11-02T18:02:44.168 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc= 2023-11-02T18:02:44.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc=', 'Content-Length': '0'} 2023-11-02T18:02:44.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dbd3c3c4e88bbeb-006543e444-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.214 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:02:03.534Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:44.215 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:44.216 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:44.216 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.216 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.217 DEBUG:boto:Method: GET 2023-11-02T18:02:44.217 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:44.218 DEBUG:boto:Data: 2023-11-02T18:02:44.218 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.219 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.219 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.220 DEBUG:boto:Params: {} 2023-11-02T18:02:44.220 DEBUG:boto:Token: None 2023-11-02T18:02:44.220 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/ 2023-11-02T18:02:44.221 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc= 2023-11-02T18:02:44.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc=', 'Content-Length': '0'} 2023-11-02T18:02:44.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b19c4d7ad2d719ed-006543e444-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.227 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:02:03.534Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:44.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:02:44.228 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.228 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.229 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.229 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.230 DEBUG:boto:Method: GET 2023-11-02T18:02:44.230 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.231 DEBUG:boto:Data: 2023-11-02T18:02:44.231 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.232 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.232 DEBUG:boto:Params: {} 2023-11-02T18:02:44.233 DEBUG:boto:Token: None 2023-11-02T18:02:44.233 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.234 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aPILgdLOseu9WeAZL4XwuIQ6ulM= 2023-11-02T18:02:44.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aPILgdLOseu9WeAZL4XwuIQ6ulM=', 'Content-Length': '0'} 2023-11-02T18:02:44.237 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006e40d3cb8b2014a8-006543e444-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.238 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.238 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.239 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.239 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.239 DEBUG:boto:Method: GET 2023-11-02T18:02:44.240 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.240 DEBUG:boto:Data: 2023-11-02T18:02:44.241 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.241 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.242 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.242 DEBUG:boto:Params: {} 2023-11-02T18:02:44.243 DEBUG:boto:Token: None 2023-11-02T18:02:44.243 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.244 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aPILgdLOseu9WeAZL4XwuIQ6ulM= 2023-11-02T18:02:44.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aPILgdLOseu9WeAZL4XwuIQ6ulM=', 'Content-Length': '0'} 2023-11-02T18:02:44.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ececf4cebdc83459-006543e444-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.248 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.248 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.249 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.249 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.249 DEBUG:boto:Method: HEAD 2023-11-02T18:02:44.250 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.250 DEBUG:boto:Data: 2023-11-02T18:02:44.251 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.251 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.252 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.252 DEBUG:boto:Params: {} 2023-11-02T18:02:44.252 DEBUG:boto:Token: None 2023-11-02T18:02:44.253 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.253 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JiTpSsGaCftti1HS6sf77M9P1cw= 2023-11-02T18:02:44.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JiTpSsGaCftti1HS6sf77M9P1cw=', 'Content-Length': '0'} 2023-11-02T18:02:44.257 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001cfb72350c20c358-006543e444-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.257 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.258 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.258 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.259 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.259 DEBUG:boto:Method: HEAD 2023-11-02T18:02:44.260 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.260 DEBUG:boto:Data: 2023-11-02T18:02:44.260 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.261 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.261 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.262 DEBUG:boto:Params: {} 2023-11-02T18:02:44.262 DEBUG:boto:Token: None 2023-11-02T18:02:44.262 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.263 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JiTpSsGaCftti1HS6sf77M9P1cw= 2023-11-02T18:02:44.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JiTpSsGaCftti1HS6sf77M9P1cw=', 'Content-Length': '0'} 2023-11-02T18:02:44.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000908e27648742305e-006543e444-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:02:44.267 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.268 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.268 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.268 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.269 DEBUG:boto:Method: GET 2023-11-02T18:02:44.269 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.270 DEBUG:boto:Data: 2023-11-02T18:02:44.270 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.271 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.271 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.271 DEBUG:boto:Params: {} 2023-11-02T18:02:44.272 DEBUG:boto:Token: None 2023-11-02T18:02:44.272 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.273 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aPILgdLOseu9WeAZL4XwuIQ6ulM= 2023-11-02T18:02:44.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aPILgdLOseu9WeAZL4XwuIQ6ulM=', 'Content-Length': '0'} 2023-11-02T18:02:44.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000032f9ab557ed8fecc-006543e444-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.277 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.277 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.278 DEBUG:boto:path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.278 DEBUG:boto:auth_path=/ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.278 DEBUG:boto:Method: GET 2023-11-02T18:02:44.279 DEBUG:boto:Path: /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.279 DEBUG:boto:Data: 2023-11-02T18:02:44.280 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.280 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.281 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.281 DEBUG:boto:Params: {} 2023-11-02T18:02:44.281 DEBUG:boto:Token: None 2023-11-02T18:02:44.282 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj?versionId=nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl 2023-11-02T18:02:44.282 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aPILgdLOseu9WeAZL4XwuIQ6ulM= 2023-11-02T18:02:44.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aPILgdLOseu9WeAZL4XwuIQ6ulM=', 'Content-Length': '0'} 2023-11-02T18:02:44.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:01:22 GMT'), ('x-amz-version-id', 'nXZq0ifaOrZEYes8.J-UF5LgfHuiXUl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000134912e8f7cd07c8-006543e444-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.286 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:44.287 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:44.287 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.288 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.288 DEBUG:boto:Method: GET 2023-11-02T18:02:44.289 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:44.289 DEBUG:boto:Data: 2023-11-02T18:02:44.290 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.290 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.290 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.291 DEBUG:boto:Params: {} 2023-11-02T18:02:44.291 DEBUG:boto:Token: None 2023-11-02T18:02:44.292 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/ 2023-11-02T18:02:44.292 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc= 2023-11-02T18:02:44.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc=', 'Content-Length': '0'} 2023-11-02T18:02:44.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a69d996ccf1396f4-006543e444-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.338 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:02:03.534Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:44.339 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:44.340 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:44.340 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.340 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.341 DEBUG:boto:Method: GET 2023-11-02T18:02:44.341 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:44.342 DEBUG:boto:Data: 2023-11-02T18:02:44.342 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.343 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.343 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.344 DEBUG:boto:Params: {} 2023-11-02T18:02:44.344 DEBUG:boto:Token: None 2023-11-02T18:02:44.345 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/ 2023-11-02T18:02:44.345 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc= 2023-11-02T18:02:44.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc=', 'Content-Length': '0'} 2023-11-02T18:02:44.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f6a8785b8253105-006543e444-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.351 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:02:03.534Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:44.351 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.352 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.352 DEBUG:boto:Method: HEAD 2023-11-02T18:02:44.353 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:44.353 DEBUG:boto:Data: 2023-11-02T18:02:44.353 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.354 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.354 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.355 DEBUG:boto:Params: {} 2023-11-02T18:02:44.355 DEBUG:boto:Token: None 2023-11-02T18:02:44.356 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj 2023-11-02T18:02:44.356 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:03P/MKq/Fe0L8MzHQy5wdT4XvHQ= 2023-11-02T18:02:44.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:03P/MKq/Fe0L8MzHQy5wdT4XvHQ=', 'Content-Length': '0'} 2023-11-02T18:02:44.360 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007096612dff345d06-006543e444-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.361 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.361 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.362 DEBUG:boto:Method: HEAD 2023-11-02T18:02:44.362 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:44.362 DEBUG:boto:Data: 2023-11-02T18:02:44.363 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.363 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.364 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.364 DEBUG:boto:Params: {} 2023-11-02T18:02:44.365 DEBUG:boto:Token: None 2023-11-02T18:02:44.365 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj 2023-11-02T18:02:44.366 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:03P/MKq/Fe0L8MzHQy5wdT4XvHQ= 2023-11-02T18:02:44.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:03P/MKq/Fe0L8MzHQy5wdT4XvHQ=', 'Content-Length': '0'} 2023-11-02T18:02:44.369 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b2b2df7d3013d25-006543e444-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:02:44.370 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.371 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.371 DEBUG:boto:Method: GET 2023-11-02T18:02:44.372 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:44.372 DEBUG:boto:Data: 2023-11-02T18:02:44.373 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.373 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.373 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.374 DEBUG:boto:Params: {} 2023-11-02T18:02:44.374 DEBUG:boto:Token: None 2023-11-02T18:02:44.375 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj 2023-11-02T18:02:44.375 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tew/WdoPHYk4ArgwaH3c3p/aflc= 2023-11-02T18:02:44.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tew/WdoPHYk4ArgwaH3c3p/aflc=', 'Content-Length': '0'} 2023-11-02T18:02:44.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048c82ebd79c9cc87-006543e444-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.379 DEBUG:boto:path=/ibtljn-14/obj 2023-11-02T18:02:44.380 DEBUG:boto:auth_path=/ibtljn-14/obj 2023-11-02T18:02:44.380 DEBUG:boto:Method: GET 2023-11-02T18:02:44.381 DEBUG:boto:Path: /ibtljn-14/obj 2023-11-02T18:02:44.381 DEBUG:boto:Data: 2023-11-02T18:02:44.382 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.382 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.383 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.383 DEBUG:boto:Params: {} 2023-11-02T18:02:44.384 DEBUG:boto:Token: None 2023-11-02T18:02:44.384 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/obj 2023-11-02T18:02:44.384 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tew/WdoPHYk4ArgwaH3c3p/aflc= 2023-11-02T18:02:44.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tew/WdoPHYk4ArgwaH3c3p/aflc=', 'Content-Length': '0'} 2023-11-02T18:02:44.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc7a61d99b76ac21-006543e444-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.388 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:44.389 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:44.389 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.390 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.390 DEBUG:boto:Method: GET 2023-11-02T18:02:44.391 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:44.391 DEBUG:boto:Data: 2023-11-02T18:02:44.391 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.392 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.392 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.393 DEBUG:boto:Params: {} 2023-11-02T18:02:44.393 DEBUG:boto:Token: None 2023-11-02T18:02:44.394 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/ 2023-11-02T18:02:44.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc= 2023-11-02T18:02:44.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc=', 'Content-Length': '0'} 2023-11-02T18:02:44.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077f48c7b5095917c-006543e444-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.443 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:02:03.534Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:44.444 DEBUG:boto:path=/ibtljn-14/ 2023-11-02T18:02:44.444 DEBUG:boto:auth_path=/ibtljn-14/ 2023-11-02T18:02:44.445 DEBUG:boto:path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.445 DEBUG:boto:auth_path=/ibtljn-14/?prefix=obj 2023-11-02T18:02:44.445 DEBUG:boto:Method: GET 2023-11-02T18:02:44.446 DEBUG:boto:Path: /ibtljn-14/?prefix=obj 2023-11-02T18:02:44.446 DEBUG:boto:Data: 2023-11-02T18:02:44.447 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.448 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.448 DEBUG:boto:Params: {} 2023-11-02T18:02:44.449 DEBUG:boto:Token: None 2023-11-02T18:02:44.449 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-14/ 2023-11-02T18:02:44.450 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc= 2023-11-02T18:02:44.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:edFTJgKW31BnZ+750kazvl/t9uc=', 'Content-Length': '0'} 2023-11-02T18:02:44.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000831e38c15dc97cd2-006543e444-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.456 DEBUG:boto:b'ibtljn-14obj1000falseobj2023-11-02T18:02:03.534Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:02:44.456 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-14 zones={a2, a1} 2023-11-02T18:02:44.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2023-11-02T18:02:44.458 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-15 2023-11-02T18:02:44.459 DEBUG:boto:path=/ibtljn-15/ 2023-11-02T18:02:44.459 DEBUG:boto:auth_path=/ibtljn-15/ 2023-11-02T18:02:44.460 DEBUG:boto:Method: PUT 2023-11-02T18:02:44.460 DEBUG:boto:Path: /ibtljn-15/ 2023-11-02T18:02:44.460 DEBUG:boto:Data: 2023-11-02T18:02:44.461 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.461 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.462 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.462 DEBUG:boto:Params: {} 2023-11-02T18:02:44.462 DEBUG:boto:Token: None 2023-11-02T18:02:44.463 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-15/ 2023-11-02T18:02:44.463 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZNU+zn3EUNtBmtOtaqxsIDpPWVc= 2023-11-02T18:02:44.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZNU+zn3EUNtBmtOtaqxsIDpPWVc=', 'Content-Length': '0'} 2023-11-02T18:02:44.491 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000a68cc84a2f33820-006543e444-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.492 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-16 2023-11-02T18:02:44.492 DEBUG:boto:path=/ibtljn-16/ 2023-11-02T18:02:44.493 DEBUG:boto:auth_path=/ibtljn-16/ 2023-11-02T18:02:44.493 DEBUG:boto:Method: PUT 2023-11-02T18:02:44.494 DEBUG:boto:Path: /ibtljn-16/ 2023-11-02T18:02:44.494 DEBUG:boto:Data: 2023-11-02T18:02:44.494 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.495 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.495 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.496 DEBUG:boto:Params: {} 2023-11-02T18:02:44.496 DEBUG:boto:Token: None 2023-11-02T18:02:44.497 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-16/ 2023-11-02T18:02:44.497 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QbQ9infiDRhluj1V9vm3qbywz7A= 2023-11-02T18:02:44.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QbQ9infiDRhluj1V9vm3qbywz7A=', 'Content-Length': '0'} 2023-11-02T18:02:44.538 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000021ec7f9353a5bf95-006543e444-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.539 DEBUG:boto:path=/ibtljn-15/ 2023-11-02T18:02:44.540 DEBUG:boto:auth_path=/ibtljn-15/ 2023-11-02T18:02:44.540 DEBUG:boto:path=/ibtljn-15/?versioning 2023-11-02T18:02:44.540 DEBUG:boto:auth_path=/ibtljn-15/?versioning 2023-11-02T18:02:44.541 DEBUG:boto:Method: PUT 2023-11-02T18:02:44.541 DEBUG:boto:Path: /ibtljn-15/?versioning 2023-11-02T18:02:44.542 DEBUG:boto:Data: Enabled Disabled 2023-11-02T18:02:44.542 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.542 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:44.543 DEBUG:boto:Port: 8000 2023-11-02T18:02:44.543 DEBUG:boto:Params: {} 2023-11-02T18:02:44.544 DEBUG:boto:Token: None 2023-11-02T18:02:44.544 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-15/?versioning 2023-11-02T18:02:44.545 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NthiDLPhD9jP3MU178djYwEdru4= 2023-11-02T18:02:44.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NthiDLPhD9jP3MU178djYwEdru4=', 'Content-Length': '228'} 2023-11-02T18:02:44.554 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002eb86d27612b7723-006543e444-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.554 DEBUG:boto:path=/ibtljn-16/ 2023-11-02T18:02:44.555 DEBUG:boto:auth_path=/ibtljn-16/ 2023-11-02T18:02:44.555 DEBUG:boto:path=/ibtljn-16/?versioning 2023-11-02T18:02:44.556 DEBUG:boto:auth_path=/ibtljn-16/?versioning 2023-11-02T18:02:44.556 DEBUG:boto:Method: PUT 2023-11-02T18:02:44.556 DEBUG:boto:Path: /ibtljn-16/?versioning 2023-11-02T18:02:44.557 DEBUG:boto:Data: Enabled Disabled 2023-11-02T18:02:44.557 DEBUG:boto:Headers: {} 2023-11-02T18:02:44.558 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:44.558 DEBUG:boto:Port: 8001 2023-11-02T18:02:44.559 DEBUG:boto:Params: {} 2023-11-02T18:02:44.559 DEBUG:boto:Token: None 2023-11-02T18:02:44.559 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:02:44 GMT /ibtljn-16/?versioning 2023-11-02T18:02:44.560 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JvBMkqaUJ3TlDBHkHtyrq3NDVfs= 2023-11-02T18:02:44.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JvBMkqaUJ3TlDBHkHtyrq3NDVfs=', 'Content-Length': '228'} 2023-11-02T18:02:44.576 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000493b586f8969f071-006543e444-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:44.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:02:44.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:02:44.577 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:02:44.632 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:02:44.633 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:02:44.682 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:02:44.708 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:02:44.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:44.709 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:02:44.709 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:02:44.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:44.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:44.710 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:02:44.710 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:02:44.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:44.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:44.711 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948164.563943_100.1", 2023-11-02T18:02:44.711 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:02:44.563943Z" 2023-11-02T18:02:44.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:44.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:44.711 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:02:44.712 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:02:44.712 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:44.712 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:02:44.713 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948164.563943_100.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:02:44.714 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:02:44.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:02:44.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:02:44.715 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:02:44.761 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:02:44.761 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:02:44.820 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:44.809+0000 7f32fd795f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:02:44.821 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:44.809+0000 7f32fd795f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:02:44.848 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:02:44.848 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:02:44.848 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:02:44.849 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:02:44.849 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:02:44.849 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:02:44.849 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:44.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:44.850 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:02:44.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:44.850 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:02:44.850 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:02:44.851 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:02:44.851 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:02:44.851 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:02:44.851 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:02:44.851 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:02:44.852 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:02:44.852 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:44.852 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:44.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:44.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:44.853 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:02:44.853 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:02:44.853 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:02:44.853 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:02:44.854 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:02:44.854 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:02:44.854 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:02:44.854 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:02:44.854 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:44.855 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:44.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:44.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:44.855 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:02:44.856 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:02:44.856 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:02:44.856 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948164.472257_89.1", 2023-11-02T18:02:44.856 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:02:44.856 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:02:44.857 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:02:44.857 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:02:44.472257Z", 2023-11-02T18:02:44.857 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:44.857 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:44.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:44.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:44.858 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:02:44.858 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:02:44.858 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:02:44.859 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:02:44.859 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:02:44.859 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:02:44.859 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:02:44.859 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:02:44.860 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:44.860 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:44.860 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:44.860 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:02:44.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:44.861 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:02:44.861 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:02:44.861 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:02:44.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:44.862 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:02:44Z" 2023-11-02T18:02:44.862 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:02:44.863 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948164.472257_89.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:02:44.472257Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:02:44Z" } 2023-11-02T18:02:44.864 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948164.472257_89.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:02:44.472257Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:02:44.864 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948164.563943_100.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:02:44.864 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948164.472257_89.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:02:44.865 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948164.563943_100.1 target=1_1698948164.472257_89.1 2023-11-02T18:02:49.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:02:49.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:02:49.867 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:02:49.915 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:02:49.916 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:02:49.970 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:49.957+0000 7f32dcf75f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:02:49.971 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:49.961+0000 7f32dcf75f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:02:49.997 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:02:49.998 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:02:49.998 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:02:49.998 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:02:49.999 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:02:49.999 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:02:49.999 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:49.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:50.000 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:02:50.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:50.000 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:02:50.000 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:02:50.001 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:02:50.001 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:02:50.001 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:02:50.001 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:02:50.002 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:02:50.002 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:02:50.002 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:50.002 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:50.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:50.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:50.003 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:02:50.003 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:02:50.003 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:02:50.004 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:02:50.004 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:02:50.004 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:02:50.004 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:02:50.005 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:02:50.005 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:50.005 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:50.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:50.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:50.006 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:02:50.006 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:02:50.006 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:02:50.006 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948164.563943_100.1", 2023-11-02T18:02:50.007 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:02:50.007 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:02:50.007 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:02:50.007 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:02:44.563943Z", 2023-11-02T18:02:50.008 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:50.008 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:50.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:50.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:02:50.009 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:02:50.009 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:02:50.009 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:02:50.009 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:02:50.010 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:02:50.010 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:02:50.010 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:02:50.010 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:02:50.011 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:02:50.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:50.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:02:50.011 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:02:50.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:50.012 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:02:50.012 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:02:50.012 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:02:50.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:02:50.013 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:02:49Z" 2023-11-02T18:02:50.013 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:02:50.014 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948164.563943_100.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:02:44.563943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:02:49Z" } 2023-11-02T18:02:50.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948164.563943_100.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:02:44.563943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:02:50.015 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948164.563943_100.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:02:50.015 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948164.563943_100.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:02:50.016 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:02:50.016 DEBUG:boto:path=/ibtljn-15/ 2023-11-02T18:02:50.017 DEBUG:boto:auth_path=/ibtljn-15/ 2023-11-02T18:02:50.017 DEBUG:boto:Method: HEAD 2023-11-02T18:02:50.018 DEBUG:boto:Path: /ibtljn-15/ 2023-11-02T18:02:50.018 DEBUG:boto:Data: 2023-11-02T18:02:50.019 DEBUG:boto:Headers: {} 2023-11-02T18:02:50.019 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:50.020 DEBUG:boto:Port: 8000 2023-11-02T18:02:50.020 DEBUG:boto:Params: {} 2023-11-02T18:02:50.021 DEBUG:boto:Token: None 2023-11-02T18:02:50.021 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:50 GMT /ibtljn-15/ 2023-11-02T18:02:50.022 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xGoJOPEW3D8X2eRRzS5XaO0eajw= 2023-11-02T18:02:50.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xGoJOPEW3D8X2eRRzS5XaO0eajw=', 'Content-Length': '0'} 2023-11-02T18:02:50.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d036263be551352-006543e44a-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:50.028 DEBUG:boto:path=/ibtljn-15/obj 2023-11-02T18:02:50.029 DEBUG:boto:auth_path=/ibtljn-15/obj 2023-11-02T18:02:50.029 DEBUG:boto:Method: PUT 2023-11-02T18:02:50.030 DEBUG:boto:Path: /ibtljn-15/obj 2023-11-02T18:02:50.031 DEBUG:boto:Data: 2023-11-02T18:02:50.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-11-02T18:02:50.033 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:50.033 DEBUG:boto:Port: 8000 2023-11-02T18:02:50.034 DEBUG:boto:Params: {} 2023-11-02T18:02:50.035 DEBUG:boto:Token: None 2023-11-02T18:02:50.036 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 02 Nov 2023 18:02:50 GMT /ibtljn-15/obj 2023-11-02T18:02:50.037 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pJjSs7wSxgb4rYGZC9Vtd2vdGp4= 2023-11-02T18:02:50.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:02:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pJjSs7wSxgb4rYGZC9Vtd2vdGp4='} 2023-11-02T18:02:50.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ'), ('x-amz-request-id', 'tx000003e3bbdc58f61f05e-006543e44a-1111-a1'), ('Date', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:50.061 DEBUG:boto:path=/ibtljn-15/ 2023-11-02T18:02:50.062 DEBUG:boto:auth_path=/ibtljn-15/ 2023-11-02T18:02:50.062 DEBUG:boto:Method: HEAD 2023-11-02T18:02:50.063 DEBUG:boto:Path: /ibtljn-15/ 2023-11-02T18:02:50.063 DEBUG:boto:Data: 2023-11-02T18:02:50.064 DEBUG:boto:Headers: {} 2023-11-02T18:02:50.064 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:50.064 DEBUG:boto:Port: 8000 2023-11-02T18:02:50.065 DEBUG:boto:Params: {} 2023-11-02T18:02:50.065 DEBUG:boto:Token: None 2023-11-02T18:02:50.066 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:50 GMT /ibtljn-15/ 2023-11-02T18:02:50.066 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xGoJOPEW3D8X2eRRzS5XaO0eajw= 2023-11-02T18:02:50.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xGoJOPEW3D8X2eRRzS5XaO0eajw=', 'Content-Length': '0'} 2023-11-02T18:02:50.070 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f910a4e38333629-006543e44a-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:50.071 DEBUG:boto:path=/ibtljn-15/obj 2023-11-02T18:02:50.071 DEBUG:boto:auth_path=/ibtljn-15/obj 2023-11-02T18:02:50.072 DEBUG:boto:Method: DELETE 2023-11-02T18:02:50.072 DEBUG:boto:Path: /ibtljn-15/obj 2023-11-02T18:02:50.073 DEBUG:boto:Data: 2023-11-02T18:02:50.073 DEBUG:boto:Headers: {} 2023-11-02T18:02:50.073 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:02:50.074 DEBUG:boto:Port: 8000 2023-11-02T18:02:50.074 DEBUG:boto:Params: {} 2023-11-02T18:02:50.075 DEBUG:boto:Token: None 2023-11-02T18:02:50.075 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 18:02:50 GMT /ibtljn-15/obj 2023-11-02T18:02:50.076 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0bAVCffLLtt+P9WnJjTtk6+pAlQ= 2023-11-02T18:02:50.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0bAVCffLLtt+P9WnJjTtk6+pAlQ=', 'Content-Length': '0'} 2023-11-02T18:02:50.090 DEBUG:boto:Response headers: [('x-amz-version-id', 'PtYW3TRLj77BZBSY9AeBDkfLweFnXZC'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000345333053329490f-006543e44a-1111-a1'), ('Date', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:50.091 DEBUG:boto:path=/ibtljn-16/ 2023-11-02T18:02:50.091 DEBUG:boto:auth_path=/ibtljn-16/ 2023-11-02T18:02:50.092 DEBUG:boto:Method: HEAD 2023-11-02T18:02:50.093 DEBUG:boto:Path: /ibtljn-16/ 2023-11-02T18:02:50.093 DEBUG:boto:Data: 2023-11-02T18:02:50.094 DEBUG:boto:Headers: {} 2023-11-02T18:02:50.095 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:50.095 DEBUG:boto:Port: 8001 2023-11-02T18:02:50.096 DEBUG:boto:Params: {} 2023-11-02T18:02:50.097 DEBUG:boto:Token: None 2023-11-02T18:02:50.097 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:50 GMT /ibtljn-16/ 2023-11-02T18:02:50.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HZTHvzai6uMi8+RC4semxW08vNg= 2023-11-02T18:02:50.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HZTHvzai6uMi8+RC4semxW08vNg=', 'Content-Length': '0'} 2023-11-02T18:02:50.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b89c4babd7e9ae45-006543e44a-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:50.103 DEBUG:boto:path=/ibtljn-16/obj 2023-11-02T18:02:50.103 DEBUG:boto:auth_path=/ibtljn-16/obj 2023-11-02T18:02:50.104 DEBUG:boto:Method: PUT 2023-11-02T18:02:50.104 DEBUG:boto:Path: /ibtljn-16/obj 2023-11-02T18:02:50.105 DEBUG:boto:Data: 2023-11-02T18:02:50.105 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-11-02T18:02:50.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:50.106 DEBUG:boto:Port: 8001 2023-11-02T18:02:50.106 DEBUG:boto:Params: {} 2023-11-02T18:02:50.107 DEBUG:boto:Token: None 2023-11-02T18:02:50.107 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 02 Nov 2023 18:02:50 GMT /ibtljn-16/obj 2023-11-02T18:02:50.108 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BIMuKyo0NBnbOyEg4ga1OOvk7O0= 2023-11-02T18:02:50.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:02:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BIMuKyo0NBnbOyEg4ga1OOvk7O0='} 2023-11-02T18:02:50.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE'), ('x-amz-request-id', 'tx000007759887c50baec27-006543e44a-1115-a2'), ('Date', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:50.155 DEBUG:boto:path=/ibtljn-16/ 2023-11-02T18:02:50.156 DEBUG:boto:auth_path=/ibtljn-16/ 2023-11-02T18:02:50.156 DEBUG:boto:Method: HEAD 2023-11-02T18:02:50.157 DEBUG:boto:Path: /ibtljn-16/ 2023-11-02T18:02:50.157 DEBUG:boto:Data: 2023-11-02T18:02:50.157 DEBUG:boto:Headers: {} 2023-11-02T18:02:50.158 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:50.158 DEBUG:boto:Port: 8001 2023-11-02T18:02:50.159 DEBUG:boto:Params: {} 2023-11-02T18:02:50.159 DEBUG:boto:Token: None 2023-11-02T18:02:50.160 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:02:50 GMT /ibtljn-16/ 2023-11-02T18:02:50.160 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HZTHvzai6uMi8+RC4semxW08vNg= 2023-11-02T18:02:50.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HZTHvzai6uMi8+RC4semxW08vNg=', 'Content-Length': '0'} 2023-11-02T18:02:50.164 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6b91e316bf8c703-006543e44a-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:50.165 DEBUG:boto:path=/ibtljn-16/obj 2023-11-02T18:02:50.165 DEBUG:boto:auth_path=/ibtljn-16/obj 2023-11-02T18:02:50.166 DEBUG:boto:Method: DELETE 2023-11-02T18:02:50.166 DEBUG:boto:Path: /ibtljn-16/obj 2023-11-02T18:02:50.167 DEBUG:boto:Data: 2023-11-02T18:02:50.167 DEBUG:boto:Headers: {} 2023-11-02T18:02:50.168 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:02:50.168 DEBUG:boto:Port: 8001 2023-11-02T18:02:50.168 DEBUG:boto:Params: {} 2023-11-02T18:02:50.169 DEBUG:boto:Token: None 2023-11-02T18:02:50.170 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 18:02:50 GMT /ibtljn-16/obj 2023-11-02T18:02:50.170 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UtzxCyuPbzIsyWULiNC91zrPVOs= 2023-11-02T18:02:50.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:02:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UtzxCyuPbzIsyWULiNC91zrPVOs=', 'Content-Length': '0'} 2023-11-02T18:02:50.187 DEBUG:boto:Response headers: [('x-amz-version-id', 'WgCUiZzPDmYJWWDHPbAJHMCacalyJiP'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000187502c0f23be986-006543e44a-1115-a2'), ('Date', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:02:50.188 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-15 2023-11-02T18:02:50.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:02:50.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:02:50.189 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:02:50.243 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:02:50.244 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:02:50.311 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:50.301+0000 7f44c6f3af40 1 waiting to reach incremental sync.. 2023-11-02T18:02:55.312 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:02:55.301+0000 7f44c6f3af40 1 waiting to reach incremental sync.. 2023-11-02T18:03:00.313 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:00.301+0000 7f44c6f3af40 1 waiting to reach incremental sync.. 2023-11-02T18:03:05.317 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:05.305+0000 7f44c6f3af40 1 bucket sync caught up with source: 2023-11-02T18:03:05.317 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.6.12, , ] 2023-11-02T18:03:05.318 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.6.12, , ] 2023-11-02T18:03:05.318 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:05.305+0000 7f44c6f3af40 0 bucket checkpoint complete 2023-11-02T18:03:05.345 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-15 2023-11-02T18:03:05.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:03:05.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:03:05.347 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:03:05.407 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:03:05.408 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:03:05.476 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:05.465+0000 7fd8e9fc5f40 1 bucket sync caught up with source: 2023-11-02T18:03:05.476 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000002.8.6, , ] 2023-11-02T18:03:05.477 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000002.8.6, , ] 2023-11-02T18:03:05.477 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:05.465+0000 7fd8e9fc5f40 0 bucket checkpoint complete 2023-11-02T18:03:05.503 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-15 zones={a2, a1} 2023-11-02T18:03:05.504 DEBUG:boto:path=/ibtljn-15/ 2023-11-02T18:03:05.505 DEBUG:boto:auth_path=/ibtljn-15/ 2023-11-02T18:03:05.505 DEBUG:boto:Method: HEAD 2023-11-02T18:03:05.506 DEBUG:boto:Path: /ibtljn-15/ 2023-11-02T18:03:05.506 DEBUG:boto:Data: 2023-11-02T18:03:05.507 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.507 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:05.507 DEBUG:boto:Port: 8001 2023-11-02T18:03:05.508 DEBUG:boto:Params: {} 2023-11-02T18:03:05.508 DEBUG:boto:Token: None 2023-11-02T18:03:05.509 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/ 2023-11-02T18:03:05.510 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Sa6HmdpbCY3j9ptgZjs5XPvrfHQ= 2023-11-02T18:03:05.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Sa6HmdpbCY3j9ptgZjs5XPvrfHQ=', 'Content-Length': '0'} 2023-11-02T18:03:05.514 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037edec476aaf9fd5-006543e459-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.515 DEBUG:boto:path=/ibtljn-15/ 2023-11-02T18:03:05.515 DEBUG:boto:auth_path=/ibtljn-15/ 2023-11-02T18:03:05.515 DEBUG:boto:Method: HEAD 2023-11-02T18:03:05.516 DEBUG:boto:Path: /ibtljn-15/ 2023-11-02T18:03:05.516 DEBUG:boto:Data: 2023-11-02T18:03:05.517 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.517 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:05.518 DEBUG:boto:Port: 8000 2023-11-02T18:03:05.518 DEBUG:boto:Params: {} 2023-11-02T18:03:05.518 DEBUG:boto:Token: None 2023-11-02T18:03:05.519 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/ 2023-11-02T18:03:05.519 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Sa6HmdpbCY3j9ptgZjs5XPvrfHQ= 2023-11-02T18:03:05.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Sa6HmdpbCY3j9ptgZjs5XPvrfHQ=', 'Content-Length': '0'} 2023-11-02T18:03:05.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dbde40e849920e9-006543e459-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.524 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:03:05.524 DEBUG:boto:path=/ibtljn-15/ 2023-11-02T18:03:05.525 DEBUG:boto:auth_path=/ibtljn-15/ 2023-11-02T18:03:05.525 DEBUG:boto:path=/ibtljn-15/?versions&max-keys=999 2023-11-02T18:03:05.526 DEBUG:boto:auth_path=/ibtljn-15/?versions&max-keys=999 2023-11-02T18:03:05.526 DEBUG:boto:Method: GET 2023-11-02T18:03:05.527 DEBUG:boto:Path: /ibtljn-15/?versions&max-keys=999 2023-11-02T18:03:05.527 DEBUG:boto:Data: 2023-11-02T18:03:05.528 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.528 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:05.529 DEBUG:boto:Port: 8001 2023-11-02T18:03:05.529 DEBUG:boto:Params: {} 2023-11-02T18:03:05.529 DEBUG:boto:Token: None 2023-11-02T18:03:05.530 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/?versions 2023-11-02T18:03:05.530 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:77lkB6gBI3zfyI7r8XWryOBuniE= 2023-11-02T18:03:05.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:77lkB6gBI3zfyI7r8XWryOBuniE=', 'Content-Length': '0'} 2023-11-02T18:03:05.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cc01c7c5e1db9b1-006543e459-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.536 DEBUG:boto:b'ibtljn-15999falseobjPtYW3TRLj77BZBSY9AeBDkfLweFnXZCtrue2023-11-02T18:02:50.073Zrgw-multisite-test-userMultisite Test UserNormalobjZ83c7j2TAT8cbMIZ-591DnfjcAp7PDZfalse2023-11-02T18:02:50.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:03:05.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:03:05.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:03:05.537 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:03:05.538 DEBUG:boto:path=/ibtljn-15/ 2023-11-02T18:03:05.538 DEBUG:boto:auth_path=/ibtljn-15/ 2023-11-02T18:03:05.539 DEBUG:boto:path=/ibtljn-15/?versions&max-keys=999 2023-11-02T18:03:05.539 DEBUG:boto:auth_path=/ibtljn-15/?versions&max-keys=999 2023-11-02T18:03:05.540 DEBUG:boto:Method: GET 2023-11-02T18:03:05.540 DEBUG:boto:Path: /ibtljn-15/?versions&max-keys=999 2023-11-02T18:03:05.540 DEBUG:boto:Data: 2023-11-02T18:03:05.541 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:05.542 DEBUG:boto:Port: 8000 2023-11-02T18:03:05.542 DEBUG:boto:Params: {} 2023-11-02T18:03:05.543 DEBUG:boto:Token: None 2023-11-02T18:03:05.543 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/?versions 2023-11-02T18:03:05.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:77lkB6gBI3zfyI7r8XWryOBuniE= 2023-11-02T18:03:05.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:77lkB6gBI3zfyI7r8XWryOBuniE=', 'Content-Length': '0'} 2023-11-02T18:03:05.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008d35c61b55ee362-006543e459-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.594 DEBUG:boto:b'ibtljn-15999falseobjPtYW3TRLj77BZBSY9AeBDkfLweFnXZCtrue2023-11-02T18:02:50.073Zrgw-multisite-test-userMultisite Test UserNormalobjZ83c7j2TAT8cbMIZ-591DnfjcAp7PDZfalse2023-11-02T18:02:50.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:03:05.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:03:05.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:03:05.596 DEBUG:boto:path=/ibtljn-15/ 2023-11-02T18:03:05.597 DEBUG:boto:auth_path=/ibtljn-15/ 2023-11-02T18:03:05.597 DEBUG:boto:path=/ibtljn-15/?versions&max-keys=999 2023-11-02T18:03:05.598 DEBUG:boto:auth_path=/ibtljn-15/?versions&max-keys=999 2023-11-02T18:03:05.598 DEBUG:boto:Method: GET 2023-11-02T18:03:05.598 DEBUG:boto:Path: /ibtljn-15/?versions&max-keys=999 2023-11-02T18:03:05.599 DEBUG:boto:Data: 2023-11-02T18:03:05.599 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:05.600 DEBUG:boto:Port: 8001 2023-11-02T18:03:05.601 DEBUG:boto:Params: {} 2023-11-02T18:03:05.601 DEBUG:boto:Token: None 2023-11-02T18:03:05.602 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/?versions 2023-11-02T18:03:05.603 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:77lkB6gBI3zfyI7r8XWryOBuniE= 2023-11-02T18:03:05.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:77lkB6gBI3zfyI7r8XWryOBuniE=', 'Content-Length': '0'} 2023-11-02T18:03:05.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c110f9a87eacbabb-006543e459-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.609 DEBUG:boto:b'ibtljn-15999falseobjPtYW3TRLj77BZBSY9AeBDkfLweFnXZCtrue2023-11-02T18:02:50.073Zrgw-multisite-test-userMultisite Test UserNormalobjZ83c7j2TAT8cbMIZ-591DnfjcAp7PDZfalse2023-11-02T18:02:50.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:03:05.610 DEBUG:boto:path=/ibtljn-15/ 2023-11-02T18:03:05.610 DEBUG:boto:auth_path=/ibtljn-15/ 2023-11-02T18:03:05.610 DEBUG:boto:path=/ibtljn-15/?versions&max-keys=999 2023-11-02T18:03:05.611 DEBUG:boto:auth_path=/ibtljn-15/?versions&max-keys=999 2023-11-02T18:03:05.611 DEBUG:boto:Method: GET 2023-11-02T18:03:05.612 DEBUG:boto:Path: /ibtljn-15/?versions&max-keys=999 2023-11-02T18:03:05.612 DEBUG:boto:Data: 2023-11-02T18:03:05.612 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.613 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:05.613 DEBUG:boto:Port: 8000 2023-11-02T18:03:05.614 DEBUG:boto:Params: {} 2023-11-02T18:03:05.614 DEBUG:boto:Token: None 2023-11-02T18:03:05.615 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/?versions 2023-11-02T18:03:05.615 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:77lkB6gBI3zfyI7r8XWryOBuniE= 2023-11-02T18:03:05.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:77lkB6gBI3zfyI7r8XWryOBuniE=', 'Content-Length': '0'} 2023-11-02T18:03:05.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ed18daea415f009-006543e459-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.621 DEBUG:boto:b'ibtljn-15999falseobjPtYW3TRLj77BZBSY9AeBDkfLweFnXZCtrue2023-11-02T18:02:50.073Zrgw-multisite-test-userMultisite Test UserNormalobjZ83c7j2TAT8cbMIZ-591DnfjcAp7PDZfalse2023-11-02T18:02:50.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:03:05.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:03:05.622 DEBUG:boto:path=/ibtljn-15/obj 2023-11-02T18:03:05.622 DEBUG:boto:auth_path=/ibtljn-15/obj 2023-11-02T18:03:05.623 DEBUG:boto:Method: HEAD 2023-11-02T18:03:05.623 DEBUG:boto:Path: /ibtljn-15/obj 2023-11-02T18:03:05.624 DEBUG:boto:Data: 2023-11-02T18:03:05.624 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.625 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:05.625 DEBUG:boto:Port: 8001 2023-11-02T18:03:05.625 DEBUG:boto:Params: {} 2023-11-02T18:03:05.626 DEBUG:boto:Token: None 2023-11-02T18:03:05.626 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/obj 2023-11-02T18:03:05.627 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cwmAsTrNXaurDj1GaFWrcGolUXw= 2023-11-02T18:03:05.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cwmAsTrNXaurDj1GaFWrcGolUXw=', 'Content-Length': '0'} 2023-11-02T18:03:05.631 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007aad362c18cc3471-006543e459-1115-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.632 DEBUG:boto:path=/ibtljn-15/obj 2023-11-02T18:03:05.632 DEBUG:boto:auth_path=/ibtljn-15/obj 2023-11-02T18:03:05.633 DEBUG:boto:Method: HEAD 2023-11-02T18:03:05.633 DEBUG:boto:Path: /ibtljn-15/obj 2023-11-02T18:03:05.633 DEBUG:boto:Data: 2023-11-02T18:03:05.634 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.634 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:05.635 DEBUG:boto:Port: 8000 2023-11-02T18:03:05.635 DEBUG:boto:Params: {} 2023-11-02T18:03:05.635 DEBUG:boto:Token: None 2023-11-02T18:03:05.636 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/obj 2023-11-02T18:03:05.636 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cwmAsTrNXaurDj1GaFWrcGolUXw= 2023-11-02T18:03:05.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cwmAsTrNXaurDj1GaFWrcGolUXw=', 'Content-Length': '0'} 2023-11-02T18:03:05.641 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006aa473ff7ae6fbfc-006543e459-1111-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:03:05.641 DEBUG:boto:path=/ibtljn-15/obj 2023-11-02T18:03:05.642 DEBUG:boto:auth_path=/ibtljn-15/obj 2023-11-02T18:03:05.642 DEBUG:boto:path=/ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.643 DEBUG:boto:auth_path=/ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.643 DEBUG:boto:Method: GET 2023-11-02T18:03:05.644 DEBUG:boto:Path: /ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.644 DEBUG:boto:Data: 2023-11-02T18:03:05.644 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.645 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:05.645 DEBUG:boto:Port: 8001 2023-11-02T18:03:05.646 DEBUG:boto:Params: {} 2023-11-02T18:03:05.646 DEBUG:boto:Token: None 2023-11-02T18:03:05.647 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.647 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:14kjRK7NN32oanQ0uWQNUMA8sEM= 2023-11-02T18:03:05.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:14kjRK7NN32oanQ0uWQNUMA8sEM=', 'Content-Length': '0'} 2023-11-02T18:03:05.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('x-amz-version-id', 'Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7c520ca3ec687d9-006543e459-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.652 DEBUG:boto:path=/ibtljn-15/obj 2023-11-02T18:03:05.652 DEBUG:boto:auth_path=/ibtljn-15/obj 2023-11-02T18:03:05.653 DEBUG:boto:path=/ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.653 DEBUG:boto:auth_path=/ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.654 DEBUG:boto:Method: GET 2023-11-02T18:03:05.654 DEBUG:boto:Path: /ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.655 DEBUG:boto:Data: 2023-11-02T18:03:05.655 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.656 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:05.656 DEBUG:boto:Port: 8000 2023-11-02T18:03:05.657 DEBUG:boto:Params: {} 2023-11-02T18:03:05.657 DEBUG:boto:Token: None 2023-11-02T18:03:05.658 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.658 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:14kjRK7NN32oanQ0uWQNUMA8sEM= 2023-11-02T18:03:05.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:14kjRK7NN32oanQ0uWQNUMA8sEM=', 'Content-Length': '0'} 2023-11-02T18:03:05.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('x-amz-version-id', 'Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003834acdbb3af488b-006543e459-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.662 DEBUG:boto:path=/ibtljn-15/obj 2023-11-02T18:03:05.663 DEBUG:boto:auth_path=/ibtljn-15/obj 2023-11-02T18:03:05.663 DEBUG:boto:path=/ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.664 DEBUG:boto:auth_path=/ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.664 DEBUG:boto:Method: HEAD 2023-11-02T18:03:05.664 DEBUG:boto:Path: /ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.665 DEBUG:boto:Data: 2023-11-02T18:03:05.665 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.666 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:05.666 DEBUG:boto:Port: 8001 2023-11-02T18:03:05.667 DEBUG:boto:Params: {} 2023-11-02T18:03:05.667 DEBUG:boto:Token: None 2023-11-02T18:03:05.668 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.668 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ivwuQ3wdXMFhtyL/H4AuY1tf3cs= 2023-11-02T18:03:05.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ivwuQ3wdXMFhtyL/H4AuY1tf3cs=', 'Content-Length': '0'} 2023-11-02T18:03:05.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('x-amz-version-id', 'Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f1d6556695abed53-006543e459-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.673 DEBUG:boto:path=/ibtljn-15/obj 2023-11-02T18:03:05.673 DEBUG:boto:auth_path=/ibtljn-15/obj 2023-11-02T18:03:05.673 DEBUG:boto:path=/ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.674 DEBUG:boto:auth_path=/ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.674 DEBUG:boto:Method: HEAD 2023-11-02T18:03:05.675 DEBUG:boto:Path: /ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.675 DEBUG:boto:Data: 2023-11-02T18:03:05.676 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.676 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:05.676 DEBUG:boto:Port: 8000 2023-11-02T18:03:05.677 DEBUG:boto:Params: {} 2023-11-02T18:03:05.677 DEBUG:boto:Token: None 2023-11-02T18:03:05.678 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.678 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ivwuQ3wdXMFhtyL/H4AuY1tf3cs= 2023-11-02T18:03:05.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ivwuQ3wdXMFhtyL/H4AuY1tf3cs=', 'Content-Length': '0'} 2023-11-02T18:03:05.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('x-amz-version-id', 'Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040388b27ff8efa08-006543e459-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:03:05.683 DEBUG:boto:path=/ibtljn-15/obj 2023-11-02T18:03:05.683 DEBUG:boto:auth_path=/ibtljn-15/obj 2023-11-02T18:03:05.684 DEBUG:boto:path=/ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.684 DEBUG:boto:auth_path=/ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.685 DEBUG:boto:Method: GET 2023-11-02T18:03:05.685 DEBUG:boto:Path: /ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.686 DEBUG:boto:Data: 2023-11-02T18:03:05.686 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:05.687 DEBUG:boto:Port: 8001 2023-11-02T18:03:05.687 DEBUG:boto:Params: {} 2023-11-02T18:03:05.688 DEBUG:boto:Token: None 2023-11-02T18:03:05.688 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.689 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:14kjRK7NN32oanQ0uWQNUMA8sEM= 2023-11-02T18:03:05.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:14kjRK7NN32oanQ0uWQNUMA8sEM=', 'Content-Length': '0'} 2023-11-02T18:03:05.692 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('x-amz-version-id', 'Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e2a734ad1460b74-006543e459-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.693 DEBUG:boto:path=/ibtljn-15/obj 2023-11-02T18:03:05.693 DEBUG:boto:auth_path=/ibtljn-15/obj 2023-11-02T18:03:05.694 DEBUG:boto:path=/ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.694 DEBUG:boto:auth_path=/ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.695 DEBUG:boto:Method: GET 2023-11-02T18:03:05.695 DEBUG:boto:Path: /ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.696 DEBUG:boto:Data: 2023-11-02T18:03:05.696 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.696 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:05.697 DEBUG:boto:Port: 8000 2023-11-02T18:03:05.697 DEBUG:boto:Params: {} 2023-11-02T18:03:05.698 DEBUG:boto:Token: None 2023-11-02T18:03:05.698 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/obj?versionId=Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ 2023-11-02T18:03:05.699 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:14kjRK7NN32oanQ0uWQNUMA8sEM= 2023-11-02T18:03:05.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:14kjRK7NN32oanQ0uWQNUMA8sEM=', 'Content-Length': '0'} 2023-11-02T18:03:05.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('x-amz-version-id', 'Z83c7j2TAT8cbMIZ-591DnfjcAp7PDZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000abf75b98c9624129-006543e459-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.703 DEBUG:boto:path=/ibtljn-15/ 2023-11-02T18:03:05.703 DEBUG:boto:auth_path=/ibtljn-15/ 2023-11-02T18:03:05.704 DEBUG:boto:path=/ibtljn-15/?prefix=obj 2023-11-02T18:03:05.704 DEBUG:boto:auth_path=/ibtljn-15/?prefix=obj 2023-11-02T18:03:05.704 DEBUG:boto:Method: GET 2023-11-02T18:03:05.705 DEBUG:boto:Path: /ibtljn-15/?prefix=obj 2023-11-02T18:03:05.705 DEBUG:boto:Data: 2023-11-02T18:03:05.706 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.706 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:05.706 DEBUG:boto:Port: 8001 2023-11-02T18:03:05.707 DEBUG:boto:Params: {} 2023-11-02T18:03:05.707 DEBUG:boto:Token: None 2023-11-02T18:03:05.708 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/ 2023-11-02T18:03:05.708 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I6g2Z/LJ7FxgA5WRhljFshJseOw= 2023-11-02T18:03:05.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I6g2Z/LJ7FxgA5WRhljFshJseOw=', 'Content-Length': '0'} 2023-11-02T18:03:05.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076040d15afdfcba8-006543e459-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.754 DEBUG:boto:b'ibtljn-15obj1000false' 2023-11-02T18:03:05.755 DEBUG:boto:path=/ibtljn-15/ 2023-11-02T18:03:05.756 DEBUG:boto:auth_path=/ibtljn-15/ 2023-11-02T18:03:05.757 DEBUG:boto:path=/ibtljn-15/?prefix=obj 2023-11-02T18:03:05.757 DEBUG:boto:auth_path=/ibtljn-15/?prefix=obj 2023-11-02T18:03:05.758 DEBUG:boto:Method: GET 2023-11-02T18:03:05.758 DEBUG:boto:Path: /ibtljn-15/?prefix=obj 2023-11-02T18:03:05.759 DEBUG:boto:Data: 2023-11-02T18:03:05.759 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.760 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:05.760 DEBUG:boto:Port: 8000 2023-11-02T18:03:05.761 DEBUG:boto:Params: {} 2023-11-02T18:03:05.761 DEBUG:boto:Token: None 2023-11-02T18:03:05.762 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/ 2023-11-02T18:03:05.763 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I6g2Z/LJ7FxgA5WRhljFshJseOw= 2023-11-02T18:03:05.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I6g2Z/LJ7FxgA5WRhljFshJseOw=', 'Content-Length': '0'} 2023-11-02T18:03:05.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009124114ee94ace41-006543e459-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.769 DEBUG:boto:b'ibtljn-15obj1000false' 2023-11-02T18:03:05.770 DEBUG:boto:path=/ibtljn-15/obj 2023-11-02T18:03:05.770 DEBUG:boto:auth_path=/ibtljn-15/obj 2023-11-02T18:03:05.771 DEBUG:boto:Method: HEAD 2023-11-02T18:03:05.771 DEBUG:boto:Path: /ibtljn-15/obj 2023-11-02T18:03:05.771 DEBUG:boto:Data: 2023-11-02T18:03:05.772 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.772 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:05.773 DEBUG:boto:Port: 8001 2023-11-02T18:03:05.773 DEBUG:boto:Params: {} 2023-11-02T18:03:05.774 DEBUG:boto:Token: None 2023-11-02T18:03:05.774 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/obj 2023-11-02T18:03:05.775 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cwmAsTrNXaurDj1GaFWrcGolUXw= 2023-11-02T18:03:05.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cwmAsTrNXaurDj1GaFWrcGolUXw=', 'Content-Length': '0'} 2023-11-02T18:03:05.779 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000b78dcf0a1ff2c05b-006543e459-1115-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.779 DEBUG:boto:path=/ibtljn-15/obj 2023-11-02T18:03:05.780 DEBUG:boto:auth_path=/ibtljn-15/obj 2023-11-02T18:03:05.780 DEBUG:boto:Method: HEAD 2023-11-02T18:03:05.781 DEBUG:boto:Path: /ibtljn-15/obj 2023-11-02T18:03:05.781 DEBUG:boto:Data: 2023-11-02T18:03:05.782 DEBUG:boto:Headers: {} 2023-11-02T18:03:05.782 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:05.783 DEBUG:boto:Port: 8000 2023-11-02T18:03:05.783 DEBUG:boto:Params: {} 2023-11-02T18:03:05.784 DEBUG:boto:Token: None 2023-11-02T18:03:05.784 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:05 GMT /ibtljn-15/obj 2023-11-02T18:03:05.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cwmAsTrNXaurDj1GaFWrcGolUXw= 2023-11-02T18:03:05.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cwmAsTrNXaurDj1GaFWrcGolUXw=', 'Content-Length': '0'} 2023-11-02T18:03:05.789 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000008b795ce76ba43c8-006543e459-1111-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:05.790 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-15 zones={a2, a1} 2023-11-02T18:03:05.790 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-16 2023-11-02T18:03:05.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:03:05.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:03:05.791 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:03:05.844 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:03:05.845 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:03:05.911 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:05.901+0000 7f566692bf40 1 waiting to reach incremental sync.. 2023-11-02T18:03:10.912 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:10.901+0000 7f566692bf40 1 waiting to reach incremental sync.. 2023-11-02T18:03:15.913 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:15.901+0000 7f566692bf40 1 waiting to reach incremental sync.. 2023-11-02T18:03:20.916 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:20.905+0000 7f566692bf40 1 bucket sync caught up with source: 2023-11-02T18:03:20.917 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000002.80.6, , ] 2023-11-02T18:03:20.917 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000002.80.6, , ] 2023-11-02T18:03:20.917 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:20.905+0000 7f566692bf40 0 bucket checkpoint complete 2023-11-02T18:03:20.947 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-16 2023-11-02T18:03:20.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:03:20.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:03:20.949 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:03:20.999 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:03:21.000 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:03:21.064 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:21.053+0000 7f640ff92f40 1 bucket sync caught up with source: 2023-11-02T18:03:21.065 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.95.12, , ] 2023-11-02T18:03:21.065 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.95.12, , ] 2023-11-02T18:03:21.065 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:21.053+0000 7f640ff92f40 0 bucket checkpoint complete 2023-11-02T18:03:21.090 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-16 zones={a2, a1} 2023-11-02T18:03:21.091 DEBUG:boto:path=/ibtljn-16/ 2023-11-02T18:03:21.091 DEBUG:boto:auth_path=/ibtljn-16/ 2023-11-02T18:03:21.092 DEBUG:boto:Method: HEAD 2023-11-02T18:03:21.092 DEBUG:boto:Path: /ibtljn-16/ 2023-11-02T18:03:21.093 DEBUG:boto:Data: 2023-11-02T18:03:21.093 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.094 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:21.094 DEBUG:boto:Port: 8001 2023-11-02T18:03:21.094 DEBUG:boto:Params: {} 2023-11-02T18:03:21.095 DEBUG:boto:Token: None 2023-11-02T18:03:21.096 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/ 2023-11-02T18:03:21.096 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9urbskzJwaALpcCheM1JyPseRig= 2023-11-02T18:03:21.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9urbskzJwaALpcCheM1JyPseRig=', 'Content-Length': '0'} 2023-11-02T18:03:21.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d02cea0d28e8a22-006543e469-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.101 DEBUG:boto:path=/ibtljn-16/ 2023-11-02T18:03:21.101 DEBUG:boto:auth_path=/ibtljn-16/ 2023-11-02T18:03:21.102 DEBUG:boto:Method: HEAD 2023-11-02T18:03:21.102 DEBUG:boto:Path: /ibtljn-16/ 2023-11-02T18:03:21.102 DEBUG:boto:Data: 2023-11-02T18:03:21.103 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.103 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:21.104 DEBUG:boto:Port: 8000 2023-11-02T18:03:21.104 DEBUG:boto:Params: {} 2023-11-02T18:03:21.104 DEBUG:boto:Token: None 2023-11-02T18:03:21.105 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/ 2023-11-02T18:03:21.105 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9urbskzJwaALpcCheM1JyPseRig= 2023-11-02T18:03:21.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9urbskzJwaALpcCheM1JyPseRig=', 'Content-Length': '0'} 2023-11-02T18:03:21.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000133e5f21fab63b37-006543e469-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.110 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:03:21.111 DEBUG:boto:path=/ibtljn-16/ 2023-11-02T18:03:21.111 DEBUG:boto:auth_path=/ibtljn-16/ 2023-11-02T18:03:21.111 DEBUG:boto:path=/ibtljn-16/?versions&max-keys=999 2023-11-02T18:03:21.112 DEBUG:boto:auth_path=/ibtljn-16/?versions&max-keys=999 2023-11-02T18:03:21.112 DEBUG:boto:Method: GET 2023-11-02T18:03:21.113 DEBUG:boto:Path: /ibtljn-16/?versions&max-keys=999 2023-11-02T18:03:21.113 DEBUG:boto:Data: 2023-11-02T18:03:21.113 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.114 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:21.114 DEBUG:boto:Port: 8001 2023-11-02T18:03:21.115 DEBUG:boto:Params: {} 2023-11-02T18:03:21.115 DEBUG:boto:Token: None 2023-11-02T18:03:21.115 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/?versions 2023-11-02T18:03:21.116 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TaL1aFJ9HhNRCNCmunRQcif1y5k= 2023-11-02T18:03:21.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TaL1aFJ9HhNRCNCmunRQcif1y5k=', 'Content-Length': '0'} 2023-11-02T18:03:21.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052c21d5ac12368ca-006543e469-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.121 DEBUG:boto:b'ibtljn-16999falseobjWgCUiZzPDmYJWWDHPbAJHMCacalyJiPtrue2023-11-02T18:02:50.168Zrgw-multisite-test-userMultisite Test UserNormalobjalBQ3ksKhH4QP4ZiIgpptSsI8eZizfEfalse2023-11-02T18:02:50.105Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:03:21.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:03:21.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:03:21.122 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:03:21.123 DEBUG:boto:path=/ibtljn-16/ 2023-11-02T18:03:21.123 DEBUG:boto:auth_path=/ibtljn-16/ 2023-11-02T18:03:21.123 DEBUG:boto:path=/ibtljn-16/?versions&max-keys=999 2023-11-02T18:03:21.124 DEBUG:boto:auth_path=/ibtljn-16/?versions&max-keys=999 2023-11-02T18:03:21.124 DEBUG:boto:Method: GET 2023-11-02T18:03:21.125 DEBUG:boto:Path: /ibtljn-16/?versions&max-keys=999 2023-11-02T18:03:21.125 DEBUG:boto:Data: 2023-11-02T18:03:21.125 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.126 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:21.126 DEBUG:boto:Port: 8000 2023-11-02T18:03:21.127 DEBUG:boto:Params: {} 2023-11-02T18:03:21.127 DEBUG:boto:Token: None 2023-11-02T18:03:21.127 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/?versions 2023-11-02T18:03:21.128 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TaL1aFJ9HhNRCNCmunRQcif1y5k= 2023-11-02T18:03:21.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TaL1aFJ9HhNRCNCmunRQcif1y5k=', 'Content-Length': '0'} 2023-11-02T18:03:21.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ee171118a08af98-006543e469-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.133 DEBUG:boto:b'ibtljn-16999falseobjWgCUiZzPDmYJWWDHPbAJHMCacalyJiPtrue2023-11-02T18:02:50.168Zrgw-multisite-test-userMultisite Test UserNormalobjalBQ3ksKhH4QP4ZiIgpptSsI8eZizfEfalse2023-11-02T18:02:50.105Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:03:21.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:03:21.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:03:21.135 DEBUG:boto:path=/ibtljn-16/ 2023-11-02T18:03:21.135 DEBUG:boto:auth_path=/ibtljn-16/ 2023-11-02T18:03:21.136 DEBUG:boto:path=/ibtljn-16/?versions&max-keys=999 2023-11-02T18:03:21.136 DEBUG:boto:auth_path=/ibtljn-16/?versions&max-keys=999 2023-11-02T18:03:21.137 DEBUG:boto:Method: GET 2023-11-02T18:03:21.137 DEBUG:boto:Path: /ibtljn-16/?versions&max-keys=999 2023-11-02T18:03:21.138 DEBUG:boto:Data: 2023-11-02T18:03:21.138 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:21.139 DEBUG:boto:Port: 8001 2023-11-02T18:03:21.139 DEBUG:boto:Params: {} 2023-11-02T18:03:21.140 DEBUG:boto:Token: None 2023-11-02T18:03:21.140 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/?versions 2023-11-02T18:03:21.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TaL1aFJ9HhNRCNCmunRQcif1y5k= 2023-11-02T18:03:21.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TaL1aFJ9HhNRCNCmunRQcif1y5k=', 'Content-Length': '0'} 2023-11-02T18:03:21.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f961c2e51d1492c-006543e469-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.186 DEBUG:boto:b'ibtljn-16999falseobjWgCUiZzPDmYJWWDHPbAJHMCacalyJiPtrue2023-11-02T18:02:50.168Zrgw-multisite-test-userMultisite Test UserNormalobjalBQ3ksKhH4QP4ZiIgpptSsI8eZizfEfalse2023-11-02T18:02:50.105Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:03:21.187 DEBUG:boto:path=/ibtljn-16/ 2023-11-02T18:03:21.187 DEBUG:boto:auth_path=/ibtljn-16/ 2023-11-02T18:03:21.188 DEBUG:boto:path=/ibtljn-16/?versions&max-keys=999 2023-11-02T18:03:21.188 DEBUG:boto:auth_path=/ibtljn-16/?versions&max-keys=999 2023-11-02T18:03:21.189 DEBUG:boto:Method: GET 2023-11-02T18:03:21.189 DEBUG:boto:Path: /ibtljn-16/?versions&max-keys=999 2023-11-02T18:03:21.189 DEBUG:boto:Data: 2023-11-02T18:03:21.190 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.190 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:21.191 DEBUG:boto:Port: 8000 2023-11-02T18:03:21.191 DEBUG:boto:Params: {} 2023-11-02T18:03:21.192 DEBUG:boto:Token: None 2023-11-02T18:03:21.192 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/?versions 2023-11-02T18:03:21.193 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TaL1aFJ9HhNRCNCmunRQcif1y5k= 2023-11-02T18:03:21.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TaL1aFJ9HhNRCNCmunRQcif1y5k=', 'Content-Length': '0'} 2023-11-02T18:03:21.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eeb5c9e006e5318-006543e469-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.198 DEBUG:boto:b'ibtljn-16999falseobjWgCUiZzPDmYJWWDHPbAJHMCacalyJiPtrue2023-11-02T18:02:50.168Zrgw-multisite-test-userMultisite Test UserNormalobjalBQ3ksKhH4QP4ZiIgpptSsI8eZizfEfalse2023-11-02T18:02:50.105Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:03:21.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:03:21.199 DEBUG:boto:path=/ibtljn-16/obj 2023-11-02T18:03:21.200 DEBUG:boto:auth_path=/ibtljn-16/obj 2023-11-02T18:03:21.200 DEBUG:boto:Method: HEAD 2023-11-02T18:03:21.201 DEBUG:boto:Path: /ibtljn-16/obj 2023-11-02T18:03:21.201 DEBUG:boto:Data: 2023-11-02T18:03:21.201 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:21.202 DEBUG:boto:Port: 8001 2023-11-02T18:03:21.203 DEBUG:boto:Params: {} 2023-11-02T18:03:21.203 DEBUG:boto:Token: None 2023-11-02T18:03:21.204 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/obj 2023-11-02T18:03:21.204 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EUyTDuw0vPbFfdICzV3eYS15i9w= 2023-11-02T18:03:21.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EUyTDuw0vPbFfdICzV3eYS15i9w=', 'Content-Length': '0'} 2023-11-02T18:03:21.208 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000cfb6f4f72a4099f2-006543e469-1115-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.209 DEBUG:boto:path=/ibtljn-16/obj 2023-11-02T18:03:21.209 DEBUG:boto:auth_path=/ibtljn-16/obj 2023-11-02T18:03:21.209 DEBUG:boto:Method: HEAD 2023-11-02T18:03:21.210 DEBUG:boto:Path: /ibtljn-16/obj 2023-11-02T18:03:21.210 DEBUG:boto:Data: 2023-11-02T18:03:21.211 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.211 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:21.211 DEBUG:boto:Port: 8000 2023-11-02T18:03:21.212 DEBUG:boto:Params: {} 2023-11-02T18:03:21.212 DEBUG:boto:Token: None 2023-11-02T18:03:21.213 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/obj 2023-11-02T18:03:21.213 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EUyTDuw0vPbFfdICzV3eYS15i9w= 2023-11-02T18:03:21.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EUyTDuw0vPbFfdICzV3eYS15i9w=', 'Content-Length': '0'} 2023-11-02T18:03:21.218 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002f2fa22203bcf786-006543e469-1111-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:03:21.219 DEBUG:boto:path=/ibtljn-16/obj 2023-11-02T18:03:21.219 DEBUG:boto:auth_path=/ibtljn-16/obj 2023-11-02T18:03:21.219 DEBUG:boto:path=/ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.220 DEBUG:boto:auth_path=/ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.220 DEBUG:boto:Method: GET 2023-11-02T18:03:21.221 DEBUG:boto:Path: /ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.221 DEBUG:boto:Data: 2023-11-02T18:03:21.222 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.222 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:21.223 DEBUG:boto:Port: 8001 2023-11-02T18:03:21.223 DEBUG:boto:Params: {} 2023-11-02T18:03:21.223 DEBUG:boto:Token: None 2023-11-02T18:03:21.224 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.224 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0B9iblwPm18hBHiohzpXj/38ps8= 2023-11-02T18:03:21.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0B9iblwPm18hBHiohzpXj/38ps8=', 'Content-Length': '0'} 2023-11-02T18:03:21.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('x-amz-version-id', 'alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002dfebfbf27c2dcaf-006543e469-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.229 DEBUG:boto:path=/ibtljn-16/obj 2023-11-02T18:03:21.229 DEBUG:boto:auth_path=/ibtljn-16/obj 2023-11-02T18:03:21.230 DEBUG:boto:path=/ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.230 DEBUG:boto:auth_path=/ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.230 DEBUG:boto:Method: GET 2023-11-02T18:03:21.231 DEBUG:boto:Path: /ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.231 DEBUG:boto:Data: 2023-11-02T18:03:21.232 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.232 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:21.233 DEBUG:boto:Port: 8000 2023-11-02T18:03:21.233 DEBUG:boto:Params: {} 2023-11-02T18:03:21.234 DEBUG:boto:Token: None 2023-11-02T18:03:21.234 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.235 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0B9iblwPm18hBHiohzpXj/38ps8= 2023-11-02T18:03:21.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0B9iblwPm18hBHiohzpXj/38ps8=', 'Content-Length': '0'} 2023-11-02T18:03:21.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('x-amz-version-id', 'alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f5374916ac65b350-006543e469-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.239 DEBUG:boto:path=/ibtljn-16/obj 2023-11-02T18:03:21.239 DEBUG:boto:auth_path=/ibtljn-16/obj 2023-11-02T18:03:21.240 DEBUG:boto:path=/ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.240 DEBUG:boto:auth_path=/ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.241 DEBUG:boto:Method: HEAD 2023-11-02T18:03:21.241 DEBUG:boto:Path: /ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.241 DEBUG:boto:Data: 2023-11-02T18:03:21.242 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.242 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:21.243 DEBUG:boto:Port: 8001 2023-11-02T18:03:21.243 DEBUG:boto:Params: {} 2023-11-02T18:03:21.244 DEBUG:boto:Token: None 2023-11-02T18:03:21.244 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.245 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XkDpiq4hvwhrBnEVL+MeZbjx0gk= 2023-11-02T18:03:21.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XkDpiq4hvwhrBnEVL+MeZbjx0gk=', 'Content-Length': '0'} 2023-11-02T18:03:21.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('x-amz-version-id', 'alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fdb44b6718b798ca-006543e469-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.248 DEBUG:boto:path=/ibtljn-16/obj 2023-11-02T18:03:21.249 DEBUG:boto:auth_path=/ibtljn-16/obj 2023-11-02T18:03:21.249 DEBUG:boto:path=/ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.250 DEBUG:boto:auth_path=/ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.250 DEBUG:boto:Method: HEAD 2023-11-02T18:03:21.250 DEBUG:boto:Path: /ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.251 DEBUG:boto:Data: 2023-11-02T18:03:21.251 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.252 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:21.252 DEBUG:boto:Port: 8000 2023-11-02T18:03:21.253 DEBUG:boto:Params: {} 2023-11-02T18:03:21.253 DEBUG:boto:Token: None 2023-11-02T18:03:21.253 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.254 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XkDpiq4hvwhrBnEVL+MeZbjx0gk= 2023-11-02T18:03:21.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XkDpiq4hvwhrBnEVL+MeZbjx0gk=', 'Content-Length': '0'} 2023-11-02T18:03:21.257 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('x-amz-version-id', 'alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d644cdbaa52d53ae-006543e469-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:03:21.258 DEBUG:boto:path=/ibtljn-16/obj 2023-11-02T18:03:21.259 DEBUG:boto:auth_path=/ibtljn-16/obj 2023-11-02T18:03:21.259 DEBUG:boto:path=/ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.259 DEBUG:boto:auth_path=/ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.260 DEBUG:boto:Method: GET 2023-11-02T18:03:21.260 DEBUG:boto:Path: /ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.261 DEBUG:boto:Data: 2023-11-02T18:03:21.261 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.262 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:21.262 DEBUG:boto:Port: 8001 2023-11-02T18:03:21.263 DEBUG:boto:Params: {} 2023-11-02T18:03:21.263 DEBUG:boto:Token: None 2023-11-02T18:03:21.264 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.264 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0B9iblwPm18hBHiohzpXj/38ps8= 2023-11-02T18:03:21.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0B9iblwPm18hBHiohzpXj/38ps8=', 'Content-Length': '0'} 2023-11-02T18:03:21.267 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('x-amz-version-id', 'alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001acd7f7be7a3aafc-006543e469-1115-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.268 DEBUG:boto:path=/ibtljn-16/obj 2023-11-02T18:03:21.269 DEBUG:boto:auth_path=/ibtljn-16/obj 2023-11-02T18:03:21.269 DEBUG:boto:path=/ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.269 DEBUG:boto:auth_path=/ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.270 DEBUG:boto:Method: GET 2023-11-02T18:03:21.270 DEBUG:boto:Path: /ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.271 DEBUG:boto:Data: 2023-11-02T18:03:21.271 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.272 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:21.272 DEBUG:boto:Port: 8000 2023-11-02T18:03:21.273 DEBUG:boto:Params: {} 2023-11-02T18:03:21.273 DEBUG:boto:Token: None 2023-11-02T18:03:21.274 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/obj?versionId=alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE 2023-11-02T18:03:21.274 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0B9iblwPm18hBHiohzpXj/38ps8= 2023-11-02T18:03:21.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0B9iblwPm18hBHiohzpXj/38ps8=', 'Content-Length': '0'} 2023-11-02T18:03:21.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:02:50 GMT'), ('x-amz-version-id', 'alBQ3ksKhH4QP4ZiIgpptSsI8eZizfE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000434f52658c190d7a-006543e469-1111-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.278 DEBUG:boto:path=/ibtljn-16/ 2023-11-02T18:03:21.279 DEBUG:boto:auth_path=/ibtljn-16/ 2023-11-02T18:03:21.279 DEBUG:boto:path=/ibtljn-16/?prefix=obj 2023-11-02T18:03:21.280 DEBUG:boto:auth_path=/ibtljn-16/?prefix=obj 2023-11-02T18:03:21.280 DEBUG:boto:Method: GET 2023-11-02T18:03:21.280 DEBUG:boto:Path: /ibtljn-16/?prefix=obj 2023-11-02T18:03:21.281 DEBUG:boto:Data: 2023-11-02T18:03:21.281 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.282 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:21.282 DEBUG:boto:Port: 8001 2023-11-02T18:03:21.283 DEBUG:boto:Params: {} 2023-11-02T18:03:21.283 DEBUG:boto:Token: None 2023-11-02T18:03:21.284 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/ 2023-11-02T18:03:21.284 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OWbaoEfi+nt7+fHR+DuDTPj+L8I= 2023-11-02T18:03:21.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OWbaoEfi+nt7+fHR+DuDTPj+L8I=', 'Content-Length': '0'} 2023-11-02T18:03:21.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068d2a0fc1545309b-006543e469-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.330 DEBUG:boto:b'ibtljn-16obj1000false' 2023-11-02T18:03:21.331 DEBUG:boto:path=/ibtljn-16/ 2023-11-02T18:03:21.331 DEBUG:boto:auth_path=/ibtljn-16/ 2023-11-02T18:03:21.332 DEBUG:boto:path=/ibtljn-16/?prefix=obj 2023-11-02T18:03:21.332 DEBUG:boto:auth_path=/ibtljn-16/?prefix=obj 2023-11-02T18:03:21.333 DEBUG:boto:Method: GET 2023-11-02T18:03:21.333 DEBUG:boto:Path: /ibtljn-16/?prefix=obj 2023-11-02T18:03:21.334 DEBUG:boto:Data: 2023-11-02T18:03:21.334 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.334 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:21.335 DEBUG:boto:Port: 8000 2023-11-02T18:03:21.335 DEBUG:boto:Params: {} 2023-11-02T18:03:21.336 DEBUG:boto:Token: None 2023-11-02T18:03:21.336 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/ 2023-11-02T18:03:21.337 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OWbaoEfi+nt7+fHR+DuDTPj+L8I= 2023-11-02T18:03:21.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OWbaoEfi+nt7+fHR+DuDTPj+L8I=', 'Content-Length': '0'} 2023-11-02T18:03:21.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005da73827a4141a39-006543e469-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.342 DEBUG:boto:b'ibtljn-16obj1000false' 2023-11-02T18:03:21.343 DEBUG:boto:path=/ibtljn-16/obj 2023-11-02T18:03:21.343 DEBUG:boto:auth_path=/ibtljn-16/obj 2023-11-02T18:03:21.344 DEBUG:boto:Method: HEAD 2023-11-02T18:03:21.344 DEBUG:boto:Path: /ibtljn-16/obj 2023-11-02T18:03:21.345 DEBUG:boto:Data: 2023-11-02T18:03:21.345 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.346 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:21.346 DEBUG:boto:Port: 8001 2023-11-02T18:03:21.346 DEBUG:boto:Params: {} 2023-11-02T18:03:21.347 DEBUG:boto:Token: None 2023-11-02T18:03:21.347 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/obj 2023-11-02T18:03:21.348 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EUyTDuw0vPbFfdICzV3eYS15i9w= 2023-11-02T18:03:21.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EUyTDuw0vPbFfdICzV3eYS15i9w=', 'Content-Length': '0'} 2023-11-02T18:03:21.352 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006dfb3ddbdb492ea6-006543e469-1115-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.352 DEBUG:boto:path=/ibtljn-16/obj 2023-11-02T18:03:21.353 DEBUG:boto:auth_path=/ibtljn-16/obj 2023-11-02T18:03:21.353 DEBUG:boto:Method: HEAD 2023-11-02T18:03:21.353 DEBUG:boto:Path: /ibtljn-16/obj 2023-11-02T18:03:21.354 DEBUG:boto:Data: 2023-11-02T18:03:21.354 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.355 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:21.355 DEBUG:boto:Port: 8000 2023-11-02T18:03:21.356 DEBUG:boto:Params: {} 2023-11-02T18:03:21.356 DEBUG:boto:Token: None 2023-11-02T18:03:21.356 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-16/obj 2023-11-02T18:03:21.357 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EUyTDuw0vPbFfdICzV3eYS15i9w= 2023-11-02T18:03:21.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EUyTDuw0vPbFfdICzV3eYS15i9w=', 'Content-Length': '0'} 2023-11-02T18:03:21.361 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000018593bd0b367980a-006543e469-1111-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.361 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-16 zones={a2, a1} 2023-11-02T18:03:21.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2023-11-02T18:03:21.363 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-17 2023-11-02T18:03:21.364 DEBUG:boto:path=/ibtljn-17/ 2023-11-02T18:03:21.364 DEBUG:boto:auth_path=/ibtljn-17/ 2023-11-02T18:03:21.364 DEBUG:boto:Method: PUT 2023-11-02T18:03:21.365 DEBUG:boto:Path: /ibtljn-17/ 2023-11-02T18:03:21.365 DEBUG:boto:Data: 2023-11-02T18:03:21.365 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:21.366 DEBUG:boto:Port: 8000 2023-11-02T18:03:21.367 DEBUG:boto:Params: {} 2023-11-02T18:03:21.367 DEBUG:boto:Token: None 2023-11-02T18:03:21.368 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-17/ 2023-11-02T18:03:21.368 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uOmZb3GEcMGndfUjnE2zYYTqZ2M= 2023-11-02T18:03:21.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uOmZb3GEcMGndfUjnE2zYYTqZ2M=', 'Content-Length': '0'} 2023-11-02T18:03:21.394 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b9f4a4c7710c8419-006543e469-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.395 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-18 2023-11-02T18:03:21.395 DEBUG:boto:path=/ibtljn-18/ 2023-11-02T18:03:21.395 DEBUG:boto:auth_path=/ibtljn-18/ 2023-11-02T18:03:21.396 DEBUG:boto:Method: PUT 2023-11-02T18:03:21.396 DEBUG:boto:Path: /ibtljn-18/ 2023-11-02T18:03:21.396 DEBUG:boto:Data: 2023-11-02T18:03:21.397 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.397 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:21.398 DEBUG:boto:Port: 8001 2023-11-02T18:03:21.398 DEBUG:boto:Params: {} 2023-11-02T18:03:21.399 DEBUG:boto:Token: None 2023-11-02T18:03:21.399 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-18/ 2023-11-02T18:03:21.399 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kFywQnCazY+TZ8pnZ5Aw3toz950= 2023-11-02T18:03:21.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kFywQnCazY+TZ8pnZ5Aw3toz950=', 'Content-Length': '0'} 2023-11-02T18:03:21.438 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f864cd4862334161-006543e469-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.439 DEBUG:boto:path=/ibtljn-17/ 2023-11-02T18:03:21.439 DEBUG:boto:auth_path=/ibtljn-17/ 2023-11-02T18:03:21.440 DEBUG:boto:path=/ibtljn-17/?versioning 2023-11-02T18:03:21.440 DEBUG:boto:auth_path=/ibtljn-17/?versioning 2023-11-02T18:03:21.441 DEBUG:boto:Method: PUT 2023-11-02T18:03:21.441 DEBUG:boto:Path: /ibtljn-17/?versioning 2023-11-02T18:03:21.441 DEBUG:boto:Data: Enabled Disabled 2023-11-02T18:03:21.442 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.442 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:21.443 DEBUG:boto:Port: 8000 2023-11-02T18:03:21.443 DEBUG:boto:Params: {} 2023-11-02T18:03:21.443 DEBUG:boto:Token: None 2023-11-02T18:03:21.444 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-17/?versioning 2023-11-02T18:03:21.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3vdr6ZCbwPShrg/1+DJp9PTGsno= 2023-11-02T18:03:21.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3vdr6ZCbwPShrg/1+DJp9PTGsno=', 'Content-Length': '228'} 2023-11-02T18:03:21.453 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f4088b4ee9c57e49-006543e469-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.454 DEBUG:boto:path=/ibtljn-17/ 2023-11-02T18:03:21.454 DEBUG:boto:auth_path=/ibtljn-17/ 2023-11-02T18:03:21.455 DEBUG:boto:path=/ibtljn-17/?versioning 2023-11-02T18:03:21.455 DEBUG:boto:auth_path=/ibtljn-17/?versioning 2023-11-02T18:03:21.456 DEBUG:boto:Method: PUT 2023-11-02T18:03:21.456 DEBUG:boto:Path: /ibtljn-17/?versioning 2023-11-02T18:03:21.456 DEBUG:boto:Data: Suspended Disabled 2023-11-02T18:03:21.457 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.457 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:21.458 DEBUG:boto:Port: 8000 2023-11-02T18:03:21.458 DEBUG:boto:Params: {} 2023-11-02T18:03:21.458 DEBUG:boto:Token: None 2023-11-02T18:03:21.459 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-17/?versioning 2023-11-02T18:03:21.459 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3vdr6ZCbwPShrg/1+DJp9PTGsno= 2023-11-02T18:03:21.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3vdr6ZCbwPShrg/1+DJp9PTGsno=', 'Content-Length': '230'} 2023-11-02T18:03:21.475 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000113db82491123b62-006543e469-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.475 DEBUG:boto:path=/ibtljn-18/ 2023-11-02T18:03:21.476 DEBUG:boto:auth_path=/ibtljn-18/ 2023-11-02T18:03:21.476 DEBUG:boto:path=/ibtljn-18/?versioning 2023-11-02T18:03:21.477 DEBUG:boto:auth_path=/ibtljn-18/?versioning 2023-11-02T18:03:21.477 DEBUG:boto:Method: PUT 2023-11-02T18:03:21.477 DEBUG:boto:Path: /ibtljn-18/?versioning 2023-11-02T18:03:21.478 DEBUG:boto:Data: Enabled Disabled 2023-11-02T18:03:21.478 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.479 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:21.479 DEBUG:boto:Port: 8001 2023-11-02T18:03:21.479 DEBUG:boto:Params: {} 2023-11-02T18:03:21.480 DEBUG:boto:Token: None 2023-11-02T18:03:21.480 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-18/?versioning 2023-11-02T18:03:21.481 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sPPgdf93mT3YuEmrEFbHQDYfmZY= 2023-11-02T18:03:21.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sPPgdf93mT3YuEmrEFbHQDYfmZY=', 'Content-Length': '228'} 2023-11-02T18:03:21.493 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ec9f5a3ff15b7b3-006543e469-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.493 DEBUG:boto:path=/ibtljn-18/ 2023-11-02T18:03:21.494 DEBUG:boto:auth_path=/ibtljn-18/ 2023-11-02T18:03:21.494 DEBUG:boto:path=/ibtljn-18/?versioning 2023-11-02T18:03:21.495 DEBUG:boto:auth_path=/ibtljn-18/?versioning 2023-11-02T18:03:21.495 DEBUG:boto:Method: PUT 2023-11-02T18:03:21.495 DEBUG:boto:Path: /ibtljn-18/?versioning 2023-11-02T18:03:21.496 DEBUG:boto:Data: Suspended Disabled 2023-11-02T18:03:21.496 DEBUG:boto:Headers: {} 2023-11-02T18:03:21.497 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:21.497 DEBUG:boto:Port: 8001 2023-11-02T18:03:21.498 DEBUG:boto:Params: {} 2023-11-02T18:03:21.498 DEBUG:boto:Token: None 2023-11-02T18:03:21.498 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:03:21 GMT /ibtljn-18/?versioning 2023-11-02T18:03:21.499 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sPPgdf93mT3YuEmrEFbHQDYfmZY= 2023-11-02T18:03:21.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sPPgdf93mT3YuEmrEFbHQDYfmZY=', 'Content-Length': '230'} 2023-11-02T18:03:21.516 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002add9d9ecbb85a94-006543e469-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:21.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:03:21.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:03:21.519 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:03:21.565 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:03:21.565 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:03:21.621 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:03:21.649 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:03:21.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:03:21.650 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:03:21.650 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:03:21.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:21.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:03:21.651 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:03:21.651 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:03:21.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:21.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:03:21.652 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948201.503855_116.1", 2023-11-02T18:03:21.652 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:03:21.503855Z" 2023-11-02T18:03:21.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:21.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:03:21.653 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:03:21.653 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:03:21.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:03:21.654 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:03:21.655 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948201.503855_116.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:03:21.655 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:03:21.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:03:21.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:03:21.657 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:03:21.699 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:03:21.700 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:03:21.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:21.745+0000 7f4b7230ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:21.745+0000 7f4b7230ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:03:21.779 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:03:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:03:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:03:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:03:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:03:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:03:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:03:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:03:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:03:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:03:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:03:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:03:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:03:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:03:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:03:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:03:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:03:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:03:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:03:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:03:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:03:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:03:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:03:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:03:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:03:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:03:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:03:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:03:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:03:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:03:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:03:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:03:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:03:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:03:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948201.482414_114.1", 2023-11-02T18:03:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:03:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:03:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:03:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:03:21.482414Z", 2023-11-02T18:03:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:03:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:03:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:03:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:03:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:03:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:03:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:03:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:03:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:03:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:03:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:03:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:03:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:03:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:03:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:03:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:03:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:03:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:03:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:03:21Z" 2023-11-02T18:03:21.794 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:03:21.795 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948201.482414_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:03:21.482414Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:03:21Z" } 2023-11-02T18:03:21.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948201.482414_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:03:21.482414Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:03:21.796 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948201.503855_116.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:03:21.796 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948201.482414_114.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:03:21.797 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948201.503855_116.1 target=1_1698948201.482414_114.1 2023-11-02T18:03:26.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:03:26.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:03:26.799 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:03:26.845 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:03:26.845 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:03:26.902 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:26.893+0000 7fb7a32f7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:03:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:26.893+0000 7fb7a32f7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:03:26.929 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:03:26.930 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:03:26.930 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:03:26.930 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:03:26.930 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:03:26.931 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:03:26.931 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:03:26.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:26.931 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:03:26.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:03:26.932 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:03:26.932 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:03:26.932 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:03:26.933 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:03:26.933 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:03:26.933 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:03:26.933 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:03:26.933 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:03:26.934 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:03:26.934 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:03:26.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:26.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:03:26.935 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:03:26.935 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:03:26.935 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:03:26.935 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:03:26.935 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:03:26.936 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:03:26.936 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:03:26.936 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:03:26.936 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:03:26.936 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:03:26.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:26.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:03:26.937 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:03:26.937 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:03:26.937 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:03:26.937 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948201.503855_116.1", 2023-11-02T18:03:26.938 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:03:26.938 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:03:26.938 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:03:26.938 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:03:21.503855Z", 2023-11-02T18:03:26.939 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:03:26.939 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:03:26.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:26.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:03:26.939 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:03:26.940 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:03:26.940 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:03:26.940 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:03:26.940 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:03:26.941 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:03:26.941 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:03:26.941 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:03:26.941 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:03:26.941 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:03:26.942 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:03:26.942 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:03:26.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:26.942 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:03:26.942 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:03:26.943 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:03:26.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:03:26.943 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:03:26Z" 2023-11-02T18:03:26.943 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:03:26.944 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948201.503855_116.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:03:21.503855Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:03:26Z" } 2023-11-02T18:03:26.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948201.503855_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:03:21.503855Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:03:26.945 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948201.503855_116.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:03:26.946 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948201.503855_116.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:03:26.946 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:03:26.947 DEBUG:boto:path=/ibtljn-17/ 2023-11-02T18:03:26.947 DEBUG:boto:auth_path=/ibtljn-17/ 2023-11-02T18:03:26.947 DEBUG:boto:Method: HEAD 2023-11-02T18:03:26.948 DEBUG:boto:Path: /ibtljn-17/ 2023-11-02T18:03:26.948 DEBUG:boto:Data: 2023-11-02T18:03:26.949 DEBUG:boto:Headers: {} 2023-11-02T18:03:26.949 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:26.950 DEBUG:boto:Port: 8000 2023-11-02T18:03:26.950 DEBUG:boto:Params: {} 2023-11-02T18:03:26.950 DEBUG:boto:Token: None 2023-11-02T18:03:26.951 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:26 GMT /ibtljn-17/ 2023-11-02T18:03:26.951 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ujzAzltXW5VfNKPYbAP6wR6qfP8= 2023-11-02T18:03:26.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ujzAzltXW5VfNKPYbAP6wR6qfP8=', 'Content-Length': '0'} 2023-11-02T18:03:26.956 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000790a735b560eef59-006543e46e-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:26.957 DEBUG:boto:path=/ibtljn-17/obj 2023-11-02T18:03:26.957 DEBUG:boto:auth_path=/ibtljn-17/obj 2023-11-02T18:03:26.957 DEBUG:boto:Method: PUT 2023-11-02T18:03:26.958 DEBUG:boto:Path: /ibtljn-17/obj 2023-11-02T18:03:26.958 DEBUG:boto:Data: 2023-11-02T18:03:26.959 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-11-02T18:03:26.959 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:26.960 DEBUG:boto:Port: 8000 2023-11-02T18:03:26.960 DEBUG:boto:Params: {} 2023-11-02T18:03:26.960 DEBUG:boto:Token: None 2023-11-02T18:03:26.961 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 02 Nov 2023 18:03:26 GMT /ibtljn-17/obj 2023-11-02T18:03:26.961 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y0AIb26dBwmQiDcnDu/qNqhIUAo= 2023-11-02T18:03:26.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:03:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y0AIb26dBwmQiDcnDu/qNqhIUAo='} 2023-11-02T18:03:26.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000daae8bfc0819dba4-006543e46e-1111-a1'), ('Date', 'Thu, 02 Nov 2023 18:03:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:26.977 DEBUG:boto:path=/ibtljn-17/ 2023-11-02T18:03:26.977 DEBUG:boto:auth_path=/ibtljn-17/ 2023-11-02T18:03:26.978 DEBUG:boto:Method: HEAD 2023-11-02T18:03:26.978 DEBUG:boto:Path: /ibtljn-17/ 2023-11-02T18:03:26.978 DEBUG:boto:Data: 2023-11-02T18:03:26.979 DEBUG:boto:Headers: {} 2023-11-02T18:03:26.979 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:26.980 DEBUG:boto:Port: 8000 2023-11-02T18:03:26.980 DEBUG:boto:Params: {} 2023-11-02T18:03:26.980 DEBUG:boto:Token: None 2023-11-02T18:03:26.981 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:26 GMT /ibtljn-17/ 2023-11-02T18:03:26.981 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ujzAzltXW5VfNKPYbAP6wR6qfP8= 2023-11-02T18:03:26.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ujzAzltXW5VfNKPYbAP6wR6qfP8=', 'Content-Length': '0'} 2023-11-02T18:03:26.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee42680c5a743fa4-006543e46e-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:26.986 DEBUG:boto:path=/ibtljn-17/obj 2023-11-02T18:03:26.986 DEBUG:boto:auth_path=/ibtljn-17/obj 2023-11-02T18:03:26.987 DEBUG:boto:Method: DELETE 2023-11-02T18:03:26.987 DEBUG:boto:Path: /ibtljn-17/obj 2023-11-02T18:03:26.988 DEBUG:boto:Data: 2023-11-02T18:03:26.988 DEBUG:boto:Headers: {} 2023-11-02T18:03:26.989 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:03:26.989 DEBUG:boto:Port: 8000 2023-11-02T18:03:26.989 DEBUG:boto:Params: {} 2023-11-02T18:03:26.990 DEBUG:boto:Token: None 2023-11-02T18:03:26.990 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 18:03:26 GMT /ibtljn-17/obj 2023-11-02T18:03:26.991 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uY7JNmbz5rqa1F/EpAlsRLnfrPw= 2023-11-02T18:03:26.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uY7JNmbz5rqa1F/EpAlsRLnfrPw=', 'Content-Length': '0'} 2023-11-02T18:03:27.012 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000002bd7bcdb12015892-006543e46e-1111-a1'), ('Date', 'Thu, 02 Nov 2023 18:03:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:27.013 DEBUG:boto:path=/ibtljn-18/ 2023-11-02T18:03:27.014 DEBUG:boto:auth_path=/ibtljn-18/ 2023-11-02T18:03:27.014 DEBUG:boto:Method: HEAD 2023-11-02T18:03:27.014 DEBUG:boto:Path: /ibtljn-18/ 2023-11-02T18:03:27.015 DEBUG:boto:Data: 2023-11-02T18:03:27.017 DEBUG:boto:Headers: {} 2023-11-02T18:03:27.018 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:27.018 DEBUG:boto:Port: 8001 2023-11-02T18:03:27.019 DEBUG:boto:Params: {} 2023-11-02T18:03:27.019 DEBUG:boto:Token: None 2023-11-02T18:03:27.020 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:27 GMT /ibtljn-18/ 2023-11-02T18:03:27.020 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hBL3qZhVYuUcMt0MXDnGjlux0Kc= 2023-11-02T18:03:27.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hBL3qZhVYuUcMt0MXDnGjlux0Kc=', 'Content-Length': '0'} 2023-11-02T18:03:27.025 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b6d01dc75782615-006543e46f-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:27.025 DEBUG:boto:path=/ibtljn-18/obj 2023-11-02T18:03:27.026 DEBUG:boto:auth_path=/ibtljn-18/obj 2023-11-02T18:03:27.026 DEBUG:boto:Method: PUT 2023-11-02T18:03:27.027 DEBUG:boto:Path: /ibtljn-18/obj 2023-11-02T18:03:27.027 DEBUG:boto:Data: 2023-11-02T18:03:27.027 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-11-02T18:03:27.028 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:27.028 DEBUG:boto:Port: 8001 2023-11-02T18:03:27.029 DEBUG:boto:Params: {} 2023-11-02T18:03:27.029 DEBUG:boto:Token: None 2023-11-02T18:03:27.030 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 02 Nov 2023 18:03:27 GMT /ibtljn-18/obj 2023-11-02T18:03:27.030 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AUqLzVA5BcZgK0sGOItUHIqjwYc= 2023-11-02T18:03:27.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:03:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AUqLzVA5BcZgK0sGOItUHIqjwYc='} 2023-11-02T18:03:27.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf20a10871d8b731-006543e46f-1115-a2'), ('Date', 'Thu, 02 Nov 2023 18:03:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:27.075 DEBUG:boto:path=/ibtljn-18/ 2023-11-02T18:03:27.075 DEBUG:boto:auth_path=/ibtljn-18/ 2023-11-02T18:03:27.076 DEBUG:boto:Method: HEAD 2023-11-02T18:03:27.076 DEBUG:boto:Path: /ibtljn-18/ 2023-11-02T18:03:27.077 DEBUG:boto:Data: 2023-11-02T18:03:27.077 DEBUG:boto:Headers: {} 2023-11-02T18:03:27.077 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:27.078 DEBUG:boto:Port: 8001 2023-11-02T18:03:27.078 DEBUG:boto:Params: {} 2023-11-02T18:03:27.079 DEBUG:boto:Token: None 2023-11-02T18:03:27.079 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:03:27 GMT /ibtljn-18/ 2023-11-02T18:03:27.079 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hBL3qZhVYuUcMt0MXDnGjlux0Kc= 2023-11-02T18:03:27.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hBL3qZhVYuUcMt0MXDnGjlux0Kc=', 'Content-Length': '0'} 2023-11-02T18:03:27.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068a2f8ca246cfbf6-006543e46f-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:03:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:27.085 DEBUG:boto:path=/ibtljn-18/obj 2023-11-02T18:03:27.085 DEBUG:boto:auth_path=/ibtljn-18/obj 2023-11-02T18:03:27.086 DEBUG:boto:Method: DELETE 2023-11-02T18:03:27.086 DEBUG:boto:Path: /ibtljn-18/obj 2023-11-02T18:03:27.087 DEBUG:boto:Data: 2023-11-02T18:03:27.087 DEBUG:boto:Headers: {} 2023-11-02T18:03:27.087 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:03:27.088 DEBUG:boto:Port: 8001 2023-11-02T18:03:27.088 DEBUG:boto:Params: {} 2023-11-02T18:03:27.089 DEBUG:boto:Token: None 2023-11-02T18:03:27.089 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 18:03:27 GMT /ibtljn-18/obj 2023-11-02T18:03:27.090 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZTSKUIWBY89qrp6eR/wn5AvOxmM= 2023-11-02T18:03:27.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:03:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZTSKUIWBY89qrp6eR/wn5AvOxmM=', 'Content-Length': '0'} 2023-11-02T18:03:27.112 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000079e2be4b42ed6997-006543e46f-1115-a2'), ('Date', 'Thu, 02 Nov 2023 18:03:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:03:27.113 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-17 2023-11-02T18:03:27.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:03:27.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:03:27.114 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:03:27.166 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:03:27.166 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:03:27.230 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:27.221+0000 7f667cb00f40 1 waiting to reach incremental sync.. 2023-11-02T18:03:32.231 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:32.221+0000 7f667cb00f40 1 waiting to reach incremental sync.. 2023-11-02T18:03:37.232 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:37.221+0000 7f667cb00f40 1 waiting to reach incremental sync.. 2023-11-02T18:03:42.235 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:42.225+0000 7f667cb00f40 1 bucket sync caught up with source: 2023-11-02T18:03:42.235 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.13.7, , ] 2023-11-02T18:03:42.236 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.13.7, , ] 2023-11-02T18:03:42.236 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:42.225+0000 7f667cb00f40 0 bucket checkpoint complete 2023-11-02T18:03:42.264 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-17 2023-11-02T18:03:42.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:03:42.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:03:42.265 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:03:42.317 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:03:42.318 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:03:42.386 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:42.377+0000 7f83f0ebef40 1 waiting to reach incremental sync.. 2023-11-02T18:03:47.387 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:47.377+0000 7f83f0ebef40 1 waiting to reach incremental sync.. 2023-11-02T18:03:52.388 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:52.377+0000 7f83f0ebef40 1 waiting to reach incremental sync.. 2023-11-02T18:03:57.389 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:03:57.377+0000 7f83f0ebef40 1 waiting to reach incremental sync.. 2023-11-02T18:04:02.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:02.381+0000 7f83f0ebef40 1 bucket sync caught up with source: 2023-11-02T18:04:02.392 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.19.14, , ] 2023-11-02T18:04:02.393 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000001.19.14, , ] 2023-11-02T18:04:02.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:02.381+0000 7f83f0ebef40 0 bucket checkpoint complete 2023-11-02T18:04:02.420 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-17 zones={a2, a1} 2023-11-02T18:04:02.421 DEBUG:boto:path=/ibtljn-17/ 2023-11-02T18:04:02.421 DEBUG:boto:auth_path=/ibtljn-17/ 2023-11-02T18:04:02.422 DEBUG:boto:Method: HEAD 2023-11-02T18:04:02.422 DEBUG:boto:Path: /ibtljn-17/ 2023-11-02T18:04:02.423 DEBUG:boto:Data: 2023-11-02T18:04:02.423 DEBUG:boto:Headers: {} 2023-11-02T18:04:02.423 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:02.424 DEBUG:boto:Port: 8001 2023-11-02T18:04:02.424 DEBUG:boto:Params: {} 2023-11-02T18:04:02.425 DEBUG:boto:Token: None 2023-11-02T18:04:02.425 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:02 GMT /ibtljn-17/ 2023-11-02T18:04:02.426 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nbZIZKmw4OwRnE2ugBN9aAf7XHw= 2023-11-02T18:04:02.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nbZIZKmw4OwRnE2ugBN9aAf7XHw=', 'Content-Length': '0'} 2023-11-02T18:04:02.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000201a4c4e3ca69cc5-006543e492-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:02.431 DEBUG:boto:path=/ibtljn-17/ 2023-11-02T18:04:02.432 DEBUG:boto:auth_path=/ibtljn-17/ 2023-11-02T18:04:02.432 DEBUG:boto:Method: HEAD 2023-11-02T18:04:02.433 DEBUG:boto:Path: /ibtljn-17/ 2023-11-02T18:04:02.433 DEBUG:boto:Data: 2023-11-02T18:04:02.433 DEBUG:boto:Headers: {} 2023-11-02T18:04:02.434 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:02.434 DEBUG:boto:Port: 8000 2023-11-02T18:04:02.435 DEBUG:boto:Params: {} 2023-11-02T18:04:02.435 DEBUG:boto:Token: None 2023-11-02T18:04:02.436 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:02 GMT /ibtljn-17/ 2023-11-02T18:04:02.436 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nbZIZKmw4OwRnE2ugBN9aAf7XHw= 2023-11-02T18:04:02.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nbZIZKmw4OwRnE2ugBN9aAf7XHw=', 'Content-Length': '0'} 2023-11-02T18:04:02.441 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a36ccd77eaacf83-006543e492-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:02.441 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:04:02.442 DEBUG:boto:path=/ibtljn-17/ 2023-11-02T18:04:02.442 DEBUG:boto:auth_path=/ibtljn-17/ 2023-11-02T18:04:02.443 DEBUG:boto:path=/ibtljn-17/?versions&max-keys=999 2023-11-02T18:04:02.445 DEBUG:boto:auth_path=/ibtljn-17/?versions&max-keys=999 2023-11-02T18:04:02.446 DEBUG:boto:Method: GET 2023-11-02T18:04:02.446 DEBUG:boto:Path: /ibtljn-17/?versions&max-keys=999 2023-11-02T18:04:02.447 DEBUG:boto:Data: 2023-11-02T18:04:02.447 DEBUG:boto:Headers: {} 2023-11-02T18:04:02.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:02.448 DEBUG:boto:Port: 8001 2023-11-02T18:04:02.448 DEBUG:boto:Params: {} 2023-11-02T18:04:02.449 DEBUG:boto:Token: None 2023-11-02T18:04:02.449 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:02 GMT /ibtljn-17/?versions 2023-11-02T18:04:02.450 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Od7lMcP2K2FWcuzOSUbkMY13fbQ= 2023-11-02T18:04:02.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Od7lMcP2K2FWcuzOSUbkMY13fbQ=', 'Content-Length': '0'} 2023-11-02T18:04:02.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeed2c636e487fe1-006543e492-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:02.498 DEBUG:boto:b'ibtljn-17999falseobjnulltrue2023-11-02T18:03:26.988Zrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:04:02.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:04:02.500 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:04:02.500 DEBUG:boto:path=/ibtljn-17/ 2023-11-02T18:04:02.501 DEBUG:boto:auth_path=/ibtljn-17/ 2023-11-02T18:04:02.501 DEBUG:boto:path=/ibtljn-17/?versions&max-keys=999 2023-11-02T18:04:02.502 DEBUG:boto:auth_path=/ibtljn-17/?versions&max-keys=999 2023-11-02T18:04:02.502 DEBUG:boto:Method: GET 2023-11-02T18:04:02.502 DEBUG:boto:Path: /ibtljn-17/?versions&max-keys=999 2023-11-02T18:04:02.503 DEBUG:boto:Data: 2023-11-02T18:04:02.503 DEBUG:boto:Headers: {} 2023-11-02T18:04:02.504 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:02.504 DEBUG:boto:Port: 8000 2023-11-02T18:04:02.505 DEBUG:boto:Params: {} 2023-11-02T18:04:02.505 DEBUG:boto:Token: None 2023-11-02T18:04:02.506 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:02 GMT /ibtljn-17/?versions 2023-11-02T18:04:02.506 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Od7lMcP2K2FWcuzOSUbkMY13fbQ= 2023-11-02T18:04:02.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Od7lMcP2K2FWcuzOSUbkMY13fbQ=', 'Content-Length': '0'} 2023-11-02T18:04:02.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df66c0e2a86b6042-006543e492-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:02.512 DEBUG:boto:b'ibtljn-17999falseobjnulltrue2023-11-02T18:03:26.988Zrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:04:02.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:04:02.513 DEBUG:boto:path=/ibtljn-17/ 2023-11-02T18:04:02.513 DEBUG:boto:auth_path=/ibtljn-17/ 2023-11-02T18:04:02.514 DEBUG:boto:path=/ibtljn-17/?versions&max-keys=999 2023-11-02T18:04:02.514 DEBUG:boto:auth_path=/ibtljn-17/?versions&max-keys=999 2023-11-02T18:04:02.515 DEBUG:boto:Method: GET 2023-11-02T18:04:02.515 DEBUG:boto:Path: /ibtljn-17/?versions&max-keys=999 2023-11-02T18:04:02.516 DEBUG:boto:Data: 2023-11-02T18:04:02.516 DEBUG:boto:Headers: {} 2023-11-02T18:04:02.517 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:02.517 DEBUG:boto:Port: 8001 2023-11-02T18:04:02.518 DEBUG:boto:Params: {} 2023-11-02T18:04:02.518 DEBUG:boto:Token: None 2023-11-02T18:04:02.519 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:02 GMT /ibtljn-17/?versions 2023-11-02T18:04:02.519 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Od7lMcP2K2FWcuzOSUbkMY13fbQ= 2023-11-02T18:04:02.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Od7lMcP2K2FWcuzOSUbkMY13fbQ=', 'Content-Length': '0'} 2023-11-02T18:04:02.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006755f6125bc60970-006543e492-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:02.525 DEBUG:boto:b'ibtljn-17999falseobjnulltrue2023-11-02T18:03:26.988Zrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:04:02.526 DEBUG:boto:path=/ibtljn-17/ 2023-11-02T18:04:02.526 DEBUG:boto:auth_path=/ibtljn-17/ 2023-11-02T18:04:02.526 DEBUG:boto:path=/ibtljn-17/?versions&max-keys=999 2023-11-02T18:04:02.527 DEBUG:boto:auth_path=/ibtljn-17/?versions&max-keys=999 2023-11-02T18:04:02.527 DEBUG:boto:Method: GET 2023-11-02T18:04:02.528 DEBUG:boto:Path: /ibtljn-17/?versions&max-keys=999 2023-11-02T18:04:02.528 DEBUG:boto:Data: 2023-11-02T18:04:02.529 DEBUG:boto:Headers: {} 2023-11-02T18:04:02.529 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:02.530 DEBUG:boto:Port: 8000 2023-11-02T18:04:02.530 DEBUG:boto:Params: {} 2023-11-02T18:04:02.530 DEBUG:boto:Token: None 2023-11-02T18:04:02.531 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:02 GMT /ibtljn-17/?versions 2023-11-02T18:04:02.531 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Od7lMcP2K2FWcuzOSUbkMY13fbQ= 2023-11-02T18:04:02.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Od7lMcP2K2FWcuzOSUbkMY13fbQ=', 'Content-Length': '0'} 2023-11-02T18:04:02.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000220b1b7a0646057b-006543e492-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:02.537 DEBUG:boto:b'ibtljn-17999falseobjnulltrue2023-11-02T18:03:26.988Zrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:04:02.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:04:02.538 DEBUG:boto:path=/ibtljn-17/obj 2023-11-02T18:04:02.539 DEBUG:boto:auth_path=/ibtljn-17/obj 2023-11-02T18:04:02.539 DEBUG:boto:Method: HEAD 2023-11-02T18:04:02.540 DEBUG:boto:Path: /ibtljn-17/obj 2023-11-02T18:04:02.540 DEBUG:boto:Data: 2023-11-02T18:04:02.541 DEBUG:boto:Headers: {} 2023-11-02T18:04:02.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:02.542 DEBUG:boto:Port: 8001 2023-11-02T18:04:02.542 DEBUG:boto:Params: {} 2023-11-02T18:04:02.543 DEBUG:boto:Token: None 2023-11-02T18:04:02.543 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:02 GMT /ibtljn-17/obj 2023-11-02T18:04:02.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sbdSGf+35wjYWV8kLpLNVGLjuh8= 2023-11-02T18:04:02.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sbdSGf+35wjYWV8kLpLNVGLjuh8=', 'Content-Length': '0'} 2023-11-02T18:04:02.547 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f7a4200691d86587-006543e492-1115-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:02.547 DEBUG:boto:path=/ibtljn-17/obj 2023-11-02T18:04:02.548 DEBUG:boto:auth_path=/ibtljn-17/obj 2023-11-02T18:04:02.548 DEBUG:boto:Method: HEAD 2023-11-02T18:04:02.548 DEBUG:boto:Path: /ibtljn-17/obj 2023-11-02T18:04:02.549 DEBUG:boto:Data: 2023-11-02T18:04:02.549 DEBUG:boto:Headers: {} 2023-11-02T18:04:02.550 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:02.550 DEBUG:boto:Port: 8000 2023-11-02T18:04:02.551 DEBUG:boto:Params: {} 2023-11-02T18:04:02.551 DEBUG:boto:Token: None 2023-11-02T18:04:02.551 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:02 GMT /ibtljn-17/obj 2023-11-02T18:04:02.552 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sbdSGf+35wjYWV8kLpLNVGLjuh8= 2023-11-02T18:04:02.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sbdSGf+35wjYWV8kLpLNVGLjuh8=', 'Content-Length': '0'} 2023-11-02T18:04:02.555 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000334ac1e714d5f9ae-006543e492-1111-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:02.555 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-17 zones={a2, a1} 2023-11-02T18:04:02.556 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-18 2023-11-02T18:04:02.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:04:02.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:04:02.557 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:04:02.611 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:04:02.612 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:04:02.679 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:02.669+0000 7f734ec5bf40 1 waiting to reach incremental sync.. 2023-11-02T18:04:07.680 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:07.668+0000 7f734ec5bf40 1 waiting to reach incremental sync.. 2023-11-02T18:04:12.681 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:12.668+0000 7f734ec5bf40 1 waiting to reach incremental sync.. 2023-11-02T18:04:17.682 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:17.672+0000 7f734ec5bf40 1 waiting to reach incremental sync.. 2023-11-02T18:04:22.683 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:22.672+0000 7f734ec5bf40 1 waiting to reach incremental sync.. 2023-11-02T18:04:27.684 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:27.672+0000 7f734ec5bf40 1 waiting to reach incremental sync.. 2023-11-02T18:04:32.685 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:32.672+0000 7f734ec5bf40 1 waiting to reach incremental sync.. 2023-11-02T18:04:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:37.676+0000 7f734ec5bf40 1 waiting to reach incremental sync.. 2023-11-02T18:04:42.689 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:42.676+0000 7f734ec5bf40 1 bucket sync caught up with source: 2023-11-02T18:04:42.690 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.9.14, , ] 2023-11-02T18:04:42.690 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000001.9.14, , ] 2023-11-02T18:04:42.690 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:42.676+0000 7f734ec5bf40 0 bucket checkpoint complete 2023-11-02T18:04:42.718 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-18 2023-11-02T18:04:42.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:04:42.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:04:42.721 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:04:42.772 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:04:42.773 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:04:42.846 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:42.832+0000 7f1023e81f40 1 bucket sync caught up with source: 2023-11-02T18:04:42.846 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000003.23.7, , ] 2023-11-02T18:04:42.847 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000003.23.7, , ] 2023-11-02T18:04:42.847 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:42.832+0000 7f1023e81f40 0 bucket checkpoint complete 2023-11-02T18:04:42.872 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-18 zones={a2, a1} 2023-11-02T18:04:42.873 DEBUG:boto:path=/ibtljn-18/ 2023-11-02T18:04:42.873 DEBUG:boto:auth_path=/ibtljn-18/ 2023-11-02T18:04:42.874 DEBUG:boto:Method: HEAD 2023-11-02T18:04:42.874 DEBUG:boto:Path: /ibtljn-18/ 2023-11-02T18:04:42.875 DEBUG:boto:Data: 2023-11-02T18:04:42.875 DEBUG:boto:Headers: {} 2023-11-02T18:04:42.876 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:42.876 DEBUG:boto:Port: 8001 2023-11-02T18:04:42.876 DEBUG:boto:Params: {} 2023-11-02T18:04:42.877 DEBUG:boto:Token: None 2023-11-02T18:04:42.878 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:42 GMT /ibtljn-18/ 2023-11-02T18:04:42.878 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z375Jms56CKJzEN/hSl8+TnPpXo= 2023-11-02T18:04:42.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z375Jms56CKJzEN/hSl8+TnPpXo=', 'Content-Length': '0'} 2023-11-02T18:04:42.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7ae216533706808-006543e4ba-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:42.883 DEBUG:boto:path=/ibtljn-18/ 2023-11-02T18:04:42.884 DEBUG:boto:auth_path=/ibtljn-18/ 2023-11-02T18:04:42.884 DEBUG:boto:Method: HEAD 2023-11-02T18:04:42.884 DEBUG:boto:Path: /ibtljn-18/ 2023-11-02T18:04:42.885 DEBUG:boto:Data: 2023-11-02T18:04:42.885 DEBUG:boto:Headers: {} 2023-11-02T18:04:42.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:42.886 DEBUG:boto:Port: 8000 2023-11-02T18:04:42.886 DEBUG:boto:Params: {} 2023-11-02T18:04:42.887 DEBUG:boto:Token: None 2023-11-02T18:04:42.887 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:42 GMT /ibtljn-18/ 2023-11-02T18:04:42.888 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z375Jms56CKJzEN/hSl8+TnPpXo= 2023-11-02T18:04:42.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z375Jms56CKJzEN/hSl8+TnPpXo=', 'Content-Length': '0'} 2023-11-02T18:04:42.892 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003bc721152d7e5154-006543e4ba-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:42.892 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:04:42.893 DEBUG:boto:path=/ibtljn-18/ 2023-11-02T18:04:42.894 DEBUG:boto:auth_path=/ibtljn-18/ 2023-11-02T18:04:42.894 DEBUG:boto:path=/ibtljn-18/?versions&max-keys=999 2023-11-02T18:04:42.895 DEBUG:boto:auth_path=/ibtljn-18/?versions&max-keys=999 2023-11-02T18:04:42.895 DEBUG:boto:Method: GET 2023-11-02T18:04:42.895 DEBUG:boto:Path: /ibtljn-18/?versions&max-keys=999 2023-11-02T18:04:42.896 DEBUG:boto:Data: 2023-11-02T18:04:42.897 DEBUG:boto:Headers: {} 2023-11-02T18:04:42.897 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:42.898 DEBUG:boto:Port: 8001 2023-11-02T18:04:42.898 DEBUG:boto:Params: {} 2023-11-02T18:04:42.899 DEBUG:boto:Token: None 2023-11-02T18:04:42.899 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:42 GMT /ibtljn-18/?versions 2023-11-02T18:04:42.900 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gSmbiCytTrxagYa5oPESAcpx8VE= 2023-11-02T18:04:42.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gSmbiCytTrxagYa5oPESAcpx8VE=', 'Content-Length': '0'} 2023-11-02T18:04:42.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011ac653bd3a34b1f-006543e4ba-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:42.946 DEBUG:boto:b'ibtljn-18999falseobjnulltrue2023-11-02T18:03:27.087Zrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:04:42.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:04:42.948 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:04:42.948 DEBUG:boto:path=/ibtljn-18/ 2023-11-02T18:04:42.949 DEBUG:boto:auth_path=/ibtljn-18/ 2023-11-02T18:04:42.949 DEBUG:boto:path=/ibtljn-18/?versions&max-keys=999 2023-11-02T18:04:42.950 DEBUG:boto:auth_path=/ibtljn-18/?versions&max-keys=999 2023-11-02T18:04:42.950 DEBUG:boto:Method: GET 2023-11-02T18:04:42.951 DEBUG:boto:Path: /ibtljn-18/?versions&max-keys=999 2023-11-02T18:04:42.951 DEBUG:boto:Data: 2023-11-02T18:04:42.951 DEBUG:boto:Headers: {} 2023-11-02T18:04:42.952 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:42.952 DEBUG:boto:Port: 8000 2023-11-02T18:04:42.953 DEBUG:boto:Params: {} 2023-11-02T18:04:42.953 DEBUG:boto:Token: None 2023-11-02T18:04:42.953 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:42 GMT /ibtljn-18/?versions 2023-11-02T18:04:42.954 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gSmbiCytTrxagYa5oPESAcpx8VE= 2023-11-02T18:04:42.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gSmbiCytTrxagYa5oPESAcpx8VE=', 'Content-Length': '0'} 2023-11-02T18:04:42.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bec00c64e9de150b-006543e4ba-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:42.960 DEBUG:boto:b'ibtljn-18999falseobjnulltrue2023-11-02T18:03:27.087Zrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:04:42.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-11-02T18:04:42.961 DEBUG:boto:path=/ibtljn-18/ 2023-11-02T18:04:42.961 DEBUG:boto:auth_path=/ibtljn-18/ 2023-11-02T18:04:42.962 DEBUG:boto:path=/ibtljn-18/?versions&max-keys=999 2023-11-02T18:04:42.962 DEBUG:boto:auth_path=/ibtljn-18/?versions&max-keys=999 2023-11-02T18:04:42.963 DEBUG:boto:Method: GET 2023-11-02T18:04:42.963 DEBUG:boto:Path: /ibtljn-18/?versions&max-keys=999 2023-11-02T18:04:42.964 DEBUG:boto:Data: 2023-11-02T18:04:42.964 DEBUG:boto:Headers: {} 2023-11-02T18:04:42.964 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:42.965 DEBUG:boto:Port: 8001 2023-11-02T18:04:42.965 DEBUG:boto:Params: {} 2023-11-02T18:04:42.966 DEBUG:boto:Token: None 2023-11-02T18:04:42.966 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:42 GMT /ibtljn-18/?versions 2023-11-02T18:04:42.967 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gSmbiCytTrxagYa5oPESAcpx8VE= 2023-11-02T18:04:42.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gSmbiCytTrxagYa5oPESAcpx8VE=', 'Content-Length': '0'} 2023-11-02T18:04:42.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a6b6a26da606a16-006543e4ba-1115-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:42.973 DEBUG:boto:b'ibtljn-18999falseobjnulltrue2023-11-02T18:03:27.087Zrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:04:42.973 DEBUG:boto:path=/ibtljn-18/ 2023-11-02T18:04:42.974 DEBUG:boto:auth_path=/ibtljn-18/ 2023-11-02T18:04:42.974 DEBUG:boto:path=/ibtljn-18/?versions&max-keys=999 2023-11-02T18:04:42.975 DEBUG:boto:auth_path=/ibtljn-18/?versions&max-keys=999 2023-11-02T18:04:42.975 DEBUG:boto:Method: GET 2023-11-02T18:04:42.976 DEBUG:boto:Path: /ibtljn-18/?versions&max-keys=999 2023-11-02T18:04:42.976 DEBUG:boto:Data: 2023-11-02T18:04:42.976 DEBUG:boto:Headers: {} 2023-11-02T18:04:42.977 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:42.977 DEBUG:boto:Port: 8000 2023-11-02T18:04:42.978 DEBUG:boto:Params: {} 2023-11-02T18:04:42.978 DEBUG:boto:Token: None 2023-11-02T18:04:42.979 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:42 GMT /ibtljn-18/?versions 2023-11-02T18:04:42.979 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gSmbiCytTrxagYa5oPESAcpx8VE= 2023-11-02T18:04:42.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gSmbiCytTrxagYa5oPESAcpx8VE=', 'Content-Length': '0'} 2023-11-02T18:04:42.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000572f4200e270f750-006543e4ba-1111-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.030 DEBUG:boto:b'ibtljn-18999falseobjnulltrue2023-11-02T18:03:27.087Zrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:04:43.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-11-02T18:04:43.031 DEBUG:boto:path=/ibtljn-18/obj 2023-11-02T18:04:43.032 DEBUG:boto:auth_path=/ibtljn-18/obj 2023-11-02T18:04:43.032 DEBUG:boto:Method: HEAD 2023-11-02T18:04:43.033 DEBUG:boto:Path: /ibtljn-18/obj 2023-11-02T18:04:43.033 DEBUG:boto:Data: 2023-11-02T18:04:43.034 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.034 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:43.034 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.035 DEBUG:boto:Params: {} 2023-11-02T18:04:43.035 DEBUG:boto:Token: None 2023-11-02T18:04:43.036 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-18/obj 2023-11-02T18:04:43.036 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zRTkLX/MgLtR3pajybFZaoAgmj0= 2023-11-02T18:04:43.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zRTkLX/MgLtR3pajybFZaoAgmj0=', 'Content-Length': '0'} 2023-11-02T18:04:43.039 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000b34385835ed9a794-006543e4bb-1115-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.040 DEBUG:boto:path=/ibtljn-18/obj 2023-11-02T18:04:43.040 DEBUG:boto:auth_path=/ibtljn-18/obj 2023-11-02T18:04:43.041 DEBUG:boto:Method: HEAD 2023-11-02T18:04:43.041 DEBUG:boto:Path: /ibtljn-18/obj 2023-11-02T18:04:43.042 DEBUG:boto:Data: 2023-11-02T18:04:43.042 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.043 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:43.043 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.043 DEBUG:boto:Params: {} 2023-11-02T18:04:43.044 DEBUG:boto:Token: None 2023-11-02T18:04:43.044 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-18/obj 2023-11-02T18:04:43.045 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zRTkLX/MgLtR3pajybFZaoAgmj0= 2023-11-02T18:04:43.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zRTkLX/MgLtR3pajybFZaoAgmj0=', 'Content-Length': '0'} 2023-11-02T18:04:43.048 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004a69f987bd53ca97-006543e4bb-1111-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.049 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-18 zones={a2, a1} 2023-11-02T18:04:43.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2023-11-02T18:04:43.050 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-19 2023-11-02T18:04:43.051 DEBUG:boto:path=/ibtljn-19/ 2023-11-02T18:04:43.051 DEBUG:boto:auth_path=/ibtljn-19/ 2023-11-02T18:04:43.052 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.052 DEBUG:boto:Path: /ibtljn-19/ 2023-11-02T18:04:43.053 DEBUG:boto:Data: 2023-11-02T18:04:43.053 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.054 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:43.054 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.054 DEBUG:boto:Params: {} 2023-11-02T18:04:43.055 DEBUG:boto:Token: None 2023-11-02T18:04:43.055 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-19/ 2023-11-02T18:04:43.056 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4Oyn8wDlREYgUGIIoVSgthP/9Ls= 2023-11-02T18:04:43.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4Oyn8wDlREYgUGIIoVSgthP/9Ls=', 'Content-Length': '0'} 2023-11-02T18:04:43.082 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e450d8ebde62f2e6-006543e4bb-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.082 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-20 2023-11-02T18:04:43.083 DEBUG:boto:path=/ibtljn-20/ 2023-11-02T18:04:43.083 DEBUG:boto:auth_path=/ibtljn-20/ 2023-11-02T18:04:43.084 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.084 DEBUG:boto:Path: /ibtljn-20/ 2023-11-02T18:04:43.085 DEBUG:boto:Data: 2023-11-02T18:04:43.085 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.085 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:43.086 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.086 DEBUG:boto:Params: {} 2023-11-02T18:04:43.087 DEBUG:boto:Token: None 2023-11-02T18:04:43.087 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-20/ 2023-11-02T18:04:43.088 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cIfPnJ8pTpIUdqDaWlFfTjpOYVs= 2023-11-02T18:04:43.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cIfPnJ8pTpIUdqDaWlFfTjpOYVs=', 'Content-Length': '0'} 2023-11-02T18:04:43.127 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000023c20d5c22ee9891-006543e4bb-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.129 DEBUG:boto:Using access key provided by client. 2023-11-02T18:04:43.129 DEBUG:boto:Using secret key provided by client. 2023-11-02T18:04:43.130 DEBUG:boto:Using access key provided by client. 2023-11-02T18:04:43.131 DEBUG:boto:Using secret key provided by client. 2023-11-02T18:04:43.132 DEBUG:boto:Using access key provided by client. 2023-11-02T18:04:43.132 DEBUG:boto:Using secret key provided by client. 2023-11-02T18:04:43.133 DEBUG:boto:Using access key provided by client. 2023-11-02T18:04:43.133 DEBUG:boto:Using secret key provided by client. 2023-11-02T18:04:43.134 INFO:rgw_multi.tests:create bucket zone=b1 name=ibtljn-21 2023-11-02T18:04:43.134 DEBUG:boto:path=/ibtljn-21/ 2023-11-02T18:04:43.135 DEBUG:boto:auth_path=/ibtljn-21/ 2023-11-02T18:04:43.135 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.136 DEBUG:boto:Path: /ibtljn-21/ 2023-11-02T18:04:43.136 DEBUG:boto:Data: 2023-11-02T18:04:43.136 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.137 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:04:43.137 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.138 DEBUG:boto:Params: {} 2023-11-02T18:04:43.138 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:04:43.139 DEBUG:boto:Token: None 2023-11-02T18:04:43.139 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-21/ 2023-11-02T18:04:43.139 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iH3U+smaw/l1tp7aOTPRkaxtU9M= 2023-11-02T18:04:43.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iH3U+smaw/l1tp7aOTPRkaxtU9M=', 'Content-Length': '0'} 2023-11-02T18:04:43.192 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004cd9afa29bce41e5-006543e4bb-1107-b1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.192 INFO:rgw_multi.tests:create bucket zone=b2 name=ibtljn-22 2023-11-02T18:04:43.193 DEBUG:boto:path=/ibtljn-22/ 2023-11-02T18:04:43.193 DEBUG:boto:auth_path=/ibtljn-22/ 2023-11-02T18:04:43.194 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.194 DEBUG:boto:Path: /ibtljn-22/ 2023-11-02T18:04:43.194 DEBUG:boto:Data: 2023-11-02T18:04:43.195 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.195 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:04:43.196 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.196 DEBUG:boto:Params: {} 2023-11-02T18:04:43.197 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:04:43.197 DEBUG:boto:Token: None 2023-11-02T18:04:43.197 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-22/ 2023-11-02T18:04:43.198 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QDhR2GVRSSb33233zl5ggEhne3o= 2023-11-02T18:04:43.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QDhR2GVRSSb33233zl5ggEhne3o=', 'Content-Length': '0'} 2023-11-02T18:04:43.246 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ca2ab6794bbc09d5-006543e4bb-110d-b2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.246 DEBUG:boto:path=/ibtljn-19/ 2023-11-02T18:04:43.247 DEBUG:boto:auth_path=/ibtljn-19/ 2023-11-02T18:04:43.247 DEBUG:boto:path=/ibtljn-19/?versioning 2023-11-02T18:04:43.247 DEBUG:boto:auth_path=/ibtljn-19/?versioning 2023-11-02T18:04:43.248 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.248 DEBUG:boto:Path: /ibtljn-19/?versioning 2023-11-02T18:04:43.249 DEBUG:boto:Data: Enabled Disabled 2023-11-02T18:04:43.249 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.250 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:43.250 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.250 DEBUG:boto:Params: {} 2023-11-02T18:04:43.251 DEBUG:boto:Token: None 2023-11-02T18:04:43.251 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-19/?versioning 2023-11-02T18:04:43.252 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uOl2OQYlpnNQIf0NTfvs4FSIKrY= 2023-11-02T18:04:43.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uOl2OQYlpnNQIf0NTfvs4FSIKrY=', 'Content-Length': '228'} 2023-11-02T18:04:43.264 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007018f86e2e59d505-006543e4bb-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.265 DEBUG:boto:path=/ibtljn-19/ 2023-11-02T18:04:43.266 DEBUG:boto:auth_path=/ibtljn-19/ 2023-11-02T18:04:43.266 DEBUG:boto:path=/ibtljn-19/?versioning 2023-11-02T18:04:43.266 DEBUG:boto:auth_path=/ibtljn-19/?versioning 2023-11-02T18:04:43.267 DEBUG:boto:Method: GET 2023-11-02T18:04:43.267 DEBUG:boto:Path: /ibtljn-19/?versioning 2023-11-02T18:04:43.268 DEBUG:boto:Data: 2023-11-02T18:04:43.268 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.268 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:43.269 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.269 DEBUG:boto:Params: {} 2023-11-02T18:04:43.270 DEBUG:boto:Token: None 2023-11-02T18:04:43.270 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-19/?versioning 2023-11-02T18:04:43.271 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GnopRt2OzBuf595a+FE5L3u7MsY= 2023-11-02T18:04:43.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GnopRt2OzBuf595a+FE5L3u7MsY=', 'Content-Length': '0'} 2023-11-02T18:04:43.272 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e53e2c2914957c21-006543e4bb-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.273 DEBUG:boto:EnabledDisabled 2023-11-02T18:04:43.274 DEBUG:boto:path=/ibtljn-20/ 2023-11-02T18:04:43.274 DEBUG:boto:auth_path=/ibtljn-20/ 2023-11-02T18:04:43.275 DEBUG:boto:path=/ibtljn-20/?versioning 2023-11-02T18:04:43.275 DEBUG:boto:auth_path=/ibtljn-20/?versioning 2023-11-02T18:04:43.275 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.276 DEBUG:boto:Path: /ibtljn-20/?versioning 2023-11-02T18:04:43.276 DEBUG:boto:Data: Enabled Disabled 2023-11-02T18:04:43.277 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.277 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:43.277 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.278 DEBUG:boto:Params: {} 2023-11-02T18:04:43.278 DEBUG:boto:Token: None 2023-11-02T18:04:43.279 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-20/?versioning 2023-11-02T18:04:43.279 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FhnnI45uwss8Z3HCFxzOjz6fvz0= 2023-11-02T18:04:43.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FhnnI45uwss8Z3HCFxzOjz6fvz0=', 'Content-Length': '228'} 2023-11-02T18:04:43.290 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bd2ca02204da7f0f-006543e4bb-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.291 DEBUG:boto:path=/ibtljn-20/ 2023-11-02T18:04:43.291 DEBUG:boto:auth_path=/ibtljn-20/ 2023-11-02T18:04:43.291 DEBUG:boto:path=/ibtljn-20/?versioning 2023-11-02T18:04:43.292 DEBUG:boto:auth_path=/ibtljn-20/?versioning 2023-11-02T18:04:43.292 DEBUG:boto:Method: GET 2023-11-02T18:04:43.293 DEBUG:boto:Path: /ibtljn-20/?versioning 2023-11-02T18:04:43.293 DEBUG:boto:Data: 2023-11-02T18:04:43.293 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.294 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:43.294 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.295 DEBUG:boto:Params: {} 2023-11-02T18:04:43.295 DEBUG:boto:Token: None 2023-11-02T18:04:43.296 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-20/?versioning 2023-11-02T18:04:43.296 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gaV31n9W9rRPHaCiR0XIgqjpP7Y= 2023-11-02T18:04:43.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gaV31n9W9rRPHaCiR0XIgqjpP7Y=', 'Content-Length': '0'} 2023-11-02T18:04:43.298 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000afaf8dcb1e509f13-006543e4bb-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.298 DEBUG:boto:EnabledDisabled 2023-11-02T18:04:43.299 DEBUG:boto:path=/ibtljn-21/ 2023-11-02T18:04:43.299 DEBUG:boto:auth_path=/ibtljn-21/ 2023-11-02T18:04:43.300 DEBUG:boto:path=/ibtljn-21/?versioning 2023-11-02T18:04:43.300 DEBUG:boto:auth_path=/ibtljn-21/?versioning 2023-11-02T18:04:43.300 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.301 DEBUG:boto:Path: /ibtljn-21/?versioning 2023-11-02T18:04:43.301 DEBUG:boto:Data: Enabled Disabled 2023-11-02T18:04:43.301 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.302 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:04:43.302 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.303 DEBUG:boto:Params: {} 2023-11-02T18:04:43.303 DEBUG:boto:Token: None 2023-11-02T18:04:43.304 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-21/?versioning 2023-11-02T18:04:43.304 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hMK2KX6CIGgbZVvietJhKz6I1Dc= 2023-11-02T18:04:43.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hMK2KX6CIGgbZVvietJhKz6I1Dc=', 'Content-Length': '228'} 2023-11-02T18:04:43.325 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001d32994e5513dbca-006543e4bb-1107-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.325 DEBUG:boto:path=/ibtljn-21/ 2023-11-02T18:04:43.326 DEBUG:boto:auth_path=/ibtljn-21/ 2023-11-02T18:04:43.326 DEBUG:boto:path=/ibtljn-21/?versioning 2023-11-02T18:04:43.327 DEBUG:boto:auth_path=/ibtljn-21/?versioning 2023-11-02T18:04:43.327 DEBUG:boto:Method: GET 2023-11-02T18:04:43.327 DEBUG:boto:Path: /ibtljn-21/?versioning 2023-11-02T18:04:43.328 DEBUG:boto:Data: 2023-11-02T18:04:43.328 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.329 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:04:43.329 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.329 DEBUG:boto:Params: {} 2023-11-02T18:04:43.330 DEBUG:boto:Token: None 2023-11-02T18:04:43.330 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-21/?versioning 2023-11-02T18:04:43.331 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V73cFYtV0MSoQfJHEGrRoS7kjIk= 2023-11-02T18:04:43.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V73cFYtV0MSoQfJHEGrRoS7kjIk=', 'Content-Length': '0'} 2023-11-02T18:04:43.333 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000539757e51bbaf388-006543e4bb-1107-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.333 DEBUG:boto:EnabledDisabled 2023-11-02T18:04:43.334 DEBUG:boto:path=/ibtljn-22/ 2023-11-02T18:04:43.334 DEBUG:boto:auth_path=/ibtljn-22/ 2023-11-02T18:04:43.335 DEBUG:boto:path=/ibtljn-22/?versioning 2023-11-02T18:04:43.335 DEBUG:boto:auth_path=/ibtljn-22/?versioning 2023-11-02T18:04:43.335 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.336 DEBUG:boto:Path: /ibtljn-22/?versioning 2023-11-02T18:04:43.336 DEBUG:boto:Data: Enabled Disabled 2023-11-02T18:04:43.337 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.337 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:04:43.338 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.338 DEBUG:boto:Params: {} 2023-11-02T18:04:43.338 DEBUG:boto:Token: None 2023-11-02T18:04:43.339 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-22/?versioning 2023-11-02T18:04:43.339 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YelmuP7bjm8kPL1JFNyNj2HM470= 2023-11-02T18:04:43.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YelmuP7bjm8kPL1JFNyNj2HM470=', 'Content-Length': '228'} 2023-11-02T18:04:43.356 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004773551cabe57f9f-006543e4bb-110d-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.357 DEBUG:boto:path=/ibtljn-22/ 2023-11-02T18:04:43.357 DEBUG:boto:auth_path=/ibtljn-22/ 2023-11-02T18:04:43.358 DEBUG:boto:path=/ibtljn-22/?versioning 2023-11-02T18:04:43.358 DEBUG:boto:auth_path=/ibtljn-22/?versioning 2023-11-02T18:04:43.358 DEBUG:boto:Method: GET 2023-11-02T18:04:43.359 DEBUG:boto:Path: /ibtljn-22/?versioning 2023-11-02T18:04:43.359 DEBUG:boto:Data: 2023-11-02T18:04:43.360 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.360 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:04:43.360 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.361 DEBUG:boto:Params: {} 2023-11-02T18:04:43.361 DEBUG:boto:Token: None 2023-11-02T18:04:43.362 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-22/?versioning 2023-11-02T18:04:43.362 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x1ctvEX8AusFhNgCyKBPik2GWdM= 2023-11-02T18:04:43.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x1ctvEX8AusFhNgCyKBPik2GWdM=', 'Content-Length': '0'} 2023-11-02T18:04:43.365 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000039eb83f142997704-006543e4bb-110d-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.365 DEBUG:boto:EnabledDisabled 2023-11-02T18:04:43.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2023-11-02T18:04:43.366 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-23 2023-11-02T18:04:43.367 DEBUG:boto:path=/ibtljn-23/ 2023-11-02T18:04:43.367 DEBUG:boto:auth_path=/ibtljn-23/ 2023-11-02T18:04:43.368 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.368 DEBUG:boto:Path: /ibtljn-23/ 2023-11-02T18:04:43.369 DEBUG:boto:Data: 2023-11-02T18:04:43.369 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:43.370 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.370 DEBUG:boto:Params: {} 2023-11-02T18:04:43.371 DEBUG:boto:Token: None 2023-11-02T18:04:43.371 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-23/ 2023-11-02T18:04:43.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w5Ei3iEKTPZVbIzCoeutcuU39B8= 2023-11-02T18:04:43.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w5Ei3iEKTPZVbIzCoeutcuU39B8=', 'Content-Length': '0'} 2023-11-02T18:04:43.399 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000488dd4bf0cde8d3a-006543e4bb-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.400 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-24 2023-11-02T18:04:43.400 DEBUG:boto:path=/ibtljn-24/ 2023-11-02T18:04:43.401 DEBUG:boto:auth_path=/ibtljn-24/ 2023-11-02T18:04:43.401 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.402 DEBUG:boto:Path: /ibtljn-24/ 2023-11-02T18:04:43.402 DEBUG:boto:Data: 2023-11-02T18:04:43.402 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:43.403 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.404 DEBUG:boto:Params: {} 2023-11-02T18:04:43.404 DEBUG:boto:Token: None 2023-11-02T18:04:43.405 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-24/ 2023-11-02T18:04:43.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cd6aLjdt1mTS5nqnWr9rs6KXA0o= 2023-11-02T18:04:43.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cd6aLjdt1mTS5nqnWr9rs6KXA0o=', 'Content-Length': '0'} 2023-11-02T18:04:43.455 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008097da6ae0abf1ba-006543e4bb-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.456 INFO:rgw_multi.tests:create bucket zone=b1 name=ibtljn-25 2023-11-02T18:04:43.456 DEBUG:boto:path=/ibtljn-25/ 2023-11-02T18:04:43.457 DEBUG:boto:auth_path=/ibtljn-25/ 2023-11-02T18:04:43.457 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.458 DEBUG:boto:Path: /ibtljn-25/ 2023-11-02T18:04:43.458 DEBUG:boto:Data: 2023-11-02T18:04:43.458 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.459 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:04:43.459 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.460 DEBUG:boto:Params: {} 2023-11-02T18:04:43.460 DEBUG:boto:Token: None 2023-11-02T18:04:43.461 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-25/ 2023-11-02T18:04:43.461 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n0IpyOVsQd13MyBYYyNWgQDFLPI= 2023-11-02T18:04:43.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n0IpyOVsQd13MyBYYyNWgQDFLPI=', 'Content-Length': '0'} 2023-11-02T18:04:43.507 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b3b46a4ac48de033-006543e4bb-1107-b1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.507 INFO:rgw_multi.tests:create bucket zone=b2 name=ibtljn-26 2023-11-02T18:04:43.508 DEBUG:boto:path=/ibtljn-26/ 2023-11-02T18:04:43.508 DEBUG:boto:auth_path=/ibtljn-26/ 2023-11-02T18:04:43.509 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.509 DEBUG:boto:Path: /ibtljn-26/ 2023-11-02T18:04:43.510 DEBUG:boto:Data: 2023-11-02T18:04:43.510 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.511 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:04:43.511 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.512 DEBUG:boto:Params: {} 2023-11-02T18:04:43.512 DEBUG:boto:Token: None 2023-11-02T18:04:43.513 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-26/ 2023-11-02T18:04:43.514 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bFbqLwvxJocyspCXirf36UiKNME= 2023-11-02T18:04:43.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bFbqLwvxJocyspCXirf36UiKNME=', 'Content-Length': '0'} 2023-11-02T18:04:43.558 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000099fb57bd208fbec4-006543e4bb-110d-b2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.558 DEBUG:boto:path=/ibtljn-23/ 2023-11-02T18:04:43.559 DEBUG:boto:auth_path=/ibtljn-23/ 2023-11-02T18:04:43.559 DEBUG:boto:path=/ibtljn-23/?acl 2023-11-02T18:04:43.560 DEBUG:boto:auth_path=/ibtljn-23/?acl 2023-11-02T18:04:43.560 DEBUG:boto:Method: GET 2023-11-02T18:04:43.561 DEBUG:boto:Path: /ibtljn-23/?acl 2023-11-02T18:04:43.561 DEBUG:boto:Data: 2023-11-02T18:04:43.561 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.562 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:43.562 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.562 DEBUG:boto:Params: {} 2023-11-02T18:04:43.563 DEBUG:boto:Token: None 2023-11-02T18:04:43.563 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-23/?acl 2023-11-02T18:04:43.564 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5gq0AiJR/FGTMbfwPkFfV+ruhrQ= 2023-11-02T18:04:43.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5gq0AiJR/FGTMbfwPkFfV+ruhrQ=', 'Content-Length': '0'} 2023-11-02T18:04:43.566 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000081ed424c77845ec7-006543e4bb-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.567 DEBUG:boto:path=/ibtljn-23/ 2023-11-02T18:04:43.567 DEBUG:boto:auth_path=/ibtljn-23/ 2023-11-02T18:04:43.568 DEBUG:boto:path=/ibtljn-23/?acl 2023-11-02T18:04:43.568 DEBUG:boto:auth_path=/ibtljn-23/?acl 2023-11-02T18:04:43.568 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.569 DEBUG:boto:Path: /ibtljn-23/?acl 2023-11-02T18:04:43.569 DEBUG:boto:Data: 2023-11-02T18:04:43.570 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-11-02T18:04:43.570 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:43.570 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.571 DEBUG:boto:Params: {} 2023-11-02T18:04:43.571 DEBUG:boto:Token: None 2023-11-02T18:04:43.572 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT x-amz-acl:public-read /ibtljn-23/?acl 2023-11-02T18:04:43.572 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NFe+q4TTDFeyDQLnQ4OFrvmobn8= 2023-11-02T18:04:43.573 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NFe+q4TTDFeyDQLnQ4OFrvmobn8=', 'Content-Length': '0'} 2023-11-02T18:04:43.585 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000010442e8d97882fbc-006543e4bb-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.586 DEBUG:boto:path=/ibtljn-23/ 2023-11-02T18:04:43.586 DEBUG:boto:auth_path=/ibtljn-23/ 2023-11-02T18:04:43.586 DEBUG:boto:path=/ibtljn-23/?acl 2023-11-02T18:04:43.587 DEBUG:boto:auth_path=/ibtljn-23/?acl 2023-11-02T18:04:43.587 DEBUG:boto:Method: GET 2023-11-02T18:04:43.588 DEBUG:boto:Path: /ibtljn-23/?acl 2023-11-02T18:04:43.588 DEBUG:boto:Data: 2023-11-02T18:04:43.588 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.589 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:43.589 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.589 DEBUG:boto:Params: {} 2023-11-02T18:04:43.590 DEBUG:boto:Token: None 2023-11-02T18:04:43.590 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-23/?acl 2023-11-02T18:04:43.591 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5gq0AiJR/FGTMbfwPkFfV+ruhrQ= 2023-11-02T18:04:43.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5gq0AiJR/FGTMbfwPkFfV+ruhrQ=', 'Content-Length': '0'} 2023-11-02T18:04:43.593 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000976806bf7f8858b3-006543e4bb-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.639 DEBUG:boto:path=/ibtljn-24/ 2023-11-02T18:04:43.639 DEBUG:boto:auth_path=/ibtljn-24/ 2023-11-02T18:04:43.640 DEBUG:boto:path=/ibtljn-24/?acl 2023-11-02T18:04:43.640 DEBUG:boto:auth_path=/ibtljn-24/?acl 2023-11-02T18:04:43.640 DEBUG:boto:Method: GET 2023-11-02T18:04:43.641 DEBUG:boto:Path: /ibtljn-24/?acl 2023-11-02T18:04:43.641 DEBUG:boto:Data: 2023-11-02T18:04:43.641 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.642 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:43.642 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.643 DEBUG:boto:Params: {} 2023-11-02T18:04:43.643 DEBUG:boto:Token: None 2023-11-02T18:04:43.644 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-24/?acl 2023-11-02T18:04:43.644 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:r6FypovuvJbI9cM4U+YoGXcNU6k= 2023-11-02T18:04:43.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:r6FypovuvJbI9cM4U+YoGXcNU6k=', 'Content-Length': '0'} 2023-11-02T18:04:43.647 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b3ed6dee81e59830-006543e4bb-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.647 DEBUG:boto:path=/ibtljn-24/ 2023-11-02T18:04:43.648 DEBUG:boto:auth_path=/ibtljn-24/ 2023-11-02T18:04:43.648 DEBUG:boto:path=/ibtljn-24/?acl 2023-11-02T18:04:43.649 DEBUG:boto:auth_path=/ibtljn-24/?acl 2023-11-02T18:04:43.649 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.650 DEBUG:boto:Path: /ibtljn-24/?acl 2023-11-02T18:04:43.650 DEBUG:boto:Data: 2023-11-02T18:04:43.650 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-11-02T18:04:43.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:43.651 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.652 DEBUG:boto:Params: {} 2023-11-02T18:04:43.652 DEBUG:boto:Token: None 2023-11-02T18:04:43.653 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT x-amz-acl:public-read /ibtljn-24/?acl 2023-11-02T18:04:43.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ljTagHDKkdu+AVT4z//8krW+KyI= 2023-11-02T18:04:43.653 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ljTagHDKkdu+AVT4z//8krW+KyI=', 'Content-Length': '0'} 2023-11-02T18:04:43.674 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dabf84c1bc4f16f2-006543e4bb-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.675 DEBUG:boto:path=/ibtljn-24/ 2023-11-02T18:04:43.675 DEBUG:boto:auth_path=/ibtljn-24/ 2023-11-02T18:04:43.676 DEBUG:boto:path=/ibtljn-24/?acl 2023-11-02T18:04:43.676 DEBUG:boto:auth_path=/ibtljn-24/?acl 2023-11-02T18:04:43.677 DEBUG:boto:Method: GET 2023-11-02T18:04:43.677 DEBUG:boto:Path: /ibtljn-24/?acl 2023-11-02T18:04:43.677 DEBUG:boto:Data: 2023-11-02T18:04:43.678 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.678 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:43.679 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.679 DEBUG:boto:Params: {} 2023-11-02T18:04:43.680 DEBUG:boto:Token: None 2023-11-02T18:04:43.680 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-24/?acl 2023-11-02T18:04:43.681 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:r6FypovuvJbI9cM4U+YoGXcNU6k= 2023-11-02T18:04:43.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:r6FypovuvJbI9cM4U+YoGXcNU6k=', 'Content-Length': '0'} 2023-11-02T18:04:43.683 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff32791b1893ec5a-006543e4bb-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.726 DEBUG:boto:path=/ibtljn-25/ 2023-11-02T18:04:43.727 DEBUG:boto:auth_path=/ibtljn-25/ 2023-11-02T18:04:43.727 DEBUG:boto:path=/ibtljn-25/?acl 2023-11-02T18:04:43.728 DEBUG:boto:auth_path=/ibtljn-25/?acl 2023-11-02T18:04:43.728 DEBUG:boto:Method: GET 2023-11-02T18:04:43.729 DEBUG:boto:Path: /ibtljn-25/?acl 2023-11-02T18:04:43.729 DEBUG:boto:Data: 2023-11-02T18:04:43.729 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.730 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:04:43.730 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.731 DEBUG:boto:Params: {} 2023-11-02T18:04:43.731 DEBUG:boto:Token: None 2023-11-02T18:04:43.732 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-25/?acl 2023-11-02T18:04:43.732 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9xSoJkwWmWFiBqI5ez9nBY/O1JU= 2023-11-02T18:04:43.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9xSoJkwWmWFiBqI5ez9nBY/O1JU=', 'Content-Length': '0'} 2023-11-02T18:04:43.734 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b91bb06a2d4a4959-006543e4bb-1107-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.735 DEBUG:boto:path=/ibtljn-25/ 2023-11-02T18:04:43.735 DEBUG:boto:auth_path=/ibtljn-25/ 2023-11-02T18:04:43.736 DEBUG:boto:path=/ibtljn-25/?acl 2023-11-02T18:04:43.736 DEBUG:boto:auth_path=/ibtljn-25/?acl 2023-11-02T18:04:43.737 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.737 DEBUG:boto:Path: /ibtljn-25/?acl 2023-11-02T18:04:43.738 DEBUG:boto:Data: 2023-11-02T18:04:43.738 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-11-02T18:04:43.738 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:04:43.739 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.739 DEBUG:boto:Params: {} 2023-11-02T18:04:43.740 DEBUG:boto:Token: None 2023-11-02T18:04:43.740 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT x-amz-acl:public-read /ibtljn-25/?acl 2023-11-02T18:04:43.741 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MQ65KU/ynFn+45cDoRP/H9LoREs= 2023-11-02T18:04:43.741 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MQ65KU/ynFn+45cDoRP/H9LoREs=', 'Content-Length': '0'} 2023-11-02T18:04:43.761 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a7b9afcc00a4ee33-006543e4bb-1107-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.762 DEBUG:boto:path=/ibtljn-25/ 2023-11-02T18:04:43.762 DEBUG:boto:auth_path=/ibtljn-25/ 2023-11-02T18:04:43.762 DEBUG:boto:path=/ibtljn-25/?acl 2023-11-02T18:04:43.763 DEBUG:boto:auth_path=/ibtljn-25/?acl 2023-11-02T18:04:43.763 DEBUG:boto:Method: GET 2023-11-02T18:04:43.764 DEBUG:boto:Path: /ibtljn-25/?acl 2023-11-02T18:04:43.764 DEBUG:boto:Data: 2023-11-02T18:04:43.765 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.765 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:04:43.766 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.766 DEBUG:boto:Params: {} 2023-11-02T18:04:43.766 DEBUG:boto:Token: None 2023-11-02T18:04:43.767 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-25/?acl 2023-11-02T18:04:43.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9xSoJkwWmWFiBqI5ez9nBY/O1JU= 2023-11-02T18:04:43.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9xSoJkwWmWFiBqI5ez9nBY/O1JU=', 'Content-Length': '0'} 2023-11-02T18:04:43.770 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000563fd6a9194de6ea-006543e4bb-1107-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.810 DEBUG:boto:path=/ibtljn-26/ 2023-11-02T18:04:43.811 DEBUG:boto:auth_path=/ibtljn-26/ 2023-11-02T18:04:43.811 DEBUG:boto:path=/ibtljn-26/?acl 2023-11-02T18:04:43.812 DEBUG:boto:auth_path=/ibtljn-26/?acl 2023-11-02T18:04:43.812 DEBUG:boto:Method: GET 2023-11-02T18:04:43.813 DEBUG:boto:Path: /ibtljn-26/?acl 2023-11-02T18:04:43.813 DEBUG:boto:Data: 2023-11-02T18:04:43.814 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.814 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:04:43.814 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.815 DEBUG:boto:Params: {} 2023-11-02T18:04:43.815 DEBUG:boto:Token: None 2023-11-02T18:04:43.816 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-26/?acl 2023-11-02T18:04:43.816 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KZkruVXW7jUkrRpobboVrwL1P2I= 2023-11-02T18:04:43.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KZkruVXW7jUkrRpobboVrwL1P2I=', 'Content-Length': '0'} 2023-11-02T18:04:43.819 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f914c8db22ce3394-006543e4bb-110d-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.819 DEBUG:boto:path=/ibtljn-26/ 2023-11-02T18:04:43.820 DEBUG:boto:auth_path=/ibtljn-26/ 2023-11-02T18:04:43.820 DEBUG:boto:path=/ibtljn-26/?acl 2023-11-02T18:04:43.821 DEBUG:boto:auth_path=/ibtljn-26/?acl 2023-11-02T18:04:43.821 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.821 DEBUG:boto:Path: /ibtljn-26/?acl 2023-11-02T18:04:43.822 DEBUG:boto:Data: 2023-11-02T18:04:43.822 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-11-02T18:04:43.823 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:04:43.823 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.823 DEBUG:boto:Params: {} 2023-11-02T18:04:43.824 DEBUG:boto:Token: None 2023-11-02T18:04:43.824 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT x-amz-acl:public-read /ibtljn-26/?acl 2023-11-02T18:04:43.825 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zf2SI+7Lx60nVygKn6afJnxIjeg= 2023-11-02T18:04:43.825 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zf2SI+7Lx60nVygKn6afJnxIjeg=', 'Content-Length': '0'} 2023-11-02T18:04:43.844 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000041927a95e9133991-006543e4bb-110d-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.845 DEBUG:boto:path=/ibtljn-26/ 2023-11-02T18:04:43.845 DEBUG:boto:auth_path=/ibtljn-26/ 2023-11-02T18:04:43.846 DEBUG:boto:path=/ibtljn-26/?acl 2023-11-02T18:04:43.846 DEBUG:boto:auth_path=/ibtljn-26/?acl 2023-11-02T18:04:43.847 DEBUG:boto:Method: GET 2023-11-02T18:04:43.847 DEBUG:boto:Path: /ibtljn-26/?acl 2023-11-02T18:04:43.847 DEBUG:boto:Data: 2023-11-02T18:04:43.848 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.848 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:04:43.849 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.849 DEBUG:boto:Params: {} 2023-11-02T18:04:43.850 DEBUG:boto:Token: None 2023-11-02T18:04:43.850 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-26/?acl 2023-11-02T18:04:43.851 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KZkruVXW7jUkrRpobboVrwL1P2I= 2023-11-02T18:04:43.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KZkruVXW7jUkrRpobboVrwL1P2I=', 'Content-Length': '0'} 2023-11-02T18:04:43.853 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e26fba72a0b836ff-006543e4bb-110d-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2023-11-02T18:04:43.896 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-27 2023-11-02T18:04:43.896 DEBUG:boto:path=/ibtljn-27/ 2023-11-02T18:04:43.896 DEBUG:boto:auth_path=/ibtljn-27/ 2023-11-02T18:04:43.897 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.897 DEBUG:boto:Path: /ibtljn-27/ 2023-11-02T18:04:43.898 DEBUG:boto:Data: 2023-11-02T18:04:43.898 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.899 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:43.899 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.899 DEBUG:boto:Params: {} 2023-11-02T18:04:43.900 DEBUG:boto:Token: None 2023-11-02T18:04:43.900 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-27/ 2023-11-02T18:04:43.901 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:thMw5H203YfFztz0m5vmkAi2dqo= 2023-11-02T18:04:43.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:thMw5H203YfFztz0m5vmkAi2dqo=', 'Content-Length': '0'} 2023-11-02T18:04:43.929 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007c84872acf7bdb65-006543e4bb-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.930 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-28 2023-11-02T18:04:43.930 DEBUG:boto:path=/ibtljn-28/ 2023-11-02T18:04:43.931 DEBUG:boto:auth_path=/ibtljn-28/ 2023-11-02T18:04:43.931 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.931 DEBUG:boto:Path: /ibtljn-28/ 2023-11-02T18:04:43.932 DEBUG:boto:Data: 2023-11-02T18:04:43.932 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.933 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:43.933 DEBUG:boto:Port: 8001 2023-11-02T18:04:43.933 DEBUG:boto:Params: {} 2023-11-02T18:04:43.934 DEBUG:boto:Token: None 2023-11-02T18:04:43.934 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-28/ 2023-11-02T18:04:43.935 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ypz0/Ik0yNEy3QcQhaW/KWAtPmc= 2023-11-02T18:04:43.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ypz0/Ik0yNEy3QcQhaW/KWAtPmc=', 'Content-Length': '0'} 2023-11-02T18:04:43.975 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005929ff2243b3c987-006543e4bb-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:43.976 INFO:rgw_multi.tests:create bucket zone=b1 name=ibtljn-29 2023-11-02T18:04:43.977 DEBUG:boto:path=/ibtljn-29/ 2023-11-02T18:04:43.977 DEBUG:boto:auth_path=/ibtljn-29/ 2023-11-02T18:04:43.977 DEBUG:boto:Method: PUT 2023-11-02T18:04:43.978 DEBUG:boto:Path: /ibtljn-29/ 2023-11-02T18:04:43.978 DEBUG:boto:Data: 2023-11-02T18:04:43.979 DEBUG:boto:Headers: {} 2023-11-02T18:04:43.979 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:04:43.979 DEBUG:boto:Port: 8000 2023-11-02T18:04:43.980 DEBUG:boto:Params: {} 2023-11-02T18:04:43.980 DEBUG:boto:Token: None 2023-11-02T18:04:43.981 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:43 GMT /ibtljn-29/ 2023-11-02T18:04:43.981 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zch43vv2XR1v1+oRZTbLGnbAROA= 2023-11-02T18:04:43.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zch43vv2XR1v1+oRZTbLGnbAROA=', 'Content-Length': '0'} 2023-11-02T18:04:44.031 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d0d52e67f46208c2-006543e4bb-1107-b1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:44.031 INFO:rgw_multi.tests:create bucket zone=b2 name=ibtljn-30 2023-11-02T18:04:44.032 DEBUG:boto:path=/ibtljn-30/ 2023-11-02T18:04:44.032 DEBUG:boto:auth_path=/ibtljn-30/ 2023-11-02T18:04:44.033 DEBUG:boto:Method: PUT 2023-11-02T18:04:44.033 DEBUG:boto:Path: /ibtljn-30/ 2023-11-02T18:04:44.034 DEBUG:boto:Data: 2023-11-02T18:04:44.034 DEBUG:boto:Headers: {} 2023-11-02T18:04:44.034 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:04:44.035 DEBUG:boto:Port: 8001 2023-11-02T18:04:44.035 DEBUG:boto:Params: {} 2023-11-02T18:04:44.036 DEBUG:boto:Token: None 2023-11-02T18:04:44.036 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:44 GMT /ibtljn-30/ 2023-11-02T18:04:44.037 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iXaCjFSBEhkA9Ui9i3ikXuVPk/I= 2023-11-02T18:04:44.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iXaCjFSBEhkA9Ui9i3ikXuVPk/I=', 'Content-Length': '0'} 2023-11-02T18:04:44.095 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000077cc3ff225c556b9-006543e4bc-110d-b2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:44.096 DEBUG:boto:path=/ibtljn-27/ 2023-11-02T18:04:44.096 DEBUG:boto:auth_path=/ibtljn-27/ 2023-11-02T18:04:44.096 DEBUG:boto:path=/ibtljn-27/?cors 2023-11-02T18:04:44.097 DEBUG:boto:auth_path=/ibtljn-27/?cors 2023-11-02T18:04:44.097 DEBUG:boto:Method: PUT 2023-11-02T18:04:44.098 DEBUG:boto:Path: /ibtljn-27/?cors 2023-11-02T18:04:44.098 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-11-02T18:04:44.099 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-11-02T18:04:44.099 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:44.100 DEBUG:boto:Port: 8000 2023-11-02T18:04:44.100 DEBUG:boto:Params: {} 2023-11-02T18:04:44.101 DEBUG:boto:Token: None 2023-11-02T18:04:44.101 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Thu, 02 Nov 2023 18:04:44 GMT /ibtljn-27/?cors 2023-11-02T18:04:44.102 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5vF5/16DLgktfVIfVUnpnCmMaVo= 2023-11-02T18:04:44.102 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5vF5/16DLgktfVIfVUnpnCmMaVo=', 'Content-Length': '218'} 2023-11-02T18:04:44.112 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004e78a49ceb1d4378-006543e4bc-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:44.113 DEBUG:boto:path=/ibtljn-27/ 2023-11-02T18:04:44.113 DEBUG:boto:auth_path=/ibtljn-27/ 2023-11-02T18:04:44.113 DEBUG:boto:path=/ibtljn-27/?cors 2023-11-02T18:04:44.114 DEBUG:boto:auth_path=/ibtljn-27/?cors 2023-11-02T18:04:44.114 DEBUG:boto:Method: GET 2023-11-02T18:04:44.115 DEBUG:boto:Path: /ibtljn-27/?cors 2023-11-02T18:04:44.115 DEBUG:boto:Data: 2023-11-02T18:04:44.116 DEBUG:boto:Headers: {} 2023-11-02T18:04:44.116 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:44.116 DEBUG:boto:Port: 8000 2023-11-02T18:04:44.117 DEBUG:boto:Params: {} 2023-11-02T18:04:44.117 DEBUG:boto:Token: None 2023-11-02T18:04:44.118 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:44 GMT /ibtljn-27/?cors 2023-11-02T18:04:44.118 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C638hgCS63Qi9B7QJV12MTg9G80= 2023-11-02T18:04:44.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C638hgCS63Qi9B7QJV12MTg9G80=', 'Content-Length': '0'} 2023-11-02T18:04:44.121 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aa74d9a69129d9a9-006543e4bc-1111-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Thu, 02 Nov 2023 18:04:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:44.121 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-11-02T18:04:44.122 DEBUG:boto:path=/ibtljn-28/ 2023-11-02T18:04:44.122 DEBUG:boto:auth_path=/ibtljn-28/ 2023-11-02T18:04:44.123 DEBUG:boto:path=/ibtljn-28/?cors 2023-11-02T18:04:44.123 DEBUG:boto:auth_path=/ibtljn-28/?cors 2023-11-02T18:04:44.124 DEBUG:boto:Method: PUT 2023-11-02T18:04:44.124 DEBUG:boto:Path: /ibtljn-28/?cors 2023-11-02T18:04:44.124 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-11-02T18:04:44.125 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-11-02T18:04:44.125 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:44.126 DEBUG:boto:Port: 8001 2023-11-02T18:04:44.126 DEBUG:boto:Params: {} 2023-11-02T18:04:44.127 DEBUG:boto:Token: None 2023-11-02T18:04:44.127 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Thu, 02 Nov 2023 18:04:44 GMT /ibtljn-28/?cors 2023-11-02T18:04:44.128 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NKJHD4MwdOqIvbWvObtPNQOr54s= 2023-11-02T18:04:44.128 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NKJHD4MwdOqIvbWvObtPNQOr54s=', 'Content-Length': '218'} 2023-11-02T18:04:45.148 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f39da209957f7f44-006543e4bc-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:45.148 DEBUG:boto:path=/ibtljn-28/ 2023-11-02T18:04:45.149 DEBUG:boto:auth_path=/ibtljn-28/ 2023-11-02T18:04:45.149 DEBUG:boto:path=/ibtljn-28/?cors 2023-11-02T18:04:45.150 DEBUG:boto:auth_path=/ibtljn-28/?cors 2023-11-02T18:04:45.150 DEBUG:boto:Method: GET 2023-11-02T18:04:45.150 DEBUG:boto:Path: /ibtljn-28/?cors 2023-11-02T18:04:45.151 DEBUG:boto:Data: 2023-11-02T18:04:45.151 DEBUG:boto:Headers: {} 2023-11-02T18:04:45.152 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:45.152 DEBUG:boto:Port: 8001 2023-11-02T18:04:45.153 DEBUG:boto:Params: {} 2023-11-02T18:04:45.153 DEBUG:boto:Token: None 2023-11-02T18:04:45.154 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:45 GMT /ibtljn-28/?cors 2023-11-02T18:04:45.154 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/beDKiwtx0aZ43Rr1hrsgrHr6Bs= 2023-11-02T18:04:45.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/beDKiwtx0aZ43Rr1hrsgrHr6Bs=', 'Content-Length': '0'} 2023-11-02T18:04:45.156 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d261f04aee54877-006543e4bd-1115-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Thu, 02 Nov 2023 18:04:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:45.157 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-11-02T18:04:45.157 DEBUG:boto:path=/ibtljn-29/ 2023-11-02T18:04:45.158 DEBUG:boto:auth_path=/ibtljn-29/ 2023-11-02T18:04:45.158 DEBUG:boto:path=/ibtljn-29/?cors 2023-11-02T18:04:45.159 DEBUG:boto:auth_path=/ibtljn-29/?cors 2023-11-02T18:04:45.159 DEBUG:boto:Method: PUT 2023-11-02T18:04:45.159 DEBUG:boto:Path: /ibtljn-29/?cors 2023-11-02T18:04:45.160 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-11-02T18:04:45.160 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-11-02T18:04:45.160 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:04:45.161 DEBUG:boto:Port: 8000 2023-11-02T18:04:45.161 DEBUG:boto:Params: {} 2023-11-02T18:04:45.162 DEBUG:boto:Token: None 2023-11-02T18:04:45.162 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Thu, 02 Nov 2023 18:04:45 GMT /ibtljn-29/?cors 2023-11-02T18:04:45.163 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RTzAN6dEk/7mvrapjGJzPMo9NwY= 2023-11-02T18:04:45.163 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RTzAN6dEk/7mvrapjGJzPMo9NwY=', 'Content-Length': '218'} 2023-11-02T18:04:46.183 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000969b24f2ecc36d44-006543e4bd-1107-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:46.184 DEBUG:boto:path=/ibtljn-29/ 2023-11-02T18:04:46.185 DEBUG:boto:auth_path=/ibtljn-29/ 2023-11-02T18:04:46.185 DEBUG:boto:path=/ibtljn-29/?cors 2023-11-02T18:04:46.185 DEBUG:boto:auth_path=/ibtljn-29/?cors 2023-11-02T18:04:46.186 DEBUG:boto:Method: GET 2023-11-02T18:04:46.186 DEBUG:boto:Path: /ibtljn-29/?cors 2023-11-02T18:04:46.187 DEBUG:boto:Data: 2023-11-02T18:04:46.187 DEBUG:boto:Headers: {} 2023-11-02T18:04:46.188 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:04:46.188 DEBUG:boto:Port: 8000 2023-11-02T18:04:46.188 DEBUG:boto:Params: {} 2023-11-02T18:04:46.189 DEBUG:boto:Token: None 2023-11-02T18:04:46.189 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:46 GMT /ibtljn-29/?cors 2023-11-02T18:04:46.190 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rW3hiZjGmQUS7jIwtkH2xFvvK0k= 2023-11-02T18:04:46.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rW3hiZjGmQUS7jIwtkH2xFvvK0k=', 'Content-Length': '0'} 2023-11-02T18:04:46.192 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009b69a5777c3865ff-006543e4be-1107-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Thu, 02 Nov 2023 18:04:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:46.192 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-11-02T18:04:46.193 DEBUG:boto:path=/ibtljn-30/ 2023-11-02T18:04:46.194 DEBUG:boto:auth_path=/ibtljn-30/ 2023-11-02T18:04:46.194 DEBUG:boto:path=/ibtljn-30/?cors 2023-11-02T18:04:46.194 DEBUG:boto:auth_path=/ibtljn-30/?cors 2023-11-02T18:04:46.195 DEBUG:boto:Method: PUT 2023-11-02T18:04:46.195 DEBUG:boto:Path: /ibtljn-30/?cors 2023-11-02T18:04:46.196 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-11-02T18:04:46.196 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-11-02T18:04:46.196 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:04:46.197 DEBUG:boto:Port: 8001 2023-11-02T18:04:46.197 DEBUG:boto:Params: {} 2023-11-02T18:04:46.198 DEBUG:boto:Token: None 2023-11-02T18:04:46.198 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Thu, 02 Nov 2023 18:04:46 GMT /ibtljn-30/?cors 2023-11-02T18:04:46.199 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuiWxhiBjl9nk8bMLbiFcBpWsMs= 2023-11-02T18:04:46.199 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuiWxhiBjl9nk8bMLbiFcBpWsMs=', 'Content-Length': '218'} 2023-11-02T18:04:47.222 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000044508a7bf626c883-006543e4be-110d-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.223 DEBUG:boto:path=/ibtljn-30/ 2023-11-02T18:04:47.223 DEBUG:boto:auth_path=/ibtljn-30/ 2023-11-02T18:04:47.224 DEBUG:boto:path=/ibtljn-30/?cors 2023-11-02T18:04:47.224 DEBUG:boto:auth_path=/ibtljn-30/?cors 2023-11-02T18:04:47.224 DEBUG:boto:Method: GET 2023-11-02T18:04:47.225 DEBUG:boto:Path: /ibtljn-30/?cors 2023-11-02T18:04:47.225 DEBUG:boto:Data: 2023-11-02T18:04:47.226 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.226 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:04:47.226 DEBUG:boto:Port: 8001 2023-11-02T18:04:47.227 DEBUG:boto:Params: {} 2023-11-02T18:04:47.227 DEBUG:boto:Token: None 2023-11-02T18:04:47.228 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-30/?cors 2023-11-02T18:04:47.228 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/2JPA1WgBp4Ah0JvrZTjEO9hyYY= 2023-11-02T18:04:47.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/2JPA1WgBp4Ah0JvrZTjEO9hyYY=', 'Content-Length': '0'} 2023-11-02T18:04:47.231 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ebacdc5e94bfd2b1-006543e4bf-110d-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.232 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-11-02T18:04:47.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2023-11-02T18:04:47.233 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-31 2023-11-02T18:04:47.234 DEBUG:boto:path=/ibtljn-31/ 2023-11-02T18:04:47.234 DEBUG:boto:auth_path=/ibtljn-31/ 2023-11-02T18:04:47.235 DEBUG:boto:Method: PUT 2023-11-02T18:04:47.235 DEBUG:boto:Path: /ibtljn-31/ 2023-11-02T18:04:47.235 DEBUG:boto:Data: 2023-11-02T18:04:47.236 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.236 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:47.237 DEBUG:boto:Port: 8000 2023-11-02T18:04:47.237 DEBUG:boto:Params: {} 2023-11-02T18:04:47.237 DEBUG:boto:Token: None 2023-11-02T18:04:47.238 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-31/ 2023-11-02T18:04:47.238 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c2IRi+hXBmNG7gavOcM49D3s2Nw= 2023-11-02T18:04:47.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c2IRi+hXBmNG7gavOcM49D3s2Nw=', 'Content-Length': '0'} 2023-11-02T18:04:47.270 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e4ba2a03e153ac42-006543e4bf-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.271 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-32 2023-11-02T18:04:47.271 DEBUG:boto:path=/ibtljn-32/ 2023-11-02T18:04:47.272 DEBUG:boto:auth_path=/ibtljn-32/ 2023-11-02T18:04:47.272 DEBUG:boto:Method: PUT 2023-11-02T18:04:47.272 DEBUG:boto:Path: /ibtljn-32/ 2023-11-02T18:04:47.273 DEBUG:boto:Data: 2023-11-02T18:04:47.273 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.274 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:47.274 DEBUG:boto:Port: 8001 2023-11-02T18:04:47.275 DEBUG:boto:Params: {} 2023-11-02T18:04:47.275 DEBUG:boto:Token: None 2023-11-02T18:04:47.275 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-32/ 2023-11-02T18:04:47.276 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5ztD7eEXJWk9YFobsMlk1tYam5A= 2023-11-02T18:04:47.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5ztD7eEXJWk9YFobsMlk1tYam5A=', 'Content-Length': '0'} 2023-11-02T18:04:47.315 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d219257519731ad0-006543e4bf-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:47.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:47.317 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:04:47.361 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:04:47.362 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:04:47.417 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:04:47.444 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:04:47.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:47.445 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:04:47.445 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:04:47.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:47.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:47.446 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:04:47.446 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:04:47.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:47.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:47.447 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.292650_197.1", 2023-11-02T18:04:47.447 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:04:47.292650Z" 2023-11-02T18:04:47.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:47.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:47.448 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:04:47.448 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:04:47.448 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:47.449 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:04:47.450 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.292650_197.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:04:47.450 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:04:47.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:47.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:47.452 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:04:47.533 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:04:47.533 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:04:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:47.568+0000 7f4857897f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:04:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:47.572+0000 7f4857897f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:04:47.606 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:04:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:04:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:04:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:04:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:04:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:04:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:04:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:04:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:04:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:04:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:04:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:04:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:04:47.612 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:04:47.612 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:47.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:47.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:47.613 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:04:47.613 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:47.613 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:04:47.613 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:04:47.614 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:47.614 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:47.614 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:47.614 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:04:47.614 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:04:47.615 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:47.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:47.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:47.615 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:04:47.616 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:47.616 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:04:47.616 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.292650_197.1", 2023-11-02T18:04:47.616 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:47.616 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:47.616 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:47.617 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.292650Z", 2023-11-02T18:04:47.617 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:04:47.617 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:47.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:47.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:47.618 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:04:47.618 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:47.618 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:04:47.618 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:04:47.619 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:47.619 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:47.619 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:47.619 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:04:47.620 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:04:47.620 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:47.620 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:47.620 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:04:47.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:47.621 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:04:47.621 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:04:47.621 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:04:47.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:47.622 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:04:47Z" 2023-11-02T18:04:47.622 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:04:47.623 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.292650_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.292650Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:04:47Z" } 2023-11-02T18:04:47.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.292650_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.292650Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:04:47.624 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.292650_197.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:04:47.624 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.292650_197.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:04:47.625 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:04:47.625 DEBUG:boto:path=/ibtljn-31/ 2023-11-02T18:04:47.626 DEBUG:boto:auth_path=/ibtljn-31/ 2023-11-02T18:04:47.626 DEBUG:boto:Method: HEAD 2023-11-02T18:04:47.626 DEBUG:boto:Path: /ibtljn-31/ 2023-11-02T18:04:47.627 DEBUG:boto:Data: 2023-11-02T18:04:47.627 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.628 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:47.628 DEBUG:boto:Port: 8000 2023-11-02T18:04:47.628 DEBUG:boto:Params: {} 2023-11-02T18:04:47.629 DEBUG:boto:Token: None 2023-11-02T18:04:47.629 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-31/ 2023-11-02T18:04:47.630 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qNi848Dl7J8q3/ielYaDN7JlGhY= 2023-11-02T18:04:47.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qNi848Dl7J8q3/ielYaDN7JlGhY=', 'Content-Length': '0'} 2023-11-02T18:04:47.636 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc857a5c0c35e547-006543e4bf-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.637 DEBUG:boto:path=/ibtljn-31/foo 2023-11-02T18:04:47.637 DEBUG:boto:auth_path=/ibtljn-31/foo 2023-11-02T18:04:47.638 DEBUG:boto:Method: PUT 2023-11-02T18:04:47.638 DEBUG:boto:Path: /ibtljn-31/foo 2023-11-02T18:04:47.639 DEBUG:boto:Data: 2023-11-02T18:04:47.639 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:04:47.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:47.640 DEBUG:boto:Port: 8000 2023-11-02T18:04:47.640 DEBUG:boto:Params: {} 2023-11-02T18:04:47.641 DEBUG:boto:Token: None 2023-11-02T18:04:47.641 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-31/foo 2023-11-02T18:04:47.642 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1Df98vxVL69qpgXa727YG5YH8Q= 2023-11-02T18:04:47.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1Df98vxVL69qpgXa727YG5YH8Q='} 2023-11-02T18:04:47.657 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e049e50ea5b93a1-006543e4bf-1111-a1'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.657 DEBUG:boto:path=/ibtljn-31/ 2023-11-02T18:04:47.658 DEBUG:boto:auth_path=/ibtljn-31/ 2023-11-02T18:04:47.658 DEBUG:boto:Method: DELETE 2023-11-02T18:04:47.658 DEBUG:boto:Path: /ibtljn-31/ 2023-11-02T18:04:47.659 DEBUG:boto:Data: 2023-11-02T18:04:47.659 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.660 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:47.660 DEBUG:boto:Port: 8000 2023-11-02T18:04:47.660 DEBUG:boto:Params: {} 2023-11-02T18:04:47.661 DEBUG:boto:Token: None 2023-11-02T18:04:47.661 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-31/ 2023-11-02T18:04:47.662 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vRn9oujBzv8GHObq6s1TrKG7O9w= 2023-11-02T18:04:47.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vRn9oujBzv8GHObq6s1TrKG7O9w=', 'Content-Length': '0'} 2023-11-02T18:04:47.679 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx00000f20dd2db0ae8ceb9-006543e4bf-1111-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.723 DEBUG:boto:path=/ibtljn-32/ 2023-11-02T18:04:47.723 DEBUG:boto:auth_path=/ibtljn-32/ 2023-11-02T18:04:47.724 DEBUG:boto:Method: HEAD 2023-11-02T18:04:47.724 DEBUG:boto:Path: /ibtljn-32/ 2023-11-02T18:04:47.724 DEBUG:boto:Data: 2023-11-02T18:04:47.725 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.725 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:47.726 DEBUG:boto:Port: 8001 2023-11-02T18:04:47.726 DEBUG:boto:Params: {} 2023-11-02T18:04:47.727 DEBUG:boto:Token: None 2023-11-02T18:04:47.727 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-32/ 2023-11-02T18:04:47.727 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DdSR8X2V9k4ZS4lxPl8BH6fjT8Q= 2023-11-02T18:04:47.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DdSR8X2V9k4ZS4lxPl8BH6fjT8Q=', 'Content-Length': '0'} 2023-11-02T18:04:47.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f977030a030e9d0-006543e4bf-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.733 DEBUG:boto:path=/ibtljn-32/foo 2023-11-02T18:04:47.734 DEBUG:boto:auth_path=/ibtljn-32/foo 2023-11-02T18:04:47.734 DEBUG:boto:Method: PUT 2023-11-02T18:04:47.735 DEBUG:boto:Path: /ibtljn-32/foo 2023-11-02T18:04:47.735 DEBUG:boto:Data: 2023-11-02T18:04:47.736 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:04:47.736 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:47.737 DEBUG:boto:Port: 8001 2023-11-02T18:04:47.737 DEBUG:boto:Params: {} 2023-11-02T18:04:47.737 DEBUG:boto:Token: None 2023-11-02T18:04:47.738 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-32/foo 2023-11-02T18:04:47.738 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Wh31yDn9HM5ftkl8wneGW1vlbXs= 2023-11-02T18:04:47.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Wh31yDn9HM5ftkl8wneGW1vlbXs='} 2023-11-02T18:04:47.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000255e06d4eaada232-006543e4bf-1115-a2'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.753 DEBUG:boto:path=/ibtljn-32/ 2023-11-02T18:04:47.754 DEBUG:boto:auth_path=/ibtljn-32/ 2023-11-02T18:04:47.754 DEBUG:boto:Method: DELETE 2023-11-02T18:04:47.754 DEBUG:boto:Path: /ibtljn-32/ 2023-11-02T18:04:47.755 DEBUG:boto:Data: 2023-11-02T18:04:47.755 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.756 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:47.756 DEBUG:boto:Port: 8001 2023-11-02T18:04:47.756 DEBUG:boto:Params: {} 2023-11-02T18:04:47.757 DEBUG:boto:Token: None 2023-11-02T18:04:47.757 DEBUG:boto:StringToSign: DELETE Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-32/ 2023-11-02T18:04:47.758 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:znsSyTuTVhHyjdjh/l6rOPFKlfs= 2023-11-02T18:04:47.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:znsSyTuTVhHyjdjh/l6rOPFKlfs=', 'Content-Length': '0'} 2023-11-02T18:04:47.775 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx000009de400a8d0411959-006543e4bf-1115-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.819 DEBUG:boto:path=/ibtljn-31/ 2023-11-02T18:04:47.820 DEBUG:boto:auth_path=/ibtljn-31/ 2023-11-02T18:04:47.820 DEBUG:boto:Method: HEAD 2023-11-02T18:04:47.821 DEBUG:boto:Path: /ibtljn-31/ 2023-11-02T18:04:47.821 DEBUG:boto:Data: 2023-11-02T18:04:47.822 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.822 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:47.823 DEBUG:boto:Port: 8000 2023-11-02T18:04:47.823 DEBUG:boto:Params: {} 2023-11-02T18:04:47.823 DEBUG:boto:Token: None 2023-11-02T18:04:47.824 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-31/ 2023-11-02T18:04:47.824 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qNi848Dl7J8q3/ielYaDN7JlGhY= 2023-11-02T18:04:47.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qNi848Dl7J8q3/ielYaDN7JlGhY=', 'Content-Length': '0'} 2023-11-02T18:04:47.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d5c62f123563c04-006543e4bf-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.830 DEBUG:boto:path=/ibtljn-32/ 2023-11-02T18:04:47.830 DEBUG:boto:auth_path=/ibtljn-32/ 2023-11-02T18:04:47.831 DEBUG:boto:Method: HEAD 2023-11-02T18:04:47.831 DEBUG:boto:Path: /ibtljn-32/ 2023-11-02T18:04:47.832 DEBUG:boto:Data: 2023-11-02T18:04:47.832 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.832 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:47.833 DEBUG:boto:Port: 8000 2023-11-02T18:04:47.833 DEBUG:boto:Params: {} 2023-11-02T18:04:47.834 DEBUG:boto:Token: None 2023-11-02T18:04:47.834 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-32/ 2023-11-02T18:04:47.835 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DdSR8X2V9k4ZS4lxPl8BH6fjT8Q= 2023-11-02T18:04:47.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DdSR8X2V9k4ZS4lxPl8BH6fjT8Q=', 'Content-Length': '0'} 2023-11-02T18:04:47.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b0677a5bf18f5c4-006543e4bf-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2023-11-02T18:04:47.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2023-11-02T18:04:47.842 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-33 2023-11-02T18:04:47.842 DEBUG:boto:path=/ibtljn-33/ 2023-11-02T18:04:47.843 DEBUG:boto:auth_path=/ibtljn-33/ 2023-11-02T18:04:47.843 DEBUG:boto:Method: PUT 2023-11-02T18:04:47.844 DEBUG:boto:Path: /ibtljn-33/ 2023-11-02T18:04:47.844 DEBUG:boto:Data: 2023-11-02T18:04:47.844 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.845 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:47.845 DEBUG:boto:Port: 8000 2023-11-02T18:04:47.846 DEBUG:boto:Params: {} 2023-11-02T18:04:47.846 DEBUG:boto:Token: None 2023-11-02T18:04:47.847 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-33/ 2023-11-02T18:04:47.847 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tj0fVdoLxRkfPTFd1fthZTvF8bU= 2023-11-02T18:04:47.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tj0fVdoLxRkfPTFd1fthZTvF8bU=', 'Content-Length': '0'} 2023-11-02T18:04:47.875 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001424d40dab288ab6-006543e4bf-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.876 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-34 2023-11-02T18:04:47.876 DEBUG:boto:path=/ibtljn-34/ 2023-11-02T18:04:47.877 DEBUG:boto:auth_path=/ibtljn-34/ 2023-11-02T18:04:47.877 DEBUG:boto:Method: PUT 2023-11-02T18:04:47.878 DEBUG:boto:Path: /ibtljn-34/ 2023-11-02T18:04:47.878 DEBUG:boto:Data: 2023-11-02T18:04:47.879 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.879 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:47.880 DEBUG:boto:Port: 8001 2023-11-02T18:04:47.880 DEBUG:boto:Params: {} 2023-11-02T18:04:47.880 DEBUG:boto:Token: None 2023-11-02T18:04:47.881 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-34/ 2023-11-02T18:04:47.881 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ClwihzXy8IcYpfI8A9PTd/x4ik= 2023-11-02T18:04:47.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ClwihzXy8IcYpfI8A9PTd/x4ik=', 'Content-Length': '0'} 2023-11-02T18:04:47.926 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f54914ecccd9213e-006543e4bf-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.927 DEBUG:boto:path=/ibtljn-33/ 2023-11-02T18:04:47.927 DEBUG:boto:auth_path=/ibtljn-33/ 2023-11-02T18:04:47.928 DEBUG:boto:Method: HEAD 2023-11-02T18:04:47.928 DEBUG:boto:Path: /ibtljn-33/ 2023-11-02T18:04:47.929 DEBUG:boto:Data: 2023-11-02T18:04:47.929 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:47.930 DEBUG:boto:Port: 8000 2023-11-02T18:04:47.931 DEBUG:boto:Params: {} 2023-11-02T18:04:47.931 DEBUG:boto:Token: None 2023-11-02T18:04:47.932 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-33/ 2023-11-02T18:04:47.932 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RAyZ0yimJBQDXctps3JoCHJe4Ts= 2023-11-02T18:04:47.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RAyZ0yimJBQDXctps3JoCHJe4Ts=', 'Content-Length': '0'} 2023-11-02T18:04:47.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fa5adf2ede909f2-006543e4bf-1111-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.937 DEBUG:boto:path=/ibtljn-33/key 2023-11-02T18:04:47.938 DEBUG:boto:auth_path=/ibtljn-33/key 2023-11-02T18:04:47.938 DEBUG:boto:Method: PUT 2023-11-02T18:04:47.938 DEBUG:boto:Path: /ibtljn-33/key 2023-11-02T18:04:47.939 DEBUG:boto:Data: 2023-11-02T18:04:47.939 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-11-02T18:04:47.940 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:04:47.940 DEBUG:boto:Port: 8000 2023-11-02T18:04:47.941 DEBUG:boto:Params: {} 2023-11-02T18:04:47.941 DEBUG:boto:Token: None 2023-11-02T18:04:47.942 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-33/key 2023-11-02T18:04:47.942 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CeUo4pDqJlwdUhfiruHLvphiEWw= 2023-11-02T18:04:47.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CeUo4pDqJlwdUhfiruHLvphiEWw='} 2023-11-02T18:04:47.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7b1928d77558f03-006543e4bf-1111-a1'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:47.988 DEBUG:boto:path=/ibtljn-34/ 2023-11-02T18:04:47.989 DEBUG:boto:auth_path=/ibtljn-34/ 2023-11-02T18:04:47.989 DEBUG:boto:Method: HEAD 2023-11-02T18:04:47.989 DEBUG:boto:Path: /ibtljn-34/ 2023-11-02T18:04:47.991 DEBUG:boto:Data: 2023-11-02T18:04:47.992 DEBUG:boto:Headers: {} 2023-11-02T18:04:47.992 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:47.993 DEBUG:boto:Port: 8001 2023-11-02T18:04:47.993 DEBUG:boto:Params: {} 2023-11-02T18:04:47.993 DEBUG:boto:Token: None 2023-11-02T18:04:47.994 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:04:47 GMT /ibtljn-34/ 2023-11-02T18:04:47.994 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wBz7ndzhG57CoKt3tm62s/fKIhI= 2023-11-02T18:04:47.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:04:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wBz7ndzhG57CoKt3tm62s/fKIhI=', 'Content-Length': '0'} 2023-11-02T18:04:47.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000271b8b58e6457e23-006543e4bf-1115-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:04:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:48.000 DEBUG:boto:path=/ibtljn-34/key 2023-11-02T18:04:48.001 DEBUG:boto:auth_path=/ibtljn-34/key 2023-11-02T18:04:48.001 DEBUG:boto:Method: PUT 2023-11-02T18:04:48.002 DEBUG:boto:Path: /ibtljn-34/key 2023-11-02T18:04:48.002 DEBUG:boto:Data: 2023-11-02T18:04:48.003 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-11-02T18:04:48.003 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:04:48.004 DEBUG:boto:Port: 8001 2023-11-02T18:04:48.005 DEBUG:boto:Params: {} 2023-11-02T18:04:48.006 DEBUG:boto:Token: None 2023-11-02T18:04:48.006 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Thu, 02 Nov 2023 18:04:48 GMT /ibtljn-34/key 2023-11-02T18:04:48.007 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3yDHVdf8Fik+ogTVRJE5IcIJkmU= 2023-11-02T18:04:48.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:04:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3yDHVdf8Fik+ogTVRJE5IcIJkmU='} 2023-11-02T18:04:48.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab7835d6944f1093-006543e4bf-1115-a2'), ('Date', 'Thu, 02 Nov 2023 18:04:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:04:48.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:48.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:48.024 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:04:48.071 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:04:48.071 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:04:48.122 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:04:48.148 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:04:48.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.149 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:04:48.149 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:04:48.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.150 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:04:48.150 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:04:48.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.151 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:04:48.151 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:04:47.903346Z" 2023-11-02T18:04:48.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.152 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:04:48.152 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:04:48.152 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:48.152 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:04:48.153 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:04:48.154 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:04:48.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:48.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:48.155 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:04:48.200 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:04:48.200 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:04:48.255 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:48.244+0000 7fb03aa31f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:04:48.257 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:04:48.244+0000 7fb03aa31f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:04:48.281 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:04:48.281 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:04:48.282 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:04:48.282 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:04:48.282 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:04:48.282 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:04:48.283 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:04:48.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.286 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:04:48.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.286 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:04:48.286 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:48.287 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:04:48.287 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:04:48.287 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:48.287 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:04:48.287 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:48.288 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:04:48.288 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:04:48.288 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:48.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.289 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:04:48.289 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:48.289 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:04:48.289 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:04:48.290 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:48.290 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:48.290 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:48.290 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:04:48.290 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:04:48.291 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:48.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.291 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:04:48.291 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:48.292 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:04:48.292 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:04:48.292 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:48.292 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:48.292 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:48.293 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:04:48.293 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:04:48.293 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:48.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.294 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:04:48.294 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:48.294 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:04:48.294 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:04:48.295 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:48.295 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:48.296 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:48.296 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:04:48.296 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:04:48.297 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:48.297 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:48.297 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:04:48.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.297 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:04:48.298 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:04:48.298 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:04:48.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.298 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:04:48Z" 2023-11-02T18:04:48.298 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:04:48.300 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:04:48Z" } 2023-11-02T18:04:48.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:04:48.301 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:04:48.301 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:04:48.301 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:04:48.302 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-11-02T18:04:48.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:48.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:48.303 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:04:48.349 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:04:48.349 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:04:48.446 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:04:48.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.447 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:04:48.447 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:00.706996Z" 2023-11-02T18:04:48.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.448 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-11-02T18:04:48.448 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:04:47.947886Z" 2023-11-02T18:04:48.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.449 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-11-02T18:04:48.449 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:59:11.495487Z" 2023-11-02T18:04:48.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.450 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-11-02T18:04:48.450 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:04:47.648707Z" 2023-11-02T18:04:48.450 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:48.450 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:04:48.452 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001068'} 2023-11-02T18:04:48.452 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:04:48.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:48.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:48.454 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:04:48.501 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:04:48.502 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:04:48.589 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:04:48.590 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:04:48.590 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:04:48.590 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:04:48.590 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:04:48.591 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 14562503763241451158 2023-11-02T18:04:48.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.591 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:04:48.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.591 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:04:48.592 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:48.592 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:04:48.592 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:04:48.592 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:48.592 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:48.592 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:48.593 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:00.706996Z" 2023-11-02T18:04:48.593 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:48.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.593 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:04:48.594 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:48.594 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:04:48.594 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000003183", 2023-11-02T18:04:48.594 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:03:41.866421Z" 2023-11-02T18:04:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:04:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:04:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-11-02T18:04:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:59:11.495487Z" 2023-11-02T18:04:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:04:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:04:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-11-02T18:04:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:52:00.720077Z" 2023-11-02T18:04:48.600 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:48.600 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:48.600 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:04:48.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.601 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:04:48.601 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:04:48.601 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:04:48.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:48.602 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:04:48Z" 2023-11-02T18:04:48.602 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:04:48.603 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14562503763241451158 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:00.706996Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003183", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:03:41.866421Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:59:11.495487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:52:00.720077Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:04:48Z" } 2023-11-02T18:04:48.604 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:00.706996Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003183', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:03:41.866421Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:59:11.495487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:52:00.720077Z'}}] 2023-11-02T18:04:48.604 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001068'} 2023-11-02T18:04:48.604 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003183', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-11-02T18:04:48.605 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000003324 target=00000000000000000000:00000000000000003183, shard=3 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929 2023-11-02T18:04:53.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:53.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:53.607 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:04:53.656 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:04:53.656 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:04:53.739 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:04:53.739 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:04:53.739 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:04:53.740 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:04:53.740 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:04:53.740 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 14562503763241451158 2023-11-02T18:04:53.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:53.741 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:04:53.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:53.741 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:04:53.741 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:53.742 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:04:53.742 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:04:53.742 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:53.742 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:53.743 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:53.743 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:00.706996Z" 2023-11-02T18:04:53.743 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:53.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:53.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:53.744 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:04:53.744 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:53.744 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:04:53.744 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000003183", 2023-11-02T18:04:53.745 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:53.745 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:53.745 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:53.745 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:03:41.866421Z" 2023-11-02T18:04:53.746 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:53.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:53.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:53.746 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:04:53.747 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:53.747 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:04:53.747 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-11-02T18:04:53.747 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:53.747 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:53.748 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:53.748 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:59:11.495487Z" 2023-11-02T18:04:53.748 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:53.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:53.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:53.749 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:04:53.749 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:53.749 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:04:53.750 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-11-02T18:04:53.750 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:53.750 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:53.750 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:53.750 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:52:00.720077Z" 2023-11-02T18:04:53.751 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:53.751 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:53.751 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:04:53.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:53.752 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:04:53.752 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:04:53.752 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:04:53.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:53.753 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:04:53Z" 2023-11-02T18:04:53.753 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:04:53.754 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14562503763241451158 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:00.706996Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003183", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:03:41.866421Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:59:11.495487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:52:00.720077Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:04:53Z" } 2023-11-02T18:04:53.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:00.706996Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003183', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:03:41.866421Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:59:11.495487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:52:00.720077Z'}}] 2023-11-02T18:04:53.755 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001068'} 2023-11-02T18:04:53.756 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003183', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-11-02T18:04:53.756 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000003324 target=00000000000000000000:00000000000000003183, shard=3 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929 2023-11-02T18:04:58.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:58.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:04:58.758 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:04:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:04:58.808 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:04:58.891 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:04:58.891 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:04:58.891 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:04:58.892 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:04:58.892 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:04:58.892 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 14562503763241451158 2023-11-02T18:04:58.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:58.892 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:04:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:04:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:04:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:04:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:00.706996Z" 2023-11-02T18:04:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:04:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:04:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000003183", 2023-11-02T18:04:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:58.897 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:58.897 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:03:41.866421Z" 2023-11-02T18:04:58.897 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:58.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:04:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:04:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-11-02T18:04:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:59:11.495487Z" 2023-11-02T18:04:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:04:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:04:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:04:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:04:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-11-02T18:04:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:04:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:04:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:04:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:52:00.720077Z" 2023-11-02T18:04:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:04:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:04:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:04:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:04:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:04:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:04:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:04:58Z" 2023-11-02T18:04:58.904 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:04:58.905 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14562503763241451158 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:00.706996Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003183", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:03:41.866421Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:59:11.495487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:52:00.720077Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:04:58Z" } 2023-11-02T18:04:58.906 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:00.706996Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003183', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:03:41.866421Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:59:11.495487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:52:00.720077Z'}}] 2023-11-02T18:04:58.907 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001068'} 2023-11-02T18:04:58.907 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003183', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-11-02T18:04:58.907 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000003324 target=00000000000000000000:00000000000000003183, shard=3 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929 2023-11-02T18:05:03.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:03.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:03.909 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:03.958 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:03.959 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:04.045 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:05:04.046 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:05:04.046 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:05:04.046 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:05:04.046 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:05:04.047 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 14562503763241451158 2023-11-02T18:05:04.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.047 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:05:04.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:04.048 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:05:04.048 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:04.048 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:04.048 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:05:04.049 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:04.049 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:04.049 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:04.049 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:00.706996Z" 2023-11-02T18:05:04.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:04.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:04.050 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:05:04.051 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:04.051 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:04.051 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-11-02T18:05:04.051 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:04.052 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:04.052 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:04.052 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.947886Z" 2023-11-02T18:05:04.052 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:04.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:04.053 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:05:04.053 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:04.054 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:04.054 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-11-02T18:05:04.054 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:04.054 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:04.055 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:04.055 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:59:11.495487Z" 2023-11-02T18:05:04.055 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:04.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:04.056 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:05:04.056 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:04.056 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:04.057 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-11-02T18:05:04.057 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:04.057 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:04.057 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:04.058 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.648707Z" 2023-11-02T18:05:04.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:04.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:04.058 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:04.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.059 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:05:04.059 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:05:04.059 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:05:04.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.060 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:05:04Z" 2023-11-02T18:05:04.060 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:05:04.061 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14562503763241451158 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:00.706996Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.947886Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:59:11.495487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.648707Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:05:04Z" } 2023-11-02T18:05:04.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:00.706996Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.947886Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:59:11.495487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.648707Z'}}] 2023-11-02T18:05:04.062 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001068'} 2023-11-02T18:05:04.063 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001068'} 2023-11-02T18:05:04.063 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:05:04.064 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-11-02T18:05:04.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:04.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:04.065 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:04.110 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:04.111 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:04.207 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:05:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:05:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:52:00.715412Z" 2023-11-02T18:05:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-11-02T18:05:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:05:01.920415Z" 2023-11-02T18:05:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-11-02T18:05:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:58:40.713454Z" 2023-11-02T18:05:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-11-02T18:05:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:05:00.782707Z" 2023-11-02T18:05:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:04.211 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:05:04.212 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004170', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-11-02T18:05:04.213 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:05:04.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:04.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:04.214 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:04.260 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:04.261 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:04.348 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:05:04.348 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:05:04.348 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:05:04.349 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:05:04.349 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:05:04.349 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:05:04.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.350 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:05:04.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:04.350 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:05:04.351 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:04.351 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:04.351 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:05:04.351 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:04.352 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:04.352 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:04.352 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:52:00.715412Z" 2023-11-02T18:05:04.352 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:04.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:04.353 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:05:04.353 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:04.354 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:04.354 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-11-02T18:05:04.354 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:04.354 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:04.354 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:04.355 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:48.013637Z" 2023-11-02T18:05:04.355 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:04.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:04.356 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:05:04.356 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:04.356 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:04.356 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-11-02T18:05:04.356 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:04.357 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:04.357 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:04.357 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:58:40.713454Z" 2023-11-02T18:05:04.357 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:04.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:04.358 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:05:04.358 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:04.359 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:04.359 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-11-02T18:05:04.359 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:04.359 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:04.359 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:04.360 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.744614Z" 2023-11-02T18:05:04.360 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:04.360 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:04.360 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:04.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.361 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:05:04.361 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:05:04.361 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:05:04.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:04.362 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:05:04Z" 2023-11-02T18:05:04.362 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:05:04.363 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:52:00.715412Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:48.013637Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:58:40.713454Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.744614Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:05:04Z" } 2023-11-02T18:05:04.363 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:52:00.715412Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:48.013637Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:58:40.713454Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.744614Z'}}] 2023-11-02T18:05:04.364 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004170', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-11-02T18:05:04.364 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001068'} 2023-11-02T18:05:04.365 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029, shard=3 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-11-02T18:05:09.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:09.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:09.367 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:09.415 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:09.415 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:09.499 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:05:09.499 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:05:09.500 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:05:09.500 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:05:09.500 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:05:09.501 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:05:09.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:09.501 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:05:09.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:09.501 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:05:09.502 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:09.502 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:09.502 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:05:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:52:00.715412Z" 2023-11-02T18:05:09.505 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:09.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:09.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:09.506 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:05:09.506 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:09.506 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:09.506 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-11-02T18:05:09.506 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:09.507 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:09.507 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:09.507 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:48.013637Z" 2023-11-02T18:05:09.507 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:09.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:09.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:09.508 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:05:09.508 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:09.508 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:09.509 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-11-02T18:05:09.509 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:09.509 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:09.509 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:09.510 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:58:40.713454Z" 2023-11-02T18:05:09.510 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:09.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:09.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:09.511 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:05:09.511 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:09.511 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:09.511 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-11-02T18:05:09.511 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:09.512 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:09.512 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:09.512 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.744614Z" 2023-11-02T18:05:09.513 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:09.513 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:09.513 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:09.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:09.514 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:05:09.514 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:05:09.514 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:05:09.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:09.514 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:05:09Z" 2023-11-02T18:05:09.515 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:05:09.516 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:52:00.715412Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:48.013637Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:58:40.713454Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.744614Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:05:09Z" } 2023-11-02T18:05:09.516 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:52:00.715412Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:48.013637Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:58:40.713454Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.744614Z'}}] 2023-11-02T18:05:09.517 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004170', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-11-02T18:05:09.517 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001068'} 2023-11-02T18:05:09.518 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029, shard=3 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-11-02T18:05:14.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:14.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:14.520 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:14.653 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:05:14.653 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:05:14.654 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:05:14.654 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:05:14.654 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:05:14.654 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:05:14.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:14.655 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:05:14.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:14.655 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:05:14.655 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:14.656 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:14.656 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:05:14.656 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:14.656 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:14.656 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:14.657 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:52:00.715412Z" 2023-11-02T18:05:14.657 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:14.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:14.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:14.657 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:05:14.658 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:14.658 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:14.658 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-11-02T18:05:14.658 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:14.659 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:14.659 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:14.659 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:48.013637Z" 2023-11-02T18:05:14.659 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:14.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:14.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:14.660 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:05:14.660 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:14.660 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:14.660 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-11-02T18:05:14.661 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:14.661 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:14.661 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:14.661 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:58:40.713454Z" 2023-11-02T18:05:14.662 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:14.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:14.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:14.662 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:05:14.662 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:14.663 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:14.663 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-11-02T18:05:14.663 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:14.663 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:14.664 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:14.664 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.744614Z" 2023-11-02T18:05:14.664 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:14.664 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:14.664 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:14.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:14.665 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:05:14.665 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:05:14.665 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:05:14.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:14.666 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:05:14Z" 2023-11-02T18:05:14.666 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:05:14.667 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:52:00.715412Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:48.013637Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:58:40.713454Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.744614Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:05:14Z" } 2023-11-02T18:05:14.668 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:52:00.715412Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:48.013637Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:58:40.713454Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.744614Z'}}] 2023-11-02T18:05:14.668 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004170', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-11-02T18:05:14.669 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001068'} 2023-11-02T18:05:14.669 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029, shard=3 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-11-02T18:05:19.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:19.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:19.672 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:19.718 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:19.718 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:19.806 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:05:19.806 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:05:19.806 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:05:19.806 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:05:19.807 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:05:19.807 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:05:19.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:19.807 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:05:19.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:19.808 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:05:19.808 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:19.808 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:19.808 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:05:19.809 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:19.809 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:19.809 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:19.809 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:52:00.715412Z" 2023-11-02T18:05:19.809 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:05:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:19.811 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:19.811 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-11-02T18:05:19.811 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:19.811 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:19.811 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:19.812 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:48.013637Z" 2023-11-02T18:05:19.812 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:19.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:19.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:19.813 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:05:19.813 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:19.813 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:19.813 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-11-02T18:05:19.814 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:19.814 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:19.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:19.814 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:58:40.713454Z" 2023-11-02T18:05:19.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:19.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:19.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:19.815 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:05:19.815 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:19.816 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:19.816 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-11-02T18:05:19.816 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:19.816 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:19.816 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:19.817 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.744614Z" 2023-11-02T18:05:19.817 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:19.817 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:19.817 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:19.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:19.818 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:05:19.818 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:05:19.818 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:05:19.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:19.819 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:05:19Z" 2023-11-02T18:05:19.819 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:05:19.820 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:52:00.715412Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:48.013637Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:58:40.713454Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.744614Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:05:19Z" } 2023-11-02T18:05:19.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:52:00.715412Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:48.013637Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:58:40.713454Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.744614Z'}}] 2023-11-02T18:05:19.821 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004170', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-11-02T18:05:19.821 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001068'} 2023-11-02T18:05:19.822 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029, shard=3 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-11-02T18:05:24.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:24.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:24.825 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:24.874 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:24.874 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:24.960 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:05:24.961 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:05:24.961 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:05:24.961 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:05:24.962 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:05:24.962 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:05:24.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:24.962 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:05:24.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:24.963 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:05:24.963 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:24.963 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:24.963 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:05:24.964 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:24.964 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:24.964 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:24.964 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:52:00.715412Z" 2023-11-02T18:05:24.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:24.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:24.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:24.965 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:05:24.965 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:24.966 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:24.966 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-11-02T18:05:24.966 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:24.966 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:24.967 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:24.967 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:05:01.920415Z" 2023-11-02T18:05:24.967 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:24.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:24.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:24.968 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:05:24.968 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:24.968 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:24.968 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-11-02T18:05:24.968 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:24.969 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:24.969 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:24.969 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:58:40.713454Z" 2023-11-02T18:05:24.969 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:24.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:24.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:24.970 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:05:24.970 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:24.970 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:05:24.971 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-11-02T18:05:24.971 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:05:24.971 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:05:24.971 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:05:24.971 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:05:00.782707Z" 2023-11-02T18:05:24.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:24.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:24.972 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:24.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:24.972 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:05:24.973 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:05:24.973 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:05:24.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:24.973 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:05:24Z" 2023-11-02T18:05:24.973 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:05:24.975 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:52:00.715412Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:05:01.920415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:58:40.713454Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:05:00.782707Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:05:24Z" } 2023-11-02T18:05:24.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:52:00.715412Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:05:01.920415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:58:40.713454Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:05:00.782707Z'}}] 2023-11-02T18:05:24.976 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004170', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-11-02T18:05:24.976 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004170', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-11-02T18:05:24.976 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:05:24.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:24.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:24.978 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:25.021 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:25.021 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:25.119 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:05:25.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:25.119 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:05:25.119 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:00.706996Z" 2023-11-02T18:05:25.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:25.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:25.121 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000003465", 2023-11-02T18:05:25.121 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:05:01.857110Z" 2023-11-02T18:05:25.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:25.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:25.122 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-11-02T18:05:25.122 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:59:11.495487Z" 2023-11-02T18:05:25.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:25.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:25.123 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-11-02T18:05:25.123 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:05:00.792295Z" 2023-11-02T18:05:25.123 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:25.123 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:05:25.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:05:25.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:05:25.126 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:05:25.171 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:25.171 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:25.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:25.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:25.277 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:25.364 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:25.365 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:25.449 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T18:05:25.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:25.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:25.453 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:25.578 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T18:05:25.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:25.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:25.582 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:25.627 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:25.627 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:25.714 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T18:05:25.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:25.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:25.718 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:25.848 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T18:05:25.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:25.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:25.852 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:25.890 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:25.891 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:25.991 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:05:25.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:25.991 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:05:25.991 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:00.706996Z" 2023-11-02T18:05:25.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:25.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:25.992 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000003465", 2023-11-02T18:05:25.992 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:05:01.857110Z" 2023-11-02T18:05:25.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:25.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:25.993 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-11-02T18:05:25.993 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:59:11.495487Z" 2023-11-02T18:05:25.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:25.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:25.994 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-11-02T18:05:25.994 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:05:00.792295Z" 2023-11-02T18:05:25.994 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:25.995 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:05:25.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:05:25.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:05:25.997 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:05:26.038 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:26.039 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:26.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:26.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:26.147 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:26.230 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:26.231 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:26.314 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T18:05:26.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:26.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:26.318 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:26.361 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:26.448 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T18:05:26.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:26.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:26.451 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:26.498 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:26.498 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:26.577 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T18:05:26.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:26.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:05:26.581 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:05:26.621 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:26.622 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:26.706 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T18:05:26.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2023-11-02T18:05:26.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:05:26.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:05:26.712 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:05:26.797 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:26.798 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:26.839 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:05:26.839 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:05:26.840 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:05:26.840 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "a2.rgw.meta:root", 2023-11-02T18:05:26.840 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "a2.rgw.control", 2023-11-02T18:05:26.840 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "a2.rgw.log:gc", 2023-11-02T18:05:26.841 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "a2.rgw.log:lc", 2023-11-02T18:05:26.841 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "a2.rgw.log", 2023-11-02T18:05:26.841 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-11-02T18:05:26.841 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-11-02T18:05:26.842 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-11-02T18:05:26.842 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-11-02T18:05:26.842 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-11-02T18:05:26.842 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-11-02T18:05:26.843 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-11-02T18:05:26.843 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-11-02T18:05:26.843 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "a2.rgw.otp", 2023-11-02T18:05:26.843 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2023-11-02T18:05:26.844 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T18:05:26.844 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T18:05:26.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:26.844 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2023-11-02T18:05:26.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:26.845 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2023-11-02T18:05:26.845 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:05:26.845 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "a2.rgw.buckets.index", 2023-11-02T18:05:26.845 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2023-11-02T18:05:26.846 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2023-11-02T18:05:26.846 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "a2.rgw.buckets.data", 2023-11-02T18:05:26.846 INFO:teuthology.orchestra.run.smithi017.stdout: "compression_type": "random" 2023-11-02T18:05:26.846 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:26.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:26.847 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-11-02T18:05:26.847 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0, 2023-11-02T18:05:26.847 INFO:teuthology.orchestra.run.smithi017.stdout: "inline_data": true 2023-11-02T18:05:26.848 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:26.848 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:26.848 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:05:26.848 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:05:26.849 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "a2.rgw.log:notif" 2023-11-02T18:05:26.849 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:05:26.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:05:26.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:05:26.852 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:05:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:05:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:05:26.991 INFO:teuthology.orchestra.run.smithi017.stderr:Sending period to new master zone d61b2bea-59a8-4454-8491-ac3afbd04937 2023-11-02T18:05:27.042 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:05:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:05:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5, 2023-11-02T18:05:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T18:05:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T18:05:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T18:05:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:05:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T18:05:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T18:05:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T18:05:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:05:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T18:05:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:05:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:05:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:05:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:05:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:05:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:05:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:05:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:05:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:05:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:05:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T18:05:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:05:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:05:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:05:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:05:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:05:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:05:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": false, 2023-11-02T18:05:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:05:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:05:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:05:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:05:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:05:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:05:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T18:05:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:05:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:05:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:05:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:05:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:05:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:05:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:05:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:05:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:05:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:05:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:05:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:05:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:05:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:05:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:05:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:05:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:05:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:05:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:05:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:05:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:05:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:05:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:05:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:05:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:05:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:05:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:05:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T18:05:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T18:05:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:05:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T18:05:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:05:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:05:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:05:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:05:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:05:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:05:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:05:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:05:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T18:05:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:05:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:05:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:05:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:05:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:05:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:05:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:05:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:05:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:05:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:05:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:05:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:05:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:05:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:05:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:05:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T18:05:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:05:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T18:05:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:05:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:05:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:05:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:05:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:05:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:05:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:05:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:05:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:05:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:05:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:05:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:05:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:05:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:05:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:05:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:05:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:05:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:05:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:05:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:05:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:05:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:05:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:05:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:05:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:05:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:05:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T18:05:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:05:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T18:05:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:05:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T18:05:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:05:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T18:05:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:05:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:05:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T18:05:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:05:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T18:05:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:05:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T18:05:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T18:05:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:05:27.087 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:05:27.087 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:05:27.087 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:05:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:05:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T18:05:27.089 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:05:27.089 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:05:27.090 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:05:27.090 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:05:27.091 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:05:27.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.091 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T18:05:27.092 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:05:27.092 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:05:27.092 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:05:27.092 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:05:27.093 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:05:27.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.093 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T18:05:27.093 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:05:27.094 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:05:27.094 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:05:27.094 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:05:27.094 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:05:27.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.095 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T18:05:27.095 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:05:27.095 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:05:27.096 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:05:27.096 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:05:27.096 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:05:27.097 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:05:27.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:05:27.097 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:05:27.097 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T18:05:27.098 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:05:27.098 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:05:27.099 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2023-11-02T18:06:27.100 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-35 2023-11-02T18:06:27.101 DEBUG:boto:path=/ibtljn-35/ 2023-11-02T18:06:27.101 DEBUG:boto:auth_path=/ibtljn-35/ 2023-11-02T18:06:27.102 DEBUG:boto:Method: PUT 2023-11-02T18:06:27.102 DEBUG:boto:Path: /ibtljn-35/ 2023-11-02T18:06:27.103 DEBUG:boto:Data: 2023-11-02T18:06:27.103 DEBUG:boto:Headers: {} 2023-11-02T18:06:27.103 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:06:27.104 DEBUG:boto:Port: 8000 2023-11-02T18:06:27.104 DEBUG:boto:Params: {} 2023-11-02T18:06:27.105 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:06:27.106 DEBUG:boto:Token: None 2023-11-02T18:06:27.107 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:06:27 GMT /ibtljn-35/ 2023-11-02T18:06:27.107 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MKkSQT3KcQqdfdMeqLAPnByxdug= 2023-11-02T18:06:27.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:06:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MKkSQT3KcQqdfdMeqLAPnByxdug=', 'Content-Length': '0'} 2023-11-02T18:06:27.143 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007cc32e47711bbf62-006543e523-1517-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:06:27.144 DEBUG:boto:path=/ibtljn-35/testredirect 2023-11-02T18:06:27.145 DEBUG:boto:auth_path=/ibtljn-35/testredirect 2023-11-02T18:06:27.145 DEBUG:boto:Method: PUT 2023-11-02T18:06:27.145 DEBUG:boto:Path: /ibtljn-35/testredirect 2023-11-02T18:06:27.146 DEBUG:boto:Data: 2023-11-02T18:06:27.146 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-11-02T18:06:27.147 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:06:27.147 DEBUG:boto:Port: 8000 2023-11-02T18:06:27.148 DEBUG:boto:Params: {} 2023-11-02T18:06:27.148 DEBUG:boto:Token: None 2023-11-02T18:06:27.149 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Thu, 02 Nov 2023 18:06:27 GMT /ibtljn-35/testredirect 2023-11-02T18:06:27.149 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GA/HNakJLi99sbbo2IZPXC7Rc0M= 2023-11-02T18:06:27.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:06:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GA/HNakJLi99sbbo2IZPXC7Rc0M='} 2023-11-02T18:06:27.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000234f6d5359e7ba81-006543e523-1517-a1'), ('Date', 'Thu, 02 Nov 2023 18:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:06:27.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:27.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:27.164 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:06:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:06:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:06:27.266 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:06:27.292 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:06:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:06:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:06:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:06:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:06:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:06:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:06:27.133683Z" 2023-11-02T18:06:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:06:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:06:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:27.297 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:06:27.298 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:27.298 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:06:27.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:27.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:27.300 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:06:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:06:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:06:27.405 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:27.394+0000 7f0911346f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:27.407 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:27.398+0000 7f0911346f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:27.434 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:06:27.434 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:06:27.435 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:06:27.435 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:06:27.435 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:06:27.436 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:06:27.436 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:27.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:27.436 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:06:27.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:27.437 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:06:27.437 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:27.437 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:27.438 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:06:27.438 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:27.438 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:06:27.438 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:06:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:27.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:27.440 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:06:27.440 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:27.440 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:27.441 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:06:27.441 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:27.441 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:27.442 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:27.442 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:06:27.442 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:27.442 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:27.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:27.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:27.443 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:06:27.443 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:27.444 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:27.444 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:06:27.444 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:27.444 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:27.445 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:27.445 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:06:27.445 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:27.445 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:27.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:27.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:27.446 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:06:27.446 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:27.446 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:27.446 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:06:27.447 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:27.447 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:27.447 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:27.447 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:06:27.448 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:27.448 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:27.448 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:27.448 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:06:27.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:27.449 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:06:27.449 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:06:27.449 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:06:27.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:27.450 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:06:27Z" 2023-11-02T18:06:27.450 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:06:27.451 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:06:27Z" } 2023-11-02T18:06:27.452 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:06:27.452 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:27.452 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:27.453 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948387.133683_219.1 target=1_1698948287.903346_205.1 2023-11-02T18:06:32.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:32.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:32.455 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:06:32.501 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:06:32.502 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:06:32.560 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:32.550+0000 7f4bffbf2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:32.562 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:32.550+0000 7f4bffbf2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:32.588 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:06:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:06:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:06:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:06:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:06:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:06:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:06:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:06:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:06:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:06:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:06:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:06:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:06:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:06:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:06:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:06:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:06:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:06:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:06:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:06:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:06:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:06:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:06:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:06:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:06:32Z" 2023-11-02T18:06:32.602 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:06:32.603 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:06:32Z" } 2023-11-02T18:06:32.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:06:32.604 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:32.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:32.605 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948387.133683_219.1 target=1_1698948287.903346_205.1 2023-11-02T18:06:37.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:37.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:37.607 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:06:37.653 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:06:37.653 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:06:37.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:37.702+0000 7fa3c1e13f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:37.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:37.706+0000 7fa3c1e13f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:37.739 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:06:37.740 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:06:37.740 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:06:37.740 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:06:37.740 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:06:37.740 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:06:37.741 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:37.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:37.741 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:06:37.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:37.742 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:06:37.742 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:37.742 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:37.742 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:06:37.743 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:37.743 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:06:37.743 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:37.743 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:06:37.744 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:37.744 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:37.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:37.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:37.745 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:06:37.745 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:37.745 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:37.745 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:06:37.745 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:37.746 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:37.746 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:37.746 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:06:37.746 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:37.747 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:37.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:37.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:37.747 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:06:37.747 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:37.748 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:37.748 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:06:37.748 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:37.749 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:37.749 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:37.749 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:06:37.749 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:37.750 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:37.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:37.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:37.750 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:06:37.750 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:37.751 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:37.751 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:06:37.751 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:37.751 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:37.752 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:37.752 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:06:37.752 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:37.752 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:37.752 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:37.753 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:06:37.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:37.753 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:06:37.753 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:06:37.753 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:06:37.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:37.754 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:06:37Z" 2023-11-02T18:06:37.754 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:06:37.755 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:06:37Z" } 2023-11-02T18:06:37.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:06:37.757 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:37.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:37.757 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948387.133683_219.1 target=1_1698948287.903346_205.1 2023-11-02T18:06:42.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:42.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:42.759 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:06:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:06:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:06:42.863 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:42.854+0000 7ff84070df40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:42.864 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:42.854+0000 7ff84070df40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:42.889 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:06:42.890 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:06:42.890 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:06:42.890 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:06:42.890 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:06:42.891 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:06:42.891 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:42.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:42.892 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:06:42.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:42.892 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:06:42.892 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:42.893 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:42.893 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:06:42.893 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:42.894 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:06:42.894 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:42.894 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:06:42.894 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:42.895 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:42.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:42.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:42.895 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:06:42.896 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:42.896 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:42.896 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:06:42.896 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:42.897 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:42.897 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:42.897 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:06:42.898 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:42.898 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:42.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:42.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:42.899 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:06:42.899 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:42.899 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:42.899 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:06:42.900 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:42.900 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:42.900 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:42.900 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:06:42.901 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:42.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:42.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:42.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:42.902 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:06:42.902 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:42.902 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:42.903 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:06:42.903 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:42.903 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:42.903 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:42.903 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:06:42.904 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:42.904 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:42.904 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:42.904 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:06:42.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:42.905 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:06:42.905 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:06:42.905 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:06:42.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:42.906 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:06:42Z" 2023-11-02T18:06:42.906 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:06:42.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:06:42Z" } 2023-11-02T18:06:42.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:06:42.908 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:42.909 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:42.909 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948387.133683_219.1 target=1_1698948287.903346_205.1 2023-11-02T18:06:47.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:47.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:47.912 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:06:47.959 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:06:47.959 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:06:48.015 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:48.006+0000 7f29264bcf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:48.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:48.006+0000 7f29264bcf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:48.043 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:06:48.044 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:06:48.044 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:06:48.044 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:06:48.045 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:06:48.045 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:06:48.045 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:48.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:48.046 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:06:48.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:48.046 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:06:48.046 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:48.047 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:48.047 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:06:48.047 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:48.047 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:06:48.048 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:48.048 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:06:48.048 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:48.048 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:48.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:48.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:48.049 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:06:48.049 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:48.049 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:48.050 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:06:48.050 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:48.050 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:48.050 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:48.051 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:06:48.051 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:48.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:48.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:48.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:48.052 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:06:48.052 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:48.052 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:48.052 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:06:48.053 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:48.053 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:48.053 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:48.053 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:06:48.053 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:48.054 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:48.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:48.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:48.054 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:06:48.054 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:48.055 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:48.055 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:06:48.055 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:48.055 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:48.056 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:48.056 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:06:48.056 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:48.056 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:48.056 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:48.057 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:06:48.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:48.057 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:06:48.057 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:06:48.057 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:06:48.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:48.058 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:06:48Z" 2023-11-02T18:06:48.058 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:06:48.059 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:06:48Z" } 2023-11-02T18:06:48.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:06:48.060 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:48.061 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:48.061 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948387.133683_219.1 target=1_1698948287.903346_205.1 2023-11-02T18:06:53.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:53.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:53.064 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:06:53.114 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:06:53.114 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:06:53.173 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:53.162+0000 7f964e81af40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:53.175 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:53.166+0000 7f964e81af40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:53.200 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:06:53.200 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:06:53.201 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:06:53.201 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:06:53.201 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:06:53.201 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:06:53.202 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:53.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:53.202 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:06:53.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:53.203 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:06:53.203 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:53.203 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:53.203 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:06:53.203 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:53.204 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:06:53.204 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:53.204 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:06:53.204 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:53.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:53.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:53.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:53.205 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:06:53.205 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:53.206 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:53.206 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:06:53.206 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:53.206 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:53.207 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:53.207 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:06:53.207 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:53.207 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:53.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:53.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:53.208 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:06:53.208 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:53.208 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:53.209 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:06:53.209 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:53.209 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:53.209 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:53.209 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:06:53.210 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:53.210 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:53.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:53.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:53.211 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:06:53.211 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:53.211 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:53.211 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:06:53.212 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:53.212 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:53.212 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:53.212 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:06:53.212 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:53.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:53.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:53.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:06:53.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:53.213 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:06:53.214 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:06:53.214 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:06:53.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:53.214 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:06:53Z" 2023-11-02T18:06:53.214 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:06:53.215 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:06:53Z" } 2023-11-02T18:06:53.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:06:53.217 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:53.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:53.217 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948387.133683_219.1 target=1_1698948287.903346_205.1 2023-11-02T18:06:58.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:58.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:06:58.219 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:06:58.268 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:06:58.269 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:06:58.327 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:58.318+0000 7f8f73d2bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:58.328 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:06:58.318+0000 7f8f73d2bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:06:58.348 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:06:58.348 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:06:58.348 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:06:58.349 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:06:58.349 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:06:58.349 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:06:58.349 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:58.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:58.350 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:06:58.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:58.350 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:06:58.350 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:58.351 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:58.351 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:06:58.351 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:58.351 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:06:58.352 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:58.352 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:06:58.352 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:58.352 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:58.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:58.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:58.353 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:06:58.353 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:58.353 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:58.354 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:06:58.354 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:58.354 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:58.354 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:58.355 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:06:58.355 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:58.355 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:58.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:58.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:58.356 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:06:58.356 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:58.356 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:58.356 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:06:58.357 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:58.357 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:58.357 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:58.357 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:06:58.357 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:58.358 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:58.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:58.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:06:58.358 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:06:58.359 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:06:58.359 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:06:58.359 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:06:58.359 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:06:58.359 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:06:58.360 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:06:58.360 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:06:58.360 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:06:58.360 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:58.361 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:06:58.361 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:06:58.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:58.361 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:06:58.361 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:06:58.362 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:06:58.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:06:58.362 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:06:58Z" 2023-11-02T18:06:58.362 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:06:58.363 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:06:58Z" } 2023-11-02T18:06:58.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:06:58.364 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:58.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:06:58.365 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948387.133683_219.1 target=1_1698948287.903346_205.1 2023-11-02T18:07:03.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:07:03.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:07:03.368 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:07:03.415 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:07:03.416 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:07:03.474 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:07:03.462+0000 7ff925ad4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:07:03.476 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:07:03.466+0000 7ff925ad4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:07:03.503 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:07:03.503 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:07:03.503 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:07:03.503 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:07:03.504 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:07:03.504 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:07:03.504 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:03.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:03.505 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:07:03.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:03.505 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:07:03.505 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:03.505 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:03.506 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:07:03.506 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:03.506 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:07:03.506 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:03.507 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:07:03.507 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:03.507 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:03.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:03.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:03.508 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:07:03.508 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:03.508 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:03.508 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:07:03.509 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:03.509 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:03.509 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:03.509 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:07:03.509 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:03.510 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:03.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:03.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:03.510 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:07:03.511 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:03.511 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:03.511 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:07:03.511 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:03.511 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:03.512 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:03.512 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:07:03.512 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:03.512 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:03.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:03.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:03.513 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:07:03.513 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:03.513 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:03.514 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:07:03.514 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:03.514 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:03.514 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:03.514 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:07:03.515 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:03.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:03.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:03.515 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:03.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:03.516 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:07:03.516 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:07:03.516 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:07:03.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:03.517 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:07:03Z" 2023-11-02T18:07:03.517 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:07:03.518 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:07:03Z" } 2023-11-02T18:07:03.518 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:07:03.519 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:07:03.519 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:07:03.520 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948387.133683_219.1 target=1_1698948287.903346_205.1 2023-11-02T18:07:08.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:07:08.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:07:08.523 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:07:08.569 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:07:08.569 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:07:08.621 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:07:08.610+0000 7ff520d8ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:07:08.623 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:07:08.614+0000 7ff520d8ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:07:08.649 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:07:08.650 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:07:08.650 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:07:08.650 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:07:08.650 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:07:08.651 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:07:08.651 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:08.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:08.651 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:07:08.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:08.652 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:07:08.652 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:08.652 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:08.653 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:07:08.653 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:08.653 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:07:08.653 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:08.653 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:07:08.654 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:08.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:08.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:08.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:08.655 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:07:08.655 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:08.655 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:08.655 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:07:08.655 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:08.656 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:08.656 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:08.656 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:07:08.656 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:08.656 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:08.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:08.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:08.657 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:07:08.657 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:08.657 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:08.658 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:07:08.658 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:08.658 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:08.658 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:08.659 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:07:08.659 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:08.659 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:08.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:08.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:08.660 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:07:08.660 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:08.660 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:08.660 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:07:08.660 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:08.661 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:08.661 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:08.661 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:07:08.661 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:08.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:08.662 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:08.662 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:08.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:08.662 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:07:08.663 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:07:08.663 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:07:08.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:08.663 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:07:08Z" 2023-11-02T18:07:08.663 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:07:08.664 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:07:08Z" } 2023-11-02T18:07:08.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:07:08.665 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:07:08.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:07:08.666 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948387.133683_219.1 target=1_1698948287.903346_205.1 2023-11-02T18:07:13.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:07:13.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:07:13.669 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:07:13.716 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:07:13.716 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:07:13.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:07:13.766+0000 7ff49851cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:07:13.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:07:13.766+0000 7ff49851cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:07:13.803 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:07:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:07:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:07:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:07:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:07:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:07:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:07:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:07:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:07:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:07:13.808 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:13.808 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:07:13.808 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:13.808 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:13.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:13.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:13.809 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:07:13.809 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:13.810 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:13.810 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:07:13.810 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:13.810 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:13.811 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:13.811 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:07:13.811 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:13.811 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:13.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:13.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:13.812 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:07:13.812 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:13.813 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:13.813 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:07:13.813 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:13.813 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:13.813 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:13.814 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:07:13.814 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:13.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:13.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:13.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:13.815 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:07:13.815 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:13.815 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:13.816 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:07:13.816 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:13.816 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:13.816 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:13.816 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:07:13.817 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:13.817 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:13.817 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:13.817 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:13.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:13.818 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:07:13.818 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:07:13.818 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:07:13.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:13.819 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:07:13Z" 2023-11-02T18:07:13.819 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:07:13.820 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:07:13Z" } 2023-11-02T18:07:13.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:07:13.821 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:07:13.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:07:13.822 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948387.133683_219.1 target=1_1698948287.903346_205.1 2023-11-02T18:07:18.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:07:18.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:07:18.825 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:07:18.870 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:07:18.871 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:07:18.931 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:07:18.922+0000 7f628bbb8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:07:18.932 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:07:18.922+0000 7f628bbb8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:07:18.958 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:07:18.959 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:07:18.959 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:07:18.959 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:07:18.959 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:07:18.959 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:07:18.960 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:18.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:18.960 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:07:18.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:18.960 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:07:18.961 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:18.961 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:18.961 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:07:18.961 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:18.961 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:07:18.962 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:18.962 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:07:18.962 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:18.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:18.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:18.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:18.963 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:07:18.963 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:18.963 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:18.964 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:07:18.964 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:18.964 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:18.964 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:18.964 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:07:18.965 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:18.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:18.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:18.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:18.966 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:07:18.966 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:18.966 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:18.966 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:07:18.966 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:18.967 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:18.967 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:18.967 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:07:18.967 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:18.967 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:18.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:18.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:18.968 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:07:18.968 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:18.969 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:18.969 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:07:18.969 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:18.969 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:18.970 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:18.970 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:07:18.970 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:18.970 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:18.970 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:18.971 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:18.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:18.971 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:07:18.971 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:07:18.971 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:07:18.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:18.972 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:07:18Z" 2023-11-02T18:07:18.972 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:07:18.973 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:07:18Z" } 2023-11-02T18:07:18.974 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:07:18.974 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:07:18.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:07:18.975 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948387.133683_219.1 target=1_1698948287.903346_205.1 2023-11-02T18:07:23.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:07:23.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:07:23.977 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:07:24.022 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:07:24.023 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:07:24.080 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:07:24.070+0000 7ff964edcf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:07:24.081 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:07:24.070+0000 7ff964edcf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:07:24.107 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:07:24.107 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:07:24.107 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:07:24.107 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:07:24.108 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:07:24.108 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:07:24.108 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:24.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:24.109 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:07:24.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:24.109 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:07:24.109 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:24.109 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:24.110 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:07:24.110 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:24.110 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:07:24.110 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:24.111 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:07:24.111 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:24.111 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:24.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:24.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:24.112 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:07:24.112 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:24.112 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:24.112 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:07:24.112 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:24.113 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:24.113 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:24.113 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:07:24.113 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:24.114 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:24.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:24.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:24.114 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:07:24.114 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:24.115 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:24.115 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948287.903346_205.1", 2023-11-02T18:07:24.115 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:24.115 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:24.116 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:24.116 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:04:47.903346Z", 2023-11-02T18:07:24.116 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:24.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:24.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:24.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:24.117 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:07:24.117 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:24.117 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:24.117 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:07:24.118 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:24.118 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:24.118 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:24.118 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:07:24.119 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:24.119 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:24.119 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:24.119 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:24.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:24.120 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:07:24.120 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:07:24.120 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:07:24.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:24.121 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:07:24Z" 2023-11-02T18:07:24.121 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:07:24.122 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948287.903346_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:04:47.903346Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:07:24Z" } 2023-11-02T18:07:24.122 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948287.903346_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:04:47.903346Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:07:24.123 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:07:24.123 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948287.903346_205.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:07:24.124 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948387.133683_219.1 target=1_1698948287.903346_205.1 2023-11-02T18:07:29.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:07:29.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:07:29.126 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:07:29.174 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:07:29.175 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:07:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:07:29.221+0000 7f741d76af40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:07:29.233 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:07:29.221+0000 7f741d76af40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:07:29.261 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:07:29.261 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:07:29.261 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:07:29.262 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:07:29.262 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:07:29.262 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:07:29.262 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:29.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.263 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:07:29.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.263 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:07:29.263 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:29.264 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:29.264 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:07:29.264 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:29.264 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:07:29.264 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:29.265 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:07:29.265 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:29.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.266 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:07:29.267 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:29.267 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:29.267 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:07:29.268 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:29.268 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:29.268 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:29.268 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:07:29.269 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:29.269 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.270 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:07:29.270 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:29.270 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:29.270 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:07:29.271 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:29.271 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:29.271 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:29.271 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:07:29.271 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:29.272 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.272 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:07:29.273 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:29.273 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:07:29.273 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:07:29.273 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:07:29.273 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:07:29.274 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:07:29.274 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:07:29.274 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:29.274 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.275 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.275 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:29.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.275 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:07:29.276 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:07:29.276 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:07:29.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.276 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:07:29Z" 2023-11-02T18:07:29.276 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:07:29.278 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:07:29Z" } 2023-11-02T18:07:29.278 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:07:29.279 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:07:29.279 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:07:29.280 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:07:29.281 DEBUG:boto:path=/ibtljn-35/ 2023-11-02T18:07:29.282 DEBUG:boto:auth_path=/ibtljn-35/ 2023-11-02T18:07:29.282 DEBUG:boto:Method: HEAD 2023-11-02T18:07:29.283 DEBUG:boto:Path: /ibtljn-35/ 2023-11-02T18:07:29.283 DEBUG:boto:Data: 2023-11-02T18:07:29.284 DEBUG:boto:Headers: {} 2023-11-02T18:07:29.284 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:07:29.285 DEBUG:boto:Port: 8001 2023-11-02T18:07:29.285 DEBUG:boto:Params: {} 2023-11-02T18:07:29.286 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:07:29.286 DEBUG:boto:Token: None 2023-11-02T18:07:29.287 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:07:29 GMT /ibtljn-35/ 2023-11-02T18:07:29.287 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RJ7p4b3YIPmrI8mW6nD/l2+YNGQ= 2023-11-02T18:07:29.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:07:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RJ7p4b3YIPmrI8mW6nD/l2+YNGQ=', 'Content-Length': '0'} 2023-11-02T18:07:29.296 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000898e8c0534dd6115-006543e561-151b-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:07:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:07:29.297 DEBUG:boto:path=/ibtljn-35/testredirect 2023-11-02T18:07:29.298 DEBUG:boto:auth_path=/ibtljn-35/testredirect 2023-11-02T18:07:29.298 DEBUG:boto:Method: HEAD 2023-11-02T18:07:29.299 DEBUG:boto:Path: /ibtljn-35/testredirect 2023-11-02T18:07:29.299 DEBUG:boto:Data: 2023-11-02T18:07:29.300 DEBUG:boto:Headers: {} 2023-11-02T18:07:29.300 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:07:29.301 DEBUG:boto:Port: 8001 2023-11-02T18:07:29.301 DEBUG:boto:Params: {} 2023-11-02T18:07:29.301 DEBUG:boto:Token: None 2023-11-02T18:07:29.302 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:07:29 GMT /ibtljn-35/testredirect 2023-11-02T18:07:29.302 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PHQg8URo1Rg2aysM85iJu9LpHeY= 2023-11-02T18:07:29.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:07:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PHQg8URo1Rg2aysM85iJu9LpHeY=', 'Content-Length': '0'} 2023-11-02T18:07:29.306 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000003c59019afd8f963-006543e561-151b-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:07:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:07:29.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=d61b2bea-59a8-4454-8491-ac3afbd04937', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:07:29.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=d61b2bea-59a8-4454-8491-ac3afbd04937', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:07:29.307 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=d61b2bea-59a8-4454-8491-ac3afbd04937 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:07:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:07:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:07:29.396 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:07:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:07:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:07:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "a2.rgw.meta:root", 2023-11-02T18:07:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "a2.rgw.control", 2023-11-02T18:07:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "a2.rgw.log:gc", 2023-11-02T18:07:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "a2.rgw.log:lc", 2023-11-02T18:07:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "a2.rgw.log", 2023-11-02T18:07:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-11-02T18:07:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-11-02T18:07:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-11-02T18:07:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-11-02T18:07:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-11-02T18:07:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-11-02T18:07:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-11-02T18:07:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-11-02T18:07:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "a2.rgw.otp", 2023-11-02T18:07:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2023-11-02T18:07:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T18:07:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T18:07:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2023-11-02T18:07:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2023-11-02T18:07:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:07:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "a2.rgw.buckets.index", 2023-11-02T18:07:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2023-11-02T18:07:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2023-11-02T18:07:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "a2.rgw.buckets.data", 2023-11-02T18:07:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: "compression_type": "random" 2023-11-02T18:07:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-11-02T18:07:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0, 2023-11-02T18:07:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: "inline_data": true 2023-11-02T18:07:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:07:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:07:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "a2.rgw.log:notif" 2023-11-02T18:07:29.405 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:07:29.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:07:29.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:07:29.407 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:07:29.454 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:07:29.455 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:07:29.544 INFO:teuthology.orchestra.run.smithi017.stderr:Sending period to new master zone d61b2bea-59a8-4454-8491-ac3afbd04937 2023-11-02T18:07:29.592 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:07:29.593 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:07:29.593 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6, 2023-11-02T18:07:29.593 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T18:07:29.593 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T18:07:29.594 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T18:07:29.594 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:07:29.594 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T18:07:29.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.595 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T18:07:29.595 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T18:07:29.595 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:07:29.595 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T18:07:29.596 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:07:29.596 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:07:29.596 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:07:29.596 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:07:29.596 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:07:29.597 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:07:29.597 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:07:29.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.597 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:07:29.597 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:07:29.598 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:07:29.598 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T18:07:29.598 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:07:29.598 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:07:29.599 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:07:29.599 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:07:29.599 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:07:29.599 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:07:29.599 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": false, 2023-11-02T18:07:29.600 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:07:29.600 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:07:29.600 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:07:29.600 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:07:29.601 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:07:29.601 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:29.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.601 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:07:29.602 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T18:07:29.602 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:07:29.602 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:07:29.602 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:07:29.603 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:07:29.603 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:07:29.603 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:07:29.603 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:07:29.603 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:07:29.604 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:07:29.604 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:07:29.604 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:07:29.604 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:07:29.605 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:07:29.605 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:07:29.605 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:29.605 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.605 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:07:29.606 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:07:29.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.606 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:07:29.606 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:07:29.607 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:07:29.607 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:07:29.607 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:29.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.608 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:07:29.609 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:07:29.609 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:07:29.609 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:07:29.610 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:07:29.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.610 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:07:29.610 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:07:29.611 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:29.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.611 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T18:07:29.612 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T18:07:29.612 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:07:29.612 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T18:07:29.612 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:07:29.612 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:07:29.613 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:07:29.613 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:07:29.613 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:07:29.613 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:07:29.614 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:07:29.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.614 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:07:29.615 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T18:07:29.615 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:07:29.615 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:07:29.615 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:07:29.616 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:07:29.616 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:07:29.616 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:07:29.616 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:07:29.617 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:07:29.617 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:07:29.617 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:07:29.617 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:07:29.617 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:07:29.619 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:07:29.619 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:07:29.619 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:29.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.620 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:07:29.620 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T18:07:29.620 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:07:29.620 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T18:07:29.621 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:07:29.621 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:07:29.621 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:07:29.621 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:07:29.622 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:07:29.622 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:07:29.622 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:07:29.622 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:07:29.622 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:07:29.623 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:07:29.623 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:07:29.623 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:07:29.623 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:29.624 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.624 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:07:29.624 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:07:29.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.625 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:07:29.625 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:07:29.625 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:07:29.625 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:07:29.625 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:29.626 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.626 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:07:29.626 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:07:29.626 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:07:29.627 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:07:29.627 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:07:29.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.627 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:07:29.628 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:07:29.628 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:29.628 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.628 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:07:29.628 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T18:07:29.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.629 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:07:29.629 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T18:07:29.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.630 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:07:29.630 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T18:07:29.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.630 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:07:29.631 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T18:07:29.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:07:29.631 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:07:29.631 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T18:07:29.631 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.632 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:07:29.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.632 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T18:07:29.632 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:07:29.632 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T18:07:29.633 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T18:07:29.633 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:07:29.633 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:07:29.633 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:07:29.633 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:07:29.634 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:07:29.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.634 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T18:07:29.634 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:07:29.635 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:07:29.635 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:07:29.635 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:07:29.635 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:07:29.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.636 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T18:07:29.636 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:07:29.636 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:07:29.636 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:07:29.637 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:07:29.637 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:07:29.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.637 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T18:07:29.637 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:07:29.638 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:07:29.638 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:07:29.638 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:07:29.638 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:07:29.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.639 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T18:07:29.639 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:07:29.639 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:07:29.639 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:07:29.640 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:07:29.640 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:07:29.640 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:07:29.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:07:29.640 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:07:29.641 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T18:07:29.641 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:07:29.641 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:07:29.642 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T18:08:29.643 DEBUG:boto:path=/ibtljn-35/testredirect 2023-11-02T18:08:29.644 DEBUG:boto:auth_path=/ibtljn-35/testredirect 2023-11-02T18:08:29.645 DEBUG:boto:Method: HEAD 2023-11-02T18:08:29.645 DEBUG:boto:Path: /ibtljn-35/testredirect 2023-11-02T18:08:29.646 DEBUG:boto:Data: 2023-11-02T18:08:29.646 DEBUG:boto:Headers: {} 2023-11-02T18:08:29.646 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:08:29.647 DEBUG:boto:Port: 8001 2023-11-02T18:08:29.647 DEBUG:boto:Params: {} 2023-11-02T18:08:29.648 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:08:29.648 DEBUG:boto:Token: None 2023-11-02T18:08:29.649 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.649 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fDbBT//IyJWLtA1A3bQ1juhVRoA= 2023-11-02T18:08:29.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fDbBT//IyJWLtA1A3bQ1juhVRoA=', 'Content-Length': '0'} 2023-11-02T18:08:29.658 DEBUG:boto:Response headers: [('Location', 'http://smithi017.front.sepia.ceph.com:8000/ibtljn-35/testredirect?'), ('x-amz-request-id', 'tx000000f8503cf8d669a0c-006543e59d-15a3-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.659 DEBUG:boto:Redirecting: http://smithi017.front.sepia.ceph.com/ibtljn-35/testredirect 2023-11-02T18:08:29.659 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2023-11-02T18:08:29.660 DEBUG:boto:Token: None 2023-11-02T18:08:29.660 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.661 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fDbBT//IyJWLtA1A3bQ1juhVRoA= 2023-11-02T18:08:29.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fDbBT//IyJWLtA1A3bQ1juhVRoA=', 'Content-Length': '0'} 2023-11-02T18:08:29.669 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000350a1a87807fbb5e-006543e59d-159f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.670 DEBUG:boto:path=/ibtljn-35/testredirect 2023-11-02T18:08:29.670 DEBUG:boto:auth_path=/ibtljn-35/testredirect 2023-11-02T18:08:29.670 DEBUG:boto:Method: GET 2023-11-02T18:08:29.671 DEBUG:boto:Path: /ibtljn-35/testredirect 2023-11-02T18:08:29.671 DEBUG:boto:Data: 2023-11-02T18:08:29.672 DEBUG:boto:Headers: {} 2023-11-02T18:08:29.672 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:08:29.672 DEBUG:boto:Port: 8001 2023-11-02T18:08:29.673 DEBUG:boto:Params: {} 2023-11-02T18:08:29.673 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:08:29.674 DEBUG:boto:Token: None 2023-11-02T18:08:29.674 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.675 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY= 2023-11-02T18:08:29.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY=', 'Content-Length': '0'} 2023-11-02T18:08:29.679 DEBUG:boto:Response headers: [('Location', 'http://smithi017.front.sepia.ceph.com:8000/ibtljn-35/testredirect?'), ('x-amz-request-id', 'tx00000278672fe39b5e342-006543e59d-15a3-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.679 DEBUG:boto:Redirecting: http://smithi017.front.sepia.ceph.com/ibtljn-35/testredirect 2023-11-02T18:08:29.680 DEBUG:boto:Token: None 2023-11-02T18:08:29.680 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.681 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY= 2023-11-02T18:08:29.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY=', 'Content-Length': '0'} 2023-11-02T18:08:29.684 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx0000070731727e61fe526-006543e59d-159f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.685 DEBUG:boto:path=/ibtljn-35/testredirect 2023-11-02T18:08:29.685 DEBUG:boto:auth_path=/ibtljn-35/testredirect 2023-11-02T18:08:29.686 DEBUG:boto:Method: PUT 2023-11-02T18:08:29.686 DEBUG:boto:Path: /ibtljn-35/testredirect 2023-11-02T18:08:29.687 DEBUG:boto:Data: 2023-11-02T18:08:29.687 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-11-02T18:08:29.688 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:08:29.688 DEBUG:boto:Port: 8000 2023-11-02T18:08:29.688 DEBUG:boto:Params: {} 2023-11-02T18:08:29.690 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:08:29.690 DEBUG:boto:Token: None 2023-11-02T18:08:29.691 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.691 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R8XWF8kFVxxUZfSgrP8vr0a40hc= 2023-11-02T18:08:29.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R8XWF8kFVxxUZfSgrP8vr0a40hc='} 2023-11-02T18:08:29.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048568d45335075bd-006543e59d-159f-a1'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.716 DEBUG:boto:path=/ibtljn-35/testredirect 2023-11-02T18:08:29.716 DEBUG:boto:auth_path=/ibtljn-35/testredirect 2023-11-02T18:08:29.717 DEBUG:boto:Method: GET 2023-11-02T18:08:29.717 DEBUG:boto:Path: /ibtljn-35/testredirect 2023-11-02T18:08:29.717 DEBUG:boto:Data: 2023-11-02T18:08:29.718 DEBUG:boto:Headers: {} 2023-11-02T18:08:29.718 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:08:29.719 DEBUG:boto:Port: 8001 2023-11-02T18:08:29.719 DEBUG:boto:Params: {} 2023-11-02T18:08:29.720 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:08:29.720 DEBUG:boto:Token: None 2023-11-02T18:08:29.720 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY= 2023-11-02T18:08:29.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY=', 'Content-Length': '0'} 2023-11-02T18:08:29.725 DEBUG:boto:Response headers: [('Location', 'http://smithi017.front.sepia.ceph.com:8000/ibtljn-35/testredirect?'), ('x-amz-request-id', 'tx000003abaa3e1fbd0c4dd-006543e59d-15a3-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.726 DEBUG:boto:Redirecting: http://smithi017.front.sepia.ceph.com/ibtljn-35/testredirect 2023-11-02T18:08:29.729 DEBUG:boto:Token: None 2023-11-02T18:08:29.730 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.730 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY= 2023-11-02T18:08:29.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY=', 'Content-Length': '0'} 2023-11-02T18:08:29.734 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx00000d9968407eb15b1fd-006543e59d-159f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.734 DEBUG:boto:path=/ibtljn-35/testredirect 2023-11-02T18:08:29.735 DEBUG:boto:auth_path=/ibtljn-35/testredirect 2023-11-02T18:08:29.735 DEBUG:boto:Method: PUT 2023-11-02T18:08:29.736 DEBUG:boto:Path: /ibtljn-35/testredirect 2023-11-02T18:08:29.736 DEBUG:boto:Data: 2023-11-02T18:08:29.737 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-11-02T18:08:29.737 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:08:29.737 DEBUG:boto:Port: 8000 2023-11-02T18:08:29.738 DEBUG:boto:Params: {} 2023-11-02T18:08:29.738 DEBUG:boto:Token: None 2023-11-02T18:08:29.739 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.739 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GYG1BTfDkAJpu/QI+0XcjmdxkRY= 2023-11-02T18:08:29.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GYG1BTfDkAJpu/QI+0XcjmdxkRY='} 2023-11-02T18:08:29.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d1edda3d657aefc-006543e59d-159f-a1'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.752 DEBUG:boto:path=/ibtljn-35/testredirect 2023-11-02T18:08:29.753 DEBUG:boto:auth_path=/ibtljn-35/testredirect 2023-11-02T18:08:29.753 DEBUG:boto:Method: GET 2023-11-02T18:08:29.754 DEBUG:boto:Path: /ibtljn-35/testredirect 2023-11-02T18:08:29.754 DEBUG:boto:Data: 2023-11-02T18:08:29.755 DEBUG:boto:Headers: {} 2023-11-02T18:08:29.755 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:08:29.755 DEBUG:boto:Port: 8001 2023-11-02T18:08:29.756 DEBUG:boto:Params: {} 2023-11-02T18:08:29.756 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:08:29.757 DEBUG:boto:Token: None 2023-11-02T18:08:29.757 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.758 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY= 2023-11-02T18:08:29.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY=', 'Content-Length': '0'} 2023-11-02T18:08:29.762 DEBUG:boto:Response headers: [('Location', 'http://smithi017.front.sepia.ceph.com:8000/ibtljn-35/testredirect?'), ('x-amz-request-id', 'tx00000201235fb96e3f164-006543e59d-15a3-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.763 DEBUG:boto:Redirecting: http://smithi017.front.sepia.ceph.com/ibtljn-35/testredirect 2023-11-02T18:08:29.763 DEBUG:boto:Token: None 2023-11-02T18:08:29.764 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.764 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY= 2023-11-02T18:08:29.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY=', 'Content-Length': '0'} 2023-11-02T18:08:29.767 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx00000ab0fe54c94f12a44-006543e59d-159f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.768 DEBUG:boto:path=/ibtljn-35/testredirect 2023-11-02T18:08:29.768 DEBUG:boto:auth_path=/ibtljn-35/testredirect 2023-11-02T18:08:29.769 DEBUG:boto:Method: PUT 2023-11-02T18:08:29.769 DEBUG:boto:Path: /ibtljn-35/testredirect 2023-11-02T18:08:29.770 DEBUG:boto:Data: 2023-11-02T18:08:29.770 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-11-02T18:08:29.770 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:08:29.771 DEBUG:boto:Port: 8000 2023-11-02T18:08:29.771 DEBUG:boto:Params: {} 2023-11-02T18:08:29.772 DEBUG:boto:Token: None 2023-11-02T18:08:29.772 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.773 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JcSn55wAC89CD4fTwcyVK5E9Gh4= 2023-11-02T18:08:29.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JcSn55wAC89CD4fTwcyVK5E9Gh4='} 2023-11-02T18:08:29.785 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029c39c602b9101e2-006543e59d-159f-a1'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.785 DEBUG:boto:path=/ibtljn-35/testredirect 2023-11-02T18:08:29.786 DEBUG:boto:auth_path=/ibtljn-35/testredirect 2023-11-02T18:08:29.786 DEBUG:boto:Method: GET 2023-11-02T18:08:29.786 DEBUG:boto:Path: /ibtljn-35/testredirect 2023-11-02T18:08:29.787 DEBUG:boto:Data: 2023-11-02T18:08:29.787 DEBUG:boto:Headers: {} 2023-11-02T18:08:29.788 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:08:29.788 DEBUG:boto:Port: 8001 2023-11-02T18:08:29.788 DEBUG:boto:Params: {} 2023-11-02T18:08:29.789 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:08:29.789 DEBUG:boto:Token: None 2023-11-02T18:08:29.790 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.790 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY= 2023-11-02T18:08:29.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY=', 'Content-Length': '0'} 2023-11-02T18:08:29.794 DEBUG:boto:Response headers: [('Location', 'http://smithi017.front.sepia.ceph.com:8000/ibtljn-35/testredirect?'), ('x-amz-request-id', 'tx0000015c0fcaa8b14a0bf-006543e59d-15a3-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.795 DEBUG:boto:Redirecting: http://smithi017.front.sepia.ceph.com/ibtljn-35/testredirect 2023-11-02T18:08:29.795 DEBUG:boto:Token: None 2023-11-02T18:08:29.796 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.796 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY= 2023-11-02T18:08:29.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY=', 'Content-Length': '0'} 2023-11-02T18:08:29.799 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx00000cc740f327d569e4d-006543e59d-159f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.800 DEBUG:boto:path=/ibtljn-35/testredirect 2023-11-02T18:08:29.801 DEBUG:boto:auth_path=/ibtljn-35/testredirect 2023-11-02T18:08:29.801 DEBUG:boto:Method: PUT 2023-11-02T18:08:29.801 DEBUG:boto:Path: /ibtljn-35/testredirect 2023-11-02T18:08:29.802 DEBUG:boto:Data: 2023-11-02T18:08:29.802 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-11-02T18:08:29.803 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:08:29.803 DEBUG:boto:Port: 8000 2023-11-02T18:08:29.803 DEBUG:boto:Params: {} 2023-11-02T18:08:29.804 DEBUG:boto:Token: None 2023-11-02T18:08:29.804 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.805 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:idBMalSEznCh0G8+kD8pJOCJitI= 2023-11-02T18:08:29.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:idBMalSEznCh0G8+kD8pJOCJitI='} 2023-11-02T18:08:29.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d663c13704c0a00-006543e59d-159f-a1'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.851 DEBUG:boto:path=/ibtljn-35/testredirect 2023-11-02T18:08:29.851 DEBUG:boto:auth_path=/ibtljn-35/testredirect 2023-11-02T18:08:29.852 DEBUG:boto:Method: GET 2023-11-02T18:08:29.852 DEBUG:boto:Path: /ibtljn-35/testredirect 2023-11-02T18:08:29.853 DEBUG:boto:Data: 2023-11-02T18:08:29.853 DEBUG:boto:Headers: {} 2023-11-02T18:08:29.853 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:08:29.854 DEBUG:boto:Port: 8001 2023-11-02T18:08:29.854 DEBUG:boto:Params: {} 2023-11-02T18:08:29.855 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:08:29.855 DEBUG:boto:Token: None 2023-11-02T18:08:29.856 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.856 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY= 2023-11-02T18:08:29.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY=', 'Content-Length': '0'} 2023-11-02T18:08:29.861 DEBUG:boto:Response headers: [('Location', 'http://smithi017.front.sepia.ceph.com:8000/ibtljn-35/testredirect?'), ('x-amz-request-id', 'tx00000297ca00443e8fb01-006543e59d-15a3-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.862 DEBUG:boto:Redirecting: http://smithi017.front.sepia.ceph.com/ibtljn-35/testredirect 2023-11-02T18:08:29.862 DEBUG:boto:Token: None 2023-11-02T18:08:29.863 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:08:29 GMT /ibtljn-35/testredirect 2023-11-02T18:08:29.863 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY= 2023-11-02T18:08:29.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:08:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zVXLpmMwmBorHEZDzc760RcUsxY=', 'Content-Length': '0'} 2023-11-02T18:08:29.866 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx0000068b2561b859eaa15-006543e59d-159f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:08:29.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:08:29.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:08:29.868 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:08:29.913 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:08:29.914 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:08:29.956 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:08:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:08:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:08:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "a2.rgw.meta:root", 2023-11-02T18:08:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "a2.rgw.control", 2023-11-02T18:08:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "a2.rgw.log:gc", 2023-11-02T18:08:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "a2.rgw.log:lc", 2023-11-02T18:08:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "a2.rgw.log", 2023-11-02T18:08:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-11-02T18:08:29.958 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-11-02T18:08:29.958 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-11-02T18:08:29.958 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-11-02T18:08:29.958 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-11-02T18:08:29.959 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-11-02T18:08:29.959 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-11-02T18:08:29.959 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-11-02T18:08:29.959 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "a2.rgw.otp", 2023-11-02T18:08:29.959 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2023-11-02T18:08:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T18:08:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T18:08:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2023-11-02T18:08:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:29.961 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2023-11-02T18:08:29.961 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:08:29.961 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "a2.rgw.buckets.index", 2023-11-02T18:08:29.961 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2023-11-02T18:08:29.961 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2023-11-02T18:08:29.962 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "a2.rgw.buckets.data", 2023-11-02T18:08:29.962 INFO:teuthology.orchestra.run.smithi017.stdout: "compression_type": "random" 2023-11-02T18:08:29.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:08:29.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:29.962 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-11-02T18:08:29.963 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0, 2023-11-02T18:08:29.963 INFO:teuthology.orchestra.run.smithi017.stdout: "inline_data": true 2023-11-02T18:08:29.963 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:08:29.963 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:08:29.963 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:08:29.964 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:08:29.964 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "a2.rgw.log:notif" 2023-11-02T18:08:29.964 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:08:29.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:08:29.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:08:29.966 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:08:30.011 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:08:30.012 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:08:30.099 INFO:teuthology.orchestra.run.smithi017.stderr:Sending period to new master zone d61b2bea-59a8-4454-8491-ac3afbd04937 2023-11-02T18:08:30.152 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:08:30.152 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:08:30.152 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7, 2023-11-02T18:08:30.153 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T18:08:30.153 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T18:08:30.153 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T18:08:30.153 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:08:30.154 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T18:08:30.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:30.154 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T18:08:30.154 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T18:08:30.154 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:08:30.155 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T18:08:30.155 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:08:30.155 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:08:30.155 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:08:30.156 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:08:30.156 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:08:30.156 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:08:30.156 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:08:30.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:30.157 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:08:30.157 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:08:30.157 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:08:30.157 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T18:08:30.158 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:08:30.158 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:08:30.158 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:08:30.158 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:08:30.158 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:08:30.159 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:08:30.159 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:08:30.159 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:08:30.159 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:08:30.159 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:08:30.160 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:08:30.160 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:08:30.160 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:08:30.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:30.161 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:08:30.161 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T18:08:30.161 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:08:30.161 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:08:30.162 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:08:30.162 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:08:30.162 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:08:30.162 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:08:30.163 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:08:30.163 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:08:30.163 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:08:30.163 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:08:30.163 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:08:30.164 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:08:30.164 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:08:30.164 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:08:30.164 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:08:30.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:08:30.165 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:08:30.165 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:08:30.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:30.165 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:08:30.166 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:08:30.166 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:08:30.166 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:08:30.166 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:08:30.166 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:08:30.167 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:08:30.167 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:08:30.167 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:08:30.167 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:08:30.167 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:08:30.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.168 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:08:30.168 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:08:30.168 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:08:30.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:30.169 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T18:08:30.169 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T18:08:30.169 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:08:30.170 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T18:08:30.170 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:08:30.170 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:08:30.170 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:08:30.171 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:08:30.171 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:08:30.171 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:08:30.171 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:08:30.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:30.172 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:08:30.172 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T18:08:30.172 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:08:30.172 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:08:30.172 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:08:30.173 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:08:30.173 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:08:30.173 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:08:30.173 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:08:30.173 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:08:30.174 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:08:30.174 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:08:30.174 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:08:30.174 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:08:30.175 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:08:30.175 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:08:30.175 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:08:30.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:30.176 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:08:30.176 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T18:08:30.176 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:08:30.177 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T18:08:30.177 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:08:30.177 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:08:30.177 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:08:30.178 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:08:30.178 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:08:30.178 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:08:30.178 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:08:30.178 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:08:30.179 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:08:30.179 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:08:30.179 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:08:30.179 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:08:30.180 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:08:30.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:08:30.180 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:08:30.180 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:08:30.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:30.181 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:08:30.181 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:08:30.181 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:08:30.181 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:08:30.181 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:08:30.182 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:08:30.182 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:08:30.182 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:08:30.183 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:08:30.183 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:08:30.183 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:08:30.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.183 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:08:30.184 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:08:30.184 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:08:30.184 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:08:30.184 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:08:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T18:08:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:08:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T18:08:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:30.186 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:08:30.186 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T18:08:30.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:30.187 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:08:30.187 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T18:08:30.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:08:30.188 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:08:30.188 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T18:08:30.188 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:08:30.188 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:08:30.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.189 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T18:08:30.189 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:08:30.189 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T18:08:30.189 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T18:08:30.190 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:08:30.190 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:08:30.190 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:08:30.190 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:08:30.191 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:08:30.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.191 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T18:08:30.191 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:08:30.191 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:08:30.192 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:08:30.192 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:08:30.192 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:08:30.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.192 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T18:08:30.193 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:08:30.193 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:08:30.193 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:08:30.193 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:08:30.194 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:08:30.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.194 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T18:08:30.194 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:08:30.194 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:08:30.195 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:08:30.195 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:08:30.195 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:08:30.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.196 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T18:08:30.196 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:08:30.196 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:08:30.196 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:08:30.196 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:08:30.197 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:08:30.197 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:08:30.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:08:30.197 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:08:30.197 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T18:08:30.198 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:08:30.198 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:08:30.199 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2023-11-02T18:09:30.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:09:30.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:09:30.202 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:09:30.249 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:09:30.249 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:09:30.293 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:09:30.294 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:09:30.294 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:09:30.294 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "a2.rgw.meta:root", 2023-11-02T18:09:30.295 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "a2.rgw.control", 2023-11-02T18:09:30.295 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "a2.rgw.log:gc", 2023-11-02T18:09:30.295 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "a2.rgw.log:lc", 2023-11-02T18:09:30.295 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "a2.rgw.log", 2023-11-02T18:09:30.295 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-11-02T18:09:30.296 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-11-02T18:09:30.296 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-11-02T18:09:30.296 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-11-02T18:09:30.296 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-11-02T18:09:30.296 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-11-02T18:09:30.297 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-11-02T18:09:30.297 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-11-02T18:09:30.297 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "a2.rgw.otp", 2023-11-02T18:09:30.297 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2023-11-02T18:09:30.297 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T18:09:30.298 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T18:09:30.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.298 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2023-11-02T18:09:30.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.299 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2023-11-02T18:09:30.299 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:09:30.299 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "a2.rgw.buckets.index", 2023-11-02T18:09:30.299 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2023-11-02T18:09:30.299 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2023-11-02T18:09:30.300 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "a2.rgw.buckets.data", 2023-11-02T18:09:30.300 INFO:teuthology.orchestra.run.smithi017.stdout: "compression_type": "random" 2023-11-02T18:09:30.300 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:09:30.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.301 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-11-02T18:09:30.301 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0, 2023-11-02T18:09:30.301 INFO:teuthology.orchestra.run.smithi017.stdout: "inline_data": true 2023-11-02T18:09:30.301 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:09:30.301 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:09:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:09:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:09:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "a2.rgw.log:notif" 2023-11-02T18:09:30.302 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:09:30.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:09:30.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:09:30.304 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:09:30.348 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:09:30.349 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:09:30.443 INFO:teuthology.orchestra.run.smithi017.stderr:Sending period to new master zone d61b2bea-59a8-4454-8491-ac3afbd04937 2023-11-02T18:09:30.493 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:09:30.493 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:09:30.493 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 8, 2023-11-02T18:09:30.494 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T18:09:30.494 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T18:09:30.494 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T18:09:30.494 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:09:30.494 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T18:09:30.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.495 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T18:09:30.495 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T18:09:30.495 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:09:30.496 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T18:09:30.496 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:09:30.496 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:09:30.496 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:09:30.497 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:09:30.497 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:09:30.497 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:09:30.497 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:09:30.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.498 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:09:30.498 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:09:30.498 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:09:30.498 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T18:09:30.499 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:09:30.499 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:09:30.499 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:09:30.499 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:09:30.500 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:09:30.500 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:09:30.500 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:09:30.500 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:09:30.500 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:09:30.501 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:09:30.501 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:09:30.501 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:09:30.501 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:09:30.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.502 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:09:30.502 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T18:09:30.502 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:09:30.503 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:09:30.503 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:09:30.503 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:09:30.503 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:09:30.503 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:09:30.504 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:09:30.504 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:09:30.504 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:09:30.504 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:09:30.505 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:09:30.505 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:09:30.505 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:09:30.505 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:09:30.505 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:09:30.506 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:09:30.506 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:09:30.506 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:09:30.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.507 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:09:30.507 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:09:30.507 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:09:30.507 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:09:30.507 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:09:30.508 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:09:30.508 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:09:30.508 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:09:30.508 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:09:30.508 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:09:30.509 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:09:30.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.509 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:09:30.509 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:09:30.510 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:09:30.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.510 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T18:09:30.510 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T18:09:30.511 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:09:30.511 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T18:09:30.511 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:09:30.511 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:09:30.511 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:09:30.512 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:09:30.512 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:09:30.512 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:09:30.512 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:09:30.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.513 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:09:30.513 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T18:09:30.513 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:09:30.513 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:09:30.514 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:09:30.514 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:09:30.514 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:09:30.514 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:09:30.515 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:09:30.515 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:09:30.515 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:09:30.515 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:09:30.515 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:09:30.516 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:09:30.516 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:09:30.517 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:09:30.517 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:09:30.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.518 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:09:30.518 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T18:09:30.518 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:09:30.519 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T18:09:30.519 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:09:30.519 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:09:30.519 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:09:30.520 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:09:30.520 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:09:30.520 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:09:30.520 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:09:30.520 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:09:30.521 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:09:30.521 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:09:30.521 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:09:30.521 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:09:30.522 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:09:30.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:09:30.522 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:09:30.522 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:09:30.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.523 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:09:30.523 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:09:30.523 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:09:30.523 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:09:30.524 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:09:30.524 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:09:30.524 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:09:30.524 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:09:30.524 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:09:30.525 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:09:30.525 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:09:30.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.525 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:09:30.525 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:09:30.526 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:09:30.526 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:09:30.526 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:09:30.526 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T18:09:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:09:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T18:09:30.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.529 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:09:30.529 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T18:09:30.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.530 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:09:30.530 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T18:09:30.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:09:30.531 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:09:30.531 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T18:09:30.531 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:09:30.532 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:09:30.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.532 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T18:09:30.532 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:09:30.533 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T18:09:30.533 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T18:09:30.533 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:09:30.533 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:09:30.533 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:09:30.534 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:09:30.534 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:09:30.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.534 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T18:09:30.535 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:09:30.535 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:09:30.535 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:09:30.535 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:09:30.535 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:09:30.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.536 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T18:09:30.536 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:09:30.536 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:09:30.537 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:09:30.537 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:09:30.537 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:09:30.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.537 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T18:09:30.538 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:09:30.538 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:09:30.538 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:09:30.538 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:09:30.539 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:09:30.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.539 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T18:09:30.539 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:09:30.539 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:09:30.540 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:09:30.540 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:09:30.540 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:09:30.540 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:09:30.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:09:30.541 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:09:30.541 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T18:09:30.541 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:09:30.541 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:09:30.542 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2023-11-02T18:10:30.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2023-11-02T18:10:30.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:30.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:30.547 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:10:30.587 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:30.587 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:30.641 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:10:30.670 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:10:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:10:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:10:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:10:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:10:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:10:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:06:27.133683Z" 2023-11-02T18:10:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:10:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:10:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:30.674 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:10:30.675 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:10:30.676 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:10:30.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:30.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:30.678 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:10:30.724 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:30.725 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:30.775+0000 7f6dc822df40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:10:30.785 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:30.775+0000 7f6dc822df40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:10:30.810 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:30.810 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:10:30.811 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:10:30.811 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:10:30.811 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:10:30.811 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:10:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:10:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:10:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:10:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:10:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:10:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:10:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:10:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:10:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:10:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:10:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:10:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:10:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:10:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:10:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:10:30.824 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:10:30.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:30.824 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:10:30Z" 2023-11-02T18:10:30.824 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:30.825 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:10:30Z" } 2023-11-02T18:10:30.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:10:30.826 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:10:30.827 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:10:30.827 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:10:30.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:30.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'J46L5YZ9VLF35XGJ', '--secret', 'u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:30.829 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key J46L5YZ9VLF35XGJ --secret u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:30.877 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:30.877 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:30.905 INFO:teuthology.orchestra.run.smithi017.stderr:NOTICE: set zone's realm_id=a2bfe17b-2975-4441-84f6-480a071e229b 2023-11-02T18:10:30.945 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:30.946 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "7e35c7ad-8c13-454f-85bc-ef20f4df8d98", 2023-11-02T18:10:30.946 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "remove", 2023-11-02T18:10:30.946 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "remove.rgw.meta:root", 2023-11-02T18:10:30.946 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "remove.rgw.control", 2023-11-02T18:10:30.947 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "remove.rgw.log:gc", 2023-11-02T18:10:30.947 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "remove.rgw.log:lc", 2023-11-02T18:10:30.947 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "remove.rgw.log", 2023-11-02T18:10:30.947 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-11-02T18:10:30.947 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-11-02T18:10:30.948 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-11-02T18:10:30.948 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-11-02T18:10:30.948 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-11-02T18:10:30.948 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-11-02T18:10:30.949 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-11-02T18:10:30.949 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-11-02T18:10:30.949 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "remove.rgw.otp", 2023-11-02T18:10:30.949 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2023-11-02T18:10:30.949 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "J46L5YZ9VLF35XGJ", 2023-11-02T18:10:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "u7Al2pOXQLcd6OiWDAKPt6bmwLx3qop9" 2023-11-02T18:10:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2023-11-02T18:10:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2023-11-02T18:10:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "remove.rgw.buckets.index", 2023-11-02T18:10:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2023-11-02T18:10:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2023-11-02T18:10:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "remove.rgw.buckets.data" 2023-11-02T18:10:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-11-02T18:10:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0, 2023-11-02T18:10:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: "inline_data": true 2023-11-02T18:10:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "remove.rgw.log:notif" 2023-11-02T18:10:30.954 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:30.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:30.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:30.956 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:31.000 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:31.001 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:31.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:31.015+0000 7f51719ecf40 0 period (b3a94447-46df-4768-8340-cf63e4392a5c does not have zone 7e35c7ad-8c13-454f-85bc-ef20f4df8d98 configured 2023-11-02T18:10:37.304 INFO:teuthology.orchestra.run.smithi017.stderr:Sending period to new master zone d61b2bea-59a8-4454-8491-ac3afbd04937 2023-11-02T18:10:37.351 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:37.352 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:37.352 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 9, 2023-11-02T18:10:37.352 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T18:10:37.352 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T18:10:37.353 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T18:10:37.353 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:37.353 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T18:10:37.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.353 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T18:10:37.354 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T18:10:37.354 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:10:37.354 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T18:10:37.354 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.355 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:10:37.355 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.355 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:10:37.355 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:10:37.355 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:37.356 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:10:37.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.356 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:10:37.356 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:10:37.357 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.357 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T18:10:37.357 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.357 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:37.358 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:37.358 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:37.358 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:37.358 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:37.358 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:37.359 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:37.359 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:37.359 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:37.359 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:37.360 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.360 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.360 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "7e35c7ad-8c13-454f-85bc-ef20f4df8d98", 2023-11-02T18:10:37.361 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "remove", 2023-11-02T18:10:37.361 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2023-11-02T18:10:37.361 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:37.361 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:37.361 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:37.362 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:37.362 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:37.362 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:37.362 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:37.363 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:37.363 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:37.363 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:37.363 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.363 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.364 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:37.364 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T18:10:37.364 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.365 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:10:37.365 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.365 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:37.365 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:37.366 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:37.366 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:37.366 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:37.366 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:37.366 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:37.367 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:37.367 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:37.367 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:37.367 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.368 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.368 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.368 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.368 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:10:37.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.369 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:10:37.369 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:10:37.369 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:10:37.370 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:10:37.370 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.370 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.370 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.370 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:10:37.371 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:37.371 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:10:37.371 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:10:37.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.371 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:10:37.372 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.372 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.373 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T18:10:37.373 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T18:10:37.373 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:10:37.373 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T18:10:37.373 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.374 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:10:37.374 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.374 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:10:37.374 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:10:37.375 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:10:37.375 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:10:37.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.375 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:10:37.375 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T18:10:37.376 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.376 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:10:37.376 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.376 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:37.377 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:37.377 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:37.377 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:37.377 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:37.377 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:37.378 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:37.378 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:37.378 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:37.378 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:37.379 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.379 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.379 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:10:37.380 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T18:10:37.380 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.380 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T18:10:37.380 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.381 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:37.381 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:37.381 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:37.381 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:37.382 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:37.382 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:37.382 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:37.382 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:37.382 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:37.383 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:37.383 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.383 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.383 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.383 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.384 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:10:37.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.384 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:10:37.384 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:10:37.384 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:10:37.385 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:10:37.385 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.385 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.385 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.386 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:10:37.386 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:37.386 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:10:37.386 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:10:37.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.387 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:10:37.387 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.387 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.387 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.387 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.388 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T18:10:37.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.388 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:10:37.388 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T18:10:37.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.389 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "7e35c7ad-8c13-454f-85bc-ef20f4df8d98", 2023-11-02T18:10:37.389 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2028723067 2023-11-02T18:10:37.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:10:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T18:10:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T18:10:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:10:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T18:10:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T18:10:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T18:10:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T18:10:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:10:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:10:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:10:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:10:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:10:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T18:10:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:10:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:10:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:10:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:10:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:10:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T18:10:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:10:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:10:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:10:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:10:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:10:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T18:10:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:10:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:10:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:10:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:10:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:10:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T18:10:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:10:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:10:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:10:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:10:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:10:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T18:10:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:37.403 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:37.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:37.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:37.405 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:37.499 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:37.500 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:37.544 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:37.545 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T18:10:37.545 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T18:10:37.545 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:10:37.545 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T18:10:37.545 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.546 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:10:37.546 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.546 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:10:37.546 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:10:37.547 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:37.547 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:10:37.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.547 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:10:37.548 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:10:37.548 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.548 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T18:10:37.548 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.549 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:37.549 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:37.549 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:37.549 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:37.549 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:37.550 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:37.550 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:37.550 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:37.550 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:37.551 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:37.551 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.551 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.552 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:37.552 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T18:10:37.552 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.552 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:10:37.552 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.553 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:37.553 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:37.553 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:37.553 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:37.554 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:37.554 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:37.554 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:37.554 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:37.554 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:37.555 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:37.555 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.555 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.555 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.555 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.556 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:10:37.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.556 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:10:37.556 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:10:37.557 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:10:37.557 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:10:37.557 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.557 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.558 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:10:37.558 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:37.558 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:10:37.558 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:10:37.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.559 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:10:37.559 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.559 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.559 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:37.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:37.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:37.562 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:37.611 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:37.612 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:37.654 INFO:teuthology.orchestra.run.smithi017.stderr:zone name remove not found in zonegroup a 2023-11-02T18:10:37.665 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-02T18:10:37.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:37.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:37.667 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:37.713 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:37.713 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:37.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:37.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:37.773 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:37.865 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:37.866 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:37.889 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:37.879+0000 7fde53c1cf40 0 WARNING: can't generate connection for zone 7e35c7ad-8c13-454f-85bc-ef20f4df8d98 id remove: no endpoints defined 2023-11-02T18:10:37.952 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:37.952 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:37.952 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10, 2023-11-02T18:10:37.953 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T18:10:37.953 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T18:10:37.953 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T18:10:37.953 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:37.954 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T18:10:37.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.954 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T18:10:37.954 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T18:10:37.954 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:10:37.955 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T18:10:37.955 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.955 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:10:37.955 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.956 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:10:37.956 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:10:37.956 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:37.956 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:10:37.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.957 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:10:37.957 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:10:37.957 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.957 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T18:10:37.958 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.958 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:37.958 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:37.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:37.958 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:37.959 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:37.959 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:37.959 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:37.959 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:37.960 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:37.960 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:37.960 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.960 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.961 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:37.961 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T18:10:37.962 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.962 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:10:37.962 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.962 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:37.963 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:37.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:37.963 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:37.963 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:37.963 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:37.964 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:37.964 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:37.964 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:37.964 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:37.965 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.965 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.965 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.965 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:10:37.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.966 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:10:37.966 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:10:37.966 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:10:37.967 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:10:37.967 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.967 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.967 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.967 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:10:37.968 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:37.968 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:10:37.968 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:10:37.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.968 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:10:37.969 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.969 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.970 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T18:10:37.970 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T18:10:37.970 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:10:37.970 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T18:10:37.970 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.971 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:10:37.971 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.971 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:10:37.971 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:10:37.971 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:10:37.972 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:10:37.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.972 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:10:37.972 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T18:10:37.973 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.973 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:10:37.973 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.973 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:37.974 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:37.974 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:37.974 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:37.974 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:37.974 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:37.975 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:37.975 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:37.975 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:37.975 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:37.976 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.977 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.977 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:10:37.978 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T18:10:37.978 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:37.978 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T18:10:37.978 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.978 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:37.979 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:37.979 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:37.979 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:37.979 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:37.980 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:37.980 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:37.980 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:37.980 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:37.980 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:37.981 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.981 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.981 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.981 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.982 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:10:37.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.982 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:10:37.982 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:10:37.982 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:10:37.983 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:10:37.983 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.983 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.983 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.984 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:10:37.984 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:37.984 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:10:37.984 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:10:37.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.985 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:10:37.985 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:37.985 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.985 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.985 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:37.986 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T18:10:37.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.986 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:10:37.986 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T18:10:37.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.987 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:10:37.987 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T18:10:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T18:10:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:10:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T18:10:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T18:10:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T18:10:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T18:10:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:10:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:10:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:10:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:10:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:10:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T18:10:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:10:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:10:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:10:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:10:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:10:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T18:10:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:10:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:10:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:10:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:10:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:10:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T18:10:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:10:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:10:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:10:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:10:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:10:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T18:10:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:10:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:10:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:10:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:10:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:10:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T18:10:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:37.999 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:38.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2023-11-02T18:10:38.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.002 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:38.041 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:38.042 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:38.079 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "2c8c7c88-5a26-4395-8087-97ed91fad66d", 2023-11-02T18:10:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "remove_zg", 2023-11-02T18:10:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "remove_zg", 2023-11-02T18:10:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T18:10:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2023-11-02T18:10:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:10:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:10:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "", 2023-11-02T18:10:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [], 2023-11-02T18:10:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:10:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:10:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:10:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [] 2023-11-02T18:10:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:10:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:10:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:10:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:10:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.085 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:38.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.088 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:38.132 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:38.132 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:38.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:38.179+0000 7f6eac685f40 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2023-11-02T18:10:38.196 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:38.196 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "6215c1fb-d062-4f75-b083-2b8f6a4464fe", 2023-11-02T18:10:38.197 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "remove", 2023-11-02T18:10:38.197 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "remove.rgw.meta:root", 2023-11-02T18:10:38.197 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "remove.rgw.control", 2023-11-02T18:10:38.197 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "remove.rgw.log:gc", 2023-11-02T18:10:38.198 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "remove.rgw.log:lc", 2023-11-02T18:10:38.198 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "remove.rgw.log", 2023-11-02T18:10:38.198 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-11-02T18:10:38.198 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-11-02T18:10:38.198 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-11-02T18:10:38.199 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-11-02T18:10:38.199 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-11-02T18:10:38.199 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-11-02T18:10:38.199 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-11-02T18:10:38.199 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-11-02T18:10:38.200 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "remove.rgw.otp", 2023-11-02T18:10:38.200 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2023-11-02T18:10:38.200 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "", 2023-11-02T18:10:38.200 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "" 2023-11-02T18:10:38.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.201 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2023-11-02T18:10:38.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.201 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2023-11-02T18:10:38.201 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:38.202 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "remove.rgw.buckets.index", 2023-11-02T18:10:38.202 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2023-11-02T18:10:38.202 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2023-11-02T18:10:38.202 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "remove.rgw.buckets.data" 2023-11-02T18:10:38.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.203 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-11-02T18:10:38.203 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0, 2023-11-02T18:10:38.204 INFO:teuthology.orchestra.run.smithi017.stdout: "inline_data": true 2023-11-02T18:10:38.204 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.204 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.204 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.204 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:38.205 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "remove.rgw.log:notif" 2023-11-02T18:10:38.205 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:38.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.207 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:38.263 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:38.264 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:38.374 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:38.374 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:38.374 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 11, 2023-11-02T18:10:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T18:10:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T18:10:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T18:10:38.376 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:38.376 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T18:10:38.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.376 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "2c8c7c88-5a26-4395-8087-97ed91fad66d", 2023-11-02T18:10:38.376 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "remove_zg", 2023-11-02T18:10:38.377 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "remove_zg", 2023-11-02T18:10:38.377 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T18:10:38.377 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2023-11-02T18:10:38.377 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:10:38.378 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:10:38.378 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "6215c1fb-d062-4f75-b083-2b8f6a4464fe", 2023-11-02T18:10:38.378 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:10:38.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.379 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "6215c1fb-d062-4f75-b083-2b8f6a4464fe", 2023-11-02T18:10:38.379 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "remove", 2023-11-02T18:10:38.379 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2023-11-02T18:10:38.379 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:38.380 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": false, 2023-11-02T18:10:38.380 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:38.380 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:38.380 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:38.380 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:38.381 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:38.381 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:38.381 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:38.381 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:38.382 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.382 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.382 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.382 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.382 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:10:38.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.383 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:10:38.383 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:10:38.383 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:10:38.384 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:10:38.384 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.384 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.384 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.384 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:10:38.385 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:38.385 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:10:38.385 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:10:38.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.385 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:10:38.386 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.386 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.387 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T18:10:38.387 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T18:10:38.387 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:10:38.387 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T18:10:38.388 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:38.388 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:10:38.388 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.388 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:10:38.388 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:10:38.389 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:38.389 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:10:38.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.389 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:10:38.390 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:10:38.390 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:38.390 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T18:10:38.390 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.391 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:38.391 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:38.391 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:38.391 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:38.392 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:38.392 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:38.392 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:38.392 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:38.392 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:38.393 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:38.393 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.393 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.394 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:38.394 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T18:10:38.394 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:38.395 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:10:38.395 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.395 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:38.395 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:38.395 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:38.396 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:38.396 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:38.396 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:38.396 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:38.397 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:38.397 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:38.397 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:38.397 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.398 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.398 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.398 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.398 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:10:38.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.399 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:10:38.399 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:10:38.399 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:10:38.399 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:10:38.399 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.400 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.400 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.400 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:10:38.400 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:38.401 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:10:38.401 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:10:38.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.401 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:10:38.401 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.402 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.402 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T18:10:38.403 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T18:10:38.403 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:10:38.403 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T18:10:38.403 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:38.403 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:10:38.404 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.404 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:10:38.404 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:10:38.404 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:10:38.405 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:10:38.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.405 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:10:38.405 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T18:10:38.405 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:38.406 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:10:38.406 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.406 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:38.406 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:38.407 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:38.407 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:38.407 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:38.407 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:38.407 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:38.408 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:38.408 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:38.408 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:38.408 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.408 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.409 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:10:38.409 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T18:10:38.409 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:38.410 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T18:10:38.410 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.410 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:38.410 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:38.411 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:38.411 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:38.411 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:38.411 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:38.411 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:38.411 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:38.412 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:38.412 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:38.412 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.412 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.412 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.413 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.413 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:10:38.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.413 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:10:38.414 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:10:38.414 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:10:38.414 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:10:38.414 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.415 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.415 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.416 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:10:38.416 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:38.417 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:10:38.417 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:10:38.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.418 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:10:38.418 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.418 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.418 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.419 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.419 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T18:10:38.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.420 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:10:38.420 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T18:10:38.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.421 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "6215c1fb-d062-4f75-b083-2b8f6a4464fe", 2023-11-02T18:10:38.421 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3714432532 2023-11-02T18:10:38.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.422 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:10:38.423 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T18:10:38.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.423 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:38.424 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T18:10:38.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.424 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:10:38.425 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T18:10:38.425 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.425 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.425 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T18:10:38.426 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:38.426 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T18:10:38.427 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T18:10:38.427 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:10:38.428 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:10:38.428 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:10:38.428 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:10:38.428 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:10:38.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.429 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T18:10:38.429 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:10:38.429 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:10:38.429 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:10:38.430 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:10:38.430 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:10:38.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.431 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T18:10:38.432 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:10:38.432 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:10:38.432 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:10:38.432 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:10:38.432 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:10:38.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.433 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T18:10:38.433 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:10:38.433 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:10:38.434 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:10:38.434 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:10:38.434 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:10:38.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.434 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T18:10:38.435 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:10:38.435 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:10:38.435 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:10:38.435 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:10:38.435 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:10:38.436 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.436 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:38.436 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T18:10:38.437 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:38.437 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:38.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.438 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:38.481 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:38.481 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:38.515+0000 7f59242aef40 0 NOTICE: removed master_zone of zonegroup remove_zg 2023-11-02T18:10:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:38.515+0000 7f59242aef40 0 Removed zone from zonegroup remove_zg 2023-11-02T18:10:38.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.541 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:38.585 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:38.585 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:38.624 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:38.615+0000 7f0f716d8f40 0 ERROR: zonegroup remove_zg has a non existent master zone 2023-11-02T18:10:38.633 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-02T18:10:38.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.635 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:38.716 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:38.717 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:38.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:10:38.751 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:10:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:38.923 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 12, 2023-11-02T18:10:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T18:10:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T18:10:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T18:10:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T18:10:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T18:10:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T18:10:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:10:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T18:10:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:10:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:10:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:10:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:10:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:10:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T18:10:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T18:10:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T18:10:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T18:10:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:10:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:10:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:10:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:10:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:10:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:10:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:10:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:10:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:10:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T18:10:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T18:10:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T18:10:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T18:10:38.942 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:38.942 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:10:38.942 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.942 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T18:10:38.943 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T18:10:38.943 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:10:38.943 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T18:10:38.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.944 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:10:38.944 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T18:10:38.944 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:38.944 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T18:10:38.944 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:38.947 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:10:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T18:10:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T18:10:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T18:10:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T18:10:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T18:10:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T18:10:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T18:10:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T18:10:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T18:10:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T18:10:38.952 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T18:10:38.952 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T18:10:38.952 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T18:10:38.952 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.953 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.953 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.953 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T18:10:38.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T18:10:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T18:10:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T18:10:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T18:10:38.955 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.955 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.955 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.955 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T18:10:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T18:10:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T18:10:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T18:10:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T18:10:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T18:10:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T18:10:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:10:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T18:10:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T18:10:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T18:10:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T18:10:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T18:10:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T18:10:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T18:10:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:10:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T18:10:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T18:10:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:10:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:10:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:10:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:10:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:10:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T18:10:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T18:10:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T18:10:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T18:10:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T18:10:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T18:10:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T18:10:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:10:38.966 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:10:38.966 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:10:38.966 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:10:38.966 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:10:38.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T18:10:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:10:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:10:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:10:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:10:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:10:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T18:10:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T18:10:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T18:10:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T18:10:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T18:10:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T18:10:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T18:10:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T18:10:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:38.971 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:38.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2023-11-02T18:10:38.972 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-36 2023-11-02T18:10:38.972 DEBUG:boto:path=/ibtljn-36/ 2023-11-02T18:10:38.973 DEBUG:boto:auth_path=/ibtljn-36/ 2023-11-02T18:10:38.973 DEBUG:boto:Method: PUT 2023-11-02T18:10:38.974 DEBUG:boto:Path: /ibtljn-36/ 2023-11-02T18:10:38.974 DEBUG:boto:Data: 2023-11-02T18:10:38.975 DEBUG:boto:Headers: {} 2023-11-02T18:10:38.975 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:10:38.975 DEBUG:boto:Port: 8000 2023-11-02T18:10:38.976 DEBUG:boto:Params: {} 2023-11-02T18:10:38.977 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:10:38.977 DEBUG:boto:Token: None 2023-11-02T18:10:38.978 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:38 GMT /ibtljn-36/ 2023-11-02T18:10:38.978 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ehd05AKBPI2l2+BjnnRst4fz2Jw= 2023-11-02T18:10:38.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ehd05AKBPI2l2+BjnnRst4fz2Jw=', 'Content-Length': '0'} 2023-11-02T18:10:39.459 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004f9bd595c15895f7-006543e61f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:39.460 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-37 2023-11-02T18:10:39.460 DEBUG:boto:path=/ibtljn-37/ 2023-11-02T18:10:39.460 DEBUG:boto:auth_path=/ibtljn-37/ 2023-11-02T18:10:39.461 DEBUG:boto:Method: PUT 2023-11-02T18:10:39.461 DEBUG:boto:Path: /ibtljn-37/ 2023-11-02T18:10:39.462 DEBUG:boto:Data: 2023-11-02T18:10:39.462 DEBUG:boto:Headers: {} 2023-11-02T18:10:39.463 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:10:39.463 DEBUG:boto:Port: 8001 2023-11-02T18:10:39.463 DEBUG:boto:Params: {} 2023-11-02T18:10:39.464 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:10:39.464 DEBUG:boto:Token: None 2023-11-02T18:10:39.465 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:39 GMT /ibtljn-37/ 2023-11-02T18:10:39.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AClDmcHi+8I+zOEwyebdXagjGkU= 2023-11-02T18:10:39.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AClDmcHi+8I+zOEwyebdXagjGkU=', 'Content-Length': '0'} 2023-11-02T18:10:39.512 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000efdd62ce1f91c1b8-006543e61f-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:39.513 INFO:rgw_multi.tests:create bucket zone=b1 name=ibtljn-38 2023-11-02T18:10:39.513 DEBUG:boto:path=/ibtljn-38/ 2023-11-02T18:10:39.514 DEBUG:boto:auth_path=/ibtljn-38/ 2023-11-02T18:10:39.514 DEBUG:boto:Method: PUT 2023-11-02T18:10:39.514 DEBUG:boto:Path: /ibtljn-38/ 2023-11-02T18:10:39.515 DEBUG:boto:Data: 2023-11-02T18:10:39.515 DEBUG:boto:Headers: {} 2023-11-02T18:10:39.516 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:10:39.516 DEBUG:boto:Port: 8000 2023-11-02T18:10:39.517 DEBUG:boto:Params: {} 2023-11-02T18:10:39.517 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:10:39.518 DEBUG:boto:Token: None 2023-11-02T18:10:39.518 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:39 GMT /ibtljn-38/ 2023-11-02T18:10:39.519 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A3YLXASHjX6Gf+DGje/ceRdI000= 2023-11-02T18:10:39.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A3YLXASHjX6Gf+DGje/ceRdI000=', 'Content-Length': '0'} 2023-11-02T18:10:39.563 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a775dcafb8025230-006543e61f-116b-b1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:39.564 INFO:rgw_multi.tests:create bucket zone=b2 name=ibtljn-39 2023-11-02T18:10:39.564 DEBUG:boto:path=/ibtljn-39/ 2023-11-02T18:10:39.564 DEBUG:boto:auth_path=/ibtljn-39/ 2023-11-02T18:10:39.565 DEBUG:boto:Method: PUT 2023-11-02T18:10:39.565 DEBUG:boto:Path: /ibtljn-39/ 2023-11-02T18:10:39.566 DEBUG:boto:Data: 2023-11-02T18:10:39.566 DEBUG:boto:Headers: {} 2023-11-02T18:10:39.567 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:10:39.567 DEBUG:boto:Port: 8001 2023-11-02T18:10:39.567 DEBUG:boto:Params: {} 2023-11-02T18:10:39.568 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:10:39.568 DEBUG:boto:Token: None 2023-11-02T18:10:39.569 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:39 GMT /ibtljn-39/ 2023-11-02T18:10:39.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vN4vEVaVsG89JhcCeiBHIKiVMys= 2023-11-02T18:10:39.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vN4vEVaVsG89JhcCeiBHIKiVMys=', 'Content-Length': '0'} 2023-11-02T18:10:39.614 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a9bdba8ca7508d69-006543e61f-116d-b2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:39.615 DEBUG:boto:path=/ibtljn-36/ 2023-11-02T18:10:39.615 DEBUG:boto:auth_path=/ibtljn-36/ 2023-11-02T18:10:39.615 DEBUG:boto:path=/ibtljn-36/?website 2023-11-02T18:10:39.616 DEBUG:boto:auth_path=/ibtljn-36/?website 2023-11-02T18:10:39.616 DEBUG:boto:Method: PUT 2023-11-02T18:10:39.617 DEBUG:boto:Path: /ibtljn-36/?website 2023-11-02T18:10:39.617 DEBUG:boto:Data: index.htmlerror.html 2023-11-02T18:10:39.617 DEBUG:boto:Headers: {} 2023-11-02T18:10:39.618 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:10:39.618 DEBUG:boto:Port: 8000 2023-11-02T18:10:39.619 DEBUG:boto:Params: {} 2023-11-02T18:10:39.619 DEBUG:boto:Token: None 2023-11-02T18:10:39.620 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:39 GMT /ibtljn-36/?website 2023-11-02T18:10:39.620 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:U3JvxYMFIWNxzKS/UMSSaDu8MEk= 2023-11-02T18:10:39.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:U3JvxYMFIWNxzKS/UMSSaDu8MEk=', 'Content-Length': '241'} 2023-11-02T18:10:39.622 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx00000043566708adb73fc-006543e61f-1669-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:39.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2023-11-02T18:10:39.623 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-40 2023-11-02T18:10:39.624 DEBUG:boto:path=/ibtljn-40/ 2023-11-02T18:10:39.624 DEBUG:boto:auth_path=/ibtljn-40/ 2023-11-02T18:10:39.625 DEBUG:boto:Method: PUT 2023-11-02T18:10:39.625 DEBUG:boto:Path: /ibtljn-40/ 2023-11-02T18:10:39.625 DEBUG:boto:Data: 2023-11-02T18:10:39.626 DEBUG:boto:Headers: {} 2023-11-02T18:10:39.626 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:10:39.627 DEBUG:boto:Port: 8000 2023-11-02T18:10:39.627 DEBUG:boto:Params: {} 2023-11-02T18:10:39.628 DEBUG:boto:Token: None 2023-11-02T18:10:39.628 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:39 GMT /ibtljn-40/ 2023-11-02T18:10:39.628 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1Hbsqcee7oNRUhPU3fOSgW8wGg0= 2023-11-02T18:10:39.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1Hbsqcee7oNRUhPU3fOSgW8wGg0=', 'Content-Length': '0'} 2023-11-02T18:10:39.652 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff7d7a9b90071d4c-006543e61f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:39.652 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-41 2023-11-02T18:10:39.653 DEBUG:boto:path=/ibtljn-41/ 2023-11-02T18:10:39.653 DEBUG:boto:auth_path=/ibtljn-41/ 2023-11-02T18:10:39.654 DEBUG:boto:Method: PUT 2023-11-02T18:10:39.654 DEBUG:boto:Path: /ibtljn-41/ 2023-11-02T18:10:39.655 DEBUG:boto:Data: 2023-11-02T18:10:39.655 DEBUG:boto:Headers: {} 2023-11-02T18:10:39.655 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:10:39.656 DEBUG:boto:Port: 8001 2023-11-02T18:10:39.656 DEBUG:boto:Params: {} 2023-11-02T18:10:39.657 DEBUG:boto:Token: None 2023-11-02T18:10:39.657 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:39 GMT /ibtljn-41/ 2023-11-02T18:10:39.658 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7cbBsRZgALwPjHUTtW97GVUWbE8= 2023-11-02T18:10:39.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7cbBsRZgALwPjHUTtW97GVUWbE8=', 'Content-Length': '0'} 2023-11-02T18:10:39.695 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000073099253a6a6c4e0-006543e61f-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:39.695 INFO:rgw_multi.tests:create bucket zone=b1 name=ibtljn-42 2023-11-02T18:10:39.696 DEBUG:boto:path=/ibtljn-42/ 2023-11-02T18:10:39.696 DEBUG:boto:auth_path=/ibtljn-42/ 2023-11-02T18:10:39.697 DEBUG:boto:Method: PUT 2023-11-02T18:10:39.697 DEBUG:boto:Path: /ibtljn-42/ 2023-11-02T18:10:39.697 DEBUG:boto:Data: 2023-11-02T18:10:39.698 DEBUG:boto:Headers: {} 2023-11-02T18:10:39.698 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:10:39.699 DEBUG:boto:Port: 8000 2023-11-02T18:10:39.699 DEBUG:boto:Params: {} 2023-11-02T18:10:39.699 DEBUG:boto:Token: None 2023-11-02T18:10:39.700 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:39 GMT /ibtljn-42/ 2023-11-02T18:10:39.700 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gokzZ8+Xgvyj35RHzNirgfXY44Q= 2023-11-02T18:10:39.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gokzZ8+Xgvyj35RHzNirgfXY44Q=', 'Content-Length': '0'} 2023-11-02T18:10:39.741 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a24f17ace3fcd109-006543e61f-116b-b1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:39.742 INFO:rgw_multi.tests:create bucket zone=b2 name=ibtljn-43 2023-11-02T18:10:39.742 DEBUG:boto:path=/ibtljn-43/ 2023-11-02T18:10:39.743 DEBUG:boto:auth_path=/ibtljn-43/ 2023-11-02T18:10:39.743 DEBUG:boto:Method: PUT 2023-11-02T18:10:39.743 DEBUG:boto:Path: /ibtljn-43/ 2023-11-02T18:10:39.744 DEBUG:boto:Data: 2023-11-02T18:10:39.744 DEBUG:boto:Headers: {} 2023-11-02T18:10:39.745 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:10:39.745 DEBUG:boto:Port: 8001 2023-11-02T18:10:39.745 DEBUG:boto:Params: {} 2023-11-02T18:10:39.746 DEBUG:boto:Token: None 2023-11-02T18:10:39.746 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:39 GMT /ibtljn-43/ 2023-11-02T18:10:39.747 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rTpsSwwBwZRZpaSbIM+bb/RdicA= 2023-11-02T18:10:39.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rTpsSwwBwZRZpaSbIM+bb/RdicA=', 'Content-Length': '0'} 2023-11-02T18:10:39.787 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000041a757539c118ad-006543e61f-116d-b2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:39.788 DEBUG:boto:path=/ibtljn-40/ 2023-11-02T18:10:39.789 DEBUG:boto:auth_path=/ibtljn-40/ 2023-11-02T18:10:39.789 DEBUG:boto:path=/ibtljn-40/?policy 2023-11-02T18:10:39.790 DEBUG:boto:auth_path=/ibtljn-40/?policy 2023-11-02T18:10:39.790 DEBUG:boto:Method: PUT 2023-11-02T18:10:39.791 DEBUG:boto:Path: /ibtljn-40/?policy 2023-11-02T18:10:39.791 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-11-02T18:10:39.792 DEBUG:boto:Headers: {} 2023-11-02T18:10:39.792 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:10:39.792 DEBUG:boto:Port: 8000 2023-11-02T18:10:39.793 DEBUG:boto:Params: {} 2023-11-02T18:10:39.793 DEBUG:boto:Token: None 2023-11-02T18:10:39.794 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:39 GMT /ibtljn-40/?policy 2023-11-02T18:10:39.794 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1y/ACuL98xxTUuU1nsS5+5P22W0= 2023-11-02T18:10:39.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1y/ACuL98xxTUuU1nsS5+5P22W0=', 'Content-Length': '97'} 2023-11-02T18:10:39.802 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a6533480a7eda670-006543e61f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:39.803 DEBUG:boto:path=/ibtljn-40/ 2023-11-02T18:10:39.803 DEBUG:boto:auth_path=/ibtljn-40/ 2023-11-02T18:10:39.804 DEBUG:boto:path=/ibtljn-40/?policy 2023-11-02T18:10:39.804 DEBUG:boto:auth_path=/ibtljn-40/?policy 2023-11-02T18:10:39.804 DEBUG:boto:Method: GET 2023-11-02T18:10:39.805 DEBUG:boto:Path: /ibtljn-40/?policy 2023-11-02T18:10:39.805 DEBUG:boto:Data: 2023-11-02T18:10:39.806 DEBUG:boto:Headers: {} 2023-11-02T18:10:39.806 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:10:39.806 DEBUG:boto:Port: 8000 2023-11-02T18:10:39.807 DEBUG:boto:Params: {} 2023-11-02T18:10:39.807 DEBUG:boto:Token: None 2023-11-02T18:10:39.808 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:10:39 GMT /ibtljn-40/?policy 2023-11-02T18:10:39.808 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JIVImojuYXOFlRp9Vaa4lRfLKnw= 2023-11-02T18:10:39.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JIVImojuYXOFlRp9Vaa4lRfLKnw=', 'Content-Length': '0'} 2023-11-02T18:10:39.810 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b6c5b42a0066a9e-006543e61f-1669-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Thu, 02 Nov 2023 18:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:39.811 DEBUG:boto:path=/ibtljn-41/ 2023-11-02T18:10:39.811 DEBUG:boto:auth_path=/ibtljn-41/ 2023-11-02T18:10:39.812 DEBUG:boto:path=/ibtljn-41/?policy 2023-11-02T18:10:39.812 DEBUG:boto:auth_path=/ibtljn-41/?policy 2023-11-02T18:10:39.813 DEBUG:boto:Method: PUT 2023-11-02T18:10:39.813 DEBUG:boto:Path: /ibtljn-41/?policy 2023-11-02T18:10:39.813 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-11-02T18:10:39.814 DEBUG:boto:Headers: {} 2023-11-02T18:10:39.814 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:10:39.815 DEBUG:boto:Port: 8001 2023-11-02T18:10:39.815 DEBUG:boto:Params: {} 2023-11-02T18:10:39.815 DEBUG:boto:Token: None 2023-11-02T18:10:39.816 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:39 GMT /ibtljn-41/?policy 2023-11-02T18:10:39.816 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:laeXDstWNA6PG6t8Hcuhl2Ogwi0= 2023-11-02T18:10:39.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:laeXDstWNA6PG6t8Hcuhl2Ogwi0=', 'Content-Length': '97'} 2023-11-02T18:10:40.835 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006846ecdbefb6becc-006543e61f-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:40.836 DEBUG:boto:path=/ibtljn-41/ 2023-11-02T18:10:40.837 DEBUG:boto:auth_path=/ibtljn-41/ 2023-11-02T18:10:40.837 DEBUG:boto:path=/ibtljn-41/?policy 2023-11-02T18:10:40.837 DEBUG:boto:auth_path=/ibtljn-41/?policy 2023-11-02T18:10:40.838 DEBUG:boto:Method: GET 2023-11-02T18:10:40.838 DEBUG:boto:Path: /ibtljn-41/?policy 2023-11-02T18:10:40.839 DEBUG:boto:Data: 2023-11-02T18:10:40.839 DEBUG:boto:Headers: {} 2023-11-02T18:10:40.839 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:10:40.840 DEBUG:boto:Port: 8001 2023-11-02T18:10:40.840 DEBUG:boto:Params: {} 2023-11-02T18:10:40.841 DEBUG:boto:Token: None 2023-11-02T18:10:40.841 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:10:40 GMT /ibtljn-41/?policy 2023-11-02T18:10:40.842 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:72Z8Gm1xubocAUE+j7Q8XoV5hDA= 2023-11-02T18:10:40.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:72Z8Gm1xubocAUE+j7Q8XoV5hDA=', 'Content-Length': '0'} 2023-11-02T18:10:40.844 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aa7d1c27cd9515fb-006543e620-1665-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Thu, 02 Nov 2023 18:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:40.844 DEBUG:boto:path=/ibtljn-42/ 2023-11-02T18:10:40.845 DEBUG:boto:auth_path=/ibtljn-42/ 2023-11-02T18:10:40.845 DEBUG:boto:path=/ibtljn-42/?policy 2023-11-02T18:10:40.846 DEBUG:boto:auth_path=/ibtljn-42/?policy 2023-11-02T18:10:40.846 DEBUG:boto:Method: PUT 2023-11-02T18:10:40.847 DEBUG:boto:Path: /ibtljn-42/?policy 2023-11-02T18:10:40.847 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-11-02T18:10:40.848 DEBUG:boto:Headers: {} 2023-11-02T18:10:40.848 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:10:40.849 DEBUG:boto:Port: 8000 2023-11-02T18:10:40.849 DEBUG:boto:Params: {} 2023-11-02T18:10:40.849 DEBUG:boto:Token: None 2023-11-02T18:10:40.850 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:40 GMT /ibtljn-42/?policy 2023-11-02T18:10:40.850 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NJpG1IJOMIAuor+bH9TqOTr9Nc4= 2023-11-02T18:10:40.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NJpG1IJOMIAuor+bH9TqOTr9Nc4=', 'Content-Length': '97'} 2023-11-02T18:10:41.872 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000604c1619454ed93e-006543e620-116b-b1'), ('Date', 'Thu, 02 Nov 2023 18:10:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:41.872 DEBUG:boto:path=/ibtljn-42/ 2023-11-02T18:10:41.873 DEBUG:boto:auth_path=/ibtljn-42/ 2023-11-02T18:10:41.873 DEBUG:boto:path=/ibtljn-42/?policy 2023-11-02T18:10:41.874 DEBUG:boto:auth_path=/ibtljn-42/?policy 2023-11-02T18:10:41.874 DEBUG:boto:Method: GET 2023-11-02T18:10:41.874 DEBUG:boto:Path: /ibtljn-42/?policy 2023-11-02T18:10:41.875 DEBUG:boto:Data: 2023-11-02T18:10:41.875 DEBUG:boto:Headers: {} 2023-11-02T18:10:41.876 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8000 2023-11-02T18:10:41.876 DEBUG:boto:Port: 8000 2023-11-02T18:10:41.877 DEBUG:boto:Params: {} 2023-11-02T18:10:41.877 DEBUG:boto:Token: None 2023-11-02T18:10:41.878 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:10:41 GMT /ibtljn-42/?policy 2023-11-02T18:10:41.879 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JiN6cD332raBN9wROIkfMYvkW04= 2023-11-02T18:10:41.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JiN6cD332raBN9wROIkfMYvkW04=', 'Content-Length': '0'} 2023-11-02T18:10:41.882 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000000775f083822d270-006543e621-116b-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Thu, 02 Nov 2023 18:10:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:41.883 DEBUG:boto:path=/ibtljn-43/ 2023-11-02T18:10:41.883 DEBUG:boto:auth_path=/ibtljn-43/ 2023-11-02T18:10:41.884 DEBUG:boto:path=/ibtljn-43/?policy 2023-11-02T18:10:41.884 DEBUG:boto:auth_path=/ibtljn-43/?policy 2023-11-02T18:10:41.885 DEBUG:boto:Method: PUT 2023-11-02T18:10:41.886 DEBUG:boto:Path: /ibtljn-43/?policy 2023-11-02T18:10:41.886 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-11-02T18:10:41.887 DEBUG:boto:Headers: {} 2023-11-02T18:10:41.888 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:10:41.888 DEBUG:boto:Port: 8001 2023-11-02T18:10:41.889 DEBUG:boto:Params: {} 2023-11-02T18:10:41.889 DEBUG:boto:Token: None 2023-11-02T18:10:41.890 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:41 GMT /ibtljn-43/?policy 2023-11-02T18:10:41.890 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:auSl6c2qRfOpT5rn7NMWQtA6KVM= 2023-11-02T18:10:41.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:auSl6c2qRfOpT5rn7NMWQtA6KVM=', 'Content-Length': '97'} 2023-11-02T18:10:42.912 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000432311a95a2de14f-006543e621-116d-b2'), ('Date', 'Thu, 02 Nov 2023 18:10:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:42.913 DEBUG:boto:path=/ibtljn-43/ 2023-11-02T18:10:42.914 DEBUG:boto:auth_path=/ibtljn-43/ 2023-11-02T18:10:42.915 DEBUG:boto:path=/ibtljn-43/?policy 2023-11-02T18:10:42.915 DEBUG:boto:auth_path=/ibtljn-43/?policy 2023-11-02T18:10:42.916 DEBUG:boto:Method: GET 2023-11-02T18:10:42.916 DEBUG:boto:Path: /ibtljn-43/?policy 2023-11-02T18:10:42.917 DEBUG:boto:Data: 2023-11-02T18:10:42.918 DEBUG:boto:Headers: {} 2023-11-02T18:10:42.918 DEBUG:boto:Host: smithi069.front.sepia.ceph.com:8001 2023-11-02T18:10:42.919 DEBUG:boto:Port: 8001 2023-11-02T18:10:42.919 DEBUG:boto:Params: {} 2023-11-02T18:10:42.920 DEBUG:boto:Token: None 2023-11-02T18:10:42.920 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:10:42 GMT /ibtljn-43/?policy 2023-11-02T18:10:42.921 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Spl/79yNW9wuD0SLU0ZEgj9Abaw= 2023-11-02T18:10:42.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Spl/79yNW9wuD0SLU0ZEgj9Abaw=', 'Content-Length': '0'} 2023-11-02T18:10:42.924 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000023b53df391182acf-006543e622-116d-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Thu, 02 Nov 2023 18:10:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:42.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2023-11-02T18:10:42.925 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-44 2023-11-02T18:10:42.926 DEBUG:boto:path=/ibtljn-44/ 2023-11-02T18:10:42.927 DEBUG:boto:auth_path=/ibtljn-44/ 2023-11-02T18:10:42.927 DEBUG:boto:Method: PUT 2023-11-02T18:10:42.928 DEBUG:boto:Path: /ibtljn-44/ 2023-11-02T18:10:42.929 DEBUG:boto:Data: 2023-11-02T18:10:42.929 DEBUG:boto:Headers: {} 2023-11-02T18:10:42.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:10:42.930 DEBUG:boto:Port: 8000 2023-11-02T18:10:42.931 DEBUG:boto:Params: {} 2023-11-02T18:10:42.931 DEBUG:boto:Token: None 2023-11-02T18:10:42.932 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:42 GMT /ibtljn-44/ 2023-11-02T18:10:42.932 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kFyOindPqhvag0CC/ueOjwvbnVc= 2023-11-02T18:10:42.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kFyOindPqhvag0CC/ueOjwvbnVc=', 'Content-Length': '0'} 2023-11-02T18:10:42.965 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000953546a2648df710-006543e622-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:10:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:42.966 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-45 2023-11-02T18:10:42.967 DEBUG:boto:path=/ibtljn-45/ 2023-11-02T18:10:42.968 DEBUG:boto:auth_path=/ibtljn-45/ 2023-11-02T18:10:42.968 DEBUG:boto:Method: PUT 2023-11-02T18:10:42.969 DEBUG:boto:Path: /ibtljn-45/ 2023-11-02T18:10:42.970 DEBUG:boto:Data: 2023-11-02T18:10:42.971 DEBUG:boto:Headers: {} 2023-11-02T18:10:42.971 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:10:42.971 DEBUG:boto:Port: 8001 2023-11-02T18:10:42.972 DEBUG:boto:Params: {} 2023-11-02T18:10:42.973 DEBUG:boto:Token: None 2023-11-02T18:10:42.973 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:10:42 GMT /ibtljn-45/ 2023-11-02T18:10:42.974 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/3Wc1OQiDskKNTjrpC6eu1RhfLU= 2023-11-02T18:10:42.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:10:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/3Wc1OQiDskKNTjrpC6eu1RhfLU=', 'Content-Length': '0'} 2023-11-02T18:10:43.022 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000cc9a72e62d76db5-006543e622-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:10:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:10:43.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:43.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:43.025 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:10:43.066 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:43.066 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:43.124 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:10:43.152 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:10:43.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:43.153 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:10:43.153 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:10:43.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:43.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:43.154 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:10:43.154 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:10:43.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:43.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:43.155 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948643.000182_337.1", 2023-11-02T18:10:43.155 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:10:43.000182Z" 2023-11-02T18:10:43.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:43.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:43.156 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:10:43.156 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:10:43.156 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:43.156 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:10:43.157 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:10:43.158 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:10:43.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:43.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:43.160 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:10:43.243 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:43.243 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:43.295+0000 7f2add276f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:10:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:43.295+0000 7f2add276f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:10:43.332 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:43.332 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:10:43.332 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:10:43.333 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:10:43.333 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:10:43.333 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:43.333 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:43.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:43.334 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:10:43.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:43.335 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:10:43.335 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:43.335 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:43.335 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:10:43.335 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:43.336 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:10:43.336 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:43.336 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:10:43.336 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:43.337 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:43.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:43.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:43.337 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:10:43.337 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:43.338 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:43.338 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:10:43.338 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:43.338 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:43.339 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:43.339 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:10:43.339 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:43.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:43.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:43.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:43.340 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:10:43.340 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:43.340 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:43.341 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:10:43.341 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:43.341 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:43.341 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:43.341 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:10:43.342 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:43.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:43.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:43.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:43.343 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:10:43.343 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:43.344 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:43.344 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:10:43.344 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:43.345 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:43.345 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:43.345 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:10:43.346 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:43.346 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:43.347 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:43.347 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:43.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:43.348 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:10:43.348 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:10:43.349 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:10:43.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:43.349 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:10:43Z" 2023-11-02T18:10:43.350 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:43.352 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:10:43Z" } 2023-11-02T18:10:43.352 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:10:43.353 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:10:43.354 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:10:43.354 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:10:48.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:48.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:48.357 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:10:48.403 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:48.403 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:48.458 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:48.447+0000 7fb567c12f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:10:48.459 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:48.451+0000 7fb567c12f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:10:48.485 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:48.485 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:10:48.485 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:10:48.485 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:10:48.486 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:10:48.486 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:48.486 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:48.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:48.486 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:10:48.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:48.487 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:10:48.487 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:48.487 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:48.488 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:10:48.488 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:48.488 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:10:48.488 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:48.489 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:10:48.489 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:48.489 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:48.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:48.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:48.490 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:10:48.490 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:48.490 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:48.490 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:10:48.490 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:48.491 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:48.491 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:48.491 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:10:48.491 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:48.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:48.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:48.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:48.492 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:10:48.492 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:48.493 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:48.493 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:10:48.493 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:48.493 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:48.493 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:48.494 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:10:48.494 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:48.494 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:48.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:48.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:48.495 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:10:48.495 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:48.495 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:48.495 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:10:48.496 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:48.496 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:48.496 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:48.496 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:10:48.497 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:48.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:48.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:48.497 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:48.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:48.498 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:10:48.498 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:10:48.498 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:10:48.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:48.499 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:10:48Z" 2023-11-02T18:10:48.499 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:48.500 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:10:48Z" } 2023-11-02T18:10:48.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:10:48.501 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:10:48.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:10:48.502 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:10:53.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:53.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:53.504 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:10:53.554 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:53.554 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:53.604 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:53.595+0000 7f5dfc6f5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:10:53.605 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:53.595+0000 7f5dfc6f5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:10:53.632 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:53.633 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:10:53.633 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:10:53.633 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:10:53.633 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:10:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:10:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:10:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:10:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:10:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:10:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:10:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:10:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:10:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:10:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:10:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:10:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:10:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:53.646 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:53.646 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:10:53.646 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:53.647 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:53.647 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:53.647 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:10:53.647 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:53.648 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:53.648 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:53.648 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:53.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:53.649 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:10:53.649 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:10:53.649 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:10:53.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:53.650 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:10:53Z" 2023-11-02T18:10:53.650 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:53.652 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:10:53Z" } 2023-11-02T18:10:53.653 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:10:53.654 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:10:53.654 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:10:53.655 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:10:58.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:58.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:10:58.657 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:10:58.702 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:10:58.703 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:10:58.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:58.751+0000 7fa6f8251f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:10:58.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:10:58.751+0000 7fa6f8251f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:10:58.788 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:10:58.788 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:10:58.788 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:10:58.789 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:10:58.789 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:10:58.789 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:10:58.789 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:58.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:58.790 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:10:58.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:58.790 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:10:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:10:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:58.792 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:10:58.792 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:58.792 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:10:58.792 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:58.792 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:58.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:58.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:58.793 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:10:58.793 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:58.793 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:58.794 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:10:58.794 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:58.794 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:58.794 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:58.795 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:10:58.795 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:58.795 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:58.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:58.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:58.796 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:10:58.796 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:58.796 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:58.796 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:10:58.797 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:58.797 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:58.797 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:58.797 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:10:58.797 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:58.798 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:58.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:58.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:10:58.798 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:10:58.799 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:10:58.799 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:10:58.799 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:10:58.799 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:10:58.799 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:10:58.800 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:10:58.800 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:10:58.800 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:10:58.801 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:58.801 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:10:58.801 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:10:58.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:58.801 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:10:58.802 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:10:58.802 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:10:58.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:10:58.802 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:10:58Z" 2023-11-02T18:10:58.803 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:10:58.804 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:10:58Z" } 2023-11-02T18:10:58.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:10:58.805 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:10:58.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:10:58.806 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:11:03.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:03.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:03.808 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:11:03.854 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:11:03.854 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:11:03.913 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:03.903+0000 7f847b9c4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:03.914 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:03.903+0000 7f847b9c4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:03.942 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:11:03.942 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:11:03.942 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:11:03.942 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:11:03.943 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:11:03.943 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:11:03.943 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:03.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:03.944 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:11:03.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:03.944 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:11:03.944 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:03.945 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:03.945 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:11:03.945 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:03.945 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:11:03.946 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:03.946 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:11:03.946 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:03.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:03.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:03.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:03.947 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:11:03.947 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:03.947 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:03.948 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:11:03.948 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:03.948 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:03.948 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:03.949 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:11:03.949 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:03.949 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:03.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:03.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:03.950 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:11:03.950 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:03.950 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:03.950 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:11:03.951 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:03.951 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:03.951 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:03.951 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:11:03.952 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:03.952 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:03.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:03.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:03.952 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:11:03.953 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:03.954 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:03.954 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:11:03.954 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:03.955 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:03.955 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:03.955 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:11:03.955 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:03.955 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:03.956 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:03.956 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:11:03.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:03.956 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:11:03.957 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:11:03.957 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:11:03.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:03.957 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:11:03Z" 2023-11-02T18:11:03.957 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:11:03.959 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:11:03Z" } 2023-11-02T18:11:03.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:11:03.960 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:03.960 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:03.961 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:11:08.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:08.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:08.963 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:11:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:11:09.010 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:11:09.069 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:09.059+0000 7f23d9179f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:09.071 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:09.063+0000 7f23d9179f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:11:09Z" 2023-11-02T18:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:11:09.112 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:11:09Z" } 2023-11-02T18:11:09.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:11:09.113 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:09.114 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:09.114 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:11:14.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:14.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:14.116 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:11:14.164 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:11:14.165 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:11:14.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:14.215+0000 7ff07dca8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:14.225 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:14.215+0000 7ff07dca8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:14.251 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:11:14.251 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:11:14.252 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:11:14.252 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:11:14.252 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:11:14.252 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:11:14.253 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:14.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:14.253 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:11:14.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:14.253 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:11:14.254 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:14.254 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:14.254 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:11:14.254 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:14.254 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:11:14.255 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:14.255 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:11:14.255 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:14.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:14.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:14.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:14.256 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:11:14.257 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:14.257 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:14.257 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:11:14.257 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:14.257 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:14.258 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:14.258 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:11:14.258 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:14.258 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:14.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:14.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:14.259 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:11:14.259 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:14.260 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:14.260 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:11:14.260 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:14.260 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:14.261 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:14.261 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:11:14.261 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:14.261 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:14.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:14.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:14.262 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:11:14.262 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:14.263 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:14.263 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:11:14.263 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:14.263 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:14.263 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:14.264 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:11:14.264 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:14.264 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:14.264 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:14.265 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:11:14.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:14.265 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:11:14.265 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:11:14.265 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:11:14.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:14.266 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:11:14Z" 2023-11-02T18:11:14.266 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:11:14.267 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:11:14Z" } 2023-11-02T18:11:14.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:11:14.268 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:14.269 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:14.269 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:11:19.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:19.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:19.272 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:11:19.318 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:11:19.319 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:11:19.374 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:19.363+0000 7f1153f88f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:19.375 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:19.367+0000 7f1153f88f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:19.401 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:11:19.401 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:11:19.402 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:11:19.402 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:11:19.402 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:11:19.402 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:11:19.403 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:19.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:19.403 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:11:19.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:19.403 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:11:19.404 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:19.404 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:19.404 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:11:19.404 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:19.405 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:11:19.405 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:19.405 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:11:19.405 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:19.406 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:19.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:19.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:19.406 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:11:19.406 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:19.407 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:19.407 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:11:19.407 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:19.408 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:19.408 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:19.408 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:11:19.408 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:19.409 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:19.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:19.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:19.409 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:11:19.409 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:19.410 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:19.410 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:11:19.410 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:19.410 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:19.411 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:19.411 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:11:19.411 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:19.411 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:19.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:19.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:19.412 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:11:19.412 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:19.412 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:19.413 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:11:19.413 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:19.413 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:19.413 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:19.414 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:11:19.414 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:19.414 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:19.414 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:19.414 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:11:19.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:19.415 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:11:19.415 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:11:19.415 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:11:19.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:19.416 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:11:19Z" 2023-11-02T18:11:19.416 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:11:19.417 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:11:19Z" } 2023-11-02T18:11:19.418 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:11:19.418 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:19.419 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:19.419 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:11:24.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:24.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:24.422 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:11:24.469 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:11:24.470 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:11:24.524 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:24.515+0000 7f4dd0279f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:24.515+0000 7f4dd0279f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:24.552 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:11:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:11:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:11:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:11:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:11:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:11:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:11:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:11:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:11:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:11:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:11:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:11:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:11:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:11:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:11:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:11:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:11:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:11:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:11:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:11:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:11:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:11:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:11:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:11:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:11:24Z" 2023-11-02T18:11:24.566 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:11:24.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:11:24Z" } 2023-11-02T18:11:24.568 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:11:24.568 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:24.569 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:24.569 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:11:29.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:29.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:29.572 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:11:29.615 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:11:29.615 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:11:29.670 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:29.659+0000 7f5e8f889f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:29.672 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:29.663+0000 7f5e8f889f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:29.699 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:11:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:11:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:11:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:11:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:11:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:11:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:11:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:11:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:11:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:11:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:11:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:11:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:11:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:11:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:11:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:11:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:11:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:11:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:11:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:11:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:29.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:29.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:29.711 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:11:29.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:11:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:11:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:11:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:11:29Z" 2023-11-02T18:11:29.713 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:11:29.714 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:11:29Z" } 2023-11-02T18:11:29.714 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:11:29.715 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:29.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:29.715 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:11:34.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:34.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:34.718 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:11:34.765 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:11:34.765 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:11:34.822 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:34.811+0000 7ff65aadaf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:34.823 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:34.815+0000 7ff65aadaf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:34.850 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:11:34.851 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:11:34.851 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:11:34.851 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:11:34.852 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:11:34.852 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:11:34.852 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:34.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:34.852 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:11:34.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:34.853 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:11:34.853 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:34.853 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:34.854 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:11:34.854 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:34.854 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:11:34.854 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:34.854 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:11:34.855 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:34.855 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:34.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:34.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:34.856 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:11:34.856 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:34.856 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:34.856 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:11:34.856 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:34.857 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:34.857 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:34.857 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:11:34.857 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:34.857 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:34.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:34.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:34.858 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:11:34.858 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:34.858 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:34.859 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:11:34.859 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:34.859 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:34.859 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:34.860 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:11:34.860 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:34.860 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:34.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:34.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:34.861 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:11:34.861 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:34.861 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:34.861 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:11:34.862 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:34.862 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:34.862 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:34.862 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:11:34.862 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:34.863 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:34.863 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:34.863 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:11:34.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:34.863 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:11:34.864 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:11:34.864 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:11:34.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:34.864 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:11:34Z" 2023-11-02T18:11:34.864 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:11:34.866 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:11:34Z" } 2023-11-02T18:11:34.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:11:34.867 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:34.867 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:34.867 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:11:39.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:39.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:39.870 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:11:39.917 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:11:39.918 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:11:39.977 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:39.967+0000 7f312db98f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:39.978 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:39.967+0000 7f312db98f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:40.005 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:11:40.005 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:11:40.006 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:11:40.006 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:11:40.006 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:11:40.006 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:11:40.007 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:40.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:40.007 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:11:40.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:40.007 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:11:40.008 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:40.008 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:40.008 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:11:40.008 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:40.009 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:11:40.009 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:40.009 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:11:40.009 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:40.009 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:40.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:40.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:40.010 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:11:40.010 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:40.011 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:40.011 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:11:40.011 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:40.011 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:40.011 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:40.012 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:11:40.012 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:40.012 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:40.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:40.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:40.013 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:11:40.013 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:40.013 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:40.013 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:11:40.014 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:40.014 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:40.014 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:40.014 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:11:40.015 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:40.015 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:40.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:40.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:40.015 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:11:40.016 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:40.016 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:40.016 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:11:40.016 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:40.017 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:40.017 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:40.017 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:11:40.023 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:40.023 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:40.024 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:40.024 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:11:40.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:40.024 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:11:40.024 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:11:40.025 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:11:40.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:40.025 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:11:39Z" 2023-11-02T18:11:40.025 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:11:40.026 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:11:39Z" } 2023-11-02T18:11:40.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:11:40.029 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:40.029 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:40.030 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:11:45.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:45.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:45.033 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:11:45.078 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:11:45.079 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:11:45.137 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:45.127+0000 7f85ef530f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:45.139 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:45.131+0000 7f85ef530f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:45.165 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:11:45.165 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:11:45.166 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:11:45.166 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:11:45.166 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:11:45.166 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:11:45.167 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:45.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:45.167 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:11:45.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:45.168 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:11:45.168 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:45.168 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:45.168 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:11:45.168 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:45.169 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:11:45.169 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:45.169 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:11:45.170 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:45.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:45.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:45.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:45.171 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:11:45.171 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:45.171 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:45.171 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:11:45.172 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:45.172 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:45.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:45.172 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:11:45.172 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:45.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:45.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:45.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:45.173 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:11:45.174 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:45.174 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:45.174 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:11:45.174 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:45.174 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:45.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:45.175 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:11:45.175 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:45.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:45.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:45.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:45.176 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:11:45.176 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:45.176 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:45.177 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:11:45.177 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:45.177 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:45.177 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:45.178 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:11:45.178 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:45.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:45.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:45.179 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:11:45.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:45.179 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:11:45.179 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:11:45.179 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:11:45.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:45.180 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:11:45Z" 2023-11-02T18:11:45.180 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:11:45.181 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:11:45Z" } 2023-11-02T18:11:45.182 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:11:45.182 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:45.183 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:45.183 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:11:50.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:50.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:50.186 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:11:50.230 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:11:50.230 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:11:50.288 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:50.279+0000 7fc0f7e91f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:50.289 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:50.279+0000 7fc0f7e91f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:50.315 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:11:50.315 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:11:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:11:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:11:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:11:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:11:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:11:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:11:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:11:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:11:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:11:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:11:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:11:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:11:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:11:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:11:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:11:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:11:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:11:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:11:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:11:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:11:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:11:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:11:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:11:50Z" 2023-11-02T18:11:50.331 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:11:50.332 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:11:50Z" } 2023-11-02T18:11:50.332 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:11:50.333 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:50.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:50.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:11:55.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:55.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:11:55.335 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:11:55.382 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:11:55.382 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:11:55.438 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:55.427+0000 7f287759cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:55.440 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:11:55.431+0000 7f287759cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:11:55.466 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:11:55.466 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:11:55.467 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:11:55.467 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:11:55.467 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:11:55.467 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:11:55.467 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:55.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:55.468 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:11:55.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:55.468 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:11:55.469 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:55.469 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:55.469 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:11:55.469 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:55.469 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:11:55.470 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:55.470 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:11:55.470 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:55.470 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:55.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:55.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:55.471 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:11:55.471 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:55.471 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:55.472 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:11:55.472 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:55.472 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:55.472 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:55.472 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:11:55.473 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:55.473 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:55.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:55.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:55.473 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:11:55.474 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:55.474 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:55.474 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:11:55.475 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:55.475 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:55.475 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:55.475 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:11:55.476 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:55.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:55.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:55.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:11:55.477 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:11:55.477 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:11:55.477 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:11:55.477 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:11:55.477 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:11:55.478 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:11:55.478 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:11:55.478 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:11:55.478 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:11:55.479 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:55.479 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:11:55.479 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:11:55.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:55.480 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:11:55.480 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:11:55.480 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:11:55.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:11:55.480 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:11:55Z" 2023-11-02T18:11:55.481 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:11:55.482 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:11:55Z" } 2023-11-02T18:11:55.482 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:11:55.483 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:55.483 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:11:55.484 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:12:00.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:00.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:00.486 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:12:00.532 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:12:00.533 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:12:00.591 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:00.583+0000 7f91d56b5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:00.592 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:00.583+0000 7f91d56b5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:00.620 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:12:00.620 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:12:00.620 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:12:00.621 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:12:00.621 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:12:00.621 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:12:00.621 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:00.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:00.622 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:12:00.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:00.622 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:12:00.623 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:00.623 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:00.623 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:12:00.623 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:00.623 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:12:00.624 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:00.624 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:12:00.624 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:00.624 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:00.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:00.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:00.625 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:12:00.625 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:00.625 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:00.626 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:12:00.626 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:00.626 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:00.626 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:00.627 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:12:00.627 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:00.627 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:00.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:00.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:00.628 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:12:00.628 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:00.628 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:00.628 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:12:00.629 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:00.629 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:00.629 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:00.629 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:12:00.629 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:00.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:00.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:00.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:00.630 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:12:00.631 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:00.631 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:00.631 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:12:00.631 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:00.631 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:00.632 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:00.632 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:12:00.632 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:00.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:00.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:00.633 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:12:00.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:00.633 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:12:00.633 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:12:00.633 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:12:00.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:00.634 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:12:00Z" 2023-11-02T18:12:00.634 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:12:00.635 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:12:00Z" } 2023-11-02T18:12:00.636 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:12:00.636 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:00.637 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:00.637 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:12:05.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:05.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:05.640 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:12:05.685 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:12:05.685 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:12:05.742 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:05.731+0000 7f5f3a34ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:05.744 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:05.735+0000 7f5f3a34ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:05.769 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:12:05.770 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:12:05.770 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:12:05.770 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:12:05.770 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:12:05.770 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:12:05.771 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:05.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:05.771 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:12:05.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:05.772 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:12:05.772 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:05.772 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:05.772 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:12:05.772 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:05.773 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:12:05.773 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:05.773 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:12:05.773 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:05.773 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:05.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:05.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:05.774 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:12:05.774 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:05.775 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:05.775 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:12:05.775 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:05.775 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:05.775 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:05.776 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:12:05.776 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:05.776 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:05.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:05.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:05.776 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:12:05.777 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:05.777 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:05.777 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:12:05.777 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:05.778 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:05.778 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:05.778 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:12:05.778 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:05.778 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:05.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:05.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:05.779 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:12:05.779 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:05.779 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:05.780 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:12:05.780 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:05.780 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:05.780 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:05.781 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:12:05.781 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:05.781 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:05.781 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:05.781 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:12:05.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:05.782 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:12:05.782 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:12:05.782 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:12:05.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:05.783 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:12:05Z" 2023-11-02T18:12:05.783 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:12:05.784 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:12:05Z" } 2023-11-02T18:12:05.785 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:12:05.785 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:05.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:05.786 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:12:10.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:10.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:10.789 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:12:10.834 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:12:10.834 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:12:10.893 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:10.883+0000 7f493976df40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:10.895 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:10.887+0000 7f493976df40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:10.921 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:12:10.921 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:12:10.921 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:12:10.921 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:12:10.922 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:12:10.922 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:12:10.922 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:10.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:10.922 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:12:10.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:10.923 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:12:10.923 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:10.923 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:10.924 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:12:10.924 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:10.924 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:12:10.924 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:10.924 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:12:10.925 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:10.925 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:10.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:10.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:10.925 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:12:10.926 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:10.926 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:10.926 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:12:10.926 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:10.927 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:10.927 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:10.927 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:12:10.927 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:10.927 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:10.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:10.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:10.928 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:12:10.928 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:10.928 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:10.929 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:12:10.929 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:10.929 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:10.929 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:10.930 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:12:10.930 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:10.930 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:10.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:10.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:10.931 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:12:10.931 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:10.931 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:10.931 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:12:10.931 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:10.932 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:10.932 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:10.932 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:12:10.932 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:10.933 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:10.933 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:10.933 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:12:10.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:10.933 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:12:10.934 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:12:10.934 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:12:10.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:10.934 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:12:10Z" 2023-11-02T18:12:10.934 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:12:10.935 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:12:10Z" } 2023-11-02T18:12:10.936 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:12:10.936 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:10.937 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:10.937 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:12:15.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:15.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:15.940 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:12:15.988 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:12:15.988 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:12:16.044 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:16.035+0000 7fb4576f6f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:16.045 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:16.035+0000 7fb4576f6f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:16.071 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:12:16.072 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:12:16.072 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:12:16.072 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:12:16.072 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:12:16.073 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:12:16.073 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:16.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:16.073 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:12:16.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:16.074 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:12:16.074 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:16.074 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:16.074 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:12:16.074 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:16.075 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:12:16.075 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:16.075 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:12:16.075 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:16.075 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:16.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:16.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:16.076 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:12:16.076 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:16.076 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:16.076 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:12:16.077 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:16.077 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:16.077 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:16.077 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:12:16.078 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:16.078 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:16.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:16.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:16.078 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:12:16.079 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:16.079 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:16.079 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:12:16.079 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:16.079 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:16.080 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:16.080 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:12:16.080 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:16.080 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:16.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:16.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:16.081 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:12:16.081 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:16.081 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:16.081 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:12:16.082 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:16.082 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:16.082 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:16.082 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:12:16.082 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:16.083 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:16.083 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:16.083 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:12:16.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:16.083 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:12:16.083 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:12:16.084 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:12:16.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:16.084 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:12:16Z" 2023-11-02T18:12:16.084 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:12:16.085 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:12:16Z" } 2023-11-02T18:12:16.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:12:16.086 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:16.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:16.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:12:21.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:21.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:21.090 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:12:21.139 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:12:21.139 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:12:21.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:21.187+0000 7f32cbf08f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:21.199 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:21.191+0000 7f32cbf08f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:21.225 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:12:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:12:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:12:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:12:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:12:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:12:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:12:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:12:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:21.228 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:12:21.229 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:21.229 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:12:21.229 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:21.229 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:12:21.230 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:21.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:21.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:21.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:12:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:12:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:12:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:12:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:12:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:12:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:12:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:12:21.237 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:21.237 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:21.237 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:21.237 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:12:21.237 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:21.238 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:21.238 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:21.238 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:12:21.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:21.239 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:12:21.239 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:12:21.239 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:12:21.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:21.239 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:12:21Z" 2023-11-02T18:12:21.240 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:12:21.241 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:12:21Z" } 2023-11-02T18:12:21.241 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:12:21.242 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:21.242 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:21.243 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:12:26.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:26.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:26.245 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:12:26.291 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:12:26.291 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:12:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:26.339+0000 7f90b0cb5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:26.343+0000 7f90b0cb5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:26.378 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:12:26.378 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:12:26.378 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:12:26.379 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:12:26.379 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:12:26.379 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:12:26.379 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:26.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:26.380 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:12:26.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:26.380 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:12:26.380 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:26.381 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:26.381 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:12:26.381 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:26.381 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:12:26.382 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:26.382 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:12:26.382 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:26.382 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:26.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:26.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:26.383 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:12:26.383 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:26.383 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:26.384 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:12:26.384 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:26.384 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:26.384 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:26.384 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:12:26.385 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:26.385 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:26.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:26.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:26.385 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:12:26.386 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:26.386 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:26.386 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:12:26.386 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:26.387 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:26.387 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:26.387 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:12:26.387 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:26.387 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:26.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:26.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:26.388 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:12:26.388 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:26.388 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:26.389 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:12:26.389 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:26.389 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:26.390 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:26.390 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:12:26.390 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:26.390 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:26.390 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:26.391 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:12:26.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:26.391 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:12:26.391 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:12:26.392 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:12:26.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:26.392 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:12:26Z" 2023-11-02T18:12:26.392 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:12:26.393 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:12:26Z" } 2023-11-02T18:12:26.394 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:12:26.394 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:26.395 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:26.395 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:12:31.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:31.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:31.398 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:12:31.445 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:12:31.445 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:12:31.501 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:31.491+0000 7fcd369a8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:31.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:31.495+0000 7fcd369a8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:31.527 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:12:31.528 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:12:31.528 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:12:31.528 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:12:31.529 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:12:31.529 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:12:31.529 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:31.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:31.529 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:12:31.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:31.530 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:12:31.530 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:31.530 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:31.531 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:12:31.531 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:31.531 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:12:31.531 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:31.532 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:12:31.532 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:31.532 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:31.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:31.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:31.533 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:12:31.533 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:31.533 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:31.534 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:12:31.534 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:31.534 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:31.534 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:31.535 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:12:31.535 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:31.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:31.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:31.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:31.536 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:12:31.536 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:31.536 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:31.537 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:12:31.537 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:31.537 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:31.537 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:31.538 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:12:31.538 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:31.538 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:31.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:31.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:31.539 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:12:31.539 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:31.539 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:31.539 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:12:31.540 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:31.540 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:31.540 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:31.540 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:12:31.541 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:31.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:31.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:31.541 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:12:31.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:31.542 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:12:31.542 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:12:31.542 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:12:31.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:31.543 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:12:31Z" 2023-11-02T18:12:31.543 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:12:31.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:12:31Z" } 2023-11-02T18:12:31.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:12:31.545 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:31.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:31.546 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:12:36.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:36.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:36.548 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:12:36.642 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:36.635+0000 7fb7c673cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:36.644 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:36.635+0000 7fb7c673cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:36.671 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:12:36.671 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:12:36.671 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:12:36.672 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:12:36.672 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:12:36.672 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:12:36.672 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:36.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:36.673 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:12:36.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:36.673 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:12:36.673 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:36.674 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:36.674 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:12:36.674 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:36.674 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:12:36.675 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:36.675 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:12:36.675 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:36.675 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:36.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:36.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:36.676 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:12:36.676 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:36.676 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:36.677 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:12:36.677 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:36.677 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:36.677 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:36.677 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:12:36.678 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:36.678 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:36.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:36.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:36.679 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:12:36.679 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:36.679 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:36.679 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948387.133683_219.1", 2023-11-02T18:12:36.680 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:36.680 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:36.680 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:36.680 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:06:27.133683Z", 2023-11-02T18:12:36.680 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:36.681 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:36.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:36.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:36.681 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:12:36.682 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:36.682 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:36.682 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:12:36.682 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:36.682 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:36.683 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:36.683 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:12:36.683 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:36.683 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:36.684 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:36.684 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:12:36.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:36.684 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:12:36.684 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:12:36.685 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:12:36.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:36.685 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:12:36Z" 2023-11-02T18:12:36.685 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:12:36.686 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948387.133683_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:06:27.133683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:12:36Z" } 2023-11-02T18:12:36.687 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948387.133683_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:06:27.133683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:12:36.687 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:36.688 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948387.133683_219.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:36.688 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948387.133683_219.1 2023-11-02T18:12:41.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:41.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:41.691 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:12:41.739 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:12:41.739 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:12:41.793 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:41.783+0000 7fea087a6f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:41.795 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:41.787+0000 7fea087a6f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:41.822 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:12:41.822 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:12:41.822 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:12:41.823 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:12:41.823 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:12:41.823 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:12:41.823 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:41.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:41.824 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:12:41.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:41.824 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:12:41.824 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:41.825 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:41.825 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:12:41.825 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:41.825 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:12:41.825 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:41.826 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:12:41.826 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:41.826 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:41.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:41.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:41.827 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:12:41.827 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:41.827 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:41.827 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:12:41.828 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:41.828 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:41.828 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:41.828 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:12:41.828 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:41.829 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:41.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:41.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:41.829 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:12:41.829 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:41.830 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:41.830 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948639.433213_280.1", 2023-11-02T18:12:41.830 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:41.830 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:41.831 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:41.831 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:10:39.433213Z", 2023-11-02T18:12:41.831 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:41.831 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:41.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:41.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:41.832 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:12:41.832 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:41.832 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:41.832 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:12:41.833 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:41.833 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:41.833 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:41.833 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:12:41.834 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:41.834 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:41.834 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:41.834 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:12:41.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:41.835 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:12:41.835 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:12:41.835 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:12:41.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:41.836 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:12:41Z" 2023-11-02T18:12:41.836 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:12:41.837 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948639.433213_280.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:10:39.433213Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:12:41Z" } 2023-11-02T18:12:41.837 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948639.433213_280.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:10:39.433213Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:12:41.838 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:41.841 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948639.433213_280.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:41.841 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948639.433213_280.1 2023-11-02T18:12:46.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:46.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:46.844 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:12:46.896 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:12:46.896 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:12:46.945 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:46.935+0000 7f9b72ac1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:46.946 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:46.939+0000 7f9b72ac1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:46.966 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:12:46.966 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:12:46.967 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:12:46.969 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:12:46.969 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:12:46.970 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:12:46.970 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:46.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:46.970 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:12:46.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:46.971 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:12:46.971 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:46.971 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:46.971 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:12:46.972 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:46.972 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:12:46.972 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:46.972 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:12:46.973 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:46.973 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:46.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:46.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:46.974 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:12:46.974 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:46.974 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:46.974 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:12:46.974 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:46.975 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:46.975 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:46.975 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:12:46.975 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:46.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:46.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:46.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:46.976 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:12:46.976 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:46.977 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:46.977 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948639.579667_292.1", 2023-11-02T18:12:46.977 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:46.977 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:46.978 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:46.978 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:10:39.579667Z", 2023-11-02T18:12:46.978 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:46.978 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:46.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:46.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:46.979 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:12:46.979 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:46.979 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:46.980 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:12:46.980 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:46.980 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:46.980 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:46.980 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:12:46.981 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:46.981 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:46.981 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:46.981 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:12:46.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:46.982 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:12:46.982 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:12:46.982 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:12:46.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:46.983 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:12:46Z" 2023-11-02T18:12:46.983 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:12:46.984 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948639.579667_292.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:10:39.579667Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:12:46Z" } 2023-11-02T18:12:46.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948639.579667_292.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:10:39.579667Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:12:46.985 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:46.985 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948639.579667_292.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:46.986 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948639.579667_292.1 2023-11-02T18:12:51.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:51.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:51.988 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:12:52.034 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:12:52.034 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:12:52.093 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:52.083+0000 7f8389017f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:52.095 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:52.087+0000 7f8389017f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:52.123 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:12:52.123 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:12:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:12:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:12:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:12:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:12:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:12:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:12:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:12:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:12:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:12:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:12:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:12:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:12:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:52.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:52.137 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:12:52.137 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:52.138 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:52.138 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948639.534915_290.1", 2023-11-02T18:12:52.139 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:52.139 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:52.140 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:52.140 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:10:39.534915Z", 2023-11-02T18:12:52.140 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:52.140 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:12:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:12:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:12:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:12:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:12:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:12:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:12:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:12:52Z" 2023-11-02T18:12:52.147 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:12:52.148 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948639.534915_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:10:39.534915Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:12:52Z" } 2023-11-02T18:12:52.149 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948639.534915_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:10:39.534915Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:12:52.150 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:52.150 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948639.534915_290.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:52.151 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948639.534915_290.1 2023-11-02T18:12:57.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:57.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:12:57.154 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:12:57.200 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:12:57.201 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:12:57.259 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:57.251+0000 7f9b8d7abf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:57.260 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:12:57.251+0000 7f9b8d7abf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:12:57.288 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:12:57.288 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:12:57.289 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:12:57.289 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:12:57.289 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:12:57.289 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:12:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:12:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:12:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:12:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:12:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:12:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:12:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:57.294 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:12:57.294 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:57.294 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:57.294 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:57.294 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:12:57.295 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:57.295 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:57.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:57.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:57.295 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:12:57.296 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:57.296 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:57.296 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948639.538994_291.1", 2023-11-02T18:12:57.296 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:10:39.538994Z", 2023-11-02T18:12:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:12:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:12:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:12:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:12:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:12:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:12:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:12:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:12:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:12:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:12:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:12:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:12:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:12:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:12:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:12:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:12:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:12:57Z" 2023-11-02T18:12:57.302 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:12:57.303 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948639.538994_291.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:10:39.538994Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:12:57Z" } 2023-11-02T18:12:57.304 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948639.538994_291.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:10:39.538994Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:12:57.304 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:57.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948639.538994_291.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:12:57.305 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948639.538994_291.1 2023-11-02T18:13:02.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:02.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:02.307 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:02.355 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:02.356 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:02.399+0000 7f463b370f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:02.403+0000 7f463b370f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:02.438 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:13:02.438 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:13:02.438 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:13:02.439 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:13:02.439 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:13:02.439 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:13:02.439 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:02.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:02.440 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:13:02.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:02.440 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:13:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:13:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:13:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:13:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:02.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:02.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:02.443 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:13:02.443 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:02.444 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:02.444 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:13:02.444 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:02.444 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:02.444 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:02.445 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:13:02.445 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:02.445 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:02.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:02.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:02.446 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:13:02.446 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:02.446 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:02.446 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948639.579667_292.1", 2023-11-02T18:13:02.447 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:02.447 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:02.447 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:02.447 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:10:39.579667Z", 2023-11-02T18:13:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:13:02.449 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:02.449 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:02.449 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:13:02.449 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:02.449 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:02.450 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:02.450 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:13:02.450 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:02.450 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:02.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:02.451 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:13:02.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:02.451 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:13:02.452 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:13:02.452 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:13:02.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:02.452 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:13:02Z" 2023-11-02T18:13:02.452 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:13:02.453 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948639.579667_292.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:10:39.579667Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:13:02Z" } 2023-11-02T18:13:02.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948639.579667_292.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:10:39.579667Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:13:02.455 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:02.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948639.579667_292.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:02.455 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948639.579667_292.1 2023-11-02T18:13:07.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:07.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:07.458 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:07.504 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:07.504 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:07.563 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:07.554+0000 7fe277ba0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:07.565 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:07.554+0000 7fe277ba0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:07.589 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:13:07.589 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:13:07.589 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:13:07.590 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:13:07.590 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:13:07.590 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:13:07.590 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:07.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:07.591 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:13:07.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:07.591 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:13:07.591 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:07.592 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:07.592 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:13:07.592 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:07.592 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:13:07.593 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:07.593 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:13:07.593 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:07.593 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:07.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:07.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:07.594 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:13:07.594 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:07.594 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:07.595 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:13:07.595 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:07.595 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:07.595 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:07.596 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:13:07.596 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:07.596 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:07.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:07.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:07.597 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:13:07.597 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:07.597 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:07.598 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948639.475252_284.1", 2023-11-02T18:13:07.598 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:07.598 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:07.598 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:07.599 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:10:39.475252Z", 2023-11-02T18:13:07.599 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:07.599 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:07.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:07.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:07.600 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:13:07.600 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:07.600 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:07.600 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:13:07.601 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:07.601 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:07.601 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:07.601 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:13:07.601 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:07.602 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:07.602 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:07.602 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:13:07.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:07.603 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:13:07.603 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:13:07.603 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:13:07.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:07.604 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:13:07Z" 2023-11-02T18:13:07.604 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:13:07.605 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948639.475252_284.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:10:39.475252Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:13:07Z" } 2023-11-02T18:13:07.606 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948639.475252_284.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:10:39.475252Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:13:07.606 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:07.607 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948639.475252_284.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:07.607 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948639.475252_284.1 2023-11-02T18:13:12.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:12.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:12.609 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:12.656 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:12.657 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:12.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:12.706+0000 7f5903856f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:12.719 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:12.710+0000 7f5903856f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:12.744 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:13:12.744 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:13:12.745 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:13:12.745 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:13:12.745 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:13:12.745 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:13:12.746 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:12.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:12.746 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:13:12.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:12.747 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:13:12.747 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:12.747 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:12.747 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:13:12.748 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:12.748 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:13:12.748 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:12.748 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:13:12.748 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:12.749 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:12.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:12.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:12.749 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:13:12.750 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:12.750 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:12.750 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:13:12.750 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:12.750 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:12.751 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:12.751 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:13:12.751 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:12.751 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:12.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:12.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:12.752 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:13:12.752 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:12.752 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:12.753 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948639.663657_304.1", 2023-11-02T18:13:12.753 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:12.753 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:12.753 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:12.753 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:10:39.663657Z", 2023-11-02T18:13:12.754 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:12.754 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:12.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:12.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:12.755 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:13:12.755 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:12.755 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:12.755 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:13:12.755 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:12.756 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:12.756 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:12.756 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:13:12.756 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:12.757 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:12.757 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:12.757 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:13:12.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:12.757 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:13:12.757 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:13:12.758 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:13:12.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:12.758 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:13:12Z" 2023-11-02T18:13:12.758 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:13:12.759 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948639.663657_304.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:10:39.663657Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:13:12Z" } 2023-11-02T18:13:12.760 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948639.663657_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:10:39.663657Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:13:12.761 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:12.761 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948639.663657_304.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:12.761 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948639.663657_304.1 2023-11-02T18:13:17.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:17.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:17.764 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:17.809 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:17.809 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:17.867 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:17.858+0000 7f82ab819f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:17.869 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:17.858+0000 7f82ab819f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:17.896 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:13:17.897 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:13:17.897 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:13:17.897 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:13:17.897 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:13:17.898 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:13:17.898 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:17.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:17.898 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:13:17.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:17.899 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:13:17.899 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:17.899 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:17.900 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:13:17.900 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:17.900 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:13:17.900 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:17.900 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:13:17.901 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:17.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:17.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:17.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:17.901 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:13:17.902 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:17.902 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:17.902 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:13:17.902 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:17.903 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:17.903 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:17.903 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:13:17.903 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:17.904 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:17.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:17.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:17.904 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:13:17.904 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:17.905 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:17.905 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948639.663657_304.1", 2023-11-02T18:13:17.905 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:17.905 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:17.906 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:17.906 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:10:39.663657Z", 2023-11-02T18:13:17.906 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:17.906 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:17.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:17.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:17.907 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:13:17.907 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:17.907 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:17.907 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:13:17.908 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:17.908 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:17.908 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:17.908 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:13:17.909 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:17.909 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:17.909 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:17.909 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:13:17.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:17.910 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:13:17.910 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:13:17.910 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:13:17.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:17.911 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:13:17Z" 2023-11-02T18:13:17.911 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:13:17.912 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948639.663657_304.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:10:39.663657Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:13:17Z" } 2023-11-02T18:13:17.912 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948639.663657_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:10:39.663657Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:13:17.913 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:17.913 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948639.663657_304.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:17.914 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948639.663657_304.1 2023-11-02T18:13:22.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:22.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:22.917 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:22.961 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:22.962 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:23.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:23.010+0000 7f1a547ccf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:23.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:23.014+0000 7f1a547ccf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:23.046 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:13:23.047 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:13:23.047 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:13:23.047 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:13:23.047 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:13:23.048 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:13:23.048 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:23.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:23.049 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:13:23.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:23.049 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:13:23.049 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:23.049 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:23.050 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:13:23.050 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:23.050 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:13:23.050 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:23.051 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:13:23.051 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:23.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:23.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:23.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:23.052 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:13:23.052 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:23.052 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:23.052 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:13:23.053 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:23.053 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:23.053 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:23.053 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:13:23.053 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:23.054 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:23.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:23.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:23.054 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:13:23.055 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:23.055 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:23.055 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948639.534915_290.1", 2023-11-02T18:13:23.055 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:23.055 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:23.056 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:23.056 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:10:39.534915Z", 2023-11-02T18:13:23.056 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:23.056 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:23.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:23.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:23.057 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:13:23.057 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:23.057 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:23.058 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:13:23.058 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:23.058 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:23.058 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:23.059 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:13:23.059 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:23.059 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:23.059 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:23.060 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:13:23.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:23.060 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:13:23.060 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:13:23.060 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:13:23.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:23.061 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:13:23Z" 2023-11-02T18:13:23.061 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:13:23.062 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948639.534915_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:10:39.534915Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:13:23Z" } 2023-11-02T18:13:23.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948639.534915_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:10:39.534915Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:13:23.066 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:23.066 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948639.534915_290.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:23.067 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948639.534915_290.1 2023-11-02T18:13:28.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:28.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:28.070 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:28.118 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:28.119 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:28.175 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:28.166+0000 7f7ffea50f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:28.177 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:28.166+0000 7f7ffea50f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:28.205 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:13:28.205 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:13:28.205 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:13:28.206 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:13:28.206 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:13:28.206 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:13:28.206 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:28.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:28.207 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:13:28.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:28.207 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:13:28.207 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:28.208 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:28.208 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:13:28.208 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:28.208 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:13:28.208 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:28.209 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:13:28.209 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:28.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:28.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:28.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:28.210 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:13:28.210 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:28.210 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:28.210 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:13:28.211 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:28.211 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:28.211 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:28.211 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:13:28.212 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:28.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:28.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:28.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:28.212 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:13:28.213 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:28.213 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:28.213 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948639.793666_321.1", 2023-11-02T18:13:28.213 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:28.214 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:28.214 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:28.214 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:10:39.793666Z", 2023-11-02T18:13:28.214 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:28.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:28.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:28.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:28.215 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:13:28.215 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:28.216 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:28.216 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:13:28.216 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:28.216 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:28.216 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:28.217 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:13:28.217 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:28.217 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:28.217 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:28.218 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:13:28.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:28.218 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:13:28.218 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:13:28.218 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:13:28.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:28.219 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:13:28Z" 2023-11-02T18:13:28.219 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:13:28.220 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948639.793666_321.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:10:39.793666Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:13:28Z" } 2023-11-02T18:13:28.221 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948639.793666_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:10:39.793666Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:13:28.221 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:28.222 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948639.793666_321.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:28.222 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948639.793666_321.1 2023-11-02T18:13:33.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:33.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:33.225 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:33.270 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:33.270 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:33.318+0000 7fa1116f1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:33.318+0000 7fa1116f1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:33.355 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:13:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:13:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:13:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:13:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:13:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:13:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:13:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:13:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:13:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:13:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:13:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:13:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:13:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:13:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:13:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948639.585943_294.1", 2023-11-02T18:13:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:10:39.585943Z", 2023-11-02T18:13:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:13:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:13:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:13:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:13:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:13:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:13:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:13:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:13:33Z" 2023-11-02T18:13:33.370 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:13:33.372 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948639.585943_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:10:39.585943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:13:33Z" } 2023-11-02T18:13:33.372 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948639.585943_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:10:39.585943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:13:33.373 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:33.373 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948639.585943_294.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:33.374 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698948643.000182_337.1 target=1_1698948639.585943_294.1 2023-11-02T18:13:38.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:38.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:38.376 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:38.422 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:38.422 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:38.480 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:38.470+0000 7f764a24df40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:38.482 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:38.474+0000 7f764a24df40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:38.510 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:13:38.510 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:13:38.510 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:13:38.511 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:13:38.511 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:13:38.512 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:13:38.512 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:38.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:38.513 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:13:38.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:38.513 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:13:38.513 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:38.513 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:38.514 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:13:38.514 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:38.514 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:13:38.514 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:38.515 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:13:38.515 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:38.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:38.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:38.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:38.516 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:13:38.517 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:38.517 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:38.517 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:13:38.517 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:38.518 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:38.518 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:38.519 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:13:38.519 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:38.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:38.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:38.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:38.521 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:13:38.521 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:38.521 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:38.522 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948643.000182_337.1", 2023-11-02T18:13:38.522 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:38.522 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:38.523 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:38.523 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:10:43.000182Z", 2023-11-02T18:13:38.524 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:38.524 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:38.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:38.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:38.525 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:13:38.525 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:38.525 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:38.525 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:13:38.526 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:38.526 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:38.526 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:38.526 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:13:38.527 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:38.527 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:38.527 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:38.527 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:13:38.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:38.528 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:13:38.528 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:13:38.528 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:13:38.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:38.529 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:13:38Z" 2023-11-02T18:13:38.529 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:13:38.530 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948643.000182_337.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:10:43.000182Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:13:38Z" } 2023-11-02T18:13:38.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948643.000182_337.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:10:43.000182Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:13:38.531 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:38.532 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948643.000182_337.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:38.532 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:13:38.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ibtljn-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:13:38.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ibtljn-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:13:38.534 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ibtljn-44 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:13:38.583 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:38.584 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:38.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ibtljn-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:13:38.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ibtljn-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:13:38.720 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ibtljn-45 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:13:38.765 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:38.765 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:38.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:38.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:38.902 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:38.984 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:38.984 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:39.081 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:13:39.082 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:1", 2023-11-02T18:13:39.082 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000078", 2023-11-02T18:13:39.082 INFO:teuthology.orchestra.run.smithi017.stdout: "meta.history", 2023-11-02T18:13:39.083 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000070", 2023-11-02T18:13:39.083 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000104", 2023-11-02T18:13:39.083 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000026", 2023-11-02T18:13:39.083 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:10", 2023-11-02T18:13:39.084 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:3", 2023-11-02T18:13:39.084 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.8", 2023-11-02T18:13:39.084 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000028", 2023-11-02T18:13:39.084 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.39a01552-6d59-4ce7-b7e5-915ca69b979a.1.retry", 2023-11-02T18:13:39.084 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:0", 2023-11-02T18:13:39.085 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:6", 2023-11-02T18:13:39.085 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:8", 2023-11-02T18:13:39.085 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:1", 2023-11-02T18:13:39.086 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:5", 2023-11-02T18:13:39.086 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000040", 2023-11-02T18:13:39.086 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:0", 2023-11-02T18:13:39.087 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.6", 2023-11-02T18:13:39.087 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:2", 2023-11-02T18:13:39.087 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.1.vEUoyBhExT0.0", 2023-11-02T18:13:39.087 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:10", 2023-11-02T18:13:39.088 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000015", 2023-11-02T18:13:39.088 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000069", 2023-11-02T18:13:39.088 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000095", 2023-11-02T18:13:39.088 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:4", 2023-11-02T18:13:39.089 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000003", 2023-11-02T18:13:39.089 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000047", 2023-11-02T18:13:39.089 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:9", 2023-11-02T18:13:39.089 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000035", 2023-11-02T18:13:39.089 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000037", 2023-11-02T18:13:39.090 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:2", 2023-11-02T18:13:39.090 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:4", 2023-11-02T18:13:39.090 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:3", 2023-11-02T18:13:39.090 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000024", 2023-11-02T18:13:39.090 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000000", 2023-11-02T18:13:39.091 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000031", 2023-11-02T18:13:39.091 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.2.t2U+WQNCaA2.0", 2023-11-02T18:13:39.091 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.3", 2023-11-02T18:13:39.091 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:4", 2023-11-02T18:13:39.092 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:10", 2023-11-02T18:13:39.092 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:6", 2023-11-02T18:13:39.092 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:10", 2023-11-02T18:13:39.092 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:6", 2023-11-02T18:13:39.092 INFO:teuthology.orchestra.run.smithi017.stdout: "data_loggenerations_metadata", 2023-11-02T18:13:39.093 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:2", 2023-11-02T18:13:39.093 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:3", 2023-11-02T18:13:39.093 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:7", 2023-11-02T18:13:39.093 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000076", 2023-11-02T18:13:39.094 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000113", 2023-11-02T18:13:39.094 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000005", 2023-11-02T18:13:39.094 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000011", 2023-11-02T18:13:39.094 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:10", 2023-11-02T18:13:39.095 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000058", 2023-11-02T18:13:39.095 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000052", 2023-11-02T18:13:39.095 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:1", 2023-11-02T18:13:39.095 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:4", 2023-11-02T18:13:39.096 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:7", 2023-11-02T18:13:39.096 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:6", 2023-11-02T18:13:39.096 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:9", 2023-11-02T18:13:39.096 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:1", 2023-11-02T18:13:39.096 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:1", 2023-11-02T18:13:39.097 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:7", 2023-11-02T18:13:39.097 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000088", 2023-11-02T18:13:39.097 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000080", 2023-11-02T18:13:39.097 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000090", 2023-11-02T18:13:39.097 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000110", 2023-11-02T18:13:39.098 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14", 2023-11-02T18:13:39.098 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:2", 2023-11-02T18:13:39.098 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:6", 2023-11-02T18:13:39.098 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10", 2023-11-02T18:13:39.099 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000096", 2023-11-02T18:13:39.099 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:6", 2023-11-02T18:13:39.099 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:8", 2023-11-02T18:13:39.099 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:6", 2023-11-02T18:13:39.100 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:5", 2023-11-02T18:13:39.100 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000087", 2023-11-02T18:13:39.100 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000008", 2023-11-02T18:13:39.101 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:0", 2023-11-02T18:13:39.101 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000006", 2023-11-02T18:13:39.101 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000029", 2023-11-02T18:13:39.102 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000089", 2023-11-02T18:13:39.102 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000022", 2023-11-02T18:13:39.102 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:5", 2023-11-02T18:13:39.103 INFO:teuthology.orchestra.run.smithi017.stdout: "bilog.trim", 2023-11-02T18:13:39.103 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000016", 2023-11-02T18:13:39.103 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:2", 2023-11-02T18:13:39.103 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:9", 2023-11-02T18:13:39.104 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34", 2023-11-02T18:13:39.104 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000041", 2023-11-02T18:13:39.104 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:0", 2023-11-02T18:13:39.104 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:2", 2023-11-02T18:13:39.104 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000018", 2023-11-02T18:13:39.105 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000092", 2023-11-02T18:13:39.105 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31", 2023-11-02T18:13:39.105 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:1", 2023-11-02T18:13:39.105 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.4", 2023-11-02T18:13:39.105 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:3", 2023-11-02T18:13:39.106 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000014", 2023-11-02T18:13:39.106 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.39a01552-6d59-4ce7-b7e5-915ca69b979a.2", 2023-11-02T18:13:39.106 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:0", 2023-11-02T18:13:39.106 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:3", 2023-11-02T18:13:39.107 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000112", 2023-11-02T18:13:39.107 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:5", 2023-11-02T18:13:39.107 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000007", 2023-11-02T18:13:39.108 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000021", 2023-11-02T18:13:39.108 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:0", 2023-11-02T18:13:39.108 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000064", 2023-11-02T18:13:39.108 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000071", 2023-11-02T18:13:39.108 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:5", 2023-11-02T18:13:39.109 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:1", 2023-11-02T18:13:39.109 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:3", 2023-11-02T18:13:39.109 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000074", 2023-11-02T18:13:39.109 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:9", 2023-11-02T18:13:39.110 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:8", 2023-11-02T18:13:39.110 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000081", 2023-11-02T18:13:39.110 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000009", 2023-11-02T18:13:39.110 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000121", 2023-11-02T18:13:39.110 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.1", 2023-11-02T18:13:39.111 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8", 2023-11-02T18:13:39.111 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.39a01552-6d59-4ce7-b7e5-915ca69b979a.1", 2023-11-02T18:13:39.111 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000125", 2023-11-02T18:13:39.111 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000082", 2023-11-02T18:13:39.112 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:7", 2023-11-02T18:13:39.112 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:9", 2023-11-02T18:13:39.112 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000105", 2023-11-02T18:13:39.112 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000059", 2023-11-02T18:13:39.113 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:8", 2023-11-02T18:13:39.113 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000077", 2023-11-02T18:13:39.113 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000032", 2023-11-02T18:13:39.113 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:10", 2023-11-02T18:13:39.114 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:3", 2023-11-02T18:13:39.114 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:1", 2023-11-02T18:13:39.114 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:10", 2023-11-02T18:13:39.114 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:0", 2023-11-02T18:13:39.115 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000053", 2023-11-02T18:13:39.115 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000091", 2023-11-02T18:13:39.115 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:1", 2023-11-02T18:13:39.116 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T18:13:39.116 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000065", 2023-11-02T18:13:39.116 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:5", 2023-11-02T18:13:39.116 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000083", 2023-11-02T18:13:39.116 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000010", 2023-11-02T18:13:39.117 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:10", 2023-11-02T18:13:39.117 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:0", 2023-11-02T18:13:39.117 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:6", 2023-11-02T18:13:39.117 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.39a01552-6d59-4ce7-b7e5-915ca69b979a.3", 2023-11-02T18:13:39.118 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:6", 2023-11-02T18:13:39.118 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:8", 2023-11-02T18:13:39.118 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.7", 2023-11-02T18:13:39.118 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.3", 2023-11-02T18:13:39.119 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:10", 2023-11-02T18:13:39.119 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:7", 2023-11-02T18:13:39.119 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:7", 2023-11-02T18:13:39.119 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:7", 2023-11-02T18:13:39.119 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:5", 2023-11-02T18:13:39.120 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:4", 2023-11-02T18:13:39.120 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000045", 2023-11-02T18:13:39.120 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:9", 2023-11-02T18:13:39.120 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:4", 2023-11-02T18:13:39.121 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000002", 2023-11-02T18:13:39.121 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000116", 2023-11-02T18:13:39.121 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.3.l9wRXiy7xJW.0", 2023-11-02T18:13:39.121 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:7", 2023-11-02T18:13:39.121 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:3", 2023-11-02T18:13:39.122 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000034", 2023-11-02T18:13:39.122 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:10", 2023-11-02T18:13:39.122 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:3", 2023-11-02T18:13:39.122 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000101", 2023-11-02T18:13:39.123 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:4", 2023-11-02T18:13:39.123 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:4", 2023-11-02T18:13:39.123 INFO:teuthology.orchestra.run.smithi017.stdout: "meta.log.b3a94447-46df-4768-8340-cf63e4392a5c.2", 2023-11-02T18:13:39.123 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:9", 2023-11-02T18:13:39.124 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000079", 2023-11-02T18:13:39.124 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000049", 2023-11-02T18:13:39.124 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:4", 2023-11-02T18:13:39.124 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:8", 2023-11-02T18:13:39.124 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000117", 2023-11-02T18:13:39.125 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000044", 2023-11-02T18:13:39.125 INFO:teuthology.orchestra.run.smithi017.stdout: "meta.log.b3a94447-46df-4768-8340-cf63e4392a5c.0", 2023-11-02T18:13:39.125 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:3", 2023-11-02T18:13:39.125 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000066", 2023-11-02T18:13:39.126 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000068", 2023-11-02T18:13:39.126 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:4", 2023-11-02T18:13:39.126 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000085", 2023-11-02T18:13:39.126 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:2", 2023-11-02T18:13:39.126 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:6", 2023-11-02T18:13:39.127 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.0.EZJ79UkpVtS.0", 2023-11-02T18:13:39.127 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:1", 2023-11-02T18:13:39.127 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000073", 2023-11-02T18:13:39.128 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000038", 2023-11-02T18:13:39.128 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32", 2023-11-02T18:13:39.128 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000118", 2023-11-02T18:13:39.128 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:9", 2023-11-02T18:13:39.128 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000036", 2023-11-02T18:13:39.129 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000103", 2023-11-02T18:13:39.129 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:8", 2023-11-02T18:13:39.129 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:0", 2023-11-02T18:13:39.129 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:7", 2023-11-02T18:13:39.130 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000119", 2023-11-02T18:13:39.130 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000098", 2023-11-02T18:13:39.130 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:8", 2023-11-02T18:13:39.130 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:4", 2023-11-02T18:13:39.132 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:3", 2023-11-02T18:13:39.132 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:10", 2023-11-02T18:13:39.132 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000027", 2023-11-02T18:13:39.132 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:10", 2023-11-02T18:13:39.133 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000019", 2023-11-02T18:13:39.133 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:5", 2023-11-02T18:13:39.133 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000039", 2023-11-02T18:13:39.133 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:7", 2023-11-02T18:13:39.133 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:6", 2023-11-02T18:13:39.134 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000100", 2023-11-02T18:13:39.134 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:1", 2023-11-02T18:13:39.134 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:1", 2023-11-02T18:13:39.134 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.39a01552-6d59-4ce7-b7e5-915ca69b979a.0.retry", 2023-11-02T18:13:39.135 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:5", 2023-11-02T18:13:39.135 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:4", 2023-11-02T18:13:39.135 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:8", 2023-11-02T18:13:39.135 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000093", 2023-11-02T18:13:39.135 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:8", 2023-11-02T18:13:39.136 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000004", 2023-11-02T18:13:39.136 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000063", 2023-11-02T18:13:39.136 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000122", 2023-11-02T18:13:39.136 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:0", 2023-11-02T18:13:39.137 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000057", 2023-11-02T18:13:39.137 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:0", 2023-11-02T18:13:39.137 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:7", 2023-11-02T18:13:39.137 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:8", 2023-11-02T18:13:39.138 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:8", 2023-11-02T18:13:39.138 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000054", 2023-11-02T18:13:39.138 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000114", 2023-11-02T18:13:39.138 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11", 2023-11-02T18:13:39.138 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.2", 2023-11-02T18:13:39.139 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.0", 2023-11-02T18:13:39.139 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:5", 2023-11-02T18:13:39.139 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:10", 2023-11-02T18:13:39.139 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000012", 2023-11-02T18:13:39.140 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:7", 2023-11-02T18:13:39.140 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:6", 2023-11-02T18:13:39.140 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:1", 2023-11-02T18:13:39.140 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000084", 2023-11-02T18:13:39.140 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000043", 2023-11-02T18:13:39.141 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:2", 2023-11-02T18:13:39.141 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:2", 2023-11-02T18:13:39.141 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:7", 2023-11-02T18:13:39.141 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:1", 2023-11-02T18:13:39.142 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:9", 2023-11-02T18:13:39.142 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000111", 2023-11-02T18:13:39.142 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:6", 2023-11-02T18:13:39.142 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000048", 2023-11-02T18:13:39.142 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33", 2023-11-02T18:13:39.143 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:3", 2023-11-02T18:13:39.143 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000020", 2023-11-02T18:13:39.143 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:3", 2023-11-02T18:13:39.143 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000099", 2023-11-02T18:13:39.144 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000056", 2023-11-02T18:13:39.144 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:8", 2023-11-02T18:13:39.144 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12", 2023-11-02T18:13:39.144 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:0", 2023-11-02T18:13:39.145 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:0", 2023-11-02T18:13:39.145 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:8", 2023-11-02T18:13:39.145 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:8", 2023-11-02T18:13:39.145 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000072", 2023-11-02T18:13:39.145 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000062", 2023-11-02T18:13:39.146 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000115", 2023-11-02T18:13:39.146 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:5", 2023-11-02T18:13:39.146 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.1", 2023-11-02T18:13:39.146 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000033", 2023-11-02T18:13:39.147 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:9", 2023-11-02T18:13:39.147 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:10", 2023-11-02T18:13:39.147 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:3", 2023-11-02T18:13:39.147 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16", 2023-11-02T18:13:39.147 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:1", 2023-11-02T18:13:39.148 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:10", 2023-11-02T18:13:39.148 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000025", 2023-11-02T18:13:39.148 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000123", 2023-11-02T18:13:39.148 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18", 2023-11-02T18:13:39.149 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9", 2023-11-02T18:13:39.149 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:2", 2023-11-02T18:13:39.149 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:6", 2023-11-02T18:13:39.149 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000108", 2023-11-02T18:13:39.150 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:3", 2023-11-02T18:13:39.150 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000094", 2023-11-02T18:13:39.150 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17", 2023-11-02T18:13:39.150 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000060", 2023-11-02T18:13:39.151 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:1", 2023-11-02T18:13:39.151 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000109", 2023-11-02T18:13:39.151 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:2", 2023-11-02T18:13:39.151 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:2", 2023-11-02T18:13:39.151 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000013", 2023-11-02T18:13:39.152 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:2", 2023-11-02T18:13:39.152 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:1", 2023-11-02T18:13:39.152 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7", 2023-11-02T18:13:39.152 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:3", 2023-11-02T18:13:39.153 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:0", 2023-11-02T18:13:39.153 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000051", 2023-11-02T18:13:39.153 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:10", 2023-11-02T18:13:39.153 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000106", 2023-11-02T18:13:39.153 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:9", 2023-11-02T18:13:39.154 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.39a01552-6d59-4ce7-b7e5-915ca69b979a.3.retry", 2023-11-02T18:13:39.154 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000017", 2023-11-02T18:13:39.154 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:10", 2023-11-02T18:13:39.154 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:3", 2023-11-02T18:13:39.155 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:2", 2023-11-02T18:13:39.155 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:9", 2023-11-02T18:13:39.155 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13", 2023-11-02T18:13:39.155 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:7", 2023-11-02T18:13:39.155 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000023", 2023-11-02T18:13:39.156 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:4", 2023-11-02T18:13:39.156 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000107", 2023-11-02T18:13:39.156 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:4", 2023-11-02T18:13:39.156 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000046", 2023-11-02T18:13:39.157 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:9", 2023-11-02T18:13:39.157 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:2", 2023-11-02T18:13:39.157 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:4", 2023-11-02T18:13:39.157 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.2", 2023-11-02T18:13:39.157 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000067", 2023-11-02T18:13:39.158 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:5", 2023-11-02T18:13:39.158 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.9", 2023-11-02T18:13:39.158 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000086", 2023-11-02T18:13:39.158 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:7", 2023-11-02T18:13:39.159 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.5", 2023-11-02T18:13:39.159 INFO:teuthology.orchestra.run.smithi017.stdout: "meta.log.b3a94447-46df-4768-8340-cf63e4392a5c.3", 2023-11-02T18:13:39.159 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:0", 2023-11-02T18:13:39.159 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:5", 2023-11-02T18:13:39.160 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:5", 2023-11-02T18:13:39.160 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.39a01552-6d59-4ce7-b7e5-915ca69b979a.0", 2023-11-02T18:13:39.160 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:7", 2023-11-02T18:13:39.160 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:5", 2023-11-02T18:13:39.160 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000050", 2023-11-02T18:13:39.161 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:6", 2023-11-02T18:13:39.161 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000102", 2023-11-02T18:13:39.161 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:5", 2023-11-02T18:13:39.161 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:9", 2023-11-02T18:13:39.162 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:8", 2023-11-02T18:13:39.162 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000124", 2023-11-02T18:13:39.162 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000001", 2023-11-02T18:13:39.162 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:2", 2023-11-02T18:13:39.163 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:7", 2023-11-02T18:13:39.163 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:4", 2023-11-02T18:13:39.163 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:0", 2023-11-02T18:13:39.163 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:6", 2023-11-02T18:13:39.163 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000126", 2023-11-02T18:13:39.164 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000055", 2023-11-02T18:13:39.164 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15", 2023-11-02T18:13:39.164 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:5", 2023-11-02T18:13:39.164 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:0", 2023-11-02T18:13:39.165 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:2", 2023-11-02T18:13:39.165 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000061", 2023-11-02T18:13:39.165 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000097", 2023-11-02T18:13:39.165 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:9", 2023-11-02T18:13:39.166 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000042", 2023-11-02T18:13:39.166 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:8", 2023-11-02T18:13:39.166 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1", 2023-11-02T18:13:39.166 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:4", 2023-11-02T18:13:39.166 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:6", 2023-11-02T18:13:39.167 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000075", 2023-11-02T18:13:39.167 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:9", 2023-11-02T18:13:39.167 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000120", 2023-11-02T18:13:39.167 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000030", 2023-11-02T18:13:39.168 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:9" 2023-11-02T18:13:39.168 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:13:39.168 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T18:13:39.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:39.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:39.170 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:39.215 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:39.215 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:39.317 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:13:39.317 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000078", 2023-11-02T18:13:39.318 INFO:teuthology.orchestra.run.smithi017.stdout: "meta.history", 2023-11-02T18:13:39.318 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:10", 2023-11-02T18:13:39.318 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000070", 2023-11-02T18:13:39.318 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:0", 2023-11-02T18:13:39.319 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000104", 2023-11-02T18:13:39.319 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:8", 2023-11-02T18:13:39.319 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000026", 2023-11-02T18:13:39.319 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:0", 2023-11-02T18:13:39.320 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:10", 2023-11-02T18:13:39.320 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11", 2023-11-02T18:13:39.320 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.8", 2023-11-02T18:13:39.320 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:3", 2023-11-02T18:13:39.321 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000028", 2023-11-02T18:13:39.321 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13", 2023-11-02T18:13:39.321 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:5", 2023-11-02T18:13:39.321 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:8", 2023-11-02T18:13:39.321 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:2", 2023-11-02T18:13:39.322 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:3", 2023-11-02T18:13:39.322 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:9", 2023-11-02T18:13:39.322 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:8", 2023-11-02T18:13:39.322 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:2", 2023-11-02T18:13:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18", 2023-11-02T18:13:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:3", 2023-11-02T18:13:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:0", 2023-11-02T18:13:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000040", 2023-11-02T18:13:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.6", 2023-11-02T18:13:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000015", 2023-11-02T18:13:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:6", 2023-11-02T18:13:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000069", 2023-11-02T18:13:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000095", 2023-11-02T18:13:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000003", 2023-11-02T18:13:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000047", 2023-11-02T18:13:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.3.726zz1L8x22.0", 2023-11-02T18:13:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:6", 2023-11-02T18:13:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000035", 2023-11-02T18:13:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:10", 2023-11-02T18:13:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000037", 2023-11-02T18:13:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:3", 2023-11-02T18:13:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:5", 2023-11-02T18:13:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:1", 2023-11-02T18:13:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000024", 2023-11-02T18:13:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:9", 2023-11-02T18:13:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:5", 2023-11-02T18:13:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000000", 2023-11-02T18:13:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:9", 2023-11-02T18:13:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000031", 2023-11-02T18:13:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:6", 2023-11-02T18:13:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.3", 2023-11-02T18:13:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:8", 2023-11-02T18:13:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:4", 2023-11-02T18:13:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:8", 2023-11-02T18:13:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:3", 2023-11-02T18:13:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:8", 2023-11-02T18:13:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:2", 2023-11-02T18:13:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:6", 2023-11-02T18:13:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: "data_loggenerations_metadata", 2023-11-02T18:13:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:0", 2023-11-02T18:13:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:9", 2023-11-02T18:13:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000076", 2023-11-02T18:13:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000113", 2023-11-02T18:13:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000005", 2023-11-02T18:13:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.d61b2bea-59a8-4454-8491-ac3afbd04937.2", 2023-11-02T18:13:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000011", 2023-11-02T18:13:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000058", 2023-11-02T18:13:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000052", 2023-11-02T18:13:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:7", 2023-11-02T18:13:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:6", 2023-11-02T18:13:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:7", 2023-11-02T18:13:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:1", 2023-11-02T18:13:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000088", 2023-11-02T18:13:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:5", 2023-11-02T18:13:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000080", 2023-11-02T18:13:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000090", 2023-11-02T18:13:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8", 2023-11-02T18:13:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000110", 2023-11-02T18:13:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:1", 2023-11-02T18:13:39.335 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000096", 2023-11-02T18:13:39.335 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:9", 2023-11-02T18:13:39.335 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000087", 2023-11-02T18:13:39.335 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:5", 2023-11-02T18:13:39.335 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000008", 2023-11-02T18:13:39.336 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:8", 2023-11-02T18:13:39.336 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:5", 2023-11-02T18:13:39.336 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:3", 2023-11-02T18:13:39.336 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000006", 2023-11-02T18:13:39.337 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:9", 2023-11-02T18:13:39.337 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000029", 2023-11-02T18:13:39.337 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:7", 2023-11-02T18:13:39.337 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:2", 2023-11-02T18:13:39.337 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:1", 2023-11-02T18:13:39.338 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:7", 2023-11-02T18:13:39.338 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000089", 2023-11-02T18:13:39.338 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000022", 2023-11-02T18:13:39.338 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:8", 2023-11-02T18:13:39.339 INFO:teuthology.orchestra.run.smithi017.stdout: "bilog.trim", 2023-11-02T18:13:39.339 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000016", 2023-11-02T18:13:39.339 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.d61b2bea-59a8-4454-8491-ac3afbd04937.0.retry", 2023-11-02T18:13:39.339 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000041", 2023-11-02T18:13:39.340 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.0.SuZpEvJoonv.0", 2023-11-02T18:13:39.340 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9", 2023-11-02T18:13:39.340 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000018", 2023-11-02T18:13:39.340 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:5", 2023-11-02T18:13:39.340 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000092", 2023-11-02T18:13:39.341 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:3", 2023-11-02T18:13:39.341 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.4", 2023-11-02T18:13:39.341 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000014", 2023-11-02T18:13:39.341 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:4", 2023-11-02T18:13:39.342 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:8", 2023-11-02T18:13:39.342 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:5", 2023-11-02T18:13:39.342 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000112", 2023-11-02T18:13:39.342 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000007", 2023-11-02T18:13:39.342 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000021", 2023-11-02T18:13:39.343 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.16", 2023-11-02T18:13:39.343 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000064", 2023-11-02T18:13:39.343 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000071", 2023-11-02T18:13:39.343 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000074", 2023-11-02T18:13:39.344 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:4", 2023-11-02T18:13:39.345 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:7", 2023-11-02T18:13:39.345 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:10", 2023-11-02T18:13:39.345 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:6", 2023-11-02T18:13:39.345 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:10", 2023-11-02T18:13:39.345 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:4", 2023-11-02T18:13:39.346 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:1", 2023-11-02T18:13:39.346 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:4", 2023-11-02T18:13:39.346 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000081", 2023-11-02T18:13:39.346 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:9", 2023-11-02T18:13:39.347 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000009", 2023-11-02T18:13:39.347 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000121", 2023-11-02T18:13:39.347 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.1", 2023-11-02T18:13:39.347 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31", 2023-11-02T18:13:39.348 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.17", 2023-11-02T18:13:39.348 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:0", 2023-11-02T18:13:39.348 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000125", 2023-11-02T18:13:39.348 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000082", 2023-11-02T18:13:39.348 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:9", 2023-11-02T18:13:39.349 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:3", 2023-11-02T18:13:39.349 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:5", 2023-11-02T18:13:39.349 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:2", 2023-11-02T18:13:39.349 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000105", 2023-11-02T18:13:39.350 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000059", 2023-11-02T18:13:39.350 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000077", 2023-11-02T18:13:39.350 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:2", 2023-11-02T18:13:39.350 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000032", 2023-11-02T18:13:39.350 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34", 2023-11-02T18:13:39.351 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:10", 2023-11-02T18:13:39.351 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:4", 2023-11-02T18:13:39.351 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:0", 2023-11-02T18:13:39.351 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000053", 2023-11-02T18:13:39.351 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000091", 2023-11-02T18:13:39.352 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:6", 2023-11-02T18:13:39.352 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000065", 2023-11-02T18:13:39.352 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:3", 2023-11-02T18:13:39.352 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:10", 2023-11-02T18:13:39.353 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000083", 2023-11-02T18:13:39.353 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000010", 2023-11-02T18:13:39.353 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:9", 2023-11-02T18:13:39.353 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:10", 2023-11-02T18:13:39.353 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.7", 2023-11-02T18:13:39.354 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.3", 2023-11-02T18:13:39.354 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:7", 2023-11-02T18:13:39.354 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:3", 2023-11-02T18:13:39.354 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:8", 2023-11-02T18:13:39.355 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33", 2023-11-02T18:13:39.355 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:1", 2023-11-02T18:13:39.355 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:0", 2023-11-02T18:13:39.355 INFO:teuthology.orchestra.run.smithi017.stdout: "mdlog.sync-status.shard.2", 2023-11-02T18:13:39.355 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:2", 2023-11-02T18:13:39.356 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000045", 2023-11-02T18:13:39.356 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:10", 2023-11-02T18:13:39.356 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000002", 2023-11-02T18:13:39.356 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.12", 2023-11-02T18:13:39.356 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:4", 2023-11-02T18:13:39.357 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:9", 2023-11-02T18:13:39.357 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:7", 2023-11-02T18:13:39.357 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000116", 2023-11-02T18:13:39.357 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.d61b2bea-59a8-4454-8491-ac3afbd04937.0", 2023-11-02T18:13:39.358 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000034", 2023-11-02T18:13:39.358 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000101", 2023-11-02T18:13:39.358 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.13", 2023-11-02T18:13:39.358 INFO:teuthology.orchestra.run.smithi017.stdout: "mdlog.sync-status.shard.1", 2023-11-02T18:13:39.359 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:4", 2023-11-02T18:13:39.359 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:1", 2023-11-02T18:13:39.359 INFO:teuthology.orchestra.run.smithi017.stdout: "meta.log.b3a94447-46df-4768-8340-cf63e4392a5c.2", 2023-11-02T18:13:39.359 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000079", 2023-11-02T18:13:39.359 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000049", 2023-11-02T18:13:39.360 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:7", 2023-11-02T18:13:39.360 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000117", 2023-11-02T18:13:39.360 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000044", 2023-11-02T18:13:39.360 INFO:teuthology.orchestra.run.smithi017.stdout: "meta.log.b3a94447-46df-4768-8340-cf63e4392a5c.0", 2023-11-02T18:13:39.361 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.14", 2023-11-02T18:13:39.361 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000066", 2023-11-02T18:13:39.361 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:6", 2023-11-02T18:13:39.361 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:7", 2023-11-02T18:13:39.361 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000068", 2023-11-02T18:13:39.362 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:1", 2023-11-02T18:13:39.362 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000085", 2023-11-02T18:13:39.362 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:5", 2023-11-02T18:13:39.362 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:7", 2023-11-02T18:13:39.363 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:4", 2023-11-02T18:13:39.363 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000073", 2023-11-02T18:13:39.363 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:8", 2023-11-02T18:13:39.363 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000038", 2023-11-02T18:13:39.363 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:6", 2023-11-02T18:13:39.364 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:3", 2023-11-02T18:13:39.364 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.1.9nQ9eZiofhn.0", 2023-11-02T18:13:39.364 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:4", 2023-11-02T18:13:39.364 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.10", 2023-11-02T18:13:39.365 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000118", 2023-11-02T18:13:39.365 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:0", 2023-11-02T18:13:39.365 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:10", 2023-11-02T18:13:39.365 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15", 2023-11-02T18:13:39.365 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000036", 2023-11-02T18:13:39.366 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:2", 2023-11-02T18:13:39.366 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000103", 2023-11-02T18:13:39.366 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:6", 2023-11-02T18:13:39.366 INFO:teuthology.orchestra.run.smithi017.stdout: "mdlog.sync-status.shard.3", 2023-11-02T18:13:39.367 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:5", 2023-11-02T18:13:39.367 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:1", 2023-11-02T18:13:39.367 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000119", 2023-11-02T18:13:39.368 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:8", 2023-11-02T18:13:39.368 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000098", 2023-11-02T18:13:39.369 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:6", 2023-11-02T18:13:39.369 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7", 2023-11-02T18:13:39.369 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:4", 2023-11-02T18:13:39.369 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:0", 2023-11-02T18:13:39.369 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:0", 2023-11-02T18:13:39.370 INFO:teuthology.orchestra.run.smithi017.stdout: "mdlog.sync-status.shard.0", 2023-11-02T18:13:39.370 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:6", 2023-11-02T18:13:39.370 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:1", 2023-11-02T18:13:39.370 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000027", 2023-11-02T18:13:39.371 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:1", 2023-11-02T18:13:39.371 INFO:teuthology.orchestra.run.smithi017.stdout: "mdlog.sync-status", 2023-11-02T18:13:39.371 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000019", 2023-11-02T18:13:39.371 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:8", 2023-11-02T18:13:39.372 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000039", 2023-11-02T18:13:39.372 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:2", 2023-11-02T18:13:39.372 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14", 2023-11-02T18:13:39.372 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:10", 2023-11-02T18:13:39.372 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17", 2023-11-02T18:13:39.373 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.d61b2bea-59a8-4454-8491-ac3afbd04937.3.retry", 2023-11-02T18:13:39.373 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:3", 2023-11-02T18:13:39.373 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.d61b2bea-59a8-4454-8491-ac3afbd04937.1.retry", 2023-11-02T18:13:39.373 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:7", 2023-11-02T18:13:39.374 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000100", 2023-11-02T18:13:39.374 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:5", 2023-11-02T18:13:39.374 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000093", 2023-11-02T18:13:39.374 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12", 2023-11-02T18:13:39.375 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-9:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.9:6", 2023-11-02T18:13:39.375 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:9", 2023-11-02T18:13:39.375 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000004", 2023-11-02T18:13:39.375 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:8", 2023-11-02T18:13:39.375 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000063", 2023-11-02T18:13:39.376 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:6", 2023-11-02T18:13:39.376 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000122", 2023-11-02T18:13:39.376 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000057", 2023-11-02T18:13:39.376 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:4", 2023-11-02T18:13:39.377 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:0", 2023-11-02T18:13:39.377 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000054", 2023-11-02T18:13:39.377 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000114", 2023-11-02T18:13:39.377 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.2", 2023-11-02T18:13:39.378 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.0", 2023-11-02T18:13:39.378 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000012", 2023-11-02T18:13:39.378 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:0", 2023-11-02T18:13:39.378 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1", 2023-11-02T18:13:39.378 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:8", 2023-11-02T18:13:39.379 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000084", 2023-11-02T18:13:39.379 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000043", 2023-11-02T18:13:39.379 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-35:d61b2bea-59a8-4454-8491-ac3afbd04937.5401.1:10", 2023-11-02T18:13:39.379 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.11", 2023-11-02T18:13:39.379 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:3", 2023-11-02T18:13:39.380 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:5", 2023-11-02T18:13:39.380 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:4", 2023-11-02T18:13:39.380 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000111", 2023-11-02T18:13:39.380 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32", 2023-11-02T18:13:39.381 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000048", 2023-11-02T18:13:39.381 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000020", 2023-11-02T18:13:39.381 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:2", 2023-11-02T18:13:39.381 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:5", 2023-11-02T18:13:39.381 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000099", 2023-11-02T18:13:39.382 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000056", 2023-11-02T18:13:39.382 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.19", 2023-11-02T18:13:39.382 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:10", 2023-11-02T18:13:39.382 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:2", 2023-11-02T18:13:39.383 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:2", 2023-11-02T18:13:39.383 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000072", 2023-11-02T18:13:39.383 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:0", 2023-11-02T18:13:39.383 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000062", 2023-11-02T18:13:39.384 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:9", 2023-11-02T18:13:39.384 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:10", 2023-11-02T18:13:39.384 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000115", 2023-11-02T18:13:39.384 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.1", 2023-11-02T18:13:39.384 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000033", 2023-11-02T18:13:39.385 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:7", 2023-11-02T18:13:39.385 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000025", 2023-11-02T18:13:39.385 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000123", 2023-11-02T18:13:39.385 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:3", 2023-11-02T18:13:39.386 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:2", 2023-11-02T18:13:39.386 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16", 2023-11-02T18:13:39.386 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.18", 2023-11-02T18:13:39.386 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000108", 2023-11-02T18:13:39.386 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000094", 2023-11-02T18:13:39.387 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000060", 2023-11-02T18:13:39.387 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.d61b2bea-59a8-4454-8491-ac3afbd04937.2.retry", 2023-11-02T18:13:39.387 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000109", 2023-11-02T18:13:39.387 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.15", 2023-11-02T18:13:39.388 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:0", 2023-11-02T18:13:39.388 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000013", 2023-11-02T18:13:39.388 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:7", 2023-11-02T18:13:39.388 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:0", 2023-11-02T18:13:39.388 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:5", 2023-11-02T18:13:39.389 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000051", 2023-11-02T18:13:39.389 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000106", 2023-11-02T18:13:39.389 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:8", 2023-11-02T18:13:39.389 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:4", 2023-11-02T18:13:39.390 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:10", 2023-11-02T18:13:39.390 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000017", 2023-11-02T18:13:39.390 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:9", 2023-11-02T18:13:39.390 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000023", 2023-11-02T18:13:39.391 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:6", 2023-11-02T18:13:39.391 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:9", 2023-11-02T18:13:39.391 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000107", 2023-11-02T18:13:39.391 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000046", 2023-11-02T18:13:39.391 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-14:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.14:1", 2023-11-02T18:13:39.392 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:3", 2023-11-02T18:13:39.392 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:2", 2023-11-02T18:13:39.392 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:8", 2023-11-02T18:13:39.392 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:5", 2023-11-02T18:13:39.393 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:4", 2023-11-02T18:13:39.393 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-16:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.16:3", 2023-11-02T18:13:39.393 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:7", 2023-11-02T18:13:39.393 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:7", 2023-11-02T18:13:39.393 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:1", 2023-11-02T18:13:39.394 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-31:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.31:1", 2023-11-02T18:13:39.394 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:9", 2023-11-02T18:13:39.394 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:6", 2023-11-02T18:13:39.394 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.2", 2023-11-02T18:13:39.395 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000067", 2023-11-02T18:13:39.395 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10", 2023-11-02T18:13:39.395 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-8:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.8:10", 2023-11-02T18:13:39.395 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.9", 2023-11-02T18:13:39.395 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000086", 2023-11-02T18:13:39.396 INFO:teuthology.orchestra.run.smithi017.stdout: "sync.error-log.5", 2023-11-02T18:13:39.396 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:3", 2023-11-02T18:13:39.396 INFO:teuthology.orchestra.run.smithi017.stdout: "meta.log.b3a94447-46df-4768-8340-cf63e4392a5c.3", 2023-11-02T18:13:39.396 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:0", 2023-11-02T18:13:39.397 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:9", 2023-11-02T18:13:39.397 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.d61b2bea-59a8-4454-8491-ac3afbd04937.3", 2023-11-02T18:13:39.397 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-33:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.33:1", 2023-11-02T18:13:39.397 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:9", 2023-11-02T18:13:39.397 INFO:teuthology.orchestra.run.smithi017.stdout: "data_log.2.dVFJHzP0tZK.0", 2023-11-02T18:13:39.398 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T18:13:39.398 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:5", 2023-11-02T18:13:39.398 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-15:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.15:10", 2023-11-02T18:13:39.398 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-18:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.18:1", 2023-11-02T18:13:39.399 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000050", 2023-11-02T18:13:39.399 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000102", 2023-11-02T18:13:39.399 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-10:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.10:6", 2023-11-02T18:13:39.399 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-11:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.11:2", 2023-11-02T18:13:39.399 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000124", 2023-11-02T18:13:39.400 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:4", 2023-11-02T18:13:39.400 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-7:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.7:4", 2023-11-02T18:13:39.400 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000001", 2023-11-02T18:13:39.400 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-13:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.13:7", 2023-11-02T18:13:39.400 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000126", 2023-11-02T18:13:39.401 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000055", 2023-11-02T18:13:39.401 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000061", 2023-11-02T18:13:39.401 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:0", 2023-11-02T18:13:39.401 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000097", 2023-11-02T18:13:39.402 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-12:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.12:2", 2023-11-02T18:13:39.402 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-17:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.17:7", 2023-11-02T18:13:39.402 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000042", 2023-11-02T18:13:39.402 INFO:teuthology.orchestra.run.smithi017.stdout: "datalog.sync-status.shard.d61b2bea-59a8-4454-8491-ac3afbd04937.1", 2023-11-02T18:13:39.403 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-34:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.34:1", 2023-11-02T18:13:39.403 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000075", 2023-11-02T18:13:39.403 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000120", 2023-11-02T18:13:39.403 INFO:teuthology.orchestra.run.smithi017.stdout: "obj_delete_at_hint.0000000030", 2023-11-02T18:13:39.403 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-32:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.32:2" 2023-11-02T18:13:39.404 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:13:39.404 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-11-02T18:13:39.405 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-11-02T18:13:39.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:39.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:39.407 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:39.551 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:13:39.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:39.552 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001211", 2023-11-02T18:13:39.552 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:13:38.849594Z" 2023-11-02T18:13:39.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:39.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:39.553 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004308", 2023-11-02T18:13:39.553 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:13:38.851900Z" 2023-11-02T18:13:39.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:39.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:39.554 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000003467", 2023-11-02T18:13:39.554 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:13:38.853992Z" 2023-11-02T18:13:39.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:39.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:39.555 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-11-02T18:13:39.555 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:13:38.856336Z" 2023-11-02T18:13:39.555 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:39.555 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:13:39.556 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004308', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-11-02T18:13:39.557 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:13:39.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:39.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:39.558 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:39.644 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:39.644 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:39.732 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:13:39.732 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:13:39.733 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:13:39.733 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:13:39.733 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:13:39.733 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 14562503763241451158 2023-11-02T18:13:39.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:39.734 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:13:39.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:39.734 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:13:39.735 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:39.735 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:13:39.735 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:13:39.735 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:39.735 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:39.736 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:39.736 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:00.706996Z" 2023-11-02T18:13:39.736 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:39.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:39.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:39.737 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:13:39.737 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:39.737 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:13:39.738 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000003465", 2023-11-02T18:13:39.738 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:39.738 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:39.738 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:39.738 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:05:01.857110Z" 2023-11-02T18:13:39.739 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:39.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:39.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:39.739 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:13:39.740 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:39.740 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:13:39.740 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-11-02T18:13:39.740 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:39.740 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:39.741 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:39.741 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:08:30.240395Z" 2023-11-02T18:13:39.741 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:39.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:39.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:39.742 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:13:39.742 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:39.742 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:13:39.742 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-11-02T18:13:39.743 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:39.743 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:39.743 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:39.743 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:05:00.792295Z" 2023-11-02T18:13:39.744 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:39.744 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:39.744 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:13:39.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:39.744 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:13:39.745 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:13:39.745 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:13:39.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:39.745 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:13:39Z" 2023-11-02T18:13:39.746 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:13:39.747 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14562503763241451158 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:00.706996Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:05:01.857110Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002625", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:08:30.240395Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:05:00.792295Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:13:39Z" } 2023-11-02T18:13:39.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:00.706996Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:05:01.857110Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:08:30.240395Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:05:00.792295Z'}}] 2023-11-02T18:13:39.748 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004308', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-11-02T18:13:39.748 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003465', 2: '00000000000000000000:00000000000000002625', 3: '00000000000000000000:00000000000000001210'} 2023-11-02T18:13:39.749 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000001211 target=00000000000000000000:00000000000000000651, shard=1 master=00000000000000000000:00000000000000004308 target=00000000000000000000:00000000000000003465, shard=2 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002625, shard=3 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210 2023-11-02T18:13:44.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:44.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:44.751 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:44.799 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:44.799 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:44.892 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:13:44.893 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:13:44.893 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:13:44.893 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:13:44.893 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:13:44.894 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 14562503763241451158 2023-11-02T18:13:44.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:44.894 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:13:44.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:44.895 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:13:44.895 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:44.895 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:13:44.895 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001211", 2023-11-02T18:13:44.896 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:44.896 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:44.896 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:44.896 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:13:38.849594Z" 2023-11-02T18:13:44.896 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:44.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:44.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:44.897 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:13:44.897 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:44.898 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:13:44.898 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004308", 2023-11-02T18:13:44.898 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:44.898 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:44.899 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:44.899 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:13:38.851900Z" 2023-11-02T18:13:44.899 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:44.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:44.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:44.900 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:13:44.900 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:44.900 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:13:44.900 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000003467", 2023-11-02T18:13:44.900 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:44.901 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:44.901 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:44.901 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:13:38.853992Z" 2023-11-02T18:13:44.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:44.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:44.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:44.902 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:13:44.902 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:44.903 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:13:44.903 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-11-02T18:13:44.903 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:44.903 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:44.903 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:44.904 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:13:38.856336Z" 2023-11-02T18:13:44.904 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:44.904 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:44.904 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:13:44.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:44.905 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:13:44.905 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:13:44.905 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:13:44.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:44.906 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:13:44Z" 2023-11-02T18:13:44.906 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:13:44.907 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14562503763241451158 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:13:38.849594Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004308", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:13:38.851900Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003467", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:13:38.853992Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002055", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:13:38.856336Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:13:44Z" } 2023-11-02T18:13:44.907 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:13:38.849594Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004308', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:13:38.851900Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003467', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:13:38.853992Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002055', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:13:38.856336Z'}}] 2023-11-02T18:13:44.908 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004308', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-11-02T18:13:44.908 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004308', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-11-02T18:13:44.909 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:13:44.909 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-11-02T18:13:44.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:44.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:44.911 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:44.956 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:44.957 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:45.064 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:13:45.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.065 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:13:45.065 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:52:00.715412Z" 2023-11-02T18:13:45.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.065 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-11-02T18:13:45.066 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:05:01.920415Z" 2023-11-02T18:13:45.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.066 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001781", 2023-11-02T18:13:45.067 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:08:30.498244Z" 2023-11-02T18:13:45.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.067 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-11-02T18:13:45.067 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:05:00.782707Z" 2023-11-02T18:13:45.068 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:45.068 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:13:45.069 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004170', 2: '00000000000000000000:00000000000000001781', 3: '00000000000000000000:00000000000000001210'} 2023-11-02T18:13:45.069 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:13:45.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:45.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:45.071 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:45.156 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:45.156 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:45.243 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:13:45.244 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:13:45.244 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:13:45.244 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:13:45.244 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:13:45.245 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:13:45.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.245 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:13:45.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.246 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:13:45.246 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:45.246 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:13:45.246 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-11-02T18:13:45.247 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:45.247 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:45.247 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:45.247 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:52:00.715412Z" 2023-11-02T18:13:45.247 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:45.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.248 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:13:45.248 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:45.248 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:13:45.249 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-11-02T18:13:45.249 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:45.249 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:45.249 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:45.249 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:05:01.920415Z" 2023-11-02T18:13:45.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:45.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.250 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:13:45.251 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:45.251 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:13:45.251 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001781", 2023-11-02T18:13:45.251 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:45.251 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:45.252 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:45.252 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:08:30.498244Z" 2023-11-02T18:13:45.252 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:45.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.253 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:13:45.253 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:45.253 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:13:45.253 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-11-02T18:13:45.254 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:45.254 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:45.254 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:45.254 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:05:00.782707Z" 2023-11-02T18:13:45.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:45.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:45.255 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:13:45.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.255 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:13:45.256 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:13:45.256 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:13:45.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.256 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:13:45Z" 2023-11-02T18:13:45.257 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:13:45.258 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:52:00.715412Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:05:01.920415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001781", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:08:30.498244Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:05:00.782707Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:13:45Z" } 2023-11-02T18:13:45.258 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:52:00.715412Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:05:01.920415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001781', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:08:30.498244Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:05:00.782707Z'}}] 2023-11-02T18:13:45.259 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004170', 2: '00000000000000000000:00000000000000001781', 3: '00000000000000000000:00000000000000001210'} 2023-11-02T18:13:45.259 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004170', 2: '00000000000000000000:00000000000000001781', 3: '00000000000000000000:00000000000000001210'} 2023-11-02T18:13:45.259 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:13:45.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2023-11-02T18:13:45.261 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-46 2023-11-02T18:13:45.262 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:13:45.262 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:13:45.263 DEBUG:boto:Method: PUT 2023-11-02T18:13:45.263 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:13:45.263 DEBUG:boto:Data: 2023-11-02T18:13:45.264 DEBUG:boto:Headers: {} 2023-11-02T18:13:45.264 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:13:45.264 DEBUG:boto:Port: 8000 2023-11-02T18:13:45.265 DEBUG:boto:Params: {} 2023-11-02T18:13:45.265 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:13:45.266 DEBUG:boto:Token: None 2023-11-02T18:13:45.267 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-46/ 2023-11-02T18:13:45.267 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Himvuc56cQ8XYVMsJife5CPjlvY= 2023-11-02T18:13:45.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Himvuc56cQ8XYVMsJife5CPjlvY=', 'Content-Length': '0'} 2023-11-02T18:13:45.300 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b13f62fc2bfc53d7-006543e6d9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.301 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-47 2023-11-02T18:13:45.301 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:13:45.301 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:13:45.302 DEBUG:boto:Method: PUT 2023-11-02T18:13:45.302 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:13:45.303 DEBUG:boto:Data: 2023-11-02T18:13:45.303 DEBUG:boto:Headers: {} 2023-11-02T18:13:45.304 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:13:45.304 DEBUG:boto:Port: 8001 2023-11-02T18:13:45.305 DEBUG:boto:Params: {} 2023-11-02T18:13:45.305 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:13:45.306 DEBUG:boto:Token: None 2023-11-02T18:13:45.306 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-47/ 2023-11-02T18:13:45.307 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fwbep4z2ZZFnMK6B2o2/gEFVXG4= 2023-11-02T18:13:45.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fwbep4z2ZZFnMK6B2o2/gEFVXG4=', 'Content-Length': '0'} 2023-11-02T18:13:45.365 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000044d32461f0a7f785-006543e6d9-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.366 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:13:45.366 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:13:45.367 DEBUG:boto:Method: HEAD 2023-11-02T18:13:45.367 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:13:45.368 DEBUG:boto:Data: 2023-11-02T18:13:45.368 DEBUG:boto:Headers: {} 2023-11-02T18:13:45.368 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:13:45.369 DEBUG:boto:Port: 8000 2023-11-02T18:13:45.369 DEBUG:boto:Params: {} 2023-11-02T18:13:45.370 DEBUG:boto:Token: None 2023-11-02T18:13:45.370 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-46/ 2023-11-02T18:13:45.371 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZhqH348SPbJpbazgeWneiStkCwA= 2023-11-02T18:13:45.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZhqH348SPbJpbazgeWneiStkCwA=', 'Content-Length': '0'} 2023-11-02T18:13:45.376 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000075735723f8f1221-006543e6d9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.377 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:13:45.378 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:13:45.378 DEBUG:boto:Method: PUT 2023-11-02T18:13:45.378 DEBUG:boto:Path: /ibtljn-46/obj1 2023-11-02T18:13:45.379 DEBUG:boto:Data: 2023-11-02T18:13:45.379 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:13:45.380 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:13:45.380 DEBUG:boto:Port: 8000 2023-11-02T18:13:45.380 DEBUG:boto:Params: {} 2023-11-02T18:13:45.381 DEBUG:boto:Token: None 2023-11-02T18:13:45.381 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-46/obj1 2023-11-02T18:13:45.382 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QfU38jBkkQc7zUZXnMSTVzI1BXg= 2023-11-02T18:13:45.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QfU38jBkkQc7zUZXnMSTVzI1BXg='} 2023-11-02T18:13:45.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001857e0a0015928de-006543e6d9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.391 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:13:45.391 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:13:45.392 DEBUG:boto:Method: HEAD 2023-11-02T18:13:45.392 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:13:45.392 DEBUG:boto:Data: 2023-11-02T18:13:45.393 DEBUG:boto:Headers: {} 2023-11-02T18:13:45.393 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:13:45.394 DEBUG:boto:Port: 8000 2023-11-02T18:13:45.394 DEBUG:boto:Params: {} 2023-11-02T18:13:45.395 DEBUG:boto:Token: None 2023-11-02T18:13:45.395 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-46/ 2023-11-02T18:13:45.395 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZhqH348SPbJpbazgeWneiStkCwA= 2023-11-02T18:13:45.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZhqH348SPbJpbazgeWneiStkCwA=', 'Content-Length': '0'} 2023-11-02T18:13:45.399 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d7980d6320e67fb-006543e6d9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.399 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:13:45.400 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:13:45.401 DEBUG:boto:Method: PUT 2023-11-02T18:13:45.401 DEBUG:boto:Path: /ibtljn-46/obj2 2023-11-02T18:13:45.402 DEBUG:boto:Data: 2023-11-02T18:13:45.402 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:13:45.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:13:45.403 DEBUG:boto:Port: 8000 2023-11-02T18:13:45.403 DEBUG:boto:Params: {} 2023-11-02T18:13:45.404 DEBUG:boto:Token: None 2023-11-02T18:13:45.404 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-46/obj2 2023-11-02T18:13:45.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:anJdge0zlbcQnndvtK68syif5h8= 2023-11-02T18:13:45.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:anJdge0zlbcQnndvtK68syif5h8='} 2023-11-02T18:13:45.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e89fc1b97e53acde-006543e6d9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.451 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:13:45.452 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:13:45.452 DEBUG:boto:Method: HEAD 2023-11-02T18:13:45.453 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:13:45.453 DEBUG:boto:Data: 2023-11-02T18:13:45.454 DEBUG:boto:Headers: {} 2023-11-02T18:13:45.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:13:45.454 DEBUG:boto:Port: 8000 2023-11-02T18:13:45.455 DEBUG:boto:Params: {} 2023-11-02T18:13:45.455 DEBUG:boto:Token: None 2023-11-02T18:13:45.456 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-46/ 2023-11-02T18:13:45.456 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZhqH348SPbJpbazgeWneiStkCwA= 2023-11-02T18:13:45.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZhqH348SPbJpbazgeWneiStkCwA=', 'Content-Length': '0'} 2023-11-02T18:13:45.461 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fb8af5e2d227c31-006543e6d9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.462 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:13:45.462 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:13:45.463 DEBUG:boto:Method: PUT 2023-11-02T18:13:45.463 DEBUG:boto:Path: /ibtljn-46/obj3 2023-11-02T18:13:45.463 DEBUG:boto:Data: 2023-11-02T18:13:45.464 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:13:45.464 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:13:45.465 DEBUG:boto:Port: 8000 2023-11-02T18:13:45.465 DEBUG:boto:Params: {} 2023-11-02T18:13:45.465 DEBUG:boto:Token: None 2023-11-02T18:13:45.466 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-46/obj3 2023-11-02T18:13:45.466 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dbPlqf8zII5KzwwmE+X52JUg6gI= 2023-11-02T18:13:45.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dbPlqf8zII5KzwwmE+X52JUg6gI='} 2023-11-02T18:13:45.479 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091291a69fe684114-006543e6d9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.479 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:13:45.480 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:13:45.480 DEBUG:boto:Method: HEAD 2023-11-02T18:13:45.481 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:13:45.481 DEBUG:boto:Data: 2023-11-02T18:13:45.481 DEBUG:boto:Headers: {} 2023-11-02T18:13:45.482 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:13:45.482 DEBUG:boto:Port: 8000 2023-11-02T18:13:45.483 DEBUG:boto:Params: {} 2023-11-02T18:13:45.483 DEBUG:boto:Token: None 2023-11-02T18:13:45.484 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-46/ 2023-11-02T18:13:45.484 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZhqH348SPbJpbazgeWneiStkCwA= 2023-11-02T18:13:45.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZhqH348SPbJpbazgeWneiStkCwA=', 'Content-Length': '0'} 2023-11-02T18:13:45.488 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014583ec389a08587-006543e6d9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.489 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:13:45.489 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:13:45.489 DEBUG:boto:Method: PUT 2023-11-02T18:13:45.490 DEBUG:boto:Path: /ibtljn-46/obj4 2023-11-02T18:13:45.490 DEBUG:boto:Data: 2023-11-02T18:13:45.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:13:45.491 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:13:45.491 DEBUG:boto:Port: 8000 2023-11-02T18:13:45.492 DEBUG:boto:Params: {} 2023-11-02T18:13:45.492 DEBUG:boto:Token: None 2023-11-02T18:13:45.493 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-46/obj4 2023-11-02T18:13:45.493 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n+/0/hKqeilkfUWpoRmS07775Nw= 2023-11-02T18:13:45.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n+/0/hKqeilkfUWpoRmS07775Nw='} 2023-11-02T18:13:45.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6b1630a74f7b8b5-006543e6d9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.539 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:13:45.539 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:13:45.540 DEBUG:boto:Method: HEAD 2023-11-02T18:13:45.540 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:13:45.541 DEBUG:boto:Data: 2023-11-02T18:13:45.541 DEBUG:boto:Headers: {} 2023-11-02T18:13:45.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:13:45.542 DEBUG:boto:Port: 8001 2023-11-02T18:13:45.542 DEBUG:boto:Params: {} 2023-11-02T18:13:45.543 DEBUG:boto:Token: None 2023-11-02T18:13:45.543 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-47/ 2023-11-02T18:13:45.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EZAcbkLwLDftJpZlJJE2sf4ZE5Y= 2023-11-02T18:13:45.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EZAcbkLwLDftJpZlJJE2sf4ZE5Y=', 'Content-Length': '0'} 2023-11-02T18:13:45.547 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029732ed76c447e9b-006543e6d9-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.547 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:13:45.548 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:13:45.548 DEBUG:boto:Method: PUT 2023-11-02T18:13:45.549 DEBUG:boto:Path: /ibtljn-47/obj1 2023-11-02T18:13:45.549 DEBUG:boto:Data: 2023-11-02T18:13:45.550 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:13:45.550 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:13:45.550 DEBUG:boto:Port: 8001 2023-11-02T18:13:45.551 DEBUG:boto:Params: {} 2023-11-02T18:13:45.551 DEBUG:boto:Token: None 2023-11-02T18:13:45.552 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-47/obj1 2023-11-02T18:13:45.552 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1QUJj6xRnlN+1LIPRVsz00F0wCA= 2023-11-02T18:13:45.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1QUJj6xRnlN+1LIPRVsz00F0wCA='} 2023-11-02T18:13:45.561 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efa450555a9ec129-006543e6d9-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.562 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:13:45.562 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:13:45.563 DEBUG:boto:Method: HEAD 2023-11-02T18:13:45.563 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:13:45.564 DEBUG:boto:Data: 2023-11-02T18:13:45.564 DEBUG:boto:Headers: {} 2023-11-02T18:13:45.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:13:45.565 DEBUG:boto:Port: 8001 2023-11-02T18:13:45.565 DEBUG:boto:Params: {} 2023-11-02T18:13:45.566 DEBUG:boto:Token: None 2023-11-02T18:13:45.566 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-47/ 2023-11-02T18:13:45.567 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EZAcbkLwLDftJpZlJJE2sf4ZE5Y= 2023-11-02T18:13:45.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EZAcbkLwLDftJpZlJJE2sf4ZE5Y=', 'Content-Length': '0'} 2023-11-02T18:13:45.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086946711d7a61524-006543e6d9-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.570 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:13:45.571 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:13:45.571 DEBUG:boto:Method: PUT 2023-11-02T18:13:45.572 DEBUG:boto:Path: /ibtljn-47/obj2 2023-11-02T18:13:45.572 DEBUG:boto:Data: 2023-11-02T18:13:45.572 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:13:45.573 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:13:45.573 DEBUG:boto:Port: 8001 2023-11-02T18:13:45.574 DEBUG:boto:Params: {} 2023-11-02T18:13:45.574 DEBUG:boto:Token: None 2023-11-02T18:13:45.575 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-47/obj2 2023-11-02T18:13:45.575 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BNZEMFzG+SdeoysR5fQZy6HahG0= 2023-11-02T18:13:45.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BNZEMFzG+SdeoysR5fQZy6HahG0='} 2023-11-02T18:13:45.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a22343cee9fa345-006543e6d9-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.588 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:13:45.588 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:13:45.589 DEBUG:boto:Method: HEAD 2023-11-02T18:13:45.589 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:13:45.590 DEBUG:boto:Data: 2023-11-02T18:13:45.590 DEBUG:boto:Headers: {} 2023-11-02T18:13:45.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:13:45.591 DEBUG:boto:Port: 8001 2023-11-02T18:13:45.591 DEBUG:boto:Params: {} 2023-11-02T18:13:45.592 DEBUG:boto:Token: None 2023-11-02T18:13:45.592 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-47/ 2023-11-02T18:13:45.592 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EZAcbkLwLDftJpZlJJE2sf4ZE5Y= 2023-11-02T18:13:45.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EZAcbkLwLDftJpZlJJE2sf4ZE5Y=', 'Content-Length': '0'} 2023-11-02T18:13:45.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e90f139a95677b5a-006543e6d9-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.596 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:13:45.597 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:13:45.597 DEBUG:boto:Method: PUT 2023-11-02T18:13:45.597 DEBUG:boto:Path: /ibtljn-47/obj3 2023-11-02T18:13:45.598 DEBUG:boto:Data: 2023-11-02T18:13:45.598 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:13:45.599 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:13:45.599 DEBUG:boto:Port: 8001 2023-11-02T18:13:45.599 DEBUG:boto:Params: {} 2023-11-02T18:13:45.600 DEBUG:boto:Token: None 2023-11-02T18:13:45.600 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-47/obj3 2023-11-02T18:13:45.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BpN/NlXg1PlZ5/YZzSxFsBqKIvI= 2023-11-02T18:13:45.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BpN/NlXg1PlZ5/YZzSxFsBqKIvI='} 2023-11-02T18:13:45.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf9b771c70e6d577-006543e6d9-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.647 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:13:45.647 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:13:45.648 DEBUG:boto:Method: HEAD 2023-11-02T18:13:45.648 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:13:45.649 DEBUG:boto:Data: 2023-11-02T18:13:45.649 DEBUG:boto:Headers: {} 2023-11-02T18:13:45.649 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:13:45.650 DEBUG:boto:Port: 8001 2023-11-02T18:13:45.650 DEBUG:boto:Params: {} 2023-11-02T18:13:45.651 DEBUG:boto:Token: None 2023-11-02T18:13:45.651 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-47/ 2023-11-02T18:13:45.652 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EZAcbkLwLDftJpZlJJE2sf4ZE5Y= 2023-11-02T18:13:45.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EZAcbkLwLDftJpZlJJE2sf4ZE5Y=', 'Content-Length': '0'} 2023-11-02T18:13:45.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022bd25efd5172ca8-006543e6d9-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.657 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:13:45.657 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:13:45.658 DEBUG:boto:Method: PUT 2023-11-02T18:13:45.658 DEBUG:boto:Path: /ibtljn-47/obj4 2023-11-02T18:13:45.659 DEBUG:boto:Data: 2023-11-02T18:13:45.659 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:13:45.659 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:13:45.660 DEBUG:boto:Port: 8001 2023-11-02T18:13:45.660 DEBUG:boto:Params: {} 2023-11-02T18:13:45.661 DEBUG:boto:Token: None 2023-11-02T18:13:45.661 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:13:45 GMT /ibtljn-47/obj4 2023-11-02T18:13:45.662 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+NiBWH3K2qjFkCCaLGvIL3FU3gk= 2023-11-02T18:13:45.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:13:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+NiBWH3K2qjFkCCaLGvIL3FU3gk='} 2023-11-02T18:13:45.674 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f28cd445ad58b784-006543e6d9-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:13:45.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:45.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:45.676 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:45.722 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:45.722 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:45.784 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:13:45.812 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:13:45.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.812 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:13:45.813 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:13:45.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.813 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:13:45.813 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:13:45.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.814 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948825.343746_349.1", 2023-11-02T18:13:45.814 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:13:45.343746Z" 2023-11-02T18:13:45.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.815 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:13:45.815 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:13:45.815 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:45.816 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:13:45.817 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948825.343746_349.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:45.817 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:13:45.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:45.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:13:45.818 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:13:45.864 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:45.864 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:45.914+0000 7fcf3a4d0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:45.914+0000 7fcf3a4d0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:13:45.953 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:13:45.953 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:13:45.954 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:13:45.954 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:13:45.954 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:13:45.954 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:13:45.955 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:45.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.955 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:13:45.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:13:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:13:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:13:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:13:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:13:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:13:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:13:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:13:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698948825.343746_349.1", 2023-11-02T18:13:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:13:45.343746Z", 2023-11-02T18:13:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:13:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:13:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:13:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:13:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:13:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:13:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:13:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:13:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:13:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:13:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:13:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:13:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:13:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:13:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:13:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:13:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:13:45Z" 2023-11-02T18:13:45.968 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:13:45.969 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698948825.343746_349.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:13:45.343746Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:13:45Z" } 2023-11-02T18:13:45.969 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698948825.343746_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:13:45.343746Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:13:45.970 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948825.343746_349.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:45.970 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698948825.343746_349.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:13:45.970 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:13:45.971 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-46 2023-11-02T18:13:45.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:13:45.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:13:45.972 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:13:46.026 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:13:46.027 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:13:46.094 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:46.086+0000 7f9a1464df40 1 waiting to reach incremental sync.. 2023-11-02T18:13:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:51.086+0000 7f9a1464df40 1 waiting to reach incremental sync.. 2023-11-02T18:13:56.097 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:13:56.086+0000 7f9a1464df40 1 waiting to reach incremental sync.. 2023-11-02T18:14:01.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:01.090+0000 7f9a1464df40 1 bucket sync caught up with source: 2023-11-02T18:14:01.101 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.26.6, , 00000000001.14.6, , 00000000001.19.6, , , 00000000001.100.6, , , ] 2023-11-02T18:14:01.101 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.26.6, , 00000000001.14.6, , 00000000001.19.6, , , 00000000001.100.6, , , ] 2023-11-02T18:14:01.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:01.090+0000 7f9a1464df40 0 bucket checkpoint complete 2023-11-02T18:14:01.130 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-46 2023-11-02T18:14:01.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:14:01.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:14:01.132 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:14:01.182 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:14:01.183 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:14:01.254 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:01.242+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:14:06.254 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:06.246+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:14:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:11.246+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:14:16.256 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:16.246+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:14:21.258 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:21.246+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:14:26.258 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:26.250+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:14:31.259 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:31.250+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:14:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:36.250+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:14:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:41.250+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:14:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:46.254+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:14:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:51.254+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:14:56.265 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:14:56.253+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:15:01.266 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:01.257+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:15:06.267 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:06.257+0000 7f22f502bf40 1 waiting to reach incremental sync.. 2023-11-02T18:15:11.270 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:11.261+0000 7f22f502bf40 1 bucket sync caught up with source: 2023-11-02T18:15:11.271 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.191.6, , 00000000001.84.6, , 00000000001.296.6, , , 00000000001.246.6, , , ] 2023-11-02T18:15:11.271 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.191.6, , 00000000001.84.6, , 00000000001.296.6, , , 00000000001.246.6, , , ] 2023-11-02T18:15:11.271 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:11.261+0000 7f22f502bf40 0 bucket checkpoint complete 2023-11-02T18:15:11.300 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-46 zones={a2, a1} 2023-11-02T18:15:11.300 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.301 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.301 DEBUG:boto:Method: HEAD 2023-11-02T18:15:11.302 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:15:11.302 DEBUG:boto:Data: 2023-11-02T18:15:11.303 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.303 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.304 DEBUG:boto:Params: {} 2023-11-02T18:15:11.304 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:15:11.305 DEBUG:boto:Token: None 2023-11-02T18:15:11.305 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/ 2023-11-02T18:15:11.306 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nWAMu8XWxhFADkuh1zcFR6XsuYg= 2023-11-02T18:15:11.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nWAMu8XWxhFADkuh1zcFR6XsuYg=', 'Content-Length': '0'} 2023-11-02T18:15:11.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b13b402a8d051c3-006543e72f-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.314 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.314 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.315 DEBUG:boto:Method: HEAD 2023-11-02T18:15:11.315 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:15:11.315 DEBUG:boto:Data: 2023-11-02T18:15:11.316 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.316 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.317 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.317 DEBUG:boto:Params: {} 2023-11-02T18:15:11.318 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:15:11.318 DEBUG:boto:Token: None 2023-11-02T18:15:11.319 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/ 2023-11-02T18:15:11.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nWAMu8XWxhFADkuh1zcFR6XsuYg= 2023-11-02T18:15:11.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nWAMu8XWxhFADkuh1zcFR6XsuYg=', 'Content-Length': '0'} 2023-11-02T18:15:11.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000377055c2825c0de2-006543e72f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.327 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:15:11.327 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.328 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.328 DEBUG:boto:path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:15:11.328 DEBUG:boto:auth_path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:15:11.329 DEBUG:boto:Method: GET 2023-11-02T18:15:11.329 DEBUG:boto:Path: /ibtljn-46/?versions&max-keys=999 2023-11-02T18:15:11.330 DEBUG:boto:Data: 2023-11-02T18:15:11.330 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.331 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.331 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.331 DEBUG:boto:Params: {} 2023-11-02T18:15:11.332 DEBUG:boto:Token: None 2023-11-02T18:15:11.332 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/?versions 2023-11-02T18:15:11.333 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ypWkPuaT9600ZkA0l2XRKZXBzkU= 2023-11-02T18:15:11.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ypWkPuaT9600ZkA0l2XRKZXBzkU=', 'Content-Length': '0'} 2023-11-02T18:15:11.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afb13a28fa008c41-006543e72f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.339 DEBUG:boto:b'ibtljn-46999falseobj1nulltrue2023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:11.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:15:11.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:15:11.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:15:11.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:15:11.342 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:15:11.343 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.343 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.344 DEBUG:boto:path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:15:11.344 DEBUG:boto:auth_path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:15:11.344 DEBUG:boto:Method: GET 2023-11-02T18:15:11.345 DEBUG:boto:Path: /ibtljn-46/?versions&max-keys=999 2023-11-02T18:15:11.345 DEBUG:boto:Data: 2023-11-02T18:15:11.346 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.346 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.347 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.347 DEBUG:boto:Params: {} 2023-11-02T18:15:11.347 DEBUG:boto:Token: None 2023-11-02T18:15:11.348 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/?versions 2023-11-02T18:15:11.348 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ypWkPuaT9600ZkA0l2XRKZXBzkU= 2023-11-02T18:15:11.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ypWkPuaT9600ZkA0l2XRKZXBzkU=', 'Content-Length': '0'} 2023-11-02T18:15:11.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085c60c8b83ad5795-006543e72f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.354 DEBUG:boto:b'ibtljn-46999falseobj1nulltrue2023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:11.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:15:11.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:15:11.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:15:11.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:15:11.357 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.357 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.358 DEBUG:boto:path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:15:11.358 DEBUG:boto:auth_path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:15:11.359 DEBUG:boto:Method: GET 2023-11-02T18:15:11.359 DEBUG:boto:Path: /ibtljn-46/?versions&max-keys=999 2023-11-02T18:15:11.359 DEBUG:boto:Data: 2023-11-02T18:15:11.360 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.360 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.361 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.361 DEBUG:boto:Params: {} 2023-11-02T18:15:11.362 DEBUG:boto:Token: None 2023-11-02T18:15:11.362 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/?versions 2023-11-02T18:15:11.363 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ypWkPuaT9600ZkA0l2XRKZXBzkU= 2023-11-02T18:15:11.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ypWkPuaT9600ZkA0l2XRKZXBzkU=', 'Content-Length': '0'} 2023-11-02T18:15:11.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073f56be99b2b6d2e-006543e72f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.370 DEBUG:boto:b'ibtljn-46999falseobj1nulltrue2023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:11.370 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.371 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.371 DEBUG:boto:path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:15:11.372 DEBUG:boto:auth_path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:15:11.372 DEBUG:boto:Method: GET 2023-11-02T18:15:11.373 DEBUG:boto:Path: /ibtljn-46/?versions&max-keys=999 2023-11-02T18:15:11.373 DEBUG:boto:Data: 2023-11-02T18:15:11.374 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.374 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.374 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.375 DEBUG:boto:Params: {} 2023-11-02T18:15:11.376 DEBUG:boto:Token: None 2023-11-02T18:15:11.376 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/?versions 2023-11-02T18:15:11.377 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ypWkPuaT9600ZkA0l2XRKZXBzkU= 2023-11-02T18:15:11.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ypWkPuaT9600ZkA0l2XRKZXBzkU=', 'Content-Length': '0'} 2023-11-02T18:15:11.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b73e71adec3baf72-006543e72f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.384 DEBUG:boto:b'ibtljn-46999falseobj1nulltrue2023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:11.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:15:11.386 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:15:11.386 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:15:11.387 DEBUG:boto:path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.387 DEBUG:boto:auth_path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.388 DEBUG:boto:Method: GET 2023-11-02T18:15:11.388 DEBUG:boto:Path: /ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.390 DEBUG:boto:Data: 2023-11-02T18:15:11.390 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.390 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.391 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.391 DEBUG:boto:Params: {} 2023-11-02T18:15:11.392 DEBUG:boto:Token: None 2023-11-02T18:15:11.392 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.393 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Oj0kf/bxQrzJLm2XyPeibb629O4= 2023-11-02T18:15:11.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Oj0kf/bxQrzJLm2XyPeibb629O4=', 'Content-Length': '0'} 2023-11-02T18:15:11.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026fb5376e05c6f08-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.438 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:15:11.439 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:15:11.439 DEBUG:boto:path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.440 DEBUG:boto:auth_path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.440 DEBUG:boto:Method: GET 2023-11-02T18:15:11.441 DEBUG:boto:Path: /ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.441 DEBUG:boto:Data: 2023-11-02T18:15:11.442 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.442 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.443 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.443 DEBUG:boto:Params: {} 2023-11-02T18:15:11.443 DEBUG:boto:Token: None 2023-11-02T18:15:11.444 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Oj0kf/bxQrzJLm2XyPeibb629O4= 2023-11-02T18:15:11.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Oj0kf/bxQrzJLm2XyPeibb629O4=', 'Content-Length': '0'} 2023-11-02T18:15:11.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd9dda9756802a1a-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.449 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:15:11.449 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:15:11.449 DEBUG:boto:path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.450 DEBUG:boto:auth_path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.450 DEBUG:boto:Method: HEAD 2023-11-02T18:15:11.451 DEBUG:boto:Path: /ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.451 DEBUG:boto:Data: 2023-11-02T18:15:11.452 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.453 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.453 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.453 DEBUG:boto:Params: {} 2023-11-02T18:15:11.454 DEBUG:boto:Token: None 2023-11-02T18:15:11.454 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.455 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nK2d/qT2KLnQTEr29Y8vW/5Vr5w= 2023-11-02T18:15:11.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nK2d/qT2KLnQTEr29Y8vW/5Vr5w=', 'Content-Length': '0'} 2023-11-02T18:15:11.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cedadc37ca0aaed-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.459 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:15:11.459 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:15:11.460 DEBUG:boto:path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.460 DEBUG:boto:auth_path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.461 DEBUG:boto:Method: HEAD 2023-11-02T18:15:11.461 DEBUG:boto:Path: /ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.461 DEBUG:boto:Data: 2023-11-02T18:15:11.462 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.463 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.463 DEBUG:boto:Params: {} 2023-11-02T18:15:11.464 DEBUG:boto:Token: None 2023-11-02T18:15:11.464 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nK2d/qT2KLnQTEr29Y8vW/5Vr5w= 2023-11-02T18:15:11.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nK2d/qT2KLnQTEr29Y8vW/5Vr5w=', 'Content-Length': '0'} 2023-11-02T18:15:11.468 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0cbb88f9a582ea3-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:15:11.469 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:15:11.469 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:15:11.470 DEBUG:boto:path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.470 DEBUG:boto:auth_path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.471 DEBUG:boto:Method: GET 2023-11-02T18:15:11.471 DEBUG:boto:Path: /ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.472 DEBUG:boto:Data: 2023-11-02T18:15:11.472 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.473 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.473 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.474 DEBUG:boto:Params: {} 2023-11-02T18:15:11.474 DEBUG:boto:Token: None 2023-11-02T18:15:11.475 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.475 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Oj0kf/bxQrzJLm2XyPeibb629O4= 2023-11-02T18:15:11.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Oj0kf/bxQrzJLm2XyPeibb629O4=', 'Content-Length': '0'} 2023-11-02T18:15:11.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000610f37c857f3239c-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.479 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:15:11.480 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:15:11.480 DEBUG:boto:path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.481 DEBUG:boto:auth_path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.481 DEBUG:boto:Method: GET 2023-11-02T18:15:11.482 DEBUG:boto:Path: /ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.482 DEBUG:boto:Data: 2023-11-02T18:15:11.483 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.483 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.484 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.484 DEBUG:boto:Params: {} 2023-11-02T18:15:11.485 DEBUG:boto:Token: None 2023-11-02T18:15:11.485 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj1?versionId=null 2023-11-02T18:15:11.486 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Oj0kf/bxQrzJLm2XyPeibb629O4= 2023-11-02T18:15:11.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Oj0kf/bxQrzJLm2XyPeibb629O4=', 'Content-Length': '0'} 2023-11-02T18:15:11.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7da76e7b00284f6-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.534 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.535 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.535 DEBUG:boto:path=/ibtljn-46/?prefix=obj1 2023-11-02T18:15:11.536 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj1 2023-11-02T18:15:11.536 DEBUG:boto:Method: GET 2023-11-02T18:15:11.537 DEBUG:boto:Path: /ibtljn-46/?prefix=obj1 2023-11-02T18:15:11.537 DEBUG:boto:Data: 2023-11-02T18:15:11.538 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.538 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.539 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.539 DEBUG:boto:Params: {} 2023-11-02T18:15:11.540 DEBUG:boto:Token: None 2023-11-02T18:15:11.540 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/ 2023-11-02T18:15:11.541 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ= 2023-11-02T18:15:11.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ=', 'Content-Length': '0'} 2023-11-02T18:15:11.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000244123a85013ef0b-006543e72f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.548 DEBUG:boto:b'ibtljn-46obj11000falseobj12023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:11.549 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.549 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.550 DEBUG:boto:path=/ibtljn-46/?prefix=obj1 2023-11-02T18:15:11.550 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj1 2023-11-02T18:15:11.551 DEBUG:boto:Method: GET 2023-11-02T18:15:11.551 DEBUG:boto:Path: /ibtljn-46/?prefix=obj1 2023-11-02T18:15:11.551 DEBUG:boto:Data: 2023-11-02T18:15:11.552 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.552 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.553 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.553 DEBUG:boto:Params: {} 2023-11-02T18:15:11.554 DEBUG:boto:Token: None 2023-11-02T18:15:11.554 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/ 2023-11-02T18:15:11.555 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ= 2023-11-02T18:15:11.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ=', 'Content-Length': '0'} 2023-11-02T18:15:11.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089d19bf019653c48-006543e72f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.561 DEBUG:boto:b'ibtljn-46obj11000falseobj12023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:11.561 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:15:11.562 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:15:11.562 DEBUG:boto:Method: HEAD 2023-11-02T18:15:11.563 DEBUG:boto:Path: /ibtljn-46/obj1 2023-11-02T18:15:11.563 DEBUG:boto:Data: 2023-11-02T18:15:11.564 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.564 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.565 DEBUG:boto:Params: {} 2023-11-02T18:15:11.565 DEBUG:boto:Token: None 2023-11-02T18:15:11.566 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj1 2023-11-02T18:15:11.566 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0wgv95NvNTMXfSFfs5MMHYkWjCY= 2023-11-02T18:15:11.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0wgv95NvNTMXfSFfs5MMHYkWjCY=', 'Content-Length': '0'} 2023-11-02T18:15:11.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfdc289c9e6ee65f-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.570 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:15:11.571 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:15:11.571 DEBUG:boto:Method: HEAD 2023-11-02T18:15:11.572 DEBUG:boto:Path: /ibtljn-46/obj1 2023-11-02T18:15:11.572 DEBUG:boto:Data: 2023-11-02T18:15:11.573 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.573 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.574 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.574 DEBUG:boto:Params: {} 2023-11-02T18:15:11.574 DEBUG:boto:Token: None 2023-11-02T18:15:11.575 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj1 2023-11-02T18:15:11.575 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0wgv95NvNTMXfSFfs5MMHYkWjCY= 2023-11-02T18:15:11.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0wgv95NvNTMXfSFfs5MMHYkWjCY=', 'Content-Length': '0'} 2023-11-02T18:15:11.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1454033ede2922d-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:15:11.579 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:15:11.579 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:15:11.580 DEBUG:boto:Method: GET 2023-11-02T18:15:11.580 DEBUG:boto:Path: /ibtljn-46/obj1 2023-11-02T18:15:11.581 DEBUG:boto:Data: 2023-11-02T18:15:11.581 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.582 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.582 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.583 DEBUG:boto:Params: {} 2023-11-02T18:15:11.583 DEBUG:boto:Token: None 2023-11-02T18:15:11.584 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj1 2023-11-02T18:15:11.584 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1xEVSYIoCeVr1UYedoePUIpyY4I= 2023-11-02T18:15:11.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1xEVSYIoCeVr1UYedoePUIpyY4I=', 'Content-Length': '0'} 2023-11-02T18:15:11.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a215d75807a7db1-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.630 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:15:11.631 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:15:11.631 DEBUG:boto:Method: GET 2023-11-02T18:15:11.632 DEBUG:boto:Path: /ibtljn-46/obj1 2023-11-02T18:15:11.632 DEBUG:boto:Data: 2023-11-02T18:15:11.633 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.633 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.634 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.634 DEBUG:boto:Params: {} 2023-11-02T18:15:11.635 DEBUG:boto:Token: None 2023-11-02T18:15:11.635 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj1 2023-11-02T18:15:11.636 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1xEVSYIoCeVr1UYedoePUIpyY4I= 2023-11-02T18:15:11.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1xEVSYIoCeVr1UYedoePUIpyY4I=', 'Content-Length': '0'} 2023-11-02T18:15:11.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006630da9e09ffbcbc-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.640 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.641 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.641 DEBUG:boto:path=/ibtljn-46/?prefix=obj1 2023-11-02T18:15:11.642 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj1 2023-11-02T18:15:11.642 DEBUG:boto:Method: GET 2023-11-02T18:15:11.643 DEBUG:boto:Path: /ibtljn-46/?prefix=obj1 2023-11-02T18:15:11.643 DEBUG:boto:Data: 2023-11-02T18:15:11.643 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.644 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.644 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.645 DEBUG:boto:Params: {} 2023-11-02T18:15:11.645 DEBUG:boto:Token: None 2023-11-02T18:15:11.646 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/ 2023-11-02T18:15:11.646 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ= 2023-11-02T18:15:11.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ=', 'Content-Length': '0'} 2023-11-02T18:15:11.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008829f42cd415b19-006543e72f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.653 DEBUG:boto:b'ibtljn-46obj11000falseobj12023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:11.653 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.654 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.655 DEBUG:boto:path=/ibtljn-46/?prefix=obj1 2023-11-02T18:15:11.655 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj1 2023-11-02T18:15:11.655 DEBUG:boto:Method: GET 2023-11-02T18:15:11.656 DEBUG:boto:Path: /ibtljn-46/?prefix=obj1 2023-11-02T18:15:11.656 DEBUG:boto:Data: 2023-11-02T18:15:11.657 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.657 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.658 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.658 DEBUG:boto:Params: {} 2023-11-02T18:15:11.659 DEBUG:boto:Token: None 2023-11-02T18:15:11.659 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/ 2023-11-02T18:15:11.660 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ= 2023-11-02T18:15:11.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ=', 'Content-Length': '0'} 2023-11-02T18:15:11.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007cc8cdf6d3760c1-006543e72f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.706 DEBUG:boto:b'ibtljn-46obj11000falseobj12023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:11.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:15:11.708 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:15:11.708 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:15:11.709 DEBUG:boto:path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.709 DEBUG:boto:auth_path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.709 DEBUG:boto:Method: GET 2023-11-02T18:15:11.710 DEBUG:boto:Path: /ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.710 DEBUG:boto:Data: 2023-11-02T18:15:11.711 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.711 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.712 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.712 DEBUG:boto:Params: {} 2023-11-02T18:15:11.713 DEBUG:boto:Token: None 2023-11-02T18:15:11.713 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.714 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W9rMQaGvAKa5T/5HtwcaNrXk6DE= 2023-11-02T18:15:11.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W9rMQaGvAKa5T/5HtwcaNrXk6DE=', 'Content-Length': '0'} 2023-11-02T18:15:11.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce966f973299eb0a-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.718 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:15:11.719 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:15:11.719 DEBUG:boto:path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.720 DEBUG:boto:auth_path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.720 DEBUG:boto:Method: GET 2023-11-02T18:15:11.720 DEBUG:boto:Path: /ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.721 DEBUG:boto:Data: 2023-11-02T18:15:11.721 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.722 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.722 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.723 DEBUG:boto:Params: {} 2023-11-02T18:15:11.723 DEBUG:boto:Token: None 2023-11-02T18:15:11.723 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.724 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W9rMQaGvAKa5T/5HtwcaNrXk6DE= 2023-11-02T18:15:11.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W9rMQaGvAKa5T/5HtwcaNrXk6DE=', 'Content-Length': '0'} 2023-11-02T18:15:11.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ec05d8453773d96-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.728 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:15:11.728 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:15:11.729 DEBUG:boto:path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.729 DEBUG:boto:auth_path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.730 DEBUG:boto:Method: HEAD 2023-11-02T18:15:11.730 DEBUG:boto:Path: /ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.730 DEBUG:boto:Data: 2023-11-02T18:15:11.731 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.731 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.732 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.732 DEBUG:boto:Params: {} 2023-11-02T18:15:11.733 DEBUG:boto:Token: None 2023-11-02T18:15:11.733 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.734 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cV21l58cko1XZ3VTeadhKsxEdiw= 2023-11-02T18:15:11.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cV21l58cko1XZ3VTeadhKsxEdiw=', 'Content-Length': '0'} 2023-11-02T18:15:11.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000801c408e7d35a1fe-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.739 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:15:11.739 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:15:11.741 DEBUG:boto:path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.741 DEBUG:boto:auth_path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.742 DEBUG:boto:Method: HEAD 2023-11-02T18:15:11.742 DEBUG:boto:Path: /ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.742 DEBUG:boto:Data: 2023-11-02T18:15:11.743 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.743 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.744 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.744 DEBUG:boto:Params: {} 2023-11-02T18:15:11.745 DEBUG:boto:Token: None 2023-11-02T18:15:11.745 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.746 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cV21l58cko1XZ3VTeadhKsxEdiw= 2023-11-02T18:15:11.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cV21l58cko1XZ3VTeadhKsxEdiw=', 'Content-Length': '0'} 2023-11-02T18:15:11.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000323d91d1876b11e3-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:15:11.750 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:15:11.751 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:15:11.751 DEBUG:boto:path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.752 DEBUG:boto:auth_path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.752 DEBUG:boto:Method: GET 2023-11-02T18:15:11.752 DEBUG:boto:Path: /ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.753 DEBUG:boto:Data: 2023-11-02T18:15:11.753 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.754 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.755 DEBUG:boto:Params: {} 2023-11-02T18:15:11.755 DEBUG:boto:Token: None 2023-11-02T18:15:11.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.756 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W9rMQaGvAKa5T/5HtwcaNrXk6DE= 2023-11-02T18:15:11.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W9rMQaGvAKa5T/5HtwcaNrXk6DE=', 'Content-Length': '0'} 2023-11-02T18:15:11.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b295e6935454e582-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.761 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:15:11.761 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:15:11.762 DEBUG:boto:path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.762 DEBUG:boto:auth_path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.763 DEBUG:boto:Method: GET 2023-11-02T18:15:11.763 DEBUG:boto:Path: /ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.764 DEBUG:boto:Data: 2023-11-02T18:15:11.764 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.765 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.765 DEBUG:boto:Params: {} 2023-11-02T18:15:11.766 DEBUG:boto:Token: None 2023-11-02T18:15:11.766 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj2?versionId=null 2023-11-02T18:15:11.767 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W9rMQaGvAKa5T/5HtwcaNrXk6DE= 2023-11-02T18:15:11.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W9rMQaGvAKa5T/5HtwcaNrXk6DE=', 'Content-Length': '0'} 2023-11-02T18:15:11.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3470e1e0ec729ef-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.770 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.771 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.771 DEBUG:boto:path=/ibtljn-46/?prefix=obj2 2023-11-02T18:15:11.772 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj2 2023-11-02T18:15:11.772 DEBUG:boto:Method: GET 2023-11-02T18:15:11.773 DEBUG:boto:Path: /ibtljn-46/?prefix=obj2 2023-11-02T18:15:11.773 DEBUG:boto:Data: 2023-11-02T18:15:11.774 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.774 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.775 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.775 DEBUG:boto:Params: {} 2023-11-02T18:15:11.776 DEBUG:boto:Token: None 2023-11-02T18:15:11.776 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/ 2023-11-02T18:15:11.777 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ= 2023-11-02T18:15:11.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ=', 'Content-Length': '0'} 2023-11-02T18:15:11.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e29382a4fa85c5f-006543e72f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.783 DEBUG:boto:b'ibtljn-46obj21000falseobj22023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:11.784 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.785 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.785 DEBUG:boto:path=/ibtljn-46/?prefix=obj2 2023-11-02T18:15:11.786 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj2 2023-11-02T18:15:11.786 DEBUG:boto:Method: GET 2023-11-02T18:15:11.787 DEBUG:boto:Path: /ibtljn-46/?prefix=obj2 2023-11-02T18:15:11.787 DEBUG:boto:Data: 2023-11-02T18:15:11.787 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.788 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.788 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.789 DEBUG:boto:Params: {} 2023-11-02T18:15:11.789 DEBUG:boto:Token: None 2023-11-02T18:15:11.790 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/ 2023-11-02T18:15:11.790 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ= 2023-11-02T18:15:11.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ=', 'Content-Length': '0'} 2023-11-02T18:15:11.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031548a7f854ffd40-006543e72f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.797 DEBUG:boto:b'ibtljn-46obj21000falseobj22023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:11.797 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:15:11.798 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:15:11.798 DEBUG:boto:Method: HEAD 2023-11-02T18:15:11.799 DEBUG:boto:Path: /ibtljn-46/obj2 2023-11-02T18:15:11.799 DEBUG:boto:Data: 2023-11-02T18:15:11.800 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.800 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.800 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.801 DEBUG:boto:Params: {} 2023-11-02T18:15:11.801 DEBUG:boto:Token: None 2023-11-02T18:15:11.802 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj2 2023-11-02T18:15:11.802 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HUObZgdrl+8MbUBEa8RvPoclnH0= 2023-11-02T18:15:11.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HUObZgdrl+8MbUBEa8RvPoclnH0=', 'Content-Length': '0'} 2023-11-02T18:15:11.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfc2fe49f680d566-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.806 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:15:11.807 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:15:11.807 DEBUG:boto:Method: HEAD 2023-11-02T18:15:11.808 DEBUG:boto:Path: /ibtljn-46/obj2 2023-11-02T18:15:11.808 DEBUG:boto:Data: 2023-11-02T18:15:11.808 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.809 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.809 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.810 DEBUG:boto:Params: {} 2023-11-02T18:15:11.810 DEBUG:boto:Token: None 2023-11-02T18:15:11.811 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj2 2023-11-02T18:15:11.811 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HUObZgdrl+8MbUBEa8RvPoclnH0= 2023-11-02T18:15:11.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HUObZgdrl+8MbUBEa8RvPoclnH0=', 'Content-Length': '0'} 2023-11-02T18:15:11.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b0bd78f60bc21f7-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:15:11.815 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:15:11.816 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:15:11.816 DEBUG:boto:Method: GET 2023-11-02T18:15:11.817 DEBUG:boto:Path: /ibtljn-46/obj2 2023-11-02T18:15:11.817 DEBUG:boto:Data: 2023-11-02T18:15:11.817 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.818 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.818 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.819 DEBUG:boto:Params: {} 2023-11-02T18:15:11.819 DEBUG:boto:Token: None 2023-11-02T18:15:11.820 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj2 2023-11-02T18:15:11.820 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:98ocP0XDaTTiNQ65uNCdWdTOnk0= 2023-11-02T18:15:11.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:98ocP0XDaTTiNQ65uNCdWdTOnk0=', 'Content-Length': '0'} 2023-11-02T18:15:11.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007690006cf5d9a85d-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.870 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:15:11.871 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:15:11.871 DEBUG:boto:Method: GET 2023-11-02T18:15:11.872 DEBUG:boto:Path: /ibtljn-46/obj2 2023-11-02T18:15:11.872 DEBUG:boto:Data: 2023-11-02T18:15:11.873 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.873 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.874 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.874 DEBUG:boto:Params: {} 2023-11-02T18:15:11.874 DEBUG:boto:Token: None 2023-11-02T18:15:11.875 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj2 2023-11-02T18:15:11.875 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:98ocP0XDaTTiNQ65uNCdWdTOnk0= 2023-11-02T18:15:11.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:98ocP0XDaTTiNQ65uNCdWdTOnk0=', 'Content-Length': '0'} 2023-11-02T18:15:11.879 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068d3551771a3530f-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.880 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.880 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.881 DEBUG:boto:path=/ibtljn-46/?prefix=obj2 2023-11-02T18:15:11.881 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj2 2023-11-02T18:15:11.881 DEBUG:boto:Method: GET 2023-11-02T18:15:11.882 DEBUG:boto:Path: /ibtljn-46/?prefix=obj2 2023-11-02T18:15:11.882 DEBUG:boto:Data: 2023-11-02T18:15:11.883 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.883 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.884 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.884 DEBUG:boto:Params: {} 2023-11-02T18:15:11.885 DEBUG:boto:Token: None 2023-11-02T18:15:11.885 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/ 2023-11-02T18:15:11.886 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ= 2023-11-02T18:15:11.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ=', 'Content-Length': '0'} 2023-11-02T18:15:11.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bb86dcb437eeb59-006543e72f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.893 DEBUG:boto:b'ibtljn-46obj21000falseobj22023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:11.894 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:11.894 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:11.894 DEBUG:boto:path=/ibtljn-46/?prefix=obj2 2023-11-02T18:15:11.895 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj2 2023-11-02T18:15:11.895 DEBUG:boto:Method: GET 2023-11-02T18:15:11.896 DEBUG:boto:Path: /ibtljn-46/?prefix=obj2 2023-11-02T18:15:11.896 DEBUG:boto:Data: 2023-11-02T18:15:11.897 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.897 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.897 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.898 DEBUG:boto:Params: {} 2023-11-02T18:15:11.898 DEBUG:boto:Token: None 2023-11-02T18:15:11.899 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/ 2023-11-02T18:15:11.899 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ= 2023-11-02T18:15:11.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YEX9xOK5yJ9UaJGWJ9VIEDLhAnQ=', 'Content-Length': '0'} 2023-11-02T18:15:11.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f0706181028bb3f-006543e72f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.906 DEBUG:boto:b'ibtljn-46obj21000falseobj22023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:11.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:15:11.907 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:15:11.907 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:15:11.908 DEBUG:boto:path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.908 DEBUG:boto:auth_path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.909 DEBUG:boto:Method: GET 2023-11-02T18:15:11.909 DEBUG:boto:Path: /ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.909 DEBUG:boto:Data: 2023-11-02T18:15:11.910 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.911 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.911 DEBUG:boto:Params: {} 2023-11-02T18:15:11.912 DEBUG:boto:Token: None 2023-11-02T18:15:11.912 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.913 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a3I98fsdBpCG9bfPBI6y0XQ3u0c= 2023-11-02T18:15:11.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a3I98fsdBpCG9bfPBI6y0XQ3u0c=', 'Content-Length': '0'} 2023-11-02T18:15:11.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccf2f7d993135415-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.917 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:15:11.918 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:15:11.918 DEBUG:boto:path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.918 DEBUG:boto:auth_path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.919 DEBUG:boto:Method: GET 2023-11-02T18:15:11.919 DEBUG:boto:Path: /ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.920 DEBUG:boto:Data: 2023-11-02T18:15:11.920 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.921 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.921 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.921 DEBUG:boto:Params: {} 2023-11-02T18:15:11.922 DEBUG:boto:Token: None 2023-11-02T18:15:11.923 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.923 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a3I98fsdBpCG9bfPBI6y0XQ3u0c= 2023-11-02T18:15:11.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a3I98fsdBpCG9bfPBI6y0XQ3u0c=', 'Content-Length': '0'} 2023-11-02T18:15:11.927 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000323d5a3d6aea451b-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.928 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:15:11.928 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:15:11.928 DEBUG:boto:path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.929 DEBUG:boto:auth_path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.929 DEBUG:boto:Method: HEAD 2023-11-02T18:15:11.930 DEBUG:boto:Path: /ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.931 DEBUG:boto:Data: 2023-11-02T18:15:11.931 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.931 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.932 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.932 DEBUG:boto:Params: {} 2023-11-02T18:15:11.933 DEBUG:boto:Token: None 2023-11-02T18:15:11.933 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.934 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I7CZ1BxLmmZHqtEI1NKBBGGbOjk= 2023-11-02T18:15:11.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I7CZ1BxLmmZHqtEI1NKBBGGbOjk=', 'Content-Length': '0'} 2023-11-02T18:15:11.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018520aa7832b2583-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.937 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:15:11.938 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:15:11.938 DEBUG:boto:path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.939 DEBUG:boto:auth_path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.939 DEBUG:boto:Method: HEAD 2023-11-02T18:15:11.940 DEBUG:boto:Path: /ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.940 DEBUG:boto:Data: 2023-11-02T18:15:11.940 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.941 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:11.941 DEBUG:boto:Port: 8000 2023-11-02T18:15:11.942 DEBUG:boto:Params: {} 2023-11-02T18:15:11.942 DEBUG:boto:Token: None 2023-11-02T18:15:11.943 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.943 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I7CZ1BxLmmZHqtEI1NKBBGGbOjk= 2023-11-02T18:15:11.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I7CZ1BxLmmZHqtEI1NKBBGGbOjk=', 'Content-Length': '0'} 2023-11-02T18:15:11.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098a9d2e688055243-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:15:11.947 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:15:11.948 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:15:11.948 DEBUG:boto:path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.949 DEBUG:boto:auth_path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.949 DEBUG:boto:Method: GET 2023-11-02T18:15:11.950 DEBUG:boto:Path: /ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.950 DEBUG:boto:Data: 2023-11-02T18:15:11.950 DEBUG:boto:Headers: {} 2023-11-02T18:15:11.951 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:11.951 DEBUG:boto:Port: 8001 2023-11-02T18:15:11.952 DEBUG:boto:Params: {} 2023-11-02T18:15:11.952 DEBUG:boto:Token: None 2023-11-02T18:15:11.953 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:11 GMT /ibtljn-46/obj3?versionId=null 2023-11-02T18:15:11.953 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a3I98fsdBpCG9bfPBI6y0XQ3u0c= 2023-11-02T18:15:11.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a3I98fsdBpCG9bfPBI6y0XQ3u0c=', 'Content-Length': '0'} 2023-11-02T18:15:11.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000233d5cd2963f82a-006543e72f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:11.998 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:15:11.999 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:15:11.999 DEBUG:boto:path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:15:12.000 DEBUG:boto:auth_path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:15:12.000 DEBUG:boto:Method: GET 2023-11-02T18:15:12.001 DEBUG:boto:Path: /ibtljn-46/obj3?versionId=null 2023-11-02T18:15:12.001 DEBUG:boto:Data: 2023-11-02T18:15:12.002 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.002 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:12.002 DEBUG:boto:Port: 8000 2023-11-02T18:15:12.003 DEBUG:boto:Params: {} 2023-11-02T18:15:12.003 DEBUG:boto:Token: None 2023-11-02T18:15:12.004 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj3?versionId=null 2023-11-02T18:15:12.004 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IFGZ72tWSjaeuYXm+doc2Avq7Vc= 2023-11-02T18:15:12.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IFGZ72tWSjaeuYXm+doc2Avq7Vc=', 'Content-Length': '0'} 2023-11-02T18:15:12.008 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000157705671c2b9c70-006543e72f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.009 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:12.009 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:12.010 DEBUG:boto:path=/ibtljn-46/?prefix=obj3 2023-11-02T18:15:12.010 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj3 2023-11-02T18:15:12.010 DEBUG:boto:Method: GET 2023-11-02T18:15:12.011 DEBUG:boto:Path: /ibtljn-46/?prefix=obj3 2023-11-02T18:15:12.011 DEBUG:boto:Data: 2023-11-02T18:15:12.012 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.012 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:12.013 DEBUG:boto:Port: 8001 2023-11-02T18:15:12.013 DEBUG:boto:Params: {} 2023-11-02T18:15:12.014 DEBUG:boto:Token: None 2023-11-02T18:15:12.014 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/ 2023-11-02T18:15:12.015 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8= 2023-11-02T18:15:12.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8=', 'Content-Length': '0'} 2023-11-02T18:15:12.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dff025289e7335a-006543e730-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.022 DEBUG:boto:b'ibtljn-46obj31000falseobj32023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:12.022 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:12.023 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:12.023 DEBUG:boto:path=/ibtljn-46/?prefix=obj3 2023-11-02T18:15:12.024 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj3 2023-11-02T18:15:12.024 DEBUG:boto:Method: GET 2023-11-02T18:15:12.025 DEBUG:boto:Path: /ibtljn-46/?prefix=obj3 2023-11-02T18:15:12.025 DEBUG:boto:Data: 2023-11-02T18:15:12.026 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.026 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:12.026 DEBUG:boto:Port: 8000 2023-11-02T18:15:12.027 DEBUG:boto:Params: {} 2023-11-02T18:15:12.027 DEBUG:boto:Token: None 2023-11-02T18:15:12.028 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/ 2023-11-02T18:15:12.028 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8= 2023-11-02T18:15:12.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8=', 'Content-Length': '0'} 2023-11-02T18:15:12.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c48d7b9e372aefc5-006543e730-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.034 DEBUG:boto:b'ibtljn-46obj31000falseobj32023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:12.035 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:15:12.036 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:15:12.036 DEBUG:boto:Method: HEAD 2023-11-02T18:15:12.036 DEBUG:boto:Path: /ibtljn-46/obj3 2023-11-02T18:15:12.037 DEBUG:boto:Data: 2023-11-02T18:15:12.037 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:12.038 DEBUG:boto:Port: 8001 2023-11-02T18:15:12.039 DEBUG:boto:Params: {} 2023-11-02T18:15:12.039 DEBUG:boto:Token: None 2023-11-02T18:15:12.040 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj3 2023-11-02T18:15:12.040 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p/6kxcQwZiP06wpOFFYcGNKZPR4= 2023-11-02T18:15:12.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p/6kxcQwZiP06wpOFFYcGNKZPR4=', 'Content-Length': '0'} 2023-11-02T18:15:12.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024428c90092afa1d-006543e730-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.044 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:15:12.044 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:15:12.044 DEBUG:boto:Method: HEAD 2023-11-02T18:15:12.045 DEBUG:boto:Path: /ibtljn-46/obj3 2023-11-02T18:15:12.045 DEBUG:boto:Data: 2023-11-02T18:15:12.046 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:12.046 DEBUG:boto:Port: 8000 2023-11-02T18:15:12.047 DEBUG:boto:Params: {} 2023-11-02T18:15:12.047 DEBUG:boto:Token: None 2023-11-02T18:15:12.048 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj3 2023-11-02T18:15:12.048 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p/6kxcQwZiP06wpOFFYcGNKZPR4= 2023-11-02T18:15:12.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p/6kxcQwZiP06wpOFFYcGNKZPR4=', 'Content-Length': '0'} 2023-11-02T18:15:12.051 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3275301c8db7e11-006543e730-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:15:12.052 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:15:12.053 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:15:12.053 DEBUG:boto:Method: GET 2023-11-02T18:15:12.054 DEBUG:boto:Path: /ibtljn-46/obj3 2023-11-02T18:15:12.054 DEBUG:boto:Data: 2023-11-02T18:15:12.055 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.055 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:12.055 DEBUG:boto:Port: 8001 2023-11-02T18:15:12.056 DEBUG:boto:Params: {} 2023-11-02T18:15:12.056 DEBUG:boto:Token: None 2023-11-02T18:15:12.057 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj3 2023-11-02T18:15:12.057 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aayIgX+4d9FkTBPI5J24BPUsquA= 2023-11-02T18:15:12.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aayIgX+4d9FkTBPI5J24BPUsquA=', 'Content-Length': '0'} 2023-11-02T18:15:12.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008e5b0771bb2322c-006543e730-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.102 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:15:12.103 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:15:12.104 DEBUG:boto:Method: GET 2023-11-02T18:15:12.104 DEBUG:boto:Path: /ibtljn-46/obj3 2023-11-02T18:15:12.104 DEBUG:boto:Data: 2023-11-02T18:15:12.105 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.105 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:12.106 DEBUG:boto:Port: 8000 2023-11-02T18:15:12.106 DEBUG:boto:Params: {} 2023-11-02T18:15:12.110 DEBUG:boto:Token: None 2023-11-02T18:15:12.111 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj3 2023-11-02T18:15:12.112 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aayIgX+4d9FkTBPI5J24BPUsquA= 2023-11-02T18:15:12.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aayIgX+4d9FkTBPI5J24BPUsquA=', 'Content-Length': '0'} 2023-11-02T18:15:12.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fdf208d642a40ee-006543e730-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.117 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:12.118 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:12.119 DEBUG:boto:path=/ibtljn-46/?prefix=obj3 2023-11-02T18:15:12.119 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj3 2023-11-02T18:15:12.120 DEBUG:boto:Method: GET 2023-11-02T18:15:12.121 DEBUG:boto:Path: /ibtljn-46/?prefix=obj3 2023-11-02T18:15:12.122 DEBUG:boto:Data: 2023-11-02T18:15:12.122 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.123 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:12.124 DEBUG:boto:Port: 8001 2023-11-02T18:15:12.124 DEBUG:boto:Params: {} 2023-11-02T18:15:12.125 DEBUG:boto:Token: None 2023-11-02T18:15:12.126 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/ 2023-11-02T18:15:12.126 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8= 2023-11-02T18:15:12.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8=', 'Content-Length': '0'} 2023-11-02T18:15:12.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000376e1eebdfa8705f-006543e730-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.134 DEBUG:boto:b'ibtljn-46obj31000falseobj32023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:12.135 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:12.135 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:12.136 DEBUG:boto:path=/ibtljn-46/?prefix=obj3 2023-11-02T18:15:12.136 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj3 2023-11-02T18:15:12.137 DEBUG:boto:Method: GET 2023-11-02T18:15:12.137 DEBUG:boto:Path: /ibtljn-46/?prefix=obj3 2023-11-02T18:15:12.138 DEBUG:boto:Data: 2023-11-02T18:15:12.139 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:12.140 DEBUG:boto:Port: 8000 2023-11-02T18:15:12.141 DEBUG:boto:Params: {} 2023-11-02T18:15:12.142 DEBUG:boto:Token: None 2023-11-02T18:15:12.142 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/ 2023-11-02T18:15:12.143 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8= 2023-11-02T18:15:12.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8=', 'Content-Length': '0'} 2023-11-02T18:15:12.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e42168d10a654af-006543e730-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.194 DEBUG:boto:b'ibtljn-46obj31000falseobj32023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:12.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:15:12.196 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:15:12.196 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:15:12.197 DEBUG:boto:path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.197 DEBUG:boto:auth_path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.197 DEBUG:boto:Method: GET 2023-11-02T18:15:12.198 DEBUG:boto:Path: /ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.198 DEBUG:boto:Data: 2023-11-02T18:15:12.199 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.199 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:12.200 DEBUG:boto:Port: 8001 2023-11-02T18:15:12.200 DEBUG:boto:Params: {} 2023-11-02T18:15:12.201 DEBUG:boto:Token: None 2023-11-02T18:15:12.202 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.203 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dmUTjHgmZOWuXR553x9JDOyUnm8= 2023-11-02T18:15:12.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dmUTjHgmZOWuXR553x9JDOyUnm8=', 'Content-Length': '0'} 2023-11-02T18:15:12.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091c1ace13d75bebc-006543e730-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.208 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:15:12.209 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:15:12.209 DEBUG:boto:path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.209 DEBUG:boto:auth_path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.210 DEBUG:boto:Method: GET 2023-11-02T18:15:12.210 DEBUG:boto:Path: /ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.211 DEBUG:boto:Data: 2023-11-02T18:15:12.212 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.213 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:12.213 DEBUG:boto:Port: 8000 2023-11-02T18:15:12.214 DEBUG:boto:Params: {} 2023-11-02T18:15:12.215 DEBUG:boto:Token: None 2023-11-02T18:15:12.216 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dmUTjHgmZOWuXR553x9JDOyUnm8= 2023-11-02T18:15:12.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dmUTjHgmZOWuXR553x9JDOyUnm8=', 'Content-Length': '0'} 2023-11-02T18:15:12.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec5a26d073efae57-006543e730-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.222 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:15:12.222 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:15:12.223 DEBUG:boto:path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.223 DEBUG:boto:auth_path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.223 DEBUG:boto:Method: HEAD 2023-11-02T18:15:12.224 DEBUG:boto:Path: /ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.225 DEBUG:boto:Data: 2023-11-02T18:15:12.225 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.226 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:12.226 DEBUG:boto:Port: 8001 2023-11-02T18:15:12.227 DEBUG:boto:Params: {} 2023-11-02T18:15:12.227 DEBUG:boto:Token: None 2023-11-02T18:15:12.228 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.228 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wC96r7MTYOzzPiX44ApxOvn/tVA= 2023-11-02T18:15:12.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wC96r7MTYOzzPiX44ApxOvn/tVA=', 'Content-Length': '0'} 2023-11-02T18:15:12.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087fedfa33c1f6527-006543e730-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.233 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:15:12.233 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:15:12.233 DEBUG:boto:path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.234 DEBUG:boto:auth_path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.234 DEBUG:boto:Method: HEAD 2023-11-02T18:15:12.235 DEBUG:boto:Path: /ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.235 DEBUG:boto:Data: 2023-11-02T18:15:12.236 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.236 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:12.237 DEBUG:boto:Port: 8000 2023-11-02T18:15:12.237 DEBUG:boto:Params: {} 2023-11-02T18:15:12.238 DEBUG:boto:Token: None 2023-11-02T18:15:12.238 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.239 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wC96r7MTYOzzPiX44ApxOvn/tVA= 2023-11-02T18:15:12.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wC96r7MTYOzzPiX44ApxOvn/tVA=', 'Content-Length': '0'} 2023-11-02T18:15:12.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d075910a6b37b2d9-006543e730-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:15:12.244 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:15:12.244 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:15:12.244 DEBUG:boto:path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.245 DEBUG:boto:auth_path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.245 DEBUG:boto:Method: GET 2023-11-02T18:15:12.246 DEBUG:boto:Path: /ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.246 DEBUG:boto:Data: 2023-11-02T18:15:12.247 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.247 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:12.248 DEBUG:boto:Port: 8001 2023-11-02T18:15:12.248 DEBUG:boto:Params: {} 2023-11-02T18:15:12.249 DEBUG:boto:Token: None 2023-11-02T18:15:12.249 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.250 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dmUTjHgmZOWuXR553x9JDOyUnm8= 2023-11-02T18:15:12.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dmUTjHgmZOWuXR553x9JDOyUnm8=', 'Content-Length': '0'} 2023-11-02T18:15:12.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006334d83237e9c274-006543e730-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.294 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:15:12.295 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:15:12.295 DEBUG:boto:path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.296 DEBUG:boto:auth_path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.296 DEBUG:boto:Method: GET 2023-11-02T18:15:12.297 DEBUG:boto:Path: /ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.297 DEBUG:boto:Data: 2023-11-02T18:15:12.297 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.298 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:12.298 DEBUG:boto:Port: 8000 2023-11-02T18:15:12.299 DEBUG:boto:Params: {} 2023-11-02T18:15:12.299 DEBUG:boto:Token: None 2023-11-02T18:15:12.300 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj4?versionId=null 2023-11-02T18:15:12.300 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dmUTjHgmZOWuXR553x9JDOyUnm8= 2023-11-02T18:15:12.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dmUTjHgmZOWuXR553x9JDOyUnm8=', 'Content-Length': '0'} 2023-11-02T18:15:12.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebfe246ec41ed7a4-006543e730-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.304 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:12.305 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:12.305 DEBUG:boto:path=/ibtljn-46/?prefix=obj4 2023-11-02T18:15:12.306 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj4 2023-11-02T18:15:12.306 DEBUG:boto:Method: GET 2023-11-02T18:15:12.307 DEBUG:boto:Path: /ibtljn-46/?prefix=obj4 2023-11-02T18:15:12.307 DEBUG:boto:Data: 2023-11-02T18:15:12.308 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.308 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:12.309 DEBUG:boto:Port: 8001 2023-11-02T18:15:12.310 DEBUG:boto:Params: {} 2023-11-02T18:15:12.310 DEBUG:boto:Token: None 2023-11-02T18:15:12.311 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/ 2023-11-02T18:15:12.313 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8= 2023-11-02T18:15:12.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8=', 'Content-Length': '0'} 2023-11-02T18:15:12.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d966d7f67841003d-006543e730-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.322 DEBUG:boto:b'ibtljn-46obj41000falseobj42023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:12.323 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:12.323 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:12.323 DEBUG:boto:path=/ibtljn-46/?prefix=obj4 2023-11-02T18:15:12.324 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj4 2023-11-02T18:15:12.324 DEBUG:boto:Method: GET 2023-11-02T18:15:12.325 DEBUG:boto:Path: /ibtljn-46/?prefix=obj4 2023-11-02T18:15:12.325 DEBUG:boto:Data: 2023-11-02T18:15:12.326 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.326 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:12.327 DEBUG:boto:Port: 8000 2023-11-02T18:15:12.327 DEBUG:boto:Params: {} 2023-11-02T18:15:12.328 DEBUG:boto:Token: None 2023-11-02T18:15:12.328 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/ 2023-11-02T18:15:12.329 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8= 2023-11-02T18:15:12.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8=', 'Content-Length': '0'} 2023-11-02T18:15:12.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea7c446bdef2cde0-006543e730-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.335 DEBUG:boto:b'ibtljn-46obj41000falseobj42023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:12.336 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:15:12.336 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:15:12.337 DEBUG:boto:Method: HEAD 2023-11-02T18:15:12.337 DEBUG:boto:Path: /ibtljn-46/obj4 2023-11-02T18:15:12.337 DEBUG:boto:Data: 2023-11-02T18:15:12.338 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.338 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:12.339 DEBUG:boto:Port: 8001 2023-11-02T18:15:12.339 DEBUG:boto:Params: {} 2023-11-02T18:15:12.340 DEBUG:boto:Token: None 2023-11-02T18:15:12.340 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj4 2023-11-02T18:15:12.340 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ciGjvsWdiIKGgzn+5mB9Vfj1fVE= 2023-11-02T18:15:12.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ciGjvsWdiIKGgzn+5mB9Vfj1fVE=', 'Content-Length': '0'} 2023-11-02T18:15:12.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3d298cb1f13c780-006543e730-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.344 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:15:12.345 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:15:12.345 DEBUG:boto:Method: HEAD 2023-11-02T18:15:12.346 DEBUG:boto:Path: /ibtljn-46/obj4 2023-11-02T18:15:12.346 DEBUG:boto:Data: 2023-11-02T18:15:12.346 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.347 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:12.347 DEBUG:boto:Port: 8000 2023-11-02T18:15:12.348 DEBUG:boto:Params: {} 2023-11-02T18:15:12.348 DEBUG:boto:Token: None 2023-11-02T18:15:12.349 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj4 2023-11-02T18:15:12.349 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ciGjvsWdiIKGgzn+5mB9Vfj1fVE= 2023-11-02T18:15:12.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ciGjvsWdiIKGgzn+5mB9Vfj1fVE=', 'Content-Length': '0'} 2023-11-02T18:15:12.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d88476b85f169fc-006543e730-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:15:12.354 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:15:12.354 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:15:12.355 DEBUG:boto:Method: GET 2023-11-02T18:15:12.355 DEBUG:boto:Path: /ibtljn-46/obj4 2023-11-02T18:15:12.356 DEBUG:boto:Data: 2023-11-02T18:15:12.356 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.356 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:12.357 DEBUG:boto:Port: 8001 2023-11-02T18:15:12.357 DEBUG:boto:Params: {} 2023-11-02T18:15:12.358 DEBUG:boto:Token: None 2023-11-02T18:15:12.358 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj4 2023-11-02T18:15:12.359 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rx5Jqgf9zRheAyXej4NJIv1cJkk= 2023-11-02T18:15:12.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rx5Jqgf9zRheAyXej4NJIv1cJkk=', 'Content-Length': '0'} 2023-11-02T18:15:12.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c40d3da7726bd59-006543e730-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.406 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:15:12.407 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:15:12.407 DEBUG:boto:Method: GET 2023-11-02T18:15:12.408 DEBUG:boto:Path: /ibtljn-46/obj4 2023-11-02T18:15:12.408 DEBUG:boto:Data: 2023-11-02T18:15:12.409 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.409 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:12.410 DEBUG:boto:Port: 8000 2023-11-02T18:15:12.410 DEBUG:boto:Params: {} 2023-11-02T18:15:12.411 DEBUG:boto:Token: None 2023-11-02T18:15:12.411 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/obj4 2023-11-02T18:15:12.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rx5Jqgf9zRheAyXej4NJIv1cJkk= 2023-11-02T18:15:12.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rx5Jqgf9zRheAyXej4NJIv1cJkk=', 'Content-Length': '0'} 2023-11-02T18:15:12.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f80a77b00e49796f-006543e730-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.416 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:12.417 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:12.417 DEBUG:boto:path=/ibtljn-46/?prefix=obj4 2023-11-02T18:15:12.417 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj4 2023-11-02T18:15:12.418 DEBUG:boto:Method: GET 2023-11-02T18:15:12.418 DEBUG:boto:Path: /ibtljn-46/?prefix=obj4 2023-11-02T18:15:12.419 DEBUG:boto:Data: 2023-11-02T18:15:12.419 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.420 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:12.420 DEBUG:boto:Port: 8001 2023-11-02T18:15:12.421 DEBUG:boto:Params: {} 2023-11-02T18:15:12.421 DEBUG:boto:Token: None 2023-11-02T18:15:12.422 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/ 2023-11-02T18:15:12.422 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8= 2023-11-02T18:15:12.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8=', 'Content-Length': '0'} 2023-11-02T18:15:12.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fba3ebbf9f1bf21-006543e730-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.430 DEBUG:boto:b'ibtljn-46obj41000falseobj42023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:12.431 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:12.432 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:12.432 DEBUG:boto:path=/ibtljn-46/?prefix=obj4 2023-11-02T18:15:12.432 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj4 2023-11-02T18:15:12.433 DEBUG:boto:Method: GET 2023-11-02T18:15:12.433 DEBUG:boto:Path: /ibtljn-46/?prefix=obj4 2023-11-02T18:15:12.434 DEBUG:boto:Data: 2023-11-02T18:15:12.434 DEBUG:boto:Headers: {} 2023-11-02T18:15:12.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:12.435 DEBUG:boto:Port: 8000 2023-11-02T18:15:12.435 DEBUG:boto:Params: {} 2023-11-02T18:15:12.436 DEBUG:boto:Token: None 2023-11-02T18:15:12.436 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:12 GMT /ibtljn-46/ 2023-11-02T18:15:12.437 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8= 2023-11-02T18:15:12.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LRl24mzTf3ov55CAiRa7/eLk3X8=', 'Content-Length': '0'} 2023-11-02T18:15:12.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eed81f3da27bd7e-006543e730-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:12.443 DEBUG:boto:b'ibtljn-46obj41000falseobj42023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:12.444 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-46 zones={a2, a1} 2023-11-02T18:15:12.444 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-47 2023-11-02T18:15:12.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:15:12.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:15:12.447 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:15:12.500 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:15:12.500 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:15:12.564 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:12.553+0000 7fe7d8adcf40 1 bucket sync caught up with source: 2023-11-02T18:15:12.565 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.17.6, , 00000000001.21.6, , 00000000001.26.6, , , 00000000001.28.6, , , ] 2023-11-02T18:15:12.565 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.17.6, , 00000000001.21.6, , 00000000001.26.6, , , 00000000001.28.6, , , ] 2023-11-02T18:15:12.565 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:12.553+0000 7fe7d8adcf40 0 bucket checkpoint complete 2023-11-02T18:15:12.592 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-47 2023-11-02T18:15:12.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:15:12.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:15:12.593 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:15:12.645 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:15:12.646 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:15:12.708 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:12.697+0000 7fae7e9f3f40 1 waiting to reach incremental sync.. 2023-11-02T18:15:17.709 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:17.697+0000 7fae7e9f3f40 1 waiting to reach incremental sync.. 2023-11-02T18:15:22.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:22.701+0000 7fae7e9f3f40 1 bucket sync caught up with source: 2023-11-02T18:15:22.713 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.216.6, , 00000000001.214.6, , 00000000001.67.6, , , 00000000001.189.6, , , ] 2023-11-02T18:15:22.713 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.216.6, , 00000000001.214.6, , 00000000001.67.6, , , 00000000001.189.6, , , ] 2023-11-02T18:15:22.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:22.701+0000 7fae7e9f3f40 0 bucket checkpoint complete 2023-11-02T18:15:22.743 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-47 zones={a2, a1} 2023-11-02T18:15:22.744 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:22.744 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:22.745 DEBUG:boto:Method: HEAD 2023-11-02T18:15:22.745 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:15:22.746 DEBUG:boto:Data: 2023-11-02T18:15:22.746 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.746 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:22.747 DEBUG:boto:Port: 8001 2023-11-02T18:15:22.747 DEBUG:boto:Params: {} 2023-11-02T18:15:22.748 DEBUG:boto:Token: None 2023-11-02T18:15:22.748 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/ 2023-11-02T18:15:22.749 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kB9XTcTHcUrYY5nExf8fDKOBCzs= 2023-11-02T18:15:22.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kB9XTcTHcUrYY5nExf8fDKOBCzs=', 'Content-Length': '0'} 2023-11-02T18:15:22.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038f9be23447dde80-006543e73a-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.757 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:22.758 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:22.758 DEBUG:boto:Method: HEAD 2023-11-02T18:15:22.758 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:15:22.759 DEBUG:boto:Data: 2023-11-02T18:15:22.759 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.760 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:22.760 DEBUG:boto:Port: 8000 2023-11-02T18:15:22.761 DEBUG:boto:Params: {} 2023-11-02T18:15:22.761 DEBUG:boto:Token: None 2023-11-02T18:15:22.762 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/ 2023-11-02T18:15:22.762 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kB9XTcTHcUrYY5nExf8fDKOBCzs= 2023-11-02T18:15:22.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kB9XTcTHcUrYY5nExf8fDKOBCzs=', 'Content-Length': '0'} 2023-11-02T18:15:22.767 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001847a5842e69005b-006543e73a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.768 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:15:22.768 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:22.769 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:22.769 DEBUG:boto:path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:15:22.770 DEBUG:boto:auth_path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:15:22.770 DEBUG:boto:Method: GET 2023-11-02T18:15:22.771 DEBUG:boto:Path: /ibtljn-47/?versions&max-keys=999 2023-11-02T18:15:22.771 DEBUG:boto:Data: 2023-11-02T18:15:22.772 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.772 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:22.772 DEBUG:boto:Port: 8001 2023-11-02T18:15:22.773 DEBUG:boto:Params: {} 2023-11-02T18:15:22.773 DEBUG:boto:Token: None 2023-11-02T18:15:22.774 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/?versions 2023-11-02T18:15:22.774 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QgXg8GJ2FAo6ak2qe8kCgHGv2AE= 2023-11-02T18:15:22.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QgXg8GJ2FAo6ak2qe8kCgHGv2AE=', 'Content-Length': '0'} 2023-11-02T18:15:22.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077ba3aef67ff5ac8-006543e73a-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.780 DEBUG:boto:b'ibtljn-47999falseobj1nulltrue2023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:22.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:15:22.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:15:22.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:15:22.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:15:22.783 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:15:22.783 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:22.784 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:22.784 DEBUG:boto:path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:15:22.785 DEBUG:boto:auth_path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:15:22.786 DEBUG:boto:Method: GET 2023-11-02T18:15:22.786 DEBUG:boto:Path: /ibtljn-47/?versions&max-keys=999 2023-11-02T18:15:22.787 DEBUG:boto:Data: 2023-11-02T18:15:22.787 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.787 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:22.788 DEBUG:boto:Port: 8000 2023-11-02T18:15:22.788 DEBUG:boto:Params: {} 2023-11-02T18:15:22.789 DEBUG:boto:Token: None 2023-11-02T18:15:22.789 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/?versions 2023-11-02T18:15:22.790 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QgXg8GJ2FAo6ak2qe8kCgHGv2AE= 2023-11-02T18:15:22.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QgXg8GJ2FAo6ak2qe8kCgHGv2AE=', 'Content-Length': '0'} 2023-11-02T18:15:22.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8e62419b7038ef5-006543e73a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.796 DEBUG:boto:b'ibtljn-47999falseobj1nulltrue2023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:22.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:15:22.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:15:22.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:15:22.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:15:22.798 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:22.799 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:22.799 DEBUG:boto:path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:15:22.799 DEBUG:boto:auth_path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:15:22.800 DEBUG:boto:Method: GET 2023-11-02T18:15:22.800 DEBUG:boto:Path: /ibtljn-47/?versions&max-keys=999 2023-11-02T18:15:22.801 DEBUG:boto:Data: 2023-11-02T18:15:22.801 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.802 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:22.802 DEBUG:boto:Port: 8001 2023-11-02T18:15:22.802 DEBUG:boto:Params: {} 2023-11-02T18:15:22.803 DEBUG:boto:Token: None 2023-11-02T18:15:22.803 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/?versions 2023-11-02T18:15:22.804 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QgXg8GJ2FAo6ak2qe8kCgHGv2AE= 2023-11-02T18:15:22.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QgXg8GJ2FAo6ak2qe8kCgHGv2AE=', 'Content-Length': '0'} 2023-11-02T18:15:22.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adf6c5541051c5e9-006543e73a-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.809 DEBUG:boto:b'ibtljn-47999falseobj1nulltrue2023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:22.810 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:22.811 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:22.811 DEBUG:boto:path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:15:22.811 DEBUG:boto:auth_path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:15:22.812 DEBUG:boto:Method: GET 2023-11-02T18:15:22.812 DEBUG:boto:Path: /ibtljn-47/?versions&max-keys=999 2023-11-02T18:15:22.813 DEBUG:boto:Data: 2023-11-02T18:15:22.813 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.814 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:22.814 DEBUG:boto:Port: 8000 2023-11-02T18:15:22.815 DEBUG:boto:Params: {} 2023-11-02T18:15:22.815 DEBUG:boto:Token: None 2023-11-02T18:15:22.816 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/?versions 2023-11-02T18:15:22.816 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QgXg8GJ2FAo6ak2qe8kCgHGv2AE= 2023-11-02T18:15:22.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QgXg8GJ2FAo6ak2qe8kCgHGv2AE=', 'Content-Length': '0'} 2023-11-02T18:15:22.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c52728b50729be09-006543e73a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.822 DEBUG:boto:b'ibtljn-47999falseobj1nulltrue2023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:22.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:15:22.823 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:15:22.824 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:15:22.824 DEBUG:boto:path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.825 DEBUG:boto:auth_path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.825 DEBUG:boto:Method: GET 2023-11-02T18:15:22.826 DEBUG:boto:Path: /ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.826 DEBUG:boto:Data: 2023-11-02T18:15:22.827 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.827 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:22.828 DEBUG:boto:Port: 8001 2023-11-02T18:15:22.828 DEBUG:boto:Params: {} 2023-11-02T18:15:22.828 DEBUG:boto:Token: None 2023-11-02T18:15:22.829 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.830 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fjVWV+SvsaEQMBO5bcbdnBcJKOA= 2023-11-02T18:15:22.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fjVWV+SvsaEQMBO5bcbdnBcJKOA=', 'Content-Length': '0'} 2023-11-02T18:15:22.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036f4cd7ea7ec1f61-006543e73a-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.874 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:15:22.875 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:15:22.875 DEBUG:boto:path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.876 DEBUG:boto:auth_path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.876 DEBUG:boto:Method: GET 2023-11-02T18:15:22.877 DEBUG:boto:Path: /ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.877 DEBUG:boto:Data: 2023-11-02T18:15:22.878 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.878 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:22.879 DEBUG:boto:Port: 8000 2023-11-02T18:15:22.879 DEBUG:boto:Params: {} 2023-11-02T18:15:22.879 DEBUG:boto:Token: None 2023-11-02T18:15:22.880 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.880 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fjVWV+SvsaEQMBO5bcbdnBcJKOA= 2023-11-02T18:15:22.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fjVWV+SvsaEQMBO5bcbdnBcJKOA=', 'Content-Length': '0'} 2023-11-02T18:15:22.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b8492e80b7fef68-006543e73a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.886 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:15:22.886 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:15:22.887 DEBUG:boto:path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.887 DEBUG:boto:auth_path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.888 DEBUG:boto:Method: HEAD 2023-11-02T18:15:22.888 DEBUG:boto:Path: /ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.889 DEBUG:boto:Data: 2023-11-02T18:15:22.889 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.889 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:22.890 DEBUG:boto:Port: 8001 2023-11-02T18:15:22.890 DEBUG:boto:Params: {} 2023-11-02T18:15:22.891 DEBUG:boto:Token: None 2023-11-02T18:15:22.891 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.891 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X4VpNrp99E7w5SEBkM5rEn7iees= 2023-11-02T18:15:22.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X4VpNrp99E7w5SEBkM5rEn7iees=', 'Content-Length': '0'} 2023-11-02T18:15:22.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a8da127e669d383-006543e73a-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.896 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:15:22.896 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:15:22.896 DEBUG:boto:path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.897 DEBUG:boto:auth_path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.897 DEBUG:boto:Method: HEAD 2023-11-02T18:15:22.898 DEBUG:boto:Path: /ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.898 DEBUG:boto:Data: 2023-11-02T18:15:22.899 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.899 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:22.899 DEBUG:boto:Port: 8000 2023-11-02T18:15:22.900 DEBUG:boto:Params: {} 2023-11-02T18:15:22.900 DEBUG:boto:Token: None 2023-11-02T18:15:22.901 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.901 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X4VpNrp99E7w5SEBkM5rEn7iees= 2023-11-02T18:15:22.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X4VpNrp99E7w5SEBkM5rEn7iees=', 'Content-Length': '0'} 2023-11-02T18:15:22.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000646108722b607628-006543e73a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:15:22.906 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:15:22.906 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:15:22.907 DEBUG:boto:path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.907 DEBUG:boto:auth_path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.908 DEBUG:boto:Method: GET 2023-11-02T18:15:22.908 DEBUG:boto:Path: /ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.908 DEBUG:boto:Data: 2023-11-02T18:15:22.909 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.909 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:22.910 DEBUG:boto:Port: 8001 2023-11-02T18:15:22.910 DEBUG:boto:Params: {} 2023-11-02T18:15:22.911 DEBUG:boto:Token: None 2023-11-02T18:15:22.911 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.912 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fjVWV+SvsaEQMBO5bcbdnBcJKOA= 2023-11-02T18:15:22.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fjVWV+SvsaEQMBO5bcbdnBcJKOA=', 'Content-Length': '0'} 2023-11-02T18:15:22.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f47aa353ed0aff30-006543e73a-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.916 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:15:22.916 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:15:22.917 DEBUG:boto:path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.917 DEBUG:boto:auth_path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.918 DEBUG:boto:Method: GET 2023-11-02T18:15:22.918 DEBUG:boto:Path: /ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.919 DEBUG:boto:Data: 2023-11-02T18:15:22.919 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.920 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:22.920 DEBUG:boto:Port: 8000 2023-11-02T18:15:22.921 DEBUG:boto:Params: {} 2023-11-02T18:15:22.921 DEBUG:boto:Token: None 2023-11-02T18:15:22.922 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/obj1?versionId=null 2023-11-02T18:15:22.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fjVWV+SvsaEQMBO5bcbdnBcJKOA= 2023-11-02T18:15:22.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fjVWV+SvsaEQMBO5bcbdnBcJKOA=', 'Content-Length': '0'} 2023-11-02T18:15:22.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b874eeca2801706e-006543e73a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.926 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:22.927 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:22.927 DEBUG:boto:path=/ibtljn-47/?prefix=obj1 2023-11-02T18:15:22.928 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj1 2023-11-02T18:15:22.928 DEBUG:boto:Method: GET 2023-11-02T18:15:22.929 DEBUG:boto:Path: /ibtljn-47/?prefix=obj1 2023-11-02T18:15:22.929 DEBUG:boto:Data: 2023-11-02T18:15:22.930 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:22.931 DEBUG:boto:Port: 8001 2023-11-02T18:15:22.931 DEBUG:boto:Params: {} 2023-11-02T18:15:22.932 DEBUG:boto:Token: None 2023-11-02T18:15:22.932 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/ 2023-11-02T18:15:22.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TpCTr4gd0YbILm7mQeV7bvClSio= 2023-11-02T18:15:22.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TpCTr4gd0YbILm7mQeV7bvClSio=', 'Content-Length': '0'} 2023-11-02T18:15:22.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f18fca19e7ff836e-006543e73a-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.978 DEBUG:boto:b'ibtljn-47obj11000falseobj12023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:22.979 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:22.980 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:22.980 DEBUG:boto:path=/ibtljn-47/?prefix=obj1 2023-11-02T18:15:22.981 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj1 2023-11-02T18:15:22.981 DEBUG:boto:Method: GET 2023-11-02T18:15:22.982 DEBUG:boto:Path: /ibtljn-47/?prefix=obj1 2023-11-02T18:15:22.982 DEBUG:boto:Data: 2023-11-02T18:15:22.983 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.983 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:22.984 DEBUG:boto:Port: 8000 2023-11-02T18:15:22.984 DEBUG:boto:Params: {} 2023-11-02T18:15:22.985 DEBUG:boto:Token: None 2023-11-02T18:15:22.985 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/ 2023-11-02T18:15:22.986 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TpCTr4gd0YbILm7mQeV7bvClSio= 2023-11-02T18:15:22.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TpCTr4gd0YbILm7mQeV7bvClSio=', 'Content-Length': '0'} 2023-11-02T18:15:22.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aeadd1cf70c2bd4-006543e73a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:22.992 DEBUG:boto:b'ibtljn-47obj11000falseobj12023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:22.992 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:15:22.993 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:15:22.993 DEBUG:boto:Method: HEAD 2023-11-02T18:15:22.994 DEBUG:boto:Path: /ibtljn-47/obj1 2023-11-02T18:15:22.994 DEBUG:boto:Data: 2023-11-02T18:15:22.995 DEBUG:boto:Headers: {} 2023-11-02T18:15:22.995 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:22.996 DEBUG:boto:Port: 8001 2023-11-02T18:15:22.996 DEBUG:boto:Params: {} 2023-11-02T18:15:22.997 DEBUG:boto:Token: None 2023-11-02T18:15:22.997 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:22 GMT /ibtljn-47/obj1 2023-11-02T18:15:22.998 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KmnEcRyi5ZCaGNH/jKYX92NmamU= 2023-11-02T18:15:22.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KmnEcRyi5ZCaGNH/jKYX92NmamU=', 'Content-Length': '0'} 2023-11-02T18:15:23.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d844081b53d8b81-006543e73a-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.002 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:15:23.002 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:15:23.003 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.003 DEBUG:boto:Path: /ibtljn-47/obj1 2023-11-02T18:15:23.004 DEBUG:boto:Data: 2023-11-02T18:15:23.004 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.005 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.005 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.005 DEBUG:boto:Params: {} 2023-11-02T18:15:23.006 DEBUG:boto:Token: None 2023-11-02T18:15:23.006 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj1 2023-11-02T18:15:23.007 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cUFNkAYkt2mnMKJ3oZhWJ6htawI= 2023-11-02T18:15:23.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cUFNkAYkt2mnMKJ3oZhWJ6htawI=', 'Content-Length': '0'} 2023-11-02T18:15:23.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a1811b07c042396-006543e73a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:15:23.012 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:15:23.012 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:15:23.013 DEBUG:boto:Method: GET 2023-11-02T18:15:23.013 DEBUG:boto:Path: /ibtljn-47/obj1 2023-11-02T18:15:23.014 DEBUG:boto:Data: 2023-11-02T18:15:23.014 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.015 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.015 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.015 DEBUG:boto:Params: {} 2023-11-02T18:15:23.016 DEBUG:boto:Token: None 2023-11-02T18:15:23.016 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj1 2023-11-02T18:15:23.017 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T91e+3HN34UDkqjLp0EO55y37jI= 2023-11-02T18:15:23.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T91e+3HN34UDkqjLp0EO55y37jI=', 'Content-Length': '0'} 2023-11-02T18:15:23.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000369706658fee8521-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.021 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:15:23.021 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:15:23.022 DEBUG:boto:Method: GET 2023-11-02T18:15:23.022 DEBUG:boto:Path: /ibtljn-47/obj1 2023-11-02T18:15:23.023 DEBUG:boto:Data: 2023-11-02T18:15:23.023 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.024 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.024 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.025 DEBUG:boto:Params: {} 2023-11-02T18:15:23.025 DEBUG:boto:Token: None 2023-11-02T18:15:23.026 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj1 2023-11-02T18:15:23.026 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T91e+3HN34UDkqjLp0EO55y37jI= 2023-11-02T18:15:23.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T91e+3HN34UDkqjLp0EO55y37jI=', 'Content-Length': '0'} 2023-11-02T18:15:23.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eeced4d85eb8172c-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.074 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.075 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.075 DEBUG:boto:path=/ibtljn-47/?prefix=obj1 2023-11-02T18:15:23.076 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj1 2023-11-02T18:15:23.076 DEBUG:boto:Method: GET 2023-11-02T18:15:23.077 DEBUG:boto:Path: /ibtljn-47/?prefix=obj1 2023-11-02T18:15:23.077 DEBUG:boto:Data: 2023-11-02T18:15:23.078 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.078 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.078 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.079 DEBUG:boto:Params: {} 2023-11-02T18:15:23.079 DEBUG:boto:Token: None 2023-11-02T18:15:23.080 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.080 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f018cc1c97a15e26-006543e73b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.086 DEBUG:boto:b'ibtljn-47obj11000falseobj12023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.087 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.087 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.088 DEBUG:boto:path=/ibtljn-47/?prefix=obj1 2023-11-02T18:15:23.088 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj1 2023-11-02T18:15:23.089 DEBUG:boto:Method: GET 2023-11-02T18:15:23.089 DEBUG:boto:Path: /ibtljn-47/?prefix=obj1 2023-11-02T18:15:23.090 DEBUG:boto:Data: 2023-11-02T18:15:23.090 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.091 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.091 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.092 DEBUG:boto:Params: {} 2023-11-02T18:15:23.092 DEBUG:boto:Token: None 2023-11-02T18:15:23.093 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.093 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cb44d0dcf8e3834-006543e73b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.099 DEBUG:boto:b'ibtljn-47obj11000falseobj12023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:15:23.100 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:15:23.100 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:15:23.101 DEBUG:boto:path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.101 DEBUG:boto:auth_path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.102 DEBUG:boto:Method: GET 2023-11-02T18:15:23.102 DEBUG:boto:Path: /ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.103 DEBUG:boto:Data: 2023-11-02T18:15:23.103 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.104 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.104 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.105 DEBUG:boto:Params: {} 2023-11-02T18:15:23.105 DEBUG:boto:Token: None 2023-11-02T18:15:23.106 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.106 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eU3MFYAxOV5xpnDzMqdzyabFKCA= 2023-11-02T18:15:23.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eU3MFYAxOV5xpnDzMqdzyabFKCA=', 'Content-Length': '0'} 2023-11-02T18:15:23.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b24c763aaac3996-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.110 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:15:23.111 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:15:23.111 DEBUG:boto:path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.112 DEBUG:boto:auth_path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.112 DEBUG:boto:Method: GET 2023-11-02T18:15:23.113 DEBUG:boto:Path: /ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.113 DEBUG:boto:Data: 2023-11-02T18:15:23.113 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.114 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.114 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.115 DEBUG:boto:Params: {} 2023-11-02T18:15:23.115 DEBUG:boto:Token: None 2023-11-02T18:15:23.116 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.117 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eU3MFYAxOV5xpnDzMqdzyabFKCA= 2023-11-02T18:15:23.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eU3MFYAxOV5xpnDzMqdzyabFKCA=', 'Content-Length': '0'} 2023-11-02T18:15:23.120 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000504f430646370c5a-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.121 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:15:23.121 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:15:23.122 DEBUG:boto:path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.122 DEBUG:boto:auth_path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.123 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.123 DEBUG:boto:Path: /ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.123 DEBUG:boto:Data: 2023-11-02T18:15:23.124 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.124 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.125 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.125 DEBUG:boto:Params: {} 2023-11-02T18:15:23.125 DEBUG:boto:Token: None 2023-11-02T18:15:23.126 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.126 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wN2cpA1pbWxUpVPLarr6UwDDDEo= 2023-11-02T18:15:23.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wN2cpA1pbWxUpVPLarr6UwDDDEo=', 'Content-Length': '0'} 2023-11-02T18:15:23.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b55a657fae149e36-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.130 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:15:23.130 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:15:23.131 DEBUG:boto:path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.131 DEBUG:boto:auth_path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.132 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.132 DEBUG:boto:Path: /ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.132 DEBUG:boto:Data: 2023-11-02T18:15:23.133 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.133 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.134 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.134 DEBUG:boto:Params: {} 2023-11-02T18:15:23.135 DEBUG:boto:Token: None 2023-11-02T18:15:23.135 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.136 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wN2cpA1pbWxUpVPLarr6UwDDDEo= 2023-11-02T18:15:23.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wN2cpA1pbWxUpVPLarr6UwDDDEo=', 'Content-Length': '0'} 2023-11-02T18:15:23.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b790a49c141e9c1-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:15:23.140 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:15:23.141 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:15:23.141 DEBUG:boto:path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.141 DEBUG:boto:auth_path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.142 DEBUG:boto:Method: GET 2023-11-02T18:15:23.142 DEBUG:boto:Path: /ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.143 DEBUG:boto:Data: 2023-11-02T18:15:23.143 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.143 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.144 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.144 DEBUG:boto:Params: {} 2023-11-02T18:15:23.145 DEBUG:boto:Token: None 2023-11-02T18:15:23.145 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.146 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eU3MFYAxOV5xpnDzMqdzyabFKCA= 2023-11-02T18:15:23.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eU3MFYAxOV5xpnDzMqdzyabFKCA=', 'Content-Length': '0'} 2023-11-02T18:15:23.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b81e404eb3bdce7-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.150 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:15:23.151 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:15:23.151 DEBUG:boto:path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.152 DEBUG:boto:auth_path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.152 DEBUG:boto:Method: GET 2023-11-02T18:15:23.153 DEBUG:boto:Path: /ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.153 DEBUG:boto:Data: 2023-11-02T18:15:23.154 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.154 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.155 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.155 DEBUG:boto:Params: {} 2023-11-02T18:15:23.156 DEBUG:boto:Token: None 2023-11-02T18:15:23.156 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj2?versionId=null 2023-11-02T18:15:23.157 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eU3MFYAxOV5xpnDzMqdzyabFKCA= 2023-11-02T18:15:23.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eU3MFYAxOV5xpnDzMqdzyabFKCA=', 'Content-Length': '0'} 2023-11-02T18:15:23.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2a362199e71227a-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.202 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.203 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.203 DEBUG:boto:path=/ibtljn-47/?prefix=obj2 2023-11-02T18:15:23.204 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj2 2023-11-02T18:15:23.204 DEBUG:boto:Method: GET 2023-11-02T18:15:23.205 DEBUG:boto:Path: /ibtljn-47/?prefix=obj2 2023-11-02T18:15:23.205 DEBUG:boto:Data: 2023-11-02T18:15:23.206 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.206 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.206 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.207 DEBUG:boto:Params: {} 2023-11-02T18:15:23.207 DEBUG:boto:Token: None 2023-11-02T18:15:23.208 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcfac715f4609d64-006543e73b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.214 DEBUG:boto:b'ibtljn-47obj21000falseobj22023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.215 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.215 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.216 DEBUG:boto:path=/ibtljn-47/?prefix=obj2 2023-11-02T18:15:23.216 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj2 2023-11-02T18:15:23.217 DEBUG:boto:Method: GET 2023-11-02T18:15:23.217 DEBUG:boto:Path: /ibtljn-47/?prefix=obj2 2023-11-02T18:15:23.218 DEBUG:boto:Data: 2023-11-02T18:15:23.218 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.219 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.219 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.219 DEBUG:boto:Params: {} 2023-11-02T18:15:23.220 DEBUG:boto:Token: None 2023-11-02T18:15:23.220 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.221 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdde7c8ebc1104e0-006543e73b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.227 DEBUG:boto:b'ibtljn-47obj21000falseobj22023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.228 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:15:23.228 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:15:23.229 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.229 DEBUG:boto:Path: /ibtljn-47/obj2 2023-11-02T18:15:23.230 DEBUG:boto:Data: 2023-11-02T18:15:23.230 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.231 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.231 DEBUG:boto:Params: {} 2023-11-02T18:15:23.232 DEBUG:boto:Token: None 2023-11-02T18:15:23.232 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj2 2023-11-02T18:15:23.233 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eaDwjPJPNlXqdbaCe1BcTAhLmaE= 2023-11-02T18:15:23.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eaDwjPJPNlXqdbaCe1BcTAhLmaE=', 'Content-Length': '0'} 2023-11-02T18:15:23.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077425b0e6dc88043-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.237 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:15:23.237 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:15:23.238 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.238 DEBUG:boto:Path: /ibtljn-47/obj2 2023-11-02T18:15:23.239 DEBUG:boto:Data: 2023-11-02T18:15:23.239 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.239 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.240 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.240 DEBUG:boto:Params: {} 2023-11-02T18:15:23.241 DEBUG:boto:Token: None 2023-11-02T18:15:23.241 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj2 2023-11-02T18:15:23.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eaDwjPJPNlXqdbaCe1BcTAhLmaE= 2023-11-02T18:15:23.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eaDwjPJPNlXqdbaCe1BcTAhLmaE=', 'Content-Length': '0'} 2023-11-02T18:15:23.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abc363ad227f1461-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:15:23.246 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:15:23.247 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:15:23.247 DEBUG:boto:Method: GET 2023-11-02T18:15:23.248 DEBUG:boto:Path: /ibtljn-47/obj2 2023-11-02T18:15:23.248 DEBUG:boto:Data: 2023-11-02T18:15:23.249 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.249 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.250 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.250 DEBUG:boto:Params: {} 2023-11-02T18:15:23.251 DEBUG:boto:Token: None 2023-11-02T18:15:23.251 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj2 2023-11-02T18:15:23.252 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NhqB1trFPU5Tu6Dz58TVnR8OarM= 2023-11-02T18:15:23.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NhqB1trFPU5Tu6Dz58TVnR8OarM=', 'Content-Length': '0'} 2023-11-02T18:15:23.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc956d457a1ffbcf-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.298 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:15:23.299 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:15:23.299 DEBUG:boto:Method: GET 2023-11-02T18:15:23.300 DEBUG:boto:Path: /ibtljn-47/obj2 2023-11-02T18:15:23.300 DEBUG:boto:Data: 2023-11-02T18:15:23.301 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.301 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.302 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.302 DEBUG:boto:Params: {} 2023-11-02T18:15:23.303 DEBUG:boto:Token: None 2023-11-02T18:15:23.303 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj2 2023-11-02T18:15:23.304 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NhqB1trFPU5Tu6Dz58TVnR8OarM= 2023-11-02T18:15:23.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NhqB1trFPU5Tu6Dz58TVnR8OarM=', 'Content-Length': '0'} 2023-11-02T18:15:23.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006229e718d1600383-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.308 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.309 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.309 DEBUG:boto:path=/ibtljn-47/?prefix=obj2 2023-11-02T18:15:23.309 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj2 2023-11-02T18:15:23.310 DEBUG:boto:Method: GET 2023-11-02T18:15:23.310 DEBUG:boto:Path: /ibtljn-47/?prefix=obj2 2023-11-02T18:15:23.311 DEBUG:boto:Data: 2023-11-02T18:15:23.311 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.312 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.312 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.313 DEBUG:boto:Params: {} 2023-11-02T18:15:23.313 DEBUG:boto:Token: None 2023-11-02T18:15:23.314 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.314 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000627cbdbbcd2777f6-006543e73b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.320 DEBUG:boto:b'ibtljn-47obj21000falseobj22023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.321 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.321 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.322 DEBUG:boto:path=/ibtljn-47/?prefix=obj2 2023-11-02T18:15:23.322 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj2 2023-11-02T18:15:23.323 DEBUG:boto:Method: GET 2023-11-02T18:15:23.323 DEBUG:boto:Path: /ibtljn-47/?prefix=obj2 2023-11-02T18:15:23.324 DEBUG:boto:Data: 2023-11-02T18:15:23.324 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.325 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.325 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.325 DEBUG:boto:Params: {} 2023-11-02T18:15:23.326 DEBUG:boto:Token: None 2023-11-02T18:15:23.326 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.327 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000639bad05748b8281-006543e73b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.378 DEBUG:boto:b'ibtljn-47obj21000falseobj22023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:15:23.380 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:15:23.380 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:15:23.380 DEBUG:boto:path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.381 DEBUG:boto:auth_path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.381 DEBUG:boto:Method: GET 2023-11-02T18:15:23.382 DEBUG:boto:Path: /ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.382 DEBUG:boto:Data: 2023-11-02T18:15:23.383 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.384 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.384 DEBUG:boto:Params: {} 2023-11-02T18:15:23.385 DEBUG:boto:Token: None 2023-11-02T18:15:23.385 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.386 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZuwC+qQBAy+Q0vntIFbFyyuPgAI= 2023-11-02T18:15:23.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZuwC+qQBAy+Q0vntIFbFyyuPgAI=', 'Content-Length': '0'} 2023-11-02T18:15:23.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7f9f57356de7be9-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.390 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:15:23.391 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:15:23.391 DEBUG:boto:path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.392 DEBUG:boto:auth_path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.392 DEBUG:boto:Method: GET 2023-11-02T18:15:23.392 DEBUG:boto:Path: /ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.393 DEBUG:boto:Data: 2023-11-02T18:15:23.393 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.394 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.394 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.395 DEBUG:boto:Params: {} 2023-11-02T18:15:23.395 DEBUG:boto:Token: None 2023-11-02T18:15:23.395 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.396 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZuwC+qQBAy+Q0vntIFbFyyuPgAI= 2023-11-02T18:15:23.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZuwC+qQBAy+Q0vntIFbFyyuPgAI=', 'Content-Length': '0'} 2023-11-02T18:15:23.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee51afed30a6e818-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.400 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:15:23.401 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:15:23.401 DEBUG:boto:path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.402 DEBUG:boto:auth_path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.402 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.402 DEBUG:boto:Path: /ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.403 DEBUG:boto:Data: 2023-11-02T18:15:23.403 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.404 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.404 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.405 DEBUG:boto:Params: {} 2023-11-02T18:15:23.405 DEBUG:boto:Token: None 2023-11-02T18:15:23.405 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.406 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IOTT2vUPGXQ/9LYoyrQmlLLcEX0= 2023-11-02T18:15:23.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IOTT2vUPGXQ/9LYoyrQmlLLcEX0=', 'Content-Length': '0'} 2023-11-02T18:15:23.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b9e40ab76d7d8a5-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.410 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:15:23.410 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:15:23.411 DEBUG:boto:path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.411 DEBUG:boto:auth_path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.412 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.412 DEBUG:boto:Path: /ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.413 DEBUG:boto:Data: 2023-11-02T18:15:23.413 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.413 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.414 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.414 DEBUG:boto:Params: {} 2023-11-02T18:15:23.415 DEBUG:boto:Token: None 2023-11-02T18:15:23.415 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.416 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IOTT2vUPGXQ/9LYoyrQmlLLcEX0= 2023-11-02T18:15:23.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IOTT2vUPGXQ/9LYoyrQmlLLcEX0=', 'Content-Length': '0'} 2023-11-02T18:15:23.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b905bdbf6bdd82b-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:15:23.421 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:15:23.421 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:15:23.422 DEBUG:boto:path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.422 DEBUG:boto:auth_path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.422 DEBUG:boto:Method: GET 2023-11-02T18:15:23.423 DEBUG:boto:Path: /ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.423 DEBUG:boto:Data: 2023-11-02T18:15:23.424 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.424 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.425 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.425 DEBUG:boto:Params: {} 2023-11-02T18:15:23.426 DEBUG:boto:Token: None 2023-11-02T18:15:23.426 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.427 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZuwC+qQBAy+Q0vntIFbFyyuPgAI= 2023-11-02T18:15:23.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZuwC+qQBAy+Q0vntIFbFyyuPgAI=', 'Content-Length': '0'} 2023-11-02T18:15:23.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb281165fa7ffcd4-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.431 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:15:23.431 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:15:23.432 DEBUG:boto:path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.432 DEBUG:boto:auth_path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.433 DEBUG:boto:Method: GET 2023-11-02T18:15:23.433 DEBUG:boto:Path: /ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.434 DEBUG:boto:Data: 2023-11-02T18:15:23.434 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.435 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.436 DEBUG:boto:Params: {} 2023-11-02T18:15:23.436 DEBUG:boto:Token: None 2023-11-02T18:15:23.437 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj3?versionId=null 2023-11-02T18:15:23.437 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZuwC+qQBAy+Q0vntIFbFyyuPgAI= 2023-11-02T18:15:23.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZuwC+qQBAy+Q0vntIFbFyyuPgAI=', 'Content-Length': '0'} 2023-11-02T18:15:23.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000674718353bf1d07f-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.482 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.483 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.483 DEBUG:boto:path=/ibtljn-47/?prefix=obj3 2023-11-02T18:15:23.484 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj3 2023-11-02T18:15:23.484 DEBUG:boto:Method: GET 2023-11-02T18:15:23.485 DEBUG:boto:Path: /ibtljn-47/?prefix=obj3 2023-11-02T18:15:23.485 DEBUG:boto:Data: 2023-11-02T18:15:23.485 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.486 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.486 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.487 DEBUG:boto:Params: {} 2023-11-02T18:15:23.487 DEBUG:boto:Token: None 2023-11-02T18:15:23.488 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.488 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000088c46f0c0d9dd6f-006543e73b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.494 DEBUG:boto:b'ibtljn-47obj31000falseobj32023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.495 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.496 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.496 DEBUG:boto:path=/ibtljn-47/?prefix=obj3 2023-11-02T18:15:23.496 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj3 2023-11-02T18:15:23.497 DEBUG:boto:Method: GET 2023-11-02T18:15:23.497 DEBUG:boto:Path: /ibtljn-47/?prefix=obj3 2023-11-02T18:15:23.498 DEBUG:boto:Data: 2023-11-02T18:15:23.498 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.499 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.499 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.499 DEBUG:boto:Params: {} 2023-11-02T18:15:23.500 DEBUG:boto:Token: None 2023-11-02T18:15:23.500 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.501 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d1d150d2c2e0bdb-006543e73b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.507 DEBUG:boto:b'ibtljn-47obj31000falseobj32023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.508 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:15:23.508 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:15:23.508 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.509 DEBUG:boto:Path: /ibtljn-47/obj3 2023-11-02T18:15:23.509 DEBUG:boto:Data: 2023-11-02T18:15:23.510 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.511 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.511 DEBUG:boto:Params: {} 2023-11-02T18:15:23.511 DEBUG:boto:Token: None 2023-11-02T18:15:23.512 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj3 2023-11-02T18:15:23.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJ30xuAXSOYGIADEOpGIJu5Ddck= 2023-11-02T18:15:23.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJ30xuAXSOYGIADEOpGIJu5Ddck=', 'Content-Length': '0'} 2023-11-02T18:15:23.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b591b00c0941176-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.516 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:15:23.517 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:15:23.517 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.518 DEBUG:boto:Path: /ibtljn-47/obj3 2023-11-02T18:15:23.518 DEBUG:boto:Data: 2023-11-02T18:15:23.519 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.519 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.520 DEBUG:boto:Params: {} 2023-11-02T18:15:23.520 DEBUG:boto:Token: None 2023-11-02T18:15:23.521 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj3 2023-11-02T18:15:23.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJ30xuAXSOYGIADEOpGIJu5Ddck= 2023-11-02T18:15:23.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJ30xuAXSOYGIADEOpGIJu5Ddck=', 'Content-Length': '0'} 2023-11-02T18:15:23.524 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6b9afe836b44fdc-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:15:23.525 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:15:23.526 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:15:23.526 DEBUG:boto:Method: GET 2023-11-02T18:15:23.527 DEBUG:boto:Path: /ibtljn-47/obj3 2023-11-02T18:15:23.527 DEBUG:boto:Data: 2023-11-02T18:15:23.528 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.528 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.529 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.529 DEBUG:boto:Params: {} 2023-11-02T18:15:23.530 DEBUG:boto:Token: None 2023-11-02T18:15:23.530 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj3 2023-11-02T18:15:23.531 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ydtQi10sttCTOdc3OipsKMElXd8= 2023-11-02T18:15:23.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ydtQi10sttCTOdc3OipsKMElXd8=', 'Content-Length': '0'} 2023-11-02T18:15:23.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048c8a4c33650ba9e-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.582 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:15:23.583 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:15:23.583 DEBUG:boto:Method: GET 2023-11-02T18:15:23.584 DEBUG:boto:Path: /ibtljn-47/obj3 2023-11-02T18:15:23.584 DEBUG:boto:Data: 2023-11-02T18:15:23.585 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.585 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.586 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.586 DEBUG:boto:Params: {} 2023-11-02T18:15:23.587 DEBUG:boto:Token: None 2023-11-02T18:15:23.587 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj3 2023-11-02T18:15:23.588 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ydtQi10sttCTOdc3OipsKMElXd8= 2023-11-02T18:15:23.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ydtQi10sttCTOdc3OipsKMElXd8=', 'Content-Length': '0'} 2023-11-02T18:15:23.591 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005911d42a5eba75be-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.592 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.593 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.593 DEBUG:boto:path=/ibtljn-47/?prefix=obj3 2023-11-02T18:15:23.593 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj3 2023-11-02T18:15:23.594 DEBUG:boto:Method: GET 2023-11-02T18:15:23.594 DEBUG:boto:Path: /ibtljn-47/?prefix=obj3 2023-11-02T18:15:23.595 DEBUG:boto:Data: 2023-11-02T18:15:23.595 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.596 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.596 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.597 DEBUG:boto:Params: {} 2023-11-02T18:15:23.597 DEBUG:boto:Token: None 2023-11-02T18:15:23.598 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.598 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d2c2e8d0a9a0884-006543e73b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.604 DEBUG:boto:b'ibtljn-47obj31000falseobj32023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.605 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.605 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.606 DEBUG:boto:path=/ibtljn-47/?prefix=obj3 2023-11-02T18:15:23.606 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj3 2023-11-02T18:15:23.607 DEBUG:boto:Method: GET 2023-11-02T18:15:23.607 DEBUG:boto:Path: /ibtljn-47/?prefix=obj3 2023-11-02T18:15:23.608 DEBUG:boto:Data: 2023-11-02T18:15:23.608 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.609 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.609 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.610 DEBUG:boto:Params: {} 2023-11-02T18:15:23.610 DEBUG:boto:Token: None 2023-11-02T18:15:23.611 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.611 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd773ed149a1332f-006543e73b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.662 DEBUG:boto:b'ibtljn-47obj31000falseobj32023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:15:23.664 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:15:23.664 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:15:23.665 DEBUG:boto:path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.665 DEBUG:boto:auth_path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.666 DEBUG:boto:Method: GET 2023-11-02T18:15:23.666 DEBUG:boto:Path: /ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.666 DEBUG:boto:Data: 2023-11-02T18:15:23.667 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.667 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.668 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.668 DEBUG:boto:Params: {} 2023-11-02T18:15:23.669 DEBUG:boto:Token: None 2023-11-02T18:15:23.669 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.670 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H85z/YGLpRgkSSMI8KsbzDuHyKQ= 2023-11-02T18:15:23.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H85z/YGLpRgkSSMI8KsbzDuHyKQ=', 'Content-Length': '0'} 2023-11-02T18:15:23.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d673fcf2d7fae3b4-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.674 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:15:23.675 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:15:23.675 DEBUG:boto:path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.676 DEBUG:boto:auth_path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.676 DEBUG:boto:Method: GET 2023-11-02T18:15:23.677 DEBUG:boto:Path: /ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.677 DEBUG:boto:Data: 2023-11-02T18:15:23.678 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.678 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.678 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.679 DEBUG:boto:Params: {} 2023-11-02T18:15:23.679 DEBUG:boto:Token: None 2023-11-02T18:15:23.680 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.680 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H85z/YGLpRgkSSMI8KsbzDuHyKQ= 2023-11-02T18:15:23.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H85z/YGLpRgkSSMI8KsbzDuHyKQ=', 'Content-Length': '0'} 2023-11-02T18:15:23.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002762a2f35be61eef-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.685 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:15:23.685 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:15:23.686 DEBUG:boto:path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.686 DEBUG:boto:auth_path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.687 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.687 DEBUG:boto:Path: /ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.688 DEBUG:boto:Data: 2023-11-02T18:15:23.688 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.689 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.689 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.689 DEBUG:boto:Params: {} 2023-11-02T18:15:23.690 DEBUG:boto:Token: None 2023-11-02T18:15:23.690 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.691 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w6VluQ2z2+3xGnBfPQ360SFMKiM= 2023-11-02T18:15:23.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w6VluQ2z2+3xGnBfPQ360SFMKiM=', 'Content-Length': '0'} 2023-11-02T18:15:23.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f98e724887abf492-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.695 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:15:23.695 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:15:23.696 DEBUG:boto:path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.696 DEBUG:boto:auth_path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.697 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.697 DEBUG:boto:Path: /ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.698 DEBUG:boto:Data: 2023-11-02T18:15:23.698 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.699 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.699 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.699 DEBUG:boto:Params: {} 2023-11-02T18:15:23.700 DEBUG:boto:Token: None 2023-11-02T18:15:23.700 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.701 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w6VluQ2z2+3xGnBfPQ360SFMKiM= 2023-11-02T18:15:23.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w6VluQ2z2+3xGnBfPQ360SFMKiM=', 'Content-Length': '0'} 2023-11-02T18:15:23.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071515c3ae3b89688-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:15:23.706 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:15:23.706 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:15:23.707 DEBUG:boto:path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.707 DEBUG:boto:auth_path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.708 DEBUG:boto:Method: GET 2023-11-02T18:15:23.708 DEBUG:boto:Path: /ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.708 DEBUG:boto:Data: 2023-11-02T18:15:23.709 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.709 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.710 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.710 DEBUG:boto:Params: {} 2023-11-02T18:15:23.711 DEBUG:boto:Token: None 2023-11-02T18:15:23.711 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H85z/YGLpRgkSSMI8KsbzDuHyKQ= 2023-11-02T18:15:23.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H85z/YGLpRgkSSMI8KsbzDuHyKQ=', 'Content-Length': '0'} 2023-11-02T18:15:23.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003568634cdafdd556-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.716 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:15:23.717 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:15:23.717 DEBUG:boto:path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.718 DEBUG:boto:auth_path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.718 DEBUG:boto:Method: GET 2023-11-02T18:15:23.719 DEBUG:boto:Path: /ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.719 DEBUG:boto:Data: 2023-11-02T18:15:23.720 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.720 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.720 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.721 DEBUG:boto:Params: {} 2023-11-02T18:15:23.721 DEBUG:boto:Token: None 2023-11-02T18:15:23.722 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj4?versionId=null 2023-11-02T18:15:23.722 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H85z/YGLpRgkSSMI8KsbzDuHyKQ= 2023-11-02T18:15:23.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H85z/YGLpRgkSSMI8KsbzDuHyKQ=', 'Content-Length': '0'} 2023-11-02T18:15:23.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecfe7d75bbc729b1-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.727 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.727 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.728 DEBUG:boto:path=/ibtljn-47/?prefix=obj4 2023-11-02T18:15:23.728 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj4 2023-11-02T18:15:23.729 DEBUG:boto:Method: GET 2023-11-02T18:15:23.729 DEBUG:boto:Path: /ibtljn-47/?prefix=obj4 2023-11-02T18:15:23.730 DEBUG:boto:Data: 2023-11-02T18:15:23.730 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.730 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.731 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.731 DEBUG:boto:Params: {} 2023-11-02T18:15:23.732 DEBUG:boto:Token: None 2023-11-02T18:15:23.732 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.733 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d4c6693309497d8-006543e73b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.782 DEBUG:boto:b'ibtljn-47obj41000falseobj42023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.783 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.784 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.784 DEBUG:boto:path=/ibtljn-47/?prefix=obj4 2023-11-02T18:15:23.785 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj4 2023-11-02T18:15:23.785 DEBUG:boto:Method: GET 2023-11-02T18:15:23.786 DEBUG:boto:Path: /ibtljn-47/?prefix=obj4 2023-11-02T18:15:23.786 DEBUG:boto:Data: 2023-11-02T18:15:23.787 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.787 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.788 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.788 DEBUG:boto:Params: {} 2023-11-02T18:15:23.788 DEBUG:boto:Token: None 2023-11-02T18:15:23.789 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085ad12fe34c1cc14-006543e73b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.795 DEBUG:boto:b'ibtljn-47obj41000falseobj42023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.796 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:15:23.797 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:15:23.797 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.798 DEBUG:boto:Path: /ibtljn-47/obj4 2023-11-02T18:15:23.798 DEBUG:boto:Data: 2023-11-02T18:15:23.798 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.799 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.800 DEBUG:boto:Params: {} 2023-11-02T18:15:23.800 DEBUG:boto:Token: None 2023-11-02T18:15:23.801 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj4 2023-11-02T18:15:23.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SGXo2QpQwlkyLcaGmzlNoiTswVo= 2023-11-02T18:15:23.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SGXo2QpQwlkyLcaGmzlNoiTswVo=', 'Content-Length': '0'} 2023-11-02T18:15:23.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035eb127e606aceab-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.805 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:15:23.805 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:15:23.806 DEBUG:boto:Method: HEAD 2023-11-02T18:15:23.806 DEBUG:boto:Path: /ibtljn-47/obj4 2023-11-02T18:15:23.807 DEBUG:boto:Data: 2023-11-02T18:15:23.807 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.808 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.808 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.808 DEBUG:boto:Params: {} 2023-11-02T18:15:23.809 DEBUG:boto:Token: None 2023-11-02T18:15:23.809 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj4 2023-11-02T18:15:23.810 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SGXo2QpQwlkyLcaGmzlNoiTswVo= 2023-11-02T18:15:23.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SGXo2QpQwlkyLcaGmzlNoiTswVo=', 'Content-Length': '0'} 2023-11-02T18:15:23.813 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac222fb0c04bd845-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:15:23.815 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:15:23.815 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:15:23.815 DEBUG:boto:Method: GET 2023-11-02T18:15:23.816 DEBUG:boto:Path: /ibtljn-47/obj4 2023-11-02T18:15:23.816 DEBUG:boto:Data: 2023-11-02T18:15:23.817 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.817 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.818 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.818 DEBUG:boto:Params: {} 2023-11-02T18:15:23.819 DEBUG:boto:Token: None 2023-11-02T18:15:23.819 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj4 2023-11-02T18:15:23.820 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qAxRq8rQ4Z+0xcHT5Ug114AbC04= 2023-11-02T18:15:23.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qAxRq8rQ4Z+0xcHT5Ug114AbC04=', 'Content-Length': '0'} 2023-11-02T18:15:23.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082666946997bd32d-006543e73b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.824 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:15:23.824 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:15:23.825 DEBUG:boto:Method: GET 2023-11-02T18:15:23.825 DEBUG:boto:Path: /ibtljn-47/obj4 2023-11-02T18:15:23.826 DEBUG:boto:Data: 2023-11-02T18:15:23.826 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.827 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.827 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.827 DEBUG:boto:Params: {} 2023-11-02T18:15:23.828 DEBUG:boto:Token: None 2023-11-02T18:15:23.828 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/obj4 2023-11-02T18:15:23.829 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qAxRq8rQ4Z+0xcHT5Ug114AbC04= 2023-11-02T18:15:23.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qAxRq8rQ4Z+0xcHT5Ug114AbC04=', 'Content-Length': '0'} 2023-11-02T18:15:23.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000658f7274c30a6397-006543e73b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.874 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.875 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.875 DEBUG:boto:path=/ibtljn-47/?prefix=obj4 2023-11-02T18:15:23.876 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj4 2023-11-02T18:15:23.876 DEBUG:boto:Method: GET 2023-11-02T18:15:23.877 DEBUG:boto:Path: /ibtljn-47/?prefix=obj4 2023-11-02T18:15:23.877 DEBUG:boto:Data: 2023-11-02T18:15:23.878 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.878 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:23.879 DEBUG:boto:Port: 8001 2023-11-02T18:15:23.879 DEBUG:boto:Params: {} 2023-11-02T18:15:23.880 DEBUG:boto:Token: None 2023-11-02T18:15:23.880 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.881 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdd629fe1768810b-006543e73b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.887 DEBUG:boto:b'ibtljn-47obj41000falseobj42023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.887 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:23.888 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:23.888 DEBUG:boto:path=/ibtljn-47/?prefix=obj4 2023-11-02T18:15:23.889 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj4 2023-11-02T18:15:23.889 DEBUG:boto:Method: GET 2023-11-02T18:15:23.890 DEBUG:boto:Path: /ibtljn-47/?prefix=obj4 2023-11-02T18:15:23.890 DEBUG:boto:Data: 2023-11-02T18:15:23.891 DEBUG:boto:Headers: {} 2023-11-02T18:15:23.891 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:23.892 DEBUG:boto:Port: 8000 2023-11-02T18:15:23.892 DEBUG:boto:Params: {} 2023-11-02T18:15:23.892 DEBUG:boto:Token: None 2023-11-02T18:15:23.893 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:15:23 GMT /ibtljn-47/ 2023-11-02T18:15:23.893 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y= 2023-11-02T18:15:23.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E3zbLEhf1ZssjLlyYt2SYLn7I2Y=', 'Content-Length': '0'} 2023-11-02T18:15:23.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7a4e2629585353e-006543e73b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:15:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:23.899 DEBUG:boto:b'ibtljn-47obj41000falseobj42023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:15:23.900 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-47 zones={a2, a1} 2023-11-02T18:15:23.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ibtljn-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:15:23.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ibtljn-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:15:23.902 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ibtljn-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:15:23.948 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:15:23.948 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:15:24.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'ibtljn-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:15:24.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'ibtljn-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:15:24.092 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket ibtljn-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:15:24.138 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:15:24.139 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:15:24.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ibtljn-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:15:24.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ibtljn-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:15:24.281 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ibtljn-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:15:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:15:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:15:24.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'ibtljn-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:15:24.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'ibtljn-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:15:24.458 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket ibtljn-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:15:24.502 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:15:24.503 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:15:24.642 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:24.642 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:24.643 DEBUG:boto:Method: HEAD 2023-11-02T18:15:24.643 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:15:24.644 DEBUG:boto:Data: 2023-11-02T18:15:24.644 DEBUG:boto:Headers: {} 2023-11-02T18:15:24.645 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:24.645 DEBUG:boto:Port: 8000 2023-11-02T18:15:24.645 DEBUG:boto:Params: {} 2023-11-02T18:15:24.646 DEBUG:boto:Token: None 2023-11-02T18:15:24.647 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-46/ 2023-11-02T18:15:24.648 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UtYsoz2Fk10XSXD7znQJ1QWrVZ8= 2023-11-02T18:15:24.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UtYsoz2Fk10XSXD7znQJ1QWrVZ8=', 'Content-Length': '0'} 2023-11-02T18:15:24.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000773d6b4ee5af89d-006543e73c-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.654 DEBUG:boto:path=/ibtljn-46/obj5 2023-11-02T18:15:24.655 DEBUG:boto:auth_path=/ibtljn-46/obj5 2023-11-02T18:15:24.655 DEBUG:boto:Method: PUT 2023-11-02T18:15:24.656 DEBUG:boto:Path: /ibtljn-46/obj5 2023-11-02T18:15:24.656 DEBUG:boto:Data: 2023-11-02T18:15:24.657 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:15:24.657 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:24.658 DEBUG:boto:Port: 8000 2023-11-02T18:15:24.658 DEBUG:boto:Params: {} 2023-11-02T18:15:24.659 DEBUG:boto:Token: None 2023-11-02T18:15:24.659 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-46/obj5 2023-11-02T18:15:24.660 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wgfyQxiYjSHgJ5x349ATDQDBlYw= 2023-11-02T18:15:24.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wgfyQxiYjSHgJ5x349ATDQDBlYw='} 2023-11-02T18:15:24.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025ac4c190ebf742a-006543e73c-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.673 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:24.673 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:24.674 DEBUG:boto:Method: HEAD 2023-11-02T18:15:24.674 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:15:24.675 DEBUG:boto:Data: 2023-11-02T18:15:24.675 DEBUG:boto:Headers: {} 2023-11-02T18:15:24.675 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:24.676 DEBUG:boto:Port: 8000 2023-11-02T18:15:24.676 DEBUG:boto:Params: {} 2023-11-02T18:15:24.676 DEBUG:boto:Token: None 2023-11-02T18:15:24.677 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-46/ 2023-11-02T18:15:24.677 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UtYsoz2Fk10XSXD7znQJ1QWrVZ8= 2023-11-02T18:15:24.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UtYsoz2Fk10XSXD7znQJ1QWrVZ8=', 'Content-Length': '0'} 2023-11-02T18:15:24.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a206d5d05588d1b-006543e73c-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.683 DEBUG:boto:path=/ibtljn-46/obj6 2023-11-02T18:15:24.684 DEBUG:boto:auth_path=/ibtljn-46/obj6 2023-11-02T18:15:24.684 DEBUG:boto:Method: PUT 2023-11-02T18:15:24.685 DEBUG:boto:Path: /ibtljn-46/obj6 2023-11-02T18:15:24.685 DEBUG:boto:Data: 2023-11-02T18:15:24.686 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:15:24.687 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:24.687 DEBUG:boto:Port: 8000 2023-11-02T18:15:24.688 DEBUG:boto:Params: {} 2023-11-02T18:15:24.688 DEBUG:boto:Token: None 2023-11-02T18:15:24.689 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-46/obj6 2023-11-02T18:15:24.689 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kTYFAxr8puTw1NvYe6YwVerS+W4= 2023-11-02T18:15:24.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kTYFAxr8puTw1NvYe6YwVerS+W4='} 2023-11-02T18:15:24.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c0d8aa169dd1262-006543e73c-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.735 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:24.736 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:24.736 DEBUG:boto:Method: HEAD 2023-11-02T18:15:24.736 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:15:24.737 DEBUG:boto:Data: 2023-11-02T18:15:24.737 DEBUG:boto:Headers: {} 2023-11-02T18:15:24.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:24.738 DEBUG:boto:Port: 8000 2023-11-02T18:15:24.738 DEBUG:boto:Params: {} 2023-11-02T18:15:24.739 DEBUG:boto:Token: None 2023-11-02T18:15:24.739 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-46/ 2023-11-02T18:15:24.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UtYsoz2Fk10XSXD7znQJ1QWrVZ8= 2023-11-02T18:15:24.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UtYsoz2Fk10XSXD7znQJ1QWrVZ8=', 'Content-Length': '0'} 2023-11-02T18:15:24.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1bc59a4c25522ca-006543e73c-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.746 DEBUG:boto:path=/ibtljn-46/obj7 2023-11-02T18:15:24.746 DEBUG:boto:auth_path=/ibtljn-46/obj7 2023-11-02T18:15:24.747 DEBUG:boto:Method: PUT 2023-11-02T18:15:24.747 DEBUG:boto:Path: /ibtljn-46/obj7 2023-11-02T18:15:24.748 DEBUG:boto:Data: 2023-11-02T18:15:24.749 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:15:24.749 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:24.750 DEBUG:boto:Port: 8000 2023-11-02T18:15:24.751 DEBUG:boto:Params: {} 2023-11-02T18:15:24.752 DEBUG:boto:Token: None 2023-11-02T18:15:24.752 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-46/obj7 2023-11-02T18:15:24.753 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sb4ziEdSWrh9Sc35NscrvWy0coE= 2023-11-02T18:15:24.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sb4ziEdSWrh9Sc35NscrvWy0coE='} 2023-11-02T18:15:24.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb9bb1b2e6f3caf6-006543e73c-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.767 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:15:24.768 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:15:24.768 DEBUG:boto:Method: HEAD 2023-11-02T18:15:24.769 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:15:24.769 DEBUG:boto:Data: 2023-11-02T18:15:24.769 DEBUG:boto:Headers: {} 2023-11-02T18:15:24.770 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:24.770 DEBUG:boto:Port: 8000 2023-11-02T18:15:24.771 DEBUG:boto:Params: {} 2023-11-02T18:15:24.771 DEBUG:boto:Token: None 2023-11-02T18:15:24.772 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-46/ 2023-11-02T18:15:24.772 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UtYsoz2Fk10XSXD7znQJ1QWrVZ8= 2023-11-02T18:15:24.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UtYsoz2Fk10XSXD7znQJ1QWrVZ8=', 'Content-Length': '0'} 2023-11-02T18:15:24.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0fc604dce3e8347-006543e73c-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.777 DEBUG:boto:path=/ibtljn-46/obj8 2023-11-02T18:15:24.777 DEBUG:boto:auth_path=/ibtljn-46/obj8 2023-11-02T18:15:24.778 DEBUG:boto:Method: PUT 2023-11-02T18:15:24.778 DEBUG:boto:Path: /ibtljn-46/obj8 2023-11-02T18:15:24.779 DEBUG:boto:Data: 2023-11-02T18:15:24.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:15:24.780 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:15:24.780 DEBUG:boto:Port: 8000 2023-11-02T18:15:24.780 DEBUG:boto:Params: {} 2023-11-02T18:15:24.781 DEBUG:boto:Token: None 2023-11-02T18:15:24.781 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-46/obj8 2023-11-02T18:15:24.782 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SjEQHD8SL4FiCzPHZ6dihq9TEtk= 2023-11-02T18:15:24.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SjEQHD8SL4FiCzPHZ6dihq9TEtk='} 2023-11-02T18:15:24.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b107b13c1bedb21e-006543e73c-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.827 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:24.828 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:24.828 DEBUG:boto:Method: HEAD 2023-11-02T18:15:24.829 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:15:24.829 DEBUG:boto:Data: 2023-11-02T18:15:24.829 DEBUG:boto:Headers: {} 2023-11-02T18:15:24.830 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:24.831 DEBUG:boto:Port: 8001 2023-11-02T18:15:24.831 DEBUG:boto:Params: {} 2023-11-02T18:15:24.832 DEBUG:boto:Token: None 2023-11-02T18:15:24.832 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-47/ 2023-11-02T18:15:24.833 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VSs9kI/GsJemQePyjpl9ZWwcQAk= 2023-11-02T18:15:24.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VSs9kI/GsJemQePyjpl9ZWwcQAk=', 'Content-Length': '0'} 2023-11-02T18:15:24.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4893590e6167be1-006543e73c-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.838 DEBUG:boto:path=/ibtljn-47/obj5 2023-11-02T18:15:24.838 DEBUG:boto:auth_path=/ibtljn-47/obj5 2023-11-02T18:15:24.839 DEBUG:boto:Method: PUT 2023-11-02T18:15:24.839 DEBUG:boto:Path: /ibtljn-47/obj5 2023-11-02T18:15:24.840 DEBUG:boto:Data: 2023-11-02T18:15:24.840 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:15:24.840 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:24.841 DEBUG:boto:Port: 8001 2023-11-02T18:15:24.841 DEBUG:boto:Params: {} 2023-11-02T18:15:24.842 DEBUG:boto:Token: None 2023-11-02T18:15:24.842 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-47/obj5 2023-11-02T18:15:24.843 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9rPXSKEtAZpYRlVWF8kiW9u/kgo= 2023-11-02T18:15:24.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9rPXSKEtAZpYRlVWF8kiW9u/kgo='} 2023-11-02T18:15:24.856 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071ae41ce080d2c65-006543e73c-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.857 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:24.858 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:24.858 DEBUG:boto:Method: HEAD 2023-11-02T18:15:24.859 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:15:24.859 DEBUG:boto:Data: 2023-11-02T18:15:24.860 DEBUG:boto:Headers: {} 2023-11-02T18:15:24.860 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:24.860 DEBUG:boto:Port: 8001 2023-11-02T18:15:24.861 DEBUG:boto:Params: {} 2023-11-02T18:15:24.861 DEBUG:boto:Token: None 2023-11-02T18:15:24.862 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-47/ 2023-11-02T18:15:24.862 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VSs9kI/GsJemQePyjpl9ZWwcQAk= 2023-11-02T18:15:24.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VSs9kI/GsJemQePyjpl9ZWwcQAk=', 'Content-Length': '0'} 2023-11-02T18:15:24.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053d1c05b9011c985-006543e73c-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.867 DEBUG:boto:path=/ibtljn-47/obj6 2023-11-02T18:15:24.868 DEBUG:boto:auth_path=/ibtljn-47/obj6 2023-11-02T18:15:24.868 DEBUG:boto:Method: PUT 2023-11-02T18:15:24.868 DEBUG:boto:Path: /ibtljn-47/obj6 2023-11-02T18:15:24.869 DEBUG:boto:Data: 2023-11-02T18:15:24.869 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:15:24.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:24.870 DEBUG:boto:Port: 8001 2023-11-02T18:15:24.871 DEBUG:boto:Params: {} 2023-11-02T18:15:24.871 DEBUG:boto:Token: None 2023-11-02T18:15:24.872 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-47/obj6 2023-11-02T18:15:24.872 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NJfADx6o6eV4iNlE2IWFZ6D3JKI= 2023-11-02T18:15:24.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NJfADx6o6eV4iNlE2IWFZ6D3JKI='} 2023-11-02T18:15:24.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2bf618d1a4124e3-006543e73c-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.919 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:24.920 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:24.920 DEBUG:boto:Method: HEAD 2023-11-02T18:15:24.920 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:15:24.921 DEBUG:boto:Data: 2023-11-02T18:15:24.921 DEBUG:boto:Headers: {} 2023-11-02T18:15:24.922 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:24.922 DEBUG:boto:Port: 8001 2023-11-02T18:15:24.922 DEBUG:boto:Params: {} 2023-11-02T18:15:24.923 DEBUG:boto:Token: None 2023-11-02T18:15:24.923 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-47/ 2023-11-02T18:15:24.924 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VSs9kI/GsJemQePyjpl9ZWwcQAk= 2023-11-02T18:15:24.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VSs9kI/GsJemQePyjpl9ZWwcQAk=', 'Content-Length': '0'} 2023-11-02T18:15:24.928 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000278d440c38a87b55-006543e73c-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.929 DEBUG:boto:path=/ibtljn-47/obj7 2023-11-02T18:15:24.930 DEBUG:boto:auth_path=/ibtljn-47/obj7 2023-11-02T18:15:24.930 DEBUG:boto:Method: PUT 2023-11-02T18:15:24.931 DEBUG:boto:Path: /ibtljn-47/obj7 2023-11-02T18:15:24.933 DEBUG:boto:Data: 2023-11-02T18:15:24.933 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:15:24.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:24.934 DEBUG:boto:Port: 8001 2023-11-02T18:15:24.935 DEBUG:boto:Params: {} 2023-11-02T18:15:24.936 DEBUG:boto:Token: None 2023-11-02T18:15:24.936 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-47/obj7 2023-11-02T18:15:24.937 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iSmQmfaD1xyB3bKgh0kITdnVomc= 2023-11-02T18:15:24.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iSmQmfaD1xyB3bKgh0kITdnVomc='} 2023-11-02T18:15:24.949 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008af28fe449c19df8-006543e73c-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.950 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:15:24.950 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:15:24.951 DEBUG:boto:Method: HEAD 2023-11-02T18:15:24.951 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:15:24.952 DEBUG:boto:Data: 2023-11-02T18:15:24.952 DEBUG:boto:Headers: {} 2023-11-02T18:15:24.952 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:24.953 DEBUG:boto:Port: 8001 2023-11-02T18:15:24.953 DEBUG:boto:Params: {} 2023-11-02T18:15:24.954 DEBUG:boto:Token: None 2023-11-02T18:15:24.954 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-47/ 2023-11-02T18:15:24.955 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VSs9kI/GsJemQePyjpl9ZWwcQAk= 2023-11-02T18:15:24.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VSs9kI/GsJemQePyjpl9ZWwcQAk=', 'Content-Length': '0'} 2023-11-02T18:15:24.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed6321c658f41c83-006543e73c-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:24.959 DEBUG:boto:path=/ibtljn-47/obj8 2023-11-02T18:15:24.960 DEBUG:boto:auth_path=/ibtljn-47/obj8 2023-11-02T18:15:24.960 DEBUG:boto:Method: PUT 2023-11-02T18:15:24.960 DEBUG:boto:Path: /ibtljn-47/obj8 2023-11-02T18:15:24.961 DEBUG:boto:Data: 2023-11-02T18:15:24.961 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:15:24.962 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:15:24.962 DEBUG:boto:Port: 8001 2023-11-02T18:15:24.962 DEBUG:boto:Params: {} 2023-11-02T18:15:24.963 DEBUG:boto:Token: None 2023-11-02T18:15:24.963 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:15:24 GMT /ibtljn-47/obj8 2023-11-02T18:15:24.964 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iF66vg5X4uXPOdhxtrBlU4zZ/0k= 2023-11-02T18:15:24.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:15:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iF66vg5X4uXPOdhxtrBlU4zZ/0k='} 2023-11-02T18:15:25.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa3283f50c233f96-006543e73c-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:15:25.007 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-46 2023-11-02T18:15:25.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:15:25.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:15:25.008 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:15:25.062 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:15:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:15:25.135 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:25.125+0000 7f41116bbf40 1 waiting for incremental sync to catch up: 2023-11-02T18:15:25.135 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.26.6, , 00000000001.14.6, , 00000000001.19.6, , , 00000000001.100.6, , , ] 2023-11-02T18:15:25.136 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.34.6, 00000000002.25.1, 00000000004.18.6, 00000000002.21.1, 00000000004.23.6, 00000000002.104.1, 00000000002.105.1, 00000000004.108.6, 00000000002.21.1, 00000000002.23.1, 00000000002.48.1] 2023-11-02T18:15:30.137 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:30.125+0000 7f41116bbf40 1 waiting for incremental sync to catch up: 2023-11-02T18:15:30.138 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.26.6, , 00000000001.14.6, , 00000000001.19.6, , , 00000000001.100.6, , , ] 2023-11-02T18:15:30.138 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.34.6, 00000000002.25.1, 00000000004.18.6, 00000000002.21.1, 00000000004.23.6, 00000000002.104.1, 00000000002.105.1, 00000000004.108.6, 00000000002.21.1, 00000000002.23.1, 00000000002.48.1] 2023-11-02T18:15:35.139 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:35.129+0000 7f41116bbf40 1 waiting for incremental sync to catch up: 2023-11-02T18:15:35.140 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.26.6, , 00000000001.14.6, , 00000000001.19.6, , , 00000000001.100.6, , , ] 2023-11-02T18:15:35.140 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.34.6, 00000000002.25.1, 00000000004.18.6, 00000000002.21.1, 00000000004.23.6, 00000000002.104.1, 00000000002.105.1, 00000000004.108.6, 00000000002.21.1, 00000000002.23.1, 00000000002.48.1] 2023-11-02T18:15:40.142 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:40.133+0000 7f41116bbf40 1 waiting for incremental sync to catch up: 2023-11-02T18:15:40.142 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.26.6, , 00000000001.14.6, , 00000000001.19.6, , , 00000000001.100.6, , , ] 2023-11-02T18:15:40.143 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.34.6, 00000000002.25.1, 00000000004.18.6, 00000000002.21.1, 00000000004.23.6, 00000000002.104.1, 00000000002.105.1, 00000000004.108.6, 00000000002.21.1, 00000000002.23.1, 00000000002.48.1] 2023-11-02T18:15:45.144 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:45.133+0000 7f41116bbf40 1 bucket sync caught up with source: 2023-11-02T18:15:45.145 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000004.34.6, 00000000002.25.1, 00000000004.18.6, 00000000002.21.1, 00000000004.23.6, 00000000002.104.1, 00000000002.105.1, 00000000004.108.6, 00000000002.21.1, 00000000002.23.1, 00000000002.48.1] 2023-11-02T18:15:45.145 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.34.6, 00000000002.25.1, 00000000004.18.6, 00000000002.21.1, 00000000004.23.6, 00000000002.104.1, 00000000002.105.1, 00000000004.108.6, 00000000002.21.1, 00000000002.23.1, 00000000002.48.1] 2023-11-02T18:15:45.145 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:45.133+0000 7f41116bbf40 0 bucket checkpoint complete 2023-11-02T18:15:45.175 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-46 2023-11-02T18:15:45.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:15:45.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:15:45.176 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:15:45.225 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:15:45.225 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:15:45.293 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:45.285+0000 7f1a2dfd2f40 1 waiting for incremental sync to catch up: 2023-11-02T18:15:45.294 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.191.6, , 00000000001.84.6, , 00000000001.296.6, , , 00000000001.246.6, , , ] 2023-11-02T18:15:45.294 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.199.6, , 00000000002.88.6, , 00000000002.300.6, , , 00000000002.254.6, , , ] 2023-11-02T18:15:50.296 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:50.285+0000 7f1a2dfd2f40 1 waiting for incremental sync to catch up: 2023-11-02T18:15:50.296 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.191.6, , 00000000001.84.6, , 00000000001.296.6, , , 00000000001.246.6, , , ] 2023-11-02T18:15:50.297 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.199.6, , 00000000002.88.6, , 00000000002.300.6, , , 00000000002.254.6, , , ] 2023-11-02T18:15:55.298 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:15:55.289+0000 7f1a2dfd2f40 1 waiting for incremental sync to catch up: 2023-11-02T18:15:55.299 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.191.6, , 00000000002.88.6, , 00000000001.296.6, , , 00000000002.254.6, , , ] 2023-11-02T18:15:55.299 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.199.6, , 00000000002.88.6, , 00000000002.300.6, , , 00000000002.254.6, , , ] 2023-11-02T18:16:00.301 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:00.289+0000 7f1a2dfd2f40 1 waiting for incremental sync to catch up: 2023-11-02T18:16:00.301 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.191.6, , 00000000002.88.6, , 00000000001.296.6, , , 00000000002.254.6, , , ] 2023-11-02T18:16:00.301 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.199.6, , 00000000002.88.6, , 00000000002.300.6, , , 00000000002.254.6, , , ] 2023-11-02T18:16:05.303 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:05.293+0000 7f1a2dfd2f40 1 bucket sync caught up with source: 2023-11-02T18:16:05.304 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.199.6, , 00000000002.88.6, , 00000000002.300.6, , , 00000000002.254.6, , , ] 2023-11-02T18:16:05.304 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.199.6, , 00000000002.88.6, , 00000000002.300.6, , , 00000000002.254.6, , , ] 2023-11-02T18:16:05.304 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:05.293+0000 7f1a2dfd2f40 0 bucket checkpoint complete 2023-11-02T18:16:05.333 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-46 zones={a2, a1} 2023-11-02T18:16:05.333 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.334 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.334 DEBUG:boto:Method: HEAD 2023-11-02T18:16:05.335 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:16:05.335 DEBUG:boto:Data: 2023-11-02T18:16:05.336 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.336 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.337 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.337 DEBUG:boto:Params: {} 2023-11-02T18:16:05.338 DEBUG:boto:Token: None 2023-11-02T18:16:05.338 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/ 2023-11-02T18:16:05.339 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EJplaEVVzxpsehaLpbm+7CCYrPI= 2023-11-02T18:16:05.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EJplaEVVzxpsehaLpbm+7CCYrPI=', 'Content-Length': '0'} 2023-11-02T18:16:05.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbd2d60fae7ede79-006543e765-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.344 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.345 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.345 DEBUG:boto:Method: HEAD 2023-11-02T18:16:05.346 DEBUG:boto:Path: /ibtljn-46/ 2023-11-02T18:16:05.346 DEBUG:boto:Data: 2023-11-02T18:16:05.347 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.347 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.348 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.348 DEBUG:boto:Params: {} 2023-11-02T18:16:05.348 DEBUG:boto:Token: None 2023-11-02T18:16:05.349 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/ 2023-11-02T18:16:05.350 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EJplaEVVzxpsehaLpbm+7CCYrPI= 2023-11-02T18:16:05.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EJplaEVVzxpsehaLpbm+7CCYrPI=', 'Content-Length': '0'} 2023-11-02T18:16:05.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006add22000c5389db-006543e765-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.355 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:16:05.355 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.355 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.356 DEBUG:boto:path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:16:05.356 DEBUG:boto:auth_path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:16:05.357 DEBUG:boto:Method: GET 2023-11-02T18:16:05.357 DEBUG:boto:Path: /ibtljn-46/?versions&max-keys=999 2023-11-02T18:16:05.358 DEBUG:boto:Data: 2023-11-02T18:16:05.358 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.359 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.359 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.360 DEBUG:boto:Params: {} 2023-11-02T18:16:05.360 DEBUG:boto:Token: None 2023-11-02T18:16:05.361 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/?versions 2023-11-02T18:16:05.361 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pU9Twr4ztNRfSfMO57nPh1f0YyY= 2023-11-02T18:16:05.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pU9Twr4ztNRfSfMO57nPh1f0YyY=', 'Content-Length': '0'} 2023-11-02T18:16:05.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d9a3a5d9a747031-006543e765-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.368 DEBUG:boto:b'ibtljn-46999falseobj1nulltrue2023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:15:24.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:15:24.686Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:15:24.750Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:15:24.778Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:05.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:16:05.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:16:05.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:16:05.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:16:05.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-11-02T18:16:05.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-11-02T18:16:05.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-11-02T18:16:05.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-11-02T18:16:05.373 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:16:05.373 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.374 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.374 DEBUG:boto:path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:16:05.374 DEBUG:boto:auth_path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:16:05.375 DEBUG:boto:Method: GET 2023-11-02T18:16:05.375 DEBUG:boto:Path: /ibtljn-46/?versions&max-keys=999 2023-11-02T18:16:05.376 DEBUG:boto:Data: 2023-11-02T18:16:05.376 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.377 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.377 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.378 DEBUG:boto:Params: {} 2023-11-02T18:16:05.378 DEBUG:boto:Token: None 2023-11-02T18:16:05.379 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/?versions 2023-11-02T18:16:05.379 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pU9Twr4ztNRfSfMO57nPh1f0YyY= 2023-11-02T18:16:05.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pU9Twr4ztNRfSfMO57nPh1f0YyY=', 'Content-Length': '0'} 2023-11-02T18:16:05.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000000dd7c01967df34-006543e765-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.386 DEBUG:boto:b'ibtljn-46999falseobj1nulltrue2023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:15:24.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:15:24.686Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:15:24.750Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:15:24.778Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:05.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:16:05.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:16:05.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:16:05.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:16:05.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-11-02T18:16:05.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-11-02T18:16:05.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-11-02T18:16:05.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-11-02T18:16:05.390 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.391 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.391 DEBUG:boto:path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:16:05.392 DEBUG:boto:auth_path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:16:05.392 DEBUG:boto:Method: GET 2023-11-02T18:16:05.392 DEBUG:boto:Path: /ibtljn-46/?versions&max-keys=999 2023-11-02T18:16:05.393 DEBUG:boto:Data: 2023-11-02T18:16:05.393 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.394 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.394 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.395 DEBUG:boto:Params: {} 2023-11-02T18:16:05.395 DEBUG:boto:Token: None 2023-11-02T18:16:05.396 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/?versions 2023-11-02T18:16:05.396 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pU9Twr4ztNRfSfMO57nPh1f0YyY= 2023-11-02T18:16:05.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pU9Twr4ztNRfSfMO57nPh1f0YyY=', 'Content-Length': '0'} 2023-11-02T18:16:05.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc36d01734579faa-006543e765-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.402 DEBUG:boto:b'ibtljn-46999falseobj1nulltrue2023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:15:24.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:15:24.686Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:15:24.750Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:15:24.778Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:05.403 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.404 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.404 DEBUG:boto:path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:16:05.405 DEBUG:boto:auth_path=/ibtljn-46/?versions&max-keys=999 2023-11-02T18:16:05.405 DEBUG:boto:Method: GET 2023-11-02T18:16:05.405 DEBUG:boto:Path: /ibtljn-46/?versions&max-keys=999 2023-11-02T18:16:05.406 DEBUG:boto:Data: 2023-11-02T18:16:05.406 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.407 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.407 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.408 DEBUG:boto:Params: {} 2023-11-02T18:16:05.408 DEBUG:boto:Token: None 2023-11-02T18:16:05.409 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/?versions 2023-11-02T18:16:05.409 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pU9Twr4ztNRfSfMO57nPh1f0YyY= 2023-11-02T18:16:05.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pU9Twr4ztNRfSfMO57nPh1f0YyY=', 'Content-Length': '0'} 2023-11-02T18:16:05.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d37108cbf646661-006543e765-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.417 DEBUG:boto:b'ibtljn-46999falseobj1nulltrue2023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:15:24.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:15:24.686Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:15:24.750Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:15:24.778Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:05.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:16:05.418 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:16:05.419 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:16:05.419 DEBUG:boto:path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.420 DEBUG:boto:auth_path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.420 DEBUG:boto:Method: GET 2023-11-02T18:16:05.420 DEBUG:boto:Path: /ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.421 DEBUG:boto:Data: 2023-11-02T18:16:05.421 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.422 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.423 DEBUG:boto:Params: {} 2023-11-02T18:16:05.423 DEBUG:boto:Token: None 2023-11-02T18:16:05.424 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CPS0DuOax986NYrWj/8OblRnajI= 2023-11-02T18:16:05.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CPS0DuOax986NYrWj/8OblRnajI=', 'Content-Length': '0'} 2023-11-02T18:16:05.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019cabad13233912a-006543e765-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.470 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:16:05.471 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:16:05.471 DEBUG:boto:path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.472 DEBUG:boto:auth_path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.472 DEBUG:boto:Method: GET 2023-11-02T18:16:05.473 DEBUG:boto:Path: /ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.473 DEBUG:boto:Data: 2023-11-02T18:16:05.473 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.474 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.474 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.475 DEBUG:boto:Params: {} 2023-11-02T18:16:05.475 DEBUG:boto:Token: None 2023-11-02T18:16:05.476 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.476 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CPS0DuOax986NYrWj/8OblRnajI= 2023-11-02T18:16:05.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CPS0DuOax986NYrWj/8OblRnajI=', 'Content-Length': '0'} 2023-11-02T18:16:05.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fecce90f82822479-006543e765-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.481 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:16:05.481 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:16:05.481 DEBUG:boto:path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.482 DEBUG:boto:auth_path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.482 DEBUG:boto:Method: HEAD 2023-11-02T18:16:05.483 DEBUG:boto:Path: /ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.483 DEBUG:boto:Data: 2023-11-02T18:16:05.484 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.484 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.485 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.485 DEBUG:boto:Params: {} 2023-11-02T18:16:05.486 DEBUG:boto:Token: None 2023-11-02T18:16:05.486 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.487 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W95958zQHtj4/cJIttXQtLkwExo= 2023-11-02T18:16:05.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W95958zQHtj4/cJIttXQtLkwExo=', 'Content-Length': '0'} 2023-11-02T18:16:05.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d63f1807378c34eb-006543e765-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.490 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:16:05.491 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:16:05.492 DEBUG:boto:path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.492 DEBUG:boto:auth_path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.493 DEBUG:boto:Method: HEAD 2023-11-02T18:16:05.493 DEBUG:boto:Path: /ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.493 DEBUG:boto:Data: 2023-11-02T18:16:05.494 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.494 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.495 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.495 DEBUG:boto:Params: {} 2023-11-02T18:16:05.496 DEBUG:boto:Token: None 2023-11-02T18:16:05.496 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W95958zQHtj4/cJIttXQtLkwExo= 2023-11-02T18:16:05.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W95958zQHtj4/cJIttXQtLkwExo=', 'Content-Length': '0'} 2023-11-02T18:16:05.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af76047f1c6d0ae0-006543e765-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:16:05.501 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:16:05.501 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:16:05.502 DEBUG:boto:path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.502 DEBUG:boto:auth_path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.503 DEBUG:boto:Method: GET 2023-11-02T18:16:05.503 DEBUG:boto:Path: /ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.504 DEBUG:boto:Data: 2023-11-02T18:16:05.504 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.505 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.505 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.506 DEBUG:boto:Params: {} 2023-11-02T18:16:05.506 DEBUG:boto:Token: None 2023-11-02T18:16:05.507 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.507 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CPS0DuOax986NYrWj/8OblRnajI= 2023-11-02T18:16:05.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CPS0DuOax986NYrWj/8OblRnajI=', 'Content-Length': '0'} 2023-11-02T18:16:05.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e43bdf6b9ca653d1-006543e765-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.511 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:16:05.512 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:16:05.512 DEBUG:boto:path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.512 DEBUG:boto:auth_path=/ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.513 DEBUG:boto:Method: GET 2023-11-02T18:16:05.513 DEBUG:boto:Path: /ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.514 DEBUG:boto:Data: 2023-11-02T18:16:05.514 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.515 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.515 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.516 DEBUG:boto:Params: {} 2023-11-02T18:16:05.516 DEBUG:boto:Token: None 2023-11-02T18:16:05.517 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj1?versionId=null 2023-11-02T18:16:05.517 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CPS0DuOax986NYrWj/8OblRnajI= 2023-11-02T18:16:05.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CPS0DuOax986NYrWj/8OblRnajI=', 'Content-Length': '0'} 2023-11-02T18:16:05.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000718408ac609c840b-006543e765-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.562 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.563 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.563 DEBUG:boto:path=/ibtljn-46/?prefix=obj1 2023-11-02T18:16:05.564 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj1 2023-11-02T18:16:05.564 DEBUG:boto:Method: GET 2023-11-02T18:16:05.565 DEBUG:boto:Path: /ibtljn-46/?prefix=obj1 2023-11-02T18:16:05.565 DEBUG:boto:Data: 2023-11-02T18:16:05.566 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.567 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.567 DEBUG:boto:Params: {} 2023-11-02T18:16:05.568 DEBUG:boto:Token: None 2023-11-02T18:16:05.568 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/ 2023-11-02T18:16:05.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc= 2023-11-02T18:16:05.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc=', 'Content-Length': '0'} 2023-11-02T18:16:05.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a15d676170eac1a-006543e765-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.574 DEBUG:boto:b'ibtljn-46obj11000falseobj12023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:05.575 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.575 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.576 DEBUG:boto:path=/ibtljn-46/?prefix=obj1 2023-11-02T18:16:05.576 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj1 2023-11-02T18:16:05.577 DEBUG:boto:Method: GET 2023-11-02T18:16:05.577 DEBUG:boto:Path: /ibtljn-46/?prefix=obj1 2023-11-02T18:16:05.578 DEBUG:boto:Data: 2023-11-02T18:16:05.578 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.579 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.579 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.579 DEBUG:boto:Params: {} 2023-11-02T18:16:05.580 DEBUG:boto:Token: None 2023-11-02T18:16:05.580 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/ 2023-11-02T18:16:05.581 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc= 2023-11-02T18:16:05.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc=', 'Content-Length': '0'} 2023-11-02T18:16:05.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4d96e410a48ef13-006543e765-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.586 DEBUG:boto:b'ibtljn-46obj11000falseobj12023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:05.587 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:16:05.588 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:16:05.589 DEBUG:boto:Method: HEAD 2023-11-02T18:16:05.590 DEBUG:boto:Path: /ibtljn-46/obj1 2023-11-02T18:16:05.590 DEBUG:boto:Data: 2023-11-02T18:16:05.590 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.592 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.592 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.592 DEBUG:boto:Params: {} 2023-11-02T18:16:05.593 DEBUG:boto:Token: None 2023-11-02T18:16:05.593 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj1 2023-11-02T18:16:05.594 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S9f+egX34RaxGadcKNpvywb7pCw= 2023-11-02T18:16:05.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S9f+egX34RaxGadcKNpvywb7pCw=', 'Content-Length': '0'} 2023-11-02T18:16:05.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e017b13ee68fef8f-006543e765-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.598 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:16:05.598 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:16:05.599 DEBUG:boto:Method: HEAD 2023-11-02T18:16:05.599 DEBUG:boto:Path: /ibtljn-46/obj1 2023-11-02T18:16:05.600 DEBUG:boto:Data: 2023-11-02T18:16:05.600 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.601 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.601 DEBUG:boto:Params: {} 2023-11-02T18:16:05.602 DEBUG:boto:Token: None 2023-11-02T18:16:05.602 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj1 2023-11-02T18:16:05.602 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S9f+egX34RaxGadcKNpvywb7pCw= 2023-11-02T18:16:05.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S9f+egX34RaxGadcKNpvywb7pCw=', 'Content-Length': '0'} 2023-11-02T18:16:05.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc1cb71492007fcf-006543e765-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:16:05.607 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:16:05.607 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:16:05.608 DEBUG:boto:Method: GET 2023-11-02T18:16:05.608 DEBUG:boto:Path: /ibtljn-46/obj1 2023-11-02T18:16:05.608 DEBUG:boto:Data: 2023-11-02T18:16:05.609 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.609 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.610 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.610 DEBUG:boto:Params: {} 2023-11-02T18:16:05.611 DEBUG:boto:Token: None 2023-11-02T18:16:05.611 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj1 2023-11-02T18:16:05.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i5PEorBCf73IhbgIOSABUNXXWtU= 2023-11-02T18:16:05.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i5PEorBCf73IhbgIOSABUNXXWtU=', 'Content-Length': '0'} 2023-11-02T18:16:05.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000486254e3bdde451a-006543e765-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.616 DEBUG:boto:path=/ibtljn-46/obj1 2023-11-02T18:16:05.616 DEBUG:boto:auth_path=/ibtljn-46/obj1 2023-11-02T18:16:05.617 DEBUG:boto:Method: GET 2023-11-02T18:16:05.617 DEBUG:boto:Path: /ibtljn-46/obj1 2023-11-02T18:16:05.618 DEBUG:boto:Data: 2023-11-02T18:16:05.618 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.619 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.620 DEBUG:boto:Params: {} 2023-11-02T18:16:05.620 DEBUG:boto:Token: None 2023-11-02T18:16:05.621 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj1 2023-11-02T18:16:05.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i5PEorBCf73IhbgIOSABUNXXWtU= 2023-11-02T18:16:05.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i5PEorBCf73IhbgIOSABUNXXWtU=', 'Content-Length': '0'} 2023-11-02T18:16:05.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb6c67a62663fda9-006543e765-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.625 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.626 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.626 DEBUG:boto:path=/ibtljn-46/?prefix=obj1 2023-11-02T18:16:05.627 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj1 2023-11-02T18:16:05.627 DEBUG:boto:Method: GET 2023-11-02T18:16:05.627 DEBUG:boto:Path: /ibtljn-46/?prefix=obj1 2023-11-02T18:16:05.628 DEBUG:boto:Data: 2023-11-02T18:16:05.628 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.629 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.629 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.630 DEBUG:boto:Params: {} 2023-11-02T18:16:05.630 DEBUG:boto:Token: None 2023-11-02T18:16:05.631 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/ 2023-11-02T18:16:05.631 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc= 2023-11-02T18:16:05.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc=', 'Content-Length': '0'} 2023-11-02T18:16:05.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000732f793b96c43c17-006543e765-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.679 DEBUG:boto:b'ibtljn-46obj11000falseobj12023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:05.680 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.681 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.681 DEBUG:boto:path=/ibtljn-46/?prefix=obj1 2023-11-02T18:16:05.682 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj1 2023-11-02T18:16:05.682 DEBUG:boto:Method: GET 2023-11-02T18:16:05.683 DEBUG:boto:Path: /ibtljn-46/?prefix=obj1 2023-11-02T18:16:05.683 DEBUG:boto:Data: 2023-11-02T18:16:05.683 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.684 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.684 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.685 DEBUG:boto:Params: {} 2023-11-02T18:16:05.685 DEBUG:boto:Token: None 2023-11-02T18:16:05.686 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/ 2023-11-02T18:16:05.686 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc= 2023-11-02T18:16:05.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc=', 'Content-Length': '0'} 2023-11-02T18:16:05.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ea2b1fd7c1e89a0-006543e765-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.692 DEBUG:boto:b'ibtljn-46obj11000falseobj12023-11-02T18:13:45.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:05.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:16:05.694 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:16:05.694 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:16:05.694 DEBUG:boto:path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.695 DEBUG:boto:auth_path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.695 DEBUG:boto:Method: GET 2023-11-02T18:16:05.696 DEBUG:boto:Path: /ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.696 DEBUG:boto:Data: 2023-11-02T18:16:05.697 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.697 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.698 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.698 DEBUG:boto:Params: {} 2023-11-02T18:16:05.699 DEBUG:boto:Token: None 2023-11-02T18:16:05.699 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.700 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vr0Uy1Tvc9w5wO7y9PnduATzuH4= 2023-11-02T18:16:05.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vr0Uy1Tvc9w5wO7y9PnduATzuH4=', 'Content-Length': '0'} 2023-11-02T18:16:05.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbfd79852c77912c-006543e765-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.704 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:16:05.704 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:16:05.705 DEBUG:boto:path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.705 DEBUG:boto:auth_path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.706 DEBUG:boto:Method: GET 2023-11-02T18:16:05.706 DEBUG:boto:Path: /ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.707 DEBUG:boto:Data: 2023-11-02T18:16:05.707 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.707 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.708 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.708 DEBUG:boto:Params: {} 2023-11-02T18:16:05.709 DEBUG:boto:Token: None 2023-11-02T18:16:05.709 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.710 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vr0Uy1Tvc9w5wO7y9PnduATzuH4= 2023-11-02T18:16:05.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vr0Uy1Tvc9w5wO7y9PnduATzuH4=', 'Content-Length': '0'} 2023-11-02T18:16:05.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000714f9768fb7e3bd2-006543e765-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.714 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:16:05.714 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:16:05.715 DEBUG:boto:path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.715 DEBUG:boto:auth_path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.716 DEBUG:boto:Method: HEAD 2023-11-02T18:16:05.716 DEBUG:boto:Path: /ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.716 DEBUG:boto:Data: 2023-11-02T18:16:05.717 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.717 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.718 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.718 DEBUG:boto:Params: {} 2023-11-02T18:16:05.719 DEBUG:boto:Token: None 2023-11-02T18:16:05.719 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.720 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1+kXOqhyQHW357mL1LLrkCC1FJ0= 2023-11-02T18:16:05.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1+kXOqhyQHW357mL1LLrkCC1FJ0=', 'Content-Length': '0'} 2023-11-02T18:16:05.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6e5e9978501dbae-006543e765-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.723 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:16:05.724 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:16:05.724 DEBUG:boto:path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.724 DEBUG:boto:auth_path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.725 DEBUG:boto:Method: HEAD 2023-11-02T18:16:05.725 DEBUG:boto:Path: /ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.726 DEBUG:boto:Data: 2023-11-02T18:16:05.726 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.727 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.727 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.728 DEBUG:boto:Params: {} 2023-11-02T18:16:05.728 DEBUG:boto:Token: None 2023-11-02T18:16:05.729 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.729 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1+kXOqhyQHW357mL1LLrkCC1FJ0= 2023-11-02T18:16:05.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1+kXOqhyQHW357mL1LLrkCC1FJ0=', 'Content-Length': '0'} 2023-11-02T18:16:05.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005224cf6614c68056-006543e765-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:16:05.734 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:16:05.734 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:16:05.735 DEBUG:boto:path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.735 DEBUG:boto:auth_path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.735 DEBUG:boto:Method: GET 2023-11-02T18:16:05.736 DEBUG:boto:Path: /ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.736 DEBUG:boto:Data: 2023-11-02T18:16:05.737 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.737 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.738 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.738 DEBUG:boto:Params: {} 2023-11-02T18:16:05.739 DEBUG:boto:Token: None 2023-11-02T18:16:05.739 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vr0Uy1Tvc9w5wO7y9PnduATzuH4= 2023-11-02T18:16:05.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vr0Uy1Tvc9w5wO7y9PnduATzuH4=', 'Content-Length': '0'} 2023-11-02T18:16:05.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb7d9bcb790bff6c-006543e765-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.786 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:16:05.787 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:16:05.787 DEBUG:boto:path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.788 DEBUG:boto:auth_path=/ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.788 DEBUG:boto:Method: GET 2023-11-02T18:16:05.789 DEBUG:boto:Path: /ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.789 DEBUG:boto:Data: 2023-11-02T18:16:05.790 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.790 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.791 DEBUG:boto:Params: {} 2023-11-02T18:16:05.791 DEBUG:boto:Token: None 2023-11-02T18:16:05.792 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj2?versionId=null 2023-11-02T18:16:05.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vr0Uy1Tvc9w5wO7y9PnduATzuH4= 2023-11-02T18:16:05.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vr0Uy1Tvc9w5wO7y9PnduATzuH4=', 'Content-Length': '0'} 2023-11-02T18:16:05.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e5b0b80bd367234-006543e765-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.797 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.797 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.797 DEBUG:boto:path=/ibtljn-46/?prefix=obj2 2023-11-02T18:16:05.798 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj2 2023-11-02T18:16:05.798 DEBUG:boto:Method: GET 2023-11-02T18:16:05.799 DEBUG:boto:Path: /ibtljn-46/?prefix=obj2 2023-11-02T18:16:05.799 DEBUG:boto:Data: 2023-11-02T18:16:05.800 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.800 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.801 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.801 DEBUG:boto:Params: {} 2023-11-02T18:16:05.802 DEBUG:boto:Token: None 2023-11-02T18:16:05.802 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/ 2023-11-02T18:16:05.803 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc= 2023-11-02T18:16:05.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc=', 'Content-Length': '0'} 2023-11-02T18:16:05.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9505e58318660f3-006543e765-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.810 DEBUG:boto:b'ibtljn-46obj21000falseobj22023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:05.811 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.811 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.812 DEBUG:boto:path=/ibtljn-46/?prefix=obj2 2023-11-02T18:16:05.812 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj2 2023-11-02T18:16:05.813 DEBUG:boto:Method: GET 2023-11-02T18:16:05.813 DEBUG:boto:Path: /ibtljn-46/?prefix=obj2 2023-11-02T18:16:05.814 DEBUG:boto:Data: 2023-11-02T18:16:05.814 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.815 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.815 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.815 DEBUG:boto:Params: {} 2023-11-02T18:16:05.816 DEBUG:boto:Token: None 2023-11-02T18:16:05.816 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/ 2023-11-02T18:16:05.817 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc= 2023-11-02T18:16:05.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc=', 'Content-Length': '0'} 2023-11-02T18:16:05.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000761a3a1e8f5429b3-006543e765-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.866 DEBUG:boto:b'ibtljn-46obj21000falseobj22023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:05.867 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:16:05.868 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:16:05.868 DEBUG:boto:Method: HEAD 2023-11-02T18:16:05.868 DEBUG:boto:Path: /ibtljn-46/obj2 2023-11-02T18:16:05.869 DEBUG:boto:Data: 2023-11-02T18:16:05.869 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.870 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.871 DEBUG:boto:Params: {} 2023-11-02T18:16:05.871 DEBUG:boto:Token: None 2023-11-02T18:16:05.871 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj2 2023-11-02T18:16:05.872 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a3EpMUVHk1aeuFH/Ecv72eTJh58= 2023-11-02T18:16:05.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a3EpMUVHk1aeuFH/Ecv72eTJh58=', 'Content-Length': '0'} 2023-11-02T18:16:05.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009be1998a00ca6b53-006543e765-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.877 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:16:05.877 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:16:05.878 DEBUG:boto:Method: HEAD 2023-11-02T18:16:05.878 DEBUG:boto:Path: /ibtljn-46/obj2 2023-11-02T18:16:05.878 DEBUG:boto:Data: 2023-11-02T18:16:05.879 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.879 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.880 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.880 DEBUG:boto:Params: {} 2023-11-02T18:16:05.881 DEBUG:boto:Token: None 2023-11-02T18:16:05.881 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj2 2023-11-02T18:16:05.882 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a3EpMUVHk1aeuFH/Ecv72eTJh58= 2023-11-02T18:16:05.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a3EpMUVHk1aeuFH/Ecv72eTJh58=', 'Content-Length': '0'} 2023-11-02T18:16:05.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dbe3f9e538edc40-006543e765-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:16:05.886 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:16:05.887 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:16:05.887 DEBUG:boto:Method: GET 2023-11-02T18:16:05.888 DEBUG:boto:Path: /ibtljn-46/obj2 2023-11-02T18:16:05.888 DEBUG:boto:Data: 2023-11-02T18:16:05.889 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.889 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.890 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.890 DEBUG:boto:Params: {} 2023-11-02T18:16:05.891 DEBUG:boto:Token: None 2023-11-02T18:16:05.892 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj2 2023-11-02T18:16:05.892 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LTGlf+NSmsI5HL4P06jE34OsglY= 2023-11-02T18:16:05.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LTGlf+NSmsI5HL4P06jE34OsglY=', 'Content-Length': '0'} 2023-11-02T18:16:05.896 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e685c825182381f-006543e765-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.896 DEBUG:boto:path=/ibtljn-46/obj2 2023-11-02T18:16:05.897 DEBUG:boto:auth_path=/ibtljn-46/obj2 2023-11-02T18:16:05.897 DEBUG:boto:Method: GET 2023-11-02T18:16:05.898 DEBUG:boto:Path: /ibtljn-46/obj2 2023-11-02T18:16:05.898 DEBUG:boto:Data: 2023-11-02T18:16:05.899 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.899 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.900 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.900 DEBUG:boto:Params: {} 2023-11-02T18:16:05.900 DEBUG:boto:Token: None 2023-11-02T18:16:05.901 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj2 2023-11-02T18:16:05.901 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LTGlf+NSmsI5HL4P06jE34OsglY= 2023-11-02T18:16:05.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LTGlf+NSmsI5HL4P06jE34OsglY=', 'Content-Length': '0'} 2023-11-02T18:16:05.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c4b4cd2d615e551-006543e765-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.906 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.906 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.907 DEBUG:boto:path=/ibtljn-46/?prefix=obj2 2023-11-02T18:16:05.907 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj2 2023-11-02T18:16:05.907 DEBUG:boto:Method: GET 2023-11-02T18:16:05.908 DEBUG:boto:Path: /ibtljn-46/?prefix=obj2 2023-11-02T18:16:05.908 DEBUG:boto:Data: 2023-11-02T18:16:05.909 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.909 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.910 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.910 DEBUG:boto:Params: {} 2023-11-02T18:16:05.911 DEBUG:boto:Token: None 2023-11-02T18:16:05.911 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/ 2023-11-02T18:16:05.912 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc= 2023-11-02T18:16:05.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc=', 'Content-Length': '0'} 2023-11-02T18:16:05.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066a8b6ee02625c57-006543e765-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.962 DEBUG:boto:b'ibtljn-46obj21000falseobj22023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:05.963 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:05.964 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:05.964 DEBUG:boto:path=/ibtljn-46/?prefix=obj2 2023-11-02T18:16:05.965 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj2 2023-11-02T18:16:05.965 DEBUG:boto:Method: GET 2023-11-02T18:16:05.966 DEBUG:boto:Path: /ibtljn-46/?prefix=obj2 2023-11-02T18:16:05.966 DEBUG:boto:Data: 2023-11-02T18:16:05.967 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.967 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.968 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.968 DEBUG:boto:Params: {} 2023-11-02T18:16:05.969 DEBUG:boto:Token: None 2023-11-02T18:16:05.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/ 2023-11-02T18:16:05.970 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc= 2023-11-02T18:16:05.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WNl2kwYhngr5ShRbAyBRlKMB4Wc=', 'Content-Length': '0'} 2023-11-02T18:16:05.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000679f3feef1c17a9e-006543e765-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.976 DEBUG:boto:b'ibtljn-46obj21000falseobj22023-11-02T18:13:45.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:05.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:16:05.977 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:16:05.977 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:16:05.978 DEBUG:boto:path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:16:05.978 DEBUG:boto:auth_path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:16:05.979 DEBUG:boto:Method: GET 2023-11-02T18:16:05.979 DEBUG:boto:Path: /ibtljn-46/obj3?versionId=null 2023-11-02T18:16:05.980 DEBUG:boto:Data: 2023-11-02T18:16:05.980 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.981 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:05.981 DEBUG:boto:Port: 8001 2023-11-02T18:16:05.982 DEBUG:boto:Params: {} 2023-11-02T18:16:05.982 DEBUG:boto:Token: None 2023-11-02T18:16:05.983 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj3?versionId=null 2023-11-02T18:16:05.984 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6LYCdCuc6VnGUw5CdgfUViLKUuA= 2023-11-02T18:16:05.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6LYCdCuc6VnGUw5CdgfUViLKUuA=', 'Content-Length': '0'} 2023-11-02T18:16:05.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db116c4bbd77a32a-006543e765-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:05.988 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:16:05.989 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:16:05.989 DEBUG:boto:path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:16:05.990 DEBUG:boto:auth_path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:16:05.990 DEBUG:boto:Method: GET 2023-11-02T18:16:05.991 DEBUG:boto:Path: /ibtljn-46/obj3?versionId=null 2023-11-02T18:16:05.991 DEBUG:boto:Data: 2023-11-02T18:16:05.992 DEBUG:boto:Headers: {} 2023-11-02T18:16:05.992 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:05.993 DEBUG:boto:Port: 8000 2023-11-02T18:16:05.993 DEBUG:boto:Params: {} 2023-11-02T18:16:05.994 DEBUG:boto:Token: None 2023-11-02T18:16:05.995 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:05 GMT /ibtljn-46/obj3?versionId=null 2023-11-02T18:16:05.997 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6LYCdCuc6VnGUw5CdgfUViLKUuA= 2023-11-02T18:16:05.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6LYCdCuc6VnGUw5CdgfUViLKUuA=', 'Content-Length': '0'} 2023-11-02T18:16:06.005 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003769dba9b0023947-006543e765-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.006 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:16:06.006 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:16:06.006 DEBUG:boto:path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.007 DEBUG:boto:auth_path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.007 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.008 DEBUG:boto:Path: /ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.008 DEBUG:boto:Data: 2023-11-02T18:16:06.009 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.009 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.010 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.010 DEBUG:boto:Params: {} 2023-11-02T18:16:06.010 DEBUG:boto:Token: None 2023-11-02T18:16:06.011 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.011 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jku4GWgKZcLCHtXUIWk4Cfmf9LA= 2023-11-02T18:16:06.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jku4GWgKZcLCHtXUIWk4Cfmf9LA=', 'Content-Length': '0'} 2023-11-02T18:16:06.015 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062aae6abf9725f2b-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.015 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:16:06.016 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:16:06.016 DEBUG:boto:path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.017 DEBUG:boto:auth_path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.017 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.017 DEBUG:boto:Path: /ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.018 DEBUG:boto:Data: 2023-11-02T18:16:06.018 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.019 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.019 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.020 DEBUG:boto:Params: {} 2023-11-02T18:16:06.020 DEBUG:boto:Token: None 2023-11-02T18:16:06.021 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.021 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jku4GWgKZcLCHtXUIWk4Cfmf9LA= 2023-11-02T18:16:06.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jku4GWgKZcLCHtXUIWk4Cfmf9LA=', 'Content-Length': '0'} 2023-11-02T18:16:06.024 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009dcbc8c9e4a9d5ca-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:16:06.025 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:16:06.026 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:16:06.026 DEBUG:boto:path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.027 DEBUG:boto:auth_path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.027 DEBUG:boto:Method: GET 2023-11-02T18:16:06.028 DEBUG:boto:Path: /ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.028 DEBUG:boto:Data: 2023-11-02T18:16:06.029 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.029 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.030 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.030 DEBUG:boto:Params: {} 2023-11-02T18:16:06.031 DEBUG:boto:Token: None 2023-11-02T18:16:06.031 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNcq8/tT3KW5Z5aLl2pvfkWdPAI= 2023-11-02T18:16:06.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNcq8/tT3KW5Z5aLl2pvfkWdPAI=', 'Content-Length': '0'} 2023-11-02T18:16:06.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024164d7fd81b4daa-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.082 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:16:06.083 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:16:06.083 DEBUG:boto:path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.084 DEBUG:boto:auth_path=/ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.084 DEBUG:boto:Method: GET 2023-11-02T18:16:06.085 DEBUG:boto:Path: /ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.085 DEBUG:boto:Data: 2023-11-02T18:16:06.086 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.086 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.086 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.087 DEBUG:boto:Params: {} 2023-11-02T18:16:06.087 DEBUG:boto:Token: None 2023-11-02T18:16:06.088 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj3?versionId=null 2023-11-02T18:16:06.089 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNcq8/tT3KW5Z5aLl2pvfkWdPAI= 2023-11-02T18:16:06.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNcq8/tT3KW5Z5aLl2pvfkWdPAI=', 'Content-Length': '0'} 2023-11-02T18:16:06.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bc288667209356f-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.093 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.093 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.094 DEBUG:boto:path=/ibtljn-46/?prefix=obj3 2023-11-02T18:16:06.094 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj3 2023-11-02T18:16:06.095 DEBUG:boto:Method: GET 2023-11-02T18:16:06.095 DEBUG:boto:Path: /ibtljn-46/?prefix=obj3 2023-11-02T18:16:06.095 DEBUG:boto:Data: 2023-11-02T18:16:06.096 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.097 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.097 DEBUG:boto:Params: {} 2023-11-02T18:16:06.098 DEBUG:boto:Token: None 2023-11-02T18:16:06.098 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.099 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9501063c98b584d-006543e766-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.106 DEBUG:boto:b'ibtljn-46obj31000falseobj32023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.107 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.107 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.108 DEBUG:boto:path=/ibtljn-46/?prefix=obj3 2023-11-02T18:16:06.108 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj3 2023-11-02T18:16:06.108 DEBUG:boto:Method: GET 2023-11-02T18:16:06.109 DEBUG:boto:Path: /ibtljn-46/?prefix=obj3 2023-11-02T18:16:06.109 DEBUG:boto:Data: 2023-11-02T18:16:06.110 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.110 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.111 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.111 DEBUG:boto:Params: {} 2023-11-02T18:16:06.112 DEBUG:boto:Token: None 2023-11-02T18:16:06.112 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.113 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ccd21f08baaa2ae-006543e766-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.166 DEBUG:boto:b'ibtljn-46obj31000falseobj32023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.168 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:16:06.168 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:16:06.169 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.169 DEBUG:boto:Path: /ibtljn-46/obj3 2023-11-02T18:16:06.170 DEBUG:boto:Data: 2023-11-02T18:16:06.170 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.171 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.171 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.172 DEBUG:boto:Params: {} 2023-11-02T18:16:06.172 DEBUG:boto:Token: None 2023-11-02T18:16:06.173 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj3 2023-11-02T18:16:06.173 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AugHgYmkzxBZnFOyhpOwvTw3eGE= 2023-11-02T18:16:06.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AugHgYmkzxBZnFOyhpOwvTw3eGE=', 'Content-Length': '0'} 2023-11-02T18:16:06.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ac41802f4f9cd27-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.177 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:16:06.177 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:16:06.178 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.179 DEBUG:boto:Path: /ibtljn-46/obj3 2023-11-02T18:16:06.179 DEBUG:boto:Data: 2023-11-02T18:16:06.180 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.180 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.180 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.181 DEBUG:boto:Params: {} 2023-11-02T18:16:06.181 DEBUG:boto:Token: None 2023-11-02T18:16:06.182 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj3 2023-11-02T18:16:06.182 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AugHgYmkzxBZnFOyhpOwvTw3eGE= 2023-11-02T18:16:06.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AugHgYmkzxBZnFOyhpOwvTw3eGE=', 'Content-Length': '0'} 2023-11-02T18:16:06.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000716f5b1c45d1a856-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:16:06.187 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:16:06.187 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:16:06.188 DEBUG:boto:Method: GET 2023-11-02T18:16:06.188 DEBUG:boto:Path: /ibtljn-46/obj3 2023-11-02T18:16:06.189 DEBUG:boto:Data: 2023-11-02T18:16:06.189 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.189 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.190 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.190 DEBUG:boto:Params: {} 2023-11-02T18:16:06.191 DEBUG:boto:Token: None 2023-11-02T18:16:06.191 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj3 2023-11-02T18:16:06.192 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:veMxFMe8tOE4sO8gSINP+hxf+qI= 2023-11-02T18:16:06.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:veMxFMe8tOE4sO8gSINP+hxf+qI=', 'Content-Length': '0'} 2023-11-02T18:16:06.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003316fb8a075d0b97-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.197 DEBUG:boto:path=/ibtljn-46/obj3 2023-11-02T18:16:06.197 DEBUG:boto:auth_path=/ibtljn-46/obj3 2023-11-02T18:16:06.199 DEBUG:boto:Method: GET 2023-11-02T18:16:06.199 DEBUG:boto:Path: /ibtljn-46/obj3 2023-11-02T18:16:06.200 DEBUG:boto:Data: 2023-11-02T18:16:06.200 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.201 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.201 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.201 DEBUG:boto:Params: {} 2023-11-02T18:16:06.202 DEBUG:boto:Token: None 2023-11-02T18:16:06.203 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj3 2023-11-02T18:16:06.203 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:veMxFMe8tOE4sO8gSINP+hxf+qI= 2023-11-02T18:16:06.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:veMxFMe8tOE4sO8gSINP+hxf+qI=', 'Content-Length': '0'} 2023-11-02T18:16:06.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df98ee25e415bed3-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.207 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.208 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.208 DEBUG:boto:path=/ibtljn-46/?prefix=obj3 2023-11-02T18:16:06.209 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj3 2023-11-02T18:16:06.209 DEBUG:boto:Method: GET 2023-11-02T18:16:06.210 DEBUG:boto:Path: /ibtljn-46/?prefix=obj3 2023-11-02T18:16:06.210 DEBUG:boto:Data: 2023-11-02T18:16:06.211 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.211 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.212 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.212 DEBUG:boto:Params: {} 2023-11-02T18:16:06.212 DEBUG:boto:Token: None 2023-11-02T18:16:06.213 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.213 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bac918e1e5dfa41c-006543e766-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.220 DEBUG:boto:b'ibtljn-46obj31000falseobj32023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.221 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.222 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.222 DEBUG:boto:path=/ibtljn-46/?prefix=obj3 2023-11-02T18:16:06.223 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj3 2023-11-02T18:16:06.223 DEBUG:boto:Method: GET 2023-11-02T18:16:06.224 DEBUG:boto:Path: /ibtljn-46/?prefix=obj3 2023-11-02T18:16:06.224 DEBUG:boto:Data: 2023-11-02T18:16:06.224 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.225 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.226 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.227 DEBUG:boto:Params: {} 2023-11-02T18:16:06.227 DEBUG:boto:Token: None 2023-11-02T18:16:06.228 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.228 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f0ec03ba6a2b743-006543e766-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.234 DEBUG:boto:b'ibtljn-46obj31000falseobj32023-11-02T18:13:45.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:16:06.236 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:16:06.236 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:16:06.237 DEBUG:boto:path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.237 DEBUG:boto:auth_path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.238 DEBUG:boto:Method: GET 2023-11-02T18:16:06.238 DEBUG:boto:Path: /ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.239 DEBUG:boto:Data: 2023-11-02T18:16:06.239 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.240 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.240 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.240 DEBUG:boto:Params: {} 2023-11-02T18:16:06.241 DEBUG:boto:Token: None 2023-11-02T18:16:06.241 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ijaPsfNBMeh4DQ1SX2hirWBhiPs= 2023-11-02T18:16:06.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ijaPsfNBMeh4DQ1SX2hirWBhiPs=', 'Content-Length': '0'} 2023-11-02T18:16:06.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f10add863c961c7-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.286 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:16:06.287 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:16:06.287 DEBUG:boto:path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.288 DEBUG:boto:auth_path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.288 DEBUG:boto:Method: GET 2023-11-02T18:16:06.289 DEBUG:boto:Path: /ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.289 DEBUG:boto:Data: 2023-11-02T18:16:06.290 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.290 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.291 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.291 DEBUG:boto:Params: {} 2023-11-02T18:16:06.292 DEBUG:boto:Token: None 2023-11-02T18:16:06.292 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.293 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ijaPsfNBMeh4DQ1SX2hirWBhiPs= 2023-11-02T18:16:06.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ijaPsfNBMeh4DQ1SX2hirWBhiPs=', 'Content-Length': '0'} 2023-11-02T18:16:06.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4fda4ac9c98a633-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.298 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:16:06.298 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:16:06.299 DEBUG:boto:path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.299 DEBUG:boto:auth_path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.300 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.300 DEBUG:boto:Path: /ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.301 DEBUG:boto:Data: 2023-11-02T18:16:06.301 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.301 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.302 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.302 DEBUG:boto:Params: {} 2023-11-02T18:16:06.303 DEBUG:boto:Token: None 2023-11-02T18:16:06.304 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.304 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LeF/Yq9KbumMze5Y4FQ9/4o7omk= 2023-11-02T18:16:06.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LeF/Yq9KbumMze5Y4FQ9/4o7omk=', 'Content-Length': '0'} 2023-11-02T18:16:06.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e2f9971f54a0e57-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.308 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:16:06.309 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:16:06.309 DEBUG:boto:path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.310 DEBUG:boto:auth_path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.310 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.311 DEBUG:boto:Path: /ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.311 DEBUG:boto:Data: 2023-11-02T18:16:06.311 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.312 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.312 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.313 DEBUG:boto:Params: {} 2023-11-02T18:16:06.313 DEBUG:boto:Token: None 2023-11-02T18:16:06.314 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.314 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LeF/Yq9KbumMze5Y4FQ9/4o7omk= 2023-11-02T18:16:06.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LeF/Yq9KbumMze5Y4FQ9/4o7omk=', 'Content-Length': '0'} 2023-11-02T18:16:06.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077f986945e00a7bb-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:16:06.319 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:16:06.319 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:16:06.319 DEBUG:boto:path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.320 DEBUG:boto:auth_path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.320 DEBUG:boto:Method: GET 2023-11-02T18:16:06.321 DEBUG:boto:Path: /ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.321 DEBUG:boto:Data: 2023-11-02T18:16:06.322 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.322 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.323 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.323 DEBUG:boto:Params: {} 2023-11-02T18:16:06.323 DEBUG:boto:Token: None 2023-11-02T18:16:06.324 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.324 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ijaPsfNBMeh4DQ1SX2hirWBhiPs= 2023-11-02T18:16:06.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ijaPsfNBMeh4DQ1SX2hirWBhiPs=', 'Content-Length': '0'} 2023-11-02T18:16:06.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c4b2020ff2a87fc-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.329 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:16:06.329 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:16:06.329 DEBUG:boto:path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.330 DEBUG:boto:auth_path=/ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.330 DEBUG:boto:Method: GET 2023-11-02T18:16:06.331 DEBUG:boto:Path: /ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.331 DEBUG:boto:Data: 2023-11-02T18:16:06.332 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.332 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.333 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.333 DEBUG:boto:Params: {} 2023-11-02T18:16:06.334 DEBUG:boto:Token: None 2023-11-02T18:16:06.335 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj4?versionId=null 2023-11-02T18:16:06.335 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ijaPsfNBMeh4DQ1SX2hirWBhiPs= 2023-11-02T18:16:06.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ijaPsfNBMeh4DQ1SX2hirWBhiPs=', 'Content-Length': '0'} 2023-11-02T18:16:06.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8f2ef7d86c9e1cc-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.339 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.340 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.340 DEBUG:boto:path=/ibtljn-46/?prefix=obj4 2023-11-02T18:16:06.341 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj4 2023-11-02T18:16:06.341 DEBUG:boto:Method: GET 2023-11-02T18:16:06.341 DEBUG:boto:Path: /ibtljn-46/?prefix=obj4 2023-11-02T18:16:06.342 DEBUG:boto:Data: 2023-11-02T18:16:06.342 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.343 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.343 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.344 DEBUG:boto:Params: {} 2023-11-02T18:16:06.344 DEBUG:boto:Token: None 2023-11-02T18:16:06.345 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.345 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b6609eae292df27-006543e766-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.394 DEBUG:boto:b'ibtljn-46obj41000falseobj42023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.395 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.396 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.396 DEBUG:boto:path=/ibtljn-46/?prefix=obj4 2023-11-02T18:16:06.397 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj4 2023-11-02T18:16:06.397 DEBUG:boto:Method: GET 2023-11-02T18:16:06.398 DEBUG:boto:Path: /ibtljn-46/?prefix=obj4 2023-11-02T18:16:06.398 DEBUG:boto:Data: 2023-11-02T18:16:06.398 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.399 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.399 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.400 DEBUG:boto:Params: {} 2023-11-02T18:16:06.400 DEBUG:boto:Token: None 2023-11-02T18:16:06.401 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.401 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029ac6b1a46a5f18e-006543e766-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.408 DEBUG:boto:b'ibtljn-46obj41000falseobj42023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.409 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:16:06.409 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:16:06.410 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.410 DEBUG:boto:Path: /ibtljn-46/obj4 2023-11-02T18:16:06.411 DEBUG:boto:Data: 2023-11-02T18:16:06.411 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.411 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.412 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.412 DEBUG:boto:Params: {} 2023-11-02T18:16:06.413 DEBUG:boto:Token: None 2023-11-02T18:16:06.413 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj4 2023-11-02T18:16:06.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Quhsby7jHgmd3ehjk/doCFc4QsQ= 2023-11-02T18:16:06.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Quhsby7jHgmd3ehjk/doCFc4QsQ=', 'Content-Length': '0'} 2023-11-02T18:16:06.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ac44f15313739d2-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.418 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:16:06.418 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:16:06.419 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.419 DEBUG:boto:Path: /ibtljn-46/obj4 2023-11-02T18:16:06.420 DEBUG:boto:Data: 2023-11-02T18:16:06.420 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.420 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.421 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.421 DEBUG:boto:Params: {} 2023-11-02T18:16:06.422 DEBUG:boto:Token: None 2023-11-02T18:16:06.422 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj4 2023-11-02T18:16:06.423 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Quhsby7jHgmd3ehjk/doCFc4QsQ= 2023-11-02T18:16:06.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Quhsby7jHgmd3ehjk/doCFc4QsQ=', 'Content-Length': '0'} 2023-11-02T18:16:06.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009492976f75e038ab-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:16:06.427 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:16:06.427 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:16:06.428 DEBUG:boto:Method: GET 2023-11-02T18:16:06.428 DEBUG:boto:Path: /ibtljn-46/obj4 2023-11-02T18:16:06.429 DEBUG:boto:Data: 2023-11-02T18:16:06.429 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.430 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.430 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.431 DEBUG:boto:Params: {} 2023-11-02T18:16:06.431 DEBUG:boto:Token: None 2023-11-02T18:16:06.432 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj4 2023-11-02T18:16:06.432 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hVCqXk34flBI158B0QejH/VujXM= 2023-11-02T18:16:06.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hVCqXk34flBI158B0QejH/VujXM=', 'Content-Length': '0'} 2023-11-02T18:16:06.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056c948d19673a0ac-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.436 DEBUG:boto:path=/ibtljn-46/obj4 2023-11-02T18:16:06.437 DEBUG:boto:auth_path=/ibtljn-46/obj4 2023-11-02T18:16:06.437 DEBUG:boto:Method: GET 2023-11-02T18:16:06.438 DEBUG:boto:Path: /ibtljn-46/obj4 2023-11-02T18:16:06.438 DEBUG:boto:Data: 2023-11-02T18:16:06.438 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.439 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.439 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.440 DEBUG:boto:Params: {} 2023-11-02T18:16:06.440 DEBUG:boto:Token: None 2023-11-02T18:16:06.441 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj4 2023-11-02T18:16:06.441 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hVCqXk34flBI158B0QejH/VujXM= 2023-11-02T18:16:06.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hVCqXk34flBI158B0QejH/VujXM=', 'Content-Length': '0'} 2023-11-02T18:16:06.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b71e7c1d886fa88-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.486 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.487 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.488 DEBUG:boto:path=/ibtljn-46/?prefix=obj4 2023-11-02T18:16:06.488 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj4 2023-11-02T18:16:06.489 DEBUG:boto:Method: GET 2023-11-02T18:16:06.489 DEBUG:boto:Path: /ibtljn-46/?prefix=obj4 2023-11-02T18:16:06.489 DEBUG:boto:Data: 2023-11-02T18:16:06.490 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.490 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.491 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.491 DEBUG:boto:Params: {} 2023-11-02T18:16:06.492 DEBUG:boto:Token: None 2023-11-02T18:16:06.492 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.493 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2020e8ce20ccd7a-006543e766-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.500 DEBUG:boto:b'ibtljn-46obj41000falseobj42023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.500 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.501 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.501 DEBUG:boto:path=/ibtljn-46/?prefix=obj4 2023-11-02T18:16:06.502 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj4 2023-11-02T18:16:06.502 DEBUG:boto:Method: GET 2023-11-02T18:16:06.503 DEBUG:boto:Path: /ibtljn-46/?prefix=obj4 2023-11-02T18:16:06.503 DEBUG:boto:Data: 2023-11-02T18:16:06.503 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.504 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.504 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.505 DEBUG:boto:Params: {} 2023-11-02T18:16:06.505 DEBUG:boto:Token: None 2023-11-02T18:16:06.507 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.507 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c759a2937e1dd61b-006543e766-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.514 DEBUG:boto:b'ibtljn-46obj41000falseobj42023-11-02T18:13:45.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-11-02T18:16:06.515 DEBUG:boto:path=/ibtljn-46/obj5 2023-11-02T18:16:06.515 DEBUG:boto:auth_path=/ibtljn-46/obj5 2023-11-02T18:16:06.516 DEBUG:boto:path=/ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.516 DEBUG:boto:auth_path=/ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.517 DEBUG:boto:Method: GET 2023-11-02T18:16:06.517 DEBUG:boto:Path: /ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.518 DEBUG:boto:Data: 2023-11-02T18:16:06.518 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.519 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.520 DEBUG:boto:Params: {} 2023-11-02T18:16:06.520 DEBUG:boto:Token: None 2023-11-02T18:16:06.521 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2eBL03mBpXm4Bg1M5BgVXvNvQ64= 2023-11-02T18:16:06.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2eBL03mBpXm4Bg1M5BgVXvNvQ64=', 'Content-Length': '0'} 2023-11-02T18:16:06.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000979e258a40ef2099-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.526 DEBUG:boto:path=/ibtljn-46/obj5 2023-11-02T18:16:06.526 DEBUG:boto:auth_path=/ibtljn-46/obj5 2023-11-02T18:16:06.527 DEBUG:boto:path=/ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.527 DEBUG:boto:auth_path=/ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.527 DEBUG:boto:Method: GET 2023-11-02T18:16:06.528 DEBUG:boto:Path: /ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.528 DEBUG:boto:Data: 2023-11-02T18:16:06.529 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.529 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.530 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.530 DEBUG:boto:Params: {} 2023-11-02T18:16:06.531 DEBUG:boto:Token: None 2023-11-02T18:16:06.531 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.532 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2eBL03mBpXm4Bg1M5BgVXvNvQ64= 2023-11-02T18:16:06.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2eBL03mBpXm4Bg1M5BgVXvNvQ64=', 'Content-Length': '0'} 2023-11-02T18:16:06.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a1f6a3ec07466a9-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.536 DEBUG:boto:path=/ibtljn-46/obj5 2023-11-02T18:16:06.537 DEBUG:boto:auth_path=/ibtljn-46/obj5 2023-11-02T18:16:06.537 DEBUG:boto:path=/ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.537 DEBUG:boto:auth_path=/ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.538 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.538 DEBUG:boto:Path: /ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.539 DEBUG:boto:Data: 2023-11-02T18:16:06.539 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.540 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.540 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.541 DEBUG:boto:Params: {} 2023-11-02T18:16:06.541 DEBUG:boto:Token: None 2023-11-02T18:16:06.541 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.542 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QUmvm8fV00zexgQUSD4tLrdz0FI= 2023-11-02T18:16:06.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QUmvm8fV00zexgQUSD4tLrdz0FI=', 'Content-Length': '0'} 2023-11-02T18:16:06.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c23a494c785ead54-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.546 DEBUG:boto:path=/ibtljn-46/obj5 2023-11-02T18:16:06.547 DEBUG:boto:auth_path=/ibtljn-46/obj5 2023-11-02T18:16:06.547 DEBUG:boto:path=/ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.547 DEBUG:boto:auth_path=/ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.548 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.548 DEBUG:boto:Path: /ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.549 DEBUG:boto:Data: 2023-11-02T18:16:06.549 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.550 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.550 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.551 DEBUG:boto:Params: {} 2023-11-02T18:16:06.551 DEBUG:boto:Token: None 2023-11-02T18:16:06.552 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.552 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QUmvm8fV00zexgQUSD4tLrdz0FI= 2023-11-02T18:16:06.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QUmvm8fV00zexgQUSD4tLrdz0FI=', 'Content-Length': '0'} 2023-11-02T18:16:06.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f5407742e1b375f-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-11-02T18:16:06.557 DEBUG:boto:path=/ibtljn-46/obj5 2023-11-02T18:16:06.557 DEBUG:boto:auth_path=/ibtljn-46/obj5 2023-11-02T18:16:06.558 DEBUG:boto:path=/ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.558 DEBUG:boto:auth_path=/ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.559 DEBUG:boto:Method: GET 2023-11-02T18:16:06.559 DEBUG:boto:Path: /ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.560 DEBUG:boto:Data: 2023-11-02T18:16:06.560 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.561 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.561 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.562 DEBUG:boto:Params: {} 2023-11-02T18:16:06.562 DEBUG:boto:Token: None 2023-11-02T18:16:06.563 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.563 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2eBL03mBpXm4Bg1M5BgVXvNvQ64= 2023-11-02T18:16:06.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2eBL03mBpXm4Bg1M5BgVXvNvQ64=', 'Content-Length': '0'} 2023-11-02T18:16:06.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003993d891cc900e8f-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.567 DEBUG:boto:path=/ibtljn-46/obj5 2023-11-02T18:16:06.567 DEBUG:boto:auth_path=/ibtljn-46/obj5 2023-11-02T18:16:06.568 DEBUG:boto:path=/ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.568 DEBUG:boto:auth_path=/ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.569 DEBUG:boto:Method: GET 2023-11-02T18:16:06.569 DEBUG:boto:Path: /ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.570 DEBUG:boto:Data: 2023-11-02T18:16:06.570 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.571 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.571 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.571 DEBUG:boto:Params: {} 2023-11-02T18:16:06.572 DEBUG:boto:Token: None 2023-11-02T18:16:06.572 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj5?versionId=null 2023-11-02T18:16:06.573 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2eBL03mBpXm4Bg1M5BgVXvNvQ64= 2023-11-02T18:16:06.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2eBL03mBpXm4Bg1M5BgVXvNvQ64=', 'Content-Length': '0'} 2023-11-02T18:16:06.576 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011ffa8c987463adb-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.618 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.619 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.620 DEBUG:boto:path=/ibtljn-46/?prefix=obj5 2023-11-02T18:16:06.620 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj5 2023-11-02T18:16:06.621 DEBUG:boto:Method: GET 2023-11-02T18:16:06.621 DEBUG:boto:Path: /ibtljn-46/?prefix=obj5 2023-11-02T18:16:06.622 DEBUG:boto:Data: 2023-11-02T18:16:06.622 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.623 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.623 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.624 DEBUG:boto:Params: {} 2023-11-02T18:16:06.624 DEBUG:boto:Token: None 2023-11-02T18:16:06.625 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009af3c3e287b5f174-006543e766-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.632 DEBUG:boto:b'ibtljn-46obj51000falseobj52023-11-02T18:15:24.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.633 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.633 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.634 DEBUG:boto:path=/ibtljn-46/?prefix=obj5 2023-11-02T18:16:06.634 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj5 2023-11-02T18:16:06.635 DEBUG:boto:Method: GET 2023-11-02T18:16:06.635 DEBUG:boto:Path: /ibtljn-46/?prefix=obj5 2023-11-02T18:16:06.636 DEBUG:boto:Data: 2023-11-02T18:16:06.636 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.637 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.638 DEBUG:boto:Params: {} 2023-11-02T18:16:06.638 DEBUG:boto:Token: None 2023-11-02T18:16:06.639 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.639 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e675a116cf7c0dd-006543e766-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.645 DEBUG:boto:b'ibtljn-46obj51000falseobj52023-11-02T18:15:24.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.646 DEBUG:boto:path=/ibtljn-46/obj5 2023-11-02T18:16:06.647 DEBUG:boto:auth_path=/ibtljn-46/obj5 2023-11-02T18:16:06.647 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.648 DEBUG:boto:Path: /ibtljn-46/obj5 2023-11-02T18:16:06.648 DEBUG:boto:Data: 2023-11-02T18:16:06.649 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.649 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.649 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.650 DEBUG:boto:Params: {} 2023-11-02T18:16:06.650 DEBUG:boto:Token: None 2023-11-02T18:16:06.651 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj5 2023-11-02T18:16:06.652 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jXjKxRE+SGh4eXKMYos6qXSF/uc= 2023-11-02T18:16:06.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jXjKxRE+SGh4eXKMYos6qXSF/uc=', 'Content-Length': '0'} 2023-11-02T18:16:06.655 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d038c523bd4d8cd4-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.656 DEBUG:boto:path=/ibtljn-46/obj5 2023-11-02T18:16:06.656 DEBUG:boto:auth_path=/ibtljn-46/obj5 2023-11-02T18:16:06.657 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.657 DEBUG:boto:Path: /ibtljn-46/obj5 2023-11-02T18:16:06.658 DEBUG:boto:Data: 2023-11-02T18:16:06.658 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.658 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.659 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.659 DEBUG:boto:Params: {} 2023-11-02T18:16:06.660 DEBUG:boto:Token: None 2023-11-02T18:16:06.660 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj5 2023-11-02T18:16:06.661 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jXjKxRE+SGh4eXKMYos6qXSF/uc= 2023-11-02T18:16:06.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jXjKxRE+SGh4eXKMYos6qXSF/uc=', 'Content-Length': '0'} 2023-11-02T18:16:06.664 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075e34591e04d935e-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-11-02T18:16:06.665 DEBUG:boto:path=/ibtljn-46/obj5 2023-11-02T18:16:06.666 DEBUG:boto:auth_path=/ibtljn-46/obj5 2023-11-02T18:16:06.666 DEBUG:boto:Method: GET 2023-11-02T18:16:06.667 DEBUG:boto:Path: /ibtljn-46/obj5 2023-11-02T18:16:06.667 DEBUG:boto:Data: 2023-11-02T18:16:06.668 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.668 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.669 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.669 DEBUG:boto:Params: {} 2023-11-02T18:16:06.670 DEBUG:boto:Token: None 2023-11-02T18:16:06.670 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj5 2023-11-02T18:16:06.671 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:y6ooWqT8b1qAiovh6kOdmXeVc/A= 2023-11-02T18:16:06.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:y6ooWqT8b1qAiovh6kOdmXeVc/A=', 'Content-Length': '0'} 2023-11-02T18:16:06.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdf1b02e150788f7-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.718 DEBUG:boto:path=/ibtljn-46/obj5 2023-11-02T18:16:06.719 DEBUG:boto:auth_path=/ibtljn-46/obj5 2023-11-02T18:16:06.720 DEBUG:boto:Method: GET 2023-11-02T18:16:06.720 DEBUG:boto:Path: /ibtljn-46/obj5 2023-11-02T18:16:06.721 DEBUG:boto:Data: 2023-11-02T18:16:06.721 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.722 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.722 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.723 DEBUG:boto:Params: {} 2023-11-02T18:16:06.723 DEBUG:boto:Token: None 2023-11-02T18:16:06.724 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj5 2023-11-02T18:16:06.725 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:y6ooWqT8b1qAiovh6kOdmXeVc/A= 2023-11-02T18:16:06.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:y6ooWqT8b1qAiovh6kOdmXeVc/A=', 'Content-Length': '0'} 2023-11-02T18:16:06.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015998c50d1305a38-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.729 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.730 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.730 DEBUG:boto:path=/ibtljn-46/?prefix=obj5 2023-11-02T18:16:06.731 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj5 2023-11-02T18:16:06.731 DEBUG:boto:Method: GET 2023-11-02T18:16:06.732 DEBUG:boto:Path: /ibtljn-46/?prefix=obj5 2023-11-02T18:16:06.732 DEBUG:boto:Data: 2023-11-02T18:16:06.732 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.733 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.733 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.734 DEBUG:boto:Params: {} 2023-11-02T18:16:06.734 DEBUG:boto:Token: None 2023-11-02T18:16:06.735 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.735 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059ac6030a10e43be-006543e766-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.742 DEBUG:boto:b'ibtljn-46obj51000falseobj52023-11-02T18:15:24.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.743 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.744 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.744 DEBUG:boto:path=/ibtljn-46/?prefix=obj5 2023-11-02T18:16:06.745 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj5 2023-11-02T18:16:06.745 DEBUG:boto:Method: GET 2023-11-02T18:16:06.746 DEBUG:boto:Path: /ibtljn-46/?prefix=obj5 2023-11-02T18:16:06.747 DEBUG:boto:Data: 2023-11-02T18:16:06.747 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.748 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.748 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.749 DEBUG:boto:Params: {} 2023-11-02T18:16:06.749 DEBUG:boto:Token: None 2023-11-02T18:16:06.750 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.750 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e78b0bff1558b459-006543e766-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.757 DEBUG:boto:b'ibtljn-46obj51000falseobj52023-11-02T18:15:24.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-11-02T18:16:06.759 DEBUG:boto:path=/ibtljn-46/obj6 2023-11-02T18:16:06.759 DEBUG:boto:auth_path=/ibtljn-46/obj6 2023-11-02T18:16:06.760 DEBUG:boto:path=/ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.760 DEBUG:boto:auth_path=/ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.760 DEBUG:boto:Method: GET 2023-11-02T18:16:06.761 DEBUG:boto:Path: /ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.761 DEBUG:boto:Data: 2023-11-02T18:16:06.762 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.762 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.763 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.763 DEBUG:boto:Params: {} 2023-11-02T18:16:06.764 DEBUG:boto:Token: None 2023-11-02T18:16:06.764 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.765 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ybck+TD+C7x5wjNur06JwYEE3V8= 2023-11-02T18:16:06.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ybck+TD+C7x5wjNur06JwYEE3V8=', 'Content-Length': '0'} 2023-11-02T18:16:06.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000955f2e55e23bcb9e-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.769 DEBUG:boto:path=/ibtljn-46/obj6 2023-11-02T18:16:06.770 DEBUG:boto:auth_path=/ibtljn-46/obj6 2023-11-02T18:16:06.770 DEBUG:boto:path=/ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.771 DEBUG:boto:auth_path=/ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.771 DEBUG:boto:Method: GET 2023-11-02T18:16:06.772 DEBUG:boto:Path: /ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.772 DEBUG:boto:Data: 2023-11-02T18:16:06.773 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.773 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.773 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.774 DEBUG:boto:Params: {} 2023-11-02T18:16:06.774 DEBUG:boto:Token: None 2023-11-02T18:16:06.775 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.775 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ybck+TD+C7x5wjNur06JwYEE3V8= 2023-11-02T18:16:06.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ybck+TD+C7x5wjNur06JwYEE3V8=', 'Content-Length': '0'} 2023-11-02T18:16:06.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2e62d9abfe2e89f-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.826 DEBUG:boto:path=/ibtljn-46/obj6 2023-11-02T18:16:06.827 DEBUG:boto:auth_path=/ibtljn-46/obj6 2023-11-02T18:16:06.827 DEBUG:boto:path=/ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.828 DEBUG:boto:auth_path=/ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.828 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.829 DEBUG:boto:Path: /ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.829 DEBUG:boto:Data: 2023-11-02T18:16:06.830 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.830 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.830 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.831 DEBUG:boto:Params: {} 2023-11-02T18:16:06.831 DEBUG:boto:Token: None 2023-11-02T18:16:06.832 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.832 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nBiDijKs/UbwXVccaQ76Md1Su2Y= 2023-11-02T18:16:06.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nBiDijKs/UbwXVccaQ76Md1Su2Y=', 'Content-Length': '0'} 2023-11-02T18:16:06.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a8b3511a8675325-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.837 DEBUG:boto:path=/ibtljn-46/obj6 2023-11-02T18:16:06.837 DEBUG:boto:auth_path=/ibtljn-46/obj6 2023-11-02T18:16:06.838 DEBUG:boto:path=/ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.838 DEBUG:boto:auth_path=/ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.839 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.839 DEBUG:boto:Path: /ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.840 DEBUG:boto:Data: 2023-11-02T18:16:06.840 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.841 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.842 DEBUG:boto:Params: {} 2023-11-02T18:16:06.842 DEBUG:boto:Token: None 2023-11-02T18:16:06.843 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.843 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nBiDijKs/UbwXVccaQ76Md1Su2Y= 2023-11-02T18:16:06.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nBiDijKs/UbwXVccaQ76Md1Su2Y=', 'Content-Length': '0'} 2023-11-02T18:16:06.847 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000879f2a7326bd74e5-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-11-02T18:16:06.848 DEBUG:boto:path=/ibtljn-46/obj6 2023-11-02T18:16:06.848 DEBUG:boto:auth_path=/ibtljn-46/obj6 2023-11-02T18:16:06.849 DEBUG:boto:path=/ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.849 DEBUG:boto:auth_path=/ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.850 DEBUG:boto:Method: GET 2023-11-02T18:16:06.850 DEBUG:boto:Path: /ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.851 DEBUG:boto:Data: 2023-11-02T18:16:06.851 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.852 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.853 DEBUG:boto:Params: {} 2023-11-02T18:16:06.853 DEBUG:boto:Token: None 2023-11-02T18:16:06.854 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.854 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ybck+TD+C7x5wjNur06JwYEE3V8= 2023-11-02T18:16:06.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ybck+TD+C7x5wjNur06JwYEE3V8=', 'Content-Length': '0'} 2023-11-02T18:16:06.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000270c9c26bd8dc3a-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.902 DEBUG:boto:path=/ibtljn-46/obj6 2023-11-02T18:16:06.903 DEBUG:boto:auth_path=/ibtljn-46/obj6 2023-11-02T18:16:06.903 DEBUG:boto:path=/ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.904 DEBUG:boto:auth_path=/ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.904 DEBUG:boto:Method: GET 2023-11-02T18:16:06.905 DEBUG:boto:Path: /ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.905 DEBUG:boto:Data: 2023-11-02T18:16:06.906 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.907 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.907 DEBUG:boto:Params: {} 2023-11-02T18:16:06.908 DEBUG:boto:Token: None 2023-11-02T18:16:06.908 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj6?versionId=null 2023-11-02T18:16:06.909 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ybck+TD+C7x5wjNur06JwYEE3V8= 2023-11-02T18:16:06.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ybck+TD+C7x5wjNur06JwYEE3V8=', 'Content-Length': '0'} 2023-11-02T18:16:06.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f0ec15a673791f1-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.913 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.914 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.914 DEBUG:boto:path=/ibtljn-46/?prefix=obj6 2023-11-02T18:16:06.915 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj6 2023-11-02T18:16:06.915 DEBUG:boto:Method: GET 2023-11-02T18:16:06.916 DEBUG:boto:Path: /ibtljn-46/?prefix=obj6 2023-11-02T18:16:06.916 DEBUG:boto:Data: 2023-11-02T18:16:06.917 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.917 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.917 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.918 DEBUG:boto:Params: {} 2023-11-02T18:16:06.918 DEBUG:boto:Token: None 2023-11-02T18:16:06.919 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.919 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b57dca26d4d12ba0-006543e766-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.926 DEBUG:boto:b'ibtljn-46obj61000falseobj62023-11-02T18:15:24.686Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.927 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:06.927 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:06.927 DEBUG:boto:path=/ibtljn-46/?prefix=obj6 2023-11-02T18:16:06.928 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj6 2023-11-02T18:16:06.928 DEBUG:boto:Method: GET 2023-11-02T18:16:06.929 DEBUG:boto:Path: /ibtljn-46/?prefix=obj6 2023-11-02T18:16:06.929 DEBUG:boto:Data: 2023-11-02T18:16:06.929 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.930 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.931 DEBUG:boto:Params: {} 2023-11-02T18:16:06.931 DEBUG:boto:Token: None 2023-11-02T18:16:06.932 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/ 2023-11-02T18:16:06.932 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs= 2023-11-02T18:16:06.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:if6BeMayElvIAE3W3wGJVZcfTQs=', 'Content-Length': '0'} 2023-11-02T18:16:06.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d1158854750468e-006543e766-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.938 DEBUG:boto:b'ibtljn-46obj61000falseobj62023-11-02T18:15:24.686Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:06.939 DEBUG:boto:path=/ibtljn-46/obj6 2023-11-02T18:16:06.939 DEBUG:boto:auth_path=/ibtljn-46/obj6 2023-11-02T18:16:06.940 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.940 DEBUG:boto:Path: /ibtljn-46/obj6 2023-11-02T18:16:06.941 DEBUG:boto:Data: 2023-11-02T18:16:06.941 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.941 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.942 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.942 DEBUG:boto:Params: {} 2023-11-02T18:16:06.943 DEBUG:boto:Token: None 2023-11-02T18:16:06.943 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj6 2023-11-02T18:16:06.944 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k2aYdet3OLQplnRkik+RZAF2Bh4= 2023-11-02T18:16:06.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k2aYdet3OLQplnRkik+RZAF2Bh4=', 'Content-Length': '0'} 2023-11-02T18:16:06.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087be7aafb5d99c02-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.947 DEBUG:boto:path=/ibtljn-46/obj6 2023-11-02T18:16:06.948 DEBUG:boto:auth_path=/ibtljn-46/obj6 2023-11-02T18:16:06.948 DEBUG:boto:Method: HEAD 2023-11-02T18:16:06.948 DEBUG:boto:Path: /ibtljn-46/obj6 2023-11-02T18:16:06.949 DEBUG:boto:Data: 2023-11-02T18:16:06.949 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.949 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:06.950 DEBUG:boto:Port: 8000 2023-11-02T18:16:06.950 DEBUG:boto:Params: {} 2023-11-02T18:16:06.951 DEBUG:boto:Token: None 2023-11-02T18:16:06.951 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj6 2023-11-02T18:16:06.952 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k2aYdet3OLQplnRkik+RZAF2Bh4= 2023-11-02T18:16:06.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k2aYdet3OLQplnRkik+RZAF2Bh4=', 'Content-Length': '0'} 2023-11-02T18:16:06.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090195fd22360880d-006543e766-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:06.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-11-02T18:16:06.956 DEBUG:boto:path=/ibtljn-46/obj6 2023-11-02T18:16:06.956 DEBUG:boto:auth_path=/ibtljn-46/obj6 2023-11-02T18:16:06.957 DEBUG:boto:Method: GET 2023-11-02T18:16:06.957 DEBUG:boto:Path: /ibtljn-46/obj6 2023-11-02T18:16:06.958 DEBUG:boto:Data: 2023-11-02T18:16:06.958 DEBUG:boto:Headers: {} 2023-11-02T18:16:06.959 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:06.959 DEBUG:boto:Port: 8001 2023-11-02T18:16:06.959 DEBUG:boto:Params: {} 2023-11-02T18:16:06.960 DEBUG:boto:Token: None 2023-11-02T18:16:06.960 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:06 GMT /ibtljn-46/obj6 2023-11-02T18:16:06.961 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T/LqZfkdC+ccuRj6DDRK+Ph8ZGg= 2023-11-02T18:16:06.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T/LqZfkdC+ccuRj6DDRK+Ph8ZGg=', 'Content-Length': '0'} 2023-11-02T18:16:06.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ef89be42ba46590-006543e766-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.010 DEBUG:boto:path=/ibtljn-46/obj6 2023-11-02T18:16:07.011 DEBUG:boto:auth_path=/ibtljn-46/obj6 2023-11-02T18:16:07.012 DEBUG:boto:Method: GET 2023-11-02T18:16:07.012 DEBUG:boto:Path: /ibtljn-46/obj6 2023-11-02T18:16:07.012 DEBUG:boto:Data: 2023-11-02T18:16:07.013 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.013 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.014 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.014 DEBUG:boto:Params: {} 2023-11-02T18:16:07.015 DEBUG:boto:Token: None 2023-11-02T18:16:07.015 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj6 2023-11-02T18:16:07.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZxQKcMgVFQE/SyR8QRpyj+yI8g= 2023-11-02T18:16:07.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZxQKcMgVFQE/SyR8QRpyj+yI8g=', 'Content-Length': '0'} 2023-11-02T18:16:07.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f60dc15529ec71bd-006543e767-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.020 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:07.021 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:07.021 DEBUG:boto:path=/ibtljn-46/?prefix=obj6 2023-11-02T18:16:07.021 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj6 2023-11-02T18:16:07.022 DEBUG:boto:Method: GET 2023-11-02T18:16:07.022 DEBUG:boto:Path: /ibtljn-46/?prefix=obj6 2023-11-02T18:16:07.023 DEBUG:boto:Data: 2023-11-02T18:16:07.023 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.025 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.026 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.026 DEBUG:boto:Params: {} 2023-11-02T18:16:07.027 DEBUG:boto:Token: None 2023-11-02T18:16:07.027 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/ 2023-11-02T18:16:07.027 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss= 2023-11-02T18:16:07.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss=', 'Content-Length': '0'} 2023-11-02T18:16:07.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db4e878f14150f6c-006543e767-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.034 DEBUG:boto:b'ibtljn-46obj61000falseobj62023-11-02T18:15:24.686Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:07.035 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:07.036 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:07.036 DEBUG:boto:path=/ibtljn-46/?prefix=obj6 2023-11-02T18:16:07.036 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj6 2023-11-02T18:16:07.037 DEBUG:boto:Method: GET 2023-11-02T18:16:07.037 DEBUG:boto:Path: /ibtljn-46/?prefix=obj6 2023-11-02T18:16:07.038 DEBUG:boto:Data: 2023-11-02T18:16:07.038 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.039 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.039 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.039 DEBUG:boto:Params: {} 2023-11-02T18:16:07.040 DEBUG:boto:Token: None 2023-11-02T18:16:07.040 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/ 2023-11-02T18:16:07.041 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss= 2023-11-02T18:16:07.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss=', 'Content-Length': '0'} 2023-11-02T18:16:07.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000100c4a7aedaacaaa-006543e767-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.090 DEBUG:boto:b'ibtljn-46obj61000falseobj62023-11-02T18:15:24.686Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:07.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-11-02T18:16:07.092 DEBUG:boto:path=/ibtljn-46/obj7 2023-11-02T18:16:07.092 DEBUG:boto:auth_path=/ibtljn-46/obj7 2023-11-02T18:16:07.093 DEBUG:boto:path=/ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.093 DEBUG:boto:auth_path=/ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.094 DEBUG:boto:Method: GET 2023-11-02T18:16:07.094 DEBUG:boto:Path: /ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.095 DEBUG:boto:Data: 2023-11-02T18:16:07.095 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.095 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.096 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.096 DEBUG:boto:Params: {} 2023-11-02T18:16:07.097 DEBUG:boto:Token: None 2023-11-02T18:16:07.097 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EHotj1xX/eKM22eZBhkjvVsW1Bw= 2023-11-02T18:16:07.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EHotj1xX/eKM22eZBhkjvVsW1Bw=', 'Content-Length': '0'} 2023-11-02T18:16:07.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027f33a7b833cc679-006543e767-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.102 DEBUG:boto:path=/ibtljn-46/obj7 2023-11-02T18:16:07.103 DEBUG:boto:auth_path=/ibtljn-46/obj7 2023-11-02T18:16:07.103 DEBUG:boto:path=/ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.104 DEBUG:boto:auth_path=/ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.104 DEBUG:boto:Method: GET 2023-11-02T18:16:07.104 DEBUG:boto:Path: /ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.105 DEBUG:boto:Data: 2023-11-02T18:16:07.105 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.106 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.107 DEBUG:boto:Params: {} 2023-11-02T18:16:07.107 DEBUG:boto:Token: None 2023-11-02T18:16:07.108 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.108 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EHotj1xX/eKM22eZBhkjvVsW1Bw= 2023-11-02T18:16:07.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EHotj1xX/eKM22eZBhkjvVsW1Bw=', 'Content-Length': '0'} 2023-11-02T18:16:07.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbb8ec1e644aaa0f-006543e767-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.112 DEBUG:boto:path=/ibtljn-46/obj7 2023-11-02T18:16:07.113 DEBUG:boto:auth_path=/ibtljn-46/obj7 2023-11-02T18:16:07.113 DEBUG:boto:path=/ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.114 DEBUG:boto:auth_path=/ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.114 DEBUG:boto:Method: HEAD 2023-11-02T18:16:07.114 DEBUG:boto:Path: /ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.115 DEBUG:boto:Data: 2023-11-02T18:16:07.115 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.116 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.116 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.116 DEBUG:boto:Params: {} 2023-11-02T18:16:07.117 DEBUG:boto:Token: None 2023-11-02T18:16:07.117 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.118 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7TqXqN8BlUzSkYZO7BLa9YKBvww= 2023-11-02T18:16:07.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7TqXqN8BlUzSkYZO7BLa9YKBvww=', 'Content-Length': '0'} 2023-11-02T18:16:07.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000270675be961f2fe4-006543e767-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.122 DEBUG:boto:path=/ibtljn-46/obj7 2023-11-02T18:16:07.123 DEBUG:boto:auth_path=/ibtljn-46/obj7 2023-11-02T18:16:07.123 DEBUG:boto:path=/ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.123 DEBUG:boto:auth_path=/ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.124 DEBUG:boto:Method: HEAD 2023-11-02T18:16:07.124 DEBUG:boto:Path: /ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.125 DEBUG:boto:Data: 2023-11-02T18:16:07.125 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.125 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.126 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.126 DEBUG:boto:Params: {} 2023-11-02T18:16:07.127 DEBUG:boto:Token: None 2023-11-02T18:16:07.127 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.128 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7TqXqN8BlUzSkYZO7BLa9YKBvww= 2023-11-02T18:16:07.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7TqXqN8BlUzSkYZO7BLa9YKBvww=', 'Content-Length': '0'} 2023-11-02T18:16:07.131 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030edd09e56801ecd-006543e767-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-11-02T18:16:07.132 DEBUG:boto:path=/ibtljn-46/obj7 2023-11-02T18:16:07.133 DEBUG:boto:auth_path=/ibtljn-46/obj7 2023-11-02T18:16:07.133 DEBUG:boto:path=/ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.134 DEBUG:boto:auth_path=/ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.134 DEBUG:boto:Method: GET 2023-11-02T18:16:07.134 DEBUG:boto:Path: /ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.135 DEBUG:boto:Data: 2023-11-02T18:16:07.135 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.136 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.136 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.137 DEBUG:boto:Params: {} 2023-11-02T18:16:07.137 DEBUG:boto:Token: None 2023-11-02T18:16:07.138 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.138 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EHotj1xX/eKM22eZBhkjvVsW1Bw= 2023-11-02T18:16:07.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EHotj1xX/eKM22eZBhkjvVsW1Bw=', 'Content-Length': '0'} 2023-11-02T18:16:07.142 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b865603fad915cd5-006543e767-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.142 DEBUG:boto:path=/ibtljn-46/obj7 2023-11-02T18:16:07.143 DEBUG:boto:auth_path=/ibtljn-46/obj7 2023-11-02T18:16:07.143 DEBUG:boto:path=/ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.144 DEBUG:boto:auth_path=/ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.144 DEBUG:boto:Method: GET 2023-11-02T18:16:07.145 DEBUG:boto:Path: /ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.145 DEBUG:boto:Data: 2023-11-02T18:16:07.145 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.146 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.146 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.147 DEBUG:boto:Params: {} 2023-11-02T18:16:07.147 DEBUG:boto:Token: None 2023-11-02T18:16:07.148 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj7?versionId=null 2023-11-02T18:16:07.148 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EHotj1xX/eKM22eZBhkjvVsW1Bw= 2023-11-02T18:16:07.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EHotj1xX/eKM22eZBhkjvVsW1Bw=', 'Content-Length': '0'} 2023-11-02T18:16:07.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a93959264334f1c0-006543e767-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.194 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:07.195 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:07.195 DEBUG:boto:path=/ibtljn-46/?prefix=obj7 2023-11-02T18:16:07.196 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj7 2023-11-02T18:16:07.196 DEBUG:boto:Method: GET 2023-11-02T18:16:07.197 DEBUG:boto:Path: /ibtljn-46/?prefix=obj7 2023-11-02T18:16:07.197 DEBUG:boto:Data: 2023-11-02T18:16:07.197 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.198 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.198 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.199 DEBUG:boto:Params: {} 2023-11-02T18:16:07.199 DEBUG:boto:Token: None 2023-11-02T18:16:07.200 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/ 2023-11-02T18:16:07.200 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss= 2023-11-02T18:16:07.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss=', 'Content-Length': '0'} 2023-11-02T18:16:07.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e74a82e5fc8b96b-006543e767-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.207 DEBUG:boto:b'ibtljn-46obj71000falseobj72023-11-02T18:15:24.750Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:07.208 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:07.208 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:07.209 DEBUG:boto:path=/ibtljn-46/?prefix=obj7 2023-11-02T18:16:07.209 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj7 2023-11-02T18:16:07.210 DEBUG:boto:Method: GET 2023-11-02T18:16:07.210 DEBUG:boto:Path: /ibtljn-46/?prefix=obj7 2023-11-02T18:16:07.211 DEBUG:boto:Data: 2023-11-02T18:16:07.211 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.211 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.212 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.212 DEBUG:boto:Params: {} 2023-11-02T18:16:07.213 DEBUG:boto:Token: None 2023-11-02T18:16:07.213 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/ 2023-11-02T18:16:07.214 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss= 2023-11-02T18:16:07.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss=', 'Content-Length': '0'} 2023-11-02T18:16:07.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000392b95ca5440a9f9-006543e767-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.219 DEBUG:boto:b'ibtljn-46obj71000falseobj72023-11-02T18:15:24.750Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:07.220 DEBUG:boto:path=/ibtljn-46/obj7 2023-11-02T18:16:07.220 DEBUG:boto:auth_path=/ibtljn-46/obj7 2023-11-02T18:16:07.221 DEBUG:boto:Method: HEAD 2023-11-02T18:16:07.221 DEBUG:boto:Path: /ibtljn-46/obj7 2023-11-02T18:16:07.222 DEBUG:boto:Data: 2023-11-02T18:16:07.222 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.222 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.223 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.223 DEBUG:boto:Params: {} 2023-11-02T18:16:07.224 DEBUG:boto:Token: None 2023-11-02T18:16:07.224 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj7 2023-11-02T18:16:07.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EnpI8/iKxXANJsEbapK/edXAYFc= 2023-11-02T18:16:07.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EnpI8/iKxXANJsEbapK/edXAYFc=', 'Content-Length': '0'} 2023-11-02T18:16:07.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077b08fca729ff30c-006543e767-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.229 DEBUG:boto:path=/ibtljn-46/obj7 2023-11-02T18:16:07.229 DEBUG:boto:auth_path=/ibtljn-46/obj7 2023-11-02T18:16:07.229 DEBUG:boto:Method: HEAD 2023-11-02T18:16:07.230 DEBUG:boto:Path: /ibtljn-46/obj7 2023-11-02T18:16:07.230 DEBUG:boto:Data: 2023-11-02T18:16:07.231 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.231 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.232 DEBUG:boto:Params: {} 2023-11-02T18:16:07.232 DEBUG:boto:Token: None 2023-11-02T18:16:07.233 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj7 2023-11-02T18:16:07.233 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EnpI8/iKxXANJsEbapK/edXAYFc= 2023-11-02T18:16:07.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EnpI8/iKxXANJsEbapK/edXAYFc=', 'Content-Length': '0'} 2023-11-02T18:16:07.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004799cea86737463f-006543e767-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-11-02T18:16:07.237 DEBUG:boto:path=/ibtljn-46/obj7 2023-11-02T18:16:07.238 DEBUG:boto:auth_path=/ibtljn-46/obj7 2023-11-02T18:16:07.238 DEBUG:boto:Method: GET 2023-11-02T18:16:07.239 DEBUG:boto:Path: /ibtljn-46/obj7 2023-11-02T18:16:07.239 DEBUG:boto:Data: 2023-11-02T18:16:07.240 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.240 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.240 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.241 DEBUG:boto:Params: {} 2023-11-02T18:16:07.241 DEBUG:boto:Token: None 2023-11-02T18:16:07.242 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj7 2023-11-02T18:16:07.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4ncCXYrdVWs9IjD6MghjaPRKu0E= 2023-11-02T18:16:07.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4ncCXYrdVWs9IjD6MghjaPRKu0E=', 'Content-Length': '0'} 2023-11-02T18:16:07.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2240a776fc8f0f5-006543e767-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.290 DEBUG:boto:path=/ibtljn-46/obj7 2023-11-02T18:16:07.291 DEBUG:boto:auth_path=/ibtljn-46/obj7 2023-11-02T18:16:07.291 DEBUG:boto:Method: GET 2023-11-02T18:16:07.292 DEBUG:boto:Path: /ibtljn-46/obj7 2023-11-02T18:16:07.292 DEBUG:boto:Data: 2023-11-02T18:16:07.293 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.293 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.294 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.294 DEBUG:boto:Params: {} 2023-11-02T18:16:07.295 DEBUG:boto:Token: None 2023-11-02T18:16:07.295 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj7 2023-11-02T18:16:07.296 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4ncCXYrdVWs9IjD6MghjaPRKu0E= 2023-11-02T18:16:07.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4ncCXYrdVWs9IjD6MghjaPRKu0E=', 'Content-Length': '0'} 2023-11-02T18:16:07.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e9cdc8efe353ef9-006543e767-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.300 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:07.300 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:07.300 DEBUG:boto:path=/ibtljn-46/?prefix=obj7 2023-11-02T18:16:07.301 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj7 2023-11-02T18:16:07.301 DEBUG:boto:Method: GET 2023-11-02T18:16:07.302 DEBUG:boto:Path: /ibtljn-46/?prefix=obj7 2023-11-02T18:16:07.302 DEBUG:boto:Data: 2023-11-02T18:16:07.303 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.303 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.304 DEBUG:boto:Params: {} 2023-11-02T18:16:07.304 DEBUG:boto:Token: None 2023-11-02T18:16:07.305 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/ 2023-11-02T18:16:07.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss= 2023-11-02T18:16:07.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss=', 'Content-Length': '0'} 2023-11-02T18:16:07.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3e6e2fb26e6d346-006543e767-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.312 DEBUG:boto:b'ibtljn-46obj71000falseobj72023-11-02T18:15:24.750Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:07.313 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:07.313 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:07.314 DEBUG:boto:path=/ibtljn-46/?prefix=obj7 2023-11-02T18:16:07.314 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj7 2023-11-02T18:16:07.314 DEBUG:boto:Method: GET 2023-11-02T18:16:07.315 DEBUG:boto:Path: /ibtljn-46/?prefix=obj7 2023-11-02T18:16:07.315 DEBUG:boto:Data: 2023-11-02T18:16:07.316 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.316 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.317 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.317 DEBUG:boto:Params: {} 2023-11-02T18:16:07.317 DEBUG:boto:Token: None 2023-11-02T18:16:07.318 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/ 2023-11-02T18:16:07.318 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss= 2023-11-02T18:16:07.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss=', 'Content-Length': '0'} 2023-11-02T18:16:07.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0a42908b0768c0b-006543e767-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.366 DEBUG:boto:b'ibtljn-46obj71000falseobj72023-11-02T18:15:24.750Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:07.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-11-02T18:16:07.367 DEBUG:boto:path=/ibtljn-46/obj8 2023-11-02T18:16:07.368 DEBUG:boto:auth_path=/ibtljn-46/obj8 2023-11-02T18:16:07.368 DEBUG:boto:path=/ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.369 DEBUG:boto:auth_path=/ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.369 DEBUG:boto:Method: GET 2023-11-02T18:16:07.369 DEBUG:boto:Path: /ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.370 DEBUG:boto:Data: 2023-11-02T18:16:07.370 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.371 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.371 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.372 DEBUG:boto:Params: {} 2023-11-02T18:16:07.372 DEBUG:boto:Token: None 2023-11-02T18:16:07.372 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.373 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6NgaTYptEmN00WJUhrLs5YHcvME= 2023-11-02T18:16:07.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6NgaTYptEmN00WJUhrLs5YHcvME=', 'Content-Length': '0'} 2023-11-02T18:16:07.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcef104a32744825-006543e767-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.377 DEBUG:boto:path=/ibtljn-46/obj8 2023-11-02T18:16:07.378 DEBUG:boto:auth_path=/ibtljn-46/obj8 2023-11-02T18:16:07.378 DEBUG:boto:path=/ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.379 DEBUG:boto:auth_path=/ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.379 DEBUG:boto:Method: GET 2023-11-02T18:16:07.379 DEBUG:boto:Path: /ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.380 DEBUG:boto:Data: 2023-11-02T18:16:07.380 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.380 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.381 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.381 DEBUG:boto:Params: {} 2023-11-02T18:16:07.382 DEBUG:boto:Token: None 2023-11-02T18:16:07.382 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.383 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6NgaTYptEmN00WJUhrLs5YHcvME= 2023-11-02T18:16:07.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6NgaTYptEmN00WJUhrLs5YHcvME=', 'Content-Length': '0'} 2023-11-02T18:16:07.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c550a2db4cbe09a-006543e767-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.387 DEBUG:boto:path=/ibtljn-46/obj8 2023-11-02T18:16:07.387 DEBUG:boto:auth_path=/ibtljn-46/obj8 2023-11-02T18:16:07.388 DEBUG:boto:path=/ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.388 DEBUG:boto:auth_path=/ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.389 DEBUG:boto:Method: HEAD 2023-11-02T18:16:07.389 DEBUG:boto:Path: /ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.389 DEBUG:boto:Data: 2023-11-02T18:16:07.390 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.390 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.391 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.391 DEBUG:boto:Params: {} 2023-11-02T18:16:07.391 DEBUG:boto:Token: None 2023-11-02T18:16:07.392 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.392 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZKbHcgKueAyuoh5tWtPjGVzoeLk= 2023-11-02T18:16:07.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZKbHcgKueAyuoh5tWtPjGVzoeLk=', 'Content-Length': '0'} 2023-11-02T18:16:07.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000070ac19c8a30df8-006543e767-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.396 DEBUG:boto:path=/ibtljn-46/obj8 2023-11-02T18:16:07.397 DEBUG:boto:auth_path=/ibtljn-46/obj8 2023-11-02T18:16:07.397 DEBUG:boto:path=/ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.397 DEBUG:boto:auth_path=/ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.398 DEBUG:boto:Method: HEAD 2023-11-02T18:16:07.398 DEBUG:boto:Path: /ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.399 DEBUG:boto:Data: 2023-11-02T18:16:07.399 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.399 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.400 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.400 DEBUG:boto:Params: {} 2023-11-02T18:16:07.401 DEBUG:boto:Token: None 2023-11-02T18:16:07.401 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.401 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZKbHcgKueAyuoh5tWtPjGVzoeLk= 2023-11-02T18:16:07.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZKbHcgKueAyuoh5tWtPjGVzoeLk=', 'Content-Length': '0'} 2023-11-02T18:16:07.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064e2af91295fa2a3-006543e767-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-11-02T18:16:07.406 DEBUG:boto:path=/ibtljn-46/obj8 2023-11-02T18:16:07.406 DEBUG:boto:auth_path=/ibtljn-46/obj8 2023-11-02T18:16:07.407 DEBUG:boto:path=/ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.407 DEBUG:boto:auth_path=/ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.407 DEBUG:boto:Method: GET 2023-11-02T18:16:07.408 DEBUG:boto:Path: /ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.408 DEBUG:boto:Data: 2023-11-02T18:16:07.409 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.409 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.410 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.410 DEBUG:boto:Params: {} 2023-11-02T18:16:07.410 DEBUG:boto:Token: None 2023-11-02T18:16:07.411 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.411 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6NgaTYptEmN00WJUhrLs5YHcvME= 2023-11-02T18:16:07.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6NgaTYptEmN00WJUhrLs5YHcvME=', 'Content-Length': '0'} 2023-11-02T18:16:07.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce31dcf21e7d17f1-006543e767-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.458 DEBUG:boto:path=/ibtljn-46/obj8 2023-11-02T18:16:07.459 DEBUG:boto:auth_path=/ibtljn-46/obj8 2023-11-02T18:16:07.459 DEBUG:boto:path=/ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.460 DEBUG:boto:auth_path=/ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.460 DEBUG:boto:Method: GET 2023-11-02T18:16:07.461 DEBUG:boto:Path: /ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.461 DEBUG:boto:Data: 2023-11-02T18:16:07.461 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.462 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.463 DEBUG:boto:Params: {} 2023-11-02T18:16:07.463 DEBUG:boto:Token: None 2023-11-02T18:16:07.464 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj8?versionId=null 2023-11-02T18:16:07.464 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6NgaTYptEmN00WJUhrLs5YHcvME= 2023-11-02T18:16:07.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6NgaTYptEmN00WJUhrLs5YHcvME=', 'Content-Length': '0'} 2023-11-02T18:16:07.468 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002391d43567bf1e19-006543e767-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.468 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:07.469 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:07.469 DEBUG:boto:path=/ibtljn-46/?prefix=obj8 2023-11-02T18:16:07.470 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj8 2023-11-02T18:16:07.470 DEBUG:boto:Method: GET 2023-11-02T18:16:07.471 DEBUG:boto:Path: /ibtljn-46/?prefix=obj8 2023-11-02T18:16:07.471 DEBUG:boto:Data: 2023-11-02T18:16:07.472 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.472 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.472 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.473 DEBUG:boto:Params: {} 2023-11-02T18:16:07.473 DEBUG:boto:Token: None 2023-11-02T18:16:07.474 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/ 2023-11-02T18:16:07.474 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss= 2023-11-02T18:16:07.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss=', 'Content-Length': '0'} 2023-11-02T18:16:07.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ad0d64f5e0a67b3-006543e767-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.481 DEBUG:boto:b'ibtljn-46obj81000falseobj82023-11-02T18:15:24.778Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:07.482 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:07.482 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:07.483 DEBUG:boto:path=/ibtljn-46/?prefix=obj8 2023-11-02T18:16:07.483 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj8 2023-11-02T18:16:07.484 DEBUG:boto:Method: GET 2023-11-02T18:16:07.484 DEBUG:boto:Path: /ibtljn-46/?prefix=obj8 2023-11-02T18:16:07.484 DEBUG:boto:Data: 2023-11-02T18:16:07.485 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.485 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.486 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.486 DEBUG:boto:Params: {} 2023-11-02T18:16:07.487 DEBUG:boto:Token: None 2023-11-02T18:16:07.487 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/ 2023-11-02T18:16:07.488 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss= 2023-11-02T18:16:07.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss=', 'Content-Length': '0'} 2023-11-02T18:16:07.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041ff418f32566aa0-006543e767-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.534 DEBUG:boto:b'ibtljn-46obj81000falseobj82023-11-02T18:15:24.778Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:07.535 DEBUG:boto:path=/ibtljn-46/obj8 2023-11-02T18:16:07.536 DEBUG:boto:auth_path=/ibtljn-46/obj8 2023-11-02T18:16:07.536 DEBUG:boto:Method: HEAD 2023-11-02T18:16:07.536 DEBUG:boto:Path: /ibtljn-46/obj8 2023-11-02T18:16:07.537 DEBUG:boto:Data: 2023-11-02T18:16:07.537 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.538 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.538 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.539 DEBUG:boto:Params: {} 2023-11-02T18:16:07.539 DEBUG:boto:Token: None 2023-11-02T18:16:07.539 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj8 2023-11-02T18:16:07.540 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CC0aQjkncjp+diWDbXV72wqSM58= 2023-11-02T18:16:07.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CC0aQjkncjp+diWDbXV72wqSM58=', 'Content-Length': '0'} 2023-11-02T18:16:07.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a87e2385c0a64f1b-006543e767-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.543 DEBUG:boto:path=/ibtljn-46/obj8 2023-11-02T18:16:07.544 DEBUG:boto:auth_path=/ibtljn-46/obj8 2023-11-02T18:16:07.544 DEBUG:boto:Method: HEAD 2023-11-02T18:16:07.545 DEBUG:boto:Path: /ibtljn-46/obj8 2023-11-02T18:16:07.545 DEBUG:boto:Data: 2023-11-02T18:16:07.545 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.546 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.546 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.547 DEBUG:boto:Params: {} 2023-11-02T18:16:07.547 DEBUG:boto:Token: None 2023-11-02T18:16:07.548 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj8 2023-11-02T18:16:07.548 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CC0aQjkncjp+diWDbXV72wqSM58= 2023-11-02T18:16:07.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CC0aQjkncjp+diWDbXV72wqSM58=', 'Content-Length': '0'} 2023-11-02T18:16:07.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000727df3890c093e48-006543e767-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-11-02T18:16:07.552 DEBUG:boto:path=/ibtljn-46/obj8 2023-11-02T18:16:07.553 DEBUG:boto:auth_path=/ibtljn-46/obj8 2023-11-02T18:16:07.553 DEBUG:boto:Method: GET 2023-11-02T18:16:07.553 DEBUG:boto:Path: /ibtljn-46/obj8 2023-11-02T18:16:07.554 DEBUG:boto:Data: 2023-11-02T18:16:07.554 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.555 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.555 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.556 DEBUG:boto:Params: {} 2023-11-02T18:16:07.556 DEBUG:boto:Token: None 2023-11-02T18:16:07.556 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj8 2023-11-02T18:16:07.557 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jacCxW6C0aD6HGR++p4UyfApO8U= 2023-11-02T18:16:07.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jacCxW6C0aD6HGR++p4UyfApO8U=', 'Content-Length': '0'} 2023-11-02T18:16:07.560 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-46:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000461b2e052c692af3-006543e767-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.561 DEBUG:boto:path=/ibtljn-46/obj8 2023-11-02T18:16:07.562 DEBUG:boto:auth_path=/ibtljn-46/obj8 2023-11-02T18:16:07.562 DEBUG:boto:Method: GET 2023-11-02T18:16:07.562 DEBUG:boto:Path: /ibtljn-46/obj8 2023-11-02T18:16:07.563 DEBUG:boto:Data: 2023-11-02T18:16:07.563 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.564 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.565 DEBUG:boto:Params: {} 2023-11-02T18:16:07.565 DEBUG:boto:Token: None 2023-11-02T18:16:07.565 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/obj8 2023-11-02T18:16:07.566 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jacCxW6C0aD6HGR++p4UyfApO8U= 2023-11-02T18:16:07.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jacCxW6C0aD6HGR++p4UyfApO8U=', 'Content-Length': '0'} 2023-11-02T18:16:07.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ec25486f56f3de0-006543e767-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.571 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:07.571 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:07.572 DEBUG:boto:path=/ibtljn-46/?prefix=obj8 2023-11-02T18:16:07.572 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj8 2023-11-02T18:16:07.572 DEBUG:boto:Method: GET 2023-11-02T18:16:07.573 DEBUG:boto:Path: /ibtljn-46/?prefix=obj8 2023-11-02T18:16:07.573 DEBUG:boto:Data: 2023-11-02T18:16:07.574 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:07.575 DEBUG:boto:Port: 8001 2023-11-02T18:16:07.575 DEBUG:boto:Params: {} 2023-11-02T18:16:07.575 DEBUG:boto:Token: None 2023-11-02T18:16:07.576 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/ 2023-11-02T18:16:07.576 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss= 2023-11-02T18:16:07.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss=', 'Content-Length': '0'} 2023-11-02T18:16:07.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eeaa179c075e5b1-006543e767-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.622 DEBUG:boto:b'ibtljn-46obj81000falseobj82023-11-02T18:15:24.778Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:07.623 DEBUG:boto:path=/ibtljn-46/ 2023-11-02T18:16:07.624 DEBUG:boto:auth_path=/ibtljn-46/ 2023-11-02T18:16:07.624 DEBUG:boto:path=/ibtljn-46/?prefix=obj8 2023-11-02T18:16:07.625 DEBUG:boto:auth_path=/ibtljn-46/?prefix=obj8 2023-11-02T18:16:07.625 DEBUG:boto:Method: GET 2023-11-02T18:16:07.625 DEBUG:boto:Path: /ibtljn-46/?prefix=obj8 2023-11-02T18:16:07.626 DEBUG:boto:Data: 2023-11-02T18:16:07.626 DEBUG:boto:Headers: {} 2023-11-02T18:16:07.627 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:07.627 DEBUG:boto:Port: 8000 2023-11-02T18:16:07.628 DEBUG:boto:Params: {} 2023-11-02T18:16:07.628 DEBUG:boto:Token: None 2023-11-02T18:16:07.629 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:07 GMT /ibtljn-46/ 2023-11-02T18:16:07.629 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss= 2023-11-02T18:16:07.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3fj/pL+bCnKKyKKT/zibr1Zx1ss=', 'Content-Length': '0'} 2023-11-02T18:16:07.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a232318191da477e-006543e767-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:07.636 DEBUG:boto:b'ibtljn-46obj81000falseobj82023-11-02T18:15:24.778Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:07.636 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-46 zones={a2, a1} 2023-11-02T18:16:07.637 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-47 2023-11-02T18:16:07.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:16:07.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:16:07.638 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:16:07.689 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:16:07.689 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:16:07.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:07.749+0000 7f2193423f40 1 waiting to reach incremental sync.. 2023-11-02T18:16:12.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:12.749+0000 7f2193423f40 1 waiting to reach incremental sync.. 2023-11-02T18:16:17.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:17.749+0000 7f2193423f40 1 waiting to reach incremental sync.. 2023-11-02T18:16:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:22.754+0000 7f2193423f40 1 waiting to reach incremental sync.. 2023-11-02T18:16:27.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:27.754+0000 7f2193423f40 1 waiting to reach incremental sync.. 2023-11-02T18:16:32.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:32.754+0000 7f2193423f40 1 waiting to reach incremental sync.. 2023-11-02T18:16:37.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:37.754+0000 7f2193423f40 1 waiting to reach incremental sync.. 2023-11-02T18:16:42.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:42.758+0000 7f2193423f40 1 bucket sync caught up with source: 2023-11-02T18:16:42.768 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000004.21.6, 00000000002.25.1, 00000000004.27.6, 00000000002.17.1, 00000000004.30.6, 00000000002.37.1, 00000000002.27.1, 00000000004.36.6, 00000000002.23.1, 00000000002.50.1, 00000000002.16.1] 2023-11-02T18:16:42.769 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.21.6, 00000000002.25.1, 00000000004.27.6, 00000000002.17.1, 00000000004.30.6, 00000000002.37.1, 00000000002.27.1, 00000000004.36.6, 00000000002.23.1, 00000000002.50.1, 00000000002.16.1] 2023-11-02T18:16:42.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:42.758+0000 7f2193423f40 0 bucket checkpoint complete 2023-11-02T18:16:42.797 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-47 2023-11-02T18:16:42.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:16:42.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:16:42.799 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:16:42.852 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:16:42.852 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:16:42.918 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:42.910+0000 7f3cf2833f40 1 bucket sync caught up with source: 2023-11-02T18:16:42.918 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.218.6, , 00000000002.216.6, , 00000000002.69.6, , , 00000000002.193.6, , , ] 2023-11-02T18:16:42.919 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.218.6, , 00000000002.216.6, , 00000000002.69.6, , , 00000000002.193.6, , , ] 2023-11-02T18:16:42.919 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:42.910+0000 7f3cf2833f40 0 bucket checkpoint complete 2023-11-02T18:16:42.945 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-47 zones={a2, a1} 2023-11-02T18:16:42.946 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:42.946 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:42.947 DEBUG:boto:Method: HEAD 2023-11-02T18:16:42.947 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:16:42.948 DEBUG:boto:Data: 2023-11-02T18:16:42.948 DEBUG:boto:Headers: {} 2023-11-02T18:16:42.949 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:42.949 DEBUG:boto:Port: 8001 2023-11-02T18:16:42.949 DEBUG:boto:Params: {} 2023-11-02T18:16:42.950 DEBUG:boto:Token: None 2023-11-02T18:16:42.950 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:42 GMT /ibtljn-47/ 2023-11-02T18:16:42.951 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:74hfMD0FeyD7qJW3ytfavm9Ip40= 2023-11-02T18:16:42.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:74hfMD0FeyD7qJW3ytfavm9Ip40=', 'Content-Length': '0'} 2023-11-02T18:16:42.956 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000678e5ca8c892b4ab-006543e78a-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:42.957 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:42.957 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:42.958 DEBUG:boto:Method: HEAD 2023-11-02T18:16:42.958 DEBUG:boto:Path: /ibtljn-47/ 2023-11-02T18:16:42.959 DEBUG:boto:Data: 2023-11-02T18:16:42.959 DEBUG:boto:Headers: {} 2023-11-02T18:16:42.959 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:42.960 DEBUG:boto:Port: 8000 2023-11-02T18:16:42.960 DEBUG:boto:Params: {} 2023-11-02T18:16:42.961 DEBUG:boto:Token: None 2023-11-02T18:16:42.961 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:42 GMT /ibtljn-47/ 2023-11-02T18:16:42.962 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:74hfMD0FeyD7qJW3ytfavm9Ip40= 2023-11-02T18:16:42.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:74hfMD0FeyD7qJW3ytfavm9Ip40=', 'Content-Length': '0'} 2023-11-02T18:16:42.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a38e3e76911766b5-006543e78a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:42.966 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:16:42.967 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:42.967 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:42.968 DEBUG:boto:path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:16:42.968 DEBUG:boto:auth_path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:16:42.968 DEBUG:boto:Method: GET 2023-11-02T18:16:42.969 DEBUG:boto:Path: /ibtljn-47/?versions&max-keys=999 2023-11-02T18:16:42.969 DEBUG:boto:Data: 2023-11-02T18:16:42.970 DEBUG:boto:Headers: {} 2023-11-02T18:16:42.970 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:42.971 DEBUG:boto:Port: 8001 2023-11-02T18:16:42.971 DEBUG:boto:Params: {} 2023-11-02T18:16:42.972 DEBUG:boto:Token: None 2023-11-02T18:16:42.972 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:42 GMT /ibtljn-47/?versions 2023-11-02T18:16:42.972 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:easGyjxOq8vI8xayjDoDTUND90c= 2023-11-02T18:16:42.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:easGyjxOq8vI8xayjDoDTUND90c=', 'Content-Length': '0'} 2023-11-02T18:16:42.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c25d4561e991a08-006543e78a-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:42.978 DEBUG:boto:b'ibtljn-47999falseobj1nulltrue2023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:15:24.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:15:24.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:15:24.933Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:15:24.960Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:42.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:16:42.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:16:42.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:16:42.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:16:42.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-11-02T18:16:42.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-11-02T18:16:42.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-11-02T18:16:42.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-11-02T18:16:42.982 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:16:42.982 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:42.983 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:42.983 DEBUG:boto:path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:16:42.983 DEBUG:boto:auth_path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:16:42.984 DEBUG:boto:Method: GET 2023-11-02T18:16:42.984 DEBUG:boto:Path: /ibtljn-47/?versions&max-keys=999 2023-11-02T18:16:42.985 DEBUG:boto:Data: 2023-11-02T18:16:42.985 DEBUG:boto:Headers: {} 2023-11-02T18:16:42.986 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:42.986 DEBUG:boto:Port: 8000 2023-11-02T18:16:42.986 DEBUG:boto:Params: {} 2023-11-02T18:16:42.987 DEBUG:boto:Token: None 2023-11-02T18:16:42.987 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:42 GMT /ibtljn-47/?versions 2023-11-02T18:16:42.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:easGyjxOq8vI8xayjDoDTUND90c= 2023-11-02T18:16:42.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:easGyjxOq8vI8xayjDoDTUND90c=', 'Content-Length': '0'} 2023-11-02T18:16:42.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbe984c91076f0cf-006543e78a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:42.994 DEBUG:boto:b'ibtljn-47999falseobj1nulltrue2023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:15:24.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:15:24.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:15:24.933Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:15:24.960Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:42.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:16:42.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:16:42.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:16:42.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:16:42.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-11-02T18:16:42.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-11-02T18:16:42.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-11-02T18:16:42.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-11-02T18:16:42.999 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.000 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.000 DEBUG:boto:path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:16:43.001 DEBUG:boto:auth_path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:16:43.001 DEBUG:boto:Method: GET 2023-11-02T18:16:43.001 DEBUG:boto:Path: /ibtljn-47/?versions&max-keys=999 2023-11-02T18:16:43.002 DEBUG:boto:Data: 2023-11-02T18:16:43.002 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.003 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.003 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.004 DEBUG:boto:Params: {} 2023-11-02T18:16:43.004 DEBUG:boto:Token: None 2023-11-02T18:16:43.004 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/?versions 2023-11-02T18:16:43.005 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LDI62JJTs6MQJmkMylXgnhShBlk= 2023-11-02T18:16:43.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LDI62JJTs6MQJmkMylXgnhShBlk=', 'Content-Length': '0'} 2023-11-02T18:16:43.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f9aa7d346961fb4-006543e78a-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.013 DEBUG:boto:b'ibtljn-47999falseobj1nulltrue2023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:15:24.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:15:24.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:15:24.933Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:15:24.960Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.014 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.014 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.015 DEBUG:boto:path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:16:43.015 DEBUG:boto:auth_path=/ibtljn-47/?versions&max-keys=999 2023-11-02T18:16:43.016 DEBUG:boto:Method: GET 2023-11-02T18:16:43.016 DEBUG:boto:Path: /ibtljn-47/?versions&max-keys=999 2023-11-02T18:16:43.016 DEBUG:boto:Data: 2023-11-02T18:16:43.017 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.017 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.018 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.018 DEBUG:boto:Params: {} 2023-11-02T18:16:43.019 DEBUG:boto:Token: None 2023-11-02T18:16:43.019 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/?versions 2023-11-02T18:16:43.020 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LDI62JJTs6MQJmkMylXgnhShBlk= 2023-11-02T18:16:43.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LDI62JJTs6MQJmkMylXgnhShBlk=', 'Content-Length': '0'} 2023-11-02T18:16:43.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adc17a55bb89dca1-006543e78b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.066 DEBUG:boto:b'ibtljn-47999falseobj1nulltrue2023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:15:24.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:15:24.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:15:24.933Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:15:24.960Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:16:43.067 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:16:43.068 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:16:43.068 DEBUG:boto:path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.069 DEBUG:boto:auth_path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.069 DEBUG:boto:Method: GET 2023-11-02T18:16:43.070 DEBUG:boto:Path: /ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.070 DEBUG:boto:Data: 2023-11-02T18:16:43.071 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.071 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.071 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.072 DEBUG:boto:Params: {} 2023-11-02T18:16:43.072 DEBUG:boto:Token: None 2023-11-02T18:16:43.073 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.073 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bl6illc4EqtLGh35nC+dh+mXTr8= 2023-11-02T18:16:43.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bl6illc4EqtLGh35nC+dh+mXTr8=', 'Content-Length': '0'} 2023-11-02T18:16:43.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006e3d7d8955076cc-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.078 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:16:43.078 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:16:43.079 DEBUG:boto:path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.079 DEBUG:boto:auth_path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.080 DEBUG:boto:Method: GET 2023-11-02T18:16:43.080 DEBUG:boto:Path: /ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.081 DEBUG:boto:Data: 2023-11-02T18:16:43.081 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.082 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.082 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.083 DEBUG:boto:Params: {} 2023-11-02T18:16:43.083 DEBUG:boto:Token: None 2023-11-02T18:16:43.084 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.084 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bl6illc4EqtLGh35nC+dh+mXTr8= 2023-11-02T18:16:43.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bl6illc4EqtLGh35nC+dh+mXTr8=', 'Content-Length': '0'} 2023-11-02T18:16:43.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050e69807e27563e4-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.088 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:16:43.089 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:16:43.089 DEBUG:boto:path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.090 DEBUG:boto:auth_path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.090 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.091 DEBUG:boto:Path: /ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.091 DEBUG:boto:Data: 2023-11-02T18:16:43.091 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.092 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.092 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.092 DEBUG:boto:Params: {} 2023-11-02T18:16:43.093 DEBUG:boto:Token: None 2023-11-02T18:16:43.093 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0wYSCxKow093kUxYz+7dXavNzPk= 2023-11-02T18:16:43.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0wYSCxKow093kUxYz+7dXavNzPk=', 'Content-Length': '0'} 2023-11-02T18:16:43.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1e4eb13107a76bf-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.098 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:16:43.098 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:16:43.099 DEBUG:boto:path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.099 DEBUG:boto:auth_path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.100 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.100 DEBUG:boto:Path: /ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.101 DEBUG:boto:Data: 2023-11-02T18:16:43.102 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.102 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.103 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.103 DEBUG:boto:Params: {} 2023-11-02T18:16:43.104 DEBUG:boto:Token: None 2023-11-02T18:16:43.105 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.105 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0wYSCxKow093kUxYz+7dXavNzPk= 2023-11-02T18:16:43.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0wYSCxKow093kUxYz+7dXavNzPk=', 'Content-Length': '0'} 2023-11-02T18:16:43.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edd5a0a910fab1f1-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:16:43.111 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:16:43.111 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:16:43.112 DEBUG:boto:path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.112 DEBUG:boto:auth_path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.113 DEBUG:boto:Method: GET 2023-11-02T18:16:43.114 DEBUG:boto:Path: /ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.115 DEBUG:boto:Data: 2023-11-02T18:16:43.116 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.116 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.117 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.117 DEBUG:boto:Params: {} 2023-11-02T18:16:43.118 DEBUG:boto:Token: None 2023-11-02T18:16:43.119 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.119 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bl6illc4EqtLGh35nC+dh+mXTr8= 2023-11-02T18:16:43.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bl6illc4EqtLGh35nC+dh+mXTr8=', 'Content-Length': '0'} 2023-11-02T18:16:43.123 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ef12212b623a1d2-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.124 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:16:43.124 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:16:43.125 DEBUG:boto:path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.125 DEBUG:boto:auth_path=/ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.126 DEBUG:boto:Method: GET 2023-11-02T18:16:43.126 DEBUG:boto:Path: /ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.127 DEBUG:boto:Data: 2023-11-02T18:16:43.127 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.127 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.128 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.128 DEBUG:boto:Params: {} 2023-11-02T18:16:43.129 DEBUG:boto:Token: None 2023-11-02T18:16:43.129 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj1?versionId=null 2023-11-02T18:16:43.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bl6illc4EqtLGh35nC+dh+mXTr8= 2023-11-02T18:16:43.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bl6illc4EqtLGh35nC+dh+mXTr8=', 'Content-Length': '0'} 2023-11-02T18:16:43.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fd83628b837b9b3-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.134 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.134 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.135 DEBUG:boto:path=/ibtljn-47/?prefix=obj1 2023-11-02T18:16:43.135 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj1 2023-11-02T18:16:43.136 DEBUG:boto:Method: GET 2023-11-02T18:16:43.136 DEBUG:boto:Path: /ibtljn-47/?prefix=obj1 2023-11-02T18:16:43.137 DEBUG:boto:Data: 2023-11-02T18:16:43.137 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.137 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.138 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.138 DEBUG:boto:Params: {} 2023-11-02T18:16:43.139 DEBUG:boto:Token: None 2023-11-02T18:16:43.139 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/ 2023-11-02T18:16:43.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA= 2023-11-02T18:16:43.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA=', 'Content-Length': '0'} 2023-11-02T18:16:43.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b06423b3bb39ee81-006543e78b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.190 DEBUG:boto:b'ibtljn-47obj11000falseobj12023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.192 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.192 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.193 DEBUG:boto:path=/ibtljn-47/?prefix=obj1 2023-11-02T18:16:43.193 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj1 2023-11-02T18:16:43.194 DEBUG:boto:Method: GET 2023-11-02T18:16:43.194 DEBUG:boto:Path: /ibtljn-47/?prefix=obj1 2023-11-02T18:16:43.194 DEBUG:boto:Data: 2023-11-02T18:16:43.195 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.195 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.196 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.196 DEBUG:boto:Params: {} 2023-11-02T18:16:43.197 DEBUG:boto:Token: None 2023-11-02T18:16:43.197 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/ 2023-11-02T18:16:43.198 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA= 2023-11-02T18:16:43.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA=', 'Content-Length': '0'} 2023-11-02T18:16:43.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033fc9fc6ccc9bd25-006543e78b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.203 DEBUG:boto:b'ibtljn-47obj11000falseobj12023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.204 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:16:43.204 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:16:43.205 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.205 DEBUG:boto:Path: /ibtljn-47/obj1 2023-11-02T18:16:43.206 DEBUG:boto:Data: 2023-11-02T18:16:43.206 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.206 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.207 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.207 DEBUG:boto:Params: {} 2023-11-02T18:16:43.207 DEBUG:boto:Token: None 2023-11-02T18:16:43.208 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj1 2023-11-02T18:16:43.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNtL7GWGp6Wg3IC5LkZ5s1zs/+4= 2023-11-02T18:16:43.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNtL7GWGp6Wg3IC5LkZ5s1zs/+4=', 'Content-Length': '0'} 2023-11-02T18:16:43.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003860a7144170868d-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.212 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:16:43.213 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:16:43.213 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.214 DEBUG:boto:Path: /ibtljn-47/obj1 2023-11-02T18:16:43.214 DEBUG:boto:Data: 2023-11-02T18:16:43.214 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.215 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.215 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.216 DEBUG:boto:Params: {} 2023-11-02T18:16:43.216 DEBUG:boto:Token: None 2023-11-02T18:16:43.217 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj1 2023-11-02T18:16:43.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNtL7GWGp6Wg3IC5LkZ5s1zs/+4= 2023-11-02T18:16:43.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNtL7GWGp6Wg3IC5LkZ5s1zs/+4=', 'Content-Length': '0'} 2023-11-02T18:16:43.220 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbbd7995da0048d5-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:16:43.221 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:16:43.222 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:16:43.222 DEBUG:boto:Method: GET 2023-11-02T18:16:43.223 DEBUG:boto:Path: /ibtljn-47/obj1 2023-11-02T18:16:43.223 DEBUG:boto:Data: 2023-11-02T18:16:43.224 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.224 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.225 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.225 DEBUG:boto:Params: {} 2023-11-02T18:16:43.226 DEBUG:boto:Token: None 2023-11-02T18:16:43.226 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj1 2023-11-02T18:16:43.227 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1FQ0l2Z37bRkuTwvjt6Hlk8ryyw= 2023-11-02T18:16:43.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1FQ0l2Z37bRkuTwvjt6Hlk8ryyw=', 'Content-Length': '0'} 2023-11-02T18:16:43.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f7976db20efc46b-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.231 DEBUG:boto:path=/ibtljn-47/obj1 2023-11-02T18:16:43.231 DEBUG:boto:auth_path=/ibtljn-47/obj1 2023-11-02T18:16:43.232 DEBUG:boto:Method: GET 2023-11-02T18:16:43.232 DEBUG:boto:Path: /ibtljn-47/obj1 2023-11-02T18:16:43.233 DEBUG:boto:Data: 2023-11-02T18:16:43.233 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.234 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.234 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.234 DEBUG:boto:Params: {} 2023-11-02T18:16:43.235 DEBUG:boto:Token: None 2023-11-02T18:16:43.235 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj1 2023-11-02T18:16:43.236 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1FQ0l2Z37bRkuTwvjt6Hlk8ryyw= 2023-11-02T18:16:43.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1FQ0l2Z37bRkuTwvjt6Hlk8ryyw=', 'Content-Length': '0'} 2023-11-02T18:16:43.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9ad4e02fb077539-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.286 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.287 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.287 DEBUG:boto:path=/ibtljn-47/?prefix=obj1 2023-11-02T18:16:43.288 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj1 2023-11-02T18:16:43.288 DEBUG:boto:Method: GET 2023-11-02T18:16:43.289 DEBUG:boto:Path: /ibtljn-47/?prefix=obj1 2023-11-02T18:16:43.289 DEBUG:boto:Data: 2023-11-02T18:16:43.290 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.290 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.291 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.291 DEBUG:boto:Params: {} 2023-11-02T18:16:43.291 DEBUG:boto:Token: None 2023-11-02T18:16:43.292 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/ 2023-11-02T18:16:43.292 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA= 2023-11-02T18:16:43.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA=', 'Content-Length': '0'} 2023-11-02T18:16:43.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbb4f7d27d23bbc9-006543e78b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.298 DEBUG:boto:b'ibtljn-47obj11000falseobj12023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.299 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.300 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.300 DEBUG:boto:path=/ibtljn-47/?prefix=obj1 2023-11-02T18:16:43.301 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj1 2023-11-02T18:16:43.301 DEBUG:boto:Method: GET 2023-11-02T18:16:43.302 DEBUG:boto:Path: /ibtljn-47/?prefix=obj1 2023-11-02T18:16:43.302 DEBUG:boto:Data: 2023-11-02T18:16:43.302 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.303 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.304 DEBUG:boto:Params: {} 2023-11-02T18:16:43.304 DEBUG:boto:Token: None 2023-11-02T18:16:43.305 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/ 2023-11-02T18:16:43.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA= 2023-11-02T18:16:43.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA=', 'Content-Length': '0'} 2023-11-02T18:16:43.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf2ff9f6f0cb0b68-006543e78b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.312 DEBUG:boto:b'ibtljn-47obj11000falseobj12023-11-02T18:13:45.549Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:16:43.315 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:16:43.315 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:16:43.316 DEBUG:boto:path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.316 DEBUG:boto:auth_path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.317 DEBUG:boto:Method: GET 2023-11-02T18:16:43.317 DEBUG:boto:Path: /ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.318 DEBUG:boto:Data: 2023-11-02T18:16:43.318 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.319 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.319 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.320 DEBUG:boto:Params: {} 2023-11-02T18:16:43.320 DEBUG:boto:Token: None 2023-11-02T18:16:43.321 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.321 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ldr8gaDVNBbKIYDTFCHfuPiD9So= 2023-11-02T18:16:43.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ldr8gaDVNBbKIYDTFCHfuPiD9So=', 'Content-Length': '0'} 2023-11-02T18:16:43.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d195a4402869bce-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.366 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:16:43.367 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:16:43.367 DEBUG:boto:path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.368 DEBUG:boto:auth_path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.368 DEBUG:boto:Method: GET 2023-11-02T18:16:43.369 DEBUG:boto:Path: /ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.369 DEBUG:boto:Data: 2023-11-02T18:16:43.370 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.370 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.371 DEBUG:boto:Params: {} 2023-11-02T18:16:43.371 DEBUG:boto:Token: None 2023-11-02T18:16:43.372 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ldr8gaDVNBbKIYDTFCHfuPiD9So= 2023-11-02T18:16:43.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ldr8gaDVNBbKIYDTFCHfuPiD9So=', 'Content-Length': '0'} 2023-11-02T18:16:43.376 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cd016472f8f2180-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.376 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:16:43.377 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:16:43.377 DEBUG:boto:path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.378 DEBUG:boto:auth_path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.378 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.378 DEBUG:boto:Path: /ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.379 DEBUG:boto:Data: 2023-11-02T18:16:43.379 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.379 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.380 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.380 DEBUG:boto:Params: {} 2023-11-02T18:16:43.381 DEBUG:boto:Token: None 2023-11-02T18:16:43.381 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.382 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5+ivbkoBKYHBk9cVC0Vp3H9Mw70= 2023-11-02T18:16:43.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5+ivbkoBKYHBk9cVC0Vp3H9Mw70=', 'Content-Length': '0'} 2023-11-02T18:16:43.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ec644563a491a41-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.386 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:16:43.386 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:16:43.387 DEBUG:boto:path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.387 DEBUG:boto:auth_path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.387 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.388 DEBUG:boto:Path: /ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.388 DEBUG:boto:Data: 2023-11-02T18:16:43.389 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.389 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.389 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.390 DEBUG:boto:Params: {} 2023-11-02T18:16:43.390 DEBUG:boto:Token: None 2023-11-02T18:16:43.391 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.391 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5+ivbkoBKYHBk9cVC0Vp3H9Mw70= 2023-11-02T18:16:43.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5+ivbkoBKYHBk9cVC0Vp3H9Mw70=', 'Content-Length': '0'} 2023-11-02T18:16:43.395 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f4c1754fbf1ac82-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:16:43.396 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:16:43.396 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:16:43.397 DEBUG:boto:path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.397 DEBUG:boto:auth_path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.397 DEBUG:boto:Method: GET 2023-11-02T18:16:43.398 DEBUG:boto:Path: /ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.398 DEBUG:boto:Data: 2023-11-02T18:16:43.399 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.399 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.399 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.400 DEBUG:boto:Params: {} 2023-11-02T18:16:43.400 DEBUG:boto:Token: None 2023-11-02T18:16:43.401 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.401 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ldr8gaDVNBbKIYDTFCHfuPiD9So= 2023-11-02T18:16:43.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ldr8gaDVNBbKIYDTFCHfuPiD9So=', 'Content-Length': '0'} 2023-11-02T18:16:43.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7e37f97b210c5d0-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.405 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:16:43.406 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:16:43.406 DEBUG:boto:path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.407 DEBUG:boto:auth_path=/ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.407 DEBUG:boto:Method: GET 2023-11-02T18:16:43.408 DEBUG:boto:Path: /ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.409 DEBUG:boto:Data: 2023-11-02T18:16:43.409 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.410 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.410 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.411 DEBUG:boto:Params: {} 2023-11-02T18:16:43.411 DEBUG:boto:Token: None 2023-11-02T18:16:43.411 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj2?versionId=null 2023-11-02T18:16:43.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ldr8gaDVNBbKIYDTFCHfuPiD9So= 2023-11-02T18:16:43.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ldr8gaDVNBbKIYDTFCHfuPiD9So=', 'Content-Length': '0'} 2023-11-02T18:16:43.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019dd3dafb085410e-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.462 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.463 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.464 DEBUG:boto:path=/ibtljn-47/?prefix=obj2 2023-11-02T18:16:43.464 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj2 2023-11-02T18:16:43.464 DEBUG:boto:Method: GET 2023-11-02T18:16:43.465 DEBUG:boto:Path: /ibtljn-47/?prefix=obj2 2023-11-02T18:16:43.465 DEBUG:boto:Data: 2023-11-02T18:16:43.466 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.466 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.467 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.467 DEBUG:boto:Params: {} 2023-11-02T18:16:43.468 DEBUG:boto:Token: None 2023-11-02T18:16:43.468 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/ 2023-11-02T18:16:43.469 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA= 2023-11-02T18:16:43.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA=', 'Content-Length': '0'} 2023-11-02T18:16:43.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f7b974fb828c39a-006543e78b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.475 DEBUG:boto:b'ibtljn-47obj21000falseobj22023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.475 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.476 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.476 DEBUG:boto:path=/ibtljn-47/?prefix=obj2 2023-11-02T18:16:43.477 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj2 2023-11-02T18:16:43.477 DEBUG:boto:Method: GET 2023-11-02T18:16:43.478 DEBUG:boto:Path: /ibtljn-47/?prefix=obj2 2023-11-02T18:16:43.478 DEBUG:boto:Data: 2023-11-02T18:16:43.478 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.479 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.479 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.480 DEBUG:boto:Params: {} 2023-11-02T18:16:43.480 DEBUG:boto:Token: None 2023-11-02T18:16:43.481 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/ 2023-11-02T18:16:43.481 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA= 2023-11-02T18:16:43.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA=', 'Content-Length': '0'} 2023-11-02T18:16:43.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066eae74cf444f511-006543e78b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.487 DEBUG:boto:b'ibtljn-47obj21000falseobj22023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.488 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:16:43.488 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:16:43.489 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.489 DEBUG:boto:Path: /ibtljn-47/obj2 2023-11-02T18:16:43.490 DEBUG:boto:Data: 2023-11-02T18:16:43.490 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.491 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.491 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.491 DEBUG:boto:Params: {} 2023-11-02T18:16:43.492 DEBUG:boto:Token: None 2023-11-02T18:16:43.492 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj2 2023-11-02T18:16:43.493 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TvUqkua5snVYLydh4jSlvABs1T0= 2023-11-02T18:16:43.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TvUqkua5snVYLydh4jSlvABs1T0=', 'Content-Length': '0'} 2023-11-02T18:16:43.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de2a87147b6dfdcc-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.497 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:16:43.497 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:16:43.498 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.498 DEBUG:boto:Path: /ibtljn-47/obj2 2023-11-02T18:16:43.498 DEBUG:boto:Data: 2023-11-02T18:16:43.499 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.499 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.500 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.500 DEBUG:boto:Params: {} 2023-11-02T18:16:43.500 DEBUG:boto:Token: None 2023-11-02T18:16:43.501 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj2 2023-11-02T18:16:43.501 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TvUqkua5snVYLydh4jSlvABs1T0= 2023-11-02T18:16:43.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TvUqkua5snVYLydh4jSlvABs1T0=', 'Content-Length': '0'} 2023-11-02T18:16:43.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d7531e8dc6af7ce-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:16:43.506 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:16:43.506 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:16:43.507 DEBUG:boto:Method: GET 2023-11-02T18:16:43.507 DEBUG:boto:Path: /ibtljn-47/obj2 2023-11-02T18:16:43.507 DEBUG:boto:Data: 2023-11-02T18:16:43.508 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.508 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.509 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.509 DEBUG:boto:Params: {} 2023-11-02T18:16:43.510 DEBUG:boto:Token: None 2023-11-02T18:16:43.510 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj2 2023-11-02T18:16:43.511 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:so9UmovAYuUV/T0T8xBGM+XCS4Q= 2023-11-02T18:16:43.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:so9UmovAYuUV/T0T8xBGM+XCS4Q=', 'Content-Length': '0'} 2023-11-02T18:16:43.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001022ec4288fea4ec-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.554 DEBUG:boto:path=/ibtljn-47/obj2 2023-11-02T18:16:43.555 DEBUG:boto:auth_path=/ibtljn-47/obj2 2023-11-02T18:16:43.556 DEBUG:boto:Method: GET 2023-11-02T18:16:43.556 DEBUG:boto:Path: /ibtljn-47/obj2 2023-11-02T18:16:43.556 DEBUG:boto:Data: 2023-11-02T18:16:43.557 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.557 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.558 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.558 DEBUG:boto:Params: {} 2023-11-02T18:16:43.559 DEBUG:boto:Token: None 2023-11-02T18:16:43.559 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj2 2023-11-02T18:16:43.560 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:so9UmovAYuUV/T0T8xBGM+XCS4Q= 2023-11-02T18:16:43.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:so9UmovAYuUV/T0T8xBGM+XCS4Q=', 'Content-Length': '0'} 2023-11-02T18:16:43.563 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000041bcdd26a5c3a90-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.563 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.564 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.564 DEBUG:boto:path=/ibtljn-47/?prefix=obj2 2023-11-02T18:16:43.565 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj2 2023-11-02T18:16:43.565 DEBUG:boto:Method: GET 2023-11-02T18:16:43.566 DEBUG:boto:Path: /ibtljn-47/?prefix=obj2 2023-11-02T18:16:43.566 DEBUG:boto:Data: 2023-11-02T18:16:43.567 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.567 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.567 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.568 DEBUG:boto:Params: {} 2023-11-02T18:16:43.568 DEBUG:boto:Token: None 2023-11-02T18:16:43.569 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/ 2023-11-02T18:16:43.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA= 2023-11-02T18:16:43.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA=', 'Content-Length': '0'} 2023-11-02T18:16:43.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b632fc0141b46503-006543e78b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.574 DEBUG:boto:b'ibtljn-47obj21000falseobj22023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.575 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.576 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.576 DEBUG:boto:path=/ibtljn-47/?prefix=obj2 2023-11-02T18:16:43.576 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj2 2023-11-02T18:16:43.577 DEBUG:boto:Method: GET 2023-11-02T18:16:43.577 DEBUG:boto:Path: /ibtljn-47/?prefix=obj2 2023-11-02T18:16:43.578 DEBUG:boto:Data: 2023-11-02T18:16:43.578 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.579 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.579 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.580 DEBUG:boto:Params: {} 2023-11-02T18:16:43.580 DEBUG:boto:Token: None 2023-11-02T18:16:43.581 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/ 2023-11-02T18:16:43.581 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA= 2023-11-02T18:16:43.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA=', 'Content-Length': '0'} 2023-11-02T18:16:43.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dc4b62a24157c81-006543e78b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.626 DEBUG:boto:b'ibtljn-47obj21000falseobj22023-11-02T18:13:45.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:16:43.627 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:16:43.628 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:16:43.628 DEBUG:boto:path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.629 DEBUG:boto:auth_path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.629 DEBUG:boto:Method: GET 2023-11-02T18:16:43.630 DEBUG:boto:Path: /ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.630 DEBUG:boto:Data: 2023-11-02T18:16:43.631 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.631 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.632 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.632 DEBUG:boto:Params: {} 2023-11-02T18:16:43.633 DEBUG:boto:Token: None 2023-11-02T18:16:43.633 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.634 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MeMTpf3qbgPrc+MQlmJ6woixVZE= 2023-11-02T18:16:43.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MeMTpf3qbgPrc+MQlmJ6woixVZE=', 'Content-Length': '0'} 2023-11-02T18:16:43.637 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042724f8ab3c02849-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.638 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:16:43.639 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:16:43.639 DEBUG:boto:path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.640 DEBUG:boto:auth_path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.640 DEBUG:boto:Method: GET 2023-11-02T18:16:43.641 DEBUG:boto:Path: /ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.641 DEBUG:boto:Data: 2023-11-02T18:16:43.642 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.642 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.648 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.648 DEBUG:boto:Params: {} 2023-11-02T18:16:43.649 DEBUG:boto:Token: None 2023-11-02T18:16:43.649 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.650 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MeMTpf3qbgPrc+MQlmJ6woixVZE= 2023-11-02T18:16:43.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MeMTpf3qbgPrc+MQlmJ6woixVZE=', 'Content-Length': '0'} 2023-11-02T18:16:43.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e9ef79272a44b60-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.654 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:16:43.655 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:16:43.655 DEBUG:boto:path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.656 DEBUG:boto:auth_path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.656 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.657 DEBUG:boto:Path: /ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.657 DEBUG:boto:Data: 2023-11-02T18:16:43.658 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.658 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.659 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.659 DEBUG:boto:Params: {} 2023-11-02T18:16:43.660 DEBUG:boto:Token: None 2023-11-02T18:16:43.660 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.661 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ciQCCr8w7AXHtEXoo9NdtZga070= 2023-11-02T18:16:43.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ciQCCr8w7AXHtEXoo9NdtZga070=', 'Content-Length': '0'} 2023-11-02T18:16:43.664 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000da6aa6dbb639bcd-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.665 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:16:43.665 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:16:43.666 DEBUG:boto:path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.666 DEBUG:boto:auth_path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.667 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.667 DEBUG:boto:Path: /ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.668 DEBUG:boto:Data: 2023-11-02T18:16:43.668 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.668 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.669 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.669 DEBUG:boto:Params: {} 2023-11-02T18:16:43.670 DEBUG:boto:Token: None 2023-11-02T18:16:43.670 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.671 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ciQCCr8w7AXHtEXoo9NdtZga070= 2023-11-02T18:16:43.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ciQCCr8w7AXHtEXoo9NdtZga070=', 'Content-Length': '0'} 2023-11-02T18:16:43.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f43cc005afef404-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:16:43.676 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:16:43.676 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:16:43.677 DEBUG:boto:path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.677 DEBUG:boto:auth_path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.678 DEBUG:boto:Method: GET 2023-11-02T18:16:43.678 DEBUG:boto:Path: /ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.679 DEBUG:boto:Data: 2023-11-02T18:16:43.679 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.680 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.680 DEBUG:boto:Params: {} 2023-11-02T18:16:43.681 DEBUG:boto:Token: None 2023-11-02T18:16:43.681 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.682 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MeMTpf3qbgPrc+MQlmJ6woixVZE= 2023-11-02T18:16:43.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MeMTpf3qbgPrc+MQlmJ6woixVZE=', 'Content-Length': '0'} 2023-11-02T18:16:43.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055aae8cfb88b1a98-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.686 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:16:43.687 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:16:43.687 DEBUG:boto:path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.687 DEBUG:boto:auth_path=/ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.688 DEBUG:boto:Method: GET 2023-11-02T18:16:43.688 DEBUG:boto:Path: /ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.689 DEBUG:boto:Data: 2023-11-02T18:16:43.689 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.690 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.690 DEBUG:boto:Params: {} 2023-11-02T18:16:43.691 DEBUG:boto:Token: None 2023-11-02T18:16:43.691 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj3?versionId=null 2023-11-02T18:16:43.692 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MeMTpf3qbgPrc+MQlmJ6woixVZE= 2023-11-02T18:16:43.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MeMTpf3qbgPrc+MQlmJ6woixVZE=', 'Content-Length': '0'} 2023-11-02T18:16:43.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dde100f8c68d270e-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.696 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.697 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.697 DEBUG:boto:path=/ibtljn-47/?prefix=obj3 2023-11-02T18:16:43.697 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj3 2023-11-02T18:16:43.698 DEBUG:boto:Method: GET 2023-11-02T18:16:43.698 DEBUG:boto:Path: /ibtljn-47/?prefix=obj3 2023-11-02T18:16:43.699 DEBUG:boto:Data: 2023-11-02T18:16:43.699 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.700 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.701 DEBUG:boto:Params: {} 2023-11-02T18:16:43.701 DEBUG:boto:Token: None 2023-11-02T18:16:43.702 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/ 2023-11-02T18:16:43.702 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA= 2023-11-02T18:16:43.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA=', 'Content-Length': '0'} 2023-11-02T18:16:43.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b0f6766b45efcc6-006543e78b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.708 DEBUG:boto:b'ibtljn-47obj31000falseobj32023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.709 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.709 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.710 DEBUG:boto:path=/ibtljn-47/?prefix=obj3 2023-11-02T18:16:43.710 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj3 2023-11-02T18:16:43.711 DEBUG:boto:Method: GET 2023-11-02T18:16:43.711 DEBUG:boto:Path: /ibtljn-47/?prefix=obj3 2023-11-02T18:16:43.711 DEBUG:boto:Data: 2023-11-02T18:16:43.712 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.712 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.712 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.713 DEBUG:boto:Params: {} 2023-11-02T18:16:43.713 DEBUG:boto:Token: None 2023-11-02T18:16:43.714 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/ 2023-11-02T18:16:43.714 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA= 2023-11-02T18:16:43.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA=', 'Content-Length': '0'} 2023-11-02T18:16:43.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029b1f7301eefb1f0-006543e78b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.762 DEBUG:boto:b'ibtljn-47obj31000falseobj32023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.763 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:16:43.763 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:16:43.764 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.764 DEBUG:boto:Path: /ibtljn-47/obj3 2023-11-02T18:16:43.765 DEBUG:boto:Data: 2023-11-02T18:16:43.765 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.765 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.766 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.766 DEBUG:boto:Params: {} 2023-11-02T18:16:43.767 DEBUG:boto:Token: None 2023-11-02T18:16:43.767 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj3 2023-11-02T18:16:43.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5Xtt97/x02alc2T0IE82cMeEnUU= 2023-11-02T18:16:43.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5Xtt97/x02alc2T0IE82cMeEnUU=', 'Content-Length': '0'} 2023-11-02T18:16:43.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002575142f3d2c8c4d-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.772 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:16:43.772 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:16:43.773 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.773 DEBUG:boto:Path: /ibtljn-47/obj3 2023-11-02T18:16:43.774 DEBUG:boto:Data: 2023-11-02T18:16:43.774 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.774 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.775 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.775 DEBUG:boto:Params: {} 2023-11-02T18:16:43.776 DEBUG:boto:Token: None 2023-11-02T18:16:43.776 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj3 2023-11-02T18:16:43.777 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5Xtt97/x02alc2T0IE82cMeEnUU= 2023-11-02T18:16:43.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5Xtt97/x02alc2T0IE82cMeEnUU=', 'Content-Length': '0'} 2023-11-02T18:16:43.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002465a5f5894f4218-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:16:43.781 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:16:43.781 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:16:43.782 DEBUG:boto:Method: GET 2023-11-02T18:16:43.782 DEBUG:boto:Path: /ibtljn-47/obj3 2023-11-02T18:16:43.783 DEBUG:boto:Data: 2023-11-02T18:16:43.783 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.784 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.784 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.784 DEBUG:boto:Params: {} 2023-11-02T18:16:43.785 DEBUG:boto:Token: None 2023-11-02T18:16:43.785 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj3 2023-11-02T18:16:43.786 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:khFFEbZcXCyvgvaQf4K89pa1WLQ= 2023-11-02T18:16:43.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:khFFEbZcXCyvgvaQf4K89pa1WLQ=', 'Content-Length': '0'} 2023-11-02T18:16:43.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063701ef69e42c4fe-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.830 DEBUG:boto:path=/ibtljn-47/obj3 2023-11-02T18:16:43.831 DEBUG:boto:auth_path=/ibtljn-47/obj3 2023-11-02T18:16:43.831 DEBUG:boto:Method: GET 2023-11-02T18:16:43.832 DEBUG:boto:Path: /ibtljn-47/obj3 2023-11-02T18:16:43.832 DEBUG:boto:Data: 2023-11-02T18:16:43.833 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.833 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.834 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.834 DEBUG:boto:Params: {} 2023-11-02T18:16:43.835 DEBUG:boto:Token: None 2023-11-02T18:16:43.835 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj3 2023-11-02T18:16:43.836 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:khFFEbZcXCyvgvaQf4K89pa1WLQ= 2023-11-02T18:16:43.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:khFFEbZcXCyvgvaQf4K89pa1WLQ=', 'Content-Length': '0'} 2023-11-02T18:16:43.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dded37f30a373b04-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.840 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.840 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.841 DEBUG:boto:path=/ibtljn-47/?prefix=obj3 2023-11-02T18:16:43.841 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj3 2023-11-02T18:16:43.842 DEBUG:boto:Method: GET 2023-11-02T18:16:43.842 DEBUG:boto:Path: /ibtljn-47/?prefix=obj3 2023-11-02T18:16:43.843 DEBUG:boto:Data: 2023-11-02T18:16:43.843 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.843 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.844 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.844 DEBUG:boto:Params: {} 2023-11-02T18:16:43.845 DEBUG:boto:Token: None 2023-11-02T18:16:43.845 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/ 2023-11-02T18:16:43.846 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA= 2023-11-02T18:16:43.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA=', 'Content-Length': '0'} 2023-11-02T18:16:43.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046676045c026b366-006543e78b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.852 DEBUG:boto:b'ibtljn-47obj31000falseobj32023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.852 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:43.853 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:43.853 DEBUG:boto:path=/ibtljn-47/?prefix=obj3 2023-11-02T18:16:43.854 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj3 2023-11-02T18:16:43.854 DEBUG:boto:Method: GET 2023-11-02T18:16:43.855 DEBUG:boto:Path: /ibtljn-47/?prefix=obj3 2023-11-02T18:16:43.855 DEBUG:boto:Data: 2023-11-02T18:16:43.855 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.856 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.856 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.857 DEBUG:boto:Params: {} 2023-11-02T18:16:43.857 DEBUG:boto:Token: None 2023-11-02T18:16:43.858 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/ 2023-11-02T18:16:43.858 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA= 2023-11-02T18:16:43.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qYecatJETQMZeK1SHK4mGIDMchA=', 'Content-Length': '0'} 2023-11-02T18:16:43.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ddc32044f3f2314-006543e78b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.864 DEBUG:boto:b'ibtljn-47obj31000falseobj32023-11-02T18:13:45.597Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:43.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:16:43.865 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:16:43.866 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:16:43.866 DEBUG:boto:path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.867 DEBUG:boto:auth_path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.867 DEBUG:boto:Method: GET 2023-11-02T18:16:43.868 DEBUG:boto:Path: /ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.868 DEBUG:boto:Data: 2023-11-02T18:16:43.868 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.869 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.869 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.870 DEBUG:boto:Params: {} 2023-11-02T18:16:43.870 DEBUG:boto:Token: None 2023-11-02T18:16:43.871 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.871 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l0+eDsRHSI2nYydjaEJZrif4vmg= 2023-11-02T18:16:43.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l0+eDsRHSI2nYydjaEJZrif4vmg=', 'Content-Length': '0'} 2023-11-02T18:16:43.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4824043148c46d5-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.875 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:16:43.876 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:16:43.876 DEBUG:boto:path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.877 DEBUG:boto:auth_path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.877 DEBUG:boto:Method: GET 2023-11-02T18:16:43.877 DEBUG:boto:Path: /ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.878 DEBUG:boto:Data: 2023-11-02T18:16:43.878 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.879 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.879 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.880 DEBUG:boto:Params: {} 2023-11-02T18:16:43.880 DEBUG:boto:Token: None 2023-11-02T18:16:43.881 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.881 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l0+eDsRHSI2nYydjaEJZrif4vmg= 2023-11-02T18:16:43.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l0+eDsRHSI2nYydjaEJZrif4vmg=', 'Content-Length': '0'} 2023-11-02T18:16:43.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000738373cd28431efc-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.930 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:16:43.931 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:16:43.931 DEBUG:boto:path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.932 DEBUG:boto:auth_path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.932 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.933 DEBUG:boto:Path: /ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.933 DEBUG:boto:Data: 2023-11-02T18:16:43.933 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.934 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.935 DEBUG:boto:Params: {} 2023-11-02T18:16:43.935 DEBUG:boto:Token: None 2023-11-02T18:16:43.936 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.936 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NFbh+zhrDDMIcJAv7ABJBCN3XYA= 2023-11-02T18:16:43.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NFbh+zhrDDMIcJAv7ABJBCN3XYA=', 'Content-Length': '0'} 2023-11-02T18:16:43.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008de98a478746ec46-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.940 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:16:43.941 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:16:43.941 DEBUG:boto:path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.941 DEBUG:boto:auth_path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.942 DEBUG:boto:Method: HEAD 2023-11-02T18:16:43.942 DEBUG:boto:Path: /ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.943 DEBUG:boto:Data: 2023-11-02T18:16:43.943 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.943 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:43.944 DEBUG:boto:Port: 8000 2023-11-02T18:16:43.944 DEBUG:boto:Params: {} 2023-11-02T18:16:43.945 DEBUG:boto:Token: None 2023-11-02T18:16:43.945 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.945 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NFbh+zhrDDMIcJAv7ABJBCN3XYA= 2023-11-02T18:16:43.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NFbh+zhrDDMIcJAv7ABJBCN3XYA=', 'Content-Length': '0'} 2023-11-02T18:16:43.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c4fbbe5dc3350c3-006543e78b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:43.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:16:43.950 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:16:43.951 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:16:43.951 DEBUG:boto:path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.952 DEBUG:boto:auth_path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.952 DEBUG:boto:Method: GET 2023-11-02T18:16:43.952 DEBUG:boto:Path: /ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.953 DEBUG:boto:Data: 2023-11-02T18:16:43.953 DEBUG:boto:Headers: {} 2023-11-02T18:16:43.954 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:43.954 DEBUG:boto:Port: 8001 2023-11-02T18:16:43.955 DEBUG:boto:Params: {} 2023-11-02T18:16:43.955 DEBUG:boto:Token: None 2023-11-02T18:16:43.955 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:43 GMT /ibtljn-47/obj4?versionId=null 2023-11-02T18:16:43.956 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l0+eDsRHSI2nYydjaEJZrif4vmg= 2023-11-02T18:16:43.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l0+eDsRHSI2nYydjaEJZrif4vmg=', 'Content-Length': '0'} 2023-11-02T18:16:43.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9d2aa08a3eadf76-006543e78b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.002 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:16:44.003 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:16:44.003 DEBUG:boto:path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:16:44.004 DEBUG:boto:auth_path=/ibtljn-47/obj4?versionId=null 2023-11-02T18:16:44.004 DEBUG:boto:Method: GET 2023-11-02T18:16:44.005 DEBUG:boto:Path: /ibtljn-47/obj4?versionId=null 2023-11-02T18:16:44.005 DEBUG:boto:Data: 2023-11-02T18:16:44.006 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.006 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.006 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.007 DEBUG:boto:Params: {} 2023-11-02T18:16:44.007 DEBUG:boto:Token: None 2023-11-02T18:16:44.008 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj4?versionId=null 2023-11-02T18:16:44.009 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FUHQcio6MfUZPOQOBPXTR5sz2nU= 2023-11-02T18:16:44.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FUHQcio6MfUZPOQOBPXTR5sz2nU=', 'Content-Length': '0'} 2023-11-02T18:16:44.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b83fefd77a3964f9-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.013 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.014 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.014 DEBUG:boto:path=/ibtljn-47/?prefix=obj4 2023-11-02T18:16:44.015 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj4 2023-11-02T18:16:44.015 DEBUG:boto:Method: GET 2023-11-02T18:16:44.016 DEBUG:boto:Path: /ibtljn-47/?prefix=obj4 2023-11-02T18:16:44.016 DEBUG:boto:Data: 2023-11-02T18:16:44.017 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.017 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.017 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.018 DEBUG:boto:Params: {} 2023-11-02T18:16:44.019 DEBUG:boto:Token: None 2023-11-02T18:16:44.019 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.020 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001608d6f0648b8b45-006543e78c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.025 DEBUG:boto:b'ibtljn-47obj41000falseobj42023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.026 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.026 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.026 DEBUG:boto:path=/ibtljn-47/?prefix=obj4 2023-11-02T18:16:44.027 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj4 2023-11-02T18:16:44.027 DEBUG:boto:Method: GET 2023-11-02T18:16:44.028 DEBUG:boto:Path: /ibtljn-47/?prefix=obj4 2023-11-02T18:16:44.028 DEBUG:boto:Data: 2023-11-02T18:16:44.029 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.029 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.030 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.030 DEBUG:boto:Params: {} 2023-11-02T18:16:44.031 DEBUG:boto:Token: None 2023-11-02T18:16:44.031 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.031 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028e8e8388228b717-006543e78c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.038 DEBUG:boto:b'ibtljn-47obj41000falseobj42023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.038 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:16:44.039 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:16:44.039 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.040 DEBUG:boto:Path: /ibtljn-47/obj4 2023-11-02T18:16:44.040 DEBUG:boto:Data: 2023-11-02T18:16:44.041 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.041 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.041 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.042 DEBUG:boto:Params: {} 2023-11-02T18:16:44.042 DEBUG:boto:Token: None 2023-11-02T18:16:44.043 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj4 2023-11-02T18:16:44.043 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I66Cv2nvSZPkl4WI3NIoJgA09OA= 2023-11-02T18:16:44.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I66Cv2nvSZPkl4WI3NIoJgA09OA=', 'Content-Length': '0'} 2023-11-02T18:16:44.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000855f11bd33912bfa-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.047 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:16:44.047 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:16:44.048 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.048 DEBUG:boto:Path: /ibtljn-47/obj4 2023-11-02T18:16:44.049 DEBUG:boto:Data: 2023-11-02T18:16:44.049 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.050 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.051 DEBUG:boto:Params: {} 2023-11-02T18:16:44.051 DEBUG:boto:Token: None 2023-11-02T18:16:44.052 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj4 2023-11-02T18:16:44.052 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I66Cv2nvSZPkl4WI3NIoJgA09OA= 2023-11-02T18:16:44.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I66Cv2nvSZPkl4WI3NIoJgA09OA=', 'Content-Length': '0'} 2023-11-02T18:16:44.056 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e10ad119962436d7-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:16:44.057 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:16:44.057 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:16:44.058 DEBUG:boto:Method: GET 2023-11-02T18:16:44.058 DEBUG:boto:Path: /ibtljn-47/obj4 2023-11-02T18:16:44.059 DEBUG:boto:Data: 2023-11-02T18:16:44.059 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.060 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.060 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.061 DEBUG:boto:Params: {} 2023-11-02T18:16:44.061 DEBUG:boto:Token: None 2023-11-02T18:16:44.062 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj4 2023-11-02T18:16:44.062 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KK/wbBNAl+w9wUseKeCy0Z3xnGM= 2023-11-02T18:16:44.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KK/wbBNAl+w9wUseKeCy0Z3xnGM=', 'Content-Length': '0'} 2023-11-02T18:16:44.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084f4c9a931f619c5-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.106 DEBUG:boto:path=/ibtljn-47/obj4 2023-11-02T18:16:44.107 DEBUG:boto:auth_path=/ibtljn-47/obj4 2023-11-02T18:16:44.108 DEBUG:boto:Method: GET 2023-11-02T18:16:44.108 DEBUG:boto:Path: /ibtljn-47/obj4 2023-11-02T18:16:44.109 DEBUG:boto:Data: 2023-11-02T18:16:44.109 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.110 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.110 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.111 DEBUG:boto:Params: {} 2023-11-02T18:16:44.111 DEBUG:boto:Token: None 2023-11-02T18:16:44.112 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj4 2023-11-02T18:16:44.112 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KK/wbBNAl+w9wUseKeCy0Z3xnGM= 2023-11-02T18:16:44.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KK/wbBNAl+w9wUseKeCy0Z3xnGM=', 'Content-Length': '0'} 2023-11-02T18:16:44.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:13:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bae4223e437aae75-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.116 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.117 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.117 DEBUG:boto:path=/ibtljn-47/?prefix=obj4 2023-11-02T18:16:44.118 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj4 2023-11-02T18:16:44.118 DEBUG:boto:Method: GET 2023-11-02T18:16:44.119 DEBUG:boto:Path: /ibtljn-47/?prefix=obj4 2023-11-02T18:16:44.119 DEBUG:boto:Data: 2023-11-02T18:16:44.120 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.120 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.121 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.121 DEBUG:boto:Params: {} 2023-11-02T18:16:44.122 DEBUG:boto:Token: None 2023-11-02T18:16:44.122 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.123 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8ee2e3bf6856fef-006543e78c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.129 DEBUG:boto:b'ibtljn-47obj41000falseobj42023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.130 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.130 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.131 DEBUG:boto:path=/ibtljn-47/?prefix=obj4 2023-11-02T18:16:44.131 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj4 2023-11-02T18:16:44.131 DEBUG:boto:Method: GET 2023-11-02T18:16:44.132 DEBUG:boto:Path: /ibtljn-47/?prefix=obj4 2023-11-02T18:16:44.132 DEBUG:boto:Data: 2023-11-02T18:16:44.133 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.133 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.134 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.134 DEBUG:boto:Params: {} 2023-11-02T18:16:44.135 DEBUG:boto:Token: None 2023-11-02T18:16:44.135 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.136 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f716638461696162-006543e78c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.142 DEBUG:boto:b'ibtljn-47obj41000falseobj42023-11-02T18:13:45.658Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-11-02T18:16:44.143 DEBUG:boto:path=/ibtljn-47/obj5 2023-11-02T18:16:44.144 DEBUG:boto:auth_path=/ibtljn-47/obj5 2023-11-02T18:16:44.144 DEBUG:boto:path=/ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.145 DEBUG:boto:auth_path=/ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.145 DEBUG:boto:Method: GET 2023-11-02T18:16:44.145 DEBUG:boto:Path: /ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.146 DEBUG:boto:Data: 2023-11-02T18:16:44.146 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.147 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.147 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.148 DEBUG:boto:Params: {} 2023-11-02T18:16:44.148 DEBUG:boto:Token: None 2023-11-02T18:16:44.149 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.149 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4tWI3hJs7TjZbz/mmosIlDjUH3M= 2023-11-02T18:16:44.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4tWI3hJs7TjZbz/mmosIlDjUH3M=', 'Content-Length': '0'} 2023-11-02T18:16:44.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c6fa0a9977c306e-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.153 DEBUG:boto:path=/ibtljn-47/obj5 2023-11-02T18:16:44.154 DEBUG:boto:auth_path=/ibtljn-47/obj5 2023-11-02T18:16:44.154 DEBUG:boto:path=/ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.155 DEBUG:boto:auth_path=/ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.155 DEBUG:boto:Method: GET 2023-11-02T18:16:44.156 DEBUG:boto:Path: /ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.156 DEBUG:boto:Data: 2023-11-02T18:16:44.157 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.157 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.158 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.158 DEBUG:boto:Params: {} 2023-11-02T18:16:44.158 DEBUG:boto:Token: None 2023-11-02T18:16:44.159 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.159 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4tWI3hJs7TjZbz/mmosIlDjUH3M= 2023-11-02T18:16:44.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4tWI3hJs7TjZbz/mmosIlDjUH3M=', 'Content-Length': '0'} 2023-11-02T18:16:44.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b60ed483b70bc217-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.163 DEBUG:boto:path=/ibtljn-47/obj5 2023-11-02T18:16:44.164 DEBUG:boto:auth_path=/ibtljn-47/obj5 2023-11-02T18:16:44.164 DEBUG:boto:path=/ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.165 DEBUG:boto:auth_path=/ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.165 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.166 DEBUG:boto:Path: /ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.166 DEBUG:boto:Data: 2023-11-02T18:16:44.166 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.167 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.167 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.168 DEBUG:boto:Params: {} 2023-11-02T18:16:44.168 DEBUG:boto:Token: None 2023-11-02T18:16:44.169 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.169 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6jyR1zI92LyynNwgdijZ9bcxSp0= 2023-11-02T18:16:44.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6jyR1zI92LyynNwgdijZ9bcxSp0=', 'Content-Length': '0'} 2023-11-02T18:16:44.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000772613766cf90636-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.173 DEBUG:boto:path=/ibtljn-47/obj5 2023-11-02T18:16:44.174 DEBUG:boto:auth_path=/ibtljn-47/obj5 2023-11-02T18:16:44.174 DEBUG:boto:path=/ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.175 DEBUG:boto:auth_path=/ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.175 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.175 DEBUG:boto:Path: /ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.176 DEBUG:boto:Data: 2023-11-02T18:16:44.176 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.177 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.177 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.178 DEBUG:boto:Params: {} 2023-11-02T18:16:44.178 DEBUG:boto:Token: None 2023-11-02T18:16:44.179 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.179 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6jyR1zI92LyynNwgdijZ9bcxSp0= 2023-11-02T18:16:44.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6jyR1zI92LyynNwgdijZ9bcxSp0=', 'Content-Length': '0'} 2023-11-02T18:16:44.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000372bca86db4c5be9-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-11-02T18:16:44.183 DEBUG:boto:path=/ibtljn-47/obj5 2023-11-02T18:16:44.184 DEBUG:boto:auth_path=/ibtljn-47/obj5 2023-11-02T18:16:44.184 DEBUG:boto:path=/ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.185 DEBUG:boto:auth_path=/ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.185 DEBUG:boto:Method: GET 2023-11-02T18:16:44.186 DEBUG:boto:Path: /ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.186 DEBUG:boto:Data: 2023-11-02T18:16:44.187 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.187 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.187 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.188 DEBUG:boto:Params: {} 2023-11-02T18:16:44.188 DEBUG:boto:Token: None 2023-11-02T18:16:44.189 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.190 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4tWI3hJs7TjZbz/mmosIlDjUH3M= 2023-11-02T18:16:44.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4tWI3hJs7TjZbz/mmosIlDjUH3M=', 'Content-Length': '0'} 2023-11-02T18:16:44.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9b022b2eeaf001c-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.194 DEBUG:boto:path=/ibtljn-47/obj5 2023-11-02T18:16:44.194 DEBUG:boto:auth_path=/ibtljn-47/obj5 2023-11-02T18:16:44.195 DEBUG:boto:path=/ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.195 DEBUG:boto:auth_path=/ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.196 DEBUG:boto:Method: GET 2023-11-02T18:16:44.196 DEBUG:boto:Path: /ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.196 DEBUG:boto:Data: 2023-11-02T18:16:44.197 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.197 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.198 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.198 DEBUG:boto:Params: {} 2023-11-02T18:16:44.199 DEBUG:boto:Token: None 2023-11-02T18:16:44.199 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj5?versionId=null 2023-11-02T18:16:44.200 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4tWI3hJs7TjZbz/mmosIlDjUH3M= 2023-11-02T18:16:44.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4tWI3hJs7TjZbz/mmosIlDjUH3M=', 'Content-Length': '0'} 2023-11-02T18:16:44.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4fbf215ddb85541-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.250 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.251 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.251 DEBUG:boto:path=/ibtljn-47/?prefix=obj5 2023-11-02T18:16:44.252 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj5 2023-11-02T18:16:44.252 DEBUG:boto:Method: GET 2023-11-02T18:16:44.253 DEBUG:boto:Path: /ibtljn-47/?prefix=obj5 2023-11-02T18:16:44.253 DEBUG:boto:Data: 2023-11-02T18:16:44.254 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.254 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.255 DEBUG:boto:Params: {} 2023-11-02T18:16:44.255 DEBUG:boto:Token: None 2023-11-02T18:16:44.256 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.256 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c3e6348f6d862e7-006543e78c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.262 DEBUG:boto:b'ibtljn-47obj51000falseobj52023-11-02T18:15:24.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.263 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.263 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.264 DEBUG:boto:path=/ibtljn-47/?prefix=obj5 2023-11-02T18:16:44.264 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj5 2023-11-02T18:16:44.265 DEBUG:boto:Method: GET 2023-11-02T18:16:44.265 DEBUG:boto:Path: /ibtljn-47/?prefix=obj5 2023-11-02T18:16:44.265 DEBUG:boto:Data: 2023-11-02T18:16:44.266 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.266 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.267 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.267 DEBUG:boto:Params: {} 2023-11-02T18:16:44.268 DEBUG:boto:Token: None 2023-11-02T18:16:44.268 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.269 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feb50e6ac6bac354-006543e78c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.276 DEBUG:boto:b'ibtljn-47obj51000falseobj52023-11-02T18:15:24.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.277 DEBUG:boto:path=/ibtljn-47/obj5 2023-11-02T18:16:44.277 DEBUG:boto:auth_path=/ibtljn-47/obj5 2023-11-02T18:16:44.278 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.278 DEBUG:boto:Path: /ibtljn-47/obj5 2023-11-02T18:16:44.279 DEBUG:boto:Data: 2023-11-02T18:16:44.279 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.280 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.280 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.280 DEBUG:boto:Params: {} 2023-11-02T18:16:44.281 DEBUG:boto:Token: None 2023-11-02T18:16:44.281 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj5 2023-11-02T18:16:44.282 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:53ShqKLbxsaP+5LstOZeQ867tjg= 2023-11-02T18:16:44.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:53ShqKLbxsaP+5LstOZeQ867tjg=', 'Content-Length': '0'} 2023-11-02T18:16:44.284 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a36a00e27b1addc8-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.285 DEBUG:boto:path=/ibtljn-47/obj5 2023-11-02T18:16:44.286 DEBUG:boto:auth_path=/ibtljn-47/obj5 2023-11-02T18:16:44.286 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.286 DEBUG:boto:Path: /ibtljn-47/obj5 2023-11-02T18:16:44.287 DEBUG:boto:Data: 2023-11-02T18:16:44.287 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.288 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.288 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.289 DEBUG:boto:Params: {} 2023-11-02T18:16:44.289 DEBUG:boto:Token: None 2023-11-02T18:16:44.289 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj5 2023-11-02T18:16:44.290 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:53ShqKLbxsaP+5LstOZeQ867tjg= 2023-11-02T18:16:44.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:53ShqKLbxsaP+5LstOZeQ867tjg=', 'Content-Length': '0'} 2023-11-02T18:16:44.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eecdc643d3f3e2be-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-11-02T18:16:44.294 DEBUG:boto:path=/ibtljn-47/obj5 2023-11-02T18:16:44.294 DEBUG:boto:auth_path=/ibtljn-47/obj5 2023-11-02T18:16:44.295 DEBUG:boto:Method: GET 2023-11-02T18:16:44.295 DEBUG:boto:Path: /ibtljn-47/obj5 2023-11-02T18:16:44.296 DEBUG:boto:Data: 2023-11-02T18:16:44.296 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.296 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.297 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.297 DEBUG:boto:Params: {} 2023-11-02T18:16:44.298 DEBUG:boto:Token: None 2023-11-02T18:16:44.298 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj5 2023-11-02T18:16:44.299 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DQBhQEXS5X8p/FGjZtzYg/Ttj9o= 2023-11-02T18:16:44.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DQBhQEXS5X8p/FGjZtzYg/Ttj9o=', 'Content-Length': '0'} 2023-11-02T18:16:44.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ec75dc1aa9d2a5d-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.342 DEBUG:boto:path=/ibtljn-47/obj5 2023-11-02T18:16:44.343 DEBUG:boto:auth_path=/ibtljn-47/obj5 2023-11-02T18:16:44.343 DEBUG:boto:Method: GET 2023-11-02T18:16:44.344 DEBUG:boto:Path: /ibtljn-47/obj5 2023-11-02T18:16:44.344 DEBUG:boto:Data: 2023-11-02T18:16:44.345 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.345 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.346 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.346 DEBUG:boto:Params: {} 2023-11-02T18:16:44.347 DEBUG:boto:Token: None 2023-11-02T18:16:44.347 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj5 2023-11-02T18:16:44.348 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DQBhQEXS5X8p/FGjZtzYg/Ttj9o= 2023-11-02T18:16:44.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DQBhQEXS5X8p/FGjZtzYg/Ttj9o=', 'Content-Length': '0'} 2023-11-02T18:16:44.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000074dafce5d6b84af-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.352 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.353 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.353 DEBUG:boto:path=/ibtljn-47/?prefix=obj5 2023-11-02T18:16:44.354 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj5 2023-11-02T18:16:44.354 DEBUG:boto:Method: GET 2023-11-02T18:16:44.355 DEBUG:boto:Path: /ibtljn-47/?prefix=obj5 2023-11-02T18:16:44.355 DEBUG:boto:Data: 2023-11-02T18:16:44.356 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.356 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.356 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.357 DEBUG:boto:Params: {} 2023-11-02T18:16:44.357 DEBUG:boto:Token: None 2023-11-02T18:16:44.358 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.358 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c72551e229b3ae16-006543e78c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.364 DEBUG:boto:b'ibtljn-47obj51000falseobj52023-11-02T18:15:24.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.365 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.365 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.365 DEBUG:boto:path=/ibtljn-47/?prefix=obj5 2023-11-02T18:16:44.366 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj5 2023-11-02T18:16:44.366 DEBUG:boto:Method: GET 2023-11-02T18:16:44.367 DEBUG:boto:Path: /ibtljn-47/?prefix=obj5 2023-11-02T18:16:44.367 DEBUG:boto:Data: 2023-11-02T18:16:44.368 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.368 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.368 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.369 DEBUG:boto:Params: {} 2023-11-02T18:16:44.369 DEBUG:boto:Token: None 2023-11-02T18:16:44.370 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.370 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052c18911f51730af-006543e78c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.418 DEBUG:boto:b'ibtljn-47obj51000falseobj52023-11-02T18:15:24.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-11-02T18:16:44.420 DEBUG:boto:path=/ibtljn-47/obj6 2023-11-02T18:16:44.420 DEBUG:boto:auth_path=/ibtljn-47/obj6 2023-11-02T18:16:44.421 DEBUG:boto:path=/ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.421 DEBUG:boto:auth_path=/ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.422 DEBUG:boto:Method: GET 2023-11-02T18:16:44.422 DEBUG:boto:Path: /ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.422 DEBUG:boto:Data: 2023-11-02T18:16:44.423 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.423 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.424 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.424 DEBUG:boto:Params: {} 2023-11-02T18:16:44.425 DEBUG:boto:Token: None 2023-11-02T18:16:44.425 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.426 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jxLz5MvSQsV6fqUG8wDq8d+wCL0= 2023-11-02T18:16:44.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jxLz5MvSQsV6fqUG8wDq8d+wCL0=', 'Content-Length': '0'} 2023-11-02T18:16:44.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050d7994db1c6526b-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.430 DEBUG:boto:path=/ibtljn-47/obj6 2023-11-02T18:16:44.431 DEBUG:boto:auth_path=/ibtljn-47/obj6 2023-11-02T18:16:44.431 DEBUG:boto:path=/ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.432 DEBUG:boto:auth_path=/ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.432 DEBUG:boto:Method: GET 2023-11-02T18:16:44.432 DEBUG:boto:Path: /ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.433 DEBUG:boto:Data: 2023-11-02T18:16:44.433 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.434 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.434 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.435 DEBUG:boto:Params: {} 2023-11-02T18:16:44.435 DEBUG:boto:Token: None 2023-11-02T18:16:44.436 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.436 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jxLz5MvSQsV6fqUG8wDq8d+wCL0= 2023-11-02T18:16:44.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jxLz5MvSQsV6fqUG8wDq8d+wCL0=', 'Content-Length': '0'} 2023-11-02T18:16:44.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0d4745854f6b43f-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.440 DEBUG:boto:path=/ibtljn-47/obj6 2023-11-02T18:16:44.441 DEBUG:boto:auth_path=/ibtljn-47/obj6 2023-11-02T18:16:44.441 DEBUG:boto:path=/ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.442 DEBUG:boto:auth_path=/ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.442 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.443 DEBUG:boto:Path: /ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.443 DEBUG:boto:Data: 2023-11-02T18:16:44.444 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.444 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.444 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.445 DEBUG:boto:Params: {} 2023-11-02T18:16:44.445 DEBUG:boto:Token: None 2023-11-02T18:16:44.446 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.446 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TL/pU1SjDkmWvTt999rcaLKxz6g= 2023-11-02T18:16:44.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TL/pU1SjDkmWvTt999rcaLKxz6g=', 'Content-Length': '0'} 2023-11-02T18:16:44.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031da1bf20559a062-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.450 DEBUG:boto:path=/ibtljn-47/obj6 2023-11-02T18:16:44.451 DEBUG:boto:auth_path=/ibtljn-47/obj6 2023-11-02T18:16:44.451 DEBUG:boto:path=/ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.452 DEBUG:boto:auth_path=/ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.452 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.453 DEBUG:boto:Path: /ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.453 DEBUG:boto:Data: 2023-11-02T18:16:44.453 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.454 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.455 DEBUG:boto:Params: {} 2023-11-02T18:16:44.455 DEBUG:boto:Token: None 2023-11-02T18:16:44.456 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.456 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TL/pU1SjDkmWvTt999rcaLKxz6g= 2023-11-02T18:16:44.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TL/pU1SjDkmWvTt999rcaLKxz6g=', 'Content-Length': '0'} 2023-11-02T18:16:44.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b1a8b27cae1bcf4-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-11-02T18:16:44.460 DEBUG:boto:path=/ibtljn-47/obj6 2023-11-02T18:16:44.461 DEBUG:boto:auth_path=/ibtljn-47/obj6 2023-11-02T18:16:44.461 DEBUG:boto:path=/ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.462 DEBUG:boto:auth_path=/ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.462 DEBUG:boto:Method: GET 2023-11-02T18:16:44.463 DEBUG:boto:Path: /ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.463 DEBUG:boto:Data: 2023-11-02T18:16:44.464 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.464 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.465 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.465 DEBUG:boto:Params: {} 2023-11-02T18:16:44.465 DEBUG:boto:Token: None 2023-11-02T18:16:44.466 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.466 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jxLz5MvSQsV6fqUG8wDq8d+wCL0= 2023-11-02T18:16:44.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jxLz5MvSQsV6fqUG8wDq8d+wCL0=', 'Content-Length': '0'} 2023-11-02T18:16:44.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a0bf26c8481e4d1-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.471 DEBUG:boto:path=/ibtljn-47/obj6 2023-11-02T18:16:44.471 DEBUG:boto:auth_path=/ibtljn-47/obj6 2023-11-02T18:16:44.471 DEBUG:boto:path=/ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.472 DEBUG:boto:auth_path=/ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.472 DEBUG:boto:Method: GET 2023-11-02T18:16:44.473 DEBUG:boto:Path: /ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.473 DEBUG:boto:Data: 2023-11-02T18:16:44.474 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.474 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.475 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.475 DEBUG:boto:Params: {} 2023-11-02T18:16:44.476 DEBUG:boto:Token: None 2023-11-02T18:16:44.476 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj6?versionId=null 2023-11-02T18:16:44.477 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jxLz5MvSQsV6fqUG8wDq8d+wCL0= 2023-11-02T18:16:44.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jxLz5MvSQsV6fqUG8wDq8d+wCL0=', 'Content-Length': '0'} 2023-11-02T18:16:44.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000726875600643cb58-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.526 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.527 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.527 DEBUG:boto:path=/ibtljn-47/?prefix=obj6 2023-11-02T18:16:44.528 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj6 2023-11-02T18:16:44.528 DEBUG:boto:Method: GET 2023-11-02T18:16:44.529 DEBUG:boto:Path: /ibtljn-47/?prefix=obj6 2023-11-02T18:16:44.529 DEBUG:boto:Data: 2023-11-02T18:16:44.530 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.530 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.531 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.531 DEBUG:boto:Params: {} 2023-11-02T18:16:44.531 DEBUG:boto:Token: None 2023-11-02T18:16:44.532 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.533 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059a887662801f9b1-006543e78c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.540 DEBUG:boto:b'ibtljn-47obj61000falseobj62023-11-02T18:15:24.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.540 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.541 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.541 DEBUG:boto:path=/ibtljn-47/?prefix=obj6 2023-11-02T18:16:44.542 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj6 2023-11-02T18:16:44.542 DEBUG:boto:Method: GET 2023-11-02T18:16:44.543 DEBUG:boto:Path: /ibtljn-47/?prefix=obj6 2023-11-02T18:16:44.543 DEBUG:boto:Data: 2023-11-02T18:16:44.543 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.544 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.544 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.545 DEBUG:boto:Params: {} 2023-11-02T18:16:44.545 DEBUG:boto:Token: None 2023-11-02T18:16:44.546 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.546 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb28cec47b9d1f77-006543e78c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.552 DEBUG:boto:b'ibtljn-47obj61000falseobj62023-11-02T18:15:24.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.553 DEBUG:boto:path=/ibtljn-47/obj6 2023-11-02T18:16:44.554 DEBUG:boto:auth_path=/ibtljn-47/obj6 2023-11-02T18:16:44.554 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.555 DEBUG:boto:Path: /ibtljn-47/obj6 2023-11-02T18:16:44.555 DEBUG:boto:Data: 2023-11-02T18:16:44.555 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.556 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.556 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.557 DEBUG:boto:Params: {} 2023-11-02T18:16:44.557 DEBUG:boto:Token: None 2023-11-02T18:16:44.558 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj6 2023-11-02T18:16:44.558 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DJgppPdWEVPPdXlDYchDL45h8h0= 2023-11-02T18:16:44.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DJgppPdWEVPPdXlDYchDL45h8h0=', 'Content-Length': '0'} 2023-11-02T18:16:44.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000209aa6040fa8ade5-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.561 DEBUG:boto:path=/ibtljn-47/obj6 2023-11-02T18:16:44.562 DEBUG:boto:auth_path=/ibtljn-47/obj6 2023-11-02T18:16:44.562 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.563 DEBUG:boto:Path: /ibtljn-47/obj6 2023-11-02T18:16:44.563 DEBUG:boto:Data: 2023-11-02T18:16:44.564 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.564 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.565 DEBUG:boto:Params: {} 2023-11-02T18:16:44.565 DEBUG:boto:Token: None 2023-11-02T18:16:44.566 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj6 2023-11-02T18:16:44.566 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DJgppPdWEVPPdXlDYchDL45h8h0= 2023-11-02T18:16:44.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DJgppPdWEVPPdXlDYchDL45h8h0=', 'Content-Length': '0'} 2023-11-02T18:16:44.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f4771ab59d26247-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-11-02T18:16:44.570 DEBUG:boto:path=/ibtljn-47/obj6 2023-11-02T18:16:44.570 DEBUG:boto:auth_path=/ibtljn-47/obj6 2023-11-02T18:16:44.571 DEBUG:boto:Method: GET 2023-11-02T18:16:44.571 DEBUG:boto:Path: /ibtljn-47/obj6 2023-11-02T18:16:44.572 DEBUG:boto:Data: 2023-11-02T18:16:44.572 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.572 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.573 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.573 DEBUG:boto:Params: {} 2023-11-02T18:16:44.574 DEBUG:boto:Token: None 2023-11-02T18:16:44.574 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj6 2023-11-02T18:16:44.575 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/l21xNjXOz0qTLyC+UPz/TbtaCg= 2023-11-02T18:16:44.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/l21xNjXOz0qTLyC+UPz/TbtaCg=', 'Content-Length': '0'} 2023-11-02T18:16:44.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aac09e86eccfa45c-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.578 DEBUG:boto:path=/ibtljn-47/obj6 2023-11-02T18:16:44.579 DEBUG:boto:auth_path=/ibtljn-47/obj6 2023-11-02T18:16:44.579 DEBUG:boto:Method: GET 2023-11-02T18:16:44.580 DEBUG:boto:Path: /ibtljn-47/obj6 2023-11-02T18:16:44.580 DEBUG:boto:Data: 2023-11-02T18:16:44.580 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.581 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.581 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.582 DEBUG:boto:Params: {} 2023-11-02T18:16:44.582 DEBUG:boto:Token: None 2023-11-02T18:16:44.583 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj6 2023-11-02T18:16:44.583 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/l21xNjXOz0qTLyC+UPz/TbtaCg= 2023-11-02T18:16:44.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/l21xNjXOz0qTLyC+UPz/TbtaCg=', 'Content-Length': '0'} 2023-11-02T18:16:44.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f746e59b4200c13a-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.587 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.588 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.588 DEBUG:boto:path=/ibtljn-47/?prefix=obj6 2023-11-02T18:16:44.588 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj6 2023-11-02T18:16:44.589 DEBUG:boto:Method: GET 2023-11-02T18:16:44.589 DEBUG:boto:Path: /ibtljn-47/?prefix=obj6 2023-11-02T18:16:44.590 DEBUG:boto:Data: 2023-11-02T18:16:44.590 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.591 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.591 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.591 DEBUG:boto:Params: {} 2023-11-02T18:16:44.592 DEBUG:boto:Token: None 2023-11-02T18:16:44.592 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000275303af58e24937-006543e78c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.598 DEBUG:boto:b'ibtljn-47obj61000falseobj62023-11-02T18:15:24.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.599 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.599 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.600 DEBUG:boto:path=/ibtljn-47/?prefix=obj6 2023-11-02T18:16:44.600 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj6 2023-11-02T18:16:44.601 DEBUG:boto:Method: GET 2023-11-02T18:16:44.601 DEBUG:boto:Path: /ibtljn-47/?prefix=obj6 2023-11-02T18:16:44.601 DEBUG:boto:Data: 2023-11-02T18:16:44.602 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.602 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.603 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.603 DEBUG:boto:Params: {} 2023-11-02T18:16:44.603 DEBUG:boto:Token: None 2023-11-02T18:16:44.604 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.604 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000062518d7426f7b19-006543e78c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.650 DEBUG:boto:b'ibtljn-47obj61000falseobj62023-11-02T18:15:24.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-11-02T18:16:44.652 DEBUG:boto:path=/ibtljn-47/obj7 2023-11-02T18:16:44.652 DEBUG:boto:auth_path=/ibtljn-47/obj7 2023-11-02T18:16:44.653 DEBUG:boto:path=/ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.653 DEBUG:boto:auth_path=/ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.653 DEBUG:boto:Method: GET 2023-11-02T18:16:44.654 DEBUG:boto:Path: /ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.654 DEBUG:boto:Data: 2023-11-02T18:16:44.655 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.655 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.656 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.656 DEBUG:boto:Params: {} 2023-11-02T18:16:44.656 DEBUG:boto:Token: None 2023-11-02T18:16:44.657 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.658 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rfhRzIcI0mYKncwlUyhF2L5mYAw= 2023-11-02T18:16:44.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rfhRzIcI0mYKncwlUyhF2L5mYAw=', 'Content-Length': '0'} 2023-11-02T18:16:44.661 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b3c94651f749c7e-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.662 DEBUG:boto:path=/ibtljn-47/obj7 2023-11-02T18:16:44.662 DEBUG:boto:auth_path=/ibtljn-47/obj7 2023-11-02T18:16:44.663 DEBUG:boto:path=/ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.663 DEBUG:boto:auth_path=/ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.664 DEBUG:boto:Method: GET 2023-11-02T18:16:44.664 DEBUG:boto:Path: /ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.665 DEBUG:boto:Data: 2023-11-02T18:16:44.665 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.665 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.666 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.666 DEBUG:boto:Params: {} 2023-11-02T18:16:44.667 DEBUG:boto:Token: None 2023-11-02T18:16:44.667 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.668 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rfhRzIcI0mYKncwlUyhF2L5mYAw= 2023-11-02T18:16:44.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rfhRzIcI0mYKncwlUyhF2L5mYAw=', 'Content-Length': '0'} 2023-11-02T18:16:44.671 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f95561eec3cf6df9-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.672 DEBUG:boto:path=/ibtljn-47/obj7 2023-11-02T18:16:44.672 DEBUG:boto:auth_path=/ibtljn-47/obj7 2023-11-02T18:16:44.673 DEBUG:boto:path=/ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.673 DEBUG:boto:auth_path=/ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.673 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.674 DEBUG:boto:Path: /ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.674 DEBUG:boto:Data: 2023-11-02T18:16:44.675 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.675 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.676 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.676 DEBUG:boto:Params: {} 2023-11-02T18:16:44.676 DEBUG:boto:Token: None 2023-11-02T18:16:44.677 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.677 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tBl1/MAW4HXDlH0EsD2aE3gnG/I= 2023-11-02T18:16:44.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tBl1/MAW4HXDlH0EsD2aE3gnG/I=', 'Content-Length': '0'} 2023-11-02T18:16:44.681 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7ca2262300973bf-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.681 DEBUG:boto:path=/ibtljn-47/obj7 2023-11-02T18:16:44.682 DEBUG:boto:auth_path=/ibtljn-47/obj7 2023-11-02T18:16:44.682 DEBUG:boto:path=/ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.683 DEBUG:boto:auth_path=/ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.683 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.683 DEBUG:boto:Path: /ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.684 DEBUG:boto:Data: 2023-11-02T18:16:44.684 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.685 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.685 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.685 DEBUG:boto:Params: {} 2023-11-02T18:16:44.686 DEBUG:boto:Token: None 2023-11-02T18:16:44.686 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.687 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tBl1/MAW4HXDlH0EsD2aE3gnG/I= 2023-11-02T18:16:44.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tBl1/MAW4HXDlH0EsD2aE3gnG/I=', 'Content-Length': '0'} 2023-11-02T18:16:44.690 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006eb65b2cd1601bd0-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-11-02T18:16:44.691 DEBUG:boto:path=/ibtljn-47/obj7 2023-11-02T18:16:44.692 DEBUG:boto:auth_path=/ibtljn-47/obj7 2023-11-02T18:16:44.692 DEBUG:boto:path=/ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.692 DEBUG:boto:auth_path=/ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.693 DEBUG:boto:Method: GET 2023-11-02T18:16:44.693 DEBUG:boto:Path: /ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.694 DEBUG:boto:Data: 2023-11-02T18:16:44.694 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.695 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.695 DEBUG:boto:Params: {} 2023-11-02T18:16:44.696 DEBUG:boto:Token: None 2023-11-02T18:16:44.696 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.697 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rfhRzIcI0mYKncwlUyhF2L5mYAw= 2023-11-02T18:16:44.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rfhRzIcI0mYKncwlUyhF2L5mYAw=', 'Content-Length': '0'} 2023-11-02T18:16:44.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008c30906d5ff3a03-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.701 DEBUG:boto:path=/ibtljn-47/obj7 2023-11-02T18:16:44.701 DEBUG:boto:auth_path=/ibtljn-47/obj7 2023-11-02T18:16:44.702 DEBUG:boto:path=/ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.702 DEBUG:boto:auth_path=/ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.702 DEBUG:boto:Method: GET 2023-11-02T18:16:44.703 DEBUG:boto:Path: /ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.703 DEBUG:boto:Data: 2023-11-02T18:16:44.704 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.704 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.705 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.705 DEBUG:boto:Params: {} 2023-11-02T18:16:44.705 DEBUG:boto:Token: None 2023-11-02T18:16:44.706 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj7?versionId=null 2023-11-02T18:16:44.707 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rfhRzIcI0mYKncwlUyhF2L5mYAw= 2023-11-02T18:16:44.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rfhRzIcI0mYKncwlUyhF2L5mYAw=', 'Content-Length': '0'} 2023-11-02T18:16:44.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037ea390f84e829a1-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.711 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.712 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.712 DEBUG:boto:path=/ibtljn-47/?prefix=obj7 2023-11-02T18:16:44.713 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj7 2023-11-02T18:16:44.713 DEBUG:boto:Method: GET 2023-11-02T18:16:44.714 DEBUG:boto:Path: /ibtljn-47/?prefix=obj7 2023-11-02T18:16:44.714 DEBUG:boto:Data: 2023-11-02T18:16:44.714 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.715 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.715 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.717 DEBUG:boto:Params: {} 2023-11-02T18:16:44.717 DEBUG:boto:Token: None 2023-11-02T18:16:44.718 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.718 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072d09a7def5ce06f-006543e78c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.766 DEBUG:boto:b'ibtljn-47obj71000falseobj72023-11-02T18:15:24.933Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.767 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.768 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.768 DEBUG:boto:path=/ibtljn-47/?prefix=obj7 2023-11-02T18:16:44.769 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj7 2023-11-02T18:16:44.769 DEBUG:boto:Method: GET 2023-11-02T18:16:44.770 DEBUG:boto:Path: /ibtljn-47/?prefix=obj7 2023-11-02T18:16:44.770 DEBUG:boto:Data: 2023-11-02T18:16:44.770 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.771 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.771 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.772 DEBUG:boto:Params: {} 2023-11-02T18:16:44.772 DEBUG:boto:Token: None 2023-11-02T18:16:44.773 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.773 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a55d8f8aad996a5-006543e78c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.779 DEBUG:boto:b'ibtljn-47obj71000falseobj72023-11-02T18:15:24.933Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.780 DEBUG:boto:path=/ibtljn-47/obj7 2023-11-02T18:16:44.780 DEBUG:boto:auth_path=/ibtljn-47/obj7 2023-11-02T18:16:44.781 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.781 DEBUG:boto:Path: /ibtljn-47/obj7 2023-11-02T18:16:44.782 DEBUG:boto:Data: 2023-11-02T18:16:44.782 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.783 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.783 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.783 DEBUG:boto:Params: {} 2023-11-02T18:16:44.784 DEBUG:boto:Token: None 2023-11-02T18:16:44.784 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj7 2023-11-02T18:16:44.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CcKHarBVNPdDymZt3DApF5rQ0Fc= 2023-11-02T18:16:44.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CcKHarBVNPdDymZt3DApF5rQ0Fc=', 'Content-Length': '0'} 2023-11-02T18:16:44.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da7a894adbb08145-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.789 DEBUG:boto:path=/ibtljn-47/obj7 2023-11-02T18:16:44.789 DEBUG:boto:auth_path=/ibtljn-47/obj7 2023-11-02T18:16:44.789 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.790 DEBUG:boto:Path: /ibtljn-47/obj7 2023-11-02T18:16:44.790 DEBUG:boto:Data: 2023-11-02T18:16:44.791 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.791 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.791 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.792 DEBUG:boto:Params: {} 2023-11-02T18:16:44.792 DEBUG:boto:Token: None 2023-11-02T18:16:44.793 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj7 2023-11-02T18:16:44.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CcKHarBVNPdDymZt3DApF5rQ0Fc= 2023-11-02T18:16:44.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CcKHarBVNPdDymZt3DApF5rQ0Fc=', 'Content-Length': '0'} 2023-11-02T18:16:44.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b04b65e18b024b7-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-11-02T18:16:44.798 DEBUG:boto:path=/ibtljn-47/obj7 2023-11-02T18:16:44.799 DEBUG:boto:auth_path=/ibtljn-47/obj7 2023-11-02T18:16:44.799 DEBUG:boto:Method: GET 2023-11-02T18:16:44.799 DEBUG:boto:Path: /ibtljn-47/obj7 2023-11-02T18:16:44.800 DEBUG:boto:Data: 2023-11-02T18:16:44.800 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.801 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.801 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.802 DEBUG:boto:Params: {} 2023-11-02T18:16:44.802 DEBUG:boto:Token: None 2023-11-02T18:16:44.803 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj7 2023-11-02T18:16:44.803 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XVcSLhiE4KWq65N/JOzNX8pn0kU= 2023-11-02T18:16:44.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XVcSLhiE4KWq65N/JOzNX8pn0kU=', 'Content-Length': '0'} 2023-11-02T18:16:44.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040fcdceaae25248a-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.807 DEBUG:boto:path=/ibtljn-47/obj7 2023-11-02T18:16:44.807 DEBUG:boto:auth_path=/ibtljn-47/obj7 2023-11-02T18:16:44.808 DEBUG:boto:Method: GET 2023-11-02T18:16:44.808 DEBUG:boto:Path: /ibtljn-47/obj7 2023-11-02T18:16:44.809 DEBUG:boto:Data: 2023-11-02T18:16:44.809 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.809 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.810 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.810 DEBUG:boto:Params: {} 2023-11-02T18:16:44.811 DEBUG:boto:Token: None 2023-11-02T18:16:44.811 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj7 2023-11-02T18:16:44.812 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XVcSLhiE4KWq65N/JOzNX8pn0kU= 2023-11-02T18:16:44.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XVcSLhiE4KWq65N/JOzNX8pn0kU=', 'Content-Length': '0'} 2023-11-02T18:16:44.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000510e09641cde3ca4-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.858 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.859 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.860 DEBUG:boto:path=/ibtljn-47/?prefix=obj7 2023-11-02T18:16:44.860 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj7 2023-11-02T18:16:44.860 DEBUG:boto:Method: GET 2023-11-02T18:16:44.861 DEBUG:boto:Path: /ibtljn-47/?prefix=obj7 2023-11-02T18:16:44.861 DEBUG:boto:Data: 2023-11-02T18:16:44.862 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.862 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.863 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.863 DEBUG:boto:Params: {} 2023-11-02T18:16:44.863 DEBUG:boto:Token: None 2023-11-02T18:16:44.864 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.865 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088fa0ac58bcc85a2-006543e78c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.871 DEBUG:boto:b'ibtljn-47obj71000falseobj72023-11-02T18:15:24.933Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.872 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.872 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.873 DEBUG:boto:path=/ibtljn-47/?prefix=obj7 2023-11-02T18:16:44.873 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj7 2023-11-02T18:16:44.873 DEBUG:boto:Method: GET 2023-11-02T18:16:44.874 DEBUG:boto:Path: /ibtljn-47/?prefix=obj7 2023-11-02T18:16:44.874 DEBUG:boto:Data: 2023-11-02T18:16:44.875 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.875 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.876 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.876 DEBUG:boto:Params: {} 2023-11-02T18:16:44.876 DEBUG:boto:Token: None 2023-11-02T18:16:44.877 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:44.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6ce3475dc7654cf-006543e78c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.884 DEBUG:boto:b'ibtljn-47obj71000falseobj72023-11-02T18:15:24.933Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:44.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-11-02T18:16:44.885 DEBUG:boto:path=/ibtljn-47/obj8 2023-11-02T18:16:44.885 DEBUG:boto:auth_path=/ibtljn-47/obj8 2023-11-02T18:16:44.886 DEBUG:boto:path=/ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.886 DEBUG:boto:auth_path=/ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.887 DEBUG:boto:Method: GET 2023-11-02T18:16:44.887 DEBUG:boto:Path: /ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.887 DEBUG:boto:Data: 2023-11-02T18:16:44.888 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.888 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.889 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.889 DEBUG:boto:Params: {} 2023-11-02T18:16:44.890 DEBUG:boto:Token: None 2023-11-02T18:16:44.890 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.891 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ANGCleCi+ySIUoOjbf4KTuFuQks= 2023-11-02T18:16:44.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ANGCleCi+ySIUoOjbf4KTuFuQks=', 'Content-Length': '0'} 2023-11-02T18:16:44.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074ea82e1ccc53b7f-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.934 DEBUG:boto:path=/ibtljn-47/obj8 2023-11-02T18:16:44.935 DEBUG:boto:auth_path=/ibtljn-47/obj8 2023-11-02T18:16:44.936 DEBUG:boto:path=/ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.936 DEBUG:boto:auth_path=/ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.936 DEBUG:boto:Method: GET 2023-11-02T18:16:44.937 DEBUG:boto:Path: /ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.937 DEBUG:boto:Data: 2023-11-02T18:16:44.938 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.938 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.939 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.939 DEBUG:boto:Params: {} 2023-11-02T18:16:44.940 DEBUG:boto:Token: None 2023-11-02T18:16:44.940 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.941 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ANGCleCi+ySIUoOjbf4KTuFuQks= 2023-11-02T18:16:44.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ANGCleCi+ySIUoOjbf4KTuFuQks=', 'Content-Length': '0'} 2023-11-02T18:16:44.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ae70afd4169a57e-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.945 DEBUG:boto:path=/ibtljn-47/obj8 2023-11-02T18:16:44.946 DEBUG:boto:auth_path=/ibtljn-47/obj8 2023-11-02T18:16:44.947 DEBUG:boto:path=/ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.947 DEBUG:boto:auth_path=/ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.947 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.948 DEBUG:boto:Path: /ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.948 DEBUG:boto:Data: 2023-11-02T18:16:44.949 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.949 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.950 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.950 DEBUG:boto:Params: {} 2023-11-02T18:16:44.951 DEBUG:boto:Token: None 2023-11-02T18:16:44.951 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.952 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IjLkG4EuzwJql+CQjz678vF6YVA= 2023-11-02T18:16:44.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IjLkG4EuzwJql+CQjz678vF6YVA=', 'Content-Length': '0'} 2023-11-02T18:16:44.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017e3695c46a72464-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.956 DEBUG:boto:path=/ibtljn-47/obj8 2023-11-02T18:16:44.956 DEBUG:boto:auth_path=/ibtljn-47/obj8 2023-11-02T18:16:44.957 DEBUG:boto:path=/ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.957 DEBUG:boto:auth_path=/ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.958 DEBUG:boto:Method: HEAD 2023-11-02T18:16:44.958 DEBUG:boto:Path: /ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.959 DEBUG:boto:Data: 2023-11-02T18:16:44.959 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.960 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.960 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.961 DEBUG:boto:Params: {} 2023-11-02T18:16:44.961 DEBUG:boto:Token: None 2023-11-02T18:16:44.962 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.962 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IjLkG4EuzwJql+CQjz678vF6YVA= 2023-11-02T18:16:44.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IjLkG4EuzwJql+CQjz678vF6YVA=', 'Content-Length': '0'} 2023-11-02T18:16:44.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017ca6c5575f36b44-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-11-02T18:16:44.967 DEBUG:boto:path=/ibtljn-47/obj8 2023-11-02T18:16:44.967 DEBUG:boto:auth_path=/ibtljn-47/obj8 2023-11-02T18:16:44.968 DEBUG:boto:path=/ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.968 DEBUG:boto:auth_path=/ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.969 DEBUG:boto:Method: GET 2023-11-02T18:16:44.969 DEBUG:boto:Path: /ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.970 DEBUG:boto:Data: 2023-11-02T18:16:44.970 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.971 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.971 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.972 DEBUG:boto:Params: {} 2023-11-02T18:16:44.972 DEBUG:boto:Token: None 2023-11-02T18:16:44.973 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.974 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ANGCleCi+ySIUoOjbf4KTuFuQks= 2023-11-02T18:16:44.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ANGCleCi+ySIUoOjbf4KTuFuQks=', 'Content-Length': '0'} 2023-11-02T18:16:44.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc74b832b4032cd5-006543e78c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.978 DEBUG:boto:path=/ibtljn-47/obj8 2023-11-02T18:16:44.978 DEBUG:boto:auth_path=/ibtljn-47/obj8 2023-11-02T18:16:44.979 DEBUG:boto:path=/ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.979 DEBUG:boto:auth_path=/ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.980 DEBUG:boto:Method: GET 2023-11-02T18:16:44.980 DEBUG:boto:Path: /ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.981 DEBUG:boto:Data: 2023-11-02T18:16:44.981 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.982 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:44.982 DEBUG:boto:Port: 8000 2023-11-02T18:16:44.983 DEBUG:boto:Params: {} 2023-11-02T18:16:44.983 DEBUG:boto:Token: None 2023-11-02T18:16:44.984 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/obj8?versionId=null 2023-11-02T18:16:44.984 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ANGCleCi+ySIUoOjbf4KTuFuQks= 2023-11-02T18:16:44.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ANGCleCi+ySIUoOjbf4KTuFuQks=', 'Content-Length': '0'} 2023-11-02T18:16:44.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e42630f07e18f32d-006543e78c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:44.988 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:44.989 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:44.990 DEBUG:boto:path=/ibtljn-47/?prefix=obj8 2023-11-02T18:16:44.990 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj8 2023-11-02T18:16:44.990 DEBUG:boto:Method: GET 2023-11-02T18:16:44.991 DEBUG:boto:Path: /ibtljn-47/?prefix=obj8 2023-11-02T18:16:44.991 DEBUG:boto:Data: 2023-11-02T18:16:44.992 DEBUG:boto:Headers: {} 2023-11-02T18:16:44.992 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:44.993 DEBUG:boto:Port: 8001 2023-11-02T18:16:44.993 DEBUG:boto:Params: {} 2023-11-02T18:16:44.994 DEBUG:boto:Token: None 2023-11-02T18:16:44.994 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:44 GMT /ibtljn-47/ 2023-11-02T18:16:44.995 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI= 2023-11-02T18:16:44.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Thax7GbwldubTl8r2uC8xykKAGI=', 'Content-Length': '0'} 2023-11-02T18:16:45.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f8f8d4054c8a13b-006543e78c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.001 DEBUG:boto:b'ibtljn-47obj81000falseobj82023-11-02T18:15:24.960Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:45.002 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:45.002 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:45.003 DEBUG:boto:path=/ibtljn-47/?prefix=obj8 2023-11-02T18:16:45.003 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj8 2023-11-02T18:16:45.003 DEBUG:boto:Method: GET 2023-11-02T18:16:45.004 DEBUG:boto:Path: /ibtljn-47/?prefix=obj8 2023-11-02T18:16:45.004 DEBUG:boto:Data: 2023-11-02T18:16:45.005 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.005 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.006 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.006 DEBUG:boto:Params: {} 2023-11-02T18:16:45.007 DEBUG:boto:Token: None 2023-11-02T18:16:45.007 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-47/ 2023-11-02T18:16:45.008 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BCv+1j+JVgmsmyxdfpR9BgWyX/Q= 2023-11-02T18:16:45.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BCv+1j+JVgmsmyxdfpR9BgWyX/Q=', 'Content-Length': '0'} 2023-11-02T18:16:45.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d56f4c9da687ce9-006543e78d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.014 DEBUG:boto:b'ibtljn-47obj81000falseobj82023-11-02T18:15:24.960Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:45.015 DEBUG:boto:path=/ibtljn-47/obj8 2023-11-02T18:16:45.015 DEBUG:boto:auth_path=/ibtljn-47/obj8 2023-11-02T18:16:45.016 DEBUG:boto:Method: HEAD 2023-11-02T18:16:45.016 DEBUG:boto:Path: /ibtljn-47/obj8 2023-11-02T18:16:45.017 DEBUG:boto:Data: 2023-11-02T18:16:45.017 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.018 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:45.018 DEBUG:boto:Port: 8001 2023-11-02T18:16:45.019 DEBUG:boto:Params: {} 2023-11-02T18:16:45.019 DEBUG:boto:Token: None 2023-11-02T18:16:45.020 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-47/obj8 2023-11-02T18:16:45.020 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9FRL3MYmMoJFMWnBFW/Zi9JUH6s= 2023-11-02T18:16:45.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9FRL3MYmMoJFMWnBFW/Zi9JUH6s=', 'Content-Length': '0'} 2023-11-02T18:16:45.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ef23a6be5ea889c-006543e78d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.024 DEBUG:boto:path=/ibtljn-47/obj8 2023-11-02T18:16:45.024 DEBUG:boto:auth_path=/ibtljn-47/obj8 2023-11-02T18:16:45.025 DEBUG:boto:Method: HEAD 2023-11-02T18:16:45.025 DEBUG:boto:Path: /ibtljn-47/obj8 2023-11-02T18:16:45.026 DEBUG:boto:Data: 2023-11-02T18:16:45.026 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.026 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.027 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.027 DEBUG:boto:Params: {} 2023-11-02T18:16:45.028 DEBUG:boto:Token: None 2023-11-02T18:16:45.028 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-47/obj8 2023-11-02T18:16:45.029 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9FRL3MYmMoJFMWnBFW/Zi9JUH6s= 2023-11-02T18:16:45.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9FRL3MYmMoJFMWnBFW/Zi9JUH6s=', 'Content-Length': '0'} 2023-11-02T18:16:45.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073547ae5115073bc-006543e78d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-11-02T18:16:45.033 DEBUG:boto:path=/ibtljn-47/obj8 2023-11-02T18:16:45.034 DEBUG:boto:auth_path=/ibtljn-47/obj8 2023-11-02T18:16:45.034 DEBUG:boto:Method: GET 2023-11-02T18:16:45.035 DEBUG:boto:Path: /ibtljn-47/obj8 2023-11-02T18:16:45.035 DEBUG:boto:Data: 2023-11-02T18:16:45.036 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.036 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:45.037 DEBUG:boto:Port: 8001 2023-11-02T18:16:45.038 DEBUG:boto:Params: {} 2023-11-02T18:16:45.038 DEBUG:boto:Token: None 2023-11-02T18:16:45.039 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-47/obj8 2023-11-02T18:16:45.039 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ild71RH206DUW9NIP84jYGBd5LU= 2023-11-02T18:16:45.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ild71RH206DUW9NIP84jYGBd5LU=', 'Content-Length': '0'} 2023-11-02T18:16:45.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b42764ce3bfb952c-006543e78d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.086 DEBUG:boto:path=/ibtljn-47/obj8 2023-11-02T18:16:45.087 DEBUG:boto:auth_path=/ibtljn-47/obj8 2023-11-02T18:16:45.088 DEBUG:boto:Method: GET 2023-11-02T18:16:45.088 DEBUG:boto:Path: /ibtljn-47/obj8 2023-11-02T18:16:45.089 DEBUG:boto:Data: 2023-11-02T18:16:45.089 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.090 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.090 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.091 DEBUG:boto:Params: {} 2023-11-02T18:16:45.091 DEBUG:boto:Token: None 2023-11-02T18:16:45.092 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-47/obj8 2023-11-02T18:16:45.092 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ild71RH206DUW9NIP84jYGBd5LU= 2023-11-02T18:16:45.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ild71RH206DUW9NIP84jYGBd5LU=', 'Content-Length': '0'} 2023-11-02T18:16:45.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:15:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-47:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbb62c9ffb8193fb-006543e78d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.097 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:45.097 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:45.098 DEBUG:boto:path=/ibtljn-47/?prefix=obj8 2023-11-02T18:16:45.098 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj8 2023-11-02T18:16:45.099 DEBUG:boto:Method: GET 2023-11-02T18:16:45.099 DEBUG:boto:Path: /ibtljn-47/?prefix=obj8 2023-11-02T18:16:45.100 DEBUG:boto:Data: 2023-11-02T18:16:45.100 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.101 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:45.101 DEBUG:boto:Port: 8001 2023-11-02T18:16:45.102 DEBUG:boto:Params: {} 2023-11-02T18:16:45.103 DEBUG:boto:Token: None 2023-11-02T18:16:45.103 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-47/ 2023-11-02T18:16:45.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BCv+1j+JVgmsmyxdfpR9BgWyX/Q= 2023-11-02T18:16:45.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BCv+1j+JVgmsmyxdfpR9BgWyX/Q=', 'Content-Length': '0'} 2023-11-02T18:16:45.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de35c67ec021d221-006543e78d-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.109 DEBUG:boto:b'ibtljn-47obj81000falseobj82023-11-02T18:15:24.960Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:45.110 DEBUG:boto:path=/ibtljn-47/ 2023-11-02T18:16:45.111 DEBUG:boto:auth_path=/ibtljn-47/ 2023-11-02T18:16:45.111 DEBUG:boto:path=/ibtljn-47/?prefix=obj8 2023-11-02T18:16:45.111 DEBUG:boto:auth_path=/ibtljn-47/?prefix=obj8 2023-11-02T18:16:45.112 DEBUG:boto:Method: GET 2023-11-02T18:16:45.112 DEBUG:boto:Path: /ibtljn-47/?prefix=obj8 2023-11-02T18:16:45.113 DEBUG:boto:Data: 2023-11-02T18:16:45.113 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.114 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.114 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.115 DEBUG:boto:Params: {} 2023-11-02T18:16:45.115 DEBUG:boto:Token: None 2023-11-02T18:16:45.116 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-47/ 2023-11-02T18:16:45.116 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BCv+1j+JVgmsmyxdfpR9BgWyX/Q= 2023-11-02T18:16:45.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BCv+1j+JVgmsmyxdfpR9BgWyX/Q=', 'Content-Length': '0'} 2023-11-02T18:16:45.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d967792acb0407f-006543e78d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.122 DEBUG:boto:b'ibtljn-47obj81000falseobj82023-11-02T18:15:24.960Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:16:45.125 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-47 zones={a2, a1} 2023-11-02T18:16:45.125 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-11-02T18:16:45.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:16:45.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:16:45.127 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:16:45.171 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:16:45.171 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:16:45.272 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:16:45.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.272 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002903", 2023-11-02T18:16:45.273 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:15:30.488807Z" 2023-11-02T18:16:45.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.273 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000007128", 2023-11-02T18:16:45.273 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:15:40.518745Z" 2023-11-02T18:16:45.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.274 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000005159", 2023-11-02T18:16:45.274 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:15:24.591662Z" 2023-11-02T18:16:45.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.275 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-11-02T18:16:45.275 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:15:30.488973Z" 2023-11-02T18:16:45.275 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:45.276 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:16:45.277 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002903', 1: '00000000000000000000:00000000000000007128', 2: '00000000000000000000:00000000000000005159', 3: '00000000000000000000:00000000000000004316'} 2023-11-02T18:16:45.277 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:16:45.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:16:45.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:16:45.279 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:16:45.326 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:16:45.327 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:16:45.413 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:16:45.413 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:16:45.414 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:16:45.414 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:16:45.414 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:16:45.415 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 14562503763241451158 2023-11-02T18:16:45.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.415 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:16:45.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.415 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:16:45.416 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:16:45.416 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:16:45.416 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002903", 2023-11-02T18:16:45.416 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:16:45.417 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:16:45.417 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:16:45.417 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:15:30.488807Z" 2023-11-02T18:16:45.417 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:45.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.418 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:16:45.418 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:16:45.418 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:16:45.419 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000007128", 2023-11-02T18:16:45.419 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:16:45.419 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:16:45.419 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:16:45.419 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:15:40.518745Z" 2023-11-02T18:16:45.420 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:45.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.420 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:16:45.420 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:16:45.421 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:16:45.421 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000005159", 2023-11-02T18:16:45.421 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:16:45.421 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:16:45.421 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:16:45.422 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:15:24.591662Z" 2023-11-02T18:16:45.422 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:45.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.423 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:16:45.423 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:16:45.423 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:16:45.423 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-11-02T18:16:45.423 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:16:45.424 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:16:45.424 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:16:45.424 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:15:30.488973Z" 2023-11-02T18:16:45.424 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:45.425 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:45.425 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:16:45.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.425 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:16:45.425 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:16:45.426 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:16:45.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.426 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:16:45Z" 2023-11-02T18:16:45.426 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:16:45.427 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14562503763241451158 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002903", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:15:30.488807Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007128", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:15:40.518745Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005159", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:15:24.591662Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004316", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:15:30.488973Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:16:45Z" } 2023-11-02T18:16:45.428 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002903', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:15:30.488807Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007128', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:15:40.518745Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005159', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:15:24.591662Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004316', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:15:30.488973Z'}}] 2023-11-02T18:16:45.428 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002903', 1: '00000000000000000000:00000000000000007128', 2: '00000000000000000000:00000000000000005159', 3: '00000000000000000000:00000000000000004316'} 2023-11-02T18:16:45.429 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002903', 1: '00000000000000000000:00000000000000007128', 2: '00000000000000000000:00000000000000005159', 3: '00000000000000000000:00000000000000004316'} 2023-11-02T18:16:45.429 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:16:45.430 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-11-02T18:16:45.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:16:45.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:16:45.431 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:16:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:16:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:16:45.570 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:16:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001775", 2023-11-02T18:16:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:15:40.560486Z" 2023-11-02T18:16:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000006141", 2023-11-02T18:16:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:15:40.562919Z" 2023-11-02T18:16:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002624", 2023-11-02T18:16:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:13:45.360850Z" 2023-11-02T18:16:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-11-02T18:16:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:15:40.560965Z" 2023-11-02T18:16:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:45.575 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:16:45.576 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001775', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000002624', 3: '00000000000000000000:00000000000000002620'} 2023-11-02T18:16:45.576 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:16:45.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:16:45.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:16:45.578 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:16:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:16:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:16:45.702 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:16:45.703 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:16:45.703 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:16:45.703 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:16:45.703 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:16:45.704 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:16:45.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.704 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:16:45.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.705 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:16:45.705 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:16:45.705 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:16:45.705 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000001775", 2023-11-02T18:16:45.705 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:16:45.706 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:16:45.706 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:16:45.706 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:15:40.560486Z" 2023-11-02T18:16:45.706 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:45.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.707 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:16:45.707 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:16:45.707 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:16:45.708 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000006141", 2023-11-02T18:16:45.708 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:16:45.708 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:16:45.708 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:16:45.708 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:15:40.562919Z" 2023-11-02T18:16:45.709 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:45.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.709 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:16:45.709 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:16:45.710 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:16:45.710 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002624", 2023-11-02T18:16:45.710 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:16:45.710 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:16:45.711 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:16:45.711 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:13:45.360850Z" 2023-11-02T18:16:45.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:45.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:45.712 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:16:45.712 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:16:45.712 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:16:45.712 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-11-02T18:16:45.713 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:16:45.713 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:16:45.713 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:16:45.713 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:15:40.560965Z" 2023-11-02T18:16:45.713 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:45.714 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:45.714 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:16:45.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.714 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:16:45.714 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:16:45.715 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:16:45.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:45.715 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:16:45Z" 2023-11-02T18:16:45.715 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:16:45.716 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001775", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:15:40.560486Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006141", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:15:40.562919Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:13:45.360850Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002620", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:15:40.560965Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:16:45Z" } 2023-11-02T18:16:45.717 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001775', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:15:40.560486Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006141', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:15:40.562919Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:13:45.360850Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002620', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:15:40.560965Z'}}] 2023-11-02T18:16:45.717 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001775', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000002624', 3: '00000000000000000000:00000000000000002620'} 2023-11-02T18:16:45.718 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001775', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000002624', 3: '00000000000000000000:00000000000000002620'} 2023-11-02T18:16:45.718 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:16:45.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2023-11-02T18:16:45.720 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-48 2023-11-02T18:16:45.721 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:16:45.721 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:16:45.722 DEBUG:boto:Method: PUT 2023-11-02T18:16:45.722 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:16:45.723 DEBUG:boto:Data: 2023-11-02T18:16:45.723 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.723 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.724 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.724 DEBUG:boto:Params: {} 2023-11-02T18:16:45.725 DEBUG:boto:Token: None 2023-11-02T18:16:45.725 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-48/ 2023-11-02T18:16:45.726 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xLoe/0DU1L2N18YSdK2YI9u/T8g= 2023-11-02T18:16:45.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xLoe/0DU1L2N18YSdK2YI9u/T8g=', 'Content-Length': '0'} 2023-11-02T18:16:45.758 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a01a041a97a17092-006543e78d-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.758 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-49 2023-11-02T18:16:45.759 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:16:45.759 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:16:45.760 DEBUG:boto:Method: PUT 2023-11-02T18:16:45.760 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:16:45.760 DEBUG:boto:Data: 2023-11-02T18:16:45.761 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.761 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:45.762 DEBUG:boto:Port: 8001 2023-11-02T18:16:45.762 DEBUG:boto:Params: {} 2023-11-02T18:16:45.763 DEBUG:boto:Token: None 2023-11-02T18:16:45.763 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-49/ 2023-11-02T18:16:45.764 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vCmay7vsIhlbTptLL8NAJ8JO2EQ= 2023-11-02T18:16:45.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vCmay7vsIhlbTptLL8NAJ8JO2EQ=', 'Content-Length': '0'} 2023-11-02T18:16:45.810 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ed2c5681bf190b58-006543e78d-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.811 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:16:45.811 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:16:45.812 DEBUG:boto:Method: HEAD 2023-11-02T18:16:45.812 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:16:45.813 DEBUG:boto:Data: 2023-11-02T18:16:45.813 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.813 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.814 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.814 DEBUG:boto:Params: {} 2023-11-02T18:16:45.815 DEBUG:boto:Token: None 2023-11-02T18:16:45.815 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-48/ 2023-11-02T18:16:45.816 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QqlTci9Xp3cLHJot+IVh93KLk70= 2023-11-02T18:16:45.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QqlTci9Xp3cLHJot+IVh93KLk70=', 'Content-Length': '0'} 2023-11-02T18:16:45.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b9e65a109cc281c-006543e78d-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.820 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:16:45.821 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:16:45.821 DEBUG:boto:Method: PUT 2023-11-02T18:16:45.822 DEBUG:boto:Path: /ibtljn-48/obj1 2023-11-02T18:16:45.822 DEBUG:boto:Data: 2023-11-02T18:16:45.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:16:45.823 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.823 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.824 DEBUG:boto:Params: {} 2023-11-02T18:16:45.824 DEBUG:boto:Token: None 2023-11-02T18:16:45.825 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-48/obj1 2023-11-02T18:16:45.825 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e3pW8q0WkqIm01SddmxU8oI09DM= 2023-11-02T18:16:45.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e3pW8q0WkqIm01SddmxU8oI09DM='} 2023-11-02T18:16:45.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8eeca00866866b5-006543e78d-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.846 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:16:45.846 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:16:45.847 DEBUG:boto:Method: HEAD 2023-11-02T18:16:45.847 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:16:45.848 DEBUG:boto:Data: 2023-11-02T18:16:45.848 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.848 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.849 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.849 DEBUG:boto:Params: {} 2023-11-02T18:16:45.850 DEBUG:boto:Token: None 2023-11-02T18:16:45.850 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-48/ 2023-11-02T18:16:45.851 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QqlTci9Xp3cLHJot+IVh93KLk70= 2023-11-02T18:16:45.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QqlTci9Xp3cLHJot+IVh93KLk70=', 'Content-Length': '0'} 2023-11-02T18:16:45.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066d5fea3bb2af77c-006543e78d-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.855 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:16:45.855 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:16:45.855 DEBUG:boto:Method: PUT 2023-11-02T18:16:45.856 DEBUG:boto:Path: /ibtljn-48/obj2 2023-11-02T18:16:45.856 DEBUG:boto:Data: 2023-11-02T18:16:45.857 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:16:45.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.858 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.858 DEBUG:boto:Params: {} 2023-11-02T18:16:45.858 DEBUG:boto:Token: None 2023-11-02T18:16:45.859 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-48/obj2 2023-11-02T18:16:45.859 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRaJgyWhGYANOM618Ak2maKgiiU= 2023-11-02T18:16:45.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRaJgyWhGYANOM618Ak2maKgiiU='} 2023-11-02T18:16:45.907 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012fbe52be31617d7-006543e78d-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.907 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:16:45.908 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:16:45.908 DEBUG:boto:Method: HEAD 2023-11-02T18:16:45.909 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:16:45.909 DEBUG:boto:Data: 2023-11-02T18:16:45.910 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.910 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.911 DEBUG:boto:Params: {} 2023-11-02T18:16:45.911 DEBUG:boto:Token: None 2023-11-02T18:16:45.912 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-48/ 2023-11-02T18:16:45.912 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QqlTci9Xp3cLHJot+IVh93KLk70= 2023-11-02T18:16:45.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QqlTci9Xp3cLHJot+IVh93KLk70=', 'Content-Length': '0'} 2023-11-02T18:16:45.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3d14c4a948df943-006543e78d-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.918 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:16:45.918 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:16:45.918 DEBUG:boto:Method: PUT 2023-11-02T18:16:45.919 DEBUG:boto:Path: /ibtljn-48/obj3 2023-11-02T18:16:45.919 DEBUG:boto:Data: 2023-11-02T18:16:45.920 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:16:45.920 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.921 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.921 DEBUG:boto:Params: {} 2023-11-02T18:16:45.921 DEBUG:boto:Token: None 2023-11-02T18:16:45.922 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-48/obj3 2023-11-02T18:16:45.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xDrjSRH8NItZafXDyOgSygFpld0= 2023-11-02T18:16:45.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xDrjSRH8NItZafXDyOgSygFpld0='} 2023-11-02T18:16:45.935 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000199fdae8353a9ca4-006543e78d-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.936 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:16:45.936 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:16:45.937 DEBUG:boto:Method: HEAD 2023-11-02T18:16:45.937 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:16:45.938 DEBUG:boto:Data: 2023-11-02T18:16:45.938 DEBUG:boto:Headers: {} 2023-11-02T18:16:45.938 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.939 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.939 DEBUG:boto:Params: {} 2023-11-02T18:16:45.940 DEBUG:boto:Token: None 2023-11-02T18:16:45.940 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-48/ 2023-11-02T18:16:45.941 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QqlTci9Xp3cLHJot+IVh93KLk70= 2023-11-02T18:16:45.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QqlTci9Xp3cLHJot+IVh93KLk70=', 'Content-Length': '0'} 2023-11-02T18:16:45.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007b6b83a3b173853-006543e78d-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.946 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:16:45.949 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:16:45.950 DEBUG:boto:Method: PUT 2023-11-02T18:16:45.950 DEBUG:boto:Path: /ibtljn-48/obj4 2023-11-02T18:16:45.951 DEBUG:boto:Data: 2023-11-02T18:16:45.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:16:45.951 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:16:45.952 DEBUG:boto:Port: 8000 2023-11-02T18:16:45.952 DEBUG:boto:Params: {} 2023-11-02T18:16:45.953 DEBUG:boto:Token: None 2023-11-02T18:16:45.953 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:16:45 GMT /ibtljn-48/obj4 2023-11-02T18:16:45.954 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MczD4AzxCCz3gBKr7/lPcuRNFIU= 2023-11-02T18:16:45.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:16:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MczD4AzxCCz3gBKr7/lPcuRNFIU='} 2023-11-02T18:16:45.999 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000025e5bb6f835ada-006543e78d-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:45.999 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:16:46.000 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:16:46.000 DEBUG:boto:Method: HEAD 2023-11-02T18:16:46.001 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:16:46.001 DEBUG:boto:Data: 2023-11-02T18:16:46.002 DEBUG:boto:Headers: {} 2023-11-02T18:16:46.002 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:46.002 DEBUG:boto:Port: 8001 2023-11-02T18:16:46.003 DEBUG:boto:Params: {} 2023-11-02T18:16:46.003 DEBUG:boto:Token: None 2023-11-02T18:16:46.004 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:46 GMT /ibtljn-49/ 2023-11-02T18:16:46.004 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R5BtU8Avt+gZ+KYlSR6uaQX5iog= 2023-11-02T18:16:46.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R5BtU8Avt+gZ+KYlSR6uaQX5iog=', 'Content-Length': '0'} 2023-11-02T18:16:46.009 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3d6b25fd95a7c00-006543e78d-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:46.010 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:16:46.010 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:16:46.010 DEBUG:boto:Method: PUT 2023-11-02T18:16:46.011 DEBUG:boto:Path: /ibtljn-49/obj1 2023-11-02T18:16:46.011 DEBUG:boto:Data: 2023-11-02T18:16:46.012 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:16:46.012 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:46.013 DEBUG:boto:Port: 8001 2023-11-02T18:16:46.013 DEBUG:boto:Params: {} 2023-11-02T18:16:46.013 DEBUG:boto:Token: None 2023-11-02T18:16:46.014 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:16:46 GMT /ibtljn-49/obj1 2023-11-02T18:16:46.014 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HqWJQLtZEgPSi/LeLS08eTMwGJQ= 2023-11-02T18:16:46.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:16:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HqWJQLtZEgPSi/LeLS08eTMwGJQ='} 2023-11-02T18:16:46.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029e7847095b2884c-006543e78e-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:46.030 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:16:46.030 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:16:46.031 DEBUG:boto:Method: HEAD 2023-11-02T18:16:46.031 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:16:46.031 DEBUG:boto:Data: 2023-11-02T18:16:46.032 DEBUG:boto:Headers: {} 2023-11-02T18:16:46.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:46.033 DEBUG:boto:Port: 8001 2023-11-02T18:16:46.033 DEBUG:boto:Params: {} 2023-11-02T18:16:46.034 DEBUG:boto:Token: None 2023-11-02T18:16:46.034 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:46 GMT /ibtljn-49/ 2023-11-02T18:16:46.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R5BtU8Avt+gZ+KYlSR6uaQX5iog= 2023-11-02T18:16:46.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R5BtU8Avt+gZ+KYlSR6uaQX5iog=', 'Content-Length': '0'} 2023-11-02T18:16:46.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027e6e756b120ceba-006543e78e-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:46.040 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:16:46.040 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:16:46.041 DEBUG:boto:Method: PUT 2023-11-02T18:16:46.041 DEBUG:boto:Path: /ibtljn-49/obj2 2023-11-02T18:16:46.042 DEBUG:boto:Data: 2023-11-02T18:16:46.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:16:46.042 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:46.043 DEBUG:boto:Port: 8001 2023-11-02T18:16:46.043 DEBUG:boto:Params: {} 2023-11-02T18:16:46.044 DEBUG:boto:Token: None 2023-11-02T18:16:46.044 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:16:46 GMT /ibtljn-49/obj2 2023-11-02T18:16:46.045 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWJwpLQFkytOttHT75c6KssNbJU= 2023-11-02T18:16:46.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:16:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWJwpLQFkytOttHT75c6KssNbJU='} 2023-11-02T18:16:46.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064e4574482aaf708-006543e78e-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:46.095 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:16:46.096 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:16:46.096 DEBUG:boto:Method: HEAD 2023-11-02T18:16:46.097 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:16:46.097 DEBUG:boto:Data: 2023-11-02T18:16:46.097 DEBUG:boto:Headers: {} 2023-11-02T18:16:46.098 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:46.098 DEBUG:boto:Port: 8001 2023-11-02T18:16:46.099 DEBUG:boto:Params: {} 2023-11-02T18:16:46.099 DEBUG:boto:Token: None 2023-11-02T18:16:46.100 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:46 GMT /ibtljn-49/ 2023-11-02T18:16:46.100 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R5BtU8Avt+gZ+KYlSR6uaQX5iog= 2023-11-02T18:16:46.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R5BtU8Avt+gZ+KYlSR6uaQX5iog=', 'Content-Length': '0'} 2023-11-02T18:16:46.105 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000589856d090772b3d-006543e78e-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:46.105 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:16:46.106 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:16:46.106 DEBUG:boto:Method: PUT 2023-11-02T18:16:46.107 DEBUG:boto:Path: /ibtljn-49/obj3 2023-11-02T18:16:46.107 DEBUG:boto:Data: 2023-11-02T18:16:46.107 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:16:46.108 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:46.108 DEBUG:boto:Port: 8001 2023-11-02T18:16:46.109 DEBUG:boto:Params: {} 2023-11-02T18:16:46.109 DEBUG:boto:Token: None 2023-11-02T18:16:46.110 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:16:46 GMT /ibtljn-49/obj3 2023-11-02T18:16:46.110 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lFEsv/Q8k3snn2+w6dAFR9s7na4= 2023-11-02T18:16:46.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:16:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lFEsv/Q8k3snn2+w6dAFR9s7na4='} 2023-11-02T18:16:46.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000312f6d7f7cbe9b1a-006543e78e-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:46.124 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:16:46.124 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:16:46.125 DEBUG:boto:Method: HEAD 2023-11-02T18:16:46.125 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:16:46.126 DEBUG:boto:Data: 2023-11-02T18:16:46.126 DEBUG:boto:Headers: {} 2023-11-02T18:16:46.127 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:46.127 DEBUG:boto:Port: 8001 2023-11-02T18:16:46.128 DEBUG:boto:Params: {} 2023-11-02T18:16:46.128 DEBUG:boto:Token: None 2023-11-02T18:16:46.129 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:16:46 GMT /ibtljn-49/ 2023-11-02T18:16:46.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R5BtU8Avt+gZ+KYlSR6uaQX5iog= 2023-11-02T18:16:46.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:16:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R5BtU8Avt+gZ+KYlSR6uaQX5iog=', 'Content-Length': '0'} 2023-11-02T18:16:46.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb4969703430dfe9-006543e78e-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:46.135 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:16:46.135 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:16:46.136 DEBUG:boto:Method: PUT 2023-11-02T18:16:46.136 DEBUG:boto:Path: /ibtljn-49/obj4 2023-11-02T18:16:46.137 DEBUG:boto:Data: 2023-11-02T18:16:46.137 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:16:46.137 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:16:46.138 DEBUG:boto:Port: 8001 2023-11-02T18:16:46.138 DEBUG:boto:Params: {} 2023-11-02T18:16:46.139 DEBUG:boto:Token: None 2023-11-02T18:16:46.139 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:16:46 GMT /ibtljn-49/obj4 2023-11-02T18:16:46.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ByPkDZ8X1pCjvqlchDSAdVyvodM= 2023-11-02T18:16:46.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:16:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ByPkDZ8X1pCjvqlchDSAdVyvodM='} 2023-11-02T18:16:46.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f62b175c356ce4e1-006543e78e-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:16:46.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:16:46.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:16:46.184 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:16:46.230 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:16:46.230 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:16:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:16:46.314 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:16:46.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:46.314 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:16:46.315 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:16:46.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:46.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:46.315 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:16:46.315 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:16:46.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:46.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:46.316 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949005.787608_369.1", 2023-11-02T18:16:46.316 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:16:45.787608Z" 2023-11-02T18:16:46.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:46.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:46.317 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:16:46.317 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:16:46.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:46.318 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:16:46.319 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949005.787608_369.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:16:46.319 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:16:46.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:16:46.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:16:46.321 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:16:46.366 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:16:46.366 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:16:46.424 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:46.414+0000 7f04d1367f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:16:46.425 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:46.418+0000 7f04d1367f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:16:46.453 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:16:46.453 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:16:46.454 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:16:46.454 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:16:46.454 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:16:46.454 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:16:46.455 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:16:46.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:46.455 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:16:46.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:46.456 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:16:46.456 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:16:46.456 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:16:46.456 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:16:46.456 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:16:46.457 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:16:46.457 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:16:46.457 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:16:46.457 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:16:46.457 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:46.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:46.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:46.458 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:16:46.458 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:16:46.459 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:16:46.459 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:16:46.459 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:16:46.459 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:16:46.459 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:16:46.460 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:16:46.460 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:16:46.460 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:46.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:46.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:46.461 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:16:46.461 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:16:46.461 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:16:46.461 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949005.787608_369.1", 2023-11-02T18:16:46.462 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:16:46.462 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:16:46.462 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:16:46.462 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:16:45.787608Z", 2023-11-02T18:16:46.463 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:16:46.463 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:46.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:46.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:16:46.463 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:16:46.464 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:16:46.464 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:16:46.464 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:16:46.464 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:16:46.464 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:16:46.465 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:16:46.465 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:16:46.465 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:16:46.465 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:46.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:16:46.466 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:16:46.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:46.466 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:16:46.467 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:16:46.467 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:16:46.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:16:46.467 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:16:46Z" 2023-11-02T18:16:46.467 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:16:46.468 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949005.787608_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:16:45.787608Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:16:46Z" } 2023-11-02T18:16:46.469 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949005.787608_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:16:45.787608Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:16:46.469 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949005.787608_369.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:16:46.470 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949005.787608_369.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:16:46.470 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:16:46.471 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-48 2023-11-02T18:16:46.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:16:46.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:16:46.472 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:16:46.524 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:16:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:16:46.587 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:46.578+0000 7efc399a4f40 1 waiting to reach incremental sync.. 2023-11-02T18:16:51.588 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:51.578+0000 7efc399a4f40 1 waiting to reach incremental sync.. 2023-11-02T18:16:56.589 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:16:56.582+0000 7efc399a4f40 1 waiting to reach incremental sync.. 2023-11-02T18:17:01.594 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:01.586+0000 7efc399a4f40 1 bucket sync caught up with source: 2023-11-02T18:17:01.594 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.26.6, , 00000000001.19.6, , 00000000001.26.6, , , 00000000001.31.6, , , ] 2023-11-02T18:17:01.595 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.26.6, , 00000000001.19.6, , 00000000001.26.6, , , 00000000001.31.6, , , ] 2023-11-02T18:17:01.595 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:01.586+0000 7efc399a4f40 0 bucket checkpoint complete 2023-11-02T18:17:01.620 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-48 2023-11-02T18:17:01.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:17:01.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:17:01.621 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:17:01.670 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:17:01.671 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:17:01.730 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:01.722+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:17:06.731 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:06.722+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:17:11.732 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:11.722+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:17:16.733 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:16.726+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:17:21.734 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:21.726+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:17:26.735 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:26.726+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:17:31.736 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:31.730+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:17:36.738 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:36.730+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:17:41.739 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:41.730+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:17:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:46.730+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:17:51.740 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:51.734+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:17:56.742 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:17:56.734+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:18:01.742 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:01.734+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:18:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:06.734+0000 7f67608fdf40 1 waiting to reach incremental sync.. 2023-11-02T18:18:11.747 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:11.738+0000 7f67608fdf40 1 bucket sync caught up with source: 2023-11-02T18:18:11.747 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.413.6, , 00000000001.68.6, , 00000000001.303.6, , , 00000000001.203.6, , , ] 2023-11-02T18:18:11.747 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.413.6, , 00000000001.68.6, , 00000000001.303.6, , , 00000000001.203.6, , , ] 2023-11-02T18:18:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:11.738+0000 7f67608fdf40 0 bucket checkpoint complete 2023-11-02T18:18:11.777 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-48 zones={a2, a1} 2023-11-02T18:18:11.778 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:11.779 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:11.779 DEBUG:boto:Method: HEAD 2023-11-02T18:18:11.780 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:18:11.780 DEBUG:boto:Data: 2023-11-02T18:18:11.781 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.781 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:11.782 DEBUG:boto:Port: 8001 2023-11-02T18:18:11.782 DEBUG:boto:Params: {} 2023-11-02T18:18:11.783 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:18:11.784 DEBUG:boto:Token: None 2023-11-02T18:18:11.784 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/ 2023-11-02T18:18:11.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eWSurx++TBamoNy/IQkrcQk3O/U= 2023-11-02T18:18:11.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eWSurx++TBamoNy/IQkrcQk3O/U=', 'Content-Length': '0'} 2023-11-02T18:18:11.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004db18c8b503f591d-006543e7e3-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.792 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:11.793 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:11.793 DEBUG:boto:Method: HEAD 2023-11-02T18:18:11.794 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:18:11.794 DEBUG:boto:Data: 2023-11-02T18:18:11.795 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.795 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:11.795 DEBUG:boto:Port: 8000 2023-11-02T18:18:11.796 DEBUG:boto:Params: {} 2023-11-02T18:18:11.796 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:18:11.797 DEBUG:boto:Token: None 2023-11-02T18:18:11.798 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/ 2023-11-02T18:18:11.798 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eWSurx++TBamoNy/IQkrcQk3O/U= 2023-11-02T18:18:11.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eWSurx++TBamoNy/IQkrcQk3O/U=', 'Content-Length': '0'} 2023-11-02T18:18:11.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f68fa4c846dc0a80-006543e7e3-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.805 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:18:11.806 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:11.806 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:11.806 DEBUG:boto:path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:18:11.807 DEBUG:boto:auth_path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:18:11.807 DEBUG:boto:Method: GET 2023-11-02T18:18:11.808 DEBUG:boto:Path: /ibtljn-48/?versions&max-keys=999 2023-11-02T18:18:11.808 DEBUG:boto:Data: 2023-11-02T18:18:11.809 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.809 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:11.809 DEBUG:boto:Port: 8001 2023-11-02T18:18:11.810 DEBUG:boto:Params: {} 2023-11-02T18:18:11.810 DEBUG:boto:Token: None 2023-11-02T18:18:11.811 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/?versions 2023-11-02T18:18:11.811 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:clmLqitVfoH2Z+sL/WcDtiQfktg= 2023-11-02T18:18:11.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:clmLqitVfoH2Z+sL/WcDtiQfktg=', 'Content-Length': '0'} 2023-11-02T18:18:11.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0d9b2ff0588fd4d-006543e7e3-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.817 DEBUG:boto:b'ibtljn-48999falseobj1nulltrue2023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:11.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:18:11.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:18:11.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:18:11.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:18:11.820 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:18:11.820 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:11.820 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:11.821 DEBUG:boto:path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:18:11.821 DEBUG:boto:auth_path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:18:11.822 DEBUG:boto:Method: GET 2023-11-02T18:18:11.822 DEBUG:boto:Path: /ibtljn-48/?versions&max-keys=999 2023-11-02T18:18:11.823 DEBUG:boto:Data: 2023-11-02T18:18:11.823 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.823 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:11.824 DEBUG:boto:Port: 8000 2023-11-02T18:18:11.824 DEBUG:boto:Params: {} 2023-11-02T18:18:11.825 DEBUG:boto:Token: None 2023-11-02T18:18:11.825 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/?versions 2023-11-02T18:18:11.826 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:clmLqitVfoH2Z+sL/WcDtiQfktg= 2023-11-02T18:18:11.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:clmLqitVfoH2Z+sL/WcDtiQfktg=', 'Content-Length': '0'} 2023-11-02T18:18:11.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c4e59bc86932dbc-006543e7e3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.832 DEBUG:boto:b'ibtljn-48999falseobj1nulltrue2023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:11.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:18:11.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:18:11.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:18:11.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:18:11.834 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:11.835 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:11.835 DEBUG:boto:path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:18:11.836 DEBUG:boto:auth_path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:18:11.836 DEBUG:boto:Method: GET 2023-11-02T18:18:11.836 DEBUG:boto:Path: /ibtljn-48/?versions&max-keys=999 2023-11-02T18:18:11.837 DEBUG:boto:Data: 2023-11-02T18:18:11.837 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.838 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:11.838 DEBUG:boto:Port: 8001 2023-11-02T18:18:11.839 DEBUG:boto:Params: {} 2023-11-02T18:18:11.839 DEBUG:boto:Token: None 2023-11-02T18:18:11.840 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/?versions 2023-11-02T18:18:11.840 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:clmLqitVfoH2Z+sL/WcDtiQfktg= 2023-11-02T18:18:11.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:clmLqitVfoH2Z+sL/WcDtiQfktg=', 'Content-Length': '0'} 2023-11-02T18:18:11.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf99c800b3cf7713-006543e7e3-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.846 DEBUG:boto:b'ibtljn-48999falseobj1nulltrue2023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:11.847 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:11.847 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:11.848 DEBUG:boto:path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:18:11.848 DEBUG:boto:auth_path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:18:11.849 DEBUG:boto:Method: GET 2023-11-02T18:18:11.849 DEBUG:boto:Path: /ibtljn-48/?versions&max-keys=999 2023-11-02T18:18:11.850 DEBUG:boto:Data: 2023-11-02T18:18:11.850 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:11.851 DEBUG:boto:Port: 8000 2023-11-02T18:18:11.851 DEBUG:boto:Params: {} 2023-11-02T18:18:11.852 DEBUG:boto:Token: None 2023-11-02T18:18:11.852 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/?versions 2023-11-02T18:18:11.853 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:clmLqitVfoH2Z+sL/WcDtiQfktg= 2023-11-02T18:18:11.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:clmLqitVfoH2Z+sL/WcDtiQfktg=', 'Content-Length': '0'} 2023-11-02T18:18:11.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000207e8fb6a82092e1-006543e7e3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.860 DEBUG:boto:b'ibtljn-48999falseobj1nulltrue2023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:11.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:18:11.862 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:18:11.862 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:18:11.863 DEBUG:boto:path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.863 DEBUG:boto:auth_path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.863 DEBUG:boto:Method: GET 2023-11-02T18:18:11.864 DEBUG:boto:Path: /ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.864 DEBUG:boto:Data: 2023-11-02T18:18:11.865 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.866 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:11.866 DEBUG:boto:Port: 8001 2023-11-02T18:18:11.867 DEBUG:boto:Params: {} 2023-11-02T18:18:11.867 DEBUG:boto:Token: None 2023-11-02T18:18:11.868 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.869 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:it+fAtUo+/AHofU0dsHfIKurfPg= 2023-11-02T18:18:11.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:it+fAtUo+/AHofU0dsHfIKurfPg=', 'Content-Length': '0'} 2023-11-02T18:18:11.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000268afee0449af794-006543e7e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.914 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:18:11.915 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:18:11.916 DEBUG:boto:path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.916 DEBUG:boto:auth_path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.917 DEBUG:boto:Method: GET 2023-11-02T18:18:11.917 DEBUG:boto:Path: /ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.918 DEBUG:boto:Data: 2023-11-02T18:18:11.918 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.919 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:11.919 DEBUG:boto:Port: 8000 2023-11-02T18:18:11.919 DEBUG:boto:Params: {} 2023-11-02T18:18:11.920 DEBUG:boto:Token: None 2023-11-02T18:18:11.921 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.921 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:it+fAtUo+/AHofU0dsHfIKurfPg= 2023-11-02T18:18:11.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:it+fAtUo+/AHofU0dsHfIKurfPg=', 'Content-Length': '0'} 2023-11-02T18:18:11.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c2bb50022430932-006543e7e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.925 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:18:11.926 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:18:11.926 DEBUG:boto:path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.927 DEBUG:boto:auth_path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.927 DEBUG:boto:Method: HEAD 2023-11-02T18:18:11.928 DEBUG:boto:Path: /ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.928 DEBUG:boto:Data: 2023-11-02T18:18:11.928 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.929 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:11.929 DEBUG:boto:Port: 8001 2023-11-02T18:18:11.930 DEBUG:boto:Params: {} 2023-11-02T18:18:11.930 DEBUG:boto:Token: None 2023-11-02T18:18:11.931 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.931 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O2Ee14psKqkytbzQWOhP6gDxPnI= 2023-11-02T18:18:11.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O2Ee14psKqkytbzQWOhP6gDxPnI=', 'Content-Length': '0'} 2023-11-02T18:18:11.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8f7a98db35d77f7-006543e7e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.935 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:18:11.936 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:18:11.936 DEBUG:boto:path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.937 DEBUG:boto:auth_path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.937 DEBUG:boto:Method: HEAD 2023-11-02T18:18:11.937 DEBUG:boto:Path: /ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.938 DEBUG:boto:Data: 2023-11-02T18:18:11.938 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.939 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:11.939 DEBUG:boto:Port: 8000 2023-11-02T18:18:11.941 DEBUG:boto:Params: {} 2023-11-02T18:18:11.941 DEBUG:boto:Token: None 2023-11-02T18:18:11.942 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.942 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O2Ee14psKqkytbzQWOhP6gDxPnI= 2023-11-02T18:18:11.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O2Ee14psKqkytbzQWOhP6gDxPnI=', 'Content-Length': '0'} 2023-11-02T18:18:11.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ad30d5f40dc2675-006543e7e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:18:11.947 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:18:11.947 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:18:11.948 DEBUG:boto:path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.948 DEBUG:boto:auth_path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.949 DEBUG:boto:Method: GET 2023-11-02T18:18:11.949 DEBUG:boto:Path: /ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.950 DEBUG:boto:Data: 2023-11-02T18:18:11.950 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.951 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:11.951 DEBUG:boto:Port: 8001 2023-11-02T18:18:11.952 DEBUG:boto:Params: {} 2023-11-02T18:18:11.952 DEBUG:boto:Token: None 2023-11-02T18:18:11.953 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.953 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:it+fAtUo+/AHofU0dsHfIKurfPg= 2023-11-02T18:18:11.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:it+fAtUo+/AHofU0dsHfIKurfPg=', 'Content-Length': '0'} 2023-11-02T18:18:11.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d7b87db3c3d0507-006543e7e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.958 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:18:11.958 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:18:11.959 DEBUG:boto:path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.959 DEBUG:boto:auth_path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.960 DEBUG:boto:Method: GET 2023-11-02T18:18:11.960 DEBUG:boto:Path: /ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.961 DEBUG:boto:Data: 2023-11-02T18:18:11.961 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.962 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:11.962 DEBUG:boto:Port: 8000 2023-11-02T18:18:11.963 DEBUG:boto:Params: {} 2023-11-02T18:18:11.963 DEBUG:boto:Token: None 2023-11-02T18:18:11.964 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/obj1?versionId=null 2023-11-02T18:18:11.964 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:it+fAtUo+/AHofU0dsHfIKurfPg= 2023-11-02T18:18:11.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:it+fAtUo+/AHofU0dsHfIKurfPg=', 'Content-Length': '0'} 2023-11-02T18:18:11.967 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7c174b5fca797df-006543e7e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.968 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:11.969 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:11.969 DEBUG:boto:path=/ibtljn-48/?prefix=obj1 2023-11-02T18:18:11.970 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj1 2023-11-02T18:18:11.970 DEBUG:boto:Method: GET 2023-11-02T18:18:11.970 DEBUG:boto:Path: /ibtljn-48/?prefix=obj1 2023-11-02T18:18:11.971 DEBUG:boto:Data: 2023-11-02T18:18:11.971 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.972 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:11.972 DEBUG:boto:Port: 8001 2023-11-02T18:18:11.973 DEBUG:boto:Params: {} 2023-11-02T18:18:11.974 DEBUG:boto:Token: None 2023-11-02T18:18:11.974 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/ 2023-11-02T18:18:11.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ezekmP5ZWxvihJEz+Bk4zInJSfY= 2023-11-02T18:18:11.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ezekmP5ZWxvihJEz+Bk4zInJSfY=', 'Content-Length': '0'} 2023-11-02T18:18:11.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee50d4d7404b51bb-006543e7e3-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:11.981 DEBUG:boto:b'ibtljn-48obj11000falseobj12023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:11.982 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:11.982 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:11.983 DEBUG:boto:path=/ibtljn-48/?prefix=obj1 2023-11-02T18:18:11.983 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj1 2023-11-02T18:18:11.984 DEBUG:boto:Method: GET 2023-11-02T18:18:11.984 DEBUG:boto:Path: /ibtljn-48/?prefix=obj1 2023-11-02T18:18:11.985 DEBUG:boto:Data: 2023-11-02T18:18:11.985 DEBUG:boto:Headers: {} 2023-11-02T18:18:11.986 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:11.986 DEBUG:boto:Port: 8000 2023-11-02T18:18:11.987 DEBUG:boto:Params: {} 2023-11-02T18:18:11.987 DEBUG:boto:Token: None 2023-11-02T18:18:11.988 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:11 GMT /ibtljn-48/ 2023-11-02T18:18:11.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ezekmP5ZWxvihJEz+Bk4zInJSfY= 2023-11-02T18:18:11.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ezekmP5ZWxvihJEz+Bk4zInJSfY=', 'Content-Length': '0'} 2023-11-02T18:18:11.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004acc899a498ae7d5-006543e7e3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.034 DEBUG:boto:b'ibtljn-48obj11000falseobj12023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.035 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:18:12.036 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:18:12.036 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.037 DEBUG:boto:Path: /ibtljn-48/obj1 2023-11-02T18:18:12.037 DEBUG:boto:Data: 2023-11-02T18:18:12.038 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.039 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.039 DEBUG:boto:Params: {} 2023-11-02T18:18:12.040 DEBUG:boto:Token: None 2023-11-02T18:18:12.040 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj1 2023-11-02T18:18:12.041 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JTGfarKGq1ZxtlZG+1aRn1xYN8M= 2023-11-02T18:18:12.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JTGfarKGq1ZxtlZG+1aRn1xYN8M=', 'Content-Length': '0'} 2023-11-02T18:18:12.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000755f80f14c989af2-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.044 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:18:12.045 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:18:12.045 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.046 DEBUG:boto:Path: /ibtljn-48/obj1 2023-11-02T18:18:12.046 DEBUG:boto:Data: 2023-11-02T18:18:12.047 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.047 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.048 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.048 DEBUG:boto:Params: {} 2023-11-02T18:18:12.048 DEBUG:boto:Token: None 2023-11-02T18:18:12.049 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj1 2023-11-02T18:18:12.049 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JTGfarKGq1ZxtlZG+1aRn1xYN8M= 2023-11-02T18:18:12.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JTGfarKGq1ZxtlZG+1aRn1xYN8M=', 'Content-Length': '0'} 2023-11-02T18:18:12.053 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007acf6a04a336faa9-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:18:12.054 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:18:12.054 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:18:12.055 DEBUG:boto:Method: GET 2023-11-02T18:18:12.055 DEBUG:boto:Path: /ibtljn-48/obj1 2023-11-02T18:18:12.056 DEBUG:boto:Data: 2023-11-02T18:18:12.056 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.057 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.057 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.058 DEBUG:boto:Params: {} 2023-11-02T18:18:12.058 DEBUG:boto:Token: None 2023-11-02T18:18:12.059 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj1 2023-11-02T18:18:12.059 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VozHsGJnotGv3I7sXUoakV0Hc3w= 2023-11-02T18:18:12.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VozHsGJnotGv3I7sXUoakV0Hc3w=', 'Content-Length': '0'} 2023-11-02T18:18:12.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026e17e7fa0ea7a88-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.106 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:18:12.107 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:18:12.107 DEBUG:boto:Method: GET 2023-11-02T18:18:12.108 DEBUG:boto:Path: /ibtljn-48/obj1 2023-11-02T18:18:12.108 DEBUG:boto:Data: 2023-11-02T18:18:12.109 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.109 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.110 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.110 DEBUG:boto:Params: {} 2023-11-02T18:18:12.111 DEBUG:boto:Token: None 2023-11-02T18:18:12.111 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj1 2023-11-02T18:18:12.112 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VozHsGJnotGv3I7sXUoakV0Hc3w= 2023-11-02T18:18:12.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VozHsGJnotGv3I7sXUoakV0Hc3w=', 'Content-Length': '0'} 2023-11-02T18:18:12.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f373ef50b30979b-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.116 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.117 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.117 DEBUG:boto:path=/ibtljn-48/?prefix=obj1 2023-11-02T18:18:12.118 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj1 2023-11-02T18:18:12.118 DEBUG:boto:Method: GET 2023-11-02T18:18:12.119 DEBUG:boto:Path: /ibtljn-48/?prefix=obj1 2023-11-02T18:18:12.119 DEBUG:boto:Data: 2023-11-02T18:18:12.119 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.120 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.120 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.121 DEBUG:boto:Params: {} 2023-11-02T18:18:12.121 DEBUG:boto:Token: None 2023-11-02T18:18:12.122 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.122 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009027683755bae5f3-006543e7e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.128 DEBUG:boto:b'ibtljn-48obj11000falseobj12023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.129 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.129 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.130 DEBUG:boto:path=/ibtljn-48/?prefix=obj1 2023-11-02T18:18:12.130 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj1 2023-11-02T18:18:12.131 DEBUG:boto:Method: GET 2023-11-02T18:18:12.131 DEBUG:boto:Path: /ibtljn-48/?prefix=obj1 2023-11-02T18:18:12.132 DEBUG:boto:Data: 2023-11-02T18:18:12.132 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.132 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.133 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.133 DEBUG:boto:Params: {} 2023-11-02T18:18:12.134 DEBUG:boto:Token: None 2023-11-02T18:18:12.134 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.135 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f610125eda0fbafe-006543e7e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.140 DEBUG:boto:b'ibtljn-48obj11000falseobj12023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:18:12.141 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:18:12.142 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:18:12.142 DEBUG:boto:path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.143 DEBUG:boto:auth_path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.143 DEBUG:boto:Method: GET 2023-11-02T18:18:12.145 DEBUG:boto:Path: /ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.145 DEBUG:boto:Data: 2023-11-02T18:18:12.146 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.146 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.147 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.147 DEBUG:boto:Params: {} 2023-11-02T18:18:12.148 DEBUG:boto:Token: None 2023-11-02T18:18:12.148 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.149 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QOHiQ5ezVlO7E+I7kYTBQGx14sg= 2023-11-02T18:18:12.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QOHiQ5ezVlO7E+I7kYTBQGx14sg=', 'Content-Length': '0'} 2023-11-02T18:18:12.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca6e1ed55d9f8a58-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.153 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:18:12.154 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:18:12.154 DEBUG:boto:path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.155 DEBUG:boto:auth_path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.155 DEBUG:boto:Method: GET 2023-11-02T18:18:12.155 DEBUG:boto:Path: /ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.156 DEBUG:boto:Data: 2023-11-02T18:18:12.156 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.157 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.157 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.158 DEBUG:boto:Params: {} 2023-11-02T18:18:12.158 DEBUG:boto:Token: None 2023-11-02T18:18:12.159 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.159 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QOHiQ5ezVlO7E+I7kYTBQGx14sg= 2023-11-02T18:18:12.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QOHiQ5ezVlO7E+I7kYTBQGx14sg=', 'Content-Length': '0'} 2023-11-02T18:18:12.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c709e8a99afb679-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.206 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:18:12.207 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:18:12.207 DEBUG:boto:path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.208 DEBUG:boto:auth_path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.208 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.209 DEBUG:boto:Path: /ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.209 DEBUG:boto:Data: 2023-11-02T18:18:12.210 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.210 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.211 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.211 DEBUG:boto:Params: {} 2023-11-02T18:18:12.212 DEBUG:boto:Token: None 2023-11-02T18:18:12.212 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.213 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IeOKPYdHDaRv0IXMOmX/VCWr2kw= 2023-11-02T18:18:12.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IeOKPYdHDaRv0IXMOmX/VCWr2kw=', 'Content-Length': '0'} 2023-11-02T18:18:12.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ebec2a100be0d51-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.217 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:18:12.217 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:18:12.218 DEBUG:boto:path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.218 DEBUG:boto:auth_path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.219 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.219 DEBUG:boto:Path: /ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.220 DEBUG:boto:Data: 2023-11-02T18:18:12.220 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.220 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.221 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.221 DEBUG:boto:Params: {} 2023-11-02T18:18:12.222 DEBUG:boto:Token: None 2023-11-02T18:18:12.222 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.223 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IeOKPYdHDaRv0IXMOmX/VCWr2kw= 2023-11-02T18:18:12.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IeOKPYdHDaRv0IXMOmX/VCWr2kw=', 'Content-Length': '0'} 2023-11-02T18:18:12.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0c41454af18848d-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:18:12.227 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:18:12.227 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:18:12.228 DEBUG:boto:path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.228 DEBUG:boto:auth_path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.229 DEBUG:boto:Method: GET 2023-11-02T18:18:12.229 DEBUG:boto:Path: /ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.230 DEBUG:boto:Data: 2023-11-02T18:18:12.230 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.231 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.232 DEBUG:boto:Params: {} 2023-11-02T18:18:12.232 DEBUG:boto:Token: None 2023-11-02T18:18:12.233 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.233 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QOHiQ5ezVlO7E+I7kYTBQGx14sg= 2023-11-02T18:18:12.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QOHiQ5ezVlO7E+I7kYTBQGx14sg=', 'Content-Length': '0'} 2023-11-02T18:18:12.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062b71a3d3b5c7a21-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.278 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:18:12.279 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:18:12.279 DEBUG:boto:path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.280 DEBUG:boto:auth_path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.280 DEBUG:boto:Method: GET 2023-11-02T18:18:12.281 DEBUG:boto:Path: /ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.281 DEBUG:boto:Data: 2023-11-02T18:18:12.282 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.282 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.283 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.283 DEBUG:boto:Params: {} 2023-11-02T18:18:12.284 DEBUG:boto:Token: None 2023-11-02T18:18:12.284 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj2?versionId=null 2023-11-02T18:18:12.285 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QOHiQ5ezVlO7E+I7kYTBQGx14sg= 2023-11-02T18:18:12.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QOHiQ5ezVlO7E+I7kYTBQGx14sg=', 'Content-Length': '0'} 2023-11-02T18:18:12.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a65c4908500eec24-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.289 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.289 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.290 DEBUG:boto:path=/ibtljn-48/?prefix=obj2 2023-11-02T18:18:12.290 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj2 2023-11-02T18:18:12.291 DEBUG:boto:Method: GET 2023-11-02T18:18:12.291 DEBUG:boto:Path: /ibtljn-48/?prefix=obj2 2023-11-02T18:18:12.292 DEBUG:boto:Data: 2023-11-02T18:18:12.292 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.293 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.293 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.294 DEBUG:boto:Params: {} 2023-11-02T18:18:12.294 DEBUG:boto:Token: None 2023-11-02T18:18:12.295 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.295 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d410004cfccae744-006543e7e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.301 DEBUG:boto:b'ibtljn-48obj21000falseobj22023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.302 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.303 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.303 DEBUG:boto:path=/ibtljn-48/?prefix=obj2 2023-11-02T18:18:12.304 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj2 2023-11-02T18:18:12.304 DEBUG:boto:Method: GET 2023-11-02T18:18:12.305 DEBUG:boto:Path: /ibtljn-48/?prefix=obj2 2023-11-02T18:18:12.305 DEBUG:boto:Data: 2023-11-02T18:18:12.306 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.306 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.306 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.307 DEBUG:boto:Params: {} 2023-11-02T18:18:12.307 DEBUG:boto:Token: None 2023-11-02T18:18:12.308 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.308 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b16a8fa9d0544850-006543e7e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.314 DEBUG:boto:b'ibtljn-48obj21000falseobj22023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.314 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:18:12.315 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:18:12.315 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.316 DEBUG:boto:Path: /ibtljn-48/obj2 2023-11-02T18:18:12.316 DEBUG:boto:Data: 2023-11-02T18:18:12.316 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.317 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.318 DEBUG:boto:Params: {} 2023-11-02T18:18:12.318 DEBUG:boto:Token: None 2023-11-02T18:18:12.319 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj2 2023-11-02T18:18:12.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iuTalEs/kwooZvD2S/QWgUq2V7Y= 2023-11-02T18:18:12.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iuTalEs/kwooZvD2S/QWgUq2V7Y=', 'Content-Length': '0'} 2023-11-02T18:18:12.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c20841024fbb14e-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.324 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:18:12.324 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:18:12.324 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.325 DEBUG:boto:Path: /ibtljn-48/obj2 2023-11-02T18:18:12.325 DEBUG:boto:Data: 2023-11-02T18:18:12.326 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.326 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.327 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.327 DEBUG:boto:Params: {} 2023-11-02T18:18:12.328 DEBUG:boto:Token: None 2023-11-02T18:18:12.328 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj2 2023-11-02T18:18:12.329 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iuTalEs/kwooZvD2S/QWgUq2V7Y= 2023-11-02T18:18:12.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iuTalEs/kwooZvD2S/QWgUq2V7Y=', 'Content-Length': '0'} 2023-11-02T18:18:12.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0fedff9564cd17c-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:18:12.333 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:18:12.334 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:18:12.334 DEBUG:boto:Method: GET 2023-11-02T18:18:12.335 DEBUG:boto:Path: /ibtljn-48/obj2 2023-11-02T18:18:12.335 DEBUG:boto:Data: 2023-11-02T18:18:12.336 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.336 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.337 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.337 DEBUG:boto:Params: {} 2023-11-02T18:18:12.338 DEBUG:boto:Token: None 2023-11-02T18:18:12.338 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj2 2023-11-02T18:18:12.339 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q1Hyd7clo/rdF9NGe2RvUolIeKA= 2023-11-02T18:18:12.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q1Hyd7clo/rdF9NGe2RvUolIeKA=', 'Content-Length': '0'} 2023-11-02T18:18:12.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033fc42359ccd3c95-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.386 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:18:12.387 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:18:12.388 DEBUG:boto:Method: GET 2023-11-02T18:18:12.388 DEBUG:boto:Path: /ibtljn-48/obj2 2023-11-02T18:18:12.389 DEBUG:boto:Data: 2023-11-02T18:18:12.389 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.390 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.390 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.391 DEBUG:boto:Params: {} 2023-11-02T18:18:12.391 DEBUG:boto:Token: None 2023-11-02T18:18:12.392 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj2 2023-11-02T18:18:12.392 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q1Hyd7clo/rdF9NGe2RvUolIeKA= 2023-11-02T18:18:12.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q1Hyd7clo/rdF9NGe2RvUolIeKA=', 'Content-Length': '0'} 2023-11-02T18:18:12.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000483ea66cf1473335-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.397 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.397 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.398 DEBUG:boto:path=/ibtljn-48/?prefix=obj2 2023-11-02T18:18:12.398 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj2 2023-11-02T18:18:12.399 DEBUG:boto:Method: GET 2023-11-02T18:18:12.399 DEBUG:boto:Path: /ibtljn-48/?prefix=obj2 2023-11-02T18:18:12.400 DEBUG:boto:Data: 2023-11-02T18:18:12.400 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.401 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.401 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.401 DEBUG:boto:Params: {} 2023-11-02T18:18:12.402 DEBUG:boto:Token: None 2023-11-02T18:18:12.403 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.403 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa15c8b87006417d-006543e7e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.409 DEBUG:boto:b'ibtljn-48obj21000falseobj22023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.410 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.410 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.411 DEBUG:boto:path=/ibtljn-48/?prefix=obj2 2023-11-02T18:18:12.411 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj2 2023-11-02T18:18:12.411 DEBUG:boto:Method: GET 2023-11-02T18:18:12.412 DEBUG:boto:Path: /ibtljn-48/?prefix=obj2 2023-11-02T18:18:12.412 DEBUG:boto:Data: 2023-11-02T18:18:12.413 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.413 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.414 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.414 DEBUG:boto:Params: {} 2023-11-02T18:18:12.415 DEBUG:boto:Token: None 2023-11-02T18:18:12.415 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.415 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003db5c8a55c3972ef-006543e7e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.422 DEBUG:boto:b'ibtljn-48obj21000falseobj22023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:18:12.423 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:18:12.424 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:18:12.424 DEBUG:boto:path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.424 DEBUG:boto:auth_path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.425 DEBUG:boto:Method: GET 2023-11-02T18:18:12.425 DEBUG:boto:Path: /ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.426 DEBUG:boto:Data: 2023-11-02T18:18:12.426 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.427 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.427 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.427 DEBUG:boto:Params: {} 2023-11-02T18:18:12.428 DEBUG:boto:Token: None 2023-11-02T18:18:12.428 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.429 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IYF87y8UfXUbgLu3/576siAueMw= 2023-11-02T18:18:12.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IYF87y8UfXUbgLu3/576siAueMw=', 'Content-Length': '0'} 2023-11-02T18:18:12.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000579f22c3c2cbb921-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.433 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:18:12.434 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:18:12.434 DEBUG:boto:path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.435 DEBUG:boto:auth_path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.435 DEBUG:boto:Method: GET 2023-11-02T18:18:12.436 DEBUG:boto:Path: /ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.436 DEBUG:boto:Data: 2023-11-02T18:18:12.437 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.437 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.438 DEBUG:boto:Params: {} 2023-11-02T18:18:12.438 DEBUG:boto:Token: None 2023-11-02T18:18:12.439 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.440 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IYF87y8UfXUbgLu3/576siAueMw= 2023-11-02T18:18:12.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IYF87y8UfXUbgLu3/576siAueMw=', 'Content-Length': '0'} 2023-11-02T18:18:12.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad80b37b830ed35b-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.444 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:18:12.444 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:18:12.445 DEBUG:boto:path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.445 DEBUG:boto:auth_path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.445 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.446 DEBUG:boto:Path: /ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.446 DEBUG:boto:Data: 2023-11-02T18:18:12.447 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.448 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.448 DEBUG:boto:Params: {} 2023-11-02T18:18:12.449 DEBUG:boto:Token: None 2023-11-02T18:18:12.449 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.450 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfnYS/bDTkHLIFk00YJrHYHWvrg= 2023-11-02T18:18:12.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfnYS/bDTkHLIFk00YJrHYHWvrg=', 'Content-Length': '0'} 2023-11-02T18:18:12.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9379f1d40838653-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.454 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:18:12.454 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:18:12.455 DEBUG:boto:path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.455 DEBUG:boto:auth_path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.456 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.456 DEBUG:boto:Path: /ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.457 DEBUG:boto:Data: 2023-11-02T18:18:12.457 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.457 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.458 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.458 DEBUG:boto:Params: {} 2023-11-02T18:18:12.459 DEBUG:boto:Token: None 2023-11-02T18:18:12.459 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.460 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfnYS/bDTkHLIFk00YJrHYHWvrg= 2023-11-02T18:18:12.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfnYS/bDTkHLIFk00YJrHYHWvrg=', 'Content-Length': '0'} 2023-11-02T18:18:12.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000170ed32c405929aa-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:18:12.464 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:18:12.465 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:18:12.465 DEBUG:boto:path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.466 DEBUG:boto:auth_path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.466 DEBUG:boto:Method: GET 2023-11-02T18:18:12.467 DEBUG:boto:Path: /ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.467 DEBUG:boto:Data: 2023-11-02T18:18:12.467 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.468 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.468 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.469 DEBUG:boto:Params: {} 2023-11-02T18:18:12.469 DEBUG:boto:Token: None 2023-11-02T18:18:12.470 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.470 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IYF87y8UfXUbgLu3/576siAueMw= 2023-11-02T18:18:12.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IYF87y8UfXUbgLu3/576siAueMw=', 'Content-Length': '0'} 2023-11-02T18:18:12.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef27f93d161c0a14-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.475 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:18:12.475 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:18:12.476 DEBUG:boto:path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.476 DEBUG:boto:auth_path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.476 DEBUG:boto:Method: GET 2023-11-02T18:18:12.477 DEBUG:boto:Path: /ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.477 DEBUG:boto:Data: 2023-11-02T18:18:12.478 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.478 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.479 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.479 DEBUG:boto:Params: {} 2023-11-02T18:18:12.480 DEBUG:boto:Token: None 2023-11-02T18:18:12.480 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj3?versionId=null 2023-11-02T18:18:12.481 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IYF87y8UfXUbgLu3/576siAueMw= 2023-11-02T18:18:12.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IYF87y8UfXUbgLu3/576siAueMw=', 'Content-Length': '0'} 2023-11-02T18:18:12.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007131f9dd4b504ee-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.526 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.527 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.527 DEBUG:boto:path=/ibtljn-48/?prefix=obj3 2023-11-02T18:18:12.528 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj3 2023-11-02T18:18:12.528 DEBUG:boto:Method: GET 2023-11-02T18:18:12.529 DEBUG:boto:Path: /ibtljn-48/?prefix=obj3 2023-11-02T18:18:12.529 DEBUG:boto:Data: 2023-11-02T18:18:12.530 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.530 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.531 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.531 DEBUG:boto:Params: {} 2023-11-02T18:18:12.532 DEBUG:boto:Token: None 2023-11-02T18:18:12.532 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.533 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec860d67f79fdccf-006543e7e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.538 DEBUG:boto:b'ibtljn-48obj31000falseobj32023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.539 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.540 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.540 DEBUG:boto:path=/ibtljn-48/?prefix=obj3 2023-11-02T18:18:12.540 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj3 2023-11-02T18:18:12.541 DEBUG:boto:Method: GET 2023-11-02T18:18:12.541 DEBUG:boto:Path: /ibtljn-48/?prefix=obj3 2023-11-02T18:18:12.542 DEBUG:boto:Data: 2023-11-02T18:18:12.542 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.543 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.543 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.544 DEBUG:boto:Params: {} 2023-11-02T18:18:12.544 DEBUG:boto:Token: None 2023-11-02T18:18:12.545 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.545 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e11a104367deb565-006543e7e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.554 DEBUG:boto:b'ibtljn-48obj31000falseobj32023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.555 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:18:12.555 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:18:12.556 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.556 DEBUG:boto:Path: /ibtljn-48/obj3 2023-11-02T18:18:12.556 DEBUG:boto:Data: 2023-11-02T18:18:12.557 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.557 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.558 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.558 DEBUG:boto:Params: {} 2023-11-02T18:18:12.559 DEBUG:boto:Token: None 2023-11-02T18:18:12.559 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj3 2023-11-02T18:18:12.560 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Binh+4jErPj0CYGzHXKv8geHUjs= 2023-11-02T18:18:12.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Binh+4jErPj0CYGzHXKv8geHUjs=', 'Content-Length': '0'} 2023-11-02T18:18:12.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000773a2462376eae83-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.563 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:18:12.563 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:18:12.564 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.564 DEBUG:boto:Path: /ibtljn-48/obj3 2023-11-02T18:18:12.565 DEBUG:boto:Data: 2023-11-02T18:18:12.565 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.565 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.566 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.566 DEBUG:boto:Params: {} 2023-11-02T18:18:12.567 DEBUG:boto:Token: None 2023-11-02T18:18:12.567 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj3 2023-11-02T18:18:12.568 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Binh+4jErPj0CYGzHXKv8geHUjs= 2023-11-02T18:18:12.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Binh+4jErPj0CYGzHXKv8geHUjs=', 'Content-Length': '0'} 2023-11-02T18:18:12.571 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d3ed64d5bd5639d-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:18:12.572 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:18:12.572 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:18:12.573 DEBUG:boto:Method: GET 2023-11-02T18:18:12.573 DEBUG:boto:Path: /ibtljn-48/obj3 2023-11-02T18:18:12.573 DEBUG:boto:Data: 2023-11-02T18:18:12.574 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.575 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.575 DEBUG:boto:Params: {} 2023-11-02T18:18:12.576 DEBUG:boto:Token: None 2023-11-02T18:18:12.576 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj3 2023-11-02T18:18:12.577 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uLDl5qYUfC4S734mzX6QrKK+AZw= 2023-11-02T18:18:12.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uLDl5qYUfC4S734mzX6QrKK+AZw=', 'Content-Length': '0'} 2023-11-02T18:18:12.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe0eb86478a09148-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.622 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:18:12.623 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:18:12.623 DEBUG:boto:Method: GET 2023-11-02T18:18:12.624 DEBUG:boto:Path: /ibtljn-48/obj3 2023-11-02T18:18:12.624 DEBUG:boto:Data: 2023-11-02T18:18:12.625 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.625 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.625 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.626 DEBUG:boto:Params: {} 2023-11-02T18:18:12.626 DEBUG:boto:Token: None 2023-11-02T18:18:12.627 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj3 2023-11-02T18:18:12.627 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uLDl5qYUfC4S734mzX6QrKK+AZw= 2023-11-02T18:18:12.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uLDl5qYUfC4S734mzX6QrKK+AZw=', 'Content-Length': '0'} 2023-11-02T18:18:12.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003408a005f4620c67-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.631 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.632 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.632 DEBUG:boto:path=/ibtljn-48/?prefix=obj3 2023-11-02T18:18:12.633 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj3 2023-11-02T18:18:12.633 DEBUG:boto:Method: GET 2023-11-02T18:18:12.634 DEBUG:boto:Path: /ibtljn-48/?prefix=obj3 2023-11-02T18:18:12.634 DEBUG:boto:Data: 2023-11-02T18:18:12.634 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.635 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.635 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.636 DEBUG:boto:Params: {} 2023-11-02T18:18:12.636 DEBUG:boto:Token: None 2023-11-02T18:18:12.637 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.637 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a666c8c8b27a2d7-006543e7e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.644 DEBUG:boto:b'ibtljn-48obj31000falseobj32023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.644 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.645 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.645 DEBUG:boto:path=/ibtljn-48/?prefix=obj3 2023-11-02T18:18:12.646 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj3 2023-11-02T18:18:12.646 DEBUG:boto:Method: GET 2023-11-02T18:18:12.647 DEBUG:boto:Path: /ibtljn-48/?prefix=obj3 2023-11-02T18:18:12.647 DEBUG:boto:Data: 2023-11-02T18:18:12.648 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.648 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.648 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.649 DEBUG:boto:Params: {} 2023-11-02T18:18:12.649 DEBUG:boto:Token: None 2023-11-02T18:18:12.650 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.650 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a15afad1b2a867c1-006543e7e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.698 DEBUG:boto:b'ibtljn-48obj31000falseobj32023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:18:12.700 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:18:12.700 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:18:12.701 DEBUG:boto:path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.701 DEBUG:boto:auth_path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.702 DEBUG:boto:Method: GET 2023-11-02T18:18:12.702 DEBUG:boto:Path: /ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.703 DEBUG:boto:Data: 2023-11-02T18:18:12.703 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.703 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.704 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.704 DEBUG:boto:Params: {} 2023-11-02T18:18:12.705 DEBUG:boto:Token: None 2023-11-02T18:18:12.706 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.706 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S9J2r4PhpQek76wW0t/VfAjDOO0= 2023-11-02T18:18:12.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S9J2r4PhpQek76wW0t/VfAjDOO0=', 'Content-Length': '0'} 2023-11-02T18:18:12.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd4b292cedcf617c-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.710 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:18:12.711 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:18:12.711 DEBUG:boto:path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.712 DEBUG:boto:auth_path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.712 DEBUG:boto:Method: GET 2023-11-02T18:18:12.713 DEBUG:boto:Path: /ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.713 DEBUG:boto:Data: 2023-11-02T18:18:12.714 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.714 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.715 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.715 DEBUG:boto:Params: {} 2023-11-02T18:18:12.716 DEBUG:boto:Token: None 2023-11-02T18:18:12.716 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.717 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S9J2r4PhpQek76wW0t/VfAjDOO0= 2023-11-02T18:18:12.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S9J2r4PhpQek76wW0t/VfAjDOO0=', 'Content-Length': '0'} 2023-11-02T18:18:12.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9fa8930d6a2350f-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.721 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:18:12.722 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:18:12.722 DEBUG:boto:path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.723 DEBUG:boto:auth_path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.723 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.723 DEBUG:boto:Path: /ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.724 DEBUG:boto:Data: 2023-11-02T18:18:12.724 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.725 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.725 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.726 DEBUG:boto:Params: {} 2023-11-02T18:18:12.726 DEBUG:boto:Token: None 2023-11-02T18:18:12.727 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.727 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NHJjTFgggsFdzzv70pwO1sanAas= 2023-11-02T18:18:12.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NHJjTFgggsFdzzv70pwO1sanAas=', 'Content-Length': '0'} 2023-11-02T18:18:12.731 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a316fa7ee6d5d88a-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.731 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:18:12.732 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:18:12.732 DEBUG:boto:path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.732 DEBUG:boto:auth_path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.733 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.733 DEBUG:boto:Path: /ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.734 DEBUG:boto:Data: 2023-11-02T18:18:12.734 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.735 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.735 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.735 DEBUG:boto:Params: {} 2023-11-02T18:18:12.736 DEBUG:boto:Token: None 2023-11-02T18:18:12.736 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.737 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NHJjTFgggsFdzzv70pwO1sanAas= 2023-11-02T18:18:12.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NHJjTFgggsFdzzv70pwO1sanAas=', 'Content-Length': '0'} 2023-11-02T18:18:12.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006898183c105f68df-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:18:12.741 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:18:12.741 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:18:12.742 DEBUG:boto:path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.742 DEBUG:boto:auth_path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.743 DEBUG:boto:Method: GET 2023-11-02T18:18:12.743 DEBUG:boto:Path: /ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.744 DEBUG:boto:Data: 2023-11-02T18:18:12.744 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.745 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.745 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.746 DEBUG:boto:Params: {} 2023-11-02T18:18:12.746 DEBUG:boto:Token: None 2023-11-02T18:18:12.747 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.747 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S9J2r4PhpQek76wW0t/VfAjDOO0= 2023-11-02T18:18:12.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S9J2r4PhpQek76wW0t/VfAjDOO0=', 'Content-Length': '0'} 2023-11-02T18:18:12.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000375b7db9d1c941d8-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.751 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:18:12.752 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:18:12.752 DEBUG:boto:path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.753 DEBUG:boto:auth_path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.753 DEBUG:boto:Method: GET 2023-11-02T18:18:12.754 DEBUG:boto:Path: /ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.754 DEBUG:boto:Data: 2023-11-02T18:18:12.755 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.755 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.756 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.756 DEBUG:boto:Params: {} 2023-11-02T18:18:12.757 DEBUG:boto:Token: None 2023-11-02T18:18:12.757 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj4?versionId=null 2023-11-02T18:18:12.758 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S9J2r4PhpQek76wW0t/VfAjDOO0= 2023-11-02T18:18:12.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S9J2r4PhpQek76wW0t/VfAjDOO0=', 'Content-Length': '0'} 2023-11-02T18:18:12.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7cdf7a14744a708-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.762 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.762 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.763 DEBUG:boto:path=/ibtljn-48/?prefix=obj4 2023-11-02T18:18:12.763 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj4 2023-11-02T18:18:12.764 DEBUG:boto:Method: GET 2023-11-02T18:18:12.764 DEBUG:boto:Path: /ibtljn-48/?prefix=obj4 2023-11-02T18:18:12.765 DEBUG:boto:Data: 2023-11-02T18:18:12.765 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.766 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.766 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.767 DEBUG:boto:Params: {} 2023-11-02T18:18:12.767 DEBUG:boto:Token: None 2023-11-02T18:18:12.768 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ab868cd16f1de26-006543e7e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.818 DEBUG:boto:b'ibtljn-48obj41000falseobj42023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.819 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.820 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.820 DEBUG:boto:path=/ibtljn-48/?prefix=obj4 2023-11-02T18:18:12.821 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj4 2023-11-02T18:18:12.821 DEBUG:boto:Method: GET 2023-11-02T18:18:12.822 DEBUG:boto:Path: /ibtljn-48/?prefix=obj4 2023-11-02T18:18:12.822 DEBUG:boto:Data: 2023-11-02T18:18:12.823 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.823 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.824 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.824 DEBUG:boto:Params: {} 2023-11-02T18:18:12.825 DEBUG:boto:Token: None 2023-11-02T18:18:12.825 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.826 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037fc48754801cf30-006543e7e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.831 DEBUG:boto:b'ibtljn-48obj41000falseobj42023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.832 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:18:12.833 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:18:12.833 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.833 DEBUG:boto:Path: /ibtljn-48/obj4 2023-11-02T18:18:12.834 DEBUG:boto:Data: 2023-11-02T18:18:12.834 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.835 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.835 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.836 DEBUG:boto:Params: {} 2023-11-02T18:18:12.836 DEBUG:boto:Token: None 2023-11-02T18:18:12.837 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj4 2023-11-02T18:18:12.837 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ts5r9Y1OaCsdgdDfQXoyBHJUVM= 2023-11-02T18:18:12.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ts5r9Y1OaCsdgdDfQXoyBHJUVM=', 'Content-Length': '0'} 2023-11-02T18:18:12.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008408e70138ca9c43-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.841 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:18:12.842 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:18:12.842 DEBUG:boto:Method: HEAD 2023-11-02T18:18:12.843 DEBUG:boto:Path: /ibtljn-48/obj4 2023-11-02T18:18:12.843 DEBUG:boto:Data: 2023-11-02T18:18:12.844 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.844 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.845 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.845 DEBUG:boto:Params: {} 2023-11-02T18:18:12.845 DEBUG:boto:Token: None 2023-11-02T18:18:12.846 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj4 2023-11-02T18:18:12.846 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ts5r9Y1OaCsdgdDfQXoyBHJUVM= 2023-11-02T18:18:12.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ts5r9Y1OaCsdgdDfQXoyBHJUVM=', 'Content-Length': '0'} 2023-11-02T18:18:12.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042391f735f9c3ac3-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:18:12.851 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:18:12.851 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:18:12.852 DEBUG:boto:Method: GET 2023-11-02T18:18:12.852 DEBUG:boto:Path: /ibtljn-48/obj4 2023-11-02T18:18:12.853 DEBUG:boto:Data: 2023-11-02T18:18:12.853 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.854 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.854 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.855 DEBUG:boto:Params: {} 2023-11-02T18:18:12.855 DEBUG:boto:Token: None 2023-11-02T18:18:12.856 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj4 2023-11-02T18:18:12.856 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oM7N9QFmKSl8vYB06vwJ9ekRZMg= 2023-11-02T18:18:12.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oM7N9QFmKSl8vYB06vwJ9ekRZMg=', 'Content-Length': '0'} 2023-11-02T18:18:12.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055f952b26f649de1-006543e7e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.861 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:18:12.861 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:18:12.862 DEBUG:boto:Method: GET 2023-11-02T18:18:12.862 DEBUG:boto:Path: /ibtljn-48/obj4 2023-11-02T18:18:12.863 DEBUG:boto:Data: 2023-11-02T18:18:12.863 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.864 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.864 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.864 DEBUG:boto:Params: {} 2023-11-02T18:18:12.865 DEBUG:boto:Token: None 2023-11-02T18:18:12.865 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/obj4 2023-11-02T18:18:12.866 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oM7N9QFmKSl8vYB06vwJ9ekRZMg= 2023-11-02T18:18:12.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oM7N9QFmKSl8vYB06vwJ9ekRZMg=', 'Content-Length': '0'} 2023-11-02T18:18:12.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a01378903c553b02-006543e7e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.910 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.911 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.912 DEBUG:boto:path=/ibtljn-48/?prefix=obj4 2023-11-02T18:18:12.912 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj4 2023-11-02T18:18:12.913 DEBUG:boto:Method: GET 2023-11-02T18:18:12.913 DEBUG:boto:Path: /ibtljn-48/?prefix=obj4 2023-11-02T18:18:12.913 DEBUG:boto:Data: 2023-11-02T18:18:12.914 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.914 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:12.915 DEBUG:boto:Port: 8001 2023-11-02T18:18:12.915 DEBUG:boto:Params: {} 2023-11-02T18:18:12.916 DEBUG:boto:Token: None 2023-11-02T18:18:12.916 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.917 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000725ffced2e2e2111-006543e7e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.923 DEBUG:boto:b'ibtljn-48obj41000falseobj42023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.924 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:12.924 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:12.925 DEBUG:boto:path=/ibtljn-48/?prefix=obj4 2023-11-02T18:18:12.925 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj4 2023-11-02T18:18:12.926 DEBUG:boto:Method: GET 2023-11-02T18:18:12.926 DEBUG:boto:Path: /ibtljn-48/?prefix=obj4 2023-11-02T18:18:12.927 DEBUG:boto:Data: 2023-11-02T18:18:12.927 DEBUG:boto:Headers: {} 2023-11-02T18:18:12.928 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:12.928 DEBUG:boto:Port: 8000 2023-11-02T18:18:12.929 DEBUG:boto:Params: {} 2023-11-02T18:18:12.929 DEBUG:boto:Token: None 2023-11-02T18:18:12.930 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:12 GMT /ibtljn-48/ 2023-11-02T18:18:12.930 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88= 2023-11-02T18:18:12.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Lv1m50xHWJGqU+JDHUnCGtXl88=', 'Content-Length': '0'} 2023-11-02T18:18:12.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc5d4cb2d13b26db-006543e7e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:12.937 DEBUG:boto:b'ibtljn-48obj41000falseobj42023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:12.938 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-48 zones={a2, a1} 2023-11-02T18:18:12.938 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-49 2023-11-02T18:18:12.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:18:12.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:18:12.940 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:18:12.994 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:12.995 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:13.067 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:13.062+0000 7f3a35b0ef40 1 bucket sync caught up with source: 2023-11-02T18:18:13.068 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.16.6, , 00000000001.25.6, , 00000000001.112.6, , , 00000000001.30.6, , , ] 2023-11-02T18:18:13.068 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.16.6, , 00000000001.25.6, , 00000000001.112.6, , , 00000000001.30.6, , , ] 2023-11-02T18:18:13.068 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:13.062+0000 7f3a35b0ef40 0 bucket checkpoint complete 2023-11-02T18:18:13.095 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-49 2023-11-02T18:18:13.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:18:13.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:18:13.097 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:18:13.149 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:13.149 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:13.221 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:13.214+0000 7fbaba1b1f40 1 bucket sync caught up with source: 2023-11-02T18:18:13.221 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.236.6, , 00000000001.230.6, , 00000000001.260.6, , , 00000000001.53.6, , , ] 2023-11-02T18:18:13.222 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.236.6, , 00000000001.230.6, , 00000000001.260.6, , , 00000000001.53.6, , , ] 2023-11-02T18:18:13.222 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:13.214+0000 7fbaba1b1f40 0 bucket checkpoint complete 2023-11-02T18:18:13.248 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-49 zones={a2, a1} 2023-11-02T18:18:13.249 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.249 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.249 DEBUG:boto:Method: HEAD 2023-11-02T18:18:13.250 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:18:13.250 DEBUG:boto:Data: 2023-11-02T18:18:13.251 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.251 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.252 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.252 DEBUG:boto:Params: {} 2023-11-02T18:18:13.253 DEBUG:boto:Token: None 2023-11-02T18:18:13.253 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/ 2023-11-02T18:18:13.254 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:49XoEfB9hipkImtFvPUIgdI6Q/0= 2023-11-02T18:18:13.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:49XoEfB9hipkImtFvPUIgdI6Q/0=', 'Content-Length': '0'} 2023-11-02T18:18:13.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d58187ccfff293bf-006543e7e5-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.259 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.260 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.260 DEBUG:boto:Method: HEAD 2023-11-02T18:18:13.261 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:18:13.261 DEBUG:boto:Data: 2023-11-02T18:18:13.262 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.262 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.262 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.263 DEBUG:boto:Params: {} 2023-11-02T18:18:13.263 DEBUG:boto:Token: None 2023-11-02T18:18:13.264 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/ 2023-11-02T18:18:13.264 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:49XoEfB9hipkImtFvPUIgdI6Q/0= 2023-11-02T18:18:13.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:49XoEfB9hipkImtFvPUIgdI6Q/0=', 'Content-Length': '0'} 2023-11-02T18:18:13.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c61bd355813c2c9-006543e7e5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.269 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:18:13.270 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.270 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.271 DEBUG:boto:path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:18:13.271 DEBUG:boto:auth_path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:18:13.272 DEBUG:boto:Method: GET 2023-11-02T18:18:13.272 DEBUG:boto:Path: /ibtljn-49/?versions&max-keys=999 2023-11-02T18:18:13.272 DEBUG:boto:Data: 2023-11-02T18:18:13.273 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.273 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.274 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.274 DEBUG:boto:Params: {} 2023-11-02T18:18:13.275 DEBUG:boto:Token: None 2023-11-02T18:18:13.275 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/?versions 2023-11-02T18:18:13.276 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i+mDndpamF8wpjCnEzvX8q7YSSY= 2023-11-02T18:18:13.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i+mDndpamF8wpjCnEzvX8q7YSSY=', 'Content-Length': '0'} 2023-11-02T18:18:13.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e08227397fc1475-006543e7e5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.282 DEBUG:boto:b'ibtljn-49999falseobj1nulltrue2023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:13.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:18:13.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:18:13.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:18:13.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:18:13.285 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:18:13.285 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.286 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.286 DEBUG:boto:path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:18:13.287 DEBUG:boto:auth_path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:18:13.287 DEBUG:boto:Method: GET 2023-11-02T18:18:13.288 DEBUG:boto:Path: /ibtljn-49/?versions&max-keys=999 2023-11-02T18:18:13.288 DEBUG:boto:Data: 2023-11-02T18:18:13.288 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.289 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.289 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.290 DEBUG:boto:Params: {} 2023-11-02T18:18:13.290 DEBUG:boto:Token: None 2023-11-02T18:18:13.291 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/?versions 2023-11-02T18:18:13.291 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i+mDndpamF8wpjCnEzvX8q7YSSY= 2023-11-02T18:18:13.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i+mDndpamF8wpjCnEzvX8q7YSSY=', 'Content-Length': '0'} 2023-11-02T18:18:13.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b522719bfc1a7214-006543e7e5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.297 DEBUG:boto:b'ibtljn-49999falseobj1nulltrue2023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:13.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:18:13.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:18:13.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:18:13.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:18:13.299 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.300 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.300 DEBUG:boto:path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:18:13.301 DEBUG:boto:auth_path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:18:13.301 DEBUG:boto:Method: GET 2023-11-02T18:18:13.301 DEBUG:boto:Path: /ibtljn-49/?versions&max-keys=999 2023-11-02T18:18:13.302 DEBUG:boto:Data: 2023-11-02T18:18:13.302 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.303 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.304 DEBUG:boto:Params: {} 2023-11-02T18:18:13.304 DEBUG:boto:Token: None 2023-11-02T18:18:13.305 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/?versions 2023-11-02T18:18:13.306 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i+mDndpamF8wpjCnEzvX8q7YSSY= 2023-11-02T18:18:13.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i+mDndpamF8wpjCnEzvX8q7YSSY=', 'Content-Length': '0'} 2023-11-02T18:18:13.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef6f24f8960fa110-006543e7e5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.312 DEBUG:boto:b'ibtljn-49999falseobj1nulltrue2023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:13.313 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.314 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.314 DEBUG:boto:path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:18:13.314 DEBUG:boto:auth_path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:18:13.315 DEBUG:boto:Method: GET 2023-11-02T18:18:13.315 DEBUG:boto:Path: /ibtljn-49/?versions&max-keys=999 2023-11-02T18:18:13.316 DEBUG:boto:Data: 2023-11-02T18:18:13.316 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.317 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.317 DEBUG:boto:Params: {} 2023-11-02T18:18:13.318 DEBUG:boto:Token: None 2023-11-02T18:18:13.318 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/?versions 2023-11-02T18:18:13.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i+mDndpamF8wpjCnEzvX8q7YSSY= 2023-11-02T18:18:13.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i+mDndpamF8wpjCnEzvX8q7YSSY=', 'Content-Length': '0'} 2023-11-02T18:18:13.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b92d7ba3cd6d748a-006543e7e5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.325 DEBUG:boto:b'ibtljn-49999falseobj1nulltrue2023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:13.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:18:13.326 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:18:13.326 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:18:13.327 DEBUG:boto:path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.327 DEBUG:boto:auth_path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.328 DEBUG:boto:Method: GET 2023-11-02T18:18:13.328 DEBUG:boto:Path: /ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.329 DEBUG:boto:Data: 2023-11-02T18:18:13.329 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.330 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.330 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.330 DEBUG:boto:Params: {} 2023-11-02T18:18:13.331 DEBUG:boto:Token: None 2023-11-02T18:18:13.331 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.332 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OjvLEEHO2lm+JVTJfdML7ON4WTQ= 2023-11-02T18:18:13.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OjvLEEHO2lm+JVTJfdML7ON4WTQ=', 'Content-Length': '0'} 2023-11-02T18:18:13.335 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5e1e2724a2f266a-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.382 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:18:13.383 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:18:13.383 DEBUG:boto:path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.384 DEBUG:boto:auth_path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.384 DEBUG:boto:Method: GET 2023-11-02T18:18:13.385 DEBUG:boto:Path: /ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.385 DEBUG:boto:Data: 2023-11-02T18:18:13.386 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.386 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.386 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.387 DEBUG:boto:Params: {} 2023-11-02T18:18:13.387 DEBUG:boto:Token: None 2023-11-02T18:18:13.388 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.388 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OjvLEEHO2lm+JVTJfdML7ON4WTQ= 2023-11-02T18:18:13.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OjvLEEHO2lm+JVTJfdML7ON4WTQ=', 'Content-Length': '0'} 2023-11-02T18:18:13.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003dbb0a1c0e3e445b-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.393 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:18:13.394 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:18:13.394 DEBUG:boto:path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.395 DEBUG:boto:auth_path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.395 DEBUG:boto:Method: HEAD 2023-11-02T18:18:13.395 DEBUG:boto:Path: /ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.396 DEBUG:boto:Data: 2023-11-02T18:18:13.396 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.396 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.397 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.397 DEBUG:boto:Params: {} 2023-11-02T18:18:13.398 DEBUG:boto:Token: None 2023-11-02T18:18:13.398 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.399 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vo3TlyxF6BVBXTgzJXmrCF0omAo= 2023-11-02T18:18:13.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vo3TlyxF6BVBXTgzJXmrCF0omAo=', 'Content-Length': '0'} 2023-11-02T18:18:13.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd859569e415e25a-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.403 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:18:13.404 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:18:13.404 DEBUG:boto:path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.404 DEBUG:boto:auth_path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.405 DEBUG:boto:Method: HEAD 2023-11-02T18:18:13.405 DEBUG:boto:Path: /ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.406 DEBUG:boto:Data: 2023-11-02T18:18:13.406 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.407 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.407 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.407 DEBUG:boto:Params: {} 2023-11-02T18:18:13.408 DEBUG:boto:Token: None 2023-11-02T18:18:13.408 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.409 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vo3TlyxF6BVBXTgzJXmrCF0omAo= 2023-11-02T18:18:13.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vo3TlyxF6BVBXTgzJXmrCF0omAo=', 'Content-Length': '0'} 2023-11-02T18:18:13.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c502842dbc60d838-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:18:13.413 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:18:13.414 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:18:13.414 DEBUG:boto:path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.415 DEBUG:boto:auth_path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.415 DEBUG:boto:Method: GET 2023-11-02T18:18:13.416 DEBUG:boto:Path: /ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.416 DEBUG:boto:Data: 2023-11-02T18:18:13.416 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.417 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.417 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.418 DEBUG:boto:Params: {} 2023-11-02T18:18:13.418 DEBUG:boto:Token: None 2023-11-02T18:18:13.419 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.419 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OjvLEEHO2lm+JVTJfdML7ON4WTQ= 2023-11-02T18:18:13.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OjvLEEHO2lm+JVTJfdML7ON4WTQ=', 'Content-Length': '0'} 2023-11-02T18:18:13.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc71a4d040337651-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.423 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:18:13.423 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:18:13.424 DEBUG:boto:path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.424 DEBUG:boto:auth_path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.425 DEBUG:boto:Method: GET 2023-11-02T18:18:13.425 DEBUG:boto:Path: /ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.425 DEBUG:boto:Data: 2023-11-02T18:18:13.426 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.427 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.427 DEBUG:boto:Params: {} 2023-11-02T18:18:13.428 DEBUG:boto:Token: None 2023-11-02T18:18:13.428 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj1?versionId=null 2023-11-02T18:18:13.429 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OjvLEEHO2lm+JVTJfdML7ON4WTQ= 2023-11-02T18:18:13.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OjvLEEHO2lm+JVTJfdML7ON4WTQ=', 'Content-Length': '0'} 2023-11-02T18:18:13.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020bc9754f7f4a0cc-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.433 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.434 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.434 DEBUG:boto:path=/ibtljn-49/?prefix=obj1 2023-11-02T18:18:13.434 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj1 2023-11-02T18:18:13.435 DEBUG:boto:Method: GET 2023-11-02T18:18:13.435 DEBUG:boto:Path: /ibtljn-49/?prefix=obj1 2023-11-02T18:18:13.436 DEBUG:boto:Data: 2023-11-02T18:18:13.436 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.437 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.437 DEBUG:boto:Params: {} 2023-11-02T18:18:13.438 DEBUG:boto:Token: None 2023-11-02T18:18:13.438 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/ 2023-11-02T18:18:13.439 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM= 2023-11-02T18:18:13.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM=', 'Content-Length': '0'} 2023-11-02T18:18:13.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000306b8c94080713be-006543e7e5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.446 DEBUG:boto:b'ibtljn-49obj11000falseobj12023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:13.447 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.447 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.448 DEBUG:boto:path=/ibtljn-49/?prefix=obj1 2023-11-02T18:18:13.448 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj1 2023-11-02T18:18:13.449 DEBUG:boto:Method: GET 2023-11-02T18:18:13.449 DEBUG:boto:Path: /ibtljn-49/?prefix=obj1 2023-11-02T18:18:13.450 DEBUG:boto:Data: 2023-11-02T18:18:13.450 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.451 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.451 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.452 DEBUG:boto:Params: {} 2023-11-02T18:18:13.452 DEBUG:boto:Token: None 2023-11-02T18:18:13.453 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/ 2023-11-02T18:18:13.453 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM= 2023-11-02T18:18:13.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM=', 'Content-Length': '0'} 2023-11-02T18:18:13.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0f3e59dd502aa4f-006543e7e5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.506 DEBUG:boto:b'ibtljn-49obj11000falseobj12023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:13.507 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:18:13.508 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:18:13.508 DEBUG:boto:Method: HEAD 2023-11-02T18:18:13.509 DEBUG:boto:Path: /ibtljn-49/obj1 2023-11-02T18:18:13.509 DEBUG:boto:Data: 2023-11-02T18:18:13.509 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.510 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.511 DEBUG:boto:Params: {} 2023-11-02T18:18:13.511 DEBUG:boto:Token: None 2023-11-02T18:18:13.512 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj1 2023-11-02T18:18:13.512 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OmIsJL3w1ADErdivg58GxjCyKmA= 2023-11-02T18:18:13.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OmIsJL3w1ADErdivg58GxjCyKmA=', 'Content-Length': '0'} 2023-11-02T18:18:13.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005621c190abd085dc-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.517 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:18:13.517 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:18:13.517 DEBUG:boto:Method: HEAD 2023-11-02T18:18:13.518 DEBUG:boto:Path: /ibtljn-49/obj1 2023-11-02T18:18:13.518 DEBUG:boto:Data: 2023-11-02T18:18:13.519 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.519 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.520 DEBUG:boto:Params: {} 2023-11-02T18:18:13.520 DEBUG:boto:Token: None 2023-11-02T18:18:13.521 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj1 2023-11-02T18:18:13.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OmIsJL3w1ADErdivg58GxjCyKmA= 2023-11-02T18:18:13.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OmIsJL3w1ADErdivg58GxjCyKmA=', 'Content-Length': '0'} 2023-11-02T18:18:13.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fc6a03c7369fb4a-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:18:13.526 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:18:13.526 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:18:13.526 DEBUG:boto:Method: GET 2023-11-02T18:18:13.527 DEBUG:boto:Path: /ibtljn-49/obj1 2023-11-02T18:18:13.527 DEBUG:boto:Data: 2023-11-02T18:18:13.528 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.528 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.529 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.529 DEBUG:boto:Params: {} 2023-11-02T18:18:13.530 DEBUG:boto:Token: None 2023-11-02T18:18:13.530 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj1 2023-11-02T18:18:13.531 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sPqJY4J3su8aIkMODntSN478Y1g= 2023-11-02T18:18:13.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sPqJY4J3su8aIkMODntSN478Y1g=', 'Content-Length': '0'} 2023-11-02T18:18:13.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7fe0fa8917679b6-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.574 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:18:13.575 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:18:13.575 DEBUG:boto:Method: GET 2023-11-02T18:18:13.576 DEBUG:boto:Path: /ibtljn-49/obj1 2023-11-02T18:18:13.576 DEBUG:boto:Data: 2023-11-02T18:18:13.577 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.577 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.578 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.578 DEBUG:boto:Params: {} 2023-11-02T18:18:13.579 DEBUG:boto:Token: None 2023-11-02T18:18:13.580 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj1 2023-11-02T18:18:13.580 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sPqJY4J3su8aIkMODntSN478Y1g= 2023-11-02T18:18:13.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sPqJY4J3su8aIkMODntSN478Y1g=', 'Content-Length': '0'} 2023-11-02T18:18:13.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000457d5d533d014ac-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.584 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.584 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.585 DEBUG:boto:path=/ibtljn-49/?prefix=obj1 2023-11-02T18:18:13.585 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj1 2023-11-02T18:18:13.586 DEBUG:boto:Method: GET 2023-11-02T18:18:13.586 DEBUG:boto:Path: /ibtljn-49/?prefix=obj1 2023-11-02T18:18:13.587 DEBUG:boto:Data: 2023-11-02T18:18:13.587 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.588 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.588 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.588 DEBUG:boto:Params: {} 2023-11-02T18:18:13.589 DEBUG:boto:Token: None 2023-11-02T18:18:13.589 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/ 2023-11-02T18:18:13.590 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM= 2023-11-02T18:18:13.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM=', 'Content-Length': '0'} 2023-11-02T18:18:13.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc189c4677191499-006543e7e5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.595 DEBUG:boto:b'ibtljn-49obj11000falseobj12023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:13.596 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.597 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.597 DEBUG:boto:path=/ibtljn-49/?prefix=obj1 2023-11-02T18:18:13.597 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj1 2023-11-02T18:18:13.598 DEBUG:boto:Method: GET 2023-11-02T18:18:13.598 DEBUG:boto:Path: /ibtljn-49/?prefix=obj1 2023-11-02T18:18:13.599 DEBUG:boto:Data: 2023-11-02T18:18:13.599 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.600 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.600 DEBUG:boto:Params: {} 2023-11-02T18:18:13.601 DEBUG:boto:Token: None 2023-11-02T18:18:13.601 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/ 2023-11-02T18:18:13.602 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM= 2023-11-02T18:18:13.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM=', 'Content-Length': '0'} 2023-11-02T18:18:13.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce0b3596cbad267c-006543e7e5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.608 DEBUG:boto:b'ibtljn-49obj11000falseobj12023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:13.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:18:13.609 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:18:13.609 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:18:13.610 DEBUG:boto:path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.610 DEBUG:boto:auth_path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.611 DEBUG:boto:Method: GET 2023-11-02T18:18:13.611 DEBUG:boto:Path: /ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.611 DEBUG:boto:Data: 2023-11-02T18:18:13.612 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.612 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.613 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.613 DEBUG:boto:Params: {} 2023-11-02T18:18:13.613 DEBUG:boto:Token: None 2023-11-02T18:18:13.614 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.614 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kDzGt/foSpwZTXdlTlGzGWnduBI= 2023-11-02T18:18:13.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kDzGt/foSpwZTXdlTlGzGWnduBI=', 'Content-Length': '0'} 2023-11-02T18:18:13.618 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002bfadd1c95aa3025-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.618 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:18:13.619 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:18:13.619 DEBUG:boto:path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.620 DEBUG:boto:auth_path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.620 DEBUG:boto:Method: GET 2023-11-02T18:18:13.620 DEBUG:boto:Path: /ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.621 DEBUG:boto:Data: 2023-11-02T18:18:13.621 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.622 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.622 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.622 DEBUG:boto:Params: {} 2023-11-02T18:18:13.623 DEBUG:boto:Token: None 2023-11-02T18:18:13.623 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kDzGt/foSpwZTXdlTlGzGWnduBI= 2023-11-02T18:18:13.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kDzGt/foSpwZTXdlTlGzGWnduBI=', 'Content-Length': '0'} 2023-11-02T18:18:13.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fd06769972aedfd-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.670 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:18:13.671 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:18:13.671 DEBUG:boto:path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.672 DEBUG:boto:auth_path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.672 DEBUG:boto:Method: HEAD 2023-11-02T18:18:13.672 DEBUG:boto:Path: /ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.673 DEBUG:boto:Data: 2023-11-02T18:18:13.673 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.673 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.674 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.674 DEBUG:boto:Params: {} 2023-11-02T18:18:13.675 DEBUG:boto:Token: None 2023-11-02T18:18:13.675 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bAEJPlzN3ElDBjNHMKBNn3N/8UQ= 2023-11-02T18:18:13.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bAEJPlzN3ElDBjNHMKBNn3N/8UQ=', 'Content-Length': '0'} 2023-11-02T18:18:13.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000723efb3dc0c7175a-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.680 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:18:13.680 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:18:13.680 DEBUG:boto:path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.681 DEBUG:boto:auth_path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.681 DEBUG:boto:Method: HEAD 2023-11-02T18:18:13.682 DEBUG:boto:Path: /ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.682 DEBUG:boto:Data: 2023-11-02T18:18:13.683 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.683 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.683 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.684 DEBUG:boto:Params: {} 2023-11-02T18:18:13.684 DEBUG:boto:Token: None 2023-11-02T18:18:13.685 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.685 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bAEJPlzN3ElDBjNHMKBNn3N/8UQ= 2023-11-02T18:18:13.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bAEJPlzN3ElDBjNHMKBNn3N/8UQ=', 'Content-Length': '0'} 2023-11-02T18:18:13.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a1ae10b6cee9e4f-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:18:13.690 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:18:13.690 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:18:13.691 DEBUG:boto:path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.691 DEBUG:boto:auth_path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.691 DEBUG:boto:Method: GET 2023-11-02T18:18:13.692 DEBUG:boto:Path: /ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.692 DEBUG:boto:Data: 2023-11-02T18:18:13.693 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.693 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.694 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.694 DEBUG:boto:Params: {} 2023-11-02T18:18:13.694 DEBUG:boto:Token: None 2023-11-02T18:18:13.695 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.695 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kDzGt/foSpwZTXdlTlGzGWnduBI= 2023-11-02T18:18:13.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kDzGt/foSpwZTXdlTlGzGWnduBI=', 'Content-Length': '0'} 2023-11-02T18:18:13.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba478b21f4e36a17-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.742 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:18:13.743 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:18:13.743 DEBUG:boto:path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.744 DEBUG:boto:auth_path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.744 DEBUG:boto:Method: GET 2023-11-02T18:18:13.744 DEBUG:boto:Path: /ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.745 DEBUG:boto:Data: 2023-11-02T18:18:13.745 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.746 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.746 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.746 DEBUG:boto:Params: {} 2023-11-02T18:18:13.747 DEBUG:boto:Token: None 2023-11-02T18:18:13.747 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj2?versionId=null 2023-11-02T18:18:13.748 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kDzGt/foSpwZTXdlTlGzGWnduBI= 2023-11-02T18:18:13.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kDzGt/foSpwZTXdlTlGzGWnduBI=', 'Content-Length': '0'} 2023-11-02T18:18:13.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089af8f36230a6061-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.752 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.753 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.753 DEBUG:boto:path=/ibtljn-49/?prefix=obj2 2023-11-02T18:18:13.753 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj2 2023-11-02T18:18:13.754 DEBUG:boto:Method: GET 2023-11-02T18:18:13.754 DEBUG:boto:Path: /ibtljn-49/?prefix=obj2 2023-11-02T18:18:13.755 DEBUG:boto:Data: 2023-11-02T18:18:13.755 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.755 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.756 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.756 DEBUG:boto:Params: {} 2023-11-02T18:18:13.757 DEBUG:boto:Token: None 2023-11-02T18:18:13.757 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/ 2023-11-02T18:18:13.758 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM= 2023-11-02T18:18:13.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM=', 'Content-Length': '0'} 2023-11-02T18:18:13.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c31ca7048fc2ed6c-006543e7e5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.764 DEBUG:boto:b'ibtljn-49obj21000falseobj22023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:13.765 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.765 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.766 DEBUG:boto:path=/ibtljn-49/?prefix=obj2 2023-11-02T18:18:13.766 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj2 2023-11-02T18:18:13.767 DEBUG:boto:Method: GET 2023-11-02T18:18:13.767 DEBUG:boto:Path: /ibtljn-49/?prefix=obj2 2023-11-02T18:18:13.767 DEBUG:boto:Data: 2023-11-02T18:18:13.768 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.768 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.769 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.769 DEBUG:boto:Params: {} 2023-11-02T18:18:13.769 DEBUG:boto:Token: None 2023-11-02T18:18:13.770 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/ 2023-11-02T18:18:13.771 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM= 2023-11-02T18:18:13.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM=', 'Content-Length': '0'} 2023-11-02T18:18:13.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7f53908a914e17c-006543e7e5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.777 DEBUG:boto:b'ibtljn-49obj21000falseobj22023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:13.777 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:18:13.778 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:18:13.778 DEBUG:boto:Method: HEAD 2023-11-02T18:18:13.779 DEBUG:boto:Path: /ibtljn-49/obj2 2023-11-02T18:18:13.779 DEBUG:boto:Data: 2023-11-02T18:18:13.779 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.780 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.780 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.781 DEBUG:boto:Params: {} 2023-11-02T18:18:13.781 DEBUG:boto:Token: None 2023-11-02T18:18:13.782 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj2 2023-11-02T18:18:13.782 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p3gj1b/tAucXF+CjcqVxF0kqRq0= 2023-11-02T18:18:13.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p3gj1b/tAucXF+CjcqVxF0kqRq0=', 'Content-Length': '0'} 2023-11-02T18:18:13.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c579f44e7d4a741-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.786 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:18:13.786 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:18:13.787 DEBUG:boto:Method: HEAD 2023-11-02T18:18:13.787 DEBUG:boto:Path: /ibtljn-49/obj2 2023-11-02T18:18:13.787 DEBUG:boto:Data: 2023-11-02T18:18:13.788 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.788 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.789 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.789 DEBUG:boto:Params: {} 2023-11-02T18:18:13.789 DEBUG:boto:Token: None 2023-11-02T18:18:13.790 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj2 2023-11-02T18:18:13.790 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p3gj1b/tAucXF+CjcqVxF0kqRq0= 2023-11-02T18:18:13.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p3gj1b/tAucXF+CjcqVxF0kqRq0=', 'Content-Length': '0'} 2023-11-02T18:18:13.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008af42c851d5f4784-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:18:13.795 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:18:13.795 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:18:13.795 DEBUG:boto:Method: GET 2023-11-02T18:18:13.796 DEBUG:boto:Path: /ibtljn-49/obj2 2023-11-02T18:18:13.796 DEBUG:boto:Data: 2023-11-02T18:18:13.797 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.797 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.798 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.798 DEBUG:boto:Params: {} 2023-11-02T18:18:13.798 DEBUG:boto:Token: None 2023-11-02T18:18:13.799 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj2 2023-11-02T18:18:13.799 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0o7A3+ioBrJo/7i3Ia/RJriNj6w= 2023-11-02T18:18:13.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0o7A3+ioBrJo/7i3Ia/RJriNj6w=', 'Content-Length': '0'} 2023-11-02T18:18:13.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff07c69eb2e69eb1-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.803 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:18:13.804 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:18:13.804 DEBUG:boto:Method: GET 2023-11-02T18:18:13.805 DEBUG:boto:Path: /ibtljn-49/obj2 2023-11-02T18:18:13.805 DEBUG:boto:Data: 2023-11-02T18:18:13.806 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.806 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.806 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.807 DEBUG:boto:Params: {} 2023-11-02T18:18:13.807 DEBUG:boto:Token: None 2023-11-02T18:18:13.808 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj2 2023-11-02T18:18:13.808 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0o7A3+ioBrJo/7i3Ia/RJriNj6w= 2023-11-02T18:18:13.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0o7A3+ioBrJo/7i3Ia/RJriNj6w=', 'Content-Length': '0'} 2023-11-02T18:18:13.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065221e0a313d7cde-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.854 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.855 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.855 DEBUG:boto:path=/ibtljn-49/?prefix=obj2 2023-11-02T18:18:13.856 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj2 2023-11-02T18:18:13.856 DEBUG:boto:Method: GET 2023-11-02T18:18:13.857 DEBUG:boto:Path: /ibtljn-49/?prefix=obj2 2023-11-02T18:18:13.857 DEBUG:boto:Data: 2023-11-02T18:18:13.857 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.858 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.858 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.859 DEBUG:boto:Params: {} 2023-11-02T18:18:13.859 DEBUG:boto:Token: None 2023-11-02T18:18:13.860 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/ 2023-11-02T18:18:13.860 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM= 2023-11-02T18:18:13.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM=', 'Content-Length': '0'} 2023-11-02T18:18:13.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001778e43625ba34fb-006543e7e5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.866 DEBUG:boto:b'ibtljn-49obj21000falseobj22023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:13.867 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:13.867 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:13.868 DEBUG:boto:path=/ibtljn-49/?prefix=obj2 2023-11-02T18:18:13.868 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj2 2023-11-02T18:18:13.869 DEBUG:boto:Method: GET 2023-11-02T18:18:13.869 DEBUG:boto:Path: /ibtljn-49/?prefix=obj2 2023-11-02T18:18:13.870 DEBUG:boto:Data: 2023-11-02T18:18:13.870 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.871 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.871 DEBUG:boto:Params: {} 2023-11-02T18:18:13.872 DEBUG:boto:Token: None 2023-11-02T18:18:13.872 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/ 2023-11-02T18:18:13.873 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM= 2023-11-02T18:18:13.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XeekHgzB7b+X1ayXiwxyQ/RttIM=', 'Content-Length': '0'} 2023-11-02T18:18:13.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cd35acbbc8f64a8-006543e7e5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.879 DEBUG:boto:b'ibtljn-49obj21000falseobj22023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:13.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:18:13.880 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:18:13.880 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:18:13.881 DEBUG:boto:path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.881 DEBUG:boto:auth_path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.882 DEBUG:boto:Method: GET 2023-11-02T18:18:13.882 DEBUG:boto:Path: /ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.883 DEBUG:boto:Data: 2023-11-02T18:18:13.883 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.883 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.884 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.884 DEBUG:boto:Params: {} 2023-11-02T18:18:13.885 DEBUG:boto:Token: None 2023-11-02T18:18:13.885 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.886 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rVXvGgyGIkP21a3VfocVfYvbIXE= 2023-11-02T18:18:13.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rVXvGgyGIkP21a3VfocVfYvbIXE=', 'Content-Length': '0'} 2023-11-02T18:18:13.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c3afe3ed92a2701-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.930 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:18:13.931 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:18:13.931 DEBUG:boto:path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.932 DEBUG:boto:auth_path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.932 DEBUG:boto:Method: GET 2023-11-02T18:18:13.933 DEBUG:boto:Path: /ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.933 DEBUG:boto:Data: 2023-11-02T18:18:13.933 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.934 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.935 DEBUG:boto:Params: {} 2023-11-02T18:18:13.935 DEBUG:boto:Token: None 2023-11-02T18:18:13.936 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.936 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rVXvGgyGIkP21a3VfocVfYvbIXE= 2023-11-02T18:18:13.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rVXvGgyGIkP21a3VfocVfYvbIXE=', 'Content-Length': '0'} 2023-11-02T18:18:13.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a4a60b2e8fc4bd4-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.940 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:18:13.941 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:18:13.941 DEBUG:boto:path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.942 DEBUG:boto:auth_path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.942 DEBUG:boto:Method: HEAD 2023-11-02T18:18:13.942 DEBUG:boto:Path: /ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.943 DEBUG:boto:Data: 2023-11-02T18:18:13.943 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.943 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.944 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.944 DEBUG:boto:Params: {} 2023-11-02T18:18:13.945 DEBUG:boto:Token: None 2023-11-02T18:18:13.945 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.946 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3jYvqXnk0QbUxEJ12ndlMuO9o7o= 2023-11-02T18:18:13.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3jYvqXnk0QbUxEJ12ndlMuO9o7o=', 'Content-Length': '0'} 2023-11-02T18:18:13.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1c7586505de9373-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.951 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:18:13.951 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:18:13.952 DEBUG:boto:path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.952 DEBUG:boto:auth_path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.953 DEBUG:boto:Method: HEAD 2023-11-02T18:18:13.953 DEBUG:boto:Path: /ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.953 DEBUG:boto:Data: 2023-11-02T18:18:13.954 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.954 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.955 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.955 DEBUG:boto:Params: {} 2023-11-02T18:18:13.956 DEBUG:boto:Token: None 2023-11-02T18:18:13.956 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.957 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3jYvqXnk0QbUxEJ12ndlMuO9o7o= 2023-11-02T18:18:13.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3jYvqXnk0QbUxEJ12ndlMuO9o7o=', 'Content-Length': '0'} 2023-11-02T18:18:13.960 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e984067d8f7fda70-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:18:13.961 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:18:13.961 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:18:13.962 DEBUG:boto:path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.962 DEBUG:boto:auth_path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.963 DEBUG:boto:Method: GET 2023-11-02T18:18:13.963 DEBUG:boto:Path: /ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.964 DEBUG:boto:Data: 2023-11-02T18:18:13.964 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.964 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:13.965 DEBUG:boto:Port: 8001 2023-11-02T18:18:13.965 DEBUG:boto:Params: {} 2023-11-02T18:18:13.966 DEBUG:boto:Token: None 2023-11-02T18:18:13.966 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.967 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rVXvGgyGIkP21a3VfocVfYvbIXE= 2023-11-02T18:18:13.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rVXvGgyGIkP21a3VfocVfYvbIXE=', 'Content-Length': '0'} 2023-11-02T18:18:13.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d15b6a59cbdc3340-006543e7e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:13.971 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:18:13.971 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:18:13.972 DEBUG:boto:path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.972 DEBUG:boto:auth_path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.972 DEBUG:boto:Method: GET 2023-11-02T18:18:13.973 DEBUG:boto:Path: /ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.973 DEBUG:boto:Data: 2023-11-02T18:18:13.974 DEBUG:boto:Headers: {} 2023-11-02T18:18:13.974 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:13.974 DEBUG:boto:Port: 8000 2023-11-02T18:18:13.975 DEBUG:boto:Params: {} 2023-11-02T18:18:13.975 DEBUG:boto:Token: None 2023-11-02T18:18:13.976 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:13 GMT /ibtljn-49/obj3?versionId=null 2023-11-02T18:18:13.976 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rVXvGgyGIkP21a3VfocVfYvbIXE= 2023-11-02T18:18:13.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rVXvGgyGIkP21a3VfocVfYvbIXE=', 'Content-Length': '0'} 2023-11-02T18:18:13.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ba9d9d149903d9e-006543e7e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.022 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:14.023 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:14.023 DEBUG:boto:path=/ibtljn-49/?prefix=obj3 2023-11-02T18:18:14.024 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj3 2023-11-02T18:18:14.024 DEBUG:boto:Method: GET 2023-11-02T18:18:14.025 DEBUG:boto:Path: /ibtljn-49/?prefix=obj3 2023-11-02T18:18:14.025 DEBUG:boto:Data: 2023-11-02T18:18:14.025 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.026 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:14.026 DEBUG:boto:Port: 8001 2023-11-02T18:18:14.027 DEBUG:boto:Params: {} 2023-11-02T18:18:14.027 DEBUG:boto:Token: None 2023-11-02T18:18:14.028 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/ 2023-11-02T18:18:14.028 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY= 2023-11-02T18:18:14.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY=', 'Content-Length': '0'} 2023-11-02T18:18:14.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cea3b3a38067ae8-006543e7e6-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.034 DEBUG:boto:b'ibtljn-49obj31000falseobj32023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:14.035 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:14.035 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:14.036 DEBUG:boto:path=/ibtljn-49/?prefix=obj3 2023-11-02T18:18:14.036 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj3 2023-11-02T18:18:14.036 DEBUG:boto:Method: GET 2023-11-02T18:18:14.037 DEBUG:boto:Path: /ibtljn-49/?prefix=obj3 2023-11-02T18:18:14.037 DEBUG:boto:Data: 2023-11-02T18:18:14.038 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:14.039 DEBUG:boto:Port: 8000 2023-11-02T18:18:14.039 DEBUG:boto:Params: {} 2023-11-02T18:18:14.039 DEBUG:boto:Token: None 2023-11-02T18:18:14.040 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/ 2023-11-02T18:18:14.040 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY= 2023-11-02T18:18:14.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY=', 'Content-Length': '0'} 2023-11-02T18:18:14.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000670ef212f0336472-006543e7e6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.046 DEBUG:boto:b'ibtljn-49obj31000falseobj32023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:14.047 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:18:14.047 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:18:14.047 DEBUG:boto:Method: HEAD 2023-11-02T18:18:14.048 DEBUG:boto:Path: /ibtljn-49/obj3 2023-11-02T18:18:14.048 DEBUG:boto:Data: 2023-11-02T18:18:14.049 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.049 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:14.050 DEBUG:boto:Port: 8001 2023-11-02T18:18:14.050 DEBUG:boto:Params: {} 2023-11-02T18:18:14.050 DEBUG:boto:Token: None 2023-11-02T18:18:14.051 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj3 2023-11-02T18:18:14.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V3+6AcOyj69reAIOHg37tZ0hT3E= 2023-11-02T18:18:14.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V3+6AcOyj69reAIOHg37tZ0hT3E=', 'Content-Length': '0'} 2023-11-02T18:18:14.054 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2d5512a3a34a85a-006543e7e6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.055 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:18:14.055 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:18:14.056 DEBUG:boto:Method: HEAD 2023-11-02T18:18:14.056 DEBUG:boto:Path: /ibtljn-49/obj3 2023-11-02T18:18:14.056 DEBUG:boto:Data: 2023-11-02T18:18:14.057 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.057 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:14.058 DEBUG:boto:Port: 8000 2023-11-02T18:18:14.058 DEBUG:boto:Params: {} 2023-11-02T18:18:14.059 DEBUG:boto:Token: None 2023-11-02T18:18:14.059 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj3 2023-11-02T18:18:14.059 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V3+6AcOyj69reAIOHg37tZ0hT3E= 2023-11-02T18:18:14.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V3+6AcOyj69reAIOHg37tZ0hT3E=', 'Content-Length': '0'} 2023-11-02T18:18:14.063 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001073b08074c3192b-006543e7e6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:18:14.064 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:18:14.064 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:18:14.064 DEBUG:boto:Method: GET 2023-11-02T18:18:14.065 DEBUG:boto:Path: /ibtljn-49/obj3 2023-11-02T18:18:14.065 DEBUG:boto:Data: 2023-11-02T18:18:14.066 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.066 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:14.067 DEBUG:boto:Port: 8001 2023-11-02T18:18:14.067 DEBUG:boto:Params: {} 2023-11-02T18:18:14.068 DEBUG:boto:Token: None 2023-11-02T18:18:14.068 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj3 2023-11-02T18:18:14.069 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+lI2AgMCS4TDMp+01vLr4hHiv1s= 2023-11-02T18:18:14.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+lI2AgMCS4TDMp+01vLr4hHiv1s=', 'Content-Length': '0'} 2023-11-02T18:18:14.072 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000466d346dbd667efc-006543e7e6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.114 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:18:14.115 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:18:14.115 DEBUG:boto:Method: GET 2023-11-02T18:18:14.116 DEBUG:boto:Path: /ibtljn-49/obj3 2023-11-02T18:18:14.116 DEBUG:boto:Data: 2023-11-02T18:18:14.117 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.117 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:14.118 DEBUG:boto:Port: 8000 2023-11-02T18:18:14.118 DEBUG:boto:Params: {} 2023-11-02T18:18:14.119 DEBUG:boto:Token: None 2023-11-02T18:18:14.119 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj3 2023-11-02T18:18:14.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+lI2AgMCS4TDMp+01vLr4hHiv1s= 2023-11-02T18:18:14.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+lI2AgMCS4TDMp+01vLr4hHiv1s=', 'Content-Length': '0'} 2023-11-02T18:18:14.123 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000b95d3310464c22-006543e7e6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.124 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:14.124 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:14.125 DEBUG:boto:path=/ibtljn-49/?prefix=obj3 2023-11-02T18:18:14.125 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj3 2023-11-02T18:18:14.126 DEBUG:boto:Method: GET 2023-11-02T18:18:14.126 DEBUG:boto:Path: /ibtljn-49/?prefix=obj3 2023-11-02T18:18:14.127 DEBUG:boto:Data: 2023-11-02T18:18:14.127 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.127 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:14.128 DEBUG:boto:Port: 8001 2023-11-02T18:18:14.128 DEBUG:boto:Params: {} 2023-11-02T18:18:14.129 DEBUG:boto:Token: None 2023-11-02T18:18:14.129 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/ 2023-11-02T18:18:14.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY= 2023-11-02T18:18:14.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY=', 'Content-Length': '0'} 2023-11-02T18:18:14.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9eb378ffb6f7b1b-006543e7e6-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.136 DEBUG:boto:b'ibtljn-49obj31000falseobj32023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:14.137 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:14.137 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:14.137 DEBUG:boto:path=/ibtljn-49/?prefix=obj3 2023-11-02T18:18:14.138 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj3 2023-11-02T18:18:14.138 DEBUG:boto:Method: GET 2023-11-02T18:18:14.139 DEBUG:boto:Path: /ibtljn-49/?prefix=obj3 2023-11-02T18:18:14.139 DEBUG:boto:Data: 2023-11-02T18:18:14.140 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.140 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:14.141 DEBUG:boto:Port: 8000 2023-11-02T18:18:14.141 DEBUG:boto:Params: {} 2023-11-02T18:18:14.141 DEBUG:boto:Token: None 2023-11-02T18:18:14.142 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/ 2023-11-02T18:18:14.142 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY= 2023-11-02T18:18:14.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY=', 'Content-Length': '0'} 2023-11-02T18:18:14.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001de34ccef4028124-006543e7e6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.190 DEBUG:boto:b'ibtljn-49obj31000falseobj32023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:14.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:18:14.192 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:18:14.192 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:18:14.193 DEBUG:boto:path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.193 DEBUG:boto:auth_path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.194 DEBUG:boto:Method: GET 2023-11-02T18:18:14.194 DEBUG:boto:Path: /ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.194 DEBUG:boto:Data: 2023-11-02T18:18:14.195 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.195 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:14.196 DEBUG:boto:Port: 8001 2023-11-02T18:18:14.196 DEBUG:boto:Params: {} 2023-11-02T18:18:14.197 DEBUG:boto:Token: None 2023-11-02T18:18:14.197 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.198 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ucbY7/HuA7S0yLBXna2tVCrpmvI= 2023-11-02T18:18:14.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ucbY7/HuA7S0yLBXna2tVCrpmvI=', 'Content-Length': '0'} 2023-11-02T18:18:14.201 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084b0afb444fe39ad-006543e7e6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.202 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:18:14.202 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:18:14.203 DEBUG:boto:path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.203 DEBUG:boto:auth_path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.204 DEBUG:boto:Method: GET 2023-11-02T18:18:14.204 DEBUG:boto:Path: /ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.205 DEBUG:boto:Data: 2023-11-02T18:18:14.205 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.205 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:14.206 DEBUG:boto:Port: 8000 2023-11-02T18:18:14.206 DEBUG:boto:Params: {} 2023-11-02T18:18:14.207 DEBUG:boto:Token: None 2023-11-02T18:18:14.207 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ucbY7/HuA7S0yLBXna2tVCrpmvI= 2023-11-02T18:18:14.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ucbY7/HuA7S0yLBXna2tVCrpmvI=', 'Content-Length': '0'} 2023-11-02T18:18:14.212 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cae8db7fcfa6dd8-006543e7e6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.212 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:18:14.213 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:18:14.213 DEBUG:boto:path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.214 DEBUG:boto:auth_path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.214 DEBUG:boto:Method: HEAD 2023-11-02T18:18:14.214 DEBUG:boto:Path: /ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.215 DEBUG:boto:Data: 2023-11-02T18:18:14.215 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.216 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:14.216 DEBUG:boto:Port: 8001 2023-11-02T18:18:14.216 DEBUG:boto:Params: {} 2023-11-02T18:18:14.217 DEBUG:boto:Token: None 2023-11-02T18:18:14.217 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.218 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FlSpVwE8mZLom3c1BdJE7oK8syE= 2023-11-02T18:18:14.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FlSpVwE8mZLom3c1BdJE7oK8syE=', 'Content-Length': '0'} 2023-11-02T18:18:14.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001268cbaf728603c3-006543e7e6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.222 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:18:14.222 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:18:14.223 DEBUG:boto:path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.223 DEBUG:boto:auth_path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.223 DEBUG:boto:Method: HEAD 2023-11-02T18:18:14.224 DEBUG:boto:Path: /ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.224 DEBUG:boto:Data: 2023-11-02T18:18:14.224 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.225 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:14.225 DEBUG:boto:Port: 8000 2023-11-02T18:18:14.226 DEBUG:boto:Params: {} 2023-11-02T18:18:14.226 DEBUG:boto:Token: None 2023-11-02T18:18:14.227 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.227 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FlSpVwE8mZLom3c1BdJE7oK8syE= 2023-11-02T18:18:14.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FlSpVwE8mZLom3c1BdJE7oK8syE=', 'Content-Length': '0'} 2023-11-02T18:18:14.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000761df9a17b698e95-006543e7e6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:18:14.231 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:18:14.232 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:18:14.232 DEBUG:boto:path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.232 DEBUG:boto:auth_path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.233 DEBUG:boto:Method: GET 2023-11-02T18:18:14.233 DEBUG:boto:Path: /ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.234 DEBUG:boto:Data: 2023-11-02T18:18:14.234 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.235 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:14.235 DEBUG:boto:Port: 8001 2023-11-02T18:18:14.235 DEBUG:boto:Params: {} 2023-11-02T18:18:14.236 DEBUG:boto:Token: None 2023-11-02T18:18:14.236 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.237 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ucbY7/HuA7S0yLBXna2tVCrpmvI= 2023-11-02T18:18:14.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ucbY7/HuA7S0yLBXna2tVCrpmvI=', 'Content-Length': '0'} 2023-11-02T18:18:14.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007a23518360eea6a-006543e7e6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.241 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:18:14.241 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:18:14.241 DEBUG:boto:path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.242 DEBUG:boto:auth_path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.242 DEBUG:boto:Method: GET 2023-11-02T18:18:14.243 DEBUG:boto:Path: /ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.243 DEBUG:boto:Data: 2023-11-02T18:18:14.244 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.244 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:14.245 DEBUG:boto:Port: 8000 2023-11-02T18:18:14.245 DEBUG:boto:Params: {} 2023-11-02T18:18:14.246 DEBUG:boto:Token: None 2023-11-02T18:18:14.246 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj4?versionId=null 2023-11-02T18:18:14.246 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ucbY7/HuA7S0yLBXna2tVCrpmvI= 2023-11-02T18:18:14.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ucbY7/HuA7S0yLBXna2tVCrpmvI=', 'Content-Length': '0'} 2023-11-02T18:18:14.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae123a50251e8add-006543e7e6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.250 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:14.251 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:14.251 DEBUG:boto:path=/ibtljn-49/?prefix=obj4 2023-11-02T18:18:14.252 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj4 2023-11-02T18:18:14.252 DEBUG:boto:Method: GET 2023-11-02T18:18:14.253 DEBUG:boto:Path: /ibtljn-49/?prefix=obj4 2023-11-02T18:18:14.253 DEBUG:boto:Data: 2023-11-02T18:18:14.253 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:14.254 DEBUG:boto:Port: 8001 2023-11-02T18:18:14.255 DEBUG:boto:Params: {} 2023-11-02T18:18:14.255 DEBUG:boto:Token: None 2023-11-02T18:18:14.256 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/ 2023-11-02T18:18:14.256 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY= 2023-11-02T18:18:14.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY=', 'Content-Length': '0'} 2023-11-02T18:18:14.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000097d468bd9501234-006543e7e6-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.263 DEBUG:boto:b'ibtljn-49obj41000falseobj42023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:14.263 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:14.264 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:14.264 DEBUG:boto:path=/ibtljn-49/?prefix=obj4 2023-11-02T18:18:14.264 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj4 2023-11-02T18:18:14.265 DEBUG:boto:Method: GET 2023-11-02T18:18:14.265 DEBUG:boto:Path: /ibtljn-49/?prefix=obj4 2023-11-02T18:18:14.266 DEBUG:boto:Data: 2023-11-02T18:18:14.266 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.267 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:14.267 DEBUG:boto:Port: 8000 2023-11-02T18:18:14.267 DEBUG:boto:Params: {} 2023-11-02T18:18:14.268 DEBUG:boto:Token: None 2023-11-02T18:18:14.268 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/ 2023-11-02T18:18:14.269 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY= 2023-11-02T18:18:14.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY=', 'Content-Length': '0'} 2023-11-02T18:18:14.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000911a47826432d39a-006543e7e6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.275 DEBUG:boto:b'ibtljn-49obj41000falseobj42023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:14.276 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:18:14.276 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:18:14.277 DEBUG:boto:Method: HEAD 2023-11-02T18:18:14.277 DEBUG:boto:Path: /ibtljn-49/obj4 2023-11-02T18:18:14.277 DEBUG:boto:Data: 2023-11-02T18:18:14.278 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.278 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:14.279 DEBUG:boto:Port: 8001 2023-11-02T18:18:14.279 DEBUG:boto:Params: {} 2023-11-02T18:18:14.280 DEBUG:boto:Token: None 2023-11-02T18:18:14.280 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj4 2023-11-02T18:18:14.280 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a9nckczQjuaBh24eWo2eqIaOFv4= 2023-11-02T18:18:14.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a9nckczQjuaBh24eWo2eqIaOFv4=', 'Content-Length': '0'} 2023-11-02T18:18:14.284 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b98e08afd1835e65-006543e7e6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.284 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:18:14.285 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:18:14.285 DEBUG:boto:Method: HEAD 2023-11-02T18:18:14.285 DEBUG:boto:Path: /ibtljn-49/obj4 2023-11-02T18:18:14.286 DEBUG:boto:Data: 2023-11-02T18:18:14.286 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.287 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:14.287 DEBUG:boto:Port: 8000 2023-11-02T18:18:14.287 DEBUG:boto:Params: {} 2023-11-02T18:18:14.288 DEBUG:boto:Token: None 2023-11-02T18:18:14.288 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj4 2023-11-02T18:18:14.289 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a9nckczQjuaBh24eWo2eqIaOFv4= 2023-11-02T18:18:14.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a9nckczQjuaBh24eWo2eqIaOFv4=', 'Content-Length': '0'} 2023-11-02T18:18:14.292 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf735f69812ff646-006543e7e6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:18:14.293 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:18:14.293 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:18:14.294 DEBUG:boto:Method: GET 2023-11-02T18:18:14.294 DEBUG:boto:Path: /ibtljn-49/obj4 2023-11-02T18:18:14.295 DEBUG:boto:Data: 2023-11-02T18:18:14.295 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.296 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:14.296 DEBUG:boto:Port: 8001 2023-11-02T18:18:14.297 DEBUG:boto:Params: {} 2023-11-02T18:18:14.297 DEBUG:boto:Token: None 2023-11-02T18:18:14.298 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj4 2023-11-02T18:18:14.298 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e5XcJqyei8gnRB5QPoCJmV+W8YU= 2023-11-02T18:18:14.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e5XcJqyei8gnRB5QPoCJmV+W8YU=', 'Content-Length': '0'} 2023-11-02T18:18:14.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd3927b491da3aab-006543e7e6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.342 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:18:14.343 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:18:14.343 DEBUG:boto:Method: GET 2023-11-02T18:18:14.343 DEBUG:boto:Path: /ibtljn-49/obj4 2023-11-02T18:18:14.344 DEBUG:boto:Data: 2023-11-02T18:18:14.344 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.345 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:14.345 DEBUG:boto:Port: 8000 2023-11-02T18:18:14.346 DEBUG:boto:Params: {} 2023-11-02T18:18:14.346 DEBUG:boto:Token: None 2023-11-02T18:18:14.346 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/obj4 2023-11-02T18:18:14.347 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e5XcJqyei8gnRB5QPoCJmV+W8YU= 2023-11-02T18:18:14.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e5XcJqyei8gnRB5QPoCJmV+W8YU=', 'Content-Length': '0'} 2023-11-02T18:18:14.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008657c8acc1129cb4-006543e7e6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.351 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:14.351 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:14.352 DEBUG:boto:path=/ibtljn-49/?prefix=obj4 2023-11-02T18:18:14.352 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj4 2023-11-02T18:18:14.353 DEBUG:boto:Method: GET 2023-11-02T18:18:14.353 DEBUG:boto:Path: /ibtljn-49/?prefix=obj4 2023-11-02T18:18:14.353 DEBUG:boto:Data: 2023-11-02T18:18:14.354 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.354 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:14.355 DEBUG:boto:Port: 8001 2023-11-02T18:18:14.355 DEBUG:boto:Params: {} 2023-11-02T18:18:14.356 DEBUG:boto:Token: None 2023-11-02T18:18:14.356 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/ 2023-11-02T18:18:14.357 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY= 2023-11-02T18:18:14.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY=', 'Content-Length': '0'} 2023-11-02T18:18:14.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d71984cdbc593aac-006543e7e6-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.363 DEBUG:boto:b'ibtljn-49obj41000falseobj42023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:14.364 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:14.364 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:14.365 DEBUG:boto:path=/ibtljn-49/?prefix=obj4 2023-11-02T18:18:14.365 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj4 2023-11-02T18:18:14.366 DEBUG:boto:Method: GET 2023-11-02T18:18:14.366 DEBUG:boto:Path: /ibtljn-49/?prefix=obj4 2023-11-02T18:18:14.367 DEBUG:boto:Data: 2023-11-02T18:18:14.367 DEBUG:boto:Headers: {} 2023-11-02T18:18:14.367 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:14.368 DEBUG:boto:Port: 8000 2023-11-02T18:18:14.368 DEBUG:boto:Params: {} 2023-11-02T18:18:14.369 DEBUG:boto:Token: None 2023-11-02T18:18:14.369 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:18:14 GMT /ibtljn-49/ 2023-11-02T18:18:14.370 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY= 2023-11-02T18:18:14.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTHB0/rTg0/r2DrVv8JAkCYFgBY=', 'Content-Length': '0'} 2023-11-02T18:18:14.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edf23d493d65d84e-006543e7e6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:14.376 DEBUG:boto:b'ibtljn-49obj41000falseobj42023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:18:14.376 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-49 zones={a2, a1} 2023-11-02T18:18:14.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ibtljn-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:18:14.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ibtljn-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:18:14.378 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ibtljn-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:18:14.425 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:14.426 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:14.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ibtljn-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:18:14.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ibtljn-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:18:14.563 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ibtljn-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:18:14.606 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:14.606 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:14.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:18:14.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:18:14.738 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:18:14.782 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:14.782 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:14.843 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:18:14.872 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:18:14.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:14.872 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:18:14.873 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:18:14.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:14.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:14.874 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:18:14.874 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:18:14.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:14.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:14.875 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949094.703369_375.1", 2023-11-02T18:18:14.876 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:18:14.703369Z" 2023-11-02T18:18:14.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:14.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:14.876 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:18:14.877 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:18:14.877 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:14.877 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:18:14.878 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949094.703369_375.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:18:14.879 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:18:14.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:18:14.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:18:14.880 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:18:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:14.978+0000 7f3633d22f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:18:14.989 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:14.982+0000 7f3633d22f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:18:15.017 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:18:15.017 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:18:15.018 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:18:15.018 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:18:15.019 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:18:15.019 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:18:15.019 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:15.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:15.019 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:18:15.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:15.020 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:18:15.020 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:15.020 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:15.020 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:18:15.021 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:15.021 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:18:15.021 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:15.021 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:18:15.022 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:15.022 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:15.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:15.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:15.023 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:18:15.023 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:15.023 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:15.023 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:18:15.023 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:15.024 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:18:15.024 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:15.024 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:18:15.024 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:15.025 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:15.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:15.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:15.025 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:18:15.025 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:15.026 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:15.026 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949005.787608_369.1", 2023-11-02T18:18:15.026 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:15.026 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:18:15.027 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:15.027 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:16:45.787608Z", 2023-11-02T18:18:15.027 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:15.027 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:15.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:15.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:15.028 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:18:15.028 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:15.028 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:15.029 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:18:15.029 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:15.029 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:18:15.029 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:15.029 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:18:15.030 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:15.030 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:15.030 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:15.030 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:18:15.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:15.031 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:18:15.031 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:18:15.031 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:18:15.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:15.031 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:18:14Z" 2023-11-02T18:18:15.032 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:18:15.033 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949005.787608_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:16:45.787608Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:18:14Z" } 2023-11-02T18:18:15.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949005.787608_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:16:45.787608Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:18:15.034 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949094.703369_375.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:18:15.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949005.787608_369.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:18:15.035 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949094.703369_375.1 target=1_1698949005.787608_369.1 2023-11-02T18:18:20.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:18:20.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:18:20.037 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:18:20.083 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:20.083 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:20.143 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:20.138+0000 7f1445bd0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:18:20.145 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:20.138+0000 7f1445bd0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:18:20.171 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:18:20.171 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:18:20.172 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:18:20.172 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:18:20.172 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:18:20.172 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:18:20.172 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:20.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:20.173 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:18:20.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:20.173 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:18:20.173 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:20.174 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:20.174 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:18:20.174 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:20.174 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:18:20.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:20.175 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:18:20.175 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:20.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:20.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:20.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:20.176 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:18:20.176 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:20.176 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:20.176 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:18:20.177 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:20.177 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:18:20.177 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:20.177 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:18:20.177 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:20.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:20.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:20.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:20.178 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:18:20.179 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:20.179 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:20.179 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949005.787608_369.1", 2023-11-02T18:18:20.179 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:20.179 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:18:20.180 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:20.180 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:16:45.787608Z", 2023-11-02T18:18:20.180 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:20.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:20.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:20.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:20.181 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:18:20.181 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:20.181 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:20.182 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:18:20.182 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:20.182 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:18:20.182 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:20.183 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:18:20.183 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:20.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:20.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:20.183 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:18:20.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:20.184 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:18:20.184 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:18:20.184 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:18:20.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:20.185 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:18:20Z" 2023-11-02T18:18:20.185 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:18:20.186 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949005.787608_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:16:45.787608Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:18:20Z" } 2023-11-02T18:18:20.187 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949005.787608_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:16:45.787608Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:18:20.187 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949094.703369_375.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:18:20.187 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949005.787608_369.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:18:20.188 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949094.703369_375.1 target=1_1698949005.787608_369.1 2023-11-02T18:18:25.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:18:25.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:18:25.191 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:18:25.238 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:25.238 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:25.296 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:25.290+0000 7f76948cbf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:18:25.298 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:25.290+0000 7f76948cbf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:18:25.326 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:18:25.327 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:18:25.327 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:18:25.327 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:18:25.327 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:18:25.328 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:18:25.328 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:25.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:25.328 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:18:25.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:25.329 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:18:25.329 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:25.329 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:25.330 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:18:25.330 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:25.330 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:18:25.330 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:25.331 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:18:25.331 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:25.331 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:25.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:25.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:25.332 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:18:25.332 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:25.332 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:25.333 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:18:25.333 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:25.333 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:18:25.333 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:25.333 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:18:25.334 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:25.334 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:25.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:25.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:25.335 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:18:25.335 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:25.335 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:25.335 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949005.787608_369.1", 2023-11-02T18:18:25.336 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:25.336 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:18:25.336 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:25.336 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:16:45.787608Z", 2023-11-02T18:18:25.336 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:25.337 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:25.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:25.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:25.337 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:18:25.338 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:25.338 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:25.338 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:18:25.338 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:25.339 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:18:25.339 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:25.339 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:18:25.339 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:25.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:25.340 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:25.340 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:18:25.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:25.340 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:18:25.341 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:18:25.341 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:18:25.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:25.341 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:18:25Z" 2023-11-02T18:18:25.341 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:18:25.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949005.787608_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:16:45.787608Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:18:25Z" } 2023-11-02T18:18:25.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949005.787608_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:16:45.787608Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:18:25.344 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949094.703369_375.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:18:25.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949005.787608_369.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:18:25.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949094.703369_375.1 target=1_1698949005.787608_369.1 2023-11-02T18:18:30.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:18:30.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:18:30.347 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:18:30.394 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:30.394 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:30.454 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:30.446+0000 7f69070fef40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:18:30.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:30.450+0000 7f69070fef40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:18:30.482 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:18:30.483 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:18:30.483 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:18:30.483 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:18:30.483 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:18:30.484 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:18:30.484 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:30.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:30.484 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:18:30.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:30.485 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:18:30.485 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:30.485 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:30.485 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:18:30.486 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:30.486 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:18:30.486 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:30.486 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:18:30.487 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:30.487 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:30.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:30.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:30.488 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:18:30.488 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:30.488 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:30.488 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:18:30.489 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:30.489 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:18:30.489 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:30.489 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:18:30.489 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:30.490 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:30.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:30.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:30.490 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:18:30.491 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:30.491 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:30.491 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949094.703369_375.1", 2023-11-02T18:18:30.491 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:30.492 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:18:30.492 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:30.492 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:18:14.703369Z", 2023-11-02T18:18:30.492 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:30.493 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:30.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:30.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:18:30.493 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:18:30.494 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:18:30.494 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:18:30.494 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:18:30.494 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:18:30.494 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:18:30.495 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:18:30.495 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:18:30.495 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:18:30.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:30.496 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:18:30.496 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:18:30.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:30.496 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:18:30.496 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:18:30.497 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:18:30.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:18:30.497 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:18:30Z" 2023-11-02T18:18:30.497 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:18:30.498 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949094.703369_375.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:18:14.703369Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:18:30Z" } 2023-11-02T18:18:30.499 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949094.703369_375.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:18:14.703369Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:18:30.500 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949094.703369_375.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:18:30.500 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949094.703369_375.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:18:30.500 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:18:30.501 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:30.501 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:30.502 DEBUG:boto:Method: HEAD 2023-11-02T18:18:30.502 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:18:30.503 DEBUG:boto:Data: 2023-11-02T18:18:30.503 DEBUG:boto:Headers: {} 2023-11-02T18:18:30.503 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:30.504 DEBUG:boto:Port: 8000 2023-11-02T18:18:30.504 DEBUG:boto:Params: {} 2023-11-02T18:18:30.505 DEBUG:boto:Token: None 2023-11-02T18:18:30.505 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-48/ 2023-11-02T18:18:30.506 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tzQmSzisQNmNOYoy9NkW69Ows8k= 2023-11-02T18:18:30.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tzQmSzisQNmNOYoy9NkW69Ows8k=', 'Content-Length': '0'} 2023-11-02T18:18:30.509 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035c295bd9cdffbb7-006543e7f6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.510 DEBUG:boto:path=/ibtljn-48/obj5 2023-11-02T18:18:30.510 DEBUG:boto:auth_path=/ibtljn-48/obj5 2023-11-02T18:18:30.510 DEBUG:boto:Method: PUT 2023-11-02T18:18:30.511 DEBUG:boto:Path: /ibtljn-48/obj5 2023-11-02T18:18:30.511 DEBUG:boto:Data: 2023-11-02T18:18:30.512 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:18:30.512 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:30.512 DEBUG:boto:Port: 8000 2023-11-02T18:18:30.513 DEBUG:boto:Params: {} 2023-11-02T18:18:30.513 DEBUG:boto:Token: None 2023-11-02T18:18:30.514 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-48/obj5 2023-11-02T18:18:30.514 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bAgC0Oy9idxFjCKZ6zMO0SvaNxY= 2023-11-02T18:18:30.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bAgC0Oy9idxFjCKZ6zMO0SvaNxY='} 2023-11-02T18:18:30.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000754f1364dfca0ac4-006543e7f6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.523 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:30.523 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:30.524 DEBUG:boto:Method: HEAD 2023-11-02T18:18:30.524 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:18:30.525 DEBUG:boto:Data: 2023-11-02T18:18:30.525 DEBUG:boto:Headers: {} 2023-11-02T18:18:30.525 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:30.526 DEBUG:boto:Port: 8000 2023-11-02T18:18:30.526 DEBUG:boto:Params: {} 2023-11-02T18:18:30.527 DEBUG:boto:Token: None 2023-11-02T18:18:30.527 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-48/ 2023-11-02T18:18:30.528 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tzQmSzisQNmNOYoy9NkW69Ows8k= 2023-11-02T18:18:30.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tzQmSzisQNmNOYoy9NkW69Ows8k=', 'Content-Length': '0'} 2023-11-02T18:18:30.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002624c1add5bc6f12-006543e7f6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.532 DEBUG:boto:path=/ibtljn-48/obj6 2023-11-02T18:18:30.533 DEBUG:boto:auth_path=/ibtljn-48/obj6 2023-11-02T18:18:30.533 DEBUG:boto:Method: PUT 2023-11-02T18:18:30.533 DEBUG:boto:Path: /ibtljn-48/obj6 2023-11-02T18:18:30.535 DEBUG:boto:Data: 2023-11-02T18:18:30.536 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:18:30.536 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:30.537 DEBUG:boto:Port: 8000 2023-11-02T18:18:30.537 DEBUG:boto:Params: {} 2023-11-02T18:18:30.538 DEBUG:boto:Token: None 2023-11-02T18:18:30.538 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-48/obj6 2023-11-02T18:18:30.539 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OETKZ5k6rwdgsfVuS50fRSmB5RQ= 2023-11-02T18:18:30.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OETKZ5k6rwdgsfVuS50fRSmB5RQ='} 2023-11-02T18:18:30.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077be7386122cb9cd-006543e7f6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.587 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:30.587 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:30.588 DEBUG:boto:Method: HEAD 2023-11-02T18:18:30.588 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:18:30.589 DEBUG:boto:Data: 2023-11-02T18:18:30.589 DEBUG:boto:Headers: {} 2023-11-02T18:18:30.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:30.590 DEBUG:boto:Port: 8000 2023-11-02T18:18:30.590 DEBUG:boto:Params: {} 2023-11-02T18:18:30.591 DEBUG:boto:Token: None 2023-11-02T18:18:30.591 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-48/ 2023-11-02T18:18:30.592 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tzQmSzisQNmNOYoy9NkW69Ows8k= 2023-11-02T18:18:30.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tzQmSzisQNmNOYoy9NkW69Ows8k=', 'Content-Length': '0'} 2023-11-02T18:18:30.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1f1251fb5b68ce7-006543e7f6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.597 DEBUG:boto:path=/ibtljn-48/obj7 2023-11-02T18:18:30.598 DEBUG:boto:auth_path=/ibtljn-48/obj7 2023-11-02T18:18:30.598 DEBUG:boto:Method: PUT 2023-11-02T18:18:30.598 DEBUG:boto:Path: /ibtljn-48/obj7 2023-11-02T18:18:30.599 DEBUG:boto:Data: 2023-11-02T18:18:30.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:18:30.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:30.600 DEBUG:boto:Port: 8000 2023-11-02T18:18:30.601 DEBUG:boto:Params: {} 2023-11-02T18:18:30.601 DEBUG:boto:Token: None 2023-11-02T18:18:30.601 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-48/obj7 2023-11-02T18:18:30.602 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5iktDpMbGQ9JMeun3pArlU6ZtsY= 2023-11-02T18:18:30.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5iktDpMbGQ9JMeun3pArlU6ZtsY='} 2023-11-02T18:18:30.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085813956b7b42afc-006543e7f6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.612 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:18:30.613 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:18:30.613 DEBUG:boto:Method: HEAD 2023-11-02T18:18:30.614 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:18:30.614 DEBUG:boto:Data: 2023-11-02T18:18:30.615 DEBUG:boto:Headers: {} 2023-11-02T18:18:30.615 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:30.615 DEBUG:boto:Port: 8000 2023-11-02T18:18:30.616 DEBUG:boto:Params: {} 2023-11-02T18:18:30.616 DEBUG:boto:Token: None 2023-11-02T18:18:30.617 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-48/ 2023-11-02T18:18:30.617 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tzQmSzisQNmNOYoy9NkW69Ows8k= 2023-11-02T18:18:30.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tzQmSzisQNmNOYoy9NkW69Ows8k=', 'Content-Length': '0'} 2023-11-02T18:18:30.621 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b52d4fd48f0ae0c-006543e7f6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.621 DEBUG:boto:path=/ibtljn-48/obj8 2023-11-02T18:18:30.622 DEBUG:boto:auth_path=/ibtljn-48/obj8 2023-11-02T18:18:30.622 DEBUG:boto:Method: PUT 2023-11-02T18:18:30.623 DEBUG:boto:Path: /ibtljn-48/obj8 2023-11-02T18:18:30.623 DEBUG:boto:Data: 2023-11-02T18:18:30.624 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:18:30.624 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:18:30.624 DEBUG:boto:Port: 8000 2023-11-02T18:18:30.625 DEBUG:boto:Params: {} 2023-11-02T18:18:30.625 DEBUG:boto:Token: None 2023-11-02T18:18:30.626 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-48/obj8 2023-11-02T18:18:30.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nG90jBR14TmulrPP0EmITTDJODg= 2023-11-02T18:18:30.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nG90jBR14TmulrPP0EmITTDJODg='} 2023-11-02T18:18:30.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e4d2ff321da8c27-006543e7f6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.671 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:30.672 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:30.672 DEBUG:boto:Method: HEAD 2023-11-02T18:18:30.672 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:18:30.673 DEBUG:boto:Data: 2023-11-02T18:18:30.673 DEBUG:boto:Headers: {} 2023-11-02T18:18:30.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:30.674 DEBUG:boto:Port: 8001 2023-11-02T18:18:30.675 DEBUG:boto:Params: {} 2023-11-02T18:18:30.675 DEBUG:boto:Token: None 2023-11-02T18:18:30.676 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-49/ 2023-11-02T18:18:30.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jee/5D4LfgmxFzFSdvEKqtA0aBo= 2023-11-02T18:18:30.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jee/5D4LfgmxFzFSdvEKqtA0aBo=', 'Content-Length': '0'} 2023-11-02T18:18:30.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004b7770dbef498e3-006543e7f6-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.682 DEBUG:boto:path=/ibtljn-49/obj5 2023-11-02T18:18:30.682 DEBUG:boto:auth_path=/ibtljn-49/obj5 2023-11-02T18:18:30.683 DEBUG:boto:Method: PUT 2023-11-02T18:18:30.683 DEBUG:boto:Path: /ibtljn-49/obj5 2023-11-02T18:18:30.684 DEBUG:boto:Data: 2023-11-02T18:18:30.684 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:18:30.684 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:30.685 DEBUG:boto:Port: 8001 2023-11-02T18:18:30.685 DEBUG:boto:Params: {} 2023-11-02T18:18:30.686 DEBUG:boto:Token: None 2023-11-02T18:18:30.686 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-49/obj5 2023-11-02T18:18:30.687 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kP2TfV+HCMFHa7bAz5gKDs2f1Uw= 2023-11-02T18:18:30.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kP2TfV+HCMFHa7bAz5gKDs2f1Uw='} 2023-11-02T18:18:30.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008348e93d426ff4d6-006543e7f6-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.696 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:30.697 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:30.697 DEBUG:boto:Method: HEAD 2023-11-02T18:18:30.698 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:18:30.698 DEBUG:boto:Data: 2023-11-02T18:18:30.698 DEBUG:boto:Headers: {} 2023-11-02T18:18:30.699 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:30.699 DEBUG:boto:Port: 8001 2023-11-02T18:18:30.700 DEBUG:boto:Params: {} 2023-11-02T18:18:30.700 DEBUG:boto:Token: None 2023-11-02T18:18:30.701 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-49/ 2023-11-02T18:18:30.701 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jee/5D4LfgmxFzFSdvEKqtA0aBo= 2023-11-02T18:18:30.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jee/5D4LfgmxFzFSdvEKqtA0aBo=', 'Content-Length': '0'} 2023-11-02T18:18:30.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a57a6c537af0b53-006543e7f6-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.706 DEBUG:boto:path=/ibtljn-49/obj6 2023-11-02T18:18:30.707 DEBUG:boto:auth_path=/ibtljn-49/obj6 2023-11-02T18:18:30.707 DEBUG:boto:Method: PUT 2023-11-02T18:18:30.708 DEBUG:boto:Path: /ibtljn-49/obj6 2023-11-02T18:18:30.708 DEBUG:boto:Data: 2023-11-02T18:18:30.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:18:30.709 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:30.709 DEBUG:boto:Port: 8001 2023-11-02T18:18:30.710 DEBUG:boto:Params: {} 2023-11-02T18:18:30.710 DEBUG:boto:Token: None 2023-11-02T18:18:30.711 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-49/obj6 2023-11-02T18:18:30.711 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SZ92Fe26G6yCwRp3Jp6SlZNsYOM= 2023-11-02T18:18:30.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SZ92Fe26G6yCwRp3Jp6SlZNsYOM='} 2023-11-02T18:18:30.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f91888cc8f1c0241-006543e7f6-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.759 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:30.759 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:30.761 DEBUG:boto:Method: HEAD 2023-11-02T18:18:30.762 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:18:30.762 DEBUG:boto:Data: 2023-11-02T18:18:30.763 DEBUG:boto:Headers: {} 2023-11-02T18:18:30.763 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:30.764 DEBUG:boto:Port: 8001 2023-11-02T18:18:30.765 DEBUG:boto:Params: {} 2023-11-02T18:18:30.765 DEBUG:boto:Token: None 2023-11-02T18:18:30.766 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-49/ 2023-11-02T18:18:30.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jee/5D4LfgmxFzFSdvEKqtA0aBo= 2023-11-02T18:18:30.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jee/5D4LfgmxFzFSdvEKqtA0aBo=', 'Content-Length': '0'} 2023-11-02T18:18:30.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a65b6ed96029cd8d-006543e7f6-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.772 DEBUG:boto:path=/ibtljn-49/obj7 2023-11-02T18:18:30.772 DEBUG:boto:auth_path=/ibtljn-49/obj7 2023-11-02T18:18:30.772 DEBUG:boto:Method: PUT 2023-11-02T18:18:30.773 DEBUG:boto:Path: /ibtljn-49/obj7 2023-11-02T18:18:30.773 DEBUG:boto:Data: 2023-11-02T18:18:30.774 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:18:30.774 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:30.775 DEBUG:boto:Port: 8001 2023-11-02T18:18:30.775 DEBUG:boto:Params: {} 2023-11-02T18:18:30.775 DEBUG:boto:Token: None 2023-11-02T18:18:30.776 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-49/obj7 2023-11-02T18:18:30.776 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2hPDOHPH+KSALeJ3T/bFqwAsgEI= 2023-11-02T18:18:30.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2hPDOHPH+KSALeJ3T/bFqwAsgEI='} 2023-11-02T18:18:30.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092b067e7ac2be16c-006543e7f6-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.786 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:18:30.787 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:18:30.787 DEBUG:boto:Method: HEAD 2023-11-02T18:18:30.788 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:18:30.788 DEBUG:boto:Data: 2023-11-02T18:18:30.789 DEBUG:boto:Headers: {} 2023-11-02T18:18:30.789 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:30.790 DEBUG:boto:Port: 8001 2023-11-02T18:18:30.790 DEBUG:boto:Params: {} 2023-11-02T18:18:30.791 DEBUG:boto:Token: None 2023-11-02T18:18:30.791 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-49/ 2023-11-02T18:18:30.792 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jee/5D4LfgmxFzFSdvEKqtA0aBo= 2023-11-02T18:18:30.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jee/5D4LfgmxFzFSdvEKqtA0aBo=', 'Content-Length': '0'} 2023-11-02T18:18:30.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d297963071afa405-006543e7f6-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.796 DEBUG:boto:path=/ibtljn-49/obj8 2023-11-02T18:18:30.797 DEBUG:boto:auth_path=/ibtljn-49/obj8 2023-11-02T18:18:30.797 DEBUG:boto:Method: PUT 2023-11-02T18:18:30.798 DEBUG:boto:Path: /ibtljn-49/obj8 2023-11-02T18:18:30.798 DEBUG:boto:Data: 2023-11-02T18:18:30.799 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T18:18:30.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:18:30.799 DEBUG:boto:Port: 8001 2023-11-02T18:18:30.800 DEBUG:boto:Params: {} 2023-11-02T18:18:30.800 DEBUG:boto:Token: None 2023-11-02T18:18:30.801 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 18:18:30 GMT /ibtljn-49/obj8 2023-11-02T18:18:30.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0ulvog0oyFu3PPMscBwArigy85Q= 2023-11-02T18:18:30.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:18:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0ulvog0oyFu3PPMscBwArigy85Q='} 2023-11-02T18:18:30.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b48a9e359f09e257-006543e7f6-1665-a2'), ('Date', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:18:30.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'ibtljn-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:18:30.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'ibtljn-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:18:30.848 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket ibtljn-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:18:30.897 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:30.897 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:31.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'ibtljn-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:18:31.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'ibtljn-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:18:31.034 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket ibtljn-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:18:31.078 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:31.079 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:31.214 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-48 2023-11-02T18:18:31.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:18:31.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:18:31.216 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:18:31.271 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:31.272 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:31.336 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:31.330+0000 7f96793c8f40 1 waiting to reach incremental sync.. 2023-11-02T18:18:36.337 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:36.330+0000 7f96793c8f40 1 waiting to reach incremental sync.. 2023-11-02T18:18:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:41.330+0000 7f96793c8f40 1 waiting to reach incremental sync.. 2023-11-02T18:18:46.342 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:46.334+0000 7f96793c8f40 1 bucket sync caught up with source: 2023-11-02T18:18:46.342 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000004.30.1, 00000000002.25.1, 00000000004.23.1, 00000000002.14.1, 00000000004.30.1, 00000000002.29.1, 00000000002.36.1, 00000000004.37.1, 00000000002.20.1, 00000000002.39.1, 00000000002.23.1] 2023-11-02T18:18:46.342 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.30.1, 00000000002.25.1, 00000000004.23.1, 00000000002.14.1, 00000000004.30.1, 00000000002.29.1, 00000000002.36.1, 00000000004.37.1, 00000000002.20.1, 00000000002.39.1, 00000000002.23.1] 2023-11-02T18:18:46.343 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:46.334+0000 7f96793c8f40 0 bucket checkpoint complete 2023-11-02T18:18:46.372 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-48 2023-11-02T18:18:46.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:18:46.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:18:46.373 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:18:46.426 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:18:46.426 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:18:46.496 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:46.490+0000 7f09df453f40 1 waiting for incremental sync to catch up: 2023-11-02T18:18:46.496 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000003.415.1, 00000000002.109.1, 00000000003.70.1, 00000000002.54.1, 00000000003.305.1, 00000000002.247.1, 00000000002.206.1, 00000000003.213.1, 00000000002.188.1, 00000000002.202.1, 00000000002.165.1] 2023-11-02T18:18:46.496 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.421.6, 00000000002.109.1, 00000000004.76.6, 00000000002.54.1, 00000000004.311.6, 00000000002.247.1, 00000000002.206.1, 00000000004.217.6, 00000000002.188.1, 00000000002.202.1, 00000000002.165.1] 2023-11-02T18:18:51.498 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:51.490+0000 7f09df453f40 1 waiting for incremental sync to catch up: 2023-11-02T18:18:51.499 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000003.415.1, 00000000002.109.1, 00000000003.70.1, 00000000002.54.1, 00000000003.305.1, 00000000002.247.1, 00000000002.206.1, 00000000003.213.1, 00000000002.188.1, 00000000002.202.1, 00000000002.165.1] 2023-11-02T18:18:51.499 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.421.6, 00000000002.109.1, 00000000004.76.6, 00000000002.54.1, 00000000004.311.6, 00000000002.247.1, 00000000002.206.1, 00000000004.217.6, 00000000002.188.1, 00000000002.202.1, 00000000002.165.1] 2023-11-02T18:18:56.500 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:18:56.494+0000 7f09df453f40 1 waiting for incremental sync to catch up: 2023-11-02T18:18:56.501 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000003.415.1, 00000000002.109.1, 00000000004.76.6, 00000000002.54.1, 00000000003.305.1, 00000000002.247.1, 00000000002.206.1, 00000000004.217.6, 00000000002.188.1, 00000000002.202.1, 00000000002.165.1] 2023-11-02T18:18:56.501 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.421.6, 00000000002.109.1, 00000000004.76.6, 00000000002.54.1, 00000000004.311.6, 00000000002.247.1, 00000000002.206.1, 00000000004.217.6, 00000000002.188.1, 00000000002.202.1, 00000000002.165.1] 2023-11-02T18:19:01.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:01.494+0000 7f09df453f40 1 waiting for incremental sync to catch up: 2023-11-02T18:19:01.503 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000003.415.1, 00000000002.109.1, 00000000004.76.6, 00000000002.54.1, 00000000003.305.1, 00000000002.247.1, 00000000002.206.1, 00000000004.217.6, 00000000002.188.1, 00000000002.202.1, 00000000002.165.1] 2023-11-02T18:19:01.503 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.421.6, 00000000002.109.1, 00000000004.76.6, 00000000002.54.1, 00000000004.311.6, 00000000002.247.1, 00000000002.206.1, 00000000004.217.6, 00000000002.188.1, 00000000002.202.1, 00000000002.165.1] 2023-11-02T18:19:06.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:06.498+0000 7f09df453f40 1 bucket sync caught up with source: 2023-11-02T18:19:06.506 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000004.421.6, 00000000002.109.1, 00000000004.76.6, 00000000002.54.1, 00000000004.311.6, 00000000002.247.1, 00000000002.206.1, 00000000004.217.6, 00000000002.188.1, 00000000002.202.1, 00000000002.165.1] 2023-11-02T18:19:06.506 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.421.6, 00000000002.109.1, 00000000004.76.6, 00000000002.54.1, 00000000004.311.6, 00000000002.247.1, 00000000002.206.1, 00000000004.217.6, 00000000002.188.1, 00000000002.202.1, 00000000002.165.1] 2023-11-02T18:19:06.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:06.498+0000 7f09df453f40 0 bucket checkpoint complete 2023-11-02T18:19:06.536 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-48 zones={a2, a1} 2023-11-02T18:19:06.537 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:06.537 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:06.538 DEBUG:boto:Method: HEAD 2023-11-02T18:19:06.538 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:19:06.539 DEBUG:boto:Data: 2023-11-02T18:19:06.539 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.539 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.540 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.540 DEBUG:boto:Params: {} 2023-11-02T18:19:06.541 DEBUG:boto:Token: None 2023-11-02T18:19:06.541 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/ 2023-11-02T18:19:06.542 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZCbkFf+DxEiT3M/75xRKsGa05Wc= 2023-11-02T18:19:06.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZCbkFf+DxEiT3M/75xRKsGa05Wc=', 'Content-Length': '0'} 2023-11-02T18:19:06.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093c382efc2870c7c-006543e81a-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.547 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:06.547 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:06.548 DEBUG:boto:Method: HEAD 2023-11-02T18:19:06.548 DEBUG:boto:Path: /ibtljn-48/ 2023-11-02T18:19:06.548 DEBUG:boto:Data: 2023-11-02T18:19:06.549 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.549 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.550 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.550 DEBUG:boto:Params: {} 2023-11-02T18:19:06.551 DEBUG:boto:Token: None 2023-11-02T18:19:06.551 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/ 2023-11-02T18:19:06.552 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZCbkFf+DxEiT3M/75xRKsGa05Wc= 2023-11-02T18:19:06.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZCbkFf+DxEiT3M/75xRKsGa05Wc=', 'Content-Length': '0'} 2023-11-02T18:19:06.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000481c908af63a39be-006543e81a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.557 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:19:06.557 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:06.558 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:06.558 DEBUG:boto:path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:19:06.559 DEBUG:boto:auth_path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:19:06.559 DEBUG:boto:Method: GET 2023-11-02T18:19:06.559 DEBUG:boto:Path: /ibtljn-48/?versions&max-keys=999 2023-11-02T18:19:06.560 DEBUG:boto:Data: 2023-11-02T18:19:06.560 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.561 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.561 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.562 DEBUG:boto:Params: {} 2023-11-02T18:19:06.562 DEBUG:boto:Token: None 2023-11-02T18:19:06.563 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/?versions 2023-11-02T18:19:06.563 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a0hgAfS6hbfyaLp3yY0pUKhuiiU= 2023-11-02T18:19:06.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a0hgAfS6hbfyaLp3yY0pUKhuiiU=', 'Content-Length': '0'} 2023-11-02T18:19:06.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad6654d17d26bae-006543e81a-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.569 DEBUG:boto:b'ibtljn-48999falseobj1nulltrue2023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:18:30.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:18:30.540Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:18:30.601Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:18:30.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:06.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:19:06.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:19:06.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:19:06.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:19:06.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-11-02T18:19:06.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-11-02T18:19:06.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-11-02T18:19:06.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-11-02T18:19:06.573 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:19:06.573 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:06.574 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:06.574 DEBUG:boto:path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:19:06.574 DEBUG:boto:auth_path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:19:06.575 DEBUG:boto:Method: GET 2023-11-02T18:19:06.575 DEBUG:boto:Path: /ibtljn-48/?versions&max-keys=999 2023-11-02T18:19:06.575 DEBUG:boto:Data: 2023-11-02T18:19:06.576 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.576 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.577 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.577 DEBUG:boto:Params: {} 2023-11-02T18:19:06.578 DEBUG:boto:Token: None 2023-11-02T18:19:06.578 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/?versions 2023-11-02T18:19:06.579 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a0hgAfS6hbfyaLp3yY0pUKhuiiU= 2023-11-02T18:19:06.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a0hgAfS6hbfyaLp3yY0pUKhuiiU=', 'Content-Length': '0'} 2023-11-02T18:19:06.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b3a417dee5b2878-006543e81a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.584 DEBUG:boto:b'ibtljn-48999falseobj1nulltrue2023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:18:30.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:18:30.540Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:18:30.601Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:18:30.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:06.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:19:06.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:19:06.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:19:06.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:19:06.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-11-02T18:19:06.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-11-02T18:19:06.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-11-02T18:19:06.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-11-02T18:19:06.589 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:06.589 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:06.589 DEBUG:boto:path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:19:06.590 DEBUG:boto:auth_path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:19:06.590 DEBUG:boto:Method: GET 2023-11-02T18:19:06.591 DEBUG:boto:Path: /ibtljn-48/?versions&max-keys=999 2023-11-02T18:19:06.591 DEBUG:boto:Data: 2023-11-02T18:19:06.592 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.592 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.592 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.593 DEBUG:boto:Params: {} 2023-11-02T18:19:06.593 DEBUG:boto:Token: None 2023-11-02T18:19:06.594 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/?versions 2023-11-02T18:19:06.594 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a0hgAfS6hbfyaLp3yY0pUKhuiiU= 2023-11-02T18:19:06.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a0hgAfS6hbfyaLp3yY0pUKhuiiU=', 'Content-Length': '0'} 2023-11-02T18:19:06.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9d76c199499ab65-006543e81a-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.600 DEBUG:boto:b'ibtljn-48999falseobj1nulltrue2023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:18:30.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:18:30.540Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:18:30.601Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:18:30.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:06.600 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:06.601 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:06.601 DEBUG:boto:path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:19:06.602 DEBUG:boto:auth_path=/ibtljn-48/?versions&max-keys=999 2023-11-02T18:19:06.602 DEBUG:boto:Method: GET 2023-11-02T18:19:06.603 DEBUG:boto:Path: /ibtljn-48/?versions&max-keys=999 2023-11-02T18:19:06.603 DEBUG:boto:Data: 2023-11-02T18:19:06.603 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.604 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.604 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.605 DEBUG:boto:Params: {} 2023-11-02T18:19:06.605 DEBUG:boto:Token: None 2023-11-02T18:19:06.606 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/?versions 2023-11-02T18:19:06.606 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a0hgAfS6hbfyaLp3yY0pUKhuiiU= 2023-11-02T18:19:06.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a0hgAfS6hbfyaLp3yY0pUKhuiiU=', 'Content-Length': '0'} 2023-11-02T18:19:06.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d65108ce6af1a383-006543e81a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.612 DEBUG:boto:b'ibtljn-48999falseobj1nulltrue2023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:18:30.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:18:30.540Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:18:30.601Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:18:30.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:06.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:19:06.614 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:19:06.614 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:19:06.615 DEBUG:boto:path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.615 DEBUG:boto:auth_path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.615 DEBUG:boto:Method: GET 2023-11-02T18:19:06.616 DEBUG:boto:Path: /ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.616 DEBUG:boto:Data: 2023-11-02T18:19:06.617 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.617 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.618 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.618 DEBUG:boto:Params: {} 2023-11-02T18:19:06.619 DEBUG:boto:Token: None 2023-11-02T18:19:06.619 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.620 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AJltK8D5bOv2sGW75A6j8/pIKag= 2023-11-02T18:19:06.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AJltK8D5bOv2sGW75A6j8/pIKag=', 'Content-Length': '0'} 2023-11-02T18:19:06.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d56bfa6c725f5444-006543e81a-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.670 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:19:06.671 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:19:06.671 DEBUG:boto:path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.672 DEBUG:boto:auth_path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.672 DEBUG:boto:Method: GET 2023-11-02T18:19:06.673 DEBUG:boto:Path: /ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.673 DEBUG:boto:Data: 2023-11-02T18:19:06.674 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.675 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.675 DEBUG:boto:Params: {} 2023-11-02T18:19:06.676 DEBUG:boto:Token: None 2023-11-02T18:19:06.676 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AJltK8D5bOv2sGW75A6j8/pIKag= 2023-11-02T18:19:06.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AJltK8D5bOv2sGW75A6j8/pIKag=', 'Content-Length': '0'} 2023-11-02T18:19:06.680 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0c791a9b93da09b-006543e81a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.681 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:19:06.682 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:19:06.682 DEBUG:boto:path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.682 DEBUG:boto:auth_path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.683 DEBUG:boto:Method: HEAD 2023-11-02T18:19:06.683 DEBUG:boto:Path: /ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.684 DEBUG:boto:Data: 2023-11-02T18:19:06.684 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.684 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.685 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.685 DEBUG:boto:Params: {} 2023-11-02T18:19:06.686 DEBUG:boto:Token: None 2023-11-02T18:19:06.686 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.687 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ai8bHQip7oxbyeqZ0aRWyY7N3BQ= 2023-11-02T18:19:06.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ai8bHQip7oxbyeqZ0aRWyY7N3BQ=', 'Content-Length': '0'} 2023-11-02T18:19:06.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7e2eda2a73591de-006543e81a-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.691 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:19:06.692 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:19:06.692 DEBUG:boto:path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.693 DEBUG:boto:auth_path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.693 DEBUG:boto:Method: HEAD 2023-11-02T18:19:06.693 DEBUG:boto:Path: /ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.694 DEBUG:boto:Data: 2023-11-02T18:19:06.694 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.695 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.696 DEBUG:boto:Params: {} 2023-11-02T18:19:06.696 DEBUG:boto:Token: None 2023-11-02T18:19:06.697 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.697 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ai8bHQip7oxbyeqZ0aRWyY7N3BQ= 2023-11-02T18:19:06.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ai8bHQip7oxbyeqZ0aRWyY7N3BQ=', 'Content-Length': '0'} 2023-11-02T18:19:06.701 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e40ab675667e319-006543e81a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:19:06.701 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:19:06.702 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:19:06.702 DEBUG:boto:path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.703 DEBUG:boto:auth_path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.703 DEBUG:boto:Method: GET 2023-11-02T18:19:06.704 DEBUG:boto:Path: /ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.704 DEBUG:boto:Data: 2023-11-02T18:19:06.705 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.705 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.705 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.706 DEBUG:boto:Params: {} 2023-11-02T18:19:06.706 DEBUG:boto:Token: None 2023-11-02T18:19:06.707 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.707 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AJltK8D5bOv2sGW75A6j8/pIKag= 2023-11-02T18:19:06.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AJltK8D5bOv2sGW75A6j8/pIKag=', 'Content-Length': '0'} 2023-11-02T18:19:06.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eada218b023b9636-006543e81a-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.711 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:19:06.712 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:19:06.712 DEBUG:boto:path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.713 DEBUG:boto:auth_path=/ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.713 DEBUG:boto:Method: GET 2023-11-02T18:19:06.714 DEBUG:boto:Path: /ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.714 DEBUG:boto:Data: 2023-11-02T18:19:06.715 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.715 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.715 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.716 DEBUG:boto:Params: {} 2023-11-02T18:19:06.716 DEBUG:boto:Token: None 2023-11-02T18:19:06.717 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj1?versionId=null 2023-11-02T18:19:06.717 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AJltK8D5bOv2sGW75A6j8/pIKag= 2023-11-02T18:19:06.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AJltK8D5bOv2sGW75A6j8/pIKag=', 'Content-Length': '0'} 2023-11-02T18:19:06.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003320d8e3167b925b-006543e81a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.762 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:06.763 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:06.763 DEBUG:boto:path=/ibtljn-48/?prefix=obj1 2023-11-02T18:19:06.764 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj1 2023-11-02T18:19:06.764 DEBUG:boto:Method: GET 2023-11-02T18:19:06.765 DEBUG:boto:Path: /ibtljn-48/?prefix=obj1 2023-11-02T18:19:06.765 DEBUG:boto:Data: 2023-11-02T18:19:06.766 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.766 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.766 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.767 DEBUG:boto:Params: {} 2023-11-02T18:19:06.767 DEBUG:boto:Token: None 2023-11-02T18:19:06.768 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/ 2023-11-02T18:19:06.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2eL2Tg+SE72guert5ac08LIugIg= 2023-11-02T18:19:06.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2eL2Tg+SE72guert5ac08LIugIg=', 'Content-Length': '0'} 2023-11-02T18:19:06.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8c6c7eb61c6bb6b-006543e81a-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.774 DEBUG:boto:b'ibtljn-48obj11000falseobj12023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:06.775 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:06.775 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:06.776 DEBUG:boto:path=/ibtljn-48/?prefix=obj1 2023-11-02T18:19:06.776 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj1 2023-11-02T18:19:06.777 DEBUG:boto:Method: GET 2023-11-02T18:19:06.777 DEBUG:boto:Path: /ibtljn-48/?prefix=obj1 2023-11-02T18:19:06.777 DEBUG:boto:Data: 2023-11-02T18:19:06.778 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.778 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.779 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.779 DEBUG:boto:Params: {} 2023-11-02T18:19:06.780 DEBUG:boto:Token: None 2023-11-02T18:19:06.780 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/ 2023-11-02T18:19:06.781 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2eL2Tg+SE72guert5ac08LIugIg= 2023-11-02T18:19:06.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2eL2Tg+SE72guert5ac08LIugIg=', 'Content-Length': '0'} 2023-11-02T18:19:06.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa17855e340e62d4-006543e81a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.786 DEBUG:boto:b'ibtljn-48obj11000falseobj12023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:06.787 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:19:06.787 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:19:06.788 DEBUG:boto:Method: HEAD 2023-11-02T18:19:06.788 DEBUG:boto:Path: /ibtljn-48/obj1 2023-11-02T18:19:06.789 DEBUG:boto:Data: 2023-11-02T18:19:06.789 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.790 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.790 DEBUG:boto:Params: {} 2023-11-02T18:19:06.791 DEBUG:boto:Token: None 2023-11-02T18:19:06.791 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj1 2023-11-02T18:19:06.792 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i8f8EyuDucODNMSl9o7p9kjMd/0= 2023-11-02T18:19:06.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i8f8EyuDucODNMSl9o7p9kjMd/0=', 'Content-Length': '0'} 2023-11-02T18:19:06.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097df9cf953f973f2-006543e81a-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.796 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:19:06.796 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:19:06.796 DEBUG:boto:Method: HEAD 2023-11-02T18:19:06.797 DEBUG:boto:Path: /ibtljn-48/obj1 2023-11-02T18:19:06.797 DEBUG:boto:Data: 2023-11-02T18:19:06.798 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.798 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.798 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.799 DEBUG:boto:Params: {} 2023-11-02T18:19:06.799 DEBUG:boto:Token: None 2023-11-02T18:19:06.800 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj1 2023-11-02T18:19:06.800 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i8f8EyuDucODNMSl9o7p9kjMd/0= 2023-11-02T18:19:06.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i8f8EyuDucODNMSl9o7p9kjMd/0=', 'Content-Length': '0'} 2023-11-02T18:19:06.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c104a00d2953be8-006543e81a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:19:06.804 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:19:06.805 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:19:06.805 DEBUG:boto:Method: GET 2023-11-02T18:19:06.805 DEBUG:boto:Path: /ibtljn-48/obj1 2023-11-02T18:19:06.806 DEBUG:boto:Data: 2023-11-02T18:19:06.806 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.807 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.807 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.807 DEBUG:boto:Params: {} 2023-11-02T18:19:06.808 DEBUG:boto:Token: None 2023-11-02T18:19:06.808 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj1 2023-11-02T18:19:06.809 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Kg4OnkeAM9DUzBHtRor0yA48jsA= 2023-11-02T18:19:06.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Kg4OnkeAM9DUzBHtRor0yA48jsA=', 'Content-Length': '0'} 2023-11-02T18:19:06.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dec12156b7e69d20-006543e81a-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.854 DEBUG:boto:path=/ibtljn-48/obj1 2023-11-02T18:19:06.855 DEBUG:boto:auth_path=/ibtljn-48/obj1 2023-11-02T18:19:06.855 DEBUG:boto:Method: GET 2023-11-02T18:19:06.856 DEBUG:boto:Path: /ibtljn-48/obj1 2023-11-02T18:19:06.856 DEBUG:boto:Data: 2023-11-02T18:19:06.857 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.858 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.858 DEBUG:boto:Params: {} 2023-11-02T18:19:06.859 DEBUG:boto:Token: None 2023-11-02T18:19:06.859 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj1 2023-11-02T18:19:06.860 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Kg4OnkeAM9DUzBHtRor0yA48jsA= 2023-11-02T18:19:06.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Kg4OnkeAM9DUzBHtRor0yA48jsA=', 'Content-Length': '0'} 2023-11-02T18:19:06.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1aa9dce8a1602cd-006543e81a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.864 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:06.864 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:06.865 DEBUG:boto:path=/ibtljn-48/?prefix=obj1 2023-11-02T18:19:06.865 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj1 2023-11-02T18:19:06.865 DEBUG:boto:Method: GET 2023-11-02T18:19:06.866 DEBUG:boto:Path: /ibtljn-48/?prefix=obj1 2023-11-02T18:19:06.866 DEBUG:boto:Data: 2023-11-02T18:19:06.867 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.867 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.868 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.868 DEBUG:boto:Params: {} 2023-11-02T18:19:06.868 DEBUG:boto:Token: None 2023-11-02T18:19:06.869 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/ 2023-11-02T18:19:06.869 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2eL2Tg+SE72guert5ac08LIugIg= 2023-11-02T18:19:06.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2eL2Tg+SE72guert5ac08LIugIg=', 'Content-Length': '0'} 2023-11-02T18:19:06.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019cfd948d245863a-006543e81a-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.875 DEBUG:boto:b'ibtljn-48obj11000falseobj12023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:06.876 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:06.876 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:06.877 DEBUG:boto:path=/ibtljn-48/?prefix=obj1 2023-11-02T18:19:06.877 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj1 2023-11-02T18:19:06.878 DEBUG:boto:Method: GET 2023-11-02T18:19:06.878 DEBUG:boto:Path: /ibtljn-48/?prefix=obj1 2023-11-02T18:19:06.879 DEBUG:boto:Data: 2023-11-02T18:19:06.879 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.879 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.880 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.880 DEBUG:boto:Params: {} 2023-11-02T18:19:06.881 DEBUG:boto:Token: None 2023-11-02T18:19:06.881 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/ 2023-11-02T18:19:06.882 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2eL2Tg+SE72guert5ac08LIugIg= 2023-11-02T18:19:06.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2eL2Tg+SE72guert5ac08LIugIg=', 'Content-Length': '0'} 2023-11-02T18:19:06.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c70dcb479918f17-006543e81a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.934 DEBUG:boto:b'ibtljn-48obj11000falseobj12023-11-02T18:16:45.823Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:06.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:19:06.936 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:19:06.936 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:19:06.936 DEBUG:boto:path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.937 DEBUG:boto:auth_path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.937 DEBUG:boto:Method: GET 2023-11-02T18:19:06.938 DEBUG:boto:Path: /ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.938 DEBUG:boto:Data: 2023-11-02T18:19:06.939 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.939 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.939 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.940 DEBUG:boto:Params: {} 2023-11-02T18:19:06.940 DEBUG:boto:Token: None 2023-11-02T18:19:06.941 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.941 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bi7YxphLKqbiI35Mnjnq0cS6kWQ= 2023-11-02T18:19:06.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bi7YxphLKqbiI35Mnjnq0cS6kWQ=', 'Content-Length': '0'} 2023-11-02T18:19:06.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee4ccf5a3052c495-006543e81a-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.946 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:19:06.947 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:19:06.947 DEBUG:boto:path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.947 DEBUG:boto:auth_path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.948 DEBUG:boto:Method: GET 2023-11-02T18:19:06.948 DEBUG:boto:Path: /ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.949 DEBUG:boto:Data: 2023-11-02T18:19:06.949 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.950 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.951 DEBUG:boto:Params: {} 2023-11-02T18:19:06.951 DEBUG:boto:Token: None 2023-11-02T18:19:06.952 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.952 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bi7YxphLKqbiI35Mnjnq0cS6kWQ= 2023-11-02T18:19:06.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bi7YxphLKqbiI35Mnjnq0cS6kWQ=', 'Content-Length': '0'} 2023-11-02T18:19:06.956 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9610bab7f099189-006543e81a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.956 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:19:06.957 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:19:06.957 DEBUG:boto:path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.957 DEBUG:boto:auth_path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.958 DEBUG:boto:Method: HEAD 2023-11-02T18:19:06.958 DEBUG:boto:Path: /ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.959 DEBUG:boto:Data: 2023-11-02T18:19:06.959 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.960 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.960 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.960 DEBUG:boto:Params: {} 2023-11-02T18:19:06.961 DEBUG:boto:Token: None 2023-11-02T18:19:06.961 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.962 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oohUPS0SuG7ZvVOsKwnt+26qf1s= 2023-11-02T18:19:06.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oohUPS0SuG7ZvVOsKwnt+26qf1s=', 'Content-Length': '0'} 2023-11-02T18:19:06.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb0c9819837cd784-006543e81a-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.966 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:19:06.966 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:19:06.967 DEBUG:boto:path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.967 DEBUG:boto:auth_path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.967 DEBUG:boto:Method: HEAD 2023-11-02T18:19:06.968 DEBUG:boto:Path: /ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.968 DEBUG:boto:Data: 2023-11-02T18:19:06.969 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.969 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.969 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.970 DEBUG:boto:Params: {} 2023-11-02T18:19:06.970 DEBUG:boto:Token: None 2023-11-02T18:19:06.971 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.971 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oohUPS0SuG7ZvVOsKwnt+26qf1s= 2023-11-02T18:19:06.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oohUPS0SuG7ZvVOsKwnt+26qf1s=', 'Content-Length': '0'} 2023-11-02T18:19:06.975 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1142d18e2eb44fe-006543e81a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:19:06.976 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:19:06.976 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:19:06.976 DEBUG:boto:path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.977 DEBUG:boto:auth_path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.977 DEBUG:boto:Method: GET 2023-11-02T18:19:06.978 DEBUG:boto:Path: /ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.978 DEBUG:boto:Data: 2023-11-02T18:19:06.978 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.979 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:06.979 DEBUG:boto:Port: 8001 2023-11-02T18:19:06.980 DEBUG:boto:Params: {} 2023-11-02T18:19:06.980 DEBUG:boto:Token: None 2023-11-02T18:19:06.981 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.981 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bi7YxphLKqbiI35Mnjnq0cS6kWQ= 2023-11-02T18:19:06.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bi7YxphLKqbiI35Mnjnq0cS6kWQ=', 'Content-Length': '0'} 2023-11-02T18:19:06.984 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2e35de98ecb6d45-006543e81a-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:06.985 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:19:06.985 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:19:06.986 DEBUG:boto:path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.986 DEBUG:boto:auth_path=/ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.987 DEBUG:boto:Method: GET 2023-11-02T18:19:06.987 DEBUG:boto:Path: /ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.988 DEBUG:boto:Data: 2023-11-02T18:19:06.988 DEBUG:boto:Headers: {} 2023-11-02T18:19:06.988 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:06.989 DEBUG:boto:Port: 8000 2023-11-02T18:19:06.989 DEBUG:boto:Params: {} 2023-11-02T18:19:06.990 DEBUG:boto:Token: None 2023-11-02T18:19:06.990 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:06 GMT /ibtljn-48/obj2?versionId=null 2023-11-02T18:19:06.991 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bi7YxphLKqbiI35Mnjnq0cS6kWQ= 2023-11-02T18:19:06.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bi7YxphLKqbiI35Mnjnq0cS6kWQ=', 'Content-Length': '0'} 2023-11-02T18:19:06.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090d62564f30bde4d-006543e81a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.034 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.035 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.035 DEBUG:boto:path=/ibtljn-48/?prefix=obj2 2023-11-02T18:19:07.036 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj2 2023-11-02T18:19:07.036 DEBUG:boto:Method: GET 2023-11-02T18:19:07.036 DEBUG:boto:Path: /ibtljn-48/?prefix=obj2 2023-11-02T18:19:07.037 DEBUG:boto:Data: 2023-11-02T18:19:07.037 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.038 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.038 DEBUG:boto:Params: {} 2023-11-02T18:19:07.039 DEBUG:boto:Token: None 2023-11-02T18:19:07.039 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.040 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bae0de7d16e7d93-006543e81b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.045 DEBUG:boto:b'ibtljn-48obj21000falseobj22023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.046 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.046 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.047 DEBUG:boto:path=/ibtljn-48/?prefix=obj2 2023-11-02T18:19:07.047 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj2 2023-11-02T18:19:07.047 DEBUG:boto:Method: GET 2023-11-02T18:19:07.048 DEBUG:boto:Path: /ibtljn-48/?prefix=obj2 2023-11-02T18:19:07.048 DEBUG:boto:Data: 2023-11-02T18:19:07.049 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.049 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.050 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.050 DEBUG:boto:Params: {} 2023-11-02T18:19:07.051 DEBUG:boto:Token: None 2023-11-02T18:19:07.051 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5670a638a1da8b4-006543e81b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.057 DEBUG:boto:b'ibtljn-48obj21000falseobj22023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.057 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:19:07.058 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:19:07.058 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.059 DEBUG:boto:Path: /ibtljn-48/obj2 2023-11-02T18:19:07.059 DEBUG:boto:Data: 2023-11-02T18:19:07.059 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.060 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.060 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.061 DEBUG:boto:Params: {} 2023-11-02T18:19:07.061 DEBUG:boto:Token: None 2023-11-02T18:19:07.062 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj2 2023-11-02T18:19:07.062 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FvKFHUtz7UJXiOXCjwCwijqQFwQ= 2023-11-02T18:19:07.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FvKFHUtz7UJXiOXCjwCwijqQFwQ=', 'Content-Length': '0'} 2023-11-02T18:19:07.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c490af84dbc1d0f-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.066 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:19:07.067 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:19:07.067 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.067 DEBUG:boto:Path: /ibtljn-48/obj2 2023-11-02T18:19:07.068 DEBUG:boto:Data: 2023-11-02T18:19:07.068 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.069 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.069 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.069 DEBUG:boto:Params: {} 2023-11-02T18:19:07.070 DEBUG:boto:Token: None 2023-11-02T18:19:07.070 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj2 2023-11-02T18:19:07.071 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FvKFHUtz7UJXiOXCjwCwijqQFwQ= 2023-11-02T18:19:07.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FvKFHUtz7UJXiOXCjwCwijqQFwQ=', 'Content-Length': '0'} 2023-11-02T18:19:07.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006eb26c18cce5c404-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:19:07.075 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:19:07.075 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:19:07.076 DEBUG:boto:Method: GET 2023-11-02T18:19:07.076 DEBUG:boto:Path: /ibtljn-48/obj2 2023-11-02T18:19:07.076 DEBUG:boto:Data: 2023-11-02T18:19:07.077 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.077 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.078 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.078 DEBUG:boto:Params: {} 2023-11-02T18:19:07.079 DEBUG:boto:Token: None 2023-11-02T18:19:07.079 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj2 2023-11-02T18:19:07.080 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HulEtZSDINESVr2Zxl5INk1OUIw= 2023-11-02T18:19:07.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HulEtZSDINESVr2Zxl5INk1OUIw=', 'Content-Length': '0'} 2023-11-02T18:19:07.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f481a31704155bb-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.084 DEBUG:boto:path=/ibtljn-48/obj2 2023-11-02T18:19:07.084 DEBUG:boto:auth_path=/ibtljn-48/obj2 2023-11-02T18:19:07.085 DEBUG:boto:Method: GET 2023-11-02T18:19:07.085 DEBUG:boto:Path: /ibtljn-48/obj2 2023-11-02T18:19:07.085 DEBUG:boto:Data: 2023-11-02T18:19:07.086 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.086 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.087 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.087 DEBUG:boto:Params: {} 2023-11-02T18:19:07.087 DEBUG:boto:Token: None 2023-11-02T18:19:07.088 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj2 2023-11-02T18:19:07.088 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HulEtZSDINESVr2Zxl5INk1OUIw= 2023-11-02T18:19:07.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HulEtZSDINESVr2Zxl5INk1OUIw=', 'Content-Length': '0'} 2023-11-02T18:19:07.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000042ec81525e8db02-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.093 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.093 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.093 DEBUG:boto:path=/ibtljn-48/?prefix=obj2 2023-11-02T18:19:07.094 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj2 2023-11-02T18:19:07.094 DEBUG:boto:Method: GET 2023-11-02T18:19:07.095 DEBUG:boto:Path: /ibtljn-48/?prefix=obj2 2023-11-02T18:19:07.095 DEBUG:boto:Data: 2023-11-02T18:19:07.096 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.096 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.097 DEBUG:boto:Params: {} 2023-11-02T18:19:07.097 DEBUG:boto:Token: None 2023-11-02T18:19:07.098 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a664e0cb3833938b-006543e81b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.146 DEBUG:boto:b'ibtljn-48obj21000falseobj22023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.147 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.148 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.148 DEBUG:boto:path=/ibtljn-48/?prefix=obj2 2023-11-02T18:19:07.149 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj2 2023-11-02T18:19:07.149 DEBUG:boto:Method: GET 2023-11-02T18:19:07.150 DEBUG:boto:Path: /ibtljn-48/?prefix=obj2 2023-11-02T18:19:07.150 DEBUG:boto:Data: 2023-11-02T18:19:07.151 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.151 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.151 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.152 DEBUG:boto:Params: {} 2023-11-02T18:19:07.152 DEBUG:boto:Token: None 2023-11-02T18:19:07.153 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.153 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fced068d66e222f9-006543e81b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.159 DEBUG:boto:b'ibtljn-48obj21000falseobj22023-11-02T18:16:45.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:19:07.160 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:19:07.160 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:19:07.161 DEBUG:boto:path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.161 DEBUG:boto:auth_path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.162 DEBUG:boto:Method: GET 2023-11-02T18:19:07.162 DEBUG:boto:Path: /ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.163 DEBUG:boto:Data: 2023-11-02T18:19:07.163 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.164 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.164 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.164 DEBUG:boto:Params: {} 2023-11-02T18:19:07.165 DEBUG:boto:Token: None 2023-11-02T18:19:07.165 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.166 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bXlVFoqfmWQWcid3FuLiNVVwT/U= 2023-11-02T18:19:07.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bXlVFoqfmWQWcid3FuLiNVVwT/U=', 'Content-Length': '0'} 2023-11-02T18:19:07.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084f7d18e3fe090fa-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.171 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:19:07.171 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:19:07.172 DEBUG:boto:path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.172 DEBUG:boto:auth_path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.173 DEBUG:boto:Method: GET 2023-11-02T18:19:07.173 DEBUG:boto:Path: /ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.173 DEBUG:boto:Data: 2023-11-02T18:19:07.174 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.174 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.175 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.175 DEBUG:boto:Params: {} 2023-11-02T18:19:07.176 DEBUG:boto:Token: None 2023-11-02T18:19:07.176 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bXlVFoqfmWQWcid3FuLiNVVwT/U= 2023-11-02T18:19:07.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bXlVFoqfmWQWcid3FuLiNVVwT/U=', 'Content-Length': '0'} 2023-11-02T18:19:07.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd60cd2066f6500c-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.226 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:19:07.227 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:19:07.229 DEBUG:boto:path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.230 DEBUG:boto:auth_path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.230 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.231 DEBUG:boto:Path: /ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.231 DEBUG:boto:Data: 2023-11-02T18:19:07.231 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.232 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.232 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.233 DEBUG:boto:Params: {} 2023-11-02T18:19:07.233 DEBUG:boto:Token: None 2023-11-02T18:19:07.234 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.234 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UchGReh71D82TUl6CPRwS3et5hw= 2023-11-02T18:19:07.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UchGReh71D82TUl6CPRwS3et5hw=', 'Content-Length': '0'} 2023-11-02T18:19:07.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081ce2b8e05e491bc-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.240 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:19:07.240 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:19:07.241 DEBUG:boto:path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.241 DEBUG:boto:auth_path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.241 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.242 DEBUG:boto:Path: /ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.242 DEBUG:boto:Data: 2023-11-02T18:19:07.243 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.243 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.244 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.244 DEBUG:boto:Params: {} 2023-11-02T18:19:07.245 DEBUG:boto:Token: None 2023-11-02T18:19:07.245 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.246 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UchGReh71D82TUl6CPRwS3et5hw= 2023-11-02T18:19:07.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UchGReh71D82TUl6CPRwS3et5hw=', 'Content-Length': '0'} 2023-11-02T18:19:07.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a16c57ab1a362f7-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:19:07.250 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:19:07.250 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:19:07.251 DEBUG:boto:path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.251 DEBUG:boto:auth_path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.252 DEBUG:boto:Method: GET 2023-11-02T18:19:07.252 DEBUG:boto:Path: /ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.253 DEBUG:boto:Data: 2023-11-02T18:19:07.253 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.253 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.254 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.254 DEBUG:boto:Params: {} 2023-11-02T18:19:07.255 DEBUG:boto:Token: None 2023-11-02T18:19:07.255 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.256 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bXlVFoqfmWQWcid3FuLiNVVwT/U= 2023-11-02T18:19:07.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bXlVFoqfmWQWcid3FuLiNVVwT/U=', 'Content-Length': '0'} 2023-11-02T18:19:07.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000696c3163568da3fe-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.260 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:19:07.261 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:19:07.261 DEBUG:boto:path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.262 DEBUG:boto:auth_path=/ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.262 DEBUG:boto:Method: GET 2023-11-02T18:19:07.262 DEBUG:boto:Path: /ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.263 DEBUG:boto:Data: 2023-11-02T18:19:07.263 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.264 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.264 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.265 DEBUG:boto:Params: {} 2023-11-02T18:19:07.265 DEBUG:boto:Token: None 2023-11-02T18:19:07.266 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj3?versionId=null 2023-11-02T18:19:07.266 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bXlVFoqfmWQWcid3FuLiNVVwT/U= 2023-11-02T18:19:07.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bXlVFoqfmWQWcid3FuLiNVVwT/U=', 'Content-Length': '0'} 2023-11-02T18:19:07.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e87f70b1eb856a12-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.270 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.271 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.271 DEBUG:boto:path=/ibtljn-48/?prefix=obj3 2023-11-02T18:19:07.272 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj3 2023-11-02T18:19:07.272 DEBUG:boto:Method: GET 2023-11-02T18:19:07.273 DEBUG:boto:Path: /ibtljn-48/?prefix=obj3 2023-11-02T18:19:07.273 DEBUG:boto:Data: 2023-11-02T18:19:07.273 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.274 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.274 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.275 DEBUG:boto:Params: {} 2023-11-02T18:19:07.275 DEBUG:boto:Token: None 2023-11-02T18:19:07.276 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.276 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa9fb5a7ac234dc6-006543e81b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.326 DEBUG:boto:b'ibtljn-48obj31000falseobj32023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.327 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.328 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.328 DEBUG:boto:path=/ibtljn-48/?prefix=obj3 2023-11-02T18:19:07.329 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj3 2023-11-02T18:19:07.329 DEBUG:boto:Method: GET 2023-11-02T18:19:07.330 DEBUG:boto:Path: /ibtljn-48/?prefix=obj3 2023-11-02T18:19:07.330 DEBUG:boto:Data: 2023-11-02T18:19:07.331 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.331 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.331 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.332 DEBUG:boto:Params: {} 2023-11-02T18:19:07.332 DEBUG:boto:Token: None 2023-11-02T18:19:07.333 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.333 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e61ae8fe49051195-006543e81b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.340 DEBUG:boto:b'ibtljn-48obj31000falseobj32023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.341 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:19:07.341 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:19:07.341 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.342 DEBUG:boto:Path: /ibtljn-48/obj3 2023-11-02T18:19:07.342 DEBUG:boto:Data: 2023-11-02T18:19:07.343 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.343 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.344 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.344 DEBUG:boto:Params: {} 2023-11-02T18:19:07.344 DEBUG:boto:Token: None 2023-11-02T18:19:07.345 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj3 2023-11-02T18:19:07.345 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w0GZkFcrAdvZjrDClxMOmp5gaZI= 2023-11-02T18:19:07.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w0GZkFcrAdvZjrDClxMOmp5gaZI=', 'Content-Length': '0'} 2023-11-02T18:19:07.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ab91158ad6154da-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.349 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:19:07.350 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:19:07.350 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.351 DEBUG:boto:Path: /ibtljn-48/obj3 2023-11-02T18:19:07.351 DEBUG:boto:Data: 2023-11-02T18:19:07.351 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.352 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.352 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.353 DEBUG:boto:Params: {} 2023-11-02T18:19:07.353 DEBUG:boto:Token: None 2023-11-02T18:19:07.354 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj3 2023-11-02T18:19:07.354 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w0GZkFcrAdvZjrDClxMOmp5gaZI= 2023-11-02T18:19:07.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w0GZkFcrAdvZjrDClxMOmp5gaZI=', 'Content-Length': '0'} 2023-11-02T18:19:07.357 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1c7cc6ea1616d94-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:19:07.358 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:19:07.359 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:19:07.359 DEBUG:boto:Method: GET 2023-11-02T18:19:07.360 DEBUG:boto:Path: /ibtljn-48/obj3 2023-11-02T18:19:07.360 DEBUG:boto:Data: 2023-11-02T18:19:07.360 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.361 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.361 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.362 DEBUG:boto:Params: {} 2023-11-02T18:19:07.362 DEBUG:boto:Token: None 2023-11-02T18:19:07.363 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj3 2023-11-02T18:19:07.363 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fe/7BUIPVuVw080/p9Hdnc3x3iw= 2023-11-02T18:19:07.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fe/7BUIPVuVw080/p9Hdnc3x3iw=', 'Content-Length': '0'} 2023-11-02T18:19:07.367 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ebe7d19dd14d8b3-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.367 DEBUG:boto:path=/ibtljn-48/obj3 2023-11-02T18:19:07.368 DEBUG:boto:auth_path=/ibtljn-48/obj3 2023-11-02T18:19:07.368 DEBUG:boto:Method: GET 2023-11-02T18:19:07.369 DEBUG:boto:Path: /ibtljn-48/obj3 2023-11-02T18:19:07.369 DEBUG:boto:Data: 2023-11-02T18:19:07.370 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.370 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.371 DEBUG:boto:Params: {} 2023-11-02T18:19:07.371 DEBUG:boto:Token: None 2023-11-02T18:19:07.372 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj3 2023-11-02T18:19:07.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fe/7BUIPVuVw080/p9Hdnc3x3iw= 2023-11-02T18:19:07.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fe/7BUIPVuVw080/p9Hdnc3x3iw=', 'Content-Length': '0'} 2023-11-02T18:19:07.376 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000287753f742765733-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.376 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.377 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.377 DEBUG:boto:path=/ibtljn-48/?prefix=obj3 2023-11-02T18:19:07.378 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj3 2023-11-02T18:19:07.378 DEBUG:boto:Method: GET 2023-11-02T18:19:07.379 DEBUG:boto:Path: /ibtljn-48/?prefix=obj3 2023-11-02T18:19:07.379 DEBUG:boto:Data: 2023-11-02T18:19:07.380 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.380 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.381 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.381 DEBUG:boto:Params: {} 2023-11-02T18:19:07.382 DEBUG:boto:Token: None 2023-11-02T18:19:07.382 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.383 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb2b7800d8fc0077-006543e81b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.388 DEBUG:boto:b'ibtljn-48obj31000falseobj32023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.389 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.390 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.390 DEBUG:boto:path=/ibtljn-48/?prefix=obj3 2023-11-02T18:19:07.391 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj3 2023-11-02T18:19:07.391 DEBUG:boto:Method: GET 2023-11-02T18:19:07.391 DEBUG:boto:Path: /ibtljn-48/?prefix=obj3 2023-11-02T18:19:07.392 DEBUG:boto:Data: 2023-11-02T18:19:07.392 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.393 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.393 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.394 DEBUG:boto:Params: {} 2023-11-02T18:19:07.394 DEBUG:boto:Token: None 2023-11-02T18:19:07.395 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.395 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000624cef9048d59cb2-006543e81b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.442 DEBUG:boto:b'ibtljn-48obj31000falseobj32023-11-02T18:16:45.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:19:07.444 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:19:07.444 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:19:07.445 DEBUG:boto:path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.445 DEBUG:boto:auth_path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.445 DEBUG:boto:Method: GET 2023-11-02T18:19:07.446 DEBUG:boto:Path: /ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.446 DEBUG:boto:Data: 2023-11-02T18:19:07.447 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.448 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.448 DEBUG:boto:Params: {} 2023-11-02T18:19:07.449 DEBUG:boto:Token: None 2023-11-02T18:19:07.449 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.450 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:K0ZI5LtTNPT4rlzhQrFEAZ1kfFo= 2023-11-02T18:19:07.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:K0ZI5LtTNPT4rlzhQrFEAZ1kfFo=', 'Content-Length': '0'} 2023-11-02T18:19:07.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da0e7cc4f20c501a-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.454 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:19:07.455 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:19:07.455 DEBUG:boto:path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.455 DEBUG:boto:auth_path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.456 DEBUG:boto:Method: GET 2023-11-02T18:19:07.456 DEBUG:boto:Path: /ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.457 DEBUG:boto:Data: 2023-11-02T18:19:07.457 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.458 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.458 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.458 DEBUG:boto:Params: {} 2023-11-02T18:19:07.459 DEBUG:boto:Token: None 2023-11-02T18:19:07.459 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.460 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:K0ZI5LtTNPT4rlzhQrFEAZ1kfFo= 2023-11-02T18:19:07.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:K0ZI5LtTNPT4rlzhQrFEAZ1kfFo=', 'Content-Length': '0'} 2023-11-02T18:19:07.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e53af99070c4d289-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.464 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:19:07.464 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:19:07.465 DEBUG:boto:path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.465 DEBUG:boto:auth_path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.466 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.466 DEBUG:boto:Path: /ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.466 DEBUG:boto:Data: 2023-11-02T18:19:07.467 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.467 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.468 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.468 DEBUG:boto:Params: {} 2023-11-02T18:19:07.469 DEBUG:boto:Token: None 2023-11-02T18:19:07.469 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.470 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e/pKcqwXpyl7/v1LgVdWWfLXtjE= 2023-11-02T18:19:07.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e/pKcqwXpyl7/v1LgVdWWfLXtjE=', 'Content-Length': '0'} 2023-11-02T18:19:07.473 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002340b1ed4bc05cf6-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.473 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:19:07.474 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:19:07.474 DEBUG:boto:path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.475 DEBUG:boto:auth_path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.475 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.476 DEBUG:boto:Path: /ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.476 DEBUG:boto:Data: 2023-11-02T18:19:07.476 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.477 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.477 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.478 DEBUG:boto:Params: {} 2023-11-02T18:19:07.478 DEBUG:boto:Token: None 2023-11-02T18:19:07.479 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.479 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e/pKcqwXpyl7/v1LgVdWWfLXtjE= 2023-11-02T18:19:07.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e/pKcqwXpyl7/v1LgVdWWfLXtjE=', 'Content-Length': '0'} 2023-11-02T18:19:07.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000eacbfed24261235-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:19:07.484 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:19:07.484 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:19:07.484 DEBUG:boto:path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.485 DEBUG:boto:auth_path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.485 DEBUG:boto:Method: GET 2023-11-02T18:19:07.485 DEBUG:boto:Path: /ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.486 DEBUG:boto:Data: 2023-11-02T18:19:07.486 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.487 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.487 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.488 DEBUG:boto:Params: {} 2023-11-02T18:19:07.488 DEBUG:boto:Token: None 2023-11-02T18:19:07.489 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.489 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:K0ZI5LtTNPT4rlzhQrFEAZ1kfFo= 2023-11-02T18:19:07.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:K0ZI5LtTNPT4rlzhQrFEAZ1kfFo=', 'Content-Length': '0'} 2023-11-02T18:19:07.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061f121f2341452a8-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.493 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:19:07.493 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:19:07.494 DEBUG:boto:path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.494 DEBUG:boto:auth_path=/ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.495 DEBUG:boto:Method: GET 2023-11-02T18:19:07.495 DEBUG:boto:Path: /ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.496 DEBUG:boto:Data: 2023-11-02T18:19:07.496 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.497 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.497 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.497 DEBUG:boto:Params: {} 2023-11-02T18:19:07.498 DEBUG:boto:Token: None 2023-11-02T18:19:07.498 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj4?versionId=null 2023-11-02T18:19:07.499 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:K0ZI5LtTNPT4rlzhQrFEAZ1kfFo= 2023-11-02T18:19:07.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:K0ZI5LtTNPT4rlzhQrFEAZ1kfFo=', 'Content-Length': '0'} 2023-11-02T18:19:07.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004368005bb44038af-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.546 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.547 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.547 DEBUG:boto:path=/ibtljn-48/?prefix=obj4 2023-11-02T18:19:07.548 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj4 2023-11-02T18:19:07.548 DEBUG:boto:Method: GET 2023-11-02T18:19:07.549 DEBUG:boto:Path: /ibtljn-48/?prefix=obj4 2023-11-02T18:19:07.549 DEBUG:boto:Data: 2023-11-02T18:19:07.550 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.550 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.550 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.551 DEBUG:boto:Params: {} 2023-11-02T18:19:07.551 DEBUG:boto:Token: None 2023-11-02T18:19:07.552 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.552 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000858432553dfa6010-006543e81b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.557 DEBUG:boto:b'ibtljn-48obj41000falseobj42023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.557 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.558 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.558 DEBUG:boto:path=/ibtljn-48/?prefix=obj4 2023-11-02T18:19:07.559 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj4 2023-11-02T18:19:07.559 DEBUG:boto:Method: GET 2023-11-02T18:19:07.560 DEBUG:boto:Path: /ibtljn-48/?prefix=obj4 2023-11-02T18:19:07.560 DEBUG:boto:Data: 2023-11-02T18:19:07.560 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.561 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.561 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.562 DEBUG:boto:Params: {} 2023-11-02T18:19:07.562 DEBUG:boto:Token: None 2023-11-02T18:19:07.563 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.563 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5eac241374779d1-006543e81b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.568 DEBUG:boto:b'ibtljn-48obj41000falseobj42023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.569 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:19:07.569 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:19:07.570 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.570 DEBUG:boto:Path: /ibtljn-48/obj4 2023-11-02T18:19:07.571 DEBUG:boto:Data: 2023-11-02T18:19:07.571 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.572 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.572 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.572 DEBUG:boto:Params: {} 2023-11-02T18:19:07.573 DEBUG:boto:Token: None 2023-11-02T18:19:07.573 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj4 2023-11-02T18:19:07.574 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dhs85SPuRzXeuShYseqGKpzhnNY= 2023-11-02T18:19:07.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dhs85SPuRzXeuShYseqGKpzhnNY=', 'Content-Length': '0'} 2023-11-02T18:19:07.576 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000475080584ef3712e-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.577 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:19:07.577 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:19:07.578 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.578 DEBUG:boto:Path: /ibtljn-48/obj4 2023-11-02T18:19:07.579 DEBUG:boto:Data: 2023-11-02T18:19:07.579 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.580 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.580 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.581 DEBUG:boto:Params: {} 2023-11-02T18:19:07.581 DEBUG:boto:Token: None 2023-11-02T18:19:07.581 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj4 2023-11-02T18:19:07.582 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dhs85SPuRzXeuShYseqGKpzhnNY= 2023-11-02T18:19:07.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dhs85SPuRzXeuShYseqGKpzhnNY=', 'Content-Length': '0'} 2023-11-02T18:19:07.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebfbb838860bee06-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:19:07.588 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:19:07.588 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:19:07.589 DEBUG:boto:Method: GET 2023-11-02T18:19:07.589 DEBUG:boto:Path: /ibtljn-48/obj4 2023-11-02T18:19:07.589 DEBUG:boto:Data: 2023-11-02T18:19:07.590 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.591 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.591 DEBUG:boto:Params: {} 2023-11-02T18:19:07.592 DEBUG:boto:Token: None 2023-11-02T18:19:07.592 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj4 2023-11-02T18:19:07.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gX9lYfBIVlea1ZA4p3yTYTTnjRM= 2023-11-02T18:19:07.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gX9lYfBIVlea1ZA4p3yTYTTnjRM=', 'Content-Length': '0'} 2023-11-02T18:19:07.596 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c344f42cbae3ee3-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.638 DEBUG:boto:path=/ibtljn-48/obj4 2023-11-02T18:19:07.639 DEBUG:boto:auth_path=/ibtljn-48/obj4 2023-11-02T18:19:07.639 DEBUG:boto:Method: GET 2023-11-02T18:19:07.640 DEBUG:boto:Path: /ibtljn-48/obj4 2023-11-02T18:19:07.641 DEBUG:boto:Data: 2023-11-02T18:19:07.641 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.641 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.642 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.643 DEBUG:boto:Params: {} 2023-11-02T18:19:07.643 DEBUG:boto:Token: None 2023-11-02T18:19:07.644 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj4 2023-11-02T18:19:07.644 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gX9lYfBIVlea1ZA4p3yTYTTnjRM= 2023-11-02T18:19:07.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gX9lYfBIVlea1ZA4p3yTYTTnjRM=', 'Content-Length': '0'} 2023-11-02T18:19:07.648 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ad9b32189993f8e-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.649 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.649 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.650 DEBUG:boto:path=/ibtljn-48/?prefix=obj4 2023-11-02T18:19:07.651 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj4 2023-11-02T18:19:07.651 DEBUG:boto:Method: GET 2023-11-02T18:19:07.652 DEBUG:boto:Path: /ibtljn-48/?prefix=obj4 2023-11-02T18:19:07.652 DEBUG:boto:Data: 2023-11-02T18:19:07.653 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.653 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.654 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.654 DEBUG:boto:Params: {} 2023-11-02T18:19:07.655 DEBUG:boto:Token: None 2023-11-02T18:19:07.655 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009af320fa342a2264-006543e81b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.661 DEBUG:boto:b'ibtljn-48obj41000falseobj42023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.662 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.663 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.663 DEBUG:boto:path=/ibtljn-48/?prefix=obj4 2023-11-02T18:19:07.664 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj4 2023-11-02T18:19:07.664 DEBUG:boto:Method: GET 2023-11-02T18:19:07.664 DEBUG:boto:Path: /ibtljn-48/?prefix=obj4 2023-11-02T18:19:07.665 DEBUG:boto:Data: 2023-11-02T18:19:07.665 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.666 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.666 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.667 DEBUG:boto:Params: {} 2023-11-02T18:19:07.667 DEBUG:boto:Token: None 2023-11-02T18:19:07.667 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.668 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1a2ae55383f3aad-006543e81b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.718 DEBUG:boto:b'ibtljn-48obj41000falseobj42023-11-02T18:16:45.951Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-11-02T18:19:07.720 DEBUG:boto:path=/ibtljn-48/obj5 2023-11-02T18:19:07.720 DEBUG:boto:auth_path=/ibtljn-48/obj5 2023-11-02T18:19:07.721 DEBUG:boto:path=/ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.722 DEBUG:boto:auth_path=/ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.723 DEBUG:boto:Method: GET 2023-11-02T18:19:07.723 DEBUG:boto:Path: /ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.724 DEBUG:boto:Data: 2023-11-02T18:19:07.724 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.725 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.726 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.726 DEBUG:boto:Params: {} 2023-11-02T18:19:07.727 DEBUG:boto:Token: None 2023-11-02T18:19:07.727 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.728 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wuRnss3zqov0ljWmpaOY+cHhPok= 2023-11-02T18:19:07.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wuRnss3zqov0ljWmpaOY+cHhPok=', 'Content-Length': '0'} 2023-11-02T18:19:07.732 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4d19c60c7a041d3-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.732 DEBUG:boto:path=/ibtljn-48/obj5 2023-11-02T18:19:07.733 DEBUG:boto:auth_path=/ibtljn-48/obj5 2023-11-02T18:19:07.733 DEBUG:boto:path=/ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.734 DEBUG:boto:auth_path=/ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.734 DEBUG:boto:Method: GET 2023-11-02T18:19:07.734 DEBUG:boto:Path: /ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.735 DEBUG:boto:Data: 2023-11-02T18:19:07.735 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.736 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.736 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.736 DEBUG:boto:Params: {} 2023-11-02T18:19:07.737 DEBUG:boto:Token: None 2023-11-02T18:19:07.737 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.738 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wuRnss3zqov0ljWmpaOY+cHhPok= 2023-11-02T18:19:07.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wuRnss3zqov0ljWmpaOY+cHhPok=', 'Content-Length': '0'} 2023-11-02T18:19:07.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc4bcba1371f4f84-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.742 DEBUG:boto:path=/ibtljn-48/obj5 2023-11-02T18:19:07.743 DEBUG:boto:auth_path=/ibtljn-48/obj5 2023-11-02T18:19:07.743 DEBUG:boto:path=/ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.744 DEBUG:boto:auth_path=/ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.744 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.744 DEBUG:boto:Path: /ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.745 DEBUG:boto:Data: 2023-11-02T18:19:07.745 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.746 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.746 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.747 DEBUG:boto:Params: {} 2023-11-02T18:19:07.747 DEBUG:boto:Token: None 2023-11-02T18:19:07.748 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.748 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RfJvb3e87PaLZ7dMtfuZu3aYlXk= 2023-11-02T18:19:07.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RfJvb3e87PaLZ7dMtfuZu3aYlXk=', 'Content-Length': '0'} 2023-11-02T18:19:07.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007774ababbb94667e-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.752 DEBUG:boto:path=/ibtljn-48/obj5 2023-11-02T18:19:07.753 DEBUG:boto:auth_path=/ibtljn-48/obj5 2023-11-02T18:19:07.753 DEBUG:boto:path=/ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.753 DEBUG:boto:auth_path=/ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.754 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.754 DEBUG:boto:Path: /ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.755 DEBUG:boto:Data: 2023-11-02T18:19:07.755 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.755 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.756 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.756 DEBUG:boto:Params: {} 2023-11-02T18:19:07.757 DEBUG:boto:Token: None 2023-11-02T18:19:07.758 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.758 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RfJvb3e87PaLZ7dMtfuZu3aYlXk= 2023-11-02T18:19:07.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RfJvb3e87PaLZ7dMtfuZu3aYlXk=', 'Content-Length': '0'} 2023-11-02T18:19:07.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cafbe41b928d6325-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-11-02T18:19:07.762 DEBUG:boto:path=/ibtljn-48/obj5 2023-11-02T18:19:07.763 DEBUG:boto:auth_path=/ibtljn-48/obj5 2023-11-02T18:19:07.763 DEBUG:boto:path=/ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.764 DEBUG:boto:auth_path=/ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.764 DEBUG:boto:Method: GET 2023-11-02T18:19:07.765 DEBUG:boto:Path: /ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.765 DEBUG:boto:Data: 2023-11-02T18:19:07.765 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.766 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.766 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.767 DEBUG:boto:Params: {} 2023-11-02T18:19:07.767 DEBUG:boto:Token: None 2023-11-02T18:19:07.767 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wuRnss3zqov0ljWmpaOY+cHhPok= 2023-11-02T18:19:07.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wuRnss3zqov0ljWmpaOY+cHhPok=', 'Content-Length': '0'} 2023-11-02T18:19:07.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076357745877ecc15-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.814 DEBUG:boto:path=/ibtljn-48/obj5 2023-11-02T18:19:07.815 DEBUG:boto:auth_path=/ibtljn-48/obj5 2023-11-02T18:19:07.815 DEBUG:boto:path=/ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.816 DEBUG:boto:auth_path=/ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.816 DEBUG:boto:Method: GET 2023-11-02T18:19:07.817 DEBUG:boto:Path: /ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.817 DEBUG:boto:Data: 2023-11-02T18:19:07.817 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.818 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.818 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.819 DEBUG:boto:Params: {} 2023-11-02T18:19:07.819 DEBUG:boto:Token: None 2023-11-02T18:19:07.820 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj5?versionId=null 2023-11-02T18:19:07.820 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wuRnss3zqov0ljWmpaOY+cHhPok= 2023-11-02T18:19:07.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wuRnss3zqov0ljWmpaOY+cHhPok=', 'Content-Length': '0'} 2023-11-02T18:19:07.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d82570afa399329-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.825 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.826 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.826 DEBUG:boto:path=/ibtljn-48/?prefix=obj5 2023-11-02T18:19:07.827 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj5 2023-11-02T18:19:07.827 DEBUG:boto:Method: GET 2023-11-02T18:19:07.828 DEBUG:boto:Path: /ibtljn-48/?prefix=obj5 2023-11-02T18:19:07.829 DEBUG:boto:Data: 2023-11-02T18:19:07.829 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.830 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.831 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.831 DEBUG:boto:Params: {} 2023-11-02T18:19:07.832 DEBUG:boto:Token: None 2023-11-02T18:19:07.832 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.833 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cbb4ce5e67fb4ab-006543e81b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.838 DEBUG:boto:b'ibtljn-48obj51000falseobj52023-11-02T18:18:30.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.839 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.839 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.840 DEBUG:boto:path=/ibtljn-48/?prefix=obj5 2023-11-02T18:19:07.840 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj5 2023-11-02T18:19:07.841 DEBUG:boto:Method: GET 2023-11-02T18:19:07.841 DEBUG:boto:Path: /ibtljn-48/?prefix=obj5 2023-11-02T18:19:07.842 DEBUG:boto:Data: 2023-11-02T18:19:07.842 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.842 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.843 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.843 DEBUG:boto:Params: {} 2023-11-02T18:19:07.844 DEBUG:boto:Token: None 2023-11-02T18:19:07.845 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.845 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8b7ef3fc49c5749-006543e81b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.851 DEBUG:boto:b'ibtljn-48obj51000falseobj52023-11-02T18:18:30.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.852 DEBUG:boto:path=/ibtljn-48/obj5 2023-11-02T18:19:07.852 DEBUG:boto:auth_path=/ibtljn-48/obj5 2023-11-02T18:19:07.853 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.853 DEBUG:boto:Path: /ibtljn-48/obj5 2023-11-02T18:19:07.854 DEBUG:boto:Data: 2023-11-02T18:19:07.854 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.855 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.855 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.856 DEBUG:boto:Params: {} 2023-11-02T18:19:07.856 DEBUG:boto:Token: None 2023-11-02T18:19:07.857 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj5 2023-11-02T18:19:07.858 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BswniWrOmDAEERE4AKASsL2Qy2k= 2023-11-02T18:19:07.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BswniWrOmDAEERE4AKASsL2Qy2k=', 'Content-Length': '0'} 2023-11-02T18:19:07.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc224b07ee529d12-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.863 DEBUG:boto:path=/ibtljn-48/obj5 2023-11-02T18:19:07.863 DEBUG:boto:auth_path=/ibtljn-48/obj5 2023-11-02T18:19:07.864 DEBUG:boto:Method: HEAD 2023-11-02T18:19:07.864 DEBUG:boto:Path: /ibtljn-48/obj5 2023-11-02T18:19:07.864 DEBUG:boto:Data: 2023-11-02T18:19:07.865 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.865 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.866 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.867 DEBUG:boto:Params: {} 2023-11-02T18:19:07.867 DEBUG:boto:Token: None 2023-11-02T18:19:07.868 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj5 2023-11-02T18:19:07.868 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BswniWrOmDAEERE4AKASsL2Qy2k= 2023-11-02T18:19:07.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BswniWrOmDAEERE4AKASsL2Qy2k=', 'Content-Length': '0'} 2023-11-02T18:19:07.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c493ba006af7118-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-11-02T18:19:07.872 DEBUG:boto:path=/ibtljn-48/obj5 2023-11-02T18:19:07.873 DEBUG:boto:auth_path=/ibtljn-48/obj5 2023-11-02T18:19:07.873 DEBUG:boto:Method: GET 2023-11-02T18:19:07.874 DEBUG:boto:Path: /ibtljn-48/obj5 2023-11-02T18:19:07.874 DEBUG:boto:Data: 2023-11-02T18:19:07.875 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.875 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.876 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.876 DEBUG:boto:Params: {} 2023-11-02T18:19:07.877 DEBUG:boto:Token: None 2023-11-02T18:19:07.877 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj5 2023-11-02T18:19:07.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jdmQG2tkZR9r9W0i5zLFY9CdH1E= 2023-11-02T18:19:07.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdmQG2tkZR9r9W0i5zLFY9CdH1E=', 'Content-Length': '0'} 2023-11-02T18:19:07.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3ea68c13560b64b-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.882 DEBUG:boto:path=/ibtljn-48/obj5 2023-11-02T18:19:07.882 DEBUG:boto:auth_path=/ibtljn-48/obj5 2023-11-02T18:19:07.882 DEBUG:boto:Method: GET 2023-11-02T18:19:07.883 DEBUG:boto:Path: /ibtljn-48/obj5 2023-11-02T18:19:07.883 DEBUG:boto:Data: 2023-11-02T18:19:07.884 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.885 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.885 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.886 DEBUG:boto:Params: {} 2023-11-02T18:19:07.886 DEBUG:boto:Token: None 2023-11-02T18:19:07.887 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj5 2023-11-02T18:19:07.887 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jdmQG2tkZR9r9W0i5zLFY9CdH1E= 2023-11-02T18:19:07.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdmQG2tkZR9r9W0i5zLFY9CdH1E=', 'Content-Length': '0'} 2023-11-02T18:19:07.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b414437e6c6803f8-006543e81b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.938 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.939 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.939 DEBUG:boto:path=/ibtljn-48/?prefix=obj5 2023-11-02T18:19:07.940 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj5 2023-11-02T18:19:07.940 DEBUG:boto:Method: GET 2023-11-02T18:19:07.940 DEBUG:boto:Path: /ibtljn-48/?prefix=obj5 2023-11-02T18:19:07.941 DEBUG:boto:Data: 2023-11-02T18:19:07.941 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.942 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.942 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.943 DEBUG:boto:Params: {} 2023-11-02T18:19:07.943 DEBUG:boto:Token: None 2023-11-02T18:19:07.943 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.944 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099ec0940c6f95937-006543e81b-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.950 DEBUG:boto:b'ibtljn-48obj51000falseobj52023-11-02T18:18:30.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.950 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:07.951 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:07.951 DEBUG:boto:path=/ibtljn-48/?prefix=obj5 2023-11-02T18:19:07.952 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj5 2023-11-02T18:19:07.952 DEBUG:boto:Method: GET 2023-11-02T18:19:07.953 DEBUG:boto:Path: /ibtljn-48/?prefix=obj5 2023-11-02T18:19:07.953 DEBUG:boto:Data: 2023-11-02T18:19:07.953 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.954 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:07.954 DEBUG:boto:Port: 8000 2023-11-02T18:19:07.955 DEBUG:boto:Params: {} 2023-11-02T18:19:07.955 DEBUG:boto:Token: None 2023-11-02T18:19:07.955 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/ 2023-11-02T18:19:07.956 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA= 2023-11-02T18:19:07.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGO3CNKW5XimM1rlB69B+7O3lOA=', 'Content-Length': '0'} 2023-11-02T18:19:07.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000240d12fd1c7f9c68-006543e81b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:07.962 DEBUG:boto:b'ibtljn-48obj51000falseobj52023-11-02T18:18:30.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:07.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-11-02T18:19:07.963 DEBUG:boto:path=/ibtljn-48/obj6 2023-11-02T18:19:07.964 DEBUG:boto:auth_path=/ibtljn-48/obj6 2023-11-02T18:19:07.964 DEBUG:boto:path=/ibtljn-48/obj6?versionId=null 2023-11-02T18:19:07.964 DEBUG:boto:auth_path=/ibtljn-48/obj6?versionId=null 2023-11-02T18:19:07.965 DEBUG:boto:Method: GET 2023-11-02T18:19:07.965 DEBUG:boto:Path: /ibtljn-48/obj6?versionId=null 2023-11-02T18:19:07.966 DEBUG:boto:Data: 2023-11-02T18:19:07.966 DEBUG:boto:Headers: {} 2023-11-02T18:19:07.967 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:07.967 DEBUG:boto:Port: 8001 2023-11-02T18:19:07.967 DEBUG:boto:Params: {} 2023-11-02T18:19:07.968 DEBUG:boto:Token: None 2023-11-02T18:19:07.968 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:07 GMT /ibtljn-48/obj6?versionId=null 2023-11-02T18:19:07.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1NuxVuZxPVOTuzBdQLH8oEY+38Y= 2023-11-02T18:19:07.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1NuxVuZxPVOTuzBdQLH8oEY+38Y=', 'Content-Length': '0'} 2023-11-02T18:19:07.972 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce8e78f71d18c0c2-006543e81b-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.014 DEBUG:boto:path=/ibtljn-48/obj6 2023-11-02T18:19:08.015 DEBUG:boto:auth_path=/ibtljn-48/obj6 2023-11-02T18:19:08.015 DEBUG:boto:path=/ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.016 DEBUG:boto:auth_path=/ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.016 DEBUG:boto:Method: GET 2023-11-02T18:19:08.016 DEBUG:boto:Path: /ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.017 DEBUG:boto:Data: 2023-11-02T18:19:08.017 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.018 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.018 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.019 DEBUG:boto:Params: {} 2023-11-02T18:19:08.019 DEBUG:boto:Token: None 2023-11-02T18:19:08.020 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.020 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F57LbNuCZhmMyPflukjG7Qhj4jw= 2023-11-02T18:19:08.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F57LbNuCZhmMyPflukjG7Qhj4jw=', 'Content-Length': '0'} 2023-11-02T18:19:08.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1e5f279978f0749-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.024 DEBUG:boto:path=/ibtljn-48/obj6 2023-11-02T18:19:08.025 DEBUG:boto:auth_path=/ibtljn-48/obj6 2023-11-02T18:19:08.025 DEBUG:boto:path=/ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.025 DEBUG:boto:auth_path=/ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.026 DEBUG:boto:Method: HEAD 2023-11-02T18:19:08.026 DEBUG:boto:Path: /ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.027 DEBUG:boto:Data: 2023-11-02T18:19:08.027 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.028 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.028 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.028 DEBUG:boto:Params: {} 2023-11-02T18:19:08.029 DEBUG:boto:Token: None 2023-11-02T18:19:08.029 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.030 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0XIVnX524HuLy68Q4jw4HyaMeV0= 2023-11-02T18:19:08.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0XIVnX524HuLy68Q4jw4HyaMeV0=', 'Content-Length': '0'} 2023-11-02T18:19:08.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084cfdb46c14cc9a6-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.033 DEBUG:boto:path=/ibtljn-48/obj6 2023-11-02T18:19:08.034 DEBUG:boto:auth_path=/ibtljn-48/obj6 2023-11-02T18:19:08.034 DEBUG:boto:path=/ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.035 DEBUG:boto:auth_path=/ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.035 DEBUG:boto:Method: HEAD 2023-11-02T18:19:08.036 DEBUG:boto:Path: /ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.036 DEBUG:boto:Data: 2023-11-02T18:19:08.036 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.037 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.037 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.038 DEBUG:boto:Params: {} 2023-11-02T18:19:08.038 DEBUG:boto:Token: None 2023-11-02T18:19:08.039 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.039 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0XIVnX524HuLy68Q4jw4HyaMeV0= 2023-11-02T18:19:08.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0XIVnX524HuLy68Q4jw4HyaMeV0=', 'Content-Length': '0'} 2023-11-02T18:19:08.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebf3d2667929094a-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-11-02T18:19:08.043 DEBUG:boto:path=/ibtljn-48/obj6 2023-11-02T18:19:08.044 DEBUG:boto:auth_path=/ibtljn-48/obj6 2023-11-02T18:19:08.044 DEBUG:boto:path=/ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.044 DEBUG:boto:auth_path=/ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.045 DEBUG:boto:Method: GET 2023-11-02T18:19:08.045 DEBUG:boto:Path: /ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.046 DEBUG:boto:Data: 2023-11-02T18:19:08.046 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.047 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.047 DEBUG:boto:Params: {} 2023-11-02T18:19:08.048 DEBUG:boto:Token: None 2023-11-02T18:19:08.048 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.049 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F57LbNuCZhmMyPflukjG7Qhj4jw= 2023-11-02T18:19:08.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F57LbNuCZhmMyPflukjG7Qhj4jw=', 'Content-Length': '0'} 2023-11-02T18:19:08.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005de4828b12dc85bb-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.053 DEBUG:boto:path=/ibtljn-48/obj6 2023-11-02T18:19:08.053 DEBUG:boto:auth_path=/ibtljn-48/obj6 2023-11-02T18:19:08.054 DEBUG:boto:path=/ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.054 DEBUG:boto:auth_path=/ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.055 DEBUG:boto:Method: GET 2023-11-02T18:19:08.055 DEBUG:boto:Path: /ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.056 DEBUG:boto:Data: 2023-11-02T18:19:08.056 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.056 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.057 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.057 DEBUG:boto:Params: {} 2023-11-02T18:19:08.058 DEBUG:boto:Token: None 2023-11-02T18:19:08.058 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj6?versionId=null 2023-11-02T18:19:08.059 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F57LbNuCZhmMyPflukjG7Qhj4jw= 2023-11-02T18:19:08.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F57LbNuCZhmMyPflukjG7Qhj4jw=', 'Content-Length': '0'} 2023-11-02T18:19:08.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab106fbddbf14394-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.063 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:08.063 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:08.064 DEBUG:boto:path=/ibtljn-48/?prefix=obj6 2023-11-02T18:19:08.064 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj6 2023-11-02T18:19:08.065 DEBUG:boto:Method: GET 2023-11-02T18:19:08.065 DEBUG:boto:Path: /ibtljn-48/?prefix=obj6 2023-11-02T18:19:08.065 DEBUG:boto:Data: 2023-11-02T18:19:08.066 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.066 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.067 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.067 DEBUG:boto:Params: {} 2023-11-02T18:19:08.068 DEBUG:boto:Token: None 2023-11-02T18:19:08.068 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/ 2023-11-02T18:19:08.069 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM= 2023-11-02T18:19:08.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM=', 'Content-Length': '0'} 2023-11-02T18:19:08.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f77dfbed025bbba-006543e81c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.075 DEBUG:boto:b'ibtljn-48obj61000falseobj62023-11-02T18:18:30.540Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:08.075 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:08.076 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:08.076 DEBUG:boto:path=/ibtljn-48/?prefix=obj6 2023-11-02T18:19:08.077 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj6 2023-11-02T18:19:08.077 DEBUG:boto:Method: GET 2023-11-02T18:19:08.077 DEBUG:boto:Path: /ibtljn-48/?prefix=obj6 2023-11-02T18:19:08.078 DEBUG:boto:Data: 2023-11-02T18:19:08.078 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.079 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.079 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.079 DEBUG:boto:Params: {} 2023-11-02T18:19:08.080 DEBUG:boto:Token: None 2023-11-02T18:19:08.080 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/ 2023-11-02T18:19:08.081 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM= 2023-11-02T18:19:08.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM=', 'Content-Length': '0'} 2023-11-02T18:19:08.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d23b24d688f5d1a2-006543e81c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.130 DEBUG:boto:b'ibtljn-48obj61000falseobj62023-11-02T18:18:30.540Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:08.131 DEBUG:boto:path=/ibtljn-48/obj6 2023-11-02T18:19:08.132 DEBUG:boto:auth_path=/ibtljn-48/obj6 2023-11-02T18:19:08.132 DEBUG:boto:Method: HEAD 2023-11-02T18:19:08.132 DEBUG:boto:Path: /ibtljn-48/obj6 2023-11-02T18:19:08.133 DEBUG:boto:Data: 2023-11-02T18:19:08.133 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.134 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.134 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.135 DEBUG:boto:Params: {} 2023-11-02T18:19:08.135 DEBUG:boto:Token: None 2023-11-02T18:19:08.136 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj6 2023-11-02T18:19:08.136 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8P6DHibb/oz5n4cCMCvwh0rfbcw= 2023-11-02T18:19:08.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8P6DHibb/oz5n4cCMCvwh0rfbcw=', 'Content-Length': '0'} 2023-11-02T18:19:08.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e23822721de98f4-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.140 DEBUG:boto:path=/ibtljn-48/obj6 2023-11-02T18:19:08.140 DEBUG:boto:auth_path=/ibtljn-48/obj6 2023-11-02T18:19:08.141 DEBUG:boto:Method: HEAD 2023-11-02T18:19:08.141 DEBUG:boto:Path: /ibtljn-48/obj6 2023-11-02T18:19:08.142 DEBUG:boto:Data: 2023-11-02T18:19:08.142 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.143 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.143 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.143 DEBUG:boto:Params: {} 2023-11-02T18:19:08.144 DEBUG:boto:Token: None 2023-11-02T18:19:08.144 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj6 2023-11-02T18:19:08.145 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8P6DHibb/oz5n4cCMCvwh0rfbcw= 2023-11-02T18:19:08.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8P6DHibb/oz5n4cCMCvwh0rfbcw=', 'Content-Length': '0'} 2023-11-02T18:19:08.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089a4a20f0886de10-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-11-02T18:19:08.149 DEBUG:boto:path=/ibtljn-48/obj6 2023-11-02T18:19:08.150 DEBUG:boto:auth_path=/ibtljn-48/obj6 2023-11-02T18:19:08.150 DEBUG:boto:Method: GET 2023-11-02T18:19:08.151 DEBUG:boto:Path: /ibtljn-48/obj6 2023-11-02T18:19:08.151 DEBUG:boto:Data: 2023-11-02T18:19:08.152 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.152 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.153 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.153 DEBUG:boto:Params: {} 2023-11-02T18:19:08.154 DEBUG:boto:Token: None 2023-11-02T18:19:08.154 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj6 2023-11-02T18:19:08.155 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kTIckN6F+Ak2IvHAcJOdmartyZY= 2023-11-02T18:19:08.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kTIckN6F+Ak2IvHAcJOdmartyZY=', 'Content-Length': '0'} 2023-11-02T18:19:08.159 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc51bfdae4d1e85f-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.202 DEBUG:boto:path=/ibtljn-48/obj6 2023-11-02T18:19:08.203 DEBUG:boto:auth_path=/ibtljn-48/obj6 2023-11-02T18:19:08.203 DEBUG:boto:Method: GET 2023-11-02T18:19:08.204 DEBUG:boto:Path: /ibtljn-48/obj6 2023-11-02T18:19:08.204 DEBUG:boto:Data: 2023-11-02T18:19:08.205 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.205 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.205 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.206 DEBUG:boto:Params: {} 2023-11-02T18:19:08.206 DEBUG:boto:Token: None 2023-11-02T18:19:08.207 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj6 2023-11-02T18:19:08.207 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kTIckN6F+Ak2IvHAcJOdmartyZY= 2023-11-02T18:19:08.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kTIckN6F+Ak2IvHAcJOdmartyZY=', 'Content-Length': '0'} 2023-11-02T18:19:08.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000690400946775dc63-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.212 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:08.212 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:08.213 DEBUG:boto:path=/ibtljn-48/?prefix=obj6 2023-11-02T18:19:08.213 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj6 2023-11-02T18:19:08.214 DEBUG:boto:Method: GET 2023-11-02T18:19:08.214 DEBUG:boto:Path: /ibtljn-48/?prefix=obj6 2023-11-02T18:19:08.214 DEBUG:boto:Data: 2023-11-02T18:19:08.215 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.215 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.216 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.216 DEBUG:boto:Params: {} 2023-11-02T18:19:08.216 DEBUG:boto:Token: None 2023-11-02T18:19:08.217 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/ 2023-11-02T18:19:08.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM= 2023-11-02T18:19:08.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM=', 'Content-Length': '0'} 2023-11-02T18:19:08.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d8d5e22e20b8c44-006543e81c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.223 DEBUG:boto:b'ibtljn-48obj61000falseobj62023-11-02T18:18:30.540Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:08.224 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:08.225 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:08.225 DEBUG:boto:path=/ibtljn-48/?prefix=obj6 2023-11-02T18:19:08.225 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj6 2023-11-02T18:19:08.226 DEBUG:boto:Method: GET 2023-11-02T18:19:08.226 DEBUG:boto:Path: /ibtljn-48/?prefix=obj6 2023-11-02T18:19:08.227 DEBUG:boto:Data: 2023-11-02T18:19:08.227 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.228 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.228 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.229 DEBUG:boto:Params: {} 2023-11-02T18:19:08.229 DEBUG:boto:Token: None 2023-11-02T18:19:08.229 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/ 2023-11-02T18:19:08.230 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM= 2023-11-02T18:19:08.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM=', 'Content-Length': '0'} 2023-11-02T18:19:08.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a744a1e478b7583-006543e81c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.236 DEBUG:boto:b'ibtljn-48obj61000falseobj62023-11-02T18:18:30.540Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:08.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-11-02T18:19:08.237 DEBUG:boto:path=/ibtljn-48/obj7 2023-11-02T18:19:08.237 DEBUG:boto:auth_path=/ibtljn-48/obj7 2023-11-02T18:19:08.237 DEBUG:boto:path=/ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.238 DEBUG:boto:auth_path=/ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.238 DEBUG:boto:Method: GET 2023-11-02T18:19:08.239 DEBUG:boto:Path: /ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.239 DEBUG:boto:Data: 2023-11-02T18:19:08.239 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.240 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.240 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.241 DEBUG:boto:Params: {} 2023-11-02T18:19:08.241 DEBUG:boto:Token: None 2023-11-02T18:19:08.241 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rfFMCmfl6QdcIwqu9flAgfGBKkc= 2023-11-02T18:19:08.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rfFMCmfl6QdcIwqu9flAgfGBKkc=', 'Content-Length': '0'} 2023-11-02T18:19:08.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e25face3e5241e98-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.246 DEBUG:boto:path=/ibtljn-48/obj7 2023-11-02T18:19:08.247 DEBUG:boto:auth_path=/ibtljn-48/obj7 2023-11-02T18:19:08.247 DEBUG:boto:path=/ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.248 DEBUG:boto:auth_path=/ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.248 DEBUG:boto:Method: GET 2023-11-02T18:19:08.249 DEBUG:boto:Path: /ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.249 DEBUG:boto:Data: 2023-11-02T18:19:08.250 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.250 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.250 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.251 DEBUG:boto:Params: {} 2023-11-02T18:19:08.251 DEBUG:boto:Token: None 2023-11-02T18:19:08.252 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.252 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rfFMCmfl6QdcIwqu9flAgfGBKkc= 2023-11-02T18:19:08.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rfFMCmfl6QdcIwqu9flAgfGBKkc=', 'Content-Length': '0'} 2023-11-02T18:19:08.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea02da519a8e923a-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.298 DEBUG:boto:path=/ibtljn-48/obj7 2023-11-02T18:19:08.299 DEBUG:boto:auth_path=/ibtljn-48/obj7 2023-11-02T18:19:08.299 DEBUG:boto:path=/ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.300 DEBUG:boto:auth_path=/ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.300 DEBUG:boto:Method: HEAD 2023-11-02T18:19:08.300 DEBUG:boto:Path: /ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.301 DEBUG:boto:Data: 2023-11-02T18:19:08.301 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.302 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.302 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.303 DEBUG:boto:Params: {} 2023-11-02T18:19:08.303 DEBUG:boto:Token: None 2023-11-02T18:19:08.304 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.304 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0CNErk2Z/MmXhPfqJc7dAZqHGx0= 2023-11-02T18:19:08.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0CNErk2Z/MmXhPfqJc7dAZqHGx0=', 'Content-Length': '0'} 2023-11-02T18:19:08.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e161f72647cfeb2-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.308 DEBUG:boto:path=/ibtljn-48/obj7 2023-11-02T18:19:08.308 DEBUG:boto:auth_path=/ibtljn-48/obj7 2023-11-02T18:19:08.309 DEBUG:boto:path=/ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.309 DEBUG:boto:auth_path=/ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.310 DEBUG:boto:Method: HEAD 2023-11-02T18:19:08.310 DEBUG:boto:Path: /ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.310 DEBUG:boto:Data: 2023-11-02T18:19:08.311 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.311 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.312 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.312 DEBUG:boto:Params: {} 2023-11-02T18:19:08.312 DEBUG:boto:Token: None 2023-11-02T18:19:08.313 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.313 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0CNErk2Z/MmXhPfqJc7dAZqHGx0= 2023-11-02T18:19:08.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0CNErk2Z/MmXhPfqJc7dAZqHGx0=', 'Content-Length': '0'} 2023-11-02T18:19:08.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000146a5f4b558384ea-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-11-02T18:19:08.317 DEBUG:boto:path=/ibtljn-48/obj7 2023-11-02T18:19:08.318 DEBUG:boto:auth_path=/ibtljn-48/obj7 2023-11-02T18:19:08.318 DEBUG:boto:path=/ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.319 DEBUG:boto:auth_path=/ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.319 DEBUG:boto:Method: GET 2023-11-02T18:19:08.319 DEBUG:boto:Path: /ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.320 DEBUG:boto:Data: 2023-11-02T18:19:08.320 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.321 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.321 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.321 DEBUG:boto:Params: {} 2023-11-02T18:19:08.322 DEBUG:boto:Token: None 2023-11-02T18:19:08.322 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.323 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rfFMCmfl6QdcIwqu9flAgfGBKkc= 2023-11-02T18:19:08.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rfFMCmfl6QdcIwqu9flAgfGBKkc=', 'Content-Length': '0'} 2023-11-02T18:19:08.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000323cb6c197128f42-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.370 DEBUG:boto:path=/ibtljn-48/obj7 2023-11-02T18:19:08.371 DEBUG:boto:auth_path=/ibtljn-48/obj7 2023-11-02T18:19:08.371 DEBUG:boto:path=/ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.372 DEBUG:boto:auth_path=/ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.373 DEBUG:boto:Method: GET 2023-11-02T18:19:08.373 DEBUG:boto:Path: /ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.374 DEBUG:boto:Data: 2023-11-02T18:19:08.374 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.375 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.375 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.376 DEBUG:boto:Params: {} 2023-11-02T18:19:08.377 DEBUG:boto:Token: None 2023-11-02T18:19:08.378 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj7?versionId=null 2023-11-02T18:19:08.379 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rfFMCmfl6QdcIwqu9flAgfGBKkc= 2023-11-02T18:19:08.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rfFMCmfl6QdcIwqu9flAgfGBKkc=', 'Content-Length': '0'} 2023-11-02T18:19:08.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081b818d9b3f0cbf8-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.383 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:08.384 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:08.384 DEBUG:boto:path=/ibtljn-48/?prefix=obj7 2023-11-02T18:19:08.385 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj7 2023-11-02T18:19:08.385 DEBUG:boto:Method: GET 2023-11-02T18:19:08.385 DEBUG:boto:Path: /ibtljn-48/?prefix=obj7 2023-11-02T18:19:08.386 DEBUG:boto:Data: 2023-11-02T18:19:08.386 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.387 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.387 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.388 DEBUG:boto:Params: {} 2023-11-02T18:19:08.388 DEBUG:boto:Token: None 2023-11-02T18:19:08.388 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/ 2023-11-02T18:19:08.389 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM= 2023-11-02T18:19:08.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM=', 'Content-Length': '0'} 2023-11-02T18:19:08.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f90d31733e22377c-006543e81c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.395 DEBUG:boto:b'ibtljn-48obj71000falseobj72023-11-02T18:18:30.601Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:08.396 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:08.397 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:08.397 DEBUG:boto:path=/ibtljn-48/?prefix=obj7 2023-11-02T18:19:08.397 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj7 2023-11-02T18:19:08.398 DEBUG:boto:Method: GET 2023-11-02T18:19:08.398 DEBUG:boto:Path: /ibtljn-48/?prefix=obj7 2023-11-02T18:19:08.399 DEBUG:boto:Data: 2023-11-02T18:19:08.399 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.400 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.400 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.401 DEBUG:boto:Params: {} 2023-11-02T18:19:08.401 DEBUG:boto:Token: None 2023-11-02T18:19:08.402 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/ 2023-11-02T18:19:08.402 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM= 2023-11-02T18:19:08.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM=', 'Content-Length': '0'} 2023-11-02T18:19:08.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2769bcbc80637cf-006543e81c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.409 DEBUG:boto:b'ibtljn-48obj71000falseobj72023-11-02T18:18:30.601Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:08.409 DEBUG:boto:path=/ibtljn-48/obj7 2023-11-02T18:19:08.410 DEBUG:boto:auth_path=/ibtljn-48/obj7 2023-11-02T18:19:08.410 DEBUG:boto:Method: HEAD 2023-11-02T18:19:08.411 DEBUG:boto:Path: /ibtljn-48/obj7 2023-11-02T18:19:08.411 DEBUG:boto:Data: 2023-11-02T18:19:08.412 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.413 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.413 DEBUG:boto:Params: {} 2023-11-02T18:19:08.414 DEBUG:boto:Token: None 2023-11-02T18:19:08.414 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj7 2023-11-02T18:19:08.415 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M7nTS80UE/ZLxgEl+T0XRPVvRkw= 2023-11-02T18:19:08.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M7nTS80UE/ZLxgEl+T0XRPVvRkw=', 'Content-Length': '0'} 2023-11-02T18:19:08.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbb8759e67858d2e-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.419 DEBUG:boto:path=/ibtljn-48/obj7 2023-11-02T18:19:08.420 DEBUG:boto:auth_path=/ibtljn-48/obj7 2023-11-02T18:19:08.420 DEBUG:boto:Method: HEAD 2023-11-02T18:19:08.420 DEBUG:boto:Path: /ibtljn-48/obj7 2023-11-02T18:19:08.421 DEBUG:boto:Data: 2023-11-02T18:19:08.421 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.422 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.423 DEBUG:boto:Params: {} 2023-11-02T18:19:08.423 DEBUG:boto:Token: None 2023-11-02T18:19:08.424 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj7 2023-11-02T18:19:08.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M7nTS80UE/ZLxgEl+T0XRPVvRkw= 2023-11-02T18:19:08.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M7nTS80UE/ZLxgEl+T0XRPVvRkw=', 'Content-Length': '0'} 2023-11-02T18:19:08.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e7248a348350aa4-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-11-02T18:19:08.429 DEBUG:boto:path=/ibtljn-48/obj7 2023-11-02T18:19:08.429 DEBUG:boto:auth_path=/ibtljn-48/obj7 2023-11-02T18:19:08.430 DEBUG:boto:Method: GET 2023-11-02T18:19:08.430 DEBUG:boto:Path: /ibtljn-48/obj7 2023-11-02T18:19:08.431 DEBUG:boto:Data: 2023-11-02T18:19:08.431 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.432 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.432 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.433 DEBUG:boto:Params: {} 2023-11-02T18:19:08.433 DEBUG:boto:Token: None 2023-11-02T18:19:08.434 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj7 2023-11-02T18:19:08.434 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GZVtzzmH7wDtpLbQFV4hUbRrRW4= 2023-11-02T18:19:08.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GZVtzzmH7wDtpLbQFV4hUbRrRW4=', 'Content-Length': '0'} 2023-11-02T18:19:08.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008df9f50572a50143-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.482 DEBUG:boto:path=/ibtljn-48/obj7 2023-11-02T18:19:08.483 DEBUG:boto:auth_path=/ibtljn-48/obj7 2023-11-02T18:19:08.483 DEBUG:boto:Method: GET 2023-11-02T18:19:08.484 DEBUG:boto:Path: /ibtljn-48/obj7 2023-11-02T18:19:08.484 DEBUG:boto:Data: 2023-11-02T18:19:08.485 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.485 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.486 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.486 DEBUG:boto:Params: {} 2023-11-02T18:19:08.487 DEBUG:boto:Token: None 2023-11-02T18:19:08.487 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj7 2023-11-02T18:19:08.488 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GZVtzzmH7wDtpLbQFV4hUbRrRW4= 2023-11-02T18:19:08.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GZVtzzmH7wDtpLbQFV4hUbRrRW4=', 'Content-Length': '0'} 2023-11-02T18:19:08.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047c3befce951fc57-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.492 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:08.493 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:08.493 DEBUG:boto:path=/ibtljn-48/?prefix=obj7 2023-11-02T18:19:08.494 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj7 2023-11-02T18:19:08.494 DEBUG:boto:Method: GET 2023-11-02T18:19:08.495 DEBUG:boto:Path: /ibtljn-48/?prefix=obj7 2023-11-02T18:19:08.495 DEBUG:boto:Data: 2023-11-02T18:19:08.495 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.496 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.496 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.497 DEBUG:boto:Params: {} 2023-11-02T18:19:08.497 DEBUG:boto:Token: None 2023-11-02T18:19:08.498 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/ 2023-11-02T18:19:08.498 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM= 2023-11-02T18:19:08.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM=', 'Content-Length': '0'} 2023-11-02T18:19:08.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037567bc6c0575895-006543e81c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.504 DEBUG:boto:b'ibtljn-48obj71000falseobj72023-11-02T18:18:30.601Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:08.505 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:08.505 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:08.506 DEBUG:boto:path=/ibtljn-48/?prefix=obj7 2023-11-02T18:19:08.506 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj7 2023-11-02T18:19:08.507 DEBUG:boto:Method: GET 2023-11-02T18:19:08.507 DEBUG:boto:Path: /ibtljn-48/?prefix=obj7 2023-11-02T18:19:08.507 DEBUG:boto:Data: 2023-11-02T18:19:08.508 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.508 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.509 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.509 DEBUG:boto:Params: {} 2023-11-02T18:19:08.510 DEBUG:boto:Token: None 2023-11-02T18:19:08.510 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/ 2023-11-02T18:19:08.511 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM= 2023-11-02T18:19:08.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM=', 'Content-Length': '0'} 2023-11-02T18:19:08.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000643d4adfd709b22a-006543e81c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.516 DEBUG:boto:b'ibtljn-48obj71000falseobj72023-11-02T18:18:30.601Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:08.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-11-02T18:19:08.517 DEBUG:boto:path=/ibtljn-48/obj8 2023-11-02T18:19:08.518 DEBUG:boto:auth_path=/ibtljn-48/obj8 2023-11-02T18:19:08.518 DEBUG:boto:path=/ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.519 DEBUG:boto:auth_path=/ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.519 DEBUG:boto:Method: GET 2023-11-02T18:19:08.520 DEBUG:boto:Path: /ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.520 DEBUG:boto:Data: 2023-11-02T18:19:08.520 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.521 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.521 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.522 DEBUG:boto:Params: {} 2023-11-02T18:19:08.522 DEBUG:boto:Token: None 2023-11-02T18:19:08.523 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.523 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gwaTZ1w7EC0CL+1p0SZ4oApEBTE= 2023-11-02T18:19:08.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gwaTZ1w7EC0CL+1p0SZ4oApEBTE=', 'Content-Length': '0'} 2023-11-02T18:19:08.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f7e4520b7e14716-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.528 DEBUG:boto:path=/ibtljn-48/obj8 2023-11-02T18:19:08.528 DEBUG:boto:auth_path=/ibtljn-48/obj8 2023-11-02T18:19:08.529 DEBUG:boto:path=/ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.529 DEBUG:boto:auth_path=/ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.530 DEBUG:boto:Method: GET 2023-11-02T18:19:08.530 DEBUG:boto:Path: /ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.530 DEBUG:boto:Data: 2023-11-02T18:19:08.531 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.531 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.532 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.532 DEBUG:boto:Params: {} 2023-11-02T18:19:08.533 DEBUG:boto:Token: None 2023-11-02T18:19:08.533 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.534 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gwaTZ1w7EC0CL+1p0SZ4oApEBTE= 2023-11-02T18:19:08.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gwaTZ1w7EC0CL+1p0SZ4oApEBTE=', 'Content-Length': '0'} 2023-11-02T18:19:08.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc729a779e2edd2d-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.538 DEBUG:boto:path=/ibtljn-48/obj8 2023-11-02T18:19:08.538 DEBUG:boto:auth_path=/ibtljn-48/obj8 2023-11-02T18:19:08.538 DEBUG:boto:path=/ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.539 DEBUG:boto:auth_path=/ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.539 DEBUG:boto:Method: HEAD 2023-11-02T18:19:08.540 DEBUG:boto:Path: /ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.540 DEBUG:boto:Data: 2023-11-02T18:19:08.541 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.541 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.542 DEBUG:boto:Params: {} 2023-11-02T18:19:08.542 DEBUG:boto:Token: None 2023-11-02T18:19:08.543 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.543 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QB9EUP4KFCzJXu2K0kCFXyG7Q1g= 2023-11-02T18:19:08.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QB9EUP4KFCzJXu2K0kCFXyG7Q1g=', 'Content-Length': '0'} 2023-11-02T18:19:08.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042afab972b9729a5-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.547 DEBUG:boto:path=/ibtljn-48/obj8 2023-11-02T18:19:08.548 DEBUG:boto:auth_path=/ibtljn-48/obj8 2023-11-02T18:19:08.548 DEBUG:boto:path=/ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.549 DEBUG:boto:auth_path=/ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.549 DEBUG:boto:Method: HEAD 2023-11-02T18:19:08.549 DEBUG:boto:Path: /ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.550 DEBUG:boto:Data: 2023-11-02T18:19:08.550 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.551 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.551 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.551 DEBUG:boto:Params: {} 2023-11-02T18:19:08.552 DEBUG:boto:Token: None 2023-11-02T18:19:08.552 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.553 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QB9EUP4KFCzJXu2K0kCFXyG7Q1g= 2023-11-02T18:19:08.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QB9EUP4KFCzJXu2K0kCFXyG7Q1g=', 'Content-Length': '0'} 2023-11-02T18:19:08.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000112037556fa1126e-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-11-02T18:19:08.559 DEBUG:boto:path=/ibtljn-48/obj8 2023-11-02T18:19:08.559 DEBUG:boto:auth_path=/ibtljn-48/obj8 2023-11-02T18:19:08.560 DEBUG:boto:path=/ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.560 DEBUG:boto:auth_path=/ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.561 DEBUG:boto:Method: GET 2023-11-02T18:19:08.561 DEBUG:boto:Path: /ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.561 DEBUG:boto:Data: 2023-11-02T18:19:08.562 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.562 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.563 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.563 DEBUG:boto:Params: {} 2023-11-02T18:19:08.564 DEBUG:boto:Token: None 2023-11-02T18:19:08.564 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.565 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gwaTZ1w7EC0CL+1p0SZ4oApEBTE= 2023-11-02T18:19:08.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gwaTZ1w7EC0CL+1p0SZ4oApEBTE=', 'Content-Length': '0'} 2023-11-02T18:19:08.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010b27edeaa84da97-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.610 DEBUG:boto:path=/ibtljn-48/obj8 2023-11-02T18:19:08.611 DEBUG:boto:auth_path=/ibtljn-48/obj8 2023-11-02T18:19:08.611 DEBUG:boto:path=/ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.612 DEBUG:boto:auth_path=/ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.612 DEBUG:boto:Method: GET 2023-11-02T18:19:08.612 DEBUG:boto:Path: /ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.613 DEBUG:boto:Data: 2023-11-02T18:19:08.613 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.614 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.614 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.615 DEBUG:boto:Params: {} 2023-11-02T18:19:08.615 DEBUG:boto:Token: None 2023-11-02T18:19:08.616 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj8?versionId=null 2023-11-02T18:19:08.616 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gwaTZ1w7EC0CL+1p0SZ4oApEBTE= 2023-11-02T18:19:08.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gwaTZ1w7EC0CL+1p0SZ4oApEBTE=', 'Content-Length': '0'} 2023-11-02T18:19:08.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091e89abb346ccc42-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.620 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:08.621 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:08.621 DEBUG:boto:path=/ibtljn-48/?prefix=obj8 2023-11-02T18:19:08.621 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj8 2023-11-02T18:19:08.622 DEBUG:boto:Method: GET 2023-11-02T18:19:08.622 DEBUG:boto:Path: /ibtljn-48/?prefix=obj8 2023-11-02T18:19:08.623 DEBUG:boto:Data: 2023-11-02T18:19:08.623 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.624 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.624 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.625 DEBUG:boto:Params: {} 2023-11-02T18:19:08.625 DEBUG:boto:Token: None 2023-11-02T18:19:08.626 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/ 2023-11-02T18:19:08.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM= 2023-11-02T18:19:08.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM=', 'Content-Length': '0'} 2023-11-02T18:19:08.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9a5eacd911fd384-006543e81c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.632 DEBUG:boto:b'ibtljn-48obj81000falseobj82023-11-02T18:18:30.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:08.633 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:08.633 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:08.634 DEBUG:boto:path=/ibtljn-48/?prefix=obj8 2023-11-02T18:19:08.634 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj8 2023-11-02T18:19:08.635 DEBUG:boto:Method: GET 2023-11-02T18:19:08.635 DEBUG:boto:Path: /ibtljn-48/?prefix=obj8 2023-11-02T18:19:08.636 DEBUG:boto:Data: 2023-11-02T18:19:08.636 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.636 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.637 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.637 DEBUG:boto:Params: {} 2023-11-02T18:19:08.638 DEBUG:boto:Token: None 2023-11-02T18:19:08.638 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/ 2023-11-02T18:19:08.639 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM= 2023-11-02T18:19:08.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM=', 'Content-Length': '0'} 2023-11-02T18:19:08.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b49a4f332b2b6f0-006543e81c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.690 DEBUG:boto:b'ibtljn-48obj81000falseobj82023-11-02T18:18:30.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:08.691 DEBUG:boto:path=/ibtljn-48/obj8 2023-11-02T18:19:08.692 DEBUG:boto:auth_path=/ibtljn-48/obj8 2023-11-02T18:19:08.692 DEBUG:boto:Method: HEAD 2023-11-02T18:19:08.692 DEBUG:boto:Path: /ibtljn-48/obj8 2023-11-02T18:19:08.693 DEBUG:boto:Data: 2023-11-02T18:19:08.693 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.694 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.694 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.695 DEBUG:boto:Params: {} 2023-11-02T18:19:08.695 DEBUG:boto:Token: None 2023-11-02T18:19:08.696 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj8 2023-11-02T18:19:08.696 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SB15be084n30m6t8yox9+XPSHpk= 2023-11-02T18:19:08.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SB15be084n30m6t8yox9+XPSHpk=', 'Content-Length': '0'} 2023-11-02T18:19:08.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086102d2387c88bd1-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.700 DEBUG:boto:path=/ibtljn-48/obj8 2023-11-02T18:19:08.701 DEBUG:boto:auth_path=/ibtljn-48/obj8 2023-11-02T18:19:08.701 DEBUG:boto:Method: HEAD 2023-11-02T18:19:08.701 DEBUG:boto:Path: /ibtljn-48/obj8 2023-11-02T18:19:08.702 DEBUG:boto:Data: 2023-11-02T18:19:08.702 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.703 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.703 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.703 DEBUG:boto:Params: {} 2023-11-02T18:19:08.704 DEBUG:boto:Token: None 2023-11-02T18:19:08.704 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj8 2023-11-02T18:19:08.705 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SB15be084n30m6t8yox9+XPSHpk= 2023-11-02T18:19:08.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SB15be084n30m6t8yox9+XPSHpk=', 'Content-Length': '0'} 2023-11-02T18:19:08.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddb953ae6d67efdc-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-11-02T18:19:08.709 DEBUG:boto:path=/ibtljn-48/obj8 2023-11-02T18:19:08.709 DEBUG:boto:auth_path=/ibtljn-48/obj8 2023-11-02T18:19:08.710 DEBUG:boto:Method: GET 2023-11-02T18:19:08.710 DEBUG:boto:Path: /ibtljn-48/obj8 2023-11-02T18:19:08.711 DEBUG:boto:Data: 2023-11-02T18:19:08.711 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.712 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.712 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.712 DEBUG:boto:Params: {} 2023-11-02T18:19:08.713 DEBUG:boto:Token: None 2023-11-02T18:19:08.713 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj8 2023-11-02T18:19:08.714 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l2NJsoid2tOtuySCACu45sL7mQE= 2023-11-02T18:19:08.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l2NJsoid2tOtuySCACu45sL7mQE=', 'Content-Length': '0'} 2023-11-02T18:19:08.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-48:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a40d6ad94990a330-006543e81c-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.718 DEBUG:boto:path=/ibtljn-48/obj8 2023-11-02T18:19:08.718 DEBUG:boto:auth_path=/ibtljn-48/obj8 2023-11-02T18:19:08.719 DEBUG:boto:Method: GET 2023-11-02T18:19:08.719 DEBUG:boto:Path: /ibtljn-48/obj8 2023-11-02T18:19:08.720 DEBUG:boto:Data: 2023-11-02T18:19:08.720 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.721 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.721 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.721 DEBUG:boto:Params: {} 2023-11-02T18:19:08.722 DEBUG:boto:Token: None 2023-11-02T18:19:08.722 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/obj8 2023-11-02T18:19:08.723 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l2NJsoid2tOtuySCACu45sL7mQE= 2023-11-02T18:19:08.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l2NJsoid2tOtuySCACu45sL7mQE=', 'Content-Length': '0'} 2023-11-02T18:19:08.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efbd3da423428573-006543e81c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.727 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:08.727 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:08.728 DEBUG:boto:path=/ibtljn-48/?prefix=obj8 2023-11-02T18:19:08.728 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj8 2023-11-02T18:19:08.729 DEBUG:boto:Method: GET 2023-11-02T18:19:08.729 DEBUG:boto:Path: /ibtljn-48/?prefix=obj8 2023-11-02T18:19:08.730 DEBUG:boto:Data: 2023-11-02T18:19:08.730 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.730 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:08.731 DEBUG:boto:Port: 8001 2023-11-02T18:19:08.731 DEBUG:boto:Params: {} 2023-11-02T18:19:08.732 DEBUG:boto:Token: None 2023-11-02T18:19:08.732 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/ 2023-11-02T18:19:08.733 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM= 2023-11-02T18:19:08.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM=', 'Content-Length': '0'} 2023-11-02T18:19:08.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023374317f54f545b-006543e81c-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.739 DEBUG:boto:b'ibtljn-48obj81000falseobj82023-11-02T18:18:30.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:08.739 DEBUG:boto:path=/ibtljn-48/ 2023-11-02T18:19:08.740 DEBUG:boto:auth_path=/ibtljn-48/ 2023-11-02T18:19:08.740 DEBUG:boto:path=/ibtljn-48/?prefix=obj8 2023-11-02T18:19:08.741 DEBUG:boto:auth_path=/ibtljn-48/?prefix=obj8 2023-11-02T18:19:08.741 DEBUG:boto:Method: GET 2023-11-02T18:19:08.742 DEBUG:boto:Path: /ibtljn-48/?prefix=obj8 2023-11-02T18:19:08.742 DEBUG:boto:Data: 2023-11-02T18:19:08.743 DEBUG:boto:Headers: {} 2023-11-02T18:19:08.743 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:08.743 DEBUG:boto:Port: 8000 2023-11-02T18:19:08.744 DEBUG:boto:Params: {} 2023-11-02T18:19:08.744 DEBUG:boto:Token: None 2023-11-02T18:19:08.745 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:08 GMT /ibtljn-48/ 2023-11-02T18:19:08.745 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM= 2023-11-02T18:19:08.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ss8lLlk6e1rY60ktDV4UYEVIODM=', 'Content-Length': '0'} 2023-11-02T18:19:08.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b38b12a490c05667-006543e81c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:08.798 DEBUG:boto:b'ibtljn-48obj81000falseobj82023-11-02T18:18:30.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:08.799 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-48 zones={a2, a1} 2023-11-02T18:19:08.800 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-49 2023-11-02T18:19:08.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:19:08.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:19:08.801 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:19:08.853 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:08.854 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:08.922 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:08.914+0000 7fcc211a6f40 1 bucket sync caught up with source: 2023-11-02T18:19:08.923 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000004.20.6, 00000000002.34.1, 00000000004.33.6, 00000000002.49.1, 00000000004.118.6, 00000000002.24.1, 00000000002.31.1, 00000000004.38.6, 00000000002.116.1, 00000000002.14.1, 00000000002.36.1] 2023-11-02T18:19:08.923 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.20.6, 00000000002.34.1, 00000000004.33.6, 00000000002.49.1, 00000000004.118.6, 00000000002.24.1, 00000000002.31.1, 00000000004.38.6, 00000000002.116.1, 00000000002.14.1, 00000000002.36.1] 2023-11-02T18:19:08.923 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:08.914+0000 7fcc211a6f40 0 bucket checkpoint complete 2023-11-02T18:19:08.951 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-49 2023-11-02T18:19:08.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:19:08.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:19:08.953 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:19:09.000 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:09.001 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:09.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:09.066+0000 7f5e406e5f40 1 bucket sync caught up with source: 2023-11-02T18:19:09.074 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000004.243.1, 00000000002.68.1, 00000000004.245.1, 00000000002.113.1, 00000000004.271.1, 00000000002.68.1, 00000000002.246.1, 00000000004.69.1, 00000000002.272.1, 00000000002.55.1, 00000000002.70.1] 2023-11-02T18:19:09.074 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.243.1, 00000000002.68.1, 00000000004.245.1, 00000000002.113.1, 00000000004.271.1, 00000000002.68.1, 00000000002.246.1, 00000000004.69.1, 00000000002.272.1, 00000000002.55.1, 00000000002.70.1] 2023-11-02T18:19:09.074 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:09.066+0000 7f5e406e5f40 0 bucket checkpoint complete 2023-11-02T18:19:09.099 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-49 zones={a2, a1} 2023-11-02T18:19:09.099 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.100 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.100 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.100 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:19:09.101 DEBUG:boto:Data: 2023-11-02T18:19:09.101 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.102 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.102 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.102 DEBUG:boto:Params: {} 2023-11-02T18:19:09.103 DEBUG:boto:Token: None 2023-11-02T18:19:09.103 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uZOCsjwD1AUxEIFgaDGjEmQL6hU= 2023-11-02T18:19:09.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uZOCsjwD1AUxEIFgaDGjEmQL6hU=', 'Content-Length': '0'} 2023-11-02T18:19:09.108 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3056b814597c0e0-006543e81d-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.108 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.109 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.109 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.110 DEBUG:boto:Path: /ibtljn-49/ 2023-11-02T18:19:09.110 DEBUG:boto:Data: 2023-11-02T18:19:09.110 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.111 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.111 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.111 DEBUG:boto:Params: {} 2023-11-02T18:19:09.112 DEBUG:boto:Token: None 2023-11-02T18:19:09.112 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.113 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uZOCsjwD1AUxEIFgaDGjEmQL6hU= 2023-11-02T18:19:09.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uZOCsjwD1AUxEIFgaDGjEmQL6hU=', 'Content-Length': '0'} 2023-11-02T18:19:09.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018b3b2a01859dd3c-006543e81d-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.117 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:19:09.118 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.118 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.119 DEBUG:boto:path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:19:09.119 DEBUG:boto:auth_path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:19:09.120 DEBUG:boto:Method: GET 2023-11-02T18:19:09.120 DEBUG:boto:Path: /ibtljn-49/?versions&max-keys=999 2023-11-02T18:19:09.120 DEBUG:boto:Data: 2023-11-02T18:19:09.121 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.121 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.122 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.122 DEBUG:boto:Params: {} 2023-11-02T18:19:09.122 DEBUG:boto:Token: None 2023-11-02T18:19:09.123 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/?versions 2023-11-02T18:19:09.123 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LeIZxjOzVFyF3tPQ6dbGpx1kiJE= 2023-11-02T18:19:09.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LeIZxjOzVFyF3tPQ6dbGpx1kiJE=', 'Content-Length': '0'} 2023-11-02T18:19:09.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa5a0e7af374f37c-006543e81d-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.129 DEBUG:boto:b'ibtljn-49999falseobj1nulltrue2023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:18:30.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:18:30.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:18:30.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:18:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:19:09.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:19:09.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:19:09.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:19:09.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-11-02T18:19:09.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-11-02T18:19:09.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-11-02T18:19:09.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-11-02T18:19:09.133 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:19:09.134 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.134 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.135 DEBUG:boto:path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:19:09.135 DEBUG:boto:auth_path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:19:09.135 DEBUG:boto:Method: GET 2023-11-02T18:19:09.136 DEBUG:boto:Path: /ibtljn-49/?versions&max-keys=999 2023-11-02T18:19:09.136 DEBUG:boto:Data: 2023-11-02T18:19:09.137 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.137 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.137 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.138 DEBUG:boto:Params: {} 2023-11-02T18:19:09.138 DEBUG:boto:Token: None 2023-11-02T18:19:09.139 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/?versions 2023-11-02T18:19:09.139 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LeIZxjOzVFyF3tPQ6dbGpx1kiJE= 2023-11-02T18:19:09.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LeIZxjOzVFyF3tPQ6dbGpx1kiJE=', 'Content-Length': '0'} 2023-11-02T18:19:09.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bda50d8847bef5d-006543e81d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.145 DEBUG:boto:b'ibtljn-49999falseobj1nulltrue2023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:18:30.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:18:30.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:18:30.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:18:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-11-02T18:19:09.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-11-02T18:19:09.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-11-02T18:19:09.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-11-02T18:19:09.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-11-02T18:19:09.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-11-02T18:19:09.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-11-02T18:19:09.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-11-02T18:19:09.149 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.149 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.150 DEBUG:boto:path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:19:09.150 DEBUG:boto:auth_path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:19:09.151 DEBUG:boto:Method: GET 2023-11-02T18:19:09.151 DEBUG:boto:Path: /ibtljn-49/?versions&max-keys=999 2023-11-02T18:19:09.151 DEBUG:boto:Data: 2023-11-02T18:19:09.152 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.152 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.153 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.153 DEBUG:boto:Params: {} 2023-11-02T18:19:09.153 DEBUG:boto:Token: None 2023-11-02T18:19:09.154 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/?versions 2023-11-02T18:19:09.154 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LeIZxjOzVFyF3tPQ6dbGpx1kiJE= 2023-11-02T18:19:09.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LeIZxjOzVFyF3tPQ6dbGpx1kiJE=', 'Content-Length': '0'} 2023-11-02T18:19:09.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ffef77618553c2a-006543e81d-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.160 DEBUG:boto:b'ibtljn-49999falseobj1nulltrue2023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:18:30.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:18:30.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:18:30.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:18:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.161 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.162 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.162 DEBUG:boto:path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:19:09.163 DEBUG:boto:auth_path=/ibtljn-49/?versions&max-keys=999 2023-11-02T18:19:09.163 DEBUG:boto:Method: GET 2023-11-02T18:19:09.163 DEBUG:boto:Path: /ibtljn-49/?versions&max-keys=999 2023-11-02T18:19:09.164 DEBUG:boto:Data: 2023-11-02T18:19:09.164 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.165 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.165 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.165 DEBUG:boto:Params: {} 2023-11-02T18:19:09.166 DEBUG:boto:Token: None 2023-11-02T18:19:09.166 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/?versions 2023-11-02T18:19:09.167 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LeIZxjOzVFyF3tPQ6dbGpx1kiJE= 2023-11-02T18:19:09.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LeIZxjOzVFyF3tPQ6dbGpx1kiJE=', 'Content-Length': '0'} 2023-11-02T18:19:09.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d90bd5dfcc033ec-006543e81d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.173 DEBUG:boto:b'ibtljn-49999falseobj1nulltrue2023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-11-02T18:18:30.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-11-02T18:18:30.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-11-02T18:18:30.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-11-02T18:18:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:19:09.174 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:19:09.175 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:19:09.175 DEBUG:boto:path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.175 DEBUG:boto:auth_path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.176 DEBUG:boto:Method: GET 2023-11-02T18:19:09.176 DEBUG:boto:Path: /ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.177 DEBUG:boto:Data: 2023-11-02T18:19:09.177 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.177 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.178 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.178 DEBUG:boto:Params: {} 2023-11-02T18:19:09.179 DEBUG:boto:Token: None 2023-11-02T18:19:09.179 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.180 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RUWcb40Q4vkLuE1gUe8XCT44qTk= 2023-11-02T18:19:09.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RUWcb40Q4vkLuE1gUe8XCT44qTk=', 'Content-Length': '0'} 2023-11-02T18:19:09.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd1cfd197fe90ba2-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.227 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:19:09.227 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:19:09.228 DEBUG:boto:path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.228 DEBUG:boto:auth_path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.228 DEBUG:boto:Method: GET 2023-11-02T18:19:09.229 DEBUG:boto:Path: /ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.229 DEBUG:boto:Data: 2023-11-02T18:19:09.230 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.230 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.231 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.231 DEBUG:boto:Params: {} 2023-11-02T18:19:09.231 DEBUG:boto:Token: None 2023-11-02T18:19:09.232 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.232 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RUWcb40Q4vkLuE1gUe8XCT44qTk= 2023-11-02T18:19:09.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RUWcb40Q4vkLuE1gUe8XCT44qTk=', 'Content-Length': '0'} 2023-11-02T18:19:09.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010a73aebe37e1ef3-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.237 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:19:09.237 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:19:09.237 DEBUG:boto:path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.238 DEBUG:boto:auth_path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.238 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.239 DEBUG:boto:Path: /ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.239 DEBUG:boto:Data: 2023-11-02T18:19:09.239 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.240 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.240 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.241 DEBUG:boto:Params: {} 2023-11-02T18:19:09.241 DEBUG:boto:Token: None 2023-11-02T18:19:09.241 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JrfpcQ3Kwf9B825WbjVFWRC9iak= 2023-11-02T18:19:09.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JrfpcQ3Kwf9B825WbjVFWRC9iak=', 'Content-Length': '0'} 2023-11-02T18:19:09.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed090839a3d59264-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.246 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:19:09.246 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:19:09.247 DEBUG:boto:path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.247 DEBUG:boto:auth_path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.247 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.248 DEBUG:boto:Path: /ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.248 DEBUG:boto:Data: 2023-11-02T18:19:09.249 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.249 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.249 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.250 DEBUG:boto:Params: {} 2023-11-02T18:19:09.250 DEBUG:boto:Token: None 2023-11-02T18:19:09.251 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.251 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JrfpcQ3Kwf9B825WbjVFWRC9iak= 2023-11-02T18:19:09.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JrfpcQ3Kwf9B825WbjVFWRC9iak=', 'Content-Length': '0'} 2023-11-02T18:19:09.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000961ba28088272028-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:19:09.255 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:19:09.256 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:19:09.256 DEBUG:boto:path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.256 DEBUG:boto:auth_path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.257 DEBUG:boto:Method: GET 2023-11-02T18:19:09.257 DEBUG:boto:Path: /ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.258 DEBUG:boto:Data: 2023-11-02T18:19:09.258 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.259 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.259 DEBUG:boto:Params: {} 2023-11-02T18:19:09.260 DEBUG:boto:Token: None 2023-11-02T18:19:09.260 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.261 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RUWcb40Q4vkLuE1gUe8XCT44qTk= 2023-11-02T18:19:09.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RUWcb40Q4vkLuE1gUe8XCT44qTk=', 'Content-Length': '0'} 2023-11-02T18:19:09.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068340af140a62fca-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.265 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:19:09.265 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:19:09.266 DEBUG:boto:path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.266 DEBUG:boto:auth_path=/ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.266 DEBUG:boto:Method: GET 2023-11-02T18:19:09.267 DEBUG:boto:Path: /ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.267 DEBUG:boto:Data: 2023-11-02T18:19:09.268 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.268 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.269 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.269 DEBUG:boto:Params: {} 2023-11-02T18:19:09.270 DEBUG:boto:Token: None 2023-11-02T18:19:09.270 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj1?versionId=null 2023-11-02T18:19:09.271 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RUWcb40Q4vkLuE1gUe8XCT44qTk= 2023-11-02T18:19:09.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RUWcb40Q4vkLuE1gUe8XCT44qTk=', 'Content-Length': '0'} 2023-11-02T18:19:09.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d18f3e4c604c8f0e-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.318 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.319 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.319 DEBUG:boto:path=/ibtljn-49/?prefix=obj1 2023-11-02T18:19:09.320 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj1 2023-11-02T18:19:09.320 DEBUG:boto:Method: GET 2023-11-02T18:19:09.321 DEBUG:boto:Path: /ibtljn-49/?prefix=obj1 2023-11-02T18:19:09.321 DEBUG:boto:Data: 2023-11-02T18:19:09.322 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.322 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.323 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.323 DEBUG:boto:Params: {} 2023-11-02T18:19:09.323 DEBUG:boto:Token: None 2023-11-02T18:19:09.324 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.324 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs= 2023-11-02T18:19:09.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs=', 'Content-Length': '0'} 2023-11-02T18:19:09.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023adc1713a37658e-006543e81d-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.330 DEBUG:boto:b'ibtljn-49obj11000falseobj12023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.331 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.332 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.332 DEBUG:boto:path=/ibtljn-49/?prefix=obj1 2023-11-02T18:19:09.332 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj1 2023-11-02T18:19:09.333 DEBUG:boto:Method: GET 2023-11-02T18:19:09.333 DEBUG:boto:Path: /ibtljn-49/?prefix=obj1 2023-11-02T18:19:09.334 DEBUG:boto:Data: 2023-11-02T18:19:09.334 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.335 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.335 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.336 DEBUG:boto:Params: {} 2023-11-02T18:19:09.336 DEBUG:boto:Token: None 2023-11-02T18:19:09.337 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.337 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs= 2023-11-02T18:19:09.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs=', 'Content-Length': '0'} 2023-11-02T18:19:09.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c02dc3937852171a-006543e81d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.343 DEBUG:boto:b'ibtljn-49obj11000falseobj12023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.343 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:19:09.344 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:19:09.344 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.345 DEBUG:boto:Path: /ibtljn-49/obj1 2023-11-02T18:19:09.345 DEBUG:boto:Data: 2023-11-02T18:19:09.345 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.346 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.346 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.347 DEBUG:boto:Params: {} 2023-11-02T18:19:09.347 DEBUG:boto:Token: None 2023-11-02T18:19:09.348 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj1 2023-11-02T18:19:09.348 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1VSURPte0vrA7Fu34n7VUaWEn1w= 2023-11-02T18:19:09.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1VSURPte0vrA7Fu34n7VUaWEn1w=', 'Content-Length': '0'} 2023-11-02T18:19:09.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000baf878d631ab7e7a-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.352 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:19:09.352 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:19:09.353 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.353 DEBUG:boto:Path: /ibtljn-49/obj1 2023-11-02T18:19:09.353 DEBUG:boto:Data: 2023-11-02T18:19:09.354 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.354 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.355 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.355 DEBUG:boto:Params: {} 2023-11-02T18:19:09.356 DEBUG:boto:Token: None 2023-11-02T18:19:09.356 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj1 2023-11-02T18:19:09.356 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1VSURPte0vrA7Fu34n7VUaWEn1w= 2023-11-02T18:19:09.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1VSURPte0vrA7Fu34n7VUaWEn1w=', 'Content-Length': '0'} 2023-11-02T18:19:09.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000347aa32aecf2e48e-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-11-02T18:19:09.361 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:19:09.361 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:19:09.362 DEBUG:boto:Method: GET 2023-11-02T18:19:09.362 DEBUG:boto:Path: /ibtljn-49/obj1 2023-11-02T18:19:09.363 DEBUG:boto:Data: 2023-11-02T18:19:09.363 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.363 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.364 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.364 DEBUG:boto:Params: {} 2023-11-02T18:19:09.365 DEBUG:boto:Token: None 2023-11-02T18:19:09.365 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj1 2023-11-02T18:19:09.366 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lt2sFav0BLgFsD59g3PNIr/5mTI= 2023-11-02T18:19:09.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lt2sFav0BLgFsD59g3PNIr/5mTI=', 'Content-Length': '0'} 2023-11-02T18:19:09.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000334d90fb859d531e-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.410 DEBUG:boto:path=/ibtljn-49/obj1 2023-11-02T18:19:09.411 DEBUG:boto:auth_path=/ibtljn-49/obj1 2023-11-02T18:19:09.411 DEBUG:boto:Method: GET 2023-11-02T18:19:09.412 DEBUG:boto:Path: /ibtljn-49/obj1 2023-11-02T18:19:09.412 DEBUG:boto:Data: 2023-11-02T18:19:09.413 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.413 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.413 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.414 DEBUG:boto:Params: {} 2023-11-02T18:19:09.414 DEBUG:boto:Token: None 2023-11-02T18:19:09.415 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj1 2023-11-02T18:19:09.415 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lt2sFav0BLgFsD59g3PNIr/5mTI= 2023-11-02T18:19:09.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lt2sFav0BLgFsD59g3PNIr/5mTI=', 'Content-Length': '0'} 2023-11-02T18:19:09.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f94688872646296-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.420 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.420 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.421 DEBUG:boto:path=/ibtljn-49/?prefix=obj1 2023-11-02T18:19:09.421 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj1 2023-11-02T18:19:09.421 DEBUG:boto:Method: GET 2023-11-02T18:19:09.422 DEBUG:boto:Path: /ibtljn-49/?prefix=obj1 2023-11-02T18:19:09.422 DEBUG:boto:Data: 2023-11-02T18:19:09.423 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.423 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.423 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.424 DEBUG:boto:Params: {} 2023-11-02T18:19:09.424 DEBUG:boto:Token: None 2023-11-02T18:19:09.425 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.425 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs= 2023-11-02T18:19:09.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs=', 'Content-Length': '0'} 2023-11-02T18:19:09.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009e3fb81059a0215-006543e81d-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.431 DEBUG:boto:b'ibtljn-49obj11000falseobj12023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.432 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.432 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.433 DEBUG:boto:path=/ibtljn-49/?prefix=obj1 2023-11-02T18:19:09.433 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj1 2023-11-02T18:19:09.434 DEBUG:boto:Method: GET 2023-11-02T18:19:09.434 DEBUG:boto:Path: /ibtljn-49/?prefix=obj1 2023-11-02T18:19:09.435 DEBUG:boto:Data: 2023-11-02T18:19:09.435 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.436 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.436 DEBUG:boto:Params: {} 2023-11-02T18:19:09.437 DEBUG:boto:Token: None 2023-11-02T18:19:09.437 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.438 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs= 2023-11-02T18:19:09.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs=', 'Content-Length': '0'} 2023-11-02T18:19:09.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000454ca95542d0bf3f-006543e81d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.486 DEBUG:boto:b'ibtljn-49obj11000falseobj12023-11-02T18:16:46.013Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:19:09.488 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:19:09.488 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:19:09.489 DEBUG:boto:path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.489 DEBUG:boto:auth_path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.489 DEBUG:boto:Method: GET 2023-11-02T18:19:09.490 DEBUG:boto:Path: /ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.490 DEBUG:boto:Data: 2023-11-02T18:19:09.491 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.491 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.492 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.492 DEBUG:boto:Params: {} 2023-11-02T18:19:09.493 DEBUG:boto:Token: None 2023-11-02T18:19:09.493 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.494 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jpYfbjgfvu8kNOLlku5gMG8qksE= 2023-11-02T18:19:09.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jpYfbjgfvu8kNOLlku5gMG8qksE=', 'Content-Length': '0'} 2023-11-02T18:19:09.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6033570214879d9-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.498 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:19:09.499 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:19:09.499 DEBUG:boto:path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.500 DEBUG:boto:auth_path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.500 DEBUG:boto:Method: GET 2023-11-02T18:19:09.501 DEBUG:boto:Path: /ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.501 DEBUG:boto:Data: 2023-11-02T18:19:09.501 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.502 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.502 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.503 DEBUG:boto:Params: {} 2023-11-02T18:19:09.503 DEBUG:boto:Token: None 2023-11-02T18:19:09.504 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.504 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jpYfbjgfvu8kNOLlku5gMG8qksE= 2023-11-02T18:19:09.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jpYfbjgfvu8kNOLlku5gMG8qksE=', 'Content-Length': '0'} 2023-11-02T18:19:09.508 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1c04071228f26c4-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.509 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:19:09.509 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:19:09.509 DEBUG:boto:path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.510 DEBUG:boto:auth_path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.510 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.511 DEBUG:boto:Path: /ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.511 DEBUG:boto:Data: 2023-11-02T18:19:09.512 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.512 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.512 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.513 DEBUG:boto:Params: {} 2023-11-02T18:19:09.513 DEBUG:boto:Token: None 2023-11-02T18:19:09.514 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.514 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:COzv1SHpCVtAFJeIpyHhTpw+bkA= 2023-11-02T18:19:09.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:COzv1SHpCVtAFJeIpyHhTpw+bkA=', 'Content-Length': '0'} 2023-11-02T18:19:09.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dc40b230c5ed35b-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.518 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:19:09.519 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:19:09.519 DEBUG:boto:path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.519 DEBUG:boto:auth_path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.520 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.520 DEBUG:boto:Path: /ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.521 DEBUG:boto:Data: 2023-11-02T18:19:09.521 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.521 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.522 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.522 DEBUG:boto:Params: {} 2023-11-02T18:19:09.523 DEBUG:boto:Token: None 2023-11-02T18:19:09.523 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:COzv1SHpCVtAFJeIpyHhTpw+bkA= 2023-11-02T18:19:09.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:COzv1SHpCVtAFJeIpyHhTpw+bkA=', 'Content-Length': '0'} 2023-11-02T18:19:09.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9185594d4d77ba7-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:19:09.528 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:19:09.528 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:19:09.529 DEBUG:boto:path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.529 DEBUG:boto:auth_path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.530 DEBUG:boto:Method: GET 2023-11-02T18:19:09.530 DEBUG:boto:Path: /ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.531 DEBUG:boto:Data: 2023-11-02T18:19:09.531 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.531 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.532 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.532 DEBUG:boto:Params: {} 2023-11-02T18:19:09.533 DEBUG:boto:Token: None 2023-11-02T18:19:09.533 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.534 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jpYfbjgfvu8kNOLlku5gMG8qksE= 2023-11-02T18:19:09.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jpYfbjgfvu8kNOLlku5gMG8qksE=', 'Content-Length': '0'} 2023-11-02T18:19:09.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6a7f6b37fce970e-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.538 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:19:09.538 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:19:09.539 DEBUG:boto:path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.539 DEBUG:boto:auth_path=/ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.540 DEBUG:boto:Method: GET 2023-11-02T18:19:09.540 DEBUG:boto:Path: /ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.541 DEBUG:boto:Data: 2023-11-02T18:19:09.541 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.542 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.542 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.543 DEBUG:boto:Params: {} 2023-11-02T18:19:09.543 DEBUG:boto:Token: None 2023-11-02T18:19:09.544 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj2?versionId=null 2023-11-02T18:19:09.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jpYfbjgfvu8kNOLlku5gMG8qksE= 2023-11-02T18:19:09.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jpYfbjgfvu8kNOLlku5gMG8qksE=', 'Content-Length': '0'} 2023-11-02T18:19:09.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000894f6d6455f2234a-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.590 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.591 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.591 DEBUG:boto:path=/ibtljn-49/?prefix=obj2 2023-11-02T18:19:09.592 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj2 2023-11-02T18:19:09.592 DEBUG:boto:Method: GET 2023-11-02T18:19:09.593 DEBUG:boto:Path: /ibtljn-49/?prefix=obj2 2023-11-02T18:19:09.593 DEBUG:boto:Data: 2023-11-02T18:19:09.594 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.594 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.595 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.595 DEBUG:boto:Params: {} 2023-11-02T18:19:09.595 DEBUG:boto:Token: None 2023-11-02T18:19:09.596 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.596 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs= 2023-11-02T18:19:09.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs=', 'Content-Length': '0'} 2023-11-02T18:19:09.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abf0b3f65fade64f-006543e81d-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.603 DEBUG:boto:b'ibtljn-49obj21000falseobj22023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.604 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.604 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.605 DEBUG:boto:path=/ibtljn-49/?prefix=obj2 2023-11-02T18:19:09.605 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj2 2023-11-02T18:19:09.606 DEBUG:boto:Method: GET 2023-11-02T18:19:09.606 DEBUG:boto:Path: /ibtljn-49/?prefix=obj2 2023-11-02T18:19:09.607 DEBUG:boto:Data: 2023-11-02T18:19:09.607 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.608 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.608 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.608 DEBUG:boto:Params: {} 2023-11-02T18:19:09.609 DEBUG:boto:Token: None 2023-11-02T18:19:09.609 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.610 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs= 2023-11-02T18:19:09.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs=', 'Content-Length': '0'} 2023-11-02T18:19:09.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042375610f50a9cc8-006543e81d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.616 DEBUG:boto:b'ibtljn-49obj21000falseobj22023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.616 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:19:09.617 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:19:09.617 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.618 DEBUG:boto:Path: /ibtljn-49/obj2 2023-11-02T18:19:09.618 DEBUG:boto:Data: 2023-11-02T18:19:09.618 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.619 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.620 DEBUG:boto:Params: {} 2023-11-02T18:19:09.620 DEBUG:boto:Token: None 2023-11-02T18:19:09.621 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj2 2023-11-02T18:19:09.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BypQY2/Gg8qCLSyJrN22fJJ+uGg= 2023-11-02T18:19:09.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BypQY2/Gg8qCLSyJrN22fJJ+uGg=', 'Content-Length': '0'} 2023-11-02T18:19:09.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039b0da44acd1256b-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.625 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:19:09.625 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:19:09.626 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.626 DEBUG:boto:Path: /ibtljn-49/obj2 2023-11-02T18:19:09.627 DEBUG:boto:Data: 2023-11-02T18:19:09.627 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.628 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.628 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.628 DEBUG:boto:Params: {} 2023-11-02T18:19:09.629 DEBUG:boto:Token: None 2023-11-02T18:19:09.629 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj2 2023-11-02T18:19:09.630 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BypQY2/Gg8qCLSyJrN22fJJ+uGg= 2023-11-02T18:19:09.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BypQY2/Gg8qCLSyJrN22fJJ+uGg=', 'Content-Length': '0'} 2023-11-02T18:19:09.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab0615380ca086cd-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-11-02T18:19:09.634 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:19:09.635 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:19:09.635 DEBUG:boto:Method: GET 2023-11-02T18:19:09.636 DEBUG:boto:Path: /ibtljn-49/obj2 2023-11-02T18:19:09.636 DEBUG:boto:Data: 2023-11-02T18:19:09.636 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.637 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.638 DEBUG:boto:Params: {} 2023-11-02T18:19:09.638 DEBUG:boto:Token: None 2023-11-02T18:19:09.639 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj2 2023-11-02T18:19:09.639 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m98KkpBgHbusmhuzs6ci1ig0JsA= 2023-11-02T18:19:09.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m98KkpBgHbusmhuzs6ci1ig0JsA=', 'Content-Length': '0'} 2023-11-02T18:19:09.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5d16eb33a33a8ab-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.643 DEBUG:boto:path=/ibtljn-49/obj2 2023-11-02T18:19:09.644 DEBUG:boto:auth_path=/ibtljn-49/obj2 2023-11-02T18:19:09.644 DEBUG:boto:Method: GET 2023-11-02T18:19:09.645 DEBUG:boto:Path: /ibtljn-49/obj2 2023-11-02T18:19:09.645 DEBUG:boto:Data: 2023-11-02T18:19:09.645 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.646 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.646 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.647 DEBUG:boto:Params: {} 2023-11-02T18:19:09.647 DEBUG:boto:Token: None 2023-11-02T18:19:09.648 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj2 2023-11-02T18:19:09.648 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m98KkpBgHbusmhuzs6ci1ig0JsA= 2023-11-02T18:19:09.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m98KkpBgHbusmhuzs6ci1ig0JsA=', 'Content-Length': '0'} 2023-11-02T18:19:09.651 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4b02119bbdc53fc-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.652 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.653 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.653 DEBUG:boto:path=/ibtljn-49/?prefix=obj2 2023-11-02T18:19:09.654 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj2 2023-11-02T18:19:09.654 DEBUG:boto:Method: GET 2023-11-02T18:19:09.655 DEBUG:boto:Path: /ibtljn-49/?prefix=obj2 2023-11-02T18:19:09.655 DEBUG:boto:Data: 2023-11-02T18:19:09.655 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.656 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.656 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.657 DEBUG:boto:Params: {} 2023-11-02T18:19:09.657 DEBUG:boto:Token: None 2023-11-02T18:19:09.658 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.658 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs= 2023-11-02T18:19:09.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs=', 'Content-Length': '0'} 2023-11-02T18:19:09.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043b84f4563d88500-006543e81d-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.710 DEBUG:boto:b'ibtljn-49obj21000falseobj22023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.711 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.712 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.712 DEBUG:boto:path=/ibtljn-49/?prefix=obj2 2023-11-02T18:19:09.712 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj2 2023-11-02T18:19:09.713 DEBUG:boto:Method: GET 2023-11-02T18:19:09.713 DEBUG:boto:Path: /ibtljn-49/?prefix=obj2 2023-11-02T18:19:09.714 DEBUG:boto:Data: 2023-11-02T18:19:09.714 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.715 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.715 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.716 DEBUG:boto:Params: {} 2023-11-02T18:19:09.716 DEBUG:boto:Token: None 2023-11-02T18:19:09.717 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.717 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs= 2023-11-02T18:19:09.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs=', 'Content-Length': '0'} 2023-11-02T18:19:09.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025b4cb7c6a643176-006543e81d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.723 DEBUG:boto:b'ibtljn-49obj21000falseobj22023-11-02T18:16:46.042Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:19:09.724 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:19:09.725 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:19:09.725 DEBUG:boto:path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.726 DEBUG:boto:auth_path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.726 DEBUG:boto:Method: GET 2023-11-02T18:19:09.727 DEBUG:boto:Path: /ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.727 DEBUG:boto:Data: 2023-11-02T18:19:09.727 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.728 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.728 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.729 DEBUG:boto:Params: {} 2023-11-02T18:19:09.729 DEBUG:boto:Token: None 2023-11-02T18:19:09.730 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.730 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IIjQ33utYWjp+7l8g7Zfm5KLZfQ= 2023-11-02T18:19:09.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IIjQ33utYWjp+7l8g7Zfm5KLZfQ=', 'Content-Length': '0'} 2023-11-02T18:19:09.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008960fa6d563082f6-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.735 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:19:09.735 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:19:09.735 DEBUG:boto:path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.736 DEBUG:boto:auth_path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.736 DEBUG:boto:Method: GET 2023-11-02T18:19:09.737 DEBUG:boto:Path: /ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.737 DEBUG:boto:Data: 2023-11-02T18:19:09.738 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.739 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.739 DEBUG:boto:Params: {} 2023-11-02T18:19:09.739 DEBUG:boto:Token: None 2023-11-02T18:19:09.740 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.741 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IIjQ33utYWjp+7l8g7Zfm5KLZfQ= 2023-11-02T18:19:09.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IIjQ33utYWjp+7l8g7Zfm5KLZfQ=', 'Content-Length': '0'} 2023-11-02T18:19:09.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7050f1d1e359203-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.786 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:19:09.787 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:19:09.787 DEBUG:boto:path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.788 DEBUG:boto:auth_path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.788 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.789 DEBUG:boto:Path: /ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.789 DEBUG:boto:Data: 2023-11-02T18:19:09.789 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.790 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.791 DEBUG:boto:Params: {} 2023-11-02T18:19:09.791 DEBUG:boto:Token: None 2023-11-02T18:19:09.792 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.792 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4wnufAgyqzCERy6t0gfbzzpKzDk= 2023-11-02T18:19:09.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4wnufAgyqzCERy6t0gfbzzpKzDk=', 'Content-Length': '0'} 2023-11-02T18:19:09.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce9430750ba54534-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.796 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:19:09.797 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:19:09.797 DEBUG:boto:path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.798 DEBUG:boto:auth_path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.798 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.799 DEBUG:boto:Path: /ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.799 DEBUG:boto:Data: 2023-11-02T18:19:09.799 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.800 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.800 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.801 DEBUG:boto:Params: {} 2023-11-02T18:19:09.801 DEBUG:boto:Token: None 2023-11-02T18:19:09.802 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.802 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4wnufAgyqzCERy6t0gfbzzpKzDk= 2023-11-02T18:19:09.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4wnufAgyqzCERy6t0gfbzzpKzDk=', 'Content-Length': '0'} 2023-11-02T18:19:09.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000170dad9a29ca9775-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:19:09.807 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:19:09.807 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:19:09.808 DEBUG:boto:path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.808 DEBUG:boto:auth_path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.808 DEBUG:boto:Method: GET 2023-11-02T18:19:09.809 DEBUG:boto:Path: /ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.809 DEBUG:boto:Data: 2023-11-02T18:19:09.810 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.810 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.810 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.811 DEBUG:boto:Params: {} 2023-11-02T18:19:09.811 DEBUG:boto:Token: None 2023-11-02T18:19:09.812 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.812 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IIjQ33utYWjp+7l8g7Zfm5KLZfQ= 2023-11-02T18:19:09.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IIjQ33utYWjp+7l8g7Zfm5KLZfQ=', 'Content-Length': '0'} 2023-11-02T18:19:09.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7deadcc19506ac2-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.816 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:19:09.817 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:19:09.817 DEBUG:boto:path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.818 DEBUG:boto:auth_path=/ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.818 DEBUG:boto:Method: GET 2023-11-02T18:19:09.819 DEBUG:boto:Path: /ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.819 DEBUG:boto:Data: 2023-11-02T18:19:09.820 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.820 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.820 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.821 DEBUG:boto:Params: {} 2023-11-02T18:19:09.821 DEBUG:boto:Token: None 2023-11-02T18:19:09.822 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj3?versionId=null 2023-11-02T18:19:09.822 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IIjQ33utYWjp+7l8g7Zfm5KLZfQ= 2023-11-02T18:19:09.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IIjQ33utYWjp+7l8g7Zfm5KLZfQ=', 'Content-Length': '0'} 2023-11-02T18:19:09.826 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037fa1a6fa9d3b117-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.826 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.827 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.827 DEBUG:boto:path=/ibtljn-49/?prefix=obj3 2023-11-02T18:19:09.828 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj3 2023-11-02T18:19:09.828 DEBUG:boto:Method: GET 2023-11-02T18:19:09.829 DEBUG:boto:Path: /ibtljn-49/?prefix=obj3 2023-11-02T18:19:09.829 DEBUG:boto:Data: 2023-11-02T18:19:09.829 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.830 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.830 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.831 DEBUG:boto:Params: {} 2023-11-02T18:19:09.831 DEBUG:boto:Token: None 2023-11-02T18:19:09.832 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.832 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs= 2023-11-02T18:19:09.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs=', 'Content-Length': '0'} 2023-11-02T18:19:09.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec51a943193bfae7-006543e81d-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.878 DEBUG:boto:b'ibtljn-49obj31000falseobj32023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.879 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.880 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.880 DEBUG:boto:path=/ibtljn-49/?prefix=obj3 2023-11-02T18:19:09.880 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj3 2023-11-02T18:19:09.881 DEBUG:boto:Method: GET 2023-11-02T18:19:09.881 DEBUG:boto:Path: /ibtljn-49/?prefix=obj3 2023-11-02T18:19:09.882 DEBUG:boto:Data: 2023-11-02T18:19:09.882 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.883 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.883 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.884 DEBUG:boto:Params: {} 2023-11-02T18:19:09.884 DEBUG:boto:Token: None 2023-11-02T18:19:09.885 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs= 2023-11-02T18:19:09.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs=', 'Content-Length': '0'} 2023-11-02T18:19:09.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0a1f7a970911c75-006543e81d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.893 DEBUG:boto:b'ibtljn-49obj31000falseobj32023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.894 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:19:09.894 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:19:09.895 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.898 DEBUG:boto:Path: /ibtljn-49/obj3 2023-11-02T18:19:09.898 DEBUG:boto:Data: 2023-11-02T18:19:09.899 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.899 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.900 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.900 DEBUG:boto:Params: {} 2023-11-02T18:19:09.901 DEBUG:boto:Token: None 2023-11-02T18:19:09.901 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj3 2023-11-02T18:19:09.902 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SxiDVN8aWfSJJVvkxiLMNgzD3Kc= 2023-11-02T18:19:09.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SxiDVN8aWfSJJVvkxiLMNgzD3Kc=', 'Content-Length': '0'} 2023-11-02T18:19:09.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017173da491435862-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.906 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:19:09.906 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:19:09.907 DEBUG:boto:Method: HEAD 2023-11-02T18:19:09.907 DEBUG:boto:Path: /ibtljn-49/obj3 2023-11-02T18:19:09.908 DEBUG:boto:Data: 2023-11-02T18:19:09.908 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.909 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.909 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.909 DEBUG:boto:Params: {} 2023-11-02T18:19:09.910 DEBUG:boto:Token: None 2023-11-02T18:19:09.911 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj3 2023-11-02T18:19:09.911 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SxiDVN8aWfSJJVvkxiLMNgzD3Kc= 2023-11-02T18:19:09.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SxiDVN8aWfSJJVvkxiLMNgzD3Kc=', 'Content-Length': '0'} 2023-11-02T18:19:09.914 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5d86213cac06417-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-11-02T18:19:09.916 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:19:09.916 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:19:09.917 DEBUG:boto:Method: GET 2023-11-02T18:19:09.917 DEBUG:boto:Path: /ibtljn-49/obj3 2023-11-02T18:19:09.918 DEBUG:boto:Data: 2023-11-02T18:19:09.918 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.919 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.919 DEBUG:boto:Params: {} 2023-11-02T18:19:09.920 DEBUG:boto:Token: None 2023-11-02T18:19:09.920 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj3 2023-11-02T18:19:09.921 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZ0gl8TAKrqhSb5VCgNpcfCJeTU= 2023-11-02T18:19:09.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZ0gl8TAKrqhSb5VCgNpcfCJeTU=', 'Content-Length': '0'} 2023-11-02T18:19:09.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd0d716306eb3eb9-006543e81d-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.925 DEBUG:boto:path=/ibtljn-49/obj3 2023-11-02T18:19:09.926 DEBUG:boto:auth_path=/ibtljn-49/obj3 2023-11-02T18:19:09.926 DEBUG:boto:Method: GET 2023-11-02T18:19:09.927 DEBUG:boto:Path: /ibtljn-49/obj3 2023-11-02T18:19:09.927 DEBUG:boto:Data: 2023-11-02T18:19:09.928 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.928 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.929 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.929 DEBUG:boto:Params: {} 2023-11-02T18:19:09.930 DEBUG:boto:Token: None 2023-11-02T18:19:09.930 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/obj3 2023-11-02T18:19:09.931 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZ0gl8TAKrqhSb5VCgNpcfCJeTU= 2023-11-02T18:19:09.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZ0gl8TAKrqhSb5VCgNpcfCJeTU=', 'Content-Length': '0'} 2023-11-02T18:19:09.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de6ccbec77d06c80-006543e81d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.979 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.979 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.980 DEBUG:boto:path=/ibtljn-49/?prefix=obj3 2023-11-02T18:19:09.980 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj3 2023-11-02T18:19:09.981 DEBUG:boto:Method: GET 2023-11-02T18:19:09.981 DEBUG:boto:Path: /ibtljn-49/?prefix=obj3 2023-11-02T18:19:09.982 DEBUG:boto:Data: 2023-11-02T18:19:09.982 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.982 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:09.983 DEBUG:boto:Port: 8001 2023-11-02T18:19:09.983 DEBUG:boto:Params: {} 2023-11-02T18:19:09.984 DEBUG:boto:Token: None 2023-11-02T18:19:09.984 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs= 2023-11-02T18:19:09.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs=', 'Content-Length': '0'} 2023-11-02T18:19:09.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000673d438eb14924d4-006543e81d-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:09.991 DEBUG:boto:b'ibtljn-49obj31000falseobj32023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:09.992 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:09.993 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:09.993 DEBUG:boto:path=/ibtljn-49/?prefix=obj3 2023-11-02T18:19:09.993 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj3 2023-11-02T18:19:09.994 DEBUG:boto:Method: GET 2023-11-02T18:19:09.994 DEBUG:boto:Path: /ibtljn-49/?prefix=obj3 2023-11-02T18:19:09.995 DEBUG:boto:Data: 2023-11-02T18:19:09.995 DEBUG:boto:Headers: {} 2023-11-02T18:19:09.996 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:09.996 DEBUG:boto:Port: 8000 2023-11-02T18:19:09.997 DEBUG:boto:Params: {} 2023-11-02T18:19:09.997 DEBUG:boto:Token: None 2023-11-02T18:19:09.998 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:09 GMT /ibtljn-49/ 2023-11-02T18:19:09.998 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs= 2023-11-02T18:19:09.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kmhlEA5iONsFCQrJdm+6XKdy/bs=', 'Content-Length': '0'} 2023-11-02T18:19:10.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecec33d2dacfb926-006543e81d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.005 DEBUG:boto:b'ibtljn-49obj31000falseobj32023-11-02T18:16:46.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:19:10.007 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:19:10.007 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:19:10.007 DEBUG:boto:path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.008 DEBUG:boto:auth_path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.008 DEBUG:boto:Method: GET 2023-11-02T18:19:10.009 DEBUG:boto:Path: /ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.009 DEBUG:boto:Data: 2023-11-02T18:19:10.010 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.010 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.011 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.011 DEBUG:boto:Params: {} 2023-11-02T18:19:10.011 DEBUG:boto:Token: None 2023-11-02T18:19:10.012 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.013 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PuzeXP9PWxPYZMldOrAHeJmuo40= 2023-11-02T18:19:10.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PuzeXP9PWxPYZMldOrAHeJmuo40=', 'Content-Length': '0'} 2023-11-02T18:19:10.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f489fad9cb43ddfc-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.017 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:19:10.017 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:19:10.018 DEBUG:boto:path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.018 DEBUG:boto:auth_path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.019 DEBUG:boto:Method: GET 2023-11-02T18:19:10.019 DEBUG:boto:Path: /ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.019 DEBUG:boto:Data: 2023-11-02T18:19:10.020 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.020 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.021 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.021 DEBUG:boto:Params: {} 2023-11-02T18:19:10.022 DEBUG:boto:Token: None 2023-11-02T18:19:10.022 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.023 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PuzeXP9PWxPYZMldOrAHeJmuo40= 2023-11-02T18:19:10.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PuzeXP9PWxPYZMldOrAHeJmuo40=', 'Content-Length': '0'} 2023-11-02T18:19:10.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094462542415a305b-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.027 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:19:10.027 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:19:10.028 DEBUG:boto:path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.028 DEBUG:boto:auth_path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.029 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.029 DEBUG:boto:Path: /ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.030 DEBUG:boto:Data: 2023-11-02T18:19:10.030 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.031 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.031 DEBUG:boto:Params: {} 2023-11-02T18:19:10.032 DEBUG:boto:Token: None 2023-11-02T18:19:10.032 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.033 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p6Kay1EYJKvktk2OqNP90Brg9G0= 2023-11-02T18:19:10.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p6Kay1EYJKvktk2OqNP90Brg9G0=', 'Content-Length': '0'} 2023-11-02T18:19:10.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee9a19515383ccf0-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.036 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:19:10.037 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:19:10.037 DEBUG:boto:path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.039 DEBUG:boto:auth_path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.039 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.039 DEBUG:boto:Path: /ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.040 DEBUG:boto:Data: 2023-11-02T18:19:10.040 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.041 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.041 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.042 DEBUG:boto:Params: {} 2023-11-02T18:19:10.042 DEBUG:boto:Token: None 2023-11-02T18:19:10.043 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.043 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p6Kay1EYJKvktk2OqNP90Brg9G0= 2023-11-02T18:19:10.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p6Kay1EYJKvktk2OqNP90Brg9G0=', 'Content-Length': '0'} 2023-11-02T18:19:10.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050b3e130603c42e5-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:19:10.048 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:19:10.048 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:19:10.048 DEBUG:boto:path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.049 DEBUG:boto:auth_path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.049 DEBUG:boto:Method: GET 2023-11-02T18:19:10.050 DEBUG:boto:Path: /ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.050 DEBUG:boto:Data: 2023-11-02T18:19:10.051 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.051 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.052 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.052 DEBUG:boto:Params: {} 2023-11-02T18:19:10.052 DEBUG:boto:Token: None 2023-11-02T18:19:10.053 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.053 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PuzeXP9PWxPYZMldOrAHeJmuo40= 2023-11-02T18:19:10.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PuzeXP9PWxPYZMldOrAHeJmuo40=', 'Content-Length': '0'} 2023-11-02T18:19:10.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3af6907b3774068-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.098 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:19:10.099 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:19:10.099 DEBUG:boto:path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.100 DEBUG:boto:auth_path=/ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.100 DEBUG:boto:Method: GET 2023-11-02T18:19:10.101 DEBUG:boto:Path: /ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.101 DEBUG:boto:Data: 2023-11-02T18:19:10.102 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.102 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.102 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.103 DEBUG:boto:Params: {} 2023-11-02T18:19:10.103 DEBUG:boto:Token: None 2023-11-02T18:19:10.104 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj4?versionId=null 2023-11-02T18:19:10.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PuzeXP9PWxPYZMldOrAHeJmuo40= 2023-11-02T18:19:10.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PuzeXP9PWxPYZMldOrAHeJmuo40=', 'Content-Length': '0'} 2023-11-02T18:19:10.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d36a5f0d8b66cbbf-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.109 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.109 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.109 DEBUG:boto:path=/ibtljn-49/?prefix=obj4 2023-11-02T18:19:10.110 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj4 2023-11-02T18:19:10.110 DEBUG:boto:Method: GET 2023-11-02T18:19:10.111 DEBUG:boto:Path: /ibtljn-49/?prefix=obj4 2023-11-02T18:19:10.111 DEBUG:boto:Data: 2023-11-02T18:19:10.112 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.112 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.112 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.113 DEBUG:boto:Params: {} 2023-11-02T18:19:10.113 DEBUG:boto:Token: None 2023-11-02T18:19:10.114 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.114 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005b6b4ad25450884-006543e81e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.122 DEBUG:boto:b'ibtljn-49obj41000falseobj42023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.123 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.123 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.124 DEBUG:boto:path=/ibtljn-49/?prefix=obj4 2023-11-02T18:19:10.124 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj4 2023-11-02T18:19:10.125 DEBUG:boto:Method: GET 2023-11-02T18:19:10.125 DEBUG:boto:Path: /ibtljn-49/?prefix=obj4 2023-11-02T18:19:10.126 DEBUG:boto:Data: 2023-11-02T18:19:10.126 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.127 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.127 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.128 DEBUG:boto:Params: {} 2023-11-02T18:19:10.128 DEBUG:boto:Token: None 2023-11-02T18:19:10.129 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef8b211d702d72df-006543e81e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.136 DEBUG:boto:b'ibtljn-49obj41000falseobj42023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.137 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:19:10.137 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:19:10.138 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.138 DEBUG:boto:Path: /ibtljn-49/obj4 2023-11-02T18:19:10.139 DEBUG:boto:Data: 2023-11-02T18:19:10.139 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.140 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.140 DEBUG:boto:Params: {} 2023-11-02T18:19:10.141 DEBUG:boto:Token: None 2023-11-02T18:19:10.141 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj4 2023-11-02T18:19:10.142 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9/caI81Wpe64DBi80wC3f+q1PjE= 2023-11-02T18:19:10.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9/caI81Wpe64DBi80wC3f+q1PjE=', 'Content-Length': '0'} 2023-11-02T18:19:10.145 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000093cf31f5c07b0bb-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.146 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:19:10.146 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:19:10.147 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.147 DEBUG:boto:Path: /ibtljn-49/obj4 2023-11-02T18:19:10.148 DEBUG:boto:Data: 2023-11-02T18:19:10.148 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.149 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.149 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.149 DEBUG:boto:Params: {} 2023-11-02T18:19:10.150 DEBUG:boto:Token: None 2023-11-02T18:19:10.150 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj4 2023-11-02T18:19:10.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9/caI81Wpe64DBi80wC3f+q1PjE= 2023-11-02T18:19:10.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9/caI81Wpe64DBi80wC3f+q1PjE=', 'Content-Length': '0'} 2023-11-02T18:19:10.154 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095350792c436ce15-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-11-02T18:19:10.155 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:19:10.156 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:19:10.156 DEBUG:boto:Method: GET 2023-11-02T18:19:10.157 DEBUG:boto:Path: /ibtljn-49/obj4 2023-11-02T18:19:10.157 DEBUG:boto:Data: 2023-11-02T18:19:10.157 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.158 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.158 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.159 DEBUG:boto:Params: {} 2023-11-02T18:19:10.159 DEBUG:boto:Token: None 2023-11-02T18:19:10.160 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj4 2023-11-02T18:19:10.160 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a1WiNG3ET7UDlOy126zlJeSRpwQ= 2023-11-02T18:19:10.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a1WiNG3ET7UDlOy126zlJeSRpwQ=', 'Content-Length': '0'} 2023-11-02T18:19:10.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bde797ce67c808e-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.164 DEBUG:boto:path=/ibtljn-49/obj4 2023-11-02T18:19:10.165 DEBUG:boto:auth_path=/ibtljn-49/obj4 2023-11-02T18:19:10.165 DEBUG:boto:Method: GET 2023-11-02T18:19:10.166 DEBUG:boto:Path: /ibtljn-49/obj4 2023-11-02T18:19:10.166 DEBUG:boto:Data: 2023-11-02T18:19:10.166 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.167 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.167 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.168 DEBUG:boto:Params: {} 2023-11-02T18:19:10.168 DEBUG:boto:Token: None 2023-11-02T18:19:10.170 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj4 2023-11-02T18:19:10.170 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a1WiNG3ET7UDlOy126zlJeSRpwQ= 2023-11-02T18:19:10.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a1WiNG3ET7UDlOy126zlJeSRpwQ=', 'Content-Length': '0'} 2023-11-02T18:19:10.174 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:16:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066053ba72b0b6429-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.218 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.219 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.220 DEBUG:boto:path=/ibtljn-49/?prefix=obj4 2023-11-02T18:19:10.220 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj4 2023-11-02T18:19:10.220 DEBUG:boto:Method: GET 2023-11-02T18:19:10.221 DEBUG:boto:Path: /ibtljn-49/?prefix=obj4 2023-11-02T18:19:10.221 DEBUG:boto:Data: 2023-11-02T18:19:10.222 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.222 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.222 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.223 DEBUG:boto:Params: {} 2023-11-02T18:19:10.223 DEBUG:boto:Token: None 2023-11-02T18:19:10.224 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.224 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002527f6d930df0f22-006543e81e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.231 DEBUG:boto:b'ibtljn-49obj41000falseobj42023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.231 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.232 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.232 DEBUG:boto:path=/ibtljn-49/?prefix=obj4 2023-11-02T18:19:10.233 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj4 2023-11-02T18:19:10.233 DEBUG:boto:Method: GET 2023-11-02T18:19:10.234 DEBUG:boto:Path: /ibtljn-49/?prefix=obj4 2023-11-02T18:19:10.234 DEBUG:boto:Data: 2023-11-02T18:19:10.234 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.235 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.235 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.236 DEBUG:boto:Params: {} 2023-11-02T18:19:10.236 DEBUG:boto:Token: None 2023-11-02T18:19:10.237 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.237 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b1c52c6348d9977-006543e81e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.243 DEBUG:boto:b'ibtljn-49obj41000falseobj42023-11-02T18:16:46.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-11-02T18:19:10.245 DEBUG:boto:path=/ibtljn-49/obj5 2023-11-02T18:19:10.245 DEBUG:boto:auth_path=/ibtljn-49/obj5 2023-11-02T18:19:10.245 DEBUG:boto:path=/ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.246 DEBUG:boto:auth_path=/ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.246 DEBUG:boto:Method: GET 2023-11-02T18:19:10.247 DEBUG:boto:Path: /ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.247 DEBUG:boto:Data: 2023-11-02T18:19:10.248 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.248 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.249 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.249 DEBUG:boto:Params: {} 2023-11-02T18:19:10.249 DEBUG:boto:Token: None 2023-11-02T18:19:10.250 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.251 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l+PcYSOV24asqYruDR4nqsl8t5U= 2023-11-02T18:19:10.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l+PcYSOV24asqYruDR4nqsl8t5U=', 'Content-Length': '0'} 2023-11-02T18:19:10.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed314897c86847b9-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.298 DEBUG:boto:path=/ibtljn-49/obj5 2023-11-02T18:19:10.299 DEBUG:boto:auth_path=/ibtljn-49/obj5 2023-11-02T18:19:10.300 DEBUG:boto:path=/ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.300 DEBUG:boto:auth_path=/ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.301 DEBUG:boto:Method: GET 2023-11-02T18:19:10.301 DEBUG:boto:Path: /ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.301 DEBUG:boto:Data: 2023-11-02T18:19:10.302 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.302 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.303 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.303 DEBUG:boto:Params: {} 2023-11-02T18:19:10.306 DEBUG:boto:Token: None 2023-11-02T18:19:10.307 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.307 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l+PcYSOV24asqYruDR4nqsl8t5U= 2023-11-02T18:19:10.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l+PcYSOV24asqYruDR4nqsl8t5U=', 'Content-Length': '0'} 2023-11-02T18:19:10.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a27d72ffaa1eb037-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.311 DEBUG:boto:path=/ibtljn-49/obj5 2023-11-02T18:19:10.312 DEBUG:boto:auth_path=/ibtljn-49/obj5 2023-11-02T18:19:10.312 DEBUG:boto:path=/ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.313 DEBUG:boto:auth_path=/ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.313 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.314 DEBUG:boto:Path: /ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.314 DEBUG:boto:Data: 2023-11-02T18:19:10.314 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.315 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.315 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.316 DEBUG:boto:Params: {} 2023-11-02T18:19:10.316 DEBUG:boto:Token: None 2023-11-02T18:19:10.317 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.317 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:doye9W8W+X2JDRCfM/9Wh3Yu/JA= 2023-11-02T18:19:10.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:doye9W8W+X2JDRCfM/9Wh3Yu/JA=', 'Content-Length': '0'} 2023-11-02T18:19:10.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047e95cd3b3f65ffe-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.322 DEBUG:boto:path=/ibtljn-49/obj5 2023-11-02T18:19:10.322 DEBUG:boto:auth_path=/ibtljn-49/obj5 2023-11-02T18:19:10.323 DEBUG:boto:path=/ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.323 DEBUG:boto:auth_path=/ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.324 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.324 DEBUG:boto:Path: /ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.324 DEBUG:boto:Data: 2023-11-02T18:19:10.325 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.325 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.326 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.326 DEBUG:boto:Params: {} 2023-11-02T18:19:10.326 DEBUG:boto:Token: None 2023-11-02T18:19:10.327 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.327 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:doye9W8W+X2JDRCfM/9Wh3Yu/JA= 2023-11-02T18:19:10.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:doye9W8W+X2JDRCfM/9Wh3Yu/JA=', 'Content-Length': '0'} 2023-11-02T18:19:10.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000362b2427bbf9a66c-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-11-02T18:19:10.332 DEBUG:boto:path=/ibtljn-49/obj5 2023-11-02T18:19:10.332 DEBUG:boto:auth_path=/ibtljn-49/obj5 2023-11-02T18:19:10.333 DEBUG:boto:path=/ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.333 DEBUG:boto:auth_path=/ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.334 DEBUG:boto:Method: GET 2023-11-02T18:19:10.334 DEBUG:boto:Path: /ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.335 DEBUG:boto:Data: 2023-11-02T18:19:10.335 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.336 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.336 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.336 DEBUG:boto:Params: {} 2023-11-02T18:19:10.337 DEBUG:boto:Token: None 2023-11-02T18:19:10.337 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.338 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l+PcYSOV24asqYruDR4nqsl8t5U= 2023-11-02T18:19:10.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l+PcYSOV24asqYruDR4nqsl8t5U=', 'Content-Length': '0'} 2023-11-02T18:19:10.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097e7eb81168f7da5-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.343 DEBUG:boto:path=/ibtljn-49/obj5 2023-11-02T18:19:10.343 DEBUG:boto:auth_path=/ibtljn-49/obj5 2023-11-02T18:19:10.344 DEBUG:boto:path=/ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.344 DEBUG:boto:auth_path=/ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.345 DEBUG:boto:Method: GET 2023-11-02T18:19:10.345 DEBUG:boto:Path: /ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.345 DEBUG:boto:Data: 2023-11-02T18:19:10.346 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.346 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.347 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.347 DEBUG:boto:Params: {} 2023-11-02T18:19:10.348 DEBUG:boto:Token: None 2023-11-02T18:19:10.348 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj5?versionId=null 2023-11-02T18:19:10.349 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l+PcYSOV24asqYruDR4nqsl8t5U= 2023-11-02T18:19:10.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l+PcYSOV24asqYruDR4nqsl8t5U=', 'Content-Length': '0'} 2023-11-02T18:19:10.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a0e4515b57b4b45-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.394 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.395 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.395 DEBUG:boto:path=/ibtljn-49/?prefix=obj5 2023-11-02T18:19:10.396 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj5 2023-11-02T18:19:10.396 DEBUG:boto:Method: GET 2023-11-02T18:19:10.396 DEBUG:boto:Path: /ibtljn-49/?prefix=obj5 2023-11-02T18:19:10.397 DEBUG:boto:Data: 2023-11-02T18:19:10.397 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.398 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.398 DEBUG:boto:Params: {} 2023-11-02T18:19:10.399 DEBUG:boto:Token: None 2023-11-02T18:19:10.399 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e38f37ba10f98f7-006543e81e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.406 DEBUG:boto:b'ibtljn-49obj51000falseobj52023-11-02T18:18:30.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.407 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.408 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.408 DEBUG:boto:path=/ibtljn-49/?prefix=obj5 2023-11-02T18:19:10.408 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj5 2023-11-02T18:19:10.409 DEBUG:boto:Method: GET 2023-11-02T18:19:10.409 DEBUG:boto:Path: /ibtljn-49/?prefix=obj5 2023-11-02T18:19:10.410 DEBUG:boto:Data: 2023-11-02T18:19:10.410 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.410 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.411 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.411 DEBUG:boto:Params: {} 2023-11-02T18:19:10.412 DEBUG:boto:Token: None 2023-11-02T18:19:10.412 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.413 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b77b0140fb57c61-006543e81e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.419 DEBUG:boto:b'ibtljn-49obj51000falseobj52023-11-02T18:18:30.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.419 DEBUG:boto:path=/ibtljn-49/obj5 2023-11-02T18:19:10.420 DEBUG:boto:auth_path=/ibtljn-49/obj5 2023-11-02T18:19:10.420 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.420 DEBUG:boto:Path: /ibtljn-49/obj5 2023-11-02T18:19:10.421 DEBUG:boto:Data: 2023-11-02T18:19:10.421 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.421 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.422 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.422 DEBUG:boto:Params: {} 2023-11-02T18:19:10.423 DEBUG:boto:Token: None 2023-11-02T18:19:10.423 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj5 2023-11-02T18:19:10.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RAoAz9E/nO/Le56e980wt08WD50= 2023-11-02T18:19:10.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RAoAz9E/nO/Le56e980wt08WD50=', 'Content-Length': '0'} 2023-11-02T18:19:10.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074ebd9147c43fd99-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.428 DEBUG:boto:path=/ibtljn-49/obj5 2023-11-02T18:19:10.428 DEBUG:boto:auth_path=/ibtljn-49/obj5 2023-11-02T18:19:10.428 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.430 DEBUG:boto:Path: /ibtljn-49/obj5 2023-11-02T18:19:10.430 DEBUG:boto:Data: 2023-11-02T18:19:10.431 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.431 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.431 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.432 DEBUG:boto:Params: {} 2023-11-02T18:19:10.432 DEBUG:boto:Token: None 2023-11-02T18:19:10.432 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj5 2023-11-02T18:19:10.433 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RAoAz9E/nO/Le56e980wt08WD50= 2023-11-02T18:19:10.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RAoAz9E/nO/Le56e980wt08WD50=', 'Content-Length': '0'} 2023-11-02T18:19:10.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002fe8d021fbd0d4c-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-11-02T18:19:10.437 DEBUG:boto:path=/ibtljn-49/obj5 2023-11-02T18:19:10.438 DEBUG:boto:auth_path=/ibtljn-49/obj5 2023-11-02T18:19:10.438 DEBUG:boto:Method: GET 2023-11-02T18:19:10.438 DEBUG:boto:Path: /ibtljn-49/obj5 2023-11-02T18:19:10.439 DEBUG:boto:Data: 2023-11-02T18:19:10.439 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.440 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.440 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.440 DEBUG:boto:Params: {} 2023-11-02T18:19:10.441 DEBUG:boto:Token: None 2023-11-02T18:19:10.441 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj5 2023-11-02T18:19:10.442 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hHX0z1/eqpes4QBJPpkXKkYdg4E= 2023-11-02T18:19:10.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hHX0z1/eqpes4QBJPpkXKkYdg4E=', 'Content-Length': '0'} 2023-11-02T18:19:10.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ecab78e42876c56-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.486 DEBUG:boto:path=/ibtljn-49/obj5 2023-11-02T18:19:10.487 DEBUG:boto:auth_path=/ibtljn-49/obj5 2023-11-02T18:19:10.487 DEBUG:boto:Method: GET 2023-11-02T18:19:10.488 DEBUG:boto:Path: /ibtljn-49/obj5 2023-11-02T18:19:10.488 DEBUG:boto:Data: 2023-11-02T18:19:10.489 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.489 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.490 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.490 DEBUG:boto:Params: {} 2023-11-02T18:19:10.491 DEBUG:boto:Token: None 2023-11-02T18:19:10.491 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj5 2023-11-02T18:19:10.492 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hHX0z1/eqpes4QBJPpkXKkYdg4E= 2023-11-02T18:19:10.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hHX0z1/eqpes4QBJPpkXKkYdg4E=', 'Content-Length': '0'} 2023-11-02T18:19:10.495 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000daf40fa5ae91e910-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.496 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.497 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.497 DEBUG:boto:path=/ibtljn-49/?prefix=obj5 2023-11-02T18:19:10.498 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj5 2023-11-02T18:19:10.498 DEBUG:boto:Method: GET 2023-11-02T18:19:10.499 DEBUG:boto:Path: /ibtljn-49/?prefix=obj5 2023-11-02T18:19:10.499 DEBUG:boto:Data: 2023-11-02T18:19:10.499 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.500 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.500 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.501 DEBUG:boto:Params: {} 2023-11-02T18:19:10.501 DEBUG:boto:Token: None 2023-11-02T18:19:10.502 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.503 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a4d409722639584-006543e81e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.508 DEBUG:boto:b'ibtljn-49obj51000falseobj52023-11-02T18:18:30.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.509 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.510 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.510 DEBUG:boto:path=/ibtljn-49/?prefix=obj5 2023-11-02T18:19:10.511 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj5 2023-11-02T18:19:10.511 DEBUG:boto:Method: GET 2023-11-02T18:19:10.512 DEBUG:boto:Path: /ibtljn-49/?prefix=obj5 2023-11-02T18:19:10.512 DEBUG:boto:Data: 2023-11-02T18:19:10.513 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.513 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.514 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.514 DEBUG:boto:Params: {} 2023-11-02T18:19:10.515 DEBUG:boto:Token: None 2023-11-02T18:19:10.515 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.516 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c885d0467068c7e-006543e81e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.566 DEBUG:boto:b'ibtljn-49obj51000falseobj52023-11-02T18:18:30.685Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-11-02T18:19:10.568 DEBUG:boto:path=/ibtljn-49/obj6 2023-11-02T18:19:10.568 DEBUG:boto:auth_path=/ibtljn-49/obj6 2023-11-02T18:19:10.569 DEBUG:boto:path=/ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.569 DEBUG:boto:auth_path=/ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.569 DEBUG:boto:Method: GET 2023-11-02T18:19:10.570 DEBUG:boto:Path: /ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.570 DEBUG:boto:Data: 2023-11-02T18:19:10.571 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.571 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.572 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.572 DEBUG:boto:Params: {} 2023-11-02T18:19:10.572 DEBUG:boto:Token: None 2023-11-02T18:19:10.573 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.573 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:smjedADtOVzEzQRPOrwXr35MS7g= 2023-11-02T18:19:10.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:smjedADtOVzEzQRPOrwXr35MS7g=', 'Content-Length': '0'} 2023-11-02T18:19:10.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec8aeca77745e3af-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.577 DEBUG:boto:path=/ibtljn-49/obj6 2023-11-02T18:19:10.578 DEBUG:boto:auth_path=/ibtljn-49/obj6 2023-11-02T18:19:10.578 DEBUG:boto:path=/ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.579 DEBUG:boto:auth_path=/ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.579 DEBUG:boto:Method: GET 2023-11-02T18:19:10.579 DEBUG:boto:Path: /ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.580 DEBUG:boto:Data: 2023-11-02T18:19:10.580 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.581 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.581 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.581 DEBUG:boto:Params: {} 2023-11-02T18:19:10.582 DEBUG:boto:Token: None 2023-11-02T18:19:10.583 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.583 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:smjedADtOVzEzQRPOrwXr35MS7g= 2023-11-02T18:19:10.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:smjedADtOVzEzQRPOrwXr35MS7g=', 'Content-Length': '0'} 2023-11-02T18:19:10.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068968bd945162b3f-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.587 DEBUG:boto:path=/ibtljn-49/obj6 2023-11-02T18:19:10.587 DEBUG:boto:auth_path=/ibtljn-49/obj6 2023-11-02T18:19:10.588 DEBUG:boto:path=/ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.588 DEBUG:boto:auth_path=/ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.588 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.589 DEBUG:boto:Path: /ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.589 DEBUG:boto:Data: 2023-11-02T18:19:10.590 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.591 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.591 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.592 DEBUG:boto:Params: {} 2023-11-02T18:19:10.592 DEBUG:boto:Token: None 2023-11-02T18:19:10.592 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Uwj3DvUx/9j8mHAOjl1ItSdZOos= 2023-11-02T18:19:10.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Uwj3DvUx/9j8mHAOjl1ItSdZOos=', 'Content-Length': '0'} 2023-11-02T18:19:10.596 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fa89e686fcfb304-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.596 DEBUG:boto:path=/ibtljn-49/obj6 2023-11-02T18:19:10.597 DEBUG:boto:auth_path=/ibtljn-49/obj6 2023-11-02T18:19:10.597 DEBUG:boto:path=/ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.598 DEBUG:boto:auth_path=/ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.598 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.599 DEBUG:boto:Path: /ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.599 DEBUG:boto:Data: 2023-11-02T18:19:10.599 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.600 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.600 DEBUG:boto:Params: {} 2023-11-02T18:19:10.601 DEBUG:boto:Token: None 2023-11-02T18:19:10.601 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.602 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Uwj3DvUx/9j8mHAOjl1ItSdZOos= 2023-11-02T18:19:10.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Uwj3DvUx/9j8mHAOjl1ItSdZOos=', 'Content-Length': '0'} 2023-11-02T18:19:10.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f136eef9d62ecbd3-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-11-02T18:19:10.607 DEBUG:boto:path=/ibtljn-49/obj6 2023-11-02T18:19:10.607 DEBUG:boto:auth_path=/ibtljn-49/obj6 2023-11-02T18:19:10.608 DEBUG:boto:path=/ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.608 DEBUG:boto:auth_path=/ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.609 DEBUG:boto:Method: GET 2023-11-02T18:19:10.609 DEBUG:boto:Path: /ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.609 DEBUG:boto:Data: 2023-11-02T18:19:10.610 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.610 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.611 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.611 DEBUG:boto:Params: {} 2023-11-02T18:19:10.611 DEBUG:boto:Token: None 2023-11-02T18:19:10.612 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:smjedADtOVzEzQRPOrwXr35MS7g= 2023-11-02T18:19:10.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:smjedADtOVzEzQRPOrwXr35MS7g=', 'Content-Length': '0'} 2023-11-02T18:19:10.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f7223985f36abab-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.617 DEBUG:boto:path=/ibtljn-49/obj6 2023-11-02T18:19:10.618 DEBUG:boto:auth_path=/ibtljn-49/obj6 2023-11-02T18:19:10.618 DEBUG:boto:path=/ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.619 DEBUG:boto:auth_path=/ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.619 DEBUG:boto:Method: GET 2023-11-02T18:19:10.620 DEBUG:boto:Path: /ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.621 DEBUG:boto:Data: 2023-11-02T18:19:10.621 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.622 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.622 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.623 DEBUG:boto:Params: {} 2023-11-02T18:19:10.623 DEBUG:boto:Token: None 2023-11-02T18:19:10.624 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj6?versionId=null 2023-11-02T18:19:10.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:smjedADtOVzEzQRPOrwXr35MS7g= 2023-11-02T18:19:10.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:smjedADtOVzEzQRPOrwXr35MS7g=', 'Content-Length': '0'} 2023-11-02T18:19:10.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f89781a9632fb37-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.629 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.630 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.630 DEBUG:boto:path=/ibtljn-49/?prefix=obj6 2023-11-02T18:19:10.631 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj6 2023-11-02T18:19:10.631 DEBUG:boto:Method: GET 2023-11-02T18:19:10.631 DEBUG:boto:Path: /ibtljn-49/?prefix=obj6 2023-11-02T18:19:10.632 DEBUG:boto:Data: 2023-11-02T18:19:10.632 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.633 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.633 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.633 DEBUG:boto:Params: {} 2023-11-02T18:19:10.634 DEBUG:boto:Token: None 2023-11-02T18:19:10.634 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.635 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5050e8469e7cd43-006543e81e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.682 DEBUG:boto:b'ibtljn-49obj61000falseobj62023-11-02T18:18:30.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.683 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.683 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.684 DEBUG:boto:path=/ibtljn-49/?prefix=obj6 2023-11-02T18:19:10.684 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj6 2023-11-02T18:19:10.685 DEBUG:boto:Method: GET 2023-11-02T18:19:10.685 DEBUG:boto:Path: /ibtljn-49/?prefix=obj6 2023-11-02T18:19:10.685 DEBUG:boto:Data: 2023-11-02T18:19:10.686 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.687 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.688 DEBUG:boto:Params: {} 2023-11-02T18:19:10.688 DEBUG:boto:Token: None 2023-11-02T18:19:10.689 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.689 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b48423a14c70b789-006543e81e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.696 DEBUG:boto:b'ibtljn-49obj61000falseobj62023-11-02T18:18:30.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.697 DEBUG:boto:path=/ibtljn-49/obj6 2023-11-02T18:19:10.697 DEBUG:boto:auth_path=/ibtljn-49/obj6 2023-11-02T18:19:10.697 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.698 DEBUG:boto:Path: /ibtljn-49/obj6 2023-11-02T18:19:10.698 DEBUG:boto:Data: 2023-11-02T18:19:10.699 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.699 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.699 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.700 DEBUG:boto:Params: {} 2023-11-02T18:19:10.700 DEBUG:boto:Token: None 2023-11-02T18:19:10.701 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj6 2023-11-02T18:19:10.701 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:At1R3tt6IxPoENkOfsduT3/Atio= 2023-11-02T18:19:10.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:At1R3tt6IxPoENkOfsduT3/Atio=', 'Content-Length': '0'} 2023-11-02T18:19:10.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ffc68e968a63274-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.705 DEBUG:boto:path=/ibtljn-49/obj6 2023-11-02T18:19:10.705 DEBUG:boto:auth_path=/ibtljn-49/obj6 2023-11-02T18:19:10.706 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.706 DEBUG:boto:Path: /ibtljn-49/obj6 2023-11-02T18:19:10.707 DEBUG:boto:Data: 2023-11-02T18:19:10.707 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.707 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.708 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.708 DEBUG:boto:Params: {} 2023-11-02T18:19:10.709 DEBUG:boto:Token: None 2023-11-02T18:19:10.709 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj6 2023-11-02T18:19:10.710 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:At1R3tt6IxPoENkOfsduT3/Atio= 2023-11-02T18:19:10.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:At1R3tt6IxPoENkOfsduT3/Atio=', 'Content-Length': '0'} 2023-11-02T18:19:10.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000469b8a3c80b43a02-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-11-02T18:19:10.714 DEBUG:boto:path=/ibtljn-49/obj6 2023-11-02T18:19:10.714 DEBUG:boto:auth_path=/ibtljn-49/obj6 2023-11-02T18:19:10.715 DEBUG:boto:Method: GET 2023-11-02T18:19:10.715 DEBUG:boto:Path: /ibtljn-49/obj6 2023-11-02T18:19:10.716 DEBUG:boto:Data: 2023-11-02T18:19:10.716 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.716 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.717 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.717 DEBUG:boto:Params: {} 2023-11-02T18:19:10.718 DEBUG:boto:Token: None 2023-11-02T18:19:10.718 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj6 2023-11-02T18:19:10.719 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/TAAHlMo1iqn1PpxADnNU3KrBtM= 2023-11-02T18:19:10.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/TAAHlMo1iqn1PpxADnNU3KrBtM=', 'Content-Length': '0'} 2023-11-02T18:19:10.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5a7790f155f6909-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.723 DEBUG:boto:path=/ibtljn-49/obj6 2023-11-02T18:19:10.723 DEBUG:boto:auth_path=/ibtljn-49/obj6 2023-11-02T18:19:10.724 DEBUG:boto:Method: GET 2023-11-02T18:19:10.724 DEBUG:boto:Path: /ibtljn-49/obj6 2023-11-02T18:19:10.724 DEBUG:boto:Data: 2023-11-02T18:19:10.725 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.725 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.726 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.726 DEBUG:boto:Params: {} 2023-11-02T18:19:10.727 DEBUG:boto:Token: None 2023-11-02T18:19:10.727 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj6 2023-11-02T18:19:10.728 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/TAAHlMo1iqn1PpxADnNU3KrBtM= 2023-11-02T18:19:10.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/TAAHlMo1iqn1PpxADnNU3KrBtM=', 'Content-Length': '0'} 2023-11-02T18:19:10.731 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000008d7c968fa74938-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.778 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.779 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.779 DEBUG:boto:path=/ibtljn-49/?prefix=obj6 2023-11-02T18:19:10.780 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj6 2023-11-02T18:19:10.780 DEBUG:boto:Method: GET 2023-11-02T18:19:10.781 DEBUG:boto:Path: /ibtljn-49/?prefix=obj6 2023-11-02T18:19:10.781 DEBUG:boto:Data: 2023-11-02T18:19:10.782 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.782 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.782 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.783 DEBUG:boto:Params: {} 2023-11-02T18:19:10.783 DEBUG:boto:Token: None 2023-11-02T18:19:10.784 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.784 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fa6d60e45c41047-006543e81e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.792 DEBUG:boto:b'ibtljn-49obj61000falseobj62023-11-02T18:18:30.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.792 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.793 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.793 DEBUG:boto:path=/ibtljn-49/?prefix=obj6 2023-11-02T18:19:10.794 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj6 2023-11-02T18:19:10.794 DEBUG:boto:Method: GET 2023-11-02T18:19:10.794 DEBUG:boto:Path: /ibtljn-49/?prefix=obj6 2023-11-02T18:19:10.795 DEBUG:boto:Data: 2023-11-02T18:19:10.795 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.796 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.796 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.796 DEBUG:boto:Params: {} 2023-11-02T18:19:10.797 DEBUG:boto:Token: None 2023-11-02T18:19:10.797 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.798 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094fcaa385c268232-006543e81e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.804 DEBUG:boto:b'ibtljn-49obj61000falseobj62023-11-02T18:18:30.710Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-11-02T18:19:10.805 DEBUG:boto:path=/ibtljn-49/obj7 2023-11-02T18:19:10.806 DEBUG:boto:auth_path=/ibtljn-49/obj7 2023-11-02T18:19:10.806 DEBUG:boto:path=/ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.806 DEBUG:boto:auth_path=/ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.807 DEBUG:boto:Method: GET 2023-11-02T18:19:10.807 DEBUG:boto:Path: /ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.808 DEBUG:boto:Data: 2023-11-02T18:19:10.808 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.809 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.809 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.810 DEBUG:boto:Params: {} 2023-11-02T18:19:10.810 DEBUG:boto:Token: None 2023-11-02T18:19:10.811 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.811 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W+l+2NgYsk2db7+VuhmglJ1/izY= 2023-11-02T18:19:10.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W+l+2NgYsk2db7+VuhmglJ1/izY=', 'Content-Length': '0'} 2023-11-02T18:19:10.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5d56150207cc86a-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.816 DEBUG:boto:path=/ibtljn-49/obj7 2023-11-02T18:19:10.816 DEBUG:boto:auth_path=/ibtljn-49/obj7 2023-11-02T18:19:10.816 DEBUG:boto:path=/ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.817 DEBUG:boto:auth_path=/ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.817 DEBUG:boto:Method: GET 2023-11-02T18:19:10.818 DEBUG:boto:Path: /ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.818 DEBUG:boto:Data: 2023-11-02T18:19:10.819 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.819 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.819 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.820 DEBUG:boto:Params: {} 2023-11-02T18:19:10.820 DEBUG:boto:Token: None 2023-11-02T18:19:10.821 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.821 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W+l+2NgYsk2db7+VuhmglJ1/izY= 2023-11-02T18:19:10.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W+l+2NgYsk2db7+VuhmglJ1/izY=', 'Content-Length': '0'} 2023-11-02T18:19:10.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c6591fc4f111953-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.826 DEBUG:boto:path=/ibtljn-49/obj7 2023-11-02T18:19:10.827 DEBUG:boto:auth_path=/ibtljn-49/obj7 2023-11-02T18:19:10.827 DEBUG:boto:path=/ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.827 DEBUG:boto:auth_path=/ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.828 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.828 DEBUG:boto:Path: /ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.829 DEBUG:boto:Data: 2023-11-02T18:19:10.829 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.829 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.830 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.830 DEBUG:boto:Params: {} 2023-11-02T18:19:10.831 DEBUG:boto:Token: None 2023-11-02T18:19:10.831 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.831 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hT+hv/O7DfezL4TDG0PJCAiOFZ4= 2023-11-02T18:19:10.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hT+hv/O7DfezL4TDG0PJCAiOFZ4=', 'Content-Length': '0'} 2023-11-02T18:19:10.835 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000beac4252c3be75c3-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.835 DEBUG:boto:path=/ibtljn-49/obj7 2023-11-02T18:19:10.836 DEBUG:boto:auth_path=/ibtljn-49/obj7 2023-11-02T18:19:10.836 DEBUG:boto:path=/ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.837 DEBUG:boto:auth_path=/ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.837 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.837 DEBUG:boto:Path: /ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.838 DEBUG:boto:Data: 2023-11-02T18:19:10.838 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.839 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.839 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.839 DEBUG:boto:Params: {} 2023-11-02T18:19:10.840 DEBUG:boto:Token: None 2023-11-02T18:19:10.840 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.841 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hT+hv/O7DfezL4TDG0PJCAiOFZ4= 2023-11-02T18:19:10.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hT+hv/O7DfezL4TDG0PJCAiOFZ4=', 'Content-Length': '0'} 2023-11-02T18:19:10.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd2ab460a8a8e854-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-11-02T18:19:10.845 DEBUG:boto:path=/ibtljn-49/obj7 2023-11-02T18:19:10.845 DEBUG:boto:auth_path=/ibtljn-49/obj7 2023-11-02T18:19:10.846 DEBUG:boto:path=/ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.846 DEBUG:boto:auth_path=/ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.847 DEBUG:boto:Method: GET 2023-11-02T18:19:10.847 DEBUG:boto:Path: /ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.848 DEBUG:boto:Data: 2023-11-02T18:19:10.848 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.848 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.849 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.849 DEBUG:boto:Params: {} 2023-11-02T18:19:10.850 DEBUG:boto:Token: None 2023-11-02T18:19:10.850 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.851 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W+l+2NgYsk2db7+VuhmglJ1/izY= 2023-11-02T18:19:10.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W+l+2NgYsk2db7+VuhmglJ1/izY=', 'Content-Length': '0'} 2023-11-02T18:19:10.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b1244b74c72315f-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.898 DEBUG:boto:path=/ibtljn-49/obj7 2023-11-02T18:19:10.899 DEBUG:boto:auth_path=/ibtljn-49/obj7 2023-11-02T18:19:10.899 DEBUG:boto:path=/ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.900 DEBUG:boto:auth_path=/ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.900 DEBUG:boto:Method: GET 2023-11-02T18:19:10.901 DEBUG:boto:Path: /ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.901 DEBUG:boto:Data: 2023-11-02T18:19:10.902 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.902 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.902 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.903 DEBUG:boto:Params: {} 2023-11-02T18:19:10.903 DEBUG:boto:Token: None 2023-11-02T18:19:10.904 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj7?versionId=null 2023-11-02T18:19:10.904 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W+l+2NgYsk2db7+VuhmglJ1/izY= 2023-11-02T18:19:10.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W+l+2NgYsk2db7+VuhmglJ1/izY=', 'Content-Length': '0'} 2023-11-02T18:19:10.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000581b2a88de9fc792-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.909 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.909 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.910 DEBUG:boto:path=/ibtljn-49/?prefix=obj7 2023-11-02T18:19:10.910 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj7 2023-11-02T18:19:10.911 DEBUG:boto:Method: GET 2023-11-02T18:19:10.911 DEBUG:boto:Path: /ibtljn-49/?prefix=obj7 2023-11-02T18:19:10.912 DEBUG:boto:Data: 2023-11-02T18:19:10.912 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.913 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.913 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.914 DEBUG:boto:Params: {} 2023-11-02T18:19:10.914 DEBUG:boto:Token: None 2023-11-02T18:19:10.917 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.918 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000132ad381749cbd3e-006543e81e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.926 DEBUG:boto:b'ibtljn-49obj71000falseobj72023-11-02T18:18:30.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.926 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:10.927 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:10.927 DEBUG:boto:path=/ibtljn-49/?prefix=obj7 2023-11-02T18:19:10.928 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj7 2023-11-02T18:19:10.928 DEBUG:boto:Method: GET 2023-11-02T18:19:10.929 DEBUG:boto:Path: /ibtljn-49/?prefix=obj7 2023-11-02T18:19:10.929 DEBUG:boto:Data: 2023-11-02T18:19:10.930 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.931 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.931 DEBUG:boto:Params: {} 2023-11-02T18:19:10.932 DEBUG:boto:Token: None 2023-11-02T18:19:10.932 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/ 2023-11-02T18:19:10.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8= 2023-11-02T18:19:10.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrZKAFNr7tec24CdSdDobJFCEq8=', 'Content-Length': '0'} 2023-11-02T18:19:10.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adb128dadd209fa8-006543e81e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.982 DEBUG:boto:b'ibtljn-49obj71000falseobj72023-11-02T18:18:30.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:10.983 DEBUG:boto:path=/ibtljn-49/obj7 2023-11-02T18:19:10.983 DEBUG:boto:auth_path=/ibtljn-49/obj7 2023-11-02T18:19:10.984 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.984 DEBUG:boto:Path: /ibtljn-49/obj7 2023-11-02T18:19:10.985 DEBUG:boto:Data: 2023-11-02T18:19:10.985 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.986 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:10.986 DEBUG:boto:Port: 8001 2023-11-02T18:19:10.986 DEBUG:boto:Params: {} 2023-11-02T18:19:10.987 DEBUG:boto:Token: None 2023-11-02T18:19:10.987 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj7 2023-11-02T18:19:10.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j+BiLNtW6pICHrqeq7Cs7u3ITZ8= 2023-11-02T18:19:10.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j+BiLNtW6pICHrqeq7Cs7u3ITZ8=', 'Content-Length': '0'} 2023-11-02T18:19:10.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033ac97e2c19dc8eb-006543e81e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:10.992 DEBUG:boto:path=/ibtljn-49/obj7 2023-11-02T18:19:10.992 DEBUG:boto:auth_path=/ibtljn-49/obj7 2023-11-02T18:19:10.993 DEBUG:boto:Method: HEAD 2023-11-02T18:19:10.993 DEBUG:boto:Path: /ibtljn-49/obj7 2023-11-02T18:19:10.994 DEBUG:boto:Data: 2023-11-02T18:19:10.994 DEBUG:boto:Headers: {} 2023-11-02T18:19:10.995 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:10.995 DEBUG:boto:Port: 8000 2023-11-02T18:19:10.995 DEBUG:boto:Params: {} 2023-11-02T18:19:10.996 DEBUG:boto:Token: None 2023-11-02T18:19:10.996 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:10 GMT /ibtljn-49/obj7 2023-11-02T18:19:10.997 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j+BiLNtW6pICHrqeq7Cs7u3ITZ8= 2023-11-02T18:19:10.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j+BiLNtW6pICHrqeq7Cs7u3ITZ8=', 'Content-Length': '0'} 2023-11-02T18:19:11.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cb48b11af5ac84c-006543e81e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-11-02T18:19:11.001 DEBUG:boto:path=/ibtljn-49/obj7 2023-11-02T18:19:11.002 DEBUG:boto:auth_path=/ibtljn-49/obj7 2023-11-02T18:19:11.002 DEBUG:boto:Method: GET 2023-11-02T18:19:11.003 DEBUG:boto:Path: /ibtljn-49/obj7 2023-11-02T18:19:11.003 DEBUG:boto:Data: 2023-11-02T18:19:11.004 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.004 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:11.004 DEBUG:boto:Port: 8001 2023-11-02T18:19:11.005 DEBUG:boto:Params: {} 2023-11-02T18:19:11.005 DEBUG:boto:Token: None 2023-11-02T18:19:11.006 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/obj7 2023-11-02T18:19:11.006 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1MIrbpFrGECO1GYtKx8ixo3fVQ= 2023-11-02T18:19:11.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1MIrbpFrGECO1GYtKx8ixo3fVQ=', 'Content-Length': '0'} 2023-11-02T18:19:11.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c99d0d20a520cc2f-006543e81f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.011 DEBUG:boto:path=/ibtljn-49/obj7 2023-11-02T18:19:11.011 DEBUG:boto:auth_path=/ibtljn-49/obj7 2023-11-02T18:19:11.012 DEBUG:boto:Method: GET 2023-11-02T18:19:11.012 DEBUG:boto:Path: /ibtljn-49/obj7 2023-11-02T18:19:11.012 DEBUG:boto:Data: 2023-11-02T18:19:11.013 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.013 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:11.014 DEBUG:boto:Port: 8000 2023-11-02T18:19:11.014 DEBUG:boto:Params: {} 2023-11-02T18:19:11.015 DEBUG:boto:Token: None 2023-11-02T18:19:11.015 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/obj7 2023-11-02T18:19:11.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1MIrbpFrGECO1GYtKx8ixo3fVQ= 2023-11-02T18:19:11.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1MIrbpFrGECO1GYtKx8ixo3fVQ=', 'Content-Length': '0'} 2023-11-02T18:19:11.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000436b2e4e55ae984b-006543e81f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.020 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:11.020 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:11.021 DEBUG:boto:path=/ibtljn-49/?prefix=obj7 2023-11-02T18:19:11.021 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj7 2023-11-02T18:19:11.022 DEBUG:boto:Method: GET 2023-11-02T18:19:11.022 DEBUG:boto:Path: /ibtljn-49/?prefix=obj7 2023-11-02T18:19:11.023 DEBUG:boto:Data: 2023-11-02T18:19:11.023 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.023 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:11.024 DEBUG:boto:Port: 8001 2023-11-02T18:19:11.024 DEBUG:boto:Params: {} 2023-11-02T18:19:11.025 DEBUG:boto:Token: None 2023-11-02T18:19:11.025 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/ 2023-11-02T18:19:11.026 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PxVNMI/rhmSZ4ogqawfA/Qsz1Ws= 2023-11-02T18:19:11.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PxVNMI/rhmSZ4ogqawfA/Qsz1Ws=', 'Content-Length': '0'} 2023-11-02T18:19:11.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e07728fb5c6e01ee-006543e81f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.032 DEBUG:boto:b'ibtljn-49obj71000falseobj72023-11-02T18:18:30.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:11.033 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:11.033 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:11.034 DEBUG:boto:path=/ibtljn-49/?prefix=obj7 2023-11-02T18:19:11.034 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj7 2023-11-02T18:19:11.034 DEBUG:boto:Method: GET 2023-11-02T18:19:11.035 DEBUG:boto:Path: /ibtljn-49/?prefix=obj7 2023-11-02T18:19:11.035 DEBUG:boto:Data: 2023-11-02T18:19:11.036 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.036 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:11.037 DEBUG:boto:Port: 8000 2023-11-02T18:19:11.037 DEBUG:boto:Params: {} 2023-11-02T18:19:11.038 DEBUG:boto:Token: None 2023-11-02T18:19:11.038 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/ 2023-11-02T18:19:11.039 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PxVNMI/rhmSZ4ogqawfA/Qsz1Ws= 2023-11-02T18:19:11.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PxVNMI/rhmSZ4ogqawfA/Qsz1Ws=', 'Content-Length': '0'} 2023-11-02T18:19:11.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f07c7e6bb8b15662-006543e81f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.045 DEBUG:boto:b'ibtljn-49obj71000falseobj72023-11-02T18:18:30.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:11.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-11-02T18:19:11.046 DEBUG:boto:path=/ibtljn-49/obj8 2023-11-02T18:19:11.046 DEBUG:boto:auth_path=/ibtljn-49/obj8 2023-11-02T18:19:11.047 DEBUG:boto:path=/ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.047 DEBUG:boto:auth_path=/ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.048 DEBUG:boto:Method: GET 2023-11-02T18:19:11.048 DEBUG:boto:Path: /ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.048 DEBUG:boto:Data: 2023-11-02T18:19:11.049 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.049 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:11.049 DEBUG:boto:Port: 8001 2023-11-02T18:19:11.050 DEBUG:boto:Params: {} 2023-11-02T18:19:11.050 DEBUG:boto:Token: None 2023-11-02T18:19:11.051 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JdikYUSuxHbYYITay7nEkHvhni4= 2023-11-02T18:19:11.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JdikYUSuxHbYYITay7nEkHvhni4=', 'Content-Length': '0'} 2023-11-02T18:19:11.055 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ac9471c9d927073-006543e81f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.098 DEBUG:boto:path=/ibtljn-49/obj8 2023-11-02T18:19:11.099 DEBUG:boto:auth_path=/ibtljn-49/obj8 2023-11-02T18:19:11.099 DEBUG:boto:path=/ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.099 DEBUG:boto:auth_path=/ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.100 DEBUG:boto:Method: GET 2023-11-02T18:19:11.100 DEBUG:boto:Path: /ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.101 DEBUG:boto:Data: 2023-11-02T18:19:11.101 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.101 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:11.102 DEBUG:boto:Port: 8000 2023-11-02T18:19:11.102 DEBUG:boto:Params: {} 2023-11-02T18:19:11.103 DEBUG:boto:Token: None 2023-11-02T18:19:11.103 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JdikYUSuxHbYYITay7nEkHvhni4= 2023-11-02T18:19:11.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JdikYUSuxHbYYITay7nEkHvhni4=', 'Content-Length': '0'} 2023-11-02T18:19:11.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a868f70c64e0f96-006543e81f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.108 DEBUG:boto:path=/ibtljn-49/obj8 2023-11-02T18:19:11.109 DEBUG:boto:auth_path=/ibtljn-49/obj8 2023-11-02T18:19:11.109 DEBUG:boto:path=/ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.110 DEBUG:boto:auth_path=/ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.110 DEBUG:boto:Method: HEAD 2023-11-02T18:19:11.110 DEBUG:boto:Path: /ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.111 DEBUG:boto:Data: 2023-11-02T18:19:11.111 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.112 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:11.112 DEBUG:boto:Port: 8001 2023-11-02T18:19:11.113 DEBUG:boto:Params: {} 2023-11-02T18:19:11.113 DEBUG:boto:Token: None 2023-11-02T18:19:11.114 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.114 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BR8v99ZDfo1IbK212BKWQ1PVZ6U= 2023-11-02T18:19:11.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BR8v99ZDfo1IbK212BKWQ1PVZ6U=', 'Content-Length': '0'} 2023-11-02T18:19:11.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b601ae85f8cdcc9d-006543e81f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.118 DEBUG:boto:path=/ibtljn-49/obj8 2023-11-02T18:19:11.119 DEBUG:boto:auth_path=/ibtljn-49/obj8 2023-11-02T18:19:11.119 DEBUG:boto:path=/ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.119 DEBUG:boto:auth_path=/ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.120 DEBUG:boto:Method: HEAD 2023-11-02T18:19:11.120 DEBUG:boto:Path: /ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.121 DEBUG:boto:Data: 2023-11-02T18:19:11.121 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.122 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:11.122 DEBUG:boto:Port: 8000 2023-11-02T18:19:11.122 DEBUG:boto:Params: {} 2023-11-02T18:19:11.123 DEBUG:boto:Token: None 2023-11-02T18:19:11.123 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.124 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BR8v99ZDfo1IbK212BKWQ1PVZ6U= 2023-11-02T18:19:11.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BR8v99ZDfo1IbK212BKWQ1PVZ6U=', 'Content-Length': '0'} 2023-11-02T18:19:11.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adefcffa7ec20d83-006543e81f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-11-02T18:19:11.128 DEBUG:boto:path=/ibtljn-49/obj8 2023-11-02T18:19:11.129 DEBUG:boto:auth_path=/ibtljn-49/obj8 2023-11-02T18:19:11.129 DEBUG:boto:path=/ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.130 DEBUG:boto:auth_path=/ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.130 DEBUG:boto:Method: GET 2023-11-02T18:19:11.130 DEBUG:boto:Path: /ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.131 DEBUG:boto:Data: 2023-11-02T18:19:11.131 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.132 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:11.132 DEBUG:boto:Port: 8001 2023-11-02T18:19:11.132 DEBUG:boto:Params: {} 2023-11-02T18:19:11.134 DEBUG:boto:Token: None 2023-11-02T18:19:11.134 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.135 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JdikYUSuxHbYYITay7nEkHvhni4= 2023-11-02T18:19:11.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JdikYUSuxHbYYITay7nEkHvhni4=', 'Content-Length': '0'} 2023-11-02T18:19:11.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7e835a5fc963dd0-006543e81f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.139 DEBUG:boto:path=/ibtljn-49/obj8 2023-11-02T18:19:11.140 DEBUG:boto:auth_path=/ibtljn-49/obj8 2023-11-02T18:19:11.140 DEBUG:boto:path=/ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.140 DEBUG:boto:auth_path=/ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.141 DEBUG:boto:Method: GET 2023-11-02T18:19:11.141 DEBUG:boto:Path: /ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.142 DEBUG:boto:Data: 2023-11-02T18:19:11.142 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.143 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:11.143 DEBUG:boto:Port: 8000 2023-11-02T18:19:11.143 DEBUG:boto:Params: {} 2023-11-02T18:19:11.144 DEBUG:boto:Token: None 2023-11-02T18:19:11.144 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/obj8?versionId=null 2023-11-02T18:19:11.145 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JdikYUSuxHbYYITay7nEkHvhni4= 2023-11-02T18:19:11.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JdikYUSuxHbYYITay7nEkHvhni4=', 'Content-Length': '0'} 2023-11-02T18:19:11.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ef916636a77d1de-006543e81f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.190 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:11.191 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:11.191 DEBUG:boto:path=/ibtljn-49/?prefix=obj8 2023-11-02T18:19:11.192 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj8 2023-11-02T18:19:11.192 DEBUG:boto:Method: GET 2023-11-02T18:19:11.192 DEBUG:boto:Path: /ibtljn-49/?prefix=obj8 2023-11-02T18:19:11.193 DEBUG:boto:Data: 2023-11-02T18:19:11.193 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.194 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:11.194 DEBUG:boto:Port: 8001 2023-11-02T18:19:11.195 DEBUG:boto:Params: {} 2023-11-02T18:19:11.195 DEBUG:boto:Token: None 2023-11-02T18:19:11.196 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/ 2023-11-02T18:19:11.196 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PxVNMI/rhmSZ4ogqawfA/Qsz1Ws= 2023-11-02T18:19:11.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PxVNMI/rhmSZ4ogqawfA/Qsz1Ws=', 'Content-Length': '0'} 2023-11-02T18:19:11.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c807609c18951ec2-006543e81f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.203 DEBUG:boto:b'ibtljn-49obj81000falseobj82023-11-02T18:18:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:11.204 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:11.204 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:11.205 DEBUG:boto:path=/ibtljn-49/?prefix=obj8 2023-11-02T18:19:11.205 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj8 2023-11-02T18:19:11.205 DEBUG:boto:Method: GET 2023-11-02T18:19:11.206 DEBUG:boto:Path: /ibtljn-49/?prefix=obj8 2023-11-02T18:19:11.206 DEBUG:boto:Data: 2023-11-02T18:19:11.207 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.207 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:11.208 DEBUG:boto:Port: 8000 2023-11-02T18:19:11.208 DEBUG:boto:Params: {} 2023-11-02T18:19:11.209 DEBUG:boto:Token: None 2023-11-02T18:19:11.209 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/ 2023-11-02T18:19:11.210 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PxVNMI/rhmSZ4ogqawfA/Qsz1Ws= 2023-11-02T18:19:11.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PxVNMI/rhmSZ4ogqawfA/Qsz1Ws=', 'Content-Length': '0'} 2023-11-02T18:19:11.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000065d5bcf1cf7bf03-006543e81f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.216 DEBUG:boto:b'ibtljn-49obj81000falseobj82023-11-02T18:18:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:11.217 DEBUG:boto:path=/ibtljn-49/obj8 2023-11-02T18:19:11.217 DEBUG:boto:auth_path=/ibtljn-49/obj8 2023-11-02T18:19:11.217 DEBUG:boto:Method: HEAD 2023-11-02T18:19:11.218 DEBUG:boto:Path: /ibtljn-49/obj8 2023-11-02T18:19:11.218 DEBUG:boto:Data: 2023-11-02T18:19:11.219 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.219 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:11.220 DEBUG:boto:Port: 8001 2023-11-02T18:19:11.220 DEBUG:boto:Params: {} 2023-11-02T18:19:11.220 DEBUG:boto:Token: None 2023-11-02T18:19:11.221 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/obj8 2023-11-02T18:19:11.221 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HVCW9uAPfwhH4JLHsToN6AL7PKI= 2023-11-02T18:19:11.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HVCW9uAPfwhH4JLHsToN6AL7PKI=', 'Content-Length': '0'} 2023-11-02T18:19:11.224 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c41b2d0f17ed0cfa-006543e81f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.225 DEBUG:boto:path=/ibtljn-49/obj8 2023-11-02T18:19:11.225 DEBUG:boto:auth_path=/ibtljn-49/obj8 2023-11-02T18:19:11.226 DEBUG:boto:Method: HEAD 2023-11-02T18:19:11.226 DEBUG:boto:Path: /ibtljn-49/obj8 2023-11-02T18:19:11.227 DEBUG:boto:Data: 2023-11-02T18:19:11.227 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.227 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:11.228 DEBUG:boto:Port: 8000 2023-11-02T18:19:11.228 DEBUG:boto:Params: {} 2023-11-02T18:19:11.229 DEBUG:boto:Token: None 2023-11-02T18:19:11.229 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/obj8 2023-11-02T18:19:11.230 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HVCW9uAPfwhH4JLHsToN6AL7PKI= 2023-11-02T18:19:11.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HVCW9uAPfwhH4JLHsToN6AL7PKI=', 'Content-Length': '0'} 2023-11-02T18:19:11.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b7739ce330d2814-006543e81f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-11-02T18:19:11.234 DEBUG:boto:path=/ibtljn-49/obj8 2023-11-02T18:19:11.234 DEBUG:boto:auth_path=/ibtljn-49/obj8 2023-11-02T18:19:11.235 DEBUG:boto:Method: GET 2023-11-02T18:19:11.235 DEBUG:boto:Path: /ibtljn-49/obj8 2023-11-02T18:19:11.236 DEBUG:boto:Data: 2023-11-02T18:19:11.236 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.237 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:11.237 DEBUG:boto:Port: 8001 2023-11-02T18:19:11.237 DEBUG:boto:Params: {} 2023-11-02T18:19:11.238 DEBUG:boto:Token: None 2023-11-02T18:19:11.238 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/obj8 2023-11-02T18:19:11.239 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CcX0JxzKHtLZPRExbtwfhREY320= 2023-11-02T18:19:11.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CcX0JxzKHtLZPRExbtwfhREY320=', 'Content-Length': '0'} 2023-11-02T18:19:11.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000593748df9446d136-006543e81f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.243 DEBUG:boto:path=/ibtljn-49/obj8 2023-11-02T18:19:11.243 DEBUG:boto:auth_path=/ibtljn-49/obj8 2023-11-02T18:19:11.244 DEBUG:boto:Method: GET 2023-11-02T18:19:11.244 DEBUG:boto:Path: /ibtljn-49/obj8 2023-11-02T18:19:11.245 DEBUG:boto:Data: 2023-11-02T18:19:11.245 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:11.246 DEBUG:boto:Port: 8000 2023-11-02T18:19:11.247 DEBUG:boto:Params: {} 2023-11-02T18:19:11.247 DEBUG:boto:Token: None 2023-11-02T18:19:11.248 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/obj8 2023-11-02T18:19:11.248 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CcX0JxzKHtLZPRExbtwfhREY320= 2023-11-02T18:19:11.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CcX0JxzKHtLZPRExbtwfhREY320=', 'Content-Length': '0'} 2023-11-02T18:19:11.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:18:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-49:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d7916dddbb485a4-006543e81f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.252 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:11.253 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:11.253 DEBUG:boto:path=/ibtljn-49/?prefix=obj8 2023-11-02T18:19:11.253 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj8 2023-11-02T18:19:11.254 DEBUG:boto:Method: GET 2023-11-02T18:19:11.254 DEBUG:boto:Path: /ibtljn-49/?prefix=obj8 2023-11-02T18:19:11.255 DEBUG:boto:Data: 2023-11-02T18:19:11.255 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.256 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:11.256 DEBUG:boto:Port: 8001 2023-11-02T18:19:11.256 DEBUG:boto:Params: {} 2023-11-02T18:19:11.257 DEBUG:boto:Token: None 2023-11-02T18:19:11.257 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/ 2023-11-02T18:19:11.258 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PxVNMI/rhmSZ4ogqawfA/Qsz1Ws= 2023-11-02T18:19:11.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PxVNMI/rhmSZ4ogqawfA/Qsz1Ws=', 'Content-Length': '0'} 2023-11-02T18:19:11.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000088df3479c863397-006543e81f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.306 DEBUG:boto:b'ibtljn-49obj81000falseobj82023-11-02T18:18:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:11.307 DEBUG:boto:path=/ibtljn-49/ 2023-11-02T18:19:11.308 DEBUG:boto:auth_path=/ibtljn-49/ 2023-11-02T18:19:11.308 DEBUG:boto:path=/ibtljn-49/?prefix=obj8 2023-11-02T18:19:11.309 DEBUG:boto:auth_path=/ibtljn-49/?prefix=obj8 2023-11-02T18:19:11.309 DEBUG:boto:Method: GET 2023-11-02T18:19:11.309 DEBUG:boto:Path: /ibtljn-49/?prefix=obj8 2023-11-02T18:19:11.310 DEBUG:boto:Data: 2023-11-02T18:19:11.310 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.311 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:11.311 DEBUG:boto:Port: 8000 2023-11-02T18:19:11.312 DEBUG:boto:Params: {} 2023-11-02T18:19:11.312 DEBUG:boto:Token: None 2023-11-02T18:19:11.313 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-49/ 2023-11-02T18:19:11.313 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PxVNMI/rhmSZ4ogqawfA/Qsz1Ws= 2023-11-02T18:19:11.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PxVNMI/rhmSZ4ogqawfA/Qsz1Ws=', 'Content-Length': '0'} 2023-11-02T18:19:11.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dbac2aa899adb1c-006543e81f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.319 DEBUG:boto:b'ibtljn-49obj81000falseobj82023-11-02T18:18:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:11.320 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-49 zones={a2, a1} 2023-11-02T18:19:11.320 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-11-02T18:19:11.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:11.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:11.322 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:19:11.368 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:11.368 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:11.467 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:19:11.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.468 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004595", 2023-11-02T18:19:11.468 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:18:41.615924Z" 2023-11-02T18:19:11.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.469 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000009948", 2023-11-02T18:19:11.469 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:18:51.545038Z" 2023-11-02T18:19:11.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.469 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000006851", 2023-11-02T18:19:11.470 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:18:31.169748Z" 2023-11-02T18:19:11.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.470 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-11-02T18:19:11.471 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:18:51.542801Z" 2023-11-02T18:19:11.471 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:11.471 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:19:11.472 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000009948', 2: '00000000000000000000:00000000000000006851', 3: '00000000000000000000:00000000000000006576'} 2023-11-02T18:19:11.473 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:19:11.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:11.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:11.474 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:19:11.524 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:11.525 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:11.601 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:19:11.601 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:19:11.602 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:19:11.602 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:19:11.602 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:19:11.602 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 14562503763241451158 2023-11-02T18:19:11.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.603 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:19:11.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.603 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:19:11.603 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:11.603 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:19:11.604 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004595", 2023-11-02T18:19:11.604 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:11.604 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:11.604 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:11.604 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:18:41.615924Z" 2023-11-02T18:19:11.605 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:11.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.605 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:19:11.606 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:11.606 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:19:11.606 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000009948", 2023-11-02T18:19:11.606 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:11.606 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:11.607 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:11.607 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:18:51.545038Z" 2023-11-02T18:19:11.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:11.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.608 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:19:11.608 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:11.608 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:19:11.608 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000006851", 2023-11-02T18:19:11.609 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:11.609 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:11.609 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:11.609 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:18:31.169748Z" 2023-11-02T18:19:11.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:11.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.610 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:19:11.610 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:11.611 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:19:11.611 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-11-02T18:19:11.611 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:11.611 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:11.611 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:11.612 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:18:51.542801Z" 2023-11-02T18:19:11.612 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:11.612 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:11.612 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:19:11.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.613 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:19:11.613 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:19:11.613 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:19:11.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.614 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:19:11Z" 2023-11-02T18:19:11.614 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:19:11.615 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14562503763241451158 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004595", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:18:41.615924Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009948", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:18:51.545038Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006851", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:18:31.169748Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006576", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:18:51.542801Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:19:11Z" } 2023-11-02T18:19:11.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004595', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:18:41.615924Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009948', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:18:51.545038Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006851', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:18:31.169748Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006576', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:18:51.542801Z'}}] 2023-11-02T18:19:11.616 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000009948', 2: '00000000000000000000:00000000000000006851', 3: '00000000000000000000:00000000000000006576'} 2023-11-02T18:19:11.616 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000009948', 2: '00000000000000000000:00000000000000006851', 3: '00000000000000000000:00000000000000006576'} 2023-11-02T18:19:11.616 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:19:11.617 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-11-02T18:19:11.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:11.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:11.618 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:19:11.665 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:11.666 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:11.764 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:19:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000003185", 2023-11-02T18:19:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:18:43.272907Z" 2023-11-02T18:19:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000008397", 2023-11-02T18:19:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:18:43.276547Z" 2023-11-02T18:19:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004315", 2023-11-02T18:19:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:18:43.279750Z" 2023-11-02T18:19:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004597", 2023-11-02T18:19:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:18:43.282909Z" 2023-11-02T18:19:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:11.768 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:19:11.769 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000003185', 1: '00000000000000000000:00000000000000008397', 2: '00000000000000000000:00000000000000004315', 3: '00000000000000000000:00000000000000004597'} 2023-11-02T18:19:11.770 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:19:11.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:11.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:11.771 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:19:11.818 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:11.818 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:11.904 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:19:11.904 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:19:11.905 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:19:11.905 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:19:11.905 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:19:11.905 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:19:11.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.906 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:19:11.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.906 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:19:11.906 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:11.907 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:19:11.907 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000003185", 2023-11-02T18:19:11.907 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:11.907 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:11.908 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:11.908 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:18:43.272907Z" 2023-11-02T18:19:11.908 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:11.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.909 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:19:11.909 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:11.909 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:19:11.909 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000008397", 2023-11-02T18:19:11.910 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:11.910 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:11.910 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:11.910 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:18:43.276547Z" 2023-11-02T18:19:11.910 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:11.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.911 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:19:11.911 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:11.912 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:19:11.912 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004315", 2023-11-02T18:19:11.912 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:11.912 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:11.913 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:11.913 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:18:43.279750Z" 2023-11-02T18:19:11.913 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:11.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:11.914 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:19:11.914 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:11.914 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:19:11.914 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000004597", 2023-11-02T18:19:11.915 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:11.915 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:11.915 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:11.915 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:18:43.282909Z" 2023-11-02T18:19:11.915 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:11.916 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:11.916 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:19:11.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.916 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:19:11.916 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:19:11.917 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:19:11.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:11.917 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:19:11Z" 2023-11-02T18:19:11.917 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:19:11.918 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003185", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:18:43.272907Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008397", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:18:43.276547Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004315", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:18:43.279750Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:18:43.282909Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:19:11Z" } 2023-11-02T18:19:11.919 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003185', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:18:43.272907Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008397', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:18:43.276547Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004315', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:18:43.279750Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:18:43.282909Z'}}] 2023-11-02T18:19:11.919 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003185', 1: '00000000000000000000:00000000000000008397', 2: '00000000000000000000:00000000000000004315', 3: '00000000000000000000:00000000000000004597'} 2023-11-02T18:19:11.920 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003185', 1: '00000000000000000000:00000000000000008397', 2: '00000000000000000000:00000000000000004315', 3: '00000000000000000000:00000000000000004597'} 2023-11-02T18:19:11.920 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:19:11.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2023-11-02T18:19:11.922 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-50 2023-11-02T18:19:11.923 DEBUG:boto:path=/ibtljn-50/ 2023-11-02T18:19:11.923 DEBUG:boto:auth_path=/ibtljn-50/ 2023-11-02T18:19:11.923 DEBUG:boto:Method: PUT 2023-11-02T18:19:11.924 DEBUG:boto:Path: /ibtljn-50/ 2023-11-02T18:19:11.924 DEBUG:boto:Data: 2023-11-02T18:19:11.925 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.925 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:11.925 DEBUG:boto:Port: 8000 2023-11-02T18:19:11.926 DEBUG:boto:Params: {} 2023-11-02T18:19:11.926 DEBUG:boto:Token: None 2023-11-02T18:19:11.927 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-50/ 2023-11-02T18:19:11.927 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0mkLFTF0QkF/adhxuuVWKaKAer8= 2023-11-02T18:19:11.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0mkLFTF0QkF/adhxuuVWKaKAer8=', 'Content-Length': '0'} 2023-11-02T18:19:11.956 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000049ea91a8e658e844-006543e81f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:11.956 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-51 2023-11-02T18:19:11.957 DEBUG:boto:path=/ibtljn-51/ 2023-11-02T18:19:11.957 DEBUG:boto:auth_path=/ibtljn-51/ 2023-11-02T18:19:11.958 DEBUG:boto:Method: PUT 2023-11-02T18:19:11.958 DEBUG:boto:Path: /ibtljn-51/ 2023-11-02T18:19:11.958 DEBUG:boto:Data: 2023-11-02T18:19:11.959 DEBUG:boto:Headers: {} 2023-11-02T18:19:11.959 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:11.960 DEBUG:boto:Port: 8001 2023-11-02T18:19:11.960 DEBUG:boto:Params: {} 2023-11-02T18:19:11.961 DEBUG:boto:Token: None 2023-11-02T18:19:11.961 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:19:11 GMT /ibtljn-51/ 2023-11-02T18:19:11.961 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N5H6ljvHUNorpGEBR20Tuw6szvU= 2023-11-02T18:19:11.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N5H6ljvHUNorpGEBR20Tuw6szvU=', 'Content-Length': '0'} 2023-11-02T18:19:12.011 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006f7ca296013b7bf5-006543e81f-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:12.012 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:12.012 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:12.013 DEBUG:boto:path=/ibtljn-50/MULTIPART?uploads 2023-11-02T18:19:12.013 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?uploads 2023-11-02T18:19:12.014 DEBUG:boto:Method: POST 2023-11-02T18:19:12.014 DEBUG:boto:Path: /ibtljn-50/MULTIPART?uploads 2023-11-02T18:19:12.014 DEBUG:boto:Data: 2023-11-02T18:19:12.015 DEBUG:boto:Headers: {} 2023-11-02T18:19:12.015 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:12.016 DEBUG:boto:Port: 8000 2023-11-02T18:19:12.016 DEBUG:boto:Params: {} 2023-11-02T18:19:12.016 DEBUG:boto:Token: None 2023-11-02T18:19:12.017 DEBUG:boto:StringToSign: POST Thu, 02 Nov 2023 18:19:12 GMT /ibtljn-50/MULTIPART?uploads 2023-11-02T18:19:12.017 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G4PFzAPlat4wNF4ABDmZt+VKNak= 2023-11-02T18:19:12.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G4PFzAPlat4wNF4ABDmZt+VKNak=', 'Content-Length': '0'} 2023-11-02T18:19:13.842 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a4072d5496480d5d-006543e820-1669-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Thu, 02 Nov 2023 18:19:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:13.843 DEBUG:boto:b'ibtljn-50MULTIPART2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP' 2023-11-02T18:19:13.877 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:13.878 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:13.878 DEBUG:boto:path=/ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP&partNumber=1 2023-11-02T18:19:13.879 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP&partNumber=1 2023-11-02T18:19:13.879 DEBUG:boto:Method: PUT 2023-11-02T18:19:13.880 DEBUG:boto:Path: /ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP&partNumber=1 2023-11-02T18:19:13.880 DEBUG:boto:Data: 2023-11-02T18:19:13.881 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-11-02T18:19:13.881 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:13.882 DEBUG:boto:Port: 8000 2023-11-02T18:19:13.883 DEBUG:boto:Params: {} 2023-11-02T18:19:13.883 DEBUG:boto:Token: None 2023-11-02T18:19:13.884 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Thu, 02 Nov 2023 18:19:13 GMT /ibtljn-50/MULTIPART?partNumber=1&uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP 2023-11-02T18:19:13.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pf6/uVCsxFMKIlEi7xeIFvjXQGw= 2023-11-02T18:19:13.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:19:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pf6/uVCsxFMKIlEi7xeIFvjXQGw='} 2023-11-02T18:19:13.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008be8d62016e2a04d-006543e821-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:19:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:13.985 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:13.985 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:13.986 DEBUG:boto:path=/ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP&partNumber=2 2023-11-02T18:19:13.986 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP&partNumber=2 2023-11-02T18:19:13.987 DEBUG:boto:Method: PUT 2023-11-02T18:19:13.987 DEBUG:boto:Path: /ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP&partNumber=2 2023-11-02T18:19:13.988 DEBUG:boto:Data: 2023-11-02T18:19:13.988 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-11-02T18:19:13.989 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:13.989 DEBUG:boto:Port: 8000 2023-11-02T18:19:13.989 DEBUG:boto:Params: {} 2023-11-02T18:19:13.990 DEBUG:boto:Token: None 2023-11-02T18:19:13.990 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Thu, 02 Nov 2023 18:19:13 GMT /ibtljn-50/MULTIPART?partNumber=2&uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP 2023-11-02T18:19:13.991 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pwx7sbsnR3eN9Zem0dktsTA+Pr0= 2023-11-02T18:19:13.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:19:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pwx7sbsnR3eN9Zem0dktsTA+Pr0='} 2023-11-02T18:19:14.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061064fc7c6a7d1ee-006543e821-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:14.072 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:14.072 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:14.073 DEBUG:boto:path=/ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP&partNumber=3 2023-11-02T18:19:14.074 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP&partNumber=3 2023-11-02T18:19:14.074 DEBUG:boto:Method: PUT 2023-11-02T18:19:14.075 DEBUG:boto:Path: /ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP&partNumber=3 2023-11-02T18:19:14.075 DEBUG:boto:Data: 2023-11-02T18:19:14.076 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-11-02T18:19:14.076 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:14.077 DEBUG:boto:Port: 8000 2023-11-02T18:19:14.077 DEBUG:boto:Params: {} 2023-11-02T18:19:14.078 DEBUG:boto:Token: None 2023-11-02T18:19:14.078 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Thu, 02 Nov 2023 18:19:14 GMT /ibtljn-50/MULTIPART?partNumber=3&uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP 2023-11-02T18:19:14.079 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5afPwhA+j9yQWeAuAaNNHuxb1+E= 2023-11-02T18:19:14.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:19:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5afPwhA+j9yQWeAuAaNNHuxb1+E='} 2023-11-02T18:19:14.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b1957ed2d51f6556-006543e822-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:14.156 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:14.157 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:14.157 DEBUG:boto:path=/ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP&partNumber=4 2023-11-02T18:19:14.158 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP&partNumber=4 2023-11-02T18:19:14.159 DEBUG:boto:Method: PUT 2023-11-02T18:19:14.159 DEBUG:boto:Path: /ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP&partNumber=4 2023-11-02T18:19:14.160 DEBUG:boto:Data: 2023-11-02T18:19:14.160 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-11-02T18:19:14.161 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:14.161 DEBUG:boto:Port: 8000 2023-11-02T18:19:14.162 DEBUG:boto:Params: {} 2023-11-02T18:19:14.163 DEBUG:boto:Token: None 2023-11-02T18:19:14.163 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Thu, 02 Nov 2023 18:19:14 GMT /ibtljn-50/MULTIPART?partNumber=4&uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP 2023-11-02T18:19:14.164 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5whM3g6n1KHX20/0KwFcM3cmL14= 2023-11-02T18:19:14.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:19:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5whM3g6n1KHX20/0KwFcM3cmL14='} 2023-11-02T18:19:14.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000820baa4b8f862d42-006543e822-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:14.210 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:14.211 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:14.211 DEBUG:boto:path=/ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP 2023-11-02T18:19:14.211 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP 2023-11-02T18:19:14.212 DEBUG:boto:Method: GET 2023-11-02T18:19:14.212 DEBUG:boto:Path: /ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP 2023-11-02T18:19:14.213 DEBUG:boto:Data: 2023-11-02T18:19:14.213 DEBUG:boto:Headers: {} 2023-11-02T18:19:14.213 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:14.214 DEBUG:boto:Port: 8000 2023-11-02T18:19:14.214 DEBUG:boto:Params: {} 2023-11-02T18:19:14.215 DEBUG:boto:Token: None 2023-11-02T18:19:14.215 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:14 GMT /ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP 2023-11-02T18:19:14.216 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XB3H1t+bT/HrypFvoMbdR1X2Pew= 2023-11-02T18:19:14.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XB3H1t+bT/HrypFvoMbdR1X2Pew=', 'Content-Length': '0'} 2023-11-02T18:19:14.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e13b5ef67ccd84a-006543e822-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:14.263 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:14.263 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:14.264 DEBUG:boto:path=/ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP 2023-11-02T18:19:14.264 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP 2023-11-02T18:19:14.265 DEBUG:boto:Method: POST 2023-11-02T18:19:14.265 DEBUG:boto:Path: /ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP 2023-11-02T18:19:14.265 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2023-11-02T18:19:14.266 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2023-11-02T18:19:14.266 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:14.267 DEBUG:boto:Port: 8000 2023-11-02T18:19:14.267 DEBUG:boto:Params: {} 2023-11-02T18:19:14.268 DEBUG:boto:Token: None 2023-11-02T18:19:14.268 DEBUG:boto:StringToSign: POST text/xml Thu, 02 Nov 2023 18:19:14 GMT /ibtljn-50/MULTIPART?uploadId=2~iq4ReYgjLb9fS17hlWPsb_WV8UXMIpP 2023-11-02T18:19:14.269 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g1ZarLYQuX6gf4+j4PaeDWx5sFk= 2023-11-02T18:19:14.269 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g1ZarLYQuX6gf4+j4PaeDWx5sFk=', 'Content-Length': '460'} 2023-11-02T18:19:14.289 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001fd6182f4243ae89-006543e822-1669-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:14.290 DEBUG:boto:http://smithi017.front.sepia.ceph.com:8000/ibtljn-50/MULTIPARTibtljn-50MULTIPART 2023-11-02T18:19:14.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:14.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:14.292 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:19:14.336 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:14.336 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:14.398 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:19:14.425 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:19:14.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:14.425 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:19:14.425 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:19:14.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:14.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:14.426 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:19:14.426 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:19:14.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:14.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:14.427 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949151.989112_390.1", 2023-11-02T18:19:14.427 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:19:11.989112Z" 2023-11-02T18:19:14.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:14.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:14.428 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:19:14.428 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:19:14.428 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:14.429 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:19:14.430 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949151.989112_390.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:19:14.430 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:19:14.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:14.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:14.432 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:19:14.477 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:14.477 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:14.533 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:14.526+0000 7ff4f5985f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:19:14.535 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:14.530+0000 7ff4f5985f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:19:14.564 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:19:14.564 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:19:14.564 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:19:14.564 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:19:14.565 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:19:14.565 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:19:14.565 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:14.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:14.565 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:19:14.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:14.566 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:19:14.566 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:14.566 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:19:14.567 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:19:14.567 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:14.567 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:19:14.567 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:14.568 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:19:14.568 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:14.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:14.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:14.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:14.569 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:19:14.569 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:14.569 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:19:14.570 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:19:14.570 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:14.570 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:14.570 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:14.571 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:19:14.571 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:14.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:14.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:14.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:14.572 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:19:14.572 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:14.572 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:19:14.573 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949151.989112_390.1", 2023-11-02T18:19:14.573 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:14.573 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:14.573 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:14.574 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:19:11.989112Z", 2023-11-02T18:19:14.574 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:14.574 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:14.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:14.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:14.575 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:19:14.576 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:14.576 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:19:14.576 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:19:14.576 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:14.576 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:14.577 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:14.577 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:19:14.577 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:14.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:14.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:14.578 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:19:14.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:14.579 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:19:14.579 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:19:14.579 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:19:14.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:14.580 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:19:14Z" 2023-11-02T18:19:14.581 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:19:14.582 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949151.989112_390.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:19:11.989112Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:19:14Z" } 2023-11-02T18:19:14.582 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949151.989112_390.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:19:11.989112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:19:14.583 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949151.989112_390.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:19:14.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949151.989112_390.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:19:14.584 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:19:14.584 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-50 2023-11-02T18:19:14.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:19:14.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:19:14.585 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:19:14.639 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:14.639 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:14.708 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:14.702+0000 7f9a89607f40 1 waiting to reach incremental sync.. 2023-11-02T18:19:19.709 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:19.702+0000 7f9a89607f40 1 waiting to reach incremental sync.. 2023-11-02T18:19:24.712 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:24.706+0000 7f9a89607f40 1 bucket sync caught up with source: 2023-11-02T18:19:24.712 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000011.41.6, , , , , ] 2023-11-02T18:19:24.713 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000011.41.6, , , , , ] 2023-11-02T18:19:24.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:24.706+0000 7f9a89607f40 0 bucket checkpoint complete 2023-11-02T18:19:24.743 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-50 2023-11-02T18:19:24.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:19:24.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:19:24.744 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:19:24.798 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:24.798 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:24.864 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:24.858+0000 7f639ad59f40 1 waiting to reach incremental sync.. 2023-11-02T18:19:29.865 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:29.858+0000 7f639ad59f40 1 waiting to reach incremental sync.. 2023-11-02T18:19:34.868 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:34.862+0000 7f639ad59f40 1 bucket sync caught up with source: 2023-11-02T18:19:34.869 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:19:34.869 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:19:34.869 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:34.862+0000 7f639ad59f40 0 bucket checkpoint complete 2023-11-02T18:19:34.899 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-50 zones={a2, a1} 2023-11-02T18:19:34.899 DEBUG:boto:path=/ibtljn-50/ 2023-11-02T18:19:34.900 DEBUG:boto:auth_path=/ibtljn-50/ 2023-11-02T18:19:34.900 DEBUG:boto:Method: HEAD 2023-11-02T18:19:34.900 DEBUG:boto:Path: /ibtljn-50/ 2023-11-02T18:19:34.901 DEBUG:boto:Data: 2023-11-02T18:19:34.901 DEBUG:boto:Headers: {} 2023-11-02T18:19:34.902 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:34.902 DEBUG:boto:Port: 8001 2023-11-02T18:19:34.902 DEBUG:boto:Params: {} 2023-11-02T18:19:34.903 DEBUG:boto:Token: None 2023-11-02T18:19:34.903 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:34 GMT /ibtljn-50/ 2023-11-02T18:19:34.904 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yX1BqC/XacJNJe9ufsio50X5PLM= 2023-11-02T18:19:34.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yX1BqC/XacJNJe9ufsio50X5PLM=', 'Content-Length': '0'} 2023-11-02T18:19:34.908 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009314160a4b1820f1-006543e836-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:19:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:34.908 DEBUG:boto:path=/ibtljn-50/ 2023-11-02T18:19:34.909 DEBUG:boto:auth_path=/ibtljn-50/ 2023-11-02T18:19:34.909 DEBUG:boto:Method: HEAD 2023-11-02T18:19:34.910 DEBUG:boto:Path: /ibtljn-50/ 2023-11-02T18:19:34.910 DEBUG:boto:Data: 2023-11-02T18:19:34.910 DEBUG:boto:Headers: {} 2023-11-02T18:19:34.911 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:34.911 DEBUG:boto:Port: 8000 2023-11-02T18:19:34.912 DEBUG:boto:Params: {} 2023-11-02T18:19:34.912 DEBUG:boto:Token: None 2023-11-02T18:19:34.913 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:34 GMT /ibtljn-50/ 2023-11-02T18:19:34.913 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yX1BqC/XacJNJe9ufsio50X5PLM= 2023-11-02T18:19:34.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yX1BqC/XacJNJe9ufsio50X5PLM=', 'Content-Length': '0'} 2023-11-02T18:19:34.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8e89ab71a6bbc77-006543e836-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:19:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:34.919 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:19:34.919 DEBUG:boto:path=/ibtljn-50/ 2023-11-02T18:19:34.920 DEBUG:boto:auth_path=/ibtljn-50/ 2023-11-02T18:19:34.920 DEBUG:boto:path=/ibtljn-50/?versions&max-keys=999 2023-11-02T18:19:34.921 DEBUG:boto:auth_path=/ibtljn-50/?versions&max-keys=999 2023-11-02T18:19:34.921 DEBUG:boto:Method: GET 2023-11-02T18:19:34.922 DEBUG:boto:Path: /ibtljn-50/?versions&max-keys=999 2023-11-02T18:19:34.922 DEBUG:boto:Data: 2023-11-02T18:19:34.922 DEBUG:boto:Headers: {} 2023-11-02T18:19:34.923 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:34.923 DEBUG:boto:Port: 8001 2023-11-02T18:19:34.924 DEBUG:boto:Params: {} 2023-11-02T18:19:34.925 DEBUG:boto:Token: None 2023-11-02T18:19:34.925 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:34 GMT /ibtljn-50/?versions 2023-11-02T18:19:34.926 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+JgVRyuPpZVfhwfdloUtQ6mTD3k= 2023-11-02T18:19:34.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+JgVRyuPpZVfhwfdloUtQ6mTD3k=', 'Content-Length': '0'} 2023-11-02T18:19:34.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e9053a7109d434-006543e836-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:34.931 DEBUG:boto:b'ibtljn-50999falseMULTIPARTnulltrue2023-11-02T18:19:14.273Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:34.932 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-11-02T18:19:34.932 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:19:34.932 DEBUG:boto:path=/ibtljn-50/ 2023-11-02T18:19:34.933 DEBUG:boto:auth_path=/ibtljn-50/ 2023-11-02T18:19:34.933 DEBUG:boto:path=/ibtljn-50/?versions&max-keys=999 2023-11-02T18:19:34.934 DEBUG:boto:auth_path=/ibtljn-50/?versions&max-keys=999 2023-11-02T18:19:34.934 DEBUG:boto:Method: GET 2023-11-02T18:19:34.935 DEBUG:boto:Path: /ibtljn-50/?versions&max-keys=999 2023-11-02T18:19:34.935 DEBUG:boto:Data: 2023-11-02T18:19:34.936 DEBUG:boto:Headers: {} 2023-11-02T18:19:34.936 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:34.937 DEBUG:boto:Port: 8000 2023-11-02T18:19:34.937 DEBUG:boto:Params: {} 2023-11-02T18:19:34.937 DEBUG:boto:Token: None 2023-11-02T18:19:34.938 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:34 GMT /ibtljn-50/?versions 2023-11-02T18:19:34.939 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+JgVRyuPpZVfhwfdloUtQ6mTD3k= 2023-11-02T18:19:34.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+JgVRyuPpZVfhwfdloUtQ6mTD3k=', 'Content-Length': '0'} 2023-11-02T18:19:34.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647cfbcd80fca138-006543e836-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:34.944 DEBUG:boto:b'ibtljn-50999falseMULTIPARTnulltrue2023-11-02T18:19:14.273Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:34.945 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-11-02T18:19:34.945 DEBUG:boto:path=/ibtljn-50/ 2023-11-02T18:19:34.945 DEBUG:boto:auth_path=/ibtljn-50/ 2023-11-02T18:19:34.946 DEBUG:boto:path=/ibtljn-50/?versions&max-keys=999 2023-11-02T18:19:34.946 DEBUG:boto:auth_path=/ibtljn-50/?versions&max-keys=999 2023-11-02T18:19:34.947 DEBUG:boto:Method: GET 2023-11-02T18:19:34.947 DEBUG:boto:Path: /ibtljn-50/?versions&max-keys=999 2023-11-02T18:19:34.947 DEBUG:boto:Data: 2023-11-02T18:19:34.948 DEBUG:boto:Headers: {} 2023-11-02T18:19:34.948 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:34.949 DEBUG:boto:Port: 8001 2023-11-02T18:19:34.949 DEBUG:boto:Params: {} 2023-11-02T18:19:34.949 DEBUG:boto:Token: None 2023-11-02T18:19:34.950 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:34 GMT /ibtljn-50/?versions 2023-11-02T18:19:34.950 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+JgVRyuPpZVfhwfdloUtQ6mTD3k= 2023-11-02T18:19:34.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+JgVRyuPpZVfhwfdloUtQ6mTD3k=', 'Content-Length': '0'} 2023-11-02T18:19:34.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e20f874a4abbb367-006543e836-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:34.957 DEBUG:boto:b'ibtljn-50999falseMULTIPARTnulltrue2023-11-02T18:19:14.273Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:34.958 DEBUG:boto:path=/ibtljn-50/ 2023-11-02T18:19:34.958 DEBUG:boto:auth_path=/ibtljn-50/ 2023-11-02T18:19:34.959 DEBUG:boto:path=/ibtljn-50/?versions&max-keys=999 2023-11-02T18:19:34.959 DEBUG:boto:auth_path=/ibtljn-50/?versions&max-keys=999 2023-11-02T18:19:34.960 DEBUG:boto:Method: GET 2023-11-02T18:19:34.960 DEBUG:boto:Path: /ibtljn-50/?versions&max-keys=999 2023-11-02T18:19:34.960 DEBUG:boto:Data: 2023-11-02T18:19:34.961 DEBUG:boto:Headers: {} 2023-11-02T18:19:34.961 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:34.962 DEBUG:boto:Port: 8000 2023-11-02T18:19:34.962 DEBUG:boto:Params: {} 2023-11-02T18:19:34.963 DEBUG:boto:Token: None 2023-11-02T18:19:34.963 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:34 GMT /ibtljn-50/?versions 2023-11-02T18:19:34.963 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+JgVRyuPpZVfhwfdloUtQ6mTD3k= 2023-11-02T18:19:34.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+JgVRyuPpZVfhwfdloUtQ6mTD3k=', 'Content-Length': '0'} 2023-11-02T18:19:34.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062d1bf8a65a6a97e-006543e836-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:34.969 DEBUG:boto:b'ibtljn-50999falseMULTIPARTnulltrue2023-11-02T18:19:14.273Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:34.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-11-02T18:19:34.970 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:34.971 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:34.971 DEBUG:boto:path=/ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:34.971 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:34.972 DEBUG:boto:Method: GET 2023-11-02T18:19:34.972 DEBUG:boto:Path: /ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:34.973 DEBUG:boto:Data: 2023-11-02T18:19:34.973 DEBUG:boto:Headers: {} 2023-11-02T18:19:34.974 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:34.974 DEBUG:boto:Port: 8001 2023-11-02T18:19:34.974 DEBUG:boto:Params: {} 2023-11-02T18:19:34.975 DEBUG:boto:Token: None 2023-11-02T18:19:34.975 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:34 GMT /ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:34.976 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JW7jgeqOSEkjmyzpaDpDmF6Q/+w= 2023-11-02T18:19:34.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JW7jgeqOSEkjmyzpaDpDmF6Q/+w=', 'Content-Length': '0'} 2023-11-02T18:19:34.981 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-50:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000004304a1e3be2f8199-006543e836-1665-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.052 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.053 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.054 DEBUG:boto:path=/ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.054 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.055 DEBUG:boto:Method: GET 2023-11-02T18:19:35.055 DEBUG:boto:Path: /ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.055 DEBUG:boto:Data: 2023-11-02T18:19:35.056 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.056 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:35.057 DEBUG:boto:Port: 8000 2023-11-02T18:19:35.057 DEBUG:boto:Params: {} 2023-11-02T18:19:35.058 DEBUG:boto:Token: None 2023-11-02T18:19:35.058 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.059 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dSf5iHJHleR+zLztF0mpV0O2xN8= 2023-11-02T18:19:35.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dSf5iHJHleR+zLztF0mpV0O2xN8=', 'Content-Length': '0'} 2023-11-02T18:19:35.070 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000008245d4bde40ec871-006543e837-1669-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.155 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.156 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.156 DEBUG:boto:path=/ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.156 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.157 DEBUG:boto:Method: HEAD 2023-11-02T18:19:35.157 DEBUG:boto:Path: /ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.158 DEBUG:boto:Data: 2023-11-02T18:19:35.158 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.159 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:35.159 DEBUG:boto:Port: 8001 2023-11-02T18:19:35.159 DEBUG:boto:Params: {} 2023-11-02T18:19:35.160 DEBUG:boto:Token: None 2023-11-02T18:19:35.160 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.161 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bF+ZziDEwVHo5oU2YmdxteAeIqA= 2023-11-02T18:19:35.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bF+ZziDEwVHo5oU2YmdxteAeIqA=', 'Content-Length': '0'} 2023-11-02T18:19:35.165 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-50:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000002bd66f7ee70ae7f9-006543e837-1665-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.165 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.166 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.166 DEBUG:boto:path=/ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.167 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.167 DEBUG:boto:Method: HEAD 2023-11-02T18:19:35.167 DEBUG:boto:Path: /ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.168 DEBUG:boto:Data: 2023-11-02T18:19:35.168 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.169 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:35.169 DEBUG:boto:Port: 8000 2023-11-02T18:19:35.170 DEBUG:boto:Params: {} 2023-11-02T18:19:35.170 DEBUG:boto:Token: None 2023-11-02T18:19:35.171 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.171 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bF+ZziDEwVHo5oU2YmdxteAeIqA= 2023-11-02T18:19:35.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bF+ZziDEwVHo5oU2YmdxteAeIqA=', 'Content-Length': '0'} 2023-11-02T18:19:35.174 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000009e5e6aa817e3896-006543e837-1669-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-11-02T18:19:35.175 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.176 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.176 DEBUG:boto:path=/ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.177 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.177 DEBUG:boto:Method: GET 2023-11-02T18:19:35.178 DEBUG:boto:Path: /ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.178 DEBUG:boto:Data: 2023-11-02T18:19:35.178 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:35.179 DEBUG:boto:Port: 8001 2023-11-02T18:19:35.180 DEBUG:boto:Params: {} 2023-11-02T18:19:35.180 DEBUG:boto:Token: None 2023-11-02T18:19:35.181 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.181 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dSf5iHJHleR+zLztF0mpV0O2xN8= 2023-11-02T18:19:35.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dSf5iHJHleR+zLztF0mpV0O2xN8=', 'Content-Length': '0'} 2023-11-02T18:19:35.186 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-50:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000f4d8e8ecc5e868cb-006543e837-1665-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.266 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.267 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.267 DEBUG:boto:path=/ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.268 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.268 DEBUG:boto:Method: GET 2023-11-02T18:19:35.269 DEBUG:boto:Path: /ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.269 DEBUG:boto:Data: 2023-11-02T18:19:35.269 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.270 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:35.270 DEBUG:boto:Port: 8000 2023-11-02T18:19:35.271 DEBUG:boto:Params: {} 2023-11-02T18:19:35.271 DEBUG:boto:Token: None 2023-11-02T18:19:35.272 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/MULTIPART?versionId=null 2023-11-02T18:19:35.272 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dSf5iHJHleR+zLztF0mpV0O2xN8= 2023-11-02T18:19:35.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dSf5iHJHleR+zLztF0mpV0O2xN8=', 'Content-Length': '0'} 2023-11-02T18:19:35.282 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000688edcfa546181c1-006543e837-1669-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.371 DEBUG:boto:path=/ibtljn-50/ 2023-11-02T18:19:35.372 DEBUG:boto:auth_path=/ibtljn-50/ 2023-11-02T18:19:35.373 DEBUG:boto:path=/ibtljn-50/?prefix=MULTIPART 2023-11-02T18:19:35.373 DEBUG:boto:auth_path=/ibtljn-50/?prefix=MULTIPART 2023-11-02T18:19:35.373 DEBUG:boto:Method: GET 2023-11-02T18:19:35.374 DEBUG:boto:Path: /ibtljn-50/?prefix=MULTIPART 2023-11-02T18:19:35.374 DEBUG:boto:Data: 2023-11-02T18:19:35.375 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.375 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:35.376 DEBUG:boto:Port: 8001 2023-11-02T18:19:35.376 DEBUG:boto:Params: {} 2023-11-02T18:19:35.377 DEBUG:boto:Token: None 2023-11-02T18:19:35.377 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/ 2023-11-02T18:19:35.378 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+bwaw2I7MuRR5um5XAPGeu7l8TU= 2023-11-02T18:19:35.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+bwaw2I7MuRR5um5XAPGeu7l8TU=', 'Content-Length': '0'} 2023-11-02T18:19:35.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094dbfc6d99c56462-006543e837-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.384 DEBUG:boto:b'ibtljn-50MULTIPART1000falseMULTIPART2023-11-02T18:19:14.273Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:35.385 DEBUG:boto:path=/ibtljn-50/ 2023-11-02T18:19:35.385 DEBUG:boto:auth_path=/ibtljn-50/ 2023-11-02T18:19:35.386 DEBUG:boto:path=/ibtljn-50/?prefix=MULTIPART 2023-11-02T18:19:35.386 DEBUG:boto:auth_path=/ibtljn-50/?prefix=MULTIPART 2023-11-02T18:19:35.387 DEBUG:boto:Method: GET 2023-11-02T18:19:35.387 DEBUG:boto:Path: /ibtljn-50/?prefix=MULTIPART 2023-11-02T18:19:35.388 DEBUG:boto:Data: 2023-11-02T18:19:35.388 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.388 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:35.389 DEBUG:boto:Port: 8000 2023-11-02T18:19:35.389 DEBUG:boto:Params: {} 2023-11-02T18:19:35.390 DEBUG:boto:Token: None 2023-11-02T18:19:35.390 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/ 2023-11-02T18:19:35.391 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+bwaw2I7MuRR5um5XAPGeu7l8TU= 2023-11-02T18:19:35.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+bwaw2I7MuRR5um5XAPGeu7l8TU=', 'Content-Length': '0'} 2023-11-02T18:19:35.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065c3d612a1290979-006543e837-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.397 DEBUG:boto:b'ibtljn-50MULTIPART1000falseMULTIPART2023-11-02T18:19:14.273Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:35.397 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.398 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.398 DEBUG:boto:Method: HEAD 2023-11-02T18:19:35.399 DEBUG:boto:Path: /ibtljn-50/MULTIPART 2023-11-02T18:19:35.399 DEBUG:boto:Data: 2023-11-02T18:19:35.400 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.400 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:35.400 DEBUG:boto:Port: 8001 2023-11-02T18:19:35.401 DEBUG:boto:Params: {} 2023-11-02T18:19:35.401 DEBUG:boto:Token: None 2023-11-02T18:19:35.402 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/MULTIPART 2023-11-02T18:19:35.402 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:shwTxR+ceGNYfWrNVCpMAqziEHk= 2023-11-02T18:19:35.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:shwTxR+ceGNYfWrNVCpMAqziEHk=', 'Content-Length': '0'} 2023-11-02T18:19:35.406 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-50:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000006d8376306d7cb418-006543e837-1665-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.406 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.407 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.407 DEBUG:boto:Method: HEAD 2023-11-02T18:19:35.408 DEBUG:boto:Path: /ibtljn-50/MULTIPART 2023-11-02T18:19:35.408 DEBUG:boto:Data: 2023-11-02T18:19:35.408 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.409 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:35.409 DEBUG:boto:Port: 8000 2023-11-02T18:19:35.410 DEBUG:boto:Params: {} 2023-11-02T18:19:35.410 DEBUG:boto:Token: None 2023-11-02T18:19:35.411 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/MULTIPART 2023-11-02T18:19:35.411 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:shwTxR+ceGNYfWrNVCpMAqziEHk= 2023-11-02T18:19:35.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:shwTxR+ceGNYfWrNVCpMAqziEHk=', 'Content-Length': '0'} 2023-11-02T18:19:35.414 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000ffea87147cee6210-006543e837-1669-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-11-02T18:19:35.415 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.416 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.416 DEBUG:boto:Method: GET 2023-11-02T18:19:35.417 DEBUG:boto:Path: /ibtljn-50/MULTIPART 2023-11-02T18:19:35.417 DEBUG:boto:Data: 2023-11-02T18:19:35.418 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.418 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:35.419 DEBUG:boto:Port: 8001 2023-11-02T18:19:35.419 DEBUG:boto:Params: {} 2023-11-02T18:19:35.420 DEBUG:boto:Token: None 2023-11-02T18:19:35.420 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/MULTIPART 2023-11-02T18:19:35.421 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yRO8/ZG3ccruNfrmtfYHbjhYkdM= 2023-11-02T18:19:35.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yRO8/ZG3ccruNfrmtfYHbjhYkdM=', 'Content-Length': '0'} 2023-11-02T18:19:35.425 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-50:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000006758a7de3d88d26b-006543e837-1665-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.507 DEBUG:boto:path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.508 DEBUG:boto:auth_path=/ibtljn-50/MULTIPART 2023-11-02T18:19:35.509 DEBUG:boto:Method: GET 2023-11-02T18:19:35.509 DEBUG:boto:Path: /ibtljn-50/MULTIPART 2023-11-02T18:19:35.510 DEBUG:boto:Data: 2023-11-02T18:19:35.510 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.511 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:35.511 DEBUG:boto:Port: 8000 2023-11-02T18:19:35.511 DEBUG:boto:Params: {} 2023-11-02T18:19:35.512 DEBUG:boto:Token: None 2023-11-02T18:19:35.513 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/MULTIPART 2023-11-02T18:19:35.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yRO8/ZG3ccruNfrmtfYHbjhYkdM= 2023-11-02T18:19:35.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yRO8/ZG3ccruNfrmtfYHbjhYkdM=', 'Content-Length': '0'} 2023-11-02T18:19:35.523 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000060b6edd59096aba9-006543e837-1669-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.613 DEBUG:boto:path=/ibtljn-50/ 2023-11-02T18:19:35.614 DEBUG:boto:auth_path=/ibtljn-50/ 2023-11-02T18:19:35.614 DEBUG:boto:path=/ibtljn-50/?prefix=MULTIPART 2023-11-02T18:19:35.615 DEBUG:boto:auth_path=/ibtljn-50/?prefix=MULTIPART 2023-11-02T18:19:35.615 DEBUG:boto:Method: GET 2023-11-02T18:19:35.616 DEBUG:boto:Path: /ibtljn-50/?prefix=MULTIPART 2023-11-02T18:19:35.616 DEBUG:boto:Data: 2023-11-02T18:19:35.617 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.617 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:19:35.618 DEBUG:boto:Port: 8001 2023-11-02T18:19:35.619 DEBUG:boto:Params: {} 2023-11-02T18:19:35.619 DEBUG:boto:Token: None 2023-11-02T18:19:35.620 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/ 2023-11-02T18:19:35.620 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+bwaw2I7MuRR5um5XAPGeu7l8TU= 2023-11-02T18:19:35.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+bwaw2I7MuRR5um5XAPGeu7l8TU=', 'Content-Length': '0'} 2023-11-02T18:19:35.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dfb54de67151647-006543e837-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.626 DEBUG:boto:b'ibtljn-50MULTIPART1000falseMULTIPART2023-11-02T18:19:14.273Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:35.627 DEBUG:boto:path=/ibtljn-50/ 2023-11-02T18:19:35.627 DEBUG:boto:auth_path=/ibtljn-50/ 2023-11-02T18:19:35.628 DEBUG:boto:path=/ibtljn-50/?prefix=MULTIPART 2023-11-02T18:19:35.628 DEBUG:boto:auth_path=/ibtljn-50/?prefix=MULTIPART 2023-11-02T18:19:35.629 DEBUG:boto:Method: GET 2023-11-02T18:19:35.629 DEBUG:boto:Path: /ibtljn-50/?prefix=MULTIPART 2023-11-02T18:19:35.630 DEBUG:boto:Data: 2023-11-02T18:19:35.630 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.631 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:35.631 DEBUG:boto:Port: 8000 2023-11-02T18:19:35.632 DEBUG:boto:Params: {} 2023-11-02T18:19:35.632 DEBUG:boto:Token: None 2023-11-02T18:19:35.633 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-50/ 2023-11-02T18:19:35.633 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+bwaw2I7MuRR5um5XAPGeu7l8TU= 2023-11-02T18:19:35.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+bwaw2I7MuRR5um5XAPGeu7l8TU=', 'Content-Length': '0'} 2023-11-02T18:19:35.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b019f848b191a047-006543e837-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.640 DEBUG:boto:b'ibtljn-50MULTIPART1000falseMULTIPART2023-11-02T18:19:14.273Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:19:35.640 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-50 zones={a2, a1} 2023-11-02T18:19:35.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2023-11-02T18:19:35.642 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-52 2023-11-02T18:19:35.643 DEBUG:boto:path=/ibtljn-52/ 2023-11-02T18:19:35.643 DEBUG:boto:auth_path=/ibtljn-52/ 2023-11-02T18:19:35.643 DEBUG:boto:Method: PUT 2023-11-02T18:19:35.644 DEBUG:boto:Path: /ibtljn-52/ 2023-11-02T18:19:35.644 DEBUG:boto:Data: 2023-11-02T18:19:35.645 DEBUG:boto:Headers: {} 2023-11-02T18:19:35.645 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:35.645 DEBUG:boto:Port: 8000 2023-11-02T18:19:35.646 DEBUG:boto:Params: {} 2023-11-02T18:19:35.646 DEBUG:boto:Token: None 2023-11-02T18:19:35.647 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:19:35 GMT /ibtljn-52/ 2023-11-02T18:19:35.647 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JPn9PZjOffgvwdWp0sENjiTML3Q= 2023-11-02T18:19:35.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JPn9PZjOffgvwdWp0sENjiTML3Q=', 'Content-Length': '0'} 2023-11-02T18:19:35.680 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eab28bedb53abe06-006543e837-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.681 DEBUG:boto:path=/ibtljn-52/testobj-sse-c 2023-11-02T18:19:35.681 DEBUG:boto:auth_path=/ibtljn-52/testobj-sse-c 2023-11-02T18:19:35.682 DEBUG:boto:Method: PUT 2023-11-02T18:19:35.682 DEBUG:boto:Path: /ibtljn-52/testobj-sse-c 2023-11-02T18:19:35.683 DEBUG:boto:Data: 2023-11-02T18:19:35.683 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-11-02T18:19:35.683 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:35.684 DEBUG:boto:Port: 8000 2023-11-02T18:19:35.684 DEBUG:boto:Params: {} 2023-11-02T18:19:35.685 DEBUG:boto:Token: None 2023-11-02T18:19:35.685 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Thu, 02 Nov 2023 18:19:35 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /ibtljn-52/testobj-sse-c 2023-11-02T18:19:35.686 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZKmDHUPTa0hDkz4fvFK8h+tY8co= 2023-11-02T18:19:35.686 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZKmDHUPTa0hDkz4fvFK8h+tY8co='} 2023-11-02T18:19:35.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx000007c1bf90eb08a72e2-006543e837-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.735 DEBUG:boto:path=/ibtljn-52/testobj-sse-kms 2023-11-02T18:19:35.736 DEBUG:boto:auth_path=/ibtljn-52/testobj-sse-kms 2023-11-02T18:19:35.736 DEBUG:boto:Method: PUT 2023-11-02T18:19:35.737 DEBUG:boto:Path: /ibtljn-52/testobj-sse-kms 2023-11-02T18:19:35.737 DEBUG:boto:Data: 2023-11-02T18:19:35.737 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-11-02T18:19:35.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:19:35.738 DEBUG:boto:Port: 8000 2023-11-02T18:19:35.739 DEBUG:boto:Params: {} 2023-11-02T18:19:35.739 DEBUG:boto:Token: None 2023-11-02T18:19:35.740 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Thu, 02 Nov 2023 18:19:35 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /ibtljn-52/testobj-sse-kms 2023-11-02T18:19:35.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xYsslVsAPKrkmeo9LflitQQyfO8= 2023-11-02T18:19:35.741 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:19:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xYsslVsAPKrkmeo9LflitQQyfO8='} 2023-11-02T18:19:35.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6aWJ0bGpuLTUyL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000d17c5bfab1072d46-006543e837-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:19:35.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:35.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:35.768 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:19:35.812 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:35.812 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:35.870 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:19:35.898 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:19:35.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:35.899 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:19:35.899 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:19:35.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:35.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:35.899 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:19:35.900 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:19:35.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:35.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:35.900 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949175.674176_394.1", 2023-11-02T18:19:35.901 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:19:35.674176Z" 2023-11-02T18:19:35.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:35.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:35.901 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:19:35.901 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:19:35.902 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:35.902 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:19:35.903 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949175.674176_394.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:19:35.903 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:19:35.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:35.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:35.905 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:19:35.994 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:35.994 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:36.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:36.046+0000 7f90c763df40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:19:36.055 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:36.050+0000 7f90c763df40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:19:36.082 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:19:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:19:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:19:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:19:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:19:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:19:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:36.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:36.084 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:19:36.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:36.084 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:19:36.085 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:36.085 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:19:36.085 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:19:36.085 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:36.085 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:19:36.086 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:36.086 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:19:36.086 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:36.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:19:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:19:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:19:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:19:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:19:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:19:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949175.662520_391.1", 2023-11-02T18:19:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:19:35.662520Z", 2023-11-02T18:19:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:19:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:19:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:19:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:19:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:19:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:19:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:19:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:19:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:19:36Z" 2023-11-02T18:19:36.096 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:19:36.097 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949175.662520_391.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:19:35.662520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:19:36Z" } 2023-11-02T18:19:36.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949175.662520_391.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:19:35.662520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:19:36.099 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949175.674176_394.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:19:36.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949175.662520_391.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:19:36.099 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949175.674176_394.1 target=1_1698949175.662520_391.1 2023-11-02T18:19:41.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:41.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:19:41.102 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:19:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:41.206 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:41.198+0000 7f9384cfff40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:19:41.208 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:41.202+0000 7f9384cfff40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:19:41.234 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:19:41.235 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:19:41.235 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:19:41.235 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:19:41.235 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:19:41.236 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:19:41.236 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:41.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:41.236 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:19:41.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:41.237 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:19:41.237 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:41.237 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:19:41.237 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:19:41.237 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:41.238 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:19:41.238 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:41.238 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:19:41.238 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:41.239 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:41.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:41.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:41.239 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:19:41.240 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:41.240 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:19:41.240 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:19:41.240 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:41.240 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:41.241 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:41.241 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:19:41.241 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:41.241 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:41.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:41.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:41.242 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:19:41.242 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:41.242 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:19:41.243 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949175.674176_394.1", 2023-11-02T18:19:41.243 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:41.243 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:41.243 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:41.243 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:19:35.674176Z", 2023-11-02T18:19:41.244 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:41.244 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:41.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:41.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:19:41.244 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:19:41.245 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:19:41.245 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:19:41.245 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:19:41.245 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:19:41.245 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:19:41.246 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:19:41.246 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:19:41.246 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:19:41.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:41.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:19:41.247 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:19:41.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:41.247 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:19:41.247 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:19:41.248 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:19:41.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:19:41.248 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:19:41Z" 2023-11-02T18:19:41.248 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:19:41.249 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949175.674176_394.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:19:35.674176Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:19:41Z" } 2023-11-02T18:19:41.250 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949175.674176_394.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:19:35.674176Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:19:41.250 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949175.674176_394.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:19:41.251 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949175.674176_394.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:19:41.251 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:19:41.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:19:41.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:19:41.252 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:19:41.306 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:19:41.306 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:19:41.374 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:41.366+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:19:46.374 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:46.366+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:19:51.375 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:51.370+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:19:56.376 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:19:56.370+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:20:01.377 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:01.370+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:20:06.378 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:06.370+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:20:11.379 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:11.374+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:20:16.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:16.374+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:20:21.381 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:21.374+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:20:26.382 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:26.378+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:20:31.383 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:31.378+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:20:36.384 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:36.378+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:20:41.385 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:41.378+0000 7fa342febf40 1 waiting to reach incremental sync.. 2023-11-02T18:20:46.389 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:46.382+0000 7fa342febf40 1 bucket sync caught up with source: 2023-11-02T18:20:46.389 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.55.6, , , , 00000000001.41.6, , , , , ] 2023-11-02T18:20:46.389 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.55.6, , , , 00000000001.41.6, , , , , ] 2023-11-02T18:20:46.389 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:46.382+0000 7fa342febf40 0 bucket checkpoint complete 2023-11-02T18:20:46.419 DEBUG:boto:path=/ibtljn-52/ 2023-11-02T18:20:46.419 DEBUG:boto:auth_path=/ibtljn-52/ 2023-11-02T18:20:46.420 DEBUG:boto:Method: HEAD 2023-11-02T18:20:46.420 DEBUG:boto:Path: /ibtljn-52/ 2023-11-02T18:20:46.421 DEBUG:boto:Data: 2023-11-02T18:20:46.421 DEBUG:boto:Headers: {} 2023-11-02T18:20:46.421 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:20:46.422 DEBUG:boto:Port: 8001 2023-11-02T18:20:46.422 DEBUG:boto:Params: {} 2023-11-02T18:20:46.423 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:20:46.423 DEBUG:boto:Token: None 2023-11-02T18:20:46.424 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:20:46 GMT /ibtljn-52/ 2023-11-02T18:20:46.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V0Vc1IhGev0BEMZfpXkCCs9kUr8= 2023-11-02T18:20:46.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V0Vc1IhGev0BEMZfpXkCCs9kUr8=', 'Content-Length': '0'} 2023-11-02T18:20:46.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008813b3d96abe2a6-006543e87e-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.431 DEBUG:boto:path=/ibtljn-52/testobj-sse-c 2023-11-02T18:20:46.431 DEBUG:boto:auth_path=/ibtljn-52/testobj-sse-c 2023-11-02T18:20:46.431 DEBUG:boto:Method: HEAD 2023-11-02T18:20:46.432 DEBUG:boto:Path: /ibtljn-52/testobj-sse-c 2023-11-02T18:20:46.432 DEBUG:boto:Data: 2023-11-02T18:20:46.433 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-11-02T18:20:46.433 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:20:46.433 DEBUG:boto:Port: 8001 2023-11-02T18:20:46.434 DEBUG:boto:Params: {} 2023-11-02T18:20:46.434 DEBUG:boto:Token: None 2023-11-02T18:20:46.435 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:20:46 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /ibtljn-52/testobj-sse-c 2023-11-02T18:20:46.435 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A5fOIOWdJwdrUKarZ719t+hYtTU= 2023-11-02T18:20:46.435 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A5fOIOWdJwdrUKarZ719t+hYtTU=', 'Content-Length': '0'} 2023-11-02T18:20:46.438 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-52:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000349ed4e3fcdaef2d-006543e87e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.439 DEBUG:boto:path=/ibtljn-52/testobj-sse-c 2023-11-02T18:20:46.439 DEBUG:boto:auth_path=/ibtljn-52/testobj-sse-c 2023-11-02T18:20:46.440 DEBUG:boto:Method: GET 2023-11-02T18:20:46.440 DEBUG:boto:Path: /ibtljn-52/testobj-sse-c 2023-11-02T18:20:46.441 DEBUG:boto:Data: 2023-11-02T18:20:46.441 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-11-02T18:20:46.441 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:20:46.442 DEBUG:boto:Port: 8001 2023-11-02T18:20:46.442 DEBUG:boto:Params: {} 2023-11-02T18:20:46.443 DEBUG:boto:Token: None 2023-11-02T18:20:46.443 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:20:46 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /ibtljn-52/testobj-sse-c 2023-11-02T18:20:46.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JDTzfdQGicWcScsgHbbbCEkhEtU= 2023-11-02T18:20:46.444 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JDTzfdQGicWcScsgHbbbCEkhEtU=', 'Content-Length': '0'} 2023-11-02T18:20:46.450 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-52:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx0000021bb4b8997cbd824-006543e87e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.450 DEBUG:boto:path=/ibtljn-52/testobj-sse-kms 2023-11-02T18:20:46.451 DEBUG:boto:auth_path=/ibtljn-52/testobj-sse-kms 2023-11-02T18:20:46.451 DEBUG:boto:Method: HEAD 2023-11-02T18:20:46.451 DEBUG:boto:Path: /ibtljn-52/testobj-sse-kms 2023-11-02T18:20:46.452 DEBUG:boto:Data: 2023-11-02T18:20:46.452 DEBUG:boto:Headers: {} 2023-11-02T18:20:46.453 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:20:46.453 DEBUG:boto:Port: 8001 2023-11-02T18:20:46.453 DEBUG:boto:Params: {} 2023-11-02T18:20:46.454 DEBUG:boto:Token: None 2023-11-02T18:20:46.454 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:20:46 GMT /ibtljn-52/testobj-sse-kms 2023-11-02T18:20:46.455 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JFoYWc3tGCihvzTO1ccslCjABq0= 2023-11-02T18:20:46.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JFoYWc3tGCihvzTO1ccslCjABq0=', 'Content-Length': '0'} 2023-11-02T18:20:46.458 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-52:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000008810b1db5fd9130f-006543e87e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.458 DEBUG:boto:path=/ibtljn-52/testobj-sse-kms 2023-11-02T18:20:46.459 DEBUG:boto:auth_path=/ibtljn-52/testobj-sse-kms 2023-11-02T18:20:46.459 DEBUG:boto:Method: GET 2023-11-02T18:20:46.460 DEBUG:boto:Path: /ibtljn-52/testobj-sse-kms 2023-11-02T18:20:46.460 DEBUG:boto:Data: 2023-11-02T18:20:46.460 DEBUG:boto:Headers: {} 2023-11-02T18:20:46.461 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:20:46.461 DEBUG:boto:Port: 8001 2023-11-02T18:20:46.461 DEBUG:boto:Params: {} 2023-11-02T18:20:46.462 DEBUG:boto:Token: None 2023-11-02T18:20:46.462 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:20:46 GMT /ibtljn-52/testobj-sse-kms 2023-11-02T18:20:46.463 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O2pdJfutBxO4f3pkhWymTP0SEJo= 2023-11-02T18:20:46.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O2pdJfutBxO4f3pkhWymTP0SEJo=', 'Content-Length': '0'} 2023-11-02T18:20:46.466 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:19:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-52:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000b955fff514d9271d-006543e87e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2023-11-02T18:20:46.468 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-53 2023-11-02T18:20:46.468 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:20:46.468 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:20:46.469 DEBUG:boto:Method: PUT 2023-11-02T18:20:46.469 DEBUG:boto:Path: /ibtljn-53/ 2023-11-02T18:20:46.469 DEBUG:boto:Data: 2023-11-02T18:20:46.470 DEBUG:boto:Headers: {} 2023-11-02T18:20:46.470 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:20:46.471 DEBUG:boto:Port: 8000 2023-11-02T18:20:46.471 DEBUG:boto:Params: {} 2023-11-02T18:20:46.471 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:20:46.472 DEBUG:boto:Token: None 2023-11-02T18:20:46.472 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:20:46 GMT /ibtljn-53/ 2023-11-02T18:20:46.473 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NWYm6bbt12yu1Hf1N1X8Rs96f2g= 2023-11-02T18:20:46.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NWYm6bbt12yu1Hf1N1X8Rs96f2g=', 'Content-Length': '0'} 2023-11-02T18:20:46.507 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e613fd26bc559ec3-006543e87e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.508 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:20:46.508 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:20:46.509 DEBUG:boto:Method: HEAD 2023-11-02T18:20:46.509 DEBUG:boto:Path: /ibtljn-53/ 2023-11-02T18:20:46.509 DEBUG:boto:Data: 2023-11-02T18:20:46.510 DEBUG:boto:Headers: {} 2023-11-02T18:20:46.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:20:46.511 DEBUG:boto:Port: 8000 2023-11-02T18:20:46.511 DEBUG:boto:Params: {} 2023-11-02T18:20:46.511 DEBUG:boto:Token: None 2023-11-02T18:20:46.512 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:20:46 GMT /ibtljn-53/ 2023-11-02T18:20:46.512 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bsJ3bDHMUJWFl7mbl9s7DXBk+Ds= 2023-11-02T18:20:46.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bsJ3bDHMUJWFl7mbl9s7DXBk+Ds=', 'Content-Length': '0'} 2023-11-02T18:20:46.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e3becd047827c9d-006543e87e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.516 DEBUG:boto:path=/ibtljn-53/a 2023-11-02T18:20:46.517 DEBUG:boto:auth_path=/ibtljn-53/a 2023-11-02T18:20:46.517 DEBUG:boto:Method: PUT 2023-11-02T18:20:46.517 DEBUG:boto:Path: /ibtljn-53/a 2023-11-02T18:20:46.518 DEBUG:boto:Data: 2023-11-02T18:20:46.518 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:20:46.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:20:46.519 DEBUG:boto:Port: 8000 2023-11-02T18:20:46.519 DEBUG:boto:Params: {} 2023-11-02T18:20:46.520 DEBUG:boto:Token: None 2023-11-02T18:20:46.520 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:20:46 GMT /ibtljn-53/a 2023-11-02T18:20:46.520 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9yVOUFPmd91H6xAEGQVI8VwoKrA= 2023-11-02T18:20:46.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9yVOUFPmd91H6xAEGQVI8VwoKrA='} 2023-11-02T18:20:46.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000561f4b50689755b4-006543e87e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.537 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:20:46.538 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:20:46.538 DEBUG:boto:Method: HEAD 2023-11-02T18:20:46.539 DEBUG:boto:Path: /ibtljn-53/ 2023-11-02T18:20:46.539 DEBUG:boto:Data: 2023-11-02T18:20:46.540 DEBUG:boto:Headers: {} 2023-11-02T18:20:46.540 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:20:46.540 DEBUG:boto:Port: 8000 2023-11-02T18:20:46.541 DEBUG:boto:Params: {} 2023-11-02T18:20:46.541 DEBUG:boto:Token: None 2023-11-02T18:20:46.541 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:20:46 GMT /ibtljn-53/ 2023-11-02T18:20:46.542 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bsJ3bDHMUJWFl7mbl9s7DXBk+Ds= 2023-11-02T18:20:46.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bsJ3bDHMUJWFl7mbl9s7DXBk+Ds=', 'Content-Length': '0'} 2023-11-02T18:20:46.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c20f8f040ecce96-006543e87e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.545 DEBUG:boto:path=/ibtljn-53/b 2023-11-02T18:20:46.546 DEBUG:boto:auth_path=/ibtljn-53/b 2023-11-02T18:20:46.546 DEBUG:boto:Method: PUT 2023-11-02T18:20:46.547 DEBUG:boto:Path: /ibtljn-53/b 2023-11-02T18:20:46.547 DEBUG:boto:Data: 2023-11-02T18:20:46.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:20:46.548 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:20:46.548 DEBUG:boto:Port: 8000 2023-11-02T18:20:46.549 DEBUG:boto:Params: {} 2023-11-02T18:20:46.549 DEBUG:boto:Token: None 2023-11-02T18:20:46.550 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:20:46 GMT /ibtljn-53/b 2023-11-02T18:20:46.550 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0lFT+WRm1Kodgc9hYUTs2p3XF/g= 2023-11-02T18:20:46.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0lFT+WRm1Kodgc9hYUTs2p3XF/g='} 2023-11-02T18:20:46.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000140b5fd49d53c722-006543e87e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.595 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:20:46.595 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:20:46.596 DEBUG:boto:Method: HEAD 2023-11-02T18:20:46.596 DEBUG:boto:Path: /ibtljn-53/ 2023-11-02T18:20:46.596 DEBUG:boto:Data: 2023-11-02T18:20:46.597 DEBUG:boto:Headers: {} 2023-11-02T18:20:46.597 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:20:46.597 DEBUG:boto:Port: 8000 2023-11-02T18:20:46.598 DEBUG:boto:Params: {} 2023-11-02T18:20:46.598 DEBUG:boto:Token: None 2023-11-02T18:20:46.599 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:20:46 GMT /ibtljn-53/ 2023-11-02T18:20:46.599 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bsJ3bDHMUJWFl7mbl9s7DXBk+Ds= 2023-11-02T18:20:46.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bsJ3bDHMUJWFl7mbl9s7DXBk+Ds=', 'Content-Length': '0'} 2023-11-02T18:20:46.604 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000497679f15ca72642-006543e87e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.605 DEBUG:boto:path=/ibtljn-53/c 2023-11-02T18:20:46.605 DEBUG:boto:auth_path=/ibtljn-53/c 2023-11-02T18:20:46.606 DEBUG:boto:Method: PUT 2023-11-02T18:20:46.606 DEBUG:boto:Path: /ibtljn-53/c 2023-11-02T18:20:46.607 DEBUG:boto:Data: 2023-11-02T18:20:46.607 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:20:46.607 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:20:46.608 DEBUG:boto:Port: 8000 2023-11-02T18:20:46.608 DEBUG:boto:Params: {} 2023-11-02T18:20:46.609 DEBUG:boto:Token: None 2023-11-02T18:20:46.609 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:20:46 GMT /ibtljn-53/c 2023-11-02T18:20:46.610 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RYDheu8ARf1NgzyZjC7yOhVVqzU= 2023-11-02T18:20:46.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RYDheu8ARf1NgzyZjC7yOhVVqzU='} 2023-11-02T18:20:46.619 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d3b37e040b66807-006543e87e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.619 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:20:46.620 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:20:46.620 DEBUG:boto:Method: HEAD 2023-11-02T18:20:46.621 DEBUG:boto:Path: /ibtljn-53/ 2023-11-02T18:20:46.621 DEBUG:boto:Data: 2023-11-02T18:20:46.621 DEBUG:boto:Headers: {} 2023-11-02T18:20:46.622 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:20:46.622 DEBUG:boto:Port: 8000 2023-11-02T18:20:46.623 DEBUG:boto:Params: {} 2023-11-02T18:20:46.623 DEBUG:boto:Token: None 2023-11-02T18:20:46.624 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:20:46 GMT /ibtljn-53/ 2023-11-02T18:20:46.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bsJ3bDHMUJWFl7mbl9s7DXBk+Ds= 2023-11-02T18:20:46.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bsJ3bDHMUJWFl7mbl9s7DXBk+Ds=', 'Content-Length': '0'} 2023-11-02T18:20:46.628 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a132386c4b6645c0-006543e87e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.629 DEBUG:boto:path=/ibtljn-53/d 2023-11-02T18:20:46.629 DEBUG:boto:auth_path=/ibtljn-53/d 2023-11-02T18:20:46.630 DEBUG:boto:Method: PUT 2023-11-02T18:20:46.630 DEBUG:boto:Path: /ibtljn-53/d 2023-11-02T18:20:46.630 DEBUG:boto:Data: 2023-11-02T18:20:46.631 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:20:46.631 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:20:46.632 DEBUG:boto:Port: 8000 2023-11-02T18:20:46.632 DEBUG:boto:Params: {} 2023-11-02T18:20:46.633 DEBUG:boto:Token: None 2023-11-02T18:20:46.633 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:20:46 GMT /ibtljn-53/d 2023-11-02T18:20:46.634 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w+QAHZvdozTU3pNWvO+ArqkC/YA= 2023-11-02T18:20:46.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:20:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w+QAHZvdozTU3pNWvO+ArqkC/YA='} 2023-11-02T18:20:46.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fe1fa89da8872d6-006543e87e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:20:46.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:20:46.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:20:46.680 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:20:46.725 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:20:46.725 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:20:46.786 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:20:46.815 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:20:46.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:20:46.815 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:20:46.815 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:20:46.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:20:46.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:20:46.816 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:20:46.816 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:20:46.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:20:46.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:20:46.817 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949246.502324_410.1", 2023-11-02T18:20:46.817 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:20:46.502324Z" 2023-11-02T18:20:46.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:20:46.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:20:46.818 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:20:46.818 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:20:46.819 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:20:46.819 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:20:46.820 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949246.502324_410.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:20:46.820 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:20:46.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:20:46.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:20:46.822 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:20:46.866 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:20:46.866 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:20:46.934 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:46.930+0000 7f80984f1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:20:46.935 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:46.930+0000 7f80984f1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:20:46.964 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:20:46.964 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:20:46.964 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:20:46.965 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:20:46.965 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:20:46.965 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:20:46.965 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:20:46.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:20:46.966 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:20:46.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:20:46.967 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:20:46.967 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:20:46.967 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:20:46.967 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:20:46.968 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:20:46.968 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:20:46.968 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:20:46.968 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:20:46.969 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:20:46.969 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:20:46.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:20:46.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:20:46.970 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:20:46.970 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:20:46.970 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:20:46.970 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:20:46.971 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:20:46.971 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:20:46.971 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:20:46.972 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:20:46.972 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:20:46.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:20:46.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:20:46.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:20:46.973 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:20:46.973 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:20:46.973 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:20:46.974 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949246.502324_410.1", 2023-11-02T18:20:46.974 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:20:46.974 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:20:46.974 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:20:46.975 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:20:46.502324Z", 2023-11-02T18:20:46.975 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:20:46.975 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:20:46.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:20:46.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:20:46.976 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:20:46.976 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:20:46.976 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:20:46.977 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:20:46.977 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:20:46.977 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:20:46.978 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:20:46.978 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:20:46.978 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:20:46.978 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:20:46.979 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:20:46.979 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:20:46.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:20:46.979 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:20:46.980 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:20:46.980 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:20:46.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:20:46.980 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:20:46Z" 2023-11-02T18:20:46.981 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:20:46.982 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949246.502324_410.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:20:46.502324Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:20:46Z" } 2023-11-02T18:20:46.982 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949246.502324_410.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:20:46.502324Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:20:46.983 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949246.502324_410.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:20:46.983 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949246.502324_410.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:20:46.984 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:20:46.984 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-53 2023-11-02T18:20:46.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:20:46.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:20:46.986 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-53 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:20:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:20:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:20:47.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:47.098+0000 7f8fac214f40 1 waiting to reach incremental sync.. 2023-11-02T18:20:52.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:52.098+0000 7f8fac214f40 1 waiting to reach incremental sync.. 2023-11-02T18:20:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:20:57.098+0000 7f8fac214f40 1 waiting to reach incremental sync.. 2023-11-02T18:21:02.106 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:02.102+0000 7f8fac214f40 1 waiting to reach incremental sync.. 2023-11-02T18:21:07.109 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:07.102+0000 7f8fac214f40 1 bucket sync caught up with source: 2023-11-02T18:21:07.110 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.27.6, , , , 00000000001.21.6, , , 00000000001.52.6, , 00000000001.23.6] 2023-11-02T18:21:07.110 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.27.6, , , , 00000000001.21.6, , , 00000000001.52.6, , 00000000001.23.6] 2023-11-02T18:21:07.110 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:07.102+0000 7f8fac214f40 0 bucket checkpoint complete 2023-11-02T18:21:07.139 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-53 2023-11-02T18:21:07.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:21:07.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:21:07.141 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-53 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:21:07.192 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:07.192 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:07.254 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:07.250+0000 7fd7b4868f40 1 waiting to reach incremental sync.. 2023-11-02T18:21:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:12.254+0000 7fd7b4868f40 1 bucket sync caught up with source: 2023-11-02T18:21:12.258 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.124.6, , , , 00000000001.68.6, , , 00000000001.123.6, , 00000000001.97.6] 2023-11-02T18:21:12.258 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.124.6, , , , 00000000001.68.6, , , 00000000001.123.6, , 00000000001.97.6] 2023-11-02T18:21:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:12.254+0000 7fd7b4868f40 0 bucket checkpoint complete 2023-11-02T18:21:12.285 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-53 zones={a2, a1} 2023-11-02T18:21:12.286 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.287 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.287 DEBUG:boto:Method: HEAD 2023-11-02T18:21:12.287 DEBUG:boto:Path: /ibtljn-53/ 2023-11-02T18:21:12.288 DEBUG:boto:Data: 2023-11-02T18:21:12.288 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.289 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.289 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.290 DEBUG:boto:Params: {} 2023-11-02T18:21:12.290 DEBUG:boto:Token: None 2023-11-02T18:21:12.291 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/ 2023-11-02T18:21:12.292 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qM6bXwhIM/EOf5m4uqAi+9D7pmw= 2023-11-02T18:21:12.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qM6bXwhIM/EOf5m4uqAi+9D7pmw=', 'Content-Length': '0'} 2023-11-02T18:21:12.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5116e49a27d3ad8-006543e898-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.297 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.298 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.299 DEBUG:boto:Method: HEAD 2023-11-02T18:21:12.299 DEBUG:boto:Path: /ibtljn-53/ 2023-11-02T18:21:12.300 DEBUG:boto:Data: 2023-11-02T18:21:12.300 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.301 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.301 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.302 DEBUG:boto:Params: {} 2023-11-02T18:21:12.302 DEBUG:boto:Token: None 2023-11-02T18:21:12.303 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/ 2023-11-02T18:21:12.303 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qM6bXwhIM/EOf5m4uqAi+9D7pmw= 2023-11-02T18:21:12.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qM6bXwhIM/EOf5m4uqAi+9D7pmw=', 'Content-Length': '0'} 2023-11-02T18:21:12.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007812c158ce5e3222-006543e898-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.307 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:21:12.308 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.308 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.309 DEBUG:boto:path=/ibtljn-53/?versions&max-keys=999 2023-11-02T18:21:12.309 DEBUG:boto:auth_path=/ibtljn-53/?versions&max-keys=999 2023-11-02T18:21:12.310 DEBUG:boto:Method: GET 2023-11-02T18:21:12.310 DEBUG:boto:Path: /ibtljn-53/?versions&max-keys=999 2023-11-02T18:21:12.311 DEBUG:boto:Data: 2023-11-02T18:21:12.311 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.311 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.312 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.312 DEBUG:boto:Params: {} 2023-11-02T18:21:12.313 DEBUG:boto:Token: None 2023-11-02T18:21:12.313 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/?versions 2023-11-02T18:21:12.314 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XO1R02X1y8FeXnzAaNbT+3Guoiw= 2023-11-02T18:21:12.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XO1R02X1y8FeXnzAaNbT+3Guoiw=', 'Content-Length': '0'} 2023-11-02T18:21:12.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ccf28ddf5cd2206-006543e898-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.319 DEBUG:boto:b'ibtljn-53999falseanulltrue2023-11-02T18:20:46.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:20:46.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:20:46.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:20:46.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:12.320 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:21:12.320 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:21:12.321 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:21:12.321 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:21:12.321 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:21:12.322 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.322 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.323 DEBUG:boto:path=/ibtljn-53/?versions&max-keys=999 2023-11-02T18:21:12.323 DEBUG:boto:auth_path=/ibtljn-53/?versions&max-keys=999 2023-11-02T18:21:12.324 DEBUG:boto:Method: GET 2023-11-02T18:21:12.324 DEBUG:boto:Path: /ibtljn-53/?versions&max-keys=999 2023-11-02T18:21:12.324 DEBUG:boto:Data: 2023-11-02T18:21:12.325 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.325 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.326 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.326 DEBUG:boto:Params: {} 2023-11-02T18:21:12.327 DEBUG:boto:Token: None 2023-11-02T18:21:12.327 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/?versions 2023-11-02T18:21:12.328 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XO1R02X1y8FeXnzAaNbT+3Guoiw= 2023-11-02T18:21:12.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XO1R02X1y8FeXnzAaNbT+3Guoiw=', 'Content-Length': '0'} 2023-11-02T18:21:12.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a36a9e8c4bf62d47-006543e898-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.334 DEBUG:boto:b'ibtljn-53999falseanulltrue2023-11-02T18:20:46.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:20:46.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:20:46.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:20:46.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:12.335 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:21:12.335 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:21:12.336 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:21:12.336 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:21:12.337 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.337 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.338 DEBUG:boto:path=/ibtljn-53/?versions&max-keys=999 2023-11-02T18:21:12.338 DEBUG:boto:auth_path=/ibtljn-53/?versions&max-keys=999 2023-11-02T18:21:12.339 DEBUG:boto:Method: GET 2023-11-02T18:21:12.339 DEBUG:boto:Path: /ibtljn-53/?versions&max-keys=999 2023-11-02T18:21:12.340 DEBUG:boto:Data: 2023-11-02T18:21:12.340 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.341 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.341 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.342 DEBUG:boto:Params: {} 2023-11-02T18:21:12.342 DEBUG:boto:Token: None 2023-11-02T18:21:12.343 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/?versions 2023-11-02T18:21:12.343 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XO1R02X1y8FeXnzAaNbT+3Guoiw= 2023-11-02T18:21:12.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XO1R02X1y8FeXnzAaNbT+3Guoiw=', 'Content-Length': '0'} 2023-11-02T18:21:12.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000629970e10521c04-006543e898-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.394 DEBUG:boto:b'ibtljn-53999falseanulltrue2023-11-02T18:20:46.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:20:46.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:20:46.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:20:46.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:12.395 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.396 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.396 DEBUG:boto:path=/ibtljn-53/?versions&max-keys=999 2023-11-02T18:21:12.397 DEBUG:boto:auth_path=/ibtljn-53/?versions&max-keys=999 2023-11-02T18:21:12.397 DEBUG:boto:Method: GET 2023-11-02T18:21:12.397 DEBUG:boto:Path: /ibtljn-53/?versions&max-keys=999 2023-11-02T18:21:12.398 DEBUG:boto:Data: 2023-11-02T18:21:12.398 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.399 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.399 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.399 DEBUG:boto:Params: {} 2023-11-02T18:21:12.400 DEBUG:boto:Token: None 2023-11-02T18:21:12.401 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/?versions 2023-11-02T18:21:12.401 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XO1R02X1y8FeXnzAaNbT+3Guoiw= 2023-11-02T18:21:12.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XO1R02X1y8FeXnzAaNbT+3Guoiw=', 'Content-Length': '0'} 2023-11-02T18:21:12.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019bbfed123bd391b-006543e898-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.408 DEBUG:boto:b'ibtljn-53999falseanulltrue2023-11-02T18:20:46.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:20:46.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:20:46.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:20:46.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:12.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:21:12.409 DEBUG:boto:path=/ibtljn-53/a 2023-11-02T18:21:12.409 DEBUG:boto:auth_path=/ibtljn-53/a 2023-11-02T18:21:12.410 DEBUG:boto:path=/ibtljn-53/a?versionId=null 2023-11-02T18:21:12.410 DEBUG:boto:auth_path=/ibtljn-53/a?versionId=null 2023-11-02T18:21:12.411 DEBUG:boto:Method: GET 2023-11-02T18:21:12.411 DEBUG:boto:Path: /ibtljn-53/a?versionId=null 2023-11-02T18:21:12.412 DEBUG:boto:Data: 2023-11-02T18:21:12.412 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.413 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.413 DEBUG:boto:Params: {} 2023-11-02T18:21:12.414 DEBUG:boto:Token: None 2023-11-02T18:21:12.414 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/a?versionId=null 2023-11-02T18:21:12.415 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0faJXpMaOjRzPNcWz+t1hhz/DCk= 2023-11-02T18:21:12.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0faJXpMaOjRzPNcWz+t1hhz/DCk=', 'Content-Length': '0'} 2023-11-02T18:21:12.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3c3477e840bbf5b-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.419 DEBUG:boto:path=/ibtljn-53/a 2023-11-02T18:21:12.420 DEBUG:boto:auth_path=/ibtljn-53/a 2023-11-02T18:21:12.420 DEBUG:boto:path=/ibtljn-53/a?versionId=null 2023-11-02T18:21:12.421 DEBUG:boto:auth_path=/ibtljn-53/a?versionId=null 2023-11-02T18:21:12.421 DEBUG:boto:Method: GET 2023-11-02T18:21:12.422 DEBUG:boto:Path: /ibtljn-53/a?versionId=null 2023-11-02T18:21:12.422 DEBUG:boto:Data: 2023-11-02T18:21:12.423 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.423 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.424 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.424 DEBUG:boto:Params: {} 2023-11-02T18:21:12.425 DEBUG:boto:Token: None 2023-11-02T18:21:12.426 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/a?versionId=null 2023-11-02T18:21:12.427 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0faJXpMaOjRzPNcWz+t1hhz/DCk= 2023-11-02T18:21:12.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0faJXpMaOjRzPNcWz+t1hhz/DCk=', 'Content-Length': '0'} 2023-11-02T18:21:12.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081a180001d315082-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.431 DEBUG:boto:path=/ibtljn-53/a 2023-11-02T18:21:12.431 DEBUG:boto:auth_path=/ibtljn-53/a 2023-11-02T18:21:12.432 DEBUG:boto:path=/ibtljn-53/a?versionId=null 2023-11-02T18:21:12.432 DEBUG:boto:auth_path=/ibtljn-53/a?versionId=null 2023-11-02T18:21:12.433 DEBUG:boto:Method: HEAD 2023-11-02T18:21:12.433 DEBUG:boto:Path: /ibtljn-53/a?versionId=null 2023-11-02T18:21:12.433 DEBUG:boto:Data: 2023-11-02T18:21:12.434 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.434 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.435 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.435 DEBUG:boto:Params: {} 2023-11-02T18:21:12.435 DEBUG:boto:Token: None 2023-11-02T18:21:12.436 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/a?versionId=null 2023-11-02T18:21:12.436 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fida9M/Rn1j2C5yS0sjjTrr4+bY= 2023-11-02T18:21:12.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fida9M/Rn1j2C5yS0sjjTrr4+bY=', 'Content-Length': '0'} 2023-11-02T18:21:12.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f51c20883ed23ca-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.440 DEBUG:boto:path=/ibtljn-53/a 2023-11-02T18:21:12.441 DEBUG:boto:auth_path=/ibtljn-53/a 2023-11-02T18:21:12.441 DEBUG:boto:path=/ibtljn-53/a?versionId=null 2023-11-02T18:21:12.442 DEBUG:boto:auth_path=/ibtljn-53/a?versionId=null 2023-11-02T18:21:12.442 DEBUG:boto:Method: HEAD 2023-11-02T18:21:12.442 DEBUG:boto:Path: /ibtljn-53/a?versionId=null 2023-11-02T18:21:12.443 DEBUG:boto:Data: 2023-11-02T18:21:12.443 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.444 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.444 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.445 DEBUG:boto:Params: {} 2023-11-02T18:21:12.445 DEBUG:boto:Token: None 2023-11-02T18:21:12.445 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/a?versionId=null 2023-11-02T18:21:12.446 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fida9M/Rn1j2C5yS0sjjTrr4+bY= 2023-11-02T18:21:12.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fida9M/Rn1j2C5yS0sjjTrr4+bY=', 'Content-Length': '0'} 2023-11-02T18:21:12.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044e76c93d7762a68-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:21:12.450 DEBUG:boto:path=/ibtljn-53/a 2023-11-02T18:21:12.451 DEBUG:boto:auth_path=/ibtljn-53/a 2023-11-02T18:21:12.451 DEBUG:boto:path=/ibtljn-53/a?versionId=null 2023-11-02T18:21:12.452 DEBUG:boto:auth_path=/ibtljn-53/a?versionId=null 2023-11-02T18:21:12.452 DEBUG:boto:Method: GET 2023-11-02T18:21:12.453 DEBUG:boto:Path: /ibtljn-53/a?versionId=null 2023-11-02T18:21:12.454 DEBUG:boto:Data: 2023-11-02T18:21:12.454 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.455 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.456 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.457 DEBUG:boto:Params: {} 2023-11-02T18:21:12.457 DEBUG:boto:Token: None 2023-11-02T18:21:12.458 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/a?versionId=null 2023-11-02T18:21:12.459 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0faJXpMaOjRzPNcWz+t1hhz/DCk= 2023-11-02T18:21:12.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0faJXpMaOjRzPNcWz+t1hhz/DCk=', 'Content-Length': '0'} 2023-11-02T18:21:12.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001561ac29a9555813-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.506 DEBUG:boto:path=/ibtljn-53/a 2023-11-02T18:21:12.507 DEBUG:boto:auth_path=/ibtljn-53/a 2023-11-02T18:21:12.507 DEBUG:boto:path=/ibtljn-53/a?versionId=null 2023-11-02T18:21:12.508 DEBUG:boto:auth_path=/ibtljn-53/a?versionId=null 2023-11-02T18:21:12.508 DEBUG:boto:Method: GET 2023-11-02T18:21:12.508 DEBUG:boto:Path: /ibtljn-53/a?versionId=null 2023-11-02T18:21:12.509 DEBUG:boto:Data: 2023-11-02T18:21:12.509 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.510 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.511 DEBUG:boto:Params: {} 2023-11-02T18:21:12.511 DEBUG:boto:Token: None 2023-11-02T18:21:12.512 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/a?versionId=null 2023-11-02T18:21:12.512 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0faJXpMaOjRzPNcWz+t1hhz/DCk= 2023-11-02T18:21:12.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0faJXpMaOjRzPNcWz+t1hhz/DCk=', 'Content-Length': '0'} 2023-11-02T18:21:12.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc957812ffe1247-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.517 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.517 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.518 DEBUG:boto:path=/ibtljn-53/?prefix=a 2023-11-02T18:21:12.518 DEBUG:boto:auth_path=/ibtljn-53/?prefix=a 2023-11-02T18:21:12.518 DEBUG:boto:Method: GET 2023-11-02T18:21:12.519 DEBUG:boto:Path: /ibtljn-53/?prefix=a 2023-11-02T18:21:12.519 DEBUG:boto:Data: 2023-11-02T18:21:12.520 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.520 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.521 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.521 DEBUG:boto:Params: {} 2023-11-02T18:21:12.521 DEBUG:boto:Token: None 2023-11-02T18:21:12.522 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/ 2023-11-02T18:21:12.522 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI= 2023-11-02T18:21:12.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI=', 'Content-Length': '0'} 2023-11-02T18:21:12.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000434b098e5995b8aa-006543e898-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.528 DEBUG:boto:b'ibtljn-53a1000falsea2023-11-02T18:20:46.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:12.529 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.529 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.530 DEBUG:boto:path=/ibtljn-53/?prefix=a 2023-11-02T18:21:12.530 DEBUG:boto:auth_path=/ibtljn-53/?prefix=a 2023-11-02T18:21:12.531 DEBUG:boto:Method: GET 2023-11-02T18:21:12.531 DEBUG:boto:Path: /ibtljn-53/?prefix=a 2023-11-02T18:21:12.532 DEBUG:boto:Data: 2023-11-02T18:21:12.532 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.533 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.533 DEBUG:boto:Params: {} 2023-11-02T18:21:12.534 DEBUG:boto:Token: None 2023-11-02T18:21:12.534 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/ 2023-11-02T18:21:12.535 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI= 2023-11-02T18:21:12.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI=', 'Content-Length': '0'} 2023-11-02T18:21:12.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9f8a146fc68bf78-006543e898-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.540 DEBUG:boto:b'ibtljn-53a1000falsea2023-11-02T18:20:46.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:12.541 DEBUG:boto:path=/ibtljn-53/a 2023-11-02T18:21:12.541 DEBUG:boto:auth_path=/ibtljn-53/a 2023-11-02T18:21:12.542 DEBUG:boto:Method: HEAD 2023-11-02T18:21:12.542 DEBUG:boto:Path: /ibtljn-53/a 2023-11-02T18:21:12.543 DEBUG:boto:Data: 2023-11-02T18:21:12.543 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.543 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.544 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.544 DEBUG:boto:Params: {} 2023-11-02T18:21:12.545 DEBUG:boto:Token: None 2023-11-02T18:21:12.545 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/a 2023-11-02T18:21:12.546 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KSdV4J5siqdGH2b71fyjUskGoRY= 2023-11-02T18:21:12.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KSdV4J5siqdGH2b71fyjUskGoRY=', 'Content-Length': '0'} 2023-11-02T18:21:12.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa599157b347de5e-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.550 DEBUG:boto:path=/ibtljn-53/a 2023-11-02T18:21:12.551 DEBUG:boto:auth_path=/ibtljn-53/a 2023-11-02T18:21:12.552 DEBUG:boto:Method: HEAD 2023-11-02T18:21:12.552 DEBUG:boto:Path: /ibtljn-53/a 2023-11-02T18:21:12.553 DEBUG:boto:Data: 2023-11-02T18:21:12.554 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.555 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.555 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.556 DEBUG:boto:Params: {} 2023-11-02T18:21:12.557 DEBUG:boto:Token: None 2023-11-02T18:21:12.558 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/a 2023-11-02T18:21:12.558 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KSdV4J5siqdGH2b71fyjUskGoRY= 2023-11-02T18:21:12.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KSdV4J5siqdGH2b71fyjUskGoRY=', 'Content-Length': '0'} 2023-11-02T18:21:12.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000032e364d91259fd0-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:21:12.562 DEBUG:boto:path=/ibtljn-53/a 2023-11-02T18:21:12.563 DEBUG:boto:auth_path=/ibtljn-53/a 2023-11-02T18:21:12.563 DEBUG:boto:Method: GET 2023-11-02T18:21:12.564 DEBUG:boto:Path: /ibtljn-53/a 2023-11-02T18:21:12.564 DEBUG:boto:Data: 2023-11-02T18:21:12.565 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.565 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.566 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.566 DEBUG:boto:Params: {} 2023-11-02T18:21:12.567 DEBUG:boto:Token: None 2023-11-02T18:21:12.567 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/a 2023-11-02T18:21:12.568 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sLXXgKrWlXHnFIWBiS3ostWX7nc= 2023-11-02T18:21:12.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sLXXgKrWlXHnFIWBiS3ostWX7nc=', 'Content-Length': '0'} 2023-11-02T18:21:12.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6154141628bd755-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.572 DEBUG:boto:path=/ibtljn-53/a 2023-11-02T18:21:12.572 DEBUG:boto:auth_path=/ibtljn-53/a 2023-11-02T18:21:12.572 DEBUG:boto:Method: GET 2023-11-02T18:21:12.573 DEBUG:boto:Path: /ibtljn-53/a 2023-11-02T18:21:12.573 DEBUG:boto:Data: 2023-11-02T18:21:12.574 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.575 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.575 DEBUG:boto:Params: {} 2023-11-02T18:21:12.575 DEBUG:boto:Token: None 2023-11-02T18:21:12.576 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/a 2023-11-02T18:21:12.577 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sLXXgKrWlXHnFIWBiS3ostWX7nc= 2023-11-02T18:21:12.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sLXXgKrWlXHnFIWBiS3ostWX7nc=', 'Content-Length': '0'} 2023-11-02T18:21:12.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007919a520a0eb6300-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.581 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.581 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.582 DEBUG:boto:path=/ibtljn-53/?prefix=a 2023-11-02T18:21:12.582 DEBUG:boto:auth_path=/ibtljn-53/?prefix=a 2023-11-02T18:21:12.583 DEBUG:boto:Method: GET 2023-11-02T18:21:12.583 DEBUG:boto:Path: /ibtljn-53/?prefix=a 2023-11-02T18:21:12.583 DEBUG:boto:Data: 2023-11-02T18:21:12.584 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.584 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.585 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.585 DEBUG:boto:Params: {} 2023-11-02T18:21:12.585 DEBUG:boto:Token: None 2023-11-02T18:21:12.586 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/ 2023-11-02T18:21:12.587 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI= 2023-11-02T18:21:12.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI=', 'Content-Length': '0'} 2023-11-02T18:21:12.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1d0427cf83cf94e-006543e898-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.634 DEBUG:boto:b'ibtljn-53a1000falsea2023-11-02T18:20:46.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:12.635 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.635 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.636 DEBUG:boto:path=/ibtljn-53/?prefix=a 2023-11-02T18:21:12.636 DEBUG:boto:auth_path=/ibtljn-53/?prefix=a 2023-11-02T18:21:12.637 DEBUG:boto:Method: GET 2023-11-02T18:21:12.637 DEBUG:boto:Path: /ibtljn-53/?prefix=a 2023-11-02T18:21:12.637 DEBUG:boto:Data: 2023-11-02T18:21:12.638 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.638 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.639 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.639 DEBUG:boto:Params: {} 2023-11-02T18:21:12.640 DEBUG:boto:Token: None 2023-11-02T18:21:12.640 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/ 2023-11-02T18:21:12.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI= 2023-11-02T18:21:12.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI=', 'Content-Length': '0'} 2023-11-02T18:21:12.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007793c6702f44b0eb-006543e898-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.647 DEBUG:boto:b'ibtljn-53a1000falsea2023-11-02T18:20:46.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:12.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:21:12.648 DEBUG:boto:path=/ibtljn-53/b 2023-11-02T18:21:12.649 DEBUG:boto:auth_path=/ibtljn-53/b 2023-11-02T18:21:12.649 DEBUG:boto:path=/ibtljn-53/b?versionId=null 2023-11-02T18:21:12.649 DEBUG:boto:auth_path=/ibtljn-53/b?versionId=null 2023-11-02T18:21:12.650 DEBUG:boto:Method: GET 2023-11-02T18:21:12.650 DEBUG:boto:Path: /ibtljn-53/b?versionId=null 2023-11-02T18:21:12.651 DEBUG:boto:Data: 2023-11-02T18:21:12.651 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.652 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.652 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.653 DEBUG:boto:Params: {} 2023-11-02T18:21:12.653 DEBUG:boto:Token: None 2023-11-02T18:21:12.653 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/b?versionId=null 2023-11-02T18:21:12.654 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3b62s9WE3BD0B+ZpfJZq5g9qFY4= 2023-11-02T18:21:12.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3b62s9WE3BD0B+ZpfJZq5g9qFY4=', 'Content-Length': '0'} 2023-11-02T18:21:12.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005802d931e5f8bf53-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.658 DEBUG:boto:path=/ibtljn-53/b 2023-11-02T18:21:12.659 DEBUG:boto:auth_path=/ibtljn-53/b 2023-11-02T18:21:12.659 DEBUG:boto:path=/ibtljn-53/b?versionId=null 2023-11-02T18:21:12.660 DEBUG:boto:auth_path=/ibtljn-53/b?versionId=null 2023-11-02T18:21:12.660 DEBUG:boto:Method: GET 2023-11-02T18:21:12.661 DEBUG:boto:Path: /ibtljn-53/b?versionId=null 2023-11-02T18:21:12.661 DEBUG:boto:Data: 2023-11-02T18:21:12.661 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.662 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.662 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.663 DEBUG:boto:Params: {} 2023-11-02T18:21:12.663 DEBUG:boto:Token: None 2023-11-02T18:21:12.664 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/b?versionId=null 2023-11-02T18:21:12.664 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3b62s9WE3BD0B+ZpfJZq5g9qFY4= 2023-11-02T18:21:12.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3b62s9WE3BD0B+ZpfJZq5g9qFY4=', 'Content-Length': '0'} 2023-11-02T18:21:12.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c69b1681455be07e-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.710 DEBUG:boto:path=/ibtljn-53/b 2023-11-02T18:21:12.711 DEBUG:boto:auth_path=/ibtljn-53/b 2023-11-02T18:21:12.711 DEBUG:boto:path=/ibtljn-53/b?versionId=null 2023-11-02T18:21:12.712 DEBUG:boto:auth_path=/ibtljn-53/b?versionId=null 2023-11-02T18:21:12.712 DEBUG:boto:Method: HEAD 2023-11-02T18:21:12.713 DEBUG:boto:Path: /ibtljn-53/b?versionId=null 2023-11-02T18:21:12.713 DEBUG:boto:Data: 2023-11-02T18:21:12.713 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.714 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.714 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.715 DEBUG:boto:Params: {} 2023-11-02T18:21:12.715 DEBUG:boto:Token: None 2023-11-02T18:21:12.716 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/b?versionId=null 2023-11-02T18:21:12.716 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rn3SniTHk1jn71e0Yav3ZbkcaSA= 2023-11-02T18:21:12.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rn3SniTHk1jn71e0Yav3ZbkcaSA=', 'Content-Length': '0'} 2023-11-02T18:21:12.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff45cebc831093d-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.720 DEBUG:boto:path=/ibtljn-53/b 2023-11-02T18:21:12.721 DEBUG:boto:auth_path=/ibtljn-53/b 2023-11-02T18:21:12.721 DEBUG:boto:path=/ibtljn-53/b?versionId=null 2023-11-02T18:21:12.722 DEBUG:boto:auth_path=/ibtljn-53/b?versionId=null 2023-11-02T18:21:12.722 DEBUG:boto:Method: HEAD 2023-11-02T18:21:12.723 DEBUG:boto:Path: /ibtljn-53/b?versionId=null 2023-11-02T18:21:12.723 DEBUG:boto:Data: 2023-11-02T18:21:12.723 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.724 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.724 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.725 DEBUG:boto:Params: {} 2023-11-02T18:21:12.725 DEBUG:boto:Token: None 2023-11-02T18:21:12.725 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/b?versionId=null 2023-11-02T18:21:12.726 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rn3SniTHk1jn71e0Yav3ZbkcaSA= 2023-11-02T18:21:12.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rn3SniTHk1jn71e0Yav3ZbkcaSA=', 'Content-Length': '0'} 2023-11-02T18:21:12.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009faa0cb02ba86bd6-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:21:12.730 DEBUG:boto:path=/ibtljn-53/b 2023-11-02T18:21:12.731 DEBUG:boto:auth_path=/ibtljn-53/b 2023-11-02T18:21:12.731 DEBUG:boto:path=/ibtljn-53/b?versionId=null 2023-11-02T18:21:12.732 DEBUG:boto:auth_path=/ibtljn-53/b?versionId=null 2023-11-02T18:21:12.732 DEBUG:boto:Method: GET 2023-11-02T18:21:12.733 DEBUG:boto:Path: /ibtljn-53/b?versionId=null 2023-11-02T18:21:12.733 DEBUG:boto:Data: 2023-11-02T18:21:12.734 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.735 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.735 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.736 DEBUG:boto:Params: {} 2023-11-02T18:21:12.736 DEBUG:boto:Token: None 2023-11-02T18:21:12.737 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/b?versionId=null 2023-11-02T18:21:12.737 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3b62s9WE3BD0B+ZpfJZq5g9qFY4= 2023-11-02T18:21:12.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3b62s9WE3BD0B+ZpfJZq5g9qFY4=', 'Content-Length': '0'} 2023-11-02T18:21:12.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000956cd6ede3744f9c-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.741 DEBUG:boto:path=/ibtljn-53/b 2023-11-02T18:21:12.742 DEBUG:boto:auth_path=/ibtljn-53/b 2023-11-02T18:21:12.742 DEBUG:boto:path=/ibtljn-53/b?versionId=null 2023-11-02T18:21:12.743 DEBUG:boto:auth_path=/ibtljn-53/b?versionId=null 2023-11-02T18:21:12.743 DEBUG:boto:Method: GET 2023-11-02T18:21:12.744 DEBUG:boto:Path: /ibtljn-53/b?versionId=null 2023-11-02T18:21:12.744 DEBUG:boto:Data: 2023-11-02T18:21:12.745 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.745 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.746 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.746 DEBUG:boto:Params: {} 2023-11-02T18:21:12.747 DEBUG:boto:Token: None 2023-11-02T18:21:12.747 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/b?versionId=null 2023-11-02T18:21:12.748 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3b62s9WE3BD0B+ZpfJZq5g9qFY4= 2023-11-02T18:21:12.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3b62s9WE3BD0B+ZpfJZq5g9qFY4=', 'Content-Length': '0'} 2023-11-02T18:21:12.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb2e938af0f9945e-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.752 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.753 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.753 DEBUG:boto:path=/ibtljn-53/?prefix=b 2023-11-02T18:21:12.754 DEBUG:boto:auth_path=/ibtljn-53/?prefix=b 2023-11-02T18:21:12.754 DEBUG:boto:Method: GET 2023-11-02T18:21:12.755 DEBUG:boto:Path: /ibtljn-53/?prefix=b 2023-11-02T18:21:12.755 DEBUG:boto:Data: 2023-11-02T18:21:12.755 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.756 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.756 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.757 DEBUG:boto:Params: {} 2023-11-02T18:21:12.757 DEBUG:boto:Token: None 2023-11-02T18:21:12.758 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/ 2023-11-02T18:21:12.758 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI= 2023-11-02T18:21:12.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI=', 'Content-Length': '0'} 2023-11-02T18:21:12.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dfcb2e27a980a91-006543e898-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.764 DEBUG:boto:b'ibtljn-53b1000falseb2023-11-02T18:20:46.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:12.765 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.765 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.766 DEBUG:boto:path=/ibtljn-53/?prefix=b 2023-11-02T18:21:12.766 DEBUG:boto:auth_path=/ibtljn-53/?prefix=b 2023-11-02T18:21:12.767 DEBUG:boto:Method: GET 2023-11-02T18:21:12.767 DEBUG:boto:Path: /ibtljn-53/?prefix=b 2023-11-02T18:21:12.768 DEBUG:boto:Data: 2023-11-02T18:21:12.768 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.769 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.769 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.770 DEBUG:boto:Params: {} 2023-11-02T18:21:12.770 DEBUG:boto:Token: None 2023-11-02T18:21:12.771 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/ 2023-11-02T18:21:12.771 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI= 2023-11-02T18:21:12.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI=', 'Content-Length': '0'} 2023-11-02T18:21:12.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4aa9e704868c8a6-006543e898-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.777 DEBUG:boto:b'ibtljn-53b1000falseb2023-11-02T18:20:46.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:12.777 DEBUG:boto:path=/ibtljn-53/b 2023-11-02T18:21:12.778 DEBUG:boto:auth_path=/ibtljn-53/b 2023-11-02T18:21:12.778 DEBUG:boto:Method: HEAD 2023-11-02T18:21:12.779 DEBUG:boto:Path: /ibtljn-53/b 2023-11-02T18:21:12.779 DEBUG:boto:Data: 2023-11-02T18:21:12.780 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.780 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.781 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.781 DEBUG:boto:Params: {} 2023-11-02T18:21:12.782 DEBUG:boto:Token: None 2023-11-02T18:21:12.782 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/b 2023-11-02T18:21:12.783 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qR9aCJY2nAw0hj1Wj7Mt1Z5cEfw= 2023-11-02T18:21:12.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qR9aCJY2nAw0hj1Wj7Mt1Z5cEfw=', 'Content-Length': '0'} 2023-11-02T18:21:12.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba7ebe5b440ea56f-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.787 DEBUG:boto:path=/ibtljn-53/b 2023-11-02T18:21:12.787 DEBUG:boto:auth_path=/ibtljn-53/b 2023-11-02T18:21:12.788 DEBUG:boto:Method: HEAD 2023-11-02T18:21:12.788 DEBUG:boto:Path: /ibtljn-53/b 2023-11-02T18:21:12.788 DEBUG:boto:Data: 2023-11-02T18:21:12.789 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.789 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.790 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.790 DEBUG:boto:Params: {} 2023-11-02T18:21:12.791 DEBUG:boto:Token: None 2023-11-02T18:21:12.791 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/b 2023-11-02T18:21:12.792 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qR9aCJY2nAw0hj1Wj7Mt1Z5cEfw= 2023-11-02T18:21:12.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qR9aCJY2nAw0hj1Wj7Mt1Z5cEfw=', 'Content-Length': '0'} 2023-11-02T18:21:12.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edeb63a33a0afc4b-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:21:12.796 DEBUG:boto:path=/ibtljn-53/b 2023-11-02T18:21:12.796 DEBUG:boto:auth_path=/ibtljn-53/b 2023-11-02T18:21:12.797 DEBUG:boto:Method: GET 2023-11-02T18:21:12.797 DEBUG:boto:Path: /ibtljn-53/b 2023-11-02T18:21:12.798 DEBUG:boto:Data: 2023-11-02T18:21:12.798 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.799 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.800 DEBUG:boto:Params: {} 2023-11-02T18:21:12.800 DEBUG:boto:Token: None 2023-11-02T18:21:12.801 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/b 2023-11-02T18:21:12.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ot1KRqL8jtYlkfzB1POWhDokzfE= 2023-11-02T18:21:12.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ot1KRqL8jtYlkfzB1POWhDokzfE=', 'Content-Length': '0'} 2023-11-02T18:21:12.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fefb38d042cc8d1-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.846 DEBUG:boto:path=/ibtljn-53/b 2023-11-02T18:21:12.847 DEBUG:boto:auth_path=/ibtljn-53/b 2023-11-02T18:21:12.847 DEBUG:boto:Method: GET 2023-11-02T18:21:12.848 DEBUG:boto:Path: /ibtljn-53/b 2023-11-02T18:21:12.849 DEBUG:boto:Data: 2023-11-02T18:21:12.849 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.849 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.850 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.850 DEBUG:boto:Params: {} 2023-11-02T18:21:12.851 DEBUG:boto:Token: None 2023-11-02T18:21:12.851 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/b 2023-11-02T18:21:12.852 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ot1KRqL8jtYlkfzB1POWhDokzfE= 2023-11-02T18:21:12.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ot1KRqL8jtYlkfzB1POWhDokzfE=', 'Content-Length': '0'} 2023-11-02T18:21:12.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099b95ce94a22c138-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.856 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.857 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.857 DEBUG:boto:path=/ibtljn-53/?prefix=b 2023-11-02T18:21:12.858 DEBUG:boto:auth_path=/ibtljn-53/?prefix=b 2023-11-02T18:21:12.858 DEBUG:boto:Method: GET 2023-11-02T18:21:12.859 DEBUG:boto:Path: /ibtljn-53/?prefix=b 2023-11-02T18:21:12.859 DEBUG:boto:Data: 2023-11-02T18:21:12.860 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.860 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.861 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.861 DEBUG:boto:Params: {} 2023-11-02T18:21:12.861 DEBUG:boto:Token: None 2023-11-02T18:21:12.862 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/ 2023-11-02T18:21:12.863 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI= 2023-11-02T18:21:12.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI=', 'Content-Length': '0'} 2023-11-02T18:21:12.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef82b1f29cda4f68-006543e898-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.869 DEBUG:boto:b'ibtljn-53b1000falseb2023-11-02T18:20:46.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:12.870 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.870 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.871 DEBUG:boto:path=/ibtljn-53/?prefix=b 2023-11-02T18:21:12.871 DEBUG:boto:auth_path=/ibtljn-53/?prefix=b 2023-11-02T18:21:12.871 DEBUG:boto:Method: GET 2023-11-02T18:21:12.872 DEBUG:boto:Path: /ibtljn-53/?prefix=b 2023-11-02T18:21:12.872 DEBUG:boto:Data: 2023-11-02T18:21:12.873 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.873 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.874 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.874 DEBUG:boto:Params: {} 2023-11-02T18:21:12.875 DEBUG:boto:Token: None 2023-11-02T18:21:12.875 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/ 2023-11-02T18:21:12.876 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI= 2023-11-02T18:21:12.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI=', 'Content-Length': '0'} 2023-11-02T18:21:12.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003442d8bd503d1d2-006543e898-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.882 DEBUG:boto:b'ibtljn-53b1000falseb2023-11-02T18:20:46.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:12.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:21:12.884 DEBUG:boto:path=/ibtljn-53/c 2023-11-02T18:21:12.884 DEBUG:boto:auth_path=/ibtljn-53/c 2023-11-02T18:21:12.884 DEBUG:boto:path=/ibtljn-53/c?versionId=null 2023-11-02T18:21:12.885 DEBUG:boto:auth_path=/ibtljn-53/c?versionId=null 2023-11-02T18:21:12.885 DEBUG:boto:Method: GET 2023-11-02T18:21:12.886 DEBUG:boto:Path: /ibtljn-53/c?versionId=null 2023-11-02T18:21:12.887 DEBUG:boto:Data: 2023-11-02T18:21:12.887 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.887 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.888 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.888 DEBUG:boto:Params: {} 2023-11-02T18:21:12.889 DEBUG:boto:Token: None 2023-11-02T18:21:12.889 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/c?versionId=null 2023-11-02T18:21:12.890 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6b6SoI4WUm1ARjcfm0FH9RjeUY8= 2023-11-02T18:21:12.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6b6SoI4WUm1ARjcfm0FH9RjeUY8=', 'Content-Length': '0'} 2023-11-02T18:21:12.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040e027f2f53decbc-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.895 DEBUG:boto:path=/ibtljn-53/c 2023-11-02T18:21:12.895 DEBUG:boto:auth_path=/ibtljn-53/c 2023-11-02T18:21:12.896 DEBUG:boto:path=/ibtljn-53/c?versionId=null 2023-11-02T18:21:12.896 DEBUG:boto:auth_path=/ibtljn-53/c?versionId=null 2023-11-02T18:21:12.896 DEBUG:boto:Method: GET 2023-11-02T18:21:12.897 DEBUG:boto:Path: /ibtljn-53/c?versionId=null 2023-11-02T18:21:12.897 DEBUG:boto:Data: 2023-11-02T18:21:12.898 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.898 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.899 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.899 DEBUG:boto:Params: {} 2023-11-02T18:21:12.900 DEBUG:boto:Token: None 2023-11-02T18:21:12.900 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/c?versionId=null 2023-11-02T18:21:12.901 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6b6SoI4WUm1ARjcfm0FH9RjeUY8= 2023-11-02T18:21:12.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6b6SoI4WUm1ARjcfm0FH9RjeUY8=', 'Content-Length': '0'} 2023-11-02T18:21:12.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086aa8029ace09e2b-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.905 DEBUG:boto:path=/ibtljn-53/c 2023-11-02T18:21:12.905 DEBUG:boto:auth_path=/ibtljn-53/c 2023-11-02T18:21:12.906 DEBUG:boto:path=/ibtljn-53/c?versionId=null 2023-11-02T18:21:12.906 DEBUG:boto:auth_path=/ibtljn-53/c?versionId=null 2023-11-02T18:21:12.907 DEBUG:boto:Method: HEAD 2023-11-02T18:21:12.907 DEBUG:boto:Path: /ibtljn-53/c?versionId=null 2023-11-02T18:21:12.908 DEBUG:boto:Data: 2023-11-02T18:21:12.908 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.909 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.909 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.909 DEBUG:boto:Params: {} 2023-11-02T18:21:12.910 DEBUG:boto:Token: None 2023-11-02T18:21:12.910 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/c?versionId=null 2023-11-02T18:21:12.911 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NNrxOenoS+iR47eFvP9HyRyQHxs= 2023-11-02T18:21:12.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NNrxOenoS+iR47eFvP9HyRyQHxs=', 'Content-Length': '0'} 2023-11-02T18:21:12.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026a254d7bc5777b7-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.915 DEBUG:boto:path=/ibtljn-53/c 2023-11-02T18:21:12.916 DEBUG:boto:auth_path=/ibtljn-53/c 2023-11-02T18:21:12.916 DEBUG:boto:path=/ibtljn-53/c?versionId=null 2023-11-02T18:21:12.917 DEBUG:boto:auth_path=/ibtljn-53/c?versionId=null 2023-11-02T18:21:12.917 DEBUG:boto:Method: HEAD 2023-11-02T18:21:12.918 DEBUG:boto:Path: /ibtljn-53/c?versionId=null 2023-11-02T18:21:12.918 DEBUG:boto:Data: 2023-11-02T18:21:12.918 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.919 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.919 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.920 DEBUG:boto:Params: {} 2023-11-02T18:21:12.921 DEBUG:boto:Token: None 2023-11-02T18:21:12.921 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/c?versionId=null 2023-11-02T18:21:12.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NNrxOenoS+iR47eFvP9HyRyQHxs= 2023-11-02T18:21:12.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NNrxOenoS+iR47eFvP9HyRyQHxs=', 'Content-Length': '0'} 2023-11-02T18:21:12.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa47021bae9ec2f3-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:21:12.926 DEBUG:boto:path=/ibtljn-53/c 2023-11-02T18:21:12.927 DEBUG:boto:auth_path=/ibtljn-53/c 2023-11-02T18:21:12.927 DEBUG:boto:path=/ibtljn-53/c?versionId=null 2023-11-02T18:21:12.928 DEBUG:boto:auth_path=/ibtljn-53/c?versionId=null 2023-11-02T18:21:12.928 DEBUG:boto:Method: GET 2023-11-02T18:21:12.929 DEBUG:boto:Path: /ibtljn-53/c?versionId=null 2023-11-02T18:21:12.929 DEBUG:boto:Data: 2023-11-02T18:21:12.929 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.930 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.931 DEBUG:boto:Params: {} 2023-11-02T18:21:12.931 DEBUG:boto:Token: None 2023-11-02T18:21:12.932 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/c?versionId=null 2023-11-02T18:21:12.932 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6b6SoI4WUm1ARjcfm0FH9RjeUY8= 2023-11-02T18:21:12.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6b6SoI4WUm1ARjcfm0FH9RjeUY8=', 'Content-Length': '0'} 2023-11-02T18:21:12.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ff5b797da05747-006543e898-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.978 DEBUG:boto:path=/ibtljn-53/c 2023-11-02T18:21:12.979 DEBUG:boto:auth_path=/ibtljn-53/c 2023-11-02T18:21:12.979 DEBUG:boto:path=/ibtljn-53/c?versionId=null 2023-11-02T18:21:12.980 DEBUG:boto:auth_path=/ibtljn-53/c?versionId=null 2023-11-02T18:21:12.980 DEBUG:boto:Method: GET 2023-11-02T18:21:12.981 DEBUG:boto:Path: /ibtljn-53/c?versionId=null 2023-11-02T18:21:12.981 DEBUG:boto:Data: 2023-11-02T18:21:12.982 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.982 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:12.983 DEBUG:boto:Port: 8000 2023-11-02T18:21:12.983 DEBUG:boto:Params: {} 2023-11-02T18:21:12.984 DEBUG:boto:Token: None 2023-11-02T18:21:12.984 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/c?versionId=null 2023-11-02T18:21:12.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6b6SoI4WUm1ARjcfm0FH9RjeUY8= 2023-11-02T18:21:12.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6b6SoI4WUm1ARjcfm0FH9RjeUY8=', 'Content-Length': '0'} 2023-11-02T18:21:12.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064f55f523eb63380-006543e898-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:12.989 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:12.989 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:12.990 DEBUG:boto:path=/ibtljn-53/?prefix=c 2023-11-02T18:21:12.990 DEBUG:boto:auth_path=/ibtljn-53/?prefix=c 2023-11-02T18:21:12.991 DEBUG:boto:Method: GET 2023-11-02T18:21:12.991 DEBUG:boto:Path: /ibtljn-53/?prefix=c 2023-11-02T18:21:12.992 DEBUG:boto:Data: 2023-11-02T18:21:12.992 DEBUG:boto:Headers: {} 2023-11-02T18:21:12.993 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:12.993 DEBUG:boto:Port: 8001 2023-11-02T18:21:12.994 DEBUG:boto:Params: {} 2023-11-02T18:21:12.994 DEBUG:boto:Token: None 2023-11-02T18:21:12.995 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:12 GMT /ibtljn-53/ 2023-11-02T18:21:12.995 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI= 2023-11-02T18:21:12.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KYYff63aLR/jj572uaKBu3gyPtI=', 'Content-Length': '0'} 2023-11-02T18:21:13.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5ceaef8b14a7498-006543e898-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.002 DEBUG:boto:b'ibtljn-53c1000falsec2023-11-02T18:20:46.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:13.003 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:13.003 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:13.004 DEBUG:boto:path=/ibtljn-53/?prefix=c 2023-11-02T18:21:13.004 DEBUG:boto:auth_path=/ibtljn-53/?prefix=c 2023-11-02T18:21:13.005 DEBUG:boto:Method: GET 2023-11-02T18:21:13.005 DEBUG:boto:Path: /ibtljn-53/?prefix=c 2023-11-02T18:21:13.006 DEBUG:boto:Data: 2023-11-02T18:21:13.006 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.007 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.007 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.007 DEBUG:boto:Params: {} 2023-11-02T18:21:13.008 DEBUG:boto:Token: None 2023-11-02T18:21:13.008 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/ 2023-11-02T18:21:13.009 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc= 2023-11-02T18:21:13.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc=', 'Content-Length': '0'} 2023-11-02T18:21:13.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acce45f3016e4faf-006543e899-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.015 DEBUG:boto:b'ibtljn-53c1000falsec2023-11-02T18:20:46.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:13.015 DEBUG:boto:path=/ibtljn-53/c 2023-11-02T18:21:13.016 DEBUG:boto:auth_path=/ibtljn-53/c 2023-11-02T18:21:13.016 DEBUG:boto:Method: HEAD 2023-11-02T18:21:13.017 DEBUG:boto:Path: /ibtljn-53/c 2023-11-02T18:21:13.017 DEBUG:boto:Data: 2023-11-02T18:21:13.018 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.018 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:13.019 DEBUG:boto:Port: 8001 2023-11-02T18:21:13.019 DEBUG:boto:Params: {} 2023-11-02T18:21:13.020 DEBUG:boto:Token: None 2023-11-02T18:21:13.020 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/c 2023-11-02T18:21:13.021 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E2owu9Mh0MGejZ0INq/RgZHa18g= 2023-11-02T18:21:13.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E2owu9Mh0MGejZ0INq/RgZHa18g=', 'Content-Length': '0'} 2023-11-02T18:21:13.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b12109c71b906b66-006543e899-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.024 DEBUG:boto:path=/ibtljn-53/c 2023-11-02T18:21:13.025 DEBUG:boto:auth_path=/ibtljn-53/c 2023-11-02T18:21:13.025 DEBUG:boto:Method: HEAD 2023-11-02T18:21:13.026 DEBUG:boto:Path: /ibtljn-53/c 2023-11-02T18:21:13.026 DEBUG:boto:Data: 2023-11-02T18:21:13.027 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.027 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.027 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.028 DEBUG:boto:Params: {} 2023-11-02T18:21:13.028 DEBUG:boto:Token: None 2023-11-02T18:21:13.029 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/c 2023-11-02T18:21:13.029 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E2owu9Mh0MGejZ0INq/RgZHa18g= 2023-11-02T18:21:13.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E2owu9Mh0MGejZ0INq/RgZHa18g=', 'Content-Length': '0'} 2023-11-02T18:21:13.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059621fc90ca26d61-006543e899-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:21:13.034 DEBUG:boto:path=/ibtljn-53/c 2023-11-02T18:21:13.034 DEBUG:boto:auth_path=/ibtljn-53/c 2023-11-02T18:21:13.035 DEBUG:boto:Method: GET 2023-11-02T18:21:13.035 DEBUG:boto:Path: /ibtljn-53/c 2023-11-02T18:21:13.036 DEBUG:boto:Data: 2023-11-02T18:21:13.036 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.037 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:13.037 DEBUG:boto:Port: 8001 2023-11-02T18:21:13.037 DEBUG:boto:Params: {} 2023-11-02T18:21:13.038 DEBUG:boto:Token: None 2023-11-02T18:21:13.038 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/c 2023-11-02T18:21:13.039 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aIOdKxGxvz9MfTHXB3ncqtpRtFM= 2023-11-02T18:21:13.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aIOdKxGxvz9MfTHXB3ncqtpRtFM=', 'Content-Length': '0'} 2023-11-02T18:21:13.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d4b9c4e48d8d0e5-006543e899-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.086 DEBUG:boto:path=/ibtljn-53/c 2023-11-02T18:21:13.087 DEBUG:boto:auth_path=/ibtljn-53/c 2023-11-02T18:21:13.087 DEBUG:boto:Method: GET 2023-11-02T18:21:13.088 DEBUG:boto:Path: /ibtljn-53/c 2023-11-02T18:21:13.088 DEBUG:boto:Data: 2023-11-02T18:21:13.089 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.089 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.089 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.090 DEBUG:boto:Params: {} 2023-11-02T18:21:13.090 DEBUG:boto:Token: None 2023-11-02T18:21:13.094 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/c 2023-11-02T18:21:13.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aIOdKxGxvz9MfTHXB3ncqtpRtFM= 2023-11-02T18:21:13.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aIOdKxGxvz9MfTHXB3ncqtpRtFM=', 'Content-Length': '0'} 2023-11-02T18:21:13.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b110834ea1f9e300-006543e899-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.098 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:13.099 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:13.099 DEBUG:boto:path=/ibtljn-53/?prefix=c 2023-11-02T18:21:13.100 DEBUG:boto:auth_path=/ibtljn-53/?prefix=c 2023-11-02T18:21:13.100 DEBUG:boto:Method: GET 2023-11-02T18:21:13.101 DEBUG:boto:Path: /ibtljn-53/?prefix=c 2023-11-02T18:21:13.101 DEBUG:boto:Data: 2023-11-02T18:21:13.101 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.102 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:13.102 DEBUG:boto:Port: 8001 2023-11-02T18:21:13.103 DEBUG:boto:Params: {} 2023-11-02T18:21:13.103 DEBUG:boto:Token: None 2023-11-02T18:21:13.104 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/ 2023-11-02T18:21:13.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc= 2023-11-02T18:21:13.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc=', 'Content-Length': '0'} 2023-11-02T18:21:13.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfbb87c402b1f039-006543e899-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.110 DEBUG:boto:b'ibtljn-53c1000falsec2023-11-02T18:20:46.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:13.111 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:13.111 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:13.112 DEBUG:boto:path=/ibtljn-53/?prefix=c 2023-11-02T18:21:13.112 DEBUG:boto:auth_path=/ibtljn-53/?prefix=c 2023-11-02T18:21:13.113 DEBUG:boto:Method: GET 2023-11-02T18:21:13.113 DEBUG:boto:Path: /ibtljn-53/?prefix=c 2023-11-02T18:21:13.114 DEBUG:boto:Data: 2023-11-02T18:21:13.114 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.115 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.115 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.116 DEBUG:boto:Params: {} 2023-11-02T18:21:13.116 DEBUG:boto:Token: None 2023-11-02T18:21:13.117 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/ 2023-11-02T18:21:13.117 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc= 2023-11-02T18:21:13.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc=', 'Content-Length': '0'} 2023-11-02T18:21:13.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e4be7ccd990b20c-006543e899-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.123 DEBUG:boto:b'ibtljn-53c1000falsec2023-11-02T18:20:46.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:13.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:21:13.124 DEBUG:boto:path=/ibtljn-53/d 2023-11-02T18:21:13.124 DEBUG:boto:auth_path=/ibtljn-53/d 2023-11-02T18:21:13.125 DEBUG:boto:path=/ibtljn-53/d?versionId=null 2023-11-02T18:21:13.125 DEBUG:boto:auth_path=/ibtljn-53/d?versionId=null 2023-11-02T18:21:13.126 DEBUG:boto:Method: GET 2023-11-02T18:21:13.126 DEBUG:boto:Path: /ibtljn-53/d?versionId=null 2023-11-02T18:21:13.127 DEBUG:boto:Data: 2023-11-02T18:21:13.127 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.128 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:13.128 DEBUG:boto:Port: 8001 2023-11-02T18:21:13.128 DEBUG:boto:Params: {} 2023-11-02T18:21:13.129 DEBUG:boto:Token: None 2023-11-02T18:21:13.129 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/d?versionId=null 2023-11-02T18:21:13.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CBTfAIHEwBNY8xRQF7xdmcXqrOk= 2023-11-02T18:21:13.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CBTfAIHEwBNY8xRQF7xdmcXqrOk=', 'Content-Length': '0'} 2023-11-02T18:21:13.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfce2fb214054518-006543e899-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.134 DEBUG:boto:path=/ibtljn-53/d 2023-11-02T18:21:13.135 DEBUG:boto:auth_path=/ibtljn-53/d 2023-11-02T18:21:13.135 DEBUG:boto:path=/ibtljn-53/d?versionId=null 2023-11-02T18:21:13.136 DEBUG:boto:auth_path=/ibtljn-53/d?versionId=null 2023-11-02T18:21:13.136 DEBUG:boto:Method: GET 2023-11-02T18:21:13.137 DEBUG:boto:Path: /ibtljn-53/d?versionId=null 2023-11-02T18:21:13.137 DEBUG:boto:Data: 2023-11-02T18:21:13.138 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.139 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.139 DEBUG:boto:Params: {} 2023-11-02T18:21:13.140 DEBUG:boto:Token: None 2023-11-02T18:21:13.140 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/d?versionId=null 2023-11-02T18:21:13.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CBTfAIHEwBNY8xRQF7xdmcXqrOk= 2023-11-02T18:21:13.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CBTfAIHEwBNY8xRQF7xdmcXqrOk=', 'Content-Length': '0'} 2023-11-02T18:21:13.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfbe79a205507468-006543e899-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.145 DEBUG:boto:path=/ibtljn-53/d 2023-11-02T18:21:13.145 DEBUG:boto:auth_path=/ibtljn-53/d 2023-11-02T18:21:13.146 DEBUG:boto:path=/ibtljn-53/d?versionId=null 2023-11-02T18:21:13.146 DEBUG:boto:auth_path=/ibtljn-53/d?versionId=null 2023-11-02T18:21:13.147 DEBUG:boto:Method: HEAD 2023-11-02T18:21:13.147 DEBUG:boto:Path: /ibtljn-53/d?versionId=null 2023-11-02T18:21:13.147 DEBUG:boto:Data: 2023-11-02T18:21:13.148 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:13.149 DEBUG:boto:Port: 8001 2023-11-02T18:21:13.149 DEBUG:boto:Params: {} 2023-11-02T18:21:13.150 DEBUG:boto:Token: None 2023-11-02T18:21:13.150 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/d?versionId=null 2023-11-02T18:21:13.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZyY5tTH4Jnx4P/QO829f9DQdN4s= 2023-11-02T18:21:13.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZyY5tTH4Jnx4P/QO829f9DQdN4s=', 'Content-Length': '0'} 2023-11-02T18:21:13.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa006de8fc61d2ec-006543e899-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.155 DEBUG:boto:path=/ibtljn-53/d 2023-11-02T18:21:13.155 DEBUG:boto:auth_path=/ibtljn-53/d 2023-11-02T18:21:13.156 DEBUG:boto:path=/ibtljn-53/d?versionId=null 2023-11-02T18:21:13.156 DEBUG:boto:auth_path=/ibtljn-53/d?versionId=null 2023-11-02T18:21:13.156 DEBUG:boto:Method: HEAD 2023-11-02T18:21:13.157 DEBUG:boto:Path: /ibtljn-53/d?versionId=null 2023-11-02T18:21:13.157 DEBUG:boto:Data: 2023-11-02T18:21:13.158 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.158 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.159 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.159 DEBUG:boto:Params: {} 2023-11-02T18:21:13.160 DEBUG:boto:Token: None 2023-11-02T18:21:13.160 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/d?versionId=null 2023-11-02T18:21:13.161 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZyY5tTH4Jnx4P/QO829f9DQdN4s= 2023-11-02T18:21:13.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZyY5tTH4Jnx4P/QO829f9DQdN4s=', 'Content-Length': '0'} 2023-11-02T18:21:13.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f22f98e05d080b1-006543e899-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:21:13.165 DEBUG:boto:path=/ibtljn-53/d 2023-11-02T18:21:13.165 DEBUG:boto:auth_path=/ibtljn-53/d 2023-11-02T18:21:13.166 DEBUG:boto:path=/ibtljn-53/d?versionId=null 2023-11-02T18:21:13.166 DEBUG:boto:auth_path=/ibtljn-53/d?versionId=null 2023-11-02T18:21:13.167 DEBUG:boto:Method: GET 2023-11-02T18:21:13.167 DEBUG:boto:Path: /ibtljn-53/d?versionId=null 2023-11-02T18:21:13.168 DEBUG:boto:Data: 2023-11-02T18:21:13.168 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.169 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:13.169 DEBUG:boto:Port: 8001 2023-11-02T18:21:13.170 DEBUG:boto:Params: {} 2023-11-02T18:21:13.170 DEBUG:boto:Token: None 2023-11-02T18:21:13.171 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/d?versionId=null 2023-11-02T18:21:13.171 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CBTfAIHEwBNY8xRQF7xdmcXqrOk= 2023-11-02T18:21:13.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CBTfAIHEwBNY8xRQF7xdmcXqrOk=', 'Content-Length': '0'} 2023-11-02T18:21:13.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000861c865cdd028157-006543e899-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.176 DEBUG:boto:path=/ibtljn-53/d 2023-11-02T18:21:13.176 DEBUG:boto:auth_path=/ibtljn-53/d 2023-11-02T18:21:13.176 DEBUG:boto:path=/ibtljn-53/d?versionId=null 2023-11-02T18:21:13.177 DEBUG:boto:auth_path=/ibtljn-53/d?versionId=null 2023-11-02T18:21:13.177 DEBUG:boto:Method: GET 2023-11-02T18:21:13.178 DEBUG:boto:Path: /ibtljn-53/d?versionId=null 2023-11-02T18:21:13.178 DEBUG:boto:Data: 2023-11-02T18:21:13.179 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.180 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.180 DEBUG:boto:Params: {} 2023-11-02T18:21:13.181 DEBUG:boto:Token: None 2023-11-02T18:21:13.181 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/d?versionId=null 2023-11-02T18:21:13.182 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CBTfAIHEwBNY8xRQF7xdmcXqrOk= 2023-11-02T18:21:13.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CBTfAIHEwBNY8xRQF7xdmcXqrOk=', 'Content-Length': '0'} 2023-11-02T18:21:13.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064dd500867eeea03-006543e899-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.226 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:13.227 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:13.228 DEBUG:boto:path=/ibtljn-53/?prefix=d 2023-11-02T18:21:13.228 DEBUG:boto:auth_path=/ibtljn-53/?prefix=d 2023-11-02T18:21:13.229 DEBUG:boto:Method: GET 2023-11-02T18:21:13.229 DEBUG:boto:Path: /ibtljn-53/?prefix=d 2023-11-02T18:21:13.230 DEBUG:boto:Data: 2023-11-02T18:21:13.230 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:13.231 DEBUG:boto:Port: 8001 2023-11-02T18:21:13.232 DEBUG:boto:Params: {} 2023-11-02T18:21:13.232 DEBUG:boto:Token: None 2023-11-02T18:21:13.233 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/ 2023-11-02T18:21:13.233 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc= 2023-11-02T18:21:13.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc=', 'Content-Length': '0'} 2023-11-02T18:21:13.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000298a8167e2424df8-006543e899-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.240 DEBUG:boto:b'ibtljn-53d1000falsed2023-11-02T18:20:46.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:13.241 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:13.241 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:13.242 DEBUG:boto:path=/ibtljn-53/?prefix=d 2023-11-02T18:21:13.242 DEBUG:boto:auth_path=/ibtljn-53/?prefix=d 2023-11-02T18:21:13.242 DEBUG:boto:Method: GET 2023-11-02T18:21:13.243 DEBUG:boto:Path: /ibtljn-53/?prefix=d 2023-11-02T18:21:13.243 DEBUG:boto:Data: 2023-11-02T18:21:13.244 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.244 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.245 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.245 DEBUG:boto:Params: {} 2023-11-02T18:21:13.246 DEBUG:boto:Token: None 2023-11-02T18:21:13.246 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/ 2023-11-02T18:21:13.247 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc= 2023-11-02T18:21:13.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc=', 'Content-Length': '0'} 2023-11-02T18:21:13.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffb318e10f846e1a-006543e899-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.253 DEBUG:boto:b'ibtljn-53d1000falsed2023-11-02T18:20:46.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:13.253 DEBUG:boto:path=/ibtljn-53/d 2023-11-02T18:21:13.254 DEBUG:boto:auth_path=/ibtljn-53/d 2023-11-02T18:21:13.254 DEBUG:boto:Method: HEAD 2023-11-02T18:21:13.255 DEBUG:boto:Path: /ibtljn-53/d 2023-11-02T18:21:13.255 DEBUG:boto:Data: 2023-11-02T18:21:13.256 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.256 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:13.257 DEBUG:boto:Port: 8001 2023-11-02T18:21:13.257 DEBUG:boto:Params: {} 2023-11-02T18:21:13.258 DEBUG:boto:Token: None 2023-11-02T18:21:13.258 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/d 2023-11-02T18:21:13.259 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IJpMSWMZzcn/FF3GLlt6Q+BU4To= 2023-11-02T18:21:13.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IJpMSWMZzcn/FF3GLlt6Q+BU4To=', 'Content-Length': '0'} 2023-11-02T18:21:13.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048eab94430a8ab0b-006543e899-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.263 DEBUG:boto:path=/ibtljn-53/d 2023-11-02T18:21:13.263 DEBUG:boto:auth_path=/ibtljn-53/d 2023-11-02T18:21:13.263 DEBUG:boto:Method: HEAD 2023-11-02T18:21:13.264 DEBUG:boto:Path: /ibtljn-53/d 2023-11-02T18:21:13.264 DEBUG:boto:Data: 2023-11-02T18:21:13.264 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.265 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.265 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.266 DEBUG:boto:Params: {} 2023-11-02T18:21:13.266 DEBUG:boto:Token: None 2023-11-02T18:21:13.267 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/d 2023-11-02T18:21:13.267 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IJpMSWMZzcn/FF3GLlt6Q+BU4To= 2023-11-02T18:21:13.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IJpMSWMZzcn/FF3GLlt6Q+BU4To=', 'Content-Length': '0'} 2023-11-02T18:21:13.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008039afe595b197e8-006543e899-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:21:13.271 DEBUG:boto:path=/ibtljn-53/d 2023-11-02T18:21:13.272 DEBUG:boto:auth_path=/ibtljn-53/d 2023-11-02T18:21:13.272 DEBUG:boto:Method: GET 2023-11-02T18:21:13.273 DEBUG:boto:Path: /ibtljn-53/d 2023-11-02T18:21:13.273 DEBUG:boto:Data: 2023-11-02T18:21:13.273 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.274 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:13.274 DEBUG:boto:Port: 8001 2023-11-02T18:21:13.275 DEBUG:boto:Params: {} 2023-11-02T18:21:13.275 DEBUG:boto:Token: None 2023-11-02T18:21:13.276 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/d 2023-11-02T18:21:13.276 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RB2NCIBpjrslCB5tMUJwlA2yC2k= 2023-11-02T18:21:13.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RB2NCIBpjrslCB5tMUJwlA2yC2k=', 'Content-Length': '0'} 2023-11-02T18:21:13.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b6ef278f451d72-006543e899-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.326 DEBUG:boto:path=/ibtljn-53/d 2023-11-02T18:21:13.327 DEBUG:boto:auth_path=/ibtljn-53/d 2023-11-02T18:21:13.327 DEBUG:boto:Method: GET 2023-11-02T18:21:13.328 DEBUG:boto:Path: /ibtljn-53/d 2023-11-02T18:21:13.328 DEBUG:boto:Data: 2023-11-02T18:21:13.329 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.329 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.330 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.330 DEBUG:boto:Params: {} 2023-11-02T18:21:13.331 DEBUG:boto:Token: None 2023-11-02T18:21:13.332 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/d 2023-11-02T18:21:13.332 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RB2NCIBpjrslCB5tMUJwlA2yC2k= 2023-11-02T18:21:13.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RB2NCIBpjrslCB5tMUJwlA2yC2k=', 'Content-Length': '0'} 2023-11-02T18:21:13.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:20:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096a4761fb3feeb50-006543e899-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.336 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:13.337 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:13.338 DEBUG:boto:path=/ibtljn-53/?prefix=d 2023-11-02T18:21:13.338 DEBUG:boto:auth_path=/ibtljn-53/?prefix=d 2023-11-02T18:21:13.339 DEBUG:boto:Method: GET 2023-11-02T18:21:13.339 DEBUG:boto:Path: /ibtljn-53/?prefix=d 2023-11-02T18:21:13.340 DEBUG:boto:Data: 2023-11-02T18:21:13.340 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.341 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:13.341 DEBUG:boto:Port: 8001 2023-11-02T18:21:13.342 DEBUG:boto:Params: {} 2023-11-02T18:21:13.342 DEBUG:boto:Token: None 2023-11-02T18:21:13.343 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/ 2023-11-02T18:21:13.343 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc= 2023-11-02T18:21:13.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc=', 'Content-Length': '0'} 2023-11-02T18:21:13.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb0cdb0fc5d3a1be-006543e899-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.350 DEBUG:boto:b'ibtljn-53d1000falsed2023-11-02T18:20:46.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:13.351 DEBUG:boto:path=/ibtljn-53/ 2023-11-02T18:21:13.351 DEBUG:boto:auth_path=/ibtljn-53/ 2023-11-02T18:21:13.352 DEBUG:boto:path=/ibtljn-53/?prefix=d 2023-11-02T18:21:13.352 DEBUG:boto:auth_path=/ibtljn-53/?prefix=d 2023-11-02T18:21:13.353 DEBUG:boto:Method: GET 2023-11-02T18:21:13.353 DEBUG:boto:Path: /ibtljn-53/?prefix=d 2023-11-02T18:21:13.354 DEBUG:boto:Data: 2023-11-02T18:21:13.354 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.354 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.355 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.355 DEBUG:boto:Params: {} 2023-11-02T18:21:13.356 DEBUG:boto:Token: None 2023-11-02T18:21:13.356 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-53/ 2023-11-02T18:21:13.357 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc= 2023-11-02T18:21:13.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:igRFDrDSMaFlKW3EarF7GJFpEZc=', 'Content-Length': '0'} 2023-11-02T18:21:13.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f93c82d6027525a5-006543e899-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.406 DEBUG:boto:b'ibtljn-53d1000falsed2023-11-02T18:20:46.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:13.407 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-53 zones={a2, a1} 2023-11-02T18:21:13.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:21:13.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:21:13.409 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:21:13.454 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:13.455 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:13.548 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-54 2023-11-02T18:21:13.548 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:13.549 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:13.549 DEBUG:boto:Method: PUT 2023-11-02T18:21:13.550 DEBUG:boto:Path: /ibtljn-54/ 2023-11-02T18:21:13.550 DEBUG:boto:Data: 2023-11-02T18:21:13.550 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.551 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.551 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.552 DEBUG:boto:Params: {} 2023-11-02T18:21:13.552 DEBUG:boto:Token: None 2023-11-02T18:21:13.553 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-54/ 2023-11-02T18:21:13.553 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Er4Lg14Oelrar7y8osxPgHDXCOs= 2023-11-02T18:21:13.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Er4Lg14Oelrar7y8osxPgHDXCOs=', 'Content-Length': '0'} 2023-11-02T18:21:13.584 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e47ea5675b434148-006543e899-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.585 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:13.585 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:13.586 DEBUG:boto:Method: HEAD 2023-11-02T18:21:13.586 DEBUG:boto:Path: /ibtljn-54/ 2023-11-02T18:21:13.587 DEBUG:boto:Data: 2023-11-02T18:21:13.587 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.587 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.588 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.588 DEBUG:boto:Params: {} 2023-11-02T18:21:13.589 DEBUG:boto:Token: None 2023-11-02T18:21:13.589 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-54/ 2023-11-02T18:21:13.590 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MVYbmzW0ySLzVg+Q75JvRd2UuNQ= 2023-11-02T18:21:13.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MVYbmzW0ySLzVg+Q75JvRd2UuNQ=', 'Content-Length': '0'} 2023-11-02T18:21:13.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c6ffcfd06d5fd8b-006543e899-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.595 DEBUG:boto:path=/ibtljn-54/a 2023-11-02T18:21:13.596 DEBUG:boto:auth_path=/ibtljn-54/a 2023-11-02T18:21:13.596 DEBUG:boto:Method: PUT 2023-11-02T18:21:13.597 DEBUG:boto:Path: /ibtljn-54/a 2023-11-02T18:21:13.597 DEBUG:boto:Data: 2023-11-02T18:21:13.598 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:21:13.598 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.599 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.599 DEBUG:boto:Params: {} 2023-11-02T18:21:13.600 DEBUG:boto:Token: None 2023-11-02T18:21:13.600 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-54/a 2023-11-02T18:21:13.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s1FYxm1JkJDJmx0Kvvydf1l4sGE= 2023-11-02T18:21:13.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s1FYxm1JkJDJmx0Kvvydf1l4sGE='} 2023-11-02T18:21:13.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac9b2de4cd59efc9-006543e899-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.610 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:13.611 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:13.611 DEBUG:boto:Method: HEAD 2023-11-02T18:21:13.612 DEBUG:boto:Path: /ibtljn-54/ 2023-11-02T18:21:13.612 DEBUG:boto:Data: 2023-11-02T18:21:13.612 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.613 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.613 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.614 DEBUG:boto:Params: {} 2023-11-02T18:21:13.614 DEBUG:boto:Token: None 2023-11-02T18:21:13.615 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-54/ 2023-11-02T18:21:13.615 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MVYbmzW0ySLzVg+Q75JvRd2UuNQ= 2023-11-02T18:21:13.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MVYbmzW0ySLzVg+Q75JvRd2UuNQ=', 'Content-Length': '0'} 2023-11-02T18:21:13.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd34dd6abe9997ca-006543e899-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.620 DEBUG:boto:path=/ibtljn-54/b 2023-11-02T18:21:13.620 DEBUG:boto:auth_path=/ibtljn-54/b 2023-11-02T18:21:13.621 DEBUG:boto:Method: PUT 2023-11-02T18:21:13.621 DEBUG:boto:Path: /ibtljn-54/b 2023-11-02T18:21:13.621 DEBUG:boto:Data: 2023-11-02T18:21:13.622 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:21:13.622 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.623 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.623 DEBUG:boto:Params: {} 2023-11-02T18:21:13.623 DEBUG:boto:Token: None 2023-11-02T18:21:13.624 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-54/b 2023-11-02T18:21:13.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pt7qwBPMkeOk8hm++JHXSLqj6oQ= 2023-11-02T18:21:13.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pt7qwBPMkeOk8hm++JHXSLqj6oQ='} 2023-11-02T18:21:13.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089f23ff956c54203-006543e899-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.671 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:13.672 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:13.672 DEBUG:boto:Method: HEAD 2023-11-02T18:21:13.672 DEBUG:boto:Path: /ibtljn-54/ 2023-11-02T18:21:13.673 DEBUG:boto:Data: 2023-11-02T18:21:13.673 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.674 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.675 DEBUG:boto:Params: {} 2023-11-02T18:21:13.675 DEBUG:boto:Token: None 2023-11-02T18:21:13.675 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-54/ 2023-11-02T18:21:13.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MVYbmzW0ySLzVg+Q75JvRd2UuNQ= 2023-11-02T18:21:13.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MVYbmzW0ySLzVg+Q75JvRd2UuNQ=', 'Content-Length': '0'} 2023-11-02T18:21:13.680 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf05d232d2424095-006543e899-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.681 DEBUG:boto:path=/ibtljn-54/c 2023-11-02T18:21:13.681 DEBUG:boto:auth_path=/ibtljn-54/c 2023-11-02T18:21:13.682 DEBUG:boto:Method: PUT 2023-11-02T18:21:13.682 DEBUG:boto:Path: /ibtljn-54/c 2023-11-02T18:21:13.683 DEBUG:boto:Data: 2023-11-02T18:21:13.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:21:13.683 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.684 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.684 DEBUG:boto:Params: {} 2023-11-02T18:21:13.685 DEBUG:boto:Token: None 2023-11-02T18:21:13.685 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-54/c 2023-11-02T18:21:13.686 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9MWk8QARsb/2RgSztyCEqgc+RhQ= 2023-11-02T18:21:13.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9MWk8QARsb/2RgSztyCEqgc+RhQ='} 2023-11-02T18:21:13.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b303ab899fd9b70e-006543e899-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.699 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:13.700 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:13.700 DEBUG:boto:Method: HEAD 2023-11-02T18:21:13.701 DEBUG:boto:Path: /ibtljn-54/ 2023-11-02T18:21:13.701 DEBUG:boto:Data: 2023-11-02T18:21:13.701 DEBUG:boto:Headers: {} 2023-11-02T18:21:13.702 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.702 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.703 DEBUG:boto:Params: {} 2023-11-02T18:21:13.703 DEBUG:boto:Token: None 2023-11-02T18:21:13.703 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-54/ 2023-11-02T18:21:13.704 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MVYbmzW0ySLzVg+Q75JvRd2UuNQ= 2023-11-02T18:21:13.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MVYbmzW0ySLzVg+Q75JvRd2UuNQ=', 'Content-Length': '0'} 2023-11-02T18:21:13.708 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037bbb1eba84deeaf-006543e899-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.709 DEBUG:boto:path=/ibtljn-54/d 2023-11-02T18:21:13.709 DEBUG:boto:auth_path=/ibtljn-54/d 2023-11-02T18:21:13.710 DEBUG:boto:Method: PUT 2023-11-02T18:21:13.710 DEBUG:boto:Path: /ibtljn-54/d 2023-11-02T18:21:13.711 DEBUG:boto:Data: 2023-11-02T18:21:13.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:21:13.712 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:13.712 DEBUG:boto:Port: 8000 2023-11-02T18:21:13.712 DEBUG:boto:Params: {} 2023-11-02T18:21:13.713 DEBUG:boto:Token: None 2023-11-02T18:21:13.713 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:21:13 GMT /ibtljn-54/d 2023-11-02T18:21:13.714 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I40AmCMpeBwioVg3JudhoIONqpM= 2023-11-02T18:21:13.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I40AmCMpeBwioVg3JudhoIONqpM='} 2023-11-02T18:21:13.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003dd3b8a336afcc7a-006543e899-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:13.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:13.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:13.760 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:21:13.807 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:13.807 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:13.862 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:21:13.886 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:21:13.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:13.887 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:21:13.887 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:21:13.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:13.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:13.887 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:21:13.888 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:21:13.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:13.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:13.888 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949273.579441_415.1", 2023-11-02T18:21:13.889 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:21:13.579441Z" 2023-11-02T18:21:13.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:13.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:13.889 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:21:13.889 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:21:13.890 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:13.890 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:21:13.891 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949273.579441_415.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:21:13.892 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:21:13.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:13.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:13.893 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:21:13.939 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:13.940 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:13.994+0000 7f4bee70ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:21:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:13.994+0000 7f4bee70ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:21:14.030 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:21:14.030 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:21:14.031 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:21:14.031 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:21:14.031 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:21:14.031 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:21:14.031 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:21:14.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:14.032 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:21:14.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:14.032 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:21:14.032 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:21:14.033 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:21:14.033 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:21:14.033 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:21:14.033 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:21:14.033 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:21:14.034 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:21:14.034 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:21:14.034 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:14.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:14.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:14.035 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:21:14.035 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:21:14.035 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:21:14.035 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:21:14.036 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:21:14.036 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:21:14.036 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:21:14.036 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:21:14.036 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:21:14.037 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:14.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:14.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:14.037 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:21:14.037 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:21:14.038 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:21:14.038 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949273.579441_415.1", 2023-11-02T18:21:14.038 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:21:14.038 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:21:14.039 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:21:14.039 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:21:13.579441Z", 2023-11-02T18:21:14.039 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:21:14.039 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:14.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:14.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:14.040 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:21:14.040 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:21:14.040 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:21:14.041 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:21:14.041 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:21:14.041 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:21:14.041 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:21:14.041 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:21:14.042 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:21:14.042 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:14.042 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:14.042 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:21:14.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:14.043 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:21:14.043 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:21:14.043 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:21:14.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:14.044 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:21:14Z" 2023-11-02T18:21:14.044 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:21:14.045 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949273.579441_415.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:21:13.579441Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:21:14Z" } 2023-11-02T18:21:14.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949273.579441_415.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:21:13.579441Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:21:14.046 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949273.579441_415.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:21:14.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949273.579441_415.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:21:14.047 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:21:14.047 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-54 2023-11-02T18:21:14.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:21:14.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:21:14.048 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:21:14.101 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:14.101 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:14.168 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:14.162+0000 7ff3eef92f40 1 waiting to reach incremental sync.. 2023-11-02T18:21:19.169 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:19.162+0000 7ff3eef92f40 1 waiting to reach incremental sync.. 2023-11-02T18:21:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:24.166+0000 7ff3eef92f40 1 bucket sync caught up with source: 2023-11-02T18:21:24.173 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.32.6, , , , 00000000001.122.6, , , 00000000001.34.6, , 00000000001.43.6] 2023-11-02T18:21:24.174 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.32.6, , , , 00000000001.122.6, , , 00000000001.34.6, , 00000000001.43.6] 2023-11-02T18:21:24.174 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:24.166+0000 7ff3eef92f40 0 bucket checkpoint complete 2023-11-02T18:21:24.204 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-54 2023-11-02T18:21:24.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:21:24.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:21:24.206 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-54 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:21:24.262 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:24.262 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:24.322+0000 7f5f77fc8f40 1 waiting to reach incremental sync.. 2023-11-02T18:21:29.329 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:29.322+0000 7f5f77fc8f40 1 waiting to reach incremental sync.. 2023-11-02T18:21:34.333 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:34.326+0000 7f5f77fc8f40 1 bucket sync caught up with source: 2023-11-02T18:21:34.333 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.227.6, , , , 00000000001.282.6, , , 00000000001.317.6, , 00000000001.251.6] 2023-11-02T18:21:34.333 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.227.6, , , , 00000000001.282.6, , , 00000000001.317.6, , 00000000001.251.6] 2023-11-02T18:21:34.333 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:34.326+0000 7f5f77fc8f40 0 bucket checkpoint complete 2023-11-02T18:21:34.363 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-54 zones={a2, a1} 2023-11-02T18:21:34.363 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.364 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.364 DEBUG:boto:Method: HEAD 2023-11-02T18:21:34.365 DEBUG:boto:Path: /ibtljn-54/ 2023-11-02T18:21:34.365 DEBUG:boto:Data: 2023-11-02T18:21:34.365 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.366 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.367 DEBUG:boto:Params: {} 2023-11-02T18:21:34.367 DEBUG:boto:Token: None 2023-11-02T18:21:34.368 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/ 2023-11-02T18:21:34.368 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nibpiI62hJ3kfv9vFJUW4wh2Ifc= 2023-11-02T18:21:34.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nibpiI62hJ3kfv9vFJUW4wh2Ifc=', 'Content-Length': '0'} 2023-11-02T18:21:34.372 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d21f2442ac91eabd-006543e8ae-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.373 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.373 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.373 DEBUG:boto:Method: HEAD 2023-11-02T18:21:34.374 DEBUG:boto:Path: /ibtljn-54/ 2023-11-02T18:21:34.374 DEBUG:boto:Data: 2023-11-02T18:21:34.375 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.375 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.375 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.376 DEBUG:boto:Params: {} 2023-11-02T18:21:34.376 DEBUG:boto:Token: None 2023-11-02T18:21:34.377 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/ 2023-11-02T18:21:34.377 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nibpiI62hJ3kfv9vFJUW4wh2Ifc= 2023-11-02T18:21:34.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nibpiI62hJ3kfv9vFJUW4wh2Ifc=', 'Content-Length': '0'} 2023-11-02T18:21:34.381 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a9d22b79a3194bb-006543e8ae-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.381 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:21:34.382 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.382 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.383 DEBUG:boto:path=/ibtljn-54/?versions&max-keys=999 2023-11-02T18:21:34.383 DEBUG:boto:auth_path=/ibtljn-54/?versions&max-keys=999 2023-11-02T18:21:34.383 DEBUG:boto:Method: GET 2023-11-02T18:21:34.384 DEBUG:boto:Path: /ibtljn-54/?versions&max-keys=999 2023-11-02T18:21:34.384 DEBUG:boto:Data: 2023-11-02T18:21:34.384 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.385 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.385 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.386 DEBUG:boto:Params: {} 2023-11-02T18:21:34.386 DEBUG:boto:Token: None 2023-11-02T18:21:34.387 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/?versions 2023-11-02T18:21:34.387 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kSVSSX94rENyEnjNwZKWyWlXUL8= 2023-11-02T18:21:34.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kSVSSX94rENyEnjNwZKWyWlXUL8=', 'Content-Length': '0'} 2023-11-02T18:21:34.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1a9dc64a6e4dc28-006543e8ae-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.392 DEBUG:boto:b'ibtljn-54999falseanulltrue2023-11-02T18:21:13.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:13.625Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:13.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:13.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:34.393 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:21:34.394 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:21:34.394 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:21:34.394 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:21:34.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:21:34.395 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.395 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.396 DEBUG:boto:path=/ibtljn-54/?versions&max-keys=999 2023-11-02T18:21:34.396 DEBUG:boto:auth_path=/ibtljn-54/?versions&max-keys=999 2023-11-02T18:21:34.396 DEBUG:boto:Method: GET 2023-11-02T18:21:34.397 DEBUG:boto:Path: /ibtljn-54/?versions&max-keys=999 2023-11-02T18:21:34.397 DEBUG:boto:Data: 2023-11-02T18:21:34.398 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.398 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.399 DEBUG:boto:Params: {} 2023-11-02T18:21:34.399 DEBUG:boto:Token: None 2023-11-02T18:21:34.400 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/?versions 2023-11-02T18:21:34.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kSVSSX94rENyEnjNwZKWyWlXUL8= 2023-11-02T18:21:34.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kSVSSX94rENyEnjNwZKWyWlXUL8=', 'Content-Length': '0'} 2023-11-02T18:21:34.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3267c863adc3b74-006543e8ae-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.406 DEBUG:boto:b'ibtljn-54999falseanulltrue2023-11-02T18:21:13.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:13.625Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:13.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:13.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:34.407 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:21:34.407 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:21:34.408 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:21:34.408 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:21:34.408 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.409 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.409 DEBUG:boto:path=/ibtljn-54/?versions&max-keys=999 2023-11-02T18:21:34.410 DEBUG:boto:auth_path=/ibtljn-54/?versions&max-keys=999 2023-11-02T18:21:34.410 DEBUG:boto:Method: GET 2023-11-02T18:21:34.411 DEBUG:boto:Path: /ibtljn-54/?versions&max-keys=999 2023-11-02T18:21:34.411 DEBUG:boto:Data: 2023-11-02T18:21:34.411 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.412 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.413 DEBUG:boto:Params: {} 2023-11-02T18:21:34.413 DEBUG:boto:Token: None 2023-11-02T18:21:34.414 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/?versions 2023-11-02T18:21:34.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kSVSSX94rENyEnjNwZKWyWlXUL8= 2023-11-02T18:21:34.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kSVSSX94rENyEnjNwZKWyWlXUL8=', 'Content-Length': '0'} 2023-11-02T18:21:34.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ae35b147d562b6e-006543e8ae-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.466 DEBUG:boto:b'ibtljn-54999falseanulltrue2023-11-02T18:21:13.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:13.625Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:13.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:13.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:34.467 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.468 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.468 DEBUG:boto:path=/ibtljn-54/?versions&max-keys=999 2023-11-02T18:21:34.469 DEBUG:boto:auth_path=/ibtljn-54/?versions&max-keys=999 2023-11-02T18:21:34.469 DEBUG:boto:Method: GET 2023-11-02T18:21:34.469 DEBUG:boto:Path: /ibtljn-54/?versions&max-keys=999 2023-11-02T18:21:34.470 DEBUG:boto:Data: 2023-11-02T18:21:34.470 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.470 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.471 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.471 DEBUG:boto:Params: {} 2023-11-02T18:21:34.472 DEBUG:boto:Token: None 2023-11-02T18:21:34.472 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/?versions 2023-11-02T18:21:34.473 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kSVSSX94rENyEnjNwZKWyWlXUL8= 2023-11-02T18:21:34.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kSVSSX94rENyEnjNwZKWyWlXUL8=', 'Content-Length': '0'} 2023-11-02T18:21:34.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004421bed8f66b8235-006543e8ae-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.478 DEBUG:boto:b'ibtljn-54999falseanulltrue2023-11-02T18:21:13.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:13.625Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:13.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:13.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:34.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:21:34.479 DEBUG:boto:path=/ibtljn-54/a 2023-11-02T18:21:34.480 DEBUG:boto:auth_path=/ibtljn-54/a 2023-11-02T18:21:34.480 DEBUG:boto:path=/ibtljn-54/a?versionId=null 2023-11-02T18:21:34.481 DEBUG:boto:auth_path=/ibtljn-54/a?versionId=null 2023-11-02T18:21:34.481 DEBUG:boto:Method: GET 2023-11-02T18:21:34.482 DEBUG:boto:Path: /ibtljn-54/a?versionId=null 2023-11-02T18:21:34.482 DEBUG:boto:Data: 2023-11-02T18:21:34.483 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.483 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.483 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.484 DEBUG:boto:Params: {} 2023-11-02T18:21:34.484 DEBUG:boto:Token: None 2023-11-02T18:21:34.485 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/a?versionId=null 2023-11-02T18:21:34.485 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2Bg+QfGmqLsPwGmLWK2nRCPQFCc= 2023-11-02T18:21:34.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2Bg+QfGmqLsPwGmLWK2nRCPQFCc=', 'Content-Length': '0'} 2023-11-02T18:21:34.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba60e21d73edd35-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.490 DEBUG:boto:path=/ibtljn-54/a 2023-11-02T18:21:34.491 DEBUG:boto:auth_path=/ibtljn-54/a 2023-11-02T18:21:34.491 DEBUG:boto:path=/ibtljn-54/a?versionId=null 2023-11-02T18:21:34.492 DEBUG:boto:auth_path=/ibtljn-54/a?versionId=null 2023-11-02T18:21:34.492 DEBUG:boto:Method: GET 2023-11-02T18:21:34.492 DEBUG:boto:Path: /ibtljn-54/a?versionId=null 2023-11-02T18:21:34.493 DEBUG:boto:Data: 2023-11-02T18:21:34.493 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.494 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.494 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.495 DEBUG:boto:Params: {} 2023-11-02T18:21:34.495 DEBUG:boto:Token: None 2023-11-02T18:21:34.496 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/a?versionId=null 2023-11-02T18:21:34.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2Bg+QfGmqLsPwGmLWK2nRCPQFCc= 2023-11-02T18:21:34.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2Bg+QfGmqLsPwGmLWK2nRCPQFCc=', 'Content-Length': '0'} 2023-11-02T18:21:34.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000797746df25c09ff1-006543e8ae-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.500 DEBUG:boto:path=/ibtljn-54/a 2023-11-02T18:21:34.500 DEBUG:boto:auth_path=/ibtljn-54/a 2023-11-02T18:21:34.501 DEBUG:boto:path=/ibtljn-54/a?versionId=null 2023-11-02T18:21:34.501 DEBUG:boto:auth_path=/ibtljn-54/a?versionId=null 2023-11-02T18:21:34.502 DEBUG:boto:Method: HEAD 2023-11-02T18:21:34.502 DEBUG:boto:Path: /ibtljn-54/a?versionId=null 2023-11-02T18:21:34.502 DEBUG:boto:Data: 2023-11-02T18:21:34.503 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.503 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.504 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.504 DEBUG:boto:Params: {} 2023-11-02T18:21:34.504 DEBUG:boto:Token: None 2023-11-02T18:21:34.505 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/a?versionId=null 2023-11-02T18:21:34.505 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:URsRUtLNYDBq/1gzVp9IJkuH7vk= 2023-11-02T18:21:34.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:URsRUtLNYDBq/1gzVp9IJkuH7vk=', 'Content-Length': '0'} 2023-11-02T18:21:34.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf884557b8e33727-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.509 DEBUG:boto:path=/ibtljn-54/a 2023-11-02T18:21:34.510 DEBUG:boto:auth_path=/ibtljn-54/a 2023-11-02T18:21:34.510 DEBUG:boto:path=/ibtljn-54/a?versionId=null 2023-11-02T18:21:34.511 DEBUG:boto:auth_path=/ibtljn-54/a?versionId=null 2023-11-02T18:21:34.511 DEBUG:boto:Method: HEAD 2023-11-02T18:21:34.511 DEBUG:boto:Path: /ibtljn-54/a?versionId=null 2023-11-02T18:21:34.512 DEBUG:boto:Data: 2023-11-02T18:21:34.512 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.513 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.513 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.513 DEBUG:boto:Params: {} 2023-11-02T18:21:34.514 DEBUG:boto:Token: None 2023-11-02T18:21:34.514 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/a?versionId=null 2023-11-02T18:21:34.515 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:URsRUtLNYDBq/1gzVp9IJkuH7vk= 2023-11-02T18:21:34.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:URsRUtLNYDBq/1gzVp9IJkuH7vk=', 'Content-Length': '0'} 2023-11-02T18:21:34.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7a63ca8871b81f4-006543e8ae-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:21:34.519 DEBUG:boto:path=/ibtljn-54/a 2023-11-02T18:21:34.520 DEBUG:boto:auth_path=/ibtljn-54/a 2023-11-02T18:21:34.520 DEBUG:boto:path=/ibtljn-54/a?versionId=null 2023-11-02T18:21:34.520 DEBUG:boto:auth_path=/ibtljn-54/a?versionId=null 2023-11-02T18:21:34.521 DEBUG:boto:Method: GET 2023-11-02T18:21:34.521 DEBUG:boto:Path: /ibtljn-54/a?versionId=null 2023-11-02T18:21:34.522 DEBUG:boto:Data: 2023-11-02T18:21:34.522 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.523 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.523 DEBUG:boto:Params: {} 2023-11-02T18:21:34.524 DEBUG:boto:Token: None 2023-11-02T18:21:34.524 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/a?versionId=null 2023-11-02T18:21:34.525 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2Bg+QfGmqLsPwGmLWK2nRCPQFCc= 2023-11-02T18:21:34.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2Bg+QfGmqLsPwGmLWK2nRCPQFCc=', 'Content-Length': '0'} 2023-11-02T18:21:34.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6eda3bcdb7d8266-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.528 DEBUG:boto:path=/ibtljn-54/a 2023-11-02T18:21:34.529 DEBUG:boto:auth_path=/ibtljn-54/a 2023-11-02T18:21:34.529 DEBUG:boto:path=/ibtljn-54/a?versionId=null 2023-11-02T18:21:34.530 DEBUG:boto:auth_path=/ibtljn-54/a?versionId=null 2023-11-02T18:21:34.530 DEBUG:boto:Method: GET 2023-11-02T18:21:34.530 DEBUG:boto:Path: /ibtljn-54/a?versionId=null 2023-11-02T18:21:34.531 DEBUG:boto:Data: 2023-11-02T18:21:34.531 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.532 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.532 DEBUG:boto:Params: {} 2023-11-02T18:21:34.533 DEBUG:boto:Token: None 2023-11-02T18:21:34.533 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/a?versionId=null 2023-11-02T18:21:34.534 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2Bg+QfGmqLsPwGmLWK2nRCPQFCc= 2023-11-02T18:21:34.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2Bg+QfGmqLsPwGmLWK2nRCPQFCc=', 'Content-Length': '0'} 2023-11-02T18:21:34.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d52ee38b1e015256-006543e8ae-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.538 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.539 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.539 DEBUG:boto:path=/ibtljn-54/?prefix=a 2023-11-02T18:21:34.539 DEBUG:boto:auth_path=/ibtljn-54/?prefix=a 2023-11-02T18:21:34.540 DEBUG:boto:Method: GET 2023-11-02T18:21:34.540 DEBUG:boto:Path: /ibtljn-54/?prefix=a 2023-11-02T18:21:34.541 DEBUG:boto:Data: 2023-11-02T18:21:34.541 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.542 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.542 DEBUG:boto:Params: {} 2023-11-02T18:21:34.543 DEBUG:boto:Token: None 2023-11-02T18:21:34.543 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/ 2023-11-02T18:21:34.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg= 2023-11-02T18:21:34.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg=', 'Content-Length': '0'} 2023-11-02T18:21:34.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2dbce6e8e2ff246-006543e8ae-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.549 DEBUG:boto:b'ibtljn-54a1000falsea2023-11-02T18:21:13.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:34.550 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.550 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.551 DEBUG:boto:path=/ibtljn-54/?prefix=a 2023-11-02T18:21:34.551 DEBUG:boto:auth_path=/ibtljn-54/?prefix=a 2023-11-02T18:21:34.552 DEBUG:boto:Method: GET 2023-11-02T18:21:34.552 DEBUG:boto:Path: /ibtljn-54/?prefix=a 2023-11-02T18:21:34.552 DEBUG:boto:Data: 2023-11-02T18:21:34.553 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.553 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.554 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.554 DEBUG:boto:Params: {} 2023-11-02T18:21:34.554 DEBUG:boto:Token: None 2023-11-02T18:21:34.555 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/ 2023-11-02T18:21:34.555 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg= 2023-11-02T18:21:34.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg=', 'Content-Length': '0'} 2023-11-02T18:21:34.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000502ed47424421e5f-006543e8ae-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.567 DEBUG:boto:b'ibtljn-54a1000falsea2023-11-02T18:21:13.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:34.568 DEBUG:boto:path=/ibtljn-54/a 2023-11-02T18:21:34.568 DEBUG:boto:auth_path=/ibtljn-54/a 2023-11-02T18:21:34.568 DEBUG:boto:Method: HEAD 2023-11-02T18:21:34.569 DEBUG:boto:Path: /ibtljn-54/a 2023-11-02T18:21:34.569 DEBUG:boto:Data: 2023-11-02T18:21:34.570 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.570 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.570 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.571 DEBUG:boto:Params: {} 2023-11-02T18:21:34.571 DEBUG:boto:Token: None 2023-11-02T18:21:34.572 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/a 2023-11-02T18:21:34.572 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzhgprlz1TexLP19fApHXxSweYU= 2023-11-02T18:21:34.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzhgprlz1TexLP19fApHXxSweYU=', 'Content-Length': '0'} 2023-11-02T18:21:34.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d67243be75bdf36-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.575 DEBUG:boto:path=/ibtljn-54/a 2023-11-02T18:21:34.575 DEBUG:boto:auth_path=/ibtljn-54/a 2023-11-02T18:21:34.576 DEBUG:boto:Method: HEAD 2023-11-02T18:21:34.576 DEBUG:boto:Path: /ibtljn-54/a 2023-11-02T18:21:34.576 DEBUG:boto:Data: 2023-11-02T18:21:34.577 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.577 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.578 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.578 DEBUG:boto:Params: {} 2023-11-02T18:21:34.578 DEBUG:boto:Token: None 2023-11-02T18:21:34.579 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/a 2023-11-02T18:21:34.579 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzhgprlz1TexLP19fApHXxSweYU= 2023-11-02T18:21:34.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzhgprlz1TexLP19fApHXxSweYU=', 'Content-Length': '0'} 2023-11-02T18:21:34.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dca6e7c530e57c1-006543e8ae-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:21:34.583 DEBUG:boto:path=/ibtljn-54/a 2023-11-02T18:21:34.583 DEBUG:boto:auth_path=/ibtljn-54/a 2023-11-02T18:21:34.583 DEBUG:boto:Method: GET 2023-11-02T18:21:34.584 DEBUG:boto:Path: /ibtljn-54/a 2023-11-02T18:21:34.584 DEBUG:boto:Data: 2023-11-02T18:21:34.585 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.585 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.586 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.586 DEBUG:boto:Params: {} 2023-11-02T18:21:34.586 DEBUG:boto:Token: None 2023-11-02T18:21:34.587 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/a 2023-11-02T18:21:34.587 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jAZlfZBELARZ9o4qJZypf4GIdNE= 2023-11-02T18:21:34.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jAZlfZBELARZ9o4qJZypf4GIdNE=', 'Content-Length': '0'} 2023-11-02T18:21:34.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bd0c10f5d231b29-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.634 DEBUG:boto:path=/ibtljn-54/a 2023-11-02T18:21:34.635 DEBUG:boto:auth_path=/ibtljn-54/a 2023-11-02T18:21:34.636 DEBUG:boto:Method: GET 2023-11-02T18:21:34.636 DEBUG:boto:Path: /ibtljn-54/a 2023-11-02T18:21:34.636 DEBUG:boto:Data: 2023-11-02T18:21:34.637 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.638 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.638 DEBUG:boto:Params: {} 2023-11-02T18:21:34.638 DEBUG:boto:Token: None 2023-11-02T18:21:34.639 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/a 2023-11-02T18:21:34.639 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jAZlfZBELARZ9o4qJZypf4GIdNE= 2023-11-02T18:21:34.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jAZlfZBELARZ9o4qJZypf4GIdNE=', 'Content-Length': '0'} 2023-11-02T18:21:34.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000620bfb94f935a4e2-006543e8ae-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.644 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.644 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.645 DEBUG:boto:path=/ibtljn-54/?prefix=a 2023-11-02T18:21:34.645 DEBUG:boto:auth_path=/ibtljn-54/?prefix=a 2023-11-02T18:21:34.645 DEBUG:boto:Method: GET 2023-11-02T18:21:34.646 DEBUG:boto:Path: /ibtljn-54/?prefix=a 2023-11-02T18:21:34.646 DEBUG:boto:Data: 2023-11-02T18:21:34.647 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.647 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.648 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.648 DEBUG:boto:Params: {} 2023-11-02T18:21:34.649 DEBUG:boto:Token: None 2023-11-02T18:21:34.649 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/ 2023-11-02T18:21:34.649 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg= 2023-11-02T18:21:34.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg=', 'Content-Length': '0'} 2023-11-02T18:21:34.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f058252b9483c9bd-006543e8ae-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.655 DEBUG:boto:b'ibtljn-54a1000falsea2023-11-02T18:21:13.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:34.656 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.656 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.657 DEBUG:boto:path=/ibtljn-54/?prefix=a 2023-11-02T18:21:34.657 DEBUG:boto:auth_path=/ibtljn-54/?prefix=a 2023-11-02T18:21:34.657 DEBUG:boto:Method: GET 2023-11-02T18:21:34.658 DEBUG:boto:Path: /ibtljn-54/?prefix=a 2023-11-02T18:21:34.658 DEBUG:boto:Data: 2023-11-02T18:21:34.659 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.659 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.660 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.660 DEBUG:boto:Params: {} 2023-11-02T18:21:34.660 DEBUG:boto:Token: None 2023-11-02T18:21:34.661 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/ 2023-11-02T18:21:34.661 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg= 2023-11-02T18:21:34.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg=', 'Content-Length': '0'} 2023-11-02T18:21:34.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052a148006a786cff-006543e8ae-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.667 DEBUG:boto:b'ibtljn-54a1000falsea2023-11-02T18:21:13.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:34.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:21:34.668 DEBUG:boto:path=/ibtljn-54/b 2023-11-02T18:21:34.668 DEBUG:boto:auth_path=/ibtljn-54/b 2023-11-02T18:21:34.669 DEBUG:boto:path=/ibtljn-54/b?versionId=null 2023-11-02T18:21:34.669 DEBUG:boto:auth_path=/ibtljn-54/b?versionId=null 2023-11-02T18:21:34.669 DEBUG:boto:Method: GET 2023-11-02T18:21:34.670 DEBUG:boto:Path: /ibtljn-54/b?versionId=null 2023-11-02T18:21:34.670 DEBUG:boto:Data: 2023-11-02T18:21:34.671 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.671 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.672 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.672 DEBUG:boto:Params: {} 2023-11-02T18:21:34.673 DEBUG:boto:Token: None 2023-11-02T18:21:34.673 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/b?versionId=null 2023-11-02T18:21:34.674 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FJ5/uvacnF0cyW+uOccjkaJ5fdg= 2023-11-02T18:21:34.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FJ5/uvacnF0cyW+uOccjkaJ5fdg=', 'Content-Length': '0'} 2023-11-02T18:21:34.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069c4b9fa3e0a64ff-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.678 DEBUG:boto:path=/ibtljn-54/b 2023-11-02T18:21:34.678 DEBUG:boto:auth_path=/ibtljn-54/b 2023-11-02T18:21:34.678 DEBUG:boto:path=/ibtljn-54/b?versionId=null 2023-11-02T18:21:34.679 DEBUG:boto:auth_path=/ibtljn-54/b?versionId=null 2023-11-02T18:21:34.679 DEBUG:boto:Method: GET 2023-11-02T18:21:34.680 DEBUG:boto:Path: /ibtljn-54/b?versionId=null 2023-11-02T18:21:34.680 DEBUG:boto:Data: 2023-11-02T18:21:34.680 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.681 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.681 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.682 DEBUG:boto:Params: {} 2023-11-02T18:21:34.682 DEBUG:boto:Token: None 2023-11-02T18:21:34.683 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/b?versionId=null 2023-11-02T18:21:34.683 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FJ5/uvacnF0cyW+uOccjkaJ5fdg= 2023-11-02T18:21:34.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FJ5/uvacnF0cyW+uOccjkaJ5fdg=', 'Content-Length': '0'} 2023-11-02T18:21:34.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000493edaebd6331c4c-006543e8ae-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.687 DEBUG:boto:path=/ibtljn-54/b 2023-11-02T18:21:34.687 DEBUG:boto:auth_path=/ibtljn-54/b 2023-11-02T18:21:34.688 DEBUG:boto:path=/ibtljn-54/b?versionId=null 2023-11-02T18:21:34.688 DEBUG:boto:auth_path=/ibtljn-54/b?versionId=null 2023-11-02T18:21:34.688 DEBUG:boto:Method: HEAD 2023-11-02T18:21:34.689 DEBUG:boto:Path: /ibtljn-54/b?versionId=null 2023-11-02T18:21:34.689 DEBUG:boto:Data: 2023-11-02T18:21:34.690 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.691 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.691 DEBUG:boto:Params: {} 2023-11-02T18:21:34.692 DEBUG:boto:Token: None 2023-11-02T18:21:34.692 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/b?versionId=null 2023-11-02T18:21:34.692 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PFxASJaQILcd347WpsEFNrgMOIw= 2023-11-02T18:21:34.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PFxASJaQILcd347WpsEFNrgMOIw=', 'Content-Length': '0'} 2023-11-02T18:21:34.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee92bf5e8e5c73aa-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.696 DEBUG:boto:path=/ibtljn-54/b 2023-11-02T18:21:34.697 DEBUG:boto:auth_path=/ibtljn-54/b 2023-11-02T18:21:34.697 DEBUG:boto:path=/ibtljn-54/b?versionId=null 2023-11-02T18:21:34.697 DEBUG:boto:auth_path=/ibtljn-54/b?versionId=null 2023-11-02T18:21:34.698 DEBUG:boto:Method: HEAD 2023-11-02T18:21:34.698 DEBUG:boto:Path: /ibtljn-54/b?versionId=null 2023-11-02T18:21:34.698 DEBUG:boto:Data: 2023-11-02T18:21:34.699 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.699 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.700 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.700 DEBUG:boto:Params: {} 2023-11-02T18:21:34.700 DEBUG:boto:Token: None 2023-11-02T18:21:34.701 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/b?versionId=null 2023-11-02T18:21:34.701 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PFxASJaQILcd347WpsEFNrgMOIw= 2023-11-02T18:21:34.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PFxASJaQILcd347WpsEFNrgMOIw=', 'Content-Length': '0'} 2023-11-02T18:21:34.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d12c43761d56d9a-006543e8ae-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:21:34.705 DEBUG:boto:path=/ibtljn-54/b 2023-11-02T18:21:34.706 DEBUG:boto:auth_path=/ibtljn-54/b 2023-11-02T18:21:34.706 DEBUG:boto:path=/ibtljn-54/b?versionId=null 2023-11-02T18:21:34.707 DEBUG:boto:auth_path=/ibtljn-54/b?versionId=null 2023-11-02T18:21:34.707 DEBUG:boto:Method: GET 2023-11-02T18:21:34.708 DEBUG:boto:Path: /ibtljn-54/b?versionId=null 2023-11-02T18:21:34.708 DEBUG:boto:Data: 2023-11-02T18:21:34.708 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.709 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.709 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.709 DEBUG:boto:Params: {} 2023-11-02T18:21:34.710 DEBUG:boto:Token: None 2023-11-02T18:21:34.710 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/b?versionId=null 2023-11-02T18:21:34.711 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FJ5/uvacnF0cyW+uOccjkaJ5fdg= 2023-11-02T18:21:34.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FJ5/uvacnF0cyW+uOccjkaJ5fdg=', 'Content-Length': '0'} 2023-11-02T18:21:34.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5fc3ac72d12d60-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.754 DEBUG:boto:path=/ibtljn-54/b 2023-11-02T18:21:34.755 DEBUG:boto:auth_path=/ibtljn-54/b 2023-11-02T18:21:34.755 DEBUG:boto:path=/ibtljn-54/b?versionId=null 2023-11-02T18:21:34.755 DEBUG:boto:auth_path=/ibtljn-54/b?versionId=null 2023-11-02T18:21:34.756 DEBUG:boto:Method: GET 2023-11-02T18:21:34.756 DEBUG:boto:Path: /ibtljn-54/b?versionId=null 2023-11-02T18:21:34.757 DEBUG:boto:Data: 2023-11-02T18:21:34.757 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.757 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.758 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.758 DEBUG:boto:Params: {} 2023-11-02T18:21:34.759 DEBUG:boto:Token: None 2023-11-02T18:21:34.759 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/b?versionId=null 2023-11-02T18:21:34.760 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FJ5/uvacnF0cyW+uOccjkaJ5fdg= 2023-11-02T18:21:34.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FJ5/uvacnF0cyW+uOccjkaJ5fdg=', 'Content-Length': '0'} 2023-11-02T18:21:34.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfcd0c477e9188ce-006543e8ae-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.763 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.764 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.764 DEBUG:boto:path=/ibtljn-54/?prefix=b 2023-11-02T18:21:34.765 DEBUG:boto:auth_path=/ibtljn-54/?prefix=b 2023-11-02T18:21:34.765 DEBUG:boto:Method: GET 2023-11-02T18:21:34.765 DEBUG:boto:Path: /ibtljn-54/?prefix=b 2023-11-02T18:21:34.766 DEBUG:boto:Data: 2023-11-02T18:21:34.766 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.767 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.767 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.767 DEBUG:boto:Params: {} 2023-11-02T18:21:34.768 DEBUG:boto:Token: None 2023-11-02T18:21:34.768 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/ 2023-11-02T18:21:34.769 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg= 2023-11-02T18:21:34.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg=', 'Content-Length': '0'} 2023-11-02T18:21:34.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9e2557a0d2df5b7-006543e8ae-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.774 DEBUG:boto:b'ibtljn-54b1000falseb2023-11-02T18:21:13.625Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:34.775 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.775 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.776 DEBUG:boto:path=/ibtljn-54/?prefix=b 2023-11-02T18:21:34.776 DEBUG:boto:auth_path=/ibtljn-54/?prefix=b 2023-11-02T18:21:34.777 DEBUG:boto:Method: GET 2023-11-02T18:21:34.777 DEBUG:boto:Path: /ibtljn-54/?prefix=b 2023-11-02T18:21:34.777 DEBUG:boto:Data: 2023-11-02T18:21:34.778 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.778 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.779 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.779 DEBUG:boto:Params: {} 2023-11-02T18:21:34.780 DEBUG:boto:Token: None 2023-11-02T18:21:34.780 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/ 2023-11-02T18:21:34.781 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg= 2023-11-02T18:21:34.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg=', 'Content-Length': '0'} 2023-11-02T18:21:34.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea503687c71adc25-006543e8ae-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.786 DEBUG:boto:b'ibtljn-54b1000falseb2023-11-02T18:21:13.625Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:34.787 DEBUG:boto:path=/ibtljn-54/b 2023-11-02T18:21:34.787 DEBUG:boto:auth_path=/ibtljn-54/b 2023-11-02T18:21:34.788 DEBUG:boto:Method: HEAD 2023-11-02T18:21:34.788 DEBUG:boto:Path: /ibtljn-54/b 2023-11-02T18:21:34.789 DEBUG:boto:Data: 2023-11-02T18:21:34.789 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.789 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.790 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.790 DEBUG:boto:Params: {} 2023-11-02T18:21:34.791 DEBUG:boto:Token: None 2023-11-02T18:21:34.791 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/b 2023-11-02T18:21:34.791 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tlp7xF0/hBwQQh1KTYkasFSsYRA= 2023-11-02T18:21:34.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tlp7xF0/hBwQQh1KTYkasFSsYRA=', 'Content-Length': '0'} 2023-11-02T18:21:34.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc8dee64a20766cb-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.795 DEBUG:boto:path=/ibtljn-54/b 2023-11-02T18:21:34.796 DEBUG:boto:auth_path=/ibtljn-54/b 2023-11-02T18:21:34.796 DEBUG:boto:Method: HEAD 2023-11-02T18:21:34.797 DEBUG:boto:Path: /ibtljn-54/b 2023-11-02T18:21:34.797 DEBUG:boto:Data: 2023-11-02T18:21:34.797 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.798 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.798 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.799 DEBUG:boto:Params: {} 2023-11-02T18:21:34.799 DEBUG:boto:Token: None 2023-11-02T18:21:34.799 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/b 2023-11-02T18:21:34.800 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tlp7xF0/hBwQQh1KTYkasFSsYRA= 2023-11-02T18:21:34.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tlp7xF0/hBwQQh1KTYkasFSsYRA=', 'Content-Length': '0'} 2023-11-02T18:21:34.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6069be27fb1c577-006543e8ae-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:21:34.804 DEBUG:boto:path=/ibtljn-54/b 2023-11-02T18:21:34.804 DEBUG:boto:auth_path=/ibtljn-54/b 2023-11-02T18:21:34.805 DEBUG:boto:Method: GET 2023-11-02T18:21:34.805 DEBUG:boto:Path: /ibtljn-54/b 2023-11-02T18:21:34.806 DEBUG:boto:Data: 2023-11-02T18:21:34.806 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.806 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.807 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.807 DEBUG:boto:Params: {} 2023-11-02T18:21:34.808 DEBUG:boto:Token: None 2023-11-02T18:21:34.808 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/b 2023-11-02T18:21:34.809 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yy+UwQfdxQ3blRt3er8+NAWVBwE= 2023-11-02T18:21:34.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yy+UwQfdxQ3blRt3er8+NAWVBwE=', 'Content-Length': '0'} 2023-11-02T18:21:34.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ca268792269211d-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.854 DEBUG:boto:path=/ibtljn-54/b 2023-11-02T18:21:34.855 DEBUG:boto:auth_path=/ibtljn-54/b 2023-11-02T18:21:34.855 DEBUG:boto:Method: GET 2023-11-02T18:21:34.856 DEBUG:boto:Path: /ibtljn-54/b 2023-11-02T18:21:34.856 DEBUG:boto:Data: 2023-11-02T18:21:34.856 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.857 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.857 DEBUG:boto:Params: {} 2023-11-02T18:21:34.858 DEBUG:boto:Token: None 2023-11-02T18:21:34.858 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/b 2023-11-02T18:21:34.859 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yy+UwQfdxQ3blRt3er8+NAWVBwE= 2023-11-02T18:21:34.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yy+UwQfdxQ3blRt3er8+NAWVBwE=', 'Content-Length': '0'} 2023-11-02T18:21:34.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2b6e798a164afac-006543e8ae-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.863 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.863 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.864 DEBUG:boto:path=/ibtljn-54/?prefix=b 2023-11-02T18:21:34.864 DEBUG:boto:auth_path=/ibtljn-54/?prefix=b 2023-11-02T18:21:34.864 DEBUG:boto:Method: GET 2023-11-02T18:21:34.865 DEBUG:boto:Path: /ibtljn-54/?prefix=b 2023-11-02T18:21:34.865 DEBUG:boto:Data: 2023-11-02T18:21:34.865 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.866 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.866 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.867 DEBUG:boto:Params: {} 2023-11-02T18:21:34.867 DEBUG:boto:Token: None 2023-11-02T18:21:34.868 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/ 2023-11-02T18:21:34.868 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg= 2023-11-02T18:21:34.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg=', 'Content-Length': '0'} 2023-11-02T18:21:34.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093455c55904ed3d1-006543e8ae-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.874 DEBUG:boto:b'ibtljn-54b1000falseb2023-11-02T18:21:13.625Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:34.875 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:34.875 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:34.875 DEBUG:boto:path=/ibtljn-54/?prefix=b 2023-11-02T18:21:34.876 DEBUG:boto:auth_path=/ibtljn-54/?prefix=b 2023-11-02T18:21:34.876 DEBUG:boto:Method: GET 2023-11-02T18:21:34.877 DEBUG:boto:Path: /ibtljn-54/?prefix=b 2023-11-02T18:21:34.877 DEBUG:boto:Data: 2023-11-02T18:21:34.878 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.878 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.878 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.879 DEBUG:boto:Params: {} 2023-11-02T18:21:34.879 DEBUG:boto:Token: None 2023-11-02T18:21:34.880 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/ 2023-11-02T18:21:34.880 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg= 2023-11-02T18:21:34.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0UFEMcNMzgNYW0gYFkXLx9WRqdg=', 'Content-Length': '0'} 2023-11-02T18:21:34.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe16adb90d77fae3-006543e8ae-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.926 DEBUG:boto:b'ibtljn-54b1000falseb2023-11-02T18:21:13.625Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:34.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:21:34.927 DEBUG:boto:path=/ibtljn-54/c 2023-11-02T18:21:34.928 DEBUG:boto:auth_path=/ibtljn-54/c 2023-11-02T18:21:34.928 DEBUG:boto:path=/ibtljn-54/c?versionId=null 2023-11-02T18:21:34.928 DEBUG:boto:auth_path=/ibtljn-54/c?versionId=null 2023-11-02T18:21:34.929 DEBUG:boto:Method: GET 2023-11-02T18:21:34.929 DEBUG:boto:Path: /ibtljn-54/c?versionId=null 2023-11-02T18:21:34.930 DEBUG:boto:Data: 2023-11-02T18:21:34.930 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.931 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.931 DEBUG:boto:Params: {} 2023-11-02T18:21:34.932 DEBUG:boto:Token: None 2023-11-02T18:21:34.932 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/c?versionId=null 2023-11-02T18:21:34.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pLytC1fIq5QuBLd0zhJVyuLAqIM= 2023-11-02T18:21:34.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pLytC1fIq5QuBLd0zhJVyuLAqIM=', 'Content-Length': '0'} 2023-11-02T18:21:34.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000851a0aa6ccfc9bb2-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.937 DEBUG:boto:path=/ibtljn-54/c 2023-11-02T18:21:34.937 DEBUG:boto:auth_path=/ibtljn-54/c 2023-11-02T18:21:34.937 DEBUG:boto:path=/ibtljn-54/c?versionId=null 2023-11-02T18:21:34.938 DEBUG:boto:auth_path=/ibtljn-54/c?versionId=null 2023-11-02T18:21:34.938 DEBUG:boto:Method: GET 2023-11-02T18:21:34.939 DEBUG:boto:Path: /ibtljn-54/c?versionId=null 2023-11-02T18:21:34.939 DEBUG:boto:Data: 2023-11-02T18:21:34.939 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.940 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.940 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.941 DEBUG:boto:Params: {} 2023-11-02T18:21:34.941 DEBUG:boto:Token: None 2023-11-02T18:21:34.941 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/c?versionId=null 2023-11-02T18:21:34.942 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pLytC1fIq5QuBLd0zhJVyuLAqIM= 2023-11-02T18:21:34.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pLytC1fIq5QuBLd0zhJVyuLAqIM=', 'Content-Length': '0'} 2023-11-02T18:21:34.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5d1a11c540cef6d-006543e8ae-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.946 DEBUG:boto:path=/ibtljn-54/c 2023-11-02T18:21:34.946 DEBUG:boto:auth_path=/ibtljn-54/c 2023-11-02T18:21:34.947 DEBUG:boto:path=/ibtljn-54/c?versionId=null 2023-11-02T18:21:34.947 DEBUG:boto:auth_path=/ibtljn-54/c?versionId=null 2023-11-02T18:21:34.947 DEBUG:boto:Method: HEAD 2023-11-02T18:21:34.948 DEBUG:boto:Path: /ibtljn-54/c?versionId=null 2023-11-02T18:21:34.948 DEBUG:boto:Data: 2023-11-02T18:21:34.948 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.949 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.949 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.950 DEBUG:boto:Params: {} 2023-11-02T18:21:34.950 DEBUG:boto:Token: None 2023-11-02T18:21:34.951 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/c?versionId=null 2023-11-02T18:21:34.951 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eGSkc3j0d45Lk2vUV9iI7kj5nFI= 2023-11-02T18:21:34.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eGSkc3j0d45Lk2vUV9iI7kj5nFI=', 'Content-Length': '0'} 2023-11-02T18:21:34.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae1104b58656fcb3-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.955 DEBUG:boto:path=/ibtljn-54/c 2023-11-02T18:21:34.955 DEBUG:boto:auth_path=/ibtljn-54/c 2023-11-02T18:21:34.956 DEBUG:boto:path=/ibtljn-54/c?versionId=null 2023-11-02T18:21:34.956 DEBUG:boto:auth_path=/ibtljn-54/c?versionId=null 2023-11-02T18:21:34.957 DEBUG:boto:Method: HEAD 2023-11-02T18:21:34.957 DEBUG:boto:Path: /ibtljn-54/c?versionId=null 2023-11-02T18:21:34.957 DEBUG:boto:Data: 2023-11-02T18:21:34.958 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.958 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:34.958 DEBUG:boto:Port: 8000 2023-11-02T18:21:34.959 DEBUG:boto:Params: {} 2023-11-02T18:21:34.959 DEBUG:boto:Token: None 2023-11-02T18:21:34.960 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/c?versionId=null 2023-11-02T18:21:34.960 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eGSkc3j0d45Lk2vUV9iI7kj5nFI= 2023-11-02T18:21:34.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eGSkc3j0d45Lk2vUV9iI7kj5nFI=', 'Content-Length': '0'} 2023-11-02T18:21:34.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000255367aaf8292203-006543e8ae-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:34.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:21:34.964 DEBUG:boto:path=/ibtljn-54/c 2023-11-02T18:21:34.964 DEBUG:boto:auth_path=/ibtljn-54/c 2023-11-02T18:21:34.965 DEBUG:boto:path=/ibtljn-54/c?versionId=null 2023-11-02T18:21:34.965 DEBUG:boto:auth_path=/ibtljn-54/c?versionId=null 2023-11-02T18:21:34.966 DEBUG:boto:Method: GET 2023-11-02T18:21:34.966 DEBUG:boto:Path: /ibtljn-54/c?versionId=null 2023-11-02T18:21:34.966 DEBUG:boto:Data: 2023-11-02T18:21:34.967 DEBUG:boto:Headers: {} 2023-11-02T18:21:34.967 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:34.968 DEBUG:boto:Port: 8001 2023-11-02T18:21:34.968 DEBUG:boto:Params: {} 2023-11-02T18:21:34.968 DEBUG:boto:Token: None 2023-11-02T18:21:34.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:34 GMT /ibtljn-54/c?versionId=null 2023-11-02T18:21:34.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pLytC1fIq5QuBLd0zhJVyuLAqIM= 2023-11-02T18:21:34.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pLytC1fIq5QuBLd0zhJVyuLAqIM=', 'Content-Length': '0'} 2023-11-02T18:21:34.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd4035878ff7e738-006543e8ae-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.014 DEBUG:boto:path=/ibtljn-54/c 2023-11-02T18:21:35.015 DEBUG:boto:auth_path=/ibtljn-54/c 2023-11-02T18:21:35.015 DEBUG:boto:path=/ibtljn-54/c?versionId=null 2023-11-02T18:21:35.016 DEBUG:boto:auth_path=/ibtljn-54/c?versionId=null 2023-11-02T18:21:35.016 DEBUG:boto:Method: GET 2023-11-02T18:21:35.016 DEBUG:boto:Path: /ibtljn-54/c?versionId=null 2023-11-02T18:21:35.017 DEBUG:boto:Data: 2023-11-02T18:21:35.017 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.017 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:35.018 DEBUG:boto:Port: 8000 2023-11-02T18:21:35.018 DEBUG:boto:Params: {} 2023-11-02T18:21:35.019 DEBUG:boto:Token: None 2023-11-02T18:21:35.019 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/c?versionId=null 2023-11-02T18:21:35.020 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Nzg52pVLXQj0t7GrSUvXpeGnbc4= 2023-11-02T18:21:35.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Nzg52pVLXQj0t7GrSUvXpeGnbc4=', 'Content-Length': '0'} 2023-11-02T18:21:35.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b12947ced5b071bf-006543e8af-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.023 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:35.024 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:35.024 DEBUG:boto:path=/ibtljn-54/?prefix=c 2023-11-02T18:21:35.025 DEBUG:boto:auth_path=/ibtljn-54/?prefix=c 2023-11-02T18:21:35.025 DEBUG:boto:Method: GET 2023-11-02T18:21:35.025 DEBUG:boto:Path: /ibtljn-54/?prefix=c 2023-11-02T18:21:35.026 DEBUG:boto:Data: 2023-11-02T18:21:35.026 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.027 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:35.027 DEBUG:boto:Port: 8001 2023-11-02T18:21:35.027 DEBUG:boto:Params: {} 2023-11-02T18:21:35.028 DEBUG:boto:Token: None 2023-11-02T18:21:35.028 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/ 2023-11-02T18:21:35.029 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE= 2023-11-02T18:21:35.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE=', 'Content-Length': '0'} 2023-11-02T18:21:35.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d8c6aa8852cc6d1-006543e8af-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.034 DEBUG:boto:b'ibtljn-54c1000falsec2023-11-02T18:21:13.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:35.035 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:35.035 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:35.036 DEBUG:boto:path=/ibtljn-54/?prefix=c 2023-11-02T18:21:35.036 DEBUG:boto:auth_path=/ibtljn-54/?prefix=c 2023-11-02T18:21:35.037 DEBUG:boto:Method: GET 2023-11-02T18:21:35.037 DEBUG:boto:Path: /ibtljn-54/?prefix=c 2023-11-02T18:21:35.037 DEBUG:boto:Data: 2023-11-02T18:21:35.038 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:35.039 DEBUG:boto:Port: 8000 2023-11-02T18:21:35.039 DEBUG:boto:Params: {} 2023-11-02T18:21:35.040 DEBUG:boto:Token: None 2023-11-02T18:21:35.040 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/ 2023-11-02T18:21:35.041 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE= 2023-11-02T18:21:35.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE=', 'Content-Length': '0'} 2023-11-02T18:21:35.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ba1132ac95e975c-006543e8af-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.046 DEBUG:boto:b'ibtljn-54c1000falsec2023-11-02T18:21:13.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:35.047 DEBUG:boto:path=/ibtljn-54/c 2023-11-02T18:21:35.047 DEBUG:boto:auth_path=/ibtljn-54/c 2023-11-02T18:21:35.048 DEBUG:boto:Method: HEAD 2023-11-02T18:21:35.048 DEBUG:boto:Path: /ibtljn-54/c 2023-11-02T18:21:35.048 DEBUG:boto:Data: 2023-11-02T18:21:35.049 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.049 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:35.050 DEBUG:boto:Port: 8001 2023-11-02T18:21:35.050 DEBUG:boto:Params: {} 2023-11-02T18:21:35.050 DEBUG:boto:Token: None 2023-11-02T18:21:35.051 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/c 2023-11-02T18:21:35.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iCKBOgkXNbZkCuqMF9Bg1Isr2q8= 2023-11-02T18:21:35.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iCKBOgkXNbZkCuqMF9Bg1Isr2q8=', 'Content-Length': '0'} 2023-11-02T18:21:35.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e251b034b8cb86f4-006543e8af-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.055 DEBUG:boto:path=/ibtljn-54/c 2023-11-02T18:21:35.055 DEBUG:boto:auth_path=/ibtljn-54/c 2023-11-02T18:21:35.056 DEBUG:boto:Method: HEAD 2023-11-02T18:21:35.056 DEBUG:boto:Path: /ibtljn-54/c 2023-11-02T18:21:35.056 DEBUG:boto:Data: 2023-11-02T18:21:35.057 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.057 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:35.058 DEBUG:boto:Port: 8000 2023-11-02T18:21:35.058 DEBUG:boto:Params: {} 2023-11-02T18:21:35.058 DEBUG:boto:Token: None 2023-11-02T18:21:35.059 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/c 2023-11-02T18:21:35.059 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iCKBOgkXNbZkCuqMF9Bg1Isr2q8= 2023-11-02T18:21:35.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iCKBOgkXNbZkCuqMF9Bg1Isr2q8=', 'Content-Length': '0'} 2023-11-02T18:21:35.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f33142128136506-006543e8af-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:21:35.063 DEBUG:boto:path=/ibtljn-54/c 2023-11-02T18:21:35.064 DEBUG:boto:auth_path=/ibtljn-54/c 2023-11-02T18:21:35.064 DEBUG:boto:Method: GET 2023-11-02T18:21:35.065 DEBUG:boto:Path: /ibtljn-54/c 2023-11-02T18:21:35.065 DEBUG:boto:Data: 2023-11-02T18:21:35.065 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.066 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:35.066 DEBUG:boto:Port: 8001 2023-11-02T18:21:35.067 DEBUG:boto:Params: {} 2023-11-02T18:21:35.067 DEBUG:boto:Token: None 2023-11-02T18:21:35.068 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/c 2023-11-02T18:21:35.068 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B/F2cFOI0CKlrINeHQmGTRGcYuc= 2023-11-02T18:21:35.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B/F2cFOI0CKlrINeHQmGTRGcYuc=', 'Content-Length': '0'} 2023-11-02T18:21:35.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9a0fc68a9b5cf03-006543e8af-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.118 DEBUG:boto:path=/ibtljn-54/c 2023-11-02T18:21:35.119 DEBUG:boto:auth_path=/ibtljn-54/c 2023-11-02T18:21:35.119 DEBUG:boto:Method: GET 2023-11-02T18:21:35.120 DEBUG:boto:Path: /ibtljn-54/c 2023-11-02T18:21:35.120 DEBUG:boto:Data: 2023-11-02T18:21:35.120 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.121 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:35.121 DEBUG:boto:Port: 8000 2023-11-02T18:21:35.121 DEBUG:boto:Params: {} 2023-11-02T18:21:35.122 DEBUG:boto:Token: None 2023-11-02T18:21:35.122 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/c 2023-11-02T18:21:35.123 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B/F2cFOI0CKlrINeHQmGTRGcYuc= 2023-11-02T18:21:35.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B/F2cFOI0CKlrINeHQmGTRGcYuc=', 'Content-Length': '0'} 2023-11-02T18:21:35.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eab501867840e77-006543e8af-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.127 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:35.127 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:35.128 DEBUG:boto:path=/ibtljn-54/?prefix=c 2023-11-02T18:21:35.128 DEBUG:boto:auth_path=/ibtljn-54/?prefix=c 2023-11-02T18:21:35.128 DEBUG:boto:Method: GET 2023-11-02T18:21:35.129 DEBUG:boto:Path: /ibtljn-54/?prefix=c 2023-11-02T18:21:35.129 DEBUG:boto:Data: 2023-11-02T18:21:35.130 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.130 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:35.130 DEBUG:boto:Port: 8001 2023-11-02T18:21:35.131 DEBUG:boto:Params: {} 2023-11-02T18:21:35.131 DEBUG:boto:Token: None 2023-11-02T18:21:35.132 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/ 2023-11-02T18:21:35.132 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE= 2023-11-02T18:21:35.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE=', 'Content-Length': '0'} 2023-11-02T18:21:35.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026aaec99247b5a2f-006543e8af-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.138 DEBUG:boto:b'ibtljn-54c1000falsec2023-11-02T18:21:13.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:35.139 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:35.140 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:35.140 DEBUG:boto:path=/ibtljn-54/?prefix=c 2023-11-02T18:21:35.141 DEBUG:boto:auth_path=/ibtljn-54/?prefix=c 2023-11-02T18:21:35.141 DEBUG:boto:Method: GET 2023-11-02T18:21:35.141 DEBUG:boto:Path: /ibtljn-54/?prefix=c 2023-11-02T18:21:35.142 DEBUG:boto:Data: 2023-11-02T18:21:35.142 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.143 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:35.143 DEBUG:boto:Port: 8000 2023-11-02T18:21:35.143 DEBUG:boto:Params: {} 2023-11-02T18:21:35.144 DEBUG:boto:Token: None 2023-11-02T18:21:35.144 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/ 2023-11-02T18:21:35.145 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE= 2023-11-02T18:21:35.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE=', 'Content-Length': '0'} 2023-11-02T18:21:35.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099866b2b29b4a618-006543e8af-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.151 DEBUG:boto:b'ibtljn-54c1000falsec2023-11-02T18:21:13.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:35.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:21:35.152 DEBUG:boto:path=/ibtljn-54/d 2023-11-02T18:21:35.152 DEBUG:boto:auth_path=/ibtljn-54/d 2023-11-02T18:21:35.153 DEBUG:boto:path=/ibtljn-54/d?versionId=null 2023-11-02T18:21:35.153 DEBUG:boto:auth_path=/ibtljn-54/d?versionId=null 2023-11-02T18:21:35.154 DEBUG:boto:Method: GET 2023-11-02T18:21:35.154 DEBUG:boto:Path: /ibtljn-54/d?versionId=null 2023-11-02T18:21:35.155 DEBUG:boto:Data: 2023-11-02T18:21:35.155 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.156 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:35.156 DEBUG:boto:Port: 8001 2023-11-02T18:21:35.156 DEBUG:boto:Params: {} 2023-11-02T18:21:35.157 DEBUG:boto:Token: None 2023-11-02T18:21:35.157 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/d?versionId=null 2023-11-02T18:21:35.158 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dXXzB0B0CrhsYnmD7gt0MK2waWc= 2023-11-02T18:21:35.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dXXzB0B0CrhsYnmD7gt0MK2waWc=', 'Content-Length': '0'} 2023-11-02T18:21:35.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9180034290e6c42-006543e8af-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.162 DEBUG:boto:path=/ibtljn-54/d 2023-11-02T18:21:35.162 DEBUG:boto:auth_path=/ibtljn-54/d 2023-11-02T18:21:35.163 DEBUG:boto:path=/ibtljn-54/d?versionId=null 2023-11-02T18:21:35.163 DEBUG:boto:auth_path=/ibtljn-54/d?versionId=null 2023-11-02T18:21:35.164 DEBUG:boto:Method: GET 2023-11-02T18:21:35.164 DEBUG:boto:Path: /ibtljn-54/d?versionId=null 2023-11-02T18:21:35.164 DEBUG:boto:Data: 2023-11-02T18:21:35.165 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.165 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:35.166 DEBUG:boto:Port: 8000 2023-11-02T18:21:35.166 DEBUG:boto:Params: {} 2023-11-02T18:21:35.166 DEBUG:boto:Token: None 2023-11-02T18:21:35.167 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/d?versionId=null 2023-11-02T18:21:35.167 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dXXzB0B0CrhsYnmD7gt0MK2waWc= 2023-11-02T18:21:35.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dXXzB0B0CrhsYnmD7gt0MK2waWc=', 'Content-Length': '0'} 2023-11-02T18:21:35.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee91caad10dee3da-006543e8af-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.171 DEBUG:boto:path=/ibtljn-54/d 2023-11-02T18:21:35.172 DEBUG:boto:auth_path=/ibtljn-54/d 2023-11-02T18:21:35.172 DEBUG:boto:path=/ibtljn-54/d?versionId=null 2023-11-02T18:21:35.173 DEBUG:boto:auth_path=/ibtljn-54/d?versionId=null 2023-11-02T18:21:35.173 DEBUG:boto:Method: HEAD 2023-11-02T18:21:35.174 DEBUG:boto:Path: /ibtljn-54/d?versionId=null 2023-11-02T18:21:35.174 DEBUG:boto:Data: 2023-11-02T18:21:35.174 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.175 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:35.175 DEBUG:boto:Port: 8001 2023-11-02T18:21:35.176 DEBUG:boto:Params: {} 2023-11-02T18:21:35.176 DEBUG:boto:Token: None 2023-11-02T18:21:35.177 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/d?versionId=null 2023-11-02T18:21:35.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j9INccTAKegaQofSmA4lUGVS8VM= 2023-11-02T18:21:35.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j9INccTAKegaQofSmA4lUGVS8VM=', 'Content-Length': '0'} 2023-11-02T18:21:35.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b69dfdcf228f1357-006543e8af-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.181 DEBUG:boto:path=/ibtljn-54/d 2023-11-02T18:21:35.181 DEBUG:boto:auth_path=/ibtljn-54/d 2023-11-02T18:21:35.181 DEBUG:boto:path=/ibtljn-54/d?versionId=null 2023-11-02T18:21:35.182 DEBUG:boto:auth_path=/ibtljn-54/d?versionId=null 2023-11-02T18:21:35.182 DEBUG:boto:Method: HEAD 2023-11-02T18:21:35.183 DEBUG:boto:Path: /ibtljn-54/d?versionId=null 2023-11-02T18:21:35.183 DEBUG:boto:Data: 2023-11-02T18:21:35.183 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.184 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:35.184 DEBUG:boto:Port: 8000 2023-11-02T18:21:35.184 DEBUG:boto:Params: {} 2023-11-02T18:21:35.185 DEBUG:boto:Token: None 2023-11-02T18:21:35.185 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/d?versionId=null 2023-11-02T18:21:35.186 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j9INccTAKegaQofSmA4lUGVS8VM= 2023-11-02T18:21:35.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j9INccTAKegaQofSmA4lUGVS8VM=', 'Content-Length': '0'} 2023-11-02T18:21:35.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1ece5f7882bd9eb-006543e8af-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:21:35.190 DEBUG:boto:path=/ibtljn-54/d 2023-11-02T18:21:35.190 DEBUG:boto:auth_path=/ibtljn-54/d 2023-11-02T18:21:35.191 DEBUG:boto:path=/ibtljn-54/d?versionId=null 2023-11-02T18:21:35.191 DEBUG:boto:auth_path=/ibtljn-54/d?versionId=null 2023-11-02T18:21:35.191 DEBUG:boto:Method: GET 2023-11-02T18:21:35.193 DEBUG:boto:Path: /ibtljn-54/d?versionId=null 2023-11-02T18:21:35.194 DEBUG:boto:Data: 2023-11-02T18:21:35.194 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.194 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:35.195 DEBUG:boto:Port: 8001 2023-11-02T18:21:35.195 DEBUG:boto:Params: {} 2023-11-02T18:21:35.196 DEBUG:boto:Token: None 2023-11-02T18:21:35.196 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/d?versionId=null 2023-11-02T18:21:35.197 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dXXzB0B0CrhsYnmD7gt0MK2waWc= 2023-11-02T18:21:35.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dXXzB0B0CrhsYnmD7gt0MK2waWc=', 'Content-Length': '0'} 2023-11-02T18:21:35.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031936584c18f1d98-006543e8af-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.201 DEBUG:boto:path=/ibtljn-54/d 2023-11-02T18:21:35.201 DEBUG:boto:auth_path=/ibtljn-54/d 2023-11-02T18:21:35.202 DEBUG:boto:path=/ibtljn-54/d?versionId=null 2023-11-02T18:21:35.202 DEBUG:boto:auth_path=/ibtljn-54/d?versionId=null 2023-11-02T18:21:35.203 DEBUG:boto:Method: GET 2023-11-02T18:21:35.203 DEBUG:boto:Path: /ibtljn-54/d?versionId=null 2023-11-02T18:21:35.203 DEBUG:boto:Data: 2023-11-02T18:21:35.204 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.204 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:35.204 DEBUG:boto:Port: 8000 2023-11-02T18:21:35.205 DEBUG:boto:Params: {} 2023-11-02T18:21:35.205 DEBUG:boto:Token: None 2023-11-02T18:21:35.206 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/d?versionId=null 2023-11-02T18:21:35.206 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dXXzB0B0CrhsYnmD7gt0MK2waWc= 2023-11-02T18:21:35.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dXXzB0B0CrhsYnmD7gt0MK2waWc=', 'Content-Length': '0'} 2023-11-02T18:21:35.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000713f1a8c86d54e84-006543e8af-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.250 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:35.251 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:35.251 DEBUG:boto:path=/ibtljn-54/?prefix=d 2023-11-02T18:21:35.252 DEBUG:boto:auth_path=/ibtljn-54/?prefix=d 2023-11-02T18:21:35.252 DEBUG:boto:Method: GET 2023-11-02T18:21:35.253 DEBUG:boto:Path: /ibtljn-54/?prefix=d 2023-11-02T18:21:35.253 DEBUG:boto:Data: 2023-11-02T18:21:35.253 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:35.254 DEBUG:boto:Port: 8001 2023-11-02T18:21:35.255 DEBUG:boto:Params: {} 2023-11-02T18:21:35.255 DEBUG:boto:Token: None 2023-11-02T18:21:35.256 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/ 2023-11-02T18:21:35.256 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE= 2023-11-02T18:21:35.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE=', 'Content-Length': '0'} 2023-11-02T18:21:35.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e23614471005ebb1-006543e8af-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.262 DEBUG:boto:b'ibtljn-54d1000falsed2023-11-02T18:21:13.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:35.263 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:35.263 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:35.264 DEBUG:boto:path=/ibtljn-54/?prefix=d 2023-11-02T18:21:35.264 DEBUG:boto:auth_path=/ibtljn-54/?prefix=d 2023-11-02T18:21:35.265 DEBUG:boto:Method: GET 2023-11-02T18:21:35.265 DEBUG:boto:Path: /ibtljn-54/?prefix=d 2023-11-02T18:21:35.265 DEBUG:boto:Data: 2023-11-02T18:21:35.266 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.266 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:35.267 DEBUG:boto:Port: 8000 2023-11-02T18:21:35.267 DEBUG:boto:Params: {} 2023-11-02T18:21:35.268 DEBUG:boto:Token: None 2023-11-02T18:21:35.269 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/ 2023-11-02T18:21:35.269 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE= 2023-11-02T18:21:35.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE=', 'Content-Length': '0'} 2023-11-02T18:21:35.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024fe5ad45a016685-006543e8af-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.276 DEBUG:boto:b'ibtljn-54d1000falsed2023-11-02T18:21:13.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:35.276 DEBUG:boto:path=/ibtljn-54/d 2023-11-02T18:21:35.277 DEBUG:boto:auth_path=/ibtljn-54/d 2023-11-02T18:21:35.277 DEBUG:boto:Method: HEAD 2023-11-02T18:21:35.277 DEBUG:boto:Path: /ibtljn-54/d 2023-11-02T18:21:35.278 DEBUG:boto:Data: 2023-11-02T18:21:35.278 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.279 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:35.279 DEBUG:boto:Port: 8001 2023-11-02T18:21:35.279 DEBUG:boto:Params: {} 2023-11-02T18:21:35.280 DEBUG:boto:Token: None 2023-11-02T18:21:35.280 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/d 2023-11-02T18:21:35.281 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YujJWUqnf9qLrsG0aqi2nr7DAKs= 2023-11-02T18:21:35.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YujJWUqnf9qLrsG0aqi2nr7DAKs=', 'Content-Length': '0'} 2023-11-02T18:21:35.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1b0691d02cbf304-006543e8af-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.285 DEBUG:boto:path=/ibtljn-54/d 2023-11-02T18:21:35.285 DEBUG:boto:auth_path=/ibtljn-54/d 2023-11-02T18:21:35.286 DEBUG:boto:Method: HEAD 2023-11-02T18:21:35.286 DEBUG:boto:Path: /ibtljn-54/d 2023-11-02T18:21:35.286 DEBUG:boto:Data: 2023-11-02T18:21:35.287 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.287 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:35.287 DEBUG:boto:Port: 8000 2023-11-02T18:21:35.288 DEBUG:boto:Params: {} 2023-11-02T18:21:35.288 DEBUG:boto:Token: None 2023-11-02T18:21:35.289 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/d 2023-11-02T18:21:35.289 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YujJWUqnf9qLrsG0aqi2nr7DAKs= 2023-11-02T18:21:35.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YujJWUqnf9qLrsG0aqi2nr7DAKs=', 'Content-Length': '0'} 2023-11-02T18:21:35.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041e9776505967c66-006543e8af-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:21:35.293 DEBUG:boto:path=/ibtljn-54/d 2023-11-02T18:21:35.294 DEBUG:boto:auth_path=/ibtljn-54/d 2023-11-02T18:21:35.294 DEBUG:boto:Method: GET 2023-11-02T18:21:35.295 DEBUG:boto:Path: /ibtljn-54/d 2023-11-02T18:21:35.295 DEBUG:boto:Data: 2023-11-02T18:21:35.296 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.296 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:35.297 DEBUG:boto:Port: 8001 2023-11-02T18:21:35.297 DEBUG:boto:Params: {} 2023-11-02T18:21:35.297 DEBUG:boto:Token: None 2023-11-02T18:21:35.298 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/d 2023-11-02T18:21:35.298 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LFo09L72Of2IFFQv2NlfQKrKpao= 2023-11-02T18:21:35.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LFo09L72Of2IFFQv2NlfQKrKpao=', 'Content-Length': '0'} 2023-11-02T18:21:35.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-54:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dd2ec33edd732dc-006543e8af-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.346 DEBUG:boto:path=/ibtljn-54/d 2023-11-02T18:21:35.347 DEBUG:boto:auth_path=/ibtljn-54/d 2023-11-02T18:21:35.347 DEBUG:boto:Method: GET 2023-11-02T18:21:35.347 DEBUG:boto:Path: /ibtljn-54/d 2023-11-02T18:21:35.348 DEBUG:boto:Data: 2023-11-02T18:21:35.348 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.349 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:35.349 DEBUG:boto:Port: 8000 2023-11-02T18:21:35.349 DEBUG:boto:Params: {} 2023-11-02T18:21:35.350 DEBUG:boto:Token: None 2023-11-02T18:21:35.350 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/d 2023-11-02T18:21:35.351 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LFo09L72Of2IFFQv2NlfQKrKpao= 2023-11-02T18:21:35.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LFo09L72Of2IFFQv2NlfQKrKpao=', 'Content-Length': '0'} 2023-11-02T18:21:35.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079ea3b419a09ac60-006543e8af-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.355 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:35.355 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:35.356 DEBUG:boto:path=/ibtljn-54/?prefix=d 2023-11-02T18:21:35.356 DEBUG:boto:auth_path=/ibtljn-54/?prefix=d 2023-11-02T18:21:35.356 DEBUG:boto:Method: GET 2023-11-02T18:21:35.357 DEBUG:boto:Path: /ibtljn-54/?prefix=d 2023-11-02T18:21:35.357 DEBUG:boto:Data: 2023-11-02T18:21:35.358 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:35.358 DEBUG:boto:Port: 8001 2023-11-02T18:21:35.359 DEBUG:boto:Params: {} 2023-11-02T18:21:35.359 DEBUG:boto:Token: None 2023-11-02T18:21:35.360 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/ 2023-11-02T18:21:35.360 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE= 2023-11-02T18:21:35.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE=', 'Content-Length': '0'} 2023-11-02T18:21:35.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8330ee7bf6685ba-006543e8af-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.366 DEBUG:boto:b'ibtljn-54d1000falsed2023-11-02T18:21:13.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:35.366 DEBUG:boto:path=/ibtljn-54/ 2023-11-02T18:21:35.367 DEBUG:boto:auth_path=/ibtljn-54/ 2023-11-02T18:21:35.367 DEBUG:boto:path=/ibtljn-54/?prefix=d 2023-11-02T18:21:35.368 DEBUG:boto:auth_path=/ibtljn-54/?prefix=d 2023-11-02T18:21:35.368 DEBUG:boto:Method: GET 2023-11-02T18:21:35.368 DEBUG:boto:Path: /ibtljn-54/?prefix=d 2023-11-02T18:21:35.369 DEBUG:boto:Data: 2023-11-02T18:21:35.369 DEBUG:boto:Headers: {} 2023-11-02T18:21:35.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:35.370 DEBUG:boto:Port: 8000 2023-11-02T18:21:35.370 DEBUG:boto:Params: {} 2023-11-02T18:21:35.371 DEBUG:boto:Token: None 2023-11-02T18:21:35.371 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:35 GMT /ibtljn-54/ 2023-11-02T18:21:35.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE= 2023-11-02T18:21:35.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lhZvc3MQ5EL5ao8igZ95JMPq0CE=', 'Content-Length': '0'} 2023-11-02T18:21:35.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e18a6f0f88dcc276-006543e8af-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:35.418 DEBUG:boto:b'ibtljn-54d1000falsed2023-11-02T18:21:13.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:35.419 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-54 zones={a2, a1} 2023-11-02T18:21:35.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:21:35.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:21:35.420 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:21:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:35.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ibtljn-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:35.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ibtljn-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:35.566 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ibtljn-54 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:21:35.610 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:35.610 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:35.692 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T18:21:35.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ibtljn-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:35.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ibtljn-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:35.696 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ibtljn-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:21:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:35.741 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:35.826 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:21:35.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:35.826 INFO:teuthology.orchestra.run.smithi017.stdout: "op_id": "1#00000000001.27.6", 2023-11-02T18:21:35.826 INFO:teuthology.orchestra.run.smithi017.stdout: "op_tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.6205761620264048052", 2023-11-02T18:21:35.827 INFO:teuthology.orchestra.run.smithi017.stdout: "op": "write", 2023-11-02T18:21:35.827 INFO:teuthology.orchestra.run.smithi017.stdout: "object": "a", 2023-11-02T18:21:35.827 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:21:35.827 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "complete", 2023-11-02T18:21:35.828 INFO:teuthology.orchestra.run.smithi017.stdout: "index_ver": 1, 2023-11-02T18:21:35.828 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:20:46.521619342Z", 2023-11-02T18:21:35.828 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:21:35.828 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:21:35.828 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:21:35.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:35.829 INFO:teuthology.orchestra.run.smithi017.stdout: "bilog_flags": 0, 2023-11-02T18:21:35.829 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned": false, 2023-11-02T18:21:35.829 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "", 2023-11-02T18:21:35.829 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "", 2023-11-02T18:21:35.830 INFO:teuthology.orchestra.run.smithi017.stdout: "zones_trace": [ 2023-11-02T18:21:35.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:35.830 INFO:teuthology.orchestra.run.smithi017.stdout: "entry": "d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18" 2023-11-02T18:21:35.830 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:35.830 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:21:35.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:35.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:35.831 INFO:teuthology.orchestra.run.smithi017.stdout: "op_id": "5#00000000001.21.6", 2023-11-02T18:21:35.831 INFO:teuthology.orchestra.run.smithi017.stdout: "op_tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.6909078773346890454", 2023-11-02T18:21:35.831 INFO:teuthology.orchestra.run.smithi017.stdout: "op": "write", 2023-11-02T18:21:35.832 INFO:teuthology.orchestra.run.smithi017.stdout: "object": "d", 2023-11-02T18:21:35.832 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:21:35.832 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "complete", 2023-11-02T18:21:35.832 INFO:teuthology.orchestra.run.smithi017.stdout: "index_ver": 1, 2023-11-02T18:21:35.832 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:20:46.634395271Z", 2023-11-02T18:21:35.833 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:21:35.833 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:21:35.833 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 41 2023-11-02T18:21:35.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:35.833 INFO:teuthology.orchestra.run.smithi017.stdout: "bilog_flags": 0, 2023-11-02T18:21:35.834 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned": false, 2023-11-02T18:21:35.834 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "", 2023-11-02T18:21:35.834 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "", 2023-11-02T18:21:35.834 INFO:teuthology.orchestra.run.smithi017.stdout: "zones_trace": [ 2023-11-02T18:21:35.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:35.835 INFO:teuthology.orchestra.run.smithi017.stdout: "entry": "d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18" 2023-11-02T18:21:35.835 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:35.835 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:21:35.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:35.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:35.836 INFO:teuthology.orchestra.run.smithi017.stdout: "op_id": "8#00000000001.52.6", 2023-11-02T18:21:35.836 INFO:teuthology.orchestra.run.smithi017.stdout: "op_tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.2106338687023736839", 2023-11-02T18:21:35.836 INFO:teuthology.orchestra.run.smithi017.stdout: "op": "write", 2023-11-02T18:21:35.837 INFO:teuthology.orchestra.run.smithi017.stdout: "object": "c", 2023-11-02T18:21:35.837 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:21:35.837 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "complete", 2023-11-02T18:21:35.837 INFO:teuthology.orchestra.run.smithi017.stdout: "index_ver": 1, 2023-11-02T18:21:35.837 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:20:46.610597586Z", 2023-11-02T18:21:35.838 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:21:35.838 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:21:35.838 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:21:35.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:35.838 INFO:teuthology.orchestra.run.smithi017.stdout: "bilog_flags": 0, 2023-11-02T18:21:35.839 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned": false, 2023-11-02T18:21:35.839 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "", 2023-11-02T18:21:35.839 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "", 2023-11-02T18:21:35.839 INFO:teuthology.orchestra.run.smithi017.stdout: "zones_trace": [ 2023-11-02T18:21:35.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:35.840 INFO:teuthology.orchestra.run.smithi017.stdout: "entry": "d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18" 2023-11-02T18:21:35.840 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:35.840 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:21:35.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:35.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:35.841 INFO:teuthology.orchestra.run.smithi017.stdout: "op_id": "10#00000000001.23.6", 2023-11-02T18:21:35.841 INFO:teuthology.orchestra.run.smithi017.stdout: "op_tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.1444353472279594786", 2023-11-02T18:21:35.841 INFO:teuthology.orchestra.run.smithi017.stdout: "op": "write", 2023-11-02T18:21:35.841 INFO:teuthology.orchestra.run.smithi017.stdout: "object": "b", 2023-11-02T18:21:35.842 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:21:35.842 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "complete", 2023-11-02T18:21:35.842 INFO:teuthology.orchestra.run.smithi017.stdout: "index_ver": 1, 2023-11-02T18:21:35.842 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:20:46.550724490Z", 2023-11-02T18:21:35.842 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:21:35.843 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:21:35.843 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:21:35.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:35.843 INFO:teuthology.orchestra.run.smithi017.stdout: "bilog_flags": 0, 2023-11-02T18:21:35.843 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned": false, 2023-11-02T18:21:35.844 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "", 2023-11-02T18:21:35.844 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "", 2023-11-02T18:21:35.844 INFO:teuthology.orchestra.run.smithi017.stdout: "zones_trace": [ 2023-11-02T18:21:35.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:35.844 INFO:teuthology.orchestra.run.smithi017.stdout: "entry": "d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-53:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.18" 2023-11-02T18:21:35.845 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:35.845 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:21:35.845 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:35.845 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:21:35.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:21:35.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:21:35.849 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:21:35.933 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:35.933 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:36.032 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.026+0000 7f6ceeae3700 -1 req 6516395771941645095 0.003999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-28:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.28,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-28:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.28,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.033 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.026+0000 7f6cf52f0700 -1 req 9553408923846570021 0.003999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-19:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.19,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-19:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.19,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.040 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.034+0000 7f6de54d0700 -1 req 3241953015713090749 0.003999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-27:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.27,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-27:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.27,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.045 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.042+0000 7f6cf0ae7700 -1 req 4657354041503607522 0.003999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-43:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.8,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-43:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.8,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.055 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.050+0000 7f6db7474700 -1 req 6444406752794168562 0.003999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-29:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.29,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-29:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.29,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.057 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.054+0000 7f6dbf484700 -1 req 5089370575308386019 0.003999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-3:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.3,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-3:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.3,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.058 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.054+0000 7f6dbd480700 -1 req 1981037095094573765 0.003999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-42:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.7,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-42:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.7,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.061 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.058+0000 7f6dd2cab700 -1 req 6542548915082561848 0.003999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-2:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.2,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-2:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.2,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.062 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.058+0000 7f6d69bd9700 -1 req 17103273822559494006 0.003999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-36:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.1,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-36:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.1,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.065 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.058+0000 7f6dbfc85700 -1 req 17410429878208209671 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-20:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.20,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-20:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.20,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.113 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.106+0000 7f6d9d440700 -1 req 7415835184184604324 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-41:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.6,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-41:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.6,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.117 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.110+0000 7f6d91c29700 -1 req 10726120996443698885 0.003999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-21:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.21,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-21:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.21,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.119 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.114+0000 7f6d7b3fc700 -1 req 17381362690769911061 0.003999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-1:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.1,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-1:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.1,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.120 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.114+0000 7f6d7f404700 -1 req 9629236907146600923 0.003999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-5:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.5,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-5:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.5,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.122 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.118+0000 7f6d51ba9700 -1 req 14666741386521025276 0.003999979s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-40:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.5,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-40:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.5,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.122 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.118+0000 7f6d7c3fe700 -1 req 8565312186649031314 0.003999979s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-4:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.4,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-4:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.4,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.124 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.118+0000 7f6d4e3a2700 -1 req 1327436151619555298 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-25:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.25,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-25:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.25,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.126 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.122+0000 7f6d85c11700 -1 req 15077820922640515248 0.003999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-22:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.22,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-22:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.22,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.200 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.194+0000 7f6d29b59700 -1 req 8293172938379014149 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-26:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.26,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-26:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.26,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.201 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.194+0000 7f6d31b69700 -1 req 4120418828273023715 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-24:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.24,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-24:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.24,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.204 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.198+0000 7f6d01b09700 -1 req 3030634160723918603 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-38:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.3,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-38:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.3,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.217 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.210+0000 7f6da7454700 -1 req 7071734799912992378 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-37:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.2,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-37:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.2,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.271 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.266+0000 7f6cf92f8700 -1 req 18179921561964150547 0.003999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-39:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.4,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-39:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.4,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.274 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.270+0000 7f6ddbcbd700 -1 req 17696086138580368230 0.003999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-23:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.23,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-23:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.23,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.279 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.274+0000 7f6dd6cb3700 -1 req 10433206238227474695 0.003999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-30:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.30,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-30:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.30,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.279 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:21:36.274+0000 7f6d19b39700 -1 req 13287773070992948905 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-51:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.16,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-51:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.16,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:21:36.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ibtljn-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:36.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ibtljn-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:36.356 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ibtljn-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:21:36.398 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:36.399 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:36.487 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T18:21:36.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2023-11-02T18:21:36.491 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-55 2023-11-02T18:21:36.492 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:36.492 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:36.493 DEBUG:boto:Method: PUT 2023-11-02T18:21:36.493 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:21:36.493 DEBUG:boto:Data: 2023-11-02T18:21:36.494 DEBUG:boto:Headers: {} 2023-11-02T18:21:36.494 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:36.495 DEBUG:boto:Port: 8000 2023-11-02T18:21:36.495 DEBUG:boto:Params: {} 2023-11-02T18:21:36.496 DEBUG:boto:Token: None 2023-11-02T18:21:36.496 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:21:36 GMT /ibtljn-55/ 2023-11-02T18:21:36.497 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KWQJTIzst26NURitiqXZOpgkHyY= 2023-11-02T18:21:36.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KWQJTIzst26NURitiqXZOpgkHyY=', 'Content-Length': '0'} 2023-11-02T18:21:36.526 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000010dfd14f68ce8270-006543e8b0-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:36.527 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:36.527 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:36.527 DEBUG:boto:Method: HEAD 2023-11-02T18:21:36.528 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:21:36.528 DEBUG:boto:Data: 2023-11-02T18:21:36.529 DEBUG:boto:Headers: {} 2023-11-02T18:21:36.529 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:36.530 DEBUG:boto:Port: 8000 2023-11-02T18:21:36.530 DEBUG:boto:Params: {} 2023-11-02T18:21:36.530 DEBUG:boto:Token: None 2023-11-02T18:21:36.531 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:36 GMT /ibtljn-55/ 2023-11-02T18:21:36.531 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ= 2023-11-02T18:21:36.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ=', 'Content-Length': '0'} 2023-11-02T18:21:36.535 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1bafb3ccea03e40-006543e8b0-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:36.536 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:21:36.536 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:21:36.537 DEBUG:boto:Method: PUT 2023-11-02T18:21:36.537 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:21:36.538 DEBUG:boto:Data: 2023-11-02T18:21:36.538 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:21:36.539 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:36.539 DEBUG:boto:Port: 8000 2023-11-02T18:21:36.540 DEBUG:boto:Params: {} 2023-11-02T18:21:36.540 DEBUG:boto:Token: None 2023-11-02T18:21:36.541 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:21:36 GMT /ibtljn-55/a 2023-11-02T18:21:36.541 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1QItATWRWZvkb1hG0ibo43SHNIg= 2023-11-02T18:21:36.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1QItATWRWZvkb1hG0ibo43SHNIg='} 2023-11-02T18:21:36.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b1f432976685606-006543e8b0-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:36.554 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:36.555 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:36.555 DEBUG:boto:Method: HEAD 2023-11-02T18:21:36.556 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:21:36.556 DEBUG:boto:Data: 2023-11-02T18:21:36.557 DEBUG:boto:Headers: {} 2023-11-02T18:21:36.557 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:36.558 DEBUG:boto:Port: 8000 2023-11-02T18:21:36.558 DEBUG:boto:Params: {} 2023-11-02T18:21:36.559 DEBUG:boto:Token: None 2023-11-02T18:21:36.559 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:36 GMT /ibtljn-55/ 2023-11-02T18:21:36.560 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ= 2023-11-02T18:21:36.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ=', 'Content-Length': '0'} 2023-11-02T18:21:36.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093a04cba339c0061-006543e8b0-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:36.564 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:21:36.564 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:21:36.565 DEBUG:boto:Method: PUT 2023-11-02T18:21:36.565 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:21:36.566 DEBUG:boto:Data: 2023-11-02T18:21:36.566 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:21:36.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:36.567 DEBUG:boto:Port: 8000 2023-11-02T18:21:36.567 DEBUG:boto:Params: {} 2023-11-02T18:21:36.568 DEBUG:boto:Token: None 2023-11-02T18:21:36.568 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:21:36 GMT /ibtljn-55/b 2023-11-02T18:21:36.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bl/F+FQVilDek52c8cL/fv2gxf0= 2023-11-02T18:21:36.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bl/F+FQVilDek52c8cL/fv2gxf0='} 2023-11-02T18:21:36.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc15342477d33249-006543e8b0-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:36.615 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:36.616 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:36.616 DEBUG:boto:Method: HEAD 2023-11-02T18:21:36.616 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:21:36.617 DEBUG:boto:Data: 2023-11-02T18:21:36.617 DEBUG:boto:Headers: {} 2023-11-02T18:21:36.618 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:36.619 DEBUG:boto:Port: 8000 2023-11-02T18:21:36.619 DEBUG:boto:Params: {} 2023-11-02T18:21:36.620 DEBUG:boto:Token: None 2023-11-02T18:21:36.620 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:36 GMT /ibtljn-55/ 2023-11-02T18:21:36.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ= 2023-11-02T18:21:36.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ=', 'Content-Length': '0'} 2023-11-02T18:21:36.625 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000043a39e20473a750-006543e8b0-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:36.625 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:21:36.626 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:21:36.626 DEBUG:boto:Method: PUT 2023-11-02T18:21:36.627 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:21:36.627 DEBUG:boto:Data: 2023-11-02T18:21:36.628 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:21:36.628 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:36.628 DEBUG:boto:Port: 8000 2023-11-02T18:21:36.629 DEBUG:boto:Params: {} 2023-11-02T18:21:36.629 DEBUG:boto:Token: None 2023-11-02T18:21:36.629 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:21:36 GMT /ibtljn-55/c 2023-11-02T18:21:36.630 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L97vg1G1C+q1aMO+gDPeoDs5M9Q= 2023-11-02T18:21:36.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L97vg1G1C+q1aMO+gDPeoDs5M9Q='} 2023-11-02T18:21:36.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7049e1fc442de2b-006543e8b0-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:36.641 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:36.642 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:36.642 DEBUG:boto:Method: HEAD 2023-11-02T18:21:36.643 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:21:36.643 DEBUG:boto:Data: 2023-11-02T18:21:36.644 DEBUG:boto:Headers: {} 2023-11-02T18:21:36.644 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:36.645 DEBUG:boto:Port: 8000 2023-11-02T18:21:36.645 DEBUG:boto:Params: {} 2023-11-02T18:21:36.646 DEBUG:boto:Token: None 2023-11-02T18:21:36.646 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:36 GMT /ibtljn-55/ 2023-11-02T18:21:36.647 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ= 2023-11-02T18:21:36.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ=', 'Content-Length': '0'} 2023-11-02T18:21:36.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d22d035ec9bffe89-006543e8b0-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:36.652 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:21:36.652 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:21:36.653 DEBUG:boto:Method: PUT 2023-11-02T18:21:36.653 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:21:36.654 DEBUG:boto:Data: 2023-11-02T18:21:36.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:21:36.655 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:36.655 DEBUG:boto:Port: 8000 2023-11-02T18:21:36.656 DEBUG:boto:Params: {} 2023-11-02T18:21:36.656 DEBUG:boto:Token: None 2023-11-02T18:21:36.657 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:21:36 GMT /ibtljn-55/d 2023-11-02T18:21:36.657 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dIWtvkavS8dtkiNvrYJyEW4D/ws= 2023-11-02T18:21:36.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dIWtvkavS8dtkiNvrYJyEW4D/ws='} 2023-11-02T18:21:36.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b126af4600eb7a7-006543e8b0-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:36.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:36.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:36.708 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:21:36.754 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:36.812 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:21:36.841 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:21:36.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:36.842 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:21:36.842 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:21:36.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:36.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:36.843 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:21:36.843 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:21:36.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:36.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:36.844 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949296.521130_419.1", 2023-11-02T18:21:36.844 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:21:36.521130Z" 2023-11-02T18:21:36.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:36.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:36.845 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:21:36.845 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:21:36.845 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:36.846 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:21:36.847 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949296.521130_419.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:21:36.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:21:36.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:36.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:36.849 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:21:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:36.953 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:36.946+0000 7f79c7c90f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:21:36.955 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:36.950+0000 7f79c7c90f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:21:36.982 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:21:36.982 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:21:36.982 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:21:36.982 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:21:36.983 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:21:36.983 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:21:36.983 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:21:36.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:36.984 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:21:36.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:36.984 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:21:36.984 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:21:36.985 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:21:36.985 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:21:36.985 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:21:36.985 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:21:36.985 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:21:36.986 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:21:36.986 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:21:36.986 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:36.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:36.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:36.987 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:21:36.987 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:21:36.987 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:21:36.988 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:21:36.988 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:21:36.988 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:21:36.988 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:21:36.988 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:21:36.989 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:21:36.989 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:36.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:36.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:36.990 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:21:36.990 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:21:36.990 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:21:36.990 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949296.521130_419.1", 2023-11-02T18:21:36.990 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:21:36.991 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:21:36.991 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:21:36.991 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:21:36.521130Z", 2023-11-02T18:21:36.991 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:21:36.992 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:36.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:36.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:36.992 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:21:36.992 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:21:36.993 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:21:36.993 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:21:36.993 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:21:36.993 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:21:36.993 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:21:36.994 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:21:36.994 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:21:36.994 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:36.994 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:36.995 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:21:36.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:36.995 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:21:36.995 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:21:36.995 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:21:36.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:36.996 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:21:36Z" 2023-11-02T18:21:36.996 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:21:36.997 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949296.521130_419.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:21:36.521130Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:21:36Z" } 2023-11-02T18:21:36.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949296.521130_419.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:21:36.521130Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:21:36.998 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949296.521130_419.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:21:36.999 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949296.521130_419.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:21:36.999 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:21:36.999 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-55 2023-11-02T18:21:37.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:21:37.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:21:37.001 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:21:37.054 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:37.055 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:37.114+0000 7f8931096f40 1 waiting to reach incremental sync.. 2023-11-02T18:21:42.120 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:42.113+0000 7f8931096f40 1 waiting to reach incremental sync.. 2023-11-02T18:21:47.124 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:47.117+0000 7f8931096f40 1 bucket sync caught up with source: 2023-11-02T18:21:47.124 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.24.6, , , , 00000000001.66.6, , , 00000000001.64.6, , 00000000001.29.6] 2023-11-02T18:21:47.124 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.24.6, , , , 00000000001.66.6, , , 00000000001.64.6, , 00000000001.29.6] 2023-11-02T18:21:47.125 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:47.117+0000 7f8931096f40 0 bucket checkpoint complete 2023-11-02T18:21:47.154 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-55 2023-11-02T18:21:47.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:21:47.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:21:47.156 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:21:47.207 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:47.207 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:47.279 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:47.273+0000 7fdf2619cf40 1 waiting to reach incremental sync.. 2023-11-02T18:21:52.280 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:52.273+0000 7fdf2619cf40 1 waiting to reach incremental sync.. 2023-11-02T18:21:57.283 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:57.277+0000 7fdf2619cf40 1 bucket sync caught up with source: 2023-11-02T18:21:57.284 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.174.6, , , , 00000000001.191.6, , , 00000000001.192.6, , 00000000001.177.6] 2023-11-02T18:21:57.284 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.174.6, , , , 00000000001.191.6, , , 00000000001.192.6, , 00000000001.177.6] 2023-11-02T18:21:57.284 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:57.277+0000 7fdf2619cf40 0 bucket checkpoint complete 2023-11-02T18:21:57.314 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-55 zones={a2, a1} 2023-11-02T18:21:57.315 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.315 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.316 DEBUG:boto:Method: HEAD 2023-11-02T18:21:57.316 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:21:57.317 DEBUG:boto:Data: 2023-11-02T18:21:57.317 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.318 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.318 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.318 DEBUG:boto:Params: {} 2023-11-02T18:21:57.319 DEBUG:boto:Token: None 2023-11-02T18:21:57.319 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/ 2023-11-02T18:21:57.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W/QoP5vu5m/Qh/K7By+5WmKLIeQ= 2023-11-02T18:21:57.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W/QoP5vu5m/Qh/K7By+5WmKLIeQ=', 'Content-Length': '0'} 2023-11-02T18:21:57.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b283ad7bfbd95e24-006543e8c5-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.325 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.325 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.326 DEBUG:boto:Method: HEAD 2023-11-02T18:21:57.326 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:21:57.327 DEBUG:boto:Data: 2023-11-02T18:21:57.327 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.328 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.328 DEBUG:boto:Params: {} 2023-11-02T18:21:57.329 DEBUG:boto:Token: None 2023-11-02T18:21:57.329 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/ 2023-11-02T18:21:57.330 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W/QoP5vu5m/Qh/K7By+5WmKLIeQ= 2023-11-02T18:21:57.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W/QoP5vu5m/Qh/K7By+5WmKLIeQ=', 'Content-Length': '0'} 2023-11-02T18:21:57.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6040f3ff0fd9483-006543e8c5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.335 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:21:57.335 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.336 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.336 DEBUG:boto:path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:21:57.336 DEBUG:boto:auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:21:57.337 DEBUG:boto:Method: GET 2023-11-02T18:21:57.337 DEBUG:boto:Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T18:21:57.338 DEBUG:boto:Data: 2023-11-02T18:21:57.338 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.338 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.339 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.339 DEBUG:boto:Params: {} 2023-11-02T18:21:57.340 DEBUG:boto:Token: None 2023-11-02T18:21:57.340 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/?versions 2023-11-02T18:21:57.341 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ= 2023-11-02T18:21:57.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ=', 'Content-Length': '0'} 2023-11-02T18:21:57.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009da9aca866688f78-006543e8c5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.346 DEBUG:boto:b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:57.347 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:21:57.347 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:21:57.348 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:21:57.348 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:21:57.349 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:21:57.349 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.350 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.350 DEBUG:boto:path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:21:57.350 DEBUG:boto:auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:21:57.351 DEBUG:boto:Method: GET 2023-11-02T18:21:57.351 DEBUG:boto:Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T18:21:57.352 DEBUG:boto:Data: 2023-11-02T18:21:57.352 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.353 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.353 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.353 DEBUG:boto:Params: {} 2023-11-02T18:21:57.354 DEBUG:boto:Token: None 2023-11-02T18:21:57.354 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/?versions 2023-11-02T18:21:57.355 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ= 2023-11-02T18:21:57.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ=', 'Content-Length': '0'} 2023-11-02T18:21:57.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1cec51f94a6ccc4-006543e8c5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.361 DEBUG:boto:b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:57.361 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:21:57.362 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:21:57.362 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:21:57.363 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:21:57.363 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.364 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.364 DEBUG:boto:path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:21:57.364 DEBUG:boto:auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:21:57.365 DEBUG:boto:Method: GET 2023-11-02T18:21:57.365 DEBUG:boto:Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T18:21:57.366 DEBUG:boto:Data: 2023-11-02T18:21:57.366 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.367 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.367 DEBUG:boto:Params: {} 2023-11-02T18:21:57.368 DEBUG:boto:Token: None 2023-11-02T18:21:57.368 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/?versions 2023-11-02T18:21:57.369 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ= 2023-11-02T18:21:57.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ=', 'Content-Length': '0'} 2023-11-02T18:21:57.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003802765ff34a147f-006543e8c5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.374 DEBUG:boto:b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:57.375 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.376 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.376 DEBUG:boto:path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:21:57.376 DEBUG:boto:auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:21:57.377 DEBUG:boto:Method: GET 2023-11-02T18:21:57.377 DEBUG:boto:Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T18:21:57.378 DEBUG:boto:Data: 2023-11-02T18:21:57.378 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.378 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.379 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.379 DEBUG:boto:Params: {} 2023-11-02T18:21:57.380 DEBUG:boto:Token: None 2023-11-02T18:21:57.380 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/?versions 2023-11-02T18:21:57.381 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ= 2023-11-02T18:21:57.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ=', 'Content-Length': '0'} 2023-11-02T18:21:57.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003685fddc53a3f276-006543e8c5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.387 DEBUG:boto:b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:57.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:21:57.388 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:21:57.388 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:21:57.389 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:21:57.389 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:21:57.390 DEBUG:boto:Method: GET 2023-11-02T18:21:57.390 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:21:57.390 DEBUG:boto:Data: 2023-11-02T18:21:57.391 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.391 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.392 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.392 DEBUG:boto:Params: {} 2023-11-02T18:21:57.393 DEBUG:boto:Token: None 2023-11-02T18:21:57.393 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:21:57.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c= 2023-11-02T18:21:57.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c=', 'Content-Length': '0'} 2023-11-02T18:21:57.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6fcc98f203a2f67-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.438 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:21:57.439 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:21:57.439 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:21:57.440 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:21:57.440 DEBUG:boto:Method: GET 2023-11-02T18:21:57.440 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:21:57.441 DEBUG:boto:Data: 2023-11-02T18:21:57.441 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.442 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.442 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.442 DEBUG:boto:Params: {} 2023-11-02T18:21:57.443 DEBUG:boto:Token: None 2023-11-02T18:21:57.443 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:21:57.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c= 2023-11-02T18:21:57.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c=', 'Content-Length': '0'} 2023-11-02T18:21:57.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad2439f91625ebc3-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.448 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:21:57.449 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:21:57.449 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:21:57.449 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:21:57.450 DEBUG:boto:Method: HEAD 2023-11-02T18:21:57.450 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:21:57.451 DEBUG:boto:Data: 2023-11-02T18:21:57.451 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.452 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.452 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.452 DEBUG:boto:Params: {} 2023-11-02T18:21:57.453 DEBUG:boto:Token: None 2023-11-02T18:21:57.453 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:21:57.454 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Wg99zl2nwWpDdhaQWcBGN4dXL3o= 2023-11-02T18:21:57.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Wg99zl2nwWpDdhaQWcBGN4dXL3o=', 'Content-Length': '0'} 2023-11-02T18:21:57.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f27d75bd4b87cc25-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.458 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:21:57.459 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:21:57.459 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:21:57.459 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:21:57.460 DEBUG:boto:Method: HEAD 2023-11-02T18:21:57.460 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:21:57.461 DEBUG:boto:Data: 2023-11-02T18:21:57.461 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.461 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.462 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.462 DEBUG:boto:Params: {} 2023-11-02T18:21:57.463 DEBUG:boto:Token: None 2023-11-02T18:21:57.463 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:21:57.464 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Wg99zl2nwWpDdhaQWcBGN4dXL3o= 2023-11-02T18:21:57.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Wg99zl2nwWpDdhaQWcBGN4dXL3o=', 'Content-Length': '0'} 2023-11-02T18:21:57.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011de2edbdc347392-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:21:57.470 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:21:57.471 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:21:57.471 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:21:57.471 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:21:57.472 DEBUG:boto:Method: GET 2023-11-02T18:21:57.472 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:21:57.473 DEBUG:boto:Data: 2023-11-02T18:21:57.473 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.473 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.474 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.474 DEBUG:boto:Params: {} 2023-11-02T18:21:57.475 DEBUG:boto:Token: None 2023-11-02T18:21:57.475 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:21:57.476 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c= 2023-11-02T18:21:57.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c=', 'Content-Length': '0'} 2023-11-02T18:21:57.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007467a1312d726ace-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.480 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:21:57.480 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:21:57.481 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:21:57.481 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:21:57.482 DEBUG:boto:Method: GET 2023-11-02T18:21:57.482 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:21:57.482 DEBUG:boto:Data: 2023-11-02T18:21:57.483 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.483 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.484 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.484 DEBUG:boto:Params: {} 2023-11-02T18:21:57.485 DEBUG:boto:Token: None 2023-11-02T18:21:57.485 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:21:57.485 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c= 2023-11-02T18:21:57.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c=', 'Content-Length': '0'} 2023-11-02T18:21:57.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ce636c2e8925f21-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.530 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.531 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.531 DEBUG:boto:path=/ibtljn-55/?prefix=a 2023-11-02T18:21:57.532 DEBUG:boto:auth_path=/ibtljn-55/?prefix=a 2023-11-02T18:21:57.532 DEBUG:boto:Method: GET 2023-11-02T18:21:57.532 DEBUG:boto:Path: /ibtljn-55/?prefix=a 2023-11-02T18:21:57.533 DEBUG:boto:Data: 2023-11-02T18:21:57.533 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.534 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.534 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.534 DEBUG:boto:Params: {} 2023-11-02T18:21:57.535 DEBUG:boto:Token: None 2023-11-02T18:21:57.535 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/ 2023-11-02T18:21:57.536 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T18:21:57.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T18:21:57.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cdd8172d145f58f-006543e8c5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.542 DEBUG:boto:b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:57.543 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.543 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.544 DEBUG:boto:path=/ibtljn-55/?prefix=a 2023-11-02T18:21:57.544 DEBUG:boto:auth_path=/ibtljn-55/?prefix=a 2023-11-02T18:21:57.544 DEBUG:boto:Method: GET 2023-11-02T18:21:57.545 DEBUG:boto:Path: /ibtljn-55/?prefix=a 2023-11-02T18:21:57.545 DEBUG:boto:Data: 2023-11-02T18:21:57.546 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.546 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.546 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.547 DEBUG:boto:Params: {} 2023-11-02T18:21:57.547 DEBUG:boto:Token: None 2023-11-02T18:21:57.548 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/ 2023-11-02T18:21:57.548 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T18:21:57.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T18:21:57.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003a80e7f0dff731c-006543e8c5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.554 DEBUG:boto:b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:57.555 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:21:57.555 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:21:57.556 DEBUG:boto:Method: HEAD 2023-11-02T18:21:57.556 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:21:57.557 DEBUG:boto:Data: 2023-11-02T18:21:57.557 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.557 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.558 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.558 DEBUG:boto:Params: {} 2023-11-02T18:21:57.559 DEBUG:boto:Token: None 2023-11-02T18:21:57.559 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/a 2023-11-02T18:21:57.560 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9y0zeuFgq1MDsb1o4x5CNo1im60= 2023-11-02T18:21:57.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9y0zeuFgq1MDsb1o4x5CNo1im60=', 'Content-Length': '0'} 2023-11-02T18:21:57.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f81135958368ce4d-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.563 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:21:57.564 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:21:57.564 DEBUG:boto:Method: HEAD 2023-11-02T18:21:57.564 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:21:57.565 DEBUG:boto:Data: 2023-11-02T18:21:57.565 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.566 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.567 DEBUG:boto:Params: {} 2023-11-02T18:21:57.567 DEBUG:boto:Token: None 2023-11-02T18:21:57.567 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/a 2023-11-02T18:21:57.568 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9y0zeuFgq1MDsb1o4x5CNo1im60= 2023-11-02T18:21:57.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9y0zeuFgq1MDsb1o4x5CNo1im60=', 'Content-Length': '0'} 2023-11-02T18:21:57.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d66b97c909dfb89-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:21:57.571 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:21:57.572 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:21:57.572 DEBUG:boto:Method: GET 2023-11-02T18:21:57.573 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:21:57.573 DEBUG:boto:Data: 2023-11-02T18:21:57.573 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.574 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.575 DEBUG:boto:Params: {} 2023-11-02T18:21:57.575 DEBUG:boto:Token: None 2023-11-02T18:21:57.576 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/a 2023-11-02T18:21:57.576 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:h5x1IbW/feyb8fxMOP1cDOk6pU0= 2023-11-02T18:21:57.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h5x1IbW/feyb8fxMOP1cDOk6pU0=', 'Content-Length': '0'} 2023-11-02T18:21:57.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029ad2ce82a0a611b-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.622 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:21:57.623 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:21:57.623 DEBUG:boto:Method: GET 2023-11-02T18:21:57.624 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:21:57.624 DEBUG:boto:Data: 2023-11-02T18:21:57.624 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.625 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.625 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.626 DEBUG:boto:Params: {} 2023-11-02T18:21:57.626 DEBUG:boto:Token: None 2023-11-02T18:21:57.627 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/a 2023-11-02T18:21:57.627 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:h5x1IbW/feyb8fxMOP1cDOk6pU0= 2023-11-02T18:21:57.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h5x1IbW/feyb8fxMOP1cDOk6pU0=', 'Content-Length': '0'} 2023-11-02T18:21:57.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd73cc3645153ec-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.632 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.632 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.633 DEBUG:boto:path=/ibtljn-55/?prefix=a 2023-11-02T18:21:57.633 DEBUG:boto:auth_path=/ibtljn-55/?prefix=a 2023-11-02T18:21:57.633 DEBUG:boto:Method: GET 2023-11-02T18:21:57.634 DEBUG:boto:Path: /ibtljn-55/?prefix=a 2023-11-02T18:21:57.634 DEBUG:boto:Data: 2023-11-02T18:21:57.635 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.635 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.636 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.636 DEBUG:boto:Params: {} 2023-11-02T18:21:57.637 DEBUG:boto:Token: None 2023-11-02T18:21:57.637 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/ 2023-11-02T18:21:57.637 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T18:21:57.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T18:21:57.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033bbbab684d0149d-006543e8c5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.644 DEBUG:boto:b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:57.645 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.645 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.646 DEBUG:boto:path=/ibtljn-55/?prefix=a 2023-11-02T18:21:57.646 DEBUG:boto:auth_path=/ibtljn-55/?prefix=a 2023-11-02T18:21:57.647 DEBUG:boto:Method: GET 2023-11-02T18:21:57.647 DEBUG:boto:Path: /ibtljn-55/?prefix=a 2023-11-02T18:21:57.648 DEBUG:boto:Data: 2023-11-02T18:21:57.648 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.648 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.649 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.649 DEBUG:boto:Params: {} 2023-11-02T18:21:57.650 DEBUG:boto:Token: None 2023-11-02T18:21:57.650 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/ 2023-11-02T18:21:57.651 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T18:21:57.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T18:21:57.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e7217578f02e40-006543e8c5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.698 DEBUG:boto:b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:57.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:21:57.699 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:21:57.700 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:21:57.700 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:21:57.701 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:21:57.701 DEBUG:boto:Method: GET 2023-11-02T18:21:57.701 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:21:57.702 DEBUG:boto:Data: 2023-11-02T18:21:57.702 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.703 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.703 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.703 DEBUG:boto:Params: {} 2023-11-02T18:21:57.704 DEBUG:boto:Token: None 2023-11-02T18:21:57.704 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:21:57.705 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o= 2023-11-02T18:21:57.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o=', 'Content-Length': '0'} 2023-11-02T18:21:57.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c03952e8e602465d-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.709 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:21:57.709 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:21:57.710 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:21:57.710 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:21:57.711 DEBUG:boto:Method: GET 2023-11-02T18:21:57.711 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:21:57.712 DEBUG:boto:Data: 2023-11-02T18:21:57.712 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.712 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.713 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.713 DEBUG:boto:Params: {} 2023-11-02T18:21:57.714 DEBUG:boto:Token: None 2023-11-02T18:21:57.714 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:21:57.715 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o= 2023-11-02T18:21:57.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o=', 'Content-Length': '0'} 2023-11-02T18:21:57.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032260631f42a1730-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.719 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:21:57.719 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:21:57.720 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:21:57.720 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:21:57.720 DEBUG:boto:Method: HEAD 2023-11-02T18:21:57.721 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:21:57.721 DEBUG:boto:Data: 2023-11-02T18:21:57.722 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.722 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.722 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.723 DEBUG:boto:Params: {} 2023-11-02T18:21:57.723 DEBUG:boto:Token: None 2023-11-02T18:21:57.724 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:21:57.724 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ytciB+kcP/pVcP/BeTULzcqkakQ= 2023-11-02T18:21:57.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ytciB+kcP/pVcP/BeTULzcqkakQ=', 'Content-Length': '0'} 2023-11-02T18:21:57.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0ae4e70e4899425-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.728 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:21:57.728 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:21:57.729 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:21:57.729 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:21:57.730 DEBUG:boto:Method: HEAD 2023-11-02T18:21:57.730 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:21:57.730 DEBUG:boto:Data: 2023-11-02T18:21:57.731 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.731 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.732 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.732 DEBUG:boto:Params: {} 2023-11-02T18:21:57.732 DEBUG:boto:Token: None 2023-11-02T18:21:57.733 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:21:57.733 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ytciB+kcP/pVcP/BeTULzcqkakQ= 2023-11-02T18:21:57.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ytciB+kcP/pVcP/BeTULzcqkakQ=', 'Content-Length': '0'} 2023-11-02T18:21:57.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e0224a3055ec304-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:21:57.738 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:21:57.738 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:21:57.739 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:21:57.739 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:21:57.740 DEBUG:boto:Method: GET 2023-11-02T18:21:57.740 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:21:57.741 DEBUG:boto:Data: 2023-11-02T18:21:57.741 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.741 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.742 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.742 DEBUG:boto:Params: {} 2023-11-02T18:21:57.743 DEBUG:boto:Token: None 2023-11-02T18:21:57.743 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:21:57.744 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o= 2023-11-02T18:21:57.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o=', 'Content-Length': '0'} 2023-11-02T18:21:57.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003af9adce501da8fb-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.790 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:21:57.791 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:21:57.791 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:21:57.792 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:21:57.792 DEBUG:boto:Method: GET 2023-11-02T18:21:57.792 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:21:57.793 DEBUG:boto:Data: 2023-11-02T18:21:57.793 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.794 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.794 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.795 DEBUG:boto:Params: {} 2023-11-02T18:21:57.795 DEBUG:boto:Token: None 2023-11-02T18:21:57.795 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:21:57.796 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o= 2023-11-02T18:21:57.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o=', 'Content-Length': '0'} 2023-11-02T18:21:57.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cd7322874e1a5a0-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.800 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.801 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.801 DEBUG:boto:path=/ibtljn-55/?prefix=b 2023-11-02T18:21:57.802 DEBUG:boto:auth_path=/ibtljn-55/?prefix=b 2023-11-02T18:21:57.802 DEBUG:boto:Method: GET 2023-11-02T18:21:57.803 DEBUG:boto:Path: /ibtljn-55/?prefix=b 2023-11-02T18:21:57.803 DEBUG:boto:Data: 2023-11-02T18:21:57.803 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.804 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.804 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.804 DEBUG:boto:Params: {} 2023-11-02T18:21:57.805 DEBUG:boto:Token: None 2023-11-02T18:21:57.805 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/ 2023-11-02T18:21:57.806 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T18:21:57.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T18:21:57.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000145314a7e396470f-006543e8c5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.812 DEBUG:boto:b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:57.813 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.813 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.813 DEBUG:boto:path=/ibtljn-55/?prefix=b 2023-11-02T18:21:57.814 DEBUG:boto:auth_path=/ibtljn-55/?prefix=b 2023-11-02T18:21:57.814 DEBUG:boto:Method: GET 2023-11-02T18:21:57.815 DEBUG:boto:Path: /ibtljn-55/?prefix=b 2023-11-02T18:21:57.815 DEBUG:boto:Data: 2023-11-02T18:21:57.815 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.816 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.816 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.817 DEBUG:boto:Params: {} 2023-11-02T18:21:57.817 DEBUG:boto:Token: None 2023-11-02T18:21:57.817 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/ 2023-11-02T18:21:57.818 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T18:21:57.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T18:21:57.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000075002ef0ea16f3c-006543e8c5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.866 DEBUG:boto:b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:57.867 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:21:57.867 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:21:57.868 DEBUG:boto:Method: HEAD 2023-11-02T18:21:57.868 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:21:57.868 DEBUG:boto:Data: 2023-11-02T18:21:57.869 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.869 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.870 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.870 DEBUG:boto:Params: {} 2023-11-02T18:21:57.870 DEBUG:boto:Token: None 2023-11-02T18:21:57.871 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/b 2023-11-02T18:21:57.871 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9XeMFcPloEhJFcWD0fvX393mT2w= 2023-11-02T18:21:57.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9XeMFcPloEhJFcWD0fvX393mT2w=', 'Content-Length': '0'} 2023-11-02T18:21:57.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7cfdeec812c3f98-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.877 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:21:57.877 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:21:57.878 DEBUG:boto:Method: HEAD 2023-11-02T18:21:57.878 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:21:57.878 DEBUG:boto:Data: 2023-11-02T18:21:57.879 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.879 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.880 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.880 DEBUG:boto:Params: {} 2023-11-02T18:21:57.881 DEBUG:boto:Token: None 2023-11-02T18:21:57.881 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/b 2023-11-02T18:21:57.881 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9XeMFcPloEhJFcWD0fvX393mT2w= 2023-11-02T18:21:57.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9XeMFcPloEhJFcWD0fvX393mT2w=', 'Content-Length': '0'} 2023-11-02T18:21:57.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea0a5efc759e6cb0-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:21:57.886 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:21:57.886 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:21:57.887 DEBUG:boto:Method: GET 2023-11-02T18:21:57.887 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:21:57.888 DEBUG:boto:Data: 2023-11-02T18:21:57.888 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.888 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.889 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.889 DEBUG:boto:Params: {} 2023-11-02T18:21:57.890 DEBUG:boto:Token: None 2023-11-02T18:21:57.890 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/b 2023-11-02T18:21:57.890 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uzfMXCHJklvTufQCZliYSSqD+Ao= 2023-11-02T18:21:57.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uzfMXCHJklvTufQCZliYSSqD+Ao=', 'Content-Length': '0'} 2023-11-02T18:21:57.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e160489ceb00cc0c-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.895 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:21:57.895 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:21:57.896 DEBUG:boto:Method: GET 2023-11-02T18:21:57.896 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:21:57.896 DEBUG:boto:Data: 2023-11-02T18:21:57.897 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.897 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.898 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.898 DEBUG:boto:Params: {} 2023-11-02T18:21:57.899 DEBUG:boto:Token: None 2023-11-02T18:21:57.899 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/b 2023-11-02T18:21:57.899 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uzfMXCHJklvTufQCZliYSSqD+Ao= 2023-11-02T18:21:57.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uzfMXCHJklvTufQCZliYSSqD+Ao=', 'Content-Length': '0'} 2023-11-02T18:21:57.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000400fc79297b2bbb-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.903 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.904 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.904 DEBUG:boto:path=/ibtljn-55/?prefix=b 2023-11-02T18:21:57.905 DEBUG:boto:auth_path=/ibtljn-55/?prefix=b 2023-11-02T18:21:57.905 DEBUG:boto:Method: GET 2023-11-02T18:21:57.906 DEBUG:boto:Path: /ibtljn-55/?prefix=b 2023-11-02T18:21:57.906 DEBUG:boto:Data: 2023-11-02T18:21:57.906 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.907 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.907 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.908 DEBUG:boto:Params: {} 2023-11-02T18:21:57.908 DEBUG:boto:Token: None 2023-11-02T18:21:57.909 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/ 2023-11-02T18:21:57.909 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T18:21:57.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T18:21:57.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d4a6998ab8c4155-006543e8c5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.915 DEBUG:boto:b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:57.916 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:57.917 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:57.917 DEBUG:boto:path=/ibtljn-55/?prefix=b 2023-11-02T18:21:57.917 DEBUG:boto:auth_path=/ibtljn-55/?prefix=b 2023-11-02T18:21:57.918 DEBUG:boto:Method: GET 2023-11-02T18:21:57.918 DEBUG:boto:Path: /ibtljn-55/?prefix=b 2023-11-02T18:21:57.919 DEBUG:boto:Data: 2023-11-02T18:21:57.919 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.919 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.920 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.920 DEBUG:boto:Params: {} 2023-11-02T18:21:57.921 DEBUG:boto:Token: None 2023-11-02T18:21:57.921 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/ 2023-11-02T18:21:57.921 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T18:21:57.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T18:21:57.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd438608631e7afe-006543e8c5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.927 DEBUG:boto:b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:57.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:21:57.928 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:21:57.929 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:21:57.929 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:21:57.930 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:21:57.930 DEBUG:boto:Method: GET 2023-11-02T18:21:57.930 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:21:57.931 DEBUG:boto:Data: 2023-11-02T18:21:57.931 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.932 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.932 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.933 DEBUG:boto:Params: {} 2023-11-02T18:21:57.933 DEBUG:boto:Token: None 2023-11-02T18:21:57.934 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:21:57.934 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gDEJYpwC9/itU80hbyQpG80smEA= 2023-11-02T18:21:57.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gDEJYpwC9/itU80hbyQpG80smEA=', 'Content-Length': '0'} 2023-11-02T18:21:57.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018366b08d880e616-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.978 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:21:57.979 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:21:57.979 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:21:57.980 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:21:57.980 DEBUG:boto:Method: GET 2023-11-02T18:21:57.980 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:21:57.981 DEBUG:boto:Data: 2023-11-02T18:21:57.981 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.982 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:57.982 DEBUG:boto:Port: 8000 2023-11-02T18:21:57.983 DEBUG:boto:Params: {} 2023-11-02T18:21:57.983 DEBUG:boto:Token: None 2023-11-02T18:21:57.984 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:21:57.984 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gDEJYpwC9/itU80hbyQpG80smEA= 2023-11-02T18:21:57.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gDEJYpwC9/itU80hbyQpG80smEA=', 'Content-Length': '0'} 2023-11-02T18:21:57.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b57881d11f4640be-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.988 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:21:57.988 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:21:57.989 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:21:57.989 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:21:57.989 DEBUG:boto:Method: HEAD 2023-11-02T18:21:57.990 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:21:57.990 DEBUG:boto:Data: 2023-11-02T18:21:57.991 DEBUG:boto:Headers: {} 2023-11-02T18:21:57.991 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:57.991 DEBUG:boto:Port: 8001 2023-11-02T18:21:57.992 DEBUG:boto:Params: {} 2023-11-02T18:21:57.992 DEBUG:boto:Token: None 2023-11-02T18:21:57.993 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:57 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:21:57.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:myD3z9bFnmGf6OdgjKOAvqQL7sY= 2023-11-02T18:21:57.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:myD3z9bFnmGf6OdgjKOAvqQL7sY=', 'Content-Length': '0'} 2023-11-02T18:21:57.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000895508fbb62fa39b-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:57.997 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:21:57.998 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:21:57.998 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:21:57.998 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:21:57.999 DEBUG:boto:Method: HEAD 2023-11-02T18:21:57.999 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:21:58.000 DEBUG:boto:Data: 2023-11-02T18:21:58.000 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.000 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.001 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.001 DEBUG:boto:Params: {} 2023-11-02T18:21:58.001 DEBUG:boto:Token: None 2023-11-02T18:21:58.002 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:21:58.002 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Lrme/QddLhJqTjUGwd0F8VVNAo0= 2023-11-02T18:21:58.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Lrme/QddLhJqTjUGwd0F8VVNAo0=', 'Content-Length': '0'} 2023-11-02T18:21:58.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ac829582b05a0cc-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:21:58.007 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:21:58.007 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:21:58.007 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:21:58.008 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:21:58.008 DEBUG:boto:Method: GET 2023-11-02T18:21:58.009 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:21:58.009 DEBUG:boto:Data: 2023-11-02T18:21:58.009 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.010 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:58.010 DEBUG:boto:Port: 8001 2023-11-02T18:21:58.011 DEBUG:boto:Params: {} 2023-11-02T18:21:58.011 DEBUG:boto:Token: None 2023-11-02T18:21:58.011 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:21:58.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gb27Q640zqifRsOcMCkQJgObeMw= 2023-11-02T18:21:58.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gb27Q640zqifRsOcMCkQJgObeMw=', 'Content-Length': '0'} 2023-11-02T18:21:58.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1258ea0ccdb14f8-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.016 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:21:58.016 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:21:58.017 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:21:58.017 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:21:58.018 DEBUG:boto:Method: GET 2023-11-02T18:21:58.018 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:21:58.019 DEBUG:boto:Data: 2023-11-02T18:21:58.019 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.019 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.020 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.020 DEBUG:boto:Params: {} 2023-11-02T18:21:58.021 DEBUG:boto:Token: None 2023-11-02T18:21:58.021 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:21:58.021 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gb27Q640zqifRsOcMCkQJgObeMw= 2023-11-02T18:21:58.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gb27Q640zqifRsOcMCkQJgObeMw=', 'Content-Length': '0'} 2023-11-02T18:21:58.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8f5caf81ded54c7-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.066 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:58.067 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:58.067 DEBUG:boto:path=/ibtljn-55/?prefix=c 2023-11-02T18:21:58.068 DEBUG:boto:auth_path=/ibtljn-55/?prefix=c 2023-11-02T18:21:58.068 DEBUG:boto:Method: GET 2023-11-02T18:21:58.068 DEBUG:boto:Path: /ibtljn-55/?prefix=c 2023-11-02T18:21:58.069 DEBUG:boto:Data: 2023-11-02T18:21:58.069 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:58.070 DEBUG:boto:Port: 8001 2023-11-02T18:21:58.070 DEBUG:boto:Params: {} 2023-11-02T18:21:58.071 DEBUG:boto:Token: None 2023-11-02T18:21:58.071 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/ 2023-11-02T18:21:58.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T18:21:58.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T18:21:58.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c9b4b4b9c6dc217-006543e8c6-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.078 DEBUG:boto:b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:58.079 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:58.079 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:58.081 DEBUG:boto:path=/ibtljn-55/?prefix=c 2023-11-02T18:21:58.081 DEBUG:boto:auth_path=/ibtljn-55/?prefix=c 2023-11-02T18:21:58.082 DEBUG:boto:Method: GET 2023-11-02T18:21:58.082 DEBUG:boto:Path: /ibtljn-55/?prefix=c 2023-11-02T18:21:58.083 DEBUG:boto:Data: 2023-11-02T18:21:58.083 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.084 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.084 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.084 DEBUG:boto:Params: {} 2023-11-02T18:21:58.085 DEBUG:boto:Token: None 2023-11-02T18:21:58.085 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/ 2023-11-02T18:21:58.086 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T18:21:58.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T18:21:58.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c3ea9139aa32ef0-006543e8c6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.092 DEBUG:boto:b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:58.093 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:21:58.093 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:21:58.094 DEBUG:boto:Method: HEAD 2023-11-02T18:21:58.094 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:21:58.095 DEBUG:boto:Data: 2023-11-02T18:21:58.095 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.095 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:58.096 DEBUG:boto:Port: 8001 2023-11-02T18:21:58.096 DEBUG:boto:Params: {} 2023-11-02T18:21:58.097 DEBUG:boto:Token: None 2023-11-02T18:21:58.097 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/c 2023-11-02T18:21:58.097 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Osypdm7cOvEv2stQd2/LWJsxs8U= 2023-11-02T18:21:58.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Osypdm7cOvEv2stQd2/LWJsxs8U=', 'Content-Length': '0'} 2023-11-02T18:21:58.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d6c5f91d63db2cf-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.101 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:21:58.102 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:21:58.102 DEBUG:boto:Method: HEAD 2023-11-02T18:21:58.103 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:21:58.103 DEBUG:boto:Data: 2023-11-02T18:21:58.103 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.104 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.104 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.105 DEBUG:boto:Params: {} 2023-11-02T18:21:58.105 DEBUG:boto:Token: None 2023-11-02T18:21:58.105 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/c 2023-11-02T18:21:58.106 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Osypdm7cOvEv2stQd2/LWJsxs8U= 2023-11-02T18:21:58.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Osypdm7cOvEv2stQd2/LWJsxs8U=', 'Content-Length': '0'} 2023-11-02T18:21:58.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a77fe98cdf7f211c-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:21:58.110 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:21:58.110 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:21:58.111 DEBUG:boto:Method: GET 2023-11-02T18:21:58.111 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:21:58.112 DEBUG:boto:Data: 2023-11-02T18:21:58.112 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.113 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:58.113 DEBUG:boto:Port: 8001 2023-11-02T18:21:58.113 DEBUG:boto:Params: {} 2023-11-02T18:21:58.114 DEBUG:boto:Token: None 2023-11-02T18:21:58.114 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/c 2023-11-02T18:21:58.115 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NzPIT1CsBJmI5RgUUiXVnrblaMA= 2023-11-02T18:21:58.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NzPIT1CsBJmI5RgUUiXVnrblaMA=', 'Content-Length': '0'} 2023-11-02T18:21:58.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f882c99cc3a14c-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.119 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:21:58.119 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:21:58.120 DEBUG:boto:Method: GET 2023-11-02T18:21:58.120 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:21:58.121 DEBUG:boto:Data: 2023-11-02T18:21:58.121 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.122 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.122 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.122 DEBUG:boto:Params: {} 2023-11-02T18:21:58.123 DEBUG:boto:Token: None 2023-11-02T18:21:58.123 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/c 2023-11-02T18:21:58.124 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NzPIT1CsBJmI5RgUUiXVnrblaMA= 2023-11-02T18:21:58.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NzPIT1CsBJmI5RgUUiXVnrblaMA=', 'Content-Length': '0'} 2023-11-02T18:21:58.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df6ae382c7554286-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.128 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:58.128 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:58.129 DEBUG:boto:path=/ibtljn-55/?prefix=c 2023-11-02T18:21:58.129 DEBUG:boto:auth_path=/ibtljn-55/?prefix=c 2023-11-02T18:21:58.130 DEBUG:boto:Method: GET 2023-11-02T18:21:58.130 DEBUG:boto:Path: /ibtljn-55/?prefix=c 2023-11-02T18:21:58.130 DEBUG:boto:Data: 2023-11-02T18:21:58.131 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.131 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:58.132 DEBUG:boto:Port: 8001 2023-11-02T18:21:58.132 DEBUG:boto:Params: {} 2023-11-02T18:21:58.132 DEBUG:boto:Token: None 2023-11-02T18:21:58.133 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/ 2023-11-02T18:21:58.133 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T18:21:58.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T18:21:58.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f46bebc460db8126-006543e8c6-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.186 DEBUG:boto:b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:58.187 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:58.187 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:58.188 DEBUG:boto:path=/ibtljn-55/?prefix=c 2023-11-02T18:21:58.188 DEBUG:boto:auth_path=/ibtljn-55/?prefix=c 2023-11-02T18:21:58.189 DEBUG:boto:Method: GET 2023-11-02T18:21:58.189 DEBUG:boto:Path: /ibtljn-55/?prefix=c 2023-11-02T18:21:58.189 DEBUG:boto:Data: 2023-11-02T18:21:58.190 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.190 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.191 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.191 DEBUG:boto:Params: {} 2023-11-02T18:21:58.192 DEBUG:boto:Token: None 2023-11-02T18:21:58.192 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/ 2023-11-02T18:21:58.193 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T18:21:58.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T18:21:58.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f4306d2f5dfac5c-006543e8c6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.199 DEBUG:boto:b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:58.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:21:58.200 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:21:58.200 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:21:58.201 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:21:58.201 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:21:58.202 DEBUG:boto:Method: GET 2023-11-02T18:21:58.202 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:21:58.203 DEBUG:boto:Data: 2023-11-02T18:21:58.203 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.203 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:58.204 DEBUG:boto:Port: 8001 2023-11-02T18:21:58.204 DEBUG:boto:Params: {} 2023-11-02T18:21:58.205 DEBUG:boto:Token: None 2023-11-02T18:21:58.205 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:21:58.205 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA= 2023-11-02T18:21:58.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA=', 'Content-Length': '0'} 2023-11-02T18:21:58.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005881b767765a5bcc-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.209 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:21:58.210 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:21:58.210 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:21:58.211 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:21:58.211 DEBUG:boto:Method: GET 2023-11-02T18:21:58.211 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:21:58.212 DEBUG:boto:Data: 2023-11-02T18:21:58.212 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.213 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.213 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.213 DEBUG:boto:Params: {} 2023-11-02T18:21:58.214 DEBUG:boto:Token: None 2023-11-02T18:21:58.214 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:21:58.215 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA= 2023-11-02T18:21:58.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA=', 'Content-Length': '0'} 2023-11-02T18:21:58.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aca91e08f34407e5-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.262 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:21:58.263 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:21:58.263 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:21:58.263 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:21:58.264 DEBUG:boto:Method: HEAD 2023-11-02T18:21:58.264 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:21:58.265 DEBUG:boto:Data: 2023-11-02T18:21:58.265 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.266 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:58.266 DEBUG:boto:Port: 8001 2023-11-02T18:21:58.266 DEBUG:boto:Params: {} 2023-11-02T18:21:58.267 DEBUG:boto:Token: None 2023-11-02T18:21:58.267 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:21:58.268 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tgDkFm2y571d4U8TDoRhLRy4A0M= 2023-11-02T18:21:58.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tgDkFm2y571d4U8TDoRhLRy4A0M=', 'Content-Length': '0'} 2023-11-02T18:21:58.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16ad4233372a269-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.272 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:21:58.272 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:21:58.272 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:21:58.273 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:21:58.273 DEBUG:boto:Method: HEAD 2023-11-02T18:21:58.274 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:21:58.274 DEBUG:boto:Data: 2023-11-02T18:21:58.275 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.275 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.275 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.276 DEBUG:boto:Params: {} 2023-11-02T18:21:58.276 DEBUG:boto:Token: None 2023-11-02T18:21:58.277 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:21:58.277 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tgDkFm2y571d4U8TDoRhLRy4A0M= 2023-11-02T18:21:58.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tgDkFm2y571d4U8TDoRhLRy4A0M=', 'Content-Length': '0'} 2023-11-02T18:21:58.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000933bfc2941628310-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:21:58.282 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:21:58.282 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:21:58.282 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:21:58.283 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:21:58.283 DEBUG:boto:Method: GET 2023-11-02T18:21:58.284 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:21:58.284 DEBUG:boto:Data: 2023-11-02T18:21:58.285 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.285 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:58.285 DEBUG:boto:Port: 8001 2023-11-02T18:21:58.286 DEBUG:boto:Params: {} 2023-11-02T18:21:58.286 DEBUG:boto:Token: None 2023-11-02T18:21:58.289 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:21:58.289 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA= 2023-11-02T18:21:58.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA=', 'Content-Length': '0'} 2023-11-02T18:21:58.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6abfa630eda764d-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.294 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:21:58.294 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:21:58.295 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:21:58.295 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:21:58.296 DEBUG:boto:Method: GET 2023-11-02T18:21:58.296 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:21:58.296 DEBUG:boto:Data: 2023-11-02T18:21:58.297 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.297 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.298 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.298 DEBUG:boto:Params: {} 2023-11-02T18:21:58.299 DEBUG:boto:Token: None 2023-11-02T18:21:58.299 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:21:58.300 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA= 2023-11-02T18:21:58.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA=', 'Content-Length': '0'} 2023-11-02T18:21:58.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2907d1a0dcb3bdd-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.304 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:58.304 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:58.305 DEBUG:boto:path=/ibtljn-55/?prefix=d 2023-11-02T18:21:58.305 DEBUG:boto:auth_path=/ibtljn-55/?prefix=d 2023-11-02T18:21:58.306 DEBUG:boto:Method: GET 2023-11-02T18:21:58.306 DEBUG:boto:Path: /ibtljn-55/?prefix=d 2023-11-02T18:21:58.307 DEBUG:boto:Data: 2023-11-02T18:21:58.307 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.307 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:58.308 DEBUG:boto:Port: 8001 2023-11-02T18:21:58.308 DEBUG:boto:Params: {} 2023-11-02T18:21:58.309 DEBUG:boto:Token: None 2023-11-02T18:21:58.309 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/ 2023-11-02T18:21:58.310 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T18:21:58.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T18:21:58.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5e18fa9ffba4ce4-006543e8c6-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.315 DEBUG:boto:b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:58.316 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:58.316 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:58.317 DEBUG:boto:path=/ibtljn-55/?prefix=d 2023-11-02T18:21:58.317 DEBUG:boto:auth_path=/ibtljn-55/?prefix=d 2023-11-02T18:21:58.318 DEBUG:boto:Method: GET 2023-11-02T18:21:58.318 DEBUG:boto:Path: /ibtljn-55/?prefix=d 2023-11-02T18:21:58.319 DEBUG:boto:Data: 2023-11-02T18:21:58.319 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.319 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.320 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.320 DEBUG:boto:Params: {} 2023-11-02T18:21:58.321 DEBUG:boto:Token: None 2023-11-02T18:21:58.321 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/ 2023-11-02T18:21:58.322 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T18:21:58.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T18:21:58.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aa47b26be909931-006543e8c6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.328 DEBUG:boto:b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:58.328 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:21:58.329 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:21:58.329 DEBUG:boto:Method: HEAD 2023-11-02T18:21:58.329 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:21:58.330 DEBUG:boto:Data: 2023-11-02T18:21:58.330 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.331 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:58.331 DEBUG:boto:Port: 8001 2023-11-02T18:21:58.332 DEBUG:boto:Params: {} 2023-11-02T18:21:58.332 DEBUG:boto:Token: None 2023-11-02T18:21:58.332 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/d 2023-11-02T18:21:58.333 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tRJqQ3TnddLM261w4X08TzYu2OA= 2023-11-02T18:21:58.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tRJqQ3TnddLM261w4X08TzYu2OA=', 'Content-Length': '0'} 2023-11-02T18:21:58.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c7fb2257529d08-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.337 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:21:58.337 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:21:58.338 DEBUG:boto:Method: HEAD 2023-11-02T18:21:58.338 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:21:58.338 DEBUG:boto:Data: 2023-11-02T18:21:58.339 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.339 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.340 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.340 DEBUG:boto:Params: {} 2023-11-02T18:21:58.340 DEBUG:boto:Token: None 2023-11-02T18:21:58.341 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/d 2023-11-02T18:21:58.341 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tRJqQ3TnddLM261w4X08TzYu2OA= 2023-11-02T18:21:58.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tRJqQ3TnddLM261w4X08TzYu2OA=', 'Content-Length': '0'} 2023-11-02T18:21:58.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8912922220555fb-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:21:58.346 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:21:58.346 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:21:58.347 DEBUG:boto:Method: GET 2023-11-02T18:21:58.347 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:21:58.348 DEBUG:boto:Data: 2023-11-02T18:21:58.348 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.348 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:58.349 DEBUG:boto:Port: 8001 2023-11-02T18:21:58.349 DEBUG:boto:Params: {} 2023-11-02T18:21:58.350 DEBUG:boto:Token: None 2023-11-02T18:21:58.350 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/d 2023-11-02T18:21:58.351 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hbXE2TweZhV3wKXwNF/3crtBuNI= 2023-11-02T18:21:58.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hbXE2TweZhV3wKXwNF/3crtBuNI=', 'Content-Length': '0'} 2023-11-02T18:21:58.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7e48addbb57ef4-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.394 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:21:58.395 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:21:58.395 DEBUG:boto:Method: GET 2023-11-02T18:21:58.396 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:21:58.396 DEBUG:boto:Data: 2023-11-02T18:21:58.396 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.397 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.397 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.398 DEBUG:boto:Params: {} 2023-11-02T18:21:58.398 DEBUG:boto:Token: None 2023-11-02T18:21:58.398 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/d 2023-11-02T18:21:58.399 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hbXE2TweZhV3wKXwNF/3crtBuNI= 2023-11-02T18:21:58.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hbXE2TweZhV3wKXwNF/3crtBuNI=', 'Content-Length': '0'} 2023-11-02T18:21:58.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006506712b7fc7a8f1-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.403 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:58.403 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:58.404 DEBUG:boto:path=/ibtljn-55/?prefix=d 2023-11-02T18:21:58.404 DEBUG:boto:auth_path=/ibtljn-55/?prefix=d 2023-11-02T18:21:58.405 DEBUG:boto:Method: GET 2023-11-02T18:21:58.405 DEBUG:boto:Path: /ibtljn-55/?prefix=d 2023-11-02T18:21:58.406 DEBUG:boto:Data: 2023-11-02T18:21:58.406 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.406 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:21:58.407 DEBUG:boto:Port: 8001 2023-11-02T18:21:58.407 DEBUG:boto:Params: {} 2023-11-02T18:21:58.408 DEBUG:boto:Token: None 2023-11-02T18:21:58.408 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/ 2023-11-02T18:21:58.409 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T18:21:58.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T18:21:58.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096fbc9067ca8d5ca-006543e8c6-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.415 DEBUG:boto:b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:58.415 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:58.416 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:58.416 DEBUG:boto:path=/ibtljn-55/?prefix=d 2023-11-02T18:21:58.417 DEBUG:boto:auth_path=/ibtljn-55/?prefix=d 2023-11-02T18:21:58.417 DEBUG:boto:Method: GET 2023-11-02T18:21:58.417 DEBUG:boto:Path: /ibtljn-55/?prefix=d 2023-11-02T18:21:58.418 DEBUG:boto:Data: 2023-11-02T18:21:58.418 DEBUG:boto:Headers: {} 2023-11-02T18:21:58.419 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:58.419 DEBUG:boto:Port: 8000 2023-11-02T18:21:58.420 DEBUG:boto:Params: {} 2023-11-02T18:21:58.420 DEBUG:boto:Token: None 2023-11-02T18:21:58.420 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:21:58 GMT /ibtljn-55/ 2023-11-02T18:21:58.421 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T18:21:58.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T18:21:58.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000650327a5d034d63-006543e8c6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:58.426 DEBUG:boto:b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:21:58.427 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-55 zones={a2, a1} 2023-11-02T18:21:58.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:21:58.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:21:58.428 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ibtljn-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:21:58.470 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:58.471 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:58.555 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:21:58.556 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:21:58.556 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding": "None", 2023-11-02T18:21:58.556 INFO:teuthology.orchestra.run.smithi017.stdout: "current_index": { 2023-11-02T18:21:58.556 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 0, 2023-11-02T18:21:58.557 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:21:58.557 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "Normal", 2023-11-02T18:21:58.557 INFO:teuthology.orchestra.run.smithi017.stdout: "normal": { 2023-11-02T18:21:58.557 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 11, 2023-11-02T18:21:58.558 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:21:58.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:58.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:58.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:58.558 INFO:teuthology.orchestra.run.smithi017.stdout: "logs": [ 2023-11-02T18:21:58.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:58.559 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 0, 2023-11-02T18:21:58.559 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:21:58.559 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "InIndex", 2023-11-02T18:21:58.560 INFO:teuthology.orchestra.run.smithi017.stdout: "in_index": { 2023-11-02T18:21:58.560 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 0, 2023-11-02T18:21:58.560 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:21:58.560 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 11, 2023-11-02T18:21:58.560 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:21:58.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:58.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:58.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:58.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:58.561 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:21:58.562 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:58.562 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:21:58.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ibtljn-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:58.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ibtljn-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:58.564 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ibtljn-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:21:58.603 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:58.603 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:58.684 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:21:58.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:58.685 INFO:teuthology.orchestra.run.smithi017.stdout: "op_id": "1#00000000001.24.6", 2023-11-02T18:21:58.685 INFO:teuthology.orchestra.run.smithi017.stdout: "op_tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.3107276113268200966", 2023-11-02T18:21:58.685 INFO:teuthology.orchestra.run.smithi017.stdout: "op": "write", 2023-11-02T18:21:58.686 INFO:teuthology.orchestra.run.smithi017.stdout: "object": "a", 2023-11-02T18:21:58.686 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:21:58.686 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "complete", 2023-11-02T18:21:58.686 INFO:teuthology.orchestra.run.smithi017.stdout: "index_ver": 1, 2023-11-02T18:21:58.687 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:21:36.542931161Z", 2023-11-02T18:21:58.687 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:21:58.687 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:21:58.687 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 8 2023-11-02T18:21:58.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:58.688 INFO:teuthology.orchestra.run.smithi017.stdout: "bilog_flags": 0, 2023-11-02T18:21:58.688 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned": false, 2023-11-02T18:21:58.688 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "", 2023-11-02T18:21:58.688 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "", 2023-11-02T18:21:58.689 INFO:teuthology.orchestra.run.smithi017.stdout: "zones_trace": [ 2023-11-02T18:21:58.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:58.689 INFO:teuthology.orchestra.run.smithi017.stdout: "entry": "d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20" 2023-11-02T18:21:58.689 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:58.689 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:21:58.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:58.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:58.690 INFO:teuthology.orchestra.run.smithi017.stdout: "op_id": "5#00000000001.66.6", 2023-11-02T18:21:58.690 INFO:teuthology.orchestra.run.smithi017.stdout: "op_tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.3103660691027834791", 2023-11-02T18:21:58.690 INFO:teuthology.orchestra.run.smithi017.stdout: "op": "write", 2023-11-02T18:21:58.691 INFO:teuthology.orchestra.run.smithi017.stdout: "object": "d", 2023-11-02T18:21:58.691 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:21:58.691 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "complete", 2023-11-02T18:21:58.691 INFO:teuthology.orchestra.run.smithi017.stdout: "index_ver": 1, 2023-11-02T18:21:58.691 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:21:36.658842131Z", 2023-11-02T18:21:58.692 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:21:58.692 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:21:58.692 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:21:58.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:58.692 INFO:teuthology.orchestra.run.smithi017.stdout: "bilog_flags": 0, 2023-11-02T18:21:58.693 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned": false, 2023-11-02T18:21:58.693 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "", 2023-11-02T18:21:58.693 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "", 2023-11-02T18:21:58.693 INFO:teuthology.orchestra.run.smithi017.stdout: "zones_trace": [ 2023-11-02T18:21:58.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:58.694 INFO:teuthology.orchestra.run.smithi017.stdout: "entry": "d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20" 2023-11-02T18:21:58.694 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:58.694 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:21:58.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:58.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:58.695 INFO:teuthology.orchestra.run.smithi017.stdout: "op_id": "8#00000000001.64.6", 2023-11-02T18:21:58.695 INFO:teuthology.orchestra.run.smithi017.stdout: "op_tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.13187839468121546283", 2023-11-02T18:21:58.695 INFO:teuthology.orchestra.run.smithi017.stdout: "op": "write", 2023-11-02T18:21:58.695 INFO:teuthology.orchestra.run.smithi017.stdout: "object": "c", 2023-11-02T18:21:58.697 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:21:58.697 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "complete", 2023-11-02T18:21:58.697 INFO:teuthology.orchestra.run.smithi017.stdout: "index_ver": 1, 2023-11-02T18:21:58.698 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:21:36.631179381Z", 2023-11-02T18:21:58.698 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:21:58.698 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:21:58.698 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:21:58.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:58.699 INFO:teuthology.orchestra.run.smithi017.stdout: "bilog_flags": 0, 2023-11-02T18:21:58.699 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned": false, 2023-11-02T18:21:58.699 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "", 2023-11-02T18:21:58.699 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "", 2023-11-02T18:21:58.700 INFO:teuthology.orchestra.run.smithi017.stdout: "zones_trace": [ 2023-11-02T18:21:58.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:58.700 INFO:teuthology.orchestra.run.smithi017.stdout: "entry": "d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20" 2023-11-02T18:21:58.700 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:58.700 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:21:58.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:58.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:58.701 INFO:teuthology.orchestra.run.smithi017.stdout: "op_id": "10#00000000001.29.6", 2023-11-02T18:21:58.701 INFO:teuthology.orchestra.run.smithi017.stdout: "op_tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.15858638994088866377", 2023-11-02T18:21:58.701 INFO:teuthology.orchestra.run.smithi017.stdout: "op": "write", 2023-11-02T18:21:58.702 INFO:teuthology.orchestra.run.smithi017.stdout: "object": "b", 2023-11-02T18:21:58.702 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:21:58.702 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "complete", 2023-11-02T18:21:58.702 INFO:teuthology.orchestra.run.smithi017.stdout: "index_ver": 1, 2023-11-02T18:21:58.703 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:21:36.570165453Z", 2023-11-02T18:21:58.703 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:21:58.703 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:21:58.703 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10 2023-11-02T18:21:58.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:58.703 INFO:teuthology.orchestra.run.smithi017.stdout: "bilog_flags": 0, 2023-11-02T18:21:58.704 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned": false, 2023-11-02T18:21:58.704 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "", 2023-11-02T18:21:58.704 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "", 2023-11-02T18:21:58.704 INFO:teuthology.orchestra.run.smithi017.stdout: "zones_trace": [ 2023-11-02T18:21:58.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:58.705 INFO:teuthology.orchestra.run.smithi017.stdout: "entry": "d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20" 2023-11-02T18:21:58.705 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:58.705 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:21:58.705 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:58.705 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:21:58.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'ibtljn-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:21:58.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'ibtljn-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:21:58.707 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket ibtljn-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:21:58.753 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:58.753 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:58.834 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:21:58.834 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-55 2023-11-02T18:21:58.844 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 4 2023-11-02T18:21:58.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:21:58.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:21:58.926 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ibtljn-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:21:58.970 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:58.970 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:59.055 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:21:59.055 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:21:59.056 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding": "None", 2023-11-02T18:21:59.056 INFO:teuthology.orchestra.run.smithi017.stdout: "current_index": { 2023-11-02T18:21:59.056 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 1, 2023-11-02T18:21:59.056 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:21:59.057 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "Normal", 2023-11-02T18:21:59.057 INFO:teuthology.orchestra.run.smithi017.stdout: "normal": { 2023-11-02T18:21:59.057 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 3, 2023-11-02T18:21:59.057 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:21:59.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:59.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:59.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:59.058 INFO:teuthology.orchestra.run.smithi017.stdout: "logs": [ 2023-11-02T18:21:59.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:59.059 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 0, 2023-11-02T18:21:59.059 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:21:59.059 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "InIndex", 2023-11-02T18:21:59.059 INFO:teuthology.orchestra.run.smithi017.stdout: "in_index": { 2023-11-02T18:21:59.060 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 0, 2023-11-02T18:21:59.060 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:21:59.060 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 11, 2023-11-02T18:21:59.060 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:21:59.060 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:59.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:59.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:59.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:21:59.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:21:59.062 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 1, 2023-11-02T18:21:59.062 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:21:59.062 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "InIndex", 2023-11-02T18:21:59.062 INFO:teuthology.orchestra.run.smithi017.stdout: "in_index": { 2023-11-02T18:21:59.063 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 1, 2023-11-02T18:21:59.063 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:21:59.063 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 3, 2023-11-02T18:21:59.063 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:21:59.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:59.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:59.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:59.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:59.064 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:21:59.065 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:21:59.065 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:21:59.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ibtljn-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:59.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ibtljn-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:21:59.067 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ibtljn-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:21:59.114 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:59.115 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:59.203 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2023-11-02T18:21:59.206 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:59.207 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:59.207 DEBUG:boto:Method: HEAD 2023-11-02T18:21:59.208 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:21:59.208 DEBUG:boto:Data: 2023-11-02T18:21:59.208 DEBUG:boto:Headers: {} 2023-11-02T18:21:59.209 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:59.209 DEBUG:boto:Port: 8000 2023-11-02T18:21:59.210 DEBUG:boto:Params: {} 2023-11-02T18:21:59.210 DEBUG:boto:Token: None 2023-11-02T18:21:59.211 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:59 GMT /ibtljn-55/ 2023-11-02T18:21:59.211 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY= 2023-11-02T18:21:59.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY=', 'Content-Length': '0'} 2023-11-02T18:21:59.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d29cbcb815e41db5-006543e8c7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:59.215 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:21:59.216 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:21:59.216 DEBUG:boto:Method: PUT 2023-11-02T18:21:59.216 DEBUG:boto:Path: /ibtljn-55/e 2023-11-02T18:21:59.217 DEBUG:boto:Data: 2023-11-02T18:21:59.217 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:21:59.218 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:59.218 DEBUG:boto:Port: 8000 2023-11-02T18:21:59.218 DEBUG:boto:Params: {} 2023-11-02T18:21:59.219 DEBUG:boto:Token: None 2023-11-02T18:21:59.219 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:21:59 GMT /ibtljn-55/e 2023-11-02T18:21:59.220 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m2ORivXwqPxYi5Zt0PdacwHhQr4= 2023-11-02T18:21:59.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m2ORivXwqPxYi5Zt0PdacwHhQr4='} 2023-11-02T18:21:59.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038fb63939ca04287-006543e8c7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:59.231 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:59.232 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:59.232 DEBUG:boto:Method: HEAD 2023-11-02T18:21:59.233 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:21:59.233 DEBUG:boto:Data: 2023-11-02T18:21:59.233 DEBUG:boto:Headers: {} 2023-11-02T18:21:59.234 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:59.234 DEBUG:boto:Port: 8000 2023-11-02T18:21:59.234 DEBUG:boto:Params: {} 2023-11-02T18:21:59.235 DEBUG:boto:Token: None 2023-11-02T18:21:59.235 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:59 GMT /ibtljn-55/ 2023-11-02T18:21:59.236 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY= 2023-11-02T18:21:59.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY=', 'Content-Length': '0'} 2023-11-02T18:21:59.239 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de5683270a229478-006543e8c7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:59.240 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:21:59.240 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:21:59.240 DEBUG:boto:Method: PUT 2023-11-02T18:21:59.241 DEBUG:boto:Path: /ibtljn-55/f 2023-11-02T18:21:59.241 DEBUG:boto:Data: 2023-11-02T18:21:59.242 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:21:59.242 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:59.242 DEBUG:boto:Port: 8000 2023-11-02T18:21:59.243 DEBUG:boto:Params: {} 2023-11-02T18:21:59.243 DEBUG:boto:Token: None 2023-11-02T18:21:59.244 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:21:59 GMT /ibtljn-55/f 2023-11-02T18:21:59.244 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Xbtgbsgb93iiMBFOKU1zgENNuJI= 2023-11-02T18:21:59.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Xbtgbsgb93iiMBFOKU1zgENNuJI='} 2023-11-02T18:21:59.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077a1edea718f883d-006543e8c7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:59.291 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:59.292 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:59.292 DEBUG:boto:Method: HEAD 2023-11-02T18:21:59.292 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:21:59.293 DEBUG:boto:Data: 2023-11-02T18:21:59.293 DEBUG:boto:Headers: {} 2023-11-02T18:21:59.293 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:59.294 DEBUG:boto:Port: 8000 2023-11-02T18:21:59.294 DEBUG:boto:Params: {} 2023-11-02T18:21:59.295 DEBUG:boto:Token: None 2023-11-02T18:21:59.295 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:59 GMT /ibtljn-55/ 2023-11-02T18:21:59.295 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY= 2023-11-02T18:21:59.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY=', 'Content-Length': '0'} 2023-11-02T18:21:59.299 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cc56f0ad8e51957-006543e8c7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:59.299 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:21:59.300 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:21:59.300 DEBUG:boto:Method: PUT 2023-11-02T18:21:59.300 DEBUG:boto:Path: /ibtljn-55/g 2023-11-02T18:21:59.301 DEBUG:boto:Data: 2023-11-02T18:21:59.301 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:21:59.302 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:59.302 DEBUG:boto:Port: 8000 2023-11-02T18:21:59.302 DEBUG:boto:Params: {} 2023-11-02T18:21:59.303 DEBUG:boto:Token: None 2023-11-02T18:21:59.303 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:21:59 GMT /ibtljn-55/g 2023-11-02T18:21:59.304 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2Wk5nFJoKaRJ03dE2gQapgSAuuw= 2023-11-02T18:21:59.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2Wk5nFJoKaRJ03dE2gQapgSAuuw='} 2023-11-02T18:21:59.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7870befa8280cd1-006543e8c7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:59.317 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:21:59.317 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:21:59.317 DEBUG:boto:Method: HEAD 2023-11-02T18:21:59.318 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:21:59.318 DEBUG:boto:Data: 2023-11-02T18:21:59.319 DEBUG:boto:Headers: {} 2023-11-02T18:21:59.319 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:59.319 DEBUG:boto:Port: 8000 2023-11-02T18:21:59.320 DEBUG:boto:Params: {} 2023-11-02T18:21:59.320 DEBUG:boto:Token: None 2023-11-02T18:21:59.320 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:21:59 GMT /ibtljn-55/ 2023-11-02T18:21:59.321 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY= 2023-11-02T18:21:59.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY=', 'Content-Length': '0'} 2023-11-02T18:21:59.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002363a9f6bac8ec59-006543e8c7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:59.325 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:21:59.325 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:21:59.325 DEBUG:boto:Method: PUT 2023-11-02T18:21:59.326 DEBUG:boto:Path: /ibtljn-55/h 2023-11-02T18:21:59.326 DEBUG:boto:Data: 2023-11-02T18:21:59.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:21:59.327 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:21:59.327 DEBUG:boto:Port: 8000 2023-11-02T18:21:59.328 DEBUG:boto:Params: {} 2023-11-02T18:21:59.328 DEBUG:boto:Token: None 2023-11-02T18:21:59.328 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:21:59 GMT /ibtljn-55/h 2023-11-02T18:21:59.329 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zjuOqJ37PlqRYPu9nzZnqjiUY+0= 2023-11-02T18:21:59.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zjuOqJ37PlqRYPu9nzZnqjiUY+0='} 2023-11-02T18:21:59.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2963113fa8a3027-006543e8c7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:21:59.375 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-55 2023-11-02T18:21:59.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:21:59.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:21:59.376 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:21:59.429 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:21:59.430 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:21:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:21:59.489+0000 7f8dc642ff40 1 waiting to reach latest gen 1, on 0.. 2023-11-02T18:22:04.497 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:22:04.489+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:22:04.497 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:22:04.497 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:22:09.498 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:22:09.493+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:22:09.498 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:22:09.498 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:22:14.499 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:22:14.493+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:22:14.499 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:22:14.500 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:22:19.500 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:22:19.493+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:22:19.501 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:22:19.501 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:22:24.502 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:22:24.497+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:22:24.502 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:22:24.502 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:22:29.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:22:29.497+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:22:29.503 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:22:29.504 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:22:34.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:22:34.497+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:22:34.505 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:22:34.505 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:22:39.505 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:22:39.501+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:22:39.506 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:22:39.506 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:22:44.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:22:44.501+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:22:44.507 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:22:44.507 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:22:49.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:22:49.501+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:22:49.508 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:22:49.509 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:22:54.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:22:54.505+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:22:54.509 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:22:54.510 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:22:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:22:59.505+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:22:59.513 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:22:59.514 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:23:04.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:04.505+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:23:04.512 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000003.24.6, , ] 2023-11-02T18:23:04.512 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:23:09.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:09.509+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:23:09.513 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000003.24.6, , ] 2023-11-02T18:23:09.514 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:23:14.514 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:14.509+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:23:14.515 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000003.24.6, , ] 2023-11-02T18:23:14.515 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:23:19.516 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:19.509+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T18:23:19.516 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000003.24.6, , ] 2023-11-02T18:23:19.516 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:23:24.517 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:24.513+0000 7f8dc642ff40 1 bucket sync caught up with source: 2023-11-02T18:23:24.518 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:23:24.518 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T18:23:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:24.513+0000 7f8dc642ff40 0 bucket checkpoint complete 2023-11-02T18:23:24.548 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-55 2023-11-02T18:23:24.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:23:24.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:23:24.550 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:23:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:23:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:23:24.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:24.665+0000 7fba2bb45f40 1 waiting for incremental sync to catch up: 2023-11-02T18:23:24.669 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.174.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000001.192.6, , 00000000002.181.6] 2023-11-02T18:23:24.670 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T18:23:29.672 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:29.665+0000 7fba2bb45f40 1 waiting for incremental sync to catch up: 2023-11-02T18:23:29.672 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.174.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000001.192.6, , 00000000002.181.6] 2023-11-02T18:23:29.672 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T18:23:34.675 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:34.669+0000 7fba2bb45f40 1 waiting for incremental sync to catch up: 2023-11-02T18:23:34.675 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000001.192.6, , 00000000002.181.6] 2023-11-02T18:23:34.675 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T18:23:39.677 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:39.673+0000 7fba2bb45f40 1 waiting for incremental sync to catch up: 2023-11-02T18:23:39.678 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000001.192.6, , 00000000002.181.6] 2023-11-02T18:23:39.678 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T18:23:44.680 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:44.673+0000 7fba2bb45f40 1 bucket sync caught up with source: 2023-11-02T18:23:44.680 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T18:23:44.680 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T18:23:44.681 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:44.673+0000 7fba2bb45f40 0 bucket checkpoint complete 2023-11-02T18:23:44.710 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-55 zones={a2, a1} 2023-11-02T18:23:44.711 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:44.711 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:44.711 DEBUG:boto:Method: HEAD 2023-11-02T18:23:44.712 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:23:44.712 DEBUG:boto:Data: 2023-11-02T18:23:44.713 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.713 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:44.714 DEBUG:boto:Port: 8001 2023-11-02T18:23:44.714 DEBUG:boto:Params: {} 2023-11-02T18:23:44.715 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:23:44.715 DEBUG:boto:Token: None 2023-11-02T18:23:44.716 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/ 2023-11-02T18:23:44.716 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+a0H42agG1AtvF3zm5w/tCNf+p4= 2023-11-02T18:23:44.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+a0H42agG1AtvF3zm5w/tCNf+p4=', 'Content-Length': '0'} 2023-11-02T18:23:44.723 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000514e3ee38acdf241-006543e930-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.723 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:44.724 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:44.724 DEBUG:boto:Method: HEAD 2023-11-02T18:23:44.725 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:23:44.725 DEBUG:boto:Data: 2023-11-02T18:23:44.725 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.726 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:44.726 DEBUG:boto:Port: 8000 2023-11-02T18:23:44.727 DEBUG:boto:Params: {} 2023-11-02T18:23:44.727 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:23:44.728 DEBUG:boto:Token: None 2023-11-02T18:23:44.728 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/ 2023-11-02T18:23:44.729 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+a0H42agG1AtvF3zm5w/tCNf+p4= 2023-11-02T18:23:44.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+a0H42agG1AtvF3zm5w/tCNf+p4=', 'Content-Length': '0'} 2023-11-02T18:23:44.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020b245db2d47e723-006543e930-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.734 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:23:44.734 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:44.735 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:44.735 DEBUG:boto:path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:23:44.736 DEBUG:boto:auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:23:44.736 DEBUG:boto:Method: GET 2023-11-02T18:23:44.737 DEBUG:boto:Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T18:23:44.737 DEBUG:boto:Data: 2023-11-02T18:23:44.737 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:44.738 DEBUG:boto:Port: 8001 2023-11-02T18:23:44.739 DEBUG:boto:Params: {} 2023-11-02T18:23:44.739 DEBUG:boto:Token: None 2023-11-02T18:23:44.740 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/?versions 2023-11-02T18:23:44.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE= 2023-11-02T18:23:44.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE=', 'Content-Length': '0'} 2023-11-02T18:23:44.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f37251e9df0ae2a-006543e930-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.746 DEBUG:boto:b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:44.747 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:23:44.748 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:23:44.748 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:23:44.748 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:23:44.749 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-11-02T18:23:44.749 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-11-02T18:23:44.750 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-11-02T18:23:44.750 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-11-02T18:23:44.750 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:23:44.751 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:44.751 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:44.752 DEBUG:boto:path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:23:44.752 DEBUG:boto:auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:23:44.753 DEBUG:boto:Method: GET 2023-11-02T18:23:44.753 DEBUG:boto:Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T18:23:44.753 DEBUG:boto:Data: 2023-11-02T18:23:44.754 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:44.755 DEBUG:boto:Port: 8000 2023-11-02T18:23:44.755 DEBUG:boto:Params: {} 2023-11-02T18:23:44.756 DEBUG:boto:Token: None 2023-11-02T18:23:44.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/?versions 2023-11-02T18:23:44.756 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE= 2023-11-02T18:23:44.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE=', 'Content-Length': '0'} 2023-11-02T18:23:44.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bf11f0af695f87d-006543e930-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.761 DEBUG:boto:b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:44.762 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:23:44.762 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:23:44.763 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:23:44.763 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:23:44.765 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-11-02T18:23:44.766 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-11-02T18:23:44.767 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-11-02T18:23:44.767 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-11-02T18:23:44.767 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:44.768 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:44.768 DEBUG:boto:path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:23:44.769 DEBUG:boto:auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:23:44.769 DEBUG:boto:Method: GET 2023-11-02T18:23:44.770 DEBUG:boto:Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T18:23:44.770 DEBUG:boto:Data: 2023-11-02T18:23:44.771 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.771 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:44.772 DEBUG:boto:Port: 8001 2023-11-02T18:23:44.772 DEBUG:boto:Params: {} 2023-11-02T18:23:44.773 DEBUG:boto:Token: None 2023-11-02T18:23:44.773 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/?versions 2023-11-02T18:23:44.774 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE= 2023-11-02T18:23:44.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE=', 'Content-Length': '0'} 2023-11-02T18:23:44.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000afbf7b61c980986-006543e930-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.780 DEBUG:boto:b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:44.781 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:44.782 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:44.782 DEBUG:boto:path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:23:44.782 DEBUG:boto:auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:23:44.783 DEBUG:boto:Method: GET 2023-11-02T18:23:44.784 DEBUG:boto:Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T18:23:44.784 DEBUG:boto:Data: 2023-11-02T18:23:44.785 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.785 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:44.786 DEBUG:boto:Port: 8000 2023-11-02T18:23:44.787 DEBUG:boto:Params: {} 2023-11-02T18:23:44.788 DEBUG:boto:Token: None 2023-11-02T18:23:44.788 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/?versions 2023-11-02T18:23:44.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE= 2023-11-02T18:23:44.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE=', 'Content-Length': '0'} 2023-11-02T18:23:44.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000828a25bb6d3fea7e-006543e930-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.794 DEBUG:boto:b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:44.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:23:44.795 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:23:44.796 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:23:44.796 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:23:44.797 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:23:44.797 DEBUG:boto:Method: GET 2023-11-02T18:23:44.798 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:23:44.798 DEBUG:boto:Data: 2023-11-02T18:23:44.799 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:44.799 DEBUG:boto:Port: 8001 2023-11-02T18:23:44.800 DEBUG:boto:Params: {} 2023-11-02T18:23:44.800 DEBUG:boto:Token: None 2023-11-02T18:23:44.801 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:23:44.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak= 2023-11-02T18:23:44.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak=', 'Content-Length': '0'} 2023-11-02T18:23:44.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc34b0b67dabda34-006543e930-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.850 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:23:44.851 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:23:44.852 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:23:44.852 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:23:44.853 DEBUG:boto:Method: GET 2023-11-02T18:23:44.853 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:23:44.853 DEBUG:boto:Data: 2023-11-02T18:23:44.854 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.854 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:44.855 DEBUG:boto:Port: 8000 2023-11-02T18:23:44.855 DEBUG:boto:Params: {} 2023-11-02T18:23:44.856 DEBUG:boto:Token: None 2023-11-02T18:23:44.856 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:23:44.857 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak= 2023-11-02T18:23:44.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak=', 'Content-Length': '0'} 2023-11-02T18:23:44.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e71306008795bccd-006543e930-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.861 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:23:44.862 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:23:44.862 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:23:44.863 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:23:44.863 DEBUG:boto:Method: HEAD 2023-11-02T18:23:44.863 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:23:44.864 DEBUG:boto:Data: 2023-11-02T18:23:44.864 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.865 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:44.865 DEBUG:boto:Port: 8001 2023-11-02T18:23:44.866 DEBUG:boto:Params: {} 2023-11-02T18:23:44.866 DEBUG:boto:Token: None 2023-11-02T18:23:44.867 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:23:44.867 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xCIMqLHd3gWHY39njcXj/dPFDgo= 2023-11-02T18:23:44.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xCIMqLHd3gWHY39njcXj/dPFDgo=', 'Content-Length': '0'} 2023-11-02T18:23:44.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af3c2767ef905cff-006543e930-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.871 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:23:44.871 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:23:44.872 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:23:44.872 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:23:44.872 DEBUG:boto:Method: HEAD 2023-11-02T18:23:44.873 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:23:44.873 DEBUG:boto:Data: 2023-11-02T18:23:44.874 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.874 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:44.875 DEBUG:boto:Port: 8000 2023-11-02T18:23:44.875 DEBUG:boto:Params: {} 2023-11-02T18:23:44.876 DEBUG:boto:Token: None 2023-11-02T18:23:44.876 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:23:44.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xCIMqLHd3gWHY39njcXj/dPFDgo= 2023-11-02T18:23:44.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xCIMqLHd3gWHY39njcXj/dPFDgo=', 'Content-Length': '0'} 2023-11-02T18:23:44.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d16c5576fac4f9-006543e930-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:23:44.881 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:23:44.881 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:23:44.882 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:23:44.882 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:23:44.883 DEBUG:boto:Method: GET 2023-11-02T18:23:44.883 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:23:44.884 DEBUG:boto:Data: 2023-11-02T18:23:44.884 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.885 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:44.885 DEBUG:boto:Port: 8001 2023-11-02T18:23:44.886 DEBUG:boto:Params: {} 2023-11-02T18:23:44.886 DEBUG:boto:Token: None 2023-11-02T18:23:44.887 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:23:44.887 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak= 2023-11-02T18:23:44.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak=', 'Content-Length': '0'} 2023-11-02T18:23:44.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000294c65f0884036ad-006543e930-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.892 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:23:44.892 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:23:44.893 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:23:44.893 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:23:44.894 DEBUG:boto:Method: GET 2023-11-02T18:23:44.894 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:23:44.894 DEBUG:boto:Data: 2023-11-02T18:23:44.895 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.895 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:44.896 DEBUG:boto:Port: 8000 2023-11-02T18:23:44.896 DEBUG:boto:Params: {} 2023-11-02T18:23:44.897 DEBUG:boto:Token: None 2023-11-02T18:23:44.897 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:23:44.898 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak= 2023-11-02T18:23:44.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak=', 'Content-Length': '0'} 2023-11-02T18:23:44.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007159c289b28054c1-006543e930-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.902 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:44.902 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:44.903 DEBUG:boto:path=/ibtljn-55/?prefix=a 2023-11-02T18:23:44.903 DEBUG:boto:auth_path=/ibtljn-55/?prefix=a 2023-11-02T18:23:44.904 DEBUG:boto:Method: GET 2023-11-02T18:23:44.904 DEBUG:boto:Path: /ibtljn-55/?prefix=a 2023-11-02T18:23:44.905 DEBUG:boto:Data: 2023-11-02T18:23:44.905 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:44.906 DEBUG:boto:Port: 8001 2023-11-02T18:23:44.907 DEBUG:boto:Params: {} 2023-11-02T18:23:44.907 DEBUG:boto:Token: None 2023-11-02T18:23:44.908 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/ 2023-11-02T18:23:44.908 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/4eBOtjozJ7Mr+oTagkPsp2tLks= 2023-11-02T18:23:44.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/4eBOtjozJ7Mr+oTagkPsp2tLks=', 'Content-Length': '0'} 2023-11-02T18:23:44.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530f11c7b875d79b-006543e930-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.914 DEBUG:boto:b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:44.915 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:44.916 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:44.916 DEBUG:boto:path=/ibtljn-55/?prefix=a 2023-11-02T18:23:44.916 DEBUG:boto:auth_path=/ibtljn-55/?prefix=a 2023-11-02T18:23:44.917 DEBUG:boto:Method: GET 2023-11-02T18:23:44.917 DEBUG:boto:Path: /ibtljn-55/?prefix=a 2023-11-02T18:23:44.918 DEBUG:boto:Data: 2023-11-02T18:23:44.918 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.919 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:44.919 DEBUG:boto:Port: 8000 2023-11-02T18:23:44.920 DEBUG:boto:Params: {} 2023-11-02T18:23:44.920 DEBUG:boto:Token: None 2023-11-02T18:23:44.921 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/ 2023-11-02T18:23:44.921 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/4eBOtjozJ7Mr+oTagkPsp2tLks= 2023-11-02T18:23:44.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/4eBOtjozJ7Mr+oTagkPsp2tLks=', 'Content-Length': '0'} 2023-11-02T18:23:44.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f49949c32527313-006543e930-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.926 DEBUG:boto:b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:44.926 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:23:44.927 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:23:44.927 DEBUG:boto:Method: HEAD 2023-11-02T18:23:44.928 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:23:44.928 DEBUG:boto:Data: 2023-11-02T18:23:44.928 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.929 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:44.929 DEBUG:boto:Port: 8001 2023-11-02T18:23:44.930 DEBUG:boto:Params: {} 2023-11-02T18:23:44.930 DEBUG:boto:Token: None 2023-11-02T18:23:44.931 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/a 2023-11-02T18:23:44.931 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2LnUXSqDDiQbR5kNcRgEQJHtIQQ= 2023-11-02T18:23:44.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2LnUXSqDDiQbR5kNcRgEQJHtIQQ=', 'Content-Length': '0'} 2023-11-02T18:23:44.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5f37cc431434c0-006543e930-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.935 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:23:44.936 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:23:44.936 DEBUG:boto:Method: HEAD 2023-11-02T18:23:44.937 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:23:44.937 DEBUG:boto:Data: 2023-11-02T18:23:44.937 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.938 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:44.938 DEBUG:boto:Port: 8000 2023-11-02T18:23:44.938 DEBUG:boto:Params: {} 2023-11-02T18:23:44.939 DEBUG:boto:Token: None 2023-11-02T18:23:44.939 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/a 2023-11-02T18:23:44.940 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2LnUXSqDDiQbR5kNcRgEQJHtIQQ= 2023-11-02T18:23:44.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2LnUXSqDDiQbR5kNcRgEQJHtIQQ=', 'Content-Length': '0'} 2023-11-02T18:23:44.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fbcd0036b069fc6-006543e930-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:23:44.944 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:23:44.945 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:23:44.945 DEBUG:boto:Method: GET 2023-11-02T18:23:44.945 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:23:44.946 DEBUG:boto:Data: 2023-11-02T18:23:44.946 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.947 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:44.947 DEBUG:boto:Port: 8001 2023-11-02T18:23:44.948 DEBUG:boto:Params: {} 2023-11-02T18:23:44.948 DEBUG:boto:Token: None 2023-11-02T18:23:44.949 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/a 2023-11-02T18:23:44.949 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5EEOW46llURoq4x6cUCrR/ukZc8= 2023-11-02T18:23:44.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5EEOW46llURoq4x6cUCrR/ukZc8=', 'Content-Length': '0'} 2023-11-02T18:23:44.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80a827ef3d36f44-006543e930-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:44.994 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:23:44.995 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:23:44.995 DEBUG:boto:Method: GET 2023-11-02T18:23:44.996 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:23:44.997 DEBUG:boto:Data: 2023-11-02T18:23:44.997 DEBUG:boto:Headers: {} 2023-11-02T18:23:44.997 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:44.998 DEBUG:boto:Port: 8000 2023-11-02T18:23:44.998 DEBUG:boto:Params: {} 2023-11-02T18:23:44.999 DEBUG:boto:Token: None 2023-11-02T18:23:44.999 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:44 GMT /ibtljn-55/a 2023-11-02T18:23:45.000 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5EEOW46llURoq4x6cUCrR/ukZc8= 2023-11-02T18:23:45.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5EEOW46llURoq4x6cUCrR/ukZc8=', 'Content-Length': '0'} 2023-11-02T18:23:45.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008770b7e5d4b1ef4e-006543e930-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.004 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.005 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.005 DEBUG:boto:path=/ibtljn-55/?prefix=a 2023-11-02T18:23:45.006 DEBUG:boto:auth_path=/ibtljn-55/?prefix=a 2023-11-02T18:23:45.006 DEBUG:boto:Method: GET 2023-11-02T18:23:45.007 DEBUG:boto:Path: /ibtljn-55/?prefix=a 2023-11-02T18:23:45.007 DEBUG:boto:Data: 2023-11-02T18:23:45.008 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.008 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.009 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.009 DEBUG:boto:Params: {} 2023-11-02T18:23:45.009 DEBUG:boto:Token: None 2023-11-02T18:23:45.010 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.010 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045a3b2d4e5d5bcf3-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.017 DEBUG:boto:b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.017 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.018 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.018 DEBUG:boto:path=/ibtljn-55/?prefix=a 2023-11-02T18:23:45.019 DEBUG:boto:auth_path=/ibtljn-55/?prefix=a 2023-11-02T18:23:45.019 DEBUG:boto:Method: GET 2023-11-02T18:23:45.020 DEBUG:boto:Path: /ibtljn-55/?prefix=a 2023-11-02T18:23:45.020 DEBUG:boto:Data: 2023-11-02T18:23:45.021 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.021 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.022 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.022 DEBUG:boto:Params: {} 2023-11-02T18:23:45.023 DEBUG:boto:Token: None 2023-11-02T18:23:45.023 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.024 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeb3d1f4046e5383-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.070 DEBUG:boto:b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:23:45.072 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:23:45.072 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:23:45.073 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:23:45.073 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:23:45.074 DEBUG:boto:Method: GET 2023-11-02T18:23:45.074 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:23:45.075 DEBUG:boto:Data: 2023-11-02T18:23:45.075 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.075 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.076 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.076 DEBUG:boto:Params: {} 2023-11-02T18:23:45.077 DEBUG:boto:Token: None 2023-11-02T18:23:45.077 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:23:45.078 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E= 2023-11-02T18:23:45.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E=', 'Content-Length': '0'} 2023-11-02T18:23:45.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beff83737622eafd-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.083 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:23:45.083 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:23:45.084 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:23:45.084 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:23:45.085 DEBUG:boto:Method: GET 2023-11-02T18:23:45.085 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:23:45.086 DEBUG:boto:Data: 2023-11-02T18:23:45.086 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.087 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.087 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.088 DEBUG:boto:Params: {} 2023-11-02T18:23:45.088 DEBUG:boto:Token: None 2023-11-02T18:23:45.089 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:23:45.089 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E= 2023-11-02T18:23:45.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E=', 'Content-Length': '0'} 2023-11-02T18:23:45.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000270350b879446a25-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.094 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:23:45.094 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:23:45.095 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:23:45.095 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:23:45.095 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.096 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:23:45.096 DEBUG:boto:Data: 2023-11-02T18:23:45.097 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.097 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.098 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.098 DEBUG:boto:Params: {} 2023-11-02T18:23:45.098 DEBUG:boto:Token: None 2023-11-02T18:23:45.099 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:23:45.099 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s8QtRkov+rRtxMymyHvibzZXuzA= 2023-11-02T18:23:45.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s8QtRkov+rRtxMymyHvibzZXuzA=', 'Content-Length': '0'} 2023-11-02T18:23:45.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095aff3164e432d16-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.103 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:23:45.104 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:23:45.104 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:23:45.105 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:23:45.105 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.105 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:23:45.106 DEBUG:boto:Data: 2023-11-02T18:23:45.106 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.107 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.107 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.108 DEBUG:boto:Params: {} 2023-11-02T18:23:45.108 DEBUG:boto:Token: None 2023-11-02T18:23:45.109 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:23:45.109 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s8QtRkov+rRtxMymyHvibzZXuzA= 2023-11-02T18:23:45.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s8QtRkov+rRtxMymyHvibzZXuzA=', 'Content-Length': '0'} 2023-11-02T18:23:45.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c86cbbec939bb5ef-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:23:45.114 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:23:45.114 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:23:45.115 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:23:45.115 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:23:45.115 DEBUG:boto:Method: GET 2023-11-02T18:23:45.116 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:23:45.116 DEBUG:boto:Data: 2023-11-02T18:23:45.117 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.117 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.118 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.118 DEBUG:boto:Params: {} 2023-11-02T18:23:45.119 DEBUG:boto:Token: None 2023-11-02T18:23:45.119 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:23:45.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E= 2023-11-02T18:23:45.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E=', 'Content-Length': '0'} 2023-11-02T18:23:45.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5d240e9c6019b65-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.166 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:23:45.167 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:23:45.167 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:23:45.168 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:23:45.168 DEBUG:boto:Method: GET 2023-11-02T18:23:45.169 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:23:45.169 DEBUG:boto:Data: 2023-11-02T18:23:45.170 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.170 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.171 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.171 DEBUG:boto:Params: {} 2023-11-02T18:23:45.172 DEBUG:boto:Token: None 2023-11-02T18:23:45.172 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:23:45.173 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E= 2023-11-02T18:23:45.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E=', 'Content-Length': '0'} 2023-11-02T18:23:45.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8e45923aba1a930-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.178 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.178 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.179 DEBUG:boto:path=/ibtljn-55/?prefix=b 2023-11-02T18:23:45.179 DEBUG:boto:auth_path=/ibtljn-55/?prefix=b 2023-11-02T18:23:45.180 DEBUG:boto:Method: GET 2023-11-02T18:23:45.180 DEBUG:boto:Path: /ibtljn-55/?prefix=b 2023-11-02T18:23:45.180 DEBUG:boto:Data: 2023-11-02T18:23:45.181 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.181 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.182 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.182 DEBUG:boto:Params: {} 2023-11-02T18:23:45.183 DEBUG:boto:Token: None 2023-11-02T18:23:45.183 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.184 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0c9853cb9a6926-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.190 DEBUG:boto:b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.190 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.191 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.191 DEBUG:boto:path=/ibtljn-55/?prefix=b 2023-11-02T18:23:45.192 DEBUG:boto:auth_path=/ibtljn-55/?prefix=b 2023-11-02T18:23:45.192 DEBUG:boto:Method: GET 2023-11-02T18:23:45.193 DEBUG:boto:Path: /ibtljn-55/?prefix=b 2023-11-02T18:23:45.193 DEBUG:boto:Data: 2023-11-02T18:23:45.194 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.194 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.195 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.195 DEBUG:boto:Params: {} 2023-11-02T18:23:45.196 DEBUG:boto:Token: None 2023-11-02T18:23:45.196 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.197 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000369df3f21d7e1545-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.201 DEBUG:boto:b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.202 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:23:45.203 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:23:45.203 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.204 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:23:45.204 DEBUG:boto:Data: 2023-11-02T18:23:45.204 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.205 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.205 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.206 DEBUG:boto:Params: {} 2023-11-02T18:23:45.206 DEBUG:boto:Token: None 2023-11-02T18:23:45.207 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/b 2023-11-02T18:23:45.207 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sMVL5IGZSb8fW0iYYxsbD1T2E9o= 2023-11-02T18:23:45.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sMVL5IGZSb8fW0iYYxsbD1T2E9o=', 'Content-Length': '0'} 2023-11-02T18:23:45.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e74008c69d0d178-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.212 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:23:45.212 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:23:45.212 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.213 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:23:45.213 DEBUG:boto:Data: 2023-11-02T18:23:45.214 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.214 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.215 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.215 DEBUG:boto:Params: {} 2023-11-02T18:23:45.216 DEBUG:boto:Token: None 2023-11-02T18:23:45.216 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/b 2023-11-02T18:23:45.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sMVL5IGZSb8fW0iYYxsbD1T2E9o= 2023-11-02T18:23:45.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sMVL5IGZSb8fW0iYYxsbD1T2E9o=', 'Content-Length': '0'} 2023-11-02T18:23:45.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c613a15f4d6784e-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:23:45.222 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:23:45.222 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:23:45.223 DEBUG:boto:Method: GET 2023-11-02T18:23:45.223 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:23:45.224 DEBUG:boto:Data: 2023-11-02T18:23:45.224 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.225 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.225 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.226 DEBUG:boto:Params: {} 2023-11-02T18:23:45.226 DEBUG:boto:Token: None 2023-11-02T18:23:45.227 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/b 2023-11-02T18:23:45.227 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+QpjtNJDrgekXq2in5hKFPhyXo= 2023-11-02T18:23:45.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+QpjtNJDrgekXq2in5hKFPhyXo=', 'Content-Length': '0'} 2023-11-02T18:23:45.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1351a0d45cc51a-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.274 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:23:45.275 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:23:45.275 DEBUG:boto:Method: GET 2023-11-02T18:23:45.276 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:23:45.276 DEBUG:boto:Data: 2023-11-02T18:23:45.277 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.277 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.277 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.278 DEBUG:boto:Params: {} 2023-11-02T18:23:45.279 DEBUG:boto:Token: None 2023-11-02T18:23:45.279 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/b 2023-11-02T18:23:45.279 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+QpjtNJDrgekXq2in5hKFPhyXo= 2023-11-02T18:23:45.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+QpjtNJDrgekXq2in5hKFPhyXo=', 'Content-Length': '0'} 2023-11-02T18:23:45.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000310845c36817ea9a-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.284 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.284 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.285 DEBUG:boto:path=/ibtljn-55/?prefix=b 2023-11-02T18:23:45.285 DEBUG:boto:auth_path=/ibtljn-55/?prefix=b 2023-11-02T18:23:45.286 DEBUG:boto:Method: GET 2023-11-02T18:23:45.286 DEBUG:boto:Path: /ibtljn-55/?prefix=b 2023-11-02T18:23:45.287 DEBUG:boto:Data: 2023-11-02T18:23:45.287 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.287 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.288 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.288 DEBUG:boto:Params: {} 2023-11-02T18:23:45.289 DEBUG:boto:Token: None 2023-11-02T18:23:45.289 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.290 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000755ffc128bbe157c-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.296 DEBUG:boto:b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.297 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.297 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.297 DEBUG:boto:path=/ibtljn-55/?prefix=b 2023-11-02T18:23:45.298 DEBUG:boto:auth_path=/ibtljn-55/?prefix=b 2023-11-02T18:23:45.298 DEBUG:boto:Method: GET 2023-11-02T18:23:45.299 DEBUG:boto:Path: /ibtljn-55/?prefix=b 2023-11-02T18:23:45.299 DEBUG:boto:Data: 2023-11-02T18:23:45.300 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.300 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.301 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.301 DEBUG:boto:Params: {} 2023-11-02T18:23:45.301 DEBUG:boto:Token: None 2023-11-02T18:23:45.302 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.303 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002362434602d4ebd3-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.350 DEBUG:boto:b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:23:45.352 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:23:45.352 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:23:45.353 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:23:45.353 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:23:45.354 DEBUG:boto:Method: GET 2023-11-02T18:23:45.354 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:23:45.355 DEBUG:boto:Data: 2023-11-02T18:23:45.355 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.356 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.356 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.357 DEBUG:boto:Params: {} 2023-11-02T18:23:45.357 DEBUG:boto:Token: None 2023-11-02T18:23:45.358 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:23:45.358 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s= 2023-11-02T18:23:45.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s=', 'Content-Length': '0'} 2023-11-02T18:23:45.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b33e7fedcb8a89d8-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.363 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:23:45.364 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:23:45.364 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:23:45.365 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:23:45.365 DEBUG:boto:Method: GET 2023-11-02T18:23:45.365 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:23:45.366 DEBUG:boto:Data: 2023-11-02T18:23:45.366 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.367 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.367 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.368 DEBUG:boto:Params: {} 2023-11-02T18:23:45.368 DEBUG:boto:Token: None 2023-11-02T18:23:45.369 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:23:45.369 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s= 2023-11-02T18:23:45.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s=', 'Content-Length': '0'} 2023-11-02T18:23:45.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e96898977e5563d8-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.374 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:23:45.374 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:23:45.374 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:23:45.375 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:23:45.375 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.376 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:23:45.376 DEBUG:boto:Data: 2023-11-02T18:23:45.376 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.377 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.377 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.378 DEBUG:boto:Params: {} 2023-11-02T18:23:45.378 DEBUG:boto:Token: None 2023-11-02T18:23:45.379 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:23:45.379 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aDB6OxN1cjCMQutFXr4i5H/YmV0= 2023-11-02T18:23:45.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aDB6OxN1cjCMQutFXr4i5H/YmV0=', 'Content-Length': '0'} 2023-11-02T18:23:45.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000280b19ff4183ad14-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.383 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:23:45.384 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:23:45.384 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:23:45.385 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:23:45.385 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.385 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:23:45.386 DEBUG:boto:Data: 2023-11-02T18:23:45.386 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.387 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.387 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.388 DEBUG:boto:Params: {} 2023-11-02T18:23:45.388 DEBUG:boto:Token: None 2023-11-02T18:23:45.389 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:23:45.389 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aDB6OxN1cjCMQutFXr4i5H/YmV0= 2023-11-02T18:23:45.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aDB6OxN1cjCMQutFXr4i5H/YmV0=', 'Content-Length': '0'} 2023-11-02T18:23:45.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004891944ff8fb2028-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:23:45.394 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:23:45.394 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:23:45.395 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:23:45.395 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:23:45.396 DEBUG:boto:Method: GET 2023-11-02T18:23:45.396 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:23:45.396 DEBUG:boto:Data: 2023-11-02T18:23:45.397 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.397 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.398 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.398 DEBUG:boto:Params: {} 2023-11-02T18:23:45.399 DEBUG:boto:Token: None 2023-11-02T18:23:45.399 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:23:45.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s= 2023-11-02T18:23:45.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s=', 'Content-Length': '0'} 2023-11-02T18:23:45.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d20be8ec4fa5c1ce-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.446 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:23:45.447 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:23:45.447 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:23:45.448 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:23:45.448 DEBUG:boto:Method: GET 2023-11-02T18:23:45.449 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:23:45.449 DEBUG:boto:Data: 2023-11-02T18:23:45.450 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.450 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.451 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.451 DEBUG:boto:Params: {} 2023-11-02T18:23:45.452 DEBUG:boto:Token: None 2023-11-02T18:23:45.452 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:23:45.453 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s= 2023-11-02T18:23:45.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s=', 'Content-Length': '0'} 2023-11-02T18:23:45.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006921a05818c71584-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.457 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.458 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.458 DEBUG:boto:path=/ibtljn-55/?prefix=c 2023-11-02T18:23:45.459 DEBUG:boto:auth_path=/ibtljn-55/?prefix=c 2023-11-02T18:23:45.459 DEBUG:boto:Method: GET 2023-11-02T18:23:45.459 DEBUG:boto:Path: /ibtljn-55/?prefix=c 2023-11-02T18:23:45.460 DEBUG:boto:Data: 2023-11-02T18:23:45.460 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.461 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.461 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.462 DEBUG:boto:Params: {} 2023-11-02T18:23:45.462 DEBUG:boto:Token: None 2023-11-02T18:23:45.463 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.463 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ceb945f0907386d-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.469 DEBUG:boto:b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.470 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.471 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.471 DEBUG:boto:path=/ibtljn-55/?prefix=c 2023-11-02T18:23:45.472 DEBUG:boto:auth_path=/ibtljn-55/?prefix=c 2023-11-02T18:23:45.472 DEBUG:boto:Method: GET 2023-11-02T18:23:45.473 DEBUG:boto:Path: /ibtljn-55/?prefix=c 2023-11-02T18:23:45.473 DEBUG:boto:Data: 2023-11-02T18:23:45.474 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.474 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.475 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.475 DEBUG:boto:Params: {} 2023-11-02T18:23:45.476 DEBUG:boto:Token: None 2023-11-02T18:23:45.476 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.477 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81b0d132f85de3c-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.481 DEBUG:boto:b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.482 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:23:45.482 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:23:45.483 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.483 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:23:45.484 DEBUG:boto:Data: 2023-11-02T18:23:45.484 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.484 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.485 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.485 DEBUG:boto:Params: {} 2023-11-02T18:23:45.486 DEBUG:boto:Token: None 2023-11-02T18:23:45.486 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/c 2023-11-02T18:23:45.487 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0dhauboEcquEqTNzJho1KKFS5AQ= 2023-11-02T18:23:45.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0dhauboEcquEqTNzJho1KKFS5AQ=', 'Content-Length': '0'} 2023-11-02T18:23:45.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d280a9a6b01864f-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.491 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:23:45.491 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:23:45.491 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.492 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:23:45.492 DEBUG:boto:Data: 2023-11-02T18:23:45.493 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.494 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.494 DEBUG:boto:Params: {} 2023-11-02T18:23:45.495 DEBUG:boto:Token: None 2023-11-02T18:23:45.495 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/c 2023-11-02T18:23:45.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0dhauboEcquEqTNzJho1KKFS5AQ= 2023-11-02T18:23:45.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0dhauboEcquEqTNzJho1KKFS5AQ=', 'Content-Length': '0'} 2023-11-02T18:23:45.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d0bd2f5b25f9f06-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:23:45.500 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:23:45.500 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:23:45.501 DEBUG:boto:Method: GET 2023-11-02T18:23:45.501 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:23:45.501 DEBUG:boto:Data: 2023-11-02T18:23:45.502 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.502 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.503 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.503 DEBUG:boto:Params: {} 2023-11-02T18:23:45.504 DEBUG:boto:Token: None 2023-11-02T18:23:45.504 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/c 2023-11-02T18:23:45.505 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9aOnocZQXER+3ZnWoOp3MuYRyyI= 2023-11-02T18:23:45.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9aOnocZQXER+3ZnWoOp3MuYRyyI=', 'Content-Length': '0'} 2023-11-02T18:23:45.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087d95613fbe174ab-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.509 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:23:45.509 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:23:45.510 DEBUG:boto:Method: GET 2023-11-02T18:23:45.510 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:23:45.511 DEBUG:boto:Data: 2023-11-02T18:23:45.511 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.511 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.512 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.512 DEBUG:boto:Params: {} 2023-11-02T18:23:45.513 DEBUG:boto:Token: None 2023-11-02T18:23:45.513 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/c 2023-11-02T18:23:45.514 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9aOnocZQXER+3ZnWoOp3MuYRyyI= 2023-11-02T18:23:45.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9aOnocZQXER+3ZnWoOp3MuYRyyI=', 'Content-Length': '0'} 2023-11-02T18:23:45.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000865b15b77276136-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.562 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.563 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.563 DEBUG:boto:path=/ibtljn-55/?prefix=c 2023-11-02T18:23:45.564 DEBUG:boto:auth_path=/ibtljn-55/?prefix=c 2023-11-02T18:23:45.564 DEBUG:boto:Method: GET 2023-11-02T18:23:45.565 DEBUG:boto:Path: /ibtljn-55/?prefix=c 2023-11-02T18:23:45.565 DEBUG:boto:Data: 2023-11-02T18:23:45.566 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.567 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.567 DEBUG:boto:Params: {} 2023-11-02T18:23:45.568 DEBUG:boto:Token: None 2023-11-02T18:23:45.568 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1df7b3f46ade27f-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.574 DEBUG:boto:b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.575 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.575 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.576 DEBUG:boto:path=/ibtljn-55/?prefix=c 2023-11-02T18:23:45.576 DEBUG:boto:auth_path=/ibtljn-55/?prefix=c 2023-11-02T18:23:45.577 DEBUG:boto:Method: GET 2023-11-02T18:23:45.577 DEBUG:boto:Path: /ibtljn-55/?prefix=c 2023-11-02T18:23:45.578 DEBUG:boto:Data: 2023-11-02T18:23:45.578 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.579 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.579 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.579 DEBUG:boto:Params: {} 2023-11-02T18:23:45.580 DEBUG:boto:Token: None 2023-11-02T18:23:45.581 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.581 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083e145a5e5c42527-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.586 DEBUG:boto:b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:23:45.588 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:23:45.589 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:23:45.589 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:23:45.589 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:23:45.590 DEBUG:boto:Method: GET 2023-11-02T18:23:45.590 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:23:45.591 DEBUG:boto:Data: 2023-11-02T18:23:45.591 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.592 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.592 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.593 DEBUG:boto:Params: {} 2023-11-02T18:23:45.593 DEBUG:boto:Token: None 2023-11-02T18:23:45.594 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:23:45.594 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q= 2023-11-02T18:23:45.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q=', 'Content-Length': '0'} 2023-11-02T18:23:45.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005655c90f5ab06c22-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.599 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:23:45.599 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:23:45.600 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:23:45.600 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:23:45.601 DEBUG:boto:Method: GET 2023-11-02T18:23:45.601 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:23:45.602 DEBUG:boto:Data: 2023-11-02T18:23:45.602 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.603 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.603 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.603 DEBUG:boto:Params: {} 2023-11-02T18:23:45.604 DEBUG:boto:Token: None 2023-11-02T18:23:45.604 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:23:45.605 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q= 2023-11-02T18:23:45.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q=', 'Content-Length': '0'} 2023-11-02T18:23:45.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761ee09bf1de827f-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.609 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:23:45.609 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:23:45.609 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:23:45.610 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:23:45.610 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.611 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:23:45.611 DEBUG:boto:Data: 2023-11-02T18:23:45.612 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.612 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.613 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.613 DEBUG:boto:Params: {} 2023-11-02T18:23:45.613 DEBUG:boto:Token: None 2023-11-02T18:23:45.614 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:23:45.614 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhisJ7xYy6a7hFxvSyh0Hbinnk8= 2023-11-02T18:23:45.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhisJ7xYy6a7hFxvSyh0Hbinnk8=', 'Content-Length': '0'} 2023-11-02T18:23:45.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d33942175b7ae70-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.617 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:23:45.618 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:23:45.618 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:23:45.619 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:23:45.619 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.620 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:23:45.620 DEBUG:boto:Data: 2023-11-02T18:23:45.620 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.621 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.621 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.622 DEBUG:boto:Params: {} 2023-11-02T18:23:45.622 DEBUG:boto:Token: None 2023-11-02T18:23:45.623 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:23:45.623 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhisJ7xYy6a7hFxvSyh0Hbinnk8= 2023-11-02T18:23:45.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhisJ7xYy6a7hFxvSyh0Hbinnk8=', 'Content-Length': '0'} 2023-11-02T18:23:45.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c8cdd8216139739-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:23:45.627 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:23:45.627 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:23:45.628 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:23:45.628 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:23:45.629 DEBUG:boto:Method: GET 2023-11-02T18:23:45.629 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:23:45.630 DEBUG:boto:Data: 2023-11-02T18:23:45.630 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.631 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.631 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.631 DEBUG:boto:Params: {} 2023-11-02T18:23:45.632 DEBUG:boto:Token: None 2023-11-02T18:23:45.632 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:23:45.633 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q= 2023-11-02T18:23:45.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q=', 'Content-Length': '0'} 2023-11-02T18:23:45.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b006fddc760b695-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.682 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:23:45.683 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:23:45.684 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:23:45.684 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:23:45.684 DEBUG:boto:Method: GET 2023-11-02T18:23:45.685 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:23:45.685 DEBUG:boto:Data: 2023-11-02T18:23:45.686 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.687 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.687 DEBUG:boto:Params: {} 2023-11-02T18:23:45.688 DEBUG:boto:Token: None 2023-11-02T18:23:45.688 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:23:45.689 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q= 2023-11-02T18:23:45.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q=', 'Content-Length': '0'} 2023-11-02T18:23:45.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b28c3bd9df2c9f59-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.693 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.693 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.694 DEBUG:boto:path=/ibtljn-55/?prefix=d 2023-11-02T18:23:45.694 DEBUG:boto:auth_path=/ibtljn-55/?prefix=d 2023-11-02T18:23:45.695 DEBUG:boto:Method: GET 2023-11-02T18:23:45.695 DEBUG:boto:Path: /ibtljn-55/?prefix=d 2023-11-02T18:23:45.696 DEBUG:boto:Data: 2023-11-02T18:23:45.696 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.696 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.697 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.697 DEBUG:boto:Params: {} 2023-11-02T18:23:45.698 DEBUG:boto:Token: None 2023-11-02T18:23:45.698 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.699 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bc1b497e568bc87-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.705 DEBUG:boto:b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.706 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.706 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.707 DEBUG:boto:path=/ibtljn-55/?prefix=d 2023-11-02T18:23:45.707 DEBUG:boto:auth_path=/ibtljn-55/?prefix=d 2023-11-02T18:23:45.708 DEBUG:boto:Method: GET 2023-11-02T18:23:45.708 DEBUG:boto:Path: /ibtljn-55/?prefix=d 2023-11-02T18:23:45.709 DEBUG:boto:Data: 2023-11-02T18:23:45.709 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.710 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.710 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.711 DEBUG:boto:Params: {} 2023-11-02T18:23:45.711 DEBUG:boto:Token: None 2023-11-02T18:23:45.712 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009ec97546fd447c1-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.716 DEBUG:boto:b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.717 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:23:45.718 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:23:45.718 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.719 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:23:45.719 DEBUG:boto:Data: 2023-11-02T18:23:45.720 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.720 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.720 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.721 DEBUG:boto:Params: {} 2023-11-02T18:23:45.721 DEBUG:boto:Token: None 2023-11-02T18:23:45.722 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/d 2023-11-02T18:23:45.722 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VufUPEqhWGiXT6BNWEutto5ODXg= 2023-11-02T18:23:45.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VufUPEqhWGiXT6BNWEutto5ODXg=', 'Content-Length': '0'} 2023-11-02T18:23:45.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1b72434567abc47-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.726 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:23:45.726 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:23:45.727 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.727 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:23:45.728 DEBUG:boto:Data: 2023-11-02T18:23:45.728 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.729 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.729 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.730 DEBUG:boto:Params: {} 2023-11-02T18:23:45.730 DEBUG:boto:Token: None 2023-11-02T18:23:45.731 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/d 2023-11-02T18:23:45.731 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VufUPEqhWGiXT6BNWEutto5ODXg= 2023-11-02T18:23:45.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VufUPEqhWGiXT6BNWEutto5ODXg=', 'Content-Length': '0'} 2023-11-02T18:23:45.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009295ed00179fe3e6-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:23:45.735 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:23:45.736 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:23:45.736 DEBUG:boto:Method: GET 2023-11-02T18:23:45.736 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:23:45.737 DEBUG:boto:Data: 2023-11-02T18:23:45.737 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.738 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.739 DEBUG:boto:Params: {} 2023-11-02T18:23:45.739 DEBUG:boto:Token: None 2023-11-02T18:23:45.740 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/d 2023-11-02T18:23:45.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qCydm+T4m2OeThS7FCpOjNyeGtA= 2023-11-02T18:23:45.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qCydm+T4m2OeThS7FCpOjNyeGtA=', 'Content-Length': '0'} 2023-11-02T18:23:45.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c59731097be6ba6-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.744 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:23:45.745 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:23:45.745 DEBUG:boto:Method: GET 2023-11-02T18:23:45.745 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:23:45.746 DEBUG:boto:Data: 2023-11-02T18:23:45.746 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.747 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.747 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.748 DEBUG:boto:Params: {} 2023-11-02T18:23:45.748 DEBUG:boto:Token: None 2023-11-02T18:23:45.749 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/d 2023-11-02T18:23:45.749 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qCydm+T4m2OeThS7FCpOjNyeGtA= 2023-11-02T18:23:45.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qCydm+T4m2OeThS7FCpOjNyeGtA=', 'Content-Length': '0'} 2023-11-02T18:23:45.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000293a76c8d5af9d0a-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.794 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.795 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.795 DEBUG:boto:path=/ibtljn-55/?prefix=d 2023-11-02T18:23:45.796 DEBUG:boto:auth_path=/ibtljn-55/?prefix=d 2023-11-02T18:23:45.796 DEBUG:boto:Method: GET 2023-11-02T18:23:45.797 DEBUG:boto:Path: /ibtljn-55/?prefix=d 2023-11-02T18:23:45.797 DEBUG:boto:Data: 2023-11-02T18:23:45.798 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.798 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.799 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.799 DEBUG:boto:Params: {} 2023-11-02T18:23:45.800 DEBUG:boto:Token: None 2023-11-02T18:23:45.800 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddc5e7ade0c67644-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.807 DEBUG:boto:b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.808 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.809 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.809 DEBUG:boto:path=/ibtljn-55/?prefix=d 2023-11-02T18:23:45.810 DEBUG:boto:auth_path=/ibtljn-55/?prefix=d 2023-11-02T18:23:45.810 DEBUG:boto:Method: GET 2023-11-02T18:23:45.811 DEBUG:boto:Path: /ibtljn-55/?prefix=d 2023-11-02T18:23:45.811 DEBUG:boto:Data: 2023-11-02T18:23:45.812 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.812 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.813 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.813 DEBUG:boto:Params: {} 2023-11-02T18:23:45.814 DEBUG:boto:Token: None 2023-11-02T18:23:45.814 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.815 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082cd7572f27fdc4e-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.819 DEBUG:boto:b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-11-02T18:23:45.820 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:23:45.821 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:23:45.821 DEBUG:boto:path=/ibtljn-55/e?versionId=null 2023-11-02T18:23:45.822 DEBUG:boto:auth_path=/ibtljn-55/e?versionId=null 2023-11-02T18:23:45.822 DEBUG:boto:Method: GET 2023-11-02T18:23:45.823 DEBUG:boto:Path: /ibtljn-55/e?versionId=null 2023-11-02T18:23:45.823 DEBUG:boto:Data: 2023-11-02T18:23:45.824 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.824 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.825 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.825 DEBUG:boto:Params: {} 2023-11-02T18:23:45.826 DEBUG:boto:Token: None 2023-11-02T18:23:45.826 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/e?versionId=null 2023-11-02T18:23:45.827 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo= 2023-11-02T18:23:45.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo=', 'Content-Length': '0'} 2023-11-02T18:23:45.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbf89dc28e9bf650-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.878 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:23:45.879 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:23:45.879 DEBUG:boto:path=/ibtljn-55/e?versionId=null 2023-11-02T18:23:45.880 DEBUG:boto:auth_path=/ibtljn-55/e?versionId=null 2023-11-02T18:23:45.880 DEBUG:boto:Method: GET 2023-11-02T18:23:45.881 DEBUG:boto:Path: /ibtljn-55/e?versionId=null 2023-11-02T18:23:45.881 DEBUG:boto:Data: 2023-11-02T18:23:45.882 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.882 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.883 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.883 DEBUG:boto:Params: {} 2023-11-02T18:23:45.884 DEBUG:boto:Token: None 2023-11-02T18:23:45.884 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/e?versionId=null 2023-11-02T18:23:45.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo= 2023-11-02T18:23:45.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo=', 'Content-Length': '0'} 2023-11-02T18:23:45.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca0ed20c7dc87cdc-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.890 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:23:45.890 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:23:45.891 DEBUG:boto:path=/ibtljn-55/e?versionId=null 2023-11-02T18:23:45.891 DEBUG:boto:auth_path=/ibtljn-55/e?versionId=null 2023-11-02T18:23:45.892 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.892 DEBUG:boto:Path: /ibtljn-55/e?versionId=null 2023-11-02T18:23:45.892 DEBUG:boto:Data: 2023-11-02T18:23:45.893 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.893 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.894 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.894 DEBUG:boto:Params: {} 2023-11-02T18:23:45.894 DEBUG:boto:Token: None 2023-11-02T18:23:45.895 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/e?versionId=null 2023-11-02T18:23:45.895 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KMINFBuySUK3RWKayVmusWhMaSc= 2023-11-02T18:23:45.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KMINFBuySUK3RWKayVmusWhMaSc=', 'Content-Length': '0'} 2023-11-02T18:23:45.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b4f648105616df3-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.899 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:23:45.900 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:23:45.900 DEBUG:boto:path=/ibtljn-55/e?versionId=null 2023-11-02T18:23:45.901 DEBUG:boto:auth_path=/ibtljn-55/e?versionId=null 2023-11-02T18:23:45.901 DEBUG:boto:Method: HEAD 2023-11-02T18:23:45.901 DEBUG:boto:Path: /ibtljn-55/e?versionId=null 2023-11-02T18:23:45.902 DEBUG:boto:Data: 2023-11-02T18:23:45.902 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.903 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.903 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.904 DEBUG:boto:Params: {} 2023-11-02T18:23:45.904 DEBUG:boto:Token: None 2023-11-02T18:23:45.904 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/e?versionId=null 2023-11-02T18:23:45.905 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KMINFBuySUK3RWKayVmusWhMaSc= 2023-11-02T18:23:45.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KMINFBuySUK3RWKayVmusWhMaSc=', 'Content-Length': '0'} 2023-11-02T18:23:45.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e5053107dbd7c8-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-11-02T18:23:45.909 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:23:45.910 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:23:45.910 DEBUG:boto:path=/ibtljn-55/e?versionId=null 2023-11-02T18:23:45.911 DEBUG:boto:auth_path=/ibtljn-55/e?versionId=null 2023-11-02T18:23:45.911 DEBUG:boto:Method: GET 2023-11-02T18:23:45.911 DEBUG:boto:Path: /ibtljn-55/e?versionId=null 2023-11-02T18:23:45.912 DEBUG:boto:Data: 2023-11-02T18:23:45.913 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.913 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.913 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.914 DEBUG:boto:Params: {} 2023-11-02T18:23:45.914 DEBUG:boto:Token: None 2023-11-02T18:23:45.915 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/e?versionId=null 2023-11-02T18:23:45.915 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo= 2023-11-02T18:23:45.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo=', 'Content-Length': '0'} 2023-11-02T18:23:45.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5fc00b70f4bdd6-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.920 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:23:45.921 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:23:45.921 DEBUG:boto:path=/ibtljn-55/e?versionId=null 2023-11-02T18:23:45.922 DEBUG:boto:auth_path=/ibtljn-55/e?versionId=null 2023-11-02T18:23:45.922 DEBUG:boto:Method: GET 2023-11-02T18:23:45.923 DEBUG:boto:Path: /ibtljn-55/e?versionId=null 2023-11-02T18:23:45.923 DEBUG:boto:Data: 2023-11-02T18:23:45.924 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.924 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.924 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.925 DEBUG:boto:Params: {} 2023-11-02T18:23:45.925 DEBUG:boto:Token: None 2023-11-02T18:23:45.926 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/e?versionId=null 2023-11-02T18:23:45.927 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo= 2023-11-02T18:23:45.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo=', 'Content-Length': '0'} 2023-11-02T18:23:45.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ba4c68b2f3d634-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.931 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.931 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.932 DEBUG:boto:path=/ibtljn-55/?prefix=e 2023-11-02T18:23:45.932 DEBUG:boto:auth_path=/ibtljn-55/?prefix=e 2023-11-02T18:23:45.932 DEBUG:boto:Method: GET 2023-11-02T18:23:45.933 DEBUG:boto:Path: /ibtljn-55/?prefix=e 2023-11-02T18:23:45.933 DEBUG:boto:Data: 2023-11-02T18:23:45.934 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:45.935 DEBUG:boto:Port: 8001 2023-11-02T18:23:45.935 DEBUG:boto:Params: {} 2023-11-02T18:23:45.936 DEBUG:boto:Token: None 2023-11-02T18:23:45.936 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.937 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fa88bc1663ccc8e-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.986 DEBUG:boto:b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.987 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:45.988 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:45.988 DEBUG:boto:path=/ibtljn-55/?prefix=e 2023-11-02T18:23:45.988 DEBUG:boto:auth_path=/ibtljn-55/?prefix=e 2023-11-02T18:23:45.989 DEBUG:boto:Method: GET 2023-11-02T18:23:45.989 DEBUG:boto:Path: /ibtljn-55/?prefix=e 2023-11-02T18:23:45.990 DEBUG:boto:Data: 2023-11-02T18:23:45.990 DEBUG:boto:Headers: {} 2023-11-02T18:23:45.991 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:45.991 DEBUG:boto:Port: 8000 2023-11-02T18:23:45.991 DEBUG:boto:Params: {} 2023-11-02T18:23:45.993 DEBUG:boto:Token: None 2023-11-02T18:23:45.994 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:45 GMT /ibtljn-55/ 2023-11-02T18:23:45.994 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T18:23:45.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T18:23:45.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a651c42beba3da-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:45.999 DEBUG:boto:b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:45.999 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:23:46.000 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:23:46.000 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.001 DEBUG:boto:Path: /ibtljn-55/e 2023-11-02T18:23:46.001 DEBUG:boto:Data: 2023-11-02T18:23:46.001 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.002 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.002 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.003 DEBUG:boto:Params: {} 2023-11-02T18:23:46.003 DEBUG:boto:Token: None 2023-11-02T18:23:46.004 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/e 2023-11-02T18:23:46.004 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:h9PpD4i1qpfzW3GZuon6Mlfc14c= 2023-11-02T18:23:46.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h9PpD4i1qpfzW3GZuon6Mlfc14c=', 'Content-Length': '0'} 2023-11-02T18:23:46.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004153762d179a799c-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.008 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:23:46.008 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:23:46.008 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.009 DEBUG:boto:Path: /ibtljn-55/e 2023-11-02T18:23:46.009 DEBUG:boto:Data: 2023-11-02T18:23:46.010 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.010 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.011 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.011 DEBUG:boto:Params: {} 2023-11-02T18:23:46.011 DEBUG:boto:Token: None 2023-11-02T18:23:46.012 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/e 2023-11-02T18:23:46.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:h9PpD4i1qpfzW3GZuon6Mlfc14c= 2023-11-02T18:23:46.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h9PpD4i1qpfzW3GZuon6Mlfc14c=', 'Content-Length': '0'} 2023-11-02T18:23:46.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fbbf846dc638251-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-11-02T18:23:46.017 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:23:46.017 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:23:46.018 DEBUG:boto:Method: GET 2023-11-02T18:23:46.018 DEBUG:boto:Path: /ibtljn-55/e 2023-11-02T18:23:46.019 DEBUG:boto:Data: 2023-11-02T18:23:46.019 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.020 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.020 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.021 DEBUG:boto:Params: {} 2023-11-02T18:23:46.021 DEBUG:boto:Token: None 2023-11-02T18:23:46.022 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/e 2023-11-02T18:23:46.022 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P7NA20Id1QaZ96le+NgamiQEXLo= 2023-11-02T18:23:46.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P7NA20Id1QaZ96le+NgamiQEXLo=', 'Content-Length': '0'} 2023-11-02T18:23:46.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053658e0213076eb7-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.026 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:23:46.027 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:23:46.027 DEBUG:boto:Method: GET 2023-11-02T18:23:46.028 DEBUG:boto:Path: /ibtljn-55/e 2023-11-02T18:23:46.028 DEBUG:boto:Data: 2023-11-02T18:23:46.029 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.029 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.030 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.030 DEBUG:boto:Params: {} 2023-11-02T18:23:46.031 DEBUG:boto:Token: None 2023-11-02T18:23:46.031 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/e 2023-11-02T18:23:46.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P7NA20Id1QaZ96le+NgamiQEXLo= 2023-11-02T18:23:46.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P7NA20Id1QaZ96le+NgamiQEXLo=', 'Content-Length': '0'} 2023-11-02T18:23:46.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006692644d0fa616b6-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.078 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.079 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.079 DEBUG:boto:path=/ibtljn-55/?prefix=e 2023-11-02T18:23:46.080 DEBUG:boto:auth_path=/ibtljn-55/?prefix=e 2023-11-02T18:23:46.080 DEBUG:boto:Method: GET 2023-11-02T18:23:46.081 DEBUG:boto:Path: /ibtljn-55/?prefix=e 2023-11-02T18:23:46.081 DEBUG:boto:Data: 2023-11-02T18:23:46.082 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.082 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.083 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.083 DEBUG:boto:Params: {} 2023-11-02T18:23:46.084 DEBUG:boto:Token: None 2023-11-02T18:23:46.084 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.084 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045d478a16926d168-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.090 DEBUG:boto:b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.091 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.091 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.092 DEBUG:boto:path=/ibtljn-55/?prefix=e 2023-11-02T18:23:46.092 DEBUG:boto:auth_path=/ibtljn-55/?prefix=e 2023-11-02T18:23:46.093 DEBUG:boto:Method: GET 2023-11-02T18:23:46.093 DEBUG:boto:Path: /ibtljn-55/?prefix=e 2023-11-02T18:23:46.094 DEBUG:boto:Data: 2023-11-02T18:23:46.094 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.094 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.095 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.095 DEBUG:boto:Params: {} 2023-11-02T18:23:46.096 DEBUG:boto:Token: None 2023-11-02T18:23:46.096 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.097 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000210f1a7c48ebc10c-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.101 DEBUG:boto:b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-11-02T18:23:46.102 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:23:46.103 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:23:46.103 DEBUG:boto:path=/ibtljn-55/f?versionId=null 2023-11-02T18:23:46.104 DEBUG:boto:auth_path=/ibtljn-55/f?versionId=null 2023-11-02T18:23:46.104 DEBUG:boto:Method: GET 2023-11-02T18:23:46.104 DEBUG:boto:Path: /ibtljn-55/f?versionId=null 2023-11-02T18:23:46.105 DEBUG:boto:Data: 2023-11-02T18:23:46.105 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.106 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.107 DEBUG:boto:Params: {} 2023-11-02T18:23:46.107 DEBUG:boto:Token: None 2023-11-02T18:23:46.108 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/f?versionId=null 2023-11-02T18:23:46.108 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM= 2023-11-02T18:23:46.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM=', 'Content-Length': '0'} 2023-11-02T18:23:46.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065633ff7186d92a3-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.154 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:23:46.155 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:23:46.156 DEBUG:boto:path=/ibtljn-55/f?versionId=null 2023-11-02T18:23:46.156 DEBUG:boto:auth_path=/ibtljn-55/f?versionId=null 2023-11-02T18:23:46.156 DEBUG:boto:Method: GET 2023-11-02T18:23:46.157 DEBUG:boto:Path: /ibtljn-55/f?versionId=null 2023-11-02T18:23:46.157 DEBUG:boto:Data: 2023-11-02T18:23:46.158 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.158 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.159 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.159 DEBUG:boto:Params: {} 2023-11-02T18:23:46.160 DEBUG:boto:Token: None 2023-11-02T18:23:46.160 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/f?versionId=null 2023-11-02T18:23:46.161 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM= 2023-11-02T18:23:46.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM=', 'Content-Length': '0'} 2023-11-02T18:23:46.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b484262fcdc894f-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.165 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:23:46.165 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:23:46.166 DEBUG:boto:path=/ibtljn-55/f?versionId=null 2023-11-02T18:23:46.166 DEBUG:boto:auth_path=/ibtljn-55/f?versionId=null 2023-11-02T18:23:46.167 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.167 DEBUG:boto:Path: /ibtljn-55/f?versionId=null 2023-11-02T18:23:46.167 DEBUG:boto:Data: 2023-11-02T18:23:46.168 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.168 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.169 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.169 DEBUG:boto:Params: {} 2023-11-02T18:23:46.170 DEBUG:boto:Token: None 2023-11-02T18:23:46.170 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/f?versionId=null 2023-11-02T18:23:46.170 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iQdClzMjWM0C8nDKP3tG1BMS2Nk= 2023-11-02T18:23:46.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iQdClzMjWM0C8nDKP3tG1BMS2Nk=', 'Content-Length': '0'} 2023-11-02T18:23:46.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b3987236ec9f0fd-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.175 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:23:46.175 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:23:46.175 DEBUG:boto:path=/ibtljn-55/f?versionId=null 2023-11-02T18:23:46.176 DEBUG:boto:auth_path=/ibtljn-55/f?versionId=null 2023-11-02T18:23:46.176 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.177 DEBUG:boto:Path: /ibtljn-55/f?versionId=null 2023-11-02T18:23:46.177 DEBUG:boto:Data: 2023-11-02T18:23:46.177 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.178 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.178 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.179 DEBUG:boto:Params: {} 2023-11-02T18:23:46.179 DEBUG:boto:Token: None 2023-11-02T18:23:46.180 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/f?versionId=null 2023-11-02T18:23:46.180 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iQdClzMjWM0C8nDKP3tG1BMS2Nk= 2023-11-02T18:23:46.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iQdClzMjWM0C8nDKP3tG1BMS2Nk=', 'Content-Length': '0'} 2023-11-02T18:23:46.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da90ffdd60dbbf35-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-11-02T18:23:46.184 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:23:46.185 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:23:46.185 DEBUG:boto:path=/ibtljn-55/f?versionId=null 2023-11-02T18:23:46.186 DEBUG:boto:auth_path=/ibtljn-55/f?versionId=null 2023-11-02T18:23:46.186 DEBUG:boto:Method: GET 2023-11-02T18:23:46.187 DEBUG:boto:Path: /ibtljn-55/f?versionId=null 2023-11-02T18:23:46.187 DEBUG:boto:Data: 2023-11-02T18:23:46.188 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.188 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.188 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.189 DEBUG:boto:Params: {} 2023-11-02T18:23:46.189 DEBUG:boto:Token: None 2023-11-02T18:23:46.190 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/f?versionId=null 2023-11-02T18:23:46.190 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM= 2023-11-02T18:23:46.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM=', 'Content-Length': '0'} 2023-11-02T18:23:46.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090d3649409952ee0-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.197 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:23:46.198 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:23:46.198 DEBUG:boto:path=/ibtljn-55/f?versionId=null 2023-11-02T18:23:46.198 DEBUG:boto:auth_path=/ibtljn-55/f?versionId=null 2023-11-02T18:23:46.199 DEBUG:boto:Method: GET 2023-11-02T18:23:46.199 DEBUG:boto:Path: /ibtljn-55/f?versionId=null 2023-11-02T18:23:46.200 DEBUG:boto:Data: 2023-11-02T18:23:46.200 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.201 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.201 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.202 DEBUG:boto:Params: {} 2023-11-02T18:23:46.202 DEBUG:boto:Token: None 2023-11-02T18:23:46.203 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/f?versionId=null 2023-11-02T18:23:46.203 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM= 2023-11-02T18:23:46.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM=', 'Content-Length': '0'} 2023-11-02T18:23:46.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce73a4d85acbd456-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.250 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.251 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.251 DEBUG:boto:path=/ibtljn-55/?prefix=f 2023-11-02T18:23:46.252 DEBUG:boto:auth_path=/ibtljn-55/?prefix=f 2023-11-02T18:23:46.252 DEBUG:boto:Method: GET 2023-11-02T18:23:46.253 DEBUG:boto:Path: /ibtljn-55/?prefix=f 2023-11-02T18:23:46.253 DEBUG:boto:Data: 2023-11-02T18:23:46.254 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.255 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.255 DEBUG:boto:Params: {} 2023-11-02T18:23:46.256 DEBUG:boto:Token: None 2023-11-02T18:23:46.256 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.257 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea5648ab64248694-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.263 DEBUG:boto:b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.263 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.264 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.264 DEBUG:boto:path=/ibtljn-55/?prefix=f 2023-11-02T18:23:46.265 DEBUG:boto:auth_path=/ibtljn-55/?prefix=f 2023-11-02T18:23:46.265 DEBUG:boto:Method: GET 2023-11-02T18:23:46.266 DEBUG:boto:Path: /ibtljn-55/?prefix=f 2023-11-02T18:23:46.266 DEBUG:boto:Data: 2023-11-02T18:23:46.266 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.267 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.267 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.268 DEBUG:boto:Params: {} 2023-11-02T18:23:46.268 DEBUG:boto:Token: None 2023-11-02T18:23:46.269 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.269 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ff01f2abb5c7640-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.273 DEBUG:boto:b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.274 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:23:46.274 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:23:46.275 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.275 DEBUG:boto:Path: /ibtljn-55/f 2023-11-02T18:23:46.276 DEBUG:boto:Data: 2023-11-02T18:23:46.276 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.277 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.277 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.278 DEBUG:boto:Params: {} 2023-11-02T18:23:46.278 DEBUG:boto:Token: None 2023-11-02T18:23:46.279 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/f 2023-11-02T18:23:46.279 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rtcdYEBTOJPW+1jkvrOj5cXN3cw= 2023-11-02T18:23:46.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rtcdYEBTOJPW+1jkvrOj5cXN3cw=', 'Content-Length': '0'} 2023-11-02T18:23:46.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097bddae003e44bf0-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.282 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:23:46.283 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:23:46.283 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.284 DEBUG:boto:Path: /ibtljn-55/f 2023-11-02T18:23:46.284 DEBUG:boto:Data: 2023-11-02T18:23:46.285 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.285 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.285 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.286 DEBUG:boto:Params: {} 2023-11-02T18:23:46.286 DEBUG:boto:Token: None 2023-11-02T18:23:46.287 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/f 2023-11-02T18:23:46.287 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rtcdYEBTOJPW+1jkvrOj5cXN3cw= 2023-11-02T18:23:46.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rtcdYEBTOJPW+1jkvrOj5cXN3cw=', 'Content-Length': '0'} 2023-11-02T18:23:46.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006fc6a7b2f74015e-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-11-02T18:23:46.291 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:23:46.292 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:23:46.292 DEBUG:boto:Method: GET 2023-11-02T18:23:46.293 DEBUG:boto:Path: /ibtljn-55/f 2023-11-02T18:23:46.293 DEBUG:boto:Data: 2023-11-02T18:23:46.294 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.294 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.295 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.295 DEBUG:boto:Params: {} 2023-11-02T18:23:46.296 DEBUG:boto:Token: None 2023-11-02T18:23:46.297 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/f 2023-11-02T18:23:46.297 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z2lPSO8aoobcY5oFTRF8nCfWmCU= 2023-11-02T18:23:46.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z2lPSO8aoobcY5oFTRF8nCfWmCU=', 'Content-Length': '0'} 2023-11-02T18:23:46.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000783684df25c28aa8-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.301 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:23:46.302 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:23:46.302 DEBUG:boto:Method: GET 2023-11-02T18:23:46.303 DEBUG:boto:Path: /ibtljn-55/f 2023-11-02T18:23:46.303 DEBUG:boto:Data: 2023-11-02T18:23:46.304 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.304 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.305 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.305 DEBUG:boto:Params: {} 2023-11-02T18:23:46.305 DEBUG:boto:Token: None 2023-11-02T18:23:46.306 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/f 2023-11-02T18:23:46.306 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z2lPSO8aoobcY5oFTRF8nCfWmCU= 2023-11-02T18:23:46.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z2lPSO8aoobcY5oFTRF8nCfWmCU=', 'Content-Length': '0'} 2023-11-02T18:23:46.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f18afd07f1bdb7f-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.310 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.311 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.311 DEBUG:boto:path=/ibtljn-55/?prefix=f 2023-11-02T18:23:46.312 DEBUG:boto:auth_path=/ibtljn-55/?prefix=f 2023-11-02T18:23:46.312 DEBUG:boto:Method: GET 2023-11-02T18:23:46.313 DEBUG:boto:Path: /ibtljn-55/?prefix=f 2023-11-02T18:23:46.313 DEBUG:boto:Data: 2023-11-02T18:23:46.314 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.314 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.314 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.315 DEBUG:boto:Params: {} 2023-11-02T18:23:46.315 DEBUG:boto:Token: None 2023-11-02T18:23:46.316 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.316 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e7393c8f20b012c-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.322 DEBUG:boto:b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.323 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.323 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.324 DEBUG:boto:path=/ibtljn-55/?prefix=f 2023-11-02T18:23:46.324 DEBUG:boto:auth_path=/ibtljn-55/?prefix=f 2023-11-02T18:23:46.325 DEBUG:boto:Method: GET 2023-11-02T18:23:46.325 DEBUG:boto:Path: /ibtljn-55/?prefix=f 2023-11-02T18:23:46.326 DEBUG:boto:Data: 2023-11-02T18:23:46.326 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.327 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.327 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.328 DEBUG:boto:Params: {} 2023-11-02T18:23:46.328 DEBUG:boto:Token: None 2023-11-02T18:23:46.329 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.329 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8962867d21802a7-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.334 DEBUG:boto:b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-11-02T18:23:46.335 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:23:46.335 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:23:46.336 DEBUG:boto:path=/ibtljn-55/g?versionId=null 2023-11-02T18:23:46.336 DEBUG:boto:auth_path=/ibtljn-55/g?versionId=null 2023-11-02T18:23:46.337 DEBUG:boto:Method: GET 2023-11-02T18:23:46.337 DEBUG:boto:Path: /ibtljn-55/g?versionId=null 2023-11-02T18:23:46.338 DEBUG:boto:Data: 2023-11-02T18:23:46.338 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.339 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.339 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.339 DEBUG:boto:Params: {} 2023-11-02T18:23:46.340 DEBUG:boto:Token: None 2023-11-02T18:23:46.340 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/g?versionId=null 2023-11-02T18:23:46.341 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE= 2023-11-02T18:23:46.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE=', 'Content-Length': '0'} 2023-11-02T18:23:46.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3c6e5f1bf2c138c-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.386 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:23:46.387 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:23:46.387 DEBUG:boto:path=/ibtljn-55/g?versionId=null 2023-11-02T18:23:46.388 DEBUG:boto:auth_path=/ibtljn-55/g?versionId=null 2023-11-02T18:23:46.388 DEBUG:boto:Method: GET 2023-11-02T18:23:46.389 DEBUG:boto:Path: /ibtljn-55/g?versionId=null 2023-11-02T18:23:46.389 DEBUG:boto:Data: 2023-11-02T18:23:46.390 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.390 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.391 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.391 DEBUG:boto:Params: {} 2023-11-02T18:23:46.392 DEBUG:boto:Token: None 2023-11-02T18:23:46.392 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/g?versionId=null 2023-11-02T18:23:46.393 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE= 2023-11-02T18:23:46.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE=', 'Content-Length': '0'} 2023-11-02T18:23:46.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2d991ca08824da0-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.397 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:23:46.398 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:23:46.398 DEBUG:boto:path=/ibtljn-55/g?versionId=null 2023-11-02T18:23:46.399 DEBUG:boto:auth_path=/ibtljn-55/g?versionId=null 2023-11-02T18:23:46.399 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.399 DEBUG:boto:Path: /ibtljn-55/g?versionId=null 2023-11-02T18:23:46.400 DEBUG:boto:Data: 2023-11-02T18:23:46.402 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.402 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.403 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.403 DEBUG:boto:Params: {} 2023-11-02T18:23:46.404 DEBUG:boto:Token: None 2023-11-02T18:23:46.404 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/g?versionId=null 2023-11-02T18:23:46.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:r/FX1ZNLpK2vbFJbue/M20RrlLY= 2023-11-02T18:23:46.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:r/FX1ZNLpK2vbFJbue/M20RrlLY=', 'Content-Length': '0'} 2023-11-02T18:23:46.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003af77e6c313cd0a5-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.409 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:23:46.409 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:23:46.409 DEBUG:boto:path=/ibtljn-55/g?versionId=null 2023-11-02T18:23:46.410 DEBUG:boto:auth_path=/ibtljn-55/g?versionId=null 2023-11-02T18:23:46.410 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.411 DEBUG:boto:Path: /ibtljn-55/g?versionId=null 2023-11-02T18:23:46.411 DEBUG:boto:Data: 2023-11-02T18:23:46.412 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.412 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.413 DEBUG:boto:Params: {} 2023-11-02T18:23:46.413 DEBUG:boto:Token: None 2023-11-02T18:23:46.414 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/g?versionId=null 2023-11-02T18:23:46.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:r/FX1ZNLpK2vbFJbue/M20RrlLY= 2023-11-02T18:23:46.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:r/FX1ZNLpK2vbFJbue/M20RrlLY=', 'Content-Length': '0'} 2023-11-02T18:23:46.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dce58ec352820ae-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-11-02T18:23:46.419 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:23:46.419 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:23:46.420 DEBUG:boto:path=/ibtljn-55/g?versionId=null 2023-11-02T18:23:46.420 DEBUG:boto:auth_path=/ibtljn-55/g?versionId=null 2023-11-02T18:23:46.421 DEBUG:boto:Method: GET 2023-11-02T18:23:46.421 DEBUG:boto:Path: /ibtljn-55/g?versionId=null 2023-11-02T18:23:46.421 DEBUG:boto:Data: 2023-11-02T18:23:46.422 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.423 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.423 DEBUG:boto:Params: {} 2023-11-02T18:23:46.424 DEBUG:boto:Token: None 2023-11-02T18:23:46.424 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/g?versionId=null 2023-11-02T18:23:46.425 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE= 2023-11-02T18:23:46.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE=', 'Content-Length': '0'} 2023-11-02T18:23:46.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a1810ce3bcb583-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.429 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:23:46.429 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:23:46.430 DEBUG:boto:path=/ibtljn-55/g?versionId=null 2023-11-02T18:23:46.431 DEBUG:boto:auth_path=/ibtljn-55/g?versionId=null 2023-11-02T18:23:46.431 DEBUG:boto:Method: GET 2023-11-02T18:23:46.432 DEBUG:boto:Path: /ibtljn-55/g?versionId=null 2023-11-02T18:23:46.432 DEBUG:boto:Data: 2023-11-02T18:23:46.432 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.433 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.433 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.434 DEBUG:boto:Params: {} 2023-11-02T18:23:46.434 DEBUG:boto:Token: None 2023-11-02T18:23:46.435 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/g?versionId=null 2023-11-02T18:23:46.435 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE= 2023-11-02T18:23:46.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE=', 'Content-Length': '0'} 2023-11-02T18:23:46.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e7ad1e1a8e8b8d6-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.439 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.440 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.440 DEBUG:boto:path=/ibtljn-55/?prefix=g 2023-11-02T18:23:46.441 DEBUG:boto:auth_path=/ibtljn-55/?prefix=g 2023-11-02T18:23:46.441 DEBUG:boto:Method: GET 2023-11-02T18:23:46.442 DEBUG:boto:Path: /ibtljn-55/?prefix=g 2023-11-02T18:23:46.442 DEBUG:boto:Data: 2023-11-02T18:23:46.443 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.443 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.444 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.444 DEBUG:boto:Params: {} 2023-11-02T18:23:46.444 DEBUG:boto:Token: None 2023-11-02T18:23:46.445 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.445 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef4184ea2ff7f130-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.498 DEBUG:boto:b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.499 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.500 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.500 DEBUG:boto:path=/ibtljn-55/?prefix=g 2023-11-02T18:23:46.501 DEBUG:boto:auth_path=/ibtljn-55/?prefix=g 2023-11-02T18:23:46.501 DEBUG:boto:Method: GET 2023-11-02T18:23:46.501 DEBUG:boto:Path: /ibtljn-55/?prefix=g 2023-11-02T18:23:46.502 DEBUG:boto:Data: 2023-11-02T18:23:46.502 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.503 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.503 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.504 DEBUG:boto:Params: {} 2023-11-02T18:23:46.504 DEBUG:boto:Token: None 2023-11-02T18:23:46.505 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.505 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a3d870d11e71a82-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.510 DEBUG:boto:b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.511 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:23:46.511 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:23:46.511 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.512 DEBUG:boto:Path: /ibtljn-55/g 2023-11-02T18:23:46.512 DEBUG:boto:Data: 2023-11-02T18:23:46.513 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.513 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.513 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.514 DEBUG:boto:Params: {} 2023-11-02T18:23:46.514 DEBUG:boto:Token: None 2023-11-02T18:23:46.515 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/g 2023-11-02T18:23:46.515 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JCkhL1r3KNkAJucKYlMvOuz+3ys= 2023-11-02T18:23:46.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JCkhL1r3KNkAJucKYlMvOuz+3ys=', 'Content-Length': '0'} 2023-11-02T18:23:46.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027b102328d4b261b-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.519 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:23:46.520 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:23:46.520 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.521 DEBUG:boto:Path: /ibtljn-55/g 2023-11-02T18:23:46.521 DEBUG:boto:Data: 2023-11-02T18:23:46.522 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.522 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.523 DEBUG:boto:Params: {} 2023-11-02T18:23:46.523 DEBUG:boto:Token: None 2023-11-02T18:23:46.524 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/g 2023-11-02T18:23:46.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JCkhL1r3KNkAJucKYlMvOuz+3ys= 2023-11-02T18:23:46.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JCkhL1r3KNkAJucKYlMvOuz+3ys=', 'Content-Length': '0'} 2023-11-02T18:23:46.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094f5e3cbe2a7a14e-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-11-02T18:23:46.529 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:23:46.529 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:23:46.530 DEBUG:boto:Method: GET 2023-11-02T18:23:46.530 DEBUG:boto:Path: /ibtljn-55/g 2023-11-02T18:23:46.531 DEBUG:boto:Data: 2023-11-02T18:23:46.531 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.532 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.533 DEBUG:boto:Params: {} 2023-11-02T18:23:46.533 DEBUG:boto:Token: None 2023-11-02T18:23:46.533 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/g 2023-11-02T18:23:46.534 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AcqfVwEqCcMxmM+ykHd5XCuo00Q= 2023-11-02T18:23:46.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AcqfVwEqCcMxmM+ykHd5XCuo00Q=', 'Content-Length': '0'} 2023-11-02T18:23:46.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000193687b1b133aaf7-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.538 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:23:46.539 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:23:46.539 DEBUG:boto:Method: GET 2023-11-02T18:23:46.540 DEBUG:boto:Path: /ibtljn-55/g 2023-11-02T18:23:46.540 DEBUG:boto:Data: 2023-11-02T18:23:46.540 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.541 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.542 DEBUG:boto:Params: {} 2023-11-02T18:23:46.542 DEBUG:boto:Token: None 2023-11-02T18:23:46.543 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/g 2023-11-02T18:23:46.543 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AcqfVwEqCcMxmM+ykHd5XCuo00Q= 2023-11-02T18:23:46.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AcqfVwEqCcMxmM+ykHd5XCuo00Q=', 'Content-Length': '0'} 2023-11-02T18:23:46.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3cc0009bd009b0a-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.590 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.591 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.591 DEBUG:boto:path=/ibtljn-55/?prefix=g 2023-11-02T18:23:46.592 DEBUG:boto:auth_path=/ibtljn-55/?prefix=g 2023-11-02T18:23:46.592 DEBUG:boto:Method: GET 2023-11-02T18:23:46.592 DEBUG:boto:Path: /ibtljn-55/?prefix=g 2023-11-02T18:23:46.593 DEBUG:boto:Data: 2023-11-02T18:23:46.593 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.594 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.594 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.595 DEBUG:boto:Params: {} 2023-11-02T18:23:46.595 DEBUG:boto:Token: None 2023-11-02T18:23:46.596 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.596 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3379458befd31bf-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.602 DEBUG:boto:b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.605 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.606 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.606 DEBUG:boto:path=/ibtljn-55/?prefix=g 2023-11-02T18:23:46.607 DEBUG:boto:auth_path=/ibtljn-55/?prefix=g 2023-11-02T18:23:46.607 DEBUG:boto:Method: GET 2023-11-02T18:23:46.608 DEBUG:boto:Path: /ibtljn-55/?prefix=g 2023-11-02T18:23:46.608 DEBUG:boto:Data: 2023-11-02T18:23:46.609 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.609 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.610 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.610 DEBUG:boto:Params: {} 2023-11-02T18:23:46.611 DEBUG:boto:Token: None 2023-11-02T18:23:46.611 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045c34c5c9a88b664-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.616 DEBUG:boto:b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-11-02T18:23:46.617 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:23:46.618 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:23:46.618 DEBUG:boto:path=/ibtljn-55/h?versionId=null 2023-11-02T18:23:46.619 DEBUG:boto:auth_path=/ibtljn-55/h?versionId=null 2023-11-02T18:23:46.619 DEBUG:boto:Method: GET 2023-11-02T18:23:46.620 DEBUG:boto:Path: /ibtljn-55/h?versionId=null 2023-11-02T18:23:46.620 DEBUG:boto:Data: 2023-11-02T18:23:46.621 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.621 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.622 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.622 DEBUG:boto:Params: {} 2023-11-02T18:23:46.623 DEBUG:boto:Token: None 2023-11-02T18:23:46.623 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/h?versionId=null 2023-11-02T18:23:46.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk= 2023-11-02T18:23:46.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk=', 'Content-Length': '0'} 2023-11-02T18:23:46.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceb5fbdbe07fc61d-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.670 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:23:46.671 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:23:46.671 DEBUG:boto:path=/ibtljn-55/h?versionId=null 2023-11-02T18:23:46.672 DEBUG:boto:auth_path=/ibtljn-55/h?versionId=null 2023-11-02T18:23:46.672 DEBUG:boto:Method: GET 2023-11-02T18:23:46.673 DEBUG:boto:Path: /ibtljn-55/h?versionId=null 2023-11-02T18:23:46.673 DEBUG:boto:Data: 2023-11-02T18:23:46.673 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.674 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.675 DEBUG:boto:Params: {} 2023-11-02T18:23:46.675 DEBUG:boto:Token: None 2023-11-02T18:23:46.676 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/h?versionId=null 2023-11-02T18:23:46.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk= 2023-11-02T18:23:46.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk=', 'Content-Length': '0'} 2023-11-02T18:23:46.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd3d42c8ace54a77-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.680 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:23:46.681 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:23:46.682 DEBUG:boto:path=/ibtljn-55/h?versionId=null 2023-11-02T18:23:46.683 DEBUG:boto:auth_path=/ibtljn-55/h?versionId=null 2023-11-02T18:23:46.683 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.684 DEBUG:boto:Path: /ibtljn-55/h?versionId=null 2023-11-02T18:23:46.684 DEBUG:boto:Data: 2023-11-02T18:23:46.684 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.685 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.685 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.686 DEBUG:boto:Params: {} 2023-11-02T18:23:46.686 DEBUG:boto:Token: None 2023-11-02T18:23:46.687 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/h?versionId=null 2023-11-02T18:23:46.687 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oGZvOYLtMkqsI9xAhSVbMwXqEdA= 2023-11-02T18:23:46.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oGZvOYLtMkqsI9xAhSVbMwXqEdA=', 'Content-Length': '0'} 2023-11-02T18:23:46.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e44110f7468108a9-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.691 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:23:46.692 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:23:46.692 DEBUG:boto:path=/ibtljn-55/h?versionId=null 2023-11-02T18:23:46.693 DEBUG:boto:auth_path=/ibtljn-55/h?versionId=null 2023-11-02T18:23:46.693 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.693 DEBUG:boto:Path: /ibtljn-55/h?versionId=null 2023-11-02T18:23:46.694 DEBUG:boto:Data: 2023-11-02T18:23:46.694 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.695 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.695 DEBUG:boto:Params: {} 2023-11-02T18:23:46.696 DEBUG:boto:Token: None 2023-11-02T18:23:46.696 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/h?versionId=null 2023-11-02T18:23:46.697 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oGZvOYLtMkqsI9xAhSVbMwXqEdA= 2023-11-02T18:23:46.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oGZvOYLtMkqsI9xAhSVbMwXqEdA=', 'Content-Length': '0'} 2023-11-02T18:23:46.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037bc8aee9798f0e8-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-11-02T18:23:46.701 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:23:46.702 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:23:46.702 DEBUG:boto:path=/ibtljn-55/h?versionId=null 2023-11-02T18:23:46.703 DEBUG:boto:auth_path=/ibtljn-55/h?versionId=null 2023-11-02T18:23:46.703 DEBUG:boto:Method: GET 2023-11-02T18:23:46.703 DEBUG:boto:Path: /ibtljn-55/h?versionId=null 2023-11-02T18:23:46.704 DEBUG:boto:Data: 2023-11-02T18:23:46.704 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.705 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.705 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.706 DEBUG:boto:Params: {} 2023-11-02T18:23:46.706 DEBUG:boto:Token: None 2023-11-02T18:23:46.707 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/h?versionId=null 2023-11-02T18:23:46.707 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk= 2023-11-02T18:23:46.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk=', 'Content-Length': '0'} 2023-11-02T18:23:46.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005605f5faea99ea63-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.711 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:23:46.712 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:23:46.712 DEBUG:boto:path=/ibtljn-55/h?versionId=null 2023-11-02T18:23:46.712 DEBUG:boto:auth_path=/ibtljn-55/h?versionId=null 2023-11-02T18:23:46.713 DEBUG:boto:Method: GET 2023-11-02T18:23:46.713 DEBUG:boto:Path: /ibtljn-55/h?versionId=null 2023-11-02T18:23:46.714 DEBUG:boto:Data: 2023-11-02T18:23:46.714 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.715 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.715 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.715 DEBUG:boto:Params: {} 2023-11-02T18:23:46.716 DEBUG:boto:Token: None 2023-11-02T18:23:46.716 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/h?versionId=null 2023-11-02T18:23:46.717 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk= 2023-11-02T18:23:46.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk=', 'Content-Length': '0'} 2023-11-02T18:23:46.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000990a6ad111ae3bef-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.762 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.763 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.763 DEBUG:boto:path=/ibtljn-55/?prefix=h 2023-11-02T18:23:46.764 DEBUG:boto:auth_path=/ibtljn-55/?prefix=h 2023-11-02T18:23:46.764 DEBUG:boto:Method: GET 2023-11-02T18:23:46.765 DEBUG:boto:Path: /ibtljn-55/?prefix=h 2023-11-02T18:23:46.765 DEBUG:boto:Data: 2023-11-02T18:23:46.766 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.766 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.766 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.767 DEBUG:boto:Params: {} 2023-11-02T18:23:46.767 DEBUG:boto:Token: None 2023-11-02T18:23:46.768 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b0c727c30313812-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.775 DEBUG:boto:b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.776 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.776 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.777 DEBUG:boto:path=/ibtljn-55/?prefix=h 2023-11-02T18:23:46.777 DEBUG:boto:auth_path=/ibtljn-55/?prefix=h 2023-11-02T18:23:46.777 DEBUG:boto:Method: GET 2023-11-02T18:23:46.778 DEBUG:boto:Path: /ibtljn-55/?prefix=h 2023-11-02T18:23:46.778 DEBUG:boto:Data: 2023-11-02T18:23:46.779 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.779 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.780 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.780 DEBUG:boto:Params: {} 2023-11-02T18:23:46.781 DEBUG:boto:Token: None 2023-11-02T18:23:46.781 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.782 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026fbc4791a516142-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.786 DEBUG:boto:b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.787 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:23:46.787 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:23:46.788 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.788 DEBUG:boto:Path: /ibtljn-55/h 2023-11-02T18:23:46.788 DEBUG:boto:Data: 2023-11-02T18:23:46.789 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.789 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.789 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.790 DEBUG:boto:Params: {} 2023-11-02T18:23:46.790 DEBUG:boto:Token: None 2023-11-02T18:23:46.791 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/h 2023-11-02T18:23:46.791 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Kk73kx5+YCQDMSm12AEbrNxOB9s= 2023-11-02T18:23:46.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Kk73kx5+YCQDMSm12AEbrNxOB9s=', 'Content-Length': '0'} 2023-11-02T18:23:46.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6563a91d4e86c6c-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.795 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:23:46.795 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:23:46.796 DEBUG:boto:Method: HEAD 2023-11-02T18:23:46.796 DEBUG:boto:Path: /ibtljn-55/h 2023-11-02T18:23:46.797 DEBUG:boto:Data: 2023-11-02T18:23:46.797 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.798 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.798 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.798 DEBUG:boto:Params: {} 2023-11-02T18:23:46.799 DEBUG:boto:Token: None 2023-11-02T18:23:46.799 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/h 2023-11-02T18:23:46.800 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Kk73kx5+YCQDMSm12AEbrNxOB9s= 2023-11-02T18:23:46.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Kk73kx5+YCQDMSm12AEbrNxOB9s=', 'Content-Length': '0'} 2023-11-02T18:23:46.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b0bf2e3eebf3f4c-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-11-02T18:23:46.804 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:23:46.804 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:23:46.805 DEBUG:boto:Method: GET 2023-11-02T18:23:46.805 DEBUG:boto:Path: /ibtljn-55/h 2023-11-02T18:23:46.806 DEBUG:boto:Data: 2023-11-02T18:23:46.806 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.807 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.810 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.810 DEBUG:boto:Params: {} 2023-11-02T18:23:46.811 DEBUG:boto:Token: None 2023-11-02T18:23:46.812 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/h 2023-11-02T18:23:46.812 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0YZMn7mt34U/cK43a8pohoZ8+0s= 2023-11-02T18:23:46.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0YZMn7mt34U/cK43a8pohoZ8+0s=', 'Content-Length': '0'} 2023-11-02T18:23:46.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080b25521751ece16-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.858 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:23:46.859 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:23:46.859 DEBUG:boto:Method: GET 2023-11-02T18:23:46.860 DEBUG:boto:Path: /ibtljn-55/h 2023-11-02T18:23:46.861 DEBUG:boto:Data: 2023-11-02T18:23:46.861 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.862 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.862 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.863 DEBUG:boto:Params: {} 2023-11-02T18:23:46.863 DEBUG:boto:Token: None 2023-11-02T18:23:46.864 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/h 2023-11-02T18:23:46.864 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0YZMn7mt34U/cK43a8pohoZ8+0s= 2023-11-02T18:23:46.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0YZMn7mt34U/cK43a8pohoZ8+0s=', 'Content-Length': '0'} 2023-11-02T18:23:46.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003857743809c0fd46-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.871 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.871 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.872 DEBUG:boto:path=/ibtljn-55/?prefix=h 2023-11-02T18:23:46.872 DEBUG:boto:auth_path=/ibtljn-55/?prefix=h 2023-11-02T18:23:46.873 DEBUG:boto:Method: GET 2023-11-02T18:23:46.873 DEBUG:boto:Path: /ibtljn-55/?prefix=h 2023-11-02T18:23:46.874 DEBUG:boto:Data: 2023-11-02T18:23:46.874 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.874 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:23:46.875 DEBUG:boto:Port: 8001 2023-11-02T18:23:46.875 DEBUG:boto:Params: {} 2023-11-02T18:23:46.876 DEBUG:boto:Token: None 2023-11-02T18:23:46.876 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045256fcba508c0d8-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.883 DEBUG:boto:b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.884 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:23:46.884 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:23:46.885 DEBUG:boto:path=/ibtljn-55/?prefix=h 2023-11-02T18:23:46.885 DEBUG:boto:auth_path=/ibtljn-55/?prefix=h 2023-11-02T18:23:46.885 DEBUG:boto:Method: GET 2023-11-02T18:23:46.886 DEBUG:boto:Path: /ibtljn-55/?prefix=h 2023-11-02T18:23:46.886 DEBUG:boto:Data: 2023-11-02T18:23:46.887 DEBUG:boto:Headers: {} 2023-11-02T18:23:46.887 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:23:46.888 DEBUG:boto:Port: 8000 2023-11-02T18:23:46.888 DEBUG:boto:Params: {} 2023-11-02T18:23:46.889 DEBUG:boto:Token: None 2023-11-02T18:23:46.889 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:23:46 GMT /ibtljn-55/ 2023-11-02T18:23:46.890 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T18:23:46.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T18:23:46.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076a0ed7ca40d1d69-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:23:46.894 DEBUG:boto:b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:23:46.895 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-55 zones={a2, a1} 2023-11-02T18:23:46.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'ibtljn-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:23:46.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'ibtljn-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:23:46.896 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket ibtljn-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:23:46.943 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:23:46.943 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:23:47.017 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:23:47.018 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-55 2023-11-02T18:23:47.020 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 8 2023-11-02T18:23:47.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:23:47.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:23:47.081 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ibtljn-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:23:47.123 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:23:47.123 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:23:47.207 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:23:47.208 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:23:47.208 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding": "None", 2023-11-02T18:23:47.208 INFO:teuthology.orchestra.run.smithi017.stdout: "current_index": { 2023-11-02T18:23:47.209 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 2, 2023-11-02T18:23:47.209 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:23:47.209 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "Normal", 2023-11-02T18:23:47.209 INFO:teuthology.orchestra.run.smithi017.stdout: "normal": { 2023-11-02T18:23:47.210 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 3, 2023-11-02T18:23:47.210 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:23:47.210 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.210 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:23:47.211 INFO:teuthology.orchestra.run.smithi017.stdout: "logs": [ 2023-11-02T18:23:47.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:23:47.211 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 0, 2023-11-02T18:23:47.211 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:23:47.212 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "InIndex", 2023-11-02T18:23:47.212 INFO:teuthology.orchestra.run.smithi017.stdout: "in_index": { 2023-11-02T18:23:47.212 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 0, 2023-11-02T18:23:47.212 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:23:47.212 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 11, 2023-11-02T18:23:47.213 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:23:47.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:23:47.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:23:47.214 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 1, 2023-11-02T18:23:47.214 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:23:47.214 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "InIndex", 2023-11-02T18:23:47.215 INFO:teuthology.orchestra.run.smithi017.stdout: "in_index": { 2023-11-02T18:23:47.215 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 1, 2023-11-02T18:23:47.215 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:23:47.215 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 3, 2023-11-02T18:23:47.215 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:23:47.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:23:47.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:23:47.217 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 2, 2023-11-02T18:23:47.217 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:23:47.217 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "InIndex", 2023-11-02T18:23:47.218 INFO:teuthology.orchestra.run.smithi017.stdout: "in_index": { 2023-11-02T18:23:47.218 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 2, 2023-11-02T18:23:47.218 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:23:47.218 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 3, 2023-11-02T18:23:47.218 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:23:47.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.219 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:23:47.220 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:23:47.220 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:23:47.221 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-55 2023-11-02T18:23:47.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:23:47.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:23:47.222 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:23:47.316 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:23:47.316 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:23:47.385 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:47.381+0000 7fc40df17f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T18:23:52.386 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:52.380+0000 7fc40df17f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T18:23:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:23:57.380+0000 7fc40df17f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T18:24:02.388 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:02.380+0000 7fc40df17f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T18:24:07.389 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:07.384+0000 7fc40df17f40 1 bucket sync caught up with empty source 2023-11-02T18:24:07.390 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:07.384+0000 7fc40df17f40 0 bucket checkpoint complete 2023-11-02T18:24:07.422 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-55 2023-11-02T18:24:07.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:24:07.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:24:07.423 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:24:07.475 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:24:07.475 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:24:07.548 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:07.540+0000 7f29e4ef5f40 1 bucket sync caught up with source: 2023-11-02T18:24:07.548 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T18:24:07.549 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T18:24:07.549 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:07.540+0000 7f29e4ef5f40 0 bucket checkpoint complete 2023-11-02T18:24:07.574 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-55 zones={a2, a1} 2023-11-02T18:24:07.575 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:07.575 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:07.576 DEBUG:boto:Method: HEAD 2023-11-02T18:24:07.576 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:24:07.577 DEBUG:boto:Data: 2023-11-02T18:24:07.577 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.577 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.578 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.578 DEBUG:boto:Params: {} 2023-11-02T18:24:07.579 DEBUG:boto:Token: None 2023-11-02T18:24:07.579 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/ 2023-11-02T18:24:07.580 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D1uof6ZKvDDUN9XXB1PgrLzCjN8= 2023-11-02T18:24:07.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D1uof6ZKvDDUN9XXB1PgrLzCjN8=', 'Content-Length': '0'} 2023-11-02T18:24:07.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029f444051f6ae6c6-006543e947-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.588 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:07.589 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:07.589 DEBUG:boto:Method: HEAD 2023-11-02T18:24:07.589 DEBUG:boto:Path: /ibtljn-55/ 2023-11-02T18:24:07.590 DEBUG:boto:Data: 2023-11-02T18:24:07.590 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.591 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:07.591 DEBUG:boto:Port: 8000 2023-11-02T18:24:07.591 DEBUG:boto:Params: {} 2023-11-02T18:24:07.592 DEBUG:boto:Token: None 2023-11-02T18:24:07.592 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/ 2023-11-02T18:24:07.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D1uof6ZKvDDUN9XXB1PgrLzCjN8= 2023-11-02T18:24:07.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D1uof6ZKvDDUN9XXB1PgrLzCjN8=', 'Content-Length': '0'} 2023-11-02T18:24:07.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000713a767da58a3529-006543e947-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.596 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:24:07.597 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:07.597 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:07.597 DEBUG:boto:path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:24:07.598 DEBUG:boto:auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:24:07.598 DEBUG:boto:Method: GET 2023-11-02T18:24:07.599 DEBUG:boto:Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T18:24:07.599 DEBUG:boto:Data: 2023-11-02T18:24:07.600 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.600 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.601 DEBUG:boto:Params: {} 2023-11-02T18:24:07.601 DEBUG:boto:Token: None 2023-11-02T18:24:07.602 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/?versions 2023-11-02T18:24:07.602 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM= 2023-11-02T18:24:07.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM=', 'Content-Length': '0'} 2023-11-02T18:24:07.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ab5b148523ecc82-006543e947-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.607 DEBUG:boto:b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:07.608 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:24:07.608 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:24:07.609 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:24:07.609 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:24:07.609 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-11-02T18:24:07.610 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-11-02T18:24:07.610 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-11-02T18:24:07.611 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-11-02T18:24:07.611 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:24:07.612 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:07.612 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:07.612 DEBUG:boto:path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:24:07.613 DEBUG:boto:auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:24:07.613 DEBUG:boto:Method: GET 2023-11-02T18:24:07.614 DEBUG:boto:Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T18:24:07.614 DEBUG:boto:Data: 2023-11-02T18:24:07.614 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.615 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:07.615 DEBUG:boto:Port: 8000 2023-11-02T18:24:07.616 DEBUG:boto:Params: {} 2023-11-02T18:24:07.616 DEBUG:boto:Token: None 2023-11-02T18:24:07.617 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/?versions 2023-11-02T18:24:07.617 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM= 2023-11-02T18:24:07.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM=', 'Content-Length': '0'} 2023-11-02T18:24:07.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fbc73bd7bb4c5a9-006543e947-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.621 DEBUG:boto:b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:07.622 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:24:07.622 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:24:07.623 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:24:07.623 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:24:07.623 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-11-02T18:24:07.624 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-11-02T18:24:07.624 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-11-02T18:24:07.625 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-11-02T18:24:07.625 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:07.625 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:07.626 DEBUG:boto:path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:24:07.626 DEBUG:boto:auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:24:07.627 DEBUG:boto:Method: GET 2023-11-02T18:24:07.627 DEBUG:boto:Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T18:24:07.628 DEBUG:boto:Data: 2023-11-02T18:24:07.628 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.628 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.629 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.629 DEBUG:boto:Params: {} 2023-11-02T18:24:07.630 DEBUG:boto:Token: None 2023-11-02T18:24:07.630 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/?versions 2023-11-02T18:24:07.631 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM= 2023-11-02T18:24:07.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM=', 'Content-Length': '0'} 2023-11-02T18:24:07.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ba524541f542c58-006543e947-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.636 DEBUG:boto:b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:07.637 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:07.637 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:07.638 DEBUG:boto:path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:24:07.638 DEBUG:boto:auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T18:24:07.639 DEBUG:boto:Method: GET 2023-11-02T18:24:07.639 DEBUG:boto:Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T18:24:07.639 DEBUG:boto:Data: 2023-11-02T18:24:07.640 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.640 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:07.641 DEBUG:boto:Port: 8000 2023-11-02T18:24:07.641 DEBUG:boto:Params: {} 2023-11-02T18:24:07.642 DEBUG:boto:Token: None 2023-11-02T18:24:07.642 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/?versions 2023-11-02T18:24:07.643 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM= 2023-11-02T18:24:07.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM=', 'Content-Length': '0'} 2023-11-02T18:24:07.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000695ef0be0aa4853f-006543e947-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.647 DEBUG:boto:b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:07.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:24:07.648 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:24:07.649 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:24:07.649 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:24:07.650 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:24:07.650 DEBUG:boto:Method: GET 2023-11-02T18:24:07.650 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:24:07.651 DEBUG:boto:Data: 2023-11-02T18:24:07.651 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.652 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.652 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.653 DEBUG:boto:Params: {} 2023-11-02T18:24:07.653 DEBUG:boto:Token: None 2023-11-02T18:24:07.654 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:24:07.654 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc= 2023-11-02T18:24:07.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc=', 'Content-Length': '0'} 2023-11-02T18:24:07.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c7e652e2a51d2e2-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.702 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:24:07.703 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:24:07.704 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:24:07.704 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:24:07.704 DEBUG:boto:Method: GET 2023-11-02T18:24:07.705 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:24:07.705 DEBUG:boto:Data: 2023-11-02T18:24:07.706 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.706 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:07.707 DEBUG:boto:Port: 8000 2023-11-02T18:24:07.707 DEBUG:boto:Params: {} 2023-11-02T18:24:07.708 DEBUG:boto:Token: None 2023-11-02T18:24:07.708 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:24:07.709 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc= 2023-11-02T18:24:07.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc=', 'Content-Length': '0'} 2023-11-02T18:24:07.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1087e880536ee17-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.713 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:24:07.713 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:24:07.714 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:24:07.714 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:24:07.715 DEBUG:boto:Method: HEAD 2023-11-02T18:24:07.715 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:24:07.715 DEBUG:boto:Data: 2023-11-02T18:24:07.716 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.716 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.717 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.717 DEBUG:boto:Params: {} 2023-11-02T18:24:07.718 DEBUG:boto:Token: None 2023-11-02T18:24:07.718 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:24:07.718 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OqWSAtn8UWdJHKBnRZkKT8Btuck= 2023-11-02T18:24:07.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OqWSAtn8UWdJHKBnRZkKT8Btuck=', 'Content-Length': '0'} 2023-11-02T18:24:07.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031d5c35a4ac27e13-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.723 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:24:07.723 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:24:07.723 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:24:07.724 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:24:07.724 DEBUG:boto:Method: HEAD 2023-11-02T18:24:07.725 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:24:07.725 DEBUG:boto:Data: 2023-11-02T18:24:07.726 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.726 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:07.726 DEBUG:boto:Port: 8000 2023-11-02T18:24:07.727 DEBUG:boto:Params: {} 2023-11-02T18:24:07.727 DEBUG:boto:Token: None 2023-11-02T18:24:07.728 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:24:07.728 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OqWSAtn8UWdJHKBnRZkKT8Btuck= 2023-11-02T18:24:07.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OqWSAtn8UWdJHKBnRZkKT8Btuck=', 'Content-Length': '0'} 2023-11-02T18:24:07.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e5966aac562f846-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:24:07.732 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:24:07.733 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:24:07.733 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:24:07.734 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:24:07.734 DEBUG:boto:Method: GET 2023-11-02T18:24:07.735 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:24:07.735 DEBUG:boto:Data: 2023-11-02T18:24:07.736 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.736 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.736 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.737 DEBUG:boto:Params: {} 2023-11-02T18:24:07.737 DEBUG:boto:Token: None 2023-11-02T18:24:07.738 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:24:07.738 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc= 2023-11-02T18:24:07.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc=', 'Content-Length': '0'} 2023-11-02T18:24:07.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f002a7423c5ee766-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.742 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:24:07.743 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:24:07.743 DEBUG:boto:path=/ibtljn-55/a?versionId=null 2023-11-02T18:24:07.744 DEBUG:boto:auth_path=/ibtljn-55/a?versionId=null 2023-11-02T18:24:07.744 DEBUG:boto:Method: GET 2023-11-02T18:24:07.744 DEBUG:boto:Path: /ibtljn-55/a?versionId=null 2023-11-02T18:24:07.745 DEBUG:boto:Data: 2023-11-02T18:24:07.745 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.746 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:07.746 DEBUG:boto:Port: 8000 2023-11-02T18:24:07.746 DEBUG:boto:Params: {} 2023-11-02T18:24:07.747 DEBUG:boto:Token: None 2023-11-02T18:24:07.747 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/a?versionId=null 2023-11-02T18:24:07.748 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc= 2023-11-02T18:24:07.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc=', 'Content-Length': '0'} 2023-11-02T18:24:07.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006010289309709e45-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.798 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:07.799 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:07.799 DEBUG:boto:path=/ibtljn-55/?prefix=a 2023-11-02T18:24:07.800 DEBUG:boto:auth_path=/ibtljn-55/?prefix=a 2023-11-02T18:24:07.800 DEBUG:boto:Method: GET 2023-11-02T18:24:07.801 DEBUG:boto:Path: /ibtljn-55/?prefix=a 2023-11-02T18:24:07.801 DEBUG:boto:Data: 2023-11-02T18:24:07.801 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.802 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.802 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.803 DEBUG:boto:Params: {} 2023-11-02T18:24:07.803 DEBUG:boto:Token: None 2023-11-02T18:24:07.804 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/ 2023-11-02T18:24:07.804 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw= 2023-11-02T18:24:07.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw=', 'Content-Length': '0'} 2023-11-02T18:24:07.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000104e148aab1521e8-006543e947-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.810 DEBUG:boto:b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:07.811 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:07.812 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:07.812 DEBUG:boto:path=/ibtljn-55/?prefix=a 2023-11-02T18:24:07.812 DEBUG:boto:auth_path=/ibtljn-55/?prefix=a 2023-11-02T18:24:07.813 DEBUG:boto:Method: GET 2023-11-02T18:24:07.813 DEBUG:boto:Path: /ibtljn-55/?prefix=a 2023-11-02T18:24:07.814 DEBUG:boto:Data: 2023-11-02T18:24:07.814 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.815 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:07.815 DEBUG:boto:Port: 8000 2023-11-02T18:24:07.816 DEBUG:boto:Params: {} 2023-11-02T18:24:07.816 DEBUG:boto:Token: None 2023-11-02T18:24:07.817 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/ 2023-11-02T18:24:07.817 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw= 2023-11-02T18:24:07.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw=', 'Content-Length': '0'} 2023-11-02T18:24:07.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d360306bd298ac86-006543e947-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.821 DEBUG:boto:b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:07.822 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:24:07.822 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:24:07.823 DEBUG:boto:Method: HEAD 2023-11-02T18:24:07.823 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:24:07.824 DEBUG:boto:Data: 2023-11-02T18:24:07.824 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.825 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.825 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.825 DEBUG:boto:Params: {} 2023-11-02T18:24:07.826 DEBUG:boto:Token: None 2023-11-02T18:24:07.826 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/a 2023-11-02T18:24:07.827 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zLYVvIEAfo3mVe8DtH4MRmLyNa4= 2023-11-02T18:24:07.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zLYVvIEAfo3mVe8DtH4MRmLyNa4=', 'Content-Length': '0'} 2023-11-02T18:24:07.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e27b4d50903e6cc-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.831 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:24:07.831 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:24:07.832 DEBUG:boto:Method: HEAD 2023-11-02T18:24:07.832 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:24:07.832 DEBUG:boto:Data: 2023-11-02T18:24:07.833 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.833 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:07.834 DEBUG:boto:Port: 8000 2023-11-02T18:24:07.834 DEBUG:boto:Params: {} 2023-11-02T18:24:07.835 DEBUG:boto:Token: None 2023-11-02T18:24:07.835 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/a 2023-11-02T18:24:07.836 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zLYVvIEAfo3mVe8DtH4MRmLyNa4= 2023-11-02T18:24:07.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zLYVvIEAfo3mVe8DtH4MRmLyNa4=', 'Content-Length': '0'} 2023-11-02T18:24:07.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc1bd3249d50a69-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:24:07.840 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:24:07.840 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:24:07.841 DEBUG:boto:Method: GET 2023-11-02T18:24:07.841 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:24:07.841 DEBUG:boto:Data: 2023-11-02T18:24:07.842 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.842 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.843 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.843 DEBUG:boto:Params: {} 2023-11-02T18:24:07.844 DEBUG:boto:Token: None 2023-11-02T18:24:07.844 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/a 2023-11-02T18:24:07.845 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9ZvaPMigGoNKNNEfCazQU1Op018= 2023-11-02T18:24:07.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9ZvaPMigGoNKNNEfCazQU1Op018=', 'Content-Length': '0'} 2023-11-02T18:24:07.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba8f92bd65be584-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.894 DEBUG:boto:path=/ibtljn-55/a 2023-11-02T18:24:07.895 DEBUG:boto:auth_path=/ibtljn-55/a 2023-11-02T18:24:07.895 DEBUG:boto:Method: GET 2023-11-02T18:24:07.896 DEBUG:boto:Path: /ibtljn-55/a 2023-11-02T18:24:07.896 DEBUG:boto:Data: 2023-11-02T18:24:07.897 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.897 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:07.897 DEBUG:boto:Port: 8000 2023-11-02T18:24:07.898 DEBUG:boto:Params: {} 2023-11-02T18:24:07.898 DEBUG:boto:Token: None 2023-11-02T18:24:07.899 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/a 2023-11-02T18:24:07.900 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9ZvaPMigGoNKNNEfCazQU1Op018= 2023-11-02T18:24:07.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9ZvaPMigGoNKNNEfCazQU1Op018=', 'Content-Length': '0'} 2023-11-02T18:24:07.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005847dc4aca816fd4-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.904 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:07.904 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:07.905 DEBUG:boto:path=/ibtljn-55/?prefix=a 2023-11-02T18:24:07.905 DEBUG:boto:auth_path=/ibtljn-55/?prefix=a 2023-11-02T18:24:07.906 DEBUG:boto:Method: GET 2023-11-02T18:24:07.906 DEBUG:boto:Path: /ibtljn-55/?prefix=a 2023-11-02T18:24:07.907 DEBUG:boto:Data: 2023-11-02T18:24:07.907 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.907 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.908 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.908 DEBUG:boto:Params: {} 2023-11-02T18:24:07.909 DEBUG:boto:Token: None 2023-11-02T18:24:07.909 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/ 2023-11-02T18:24:07.910 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw= 2023-11-02T18:24:07.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw=', 'Content-Length': '0'} 2023-11-02T18:24:07.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000431d7ee7fe78b280-006543e947-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.918 DEBUG:boto:b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:07.919 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:07.920 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:07.920 DEBUG:boto:path=/ibtljn-55/?prefix=a 2023-11-02T18:24:07.920 DEBUG:boto:auth_path=/ibtljn-55/?prefix=a 2023-11-02T18:24:07.921 DEBUG:boto:Method: GET 2023-11-02T18:24:07.921 DEBUG:boto:Path: /ibtljn-55/?prefix=a 2023-11-02T18:24:07.922 DEBUG:boto:Data: 2023-11-02T18:24:07.922 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.923 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:07.923 DEBUG:boto:Port: 8000 2023-11-02T18:24:07.924 DEBUG:boto:Params: {} 2023-11-02T18:24:07.924 DEBUG:boto:Token: None 2023-11-02T18:24:07.925 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/ 2023-11-02T18:24:07.925 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw= 2023-11-02T18:24:07.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw=', 'Content-Length': '0'} 2023-11-02T18:24:07.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000861118878080c2ba-006543e947-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.930 DEBUG:boto:b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:07.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:24:07.931 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:24:07.931 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:24:07.932 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:24:07.932 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:24:07.933 DEBUG:boto:Method: GET 2023-11-02T18:24:07.933 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:24:07.934 DEBUG:boto:Data: 2023-11-02T18:24:07.934 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.935 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.935 DEBUG:boto:Params: {} 2023-11-02T18:24:07.936 DEBUG:boto:Token: None 2023-11-02T18:24:07.936 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:24:07.937 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PtM0taec9K1xsfdA+le+ESOf4VE= 2023-11-02T18:24:07.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PtM0taec9K1xsfdA+le+ESOf4VE=', 'Content-Length': '0'} 2023-11-02T18:24:07.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022875dea40308a5a-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.941 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:24:07.941 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:24:07.942 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:24:07.942 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:24:07.943 DEBUG:boto:Method: GET 2023-11-02T18:24:07.943 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:24:07.944 DEBUG:boto:Data: 2023-11-02T18:24:07.944 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.945 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:07.945 DEBUG:boto:Port: 8000 2023-11-02T18:24:07.945 DEBUG:boto:Params: {} 2023-11-02T18:24:07.946 DEBUG:boto:Token: None 2023-11-02T18:24:07.946 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:24:07.947 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PtM0taec9K1xsfdA+le+ESOf4VE= 2023-11-02T18:24:07.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PtM0taec9K1xsfdA+le+ESOf4VE=', 'Content-Length': '0'} 2023-11-02T18:24:07.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b5853339a1d0d53-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.951 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:24:07.951 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:24:07.952 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:24:07.952 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:24:07.953 DEBUG:boto:Method: HEAD 2023-11-02T18:24:07.953 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:24:07.953 DEBUG:boto:Data: 2023-11-02T18:24:07.954 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.954 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.955 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.955 DEBUG:boto:Params: {} 2023-11-02T18:24:07.956 DEBUG:boto:Token: None 2023-11-02T18:24:07.956 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:24:07.957 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2bNCv/m/y4a2JcvkQwGCIh0eSOA= 2023-11-02T18:24:07.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2bNCv/m/y4a2JcvkQwGCIh0eSOA=', 'Content-Length': '0'} 2023-11-02T18:24:07.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aec6cc9a402db99-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.960 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:24:07.961 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:24:07.961 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:24:07.962 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:24:07.962 DEBUG:boto:Method: HEAD 2023-11-02T18:24:07.963 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:24:07.963 DEBUG:boto:Data: 2023-11-02T18:24:07.963 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.964 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:07.964 DEBUG:boto:Port: 8000 2023-11-02T18:24:07.965 DEBUG:boto:Params: {} 2023-11-02T18:24:07.965 DEBUG:boto:Token: None 2023-11-02T18:24:07.966 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:24:07.966 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2bNCv/m/y4a2JcvkQwGCIh0eSOA= 2023-11-02T18:24:07.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2bNCv/m/y4a2JcvkQwGCIh0eSOA=', 'Content-Length': '0'} 2023-11-02T18:24:07.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f48dba733764499-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:07.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:24:07.971 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:24:07.971 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:24:07.972 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:24:07.972 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:24:07.972 DEBUG:boto:Method: GET 2023-11-02T18:24:07.973 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:24:07.973 DEBUG:boto:Data: 2023-11-02T18:24:07.974 DEBUG:boto:Headers: {} 2023-11-02T18:24:07.974 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:07.975 DEBUG:boto:Port: 8001 2023-11-02T18:24:07.975 DEBUG:boto:Params: {} 2023-11-02T18:24:07.975 DEBUG:boto:Token: None 2023-11-02T18:24:07.976 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:07 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:24:07.977 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PtM0taec9K1xsfdA+le+ESOf4VE= 2023-11-02T18:24:07.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PtM0taec9K1xsfdA+le+ESOf4VE=', 'Content-Length': '0'} 2023-11-02T18:24:07.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce4776aeb266fe04-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.022 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:24:08.023 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:24:08.023 DEBUG:boto:path=/ibtljn-55/b?versionId=null 2023-11-02T18:24:08.024 DEBUG:boto:auth_path=/ibtljn-55/b?versionId=null 2023-11-02T18:24:08.024 DEBUG:boto:Method: GET 2023-11-02T18:24:08.025 DEBUG:boto:Path: /ibtljn-55/b?versionId=null 2023-11-02T18:24:08.025 DEBUG:boto:Data: 2023-11-02T18:24:08.026 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.026 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.027 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.027 DEBUG:boto:Params: {} 2023-11-02T18:24:08.028 DEBUG:boto:Token: None 2023-11-02T18:24:08.028 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/b?versionId=null 2023-11-02T18:24:08.029 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YydcZheNKNJEivvCzNQAJNkzMFY= 2023-11-02T18:24:08.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YydcZheNKNJEivvCzNQAJNkzMFY=', 'Content-Length': '0'} 2023-11-02T18:24:08.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd03752a67f87372-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.033 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.033 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.034 DEBUG:boto:path=/ibtljn-55/?prefix=b 2023-11-02T18:24:08.034 DEBUG:boto:auth_path=/ibtljn-55/?prefix=b 2023-11-02T18:24:08.035 DEBUG:boto:Method: GET 2023-11-02T18:24:08.035 DEBUG:boto:Path: /ibtljn-55/?prefix=b 2023-11-02T18:24:08.036 DEBUG:boto:Data: 2023-11-02T18:24:08.036 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.037 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.037 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.037 DEBUG:boto:Params: {} 2023-11-02T18:24:08.038 DEBUG:boto:Token: None 2023-11-02T18:24:08.038 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.039 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000575931eb645550c4-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.045 DEBUG:boto:b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.046 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.046 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.047 DEBUG:boto:path=/ibtljn-55/?prefix=b 2023-11-02T18:24:08.047 DEBUG:boto:auth_path=/ibtljn-55/?prefix=b 2023-11-02T18:24:08.048 DEBUG:boto:Method: GET 2023-11-02T18:24:08.048 DEBUG:boto:Path: /ibtljn-55/?prefix=b 2023-11-02T18:24:08.048 DEBUG:boto:Data: 2023-11-02T18:24:08.049 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.049 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.050 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.050 DEBUG:boto:Params: {} 2023-11-02T18:24:08.051 DEBUG:boto:Token: None 2023-11-02T18:24:08.051 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.052 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a00345266d57204d-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.056 DEBUG:boto:b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.057 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:24:08.057 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:24:08.058 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.058 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:24:08.058 DEBUG:boto:Data: 2023-11-02T18:24:08.059 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.059 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.060 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.060 DEBUG:boto:Params: {} 2023-11-02T18:24:08.060 DEBUG:boto:Token: None 2023-11-02T18:24:08.061 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/b 2023-11-02T18:24:08.061 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:svPZmLgFeM2S+QTVeBGSQY4xhLI= 2023-11-02T18:24:08.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:svPZmLgFeM2S+QTVeBGSQY4xhLI=', 'Content-Length': '0'} 2023-11-02T18:24:08.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f173d69912501d07-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.065 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:24:08.066 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:24:08.066 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.067 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:24:08.067 DEBUG:boto:Data: 2023-11-02T18:24:08.067 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.068 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.068 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.069 DEBUG:boto:Params: {} 2023-11-02T18:24:08.069 DEBUG:boto:Token: None 2023-11-02T18:24:08.069 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/b 2023-11-02T18:24:08.070 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:svPZmLgFeM2S+QTVeBGSQY4xhLI= 2023-11-02T18:24:08.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:svPZmLgFeM2S+QTVeBGSQY4xhLI=', 'Content-Length': '0'} 2023-11-02T18:24:08.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002313e8e9d3f17745-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:24:08.074 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:24:08.075 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:24:08.075 DEBUG:boto:Method: GET 2023-11-02T18:24:08.076 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:24:08.076 DEBUG:boto:Data: 2023-11-02T18:24:08.077 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.077 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.077 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.078 DEBUG:boto:Params: {} 2023-11-02T18:24:08.078 DEBUG:boto:Token: None 2023-11-02T18:24:08.079 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/b 2023-11-02T18:24:08.079 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CgaudSY6A/l3oxYzXyeGELhrzqY= 2023-11-02T18:24:08.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CgaudSY6A/l3oxYzXyeGELhrzqY=', 'Content-Length': '0'} 2023-11-02T18:24:08.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008604b393a542b825-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.083 DEBUG:boto:path=/ibtljn-55/b 2023-11-02T18:24:08.084 DEBUG:boto:auth_path=/ibtljn-55/b 2023-11-02T18:24:08.084 DEBUG:boto:Method: GET 2023-11-02T18:24:08.085 DEBUG:boto:Path: /ibtljn-55/b 2023-11-02T18:24:08.085 DEBUG:boto:Data: 2023-11-02T18:24:08.086 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.086 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.087 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.087 DEBUG:boto:Params: {} 2023-11-02T18:24:08.087 DEBUG:boto:Token: None 2023-11-02T18:24:08.088 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/b 2023-11-02T18:24:08.088 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CgaudSY6A/l3oxYzXyeGELhrzqY= 2023-11-02T18:24:08.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CgaudSY6A/l3oxYzXyeGELhrzqY=', 'Content-Length': '0'} 2023-11-02T18:24:08.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b618e95362bd094-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.093 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.093 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.094 DEBUG:boto:path=/ibtljn-55/?prefix=b 2023-11-02T18:24:08.094 DEBUG:boto:auth_path=/ibtljn-55/?prefix=b 2023-11-02T18:24:08.094 DEBUG:boto:Method: GET 2023-11-02T18:24:08.095 DEBUG:boto:Path: /ibtljn-55/?prefix=b 2023-11-02T18:24:08.095 DEBUG:boto:Data: 2023-11-02T18:24:08.096 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.097 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.097 DEBUG:boto:Params: {} 2023-11-02T18:24:08.097 DEBUG:boto:Token: None 2023-11-02T18:24:08.098 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5443adacfdae205-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.150 DEBUG:boto:b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.151 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.152 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.152 DEBUG:boto:path=/ibtljn-55/?prefix=b 2023-11-02T18:24:08.153 DEBUG:boto:auth_path=/ibtljn-55/?prefix=b 2023-11-02T18:24:08.153 DEBUG:boto:Method: GET 2023-11-02T18:24:08.154 DEBUG:boto:Path: /ibtljn-55/?prefix=b 2023-11-02T18:24:08.154 DEBUG:boto:Data: 2023-11-02T18:24:08.154 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.155 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.155 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.156 DEBUG:boto:Params: {} 2023-11-02T18:24:08.156 DEBUG:boto:Token: None 2023-11-02T18:24:08.157 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.157 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000542cccef41d328ab-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.162 DEBUG:boto:b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:24:08.163 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:24:08.163 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:24:08.164 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:24:08.164 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:24:08.165 DEBUG:boto:Method: GET 2023-11-02T18:24:08.165 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:24:08.166 DEBUG:boto:Data: 2023-11-02T18:24:08.166 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.166 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.167 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.167 DEBUG:boto:Params: {} 2023-11-02T18:24:08.168 DEBUG:boto:Token: None 2023-11-02T18:24:08.168 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:24:08.169 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko= 2023-11-02T18:24:08.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko=', 'Content-Length': '0'} 2023-11-02T18:24:08.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c01c17fe68693ef6-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.173 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:24:08.174 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:24:08.174 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:24:08.175 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:24:08.176 DEBUG:boto:Method: GET 2023-11-02T18:24:08.177 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:24:08.177 DEBUG:boto:Data: 2023-11-02T18:24:08.178 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.179 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.180 DEBUG:boto:Params: {} 2023-11-02T18:24:08.180 DEBUG:boto:Token: None 2023-11-02T18:24:08.181 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:24:08.181 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko= 2023-11-02T18:24:08.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko=', 'Content-Length': '0'} 2023-11-02T18:24:08.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff05984b94a5873-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.187 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:24:08.187 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:24:08.188 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:24:08.189 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:24:08.189 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.190 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:24:08.190 DEBUG:boto:Data: 2023-11-02T18:24:08.191 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.191 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.192 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.192 DEBUG:boto:Params: {} 2023-11-02T18:24:08.193 DEBUG:boto:Token: None 2023-11-02T18:24:08.193 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:24:08.194 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dJooGhNZZPC2CNArUsW9wiaxcoc= 2023-11-02T18:24:08.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dJooGhNZZPC2CNArUsW9wiaxcoc=', 'Content-Length': '0'} 2023-11-02T18:24:08.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005513f1bb4c783090-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.198 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:24:08.199 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:24:08.199 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:24:08.199 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:24:08.200 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.200 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:24:08.201 DEBUG:boto:Data: 2023-11-02T18:24:08.201 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.201 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.202 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.202 DEBUG:boto:Params: {} 2023-11-02T18:24:08.203 DEBUG:boto:Token: None 2023-11-02T18:24:08.204 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:24:08.204 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dJooGhNZZPC2CNArUsW9wiaxcoc= 2023-11-02T18:24:08.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dJooGhNZZPC2CNArUsW9wiaxcoc=', 'Content-Length': '0'} 2023-11-02T18:24:08.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c943ba09d5f0bb1-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:24:08.209 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:24:08.209 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:24:08.210 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:24:08.210 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:24:08.211 DEBUG:boto:Method: GET 2023-11-02T18:24:08.211 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:24:08.211 DEBUG:boto:Data: 2023-11-02T18:24:08.212 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.212 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.213 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.213 DEBUG:boto:Params: {} 2023-11-02T18:24:08.214 DEBUG:boto:Token: None 2023-11-02T18:24:08.214 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:24:08.215 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko= 2023-11-02T18:24:08.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko=', 'Content-Length': '0'} 2023-11-02T18:24:08.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bd817d340a992da-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.219 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:24:08.219 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:24:08.220 DEBUG:boto:path=/ibtljn-55/c?versionId=null 2023-11-02T18:24:08.220 DEBUG:boto:auth_path=/ibtljn-55/c?versionId=null 2023-11-02T18:24:08.221 DEBUG:boto:Method: GET 2023-11-02T18:24:08.221 DEBUG:boto:Path: /ibtljn-55/c?versionId=null 2023-11-02T18:24:08.221 DEBUG:boto:Data: 2023-11-02T18:24:08.222 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.222 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.223 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.223 DEBUG:boto:Params: {} 2023-11-02T18:24:08.224 DEBUG:boto:Token: None 2023-11-02T18:24:08.224 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/c?versionId=null 2023-11-02T18:24:08.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko= 2023-11-02T18:24:08.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko=', 'Content-Length': '0'} 2023-11-02T18:24:08.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587de4470035b2e3-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.274 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.275 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.276 DEBUG:boto:path=/ibtljn-55/?prefix=c 2023-11-02T18:24:08.276 DEBUG:boto:auth_path=/ibtljn-55/?prefix=c 2023-11-02T18:24:08.276 DEBUG:boto:Method: GET 2023-11-02T18:24:08.277 DEBUG:boto:Path: /ibtljn-55/?prefix=c 2023-11-02T18:24:08.277 DEBUG:boto:Data: 2023-11-02T18:24:08.278 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.278 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.279 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.279 DEBUG:boto:Params: {} 2023-11-02T18:24:08.279 DEBUG:boto:Token: None 2023-11-02T18:24:08.280 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.280 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bda2044ab9207c2-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.286 DEBUG:boto:b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.287 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.287 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.288 DEBUG:boto:path=/ibtljn-55/?prefix=c 2023-11-02T18:24:08.288 DEBUG:boto:auth_path=/ibtljn-55/?prefix=c 2023-11-02T18:24:08.289 DEBUG:boto:Method: GET 2023-11-02T18:24:08.289 DEBUG:boto:Path: /ibtljn-55/?prefix=c 2023-11-02T18:24:08.290 DEBUG:boto:Data: 2023-11-02T18:24:08.290 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.291 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.291 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.291 DEBUG:boto:Params: {} 2023-11-02T18:24:08.292 DEBUG:boto:Token: None 2023-11-02T18:24:08.292 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.293 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005af5147a3453fece-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.297 DEBUG:boto:b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.297 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:24:08.298 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:24:08.298 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.299 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:24:08.299 DEBUG:boto:Data: 2023-11-02T18:24:08.300 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.300 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.300 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.301 DEBUG:boto:Params: {} 2023-11-02T18:24:08.301 DEBUG:boto:Token: None 2023-11-02T18:24:08.302 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/c 2023-11-02T18:24:08.302 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/ZG6GT72iKVkybZM/co9KOrjXA= 2023-11-02T18:24:08.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/ZG6GT72iKVkybZM/co9KOrjXA=', 'Content-Length': '0'} 2023-11-02T18:24:08.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1c1c18c280fedb3-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.306 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:24:08.307 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:24:08.307 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.307 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:24:08.308 DEBUG:boto:Data: 2023-11-02T18:24:08.308 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.309 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.309 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.310 DEBUG:boto:Params: {} 2023-11-02T18:24:08.310 DEBUG:boto:Token: None 2023-11-02T18:24:08.310 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/c 2023-11-02T18:24:08.311 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/ZG6GT72iKVkybZM/co9KOrjXA= 2023-11-02T18:24:08.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/ZG6GT72iKVkybZM/co9KOrjXA=', 'Content-Length': '0'} 2023-11-02T18:24:08.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd0fa749492418ed-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:24:08.315 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:24:08.316 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:24:08.316 DEBUG:boto:Method: GET 2023-11-02T18:24:08.317 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:24:08.317 DEBUG:boto:Data: 2023-11-02T18:24:08.317 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.318 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.318 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.319 DEBUG:boto:Params: {} 2023-11-02T18:24:08.319 DEBUG:boto:Token: None 2023-11-02T18:24:08.320 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/c 2023-11-02T18:24:08.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hIxFiRJtKRrybPW3yVkMJthCak0= 2023-11-02T18:24:08.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hIxFiRJtKRrybPW3yVkMJthCak0=', 'Content-Length': '0'} 2023-11-02T18:24:08.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d141061542ec29d8-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.325 DEBUG:boto:path=/ibtljn-55/c 2023-11-02T18:24:08.325 DEBUG:boto:auth_path=/ibtljn-55/c 2023-11-02T18:24:08.326 DEBUG:boto:Method: GET 2023-11-02T18:24:08.326 DEBUG:boto:Path: /ibtljn-55/c 2023-11-02T18:24:08.326 DEBUG:boto:Data: 2023-11-02T18:24:08.329 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.329 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.330 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.330 DEBUG:boto:Params: {} 2023-11-02T18:24:08.331 DEBUG:boto:Token: None 2023-11-02T18:24:08.331 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/c 2023-11-02T18:24:08.332 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hIxFiRJtKRrybPW3yVkMJthCak0= 2023-11-02T18:24:08.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hIxFiRJtKRrybPW3yVkMJthCak0=', 'Content-Length': '0'} 2023-11-02T18:24:08.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053bd522c95c7e2da-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.336 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.336 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.337 DEBUG:boto:path=/ibtljn-55/?prefix=c 2023-11-02T18:24:08.337 DEBUG:boto:auth_path=/ibtljn-55/?prefix=c 2023-11-02T18:24:08.338 DEBUG:boto:Method: GET 2023-11-02T18:24:08.338 DEBUG:boto:Path: /ibtljn-55/?prefix=c 2023-11-02T18:24:08.339 DEBUG:boto:Data: 2023-11-02T18:24:08.339 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.340 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.340 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.340 DEBUG:boto:Params: {} 2023-11-02T18:24:08.341 DEBUG:boto:Token: None 2023-11-02T18:24:08.341 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.342 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbf6c4e51c1928d5-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.390 DEBUG:boto:b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.391 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.392 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.392 DEBUG:boto:path=/ibtljn-55/?prefix=c 2023-11-02T18:24:08.392 DEBUG:boto:auth_path=/ibtljn-55/?prefix=c 2023-11-02T18:24:08.393 DEBUG:boto:Method: GET 2023-11-02T18:24:08.393 DEBUG:boto:Path: /ibtljn-55/?prefix=c 2023-11-02T18:24:08.394 DEBUG:boto:Data: 2023-11-02T18:24:08.394 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.395 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.395 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.395 DEBUG:boto:Params: {} 2023-11-02T18:24:08.396 DEBUG:boto:Token: None 2023-11-02T18:24:08.396 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.397 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd218f0357e4f0b3-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.402 DEBUG:boto:b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:24:08.403 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:24:08.403 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:24:08.404 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:24:08.404 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:24:08.404 DEBUG:boto:Method: GET 2023-11-02T18:24:08.405 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:24:08.405 DEBUG:boto:Data: 2023-11-02T18:24:08.406 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.406 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.407 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.407 DEBUG:boto:Params: {} 2023-11-02T18:24:08.408 DEBUG:boto:Token: None 2023-11-02T18:24:08.408 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:24:08.409 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0= 2023-11-02T18:24:08.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0=', 'Content-Length': '0'} 2023-11-02T18:24:08.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e3246f2d4f5af9-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.413 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:24:08.413 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:24:08.413 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:24:08.414 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:24:08.414 DEBUG:boto:Method: GET 2023-11-02T18:24:08.415 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:24:08.415 DEBUG:boto:Data: 2023-11-02T18:24:08.416 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.416 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.416 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.417 DEBUG:boto:Params: {} 2023-11-02T18:24:08.417 DEBUG:boto:Token: None 2023-11-02T18:24:08.418 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:24:08.418 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0= 2023-11-02T18:24:08.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0=', 'Content-Length': '0'} 2023-11-02T18:24:08.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da2ade5d00314e27-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.423 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:24:08.424 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:24:08.424 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:24:08.424 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:24:08.425 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.425 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:24:08.426 DEBUG:boto:Data: 2023-11-02T18:24:08.426 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.427 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.427 DEBUG:boto:Params: {} 2023-11-02T18:24:08.428 DEBUG:boto:Token: None 2023-11-02T18:24:08.428 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:24:08.429 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OcKhnlKozTwOKRS9lJw7yPFPSGo= 2023-11-02T18:24:08.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OcKhnlKozTwOKRS9lJw7yPFPSGo=', 'Content-Length': '0'} 2023-11-02T18:24:08.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d912410b1692dd5-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.433 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:24:08.433 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:24:08.433 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:24:08.434 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:24:08.434 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.435 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:24:08.435 DEBUG:boto:Data: 2023-11-02T18:24:08.436 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.436 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.436 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.437 DEBUG:boto:Params: {} 2023-11-02T18:24:08.437 DEBUG:boto:Token: None 2023-11-02T18:24:08.438 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:24:08.438 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OcKhnlKozTwOKRS9lJw7yPFPSGo= 2023-11-02T18:24:08.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OcKhnlKozTwOKRS9lJw7yPFPSGo=', 'Content-Length': '0'} 2023-11-02T18:24:08.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008b88a84743a2265-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:24:08.442 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:24:08.443 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:24:08.443 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:24:08.444 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:24:08.444 DEBUG:boto:Method: GET 2023-11-02T18:24:08.445 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:24:08.445 DEBUG:boto:Data: 2023-11-02T18:24:08.445 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.446 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.446 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.447 DEBUG:boto:Params: {} 2023-11-02T18:24:08.447 DEBUG:boto:Token: None 2023-11-02T18:24:08.448 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:24:08.448 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0= 2023-11-02T18:24:08.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0=', 'Content-Length': '0'} 2023-11-02T18:24:08.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1d81bed5ec6134c-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.494 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:24:08.495 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:24:08.495 DEBUG:boto:path=/ibtljn-55/d?versionId=null 2023-11-02T18:24:08.496 DEBUG:boto:auth_path=/ibtljn-55/d?versionId=null 2023-11-02T18:24:08.496 DEBUG:boto:Method: GET 2023-11-02T18:24:08.497 DEBUG:boto:Path: /ibtljn-55/d?versionId=null 2023-11-02T18:24:08.497 DEBUG:boto:Data: 2023-11-02T18:24:08.497 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.498 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.498 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.499 DEBUG:boto:Params: {} 2023-11-02T18:24:08.499 DEBUG:boto:Token: None 2023-11-02T18:24:08.500 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/d?versionId=null 2023-11-02T18:24:08.500 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0= 2023-11-02T18:24:08.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0=', 'Content-Length': '0'} 2023-11-02T18:24:08.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9a73809fc06c0a2-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.504 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.505 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.505 DEBUG:boto:path=/ibtljn-55/?prefix=d 2023-11-02T18:24:08.506 DEBUG:boto:auth_path=/ibtljn-55/?prefix=d 2023-11-02T18:24:08.506 DEBUG:boto:Method: GET 2023-11-02T18:24:08.507 DEBUG:boto:Path: /ibtljn-55/?prefix=d 2023-11-02T18:24:08.507 DEBUG:boto:Data: 2023-11-02T18:24:08.508 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.508 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.509 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.509 DEBUG:boto:Params: {} 2023-11-02T18:24:08.509 DEBUG:boto:Token: None 2023-11-02T18:24:08.510 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.511 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cca119f396f1c61-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.517 DEBUG:boto:b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.518 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.518 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.519 DEBUG:boto:path=/ibtljn-55/?prefix=d 2023-11-02T18:24:08.519 DEBUG:boto:auth_path=/ibtljn-55/?prefix=d 2023-11-02T18:24:08.519 DEBUG:boto:Method: GET 2023-11-02T18:24:08.520 DEBUG:boto:Path: /ibtljn-55/?prefix=d 2023-11-02T18:24:08.520 DEBUG:boto:Data: 2023-11-02T18:24:08.521 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.521 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.522 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.522 DEBUG:boto:Params: {} 2023-11-02T18:24:08.522 DEBUG:boto:Token: None 2023-11-02T18:24:08.523 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.523 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8978c00c87573c3-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.574 DEBUG:boto:b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.575 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:24:08.576 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:24:08.576 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.577 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:24:08.577 DEBUG:boto:Data: 2023-11-02T18:24:08.577 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.578 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.578 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.579 DEBUG:boto:Params: {} 2023-11-02T18:24:08.579 DEBUG:boto:Token: None 2023-11-02T18:24:08.580 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/d 2023-11-02T18:24:08.580 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qfTV4Sj6HH1z6ekJapTYjrD8yuk= 2023-11-02T18:24:08.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qfTV4Sj6HH1z6ekJapTYjrD8yuk=', 'Content-Length': '0'} 2023-11-02T18:24:08.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000412dcbeac6973abc-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.584 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:24:08.585 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:24:08.585 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.586 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:24:08.586 DEBUG:boto:Data: 2023-11-02T18:24:08.586 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.587 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.587 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.588 DEBUG:boto:Params: {} 2023-11-02T18:24:08.588 DEBUG:boto:Token: None 2023-11-02T18:24:08.589 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/d 2023-11-02T18:24:08.589 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qfTV4Sj6HH1z6ekJapTYjrD8yuk= 2023-11-02T18:24:08.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qfTV4Sj6HH1z6ekJapTYjrD8yuk=', 'Content-Length': '0'} 2023-11-02T18:24:08.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000347c1232a92dd33a-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:24:08.594 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:24:08.594 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:24:08.594 DEBUG:boto:Method: GET 2023-11-02T18:24:08.595 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:24:08.595 DEBUG:boto:Data: 2023-11-02T18:24:08.596 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.596 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.597 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.597 DEBUG:boto:Params: {} 2023-11-02T18:24:08.598 DEBUG:boto:Token: None 2023-11-02T18:24:08.598 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/d 2023-11-02T18:24:08.599 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bnDobB9W/g/BJKIf9FR7l6wPpmk= 2023-11-02T18:24:08.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bnDobB9W/g/BJKIf9FR7l6wPpmk=', 'Content-Length': '0'} 2023-11-02T18:24:08.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029f9f5250ab37a0c-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.603 DEBUG:boto:path=/ibtljn-55/d 2023-11-02T18:24:08.603 DEBUG:boto:auth_path=/ibtljn-55/d 2023-11-02T18:24:08.603 DEBUG:boto:Method: GET 2023-11-02T18:24:08.604 DEBUG:boto:Path: /ibtljn-55/d 2023-11-02T18:24:08.604 DEBUG:boto:Data: 2023-11-02T18:24:08.605 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.605 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.606 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.606 DEBUG:boto:Params: {} 2023-11-02T18:24:08.606 DEBUG:boto:Token: None 2023-11-02T18:24:08.607 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/d 2023-11-02T18:24:08.607 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bnDobB9W/g/BJKIf9FR7l6wPpmk= 2023-11-02T18:24:08.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bnDobB9W/g/BJKIf9FR7l6wPpmk=', 'Content-Length': '0'} 2023-11-02T18:24:08.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca9df948683d88c7-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.611 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.612 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.612 DEBUG:boto:path=/ibtljn-55/?prefix=d 2023-11-02T18:24:08.613 DEBUG:boto:auth_path=/ibtljn-55/?prefix=d 2023-11-02T18:24:08.613 DEBUG:boto:Method: GET 2023-11-02T18:24:08.614 DEBUG:boto:Path: /ibtljn-55/?prefix=d 2023-11-02T18:24:08.614 DEBUG:boto:Data: 2023-11-02T18:24:08.615 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.615 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.616 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.616 DEBUG:boto:Params: {} 2023-11-02T18:24:08.616 DEBUG:boto:Token: None 2023-11-02T18:24:08.617 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.617 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7168cbb568b17a1-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.624 DEBUG:boto:b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.625 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.626 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.626 DEBUG:boto:path=/ibtljn-55/?prefix=d 2023-11-02T18:24:08.627 DEBUG:boto:auth_path=/ibtljn-55/?prefix=d 2023-11-02T18:24:08.627 DEBUG:boto:Method: GET 2023-11-02T18:24:08.627 DEBUG:boto:Path: /ibtljn-55/?prefix=d 2023-11-02T18:24:08.628 DEBUG:boto:Data: 2023-11-02T18:24:08.628 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.629 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.629 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.630 DEBUG:boto:Params: {} 2023-11-02T18:24:08.630 DEBUG:boto:Token: None 2023-11-02T18:24:08.631 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.631 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e45ee1a890f0582-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.636 DEBUG:boto:b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-11-02T18:24:08.637 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:24:08.638 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:24:08.638 DEBUG:boto:path=/ibtljn-55/e?versionId=null 2023-11-02T18:24:08.639 DEBUG:boto:auth_path=/ibtljn-55/e?versionId=null 2023-11-02T18:24:08.639 DEBUG:boto:Method: GET 2023-11-02T18:24:08.639 DEBUG:boto:Path: /ibtljn-55/e?versionId=null 2023-11-02T18:24:08.640 DEBUG:boto:Data: 2023-11-02T18:24:08.640 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.641 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.641 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.642 DEBUG:boto:Params: {} 2023-11-02T18:24:08.642 DEBUG:boto:Token: None 2023-11-02T18:24:08.643 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/e?versionId=null 2023-11-02T18:24:08.643 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4= 2023-11-02T18:24:08.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4=', 'Content-Length': '0'} 2023-11-02T18:24:08.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002517bb285da23ae7-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.694 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:24:08.695 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:24:08.695 DEBUG:boto:path=/ibtljn-55/e?versionId=null 2023-11-02T18:24:08.696 DEBUG:boto:auth_path=/ibtljn-55/e?versionId=null 2023-11-02T18:24:08.696 DEBUG:boto:Method: GET 2023-11-02T18:24:08.697 DEBUG:boto:Path: /ibtljn-55/e?versionId=null 2023-11-02T18:24:08.697 DEBUG:boto:Data: 2023-11-02T18:24:08.698 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.698 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.699 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.699 DEBUG:boto:Params: {} 2023-11-02T18:24:08.700 DEBUG:boto:Token: None 2023-11-02T18:24:08.700 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/e?versionId=null 2023-11-02T18:24:08.701 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4= 2023-11-02T18:24:08.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4=', 'Content-Length': '0'} 2023-11-02T18:24:08.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007581ea57727a4491-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.705 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:24:08.705 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:24:08.706 DEBUG:boto:path=/ibtljn-55/e?versionId=null 2023-11-02T18:24:08.706 DEBUG:boto:auth_path=/ibtljn-55/e?versionId=null 2023-11-02T18:24:08.707 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.707 DEBUG:boto:Path: /ibtljn-55/e?versionId=null 2023-11-02T18:24:08.707 DEBUG:boto:Data: 2023-11-02T18:24:08.708 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.708 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.709 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.709 DEBUG:boto:Params: {} 2023-11-02T18:24:08.709 DEBUG:boto:Token: None 2023-11-02T18:24:08.710 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/e?versionId=null 2023-11-02T18:24:08.710 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:su2UvNz7iRM8XFNJC1pI7jwcPqI= 2023-11-02T18:24:08.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:su2UvNz7iRM8XFNJC1pI7jwcPqI=', 'Content-Length': '0'} 2023-11-02T18:24:08.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff9df02d35928dfc-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.715 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:24:08.715 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:24:08.716 DEBUG:boto:path=/ibtljn-55/e?versionId=null 2023-11-02T18:24:08.716 DEBUG:boto:auth_path=/ibtljn-55/e?versionId=null 2023-11-02T18:24:08.716 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.717 DEBUG:boto:Path: /ibtljn-55/e?versionId=null 2023-11-02T18:24:08.717 DEBUG:boto:Data: 2023-11-02T18:24:08.718 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.718 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.719 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.719 DEBUG:boto:Params: {} 2023-11-02T18:24:08.720 DEBUG:boto:Token: None 2023-11-02T18:24:08.720 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/e?versionId=null 2023-11-02T18:24:08.720 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:su2UvNz7iRM8XFNJC1pI7jwcPqI= 2023-11-02T18:24:08.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:su2UvNz7iRM8XFNJC1pI7jwcPqI=', 'Content-Length': '0'} 2023-11-02T18:24:08.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee90d4509286579c-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-11-02T18:24:08.725 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:24:08.725 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:24:08.726 DEBUG:boto:path=/ibtljn-55/e?versionId=null 2023-11-02T18:24:08.726 DEBUG:boto:auth_path=/ibtljn-55/e?versionId=null 2023-11-02T18:24:08.727 DEBUG:boto:Method: GET 2023-11-02T18:24:08.727 DEBUG:boto:Path: /ibtljn-55/e?versionId=null 2023-11-02T18:24:08.727 DEBUG:boto:Data: 2023-11-02T18:24:08.728 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.728 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.729 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.729 DEBUG:boto:Params: {} 2023-11-02T18:24:08.730 DEBUG:boto:Token: None 2023-11-02T18:24:08.730 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/e?versionId=null 2023-11-02T18:24:08.731 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4= 2023-11-02T18:24:08.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4=', 'Content-Length': '0'} 2023-11-02T18:24:08.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a5173a349d112a-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.737 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:24:08.737 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:24:08.738 DEBUG:boto:path=/ibtljn-55/e?versionId=null 2023-11-02T18:24:08.738 DEBUG:boto:auth_path=/ibtljn-55/e?versionId=null 2023-11-02T18:24:08.739 DEBUG:boto:Method: GET 2023-11-02T18:24:08.739 DEBUG:boto:Path: /ibtljn-55/e?versionId=null 2023-11-02T18:24:08.740 DEBUG:boto:Data: 2023-11-02T18:24:08.740 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.741 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.741 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.741 DEBUG:boto:Params: {} 2023-11-02T18:24:08.742 DEBUG:boto:Token: None 2023-11-02T18:24:08.742 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/e?versionId=null 2023-11-02T18:24:08.743 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4= 2023-11-02T18:24:08.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4=', 'Content-Length': '0'} 2023-11-02T18:24:08.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdae0416606781c1-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.794 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.795 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.795 DEBUG:boto:path=/ibtljn-55/?prefix=e 2023-11-02T18:24:08.796 DEBUG:boto:auth_path=/ibtljn-55/?prefix=e 2023-11-02T18:24:08.796 DEBUG:boto:Method: GET 2023-11-02T18:24:08.797 DEBUG:boto:Path: /ibtljn-55/?prefix=e 2023-11-02T18:24:08.797 DEBUG:boto:Data: 2023-11-02T18:24:08.798 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.798 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.798 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.799 DEBUG:boto:Params: {} 2023-11-02T18:24:08.799 DEBUG:boto:Token: None 2023-11-02T18:24:08.800 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.800 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000272924bb5fb0fa84-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.806 DEBUG:boto:b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.807 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.808 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.808 DEBUG:boto:path=/ibtljn-55/?prefix=e 2023-11-02T18:24:08.808 DEBUG:boto:auth_path=/ibtljn-55/?prefix=e 2023-11-02T18:24:08.809 DEBUG:boto:Method: GET 2023-11-02T18:24:08.809 DEBUG:boto:Path: /ibtljn-55/?prefix=e 2023-11-02T18:24:08.810 DEBUG:boto:Data: 2023-11-02T18:24:08.810 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.811 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.811 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.812 DEBUG:boto:Params: {} 2023-11-02T18:24:08.812 DEBUG:boto:Token: None 2023-11-02T18:24:08.813 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.813 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c377279af9b91cca-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.818 DEBUG:boto:b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.818 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:24:08.819 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:24:08.819 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.820 DEBUG:boto:Path: /ibtljn-55/e 2023-11-02T18:24:08.820 DEBUG:boto:Data: 2023-11-02T18:24:08.820 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.821 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.821 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.822 DEBUG:boto:Params: {} 2023-11-02T18:24:08.822 DEBUG:boto:Token: None 2023-11-02T18:24:08.823 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/e 2023-11-02T18:24:08.823 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nGuV+17ltnhpNZ+yVxSl78VY2Bk= 2023-11-02T18:24:08.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nGuV+17ltnhpNZ+yVxSl78VY2Bk=', 'Content-Length': '0'} 2023-11-02T18:24:08.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d4ed72270eb063-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.827 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:24:08.827 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:24:08.828 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.828 DEBUG:boto:Path: /ibtljn-55/e 2023-11-02T18:24:08.829 DEBUG:boto:Data: 2023-11-02T18:24:08.829 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.830 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.830 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.830 DEBUG:boto:Params: {} 2023-11-02T18:24:08.831 DEBUG:boto:Token: None 2023-11-02T18:24:08.831 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/e 2023-11-02T18:24:08.832 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nGuV+17ltnhpNZ+yVxSl78VY2Bk= 2023-11-02T18:24:08.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nGuV+17ltnhpNZ+yVxSl78VY2Bk=', 'Content-Length': '0'} 2023-11-02T18:24:08.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095abdc1735500a5c-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-11-02T18:24:08.836 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:24:08.837 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:24:08.837 DEBUG:boto:Method: GET 2023-11-02T18:24:08.838 DEBUG:boto:Path: /ibtljn-55/e 2023-11-02T18:24:08.838 DEBUG:boto:Data: 2023-11-02T18:24:08.838 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.839 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.839 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.840 DEBUG:boto:Params: {} 2023-11-02T18:24:08.840 DEBUG:boto:Token: None 2023-11-02T18:24:08.841 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/e 2023-11-02T18:24:08.841 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qN29qHJ4W3l4RsZRXMdEJtf/m3k= 2023-11-02T18:24:08.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qN29qHJ4W3l4RsZRXMdEJtf/m3k=', 'Content-Length': '0'} 2023-11-02T18:24:08.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a18ea6a7b28695d2-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.886 DEBUG:boto:path=/ibtljn-55/e 2023-11-02T18:24:08.887 DEBUG:boto:auth_path=/ibtljn-55/e 2023-11-02T18:24:08.888 DEBUG:boto:Method: GET 2023-11-02T18:24:08.888 DEBUG:boto:Path: /ibtljn-55/e 2023-11-02T18:24:08.889 DEBUG:boto:Data: 2023-11-02T18:24:08.889 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.890 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.890 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.891 DEBUG:boto:Params: {} 2023-11-02T18:24:08.891 DEBUG:boto:Token: None 2023-11-02T18:24:08.892 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/e 2023-11-02T18:24:08.892 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qN29qHJ4W3l4RsZRXMdEJtf/m3k= 2023-11-02T18:24:08.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qN29qHJ4W3l4RsZRXMdEJtf/m3k=', 'Content-Length': '0'} 2023-11-02T18:24:08.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5f9db20155e12cf-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.897 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.898 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.898 DEBUG:boto:path=/ibtljn-55/?prefix=e 2023-11-02T18:24:08.898 DEBUG:boto:auth_path=/ibtljn-55/?prefix=e 2023-11-02T18:24:08.899 DEBUG:boto:Method: GET 2023-11-02T18:24:08.899 DEBUG:boto:Path: /ibtljn-55/?prefix=e 2023-11-02T18:24:08.900 DEBUG:boto:Data: 2023-11-02T18:24:08.900 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.901 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.901 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.901 DEBUG:boto:Params: {} 2023-11-02T18:24:08.902 DEBUG:boto:Token: None 2023-11-02T18:24:08.902 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.903 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b5e53c1ee2242dd-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.909 DEBUG:boto:b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.910 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:08.911 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:08.911 DEBUG:boto:path=/ibtljn-55/?prefix=e 2023-11-02T18:24:08.911 DEBUG:boto:auth_path=/ibtljn-55/?prefix=e 2023-11-02T18:24:08.912 DEBUG:boto:Method: GET 2023-11-02T18:24:08.912 DEBUG:boto:Path: /ibtljn-55/?prefix=e 2023-11-02T18:24:08.913 DEBUG:boto:Data: 2023-11-02T18:24:08.913 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.914 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.914 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.915 DEBUG:boto:Params: {} 2023-11-02T18:24:08.915 DEBUG:boto:Token: None 2023-11-02T18:24:08.915 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/ 2023-11-02T18:24:08.916 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T18:24:08.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T18:24:08.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057b57321345eccbf-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.920 DEBUG:boto:b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:08.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-11-02T18:24:08.922 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:24:08.922 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:24:08.923 DEBUG:boto:path=/ibtljn-55/f?versionId=null 2023-11-02T18:24:08.923 DEBUG:boto:auth_path=/ibtljn-55/f?versionId=null 2023-11-02T18:24:08.923 DEBUG:boto:Method: GET 2023-11-02T18:24:08.924 DEBUG:boto:Path: /ibtljn-55/f?versionId=null 2023-11-02T18:24:08.924 DEBUG:boto:Data: 2023-11-02T18:24:08.925 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.925 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.926 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.926 DEBUG:boto:Params: {} 2023-11-02T18:24:08.927 DEBUG:boto:Token: None 2023-11-02T18:24:08.927 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/f?versionId=null 2023-11-02T18:24:08.928 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LG8cifl9QDf0NnndH6/Fv+SS4Ic= 2023-11-02T18:24:08.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LG8cifl9QDf0NnndH6/Fv+SS4Ic=', 'Content-Length': '0'} 2023-11-02T18:24:08.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db630df81acba4db-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.932 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:24:08.932 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:24:08.933 DEBUG:boto:path=/ibtljn-55/f?versionId=null 2023-11-02T18:24:08.933 DEBUG:boto:auth_path=/ibtljn-55/f?versionId=null 2023-11-02T18:24:08.934 DEBUG:boto:Method: GET 2023-11-02T18:24:08.934 DEBUG:boto:Path: /ibtljn-55/f?versionId=null 2023-11-02T18:24:08.935 DEBUG:boto:Data: 2023-11-02T18:24:08.935 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.935 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.936 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.936 DEBUG:boto:Params: {} 2023-11-02T18:24:08.937 DEBUG:boto:Token: None 2023-11-02T18:24:08.937 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/f?versionId=null 2023-11-02T18:24:08.938 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LG8cifl9QDf0NnndH6/Fv+SS4Ic= 2023-11-02T18:24:08.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LG8cifl9QDf0NnndH6/Fv+SS4Ic=', 'Content-Length': '0'} 2023-11-02T18:24:08.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bab75ef9ceaf9f1-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.982 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:24:08.983 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:24:08.983 DEBUG:boto:path=/ibtljn-55/f?versionId=null 2023-11-02T18:24:08.984 DEBUG:boto:auth_path=/ibtljn-55/f?versionId=null 2023-11-02T18:24:08.984 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.985 DEBUG:boto:Path: /ibtljn-55/f?versionId=null 2023-11-02T18:24:08.985 DEBUG:boto:Data: 2023-11-02T18:24:08.986 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.986 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:08.986 DEBUG:boto:Port: 8001 2023-11-02T18:24:08.987 DEBUG:boto:Params: {} 2023-11-02T18:24:08.987 DEBUG:boto:Token: None 2023-11-02T18:24:08.988 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/f?versionId=null 2023-11-02T18:24:08.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xw3VTYZ8pw6z09do6F9vm8EW4G4= 2023-11-02T18:24:08.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xw3VTYZ8pw6z09do6F9vm8EW4G4=', 'Content-Length': '0'} 2023-11-02T18:24:08.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d17d9587d1f7959-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:08.992 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:24:08.993 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:24:08.993 DEBUG:boto:path=/ibtljn-55/f?versionId=null 2023-11-02T18:24:08.994 DEBUG:boto:auth_path=/ibtljn-55/f?versionId=null 2023-11-02T18:24:08.994 DEBUG:boto:Method: HEAD 2023-11-02T18:24:08.995 DEBUG:boto:Path: /ibtljn-55/f?versionId=null 2023-11-02T18:24:08.995 DEBUG:boto:Data: 2023-11-02T18:24:08.995 DEBUG:boto:Headers: {} 2023-11-02T18:24:08.996 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:08.996 DEBUG:boto:Port: 8000 2023-11-02T18:24:08.997 DEBUG:boto:Params: {} 2023-11-02T18:24:08.997 DEBUG:boto:Token: None 2023-11-02T18:24:08.998 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:08 GMT /ibtljn-55/f?versionId=null 2023-11-02T18:24:08.998 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xw3VTYZ8pw6z09do6F9vm8EW4G4= 2023-11-02T18:24:08.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xw3VTYZ8pw6z09do6F9vm8EW4G4=', 'Content-Length': '0'} 2023-11-02T18:24:09.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c5961dd0eec1843-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-11-02T18:24:09.003 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:24:09.003 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:24:09.003 DEBUG:boto:path=/ibtljn-55/f?versionId=null 2023-11-02T18:24:09.004 DEBUG:boto:auth_path=/ibtljn-55/f?versionId=null 2023-11-02T18:24:09.004 DEBUG:boto:Method: GET 2023-11-02T18:24:09.005 DEBUG:boto:Path: /ibtljn-55/f?versionId=null 2023-11-02T18:24:09.005 DEBUG:boto:Data: 2023-11-02T18:24:09.006 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.006 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.006 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.007 DEBUG:boto:Params: {} 2023-11-02T18:24:09.007 DEBUG:boto:Token: None 2023-11-02T18:24:09.008 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/f?versionId=null 2023-11-02T18:24:09.008 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H5p+5YTFGaFzyoMtg9LUPLVWbV4= 2023-11-02T18:24:09.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H5p+5YTFGaFzyoMtg9LUPLVWbV4=', 'Content-Length': '0'} 2023-11-02T18:24:09.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007228a80ecdc1fead-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.054 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:24:09.055 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:24:09.055 DEBUG:boto:path=/ibtljn-55/f?versionId=null 2023-11-02T18:24:09.056 DEBUG:boto:auth_path=/ibtljn-55/f?versionId=null 2023-11-02T18:24:09.056 DEBUG:boto:Method: GET 2023-11-02T18:24:09.057 DEBUG:boto:Path: /ibtljn-55/f?versionId=null 2023-11-02T18:24:09.057 DEBUG:boto:Data: 2023-11-02T18:24:09.058 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.058 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.059 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.059 DEBUG:boto:Params: {} 2023-11-02T18:24:09.060 DEBUG:boto:Token: None 2023-11-02T18:24:09.060 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/f?versionId=null 2023-11-02T18:24:09.061 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H5p+5YTFGaFzyoMtg9LUPLVWbV4= 2023-11-02T18:24:09.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H5p+5YTFGaFzyoMtg9LUPLVWbV4=', 'Content-Length': '0'} 2023-11-02T18:24:09.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a34c45813bc3327-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.065 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:09.066 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:09.066 DEBUG:boto:path=/ibtljn-55/?prefix=f 2023-11-02T18:24:09.067 DEBUG:boto:auth_path=/ibtljn-55/?prefix=f 2023-11-02T18:24:09.067 DEBUG:boto:Method: GET 2023-11-02T18:24:09.067 DEBUG:boto:Path: /ibtljn-55/?prefix=f 2023-11-02T18:24:09.068 DEBUG:boto:Data: 2023-11-02T18:24:09.068 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.069 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.069 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.070 DEBUG:boto:Params: {} 2023-11-02T18:24:09.070 DEBUG:boto:Token: None 2023-11-02T18:24:09.071 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/ 2023-11-02T18:24:09.071 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T18:24:09.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T18:24:09.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd812aef20526b6d-006543e949-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.077 DEBUG:boto:b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:09.077 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:09.078 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:09.078 DEBUG:boto:path=/ibtljn-55/?prefix=f 2023-11-02T18:24:09.079 DEBUG:boto:auth_path=/ibtljn-55/?prefix=f 2023-11-02T18:24:09.079 DEBUG:boto:Method: GET 2023-11-02T18:24:09.080 DEBUG:boto:Path: /ibtljn-55/?prefix=f 2023-11-02T18:24:09.080 DEBUG:boto:Data: 2023-11-02T18:24:09.080 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.081 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.081 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.082 DEBUG:boto:Params: {} 2023-11-02T18:24:09.082 DEBUG:boto:Token: None 2023-11-02T18:24:09.083 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/ 2023-11-02T18:24:09.083 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T18:24:09.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T18:24:09.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cc67d09c35e5d2f-006543e949-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.088 DEBUG:boto:b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:09.088 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:24:09.089 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:24:09.089 DEBUG:boto:Method: HEAD 2023-11-02T18:24:09.090 DEBUG:boto:Path: /ibtljn-55/f 2023-11-02T18:24:09.090 DEBUG:boto:Data: 2023-11-02T18:24:09.091 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.091 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.091 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.092 DEBUG:boto:Params: {} 2023-11-02T18:24:09.092 DEBUG:boto:Token: None 2023-11-02T18:24:09.093 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/f 2023-11-02T18:24:09.093 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6JXtSCcmhdNtp/N1WBXCRxSaoJM= 2023-11-02T18:24:09.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6JXtSCcmhdNtp/N1WBXCRxSaoJM=', 'Content-Length': '0'} 2023-11-02T18:24:09.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf618cab3d313d92-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.098 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:24:09.098 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:24:09.099 DEBUG:boto:Method: HEAD 2023-11-02T18:24:09.099 DEBUG:boto:Path: /ibtljn-55/f 2023-11-02T18:24:09.100 DEBUG:boto:Data: 2023-11-02T18:24:09.100 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.100 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.101 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.101 DEBUG:boto:Params: {} 2023-11-02T18:24:09.102 DEBUG:boto:Token: None 2023-11-02T18:24:09.102 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/f 2023-11-02T18:24:09.103 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6JXtSCcmhdNtp/N1WBXCRxSaoJM= 2023-11-02T18:24:09.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6JXtSCcmhdNtp/N1WBXCRxSaoJM=', 'Content-Length': '0'} 2023-11-02T18:24:09.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f74edd0657ea0730-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-11-02T18:24:09.110 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:24:09.110 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:24:09.111 DEBUG:boto:Method: GET 2023-11-02T18:24:09.111 DEBUG:boto:Path: /ibtljn-55/f 2023-11-02T18:24:09.111 DEBUG:boto:Data: 2023-11-02T18:24:09.112 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.112 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.113 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.113 DEBUG:boto:Params: {} 2023-11-02T18:24:09.114 DEBUG:boto:Token: None 2023-11-02T18:24:09.114 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/f 2023-11-02T18:24:09.115 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:K4+LqtNTBjSicoEVxivsYDgo/og= 2023-11-02T18:24:09.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:K4+LqtNTBjSicoEVxivsYDgo/og=', 'Content-Length': '0'} 2023-11-02T18:24:09.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028c2a195e0f32477-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.119 DEBUG:boto:path=/ibtljn-55/f 2023-11-02T18:24:09.119 DEBUG:boto:auth_path=/ibtljn-55/f 2023-11-02T18:24:09.120 DEBUG:boto:Method: GET 2023-11-02T18:24:09.120 DEBUG:boto:Path: /ibtljn-55/f 2023-11-02T18:24:09.120 DEBUG:boto:Data: 2023-11-02T18:24:09.121 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.121 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.122 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.122 DEBUG:boto:Params: {} 2023-11-02T18:24:09.123 DEBUG:boto:Token: None 2023-11-02T18:24:09.123 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/f 2023-11-02T18:24:09.124 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:K4+LqtNTBjSicoEVxivsYDgo/og= 2023-11-02T18:24:09.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:K4+LqtNTBjSicoEVxivsYDgo/og=', 'Content-Length': '0'} 2023-11-02T18:24:09.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004705a6e56c0afe49-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.170 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:09.171 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:09.171 DEBUG:boto:path=/ibtljn-55/?prefix=f 2023-11-02T18:24:09.172 DEBUG:boto:auth_path=/ibtljn-55/?prefix=f 2023-11-02T18:24:09.172 DEBUG:boto:Method: GET 2023-11-02T18:24:09.173 DEBUG:boto:Path: /ibtljn-55/?prefix=f 2023-11-02T18:24:09.173 DEBUG:boto:Data: 2023-11-02T18:24:09.174 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.174 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.175 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.175 DEBUG:boto:Params: {} 2023-11-02T18:24:09.176 DEBUG:boto:Token: None 2023-11-02T18:24:09.176 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/ 2023-11-02T18:24:09.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T18:24:09.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T18:24:09.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bda91ddf5d2ed0d6-006543e949-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.183 DEBUG:boto:b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:09.183 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:09.184 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:09.184 DEBUG:boto:path=/ibtljn-55/?prefix=f 2023-11-02T18:24:09.185 DEBUG:boto:auth_path=/ibtljn-55/?prefix=f 2023-11-02T18:24:09.185 DEBUG:boto:Method: GET 2023-11-02T18:24:09.186 DEBUG:boto:Path: /ibtljn-55/?prefix=f 2023-11-02T18:24:09.186 DEBUG:boto:Data: 2023-11-02T18:24:09.187 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.187 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.187 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.188 DEBUG:boto:Params: {} 2023-11-02T18:24:09.188 DEBUG:boto:Token: None 2023-11-02T18:24:09.189 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/ 2023-11-02T18:24:09.189 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T18:24:09.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T18:24:09.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c77a1f7d2770820-006543e949-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.193 DEBUG:boto:b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:09.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-11-02T18:24:09.194 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:24:09.195 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:24:09.195 DEBUG:boto:path=/ibtljn-55/g?versionId=null 2023-11-02T18:24:09.196 DEBUG:boto:auth_path=/ibtljn-55/g?versionId=null 2023-11-02T18:24:09.196 DEBUG:boto:Method: GET 2023-11-02T18:24:09.197 DEBUG:boto:Path: /ibtljn-55/g?versionId=null 2023-11-02T18:24:09.197 DEBUG:boto:Data: 2023-11-02T18:24:09.197 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.198 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.198 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.199 DEBUG:boto:Params: {} 2023-11-02T18:24:09.199 DEBUG:boto:Token: None 2023-11-02T18:24:09.200 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/g?versionId=null 2023-11-02T18:24:09.200 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk= 2023-11-02T18:24:09.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk=', 'Content-Length': '0'} 2023-11-02T18:24:09.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7441279cf248e70-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.250 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:24:09.251 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:24:09.251 DEBUG:boto:path=/ibtljn-55/g?versionId=null 2023-11-02T18:24:09.252 DEBUG:boto:auth_path=/ibtljn-55/g?versionId=null 2023-11-02T18:24:09.252 DEBUG:boto:Method: GET 2023-11-02T18:24:09.252 DEBUG:boto:Path: /ibtljn-55/g?versionId=null 2023-11-02T18:24:09.253 DEBUG:boto:Data: 2023-11-02T18:24:09.253 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.254 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.255 DEBUG:boto:Params: {} 2023-11-02T18:24:09.255 DEBUG:boto:Token: None 2023-11-02T18:24:09.255 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/g?versionId=null 2023-11-02T18:24:09.256 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk= 2023-11-02T18:24:09.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk=', 'Content-Length': '0'} 2023-11-02T18:24:09.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b382b3a7c5dc8c1-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.260 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:24:09.261 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:24:09.261 DEBUG:boto:path=/ibtljn-55/g?versionId=null 2023-11-02T18:24:09.261 DEBUG:boto:auth_path=/ibtljn-55/g?versionId=null 2023-11-02T18:24:09.262 DEBUG:boto:Method: HEAD 2023-11-02T18:24:09.262 DEBUG:boto:Path: /ibtljn-55/g?versionId=null 2023-11-02T18:24:09.263 DEBUG:boto:Data: 2023-11-02T18:24:09.263 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.264 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.264 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.264 DEBUG:boto:Params: {} 2023-11-02T18:24:09.265 DEBUG:boto:Token: None 2023-11-02T18:24:09.265 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/g?versionId=null 2023-11-02T18:24:09.266 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6C3UgQ5IUl2YJl+QSiXs9U7msco= 2023-11-02T18:24:09.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6C3UgQ5IUl2YJl+QSiXs9U7msco=', 'Content-Length': '0'} 2023-11-02T18:24:09.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000ef480fb70b0596-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.270 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:24:09.270 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:24:09.271 DEBUG:boto:path=/ibtljn-55/g?versionId=null 2023-11-02T18:24:09.271 DEBUG:boto:auth_path=/ibtljn-55/g?versionId=null 2023-11-02T18:24:09.272 DEBUG:boto:Method: HEAD 2023-11-02T18:24:09.272 DEBUG:boto:Path: /ibtljn-55/g?versionId=null 2023-11-02T18:24:09.272 DEBUG:boto:Data: 2023-11-02T18:24:09.273 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.273 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.274 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.274 DEBUG:boto:Params: {} 2023-11-02T18:24:09.275 DEBUG:boto:Token: None 2023-11-02T18:24:09.275 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/g?versionId=null 2023-11-02T18:24:09.275 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6C3UgQ5IUl2YJl+QSiXs9U7msco= 2023-11-02T18:24:09.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6C3UgQ5IUl2YJl+QSiXs9U7msco=', 'Content-Length': '0'} 2023-11-02T18:24:09.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5d8aa78500287d-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-11-02T18:24:09.280 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:24:09.280 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:24:09.280 DEBUG:boto:path=/ibtljn-55/g?versionId=null 2023-11-02T18:24:09.281 DEBUG:boto:auth_path=/ibtljn-55/g?versionId=null 2023-11-02T18:24:09.281 DEBUG:boto:Method: GET 2023-11-02T18:24:09.282 DEBUG:boto:Path: /ibtljn-55/g?versionId=null 2023-11-02T18:24:09.282 DEBUG:boto:Data: 2023-11-02T18:24:09.283 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.283 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.284 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.284 DEBUG:boto:Params: {} 2023-11-02T18:24:09.284 DEBUG:boto:Token: None 2023-11-02T18:24:09.285 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/g?versionId=null 2023-11-02T18:24:09.285 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk= 2023-11-02T18:24:09.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk=', 'Content-Length': '0'} 2023-11-02T18:24:09.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6b4f9696c006e5-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.289 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:24:09.290 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:24:09.290 DEBUG:boto:path=/ibtljn-55/g?versionId=null 2023-11-02T18:24:09.291 DEBUG:boto:auth_path=/ibtljn-55/g?versionId=null 2023-11-02T18:24:09.291 DEBUG:boto:Method: GET 2023-11-02T18:24:09.292 DEBUG:boto:Path: /ibtljn-55/g?versionId=null 2023-11-02T18:24:09.292 DEBUG:boto:Data: 2023-11-02T18:24:09.292 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.293 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.293 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.294 DEBUG:boto:Params: {} 2023-11-02T18:24:09.294 DEBUG:boto:Token: None 2023-11-02T18:24:09.295 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/g?versionId=null 2023-11-02T18:24:09.295 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk= 2023-11-02T18:24:09.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk=', 'Content-Length': '0'} 2023-11-02T18:24:09.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000347551c232c0e2d4-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.342 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:09.343 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:09.343 DEBUG:boto:path=/ibtljn-55/?prefix=g 2023-11-02T18:24:09.344 DEBUG:boto:auth_path=/ibtljn-55/?prefix=g 2023-11-02T18:24:09.344 DEBUG:boto:Method: GET 2023-11-02T18:24:09.345 DEBUG:boto:Path: /ibtljn-55/?prefix=g 2023-11-02T18:24:09.345 DEBUG:boto:Data: 2023-11-02T18:24:09.345 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.346 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.346 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.347 DEBUG:boto:Params: {} 2023-11-02T18:24:09.347 DEBUG:boto:Token: None 2023-11-02T18:24:09.348 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/ 2023-11-02T18:24:09.348 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T18:24:09.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T18:24:09.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039b13d595fa146d8-006543e949-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.354 DEBUG:boto:b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:09.355 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:09.355 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:09.355 DEBUG:boto:path=/ibtljn-55/?prefix=g 2023-11-02T18:24:09.356 DEBUG:boto:auth_path=/ibtljn-55/?prefix=g 2023-11-02T18:24:09.356 DEBUG:boto:Method: GET 2023-11-02T18:24:09.357 DEBUG:boto:Path: /ibtljn-55/?prefix=g 2023-11-02T18:24:09.357 DEBUG:boto:Data: 2023-11-02T18:24:09.358 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.358 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.359 DEBUG:boto:Params: {} 2023-11-02T18:24:09.359 DEBUG:boto:Token: None 2023-11-02T18:24:09.360 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/ 2023-11-02T18:24:09.360 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T18:24:09.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T18:24:09.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f3472495ba71497-006543e949-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.365 DEBUG:boto:b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:09.366 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:24:09.366 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:24:09.367 DEBUG:boto:Method: HEAD 2023-11-02T18:24:09.367 DEBUG:boto:Path: /ibtljn-55/g 2023-11-02T18:24:09.367 DEBUG:boto:Data: 2023-11-02T18:24:09.368 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.368 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.369 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.369 DEBUG:boto:Params: {} 2023-11-02T18:24:09.370 DEBUG:boto:Token: None 2023-11-02T18:24:09.370 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/g 2023-11-02T18:24:09.370 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PZH0cfwKnUA3UOpTJgUKxtlWyro= 2023-11-02T18:24:09.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PZH0cfwKnUA3UOpTJgUKxtlWyro=', 'Content-Length': '0'} 2023-11-02T18:24:09.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a00d83252e166f4-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.374 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:24:09.375 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:24:09.375 DEBUG:boto:Method: HEAD 2023-11-02T18:24:09.376 DEBUG:boto:Path: /ibtljn-55/g 2023-11-02T18:24:09.376 DEBUG:boto:Data: 2023-11-02T18:24:09.376 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.377 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.377 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.378 DEBUG:boto:Params: {} 2023-11-02T18:24:09.378 DEBUG:boto:Token: None 2023-11-02T18:24:09.379 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/g 2023-11-02T18:24:09.379 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PZH0cfwKnUA3UOpTJgUKxtlWyro= 2023-11-02T18:24:09.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PZH0cfwKnUA3UOpTJgUKxtlWyro=', 'Content-Length': '0'} 2023-11-02T18:24:09.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e80a71d46b486513-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-11-02T18:24:09.383 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:24:09.384 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:24:09.384 DEBUG:boto:Method: GET 2023-11-02T18:24:09.385 DEBUG:boto:Path: /ibtljn-55/g 2023-11-02T18:24:09.385 DEBUG:boto:Data: 2023-11-02T18:24:09.386 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.386 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.387 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.387 DEBUG:boto:Params: {} 2023-11-02T18:24:09.387 DEBUG:boto:Token: None 2023-11-02T18:24:09.388 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/g 2023-11-02T18:24:09.389 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mW172XeXFFY74UKjjfyv/1BWQtk= 2023-11-02T18:24:09.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mW172XeXFFY74UKjjfyv/1BWQtk=', 'Content-Length': '0'} 2023-11-02T18:24:09.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ede57073ace1620a-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.392 DEBUG:boto:path=/ibtljn-55/g 2023-11-02T18:24:09.393 DEBUG:boto:auth_path=/ibtljn-55/g 2023-11-02T18:24:09.393 DEBUG:boto:Method: GET 2023-11-02T18:24:09.394 DEBUG:boto:Path: /ibtljn-55/g 2023-11-02T18:24:09.394 DEBUG:boto:Data: 2023-11-02T18:24:09.395 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.395 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.395 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.396 DEBUG:boto:Params: {} 2023-11-02T18:24:09.396 DEBUG:boto:Token: None 2023-11-02T18:24:09.397 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/g 2023-11-02T18:24:09.397 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mW172XeXFFY74UKjjfyv/1BWQtk= 2023-11-02T18:24:09.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mW172XeXFFY74UKjjfyv/1BWQtk=', 'Content-Length': '0'} 2023-11-02T18:24:09.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003028c3b2c7eddf70-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.442 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:09.443 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:09.443 DEBUG:boto:path=/ibtljn-55/?prefix=g 2023-11-02T18:24:09.444 DEBUG:boto:auth_path=/ibtljn-55/?prefix=g 2023-11-02T18:24:09.444 DEBUG:boto:Method: GET 2023-11-02T18:24:09.444 DEBUG:boto:Path: /ibtljn-55/?prefix=g 2023-11-02T18:24:09.445 DEBUG:boto:Data: 2023-11-02T18:24:09.445 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.446 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.446 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.447 DEBUG:boto:Params: {} 2023-11-02T18:24:09.447 DEBUG:boto:Token: None 2023-11-02T18:24:09.448 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/ 2023-11-02T18:24:09.448 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T18:24:09.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T18:24:09.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008add80fe7d0bdf0c-006543e949-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.453 DEBUG:boto:b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:09.454 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:09.455 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:09.455 DEBUG:boto:path=/ibtljn-55/?prefix=g 2023-11-02T18:24:09.456 DEBUG:boto:auth_path=/ibtljn-55/?prefix=g 2023-11-02T18:24:09.456 DEBUG:boto:Method: GET 2023-11-02T18:24:09.456 DEBUG:boto:Path: /ibtljn-55/?prefix=g 2023-11-02T18:24:09.457 DEBUG:boto:Data: 2023-11-02T18:24:09.457 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.458 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.458 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.459 DEBUG:boto:Params: {} 2023-11-02T18:24:09.459 DEBUG:boto:Token: None 2023-11-02T18:24:09.460 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/ 2023-11-02T18:24:09.460 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T18:24:09.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T18:24:09.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c8cc6f92a844417-006543e949-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.464 DEBUG:boto:b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:09.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-11-02T18:24:09.465 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:24:09.465 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:24:09.466 DEBUG:boto:path=/ibtljn-55/h?versionId=null 2023-11-02T18:24:09.466 DEBUG:boto:auth_path=/ibtljn-55/h?versionId=null 2023-11-02T18:24:09.467 DEBUG:boto:Method: GET 2023-11-02T18:24:09.467 DEBUG:boto:Path: /ibtljn-55/h?versionId=null 2023-11-02T18:24:09.468 DEBUG:boto:Data: 2023-11-02T18:24:09.468 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.468 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.469 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.469 DEBUG:boto:Params: {} 2023-11-02T18:24:09.470 DEBUG:boto:Token: None 2023-11-02T18:24:09.470 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/h?versionId=null 2023-11-02T18:24:09.471 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY= 2023-11-02T18:24:09.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY=', 'Content-Length': '0'} 2023-11-02T18:24:09.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000832c9a393dd21191-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.514 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:24:09.515 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:24:09.515 DEBUG:boto:path=/ibtljn-55/h?versionId=null 2023-11-02T18:24:09.516 DEBUG:boto:auth_path=/ibtljn-55/h?versionId=null 2023-11-02T18:24:09.516 DEBUG:boto:Method: GET 2023-11-02T18:24:09.517 DEBUG:boto:Path: /ibtljn-55/h?versionId=null 2023-11-02T18:24:09.517 DEBUG:boto:Data: 2023-11-02T18:24:09.517 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.518 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.518 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.519 DEBUG:boto:Params: {} 2023-11-02T18:24:09.519 DEBUG:boto:Token: None 2023-11-02T18:24:09.519 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/h?versionId=null 2023-11-02T18:24:09.520 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY= 2023-11-02T18:24:09.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY=', 'Content-Length': '0'} 2023-11-02T18:24:09.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7b5143c23b64d26-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.524 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:24:09.524 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:24:09.525 DEBUG:boto:path=/ibtljn-55/h?versionId=null 2023-11-02T18:24:09.525 DEBUG:boto:auth_path=/ibtljn-55/h?versionId=null 2023-11-02T18:24:09.526 DEBUG:boto:Method: HEAD 2023-11-02T18:24:09.526 DEBUG:boto:Path: /ibtljn-55/h?versionId=null 2023-11-02T18:24:09.526 DEBUG:boto:Data: 2023-11-02T18:24:09.527 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.527 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.528 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.528 DEBUG:boto:Params: {} 2023-11-02T18:24:09.528 DEBUG:boto:Token: None 2023-11-02T18:24:09.529 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/h?versionId=null 2023-11-02T18:24:09.529 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0DR3vCYFZr+I7l81PbYsu9Kw+QM= 2023-11-02T18:24:09.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0DR3vCYFZr+I7l81PbYsu9Kw+QM=', 'Content-Length': '0'} 2023-11-02T18:24:09.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f49e315e29012c38-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.533 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:24:09.534 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:24:09.534 DEBUG:boto:path=/ibtljn-55/h?versionId=null 2023-11-02T18:24:09.535 DEBUG:boto:auth_path=/ibtljn-55/h?versionId=null 2023-11-02T18:24:09.535 DEBUG:boto:Method: HEAD 2023-11-02T18:24:09.535 DEBUG:boto:Path: /ibtljn-55/h?versionId=null 2023-11-02T18:24:09.536 DEBUG:boto:Data: 2023-11-02T18:24:09.536 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.536 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.537 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.537 DEBUG:boto:Params: {} 2023-11-02T18:24:09.538 DEBUG:boto:Token: None 2023-11-02T18:24:09.538 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/h?versionId=null 2023-11-02T18:24:09.538 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0DR3vCYFZr+I7l81PbYsu9Kw+QM= 2023-11-02T18:24:09.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0DR3vCYFZr+I7l81PbYsu9Kw+QM=', 'Content-Length': '0'} 2023-11-02T18:24:09.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005688bf24fd588f0d-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-11-02T18:24:09.542 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:24:09.543 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:24:09.543 DEBUG:boto:path=/ibtljn-55/h?versionId=null 2023-11-02T18:24:09.544 DEBUG:boto:auth_path=/ibtljn-55/h?versionId=null 2023-11-02T18:24:09.544 DEBUG:boto:Method: GET 2023-11-02T18:24:09.544 DEBUG:boto:Path: /ibtljn-55/h?versionId=null 2023-11-02T18:24:09.545 DEBUG:boto:Data: 2023-11-02T18:24:09.545 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.546 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.546 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.547 DEBUG:boto:Params: {} 2023-11-02T18:24:09.547 DEBUG:boto:Token: None 2023-11-02T18:24:09.547 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/h?versionId=null 2023-11-02T18:24:09.548 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY= 2023-11-02T18:24:09.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY=', 'Content-Length': '0'} 2023-11-02T18:24:09.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1423a5180ff42a8-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.552 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:24:09.552 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:24:09.552 DEBUG:boto:path=/ibtljn-55/h?versionId=null 2023-11-02T18:24:09.553 DEBUG:boto:auth_path=/ibtljn-55/h?versionId=null 2023-11-02T18:24:09.553 DEBUG:boto:Method: GET 2023-11-02T18:24:09.554 DEBUG:boto:Path: /ibtljn-55/h?versionId=null 2023-11-02T18:24:09.554 DEBUG:boto:Data: 2023-11-02T18:24:09.555 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.555 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.555 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.556 DEBUG:boto:Params: {} 2023-11-02T18:24:09.556 DEBUG:boto:Token: None 2023-11-02T18:24:09.557 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/h?versionId=null 2023-11-02T18:24:09.557 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY= 2023-11-02T18:24:09.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY=', 'Content-Length': '0'} 2023-11-02T18:24:09.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a30270155e11d976-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.561 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:09.562 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:09.562 DEBUG:boto:path=/ibtljn-55/?prefix=h 2023-11-02T18:24:09.562 DEBUG:boto:auth_path=/ibtljn-55/?prefix=h 2023-11-02T18:24:09.563 DEBUG:boto:Method: GET 2023-11-02T18:24:09.563 DEBUG:boto:Path: /ibtljn-55/?prefix=h 2023-11-02T18:24:09.564 DEBUG:boto:Data: 2023-11-02T18:24:09.564 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.565 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.565 DEBUG:boto:Params: {} 2023-11-02T18:24:09.566 DEBUG:boto:Token: None 2023-11-02T18:24:09.566 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/ 2023-11-02T18:24:09.567 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T18:24:09.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T18:24:09.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004921f3e983a24f9b-006543e949-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.571 DEBUG:boto:b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:09.572 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:09.572 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:09.573 DEBUG:boto:path=/ibtljn-55/?prefix=h 2023-11-02T18:24:09.573 DEBUG:boto:auth_path=/ibtljn-55/?prefix=h 2023-11-02T18:24:09.574 DEBUG:boto:Method: GET 2023-11-02T18:24:09.574 DEBUG:boto:Path: /ibtljn-55/?prefix=h 2023-11-02T18:24:09.574 DEBUG:boto:Data: 2023-11-02T18:24:09.575 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.575 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.576 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.576 DEBUG:boto:Params: {} 2023-11-02T18:24:09.576 DEBUG:boto:Token: None 2023-11-02T18:24:09.577 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/ 2023-11-02T18:24:09.577 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T18:24:09.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T18:24:09.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c1c992422a30512-006543e949-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.581 DEBUG:boto:b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:09.581 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:24:09.582 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:24:09.582 DEBUG:boto:Method: HEAD 2023-11-02T18:24:09.583 DEBUG:boto:Path: /ibtljn-55/h 2023-11-02T18:24:09.583 DEBUG:boto:Data: 2023-11-02T18:24:09.583 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.584 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.584 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.585 DEBUG:boto:Params: {} 2023-11-02T18:24:09.585 DEBUG:boto:Token: None 2023-11-02T18:24:09.586 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/h 2023-11-02T18:24:09.586 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GdQAIQYhsstIIsb0AZ+xbmZuZb0= 2023-11-02T18:24:09.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GdQAIQYhsstIIsb0AZ+xbmZuZb0=', 'Content-Length': '0'} 2023-11-02T18:24:09.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b67315fba1ad0b2-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.590 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:24:09.590 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:24:09.590 DEBUG:boto:Method: HEAD 2023-11-02T18:24:09.591 DEBUG:boto:Path: /ibtljn-55/h 2023-11-02T18:24:09.591 DEBUG:boto:Data: 2023-11-02T18:24:09.592 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.592 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.592 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.593 DEBUG:boto:Params: {} 2023-11-02T18:24:09.593 DEBUG:boto:Token: None 2023-11-02T18:24:09.594 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/h 2023-11-02T18:24:09.594 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GdQAIQYhsstIIsb0AZ+xbmZuZb0= 2023-11-02T18:24:09.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GdQAIQYhsstIIsb0AZ+xbmZuZb0=', 'Content-Length': '0'} 2023-11-02T18:24:09.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000804f63bdb6ff1c6e-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-11-02T18:24:09.598 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:24:09.599 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:24:09.599 DEBUG:boto:Method: GET 2023-11-02T18:24:09.600 DEBUG:boto:Path: /ibtljn-55/h 2023-11-02T18:24:09.600 DEBUG:boto:Data: 2023-11-02T18:24:09.600 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.601 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.601 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.602 DEBUG:boto:Params: {} 2023-11-02T18:24:09.602 DEBUG:boto:Token: None 2023-11-02T18:24:09.603 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/h 2023-11-02T18:24:09.603 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8kH4kdXAxv4spFGgabodqcmIhcA= 2023-11-02T18:24:09.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8kH4kdXAxv4spFGgabodqcmIhcA=', 'Content-Length': '0'} 2023-11-02T18:24:09.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5c983f7aacad558-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.650 DEBUG:boto:path=/ibtljn-55/h 2023-11-02T18:24:09.651 DEBUG:boto:auth_path=/ibtljn-55/h 2023-11-02T18:24:09.651 DEBUG:boto:Method: GET 2023-11-02T18:24:09.652 DEBUG:boto:Path: /ibtljn-55/h 2023-11-02T18:24:09.652 DEBUG:boto:Data: 2023-11-02T18:24:09.653 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.653 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.654 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.654 DEBUG:boto:Params: {} 2023-11-02T18:24:09.655 DEBUG:boto:Token: None 2023-11-02T18:24:09.655 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/h 2023-11-02T18:24:09.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8kH4kdXAxv4spFGgabodqcmIhcA= 2023-11-02T18:24:09.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8kH4kdXAxv4spFGgabodqcmIhcA=', 'Content-Length': '0'} 2023-11-02T18:24:09.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000616dede86a86293d-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.660 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:09.660 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:09.661 DEBUG:boto:path=/ibtljn-55/?prefix=h 2023-11-02T18:24:09.661 DEBUG:boto:auth_path=/ibtljn-55/?prefix=h 2023-11-02T18:24:09.661 DEBUG:boto:Method: GET 2023-11-02T18:24:09.662 DEBUG:boto:Path: /ibtljn-55/?prefix=h 2023-11-02T18:24:09.662 DEBUG:boto:Data: 2023-11-02T18:24:09.663 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:09.664 DEBUG:boto:Port: 8001 2023-11-02T18:24:09.664 DEBUG:boto:Params: {} 2023-11-02T18:24:09.664 DEBUG:boto:Token: None 2023-11-02T18:24:09.665 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/ 2023-11-02T18:24:09.665 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T18:24:09.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T18:24:09.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d5e707230d82ce8-006543e949-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.672 DEBUG:boto:b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:09.673 DEBUG:boto:path=/ibtljn-55/ 2023-11-02T18:24:09.673 DEBUG:boto:auth_path=/ibtljn-55/ 2023-11-02T18:24:09.673 DEBUG:boto:path=/ibtljn-55/?prefix=h 2023-11-02T18:24:09.674 DEBUG:boto:auth_path=/ibtljn-55/?prefix=h 2023-11-02T18:24:09.674 DEBUG:boto:Method: GET 2023-11-02T18:24:09.675 DEBUG:boto:Path: /ibtljn-55/?prefix=h 2023-11-02T18:24:09.675 DEBUG:boto:Data: 2023-11-02T18:24:09.675 DEBUG:boto:Headers: {} 2023-11-02T18:24:09.676 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:09.676 DEBUG:boto:Port: 8000 2023-11-02T18:24:09.677 DEBUG:boto:Params: {} 2023-11-02T18:24:09.677 DEBUG:boto:Token: None 2023-11-02T18:24:09.678 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:09 GMT /ibtljn-55/ 2023-11-02T18:24:09.678 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T18:24:09.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T18:24:09.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7020fdcd4e438a-006543e949-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:09.730 DEBUG:boto:b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:09.731 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-55 zones={a2, a1} 2023-11-02T18:24:09.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:24:09.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:24:09.733 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:24:09.775 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:24:09.775 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:24:09.867 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:09.860+0000 7f6d28b57700 -1 req 6474304447377707186 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-19:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.19,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-19:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.19,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:09.874 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:09.864+0000 7f6d6cbdf700 -1 req 9826087319285350072 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-28:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.28,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-28:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.28,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:09.880 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:09.872+0000 7f6d67bd5700 -1 req 13240611044736563099 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-27:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.27,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-27:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.27,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:09.889 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:09.884+0000 7f6cfd300700 -1 req 11955232559906362647 0.003999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-43:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.8,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-43:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.8,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:09.890 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:09.884+0000 7f6d12b2b700 -1 req 14260232747133502469 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-29:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.29,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-29:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.29,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:09.891 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:09.884+0000 7f6de1cc9700 -1 req 976322668675659743 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-3:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.3,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-3:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.3,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:09.893 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:09.888+0000 7f6cf3aed700 -1 req 15793413793908589666 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-42:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.7,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-42:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.7,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:09.894 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:09.888+0000 7f6da6452700 -1 req 2843339270896427370 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-2:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.2,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-2:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.2,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:09.894 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:09.888+0000 7f6d1ab3b700 -1 req 3204166254874922139 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-36:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.1,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-36:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.1,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:09.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:24:09.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:24:09.967 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ibtljn-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:24:10.050 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:24:10.050 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:24:10.134 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:24:10.135 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:24:10.135 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding": "None", 2023-11-02T18:24:10.135 INFO:teuthology.orchestra.run.smithi017.stdout: "current_index": { 2023-11-02T18:24:10.135 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 2, 2023-11-02T18:24:10.135 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:24:10.136 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "Normal", 2023-11-02T18:24:10.136 INFO:teuthology.orchestra.run.smithi017.stdout: "normal": { 2023-11-02T18:24:10.136 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 3, 2023-11-02T18:24:10.136 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:24:10.136 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.137 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.137 INFO:teuthology.orchestra.run.smithi017.stdout: "logs": [ 2023-11-02T18:24:10.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:24:10.137 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 1, 2023-11-02T18:24:10.138 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:24:10.138 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "InIndex", 2023-11-02T18:24:10.138 INFO:teuthology.orchestra.run.smithi017.stdout: "in_index": { 2023-11-02T18:24:10.138 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 1, 2023-11-02T18:24:10.139 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:24:10.139 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 3, 2023-11-02T18:24:10.139 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:24:10.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:24:10.140 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 2, 2023-11-02T18:24:10.140 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:24:10.140 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "InIndex", 2023-11-02T18:24:10.141 INFO:teuthology.orchestra.run.smithi017.stdout: "in_index": { 2023-11-02T18:24:10.141 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 2, 2023-11-02T18:24:10.141 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:24:10.141 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 3, 2023-11-02T18:24:10.141 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:24:10.142 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.142 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.142 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.142 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.142 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:24:10.143 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.143 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:24:10.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:24:10.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:24:10.145 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:24:10.186 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:24:10.186 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:24:10.273 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:10.268+0000 7f6dce4a2700 -1 req 6348831972394980737 0.003999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-20:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.20,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-20:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.20,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:10.274 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:10.268+0000 7f6dcac9b700 -1 req 10187714122627611485 0.003999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-41:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.6,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-41:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.6,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:10.278 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:10.272+0000 7f6dc9c99700 -1 req 16383045101260734179 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-21:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.21,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-21:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.21,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:10.281 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:10.276+0000 7f6de3ccd700 -1 req 13267280008896322252 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-1:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.1,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-1:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.1,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:10.283 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:10.276+0000 7f6dc0486700 -1 req 8184490768704125574 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-4:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.4,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-4:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.4,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:10.284 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:10.276+0000 7f6d8ec23700 -1 req 13835577623689498269 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-5:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.5,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-5:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.5,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:10.284 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:10.276+0000 7f6d8fc25700 -1 req 5231040344823808504 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-40:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.5,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-40:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.5,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:10.285 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:10.280+0000 7f6dd3cad700 -1 req 17837592990121390711 0.003999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-25:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.25,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-25:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.25,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:10.286 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:24:10.280+0000 7f6d8a41a700 -1 req 13427702441716128958 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-22:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.22,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-22:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.22,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T18:24:10.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:24:10.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:24:10.385 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ibtljn-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:24:10.427 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:24:10.428 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:24:10.516 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:24:10.517 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:24:10.517 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding": "None", 2023-11-02T18:24:10.517 INFO:teuthology.orchestra.run.smithi017.stdout: "current_index": { 2023-11-02T18:24:10.517 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 2, 2023-11-02T18:24:10.518 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:24:10.518 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "Normal", 2023-11-02T18:24:10.518 INFO:teuthology.orchestra.run.smithi017.stdout: "normal": { 2023-11-02T18:24:10.518 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 3, 2023-11-02T18:24:10.518 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:24:10.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.519 INFO:teuthology.orchestra.run.smithi017.stdout: "logs": [ 2023-11-02T18:24:10.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:24:10.520 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 1, 2023-11-02T18:24:10.520 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:24:10.520 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "InIndex", 2023-11-02T18:24:10.520 INFO:teuthology.orchestra.run.smithi017.stdout: "in_index": { 2023-11-02T18:24:10.521 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 1, 2023-11-02T18:24:10.521 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:24:10.521 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 3, 2023-11-02T18:24:10.521 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:24:10.521 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.521 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:24:10.522 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 2, 2023-11-02T18:24:10.523 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:24:10.523 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "InIndex", 2023-11-02T18:24:10.523 INFO:teuthology.orchestra.run.smithi017.stdout: "in_index": { 2023-11-02T18:24:10.523 INFO:teuthology.orchestra.run.smithi017.stdout: "gen": 2, 2023-11-02T18:24:10.523 INFO:teuthology.orchestra.run.smithi017.stdout: "layout": { 2023-11-02T18:24:10.523 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 3, 2023-11-02T18:24:10.524 INFO:teuthology.orchestra.run.smithi017.stdout: "hash_type": "Mod" 2023-11-02T18:24:10.524 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.524 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.524 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.524 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.525 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:24:10.525 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.525 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:24:10.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2023-11-02T18:24:10.530 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:10.531 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:10.531 DEBUG:boto:Method: PUT 2023-11-02T18:24:10.532 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:24:10.532 DEBUG:boto:Data: 2023-11-02T18:24:10.532 DEBUG:boto:Headers: {} 2023-11-02T18:24:10.533 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:10.533 DEBUG:boto:Port: 8000 2023-11-02T18:24:10.534 DEBUG:boto:Params: {} 2023-11-02T18:24:10.534 DEBUG:boto:Token: None 2023-11-02T18:24:10.535 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:24:10 GMT /ibtljn-56/ 2023-11-02T18:24:10.535 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N7RXMh+t15q7YBjjoiLjF8Eh2vM= 2023-11-02T18:24:10.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N7RXMh+t15q7YBjjoiLjF8Eh2vM=', 'Content-Length': '0'} 2023-11-02T18:24:10.566 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b9d501e3378cc211-006543e94a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:10.566 DEBUG:rgw_multi.tests:created bucket=ibtljn-56 2023-11-02T18:24:10.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:24:10.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:24:10.568 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:24:10.617 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:24:10.618 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:24:10.687 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:24:10.706 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:24:10.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:24:10.707 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:24:10.707 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:24:10.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:24:10.707 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:24:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:24:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:24:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949450.561977_433.1", 2023-11-02T18:24:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:24:10.561977Z" 2023-11-02T18:24:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:24:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:24:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:24:10.710 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.710 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:24:10.711 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949450.561977_433.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:24:10.711 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:24:10.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:24:10.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:24:10.713 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:24:10.754 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:24:10.755 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:24:10.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:10.804+0000 7fdd1fbdbf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:24:10.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:10.808+0000 7fdd1fbdbf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:24:10.840 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:24:10.841 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:24:10.841 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:24:10.841 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:24:10.841 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:24:10.842 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:24:10.842 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:24:10.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.842 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:24:10.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:24:10.843 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:24:10.843 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:24:10.843 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:24:10.843 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:24:10.844 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:24:10.844 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:24:10.844 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:24:10.844 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:24:10.845 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:24:10.845 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:24:10.845 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:24:10.845 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:24:10.846 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:24:10.846 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:24:10.846 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:24:10.846 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:24:10.847 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:24:10.847 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:24:10.847 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:24:10.847 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:24:10.848 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:24:10.848 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:24:10.848 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:24:10.849 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949450.561977_433.1", 2023-11-02T18:24:10.849 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:24:10.849 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:24:10.849 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:24:10.849 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:24:10.561977Z", 2023-11-02T18:24:10.850 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:24:10.850 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:24:10.851 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:24:10.851 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:24:10.851 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:24:10.851 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:24:10.851 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:24:10.852 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:24:10.852 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:24:10.852 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:24:10.852 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:24:10.852 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:24:10.853 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:24:10.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.853 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:24:10.853 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:24:10.854 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:24:10.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:24:10.854 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:24:10Z" 2023-11-02T18:24:10.854 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:24:10.855 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949450.561977_433.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:24:10.561977Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:24:10Z" } 2023-11-02T18:24:10.856 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949450.561977_433.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:24:10.561977Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:24:10.856 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949450.561977_433.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:24:10.857 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949450.561977_433.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:24:10.857 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:24:10.858 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:10.858 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:10.858 DEBUG:boto:Method: HEAD 2023-11-02T18:24:10.859 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:24:10.860 DEBUG:boto:Data: 2023-11-02T18:24:10.860 DEBUG:boto:Headers: {} 2023-11-02T18:24:10.861 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:10.861 DEBUG:boto:Port: 8000 2023-11-02T18:24:10.862 DEBUG:boto:Params: {} 2023-11-02T18:24:10.862 DEBUG:boto:Token: None 2023-11-02T18:24:10.863 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:10 GMT /ibtljn-56/ 2023-11-02T18:24:10.863 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pwLofVCVurymIubReXzCy1ptUfA= 2023-11-02T18:24:10.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pwLofVCVurymIubReXzCy1ptUfA=', 'Content-Length': '0'} 2023-11-02T18:24:10.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000200b9af433a64ab2-006543e94a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:10.868 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:24:10.869 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:24:10.869 DEBUG:boto:Method: PUT 2023-11-02T18:24:10.870 DEBUG:boto:Path: /ibtljn-56/a 2023-11-02T18:24:10.870 DEBUG:boto:Data: 2023-11-02T18:24:10.871 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:24:10.871 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:10.872 DEBUG:boto:Port: 8000 2023-11-02T18:24:10.872 DEBUG:boto:Params: {} 2023-11-02T18:24:10.873 DEBUG:boto:Token: None 2023-11-02T18:24:10.873 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:24:10 GMT /ibtljn-56/a 2023-11-02T18:24:10.874 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DMZB6wpFVZQZOIygmVcNf/lfUIY= 2023-11-02T18:24:10.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:24:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DMZB6wpFVZQZOIygmVcNf/lfUIY='} 2023-11-02T18:24:10.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009dec58fd97de20f4-006543e94a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:10.889 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:10.890 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:10.890 DEBUG:boto:Method: HEAD 2023-11-02T18:24:10.891 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:24:10.891 DEBUG:boto:Data: 2023-11-02T18:24:10.891 DEBUG:boto:Headers: {} 2023-11-02T18:24:10.892 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:10.892 DEBUG:boto:Port: 8000 2023-11-02T18:24:10.893 DEBUG:boto:Params: {} 2023-11-02T18:24:10.893 DEBUG:boto:Token: None 2023-11-02T18:24:10.894 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:10 GMT /ibtljn-56/ 2023-11-02T18:24:10.894 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pwLofVCVurymIubReXzCy1ptUfA= 2023-11-02T18:24:10.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pwLofVCVurymIubReXzCy1ptUfA=', 'Content-Length': '0'} 2023-11-02T18:24:10.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000399020b7e4685b87-006543e94a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:10.899 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:24:10.899 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:24:10.900 DEBUG:boto:Method: PUT 2023-11-02T18:24:10.900 DEBUG:boto:Path: /ibtljn-56/b 2023-11-02T18:24:10.901 DEBUG:boto:Data: 2023-11-02T18:24:10.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:24:10.901 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:10.902 DEBUG:boto:Port: 8000 2023-11-02T18:24:10.902 DEBUG:boto:Params: {} 2023-11-02T18:24:10.903 DEBUG:boto:Token: None 2023-11-02T18:24:10.903 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:24:10 GMT /ibtljn-56/b 2023-11-02T18:24:10.904 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aCWVRZGTV8hlIcyqgoPowD0NiP4= 2023-11-02T18:24:10.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:24:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aCWVRZGTV8hlIcyqgoPowD0NiP4='} 2023-11-02T18:24:10.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec585731862b4d5f-006543e94a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:10.947 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:10.948 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:10.948 DEBUG:boto:Method: HEAD 2023-11-02T18:24:10.949 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:24:10.949 DEBUG:boto:Data: 2023-11-02T18:24:10.949 DEBUG:boto:Headers: {} 2023-11-02T18:24:10.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:10.950 DEBUG:boto:Port: 8000 2023-11-02T18:24:10.951 DEBUG:boto:Params: {} 2023-11-02T18:24:10.951 DEBUG:boto:Token: None 2023-11-02T18:24:10.952 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:10 GMT /ibtljn-56/ 2023-11-02T18:24:10.952 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pwLofVCVurymIubReXzCy1ptUfA= 2023-11-02T18:24:10.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pwLofVCVurymIubReXzCy1ptUfA=', 'Content-Length': '0'} 2023-11-02T18:24:10.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000212d46f5c266c8ca-006543e94a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:10.958 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:24:10.959 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:24:10.959 DEBUG:boto:Method: PUT 2023-11-02T18:24:10.959 DEBUG:boto:Path: /ibtljn-56/c 2023-11-02T18:24:10.960 DEBUG:boto:Data: 2023-11-02T18:24:10.960 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:24:10.961 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:10.961 DEBUG:boto:Port: 8000 2023-11-02T18:24:10.962 DEBUG:boto:Params: {} 2023-11-02T18:24:10.962 DEBUG:boto:Token: None 2023-11-02T18:24:10.963 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:24:10 GMT /ibtljn-56/c 2023-11-02T18:24:10.963 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BncvCVz+LSBZPCYFBTb5uexpbmQ= 2023-11-02T18:24:10.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:24:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BncvCVz+LSBZPCYFBTb5uexpbmQ='} 2023-11-02T18:24:10.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3f283e85d9d8499-006543e94a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:10.977 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:10.977 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:10.977 DEBUG:boto:Method: HEAD 2023-11-02T18:24:10.978 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:24:10.978 DEBUG:boto:Data: 2023-11-02T18:24:10.979 DEBUG:boto:Headers: {} 2023-11-02T18:24:10.979 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:10.979 DEBUG:boto:Port: 8000 2023-11-02T18:24:10.980 DEBUG:boto:Params: {} 2023-11-02T18:24:10.980 DEBUG:boto:Token: None 2023-11-02T18:24:10.981 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:10 GMT /ibtljn-56/ 2023-11-02T18:24:10.981 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pwLofVCVurymIubReXzCy1ptUfA= 2023-11-02T18:24:10.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pwLofVCVurymIubReXzCy1ptUfA=', 'Content-Length': '0'} 2023-11-02T18:24:10.987 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c63b010c7e9b0eef-006543e94a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:10.987 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:24:10.988 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:24:10.988 DEBUG:boto:Method: PUT 2023-11-02T18:24:10.989 DEBUG:boto:Path: /ibtljn-56/d 2023-11-02T18:24:10.989 DEBUG:boto:Data: 2023-11-02T18:24:10.989 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:24:10.990 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:10.990 DEBUG:boto:Port: 8000 2023-11-02T18:24:10.991 DEBUG:boto:Params: {} 2023-11-02T18:24:10.991 DEBUG:boto:Token: None 2023-11-02T18:24:10.993 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:24:10 GMT /ibtljn-56/d 2023-11-02T18:24:10.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qOn5ebyUi4FLZHiYtxMU4a4Eco4= 2023-11-02T18:24:10.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:24:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qOn5ebyUi4FLZHiYtxMU4a4Eco4='} 2023-11-02T18:24:11.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014f844589abd623d-006543e94a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:11.039 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-56 2023-11-02T18:24:11.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:24:11.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:24:11.041 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:24:11.095 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:24:11.096 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:24:11.161 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:11.156+0000 7f2bf9bdaf40 1 waiting to reach incremental sync.. 2023-11-02T18:24:16.162 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:16.156+0000 7f2bf9bdaf40 1 waiting to reach incremental sync.. 2023-11-02T18:24:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:21.156+0000 7f2bf9bdaf40 1 waiting to reach incremental sync.. 2023-11-02T18:24:26.164 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:26.156+0000 7f2bf9bdaf40 1 waiting to reach incremental sync.. 2023-11-02T18:24:31.165 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:31.160+0000 7f2bf9bdaf40 1 waiting to reach incremental sync.. 2023-11-02T18:24:36.166 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:36.160+0000 7f2bf9bdaf40 1 waiting to reach incremental sync.. 2023-11-02T18:24:41.167 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:41.160+0000 7f2bf9bdaf40 1 waiting to reach incremental sync.. 2023-11-02T18:24:46.171 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:46.164+0000 7f2bf9bdaf40 1 bucket sync caught up with source: 2023-11-02T18:24:46.172 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.35.6, , , , 00000000001.40.6, , , 00000000001.58.6, , 00000000001.75.6] 2023-11-02T18:24:46.172 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.35.6, , , , 00000000001.40.6, , , 00000000001.58.6, , 00000000001.75.6] 2023-11-02T18:24:46.172 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:46.164+0000 7f2bf9bdaf40 0 bucket checkpoint complete 2023-11-02T18:24:46.201 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-56 2023-11-02T18:24:46.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:24:46.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:24:46.203 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:24:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:24:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:24:46.325 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:46.316+0000 7fc5027f7f40 1 bucket sync caught up with source: 2023-11-02T18:24:46.325 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.189.6, , , , 00000000001.82.6, , , 00000000001.254.6, , 00000000001.211.6] 2023-11-02T18:24:46.325 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.189.6, , , , 00000000001.82.6, , , 00000000001.254.6, , 00000000001.211.6] 2023-11-02T18:24:46.325 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:46.316+0000 7fc5027f7f40 0 bucket checkpoint complete 2023-11-02T18:24:46.352 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-56 zones={a2, a1} 2023-11-02T18:24:46.353 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.354 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.354 DEBUG:boto:Method: HEAD 2023-11-02T18:24:46.354 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:24:46.355 DEBUG:boto:Data: 2023-11-02T18:24:46.355 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.356 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.356 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.357 DEBUG:boto:Params: {} 2023-11-02T18:24:46.357 DEBUG:boto:Token: None 2023-11-02T18:24:46.358 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/ 2023-11-02T18:24:46.358 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dwOStEQb0ddM4l2o0dCfc/rRMrA= 2023-11-02T18:24:46.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dwOStEQb0ddM4l2o0dCfc/rRMrA=', 'Content-Length': '0'} 2023-11-02T18:24:46.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fd4c5ffb4d4b54e-006543e96e-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.363 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.363 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.364 DEBUG:boto:Method: HEAD 2023-11-02T18:24:46.364 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:24:46.364 DEBUG:boto:Data: 2023-11-02T18:24:46.365 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.365 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.366 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.366 DEBUG:boto:Params: {} 2023-11-02T18:24:46.366 DEBUG:boto:Token: None 2023-11-02T18:24:46.367 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/ 2023-11-02T18:24:46.367 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dwOStEQb0ddM4l2o0dCfc/rRMrA= 2023-11-02T18:24:46.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dwOStEQb0ddM4l2o0dCfc/rRMrA=', 'Content-Length': '0'} 2023-11-02T18:24:46.372 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e3d3950120930a0-006543e96e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.372 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:24:46.373 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.373 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.374 DEBUG:boto:path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:24:46.374 DEBUG:boto:auth_path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:24:46.374 DEBUG:boto:Method: GET 2023-11-02T18:24:46.375 DEBUG:boto:Path: /ibtljn-56/?versions&max-keys=999 2023-11-02T18:24:46.375 DEBUG:boto:Data: 2023-11-02T18:24:46.376 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.376 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.376 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.377 DEBUG:boto:Params: {} 2023-11-02T18:24:46.377 DEBUG:boto:Token: None 2023-11-02T18:24:46.378 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/?versions 2023-11-02T18:24:46.378 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wk9mFijMzvTJanmqNzAA4MpNtsk= 2023-11-02T18:24:46.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wk9mFijMzvTJanmqNzAA4MpNtsk=', 'Content-Length': '0'} 2023-11-02T18:24:46.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a5eb1ee546e4034-006543e96e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.384 DEBUG:boto:b'ibtljn-56999falseanulltrue2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:46.385 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:24:46.385 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:24:46.385 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:24:46.386 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:24:46.386 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:24:46.387 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.387 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.388 DEBUG:boto:path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:24:46.388 DEBUG:boto:auth_path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:24:46.388 DEBUG:boto:Method: GET 2023-11-02T18:24:46.389 DEBUG:boto:Path: /ibtljn-56/?versions&max-keys=999 2023-11-02T18:24:46.389 DEBUG:boto:Data: 2023-11-02T18:24:46.390 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.390 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.390 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.391 DEBUG:boto:Params: {} 2023-11-02T18:24:46.391 DEBUG:boto:Token: None 2023-11-02T18:24:46.392 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/?versions 2023-11-02T18:24:46.392 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wk9mFijMzvTJanmqNzAA4MpNtsk= 2023-11-02T18:24:46.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wk9mFijMzvTJanmqNzAA4MpNtsk=', 'Content-Length': '0'} 2023-11-02T18:24:46.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4a4146f0de671bf-006543e96e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.398 DEBUG:boto:b'ibtljn-56999falseanulltrue2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:46.398 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:24:46.399 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:24:46.399 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:24:46.400 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:24:46.400 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.400 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.401 DEBUG:boto:path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:24:46.401 DEBUG:boto:auth_path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:24:46.402 DEBUG:boto:Method: GET 2023-11-02T18:24:46.402 DEBUG:boto:Path: /ibtljn-56/?versions&max-keys=999 2023-11-02T18:24:46.403 DEBUG:boto:Data: 2023-11-02T18:24:46.403 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.404 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.404 DEBUG:boto:Params: {} 2023-11-02T18:24:46.405 DEBUG:boto:Token: None 2023-11-02T18:24:46.405 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/?versions 2023-11-02T18:24:46.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wk9mFijMzvTJanmqNzAA4MpNtsk= 2023-11-02T18:24:46.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wk9mFijMzvTJanmqNzAA4MpNtsk=', 'Content-Length': '0'} 2023-11-02T18:24:46.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc606482c74ee548-006543e96e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.411 DEBUG:boto:b'ibtljn-56999falseanulltrue2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:46.411 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.412 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.412 DEBUG:boto:path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:24:46.413 DEBUG:boto:auth_path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:24:46.413 DEBUG:boto:Method: GET 2023-11-02T18:24:46.414 DEBUG:boto:Path: /ibtljn-56/?versions&max-keys=999 2023-11-02T18:24:46.414 DEBUG:boto:Data: 2023-11-02T18:24:46.414 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.415 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.415 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.416 DEBUG:boto:Params: {} 2023-11-02T18:24:46.416 DEBUG:boto:Token: None 2023-11-02T18:24:46.416 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/?versions 2023-11-02T18:24:46.417 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wk9mFijMzvTJanmqNzAA4MpNtsk= 2023-11-02T18:24:46.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wk9mFijMzvTJanmqNzAA4MpNtsk=', 'Content-Length': '0'} 2023-11-02T18:24:46.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b38538c1108ff01e-006543e96e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.423 DEBUG:boto:b'ibtljn-56999falseanulltrue2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:46.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:24:46.424 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:24:46.424 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:24:46.425 DEBUG:boto:path=/ibtljn-56/a?versionId=null 2023-11-02T18:24:46.425 DEBUG:boto:auth_path=/ibtljn-56/a?versionId=null 2023-11-02T18:24:46.425 DEBUG:boto:Method: GET 2023-11-02T18:24:46.426 DEBUG:boto:Path: /ibtljn-56/a?versionId=null 2023-11-02T18:24:46.426 DEBUG:boto:Data: 2023-11-02T18:24:46.427 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.427 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.428 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.428 DEBUG:boto:Params: {} 2023-11-02T18:24:46.429 DEBUG:boto:Token: None 2023-11-02T18:24:46.429 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/a?versionId=null 2023-11-02T18:24:46.429 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NjX8Igz7+ny/I4PkiOBkSBsR/4M= 2023-11-02T18:24:46.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NjX8Igz7+ny/I4PkiOBkSBsR/4M=', 'Content-Length': '0'} 2023-11-02T18:24:46.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000425157994b57e7d8-006543e96e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.474 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:24:46.475 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:24:46.475 DEBUG:boto:path=/ibtljn-56/a?versionId=null 2023-11-02T18:24:46.476 DEBUG:boto:auth_path=/ibtljn-56/a?versionId=null 2023-11-02T18:24:46.476 DEBUG:boto:Method: GET 2023-11-02T18:24:46.477 DEBUG:boto:Path: /ibtljn-56/a?versionId=null 2023-11-02T18:24:46.477 DEBUG:boto:Data: 2023-11-02T18:24:46.478 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.478 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.478 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.479 DEBUG:boto:Params: {} 2023-11-02T18:24:46.479 DEBUG:boto:Token: None 2023-11-02T18:24:46.480 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/a?versionId=null 2023-11-02T18:24:46.480 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NjX8Igz7+ny/I4PkiOBkSBsR/4M= 2023-11-02T18:24:46.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NjX8Igz7+ny/I4PkiOBkSBsR/4M=', 'Content-Length': '0'} 2023-11-02T18:24:46.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3a71fcd9398b314-006543e96e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.484 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:24:46.485 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:24:46.485 DEBUG:boto:path=/ibtljn-56/a?versionId=null 2023-11-02T18:24:46.486 DEBUG:boto:auth_path=/ibtljn-56/a?versionId=null 2023-11-02T18:24:46.486 DEBUG:boto:Method: HEAD 2023-11-02T18:24:46.487 DEBUG:boto:Path: /ibtljn-56/a?versionId=null 2023-11-02T18:24:46.487 DEBUG:boto:Data: 2023-11-02T18:24:46.487 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.488 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.488 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.489 DEBUG:boto:Params: {} 2023-11-02T18:24:46.489 DEBUG:boto:Token: None 2023-11-02T18:24:46.490 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/a?versionId=null 2023-11-02T18:24:46.490 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vXtcmaSy3KKm42Ot/Qjrxzyh45o= 2023-11-02T18:24:46.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vXtcmaSy3KKm42Ot/Qjrxzyh45o=', 'Content-Length': '0'} 2023-11-02T18:24:46.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b782d003fc44f694-006543e96e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.494 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:24:46.495 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:24:46.495 DEBUG:boto:path=/ibtljn-56/a?versionId=null 2023-11-02T18:24:46.495 DEBUG:boto:auth_path=/ibtljn-56/a?versionId=null 2023-11-02T18:24:46.496 DEBUG:boto:Method: HEAD 2023-11-02T18:24:46.496 DEBUG:boto:Path: /ibtljn-56/a?versionId=null 2023-11-02T18:24:46.497 DEBUG:boto:Data: 2023-11-02T18:24:46.497 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.497 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.498 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.498 DEBUG:boto:Params: {} 2023-11-02T18:24:46.499 DEBUG:boto:Token: None 2023-11-02T18:24:46.499 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/a?versionId=null 2023-11-02T18:24:46.500 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vXtcmaSy3KKm42Ot/Qjrxzyh45o= 2023-11-02T18:24:46.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vXtcmaSy3KKm42Ot/Qjrxzyh45o=', 'Content-Length': '0'} 2023-11-02T18:24:46.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008416475894f53c1c-006543e96e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:24:46.504 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:24:46.504 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:24:46.504 DEBUG:boto:path=/ibtljn-56/a?versionId=null 2023-11-02T18:24:46.505 DEBUG:boto:auth_path=/ibtljn-56/a?versionId=null 2023-11-02T18:24:46.505 DEBUG:boto:Method: GET 2023-11-02T18:24:46.506 DEBUG:boto:Path: /ibtljn-56/a?versionId=null 2023-11-02T18:24:46.506 DEBUG:boto:Data: 2023-11-02T18:24:46.507 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.507 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.508 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.508 DEBUG:boto:Params: {} 2023-11-02T18:24:46.509 DEBUG:boto:Token: None 2023-11-02T18:24:46.509 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/a?versionId=null 2023-11-02T18:24:46.509 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NjX8Igz7+ny/I4PkiOBkSBsR/4M= 2023-11-02T18:24:46.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NjX8Igz7+ny/I4PkiOBkSBsR/4M=', 'Content-Length': '0'} 2023-11-02T18:24:46.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c58d9bce91b68d4-006543e96e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.513 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:24:46.514 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:24:46.514 DEBUG:boto:path=/ibtljn-56/a?versionId=null 2023-11-02T18:24:46.515 DEBUG:boto:auth_path=/ibtljn-56/a?versionId=null 2023-11-02T18:24:46.515 DEBUG:boto:Method: GET 2023-11-02T18:24:46.516 DEBUG:boto:Path: /ibtljn-56/a?versionId=null 2023-11-02T18:24:46.516 DEBUG:boto:Data: 2023-11-02T18:24:46.517 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.517 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.518 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.518 DEBUG:boto:Params: {} 2023-11-02T18:24:46.519 DEBUG:boto:Token: None 2023-11-02T18:24:46.519 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/a?versionId=null 2023-11-02T18:24:46.520 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NjX8Igz7+ny/I4PkiOBkSBsR/4M= 2023-11-02T18:24:46.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NjX8Igz7+ny/I4PkiOBkSBsR/4M=', 'Content-Length': '0'} 2023-11-02T18:24:46.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d1fdb1958541374-006543e96e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.566 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.567 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.567 DEBUG:boto:path=/ibtljn-56/?prefix=a 2023-11-02T18:24:46.568 DEBUG:boto:auth_path=/ibtljn-56/?prefix=a 2023-11-02T18:24:46.568 DEBUG:boto:Method: GET 2023-11-02T18:24:46.569 DEBUG:boto:Path: /ibtljn-56/?prefix=a 2023-11-02T18:24:46.569 DEBUG:boto:Data: 2023-11-02T18:24:46.569 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.570 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.570 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.571 DEBUG:boto:Params: {} 2023-11-02T18:24:46.571 DEBUG:boto:Token: None 2023-11-02T18:24:46.572 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/ 2023-11-02T18:24:46.572 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU= 2023-11-02T18:24:46.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU=', 'Content-Length': '0'} 2023-11-02T18:24:46.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aae8129ba89daff1-006543e96e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.577 DEBUG:boto:b'ibtljn-56a1000falsea2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:46.577 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.578 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.578 DEBUG:boto:path=/ibtljn-56/?prefix=a 2023-11-02T18:24:46.579 DEBUG:boto:auth_path=/ibtljn-56/?prefix=a 2023-11-02T18:24:46.579 DEBUG:boto:Method: GET 2023-11-02T18:24:46.580 DEBUG:boto:Path: /ibtljn-56/?prefix=a 2023-11-02T18:24:46.580 DEBUG:boto:Data: 2023-11-02T18:24:46.581 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.581 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.582 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.582 DEBUG:boto:Params: {} 2023-11-02T18:24:46.583 DEBUG:boto:Token: None 2023-11-02T18:24:46.583 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/ 2023-11-02T18:24:46.584 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU= 2023-11-02T18:24:46.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU=', 'Content-Length': '0'} 2023-11-02T18:24:46.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e680c1448421a9df-006543e96e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.588 DEBUG:boto:b'ibtljn-56a1000falsea2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:46.588 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:24:46.589 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:24:46.589 DEBUG:boto:Method: HEAD 2023-11-02T18:24:46.590 DEBUG:boto:Path: /ibtljn-56/a 2023-11-02T18:24:46.590 DEBUG:boto:Data: 2023-11-02T18:24:46.591 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.591 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.591 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.592 DEBUG:boto:Params: {} 2023-11-02T18:24:46.592 DEBUG:boto:Token: None 2023-11-02T18:24:46.593 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/a 2023-11-02T18:24:46.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c4buX8EPHSV98aDLVPKRQFsd9JM= 2023-11-02T18:24:46.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c4buX8EPHSV98aDLVPKRQFsd9JM=', 'Content-Length': '0'} 2023-11-02T18:24:46.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa4807551495596c-006543e96e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.596 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:24:46.597 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:24:46.597 DEBUG:boto:Method: HEAD 2023-11-02T18:24:46.598 DEBUG:boto:Path: /ibtljn-56/a 2023-11-02T18:24:46.598 DEBUG:boto:Data: 2023-11-02T18:24:46.599 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.599 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.599 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.600 DEBUG:boto:Params: {} 2023-11-02T18:24:46.600 DEBUG:boto:Token: None 2023-11-02T18:24:46.601 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/a 2023-11-02T18:24:46.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c4buX8EPHSV98aDLVPKRQFsd9JM= 2023-11-02T18:24:46.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c4buX8EPHSV98aDLVPKRQFsd9JM=', 'Content-Length': '0'} 2023-11-02T18:24:46.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008770ae554fca3a09-006543e96e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:24:46.605 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:24:46.606 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:24:46.606 DEBUG:boto:Method: GET 2023-11-02T18:24:46.607 DEBUG:boto:Path: /ibtljn-56/a 2023-11-02T18:24:46.607 DEBUG:boto:Data: 2023-11-02T18:24:46.608 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.608 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.608 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.609 DEBUG:boto:Params: {} 2023-11-02T18:24:46.609 DEBUG:boto:Token: None 2023-11-02T18:24:46.610 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/a 2023-11-02T18:24:46.610 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+RTQGgobzCoP57SmDB7tecLfQ+Q= 2023-11-02T18:24:46.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+RTQGgobzCoP57SmDB7tecLfQ+Q=', 'Content-Length': '0'} 2023-11-02T18:24:46.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce0218249f34cbe-006543e96e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.614 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:24:46.615 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:24:46.615 DEBUG:boto:Method: GET 2023-11-02T18:24:46.616 DEBUG:boto:Path: /ibtljn-56/a 2023-11-02T18:24:46.616 DEBUG:boto:Data: 2023-11-02T18:24:46.616 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.617 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.617 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.618 DEBUG:boto:Params: {} 2023-11-02T18:24:46.618 DEBUG:boto:Token: None 2023-11-02T18:24:46.619 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/a 2023-11-02T18:24:46.619 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+RTQGgobzCoP57SmDB7tecLfQ+Q= 2023-11-02T18:24:46.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+RTQGgobzCoP57SmDB7tecLfQ+Q=', 'Content-Length': '0'} 2023-11-02T18:24:46.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d83193cfdd0d8bc-006543e96e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.623 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.623 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.623 DEBUG:boto:path=/ibtljn-56/?prefix=a 2023-11-02T18:24:46.624 DEBUG:boto:auth_path=/ibtljn-56/?prefix=a 2023-11-02T18:24:46.624 DEBUG:boto:Method: GET 2023-11-02T18:24:46.625 DEBUG:boto:Path: /ibtljn-56/?prefix=a 2023-11-02T18:24:46.625 DEBUG:boto:Data: 2023-11-02T18:24:46.626 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.626 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.626 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.627 DEBUG:boto:Params: {} 2023-11-02T18:24:46.627 DEBUG:boto:Token: None 2023-11-02T18:24:46.628 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/ 2023-11-02T18:24:46.628 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU= 2023-11-02T18:24:46.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU=', 'Content-Length': '0'} 2023-11-02T18:24:46.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8e0e69667056c03-006543e96e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.633 DEBUG:boto:b'ibtljn-56a1000falsea2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:46.634 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.634 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.635 DEBUG:boto:path=/ibtljn-56/?prefix=a 2023-11-02T18:24:46.635 DEBUG:boto:auth_path=/ibtljn-56/?prefix=a 2023-11-02T18:24:46.637 DEBUG:boto:Method: GET 2023-11-02T18:24:46.637 DEBUG:boto:Path: /ibtljn-56/?prefix=a 2023-11-02T18:24:46.638 DEBUG:boto:Data: 2023-11-02T18:24:46.638 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.639 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.640 DEBUG:boto:Params: {} 2023-11-02T18:24:46.640 DEBUG:boto:Token: None 2023-11-02T18:24:46.641 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/ 2023-11-02T18:24:46.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU= 2023-11-02T18:24:46.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU=', 'Content-Length': '0'} 2023-11-02T18:24:46.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034e0909532e70540-006543e96e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.647 DEBUG:boto:b'ibtljn-56a1000falsea2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:46.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:24:46.648 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:24:46.649 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:24:46.649 DEBUG:boto:path=/ibtljn-56/b?versionId=null 2023-11-02T18:24:46.649 DEBUG:boto:auth_path=/ibtljn-56/b?versionId=null 2023-11-02T18:24:46.650 DEBUG:boto:Method: GET 2023-11-02T18:24:46.650 DEBUG:boto:Path: /ibtljn-56/b?versionId=null 2023-11-02T18:24:46.651 DEBUG:boto:Data: 2023-11-02T18:24:46.651 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.652 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.652 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.652 DEBUG:boto:Params: {} 2023-11-02T18:24:46.653 DEBUG:boto:Token: None 2023-11-02T18:24:46.653 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/b?versionId=null 2023-11-02T18:24:46.654 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OhQJdbi8ENbch8eDp+d+zOb4rug= 2023-11-02T18:24:46.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OhQJdbi8ENbch8eDp+d+zOb4rug=', 'Content-Length': '0'} 2023-11-02T18:24:46.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dc6d9927cd294b1-006543e96e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.698 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:24:46.699 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:24:46.699 DEBUG:boto:path=/ibtljn-56/b?versionId=null 2023-11-02T18:24:46.700 DEBUG:boto:auth_path=/ibtljn-56/b?versionId=null 2023-11-02T18:24:46.700 DEBUG:boto:Method: GET 2023-11-02T18:24:46.701 DEBUG:boto:Path: /ibtljn-56/b?versionId=null 2023-11-02T18:24:46.701 DEBUG:boto:Data: 2023-11-02T18:24:46.701 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.702 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.702 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.703 DEBUG:boto:Params: {} 2023-11-02T18:24:46.703 DEBUG:boto:Token: None 2023-11-02T18:24:46.704 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/b?versionId=null 2023-11-02T18:24:46.704 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OhQJdbi8ENbch8eDp+d+zOb4rug= 2023-11-02T18:24:46.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OhQJdbi8ENbch8eDp+d+zOb4rug=', 'Content-Length': '0'} 2023-11-02T18:24:46.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000703576f8cb578a51-006543e96e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.708 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:24:46.709 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:24:46.709 DEBUG:boto:path=/ibtljn-56/b?versionId=null 2023-11-02T18:24:46.709 DEBUG:boto:auth_path=/ibtljn-56/b?versionId=null 2023-11-02T18:24:46.710 DEBUG:boto:Method: HEAD 2023-11-02T18:24:46.710 DEBUG:boto:Path: /ibtljn-56/b?versionId=null 2023-11-02T18:24:46.711 DEBUG:boto:Data: 2023-11-02T18:24:46.711 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.712 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.712 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.712 DEBUG:boto:Params: {} 2023-11-02T18:24:46.713 DEBUG:boto:Token: None 2023-11-02T18:24:46.713 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/b?versionId=null 2023-11-02T18:24:46.714 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YEA7qHKl2LQ3tbM/wocWWCtbDQo= 2023-11-02T18:24:46.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YEA7qHKl2LQ3tbM/wocWWCtbDQo=', 'Content-Length': '0'} 2023-11-02T18:24:46.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e6d6f556b9ca8ba-006543e96e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.718 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:24:46.718 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:24:46.718 DEBUG:boto:path=/ibtljn-56/b?versionId=null 2023-11-02T18:24:46.719 DEBUG:boto:auth_path=/ibtljn-56/b?versionId=null 2023-11-02T18:24:46.719 DEBUG:boto:Method: HEAD 2023-11-02T18:24:46.720 DEBUG:boto:Path: /ibtljn-56/b?versionId=null 2023-11-02T18:24:46.720 DEBUG:boto:Data: 2023-11-02T18:24:46.720 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.721 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.721 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.722 DEBUG:boto:Params: {} 2023-11-02T18:24:46.722 DEBUG:boto:Token: None 2023-11-02T18:24:46.723 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/b?versionId=null 2023-11-02T18:24:46.723 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YEA7qHKl2LQ3tbM/wocWWCtbDQo= 2023-11-02T18:24:46.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YEA7qHKl2LQ3tbM/wocWWCtbDQo=', 'Content-Length': '0'} 2023-11-02T18:24:46.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002497c9edc0358ffb-006543e96e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:24:46.727 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:24:46.728 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:24:46.728 DEBUG:boto:path=/ibtljn-56/b?versionId=null 2023-11-02T18:24:46.728 DEBUG:boto:auth_path=/ibtljn-56/b?versionId=null 2023-11-02T18:24:46.729 DEBUG:boto:Method: GET 2023-11-02T18:24:46.729 DEBUG:boto:Path: /ibtljn-56/b?versionId=null 2023-11-02T18:24:46.730 DEBUG:boto:Data: 2023-11-02T18:24:46.730 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.731 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.731 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.731 DEBUG:boto:Params: {} 2023-11-02T18:24:46.732 DEBUG:boto:Token: None 2023-11-02T18:24:46.732 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/b?versionId=null 2023-11-02T18:24:46.733 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OhQJdbi8ENbch8eDp+d+zOb4rug= 2023-11-02T18:24:46.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OhQJdbi8ENbch8eDp+d+zOb4rug=', 'Content-Length': '0'} 2023-11-02T18:24:46.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce9e2fc4ac6a80c-006543e96e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.737 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:24:46.737 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:24:46.738 DEBUG:boto:path=/ibtljn-56/b?versionId=null 2023-11-02T18:24:46.738 DEBUG:boto:auth_path=/ibtljn-56/b?versionId=null 2023-11-02T18:24:46.739 DEBUG:boto:Method: GET 2023-11-02T18:24:46.739 DEBUG:boto:Path: /ibtljn-56/b?versionId=null 2023-11-02T18:24:46.740 DEBUG:boto:Data: 2023-11-02T18:24:46.740 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.740 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.741 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.741 DEBUG:boto:Params: {} 2023-11-02T18:24:46.742 DEBUG:boto:Token: None 2023-11-02T18:24:46.742 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/b?versionId=null 2023-11-02T18:24:46.743 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OhQJdbi8ENbch8eDp+d+zOb4rug= 2023-11-02T18:24:46.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OhQJdbi8ENbch8eDp+d+zOb4rug=', 'Content-Length': '0'} 2023-11-02T18:24:46.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f51808b62573ea8-006543e96e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.790 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.791 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.791 DEBUG:boto:path=/ibtljn-56/?prefix=b 2023-11-02T18:24:46.792 DEBUG:boto:auth_path=/ibtljn-56/?prefix=b 2023-11-02T18:24:46.792 DEBUG:boto:Method: GET 2023-11-02T18:24:46.793 DEBUG:boto:Path: /ibtljn-56/?prefix=b 2023-11-02T18:24:46.793 DEBUG:boto:Data: 2023-11-02T18:24:46.793 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.794 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.794 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.795 DEBUG:boto:Params: {} 2023-11-02T18:24:46.795 DEBUG:boto:Token: None 2023-11-02T18:24:46.796 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/ 2023-11-02T18:24:46.796 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU= 2023-11-02T18:24:46.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU=', 'Content-Length': '0'} 2023-11-02T18:24:46.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049f776bf834da452-006543e96e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.802 DEBUG:boto:b'ibtljn-56b1000falseb2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:46.802 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.803 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.803 DEBUG:boto:path=/ibtljn-56/?prefix=b 2023-11-02T18:24:46.804 DEBUG:boto:auth_path=/ibtljn-56/?prefix=b 2023-11-02T18:24:46.804 DEBUG:boto:Method: GET 2023-11-02T18:24:46.805 DEBUG:boto:Path: /ibtljn-56/?prefix=b 2023-11-02T18:24:46.805 DEBUG:boto:Data: 2023-11-02T18:24:46.805 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.806 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.806 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.807 DEBUG:boto:Params: {} 2023-11-02T18:24:46.807 DEBUG:boto:Token: None 2023-11-02T18:24:46.808 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/ 2023-11-02T18:24:46.808 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU= 2023-11-02T18:24:46.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU=', 'Content-Length': '0'} 2023-11-02T18:24:46.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000127c5d867bcf30fd-006543e96e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.814 DEBUG:boto:b'ibtljn-56b1000falseb2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:46.815 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:24:46.815 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:24:46.815 DEBUG:boto:Method: HEAD 2023-11-02T18:24:46.816 DEBUG:boto:Path: /ibtljn-56/b 2023-11-02T18:24:46.816 DEBUG:boto:Data: 2023-11-02T18:24:46.817 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.817 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.817 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.818 DEBUG:boto:Params: {} 2023-11-02T18:24:46.818 DEBUG:boto:Token: None 2023-11-02T18:24:46.819 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/b 2023-11-02T18:24:46.819 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J75ECSDILgdSoxcWB0lXpJdedZA= 2023-11-02T18:24:46.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J75ECSDILgdSoxcWB0lXpJdedZA=', 'Content-Length': '0'} 2023-11-02T18:24:46.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d90e33b08291b1a6-006543e96e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.823 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:24:46.823 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:24:46.824 DEBUG:boto:Method: HEAD 2023-11-02T18:24:46.824 DEBUG:boto:Path: /ibtljn-56/b 2023-11-02T18:24:46.824 DEBUG:boto:Data: 2023-11-02T18:24:46.825 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.825 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.826 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.826 DEBUG:boto:Params: {} 2023-11-02T18:24:46.826 DEBUG:boto:Token: None 2023-11-02T18:24:46.827 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/b 2023-11-02T18:24:46.827 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J75ECSDILgdSoxcWB0lXpJdedZA= 2023-11-02T18:24:46.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J75ECSDILgdSoxcWB0lXpJdedZA=', 'Content-Length': '0'} 2023-11-02T18:24:46.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068b845c710fab42e-006543e96e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:24:46.831 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:24:46.832 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:24:46.832 DEBUG:boto:Method: GET 2023-11-02T18:24:46.833 DEBUG:boto:Path: /ibtljn-56/b 2023-11-02T18:24:46.833 DEBUG:boto:Data: 2023-11-02T18:24:46.833 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.834 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.834 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.835 DEBUG:boto:Params: {} 2023-11-02T18:24:46.835 DEBUG:boto:Token: None 2023-11-02T18:24:46.836 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/b 2023-11-02T18:24:46.836 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zAiCQVVgEyKMgRMDqcR7CIxfppo= 2023-11-02T18:24:46.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zAiCQVVgEyKMgRMDqcR7CIxfppo=', 'Content-Length': '0'} 2023-11-02T18:24:46.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a184af4d1611c61f-006543e96e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.882 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:24:46.883 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:24:46.883 DEBUG:boto:Method: GET 2023-11-02T18:24:46.884 DEBUG:boto:Path: /ibtljn-56/b 2023-11-02T18:24:46.884 DEBUG:boto:Data: 2023-11-02T18:24:46.885 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.885 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.885 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.886 DEBUG:boto:Params: {} 2023-11-02T18:24:46.886 DEBUG:boto:Token: None 2023-11-02T18:24:46.887 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/b 2023-11-02T18:24:46.887 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zAiCQVVgEyKMgRMDqcR7CIxfppo= 2023-11-02T18:24:46.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zAiCQVVgEyKMgRMDqcR7CIxfppo=', 'Content-Length': '0'} 2023-11-02T18:24:46.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057ccc246a89bde2e-006543e96e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.892 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.892 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.892 DEBUG:boto:path=/ibtljn-56/?prefix=b 2023-11-02T18:24:46.893 DEBUG:boto:auth_path=/ibtljn-56/?prefix=b 2023-11-02T18:24:46.893 DEBUG:boto:Method: GET 2023-11-02T18:24:46.894 DEBUG:boto:Path: /ibtljn-56/?prefix=b 2023-11-02T18:24:46.894 DEBUG:boto:Data: 2023-11-02T18:24:46.895 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.895 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.895 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.896 DEBUG:boto:Params: {} 2023-11-02T18:24:46.896 DEBUG:boto:Token: None 2023-11-02T18:24:46.897 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/ 2023-11-02T18:24:46.897 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU= 2023-11-02T18:24:46.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU=', 'Content-Length': '0'} 2023-11-02T18:24:46.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004694c796892503ff-006543e96e-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.903 DEBUG:boto:b'ibtljn-56b1000falseb2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:46.904 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:46.904 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:46.905 DEBUG:boto:path=/ibtljn-56/?prefix=b 2023-11-02T18:24:46.905 DEBUG:boto:auth_path=/ibtljn-56/?prefix=b 2023-11-02T18:24:46.905 DEBUG:boto:Method: GET 2023-11-02T18:24:46.906 DEBUG:boto:Path: /ibtljn-56/?prefix=b 2023-11-02T18:24:46.906 DEBUG:boto:Data: 2023-11-02T18:24:46.907 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.907 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.908 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.908 DEBUG:boto:Params: {} 2023-11-02T18:24:46.909 DEBUG:boto:Token: None 2023-11-02T18:24:46.909 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/ 2023-11-02T18:24:46.910 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU= 2023-11-02T18:24:46.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXgGWopY2JsMQbjrmev66JKvjOU=', 'Content-Length': '0'} 2023-11-02T18:24:46.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097486e06cb3fc9be-006543e96e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.958 DEBUG:boto:b'ibtljn-56b1000falseb2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:46.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:24:46.960 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:24:46.960 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:24:46.960 DEBUG:boto:path=/ibtljn-56/c?versionId=null 2023-11-02T18:24:46.961 DEBUG:boto:auth_path=/ibtljn-56/c?versionId=null 2023-11-02T18:24:46.961 DEBUG:boto:Method: GET 2023-11-02T18:24:46.962 DEBUG:boto:Path: /ibtljn-56/c?versionId=null 2023-11-02T18:24:46.962 DEBUG:boto:Data: 2023-11-02T18:24:46.963 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.963 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.964 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.964 DEBUG:boto:Params: {} 2023-11-02T18:24:46.965 DEBUG:boto:Token: None 2023-11-02T18:24:46.965 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/c?versionId=null 2023-11-02T18:24:46.966 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pxyQhKs9p32djK7u0tIIE1Ckk68= 2023-11-02T18:24:46.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pxyQhKs9p32djK7u0tIIE1Ckk68=', 'Content-Length': '0'} 2023-11-02T18:24:46.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad4ec99169ee7784-006543e96e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.970 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:24:46.971 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:24:46.971 DEBUG:boto:path=/ibtljn-56/c?versionId=null 2023-11-02T18:24:46.971 DEBUG:boto:auth_path=/ibtljn-56/c?versionId=null 2023-11-02T18:24:46.972 DEBUG:boto:Method: GET 2023-11-02T18:24:46.972 DEBUG:boto:Path: /ibtljn-56/c?versionId=null 2023-11-02T18:24:46.973 DEBUG:boto:Data: 2023-11-02T18:24:46.973 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.973 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.974 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.974 DEBUG:boto:Params: {} 2023-11-02T18:24:46.975 DEBUG:boto:Token: None 2023-11-02T18:24:46.975 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/c?versionId=null 2023-11-02T18:24:46.976 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pxyQhKs9p32djK7u0tIIE1Ckk68= 2023-11-02T18:24:46.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pxyQhKs9p32djK7u0tIIE1Ckk68=', 'Content-Length': '0'} 2023-11-02T18:24:46.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053df1a06e8d077a6-006543e96e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.980 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:24:46.980 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:24:46.981 DEBUG:boto:path=/ibtljn-56/c?versionId=null 2023-11-02T18:24:46.981 DEBUG:boto:auth_path=/ibtljn-56/c?versionId=null 2023-11-02T18:24:46.981 DEBUG:boto:Method: HEAD 2023-11-02T18:24:46.982 DEBUG:boto:Path: /ibtljn-56/c?versionId=null 2023-11-02T18:24:46.982 DEBUG:boto:Data: 2023-11-02T18:24:46.983 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.983 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:46.984 DEBUG:boto:Port: 8001 2023-11-02T18:24:46.984 DEBUG:boto:Params: {} 2023-11-02T18:24:46.984 DEBUG:boto:Token: None 2023-11-02T18:24:46.985 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/c?versionId=null 2023-11-02T18:24:46.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:duAEEE+1dFgsFrxsptedZebsfDA= 2023-11-02T18:24:46.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:duAEEE+1dFgsFrxsptedZebsfDA=', 'Content-Length': '0'} 2023-11-02T18:24:46.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001372e9376c8f2624-006543e96e-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:46.990 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:24:46.991 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:24:46.991 DEBUG:boto:path=/ibtljn-56/c?versionId=null 2023-11-02T18:24:46.991 DEBUG:boto:auth_path=/ibtljn-56/c?versionId=null 2023-11-02T18:24:46.992 DEBUG:boto:Method: HEAD 2023-11-02T18:24:46.992 DEBUG:boto:Path: /ibtljn-56/c?versionId=null 2023-11-02T18:24:46.993 DEBUG:boto:Data: 2023-11-02T18:24:46.993 DEBUG:boto:Headers: {} 2023-11-02T18:24:46.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:46.994 DEBUG:boto:Port: 8000 2023-11-02T18:24:46.994 DEBUG:boto:Params: {} 2023-11-02T18:24:46.995 DEBUG:boto:Token: None 2023-11-02T18:24:46.995 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:46 GMT /ibtljn-56/c?versionId=null 2023-11-02T18:24:46.996 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:duAEEE+1dFgsFrxsptedZebsfDA= 2023-11-02T18:24:46.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:duAEEE+1dFgsFrxsptedZebsfDA=', 'Content-Length': '0'} 2023-11-02T18:24:47.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0e656c340fe85c7-006543e96e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:24:47.001 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:24:47.001 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:24:47.002 DEBUG:boto:path=/ibtljn-56/c?versionId=null 2023-11-02T18:24:47.002 DEBUG:boto:auth_path=/ibtljn-56/c?versionId=null 2023-11-02T18:24:47.003 DEBUG:boto:Method: GET 2023-11-02T18:24:47.003 DEBUG:boto:Path: /ibtljn-56/c?versionId=null 2023-11-02T18:24:47.003 DEBUG:boto:Data: 2023-11-02T18:24:47.004 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.004 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:47.005 DEBUG:boto:Port: 8001 2023-11-02T18:24:47.005 DEBUG:boto:Params: {} 2023-11-02T18:24:47.006 DEBUG:boto:Token: None 2023-11-02T18:24:47.006 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/c?versionId=null 2023-11-02T18:24:47.007 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HuSSb5XOd0UzJQ4EpYzBKraAyrk= 2023-11-02T18:24:47.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HuSSb5XOd0UzJQ4EpYzBKraAyrk=', 'Content-Length': '0'} 2023-11-02T18:24:47.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d22b4b0767ebeebc-006543e96f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.011 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:24:47.011 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:24:47.012 DEBUG:boto:path=/ibtljn-56/c?versionId=null 2023-11-02T18:24:47.012 DEBUG:boto:auth_path=/ibtljn-56/c?versionId=null 2023-11-02T18:24:47.012 DEBUG:boto:Method: GET 2023-11-02T18:24:47.013 DEBUG:boto:Path: /ibtljn-56/c?versionId=null 2023-11-02T18:24:47.013 DEBUG:boto:Data: 2023-11-02T18:24:47.014 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.014 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.015 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.015 DEBUG:boto:Params: {} 2023-11-02T18:24:47.015 DEBUG:boto:Token: None 2023-11-02T18:24:47.016 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/c?versionId=null 2023-11-02T18:24:47.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HuSSb5XOd0UzJQ4EpYzBKraAyrk= 2023-11-02T18:24:47.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HuSSb5XOd0UzJQ4EpYzBKraAyrk=', 'Content-Length': '0'} 2023-11-02T18:24:47.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b41b7fada24b7695-006543e96f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.066 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:47.067 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:47.067 DEBUG:boto:path=/ibtljn-56/?prefix=c 2023-11-02T18:24:47.068 DEBUG:boto:auth_path=/ibtljn-56/?prefix=c 2023-11-02T18:24:47.068 DEBUG:boto:Method: GET 2023-11-02T18:24:47.069 DEBUG:boto:Path: /ibtljn-56/?prefix=c 2023-11-02T18:24:47.069 DEBUG:boto:Data: 2023-11-02T18:24:47.070 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:47.071 DEBUG:boto:Port: 8001 2023-11-02T18:24:47.071 DEBUG:boto:Params: {} 2023-11-02T18:24:47.071 DEBUG:boto:Token: None 2023-11-02T18:24:47.072 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/ 2023-11-02T18:24:47.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg= 2023-11-02T18:24:47.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg=', 'Content-Length': '0'} 2023-11-02T18:24:47.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d42ee4f6d4bbfe88-006543e96f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.078 DEBUG:boto:b'ibtljn-56c1000falsec2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:47.079 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:47.079 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:47.080 DEBUG:boto:path=/ibtljn-56/?prefix=c 2023-11-02T18:24:47.080 DEBUG:boto:auth_path=/ibtljn-56/?prefix=c 2023-11-02T18:24:47.080 DEBUG:boto:Method: GET 2023-11-02T18:24:47.081 DEBUG:boto:Path: /ibtljn-56/?prefix=c 2023-11-02T18:24:47.081 DEBUG:boto:Data: 2023-11-02T18:24:47.082 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.082 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.082 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.083 DEBUG:boto:Params: {} 2023-11-02T18:24:47.083 DEBUG:boto:Token: None 2023-11-02T18:24:47.084 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/ 2023-11-02T18:24:47.084 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg= 2023-11-02T18:24:47.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg=', 'Content-Length': '0'} 2023-11-02T18:24:47.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dd5db0d0eeaa140-006543e96f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.090 DEBUG:boto:b'ibtljn-56c1000falsec2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:47.091 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:24:47.091 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:24:47.091 DEBUG:boto:Method: HEAD 2023-11-02T18:24:47.092 DEBUG:boto:Path: /ibtljn-56/c 2023-11-02T18:24:47.092 DEBUG:boto:Data: 2023-11-02T18:24:47.093 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.093 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:47.093 DEBUG:boto:Port: 8001 2023-11-02T18:24:47.094 DEBUG:boto:Params: {} 2023-11-02T18:24:47.094 DEBUG:boto:Token: None 2023-11-02T18:24:47.095 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/c 2023-11-02T18:24:47.095 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gfjy8O+NAJfSWhKszCTxvBH5hqw= 2023-11-02T18:24:47.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gfjy8O+NAJfSWhKszCTxvBH5hqw=', 'Content-Length': '0'} 2023-11-02T18:24:47.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b474e8d9fcf5a34e-006543e96f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.099 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:24:47.099 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:24:47.100 DEBUG:boto:Method: HEAD 2023-11-02T18:24:47.100 DEBUG:boto:Path: /ibtljn-56/c 2023-11-02T18:24:47.101 DEBUG:boto:Data: 2023-11-02T18:24:47.101 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.101 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.102 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.102 DEBUG:boto:Params: {} 2023-11-02T18:24:47.103 DEBUG:boto:Token: None 2023-11-02T18:24:47.103 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/c 2023-11-02T18:24:47.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gfjy8O+NAJfSWhKszCTxvBH5hqw= 2023-11-02T18:24:47.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gfjy8O+NAJfSWhKszCTxvBH5hqw=', 'Content-Length': '0'} 2023-11-02T18:24:47.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009793af40e36152cf-006543e96f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:24:47.108 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:24:47.108 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:24:47.109 DEBUG:boto:Method: GET 2023-11-02T18:24:47.109 DEBUG:boto:Path: /ibtljn-56/c 2023-11-02T18:24:47.109 DEBUG:boto:Data: 2023-11-02T18:24:47.110 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.110 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:47.111 DEBUG:boto:Port: 8001 2023-11-02T18:24:47.111 DEBUG:boto:Params: {} 2023-11-02T18:24:47.112 DEBUG:boto:Token: None 2023-11-02T18:24:47.112 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/c 2023-11-02T18:24:47.113 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CeKa804FRP5MCzpOOnl/a4xDlf0= 2023-11-02T18:24:47.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CeKa804FRP5MCzpOOnl/a4xDlf0=', 'Content-Length': '0'} 2023-11-02T18:24:47.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3ffa3e9d8c9ec0e-006543e96f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.158 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:24:47.159 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:24:47.159 DEBUG:boto:Method: GET 2023-11-02T18:24:47.160 DEBUG:boto:Path: /ibtljn-56/c 2023-11-02T18:24:47.160 DEBUG:boto:Data: 2023-11-02T18:24:47.161 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.161 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.161 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.162 DEBUG:boto:Params: {} 2023-11-02T18:24:47.162 DEBUG:boto:Token: None 2023-11-02T18:24:47.163 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/c 2023-11-02T18:24:47.163 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CeKa804FRP5MCzpOOnl/a4xDlf0= 2023-11-02T18:24:47.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CeKa804FRP5MCzpOOnl/a4xDlf0=', 'Content-Length': '0'} 2023-11-02T18:24:47.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c31142fc384c471-006543e96f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.167 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:47.168 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:47.168 DEBUG:boto:path=/ibtljn-56/?prefix=c 2023-11-02T18:24:47.169 DEBUG:boto:auth_path=/ibtljn-56/?prefix=c 2023-11-02T18:24:47.169 DEBUG:boto:Method: GET 2023-11-02T18:24:47.170 DEBUG:boto:Path: /ibtljn-56/?prefix=c 2023-11-02T18:24:47.170 DEBUG:boto:Data: 2023-11-02T18:24:47.170 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.171 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:47.171 DEBUG:boto:Port: 8001 2023-11-02T18:24:47.172 DEBUG:boto:Params: {} 2023-11-02T18:24:47.172 DEBUG:boto:Token: None 2023-11-02T18:24:47.173 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/ 2023-11-02T18:24:47.173 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg= 2023-11-02T18:24:47.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg=', 'Content-Length': '0'} 2023-11-02T18:24:47.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000908f13ad806fd879-006543e96f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.179 DEBUG:boto:b'ibtljn-56c1000falsec2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:47.180 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:47.180 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:47.180 DEBUG:boto:path=/ibtljn-56/?prefix=c 2023-11-02T18:24:47.181 DEBUG:boto:auth_path=/ibtljn-56/?prefix=c 2023-11-02T18:24:47.181 DEBUG:boto:Method: GET 2023-11-02T18:24:47.182 DEBUG:boto:Path: /ibtljn-56/?prefix=c 2023-11-02T18:24:47.182 DEBUG:boto:Data: 2023-11-02T18:24:47.183 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.183 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.183 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.184 DEBUG:boto:Params: {} 2023-11-02T18:24:47.184 DEBUG:boto:Token: None 2023-11-02T18:24:47.185 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/ 2023-11-02T18:24:47.185 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg= 2023-11-02T18:24:47.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg=', 'Content-Length': '0'} 2023-11-02T18:24:47.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026696eb50e214a04-006543e96f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.238 DEBUG:boto:b'ibtljn-56c1000falsec2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:47.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:24:47.239 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:24:47.240 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:24:47.240 DEBUG:boto:path=/ibtljn-56/d?versionId=null 2023-11-02T18:24:47.240 DEBUG:boto:auth_path=/ibtljn-56/d?versionId=null 2023-11-02T18:24:47.241 DEBUG:boto:Method: GET 2023-11-02T18:24:47.241 DEBUG:boto:Path: /ibtljn-56/d?versionId=null 2023-11-02T18:24:47.242 DEBUG:boto:Data: 2023-11-02T18:24:47.242 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.243 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:47.243 DEBUG:boto:Port: 8001 2023-11-02T18:24:47.243 DEBUG:boto:Params: {} 2023-11-02T18:24:47.244 DEBUG:boto:Token: None 2023-11-02T18:24:47.244 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/d?versionId=null 2023-11-02T18:24:47.245 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qv5icO2ZpMyksqTx1bMf/xxjOdM= 2023-11-02T18:24:47.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qv5icO2ZpMyksqTx1bMf/xxjOdM=', 'Content-Length': '0'} 2023-11-02T18:24:47.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000510860113595d1a0-006543e96f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.250 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:24:47.250 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:24:47.251 DEBUG:boto:path=/ibtljn-56/d?versionId=null 2023-11-02T18:24:47.251 DEBUG:boto:auth_path=/ibtljn-56/d?versionId=null 2023-11-02T18:24:47.252 DEBUG:boto:Method: GET 2023-11-02T18:24:47.252 DEBUG:boto:Path: /ibtljn-56/d?versionId=null 2023-11-02T18:24:47.252 DEBUG:boto:Data: 2023-11-02T18:24:47.253 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.253 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.254 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.254 DEBUG:boto:Params: {} 2023-11-02T18:24:47.255 DEBUG:boto:Token: None 2023-11-02T18:24:47.255 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/d?versionId=null 2023-11-02T18:24:47.256 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qv5icO2ZpMyksqTx1bMf/xxjOdM= 2023-11-02T18:24:47.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qv5icO2ZpMyksqTx1bMf/xxjOdM=', 'Content-Length': '0'} 2023-11-02T18:24:47.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008234dd5fb0922d34-006543e96f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.260 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:24:47.260 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:24:47.261 DEBUG:boto:path=/ibtljn-56/d?versionId=null 2023-11-02T18:24:47.261 DEBUG:boto:auth_path=/ibtljn-56/d?versionId=null 2023-11-02T18:24:47.262 DEBUG:boto:Method: HEAD 2023-11-02T18:24:47.262 DEBUG:boto:Path: /ibtljn-56/d?versionId=null 2023-11-02T18:24:47.262 DEBUG:boto:Data: 2023-11-02T18:24:47.263 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.263 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:47.264 DEBUG:boto:Port: 8001 2023-11-02T18:24:47.264 DEBUG:boto:Params: {} 2023-11-02T18:24:47.265 DEBUG:boto:Token: None 2023-11-02T18:24:47.265 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/d?versionId=null 2023-11-02T18:24:47.265 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qhghwnzmeIr6LNV/LxtjmpQUXnA= 2023-11-02T18:24:47.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qhghwnzmeIr6LNV/LxtjmpQUXnA=', 'Content-Length': '0'} 2023-11-02T18:24:47.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7477593abd5d408-006543e96f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.269 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:24:47.270 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:24:47.270 DEBUG:boto:path=/ibtljn-56/d?versionId=null 2023-11-02T18:24:47.271 DEBUG:boto:auth_path=/ibtljn-56/d?versionId=null 2023-11-02T18:24:47.271 DEBUG:boto:Method: HEAD 2023-11-02T18:24:47.272 DEBUG:boto:Path: /ibtljn-56/d?versionId=null 2023-11-02T18:24:47.272 DEBUG:boto:Data: 2023-11-02T18:24:47.272 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.273 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.273 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.273 DEBUG:boto:Params: {} 2023-11-02T18:24:47.274 DEBUG:boto:Token: None 2023-11-02T18:24:47.274 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/d?versionId=null 2023-11-02T18:24:47.275 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qhghwnzmeIr6LNV/LxtjmpQUXnA= 2023-11-02T18:24:47.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qhghwnzmeIr6LNV/LxtjmpQUXnA=', 'Content-Length': '0'} 2023-11-02T18:24:47.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f6ea482142b6ba6-006543e96f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:24:47.279 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:24:47.280 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:24:47.280 DEBUG:boto:path=/ibtljn-56/d?versionId=null 2023-11-02T18:24:47.281 DEBUG:boto:auth_path=/ibtljn-56/d?versionId=null 2023-11-02T18:24:47.281 DEBUG:boto:Method: GET 2023-11-02T18:24:47.281 DEBUG:boto:Path: /ibtljn-56/d?versionId=null 2023-11-02T18:24:47.282 DEBUG:boto:Data: 2023-11-02T18:24:47.282 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.283 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:47.283 DEBUG:boto:Port: 8001 2023-11-02T18:24:47.283 DEBUG:boto:Params: {} 2023-11-02T18:24:47.284 DEBUG:boto:Token: None 2023-11-02T18:24:47.284 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/d?versionId=null 2023-11-02T18:24:47.285 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qv5icO2ZpMyksqTx1bMf/xxjOdM= 2023-11-02T18:24:47.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qv5icO2ZpMyksqTx1bMf/xxjOdM=', 'Content-Length': '0'} 2023-11-02T18:24:47.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000577fe93ebf56098f-006543e96f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.289 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:24:47.290 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:24:47.290 DEBUG:boto:path=/ibtljn-56/d?versionId=null 2023-11-02T18:24:47.291 DEBUG:boto:auth_path=/ibtljn-56/d?versionId=null 2023-11-02T18:24:47.291 DEBUG:boto:Method: GET 2023-11-02T18:24:47.291 DEBUG:boto:Path: /ibtljn-56/d?versionId=null 2023-11-02T18:24:47.292 DEBUG:boto:Data: 2023-11-02T18:24:47.292 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.293 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.293 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.294 DEBUG:boto:Params: {} 2023-11-02T18:24:47.294 DEBUG:boto:Token: None 2023-11-02T18:24:47.295 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/d?versionId=null 2023-11-02T18:24:47.295 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qv5icO2ZpMyksqTx1bMf/xxjOdM= 2023-11-02T18:24:47.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qv5icO2ZpMyksqTx1bMf/xxjOdM=', 'Content-Length': '0'} 2023-11-02T18:24:47.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a4ae6c7edc1779e-006543e96f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.300 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:47.300 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:47.300 DEBUG:boto:path=/ibtljn-56/?prefix=d 2023-11-02T18:24:47.301 DEBUG:boto:auth_path=/ibtljn-56/?prefix=d 2023-11-02T18:24:47.301 DEBUG:boto:Method: GET 2023-11-02T18:24:47.302 DEBUG:boto:Path: /ibtljn-56/?prefix=d 2023-11-02T18:24:47.302 DEBUG:boto:Data: 2023-11-02T18:24:47.303 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:47.303 DEBUG:boto:Port: 8001 2023-11-02T18:24:47.304 DEBUG:boto:Params: {} 2023-11-02T18:24:47.304 DEBUG:boto:Token: None 2023-11-02T18:24:47.305 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/ 2023-11-02T18:24:47.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg= 2023-11-02T18:24:47.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg=', 'Content-Length': '0'} 2023-11-02T18:24:47.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001349e37d62c55036-006543e96f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.311 DEBUG:boto:b'ibtljn-56d1000falsed2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:47.312 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:47.312 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:47.313 DEBUG:boto:path=/ibtljn-56/?prefix=d 2023-11-02T18:24:47.313 DEBUG:boto:auth_path=/ibtljn-56/?prefix=d 2023-11-02T18:24:47.313 DEBUG:boto:Method: GET 2023-11-02T18:24:47.314 DEBUG:boto:Path: /ibtljn-56/?prefix=d 2023-11-02T18:24:47.314 DEBUG:boto:Data: 2023-11-02T18:24:47.315 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.315 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.316 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.316 DEBUG:boto:Params: {} 2023-11-02T18:24:47.316 DEBUG:boto:Token: None 2023-11-02T18:24:47.317 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/ 2023-11-02T18:24:47.317 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg= 2023-11-02T18:24:47.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg=', 'Content-Length': '0'} 2023-11-02T18:24:47.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a219d2f41271aaf-006543e96f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.362 DEBUG:boto:b'ibtljn-56d1000falsed2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:47.363 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:24:47.364 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:24:47.364 DEBUG:boto:Method: HEAD 2023-11-02T18:24:47.364 DEBUG:boto:Path: /ibtljn-56/d 2023-11-02T18:24:47.365 DEBUG:boto:Data: 2023-11-02T18:24:47.365 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:47.366 DEBUG:boto:Port: 8001 2023-11-02T18:24:47.366 DEBUG:boto:Params: {} 2023-11-02T18:24:47.367 DEBUG:boto:Token: None 2023-11-02T18:24:47.367 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/d 2023-11-02T18:24:47.368 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWdDn10v9BJeg5s6pYk951F/ChY= 2023-11-02T18:24:47.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWdDn10v9BJeg5s6pYk951F/ChY=', 'Content-Length': '0'} 2023-11-02T18:24:47.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd6cbd27063d0579-006543e96f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.372 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:24:47.372 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:24:47.373 DEBUG:boto:Method: HEAD 2023-11-02T18:24:47.373 DEBUG:boto:Path: /ibtljn-56/d 2023-11-02T18:24:47.373 DEBUG:boto:Data: 2023-11-02T18:24:47.374 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.374 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.374 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.375 DEBUG:boto:Params: {} 2023-11-02T18:24:47.375 DEBUG:boto:Token: None 2023-11-02T18:24:47.376 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/d 2023-11-02T18:24:47.376 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWdDn10v9BJeg5s6pYk951F/ChY= 2023-11-02T18:24:47.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWdDn10v9BJeg5s6pYk951F/ChY=', 'Content-Length': '0'} 2023-11-02T18:24:47.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe23f9cddf7201b-006543e96f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:24:47.380 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:24:47.381 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:24:47.381 DEBUG:boto:Method: GET 2023-11-02T18:24:47.382 DEBUG:boto:Path: /ibtljn-56/d 2023-11-02T18:24:47.382 DEBUG:boto:Data: 2023-11-02T18:24:47.383 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:47.383 DEBUG:boto:Port: 8001 2023-11-02T18:24:47.384 DEBUG:boto:Params: {} 2023-11-02T18:24:47.384 DEBUG:boto:Token: None 2023-11-02T18:24:47.385 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/d 2023-11-02T18:24:47.385 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UFA+FHjCH51xcPYcF6be/u5sStw= 2023-11-02T18:24:47.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UFA+FHjCH51xcPYcF6be/u5sStw=', 'Content-Length': '0'} 2023-11-02T18:24:47.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd68a5956cfec0e6-006543e96f-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.434 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:24:47.435 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:24:47.435 DEBUG:boto:Method: GET 2023-11-02T18:24:47.436 DEBUG:boto:Path: /ibtljn-56/d 2023-11-02T18:24:47.436 DEBUG:boto:Data: 2023-11-02T18:24:47.436 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.437 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.438 DEBUG:boto:Params: {} 2023-11-02T18:24:47.438 DEBUG:boto:Token: None 2023-11-02T18:24:47.439 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/d 2023-11-02T18:24:47.439 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UFA+FHjCH51xcPYcF6be/u5sStw= 2023-11-02T18:24:47.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UFA+FHjCH51xcPYcF6be/u5sStw=', 'Content-Length': '0'} 2023-11-02T18:24:47.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000e9a3265205dd91-006543e96f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.443 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:47.444 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:47.444 DEBUG:boto:path=/ibtljn-56/?prefix=d 2023-11-02T18:24:47.445 DEBUG:boto:auth_path=/ibtljn-56/?prefix=d 2023-11-02T18:24:47.445 DEBUG:boto:Method: GET 2023-11-02T18:24:47.445 DEBUG:boto:Path: /ibtljn-56/?prefix=d 2023-11-02T18:24:47.446 DEBUG:boto:Data: 2023-11-02T18:24:47.446 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:24:47.447 DEBUG:boto:Port: 8001 2023-11-02T18:24:47.448 DEBUG:boto:Params: {} 2023-11-02T18:24:47.448 DEBUG:boto:Token: None 2023-11-02T18:24:47.449 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/ 2023-11-02T18:24:47.449 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg= 2023-11-02T18:24:47.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg=', 'Content-Length': '0'} 2023-11-02T18:24:47.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9bf2b88248c8cbd-006543e96f-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.455 DEBUG:boto:b'ibtljn-56d1000falsed2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:47.455 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:47.456 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:47.456 DEBUG:boto:path=/ibtljn-56/?prefix=d 2023-11-02T18:24:47.457 DEBUG:boto:auth_path=/ibtljn-56/?prefix=d 2023-11-02T18:24:47.457 DEBUG:boto:Method: GET 2023-11-02T18:24:47.457 DEBUG:boto:Path: /ibtljn-56/?prefix=d 2023-11-02T18:24:47.458 DEBUG:boto:Data: 2023-11-02T18:24:47.458 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.459 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.459 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.460 DEBUG:boto:Params: {} 2023-11-02T18:24:47.460 DEBUG:boto:Token: None 2023-11-02T18:24:47.460 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/ 2023-11-02T18:24:47.461 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg= 2023-11-02T18:24:47.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TFjDZx0LuKMN0WUOfvJXPOlr3Vg=', 'Content-Length': '0'} 2023-11-02T18:24:47.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3342e30a0bb843c-006543e96f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.467 DEBUG:boto:b'ibtljn-56d1000falsed2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:24:47.467 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-56 zones={a2, a1} 2023-11-02T18:24:47.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'ibtljn-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:24:47.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'ibtljn-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:24:47.469 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket ibtljn-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:24:47.513 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:24:47.514 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:24:47.593 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:24:47.594 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-56 2023-11-02T18:24:47.600 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 4 2023-11-02T18:24:47.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'ibtljn-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:24:47.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'ibtljn-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:24:47.675 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket ibtljn-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:24:47.720 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:24:47.720 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:24:47.794 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:24:47.794 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-56 2023-11-02T18:24:47.797 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 4 2023-11-02T18:24:47.855 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:47.855 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:47.856 DEBUG:boto:Method: HEAD 2023-11-02T18:24:47.856 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:24:47.857 DEBUG:boto:Data: 2023-11-02T18:24:47.857 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.858 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.858 DEBUG:boto:Params: {} 2023-11-02T18:24:47.859 DEBUG:boto:Token: None 2023-11-02T18:24:47.859 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/ 2023-11-02T18:24:47.860 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wIQnLQIS0Bgxwkxzvqmdg837RyU= 2023-11-02T18:24:47.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wIQnLQIS0Bgxwkxzvqmdg837RyU=', 'Content-Length': '0'} 2023-11-02T18:24:47.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e09a8e44038306c-006543e96f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.864 DEBUG:boto:path=/ibtljn-56/e 2023-11-02T18:24:47.865 DEBUG:boto:auth_path=/ibtljn-56/e 2023-11-02T18:24:47.865 DEBUG:boto:Method: PUT 2023-11-02T18:24:47.866 DEBUG:boto:Path: /ibtljn-56/e 2023-11-02T18:24:47.866 DEBUG:boto:Data: 2023-11-02T18:24:47.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:24:47.867 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.867 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.868 DEBUG:boto:Params: {} 2023-11-02T18:24:47.868 DEBUG:boto:Token: None 2023-11-02T18:24:47.869 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/e 2023-11-02T18:24:47.869 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q7grjJS0UPH7JHk2u2j4ArBLaDM= 2023-11-02T18:24:47.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q7grjJS0UPH7JHk2u2j4ArBLaDM='} 2023-11-02T18:24:47.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8b196dca38b2b76-006543e96f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.915 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:47.916 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:47.916 DEBUG:boto:Method: HEAD 2023-11-02T18:24:47.917 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:24:47.917 DEBUG:boto:Data: 2023-11-02T18:24:47.917 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.918 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.919 DEBUG:boto:Params: {} 2023-11-02T18:24:47.919 DEBUG:boto:Token: None 2023-11-02T18:24:47.920 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/ 2023-11-02T18:24:47.920 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wIQnLQIS0Bgxwkxzvqmdg837RyU= 2023-11-02T18:24:47.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wIQnLQIS0Bgxwkxzvqmdg837RyU=', 'Content-Length': '0'} 2023-11-02T18:24:47.924 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d12aab5ecc0a8d80-006543e96f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.924 DEBUG:boto:path=/ibtljn-56/f 2023-11-02T18:24:47.925 DEBUG:boto:auth_path=/ibtljn-56/f 2023-11-02T18:24:47.925 DEBUG:boto:Method: PUT 2023-11-02T18:24:47.926 DEBUG:boto:Path: /ibtljn-56/f 2023-11-02T18:24:47.926 DEBUG:boto:Data: 2023-11-02T18:24:47.926 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:24:47.927 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.927 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.928 DEBUG:boto:Params: {} 2023-11-02T18:24:47.928 DEBUG:boto:Token: None 2023-11-02T18:24:47.929 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/f 2023-11-02T18:24:47.929 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MPOhVZaJ9q+0NzklT62VHgI2KpA= 2023-11-02T18:24:47.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MPOhVZaJ9q+0NzklT62VHgI2KpA='} 2023-11-02T18:24:47.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b78c4389d53fc6f-006543e96f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.942 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:47.942 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:47.943 DEBUG:boto:Method: HEAD 2023-11-02T18:24:47.943 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:24:47.944 DEBUG:boto:Data: 2023-11-02T18:24:47.944 DEBUG:boto:Headers: {} 2023-11-02T18:24:47.945 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.945 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.946 DEBUG:boto:Params: {} 2023-11-02T18:24:47.946 DEBUG:boto:Token: None 2023-11-02T18:24:47.947 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/ 2023-11-02T18:24:47.947 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wIQnLQIS0Bgxwkxzvqmdg837RyU= 2023-11-02T18:24:47.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wIQnLQIS0Bgxwkxzvqmdg837RyU=', 'Content-Length': '0'} 2023-11-02T18:24:47.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086947a79e199e934-006543e96f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:47.951 DEBUG:boto:path=/ibtljn-56/g 2023-11-02T18:24:47.951 DEBUG:boto:auth_path=/ibtljn-56/g 2023-11-02T18:24:47.951 DEBUG:boto:Method: PUT 2023-11-02T18:24:47.952 DEBUG:boto:Path: /ibtljn-56/g 2023-11-02T18:24:47.952 DEBUG:boto:Data: 2023-11-02T18:24:47.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:24:47.953 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:47.954 DEBUG:boto:Port: 8000 2023-11-02T18:24:47.954 DEBUG:boto:Params: {} 2023-11-02T18:24:47.954 DEBUG:boto:Token: None 2023-11-02T18:24:47.955 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:24:47 GMT /ibtljn-56/g 2023-11-02T18:24:47.955 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P2QvlVw7Jj2K65B9WQNcDj1Lp24= 2023-11-02T18:24:47.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:24:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P2QvlVw7Jj2K65B9WQNcDj1Lp24='} 2023-11-02T18:24:48.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000dd3e1c36bc69cd8-006543e96f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:48.003 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:24:48.004 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:24:48.004 DEBUG:boto:Method: HEAD 2023-11-02T18:24:48.004 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:24:48.005 DEBUG:boto:Data: 2023-11-02T18:24:48.005 DEBUG:boto:Headers: {} 2023-11-02T18:24:48.006 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:48.006 DEBUG:boto:Port: 8000 2023-11-02T18:24:48.007 DEBUG:boto:Params: {} 2023-11-02T18:24:48.007 DEBUG:boto:Token: None 2023-11-02T18:24:48.007 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:24:48 GMT /ibtljn-56/ 2023-11-02T18:24:48.008 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:97wKB4Vxv5caPsQ2m2Wr8B27drM= 2023-11-02T18:24:48.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:97wKB4Vxv5caPsQ2m2Wr8B27drM=', 'Content-Length': '0'} 2023-11-02T18:24:48.011 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081c93252f8eb55a9-006543e970-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:48.012 DEBUG:boto:path=/ibtljn-56/h 2023-11-02T18:24:48.012 DEBUG:boto:auth_path=/ibtljn-56/h 2023-11-02T18:24:48.013 DEBUG:boto:Method: PUT 2023-11-02T18:24:48.013 DEBUG:boto:Path: /ibtljn-56/h 2023-11-02T18:24:48.013 DEBUG:boto:Data: 2023-11-02T18:24:48.014 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:24:48.014 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:24:48.015 DEBUG:boto:Port: 8000 2023-11-02T18:24:48.015 DEBUG:boto:Params: {} 2023-11-02T18:24:48.016 DEBUG:boto:Token: None 2023-11-02T18:24:48.016 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:24:48 GMT /ibtljn-56/h 2023-11-02T18:24:48.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qCTRW6Rm1zab0zA+3oY6ZF7PlqE= 2023-11-02T18:24:48.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:24:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qCTRW6Rm1zab0zA+3oY6ZF7PlqE='} 2023-11-02T18:24:48.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eec05f22f44e3026-006543e970-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:24:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:24:48.026 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-56 2023-11-02T18:24:48.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:24:48.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:24:48.027 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:24:48.079 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:24:48.080 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:24:48.147 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:48.140+0000 7f2759909f40 1 waiting to reach latest gen 2, on 0.. 2023-11-02T18:24:53.148 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:53.144+0000 7f2759909f40 1 waiting to reach latest gen 2, on 0.. 2023-11-02T18:24:58.149 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:24:58.144+0000 7f2759909f40 1 waiting to reach latest gen 2, on 0.. 2023-11-02T18:25:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:25:03.144+0000 7f2759909f40 1 waiting to reach latest gen 2, on 0.. 2023-11-02T18:25:08.151 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:25:08.144+0000 7f2759909f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T18:25:13.153 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:25:13.144+0000 7f2759909f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T18:25:18.153 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:25:18.148+0000 7f2759909f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T18:25:23.155 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:25:23.152+0000 7f2759909f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T18:25:28.157 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:25:28.152+0000 7f2759909f40 1 waiting for incremental sync to catch up: 2023-11-02T18:25:28.158 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:25:28.158 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:25:33.158 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:25:33.151+0000 7f2759909f40 1 waiting for incremental sync to catch up: 2023-11-02T18:25:33.159 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:25:33.159 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:25:38.160 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:25:38.151+0000 7f2759909f40 1 waiting for incremental sync to catch up: 2023-11-02T18:25:38.160 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:25:38.160 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:25:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:25:43.155+0000 7f2759909f40 1 waiting for incremental sync to catch up: 2023-11-02T18:25:43.161 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:25:43.162 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:25:48.162 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:25:48.159+0000 7f2759909f40 1 waiting for incremental sync to catch up: 2023-11-02T18:25:48.163 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:25:48.163 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:25:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:25:53.155+0000 7f2759909f40 1 waiting for incremental sync to catch up: 2023-11-02T18:25:53.164 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:25:53.164 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:25:58.165 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:25:58.159+0000 7f2759909f40 1 waiting for incremental sync to catch up: 2023-11-02T18:25:58.165 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:25:58.166 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:26:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:03.163+0000 7f2759909f40 1 waiting for incremental sync to catch up: 2023-11-02T18:26:03.167 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:26:03.167 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:26:08.167 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:08.163+0000 7f2759909f40 1 waiting for incremental sync to catch up: 2023-11-02T18:26:08.168 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:26:08.168 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:26:13.169 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:13.167+0000 7f2759909f40 1 waiting for incremental sync to catch up: 2023-11-02T18:26:13.169 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:26:13.169 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:26:18.170 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:18.167+0000 7f2759909f40 1 waiting for incremental sync to catch up: 2023-11-02T18:26:18.170 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:26:18.171 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:26:23.171 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:23.167+0000 7f2759909f40 1 waiting for incremental sync to catch up: 2023-11-02T18:26:23.171 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , ] 2023-11-02T18:26:23.172 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:26:28.173 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:28.167+0000 7f2759909f40 1 bucket sync caught up with source: 2023-11-02T18:26:28.173 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:26:28.173 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.29.6, 00000000002.32.6, 00000000002.32.6] 2023-11-02T18:26:28.173 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:28.171+0000 7f2759909f40 0 bucket checkpoint complete 2023-11-02T18:26:28.206 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-56 2023-11-02T18:26:28.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:26:28.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:26:28.208 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:26:28.261 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:26:28.261 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:26:28.327 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:28.323+0000 7fdb35ff2f40 1 waiting for incremental sync to catch up: 2023-11-02T18:26:28.327 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.189.6, , , , 00000000001.82.6, , , 00000000001.254.6, , 00000000001.211.6] 2023-11-02T18:26:28.328 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.202.6, , , 00000000001.266.6, 00000000001.82.6, , , 00000000002.267.6, , 00000000002.224.6] 2023-11-02T18:26:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:33.327+0000 7fdb35ff2f40 1 waiting for incremental sync to catch up: 2023-11-02T18:26:33.330 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.202.6, , , , 00000000001.82.6, , , 00000000001.254.6, , 00000000002.224.6] 2023-11-02T18:26:33.330 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.202.6, , , 00000000001.266.6, 00000000001.82.6, , , 00000000002.267.6, , 00000000002.224.6] 2023-11-02T18:26:38.332 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:38.327+0000 7fdb35ff2f40 1 waiting for incremental sync to catch up: 2023-11-02T18:26:38.333 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.202.6, , , , 00000000001.82.6, , , 00000000001.254.6, , 00000000002.224.6] 2023-11-02T18:26:38.333 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.202.6, , , 00000000001.266.6, 00000000001.82.6, , , 00000000002.267.6, , 00000000002.224.6] 2023-11-02T18:26:43.335 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:43.331+0000 7fdb35ff2f40 1 bucket sync caught up with source: 2023-11-02T18:26:43.335 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.202.6, , , 00000000001.266.6, 00000000001.82.6, , , 00000000002.267.6, , 00000000002.224.6] 2023-11-02T18:26:43.335 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.202.6, , , 00000000001.266.6, 00000000001.82.6, , , 00000000002.267.6, , 00000000002.224.6] 2023-11-02T18:26:43.335 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:43.331+0000 7fdb35ff2f40 0 bucket checkpoint complete 2023-11-02T18:26:43.365 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-56 zones={a2, a1} 2023-11-02T18:26:43.366 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.367 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.367 DEBUG:boto:Method: HEAD 2023-11-02T18:26:43.368 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:26:43.368 DEBUG:boto:Data: 2023-11-02T18:26:43.368 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.369 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.369 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.370 DEBUG:boto:Params: {} 2023-11-02T18:26:43.371 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:26:43.371 DEBUG:boto:Token: None 2023-11-02T18:26:43.371 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/ 2023-11-02T18:26:43.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HvlGvGCoo9j63LO0a/gbaE3QL7A= 2023-11-02T18:26:43.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HvlGvGCoo9j63LO0a/gbaE3QL7A=', 'Content-Length': '0'} 2023-11-02T18:26:43.382 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019f784f12a945e98-006543e9e3-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.383 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.383 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.384 DEBUG:boto:Method: HEAD 2023-11-02T18:26:43.384 DEBUG:boto:Path: /ibtljn-56/ 2023-11-02T18:26:43.385 DEBUG:boto:Data: 2023-11-02T18:26:43.385 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.385 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.386 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.386 DEBUG:boto:Params: {} 2023-11-02T18:26:43.387 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:26:43.387 DEBUG:boto:Token: None 2023-11-02T18:26:43.389 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/ 2023-11-02T18:26:43.389 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HvlGvGCoo9j63LO0a/gbaE3QL7A= 2023-11-02T18:26:43.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HvlGvGCoo9j63LO0a/gbaE3QL7A=', 'Content-Length': '0'} 2023-11-02T18:26:43.397 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a36ee49c7d0cdf91-006543e9e3-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.398 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:26:43.399 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.399 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.399 DEBUG:boto:path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:26:43.400 DEBUG:boto:auth_path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:26:43.400 DEBUG:boto:Method: GET 2023-11-02T18:26:43.401 DEBUG:boto:Path: /ibtljn-56/?versions&max-keys=999 2023-11-02T18:26:43.401 DEBUG:boto:Data: 2023-11-02T18:26:43.401 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.402 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.402 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.403 DEBUG:boto:Params: {} 2023-11-02T18:26:43.403 DEBUG:boto:Token: None 2023-11-02T18:26:43.404 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/?versions 2023-11-02T18:26:43.404 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dsgeLrw2/2+AIRLR2Vk8IUxPM24= 2023-11-02T18:26:43.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dsgeLrw2/2+AIRLR2Vk8IUxPM24=', 'Content-Length': '0'} 2023-11-02T18:26:43.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000664bb4b0cf5a3db0-006543e9e3-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.410 DEBUG:boto:b'ibtljn-56999falseanulltrue2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:24:47.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:24:47.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:24:47.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:24:48.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:43.411 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:26:43.412 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:26:43.412 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:26:43.412 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:26:43.413 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-11-02T18:26:43.413 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-11-02T18:26:43.414 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-11-02T18:26:43.414 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-11-02T18:26:43.415 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:26:43.415 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.415 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.416 DEBUG:boto:path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:26:43.416 DEBUG:boto:auth_path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:26:43.417 DEBUG:boto:Method: GET 2023-11-02T18:26:43.417 DEBUG:boto:Path: /ibtljn-56/?versions&max-keys=999 2023-11-02T18:26:43.418 DEBUG:boto:Data: 2023-11-02T18:26:43.418 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.419 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.419 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.420 DEBUG:boto:Params: {} 2023-11-02T18:26:43.420 DEBUG:boto:Token: None 2023-11-02T18:26:43.421 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/?versions 2023-11-02T18:26:43.421 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dsgeLrw2/2+AIRLR2Vk8IUxPM24= 2023-11-02T18:26:43.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dsgeLrw2/2+AIRLR2Vk8IUxPM24=', 'Content-Length': '0'} 2023-11-02T18:26:43.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f600cda1859a39f-006543e9e3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.426 DEBUG:boto:b'ibtljn-56999falseanulltrue2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:24:47.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:24:47.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:24:47.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:24:48.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:43.427 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:26:43.427 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:26:43.428 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:26:43.428 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:26:43.428 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-11-02T18:26:43.429 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-11-02T18:26:43.429 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-11-02T18:26:43.430 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-11-02T18:26:43.430 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.430 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.431 DEBUG:boto:path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:26:43.431 DEBUG:boto:auth_path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:26:43.432 DEBUG:boto:Method: GET 2023-11-02T18:26:43.432 DEBUG:boto:Path: /ibtljn-56/?versions&max-keys=999 2023-11-02T18:26:43.433 DEBUG:boto:Data: 2023-11-02T18:26:43.433 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.434 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.434 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.434 DEBUG:boto:Params: {} 2023-11-02T18:26:43.435 DEBUG:boto:Token: None 2023-11-02T18:26:43.436 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/?versions 2023-11-02T18:26:43.436 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dsgeLrw2/2+AIRLR2Vk8IUxPM24= 2023-11-02T18:26:43.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dsgeLrw2/2+AIRLR2Vk8IUxPM24=', 'Content-Length': '0'} 2023-11-02T18:26:43.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a6e49bd539e0882-006543e9e3-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.442 DEBUG:boto:b'ibtljn-56999falseanulltrue2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:24:47.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:24:47.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:24:47.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:24:48.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:43.443 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.444 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.444 DEBUG:boto:path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:26:43.445 DEBUG:boto:auth_path=/ibtljn-56/?versions&max-keys=999 2023-11-02T18:26:43.445 DEBUG:boto:Method: GET 2023-11-02T18:26:43.445 DEBUG:boto:Path: /ibtljn-56/?versions&max-keys=999 2023-11-02T18:26:43.446 DEBUG:boto:Data: 2023-11-02T18:26:43.446 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.447 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.448 DEBUG:boto:Params: {} 2023-11-02T18:26:43.448 DEBUG:boto:Token: None 2023-11-02T18:26:43.449 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/?versions 2023-11-02T18:26:43.449 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dsgeLrw2/2+AIRLR2Vk8IUxPM24= 2023-11-02T18:26:43.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dsgeLrw2/2+AIRLR2Vk8IUxPM24=', 'Content-Length': '0'} 2023-11-02T18:26:43.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000883a02c1903027b5-006543e9e3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.454 DEBUG:boto:b'ibtljn-56999falseanulltrue2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:24:47.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:24:47.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:24:47.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:24:48.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:43.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:26:43.455 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:26:43.456 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:26:43.456 DEBUG:boto:path=/ibtljn-56/a?versionId=null 2023-11-02T18:26:43.457 DEBUG:boto:auth_path=/ibtljn-56/a?versionId=null 2023-11-02T18:26:43.457 DEBUG:boto:Method: GET 2023-11-02T18:26:43.457 DEBUG:boto:Path: /ibtljn-56/a?versionId=null 2023-11-02T18:26:43.458 DEBUG:boto:Data: 2023-11-02T18:26:43.458 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.459 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.459 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.460 DEBUG:boto:Params: {} 2023-11-02T18:26:43.460 DEBUG:boto:Token: None 2023-11-02T18:26:43.461 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/a?versionId=null 2023-11-02T18:26:43.461 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cRHDxeG9JV74YFXTOQ3/57zXv0U= 2023-11-02T18:26:43.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cRHDxeG9JV74YFXTOQ3/57zXv0U=', 'Content-Length': '0'} 2023-11-02T18:26:43.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4fbc7ed48196c0c-006543e9e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.507 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:26:43.507 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:26:43.508 DEBUG:boto:path=/ibtljn-56/a?versionId=null 2023-11-02T18:26:43.508 DEBUG:boto:auth_path=/ibtljn-56/a?versionId=null 2023-11-02T18:26:43.509 DEBUG:boto:Method: GET 2023-11-02T18:26:43.509 DEBUG:boto:Path: /ibtljn-56/a?versionId=null 2023-11-02T18:26:43.509 DEBUG:boto:Data: 2023-11-02T18:26:43.510 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.511 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.511 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.511 DEBUG:boto:Params: {} 2023-11-02T18:26:43.512 DEBUG:boto:Token: None 2023-11-02T18:26:43.512 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/a?versionId=null 2023-11-02T18:26:43.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cRHDxeG9JV74YFXTOQ3/57zXv0U= 2023-11-02T18:26:43.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cRHDxeG9JV74YFXTOQ3/57zXv0U=', 'Content-Length': '0'} 2023-11-02T18:26:43.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9b8f39a9421e9f9-006543e9e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.518 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:26:43.518 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:26:43.518 DEBUG:boto:path=/ibtljn-56/a?versionId=null 2023-11-02T18:26:43.519 DEBUG:boto:auth_path=/ibtljn-56/a?versionId=null 2023-11-02T18:26:43.519 DEBUG:boto:Method: HEAD 2023-11-02T18:26:43.520 DEBUG:boto:Path: /ibtljn-56/a?versionId=null 2023-11-02T18:26:43.520 DEBUG:boto:Data: 2023-11-02T18:26:43.520 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.521 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.521 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.522 DEBUG:boto:Params: {} 2023-11-02T18:26:43.522 DEBUG:boto:Token: None 2023-11-02T18:26:43.523 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/a?versionId=null 2023-11-02T18:26:43.523 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gpCtMbwp/pxK4CnJX2YqiHHD3Eo= 2023-11-02T18:26:43.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gpCtMbwp/pxK4CnJX2YqiHHD3Eo=', 'Content-Length': '0'} 2023-11-02T18:26:43.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000029b2da92938f22c-006543e9e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.527 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:26:43.527 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:26:43.528 DEBUG:boto:path=/ibtljn-56/a?versionId=null 2023-11-02T18:26:43.528 DEBUG:boto:auth_path=/ibtljn-56/a?versionId=null 2023-11-02T18:26:43.529 DEBUG:boto:Method: HEAD 2023-11-02T18:26:43.529 DEBUG:boto:Path: /ibtljn-56/a?versionId=null 2023-11-02T18:26:43.530 DEBUG:boto:Data: 2023-11-02T18:26:43.530 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.531 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.531 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.531 DEBUG:boto:Params: {} 2023-11-02T18:26:43.532 DEBUG:boto:Token: None 2023-11-02T18:26:43.532 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/a?versionId=null 2023-11-02T18:26:43.533 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gpCtMbwp/pxK4CnJX2YqiHHD3Eo= 2023-11-02T18:26:43.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gpCtMbwp/pxK4CnJX2YqiHHD3Eo=', 'Content-Length': '0'} 2023-11-02T18:26:43.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030fcaed976848cea-006543e9e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:26:43.538 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:26:43.538 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:26:43.539 DEBUG:boto:path=/ibtljn-56/a?versionId=null 2023-11-02T18:26:43.539 DEBUG:boto:auth_path=/ibtljn-56/a?versionId=null 2023-11-02T18:26:43.540 DEBUG:boto:Method: GET 2023-11-02T18:26:43.540 DEBUG:boto:Path: /ibtljn-56/a?versionId=null 2023-11-02T18:26:43.540 DEBUG:boto:Data: 2023-11-02T18:26:43.541 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.542 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.542 DEBUG:boto:Params: {} 2023-11-02T18:26:43.543 DEBUG:boto:Token: None 2023-11-02T18:26:43.544 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/a?versionId=null 2023-11-02T18:26:43.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cRHDxeG9JV74YFXTOQ3/57zXv0U= 2023-11-02T18:26:43.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cRHDxeG9JV74YFXTOQ3/57zXv0U=', 'Content-Length': '0'} 2023-11-02T18:26:43.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acf2642f875802d1-006543e9e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.548 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:26:43.549 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:26:43.549 DEBUG:boto:path=/ibtljn-56/a?versionId=null 2023-11-02T18:26:43.550 DEBUG:boto:auth_path=/ibtljn-56/a?versionId=null 2023-11-02T18:26:43.550 DEBUG:boto:Method: GET 2023-11-02T18:26:43.551 DEBUG:boto:Path: /ibtljn-56/a?versionId=null 2023-11-02T18:26:43.551 DEBUG:boto:Data: 2023-11-02T18:26:43.551 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.552 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.552 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.553 DEBUG:boto:Params: {} 2023-11-02T18:26:43.553 DEBUG:boto:Token: None 2023-11-02T18:26:43.554 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/a?versionId=null 2023-11-02T18:26:43.554 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cRHDxeG9JV74YFXTOQ3/57zXv0U= 2023-11-02T18:26:43.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cRHDxeG9JV74YFXTOQ3/57zXv0U=', 'Content-Length': '0'} 2023-11-02T18:26:43.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000039789941534c61d-006543e9e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.559 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.559 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.560 DEBUG:boto:path=/ibtljn-56/?prefix=a 2023-11-02T18:26:43.560 DEBUG:boto:auth_path=/ibtljn-56/?prefix=a 2023-11-02T18:26:43.561 DEBUG:boto:Method: GET 2023-11-02T18:26:43.561 DEBUG:boto:Path: /ibtljn-56/?prefix=a 2023-11-02T18:26:43.562 DEBUG:boto:Data: 2023-11-02T18:26:43.562 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.563 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.563 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.564 DEBUG:boto:Params: {} 2023-11-02T18:26:43.564 DEBUG:boto:Token: None 2023-11-02T18:26:43.565 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/ 2023-11-02T18:26:43.565 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk= 2023-11-02T18:26:43.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk=', 'Content-Length': '0'} 2023-11-02T18:26:43.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040dc8076957b0ee1-006543e9e3-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.618 DEBUG:boto:b'ibtljn-56a1000falsea2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:43.619 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.620 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.620 DEBUG:boto:path=/ibtljn-56/?prefix=a 2023-11-02T18:26:43.620 DEBUG:boto:auth_path=/ibtljn-56/?prefix=a 2023-11-02T18:26:43.621 DEBUG:boto:Method: GET 2023-11-02T18:26:43.621 DEBUG:boto:Path: /ibtljn-56/?prefix=a 2023-11-02T18:26:43.622 DEBUG:boto:Data: 2023-11-02T18:26:43.622 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.623 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.623 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.624 DEBUG:boto:Params: {} 2023-11-02T18:26:43.624 DEBUG:boto:Token: None 2023-11-02T18:26:43.625 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/ 2023-11-02T18:26:43.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk= 2023-11-02T18:26:43.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk=', 'Content-Length': '0'} 2023-11-02T18:26:43.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa922d3d3b94ea7-006543e9e3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.630 DEBUG:boto:b'ibtljn-56a1000falsea2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:43.631 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:26:43.631 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:26:43.632 DEBUG:boto:Method: HEAD 2023-11-02T18:26:43.632 DEBUG:boto:Path: /ibtljn-56/a 2023-11-02T18:26:43.633 DEBUG:boto:Data: 2023-11-02T18:26:43.633 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.633 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.634 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.634 DEBUG:boto:Params: {} 2023-11-02T18:26:43.635 DEBUG:boto:Token: None 2023-11-02T18:26:43.635 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/a 2023-11-02T18:26:43.636 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8shhKbr0p4jQDU9hec7uV07bEDo= 2023-11-02T18:26:43.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8shhKbr0p4jQDU9hec7uV07bEDo=', 'Content-Length': '0'} 2023-11-02T18:26:43.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000726331a9d342ea-006543e9e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.640 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:26:43.641 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:26:43.641 DEBUG:boto:Method: HEAD 2023-11-02T18:26:43.641 DEBUG:boto:Path: /ibtljn-56/a 2023-11-02T18:26:43.642 DEBUG:boto:Data: 2023-11-02T18:26:43.642 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.643 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.643 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.644 DEBUG:boto:Params: {} 2023-11-02T18:26:43.644 DEBUG:boto:Token: None 2023-11-02T18:26:43.645 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/a 2023-11-02T18:26:43.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8shhKbr0p4jQDU9hec7uV07bEDo= 2023-11-02T18:26:43.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8shhKbr0p4jQDU9hec7uV07bEDo=', 'Content-Length': '0'} 2023-11-02T18:26:43.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000add2c9c11c2ed446-006543e9e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:26:43.650 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:26:43.651 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:26:43.651 DEBUG:boto:Method: GET 2023-11-02T18:26:43.652 DEBUG:boto:Path: /ibtljn-56/a 2023-11-02T18:26:43.652 DEBUG:boto:Data: 2023-11-02T18:26:43.653 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.653 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.653 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.654 DEBUG:boto:Params: {} 2023-11-02T18:26:43.654 DEBUG:boto:Token: None 2023-11-02T18:26:43.655 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/a 2023-11-02T18:26:43.655 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y1azJf7QBkJdp7CZIr7qXgI458A= 2023-11-02T18:26:43.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y1azJf7QBkJdp7CZIr7qXgI458A=', 'Content-Length': '0'} 2023-11-02T18:26:43.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3e273da40c0c04e-006543e9e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.659 DEBUG:boto:path=/ibtljn-56/a 2023-11-02T18:26:43.660 DEBUG:boto:auth_path=/ibtljn-56/a 2023-11-02T18:26:43.660 DEBUG:boto:Method: GET 2023-11-02T18:26:43.661 DEBUG:boto:Path: /ibtljn-56/a 2023-11-02T18:26:43.661 DEBUG:boto:Data: 2023-11-02T18:26:43.662 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.662 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.663 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.663 DEBUG:boto:Params: {} 2023-11-02T18:26:43.664 DEBUG:boto:Token: None 2023-11-02T18:26:43.664 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/a 2023-11-02T18:26:43.665 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y1azJf7QBkJdp7CZIr7qXgI458A= 2023-11-02T18:26:43.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y1azJf7QBkJdp7CZIr7qXgI458A=', 'Content-Length': '0'} 2023-11-02T18:26:43.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a09c95a80ddaec7f-006543e9e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.669 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.670 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.670 DEBUG:boto:path=/ibtljn-56/?prefix=a 2023-11-02T18:26:43.671 DEBUG:boto:auth_path=/ibtljn-56/?prefix=a 2023-11-02T18:26:43.671 DEBUG:boto:Method: GET 2023-11-02T18:26:43.672 DEBUG:boto:Path: /ibtljn-56/?prefix=a 2023-11-02T18:26:43.672 DEBUG:boto:Data: 2023-11-02T18:26:43.673 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.673 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.674 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.674 DEBUG:boto:Params: {} 2023-11-02T18:26:43.675 DEBUG:boto:Token: None 2023-11-02T18:26:43.675 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/ 2023-11-02T18:26:43.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk= 2023-11-02T18:26:43.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk=', 'Content-Length': '0'} 2023-11-02T18:26:43.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da0cfcf6ad36f48d-006543e9e3-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.722 DEBUG:boto:b'ibtljn-56a1000falsea2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:43.723 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.724 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.724 DEBUG:boto:path=/ibtljn-56/?prefix=a 2023-11-02T18:26:43.725 DEBUG:boto:auth_path=/ibtljn-56/?prefix=a 2023-11-02T18:26:43.725 DEBUG:boto:Method: GET 2023-11-02T18:26:43.725 DEBUG:boto:Path: /ibtljn-56/?prefix=a 2023-11-02T18:26:43.726 DEBUG:boto:Data: 2023-11-02T18:26:43.727 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.727 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.728 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.728 DEBUG:boto:Params: {} 2023-11-02T18:26:43.728 DEBUG:boto:Token: None 2023-11-02T18:26:43.729 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/ 2023-11-02T18:26:43.729 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk= 2023-11-02T18:26:43.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk=', 'Content-Length': '0'} 2023-11-02T18:26:43.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b91343f25b6ab51-006543e9e3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.734 DEBUG:boto:b'ibtljn-56a1000falsea2023-11-02T18:24:10.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:43.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:26:43.735 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:26:43.736 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:26:43.736 DEBUG:boto:path=/ibtljn-56/b?versionId=null 2023-11-02T18:26:43.737 DEBUG:boto:auth_path=/ibtljn-56/b?versionId=null 2023-11-02T18:26:43.737 DEBUG:boto:Method: GET 2023-11-02T18:26:43.738 DEBUG:boto:Path: /ibtljn-56/b?versionId=null 2023-11-02T18:26:43.738 DEBUG:boto:Data: 2023-11-02T18:26:43.738 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.739 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.739 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.740 DEBUG:boto:Params: {} 2023-11-02T18:26:43.740 DEBUG:boto:Token: None 2023-11-02T18:26:43.741 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/b?versionId=null 2023-11-02T18:26:43.741 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0ZTLuS+CM42Skxn7u+fGprKewgo= 2023-11-02T18:26:43.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0ZTLuS+CM42Skxn7u+fGprKewgo=', 'Content-Length': '0'} 2023-11-02T18:26:43.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080163601e410afdb-006543e9e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.746 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:26:43.746 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:26:43.747 DEBUG:boto:path=/ibtljn-56/b?versionId=null 2023-11-02T18:26:43.747 DEBUG:boto:auth_path=/ibtljn-56/b?versionId=null 2023-11-02T18:26:43.748 DEBUG:boto:Method: GET 2023-11-02T18:26:43.748 DEBUG:boto:Path: /ibtljn-56/b?versionId=null 2023-11-02T18:26:43.749 DEBUG:boto:Data: 2023-11-02T18:26:43.749 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.749 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.750 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.750 DEBUG:boto:Params: {} 2023-11-02T18:26:43.751 DEBUG:boto:Token: None 2023-11-02T18:26:43.751 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/b?versionId=null 2023-11-02T18:26:43.752 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0ZTLuS+CM42Skxn7u+fGprKewgo= 2023-11-02T18:26:43.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0ZTLuS+CM42Skxn7u+fGprKewgo=', 'Content-Length': '0'} 2023-11-02T18:26:43.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000665a14e52ebcea6c-006543e9e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.756 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:26:43.756 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:26:43.757 DEBUG:boto:path=/ibtljn-56/b?versionId=null 2023-11-02T18:26:43.757 DEBUG:boto:auth_path=/ibtljn-56/b?versionId=null 2023-11-02T18:26:43.758 DEBUG:boto:Method: HEAD 2023-11-02T18:26:43.758 DEBUG:boto:Path: /ibtljn-56/b?versionId=null 2023-11-02T18:26:43.759 DEBUG:boto:Data: 2023-11-02T18:26:43.759 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.759 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.760 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.760 DEBUG:boto:Params: {} 2023-11-02T18:26:43.761 DEBUG:boto:Token: None 2023-11-02T18:26:43.761 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/b?versionId=null 2023-11-02T18:26:43.762 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kaG0LI2WAXGCDI8BrNJQ3bah4os= 2023-11-02T18:26:43.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kaG0LI2WAXGCDI8BrNJQ3bah4os=', 'Content-Length': '0'} 2023-11-02T18:26:43.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dbf17f802828e80-006543e9e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.766 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:26:43.766 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:26:43.767 DEBUG:boto:path=/ibtljn-56/b?versionId=null 2023-11-02T18:26:43.767 DEBUG:boto:auth_path=/ibtljn-56/b?versionId=null 2023-11-02T18:26:43.768 DEBUG:boto:Method: HEAD 2023-11-02T18:26:43.768 DEBUG:boto:Path: /ibtljn-56/b?versionId=null 2023-11-02T18:26:43.769 DEBUG:boto:Data: 2023-11-02T18:26:43.769 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.770 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.770 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.770 DEBUG:boto:Params: {} 2023-11-02T18:26:43.771 DEBUG:boto:Token: None 2023-11-02T18:26:43.771 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/b?versionId=null 2023-11-02T18:26:43.772 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kaG0LI2WAXGCDI8BrNJQ3bah4os= 2023-11-02T18:26:43.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kaG0LI2WAXGCDI8BrNJQ3bah4os=', 'Content-Length': '0'} 2023-11-02T18:26:43.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffd230a301a9924e-006543e9e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:26:43.776 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:26:43.777 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:26:43.777 DEBUG:boto:path=/ibtljn-56/b?versionId=null 2023-11-02T18:26:43.778 DEBUG:boto:auth_path=/ibtljn-56/b?versionId=null 2023-11-02T18:26:43.778 DEBUG:boto:Method: GET 2023-11-02T18:26:43.779 DEBUG:boto:Path: /ibtljn-56/b?versionId=null 2023-11-02T18:26:43.779 DEBUG:boto:Data: 2023-11-02T18:26:43.780 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.780 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.780 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.781 DEBUG:boto:Params: {} 2023-11-02T18:26:43.781 DEBUG:boto:Token: None 2023-11-02T18:26:43.782 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/b?versionId=null 2023-11-02T18:26:43.782 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0ZTLuS+CM42Skxn7u+fGprKewgo= 2023-11-02T18:26:43.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0ZTLuS+CM42Skxn7u+fGprKewgo=', 'Content-Length': '0'} 2023-11-02T18:26:43.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a52cb907a1c331b-006543e9e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.787 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:26:43.787 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:26:43.788 DEBUG:boto:path=/ibtljn-56/b?versionId=null 2023-11-02T18:26:43.788 DEBUG:boto:auth_path=/ibtljn-56/b?versionId=null 2023-11-02T18:26:43.789 DEBUG:boto:Method: GET 2023-11-02T18:26:43.789 DEBUG:boto:Path: /ibtljn-56/b?versionId=null 2023-11-02T18:26:43.789 DEBUG:boto:Data: 2023-11-02T18:26:43.790 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.791 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.791 DEBUG:boto:Params: {} 2023-11-02T18:26:43.792 DEBUG:boto:Token: None 2023-11-02T18:26:43.792 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/b?versionId=null 2023-11-02T18:26:43.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0ZTLuS+CM42Skxn7u+fGprKewgo= 2023-11-02T18:26:43.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0ZTLuS+CM42Skxn7u+fGprKewgo=', 'Content-Length': '0'} 2023-11-02T18:26:43.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077498a121ceae1cd-006543e9e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.838 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.839 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.839 DEBUG:boto:path=/ibtljn-56/?prefix=b 2023-11-02T18:26:43.840 DEBUG:boto:auth_path=/ibtljn-56/?prefix=b 2023-11-02T18:26:43.840 DEBUG:boto:Method: GET 2023-11-02T18:26:43.841 DEBUG:boto:Path: /ibtljn-56/?prefix=b 2023-11-02T18:26:43.841 DEBUG:boto:Data: 2023-11-02T18:26:43.842 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.842 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.843 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.843 DEBUG:boto:Params: {} 2023-11-02T18:26:43.843 DEBUG:boto:Token: None 2023-11-02T18:26:43.844 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/ 2023-11-02T18:26:43.844 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk= 2023-11-02T18:26:43.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk=', 'Content-Length': '0'} 2023-11-02T18:26:43.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065aee9cfe6367505-006543e9e3-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.850 DEBUG:boto:b'ibtljn-56b1000falseb2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:43.851 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.851 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.852 DEBUG:boto:path=/ibtljn-56/?prefix=b 2023-11-02T18:26:43.852 DEBUG:boto:auth_path=/ibtljn-56/?prefix=b 2023-11-02T18:26:43.853 DEBUG:boto:Method: GET 2023-11-02T18:26:43.853 DEBUG:boto:Path: /ibtljn-56/?prefix=b 2023-11-02T18:26:43.854 DEBUG:boto:Data: 2023-11-02T18:26:43.854 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.855 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.857 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.858 DEBUG:boto:Params: {} 2023-11-02T18:26:43.858 DEBUG:boto:Token: None 2023-11-02T18:26:43.859 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/ 2023-11-02T18:26:43.859 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk= 2023-11-02T18:26:43.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk=', 'Content-Length': '0'} 2023-11-02T18:26:43.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b89b6226e0271ed2-006543e9e3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.864 DEBUG:boto:b'ibtljn-56b1000falseb2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:43.865 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:26:43.865 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:26:43.865 DEBUG:boto:Method: HEAD 2023-11-02T18:26:43.866 DEBUG:boto:Path: /ibtljn-56/b 2023-11-02T18:26:43.866 DEBUG:boto:Data: 2023-11-02T18:26:43.867 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.867 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.868 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.868 DEBUG:boto:Params: {} 2023-11-02T18:26:43.868 DEBUG:boto:Token: None 2023-11-02T18:26:43.869 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/b 2023-11-02T18:26:43.869 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MOx20MXk+s6GkFA7rCZAQd5UkH4= 2023-11-02T18:26:43.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MOx20MXk+s6GkFA7rCZAQd5UkH4=', 'Content-Length': '0'} 2023-11-02T18:26:43.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000388e672137f383a-006543e9e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.873 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:26:43.874 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:26:43.874 DEBUG:boto:Method: HEAD 2023-11-02T18:26:43.875 DEBUG:boto:Path: /ibtljn-56/b 2023-11-02T18:26:43.875 DEBUG:boto:Data: 2023-11-02T18:26:43.876 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.876 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.876 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.877 DEBUG:boto:Params: {} 2023-11-02T18:26:43.877 DEBUG:boto:Token: None 2023-11-02T18:26:43.878 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/b 2023-11-02T18:26:43.878 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MOx20MXk+s6GkFA7rCZAQd5UkH4= 2023-11-02T18:26:43.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MOx20MXk+s6GkFA7rCZAQd5UkH4=', 'Content-Length': '0'} 2023-11-02T18:26:43.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6149387a9740545-006543e9e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:26:43.883 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:26:43.883 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:26:43.884 DEBUG:boto:Method: GET 2023-11-02T18:26:43.884 DEBUG:boto:Path: /ibtljn-56/b 2023-11-02T18:26:43.885 DEBUG:boto:Data: 2023-11-02T18:26:43.885 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.886 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.887 DEBUG:boto:Params: {} 2023-11-02T18:26:43.887 DEBUG:boto:Token: None 2023-11-02T18:26:43.888 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/b 2023-11-02T18:26:43.888 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fObLHtfYCaC+dA2JSLOWpIW/+qM= 2023-11-02T18:26:43.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fObLHtfYCaC+dA2JSLOWpIW/+qM=', 'Content-Length': '0'} 2023-11-02T18:26:43.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007774db65d40e6042-006543e9e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.893 DEBUG:boto:path=/ibtljn-56/b 2023-11-02T18:26:43.893 DEBUG:boto:auth_path=/ibtljn-56/b 2023-11-02T18:26:43.894 DEBUG:boto:Method: GET 2023-11-02T18:26:43.894 DEBUG:boto:Path: /ibtljn-56/b 2023-11-02T18:26:43.894 DEBUG:boto:Data: 2023-11-02T18:26:43.895 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.895 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.896 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.896 DEBUG:boto:Params: {} 2023-11-02T18:26:43.897 DEBUG:boto:Token: None 2023-11-02T18:26:43.897 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/b 2023-11-02T18:26:43.898 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fObLHtfYCaC+dA2JSLOWpIW/+qM= 2023-11-02T18:26:43.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fObLHtfYCaC+dA2JSLOWpIW/+qM=', 'Content-Length': '0'} 2023-11-02T18:26:43.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e530b3982c06e58-006543e9e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.903 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.903 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.904 DEBUG:boto:path=/ibtljn-56/?prefix=b 2023-11-02T18:26:43.904 DEBUG:boto:auth_path=/ibtljn-56/?prefix=b 2023-11-02T18:26:43.905 DEBUG:boto:Method: GET 2023-11-02T18:26:43.905 DEBUG:boto:Path: /ibtljn-56/?prefix=b 2023-11-02T18:26:43.906 DEBUG:boto:Data: 2023-11-02T18:26:43.906 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.907 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.907 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.908 DEBUG:boto:Params: {} 2023-11-02T18:26:43.908 DEBUG:boto:Token: None 2023-11-02T18:26:43.909 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/ 2023-11-02T18:26:43.909 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk= 2023-11-02T18:26:43.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk=', 'Content-Length': '0'} 2023-11-02T18:26:43.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000933203aba4583499-006543e9e3-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.958 DEBUG:boto:b'ibtljn-56b1000falseb2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:43.959 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:43.959 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:43.960 DEBUG:boto:path=/ibtljn-56/?prefix=b 2023-11-02T18:26:43.960 DEBUG:boto:auth_path=/ibtljn-56/?prefix=b 2023-11-02T18:26:43.961 DEBUG:boto:Method: GET 2023-11-02T18:26:43.961 DEBUG:boto:Path: /ibtljn-56/?prefix=b 2023-11-02T18:26:43.962 DEBUG:boto:Data: 2023-11-02T18:26:43.962 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.963 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.963 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.964 DEBUG:boto:Params: {} 2023-11-02T18:26:43.964 DEBUG:boto:Token: None 2023-11-02T18:26:43.965 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/ 2023-11-02T18:26:43.965 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk= 2023-11-02T18:26:43.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aXFUyuAu0Cc0thFjXzzcGvyNMJk=', 'Content-Length': '0'} 2023-11-02T18:26:43.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002692e1d250046790-006543e9e3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.970 DEBUG:boto:b'ibtljn-56b1000falseb2023-11-02T18:24:10.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:43.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:26:43.971 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:26:43.971 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:26:43.972 DEBUG:boto:path=/ibtljn-56/c?versionId=null 2023-11-02T18:26:43.972 DEBUG:boto:auth_path=/ibtljn-56/c?versionId=null 2023-11-02T18:26:43.973 DEBUG:boto:Method: GET 2023-11-02T18:26:43.973 DEBUG:boto:Path: /ibtljn-56/c?versionId=null 2023-11-02T18:26:43.974 DEBUG:boto:Data: 2023-11-02T18:26:43.974 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.974 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.975 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.975 DEBUG:boto:Params: {} 2023-11-02T18:26:43.976 DEBUG:boto:Token: None 2023-11-02T18:26:43.976 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/c?versionId=null 2023-11-02T18:26:43.977 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mkVDI8AlNwwJKpB3sWvEAEwC3wU= 2023-11-02T18:26:43.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mkVDI8AlNwwJKpB3sWvEAEwC3wU=', 'Content-Length': '0'} 2023-11-02T18:26:43.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071fc4adb0d3d7177-006543e9e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.981 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:26:43.982 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:26:43.982 DEBUG:boto:path=/ibtljn-56/c?versionId=null 2023-11-02T18:26:43.983 DEBUG:boto:auth_path=/ibtljn-56/c?versionId=null 2023-11-02T18:26:43.983 DEBUG:boto:Method: GET 2023-11-02T18:26:43.984 DEBUG:boto:Path: /ibtljn-56/c?versionId=null 2023-11-02T18:26:43.984 DEBUG:boto:Data: 2023-11-02T18:26:43.985 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.985 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:43.986 DEBUG:boto:Port: 8000 2023-11-02T18:26:43.986 DEBUG:boto:Params: {} 2023-11-02T18:26:43.987 DEBUG:boto:Token: None 2023-11-02T18:26:43.987 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/c?versionId=null 2023-11-02T18:26:43.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mkVDI8AlNwwJKpB3sWvEAEwC3wU= 2023-11-02T18:26:43.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mkVDI8AlNwwJKpB3sWvEAEwC3wU=', 'Content-Length': '0'} 2023-11-02T18:26:43.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ea7ccc8ba4e7b62-006543e9e3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:43.992 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:26:43.993 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:26:43.993 DEBUG:boto:path=/ibtljn-56/c?versionId=null 2023-11-02T18:26:43.993 DEBUG:boto:auth_path=/ibtljn-56/c?versionId=null 2023-11-02T18:26:43.994 DEBUG:boto:Method: HEAD 2023-11-02T18:26:43.994 DEBUG:boto:Path: /ibtljn-56/c?versionId=null 2023-11-02T18:26:43.995 DEBUG:boto:Data: 2023-11-02T18:26:43.995 DEBUG:boto:Headers: {} 2023-11-02T18:26:43.996 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:43.996 DEBUG:boto:Port: 8001 2023-11-02T18:26:43.997 DEBUG:boto:Params: {} 2023-11-02T18:26:43.997 DEBUG:boto:Token: None 2023-11-02T18:26:43.998 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:43 GMT /ibtljn-56/c?versionId=null 2023-11-02T18:26:43.998 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P7aVRZmaWNDsV5DKJTQOOKJRgH0= 2023-11-02T18:26:43.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P7aVRZmaWNDsV5DKJTQOOKJRgH0=', 'Content-Length': '0'} 2023-11-02T18:26:44.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f118d6bf6728d54-006543e9e3-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.002 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:26:44.003 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:26:44.003 DEBUG:boto:path=/ibtljn-56/c?versionId=null 2023-11-02T18:26:44.004 DEBUG:boto:auth_path=/ibtljn-56/c?versionId=null 2023-11-02T18:26:44.004 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.005 DEBUG:boto:Path: /ibtljn-56/c?versionId=null 2023-11-02T18:26:44.005 DEBUG:boto:Data: 2023-11-02T18:26:44.005 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.006 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.006 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.007 DEBUG:boto:Params: {} 2023-11-02T18:26:44.007 DEBUG:boto:Token: None 2023-11-02T18:26:44.008 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/c?versionId=null 2023-11-02T18:26:44.008 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/9Ywy8FP0iJnWriy+lsDvzNIGp0= 2023-11-02T18:26:44.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/9Ywy8FP0iJnWriy+lsDvzNIGp0=', 'Content-Length': '0'} 2023-11-02T18:26:44.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077db29f5cbd4c64b-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:26:44.012 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:26:44.013 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:26:44.013 DEBUG:boto:path=/ibtljn-56/c?versionId=null 2023-11-02T18:26:44.014 DEBUG:boto:auth_path=/ibtljn-56/c?versionId=null 2023-11-02T18:26:44.014 DEBUG:boto:Method: GET 2023-11-02T18:26:44.015 DEBUG:boto:Path: /ibtljn-56/c?versionId=null 2023-11-02T18:26:44.015 DEBUG:boto:Data: 2023-11-02T18:26:44.016 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.016 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.017 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.017 DEBUG:boto:Params: {} 2023-11-02T18:26:44.018 DEBUG:boto:Token: None 2023-11-02T18:26:44.018 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/c?versionId=null 2023-11-02T18:26:44.019 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3EdhiV0flLS+vJc0BJ1CaqKhryA= 2023-11-02T18:26:44.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3EdhiV0flLS+vJc0BJ1CaqKhryA=', 'Content-Length': '0'} 2023-11-02T18:26:44.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bc1908132e77f0f-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.023 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:26:44.023 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:26:44.024 DEBUG:boto:path=/ibtljn-56/c?versionId=null 2023-11-02T18:26:44.024 DEBUG:boto:auth_path=/ibtljn-56/c?versionId=null 2023-11-02T18:26:44.025 DEBUG:boto:Method: GET 2023-11-02T18:26:44.025 DEBUG:boto:Path: /ibtljn-56/c?versionId=null 2023-11-02T18:26:44.026 DEBUG:boto:Data: 2023-11-02T18:26:44.026 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.027 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.027 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.028 DEBUG:boto:Params: {} 2023-11-02T18:26:44.028 DEBUG:boto:Token: None 2023-11-02T18:26:44.029 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/c?versionId=null 2023-11-02T18:26:44.029 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3EdhiV0flLS+vJc0BJ1CaqKhryA= 2023-11-02T18:26:44.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3EdhiV0flLS+vJc0BJ1CaqKhryA=', 'Content-Length': '0'} 2023-11-02T18:26:44.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e0e735a15371554-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.074 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.075 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.075 DEBUG:boto:path=/ibtljn-56/?prefix=c 2023-11-02T18:26:44.076 DEBUG:boto:auth_path=/ibtljn-56/?prefix=c 2023-11-02T18:26:44.076 DEBUG:boto:Method: GET 2023-11-02T18:26:44.077 DEBUG:boto:Path: /ibtljn-56/?prefix=c 2023-11-02T18:26:44.077 DEBUG:boto:Data: 2023-11-02T18:26:44.078 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.078 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.079 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.079 DEBUG:boto:Params: {} 2023-11-02T18:26:44.080 DEBUG:boto:Token: None 2023-11-02T18:26:44.080 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.081 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d318edf4c71114b-006543e9e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.087 DEBUG:boto:b'ibtljn-56c1000falsec2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.088 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.088 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.089 DEBUG:boto:path=/ibtljn-56/?prefix=c 2023-11-02T18:26:44.089 DEBUG:boto:auth_path=/ibtljn-56/?prefix=c 2023-11-02T18:26:44.090 DEBUG:boto:Method: GET 2023-11-02T18:26:44.090 DEBUG:boto:Path: /ibtljn-56/?prefix=c 2023-11-02T18:26:44.090 DEBUG:boto:Data: 2023-11-02T18:26:44.091 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.092 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.092 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.092 DEBUG:boto:Params: {} 2023-11-02T18:26:44.093 DEBUG:boto:Token: None 2023-11-02T18:26:44.094 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b902b90d5f7e315d-006543e9e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.099 DEBUG:boto:b'ibtljn-56c1000falsec2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.099 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:26:44.100 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:26:44.100 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.101 DEBUG:boto:Path: /ibtljn-56/c 2023-11-02T18:26:44.101 DEBUG:boto:Data: 2023-11-02T18:26:44.101 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.102 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.102 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.103 DEBUG:boto:Params: {} 2023-11-02T18:26:44.103 DEBUG:boto:Token: None 2023-11-02T18:26:44.104 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/c 2023-11-02T18:26:44.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oyT5+63n2ECEO1J+Dv4roVvctuE= 2023-11-02T18:26:44.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oyT5+63n2ECEO1J+Dv4roVvctuE=', 'Content-Length': '0'} 2023-11-02T18:26:44.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088631ec9a56a5a2f-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.109 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:26:44.109 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:26:44.109 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.110 DEBUG:boto:Path: /ibtljn-56/c 2023-11-02T18:26:44.110 DEBUG:boto:Data: 2023-11-02T18:26:44.111 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.111 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.111 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.112 DEBUG:boto:Params: {} 2023-11-02T18:26:44.112 DEBUG:boto:Token: None 2023-11-02T18:26:44.113 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/c 2023-11-02T18:26:44.113 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oyT5+63n2ECEO1J+Dv4roVvctuE= 2023-11-02T18:26:44.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oyT5+63n2ECEO1J+Dv4roVvctuE=', 'Content-Length': '0'} 2023-11-02T18:26:44.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067eaef3d42571177-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:26:44.117 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:26:44.118 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:26:44.119 DEBUG:boto:Method: GET 2023-11-02T18:26:44.119 DEBUG:boto:Path: /ibtljn-56/c 2023-11-02T18:26:44.120 DEBUG:boto:Data: 2023-11-02T18:26:44.120 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.120 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.121 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.121 DEBUG:boto:Params: {} 2023-11-02T18:26:44.122 DEBUG:boto:Token: None 2023-11-02T18:26:44.123 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/c 2023-11-02T18:26:44.123 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:euWMVNaX5ldi/oqLo9cjeLQKrkY= 2023-11-02T18:26:44.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:euWMVNaX5ldi/oqLo9cjeLQKrkY=', 'Content-Length': '0'} 2023-11-02T18:26:44.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b20e443ec51879a-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.170 DEBUG:boto:path=/ibtljn-56/c 2023-11-02T18:26:44.171 DEBUG:boto:auth_path=/ibtljn-56/c 2023-11-02T18:26:44.171 DEBUG:boto:Method: GET 2023-11-02T18:26:44.172 DEBUG:boto:Path: /ibtljn-56/c 2023-11-02T18:26:44.172 DEBUG:boto:Data: 2023-11-02T18:26:44.173 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.173 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.174 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.174 DEBUG:boto:Params: {} 2023-11-02T18:26:44.175 DEBUG:boto:Token: None 2023-11-02T18:26:44.175 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/c 2023-11-02T18:26:44.176 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:euWMVNaX5ldi/oqLo9cjeLQKrkY= 2023-11-02T18:26:44.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:euWMVNaX5ldi/oqLo9cjeLQKrkY=', 'Content-Length': '0'} 2023-11-02T18:26:44.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d09b655b51cc856a-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.180 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.180 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.181 DEBUG:boto:path=/ibtljn-56/?prefix=c 2023-11-02T18:26:44.181 DEBUG:boto:auth_path=/ibtljn-56/?prefix=c 2023-11-02T18:26:44.182 DEBUG:boto:Method: GET 2023-11-02T18:26:44.182 DEBUG:boto:Path: /ibtljn-56/?prefix=c 2023-11-02T18:26:44.183 DEBUG:boto:Data: 2023-11-02T18:26:44.183 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.184 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.184 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.184 DEBUG:boto:Params: {} 2023-11-02T18:26:44.185 DEBUG:boto:Token: None 2023-11-02T18:26:44.185 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.186 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad8f8c14202f78ff-006543e9e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.192 DEBUG:boto:b'ibtljn-56c1000falsec2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.193 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.193 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.194 DEBUG:boto:path=/ibtljn-56/?prefix=c 2023-11-02T18:26:44.194 DEBUG:boto:auth_path=/ibtljn-56/?prefix=c 2023-11-02T18:26:44.195 DEBUG:boto:Method: GET 2023-11-02T18:26:44.195 DEBUG:boto:Path: /ibtljn-56/?prefix=c 2023-11-02T18:26:44.195 DEBUG:boto:Data: 2023-11-02T18:26:44.196 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.196 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.197 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.197 DEBUG:boto:Params: {} 2023-11-02T18:26:44.198 DEBUG:boto:Token: None 2023-11-02T18:26:44.198 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.199 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2ab839769ee5b58-006543e9e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.203 DEBUG:boto:b'ibtljn-56c1000falsec2023-11-02T18:24:10.965Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:26:44.204 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:26:44.205 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:26:44.205 DEBUG:boto:path=/ibtljn-56/d?versionId=null 2023-11-02T18:26:44.206 DEBUG:boto:auth_path=/ibtljn-56/d?versionId=null 2023-11-02T18:26:44.206 DEBUG:boto:Method: GET 2023-11-02T18:26:44.207 DEBUG:boto:Path: /ibtljn-56/d?versionId=null 2023-11-02T18:26:44.207 DEBUG:boto:Data: 2023-11-02T18:26:44.208 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.208 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.209 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.209 DEBUG:boto:Params: {} 2023-11-02T18:26:44.210 DEBUG:boto:Token: None 2023-11-02T18:26:44.210 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/d?versionId=null 2023-11-02T18:26:44.211 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Am7JAQ+b3kfGFvmjJMjCz6D1WMk= 2023-11-02T18:26:44.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Am7JAQ+b3kfGFvmjJMjCz6D1WMk=', 'Content-Length': '0'} 2023-11-02T18:26:44.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083480a58502057a3-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.215 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:26:44.215 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:26:44.216 DEBUG:boto:path=/ibtljn-56/d?versionId=null 2023-11-02T18:26:44.216 DEBUG:boto:auth_path=/ibtljn-56/d?versionId=null 2023-11-02T18:26:44.217 DEBUG:boto:Method: GET 2023-11-02T18:26:44.217 DEBUG:boto:Path: /ibtljn-56/d?versionId=null 2023-11-02T18:26:44.218 DEBUG:boto:Data: 2023-11-02T18:26:44.218 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.219 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.219 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.219 DEBUG:boto:Params: {} 2023-11-02T18:26:44.220 DEBUG:boto:Token: None 2023-11-02T18:26:44.220 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/d?versionId=null 2023-11-02T18:26:44.221 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Am7JAQ+b3kfGFvmjJMjCz6D1WMk= 2023-11-02T18:26:44.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Am7JAQ+b3kfGFvmjJMjCz6D1WMk=', 'Content-Length': '0'} 2023-11-02T18:26:44.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ba718a55fea1df1-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.266 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:26:44.267 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:26:44.267 DEBUG:boto:path=/ibtljn-56/d?versionId=null 2023-11-02T18:26:44.268 DEBUG:boto:auth_path=/ibtljn-56/d?versionId=null 2023-11-02T18:26:44.268 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.269 DEBUG:boto:Path: /ibtljn-56/d?versionId=null 2023-11-02T18:26:44.269 DEBUG:boto:Data: 2023-11-02T18:26:44.269 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.270 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.270 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.271 DEBUG:boto:Params: {} 2023-11-02T18:26:44.271 DEBUG:boto:Token: None 2023-11-02T18:26:44.272 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/d?versionId=null 2023-11-02T18:26:44.272 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:36ZVoFwBkrgHmwQIvK32h3ntSq4= 2023-11-02T18:26:44.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:36ZVoFwBkrgHmwQIvK32h3ntSq4=', 'Content-Length': '0'} 2023-11-02T18:26:44.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006feb80e45d09ea98-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.276 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:26:44.276 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:26:44.277 DEBUG:boto:path=/ibtljn-56/d?versionId=null 2023-11-02T18:26:44.277 DEBUG:boto:auth_path=/ibtljn-56/d?versionId=null 2023-11-02T18:26:44.278 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.278 DEBUG:boto:Path: /ibtljn-56/d?versionId=null 2023-11-02T18:26:44.279 DEBUG:boto:Data: 2023-11-02T18:26:44.279 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.280 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.280 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.280 DEBUG:boto:Params: {} 2023-11-02T18:26:44.281 DEBUG:boto:Token: None 2023-11-02T18:26:44.281 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/d?versionId=null 2023-11-02T18:26:44.282 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:36ZVoFwBkrgHmwQIvK32h3ntSq4= 2023-11-02T18:26:44.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:36ZVoFwBkrgHmwQIvK32h3ntSq4=', 'Content-Length': '0'} 2023-11-02T18:26:44.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081d29209c79b63f3-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:26:44.287 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:26:44.287 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:26:44.288 DEBUG:boto:path=/ibtljn-56/d?versionId=null 2023-11-02T18:26:44.288 DEBUG:boto:auth_path=/ibtljn-56/d?versionId=null 2023-11-02T18:26:44.289 DEBUG:boto:Method: GET 2023-11-02T18:26:44.289 DEBUG:boto:Path: /ibtljn-56/d?versionId=null 2023-11-02T18:26:44.290 DEBUG:boto:Data: 2023-11-02T18:26:44.290 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.291 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.291 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.292 DEBUG:boto:Params: {} 2023-11-02T18:26:44.292 DEBUG:boto:Token: None 2023-11-02T18:26:44.293 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/d?versionId=null 2023-11-02T18:26:44.293 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Am7JAQ+b3kfGFvmjJMjCz6D1WMk= 2023-11-02T18:26:44.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Am7JAQ+b3kfGFvmjJMjCz6D1WMk=', 'Content-Length': '0'} 2023-11-02T18:26:44.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043d8742ce005a68c-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.338 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:26:44.339 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:26:44.339 DEBUG:boto:path=/ibtljn-56/d?versionId=null 2023-11-02T18:26:44.340 DEBUG:boto:auth_path=/ibtljn-56/d?versionId=null 2023-11-02T18:26:44.340 DEBUG:boto:Method: GET 2023-11-02T18:26:44.341 DEBUG:boto:Path: /ibtljn-56/d?versionId=null 2023-11-02T18:26:44.341 DEBUG:boto:Data: 2023-11-02T18:26:44.342 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.342 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.342 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.343 DEBUG:boto:Params: {} 2023-11-02T18:26:44.343 DEBUG:boto:Token: None 2023-11-02T18:26:44.344 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/d?versionId=null 2023-11-02T18:26:44.344 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Am7JAQ+b3kfGFvmjJMjCz6D1WMk= 2023-11-02T18:26:44.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Am7JAQ+b3kfGFvmjJMjCz6D1WMk=', 'Content-Length': '0'} 2023-11-02T18:26:44.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000773c08462f9bd18-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.348 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.349 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.349 DEBUG:boto:path=/ibtljn-56/?prefix=d 2023-11-02T18:26:44.350 DEBUG:boto:auth_path=/ibtljn-56/?prefix=d 2023-11-02T18:26:44.350 DEBUG:boto:Method: GET 2023-11-02T18:26:44.351 DEBUG:boto:Path: /ibtljn-56/?prefix=d 2023-11-02T18:26:44.351 DEBUG:boto:Data: 2023-11-02T18:26:44.351 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.352 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.352 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.353 DEBUG:boto:Params: {} 2023-11-02T18:26:44.353 DEBUG:boto:Token: None 2023-11-02T18:26:44.354 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.354 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067045c4b027ec097-006543e9e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.360 DEBUG:boto:b'ibtljn-56d1000falsed2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.361 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.362 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.362 DEBUG:boto:path=/ibtljn-56/?prefix=d 2023-11-02T18:26:44.363 DEBUG:boto:auth_path=/ibtljn-56/?prefix=d 2023-11-02T18:26:44.363 DEBUG:boto:Method: GET 2023-11-02T18:26:44.364 DEBUG:boto:Path: /ibtljn-56/?prefix=d 2023-11-02T18:26:44.364 DEBUG:boto:Data: 2023-11-02T18:26:44.365 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.365 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.366 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.366 DEBUG:boto:Params: {} 2023-11-02T18:26:44.367 DEBUG:boto:Token: None 2023-11-02T18:26:44.367 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.368 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd1b8689bfd0510e-006543e9e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.372 DEBUG:boto:b'ibtljn-56d1000falsed2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.373 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:26:44.373 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:26:44.374 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.374 DEBUG:boto:Path: /ibtljn-56/d 2023-11-02T18:26:44.375 DEBUG:boto:Data: 2023-11-02T18:26:44.375 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.376 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.376 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.376 DEBUG:boto:Params: {} 2023-11-02T18:26:44.377 DEBUG:boto:Token: None 2023-11-02T18:26:44.377 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/d 2023-11-02T18:26:44.378 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N38CZaMMMPoK7H/UEq/14TbvpbU= 2023-11-02T18:26:44.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N38CZaMMMPoK7H/UEq/14TbvpbU=', 'Content-Length': '0'} 2023-11-02T18:26:44.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000305fe7b50f4b4e3c-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.382 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:26:44.383 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:26:44.383 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.383 DEBUG:boto:Path: /ibtljn-56/d 2023-11-02T18:26:44.384 DEBUG:boto:Data: 2023-11-02T18:26:44.384 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.385 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.385 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.386 DEBUG:boto:Params: {} 2023-11-02T18:26:44.386 DEBUG:boto:Token: None 2023-11-02T18:26:44.387 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/d 2023-11-02T18:26:44.387 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N38CZaMMMPoK7H/UEq/14TbvpbU= 2023-11-02T18:26:44.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N38CZaMMMPoK7H/UEq/14TbvpbU=', 'Content-Length': '0'} 2023-11-02T18:26:44.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5e865a801352257-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:26:44.391 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:26:44.392 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:26:44.392 DEBUG:boto:Method: GET 2023-11-02T18:26:44.393 DEBUG:boto:Path: /ibtljn-56/d 2023-11-02T18:26:44.393 DEBUG:boto:Data: 2023-11-02T18:26:44.394 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.394 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.395 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.395 DEBUG:boto:Params: {} 2023-11-02T18:26:44.396 DEBUG:boto:Token: None 2023-11-02T18:26:44.396 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/d 2023-11-02T18:26:44.397 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5g0Agp6DyOmSIXk+1/LnVrXYSVU= 2023-11-02T18:26:44.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5g0Agp6DyOmSIXk+1/LnVrXYSVU=', 'Content-Length': '0'} 2023-11-02T18:26:44.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3d75921aedfb46f-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.401 DEBUG:boto:path=/ibtljn-56/d 2023-11-02T18:26:44.402 DEBUG:boto:auth_path=/ibtljn-56/d 2023-11-02T18:26:44.402 DEBUG:boto:Method: GET 2023-11-02T18:26:44.403 DEBUG:boto:Path: /ibtljn-56/d 2023-11-02T18:26:44.403 DEBUG:boto:Data: 2023-11-02T18:26:44.404 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.404 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.405 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.405 DEBUG:boto:Params: {} 2023-11-02T18:26:44.406 DEBUG:boto:Token: None 2023-11-02T18:26:44.406 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/d 2023-11-02T18:26:44.407 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5g0Agp6DyOmSIXk+1/LnVrXYSVU= 2023-11-02T18:26:44.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5g0Agp6DyOmSIXk+1/LnVrXYSVU=', 'Content-Length': '0'} 2023-11-02T18:26:44.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc437f047584d319-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.450 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.451 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.451 DEBUG:boto:path=/ibtljn-56/?prefix=d 2023-11-02T18:26:44.452 DEBUG:boto:auth_path=/ibtljn-56/?prefix=d 2023-11-02T18:26:44.452 DEBUG:boto:Method: GET 2023-11-02T18:26:44.453 DEBUG:boto:Path: /ibtljn-56/?prefix=d 2023-11-02T18:26:44.453 DEBUG:boto:Data: 2023-11-02T18:26:44.454 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.454 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.455 DEBUG:boto:Params: {} 2023-11-02T18:26:44.455 DEBUG:boto:Token: None 2023-11-02T18:26:44.456 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.457 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af859b6d9b2ab761-006543e9e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.463 DEBUG:boto:b'ibtljn-56d1000falsed2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.464 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.464 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.465 DEBUG:boto:path=/ibtljn-56/?prefix=d 2023-11-02T18:26:44.465 DEBUG:boto:auth_path=/ibtljn-56/?prefix=d 2023-11-02T18:26:44.466 DEBUG:boto:Method: GET 2023-11-02T18:26:44.466 DEBUG:boto:Path: /ibtljn-56/?prefix=d 2023-11-02T18:26:44.466 DEBUG:boto:Data: 2023-11-02T18:26:44.467 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.467 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.468 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.468 DEBUG:boto:Params: {} 2023-11-02T18:26:44.469 DEBUG:boto:Token: None 2023-11-02T18:26:44.469 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.470 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1b07d8eed47dfe9-006543e9e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.474 DEBUG:boto:b'ibtljn-56d1000falsed2023-11-02T18:24:10.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-11-02T18:26:44.475 DEBUG:boto:path=/ibtljn-56/e 2023-11-02T18:26:44.475 DEBUG:boto:auth_path=/ibtljn-56/e 2023-11-02T18:26:44.476 DEBUG:boto:path=/ibtljn-56/e?versionId=null 2023-11-02T18:26:44.476 DEBUG:boto:auth_path=/ibtljn-56/e?versionId=null 2023-11-02T18:26:44.477 DEBUG:boto:Method: GET 2023-11-02T18:26:44.477 DEBUG:boto:Path: /ibtljn-56/e?versionId=null 2023-11-02T18:26:44.478 DEBUG:boto:Data: 2023-11-02T18:26:44.478 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.478 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.479 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.479 DEBUG:boto:Params: {} 2023-11-02T18:26:44.480 DEBUG:boto:Token: None 2023-11-02T18:26:44.480 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/e?versionId=null 2023-11-02T18:26:44.481 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jih1ZxzYks4HIqbMgcjZ1BEaOoA= 2023-11-02T18:26:44.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jih1ZxzYks4HIqbMgcjZ1BEaOoA=', 'Content-Length': '0'} 2023-11-02T18:26:44.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a965920ff11d2392-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.530 DEBUG:boto:path=/ibtljn-56/e 2023-11-02T18:26:44.531 DEBUG:boto:auth_path=/ibtljn-56/e 2023-11-02T18:26:44.531 DEBUG:boto:path=/ibtljn-56/e?versionId=null 2023-11-02T18:26:44.532 DEBUG:boto:auth_path=/ibtljn-56/e?versionId=null 2023-11-02T18:26:44.532 DEBUG:boto:Method: GET 2023-11-02T18:26:44.533 DEBUG:boto:Path: /ibtljn-56/e?versionId=null 2023-11-02T18:26:44.533 DEBUG:boto:Data: 2023-11-02T18:26:44.533 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.534 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.534 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.535 DEBUG:boto:Params: {} 2023-11-02T18:26:44.535 DEBUG:boto:Token: None 2023-11-02T18:26:44.536 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/e?versionId=null 2023-11-02T18:26:44.536 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jih1ZxzYks4HIqbMgcjZ1BEaOoA= 2023-11-02T18:26:44.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jih1ZxzYks4HIqbMgcjZ1BEaOoA=', 'Content-Length': '0'} 2023-11-02T18:26:44.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a86e3add1814e29-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.541 DEBUG:boto:path=/ibtljn-56/e 2023-11-02T18:26:44.541 DEBUG:boto:auth_path=/ibtljn-56/e 2023-11-02T18:26:44.541 DEBUG:boto:path=/ibtljn-56/e?versionId=null 2023-11-02T18:26:44.542 DEBUG:boto:auth_path=/ibtljn-56/e?versionId=null 2023-11-02T18:26:44.542 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.543 DEBUG:boto:Path: /ibtljn-56/e?versionId=null 2023-11-02T18:26:44.543 DEBUG:boto:Data: 2023-11-02T18:26:44.544 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.544 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.544 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.545 DEBUG:boto:Params: {} 2023-11-02T18:26:44.545 DEBUG:boto:Token: None 2023-11-02T18:26:44.546 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/e?versionId=null 2023-11-02T18:26:44.546 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TlwM9WAJSBv1aTRznNJWOFQ4DGs= 2023-11-02T18:26:44.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TlwM9WAJSBv1aTRznNJWOFQ4DGs=', 'Content-Length': '0'} 2023-11-02T18:26:44.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a392845742157b1e-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.550 DEBUG:boto:path=/ibtljn-56/e 2023-11-02T18:26:44.551 DEBUG:boto:auth_path=/ibtljn-56/e 2023-11-02T18:26:44.551 DEBUG:boto:path=/ibtljn-56/e?versionId=null 2023-11-02T18:26:44.552 DEBUG:boto:auth_path=/ibtljn-56/e?versionId=null 2023-11-02T18:26:44.552 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.552 DEBUG:boto:Path: /ibtljn-56/e?versionId=null 2023-11-02T18:26:44.553 DEBUG:boto:Data: 2023-11-02T18:26:44.553 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.554 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.555 DEBUG:boto:Params: {} 2023-11-02T18:26:44.555 DEBUG:boto:Token: None 2023-11-02T18:26:44.556 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/e?versionId=null 2023-11-02T18:26:44.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TlwM9WAJSBv1aTRznNJWOFQ4DGs= 2023-11-02T18:26:44.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TlwM9WAJSBv1aTRznNJWOFQ4DGs=', 'Content-Length': '0'} 2023-11-02T18:26:44.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012e9824ff52ef02c-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-11-02T18:26:44.560 DEBUG:boto:path=/ibtljn-56/e 2023-11-02T18:26:44.561 DEBUG:boto:auth_path=/ibtljn-56/e 2023-11-02T18:26:44.561 DEBUG:boto:path=/ibtljn-56/e?versionId=null 2023-11-02T18:26:44.562 DEBUG:boto:auth_path=/ibtljn-56/e?versionId=null 2023-11-02T18:26:44.562 DEBUG:boto:Method: GET 2023-11-02T18:26:44.562 DEBUG:boto:Path: /ibtljn-56/e?versionId=null 2023-11-02T18:26:44.563 DEBUG:boto:Data: 2023-11-02T18:26:44.563 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.564 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.565 DEBUG:boto:Params: {} 2023-11-02T18:26:44.565 DEBUG:boto:Token: None 2023-11-02T18:26:44.566 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/e?versionId=null 2023-11-02T18:26:44.566 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jih1ZxzYks4HIqbMgcjZ1BEaOoA= 2023-11-02T18:26:44.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jih1ZxzYks4HIqbMgcjZ1BEaOoA=', 'Content-Length': '0'} 2023-11-02T18:26:44.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060fd0c4d0672a0fc-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.571 DEBUG:boto:path=/ibtljn-56/e 2023-11-02T18:26:44.572 DEBUG:boto:auth_path=/ibtljn-56/e 2023-11-02T18:26:44.572 DEBUG:boto:path=/ibtljn-56/e?versionId=null 2023-11-02T18:26:44.572 DEBUG:boto:auth_path=/ibtljn-56/e?versionId=null 2023-11-02T18:26:44.573 DEBUG:boto:Method: GET 2023-11-02T18:26:44.573 DEBUG:boto:Path: /ibtljn-56/e?versionId=null 2023-11-02T18:26:44.574 DEBUG:boto:Data: 2023-11-02T18:26:44.574 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.575 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.575 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.576 DEBUG:boto:Params: {} 2023-11-02T18:26:44.576 DEBUG:boto:Token: None 2023-11-02T18:26:44.577 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/e?versionId=null 2023-11-02T18:26:44.577 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jih1ZxzYks4HIqbMgcjZ1BEaOoA= 2023-11-02T18:26:44.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jih1ZxzYks4HIqbMgcjZ1BEaOoA=', 'Content-Length': '0'} 2023-11-02T18:26:44.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f258c4bd7a72c38-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.622 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.623 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.623 DEBUG:boto:path=/ibtljn-56/?prefix=e 2023-11-02T18:26:44.624 DEBUG:boto:auth_path=/ibtljn-56/?prefix=e 2023-11-02T18:26:44.624 DEBUG:boto:Method: GET 2023-11-02T18:26:44.625 DEBUG:boto:Path: /ibtljn-56/?prefix=e 2023-11-02T18:26:44.625 DEBUG:boto:Data: 2023-11-02T18:26:44.626 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.626 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.626 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.627 DEBUG:boto:Params: {} 2023-11-02T18:26:44.627 DEBUG:boto:Token: None 2023-11-02T18:26:44.628 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.629 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091ccba74e89a25ea-006543e9e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.635 DEBUG:boto:b'ibtljn-56e1000falsee2023-11-02T18:24:47.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.636 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.636 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.637 DEBUG:boto:path=/ibtljn-56/?prefix=e 2023-11-02T18:26:44.637 DEBUG:boto:auth_path=/ibtljn-56/?prefix=e 2023-11-02T18:26:44.638 DEBUG:boto:Method: GET 2023-11-02T18:26:44.638 DEBUG:boto:Path: /ibtljn-56/?prefix=e 2023-11-02T18:26:44.639 DEBUG:boto:Data: 2023-11-02T18:26:44.639 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.640 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.640 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.641 DEBUG:boto:Params: {} 2023-11-02T18:26:44.641 DEBUG:boto:Token: None 2023-11-02T18:26:44.642 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.642 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d459a464c10c9e80-006543e9e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.647 DEBUG:boto:b'ibtljn-56e1000falsee2023-11-02T18:24:47.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.648 DEBUG:boto:path=/ibtljn-56/e 2023-11-02T18:26:44.648 DEBUG:boto:auth_path=/ibtljn-56/e 2023-11-02T18:26:44.649 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.649 DEBUG:boto:Path: /ibtljn-56/e 2023-11-02T18:26:44.650 DEBUG:boto:Data: 2023-11-02T18:26:44.650 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.651 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.652 DEBUG:boto:Params: {} 2023-11-02T18:26:44.652 DEBUG:boto:Token: None 2023-11-02T18:26:44.653 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/e 2023-11-02T18:26:44.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8Z+6JCRc/iLZ8DgtNOW09jyLgj4= 2023-11-02T18:26:44.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8Z+6JCRc/iLZ8DgtNOW09jyLgj4=', 'Content-Length': '0'} 2023-11-02T18:26:44.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7d5f3bcbf954774-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.657 DEBUG:boto:path=/ibtljn-56/e 2023-11-02T18:26:44.658 DEBUG:boto:auth_path=/ibtljn-56/e 2023-11-02T18:26:44.658 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.659 DEBUG:boto:Path: /ibtljn-56/e 2023-11-02T18:26:44.659 DEBUG:boto:Data: 2023-11-02T18:26:44.659 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.660 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.660 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.661 DEBUG:boto:Params: {} 2023-11-02T18:26:44.661 DEBUG:boto:Token: None 2023-11-02T18:26:44.662 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/e 2023-11-02T18:26:44.662 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8Z+6JCRc/iLZ8DgtNOW09jyLgj4= 2023-11-02T18:26:44.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8Z+6JCRc/iLZ8DgtNOW09jyLgj4=', 'Content-Length': '0'} 2023-11-02T18:26:44.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a155c7ee081d4779-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-11-02T18:26:44.667 DEBUG:boto:path=/ibtljn-56/e 2023-11-02T18:26:44.670 DEBUG:boto:auth_path=/ibtljn-56/e 2023-11-02T18:26:44.670 DEBUG:boto:Method: GET 2023-11-02T18:26:44.671 DEBUG:boto:Path: /ibtljn-56/e 2023-11-02T18:26:44.671 DEBUG:boto:Data: 2023-11-02T18:26:44.672 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.672 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.673 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.673 DEBUG:boto:Params: {} 2023-11-02T18:26:44.674 DEBUG:boto:Token: None 2023-11-02T18:26:44.674 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/e 2023-11-02T18:26:44.675 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:q4rScms6Z0ZTXUSGxNI+lCwrQ1k= 2023-11-02T18:26:44.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:q4rScms6Z0ZTXUSGxNI+lCwrQ1k=', 'Content-Length': '0'} 2023-11-02T18:26:44.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd4764d9ff42287a-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.679 DEBUG:boto:path=/ibtljn-56/e 2023-11-02T18:26:44.680 DEBUG:boto:auth_path=/ibtljn-56/e 2023-11-02T18:26:44.680 DEBUG:boto:Method: GET 2023-11-02T18:26:44.681 DEBUG:boto:Path: /ibtljn-56/e 2023-11-02T18:26:44.681 DEBUG:boto:Data: 2023-11-02T18:26:44.682 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.682 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.682 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.683 DEBUG:boto:Params: {} 2023-11-02T18:26:44.683 DEBUG:boto:Token: None 2023-11-02T18:26:44.684 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/e 2023-11-02T18:26:44.685 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:q4rScms6Z0ZTXUSGxNI+lCwrQ1k= 2023-11-02T18:26:44.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:q4rScms6Z0ZTXUSGxNI+lCwrQ1k=', 'Content-Length': '0'} 2023-11-02T18:26:44.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cd2e0079d98e778-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.689 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.689 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.690 DEBUG:boto:path=/ibtljn-56/?prefix=e 2023-11-02T18:26:44.690 DEBUG:boto:auth_path=/ibtljn-56/?prefix=e 2023-11-02T18:26:44.691 DEBUG:boto:Method: GET 2023-11-02T18:26:44.691 DEBUG:boto:Path: /ibtljn-56/?prefix=e 2023-11-02T18:26:44.692 DEBUG:boto:Data: 2023-11-02T18:26:44.692 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.693 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.693 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.694 DEBUG:boto:Params: {} 2023-11-02T18:26:44.694 DEBUG:boto:Token: None 2023-11-02T18:26:44.695 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.696 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5ed47b9224dc1fc-006543e9e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.742 DEBUG:boto:b'ibtljn-56e1000falsee2023-11-02T18:24:47.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.743 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.744 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.744 DEBUG:boto:path=/ibtljn-56/?prefix=e 2023-11-02T18:26:44.745 DEBUG:boto:auth_path=/ibtljn-56/?prefix=e 2023-11-02T18:26:44.745 DEBUG:boto:Method: GET 2023-11-02T18:26:44.746 DEBUG:boto:Path: /ibtljn-56/?prefix=e 2023-11-02T18:26:44.746 DEBUG:boto:Data: 2023-11-02T18:26:44.747 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.747 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.747 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.748 DEBUG:boto:Params: {} 2023-11-02T18:26:44.748 DEBUG:boto:Token: None 2023-11-02T18:26:44.749 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.749 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fc5badf7dc91767-006543e9e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.754 DEBUG:boto:b'ibtljn-56e1000falsee2023-11-02T18:24:47.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-11-02T18:26:44.755 DEBUG:boto:path=/ibtljn-56/f 2023-11-02T18:26:44.755 DEBUG:boto:auth_path=/ibtljn-56/f 2023-11-02T18:26:44.756 DEBUG:boto:path=/ibtljn-56/f?versionId=null 2023-11-02T18:26:44.756 DEBUG:boto:auth_path=/ibtljn-56/f?versionId=null 2023-11-02T18:26:44.757 DEBUG:boto:Method: GET 2023-11-02T18:26:44.757 DEBUG:boto:Path: /ibtljn-56/f?versionId=null 2023-11-02T18:26:44.758 DEBUG:boto:Data: 2023-11-02T18:26:44.758 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.759 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.759 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.759 DEBUG:boto:Params: {} 2023-11-02T18:26:44.760 DEBUG:boto:Token: None 2023-11-02T18:26:44.760 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/f?versionId=null 2023-11-02T18:26:44.761 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5ONTJQvy48zp9BhA+f2K/wSe1BA= 2023-11-02T18:26:44.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5ONTJQvy48zp9BhA+f2K/wSe1BA=', 'Content-Length': '0'} 2023-11-02T18:26:44.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a725171ccb8efc5f-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.766 DEBUG:boto:path=/ibtljn-56/f 2023-11-02T18:26:44.766 DEBUG:boto:auth_path=/ibtljn-56/f 2023-11-02T18:26:44.767 DEBUG:boto:path=/ibtljn-56/f?versionId=null 2023-11-02T18:26:44.767 DEBUG:boto:auth_path=/ibtljn-56/f?versionId=null 2023-11-02T18:26:44.767 DEBUG:boto:Method: GET 2023-11-02T18:26:44.768 DEBUG:boto:Path: /ibtljn-56/f?versionId=null 2023-11-02T18:26:44.768 DEBUG:boto:Data: 2023-11-02T18:26:44.769 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.769 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.770 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.770 DEBUG:boto:Params: {} 2023-11-02T18:26:44.771 DEBUG:boto:Token: None 2023-11-02T18:26:44.771 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/f?versionId=null 2023-11-02T18:26:44.772 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5ONTJQvy48zp9BhA+f2K/wSe1BA= 2023-11-02T18:26:44.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5ONTJQvy48zp9BhA+f2K/wSe1BA=', 'Content-Length': '0'} 2023-11-02T18:26:44.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f99e46fc263b737-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.776 DEBUG:boto:path=/ibtljn-56/f 2023-11-02T18:26:44.776 DEBUG:boto:auth_path=/ibtljn-56/f 2023-11-02T18:26:44.777 DEBUG:boto:path=/ibtljn-56/f?versionId=null 2023-11-02T18:26:44.777 DEBUG:boto:auth_path=/ibtljn-56/f?versionId=null 2023-11-02T18:26:44.778 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.778 DEBUG:boto:Path: /ibtljn-56/f?versionId=null 2023-11-02T18:26:44.778 DEBUG:boto:Data: 2023-11-02T18:26:44.779 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.779 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.780 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.780 DEBUG:boto:Params: {} 2023-11-02T18:26:44.781 DEBUG:boto:Token: None 2023-11-02T18:26:44.781 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/f?versionId=null 2023-11-02T18:26:44.781 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mHYq0JCr/PbDdrrriVQnxObU5N0= 2023-11-02T18:26:44.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mHYq0JCr/PbDdrrriVQnxObU5N0=', 'Content-Length': '0'} 2023-11-02T18:26:44.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038fb7917a7a906af-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.785 DEBUG:boto:path=/ibtljn-56/f 2023-11-02T18:26:44.786 DEBUG:boto:auth_path=/ibtljn-56/f 2023-11-02T18:26:44.786 DEBUG:boto:path=/ibtljn-56/f?versionId=null 2023-11-02T18:26:44.786 DEBUG:boto:auth_path=/ibtljn-56/f?versionId=null 2023-11-02T18:26:44.787 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.787 DEBUG:boto:Path: /ibtljn-56/f?versionId=null 2023-11-02T18:26:44.788 DEBUG:boto:Data: 2023-11-02T18:26:44.788 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.788 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.789 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.789 DEBUG:boto:Params: {} 2023-11-02T18:26:44.790 DEBUG:boto:Token: None 2023-11-02T18:26:44.790 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/f?versionId=null 2023-11-02T18:26:44.791 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mHYq0JCr/PbDdrrriVQnxObU5N0= 2023-11-02T18:26:44.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mHYq0JCr/PbDdrrriVQnxObU5N0=', 'Content-Length': '0'} 2023-11-02T18:26:44.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed6691049bdefe72-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-11-02T18:26:44.795 DEBUG:boto:path=/ibtljn-56/f 2023-11-02T18:26:44.795 DEBUG:boto:auth_path=/ibtljn-56/f 2023-11-02T18:26:44.796 DEBUG:boto:path=/ibtljn-56/f?versionId=null 2023-11-02T18:26:44.796 DEBUG:boto:auth_path=/ibtljn-56/f?versionId=null 2023-11-02T18:26:44.797 DEBUG:boto:Method: GET 2023-11-02T18:26:44.797 DEBUG:boto:Path: /ibtljn-56/f?versionId=null 2023-11-02T18:26:44.798 DEBUG:boto:Data: 2023-11-02T18:26:44.798 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.799 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.799 DEBUG:boto:Params: {} 2023-11-02T18:26:44.800 DEBUG:boto:Token: None 2023-11-02T18:26:44.801 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/f?versionId=null 2023-11-02T18:26:44.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5ONTJQvy48zp9BhA+f2K/wSe1BA= 2023-11-02T18:26:44.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5ONTJQvy48zp9BhA+f2K/wSe1BA=', 'Content-Length': '0'} 2023-11-02T18:26:44.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000831aea9c2bd7f71d-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.805 DEBUG:boto:path=/ibtljn-56/f 2023-11-02T18:26:44.806 DEBUG:boto:auth_path=/ibtljn-56/f 2023-11-02T18:26:44.806 DEBUG:boto:path=/ibtljn-56/f?versionId=null 2023-11-02T18:26:44.806 DEBUG:boto:auth_path=/ibtljn-56/f?versionId=null 2023-11-02T18:26:44.807 DEBUG:boto:Method: GET 2023-11-02T18:26:44.807 DEBUG:boto:Path: /ibtljn-56/f?versionId=null 2023-11-02T18:26:44.808 DEBUG:boto:Data: 2023-11-02T18:26:44.808 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.809 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.809 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.810 DEBUG:boto:Params: {} 2023-11-02T18:26:44.810 DEBUG:boto:Token: None 2023-11-02T18:26:44.811 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/f?versionId=null 2023-11-02T18:26:44.811 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5ONTJQvy48zp9BhA+f2K/wSe1BA= 2023-11-02T18:26:44.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5ONTJQvy48zp9BhA+f2K/wSe1BA=', 'Content-Length': '0'} 2023-11-02T18:26:44.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a3be42ddb142dea-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.858 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.859 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.859 DEBUG:boto:path=/ibtljn-56/?prefix=f 2023-11-02T18:26:44.860 DEBUG:boto:auth_path=/ibtljn-56/?prefix=f 2023-11-02T18:26:44.860 DEBUG:boto:Method: GET 2023-11-02T18:26:44.861 DEBUG:boto:Path: /ibtljn-56/?prefix=f 2023-11-02T18:26:44.861 DEBUG:boto:Data: 2023-11-02T18:26:44.862 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.862 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.863 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.863 DEBUG:boto:Params: {} 2023-11-02T18:26:44.864 DEBUG:boto:Token: None 2023-11-02T18:26:44.864 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.864 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e576e90a2da9a19-006543e9e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.870 DEBUG:boto:b'ibtljn-56f1000falsef2023-11-02T18:24:47.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.871 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.872 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.872 DEBUG:boto:path=/ibtljn-56/?prefix=f 2023-11-02T18:26:44.873 DEBUG:boto:auth_path=/ibtljn-56/?prefix=f 2023-11-02T18:26:44.873 DEBUG:boto:Method: GET 2023-11-02T18:26:44.873 DEBUG:boto:Path: /ibtljn-56/?prefix=f 2023-11-02T18:26:44.874 DEBUG:boto:Data: 2023-11-02T18:26:44.875 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.875 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.875 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.876 DEBUG:boto:Params: {} 2023-11-02T18:26:44.876 DEBUG:boto:Token: None 2023-11-02T18:26:44.877 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083f65408b236f6db-006543e9e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.882 DEBUG:boto:b'ibtljn-56f1000falsef2023-11-02T18:24:47.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.883 DEBUG:boto:path=/ibtljn-56/f 2023-11-02T18:26:44.883 DEBUG:boto:auth_path=/ibtljn-56/f 2023-11-02T18:26:44.884 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.884 DEBUG:boto:Path: /ibtljn-56/f 2023-11-02T18:26:44.885 DEBUG:boto:Data: 2023-11-02T18:26:44.885 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.886 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.886 DEBUG:boto:Params: {} 2023-11-02T18:26:44.887 DEBUG:boto:Token: None 2023-11-02T18:26:44.887 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/f 2023-11-02T18:26:44.888 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rJ9ccLFWfccq40vJ0RCa5cV+XDA= 2023-11-02T18:26:44.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rJ9ccLFWfccq40vJ0RCa5cV+XDA=', 'Content-Length': '0'} 2023-11-02T18:26:44.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e179bc8478d0f29-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.892 DEBUG:boto:path=/ibtljn-56/f 2023-11-02T18:26:44.892 DEBUG:boto:auth_path=/ibtljn-56/f 2023-11-02T18:26:44.893 DEBUG:boto:Method: HEAD 2023-11-02T18:26:44.893 DEBUG:boto:Path: /ibtljn-56/f 2023-11-02T18:26:44.893 DEBUG:boto:Data: 2023-11-02T18:26:44.894 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.894 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.895 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.895 DEBUG:boto:Params: {} 2023-11-02T18:26:44.896 DEBUG:boto:Token: None 2023-11-02T18:26:44.896 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/f 2023-11-02T18:26:44.896 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rJ9ccLFWfccq40vJ0RCa5cV+XDA= 2023-11-02T18:26:44.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rJ9ccLFWfccq40vJ0RCa5cV+XDA=', 'Content-Length': '0'} 2023-11-02T18:26:44.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000798f4fb7793cca4-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-11-02T18:26:44.901 DEBUG:boto:path=/ibtljn-56/f 2023-11-02T18:26:44.902 DEBUG:boto:auth_path=/ibtljn-56/f 2023-11-02T18:26:44.902 DEBUG:boto:Method: GET 2023-11-02T18:26:44.903 DEBUG:boto:Path: /ibtljn-56/f 2023-11-02T18:26:44.903 DEBUG:boto:Data: 2023-11-02T18:26:44.904 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.904 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.905 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.905 DEBUG:boto:Params: {} 2023-11-02T18:26:44.906 DEBUG:boto:Token: None 2023-11-02T18:26:44.906 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/f 2023-11-02T18:26:44.907 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:skZSIzvg1Yvwqt5KFWk5YztBXNM= 2023-11-02T18:26:44.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:skZSIzvg1Yvwqt5KFWk5YztBXNM=', 'Content-Length': '0'} 2023-11-02T18:26:44.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc370732909b0296-006543e9e4-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.950 DEBUG:boto:path=/ibtljn-56/f 2023-11-02T18:26:44.951 DEBUG:boto:auth_path=/ibtljn-56/f 2023-11-02T18:26:44.951 DEBUG:boto:Method: GET 2023-11-02T18:26:44.952 DEBUG:boto:Path: /ibtljn-56/f 2023-11-02T18:26:44.952 DEBUG:boto:Data: 2023-11-02T18:26:44.953 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.953 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.954 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.954 DEBUG:boto:Params: {} 2023-11-02T18:26:44.955 DEBUG:boto:Token: None 2023-11-02T18:26:44.955 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/f 2023-11-02T18:26:44.956 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:skZSIzvg1Yvwqt5KFWk5YztBXNM= 2023-11-02T18:26:44.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:skZSIzvg1Yvwqt5KFWk5YztBXNM=', 'Content-Length': '0'} 2023-11-02T18:26:44.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000086d38d853a940ba-006543e9e4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.960 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.960 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.961 DEBUG:boto:path=/ibtljn-56/?prefix=f 2023-11-02T18:26:44.961 DEBUG:boto:auth_path=/ibtljn-56/?prefix=f 2023-11-02T18:26:44.962 DEBUG:boto:Method: GET 2023-11-02T18:26:44.962 DEBUG:boto:Path: /ibtljn-56/?prefix=f 2023-11-02T18:26:44.963 DEBUG:boto:Data: 2023-11-02T18:26:44.963 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.964 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:44.964 DEBUG:boto:Port: 8001 2023-11-02T18:26:44.965 DEBUG:boto:Params: {} 2023-11-02T18:26:44.965 DEBUG:boto:Token: None 2023-11-02T18:26:44.966 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.966 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000860f09cef26017e3-006543e9e4-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:44.973 DEBUG:boto:b'ibtljn-56f1000falsef2023-11-02T18:24:47.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:44.974 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:44.975 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:44.975 DEBUG:boto:path=/ibtljn-56/?prefix=f 2023-11-02T18:26:44.975 DEBUG:boto:auth_path=/ibtljn-56/?prefix=f 2023-11-02T18:26:44.976 DEBUG:boto:Method: GET 2023-11-02T18:26:44.976 DEBUG:boto:Path: /ibtljn-56/?prefix=f 2023-11-02T18:26:44.977 DEBUG:boto:Data: 2023-11-02T18:26:44.977 DEBUG:boto:Headers: {} 2023-11-02T18:26:44.978 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:44.978 DEBUG:boto:Port: 8000 2023-11-02T18:26:44.979 DEBUG:boto:Params: {} 2023-11-02T18:26:44.979 DEBUG:boto:Token: None 2023-11-02T18:26:44.980 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:44 GMT /ibtljn-56/ 2023-11-02T18:26:44.980 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg= 2023-11-02T18:26:44.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qC5I6J1N8u9CWR65nTkLc1j3pVg=', 'Content-Length': '0'} 2023-11-02T18:26:44.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000119cb08c45a341d4-006543e9e4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.026 DEBUG:boto:b'ibtljn-56f1000falsef2023-11-02T18:24:47.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:45.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-11-02T18:26:45.028 DEBUG:boto:path=/ibtljn-56/g 2023-11-02T18:26:45.028 DEBUG:boto:auth_path=/ibtljn-56/g 2023-11-02T18:26:45.029 DEBUG:boto:path=/ibtljn-56/g?versionId=null 2023-11-02T18:26:45.029 DEBUG:boto:auth_path=/ibtljn-56/g?versionId=null 2023-11-02T18:26:45.030 DEBUG:boto:Method: GET 2023-11-02T18:26:45.030 DEBUG:boto:Path: /ibtljn-56/g?versionId=null 2023-11-02T18:26:45.031 DEBUG:boto:Data: 2023-11-02T18:26:45.031 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.032 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.032 DEBUG:boto:Params: {} 2023-11-02T18:26:45.033 DEBUG:boto:Token: None 2023-11-02T18:26:45.033 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/g?versionId=null 2023-11-02T18:26:45.034 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A/0J63pJBynWtsr4eK0itKnPIJs= 2023-11-02T18:26:45.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A/0J63pJBynWtsr4eK0itKnPIJs=', 'Content-Length': '0'} 2023-11-02T18:26:45.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000180d565cc70ce834-006543e9e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.039 DEBUG:boto:path=/ibtljn-56/g 2023-11-02T18:26:45.039 DEBUG:boto:auth_path=/ibtljn-56/g 2023-11-02T18:26:45.039 DEBUG:boto:path=/ibtljn-56/g?versionId=null 2023-11-02T18:26:45.040 DEBUG:boto:auth_path=/ibtljn-56/g?versionId=null 2023-11-02T18:26:45.040 DEBUG:boto:Method: GET 2023-11-02T18:26:45.041 DEBUG:boto:Path: /ibtljn-56/g?versionId=null 2023-11-02T18:26:45.041 DEBUG:boto:Data: 2023-11-02T18:26:45.042 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.042 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.043 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.043 DEBUG:boto:Params: {} 2023-11-02T18:26:45.043 DEBUG:boto:Token: None 2023-11-02T18:26:45.044 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/g?versionId=null 2023-11-02T18:26:45.044 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A/0J63pJBynWtsr4eK0itKnPIJs= 2023-11-02T18:26:45.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A/0J63pJBynWtsr4eK0itKnPIJs=', 'Content-Length': '0'} 2023-11-02T18:26:45.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bae5867a7a2c5bce-006543e9e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.049 DEBUG:boto:path=/ibtljn-56/g 2023-11-02T18:26:45.049 DEBUG:boto:auth_path=/ibtljn-56/g 2023-11-02T18:26:45.050 DEBUG:boto:path=/ibtljn-56/g?versionId=null 2023-11-02T18:26:45.050 DEBUG:boto:auth_path=/ibtljn-56/g?versionId=null 2023-11-02T18:26:45.050 DEBUG:boto:Method: HEAD 2023-11-02T18:26:45.051 DEBUG:boto:Path: /ibtljn-56/g?versionId=null 2023-11-02T18:26:45.051 DEBUG:boto:Data: 2023-11-02T18:26:45.052 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.052 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.053 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.053 DEBUG:boto:Params: {} 2023-11-02T18:26:45.054 DEBUG:boto:Token: None 2023-11-02T18:26:45.054 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/g?versionId=null 2023-11-02T18:26:45.054 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6Bf0y+mrvlXDp82gR5pdXHh4DOg= 2023-11-02T18:26:45.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6Bf0y+mrvlXDp82gR5pdXHh4DOg=', 'Content-Length': '0'} 2023-11-02T18:26:45.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c2f5e242307298-006543e9e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.058 DEBUG:boto:path=/ibtljn-56/g 2023-11-02T18:26:45.059 DEBUG:boto:auth_path=/ibtljn-56/g 2023-11-02T18:26:45.059 DEBUG:boto:path=/ibtljn-56/g?versionId=null 2023-11-02T18:26:45.060 DEBUG:boto:auth_path=/ibtljn-56/g?versionId=null 2023-11-02T18:26:45.060 DEBUG:boto:Method: HEAD 2023-11-02T18:26:45.061 DEBUG:boto:Path: /ibtljn-56/g?versionId=null 2023-11-02T18:26:45.061 DEBUG:boto:Data: 2023-11-02T18:26:45.061 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.062 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.062 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.063 DEBUG:boto:Params: {} 2023-11-02T18:26:45.063 DEBUG:boto:Token: None 2023-11-02T18:26:45.064 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/g?versionId=null 2023-11-02T18:26:45.064 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6Bf0y+mrvlXDp82gR5pdXHh4DOg= 2023-11-02T18:26:45.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6Bf0y+mrvlXDp82gR5pdXHh4DOg=', 'Content-Length': '0'} 2023-11-02T18:26:45.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046fe2db716df2b50-006543e9e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-11-02T18:26:45.069 DEBUG:boto:path=/ibtljn-56/g 2023-11-02T18:26:45.069 DEBUG:boto:auth_path=/ibtljn-56/g 2023-11-02T18:26:45.070 DEBUG:boto:path=/ibtljn-56/g?versionId=null 2023-11-02T18:26:45.070 DEBUG:boto:auth_path=/ibtljn-56/g?versionId=null 2023-11-02T18:26:45.071 DEBUG:boto:Method: GET 2023-11-02T18:26:45.071 DEBUG:boto:Path: /ibtljn-56/g?versionId=null 2023-11-02T18:26:45.072 DEBUG:boto:Data: 2023-11-02T18:26:45.072 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.073 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.073 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.074 DEBUG:boto:Params: {} 2023-11-02T18:26:45.074 DEBUG:boto:Token: None 2023-11-02T18:26:45.075 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/g?versionId=null 2023-11-02T18:26:45.075 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A/0J63pJBynWtsr4eK0itKnPIJs= 2023-11-02T18:26:45.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A/0J63pJBynWtsr4eK0itKnPIJs=', 'Content-Length': '0'} 2023-11-02T18:26:45.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2d056922c8976fd-006543e9e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.079 DEBUG:boto:path=/ibtljn-56/g 2023-11-02T18:26:45.081 DEBUG:boto:auth_path=/ibtljn-56/g 2023-11-02T18:26:45.082 DEBUG:boto:path=/ibtljn-56/g?versionId=null 2023-11-02T18:26:45.082 DEBUG:boto:auth_path=/ibtljn-56/g?versionId=null 2023-11-02T18:26:45.082 DEBUG:boto:Method: GET 2023-11-02T18:26:45.083 DEBUG:boto:Path: /ibtljn-56/g?versionId=null 2023-11-02T18:26:45.083 DEBUG:boto:Data: 2023-11-02T18:26:45.084 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.084 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.085 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.085 DEBUG:boto:Params: {} 2023-11-02T18:26:45.086 DEBUG:boto:Token: None 2023-11-02T18:26:45.086 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/g?versionId=null 2023-11-02T18:26:45.087 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A/0J63pJBynWtsr4eK0itKnPIJs= 2023-11-02T18:26:45.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A/0J63pJBynWtsr4eK0itKnPIJs=', 'Content-Length': '0'} 2023-11-02T18:26:45.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029002a47efafbe2a-006543e9e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.134 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:45.135 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:45.136 DEBUG:boto:path=/ibtljn-56/?prefix=g 2023-11-02T18:26:45.136 DEBUG:boto:auth_path=/ibtljn-56/?prefix=g 2023-11-02T18:26:45.137 DEBUG:boto:Method: GET 2023-11-02T18:26:45.137 DEBUG:boto:Path: /ibtljn-56/?prefix=g 2023-11-02T18:26:45.138 DEBUG:boto:Data: 2023-11-02T18:26:45.138 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.139 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.140 DEBUG:boto:Params: {} 2023-11-02T18:26:45.140 DEBUG:boto:Token: None 2023-11-02T18:26:45.141 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/ 2023-11-02T18:26:45.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68= 2023-11-02T18:26:45.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68=', 'Content-Length': '0'} 2023-11-02T18:26:45.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1ec4dd33a242b45-006543e9e5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.147 DEBUG:boto:b'ibtljn-56g1000falseg2023-11-02T18:24:47.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:45.148 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:45.148 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:45.149 DEBUG:boto:path=/ibtljn-56/?prefix=g 2023-11-02T18:26:45.149 DEBUG:boto:auth_path=/ibtljn-56/?prefix=g 2023-11-02T18:26:45.150 DEBUG:boto:Method: GET 2023-11-02T18:26:45.150 DEBUG:boto:Path: /ibtljn-56/?prefix=g 2023-11-02T18:26:45.151 DEBUG:boto:Data: 2023-11-02T18:26:45.151 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.152 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.152 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.153 DEBUG:boto:Params: {} 2023-11-02T18:26:45.153 DEBUG:boto:Token: None 2023-11-02T18:26:45.154 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/ 2023-11-02T18:26:45.154 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68= 2023-11-02T18:26:45.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68=', 'Content-Length': '0'} 2023-11-02T18:26:45.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000554182ff2af9fa67-006543e9e5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.159 DEBUG:boto:b'ibtljn-56g1000falseg2023-11-02T18:24:47.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:45.159 DEBUG:boto:path=/ibtljn-56/g 2023-11-02T18:26:45.160 DEBUG:boto:auth_path=/ibtljn-56/g 2023-11-02T18:26:45.160 DEBUG:boto:Method: HEAD 2023-11-02T18:26:45.161 DEBUG:boto:Path: /ibtljn-56/g 2023-11-02T18:26:45.161 DEBUG:boto:Data: 2023-11-02T18:26:45.161 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.162 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.162 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.163 DEBUG:boto:Params: {} 2023-11-02T18:26:45.163 DEBUG:boto:Token: None 2023-11-02T18:26:45.164 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/g 2023-11-02T18:26:45.164 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XFLZ8QlnzSb5oExpV+3V+kr0LZQ= 2023-11-02T18:26:45.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XFLZ8QlnzSb5oExpV+3V+kr0LZQ=', 'Content-Length': '0'} 2023-11-02T18:26:45.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e46333e2c842c45-006543e9e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.168 DEBUG:boto:path=/ibtljn-56/g 2023-11-02T18:26:45.169 DEBUG:boto:auth_path=/ibtljn-56/g 2023-11-02T18:26:45.169 DEBUG:boto:Method: HEAD 2023-11-02T18:26:45.169 DEBUG:boto:Path: /ibtljn-56/g 2023-11-02T18:26:45.170 DEBUG:boto:Data: 2023-11-02T18:26:45.170 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.170 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.171 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.171 DEBUG:boto:Params: {} 2023-11-02T18:26:45.172 DEBUG:boto:Token: None 2023-11-02T18:26:45.172 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/g 2023-11-02T18:26:45.173 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XFLZ8QlnzSb5oExpV+3V+kr0LZQ= 2023-11-02T18:26:45.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XFLZ8QlnzSb5oExpV+3V+kr0LZQ=', 'Content-Length': '0'} 2023-11-02T18:26:45.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d63f6804e0920cb2-006543e9e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-11-02T18:26:45.177 DEBUG:boto:path=/ibtljn-56/g 2023-11-02T18:26:45.178 DEBUG:boto:auth_path=/ibtljn-56/g 2023-11-02T18:26:45.178 DEBUG:boto:Method: GET 2023-11-02T18:26:45.179 DEBUG:boto:Path: /ibtljn-56/g 2023-11-02T18:26:45.179 DEBUG:boto:Data: 2023-11-02T18:26:45.179 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.180 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.180 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.181 DEBUG:boto:Params: {} 2023-11-02T18:26:45.181 DEBUG:boto:Token: None 2023-11-02T18:26:45.182 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/g 2023-11-02T18:26:45.182 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGkR1/OGMegUEeyb8FgscD8ILe0= 2023-11-02T18:26:45.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGkR1/OGMegUEeyb8FgscD8ILe0=', 'Content-Length': '0'} 2023-11-02T18:26:45.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060270728c3f37613-006543e9e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.230 DEBUG:boto:path=/ibtljn-56/g 2023-11-02T18:26:45.231 DEBUG:boto:auth_path=/ibtljn-56/g 2023-11-02T18:26:45.231 DEBUG:boto:Method: GET 2023-11-02T18:26:45.232 DEBUG:boto:Path: /ibtljn-56/g 2023-11-02T18:26:45.232 DEBUG:boto:Data: 2023-11-02T18:26:45.233 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.233 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.234 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.234 DEBUG:boto:Params: {} 2023-11-02T18:26:45.235 DEBUG:boto:Token: None 2023-11-02T18:26:45.235 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/g 2023-11-02T18:26:45.236 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PGkR1/OGMegUEeyb8FgscD8ILe0= 2023-11-02T18:26:45.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PGkR1/OGMegUEeyb8FgscD8ILe0=', 'Content-Length': '0'} 2023-11-02T18:26:45.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000986bb00a2a38aa6c-006543e9e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.240 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:45.240 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:45.241 DEBUG:boto:path=/ibtljn-56/?prefix=g 2023-11-02T18:26:45.241 DEBUG:boto:auth_path=/ibtljn-56/?prefix=g 2023-11-02T18:26:45.242 DEBUG:boto:Method: GET 2023-11-02T18:26:45.242 DEBUG:boto:Path: /ibtljn-56/?prefix=g 2023-11-02T18:26:45.243 DEBUG:boto:Data: 2023-11-02T18:26:45.243 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.243 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.244 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.244 DEBUG:boto:Params: {} 2023-11-02T18:26:45.245 DEBUG:boto:Token: None 2023-11-02T18:26:45.245 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/ 2023-11-02T18:26:45.246 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68= 2023-11-02T18:26:45.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68=', 'Content-Length': '0'} 2023-11-02T18:26:45.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049779a8ef0ab2c4a-006543e9e5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.253 DEBUG:boto:b'ibtljn-56g1000falseg2023-11-02T18:24:47.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:45.253 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:45.254 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:45.254 DEBUG:boto:path=/ibtljn-56/?prefix=g 2023-11-02T18:26:45.255 DEBUG:boto:auth_path=/ibtljn-56/?prefix=g 2023-11-02T18:26:45.255 DEBUG:boto:Method: GET 2023-11-02T18:26:45.256 DEBUG:boto:Path: /ibtljn-56/?prefix=g 2023-11-02T18:26:45.256 DEBUG:boto:Data: 2023-11-02T18:26:45.256 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.257 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.257 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.258 DEBUG:boto:Params: {} 2023-11-02T18:26:45.258 DEBUG:boto:Token: None 2023-11-02T18:26:45.259 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/ 2023-11-02T18:26:45.259 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68= 2023-11-02T18:26:45.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68=', 'Content-Length': '0'} 2023-11-02T18:26:45.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beb5c2d6a34127f5-006543e9e5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.310 DEBUG:boto:b'ibtljn-56g1000falseg2023-11-02T18:24:47.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:45.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-11-02T18:26:45.312 DEBUG:boto:path=/ibtljn-56/h 2023-11-02T18:26:45.312 DEBUG:boto:auth_path=/ibtljn-56/h 2023-11-02T18:26:45.313 DEBUG:boto:path=/ibtljn-56/h?versionId=null 2023-11-02T18:26:45.313 DEBUG:boto:auth_path=/ibtljn-56/h?versionId=null 2023-11-02T18:26:45.313 DEBUG:boto:Method: GET 2023-11-02T18:26:45.314 DEBUG:boto:Path: /ibtljn-56/h?versionId=null 2023-11-02T18:26:45.314 DEBUG:boto:Data: 2023-11-02T18:26:45.315 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.315 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.316 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.316 DEBUG:boto:Params: {} 2023-11-02T18:26:45.317 DEBUG:boto:Token: None 2023-11-02T18:26:45.317 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/h?versionId=null 2023-11-02T18:26:45.318 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:elaM4GVUWBfnwM2M+kyltGmWH8M= 2023-11-02T18:26:45.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:elaM4GVUWBfnwM2M+kyltGmWH8M=', 'Content-Length': '0'} 2023-11-02T18:26:45.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013fee02f34657264-006543e9e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.322 DEBUG:boto:path=/ibtljn-56/h 2023-11-02T18:26:45.323 DEBUG:boto:auth_path=/ibtljn-56/h 2023-11-02T18:26:45.323 DEBUG:boto:path=/ibtljn-56/h?versionId=null 2023-11-02T18:26:45.323 DEBUG:boto:auth_path=/ibtljn-56/h?versionId=null 2023-11-02T18:26:45.324 DEBUG:boto:Method: GET 2023-11-02T18:26:45.324 DEBUG:boto:Path: /ibtljn-56/h?versionId=null 2023-11-02T18:26:45.325 DEBUG:boto:Data: 2023-11-02T18:26:45.325 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.326 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.326 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.327 DEBUG:boto:Params: {} 2023-11-02T18:26:45.327 DEBUG:boto:Token: None 2023-11-02T18:26:45.328 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/h?versionId=null 2023-11-02T18:26:45.328 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:elaM4GVUWBfnwM2M+kyltGmWH8M= 2023-11-02T18:26:45.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:elaM4GVUWBfnwM2M+kyltGmWH8M=', 'Content-Length': '0'} 2023-11-02T18:26:45.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000674c0244412325da-006543e9e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.332 DEBUG:boto:path=/ibtljn-56/h 2023-11-02T18:26:45.333 DEBUG:boto:auth_path=/ibtljn-56/h 2023-11-02T18:26:45.333 DEBUG:boto:path=/ibtljn-56/h?versionId=null 2023-11-02T18:26:45.334 DEBUG:boto:auth_path=/ibtljn-56/h?versionId=null 2023-11-02T18:26:45.334 DEBUG:boto:Method: HEAD 2023-11-02T18:26:45.335 DEBUG:boto:Path: /ibtljn-56/h?versionId=null 2023-11-02T18:26:45.335 DEBUG:boto:Data: 2023-11-02T18:26:45.336 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.336 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.336 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.337 DEBUG:boto:Params: {} 2023-11-02T18:26:45.337 DEBUG:boto:Token: None 2023-11-02T18:26:45.338 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/h?versionId=null 2023-11-02T18:26:45.338 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TLs3tz/LD/rYsN7+KGh0nOD+nPA= 2023-11-02T18:26:45.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TLs3tz/LD/rYsN7+KGh0nOD+nPA=', 'Content-Length': '0'} 2023-11-02T18:26:45.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016708f8815fc6dfa-006543e9e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.342 DEBUG:boto:path=/ibtljn-56/h 2023-11-02T18:26:45.343 DEBUG:boto:auth_path=/ibtljn-56/h 2023-11-02T18:26:45.343 DEBUG:boto:path=/ibtljn-56/h?versionId=null 2023-11-02T18:26:45.343 DEBUG:boto:auth_path=/ibtljn-56/h?versionId=null 2023-11-02T18:26:45.344 DEBUG:boto:Method: HEAD 2023-11-02T18:26:45.344 DEBUG:boto:Path: /ibtljn-56/h?versionId=null 2023-11-02T18:26:45.345 DEBUG:boto:Data: 2023-11-02T18:26:45.345 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.346 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.346 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.346 DEBUG:boto:Params: {} 2023-11-02T18:26:45.347 DEBUG:boto:Token: None 2023-11-02T18:26:45.347 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/h?versionId=null 2023-11-02T18:26:45.348 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TLs3tz/LD/rYsN7+KGh0nOD+nPA= 2023-11-02T18:26:45.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TLs3tz/LD/rYsN7+KGh0nOD+nPA=', 'Content-Length': '0'} 2023-11-02T18:26:45.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036d4b922b43b28a8-006543e9e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-11-02T18:26:45.353 DEBUG:boto:path=/ibtljn-56/h 2023-11-02T18:26:45.354 DEBUG:boto:auth_path=/ibtljn-56/h 2023-11-02T18:26:45.354 DEBUG:boto:path=/ibtljn-56/h?versionId=null 2023-11-02T18:26:45.354 DEBUG:boto:auth_path=/ibtljn-56/h?versionId=null 2023-11-02T18:26:45.355 DEBUG:boto:Method: GET 2023-11-02T18:26:45.355 DEBUG:boto:Path: /ibtljn-56/h?versionId=null 2023-11-02T18:26:45.356 DEBUG:boto:Data: 2023-11-02T18:26:45.356 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.357 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.357 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.358 DEBUG:boto:Params: {} 2023-11-02T18:26:45.358 DEBUG:boto:Token: None 2023-11-02T18:26:45.359 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/h?versionId=null 2023-11-02T18:26:45.359 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:elaM4GVUWBfnwM2M+kyltGmWH8M= 2023-11-02T18:26:45.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:elaM4GVUWBfnwM2M+kyltGmWH8M=', 'Content-Length': '0'} 2023-11-02T18:26:45.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa51080db28df84a-006543e9e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.363 DEBUG:boto:path=/ibtljn-56/h 2023-11-02T18:26:45.363 DEBUG:boto:auth_path=/ibtljn-56/h 2023-11-02T18:26:45.364 DEBUG:boto:path=/ibtljn-56/h?versionId=null 2023-11-02T18:26:45.364 DEBUG:boto:auth_path=/ibtljn-56/h?versionId=null 2023-11-02T18:26:45.365 DEBUG:boto:Method: GET 2023-11-02T18:26:45.365 DEBUG:boto:Path: /ibtljn-56/h?versionId=null 2023-11-02T18:26:45.366 DEBUG:boto:Data: 2023-11-02T18:26:45.366 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.367 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.367 DEBUG:boto:Params: {} 2023-11-02T18:26:45.368 DEBUG:boto:Token: None 2023-11-02T18:26:45.368 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/h?versionId=null 2023-11-02T18:26:45.369 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:elaM4GVUWBfnwM2M+kyltGmWH8M= 2023-11-02T18:26:45.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:elaM4GVUWBfnwM2M+kyltGmWH8M=', 'Content-Length': '0'} 2023-11-02T18:26:45.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9b7732f6efc5c19-006543e9e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.418 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:45.419 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:45.420 DEBUG:boto:path=/ibtljn-56/?prefix=h 2023-11-02T18:26:45.420 DEBUG:boto:auth_path=/ibtljn-56/?prefix=h 2023-11-02T18:26:45.421 DEBUG:boto:Method: GET 2023-11-02T18:26:45.421 DEBUG:boto:Path: /ibtljn-56/?prefix=h 2023-11-02T18:26:45.422 DEBUG:boto:Data: 2023-11-02T18:26:45.423 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.423 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.424 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.424 DEBUG:boto:Params: {} 2023-11-02T18:26:45.425 DEBUG:boto:Token: None 2023-11-02T18:26:45.425 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/ 2023-11-02T18:26:45.426 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68= 2023-11-02T18:26:45.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68=', 'Content-Length': '0'} 2023-11-02T18:26:45.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017559d678e4bd000-006543e9e5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.432 DEBUG:boto:b'ibtljn-56h1000falseh2023-11-02T18:24:48.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:45.433 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:45.434 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:45.434 DEBUG:boto:path=/ibtljn-56/?prefix=h 2023-11-02T18:26:45.435 DEBUG:boto:auth_path=/ibtljn-56/?prefix=h 2023-11-02T18:26:45.435 DEBUG:boto:Method: GET 2023-11-02T18:26:45.436 DEBUG:boto:Path: /ibtljn-56/?prefix=h 2023-11-02T18:26:45.436 DEBUG:boto:Data: 2023-11-02T18:26:45.437 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.438 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.438 DEBUG:boto:Params: {} 2023-11-02T18:26:45.439 DEBUG:boto:Token: None 2023-11-02T18:26:45.439 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/ 2023-11-02T18:26:45.440 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68= 2023-11-02T18:26:45.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68=', 'Content-Length': '0'} 2023-11-02T18:26:45.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025e29f056ae2723f-006543e9e5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.445 DEBUG:boto:b'ibtljn-56h1000falseh2023-11-02T18:24:48.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:45.446 DEBUG:boto:path=/ibtljn-56/h 2023-11-02T18:26:45.446 DEBUG:boto:auth_path=/ibtljn-56/h 2023-11-02T18:26:45.447 DEBUG:boto:Method: HEAD 2023-11-02T18:26:45.447 DEBUG:boto:Path: /ibtljn-56/h 2023-11-02T18:26:45.448 DEBUG:boto:Data: 2023-11-02T18:26:45.448 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.449 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.449 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.450 DEBUG:boto:Params: {} 2023-11-02T18:26:45.450 DEBUG:boto:Token: None 2023-11-02T18:26:45.451 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/h 2023-11-02T18:26:45.451 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MQPOL318a441WQslM9gUc1nVGu4= 2023-11-02T18:26:45.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MQPOL318a441WQslM9gUc1nVGu4=', 'Content-Length': '0'} 2023-11-02T18:26:45.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae8434a41929e39a-006543e9e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.455 DEBUG:boto:path=/ibtljn-56/h 2023-11-02T18:26:45.456 DEBUG:boto:auth_path=/ibtljn-56/h 2023-11-02T18:26:45.456 DEBUG:boto:Method: HEAD 2023-11-02T18:26:45.457 DEBUG:boto:Path: /ibtljn-56/h 2023-11-02T18:26:45.457 DEBUG:boto:Data: 2023-11-02T18:26:45.458 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.458 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.459 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.459 DEBUG:boto:Params: {} 2023-11-02T18:26:45.460 DEBUG:boto:Token: None 2023-11-02T18:26:45.460 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/h 2023-11-02T18:26:45.461 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MQPOL318a441WQslM9gUc1nVGu4= 2023-11-02T18:26:45.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MQPOL318a441WQslM9gUc1nVGu4=', 'Content-Length': '0'} 2023-11-02T18:26:45.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001feda83057c3d1eb-006543e9e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-11-02T18:26:45.465 DEBUG:boto:path=/ibtljn-56/h 2023-11-02T18:26:45.466 DEBUG:boto:auth_path=/ibtljn-56/h 2023-11-02T18:26:45.466 DEBUG:boto:Method: GET 2023-11-02T18:26:45.467 DEBUG:boto:Path: /ibtljn-56/h 2023-11-02T18:26:45.467 DEBUG:boto:Data: 2023-11-02T18:26:45.468 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.468 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.468 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.469 DEBUG:boto:Params: {} 2023-11-02T18:26:45.469 DEBUG:boto:Token: None 2023-11-02T18:26:45.470 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/h 2023-11-02T18:26:45.470 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f0PDqnPVHmxx8D5p7EIJMew/FYE= 2023-11-02T18:26:45.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f0PDqnPVHmxx8D5p7EIJMew/FYE=', 'Content-Length': '0'} 2023-11-02T18:26:45.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-56:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad83f7cd2ff6476d-006543e9e5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.514 DEBUG:boto:path=/ibtljn-56/h 2023-11-02T18:26:45.515 DEBUG:boto:auth_path=/ibtljn-56/h 2023-11-02T18:26:45.515 DEBUG:boto:Method: GET 2023-11-02T18:26:45.516 DEBUG:boto:Path: /ibtljn-56/h 2023-11-02T18:26:45.516 DEBUG:boto:Data: 2023-11-02T18:26:45.518 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.518 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.519 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.519 DEBUG:boto:Params: {} 2023-11-02T18:26:45.520 DEBUG:boto:Token: None 2023-11-02T18:26:45.521 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/h 2023-11-02T18:26:45.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f0PDqnPVHmxx8D5p7EIJMew/FYE= 2023-11-02T18:26:45.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f0PDqnPVHmxx8D5p7EIJMew/FYE=', 'Content-Length': '0'} 2023-11-02T18:26:45.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:24:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab5ba1166bedd0f6-006543e9e5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.526 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:45.526 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:45.527 DEBUG:boto:path=/ibtljn-56/?prefix=h 2023-11-02T18:26:45.527 DEBUG:boto:auth_path=/ibtljn-56/?prefix=h 2023-11-02T18:26:45.528 DEBUG:boto:Method: GET 2023-11-02T18:26:45.528 DEBUG:boto:Path: /ibtljn-56/?prefix=h 2023-11-02T18:26:45.529 DEBUG:boto:Data: 2023-11-02T18:26:45.529 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.530 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:45.530 DEBUG:boto:Port: 8001 2023-11-02T18:26:45.531 DEBUG:boto:Params: {} 2023-11-02T18:26:45.531 DEBUG:boto:Token: None 2023-11-02T18:26:45.532 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/ 2023-11-02T18:26:45.532 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68= 2023-11-02T18:26:45.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68=', 'Content-Length': '0'} 2023-11-02T18:26:45.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049fe57308a245803-006543e9e5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.538 DEBUG:boto:b'ibtljn-56h1000falseh2023-11-02T18:24:48.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:45.539 DEBUG:boto:path=/ibtljn-56/ 2023-11-02T18:26:45.540 DEBUG:boto:auth_path=/ibtljn-56/ 2023-11-02T18:26:45.540 DEBUG:boto:path=/ibtljn-56/?prefix=h 2023-11-02T18:26:45.541 DEBUG:boto:auth_path=/ibtljn-56/?prefix=h 2023-11-02T18:26:45.541 DEBUG:boto:Method: GET 2023-11-02T18:26:45.542 DEBUG:boto:Path: /ibtljn-56/?prefix=h 2023-11-02T18:26:45.543 DEBUG:boto:Data: 2023-11-02T18:26:45.543 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.544 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.544 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.544 DEBUG:boto:Params: {} 2023-11-02T18:26:45.545 DEBUG:boto:Token: None 2023-11-02T18:26:45.546 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-56/ 2023-11-02T18:26:45.546 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68= 2023-11-02T18:26:45.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X5LF83BL4NQlj61UVduNSTA7X68=', 'Content-Length': '0'} 2023-11-02T18:26:45.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068d53252e662b8c4-006543e9e5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.551 DEBUG:boto:b'ibtljn-56h1000falseh2023-11-02T18:24:48.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:26:45.552 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-56 zones={a2, a1} 2023-11-02T18:26:45.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2023-11-02T18:26:45.554 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:26:45.554 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:26:45.555 DEBUG:boto:Method: PUT 2023-11-02T18:26:45.555 DEBUG:boto:Path: /ibtljn-57/ 2023-11-02T18:26:45.556 DEBUG:boto:Data: 2023-11-02T18:26:45.556 DEBUG:boto:Headers: {} 2023-11-02T18:26:45.557 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:45.557 DEBUG:boto:Port: 8000 2023-11-02T18:26:45.558 DEBUG:boto:Params: {} 2023-11-02T18:26:45.558 DEBUG:boto:Token: None 2023-11-02T18:26:45.559 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:26:45 GMT /ibtljn-57/ 2023-11-02T18:26:45.559 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zh3Ufnyqz+s0sDgzMg6vGhNHgys= 2023-11-02T18:26:45.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zh3Ufnyqz+s0sDgzMg6vGhNHgys=', 'Content-Length': '0'} 2023-11-02T18:26:45.593 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d98b82454609a1e-006543e9e5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:26:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:45.594 DEBUG:rgw_multi.tests:created bucket=ibtljn-57 2023-11-02T18:26:45.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:26:45.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:26:45.595 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:26:45.636 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:26:45.636 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:26:45.694 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:26:45.723 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:26:45.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:26:45.724 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:26:45.724 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:26:45.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:45.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:26:45.725 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:26:45.725 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:26:45.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:45.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:26:45.726 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949605.589431_445.1", 2023-11-02T18:26:45.726 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:26:45.589431Z" 2023-11-02T18:26:45.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:45.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:26:45.727 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:26:45.727 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:26:45.727 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:26:45.728 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:26:45.729 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949605.589431_445.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:26:45.729 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:26:45.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:26:45.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:26:45.731 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:26:45.782 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:26:45.783 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:26:45.837 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:45.835+0000 7fd40eb4bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:26:45.839 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:45.835+0000 7fd40eb4bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:26:45.866 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:26:45.866 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:26:45.866 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:26:45.866 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:26:45.867 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:26:45.867 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:26:45.867 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:26:45.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:45.868 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:26:45.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:26:45.868 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:26:45.868 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:26:45.869 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:26:45.869 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:26:45.869 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:26:45.869 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:26:45.869 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:26:45.870 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:26:45.870 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:26:45.870 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:26:45.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:45.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:26:45.871 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:26:45.871 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:26:45.871 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:26:45.871 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:26:45.872 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:26:45.872 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:26:45.872 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:26:45.872 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:26:45.872 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:26:45.873 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:26:45.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:45.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:26:45.873 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:26:45.873 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:26:45.874 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:26:45.874 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949605.581183_442.1", 2023-11-02T18:26:45.874 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:26:45.874 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:26:45.874 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:26:45.875 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:26:45.581183Z", 2023-11-02T18:26:45.875 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:26:45.875 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:26:45.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:45.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:26:45.876 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:26:45.876 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:26:45.876 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:26:45.877 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:26:45.877 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:26:45.877 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:26:45.877 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:26:45.878 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:26:45.878 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:26:45.878 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:26:45.878 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:26:45.878 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:26:45.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:45.879 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:26:45.879 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:26:45.879 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:26:45.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:45.880 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:26:45Z" 2023-11-02T18:26:45.880 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:26:45.881 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949605.581183_442.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:26:45.581183Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:26:45Z" } 2023-11-02T18:26:45.882 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949605.581183_442.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:26:45.581183Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:26:45.882 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949605.589431_445.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:26:45.882 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949605.581183_442.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:26:45.883 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949605.589431_445.1 target=1_1698949605.581183_442.1 2023-11-02T18:26:50.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:26:50.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:26:50.885 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:26:50.933 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:26:50.933 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:26:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:50.991+0000 7f0fe6e04f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:26:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:50.991+0000 7f0fe6e04f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:26:51.021 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:26:51.021 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:26:51.021 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:26:51.022 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:26:51.022 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:26:51.022 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:26:51.022 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:26:51.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:51.023 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:26:51.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:26:51.023 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:26:51.023 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:26:51.024 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:26:51.024 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:26:51.024 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:26:51.024 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:26:51.024 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:26:51.025 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:26:51.025 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:26:51.025 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:26:51.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:51.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:26:51.026 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:26:51.026 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:26:51.026 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:26:51.026 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:26:51.026 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:26:51.027 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:26:51.027 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:26:51.027 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:26:51.027 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:26:51.027 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:26:51.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:51.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:26:51.028 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:26:51.028 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:26:51.028 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:26:51.029 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949605.589431_445.1", 2023-11-02T18:26:51.029 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:26:51.029 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:26:51.029 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:26:51.029 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:26:45.589431Z", 2023-11-02T18:26:51.030 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:26:51.030 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:26:51.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:51.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:26:51.030 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:26:51.031 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:26:51.031 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:26:51.031 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:26:51.031 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:26:51.031 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:26:51.032 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:26:51.032 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:26:51.032 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:26:51.032 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:26:51.032 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:26:51.033 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:26:51.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:51.033 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:26:51.033 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:26:51.033 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:26:51.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:26:51.034 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:26:50Z" 2023-11-02T18:26:51.034 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:26:51.035 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949605.589431_445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:26:45.589431Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:26:50Z" } 2023-11-02T18:26:51.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949605.589431_445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:26:45.589431Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:26:51.036 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949605.589431_445.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:26:51.036 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949605.589431_445.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:26:51.037 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:26:51.037 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-11-02T18:26:51.038 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T18:26:51.039 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:26:51.035+0000 7f6e14f46700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 35091) UID: 0 2023-11-02T18:26:51.039 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:26:51.035+0000 7f6e187a5f80 -1 shutting down 2023-11-02T18:26:53.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-11-02T18:26:57.139 INFO:tasks.rgw.c1.client.1:Stopped 2023-11-02T18:26:57.140 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:26:57.140 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:26:57.141 DEBUG:boto:Method: HEAD 2023-11-02T18:26:57.141 DEBUG:boto:Path: /ibtljn-57/ 2023-11-02T18:26:57.142 DEBUG:boto:Data: 2023-11-02T18:26:57.142 DEBUG:boto:Headers: {} 2023-11-02T18:26:57.142 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.143 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.143 DEBUG:boto:Params: {} 2023-11-02T18:26:57.144 DEBUG:boto:Token: None 2023-11-02T18:26:57.144 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/ 2023-11-02T18:26:57.145 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs= 2023-11-02T18:26:57.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs=', 'Content-Length': '0'} 2023-11-02T18:26:57.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044fadc524f42f30d-006543e9f1-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.152 DEBUG:boto:path=/ibtljn-57/a 2023-11-02T18:26:57.152 DEBUG:boto:auth_path=/ibtljn-57/a 2023-11-02T18:26:57.153 DEBUG:boto:Method: PUT 2023-11-02T18:26:57.153 DEBUG:boto:Path: /ibtljn-57/a 2023-11-02T18:26:57.153 DEBUG:boto:Data: 2023-11-02T18:26:57.154 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:26:57.154 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.155 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.155 DEBUG:boto:Params: {} 2023-11-02T18:26:57.156 DEBUG:boto:Token: None 2023-11-02T18:26:57.156 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/a 2023-11-02T18:26:57.156 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+ImeDrH/mwrGYwVUGhmq30csQZY= 2023-11-02T18:26:57.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+ImeDrH/mwrGYwVUGhmq30csQZY='} 2023-11-02T18:26:57.172 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcadd595fe8e4e15-006543e9f1-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.173 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:26:57.173 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:26:57.174 DEBUG:boto:Method: HEAD 2023-11-02T18:26:57.174 DEBUG:boto:Path: /ibtljn-57/ 2023-11-02T18:26:57.174 DEBUG:boto:Data: 2023-11-02T18:26:57.175 DEBUG:boto:Headers: {} 2023-11-02T18:26:57.175 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.176 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.176 DEBUG:boto:Params: {} 2023-11-02T18:26:57.176 DEBUG:boto:Token: None 2023-11-02T18:26:57.177 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/ 2023-11-02T18:26:57.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs= 2023-11-02T18:26:57.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs=', 'Content-Length': '0'} 2023-11-02T18:26:57.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002be97a8d2b354ab-006543e9f1-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.182 DEBUG:boto:path=/ibtljn-57/b 2023-11-02T18:26:57.183 DEBUG:boto:auth_path=/ibtljn-57/b 2023-11-02T18:26:57.183 DEBUG:boto:Method: PUT 2023-11-02T18:26:57.184 DEBUG:boto:Path: /ibtljn-57/b 2023-11-02T18:26:57.184 DEBUG:boto:Data: 2023-11-02T18:26:57.184 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:26:57.185 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.185 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.186 DEBUG:boto:Params: {} 2023-11-02T18:26:57.186 DEBUG:boto:Token: None 2023-11-02T18:26:57.187 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/b 2023-11-02T18:26:57.187 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuQTr30OPF+BH7ws6ZO/VCRVgdA= 2023-11-02T18:26:57.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuQTr30OPF+BH7ws6ZO/VCRVgdA='} 2023-11-02T18:26:57.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024289dcd3c7e2ca4-006543e9f1-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.200 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:26:57.200 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:26:57.201 DEBUG:boto:Method: HEAD 2023-11-02T18:26:57.201 DEBUG:boto:Path: /ibtljn-57/ 2023-11-02T18:26:57.201 DEBUG:boto:Data: 2023-11-02T18:26:57.202 DEBUG:boto:Headers: {} 2023-11-02T18:26:57.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.203 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.203 DEBUG:boto:Params: {} 2023-11-02T18:26:57.203 DEBUG:boto:Token: None 2023-11-02T18:26:57.204 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/ 2023-11-02T18:26:57.204 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs= 2023-11-02T18:26:57.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs=', 'Content-Length': '0'} 2023-11-02T18:26:57.209 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d36739e76e7b9cc0-006543e9f1-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.209 DEBUG:boto:path=/ibtljn-57/c 2023-11-02T18:26:57.210 DEBUG:boto:auth_path=/ibtljn-57/c 2023-11-02T18:26:57.210 DEBUG:boto:Method: PUT 2023-11-02T18:26:57.211 DEBUG:boto:Path: /ibtljn-57/c 2023-11-02T18:26:57.211 DEBUG:boto:Data: 2023-11-02T18:26:57.211 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:26:57.212 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.212 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.213 DEBUG:boto:Params: {} 2023-11-02T18:26:57.213 DEBUG:boto:Token: None 2023-11-02T18:26:57.214 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/c 2023-11-02T18:26:57.214 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fZx3Gw9+FgLHyFli5265WZFfhHA= 2023-11-02T18:26:57.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fZx3Gw9+FgLHyFli5265WZFfhHA='} 2023-11-02T18:26:57.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048811c5c86d6d934-006543e9f1-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.259 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:26:57.259 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:26:57.260 DEBUG:boto:Method: HEAD 2023-11-02T18:26:57.260 DEBUG:boto:Path: /ibtljn-57/ 2023-11-02T18:26:57.260 DEBUG:boto:Data: 2023-11-02T18:26:57.261 DEBUG:boto:Headers: {} 2023-11-02T18:26:57.261 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.262 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.262 DEBUG:boto:Params: {} 2023-11-02T18:26:57.263 DEBUG:boto:Token: None 2023-11-02T18:26:57.263 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/ 2023-11-02T18:26:57.263 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs= 2023-11-02T18:26:57.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs=', 'Content-Length': '0'} 2023-11-02T18:26:57.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085207fda9cd418c1-006543e9f1-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.270 DEBUG:boto:path=/ibtljn-57/d 2023-11-02T18:26:57.270 DEBUG:boto:auth_path=/ibtljn-57/d 2023-11-02T18:26:57.271 DEBUG:boto:Method: PUT 2023-11-02T18:26:57.271 DEBUG:boto:Path: /ibtljn-57/d 2023-11-02T18:26:57.272 DEBUG:boto:Data: 2023-11-02T18:26:57.272 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:26:57.272 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.273 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.273 DEBUG:boto:Params: {} 2023-11-02T18:26:57.274 DEBUG:boto:Token: None 2023-11-02T18:26:57.274 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/d 2023-11-02T18:26:57.275 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8RXsQstSugV7p601+R82gpceWBg= 2023-11-02T18:26:57.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8RXsQstSugV7p601+R82gpceWBg='} 2023-11-02T18:26:57.288 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddc8e3b9689d341a-006543e9f1-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'ibtljn-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:26:57.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'ibtljn-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:26:57.289 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket ibtljn-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:26:57.334 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:26:57.335 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:26:57.415 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:26:57.415 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-57 2023-11-02T18:26:57.425 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 4 2023-11-02T18:26:57.503 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:26:57.504 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:26:57.504 DEBUG:boto:Method: HEAD 2023-11-02T18:26:57.505 DEBUG:boto:Path: /ibtljn-57/ 2023-11-02T18:26:57.505 DEBUG:boto:Data: 2023-11-02T18:26:57.506 DEBUG:boto:Headers: {} 2023-11-02T18:26:57.506 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.507 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.507 DEBUG:boto:Params: {} 2023-11-02T18:26:57.508 DEBUG:boto:Token: None 2023-11-02T18:26:57.508 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/ 2023-11-02T18:26:57.508 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs= 2023-11-02T18:26:57.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs=', 'Content-Length': '0'} 2023-11-02T18:26:57.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000928edd34f8807231-006543e9f1-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.513 DEBUG:boto:path=/ibtljn-57/e 2023-11-02T18:26:57.513 DEBUG:boto:auth_path=/ibtljn-57/e 2023-11-02T18:26:57.514 DEBUG:boto:Method: PUT 2023-11-02T18:26:57.514 DEBUG:boto:Path: /ibtljn-57/e 2023-11-02T18:26:57.515 DEBUG:boto:Data: 2023-11-02T18:26:57.515 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:26:57.515 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.516 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.516 DEBUG:boto:Params: {} 2023-11-02T18:26:57.517 DEBUG:boto:Token: None 2023-11-02T18:26:57.517 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/e 2023-11-02T18:26:57.518 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8Hi/+GdPcAmPHX06k4QrzoMv6WM= 2023-11-02T18:26:57.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8Hi/+GdPcAmPHX06k4QrzoMv6WM='} 2023-11-02T18:26:57.563 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9160f90aa5b327d-006543e9f1-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.564 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:26:57.564 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:26:57.564 DEBUG:boto:Method: HEAD 2023-11-02T18:26:57.565 DEBUG:boto:Path: /ibtljn-57/ 2023-11-02T18:26:57.565 DEBUG:boto:Data: 2023-11-02T18:26:57.566 DEBUG:boto:Headers: {} 2023-11-02T18:26:57.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.566 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.567 DEBUG:boto:Params: {} 2023-11-02T18:26:57.567 DEBUG:boto:Token: None 2023-11-02T18:26:57.568 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/ 2023-11-02T18:26:57.568 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs= 2023-11-02T18:26:57.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs=', 'Content-Length': '0'} 2023-11-02T18:26:57.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7cf7c826eb422aa-006543e9f1-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.574 DEBUG:boto:path=/ibtljn-57/f 2023-11-02T18:26:57.574 DEBUG:boto:auth_path=/ibtljn-57/f 2023-11-02T18:26:57.575 DEBUG:boto:Method: PUT 2023-11-02T18:26:57.575 DEBUG:boto:Path: /ibtljn-57/f 2023-11-02T18:26:57.576 DEBUG:boto:Data: 2023-11-02T18:26:57.576 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:26:57.576 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.577 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.577 DEBUG:boto:Params: {} 2023-11-02T18:26:57.578 DEBUG:boto:Token: None 2023-11-02T18:26:57.578 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/f 2023-11-02T18:26:57.579 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qpSCuYjsmQaDyUn6Yh5Ng0WWAY0= 2023-11-02T18:26:57.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qpSCuYjsmQaDyUn6Yh5Ng0WWAY0='} 2023-11-02T18:26:57.588 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b23f4925e8c5965-006543e9f1-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.589 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:26:57.589 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:26:57.590 DEBUG:boto:Method: HEAD 2023-11-02T18:26:57.590 DEBUG:boto:Path: /ibtljn-57/ 2023-11-02T18:26:57.591 DEBUG:boto:Data: 2023-11-02T18:26:57.591 DEBUG:boto:Headers: {} 2023-11-02T18:26:57.591 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.592 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.592 DEBUG:boto:Params: {} 2023-11-02T18:26:57.593 DEBUG:boto:Token: None 2023-11-02T18:26:57.593 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/ 2023-11-02T18:26:57.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs= 2023-11-02T18:26:57.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs=', 'Content-Length': '0'} 2023-11-02T18:26:57.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ceeaabcf65c43d36-006543e9f1-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.597 DEBUG:boto:path=/ibtljn-57/g 2023-11-02T18:26:57.598 DEBUG:boto:auth_path=/ibtljn-57/g 2023-11-02T18:26:57.598 DEBUG:boto:Method: PUT 2023-11-02T18:26:57.599 DEBUG:boto:Path: /ibtljn-57/g 2023-11-02T18:26:57.599 DEBUG:boto:Data: 2023-11-02T18:26:57.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:26:57.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.600 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.601 DEBUG:boto:Params: {} 2023-11-02T18:26:57.601 DEBUG:boto:Token: None 2023-11-02T18:26:57.602 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/g 2023-11-02T18:26:57.602 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:44g1H2chdajUX3AU0JLZFcFgjBE= 2023-11-02T18:26:57.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:44g1H2chdajUX3AU0JLZFcFgjBE='} 2023-11-02T18:26:57.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ea6c91a75207b50-006543e9f1-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.651 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:26:57.652 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:26:57.652 DEBUG:boto:Method: HEAD 2023-11-02T18:26:57.653 DEBUG:boto:Path: /ibtljn-57/ 2023-11-02T18:26:57.653 DEBUG:boto:Data: 2023-11-02T18:26:57.653 DEBUG:boto:Headers: {} 2023-11-02T18:26:57.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.654 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.655 DEBUG:boto:Params: {} 2023-11-02T18:26:57.655 DEBUG:boto:Token: None 2023-11-02T18:26:57.656 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/ 2023-11-02T18:26:57.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs= 2023-11-02T18:26:57.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPMGPWYbe+ECv/UX2UxwTDlwQMs=', 'Content-Length': '0'} 2023-11-02T18:26:57.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084235db6cb2fcd55-006543e9f1-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.660 DEBUG:boto:path=/ibtljn-57/h 2023-11-02T18:26:57.661 DEBUG:boto:auth_path=/ibtljn-57/h 2023-11-02T18:26:57.661 DEBUG:boto:Method: PUT 2023-11-02T18:26:57.661 DEBUG:boto:Path: /ibtljn-57/h 2023-11-02T18:26:57.662 DEBUG:boto:Data: 2023-11-02T18:26:57.662 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:26:57.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:26:57.663 DEBUG:boto:Port: 8000 2023-11-02T18:26:57.663 DEBUG:boto:Params: {} 2023-11-02T18:26:57.664 DEBUG:boto:Token: None 2023-11-02T18:26:57.664 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:26:57 GMT /ibtljn-57/h 2023-11-02T18:26:57.665 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1tazQfYll/KW8F7TUGGXvsEnfFo= 2023-11-02T18:26:57.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:26:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1tazQfYll/KW8F7TUGGXvsEnfFo='} 2023-11-02T18:26:57.673 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c71b8c77a6db1dc3-006543e9f1-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:26:57.674 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-11-02T18:26:57.674 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-11-02T18:26:57.677 INFO:tasks.rgw.c1.client.1:Started 2023-11-02T18:26:57.678 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:57.695 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T18:26:57.695 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T18:26:57.695 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-11-02T18:26:57.696 INFO:teuthology.orchestra.run.smithi017.stderr:curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8001: Connection refused 2023-11-02T18:26:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-11-02T18:26:58.697 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T18:26:58.715 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T18:26:58.715 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T18:26:58.716 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-11-02T18:26:58.717 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2023-11-02T18:26:58.718 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-57 2023-11-02T18:26:58.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:26:58.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:26:58.719 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:26:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:26:58.814 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:26:58.882 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:26:58.878+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:26:58.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-11-02T18:27:03.886 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:27:03.878+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:27:08.884 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:27:08.878+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:27:13.885 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:27:13.882+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:27:18.886 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:27:18.882+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:27:23.887 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:27:23.882+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:27:28.888 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:27:28.882+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:27:33.889 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:27:33.886+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:27:38.890 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:27:38.886+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:27:43.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:27:43.886+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:27:48.893 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:27:48.886+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:27:53.893 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:27:53.890+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:27:58.894 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:27:58.890+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:28:03.895 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:28:03.890+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:28:08.896 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:28:08.890+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:28:13.897 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:28:13.893+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:28:18.898 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:28:18.893+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:28:23.899 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:28:23.893+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:28:28.900 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:28:28.893+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:28:33.901 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:28:33.897+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:28:38.902 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:28:38.897+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:28:43.903 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:28:43.897+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:28:48.905 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:28:48.897+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:28:53.906 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:28:53.901+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:28:58.907 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:28:58.901+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:29:03.908 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:03.901+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:29:08.909 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:08.905+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:29:13.910 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:13.905+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:29:18.911 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:18.905+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:29:23.913 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:23.908+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:29:28.914 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:28.908+0000 7fbdc378af40 1 waiting to reach incremental sync.. 2023-11-02T18:29:33.916 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:33.908+0000 7fbdc378af40 1 bucket sync caught up with source: 2023-11-02T18:29:33.917 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000003.26.6, 00000000002.33.6, 00000000002.20.6] 2023-11-02T18:29:33.917 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000003.26.6, 00000000002.33.6, 00000000002.20.6] 2023-11-02T18:29:33.917 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:33.908+0000 7fbdc378af40 0 bucket checkpoint complete 2023-11-02T18:29:33.950 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-57 2023-11-02T18:29:33.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:29:33.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:29:33.952 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:29:34.006 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:29:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:29:34.078 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:34.072+0000 7faadb47af40 1 waiting for incremental sync to catch up: 2023-11-02T18:29:34.078 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.285.6, , , 00000000001.102.6, 00000000001.137.6, , , 00000000002.60.6, , 00000000001.59.6] 2023-11-02T18:29:34.079 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.285.6, , , 00000000001.102.6, 00000000001.137.6, , , 00000000002.60.6, , 00000000002.71.6] 2023-11-02T18:29:39.080 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:39.072+0000 7faadb47af40 1 bucket sync caught up with source: 2023-11-02T18:29:39.080 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.285.6, , , 00000000001.102.6, 00000000001.137.6, , , 00000000002.60.6, , 00000000002.71.6] 2023-11-02T18:29:39.081 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.285.6, , , 00000000001.102.6, 00000000001.137.6, , , 00000000002.60.6, , 00000000002.71.6] 2023-11-02T18:29:39.081 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:39.072+0000 7faadb47af40 0 bucket checkpoint complete 2023-11-02T18:29:39.111 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-57 zones={a2, a1} 2023-11-02T18:29:39.111 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.112 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.112 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.113 DEBUG:boto:Path: /ibtljn-57/ 2023-11-02T18:29:39.113 DEBUG:boto:Data: 2023-11-02T18:29:39.114 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.114 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.115 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.115 DEBUG:boto:Params: {} 2023-11-02T18:29:39.116 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:29:39.116 DEBUG:boto:Token: None 2023-11-02T18:29:39.117 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.117 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ybWDGg0E6yKkvikHFVsvicHCfTU= 2023-11-02T18:29:39.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ybWDGg0E6yKkvikHFVsvicHCfTU=', 'Content-Length': '0'} 2023-11-02T18:29:39.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3c53bb60494bde8-006543ea93-1ac5-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.128 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.129 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.129 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.130 DEBUG:boto:Path: /ibtljn-57/ 2023-11-02T18:29:39.130 DEBUG:boto:Data: 2023-11-02T18:29:39.131 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.131 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.132 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.132 DEBUG:boto:Params: {} 2023-11-02T18:29:39.133 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:29:39.133 DEBUG:boto:Token: None 2023-11-02T18:29:39.133 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.134 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ybWDGg0E6yKkvikHFVsvicHCfTU= 2023-11-02T18:29:39.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ybWDGg0E6yKkvikHFVsvicHCfTU=', 'Content-Length': '0'} 2023-11-02T18:29:39.139 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001eb820dc1a0ab666-006543ea93-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.139 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:29:39.140 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.140 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.141 DEBUG:boto:path=/ibtljn-57/?versions&max-keys=999 2023-11-02T18:29:39.141 DEBUG:boto:auth_path=/ibtljn-57/?versions&max-keys=999 2023-11-02T18:29:39.142 DEBUG:boto:Method: GET 2023-11-02T18:29:39.142 DEBUG:boto:Path: /ibtljn-57/?versions&max-keys=999 2023-11-02T18:29:39.143 DEBUG:boto:Data: 2023-11-02T18:29:39.143 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.144 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.144 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.145 DEBUG:boto:Params: {} 2023-11-02T18:29:39.145 DEBUG:boto:Token: None 2023-11-02T18:29:39.146 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/?versions 2023-11-02T18:29:39.146 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LKidbapioOnZL/c8pAL1AKtWG2A= 2023-11-02T18:29:39.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LKidbapioOnZL/c8pAL1AKtWG2A=', 'Content-Length': '0'} 2023-11-02T18:29:39.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f87f76b20e99367b-006543ea93-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.153 DEBUG:boto:b'ibtljn-57999falseanulltrue2023-11-02T18:26:57.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:26:57.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:26:57.216Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:26:57.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:26:57.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:26:57.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:26:57.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:26:57.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.155 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:29:39.155 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:29:39.155 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:29:39.156 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:29:39.157 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-11-02T18:29:39.158 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-11-02T18:29:39.158 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-11-02T18:29:39.158 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-11-02T18:29:39.159 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:29:39.159 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.160 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.160 DEBUG:boto:path=/ibtljn-57/?versions&max-keys=999 2023-11-02T18:29:39.161 DEBUG:boto:auth_path=/ibtljn-57/?versions&max-keys=999 2023-11-02T18:29:39.161 DEBUG:boto:Method: GET 2023-11-02T18:29:39.161 DEBUG:boto:Path: /ibtljn-57/?versions&max-keys=999 2023-11-02T18:29:39.162 DEBUG:boto:Data: 2023-11-02T18:29:39.162 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.163 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.163 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.164 DEBUG:boto:Params: {} 2023-11-02T18:29:39.164 DEBUG:boto:Token: None 2023-11-02T18:29:39.165 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/?versions 2023-11-02T18:29:39.165 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LKidbapioOnZL/c8pAL1AKtWG2A= 2023-11-02T18:29:39.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LKidbapioOnZL/c8pAL1AKtWG2A=', 'Content-Length': '0'} 2023-11-02T18:29:39.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed93710c9f7bd8f0-006543ea93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.169 DEBUG:boto:b'ibtljn-57999falseanulltrue2023-11-02T18:26:57.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:26:57.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:26:57.216Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:26:57.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:26:57.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:26:57.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:26:57.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:26:57.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.170 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-11-02T18:29:39.171 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-11-02T18:29:39.171 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-11-02T18:29:39.172 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-11-02T18:29:39.172 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-11-02T18:29:39.172 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-11-02T18:29:39.173 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-11-02T18:29:39.173 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-11-02T18:29:39.173 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.174 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.174 DEBUG:boto:path=/ibtljn-57/?versions&max-keys=999 2023-11-02T18:29:39.175 DEBUG:boto:auth_path=/ibtljn-57/?versions&max-keys=999 2023-11-02T18:29:39.175 DEBUG:boto:Method: GET 2023-11-02T18:29:39.175 DEBUG:boto:Path: /ibtljn-57/?versions&max-keys=999 2023-11-02T18:29:39.176 DEBUG:boto:Data: 2023-11-02T18:29:39.176 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.177 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.177 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.178 DEBUG:boto:Params: {} 2023-11-02T18:29:39.178 DEBUG:boto:Token: None 2023-11-02T18:29:39.179 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/?versions 2023-11-02T18:29:39.179 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LKidbapioOnZL/c8pAL1AKtWG2A= 2023-11-02T18:29:39.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LKidbapioOnZL/c8pAL1AKtWG2A=', 'Content-Length': '0'} 2023-11-02T18:29:39.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa0decb2682995b9-006543ea93-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.186 DEBUG:boto:b'ibtljn-57999falseanulltrue2023-11-02T18:26:57.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:26:57.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:26:57.216Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:26:57.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:26:57.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:26:57.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:26:57.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:26:57.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.187 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.188 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.188 DEBUG:boto:path=/ibtljn-57/?versions&max-keys=999 2023-11-02T18:29:39.189 DEBUG:boto:auth_path=/ibtljn-57/?versions&max-keys=999 2023-11-02T18:29:39.189 DEBUG:boto:Method: GET 2023-11-02T18:29:39.190 DEBUG:boto:Path: /ibtljn-57/?versions&max-keys=999 2023-11-02T18:29:39.190 DEBUG:boto:Data: 2023-11-02T18:29:39.191 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.191 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.191 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.192 DEBUG:boto:Params: {} 2023-11-02T18:29:39.192 DEBUG:boto:Token: None 2023-11-02T18:29:39.193 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/?versions 2023-11-02T18:29:39.193 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LKidbapioOnZL/c8pAL1AKtWG2A= 2023-11-02T18:29:39.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LKidbapioOnZL/c8pAL1AKtWG2A=', 'Content-Length': '0'} 2023-11-02T18:29:39.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa2d537f82d8341b-006543ea93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.198 DEBUG:boto:b'ibtljn-57999falseanulltrue2023-11-02T18:26:57.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:26:57.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:26:57.216Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:26:57.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:26:57.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:26:57.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:26:57.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:26:57.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:29:39.199 DEBUG:boto:path=/ibtljn-57/a 2023-11-02T18:29:39.200 DEBUG:boto:auth_path=/ibtljn-57/a 2023-11-02T18:29:39.200 DEBUG:boto:path=/ibtljn-57/a?versionId=null 2023-11-02T18:29:39.201 DEBUG:boto:auth_path=/ibtljn-57/a?versionId=null 2023-11-02T18:29:39.201 DEBUG:boto:Method: GET 2023-11-02T18:29:39.202 DEBUG:boto:Path: /ibtljn-57/a?versionId=null 2023-11-02T18:29:39.202 DEBUG:boto:Data: 2023-11-02T18:29:39.203 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.203 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.203 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.204 DEBUG:boto:Params: {} 2023-11-02T18:29:39.204 DEBUG:boto:Token: None 2023-11-02T18:29:39.205 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/a?versionId=null 2023-11-02T18:29:39.205 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PN+ig6h//0CL4XxFlOIv33lthTE= 2023-11-02T18:29:39.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PN+ig6h//0CL4XxFlOIv33lthTE=', 'Content-Length': '0'} 2023-11-02T18:29:39.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082d8c2f9be705a60-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.254 DEBUG:boto:path=/ibtljn-57/a 2023-11-02T18:29:39.255 DEBUG:boto:auth_path=/ibtljn-57/a 2023-11-02T18:29:39.255 DEBUG:boto:path=/ibtljn-57/a?versionId=null 2023-11-02T18:29:39.256 DEBUG:boto:auth_path=/ibtljn-57/a?versionId=null 2023-11-02T18:29:39.256 DEBUG:boto:Method: GET 2023-11-02T18:29:39.257 DEBUG:boto:Path: /ibtljn-57/a?versionId=null 2023-11-02T18:29:39.257 DEBUG:boto:Data: 2023-11-02T18:29:39.258 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.258 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.259 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.259 DEBUG:boto:Params: {} 2023-11-02T18:29:39.259 DEBUG:boto:Token: None 2023-11-02T18:29:39.260 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/a?versionId=null 2023-11-02T18:29:39.260 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PN+ig6h//0CL4XxFlOIv33lthTE= 2023-11-02T18:29:39.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PN+ig6h//0CL4XxFlOIv33lthTE=', 'Content-Length': '0'} 2023-11-02T18:29:39.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000777c920648bd4c2f-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.265 DEBUG:boto:path=/ibtljn-57/a 2023-11-02T18:29:39.265 DEBUG:boto:auth_path=/ibtljn-57/a 2023-11-02T18:29:39.266 DEBUG:boto:path=/ibtljn-57/a?versionId=null 2023-11-02T18:29:39.266 DEBUG:boto:auth_path=/ibtljn-57/a?versionId=null 2023-11-02T18:29:39.266 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.267 DEBUG:boto:Path: /ibtljn-57/a?versionId=null 2023-11-02T18:29:39.267 DEBUG:boto:Data: 2023-11-02T18:29:39.268 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.268 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.268 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.269 DEBUG:boto:Params: {} 2023-11-02T18:29:39.269 DEBUG:boto:Token: None 2023-11-02T18:29:39.270 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/a?versionId=null 2023-11-02T18:29:39.270 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xk+scUoo7qY19fFeyBDvZtDfJ/k= 2023-11-02T18:29:39.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xk+scUoo7qY19fFeyBDvZtDfJ/k=', 'Content-Length': '0'} 2023-11-02T18:29:39.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df95a80f3645f8cf-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.275 DEBUG:boto:path=/ibtljn-57/a 2023-11-02T18:29:39.275 DEBUG:boto:auth_path=/ibtljn-57/a 2023-11-02T18:29:39.275 DEBUG:boto:path=/ibtljn-57/a?versionId=null 2023-11-02T18:29:39.276 DEBUG:boto:auth_path=/ibtljn-57/a?versionId=null 2023-11-02T18:29:39.276 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.277 DEBUG:boto:Path: /ibtljn-57/a?versionId=null 2023-11-02T18:29:39.277 DEBUG:boto:Data: 2023-11-02T18:29:39.278 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.278 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.278 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.279 DEBUG:boto:Params: {} 2023-11-02T18:29:39.279 DEBUG:boto:Token: None 2023-11-02T18:29:39.280 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/a?versionId=null 2023-11-02T18:29:39.280 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xk+scUoo7qY19fFeyBDvZtDfJ/k= 2023-11-02T18:29:39.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xk+scUoo7qY19fFeyBDvZtDfJ/k=', 'Content-Length': '0'} 2023-11-02T18:29:39.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027d950a63916ee27-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:29:39.285 DEBUG:boto:path=/ibtljn-57/a 2023-11-02T18:29:39.285 DEBUG:boto:auth_path=/ibtljn-57/a 2023-11-02T18:29:39.286 DEBUG:boto:path=/ibtljn-57/a?versionId=null 2023-11-02T18:29:39.286 DEBUG:boto:auth_path=/ibtljn-57/a?versionId=null 2023-11-02T18:29:39.287 DEBUG:boto:Method: GET 2023-11-02T18:29:39.287 DEBUG:boto:Path: /ibtljn-57/a?versionId=null 2023-11-02T18:29:39.287 DEBUG:boto:Data: 2023-11-02T18:29:39.288 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.288 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.289 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.289 DEBUG:boto:Params: {} 2023-11-02T18:29:39.290 DEBUG:boto:Token: None 2023-11-02T18:29:39.290 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/a?versionId=null 2023-11-02T18:29:39.291 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PN+ig6h//0CL4XxFlOIv33lthTE= 2023-11-02T18:29:39.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PN+ig6h//0CL4XxFlOIv33lthTE=', 'Content-Length': '0'} 2023-11-02T18:29:39.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f73e693867ed51b-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.295 DEBUG:boto:path=/ibtljn-57/a 2023-11-02T18:29:39.296 DEBUG:boto:auth_path=/ibtljn-57/a 2023-11-02T18:29:39.296 DEBUG:boto:path=/ibtljn-57/a?versionId=null 2023-11-02T18:29:39.296 DEBUG:boto:auth_path=/ibtljn-57/a?versionId=null 2023-11-02T18:29:39.297 DEBUG:boto:Method: GET 2023-11-02T18:29:39.297 DEBUG:boto:Path: /ibtljn-57/a?versionId=null 2023-11-02T18:29:39.298 DEBUG:boto:Data: 2023-11-02T18:29:39.298 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.299 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.299 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.300 DEBUG:boto:Params: {} 2023-11-02T18:29:39.300 DEBUG:boto:Token: None 2023-11-02T18:29:39.301 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/a?versionId=null 2023-11-02T18:29:39.301 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PN+ig6h//0CL4XxFlOIv33lthTE= 2023-11-02T18:29:39.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PN+ig6h//0CL4XxFlOIv33lthTE=', 'Content-Length': '0'} 2023-11-02T18:29:39.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003baeb49226e7ea5d-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.350 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.351 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.352 DEBUG:boto:path=/ibtljn-57/?prefix=a 2023-11-02T18:29:39.352 DEBUG:boto:auth_path=/ibtljn-57/?prefix=a 2023-11-02T18:29:39.352 DEBUG:boto:Method: GET 2023-11-02T18:29:39.353 DEBUG:boto:Path: /ibtljn-57/?prefix=a 2023-11-02T18:29:39.353 DEBUG:boto:Data: 2023-11-02T18:29:39.354 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.354 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.355 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.355 DEBUG:boto:Params: {} 2023-11-02T18:29:39.356 DEBUG:boto:Token: None 2023-11-02T18:29:39.356 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.357 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo= 2023-11-02T18:29:39.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo=', 'Content-Length': '0'} 2023-11-02T18:29:39.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056b755d861edf9a0-006543ea93-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.363 DEBUG:boto:b'ibtljn-57a1000falsea2023-11-02T18:26:57.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.364 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.365 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.365 DEBUG:boto:path=/ibtljn-57/?prefix=a 2023-11-02T18:29:39.365 DEBUG:boto:auth_path=/ibtljn-57/?prefix=a 2023-11-02T18:29:39.366 DEBUG:boto:Method: GET 2023-11-02T18:29:39.366 DEBUG:boto:Path: /ibtljn-57/?prefix=a 2023-11-02T18:29:39.367 DEBUG:boto:Data: 2023-11-02T18:29:39.367 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.369 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.369 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.370 DEBUG:boto:Params: {} 2023-11-02T18:29:39.370 DEBUG:boto:Token: None 2023-11-02T18:29:39.371 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.371 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo= 2023-11-02T18:29:39.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo=', 'Content-Length': '0'} 2023-11-02T18:29:39.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a84a70dcbd5335e-006543ea93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.376 DEBUG:boto:b'ibtljn-57a1000falsea2023-11-02T18:26:57.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.377 DEBUG:boto:path=/ibtljn-57/a 2023-11-02T18:29:39.377 DEBUG:boto:auth_path=/ibtljn-57/a 2023-11-02T18:29:39.378 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.378 DEBUG:boto:Path: /ibtljn-57/a 2023-11-02T18:29:39.378 DEBUG:boto:Data: 2023-11-02T18:29:39.379 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.379 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.380 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.380 DEBUG:boto:Params: {} 2023-11-02T18:29:39.381 DEBUG:boto:Token: None 2023-11-02T18:29:39.381 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/a 2023-11-02T18:29:39.382 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GwvThO9Po/k5+hulmO9BVpPM6JE= 2023-11-02T18:29:39.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GwvThO9Po/k5+hulmO9BVpPM6JE=', 'Content-Length': '0'} 2023-11-02T18:29:39.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac9785696d144b38-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.386 DEBUG:boto:path=/ibtljn-57/a 2023-11-02T18:29:39.386 DEBUG:boto:auth_path=/ibtljn-57/a 2023-11-02T18:29:39.386 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.387 DEBUG:boto:Path: /ibtljn-57/a 2023-11-02T18:29:39.387 DEBUG:boto:Data: 2023-11-02T18:29:39.388 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.388 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.389 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.389 DEBUG:boto:Params: {} 2023-11-02T18:29:39.390 DEBUG:boto:Token: None 2023-11-02T18:29:39.390 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/a 2023-11-02T18:29:39.390 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GwvThO9Po/k5+hulmO9BVpPM6JE= 2023-11-02T18:29:39.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GwvThO9Po/k5+hulmO9BVpPM6JE=', 'Content-Length': '0'} 2023-11-02T18:29:39.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020d6c5abe53c8b0d-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-11-02T18:29:39.395 DEBUG:boto:path=/ibtljn-57/a 2023-11-02T18:29:39.395 DEBUG:boto:auth_path=/ibtljn-57/a 2023-11-02T18:29:39.396 DEBUG:boto:Method: GET 2023-11-02T18:29:39.396 DEBUG:boto:Path: /ibtljn-57/a 2023-11-02T18:29:39.397 DEBUG:boto:Data: 2023-11-02T18:29:39.397 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.397 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.398 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.398 DEBUG:boto:Params: {} 2023-11-02T18:29:39.399 DEBUG:boto:Token: None 2023-11-02T18:29:39.399 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/a 2023-11-02T18:29:39.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nObu0jGwDRz9UPlboyVq1zasOXw= 2023-11-02T18:29:39.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nObu0jGwDRz9UPlboyVq1zasOXw=', 'Content-Length': '0'} 2023-11-02T18:29:39.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026a7eb1f19db7ee8-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.404 DEBUG:boto:path=/ibtljn-57/a 2023-11-02T18:29:39.404 DEBUG:boto:auth_path=/ibtljn-57/a 2023-11-02T18:29:39.405 DEBUG:boto:Method: GET 2023-11-02T18:29:39.405 DEBUG:boto:Path: /ibtljn-57/a 2023-11-02T18:29:39.406 DEBUG:boto:Data: 2023-11-02T18:29:39.406 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.407 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.407 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.407 DEBUG:boto:Params: {} 2023-11-02T18:29:39.408 DEBUG:boto:Token: None 2023-11-02T18:29:39.408 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/a 2023-11-02T18:29:39.409 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nObu0jGwDRz9UPlboyVq1zasOXw= 2023-11-02T18:29:39.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nObu0jGwDRz9UPlboyVq1zasOXw=', 'Content-Length': '0'} 2023-11-02T18:29:39.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020e309992f8ab641-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.413 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.414 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.414 DEBUG:boto:path=/ibtljn-57/?prefix=a 2023-11-02T18:29:39.415 DEBUG:boto:auth_path=/ibtljn-57/?prefix=a 2023-11-02T18:29:39.415 DEBUG:boto:Method: GET 2023-11-02T18:29:39.415 DEBUG:boto:Path: /ibtljn-57/?prefix=a 2023-11-02T18:29:39.416 DEBUG:boto:Data: 2023-11-02T18:29:39.416 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.417 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.417 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.418 DEBUG:boto:Params: {} 2023-11-02T18:29:39.418 DEBUG:boto:Token: None 2023-11-02T18:29:39.419 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.419 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo= 2023-11-02T18:29:39.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo=', 'Content-Length': '0'} 2023-11-02T18:29:39.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001240d3f173322cae-006543ea93-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.466 DEBUG:boto:b'ibtljn-57a1000falsea2023-11-02T18:26:57.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.467 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.468 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.468 DEBUG:boto:path=/ibtljn-57/?prefix=a 2023-11-02T18:29:39.469 DEBUG:boto:auth_path=/ibtljn-57/?prefix=a 2023-11-02T18:29:39.469 DEBUG:boto:Method: GET 2023-11-02T18:29:39.470 DEBUG:boto:Path: /ibtljn-57/?prefix=a 2023-11-02T18:29:39.470 DEBUG:boto:Data: 2023-11-02T18:29:39.470 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.471 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.471 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.472 DEBUG:boto:Params: {} 2023-11-02T18:29:39.472 DEBUG:boto:Token: None 2023-11-02T18:29:39.473 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.473 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo= 2023-11-02T18:29:39.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo=', 'Content-Length': '0'} 2023-11-02T18:29:39.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b06b56396c291b45-006543ea93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.478 DEBUG:boto:b'ibtljn-57a1000falsea2023-11-02T18:26:57.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:29:39.479 DEBUG:boto:path=/ibtljn-57/b 2023-11-02T18:29:39.480 DEBUG:boto:auth_path=/ibtljn-57/b 2023-11-02T18:29:39.480 DEBUG:boto:path=/ibtljn-57/b?versionId=null 2023-11-02T18:29:39.481 DEBUG:boto:auth_path=/ibtljn-57/b?versionId=null 2023-11-02T18:29:39.481 DEBUG:boto:Method: GET 2023-11-02T18:29:39.481 DEBUG:boto:Path: /ibtljn-57/b?versionId=null 2023-11-02T18:29:39.482 DEBUG:boto:Data: 2023-11-02T18:29:39.482 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.483 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.483 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.484 DEBUG:boto:Params: {} 2023-11-02T18:29:39.484 DEBUG:boto:Token: None 2023-11-02T18:29:39.485 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/b?versionId=null 2023-11-02T18:29:39.485 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Php+DuaM04TlM0sQMY11vPQiGmg= 2023-11-02T18:29:39.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Php+DuaM04TlM0sQMY11vPQiGmg=', 'Content-Length': '0'} 2023-11-02T18:29:39.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002d51e86e039eaee-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.490 DEBUG:boto:path=/ibtljn-57/b 2023-11-02T18:29:39.491 DEBUG:boto:auth_path=/ibtljn-57/b 2023-11-02T18:29:39.491 DEBUG:boto:path=/ibtljn-57/b?versionId=null 2023-11-02T18:29:39.491 DEBUG:boto:auth_path=/ibtljn-57/b?versionId=null 2023-11-02T18:29:39.492 DEBUG:boto:Method: GET 2023-11-02T18:29:39.492 DEBUG:boto:Path: /ibtljn-57/b?versionId=null 2023-11-02T18:29:39.494 DEBUG:boto:Data: 2023-11-02T18:29:39.494 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.494 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.495 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.495 DEBUG:boto:Params: {} 2023-11-02T18:29:39.496 DEBUG:boto:Token: None 2023-11-02T18:29:39.496 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/b?versionId=null 2023-11-02T18:29:39.497 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Php+DuaM04TlM0sQMY11vPQiGmg= 2023-11-02T18:29:39.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Php+DuaM04TlM0sQMY11vPQiGmg=', 'Content-Length': '0'} 2023-11-02T18:29:39.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2971178ed9fc2f2-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.502 DEBUG:boto:path=/ibtljn-57/b 2023-11-02T18:29:39.502 DEBUG:boto:auth_path=/ibtljn-57/b 2023-11-02T18:29:39.503 DEBUG:boto:path=/ibtljn-57/b?versionId=null 2023-11-02T18:29:39.503 DEBUG:boto:auth_path=/ibtljn-57/b?versionId=null 2023-11-02T18:29:39.504 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.504 DEBUG:boto:Path: /ibtljn-57/b?versionId=null 2023-11-02T18:29:39.504 DEBUG:boto:Data: 2023-11-02T18:29:39.505 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.505 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.506 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.506 DEBUG:boto:Params: {} 2023-11-02T18:29:39.506 DEBUG:boto:Token: None 2023-11-02T18:29:39.507 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/b?versionId=null 2023-11-02T18:29:39.507 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b2UYOC77cqh7oJGo4DccU3NzM/I= 2023-11-02T18:29:39.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b2UYOC77cqh7oJGo4DccU3NzM/I=', 'Content-Length': '0'} 2023-11-02T18:29:39.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db185918caa4fd73-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.512 DEBUG:boto:path=/ibtljn-57/b 2023-11-02T18:29:39.512 DEBUG:boto:auth_path=/ibtljn-57/b 2023-11-02T18:29:39.513 DEBUG:boto:path=/ibtljn-57/b?versionId=null 2023-11-02T18:29:39.513 DEBUG:boto:auth_path=/ibtljn-57/b?versionId=null 2023-11-02T18:29:39.513 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.514 DEBUG:boto:Path: /ibtljn-57/b?versionId=null 2023-11-02T18:29:39.514 DEBUG:boto:Data: 2023-11-02T18:29:39.515 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.515 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.515 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.516 DEBUG:boto:Params: {} 2023-11-02T18:29:39.516 DEBUG:boto:Token: None 2023-11-02T18:29:39.517 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/b?versionId=null 2023-11-02T18:29:39.517 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b2UYOC77cqh7oJGo4DccU3NzM/I= 2023-11-02T18:29:39.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b2UYOC77cqh7oJGo4DccU3NzM/I=', 'Content-Length': '0'} 2023-11-02T18:29:39.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003abef34fb1b504ae-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:29:39.522 DEBUG:boto:path=/ibtljn-57/b 2023-11-02T18:29:39.522 DEBUG:boto:auth_path=/ibtljn-57/b 2023-11-02T18:29:39.522 DEBUG:boto:path=/ibtljn-57/b?versionId=null 2023-11-02T18:29:39.523 DEBUG:boto:auth_path=/ibtljn-57/b?versionId=null 2023-11-02T18:29:39.523 DEBUG:boto:Method: GET 2023-11-02T18:29:39.524 DEBUG:boto:Path: /ibtljn-57/b?versionId=null 2023-11-02T18:29:39.524 DEBUG:boto:Data: 2023-11-02T18:29:39.525 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.525 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.525 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.526 DEBUG:boto:Params: {} 2023-11-02T18:29:39.526 DEBUG:boto:Token: None 2023-11-02T18:29:39.527 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/b?versionId=null 2023-11-02T18:29:39.527 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Php+DuaM04TlM0sQMY11vPQiGmg= 2023-11-02T18:29:39.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Php+DuaM04TlM0sQMY11vPQiGmg=', 'Content-Length': '0'} 2023-11-02T18:29:39.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000016a6e79df4c5eb8-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.532 DEBUG:boto:path=/ibtljn-57/b 2023-11-02T18:29:39.532 DEBUG:boto:auth_path=/ibtljn-57/b 2023-11-02T18:29:39.532 DEBUG:boto:path=/ibtljn-57/b?versionId=null 2023-11-02T18:29:39.533 DEBUG:boto:auth_path=/ibtljn-57/b?versionId=null 2023-11-02T18:29:39.533 DEBUG:boto:Method: GET 2023-11-02T18:29:39.534 DEBUG:boto:Path: /ibtljn-57/b?versionId=null 2023-11-02T18:29:39.534 DEBUG:boto:Data: 2023-11-02T18:29:39.535 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.535 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.536 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.536 DEBUG:boto:Params: {} 2023-11-02T18:29:39.537 DEBUG:boto:Token: None 2023-11-02T18:29:39.537 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/b?versionId=null 2023-11-02T18:29:39.538 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Php+DuaM04TlM0sQMY11vPQiGmg= 2023-11-02T18:29:39.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Php+DuaM04TlM0sQMY11vPQiGmg=', 'Content-Length': '0'} 2023-11-02T18:29:39.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064421d07cff8e9bf-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.542 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.542 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.543 DEBUG:boto:path=/ibtljn-57/?prefix=b 2023-11-02T18:29:39.543 DEBUG:boto:auth_path=/ibtljn-57/?prefix=b 2023-11-02T18:29:39.543 DEBUG:boto:Method: GET 2023-11-02T18:29:39.544 DEBUG:boto:Path: /ibtljn-57/?prefix=b 2023-11-02T18:29:39.544 DEBUG:boto:Data: 2023-11-02T18:29:39.545 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.545 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.545 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.546 DEBUG:boto:Params: {} 2023-11-02T18:29:39.546 DEBUG:boto:Token: None 2023-11-02T18:29:39.547 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.547 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo= 2023-11-02T18:29:39.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo=', 'Content-Length': '0'} 2023-11-02T18:29:39.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f9f7ed8e2057077-006543ea93-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.594 DEBUG:boto:b'ibtljn-57b1000falseb2023-11-02T18:26:57.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.595 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.596 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.596 DEBUG:boto:path=/ibtljn-57/?prefix=b 2023-11-02T18:29:39.597 DEBUG:boto:auth_path=/ibtljn-57/?prefix=b 2023-11-02T18:29:39.597 DEBUG:boto:Method: GET 2023-11-02T18:29:39.598 DEBUG:boto:Path: /ibtljn-57/?prefix=b 2023-11-02T18:29:39.598 DEBUG:boto:Data: 2023-11-02T18:29:39.598 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.599 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.599 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.600 DEBUG:boto:Params: {} 2023-11-02T18:29:39.600 DEBUG:boto:Token: None 2023-11-02T18:29:39.601 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo= 2023-11-02T18:29:39.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo=', 'Content-Length': '0'} 2023-11-02T18:29:39.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e0f866c8246b0ef-006543ea93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.605 DEBUG:boto:b'ibtljn-57b1000falseb2023-11-02T18:26:57.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.606 DEBUG:boto:path=/ibtljn-57/b 2023-11-02T18:29:39.607 DEBUG:boto:auth_path=/ibtljn-57/b 2023-11-02T18:29:39.607 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.607 DEBUG:boto:Path: /ibtljn-57/b 2023-11-02T18:29:39.608 DEBUG:boto:Data: 2023-11-02T18:29:39.608 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.609 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.609 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.609 DEBUG:boto:Params: {} 2023-11-02T18:29:39.610 DEBUG:boto:Token: None 2023-11-02T18:29:39.610 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/b 2023-11-02T18:29:39.611 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RngjnVK6u5+mk75l4ZglCw3EylE= 2023-11-02T18:29:39.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RngjnVK6u5+mk75l4ZglCw3EylE=', 'Content-Length': '0'} 2023-11-02T18:29:39.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba80d1d5d4d446a-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.615 DEBUG:boto:path=/ibtljn-57/b 2023-11-02T18:29:39.615 DEBUG:boto:auth_path=/ibtljn-57/b 2023-11-02T18:29:39.615 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.616 DEBUG:boto:Path: /ibtljn-57/b 2023-11-02T18:29:39.616 DEBUG:boto:Data: 2023-11-02T18:29:39.617 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.617 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.617 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.618 DEBUG:boto:Params: {} 2023-11-02T18:29:39.618 DEBUG:boto:Token: None 2023-11-02T18:29:39.619 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/b 2023-11-02T18:29:39.619 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RngjnVK6u5+mk75l4ZglCw3EylE= 2023-11-02T18:29:39.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RngjnVK6u5+mk75l4ZglCw3EylE=', 'Content-Length': '0'} 2023-11-02T18:29:39.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000c64bd63220b04d-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-11-02T18:29:39.623 DEBUG:boto:path=/ibtljn-57/b 2023-11-02T18:29:39.624 DEBUG:boto:auth_path=/ibtljn-57/b 2023-11-02T18:29:39.624 DEBUG:boto:Method: GET 2023-11-02T18:29:39.625 DEBUG:boto:Path: /ibtljn-57/b 2023-11-02T18:29:39.625 DEBUG:boto:Data: 2023-11-02T18:29:39.626 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.626 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.626 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.627 DEBUG:boto:Params: {} 2023-11-02T18:29:39.627 DEBUG:boto:Token: None 2023-11-02T18:29:39.628 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/b 2023-11-02T18:29:39.628 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FrNiAPIKM4lm47tfOjihiolSUmI= 2023-11-02T18:29:39.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FrNiAPIKM4lm47tfOjihiolSUmI=', 'Content-Length': '0'} 2023-11-02T18:29:39.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c9882c4cf12ee5-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.632 DEBUG:boto:path=/ibtljn-57/b 2023-11-02T18:29:39.632 DEBUG:boto:auth_path=/ibtljn-57/b 2023-11-02T18:29:39.633 DEBUG:boto:Method: GET 2023-11-02T18:29:39.633 DEBUG:boto:Path: /ibtljn-57/b 2023-11-02T18:29:39.634 DEBUG:boto:Data: 2023-11-02T18:29:39.634 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.635 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.635 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.636 DEBUG:boto:Params: {} 2023-11-02T18:29:39.636 DEBUG:boto:Token: None 2023-11-02T18:29:39.636 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/b 2023-11-02T18:29:39.637 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FrNiAPIKM4lm47tfOjihiolSUmI= 2023-11-02T18:29:39.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FrNiAPIKM4lm47tfOjihiolSUmI=', 'Content-Length': '0'} 2023-11-02T18:29:39.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db7fe35e64548674-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.641 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.641 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.642 DEBUG:boto:path=/ibtljn-57/?prefix=b 2023-11-02T18:29:39.642 DEBUG:boto:auth_path=/ibtljn-57/?prefix=b 2023-11-02T18:29:39.643 DEBUG:boto:Method: GET 2023-11-02T18:29:39.643 DEBUG:boto:Path: /ibtljn-57/?prefix=b 2023-11-02T18:29:39.643 DEBUG:boto:Data: 2023-11-02T18:29:39.644 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.644 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.645 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.645 DEBUG:boto:Params: {} 2023-11-02T18:29:39.646 DEBUG:boto:Token: None 2023-11-02T18:29:39.646 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.647 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo= 2023-11-02T18:29:39.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo=', 'Content-Length': '0'} 2023-11-02T18:29:39.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2b9eb204d136401-006543ea93-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.653 DEBUG:boto:b'ibtljn-57b1000falseb2023-11-02T18:26:57.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.654 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.655 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.655 DEBUG:boto:path=/ibtljn-57/?prefix=b 2023-11-02T18:29:39.656 DEBUG:boto:auth_path=/ibtljn-57/?prefix=b 2023-11-02T18:29:39.656 DEBUG:boto:Method: GET 2023-11-02T18:29:39.656 DEBUG:boto:Path: /ibtljn-57/?prefix=b 2023-11-02T18:29:39.657 DEBUG:boto:Data: 2023-11-02T18:29:39.657 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.658 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.658 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.659 DEBUG:boto:Params: {} 2023-11-02T18:29:39.659 DEBUG:boto:Token: None 2023-11-02T18:29:39.660 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.660 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo= 2023-11-02T18:29:39.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo=', 'Content-Length': '0'} 2023-11-02T18:29:39.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fde30b349264050-006543ea93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.664 DEBUG:boto:b'ibtljn-57b1000falseb2023-11-02T18:26:57.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:29:39.666 DEBUG:boto:path=/ibtljn-57/c 2023-11-02T18:29:39.666 DEBUG:boto:auth_path=/ibtljn-57/c 2023-11-02T18:29:39.666 DEBUG:boto:path=/ibtljn-57/c?versionId=null 2023-11-02T18:29:39.667 DEBUG:boto:auth_path=/ibtljn-57/c?versionId=null 2023-11-02T18:29:39.667 DEBUG:boto:Method: GET 2023-11-02T18:29:39.668 DEBUG:boto:Path: /ibtljn-57/c?versionId=null 2023-11-02T18:29:39.668 DEBUG:boto:Data: 2023-11-02T18:29:39.669 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.669 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.669 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.670 DEBUG:boto:Params: {} 2023-11-02T18:29:39.670 DEBUG:boto:Token: None 2023-11-02T18:29:39.671 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/c?versionId=null 2023-11-02T18:29:39.671 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:halJsGgRn7XDXQulaARK4dKVXrc= 2023-11-02T18:29:39.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:halJsGgRn7XDXQulaARK4dKVXrc=', 'Content-Length': '0'} 2023-11-02T18:29:39.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eba3deecf88ede8-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.718 DEBUG:boto:path=/ibtljn-57/c 2023-11-02T18:29:39.719 DEBUG:boto:auth_path=/ibtljn-57/c 2023-11-02T18:29:39.719 DEBUG:boto:path=/ibtljn-57/c?versionId=null 2023-11-02T18:29:39.720 DEBUG:boto:auth_path=/ibtljn-57/c?versionId=null 2023-11-02T18:29:39.720 DEBUG:boto:Method: GET 2023-11-02T18:29:39.721 DEBUG:boto:Path: /ibtljn-57/c?versionId=null 2023-11-02T18:29:39.721 DEBUG:boto:Data: 2023-11-02T18:29:39.722 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.722 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.722 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.723 DEBUG:boto:Params: {} 2023-11-02T18:29:39.723 DEBUG:boto:Token: None 2023-11-02T18:29:39.724 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/c?versionId=null 2023-11-02T18:29:39.724 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:halJsGgRn7XDXQulaARK4dKVXrc= 2023-11-02T18:29:39.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:halJsGgRn7XDXQulaARK4dKVXrc=', 'Content-Length': '0'} 2023-11-02T18:29:39.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a04ae392a95c39-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.729 DEBUG:boto:path=/ibtljn-57/c 2023-11-02T18:29:39.729 DEBUG:boto:auth_path=/ibtljn-57/c 2023-11-02T18:29:39.729 DEBUG:boto:path=/ibtljn-57/c?versionId=null 2023-11-02T18:29:39.730 DEBUG:boto:auth_path=/ibtljn-57/c?versionId=null 2023-11-02T18:29:39.730 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.731 DEBUG:boto:Path: /ibtljn-57/c?versionId=null 2023-11-02T18:29:39.731 DEBUG:boto:Data: 2023-11-02T18:29:39.732 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.732 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.732 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.733 DEBUG:boto:Params: {} 2023-11-02T18:29:39.733 DEBUG:boto:Token: None 2023-11-02T18:29:39.734 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/c?versionId=null 2023-11-02T18:29:39.734 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rPYQIdaGbqjb1Id+/5Vxhi9WlQQ= 2023-11-02T18:29:39.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rPYQIdaGbqjb1Id+/5Vxhi9WlQQ=', 'Content-Length': '0'} 2023-11-02T18:29:39.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000614f6ec2c6a537c7-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.738 DEBUG:boto:path=/ibtljn-57/c 2023-11-02T18:29:39.739 DEBUG:boto:auth_path=/ibtljn-57/c 2023-11-02T18:29:39.739 DEBUG:boto:path=/ibtljn-57/c?versionId=null 2023-11-02T18:29:39.739 DEBUG:boto:auth_path=/ibtljn-57/c?versionId=null 2023-11-02T18:29:39.740 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.740 DEBUG:boto:Path: /ibtljn-57/c?versionId=null 2023-11-02T18:29:39.741 DEBUG:boto:Data: 2023-11-02T18:29:39.741 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.741 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.742 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.742 DEBUG:boto:Params: {} 2023-11-02T18:29:39.743 DEBUG:boto:Token: None 2023-11-02T18:29:39.743 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/c?versionId=null 2023-11-02T18:29:39.744 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rPYQIdaGbqjb1Id+/5Vxhi9WlQQ= 2023-11-02T18:29:39.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rPYQIdaGbqjb1Id+/5Vxhi9WlQQ=', 'Content-Length': '0'} 2023-11-02T18:29:39.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1c70f4cc06c9da-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:29:39.748 DEBUG:boto:path=/ibtljn-57/c 2023-11-02T18:29:39.749 DEBUG:boto:auth_path=/ibtljn-57/c 2023-11-02T18:29:39.749 DEBUG:boto:path=/ibtljn-57/c?versionId=null 2023-11-02T18:29:39.749 DEBUG:boto:auth_path=/ibtljn-57/c?versionId=null 2023-11-02T18:29:39.750 DEBUG:boto:Method: GET 2023-11-02T18:29:39.750 DEBUG:boto:Path: /ibtljn-57/c?versionId=null 2023-11-02T18:29:39.751 DEBUG:boto:Data: 2023-11-02T18:29:39.751 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.752 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.752 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.752 DEBUG:boto:Params: {} 2023-11-02T18:29:39.753 DEBUG:boto:Token: None 2023-11-02T18:29:39.753 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/c?versionId=null 2023-11-02T18:29:39.754 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:halJsGgRn7XDXQulaARK4dKVXrc= 2023-11-02T18:29:39.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:halJsGgRn7XDXQulaARK4dKVXrc=', 'Content-Length': '0'} 2023-11-02T18:29:39.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be7e79fac8fd8424-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.758 DEBUG:boto:path=/ibtljn-57/c 2023-11-02T18:29:39.759 DEBUG:boto:auth_path=/ibtljn-57/c 2023-11-02T18:29:39.759 DEBUG:boto:path=/ibtljn-57/c?versionId=null 2023-11-02T18:29:39.760 DEBUG:boto:auth_path=/ibtljn-57/c?versionId=null 2023-11-02T18:29:39.760 DEBUG:boto:Method: GET 2023-11-02T18:29:39.761 DEBUG:boto:Path: /ibtljn-57/c?versionId=null 2023-11-02T18:29:39.761 DEBUG:boto:Data: 2023-11-02T18:29:39.761 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.762 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.762 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.763 DEBUG:boto:Params: {} 2023-11-02T18:29:39.763 DEBUG:boto:Token: None 2023-11-02T18:29:39.764 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/c?versionId=null 2023-11-02T18:29:39.764 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:halJsGgRn7XDXQulaARK4dKVXrc= 2023-11-02T18:29:39.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:halJsGgRn7XDXQulaARK4dKVXrc=', 'Content-Length': '0'} 2023-11-02T18:29:39.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002689b5aa9bc56d9c-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.810 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.811 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.812 DEBUG:boto:path=/ibtljn-57/?prefix=c 2023-11-02T18:29:39.812 DEBUG:boto:auth_path=/ibtljn-57/?prefix=c 2023-11-02T18:29:39.812 DEBUG:boto:Method: GET 2023-11-02T18:29:39.813 DEBUG:boto:Path: /ibtljn-57/?prefix=c 2023-11-02T18:29:39.813 DEBUG:boto:Data: 2023-11-02T18:29:39.814 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.814 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.815 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.815 DEBUG:boto:Params: {} 2023-11-02T18:29:39.816 DEBUG:boto:Token: None 2023-11-02T18:29:39.816 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.817 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo= 2023-11-02T18:29:39.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo=', 'Content-Length': '0'} 2023-11-02T18:29:39.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3c5f298a25fe55a-006543ea93-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.824 DEBUG:boto:b'ibtljn-57c1000falsec2023-11-02T18:26:57.216Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.825 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.825 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.825 DEBUG:boto:path=/ibtljn-57/?prefix=c 2023-11-02T18:29:39.826 DEBUG:boto:auth_path=/ibtljn-57/?prefix=c 2023-11-02T18:29:39.826 DEBUG:boto:Method: GET 2023-11-02T18:29:39.827 DEBUG:boto:Path: /ibtljn-57/?prefix=c 2023-11-02T18:29:39.827 DEBUG:boto:Data: 2023-11-02T18:29:39.828 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.828 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.829 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.829 DEBUG:boto:Params: {} 2023-11-02T18:29:39.830 DEBUG:boto:Token: None 2023-11-02T18:29:39.830 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.831 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo= 2023-11-02T18:29:39.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo=', 'Content-Length': '0'} 2023-11-02T18:29:39.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a652fcd88b7af47-006543ea93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.835 DEBUG:boto:b'ibtljn-57c1000falsec2023-11-02T18:26:57.216Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.836 DEBUG:boto:path=/ibtljn-57/c 2023-11-02T18:29:39.837 DEBUG:boto:auth_path=/ibtljn-57/c 2023-11-02T18:29:39.837 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.837 DEBUG:boto:Path: /ibtljn-57/c 2023-11-02T18:29:39.838 DEBUG:boto:Data: 2023-11-02T18:29:39.838 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.839 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.839 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.840 DEBUG:boto:Params: {} 2023-11-02T18:29:39.840 DEBUG:boto:Token: None 2023-11-02T18:29:39.840 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/c 2023-11-02T18:29:39.841 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:47ZRar8Y+gsW42mgGGpUDal1Q1E= 2023-11-02T18:29:39.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:47ZRar8Y+gsW42mgGGpUDal1Q1E=', 'Content-Length': '0'} 2023-11-02T18:29:39.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb01fad8b1f2accd-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.845 DEBUG:boto:path=/ibtljn-57/c 2023-11-02T18:29:39.845 DEBUG:boto:auth_path=/ibtljn-57/c 2023-11-02T18:29:39.846 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.846 DEBUG:boto:Path: /ibtljn-57/c 2023-11-02T18:29:39.847 DEBUG:boto:Data: 2023-11-02T18:29:39.847 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.848 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.848 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.848 DEBUG:boto:Params: {} 2023-11-02T18:29:39.849 DEBUG:boto:Token: None 2023-11-02T18:29:39.849 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/c 2023-11-02T18:29:39.850 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:47ZRar8Y+gsW42mgGGpUDal1Q1E= 2023-11-02T18:29:39.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:47ZRar8Y+gsW42mgGGpUDal1Q1E=', 'Content-Length': '0'} 2023-11-02T18:29:39.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009035ac51a6b3e385-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-11-02T18:29:39.854 DEBUG:boto:path=/ibtljn-57/c 2023-11-02T18:29:39.855 DEBUG:boto:auth_path=/ibtljn-57/c 2023-11-02T18:29:39.855 DEBUG:boto:Method: GET 2023-11-02T18:29:39.855 DEBUG:boto:Path: /ibtljn-57/c 2023-11-02T18:29:39.856 DEBUG:boto:Data: 2023-11-02T18:29:39.856 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.857 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.858 DEBUG:boto:Params: {} 2023-11-02T18:29:39.858 DEBUG:boto:Token: None 2023-11-02T18:29:39.859 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/c 2023-11-02T18:29:39.859 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5LLvuRm+46vGoaZj+SAzZtj+duw= 2023-11-02T18:29:39.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5LLvuRm+46vGoaZj+SAzZtj+duw=', 'Content-Length': '0'} 2023-11-02T18:29:39.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008918327602d35046-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.863 DEBUG:boto:path=/ibtljn-57/c 2023-11-02T18:29:39.864 DEBUG:boto:auth_path=/ibtljn-57/c 2023-11-02T18:29:39.864 DEBUG:boto:Method: GET 2023-11-02T18:29:39.864 DEBUG:boto:Path: /ibtljn-57/c 2023-11-02T18:29:39.865 DEBUG:boto:Data: 2023-11-02T18:29:39.865 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.866 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.866 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.867 DEBUG:boto:Params: {} 2023-11-02T18:29:39.867 DEBUG:boto:Token: None 2023-11-02T18:29:39.868 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/c 2023-11-02T18:29:39.868 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5LLvuRm+46vGoaZj+SAzZtj+duw= 2023-11-02T18:29:39.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5LLvuRm+46vGoaZj+SAzZtj+duw=', 'Content-Length': '0'} 2023-11-02T18:29:39.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004543a37ca2374521-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.873 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.873 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.873 DEBUG:boto:path=/ibtljn-57/?prefix=c 2023-11-02T18:29:39.874 DEBUG:boto:auth_path=/ibtljn-57/?prefix=c 2023-11-02T18:29:39.874 DEBUG:boto:Method: GET 2023-11-02T18:29:39.875 DEBUG:boto:Path: /ibtljn-57/?prefix=c 2023-11-02T18:29:39.875 DEBUG:boto:Data: 2023-11-02T18:29:39.876 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.876 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.876 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.877 DEBUG:boto:Params: {} 2023-11-02T18:29:39.877 DEBUG:boto:Token: None 2023-11-02T18:29:39.878 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.878 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo= 2023-11-02T18:29:39.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo=', 'Content-Length': '0'} 2023-11-02T18:29:39.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ec821318ed7f8be-006543ea93-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.926 DEBUG:boto:b'ibtljn-57c1000falsec2023-11-02T18:26:57.216Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.927 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:39.928 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:39.928 DEBUG:boto:path=/ibtljn-57/?prefix=c 2023-11-02T18:29:39.929 DEBUG:boto:auth_path=/ibtljn-57/?prefix=c 2023-11-02T18:29:39.929 DEBUG:boto:Method: GET 2023-11-02T18:29:39.930 DEBUG:boto:Path: /ibtljn-57/?prefix=c 2023-11-02T18:29:39.930 DEBUG:boto:Data: 2023-11-02T18:29:39.930 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.931 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.931 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.932 DEBUG:boto:Params: {} 2023-11-02T18:29:39.932 DEBUG:boto:Token: None 2023-11-02T18:29:39.933 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/ 2023-11-02T18:29:39.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo= 2023-11-02T18:29:39.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kwvF3O98tlf4Zh10J5azMxkUcdo=', 'Content-Length': '0'} 2023-11-02T18:29:39.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086c8f1ccc9d3e068-006543ea93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.938 DEBUG:boto:b'ibtljn-57c1000falsec2023-11-02T18:26:57.216Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:39.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:29:39.939 DEBUG:boto:path=/ibtljn-57/d 2023-11-02T18:29:39.939 DEBUG:boto:auth_path=/ibtljn-57/d 2023-11-02T18:29:39.940 DEBUG:boto:path=/ibtljn-57/d?versionId=null 2023-11-02T18:29:39.940 DEBUG:boto:auth_path=/ibtljn-57/d?versionId=null 2023-11-02T18:29:39.941 DEBUG:boto:Method: GET 2023-11-02T18:29:39.941 DEBUG:boto:Path: /ibtljn-57/d?versionId=null 2023-11-02T18:29:39.941 DEBUG:boto:Data: 2023-11-02T18:29:39.942 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.942 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.943 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.943 DEBUG:boto:Params: {} 2023-11-02T18:29:39.944 DEBUG:boto:Token: None 2023-11-02T18:29:39.944 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/d?versionId=null 2023-11-02T18:29:39.945 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xfvgJkztlyodI1SG03h4mV6z9Bk= 2023-11-02T18:29:39.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xfvgJkztlyodI1SG03h4mV6z9Bk=', 'Content-Length': '0'} 2023-11-02T18:29:39.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd506f3fe231be78-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.949 DEBUG:boto:path=/ibtljn-57/d 2023-11-02T18:29:39.949 DEBUG:boto:auth_path=/ibtljn-57/d 2023-11-02T18:29:39.950 DEBUG:boto:path=/ibtljn-57/d?versionId=null 2023-11-02T18:29:39.950 DEBUG:boto:auth_path=/ibtljn-57/d?versionId=null 2023-11-02T18:29:39.950 DEBUG:boto:Method: GET 2023-11-02T18:29:39.951 DEBUG:boto:Path: /ibtljn-57/d?versionId=null 2023-11-02T18:29:39.951 DEBUG:boto:Data: 2023-11-02T18:29:39.952 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.952 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.953 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.953 DEBUG:boto:Params: {} 2023-11-02T18:29:39.953 DEBUG:boto:Token: None 2023-11-02T18:29:39.954 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/d?versionId=null 2023-11-02T18:29:39.954 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xfvgJkztlyodI1SG03h4mV6z9Bk= 2023-11-02T18:29:39.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xfvgJkztlyodI1SG03h4mV6z9Bk=', 'Content-Length': '0'} 2023-11-02T18:29:39.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000480494a7a965ff62-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.959 DEBUG:boto:path=/ibtljn-57/d 2023-11-02T18:29:39.959 DEBUG:boto:auth_path=/ibtljn-57/d 2023-11-02T18:29:39.959 DEBUG:boto:path=/ibtljn-57/d?versionId=null 2023-11-02T18:29:39.960 DEBUG:boto:auth_path=/ibtljn-57/d?versionId=null 2023-11-02T18:29:39.960 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.961 DEBUG:boto:Path: /ibtljn-57/d?versionId=null 2023-11-02T18:29:39.961 DEBUG:boto:Data: 2023-11-02T18:29:39.962 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.962 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.962 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.963 DEBUG:boto:Params: {} 2023-11-02T18:29:39.963 DEBUG:boto:Token: None 2023-11-02T18:29:39.964 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/d?versionId=null 2023-11-02T18:29:39.964 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Oz644MH9Mo9s4jN1N7coqOrVcuE= 2023-11-02T18:29:39.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Oz644MH9Mo9s4jN1N7coqOrVcuE=', 'Content-Length': '0'} 2023-11-02T18:29:39.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d23091c01ca2c996-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.968 DEBUG:boto:path=/ibtljn-57/d 2023-11-02T18:29:39.969 DEBUG:boto:auth_path=/ibtljn-57/d 2023-11-02T18:29:39.969 DEBUG:boto:path=/ibtljn-57/d?versionId=null 2023-11-02T18:29:39.969 DEBUG:boto:auth_path=/ibtljn-57/d?versionId=null 2023-11-02T18:29:39.970 DEBUG:boto:Method: HEAD 2023-11-02T18:29:39.970 DEBUG:boto:Path: /ibtljn-57/d?versionId=null 2023-11-02T18:29:39.971 DEBUG:boto:Data: 2023-11-02T18:29:39.971 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.971 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:39.972 DEBUG:boto:Port: 8000 2023-11-02T18:29:39.972 DEBUG:boto:Params: {} 2023-11-02T18:29:39.973 DEBUG:boto:Token: None 2023-11-02T18:29:39.973 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/d?versionId=null 2023-11-02T18:29:39.974 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Oz644MH9Mo9s4jN1N7coqOrVcuE= 2023-11-02T18:29:39.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Oz644MH9Mo9s4jN1N7coqOrVcuE=', 'Content-Length': '0'} 2023-11-02T18:29:39.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e266f4caa10a1c66-006543ea93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:39.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:29:39.978 DEBUG:boto:path=/ibtljn-57/d 2023-11-02T18:29:39.978 DEBUG:boto:auth_path=/ibtljn-57/d 2023-11-02T18:29:39.979 DEBUG:boto:path=/ibtljn-57/d?versionId=null 2023-11-02T18:29:39.979 DEBUG:boto:auth_path=/ibtljn-57/d?versionId=null 2023-11-02T18:29:39.980 DEBUG:boto:Method: GET 2023-11-02T18:29:39.980 DEBUG:boto:Path: /ibtljn-57/d?versionId=null 2023-11-02T18:29:39.981 DEBUG:boto:Data: 2023-11-02T18:29:39.981 DEBUG:boto:Headers: {} 2023-11-02T18:29:39.981 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:39.982 DEBUG:boto:Port: 8001 2023-11-02T18:29:39.982 DEBUG:boto:Params: {} 2023-11-02T18:29:39.983 DEBUG:boto:Token: None 2023-11-02T18:29:39.983 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:39 GMT /ibtljn-57/d?versionId=null 2023-11-02T18:29:39.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xfvgJkztlyodI1SG03h4mV6z9Bk= 2023-11-02T18:29:39.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xfvgJkztlyodI1SG03h4mV6z9Bk=', 'Content-Length': '0'} 2023-11-02T18:29:39.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007581beab5c14a0d9-006543ea93-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.030 DEBUG:boto:path=/ibtljn-57/d 2023-11-02T18:29:40.031 DEBUG:boto:auth_path=/ibtljn-57/d 2023-11-02T18:29:40.031 DEBUG:boto:path=/ibtljn-57/d?versionId=null 2023-11-02T18:29:40.032 DEBUG:boto:auth_path=/ibtljn-57/d?versionId=null 2023-11-02T18:29:40.032 DEBUG:boto:Method: GET 2023-11-02T18:29:40.033 DEBUG:boto:Path: /ibtljn-57/d?versionId=null 2023-11-02T18:29:40.033 DEBUG:boto:Data: 2023-11-02T18:29:40.034 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.034 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.035 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.035 DEBUG:boto:Params: {} 2023-11-02T18:29:40.035 DEBUG:boto:Token: None 2023-11-02T18:29:40.036 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/d?versionId=null 2023-11-02T18:29:40.037 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oIyabP88eEAVbMt85gc5qwz0tUI= 2023-11-02T18:29:40.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oIyabP88eEAVbMt85gc5qwz0tUI=', 'Content-Length': '0'} 2023-11-02T18:29:40.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa8f5276a4a335e4-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.041 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.041 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.041 DEBUG:boto:path=/ibtljn-57/?prefix=d 2023-11-02T18:29:40.042 DEBUG:boto:auth_path=/ibtljn-57/?prefix=d 2023-11-02T18:29:40.042 DEBUG:boto:Method: GET 2023-11-02T18:29:40.043 DEBUG:boto:Path: /ibtljn-57/?prefix=d 2023-11-02T18:29:40.043 DEBUG:boto:Data: 2023-11-02T18:29:40.044 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.045 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.045 DEBUG:boto:Params: {} 2023-11-02T18:29:40.045 DEBUG:boto:Token: None 2023-11-02T18:29:40.046 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.046 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa15b04d6380718f-006543ea94-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.053 DEBUG:boto:b'ibtljn-57d1000falsed2023-11-02T18:26:57.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.054 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.054 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.055 DEBUG:boto:path=/ibtljn-57/?prefix=d 2023-11-02T18:29:40.055 DEBUG:boto:auth_path=/ibtljn-57/?prefix=d 2023-11-02T18:29:40.056 DEBUG:boto:Method: GET 2023-11-02T18:29:40.056 DEBUG:boto:Path: /ibtljn-57/?prefix=d 2023-11-02T18:29:40.056 DEBUG:boto:Data: 2023-11-02T18:29:40.057 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.057 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.058 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.058 DEBUG:boto:Params: {} 2023-11-02T18:29:40.059 DEBUG:boto:Token: None 2023-11-02T18:29:40.059 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.060 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d21064069befa556-006543ea94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.106 DEBUG:boto:b'ibtljn-57d1000falsed2023-11-02T18:26:57.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.107 DEBUG:boto:path=/ibtljn-57/d 2023-11-02T18:29:40.108 DEBUG:boto:auth_path=/ibtljn-57/d 2023-11-02T18:29:40.108 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.109 DEBUG:boto:Path: /ibtljn-57/d 2023-11-02T18:29:40.109 DEBUG:boto:Data: 2023-11-02T18:29:40.109 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.110 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.110 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.111 DEBUG:boto:Params: {} 2023-11-02T18:29:40.111 DEBUG:boto:Token: None 2023-11-02T18:29:40.111 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/d 2023-11-02T18:29:40.112 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CimQevR8Q1KvZebFPFe7OmA0nsk= 2023-11-02T18:29:40.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CimQevR8Q1KvZebFPFe7OmA0nsk=', 'Content-Length': '0'} 2023-11-02T18:29:40.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001af91a50c4796bbb-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.116 DEBUG:boto:path=/ibtljn-57/d 2023-11-02T18:29:40.117 DEBUG:boto:auth_path=/ibtljn-57/d 2023-11-02T18:29:40.117 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.117 DEBUG:boto:Path: /ibtljn-57/d 2023-11-02T18:29:40.118 DEBUG:boto:Data: 2023-11-02T18:29:40.118 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.119 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.119 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.119 DEBUG:boto:Params: {} 2023-11-02T18:29:40.120 DEBUG:boto:Token: None 2023-11-02T18:29:40.120 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/d 2023-11-02T18:29:40.121 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CimQevR8Q1KvZebFPFe7OmA0nsk= 2023-11-02T18:29:40.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CimQevR8Q1KvZebFPFe7OmA0nsk=', 'Content-Length': '0'} 2023-11-02T18:29:40.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6bde495fedf9dfe-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-11-02T18:29:40.126 DEBUG:boto:path=/ibtljn-57/d 2023-11-02T18:29:40.127 DEBUG:boto:auth_path=/ibtljn-57/d 2023-11-02T18:29:40.127 DEBUG:boto:Method: GET 2023-11-02T18:29:40.128 DEBUG:boto:Path: /ibtljn-57/d 2023-11-02T18:29:40.128 DEBUG:boto:Data: 2023-11-02T18:29:40.128 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.129 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.129 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.130 DEBUG:boto:Params: {} 2023-11-02T18:29:40.130 DEBUG:boto:Token: None 2023-11-02T18:29:40.131 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/d 2023-11-02T18:29:40.131 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sN37bUtJr5yXmTCNj1vOApBvz60= 2023-11-02T18:29:40.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sN37bUtJr5yXmTCNj1vOApBvz60=', 'Content-Length': '0'} 2023-11-02T18:29:40.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002739b971e0955ad6-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.135 DEBUG:boto:path=/ibtljn-57/d 2023-11-02T18:29:40.136 DEBUG:boto:auth_path=/ibtljn-57/d 2023-11-02T18:29:40.136 DEBUG:boto:Method: GET 2023-11-02T18:29:40.137 DEBUG:boto:Path: /ibtljn-57/d 2023-11-02T18:29:40.137 DEBUG:boto:Data: 2023-11-02T18:29:40.138 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.138 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.139 DEBUG:boto:Params: {} 2023-11-02T18:29:40.139 DEBUG:boto:Token: None 2023-11-02T18:29:40.140 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/d 2023-11-02T18:29:40.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sN37bUtJr5yXmTCNj1vOApBvz60= 2023-11-02T18:29:40.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sN37bUtJr5yXmTCNj1vOApBvz60=', 'Content-Length': '0'} 2023-11-02T18:29:40.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c5127f0170bdf4-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.145 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.145 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.145 DEBUG:boto:path=/ibtljn-57/?prefix=d 2023-11-02T18:29:40.146 DEBUG:boto:auth_path=/ibtljn-57/?prefix=d 2023-11-02T18:29:40.146 DEBUG:boto:Method: GET 2023-11-02T18:29:40.147 DEBUG:boto:Path: /ibtljn-57/?prefix=d 2023-11-02T18:29:40.147 DEBUG:boto:Data: 2023-11-02T18:29:40.148 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.148 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.149 DEBUG:boto:Params: {} 2023-11-02T18:29:40.150 DEBUG:boto:Token: None 2023-11-02T18:29:40.150 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c4c81989489273-006543ea94-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.198 DEBUG:boto:b'ibtljn-57d1000falsed2023-11-02T18:26:57.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.199 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.200 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.200 DEBUG:boto:path=/ibtljn-57/?prefix=d 2023-11-02T18:29:40.201 DEBUG:boto:auth_path=/ibtljn-57/?prefix=d 2023-11-02T18:29:40.201 DEBUG:boto:Method: GET 2023-11-02T18:29:40.202 DEBUG:boto:Path: /ibtljn-57/?prefix=d 2023-11-02T18:29:40.202 DEBUG:boto:Data: 2023-11-02T18:29:40.203 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.203 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.203 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.204 DEBUG:boto:Params: {} 2023-11-02T18:29:40.204 DEBUG:boto:Token: None 2023-11-02T18:29:40.205 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.205 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e54fe0c82939ef0-006543ea94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.210 DEBUG:boto:b'ibtljn-57d1000falsed2023-11-02T18:26:57.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-11-02T18:29:40.211 DEBUG:boto:path=/ibtljn-57/e 2023-11-02T18:29:40.211 DEBUG:boto:auth_path=/ibtljn-57/e 2023-11-02T18:29:40.212 DEBUG:boto:path=/ibtljn-57/e?versionId=null 2023-11-02T18:29:40.212 DEBUG:boto:auth_path=/ibtljn-57/e?versionId=null 2023-11-02T18:29:40.213 DEBUG:boto:Method: GET 2023-11-02T18:29:40.213 DEBUG:boto:Path: /ibtljn-57/e?versionId=null 2023-11-02T18:29:40.214 DEBUG:boto:Data: 2023-11-02T18:29:40.214 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.214 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.215 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.215 DEBUG:boto:Params: {} 2023-11-02T18:29:40.216 DEBUG:boto:Token: None 2023-11-02T18:29:40.216 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/e?versionId=null 2023-11-02T18:29:40.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVu/Ao31fH3HM3+gP3zuJMVT7dQ= 2023-11-02T18:29:40.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVu/Ao31fH3HM3+gP3zuJMVT7dQ=', 'Content-Length': '0'} 2023-11-02T18:29:40.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034435b0a130f27cb-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.221 DEBUG:boto:path=/ibtljn-57/e 2023-11-02T18:29:40.222 DEBUG:boto:auth_path=/ibtljn-57/e 2023-11-02T18:29:40.222 DEBUG:boto:path=/ibtljn-57/e?versionId=null 2023-11-02T18:29:40.223 DEBUG:boto:auth_path=/ibtljn-57/e?versionId=null 2023-11-02T18:29:40.223 DEBUG:boto:Method: GET 2023-11-02T18:29:40.223 DEBUG:boto:Path: /ibtljn-57/e?versionId=null 2023-11-02T18:29:40.224 DEBUG:boto:Data: 2023-11-02T18:29:40.224 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.225 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.225 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.226 DEBUG:boto:Params: {} 2023-11-02T18:29:40.226 DEBUG:boto:Token: None 2023-11-02T18:29:40.227 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/e?versionId=null 2023-11-02T18:29:40.227 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVu/Ao31fH3HM3+gP3zuJMVT7dQ= 2023-11-02T18:29:40.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVu/Ao31fH3HM3+gP3zuJMVT7dQ=', 'Content-Length': '0'} 2023-11-02T18:29:40.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aba9ccaa82d3260e-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.231 DEBUG:boto:path=/ibtljn-57/e 2023-11-02T18:29:40.232 DEBUG:boto:auth_path=/ibtljn-57/e 2023-11-02T18:29:40.232 DEBUG:boto:path=/ibtljn-57/e?versionId=null 2023-11-02T18:29:40.232 DEBUG:boto:auth_path=/ibtljn-57/e?versionId=null 2023-11-02T18:29:40.233 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.233 DEBUG:boto:Path: /ibtljn-57/e?versionId=null 2023-11-02T18:29:40.234 DEBUG:boto:Data: 2023-11-02T18:29:40.234 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.234 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.235 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.235 DEBUG:boto:Params: {} 2023-11-02T18:29:40.236 DEBUG:boto:Token: None 2023-11-02T18:29:40.236 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/e?versionId=null 2023-11-02T18:29:40.237 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1JRYertTlKmUS0SyR5Cj77pGHKQ= 2023-11-02T18:29:40.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1JRYertTlKmUS0SyR5Cj77pGHKQ=', 'Content-Length': '0'} 2023-11-02T18:29:40.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b681d4d2b1724663-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.241 DEBUG:boto:path=/ibtljn-57/e 2023-11-02T18:29:40.241 DEBUG:boto:auth_path=/ibtljn-57/e 2023-11-02T18:29:40.241 DEBUG:boto:path=/ibtljn-57/e?versionId=null 2023-11-02T18:29:40.242 DEBUG:boto:auth_path=/ibtljn-57/e?versionId=null 2023-11-02T18:29:40.242 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.243 DEBUG:boto:Path: /ibtljn-57/e?versionId=null 2023-11-02T18:29:40.243 DEBUG:boto:Data: 2023-11-02T18:29:40.243 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.244 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.244 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.245 DEBUG:boto:Params: {} 2023-11-02T18:29:40.245 DEBUG:boto:Token: None 2023-11-02T18:29:40.246 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/e?versionId=null 2023-11-02T18:29:40.246 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1JRYertTlKmUS0SyR5Cj77pGHKQ= 2023-11-02T18:29:40.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1JRYertTlKmUS0SyR5Cj77pGHKQ=', 'Content-Length': '0'} 2023-11-02T18:29:40.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ab439c6b1ab4ea1-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-11-02T18:29:40.250 DEBUG:boto:path=/ibtljn-57/e 2023-11-02T18:29:40.251 DEBUG:boto:auth_path=/ibtljn-57/e 2023-11-02T18:29:40.251 DEBUG:boto:path=/ibtljn-57/e?versionId=null 2023-11-02T18:29:40.252 DEBUG:boto:auth_path=/ibtljn-57/e?versionId=null 2023-11-02T18:29:40.252 DEBUG:boto:Method: GET 2023-11-02T18:29:40.252 DEBUG:boto:Path: /ibtljn-57/e?versionId=null 2023-11-02T18:29:40.253 DEBUG:boto:Data: 2023-11-02T18:29:40.253 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.254 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.255 DEBUG:boto:Params: {} 2023-11-02T18:29:40.255 DEBUG:boto:Token: None 2023-11-02T18:29:40.256 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/e?versionId=null 2023-11-02T18:29:40.256 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVu/Ao31fH3HM3+gP3zuJMVT7dQ= 2023-11-02T18:29:40.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVu/Ao31fH3HM3+gP3zuJMVT7dQ=', 'Content-Length': '0'} 2023-11-02T18:29:40.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d97819bad1e5909c-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.306 DEBUG:boto:path=/ibtljn-57/e 2023-11-02T18:29:40.307 DEBUG:boto:auth_path=/ibtljn-57/e 2023-11-02T18:29:40.307 DEBUG:boto:path=/ibtljn-57/e?versionId=null 2023-11-02T18:29:40.308 DEBUG:boto:auth_path=/ibtljn-57/e?versionId=null 2023-11-02T18:29:40.308 DEBUG:boto:Method: GET 2023-11-02T18:29:40.309 DEBUG:boto:Path: /ibtljn-57/e?versionId=null 2023-11-02T18:29:40.309 DEBUG:boto:Data: 2023-11-02T18:29:40.310 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.310 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.311 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.311 DEBUG:boto:Params: {} 2023-11-02T18:29:40.311 DEBUG:boto:Token: None 2023-11-02T18:29:40.312 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/e?versionId=null 2023-11-02T18:29:40.312 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVu/Ao31fH3HM3+gP3zuJMVT7dQ= 2023-11-02T18:29:40.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVu/Ao31fH3HM3+gP3zuJMVT7dQ=', 'Content-Length': '0'} 2023-11-02T18:29:40.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fadbbc8e75054024-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.316 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.316 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.317 DEBUG:boto:path=/ibtljn-57/?prefix=e 2023-11-02T18:29:40.317 DEBUG:boto:auth_path=/ibtljn-57/?prefix=e 2023-11-02T18:29:40.318 DEBUG:boto:Method: GET 2023-11-02T18:29:40.318 DEBUG:boto:Path: /ibtljn-57/?prefix=e 2023-11-02T18:29:40.319 DEBUG:boto:Data: 2023-11-02T18:29:40.319 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.319 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.320 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.320 DEBUG:boto:Params: {} 2023-11-02T18:29:40.321 DEBUG:boto:Token: None 2023-11-02T18:29:40.321 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.322 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ac767a007ac9a6e-006543ea94-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.328 DEBUG:boto:b'ibtljn-57e1000falsee2023-11-02T18:26:57.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.329 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.329 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.330 DEBUG:boto:path=/ibtljn-57/?prefix=e 2023-11-02T18:29:40.330 DEBUG:boto:auth_path=/ibtljn-57/?prefix=e 2023-11-02T18:29:40.331 DEBUG:boto:Method: GET 2023-11-02T18:29:40.331 DEBUG:boto:Path: /ibtljn-57/?prefix=e 2023-11-02T18:29:40.331 DEBUG:boto:Data: 2023-11-02T18:29:40.332 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.332 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.333 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.333 DEBUG:boto:Params: {} 2023-11-02T18:29:40.334 DEBUG:boto:Token: None 2023-11-02T18:29:40.334 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.335 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0adb570b6126a8e-006543ea94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.339 DEBUG:boto:b'ibtljn-57e1000falsee2023-11-02T18:26:57.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.339 DEBUG:boto:path=/ibtljn-57/e 2023-11-02T18:29:40.340 DEBUG:boto:auth_path=/ibtljn-57/e 2023-11-02T18:29:40.340 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.341 DEBUG:boto:Path: /ibtljn-57/e 2023-11-02T18:29:40.341 DEBUG:boto:Data: 2023-11-02T18:29:40.341 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.342 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.342 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.343 DEBUG:boto:Params: {} 2023-11-02T18:29:40.343 DEBUG:boto:Token: None 2023-11-02T18:29:40.344 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/e 2023-11-02T18:29:40.344 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SZUVb0qJg6Oq24cgP94l1qdnlf8= 2023-11-02T18:29:40.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SZUVb0qJg6Oq24cgP94l1qdnlf8=', 'Content-Length': '0'} 2023-11-02T18:29:40.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dbc6869e1e6c991-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.348 DEBUG:boto:path=/ibtljn-57/e 2023-11-02T18:29:40.348 DEBUG:boto:auth_path=/ibtljn-57/e 2023-11-02T18:29:40.348 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.349 DEBUG:boto:Path: /ibtljn-57/e 2023-11-02T18:29:40.349 DEBUG:boto:Data: 2023-11-02T18:29:40.350 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.350 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.351 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.351 DEBUG:boto:Params: {} 2023-11-02T18:29:40.351 DEBUG:boto:Token: None 2023-11-02T18:29:40.352 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/e 2023-11-02T18:29:40.352 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SZUVb0qJg6Oq24cgP94l1qdnlf8= 2023-11-02T18:29:40.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SZUVb0qJg6Oq24cgP94l1qdnlf8=', 'Content-Length': '0'} 2023-11-02T18:29:40.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683b06a75579146d-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-11-02T18:29:40.356 DEBUG:boto:path=/ibtljn-57/e 2023-11-02T18:29:40.357 DEBUG:boto:auth_path=/ibtljn-57/e 2023-11-02T18:29:40.357 DEBUG:boto:Method: GET 2023-11-02T18:29:40.358 DEBUG:boto:Path: /ibtljn-57/e 2023-11-02T18:29:40.358 DEBUG:boto:Data: 2023-11-02T18:29:40.358 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.359 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.359 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.360 DEBUG:boto:Params: {} 2023-11-02T18:29:40.360 DEBUG:boto:Token: None 2023-11-02T18:29:40.361 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/e 2023-11-02T18:29:40.361 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:awXVZ0hStIQcbJmRZ+qsm3JnksA= 2023-11-02T18:29:40.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:awXVZ0hStIQcbJmRZ+qsm3JnksA=', 'Content-Length': '0'} 2023-11-02T18:29:40.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002449411bf9b195-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.406 DEBUG:boto:path=/ibtljn-57/e 2023-11-02T18:29:40.407 DEBUG:boto:auth_path=/ibtljn-57/e 2023-11-02T18:29:40.407 DEBUG:boto:Method: GET 2023-11-02T18:29:40.408 DEBUG:boto:Path: /ibtljn-57/e 2023-11-02T18:29:40.408 DEBUG:boto:Data: 2023-11-02T18:29:40.409 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.409 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.410 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.410 DEBUG:boto:Params: {} 2023-11-02T18:29:40.411 DEBUG:boto:Token: None 2023-11-02T18:29:40.411 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/e 2023-11-02T18:29:40.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:awXVZ0hStIQcbJmRZ+qsm3JnksA= 2023-11-02T18:29:40.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:awXVZ0hStIQcbJmRZ+qsm3JnksA=', 'Content-Length': '0'} 2023-11-02T18:29:40.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000455286c544d311a5-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.416 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.416 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.417 DEBUG:boto:path=/ibtljn-57/?prefix=e 2023-11-02T18:29:40.417 DEBUG:boto:auth_path=/ibtljn-57/?prefix=e 2023-11-02T18:29:40.418 DEBUG:boto:Method: GET 2023-11-02T18:29:40.418 DEBUG:boto:Path: /ibtljn-57/?prefix=e 2023-11-02T18:29:40.418 DEBUG:boto:Data: 2023-11-02T18:29:40.419 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.419 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.420 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.420 DEBUG:boto:Params: {} 2023-11-02T18:29:40.421 DEBUG:boto:Token: None 2023-11-02T18:29:40.421 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.422 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005594df496b4c8620-006543ea94-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.429 DEBUG:boto:b'ibtljn-57e1000falsee2023-11-02T18:26:57.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.429 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.430 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.430 DEBUG:boto:path=/ibtljn-57/?prefix=e 2023-11-02T18:29:40.431 DEBUG:boto:auth_path=/ibtljn-57/?prefix=e 2023-11-02T18:29:40.431 DEBUG:boto:Method: GET 2023-11-02T18:29:40.432 DEBUG:boto:Path: /ibtljn-57/?prefix=e 2023-11-02T18:29:40.432 DEBUG:boto:Data: 2023-11-02T18:29:40.432 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.433 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.433 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.434 DEBUG:boto:Params: {} 2023-11-02T18:29:40.434 DEBUG:boto:Token: None 2023-11-02T18:29:40.435 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.435 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7feef13a40c284c-006543ea94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.482 DEBUG:boto:b'ibtljn-57e1000falsee2023-11-02T18:26:57.519Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-11-02T18:29:40.484 DEBUG:boto:path=/ibtljn-57/f 2023-11-02T18:29:40.484 DEBUG:boto:auth_path=/ibtljn-57/f 2023-11-02T18:29:40.484 DEBUG:boto:path=/ibtljn-57/f?versionId=null 2023-11-02T18:29:40.485 DEBUG:boto:auth_path=/ibtljn-57/f?versionId=null 2023-11-02T18:29:40.485 DEBUG:boto:Method: GET 2023-11-02T18:29:40.486 DEBUG:boto:Path: /ibtljn-57/f?versionId=null 2023-11-02T18:29:40.486 DEBUG:boto:Data: 2023-11-02T18:29:40.487 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.487 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.488 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.488 DEBUG:boto:Params: {} 2023-11-02T18:29:40.488 DEBUG:boto:Token: None 2023-11-02T18:29:40.489 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/f?versionId=null 2023-11-02T18:29:40.489 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wfbQkRfFSe74OXghcabf2cc6hII= 2023-11-02T18:29:40.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wfbQkRfFSe74OXghcabf2cc6hII=', 'Content-Length': '0'} 2023-11-02T18:29:40.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1aa4390fbce84ef-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.494 DEBUG:boto:path=/ibtljn-57/f 2023-11-02T18:29:40.494 DEBUG:boto:auth_path=/ibtljn-57/f 2023-11-02T18:29:40.495 DEBUG:boto:path=/ibtljn-57/f?versionId=null 2023-11-02T18:29:40.495 DEBUG:boto:auth_path=/ibtljn-57/f?versionId=null 2023-11-02T18:29:40.496 DEBUG:boto:Method: GET 2023-11-02T18:29:40.496 DEBUG:boto:Path: /ibtljn-57/f?versionId=null 2023-11-02T18:29:40.497 DEBUG:boto:Data: 2023-11-02T18:29:40.497 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.497 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.498 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.498 DEBUG:boto:Params: {} 2023-11-02T18:29:40.499 DEBUG:boto:Token: None 2023-11-02T18:29:40.499 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/f?versionId=null 2023-11-02T18:29:40.500 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wfbQkRfFSe74OXghcabf2cc6hII= 2023-11-02T18:29:40.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wfbQkRfFSe74OXghcabf2cc6hII=', 'Content-Length': '0'} 2023-11-02T18:29:40.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051a440d0a6a6c81b-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.504 DEBUG:boto:path=/ibtljn-57/f 2023-11-02T18:29:40.504 DEBUG:boto:auth_path=/ibtljn-57/f 2023-11-02T18:29:40.505 DEBUG:boto:path=/ibtljn-57/f?versionId=null 2023-11-02T18:29:40.505 DEBUG:boto:auth_path=/ibtljn-57/f?versionId=null 2023-11-02T18:29:40.506 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.506 DEBUG:boto:Path: /ibtljn-57/f?versionId=null 2023-11-02T18:29:40.506 DEBUG:boto:Data: 2023-11-02T18:29:40.507 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.507 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.508 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.508 DEBUG:boto:Params: {} 2023-11-02T18:29:40.509 DEBUG:boto:Token: None 2023-11-02T18:29:40.509 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/f?versionId=null 2023-11-02T18:29:40.510 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:917WtgsiYANgUR6NaNz2uPI5Zjo= 2023-11-02T18:29:40.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:917WtgsiYANgUR6NaNz2uPI5Zjo=', 'Content-Length': '0'} 2023-11-02T18:29:40.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed1209a042276d8e-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.514 DEBUG:boto:path=/ibtljn-57/f 2023-11-02T18:29:40.514 DEBUG:boto:auth_path=/ibtljn-57/f 2023-11-02T18:29:40.514 DEBUG:boto:path=/ibtljn-57/f?versionId=null 2023-11-02T18:29:40.515 DEBUG:boto:auth_path=/ibtljn-57/f?versionId=null 2023-11-02T18:29:40.515 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.516 DEBUG:boto:Path: /ibtljn-57/f?versionId=null 2023-11-02T18:29:40.516 DEBUG:boto:Data: 2023-11-02T18:29:40.517 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.517 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.517 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.518 DEBUG:boto:Params: {} 2023-11-02T18:29:40.518 DEBUG:boto:Token: None 2023-11-02T18:29:40.519 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/f?versionId=null 2023-11-02T18:29:40.519 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:917WtgsiYANgUR6NaNz2uPI5Zjo= 2023-11-02T18:29:40.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:917WtgsiYANgUR6NaNz2uPI5Zjo=', 'Content-Length': '0'} 2023-11-02T18:29:40.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000344a5efd2c1ba510-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-11-02T18:29:40.524 DEBUG:boto:path=/ibtljn-57/f 2023-11-02T18:29:40.524 DEBUG:boto:auth_path=/ibtljn-57/f 2023-11-02T18:29:40.525 DEBUG:boto:path=/ibtljn-57/f?versionId=null 2023-11-02T18:29:40.525 DEBUG:boto:auth_path=/ibtljn-57/f?versionId=null 2023-11-02T18:29:40.525 DEBUG:boto:Method: GET 2023-11-02T18:29:40.526 DEBUG:boto:Path: /ibtljn-57/f?versionId=null 2023-11-02T18:29:40.526 DEBUG:boto:Data: 2023-11-02T18:29:40.527 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.527 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.528 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.528 DEBUG:boto:Params: {} 2023-11-02T18:29:40.529 DEBUG:boto:Token: None 2023-11-02T18:29:40.529 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/f?versionId=null 2023-11-02T18:29:40.530 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wfbQkRfFSe74OXghcabf2cc6hII= 2023-11-02T18:29:40.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wfbQkRfFSe74OXghcabf2cc6hII=', 'Content-Length': '0'} 2023-11-02T18:29:40.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000322142d902d31471-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.534 DEBUG:boto:path=/ibtljn-57/f 2023-11-02T18:29:40.534 DEBUG:boto:auth_path=/ibtljn-57/f 2023-11-02T18:29:40.535 DEBUG:boto:path=/ibtljn-57/f?versionId=null 2023-11-02T18:29:40.535 DEBUG:boto:auth_path=/ibtljn-57/f?versionId=null 2023-11-02T18:29:40.536 DEBUG:boto:Method: GET 2023-11-02T18:29:40.536 DEBUG:boto:Path: /ibtljn-57/f?versionId=null 2023-11-02T18:29:40.536 DEBUG:boto:Data: 2023-11-02T18:29:40.537 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.537 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.538 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.538 DEBUG:boto:Params: {} 2023-11-02T18:29:40.539 DEBUG:boto:Token: None 2023-11-02T18:29:40.539 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/f?versionId=null 2023-11-02T18:29:40.540 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wfbQkRfFSe74OXghcabf2cc6hII= 2023-11-02T18:29:40.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wfbQkRfFSe74OXghcabf2cc6hII=', 'Content-Length': '0'} 2023-11-02T18:29:40.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc2ec94a968b0305-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.586 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.587 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.588 DEBUG:boto:path=/ibtljn-57/?prefix=f 2023-11-02T18:29:40.588 DEBUG:boto:auth_path=/ibtljn-57/?prefix=f 2023-11-02T18:29:40.588 DEBUG:boto:Method: GET 2023-11-02T18:29:40.589 DEBUG:boto:Path: /ibtljn-57/?prefix=f 2023-11-02T18:29:40.589 DEBUG:boto:Data: 2023-11-02T18:29:40.590 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.591 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.591 DEBUG:boto:Params: {} 2023-11-02T18:29:40.592 DEBUG:boto:Token: None 2023-11-02T18:29:40.592 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee04702d12dd06f-006543ea94-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.597 DEBUG:boto:b'ibtljn-57f1000falsef2023-11-02T18:26:57.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.598 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.598 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.599 DEBUG:boto:path=/ibtljn-57/?prefix=f 2023-11-02T18:29:40.599 DEBUG:boto:auth_path=/ibtljn-57/?prefix=f 2023-11-02T18:29:40.600 DEBUG:boto:Method: GET 2023-11-02T18:29:40.600 DEBUG:boto:Path: /ibtljn-57/?prefix=f 2023-11-02T18:29:40.601 DEBUG:boto:Data: 2023-11-02T18:29:40.601 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.602 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.602 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.602 DEBUG:boto:Params: {} 2023-11-02T18:29:40.603 DEBUG:boto:Token: None 2023-11-02T18:29:40.604 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.604 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000450c838fdcf1c366-006543ea94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.608 DEBUG:boto:b'ibtljn-57f1000falsef2023-11-02T18:26:57.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.608 DEBUG:boto:path=/ibtljn-57/f 2023-11-02T18:29:40.609 DEBUG:boto:auth_path=/ibtljn-57/f 2023-11-02T18:29:40.609 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.610 DEBUG:boto:Path: /ibtljn-57/f 2023-11-02T18:29:40.610 DEBUG:boto:Data: 2023-11-02T18:29:40.611 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.611 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.611 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.612 DEBUG:boto:Params: {} 2023-11-02T18:29:40.612 DEBUG:boto:Token: None 2023-11-02T18:29:40.613 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/f 2023-11-02T18:29:40.613 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x3azZe4qdPJFyC4ND5yVL5S8QqA= 2023-11-02T18:29:40.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x3azZe4qdPJFyC4ND5yVL5S8QqA=', 'Content-Length': '0'} 2023-11-02T18:29:40.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c748f1efeeeecdb-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.617 DEBUG:boto:path=/ibtljn-57/f 2023-11-02T18:29:40.617 DEBUG:boto:auth_path=/ibtljn-57/f 2023-11-02T18:29:40.618 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.618 DEBUG:boto:Path: /ibtljn-57/f 2023-11-02T18:29:40.618 DEBUG:boto:Data: 2023-11-02T18:29:40.619 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.620 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.620 DEBUG:boto:Params: {} 2023-11-02T18:29:40.621 DEBUG:boto:Token: None 2023-11-02T18:29:40.621 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/f 2023-11-02T18:29:40.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x3azZe4qdPJFyC4ND5yVL5S8QqA= 2023-11-02T18:29:40.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x3azZe4qdPJFyC4ND5yVL5S8QqA=', 'Content-Length': '0'} 2023-11-02T18:29:40.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba7ef2d55ef83eb3-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-11-02T18:29:40.625 DEBUG:boto:path=/ibtljn-57/f 2023-11-02T18:29:40.626 DEBUG:boto:auth_path=/ibtljn-57/f 2023-11-02T18:29:40.626 DEBUG:boto:Method: GET 2023-11-02T18:29:40.627 DEBUG:boto:Path: /ibtljn-57/f 2023-11-02T18:29:40.627 DEBUG:boto:Data: 2023-11-02T18:29:40.628 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.628 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.628 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.629 DEBUG:boto:Params: {} 2023-11-02T18:29:40.629 DEBUG:boto:Token: None 2023-11-02T18:29:40.630 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/f 2023-11-02T18:29:40.630 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C/O0N8x1DzSoNL/Gw+aD5bUDqws= 2023-11-02T18:29:40.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C/O0N8x1DzSoNL/Gw+aD5bUDqws=', 'Content-Length': '0'} 2023-11-02T18:29:40.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af1401dbd8e4d25-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.634 DEBUG:boto:path=/ibtljn-57/f 2023-11-02T18:29:40.635 DEBUG:boto:auth_path=/ibtljn-57/f 2023-11-02T18:29:40.635 DEBUG:boto:Method: GET 2023-11-02T18:29:40.636 DEBUG:boto:Path: /ibtljn-57/f 2023-11-02T18:29:40.636 DEBUG:boto:Data: 2023-11-02T18:29:40.637 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.638 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.638 DEBUG:boto:Params: {} 2023-11-02T18:29:40.639 DEBUG:boto:Token: None 2023-11-02T18:29:40.639 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/f 2023-11-02T18:29:40.640 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C/O0N8x1DzSoNL/Gw+aD5bUDqws= 2023-11-02T18:29:40.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C/O0N8x1DzSoNL/Gw+aD5bUDqws=', 'Content-Length': '0'} 2023-11-02T18:29:40.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eac7543d5dbd953c-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.643 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.644 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.644 DEBUG:boto:path=/ibtljn-57/?prefix=f 2023-11-02T18:29:40.645 DEBUG:boto:auth_path=/ibtljn-57/?prefix=f 2023-11-02T18:29:40.645 DEBUG:boto:Method: GET 2023-11-02T18:29:40.646 DEBUG:boto:Path: /ibtljn-57/?prefix=f 2023-11-02T18:29:40.646 DEBUG:boto:Data: 2023-11-02T18:29:40.647 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.647 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.647 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.648 DEBUG:boto:Params: {} 2023-11-02T18:29:40.648 DEBUG:boto:Token: None 2023-11-02T18:29:40.649 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.649 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000812dca2245ae23df-006543ea94-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.656 DEBUG:boto:b'ibtljn-57f1000falsef2023-11-02T18:26:57.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.657 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.657 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.657 DEBUG:boto:path=/ibtljn-57/?prefix=f 2023-11-02T18:29:40.658 DEBUG:boto:auth_path=/ibtljn-57/?prefix=f 2023-11-02T18:29:40.658 DEBUG:boto:Method: GET 2023-11-02T18:29:40.659 DEBUG:boto:Path: /ibtljn-57/?prefix=f 2023-11-02T18:29:40.659 DEBUG:boto:Data: 2023-11-02T18:29:40.660 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.660 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.660 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.661 DEBUG:boto:Params: {} 2023-11-02T18:29:40.661 DEBUG:boto:Token: None 2023-11-02T18:29:40.662 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.662 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cc79c06f68b0e31-006543ea94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.710 DEBUG:boto:b'ibtljn-57f1000falsef2023-11-02T18:26:57.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-11-02T18:29:40.712 DEBUG:boto:path=/ibtljn-57/g 2023-11-02T18:29:40.712 DEBUG:boto:auth_path=/ibtljn-57/g 2023-11-02T18:29:40.713 DEBUG:boto:path=/ibtljn-57/g?versionId=null 2023-11-02T18:29:40.713 DEBUG:boto:auth_path=/ibtljn-57/g?versionId=null 2023-11-02T18:29:40.713 DEBUG:boto:Method: GET 2023-11-02T18:29:40.714 DEBUG:boto:Path: /ibtljn-57/g?versionId=null 2023-11-02T18:29:40.714 DEBUG:boto:Data: 2023-11-02T18:29:40.715 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.715 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.716 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.716 DEBUG:boto:Params: {} 2023-11-02T18:29:40.717 DEBUG:boto:Token: None 2023-11-02T18:29:40.717 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/g?versionId=null 2023-11-02T18:29:40.718 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fZi3dxKUoF3Tj6MLH8fSKRbsNTk= 2023-11-02T18:29:40.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fZi3dxKUoF3Tj6MLH8fSKRbsNTk=', 'Content-Length': '0'} 2023-11-02T18:29:40.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e965a9097ce67ac9-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.722 DEBUG:boto:path=/ibtljn-57/g 2023-11-02T18:29:40.723 DEBUG:boto:auth_path=/ibtljn-57/g 2023-11-02T18:29:40.723 DEBUG:boto:path=/ibtljn-57/g?versionId=null 2023-11-02T18:29:40.723 DEBUG:boto:auth_path=/ibtljn-57/g?versionId=null 2023-11-02T18:29:40.724 DEBUG:boto:Method: GET 2023-11-02T18:29:40.724 DEBUG:boto:Path: /ibtljn-57/g?versionId=null 2023-11-02T18:29:40.725 DEBUG:boto:Data: 2023-11-02T18:29:40.725 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.726 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.726 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.726 DEBUG:boto:Params: {} 2023-11-02T18:29:40.727 DEBUG:boto:Token: None 2023-11-02T18:29:40.727 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/g?versionId=null 2023-11-02T18:29:40.728 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fZi3dxKUoF3Tj6MLH8fSKRbsNTk= 2023-11-02T18:29:40.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fZi3dxKUoF3Tj6MLH8fSKRbsNTk=', 'Content-Length': '0'} 2023-11-02T18:29:40.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0aa558e5b07bb28-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.732 DEBUG:boto:path=/ibtljn-57/g 2023-11-02T18:29:40.732 DEBUG:boto:auth_path=/ibtljn-57/g 2023-11-02T18:29:40.733 DEBUG:boto:path=/ibtljn-57/g?versionId=null 2023-11-02T18:29:40.733 DEBUG:boto:auth_path=/ibtljn-57/g?versionId=null 2023-11-02T18:29:40.734 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.734 DEBUG:boto:Path: /ibtljn-57/g?versionId=null 2023-11-02T18:29:40.734 DEBUG:boto:Data: 2023-11-02T18:29:40.735 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.735 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.736 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.736 DEBUG:boto:Params: {} 2023-11-02T18:29:40.736 DEBUG:boto:Token: None 2023-11-02T18:29:40.737 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/g?versionId=null 2023-11-02T18:29:40.737 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xzWlVCQr/+WwtnKuAQUcFFI289E= 2023-11-02T18:29:40.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xzWlVCQr/+WwtnKuAQUcFFI289E=', 'Content-Length': '0'} 2023-11-02T18:29:40.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e6f2265eabbed6a-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.741 DEBUG:boto:path=/ibtljn-57/g 2023-11-02T18:29:40.742 DEBUG:boto:auth_path=/ibtljn-57/g 2023-11-02T18:29:40.742 DEBUG:boto:path=/ibtljn-57/g?versionId=null 2023-11-02T18:29:40.743 DEBUG:boto:auth_path=/ibtljn-57/g?versionId=null 2023-11-02T18:29:40.743 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.743 DEBUG:boto:Path: /ibtljn-57/g?versionId=null 2023-11-02T18:29:40.744 DEBUG:boto:Data: 2023-11-02T18:29:40.744 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.745 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.745 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.745 DEBUG:boto:Params: {} 2023-11-02T18:29:40.746 DEBUG:boto:Token: None 2023-11-02T18:29:40.746 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/g?versionId=null 2023-11-02T18:29:40.747 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xzWlVCQr/+WwtnKuAQUcFFI289E= 2023-11-02T18:29:40.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xzWlVCQr/+WwtnKuAQUcFFI289E=', 'Content-Length': '0'} 2023-11-02T18:29:40.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000846e73bacd293d6-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-11-02T18:29:40.751 DEBUG:boto:path=/ibtljn-57/g 2023-11-02T18:29:40.751 DEBUG:boto:auth_path=/ibtljn-57/g 2023-11-02T18:29:40.752 DEBUG:boto:path=/ibtljn-57/g?versionId=null 2023-11-02T18:29:40.752 DEBUG:boto:auth_path=/ibtljn-57/g?versionId=null 2023-11-02T18:29:40.753 DEBUG:boto:Method: GET 2023-11-02T18:29:40.753 DEBUG:boto:Path: /ibtljn-57/g?versionId=null 2023-11-02T18:29:40.754 DEBUG:boto:Data: 2023-11-02T18:29:40.754 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.755 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.755 DEBUG:boto:Params: {} 2023-11-02T18:29:40.756 DEBUG:boto:Token: None 2023-11-02T18:29:40.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/g?versionId=null 2023-11-02T18:29:40.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fZi3dxKUoF3Tj6MLH8fSKRbsNTk= 2023-11-02T18:29:40.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fZi3dxKUoF3Tj6MLH8fSKRbsNTk=', 'Content-Length': '0'} 2023-11-02T18:29:40.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ca312d719c9d2f2-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.761 DEBUG:boto:path=/ibtljn-57/g 2023-11-02T18:29:40.762 DEBUG:boto:auth_path=/ibtljn-57/g 2023-11-02T18:29:40.762 DEBUG:boto:path=/ibtljn-57/g?versionId=null 2023-11-02T18:29:40.763 DEBUG:boto:auth_path=/ibtljn-57/g?versionId=null 2023-11-02T18:29:40.763 DEBUG:boto:Method: GET 2023-11-02T18:29:40.763 DEBUG:boto:Path: /ibtljn-57/g?versionId=null 2023-11-02T18:29:40.764 DEBUG:boto:Data: 2023-11-02T18:29:40.764 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.765 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.765 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.765 DEBUG:boto:Params: {} 2023-11-02T18:29:40.766 DEBUG:boto:Token: None 2023-11-02T18:29:40.766 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/g?versionId=null 2023-11-02T18:29:40.767 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fZi3dxKUoF3Tj6MLH8fSKRbsNTk= 2023-11-02T18:29:40.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fZi3dxKUoF3Tj6MLH8fSKRbsNTk=', 'Content-Length': '0'} 2023-11-02T18:29:40.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df6917c4db8e2398-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.771 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.771 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.772 DEBUG:boto:path=/ibtljn-57/?prefix=g 2023-11-02T18:29:40.772 DEBUG:boto:auth_path=/ibtljn-57/?prefix=g 2023-11-02T18:29:40.773 DEBUG:boto:Method: GET 2023-11-02T18:29:40.773 DEBUG:boto:Path: /ibtljn-57/?prefix=g 2023-11-02T18:29:40.774 DEBUG:boto:Data: 2023-11-02T18:29:40.774 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.775 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.775 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.775 DEBUG:boto:Params: {} 2023-11-02T18:29:40.776 DEBUG:boto:Token: None 2023-11-02T18:29:40.776 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.777 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c04831cc065d1eb0-006543ea94-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.826 DEBUG:boto:b'ibtljn-57g1000falseg2023-11-02T18:26:57.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.827 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.828 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.828 DEBUG:boto:path=/ibtljn-57/?prefix=g 2023-11-02T18:29:40.829 DEBUG:boto:auth_path=/ibtljn-57/?prefix=g 2023-11-02T18:29:40.829 DEBUG:boto:Method: GET 2023-11-02T18:29:40.830 DEBUG:boto:Path: /ibtljn-57/?prefix=g 2023-11-02T18:29:40.830 DEBUG:boto:Data: 2023-11-02T18:29:40.830 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.831 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.831 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.832 DEBUG:boto:Params: {} 2023-11-02T18:29:40.832 DEBUG:boto:Token: None 2023-11-02T18:29:40.833 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.833 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000162de44ee518c43c-006543ea94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.838 DEBUG:boto:b'ibtljn-57g1000falseg2023-11-02T18:26:57.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.839 DEBUG:boto:path=/ibtljn-57/g 2023-11-02T18:29:40.839 DEBUG:boto:auth_path=/ibtljn-57/g 2023-11-02T18:29:40.839 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.840 DEBUG:boto:Path: /ibtljn-57/g 2023-11-02T18:29:40.840 DEBUG:boto:Data: 2023-11-02T18:29:40.841 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.841 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.842 DEBUG:boto:Params: {} 2023-11-02T18:29:40.842 DEBUG:boto:Token: None 2023-11-02T18:29:40.843 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/g 2023-11-02T18:29:40.843 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d7jJaK6D6tD97M/vsFhx0ct7QOM= 2023-11-02T18:29:40.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d7jJaK6D6tD97M/vsFhx0ct7QOM=', 'Content-Length': '0'} 2023-11-02T18:29:40.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000423253a9e35d35fd-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.847 DEBUG:boto:path=/ibtljn-57/g 2023-11-02T18:29:40.848 DEBUG:boto:auth_path=/ibtljn-57/g 2023-11-02T18:29:40.848 DEBUG:boto:Method: HEAD 2023-11-02T18:29:40.848 DEBUG:boto:Path: /ibtljn-57/g 2023-11-02T18:29:40.849 DEBUG:boto:Data: 2023-11-02T18:29:40.849 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.850 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.850 DEBUG:boto:Params: {} 2023-11-02T18:29:40.851 DEBUG:boto:Token: None 2023-11-02T18:29:40.851 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/g 2023-11-02T18:29:40.852 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d7jJaK6D6tD97M/vsFhx0ct7QOM= 2023-11-02T18:29:40.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d7jJaK6D6tD97M/vsFhx0ct7QOM=', 'Content-Length': '0'} 2023-11-02T18:29:40.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004cafdf6c2a2d351-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-11-02T18:29:40.856 DEBUG:boto:path=/ibtljn-57/g 2023-11-02T18:29:40.857 DEBUG:boto:auth_path=/ibtljn-57/g 2023-11-02T18:29:40.857 DEBUG:boto:Method: GET 2023-11-02T18:29:40.858 DEBUG:boto:Path: /ibtljn-57/g 2023-11-02T18:29:40.858 DEBUG:boto:Data: 2023-11-02T18:29:40.859 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.859 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.859 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.860 DEBUG:boto:Params: {} 2023-11-02T18:29:40.860 DEBUG:boto:Token: None 2023-11-02T18:29:40.861 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/g 2023-11-02T18:29:40.861 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:okWWZ7R/ImHktd4uOlb91Tzgzs8= 2023-11-02T18:29:40.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:okWWZ7R/ImHktd4uOlb91Tzgzs8=', 'Content-Length': '0'} 2023-11-02T18:29:40.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b05a8ef6515838d6-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.865 DEBUG:boto:path=/ibtljn-57/g 2023-11-02T18:29:40.866 DEBUG:boto:auth_path=/ibtljn-57/g 2023-11-02T18:29:40.866 DEBUG:boto:Method: GET 2023-11-02T18:29:40.867 DEBUG:boto:Path: /ibtljn-57/g 2023-11-02T18:29:40.867 DEBUG:boto:Data: 2023-11-02T18:29:40.867 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.868 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.868 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.869 DEBUG:boto:Params: {} 2023-11-02T18:29:40.869 DEBUG:boto:Token: None 2023-11-02T18:29:40.870 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/g 2023-11-02T18:29:40.870 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:okWWZ7R/ImHktd4uOlb91Tzgzs8= 2023-11-02T18:29:40.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:okWWZ7R/ImHktd4uOlb91Tzgzs8=', 'Content-Length': '0'} 2023-11-02T18:29:40.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000877b2ff9b32ae121-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.914 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.915 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.916 DEBUG:boto:path=/ibtljn-57/?prefix=g 2023-11-02T18:29:40.916 DEBUG:boto:auth_path=/ibtljn-57/?prefix=g 2023-11-02T18:29:40.916 DEBUG:boto:Method: GET 2023-11-02T18:29:40.917 DEBUG:boto:Path: /ibtljn-57/?prefix=g 2023-11-02T18:29:40.917 DEBUG:boto:Data: 2023-11-02T18:29:40.918 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.919 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.919 DEBUG:boto:Params: {} 2023-11-02T18:29:40.920 DEBUG:boto:Token: None 2023-11-02T18:29:40.920 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.921 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c0c78281d17e40a-006543ea94-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.928 DEBUG:boto:b'ibtljn-57g1000falseg2023-11-02T18:26:57.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.929 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:40.929 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:40.930 DEBUG:boto:path=/ibtljn-57/?prefix=g 2023-11-02T18:29:40.930 DEBUG:boto:auth_path=/ibtljn-57/?prefix=g 2023-11-02T18:29:40.930 DEBUG:boto:Method: GET 2023-11-02T18:29:40.931 DEBUG:boto:Path: /ibtljn-57/?prefix=g 2023-11-02T18:29:40.931 DEBUG:boto:Data: 2023-11-02T18:29:40.932 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.932 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.933 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.933 DEBUG:boto:Params: {} 2023-11-02T18:29:40.933 DEBUG:boto:Token: None 2023-11-02T18:29:40.934 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/ 2023-11-02T18:29:40.934 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc= 2023-11-02T18:29:40.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I0hVcjv+eNax99BEDXSqN91/7Gc=', 'Content-Length': '0'} 2023-11-02T18:29:40.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083358ce3d4957663-006543ea94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.939 DEBUG:boto:b'ibtljn-57g1000falseg2023-11-02T18:26:57.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:40.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-11-02T18:29:40.940 DEBUG:boto:path=/ibtljn-57/h 2023-11-02T18:29:40.940 DEBUG:boto:auth_path=/ibtljn-57/h 2023-11-02T18:29:40.941 DEBUG:boto:path=/ibtljn-57/h?versionId=null 2023-11-02T18:29:40.941 DEBUG:boto:auth_path=/ibtljn-57/h?versionId=null 2023-11-02T18:29:40.942 DEBUG:boto:Method: GET 2023-11-02T18:29:40.942 DEBUG:boto:Path: /ibtljn-57/h?versionId=null 2023-11-02T18:29:40.943 DEBUG:boto:Data: 2023-11-02T18:29:40.943 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.944 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:40.944 DEBUG:boto:Port: 8001 2023-11-02T18:29:40.944 DEBUG:boto:Params: {} 2023-11-02T18:29:40.945 DEBUG:boto:Token: None 2023-11-02T18:29:40.945 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/h?versionId=null 2023-11-02T18:29:40.946 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cB4SNEWoBS6AMYRCOe/ZWUEAs7k= 2023-11-02T18:29:40.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cB4SNEWoBS6AMYRCOe/ZWUEAs7k=', 'Content-Length': '0'} 2023-11-02T18:29:40.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f6ef5ad3e092d0-006543ea94-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:40.990 DEBUG:boto:path=/ibtljn-57/h 2023-11-02T18:29:40.991 DEBUG:boto:auth_path=/ibtljn-57/h 2023-11-02T18:29:40.991 DEBUG:boto:path=/ibtljn-57/h?versionId=null 2023-11-02T18:29:40.992 DEBUG:boto:auth_path=/ibtljn-57/h?versionId=null 2023-11-02T18:29:40.992 DEBUG:boto:Method: GET 2023-11-02T18:29:40.993 DEBUG:boto:Path: /ibtljn-57/h?versionId=null 2023-11-02T18:29:40.993 DEBUG:boto:Data: 2023-11-02T18:29:40.994 DEBUG:boto:Headers: {} 2023-11-02T18:29:40.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:40.995 DEBUG:boto:Port: 8000 2023-11-02T18:29:40.995 DEBUG:boto:Params: {} 2023-11-02T18:29:40.995 DEBUG:boto:Token: None 2023-11-02T18:29:40.996 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:40 GMT /ibtljn-57/h?versionId=null 2023-11-02T18:29:40.996 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cB4SNEWoBS6AMYRCOe/ZWUEAs7k= 2023-11-02T18:29:40.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cB4SNEWoBS6AMYRCOe/ZWUEAs7k=', 'Content-Length': '0'} 2023-11-02T18:29:41.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f89b9355797b507-006543ea94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.001 DEBUG:boto:path=/ibtljn-57/h 2023-11-02T18:29:41.001 DEBUG:boto:auth_path=/ibtljn-57/h 2023-11-02T18:29:41.002 DEBUG:boto:path=/ibtljn-57/h?versionId=null 2023-11-02T18:29:41.002 DEBUG:boto:auth_path=/ibtljn-57/h?versionId=null 2023-11-02T18:29:41.002 DEBUG:boto:Method: HEAD 2023-11-02T18:29:41.003 DEBUG:boto:Path: /ibtljn-57/h?versionId=null 2023-11-02T18:29:41.003 DEBUG:boto:Data: 2023-11-02T18:29:41.005 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.005 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:41.006 DEBUG:boto:Port: 8001 2023-11-02T18:29:41.006 DEBUG:boto:Params: {} 2023-11-02T18:29:41.007 DEBUG:boto:Token: None 2023-11-02T18:29:41.007 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-57/h?versionId=null 2023-11-02T18:29:41.008 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WN0x0/sfGyALXbAIx6Ji3GIK0mg= 2023-11-02T18:29:41.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WN0x0/sfGyALXbAIx6Ji3GIK0mg=', 'Content-Length': '0'} 2023-11-02T18:29:41.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e8a52819c34c08e-006543ea95-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.012 DEBUG:boto:path=/ibtljn-57/h 2023-11-02T18:29:41.012 DEBUG:boto:auth_path=/ibtljn-57/h 2023-11-02T18:29:41.013 DEBUG:boto:path=/ibtljn-57/h?versionId=null 2023-11-02T18:29:41.013 DEBUG:boto:auth_path=/ibtljn-57/h?versionId=null 2023-11-02T18:29:41.014 DEBUG:boto:Method: HEAD 2023-11-02T18:29:41.014 DEBUG:boto:Path: /ibtljn-57/h?versionId=null 2023-11-02T18:29:41.014 DEBUG:boto:Data: 2023-11-02T18:29:41.015 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.015 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:41.016 DEBUG:boto:Port: 8000 2023-11-02T18:29:41.016 DEBUG:boto:Params: {} 2023-11-02T18:29:41.017 DEBUG:boto:Token: None 2023-11-02T18:29:41.017 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-57/h?versionId=null 2023-11-02T18:29:41.018 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WN0x0/sfGyALXbAIx6Ji3GIK0mg= 2023-11-02T18:29:41.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WN0x0/sfGyALXbAIx6Ji3GIK0mg=', 'Content-Length': '0'} 2023-11-02T18:29:41.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d73303cdcfb4f0c6-006543ea95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-11-02T18:29:41.022 DEBUG:boto:path=/ibtljn-57/h 2023-11-02T18:29:41.022 DEBUG:boto:auth_path=/ibtljn-57/h 2023-11-02T18:29:41.023 DEBUG:boto:path=/ibtljn-57/h?versionId=null 2023-11-02T18:29:41.023 DEBUG:boto:auth_path=/ibtljn-57/h?versionId=null 2023-11-02T18:29:41.024 DEBUG:boto:Method: GET 2023-11-02T18:29:41.024 DEBUG:boto:Path: /ibtljn-57/h?versionId=null 2023-11-02T18:29:41.024 DEBUG:boto:Data: 2023-11-02T18:29:41.025 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.025 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:41.026 DEBUG:boto:Port: 8001 2023-11-02T18:29:41.026 DEBUG:boto:Params: {} 2023-11-02T18:29:41.027 DEBUG:boto:Token: None 2023-11-02T18:29:41.027 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-57/h?versionId=null 2023-11-02T18:29:41.028 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8w4Ow2EDgwHtT3f6xLmhctWDISA= 2023-11-02T18:29:41.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8w4Ow2EDgwHtT3f6xLmhctWDISA=', 'Content-Length': '0'} 2023-11-02T18:29:41.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5bc3b2f5ab6386d-006543ea95-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.032 DEBUG:boto:path=/ibtljn-57/h 2023-11-02T18:29:41.033 DEBUG:boto:auth_path=/ibtljn-57/h 2023-11-02T18:29:41.033 DEBUG:boto:path=/ibtljn-57/h?versionId=null 2023-11-02T18:29:41.034 DEBUG:boto:auth_path=/ibtljn-57/h?versionId=null 2023-11-02T18:29:41.034 DEBUG:boto:Method: GET 2023-11-02T18:29:41.035 DEBUG:boto:Path: /ibtljn-57/h?versionId=null 2023-11-02T18:29:41.035 DEBUG:boto:Data: 2023-11-02T18:29:41.036 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.036 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:41.036 DEBUG:boto:Port: 8000 2023-11-02T18:29:41.037 DEBUG:boto:Params: {} 2023-11-02T18:29:41.037 DEBUG:boto:Token: None 2023-11-02T18:29:41.038 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-57/h?versionId=null 2023-11-02T18:29:41.039 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8w4Ow2EDgwHtT3f6xLmhctWDISA= 2023-11-02T18:29:41.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8w4Ow2EDgwHtT3f6xLmhctWDISA=', 'Content-Length': '0'} 2023-11-02T18:29:41.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075ff0dcb9fcae099-006543ea95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.042 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:41.043 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:41.043 DEBUG:boto:path=/ibtljn-57/?prefix=h 2023-11-02T18:29:41.044 DEBUG:boto:auth_path=/ibtljn-57/?prefix=h 2023-11-02T18:29:41.044 DEBUG:boto:Method: GET 2023-11-02T18:29:41.044 DEBUG:boto:Path: /ibtljn-57/?prefix=h 2023-11-02T18:29:41.045 DEBUG:boto:Data: 2023-11-02T18:29:41.045 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:41.046 DEBUG:boto:Port: 8001 2023-11-02T18:29:41.046 DEBUG:boto:Params: {} 2023-11-02T18:29:41.047 DEBUG:boto:Token: None 2023-11-02T18:29:41.047 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-57/ 2023-11-02T18:29:41.048 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VpgSSmIm+noQwZNMs5CzXs/itYY= 2023-11-02T18:29:41.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VpgSSmIm+noQwZNMs5CzXs/itYY=', 'Content-Length': '0'} 2023-11-02T18:29:41.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ff00908f0039031-006543ea95-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.102 DEBUG:boto:b'ibtljn-57h1000falseh2023-11-02T18:26:57.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:41.103 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:41.104 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:41.104 DEBUG:boto:path=/ibtljn-57/?prefix=h 2023-11-02T18:29:41.105 DEBUG:boto:auth_path=/ibtljn-57/?prefix=h 2023-11-02T18:29:41.105 DEBUG:boto:Method: GET 2023-11-02T18:29:41.105 DEBUG:boto:Path: /ibtljn-57/?prefix=h 2023-11-02T18:29:41.106 DEBUG:boto:Data: 2023-11-02T18:29:41.106 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.107 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:41.107 DEBUG:boto:Port: 8000 2023-11-02T18:29:41.108 DEBUG:boto:Params: {} 2023-11-02T18:29:41.108 DEBUG:boto:Token: None 2023-11-02T18:29:41.109 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-57/ 2023-11-02T18:29:41.109 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VpgSSmIm+noQwZNMs5CzXs/itYY= 2023-11-02T18:29:41.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VpgSSmIm+noQwZNMs5CzXs/itYY=', 'Content-Length': '0'} 2023-11-02T18:29:41.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000524cc614938f1a70-006543ea95-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.114 DEBUG:boto:b'ibtljn-57h1000falseh2023-11-02T18:26:57.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:41.114 DEBUG:boto:path=/ibtljn-57/h 2023-11-02T18:29:41.115 DEBUG:boto:auth_path=/ibtljn-57/h 2023-11-02T18:29:41.115 DEBUG:boto:Method: HEAD 2023-11-02T18:29:41.116 DEBUG:boto:Path: /ibtljn-57/h 2023-11-02T18:29:41.116 DEBUG:boto:Data: 2023-11-02T18:29:41.116 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.117 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:41.117 DEBUG:boto:Port: 8001 2023-11-02T18:29:41.118 DEBUG:boto:Params: {} 2023-11-02T18:29:41.118 DEBUG:boto:Token: None 2023-11-02T18:29:41.119 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-57/h 2023-11-02T18:29:41.119 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7S14iIt06sH6HKWfnqZpuXTe24k= 2023-11-02T18:29:41.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7S14iIt06sH6HKWfnqZpuXTe24k=', 'Content-Length': '0'} 2023-11-02T18:29:41.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac88a7884d3dc379-006543ea95-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.123 DEBUG:boto:path=/ibtljn-57/h 2023-11-02T18:29:41.124 DEBUG:boto:auth_path=/ibtljn-57/h 2023-11-02T18:29:41.124 DEBUG:boto:Method: HEAD 2023-11-02T18:29:41.125 DEBUG:boto:Path: /ibtljn-57/h 2023-11-02T18:29:41.125 DEBUG:boto:Data: 2023-11-02T18:29:41.125 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.126 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:41.126 DEBUG:boto:Port: 8000 2023-11-02T18:29:41.127 DEBUG:boto:Params: {} 2023-11-02T18:29:41.127 DEBUG:boto:Token: None 2023-11-02T18:29:41.128 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-57/h 2023-11-02T18:29:41.128 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7S14iIt06sH6HKWfnqZpuXTe24k= 2023-11-02T18:29:41.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7S14iIt06sH6HKWfnqZpuXTe24k=', 'Content-Length': '0'} 2023-11-02T18:29:41.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a360fdb75be0b0ab-006543ea95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-11-02T18:29:41.132 DEBUG:boto:path=/ibtljn-57/h 2023-11-02T18:29:41.133 DEBUG:boto:auth_path=/ibtljn-57/h 2023-11-02T18:29:41.133 DEBUG:boto:Method: GET 2023-11-02T18:29:41.133 DEBUG:boto:Path: /ibtljn-57/h 2023-11-02T18:29:41.134 DEBUG:boto:Data: 2023-11-02T18:29:41.134 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.135 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:41.135 DEBUG:boto:Port: 8001 2023-11-02T18:29:41.136 DEBUG:boto:Params: {} 2023-11-02T18:29:41.136 DEBUG:boto:Token: None 2023-11-02T18:29:41.136 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-57/h 2023-11-02T18:29:41.137 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C2z79aonDxMJmzEB4OHI1x0meDk= 2023-11-02T18:29:41.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C2z79aonDxMJmzEB4OHI1x0meDk=', 'Content-Length': '0'} 2023-11-02T18:29:41.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-57:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbce7bd2930fe3e3-006543ea95-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.141 DEBUG:boto:path=/ibtljn-57/h 2023-11-02T18:29:41.142 DEBUG:boto:auth_path=/ibtljn-57/h 2023-11-02T18:29:41.142 DEBUG:boto:Method: GET 2023-11-02T18:29:41.142 DEBUG:boto:Path: /ibtljn-57/h 2023-11-02T18:29:41.143 DEBUG:boto:Data: 2023-11-02T18:29:41.143 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.144 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:41.144 DEBUG:boto:Port: 8000 2023-11-02T18:29:41.145 DEBUG:boto:Params: {} 2023-11-02T18:29:41.145 DEBUG:boto:Token: None 2023-11-02T18:29:41.146 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-57/h 2023-11-02T18:29:41.146 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C2z79aonDxMJmzEB4OHI1x0meDk= 2023-11-02T18:29:41.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C2z79aonDxMJmzEB4OHI1x0meDk=', 'Content-Length': '0'} 2023-11-02T18:29:41.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:26:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d45983242fff005b-006543ea95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.190 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:41.191 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:41.191 DEBUG:boto:path=/ibtljn-57/?prefix=h 2023-11-02T18:29:41.192 DEBUG:boto:auth_path=/ibtljn-57/?prefix=h 2023-11-02T18:29:41.192 DEBUG:boto:Method: GET 2023-11-02T18:29:41.193 DEBUG:boto:Path: /ibtljn-57/?prefix=h 2023-11-02T18:29:41.193 DEBUG:boto:Data: 2023-11-02T18:29:41.193 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.194 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:41.194 DEBUG:boto:Port: 8001 2023-11-02T18:29:41.195 DEBUG:boto:Params: {} 2023-11-02T18:29:41.195 DEBUG:boto:Token: None 2023-11-02T18:29:41.196 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-57/ 2023-11-02T18:29:41.196 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VpgSSmIm+noQwZNMs5CzXs/itYY= 2023-11-02T18:29:41.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VpgSSmIm+noQwZNMs5CzXs/itYY=', 'Content-Length': '0'} 2023-11-02T18:29:41.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c10996e334ff1d1c-006543ea95-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.203 DEBUG:boto:b'ibtljn-57h1000falseh2023-11-02T18:26:57.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:41.204 DEBUG:boto:path=/ibtljn-57/ 2023-11-02T18:29:41.205 DEBUG:boto:auth_path=/ibtljn-57/ 2023-11-02T18:29:41.205 DEBUG:boto:path=/ibtljn-57/?prefix=h 2023-11-02T18:29:41.206 DEBUG:boto:auth_path=/ibtljn-57/?prefix=h 2023-11-02T18:29:41.206 DEBUG:boto:Method: GET 2023-11-02T18:29:41.207 DEBUG:boto:Path: /ibtljn-57/?prefix=h 2023-11-02T18:29:41.207 DEBUG:boto:Data: 2023-11-02T18:29:41.207 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.208 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:41.208 DEBUG:boto:Port: 8000 2023-11-02T18:29:41.209 DEBUG:boto:Params: {} 2023-11-02T18:29:41.209 DEBUG:boto:Token: None 2023-11-02T18:29:41.210 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-57/ 2023-11-02T18:29:41.210 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VpgSSmIm+noQwZNMs5CzXs/itYY= 2023-11-02T18:29:41.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VpgSSmIm+noQwZNMs5CzXs/itYY=', 'Content-Length': '0'} 2023-11-02T18:29:41.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e8d9b15ab4b254a-006543ea95-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.215 DEBUG:boto:b'ibtljn-57h1000falseh2023-11-02T18:26:57.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:29:41.216 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-57 zones={a2, a1} 2023-11-02T18:29:41.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2023-11-02T18:29:41.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:29:41.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:29:41.218 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:29:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:29:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:29:41.321 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:29:41.349 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949617.430527_449.1", 2023-11-02T18:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:26:57.430527Z" 2023-11-02T18:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:29:41.354 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949617.430527_449.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:29:41.355 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:29:41.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:29:41.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:29:41.356 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:29:41.441 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:29:41.442 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:29:41.498 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:41.492+0000 7fd698abdf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:29:41.500 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:41.492+0000 7fd698abdf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:29:41.527 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:29:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:29:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:29:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:29:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:29:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:29:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:29:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:29:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:29:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:29:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:29:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:29:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:29:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:29:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:29:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:29:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:29:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:29:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:29:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:29:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:29:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:29:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:29:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:29:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:29:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:29:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:29:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:29:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:29:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:29:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:29:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949617.430527_449.1", 2023-11-02T18:29:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:29:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:29:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:29:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:26:57.430527Z", 2023-11-02T18:29:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:29:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:29:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:29:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:29:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:29:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:29:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:29:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:29:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:29:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:29:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:29:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:29:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:29:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:29:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:29:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:29:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:29:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:29:41Z" 2023-11-02T18:29:41.543 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:29:41.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949617.430527_449.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:26:57.430527Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:29:41Z" } 2023-11-02T18:29:41.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949617.430527_449.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:26:57.430527Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:29:41.545 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949617.430527_449.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:29:41.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949617.430527_449.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:29:41.546 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:29:41.546 DEBUG:boto:path=/ 2023-11-02T18:29:41.546 DEBUG:boto:auth_path=/ 2023-11-02T18:29:41.547 DEBUG:boto:Method: GET 2023-11-02T18:29:41.547 DEBUG:boto:Path: / 2023-11-02T18:29:41.548 DEBUG:boto:Data: 2023-11-02T18:29:41.548 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.548 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:41.549 DEBUG:boto:Port: 8000 2023-11-02T18:29:41.549 DEBUG:boto:Params: {} 2023-11-02T18:29:41.550 DEBUG:boto:Token: None 2023-11-02T18:29:41.550 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:41 GMT / 2023-11-02T18:29:41.551 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1D78UgbgWTsLh/2fezcLzLx+onc= 2023-11-02T18:29:41.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1D78UgbgWTsLh/2fezcLzLx+onc=', 'Content-Length': '0'} 2023-11-02T18:29:41.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088c44bd1fb8cb888-006543ea95-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.561 DEBUG:boto:path=/ 2023-11-02T18:29:41.562 DEBUG:boto:auth_path=/ 2023-11-02T18:29:41.562 DEBUG:boto:Method: GET 2023-11-02T18:29:41.563 DEBUG:boto:Path: / 2023-11-02T18:29:41.563 DEBUG:boto:Data: 2023-11-02T18:29:41.564 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:29:41.564 DEBUG:boto:Port: 8001 2023-11-02T18:29:41.565 DEBUG:boto:Params: {} 2023-11-02T18:29:41.565 DEBUG:boto:Token: None 2023-11-02T18:29:41.566 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:29:41 GMT / 2023-11-02T18:29:41.566 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1D78UgbgWTsLh/2fezcLzLx+onc= 2023-11-02T18:29:41.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1D78UgbgWTsLh/2fezcLzLx+onc=', 'Content-Length': '0'} 2023-11-02T18:29:41.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afb395942f76ba31-006543ea95-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2023-11-02T18:29:41.574 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:29:41.574 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:29:41.575 DEBUG:boto:Method: PUT 2023-11-02T18:29:41.575 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:29:41.576 DEBUG:boto:Data: 2023-11-02T18:29:41.576 DEBUG:boto:Headers: {} 2023-11-02T18:29:41.577 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:41.577 DEBUG:boto:Port: 8000 2023-11-02T18:29:41.578 DEBUG:boto:Params: {} 2023-11-02T18:29:41.578 DEBUG:boto:Token: None 2023-11-02T18:29:41.579 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:29:41 GMT /ibtljn-58/ 2023-11-02T18:29:41.580 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LbgLn8X9ECr6Ie2AIrS2tNTAkrc= 2023-11-02T18:29:41.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LbgLn8X9ECr6Ie2AIrS2tNTAkrc=', 'Content-Length': '0'} 2023-11-02T18:29:41.606 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c42899bb7a6abc65-006543ea95-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:41.607 DEBUG:rgw_multi.tests:created bucket=ibtljn-58 2023-11-02T18:29:41.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:29:41.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:29:41.609 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:29:41.652 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:29:41.653 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:29:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:29:41.731 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:29:41.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.731 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:29:41.732 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:29:41.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.732 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:29:41.733 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:29:41.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.733 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:29:41.733 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:29:41.602188Z" 2023-11-02T18:29:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:29:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:29:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:29:41.735 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:29:41.736 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:29:41.736 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:29:41.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:29:41.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:29:41.739 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:29:41.783 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:29:41.784 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:29:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:41.836+0000 7ff5606c6f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:29:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:41.836+0000 7ff5606c6f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:29:41.871 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:29:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:29:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:29:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:29:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:29:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:29:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:29:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:29:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:29:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:29:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:29:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:29:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:29:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:29:41.876 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:29:41.876 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:29:41.876 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:29:41.876 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:29:41.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.877 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:29:41.877 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:29:41.877 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:29:41.878 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:29:41.878 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:29:41.878 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:29:41.878 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:29:41.879 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:29:41.879 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:29:41.879 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:29:41.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.880 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:29:41.880 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:29:41.880 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:29:41.880 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:29:41.881 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:29:41.881 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:29:41.881 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:29:41.881 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:29:41.882 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:29:41.882 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:29:41.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:29:41.883 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:29:41.883 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:29:41.883 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:29:41.883 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:29:41.883 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:29:41.884 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:29:41.884 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:29:41.884 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:29:41.884 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:29:41.884 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:29:41.885 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:29:41.885 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:29:41.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.885 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:29:41.886 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:29:41.886 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:29:41.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:29:41.886 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:29:41Z" 2023-11-02T18:29:41.887 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:29:41.889 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:29:41Z" } 2023-11-02T18:29:41.890 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:29:41.891 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:29:41.892 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:29:41.893 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:29:41.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:29:41.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:29:41.895 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:29:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:29:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:29:42.060 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:29:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: "objs": [ 2023-11-02T18:29:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000001", 2023-11-02T18:29:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000005", 2023-11-02T18:29:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000012", 2023-11-02T18:29:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000004", 2023-11-02T18:29:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000011", 2023-11-02T18:29:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000000", 2023-11-02T18:29:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000010", 2023-11-02T18:29:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000003", 2023-11-02T18:29:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000013", 2023-11-02T18:29:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000002", 2023-11-02T18:29:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000006" 2023-11-02T18:29:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:29:42.064 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:29:42.065 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:29:42.066 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:29:42.066 DEBUG:boto:Method: HEAD 2023-11-02T18:29:42.067 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:29:42.067 DEBUG:boto:Data: 2023-11-02T18:29:42.068 DEBUG:boto:Headers: {} 2023-11-02T18:29:42.068 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.069 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.069 DEBUG:boto:Params: {} 2023-11-02T18:29:42.069 DEBUG:boto:Token: None 2023-11-02T18:29:42.070 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/ 2023-11-02T18:29:42.070 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I= 2023-11-02T18:29:42.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I=', 'Content-Length': '0'} 2023-11-02T18:29:42.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004abacfe132618d49-006543ea96-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.076 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:29:42.076 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:29:42.076 DEBUG:boto:Method: PUT 2023-11-02T18:29:42.077 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:29:42.077 DEBUG:boto:Data: 2023-11-02T18:29:42.078 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:29:42.078 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.079 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.079 DEBUG:boto:Params: {} 2023-11-02T18:29:42.079 DEBUG:boto:Token: None 2023-11-02T18:29:42.080 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:29:42.080 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+kyc/vFObqF53v4JyOxD4VvJcE= 2023-11-02T18:29:42.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+kyc/vFObqF53v4JyOxD4VvJcE='} 2023-11-02T18:29:42.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1d7c8a8f4119186-006543ea96-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.127 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:29:42.128 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:29:42.128 DEBUG:boto:Method: HEAD 2023-11-02T18:29:42.129 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:29:42.129 DEBUG:boto:Data: 2023-11-02T18:29:42.129 DEBUG:boto:Headers: {} 2023-11-02T18:29:42.130 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.130 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.131 DEBUG:boto:Params: {} 2023-11-02T18:29:42.131 DEBUG:boto:Token: None 2023-11-02T18:29:42.132 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/ 2023-11-02T18:29:42.132 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I= 2023-11-02T18:29:42.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I=', 'Content-Length': '0'} 2023-11-02T18:29:42.137 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ee67082da2ff036-006543ea96-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.138 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:29:42.138 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:29:42.138 DEBUG:boto:Method: PUT 2023-11-02T18:29:42.139 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:29:42.139 DEBUG:boto:Data: 2023-11-02T18:29:42.140 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:29:42.140 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.141 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.141 DEBUG:boto:Params: {} 2023-11-02T18:29:42.141 DEBUG:boto:Token: None 2023-11-02T18:29:42.145 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:29:42.146 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LvUR20APzefGflYnvFJUaJopYW0= 2023-11-02T18:29:42.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LvUR20APzefGflYnvFJUaJopYW0='} 2023-11-02T18:29:42.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ff299b360c8b166-006543ea96-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.159 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:29:42.160 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:29:42.160 DEBUG:boto:Method: HEAD 2023-11-02T18:29:42.161 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:29:42.161 DEBUG:boto:Data: 2023-11-02T18:29:42.162 DEBUG:boto:Headers: {} 2023-11-02T18:29:42.162 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.162 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.163 DEBUG:boto:Params: {} 2023-11-02T18:29:42.163 DEBUG:boto:Token: None 2023-11-02T18:29:42.164 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/ 2023-11-02T18:29:42.164 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I= 2023-11-02T18:29:42.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I=', 'Content-Length': '0'} 2023-11-02T18:29:42.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005491c17f0827490d-006543ea96-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.169 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:29:42.169 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:29:42.170 DEBUG:boto:Method: PUT 2023-11-02T18:29:42.170 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:29:42.171 DEBUG:boto:Data: 2023-11-02T18:29:42.171 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:29:42.172 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.172 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.172 DEBUG:boto:Params: {} 2023-11-02T18:29:42.173 DEBUG:boto:Token: None 2023-11-02T18:29:42.173 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:29:42.174 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vimhpOtKZl6PGFl5PVDp4bIcmYY= 2023-11-02T18:29:42.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vimhpOtKZl6PGFl5PVDp4bIcmYY='} 2023-11-02T18:29:42.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0b8e43b323eec83-006543ea96-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.219 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:29:42.220 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:29:42.220 DEBUG:boto:Method: HEAD 2023-11-02T18:29:42.220 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:29:42.221 DEBUG:boto:Data: 2023-11-02T18:29:42.221 DEBUG:boto:Headers: {} 2023-11-02T18:29:42.222 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.222 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.222 DEBUG:boto:Params: {} 2023-11-02T18:29:42.223 DEBUG:boto:Token: None 2023-11-02T18:29:42.223 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/ 2023-11-02T18:29:42.224 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I= 2023-11-02T18:29:42.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I=', 'Content-Length': '0'} 2023-11-02T18:29:42.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000998967cdb1d8d494-006543ea96-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.229 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:29:42.229 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:29:42.230 DEBUG:boto:Method: PUT 2023-11-02T18:29:42.230 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:29:42.231 DEBUG:boto:Data: 2023-11-02T18:29:42.231 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:29:42.232 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.232 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.232 DEBUG:boto:Params: {} 2023-11-02T18:29:42.233 DEBUG:boto:Token: None 2023-11-02T18:29:42.233 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:29:42.234 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S95mBr6rGVPIt1DWJfItLfSluBI= 2023-11-02T18:29:42.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S95mBr6rGVPIt1DWJfItLfSluBI='} 2023-11-02T18:29:42.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000386f4554c0a546c1-006543ea96-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.246 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:29:42.247 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:29:42.247 DEBUG:boto:Method: HEAD 2023-11-02T18:29:42.248 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:29:42.248 DEBUG:boto:Data: 2023-11-02T18:29:42.248 DEBUG:boto:Headers: {} 2023-11-02T18:29:42.249 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.249 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.250 DEBUG:boto:Params: {} 2023-11-02T18:29:42.250 DEBUG:boto:Token: None 2023-11-02T18:29:42.250 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/ 2023-11-02T18:29:42.251 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I= 2023-11-02T18:29:42.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I=', 'Content-Length': '0'} 2023-11-02T18:29:42.255 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ef0323e3b269e14-006543ea96-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.256 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:29:42.256 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:29:42.257 DEBUG:boto:Method: PUT 2023-11-02T18:29:42.257 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:29:42.258 DEBUG:boto:Data: 2023-11-02T18:29:42.258 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:29:42.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.259 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.260 DEBUG:boto:Params: {} 2023-11-02T18:29:42.260 DEBUG:boto:Token: None 2023-11-02T18:29:42.260 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:29:42.261 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zC9fA5CdefGkzZSo5+lOqPh/yW4= 2023-11-02T18:29:42.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zC9fA5CdefGkzZSo5+lOqPh/yW4='} 2023-11-02T18:29:42.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083dfe2ceafc98e00-006543ea96-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.307 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:29:42.307 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:29:42.308 DEBUG:boto:Method: HEAD 2023-11-02T18:29:42.308 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:29:42.309 DEBUG:boto:Data: 2023-11-02T18:29:42.309 DEBUG:boto:Headers: {} 2023-11-02T18:29:42.309 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.310 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.310 DEBUG:boto:Params: {} 2023-11-02T18:29:42.311 DEBUG:boto:Token: None 2023-11-02T18:29:42.311 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/ 2023-11-02T18:29:42.312 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I= 2023-11-02T18:29:42.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I=', 'Content-Length': '0'} 2023-11-02T18:29:42.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e83b2654e5ba53ea-006543ea96-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.317 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:29:42.317 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:29:42.318 DEBUG:boto:Method: PUT 2023-11-02T18:29:42.318 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006 2023-11-02T18:29:42.319 DEBUG:boto:Data: 2023-11-02T18:29:42.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:29:42.319 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.320 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.320 DEBUG:boto:Params: {} 2023-11-02T18:29:42.321 DEBUG:boto:Token: None 2023-11-02T18:29:42.321 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/obj00000000000000000006 2023-11-02T18:29:42.322 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QvPZbea8LWP/+Qrrw4AQJJ5SOWA= 2023-11-02T18:29:42.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QvPZbea8LWP/+Qrrw4AQJJ5SOWA='} 2023-11-02T18:29:42.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fde169bcbc481e2f-006543ea96-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.335 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:29:42.336 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:29:42.336 DEBUG:boto:Method: HEAD 2023-11-02T18:29:42.336 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:29:42.337 DEBUG:boto:Data: 2023-11-02T18:29:42.337 DEBUG:boto:Headers: {} 2023-11-02T18:29:42.338 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.338 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.339 DEBUG:boto:Params: {} 2023-11-02T18:29:42.339 DEBUG:boto:Token: None 2023-11-02T18:29:42.340 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/ 2023-11-02T18:29:42.340 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I= 2023-11-02T18:29:42.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I=', 'Content-Length': '0'} 2023-11-02T18:29:42.345 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fa8aa6c15f73b66-006543ea96-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.345 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:29:42.346 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:29:42.346 DEBUG:boto:Method: PUT 2023-11-02T18:29:42.346 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:29:42.347 DEBUG:boto:Data: 2023-11-02T18:29:42.347 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:29:42.348 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.348 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.349 DEBUG:boto:Params: {} 2023-11-02T18:29:42.349 DEBUG:boto:Token: None 2023-11-02T18:29:42.349 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:29:42.350 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z8FTlv6DanaoQ6tXLI4iJpaAhQY= 2023-11-02T18:29:42.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z8FTlv6DanaoQ6tXLI4iJpaAhQY='} 2023-11-02T18:29:42.399 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a1c86fb85541068-006543ea96-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.399 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:29:42.400 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:29:42.400 DEBUG:boto:Method: HEAD 2023-11-02T18:29:42.401 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:29:42.401 DEBUG:boto:Data: 2023-11-02T18:29:42.401 DEBUG:boto:Headers: {} 2023-11-02T18:29:42.402 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.402 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.403 DEBUG:boto:Params: {} 2023-11-02T18:29:42.403 DEBUG:boto:Token: None 2023-11-02T18:29:42.404 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/ 2023-11-02T18:29:42.404 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I= 2023-11-02T18:29:42.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I=', 'Content-Length': '0'} 2023-11-02T18:29:42.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a4c9cf68e7841d9-006543ea96-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.409 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:29:42.410 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:29:42.410 DEBUG:boto:Method: PUT 2023-11-02T18:29:42.411 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:29:42.411 DEBUG:boto:Data: 2023-11-02T18:29:42.412 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:29:42.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.412 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.413 DEBUG:boto:Params: {} 2023-11-02T18:29:42.414 DEBUG:boto:Token: None 2023-11-02T18:29:42.414 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:29:42.415 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1U98lHuRXbIg4MxGEDe7N0PhEZo= 2023-11-02T18:29:42.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1U98lHuRXbIg4MxGEDe7N0PhEZo='} 2023-11-02T18:29:42.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d830d6c4fc085230-006543ea96-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.427 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:29:42.428 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:29:42.428 DEBUG:boto:Method: HEAD 2023-11-02T18:29:42.429 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:29:42.429 DEBUG:boto:Data: 2023-11-02T18:29:42.430 DEBUG:boto:Headers: {} 2023-11-02T18:29:42.430 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.430 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.431 DEBUG:boto:Params: {} 2023-11-02T18:29:42.431 DEBUG:boto:Token: None 2023-11-02T18:29:42.432 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/ 2023-11-02T18:29:42.432 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I= 2023-11-02T18:29:42.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I=', 'Content-Length': '0'} 2023-11-02T18:29:42.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095cf5cdeea795aab-006543ea96-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.438 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:29:42.439 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:29:42.439 DEBUG:boto:Method: PUT 2023-11-02T18:29:42.439 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005 2023-11-02T18:29:42.440 DEBUG:boto:Data: 2023-11-02T18:29:42.440 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:29:42.441 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.442 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.443 DEBUG:boto:Params: {} 2023-11-02T18:29:42.443 DEBUG:boto:Token: None 2023-11-02T18:29:42.444 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/obj00000000000000000005 2023-11-02T18:29:42.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k3sWYVnZwbEWV03mIwf+YJqepNs= 2023-11-02T18:29:42.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k3sWYVnZwbEWV03mIwf+YJqepNs='} 2023-11-02T18:29:42.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cfb5849ded6b8f3-006543ea96-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.491 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:29:42.492 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:29:42.492 DEBUG:boto:Method: HEAD 2023-11-02T18:29:42.493 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:29:42.493 DEBUG:boto:Data: 2023-11-02T18:29:42.493 DEBUG:boto:Headers: {} 2023-11-02T18:29:42.494 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.494 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.495 DEBUG:boto:Params: {} 2023-11-02T18:29:42.496 DEBUG:boto:Token: None 2023-11-02T18:29:42.497 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/ 2023-11-02T18:29:42.497 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I= 2023-11-02T18:29:42.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BjbK2QIR1kyVXJeUV2pvBCo/n/I=', 'Content-Length': '0'} 2023-11-02T18:29:42.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b19c2fd9c3a52663-006543ea96-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.505 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:29:42.505 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:29:42.506 DEBUG:boto:Method: PUT 2023-11-02T18:29:42.507 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:29:42.508 DEBUG:boto:Data: 2023-11-02T18:29:42.509 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:29:42.509 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:29:42.510 DEBUG:boto:Port: 8000 2023-11-02T18:29:42.511 DEBUG:boto:Params: {} 2023-11-02T18:29:42.511 DEBUG:boto:Token: None 2023-11-02T18:29:42.512 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:29:42 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:29:42.512 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L02f0UenlZawLbYsL6Rs8rsIftU= 2023-11-02T18:29:42.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:29:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L02f0UenlZawLbYsL6Rs8rsIftU='} 2023-11-02T18:29:42.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085dc0dcb4cad3286-006543ea96-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:29:42.528 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-58 2023-11-02T18:29:42.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:29:42.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:29:42.529 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:29:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:29:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:29:42.639 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:42.632+0000 7f9339e0af40 1 waiting to reach incremental sync.. 2023-11-02T18:29:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:47.632+0000 7f9339e0af40 1 waiting to reach incremental sync.. 2023-11-02T18:29:52.644 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:52.636+0000 7f9339e0af40 1 bucket sync caught up with source: 2023-11-02T18:29:52.645 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.43.6, 00000000001.54.6, 00000000001.39.6, 00000000001.36.6, 00000000001.29.6, 00000000001.39.6, 00000000001.27.6, 00000000001.24.6, 00000000001.41.6, 00000000001.37.6] 2023-11-02T18:29:52.645 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.43.6, 00000000001.54.6, 00000000001.39.6, 00000000001.36.6, 00000000001.29.6, 00000000001.39.6, 00000000001.27.6, 00000000001.24.6, 00000000001.41.6, 00000000001.37.6] 2023-11-02T18:29:52.645 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:52.636+0000 7f9339e0af40 0 bucket checkpoint complete 2023-11-02T18:29:52.671 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-58 2023-11-02T18:29:52.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:29:52.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:29:52.673 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:29:52.723 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:29:52.723 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:29:52.793 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:52.784+0000 7fa8796a4f40 1 waiting to reach incremental sync.. 2023-11-02T18:29:57.794 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:29:57.788+0000 7fa8796a4f40 1 waiting to reach incremental sync.. 2023-11-02T18:30:02.797 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:02.788+0000 7fa8796a4f40 1 bucket sync caught up with source: 2023-11-02T18:30:02.798 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.230.6, 00000000001.100.6, 00000000001.205.6, 00000000001.256.6, 00000000001.53.6, 00000000001.428.6, 00000000001.196.6, 00000000001.255.6, 00000000001.427.6, 00000000001.239.6] 2023-11-02T18:30:02.799 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.230.6, 00000000001.100.6, 00000000001.205.6, 00000000001.256.6, 00000000001.53.6, 00000000001.428.6, 00000000001.196.6, 00000000001.255.6, 00000000001.427.6, 00000000001.239.6] 2023-11-02T18:30:02.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:02.788+0000 7fa8796a4f40 0 bucket checkpoint complete 2023-11-02T18:30:02.827 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-58 zones={a2, a1} 2023-11-02T18:30:02.828 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:02.828 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:02.829 DEBUG:boto:Method: HEAD 2023-11-02T18:30:02.829 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:02.830 DEBUG:boto:Data: 2023-11-02T18:30:02.830 DEBUG:boto:Headers: {} 2023-11-02T18:30:02.831 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:02.831 DEBUG:boto:Port: 8001 2023-11-02T18:30:02.831 DEBUG:boto:Params: {} 2023-11-02T18:30:02.832 DEBUG:boto:Token: None 2023-11-02T18:30:02.832 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:02 GMT /ibtljn-58/ 2023-11-02T18:30:02.833 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ECoSeGQz9zJNWxcLHVy72cKhPW4= 2023-11-02T18:30:02.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ECoSeGQz9zJNWxcLHVy72cKhPW4=', 'Content-Length': '0'} 2023-11-02T18:30:02.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060883326198ce63e-006543eaaa-1ac5-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:02.838 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:02.839 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:02.839 DEBUG:boto:Method: HEAD 2023-11-02T18:30:02.839 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:02.840 DEBUG:boto:Data: 2023-11-02T18:30:02.840 DEBUG:boto:Headers: {} 2023-11-02T18:30:02.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:02.841 DEBUG:boto:Port: 8000 2023-11-02T18:30:02.841 DEBUG:boto:Params: {} 2023-11-02T18:30:02.842 DEBUG:boto:Token: None 2023-11-02T18:30:02.842 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:02 GMT /ibtljn-58/ 2023-11-02T18:30:02.843 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ECoSeGQz9zJNWxcLHVy72cKhPW4= 2023-11-02T18:30:02.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ECoSeGQz9zJNWxcLHVy72cKhPW4=', 'Content-Length': '0'} 2023-11-02T18:30:02.847 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002121447519a64e19-006543eaaa-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:02.848 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:30:02.848 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:02.849 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:02.849 DEBUG:boto:path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:02.850 DEBUG:boto:auth_path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:02.850 DEBUG:boto:Method: GET 2023-11-02T18:30:02.850 DEBUG:boto:Path: /ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:02.851 DEBUG:boto:Data: 2023-11-02T18:30:02.851 DEBUG:boto:Headers: {} 2023-11-02T18:30:02.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:02.852 DEBUG:boto:Port: 8001 2023-11-02T18:30:02.852 DEBUG:boto:Params: {} 2023-11-02T18:30:02.853 DEBUG:boto:Token: None 2023-11-02T18:30:02.853 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:02 GMT /ibtljn-58/?versions 2023-11-02T18:30:02.854 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XLSRDrKL7I7tu0omX/SrFAhYsus= 2023-11-02T18:30:02.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XLSRDrKL7I7tu0omX/SrFAhYsus=', 'Content-Length': '0'} 2023-11-02T18:30:02.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042250f70959e5276-006543eaaa-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:02.860 DEBUG:boto:b'ibtljn-58999falseobj00000000000000000000nulltrue2023-11-02T18:29:42.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-11-02T18:29:42.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-11-02T18:29:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-11-02T18:29:42.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-11-02T18:29:42.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-11-02T18:29:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-11-02T18:29:42.175Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-11-02T18:29:42.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-11-02T18:29:42.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-11-02T18:29:42.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:02.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-11-02T18:30:02.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-11-02T18:30:02.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-11-02T18:30:02.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-11-02T18:30:02.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-11-02T18:30:02.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-11-02T18:30:02.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-11-02T18:30:02.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-11-02T18:30:02.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-11-02T18:30:02.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-11-02T18:30:02.865 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:30:02.865 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:02.866 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:02.866 DEBUG:boto:path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:02.867 DEBUG:boto:auth_path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:02.867 DEBUG:boto:Method: GET 2023-11-02T18:30:02.867 DEBUG:boto:Path: /ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:02.868 DEBUG:boto:Data: 2023-11-02T18:30:02.868 DEBUG:boto:Headers: {} 2023-11-02T18:30:02.869 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:02.869 DEBUG:boto:Port: 8000 2023-11-02T18:30:02.869 DEBUG:boto:Params: {} 2023-11-02T18:30:02.870 DEBUG:boto:Token: None 2023-11-02T18:30:02.870 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:02 GMT /ibtljn-58/?versions 2023-11-02T18:30:02.871 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XLSRDrKL7I7tu0omX/SrFAhYsus= 2023-11-02T18:30:02.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XLSRDrKL7I7tu0omX/SrFAhYsus=', 'Content-Length': '0'} 2023-11-02T18:30:02.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006445c962323b0b6c-006543eaaa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:02.877 DEBUG:boto:b'ibtljn-58999falseobj00000000000000000000nulltrue2023-11-02T18:29:42.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-11-02T18:29:42.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-11-02T18:29:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-11-02T18:29:42.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-11-02T18:29:42.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-11-02T18:29:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-11-02T18:29:42.175Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-11-02T18:29:42.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-11-02T18:29:42.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-11-02T18:29:42.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:02.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-11-02T18:30:02.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-11-02T18:30:02.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-11-02T18:30:02.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-11-02T18:30:02.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-11-02T18:30:02.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-11-02T18:30:02.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-11-02T18:30:02.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-11-02T18:30:02.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-11-02T18:30:02.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-11-02T18:30:02.882 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:02.883 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:02.883 DEBUG:boto:path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:02.884 DEBUG:boto:auth_path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:02.884 DEBUG:boto:Method: GET 2023-11-02T18:30:02.884 DEBUG:boto:Path: /ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:02.885 DEBUG:boto:Data: 2023-11-02T18:30:02.885 DEBUG:boto:Headers: {} 2023-11-02T18:30:02.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:02.886 DEBUG:boto:Port: 8001 2023-11-02T18:30:02.887 DEBUG:boto:Params: {} 2023-11-02T18:30:02.887 DEBUG:boto:Token: None 2023-11-02T18:30:02.888 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:02 GMT /ibtljn-58/?versions 2023-11-02T18:30:02.888 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XLSRDrKL7I7tu0omX/SrFAhYsus= 2023-11-02T18:30:02.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XLSRDrKL7I7tu0omX/SrFAhYsus=', 'Content-Length': '0'} 2023-11-02T18:30:02.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c7fa60feee28c48-006543eaaa-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:02.894 DEBUG:boto:b'ibtljn-58999falseobj00000000000000000000nulltrue2023-11-02T18:29:42.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-11-02T18:29:42.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-11-02T18:29:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-11-02T18:29:42.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-11-02T18:29:42.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-11-02T18:29:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-11-02T18:29:42.175Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-11-02T18:29:42.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-11-02T18:29:42.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-11-02T18:29:42.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:02.896 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:02.896 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:02.897 DEBUG:boto:path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:02.897 DEBUG:boto:auth_path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:02.898 DEBUG:boto:Method: GET 2023-11-02T18:30:02.898 DEBUG:boto:Path: /ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:02.899 DEBUG:boto:Data: 2023-11-02T18:30:02.899 DEBUG:boto:Headers: {} 2023-11-02T18:30:02.899 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:02.900 DEBUG:boto:Port: 8000 2023-11-02T18:30:02.900 DEBUG:boto:Params: {} 2023-11-02T18:30:02.901 DEBUG:boto:Token: None 2023-11-02T18:30:02.901 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:02 GMT /ibtljn-58/?versions 2023-11-02T18:30:02.902 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XLSRDrKL7I7tu0omX/SrFAhYsus= 2023-11-02T18:30:02.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XLSRDrKL7I7tu0omX/SrFAhYsus=', 'Content-Length': '0'} 2023-11-02T18:30:02.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8d439c806eddd2e-006543eaaa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:02.908 DEBUG:boto:b'ibtljn-58999falseobj00000000000000000000nulltrue2023-11-02T18:29:42.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-11-02T18:29:42.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-11-02T18:29:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-11-02T18:29:42.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-11-02T18:29:42.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-11-02T18:29:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-11-02T18:29:42.175Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-11-02T18:29:42.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-11-02T18:29:42.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-11-02T18:29:42.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:02.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-11-02T18:30:02.910 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:02.913 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:02.913 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.915 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.915 DEBUG:boto:Method: GET 2023-11-02T18:30:02.916 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.916 DEBUG:boto:Data: 2023-11-02T18:30:02.917 DEBUG:boto:Headers: {} 2023-11-02T18:30:02.917 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:02.918 DEBUG:boto:Port: 8001 2023-11-02T18:30:02.918 DEBUG:boto:Params: {} 2023-11-02T18:30:02.919 DEBUG:boto:Token: None 2023-11-02T18:30:02.919 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:02 GMT /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.920 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mzbJ8/bd5kLMUB7v39uu02n1JHM= 2023-11-02T18:30:02.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mzbJ8/bd5kLMUB7v39uu02n1JHM=', 'Content-Length': '0'} 2023-11-02T18:30:02.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bffb77072c9c10c8-006543eaaa-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:02.966 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:02.967 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:02.968 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.968 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.969 DEBUG:boto:Method: GET 2023-11-02T18:30:02.969 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.970 DEBUG:boto:Data: 2023-11-02T18:30:02.970 DEBUG:boto:Headers: {} 2023-11-02T18:30:02.972 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:02.972 DEBUG:boto:Port: 8000 2023-11-02T18:30:02.973 DEBUG:boto:Params: {} 2023-11-02T18:30:02.974 DEBUG:boto:Token: None 2023-11-02T18:30:02.975 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:02 GMT /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mzbJ8/bd5kLMUB7v39uu02n1JHM= 2023-11-02T18:30:02.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mzbJ8/bd5kLMUB7v39uu02n1JHM=', 'Content-Length': '0'} 2023-11-02T18:30:02.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018fd53765af01f09-006543eaaa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:02.980 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:02.980 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:02.981 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.981 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.982 DEBUG:boto:Method: HEAD 2023-11-02T18:30:02.982 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.983 DEBUG:boto:Data: 2023-11-02T18:30:02.983 DEBUG:boto:Headers: {} 2023-11-02T18:30:02.984 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:02.984 DEBUG:boto:Port: 8001 2023-11-02T18:30:02.984 DEBUG:boto:Params: {} 2023-11-02T18:30:02.985 DEBUG:boto:Token: None 2023-11-02T18:30:02.985 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:02 GMT /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.986 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b7qczr4FwalvJIPoOYIhP6uEg3s= 2023-11-02T18:30:02.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b7qczr4FwalvJIPoOYIhP6uEg3s=', 'Content-Length': '0'} 2023-11-02T18:30:02.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007125f9e823ddb15d-006543eaaa-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:02.990 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:02.990 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:02.991 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.991 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.992 DEBUG:boto:Method: HEAD 2023-11-02T18:30:02.992 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.993 DEBUG:boto:Data: 2023-11-02T18:30:02.993 DEBUG:boto:Headers: {} 2023-11-02T18:30:02.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:02.994 DEBUG:boto:Port: 8000 2023-11-02T18:30:02.995 DEBUG:boto:Params: {} 2023-11-02T18:30:02.995 DEBUG:boto:Token: None 2023-11-02T18:30:02.996 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:02 GMT /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:02.996 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b7qczr4FwalvJIPoOYIhP6uEg3s= 2023-11-02T18:30:02.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b7qczr4FwalvJIPoOYIhP6uEg3s=', 'Content-Length': '0'} 2023-11-02T18:30:03.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e09fe22a3ef696ae-006543eaaa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-11-02T18:30:03.001 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.001 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.002 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:03.002 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:03.003 DEBUG:boto:Method: GET 2023-11-02T18:30:03.003 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:03.004 DEBUG:boto:Data: 2023-11-02T18:30:03.004 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.005 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.005 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.005 DEBUG:boto:Params: {} 2023-11-02T18:30:03.006 DEBUG:boto:Token: None 2023-11-02T18:30:03.006 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:03.007 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vjf2LSaYgyRKecXQx+Me2bDN7qE= 2023-11-02T18:30:03.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vjf2LSaYgyRKecXQx+Me2bDN7qE=', 'Content-Length': '0'} 2023-11-02T18:30:03.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016e716365f2aa865-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.011 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.012 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.013 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:03.013 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:03.014 DEBUG:boto:Method: GET 2023-11-02T18:30:03.014 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:03.015 DEBUG:boto:Data: 2023-11-02T18:30:03.015 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.015 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.016 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.016 DEBUG:boto:Params: {} 2023-11-02T18:30:03.017 DEBUG:boto:Token: None 2023-11-02T18:30:03.017 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:03.018 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vjf2LSaYgyRKecXQx+Me2bDN7qE= 2023-11-02T18:30:03.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vjf2LSaYgyRKecXQx+Me2bDN7qE=', 'Content-Length': '0'} 2023-11-02T18:30:03.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cbbdc52b00df12e-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.022 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.022 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.023 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:03.023 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:03.024 DEBUG:boto:Method: GET 2023-11-02T18:30:03.024 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:03.025 DEBUG:boto:Data: 2023-11-02T18:30:03.025 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.026 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.026 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.027 DEBUG:boto:Params: {} 2023-11-02T18:30:03.027 DEBUG:boto:Token: None 2023-11-02T18:30:03.028 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.028 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c587ba1a8d7ab643-006543eaab-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.034 DEBUG:boto:b'ibtljn-58obj000000000000000000001000falseobj000000000000000000002023-11-02T18:29:42.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.035 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.035 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.036 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:03.036 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:03.037 DEBUG:boto:Method: GET 2023-11-02T18:30:03.037 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:03.038 DEBUG:boto:Data: 2023-11-02T18:30:03.038 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.039 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.039 DEBUG:boto:Params: {} 2023-11-02T18:30:03.040 DEBUG:boto:Token: None 2023-11-02T18:30:03.040 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.041 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dbe90fcfd47afe1-006543eaab-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.049 DEBUG:boto:b'ibtljn-58obj000000000000000000001000falseobj000000000000000000002023-11-02T18:29:42.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.050 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.050 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.051 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.051 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.051 DEBUG:boto:Data: 2023-11-02T18:30:03.052 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.052 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.053 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.053 DEBUG:boto:Params: {} 2023-11-02T18:30:03.054 DEBUG:boto:Token: None 2023-11-02T18:30:03.054 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.055 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AhWZs2vKkR5g7coAxjVaIk6umMk= 2023-11-02T18:30:03.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AhWZs2vKkR5g7coAxjVaIk6umMk=', 'Content-Length': '0'} 2023-11-02T18:30:03.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4b80c4ceb81bbd1-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.058 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.059 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.059 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.060 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.060 DEBUG:boto:Data: 2023-11-02T18:30:03.060 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.061 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.061 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.062 DEBUG:boto:Params: {} 2023-11-02T18:30:03.062 DEBUG:boto:Token: None 2023-11-02T18:30:03.063 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.063 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AhWZs2vKkR5g7coAxjVaIk6umMk= 2023-11-02T18:30:03.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AhWZs2vKkR5g7coAxjVaIk6umMk=', 'Content-Length': '0'} 2023-11-02T18:30:03.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c78cce1cbc2f86e9-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-11-02T18:30:03.067 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.067 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.068 DEBUG:boto:Method: GET 2023-11-02T18:30:03.068 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.069 DEBUG:boto:Data: 2023-11-02T18:30:03.069 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.070 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.071 DEBUG:boto:Params: {} 2023-11-02T18:30:03.071 DEBUG:boto:Token: None 2023-11-02T18:30:03.071 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BlpBVuy2Rbfx4+/Z8rOx+lUg+7Q= 2023-11-02T18:30:03.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BlpBVuy2Rbfx4+/Z8rOx+lUg+7Q=', 'Content-Length': '0'} 2023-11-02T18:30:03.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021757a614995a8a7-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.118 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.119 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.120 DEBUG:boto:Method: GET 2023-11-02T18:30:03.120 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.121 DEBUG:boto:Data: 2023-11-02T18:30:03.121 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.122 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.122 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.123 DEBUG:boto:Params: {} 2023-11-02T18:30:03.124 DEBUG:boto:Token: None 2023-11-02T18:30:03.124 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:03.125 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BlpBVuy2Rbfx4+/Z8rOx+lUg+7Q= 2023-11-02T18:30:03.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BlpBVuy2Rbfx4+/Z8rOx+lUg+7Q=', 'Content-Length': '0'} 2023-11-02T18:30:03.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad3c0e39cea689a6-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.129 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.130 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.130 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:03.131 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:03.131 DEBUG:boto:Method: GET 2023-11-02T18:30:03.132 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:03.132 DEBUG:boto:Data: 2023-11-02T18:30:03.133 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.133 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.134 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.134 DEBUG:boto:Params: {} 2023-11-02T18:30:03.135 DEBUG:boto:Token: None 2023-11-02T18:30:03.136 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.136 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b0252971eefd40f-006543eaab-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.143 DEBUG:boto:b'ibtljn-58obj000000000000000000001000falseobj000000000000000000002023-11-02T18:29:42.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.144 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.144 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.145 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:03.145 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:03.145 DEBUG:boto:Method: GET 2023-11-02T18:30:03.146 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:03.146 DEBUG:boto:Data: 2023-11-02T18:30:03.147 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.147 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.148 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.148 DEBUG:boto:Params: {} 2023-11-02T18:30:03.149 DEBUG:boto:Token: None 2023-11-02T18:30:03.149 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.150 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071e9b2bf840766f1-006543eaab-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.156 DEBUG:boto:b'ibtljn-58obj000000000000000000001000falseobj000000000000000000002023-11-02T18:29:42.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-11-02T18:30:03.157 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.158 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.158 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.159 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.159 DEBUG:boto:Method: GET 2023-11-02T18:30:03.160 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.160 DEBUG:boto:Data: 2023-11-02T18:30:03.160 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.161 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.161 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.162 DEBUG:boto:Params: {} 2023-11-02T18:30:03.162 DEBUG:boto:Token: None 2023-11-02T18:30:03.163 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.163 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xwT7VLxZ9jSHW5WGS3a8eCOcXYY= 2023-11-02T18:30:03.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xwT7VLxZ9jSHW5WGS3a8eCOcXYY=', 'Content-Length': '0'} 2023-11-02T18:30:03.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba124e34275244ba-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.168 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.168 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.169 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.169 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.169 DEBUG:boto:Method: GET 2023-11-02T18:30:03.170 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.170 DEBUG:boto:Data: 2023-11-02T18:30:03.171 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.171 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.172 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.172 DEBUG:boto:Params: {} 2023-11-02T18:30:03.173 DEBUG:boto:Token: None 2023-11-02T18:30:03.173 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.174 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xwT7VLxZ9jSHW5WGS3a8eCOcXYY= 2023-11-02T18:30:03.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xwT7VLxZ9jSHW5WGS3a8eCOcXYY=', 'Content-Length': '0'} 2023-11-02T18:30:03.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faad0c6b03642b31-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.178 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.178 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.179 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.179 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.180 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.180 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.181 DEBUG:boto:Data: 2023-11-02T18:30:03.181 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.181 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.182 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.182 DEBUG:boto:Params: {} 2023-11-02T18:30:03.183 DEBUG:boto:Token: None 2023-11-02T18:30:03.183 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.184 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PvcdIT+Kd25Ghub/t6LHPQj+L6w= 2023-11-02T18:30:03.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PvcdIT+Kd25Ghub/t6LHPQj+L6w=', 'Content-Length': '0'} 2023-11-02T18:30:03.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6b9477e2127f3a1-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.188 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.188 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.189 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.189 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.189 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.190 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.190 DEBUG:boto:Data: 2023-11-02T18:30:03.191 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.191 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.192 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.192 DEBUG:boto:Params: {} 2023-11-02T18:30:03.192 DEBUG:boto:Token: None 2023-11-02T18:30:03.193 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.193 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PvcdIT+Kd25Ghub/t6LHPQj+L6w= 2023-11-02T18:30:03.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PvcdIT+Kd25Ghub/t6LHPQj+L6w=', 'Content-Length': '0'} 2023-11-02T18:30:03.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f99b5fcd28f3ac12-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-11-02T18:30:03.198 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.199 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.199 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.199 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.200 DEBUG:boto:Method: GET 2023-11-02T18:30:03.200 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.201 DEBUG:boto:Data: 2023-11-02T18:30:03.201 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.202 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.203 DEBUG:boto:Params: {} 2023-11-02T18:30:03.203 DEBUG:boto:Token: None 2023-11-02T18:30:03.204 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.204 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xwT7VLxZ9jSHW5WGS3a8eCOcXYY= 2023-11-02T18:30:03.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xwT7VLxZ9jSHW5WGS3a8eCOcXYY=', 'Content-Length': '0'} 2023-11-02T18:30:03.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e6d46d617084dc-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.250 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.251 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.251 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.252 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.252 DEBUG:boto:Method: GET 2023-11-02T18:30:03.253 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.253 DEBUG:boto:Data: 2023-11-02T18:30:03.254 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.255 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.255 DEBUG:boto:Params: {} 2023-11-02T18:30:03.256 DEBUG:boto:Token: None 2023-11-02T18:30:03.256 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:03.257 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xwT7VLxZ9jSHW5WGS3a8eCOcXYY= 2023-11-02T18:30:03.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xwT7VLxZ9jSHW5WGS3a8eCOcXYY=', 'Content-Length': '0'} 2023-11-02T18:30:03.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a24d3a1a6f10cd2e-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.261 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.262 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.262 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:03.262 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:03.263 DEBUG:boto:Method: GET 2023-11-02T18:30:03.263 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:03.264 DEBUG:boto:Data: 2023-11-02T18:30:03.264 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.265 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.265 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.266 DEBUG:boto:Params: {} 2023-11-02T18:30:03.266 DEBUG:boto:Token: None 2023-11-02T18:30:03.267 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.267 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2778a9f62ba382-006543eaab-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.273 DEBUG:boto:b'ibtljn-58obj000000000000000000021000falseobj000000000000000000022023-11-02T18:29:42.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.274 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.275 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.275 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:03.276 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:03.276 DEBUG:boto:Method: GET 2023-11-02T18:30:03.277 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:03.277 DEBUG:boto:Data: 2023-11-02T18:30:03.277 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.278 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.278 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.279 DEBUG:boto:Params: {} 2023-11-02T18:30:03.279 DEBUG:boto:Token: None 2023-11-02T18:30:03.280 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.280 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e55d47ef08e338a-006543eaab-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.286 DEBUG:boto:b'ibtljn-58obj000000000000000000021000falseobj000000000000000000022023-11-02T18:29:42.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.287 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.287 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.288 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.288 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.289 DEBUG:boto:Data: 2023-11-02T18:30:03.290 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.290 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.291 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.291 DEBUG:boto:Params: {} 2023-11-02T18:30:03.292 DEBUG:boto:Token: None 2023-11-02T18:30:03.293 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.293 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zEFLEcezr8e9aHiDlWMZE/7CYYk= 2023-11-02T18:30:03.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zEFLEcezr8e9aHiDlWMZE/7CYYk=', 'Content-Length': '0'} 2023-11-02T18:30:03.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32bba60eb06ec22-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.297 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.298 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.298 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.299 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.300 DEBUG:boto:Data: 2023-11-02T18:30:03.300 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.301 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.301 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.302 DEBUG:boto:Params: {} 2023-11-02T18:30:03.303 DEBUG:boto:Token: None 2023-11-02T18:30:03.303 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.304 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zEFLEcezr8e9aHiDlWMZE/7CYYk= 2023-11-02T18:30:03.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zEFLEcezr8e9aHiDlWMZE/7CYYk=', 'Content-Length': '0'} 2023-11-02T18:30:03.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012188158c54642f0-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-11-02T18:30:03.308 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.308 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.309 DEBUG:boto:Method: GET 2023-11-02T18:30:03.309 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.310 DEBUG:boto:Data: 2023-11-02T18:30:03.310 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.311 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.311 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.312 DEBUG:boto:Params: {} 2023-11-02T18:30:03.312 DEBUG:boto:Token: None 2023-11-02T18:30:03.313 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.313 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g5az3Jp0N7b0sUmGsyOL9CHcuGI= 2023-11-02T18:30:03.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g5az3Jp0N7b0sUmGsyOL9CHcuGI=', 'Content-Length': '0'} 2023-11-02T18:30:03.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fec0096fe87e03a-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.317 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.318 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.318 DEBUG:boto:Method: GET 2023-11-02T18:30:03.319 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.319 DEBUG:boto:Data: 2023-11-02T18:30:03.320 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.320 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.321 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.321 DEBUG:boto:Params: {} 2023-11-02T18:30:03.321 DEBUG:boto:Token: None 2023-11-02T18:30:03.322 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:03.322 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g5az3Jp0N7b0sUmGsyOL9CHcuGI= 2023-11-02T18:30:03.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g5az3Jp0N7b0sUmGsyOL9CHcuGI=', 'Content-Length': '0'} 2023-11-02T18:30:03.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000373de1aea26043e6-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.366 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.367 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.367 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:03.368 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:03.368 DEBUG:boto:Method: GET 2023-11-02T18:30:03.369 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:03.369 DEBUG:boto:Data: 2023-11-02T18:30:03.370 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.371 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.371 DEBUG:boto:Params: {} 2023-11-02T18:30:03.372 DEBUG:boto:Token: None 2023-11-02T18:30:03.372 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.373 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f927991fc8d174f4-006543eaab-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.379 DEBUG:boto:b'ibtljn-58obj000000000000000000021000falseobj000000000000000000022023-11-02T18:29:42.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.380 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.380 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.381 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:03.381 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:03.381 DEBUG:boto:Method: GET 2023-11-02T18:30:03.382 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:03.382 DEBUG:boto:Data: 2023-11-02T18:30:03.383 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.384 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.384 DEBUG:boto:Params: {} 2023-11-02T18:30:03.385 DEBUG:boto:Token: None 2023-11-02T18:30:03.385 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.386 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9d07d32065825f3-006543eaab-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.393 DEBUG:boto:b'ibtljn-58obj000000000000000000021000falseobj000000000000000000022023-11-02T18:29:42.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-11-02T18:30:03.394 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.395 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.395 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.395 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.396 DEBUG:boto:Method: GET 2023-11-02T18:30:03.396 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.397 DEBUG:boto:Data: 2023-11-02T18:30:03.397 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.398 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.399 DEBUG:boto:Params: {} 2023-11-02T18:30:03.399 DEBUG:boto:Token: None 2023-11-02T18:30:03.400 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ttne0Faw1Ak3Bu4ymqs22zLuj5U= 2023-11-02T18:30:03.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ttne0Faw1Ak3Bu4ymqs22zLuj5U=', 'Content-Length': '0'} 2023-11-02T18:30:03.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d173ec98ddb4cf2-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.446 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.447 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.447 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.448 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.448 DEBUG:boto:Method: GET 2023-11-02T18:30:03.449 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.449 DEBUG:boto:Data: 2023-11-02T18:30:03.450 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.450 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.451 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.451 DEBUG:boto:Params: {} 2023-11-02T18:30:03.452 DEBUG:boto:Token: None 2023-11-02T18:30:03.452 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.453 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ttne0Faw1Ak3Bu4ymqs22zLuj5U= 2023-11-02T18:30:03.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ttne0Faw1Ak3Bu4ymqs22zLuj5U=', 'Content-Length': '0'} 2023-11-02T18:30:03.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb59b93931bb168c-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.457 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.457 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.458 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.458 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.459 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.459 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.459 DEBUG:boto:Data: 2023-11-02T18:30:03.460 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.460 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.461 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.461 DEBUG:boto:Params: {} 2023-11-02T18:30:03.462 DEBUG:boto:Token: None 2023-11-02T18:30:03.462 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.463 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/8vCD2efNyAyGQqE0zeXyeRsOqw= 2023-11-02T18:30:03.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/8vCD2efNyAyGQqE0zeXyeRsOqw=', 'Content-Length': '0'} 2023-11-02T18:30:03.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000923ef365fcc1ef09-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.467 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.467 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.468 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.468 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.468 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.469 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.469 DEBUG:boto:Data: 2023-11-02T18:30:03.470 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.470 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.470 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.471 DEBUG:boto:Params: {} 2023-11-02T18:30:03.471 DEBUG:boto:Token: None 2023-11-02T18:30:03.472 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.472 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/8vCD2efNyAyGQqE0zeXyeRsOqw= 2023-11-02T18:30:03.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/8vCD2efNyAyGQqE0zeXyeRsOqw=', 'Content-Length': '0'} 2023-11-02T18:30:03.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032626eb66f1c1b3b-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-11-02T18:30:03.477 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.477 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.478 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.478 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.479 DEBUG:boto:Method: GET 2023-11-02T18:30:03.479 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.480 DEBUG:boto:Data: 2023-11-02T18:30:03.480 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.480 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.481 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.481 DEBUG:boto:Params: {} 2023-11-02T18:30:03.482 DEBUG:boto:Token: None 2023-11-02T18:30:03.482 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.483 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ttne0Faw1Ak3Bu4ymqs22zLuj5U= 2023-11-02T18:30:03.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ttne0Faw1Ak3Bu4ymqs22zLuj5U=', 'Content-Length': '0'} 2023-11-02T18:30:03.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001940eb588532aca2-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.487 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.488 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.488 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.489 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.489 DEBUG:boto:Method: GET 2023-11-02T18:30:03.489 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.490 DEBUG:boto:Data: 2023-11-02T18:30:03.490 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.491 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.491 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.492 DEBUG:boto:Params: {} 2023-11-02T18:30:03.492 DEBUG:boto:Token: None 2023-11-02T18:30:03.493 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:03.493 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ttne0Faw1Ak3Bu4ymqs22zLuj5U= 2023-11-02T18:30:03.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ttne0Faw1Ak3Bu4ymqs22zLuj5U=', 'Content-Length': '0'} 2023-11-02T18:30:03.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef551533b8754303-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.542 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.543 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.543 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:03.544 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:03.544 DEBUG:boto:Method: GET 2023-11-02T18:30:03.545 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:03.545 DEBUG:boto:Data: 2023-11-02T18:30:03.546 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.546 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.546 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.547 DEBUG:boto:Params: {} 2023-11-02T18:30:03.547 DEBUG:boto:Token: None 2023-11-02T18:30:03.548 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.548 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b377ad0e9e2936f7-006543eaab-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.555 DEBUG:boto:b'ibtljn-58obj000000000000000000031000falseobj000000000000000000032023-11-02T18:29:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.556 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.556 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.557 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:03.557 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:03.558 DEBUG:boto:Method: GET 2023-11-02T18:30:03.558 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:03.558 DEBUG:boto:Data: 2023-11-02T18:30:03.559 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.559 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.560 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.560 DEBUG:boto:Params: {} 2023-11-02T18:30:03.561 DEBUG:boto:Token: None 2023-11-02T18:30:03.561 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.562 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001825ffec6cf25a48-006543eaab-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.569 DEBUG:boto:b'ibtljn-58obj000000000000000000031000falseobj000000000000000000032023-11-02T18:29:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.570 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.570 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.571 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.571 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.572 DEBUG:boto:Data: 2023-11-02T18:30:03.572 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.572 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.573 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.573 DEBUG:boto:Params: {} 2023-11-02T18:30:03.574 DEBUG:boto:Token: None 2023-11-02T18:30:03.574 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.575 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sh3UNTe0I7pbbuk4WCEcQoFPKXs= 2023-11-02T18:30:03.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sh3UNTe0I7pbbuk4WCEcQoFPKXs=', 'Content-Length': '0'} 2023-11-02T18:30:03.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a22c2b77fb05ac39-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.579 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.579 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.579 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.580 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.580 DEBUG:boto:Data: 2023-11-02T18:30:03.581 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.581 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.582 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.582 DEBUG:boto:Params: {} 2023-11-02T18:30:03.582 DEBUG:boto:Token: None 2023-11-02T18:30:03.583 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.583 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sh3UNTe0I7pbbuk4WCEcQoFPKXs= 2023-11-02T18:30:03.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sh3UNTe0I7pbbuk4WCEcQoFPKXs=', 'Content-Length': '0'} 2023-11-02T18:30:03.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001826ad3d45d4e6b3-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-11-02T18:30:03.587 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.587 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.588 DEBUG:boto:Method: GET 2023-11-02T18:30:03.588 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.589 DEBUG:boto:Data: 2023-11-02T18:30:03.589 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.590 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.591 DEBUG:boto:Params: {} 2023-11-02T18:30:03.591 DEBUG:boto:Token: None 2023-11-02T18:30:03.592 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.592 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2U5IJwzDVc1+xa2ctWQVqjclSNw= 2023-11-02T18:30:03.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2U5IJwzDVc1+xa2ctWQVqjclSNw=', 'Content-Length': '0'} 2023-11-02T18:30:03.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000180107d97750eba4-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.596 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.596 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.596 DEBUG:boto:Method: GET 2023-11-02T18:30:03.597 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.597 DEBUG:boto:Data: 2023-11-02T18:30:03.598 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.598 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.599 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.599 DEBUG:boto:Params: {} 2023-11-02T18:30:03.600 DEBUG:boto:Token: None 2023-11-02T18:30:03.600 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:03.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2U5IJwzDVc1+xa2ctWQVqjclSNw= 2023-11-02T18:30:03.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2U5IJwzDVc1+xa2ctWQVqjclSNw=', 'Content-Length': '0'} 2023-11-02T18:30:03.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004acb6f9338e7ccce-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.604 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.605 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.605 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:03.606 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:03.606 DEBUG:boto:Method: GET 2023-11-02T18:30:03.607 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:03.607 DEBUG:boto:Data: 2023-11-02T18:30:03.607 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.608 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.608 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.609 DEBUG:boto:Params: {} 2023-11-02T18:30:03.609 DEBUG:boto:Token: None 2023-11-02T18:30:03.610 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.610 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6f9632041d3b5e4-006543eaab-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.658 DEBUG:boto:b'ibtljn-58obj000000000000000000031000falseobj000000000000000000032023-11-02T18:29:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.659 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.660 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.660 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:03.661 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:03.661 DEBUG:boto:Method: GET 2023-11-02T18:30:03.661 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:03.662 DEBUG:boto:Data: 2023-11-02T18:30:03.662 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.664 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.664 DEBUG:boto:Params: {} 2023-11-02T18:30:03.665 DEBUG:boto:Token: None 2023-11-02T18:30:03.665 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.666 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dbcd2c9187218d7-006543eaab-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.673 DEBUG:boto:b'ibtljn-58obj000000000000000000031000falseobj000000000000000000032023-11-02T18:29:42.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-11-02T18:30:03.674 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.674 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.675 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.675 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.676 DEBUG:boto:Method: GET 2023-11-02T18:30:03.676 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.676 DEBUG:boto:Data: 2023-11-02T18:30:03.677 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.677 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.678 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.678 DEBUG:boto:Params: {} 2023-11-02T18:30:03.679 DEBUG:boto:Token: None 2023-11-02T18:30:03.679 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.680 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nMH3IhJFzKl6kW1LIvMARRr7dQs= 2023-11-02T18:30:03.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nMH3IhJFzKl6kW1LIvMARRr7dQs=', 'Content-Length': '0'} 2023-11-02T18:30:03.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000366fcb65716676f9-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.684 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.684 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.685 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.685 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.686 DEBUG:boto:Method: GET 2023-11-02T18:30:03.686 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.687 DEBUG:boto:Data: 2023-11-02T18:30:03.687 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.687 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.688 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.688 DEBUG:boto:Params: {} 2023-11-02T18:30:03.689 DEBUG:boto:Token: None 2023-11-02T18:30:03.689 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.690 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nMH3IhJFzKl6kW1LIvMARRr7dQs= 2023-11-02T18:30:03.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nMH3IhJFzKl6kW1LIvMARRr7dQs=', 'Content-Length': '0'} 2023-11-02T18:30:03.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d08b9ae2b8a0135f-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.734 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.735 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.735 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.736 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.736 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.737 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.737 DEBUG:boto:Data: 2023-11-02T18:30:03.738 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.739 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.740 DEBUG:boto:Params: {} 2023-11-02T18:30:03.740 DEBUG:boto:Token: None 2023-11-02T18:30:03.741 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.741 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZowMXSUNAmIu46ufycsIukgVcM0= 2023-11-02T18:30:03.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZowMXSUNAmIu46ufycsIukgVcM0=', 'Content-Length': '0'} 2023-11-02T18:30:03.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006086701f323df136-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.746 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.746 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.747 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.747 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.748 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.748 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.748 DEBUG:boto:Data: 2023-11-02T18:30:03.749 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.750 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.750 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.750 DEBUG:boto:Params: {} 2023-11-02T18:30:03.751 DEBUG:boto:Token: None 2023-11-02T18:30:03.751 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.752 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZowMXSUNAmIu46ufycsIukgVcM0= 2023-11-02T18:30:03.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZowMXSUNAmIu46ufycsIukgVcM0=', 'Content-Length': '0'} 2023-11-02T18:30:03.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f4540e57a522c3-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-11-02T18:30:03.757 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.757 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.758 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.758 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.759 DEBUG:boto:Method: GET 2023-11-02T18:30:03.759 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.759 DEBUG:boto:Data: 2023-11-02T18:30:03.760 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.760 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.761 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.761 DEBUG:boto:Params: {} 2023-11-02T18:30:03.762 DEBUG:boto:Token: None 2023-11-02T18:30:03.763 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.763 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nMH3IhJFzKl6kW1LIvMARRr7dQs= 2023-11-02T18:30:03.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nMH3IhJFzKl6kW1LIvMARRr7dQs=', 'Content-Length': '0'} 2023-11-02T18:30:03.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048a21d3473cf9fc6-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.768 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.768 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.769 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.769 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.769 DEBUG:boto:Method: GET 2023-11-02T18:30:03.770 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.770 DEBUG:boto:Data: 2023-11-02T18:30:03.771 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.771 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.772 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.772 DEBUG:boto:Params: {} 2023-11-02T18:30:03.773 DEBUG:boto:Token: None 2023-11-02T18:30:03.774 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:03.774 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nMH3IhJFzKl6kW1LIvMARRr7dQs= 2023-11-02T18:30:03.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nMH3IhJFzKl6kW1LIvMARRr7dQs=', 'Content-Length': '0'} 2023-11-02T18:30:03.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b563ed669163ac8-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.778 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.779 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.779 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:03.780 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:03.780 DEBUG:boto:Method: GET 2023-11-02T18:30:03.781 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:03.781 DEBUG:boto:Data: 2023-11-02T18:30:03.782 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.782 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.783 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.783 DEBUG:boto:Params: {} 2023-11-02T18:30:03.784 DEBUG:boto:Token: None 2023-11-02T18:30:03.784 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000152925a72c29f046-006543eaab-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.792 DEBUG:boto:b'ibtljn-58obj000000000000000000041000falseobj000000000000000000042023-11-02T18:29:42.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.792 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.793 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.793 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:03.794 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:03.794 DEBUG:boto:Method: GET 2023-11-02T18:30:03.795 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:03.795 DEBUG:boto:Data: 2023-11-02T18:30:03.796 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.796 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.797 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.797 DEBUG:boto:Params: {} 2023-11-02T18:30:03.798 DEBUG:boto:Token: None 2023-11-02T18:30:03.798 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.799 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000605c11417772eade-006543eaab-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.805 DEBUG:boto:b'ibtljn-58obj000000000000000000041000falseobj000000000000000000042023-11-02T18:29:42.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.806 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.806 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.807 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.807 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.808 DEBUG:boto:Data: 2023-11-02T18:30:03.808 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.809 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.809 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.810 DEBUG:boto:Params: {} 2023-11-02T18:30:03.810 DEBUG:boto:Token: None 2023-11-02T18:30:03.811 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.811 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g734RhCMnIZjHoYO6q3bZAcsS4I= 2023-11-02T18:30:03.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g734RhCMnIZjHoYO6q3bZAcsS4I=', 'Content-Length': '0'} 2023-11-02T18:30:03.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ec8463b61f907b-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.815 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.816 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.816 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.817 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.817 DEBUG:boto:Data: 2023-11-02T18:30:03.818 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.818 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.819 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.819 DEBUG:boto:Params: {} 2023-11-02T18:30:03.820 DEBUG:boto:Token: None 2023-11-02T18:30:03.820 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.821 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g734RhCMnIZjHoYO6q3bZAcsS4I= 2023-11-02T18:30:03.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g734RhCMnIZjHoYO6q3bZAcsS4I=', 'Content-Length': '0'} 2023-11-02T18:30:03.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000960c718d3db6a0db-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-11-02T18:30:03.825 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.826 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.826 DEBUG:boto:Method: GET 2023-11-02T18:30:03.827 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.827 DEBUG:boto:Data: 2023-11-02T18:30:03.828 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.828 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.829 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.829 DEBUG:boto:Params: {} 2023-11-02T18:30:03.830 DEBUG:boto:Token: None 2023-11-02T18:30:03.830 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.831 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xUEZts/ENmgpFQaaGJtK1mBCAC4= 2023-11-02T18:30:03.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xUEZts/ENmgpFQaaGJtK1mBCAC4=', 'Content-Length': '0'} 2023-11-02T18:30:03.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e67fc610034f7e0-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.878 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.879 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.880 DEBUG:boto:Method: GET 2023-11-02T18:30:03.880 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.881 DEBUG:boto:Data: 2023-11-02T18:30:03.881 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.882 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.882 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.883 DEBUG:boto:Params: {} 2023-11-02T18:30:03.883 DEBUG:boto:Token: None 2023-11-02T18:30:03.884 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:03.884 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xUEZts/ENmgpFQaaGJtK1mBCAC4= 2023-11-02T18:30:03.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xUEZts/ENmgpFQaaGJtK1mBCAC4=', 'Content-Length': '0'} 2023-11-02T18:30:03.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a8b186b19b4d0ec-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.889 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.889 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.889 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:03.890 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:03.890 DEBUG:boto:Method: GET 2023-11-02T18:30:03.891 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:03.891 DEBUG:boto:Data: 2023-11-02T18:30:03.892 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.892 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.893 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.893 DEBUG:boto:Params: {} 2023-11-02T18:30:03.894 DEBUG:boto:Token: None 2023-11-02T18:30:03.894 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.895 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000931c95c9a6ebcb2f-006543eaab-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.901 DEBUG:boto:b'ibtljn-58obj000000000000000000041000falseobj000000000000000000042023-11-02T18:29:42.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.902 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:03.902 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:03.903 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:03.903 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:03.904 DEBUG:boto:Method: GET 2023-11-02T18:30:03.904 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:03.905 DEBUG:boto:Data: 2023-11-02T18:30:03.905 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.906 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.906 DEBUG:boto:Params: {} 2023-11-02T18:30:03.907 DEBUG:boto:Token: None 2023-11-02T18:30:03.907 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/ 2023-11-02T18:30:03.908 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA= 2023-11-02T18:30:03.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LHdeq8iXgwNyHtu5VerWkhUdIwA=', 'Content-Length': '0'} 2023-11-02T18:30:03.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001abf1fb188775ae4-006543eaab-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.914 DEBUG:boto:b'ibtljn-58obj000000000000000000041000falseobj000000000000000000042023-11-02T18:29:42.514Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:03.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-11-02T18:30:03.915 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:03.916 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:03.916 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.917 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.917 DEBUG:boto:Method: GET 2023-11-02T18:30:03.918 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.918 DEBUG:boto:Data: 2023-11-02T18:30:03.919 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.919 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.919 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.920 DEBUG:boto:Params: {} 2023-11-02T18:30:03.920 DEBUG:boto:Token: None 2023-11-02T18:30:03.921 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kvhPr7x6W5ymDg9Ghi+GM29oKQM= 2023-11-02T18:30:03.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kvhPr7x6W5ymDg9Ghi+GM29oKQM=', 'Content-Length': '0'} 2023-11-02T18:30:03.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086d2cdcf5842c0ab-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.926 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:03.927 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:03.927 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.928 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.928 DEBUG:boto:Method: GET 2023-11-02T18:30:03.928 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.929 DEBUG:boto:Data: 2023-11-02T18:30:03.929 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.930 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.931 DEBUG:boto:Params: {} 2023-11-02T18:30:03.931 DEBUG:boto:Token: None 2023-11-02T18:30:03.932 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.932 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kvhPr7x6W5ymDg9Ghi+GM29oKQM= 2023-11-02T18:30:03.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kvhPr7x6W5ymDg9Ghi+GM29oKQM=', 'Content-Length': '0'} 2023-11-02T18:30:03.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e870a4de54f102ba-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.936 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:03.937 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:03.937 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.938 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.938 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.939 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.939 DEBUG:boto:Data: 2023-11-02T18:30:03.940 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.940 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.940 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.941 DEBUG:boto:Params: {} 2023-11-02T18:30:03.941 DEBUG:boto:Token: None 2023-11-02T18:30:03.942 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.942 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:01nn8VtLpGaicEkzMlTolTTvZNs= 2023-11-02T18:30:03.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:01nn8VtLpGaicEkzMlTolTTvZNs=', 'Content-Length': '0'} 2023-11-02T18:30:03.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae974355df313922-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.946 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:03.947 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:03.947 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.947 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.948 DEBUG:boto:Method: HEAD 2023-11-02T18:30:03.948 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.949 DEBUG:boto:Data: 2023-11-02T18:30:03.949 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.950 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.950 DEBUG:boto:Params: {} 2023-11-02T18:30:03.951 DEBUG:boto:Token: None 2023-11-02T18:30:03.951 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.952 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:01nn8VtLpGaicEkzMlTolTTvZNs= 2023-11-02T18:30:03.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:01nn8VtLpGaicEkzMlTolTTvZNs=', 'Content-Length': '0'} 2023-11-02T18:30:03.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f46b2fdc007d3670-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-11-02T18:30:03.956 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:03.957 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:03.957 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.957 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.958 DEBUG:boto:Method: GET 2023-11-02T18:30:03.958 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.959 DEBUG:boto:Data: 2023-11-02T18:30:03.959 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.960 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:03.960 DEBUG:boto:Port: 8001 2023-11-02T18:30:03.960 DEBUG:boto:Params: {} 2023-11-02T18:30:03.961 DEBUG:boto:Token: None 2023-11-02T18:30:03.961 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.962 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kvhPr7x6W5ymDg9Ghi+GM29oKQM= 2023-11-02T18:30:03.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kvhPr7x6W5ymDg9Ghi+GM29oKQM=', 'Content-Length': '0'} 2023-11-02T18:30:03.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da830eb0ba6a42ca-006543eaab-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:03.966 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:03.967 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:03.967 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.968 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.968 DEBUG:boto:Method: GET 2023-11-02T18:30:03.968 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.969 DEBUG:boto:Data: 2023-11-02T18:30:03.969 DEBUG:boto:Headers: {} 2023-11-02T18:30:03.970 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:03.970 DEBUG:boto:Port: 8000 2023-11-02T18:30:03.971 DEBUG:boto:Params: {} 2023-11-02T18:30:03.971 DEBUG:boto:Token: None 2023-11-02T18:30:03.972 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:03 GMT /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:03.972 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kvhPr7x6W5ymDg9Ghi+GM29oKQM= 2023-11-02T18:30:03.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kvhPr7x6W5ymDg9Ghi+GM29oKQM=', 'Content-Length': '0'} 2023-11-02T18:30:03.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2aa0dcb2a3aa8b9-006543eaab-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.018 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.019 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.019 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:04.020 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:04.020 DEBUG:boto:Method: GET 2023-11-02T18:30:04.021 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:04.021 DEBUG:boto:Data: 2023-11-02T18:30:04.022 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.022 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.023 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.023 DEBUG:boto:Params: {} 2023-11-02T18:30:04.024 DEBUG:boto:Token: None 2023-11-02T18:30:04.024 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.025 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ede80580ffe66074-006543eaac-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.031 DEBUG:boto:b'ibtljn-58obj000000000000000000051000falseobj000000000000000000052023-11-02T18:29:42.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.032 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.033 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.033 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:04.034 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:04.034 DEBUG:boto:Method: GET 2023-11-02T18:30:04.035 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:04.035 DEBUG:boto:Data: 2023-11-02T18:30:04.035 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.036 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.036 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.037 DEBUG:boto:Params: {} 2023-11-02T18:30:04.037 DEBUG:boto:Token: None 2023-11-02T18:30:04.038 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.038 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044fb6d28998a948f-006543eaac-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.045 DEBUG:boto:b'ibtljn-58obj000000000000000000051000falseobj000000000000000000052023-11-02T18:29:42.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.046 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.047 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.047 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.047 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.048 DEBUG:boto:Data: 2023-11-02T18:30:04.048 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.049 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.049 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.049 DEBUG:boto:Params: {} 2023-11-02T18:30:04.050 DEBUG:boto:Token: None 2023-11-02T18:30:04.050 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lOJxEN/XQC2cgg3hB76WBEb2TCk= 2023-11-02T18:30:04.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lOJxEN/XQC2cgg3hB76WBEb2TCk=', 'Content-Length': '0'} 2023-11-02T18:30:04.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065fd9770bfce08e0-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.055 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.055 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.056 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.056 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.056 DEBUG:boto:Data: 2023-11-02T18:30:04.057 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.057 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.058 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.058 DEBUG:boto:Params: {} 2023-11-02T18:30:04.059 DEBUG:boto:Token: None 2023-11-02T18:30:04.059 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.060 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lOJxEN/XQC2cgg3hB76WBEb2TCk= 2023-11-02T18:30:04.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lOJxEN/XQC2cgg3hB76WBEb2TCk=', 'Content-Length': '0'} 2023-11-02T18:30:04.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a900f52257d6b4f-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-11-02T18:30:04.064 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.064 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.065 DEBUG:boto:Method: GET 2023-11-02T18:30:04.065 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.065 DEBUG:boto:Data: 2023-11-02T18:30:04.066 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.066 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.067 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.067 DEBUG:boto:Params: {} 2023-11-02T18:30:04.068 DEBUG:boto:Token: None 2023-11-02T18:30:04.068 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.069 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZIV6EJziPtnYJW7yBpnaozPPRpQ= 2023-11-02T18:30:04.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZIV6EJziPtnYJW7yBpnaozPPRpQ=', 'Content-Length': '0'} 2023-11-02T18:30:04.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000196b80455bb70050-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.117 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.118 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.119 DEBUG:boto:Method: GET 2023-11-02T18:30:04.119 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.120 DEBUG:boto:Data: 2023-11-02T18:30:04.120 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.120 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.121 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.121 DEBUG:boto:Params: {} 2023-11-02T18:30:04.122 DEBUG:boto:Token: None 2023-11-02T18:30:04.122 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:04.123 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZIV6EJziPtnYJW7yBpnaozPPRpQ= 2023-11-02T18:30:04.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZIV6EJziPtnYJW7yBpnaozPPRpQ=', 'Content-Length': '0'} 2023-11-02T18:30:04.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3d030a40314c109-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.127 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.128 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.128 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:04.128 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:04.129 DEBUG:boto:Method: GET 2023-11-02T18:30:04.129 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:04.130 DEBUG:boto:Data: 2023-11-02T18:30:04.130 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.131 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.131 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.132 DEBUG:boto:Params: {} 2023-11-02T18:30:04.132 DEBUG:boto:Token: None 2023-11-02T18:30:04.133 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.133 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000467a0d69d4262be1-006543eaac-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.140 DEBUG:boto:b'ibtljn-58obj000000000000000000051000falseobj000000000000000000052023-11-02T18:29:42.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.141 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.141 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.142 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:04.142 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:04.143 DEBUG:boto:Method: GET 2023-11-02T18:30:04.143 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:04.143 DEBUG:boto:Data: 2023-11-02T18:30:04.144 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.144 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.145 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.145 DEBUG:boto:Params: {} 2023-11-02T18:30:04.146 DEBUG:boto:Token: None 2023-11-02T18:30:04.146 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.147 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7e703117f5c3518-006543eaac-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.194 DEBUG:boto:b'ibtljn-58obj000000000000000000051000falseobj000000000000000000052023-11-02T18:29:42.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-11-02T18:30:04.196 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.196 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.197 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.197 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.198 DEBUG:boto:Method: GET 2023-11-02T18:30:04.198 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.199 DEBUG:boto:Data: 2023-11-02T18:30:04.199 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.200 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.200 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.201 DEBUG:boto:Params: {} 2023-11-02T18:30:04.201 DEBUG:boto:Token: None 2023-11-02T18:30:04.202 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.202 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5UTjHCvZZ6RK/zULjuZJOS5048Y= 2023-11-02T18:30:04.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5UTjHCvZZ6RK/zULjuZJOS5048Y=', 'Content-Length': '0'} 2023-11-02T18:30:04.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cacab8efdb3fa5e0-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.207 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.207 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.208 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.208 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.209 DEBUG:boto:Method: GET 2023-11-02T18:30:04.209 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.209 DEBUG:boto:Data: 2023-11-02T18:30:04.210 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.210 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.211 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.211 DEBUG:boto:Params: {} 2023-11-02T18:30:04.212 DEBUG:boto:Token: None 2023-11-02T18:30:04.212 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.213 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5UTjHCvZZ6RK/zULjuZJOS5048Y= 2023-11-02T18:30:04.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5UTjHCvZZ6RK/zULjuZJOS5048Y=', 'Content-Length': '0'} 2023-11-02T18:30:04.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e0be41a852b48b1-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.217 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.217 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.218 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.218 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.219 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.219 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.219 DEBUG:boto:Data: 2023-11-02T18:30:04.220 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.220 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.221 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.221 DEBUG:boto:Params: {} 2023-11-02T18:30:04.222 DEBUG:boto:Token: None 2023-11-02T18:30:04.222 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.223 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+hqZa6d2Qmbufoy+CJpZvFLOO/o= 2023-11-02T18:30:04.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+hqZa6d2Qmbufoy+CJpZvFLOO/o=', 'Content-Length': '0'} 2023-11-02T18:30:04.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ff0c3ae8fbb9069-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.227 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.227 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.228 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.228 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.228 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.229 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.229 DEBUG:boto:Data: 2023-11-02T18:30:04.230 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.230 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.231 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.231 DEBUG:boto:Params: {} 2023-11-02T18:30:04.231 DEBUG:boto:Token: None 2023-11-02T18:30:04.232 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.232 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+hqZa6d2Qmbufoy+CJpZvFLOO/o= 2023-11-02T18:30:04.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+hqZa6d2Qmbufoy+CJpZvFLOO/o=', 'Content-Length': '0'} 2023-11-02T18:30:04.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beb20b0fbb23336b-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-11-02T18:30:04.237 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.237 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.237 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.238 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.238 DEBUG:boto:Method: GET 2023-11-02T18:30:04.239 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.239 DEBUG:boto:Data: 2023-11-02T18:30:04.240 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.240 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.241 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.241 DEBUG:boto:Params: {} 2023-11-02T18:30:04.242 DEBUG:boto:Token: None 2023-11-02T18:30:04.242 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.243 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5UTjHCvZZ6RK/zULjuZJOS5048Y= 2023-11-02T18:30:04.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5UTjHCvZZ6RK/zULjuZJOS5048Y=', 'Content-Length': '0'} 2023-11-02T18:30:04.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d64312aa41f4110d-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.290 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.291 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.291 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.292 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.292 DEBUG:boto:Method: GET 2023-11-02T18:30:04.293 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.293 DEBUG:boto:Data: 2023-11-02T18:30:04.294 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.294 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.295 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.295 DEBUG:boto:Params: {} 2023-11-02T18:30:04.296 DEBUG:boto:Token: None 2023-11-02T18:30:04.296 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:04.297 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5UTjHCvZZ6RK/zULjuZJOS5048Y= 2023-11-02T18:30:04.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5UTjHCvZZ6RK/zULjuZJOS5048Y=', 'Content-Length': '0'} 2023-11-02T18:30:04.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f49067401697657e-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.302 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.303 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.303 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:04.304 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:04.304 DEBUG:boto:Method: GET 2023-11-02T18:30:04.304 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:04.305 DEBUG:boto:Data: 2023-11-02T18:30:04.305 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.306 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.306 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.307 DEBUG:boto:Params: {} 2023-11-02T18:30:04.307 DEBUG:boto:Token: None 2023-11-02T18:30:04.308 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.308 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003025aa0cb0e841cb-006543eaac-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.313 DEBUG:boto:b'ibtljn-58obj000000000000000000061000falseobj000000000000000000062023-11-02T18:29:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.314 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.314 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.315 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:04.315 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:04.317 DEBUG:boto:Method: GET 2023-11-02T18:30:04.317 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:04.318 DEBUG:boto:Data: 2023-11-02T18:30:04.318 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.319 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.319 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.320 DEBUG:boto:Params: {} 2023-11-02T18:30:04.320 DEBUG:boto:Token: None 2023-11-02T18:30:04.321 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.321 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcfd8ae45f60e930-006543eaac-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.327 DEBUG:boto:b'ibtljn-58obj000000000000000000061000falseobj000000000000000000062023-11-02T18:29:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.327 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.328 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.328 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.329 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.329 DEBUG:boto:Data: 2023-11-02T18:30:04.329 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.330 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.330 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.331 DEBUG:boto:Params: {} 2023-11-02T18:30:04.331 DEBUG:boto:Token: None 2023-11-02T18:30:04.332 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.332 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dvzfB8B97Hi7q4MgcTeqV5zgtME= 2023-11-02T18:30:04.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dvzfB8B97Hi7q4MgcTeqV5zgtME=', 'Content-Length': '0'} 2023-11-02T18:30:04.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2de931df9a194ec-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.336 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.336 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.337 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.337 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.337 DEBUG:boto:Data: 2023-11-02T18:30:04.338 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.338 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.339 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.339 DEBUG:boto:Params: {} 2023-11-02T18:30:04.340 DEBUG:boto:Token: None 2023-11-02T18:30:04.340 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.341 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dvzfB8B97Hi7q4MgcTeqV5zgtME= 2023-11-02T18:30:04.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dvzfB8B97Hi7q4MgcTeqV5zgtME=', 'Content-Length': '0'} 2023-11-02T18:30:04.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f58b610dccf8060c-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-11-02T18:30:04.345 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.345 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.346 DEBUG:boto:Method: GET 2023-11-02T18:30:04.346 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.347 DEBUG:boto:Data: 2023-11-02T18:30:04.347 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.348 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.348 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.348 DEBUG:boto:Params: {} 2023-11-02T18:30:04.349 DEBUG:boto:Token: None 2023-11-02T18:30:04.349 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.350 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:neFU0Ok6G3aUp2pSDEapzcgx0s4= 2023-11-02T18:30:04.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:neFU0Ok6G3aUp2pSDEapzcgx0s4=', 'Content-Length': '0'} 2023-11-02T18:30:04.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d8c8f61b8015ac2-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.394 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.395 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.395 DEBUG:boto:Method: GET 2023-11-02T18:30:04.396 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.396 DEBUG:boto:Data: 2023-11-02T18:30:04.397 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.397 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.398 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.398 DEBUG:boto:Params: {} 2023-11-02T18:30:04.399 DEBUG:boto:Token: None 2023-11-02T18:30:04.399 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:04.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:neFU0Ok6G3aUp2pSDEapzcgx0s4= 2023-11-02T18:30:04.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:neFU0Ok6G3aUp2pSDEapzcgx0s4=', 'Content-Length': '0'} 2023-11-02T18:30:04.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c70d3c58cfb817be-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.404 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.405 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.405 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:04.406 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:04.406 DEBUG:boto:Method: GET 2023-11-02T18:30:04.407 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:04.407 DEBUG:boto:Data: 2023-11-02T18:30:04.408 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.408 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.408 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.409 DEBUG:boto:Params: {} 2023-11-02T18:30:04.409 DEBUG:boto:Token: None 2023-11-02T18:30:04.410 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.410 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003135da90c86a16b0-006543eaac-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.416 DEBUG:boto:b'ibtljn-58obj000000000000000000061000falseobj000000000000000000062023-11-02T18:29:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.417 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.417 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.418 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:04.418 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:04.419 DEBUG:boto:Method: GET 2023-11-02T18:30:04.419 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:04.420 DEBUG:boto:Data: 2023-11-02T18:30:04.420 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.421 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.421 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.421 DEBUG:boto:Params: {} 2023-11-02T18:30:04.422 DEBUG:boto:Token: None 2023-11-02T18:30:04.423 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.423 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c90b3fbedee288eb-006543eaac-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.429 DEBUG:boto:b'ibtljn-58obj000000000000000000061000falseobj000000000000000000062023-11-02T18:29:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-11-02T18:30:04.431 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.431 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.432 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.432 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.433 DEBUG:boto:Method: GET 2023-11-02T18:30:04.433 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.433 DEBUG:boto:Data: 2023-11-02T18:30:04.434 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.434 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.435 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.435 DEBUG:boto:Params: {} 2023-11-02T18:30:04.436 DEBUG:boto:Token: None 2023-11-02T18:30:04.437 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.437 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v7Q4+hoYK2AUh5XR1lsOIkVxWDg= 2023-11-02T18:30:04.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v7Q4+hoYK2AUh5XR1lsOIkVxWDg=', 'Content-Length': '0'} 2023-11-02T18:30:04.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070bb281adba124ae-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.442 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.442 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.443 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.443 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.444 DEBUG:boto:Method: GET 2023-11-02T18:30:04.444 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.445 DEBUG:boto:Data: 2023-11-02T18:30:04.445 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.446 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.446 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.447 DEBUG:boto:Params: {} 2023-11-02T18:30:04.447 DEBUG:boto:Token: None 2023-11-02T18:30:04.448 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.448 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v7Q4+hoYK2AUh5XR1lsOIkVxWDg= 2023-11-02T18:30:04.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v7Q4+hoYK2AUh5XR1lsOIkVxWDg=', 'Content-Length': '0'} 2023-11-02T18:30:04.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7d74f4931d3060b-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.453 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.453 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.454 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.454 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.455 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.455 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.455 DEBUG:boto:Data: 2023-11-02T18:30:04.456 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.456 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.457 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.457 DEBUG:boto:Params: {} 2023-11-02T18:30:04.458 DEBUG:boto:Token: None 2023-11-02T18:30:04.458 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.459 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zgR6JNj5u7p4nnOIHvWtZsWBu3A= 2023-11-02T18:30:04.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zgR6JNj5u7p4nnOIHvWtZsWBu3A=', 'Content-Length': '0'} 2023-11-02T18:30:04.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef04fa6ba3992ce1-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.463 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.463 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.464 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.464 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.465 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.465 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.466 DEBUG:boto:Data: 2023-11-02T18:30:04.466 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.467 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.467 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.467 DEBUG:boto:Params: {} 2023-11-02T18:30:04.468 DEBUG:boto:Token: None 2023-11-02T18:30:04.468 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.469 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zgR6JNj5u7p4nnOIHvWtZsWBu3A= 2023-11-02T18:30:04.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zgR6JNj5u7p4nnOIHvWtZsWBu3A=', 'Content-Length': '0'} 2023-11-02T18:30:04.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20ef0987d734f98-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-11-02T18:30:04.473 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.474 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.474 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.475 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.475 DEBUG:boto:Method: GET 2023-11-02T18:30:04.476 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.476 DEBUG:boto:Data: 2023-11-02T18:30:04.476 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.477 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.477 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.478 DEBUG:boto:Params: {} 2023-11-02T18:30:04.478 DEBUG:boto:Token: None 2023-11-02T18:30:04.479 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.479 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v7Q4+hoYK2AUh5XR1lsOIkVxWDg= 2023-11-02T18:30:04.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v7Q4+hoYK2AUh5XR1lsOIkVxWDg=', 'Content-Length': '0'} 2023-11-02T18:30:04.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b90f5b725a44b78-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.526 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.527 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.527 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.528 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.528 DEBUG:boto:Method: GET 2023-11-02T18:30:04.529 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.529 DEBUG:boto:Data: 2023-11-02T18:30:04.530 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.530 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.531 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.531 DEBUG:boto:Params: {} 2023-11-02T18:30:04.532 DEBUG:boto:Token: None 2023-11-02T18:30:04.532 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:04.533 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v7Q4+hoYK2AUh5XR1lsOIkVxWDg= 2023-11-02T18:30:04.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v7Q4+hoYK2AUh5XR1lsOIkVxWDg=', 'Content-Length': '0'} 2023-11-02T18:30:04.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099f1fd7854aa57c8-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.537 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.538 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.538 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:04.539 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:04.539 DEBUG:boto:Method: GET 2023-11-02T18:30:04.540 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:04.540 DEBUG:boto:Data: 2023-11-02T18:30:04.541 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.542 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.542 DEBUG:boto:Params: {} 2023-11-02T18:30:04.543 DEBUG:boto:Token: None 2023-11-02T18:30:04.543 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030d752d9cda43669-006543eaac-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.550 DEBUG:boto:b'ibtljn-58obj000000000000000000101000falseobj000000000000000000102023-11-02T18:29:42.175Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.551 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.551 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.552 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:04.552 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:04.553 DEBUG:boto:Method: GET 2023-11-02T18:30:04.553 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:04.554 DEBUG:boto:Data: 2023-11-02T18:30:04.554 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.555 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.555 DEBUG:boto:Params: {} 2023-11-02T18:30:04.556 DEBUG:boto:Token: None 2023-11-02T18:30:04.556 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.557 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5a11d99d2bbd88b-006543eaac-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.563 DEBUG:boto:b'ibtljn-58obj000000000000000000101000falseobj000000000000000000102023-11-02T18:29:42.175Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.564 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.564 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.565 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.565 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.565 DEBUG:boto:Data: 2023-11-02T18:30:04.566 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.566 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.567 DEBUG:boto:Params: {} 2023-11-02T18:30:04.567 DEBUG:boto:Token: None 2023-11-02T18:30:04.568 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.568 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k8XpetboW6Zl8qsi8dacCr+3JmI= 2023-11-02T18:30:04.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k8XpetboW6Zl8qsi8dacCr+3JmI=', 'Content-Length': '0'} 2023-11-02T18:30:04.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000517d603b367d5c3c-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.572 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.573 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.573 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.574 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.574 DEBUG:boto:Data: 2023-11-02T18:30:04.575 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.575 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.576 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.576 DEBUG:boto:Params: {} 2023-11-02T18:30:04.577 DEBUG:boto:Token: None 2023-11-02T18:30:04.577 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.578 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k8XpetboW6Zl8qsi8dacCr+3JmI= 2023-11-02T18:30:04.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k8XpetboW6Zl8qsi8dacCr+3JmI=', 'Content-Length': '0'} 2023-11-02T18:30:04.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007be4aa31d3328fc7-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-11-02T18:30:04.584 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.585 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.585 DEBUG:boto:Method: GET 2023-11-02T18:30:04.586 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.586 DEBUG:boto:Data: 2023-11-02T18:30:04.587 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.587 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.588 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.588 DEBUG:boto:Params: {} 2023-11-02T18:30:04.588 DEBUG:boto:Token: None 2023-11-02T18:30:04.589 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.590 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8yKXki2B8wPv4A0BFIUwCOs6Jxw= 2023-11-02T18:30:04.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8yKXki2B8wPv4A0BFIUwCOs6Jxw=', 'Content-Length': '0'} 2023-11-02T18:30:04.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001261d352df6ae37f-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.634 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.635 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.636 DEBUG:boto:Method: GET 2023-11-02T18:30:04.636 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.636 DEBUG:boto:Data: 2023-11-02T18:30:04.637 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.638 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.638 DEBUG:boto:Params: {} 2023-11-02T18:30:04.639 DEBUG:boto:Token: None 2023-11-02T18:30:04.640 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:04.640 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8yKXki2B8wPv4A0BFIUwCOs6Jxw= 2023-11-02T18:30:04.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8yKXki2B8wPv4A0BFIUwCOs6Jxw=', 'Content-Length': '0'} 2023-11-02T18:30:04.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004db257f7389228b0-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.645 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.645 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.645 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:04.646 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:04.646 DEBUG:boto:Method: GET 2023-11-02T18:30:04.647 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:04.647 DEBUG:boto:Data: 2023-11-02T18:30:04.648 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.648 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.649 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.649 DEBUG:boto:Params: {} 2023-11-02T18:30:04.650 DEBUG:boto:Token: None 2023-11-02T18:30:04.650 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.651 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b6964a87afa0208-006543eaac-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.657 DEBUG:boto:b'ibtljn-58obj000000000000000000101000falseobj000000000000000000102023-11-02T18:29:42.175Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.658 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.659 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.659 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:04.660 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:04.660 DEBUG:boto:Method: GET 2023-11-02T18:30:04.661 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:04.661 DEBUG:boto:Data: 2023-11-02T18:30:04.662 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.662 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.662 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.663 DEBUG:boto:Params: {} 2023-11-02T18:30:04.663 DEBUG:boto:Token: None 2023-11-02T18:30:04.664 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.664 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039dd3f21f143647a-006543eaac-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.671 DEBUG:boto:b'ibtljn-58obj000000000000000000101000falseobj000000000000000000102023-11-02T18:29:42.175Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-11-02T18:30:04.672 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.672 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.673 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.673 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.674 DEBUG:boto:Method: GET 2023-11-02T18:30:04.674 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.675 DEBUG:boto:Data: 2023-11-02T18:30:04.675 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.675 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.676 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.676 DEBUG:boto:Params: {} 2023-11-02T18:30:04.677 DEBUG:boto:Token: None 2023-11-02T18:30:04.677 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.678 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BiHtOORNeUigtmbC1oAwobSJ5aM= 2023-11-02T18:30:04.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BiHtOORNeUigtmbC1oAwobSJ5aM=', 'Content-Length': '0'} 2023-11-02T18:30:04.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c60f7abf2c4c93db-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.683 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.683 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.683 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.684 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.684 DEBUG:boto:Method: GET 2023-11-02T18:30:04.685 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.685 DEBUG:boto:Data: 2023-11-02T18:30:04.686 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.687 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.687 DEBUG:boto:Params: {} 2023-11-02T18:30:04.688 DEBUG:boto:Token: None 2023-11-02T18:30:04.688 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.689 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BiHtOORNeUigtmbC1oAwobSJ5aM= 2023-11-02T18:30:04.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BiHtOORNeUigtmbC1oAwobSJ5aM=', 'Content-Length': '0'} 2023-11-02T18:30:04.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3cef2725fe57239-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.734 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.735 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.735 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.736 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.736 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.737 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.737 DEBUG:boto:Data: 2023-11-02T18:30:04.738 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.739 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.739 DEBUG:boto:Params: {} 2023-11-02T18:30:04.740 DEBUG:boto:Token: None 2023-11-02T18:30:04.740 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.741 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hTXQbyodOj8H/c61SSIwMuIOGnE= 2023-11-02T18:30:04.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hTXQbyodOj8H/c61SSIwMuIOGnE=', 'Content-Length': '0'} 2023-11-02T18:30:04.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab7650711b5fd0ba-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.745 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.745 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.746 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.746 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.747 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.747 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.748 DEBUG:boto:Data: 2023-11-02T18:30:04.748 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.749 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.749 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.750 DEBUG:boto:Params: {} 2023-11-02T18:30:04.750 DEBUG:boto:Token: None 2023-11-02T18:30:04.751 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.751 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hTXQbyodOj8H/c61SSIwMuIOGnE= 2023-11-02T18:30:04.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hTXQbyodOj8H/c61SSIwMuIOGnE=', 'Content-Length': '0'} 2023-11-02T18:30:04.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b15b0548d98beb-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-11-02T18:30:04.756 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.756 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.757 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.757 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.757 DEBUG:boto:Method: GET 2023-11-02T18:30:04.758 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.758 DEBUG:boto:Data: 2023-11-02T18:30:04.759 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.759 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.760 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.760 DEBUG:boto:Params: {} 2023-11-02T18:30:04.761 DEBUG:boto:Token: None 2023-11-02T18:30:04.761 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.762 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BiHtOORNeUigtmbC1oAwobSJ5aM= 2023-11-02T18:30:04.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BiHtOORNeUigtmbC1oAwobSJ5aM=', 'Content-Length': '0'} 2023-11-02T18:30:04.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001599b1d3646cf971-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.806 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.807 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.807 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.808 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.808 DEBUG:boto:Method: GET 2023-11-02T18:30:04.809 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.809 DEBUG:boto:Data: 2023-11-02T18:30:04.810 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.810 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.811 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.811 DEBUG:boto:Params: {} 2023-11-02T18:30:04.812 DEBUG:boto:Token: None 2023-11-02T18:30:04.812 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:04.813 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BiHtOORNeUigtmbC1oAwobSJ5aM= 2023-11-02T18:30:04.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BiHtOORNeUigtmbC1oAwobSJ5aM=', 'Content-Length': '0'} 2023-11-02T18:30:04.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057a0d1ed9fea97a1-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.817 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.818 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.818 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:04.819 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:04.819 DEBUG:boto:Method: GET 2023-11-02T18:30:04.820 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:04.820 DEBUG:boto:Data: 2023-11-02T18:30:04.820 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.821 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.821 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.822 DEBUG:boto:Params: {} 2023-11-02T18:30:04.822 DEBUG:boto:Token: None 2023-11-02T18:30:04.823 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.823 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017da27b1075f6647-006543eaac-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.830 DEBUG:boto:b'ibtljn-58obj000000000000000000111000falseobj000000000000000000112023-11-02T18:29:42.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.831 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.831 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.832 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:04.832 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:04.833 DEBUG:boto:Method: GET 2023-11-02T18:30:04.833 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:04.834 DEBUG:boto:Data: 2023-11-02T18:30:04.834 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.834 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.835 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.835 DEBUG:boto:Params: {} 2023-11-02T18:30:04.836 DEBUG:boto:Token: None 2023-11-02T18:30:04.836 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.837 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071eb80b4f0f1906a-006543eaac-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.843 DEBUG:boto:b'ibtljn-58obj000000000000000000111000falseobj000000000000000000112023-11-02T18:29:42.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.843 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.844 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.844 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.845 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.845 DEBUG:boto:Data: 2023-11-02T18:30:04.846 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.846 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.846 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.847 DEBUG:boto:Params: {} 2023-11-02T18:30:04.847 DEBUG:boto:Token: None 2023-11-02T18:30:04.848 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.848 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u7g0xNQ9HMymSOdXmwo++L17c2k= 2023-11-02T18:30:04.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u7g0xNQ9HMymSOdXmwo++L17c2k=', 'Content-Length': '0'} 2023-11-02T18:30:04.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000828bcdd046bd3f92-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.852 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.853 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.853 DEBUG:boto:Method: HEAD 2023-11-02T18:30:04.854 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.854 DEBUG:boto:Data: 2023-11-02T18:30:04.855 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.855 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.855 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.856 DEBUG:boto:Params: {} 2023-11-02T18:30:04.856 DEBUG:boto:Token: None 2023-11-02T18:30:04.857 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.857 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u7g0xNQ9HMymSOdXmwo++L17c2k= 2023-11-02T18:30:04.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u7g0xNQ9HMymSOdXmwo++L17c2k=', 'Content-Length': '0'} 2023-11-02T18:30:04.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c973c95e7ba0d66-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-11-02T18:30:04.862 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.862 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.863 DEBUG:boto:Method: GET 2023-11-02T18:30:04.863 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.864 DEBUG:boto:Data: 2023-11-02T18:30:04.864 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.865 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.865 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.865 DEBUG:boto:Params: {} 2023-11-02T18:30:04.866 DEBUG:boto:Token: None 2023-11-02T18:30:04.866 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.867 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LfQbKEV4Lne1snQPZXLqtBFaTR0= 2023-11-02T18:30:04.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LfQbKEV4Lne1snQPZXLqtBFaTR0=', 'Content-Length': '0'} 2023-11-02T18:30:04.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003611e4d5c946629e-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.914 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.915 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.916 DEBUG:boto:Method: GET 2023-11-02T18:30:04.916 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.917 DEBUG:boto:Data: 2023-11-02T18:30:04.917 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.918 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.919 DEBUG:boto:Params: {} 2023-11-02T18:30:04.919 DEBUG:boto:Token: None 2023-11-02T18:30:04.920 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:04.920 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LfQbKEV4Lne1snQPZXLqtBFaTR0= 2023-11-02T18:30:04.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LfQbKEV4Lne1snQPZXLqtBFaTR0=', 'Content-Length': '0'} 2023-11-02T18:30:04.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000724b25017daa3469-006543eaac-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.925 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.925 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.926 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:04.926 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:04.927 DEBUG:boto:Method: GET 2023-11-02T18:30:04.927 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:04.927 DEBUG:boto:Data: 2023-11-02T18:30:04.928 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.928 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:04.929 DEBUG:boto:Port: 8001 2023-11-02T18:30:04.929 DEBUG:boto:Params: {} 2023-11-02T18:30:04.930 DEBUG:boto:Token: None 2023-11-02T18:30:04.930 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.931 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008438fc34e18a17d7-006543eaac-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.938 DEBUG:boto:b'ibtljn-58obj000000000000000000111000falseobj000000000000000000112023-11-02T18:29:42.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.939 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:04.939 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:04.940 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:04.940 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:04.941 DEBUG:boto:Method: GET 2023-11-02T18:30:04.941 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:04.942 DEBUG:boto:Data: 2023-11-02T18:30:04.942 DEBUG:boto:Headers: {} 2023-11-02T18:30:04.943 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:04.943 DEBUG:boto:Port: 8000 2023-11-02T18:30:04.943 DEBUG:boto:Params: {} 2023-11-02T18:30:04.944 DEBUG:boto:Token: None 2023-11-02T18:30:04.944 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:04 GMT /ibtljn-58/ 2023-11-02T18:30:04.945 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo= 2023-11-02T18:30:04.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mfeW5w89Idw16UJU16DIqbpunXo=', 'Content-Length': '0'} 2023-11-02T18:30:04.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef796579d8088b0a-006543eaac-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:04.994 DEBUG:boto:b'ibtljn-58obj000000000000000000111000falseobj000000000000000000112023-11-02T18:29:42.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:04.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-11-02T18:30:04.996 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:04.996 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:04.997 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:04.997 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:04.998 DEBUG:boto:Method: GET 2023-11-02T18:30:04.998 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:04.999 DEBUG:boto:Data: 2023-11-02T18:30:04.999 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.000 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.000 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.001 DEBUG:boto:Params: {} 2023-11-02T18:30:05.002 DEBUG:boto:Token: None 2023-11-02T18:30:05.002 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UWSSj+K/lKbJf83odQowAcVyNf4= 2023-11-02T18:30:05.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UWSSj+K/lKbJf83odQowAcVyNf4=', 'Content-Length': '0'} 2023-11-02T18:30:05.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038d5454c901ceb34-006543eaac-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.007 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.008 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.008 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.009 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.009 DEBUG:boto:Method: GET 2023-11-02T18:30:05.010 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.010 DEBUG:boto:Data: 2023-11-02T18:30:05.011 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.011 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.012 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.012 DEBUG:boto:Params: {} 2023-11-02T18:30:05.013 DEBUG:boto:Token: None 2023-11-02T18:30:05.013 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.014 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UWSSj+K/lKbJf83odQowAcVyNf4= 2023-11-02T18:30:05.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UWSSj+K/lKbJf83odQowAcVyNf4=', 'Content-Length': '0'} 2023-11-02T18:30:05.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006efcec9e82333b67-006543eaad-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.018 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.019 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.019 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.020 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.020 DEBUG:boto:Method: HEAD 2023-11-02T18:30:05.021 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.021 DEBUG:boto:Data: 2023-11-02T18:30:05.022 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.022 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.023 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.023 DEBUG:boto:Params: {} 2023-11-02T18:30:05.024 DEBUG:boto:Token: None 2023-11-02T18:30:05.024 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.025 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7/y57AwiM/NzB7urvVaUoX94gAQ= 2023-11-02T18:30:05.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7/y57AwiM/NzB7urvVaUoX94gAQ=', 'Content-Length': '0'} 2023-11-02T18:30:05.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0ca634c2b535a7f-006543eaad-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.029 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.029 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.030 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.030 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.030 DEBUG:boto:Method: HEAD 2023-11-02T18:30:05.031 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.031 DEBUG:boto:Data: 2023-11-02T18:30:05.032 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.033 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.033 DEBUG:boto:Params: {} 2023-11-02T18:30:05.034 DEBUG:boto:Token: None 2023-11-02T18:30:05.035 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7/y57AwiM/NzB7urvVaUoX94gAQ= 2023-11-02T18:30:05.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7/y57AwiM/NzB7urvVaUoX94gAQ=', 'Content-Length': '0'} 2023-11-02T18:30:05.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000341a5e880dffa40d-006543eaad-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-11-02T18:30:05.040 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.040 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.041 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.041 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.042 DEBUG:boto:Method: GET 2023-11-02T18:30:05.042 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.043 DEBUG:boto:Data: 2023-11-02T18:30:05.043 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.044 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.045 DEBUG:boto:Params: {} 2023-11-02T18:30:05.046 DEBUG:boto:Token: None 2023-11-02T18:30:05.046 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.047 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UWSSj+K/lKbJf83odQowAcVyNf4= 2023-11-02T18:30:05.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UWSSj+K/lKbJf83odQowAcVyNf4=', 'Content-Length': '0'} 2023-11-02T18:30:05.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc759ef72a65272-006543eaad-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.094 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.095 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.095 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.096 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.096 DEBUG:boto:Method: GET 2023-11-02T18:30:05.097 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.097 DEBUG:boto:Data: 2023-11-02T18:30:05.098 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.098 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.099 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.099 DEBUG:boto:Params: {} 2023-11-02T18:30:05.100 DEBUG:boto:Token: None 2023-11-02T18:30:05.101 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:05.101 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UWSSj+K/lKbJf83odQowAcVyNf4= 2023-11-02T18:30:05.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UWSSj+K/lKbJf83odQowAcVyNf4=', 'Content-Length': '0'} 2023-11-02T18:30:05.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007785f51c99283294-006543eaad-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.106 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:05.106 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:05.107 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:05.107 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:05.108 DEBUG:boto:Method: GET 2023-11-02T18:30:05.108 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:05.109 DEBUG:boto:Data: 2023-11-02T18:30:05.109 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.110 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.110 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.111 DEBUG:boto:Params: {} 2023-11-02T18:30:05.111 DEBUG:boto:Token: None 2023-11-02T18:30:05.112 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/ 2023-11-02T18:30:05.112 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0= 2023-11-02T18:30:05.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0=', 'Content-Length': '0'} 2023-11-02T18:30:05.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9efc5819163bb8e-006543eaad-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.119 DEBUG:boto:b'ibtljn-58obj000000000000000000121000falseobj000000000000000000122023-11-02T18:29:42.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:05.120 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:05.120 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:05.121 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:05.121 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:05.122 DEBUG:boto:Method: GET 2023-11-02T18:30:05.122 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:05.123 DEBUG:boto:Data: 2023-11-02T18:30:05.123 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.124 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.124 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.125 DEBUG:boto:Params: {} 2023-11-02T18:30:05.125 DEBUG:boto:Token: None 2023-11-02T18:30:05.126 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/ 2023-11-02T18:30:05.126 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0= 2023-11-02T18:30:05.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0=', 'Content-Length': '0'} 2023-11-02T18:30:05.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e431947a55531577-006543eaad-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.132 DEBUG:boto:b'ibtljn-58obj000000000000000000121000falseobj000000000000000000122023-11-02T18:29:42.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:05.133 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.133 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.134 DEBUG:boto:Method: HEAD 2023-11-02T18:30:05.134 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.135 DEBUG:boto:Data: 2023-11-02T18:30:05.135 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.136 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.137 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.137 DEBUG:boto:Params: {} 2023-11-02T18:30:05.138 DEBUG:boto:Token: None 2023-11-02T18:30:05.138 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.139 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BZojemRW5st/OrO+HuDrU5Xte9s= 2023-11-02T18:30:05.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BZojemRW5st/OrO+HuDrU5Xte9s=', 'Content-Length': '0'} 2023-11-02T18:30:05.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8c9d5f092d4a357-006543eaad-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.145 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.146 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.146 DEBUG:boto:Method: HEAD 2023-11-02T18:30:05.147 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.147 DEBUG:boto:Data: 2023-11-02T18:30:05.148 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.149 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.149 DEBUG:boto:Params: {} 2023-11-02T18:30:05.149 DEBUG:boto:Token: None 2023-11-02T18:30:05.150 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.150 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BZojemRW5st/OrO+HuDrU5Xte9s= 2023-11-02T18:30:05.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BZojemRW5st/OrO+HuDrU5Xte9s=', 'Content-Length': '0'} 2023-11-02T18:30:05.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7d38ff7e5d75607-006543eaad-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-11-02T18:30:05.155 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.155 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.156 DEBUG:boto:Method: GET 2023-11-02T18:30:05.156 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.157 DEBUG:boto:Data: 2023-11-02T18:30:05.157 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.158 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.158 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.159 DEBUG:boto:Params: {} 2023-11-02T18:30:05.159 DEBUG:boto:Token: None 2023-11-02T18:30:05.160 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.160 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EFTRUOJpxvFLfOS2FKD5V5pZWpc= 2023-11-02T18:30:05.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EFTRUOJpxvFLfOS2FKD5V5pZWpc=', 'Content-Length': '0'} 2023-11-02T18:30:05.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001865f2b5b8aba549-006543eaad-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.206 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.207 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.207 DEBUG:boto:Method: GET 2023-11-02T18:30:05.208 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.208 DEBUG:boto:Data: 2023-11-02T18:30:05.209 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.209 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.210 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.210 DEBUG:boto:Params: {} 2023-11-02T18:30:05.211 DEBUG:boto:Token: None 2023-11-02T18:30:05.211 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:05.212 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EFTRUOJpxvFLfOS2FKD5V5pZWpc= 2023-11-02T18:30:05.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EFTRUOJpxvFLfOS2FKD5V5pZWpc=', 'Content-Length': '0'} 2023-11-02T18:30:05.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000948f698fd47b978d-006543eaad-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.216 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:05.217 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:05.217 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:05.218 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:05.218 DEBUG:boto:Method: GET 2023-11-02T18:30:05.219 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:05.219 DEBUG:boto:Data: 2023-11-02T18:30:05.219 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.220 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.220 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.221 DEBUG:boto:Params: {} 2023-11-02T18:30:05.221 DEBUG:boto:Token: None 2023-11-02T18:30:05.222 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/ 2023-11-02T18:30:05.222 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0= 2023-11-02T18:30:05.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0=', 'Content-Length': '0'} 2023-11-02T18:30:05.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff7252e25779b521-006543eaad-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.228 DEBUG:boto:b'ibtljn-58obj000000000000000000121000falseobj000000000000000000122023-11-02T18:29:42.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:05.229 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:05.229 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:05.230 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:05.230 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:05.231 DEBUG:boto:Method: GET 2023-11-02T18:30:05.231 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:05.232 DEBUG:boto:Data: 2023-11-02T18:30:05.232 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.232 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.233 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.233 DEBUG:boto:Params: {} 2023-11-02T18:30:05.234 DEBUG:boto:Token: None 2023-11-02T18:30:05.234 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/ 2023-11-02T18:30:05.235 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0= 2023-11-02T18:30:05.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0=', 'Content-Length': '0'} 2023-11-02T18:30:05.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000657a430765a5a55a-006543eaad-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.282 DEBUG:boto:b'ibtljn-58obj000000000000000000121000falseobj000000000000000000122023-11-02T18:29:42.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:05.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-11-02T18:30:05.284 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.284 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.285 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.285 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.285 DEBUG:boto:Method: GET 2023-11-02T18:30:05.286 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.286 DEBUG:boto:Data: 2023-11-02T18:30:05.287 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.287 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.288 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.288 DEBUG:boto:Params: {} 2023-11-02T18:30:05.289 DEBUG:boto:Token: None 2023-11-02T18:30:05.289 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.290 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cOBiXO4MFCz/+rFaiYnBWo/x+l0= 2023-11-02T18:30:05.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cOBiXO4MFCz/+rFaiYnBWo/x+l0=', 'Content-Length': '0'} 2023-11-02T18:30:05.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006346246216e65866-006543eaad-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.294 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.295 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.295 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.296 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.297 DEBUG:boto:Method: GET 2023-11-02T18:30:05.297 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.297 DEBUG:boto:Data: 2023-11-02T18:30:05.298 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.298 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.299 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.299 DEBUG:boto:Params: {} 2023-11-02T18:30:05.300 DEBUG:boto:Token: None 2023-11-02T18:30:05.300 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.301 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cOBiXO4MFCz/+rFaiYnBWo/x+l0= 2023-11-02T18:30:05.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cOBiXO4MFCz/+rFaiYnBWo/x+l0=', 'Content-Length': '0'} 2023-11-02T18:30:05.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c140e768133072c7-006543eaad-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.305 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.306 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.306 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.306 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.307 DEBUG:boto:Method: HEAD 2023-11-02T18:30:05.307 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.308 DEBUG:boto:Data: 2023-11-02T18:30:05.308 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.309 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.309 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.309 DEBUG:boto:Params: {} 2023-11-02T18:30:05.310 DEBUG:boto:Token: None 2023-11-02T18:30:05.310 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.311 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZE5kuzm9g6iL2KUYqHFbYjZa/c= 2023-11-02T18:30:05.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZE5kuzm9g6iL2KUYqHFbYjZa/c=', 'Content-Length': '0'} 2023-11-02T18:30:05.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009139b9bc04bd44d9-006543eaad-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.315 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.315 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.316 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.316 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.316 DEBUG:boto:Method: HEAD 2023-11-02T18:30:05.317 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.317 DEBUG:boto:Data: 2023-11-02T18:30:05.318 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.318 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.318 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.319 DEBUG:boto:Params: {} 2023-11-02T18:30:05.319 DEBUG:boto:Token: None 2023-11-02T18:30:05.320 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZE5kuzm9g6iL2KUYqHFbYjZa/c= 2023-11-02T18:30:05.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZE5kuzm9g6iL2KUYqHFbYjZa/c=', 'Content-Length': '0'} 2023-11-02T18:30:05.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e687bc5418c459e-006543eaad-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-11-02T18:30:05.325 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.325 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.325 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.326 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.326 DEBUG:boto:Method: GET 2023-11-02T18:30:05.327 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.327 DEBUG:boto:Data: 2023-11-02T18:30:05.328 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.328 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.329 DEBUG:boto:Params: {} 2023-11-02T18:30:05.329 DEBUG:boto:Token: None 2023-11-02T18:30:05.330 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.330 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cOBiXO4MFCz/+rFaiYnBWo/x+l0= 2023-11-02T18:30:05.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cOBiXO4MFCz/+rFaiYnBWo/x+l0=', 'Content-Length': '0'} 2023-11-02T18:30:05.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000417af3685819ddd2-006543eaad-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.335 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.335 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.336 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.336 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.337 DEBUG:boto:Method: GET 2023-11-02T18:30:05.337 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.338 DEBUG:boto:Data: 2023-11-02T18:30:05.338 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.338 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.339 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.339 DEBUG:boto:Params: {} 2023-11-02T18:30:05.340 DEBUG:boto:Token: None 2023-11-02T18:30:05.340 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:05.341 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cOBiXO4MFCz/+rFaiYnBWo/x+l0= 2023-11-02T18:30:05.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cOBiXO4MFCz/+rFaiYnBWo/x+l0=', 'Content-Length': '0'} 2023-11-02T18:30:05.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dce5031f0acf3707-006543eaad-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.345 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:05.345 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:05.346 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:05.349 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:05.350 DEBUG:boto:Method: GET 2023-11-02T18:30:05.350 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:05.350 DEBUG:boto:Data: 2023-11-02T18:30:05.351 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.351 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.352 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.352 DEBUG:boto:Params: {} 2023-11-02T18:30:05.353 DEBUG:boto:Token: None 2023-11-02T18:30:05.353 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/ 2023-11-02T18:30:05.354 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0= 2023-11-02T18:30:05.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0=', 'Content-Length': '0'} 2023-11-02T18:30:05.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eff389dcfc1022e-006543eaad-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.361 DEBUG:boto:b'ibtljn-58obj000000000000000000131000falseobj000000000000000000132023-11-02T18:29:42.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:05.362 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:05.362 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:05.363 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:05.363 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:05.364 DEBUG:boto:Method: GET 2023-11-02T18:30:05.364 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:05.364 DEBUG:boto:Data: 2023-11-02T18:30:05.365 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.365 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.366 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.366 DEBUG:boto:Params: {} 2023-11-02T18:30:05.367 DEBUG:boto:Token: None 2023-11-02T18:30:05.367 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/ 2023-11-02T18:30:05.368 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0= 2023-11-02T18:30:05.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0=', 'Content-Length': '0'} 2023-11-02T18:30:05.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056b6d92fa1ff75c5-006543eaad-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.373 DEBUG:boto:b'ibtljn-58obj000000000000000000131000falseobj000000000000000000132023-11-02T18:29:42.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:05.374 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.374 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.375 DEBUG:boto:Method: HEAD 2023-11-02T18:30:05.375 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.376 DEBUG:boto:Data: 2023-11-02T18:30:05.376 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.376 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.377 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.377 DEBUG:boto:Params: {} 2023-11-02T18:30:05.378 DEBUG:boto:Token: None 2023-11-02T18:30:05.378 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.379 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oobOGCe0tqQtDwlS+IgaEsnickQ= 2023-11-02T18:30:05.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oobOGCe0tqQtDwlS+IgaEsnickQ=', 'Content-Length': '0'} 2023-11-02T18:30:05.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d339a3dc0d865abe-006543eaad-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.383 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.383 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.383 DEBUG:boto:Method: HEAD 2023-11-02T18:30:05.384 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.384 DEBUG:boto:Data: 2023-11-02T18:30:05.385 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.385 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.386 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.386 DEBUG:boto:Params: {} 2023-11-02T18:30:05.386 DEBUG:boto:Token: None 2023-11-02T18:30:05.387 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.388 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oobOGCe0tqQtDwlS+IgaEsnickQ= 2023-11-02T18:30:05.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oobOGCe0tqQtDwlS+IgaEsnickQ=', 'Content-Length': '0'} 2023-11-02T18:30:05.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005598bc6e669461d6-006543eaad-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-11-02T18:30:05.392 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.392 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.393 DEBUG:boto:Method: GET 2023-11-02T18:30:05.393 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.394 DEBUG:boto:Data: 2023-11-02T18:30:05.394 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.395 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.395 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.396 DEBUG:boto:Params: {} 2023-11-02T18:30:05.396 DEBUG:boto:Token: None 2023-11-02T18:30:05.397 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.397 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8Gm4RRI1ED8fNT02kj9snSWUcJ8= 2023-11-02T18:30:05.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8Gm4RRI1ED8fNT02kj9snSWUcJ8=', 'Content-Length': '0'} 2023-11-02T18:30:05.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097971fbca6ef4aab-006543eaad-1ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.401 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.402 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.402 DEBUG:boto:Method: GET 2023-11-02T18:30:05.403 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.403 DEBUG:boto:Data: 2023-11-02T18:30:05.403 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.404 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.404 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.405 DEBUG:boto:Params: {} 2023-11-02T18:30:05.405 DEBUG:boto:Token: None 2023-11-02T18:30:05.406 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:05.406 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8Gm4RRI1ED8fNT02kj9snSWUcJ8= 2023-11-02T18:30:05.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8Gm4RRI1ED8fNT02kj9snSWUcJ8=', 'Content-Length': '0'} 2023-11-02T18:30:05.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:29:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000343282c1ef83caea-006543eaad-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.450 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:05.451 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:05.452 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:05.452 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:05.452 DEBUG:boto:Method: GET 2023-11-02T18:30:05.453 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:05.453 DEBUG:boto:Data: 2023-11-02T18:30:05.454 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:05.455 DEBUG:boto:Port: 8001 2023-11-02T18:30:05.455 DEBUG:boto:Params: {} 2023-11-02T18:30:05.456 DEBUG:boto:Token: None 2023-11-02T18:30:05.456 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/ 2023-11-02T18:30:05.457 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0= 2023-11-02T18:30:05.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0=', 'Content-Length': '0'} 2023-11-02T18:30:05.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000450d8eb965f539e6-006543eaad-1ac5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.464 DEBUG:boto:b'ibtljn-58obj000000000000000000131000falseobj000000000000000000132023-11-02T18:29:42.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:05.464 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:05.465 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:05.465 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:05.466 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:05.466 DEBUG:boto:Method: GET 2023-11-02T18:30:05.467 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:05.467 DEBUG:boto:Data: 2023-11-02T18:30:05.467 DEBUG:boto:Headers: {} 2023-11-02T18:30:05.468 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:05.468 DEBUG:boto:Port: 8000 2023-11-02T18:30:05.469 DEBUG:boto:Params: {} 2023-11-02T18:30:05.469 DEBUG:boto:Token: None 2023-11-02T18:30:05.470 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:05 GMT /ibtljn-58/ 2023-11-02T18:30:05.470 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0= 2023-11-02T18:30:05.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pEefZtWB7oQir4+7VFRUXNgQZE0=', 'Content-Length': '0'} 2023-11-02T18:30:05.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032253d450d1ebbad-006543eaad-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:05.477 DEBUG:boto:b'ibtljn-58obj000000000000000000131000falseobj000000000000000000132023-11-02T18:29:42.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:05.478 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-58 zones={a2, a1} 2023-11-02T18:30:05.479 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-11-02T18:30:05.479 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T18:30:05.480 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:30:05.472+0000 7fec036a7700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 63470) UID: 0 2023-11-02T18:30:05.480 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:30:05.472+0000 7fec06f06f80 -1 shutting down 2023-11-02T18:30:07.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-11-02T18:30:11.578 INFO:tasks.rgw.c1.client.1:Stopped 2023-11-02T18:30:11.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:11.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:11.581 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-58 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:30:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:11.704 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:30:11.705 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-58 2023-11-02T18:30:11.713 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 10 2023-11-02T18:30:11.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:11.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:11.797 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:11.842 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:11.843 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:11.926 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:30:11.927 INFO:teuthology.orchestra.run.smithi017.stdout: "objs": [ 2023-11-02T18:30:11.927 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000007", 2023-11-02T18:30:11.927 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000016", 2023-11-02T18:30:11.927 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000024", 2023-11-02T18:30:11.928 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000020", 2023-11-02T18:30:11.928 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000028", 2023-11-02T18:30:11.928 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000000", 2023-11-02T18:30:11.928 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000032", 2023-11-02T18:30:11.929 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000001", 2023-11-02T18:30:11.929 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000010", 2023-11-02T18:30:11.929 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000019", 2023-11-02T18:30:11.929 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000013", 2023-11-02T18:30:11.929 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000021", 2023-11-02T18:30:11.930 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000030", 2023-11-02T18:30:11.930 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000026", 2023-11-02T18:30:11.930 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000033", 2023-11-02T18:30:11.930 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000004", 2023-11-02T18:30:11.931 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000076" 2023-11-02T18:30:11.931 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:30:11.931 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:30:11.932 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:11.933 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:11.933 DEBUG:boto:Method: HEAD 2023-11-02T18:30:11.933 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:11.934 DEBUG:boto:Data: 2023-11-02T18:30:11.934 DEBUG:boto:Headers: {} 2023-11-02T18:30:11.935 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:11.935 DEBUG:boto:Port: 8000 2023-11-02T18:30:11.936 DEBUG:boto:Params: {} 2023-11-02T18:30:11.936 DEBUG:boto:Token: None 2023-11-02T18:30:11.936 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:11 GMT /ibtljn-58/ 2023-11-02T18:30:11.937 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1cJf+CxSCUfD4YBko32+vG9o2QQ= 2023-11-02T18:30:11.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1cJf+CxSCUfD4YBko32+vG9o2QQ=', 'Content-Length': '0'} 2023-11-02T18:30:11.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1ffa23543e907bc-006543eab3-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:11.943 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:11.943 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:11.944 DEBUG:boto:Method: PUT 2023-11-02T18:30:11.944 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:11.945 DEBUG:boto:Data: 2023-11-02T18:30:11.945 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:11.945 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:11.946 DEBUG:boto:Port: 8000 2023-11-02T18:30:11.946 DEBUG:boto:Params: {} 2023-11-02T18:30:11.947 DEBUG:boto:Token: None 2023-11-02T18:30:11.947 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:11 GMT /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:11.948 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+2+vquOeCHXBvkjzPNPGAb+CyfE= 2023-11-02T18:30:11.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+2+vquOeCHXBvkjzPNPGAb+CyfE='} 2023-11-02T18:30:11.959 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000806be2d55444dfd6-006543eab3-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:11.959 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:11.960 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:11.960 DEBUG:boto:Method: HEAD 2023-11-02T18:30:11.961 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:11.961 DEBUG:boto:Data: 2023-11-02T18:30:11.961 DEBUG:boto:Headers: {} 2023-11-02T18:30:11.962 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:11.962 DEBUG:boto:Port: 8000 2023-11-02T18:30:11.963 DEBUG:boto:Params: {} 2023-11-02T18:30:11.963 DEBUG:boto:Token: None 2023-11-02T18:30:11.964 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:11 GMT /ibtljn-58/ 2023-11-02T18:30:11.964 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1cJf+CxSCUfD4YBko32+vG9o2QQ= 2023-11-02T18:30:11.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1cJf+CxSCUfD4YBko32+vG9o2QQ=', 'Content-Length': '0'} 2023-11-02T18:30:11.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dae6ebf9718453a5-006543eab3-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:11.970 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:11.971 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:11.971 DEBUG:boto:Method: PUT 2023-11-02T18:30:11.972 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:11.972 DEBUG:boto:Data: 2023-11-02T18:30:11.972 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:11.973 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:11.973 DEBUG:boto:Port: 8000 2023-11-02T18:30:11.974 DEBUG:boto:Params: {} 2023-11-02T18:30:11.974 DEBUG:boto:Token: None 2023-11-02T18:30:11.975 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:11 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:11.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hh3Lci6LDCeKw1sVzzBMsBIjfPk= 2023-11-02T18:30:11.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hh3Lci6LDCeKw1sVzzBMsBIjfPk='} 2023-11-02T18:30:12.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc9451210866b8df-006543eab3-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.023 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.024 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.024 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.025 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.025 DEBUG:boto:Data: 2023-11-02T18:30:12.026 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.026 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.026 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.027 DEBUG:boto:Params: {} 2023-11-02T18:30:12.027 DEBUG:boto:Token: None 2023-11-02T18:30:12.028 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.028 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.034 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000524a2bd3bddf8249-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.034 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:12.035 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:12.035 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.036 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:12.036 DEBUG:boto:Data: 2023-11-02T18:30:12.036 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.037 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.037 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.038 DEBUG:boto:Params: {} 2023-11-02T18:30:12.038 DEBUG:boto:Token: None 2023-11-02T18:30:12.039 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:12.039 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0+elc4d48DRTwP//mHL4DQ81Q10= 2023-11-02T18:30:12.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0+elc4d48DRTwP//mHL4DQ81Q10='} 2023-11-02T18:30:12.053 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a853c3c985696e98-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.053 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.054 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.054 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.055 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.055 DEBUG:boto:Data: 2023-11-02T18:30:12.056 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.056 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.056 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.057 DEBUG:boto:Params: {} 2023-11-02T18:30:12.057 DEBUG:boto:Token: None 2023-11-02T18:30:12.058 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.058 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e46656b963f978e3-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.065 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:12.065 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:12.066 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.066 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:12.067 DEBUG:boto:Data: 2023-11-02T18:30:12.067 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.068 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.069 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.069 DEBUG:boto:Params: {} 2023-11-02T18:30:12.070 DEBUG:boto:Token: None 2023-11-02T18:30:12.071 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:12.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xfK2opM8ejieDv7jaiLAkeAo4gE= 2023-11-02T18:30:12.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xfK2opM8ejieDv7jaiLAkeAo4gE='} 2023-11-02T18:30:12.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059680cf316b5798b-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.124 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.124 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.125 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.125 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.126 DEBUG:boto:Data: 2023-11-02T18:30:12.126 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.126 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.127 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.127 DEBUG:boto:Params: {} 2023-11-02T18:30:12.128 DEBUG:boto:Token: None 2023-11-02T18:30:12.128 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c050c3f4eba77a8-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.136 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:12.137 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:12.137 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.138 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:12.138 DEBUG:boto:Data: 2023-11-02T18:30:12.139 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.140 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.140 DEBUG:boto:Params: {} 2023-11-02T18:30:12.141 DEBUG:boto:Token: None 2023-11-02T18:30:12.141 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:12.142 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XhI6txXO8ptPZRjYENq0zucOgP4= 2023-11-02T18:30:12.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XhI6txXO8ptPZRjYENq0zucOgP4='} 2023-11-02T18:30:12.155 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000765b5f4b782eeff8-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.155 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.156 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.156 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.157 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.157 DEBUG:boto:Data: 2023-11-02T18:30:12.157 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.158 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.158 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.159 DEBUG:boto:Params: {} 2023-11-02T18:30:12.159 DEBUG:boto:Token: None 2023-11-02T18:30:12.160 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.160 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab8017b39f2a717d-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.166 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:12.166 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:12.167 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.167 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000076 2023-11-02T18:30:12.168 DEBUG:boto:Data: 2023-11-02T18:30:12.168 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.169 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.169 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.169 DEBUG:boto:Params: {} 2023-11-02T18:30:12.170 DEBUG:boto:Token: None 2023-11-02T18:30:12.170 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000076 2023-11-02T18:30:12.171 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CzBuK44enuWdgx+9KGjbFyS+ARQ= 2023-11-02T18:30:12.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CzBuK44enuWdgx+9KGjbFyS+ARQ='} 2023-11-02T18:30:12.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf16a1e6d166d2f8-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.215 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.216 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.216 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.217 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.217 DEBUG:boto:Data: 2023-11-02T18:30:12.217 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.218 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.218 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.219 DEBUG:boto:Params: {} 2023-11-02T18:30:12.219 DEBUG:boto:Token: None 2023-11-02T18:30:12.220 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.220 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000676579c9d0a9030b-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.226 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:12.227 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:12.227 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.228 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000032 2023-11-02T18:30:12.228 DEBUG:boto:Data: 2023-11-02T18:30:12.228 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.229 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.229 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.230 DEBUG:boto:Params: {} 2023-11-02T18:30:12.230 DEBUG:boto:Token: None 2023-11-02T18:30:12.231 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000032 2023-11-02T18:30:12.231 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g4yrGWH8zE3Ni4Ic3S6YQ9GmD/o= 2023-11-02T18:30:12.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g4yrGWH8zE3Ni4Ic3S6YQ9GmD/o='} 2023-11-02T18:30:12.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efd6b6f2b3b2c1be-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.244 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.244 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.244 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.245 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.245 DEBUG:boto:Data: 2023-11-02T18:30:12.246 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.246 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.247 DEBUG:boto:Params: {} 2023-11-02T18:30:12.247 DEBUG:boto:Token: None 2023-11-02T18:30:12.248 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.248 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007389f609fec9fba5-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.254 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:12.254 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:12.255 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.255 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:12.256 DEBUG:boto:Data: 2023-11-02T18:30:12.256 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.256 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.257 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.257 DEBUG:boto:Params: {} 2023-11-02T18:30:12.258 DEBUG:boto:Token: None 2023-11-02T18:30:12.258 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:12.259 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qGFVdOqr5zdUlwLaCAtQwlIBIu4= 2023-11-02T18:30:12.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qGFVdOqr5zdUlwLaCAtQwlIBIu4='} 2023-11-02T18:30:12.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061830e95405a2255-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.304 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.304 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.305 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.305 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.305 DEBUG:boto:Data: 2023-11-02T18:30:12.306 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.306 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.307 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.307 DEBUG:boto:Params: {} 2023-11-02T18:30:12.308 DEBUG:boto:Token: None 2023-11-02T18:30:12.308 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.309 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fe023c8491a4364-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.315 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:12.315 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:12.315 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.316 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:12.316 DEBUG:boto:Data: 2023-11-02T18:30:12.317 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.318 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.318 DEBUG:boto:Params: {} 2023-11-02T18:30:12.319 DEBUG:boto:Token: None 2023-11-02T18:30:12.319 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:12.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lPIEdoC4iLHvj56OUaO1BxFSh8w= 2023-11-02T18:30:12.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lPIEdoC4iLHvj56OUaO1BxFSh8w='} 2023-11-02T18:30:12.331 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000208dd296fbc287b0-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.331 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.332 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.332 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.333 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.333 DEBUG:boto:Data: 2023-11-02T18:30:12.333 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.334 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.334 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.335 DEBUG:boto:Params: {} 2023-11-02T18:30:12.335 DEBUG:boto:Token: None 2023-11-02T18:30:12.336 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.336 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f55984adbac93219-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.342 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:12.343 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:12.343 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.343 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:12.344 DEBUG:boto:Data: 2023-11-02T18:30:12.344 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.345 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.345 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.345 DEBUG:boto:Params: {} 2023-11-02T18:30:12.346 DEBUG:boto:Token: None 2023-11-02T18:30:12.346 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:12.347 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gkHQW9NvIo+RCA+upT0mYAfIgz4= 2023-11-02T18:30:12.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gkHQW9NvIo+RCA+upT0mYAfIgz4='} 2023-11-02T18:30:12.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f705ebe96f7821c-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.391 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.392 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.392 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.393 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.393 DEBUG:boto:Data: 2023-11-02T18:30:12.393 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.394 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.394 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.395 DEBUG:boto:Params: {} 2023-11-02T18:30:12.395 DEBUG:boto:Token: None 2023-11-02T18:30:12.396 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.396 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.402 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c29c1e3624acafb-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.403 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:12.404 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:12.404 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.404 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:12.405 DEBUG:boto:Data: 2023-11-02T18:30:12.405 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.406 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.406 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.406 DEBUG:boto:Params: {} 2023-11-02T18:30:12.407 DEBUG:boto:Token: None 2023-11-02T18:30:12.407 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:12.408 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hp4IbJ5e/4ahqqxhQr4c9AQSp7Q= 2023-11-02T18:30:12.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hp4IbJ5e/4ahqqxhQr4c9AQSp7Q='} 2023-11-02T18:30:12.421 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cb86c9d2ce45d23-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.421 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.422 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.422 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.423 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.423 DEBUG:boto:Data: 2023-11-02T18:30:12.423 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.424 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.424 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.425 DEBUG:boto:Params: {} 2023-11-02T18:30:12.425 DEBUG:boto:Token: None 2023-11-02T18:30:12.426 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.426 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf6d73a557d3efe2-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.432 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:12.433 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:12.433 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.434 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:12.434 DEBUG:boto:Data: 2023-11-02T18:30:12.435 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.436 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.436 DEBUG:boto:Params: {} 2023-11-02T18:30:12.437 DEBUG:boto:Token: None 2023-11-02T18:30:12.437 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:12.438 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CjKosmPTNfQ9rPAHmKMvF/rtGLE= 2023-11-02T18:30:12.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CjKosmPTNfQ9rPAHmKMvF/rtGLE='} 2023-11-02T18:30:12.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6c046f917e9adcd-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.487 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.488 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.488 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.489 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.489 DEBUG:boto:Data: 2023-11-02T18:30:12.489 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.490 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.490 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.491 DEBUG:boto:Params: {} 2023-11-02T18:30:12.491 DEBUG:boto:Token: None 2023-11-02T18:30:12.492 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.492 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad55fc2c1ae827b4-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.499 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:12.500 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:12.500 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.501 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000024 2023-11-02T18:30:12.501 DEBUG:boto:Data: 2023-11-02T18:30:12.501 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.502 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.502 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.503 DEBUG:boto:Params: {} 2023-11-02T18:30:12.503 DEBUG:boto:Token: None 2023-11-02T18:30:12.504 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000024 2023-11-02T18:30:12.504 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DCvOMYv0oXogZQtaAilNFLKp8gY= 2023-11-02T18:30:12.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DCvOMYv0oXogZQtaAilNFLKp8gY='} 2023-11-02T18:30:12.517 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095668cb529a7b01b-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.517 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.518 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.518 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.519 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.519 DEBUG:boto:Data: 2023-11-02T18:30:12.519 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.520 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.520 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.521 DEBUG:boto:Params: {} 2023-11-02T18:30:12.521 DEBUG:boto:Token: None 2023-11-02T18:30:12.521 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.522 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062549f563a00bdf6-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.528 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:12.528 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:12.528 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.529 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000028 2023-11-02T18:30:12.529 DEBUG:boto:Data: 2023-11-02T18:30:12.530 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.530 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.531 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.531 DEBUG:boto:Params: {} 2023-11-02T18:30:12.532 DEBUG:boto:Token: None 2023-11-02T18:30:12.532 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000028 2023-11-02T18:30:12.532 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gQ/dcMGMzss96at3BqHUXVZJhIM= 2023-11-02T18:30:12.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gQ/dcMGMzss96at3BqHUXVZJhIM='} 2023-11-02T18:30:12.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7e2b02927aef24e-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.579 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.580 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.580 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.580 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.581 DEBUG:boto:Data: 2023-11-02T18:30:12.581 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.582 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.582 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.583 DEBUG:boto:Params: {} 2023-11-02T18:30:12.583 DEBUG:boto:Token: None 2023-11-02T18:30:12.583 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.584 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5486611732c434a-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.590 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:12.591 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:12.591 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.591 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:12.592 DEBUG:boto:Data: 2023-11-02T18:30:12.592 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.593 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.593 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.593 DEBUG:boto:Params: {} 2023-11-02T18:30:12.594 DEBUG:boto:Token: None 2023-11-02T18:30:12.594 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:12.595 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:unPv86P0HrS+fOSF/VdAhLyHDGI= 2023-11-02T18:30:12.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:unPv86P0HrS+fOSF/VdAhLyHDGI='} 2023-11-02T18:30:12.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000657915954b0d2c65-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.603 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:12.604 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:12.604 DEBUG:boto:Method: HEAD 2023-11-02T18:30:12.605 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:12.605 DEBUG:boto:Data: 2023-11-02T18:30:12.606 DEBUG:boto:Headers: {} 2023-11-02T18:30:12.606 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.607 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.607 DEBUG:boto:Params: {} 2023-11-02T18:30:12.607 DEBUG:boto:Token: None 2023-11-02T18:30:12.608 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/ 2023-11-02T18:30:12.608 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk= 2023-11-02T18:30:12.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:we+Cuolps3bgu3fqW/2Zl/JaSYk=', 'Content-Length': '0'} 2023-11-02T18:30:12.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d98d8549ce20d508-006543eab4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.614 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:12.614 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:12.615 DEBUG:boto:Method: PUT 2023-11-02T18:30:12.615 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:12.616 DEBUG:boto:Data: 2023-11-02T18:30:12.616 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:12.616 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:12.617 DEBUG:boto:Port: 8000 2023-11-02T18:30:12.617 DEBUG:boto:Params: {} 2023-11-02T18:30:12.618 DEBUG:boto:Token: None 2023-11-02T18:30:12.618 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:12 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:12.619 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OtBOhE0X+M5T2GcWPCpQl9X5fCE= 2023-11-02T18:30:12.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OtBOhE0X+M5T2GcWPCpQl9X5fCE='} 2023-11-02T18:30:12.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de45196d1bffd6a2-006543eab4-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:12.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:12.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:12.668 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-58 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:30:12.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-11-02T18:30:12.719 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:12.719 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:12.805 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:30:12.806 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-58 2023-11-02T18:30:12.820 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 22 2023-11-02T18:30:12.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:12.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:12.912 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:12.961 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:12.962 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:13.044 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:30:13.045 INFO:teuthology.orchestra.run.smithi017.stdout: "objs": [ 2023-11-02T18:30:13.045 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000005", 2023-11-02T18:30:13.045 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000010", 2023-11-02T18:30:13.046 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000016", 2023-11-02T18:30:13.046 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000021", 2023-11-02T18:30:13.046 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000004", 2023-11-02T18:30:13.046 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000025", 2023-11-02T18:30:13.047 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000030", 2023-11-02T18:30:13.047 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000015", 2023-11-02T18:30:13.047 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000019", 2023-11-02T18:30:13.048 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000003", 2023-11-02T18:30:13.048 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000009", 2023-11-02T18:30:13.049 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000014", 2023-11-02T18:30:13.049 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000012", 2023-11-02T18:30:13.049 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000002", 2023-11-02T18:30:13.049 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000000", 2023-11-02T18:30:13.050 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000006", 2023-11-02T18:30:13.050 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000011", 2023-11-02T18:30:13.050 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000001", 2023-11-02T18:30:13.050 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000007" 2023-11-02T18:30:13.051 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:30:13.051 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:30:13.052 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.053 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.053 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.054 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.054 DEBUG:boto:Data: 2023-11-02T18:30:13.054 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.055 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.055 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.056 DEBUG:boto:Params: {} 2023-11-02T18:30:13.056 DEBUG:boto:Token: None 2023-11-02T18:30:13.057 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.057 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ec5b0e56eabea3c-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.064 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:13.065 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:13.065 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.066 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:13.066 DEBUG:boto:Data: 2023-11-02T18:30:13.069 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.069 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.070 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.070 DEBUG:boto:Params: {} 2023-11-02T18:30:13.071 DEBUG:boto:Token: None 2023-11-02T18:30:13.071 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:13.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zosv8Q6hFyc1b786it6EhwtzTMg= 2023-11-02T18:30:13.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zosv8Q6hFyc1b786it6EhwtzTMg='} 2023-11-02T18:30:13.093 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ffbc004f20849def-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.093 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.094 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.094 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.095 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.095 DEBUG:boto:Data: 2023-11-02T18:30:13.095 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.096 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.097 DEBUG:boto:Params: {} 2023-11-02T18:30:13.097 DEBUG:boto:Token: None 2023-11-02T18:30:13.098 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2ee0dfe49fcafea-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.105 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:13.105 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:13.106 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.106 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:13.107 DEBUG:boto:Data: 2023-11-02T18:30:13.107 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.108 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.108 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.108 DEBUG:boto:Params: {} 2023-11-02T18:30:13.109 DEBUG:boto:Token: None 2023-11-02T18:30:13.109 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:13.110 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uOLxfcDEVjZwobhIaGJLpQ95Dpg= 2023-11-02T18:30:13.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uOLxfcDEVjZwobhIaGJLpQ95Dpg='} 2023-11-02T18:30:13.155 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000973738ee781ae717-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.156 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.157 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.158 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.158 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.159 DEBUG:boto:Data: 2023-11-02T18:30:13.159 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.159 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.160 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.160 DEBUG:boto:Params: {} 2023-11-02T18:30:13.161 DEBUG:boto:Token: None 2023-11-02T18:30:13.162 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.162 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d3ac04a836421da-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.168 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:13.169 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:13.169 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.170 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:13.170 DEBUG:boto:Data: 2023-11-02T18:30:13.171 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.171 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.171 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.172 DEBUG:boto:Params: {} 2023-11-02T18:30:13.172 DEBUG:boto:Token: None 2023-11-02T18:30:13.173 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:13.173 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kEUTN3I0z8E7Ttpan7GxczQBX1Y= 2023-11-02T18:30:13.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kEUTN3I0z8E7Ttpan7GxczQBX1Y='} 2023-11-02T18:30:13.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c79a494da63271c-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.190 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.191 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.191 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.192 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.192 DEBUG:boto:Data: 2023-11-02T18:30:13.193 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.193 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.194 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.194 DEBUG:boto:Params: {} 2023-11-02T18:30:13.195 DEBUG:boto:Token: None 2023-11-02T18:30:13.196 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.196 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.202 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f213f98232777005-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.203 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:13.203 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:13.204 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.204 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:13.204 DEBUG:boto:Data: 2023-11-02T18:30:13.205 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.205 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.206 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.206 DEBUG:boto:Params: {} 2023-11-02T18:30:13.207 DEBUG:boto:Token: None 2023-11-02T18:30:13.207 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:13.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R5nxfGzTiFyjZPSxxDBM3XHMgi4= 2023-11-02T18:30:13.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R5nxfGzTiFyjZPSxxDBM3XHMgi4='} 2023-11-02T18:30:13.251 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001aa11c5f24176f12-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.252 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.252 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.252 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.253 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.253 DEBUG:boto:Data: 2023-11-02T18:30:13.254 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.255 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.255 DEBUG:boto:Params: {} 2023-11-02T18:30:13.256 DEBUG:boto:Token: None 2023-11-02T18:30:13.256 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.257 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000122c32a45eb12557-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.264 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:13.265 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:13.265 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.266 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:13.266 DEBUG:boto:Data: 2023-11-02T18:30:13.267 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.268 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.268 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.269 DEBUG:boto:Params: {} 2023-11-02T18:30:13.269 DEBUG:boto:Token: None 2023-11-02T18:30:13.270 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:13.270 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yk1kp05E4Du5aeDkvKBaR+GVr+0= 2023-11-02T18:30:13.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yk1kp05E4Du5aeDkvKBaR+GVr+0='} 2023-11-02T18:30:13.287 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a733774caaef5a4e-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.288 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.288 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.289 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.289 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.289 DEBUG:boto:Data: 2023-11-02T18:30:13.290 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.290 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.291 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.291 DEBUG:boto:Params: {} 2023-11-02T18:30:13.292 DEBUG:boto:Token: None 2023-11-02T18:30:13.292 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.293 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.298 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039a9db3e164ad85d-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.299 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:13.300 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:13.300 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.300 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:13.301 DEBUG:boto:Data: 2023-11-02T18:30:13.301 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.302 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.302 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.303 DEBUG:boto:Params: {} 2023-11-02T18:30:13.303 DEBUG:boto:Token: None 2023-11-02T18:30:13.304 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:13.304 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9i0TsgNivQF1xa34PJEsCF6qTWM= 2023-11-02T18:30:13.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9i0TsgNivQF1xa34PJEsCF6qTWM='} 2023-11-02T18:30:13.355 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081ebbd19afeb06ad-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.355 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.356 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.356 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.357 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.358 DEBUG:boto:Data: 2023-11-02T18:30:13.358 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.359 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.359 DEBUG:boto:Params: {} 2023-11-02T18:30:13.360 DEBUG:boto:Token: None 2023-11-02T18:30:13.360 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.361 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000167b6d4fa8a1ba9-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.368 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:13.369 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:13.369 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.370 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:13.370 DEBUG:boto:Data: 2023-11-02T18:30:13.371 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.372 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.372 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.372 DEBUG:boto:Params: {} 2023-11-02T18:30:13.373 DEBUG:boto:Token: None 2023-11-02T18:30:13.374 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:13.374 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5p33FoDqsco8VgqHHAnjK9H3dY0= 2023-11-02T18:30:13.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5p33FoDqsco8VgqHHAnjK9H3dY0='} 2023-11-02T18:30:13.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a32c9d2aff5bced-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.390 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.391 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.391 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.392 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.392 DEBUG:boto:Data: 2023-11-02T18:30:13.392 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.393 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.393 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.394 DEBUG:boto:Params: {} 2023-11-02T18:30:13.394 DEBUG:boto:Token: None 2023-11-02T18:30:13.395 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.395 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.402 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1e83ffda5e3eb0a-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.403 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:13.403 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:13.404 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.404 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:13.405 DEBUG:boto:Data: 2023-11-02T18:30:13.405 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.405 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.406 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.406 DEBUG:boto:Params: {} 2023-11-02T18:30:13.407 DEBUG:boto:Token: None 2023-11-02T18:30:13.407 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:13.408 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:67DU1EN47U10fbgbP6elWmadCTg= 2023-11-02T18:30:13.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:67DU1EN47U10fbgbP6elWmadCTg='} 2023-11-02T18:30:13.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e83611e6cacde073-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.455 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.456 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.456 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.457 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.457 DEBUG:boto:Data: 2023-11-02T18:30:13.458 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.458 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.458 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.459 DEBUG:boto:Params: {} 2023-11-02T18:30:13.459 DEBUG:boto:Token: None 2023-11-02T18:30:13.460 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.460 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f48138c44d6ac33c-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.467 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:13.468 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:13.469 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.469 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:13.469 DEBUG:boto:Data: 2023-11-02T18:30:13.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.470 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.471 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.471 DEBUG:boto:Params: {} 2023-11-02T18:30:13.472 DEBUG:boto:Token: None 2023-11-02T18:30:13.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:13.473 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:669/p4Vk9VRfrJTlT4JJiFQkoE8= 2023-11-02T18:30:13.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:669/p4Vk9VRfrJTlT4JJiFQkoE8='} 2023-11-02T18:30:13.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c0367e33726852e-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.491 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.491 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.492 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.492 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.493 DEBUG:boto:Data: 2023-11-02T18:30:13.493 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.494 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.494 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.495 DEBUG:boto:Params: {} 2023-11-02T18:30:13.495 DEBUG:boto:Token: None 2023-11-02T18:30:13.496 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.501 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b194783b4e73f70a-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.502 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:13.503 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:13.503 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.504 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:13.504 DEBUG:boto:Data: 2023-11-02T18:30:13.504 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.505 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.505 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.506 DEBUG:boto:Params: {} 2023-11-02T18:30:13.506 DEBUG:boto:Token: None 2023-11-02T18:30:13.507 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:13.507 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R4zBT9kAE4e9/vhFSFVnBTM1Xao= 2023-11-02T18:30:13.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R4zBT9kAE4e9/vhFSFVnBTM1Xao='} 2023-11-02T18:30:13.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081e57aa37a196e8b-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.551 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.552 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.552 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.553 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.553 DEBUG:boto:Data: 2023-11-02T18:30:13.553 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.554 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.555 DEBUG:boto:Params: {} 2023-11-02T18:30:13.555 DEBUG:boto:Token: None 2023-11-02T18:30:13.556 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.562 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcaeaf246e22235f-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.563 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:13.563 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:13.564 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.564 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:13.565 DEBUG:boto:Data: 2023-11-02T18:30:13.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.566 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.566 DEBUG:boto:Params: {} 2023-11-02T18:30:13.567 DEBUG:boto:Token: None 2023-11-02T18:30:13.567 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:13.568 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ax6mzPVWqcvN1mmaqIxQ3/uxDrU= 2023-11-02T18:30:13.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ax6mzPVWqcvN1mmaqIxQ3/uxDrU='} 2023-11-02T18:30:13.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000aa492414a69d65a-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.591 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.592 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.592 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.593 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.593 DEBUG:boto:Data: 2023-11-02T18:30:13.593 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.594 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.594 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.595 DEBUG:boto:Params: {} 2023-11-02T18:30:13.595 DEBUG:boto:Token: None 2023-11-02T18:30:13.596 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.596 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.600 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067039ed0a1524bad-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.600 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:13.601 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:13.601 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.602 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000025 2023-11-02T18:30:13.602 DEBUG:boto:Data: 2023-11-02T18:30:13.603 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.603 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.603 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.604 DEBUG:boto:Params: {} 2023-11-02T18:30:13.604 DEBUG:boto:Token: None 2023-11-02T18:30:13.605 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000025 2023-11-02T18:30:13.605 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gzc8qbyT+Au0CspLgYiYJ0pwZTs= 2023-11-02T18:30:13.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gzc8qbyT+Au0CspLgYiYJ0pwZTs='} 2023-11-02T18:30:13.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cef4a85704e092eb-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.652 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.652 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.653 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.653 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.654 DEBUG:boto:Data: 2023-11-02T18:30:13.654 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.655 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.655 DEBUG:boto:Params: {} 2023-11-02T18:30:13.656 DEBUG:boto:Token: None 2023-11-02T18:30:13.656 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.657 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.663 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a65923ffb356000-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.664 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:13.664 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:13.665 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.665 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:13.666 DEBUG:boto:Data: 2023-11-02T18:30:13.666 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.666 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.667 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.667 DEBUG:boto:Params: {} 2023-11-02T18:30:13.668 DEBUG:boto:Token: None 2023-11-02T18:30:13.668 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:13.669 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ecWbThFvINDVwO91nT+3+MyY+/Q= 2023-11-02T18:30:13.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ecWbThFvINDVwO91nT+3+MyY+/Q='} 2023-11-02T18:30:13.684 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e550a2da0ca85e5e-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.685 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.685 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.686 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.686 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.687 DEBUG:boto:Data: 2023-11-02T18:30:13.687 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.687 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.688 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.688 DEBUG:boto:Params: {} 2023-11-02T18:30:13.689 DEBUG:boto:Token: None 2023-11-02T18:30:13.689 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.689 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.695 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2f43f2036eaa5a5-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.696 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:13.696 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:13.697 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.697 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:13.697 DEBUG:boto:Data: 2023-11-02T18:30:13.698 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.698 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.699 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.699 DEBUG:boto:Params: {} 2023-11-02T18:30:13.699 DEBUG:boto:Token: None 2023-11-02T18:30:13.700 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:13.700 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QJ05o1XHaSswzAQFVlidLf2m5bw= 2023-11-02T18:30:13.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QJ05o1XHaSswzAQFVlidLf2m5bw='} 2023-11-02T18:30:13.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a7aa66d8a9fd99d-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.748 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.748 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.749 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.749 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.750 DEBUG:boto:Data: 2023-11-02T18:30:13.750 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.751 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.751 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.752 DEBUG:boto:Params: {} 2023-11-02T18:30:13.752 DEBUG:boto:Token: None 2023-11-02T18:30:13.753 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.753 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006224515af4228091-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.761 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:13.761 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:13.761 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.762 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:13.762 DEBUG:boto:Data: 2023-11-02T18:30:13.763 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.763 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.764 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.764 DEBUG:boto:Params: {} 2023-11-02T18:30:13.764 DEBUG:boto:Token: None 2023-11-02T18:30:13.765 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:13.765 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G1a+HTpiXVdv4l+P4VK2Njm4WWA= 2023-11-02T18:30:13.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G1a+HTpiXVdv4l+P4VK2Njm4WWA='} 2023-11-02T18:30:13.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000505b098b50347dd8-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.783 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.784 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.784 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.785 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.786 DEBUG:boto:Data: 2023-11-02T18:30:13.787 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.787 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.788 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.788 DEBUG:boto:Params: {} 2023-11-02T18:30:13.789 DEBUG:boto:Token: None 2023-11-02T18:30:13.790 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.790 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a7be0feebf1d31d-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.797 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:13.797 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:13.798 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.798 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:13.799 DEBUG:boto:Data: 2023-11-02T18:30:13.799 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.800 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.800 DEBUG:boto:Params: {} 2023-11-02T18:30:13.801 DEBUG:boto:Token: None 2023-11-02T18:30:13.801 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:13.803 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6twR6bnX3VFA6hbyMuCfBbeNEJc= 2023-11-02T18:30:13.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6twR6bnX3VFA6hbyMuCfBbeNEJc='} 2023-11-02T18:30:13.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a98dbd388ae7b38-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.851 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.852 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.852 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.852 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.853 DEBUG:boto:Data: 2023-11-02T18:30:13.853 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.854 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.854 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.855 DEBUG:boto:Params: {} 2023-11-02T18:30:13.855 DEBUG:boto:Token: None 2023-11-02T18:30:13.856 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.856 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d88dcc09286f2bc-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.864 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:13.864 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:13.864 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.865 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:13.865 DEBUG:boto:Data: 2023-11-02T18:30:13.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.866 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.867 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.867 DEBUG:boto:Params: {} 2023-11-02T18:30:13.867 DEBUG:boto:Token: None 2023-11-02T18:30:13.868 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:13.868 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aUSz5qUt4iIh7dy9O7s00m8JYho= 2023-11-02T18:30:13.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aUSz5qUt4iIh7dy9O7s00m8JYho='} 2023-11-02T18:30:13.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003357b353b81e8a3e-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.881 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:13.882 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:13.882 DEBUG:boto:Method: HEAD 2023-11-02T18:30:13.883 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:13.883 DEBUG:boto:Data: 2023-11-02T18:30:13.883 DEBUG:boto:Headers: {} 2023-11-02T18:30:13.884 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.884 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.885 DEBUG:boto:Params: {} 2023-11-02T18:30:13.885 DEBUG:boto:Token: None 2023-11-02T18:30:13.885 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/ 2023-11-02T18:30:13.886 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo= 2023-11-02T18:30:13.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQPTHD2+vO2Zsypz87bk1Rj6bEo=', 'Content-Length': '0'} 2023-11-02T18:30:13.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c0aca0f608517dd-006543eab5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.892 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:13.892 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:13.893 DEBUG:boto:Method: PUT 2023-11-02T18:30:13.893 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:13.893 DEBUG:boto:Data: 2023-11-02T18:30:13.894 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:13.894 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:13.895 DEBUG:boto:Port: 8000 2023-11-02T18:30:13.895 DEBUG:boto:Params: {} 2023-11-02T18:30:13.896 DEBUG:boto:Token: None 2023-11-02T18:30:13.896 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:13 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:13.896 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pTqCJbZ6gg9sfFNotCiZUWJalvw= 2023-11-02T18:30:13.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pTqCJbZ6gg9sfFNotCiZUWJalvw='} 2023-11-02T18:30:13.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074c56c9847a52d9a-006543eab5-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:13.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:13.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:13.944 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-58 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:30:13.994 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:13.994 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:14.079 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:30:14.080 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-58 2023-11-02T18:30:14.097 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 27 2023-11-02T18:30:14.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:14.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:14.193 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:14.239 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:14.240 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:14.317 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:30:14.317 INFO:teuthology.orchestra.run.smithi017.stdout: "objs": [ 2023-11-02T18:30:14.317 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000058", 2023-11-02T18:30:14.318 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000026", 2023-11-02T18:30:14.318 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000011", 2023-11-02T18:30:14.318 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000046", 2023-11-02T18:30:14.318 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000014", 2023-11-02T18:30:14.318 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000050", 2023-11-02T18:30:14.319 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000032", 2023-11-02T18:30:14.319 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000002", 2023-11-02T18:30:14.319 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000039", 2023-11-02T18:30:14.319 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000007", 2023-11-02T18:30:14.320 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000006", 2023-11-02T18:30:14.320 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000040", 2023-11-02T18:30:14.320 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000009", 2023-11-02T18:30:14.320 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000045", 2023-11-02T18:30:14.321 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000013", 2023-11-02T18:30:14.321 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000012", 2023-11-02T18:30:14.321 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000033", 2023-11-02T18:30:14.321 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000001", 2023-11-02T18:30:14.321 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000000", 2023-11-02T18:30:14.322 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000021", 2023-11-02T18:30:14.322 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000003", 2023-11-02T18:30:14.322 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000055", 2023-11-02T18:30:14.322 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000008" 2023-11-02T18:30:14.323 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:30:14.323 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:30:14.324 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.324 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.325 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.325 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.326 DEBUG:boto:Data: 2023-11-02T18:30:14.326 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.327 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.327 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.327 DEBUG:boto:Params: {} 2023-11-02T18:30:14.328 DEBUG:boto:Token: None 2023-11-02T18:30:14.328 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.329 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.338 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f458099eae62b015-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.339 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:14.340 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:14.340 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.341 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:14.341 DEBUG:boto:Data: 2023-11-02T18:30:14.341 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.342 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.342 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.343 DEBUG:boto:Params: {} 2023-11-02T18:30:14.343 DEBUG:boto:Token: None 2023-11-02T18:30:14.344 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:14.344 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6FNKO2OWPdgGuYwDrpMbnYD0IwM= 2023-11-02T18:30:14.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6FNKO2OWPdgGuYwDrpMbnYD0IwM='} 2023-11-02T18:30:14.360 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008eba9ee24141f145-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.361 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.361 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.362 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.362 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.362 DEBUG:boto:Data: 2023-11-02T18:30:14.363 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.363 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.364 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.364 DEBUG:boto:Params: {} 2023-11-02T18:30:14.365 DEBUG:boto:Token: None 2023-11-02T18:30:14.365 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.366 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4e791beea87f706-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.372 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:14.372 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:14.373 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.373 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:14.373 DEBUG:boto:Data: 2023-11-02T18:30:14.374 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.374 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.375 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.375 DEBUG:boto:Params: {} 2023-11-02T18:30:14.376 DEBUG:boto:Token: None 2023-11-02T18:30:14.376 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:14.377 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u5TWK08wR9mwTY8N2aLXXuzIX+0= 2023-11-02T18:30:14.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u5TWK08wR9mwTY8N2aLXXuzIX+0='} 2023-11-02T18:30:14.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060b4007610462f8b-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.423 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.424 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.424 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.425 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.425 DEBUG:boto:Data: 2023-11-02T18:30:14.425 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.426 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.427 DEBUG:boto:Params: {} 2023-11-02T18:30:14.427 DEBUG:boto:Token: None 2023-11-02T18:30:14.428 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0078c74bf10f8ec-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.435 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:14.436 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:14.436 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.437 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:14.437 DEBUG:boto:Data: 2023-11-02T18:30:14.437 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.438 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.438 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.439 DEBUG:boto:Params: {} 2023-11-02T18:30:14.439 DEBUG:boto:Token: None 2023-11-02T18:30:14.440 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:14.440 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dNHH4cG4CfOSkYlkkTyje+egW+8= 2023-11-02T18:30:14.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dNHH4cG4CfOSkYlkkTyje+egW+8='} 2023-11-02T18:30:14.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022731d0edd257852-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.456 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.456 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.457 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.457 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.458 DEBUG:boto:Data: 2023-11-02T18:30:14.458 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.458 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.459 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.459 DEBUG:boto:Params: {} 2023-11-02T18:30:14.460 DEBUG:boto:Token: None 2023-11-02T18:30:14.460 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.461 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.467 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c2808af00045db4-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.467 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:14.468 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:14.468 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.469 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000055 2023-11-02T18:30:14.469 DEBUG:boto:Data: 2023-11-02T18:30:14.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.470 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.470 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.471 DEBUG:boto:Params: {} 2023-11-02T18:30:14.471 DEBUG:boto:Token: None 2023-11-02T18:30:14.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000055 2023-11-02T18:30:14.472 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fkSB7En/ObZGR7hegCOCfljUUQw= 2023-11-02T18:30:14.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fkSB7En/ObZGR7hegCOCfljUUQw='} 2023-11-02T18:30:14.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005446aa5d859c7491-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.519 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.520 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.520 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.521 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.521 DEBUG:boto:Data: 2023-11-02T18:30:14.521 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.522 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.523 DEBUG:boto:Params: {} 2023-11-02T18:30:14.523 DEBUG:boto:Token: None 2023-11-02T18:30:14.524 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3b554341c83a824-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.532 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:14.532 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:14.533 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.533 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:14.533 DEBUG:boto:Data: 2023-11-02T18:30:14.534 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.534 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.535 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.535 DEBUG:boto:Params: {} 2023-11-02T18:30:14.536 DEBUG:boto:Token: None 2023-11-02T18:30:14.536 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:14.537 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0DD6suzQD+8+scv7ynG1J0B9M7M= 2023-11-02T18:30:14.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0DD6suzQD+8+scv7ynG1J0B9M7M='} 2023-11-02T18:30:14.552 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ac9593326da350c-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.553 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.553 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.554 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.554 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.555 DEBUG:boto:Data: 2023-11-02T18:30:14.555 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.556 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.556 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.556 DEBUG:boto:Params: {} 2023-11-02T18:30:14.557 DEBUG:boto:Token: None 2023-11-02T18:30:14.557 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.558 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9de9804c2259e5a-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.565 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:14.565 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:14.566 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.566 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:14.566 DEBUG:boto:Data: 2023-11-02T18:30:14.567 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.567 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.568 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.568 DEBUG:boto:Params: {} 2023-11-02T18:30:14.569 DEBUG:boto:Token: None 2023-11-02T18:30:14.569 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:14.570 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p8BLRtHWY/imaLK6UC2Jbec7C64= 2023-11-02T18:30:14.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p8BLRtHWY/imaLK6UC2Jbec7C64='} 2023-11-02T18:30:14.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b34c1dffc7760f5-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.615 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.616 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.616 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.617 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.617 DEBUG:boto:Data: 2023-11-02T18:30:14.618 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.618 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.618 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.619 DEBUG:boto:Params: {} 2023-11-02T18:30:14.619 DEBUG:boto:Token: None 2023-11-02T18:30:14.620 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.620 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a89baeb8676d708-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.628 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:14.628 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:14.629 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.629 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:14.630 DEBUG:boto:Data: 2023-11-02T18:30:14.630 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.631 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.631 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.631 DEBUG:boto:Params: {} 2023-11-02T18:30:14.632 DEBUG:boto:Token: None 2023-11-02T18:30:14.632 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:14.633 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LUVXWyW30pVbon2Vkw8602N39bU= 2023-11-02T18:30:14.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LUVXWyW30pVbon2Vkw8602N39bU='} 2023-11-02T18:30:14.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000573a7a0c235dd22-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.643 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.644 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.644 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.645 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.645 DEBUG:boto:Data: 2023-11-02T18:30:14.645 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.646 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.646 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.647 DEBUG:boto:Params: {} 2023-11-02T18:30:14.647 DEBUG:boto:Token: None 2023-11-02T18:30:14.648 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.648 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e5e335bbb84b675-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.654 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:14.655 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:14.655 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.656 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000045 2023-11-02T18:30:14.656 DEBUG:boto:Data: 2023-11-02T18:30:14.656 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.657 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.657 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.658 DEBUG:boto:Params: {} 2023-11-02T18:30:14.658 DEBUG:boto:Token: None 2023-11-02T18:30:14.659 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000045 2023-11-02T18:30:14.659 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e/YlQ3m2mqCy+7MGULh1baUKU0U= 2023-11-02T18:30:14.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e/YlQ3m2mqCy+7MGULh1baUKU0U='} 2023-11-02T18:30:14.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005bfaa8bedacfe927-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.703 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.704 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.704 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.704 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.705 DEBUG:boto:Data: 2023-11-02T18:30:14.705 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.706 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.706 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.707 DEBUG:boto:Params: {} 2023-11-02T18:30:14.707 DEBUG:boto:Token: None 2023-11-02T18:30:14.708 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.708 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097366a2d8e12604c-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.715 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:14.716 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:14.716 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.717 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:14.717 DEBUG:boto:Data: 2023-11-02T18:30:14.718 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.718 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.718 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.719 DEBUG:boto:Params: {} 2023-11-02T18:30:14.720 DEBUG:boto:Token: None 2023-11-02T18:30:14.720 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:14.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SXyop8LDdHxXe1+DNKDE7VMXUhw= 2023-11-02T18:30:14.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SXyop8LDdHxXe1+DNKDE7VMXUhw='} 2023-11-02T18:30:14.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4ccc0f409f06fc4-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.735 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.736 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.736 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.737 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.737 DEBUG:boto:Data: 2023-11-02T18:30:14.737 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.738 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.739 DEBUG:boto:Params: {} 2023-11-02T18:30:14.739 DEBUG:boto:Token: None 2023-11-02T18:30:14.740 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.746 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008130d4d8550a5e7e-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.747 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:14.747 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:14.748 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.748 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000032 2023-11-02T18:30:14.749 DEBUG:boto:Data: 2023-11-02T18:30:14.749 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.749 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.750 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.750 DEBUG:boto:Params: {} 2023-11-02T18:30:14.751 DEBUG:boto:Token: None 2023-11-02T18:30:14.751 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000032 2023-11-02T18:30:14.752 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AoLeKpodL5NW4Px+9F4ZFEhVaow= 2023-11-02T18:30:14.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AoLeKpodL5NW4Px+9F4ZFEhVaow='} 2023-11-02T18:30:14.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b82f256a5dc91a66-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.795 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.796 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.796 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.797 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.797 DEBUG:boto:Data: 2023-11-02T18:30:14.797 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.798 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.798 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.799 DEBUG:boto:Params: {} 2023-11-02T18:30:14.799 DEBUG:boto:Token: None 2023-11-02T18:30:14.800 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.800 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1e07a06c9579eab-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.807 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:14.808 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:14.808 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.808 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:14.809 DEBUG:boto:Data: 2023-11-02T18:30:14.809 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.810 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.810 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.810 DEBUG:boto:Params: {} 2023-11-02T18:30:14.811 DEBUG:boto:Token: None 2023-11-02T18:30:14.811 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:14.812 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AGiKSRVKPoJWWgsZORWEJ5iEEQw= 2023-11-02T18:30:14.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AGiKSRVKPoJWWgsZORWEJ5iEEQw='} 2023-11-02T18:30:14.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000749dbb746c868d09-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.827 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.827 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.828 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.828 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.829 DEBUG:boto:Data: 2023-11-02T18:30:14.829 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.829 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.830 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.831 DEBUG:boto:Params: {} 2023-11-02T18:30:14.831 DEBUG:boto:Token: None 2023-11-02T18:30:14.832 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.832 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000375eeae693a49638-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.839 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:14.839 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:14.840 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.840 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:14.841 DEBUG:boto:Data: 2023-11-02T18:30:14.841 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.842 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.842 DEBUG:boto:Params: {} 2023-11-02T18:30:14.843 DEBUG:boto:Token: None 2023-11-02T18:30:14.843 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:14.844 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6RC02tnXt0htGGMQYuYLgD6dfkw= 2023-11-02T18:30:14.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6RC02tnXt0htGGMQYuYLgD6dfkw='} 2023-11-02T18:30:14.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085f0169012f48a8d-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.891 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.892 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.892 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.892 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.893 DEBUG:boto:Data: 2023-11-02T18:30:14.893 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.894 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.894 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.895 DEBUG:boto:Params: {} 2023-11-02T18:30:14.895 DEBUG:boto:Token: None 2023-11-02T18:30:14.895 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.896 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003488371ce3947e90-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.903 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:14.904 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:14.904 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.905 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000040 2023-11-02T18:30:14.905 DEBUG:boto:Data: 2023-11-02T18:30:14.905 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.906 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.907 DEBUG:boto:Params: {} 2023-11-02T18:30:14.907 DEBUG:boto:Token: None 2023-11-02T18:30:14.908 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000040 2023-11-02T18:30:14.908 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OQGzN9c7bWqPOYzzFi0dQFvIURI= 2023-11-02T18:30:14.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OQGzN9c7bWqPOYzzFi0dQFvIURI='} 2023-11-02T18:30:14.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b95fc267963570b-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.920 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.920 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.921 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.921 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.921 DEBUG:boto:Data: 2023-11-02T18:30:14.922 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.922 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.923 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.923 DEBUG:boto:Params: {} 2023-11-02T18:30:14.923 DEBUG:boto:Token: None 2023-11-02T18:30:14.924 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.924 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ea024548ce2c31b-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.932 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:14.932 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:14.933 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.933 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:14.934 DEBUG:boto:Data: 2023-11-02T18:30:14.934 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.935 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.935 DEBUG:boto:Params: {} 2023-11-02T18:30:14.936 DEBUG:boto:Token: None 2023-11-02T18:30:14.936 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:14.937 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Gw6HN9rRSOVGdLMKtNto/IcuItk= 2023-11-02T18:30:14.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Gw6HN9rRSOVGdLMKtNto/IcuItk='} 2023-11-02T18:30:14.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021e5e6b1f18229df-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.983 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:14.984 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:14.984 DEBUG:boto:Method: HEAD 2023-11-02T18:30:14.985 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:14.985 DEBUG:boto:Data: 2023-11-02T18:30:14.985 DEBUG:boto:Headers: {} 2023-11-02T18:30:14.986 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.986 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.987 DEBUG:boto:Params: {} 2023-11-02T18:30:14.987 DEBUG:boto:Token: None 2023-11-02T18:30:14.988 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/ 2023-11-02T18:30:14.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y= 2023-11-02T18:30:14.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V27OqaW9Nftiq1gxCFhWxqsrW/Y=', 'Content-Length': '0'} 2023-11-02T18:30:14.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004e91eb53619c601-006543eab6-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:14.995 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:14.995 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:14.996 DEBUG:boto:Method: PUT 2023-11-02T18:30:14.996 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:14.997 DEBUG:boto:Data: 2023-11-02T18:30:14.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:14.997 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:14.998 DEBUG:boto:Port: 8000 2023-11-02T18:30:14.998 DEBUG:boto:Params: {} 2023-11-02T18:30:14.999 DEBUG:boto:Token: None 2023-11-02T18:30:14.999 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:14 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:15.000 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J/TbtcJOtmZaNn0YpGawHCrFl0A= 2023-11-02T18:30:15.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J/TbtcJOtmZaNn0YpGawHCrFl0A='} 2023-11-02T18:30:15.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058c7afb458a38956-006543eab6-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.015 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.015 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.017 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.017 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.017 DEBUG:boto:Data: 2023-11-02T18:30:15.018 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.018 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.019 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.019 DEBUG:boto:Params: {} 2023-11-02T18:30:15.020 DEBUG:boto:Token: None 2023-11-02T18:30:15.020 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.021 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050ab5b9501671169-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.028 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:15.029 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:15.030 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.030 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:15.030 DEBUG:boto:Data: 2023-11-02T18:30:15.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.033 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.033 DEBUG:boto:Params: {} 2023-11-02T18:30:15.034 DEBUG:boto:Token: None 2023-11-02T18:30:15.034 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:15.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KirhQZTlf2dq8YaZyiT+SS+PQ4w= 2023-11-02T18:30:15.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KirhQZTlf2dq8YaZyiT+SS+PQ4w='} 2023-11-02T18:30:15.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a38d82ecb0ef78b-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.079 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.079 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.080 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.080 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.081 DEBUG:boto:Data: 2023-11-02T18:30:15.081 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.081 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.082 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.082 DEBUG:boto:Params: {} 2023-11-02T18:30:15.083 DEBUG:boto:Token: None 2023-11-02T18:30:15.083 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.084 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.091 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000eb2c4503a21ffd9-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.091 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:15.092 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:15.092 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.092 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:15.093 DEBUG:boto:Data: 2023-11-02T18:30:15.093 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.094 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.094 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.095 DEBUG:boto:Params: {} 2023-11-02T18:30:15.095 DEBUG:boto:Token: None 2023-11-02T18:30:15.096 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:15.096 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eSTbE1wNEND/2BiC0i+aH25YA6o= 2023-11-02T18:30:15.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eSTbE1wNEND/2BiC0i+aH25YA6o='} 2023-11-02T18:30:15.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fad50ce287317465-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.111 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.111 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.111 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.112 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.112 DEBUG:boto:Data: 2023-11-02T18:30:15.113 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.113 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.114 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.114 DEBUG:boto:Params: {} 2023-11-02T18:30:15.114 DEBUG:boto:Token: None 2023-11-02T18:30:15.115 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.115 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cace941bd584849c-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.122 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:15.123 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:15.123 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.124 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000050 2023-11-02T18:30:15.124 DEBUG:boto:Data: 2023-11-02T18:30:15.125 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.125 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.125 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.126 DEBUG:boto:Params: {} 2023-11-02T18:30:15.126 DEBUG:boto:Token: None 2023-11-02T18:30:15.127 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000050 2023-11-02T18:30:15.127 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HD1Gzm4Drdkj7kuZovOZIiy2dRE= 2023-11-02T18:30:15.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HD1Gzm4Drdkj7kuZovOZIiy2dRE='} 2023-11-02T18:30:15.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b22be44e7686ee23-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.171 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.172 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.172 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.173 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.173 DEBUG:boto:Data: 2023-11-02T18:30:15.174 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.174 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.175 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.175 DEBUG:boto:Params: {} 2023-11-02T18:30:15.176 DEBUG:boto:Token: None 2023-11-02T18:30:15.176 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056e316ec52610595-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.184 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:15.184 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:15.185 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.185 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000058 2023-11-02T18:30:15.186 DEBUG:boto:Data: 2023-11-02T18:30:15.186 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.187 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.187 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.188 DEBUG:boto:Params: {} 2023-11-02T18:30:15.188 DEBUG:boto:Token: None 2023-11-02T18:30:15.189 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000058 2023-11-02T18:30:15.189 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ov+sTI6cxf4qlscLwBhv1RH0Zgo= 2023-11-02T18:30:15.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ov+sTI6cxf4qlscLwBhv1RH0Zgo='} 2023-11-02T18:30:15.200 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e226de4442fcc5c3-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.201 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.201 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.201 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.202 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.202 DEBUG:boto:Data: 2023-11-02T18:30:15.203 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.203 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.204 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.204 DEBUG:boto:Params: {} 2023-11-02T18:30:15.205 DEBUG:boto:Token: None 2023-11-02T18:30:15.205 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.206 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083b64cec5c1152dc-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.212 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:15.212 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:15.213 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.213 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:15.214 DEBUG:boto:Data: 2023-11-02T18:30:15.214 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.214 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.215 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.215 DEBUG:boto:Params: {} 2023-11-02T18:30:15.216 DEBUG:boto:Token: None 2023-11-02T18:30:15.216 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:15.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cO1zlzLL6l63PdMSliVammW6EC0= 2023-11-02T18:30:15.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cO1zlzLL6l63PdMSliVammW6EC0='} 2023-11-02T18:30:15.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000445d2b4cf01837fa-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.263 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.263 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.264 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.264 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.265 DEBUG:boto:Data: 2023-11-02T18:30:15.265 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.266 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.266 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.266 DEBUG:boto:Params: {} 2023-11-02T18:30:15.267 DEBUG:boto:Token: None 2023-11-02T18:30:15.267 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.268 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.275 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5d9dd9bad63fce1-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.276 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:15.276 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:15.277 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.277 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:15.278 DEBUG:boto:Data: 2023-11-02T18:30:15.278 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.278 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.279 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.279 DEBUG:boto:Params: {} 2023-11-02T18:30:15.280 DEBUG:boto:Token: None 2023-11-02T18:30:15.280 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:15.281 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:deQRURY714qrF805Nz4iEjd8ZHY= 2023-11-02T18:30:15.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:deQRURY714qrF805Nz4iEjd8ZHY='} 2023-11-02T18:30:15.295 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092d2a0e8b93c207f-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:15.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:15.297 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-58 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:30:15.345 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:15.345 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:15.430 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:30:15.431 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-58 2023-11-02T18:30:15.449 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 33 2023-11-02T18:30:15.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:15.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:15.552 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:15.597 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:15.598 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:15.669 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:30:15.670 INFO:teuthology.orchestra.run.smithi017.stdout: "objs": [ 2023-11-02T18:30:15.670 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000006", 2023-11-02T18:30:15.670 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000059", 2023-11-02T18:30:15.670 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000027", 2023-11-02T18:30:15.671 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000004", 2023-11-02T18:30:15.671 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000011", 2023-11-02T18:30:15.671 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000002", 2023-11-02T18:30:15.671 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000019", 2023-11-02T18:30:15.672 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000049", 2023-11-02T18:30:15.672 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000017", 2023-11-02T18:30:15.672 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000008", 2023-11-02T18:30:15.672 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000015", 2023-11-02T18:30:15.672 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000031", 2023-11-02T18:30:15.673 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000022", 2023-11-02T18:30:15.673 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000013", 2023-11-02T18:30:15.673 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000020", 2023-11-02T18:30:15.673 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000037", 2023-11-02T18:30:15.673 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000005", 2023-11-02T18:30:15.674 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000012", 2023-11-02T18:30:15.674 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000003", 2023-11-02T18:30:15.674 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000042", 2023-11-02T18:30:15.674 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000010", 2023-11-02T18:30:15.674 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000001", 2023-11-02T18:30:15.675 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000009", 2023-11-02T18:30:15.675 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000025", 2023-11-02T18:30:15.675 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000016", 2023-11-02T18:30:15.675 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000000", 2023-11-02T18:30:15.676 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000014", 2023-11-02T18:30:15.676 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000030", 2023-11-02T18:30:15.676 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000021" 2023-11-02T18:30:15.676 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:30:15.676 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:30:15.677 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.678 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.678 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.679 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.679 DEBUG:boto:Data: 2023-11-02T18:30:15.680 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.680 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.681 DEBUG:boto:Params: {} 2023-11-02T18:30:15.681 DEBUG:boto:Token: None 2023-11-02T18:30:15.682 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.682 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.689 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095319b0082aa7438-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.689 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:15.690 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:15.690 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.691 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:15.691 DEBUG:boto:Data: 2023-11-02T18:30:15.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.692 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.692 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.693 DEBUG:boto:Params: {} 2023-11-02T18:30:15.693 DEBUG:boto:Token: None 2023-11-02T18:30:15.694 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:15.694 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E4sqTgd7gDyaFRH9b9on2eUCTAc= 2023-11-02T18:30:15.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E4sqTgd7gDyaFRH9b9on2eUCTAc='} 2023-11-02T18:30:15.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fdd5fccfe03d5aa1-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.739 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.740 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.740 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.741 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.741 DEBUG:boto:Data: 2023-11-02T18:30:15.741 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.742 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.742 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.743 DEBUG:boto:Params: {} 2023-11-02T18:30:15.743 DEBUG:boto:Token: None 2023-11-02T18:30:15.744 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.744 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d21fceec0417fc3d-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.752 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:15.752 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:15.753 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.753 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:15.754 DEBUG:boto:Data: 2023-11-02T18:30:15.754 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.755 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.755 DEBUG:boto:Params: {} 2023-11-02T18:30:15.756 DEBUG:boto:Token: None 2023-11-02T18:30:15.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:15.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lLg5F1p2SEAuT7/F4cQ/xVDLKlI= 2023-11-02T18:30:15.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lLg5F1p2SEAuT7/F4cQ/xVDLKlI='} 2023-11-02T18:30:15.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000611b360c7e87b0f6-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.773 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.773 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.774 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.774 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.775 DEBUG:boto:Data: 2023-11-02T18:30:15.775 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.775 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.776 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.776 DEBUG:boto:Params: {} 2023-11-02T18:30:15.777 DEBUG:boto:Token: None 2023-11-02T18:30:15.777 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.778 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.783 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ceab94c38ad7dfd-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.784 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:15.785 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:15.785 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.786 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:15.786 DEBUG:boto:Data: 2023-11-02T18:30:15.786 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.787 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.787 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.788 DEBUG:boto:Params: {} 2023-11-02T18:30:15.788 DEBUG:boto:Token: None 2023-11-02T18:30:15.789 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:15.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nLQj3OPwkCSwhrsKDtePNi78SAQ= 2023-11-02T18:30:15.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nLQj3OPwkCSwhrsKDtePNi78SAQ='} 2023-11-02T18:30:15.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6caeca30dc28864-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.839 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.840 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.840 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.841 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.841 DEBUG:boto:Data: 2023-11-02T18:30:15.841 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.842 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.842 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.843 DEBUG:boto:Params: {} 2023-11-02T18:30:15.843 DEBUG:boto:Token: None 2023-11-02T18:30:15.844 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.844 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5460f093ff4fc60-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.852 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:15.853 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:15.853 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.854 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:15.854 DEBUG:boto:Data: 2023-11-02T18:30:15.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.855 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.856 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.856 DEBUG:boto:Params: {} 2023-11-02T18:30:15.856 DEBUG:boto:Token: None 2023-11-02T18:30:15.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:15.857 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zvZmh8TSOJTi8/sX14uAOGn7UCY= 2023-11-02T18:30:15.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zvZmh8TSOJTi8/sX14uAOGn7UCY='} 2023-11-02T18:30:15.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c80cadbc509a6e4-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.871 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.871 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.872 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.872 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.873 DEBUG:boto:Data: 2023-11-02T18:30:15.873 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.874 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.874 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.874 DEBUG:boto:Params: {} 2023-11-02T18:30:15.875 DEBUG:boto:Token: None 2023-11-02T18:30:15.875 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.876 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d76d30c6a5895bfc-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.883 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:15.883 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:15.884 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.884 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:15.885 DEBUG:boto:Data: 2023-11-02T18:30:15.885 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.885 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.886 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.886 DEBUG:boto:Params: {} 2023-11-02T18:30:15.887 DEBUG:boto:Token: None 2023-11-02T18:30:15.887 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:15.888 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MaazMg1K6nfTwCrkwkcvsNgcbRs= 2023-11-02T18:30:15.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MaazMg1K6nfTwCrkwkcvsNgcbRs='} 2023-11-02T18:30:15.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de8190ea9e9f99a7-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.931 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.932 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.932 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.933 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.933 DEBUG:boto:Data: 2023-11-02T18:30:15.933 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.934 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.935 DEBUG:boto:Params: {} 2023-11-02T18:30:15.935 DEBUG:boto:Token: None 2023-11-02T18:30:15.935 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.936 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e60187ad44e54f3-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.944 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:15.944 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:15.944 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.945 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:15.945 DEBUG:boto:Data: 2023-11-02T18:30:15.946 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.946 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.947 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.947 DEBUG:boto:Params: {} 2023-11-02T18:30:15.948 DEBUG:boto:Token: None 2023-11-02T18:30:15.948 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:15.949 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9qNwPD76fmHBn4GFzXSvefINzdw= 2023-11-02T18:30:15.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9qNwPD76fmHBn4GFzXSvefINzdw='} 2023-11-02T18:30:15.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4121e02d29c025b-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.964 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:15.965 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:15.965 DEBUG:boto:Method: HEAD 2023-11-02T18:30:15.966 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:15.966 DEBUG:boto:Data: 2023-11-02T18:30:15.966 DEBUG:boto:Headers: {} 2023-11-02T18:30:15.967 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.967 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.968 DEBUG:boto:Params: {} 2023-11-02T18:30:15.968 DEBUG:boto:Token: None 2023-11-02T18:30:15.969 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/ 2023-11-02T18:30:15.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c= 2023-11-02T18:30:15.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0jXEa4L5NT1XvHtm9iV8COXn8c=', 'Content-Length': '0'} 2023-11-02T18:30:15.976 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de121708f307b7e0-006543eab7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:15.976 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:15.977 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:15.977 DEBUG:boto:Method: PUT 2023-11-02T18:30:15.978 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:15.978 DEBUG:boto:Data: 2023-11-02T18:30:15.978 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:15.979 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:15.979 DEBUG:boto:Port: 8000 2023-11-02T18:30:15.980 DEBUG:boto:Params: {} 2023-11-02T18:30:15.980 DEBUG:boto:Token: None 2023-11-02T18:30:15.981 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:15 GMT /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:15.981 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z3A2eJup1JFl4IjL1qKOVJWcvIg= 2023-11-02T18:30:15.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z3A2eJup1JFl4IjL1qKOVJWcvIg='} 2023-11-02T18:30:16.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e785acec5423bae-006543eab7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.027 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.028 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.028 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.028 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.029 DEBUG:boto:Data: 2023-11-02T18:30:16.029 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.030 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.031 DEBUG:boto:Params: {} 2023-11-02T18:30:16.031 DEBUG:boto:Token: None 2023-11-02T18:30:16.032 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015c2a301e3ad7c8c-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.040 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:16.041 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:16.041 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.041 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:16.042 DEBUG:boto:Data: 2023-11-02T18:30:16.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.043 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.043 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.044 DEBUG:boto:Params: {} 2023-11-02T18:30:16.044 DEBUG:boto:Token: None 2023-11-02T18:30:16.045 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:16.045 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0cNGbOh4l34j9RoDSX3JY/Y4H9c= 2023-11-02T18:30:16.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0cNGbOh4l34j9RoDSX3JY/Y4H9c='} 2023-11-02T18:30:16.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a6ae83397bb80d8-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.061 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.061 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.061 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.062 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.062 DEBUG:boto:Data: 2023-11-02T18:30:16.063 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.063 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.064 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.064 DEBUG:boto:Params: {} 2023-11-02T18:30:16.065 DEBUG:boto:Token: None 2023-11-02T18:30:16.065 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.065 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.072 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc185a8c756373f3-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.073 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:16.073 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:16.074 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.074 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000042 2023-11-02T18:30:16.074 DEBUG:boto:Data: 2023-11-02T18:30:16.075 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.076 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.076 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.076 DEBUG:boto:Params: {} 2023-11-02T18:30:16.077 DEBUG:boto:Token: None 2023-11-02T18:30:16.077 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000042 2023-11-02T18:30:16.078 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L/j2Wn38Si5TfrDN1cCsXlDLKbI= 2023-11-02T18:30:16.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L/j2Wn38Si5TfrDN1cCsXlDLKbI='} 2023-11-02T18:30:16.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012e0209b257912d0-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.123 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.124 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.124 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.124 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.125 DEBUG:boto:Data: 2023-11-02T18:30:16.125 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.126 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.126 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.127 DEBUG:boto:Params: {} 2023-11-02T18:30:16.127 DEBUG:boto:Token: None 2023-11-02T18:30:16.128 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.128 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000189b5afc9c77dd60-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.136 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:16.137 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:16.137 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.137 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000059 2023-11-02T18:30:16.138 DEBUG:boto:Data: 2023-11-02T18:30:16.138 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.139 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.140 DEBUG:boto:Params: {} 2023-11-02T18:30:16.140 DEBUG:boto:Token: None 2023-11-02T18:30:16.141 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000059 2023-11-02T18:30:16.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+wGdHJ70AHC7HvZEOFKHcv/Xlag= 2023-11-02T18:30:16.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+wGdHJ70AHC7HvZEOFKHcv/Xlag='} 2023-11-02T18:30:16.151 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006049aae587c081d9-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.152 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.152 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.153 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.153 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.153 DEBUG:boto:Data: 2023-11-02T18:30:16.154 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.154 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.155 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.155 DEBUG:boto:Params: {} 2023-11-02T18:30:16.156 DEBUG:boto:Token: None 2023-11-02T18:30:16.156 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.157 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.163 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c904ecec20ac9ee6-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.164 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:16.165 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:16.165 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.165 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:16.166 DEBUG:boto:Data: 2023-11-02T18:30:16.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.167 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.167 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.168 DEBUG:boto:Params: {} 2023-11-02T18:30:16.168 DEBUG:boto:Token: None 2023-11-02T18:30:16.169 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:16.169 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xjZcXVBNaETw385WnCkiE0rK2LA= 2023-11-02T18:30:16.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xjZcXVBNaETw385WnCkiE0rK2LA='} 2023-11-02T18:30:16.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eee710c6cc6475b6-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.215 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.215 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.216 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.216 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.217 DEBUG:boto:Data: 2023-11-02T18:30:16.217 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.217 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.218 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.218 DEBUG:boto:Params: {} 2023-11-02T18:30:16.219 DEBUG:boto:Token: None 2023-11-02T18:30:16.219 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.220 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f15a80bd1428211-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.228 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:16.229 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:16.229 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.230 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:16.230 DEBUG:boto:Data: 2023-11-02T18:30:16.231 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.231 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.232 DEBUG:boto:Params: {} 2023-11-02T18:30:16.232 DEBUG:boto:Token: None 2023-11-02T18:30:16.233 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:16.233 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ICvp6LMlCrJl1wc6FH8tGWXwWmA= 2023-11-02T18:30:16.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ICvp6LMlCrJl1wc6FH8tGWXwWmA='} 2023-11-02T18:30:16.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b1b3f0b9f235873b-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.247 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.248 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.248 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.249 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.249 DEBUG:boto:Data: 2023-11-02T18:30:16.250 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.250 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.250 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.251 DEBUG:boto:Params: {} 2023-11-02T18:30:16.251 DEBUG:boto:Token: None 2023-11-02T18:30:16.252 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.252 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f006d5c2e1860e0-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.259 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:16.260 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:16.260 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.261 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:16.261 DEBUG:boto:Data: 2023-11-02T18:30:16.261 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.262 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.262 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.263 DEBUG:boto:Params: {} 2023-11-02T18:30:16.263 DEBUG:boto:Token: None 2023-11-02T18:30:16.264 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:16.264 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uhN8dXUBkpnK6wmMymLANgp9KQ= 2023-11-02T18:30:16.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uhN8dXUBkpnK6wmMymLANgp9KQ='} 2023-11-02T18:30:16.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052a12e775d5ff065-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.311 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.311 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.312 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.312 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.313 DEBUG:boto:Data: 2023-11-02T18:30:16.313 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.314 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.314 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.314 DEBUG:boto:Params: {} 2023-11-02T18:30:16.315 DEBUG:boto:Token: None 2023-11-02T18:30:16.315 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.316 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000318868141eb107e2-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.324 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:16.325 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:16.325 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.326 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000027 2023-11-02T18:30:16.326 DEBUG:boto:Data: 2023-11-02T18:30:16.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.327 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.327 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.328 DEBUG:boto:Params: {} 2023-11-02T18:30:16.328 DEBUG:boto:Token: None 2023-11-02T18:30:16.329 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000027 2023-11-02T18:30:16.330 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZvX1sIZ/DQudQOppbZfag09Bld8= 2023-11-02T18:30:16.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZvX1sIZ/DQudQOppbZfag09Bld8='} 2023-11-02T18:30:16.341 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000708c2afdc286231c-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.342 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.343 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.343 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.343 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.344 DEBUG:boto:Data: 2023-11-02T18:30:16.344 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.345 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.345 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.346 DEBUG:boto:Params: {} 2023-11-02T18:30:16.346 DEBUG:boto:Token: None 2023-11-02T18:30:16.347 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.347 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d41c789ee1f7a505-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.355 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:16.355 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:16.356 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.356 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:16.356 DEBUG:boto:Data: 2023-11-02T18:30:16.357 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.357 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.358 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.358 DEBUG:boto:Params: {} 2023-11-02T18:30:16.359 DEBUG:boto:Token: None 2023-11-02T18:30:16.359 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:16.360 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l6TZHDfG0ToCcvmBgEFHZRo/hRw= 2023-11-02T18:30:16.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l6TZHDfG0ToCcvmBgEFHZRo/hRw='} 2023-11-02T18:30:16.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b2298450bedbd9d-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.403 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.404 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.404 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.404 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.405 DEBUG:boto:Data: 2023-11-02T18:30:16.405 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.406 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.406 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.407 DEBUG:boto:Params: {} 2023-11-02T18:30:16.407 DEBUG:boto:Token: None 2023-11-02T18:30:16.408 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.408 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c25879e974051f4-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.416 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:16.417 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:16.417 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.418 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:16.418 DEBUG:boto:Data: 2023-11-02T18:30:16.418 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.419 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.419 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.420 DEBUG:boto:Params: {} 2023-11-02T18:30:16.420 DEBUG:boto:Token: None 2023-11-02T18:30:16.421 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:16.421 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DWh8+Tt28tZtB/y7T3n/INvNfg8= 2023-11-02T18:30:16.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DWh8+Tt28tZtB/y7T3n/INvNfg8='} 2023-11-02T18:30:16.432 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e13356675e1167e7-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.433 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.433 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.434 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.434 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.434 DEBUG:boto:Data: 2023-11-02T18:30:16.435 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.436 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.436 DEBUG:boto:Params: {} 2023-11-02T18:30:16.437 DEBUG:boto:Token: None 2023-11-02T18:30:16.437 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.438 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.445 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e9118a8b5900d44-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.445 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:16.446 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:16.446 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.447 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:16.447 DEBUG:boto:Data: 2023-11-02T18:30:16.448 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.448 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.449 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.449 DEBUG:boto:Params: {} 2023-11-02T18:30:16.449 DEBUG:boto:Token: None 2023-11-02T18:30:16.450 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:16.451 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Sem1vA+nqSDZBNOnssxppcCMh6Q= 2023-11-02T18:30:16.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Sem1vA+nqSDZBNOnssxppcCMh6Q='} 2023-11-02T18:30:16.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce1ca57c3cf9adbf-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.495 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.496 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.496 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.496 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.497 DEBUG:boto:Data: 2023-11-02T18:30:16.497 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.498 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.498 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.499 DEBUG:boto:Params: {} 2023-11-02T18:30:16.499 DEBUG:boto:Token: None 2023-11-02T18:30:16.499 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.500 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000dcb4dd3059a16a-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.508 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:16.508 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:16.509 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.509 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:16.509 DEBUG:boto:Data: 2023-11-02T18:30:16.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.511 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.511 DEBUG:boto:Params: {} 2023-11-02T18:30:16.512 DEBUG:boto:Token: None 2023-11-02T18:30:16.512 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:16.512 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A9IyM7E4ilaoYwBPAeWAvKfZbj4= 2023-11-02T18:30:16.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A9IyM7E4ilaoYwBPAeWAvKfZbj4='} 2023-11-02T18:30:16.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ed36e5d3a983135-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.527 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.527 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.528 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.528 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.529 DEBUG:boto:Data: 2023-11-02T18:30:16.529 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.530 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.530 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.530 DEBUG:boto:Params: {} 2023-11-02T18:30:16.531 DEBUG:boto:Token: None 2023-11-02T18:30:16.531 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.532 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b2c0391a08d7e32-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.539 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:16.539 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:16.540 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.540 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000022 2023-11-02T18:30:16.541 DEBUG:boto:Data: 2023-11-02T18:30:16.541 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.542 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.542 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.542 DEBUG:boto:Params: {} 2023-11-02T18:30:16.543 DEBUG:boto:Token: None 2023-11-02T18:30:16.543 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000022 2023-11-02T18:30:16.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VvdzLkbja/BN73C1iUWEFsVH0lo= 2023-11-02T18:30:16.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VvdzLkbja/BN73C1iUWEFsVH0lo='} 2023-11-02T18:30:16.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016f48dab47300f45-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.587 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.589 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.590 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.590 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.591 DEBUG:boto:Data: 2023-11-02T18:30:16.591 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.591 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.592 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.592 DEBUG:boto:Params: {} 2023-11-02T18:30:16.593 DEBUG:boto:Token: None 2023-11-02T18:30:16.593 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.594 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af0920074409d966-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.602 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:16.602 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:16.603 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.603 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:16.604 DEBUG:boto:Data: 2023-11-02T18:30:16.604 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.604 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.605 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.605 DEBUG:boto:Params: {} 2023-11-02T18:30:16.606 DEBUG:boto:Token: None 2023-11-02T18:30:16.606 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:16.607 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:48yljRfTEm1pnwGsR12zB1ydRJg= 2023-11-02T18:30:16.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:48yljRfTEm1pnwGsR12zB1ydRJg='} 2023-11-02T18:30:16.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000677635e865c34113-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.622 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.623 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.623 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.624 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.624 DEBUG:boto:Data: 2023-11-02T18:30:16.624 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.625 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.625 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.626 DEBUG:boto:Params: {} 2023-11-02T18:30:16.626 DEBUG:boto:Token: None 2023-11-02T18:30:16.627 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.627 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000486ac6f392aeb60-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.634 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:16.635 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:16.635 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.635 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:16.636 DEBUG:boto:Data: 2023-11-02T18:30:16.636 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.637 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.638 DEBUG:boto:Params: {} 2023-11-02T18:30:16.638 DEBUG:boto:Token: None 2023-11-02T18:30:16.638 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:16.639 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g4opZs7h7mEDhYUDsHN2JfzaoIk= 2023-11-02T18:30:16.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g4opZs7h7mEDhYUDsHN2JfzaoIk='} 2023-11-02T18:30:16.683 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb133187c5e86aed-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.683 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.684 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.684 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.685 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.685 DEBUG:boto:Data: 2023-11-02T18:30:16.685 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.686 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.687 DEBUG:boto:Params: {} 2023-11-02T18:30:16.687 DEBUG:boto:Token: None 2023-11-02T18:30:16.688 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.688 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.695 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028eb3f5d356d9526-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.696 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:16.697 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:16.697 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.697 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000025 2023-11-02T18:30:16.698 DEBUG:boto:Data: 2023-11-02T18:30:16.698 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.699 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.699 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.699 DEBUG:boto:Params: {} 2023-11-02T18:30:16.700 DEBUG:boto:Token: None 2023-11-02T18:30:16.700 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000025 2023-11-02T18:30:16.701 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2FW/XCCkUvYShf0vKPda9q+rLX4= 2023-11-02T18:30:16.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2FW/XCCkUvYShf0vKPda9q+rLX4='} 2023-11-02T18:30:16.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce69051be7fc324e-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.715 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.715 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.715 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.716 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.716 DEBUG:boto:Data: 2023-11-02T18:30:16.717 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.717 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.718 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.718 DEBUG:boto:Params: {} 2023-11-02T18:30:16.719 DEBUG:boto:Token: None 2023-11-02T18:30:16.719 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.720 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.726 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000800f7d4671975721-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.727 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:16.727 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:16.728 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.728 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000031 2023-11-02T18:30:16.728 DEBUG:boto:Data: 2023-11-02T18:30:16.729 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.729 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.730 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.730 DEBUG:boto:Params: {} 2023-11-02T18:30:16.731 DEBUG:boto:Token: None 2023-11-02T18:30:16.731 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000031 2023-11-02T18:30:16.732 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1rNsHS5DHCqcQR1zYq1cae2IQMI= 2023-11-02T18:30:16.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1rNsHS5DHCqcQR1zYq1cae2IQMI='} 2023-11-02T18:30:16.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053c88d2c5837f0e6-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.775 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.775 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.776 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.776 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.777 DEBUG:boto:Data: 2023-11-02T18:30:16.777 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.778 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.778 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.778 DEBUG:boto:Params: {} 2023-11-02T18:30:16.779 DEBUG:boto:Token: None 2023-11-02T18:30:16.779 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.780 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e4b1c7971492ad0-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.788 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:16.788 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:16.789 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.789 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:16.790 DEBUG:boto:Data: 2023-11-02T18:30:16.790 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.791 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.791 DEBUG:boto:Params: {} 2023-11-02T18:30:16.792 DEBUG:boto:Token: None 2023-11-02T18:30:16.792 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:16.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NOCY2Hb12zH7yIvN1hO7EfVzyTc= 2023-11-02T18:30:16.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NOCY2Hb12zH7yIvN1hO7EfVzyTc='} 2023-11-02T18:30:16.808 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ceb755c391f5616-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.808 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.809 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.809 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.809 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.810 DEBUG:boto:Data: 2023-11-02T18:30:16.810 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.811 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.811 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.812 DEBUG:boto:Params: {} 2023-11-02T18:30:16.812 DEBUG:boto:Token: None 2023-11-02T18:30:16.813 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.813 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.819 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087549298e5790195-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.820 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:16.820 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:16.820 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.821 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000037 2023-11-02T18:30:16.821 DEBUG:boto:Data: 2023-11-02T18:30:16.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.822 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.823 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.823 DEBUG:boto:Params: {} 2023-11-02T18:30:16.824 DEBUG:boto:Token: None 2023-11-02T18:30:16.824 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000037 2023-11-02T18:30:16.824 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0D2iKciJIVOOshUZLYLB4f4+21c= 2023-11-02T18:30:16.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0D2iKciJIVOOshUZLYLB4f4+21c='} 2023-11-02T18:30:16.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008eaaf24eb76a9412-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.871 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.871 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.872 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.872 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.873 DEBUG:boto:Data: 2023-11-02T18:30:16.873 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.874 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.874 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.874 DEBUG:boto:Params: {} 2023-11-02T18:30:16.875 DEBUG:boto:Token: None 2023-11-02T18:30:16.875 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.876 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000936ac637404e154c-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.884 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:16.884 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:16.885 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.885 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:16.885 DEBUG:boto:Data: 2023-11-02T18:30:16.886 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.887 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.887 DEBUG:boto:Params: {} 2023-11-02T18:30:16.888 DEBUG:boto:Token: None 2023-11-02T18:30:16.888 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:16.889 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bWtSc+lP5at/5AoZsAkfcKPzQSw= 2023-11-02T18:30:16.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bWtSc+lP5at/5AoZsAkfcKPzQSw='} 2023-11-02T18:30:16.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009cf81ec8577df93-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.903 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:16.904 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:16.904 DEBUG:boto:Method: HEAD 2023-11-02T18:30:16.904 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:16.905 DEBUG:boto:Data: 2023-11-02T18:30:16.905 DEBUG:boto:Headers: {} 2023-11-02T18:30:16.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.906 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.906 DEBUG:boto:Params: {} 2023-11-02T18:30:16.907 DEBUG:boto:Token: None 2023-11-02T18:30:16.907 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/ 2023-11-02T18:30:16.908 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo= 2023-11-02T18:30:16.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XnybRctgfvdga4Igsbs9xstMGCo=', 'Content-Length': '0'} 2023-11-02T18:30:16.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068e6fa4584d79c6f-006543eab8-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.915 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:16.915 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:16.916 DEBUG:boto:Method: PUT 2023-11-02T18:30:16.916 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000049 2023-11-02T18:30:16.917 DEBUG:boto:Data: 2023-11-02T18:30:16.917 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:16.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:16.918 DEBUG:boto:Port: 8000 2023-11-02T18:30:16.918 DEBUG:boto:Params: {} 2023-11-02T18:30:16.919 DEBUG:boto:Token: None 2023-11-02T18:30:16.919 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:16 GMT /ibtljn-58/obj00000000000000000049 2023-11-02T18:30:16.920 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XDmXVTSP39CV59/1Gw3NTq2Ma18= 2023-11-02T18:30:16.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:16 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XDmXVTSP39CV59/1Gw3NTq2Ma18='} 2023-11-02T18:30:16.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b7fb4a78c5db5d6-006543eab8-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:16.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:16.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:16.964 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-58 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:30:17.012 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:17.013 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:17.102 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:30:17.103 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-58 2023-11-02T18:30:17.126 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 41 2023-11-02T18:30:17.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:17.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:17.252 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:17.384 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:30:17.385 INFO:teuthology.orchestra.run.smithi017.stdout: "objs": [ 2023-11-02T18:30:17.385 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000019", 2023-11-02T18:30:17.385 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000006", 2023-11-02T18:30:17.385 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000467", 2023-11-02T18:30:17.386 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000005", 2023-11-02T18:30:17.386 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000056", 2023-11-02T18:30:17.386 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000004", 2023-11-02T18:30:17.386 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000023", 2023-11-02T18:30:17.387 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000317", 2023-11-02T18:30:17.387 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000022", 2023-11-02T18:30:17.387 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000000", 2023-11-02T18:30:17.387 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000021", 2023-11-02T18:30:17.388 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000009", 2023-11-02T18:30:17.388 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000271", 2023-11-02T18:30:17.388 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000008", 2023-11-02T18:30:17.388 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000018", 2023-11-02T18:30:17.389 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000007", 2023-11-02T18:30:17.389 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000017", 2023-11-02T18:30:17.389 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000162", 2023-11-02T18:30:17.389 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000016", 2023-11-02T18:30:17.389 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000003", 2023-11-02T18:30:17.390 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000423", 2023-11-02T18:30:17.390 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000002", 2023-11-02T18:30:17.390 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000012", 2023-11-02T18:30:17.390 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000001", 2023-11-02T18:30:17.391 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000011", 2023-11-02T18:30:17.391 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000283", 2023-11-02T18:30:17.391 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000010", 2023-11-02T18:30:17.391 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000061", 2023-11-02T18:30:17.391 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000050", 2023-11-02T18:30:17.392 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000029", 2023-11-02T18:30:17.392 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000133" 2023-11-02T18:30:17.392 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:30:17.392 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:30:17.394 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.394 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.395 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.395 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.395 DEBUG:boto:Data: 2023-11-02T18:30:17.396 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.396 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.397 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.397 DEBUG:boto:Params: {} 2023-11-02T18:30:17.397 DEBUG:boto:Token: None 2023-11-02T18:30:17.398 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.398 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008bca61067828ba4f-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.406 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:17.406 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:17.407 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.407 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:17.408 DEBUG:boto:Data: 2023-11-02T18:30:17.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.409 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.409 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.409 DEBUG:boto:Params: {} 2023-11-02T18:30:17.410 DEBUG:boto:Token: None 2023-11-02T18:30:17.410 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:17.411 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hDeDaWXGFPEjzqx6BuMgJpTP9RU= 2023-11-02T18:30:17.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hDeDaWXGFPEjzqx6BuMgJpTP9RU='} 2023-11-02T18:30:17.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce76862b54ee6f57-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.427 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.427 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.427 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.428 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.428 DEBUG:boto:Data: 2023-11-02T18:30:17.429 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.429 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.429 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.430 DEBUG:boto:Params: {} 2023-11-02T18:30:17.430 DEBUG:boto:Token: None 2023-11-02T18:30:17.431 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.431 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.438 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050a7ce0f2ef496a9-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.438 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:17.439 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:17.439 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.440 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:17.440 DEBUG:boto:Data: 2023-11-02T18:30:17.441 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.441 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.441 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.442 DEBUG:boto:Params: {} 2023-11-02T18:30:17.442 DEBUG:boto:Token: None 2023-11-02T18:30:17.443 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:17.443 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tgu4Jh6qY7kqyzUVNso6nlB7qOE= 2023-11-02T18:30:17.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tgu4Jh6qY7kqyzUVNso6nlB7qOE='} 2023-11-02T18:30:17.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000578556168799c65f-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.487 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.487 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.488 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.488 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.489 DEBUG:boto:Data: 2023-11-02T18:30:17.489 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.489 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.490 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.490 DEBUG:boto:Params: {} 2023-11-02T18:30:17.491 DEBUG:boto:Token: None 2023-11-02T18:30:17.491 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.492 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd547330915e809f-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.499 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:17.500 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:17.500 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.500 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000022 2023-11-02T18:30:17.501 DEBUG:boto:Data: 2023-11-02T18:30:17.501 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.502 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.502 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.503 DEBUG:boto:Params: {} 2023-11-02T18:30:17.503 DEBUG:boto:Token: None 2023-11-02T18:30:17.504 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000022 2023-11-02T18:30:17.504 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VhJ5ZdU+qbcGalz4pFSvuzpLE44= 2023-11-02T18:30:17.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VhJ5ZdU+qbcGalz4pFSvuzpLE44='} 2023-11-02T18:30:17.517 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6cedc92b2aecaca-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.518 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.518 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.519 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.519 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.519 DEBUG:boto:Data: 2023-11-02T18:30:17.520 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.520 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.521 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.521 DEBUG:boto:Params: {} 2023-11-02T18:30:17.521 DEBUG:boto:Token: None 2023-11-02T18:30:17.522 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.522 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000161efeaec14983ed-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.529 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:17.529 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:17.530 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.530 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000283 2023-11-02T18:30:17.531 DEBUG:boto:Data: 2023-11-02T18:30:17.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.532 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.532 DEBUG:boto:Params: {} 2023-11-02T18:30:17.533 DEBUG:boto:Token: None 2023-11-02T18:30:17.533 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000283 2023-11-02T18:30:17.534 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Igde3omJpLODILLQF3yKXaMrDEY= 2023-11-02T18:30:17.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Igde3omJpLODILLQF3yKXaMrDEY='} 2023-11-02T18:30:17.579 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001b68def6966977d-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.579 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.580 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.580 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.581 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.581 DEBUG:boto:Data: 2023-11-02T18:30:17.582 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.582 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.582 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.583 DEBUG:boto:Params: {} 2023-11-02T18:30:17.583 DEBUG:boto:Token: None 2023-11-02T18:30:17.584 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.585 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.591 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af779076677f26ed-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.592 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:17.592 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:17.593 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.593 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000467 2023-11-02T18:30:17.594 DEBUG:boto:Data: 2023-11-02T18:30:17.594 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.595 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.595 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.596 DEBUG:boto:Params: {} 2023-11-02T18:30:17.596 DEBUG:boto:Token: None 2023-11-02T18:30:17.597 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000467 2023-11-02T18:30:17.597 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x+BYcXqzW/brmk1v1fc9r0ZAuDo= 2023-11-02T18:30:17.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x+BYcXqzW/brmk1v1fc9r0ZAuDo='} 2023-11-02T18:30:17.608 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d14b9e70b1f2c530-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.609 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.609 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.610 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.610 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.610 DEBUG:boto:Data: 2023-11-02T18:30:17.611 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.611 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.613 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.614 DEBUG:boto:Params: {} 2023-11-02T18:30:17.614 DEBUG:boto:Token: None 2023-11-02T18:30:17.615 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.615 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6788facf062ddf5-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.627 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:17.627 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:17.628 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.628 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:17.628 DEBUG:boto:Data: 2023-11-02T18:30:17.629 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.629 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.630 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.630 DEBUG:boto:Params: {} 2023-11-02T18:30:17.631 DEBUG:boto:Token: None 2023-11-02T18:30:17.631 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:17.632 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Lqh0r42m4H+EEiF0Y73hq9p9MNE= 2023-11-02T18:30:17.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Lqh0r42m4H+EEiF0Y73hq9p9MNE='} 2023-11-02T18:30:17.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008dcaa704300db023-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.675 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.676 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.676 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.677 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.677 DEBUG:boto:Data: 2023-11-02T18:30:17.678 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.678 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.679 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.679 DEBUG:boto:Params: {} 2023-11-02T18:30:17.680 DEBUG:boto:Token: None 2023-11-02T18:30:17.680 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.681 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.689 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdd6e5f7f7cfc550-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.690 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:17.690 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:17.691 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.691 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000018 2023-11-02T18:30:17.692 DEBUG:boto:Data: 2023-11-02T18:30:17.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.693 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.693 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.693 DEBUG:boto:Params: {} 2023-11-02T18:30:17.694 DEBUG:boto:Token: None 2023-11-02T18:30:17.695 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000018 2023-11-02T18:30:17.695 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oLGefTD5WTHql2GMOVJzCST6Zo4= 2023-11-02T18:30:17.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oLGefTD5WTHql2GMOVJzCST6Zo4='} 2023-11-02T18:30:17.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000816ddbe421498c35-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.707 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.707 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.708 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.708 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.709 DEBUG:boto:Data: 2023-11-02T18:30:17.709 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.710 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.710 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.710 DEBUG:boto:Params: {} 2023-11-02T18:30:17.711 DEBUG:boto:Token: None 2023-11-02T18:30:17.711 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.719 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3eca5107ae2e432-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.720 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:17.720 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:17.721 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.721 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:17.721 DEBUG:boto:Data: 2023-11-02T18:30:17.722 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.722 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.723 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.723 DEBUG:boto:Params: {} 2023-11-02T18:30:17.724 DEBUG:boto:Token: None 2023-11-02T18:30:17.724 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:17.725 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jVvslWa+/2ysdLVgtFmW8MU2Svk= 2023-11-02T18:30:17.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jVvslWa+/2ysdLVgtFmW8MU2Svk='} 2023-11-02T18:30:17.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bfd44bef963c9555-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.771 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.772 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.772 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.773 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.773 DEBUG:boto:Data: 2023-11-02T18:30:17.773 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.774 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.774 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.775 DEBUG:boto:Params: {} 2023-11-02T18:30:17.775 DEBUG:boto:Token: None 2023-11-02T18:30:17.776 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.776 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f283e2df974db16-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.785 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:17.786 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:17.786 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.787 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:17.787 DEBUG:boto:Data: 2023-11-02T18:30:17.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.788 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.788 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.789 DEBUG:boto:Params: {} 2023-11-02T18:30:17.789 DEBUG:boto:Token: None 2023-11-02T18:30:17.790 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:17.790 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UE+CLHJxK6wmRcfwp9KIU3M/szw= 2023-11-02T18:30:17.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UE+CLHJxK6wmRcfwp9KIU3M/szw='} 2023-11-02T18:30:17.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000250f2b73799f5332-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.803 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.804 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.804 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.804 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.805 DEBUG:boto:Data: 2023-11-02T18:30:17.805 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.806 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.806 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.807 DEBUG:boto:Params: {} 2023-11-02T18:30:17.807 DEBUG:boto:Token: None 2023-11-02T18:30:17.808 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.808 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.814 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000501a50449eb05f0-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.817 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:17.817 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:17.818 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.818 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:17.819 DEBUG:boto:Data: 2023-11-02T18:30:17.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.820 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.820 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.820 DEBUG:boto:Params: {} 2023-11-02T18:30:17.821 DEBUG:boto:Token: None 2023-11-02T18:30:17.821 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:17.822 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lKxD+KmiFmXFDOUpy2r4UmRHugI= 2023-11-02T18:30:17.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lKxD+KmiFmXFDOUpy2r4UmRHugI='} 2023-11-02T18:30:17.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf61fcb6f6a78365-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.872 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.872 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.873 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.873 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.874 DEBUG:boto:Data: 2023-11-02T18:30:17.874 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.875 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.875 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.876 DEBUG:boto:Params: {} 2023-11-02T18:30:17.876 DEBUG:boto:Token: None 2023-11-02T18:30:17.877 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c89ea7ab2dbcc03-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.887 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:17.887 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:17.887 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.888 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000029 2023-11-02T18:30:17.888 DEBUG:boto:Data: 2023-11-02T18:30:17.889 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.889 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.890 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.890 DEBUG:boto:Params: {} 2023-11-02T18:30:17.890 DEBUG:boto:Token: None 2023-11-02T18:30:17.891 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000029 2023-11-02T18:30:17.891 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:phJj42RcSXidaCSY5llAGD5arww= 2023-11-02T18:30:17.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:phJj42RcSXidaCSY5llAGD5arww='} 2023-11-02T18:30:17.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000188cc75a22b54a7e-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.904 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.905 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.905 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.906 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.906 DEBUG:boto:Data: 2023-11-02T18:30:17.907 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.907 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.908 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.908 DEBUG:boto:Params: {} 2023-11-02T18:30:17.909 DEBUG:boto:Token: None 2023-11-02T18:30:17.909 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.910 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e62660a82ab3c20-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.918 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:17.919 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:17.919 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.920 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:17.920 DEBUG:boto:Data: 2023-11-02T18:30:17.921 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.921 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.921 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.922 DEBUG:boto:Params: {} 2023-11-02T18:30:17.923 DEBUG:boto:Token: None 2023-11-02T18:30:17.923 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:17.924 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QUqb3MqhkAO5SVcgD62WcEIkkV4= 2023-11-02T18:30:17.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QUqb3MqhkAO5SVcgD62WcEIkkV4='} 2023-11-02T18:30:17.967 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000cf7664cd8dd4835-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.967 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:17.968 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:17.968 DEBUG:boto:Method: HEAD 2023-11-02T18:30:17.969 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:17.969 DEBUG:boto:Data: 2023-11-02T18:30:17.970 DEBUG:boto:Headers: {} 2023-11-02T18:30:17.970 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.971 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.971 DEBUG:boto:Params: {} 2023-11-02T18:30:17.972 DEBUG:boto:Token: None 2023-11-02T18:30:17.973 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/ 2023-11-02T18:30:17.973 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ= 2023-11-02T18:30:17.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yM4KLLH8Id1j7XfdJm/irRw9StQ=', 'Content-Length': '0'} 2023-11-02T18:30:17.981 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036a0b59c87645011-006543eab9-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:17.982 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:17.983 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:17.984 DEBUG:boto:Method: PUT 2023-11-02T18:30:17.985 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:17.985 DEBUG:boto:Data: 2023-11-02T18:30:17.986 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:17.987 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:17.987 DEBUG:boto:Port: 8000 2023-11-02T18:30:17.988 DEBUG:boto:Params: {} 2023-11-02T18:30:17.989 DEBUG:boto:Token: None 2023-11-02T18:30:17.990 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:17 GMT /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:17.990 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z2YvbFM4jMHTS3A6e6CWY3aZHdc= 2023-11-02T18:30:17.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z2YvbFM4jMHTS3A6e6CWY3aZHdc='} 2023-11-02T18:30:18.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003311c357da19371e-006543eab9-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.007 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.007 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.008 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.008 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.009 DEBUG:boto:Data: 2023-11-02T18:30:18.009 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.010 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.010 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.011 DEBUG:boto:Params: {} 2023-11-02T18:30:18.011 DEBUG:boto:Token: None 2023-11-02T18:30:18.012 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a32a60d5d9f41b3-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.022 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:18.023 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:18.023 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.024 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:18.024 DEBUG:boto:Data: 2023-11-02T18:30:18.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.025 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.025 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.026 DEBUG:boto:Params: {} 2023-11-02T18:30:18.026 DEBUG:boto:Token: None 2023-11-02T18:30:18.027 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:18.027 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b5rwaBBwobbwLVjz5l3gbnHhvn0= 2023-11-02T18:30:18.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b5rwaBBwobbwLVjz5l3gbnHhvn0='} 2023-11-02T18:30:18.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebee69f2b3257bbc-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.071 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.072 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.072 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.072 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.073 DEBUG:boto:Data: 2023-11-02T18:30:18.073 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.074 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.074 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.075 DEBUG:boto:Params: {} 2023-11-02T18:30:18.075 DEBUG:boto:Token: None 2023-11-02T18:30:18.075 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.076 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003115f2a76b849090-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.084 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:18.085 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:18.085 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.086 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000423 2023-11-02T18:30:18.086 DEBUG:boto:Data: 2023-11-02T18:30:18.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.087 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.088 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.088 DEBUG:boto:Params: {} 2023-11-02T18:30:18.088 DEBUG:boto:Token: None 2023-11-02T18:30:18.089 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000423 2023-11-02T18:30:18.089 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1a9kpmZ9d3NFmDvevCND5rPBWKQ= 2023-11-02T18:30:18.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1a9kpmZ9d3NFmDvevCND5rPBWKQ='} 2023-11-02T18:30:18.100 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd8b71f825a3eaf2-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.101 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.101 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.102 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.102 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.102 DEBUG:boto:Data: 2023-11-02T18:30:18.103 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.103 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.104 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.104 DEBUG:boto:Params: {} 2023-11-02T18:30:18.104 DEBUG:boto:Token: None 2023-11-02T18:30:18.105 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.105 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013c1eb37f16bfc1a-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.113 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:18.114 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:18.114 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.115 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000317 2023-11-02T18:30:18.115 DEBUG:boto:Data: 2023-11-02T18:30:18.115 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.116 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.116 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.117 DEBUG:boto:Params: {} 2023-11-02T18:30:18.117 DEBUG:boto:Token: None 2023-11-02T18:30:18.118 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000317 2023-11-02T18:30:18.118 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LCIK+fKewYdgIro0jvNPqxebsrc= 2023-11-02T18:30:18.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LCIK+fKewYdgIro0jvNPqxebsrc='} 2023-11-02T18:30:18.163 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000acd091b0aedbf88-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.163 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.164 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.164 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.165 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.165 DEBUG:boto:Data: 2023-11-02T18:30:18.166 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.166 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.167 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.167 DEBUG:boto:Params: {} 2023-11-02T18:30:18.168 DEBUG:boto:Token: None 2023-11-02T18:30:18.168 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.169 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e4db42cc5a9df36-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.178 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:18.178 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:18.179 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.179 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:18.180 DEBUG:boto:Data: 2023-11-02T18:30:18.180 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.181 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.181 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.182 DEBUG:boto:Params: {} 2023-11-02T18:30:18.182 DEBUG:boto:Token: None 2023-11-02T18:30:18.183 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:18.183 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oCUdOOw+us6nrAZkOsbX5HNY4No= 2023-11-02T18:30:18.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oCUdOOw+us6nrAZkOsbX5HNY4No='} 2023-11-02T18:30:18.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c69a72027110439-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.197 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.197 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.198 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.198 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.198 DEBUG:boto:Data: 2023-11-02T18:30:18.199 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.199 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.200 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.200 DEBUG:boto:Params: {} 2023-11-02T18:30:18.201 DEBUG:boto:Token: None 2023-11-02T18:30:18.201 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.208 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054fdc87deef5cc32-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.209 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:18.210 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:18.210 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.210 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:18.211 DEBUG:boto:Data: 2023-11-02T18:30:18.211 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.212 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.212 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.213 DEBUG:boto:Params: {} 2023-11-02T18:30:18.213 DEBUG:boto:Token: None 2023-11-02T18:30:18.214 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:18.214 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Hl2FApY3LWrK4P+gOfEaC5Z09ww= 2023-11-02T18:30:18.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Hl2FApY3LWrK4P+gOfEaC5Z09ww='} 2023-11-02T18:30:18.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000525d66ed1aac6296-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.259 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.260 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.260 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.261 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.261 DEBUG:boto:Data: 2023-11-02T18:30:18.262 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.262 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.262 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.263 DEBUG:boto:Params: {} 2023-11-02T18:30:18.263 DEBUG:boto:Token: None 2023-11-02T18:30:18.264 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.264 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a923147957a090a7-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.273 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:18.273 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:18.274 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.274 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000056 2023-11-02T18:30:18.275 DEBUG:boto:Data: 2023-11-02T18:30:18.275 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.276 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.276 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.276 DEBUG:boto:Params: {} 2023-11-02T18:30:18.277 DEBUG:boto:Token: None 2023-11-02T18:30:18.277 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000056 2023-11-02T18:30:18.278 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0UizP1lh9tTRFuRvDSYLj3XuZIs= 2023-11-02T18:30:18.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0UizP1lh9tTRFuRvDSYLj3XuZIs='} 2023-11-02T18:30:18.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065b7a3ab5fd78101-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.291 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.291 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.291 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.292 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.292 DEBUG:boto:Data: 2023-11-02T18:30:18.293 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.293 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.293 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.294 DEBUG:boto:Params: {} 2023-11-02T18:30:18.294 DEBUG:boto:Token: None 2023-11-02T18:30:18.295 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.295 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.300 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be5a6a27ca77f801-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.301 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:18.301 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:18.302 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.302 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000162 2023-11-02T18:30:18.302 DEBUG:boto:Data: 2023-11-02T18:30:18.303 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.304 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.304 DEBUG:boto:Params: {} 2023-11-02T18:30:18.305 DEBUG:boto:Token: None 2023-11-02T18:30:18.305 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000162 2023-11-02T18:30:18.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1cyWUR8gN73efgkTRrPg/+GkH5s= 2023-11-02T18:30:18.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1cyWUR8gN73efgkTRrPg/+GkH5s='} 2023-11-02T18:30:18.351 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a43c9aec7cd362d-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.351 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.352 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.352 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.353 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.353 DEBUG:boto:Data: 2023-11-02T18:30:18.354 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.354 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.354 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.355 DEBUG:boto:Params: {} 2023-11-02T18:30:18.355 DEBUG:boto:Token: None 2023-11-02T18:30:18.356 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.356 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d78e48c81e62cf67-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.365 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:18.365 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:18.366 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.366 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:18.366 DEBUG:boto:Data: 2023-11-02T18:30:18.367 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.367 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.368 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.368 DEBUG:boto:Params: {} 2023-11-02T18:30:18.369 DEBUG:boto:Token: None 2023-11-02T18:30:18.369 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:18.370 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JwUpDNkHVNZrjzltqi26o8+g/+I= 2023-11-02T18:30:18.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JwUpDNkHVNZrjzltqi26o8+g/+I='} 2023-11-02T18:30:18.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009aacb3d715f5acfe-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.384 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.384 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.385 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.385 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.386 DEBUG:boto:Data: 2023-11-02T18:30:18.386 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.387 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.387 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.387 DEBUG:boto:Params: {} 2023-11-02T18:30:18.388 DEBUG:boto:Token: None 2023-11-02T18:30:18.388 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.389 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087bbc2ae50e7a710-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.396 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:18.396 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:18.397 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.397 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000050 2023-11-02T18:30:18.398 DEBUG:boto:Data: 2023-11-02T18:30:18.398 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.399 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.399 DEBUG:boto:Params: {} 2023-11-02T18:30:18.400 DEBUG:boto:Token: None 2023-11-02T18:30:18.400 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000050 2023-11-02T18:30:18.401 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S5mE36wGKwgCzsndB6rnao3uceE= 2023-11-02T18:30:18.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S5mE36wGKwgCzsndB6rnao3uceE='} 2023-11-02T18:30:18.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd6046af82030cab-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.447 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.448 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.448 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.449 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.449 DEBUG:boto:Data: 2023-11-02T18:30:18.450 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.450 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.450 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.451 DEBUG:boto:Params: {} 2023-11-02T18:30:18.451 DEBUG:boto:Token: None 2023-11-02T18:30:18.452 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.452 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.460 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fca1d771d80b7239-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.461 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:18.461 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:18.462 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.462 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000133 2023-11-02T18:30:18.463 DEBUG:boto:Data: 2023-11-02T18:30:18.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.464 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.464 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.465 DEBUG:boto:Params: {} 2023-11-02T18:30:18.465 DEBUG:boto:Token: None 2023-11-02T18:30:18.466 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000133 2023-11-02T18:30:18.466 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4eV8ghYpYUydrQAvo6daVd/7mDE= 2023-11-02T18:30:18.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4eV8ghYpYUydrQAvo6daVd/7mDE='} 2023-11-02T18:30:18.477 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053c351e2295dadd7-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.478 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.478 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.479 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.479 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.479 DEBUG:boto:Data: 2023-11-02T18:30:18.480 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.480 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.481 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.481 DEBUG:boto:Params: {} 2023-11-02T18:30:18.482 DEBUG:boto:Token: None 2023-11-02T18:30:18.482 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.483 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067c0548be8ddc722-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.490 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:18.491 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:18.491 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.492 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:18.492 DEBUG:boto:Data: 2023-11-02T18:30:18.493 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.494 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.495 DEBUG:boto:Params: {} 2023-11-02T18:30:18.495 DEBUG:boto:Token: None 2023-11-02T18:30:18.496 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:18.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JnT2qjmzQpD62wxGZWZxImxj0Dw= 2023-11-02T18:30:18.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JnT2qjmzQpD62wxGZWZxImxj0Dw='} 2023-11-02T18:30:18.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006facbeabad829add-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.543 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.544 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.544 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.544 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.545 DEBUG:boto:Data: 2023-11-02T18:30:18.545 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.546 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.546 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.547 DEBUG:boto:Params: {} 2023-11-02T18:30:18.547 DEBUG:boto:Token: None 2023-11-02T18:30:18.547 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.548 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000572000060f4473b2-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.557 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:18.557 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:18.558 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.558 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000061 2023-11-02T18:30:18.558 DEBUG:boto:Data: 2023-11-02T18:30:18.559 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.559 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.560 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.560 DEBUG:boto:Params: {} 2023-11-02T18:30:18.561 DEBUG:boto:Token: None 2023-11-02T18:30:18.561 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000061 2023-11-02T18:30:18.562 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SJ4Ld7geY5k8UbSWNa42CPJV1rQ= 2023-11-02T18:30:18.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SJ4Ld7geY5k8UbSWNa42CPJV1rQ='} 2023-11-02T18:30:18.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d64ef9835c85d6d3-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.573 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.573 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.574 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.574 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.574 DEBUG:boto:Data: 2023-11-02T18:30:18.575 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.575 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.576 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.576 DEBUG:boto:Params: {} 2023-11-02T18:30:18.577 DEBUG:boto:Token: None 2023-11-02T18:30:18.577 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.578 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c15d39da99b1f42-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.586 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:18.586 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:18.586 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.587 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000023 2023-11-02T18:30:18.587 DEBUG:boto:Data: 2023-11-02T18:30:18.588 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.588 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.589 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.589 DEBUG:boto:Params: {} 2023-11-02T18:30:18.589 DEBUG:boto:Token: None 2023-11-02T18:30:18.590 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000023 2023-11-02T18:30:18.590 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XoPI4G+Ur0Etb2vbET9m3fcTRa4= 2023-11-02T18:30:18.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XoPI4G+Ur0Etb2vbET9m3fcTRa4='} 2023-11-02T18:30:18.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007894cda800422e7d-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.635 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.636 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.636 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.636 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.637 DEBUG:boto:Data: 2023-11-02T18:30:18.637 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.638 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.638 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.639 DEBUG:boto:Params: {} 2023-11-02T18:30:18.639 DEBUG:boto:Token: None 2023-11-02T18:30:18.640 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.640 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebfb90f871ead586-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.648 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:18.649 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:18.649 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.650 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:18.650 DEBUG:boto:Data: 2023-11-02T18:30:18.651 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.651 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.652 DEBUG:boto:Params: {} 2023-11-02T18:30:18.652 DEBUG:boto:Token: None 2023-11-02T18:30:18.653 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:18.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z2Iziotydgv1deY4zDk/L20lPiE= 2023-11-02T18:30:18.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z2Iziotydgv1deY4zDk/L20lPiE='} 2023-11-02T18:30:18.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3dac88f31e2d1c7-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.668 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:18.668 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:18.669 DEBUG:boto:Method: HEAD 2023-11-02T18:30:18.669 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:18.670 DEBUG:boto:Data: 2023-11-02T18:30:18.670 DEBUG:boto:Headers: {} 2023-11-02T18:30:18.671 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.671 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.671 DEBUG:boto:Params: {} 2023-11-02T18:30:18.672 DEBUG:boto:Token: None 2023-11-02T18:30:18.672 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/ 2023-11-02T18:30:18.673 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA= 2023-11-02T18:30:18.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QZbAJq/+3LMaumKxB71PQ28GwGA=', 'Content-Length': '0'} 2023-11-02T18:30:18.680 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0a5cbf5c74e476f-006543eaba-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.681 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:18.681 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:18.682 DEBUG:boto:Method: PUT 2023-11-02T18:30:18.682 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000271 2023-11-02T18:30:18.683 DEBUG:boto:Data: 2023-11-02T18:30:18.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:18.684 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:18.684 DEBUG:boto:Port: 8000 2023-11-02T18:30:18.685 DEBUG:boto:Params: {} 2023-11-02T18:30:18.685 DEBUG:boto:Token: None 2023-11-02T18:30:18.686 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:18 GMT /ibtljn-58/obj00000000000000000271 2023-11-02T18:30:18.686 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c91Y3L/nTyA/9zOSHjpKxPw7MrM= 2023-11-02T18:30:18.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c91Y3L/nTyA/9zOSHjpKxPw7MrM='} 2023-11-02T18:30:18.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d2d61cde82c7830-006543eaba-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:18.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:18.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:18.732 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-58 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:30:18.785 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:18.785 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:18.879 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:30:18.880 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-58 2023-11-02T18:30:18.907 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 53 2023-11-02T18:30:19.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:19.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:19.040 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:19.087 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:19.087 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:19.171 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:30:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: "objs": [ 2023-11-02T18:30:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000060", 2023-11-02T18:30:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000000", 2023-11-02T18:30:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000011", 2023-11-02T18:30:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000001", 2023-11-02T18:30:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000033", 2023-11-02T18:30:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000065", 2023-11-02T18:30:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000005", 2023-11-02T18:30:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000016", 2023-11-02T18:30:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000048", 2023-11-02T18:30:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000015", 2023-11-02T18:30:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000026", 2023-11-02T18:30:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000030", 2023-11-02T18:30:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000062", 2023-11-02T18:30:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000009", 2023-11-02T18:30:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000040", 2023-11-02T18:30:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000024", 2023-11-02T18:30:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000012", 2023-11-02T18:30:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000002", 2023-11-02T18:30:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000013", 2023-11-02T18:30:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000045", 2023-11-02T18:30:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000006", 2023-11-02T18:30:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000017", 2023-11-02T18:30:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000007", 2023-11-02T18:30:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000039", 2023-11-02T18:30:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000027", 2023-11-02T18:30:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000010", 2023-11-02T18:30:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000021", 2023-11-02T18:30:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000053", 2023-11-02T18:30:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000020", 2023-11-02T18:30:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000004", 2023-11-02T18:30:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000036", 2023-11-02T18:30:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000003", 2023-11-02T18:30:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000014", 2023-11-02T18:30:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000019", 2023-11-02T18:30:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000057", 2023-11-02T18:30:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000018", 2023-11-02T18:30:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: "obj00000000000000000008" 2023-11-02T18:30:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:30:19.181 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:30:19.182 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.183 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.183 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.183 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.184 DEBUG:boto:Data: 2023-11-02T18:30:19.184 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.185 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.185 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.185 DEBUG:boto:Params: {} 2023-11-02T18:30:19.186 DEBUG:boto:Token: None 2023-11-02T18:30:19.186 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.187 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.195 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000baa308cf60a7d5f-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.196 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:19.196 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:19.197 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.197 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:19.197 DEBUG:boto:Data: 2023-11-02T18:30:19.198 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.198 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.199 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.199 DEBUG:boto:Params: {} 2023-11-02T18:30:19.200 DEBUG:boto:Token: None 2023-11-02T18:30:19.200 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:19.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0HYWTMYe/SajypuhFEH9iSuhaMQ= 2023-11-02T18:30:19.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0HYWTMYe/SajypuhFEH9iSuhaMQ='} 2023-11-02T18:30:19.213 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006054545dff8e2839-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.214 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.214 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.215 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.215 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.216 DEBUG:boto:Data: 2023-11-02T18:30:19.216 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.217 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.217 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.218 DEBUG:boto:Params: {} 2023-11-02T18:30:19.218 DEBUG:boto:Token: None 2023-11-02T18:30:19.219 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.219 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005784bcc338f7fb61-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.227 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:19.228 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:19.228 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.229 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000027 2023-11-02T18:30:19.229 DEBUG:boto:Data: 2023-11-02T18:30:19.229 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.230 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.230 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.231 DEBUG:boto:Params: {} 2023-11-02T18:30:19.231 DEBUG:boto:Token: None 2023-11-02T18:30:19.232 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000027 2023-11-02T18:30:19.232 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IKdUt/j2PTuQDpy+ObVwM+Cj6N4= 2023-11-02T18:30:19.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IKdUt/j2PTuQDpy+ObVwM+Cj6N4='} 2023-11-02T18:30:19.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0da293cc22ce53a-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.279 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.280 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.280 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.280 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.281 DEBUG:boto:Data: 2023-11-02T18:30:19.281 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.282 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.282 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.283 DEBUG:boto:Params: {} 2023-11-02T18:30:19.283 DEBUG:boto:Token: None 2023-11-02T18:30:19.283 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.284 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.293 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f8a857b80592a33-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.294 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:19.294 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:19.295 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.295 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000053 2023-11-02T18:30:19.295 DEBUG:boto:Data: 2023-11-02T18:30:19.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.296 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.297 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.297 DEBUG:boto:Params: {} 2023-11-02T18:30:19.297 DEBUG:boto:Token: None 2023-11-02T18:30:19.298 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000053 2023-11-02T18:30:19.298 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:02UhKYti8kh8R8XE/B+mNITcNU0= 2023-11-02T18:30:19.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:02UhKYti8kh8R8XE/B+mNITcNU0='} 2023-11-02T18:30:19.308 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da21b74fce905d5c-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.308 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.309 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.309 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.310 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.310 DEBUG:boto:Data: 2023-11-02T18:30:19.310 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.311 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.311 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.312 DEBUG:boto:Params: {} 2023-11-02T18:30:19.312 DEBUG:boto:Token: None 2023-11-02T18:30:19.313 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.313 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.320 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d96db8947f2ec7f0-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.320 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:19.321 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:19.321 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.322 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:19.322 DEBUG:boto:Data: 2023-11-02T18:30:19.322 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.323 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.323 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.324 DEBUG:boto:Params: {} 2023-11-02T18:30:19.324 DEBUG:boto:Token: None 2023-11-02T18:30:19.325 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:19.325 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZZAK+uJYCkSOtoIxG0TE0gVpx2M= 2023-11-02T18:30:19.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZZAK+uJYCkSOtoIxG0TE0gVpx2M='} 2023-11-02T18:30:19.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8c9b9b0a459117e-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.375 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.376 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.376 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.376 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.377 DEBUG:boto:Data: 2023-11-02T18:30:19.377 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.378 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.378 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.378 DEBUG:boto:Params: {} 2023-11-02T18:30:19.379 DEBUG:boto:Token: None 2023-11-02T18:30:19.379 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.380 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.388 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098262bdf837e74b6-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.389 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:19.390 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:19.390 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.390 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:19.391 DEBUG:boto:Data: 2023-11-02T18:30:19.391 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.392 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.392 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.392 DEBUG:boto:Params: {} 2023-11-02T18:30:19.393 DEBUG:boto:Token: None 2023-11-02T18:30:19.393 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:19.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nQ/a0OWd8+nWBLroutr78hz7M20= 2023-11-02T18:30:19.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nQ/a0OWd8+nWBLroutr78hz7M20='} 2023-11-02T18:30:19.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d08122d9da2d0760-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.407 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.407 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.408 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.408 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.408 DEBUG:boto:Data: 2023-11-02T18:30:19.409 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.409 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.410 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.410 DEBUG:boto:Params: {} 2023-11-02T18:30:19.411 DEBUG:boto:Token: None 2023-11-02T18:30:19.411 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0680204571bf700-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.419 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:19.420 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:19.420 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.421 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000048 2023-11-02T18:30:19.421 DEBUG:boto:Data: 2023-11-02T18:30:19.421 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.423 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.423 DEBUG:boto:Params: {} 2023-11-02T18:30:19.423 DEBUG:boto:Token: None 2023-11-02T18:30:19.424 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000048 2023-11-02T18:30:19.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eHjbLZyzfp37OqwKYY9HfVRdf3E= 2023-11-02T18:30:19.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eHjbLZyzfp37OqwKYY9HfVRdf3E='} 2023-11-02T18:30:19.471 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051970549dd5dbfad-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.471 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.472 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.472 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.473 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.473 DEBUG:boto:Data: 2023-11-02T18:30:19.473 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.474 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.474 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.475 DEBUG:boto:Params: {} 2023-11-02T18:30:19.475 DEBUG:boto:Token: None 2023-11-02T18:30:19.476 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.476 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.485 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011d208bca12f4fea-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.485 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:19.486 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:19.486 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.487 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:19.487 DEBUG:boto:Data: 2023-11-02T18:30:19.487 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.488 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.488 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.489 DEBUG:boto:Params: {} 2023-11-02T18:30:19.489 DEBUG:boto:Token: None 2023-11-02T18:30:19.490 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:19.490 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0rCFb+8UOHFv7WoZs4LvYh77dLU= 2023-11-02T18:30:19.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0rCFb+8UOHFv7WoZs4LvYh77dLU='} 2023-11-02T18:30:19.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a39120d48bd838b-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.504 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.504 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.505 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.505 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.506 DEBUG:boto:Data: 2023-11-02T18:30:19.506 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.506 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.507 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.507 DEBUG:boto:Params: {} 2023-11-02T18:30:19.508 DEBUG:boto:Token: None 2023-11-02T18:30:19.508 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.509 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc9cf74a57ee7049-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.517 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:19.517 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:19.518 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.518 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000024 2023-11-02T18:30:19.518 DEBUG:boto:Data: 2023-11-02T18:30:19.519 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.520 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.520 DEBUG:boto:Params: {} 2023-11-02T18:30:19.521 DEBUG:boto:Token: None 2023-11-02T18:30:19.521 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000024 2023-11-02T18:30:19.522 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJMR7OhfcWc0FRFe/Gs9O2Cc1f8= 2023-11-02T18:30:19.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJMR7OhfcWc0FRFe/Gs9O2Cc1f8='} 2023-11-02T18:30:19.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4d352c1c871a6b0-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.567 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.568 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.568 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.569 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.569 DEBUG:boto:Data: 2023-11-02T18:30:19.569 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.570 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.570 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.571 DEBUG:boto:Params: {} 2023-11-02T18:30:19.571 DEBUG:boto:Token: None 2023-11-02T18:30:19.572 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.572 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c91089d3956de0b-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.587 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:19.587 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:19.588 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.588 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:19.589 DEBUG:boto:Data: 2023-11-02T18:30:19.589 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.590 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.590 DEBUG:boto:Params: {} 2023-11-02T18:30:19.591 DEBUG:boto:Token: None 2023-11-02T18:30:19.591 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:19.592 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OKxMTTxgp4thlXw0dxZ5ZBGYnGo= 2023-11-02T18:30:19.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OKxMTTxgp4thlXw0dxZ5ZBGYnGo='} 2023-11-02T18:30:19.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7cfd6ed5d13276b-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.602 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.603 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.603 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.604 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.604 DEBUG:boto:Data: 2023-11-02T18:30:19.605 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.605 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.605 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.606 DEBUG:boto:Params: {} 2023-11-02T18:30:19.606 DEBUG:boto:Token: None 2023-11-02T18:30:19.607 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.607 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.615 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000261f465f860de081-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.616 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:19.617 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:19.617 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.617 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:19.618 DEBUG:boto:Data: 2023-11-02T18:30:19.618 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.619 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.619 DEBUG:boto:Params: {} 2023-11-02T18:30:19.620 DEBUG:boto:Token: None 2023-11-02T18:30:19.620 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:19.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Od8CtAENYoM7w0vGj0O8hbvMVs0= 2023-11-02T18:30:19.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Od8CtAENYoM7w0vGj0O8hbvMVs0='} 2023-11-02T18:30:19.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000855c6872562adb49-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.668 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.668 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.669 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.669 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.670 DEBUG:boto:Data: 2023-11-02T18:30:19.670 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.671 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.671 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.671 DEBUG:boto:Params: {} 2023-11-02T18:30:19.672 DEBUG:boto:Token: None 2023-11-02T18:30:19.672 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.673 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003010dcbc85d4f896-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.681 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:19.682 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:19.682 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.683 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:19.683 DEBUG:boto:Data: 2023-11-02T18:30:19.684 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.684 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.684 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.685 DEBUG:boto:Params: {} 2023-11-02T18:30:19.685 DEBUG:boto:Token: None 2023-11-02T18:30:19.686 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:19.686 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aMQJTjCeSINJO4S4GXz3NGFM5co= 2023-11-02T18:30:19.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aMQJTjCeSINJO4S4GXz3NGFM5co='} 2023-11-02T18:30:19.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b969cb5bbba7a7f8-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.702 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.702 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.703 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.703 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.703 DEBUG:boto:Data: 2023-11-02T18:30:19.704 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.704 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.705 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.705 DEBUG:boto:Params: {} 2023-11-02T18:30:19.706 DEBUG:boto:Token: None 2023-11-02T18:30:19.706 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.706 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.713 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fc7f29980f35df4-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.713 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:19.714 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:19.714 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.715 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:19.715 DEBUG:boto:Data: 2023-11-02T18:30:19.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.716 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.716 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.717 DEBUG:boto:Params: {} 2023-11-02T18:30:19.717 DEBUG:boto:Token: None 2023-11-02T18:30:19.718 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:19.718 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gQ1gNvyPE7J7R+Ay6Es9bu4PY8o= 2023-11-02T18:30:19.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gQ1gNvyPE7J7R+Ay6Es9bu4PY8o='} 2023-11-02T18:30:19.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000980ef5b09c0ebe08-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.763 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.764 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.764 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.765 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.765 DEBUG:boto:Data: 2023-11-02T18:30:19.765 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.766 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.766 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.767 DEBUG:boto:Params: {} 2023-11-02T18:30:19.767 DEBUG:boto:Token: None 2023-11-02T18:30:19.768 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a61c6e30109d2173-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.777 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:19.777 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:19.778 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.778 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:19.779 DEBUG:boto:Data: 2023-11-02T18:30:19.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.780 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.780 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.781 DEBUG:boto:Params: {} 2023-11-02T18:30:19.781 DEBUG:boto:Token: None 2023-11-02T18:30:19.782 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:19.783 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:taesugUkAuC7A8ytRzt4yzjaGxI= 2023-11-02T18:30:19.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:taesugUkAuC7A8ytRzt4yzjaGxI='} 2023-11-02T18:30:19.793 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b31f73415f149a5b-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.794 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.794 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.795 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.795 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.796 DEBUG:boto:Data: 2023-11-02T18:30:19.796 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.796 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.797 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.797 DEBUG:boto:Params: {} 2023-11-02T18:30:19.798 DEBUG:boto:Token: None 2023-11-02T18:30:19.798 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.799 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002303add981073495-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.806 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:19.806 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:19.807 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.807 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000018 2023-11-02T18:30:19.807 DEBUG:boto:Data: 2023-11-02T18:30:19.808 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.808 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.809 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.809 DEBUG:boto:Params: {} 2023-11-02T18:30:19.810 DEBUG:boto:Token: None 2023-11-02T18:30:19.810 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000018 2023-11-02T18:30:19.811 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dpQSPyKT9qDaYYJ1XSWmExCppYM= 2023-11-02T18:30:19.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dpQSPyKT9qDaYYJ1XSWmExCppYM='} 2023-11-02T18:30:19.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d439e5bbf94f0685-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.855 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.856 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.856 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.857 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.857 DEBUG:boto:Data: 2023-11-02T18:30:19.857 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.858 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.858 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.859 DEBUG:boto:Params: {} 2023-11-02T18:30:19.859 DEBUG:boto:Token: None 2023-11-02T18:30:19.860 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.860 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cfb40a732b82f5c-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.870 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:19.870 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:19.871 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.871 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:19.871 DEBUG:boto:Data: 2023-11-02T18:30:19.872 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.872 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.873 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.873 DEBUG:boto:Params: {} 2023-11-02T18:30:19.874 DEBUG:boto:Token: None 2023-11-02T18:30:19.874 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:19.875 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zNZ/QaZKBarHEIjwInnvgOwkfY8= 2023-11-02T18:30:19.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zNZ/QaZKBarHEIjwInnvgOwkfY8='} 2023-11-02T18:30:19.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b10162dadfff670e-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.888 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.889 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.889 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.890 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.890 DEBUG:boto:Data: 2023-11-02T18:30:19.891 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.891 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.892 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.892 DEBUG:boto:Params: {} 2023-11-02T18:30:19.893 DEBUG:boto:Token: None 2023-11-02T18:30:19.893 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.894 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.901 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5e8b7572b198d93-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.901 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:19.902 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:19.902 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.903 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:19.903 DEBUG:boto:Data: 2023-11-02T18:30:19.904 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.904 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.905 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.905 DEBUG:boto:Params: {} 2023-11-02T18:30:19.905 DEBUG:boto:Token: None 2023-11-02T18:30:19.906 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:19.906 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ULj3HvwFq0vmI4NMcMpdmgt9YRI= 2023-11-02T18:30:19.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ULj3HvwFq0vmI4NMcMpdmgt9YRI='} 2023-11-02T18:30:19.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a93ea355f9b70e0-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.951 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.952 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.952 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.953 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.953 DEBUG:boto:Data: 2023-11-02T18:30:19.954 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.954 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.955 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.955 DEBUG:boto:Params: {} 2023-11-02T18:30:19.956 DEBUG:boto:Token: None 2023-11-02T18:30:19.956 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.957 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.965 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041e5d30ea770b7e1-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.966 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:19.967 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:19.967 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.968 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:19.968 DEBUG:boto:Data: 2023-11-02T18:30:19.968 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:19.969 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.969 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.970 DEBUG:boto:Params: {} 2023-11-02T18:30:19.970 DEBUG:boto:Token: None 2023-11-02T18:30:19.971 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:19.971 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wx9P2jFWm4ngNymtKpFlrTYNeJ8= 2023-11-02T18:30:19.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wx9P2jFWm4ngNymtKpFlrTYNeJ8='} 2023-11-02T18:30:19.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033285448ecca6fdf-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.985 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:19.985 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:19.985 DEBUG:boto:Method: HEAD 2023-11-02T18:30:19.986 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:19.986 DEBUG:boto:Data: 2023-11-02T18:30:19.987 DEBUG:boto:Headers: {} 2023-11-02T18:30:19.987 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:19.988 DEBUG:boto:Port: 8000 2023-11-02T18:30:19.988 DEBUG:boto:Params: {} 2023-11-02T18:30:19.988 DEBUG:boto:Token: None 2023-11-02T18:30:19.989 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:19 GMT /ibtljn-58/ 2023-11-02T18:30:19.989 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw= 2023-11-02T18:30:19.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PDs/Is9puqpFpBv7BV+HhFYCAqw=', 'Content-Length': '0'} 2023-11-02T18:30:19.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a75e92ffc6a5b42-006543eabb-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:19.998 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:19.999 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:19.999 DEBUG:boto:Method: PUT 2023-11-02T18:30:19.999 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000039 2023-11-02T18:30:20.000 DEBUG:boto:Data: 2023-11-02T18:30:20.000 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.001 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.001 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.002 DEBUG:boto:Params: {} 2023-11-02T18:30:20.002 DEBUG:boto:Token: None 2023-11-02T18:30:20.003 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000039 2023-11-02T18:30:20.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sGoroBjAHm0AT28KFClEWKa9Rfc= 2023-11-02T18:30:20.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sGoroBjAHm0AT28KFClEWKa9Rfc='} 2023-11-02T18:30:20.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dcacb8882c66139-006543eabb-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.047 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.048 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.048 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.049 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.049 DEBUG:boto:Data: 2023-11-02T18:30:20.050 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.051 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.051 DEBUG:boto:Params: {} 2023-11-02T18:30:20.052 DEBUG:boto:Token: None 2023-11-02T18:30:20.052 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.053 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ac53ca66b0d7b35-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.062 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:20.062 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:20.063 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.063 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:20.064 DEBUG:boto:Data: 2023-11-02T18:30:20.065 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.065 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.065 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.066 DEBUG:boto:Params: {} 2023-11-02T18:30:20.066 DEBUG:boto:Token: None 2023-11-02T18:30:20.067 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:20.067 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9MTugUBprsZ+3aWj9YASKts22M4= 2023-11-02T18:30:20.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9MTugUBprsZ+3aWj9YASKts22M4='} 2023-11-02T18:30:20.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017ce51407537d836-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.081 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.081 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.082 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.082 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.083 DEBUG:boto:Data: 2023-11-02T18:30:20.083 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.084 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.084 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.084 DEBUG:boto:Params: {} 2023-11-02T18:30:20.085 DEBUG:boto:Token: None 2023-11-02T18:30:20.085 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.086 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fdd74d8e8d130d6-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.093 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:20.093 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:20.094 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.094 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000040 2023-11-02T18:30:20.095 DEBUG:boto:Data: 2023-11-02T18:30:20.095 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.096 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.096 DEBUG:boto:Params: {} 2023-11-02T18:30:20.097 DEBUG:boto:Token: None 2023-11-02T18:30:20.097 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000040 2023-11-02T18:30:20.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yjQ8Ut8e1a5Gc0RSe4+Pjdz51ps= 2023-11-02T18:30:20.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yjQ8Ut8e1a5Gc0RSe4+Pjdz51ps='} 2023-11-02T18:30:20.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe0043bc8ec1224f-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.143 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.144 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.144 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.144 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.145 DEBUG:boto:Data: 2023-11-02T18:30:20.145 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.146 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.146 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.147 DEBUG:boto:Params: {} 2023-11-02T18:30:20.147 DEBUG:boto:Token: None 2023-11-02T18:30:20.148 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.148 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.157 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f99e5b4a30815537-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.158 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:20.158 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:20.159 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.159 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000060 2023-11-02T18:30:20.159 DEBUG:boto:Data: 2023-11-02T18:30:20.160 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.160 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.161 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.161 DEBUG:boto:Params: {} 2023-11-02T18:30:20.162 DEBUG:boto:Token: None 2023-11-02T18:30:20.162 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000060 2023-11-02T18:30:20.163 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9FFpv43jEjcnlnbgI3IEAEQUTkg= 2023-11-02T18:30:20.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9FFpv43jEjcnlnbgI3IEAEQUTkg='} 2023-11-02T18:30:20.172 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003cd188fccdabd1d0-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.173 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.173 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.173 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.174 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.174 DEBUG:boto:Data: 2023-11-02T18:30:20.175 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.175 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.176 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.176 DEBUG:boto:Params: {} 2023-11-02T18:30:20.176 DEBUG:boto:Token: None 2023-11-02T18:30:20.177 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.185 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f855a5c19ca38b1-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.186 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:20.186 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:20.187 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.187 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000036 2023-11-02T18:30:20.188 DEBUG:boto:Data: 2023-11-02T18:30:20.188 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.188 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.189 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.189 DEBUG:boto:Params: {} 2023-11-02T18:30:20.190 DEBUG:boto:Token: None 2023-11-02T18:30:20.190 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000036 2023-11-02T18:30:20.191 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YRSwRUqYoahGjlXquQ7nhifO01M= 2023-11-02T18:30:20.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YRSwRUqYoahGjlXquQ7nhifO01M='} 2023-11-02T18:30:20.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061b26c320af288e8-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.239 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.240 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.240 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.240 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.241 DEBUG:boto:Data: 2023-11-02T18:30:20.241 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.242 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.242 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.243 DEBUG:boto:Params: {} 2023-11-02T18:30:20.243 DEBUG:boto:Token: None 2023-11-02T18:30:20.243 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.244 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cd0d9b8b3c8f9af-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.253 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:20.254 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:20.254 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.255 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000062 2023-11-02T18:30:20.255 DEBUG:boto:Data: 2023-11-02T18:30:20.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.256 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.256 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.257 DEBUG:boto:Params: {} 2023-11-02T18:30:20.257 DEBUG:boto:Token: None 2023-11-02T18:30:20.258 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000062 2023-11-02T18:30:20.258 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwz/FA2bCpPcHZo0iSq/9mKTvvg= 2023-11-02T18:30:20.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwz/FA2bCpPcHZo0iSq/9mKTvvg='} 2023-11-02T18:30:20.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cab77d167e22c4c-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.268 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.269 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.269 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.270 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.270 DEBUG:boto:Data: 2023-11-02T18:30:20.271 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.271 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.271 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.272 DEBUG:boto:Params: {} 2023-11-02T18:30:20.272 DEBUG:boto:Token: None 2023-11-02T18:30:20.273 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.273 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.281 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009512f6654b938e4b-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.282 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:20.282 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:20.282 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.283 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:20.283 DEBUG:boto:Data: 2023-11-02T18:30:20.284 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.284 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.285 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.285 DEBUG:boto:Params: {} 2023-11-02T18:30:20.285 DEBUG:boto:Token: None 2023-11-02T18:30:20.286 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:20.286 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q1n2rhAQE76c2ec4OF7GLQvCbMQ= 2023-11-02T18:30:20.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q1n2rhAQE76c2ec4OF7GLQvCbMQ='} 2023-11-02T18:30:20.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009815c4c119fc0a54-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.331 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.332 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.332 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.333 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.333 DEBUG:boto:Data: 2023-11-02T18:30:20.333 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.334 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.334 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.335 DEBUG:boto:Params: {} 2023-11-02T18:30:20.335 DEBUG:boto:Token: None 2023-11-02T18:30:20.336 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.336 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.345 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc15de2a0f058ad9-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.345 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:20.346 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:20.346 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.347 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:20.347 DEBUG:boto:Data: 2023-11-02T18:30:20.347 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.348 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.348 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.349 DEBUG:boto:Params: {} 2023-11-02T18:30:20.349 DEBUG:boto:Token: None 2023-11-02T18:30:20.350 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:20.350 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mKR9anhvfhtSQCReHC+9jbycf4o= 2023-11-02T18:30:20.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mKR9anhvfhtSQCReHC+9jbycf4o='} 2023-11-02T18:30:20.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000443396e21d229d1-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.362 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.363 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.363 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.364 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.364 DEBUG:boto:Data: 2023-11-02T18:30:20.364 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.365 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.365 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.366 DEBUG:boto:Params: {} 2023-11-02T18:30:20.366 DEBUG:boto:Token: None 2023-11-02T18:30:20.367 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.367 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecfc1dea043328dc-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.374 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:20.375 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:20.375 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.376 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:20.376 DEBUG:boto:Data: 2023-11-02T18:30:20.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.377 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.377 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.378 DEBUG:boto:Params: {} 2023-11-02T18:30:20.378 DEBUG:boto:Token: None 2023-11-02T18:30:20.379 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:20.379 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Wa4GJkOl4xjZgjgbXu94IxmFggQ= 2023-11-02T18:30:20.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Wa4GJkOl4xjZgjgbXu94IxmFggQ='} 2023-11-02T18:30:20.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d92bc1582a9ae49-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.423 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.423 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.424 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.424 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.425 DEBUG:boto:Data: 2023-11-02T18:30:20.425 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.426 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.426 DEBUG:boto:Params: {} 2023-11-02T18:30:20.427 DEBUG:boto:Token: None 2023-11-02T18:30:20.427 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046990dce3a4fb68e-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.437 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:20.438 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:20.438 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.439 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:20.439 DEBUG:boto:Data: 2023-11-02T18:30:20.439 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.440 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.440 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.441 DEBUG:boto:Params: {} 2023-11-02T18:30:20.441 DEBUG:boto:Token: None 2023-11-02T18:30:20.442 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:20.442 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iv2QqMWlHrh/K/qTtJyAyWIN5JA= 2023-11-02T18:30:20.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iv2QqMWlHrh/K/qTtJyAyWIN5JA='} 2023-11-02T18:30:20.457 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb5f5b247537da07-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.458 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.458 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.458 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.459 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.459 DEBUG:boto:Data: 2023-11-02T18:30:20.460 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.460 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.460 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.461 DEBUG:boto:Params: {} 2023-11-02T18:30:20.461 DEBUG:boto:Token: None 2023-11-02T18:30:20.462 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.462 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.469 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8de43684cc11296-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.470 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:20.470 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:20.471 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.471 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:20.472 DEBUG:boto:Data: 2023-11-02T18:30:20.472 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.472 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.473 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.473 DEBUG:boto:Params: {} 2023-11-02T18:30:20.474 DEBUG:boto:Token: None 2023-11-02T18:30:20.474 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:20.475 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m0E7kZl8OgOo3mqz/AY21YSWmac= 2023-11-02T18:30:20.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m0E7kZl8OgOo3mqz/AY21YSWmac='} 2023-11-02T18:30:20.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008246a97e512ac997-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.519 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.520 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.520 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.520 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.521 DEBUG:boto:Data: 2023-11-02T18:30:20.521 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.522 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.522 DEBUG:boto:Params: {} 2023-11-02T18:30:20.523 DEBUG:boto:Token: None 2023-11-02T18:30:20.523 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a930b916d6d2977-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.534 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:20.534 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:20.534 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.535 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:20.535 DEBUG:boto:Data: 2023-11-02T18:30:20.536 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.536 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.537 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.537 DEBUG:boto:Params: {} 2023-11-02T18:30:20.538 DEBUG:boto:Token: None 2023-11-02T18:30:20.538 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:20.539 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:97oE+GW7Kr4T6v6Osp4rYXrBUyQ= 2023-11-02T18:30:20.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:97oE+GW7Kr4T6v6Osp4rYXrBUyQ='} 2023-11-02T18:30:20.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036115f2084b3dba6-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.552 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.552 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.552 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.553 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.553 DEBUG:boto:Data: 2023-11-02T18:30:20.554 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.555 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.555 DEBUG:boto:Params: {} 2023-11-02T18:30:20.555 DEBUG:boto:Token: None 2023-11-02T18:30:20.556 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b157bfcf2a8fd9f-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.564 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:20.564 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:20.565 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.565 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000065 2023-11-02T18:30:20.566 DEBUG:boto:Data: 2023-11-02T18:30:20.566 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.567 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.567 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.567 DEBUG:boto:Params: {} 2023-11-02T18:30:20.568 DEBUG:boto:Token: None 2023-11-02T18:30:20.568 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000065 2023-11-02T18:30:20.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KJlI5g7Zy8eDE+GUAV6pyWtCwas= 2023-11-02T18:30:20.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KJlI5g7Zy8eDE+GUAV6pyWtCwas='} 2023-11-02T18:30:20.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e7f0796319c5110-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.615 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.615 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.616 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.616 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.617 DEBUG:boto:Data: 2023-11-02T18:30:20.617 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.617 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.618 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.618 DEBUG:boto:Params: {} 2023-11-02T18:30:20.619 DEBUG:boto:Token: None 2023-11-02T18:30:20.619 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.620 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001dcc484fa60e3757-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.628 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:20.628 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:20.629 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.629 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000045 2023-11-02T18:30:20.630 DEBUG:boto:Data: 2023-11-02T18:30:20.630 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.630 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.631 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.631 DEBUG:boto:Params: {} 2023-11-02T18:30:20.632 DEBUG:boto:Token: None 2023-11-02T18:30:20.632 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000045 2023-11-02T18:30:20.633 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X/Obv5JFY2v9K7vbPsrgamq74CQ= 2023-11-02T18:30:20.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X/Obv5JFY2v9K7vbPsrgamq74CQ='} 2023-11-02T18:30:20.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3236e3509a652f8-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.648 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.648 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.649 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.649 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.649 DEBUG:boto:Data: 2023-11-02T18:30:20.650 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.650 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.651 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.651 DEBUG:boto:Params: {} 2023-11-02T18:30:20.652 DEBUG:boto:Token: None 2023-11-02T18:30:20.652 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.652 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b476f5770a2609c-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.659 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:20.659 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:20.660 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.660 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:20.661 DEBUG:boto:Data: 2023-11-02T18:30:20.661 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.661 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.662 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.662 DEBUG:boto:Params: {} 2023-11-02T18:30:20.663 DEBUG:boto:Token: None 2023-11-02T18:30:20.663 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:20.664 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CRctzL/taOFecT4/iq6A2YFmpX0= 2023-11-02T18:30:20.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CRctzL/taOFecT4/iq6A2YFmpX0='} 2023-11-02T18:30:20.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046e86b586e1ea069-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.707 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.708 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.708 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.708 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.709 DEBUG:boto:Data: 2023-11-02T18:30:20.709 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.710 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.710 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.711 DEBUG:boto:Params: {} 2023-11-02T18:30:20.711 DEBUG:boto:Token: None 2023-11-02T18:30:20.711 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e775994715716313-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.721 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:20.721 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:20.722 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.722 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:20.723 DEBUG:boto:Data: 2023-11-02T18:30:20.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.724 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.724 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.724 DEBUG:boto:Params: {} 2023-11-02T18:30:20.725 DEBUG:boto:Token: None 2023-11-02T18:30:20.725 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:20.726 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZgD0X9Qm0hhucN8Y1LEt2TOnk9E= 2023-11-02T18:30:20.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZgD0X9Qm0hhucN8Y1LEt2TOnk9E='} 2023-11-02T18:30:20.737 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df0b7f16db5ef6ec-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.738 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:20.738 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:20.739 DEBUG:boto:Method: HEAD 2023-11-02T18:30:20.739 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:20.740 DEBUG:boto:Data: 2023-11-02T18:30:20.740 DEBUG:boto:Headers: {} 2023-11-02T18:30:20.741 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.741 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.742 DEBUG:boto:Params: {} 2023-11-02T18:30:20.742 DEBUG:boto:Token: None 2023-11-02T18:30:20.743 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/ 2023-11-02T18:30:20.743 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M= 2023-11-02T18:30:20.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qd2iENbfJofpp32kQc0qFZo5g4M=', 'Content-Length': '0'} 2023-11-02T18:30:20.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e18d4072fad333bc-006543eabc-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.751 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:20.751 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:20.752 DEBUG:boto:Method: PUT 2023-11-02T18:30:20.752 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:20.753 DEBUG:boto:Data: 2023-11-02T18:30:20.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:30:20.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:20.754 DEBUG:boto:Port: 8000 2023-11-02T18:30:20.755 DEBUG:boto:Params: {} 2023-11-02T18:30:20.755 DEBUG:boto:Token: None 2023-11-02T18:30:20.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:30:20 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:20.756 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FWlo2fxot6Wb2haZKR+RpDH88hE= 2023-11-02T18:30:20.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:30:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FWlo2fxot6Wb2haZKR+RpDH88hE='} 2023-11-02T18:30:20.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007cb0973fe87340a4-006543eabc-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:20.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:20.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:30:20.805 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-58 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:30:20.852 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:20.852 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:20.999 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:20.999 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.000 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket shard: 2 2023-11-02T18:30:21.001 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.001 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.001 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.002 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket shard: 2 2023-11-02T18:30:21.002 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: listing bilog for incremental sync; position=00000000001.39.6 2023-11-02T18:30:21.003 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket shard: 2 2023-11-02T18:30:21.003 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position=00000000001.36.6 2023-11-02T18:30:21.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket shard: 2 2023-11-02T18:30:21.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position=00000000001.39.6 2023-11-02T18:30:21.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket shard: 2 2023-11-02T18:30:21.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position=00000000001.27.6 2023-11-02T18:30:21.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket shard: 2 2023-11-02T18:30:21.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position=00000000001.43.6 2023-11-02T18:30:21.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:20.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket shard: 2 2023-11-02T18:30:21.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position=00000000001.54.6 2023-11-02T18:30:21.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket shard: 2 2023-11-02T18:30:21.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket shard: 2 2023-11-02T18:30:21.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position=00000000001.29.6 2023-11-02T18:30:21.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position=00000000001.41.6 2023-11-02T18:30:21.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket shard: 2 2023-11-02T18:30:21.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket shard: 2 2023-11-02T18:30:21.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position=00000000001.24.6 2023-11-02T18:30:21.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position=00000000001.37.6 2023-11-02T18:30:21.054 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.054 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.055 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket shard: 0 2023-11-02T18:30:21.055 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket shard: 0 2023-11-02T18:30:21.057 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.057 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket shard: 0 2023-11-02T18:30:21.057 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.057 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.058 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.060 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket shard: 0 2023-11-02T18:30:21.060 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.060 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.061 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.061 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.061 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: parsed entry: id=00000000001.32.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-11-02T18:30:21.061 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0/obj00000000000000000007 2023-11-02T18:30:21.062 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: entry->timestamp=2023-11-02T18:30:12.321047+0000 2023-11-02T18:30:21.062 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.32.6 op=0 op_state=1 2023-11-02T18:30:21.062 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000007]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000007[0] 2023-11-02T18:30:21.062 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: listing bilog for incremental sync; position=00000000001.32.6 2023-11-02T18:30:21.063 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.063 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: parsed entry: id=00000000001.34.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-11-02T18:30:21.063 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1/obj00000000000000000016 2023-11-02T18:30:21.063 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: entry->timestamp=2023-11-02T18:30:12.073519+0000 2023-11-02T18:30:21.063 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.34.6 op=0 op_state=1 2023-11-02T18:30:21.064 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.064 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000016]: : injecting data sync error on key=obj00000000000000000016 2023-11-02T18:30:21.064 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position=00000000001.34.6 2023-11-02T18:30:21.064 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket shard: 0 2023-11-02T18:30:21.065 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-11-02T18:30:21.065 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2/obj00000000000000000024 2023-11-02T18:30:21.065 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: entry->timestamp=2023-11-02T18:30:12.505851+0000 2023-11-02T18:30:21.065 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.51.6 op=0 op_state=1 2023-11-02T18:30:21.065 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.066 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000016]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.066 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000016]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1/obj00000000000000000016 2023-11-02T18:30:21.066 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.066 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000024]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000024[0] 2023-11-02T18:30:21.067 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position=00000000001.51.6 2023-11-02T18:30:21.067 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.067 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket shard: 0 2023-11-02T18:30:21.067 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket shard: 0 2023-11-02T18:30:21.070 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.070 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.071 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.071 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.071 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.071 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket shard: 0 2023-11-02T18:30:21.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket shard: 0 2023-11-02T18:30:21.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket shard: 0 2023-11-02T18:30:21.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: parsed entry: id=00000000001.28.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-11-02T18:30:21.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3/obj00000000000000000020 2023-11-02T18:30:21.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: entry->timestamp=2023-11-02T18:30:12.143356+0000 2023-11-02T18:30:21.074 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.052+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.28.6 op=0 op_state=1 2023-11-02T18:30:21.074 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.074 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:entry[obj00000000000000000020]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000020[0] 2023-11-02T18:30:21.074 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: listing bilog for incremental sync; position=00000000001.28.6 2023-11-02T18:30:21.075 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: sync status for source bucket shard: 0 2023-11-02T18:30:21.075 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.075 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket shard: 0 2023-11-02T18:30:21.075 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket shard: 0 2023-11-02T18:30:21.076 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket shard: 0 2023-11-02T18:30:21.076 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.076 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.076 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.076 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket shard: 0 2023-11-02T18:30:21.077 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket shard: 0 2023-11-02T18:30:21.077 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.077 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.077 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: sync status for source bucket shard: 0 2023-11-02T18:30:21.078 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.078 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: parsed entry: id=00000000001.82.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-11-02T18:30:21.078 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6/obj00000000000000000032 2023-11-02T18:30:21.078 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: entry->timestamp=2023-11-02T18:30:12.232224+0000 2023-11-02T18:30:21.079 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.82.6 op=0 op_state=1 2023-11-02T18:30:21.079 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000032[0] 2023-11-02T18:30:21.079 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position=00000000001.82.6 2023-11-02T18:30:21.079 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: a sync operation returned error 2023-11-02T18:30:21.080 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: backing out with sync_status=-5 2023-11-02T18:30:21.080 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.080 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.080 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.080 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-11-02T18:30:21.081 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4/obj00000000000000000028 2023-11-02T18:30:21.081 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: entry->timestamp=2023-11-02T18:30:12.533710+0000 2023-11-02T18:30:21.081 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-11-02T18:30:21.081 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-11-02T18:30:21.082 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10/obj00000000000000000013 2023-11-02T18:30:21.082 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: entry->timestamp=2023-11-02T18:30:12.439006+0000 2023-11-02T18:30:21.082 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-11-02T18:30:21.082 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000028]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000028[0] 2023-11-02T18:30:21.082 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position=00000000001.52.6 2023-11-02T18:30:21.083 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-11-02T18:30:21.083 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000026 2023-11-02T18:30:21.083 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: entry->timestamp=2023-11-02T18:30:12.040678+0000 2023-11-02T18:30:21.083 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.51.6 op=0 op_state=1 2023-11-02T18:30:21.084 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000013]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000013[0] 2023-11-02T18:30:21.084 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position=00000000002.63.6 2023-11-02T18:30:21.084 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000026]: : injecting data sync error on key=obj00000000000000000026 2023-11-02T18:30:21.084 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position=00000000001.51.6 2023-11-02T18:30:21.085 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-11-02T18:30:21.085 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11/obj00000000000000000021 2023-11-02T18:30:21.085 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: entry->timestamp=2023-11-02T18:30:12.409194+0000 2023-11-02T18:30:21.085 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.50.6 op=0 op_state=1 2023-11-02T18:30:21.085 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket shard: 0 2023-11-02T18:30:21.086 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000026]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.086 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000026]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000026 2023-11-02T18:30:21.086 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000021]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000021[0] 2023-11-02T18:30:21.086 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: listing bilog for incremental sync; position=00000000001.50.6 2023-11-02T18:30:21.087 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.095 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: parsed entry: id=00000000001.80.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2023-11-02T18:30:21.096 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16/obj00000000000000000076 2023-11-02T18:30:21.096 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: entry->timestamp=2023-11-02T18:30:12.172157+0000 2023-11-02T18:30:21.096 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.80.6 op=0 op_state=1 2023-11-02T18:30:21.096 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-11-02T18:30:21.097 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8/obj00000000000000000010 2023-11-02T18:30:21.097 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: entry->timestamp=2023-11-02T18:30:11.976612+0000 2023-11-02T18:30:21.097 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-11-02T18:30:21.097 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-11-02T18:30:21.098 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5/obj00000000000000000000 2023-11-02T18:30:21.098 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: entry->timestamp=2023-11-02T18:30:12.619862+0000 2023-11-02T18:30:21.098 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.32.6 op=0 op_state=1 2023-11-02T18:30:21.098 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-11-02T18:30:21.099 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15/obj00000000000000000004 2023-11-02T18:30:21.099 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: entry->timestamp=2023-11-02T18:30:12.260002+0000 2023-11-02T18:30:21.099 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-11-02T18:30:21.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000076]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000076[0] 2023-11-02T18:30:21.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position=00000000001.80.6 2023-11-02T18:30:21.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000010]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000010[0] 2023-11-02T18:30:21.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position=00000000002.30.6 2023-11-02T18:30:21.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000000]: : injecting data sync error on key=obj00000000000000000000 2023-11-02T18:30:21.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position=00000000002.32.6 2023-11-02T18:30:21.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000004]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000004[0] 2023-11-02T18:30:21.102 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: listing bilog for incremental sync; position=00000000002.34.6 2023-11-02T18:30:21.102 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-11-02T18:30:21.102 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14/obj00000000000000000033 2023-11-02T18:30:21.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: entry->timestamp=2023-11-02T18:30:11.949086+0000 2023-11-02T18:30:21.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-11-02T18:30:21.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000000]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000000]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5/obj00000000000000000000 2023-11-02T18:30:21.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000033]: : injecting data sync error on key=obj00000000000000000033 2023-11-02T18:30:21.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position=00000000001.49.6 2023-11-02T18:30:21.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000033]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.064+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000033]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14/obj00000000000000000033 2023-11-02T18:30:21.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-11-02T18:30:21.106 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9/obj00000000000000000019 2023-11-02T18:30:21.106 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: entry->timestamp=2023-11-02T18:30:12.348296+0000 2023-11-02T18:30:21.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-11-02T18:30:21.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000019]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000019[0] 2023-11-02T18:30:21.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position=00000000001.49.6 2023-11-02T18:30:21.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-11-02T18:30:21.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7/obj00000000000000000001 2023-11-02T18:30:21.108 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: entry->timestamp=2023-11-02T18:30:12.595653+0000 2023-11-02T18:30:21.108 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-11-02T18:30:21.108 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000001]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000001[0] 2023-11-02T18:30:21.108 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position=00000000002.53.6 2023-11-02T18:30:21.109 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: parsed entry: id=00000000001.34.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-11-02T18:30:21.109 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1/obj00000000000000000016 2023-11-02T18:30:21.109 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.072+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: entry->timestamp=2023-11-02T18:30:12.073519+0000 2023-11-02T18:30:21.110 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.072+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.34.6 op=0 op_state=1 2023-11-02T18:30:21.110 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.072+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000016]: : injecting data sync error on key=obj00000000000000000016 2023-11-02T18:30:21.110 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.072+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position=00000000001.34.6 2023-11-02T18:30:21.110 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.072+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000016]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.110 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.072+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000016]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1/obj00000000000000000016 2023-11-02T18:30:21.111 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.072+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: a sync operation returned error 2023-11-02T18:30:21.111 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.072+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: backing out with sync_status=-5 2023-11-02T18:30:21.111 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.072+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.111 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.072+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.112 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.072+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: a sync operation returned error 2023-11-02T18:30:21.112 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.076+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: backing out with sync_status=-5 2023-11-02T18:30:21.112 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.076+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.112 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.112 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.076+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000007]: success 2023-11-02T18:30:21.113 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0:1 marker=00000000001.32.6 timestamp=2023-11-02T18:30:12.321047+0000 2023-11-02T18:30:21.113 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.076+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: a sync operation returned error 2023-11-02T18:30:21.113 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.076+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: backing out with sync_status=-5 2023-11-02T18:30:21.113 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.076+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.113 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.114 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.076+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000024]: success 2023-11-02T18:30:21.114 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2:1 marker=00000000001.51.6 timestamp=2023-11-02T18:30:12.505851+0000 2023-11-02T18:30:21.114 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.114 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.115 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.115 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket shard: 0 2023-11-02T18:30:21.115 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.115 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket shard: 0 2023-11-02T18:30:21.116 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.116 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket shard: 0 2023-11-02T18:30:21.116 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.116 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000076]: success 2023-11-02T18:30:21.116 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16:1 marker=00000000001.80.6 timestamp=2023-11-02T18:30:12.172157+0000 2023-11-02T18:30:21.117 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: parsed entry: id=00000000001.34.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-11-02T18:30:21.117 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1/obj00000000000000000016 2023-11-02T18:30:21.117 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: entry->timestamp=2023-11-02T18:30:12.073519+0000 2023-11-02T18:30:21.117 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.34.6 op=0 op_state=1 2023-11-02T18:30:21.117 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-11-02T18:30:21.118 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14/obj00000000000000000033 2023-11-02T18:30:21.118 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: entry->timestamp=2023-11-02T18:30:11.949086+0000 2023-11-02T18:30:21.118 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-11-02T18:30:21.118 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000016]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000016[0] 2023-11-02T18:30:21.119 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position=00000000001.34.6 2023-11-02T18:30:21.119 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000032]: success 2023-11-02T18:30:21.119 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6:1 marker=00000000001.82.6 timestamp=2023-11-02T18:30:12.232224+0000 2023-11-02T18:30:21.119 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-11-02T18:30:21.120 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5/obj00000000000000000000 2023-11-02T18:30:21.120 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: entry->timestamp=2023-11-02T18:30:12.619862+0000 2023-11-02T18:30:21.120 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.32.6 op=0 op_state=1 2023-11-02T18:30:21.120 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000033]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000033[0] 2023-11-02T18:30:21.120 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position=00000000001.49.6 2023-11-02T18:30:21.121 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000000]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000000[0] 2023-11-02T18:30:21.121 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.084+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position=00000000002.32.6 2023-11-02T18:30:21.121 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.088+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:entry[obj00000000000000000020]: success 2023-11-02T18:30:21.121 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.088+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3:1 marker=00000000001.28.6 timestamp=2023-11-02T18:30:12.143356+0000 2023-11-02T18:30:21.124 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.088+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000021]: success 2023-11-02T18:30:21.124 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.088+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11:1 marker=00000000001.50.6 timestamp=2023-11-02T18:30:12.409194+0000 2023-11-02T18:30:21.124 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.092+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000028]: success 2023-11-02T18:30:21.124 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.092+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4:1 marker=00000000001.52.6 timestamp=2023-11-02T18:30:12.533710+0000 2023-11-02T18:30:21.125 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.092+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000010]: success 2023-11-02T18:30:21.125 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.092+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8:1 marker=00000000002.30.6 timestamp=2023-11-02T18:30:11.976612+0000 2023-11-02T18:30:21.125 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.096+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000013]: success 2023-11-02T18:30:21.125 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.096+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10:1 marker=00000000002.63.6 timestamp=2023-11-02T18:30:12.439006+0000 2023-11-02T18:30:21.126 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.096+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000004]: success 2023-11-02T18:30:21.126 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.096+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15:1 marker=00000000002.34.6 timestamp=2023-11-02T18:30:12.260002+0000 2023-11-02T18:30:21.126 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.096+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000000]: success 2023-11-02T18:30:21.126 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.096+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5:1 marker=00000000002.32.6 timestamp=2023-11-02T18:30:12.619862+0000 2023-11-02T18:30:21.127 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.100+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000033]: success 2023-11-02T18:30:21.127 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.100+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14:1 marker=00000000001.49.6 timestamp=2023-11-02T18:30:11.949086+0000 2023-11-02T18:30:21.127 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.100+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000016]: success 2023-11-02T18:30:21.127 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.100+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1:1 marker=00000000001.34.6 timestamp=2023-11-02T18:30:12.073519+0000 2023-11-02T18:30:21.128 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.108+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: a sync operation returned error 2023-11-02T18:30:21.128 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.108+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: backing out with sync_status=-5 2023-11-02T18:30:21.128 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.108+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.128 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.108+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.128 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.108+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.129 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.112+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket shard: 0 2023-11-02T18:30:21.129 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.112+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.129 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.116+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-11-02T18:30:21.129 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.116+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000026 2023-11-02T18:30:21.130 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.116+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: entry->timestamp=2023-11-02T18:30:12.040678+0000 2023-11-02T18:30:21.130 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.116+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.51.6 op=0 op_state=1 2023-11-02T18:30:21.130 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.116+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000026]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000026[0] 2023-11-02T18:30:21.130 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.116+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position=00000000001.51.6 2023-11-02T18:30:21.131 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.120+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000026]: success 2023-11-02T18:30:21.131 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.120+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13:1 marker=00000000001.51.6 timestamp=2023-11-02T18:30:12.040678+0000 2023-11-02T18:30:21.134 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.128+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000001]: success 2023-11-02T18:30:21.135 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.128+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7:1 marker=00000000002.53.6 timestamp=2023-11-02T18:30:12.595653+0000 2023-11-02T18:30:21.135 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.128+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000019]: success 2023-11-02T18:30:21.136 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.128+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9:1 marker=00000000001.49.6 timestamp=2023-11-02T18:30:12.348296+0000 2023-11-02T18:30:21.156 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.157 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.157 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.157 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket shard: 0 2023-11-02T18:30:21.157 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.158 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.158 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket shard: 0 2023-11-02T18:30:21.158 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket shard: 0 2023-11-02T18:30:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.161 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket shard: 0 2023-11-02T18:30:21.161 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.148+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.161 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-11-02T18:30:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1/obj00000000000000000010 2023-11-02T18:30:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: entry->timestamp=2023-11-02T18:30:13.077825+0000 2023-11-02T18:30:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-11-02T18:30:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000010]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000010[0] 2023-11-02T18:30:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position=00000000002.57.6 2023-11-02T18:30:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-11-02T18:30:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2/obj00000000000000000016 2023-11-02T18:30:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: entry->timestamp=2023-11-02T18:30:13.474735+0000 2023-11-02T18:30:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-11-02T18:30:21.165 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000016]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000016[0] 2023-11-02T18:30:21.165 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position=00000000002.62.6 2023-11-02T18:30:21.165 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.165 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket shard: 0 2023-11-02T18:30:21.165 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.166 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.166 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-11-02T18:30:21.166 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5/obj00000000000000000025 2023-11-02T18:30:21.166 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: entry->timestamp=2023-11-02T18:30:13.608296+0000 2023-11-02T18:30:21.167 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-11-02T18:30:21.167 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.167 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000025]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000025[0] 2023-11-02T18:30:21.167 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position=00000000001.43.6 2023-11-02T18:30:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket shard: 0 2023-11-02T18:30:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket shard: 0 2023-11-02T18:30:21.174 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.152+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.174 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket shard: 0 2023-11-02T18:30:21.174 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.174 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000016]: success 2023-11-02T18:30:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2:2 marker=00000000002.62.6 timestamp=2023-11-02T18:30:13.474735+0000 2023-11-02T18:30:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket shard: 0 2023-11-02T18:30:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-11-02T18:30:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4/obj00000000000000000004 2023-11-02T18:30:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: entry->timestamp=2023-11-02T18:30:13.806055+0000 2023-11-02T18:30:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-11-02T18:30:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket shard: 0 2023-11-02T18:30:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket shard: 0 2023-11-02T18:30:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: sync status for source bucket shard: 0 2023-11-02T18:30:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket shard: 0 2023-11-02T18:30:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket shard: 0 2023-11-02T18:30:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket shard: 0 2023-11-02T18:30:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-11-02T18:30:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6/obj00000000000000000030 2023-11-02T18:30:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: entry->timestamp=2023-11-02T18:30:13.210508+0000 2023-11-02T18:30:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-11-02T18:30:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000004]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000004[0] 2023-11-02T18:30:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position=00000000002.68.6 2023-11-02T18:30:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: sync status for source bucket shard: 0 2023-11-02T18:30:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000030]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000030[0] 2023-11-02T18:30:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position=00000000002.54.6 2023-11-02T18:30:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-11-02T18:30:21.184 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7/obj00000000000000000015 2023-11-02T18:30:21.184 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: entry->timestamp=2023-11-02T18:30:13.508769+0000 2023-11-02T18:30:21.184 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.156+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-11-02T18:30:21.184 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.160+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000015]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000015[0] 2023-11-02T18:30:21.185 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.160+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position=00000000002.39.6 2023-11-02T18:30:21.185 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: sync status for source bucket shard: 0 2023-11-02T18:30:21.185 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket shard: 0 2023-11-02T18:30:21.186 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: sync status for source bucket shard: 0 2023-11-02T18:30:21.186 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.186 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.187 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.187 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-11-02T18:30:21.187 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8/obj00000000000000000019 2023-11-02T18:30:21.187 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: entry->timestamp=2023-11-02T18:30:13.569400+0000 2023-11-02T18:30:21.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.67.6 op=0 op_state=1 2023-11-02T18:30:21.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000010]: success 2023-11-02T18:30:21.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1:2 marker=00000000002.57.6 timestamp=2023-11-02T18:30:13.077825+0000 2023-11-02T18:30:21.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-11-02T18:30:21.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9/obj00000000000000000003 2023-11-02T18:30:21.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: entry->timestamp=2023-11-02T18:30:13.271872+0000 2023-11-02T18:30:21.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-11-02T18:30:21.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: parsed entry: id=00000000002.23.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-11-02T18:30:21.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0/obj00000000000000000005 2023-11-02T18:30:21.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: entry->timestamp=2023-11-02T18:30:13.409302+0000 2023-11-02T18:30:21.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.23.6 op=0 op_state=1 2023-11-02T18:30:21.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000025]: success 2023-11-02T18:30:21.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5:2 marker=00000000001.43.6 timestamp=2023-11-02T18:30:13.608296+0000 2023-11-02T18:30:21.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-11-02T18:30:21.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12/obj00000000000000000012 2023-11-02T18:30:21.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: entry->timestamp=2023-11-02T18:30:13.376220+0000 2023-11-02T18:30:21.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-11-02T18:30:21.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-11-02T18:30:21.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000002 2023-11-02T18:30:21.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: entry->timestamp=2023-11-02T18:30:13.702716+0000 2023-11-02T18:30:21.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-11-02T18:30:21.193 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000004]: success 2023-11-02T18:30:21.193 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4:2 marker=00000000002.68.6 timestamp=2023-11-02T18:30:13.806055+0000 2023-11-02T18:30:21.193 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-11-02T18:30:21.193 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14/obj00000000000000000000 2023-11-02T18:30:21.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: entry->timestamp=2023-11-02T18:30:13.670256+0000 2023-11-02T18:30:21.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-11-02T18:30:21.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000019]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000019[0] 2023-11-02T18:30:21.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position=00000000002.67.6 2023-11-02T18:30:21.195 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000003]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000003[0] 2023-11-02T18:30:21.195 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position=00000000002.39.6 2023-11-02T18:30:21.195 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000005]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000005[0] 2023-11-02T18:30:21.195 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: listing bilog for incremental sync; position=00000000002.23.6 2023-11-02T18:30:21.195 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000012]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000012[0] 2023-11-02T18:30:21.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position=00000000002.47.6 2023-11-02T18:30:21.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000002]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000002[0] 2023-11-02T18:30:21.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position=00000000002.43.6 2023-11-02T18:30:21.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000000]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000000[0] 2023-11-02T18:30:21.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.164+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position=00000000002.49.6 2023-11-02T18:30:21.199 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-11-02T18:30:21.199 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15/obj00000000000000000006 2023-11-02T18:30:21.199 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: entry->timestamp=2023-11-02T18:30:13.174589+0000 2023-11-02T18:30:21.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.65.6 op=0 op_state=1 2023-11-02T18:30:21.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-11-02T18:30:21.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16/obj00000000000000000011 2023-11-02T18:30:21.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: entry->timestamp=2023-11-02T18:30:13.897710+0000 2023-11-02T18:30:21.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-11-02T18:30:21.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-11-02T18:30:21.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18/obj00000000000000000007 2023-11-02T18:30:21.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: entry->timestamp=2023-11-02T18:30:13.305373+0000 2023-11-02T18:30:21.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-11-02T18:30:21.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-11-02T18:30:21.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17/obj00000000000000000001 2023-11-02T18:30:21.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: entry->timestamp=2023-11-02T18:30:13.767015+0000 2023-11-02T18:30:21.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-11-02T18:30:21.203 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000006]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000006[0] 2023-11-02T18:30:21.203 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: listing bilog for incremental sync; position=00000000002.65.6 2023-11-02T18:30:21.203 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000011]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000011[0] 2023-11-02T18:30:21.203 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position=00000000002.64.6 2023-11-02T18:30:21.203 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000007]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000007[0] 2023-11-02T18:30:21.204 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: listing bilog for incremental sync; position=00000000002.62.6 2023-11-02T18:30:21.204 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000019]: success 2023-11-02T18:30:21.204 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8:2 marker=00000000002.67.6 timestamp=2023-11-02T18:30:13.569400+0000 2023-11-02T18:30:21.204 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000030]: success 2023-11-02T18:30:21.205 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6:2 marker=00000000002.54.6 timestamp=2023-11-02T18:30:13.210508+0000 2023-11-02T18:30:21.205 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000000]: success 2023-11-02T18:30:21.205 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14:2 marker=00000000002.49.6 timestamp=2023-11-02T18:30:13.670256+0000 2023-11-02T18:30:21.205 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000001[0] 2023-11-02T18:30:21.205 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.172+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: listing bilog for incremental sync; position=00000000002.37.6 2023-11-02T18:30:21.206 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.176+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000001]: success 2023-11-02T18:30:21.206 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.176+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17:2 marker=00000000002.37.6 timestamp=2023-11-02T18:30:13.767015+0000 2023-11-02T18:30:21.206 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.176+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000002]: success 2023-11-02T18:30:21.206 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.176+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13:2 marker=00000000002.43.6 timestamp=2023-11-02T18:30:13.702716+0000 2023-11-02T18:30:21.207 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.180+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000007]: success 2023-11-02T18:30:21.207 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.180+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18:2 marker=00000000002.62.6 timestamp=2023-11-02T18:30:13.305373+0000 2023-11-02T18:30:21.207 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.180+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000012]: success 2023-11-02T18:30:21.207 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.180+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12:2 marker=00000000002.47.6 timestamp=2023-11-02T18:30:13.376220+0000 2023-11-02T18:30:21.207 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.184+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000005]: success 2023-11-02T18:30:21.208 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.184+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0:2 marker=00000000002.23.6 timestamp=2023-11-02T18:30:13.409302+0000 2023-11-02T18:30:21.208 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.184+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000006]: success 2023-11-02T18:30:21.208 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.184+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15:2 marker=00000000002.65.6 timestamp=2023-11-02T18:30:13.174589+0000 2023-11-02T18:30:21.209 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.200+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-11-02T18:30:21.209 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.200+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10/obj00000000000000000009 2023-11-02T18:30:21.209 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.200+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: entry->timestamp=2023-11-02T18:30:13.111250+0000 2023-11-02T18:30:21.210 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.200+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.59.6 op=0 op_state=1 2023-11-02T18:30:21.210 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.200+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000009]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000009[0] 2023-11-02T18:30:21.210 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.200+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position=00000000001.59.6 2023-11-02T18:30:21.211 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.204+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: parsed entry: id=00000000001.32.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-11-02T18:30:21.211 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.204+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11/obj00000000000000000014 2023-11-02T18:30:21.211 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.204+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: entry->timestamp=2023-11-02T18:30:13.869810+0000 2023-11-02T18:30:21.211 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.204+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.32.6 op=0 op_state=1 2023-11-02T18:30:21.212 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.204+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000014]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000014[0] 2023-11-02T18:30:21.212 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.204+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: listing bilog for incremental sync; position=00000000001.32.6 2023-11-02T18:30:21.216 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.208+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000014]: success 2023-11-02T18:30:21.216 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.208+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11:2 marker=00000000001.32.6 timestamp=2023-11-02T18:30:13.869810+0000 2023-11-02T18:30:21.225 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.216+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000015]: success 2023-11-02T18:30:21.225 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.216+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7:2 marker=00000000002.39.6 timestamp=2023-11-02T18:30:13.508769+0000 2023-11-02T18:30:21.225 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.220+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000003]: success 2023-11-02T18:30:21.226 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.220+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9:2 marker=00000000002.39.6 timestamp=2023-11-02T18:30:13.271872+0000 2023-11-02T18:30:21.230 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.224+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000011]: success 2023-11-02T18:30:21.230 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.224+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16:2 marker=00000000002.64.6 timestamp=2023-11-02T18:30:13.897710+0000 2023-11-02T18:30:21.257 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.248+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000009]: success 2023-11-02T18:30:21.257 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.248+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10:2 marker=00000000001.59.6 timestamp=2023-11-02T18:30:13.111250+0000 2023-11-02T18:30:21.266 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.260+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.267 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.260+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.268 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.260+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket shard: 0 2023-11-02T18:30:21.268 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.260+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.268 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.260+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.268 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.260+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket shard: 0 2023-11-02T18:30:21.269 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.260+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.269 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.260+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket shard: 0 2023-11-02T18:30:21.269 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.260+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.272 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.260+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.272 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.260+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.272 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.260+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket shard: 0 2023-11-02T18:30:21.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket shard: 0 2023-11-02T18:30:21.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.274 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket shard: 0 2023-11-02T18:30:21.274 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.274 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.274 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-11-02T18:30:21.275 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1/obj00000000000000000026 2023-11-02T18:30:21.275 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: entry->timestamp=2023-11-02T18:30:15.035914+0000 2023-11-02T18:30:21.275 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-11-02T18:30:21.275 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000026]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000026[0] 2023-11-02T18:30:21.276 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position=00000000002.48.6 2023-11-02T18:30:21.276 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.276 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.276 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-11-02T18:30:21.277 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2/obj00000000000000000011 2023-11-02T18:30:21.277 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: entry->timestamp=2023-11-02T18:30:15.097350+0000 2023-11-02T18:30:21.277 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-11-02T18:30:21.277 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.278 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.278 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: parsed entry: id=00000000001.78.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2023-11-02T18:30:21.278 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0/obj00000000000000000058 2023-11-02T18:30:21.278 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: entry->timestamp=2023-11-02T18:30:15.190374+0000 2023-11-02T18:30:21.279 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.78.6 op=0 op_state=1 2023-11-02T18:30:21.282 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000011[0] 2023-11-02T18:30:21.282 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position=00000000002.46.6 2023-11-02T18:30:21.283 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000058]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000058[0] 2023-11-02T18:30:21.283 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: listing bilog for incremental sync; position=00000000001.78.6 2023-11-02T18:30:21.283 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.283 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket shard: 0 2023-11-02T18:30:21.283 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.284 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.264+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.284 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket shard: 0 2023-11-02T18:30:21.284 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket shard: 0 2023-11-02T18:30:21.284 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket shard: 0 2023-11-02T18:30:21.285 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.285 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.285 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.285 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-11-02T18:30:21.286 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5/obj00000000000000000050 2023-11-02T18:30:21.286 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: entry->timestamp=2023-11-02T18:30:15.128584+0000 2023-11-02T18:30:21.286 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-11-02T18:30:21.286 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket shard: 0 2023-11-02T18:30:21.286 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: sync status for source bucket shard: 0 2023-11-02T18:30:21.287 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000050]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000050[0] 2023-11-02T18:30:21.287 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position=00000000001.44.6 2023-11-02T18:30:21.287 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.287 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket shard: 0 2023-11-02T18:30:21.288 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.268+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.288 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.288 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.288 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.289 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.289 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.289 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.289 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.290 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-11-02T18:30:21.290 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6/obj00000000000000000032 2023-11-02T18:30:21.290 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: entry->timestamp=2023-11-02T18:30:14.752953+0000 2023-11-02T18:30:21.290 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-11-02T18:30:21.290 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.291 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-11-02T18:30:21.291 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4/obj00000000000000000014 2023-11-02T18:30:21.291 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: entry->timestamp=2023-11-02T18:30:14.937919+0000 2023-11-02T18:30:21.291 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-11-02T18:30:21.292 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000026]: success 2023-11-02T18:30:21.293 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1:3 marker=00000000002.48.6 timestamp=2023-11-02T18:30:15.035914+0000 2023-11-02T18:30:21.293 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000011]: success 2023-11-02T18:30:21.294 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2:3 marker=00000000002.46.6 timestamp=2023-11-02T18:30:15.097350+0000 2023-11-02T18:30:21.294 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000032[0] 2023-11-02T18:30:21.294 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position=00000000002.68.6 2023-11-02T18:30:21.294 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000014]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000014[0] 2023-11-02T18:30:21.294 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position=00000000002.44.6 2023-11-02T18:30:21.295 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.295 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.272+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.295 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.302 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: sync status for source bucket shard: 0 2023-11-02T18:30:21.302 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket shard: 0 2023-11-02T18:30:21.302 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket shard: 0 2023-11-02T18:30:21.302 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket shard: 0 2023-11-02T18:30:21.303 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-11-02T18:30:21.303 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7/obj00000000000000000002 2023-11-02T18:30:21.303 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: entry->timestamp=2023-11-02T18:30:14.441757+0000 2023-11-02T18:30:21.303 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.76.6 op=0 op_state=1 2023-11-02T18:30:21.303 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-11-02T18:30:21.304 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10/obj00000000000000000006 2023-11-02T18:30:21.304 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: entry->timestamp=2023-11-02T18:30:15.281726+0000 2023-11-02T18:30:21.304 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.73.6 op=0 op_state=1 2023-11-02T18:30:21.304 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: sync status for source bucket shard: 0 2023-11-02T18:30:21.305 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: sync status for source bucket shard: 0 2023-11-02T18:30:21.305 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: sync status for source bucket shard: 0 2023-11-02T18:30:21.305 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.305 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.305 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.306 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.306 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000002]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000002[0] 2023-11-02T18:30:21.306 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.276+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position=00000000002.76.6 2023-11-02T18:30:21.306 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000006]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000006[0] 2023-11-02T18:30:21.307 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position=00000000002.73.6 2023-11-02T18:30:21.307 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000058]: success 2023-11-02T18:30:21.307 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0:3 marker=00000000001.78.6 timestamp=2023-11-02T18:30:15.190374+0000 2023-11-02T18:30:21.307 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.308 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.308 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.308 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-11-02T18:30:21.308 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11/obj00000000000000000040 2023-11-02T18:30:21.308 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: entry->timestamp=2023-11-02T18:30:14.909280+0000 2023-11-02T18:30:21.309 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.40.6 op=0 op_state=1 2023-11-02T18:30:21.309 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-11-02T18:30:21.309 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12/obj00000000000000000009 2023-11-02T18:30:21.309 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: entry->timestamp=2023-11-02T18:30:14.845142+0000 2023-11-02T18:30:21.309 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-11-02T18:30:21.310 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: sync status for source bucket shard: 0 2023-11-02T18:30:21.310 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000040]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000040[0] 2023-11-02T18:30:21.310 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: listing bilog for incremental sync; position=00000000001.40.6 2023-11-02T18:30:21.310 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000009]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000009[0] 2023-11-02T18:30:21.311 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position=00000000002.42.6 2023-11-02T18:30:21.311 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000032]: success 2023-11-02T18:30:21.311 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6:3 marker=00000000002.68.6 timestamp=2023-11-02T18:30:14.752953+0000 2023-11-02T18:30:21.311 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: sync status for source bucket shard: 0 2023-11-02T18:30:21.311 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.312 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.312 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000014]: success 2023-11-02T18:30:21.312 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4:3 marker=00000000002.44.6 timestamp=2023-11-02T18:30:14.937919+0000 2023-11-02T18:30:21.312 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.280+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: sync status for source bucket shard: 0 2023-11-02T18:30:21.313 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.313 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-11-02T18:30:21.313 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15/obj00000000000000000012 2023-11-02T18:30:21.313 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: entry->timestamp=2023-11-02T18:30:14.813307+0000 2023-11-02T18:30:21.314 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-11-02T18:30:21.314 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-11-02T18:30:21.314 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000045 2023-11-02T18:30:21.314 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: entry->timestamp=2023-11-02T18:30:14.660204+0000 2023-11-02T18:30:21.315 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-11-02T18:30:21.315 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000012]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000012[0] 2023-11-02T18:30:21.315 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: listing bilog for incremental sync; position=00000000002.35.6 2023-11-02T18:30:21.315 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000050]: success 2023-11-02T18:30:21.315 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5:3 marker=00000000001.44.6 timestamp=2023-11-02T18:30:15.128584+0000 2023-11-02T18:30:21.316 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000045]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000045[0] 2023-11-02T18:30:21.316 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position=00000000001.43.6 2023-11-02T18:30:21.316 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-11-02T18:30:21.316 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14/obj00000000000000000013 2023-11-02T18:30:21.317 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: entry->timestamp=2023-11-02T18:30:14.721764+0000 2023-11-02T18:30:21.317 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-11-02T18:30:21.317 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-11-02T18:30:21.317 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16/obj00000000000000000033 2023-11-02T18:30:21.317 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: entry->timestamp=2023-11-02T18:30:14.538548+0000 2023-11-02T18:30:21.318 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-11-02T18:30:21.318 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-11-02T18:30:21.318 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19/obj00000000000000000021 2023-11-02T18:30:21.318 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: entry->timestamp=2023-11-02T18:30:14.377944+0000 2023-11-02T18:30:21.319 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-11-02T18:30:21.319 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-11-02T18:30:21.319 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18/obj00000000000000000000 2023-11-02T18:30:21.319 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: entry->timestamp=2023-11-02T18:30:14.633677+0000 2023-11-02T18:30:21.319 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-11-02T18:30:21.320 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000002]: success 2023-11-02T18:30:21.320 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7:3 marker=00000000002.76.6 timestamp=2023-11-02T18:30:14.441757+0000 2023-11-02T18:30:21.320 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000006]: success 2023-11-02T18:30:21.320 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.284+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10:3 marker=00000000002.73.6 timestamp=2023-11-02T18:30:15.281726+0000 2023-11-02T18:30:21.321 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000013]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000013[0] 2023-11-02T18:30:21.321 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position=00000000002.59.6 2023-11-02T18:30:21.321 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000033]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000033[0] 2023-11-02T18:30:21.321 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position=00000000002.37.6 2023-11-02T18:30:21.322 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000021]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000021[0] 2023-11-02T18:30:21.322 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: listing bilog for incremental sync; position=00000000002.38.6 2023-11-02T18:30:21.322 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000000]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000000[0] 2023-11-02T18:30:21.322 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: listing bilog for incremental sync; position=00000000002.34.6 2023-11-02T18:30:21.323 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2023-11-02T18:30:21.323 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21/obj00000000000000000055 2023-11-02T18:30:21.323 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: entry->timestamp=2023-11-02T18:30:14.473797+0000 2023-11-02T18:30:21.323 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.66.6 op=0 op_state=1 2023-11-02T18:30:21.323 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:entry[obj00000000000000000055]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000055[0] 2023-11-02T18:30:21.324 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: listing bilog for incremental sync; position=00000000001.66.6 2023-11-02T18:30:21.324 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000009]: success 2023-11-02T18:30:21.324 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12:3 marker=00000000002.42.6 timestamp=2023-11-02T18:30:14.845142+0000 2023-11-02T18:30:21.324 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: parsed entry: id=00000000002.28.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-11-02T18:30:21.325 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22/obj00000000000000000008 2023-11-02T18:30:21.325 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: entry->timestamp=2023-11-02T18:30:14.571134+0000 2023-11-02T18:30:21.325 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.28.6 op=0 op_state=1 2023-11-02T18:30:21.325 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-11-02T18:30:21.325 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17/obj00000000000000000001 2023-11-02T18:30:21.326 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: entry->timestamp=2023-11-02T18:30:15.218195+0000 2023-11-02T18:30:21.326 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-11-02T18:30:21.326 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-11-02T18:30:21.326 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20/obj00000000000000000003 2023-11-02T18:30:21.327 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: entry->timestamp=2023-11-02T18:30:15.000824+0000 2023-11-02T18:30:21.327 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.288+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-11-02T18:30:21.327 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.292+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000008]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000008[0] 2023-11-02T18:30:21.327 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.292+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: listing bilog for incremental sync; position=00000000002.28.6 2023-11-02T18:30:21.327 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.292+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000001[0] 2023-11-02T18:30:21.328 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.292+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: listing bilog for incremental sync; position=00000000002.37.6 2023-11-02T18:30:21.328 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.292+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:entry[obj00000000000000000003]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000003[0] 2023-11-02T18:30:21.328 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.292+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: listing bilog for incremental sync; position=00000000002.36.6 2023-11-02T18:30:21.330 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000012]: success 2023-11-02T18:30:21.330 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15:3 marker=00000000002.35.6 timestamp=2023-11-02T18:30:14.813307+0000 2023-11-02T18:30:21.330 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000013]: success 2023-11-02T18:30:21.331 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14:3 marker=00000000002.59.6 timestamp=2023-11-02T18:30:14.721764+0000 2023-11-02T18:30:21.331 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000021]: success 2023-11-02T18:30:21.331 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19:3 marker=00000000002.38.6 timestamp=2023-11-02T18:30:14.377944+0000 2023-11-02T18:30:21.331 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000033]: success 2023-11-02T18:30:21.332 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16:3 marker=00000000002.37.6 timestamp=2023-11-02T18:30:14.538548+0000 2023-11-02T18:30:21.332 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000000]: success 2023-11-02T18:30:21.332 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18:3 marker=00000000002.34.6 timestamp=2023-11-02T18:30:14.633677+0000 2023-11-02T18:30:21.332 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:entry[obj00000000000000000055]: success 2023-11-02T18:30:21.333 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21:3 marker=00000000001.66.6 timestamp=2023-11-02T18:30:14.473797+0000 2023-11-02T18:30:21.333 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000045]: success 2023-11-02T18:30:21.333 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.296+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13:3 marker=00000000001.43.6 timestamp=2023-11-02T18:30:14.660204+0000 2023-11-02T18:30:21.333 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.300+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000001]: success 2023-11-02T18:30:21.334 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.300+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17:3 marker=00000000002.37.6 timestamp=2023-11-02T18:30:15.218195+0000 2023-11-02T18:30:21.334 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.300+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:entry[obj00000000000000000003]: success 2023-11-02T18:30:21.334 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.300+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20:3 marker=00000000002.36.6 timestamp=2023-11-02T18:30:15.000824+0000 2023-11-02T18:30:21.334 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.300+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000008]: success 2023-11-02T18:30:21.335 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.300+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22:3 marker=00000000002.28.6 timestamp=2023-11-02T18:30:14.571134+0000 2023-11-02T18:30:21.335 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.316+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-11-02T18:30:21.335 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.316+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9/obj00000000000000000007 2023-11-02T18:30:21.335 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.316+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: entry->timestamp=2023-11-02T18:30:14.345634+0000 2023-11-02T18:30:21.335 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.316+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.74.6 op=0 op_state=1 2023-11-02T18:30:21.336 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.316+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000007]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000007[0] 2023-11-02T18:30:21.336 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.316+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position=00000000002.74.6 2023-11-02T18:30:21.336 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.316+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000007]: success 2023-11-02T18:30:21.336 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.316+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9:3 marker=00000000002.74.6 timestamp=2023-11-02T18:30:14.345634+0000 2023-11-02T18:30:21.354 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.348+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000040]: success 2023-11-02T18:30:21.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.348+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11:3 marker=00000000001.40.6 timestamp=2023-11-02T18:30:14.909280+0000 2023-11-02T18:30:21.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.448 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket shard: 0 2023-11-02T18:30:21.448 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.448 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket shard: 0 2023-11-02T18:30:21.448 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.449 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.450 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.450 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket shard: 0 2023-11-02T18:30:21.451 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.451 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.451 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket shard: 0 2023-11-02T18:30:21.451 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket shard: 0 2023-11-02T18:30:21.452 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.452 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.452 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.452 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket shard: 0 2023-11-02T18:30:21.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.440+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.454 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.454 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.454 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket shard: 0 2023-11-02T18:30:21.454 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.455 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.457 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket shard: 0 2023-11-02T18:30:21.457 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.457 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.458 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket shard: 0 2023-11-02T18:30:21.458 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket shard: 0 2023-11-02T18:30:21.458 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: sync status for source bucket shard: 0 2023-11-02T18:30:21.458 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket shard: 0 2023-11-02T18:30:21.459 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.459 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.459 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.459 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.460 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: parsed entry: id=00000000001.67.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-11-02T18:30:21.460 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2/obj00000000000000000027 2023-11-02T18:30:21.460 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: entry->timestamp=2023-11-02T18:30:16.330994+0000 2023-11-02T18:30:21.460 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.67.6 op=0 op_state=1 2023-11-02T18:30:21.461 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.461 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.461 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.461 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000027]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000027[0] 2023-11-02T18:30:21.462 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position=00000000001.67.6 2023-11-02T18:30:21.462 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: parsed entry: id=00000000002.33.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-11-02T18:30:21.462 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5/obj00000000000000000002 2023-11-02T18:30:21.462 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: entry->timestamp=2023-11-02T18:30:16.451750+0000 2023-11-02T18:30:21.463 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.33.6 op=0 op_state=1 2023-11-02T18:30:21.463 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-11-02T18:30:21.463 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3/obj00000000000000000004 2023-11-02T18:30:21.463 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: entry->timestamp=2023-11-02T18:30:15.757912+0000 2023-11-02T18:30:21.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-11-02T18:30:21.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2023-11-02T18:30:21.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position=00000000002.33.6 2023-11-02T18:30:21.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:entry[obj00000000000000000004]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000004[0] 2023-11-02T18:30:21.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: listing bilog for incremental sync; position=00000000002.57.6 2023-11-02T18:30:21.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.466 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.444+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000002]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5/obj00000000000000000002 2023-11-02T18:30:21.466 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: sync status for source bucket shard: 0 2023-11-02T18:30:21.466 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket shard: 0 2023-11-02T18:30:21.466 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.467 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.467 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-11-02T18:30:21.467 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4/obj00000000000000000011 2023-11-02T18:30:21.467 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: entry->timestamp=2023-11-02T18:30:16.361155+0000 2023-11-02T18:30:21.468 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-11-02T18:30:21.468 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.468 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.468 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000011]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000011[0] 2023-11-02T18:30:21.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position=00000000002.46.6 2023-11-02T18:30:21.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: sync status for source bucket shard: 0 2023-11-02T18:30:21.470 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-11-02T18:30:21.470 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8/obj00000000000000000017 2023-11-02T18:30:21.470 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: entry->timestamp=2023-11-02T18:30:16.423077+0000 2023-11-02T18:30:21.470 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.39.6 op=0 op_state=1 2023-11-02T18:30:21.471 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.471 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.471 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.471 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000017]: : injecting data sync error on key=obj00000000000000000017 2023-11-02T18:30:21.472 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position=00000000001.39.6 2023-11-02T18:30:21.472 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.448+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket shard: 0 2023-11-02T18:30:21.477 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: sync status for source bucket shard: 0 2023-11-02T18:30:21.478 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000017]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.478 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000017]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8/obj00000000000000000017 2023-11-02T18:30:21.478 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.478 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-11-02T18:30:21.479 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9/obj00000000000000000008 2023-11-02T18:30:21.479 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: entry->timestamp=2023-11-02T18:30:16.234586+0000 2023-11-02T18:30:21.479 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-11-02T18:30:21.479 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.480 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000008]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000008[0] 2023-11-02T18:30:21.480 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position=00000000002.58.6 2023-11-02T18:30:21.480 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket shard: 0 2023-11-02T18:30:21.480 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: sync status for source bucket shard: 0 2023-11-02T18:30:21.480 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket shard: 0 2023-11-02T18:30:21.481 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: sync status for source bucket shard: 0 2023-11-02T18:30:21.481 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.481 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.481 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2023-11-02T18:30:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7/obj00000000000000000049 2023-11-02T18:30:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: entry->timestamp=2023-11-02T18:30:16.921273+0000 2023-11-02T18:30:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-11-02T18:30:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-11-02T18:30:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000013 2023-11-02T18:30:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: entry->timestamp=2023-11-02T18:30:15.695464+0000 2023-11-02T18:30:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.65.6 op=0 op_state=1 2023-11-02T18:30:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000049]: : injecting data sync error on key=obj00000000000000000049 2023-11-02T18:30:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position=00000000001.52.6 2023-11-02T18:30:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000013]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000013[0] 2023-11-02T18:30:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position=00000000002.65.6 2023-11-02T18:30:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: sync status for source bucket shard: 0 2023-11-02T18:30:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-11-02T18:30:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6/obj00000000000000000019 2023-11-02T18:30:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: entry->timestamp=2023-11-02T18:30:16.640268+0000 2023-11-02T18:30:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-11-02T18:30:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: parsed entry: id=00000000001.62.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2023-11-02T18:30:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11/obj00000000000000000031 2023-11-02T18:30:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: entry->timestamp=2023-11-02T18:30:16.732710+0000 2023-11-02T18:30:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.62.6 op=0 op_state=1 2023-11-02T18:30:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-11-02T18:30:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12/obj00000000000000000022 2023-11-02T18:30:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: entry->timestamp=2023-11-02T18:30:16.545377+0000 2023-11-02T18:30:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-11-02T18:30:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000049]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000049]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7/obj00000000000000000049 2023-11-02T18:30:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: sync status for source bucket shard: 0 2023-11-02T18:30:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: sync status for source bucket shard: 0 2023-11-02T18:30:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: sync status for source bucket shard: 0 2023-11-02T18:30:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: sync status for source bucket shard: 0 2023-11-02T18:30:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.452+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000019]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000019[0] 2023-11-02T18:30:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position=00000000002.48.6 2023-11-02T18:30:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000031]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000031[0] 2023-11-02T18:30:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: listing bilog for incremental sync; position=00000000001.62.6 2023-11-02T18:30:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: a sync operation returned error 2023-11-02T18:30:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000022]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000022[0] 2023-11-02T18:30:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position=00000000002.60.6 2023-11-02T18:30:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2023-11-02T18:30:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15/obj00000000000000000037 2023-11-02T18:30:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: entry->timestamp=2023-11-02T18:30:16.825429+0000 2023-11-02T18:30:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.51.6 op=0 op_state=1 2023-11-02T18:30:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: sync status for source bucket shard: 0 2023-11-02T18:30:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: sync status for source bucket shard: 0 2023-11-02T18:30:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000037]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000037[0] 2023-11-02T18:30:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: listing bilog for incremental sync; position=00000000001.51.6 2023-11-02T18:30:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:entry[obj00000000000000000004]: success 2023-11-02T18:30:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3:4 marker=00000000002.57.6 timestamp=2023-11-02T18:30:15.757912+0000 2023-11-02T18:30:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: backing out with sync_status=-5 2023-11-02T18:30:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-11-02T18:30:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14/obj00000000000000000020 2023-11-02T18:30:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: entry->timestamp=2023-11-02T18:30:15.982517+0000 2023-11-02T18:30:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.456+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.32.6 op=0 op_state=1 2023-11-02T18:30:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.460+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.497 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.460+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000020]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000020[0] 2023-11-02T18:30:21.497 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.460+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position=00000000002.32.6 2023-11-02T18:30:21.497 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.460+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.497 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.460+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-11-02T18:30:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.460+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17/obj00000000000000000012 2023-11-02T18:30:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.460+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: entry->timestamp=2023-11-02T18:30:16.794108+0000 2023-11-02T18:30:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.460+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-11-02T18:30:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.460+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: sync status for source bucket shard: 0 2023-11-02T18:30:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000012]: : injecting data sync error on key=obj00000000000000000012 2023-11-02T18:30:21.499 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: listing bilog for incremental sync; position=00000000002.52.6 2023-11-02T18:30:21.499 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000011]: success 2023-11-02T18:30:21.499 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4:4 marker=00000000002.46.6 timestamp=2023-11-02T18:30:16.361155+0000 2023-11-02T18:30:21.499 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-11-02T18:30:21.500 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20/obj00000000000000000010 2023-11-02T18:30:21.500 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: entry->timestamp=2023-11-02T18:30:15.858397+0000 2023-11-02T18:30:21.500 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.67.6 op=0 op_state=1 2023-11-02T18:30:21.500 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.501 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000012]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.501 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000012]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17/obj00000000000000000012 2023-11-02T18:30:21.501 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: a sync operation returned error 2023-11-02T18:30:21.502 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:entry[obj00000000000000000010]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000010[0] 2023-11-02T18:30:21.502 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: listing bilog for incremental sync; position=00000000002.67.6 2023-11-02T18:30:21.502 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000008]: success 2023-11-02T18:30:21.502 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9:4 marker=00000000002.58.6 timestamp=2023-11-02T18:30:16.234586+0000 2023-11-02T18:30:21.502 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-11-02T18:30:21.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18/obj00000000000000000003 2023-11-02T18:30:21.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: entry->timestamp=2023-11-02T18:30:16.170262+0000 2023-11-02T18:30:21.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-11-02T18:30:21.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000003]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000003[0] 2023-11-02T18:30:21.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: listing bilog for incremental sync; position=00000000002.49.6 2023-11-02T18:30:21.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: backing out with sync_status=-5 2023-11-02T18:30:21.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: a sync operation returned error 2023-11-02T18:30:21.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000027]: success 2023-11-02T18:30:21.505 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2:4 marker=00000000001.67.6 timestamp=2023-11-02T18:30:16.330994+0000 2023-11-02T18:30:21.505 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.505 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.464+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000013]: success 2023-11-02T18:30:21.505 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13:4 marker=00000000002.65.6 timestamp=2023-11-02T18:30:15.695464+0000 2023-11-02T18:30:21.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: parsed entry: id=00000000002.88.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-11-02T18:30:21.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22/obj00000000000000000009 2023-11-02T18:30:21.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: entry->timestamp=2023-11-02T18:30:16.608161+0000 2023-11-02T18:30:21.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.88.6 op=0 op_state=1 2023-11-02T18:30:21.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000019]: success 2023-11-02T18:30:21.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6:4 marker=00000000002.48.6 timestamp=2023-11-02T18:30:16.640268+0000 2023-11-02T18:30:21.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: backing out with sync_status=-5 2023-11-02T18:30:21.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000009]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000009[0] 2023-11-02T18:30:21.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: listing bilog for incremental sync; position=00000000002.88.6 2023-11-02T18:30:21.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-11-02T18:30:21.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23/obj00000000000000000025 2023-11-02T18:30:21.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: entry->timestamp=2023-11-02T18:30:16.702015+0000 2023-11-02T18:30:21.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-11-02T18:30:21.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-11-02T18:30:21.510 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21/obj00000000000000000001 2023-11-02T18:30:21.510 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: entry->timestamp=2023-11-02T18:30:16.513836+0000 2023-11-02T18:30:21.510 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-11-02T18:30:21.510 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-11-02T18:30:21.510 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27/obj00000000000000000030 2023-11-02T18:30:21.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: entry->timestamp=2023-11-02T18:30:16.889705+0000 2023-11-02T18:30:21.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-11-02T18:30:21.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: parsed entry: id=00000000002.33.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-11-02T18:30:21.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25/obj00000000000000000000 2023-11-02T18:30:21.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: entry->timestamp=2023-11-02T18:30:16.265300+0000 2023-11-02T18:30:21.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.33.6 op=0 op_state=1 2023-11-02T18:30:21.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-11-02T18:30:21.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26/obj00000000000000000014 2023-11-02T18:30:21.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: entry->timestamp=2023-11-02T18:30:16.046515+0000 2023-11-02T18:30:21.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-11-02T18:30:21.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000025]: : injecting data sync error on key=obj00000000000000000025 2023-11-02T18:30:21.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: listing bilog for incremental sync; position=00000000002.50.6 2023-11-02T18:30:21.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:entry[obj00000000000000000001]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000001[0] 2023-11-02T18:30:21.514 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: listing bilog for incremental sync; position=00000000002.48.6 2023-11-02T18:30:21.514 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:entry[obj00000000000000000030]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000030[0] 2023-11-02T18:30:21.514 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: listing bilog for incremental sync; position=00000000002.64.6 2023-11-02T18:30:21.514 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000000]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000000[0] 2023-11-02T18:30:21.515 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: listing bilog for incremental sync; position=00000000002.33.6 2023-11-02T18:30:21.515 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:entry[obj00000000000000000014]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000014[0] 2023-11-02T18:30:21.515 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: listing bilog for incremental sync; position=00000000002.37.6 2023-11-02T18:30:21.515 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000025]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.516 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000025]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23/obj00000000000000000025 2023-11-02T18:30:21.516 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000020]: success 2023-11-02T18:30:21.516 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.468+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14:4 marker=00000000002.32.6 timestamp=2023-11-02T18:30:15.982517+0000 2023-11-02T18:30:21.521 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000031]: success 2023-11-02T18:30:21.521 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11:4 marker=00000000001.62.6 timestamp=2023-11-02T18:30:16.732710+0000 2023-11-02T18:30:21.521 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-11-02T18:30:21.521 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28/obj00000000000000000021 2023-11-02T18:30:21.522 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: entry->timestamp=2023-11-02T18:30:15.949856+0000 2023-11-02T18:30:21.522 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-11-02T18:30:21.522 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:entry[obj00000000000000000021]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000021[0] 2023-11-02T18:30:21.523 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: listing bilog for incremental sync; position=00000000002.59.6 2023-11-02T18:30:21.523 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000022]: success 2023-11-02T18:30:21.523 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12:4 marker=00000000002.60.6 timestamp=2023-11-02T18:30:16.545377+0000 2023-11-02T18:30:21.523 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: a sync operation returned error 2023-11-02T18:30:21.524 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: backing out with sync_status=-5 2023-11-02T18:30:21.524 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.524 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.472+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.524 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:entry[obj00000000000000000010]: success 2023-11-02T18:30:21.525 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20:4 marker=00000000002.67.6 timestamp=2023-11-02T18:30:15.858397+0000 2023-11-02T18:30:21.525 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000009]: success 2023-11-02T18:30:21.525 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22:4 marker=00000000002.88.6 timestamp=2023-11-02T18:30:16.608161+0000 2023-11-02T18:30:21.525 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:entry[obj00000000000000000021]: success 2023-11-02T18:30:21.525 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28:4 marker=00000000002.59.6 timestamp=2023-11-02T18:30:15.949856+0000 2023-11-02T18:30:21.526 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000003]: success 2023-11-02T18:30:21.526 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18:4 marker=00000000002.49.6 timestamp=2023-11-02T18:30:16.170262+0000 2023-11-02T18:30:21.526 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:entry[obj00000000000000000030]: success 2023-11-02T18:30:21.526 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27:4 marker=00000000002.64.6 timestamp=2023-11-02T18:30:16.889705+0000 2023-11-02T18:30:21.527 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000000]: success 2023-11-02T18:30:21.527 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25:4 marker=00000000002.33.6 timestamp=2023-11-02T18:30:16.265300+0000 2023-11-02T18:30:21.527 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:entry[obj00000000000000000001]: success 2023-11-02T18:30:21.527 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.476+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21:4 marker=00000000002.48.6 timestamp=2023-11-02T18:30:16.513836+0000 2023-11-02T18:30:21.528 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.528 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: a sync operation returned error 2023-11-02T18:30:21.528 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.528 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.528 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: backing out with sync_status=-5 2023-11-02T18:30:21.529 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.529 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.529 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket shard: 0 2023-11-02T18:30:21.529 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket shard: 0 2023-11-02T18:30:21.530 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket shard: 0 2023-11-02T18:30:21.530 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.530 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.530 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.530 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.531 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:entry[obj00000000000000000014]: success 2023-11-02T18:30:21.531 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.480+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26:4 marker=00000000002.37.6 timestamp=2023-11-02T18:30:16.046515+0000 2023-11-02T18:30:21.531 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.484+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: sync status for source bucket shard: 0 2023-11-02T18:30:21.531 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.484+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.532 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.484+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-11-02T18:30:21.532 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.484+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8/obj00000000000000000017 2023-11-02T18:30:21.532 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.484+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: entry->timestamp=2023-11-02T18:30:16.423077+0000 2023-11-02T18:30:21.532 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.484+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.39.6 op=0 op_state=1 2023-11-02T18:30:21.532 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.484+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000017]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000017[0] 2023-11-02T18:30:21.533 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.484+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position=00000000001.39.6 2023-11-02T18:30:21.533 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.488+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.533 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.488+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: sync status for source bucket shard: 0 2023-11-02T18:30:21.533 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.488+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.534 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.488+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-11-02T18:30:21.534 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.488+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17/obj00000000000000000012 2023-11-02T18:30:21.534 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.488+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: entry->timestamp=2023-11-02T18:30:16.794108+0000 2023-11-02T18:30:21.534 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.488+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-11-02T18:30:21.535 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.488+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000012]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000012[0] 2023-11-02T18:30:21.535 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.488+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: listing bilog for incremental sync; position=00000000002.52.6 2023-11-02T18:30:21.535 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.492+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000012]: success 2023-11-02T18:30:21.535 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.492+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17:4 marker=00000000002.52.6 timestamp=2023-11-02T18:30:16.794108+0000 2023-11-02T18:30:21.535 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.496+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-11-02T18:30:21.536 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.496+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23/obj00000000000000000025 2023-11-02T18:30:21.536 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.496+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: entry->timestamp=2023-11-02T18:30:16.702015+0000 2023-11-02T18:30:21.536 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.496+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-11-02T18:30:21.536 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.496+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000025]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000025[0] 2023-11-02T18:30:21.537 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.496+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: listing bilog for incremental sync; position=00000000002.50.6 2023-11-02T18:30:21.537 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-11-02T18:30:21.537 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16/obj00000000000000000005 2023-11-02T18:30:21.537 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: entry->timestamp=2023-11-02T18:30:15.790967+0000 2023-11-02T18:30:21.538 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-11-02T18:30:21.538 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000005]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000005[0] 2023-11-02T18:30:21.538 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position=00000000002.30.6 2023-11-02T18:30:21.538 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: parsed entry: id=00000000001.46.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2023-11-02T18:30:21.539 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1/obj00000000000000000059 2023-11-02T18:30:21.539 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: entry->timestamp=2023-11-02T18:30:16.142390+0000 2023-11-02T18:30:21.539 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.46.6 op=0 op_state=1 2023-11-02T18:30:21.539 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-11-02T18:30:21.540 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10/obj00000000000000000015 2023-11-02T18:30:21.540 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: entry->timestamp=2023-11-02T18:30:15.889000+0000 2023-11-02T18:30:21.540 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-11-02T18:30:21.540 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2023-11-02T18:30:21.540 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19/obj00000000000000000042 2023-11-02T18:30:21.541 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: entry->timestamp=2023-11-02T18:30:16.079205+0000 2023-11-02T18:30:21.541 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.50.6 op=0 op_state=1 2023-11-02T18:30:21.541 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000025]: success 2023-11-02T18:30:21.541 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23:4 marker=00000000002.50.6 timestamp=2023-11-02T18:30:16.702015+0000 2023-11-02T18:30:21.542 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000059]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000059[0] 2023-11-02T18:30:21.542 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position=00000000001.46.6 2023-11-02T18:30:21.542 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000015]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000015[0] 2023-11-02T18:30:21.542 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position=00000000002.44.6 2023-11-02T18:30:21.543 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000042]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000042[0] 2023-11-02T18:30:21.543 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: listing bilog for incremental sync; position=00000000001.50.6 2023-11-02T18:30:21.543 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000005]: success 2023-11-02T18:30:21.543 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16:4 marker=00000000002.30.6 timestamp=2023-11-02T18:30:15.790967+0000 2023-11-02T18:30:21.544 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000015]: success 2023-11-02T18:30:21.544 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.500+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10:4 marker=00000000002.44.6 timestamp=2023-11-02T18:30:15.889000+0000 2023-11-02T18:30:21.544 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.508+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000042]: success 2023-11-02T18:30:21.544 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.508+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19:4 marker=00000000001.50.6 timestamp=2023-11-02T18:30:16.079205+0000 2023-11-02T18:30:21.545 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.508+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000059]: success 2023-11-02T18:30:21.545 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.508+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1:4 marker=00000000001.46.6 timestamp=2023-11-02T18:30:16.142390+0000 2023-11-02T18:30:21.546 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.516+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000037]: success 2023-11-02T18:30:21.547 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.516+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15:4 marker=00000000001.51.6 timestamp=2023-11-02T18:30:16.825429+0000 2023-11-02T18:30:21.547 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.528+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: parsed entry: id=00000000002.33.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-11-02T18:30:21.547 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.528+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5/obj00000000000000000002 2023-11-02T18:30:21.547 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.528+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: entry->timestamp=2023-11-02T18:30:16.451750+0000 2023-11-02T18:30:21.547 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.528+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.33.6 op=0 op_state=1 2023-11-02T18:30:21.548 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.528+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000002]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000002[0] 2023-11-02T18:30:21.548 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.528+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position=00000000002.33.6 2023-11-02T18:30:21.548 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.528+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2023-11-02T18:30:21.548 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.528+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7/obj00000000000000000049 2023-11-02T18:30:21.549 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.528+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: entry->timestamp=2023-11-02T18:30:16.921273+0000 2023-11-02T18:30:21.549 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.528+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-11-02T18:30:21.549 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.528+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000049]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000049[0] 2023-11-02T18:30:21.549 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.528+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position=00000000001.52.6 2023-11-02T18:30:21.549 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.532+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000002]: success 2023-11-02T18:30:21.550 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.532+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5:4 marker=00000000002.33.6 timestamp=2023-11-02T18:30:16.451750+0000 2023-11-02T18:30:21.550 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.532+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000017]: success 2023-11-02T18:30:21.550 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.532+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8:4 marker=00000000001.39.6 timestamp=2023-11-02T18:30:16.423077+0000 2023-11-02T18:30:21.550 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.536+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000049]: success 2023-11-02T18:30:21.551 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.536+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7:4 marker=00000000001.52.6 timestamp=2023-11-02T18:30:16.921273+0000 2023-11-02T18:30:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.696+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket shard: 0 2023-11-02T18:30:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket shard: 0 2023-11-02T18:30:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket shard: 0 2023-11-02T18:30:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket shard: 0 2023-11-02T18:30:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket shard: 0 2023-11-02T18:30:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket shard: 0 2023-11-02T18:30:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: parsed entry: id=00000000001.73.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2023-11-02T18:30:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2/obj00000000000000000467 2023-11-02T18:30:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: entry->timestamp=2023-11-02T18:30:17.598529+0000 2023-11-02T18:30:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.73.6 op=0 op_state=1 2023-11-02T18:30:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000467]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000467[0] 2023-11-02T18:30:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position=00000000001.73.6 2023-11-02T18:30:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: parsed entry: id=00000000001.63.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2023-11-02T18:30:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4/obj00000000000000000056 2023-11-02T18:30:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: entry->timestamp=2023-11-02T18:30:18.279179+0000 2023-11-02T18:30:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.63.6 op=0 op_state=1 2023-11-02T18:30:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000056]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000056[0] 2023-11-02T18:30:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.700+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position=00000000001.63.6 2023-11-02T18:30:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket shard: 0 2023-11-02T18:30:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket shard: 0 2023-11-02T18:30:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket shard: 0 2023-11-02T18:30:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.719 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket shard: 0 2023-11-02T18:30:21.719 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket shard: 0 2023-11-02T18:30:21.720 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.720 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.720 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-11-02T18:30:21.720 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5/obj00000000000000000004 2023-11-02T18:30:21.720 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: entry->timestamp=2023-11-02T18:30:17.725990+0000 2023-11-02T18:30:21.721 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.704+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-11-02T18:30:21.721 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000004]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000004[0] 2023-11-02T18:30:21.721 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position=00000000002.42.6 2023-11-02T18:30:21.721 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-11-02T18:30:21.721 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8/obj00000000000000000022 2023-11-02T18:30:21.722 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: entry->timestamp=2023-11-02T18:30:17.505219+0000 2023-11-02T18:30:21.722 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-11-02T18:30:21.722 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-11-02T18:30:21.722 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9/obj00000000000000000000 2023-11-02T18:30:21.723 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: entry->timestamp=2023-11-02T18:30:17.632493+0000 2023-11-02T18:30:21.723 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-11-02T18:30:21.723 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000022]: : injecting data sync error on key=obj00000000000000000022 2023-11-02T18:30:21.723 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position=00000000002.46.6 2023-11-02T18:30:21.723 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000000]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000000[0] 2023-11-02T18:30:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position=00000000002.52.6 2023-11-02T18:30:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000022]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000022]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8/obj00000000000000000022 2023-11-02T18:30:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.708+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-11-02T18:30:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10/obj00000000000000000021 2023-11-02T18:30:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: entry->timestamp=2023-11-02T18:30:17.444704+0000 2023-11-02T18:30:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-11-02T18:30:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000021]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000021[0] 2023-11-02T18:30:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position=00000000002.54.6 2023-11-02T18:30:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: parsed entry: id=00000000001.61.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2023-11-02T18:30:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7/obj00000000000000000317 2023-11-02T18:30:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: entry->timestamp=2023-11-02T18:30:18.119583+0000 2023-11-02T18:30:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.61.6 op=0 op_state=1 2023-11-02T18:30:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.732 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000317]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000317[0] 2023-11-02T18:30:21.732 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position=00000000001.61.6 2023-11-02T18:30:21.736 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket shard: 0 2023-11-02T18:30:21.736 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket shard: 0 2023-11-02T18:30:21.737 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: sync status for source bucket shard: 0 2023-11-02T18:30:21.737 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: sync status for source bucket shard: 0 2023-11-02T18:30:21.737 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: sync status for source bucket shard: 0 2023-11-02T18:30:21.737 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: sync status for source bucket shard: 0 2023-11-02T18:30:21.738 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: sync status for source bucket shard: 0 2023-11-02T18:30:21.738 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: sync status for source bucket shard: 0 2023-11-02T18:30:21.738 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: sync status for source bucket shard: 0 2023-11-02T18:30:21.738 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.739 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.739 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.739 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.739 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.740 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.740 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.740 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.740 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket shard: 0 2023-11-02T18:30:21.741 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: sync status for source bucket shard: 0 2023-11-02T18:30:21.741 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.741 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.741 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.742 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket shard: 0 2023-11-02T18:30:21.742 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: sync status for source bucket shard: 0 2023-11-02T18:30:21.742 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: sync status for source bucket shard: 0 2023-11-02T18:30:21.742 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.743 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.743 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.743 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-11-02T18:30:21.743 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0/obj00000000000000000019 2023-11-02T18:30:21.744 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: entry->timestamp=2023-11-02T18:30:17.925184+0000 2023-11-02T18:30:21.744 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-11-02T18:30:21.744 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.744 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.745 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.745 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000000]: success 2023-11-02T18:30:21.745 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9:5 marker=00000000002.52.6 timestamp=2023-11-02T18:30:17.632493+0000 2023-11-02T18:30:21.745 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000004]: success 2023-11-02T18:30:21.745 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.712+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5:5 marker=00000000002.42.6 timestamp=2023-11-02T18:30:17.725990+0000 2023-11-02T18:30:21.746 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.716+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000019]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000019[0] 2023-11-02T18:30:21.746 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.716+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: listing bilog for incremental sync; position=00000000002.53.6 2023-11-02T18:30:21.746 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.716+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: a sync operation returned error 2023-11-02T18:30:21.746 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.720+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: sync status for source bucket shard: 0 2023-11-02T18:30:21.747 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.720+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: sync status for source bucket shard: 0 2023-11-02T18:30:21.747 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.720+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: sync status for source bucket shard: 0 2023-11-02T18:30:21.747 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.720+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: backing out with sync_status=-5 2023-11-02T18:30:21.747 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.748 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.748 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.748 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.748 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.748 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: sync status for source bucket shard: 0 2023-11-02T18:30:21.749 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: sync status for source bucket shard: 0 2023-11-02T18:30:21.749 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000021]: success 2023-11-02T18:30:21.749 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10:5 marker=00000000002.54.6 timestamp=2023-11-02T18:30:17.444704+0000 2023-11-02T18:30:21.749 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: sync status for source bucket shard: 0 2023-11-02T18:30:21.749 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.750 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.750 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.750 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: parsed entry: id=00000000002.28.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-11-02T18:30:21.750 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18/obj00000000000000000016 2023-11-02T18:30:21.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: entry->timestamp=2023-11-02T18:30:17.823378+0000 2023-11-02T18:30:21.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.28.6 op=0 op_state=1 2023-11-02T18:30:21.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-11-02T18:30:21.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000008 2023-11-02T18:30:21.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: entry->timestamp=2023-11-02T18:30:18.654730+0000 2023-11-02T18:30:21.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.32.6 op=0 op_state=1 2023-11-02T18:30:21.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2023-11-02T18:30:21.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20/obj00000000000000000423 2023-11-02T18:30:21.753 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: entry->timestamp=2023-11-02T18:30:18.090986+0000 2023-11-02T18:30:21.753 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.59.6 op=0 op_state=1 2023-11-02T18:30:21.753 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2023-11-02T18:30:21.753 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25/obj00000000000000000283 2023-11-02T18:30:21.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: entry->timestamp=2023-11-02T18:30:17.535381+0000 2023-11-02T18:30:21.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-11-02T18:30:21.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000016]: : injecting data sync error on key=obj00000000000000000016 2023-11-02T18:30:21.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: listing bilog for incremental sync; position=00000000002.28.6 2023-11-02T18:30:21.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: : injecting data sync error on key=obj00000000000000000008 2023-11-02T18:30:21.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position=00000000002.32.6 2023-11-02T18:30:21.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-11-02T18:30:21.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16/obj00000000000000000017 2023-11-02T18:30:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: entry->timestamp=2023-11-02T18:30:17.412456+0000 2023-11-02T18:30:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-11-02T18:30:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-11-02T18:30:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23/obj00000000000000000001 2023-11-02T18:30:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: entry->timestamp=2023-11-02T18:30:18.028419+0000 2023-11-02T18:30:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-11-02T18:30:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-11-02T18:30:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22/obj00000000000000000012 2023-11-02T18:30:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: entry->timestamp=2023-11-02T18:30:18.215575+0000 2023-11-02T18:30:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-11-02T18:30:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-11-02T18:30:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.724+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15/obj00000000000000000007 2023-11-02T18:30:21.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: entry->timestamp=2023-11-02T18:30:17.991996+0000 2023-11-02T18:30:21.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-11-02T18:30:21.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-11-02T18:30:21.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21/obj00000000000000000002 2023-11-02T18:30:21.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: entry->timestamp=2023-11-02T18:30:18.497761+0000 2023-11-02T18:30:21.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-11-02T18:30:21.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2023-11-02T18:30:21.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12/obj00000000000000000271 2023-11-02T18:30:21.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: entry->timestamp=2023-11-02T18:30:18.687164+0000 2023-11-02T18:30:21.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.37.6 op=0 op_state=1 2023-11-02T18:30:21.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2023-11-02T18:30:21.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17/obj00000000000000000162 2023-11-02T18:30:21.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: entry->timestamp=2023-11-02T18:30:18.306794+0000 2023-11-02T18:30:21.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.53.6 op=0 op_state=1 2023-11-02T18:30:21.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-11-02T18:30:21.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19/obj00000000000000000003 2023-11-02T18:30:21.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: entry->timestamp=2023-11-02T18:30:17.791674+0000 2023-11-02T18:30:21.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-11-02T18:30:21.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000467]: success 2023-11-02T18:30:21.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2:5 marker=00000000001.73.6 timestamp=2023-11-02T18:30:17.598529+0000 2023-11-02T18:30:21.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:entry[obj00000000000000000423]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000423[0] 2023-11-02T18:30:21.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: listing bilog for incremental sync; position=00000000001.59.6 2023-11-02T18:30:21.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000019]: success 2023-11-02T18:30:21.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0:5 marker=00000000002.53.6 timestamp=2023-11-02T18:30:17.925184+0000 2023-11-02T18:30:21.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000283]: : injecting data sync error on key=obj00000000000000000283 2023-11-02T18:30:21.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: listing bilog for incremental sync; position=00000000001.52.6 2023-11-02T18:30:21.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000016]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000016]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18/obj00000000000000000016 2023-11-02T18:30:21.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000008 2023-11-02T18:30:21.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000017]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000017[0] 2023-11-02T18:30:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position=00000000002.61.6 2023-11-02T18:30:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000001]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000001[0] 2023-11-02T18:30:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: listing bilog for incremental sync; position=00000000002.30.6 2023-11-02T18:30:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000012]: : injecting data sync error on key=obj00000000000000000012 2023-11-02T18:30:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: listing bilog for incremental sync; position=00000000002.53.6 2023-11-02T18:30:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000007]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000007[0] 2023-11-02T18:30:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: listing bilog for incremental sync; position=00000000002.47.6 2023-11-02T18:30:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:entry[obj00000000000000000002]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000002[0] 2023-11-02T18:30:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: listing bilog for incremental sync; position=00000000002.48.6 2023-11-02T18:30:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000271]: : injecting data sync error on key=obj00000000000000000271 2023-11-02T18:30:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position=00000000001.37.6 2023-11-02T18:30:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000056]: success 2023-11-02T18:30:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4:5 marker=00000000001.63.6 timestamp=2023-11-02T18:30:18.279179+0000 2023-11-02T18:30:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000162]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000162[0] 2023-11-02T18:30:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: listing bilog for incremental sync; position=00000000001.53.6 2023-11-02T18:30:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000003]: : injecting data sync error on key=obj00000000000000000003 2023-11-02T18:30:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: listing bilog for incremental sync; position=00000000002.51.6 2023-11-02T18:30:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000283]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000283]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25/obj00000000000000000283 2023-11-02T18:30:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000012]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000012]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22/obj00000000000000000012 2023-11-02T18:30:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000271]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000271]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12/obj00000000000000000271 2023-11-02T18:30:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000003]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.728+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000003]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19/obj00000000000000000003 2023-11-02T18:30:21.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.736+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2023-11-02T18:30:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.736+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27/obj00000000000000000061 2023-11-02T18:30:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.736+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: entry->timestamp=2023-11-02T18:30:18.562805+0000 2023-11-02T18:30:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.736+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-11-02T18:30:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:entry[obj00000000000000000061]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000061[0] 2023-11-02T18:30:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: listing bilog for incremental sync; position=00000000001.49.6 2023-11-02T18:30:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-11-02T18:30:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29/obj00000000000000000029 2023-11-02T18:30:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: entry->timestamp=2023-11-02T18:30:17.892795+0000 2023-11-02T18:30:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-11-02T18:30:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-11-02T18:30:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28/obj00000000000000000050 2023-11-02T18:30:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: entry->timestamp=2023-11-02T18:30:18.401770+0000 2023-11-02T18:30:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-11-02T18:30:21.780 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: parsed entry: id=00000000001.63.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2023-11-02T18:30:21.780 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30/obj00000000000000000133 2023-11-02T18:30:21.780 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: entry->timestamp=2023-11-02T18:30:18.467373+0000 2023-11-02T18:30:21.780 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.63.6 op=0 op_state=1 2023-11-02T18:30:21.781 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:entry[obj00000000000000000029]: : injecting data sync error on key=obj00000000000000000029 2023-11-02T18:30:21.781 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: listing bilog for incremental sync; position=00000000001.54.6 2023-11-02T18:30:21.781 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:entry[obj00000000000000000050]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000050[0] 2023-11-02T18:30:21.781 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: listing bilog for incremental sync; position=00000000002.58.6 2023-11-02T18:30:21.782 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:entry[obj00000000000000000133]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000133[0] 2023-11-02T18:30:21.782 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: listing bilog for incremental sync; position=00000000001.63.6 2023-11-02T18:30:21.782 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000017]: success 2023-11-02T18:30:21.782 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16:5 marker=00000000002.61.6 timestamp=2023-11-02T18:30:17.412456+0000 2023-11-02T18:30:21.782 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000001]: success 2023-11-02T18:30:21.783 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23:5 marker=00000000002.30.6 timestamp=2023-11-02T18:30:18.028419+0000 2023-11-02T18:30:21.783 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000007]: success 2023-11-02T18:30:21.783 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15:5 marker=00000000002.47.6 timestamp=2023-11-02T18:30:17.991996+0000 2023-11-02T18:30:21.783 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:entry[obj00000000000000000029]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.784 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:entry[obj00000000000000000029]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29/obj00000000000000000029 2023-11-02T18:30:21.784 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: a sync operation returned error 2023-11-02T18:30:21.784 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: a sync operation returned error 2023-11-02T18:30:21.784 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: a sync operation returned error 2023-11-02T18:30:21.784 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: backing out with sync_status=-5 2023-11-02T18:30:21.785 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: backing out with sync_status=-5 2023-11-02T18:30:21.785 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: backing out with sync_status=-5 2023-11-02T18:30:21.785 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:entry[obj00000000000000000002]: success 2023-11-02T18:30:21.785 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21:5 marker=00000000002.48.6 timestamp=2023-11-02T18:30:18.497761+0000 2023-11-02T18:30:21.786 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.786 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.786 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.786 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: a sync operation returned error 2023-11-02T18:30:21.786 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: a sync operation returned error 2023-11-02T18:30:21.787 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.787 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.787 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.787 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: backing out with sync_status=-5 2023-11-02T18:30:21.788 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: backing out with sync_status=-5 2023-11-02T18:30:21.788 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.788 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.788 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.789 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.740+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.789 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.744+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000162]: success 2023-11-02T18:30:21.789 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.744+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17:5 marker=00000000001.53.6 timestamp=2023-11-02T18:30:18.306794+0000 2023-11-02T18:30:21.789 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.744+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: a sync operation returned error 2023-11-02T18:30:21.789 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.744+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: backing out with sync_status=-5 2023-11-02T18:30:21.790 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.744+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.790 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.744+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.790 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.744+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:entry[obj00000000000000000050]: success 2023-11-02T18:30:21.790 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.744+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28:5 marker=00000000002.58.6 timestamp=2023-11-02T18:30:18.401770+0000 2023-11-02T18:30:21.791 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.744+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.791 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket shard: 0 2023-11-02T18:30:21.791 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.791 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2023-11-02T18:30:21.791 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6/obj00000000000000000023 2023-11-02T18:30:21.792 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: entry->timestamp=2023-11-02T18:30:18.591898+0000 2023-11-02T18:30:21.792 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.65.6 op=0 op_state=1 2023-11-02T18:30:21.792 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000023]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000023[0] 2023-11-02T18:30:21.792 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position=00000000002.65.6 2023-11-02T18:30:21.793 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-11-02T18:30:21.793 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8/obj00000000000000000022 2023-11-02T18:30:21.793 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: entry->timestamp=2023-11-02T18:30:17.505219+0000 2023-11-02T18:30:21.793 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-11-02T18:30:21.793 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000022]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000022[0] 2023-11-02T18:30:21.794 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position=00000000002.46.6 2023-11-02T18:30:21.794 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:entry[obj00000000000000000133]: success 2023-11-02T18:30:21.794 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.748+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30:5 marker=00000000001.63.6 timestamp=2023-11-02T18:30:18.467373+0000 2023-11-02T18:30:21.794 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.795 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.795 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: sync status for source bucket shard: 0 2023-11-02T18:30:21.795 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.795 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: sync status for source bucket shard: 0 2023-11-02T18:30:21.795 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.796 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.796 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.796 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000022]: success 2023-11-02T18:30:21.796 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8:5 marker=00000000002.46.6 timestamp=2023-11-02T18:30:17.505219+0000 2023-11-02T18:30:21.796 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.797 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: sync status for source bucket shard: 0 2023-11-02T18:30:21.797 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.797 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.752+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.797 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket shard: 0 2023-11-02T18:30:21.798 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.798 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: sync status for source bucket shard: 0 2023-11-02T18:30:21.798 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.798 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: parsed entry: id=00000000002.28.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-11-02T18:30:21.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18/obj00000000000000000016 2023-11-02T18:30:21.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: entry->timestamp=2023-11-02T18:30:17.823378+0000 2023-11-02T18:30:21.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.28.6 op=0 op_state=1 2023-11-02T18:30:21.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2023-11-02T18:30:21.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25/obj00000000000000000283 2023-11-02T18:30:21.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: entry->timestamp=2023-11-02T18:30:17.535381+0000 2023-11-02T18:30:21.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-11-02T18:30:21.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000016]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000016[0] 2023-11-02T18:30:21.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: listing bilog for incremental sync; position=00000000002.28.6 2023-11-02T18:30:21.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000283]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000283[0] 2023-11-02T18:30:21.801 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: listing bilog for incremental sync; position=00000000001.52.6 2023-11-02T18:30:21.801 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-11-02T18:30:21.801 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22/obj00000000000000000012 2023-11-02T18:30:21.801 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: entry->timestamp=2023-11-02T18:30:18.215575+0000 2023-11-02T18:30:21.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-11-02T18:30:21.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000012]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000012[0] 2023-11-02T18:30:21.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: listing bilog for incremental sync; position=00000000002.53.6 2023-11-02T18:30:21.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: sync status for source bucket shard: 0 2023-11-02T18:30:21.803 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.803 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2023-11-02T18:30:21.803 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12/obj00000000000000000271 2023-11-02T18:30:21.803 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: entry->timestamp=2023-11-02T18:30:18.687164+0000 2023-11-02T18:30:21.804 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.37.6 op=0 op_state=1 2023-11-02T18:30:21.804 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000271]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000271[0] 2023-11-02T18:30:21.804 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position=00000000001.37.6 2023-11-02T18:30:21.804 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-11-02T18:30:21.805 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19/obj00000000000000000003 2023-11-02T18:30:21.805 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: entry->timestamp=2023-11-02T18:30:17.791674+0000 2023-11-02T18:30:21.805 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-11-02T18:30:21.805 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000003]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000003[0] 2023-11-02T18:30:21.806 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.756+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: listing bilog for incremental sync; position=00000000002.51.6 2023-11-02T18:30:21.806 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.760+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000016]: success 2023-11-02T18:30:21.806 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.760+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18:5 marker=00000000002.28.6 timestamp=2023-11-02T18:30:17.823378+0000 2023-11-02T18:30:21.806 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.760+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-11-02T18:30:21.806 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.760+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29/obj00000000000000000029 2023-11-02T18:30:21.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.760+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: entry->timestamp=2023-11-02T18:30:17.892795+0000 2023-11-02T18:30:21.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.760+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-11-02T18:30:21.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.760+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:entry[obj00000000000000000029]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000029[0] 2023-11-02T18:30:21.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.760+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: listing bilog for incremental sync; position=00000000001.54.6 2023-11-02T18:30:21.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.760+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:entry[obj00000000000000000012]: success 2023-11-02T18:30:21.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.760+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22:5 marker=00000000002.53.6 timestamp=2023-11-02T18:30:18.215575+0000 2023-11-02T18:30:21.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.760+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000023]: success 2023-11-02T18:30:21.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.760+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6:5 marker=00000000002.65.6 timestamp=2023-11-02T18:30:18.591898+0000 2023-11-02T18:30:21.809 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.764+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000003]: success 2023-11-02T18:30:21.809 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.764+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19:5 marker=00000000002.51.6 timestamp=2023-11-02T18:30:17.791674+0000 2023-11-02T18:30:21.809 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.764+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-11-02T18:30:21.809 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.764+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11/obj00000000000000000009 2023-11-02T18:30:21.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.764+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: entry->timestamp=2023-11-02T18:30:18.184473+0000 2023-11-02T18:30:21.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.764+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-11-02T18:30:21.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.764+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000009]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000009[0] 2023-11-02T18:30:21.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.764+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: listing bilog for incremental sync; position=00000000002.38.6 2023-11-02T18:30:21.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000009]: success 2023-11-02T18:30:21.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11:5 marker=00000000002.38.6 timestamp=2023-11-02T18:30:18.184473+0000 2023-11-02T18:30:21.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-11-02T18:30:21.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14/obj00000000000000000018 2023-11-02T18:30:21.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: entry->timestamp=2023-11-02T18:30:17.696303+0000 2023-11-02T18:30:21.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-11-02T18:30:21.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-11-02T18:30:21.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24/obj00000000000000000011 2023-11-02T18:30:21.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: entry->timestamp=2023-11-02T18:30:18.370855+0000 2023-11-02T18:30:21.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.93.6 op=0 op_state=1 2023-11-02T18:30:21.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000018]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000018[0] 2023-11-02T18:30:21.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position=00000000002.49.6 2023-11-02T18:30:21.814 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:entry[obj00000000000000000011]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000011[0] 2023-11-02T18:30:21.814 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: listing bilog for incremental sync; position=00000000002.93.6 2023-11-02T18:30:21.816 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:entry[obj00000000000000000011]: success 2023-11-02T18:30:21.816 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.768+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24:5 marker=00000000002.93.6 timestamp=2023-11-02T18:30:18.370855+0000 2023-11-02T18:30:21.816 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.772+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:entry[obj00000000000000000029]: success 2023-11-02T18:30:21.816 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.772+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29:5 marker=00000000001.54.6 timestamp=2023-11-02T18:30:17.892795+0000 2023-11-02T18:30:21.817 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.772+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000317]: success 2023-11-02T18:30:21.817 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.772+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7:5 marker=00000000001.61.6 timestamp=2023-11-02T18:30:18.119583+0000 2023-11-02T18:30:21.817 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.776+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: a sync operation returned error 2023-11-02T18:30:21.817 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.776+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: backing out with sync_status=-5 2023-11-02T18:30:21.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.776+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.776+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.776+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:entry[obj00000000000000000423]: success 2023-11-02T18:30:21.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.776+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20:5 marker=00000000001.59.6 timestamp=2023-11-02T18:30:18.090986+0000 2023-11-02T18:30:21.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.780+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.819 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.780+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket shard: 0 2023-11-02T18:30:21.819 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.780+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.819 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.784+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-11-02T18:30:21.819 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.784+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000008 2023-11-02T18:30:21.820 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.784+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: entry->timestamp=2023-11-02T18:30:18.654730+0000 2023-11-02T18:30:21.820 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.784+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.32.6 op=0 op_state=1 2023-11-02T18:30:21.820 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.784+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: : injecting data sync error on key=obj00000000000000000008 2023-11-02T18:30:21.820 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.784+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position=00000000002.32.6 2023-11-02T18:30:21.821 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.784+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.821 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.784+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000008 2023-11-02T18:30:21.821 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.788+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:entry[obj00000000000000000061]: success 2023-11-02T18:30:21.821 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.788+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27:5 marker=00000000001.49.6 timestamp=2023-11-02T18:30:18.562805+0000 2023-11-02T18:30:21.821 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.808+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000283]: success 2023-11-02T18:30:21.822 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.808+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25:5 marker=00000000001.52.6 timestamp=2023-11-02T18:30:17.535381+0000 2023-11-02T18:30:21.822 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.808+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000271]: success 2023-11-02T18:30:21.823 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.808+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12:5 marker=00000000001.37.6 timestamp=2023-11-02T18:30:18.687164+0000 2023-11-02T18:30:21.823 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.812+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000018]: success 2023-11-02T18:30:21.823 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.812+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14:5 marker=00000000002.49.6 timestamp=2023-11-02T18:30:17.696303+0000 2023-11-02T18:30:21.837 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.832+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: a sync operation returned error 2023-11-02T18:30:21.838 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.832+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: backing out with sync_status=-5 2023-11-02T18:30:21.838 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.832+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.838 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.832+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.847 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.840+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.848 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.840+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket shard: 0 2023-11-02T18:30:21.848 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.840+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.844+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-11-02T18:30:21.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.844+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000008 2023-11-02T18:30:21.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.844+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: entry->timestamp=2023-11-02T18:30:18.654730+0000 2023-11-02T18:30:21.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.844+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.32.6 op=0 op_state=1 2023-11-02T18:30:21.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.844+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: : injecting data sync error on key=obj00000000000000000008 2023-11-02T18:30:21.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.844+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position=00000000002.32.6 2023-11-02T18:30:21.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.844+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:21.852 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.844+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000008 2023-11-02T18:30:21.852 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.844+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: a sync operation returned error 2023-11-02T18:30:21.853 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.844+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: backing out with sync_status=-5 2023-11-02T18:30:21.853 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.844+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:21.853 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.844+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:21.864 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.856+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.865 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.856+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket shard: 0 2023-11-02T18:30:21.865 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.856+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.868 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.860+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-11-02T18:30:21.868 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.860+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000008 2023-11-02T18:30:21.868 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.860+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: entry->timestamp=2023-11-02T18:30:18.654730+0000 2023-11-02T18:30:21.868 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.860+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.32.6 op=0 op_state=1 2023-11-02T18:30:21.869 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.860+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000008[0] 2023-11-02T18:30:21.869 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.860+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position=00000000002.32.6 2023-11-02T18:30:21.870 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.864+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000008]: success 2023-11-02T18:30:21.871 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.864+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13:5 marker=00000000002.32.6 timestamp=2023-11-02T18:30:18.654730+0000 2023-11-02T18:30:21.993 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.984+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.993 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.984+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.994 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.984+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.994 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: sync status for source bucket shard: 0 2023-11-02T18:30:21.994 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.995 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.995 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: sync status for source bucket shard: 0 2023-11-02T18:30:21.995 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: sync status for source bucket shard: 0 2023-11-02T18:30:21.996 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.996 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.997 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.997 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.997 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: sync status for source bucket shard: 0 2023-11-02T18:30:21.998 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.998 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: sync status for source bucket shard: 0 2023-11-02T18:30:21.998 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.998 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.999 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: sync status for source bucket shard: 0 2023-11-02T18:30:21.999 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:21.999 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position= 2023-11-02T18:30:21.999 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: sync status for source bucket shard: 0 2023-11-02T18:30:22.000 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.000 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-11-02T18:30:22.000 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2/obj00000000000000000011 2023-11-02T18:30:22.000 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: entry->timestamp=2023-11-02T18:30:19.875656+0000 2023-11-02T18:30:22.001 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-11-02T18:30:22.001 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000011[0] 2023-11-02T18:30:22.001 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: listing bilog for incremental sync; position=00000000002.61.6 2023-11-02T18:30:22.001 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: sync status for source bucket shard: 0 2023-11-02T18:30:22.001 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.002 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.988+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-11-02T18:30:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4/obj00000000000000000033 2023-11-02T18:30:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: entry->timestamp=2023-11-02T18:30:19.326434+0000 2023-11-02T18:30:22.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-11-02T18:30:22.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-11-02T18:30:22.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1/obj00000000000000000000 2023-11-02T18:30:22.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: entry->timestamp=2023-11-02T18:30:19.783705+0000 2023-11-02T18:30:22.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.83.6 op=0 op_state=1 2023-11-02T18:30:22.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000033]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000033[0] 2023-11-02T18:30:22.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: listing bilog for incremental sync; position=00000000002.54.6 2023-11-02T18:30:22.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: parsed entry: id=00000000001.103.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2023-11-02T18:30:22.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0/obj00000000000000000060 2023-11-02T18:30:22.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: entry->timestamp=2023-11-02T18:30:20.163612+0000 2023-11-02T18:30:22.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.103.6 op=0 op_state=1 2023-11-02T18:30:22.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000000]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000000[0] 2023-11-02T18:30:22.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: listing bilog for incremental sync; position=00000000002.83.6 2023-11-02T18:30:22.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000060]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000060[0] 2023-11-02T18:30:22.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: listing bilog for incremental sync; position=00000000001.103.6 2023-11-02T18:30:22.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-11-02T18:30:22.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3/obj00000000000000000001 2023-11-02T18:30:22.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: entry->timestamp=2023-11-02T18:30:20.351167+0000 2023-11-02T18:30:22.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.74.6 op=0 op_state=1 2023-11-02T18:30:22.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:entry[obj00000000000000000001]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000001[0] 2023-11-02T18:30:22.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: listing bilog for incremental sync; position=00000000002.74.6 2023-11-02T18:30:22.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2023-11-02T18:30:22.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5/obj00000000000000000065 2023-11-02T18:30:22.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: entry->timestamp=2023-11-02T18:30:20.569819+0000 2023-11-02T18:30:22.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.41.6 op=0 op_state=1 2023-11-02T18:30:22.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000065]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000065[0] 2023-11-02T18:30:22.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: listing bilog for incremental sync; position=00000000001.41.6 2023-11-02T18:30:22.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-11-02T18:30:22.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6/obj00000000000000000005 2023-11-02T18:30:22.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: entry->timestamp=2023-11-02T18:30:19.687742+0000 2023-11-02T18:30:22.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-11-02T18:30:22.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000005]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000005[0] 2023-11-02T18:30:22.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: listing bilog for incremental sync; position=00000000002.79.6 2023-11-02T18:30:22.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: sync status for source bucket shard: 0 2023-11-02T18:30:22.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2023-11-02T18:30:22.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8/obj00000000000000000048 2023-11-02T18:30:22.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: entry->timestamp=2023-11-02T18:30:19.425898+0000 2023-11-02T18:30:22.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.56.6 op=0 op_state=1 2023-11-02T18:30:22.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.992+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.015 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000048]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000048[0] 2023-11-02T18:30:22.015 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: listing bilog for incremental sync; position=00000000001.56.6 2023-11-02T18:30:22.015 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:entry[obj00000000000000000011]: success 2023-11-02T18:30:22.015 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:2:6 marker=00000000002.61.6 timestamp=2023-11-02T18:30:19.875656+0000 2023-11-02T18:30:22.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: sync status for source bucket shard: 0 2023-11-02T18:30:22.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: sync status for source bucket shard: 0 2023-11-02T18:30:22.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: sync status for source bucket shard: 0 2023-11-02T18:30:22.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.020 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: sync status for source bucket shard: 0 2023-11-02T18:30:22.020 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: sync status for source bucket shard: 0 2023-11-02T18:30:22.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:entry[obj00000000000000000000]: success 2023-11-02T18:30:22.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:1:6 marker=00000000002.83.6 timestamp=2023-11-02T18:30:19.783705+0000 2023-11-02T18:30:22.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:21.996+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:entry[obj00000000000000000001]: success 2023-11-02T18:30:22.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:3:6 marker=00000000002.74.6 timestamp=2023-11-02T18:30:20.351167+0000 2023-11-02T18:30:22.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.024 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.024 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.024 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: sync status for source bucket shard: 0 2023-11-02T18:30:22.024 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.025 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: sync status for source bucket shard: 0 2023-11-02T18:30:22.025 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.025 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-11-02T18:30:22.025 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7/obj00000000000000000016 2023-11-02T18:30:22.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: entry->timestamp=2023-11-02T18:30:20.476108+0000 2023-11-02T18:30:22.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.000+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-11-02T18:30:22.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: parsed entry: id=00000000002.72.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-11-02T18:30:22.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11/obj00000000000000000030 2023-11-02T18:30:22.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: entry->timestamp=2023-11-02T18:30:19.593004+0000 2023-11-02T18:30:22.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.72.6 op=0 op_state=1 2023-11-02T18:30:22.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000016]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000016[0] 2023-11-02T18:30:22.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: listing bilog for incremental sync; position=00000000002.63.6 2023-11-02T18:30:22.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-11-02T18:30:22.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10/obj00000000000000000026 2023-11-02T18:30:22.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: entry->timestamp=2023-11-02T18:30:19.491264+0000 2023-11-02T18:30:22.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-11-02T18:30:22.029 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000030]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000030[0] 2023-11-02T18:30:22.029 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: listing bilog for incremental sync; position=00000000002.72.6 2023-11-02T18:30:22.029 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000026]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000026[0] 2023-11-02T18:30:22.029 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: listing bilog for incremental sync; position=00000000002.79.6 2023-11-02T18:30:22.029 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: sync status for source bucket shard: 0 2023-11-02T18:30:22.030 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: sync status for source bucket shard: 0 2023-11-02T18:30:22.030 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:entry[obj00000000000000000005]: success 2023-11-02T18:30:22.030 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:6:6 marker=00000000002.79.6 timestamp=2023-11-02T18:30:19.687742+0000 2023-11-02T18:30:22.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: sync status for source bucket shard: 0 2023-11-02T18:30:22.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: sync status for source bucket shard: 0 2023-11-02T18:30:22.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket shard: 0 2023-11-02T18:30:22.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: sync status for source bucket shard: 0 2023-11-02T18:30:22.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:22]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: sync status for source bucket shard: 0 2023-11-02T18:30:22.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: sync status for source bucket shard: 0 2023-11-02T18:30:22.033 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.033 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.033 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.033 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.033 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.034 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.004+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.034 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.034 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.034 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.035 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.035 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.035 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.035 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.036 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.036 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.036 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.036 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.036 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.037 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.037 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-11-02T18:30:22.037 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20/obj00000000000000000006 2023-11-02T18:30:22.037 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: entry->timestamp=2023-11-02T18:30:19.621804+0000 2023-11-02T18:30:22.037 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-11-02T18:30:22.038 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2023-11-02T18:30:22.038 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12/obj00000000000000000062 2023-11-02T18:30:22.038 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: entry->timestamp=2023-11-02T18:30:20.259397+0000 2023-11-02T18:30:22.038 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.59.6 op=0 op_state=1 2023-11-02T18:30:22.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:entry[obj00000000000000000033]: success 2023-11-02T18:30:22.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:4:6 marker=00000000002.54.6 timestamp=2023-11-02T18:30:19.326434+0000 2023-11-02T18:30:22.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-11-02T18:30:22.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13/obj00000000000000000009 2023-11-02T18:30:22.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: entry->timestamp=2023-11-02T18:30:19.201697+0000 2023-11-02T18:30:22.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-11-02T18:30:22.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-11-02T18:30:22.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14/obj00000000000000000040 2023-11-02T18:30:22.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: entry->timestamp=2023-11-02T18:30:20.099358+0000 2023-11-02T18:30:22.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-11-02T18:30:22.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:entry[obj00000000000000000006]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000006[0] 2023-11-02T18:30:22.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: listing bilog for incremental sync; position=00000000002.48.6 2023-11-02T18:30:22.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000062]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000062[0] 2023-11-02T18:30:22.042 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: listing bilog for incremental sync; position=00000000001.59.6 2023-11-02T18:30:22.042 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000009]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000009[0] 2023-11-02T18:30:22.042 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.008+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: listing bilog for incremental sync; position=00000000002.62.6 2023-11-02T18:30:22.042 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.012+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000040]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000040[0] 2023-11-02T18:30:22.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.012+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: listing bilog for incremental sync; position=00000000002.64.6 2023-11-02T18:30:22.046 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.012+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:entry[obj00000000000000000065]: success 2023-11-02T18:30:22.046 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.012+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:5:6 marker=00000000001.41.6 timestamp=2023-11-02T18:30:20.569819+0000 2023-11-02T18:30:22.046 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: sync status for source bucket shard: 0 2023-11-02T18:30:22.047 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:entry[obj00000000000000000048]: success 2023-11-02T18:30:22.047 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:8:6 marker=00000000001.56.6 timestamp=2023-11-02T18:30:19.425898+0000 2023-11-02T18:30:22.047 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:entry[obj00000000000000000030]: success 2023-11-02T18:30:22.047 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:11:6 marker=00000000002.72.6 timestamp=2023-11-02T18:30:19.593004+0000 2023-11-02T18:30:22.048 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:entry[obj00000000000000000016]: success 2023-11-02T18:30:22.048 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:7:6 marker=00000000002.63.6 timestamp=2023-11-02T18:30:20.476108+0000 2023-11-02T18:30:22.048 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: sync status for source bucket shard: 0 2023-11-02T18:30:22.048 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]: sync status for source bucket shard: 0 2023-11-02T18:30:22.049 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: sync status for source bucket shard: 0 2023-11-02T18:30:22.049 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]: sync status for source bucket shard: 0 2023-11-02T18:30:22.049 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: sync status for source bucket shard: 0 2023-11-02T18:30:22.049 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: sync status for source bucket shard: 0 2023-11-02T18:30:22.049 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: sync status for source bucket shard: 0 2023-11-02T18:30:22.050 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: sync status for source bucket shard: 0 2023-11-02T18:30:22.050 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:34]: sync status for source bucket shard: 0 2023-11-02T18:30:22.050 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: sync status for source bucket shard: 0 2023-11-02T18:30:22.050 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]: sync status for source bucket shard: 0 2023-11-02T18:30:22.051 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]: sync status for source bucket shard: 0 2023-11-02T18:30:22.051 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:entry[obj00000000000000000060]: success 2023-11-02T18:30:22.051 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:0:6 marker=00000000001.103.6 timestamp=2023-11-02T18:30:20.163612+0000 2023-11-02T18:30:22.051 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:entry[obj00000000000000000026]: success 2023-11-02T18:30:22.052 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:10:6 marker=00000000002.79.6 timestamp=2023-11-02T18:30:19.491264+0000 2023-11-02T18:30:22.052 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.052 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.052 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.054 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:26]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.054 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.054 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:34]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:34]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.054 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.055 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.055 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.055 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-11-02T18:30:22.055 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16/obj00000000000000000012 2023-11-02T18:30:22.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: entry->timestamp=2023-11-02T18:30:20.380421+0000 2023-11-02T18:30:22.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.77.6 op=0 op_state=1 2023-11-02T18:30:22.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: parsed entry: id=00000000002.105.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-11-02T18:30:22.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19/obj00000000000000000045 2023-11-02T18:30:22.057 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: entry->timestamp=2023-11-02T18:30:20.634098+0000 2023-11-02T18:30:22.057 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.105.6 op=0 op_state=1 2023-11-02T18:30:22.057 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-11-02T18:30:22.057 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17/obj00000000000000000002 2023-11-02T18:30:22.058 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: entry->timestamp=2023-11-02T18:30:20.287534+0000 2023-11-02T18:30:22.058 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-11-02T18:30:22.058 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-11-02T18:30:22.058 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15/obj00000000000000000024 2023-11-02T18:30:22.059 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: entry->timestamp=2023-11-02T18:30:19.523008+0000 2023-11-02T18:30:22.059 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-11-02T18:30:22.059 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-11-02T18:30:22.059 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18/obj00000000000000000013 2023-11-02T18:30:22.060 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: entry->timestamp=2023-11-02T18:30:20.664595+0000 2023-11-02T18:30:22.060 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-11-02T18:30:22.060 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000012]: : injecting data sync error on key=obj00000000000000000012 2023-11-02T18:30:22.060 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position=00000000002.77.6 2023-11-02T18:30:22.060 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000045]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000045[0] 2023-11-02T18:30:22.061 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: listing bilog for incremental sync; position=00000000002.105.6 2023-11-02T18:30:22.061 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000002]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000002[0] 2023-11-02T18:30:22.061 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.016+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: listing bilog for incremental sync; position=00000000002.42.6 2023-11-02T18:30:22.061 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.020+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000024]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000024[0] 2023-11-02T18:30:22.062 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.020+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: listing bilog for incremental sync; position=00000000002.57.6 2023-11-02T18:30:22.062 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.020+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:entry[obj00000000000000000006]: success 2023-11-02T18:30:22.062 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.020+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:20:6 marker=00000000002.48.6 timestamp=2023-11-02T18:30:19.621804+0000 2023-11-02T18:30:22.062 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.020+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000013]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000013[0] 2023-11-02T18:30:22.063 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.020+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: listing bilog for incremental sync; position=00000000002.63.6 2023-11-02T18:30:22.063 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.020+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000012]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:22.063 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.020+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000012]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16/obj00000000000000000012 2023-11-02T18:30:22.063 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.020+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:entry[obj00000000000000000009]: success 2023-11-02T18:30:22.064 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.020+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:13:6 marker=00000000002.62.6 timestamp=2023-11-02T18:30:19.201697+0000 2023-11-02T18:30:22.064 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-11-02T18:30:22.064 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32/obj00000000000000000014 2023-11-02T18:30:22.064 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]: entry->timestamp=2023-11-02T18:30:19.907571+0000 2023-11-02T18:30:22.064 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-11-02T18:30:22.065 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]:entry[obj00000000000000000014]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000014[0] 2023-11-02T18:30:22.065 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]: listing bilog for incremental sync; position=00000000002.38.6 2023-11-02T18:30:22.065 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-11-02T18:30:22.065 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28/obj00000000000000000020 2023-11-02T18:30:22.066 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: entry->timestamp=2023-11-02T18:30:20.445572+0000 2023-11-02T18:30:22.066 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-11-02T18:30:22.066 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-11-02T18:30:22.066 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31/obj00000000000000000003 2023-11-02T18:30:22.067 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]: entry->timestamp=2023-11-02T18:30:20.539706+0000 2023-11-02T18:30:22.067 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-11-02T18:30:22.067 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: parsed entry: id=00000000002.143.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-11-02T18:30:22.067 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36/obj00000000000000000008 2023-11-02T18:30:22.068 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: entry->timestamp=2023-11-02T18:30:20.068411+0000 2023-11-02T18:30:22.068 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.143.6 op=0 op_state=1 2023-11-02T18:30:22.068 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:entry[obj00000000000000000020]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000020[0] 2023-11-02T18:30:22.068 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: listing bilog for incremental sync; position=00000000002.36.6 2023-11-02T18:30:22.068 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]:entry[obj00000000000000000003]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000003[0] 2023-11-02T18:30:22.069 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]: listing bilog for incremental sync; position=00000000002.79.6 2023-11-02T18:30:22.069 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:entry[obj00000000000000000008]: : injecting data sync error on key=obj00000000000000000008 2023-11-02T18:30:22.069 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: listing bilog for incremental sync; position=00000000002.143.6 2023-11-02T18:30:22.069 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:entry[obj00000000000000000040]: success 2023-11-02T18:30:22.070 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:14:6 marker=00000000002.64.6 timestamp=2023-11-02T18:30:20.099358+0000 2023-11-02T18:30:22.070 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: parsed entry: id=00000000002.31.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-11-02T18:30:22.070 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24/obj00000000000000000027 2023-11-02T18:30:22.070 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: entry->timestamp=2023-11-02T18:30:19.233387+0000 2023-11-02T18:30:22.070 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.31.6 op=0 op_state=1 2023-11-02T18:30:22.071 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-11-02T18:30:22.071 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25/obj00000000000000000010 2023-11-02T18:30:22.071 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: entry->timestamp=2023-11-02T18:30:19.719315+0000 2023-11-02T18:30:22.071 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.81.6 op=0 op_state=1 2023-11-02T18:30:22.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:entry[obj00000000000000000008]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:30:22.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 0 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:entry[obj00000000000000000008]: ERROR: failed to sync object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36/obj00000000000000000008 2023-11-02T18:30:22.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:entry[obj00000000000000000027]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000027[0] 2023-11-02T18:30:22.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: listing bilog for incremental sync; position=00000000002.31.6 2023-11-02T18:30:22.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000010]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000010[0] 2023-11-02T18:30:22.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: listing bilog for incremental sync; position=00000000002.81.6 2023-11-02T18:30:22.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:entry[obj00000000000000000045]: success 2023-11-02T18:30:22.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.024+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:19:6 marker=00000000002.105.6 timestamp=2023-11-02T18:30:20.634098+0000 2023-11-02T18:30:22.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:entry[obj00000000000000000062]: success 2023-11-02T18:30:22.074 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:12:6 marker=00000000001.59.6 timestamp=2023-11-02T18:30:20.259397+0000 2023-11-02T18:30:22.074 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: parsed entry: id=00000000001.85.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2023-11-02T18:30:22.074 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30/obj00000000000000000036 2023-11-02T18:30:22.074 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: entry->timestamp=2023-11-02T18:30:20.191883+0000 2023-11-02T18:30:22.075 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.85.6 op=0 op_state=1 2023-11-02T18:30:22.075 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:entry[obj00000000000000000036]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000036[0] 2023-11-02T18:30:22.075 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: listing bilog for incremental sync; position=00000000001.85.6 2023-11-02T18:30:22.075 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-11-02T18:30:22.075 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29/obj00000000000000000004 2023-11-02T18:30:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: entry->timestamp=2023-11-02T18:30:20.757289+0000 2023-11-02T18:30:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.78.6 op=0 op_state=1 2023-11-02T18:30:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:entry[obj00000000000000000004]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000004[0] 2023-11-02T18:30:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: listing bilog for incremental sync; position=00000000002.78.6 2023-11-02T18:30:22.077 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:entry[obj00000000000000000002]: success 2023-11-02T18:30:22.077 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.028+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:17:6 marker=00000000002.42.6 timestamp=2023-11-02T18:30:20.287534+0000 2023-11-02T18:30:22.077 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.032+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: a sync operation returned error 2023-11-02T18:30:22.077 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.032+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: backing out with sync_status=-5 2023-11-02T18:30:22.077 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.032+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:22.078 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.032+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:22.078 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.032+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.078 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.032+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: sync status for source bucket shard: 0 2023-11-02T18:30:22.079 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.032+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: parsed entry: id=00000000001.76.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-11-02T18:30:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23/obj00000000000000000039 2023-11-02T18:30:22.082 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: entry->timestamp=2023-11-02T18:30:20.004452+0000 2023-11-02T18:30:22.082 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000001.76.6 op=0 op_state=1 2023-11-02T18:30:22.082 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.048+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000039]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000039[0] 2023-11-02T18:30:22.082 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.048+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: listing bilog for incremental sync; position=00000000001.76.6 2023-11-02T18:30:22.083 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-11-02T18:30:22.083 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21/obj00000000000000000017 2023-11-02T18:30:22.083 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: entry->timestamp=2023-11-02T18:30:20.727086+0000 2023-11-02T18:30:22.083 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-11-02T18:30:22.084 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.056+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:entry[obj00000000000000000017]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000017[0] 2023-11-02T18:30:22.084 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: listing bilog for incremental sync; position=00000000002.61.6 2023-11-02T18:30:22.084 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-11-02T18:30:22.084 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9/obj00000000000000000015 2023-11-02T18:30:22.084 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: entry->timestamp=2023-11-02T18:30:19.972287+0000 2023-11-02T18:30:22.085 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.78.6 op=0 op_state=1 2023-11-02T18:30:22.085 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.056+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000015]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000015[0] 2023-11-02T18:30:22.085 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.056+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: listing bilog for incremental sync; position=00000000002.78.6 2023-11-02T18:30:22.085 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:entry[obj00000000000000000013]: success 2023-11-02T18:30:22.086 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:18:6 marker=00000000002.63.6 timestamp=2023-11-02T18:30:20.664595+0000 2023-11-02T18:30:22.086 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:entry[obj00000000000000000004]: success 2023-11-02T18:30:22.086 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:29:6 marker=00000000002.78.6 timestamp=2023-11-02T18:30:20.757289+0000 2023-11-02T18:30:22.086 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:entry[obj00000000000000000020]: success 2023-11-02T18:30:22.087 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:28:6 marker=00000000002.36.6 timestamp=2023-11-02T18:30:20.445572+0000 2023-11-02T18:30:22.087 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:entry[obj00000000000000000027]: success 2023-11-02T18:30:22.087 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:24:6 marker=00000000002.31.6 timestamp=2023-11-02T18:30:19.233387+0000 2023-11-02T18:30:22.087 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]:entry[obj00000000000000000003]: success 2023-11-02T18:30:22.087 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:31:6 marker=00000000002.79.6 timestamp=2023-11-02T18:30:20.539706+0000 2023-11-02T18:30:22.088 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]:entry[obj00000000000000000014]: success 2023-11-02T18:30:22.088 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:32:6 marker=00000000002.38.6 timestamp=2023-11-02T18:30:19.907571+0000 2023-11-02T18:30:22.088 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:entry[obj00000000000000000024]: success 2023-11-02T18:30:22.088 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:15:6 marker=00000000002.57.6 timestamp=2023-11-02T18:30:19.523008+0000 2023-11-02T18:30:22.089 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:entry[obj00000000000000000010]: success 2023-11-02T18:30:22.089 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.060+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:25:6 marker=00000000002.81.6 timestamp=2023-11-02T18:30:19.719315+0000 2023-11-02T18:30:22.089 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: parsed entry: id=00000000002.101.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2023-11-02T18:30:22.089 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27/obj00000000000000000053 2023-11-02T18:30:22.089 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: entry->timestamp=2023-11-02T18:30:19.300011+0000 2023-11-02T18:30:22.090 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.101.6 op=0 op_state=1 2023-11-02T18:30:22.090 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.064+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:entry[obj00000000000000000017]: success 2023-11-02T18:30:22.090 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:21:6 marker=00000000002.61.6 timestamp=2023-11-02T18:30:20.727086+0000 2023-11-02T18:30:22.090 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.064+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:entry[obj00000000000000000015]: success 2023-11-02T18:30:22.091 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:9:6 marker=00000000002.78.6 timestamp=2023-11-02T18:30:19.972287+0000 2023-11-02T18:30:22.091 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.064+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:entry[obj00000000000000000053]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000053[0] 2023-11-02T18:30:22.091 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.064+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: listing bilog for incremental sync; position=00000000002.101.6 2023-11-02T18:30:22.091 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-11-02T18:30:22.091 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33/obj00000000000000000019 2023-11-02T18:30:22.092 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]: entry->timestamp=2023-11-02T18:30:19.395228+0000 2023-11-02T18:30:22.092 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.77.6 op=0 op_state=1 2023-11-02T18:30:22.092 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.068+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]:entry[obj00000000000000000019]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000019[0] 2023-11-02T18:30:22.092 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]: listing bilog for incremental sync; position=00000000002.77.6 2023-11-02T18:30:22.093 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.068+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:entry[obj00000000000000000036]: success 2023-11-02T18:30:22.093 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.068+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:30:6 marker=00000000001.85.6 timestamp=2023-11-02T18:30:20.191883+0000 2023-11-02T18:30:22.093 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.072+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]:entry[obj00000000000000000019]: success 2023-11-02T18:30:22.093 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.072+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:33:6 marker=00000000002.77.6 timestamp=2023-11-02T18:30:19.395228+0000 2023-11-02T18:30:22.095 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-11-02T18:30:22.095 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35/obj00000000000000000018 2023-11-02T18:30:22.095 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]: entry->timestamp=2023-11-02T18:30:19.811681+0000 2023-11-02T18:30:22.096 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-11-02T18:30:22.096 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]:entry[obj00000000000000000018]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000018[0] 2023-11-02T18:30:22.096 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]: listing bilog for incremental sync; position=00000000002.42.6 2023-11-02T18:30:22.096 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: a sync operation returned error 2023-11-02T18:30:22.097 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: backing out with sync_status=-5 2023-11-02T18:30:22.097 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: incremental sync on bucket failed, retcode=-5 2023-11-02T18:30:22.097 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: ERROR: incremental sync failed. error: -5 2023-11-02T18:30:22.097 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:30:22.098 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: sync status for source bucket shard: 0 2023-11-02T18:30:22.098 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: listing bilog for incremental sync; position= 2023-11-02T18:30:22.098 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]:entry[obj00000000000000000018]: success 2023-11-02T18:30:22.098 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:35:6 marker=00000000002.42.6 timestamp=2023-11-02T18:30:19.811681+0000 2023-11-02T18:30:22.099 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: parsed entry: id=00000000002.143.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-11-02T18:30:22.099 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36/obj00000000000000000008 2023-11-02T18:30:22.099 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: entry->timestamp=2023-11-02T18:30:20.068411+0000 2023-11-02T18:30:22.099 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.143.6 op=0 op_state=1 2023-11-02T18:30:22.099 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:entry[obj00000000000000000008]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000008[0] 2023-11-02T18:30:22.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.076+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: listing bilog for incremental sync; position=00000000002.143.6 2023-11-02T18:30:22.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-11-02T18:30:22.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: syncing object: ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16/obj00000000000000000012 2023-11-02T18:30:22.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: entry->timestamp=2023-11-02T18:30:20.380421+0000 2023-11-02T18:30:22.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=00000000002.77.6 op=0 op_state=1 2023-11-02T18:30:22.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.080+0000 7f8014085f40 5 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000012]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-58[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23])/obj00000000000000000012[0] 2023-11-02T18:30:22.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: listing bilog for incremental sync; position=00000000002.77.6 2023-11-02T18:30:22.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.080+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:entry[obj00000000000000000008]: success 2023-11-02T18:30:22.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:36:6 marker=00000000002.143.6 timestamp=2023-11-02T18:30:20.068411+0000 2023-11-02T18:30:22.102 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.080+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:entry[obj00000000000000000012]: success 2023-11-02T18:30:22.102 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.080+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:16:6 marker=00000000002.77.6 timestamp=2023-11-02T18:30:20.380421+0000 2023-11-02T18:30:22.120 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.112+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:entry[obj00000000000000000039]: success 2023-11-02T18:30:22.120 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.112+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:23:6 marker=00000000001.76.6 timestamp=2023-11-02T18:30:20.004452+0000 2023-11-02T18:30:22.124 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.116+0000 7f8014085f40 10 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:entry[obj00000000000000000053]: success 2023-11-02T18:30:22.125 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:22.116+0000 7f8014085f40 20 RGW-SYNC:bucket[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23<-ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]:inc_sync[ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27]: updating marker marker_oid=bucket.sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23:27:6 marker=00000000002.101.6 timestamp=2023-11-02T18:30:19.300011+0000 2023-11-02T18:30:22.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:22.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:22.166 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:22.254 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:22.255 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:22.321 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:30:22.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.322 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000000", 2023-11-02T18:30:22.322 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.322 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.323 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.323 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 16 2023-11-02T18:30:22.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.323 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.324 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.324 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.324 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.324 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.325 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.783705Z", 2023-11-02T18:30:22.325 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.325 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.325 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.326 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.326 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.326 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.326 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.326 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.327 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.12907161781672385115", 2023-11-02T18:30:22.327 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.327 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.328 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.328 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000001", 2023-11-02T18:30:22.329 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.329 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.329 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.329 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 29 2023-11-02T18:30:22.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.330 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.331 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.331 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.331 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.332 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.332 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.351167Z", 2023-11-02T18:30:22.332 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.332 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.332 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.333 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.333 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.333 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.333 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.334 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.334 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.307152344767932881", 2023-11-02T18:30:22.334 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.335 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.335 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.336 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000002", 2023-11-02T18:30:22.336 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.336 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.336 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.336 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 13 2023-11-02T18:30:22.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.338 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.338 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.338 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.338 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.338 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.339 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.287534Z", 2023-11-02T18:30:22.339 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.339 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.339 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.340 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.340 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.340 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.340 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.340 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.341 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.10958881601919650388", 2023-11-02T18:30:22.341 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.341 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.341 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.342 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000003", 2023-11-02T18:30:22.342 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.343 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.343 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.343 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 31 2023-11-02T18:30:22.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.343 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.344 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.344 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.344 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.344 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.345 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.539706Z", 2023-11-02T18:30:22.345 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.345 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.345 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.346 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.346 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.346 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.346 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.346 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.347 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.3895999745922161574", 2023-11-02T18:30:22.347 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.347 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.348 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.348 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000004", 2023-11-02T18:30:22.348 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.349 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.349 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.349 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 30 2023-11-02T18:30:22.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.350 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.350 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.350 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.350 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.350 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.351 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.757289Z", 2023-11-02T18:30:22.351 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.351 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.351 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.351 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.352 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.352 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.352 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.352 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.353 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.8984847557342740644", 2023-11-02T18:30:22.353 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.353 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.353 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.354 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000005", 2023-11-02T18:30:22.354 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.355 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.355 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.355 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 26 2023-11-02T18:30:22.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.356 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.356 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.356 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.356 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.356 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.357 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.687742Z", 2023-11-02T18:30:22.357 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.357 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.357 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.358 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.358 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.358 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.358 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.358 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.359 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.13360433364422076408", 2023-11-02T18:30:22.359 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.359 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.359 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.360 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000006", 2023-11-02T18:30:22.360 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.361 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.361 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.361 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 11 2023-11-02T18:30:22.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.361 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.362 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.362 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.362 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.362 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.363 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.621804Z", 2023-11-02T18:30:22.363 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.363 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.363 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.363 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.364 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.364 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.364 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.364 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.365 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.9609670545183005513", 2023-11-02T18:30:22.365 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.365 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.365 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.366 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000007", 2023-11-02T18:30:22.366 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.366 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.367 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.367 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 13 2023-11-02T18:30:22.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.367 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.367 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.368 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.368 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.368 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.368 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:17.991996Z", 2023-11-02T18:30:22.369 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.369 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.369 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.369 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.369 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.370 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.370 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.370 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.371 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.3679937152627062558", 2023-11-02T18:30:22.371 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.371 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.371 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.372 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000008", 2023-11-02T18:30:22.372 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.372 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.372 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.373 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 48 2023-11-02T18:30:22.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.373 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.373 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.374 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.374 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.374 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.374 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.068411Z", 2023-11-02T18:30:22.374 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.375 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.375 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.375 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.375 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.376 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.376 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.376 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.376 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.1715397845361088566", 2023-11-02T18:30:22.377 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.377 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.377 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.378 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000009", 2023-11-02T18:30:22.378 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.378 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.379 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.379 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 9 2023-11-02T18:30:22.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.379 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.380 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.380 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.380 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.380 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.380 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.201697Z", 2023-11-02T18:30:22.381 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.381 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.381 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.381 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.381 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.382 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.382 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.382 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.383 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.6941265688380975161", 2023-11-02T18:30:22.383 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.383 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.383 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.384 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000010", 2023-11-02T18:30:22.384 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.384 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.385 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.385 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 12 2023-11-02T18:30:22.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.385 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.385 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.719315Z", 2023-11-02T18:30:22.387 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.387 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.387 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.387 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.387 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.388 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.388 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.388 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.388 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.10956965082320256520", 2023-11-02T18:30:22.389 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.389 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.389 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.390 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000011", 2023-11-02T18:30:22.390 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.390 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.390 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.391 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 28 2023-11-02T18:30:22.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.391 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.391 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.391 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.392 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.392 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.392 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.875656Z", 2023-11-02T18:30:22.392 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.12754584311890405134", 2023-11-02T18:30:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000012", 2023-11-02T18:30:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 18 2023-11-02T18:30:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.398 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.398 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.380421Z", 2023-11-02T18:30:22.398 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.398 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.400 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.400 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.978050869665312329", 2023-11-02T18:30:22.400 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.401 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.401 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.401 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000013", 2023-11-02T18:30:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10 2023-11-02T18:30:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.664595Z", 2023-11-02T18:30:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.405 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.405 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.405 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.405 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.405 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.406 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.5109451804800622697", 2023-11-02T18:30:22.406 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.406 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.406 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.407 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000014", 2023-11-02T18:30:22.407 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.408 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.408 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.408 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 22 2023-11-02T18:30:22.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.409 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.409 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.409 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.409 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.409 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.410 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.907571Z", 2023-11-02T18:30:22.410 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.410 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.410 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.411 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.411 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.411 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.411 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.411 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.412 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.1915131781520781536", 2023-11-02T18:30:22.412 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.412 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.413 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.413 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000015", 2023-11-02T18:30:22.413 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.414 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.414 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.414 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:30:22.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.415 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.415 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.415 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.415 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.416 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.416 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.972287Z", 2023-11-02T18:30:22.416 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.416 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.416 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.417 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.417 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.417 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.417 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.418 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.418 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.3686288967189819359", 2023-11-02T18:30:22.418 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.418 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.419 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.419 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000016", 2023-11-02T18:30:22.420 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.420 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.420 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.420 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 14 2023-11-02T18:30:22.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.421 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.421 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.421 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.421 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.422 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.422 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.476108Z", 2023-11-02T18:30:22.422 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.422 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.422 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.423 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.423 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.423 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.423 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.423 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.424 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.9387376833293109655", 2023-11-02T18:30:22.424 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.424 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.425 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.425 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000017", 2023-11-02T18:30:22.425 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.426 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.426 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.426 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 13 2023-11-02T18:30:22.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.426 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.427 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.427 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.427 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.427 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.427 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.727086Z", 2023-11-02T18:30:22.428 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.428 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.428 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.428 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.429 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.429 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.429 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.429 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.430 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.16072079431348188908", 2023-11-02T18:30:22.430 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.430 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.430 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.430 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.431 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.431 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000018", 2023-11-02T18:30:22.431 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.431 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.432 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.432 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:22.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.432 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.432 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.433 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.433 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.433 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.433 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.811681Z", 2023-11-02T18:30:22.433 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.434 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.434 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.434 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.434 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.434 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.435 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.435 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.435 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.15292506605217580677", 2023-11-02T18:30:22.436 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.436 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.436 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.437 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000019", 2023-11-02T18:30:22.437 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.437 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.437 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.438 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10 2023-11-02T18:30:22.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.438 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.438 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.438 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.439 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.439 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.439 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.395228Z", 2023-11-02T18:30:22.439 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.439 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.440 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.440 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.440 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.440 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.441 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.441 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.441 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.15024328150948317024", 2023-11-02T18:30:22.442 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.442 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.442 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.443 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000020", 2023-11-02T18:30:22.443 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.443 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.443 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 12 2023-11-02T18:30:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.445572Z", 2023-11-02T18:30:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.18113296438450969095", 2023-11-02T18:30:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.448 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.448 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.448 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000021", 2023-11-02T18:30:22.449 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.449 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.449 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.449 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 9 2023-11-02T18:30:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:17.444704Z", 2023-11-02T18:30:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.453 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.453 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.453 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.6306541507966518879", 2023-11-02T18:30:22.453 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000022", 2023-11-02T18:30:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:30:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:17.505219Z", 2023-11-02T18:30:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.461 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.462 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.17784394501153802954", 2023-11-02T18:30:22.462 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.462 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.462 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000023", 2023-11-02T18:30:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.464 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.464 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.464 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10 2023-11-02T18:30:22.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.464 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.465 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.465 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.465 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.466 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.466 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.591898Z", 2023-11-02T18:30:22.466 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.466 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.467 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.467 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.467 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.467 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.467 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.468 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.468 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.8688795702547066493", 2023-11-02T18:30:22.468 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.469 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.469 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.469 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000024", 2023-11-02T18:30:22.470 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.470 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.470 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.470 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:30:22.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.471 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.471 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.471 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.471 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.472 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.472 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.523008Z", 2023-11-02T18:30:22.472 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.472 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.472 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.473 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.473 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.473 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.473 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.473 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.474 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.11876927634551187120", 2023-11-02T18:30:22.474 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.474 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.475 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.475 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000025", 2023-11-02T18:30:22.475 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.476 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.476 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.476 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:22.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.477 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.477 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.477 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.477 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.477 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.478 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:16.702015Z", 2023-11-02T18:30:22.478 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.478 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.478 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.478 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.479 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.479 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.479 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.479 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.480 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.14873424861782094414", 2023-11-02T18:30:22.480 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.480 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.480 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.481 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000026", 2023-11-02T18:30:22.481 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.481 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.482 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.482 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:22.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.482 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.482 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.483 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.483 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.483 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.483 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.491264Z", 2023-11-02T18:30:22.484 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.484 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.484 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.484 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.484 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.485 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.485 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.485 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.486 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.6501247385350275979", 2023-11-02T18:30:22.486 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.486 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.486 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.487 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000027", 2023-11-02T18:30:22.487 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.487 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.487 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.488 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:22.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.488 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.488 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.489 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.489 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.489 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.489 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.233387Z", 2023-11-02T18:30:22.489 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.490 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.490 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.490 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.490 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.491 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.491 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.491 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.491 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.12743543436530738490", 2023-11-02T18:30:22.492 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.492 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.492 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.493 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000028", 2023-11-02T18:30:22.493 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.493 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.493 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.494 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 9 2023-11-02T18:30:22.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.494 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.494 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.494 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.495 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.495 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.495 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:12.533710Z", 2023-11-02T18:30:22.495 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.495 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.496 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.496 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.496 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.496 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.497 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.497 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.497 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.13250346744483344974", 2023-11-02T18:30:22.497 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.498 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.498 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.499 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000029", 2023-11-02T18:30:22.499 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.499 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.499 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.499 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:22.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.500 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.500 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.500 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.500 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.501 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.501 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:17.892795Z", 2023-11-02T18:30:22.501 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.501 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.502 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.502 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.502 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.502 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.503 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.503 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.503 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.1769007943593052798", 2023-11-02T18:30:22.503 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.504 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.504 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.504 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000030", 2023-11-02T18:30:22.505 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.505 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.505 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.505 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 27 2023-11-02T18:30:22.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.506 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.506 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.506 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.506 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.507 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.507 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.593004Z", 2023-11-02T18:30:22.507 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.507 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.507 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.508 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.508 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.508 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.508 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.508 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.509 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.16703805857897588587", 2023-11-02T18:30:22.509 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.509 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.510 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.510 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000031", 2023-11-02T18:30:22.510 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.511 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.511 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.511 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:22.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.511 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.512 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.512 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.512 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.512 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.513 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:16.732710Z", 2023-11-02T18:30:22.513 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.513 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.513 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.513 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.514 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.514 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.514 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.514 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.515 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.6037230522088288486", 2023-11-02T18:30:22.515 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.515 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.515 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.516 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000032", 2023-11-02T18:30:22.516 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.516 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.517 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.517 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:22.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.517 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.517 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.518 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.518 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.518 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.518 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:14.752953Z", 2023-11-02T18:30:22.519 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.519 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.519 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.519 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.519 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.520 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.520 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.520 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.520 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.13271867765654362726", 2023-11-02T18:30:22.521 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.521 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.521 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.522 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000033", 2023-11-02T18:30:22.522 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.522 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.522 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.522 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 8 2023-11-02T18:30:22.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.523 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.523 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.523 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.524 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.524 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.524 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.326434Z", 2023-11-02T18:30:22.524 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.524 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.525 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.525 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.525 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.525 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.525 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.526 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.526 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.17927063960047653246", 2023-11-02T18:30:22.526 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.527 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.527 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.527 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000036", 2023-11-02T18:30:22.528 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.528 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.528 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.528 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:30:22.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.529 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.529 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.529 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.529 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.529 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.530 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.191883Z", 2023-11-02T18:30:22.530 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.530 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.530 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.531 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.531 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.531 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.531 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.531 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.532 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.7039808129721338088", 2023-11-02T18:30:22.532 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.532 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.533 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.533 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000037", 2023-11-02T18:30:22.533 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.534 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.534 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.534 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 9 2023-11-02T18:30:22.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.535 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.535 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.535 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.535 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.535 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.536 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:16.825429Z", 2023-11-02T18:30:22.536 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.536 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.536 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.537 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.537 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.537 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.537 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.537 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.538 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.10280295519325164562", 2023-11-02T18:30:22.538 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.538 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.538 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.539 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000039", 2023-11-02T18:30:22.539 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.540 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.540 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.540 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:22.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.540 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.541 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.541 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.541 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.541 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.541 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.004452Z", 2023-11-02T18:30:22.542 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.542 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.542 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.542 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.542 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.543 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.543 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.543 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.543 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.3299673464172339513", 2023-11-02T18:30:22.544 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.544 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.544 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.545 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000040", 2023-11-02T18:30:22.545 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.545 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.545 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.546 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 14 2023-11-02T18:30:22.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.546 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.546 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.546 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.547 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.547 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.547 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.099358Z", 2023-11-02T18:30:22.547 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.548 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.548 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.548 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.548 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.548 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.549 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.549 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.549 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.18302703362761630287", 2023-11-02T18:30:22.549 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.550 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.550 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.551 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000042", 2023-11-02T18:30:22.551 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.551 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.551 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.551 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 8 2023-11-02T18:30:22.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.552 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.552 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.552 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.553 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.553 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.553 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:16.079205Z", 2023-11-02T18:30:22.553 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.553 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.554 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.554 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.554 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.554 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.554 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.555 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.555 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.1360122938186601168", 2023-11-02T18:30:22.555 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.556 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.556 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.556 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000045", 2023-11-02T18:30:22.557 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.557 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.557 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.557 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 12 2023-11-02T18:30:22.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.558 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.558 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.558 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.558 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.558 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.559 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.634098Z", 2023-11-02T18:30:22.559 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.559 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.559 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.560 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.560 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.560 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.560 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.561 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.561 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.14061203635655037688", 2023-11-02T18:30:22.561 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.561 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.562 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.562 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000048", 2023-11-02T18:30:22.563 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.563 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.563 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.563 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:22.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.564 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.564 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.564 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.564 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.565 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.565 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.425898Z", 2023-11-02T18:30:22.565 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.565 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.566 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.566 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.566 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.566 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.567 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.567 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.567 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.5879173653360132013", 2023-11-02T18:30:22.568 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.568 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.568 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.569 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000049", 2023-11-02T18:30:22.569 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.569 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.569 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.569 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:22.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.570 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.570 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.570 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.571 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.571 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.571 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:16.921273Z", 2023-11-02T18:30:22.571 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.571 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.572 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.572 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.572 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.572 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.573 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.573 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.573 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.8899030020414879190", 2023-11-02T18:30:22.573 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.574 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.574 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.574 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000050", 2023-11-02T18:30:22.575 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.575 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.575 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.575 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:22.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.576 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.576 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.576 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.576 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.577 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.577 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.401770Z", 2023-11-02T18:30:22.577 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.577 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.577 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.578 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.578 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.578 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.578 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.579 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.579 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.13645984590547061931", 2023-11-02T18:30:22.579 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.579 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.580 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.580 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000053", 2023-11-02T18:30:22.580 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.581 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.581 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.581 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10 2023-11-02T18:30:22.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.581 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.582 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.582 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.582 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.582 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.583 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.300011Z", 2023-11-02T18:30:22.583 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.583 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.583 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.583 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.584 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.584 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.584 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.584 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.585 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.15718045727895608668", 2023-11-02T18:30:22.585 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.585 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.585 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.586 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000055", 2023-11-02T18:30:22.586 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.586 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.587 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.587 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:22.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.587 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.587 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.588 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.588 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.588 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.588 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:14.473797Z", 2023-11-02T18:30:22.589 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.589 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.589 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.589 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.589 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.590 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.590 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.590 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.591 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.6072728466205996177", 2023-11-02T18:30:22.591 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.591 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.591 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.592 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000056", 2023-11-02T18:30:22.592 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.592 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.593 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.593 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:22.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.593 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.593 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.594 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.594 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.594 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.594 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.279179Z", 2023-11-02T18:30:22.594 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.595 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.595 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.595 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.595 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.596 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.596 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.596 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.596 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.7329506875011465473", 2023-11-02T18:30:22.597 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.597 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.597 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.598 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000058", 2023-11-02T18:30:22.598 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.598 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.598 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.599 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:22.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.599 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.599 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.600 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.600 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.600 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.600 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:15.190374Z", 2023-11-02T18:30:22.600 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.601 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.601 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.601 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.601 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.601 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.602 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.602 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.602 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.16295956686449722819", 2023-11-02T18:30:22.603 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.603 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.603 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.604 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000059", 2023-11-02T18:30:22.604 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.604 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.604 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.605 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:22.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.605 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.605 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.605 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.606 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.606 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.606 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:16.142390Z", 2023-11-02T18:30:22.606 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.606 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.607 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.607 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.607 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.607 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.608 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.608 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.608 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.6938264603742732761", 2023-11-02T18:30:22.608 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.609 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.609 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.609 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000060", 2023-11-02T18:30:22.610 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.610 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.610 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.610 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 14 2023-11-02T18:30:22.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.611 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.611 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.611 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.611 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.611 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.612 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.163612Z", 2023-11-02T18:30:22.612 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.612 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.612 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.613 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.613 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.613 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.613 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.613 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.614 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.4382434531771929040", 2023-11-02T18:30:22.614 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.614 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.614 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.615 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000061", 2023-11-02T18:30:22.615 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.615 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.616 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.616 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:22.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.616 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.617 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.617 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.617 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.617 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.617 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.562805Z", 2023-11-02T18:30:22.618 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.618 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.618 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.618 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.618 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.619 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.619 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.619 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.620 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.15442554514888316627", 2023-11-02T18:30:22.620 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.620 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.620 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.621 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000062", 2023-11-02T18:30:22.621 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.621 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.621 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.622 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:22.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.622 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.622 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.623 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.623 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.623 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.623 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.259397Z", 2023-11-02T18:30:22.623 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.624 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.624 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.624 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.624 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.625 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.625 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.625 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.625 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.5524641109174791244", 2023-11-02T18:30:22.626 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.626 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.626 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.627 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000065", 2023-11-02T18:30:22.627 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.627 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.627 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.628 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:30:22.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.628 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.628 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.628 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.629 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.629 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.629 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.569819Z", 2023-11-02T18:30:22.629 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.630 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.630 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.630 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.630 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.630 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.631 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.631 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.631 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.10267934017086837008", 2023-11-02T18:30:22.632 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.632 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.632 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.633 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000076", 2023-11-02T18:30:22.633 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.633 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.633 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.633 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:12.172157Z", 2023-11-02T18:30:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.13769370923459597048", 2023-11-02T18:30:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000133", 2023-11-02T18:30:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:30:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.467373Z", 2023-11-02T18:30:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.6035757957405060567", 2023-11-02T18:30:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000162", 2023-11-02T18:30:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1 2023-11-02T18:30:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.646 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.646 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.646 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.646 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.646 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.647 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.306794Z", 2023-11-02T18:30:22.647 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.647 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.647 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.648 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.648 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.648 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.648 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.648 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.649 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.1892578020939347501", 2023-11-02T18:30:22.649 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.649 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.650 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.650 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000271", 2023-11-02T18:30:22.650 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.651 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.651 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.651 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:22.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.652 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.652 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.652 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.652 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.652 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.653 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.687164Z", 2023-11-02T18:30:22.653 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.653 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.653 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.654 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.654 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.654 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.654 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.655 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.655 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.9019973165684389936", 2023-11-02T18:30:22.655 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.655 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.656 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.656 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000283", 2023-11-02T18:30:22.656 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.657 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.657 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.657 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:22.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.658 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.658 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.658 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.658 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.658 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.659 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:17.535381Z", 2023-11-02T18:30:22.659 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.659 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.659 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.659 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.660 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.660 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.660 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.660 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.661 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.123442099204298621", 2023-11-02T18:30:22.661 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.661 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.661 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.662 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000317", 2023-11-02T18:30:22.662 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.663 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.663 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.663 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10 2023-11-02T18:30:22.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.663 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.664 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.664 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.664 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.664 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.665 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.119583Z", 2023-11-02T18:30:22.665 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.665 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.665 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.665 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.666 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.666 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.666 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.666 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.667 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.778288322357084040", 2023-11-02T18:30:22.667 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.667 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.667 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.668 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000423", 2023-11-02T18:30:22.668 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.669 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.669 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.669 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:22.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.669 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.670 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.670 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.670 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.670 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.670 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.090986Z", 2023-11-02T18:30:22.671 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.671 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.671 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.671 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.672 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.672 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.672 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.672 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.673 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.13658135605528488690", 2023-11-02T18:30:22.673 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.673 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.674 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.675 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000467", 2023-11-02T18:30:22.675 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.676 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.676 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:30:22.676 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10 2023-11-02T18:30:22.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.676 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.677 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.677 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.677 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.677 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.677 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:17.598529Z", 2023-11-02T18:30:22.678 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.678 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.678 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.678 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.679 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.679 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.679 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.679 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.680 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.15081321984039241008", 2023-11-02T18:30:22.680 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.680 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.680 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.681 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:22.681 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:30:22.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:22.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:22.683 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:22.727 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:22.727 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:22.784 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:30:22.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.785 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000000", 2023-11-02T18:30:22.785 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.785 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.786 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.786 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:22.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.786 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.787 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.787 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.787 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.787 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.787 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.783705Z", 2023-11-02T18:30:22.788 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.788 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.788 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.788 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.789 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.789 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.789 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.789 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.790 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_xvF_y4MTwMGpgfLzRaT_AI1bSnpPkX7", 2023-11-02T18:30:22.790 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.790 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.790 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.791 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000001", 2023-11-02T18:30:22.791 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.791 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.792 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.792 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:30:22.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.792 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.792 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.793 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.793 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.793 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.793 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.351167Z", 2023-11-02T18:30:22.794 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.794 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.794 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.794 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.794 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.795 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.795 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.795 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.795 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_XpNuVhvoXnVsPMU-VLcF40QRLVmJM_a", 2023-11-02T18:30:22.796 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.796 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.796 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.797 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000002", 2023-11-02T18:30:22.797 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.797 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.797 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.798 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:22.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.798 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.798 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.799 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.799 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.799 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.799 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.287534Z", 2023-11-02T18:30:22.799 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.800 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.800 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.800 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.800 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.800 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.801 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.801 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.801 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_FOR9gAzUGormk4dtAaQkwcOoaA_6cSQ", 2023-11-02T18:30:22.801 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.802 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.802 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.803 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000003", 2023-11-02T18:30:22.803 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.803 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.803 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.803 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 8 2023-11-02T18:30:22.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.804 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.804 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.804 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.804 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.805 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.805 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.539706Z", 2023-11-02T18:30:22.805 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.805 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.806 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.806 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.806 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.806 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.806 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.807 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.807 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_1Ag5ULuyjot_asoAV-EOJJncvUcUlmX", 2023-11-02T18:30:22.807 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.808 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.808 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.808 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000004", 2023-11-02T18:30:22.809 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.809 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.809 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.809 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:22.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.810 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.810 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.810 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.810 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.811 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.811 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.757289Z", 2023-11-02T18:30:22.811 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.811 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.811 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.812 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.812 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.812 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.812 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.813 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.813 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_FXKkJu6Dq9QcUHhXdOM_E7KmLvkNJXz", 2023-11-02T18:30:22.813 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.813 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.814 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.814 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000005", 2023-11-02T18:30:22.814 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.815 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.815 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.815 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:30:22.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.816 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.816 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.816 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.816 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.816 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.817 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.687742Z", 2023-11-02T18:30:22.817 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.817 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.818 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.818 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.818 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.818 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.818 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.819 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.819 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_ElU8eNmfMviwC1spI6ydtVH0ggXuDm2", 2023-11-02T18:30:22.819 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.820 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.820 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.820 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000006", 2023-11-02T18:30:22.821 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.821 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.821 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.821 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:22.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.822 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.822 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.822 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.822 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.822 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.823 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.621804Z", 2023-11-02T18:30:22.823 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.823 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.823 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.824 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.824 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.824 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.824 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.824 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.825 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_4DDisGF_b67Nuas6ipWWIzinv6GI2QE", 2023-11-02T18:30:22.825 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.825 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.826 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.826 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000007", 2023-11-02T18:30:22.826 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.827 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.827 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.827 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:22.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.827 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.828 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.828 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.828 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.828 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.829 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:17.991996Z", 2023-11-02T18:30:22.829 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.829 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.829 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.829 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.830 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.830 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.830 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.830 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.831 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_R5kgxcDpR5QryObgxUCXRtZDnB_mXKP", 2023-11-02T18:30:22.831 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.831 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.831 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.832 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000008", 2023-11-02T18:30:22.832 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.832 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.833 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.833 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 41 2023-11-02T18:30:22.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.833 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.833 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.834 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.834 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.834 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.834 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.068411Z", 2023-11-02T18:30:22.835 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.835 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.835 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.835 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.835 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.836 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.836 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.836 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.836 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_0m7TdD6ful4jBBJzB1bMvOeYK-izcgS", 2023-11-02T18:30:22.837 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.837 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.838 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000009", 2023-11-02T18:30:22.838 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.838 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.838 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.839 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1 2023-11-02T18:30:22.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.839 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.839 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.839 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.840 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.840 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.840 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.201697Z", 2023-11-02T18:30:22.840 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.841 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.841 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.841 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.841 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.841 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.842 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.842 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.842 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_-iB5CSDL_EBdAIIciDFYa6u3MJkkmxM", 2023-11-02T18:30:22.843 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.843 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.843 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.844 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000010", 2023-11-02T18:30:22.844 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.844 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.844 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.844 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:22.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.845 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.845 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.845 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.846 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.846 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.846 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.719315Z", 2023-11-02T18:30:22.846 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.846 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.847 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.847 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.847 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.847 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.848 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.848 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.848 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_TmJJ7N4jYFy2T1A3mhMLt68DnxfpyyZ", 2023-11-02T18:30:22.848 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.849 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.849 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.849 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000011", 2023-11-02T18:30:22.850 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.850 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.850 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.850 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:30:22.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.851 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.851 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.851 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.851 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.852 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.852 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.875656Z", 2023-11-02T18:30:22.852 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.852 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.852 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.853 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.853 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.853 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.853 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.854 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.854 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_yif7HEsuVJVeNoN54PXNmq3XTgyjiAF", 2023-11-02T18:30:22.854 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.854 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.855 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.855 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000012", 2023-11-02T18:30:22.856 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.856 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.856 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.856 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 15 2023-11-02T18:30:22.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.857 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.857 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.857 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.857 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.857 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.858 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.380421Z", 2023-11-02T18:30:22.858 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.858 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.858 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.859 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.859 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.859 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.859 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.859 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.860 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_gJlUrmwforYmktq6iA1qi71faf_0Imm", 2023-11-02T18:30:22.860 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.860 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.861 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.861 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000013", 2023-11-02T18:30:22.861 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.862 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.862 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.862 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:22.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.862 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.863 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.863 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.863 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.863 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.864 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.664595Z", 2023-11-02T18:30:22.864 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.864 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.864 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.864 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.865 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.865 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.865 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.865 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.866 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_UIvFke1yIhRXcENaDAHYT8dzWa1VEun", 2023-11-02T18:30:22.866 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.866 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.866 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.867 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000014", 2023-11-02T18:30:22.867 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.867 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.868 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.868 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:30:22.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.868 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.869 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.869 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.869 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.869 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.869 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.907571Z", 2023-11-02T18:30:22.870 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.870 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.870 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.870 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.870 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.871 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.871 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.871 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.872 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_3y7F97vk0aKJtvFPzexQzJyoZAyxgtq", 2023-11-02T18:30:22.872 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.872 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.872 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.873 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000015", 2023-11-02T18:30:22.873 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.873 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.874 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.874 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:22.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.874 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.874 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.875 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.875 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.875 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.875 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.972287Z", 2023-11-02T18:30:22.876 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.876 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.876 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.876 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.877 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.877 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.877 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.877 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.878 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_kmmkb_sOZPgVqX5iEfm8ZKHxp_7at_h", 2023-11-02T18:30:22.878 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.878 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.878 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.879 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000016", 2023-11-02T18:30:22.879 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.879 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.880 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.880 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:22.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.880 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.880 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.881 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.881 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.881 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.881 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.476108Z", 2023-11-02T18:30:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.883 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.883 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.883 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.883 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_RQgfAqsziC_qJa6C3IUVbDQNgo_9OSW", 2023-11-02T18:30:22.884 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.884 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.884 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.885 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000017", 2023-11-02T18:30:22.885 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.885 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.885 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.886 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:22.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.886 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.886 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.886 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.887 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.887 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.887 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.727086Z", 2023-11-02T18:30:22.887 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.888 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.888 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.888 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.888 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.888 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.889 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.889 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.889 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_O56pbhL2l9scijAY_wVyGhCdLN7AX2v", 2023-11-02T18:30:22.889 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.890 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.890 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.891 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000018", 2023-11-02T18:30:22.891 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.891 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.891 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.891 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:22.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.892 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.892 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.892 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.892 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.893 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.893 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.811681Z", 2023-11-02T18:30:22.893 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.893 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.893 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.894 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.894 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.894 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.894 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.895 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.895 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_QreIQE7KKTMZqQfTTamzl-wkGtut7ko", 2023-11-02T18:30:22.895 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.895 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.896 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.896 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000019", 2023-11-02T18:30:22.897 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.897 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.897 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.897 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1 2023-11-02T18:30:22.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.898 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.898 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.898 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.898 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.898 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.899 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.395228Z", 2023-11-02T18:30:22.899 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.899 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.899 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.900 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.900 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.900 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.900 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.900 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.901 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_bNoThMkFhDtTGH5wJHCkXQjXKQJWGMG", 2023-11-02T18:30:22.901 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.901 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.901 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.902 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000020", 2023-11-02T18:30:22.902 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.902 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.903 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.903 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:22.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.903 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.904 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.904 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.904 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.904 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.904 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.445572Z", 2023-11-02T18:30:22.905 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.905 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.905 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.905 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.905 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.906 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.906 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.906 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.907 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_5Gno81M526HyEa_5d-poBC_-CD8ZMtp", 2023-11-02T18:30:22.907 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.907 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.907 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.908 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000021", 2023-11-02T18:30:22.908 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.908 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.909 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.909 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:22.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.909 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.909 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.910 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.910 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.910 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.910 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:17.444704Z", 2023-11-02T18:30:22.911 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.911 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.911 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.911 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.911 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.912 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.912 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.912 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.912 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_w_TA0qZWo88mI_L-OiH6lJWxr_XnRuc", 2023-11-02T18:30:22.913 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.913 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.913 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.914 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000022", 2023-11-02T18:30:22.914 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.914 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.914 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.915 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:30:22.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.915 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.915 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.915 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.916 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.916 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.916 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:17.505219Z", 2023-11-02T18:30:22.916 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.916 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.917 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.917 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.917 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.917 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.918 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.918 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.918 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_IlzT7u-AmzxXeBTepEMIjYxC83drFyg", 2023-11-02T18:30:22.918 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.919 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.919 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.920 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000023", 2023-11-02T18:30:22.920 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.920 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.920 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.920 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:30:22.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.921 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.921 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.921 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.921 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.922 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.922 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.591898Z", 2023-11-02T18:30:22.922 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.922 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.923 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.923 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.923 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.923 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.923 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.924 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.924 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_qNsDZaeePR-DhvUlxTBKBsCcsPOi96f", 2023-11-02T18:30:22.924 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.924 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.925 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.925 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000024", 2023-11-02T18:30:22.926 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.926 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.926 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.926 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:22.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.927 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.927 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.927 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.927 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.928 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:22.928 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.523008Z", 2023-11-02T18:30:22.928 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.928 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.929 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.929 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.929 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.929 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.929 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.930 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.930 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_AlKROX8m0FC5C6cUfkleCkODWoL6uUn", 2023-11-02T18:30:22.930 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.930 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.931 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.931 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000025", 2023-11-02T18:30:22.932 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.932 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.932 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.932 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:22.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.933 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.933 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.933 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.933 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.934 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.934 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:16.702015Z", 2023-11-02T18:30:22.934 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.934 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.935 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.935 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.935 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.935 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.935 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.936 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.936 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_QYi1e6TSNqWdO7P0yU2HT-PfkQ31cC_", 2023-11-02T18:30:22.936 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.937 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.937 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.938 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000026", 2023-11-02T18:30:22.938 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.938 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.938 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.938 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:22.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.939 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.939 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.939 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.939 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.940 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.940 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.491264Z", 2023-11-02T18:30:22.940 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.940 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.941 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.941 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.941 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.941 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.941 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.942 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.942 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_-KTzg5STp7KifWjNNtoxRc6vHoYRudk", 2023-11-02T18:30:22.942 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.943 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.943 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.943 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000027", 2023-11-02T18:30:22.944 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.944 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.944 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.944 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:22.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.945 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.945 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.945 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.945 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.945 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.946 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.233387Z", 2023-11-02T18:30:22.946 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.946 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.946 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.947 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.947 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.947 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.947 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.947 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.948 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_0ZV4xZCsjfg5OG40SibEFBLf-r1dnbO", 2023-11-02T18:30:22.948 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.948 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.949 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.949 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000028", 2023-11-02T18:30:22.949 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.950 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.950 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.950 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:30:22.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.950 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.951 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.951 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.951 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.951 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.952 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:12.533710Z", 2023-11-02T18:30:22.952 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.952 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.952 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.952 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.953 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.953 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.953 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.953 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.954 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_WZJH6R18KRpZb31oD2OcOotU8IpvJ2E", 2023-11-02T18:30:22.954 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.954 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.954 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.955 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000029", 2023-11-02T18:30:22.955 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.955 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.956 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.956 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:22.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.956 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.956 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.957 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.957 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.957 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.957 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:17.892795Z", 2023-11-02T18:30:22.958 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.958 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.958 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.958 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.958 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.959 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.959 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.959 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.960 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_mMHfgPThdFs75fWBHiFBriuVxcaYlv8", 2023-11-02T18:30:22.960 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.960 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.960 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.961 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000030", 2023-11-02T18:30:22.961 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.961 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.962 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.962 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:30:22.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.962 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.962 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.963 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.963 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.963 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:22.963 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.593004Z", 2023-11-02T18:30:22.964 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.964 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.964 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.964 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.964 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.965 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.965 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.965 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.965 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_e0JX6tEcXt-WJwgXdAnXpPUJXVjWfo_", 2023-11-02T18:30:22.966 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.966 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.966 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.967 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000031", 2023-11-02T18:30:22.967 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.967 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.967 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.968 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:22.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.968 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.968 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.968 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.969 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.969 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.969 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:16.732710Z", 2023-11-02T18:30:22.969 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.970 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.970 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.970 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.970 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.970 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.971 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.971 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.971 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_O_qlv5niAJVmUEN-sxwBBGTnBpD4Ols", 2023-11-02T18:30:22.972 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.972 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.972 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.973 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000032", 2023-11-02T18:30:22.973 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.973 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.973 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.973 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:22.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.974 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.974 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.974 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.975 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.975 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.975 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:14.752953Z", 2023-11-02T18:30:22.975 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.975 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.976 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.976 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.976 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.976 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.976 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.977 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.977 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_16IAKVlZNJWcbKsD2fbf1p8Qx6-Fxv9", 2023-11-02T18:30:22.977 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.978 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.978 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.979 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000033", 2023-11-02T18:30:22.979 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.979 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.979 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.979 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:22.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.980 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.980 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.980 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.980 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.981 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.981 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.326434Z", 2023-11-02T18:30:22.981 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.981 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.982 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.982 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.982 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.982 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.982 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.983 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.983 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_vKg4qK-ow9x7Coi42NOqXkiSYsJb0xY", 2023-11-02T18:30:22.983 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.983 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.984 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.984 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000036", 2023-11-02T18:30:22.985 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.985 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.985 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.985 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:22.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.986 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.986 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.987 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.987 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.987 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:22.987 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.191883Z", 2023-11-02T18:30:22.988 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.988 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.988 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.988 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.989 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.989 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.989 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.990 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.990 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_RSZ7CtPVkgDjDA1asALPTkDHBO1oQFl", 2023-11-02T18:30:22.990 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.991 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.991 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.992 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000037", 2023-11-02T18:30:22.992 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:22.992 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:22.993 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:22.993 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:30:22.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.993 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:22.994 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:22.994 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:22.994 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:22.995 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:22.995 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:16.825429Z", 2023-11-02T18:30:22.995 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:22.995 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:22.996 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:22.996 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:22.996 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:22.996 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:22.997 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:22.997 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:22.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.997 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_XsBleDWR1zUm7KT4TpVBUwyvqQW_n7f", 2023-11-02T18:30:22.998 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:22.998 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:22.998 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:22.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:22.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:22.999 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000039", 2023-11-02T18:30:22.999 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.000 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.000 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.000 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:23.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.001 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.001 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.001 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.002 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.002 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:23.002 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.004452Z", 2023-11-02T18:30:23.002 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.002 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.003 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.003 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.003 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.003 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.003 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.004 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.004 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_U60XtMxupUtfFp43jTcvs7hYq8Z4hOt", 2023-11-02T18:30:23.004 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.005 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.005 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.005 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000040", 2023-11-02T18:30:23.006 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.006 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.006 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.006 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:23.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.007 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.007 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.007 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.007 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.008 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:23.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.099358Z", 2023-11-02T18:30:23.008 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.008 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.008 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.009 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.009 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.009 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.009 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.010 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.010 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_o0OCQNSeYvkYUaxd_dqsCoWpX21IfsR", 2023-11-02T18:30:23.010 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.011 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.011 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.011 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000042", 2023-11-02T18:30:23.012 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.012 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.012 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.012 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10 2023-11-02T18:30:23.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.013 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.013 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.013 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.013 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.014 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:23.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:16.079205Z", 2023-11-02T18:30:23.014 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.014 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.015 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.015 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.015 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.015 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.015 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.016 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.016 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_Gqa2x_OoqNGvyF2Rtd3VmNw2ERkN-gA", 2023-11-02T18:30:23.016 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.016 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.017 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.017 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000045", 2023-11-02T18:30:23.017 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.018 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.018 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.018 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:23.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.019 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.019 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.019 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.019 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.019 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:23.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.634098Z", 2023-11-02T18:30:23.020 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.020 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.020 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.020 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.021 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.021 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.021 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.021 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.022 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_pmff-T2mg6WVSRzZEYJh6E7pUF1qmMC", 2023-11-02T18:30:23.022 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.022 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.022 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.023 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000048", 2023-11-02T18:30:23.023 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.023 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.024 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.024 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:23.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.024 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.025 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.025 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.025 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.025 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:23.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.425898Z", 2023-11-02T18:30:23.026 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.026 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.026 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.026 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.026 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.027 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.027 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.027 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.028 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_wB-MQnYkuOV19MRdG_To4-_Z15kpHQF", 2023-11-02T18:30:23.028 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.028 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.028 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.029 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000049", 2023-11-02T18:30:23.029 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.029 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.030 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.030 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:23.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.030 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.030 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.031 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.031 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.031 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:23.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:16.921273Z", 2023-11-02T18:30:23.032 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.032 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.032 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.032 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.032 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.033 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.033 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.033 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.033 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_EEy_uOkyOOIy-lxFEc6S1G8ZvQ_eyQV", 2023-11-02T18:30:23.034 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.034 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.034 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.035 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000050", 2023-11-02T18:30:23.035 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.035 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.035 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.036 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:23.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.036 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.036 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.037 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.037 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.037 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:23.037 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.401770Z", 2023-11-02T18:30:23.037 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.038 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.038 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.038 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.038 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.038 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.039 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.039 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.039 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_uosGTOdUAAcfn6bu5Q9VHk1IlGTbCTY", 2023-11-02T18:30:23.040 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.040 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.040 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.041 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000053", 2023-11-02T18:30:23.041 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.041 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.042 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:30:23.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.042 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.042 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.042 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.043 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.043 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:23.043 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:19.300011Z", 2023-11-02T18:30:23.043 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.044 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.044 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.044 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.044 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.044 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.045 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.045 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.045 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_qomuQ_RNCASy4A_cyZ7gT_EZZ6KZQJh", 2023-11-02T18:30:23.046 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.046 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.046 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.047 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000055", 2023-11-02T18:30:23.047 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.047 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.047 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.047 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:23.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.048 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.048 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.048 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.049 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.049 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:23.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:14.473797Z", 2023-11-02T18:30:23.049 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.049 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.050 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.050 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.050 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.050 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.051 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.051 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.051 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_ojdriMsv281B8-x3qQNTfMs-fuDeRRk", 2023-11-02T18:30:23.052 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.052 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.052 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.053 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000056", 2023-11-02T18:30:23.053 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.053 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.053 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.053 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:23.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.054 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.054 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.054 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.055 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.055 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:23.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.279179Z", 2023-11-02T18:30:23.055 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.055 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.056 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.056 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.056 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.056 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.056 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.057 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.057 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_sCY3W120tMCdzc4Ld8CmNOPbdGpxkIE", 2023-11-02T18:30:23.057 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.058 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.058 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.058 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000058", 2023-11-02T18:30:23.059 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.059 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.059 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.059 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:23.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.060 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.060 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.060 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.060 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.061 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:23.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:15.190374Z", 2023-11-02T18:30:23.061 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.061 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.061 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.062 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.062 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.062 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.062 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.063 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.063 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_JQDDmH3O3CtrsymPb2OPKkiOGnJuJme", 2023-11-02T18:30:23.063 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.063 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.064 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.064 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000059", 2023-11-02T18:30:23.064 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.065 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.065 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.065 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:23.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.065 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.066 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.066 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.066 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.066 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:23.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:16.142390Z", 2023-11-02T18:30:23.067 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.067 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.067 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.067 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.068 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.068 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.068 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.068 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.069 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_wRVzgJaBIizmjgGJoNzb8nlNVqtRCI8", 2023-11-02T18:30:23.069 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.069 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.069 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.070 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000060", 2023-11-02T18:30:23.070 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.070 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.071 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.071 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10 2023-11-02T18:30:23.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.071 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.071 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.072 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.072 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.072 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:23.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.163612Z", 2023-11-02T18:30:23.072 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.073 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.073 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.073 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.073 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.074 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.074 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.074 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.074 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_8bZFLqUCyS2kwd7HHT1vJqcyPpmC7tr", 2023-11-02T18:30:23.075 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.075 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.075 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.076 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000061", 2023-11-02T18:30:23.076 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.076 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.076 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.077 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:23.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.077 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.077 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.077 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.078 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.078 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:23.078 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.562805Z", 2023-11-02T18:30:23.078 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.079 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.079 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.079 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.079 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.079 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.080 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.080 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.080 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_qeuAA2Dza15PAMvVdo2FAf-YgBlw3Zm", 2023-11-02T18:30:23.081 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.081 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.081 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.082 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000062", 2023-11-02T18:30:23.082 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.082 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.082 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:23.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.083 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.083 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.083 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.084 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.084 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:23.084 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.259397Z", 2023-11-02T18:30:23.084 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.084 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.085 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.085 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.085 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.085 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.086 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.086 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.086 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_8i5h3-e7TUn7aUJnHgXDr98VjV2MlK9", 2023-11-02T18:30:23.086 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.087 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.087 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.088 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000065", 2023-11-02T18:30:23.088 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.088 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.088 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.088 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:23.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.089 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.089 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.089 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.089 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.090 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:23.090 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:20.569819Z", 2023-11-02T18:30:23.090 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.090 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.091 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.091 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.091 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.091 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.091 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.092 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.092 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_iHOolzWe8pQJRV18AbbXqVZCqC0okXj", 2023-11-02T18:30:23.092 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.092 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.093 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.093 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000076", 2023-11-02T18:30:23.094 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.094 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.094 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.094 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:23.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.095 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.095 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.095 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.095 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.096 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:23.096 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:12.172157Z", 2023-11-02T18:30:23.096 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.096 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.096 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.097 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.097 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.097 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.097 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.098 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.098 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_6UQZbbLiHGXMQc_t10IoOa9w-2PhhBc", 2023-11-02T18:30:23.098 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.133 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.133 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.134 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000133", 2023-11-02T18:30:23.134 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.135 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.135 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.135 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:30:23.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.135 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.136 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.136 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.136 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.136 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:23.137 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.467373Z", 2023-11-02T18:30:23.137 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.137 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.137 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.137 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.138 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.138 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.138 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.141 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.142 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_azO_oKR-wZH3p9QNSTq8p8ZeHUwha7D", 2023-11-02T18:30:23.142 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.142 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.142 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.143 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000162", 2023-11-02T18:30:23.143 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.143 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.143 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.144 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1 2023-11-02T18:30:23.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.144 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.144 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.145 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.145 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.145 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:23.145 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.306794Z", 2023-11-02T18:30:23.145 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.146 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.146 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.146 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.146 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.147 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.147 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.147 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.147 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_icqPMyThh8MOxlpw4TbT8SNKsUMVYai", 2023-11-02T18:30:23.148 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.148 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.148 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.149 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000271", 2023-11-02T18:30:23.149 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.149 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.149 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.149 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:30:23.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.150 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.150 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.150 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.151 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.151 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:23.151 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.687164Z", 2023-11-02T18:30:23.151 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.151 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.152 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.152 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.152 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.152 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.153 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.153 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.153 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_Kwo0t6haTK7O2e0Uv6pMI6pFUhWaHhR", 2023-11-02T18:30:23.153 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.154 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.154 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.156 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000283", 2023-11-02T18:30:23.156 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.156 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.156 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.156 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:23.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.157 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.157 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.157 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.158 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.158 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:30:23.158 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:17.535381Z", 2023-11-02T18:30:23.158 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.158 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.159 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.159 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.159 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.159 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.160 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.160 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.160 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_vC_c0-y4Ircc7IvcvVfs1TJrRc8vm3m", 2023-11-02T18:30:23.160 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.161 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.161 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.161 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000317", 2023-11-02T18:30:23.162 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.162 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.162 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.162 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 8 2023-11-02T18:30:23.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.163 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.163 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.163 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.163 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.164 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:30:23.164 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.119583Z", 2023-11-02T18:30:23.164 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.164 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.164 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.165 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.165 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.165 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.165 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.166 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.166 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_eqoGY8ZLzCBPsQGlfzQ_gTdf127n1Yi", 2023-11-02T18:30:23.166 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.166 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.167 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.167 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000423", 2023-11-02T18:30:23.167 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.168 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.168 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.168 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:30:23.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.169 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.169 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.169 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.169 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.169 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:30:23.170 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:18.090986Z", 2023-11-02T18:30:23.170 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.170 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.170 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.171 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.171 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.171 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.171 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.171 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.172 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_kXG1OTI5o7DxI7ssm1G5LgFd3o_g4xR", 2023-11-02T18:30:23.172 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.172 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:23.173 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj00000000000000000467", 2023-11-02T18:30:23.173 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:30:23.173 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:30:23.174 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:30:23.174 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:30:23.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.174 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:30:23.174 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:30:23.175 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:30:23.175 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:30:23.175 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:30:23.175 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:30:17.598529Z", 2023-11-02T18:30:23.176 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:30:23.176 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:30:23.176 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:30:23.176 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:30:23.176 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:30:23.177 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:30:23.177 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:30:23.177 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:30:23.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:23.177 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_DOf7fAokmKXGhsuuQX_k3B38mCpUH_Y", 2023-11-02T18:30:23.178 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:30:23.178 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:30:23.178 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:30:23.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:23.178 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:30:23.179 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-11-02T18:30:23.180 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-11-02T18:30:23.183 INFO:tasks.rgw.c1.client.1:Started 2023-11-02T18:30:23.183 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-11-02T18:30:23.202 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T18:30:23.202 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T18:30:23.202 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-11-02T18:30:23.203 INFO:teuthology.orchestra.run.smithi017.stderr:curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8001: Connection refused 2023-11-02T18:30:24.204 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:24.223 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T18:30:24.223 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T18:30:24.225 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-11-02T18:30:24.226 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2023-11-02T18:30:24.226 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-58 2023-11-02T18:30:24.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:30:24.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:30:24.227 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:30:24.281 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:24.281 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:24.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:24.344+0000 7f304b75ff40 1 bucket sync caught up with source: 2023-11-02T18:30:24.350 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.103.6, 00000000002.83.6, 00000000002.61.6, 00000000002.74.6, 00000000002.54.6, 00000000001.41.6, 00000000002.79.6, 00000000002.63.6, 00000000001.56.6, 00000000002.78.6, 00000000002.79.6, 00000000002.72.6, 00000000001.59.6, 00000000002.62.6, 00000000002.64.6, 00000000002.57.6, 00000000002.77.6, 00000000002.42.6, 00000000002.63.6, 00000000002.105.6, 00000000002.48.6, 00000000002.61.6, , 00000000001.76.6, 00000000002.31.6, 00000000002.81.6, , 00000000002.101.6, 00000000002.36.6, 00000000002.78.6, 00000000001.85.6, 00000000002.79.6, 00000000002.38.6, 00000000002.77.6, , 00000000002.42.6, 00000000002.143.6] 2023-11-02T18:30:24.350 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.103.6, 00000000002.83.6, 00000000002.61.6, 00000000002.74.6, 00000000002.54.6, 00000000001.41.6, 00000000002.79.6, 00000000002.63.6, 00000000001.56.6, 00000000002.78.6, 00000000002.79.6, 00000000002.72.6, 00000000001.59.6, 00000000002.62.6, 00000000002.64.6, 00000000002.57.6, 00000000002.77.6, 00000000002.42.6, 00000000002.63.6, 00000000002.105.6, 00000000002.48.6, 00000000002.61.6, , 00000000001.76.6, 00000000002.31.6, 00000000002.81.6, , 00000000002.101.6, 00000000002.36.6, 00000000002.78.6, 00000000001.85.6, 00000000002.79.6, 00000000002.38.6, 00000000002.77.6, , 00000000002.42.6, 00000000002.143.6] 2023-11-02T18:30:24.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:24.344+0000 7f304b75ff40 0 bucket checkpoint complete 2023-11-02T18:30:24.374 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-58 2023-11-02T18:30:24.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:30:24.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:30:24.375 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:30:24.426 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:24.427 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:24.492 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:24.484+0000 7f123e934f40 1 bucket sync caught up with source: 2023-11-02T18:30:24.493 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000004.112.6, 00000000007.242.6, 00000000006.118.6, 00000000010.223.6, 00000000005.264.6, 00000000008.67.6, 00000000008.453.6, 00000000005.204.6, 00000000006.265.6, 00000000007.454.6, 00000000004.245.6] 2023-11-02T18:30:24.493 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000004.112.6, 00000000007.242.6, 00000000006.118.6, 00000000010.223.6, 00000000005.264.6, 00000000008.67.6, 00000000008.453.6, 00000000005.204.6, 00000000006.265.6, 00000000007.454.6, 00000000004.245.6] 2023-11-02T18:30:24.493 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:24.484+0000 7f123e934f40 0 bucket checkpoint complete 2023-11-02T18:30:24.521 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-58 zones={a2, a1} 2023-11-02T18:30:24.522 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:24.522 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:24.523 DEBUG:boto:Method: HEAD 2023-11-02T18:30:24.523 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:24.524 DEBUG:boto:Data: 2023-11-02T18:30:24.524 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.525 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:24.525 DEBUG:boto:Port: 8001 2023-11-02T18:30:24.526 DEBUG:boto:Params: {} 2023-11-02T18:30:24.526 DEBUG:boto:Token: None 2023-11-02T18:30:24.527 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/ 2023-11-02T18:30:24.527 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SEiQoxvxMPgkccuvA7rEHYR+qxE= 2023-11-02T18:30:24.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SEiQoxvxMPgkccuvA7rEHYR+qxE=', 'Content-Length': '0'} 2023-11-02T18:30:24.528 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-11-02T18:30:24.529 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:30:24.721 DEBUG:boto:Token: None 2023-11-02T18:30:24.722 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/ 2023-11-02T18:30:24.722 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SEiQoxvxMPgkccuvA7rEHYR+qxE= 2023-11-02T18:30:24.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SEiQoxvxMPgkccuvA7rEHYR+qxE=', 'Content-Length': '0'} 2023-11-02T18:30:24.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020cc17c2eb280aa8-006543eac0-1b8c-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.734 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:24.734 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:24.735 DEBUG:boto:Method: HEAD 2023-11-02T18:30:24.735 DEBUG:boto:Path: /ibtljn-58/ 2023-11-02T18:30:24.736 DEBUG:boto:Data: 2023-11-02T18:30:24.736 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.736 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:24.737 DEBUG:boto:Port: 8000 2023-11-02T18:30:24.737 DEBUG:boto:Params: {} 2023-11-02T18:30:24.738 DEBUG:boto:Token: None 2023-11-02T18:30:24.738 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/ 2023-11-02T18:30:24.739 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SEiQoxvxMPgkccuvA7rEHYR+qxE= 2023-11-02T18:30:24.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SEiQoxvxMPgkccuvA7rEHYR+qxE=', 'Content-Length': '0'} 2023-11-02T18:30:24.746 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000deee9ba88aa3ef03-006543eac0-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.747 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:30:24.747 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:24.748 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:24.748 DEBUG:boto:path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:24.749 DEBUG:boto:auth_path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:24.749 DEBUG:boto:Method: GET 2023-11-02T18:30:24.749 DEBUG:boto:Path: /ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:24.750 DEBUG:boto:Data: 2023-11-02T18:30:24.750 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.751 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:24.751 DEBUG:boto:Port: 8001 2023-11-02T18:30:24.752 DEBUG:boto:Params: {} 2023-11-02T18:30:24.752 DEBUG:boto:Token: None 2023-11-02T18:30:24.753 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/?versions 2023-11-02T18:30:24.753 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+6i/fFQ+u1utlckv+zgzLQUtkxo= 2023-11-02T18:30:24.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+6i/fFQ+u1utlckv+zgzLQUtkxo=', 'Content-Length': '0'} 2023-11-02T18:30:24.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021a2832e621b1b17-006543eac0-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.759 DEBUG:boto:b'ibtljn-58999falseobj00000000000000000000nulltrue2023-11-02T18:30:19.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-11-02T18:30:20.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-11-02T18:30:20.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-11-02T18:30:20.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-11-02T18:30:20.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-11-02T18:30:19.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-11-02T18:30:19.621Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-11-02T18:30:17.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-11-02T18:30:20.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-11-02T18:30:19.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-11-02T18:30:19.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-11-02T18:30:19.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-11-02T18:30:20.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-11-02T18:30:20.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-11-02T18:30:19.907Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-11-02T18:30:19.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-11-02T18:30:20.476Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-11-02T18:30:20.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-11-02T18:30:19.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-11-02T18:30:19.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-11-02T18:30:20.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-11-02T18:30:17.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-11-02T18:30:17.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-11-02T18:30:18.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-11-02T18:30:19.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-11-02T18:30:16.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-11-02T18:30:19.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-11-02T18:30:19.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-11-02T18:30:12.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-11-02T18:30:17.892Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-11-02T18:30:19.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-11-02T18:30:16.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-11-02T18:30:14.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-11-02T18:30:19.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-11-02T18:30:20.191Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-11-02T18:30:16.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-11-02T18:30:20.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-11-02T18:30:20.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-11-02T18:30:16.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-11-02T18:30:20.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-11-02T18:30:19.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-11-02T18:30:16.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-11-02T18:30:18.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-11-02T18:30:19.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-11-02T18:30:14.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-11-02T18:30:18.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-11-02T18:30:15.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-11-02T18:30:16.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-11-02T18:30:20.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-11-02T18:30:18.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-11-02T18:30:20.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-11-02T18:30:20.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-11-02T18:30:12.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-11-02T18:30:18.467Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-11-02T18:30:18.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-11-02T18:30:18.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-11-02T18:30:17.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-11-02T18:30:18.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-11-02T18:30:18.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-11-02T18:30:17.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:24.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-11-02T18:30:24.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-11-02T18:30:24.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-11-02T18:30:24.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-11-02T18:30:24.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-11-02T18:30:24.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-11-02T18:30:24.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-11-02T18:30:24.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-11-02T18:30:24.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-11-02T18:30:24.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-11-02T18:30:24.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-11-02T18:30:24.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-11-02T18:30:24.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-11-02T18:30:24.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-11-02T18:30:24.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-11-02T18:30:24.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-11-02T18:30:24.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-11-02T18:30:24.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-11-02T18:30:24.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-11-02T18:30:24.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-11-02T18:30:24.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-11-02T18:30:24.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-11-02T18:30:24.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-11-02T18:30:24.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-11-02T18:30:24.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-11-02T18:30:24.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-11-02T18:30:24.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-11-02T18:30:24.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-11-02T18:30:24.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-11-02T18:30:24.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-11-02T18:30:24.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-11-02T18:30:24.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-11-02T18:30:24.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-11-02T18:30:24.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-11-02T18:30:24.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-11-02T18:30:24.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-11-02T18:30:24.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-11-02T18:30:24.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-11-02T18:30:24.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-11-02T18:30:24.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-11-02T18:30:24.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-11-02T18:30:24.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-11-02T18:30:24.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-11-02T18:30:24.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-11-02T18:30:24.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-11-02T18:30:24.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2023-11-02T18:30:24.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-11-02T18:30:24.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-11-02T18:30:24.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-11-02T18:30:24.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-11-02T18:30:24.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-11-02T18:30:24.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-11-02T18:30:24.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-11-02T18:30:24.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-11-02T18:30:24.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-11-02T18:30:24.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-11-02T18:30:24.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-11-02T18:30:24.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-11-02T18:30:24.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-11-02T18:30:24.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-11-02T18:30:24.787 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:30:24.787 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:24.788 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:24.788 DEBUG:boto:path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:24.789 DEBUG:boto:auth_path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:24.789 DEBUG:boto:Method: GET 2023-11-02T18:30:24.790 DEBUG:boto:Path: /ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:24.790 DEBUG:boto:Data: 2023-11-02T18:30:24.791 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.791 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:24.792 DEBUG:boto:Port: 8000 2023-11-02T18:30:24.792 DEBUG:boto:Params: {} 2023-11-02T18:30:24.793 DEBUG:boto:Token: None 2023-11-02T18:30:24.793 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/?versions 2023-11-02T18:30:24.794 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+6i/fFQ+u1utlckv+zgzLQUtkxo= 2023-11-02T18:30:24.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+6i/fFQ+u1utlckv+zgzLQUtkxo=', 'Content-Length': '0'} 2023-11-02T18:30:24.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091eb8ded37b9aeb9-006543eac0-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.805 DEBUG:boto:b'ibtljn-58999falseobj00000000000000000000nulltrue2023-11-02T18:30:19.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-11-02T18:30:20.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-11-02T18:30:20.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-11-02T18:30:20.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-11-02T18:30:20.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-11-02T18:30:19.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-11-02T18:30:19.621Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-11-02T18:30:17.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-11-02T18:30:20.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-11-02T18:30:19.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-11-02T18:30:19.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-11-02T18:30:19.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-11-02T18:30:20.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-11-02T18:30:20.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-11-02T18:30:19.907Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-11-02T18:30:19.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-11-02T18:30:20.476Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-11-02T18:30:20.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-11-02T18:30:19.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-11-02T18:30:19.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-11-02T18:30:20.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-11-02T18:30:17.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-11-02T18:30:17.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-11-02T18:30:18.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-11-02T18:30:19.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-11-02T18:30:16.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-11-02T18:30:19.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-11-02T18:30:19.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-11-02T18:30:12.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-11-02T18:30:17.892Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-11-02T18:30:19.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-11-02T18:30:16.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-11-02T18:30:14.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-11-02T18:30:19.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-11-02T18:30:20.191Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-11-02T18:30:16.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-11-02T18:30:20.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-11-02T18:30:20.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-11-02T18:30:16.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-11-02T18:30:20.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-11-02T18:30:19.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-11-02T18:30:16.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-11-02T18:30:18.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-11-02T18:30:19.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-11-02T18:30:14.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-11-02T18:30:18.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-11-02T18:30:15.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-11-02T18:30:16.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-11-02T18:30:20.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-11-02T18:30:18.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-11-02T18:30:20.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-11-02T18:30:20.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-11-02T18:30:12.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-11-02T18:30:18.467Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-11-02T18:30:18.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-11-02T18:30:18.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-11-02T18:30:17.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-11-02T18:30:18.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-11-02T18:30:18.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-11-02T18:30:17.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:24.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-11-02T18:30:24.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-11-02T18:30:24.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-11-02T18:30:24.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-11-02T18:30:24.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-11-02T18:30:24.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-11-02T18:30:24.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-11-02T18:30:24.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-11-02T18:30:24.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-11-02T18:30:24.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-11-02T18:30:24.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-11-02T18:30:24.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-11-02T18:30:24.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-11-02T18:30:24.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-11-02T18:30:24.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-11-02T18:30:24.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-11-02T18:30:24.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-11-02T18:30:24.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-11-02T18:30:24.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-11-02T18:30:24.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-11-02T18:30:24.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-11-02T18:30:24.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-11-02T18:30:24.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-11-02T18:30:24.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-11-02T18:30:24.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-11-02T18:30:24.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-11-02T18:30:24.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-11-02T18:30:24.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-11-02T18:30:24.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-11-02T18:30:24.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-11-02T18:30:24.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-11-02T18:30:24.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-11-02T18:30:24.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-11-02T18:30:24.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-11-02T18:30:24.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-11-02T18:30:24.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-11-02T18:30:24.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-11-02T18:30:24.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-11-02T18:30:24.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-11-02T18:30:24.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-11-02T18:30:24.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-11-02T18:30:24.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-11-02T18:30:24.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-11-02T18:30:24.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-11-02T18:30:24.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-11-02T18:30:24.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2023-11-02T18:30:24.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-11-02T18:30:24.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-11-02T18:30:24.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-11-02T18:30:24.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-11-02T18:30:24.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-11-02T18:30:24.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-11-02T18:30:24.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-11-02T18:30:24.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-11-02T18:30:24.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-11-02T18:30:24.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-11-02T18:30:24.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-11-02T18:30:24.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-11-02T18:30:24.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-11-02T18:30:24.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-11-02T18:30:24.831 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:24.832 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:24.832 DEBUG:boto:path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:24.833 DEBUG:boto:auth_path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:24.833 DEBUG:boto:Method: GET 2023-11-02T18:30:24.833 DEBUG:boto:Path: /ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:24.834 DEBUG:boto:Data: 2023-11-02T18:30:24.834 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.835 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:24.835 DEBUG:boto:Port: 8001 2023-11-02T18:30:24.836 DEBUG:boto:Params: {} 2023-11-02T18:30:24.836 DEBUG:boto:Token: None 2023-11-02T18:30:24.837 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/?versions 2023-11-02T18:30:24.837 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+6i/fFQ+u1utlckv+zgzLQUtkxo= 2023-11-02T18:30:24.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+6i/fFQ+u1utlckv+zgzLQUtkxo=', 'Content-Length': '0'} 2023-11-02T18:30:24.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9b63b886f26daa3-006543eac0-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.845 DEBUG:boto:b'ibtljn-58999falseobj00000000000000000000nulltrue2023-11-02T18:30:19.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-11-02T18:30:20.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-11-02T18:30:20.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-11-02T18:30:20.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-11-02T18:30:20.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-11-02T18:30:19.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-11-02T18:30:19.621Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-11-02T18:30:17.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-11-02T18:30:20.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-11-02T18:30:19.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-11-02T18:30:19.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-11-02T18:30:19.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-11-02T18:30:20.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-11-02T18:30:20.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-11-02T18:30:19.907Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-11-02T18:30:19.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-11-02T18:30:20.476Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-11-02T18:30:20.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-11-02T18:30:19.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-11-02T18:30:19.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-11-02T18:30:20.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-11-02T18:30:17.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-11-02T18:30:17.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-11-02T18:30:18.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-11-02T18:30:19.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-11-02T18:30:16.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-11-02T18:30:19.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-11-02T18:30:19.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-11-02T18:30:12.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-11-02T18:30:17.892Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-11-02T18:30:19.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-11-02T18:30:16.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-11-02T18:30:14.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-11-02T18:30:19.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-11-02T18:30:20.191Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-11-02T18:30:16.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-11-02T18:30:20.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-11-02T18:30:20.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-11-02T18:30:16.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-11-02T18:30:20.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-11-02T18:30:19.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-11-02T18:30:16.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-11-02T18:30:18.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-11-02T18:30:19.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-11-02T18:30:14.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-11-02T18:30:18.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-11-02T18:30:15.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-11-02T18:30:16.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-11-02T18:30:20.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-11-02T18:30:18.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-11-02T18:30:20.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-11-02T18:30:20.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-11-02T18:30:12.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-11-02T18:30:18.467Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-11-02T18:30:18.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-11-02T18:30:18.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-11-02T18:30:17.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-11-02T18:30:18.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-11-02T18:30:18.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-11-02T18:30:17.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:24.848 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:24.849 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:24.849 DEBUG:boto:path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:24.850 DEBUG:boto:auth_path=/ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:24.850 DEBUG:boto:Method: GET 2023-11-02T18:30:24.851 DEBUG:boto:Path: /ibtljn-58/?versions&max-keys=999 2023-11-02T18:30:24.851 DEBUG:boto:Data: 2023-11-02T18:30:24.851 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:24.852 DEBUG:boto:Port: 8000 2023-11-02T18:30:24.853 DEBUG:boto:Params: {} 2023-11-02T18:30:24.853 DEBUG:boto:Token: None 2023-11-02T18:30:24.854 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/?versions 2023-11-02T18:30:24.854 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+6i/fFQ+u1utlckv+zgzLQUtkxo= 2023-11-02T18:30:24.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+6i/fFQ+u1utlckv+zgzLQUtkxo=', 'Content-Length': '0'} 2023-11-02T18:30:24.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cb3d4bd11d938d7-006543eac0-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.866 DEBUG:boto:b'ibtljn-58999falseobj00000000000000000000nulltrue2023-11-02T18:30:19.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-11-02T18:30:20.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-11-02T18:30:20.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-11-02T18:30:20.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-11-02T18:30:20.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-11-02T18:30:19.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-11-02T18:30:19.621Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-11-02T18:30:17.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-11-02T18:30:20.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-11-02T18:30:19.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-11-02T18:30:19.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-11-02T18:30:19.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-11-02T18:30:20.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-11-02T18:30:20.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-11-02T18:30:19.907Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-11-02T18:30:19.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-11-02T18:30:20.476Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-11-02T18:30:20.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-11-02T18:30:19.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-11-02T18:30:19.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-11-02T18:30:20.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-11-02T18:30:17.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-11-02T18:30:17.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-11-02T18:30:18.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-11-02T18:30:19.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-11-02T18:30:16.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-11-02T18:30:19.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-11-02T18:30:19.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-11-02T18:30:12.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-11-02T18:30:17.892Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-11-02T18:30:19.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-11-02T18:30:16.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-11-02T18:30:14.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-11-02T18:30:19.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-11-02T18:30:20.191Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-11-02T18:30:16.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-11-02T18:30:20.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-11-02T18:30:20.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-11-02T18:30:16.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-11-02T18:30:20.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-11-02T18:30:19.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-11-02T18:30:16.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-11-02T18:30:18.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-11-02T18:30:19.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-11-02T18:30:14.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-11-02T18:30:18.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-11-02T18:30:15.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-11-02T18:30:16.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-11-02T18:30:20.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-11-02T18:30:18.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-11-02T18:30:20.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-11-02T18:30:20.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-11-02T18:30:12.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-11-02T18:30:18.467Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-11-02T18:30:18.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-11-02T18:30:18.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-11-02T18:30:17.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-11-02T18:30:18.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-11-02T18:30:18.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-11-02T18:30:17.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:24.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-11-02T18:30:24.870 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:24.871 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:24.871 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.872 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.872 DEBUG:boto:Method: GET 2023-11-02T18:30:24.872 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.873 DEBUG:boto:Data: 2023-11-02T18:30:24.873 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.874 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:24.874 DEBUG:boto:Port: 8001 2023-11-02T18:30:24.875 DEBUG:boto:Params: {} 2023-11-02T18:30:24.875 DEBUG:boto:Token: None 2023-11-02T18:30:24.876 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.876 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vi5tNFHW7xtCBvZcMj7aPUDhs40= 2023-11-02T18:30:24.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vi5tNFHW7xtCBvZcMj7aPUDhs40=', 'Content-Length': '0'} 2023-11-02T18:30:24.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2bccfd668f1ee18-006543eac0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.881 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:24.881 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:24.882 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.882 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.882 DEBUG:boto:Method: GET 2023-11-02T18:30:24.883 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.883 DEBUG:boto:Data: 2023-11-02T18:30:24.884 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.884 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:24.885 DEBUG:boto:Port: 8000 2023-11-02T18:30:24.885 DEBUG:boto:Params: {} 2023-11-02T18:30:24.886 DEBUG:boto:Token: None 2023-11-02T18:30:24.886 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.887 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vi5tNFHW7xtCBvZcMj7aPUDhs40= 2023-11-02T18:30:24.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vi5tNFHW7xtCBvZcMj7aPUDhs40=', 'Content-Length': '0'} 2023-11-02T18:30:24.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823e62629263291c-006543eac0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.890 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:24.891 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:24.891 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.892 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.892 DEBUG:boto:Method: HEAD 2023-11-02T18:30:24.893 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.893 DEBUG:boto:Data: 2023-11-02T18:30:24.893 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.894 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:24.894 DEBUG:boto:Port: 8001 2023-11-02T18:30:24.895 DEBUG:boto:Params: {} 2023-11-02T18:30:24.895 DEBUG:boto:Token: None 2023-11-02T18:30:24.896 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.896 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:03O2z1Lgzp4VyIxJeL4LTwtEn3I= 2023-11-02T18:30:24.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:03O2z1Lgzp4VyIxJeL4LTwtEn3I=', 'Content-Length': '0'} 2023-11-02T18:30:24.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc047d87e58c537b-006543eac0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.900 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:24.900 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:24.900 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.901 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.901 DEBUG:boto:Method: HEAD 2023-11-02T18:30:24.902 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.902 DEBUG:boto:Data: 2023-11-02T18:30:24.902 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.903 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:24.903 DEBUG:boto:Port: 8000 2023-11-02T18:30:24.904 DEBUG:boto:Params: {} 2023-11-02T18:30:24.904 DEBUG:boto:Token: None 2023-11-02T18:30:24.905 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.905 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:03O2z1Lgzp4VyIxJeL4LTwtEn3I= 2023-11-02T18:30:24.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:03O2z1Lgzp4VyIxJeL4LTwtEn3I=', 'Content-Length': '0'} 2023-11-02T18:30:24.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000881e97888064fe1f-006543eac0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-11-02T18:30:24.909 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:24.910 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:24.910 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.911 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.911 DEBUG:boto:Method: GET 2023-11-02T18:30:24.912 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.912 DEBUG:boto:Data: 2023-11-02T18:30:24.912 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.913 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:24.913 DEBUG:boto:Port: 8001 2023-11-02T18:30:24.914 DEBUG:boto:Params: {} 2023-11-02T18:30:24.914 DEBUG:boto:Token: None 2023-11-02T18:30:24.915 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.915 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vi5tNFHW7xtCBvZcMj7aPUDhs40= 2023-11-02T18:30:24.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vi5tNFHW7xtCBvZcMj7aPUDhs40=', 'Content-Length': '0'} 2023-11-02T18:30:24.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f07bcc82dd73efad-006543eac0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.962 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:24.963 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:24.963 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.964 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.964 DEBUG:boto:Method: GET 2023-11-02T18:30:24.965 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.965 DEBUG:boto:Data: 2023-11-02T18:30:24.966 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.966 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:24.966 DEBUG:boto:Port: 8000 2023-11-02T18:30:24.967 DEBUG:boto:Params: {} 2023-11-02T18:30:24.967 DEBUG:boto:Token: None 2023-11-02T18:30:24.968 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/obj00000000000000000000?versionId=null 2023-11-02T18:30:24.968 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vi5tNFHW7xtCBvZcMj7aPUDhs40= 2023-11-02T18:30:24.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vi5tNFHW7xtCBvZcMj7aPUDhs40=', 'Content-Length': '0'} 2023-11-02T18:30:24.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c44d164db0b574be-006543eac0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.972 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:24.973 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:24.973 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:24.974 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:24.974 DEBUG:boto:Method: GET 2023-11-02T18:30:24.975 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:24.975 DEBUG:boto:Data: 2023-11-02T18:30:24.976 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.977 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:24.978 DEBUG:boto:Port: 8001 2023-11-02T18:30:24.978 DEBUG:boto:Params: {} 2023-11-02T18:30:24.979 DEBUG:boto:Token: None 2023-11-02T18:30:24.979 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/ 2023-11-02T18:30:24.980 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:78kaAnfoehXbivuFM5VJlE5Aj5I= 2023-11-02T18:30:24.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:78kaAnfoehXbivuFM5VJlE5Aj5I=', 'Content-Length': '0'} 2023-11-02T18:30:24.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000399594a0304f2bb-006543eac0-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:24.986 DEBUG:boto:b'ibtljn-58obj000000000000000000001000falseobj000000000000000000002023-11-02T18:30:19.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:24.987 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:24.987 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:24.988 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:24.988 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:24.989 DEBUG:boto:Method: GET 2023-11-02T18:30:24.989 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:24.989 DEBUG:boto:Data: 2023-11-02T18:30:24.990 DEBUG:boto:Headers: {} 2023-11-02T18:30:24.990 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:24.991 DEBUG:boto:Port: 8000 2023-11-02T18:30:24.991 DEBUG:boto:Params: {} 2023-11-02T18:30:24.992 DEBUG:boto:Token: None 2023-11-02T18:30:24.992 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:24 GMT /ibtljn-58/ 2023-11-02T18:30:24.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:78kaAnfoehXbivuFM5VJlE5Aj5I= 2023-11-02T18:30:24.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:78kaAnfoehXbivuFM5VJlE5Aj5I=', 'Content-Length': '0'} 2023-11-02T18:30:25.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0aa0e1bf663df07-006543eac0-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.004 DEBUG:boto:b'ibtljn-58obj000000000000000000001000falseobj000000000000000000002023-11-02T18:30:19.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.005 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.005 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.006 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.006 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.007 DEBUG:boto:Data: 2023-11-02T18:30:25.007 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.007 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.008 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.008 DEBUG:boto:Params: {} 2023-11-02T18:30:25.009 DEBUG:boto:Token: None 2023-11-02T18:30:25.009 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.010 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F4HIZwB1IKsi0bTXDTAVIx3x+Ow= 2023-11-02T18:30:25.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F4HIZwB1IKsi0bTXDTAVIx3x+Ow=', 'Content-Length': '0'} 2023-11-02T18:30:25.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077fcbec704b0bdd7-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.013 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.013 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.014 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.014 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.015 DEBUG:boto:Data: 2023-11-02T18:30:25.015 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.015 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.016 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.016 DEBUG:boto:Params: {} 2023-11-02T18:30:25.017 DEBUG:boto:Token: None 2023-11-02T18:30:25.017 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.018 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F4HIZwB1IKsi0bTXDTAVIx3x+Ow= 2023-11-02T18:30:25.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F4HIZwB1IKsi0bTXDTAVIx3x+Ow=', 'Content-Length': '0'} 2023-11-02T18:30:25.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007194a06ce30fe341-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-11-02T18:30:25.021 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.022 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.022 DEBUG:boto:Method: GET 2023-11-02T18:30:25.023 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.023 DEBUG:boto:Data: 2023-11-02T18:30:25.024 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.024 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.024 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.025 DEBUG:boto:Params: {} 2023-11-02T18:30:25.025 DEBUG:boto:Token: None 2023-11-02T18:30:25.026 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.026 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3mSAyaXFq1LsLjZ8rLeLQY9+BzY= 2023-11-02T18:30:25.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3mSAyaXFq1LsLjZ8rLeLQY9+BzY=', 'Content-Length': '0'} 2023-11-02T18:30:25.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3cb146464271ec1-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.030 DEBUG:boto:path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.031 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.031 DEBUG:boto:Method: GET 2023-11-02T18:30:25.032 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.032 DEBUG:boto:Data: 2023-11-02T18:30:25.032 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.033 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.033 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.034 DEBUG:boto:Params: {} 2023-11-02T18:30:25.034 DEBUG:boto:Token: None 2023-11-02T18:30:25.035 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000000 2023-11-02T18:30:25.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3mSAyaXFq1LsLjZ8rLeLQY9+BzY= 2023-11-02T18:30:25.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3mSAyaXFq1LsLjZ8rLeLQY9+BzY=', 'Content-Length': '0'} 2023-11-02T18:30:25.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005013ac5bb6e3fdae-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.082 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.083 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.083 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:25.084 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:25.084 DEBUG:boto:Method: GET 2023-11-02T18:30:25.085 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:25.085 DEBUG:boto:Data: 2023-11-02T18:30:25.086 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.086 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.087 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.087 DEBUG:boto:Params: {} 2023-11-02T18:30:25.087 DEBUG:boto:Token: None 2023-11-02T18:30:25.088 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.088 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009963cbc0681ed006-006543eac1-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.095 DEBUG:boto:b'ibtljn-58obj000000000000000000001000falseobj000000000000000000002023-11-02T18:30:19.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.096 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.097 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.097 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:25.098 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:25.098 DEBUG:boto:Method: GET 2023-11-02T18:30:25.099 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000000 2023-11-02T18:30:25.099 DEBUG:boto:Data: 2023-11-02T18:30:25.099 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.100 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.100 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.101 DEBUG:boto:Params: {} 2023-11-02T18:30:25.101 DEBUG:boto:Token: None 2023-11-02T18:30:25.102 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.102 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004650e57fb8d3e413-006543eac1-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.113 DEBUG:boto:b'ibtljn-58obj000000000000000000001000falseobj000000000000000000002023-11-02T18:30:19.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-11-02T18:30:25.114 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.115 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.115 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.116 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.116 DEBUG:boto:Method: GET 2023-11-02T18:30:25.117 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.117 DEBUG:boto:Data: 2023-11-02T18:30:25.118 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.118 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.118 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.119 DEBUG:boto:Params: {} 2023-11-02T18:30:25.119 DEBUG:boto:Token: None 2023-11-02T18:30:25.120 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eCEwH2BtcP2EpH9U+ul3ATU3W+s= 2023-11-02T18:30:25.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eCEwH2BtcP2EpH9U+ul3ATU3W+s=', 'Content-Length': '0'} 2023-11-02T18:30:25.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ffa7f8aadaecab2-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.170 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.171 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.172 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.172 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.173 DEBUG:boto:Method: GET 2023-11-02T18:30:25.173 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.174 DEBUG:boto:Data: 2023-11-02T18:30:25.174 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.175 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.175 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.176 DEBUG:boto:Params: {} 2023-11-02T18:30:25.176 DEBUG:boto:Token: None 2023-11-02T18:30:25.177 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.178 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eCEwH2BtcP2EpH9U+ul3ATU3W+s= 2023-11-02T18:30:25.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eCEwH2BtcP2EpH9U+ul3ATU3W+s=', 'Content-Length': '0'} 2023-11-02T18:30:25.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c86d97ba0476be2-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.182 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.183 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.183 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.184 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.184 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.185 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.185 DEBUG:boto:Data: 2023-11-02T18:30:25.185 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.186 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.186 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.187 DEBUG:boto:Params: {} 2023-11-02T18:30:25.187 DEBUG:boto:Token: None 2023-11-02T18:30:25.188 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.188 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bsqC0sZOK2HG1RHNZ5z0tt5qtHE= 2023-11-02T18:30:25.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bsqC0sZOK2HG1RHNZ5z0tt5qtHE=', 'Content-Length': '0'} 2023-11-02T18:30:25.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000952a5655c86d5b68-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.192 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.193 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.193 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.193 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.194 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.194 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.195 DEBUG:boto:Data: 2023-11-02T18:30:25.195 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.196 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.196 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.196 DEBUG:boto:Params: {} 2023-11-02T18:30:25.197 DEBUG:boto:Token: None 2023-11-02T18:30:25.197 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.198 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bsqC0sZOK2HG1RHNZ5z0tt5qtHE= 2023-11-02T18:30:25.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bsqC0sZOK2HG1RHNZ5z0tt5qtHE=', 'Content-Length': '0'} 2023-11-02T18:30:25.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c98cbe1d47fe3e6-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-11-02T18:30:25.202 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.203 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.203 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.204 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.204 DEBUG:boto:Method: GET 2023-11-02T18:30:25.205 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.205 DEBUG:boto:Data: 2023-11-02T18:30:25.206 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.206 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.206 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.207 DEBUG:boto:Params: {} 2023-11-02T18:30:25.207 DEBUG:boto:Token: None 2023-11-02T18:30:25.208 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eCEwH2BtcP2EpH9U+ul3ATU3W+s= 2023-11-02T18:30:25.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eCEwH2BtcP2EpH9U+ul3ATU3W+s=', 'Content-Length': '0'} 2023-11-02T18:30:25.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000173cf563e9317464-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.213 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.213 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.214 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.214 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.215 DEBUG:boto:Method: GET 2023-11-02T18:30:25.215 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.216 DEBUG:boto:Data: 2023-11-02T18:30:25.216 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.217 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.217 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.217 DEBUG:boto:Params: {} 2023-11-02T18:30:25.218 DEBUG:boto:Token: None 2023-11-02T18:30:25.218 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000001?versionId=null 2023-11-02T18:30:25.219 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eCEwH2BtcP2EpH9U+ul3ATU3W+s= 2023-11-02T18:30:25.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eCEwH2BtcP2EpH9U+ul3ATU3W+s=', 'Content-Length': '0'} 2023-11-02T18:30:25.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096b30e93303a1742-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.223 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.224 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.224 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000001 2023-11-02T18:30:25.224 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000001 2023-11-02T18:30:25.225 DEBUG:boto:Method: GET 2023-11-02T18:30:25.226 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000001 2023-11-02T18:30:25.226 DEBUG:boto:Data: 2023-11-02T18:30:25.226 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.227 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.227 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.228 DEBUG:boto:Params: {} 2023-11-02T18:30:25.228 DEBUG:boto:Token: None 2023-11-02T18:30:25.229 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.229 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002795068033680ce5-006543eac1-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.278 DEBUG:boto:b'ibtljn-58obj000000000000000000011000falseobj000000000000000000012023-11-02T18:30:20.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.279 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.280 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.280 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000001 2023-11-02T18:30:25.281 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000001 2023-11-02T18:30:25.281 DEBUG:boto:Method: GET 2023-11-02T18:30:25.282 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000001 2023-11-02T18:30:25.282 DEBUG:boto:Data: 2023-11-02T18:30:25.282 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.283 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.283 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.284 DEBUG:boto:Params: {} 2023-11-02T18:30:25.284 DEBUG:boto:Token: None 2023-11-02T18:30:25.285 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.285 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000824a144e91d05b60-006543eac1-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.296 DEBUG:boto:b'ibtljn-58obj000000000000000000011000falseobj000000000000000000012023-11-02T18:30:20.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.297 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.297 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.298 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.298 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.299 DEBUG:boto:Data: 2023-11-02T18:30:25.299 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.299 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.300 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.300 DEBUG:boto:Params: {} 2023-11-02T18:30:25.301 DEBUG:boto:Token: None 2023-11-02T18:30:25.301 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.302 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qyMynmzPMpft8UWaIWsBzEiXWno= 2023-11-02T18:30:25.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qyMynmzPMpft8UWaIWsBzEiXWno=', 'Content-Length': '0'} 2023-11-02T18:30:25.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032b8ec2b4a4f3e66-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.305 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.306 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.306 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.307 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.307 DEBUG:boto:Data: 2023-11-02T18:30:25.307 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.308 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.308 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.309 DEBUG:boto:Params: {} 2023-11-02T18:30:25.309 DEBUG:boto:Token: None 2023-11-02T18:30:25.310 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.310 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qyMynmzPMpft8UWaIWsBzEiXWno= 2023-11-02T18:30:25.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qyMynmzPMpft8UWaIWsBzEiXWno=', 'Content-Length': '0'} 2023-11-02T18:30:25.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008930f1bb33dae951-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-11-02T18:30:25.314 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.315 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.315 DEBUG:boto:Method: GET 2023-11-02T18:30:25.315 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.316 DEBUG:boto:Data: 2023-11-02T18:30:25.316 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.317 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.318 DEBUG:boto:Params: {} 2023-11-02T18:30:25.318 DEBUG:boto:Token: None 2023-11-02T18:30:25.319 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YVHQISEyebcb3D6LeaLuk5Zyv+w= 2023-11-02T18:30:25.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YVHQISEyebcb3D6LeaLuk5Zyv+w=', 'Content-Length': '0'} 2023-11-02T18:30:25.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd44a04d93e1b400-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.323 DEBUG:boto:path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.323 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.324 DEBUG:boto:Method: GET 2023-11-02T18:30:25.324 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.325 DEBUG:boto:Data: 2023-11-02T18:30:25.325 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.325 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.326 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.326 DEBUG:boto:Params: {} 2023-11-02T18:30:25.327 DEBUG:boto:Token: None 2023-11-02T18:30:25.327 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000001 2023-11-02T18:30:25.328 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YVHQISEyebcb3D6LeaLuk5Zyv+w= 2023-11-02T18:30:25.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YVHQISEyebcb3D6LeaLuk5Zyv+w=', 'Content-Length': '0'} 2023-11-02T18:30:25.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7437e5e6330a22b-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.332 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.332 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.333 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000001 2023-11-02T18:30:25.333 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000001 2023-11-02T18:30:25.333 DEBUG:boto:Method: GET 2023-11-02T18:30:25.334 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000001 2023-11-02T18:30:25.334 DEBUG:boto:Data: 2023-11-02T18:30:25.335 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.335 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.336 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.336 DEBUG:boto:Params: {} 2023-11-02T18:30:25.337 DEBUG:boto:Token: None 2023-11-02T18:30:25.337 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.337 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000726b3ce4732a4528-006543eac1-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.344 DEBUG:boto:b'ibtljn-58obj000000000000000000011000falseobj000000000000000000012023-11-02T18:30:20.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.345 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.345 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.346 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000001 2023-11-02T18:30:25.346 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000001 2023-11-02T18:30:25.347 DEBUG:boto:Method: GET 2023-11-02T18:30:25.347 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000001 2023-11-02T18:30:25.347 DEBUG:boto:Data: 2023-11-02T18:30:25.348 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.348 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.349 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.349 DEBUG:boto:Params: {} 2023-11-02T18:30:25.350 DEBUG:boto:Token: None 2023-11-02T18:30:25.350 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.351 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f02b51f097778981-006543eac1-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.402 DEBUG:boto:b'ibtljn-58obj000000000000000000011000falseobj000000000000000000012023-11-02T18:30:20.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-11-02T18:30:25.404 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.404 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.405 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.405 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.406 DEBUG:boto:Method: GET 2023-11-02T18:30:25.406 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.406 DEBUG:boto:Data: 2023-11-02T18:30:25.407 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.407 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.408 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.408 DEBUG:boto:Params: {} 2023-11-02T18:30:25.409 DEBUG:boto:Token: None 2023-11-02T18:30:25.409 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.410 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Gkx/TI+K+lPTV4Zlr5D1BIyNzQ= 2023-11-02T18:30:25.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Gkx/TI+K+lPTV4Zlr5D1BIyNzQ=', 'Content-Length': '0'} 2023-11-02T18:30:25.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015fbe7c5426b67ca-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.415 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.415 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.416 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.416 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.416 DEBUG:boto:Method: GET 2023-11-02T18:30:25.417 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.417 DEBUG:boto:Data: 2023-11-02T18:30:25.418 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.418 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.419 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.419 DEBUG:boto:Params: {} 2023-11-02T18:30:25.420 DEBUG:boto:Token: None 2023-11-02T18:30:25.420 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.420 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Gkx/TI+K+lPTV4Zlr5D1BIyNzQ= 2023-11-02T18:30:25.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Gkx/TI+K+lPTV4Zlr5D1BIyNzQ=', 'Content-Length': '0'} 2023-11-02T18:30:25.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae08e746dceeef3a-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.425 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.425 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.425 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.426 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.426 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.427 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.427 DEBUG:boto:Data: 2023-11-02T18:30:25.428 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.428 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.428 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.429 DEBUG:boto:Params: {} 2023-11-02T18:30:25.429 DEBUG:boto:Token: None 2023-11-02T18:30:25.430 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.430 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+L6RP6jhr/idOQbWVnt3jvt4y/8= 2023-11-02T18:30:25.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+L6RP6jhr/idOQbWVnt3jvt4y/8=', 'Content-Length': '0'} 2023-11-02T18:30:25.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b53335df9585e09-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.434 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.435 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.435 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.436 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.436 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.437 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.437 DEBUG:boto:Data: 2023-11-02T18:30:25.437 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.438 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.438 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.439 DEBUG:boto:Params: {} 2023-11-02T18:30:25.439 DEBUG:boto:Token: None 2023-11-02T18:30:25.440 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.440 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+L6RP6jhr/idOQbWVnt3jvt4y/8= 2023-11-02T18:30:25.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+L6RP6jhr/idOQbWVnt3jvt4y/8=', 'Content-Length': '0'} 2023-11-02T18:30:25.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074cab1f2fba88948-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-11-02T18:30:25.445 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.445 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.445 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.446 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.446 DEBUG:boto:Method: GET 2023-11-02T18:30:25.447 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.447 DEBUG:boto:Data: 2023-11-02T18:30:25.448 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.448 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.449 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.449 DEBUG:boto:Params: {} 2023-11-02T18:30:25.449 DEBUG:boto:Token: None 2023-11-02T18:30:25.450 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.450 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Gkx/TI+K+lPTV4Zlr5D1BIyNzQ= 2023-11-02T18:30:25.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Gkx/TI+K+lPTV4Zlr5D1BIyNzQ=', 'Content-Length': '0'} 2023-11-02T18:30:25.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026606dcaf00f4857-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.455 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.455 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.455 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.456 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.456 DEBUG:boto:Method: GET 2023-11-02T18:30:25.457 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.457 DEBUG:boto:Data: 2023-11-02T18:30:25.458 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.458 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.458 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.459 DEBUG:boto:Params: {} 2023-11-02T18:30:25.459 DEBUG:boto:Token: None 2023-11-02T18:30:25.460 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000002?versionId=null 2023-11-02T18:30:25.460 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Gkx/TI+K+lPTV4Zlr5D1BIyNzQ= 2023-11-02T18:30:25.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Gkx/TI+K+lPTV4Zlr5D1BIyNzQ=', 'Content-Length': '0'} 2023-11-02T18:30:25.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044341fac63477845-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.506 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.507 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.508 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:25.508 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:25.509 DEBUG:boto:Method: GET 2023-11-02T18:30:25.509 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:25.510 DEBUG:boto:Data: 2023-11-02T18:30:25.510 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.511 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.511 DEBUG:boto:Params: {} 2023-11-02T18:30:25.512 DEBUG:boto:Token: None 2023-11-02T18:30:25.512 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ae58c613d32e31c-006543eac1-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.520 DEBUG:boto:b'ibtljn-58obj000000000000000000021000falseobj000000000000000000022023-11-02T18:30:20.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.520 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.521 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.521 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:25.522 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:25.522 DEBUG:boto:Method: GET 2023-11-02T18:30:25.523 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:25.523 DEBUG:boto:Data: 2023-11-02T18:30:25.524 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.524 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.525 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.525 DEBUG:boto:Params: {} 2023-11-02T18:30:25.526 DEBUG:boto:Token: None 2023-11-02T18:30:25.526 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.527 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3d0e71340a5404e-006543eac1-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.539 DEBUG:boto:b'ibtljn-58obj000000000000000000021000falseobj000000000000000000022023-11-02T18:30:20.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.539 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.540 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.540 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.541 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.541 DEBUG:boto:Data: 2023-11-02T18:30:25.542 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.542 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.543 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.543 DEBUG:boto:Params: {} 2023-11-02T18:30:25.544 DEBUG:boto:Token: None 2023-11-02T18:30:25.544 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.545 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jCmuM5RiHak57LMO1tg0Dp+mB88= 2023-11-02T18:30:25.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jCmuM5RiHak57LMO1tg0Dp+mB88=', 'Content-Length': '0'} 2023-11-02T18:30:25.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c16f3386d1860f47-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.548 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.549 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.549 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.549 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.550 DEBUG:boto:Data: 2023-11-02T18:30:25.550 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.551 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.551 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.552 DEBUG:boto:Params: {} 2023-11-02T18:30:25.552 DEBUG:boto:Token: None 2023-11-02T18:30:25.553 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.553 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jCmuM5RiHak57LMO1tg0Dp+mB88= 2023-11-02T18:30:25.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jCmuM5RiHak57LMO1tg0Dp+mB88=', 'Content-Length': '0'} 2023-11-02T18:30:25.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002851e82c117c4d5f-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-11-02T18:30:25.557 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.558 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.558 DEBUG:boto:Method: GET 2023-11-02T18:30:25.559 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.559 DEBUG:boto:Data: 2023-11-02T18:30:25.560 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.560 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.561 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.561 DEBUG:boto:Params: {} 2023-11-02T18:30:25.562 DEBUG:boto:Token: None 2023-11-02T18:30:25.562 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.563 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TnxLjpuVj7KBrXtfgmJssEiWaRE= 2023-11-02T18:30:25.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TnxLjpuVj7KBrXtfgmJssEiWaRE=', 'Content-Length': '0'} 2023-11-02T18:30:25.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c2bf53fa335030c-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.610 DEBUG:boto:path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.611 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.611 DEBUG:boto:Method: GET 2023-11-02T18:30:25.612 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.612 DEBUG:boto:Data: 2023-11-02T18:30:25.613 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.613 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.613 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.614 DEBUG:boto:Params: {} 2023-11-02T18:30:25.614 DEBUG:boto:Token: None 2023-11-02T18:30:25.615 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000002 2023-11-02T18:30:25.615 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TnxLjpuVj7KBrXtfgmJssEiWaRE= 2023-11-02T18:30:25.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TnxLjpuVj7KBrXtfgmJssEiWaRE=', 'Content-Length': '0'} 2023-11-02T18:30:25.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d3f1df3a092921f-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.622 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.622 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.623 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:25.623 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:25.624 DEBUG:boto:Method: GET 2023-11-02T18:30:25.624 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:25.625 DEBUG:boto:Data: 2023-11-02T18:30:25.625 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.625 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.626 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.626 DEBUG:boto:Params: {} 2023-11-02T18:30:25.627 DEBUG:boto:Token: None 2023-11-02T18:30:25.627 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.628 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce997072bce87d00-006543eac1-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.632 DEBUG:boto:b'ibtljn-58obj000000000000000000021000falseobj000000000000000000022023-11-02T18:30:20.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.633 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.633 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.633 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:25.634 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:25.634 DEBUG:boto:Method: GET 2023-11-02T18:30:25.635 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000002 2023-11-02T18:30:25.635 DEBUG:boto:Data: 2023-11-02T18:30:25.636 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.637 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.638 DEBUG:boto:Params: {} 2023-11-02T18:30:25.638 DEBUG:boto:Token: None 2023-11-02T18:30:25.639 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.639 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000055213f009edda7b-006543eac1-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.690 DEBUG:boto:b'ibtljn-58obj000000000000000000021000falseobj000000000000000000022023-11-02T18:30:20.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-11-02T18:30:25.692 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.692 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.692 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.693 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.693 DEBUG:boto:Method: GET 2023-11-02T18:30:25.694 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.694 DEBUG:boto:Data: 2023-11-02T18:30:25.695 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.696 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.696 DEBUG:boto:Params: {} 2023-11-02T18:30:25.696 DEBUG:boto:Token: None 2023-11-02T18:30:25.697 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.698 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T2Eq971v4aLEAwBhOwa8Y/2Hzks= 2023-11-02T18:30:25.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T2Eq971v4aLEAwBhOwa8Y/2Hzks=', 'Content-Length': '0'} 2023-11-02T18:30:25.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064971d01e787d65a-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.702 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.703 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.703 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.704 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.704 DEBUG:boto:Method: GET 2023-11-02T18:30:25.704 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.705 DEBUG:boto:Data: 2023-11-02T18:30:25.705 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.706 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.706 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.707 DEBUG:boto:Params: {} 2023-11-02T18:30:25.707 DEBUG:boto:Token: None 2023-11-02T18:30:25.707 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.708 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T2Eq971v4aLEAwBhOwa8Y/2Hzks= 2023-11-02T18:30:25.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T2Eq971v4aLEAwBhOwa8Y/2Hzks=', 'Content-Length': '0'} 2023-11-02T18:30:25.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000256b08e5967eac84-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.712 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.713 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.713 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.714 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.714 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.715 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.715 DEBUG:boto:Data: 2023-11-02T18:30:25.716 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.716 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.717 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.717 DEBUG:boto:Params: {} 2023-11-02T18:30:25.718 DEBUG:boto:Token: None 2023-11-02T18:30:25.718 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.719 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7E/ZbxLlQRoOH2+SOUzXvy8Cb9U= 2023-11-02T18:30:25.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7E/ZbxLlQRoOH2+SOUzXvy8Cb9U=', 'Content-Length': '0'} 2023-11-02T18:30:25.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037bbf4fdcab387e3-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.723 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.723 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.724 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.724 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.725 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.725 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.726 DEBUG:boto:Data: 2023-11-02T18:30:25.726 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.727 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.727 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.728 DEBUG:boto:Params: {} 2023-11-02T18:30:25.728 DEBUG:boto:Token: None 2023-11-02T18:30:25.729 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.729 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7E/ZbxLlQRoOH2+SOUzXvy8Cb9U= 2023-11-02T18:30:25.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7E/ZbxLlQRoOH2+SOUzXvy8Cb9U=', 'Content-Length': '0'} 2023-11-02T18:30:25.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078672d9b1a2b438c-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-11-02T18:30:25.734 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.735 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.735 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.736 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.736 DEBUG:boto:Method: GET 2023-11-02T18:30:25.736 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.737 DEBUG:boto:Data: 2023-11-02T18:30:25.737 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.738 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.739 DEBUG:boto:Params: {} 2023-11-02T18:30:25.739 DEBUG:boto:Token: None 2023-11-02T18:30:25.740 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T2Eq971v4aLEAwBhOwa8Y/2Hzks= 2023-11-02T18:30:25.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T2Eq971v4aLEAwBhOwa8Y/2Hzks=', 'Content-Length': '0'} 2023-11-02T18:30:25.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdf50d65ceb4e81d-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.745 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.745 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.746 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.746 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.747 DEBUG:boto:Method: GET 2023-11-02T18:30:25.747 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.748 DEBUG:boto:Data: 2023-11-02T18:30:25.748 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.749 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.749 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.750 DEBUG:boto:Params: {} 2023-11-02T18:30:25.750 DEBUG:boto:Token: None 2023-11-02T18:30:25.751 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000003?versionId=null 2023-11-02T18:30:25.751 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T2Eq971v4aLEAwBhOwa8Y/2Hzks= 2023-11-02T18:30:25.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T2Eq971v4aLEAwBhOwa8Y/2Hzks=', 'Content-Length': '0'} 2023-11-02T18:30:25.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050c14deb42c16201-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.756 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.756 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.756 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:25.757 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:25.757 DEBUG:boto:Method: GET 2023-11-02T18:30:25.758 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:25.758 DEBUG:boto:Data: 2023-11-02T18:30:25.759 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.759 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.759 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.760 DEBUG:boto:Params: {} 2023-11-02T18:30:25.760 DEBUG:boto:Token: None 2023-11-02T18:30:25.761 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.761 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006207081fef322465-006543eac1-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.767 DEBUG:boto:b'ibtljn-58obj000000000000000000031000falseobj000000000000000000032023-11-02T18:30:20.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.768 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.769 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.769 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:25.770 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:25.770 DEBUG:boto:Method: GET 2023-11-02T18:30:25.770 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:25.771 DEBUG:boto:Data: 2023-11-02T18:30:25.771 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.772 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.772 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.773 DEBUG:boto:Params: {} 2023-11-02T18:30:25.773 DEBUG:boto:Token: None 2023-11-02T18:30:25.774 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.774 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000256c1d9e2469135a-006543eac1-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.826 DEBUG:boto:b'ibtljn-58obj000000000000000000031000falseobj000000000000000000032023-11-02T18:30:20.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.827 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.828 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.828 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.829 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.829 DEBUG:boto:Data: 2023-11-02T18:30:25.830 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.830 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.830 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.831 DEBUG:boto:Params: {} 2023-11-02T18:30:25.831 DEBUG:boto:Token: None 2023-11-02T18:30:25.832 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.832 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w9+hAR1IEH7tES5tA4xWRYVjkzQ= 2023-11-02T18:30:25.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w9+hAR1IEH7tES5tA4xWRYVjkzQ=', 'Content-Length': '0'} 2023-11-02T18:30:25.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdb77d7ebc461a18-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.837 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.837 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.838 DEBUG:boto:Method: HEAD 2023-11-02T18:30:25.838 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.839 DEBUG:boto:Data: 2023-11-02T18:30:25.839 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.839 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.840 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.840 DEBUG:boto:Params: {} 2023-11-02T18:30:25.841 DEBUG:boto:Token: None 2023-11-02T18:30:25.841 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.842 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w9+hAR1IEH7tES5tA4xWRYVjkzQ= 2023-11-02T18:30:25.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w9+hAR1IEH7tES5tA4xWRYVjkzQ=', 'Content-Length': '0'} 2023-11-02T18:30:25.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1d792523833054f-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-11-02T18:30:25.847 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.847 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.848 DEBUG:boto:Method: GET 2023-11-02T18:30:25.848 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.849 DEBUG:boto:Data: 2023-11-02T18:30:25.849 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.850 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.850 DEBUG:boto:Params: {} 2023-11-02T18:30:25.851 DEBUG:boto:Token: None 2023-11-02T18:30:25.851 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.852 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V8/RLxKJsWBC+D2nGfQJTlrQsRE= 2023-11-02T18:30:25.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V8/RLxKJsWBC+D2nGfQJTlrQsRE=', 'Content-Length': '0'} 2023-11-02T18:30:25.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b819bf5864675cc-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.898 DEBUG:boto:path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.899 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.899 DEBUG:boto:Method: GET 2023-11-02T18:30:25.900 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.900 DEBUG:boto:Data: 2023-11-02T18:30:25.901 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.901 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.902 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.902 DEBUG:boto:Params: {} 2023-11-02T18:30:25.903 DEBUG:boto:Token: None 2023-11-02T18:30:25.903 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000003 2023-11-02T18:30:25.904 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V8/RLxKJsWBC+D2nGfQJTlrQsRE= 2023-11-02T18:30:25.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V8/RLxKJsWBC+D2nGfQJTlrQsRE=', 'Content-Length': '0'} 2023-11-02T18:30:25.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058af5d99df983e14-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.908 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.908 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.909 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:25.910 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:25.910 DEBUG:boto:Method: GET 2023-11-02T18:30:25.910 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:25.911 DEBUG:boto:Data: 2023-11-02T18:30:25.911 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.912 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.912 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.913 DEBUG:boto:Params: {} 2023-11-02T18:30:25.913 DEBUG:boto:Token: None 2023-11-02T18:30:25.914 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.914 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000298bb60e808cfb4-006543eac1-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.920 DEBUG:boto:b'ibtljn-58obj000000000000000000031000falseobj000000000000000000032023-11-02T18:30:20.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.921 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:25.921 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:25.922 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:25.922 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:25.923 DEBUG:boto:Method: GET 2023-11-02T18:30:25.923 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000003 2023-11-02T18:30:25.924 DEBUG:boto:Data: 2023-11-02T18:30:25.924 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.925 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.925 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.925 DEBUG:boto:Params: {} 2023-11-02T18:30:25.926 DEBUG:boto:Token: None 2023-11-02T18:30:25.926 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/ 2023-11-02T18:30:25.927 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps= 2023-11-02T18:30:25.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuRGg6HuKV3WINYYUeM8pUFaVps=', 'Content-Length': '0'} 2023-11-02T18:30:25.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a44cb99241260feb-006543eac1-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.938 DEBUG:boto:b'ibtljn-58obj000000000000000000031000falseobj000000000000000000032023-11-02T18:30:20.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:25.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-11-02T18:30:25.939 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:25.940 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:25.940 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:25.940 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:25.941 DEBUG:boto:Method: GET 2023-11-02T18:30:25.941 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:25.942 DEBUG:boto:Data: 2023-11-02T18:30:25.942 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.943 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:25.943 DEBUG:boto:Port: 8001 2023-11-02T18:30:25.944 DEBUG:boto:Params: {} 2023-11-02T18:30:25.945 DEBUG:boto:Token: None 2023-11-02T18:30:25.946 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:25.946 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AKJFZX7znGcEdoB1+UFPJlfQKc4= 2023-11-02T18:30:25.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AKJFZX7znGcEdoB1+UFPJlfQKc4=', 'Content-Length': '0'} 2023-11-02T18:30:25.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0073bbdd5ace4cb-006543eac1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:25.950 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:25.950 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:25.951 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:25.951 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:25.952 DEBUG:boto:Method: GET 2023-11-02T18:30:25.952 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:25.953 DEBUG:boto:Data: 2023-11-02T18:30:25.953 DEBUG:boto:Headers: {} 2023-11-02T18:30:25.954 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:25.954 DEBUG:boto:Port: 8000 2023-11-02T18:30:25.954 DEBUG:boto:Params: {} 2023-11-02T18:30:25.955 DEBUG:boto:Token: None 2023-11-02T18:30:25.955 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:25 GMT /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:25.956 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AKJFZX7znGcEdoB1+UFPJlfQKc4= 2023-11-02T18:30:25.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AKJFZX7znGcEdoB1+UFPJlfQKc4=', 'Content-Length': '0'} 2023-11-02T18:30:25.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f37a257fe2a644b-006543eac1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.006 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.007 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.008 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.008 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.008 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.009 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.009 DEBUG:boto:Data: 2023-11-02T18:30:26.010 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.010 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.011 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.011 DEBUG:boto:Params: {} 2023-11-02T18:30:26.011 DEBUG:boto:Token: None 2023-11-02T18:30:26.012 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bM6BT2lQNL38TT8D9BhfRE10et0= 2023-11-02T18:30:26.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bM6BT2lQNL38TT8D9BhfRE10et0=', 'Content-Length': '0'} 2023-11-02T18:30:26.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb6535d155cf86a6-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.016 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.017 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.017 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.018 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.018 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.019 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.019 DEBUG:boto:Data: 2023-11-02T18:30:26.020 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.020 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.020 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.021 DEBUG:boto:Params: {} 2023-11-02T18:30:26.021 DEBUG:boto:Token: None 2023-11-02T18:30:26.022 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.022 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bM6BT2lQNL38TT8D9BhfRE10et0= 2023-11-02T18:30:26.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bM6BT2lQNL38TT8D9BhfRE10et0=', 'Content-Length': '0'} 2023-11-02T18:30:26.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000437d2bc40f953e7a-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-11-02T18:30:26.027 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.027 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.028 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.028 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.029 DEBUG:boto:Method: GET 2023-11-02T18:30:26.029 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.029 DEBUG:boto:Data: 2023-11-02T18:30:26.030 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.031 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.031 DEBUG:boto:Params: {} 2023-11-02T18:30:26.031 DEBUG:boto:Token: None 2023-11-02T18:30:26.032 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MEoI2lJEQ9ybF66vSlVBqTzWpYQ= 2023-11-02T18:30:26.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MEoI2lJEQ9ybF66vSlVBqTzWpYQ=', 'Content-Length': '0'} 2023-11-02T18:30:26.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f1edd51aa2f0c67-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.078 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.079 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.079 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.080 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.080 DEBUG:boto:Method: GET 2023-11-02T18:30:26.081 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.081 DEBUG:boto:Data: 2023-11-02T18:30:26.082 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.082 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.082 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.083 DEBUG:boto:Params: {} 2023-11-02T18:30:26.083 DEBUG:boto:Token: None 2023-11-02T18:30:26.084 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000004?versionId=null 2023-11-02T18:30:26.085 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MEoI2lJEQ9ybF66vSlVBqTzWpYQ= 2023-11-02T18:30:26.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MEoI2lJEQ9ybF66vSlVBqTzWpYQ=', 'Content-Length': '0'} 2023-11-02T18:30:26.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c53bdb17246d7472-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.089 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.089 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.090 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:26.090 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:26.091 DEBUG:boto:Method: GET 2023-11-02T18:30:26.091 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:26.092 DEBUG:boto:Data: 2023-11-02T18:30:26.092 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.092 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.093 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.093 DEBUG:boto:Params: {} 2023-11-02T18:30:26.094 DEBUG:boto:Token: None 2023-11-02T18:30:26.094 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.095 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000978cca4e5a3b0d2f-006543eac2-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.100 DEBUG:boto:b'ibtljn-58obj000000000000000000041000falseobj000000000000000000042023-11-02T18:30:20.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.101 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.102 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.102 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:26.102 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:26.103 DEBUG:boto:Method: GET 2023-11-02T18:30:26.103 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:26.104 DEBUG:boto:Data: 2023-11-02T18:30:26.104 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.105 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.105 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.106 DEBUG:boto:Params: {} 2023-11-02T18:30:26.106 DEBUG:boto:Token: None 2023-11-02T18:30:26.107 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.107 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab3349d13b9c0635-006543eac2-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.119 DEBUG:boto:b'ibtljn-58obj000000000000000000041000falseobj000000000000000000042023-11-02T18:30:20.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.120 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.120 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.120 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.121 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.121 DEBUG:boto:Data: 2023-11-02T18:30:26.122 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.122 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.123 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.123 DEBUG:boto:Params: {} 2023-11-02T18:30:26.124 DEBUG:boto:Token: None 2023-11-02T18:30:26.124 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.124 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nLfV8YRb6he+bDHlMc0P4be8Jp4= 2023-11-02T18:30:26.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nLfV8YRb6he+bDHlMc0P4be8Jp4=', 'Content-Length': '0'} 2023-11-02T18:30:26.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b733aeee678ba7c-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.128 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.128 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.129 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.129 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.129 DEBUG:boto:Data: 2023-11-02T18:30:26.130 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.130 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.131 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.131 DEBUG:boto:Params: {} 2023-11-02T18:30:26.132 DEBUG:boto:Token: None 2023-11-02T18:30:26.132 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.133 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nLfV8YRb6he+bDHlMc0P4be8Jp4= 2023-11-02T18:30:26.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nLfV8YRb6he+bDHlMc0P4be8Jp4=', 'Content-Length': '0'} 2023-11-02T18:30:26.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4a19fcd6bb5f3f-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-11-02T18:30:26.136 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.137 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.137 DEBUG:boto:Method: GET 2023-11-02T18:30:26.138 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.138 DEBUG:boto:Data: 2023-11-02T18:30:26.139 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.139 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.140 DEBUG:boto:Params: {} 2023-11-02T18:30:26.140 DEBUG:boto:Token: None 2023-11-02T18:30:26.141 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DC/u++/ZnaFHc5xfxc+zYEKa4o0= 2023-11-02T18:30:26.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DC/u++/ZnaFHc5xfxc+zYEKa4o0=', 'Content-Length': '0'} 2023-11-02T18:30:26.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b24cd8395cdeb133-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.190 DEBUG:boto:path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.191 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.192 DEBUG:boto:Method: GET 2023-11-02T18:30:26.193 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.194 DEBUG:boto:Data: 2023-11-02T18:30:26.194 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.195 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.196 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.196 DEBUG:boto:Params: {} 2023-11-02T18:30:26.197 DEBUG:boto:Token: None 2023-11-02T18:30:26.198 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000004 2023-11-02T18:30:26.199 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DC/u++/ZnaFHc5xfxc+zYEKa4o0= 2023-11-02T18:30:26.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DC/u++/ZnaFHc5xfxc+zYEKa4o0=', 'Content-Length': '0'} 2023-11-02T18:30:26.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1c5bedc53d768e6-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.203 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.203 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.204 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:26.204 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:26.205 DEBUG:boto:Method: GET 2023-11-02T18:30:26.205 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:26.206 DEBUG:boto:Data: 2023-11-02T18:30:26.207 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.207 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.208 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.209 DEBUG:boto:Params: {} 2023-11-02T18:30:26.211 DEBUG:boto:Token: None 2023-11-02T18:30:26.211 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.212 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012353568fdb67fad-006543eac2-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.219 DEBUG:boto:b'ibtljn-58obj000000000000000000041000falseobj000000000000000000042023-11-02T18:30:20.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.220 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.220 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.221 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:26.221 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:26.221 DEBUG:boto:Method: GET 2023-11-02T18:30:26.222 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000004 2023-11-02T18:30:26.222 DEBUG:boto:Data: 2023-11-02T18:30:26.223 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.224 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.224 DEBUG:boto:Params: {} 2023-11-02T18:30:26.225 DEBUG:boto:Token: None 2023-11-02T18:30:26.225 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dce1494d21a54bf5-006543eac2-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.238 DEBUG:boto:b'ibtljn-58obj000000000000000000041000falseobj000000000000000000042023-11-02T18:30:20.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-11-02T18:30:26.239 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.239 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.240 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.240 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.240 DEBUG:boto:Method: GET 2023-11-02T18:30:26.241 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.241 DEBUG:boto:Data: 2023-11-02T18:30:26.242 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.242 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.243 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.243 DEBUG:boto:Params: {} 2023-11-02T18:30:26.244 DEBUG:boto:Token: None 2023-11-02T18:30:26.244 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.245 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oJPP5ttO9OQ/fMa1/kE4rb2bmII= 2023-11-02T18:30:26.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oJPP5ttO9OQ/fMa1/kE4rb2bmII=', 'Content-Length': '0'} 2023-11-02T18:30:26.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055617b6410533514-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.249 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.249 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.249 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.250 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.250 DEBUG:boto:Method: GET 2023-11-02T18:30:26.251 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.251 DEBUG:boto:Data: 2023-11-02T18:30:26.252 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.252 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.252 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.253 DEBUG:boto:Params: {} 2023-11-02T18:30:26.253 DEBUG:boto:Token: None 2023-11-02T18:30:26.254 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.254 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oJPP5ttO9OQ/fMa1/kE4rb2bmII= 2023-11-02T18:30:26.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oJPP5ttO9OQ/fMa1/kE4rb2bmII=', 'Content-Length': '0'} 2023-11-02T18:30:26.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcf272822eb1b6d1-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.258 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.258 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.259 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.259 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.259 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.260 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.260 DEBUG:boto:Data: 2023-11-02T18:30:26.261 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.261 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.261 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.262 DEBUG:boto:Params: {} 2023-11-02T18:30:26.262 DEBUG:boto:Token: None 2023-11-02T18:30:26.263 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.263 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:njQj0/+3ynzNu9E1BFNmNMNMHAE= 2023-11-02T18:30:26.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:njQj0/+3ynzNu9E1BFNmNMNMHAE=', 'Content-Length': '0'} 2023-11-02T18:30:26.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ed5f5586665965c-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.267 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.267 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.268 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.268 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.268 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.269 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.269 DEBUG:boto:Data: 2023-11-02T18:30:26.270 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.270 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.270 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.271 DEBUG:boto:Params: {} 2023-11-02T18:30:26.271 DEBUG:boto:Token: None 2023-11-02T18:30:26.272 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.272 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:njQj0/+3ynzNu9E1BFNmNMNMHAE= 2023-11-02T18:30:26.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:njQj0/+3ynzNu9E1BFNmNMNMHAE=', 'Content-Length': '0'} 2023-11-02T18:30:26.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000112f6e183d1046f-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-11-02T18:30:26.276 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.277 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.277 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.278 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.278 DEBUG:boto:Method: GET 2023-11-02T18:30:26.279 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.279 DEBUG:boto:Data: 2023-11-02T18:30:26.279 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.280 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.280 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.281 DEBUG:boto:Params: {} 2023-11-02T18:30:26.281 DEBUG:boto:Token: None 2023-11-02T18:30:26.282 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.282 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oJPP5ttO9OQ/fMa1/kE4rb2bmII= 2023-11-02T18:30:26.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oJPP5ttO9OQ/fMa1/kE4rb2bmII=', 'Content-Length': '0'} 2023-11-02T18:30:26.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b32c29e964395ef8-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.330 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.331 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.331 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.332 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.332 DEBUG:boto:Method: GET 2023-11-02T18:30:26.333 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.333 DEBUG:boto:Data: 2023-11-02T18:30:26.334 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.334 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.334 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.335 DEBUG:boto:Params: {} 2023-11-02T18:30:26.335 DEBUG:boto:Token: None 2023-11-02T18:30:26.336 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000005?versionId=null 2023-11-02T18:30:26.336 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oJPP5ttO9OQ/fMa1/kE4rb2bmII= 2023-11-02T18:30:26.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oJPP5ttO9OQ/fMa1/kE4rb2bmII=', 'Content-Length': '0'} 2023-11-02T18:30:26.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef886fd70eaf20fb-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.341 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.341 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.341 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:26.342 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:26.342 DEBUG:boto:Method: GET 2023-11-02T18:30:26.343 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:26.343 DEBUG:boto:Data: 2023-11-02T18:30:26.344 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.344 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.345 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.345 DEBUG:boto:Params: {} 2023-11-02T18:30:26.345 DEBUG:boto:Token: None 2023-11-02T18:30:26.346 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.346 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee84ca0865c886c0-006543eac2-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.354 DEBUG:boto:b'ibtljn-58obj000000000000000000051000falseobj000000000000000000052023-11-02T18:30:19.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.354 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.355 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.355 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:26.356 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:26.356 DEBUG:boto:Method: GET 2023-11-02T18:30:26.357 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:26.357 DEBUG:boto:Data: 2023-11-02T18:30:26.358 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.359 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.359 DEBUG:boto:Params: {} 2023-11-02T18:30:26.359 DEBUG:boto:Token: None 2023-11-02T18:30:26.360 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.361 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c35558ad230f81db-006543eac2-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.414 DEBUG:boto:b'ibtljn-58obj000000000000000000051000falseobj000000000000000000052023-11-02T18:30:19.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.415 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.416 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.416 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.417 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.417 DEBUG:boto:Data: 2023-11-02T18:30:26.417 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.418 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.418 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.419 DEBUG:boto:Params: {} 2023-11-02T18:30:26.419 DEBUG:boto:Token: None 2023-11-02T18:30:26.420 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.420 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Gh3HnWkQzg+1ClYDMHoRTxAfDUo= 2023-11-02T18:30:26.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Gh3HnWkQzg+1ClYDMHoRTxAfDUo=', 'Content-Length': '0'} 2023-11-02T18:30:26.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a53d8fabc45300c1-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.424 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.425 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.425 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.426 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.426 DEBUG:boto:Data: 2023-11-02T18:30:26.427 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.427 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.427 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.428 DEBUG:boto:Params: {} 2023-11-02T18:30:26.428 DEBUG:boto:Token: None 2023-11-02T18:30:26.429 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.429 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Gh3HnWkQzg+1ClYDMHoRTxAfDUo= 2023-11-02T18:30:26.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Gh3HnWkQzg+1ClYDMHoRTxAfDUo=', 'Content-Length': '0'} 2023-11-02T18:30:26.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdacbff53a2ced36-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-11-02T18:30:26.434 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.434 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.435 DEBUG:boto:Method: GET 2023-11-02T18:30:26.436 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.436 DEBUG:boto:Data: 2023-11-02T18:30:26.437 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.437 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.438 DEBUG:boto:Params: {} 2023-11-02T18:30:26.439 DEBUG:boto:Token: None 2023-11-02T18:30:26.439 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.440 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7WX2fpO8f3ux41B/34NZez4SZvI= 2023-11-02T18:30:26.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7WX2fpO8f3ux41B/34NZez4SZvI=', 'Content-Length': '0'} 2023-11-02T18:30:26.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b68afaa0e26f9df8-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.444 DEBUG:boto:path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.444 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.445 DEBUG:boto:Method: GET 2023-11-02T18:30:26.445 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.446 DEBUG:boto:Data: 2023-11-02T18:30:26.446 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.447 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.448 DEBUG:boto:Params: {} 2023-11-02T18:30:26.448 DEBUG:boto:Token: None 2023-11-02T18:30:26.449 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000005 2023-11-02T18:30:26.449 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7WX2fpO8f3ux41B/34NZez4SZvI= 2023-11-02T18:30:26.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7WX2fpO8f3ux41B/34NZez4SZvI=', 'Content-Length': '0'} 2023-11-02T18:30:26.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005718176a00f36292-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.453 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.454 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.454 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:26.455 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:26.455 DEBUG:boto:Method: GET 2023-11-02T18:30:26.456 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:26.456 DEBUG:boto:Data: 2023-11-02T18:30:26.457 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.457 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.457 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.458 DEBUG:boto:Params: {} 2023-11-02T18:30:26.458 DEBUG:boto:Token: None 2023-11-02T18:30:26.459 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.459 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb4628fca9446f7b-006543eac2-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.506 DEBUG:boto:b'ibtljn-58obj000000000000000000051000falseobj000000000000000000052023-11-02T18:30:19.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.507 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.508 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.508 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:26.509 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:26.510 DEBUG:boto:Method: GET 2023-11-02T18:30:26.511 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000005 2023-11-02T18:30:26.511 DEBUG:boto:Data: 2023-11-02T18:30:26.512 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.512 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.513 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.513 DEBUG:boto:Params: {} 2023-11-02T18:30:26.514 DEBUG:boto:Token: None 2023-11-02T18:30:26.514 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.515 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b9ff464964d14a7-006543eac2-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.526 DEBUG:boto:b'ibtljn-58obj000000000000000000051000falseobj000000000000000000052023-11-02T18:30:19.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-11-02T18:30:26.527 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.528 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.528 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.529 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.530 DEBUG:boto:Method: GET 2023-11-02T18:30:26.530 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.530 DEBUG:boto:Data: 2023-11-02T18:30:26.531 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.531 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.532 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.532 DEBUG:boto:Params: {} 2023-11-02T18:30:26.533 DEBUG:boto:Token: None 2023-11-02T18:30:26.533 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.534 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YRzYRbJmoa/1K7PysY0HcJjileU= 2023-11-02T18:30:26.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YRzYRbJmoa/1K7PysY0HcJjileU=', 'Content-Length': '0'} 2023-11-02T18:30:26.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cb541300cc1cd90-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.540 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.541 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.541 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.542 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.542 DEBUG:boto:Method: GET 2023-11-02T18:30:26.542 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.543 DEBUG:boto:Data: 2023-11-02T18:30:26.543 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.544 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.544 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.545 DEBUG:boto:Params: {} 2023-11-02T18:30:26.545 DEBUG:boto:Token: None 2023-11-02T18:30:26.546 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.546 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YRzYRbJmoa/1K7PysY0HcJjileU= 2023-11-02T18:30:26.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YRzYRbJmoa/1K7PysY0HcJjileU=', 'Content-Length': '0'} 2023-11-02T18:30:26.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037da5ef363b6c979-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.550 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.551 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.551 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.552 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.552 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.552 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.553 DEBUG:boto:Data: 2023-11-02T18:30:26.553 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.554 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.555 DEBUG:boto:Params: {} 2023-11-02T18:30:26.555 DEBUG:boto:Token: None 2023-11-02T18:30:26.556 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KWBVtRA+CNUErcfI59Q6GCGB+Zo= 2023-11-02T18:30:26.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KWBVtRA+CNUErcfI59Q6GCGB+Zo=', 'Content-Length': '0'} 2023-11-02T18:30:26.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4ded805d0f9510b-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.560 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.560 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.561 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.561 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.562 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.562 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.563 DEBUG:boto:Data: 2023-11-02T18:30:26.563 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.563 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.564 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.564 DEBUG:boto:Params: {} 2023-11-02T18:30:26.565 DEBUG:boto:Token: None 2023-11-02T18:30:26.565 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.566 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KWBVtRA+CNUErcfI59Q6GCGB+Zo= 2023-11-02T18:30:26.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KWBVtRA+CNUErcfI59Q6GCGB+Zo=', 'Content-Length': '0'} 2023-11-02T18:30:26.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022da8048e2ced580-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-11-02T18:30:26.570 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.570 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.571 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.571 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.572 DEBUG:boto:Method: GET 2023-11-02T18:30:26.572 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.573 DEBUG:boto:Data: 2023-11-02T18:30:26.573 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.574 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.575 DEBUG:boto:Params: {} 2023-11-02T18:30:26.575 DEBUG:boto:Token: None 2023-11-02T18:30:26.576 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.576 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YRzYRbJmoa/1K7PysY0HcJjileU= 2023-11-02T18:30:26.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YRzYRbJmoa/1K7PysY0HcJjileU=', 'Content-Length': '0'} 2023-11-02T18:30:26.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d58c24187446be-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.622 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.623 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.623 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.624 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.624 DEBUG:boto:Method: GET 2023-11-02T18:30:26.625 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.625 DEBUG:boto:Data: 2023-11-02T18:30:26.626 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.626 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.627 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.627 DEBUG:boto:Params: {} 2023-11-02T18:30:26.627 DEBUG:boto:Token: None 2023-11-02T18:30:26.628 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000006?versionId=null 2023-11-02T18:30:26.629 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YRzYRbJmoa/1K7PysY0HcJjileU= 2023-11-02T18:30:26.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YRzYRbJmoa/1K7PysY0HcJjileU=', 'Content-Length': '0'} 2023-11-02T18:30:26.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000561aba7711262a50-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.633 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.634 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.634 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:26.635 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:26.635 DEBUG:boto:Method: GET 2023-11-02T18:30:26.636 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:26.636 DEBUG:boto:Data: 2023-11-02T18:30:26.637 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.638 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.638 DEBUG:boto:Params: {} 2023-11-02T18:30:26.639 DEBUG:boto:Token: None 2023-11-02T18:30:26.639 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.640 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028092f1b45c31b2a-006543eac2-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.646 DEBUG:boto:b'ibtljn-58obj000000000000000000061000falseobj000000000000000000062023-11-02T18:30:19.621Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.647 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.647 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.648 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:26.648 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:26.649 DEBUG:boto:Method: GET 2023-11-02T18:30:26.649 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:26.650 DEBUG:boto:Data: 2023-11-02T18:30:26.650 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.651 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.652 DEBUG:boto:Params: {} 2023-11-02T18:30:26.652 DEBUG:boto:Token: None 2023-11-02T18:30:26.653 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bebf2eb0f8339b0-006543eac2-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.706 DEBUG:boto:b'ibtljn-58obj000000000000000000061000falseobj000000000000000000062023-11-02T18:30:19.621Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.707 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.708 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.708 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.709 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.709 DEBUG:boto:Data: 2023-11-02T18:30:26.710 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.710 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.711 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.711 DEBUG:boto:Params: {} 2023-11-02T18:30:26.712 DEBUG:boto:Token: None 2023-11-02T18:30:26.712 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.713 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ciJPSJ2SJvYKo+XPgMvZGkmfgyk= 2023-11-02T18:30:26.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ciJPSJ2SJvYKo+XPgMvZGkmfgyk=', 'Content-Length': '0'} 2023-11-02T18:30:26.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082564886d4101b9c-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.717 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.717 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.718 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.718 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.719 DEBUG:boto:Data: 2023-11-02T18:30:26.719 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.719 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.720 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.720 DEBUG:boto:Params: {} 2023-11-02T18:30:26.721 DEBUG:boto:Token: None 2023-11-02T18:30:26.721 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.722 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ciJPSJ2SJvYKo+XPgMvZGkmfgyk= 2023-11-02T18:30:26.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ciJPSJ2SJvYKo+XPgMvZGkmfgyk=', 'Content-Length': '0'} 2023-11-02T18:30:26.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a4496b430f350dd-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-11-02T18:30:26.727 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.727 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.728 DEBUG:boto:Method: GET 2023-11-02T18:30:26.728 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.728 DEBUG:boto:Data: 2023-11-02T18:30:26.729 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.730 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.730 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.730 DEBUG:boto:Params: {} 2023-11-02T18:30:26.731 DEBUG:boto:Token: None 2023-11-02T18:30:26.731 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.732 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RR+pcVzw8goUd1rqH/hun/mblB8= 2023-11-02T18:30:26.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RR+pcVzw8goUd1rqH/hun/mblB8=', 'Content-Length': '0'} 2023-11-02T18:30:26.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f9971292743548f-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.736 DEBUG:boto:path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.737 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.737 DEBUG:boto:Method: GET 2023-11-02T18:30:26.738 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.738 DEBUG:boto:Data: 2023-11-02T18:30:26.738 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.739 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.739 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.740 DEBUG:boto:Params: {} 2023-11-02T18:30:26.741 DEBUG:boto:Token: None 2023-11-02T18:30:26.741 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000006 2023-11-02T18:30:26.742 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RR+pcVzw8goUd1rqH/hun/mblB8= 2023-11-02T18:30:26.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RR+pcVzw8goUd1rqH/hun/mblB8=', 'Content-Length': '0'} 2023-11-02T18:30:26.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f01bb55395cf69c-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.746 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.746 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.746 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:26.747 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:26.747 DEBUG:boto:Method: GET 2023-11-02T18:30:26.748 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:26.748 DEBUG:boto:Data: 2023-11-02T18:30:26.749 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.749 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.750 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.750 DEBUG:boto:Params: {} 2023-11-02T18:30:26.751 DEBUG:boto:Token: None 2023-11-02T18:30:26.751 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.752 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fe430777cc59897-006543eac2-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.798 DEBUG:boto:b'ibtljn-58obj000000000000000000061000falseobj000000000000000000062023-11-02T18:30:19.621Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.799 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.800 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.800 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:26.801 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:26.801 DEBUG:boto:Method: GET 2023-11-02T18:30:26.802 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000006 2023-11-02T18:30:26.802 DEBUG:boto:Data: 2023-11-02T18:30:26.803 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.803 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.804 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.804 DEBUG:boto:Params: {} 2023-11-02T18:30:26.805 DEBUG:boto:Token: None 2023-11-02T18:30:26.805 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.806 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058fcd31a991cf708-006543eac2-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.817 DEBUG:boto:b'ibtljn-58obj000000000000000000061000falseobj000000000000000000062023-11-02T18:30:19.621Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-11-02T18:30:26.818 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.819 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.819 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.820 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.820 DEBUG:boto:Method: GET 2023-11-02T18:30:26.821 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.821 DEBUG:boto:Data: 2023-11-02T18:30:26.822 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.822 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.825 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.826 DEBUG:boto:Params: {} 2023-11-02T18:30:26.826 DEBUG:boto:Token: None 2023-11-02T18:30:26.827 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.827 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/JDv4bi3T3NeR9opXHG6xliL8uM= 2023-11-02T18:30:26.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/JDv4bi3T3NeR9opXHG6xliL8uM=', 'Content-Length': '0'} 2023-11-02T18:30:26.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000324216fe58df97ea-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.833 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.834 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.834 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.835 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.835 DEBUG:boto:Method: GET 2023-11-02T18:30:26.836 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.836 DEBUG:boto:Data: 2023-11-02T18:30:26.836 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.837 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.837 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.838 DEBUG:boto:Params: {} 2023-11-02T18:30:26.838 DEBUG:boto:Token: None 2023-11-02T18:30:26.839 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.839 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/JDv4bi3T3NeR9opXHG6xliL8uM= 2023-11-02T18:30:26.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/JDv4bi3T3NeR9opXHG6xliL8uM=', 'Content-Length': '0'} 2023-11-02T18:30:26.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c97b37a04b2004d-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.843 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.844 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.844 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.845 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.845 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.846 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.846 DEBUG:boto:Data: 2023-11-02T18:30:26.846 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.847 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.847 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.848 DEBUG:boto:Params: {} 2023-11-02T18:30:26.848 DEBUG:boto:Token: None 2023-11-02T18:30:26.849 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.849 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Cf6h7F5OoPFqUl2+3Sw3qHmP1CQ= 2023-11-02T18:30:26.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Cf6h7F5OoPFqUl2+3Sw3qHmP1CQ=', 'Content-Length': '0'} 2023-11-02T18:30:26.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c2518ce1e6b35b6-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.853 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.854 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.854 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.854 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.855 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.855 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.856 DEBUG:boto:Data: 2023-11-02T18:30:26.856 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.857 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.857 DEBUG:boto:Params: {} 2023-11-02T18:30:26.858 DEBUG:boto:Token: None 2023-11-02T18:30:26.858 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.859 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Cf6h7F5OoPFqUl2+3Sw3qHmP1CQ= 2023-11-02T18:30:26.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Cf6h7F5OoPFqUl2+3Sw3qHmP1CQ=', 'Content-Length': '0'} 2023-11-02T18:30:26.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b19e36674d97c2a-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-11-02T18:30:26.863 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.864 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.864 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.865 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.865 DEBUG:boto:Method: GET 2023-11-02T18:30:26.866 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.866 DEBUG:boto:Data: 2023-11-02T18:30:26.867 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.867 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.868 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.868 DEBUG:boto:Params: {} 2023-11-02T18:30:26.869 DEBUG:boto:Token: None 2023-11-02T18:30:26.869 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.870 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/JDv4bi3T3NeR9opXHG6xliL8uM= 2023-11-02T18:30:26.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/JDv4bi3T3NeR9opXHG6xliL8uM=', 'Content-Length': '0'} 2023-11-02T18:30:26.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd953287e5ffa716-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.874 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.874 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.875 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.875 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.876 DEBUG:boto:Method: GET 2023-11-02T18:30:26.876 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.877 DEBUG:boto:Data: 2023-11-02T18:30:26.877 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.878 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.878 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.879 DEBUG:boto:Params: {} 2023-11-02T18:30:26.879 DEBUG:boto:Token: None 2023-11-02T18:30:26.880 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000007?versionId=null 2023-11-02T18:30:26.880 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/JDv4bi3T3NeR9opXHG6xliL8uM= 2023-11-02T18:30:26.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/JDv4bi3T3NeR9opXHG6xliL8uM=', 'Content-Length': '0'} 2023-11-02T18:30:26.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019dd71eccb4f91b0-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.926 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.927 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.928 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000007 2023-11-02T18:30:26.928 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000007 2023-11-02T18:30:26.929 DEBUG:boto:Method: GET 2023-11-02T18:30:26.929 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000007 2023-11-02T18:30:26.929 DEBUG:boto:Data: 2023-11-02T18:30:26.930 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.931 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.931 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.931 DEBUG:boto:Params: {} 2023-11-02T18:30:26.932 DEBUG:boto:Token: None 2023-11-02T18:30:26.933 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e54c5d7c5bed372d-006543eac2-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.939 DEBUG:boto:b'ibtljn-58obj000000000000000000071000falseobj000000000000000000072023-11-02T18:30:17.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.940 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:26.941 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:26.941 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000007 2023-11-02T18:30:26.942 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000007 2023-11-02T18:30:26.942 DEBUG:boto:Method: GET 2023-11-02T18:30:26.943 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000007 2023-11-02T18:30:26.943 DEBUG:boto:Data: 2023-11-02T18:30:26.944 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.944 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.945 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.945 DEBUG:boto:Params: {} 2023-11-02T18:30:26.946 DEBUG:boto:Token: None 2023-11-02T18:30:26.946 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/ 2023-11-02T18:30:26.947 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA= 2023-11-02T18:30:26.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMV8Hp9v+Q2zY2Ah5adzPKK0pbA=', 'Content-Length': '0'} 2023-11-02T18:30:26.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033b754c3435f435b-006543eac2-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.958 DEBUG:boto:b'ibtljn-58obj000000000000000000071000falseobj000000000000000000072023-11-02T18:30:17.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:26.959 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.959 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.959 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.960 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.960 DEBUG:boto:Data: 2023-11-02T18:30:26.961 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.961 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.962 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.962 DEBUG:boto:Params: {} 2023-11-02T18:30:26.963 DEBUG:boto:Token: None 2023-11-02T18:30:26.963 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.964 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EAxL0lXrcgV8XwV/+DaEjgnvVjI= 2023-11-02T18:30:26.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EAxL0lXrcgV8XwV/+DaEjgnvVjI=', 'Content-Length': '0'} 2023-11-02T18:30:26.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001905cd07339875fc-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.967 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.968 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.968 DEBUG:boto:Method: HEAD 2023-11-02T18:30:26.969 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.969 DEBUG:boto:Data: 2023-11-02T18:30:26.970 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.970 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:26.971 DEBUG:boto:Port: 8000 2023-11-02T18:30:26.971 DEBUG:boto:Params: {} 2023-11-02T18:30:26.972 DEBUG:boto:Token: None 2023-11-02T18:30:26.972 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.973 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EAxL0lXrcgV8XwV/+DaEjgnvVjI= 2023-11-02T18:30:26.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EAxL0lXrcgV8XwV/+DaEjgnvVjI=', 'Content-Length': '0'} 2023-11-02T18:30:26.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ab6d69ba8aceeae-006543eac2-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:26.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-11-02T18:30:26.977 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.977 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.978 DEBUG:boto:Method: GET 2023-11-02T18:30:26.979 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.979 DEBUG:boto:Data: 2023-11-02T18:30:26.980 DEBUG:boto:Headers: {} 2023-11-02T18:30:26.980 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:26.980 DEBUG:boto:Port: 8001 2023-11-02T18:30:26.981 DEBUG:boto:Params: {} 2023-11-02T18:30:26.981 DEBUG:boto:Token: None 2023-11-02T18:30:26.982 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:26 GMT /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:26.983 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ed0RKCDCdpk5sro8w0z2ZG+f0j0= 2023-11-02T18:30:26.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ed0RKCDCdpk5sro8w0z2ZG+f0j0=', 'Content-Length': '0'} 2023-11-02T18:30:26.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d6401584b9a6588-006543eac2-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.029 DEBUG:boto:path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:27.030 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000007 2023-11-02T18:30:27.031 DEBUG:boto:Method: GET 2023-11-02T18:30:27.031 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:27.032 DEBUG:boto:Data: 2023-11-02T18:30:27.032 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.033 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.033 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.034 DEBUG:boto:Params: {} 2023-11-02T18:30:27.035 DEBUG:boto:Token: None 2023-11-02T18:30:27.035 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000007 2023-11-02T18:30:27.036 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E9VUyAwoRoyQ1ZGm0s9ksD50RI8= 2023-11-02T18:30:27.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E9VUyAwoRoyQ1ZGm0s9ksD50RI8=', 'Content-Length': '0'} 2023-11-02T18:30:27.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d422adab10e81e83-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.042 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.042 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.043 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000007 2023-11-02T18:30:27.044 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000007 2023-11-02T18:30:27.044 DEBUG:boto:Method: GET 2023-11-02T18:30:27.045 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000007 2023-11-02T18:30:27.045 DEBUG:boto:Data: 2023-11-02T18:30:27.047 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.048 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.048 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.049 DEBUG:boto:Params: {} 2023-11-02T18:30:27.049 DEBUG:boto:Token: None 2023-11-02T18:30:27.050 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fc50e43054412d8-006543eac3-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.056 DEBUG:boto:b'ibtljn-58obj000000000000000000071000falseobj000000000000000000072023-11-02T18:30:17.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.057 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.058 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.058 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000007 2023-11-02T18:30:27.058 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000007 2023-11-02T18:30:27.059 DEBUG:boto:Method: GET 2023-11-02T18:30:27.059 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000007 2023-11-02T18:30:27.060 DEBUG:boto:Data: 2023-11-02T18:30:27.060 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.061 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.061 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.061 DEBUG:boto:Params: {} 2023-11-02T18:30:27.062 DEBUG:boto:Token: None 2023-11-02T18:30:27.062 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.063 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d2e0fd00169684c-006543eac3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.114 DEBUG:boto:b'ibtljn-58obj000000000000000000071000falseobj000000000000000000072023-11-02T18:30:17.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-11-02T18:30:27.116 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.116 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.117 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.117 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.118 DEBUG:boto:Method: GET 2023-11-02T18:30:27.118 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.119 DEBUG:boto:Data: 2023-11-02T18:30:27.119 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.120 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.120 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.121 DEBUG:boto:Params: {} 2023-11-02T18:30:27.121 DEBUG:boto:Token: None 2023-11-02T18:30:27.122 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.122 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tmf+EkPEKkxtSXCL9aRgaXcvOck= 2023-11-02T18:30:27.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tmf+EkPEKkxtSXCL9aRgaXcvOck=', 'Content-Length': '0'} 2023-11-02T18:30:27.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d97a091a5018d65-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.130 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.131 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.131 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.132 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.132 DEBUG:boto:Method: GET 2023-11-02T18:30:27.133 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.133 DEBUG:boto:Data: 2023-11-02T18:30:27.134 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.134 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.134 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.135 DEBUG:boto:Params: {} 2023-11-02T18:30:27.135 DEBUG:boto:Token: None 2023-11-02T18:30:27.136 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.136 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tmf+EkPEKkxtSXCL9aRgaXcvOck= 2023-11-02T18:30:27.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tmf+EkPEKkxtSXCL9aRgaXcvOck=', 'Content-Length': '0'} 2023-11-02T18:30:27.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd4f448068faf1ba-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.141 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.141 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.142 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.142 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.142 DEBUG:boto:Method: HEAD 2023-11-02T18:30:27.143 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.143 DEBUG:boto:Data: 2023-11-02T18:30:27.144 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.144 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.145 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.145 DEBUG:boto:Params: {} 2023-11-02T18:30:27.146 DEBUG:boto:Token: None 2023-11-02T18:30:27.146 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.147 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:scrvVA0qNqHTljJgPYDii4QoCWE= 2023-11-02T18:30:27.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:scrvVA0qNqHTljJgPYDii4QoCWE=', 'Content-Length': '0'} 2023-11-02T18:30:27.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d703e54f26fbcb-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.151 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.151 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.151 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.152 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.152 DEBUG:boto:Method: HEAD 2023-11-02T18:30:27.153 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.153 DEBUG:boto:Data: 2023-11-02T18:30:27.154 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.154 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.154 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.155 DEBUG:boto:Params: {} 2023-11-02T18:30:27.155 DEBUG:boto:Token: None 2023-11-02T18:30:27.156 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.156 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:scrvVA0qNqHTljJgPYDii4QoCWE= 2023-11-02T18:30:27.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:scrvVA0qNqHTljJgPYDii4QoCWE=', 'Content-Length': '0'} 2023-11-02T18:30:27.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e464aa4c88421c05-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-11-02T18:30:27.160 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.161 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.161 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.162 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.162 DEBUG:boto:Method: GET 2023-11-02T18:30:27.163 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.163 DEBUG:boto:Data: 2023-11-02T18:30:27.164 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.164 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.164 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.165 DEBUG:boto:Params: {} 2023-11-02T18:30:27.165 DEBUG:boto:Token: None 2023-11-02T18:30:27.166 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.166 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tmf+EkPEKkxtSXCL9aRgaXcvOck= 2023-11-02T18:30:27.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tmf+EkPEKkxtSXCL9aRgaXcvOck=', 'Content-Length': '0'} 2023-11-02T18:30:27.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009725a37b738bdac7-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.171 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.171 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.171 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.172 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.172 DEBUG:boto:Method: GET 2023-11-02T18:30:27.173 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.173 DEBUG:boto:Data: 2023-11-02T18:30:27.174 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.174 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.175 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.175 DEBUG:boto:Params: {} 2023-11-02T18:30:27.176 DEBUG:boto:Token: None 2023-11-02T18:30:27.176 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000008?versionId=null 2023-11-02T18:30:27.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tmf+EkPEKkxtSXCL9aRgaXcvOck= 2023-11-02T18:30:27.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tmf+EkPEKkxtSXCL9aRgaXcvOck=', 'Content-Length': '0'} 2023-11-02T18:30:27.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065362e4a7b5b3118-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.222 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.223 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.223 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000008 2023-11-02T18:30:27.224 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000008 2023-11-02T18:30:27.224 DEBUG:boto:Method: GET 2023-11-02T18:30:27.225 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000008 2023-11-02T18:30:27.225 DEBUG:boto:Data: 2023-11-02T18:30:27.226 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.226 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.226 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.227 DEBUG:boto:Params: {} 2023-11-02T18:30:27.227 DEBUG:boto:Token: None 2023-11-02T18:30:27.228 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.228 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a79111b52c017663-006543eac3-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.234 DEBUG:boto:b'ibtljn-58obj000000000000000000081000falseobj000000000000000000082023-11-02T18:30:20.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.235 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.236 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.236 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000008 2023-11-02T18:30:27.236 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000008 2023-11-02T18:30:27.237 DEBUG:boto:Method: GET 2023-11-02T18:30:27.237 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000008 2023-11-02T18:30:27.238 DEBUG:boto:Data: 2023-11-02T18:30:27.238 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.239 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.239 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.240 DEBUG:boto:Params: {} 2023-11-02T18:30:27.240 DEBUG:boto:Token: None 2023-11-02T18:30:27.241 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e78f37a438e2c10f-006543eac3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.252 DEBUG:boto:b'ibtljn-58obj000000000000000000081000falseobj000000000000000000082023-11-02T18:30:20.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.252 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.253 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.253 DEBUG:boto:Method: HEAD 2023-11-02T18:30:27.254 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.254 DEBUG:boto:Data: 2023-11-02T18:30:27.254 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.255 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.255 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.256 DEBUG:boto:Params: {} 2023-11-02T18:30:27.256 DEBUG:boto:Token: None 2023-11-02T18:30:27.257 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.257 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XjsBE26YbLCTybk0h/60UKoQZvU= 2023-11-02T18:30:27.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XjsBE26YbLCTybk0h/60UKoQZvU=', 'Content-Length': '0'} 2023-11-02T18:30:27.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000201f2f613e42f948-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.261 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.261 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.262 DEBUG:boto:Method: HEAD 2023-11-02T18:30:27.262 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.263 DEBUG:boto:Data: 2023-11-02T18:30:27.263 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.264 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.264 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.265 DEBUG:boto:Params: {} 2023-11-02T18:30:27.265 DEBUG:boto:Token: None 2023-11-02T18:30:27.266 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.266 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XjsBE26YbLCTybk0h/60UKoQZvU= 2023-11-02T18:30:27.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XjsBE26YbLCTybk0h/60UKoQZvU=', 'Content-Length': '0'} 2023-11-02T18:30:27.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000560eac686278ae93-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-11-02T18:30:27.270 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.270 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.271 DEBUG:boto:Method: GET 2023-11-02T18:30:27.271 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.272 DEBUG:boto:Data: 2023-11-02T18:30:27.272 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.273 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.273 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.274 DEBUG:boto:Params: {} 2023-11-02T18:30:27.274 DEBUG:boto:Token: None 2023-11-02T18:30:27.275 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.275 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pna1lTkh6PGI+pYBZ4zrYkftW14= 2023-11-02T18:30:27.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pna1lTkh6PGI+pYBZ4zrYkftW14=', 'Content-Length': '0'} 2023-11-02T18:30:27.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032bf38bc054f51b1-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.322 DEBUG:boto:path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.323 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.323 DEBUG:boto:Method: GET 2023-11-02T18:30:27.324 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.324 DEBUG:boto:Data: 2023-11-02T18:30:27.325 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.325 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.326 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.326 DEBUG:boto:Params: {} 2023-11-02T18:30:27.327 DEBUG:boto:Token: None 2023-11-02T18:30:27.327 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000008 2023-11-02T18:30:27.328 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pna1lTkh6PGI+pYBZ4zrYkftW14= 2023-11-02T18:30:27.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pna1lTkh6PGI+pYBZ4zrYkftW14=', 'Content-Length': '0'} 2023-11-02T18:30:27.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066992bf609c74897-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.332 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.333 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.334 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000008 2023-11-02T18:30:27.334 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000008 2023-11-02T18:30:27.335 DEBUG:boto:Method: GET 2023-11-02T18:30:27.336 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000008 2023-11-02T18:30:27.336 DEBUG:boto:Data: 2023-11-02T18:30:27.337 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.337 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.338 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.338 DEBUG:boto:Params: {} 2023-11-02T18:30:27.339 DEBUG:boto:Token: None 2023-11-02T18:30:27.339 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.340 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f61ea6ea544a0de0-006543eac3-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.346 DEBUG:boto:b'ibtljn-58obj000000000000000000081000falseobj000000000000000000082023-11-02T18:30:20.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.347 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.348 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.348 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000008 2023-11-02T18:30:27.349 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000008 2023-11-02T18:30:27.349 DEBUG:boto:Method: GET 2023-11-02T18:30:27.349 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000008 2023-11-02T18:30:27.350 DEBUG:boto:Data: 2023-11-02T18:30:27.350 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.351 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.351 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.352 DEBUG:boto:Params: {} 2023-11-02T18:30:27.352 DEBUG:boto:Token: None 2023-11-02T18:30:27.353 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.353 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081a44b280bf58215-006543eac3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.406 DEBUG:boto:b'ibtljn-58obj000000000000000000081000falseobj000000000000000000082023-11-02T18:30:20.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-11-02T18:30:27.408 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.408 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.409 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.409 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.409 DEBUG:boto:Method: GET 2023-11-02T18:30:27.410 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.410 DEBUG:boto:Data: 2023-11-02T18:30:27.411 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.411 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.412 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.412 DEBUG:boto:Params: {} 2023-11-02T18:30:27.413 DEBUG:boto:Token: None 2023-11-02T18:30:27.413 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zKlCQ48j+T0SmvXXIFZ6OGUrbHQ= 2023-11-02T18:30:27.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zKlCQ48j+T0SmvXXIFZ6OGUrbHQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019cb1693e1429ee3-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.418 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.419 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.419 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.420 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.420 DEBUG:boto:Method: GET 2023-11-02T18:30:27.421 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.421 DEBUG:boto:Data: 2023-11-02T18:30:27.422 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.422 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.423 DEBUG:boto:Params: {} 2023-11-02T18:30:27.423 DEBUG:boto:Token: None 2023-11-02T18:30:27.424 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zKlCQ48j+T0SmvXXIFZ6OGUrbHQ= 2023-11-02T18:30:27.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zKlCQ48j+T0SmvXXIFZ6OGUrbHQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c0bdbbd8ce1d878-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.429 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.429 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.430 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.430 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.431 DEBUG:boto:Method: HEAD 2023-11-02T18:30:27.431 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.431 DEBUG:boto:Data: 2023-11-02T18:30:27.432 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.432 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.433 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.433 DEBUG:boto:Params: {} 2023-11-02T18:30:27.434 DEBUG:boto:Token: None 2023-11-02T18:30:27.434 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.435 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zwPcKlleDABbtm4p3IBBrRDxGHM= 2023-11-02T18:30:27.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zwPcKlleDABbtm4p3IBBrRDxGHM=', 'Content-Length': '0'} 2023-11-02T18:30:27.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000111e5ef17d5d5984-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.439 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.439 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.441 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.442 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.442 DEBUG:boto:Method: HEAD 2023-11-02T18:30:27.443 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.443 DEBUG:boto:Data: 2023-11-02T18:30:27.444 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.444 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.444 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.445 DEBUG:boto:Params: {} 2023-11-02T18:30:27.446 DEBUG:boto:Token: None 2023-11-02T18:30:27.446 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.447 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zwPcKlleDABbtm4p3IBBrRDxGHM= 2023-11-02T18:30:27.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zwPcKlleDABbtm4p3IBBrRDxGHM=', 'Content-Length': '0'} 2023-11-02T18:30:27.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf7108c2ae2b22ed-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-11-02T18:30:27.452 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.452 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.453 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.453 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.454 DEBUG:boto:Method: GET 2023-11-02T18:30:27.454 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.455 DEBUG:boto:Data: 2023-11-02T18:30:27.455 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.456 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.456 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.457 DEBUG:boto:Params: {} 2023-11-02T18:30:27.457 DEBUG:boto:Token: None 2023-11-02T18:30:27.458 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.458 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zKlCQ48j+T0SmvXXIFZ6OGUrbHQ= 2023-11-02T18:30:27.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zKlCQ48j+T0SmvXXIFZ6OGUrbHQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c946aab9d7cda510-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.462 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.463 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.463 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.464 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.464 DEBUG:boto:Method: GET 2023-11-02T18:30:27.465 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.465 DEBUG:boto:Data: 2023-11-02T18:30:27.466 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.466 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.467 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.467 DEBUG:boto:Params: {} 2023-11-02T18:30:27.468 DEBUG:boto:Token: None 2023-11-02T18:30:27.468 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000009?versionId=null 2023-11-02T18:30:27.469 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zKlCQ48j+T0SmvXXIFZ6OGUrbHQ= 2023-11-02T18:30:27.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zKlCQ48j+T0SmvXXIFZ6OGUrbHQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006909a389cb7a563a-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.514 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.515 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.515 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000009 2023-11-02T18:30:27.516 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000009 2023-11-02T18:30:27.516 DEBUG:boto:Method: GET 2023-11-02T18:30:27.517 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000009 2023-11-02T18:30:27.517 DEBUG:boto:Data: 2023-11-02T18:30:27.518 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.518 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.519 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.519 DEBUG:boto:Params: {} 2023-11-02T18:30:27.520 DEBUG:boto:Token: None 2023-11-02T18:30:27.520 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013b4f3374ac40a68-006543eac3-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.527 DEBUG:boto:b'ibtljn-58obj000000000000000000091000falseobj000000000000000000092023-11-02T18:30:19.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.528 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.528 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.529 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000009 2023-11-02T18:30:27.529 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000009 2023-11-02T18:30:27.530 DEBUG:boto:Method: GET 2023-11-02T18:30:27.530 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000009 2023-11-02T18:30:27.531 DEBUG:boto:Data: 2023-11-02T18:30:27.531 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.532 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.532 DEBUG:boto:Params: {} 2023-11-02T18:30:27.533 DEBUG:boto:Token: None 2023-11-02T18:30:27.534 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.534 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063f0f75d323dc6a3-006543eac3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.545 DEBUG:boto:b'ibtljn-58obj000000000000000000091000falseobj000000000000000000092023-11-02T18:30:19.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.546 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.546 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.547 DEBUG:boto:Method: HEAD 2023-11-02T18:30:27.547 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.548 DEBUG:boto:Data: 2023-11-02T18:30:27.548 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.549 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.549 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.550 DEBUG:boto:Params: {} 2023-11-02T18:30:27.550 DEBUG:boto:Token: None 2023-11-02T18:30:27.551 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.551 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BR6pxTATxIBNnlK+xVcmbmWbofo= 2023-11-02T18:30:27.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BR6pxTATxIBNnlK+xVcmbmWbofo=', 'Content-Length': '0'} 2023-11-02T18:30:27.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e3d255967caef38-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.555 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.555 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.556 DEBUG:boto:Method: HEAD 2023-11-02T18:30:27.556 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.556 DEBUG:boto:Data: 2023-11-02T18:30:27.557 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.557 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.558 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.558 DEBUG:boto:Params: {} 2023-11-02T18:30:27.559 DEBUG:boto:Token: None 2023-11-02T18:30:27.559 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.560 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BR6pxTATxIBNnlK+xVcmbmWbofo= 2023-11-02T18:30:27.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BR6pxTATxIBNnlK+xVcmbmWbofo=', 'Content-Length': '0'} 2023-11-02T18:30:27.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c22b667e56017213-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-11-02T18:30:27.563 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.564 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.564 DEBUG:boto:Method: GET 2023-11-02T18:30:27.565 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.565 DEBUG:boto:Data: 2023-11-02T18:30:27.566 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.567 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.567 DEBUG:boto:Params: {} 2023-11-02T18:30:27.568 DEBUG:boto:Token: None 2023-11-02T18:30:27.568 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.568 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hyaLeYHE0J2+wj9lNT+VH7F2XGQ= 2023-11-02T18:30:27.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hyaLeYHE0J2+wj9lNT+VH7F2XGQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edd5f17aba6d4300-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.573 DEBUG:boto:path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.573 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.574 DEBUG:boto:Method: GET 2023-11-02T18:30:27.574 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.575 DEBUG:boto:Data: 2023-11-02T18:30:27.575 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.576 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.576 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.577 DEBUG:boto:Params: {} 2023-11-02T18:30:27.577 DEBUG:boto:Token: None 2023-11-02T18:30:27.578 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000009 2023-11-02T18:30:27.578 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hyaLeYHE0J2+wj9lNT+VH7F2XGQ= 2023-11-02T18:30:27.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hyaLeYHE0J2+wj9lNT+VH7F2XGQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e108b168ff4fbc50-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.626 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.627 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.627 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000009 2023-11-02T18:30:27.628 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000009 2023-11-02T18:30:27.628 DEBUG:boto:Method: GET 2023-11-02T18:30:27.629 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000009 2023-11-02T18:30:27.629 DEBUG:boto:Data: 2023-11-02T18:30:27.630 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.630 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.631 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.631 DEBUG:boto:Params: {} 2023-11-02T18:30:27.631 DEBUG:boto:Token: None 2023-11-02T18:30:27.632 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.632 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c96780e9aee61f12-006543eac3-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.639 DEBUG:boto:b'ibtljn-58obj000000000000000000091000falseobj000000000000000000092023-11-02T18:30:19.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.639 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.640 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.640 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000009 2023-11-02T18:30:27.641 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000009 2023-11-02T18:30:27.641 DEBUG:boto:Method: GET 2023-11-02T18:30:27.642 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000009 2023-11-02T18:30:27.642 DEBUG:boto:Data: 2023-11-02T18:30:27.643 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.646 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.646 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.646 DEBUG:boto:Params: {} 2023-11-02T18:30:27.647 DEBUG:boto:Token: None 2023-11-02T18:30:27.647 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.648 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e9e2bd5ea28e791-006543eac3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.660 DEBUG:boto:b'ibtljn-58obj000000000000000000091000falseobj000000000000000000092023-11-02T18:30:19.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-11-02T18:30:27.661 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.661 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.662 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.662 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.663 DEBUG:boto:Method: GET 2023-11-02T18:30:27.663 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.663 DEBUG:boto:Data: 2023-11-02T18:30:27.664 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.664 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.665 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.665 DEBUG:boto:Params: {} 2023-11-02T18:30:27.666 DEBUG:boto:Token: None 2023-11-02T18:30:27.666 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.667 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OKRw3DmCRWgrli6QxGRpJNjYDsQ= 2023-11-02T18:30:27.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OKRw3DmCRWgrli6QxGRpJNjYDsQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abccad53e5ded3b1-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.710 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.711 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.711 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.712 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.712 DEBUG:boto:Method: GET 2023-11-02T18:30:27.713 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.713 DEBUG:boto:Data: 2023-11-02T18:30:27.714 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.714 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.715 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.715 DEBUG:boto:Params: {} 2023-11-02T18:30:27.716 DEBUG:boto:Token: None 2023-11-02T18:30:27.716 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.717 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OKRw3DmCRWgrli6QxGRpJNjYDsQ= 2023-11-02T18:30:27.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OKRw3DmCRWgrli6QxGRpJNjYDsQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003571f660c4059528-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.721 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.722 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.722 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.722 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.723 DEBUG:boto:Method: HEAD 2023-11-02T18:30:27.723 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.724 DEBUG:boto:Data: 2023-11-02T18:30:27.724 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.725 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.725 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.725 DEBUG:boto:Params: {} 2023-11-02T18:30:27.726 DEBUG:boto:Token: None 2023-11-02T18:30:27.726 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.727 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hJA1PzpDzZSvVeZtn+QbxiOBbTE= 2023-11-02T18:30:27.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hJA1PzpDzZSvVeZtn+QbxiOBbTE=', 'Content-Length': '0'} 2023-11-02T18:30:27.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015b9cf3edcc37d25-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.731 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.731 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.732 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.732 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.732 DEBUG:boto:Method: HEAD 2023-11-02T18:30:27.733 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.733 DEBUG:boto:Data: 2023-11-02T18:30:27.734 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.734 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.734 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.735 DEBUG:boto:Params: {} 2023-11-02T18:30:27.735 DEBUG:boto:Token: None 2023-11-02T18:30:27.736 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.736 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hJA1PzpDzZSvVeZtn+QbxiOBbTE= 2023-11-02T18:30:27.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hJA1PzpDzZSvVeZtn+QbxiOBbTE=', 'Content-Length': '0'} 2023-11-02T18:30:27.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ec595bd60be98ac-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-11-02T18:30:27.740 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.741 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.741 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.742 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.742 DEBUG:boto:Method: GET 2023-11-02T18:30:27.743 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.743 DEBUG:boto:Data: 2023-11-02T18:30:27.744 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.744 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.744 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.745 DEBUG:boto:Params: {} 2023-11-02T18:30:27.745 DEBUG:boto:Token: None 2023-11-02T18:30:27.746 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.746 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OKRw3DmCRWgrli6QxGRpJNjYDsQ= 2023-11-02T18:30:27.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OKRw3DmCRWgrli6QxGRpJNjYDsQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a97e6d4fb07e0286-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.750 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.751 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.751 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.752 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.752 DEBUG:boto:Method: GET 2023-11-02T18:30:27.753 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.753 DEBUG:boto:Data: 2023-11-02T18:30:27.754 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.755 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.755 DEBUG:boto:Params: {} 2023-11-02T18:30:27.755 DEBUG:boto:Token: None 2023-11-02T18:30:27.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000010?versionId=null 2023-11-02T18:30:27.756 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OKRw3DmCRWgrli6QxGRpJNjYDsQ= 2023-11-02T18:30:27.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OKRw3DmCRWgrli6QxGRpJNjYDsQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8aeb24da7c0f3b5-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.760 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.761 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.761 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:27.762 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:27.762 DEBUG:boto:Method: GET 2023-11-02T18:30:27.763 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:27.763 DEBUG:boto:Data: 2023-11-02T18:30:27.763 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.764 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.765 DEBUG:boto:Params: {} 2023-11-02T18:30:27.765 DEBUG:boto:Token: None 2023-11-02T18:30:27.766 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015e6d825f5564519-006543eac3-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.773 DEBUG:boto:b'ibtljn-58obj000000000000000000101000falseobj000000000000000000102023-11-02T18:30:19.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.773 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.774 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.774 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:27.775 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:27.775 DEBUG:boto:Method: GET 2023-11-02T18:30:27.776 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:27.776 DEBUG:boto:Data: 2023-11-02T18:30:27.777 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.777 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.777 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.778 DEBUG:boto:Params: {} 2023-11-02T18:30:27.778 DEBUG:boto:Token: None 2023-11-02T18:30:27.779 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.779 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000193afc587a322db-006543eac3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.834 DEBUG:boto:b'ibtljn-58obj000000000000000000101000falseobj000000000000000000102023-11-02T18:30:19.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.835 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.836 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.836 DEBUG:boto:Method: HEAD 2023-11-02T18:30:27.837 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.837 DEBUG:boto:Data: 2023-11-02T18:30:27.837 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.838 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.838 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.839 DEBUG:boto:Params: {} 2023-11-02T18:30:27.839 DEBUG:boto:Token: None 2023-11-02T18:30:27.840 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.840 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y8M6d1UDp2FWqFM7S8q3VH20oIQ= 2023-11-02T18:30:27.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y8M6d1UDp2FWqFM7S8q3VH20oIQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000664ce5ceebd50c03-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.844 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.845 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.845 DEBUG:boto:Method: HEAD 2023-11-02T18:30:27.846 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.846 DEBUG:boto:Data: 2023-11-02T18:30:27.846 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.847 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.847 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.848 DEBUG:boto:Params: {} 2023-11-02T18:30:27.849 DEBUG:boto:Token: None 2023-11-02T18:30:27.850 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.850 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y8M6d1UDp2FWqFM7S8q3VH20oIQ= 2023-11-02T18:30:27.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y8M6d1UDp2FWqFM7S8q3VH20oIQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053299bd2d4c55fe4-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-11-02T18:30:27.855 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.855 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.856 DEBUG:boto:Method: GET 2023-11-02T18:30:27.856 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.857 DEBUG:boto:Data: 2023-11-02T18:30:27.857 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.858 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.858 DEBUG:boto:Params: {} 2023-11-02T18:30:27.859 DEBUG:boto:Token: None 2023-11-02T18:30:27.859 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.860 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KyIaAt/e2TpwmhlN7YlzbB8VphQ= 2023-11-02T18:30:27.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KyIaAt/e2TpwmhlN7YlzbB8VphQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db098b1bb2f4fb42-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.906 DEBUG:boto:path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.907 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.907 DEBUG:boto:Method: GET 2023-11-02T18:30:27.908 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.908 DEBUG:boto:Data: 2023-11-02T18:30:27.909 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.909 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.910 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.910 DEBUG:boto:Params: {} 2023-11-02T18:30:27.911 DEBUG:boto:Token: None 2023-11-02T18:30:27.911 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000010 2023-11-02T18:30:27.912 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KyIaAt/e2TpwmhlN7YlzbB8VphQ= 2023-11-02T18:30:27.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KyIaAt/e2TpwmhlN7YlzbB8VphQ=', 'Content-Length': '0'} 2023-11-02T18:30:27.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7a12de12f14c34b-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.916 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.916 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.917 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:27.917 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:27.917 DEBUG:boto:Method: GET 2023-11-02T18:30:27.918 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:27.918 DEBUG:boto:Data: 2023-11-02T18:30:27.919 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.919 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.920 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.920 DEBUG:boto:Params: {} 2023-11-02T18:30:27.921 DEBUG:boto:Token: None 2023-11-02T18:30:27.921 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5d11ca7101f8b8d-006543eac3-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.928 DEBUG:boto:b'ibtljn-58obj000000000000000000101000falseobj000000000000000000102023-11-02T18:30:19.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.929 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:27.929 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:27.929 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:27.930 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:27.930 DEBUG:boto:Method: GET 2023-11-02T18:30:27.931 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000010 2023-11-02T18:30:27.931 DEBUG:boto:Data: 2023-11-02T18:30:27.932 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.932 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.932 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.933 DEBUG:boto:Params: {} 2023-11-02T18:30:27.933 DEBUG:boto:Token: None 2023-11-02T18:30:27.934 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/ 2023-11-02T18:30:27.934 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA= 2023-11-02T18:30:27.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+h5KWqP334RRpQ8A1C8oJ8fTiA=', 'Content-Length': '0'} 2023-11-02T18:30:27.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef1b696124dce06f-006543eac3-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.946 DEBUG:boto:b'ibtljn-58obj000000000000000000101000falseobj000000000000000000102023-11-02T18:30:19.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:27.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-11-02T18:30:27.947 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:27.948 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:27.948 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:27.949 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:27.949 DEBUG:boto:Method: GET 2023-11-02T18:30:27.949 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:27.950 DEBUG:boto:Data: 2023-11-02T18:30:27.950 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.951 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:27.951 DEBUG:boto:Port: 8001 2023-11-02T18:30:27.952 DEBUG:boto:Params: {} 2023-11-02T18:30:27.952 DEBUG:boto:Token: None 2023-11-02T18:30:27.953 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:27.953 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s8N3cSQt6jzWKVBPY8TwvtKGt50= 2023-11-02T18:30:27.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s8N3cSQt6jzWKVBPY8TwvtKGt50=', 'Content-Length': '0'} 2023-11-02T18:30:27.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a294a0c90d6265f-006543eac3-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:27.957 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:27.958 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:27.958 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:27.959 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:27.959 DEBUG:boto:Method: GET 2023-11-02T18:30:27.960 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:27.960 DEBUG:boto:Data: 2023-11-02T18:30:27.961 DEBUG:boto:Headers: {} 2023-11-02T18:30:27.961 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:27.962 DEBUG:boto:Port: 8000 2023-11-02T18:30:27.962 DEBUG:boto:Params: {} 2023-11-02T18:30:27.963 DEBUG:boto:Token: None 2023-11-02T18:30:27.963 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:27 GMT /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:27.964 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s8N3cSQt6jzWKVBPY8TwvtKGt50= 2023-11-02T18:30:27.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s8N3cSQt6jzWKVBPY8TwvtKGt50=', 'Content-Length': '0'} 2023-11-02T18:30:27.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa750f642cb95af0-006543eac3-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.014 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.016 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.016 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.017 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.017 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.018 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.018 DEBUG:boto:Data: 2023-11-02T18:30:28.018 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.019 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.019 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.020 DEBUG:boto:Params: {} 2023-11-02T18:30:28.020 DEBUG:boto:Token: None 2023-11-02T18:30:28.021 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.021 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RXeX0BNLlCe2w5hNYmRSSv8P7R0= 2023-11-02T18:30:28.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RXeX0BNLlCe2w5hNYmRSSv8P7R0=', 'Content-Length': '0'} 2023-11-02T18:30:28.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f9fbd840228084d-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.026 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.027 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.027 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.027 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.028 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.028 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.029 DEBUG:boto:Data: 2023-11-02T18:30:28.029 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.030 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.031 DEBUG:boto:Params: {} 2023-11-02T18:30:28.031 DEBUG:boto:Token: None 2023-11-02T18:30:28.032 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RXeX0BNLlCe2w5hNYmRSSv8P7R0= 2023-11-02T18:30:28.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RXeX0BNLlCe2w5hNYmRSSv8P7R0=', 'Content-Length': '0'} 2023-11-02T18:30:28.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cdae3b019c9e43d-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-11-02T18:30:28.037 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.037 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.038 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.038 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.039 DEBUG:boto:Method: GET 2023-11-02T18:30:28.039 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.040 DEBUG:boto:Data: 2023-11-02T18:30:28.040 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.041 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.041 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.042 DEBUG:boto:Params: {} 2023-11-02T18:30:28.042 DEBUG:boto:Token: None 2023-11-02T18:30:28.043 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.043 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fe7F3ujq5AZ4YbGASJuJxzK2hbM= 2023-11-02T18:30:28.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fe7F3ujq5AZ4YbGASJuJxzK2hbM=', 'Content-Length': '0'} 2023-11-02T18:30:28.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f4d4151b4c249df-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.048 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.048 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.049 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.049 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.050 DEBUG:boto:Method: GET 2023-11-02T18:30:28.050 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.051 DEBUG:boto:Data: 2023-11-02T18:30:28.051 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.052 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.052 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.053 DEBUG:boto:Params: {} 2023-11-02T18:30:28.053 DEBUG:boto:Token: None 2023-11-02T18:30:28.054 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000011?versionId=null 2023-11-02T18:30:28.054 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fe7F3ujq5AZ4YbGASJuJxzK2hbM= 2023-11-02T18:30:28.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fe7F3ujq5AZ4YbGASJuJxzK2hbM=', 'Content-Length': '0'} 2023-11-02T18:30:28.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a023b2ac8ed8ec3-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.059 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.060 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.060 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:28.061 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:28.061 DEBUG:boto:Method: GET 2023-11-02T18:30:28.062 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:28.062 DEBUG:boto:Data: 2023-11-02T18:30:28.063 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.063 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.064 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.064 DEBUG:boto:Params: {} 2023-11-02T18:30:28.065 DEBUG:boto:Token: None 2023-11-02T18:30:28.065 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.066 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d0ce6d11fd96383-006543eac4-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.114 DEBUG:boto:b'ibtljn-58obj000000000000000000111000falseobj000000000000000000112023-11-02T18:30:19.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.115 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.116 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.116 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:28.117 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:28.117 DEBUG:boto:Method: GET 2023-11-02T18:30:28.118 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:28.118 DEBUG:boto:Data: 2023-11-02T18:30:28.119 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.119 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.119 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.120 DEBUG:boto:Params: {} 2023-11-02T18:30:28.120 DEBUG:boto:Token: None 2023-11-02T18:30:28.121 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.121 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1aa61118aeec6fc-006543eac4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.133 DEBUG:boto:b'ibtljn-58obj000000000000000000111000falseobj000000000000000000112023-11-02T18:30:19.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.134 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.134 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.135 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.135 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.136 DEBUG:boto:Data: 2023-11-02T18:30:28.136 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.137 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.137 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.137 DEBUG:boto:Params: {} 2023-11-02T18:30:28.138 DEBUG:boto:Token: None 2023-11-02T18:30:28.138 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.139 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9qRZ3CLJJ0dj6ZvMr5AzGK6RAvQ= 2023-11-02T18:30:28.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9qRZ3CLJJ0dj6ZvMr5AzGK6RAvQ=', 'Content-Length': '0'} 2023-11-02T18:30:28.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ff09bf11b94e50-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.142 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.143 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.143 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.143 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.144 DEBUG:boto:Data: 2023-11-02T18:30:28.144 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.145 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.145 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.145 DEBUG:boto:Params: {} 2023-11-02T18:30:28.146 DEBUG:boto:Token: None 2023-11-02T18:30:28.146 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.147 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9qRZ3CLJJ0dj6ZvMr5AzGK6RAvQ= 2023-11-02T18:30:28.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9qRZ3CLJJ0dj6ZvMr5AzGK6RAvQ=', 'Content-Length': '0'} 2023-11-02T18:30:28.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fd35acf1a91333f-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-11-02T18:30:28.151 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.151 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.152 DEBUG:boto:Method: GET 2023-11-02T18:30:28.152 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.153 DEBUG:boto:Data: 2023-11-02T18:30:28.153 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.154 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.154 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.155 DEBUG:boto:Params: {} 2023-11-02T18:30:28.155 DEBUG:boto:Token: None 2023-11-02T18:30:28.156 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.156 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:heQfdAynISvME92vhSJQJy5sb14= 2023-11-02T18:30:28.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:heQfdAynISvME92vhSJQJy5sb14=', 'Content-Length': '0'} 2023-11-02T18:30:28.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d89e773634d2c7f0-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.160 DEBUG:boto:path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.160 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.161 DEBUG:boto:Method: GET 2023-11-02T18:30:28.161 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.162 DEBUG:boto:Data: 2023-11-02T18:30:28.162 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.163 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.163 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.163 DEBUG:boto:Params: {} 2023-11-02T18:30:28.164 DEBUG:boto:Token: None 2023-11-02T18:30:28.164 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000011 2023-11-02T18:30:28.165 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:heQfdAynISvME92vhSJQJy5sb14= 2023-11-02T18:30:28.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:heQfdAynISvME92vhSJQJy5sb14=', 'Content-Length': '0'} 2023-11-02T18:30:28.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008851902424586dc6-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.210 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.211 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.211 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:28.212 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:28.212 DEBUG:boto:Method: GET 2023-11-02T18:30:28.213 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:28.213 DEBUG:boto:Data: 2023-11-02T18:30:28.214 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.214 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.215 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.215 DEBUG:boto:Params: {} 2023-11-02T18:30:28.215 DEBUG:boto:Token: None 2023-11-02T18:30:28.216 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.216 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4f0ba867d0c8bc2-006543eac4-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.223 DEBUG:boto:b'ibtljn-58obj000000000000000000111000falseobj000000000000000000112023-11-02T18:30:19.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.224 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.224 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.225 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:28.225 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:28.226 DEBUG:boto:Method: GET 2023-11-02T18:30:28.226 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000011 2023-11-02T18:30:28.227 DEBUG:boto:Data: 2023-11-02T18:30:28.227 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.227 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.228 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.228 DEBUG:boto:Params: {} 2023-11-02T18:30:28.229 DEBUG:boto:Token: None 2023-11-02T18:30:28.229 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.230 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009799f8279c82e97-006543eac4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.241 DEBUG:boto:b'ibtljn-58obj000000000000000000111000falseobj000000000000000000112023-11-02T18:30:19.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-11-02T18:30:28.242 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.243 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.243 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.244 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.244 DEBUG:boto:Method: GET 2023-11-02T18:30:28.244 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.245 DEBUG:boto:Data: 2023-11-02T18:30:28.245 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.246 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.247 DEBUG:boto:Params: {} 2023-11-02T18:30:28.247 DEBUG:boto:Token: None 2023-11-02T18:30:28.248 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.248 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LLZTATBsxQD7WLLujgYJ+HhWfB8= 2023-11-02T18:30:28.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LLZTATBsxQD7WLLujgYJ+HhWfB8=', 'Content-Length': '0'} 2023-11-02T18:30:28.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b32c93201a2eac39-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.252 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.252 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.253 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.253 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.254 DEBUG:boto:Method: GET 2023-11-02T18:30:28.254 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.255 DEBUG:boto:Data: 2023-11-02T18:30:28.257 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.258 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.258 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.259 DEBUG:boto:Params: {} 2023-11-02T18:30:28.259 DEBUG:boto:Token: None 2023-11-02T18:30:28.260 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.260 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LLZTATBsxQD7WLLujgYJ+HhWfB8= 2023-11-02T18:30:28.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LLZTATBsxQD7WLLujgYJ+HhWfB8=', 'Content-Length': '0'} 2023-11-02T18:30:28.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc677a4de9831995-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.264 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.265 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.265 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.265 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.266 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.266 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.267 DEBUG:boto:Data: 2023-11-02T18:30:28.267 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.267 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.268 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.268 DEBUG:boto:Params: {} 2023-11-02T18:30:28.269 DEBUG:boto:Token: None 2023-11-02T18:30:28.269 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.270 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1uTPzhj1u9B31LUmqsKKrYbpeoY= 2023-11-02T18:30:28.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1uTPzhj1u9B31LUmqsKKrYbpeoY=', 'Content-Length': '0'} 2023-11-02T18:30:28.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f63266df7539e9c6-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.274 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.274 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.275 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.275 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.275 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.276 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.276 DEBUG:boto:Data: 2023-11-02T18:30:28.277 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.277 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.278 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.278 DEBUG:boto:Params: {} 2023-11-02T18:30:28.279 DEBUG:boto:Token: None 2023-11-02T18:30:28.280 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.280 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1uTPzhj1u9B31LUmqsKKrYbpeoY= 2023-11-02T18:30:28.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1uTPzhj1u9B31LUmqsKKrYbpeoY=', 'Content-Length': '0'} 2023-11-02T18:30:28.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000240aa79cdeda3100-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-11-02T18:30:28.284 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.285 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.285 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.286 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.286 DEBUG:boto:Method: GET 2023-11-02T18:30:28.287 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.287 DEBUG:boto:Data: 2023-11-02T18:30:28.288 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.288 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.289 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.289 DEBUG:boto:Params: {} 2023-11-02T18:30:28.289 DEBUG:boto:Token: None 2023-11-02T18:30:28.290 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.290 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LLZTATBsxQD7WLLujgYJ+HhWfB8= 2023-11-02T18:30:28.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LLZTATBsxQD7WLLujgYJ+HhWfB8=', 'Content-Length': '0'} 2023-11-02T18:30:28.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e20c5779a86f43f-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.338 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.339 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.339 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.340 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.340 DEBUG:boto:Method: GET 2023-11-02T18:30:28.341 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.341 DEBUG:boto:Data: 2023-11-02T18:30:28.342 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.342 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.343 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.343 DEBUG:boto:Params: {} 2023-11-02T18:30:28.343 DEBUG:boto:Token: None 2023-11-02T18:30:28.344 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000012?versionId=null 2023-11-02T18:30:28.344 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LLZTATBsxQD7WLLujgYJ+HhWfB8= 2023-11-02T18:30:28.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LLZTATBsxQD7WLLujgYJ+HhWfB8=', 'Content-Length': '0'} 2023-11-02T18:30:28.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f82d6948fa2b53b-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.349 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.350 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.350 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:28.351 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:28.351 DEBUG:boto:Method: GET 2023-11-02T18:30:28.352 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:28.352 DEBUG:boto:Data: 2023-11-02T18:30:28.352 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.353 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.353 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.354 DEBUG:boto:Params: {} 2023-11-02T18:30:28.354 DEBUG:boto:Token: None 2023-11-02T18:30:28.355 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.355 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c869f5282729a70-006543eac4-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.362 DEBUG:boto:b'ibtljn-58obj000000000000000000121000falseobj000000000000000000122023-11-02T18:30:20.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.363 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.363 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.364 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:28.364 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:28.365 DEBUG:boto:Method: GET 2023-11-02T18:30:28.365 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:28.365 DEBUG:boto:Data: 2023-11-02T18:30:28.366 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.367 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.367 DEBUG:boto:Params: {} 2023-11-02T18:30:28.368 DEBUG:boto:Token: None 2023-11-02T18:30:28.368 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.369 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007332b0813cfb0857-006543eac4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.380 DEBUG:boto:b'ibtljn-58obj000000000000000000121000falseobj000000000000000000122023-11-02T18:30:20.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.381 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.381 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.382 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.382 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.383 DEBUG:boto:Data: 2023-11-02T18:30:28.383 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.384 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.384 DEBUG:boto:Params: {} 2023-11-02T18:30:28.385 DEBUG:boto:Token: None 2023-11-02T18:30:28.385 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.386 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ourLqJaMivNEpwBZPrvhS2kvdEQ= 2023-11-02T18:30:28.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ourLqJaMivNEpwBZPrvhS2kvdEQ=', 'Content-Length': '0'} 2023-11-02T18:30:28.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030869165e34dee8e-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.389 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.390 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.390 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.391 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.391 DEBUG:boto:Data: 2023-11-02T18:30:28.391 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.392 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.392 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.393 DEBUG:boto:Params: {} 2023-11-02T18:30:28.393 DEBUG:boto:Token: None 2023-11-02T18:30:28.394 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ourLqJaMivNEpwBZPrvhS2kvdEQ= 2023-11-02T18:30:28.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ourLqJaMivNEpwBZPrvhS2kvdEQ=', 'Content-Length': '0'} 2023-11-02T18:30:28.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a3c2c57201656d3-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-11-02T18:30:28.398 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.399 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.399 DEBUG:boto:Method: GET 2023-11-02T18:30:28.399 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.400 DEBUG:boto:Data: 2023-11-02T18:30:28.400 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.401 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.401 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.401 DEBUG:boto:Params: {} 2023-11-02T18:30:28.402 DEBUG:boto:Token: None 2023-11-02T18:30:28.402 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.403 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:raikCCRUl/M3eqJLVDl8+5X3pao= 2023-11-02T18:30:28.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:raikCCRUl/M3eqJLVDl8+5X3pao=', 'Content-Length': '0'} 2023-11-02T18:30:28.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea649644b9fc3fdc-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.407 DEBUG:boto:path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.407 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.408 DEBUG:boto:Method: GET 2023-11-02T18:30:28.408 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.409 DEBUG:boto:Data: 2023-11-02T18:30:28.409 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.410 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.410 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.411 DEBUG:boto:Params: {} 2023-11-02T18:30:28.411 DEBUG:boto:Token: None 2023-11-02T18:30:28.411 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000012 2023-11-02T18:30:28.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:raikCCRUl/M3eqJLVDl8+5X3pao= 2023-11-02T18:30:28.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:raikCCRUl/M3eqJLVDl8+5X3pao=', 'Content-Length': '0'} 2023-11-02T18:30:28.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d3c2e7b1f30bcf7-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.416 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.417 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.417 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:28.418 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:28.418 DEBUG:boto:Method: GET 2023-11-02T18:30:28.419 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:28.419 DEBUG:boto:Data: 2023-11-02T18:30:28.419 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.420 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.420 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.421 DEBUG:boto:Params: {} 2023-11-02T18:30:28.421 DEBUG:boto:Token: None 2023-11-02T18:30:28.422 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.422 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d042ed7572df2a93-006543eac4-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.470 DEBUG:boto:b'ibtljn-58obj000000000000000000121000falseobj000000000000000000122023-11-02T18:30:20.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.471 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.472 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.472 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:28.473 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:28.473 DEBUG:boto:Method: GET 2023-11-02T18:30:28.473 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000012 2023-11-02T18:30:28.474 DEBUG:boto:Data: 2023-11-02T18:30:28.474 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.475 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.475 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.476 DEBUG:boto:Params: {} 2023-11-02T18:30:28.476 DEBUG:boto:Token: None 2023-11-02T18:30:28.477 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.477 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c2f1397dca3228e-006543eac4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.488 DEBUG:boto:b'ibtljn-58obj000000000000000000121000falseobj000000000000000000122023-11-02T18:30:20.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-11-02T18:30:28.489 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.490 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.490 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.491 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.491 DEBUG:boto:Method: GET 2023-11-02T18:30:28.492 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.492 DEBUG:boto:Data: 2023-11-02T18:30:28.493 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.493 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.494 DEBUG:boto:Params: {} 2023-11-02T18:30:28.494 DEBUG:boto:Token: None 2023-11-02T18:30:28.495 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.495 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wOT2zqzDIg7rru0Py9sb2saWzhY= 2023-11-02T18:30:28.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wOT2zqzDIg7rru0Py9sb2saWzhY=', 'Content-Length': '0'} 2023-11-02T18:30:28.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000293d8a1269720e72-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.499 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.500 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.500 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.500 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.501 DEBUG:boto:Method: GET 2023-11-02T18:30:28.501 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.502 DEBUG:boto:Data: 2023-11-02T18:30:28.502 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.503 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.503 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.504 DEBUG:boto:Params: {} 2023-11-02T18:30:28.504 DEBUG:boto:Token: None 2023-11-02T18:30:28.505 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.505 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wOT2zqzDIg7rru0Py9sb2saWzhY= 2023-11-02T18:30:28.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wOT2zqzDIg7rru0Py9sb2saWzhY=', 'Content-Length': '0'} 2023-11-02T18:30:28.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad058009d42b2ea-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.554 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.555 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.555 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.556 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.556 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.557 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.557 DEBUG:boto:Data: 2023-11-02T18:30:28.558 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.558 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.558 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.559 DEBUG:boto:Params: {} 2023-11-02T18:30:28.559 DEBUG:boto:Token: None 2023-11-02T18:30:28.560 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.560 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m0yvSek/hLh8+vA0dDzfr5uN2JM= 2023-11-02T18:30:28.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m0yvSek/hLh8+vA0dDzfr5uN2JM=', 'Content-Length': '0'} 2023-11-02T18:30:28.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001201ff1b3a1f2492-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.565 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.565 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.565 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.566 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.566 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.567 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.567 DEBUG:boto:Data: 2023-11-02T18:30:28.567 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.568 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.568 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.569 DEBUG:boto:Params: {} 2023-11-02T18:30:28.569 DEBUG:boto:Token: None 2023-11-02T18:30:28.570 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.570 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m0yvSek/hLh8+vA0dDzfr5uN2JM= 2023-11-02T18:30:28.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m0yvSek/hLh8+vA0dDzfr5uN2JM=', 'Content-Length': '0'} 2023-11-02T18:30:28.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f4c71810a3030d1-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-11-02T18:30:28.575 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.575 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.575 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.576 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.576 DEBUG:boto:Method: GET 2023-11-02T18:30:28.577 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.577 DEBUG:boto:Data: 2023-11-02T18:30:28.578 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.578 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.578 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.579 DEBUG:boto:Params: {} 2023-11-02T18:30:28.579 DEBUG:boto:Token: None 2023-11-02T18:30:28.580 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.580 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wOT2zqzDIg7rru0Py9sb2saWzhY= 2023-11-02T18:30:28.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wOT2zqzDIg7rru0Py9sb2saWzhY=', 'Content-Length': '0'} 2023-11-02T18:30:28.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002801a00c65609db-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.585 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.586 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.586 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.587 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.587 DEBUG:boto:Method: GET 2023-11-02T18:30:28.587 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.588 DEBUG:boto:Data: 2023-11-02T18:30:28.588 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.589 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.589 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.590 DEBUG:boto:Params: {} 2023-11-02T18:30:28.590 DEBUG:boto:Token: None 2023-11-02T18:30:28.591 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000013?versionId=null 2023-11-02T18:30:28.591 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wOT2zqzDIg7rru0Py9sb2saWzhY= 2023-11-02T18:30:28.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wOT2zqzDIg7rru0Py9sb2saWzhY=', 'Content-Length': '0'} 2023-11-02T18:30:28.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a11980f59e5f99e6-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.595 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.595 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.595 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:28.596 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:28.596 DEBUG:boto:Method: GET 2023-11-02T18:30:28.597 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:28.597 DEBUG:boto:Data: 2023-11-02T18:30:28.598 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.598 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.598 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.599 DEBUG:boto:Params: {} 2023-11-02T18:30:28.599 DEBUG:boto:Token: None 2023-11-02T18:30:28.600 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.600 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de99030124c817c0-006543eac4-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.646 DEBUG:boto:b'ibtljn-58obj000000000000000000131000falseobj000000000000000000132023-11-02T18:30:20.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.647 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.648 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.648 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:28.649 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:28.649 DEBUG:boto:Method: GET 2023-11-02T18:30:28.650 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:28.650 DEBUG:boto:Data: 2023-11-02T18:30:28.651 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.651 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.652 DEBUG:boto:Params: {} 2023-11-02T18:30:28.652 DEBUG:boto:Token: None 2023-11-02T18:30:28.653 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baa177d81302f90f-006543eac4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.664 DEBUG:boto:b'ibtljn-58obj000000000000000000131000falseobj000000000000000000132023-11-02T18:30:20.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.665 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.665 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.665 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.666 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.666 DEBUG:boto:Data: 2023-11-02T18:30:28.667 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.667 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.668 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.668 DEBUG:boto:Params: {} 2023-11-02T18:30:28.669 DEBUG:boto:Token: None 2023-11-02T18:30:28.669 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.670 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pkhm8d8SINmGrAOGDTkuvkJBprM= 2023-11-02T18:30:28.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pkhm8d8SINmGrAOGDTkuvkJBprM=', 'Content-Length': '0'} 2023-11-02T18:30:28.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019bd50c6510f48ee-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.673 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.673 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.674 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.674 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.675 DEBUG:boto:Data: 2023-11-02T18:30:28.675 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.676 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.676 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.676 DEBUG:boto:Params: {} 2023-11-02T18:30:28.677 DEBUG:boto:Token: None 2023-11-02T18:30:28.677 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.678 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pkhm8d8SINmGrAOGDTkuvkJBprM= 2023-11-02T18:30:28.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pkhm8d8SINmGrAOGDTkuvkJBprM=', 'Content-Length': '0'} 2023-11-02T18:30:28.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cea138e53abb05a1-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-11-02T18:30:28.682 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.683 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.683 DEBUG:boto:Method: GET 2023-11-02T18:30:28.683 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.684 DEBUG:boto:Data: 2023-11-02T18:30:28.684 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.685 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.685 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.686 DEBUG:boto:Params: {} 2023-11-02T18:30:28.686 DEBUG:boto:Token: None 2023-11-02T18:30:28.687 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.687 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RgKK+NZ2rB1nULeHvqyWRpGtADA= 2023-11-02T18:30:28.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RgKK+NZ2rB1nULeHvqyWRpGtADA=', 'Content-Length': '0'} 2023-11-02T18:30:28.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b487cc9facfce06-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.691 DEBUG:boto:path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.691 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.692 DEBUG:boto:Method: GET 2023-11-02T18:30:28.692 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.693 DEBUG:boto:Data: 2023-11-02T18:30:28.693 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.694 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.694 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.695 DEBUG:boto:Params: {} 2023-11-02T18:30:28.695 DEBUG:boto:Token: None 2023-11-02T18:30:28.696 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000013 2023-11-02T18:30:28.696 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RgKK+NZ2rB1nULeHvqyWRpGtADA= 2023-11-02T18:30:28.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RgKK+NZ2rB1nULeHvqyWRpGtADA=', 'Content-Length': '0'} 2023-11-02T18:30:28.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025b9b5ff6fd10459-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.700 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.700 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.701 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:28.701 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:28.702 DEBUG:boto:Method: GET 2023-11-02T18:30:28.702 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:28.703 DEBUG:boto:Data: 2023-11-02T18:30:28.703 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.704 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.704 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.705 DEBUG:boto:Params: {} 2023-11-02T18:30:28.705 DEBUG:boto:Token: None 2023-11-02T18:30:28.705 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.706 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc3412430c685e9a-006543eac4-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.712 DEBUG:boto:b'ibtljn-58obj000000000000000000131000falseobj000000000000000000132023-11-02T18:30:20.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.712 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.713 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.713 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:28.714 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:28.714 DEBUG:boto:Method: GET 2023-11-02T18:30:28.715 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000013 2023-11-02T18:30:28.715 DEBUG:boto:Data: 2023-11-02T18:30:28.716 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.716 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.716 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.717 DEBUG:boto:Params: {} 2023-11-02T18:30:28.717 DEBUG:boto:Token: None 2023-11-02T18:30:28.718 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.718 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c57bb913657b938b-006543eac4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.770 DEBUG:boto:b'ibtljn-58obj000000000000000000131000falseobj000000000000000000132023-11-02T18:30:20.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-11-02T18:30:28.772 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.772 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.773 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.773 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.774 DEBUG:boto:Method: GET 2023-11-02T18:30:28.774 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.774 DEBUG:boto:Data: 2023-11-02T18:30:28.775 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.775 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.776 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.776 DEBUG:boto:Params: {} 2023-11-02T18:30:28.777 DEBUG:boto:Token: None 2023-11-02T18:30:28.777 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.778 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A4bZuIIJJvQZKdH6JDTC5Q1MAb8= 2023-11-02T18:30:28.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A4bZuIIJJvQZKdH6JDTC5Q1MAb8=', 'Content-Length': '0'} 2023-11-02T18:30:28.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddb4c9f3999db06f-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.782 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.783 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.783 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.784 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.784 DEBUG:boto:Method: GET 2023-11-02T18:30:28.785 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.785 DEBUG:boto:Data: 2023-11-02T18:30:28.786 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.786 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.786 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.787 DEBUG:boto:Params: {} 2023-11-02T18:30:28.787 DEBUG:boto:Token: None 2023-11-02T18:30:28.788 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.788 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A4bZuIIJJvQZKdH6JDTC5Q1MAb8= 2023-11-02T18:30:28.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A4bZuIIJJvQZKdH6JDTC5Q1MAb8=', 'Content-Length': '0'} 2023-11-02T18:30:28.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ef0d0236c1f189e-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.793 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.793 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.793 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.794 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.794 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.795 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.795 DEBUG:boto:Data: 2023-11-02T18:30:28.796 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.796 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.796 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.797 DEBUG:boto:Params: {} 2023-11-02T18:30:28.797 DEBUG:boto:Token: None 2023-11-02T18:30:28.798 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.798 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Hso1ERb86riOk9ozj8poXE9Hf/4= 2023-11-02T18:30:28.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Hso1ERb86riOk9ozj8poXE9Hf/4=', 'Content-Length': '0'} 2023-11-02T18:30:28.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000156343d8fa69ff50-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.802 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.803 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.803 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.804 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.804 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.805 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.805 DEBUG:boto:Data: 2023-11-02T18:30:28.805 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.806 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.806 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.807 DEBUG:boto:Params: {} 2023-11-02T18:30:28.807 DEBUG:boto:Token: None 2023-11-02T18:30:28.808 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.808 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Hso1ERb86riOk9ozj8poXE9Hf/4= 2023-11-02T18:30:28.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Hso1ERb86riOk9ozj8poXE9Hf/4=', 'Content-Length': '0'} 2023-11-02T18:30:28.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077e7b2fe3993bc4d-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-11-02T18:30:28.813 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.813 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.813 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.814 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.814 DEBUG:boto:Method: GET 2023-11-02T18:30:28.815 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.815 DEBUG:boto:Data: 2023-11-02T18:30:28.816 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.816 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.817 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.817 DEBUG:boto:Params: {} 2023-11-02T18:30:28.817 DEBUG:boto:Token: None 2023-11-02T18:30:28.818 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.818 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A4bZuIIJJvQZKdH6JDTC5Q1MAb8= 2023-11-02T18:30:28.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A4bZuIIJJvQZKdH6JDTC5Q1MAb8=', 'Content-Length': '0'} 2023-11-02T18:30:28.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8bbff781a150b5d-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.823 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.823 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.824 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.824 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.824 DEBUG:boto:Method: GET 2023-11-02T18:30:28.825 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.825 DEBUG:boto:Data: 2023-11-02T18:30:28.826 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.826 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.827 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.827 DEBUG:boto:Params: {} 2023-11-02T18:30:28.828 DEBUG:boto:Token: None 2023-11-02T18:30:28.828 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000014?versionId=null 2023-11-02T18:30:28.829 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A4bZuIIJJvQZKdH6JDTC5Q1MAb8= 2023-11-02T18:30:28.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A4bZuIIJJvQZKdH6JDTC5Q1MAb8=', 'Content-Length': '0'} 2023-11-02T18:30:28.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f19324162006cfb-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.874 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.875 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.875 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000014 2023-11-02T18:30:28.876 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000014 2023-11-02T18:30:28.876 DEBUG:boto:Method: GET 2023-11-02T18:30:28.877 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000014 2023-11-02T18:30:28.877 DEBUG:boto:Data: 2023-11-02T18:30:28.878 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.878 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.879 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.879 DEBUG:boto:Params: {} 2023-11-02T18:30:28.879 DEBUG:boto:Token: None 2023-11-02T18:30:28.880 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.880 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024d0c42a78bb1cd8-006543eac4-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.886 DEBUG:boto:b'ibtljn-58obj000000000000000000141000falseobj000000000000000000142023-11-02T18:30:19.907Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.887 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.888 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.888 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000014 2023-11-02T18:30:28.888 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000014 2023-11-02T18:30:28.889 DEBUG:boto:Method: GET 2023-11-02T18:30:28.889 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000014 2023-11-02T18:30:28.890 DEBUG:boto:Data: 2023-11-02T18:30:28.890 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.891 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.891 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.891 DEBUG:boto:Params: {} 2023-11-02T18:30:28.892 DEBUG:boto:Token: None 2023-11-02T18:30:28.892 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.893 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047b1360be7e3139a-006543eac4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.905 DEBUG:boto:b'ibtljn-58obj000000000000000000141000falseobj000000000000000000142023-11-02T18:30:19.907Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.906 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.906 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.907 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.907 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.907 DEBUG:boto:Data: 2023-11-02T18:30:28.908 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.908 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.909 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.909 DEBUG:boto:Params: {} 2023-11-02T18:30:28.910 DEBUG:boto:Token: None 2023-11-02T18:30:28.910 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.911 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Kc1eueabk18EK6FCJ86h09tSRvU= 2023-11-02T18:30:28.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Kc1eueabk18EK6FCJ86h09tSRvU=', 'Content-Length': '0'} 2023-11-02T18:30:28.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1b77e3224a1ca1d-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.914 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.914 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.915 DEBUG:boto:Method: HEAD 2023-11-02T18:30:28.915 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.915 DEBUG:boto:Data: 2023-11-02T18:30:28.916 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.916 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.917 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.917 DEBUG:boto:Params: {} 2023-11-02T18:30:28.918 DEBUG:boto:Token: None 2023-11-02T18:30:28.918 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.919 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Kc1eueabk18EK6FCJ86h09tSRvU= 2023-11-02T18:30:28.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Kc1eueabk18EK6FCJ86h09tSRvU=', 'Content-Length': '0'} 2023-11-02T18:30:28.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c301634a15aecd-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-11-02T18:30:28.922 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.923 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.923 DEBUG:boto:Method: GET 2023-11-02T18:30:28.924 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.924 DEBUG:boto:Data: 2023-11-02T18:30:28.925 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.925 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.925 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.926 DEBUG:boto:Params: {} 2023-11-02T18:30:28.926 DEBUG:boto:Token: None 2023-11-02T18:30:28.927 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.927 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z+tbuzKmntKlAS8r3wCORsm+U8k= 2023-11-02T18:30:28.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z+tbuzKmntKlAS8r3wCORsm+U8k=', 'Content-Length': '0'} 2023-11-02T18:30:28.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007482aac90b2bf8da-006543eac4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.974 DEBUG:boto:path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.975 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.975 DEBUG:boto:Method: GET 2023-11-02T18:30:28.976 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.976 DEBUG:boto:Data: 2023-11-02T18:30:28.977 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.977 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:28.978 DEBUG:boto:Port: 8000 2023-11-02T18:30:28.978 DEBUG:boto:Params: {} 2023-11-02T18:30:28.979 DEBUG:boto:Token: None 2023-11-02T18:30:28.979 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/obj00000000000000000014 2023-11-02T18:30:28.980 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z+tbuzKmntKlAS8r3wCORsm+U8k= 2023-11-02T18:30:28.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z+tbuzKmntKlAS8r3wCORsm+U8k=', 'Content-Length': '0'} 2023-11-02T18:30:28.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2136ba0c9b2f154-006543eac4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.984 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.985 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.985 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000014 2023-11-02T18:30:28.985 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000014 2023-11-02T18:30:28.986 DEBUG:boto:Method: GET 2023-11-02T18:30:28.986 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000014 2023-11-02T18:30:28.987 DEBUG:boto:Data: 2023-11-02T18:30:28.987 DEBUG:boto:Headers: {} 2023-11-02T18:30:28.988 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:28.988 DEBUG:boto:Port: 8001 2023-11-02T18:30:28.988 DEBUG:boto:Params: {} 2023-11-02T18:30:28.989 DEBUG:boto:Token: None 2023-11-02T18:30:28.989 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:28 GMT /ibtljn-58/ 2023-11-02T18:30:28.990 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk= 2023-11-02T18:30:28.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YNZgt/tDEy3BmqrFA9v7I9BmCrk=', 'Content-Length': '0'} 2023-11-02T18:30:28.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041a497fe335c3359-006543eac4-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:28.996 DEBUG:boto:b'ibtljn-58obj000000000000000000141000falseobj000000000000000000142023-11-02T18:30:19.907Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:28.997 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:28.997 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:28.998 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000014 2023-11-02T18:30:28.998 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000014 2023-11-02T18:30:28.999 DEBUG:boto:Method: GET 2023-11-02T18:30:28.999 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000014 2023-11-02T18:30:29.000 DEBUG:boto:Data: 2023-11-02T18:30:29.000 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.000 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.001 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.001 DEBUG:boto:Params: {} 2023-11-02T18:30:29.002 DEBUG:boto:Token: None 2023-11-02T18:30:29.002 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033666561c17b2c59-006543eac4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.054 DEBUG:boto:b'ibtljn-58obj000000000000000000141000falseobj000000000000000000142023-11-02T18:30:19.907Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-11-02T18:30:29.056 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.056 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.057 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.057 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.058 DEBUG:boto:Method: GET 2023-11-02T18:30:29.058 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.058 DEBUG:boto:Data: 2023-11-02T18:30:29.059 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.059 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.060 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.060 DEBUG:boto:Params: {} 2023-11-02T18:30:29.061 DEBUG:boto:Token: None 2023-11-02T18:30:29.061 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.062 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:36WKG4+Oz1tZ8m28IuKVHIIlcIA= 2023-11-02T18:30:29.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:36WKG4+Oz1tZ8m28IuKVHIIlcIA=', 'Content-Length': '0'} 2023-11-02T18:30:29.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000928f4251ac5ccbd-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.066 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.067 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.067 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.068 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.068 DEBUG:boto:Method: GET 2023-11-02T18:30:29.068 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.069 DEBUG:boto:Data: 2023-11-02T18:30:29.069 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.070 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.071 DEBUG:boto:Params: {} 2023-11-02T18:30:29.071 DEBUG:boto:Token: None 2023-11-02T18:30:29.072 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.074 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:36WKG4+Oz1tZ8m28IuKVHIIlcIA= 2023-11-02T18:30:29.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:36WKG4+Oz1tZ8m28IuKVHIIlcIA=', 'Content-Length': '0'} 2023-11-02T18:30:29.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e7bcd7789efe8bc-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.078 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.078 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.079 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.079 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.079 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.080 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.080 DEBUG:boto:Data: 2023-11-02T18:30:29.081 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.081 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.082 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.082 DEBUG:boto:Params: {} 2023-11-02T18:30:29.082 DEBUG:boto:Token: None 2023-11-02T18:30:29.083 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.083 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:apTHMJXTWSmmh0TcK/zhqevJMm8= 2023-11-02T18:30:29.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:apTHMJXTWSmmh0TcK/zhqevJMm8=', 'Content-Length': '0'} 2023-11-02T18:30:29.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6738aa3fd5b5b74-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.088 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.088 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.089 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.089 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.090 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.090 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.091 DEBUG:boto:Data: 2023-11-02T18:30:29.091 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.091 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.092 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.092 DEBUG:boto:Params: {} 2023-11-02T18:30:29.093 DEBUG:boto:Token: None 2023-11-02T18:30:29.093 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:apTHMJXTWSmmh0TcK/zhqevJMm8= 2023-11-02T18:30:29.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:apTHMJXTWSmmh0TcK/zhqevJMm8=', 'Content-Length': '0'} 2023-11-02T18:30:29.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086a5f1c7d6b34fd2-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-11-02T18:30:29.098 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.099 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.099 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.100 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.100 DEBUG:boto:Method: GET 2023-11-02T18:30:29.100 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.101 DEBUG:boto:Data: 2023-11-02T18:30:29.101 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.102 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.102 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.103 DEBUG:boto:Params: {} 2023-11-02T18:30:29.103 DEBUG:boto:Token: None 2023-11-02T18:30:29.104 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:36WKG4+Oz1tZ8m28IuKVHIIlcIA= 2023-11-02T18:30:29.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:36WKG4+Oz1tZ8m28IuKVHIIlcIA=', 'Content-Length': '0'} 2023-11-02T18:30:29.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd1bb26bf6523a18-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.108 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.109 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.109 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.110 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.110 DEBUG:boto:Method: GET 2023-11-02T18:30:29.112 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.112 DEBUG:boto:Data: 2023-11-02T18:30:29.112 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.113 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.113 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.114 DEBUG:boto:Params: {} 2023-11-02T18:30:29.114 DEBUG:boto:Token: None 2023-11-02T18:30:29.115 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000015?versionId=null 2023-11-02T18:30:29.115 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:36WKG4+Oz1tZ8m28IuKVHIIlcIA= 2023-11-02T18:30:29.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:36WKG4+Oz1tZ8m28IuKVHIIlcIA=', 'Content-Length': '0'} 2023-11-02T18:30:29.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e05d9634bf07a5c-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.120 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.121 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.121 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000015 2023-11-02T18:30:29.122 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000015 2023-11-02T18:30:29.122 DEBUG:boto:Method: GET 2023-11-02T18:30:29.123 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000015 2023-11-02T18:30:29.123 DEBUG:boto:Data: 2023-11-02T18:30:29.123 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.124 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.124 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.125 DEBUG:boto:Params: {} 2023-11-02T18:30:29.125 DEBUG:boto:Token: None 2023-11-02T18:30:29.126 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.126 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb8206a6dd00c282-006543eac5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.174 DEBUG:boto:b'ibtljn-58obj000000000000000000151000falseobj000000000000000000152023-11-02T18:30:19.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.175 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.176 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.176 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000015 2023-11-02T18:30:29.177 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000015 2023-11-02T18:30:29.177 DEBUG:boto:Method: GET 2023-11-02T18:30:29.178 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000015 2023-11-02T18:30:29.178 DEBUG:boto:Data: 2023-11-02T18:30:29.179 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.179 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.180 DEBUG:boto:Params: {} 2023-11-02T18:30:29.180 DEBUG:boto:Token: None 2023-11-02T18:30:29.181 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.181 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edf5d5649545e2c5-006543eac5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.193 DEBUG:boto:b'ibtljn-58obj000000000000000000151000falseobj000000000000000000152023-11-02T18:30:19.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.194 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.195 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.195 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.195 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.196 DEBUG:boto:Data: 2023-11-02T18:30:29.196 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.197 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.197 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.198 DEBUG:boto:Params: {} 2023-11-02T18:30:29.198 DEBUG:boto:Token: None 2023-11-02T18:30:29.199 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.199 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+UDCoSdSLryt+F7TQaFzxqUdJs0= 2023-11-02T18:30:29.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+UDCoSdSLryt+F7TQaFzxqUdJs0=', 'Content-Length': '0'} 2023-11-02T18:30:29.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bd9a6694d1329bf-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.203 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.203 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.204 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.204 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.205 DEBUG:boto:Data: 2023-11-02T18:30:29.205 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.205 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.206 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.206 DEBUG:boto:Params: {} 2023-11-02T18:30:29.207 DEBUG:boto:Token: None 2023-11-02T18:30:29.207 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+UDCoSdSLryt+F7TQaFzxqUdJs0= 2023-11-02T18:30:29.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+UDCoSdSLryt+F7TQaFzxqUdJs0=', 'Content-Length': '0'} 2023-11-02T18:30:29.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba061628e31c2eda-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-11-02T18:30:29.212 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.212 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.213 DEBUG:boto:Method: GET 2023-11-02T18:30:29.213 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.214 DEBUG:boto:Data: 2023-11-02T18:30:29.214 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.215 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.215 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.216 DEBUG:boto:Params: {} 2023-11-02T18:30:29.216 DEBUG:boto:Token: None 2023-11-02T18:30:29.217 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q5MYnous/6sUOLCA/5p0ZYeYf/o= 2023-11-02T18:30:29.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q5MYnous/6sUOLCA/5p0ZYeYf/o=', 'Content-Length': '0'} 2023-11-02T18:30:29.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d870079a6449b0a8-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.221 DEBUG:boto:path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.221 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.222 DEBUG:boto:Method: GET 2023-11-02T18:30:29.222 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.222 DEBUG:boto:Data: 2023-11-02T18:30:29.223 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.224 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.224 DEBUG:boto:Params: {} 2023-11-02T18:30:29.225 DEBUG:boto:Token: None 2023-11-02T18:30:29.225 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000015 2023-11-02T18:30:29.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q5MYnous/6sUOLCA/5p0ZYeYf/o= 2023-11-02T18:30:29.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q5MYnous/6sUOLCA/5p0ZYeYf/o=', 'Content-Length': '0'} 2023-11-02T18:30:29.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ccdb22cd5a349ac-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.270 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.271 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.272 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000015 2023-11-02T18:30:29.272 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000015 2023-11-02T18:30:29.272 DEBUG:boto:Method: GET 2023-11-02T18:30:29.273 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000015 2023-11-02T18:30:29.273 DEBUG:boto:Data: 2023-11-02T18:30:29.274 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.274 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.275 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.275 DEBUG:boto:Params: {} 2023-11-02T18:30:29.276 DEBUG:boto:Token: None 2023-11-02T18:30:29.278 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.279 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000395ce7819f532fb7-006543eac5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.284 DEBUG:boto:b'ibtljn-58obj000000000000000000151000falseobj000000000000000000152023-11-02T18:30:19.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.285 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.286 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.286 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000015 2023-11-02T18:30:29.287 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000015 2023-11-02T18:30:29.287 DEBUG:boto:Method: GET 2023-11-02T18:30:29.287 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000015 2023-11-02T18:30:29.288 DEBUG:boto:Data: 2023-11-02T18:30:29.288 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.289 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.289 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.290 DEBUG:boto:Params: {} 2023-11-02T18:30:29.290 DEBUG:boto:Token: None 2023-11-02T18:30:29.291 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.291 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aafb96b62af61a9-006543eac5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.302 DEBUG:boto:b'ibtljn-58obj000000000000000000151000falseobj000000000000000000152023-11-02T18:30:19.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-11-02T18:30:29.304 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.304 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.305 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.305 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.305 DEBUG:boto:Method: GET 2023-11-02T18:30:29.306 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.306 DEBUG:boto:Data: 2023-11-02T18:30:29.307 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.307 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.308 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.308 DEBUG:boto:Params: {} 2023-11-02T18:30:29.309 DEBUG:boto:Token: None 2023-11-02T18:30:29.309 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.310 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LzPLacfq6q2GRtDaNKfsu9WYkmA= 2023-11-02T18:30:29.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LzPLacfq6q2GRtDaNKfsu9WYkmA=', 'Content-Length': '0'} 2023-11-02T18:30:29.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006da31a8ca7bd4ad0-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.313 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.314 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.314 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.315 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.315 DEBUG:boto:Method: GET 2023-11-02T18:30:29.316 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.316 DEBUG:boto:Data: 2023-11-02T18:30:29.316 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.317 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.318 DEBUG:boto:Params: {} 2023-11-02T18:30:29.318 DEBUG:boto:Token: None 2023-11-02T18:30:29.319 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LzPLacfq6q2GRtDaNKfsu9WYkmA= 2023-11-02T18:30:29.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LzPLacfq6q2GRtDaNKfsu9WYkmA=', 'Content-Length': '0'} 2023-11-02T18:30:29.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eccf435a83648cb3-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.323 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.324 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.324 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.324 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.325 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.325 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.326 DEBUG:boto:Data: 2023-11-02T18:30:29.326 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.327 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.327 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.327 DEBUG:boto:Params: {} 2023-11-02T18:30:29.328 DEBUG:boto:Token: None 2023-11-02T18:30:29.328 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.329 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ask+mkGAIfhjSBKGVLuj+Erj+GY= 2023-11-02T18:30:29.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ask+mkGAIfhjSBKGVLuj+Erj+GY=', 'Content-Length': '0'} 2023-11-02T18:30:29.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074bac2a74f99bae5-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.333 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.333 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.334 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.334 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.335 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.335 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.335 DEBUG:boto:Data: 2023-11-02T18:30:29.336 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.336 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.337 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.337 DEBUG:boto:Params: {} 2023-11-02T18:30:29.338 DEBUG:boto:Token: None 2023-11-02T18:30:29.338 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.339 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ask+mkGAIfhjSBKGVLuj+Erj+GY= 2023-11-02T18:30:29.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ask+mkGAIfhjSBKGVLuj+Erj+GY=', 'Content-Length': '0'} 2023-11-02T18:30:29.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047c9afc67540b31c-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-11-02T18:30:29.343 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.344 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.344 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.345 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.345 DEBUG:boto:Method: GET 2023-11-02T18:30:29.346 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.346 DEBUG:boto:Data: 2023-11-02T18:30:29.346 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.347 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.347 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.348 DEBUG:boto:Params: {} 2023-11-02T18:30:29.348 DEBUG:boto:Token: None 2023-11-02T18:30:29.349 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.349 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LzPLacfq6q2GRtDaNKfsu9WYkmA= 2023-11-02T18:30:29.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LzPLacfq6q2GRtDaNKfsu9WYkmA=', 'Content-Length': '0'} 2023-11-02T18:30:29.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000249e0903f44c5003-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.354 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.355 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.355 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.356 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.356 DEBUG:boto:Method: GET 2023-11-02T18:30:29.357 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.357 DEBUG:boto:Data: 2023-11-02T18:30:29.358 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.359 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.359 DEBUG:boto:Params: {} 2023-11-02T18:30:29.360 DEBUG:boto:Token: None 2023-11-02T18:30:29.360 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000016?versionId=null 2023-11-02T18:30:29.361 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LzPLacfq6q2GRtDaNKfsu9WYkmA= 2023-11-02T18:30:29.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LzPLacfq6q2GRtDaNKfsu9WYkmA=', 'Content-Length': '0'} 2023-11-02T18:30:29.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000847cf67a3e4200b4-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.406 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.407 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.408 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000016 2023-11-02T18:30:29.408 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000016 2023-11-02T18:30:29.408 DEBUG:boto:Method: GET 2023-11-02T18:30:29.409 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000016 2023-11-02T18:30:29.409 DEBUG:boto:Data: 2023-11-02T18:30:29.410 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.410 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.411 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.411 DEBUG:boto:Params: {} 2023-11-02T18:30:29.412 DEBUG:boto:Token: None 2023-11-02T18:30:29.412 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.413 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aed173a636bc207d-006543eac5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.419 DEBUG:boto:b'ibtljn-58obj000000000000000000161000falseobj000000000000000000162023-11-02T18:30:20.476Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.420 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.420 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.421 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000016 2023-11-02T18:30:29.421 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000016 2023-11-02T18:30:29.422 DEBUG:boto:Method: GET 2023-11-02T18:30:29.422 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000016 2023-11-02T18:30:29.423 DEBUG:boto:Data: 2023-11-02T18:30:29.423 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.423 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.424 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.424 DEBUG:boto:Params: {} 2023-11-02T18:30:29.425 DEBUG:boto:Token: None 2023-11-02T18:30:29.425 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.426 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000516c845211158fd1-006543eac5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.438 DEBUG:boto:b'ibtljn-58obj000000000000000000161000falseobj000000000000000000162023-11-02T18:30:20.476Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.439 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.439 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.439 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.440 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.440 DEBUG:boto:Data: 2023-11-02T18:30:29.441 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.441 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.442 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.442 DEBUG:boto:Params: {} 2023-11-02T18:30:29.442 DEBUG:boto:Token: None 2023-11-02T18:30:29.443 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.443 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ieq+MQsfBnng0KdhVmgbq2On3kE= 2023-11-02T18:30:29.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ieq+MQsfBnng0KdhVmgbq2On3kE=', 'Content-Length': '0'} 2023-11-02T18:30:29.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044ae4cd28bb33964-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.447 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.447 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.448 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.448 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.449 DEBUG:boto:Data: 2023-11-02T18:30:29.449 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.449 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.450 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.450 DEBUG:boto:Params: {} 2023-11-02T18:30:29.451 DEBUG:boto:Token: None 2023-11-02T18:30:29.451 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.452 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ieq+MQsfBnng0KdhVmgbq2On3kE= 2023-11-02T18:30:29.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ieq+MQsfBnng0KdhVmgbq2On3kE=', 'Content-Length': '0'} 2023-11-02T18:30:29.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1a565b1fcdc3edc-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-11-02T18:30:29.456 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.456 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.456 DEBUG:boto:Method: GET 2023-11-02T18:30:29.457 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.457 DEBUG:boto:Data: 2023-11-02T18:30:29.458 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.458 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.459 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.459 DEBUG:boto:Params: {} 2023-11-02T18:30:29.460 DEBUG:boto:Token: None 2023-11-02T18:30:29.460 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.461 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hHcXbHLR5iT3TT4Q1bppCmwcSVw= 2023-11-02T18:30:29.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hHcXbHLR5iT3TT4Q1bppCmwcSVw=', 'Content-Length': '0'} 2023-11-02T18:30:29.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ffaab4a641f0243-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.506 DEBUG:boto:path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.507 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.508 DEBUG:boto:Method: GET 2023-11-02T18:30:29.508 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.509 DEBUG:boto:Data: 2023-11-02T18:30:29.509 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.510 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.510 DEBUG:boto:Params: {} 2023-11-02T18:30:29.511 DEBUG:boto:Token: None 2023-11-02T18:30:29.511 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000016 2023-11-02T18:30:29.512 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hHcXbHLR5iT3TT4Q1bppCmwcSVw= 2023-11-02T18:30:29.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hHcXbHLR5iT3TT4Q1bppCmwcSVw=', 'Content-Length': '0'} 2023-11-02T18:30:29.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d289e8f96e3a050-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.516 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.517 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.517 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000016 2023-11-02T18:30:29.518 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000016 2023-11-02T18:30:29.518 DEBUG:boto:Method: GET 2023-11-02T18:30:29.518 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000016 2023-11-02T18:30:29.519 DEBUG:boto:Data: 2023-11-02T18:30:29.519 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.520 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.520 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.521 DEBUG:boto:Params: {} 2023-11-02T18:30:29.521 DEBUG:boto:Token: None 2023-11-02T18:30:29.522 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.522 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000024b9c675ae42ec6-006543eac5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.529 DEBUG:boto:b'ibtljn-58obj000000000000000000161000falseobj000000000000000000162023-11-02T18:30:20.476Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.530 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.530 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.531 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000016 2023-11-02T18:30:29.531 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000016 2023-11-02T18:30:29.531 DEBUG:boto:Method: GET 2023-11-02T18:30:29.532 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000016 2023-11-02T18:30:29.532 DEBUG:boto:Data: 2023-11-02T18:30:29.533 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.533 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.534 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.534 DEBUG:boto:Params: {} 2023-11-02T18:30:29.535 DEBUG:boto:Token: None 2023-11-02T18:30:29.535 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.536 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000432b7628a8082cb3-006543eac5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.590 DEBUG:boto:b'ibtljn-58obj000000000000000000161000falseobj000000000000000000162023-11-02T18:30:20.476Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-11-02T18:30:29.592 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.592 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.593 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.593 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.593 DEBUG:boto:Method: GET 2023-11-02T18:30:29.594 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.594 DEBUG:boto:Data: 2023-11-02T18:30:29.595 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.595 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.596 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.596 DEBUG:boto:Params: {} 2023-11-02T18:30:29.596 DEBUG:boto:Token: None 2023-11-02T18:30:29.597 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.598 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NMvYzVeWjhDj2AxGsnX6pyzw0k0= 2023-11-02T18:30:29.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NMvYzVeWjhDj2AxGsnX6pyzw0k0=', 'Content-Length': '0'} 2023-11-02T18:30:29.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c61933de294f7a-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.602 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.602 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.602 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.603 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.603 DEBUG:boto:Method: GET 2023-11-02T18:30:29.604 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.604 DEBUG:boto:Data: 2023-11-02T18:30:29.605 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.605 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.605 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.606 DEBUG:boto:Params: {} 2023-11-02T18:30:29.606 DEBUG:boto:Token: None 2023-11-02T18:30:29.607 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.607 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NMvYzVeWjhDj2AxGsnX6pyzw0k0= 2023-11-02T18:30:29.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NMvYzVeWjhDj2AxGsnX6pyzw0k0=', 'Content-Length': '0'} 2023-11-02T18:30:29.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064c238f3034a7ed6-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.611 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.612 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.612 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.613 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.613 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.613 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.614 DEBUG:boto:Data: 2023-11-02T18:30:29.614 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.615 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.615 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.616 DEBUG:boto:Params: {} 2023-11-02T18:30:29.616 DEBUG:boto:Token: None 2023-11-02T18:30:29.616 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.617 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JNHrNlP8srPvEHPoVvJa6OX8rIE= 2023-11-02T18:30:29.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JNHrNlP8srPvEHPoVvJa6OX8rIE=', 'Content-Length': '0'} 2023-11-02T18:30:29.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b998ab6ec6018a2-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.621 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.621 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.621 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.622 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.622 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.623 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.623 DEBUG:boto:Data: 2023-11-02T18:30:29.624 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.624 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.624 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.625 DEBUG:boto:Params: {} 2023-11-02T18:30:29.625 DEBUG:boto:Token: None 2023-11-02T18:30:29.626 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JNHrNlP8srPvEHPoVvJa6OX8rIE= 2023-11-02T18:30:29.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JNHrNlP8srPvEHPoVvJa6OX8rIE=', 'Content-Length': '0'} 2023-11-02T18:30:29.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048ec5a37067d8a4b-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-11-02T18:30:29.630 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.630 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.631 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.631 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.632 DEBUG:boto:Method: GET 2023-11-02T18:30:29.632 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.633 DEBUG:boto:Data: 2023-11-02T18:30:29.633 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.634 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.634 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.634 DEBUG:boto:Params: {} 2023-11-02T18:30:29.635 DEBUG:boto:Token: None 2023-11-02T18:30:29.635 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.636 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NMvYzVeWjhDj2AxGsnX6pyzw0k0= 2023-11-02T18:30:29.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NMvYzVeWjhDj2AxGsnX6pyzw0k0=', 'Content-Length': '0'} 2023-11-02T18:30:29.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000481fdb60a6791bf8-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.640 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.640 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.641 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.641 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.642 DEBUG:boto:Method: GET 2023-11-02T18:30:29.642 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.643 DEBUG:boto:Data: 2023-11-02T18:30:29.643 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.643 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.644 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.644 DEBUG:boto:Params: {} 2023-11-02T18:30:29.645 DEBUG:boto:Token: None 2023-11-02T18:30:29.645 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000017?versionId=null 2023-11-02T18:30:29.646 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NMvYzVeWjhDj2AxGsnX6pyzw0k0= 2023-11-02T18:30:29.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NMvYzVeWjhDj2AxGsnX6pyzw0k0=', 'Content-Length': '0'} 2023-11-02T18:30:29.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e175438d2d7a6727-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.650 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.650 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.651 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000017 2023-11-02T18:30:29.651 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000017 2023-11-02T18:30:29.652 DEBUG:boto:Method: GET 2023-11-02T18:30:29.652 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000017 2023-11-02T18:30:29.652 DEBUG:boto:Data: 2023-11-02T18:30:29.653 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.653 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.654 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.654 DEBUG:boto:Params: {} 2023-11-02T18:30:29.655 DEBUG:boto:Token: None 2023-11-02T18:30:29.655 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005785c830c21e5189-006543eac5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.661 DEBUG:boto:b'ibtljn-58obj000000000000000000171000falseobj000000000000000000172023-11-02T18:30:20.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.662 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.663 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.663 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000017 2023-11-02T18:30:29.663 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000017 2023-11-02T18:30:29.664 DEBUG:boto:Method: GET 2023-11-02T18:30:29.664 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000017 2023-11-02T18:30:29.665 DEBUG:boto:Data: 2023-11-02T18:30:29.665 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.666 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.666 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.666 DEBUG:boto:Params: {} 2023-11-02T18:30:29.667 DEBUG:boto:Token: None 2023-11-02T18:30:29.667 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.668 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f7ff6e39656790e-006543eac5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.679 DEBUG:boto:b'ibtljn-58obj000000000000000000171000falseobj000000000000000000172023-11-02T18:30:20.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.680 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.680 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.681 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.681 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.682 DEBUG:boto:Data: 2023-11-02T18:30:29.682 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.683 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.683 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.683 DEBUG:boto:Params: {} 2023-11-02T18:30:29.686 DEBUG:boto:Token: None 2023-11-02T18:30:29.686 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.687 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CCFm8XyBZdn38hTV62tITe3uzSQ= 2023-11-02T18:30:29.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CCFm8XyBZdn38hTV62tITe3uzSQ=', 'Content-Length': '0'} 2023-11-02T18:30:29.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aac891720081b4f-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.690 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.690 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.691 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.691 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.692 DEBUG:boto:Data: 2023-11-02T18:30:29.692 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.693 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.693 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.693 DEBUG:boto:Params: {} 2023-11-02T18:30:29.694 DEBUG:boto:Token: None 2023-11-02T18:30:29.694 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.695 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CCFm8XyBZdn38hTV62tITe3uzSQ= 2023-11-02T18:30:29.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CCFm8XyBZdn38hTV62tITe3uzSQ=', 'Content-Length': '0'} 2023-11-02T18:30:29.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f43948b668c834e6-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-11-02T18:30:29.699 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.699 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.700 DEBUG:boto:Method: GET 2023-11-02T18:30:29.700 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.701 DEBUG:boto:Data: 2023-11-02T18:30:29.701 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.701 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.702 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.702 DEBUG:boto:Params: {} 2023-11-02T18:30:29.703 DEBUG:boto:Token: None 2023-11-02T18:30:29.703 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.704 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1dQLlXddlJatLrk9om22r04EpxQ= 2023-11-02T18:30:29.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1dQLlXddlJatLrk9om22r04EpxQ=', 'Content-Length': '0'} 2023-11-02T18:30:29.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032ee14748d4a3eb6-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.750 DEBUG:boto:path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.751 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.752 DEBUG:boto:Method: GET 2023-11-02T18:30:29.752 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.753 DEBUG:boto:Data: 2023-11-02T18:30:29.753 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.753 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.754 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.754 DEBUG:boto:Params: {} 2023-11-02T18:30:29.755 DEBUG:boto:Token: None 2023-11-02T18:30:29.755 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000017 2023-11-02T18:30:29.756 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1dQLlXddlJatLrk9om22r04EpxQ= 2023-11-02T18:30:29.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1dQLlXddlJatLrk9om22r04EpxQ=', 'Content-Length': '0'} 2023-11-02T18:30:29.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b42be2a089ce0f0-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.760 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.761 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.761 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000017 2023-11-02T18:30:29.762 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000017 2023-11-02T18:30:29.762 DEBUG:boto:Method: GET 2023-11-02T18:30:29.763 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000017 2023-11-02T18:30:29.763 DEBUG:boto:Data: 2023-11-02T18:30:29.764 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.765 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.765 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.766 DEBUG:boto:Params: {} 2023-11-02T18:30:29.766 DEBUG:boto:Token: None 2023-11-02T18:30:29.767 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000933c22d6d53879b7-006543eac5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.774 DEBUG:boto:b'ibtljn-58obj000000000000000000171000falseobj000000000000000000172023-11-02T18:30:20.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.774 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.775 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.776 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000017 2023-11-02T18:30:29.776 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000017 2023-11-02T18:30:29.777 DEBUG:boto:Method: GET 2023-11-02T18:30:29.777 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000017 2023-11-02T18:30:29.778 DEBUG:boto:Data: 2023-11-02T18:30:29.778 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.779 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.779 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.779 DEBUG:boto:Params: {} 2023-11-02T18:30:29.780 DEBUG:boto:Token: None 2023-11-02T18:30:29.780 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.781 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5277f9893b8b296-006543eac5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.792 DEBUG:boto:b'ibtljn-58obj000000000000000000171000falseobj000000000000000000172023-11-02T18:30:20.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-11-02T18:30:29.793 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.793 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.794 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.794 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.795 DEBUG:boto:Method: GET 2023-11-02T18:30:29.795 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.796 DEBUG:boto:Data: 2023-11-02T18:30:29.796 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.796 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.797 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.797 DEBUG:boto:Params: {} 2023-11-02T18:30:29.798 DEBUG:boto:Token: None 2023-11-02T18:30:29.798 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.799 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nl8rp1Cn1/MvStTEIMzxbN6vI1c= 2023-11-02T18:30:29.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nl8rp1Cn1/MvStTEIMzxbN6vI1c=', 'Content-Length': '0'} 2023-11-02T18:30:29.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5f80de788d549da-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.803 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.804 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.804 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.804 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.805 DEBUG:boto:Method: GET 2023-11-02T18:30:29.805 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.806 DEBUG:boto:Data: 2023-11-02T18:30:29.806 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.807 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.807 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.808 DEBUG:boto:Params: {} 2023-11-02T18:30:29.808 DEBUG:boto:Token: None 2023-11-02T18:30:29.809 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.809 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nl8rp1Cn1/MvStTEIMzxbN6vI1c= 2023-11-02T18:30:29.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nl8rp1Cn1/MvStTEIMzxbN6vI1c=', 'Content-Length': '0'} 2023-11-02T18:30:29.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab630954b869ace9-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.813 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.814 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.814 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.815 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.815 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.815 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.816 DEBUG:boto:Data: 2023-11-02T18:30:29.816 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.817 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.817 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.817 DEBUG:boto:Params: {} 2023-11-02T18:30:29.818 DEBUG:boto:Token: None 2023-11-02T18:30:29.818 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.819 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I6cucaK4sTdR+BSdmbnd17GdaRU= 2023-11-02T18:30:29.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I6cucaK4sTdR+BSdmbnd17GdaRU=', 'Content-Length': '0'} 2023-11-02T18:30:29.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c82fccc453a1782-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.823 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.823 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.823 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.824 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.824 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.825 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.825 DEBUG:boto:Data: 2023-11-02T18:30:29.826 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.826 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.826 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.827 DEBUG:boto:Params: {} 2023-11-02T18:30:29.827 DEBUG:boto:Token: None 2023-11-02T18:30:29.828 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.828 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I6cucaK4sTdR+BSdmbnd17GdaRU= 2023-11-02T18:30:29.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I6cucaK4sTdR+BSdmbnd17GdaRU=', 'Content-Length': '0'} 2023-11-02T18:30:29.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc31acbc71e9fb8-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-11-02T18:30:29.833 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.833 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.833 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.834 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.834 DEBUG:boto:Method: GET 2023-11-02T18:30:29.835 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.835 DEBUG:boto:Data: 2023-11-02T18:30:29.836 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.836 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.836 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.837 DEBUG:boto:Params: {} 2023-11-02T18:30:29.837 DEBUG:boto:Token: None 2023-11-02T18:30:29.838 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.838 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nl8rp1Cn1/MvStTEIMzxbN6vI1c= 2023-11-02T18:30:29.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nl8rp1Cn1/MvStTEIMzxbN6vI1c=', 'Content-Length': '0'} 2023-11-02T18:30:29.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc6d0c2664c098ce-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.882 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.883 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.883 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.884 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.884 DEBUG:boto:Method: GET 2023-11-02T18:30:29.885 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.885 DEBUG:boto:Data: 2023-11-02T18:30:29.886 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.886 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.887 DEBUG:boto:Params: {} 2023-11-02T18:30:29.889 DEBUG:boto:Token: None 2023-11-02T18:30:29.890 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000018?versionId=null 2023-11-02T18:30:29.890 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nl8rp1Cn1/MvStTEIMzxbN6vI1c= 2023-11-02T18:30:29.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nl8rp1Cn1/MvStTEIMzxbN6vI1c=', 'Content-Length': '0'} 2023-11-02T18:30:29.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000391b9cefa34aeb40-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.895 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.895 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.896 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000018 2023-11-02T18:30:29.896 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000018 2023-11-02T18:30:29.897 DEBUG:boto:Method: GET 2023-11-02T18:30:29.897 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000018 2023-11-02T18:30:29.898 DEBUG:boto:Data: 2023-11-02T18:30:29.898 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.898 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.899 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.900 DEBUG:boto:Params: {} 2023-11-02T18:30:29.900 DEBUG:boto:Token: None 2023-11-02T18:30:29.901 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.901 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045a53bedaf6f292f-006543eac5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.907 DEBUG:boto:b'ibtljn-58obj000000000000000000181000falseobj000000000000000000182023-11-02T18:30:19.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.908 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:29.908 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:29.909 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000018 2023-11-02T18:30:29.909 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000018 2023-11-02T18:30:29.909 DEBUG:boto:Method: GET 2023-11-02T18:30:29.910 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000018 2023-11-02T18:30:29.910 DEBUG:boto:Data: 2023-11-02T18:30:29.911 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.911 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.912 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.912 DEBUG:boto:Params: {} 2023-11-02T18:30:29.913 DEBUG:boto:Token: None 2023-11-02T18:30:29.913 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/ 2023-11-02T18:30:29.914 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g= 2023-11-02T18:30:29.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SAG9LtfGvltOxUHmo1k5hPyGT9g=', 'Content-Length': '0'} 2023-11-02T18:30:29.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd60dba68e6966d7-006543eac5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.970 DEBUG:boto:b'ibtljn-58obj000000000000000000181000falseobj000000000000000000182023-11-02T18:30:19.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:29.971 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.972 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.972 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.973 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.973 DEBUG:boto:Data: 2023-11-02T18:30:29.974 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.974 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.975 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.975 DEBUG:boto:Params: {} 2023-11-02T18:30:29.976 DEBUG:boto:Token: None 2023-11-02T18:30:29.976 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.977 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rkkZyyB68FlhM+nsealO5JN9Mb4= 2023-11-02T18:30:29.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rkkZyyB68FlhM+nsealO5JN9Mb4=', 'Content-Length': '0'} 2023-11-02T18:30:29.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000713589679ecb6ac8-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.981 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.982 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.982 DEBUG:boto:Method: HEAD 2023-11-02T18:30:29.983 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.983 DEBUG:boto:Data: 2023-11-02T18:30:29.984 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.984 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:29.984 DEBUG:boto:Port: 8000 2023-11-02T18:30:29.985 DEBUG:boto:Params: {} 2023-11-02T18:30:29.985 DEBUG:boto:Token: None 2023-11-02T18:30:29.986 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.986 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rkkZyyB68FlhM+nsealO5JN9Mb4= 2023-11-02T18:30:29.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rkkZyyB68FlhM+nsealO5JN9Mb4=', 'Content-Length': '0'} 2023-11-02T18:30:29.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002783fa9bf564f0ec-006543eac5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:29.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-11-02T18:30:29.991 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.991 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.992 DEBUG:boto:Method: GET 2023-11-02T18:30:29.992 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.993 DEBUG:boto:Data: 2023-11-02T18:30:29.993 DEBUG:boto:Headers: {} 2023-11-02T18:30:29.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:29.994 DEBUG:boto:Port: 8001 2023-11-02T18:30:29.995 DEBUG:boto:Params: {} 2023-11-02T18:30:29.995 DEBUG:boto:Token: None 2023-11-02T18:30:29.996 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:29 GMT /ibtljn-58/obj00000000000000000018 2023-11-02T18:30:29.996 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JGnC5ZfHLBTKj4ibkdP2oKoZnkc= 2023-11-02T18:30:29.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JGnC5ZfHLBTKj4ibkdP2oKoZnkc=', 'Content-Length': '0'} 2023-11-02T18:30:30.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009947661657e36ec5-006543eac5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.001 DEBUG:boto:path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:30.001 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000018 2023-11-02T18:30:30.002 DEBUG:boto:Method: GET 2023-11-02T18:30:30.002 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000018 2023-11-02T18:30:30.003 DEBUG:boto:Data: 2023-11-02T18:30:30.003 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.004 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.004 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.005 DEBUG:boto:Params: {} 2023-11-02T18:30:30.006 DEBUG:boto:Token: None 2023-11-02T18:30:30.006 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000018 2023-11-02T18:30:30.007 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rc7xFVL5LuI3mMXuD3yjOMHo9NY= 2023-11-02T18:30:30.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rc7xFVL5LuI3mMXuD3yjOMHo9NY=', 'Content-Length': '0'} 2023-11-02T18:30:30.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d19b40bb0909f61-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.011 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.012 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.012 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000018 2023-11-02T18:30:30.013 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000018 2023-11-02T18:30:30.013 DEBUG:boto:Method: GET 2023-11-02T18:30:30.014 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000018 2023-11-02T18:30:30.014 DEBUG:boto:Data: 2023-11-02T18:30:30.014 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.015 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.015 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.016 DEBUG:boto:Params: {} 2023-11-02T18:30:30.016 DEBUG:boto:Token: None 2023-11-02T18:30:30.017 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.017 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001366d68fbb860b2d-006543eac6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.023 DEBUG:boto:b'ibtljn-58obj000000000000000000181000falseobj000000000000000000182023-11-02T18:30:19.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.024 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.025 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.025 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000018 2023-11-02T18:30:30.026 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000018 2023-11-02T18:30:30.026 DEBUG:boto:Method: GET 2023-11-02T18:30:30.027 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000018 2023-11-02T18:30:30.027 DEBUG:boto:Data: 2023-11-02T18:30:30.027 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.028 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.028 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.029 DEBUG:boto:Params: {} 2023-11-02T18:30:30.029 DEBUG:boto:Token: None 2023-11-02T18:30:30.030 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.030 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9e9ee7992aa7b38-006543eac6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.041 DEBUG:boto:b'ibtljn-58obj000000000000000000181000falseobj000000000000000000182023-11-02T18:30:19.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-11-02T18:30:30.042 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.042 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.043 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.043 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.044 DEBUG:boto:Method: GET 2023-11-02T18:30:30.044 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.045 DEBUG:boto:Data: 2023-11-02T18:30:30.045 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.046 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.047 DEBUG:boto:Params: {} 2023-11-02T18:30:30.047 DEBUG:boto:Token: None 2023-11-02T18:30:30.048 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.048 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HS+4pEot/FVuXqmmMMx7eJuk85w= 2023-11-02T18:30:30.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HS+4pEot/FVuXqmmMMx7eJuk85w=', 'Content-Length': '0'} 2023-11-02T18:30:30.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7f2cae5ea0628f3-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.099 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.100 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.100 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.101 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.101 DEBUG:boto:Method: GET 2023-11-02T18:30:30.102 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.102 DEBUG:boto:Data: 2023-11-02T18:30:30.102 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.103 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.104 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.105 DEBUG:boto:Params: {} 2023-11-02T18:30:30.105 DEBUG:boto:Token: None 2023-11-02T18:30:30.106 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.106 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HS+4pEot/FVuXqmmMMx7eJuk85w= 2023-11-02T18:30:30.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HS+4pEot/FVuXqmmMMx7eJuk85w=', 'Content-Length': '0'} 2023-11-02T18:30:30.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000730aa091313e5b28-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.111 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.111 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.112 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.112 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.113 DEBUG:boto:Method: HEAD 2023-11-02T18:30:30.113 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.114 DEBUG:boto:Data: 2023-11-02T18:30:30.114 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.115 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.115 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.116 DEBUG:boto:Params: {} 2023-11-02T18:30:30.116 DEBUG:boto:Token: None 2023-11-02T18:30:30.117 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.117 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MrG0cNJ0Vbc8H8z6tSGkOLnot7M= 2023-11-02T18:30:30.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MrG0cNJ0Vbc8H8z6tSGkOLnot7M=', 'Content-Length': '0'} 2023-11-02T18:30:30.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013fbd34923e9c43f-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.123 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.124 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.124 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.125 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.125 DEBUG:boto:Method: HEAD 2023-11-02T18:30:30.126 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.126 DEBUG:boto:Data: 2023-11-02T18:30:30.126 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.127 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.127 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.128 DEBUG:boto:Params: {} 2023-11-02T18:30:30.128 DEBUG:boto:Token: None 2023-11-02T18:30:30.129 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MrG0cNJ0Vbc8H8z6tSGkOLnot7M= 2023-11-02T18:30:30.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MrG0cNJ0Vbc8H8z6tSGkOLnot7M=', 'Content-Length': '0'} 2023-11-02T18:30:30.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac001a2a456dcbc1-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-11-02T18:30:30.133 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.134 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.134 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.135 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.135 DEBUG:boto:Method: GET 2023-11-02T18:30:30.136 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.136 DEBUG:boto:Data: 2023-11-02T18:30:30.137 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.137 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.138 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.138 DEBUG:boto:Params: {} 2023-11-02T18:30:30.139 DEBUG:boto:Token: None 2023-11-02T18:30:30.139 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HS+4pEot/FVuXqmmMMx7eJuk85w= 2023-11-02T18:30:30.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HS+4pEot/FVuXqmmMMx7eJuk85w=', 'Content-Length': '0'} 2023-11-02T18:30:30.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686fbc9a5dd105e1-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.144 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.145 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.145 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.146 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.146 DEBUG:boto:Method: GET 2023-11-02T18:30:30.147 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.147 DEBUG:boto:Data: 2023-11-02T18:30:30.148 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.148 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.149 DEBUG:boto:Params: {} 2023-11-02T18:30:30.149 DEBUG:boto:Token: None 2023-11-02T18:30:30.150 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000019?versionId=null 2023-11-02T18:30:30.150 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HS+4pEot/FVuXqmmMMx7eJuk85w= 2023-11-02T18:30:30.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HS+4pEot/FVuXqmmMMx7eJuk85w=', 'Content-Length': '0'} 2023-11-02T18:30:30.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c121c6361c58ae4-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.202 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.203 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.204 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000019 2023-11-02T18:30:30.204 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000019 2023-11-02T18:30:30.205 DEBUG:boto:Method: GET 2023-11-02T18:30:30.205 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000019 2023-11-02T18:30:30.205 DEBUG:boto:Data: 2023-11-02T18:30:30.206 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.206 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.207 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.207 DEBUG:boto:Params: {} 2023-11-02T18:30:30.208 DEBUG:boto:Token: None 2023-11-02T18:30:30.208 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.209 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9601453e0303b54-006543eac6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.215 DEBUG:boto:b'ibtljn-58obj000000000000000000191000falseobj000000000000000000192023-11-02T18:30:19.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.216 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.216 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.216 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000019 2023-11-02T18:30:30.217 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000019 2023-11-02T18:30:30.218 DEBUG:boto:Method: GET 2023-11-02T18:30:30.218 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000019 2023-11-02T18:30:30.219 DEBUG:boto:Data: 2023-11-02T18:30:30.219 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.220 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.220 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.221 DEBUG:boto:Params: {} 2023-11-02T18:30:30.221 DEBUG:boto:Token: None 2023-11-02T18:30:30.222 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.223 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002790875f650dabfc-006543eac6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.234 DEBUG:boto:b'ibtljn-58obj000000000000000000191000falseobj000000000000000000192023-11-02T18:30:19.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.235 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.235 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.236 DEBUG:boto:Method: HEAD 2023-11-02T18:30:30.236 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.236 DEBUG:boto:Data: 2023-11-02T18:30:30.237 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.237 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.238 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.238 DEBUG:boto:Params: {} 2023-11-02T18:30:30.239 DEBUG:boto:Token: None 2023-11-02T18:30:30.239 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.240 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NZQn+NnXXuqdzJwF0vuc5HU+RXM= 2023-11-02T18:30:30.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NZQn+NnXXuqdzJwF0vuc5HU+RXM=', 'Content-Length': '0'} 2023-11-02T18:30:30.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27dcb99578fdfae-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.243 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.244 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.244 DEBUG:boto:Method: HEAD 2023-11-02T18:30:30.245 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.245 DEBUG:boto:Data: 2023-11-02T18:30:30.246 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.247 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.247 DEBUG:boto:Params: {} 2023-11-02T18:30:30.248 DEBUG:boto:Token: None 2023-11-02T18:30:30.249 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.249 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NZQn+NnXXuqdzJwF0vuc5HU+RXM= 2023-11-02T18:30:30.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NZQn+NnXXuqdzJwF0vuc5HU+RXM=', 'Content-Length': '0'} 2023-11-02T18:30:30.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dc3890629a55627-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-11-02T18:30:30.253 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.254 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.254 DEBUG:boto:Method: GET 2023-11-02T18:30:30.255 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.256 DEBUG:boto:Data: 2023-11-02T18:30:30.256 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.257 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.257 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.258 DEBUG:boto:Params: {} 2023-11-02T18:30:30.258 DEBUG:boto:Token: None 2023-11-02T18:30:30.259 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.259 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kJTdsvpgMOIhaEXLvDQfFQSpWtc= 2023-11-02T18:30:30.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kJTdsvpgMOIhaEXLvDQfFQSpWtc=', 'Content-Length': '0'} 2023-11-02T18:30:30.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea7eeccf3be22380-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.306 DEBUG:boto:path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.307 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.308 DEBUG:boto:Method: GET 2023-11-02T18:30:30.308 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.308 DEBUG:boto:Data: 2023-11-02T18:30:30.309 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.309 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.310 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.310 DEBUG:boto:Params: {} 2023-11-02T18:30:30.311 DEBUG:boto:Token: None 2023-11-02T18:30:30.311 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000019 2023-11-02T18:30:30.312 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kJTdsvpgMOIhaEXLvDQfFQSpWtc= 2023-11-02T18:30:30.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kJTdsvpgMOIhaEXLvDQfFQSpWtc=', 'Content-Length': '0'} 2023-11-02T18:30:30.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e08690b5c309676-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.317 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.317 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.318 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000019 2023-11-02T18:30:30.318 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000019 2023-11-02T18:30:30.318 DEBUG:boto:Method: GET 2023-11-02T18:30:30.319 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000019 2023-11-02T18:30:30.319 DEBUG:boto:Data: 2023-11-02T18:30:30.320 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.320 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.321 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.321 DEBUG:boto:Params: {} 2023-11-02T18:30:30.322 DEBUG:boto:Token: None 2023-11-02T18:30:30.322 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.323 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9ee4c86300dbe49-006543eac6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.329 DEBUG:boto:b'ibtljn-58obj000000000000000000191000falseobj000000000000000000192023-11-02T18:30:19.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.330 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.330 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.331 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000019 2023-11-02T18:30:30.331 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000019 2023-11-02T18:30:30.331 DEBUG:boto:Method: GET 2023-11-02T18:30:30.332 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000019 2023-11-02T18:30:30.332 DEBUG:boto:Data: 2023-11-02T18:30:30.333 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.333 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.334 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.334 DEBUG:boto:Params: {} 2023-11-02T18:30:30.335 DEBUG:boto:Token: None 2023-11-02T18:30:30.335 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.336 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c28390e80d31f312-006543eac6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.390 DEBUG:boto:b'ibtljn-58obj000000000000000000191000falseobj000000000000000000192023-11-02T18:30:19.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-11-02T18:30:30.392 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.392 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.393 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.393 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.394 DEBUG:boto:Method: GET 2023-11-02T18:30:30.394 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.394 DEBUG:boto:Data: 2023-11-02T18:30:30.395 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.395 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.396 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.396 DEBUG:boto:Params: {} 2023-11-02T18:30:30.397 DEBUG:boto:Token: None 2023-11-02T18:30:30.397 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.398 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4CZnrP12zXF0GzTJX7NKW4GX4Js= 2023-11-02T18:30:30.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4CZnrP12zXF0GzTJX7NKW4GX4Js=', 'Content-Length': '0'} 2023-11-02T18:30:30.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef87a9a8a90c0c09-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.403 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.403 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.404 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.404 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.405 DEBUG:boto:Method: GET 2023-11-02T18:30:30.405 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.406 DEBUG:boto:Data: 2023-11-02T18:30:30.406 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.406 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.407 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.407 DEBUG:boto:Params: {} 2023-11-02T18:30:30.408 DEBUG:boto:Token: None 2023-11-02T18:30:30.408 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.409 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4CZnrP12zXF0GzTJX7NKW4GX4Js= 2023-11-02T18:30:30.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4CZnrP12zXF0GzTJX7NKW4GX4Js=', 'Content-Length': '0'} 2023-11-02T18:30:30.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbc9c4aa07f9ab8b-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.413 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.413 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.414 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.414 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.415 DEBUG:boto:Method: HEAD 2023-11-02T18:30:30.415 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.415 DEBUG:boto:Data: 2023-11-02T18:30:30.416 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.416 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.417 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.417 DEBUG:boto:Params: {} 2023-11-02T18:30:30.418 DEBUG:boto:Token: None 2023-11-02T18:30:30.418 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.419 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CyFPAKKaC6SmCp3jmoxPQD8IS50= 2023-11-02T18:30:30.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CyFPAKKaC6SmCp3jmoxPQD8IS50=', 'Content-Length': '0'} 2023-11-02T18:30:30.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005858267aa52d71c5-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.423 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.423 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.424 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.424 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.424 DEBUG:boto:Method: HEAD 2023-11-02T18:30:30.425 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.425 DEBUG:boto:Data: 2023-11-02T18:30:30.426 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.427 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.427 DEBUG:boto:Params: {} 2023-11-02T18:30:30.427 DEBUG:boto:Token: None 2023-11-02T18:30:30.428 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CyFPAKKaC6SmCp3jmoxPQD8IS50= 2023-11-02T18:30:30.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CyFPAKKaC6SmCp3jmoxPQD8IS50=', 'Content-Length': '0'} 2023-11-02T18:30:30.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad8f3fe1791b9ae6-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-11-02T18:30:30.433 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.433 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.434 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.434 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.435 DEBUG:boto:Method: GET 2023-11-02T18:30:30.435 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.436 DEBUG:boto:Data: 2023-11-02T18:30:30.436 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.436 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.437 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.437 DEBUG:boto:Params: {} 2023-11-02T18:30:30.438 DEBUG:boto:Token: None 2023-11-02T18:30:30.438 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.439 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4CZnrP12zXF0GzTJX7NKW4GX4Js= 2023-11-02T18:30:30.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4CZnrP12zXF0GzTJX7NKW4GX4Js=', 'Content-Length': '0'} 2023-11-02T18:30:30.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cb4d301b4401835-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.443 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.443 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.444 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.444 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.445 DEBUG:boto:Method: GET 2023-11-02T18:30:30.445 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.446 DEBUG:boto:Data: 2023-11-02T18:30:30.446 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.446 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.447 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.447 DEBUG:boto:Params: {} 2023-11-02T18:30:30.448 DEBUG:boto:Token: None 2023-11-02T18:30:30.448 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000020?versionId=null 2023-11-02T18:30:30.449 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4CZnrP12zXF0GzTJX7NKW4GX4Js= 2023-11-02T18:30:30.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4CZnrP12zXF0GzTJX7NKW4GX4Js=', 'Content-Length': '0'} 2023-11-02T18:30:30.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c81726fdafbb972-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.494 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.495 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.495 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000020 2023-11-02T18:30:30.496 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000020 2023-11-02T18:30:30.496 DEBUG:boto:Method: GET 2023-11-02T18:30:30.497 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000020 2023-11-02T18:30:30.497 DEBUG:boto:Data: 2023-11-02T18:30:30.497 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.498 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.498 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.499 DEBUG:boto:Params: {} 2023-11-02T18:30:30.499 DEBUG:boto:Token: None 2023-11-02T18:30:30.500 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.503 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a297a21317c74675-006543eac6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.509 DEBUG:boto:b'ibtljn-58obj000000000000000000201000falseobj000000000000000000202023-11-02T18:30:20.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.510 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.510 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.511 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000020 2023-11-02T18:30:30.511 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000020 2023-11-02T18:30:30.512 DEBUG:boto:Method: GET 2023-11-02T18:30:30.512 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000020 2023-11-02T18:30:30.513 DEBUG:boto:Data: 2023-11-02T18:30:30.513 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.513 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.514 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.514 DEBUG:boto:Params: {} 2023-11-02T18:30:30.515 DEBUG:boto:Token: None 2023-11-02T18:30:30.516 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.516 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bd767cb26623611-006543eac6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.528 DEBUG:boto:b'ibtljn-58obj000000000000000000201000falseobj000000000000000000202023-11-02T18:30:20.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.529 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.529 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.530 DEBUG:boto:Method: HEAD 2023-11-02T18:30:30.530 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.531 DEBUG:boto:Data: 2023-11-02T18:30:30.531 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.532 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.533 DEBUG:boto:Params: {} 2023-11-02T18:30:30.533 DEBUG:boto:Token: None 2023-11-02T18:30:30.534 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.534 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tGXQpheRaMpPQVulp+aGOANvJho= 2023-11-02T18:30:30.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tGXQpheRaMpPQVulp+aGOANvJho=', 'Content-Length': '0'} 2023-11-02T18:30:30.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000361507c4650fcf5-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.538 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.538 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.539 DEBUG:boto:Method: HEAD 2023-11-02T18:30:30.539 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.540 DEBUG:boto:Data: 2023-11-02T18:30:30.540 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.541 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.541 DEBUG:boto:Params: {} 2023-11-02T18:30:30.542 DEBUG:boto:Token: None 2023-11-02T18:30:30.542 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.543 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tGXQpheRaMpPQVulp+aGOANvJho= 2023-11-02T18:30:30.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tGXQpheRaMpPQVulp+aGOANvJho=', 'Content-Length': '0'} 2023-11-02T18:30:30.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000834d53e9064cce93-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-11-02T18:30:30.547 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.548 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.548 DEBUG:boto:Method: GET 2023-11-02T18:30:30.549 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.549 DEBUG:boto:Data: 2023-11-02T18:30:30.550 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.550 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.550 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.551 DEBUG:boto:Params: {} 2023-11-02T18:30:30.551 DEBUG:boto:Token: None 2023-11-02T18:30:30.552 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.552 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HKkS/4cdCPcP+9MfzLV4p13f1HA= 2023-11-02T18:30:30.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HKkS/4cdCPcP+9MfzLV4p13f1HA=', 'Content-Length': '0'} 2023-11-02T18:30:30.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e4ee79e2355700e-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.598 DEBUG:boto:path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.599 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.600 DEBUG:boto:Method: GET 2023-11-02T18:30:30.600 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.600 DEBUG:boto:Data: 2023-11-02T18:30:30.601 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.601 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.602 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.602 DEBUG:boto:Params: {} 2023-11-02T18:30:30.603 DEBUG:boto:Token: None 2023-11-02T18:30:30.603 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000020 2023-11-02T18:30:30.604 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HKkS/4cdCPcP+9MfzLV4p13f1HA= 2023-11-02T18:30:30.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HKkS/4cdCPcP+9MfzLV4p13f1HA=', 'Content-Length': '0'} 2023-11-02T18:30:30.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b00e2b5c2471525-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.608 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.609 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.609 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000020 2023-11-02T18:30:30.610 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000020 2023-11-02T18:30:30.610 DEBUG:boto:Method: GET 2023-11-02T18:30:30.611 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000020 2023-11-02T18:30:30.611 DEBUG:boto:Data: 2023-11-02T18:30:30.612 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.612 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.613 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.613 DEBUG:boto:Params: {} 2023-11-02T18:30:30.614 DEBUG:boto:Token: None 2023-11-02T18:30:30.614 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.615 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecb0897d41667a0c-006543eac6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.620 DEBUG:boto:b'ibtljn-58obj000000000000000000201000falseobj000000000000000000202023-11-02T18:30:20.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.621 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.622 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.622 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000020 2023-11-02T18:30:30.623 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000020 2023-11-02T18:30:30.623 DEBUG:boto:Method: GET 2023-11-02T18:30:30.624 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000020 2023-11-02T18:30:30.624 DEBUG:boto:Data: 2023-11-02T18:30:30.625 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.625 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.626 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.626 DEBUG:boto:Params: {} 2023-11-02T18:30:30.627 DEBUG:boto:Token: None 2023-11-02T18:30:30.627 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.628 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdbceb6485144ef4-006543eac6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.686 DEBUG:boto:b'ibtljn-58obj000000000000000000201000falseobj000000000000000000202023-11-02T18:30:20.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-11-02T18:30:30.688 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.688 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.689 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.689 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.690 DEBUG:boto:Method: GET 2023-11-02T18:30:30.690 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.691 DEBUG:boto:Data: 2023-11-02T18:30:30.691 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.692 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.692 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.693 DEBUG:boto:Params: {} 2023-11-02T18:30:30.693 DEBUG:boto:Token: None 2023-11-02T18:30:30.694 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.694 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2L4JTUELmA5vPCKSitJGwLqdVcA= 2023-11-02T18:30:30.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2L4JTUELmA5vPCKSitJGwLqdVcA=', 'Content-Length': '0'} 2023-11-02T18:30:30.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d43cf5f08b7c9335-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.699 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.700 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.700 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.700 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.701 DEBUG:boto:Method: GET 2023-11-02T18:30:30.701 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.702 DEBUG:boto:Data: 2023-11-02T18:30:30.702 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.703 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.703 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.704 DEBUG:boto:Params: {} 2023-11-02T18:30:30.704 DEBUG:boto:Token: None 2023-11-02T18:30:30.705 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.706 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2L4JTUELmA5vPCKSitJGwLqdVcA= 2023-11-02T18:30:30.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2L4JTUELmA5vPCKSitJGwLqdVcA=', 'Content-Length': '0'} 2023-11-02T18:30:30.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069a9242198634ace-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.711 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.712 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.712 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.712 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.713 DEBUG:boto:Method: HEAD 2023-11-02T18:30:30.713 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.714 DEBUG:boto:Data: 2023-11-02T18:30:30.714 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.715 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.715 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.715 DEBUG:boto:Params: {} 2023-11-02T18:30:30.716 DEBUG:boto:Token: None 2023-11-02T18:30:30.716 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.717 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S02YLltTutE3dcracAB6H/J/shA= 2023-11-02T18:30:30.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S02YLltTutE3dcracAB6H/J/shA=', 'Content-Length': '0'} 2023-11-02T18:30:30.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2b556bd53d1844f-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.721 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.722 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.722 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.723 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.723 DEBUG:boto:Method: HEAD 2023-11-02T18:30:30.724 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.724 DEBUG:boto:Data: 2023-11-02T18:30:30.725 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.725 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.726 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.726 DEBUG:boto:Params: {} 2023-11-02T18:30:30.726 DEBUG:boto:Token: None 2023-11-02T18:30:30.727 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.728 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S02YLltTutE3dcracAB6H/J/shA= 2023-11-02T18:30:30.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S02YLltTutE3dcracAB6H/J/shA=', 'Content-Length': '0'} 2023-11-02T18:30:30.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd3f4bc898179901-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-11-02T18:30:30.732 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.733 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.733 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.733 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.734 DEBUG:boto:Method: GET 2023-11-02T18:30:30.734 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.735 DEBUG:boto:Data: 2023-11-02T18:30:30.735 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.736 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.736 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.737 DEBUG:boto:Params: {} 2023-11-02T18:30:30.737 DEBUG:boto:Token: None 2023-11-02T18:30:30.738 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.738 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2L4JTUELmA5vPCKSitJGwLqdVcA= 2023-11-02T18:30:30.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2L4JTUELmA5vPCKSitJGwLqdVcA=', 'Content-Length': '0'} 2023-11-02T18:30:30.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095c12d86a4256187-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.743 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.743 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.744 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.744 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.745 DEBUG:boto:Method: GET 2023-11-02T18:30:30.745 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.746 DEBUG:boto:Data: 2023-11-02T18:30:30.746 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.746 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.747 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.747 DEBUG:boto:Params: {} 2023-11-02T18:30:30.748 DEBUG:boto:Token: None 2023-11-02T18:30:30.748 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000021?versionId=null 2023-11-02T18:30:30.749 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2L4JTUELmA5vPCKSitJGwLqdVcA= 2023-11-02T18:30:30.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2L4JTUELmA5vPCKSitJGwLqdVcA=', 'Content-Length': '0'} 2023-11-02T18:30:30.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7c880b3b9d52054-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.798 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.799 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.800 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000021 2023-11-02T18:30:30.800 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000021 2023-11-02T18:30:30.801 DEBUG:boto:Method: GET 2023-11-02T18:30:30.801 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000021 2023-11-02T18:30:30.802 DEBUG:boto:Data: 2023-11-02T18:30:30.802 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.803 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.803 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.804 DEBUG:boto:Params: {} 2023-11-02T18:30:30.804 DEBUG:boto:Token: None 2023-11-02T18:30:30.805 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.805 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a34d20c5ac133e48-006543eac6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.812 DEBUG:boto:b'ibtljn-58obj000000000000000000211000falseobj000000000000000000212023-11-02T18:30:17.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.813 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.814 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.814 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000021 2023-11-02T18:30:30.814 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000021 2023-11-02T18:30:30.815 DEBUG:boto:Method: GET 2023-11-02T18:30:30.816 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000021 2023-11-02T18:30:30.816 DEBUG:boto:Data: 2023-11-02T18:30:30.817 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.817 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.818 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.818 DEBUG:boto:Params: {} 2023-11-02T18:30:30.819 DEBUG:boto:Token: None 2023-11-02T18:30:30.820 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.820 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a91843014b4f9bec-006543eac6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.830 DEBUG:boto:b'ibtljn-58obj000000000000000000211000falseobj000000000000000000212023-11-02T18:30:17.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.831 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.832 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.832 DEBUG:boto:Method: HEAD 2023-11-02T18:30:30.833 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.833 DEBUG:boto:Data: 2023-11-02T18:30:30.834 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.834 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.835 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.835 DEBUG:boto:Params: {} 2023-11-02T18:30:30.836 DEBUG:boto:Token: None 2023-11-02T18:30:30.836 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.837 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QmCMJ/FkxXTsOTHxR3IrWUskmE= 2023-11-02T18:30:30.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QmCMJ/FkxXTsOTHxR3IrWUskmE=', 'Content-Length': '0'} 2023-11-02T18:30:30.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e9354a97ae9f8fa-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.840 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.841 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.841 DEBUG:boto:Method: HEAD 2023-11-02T18:30:30.842 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.842 DEBUG:boto:Data: 2023-11-02T18:30:30.842 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.843 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.843 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.844 DEBUG:boto:Params: {} 2023-11-02T18:30:30.844 DEBUG:boto:Token: None 2023-11-02T18:30:30.845 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.845 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5QmCMJ/FkxXTsOTHxR3IrWUskmE= 2023-11-02T18:30:30.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5QmCMJ/FkxXTsOTHxR3IrWUskmE=', 'Content-Length': '0'} 2023-11-02T18:30:30.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbb990a458ab5e82-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-11-02T18:30:30.849 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.850 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.850 DEBUG:boto:Method: GET 2023-11-02T18:30:30.851 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.851 DEBUG:boto:Data: 2023-11-02T18:30:30.851 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.852 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.853 DEBUG:boto:Params: {} 2023-11-02T18:30:30.853 DEBUG:boto:Token: None 2023-11-02T18:30:30.854 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.854 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/ni6yHh05l2jZyBJij/6MgNGKMQ= 2023-11-02T18:30:30.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/ni6yHh05l2jZyBJij/6MgNGKMQ=', 'Content-Length': '0'} 2023-11-02T18:30:30.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008135ec80f19e6eb8-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.902 DEBUG:boto:path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.903 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.904 DEBUG:boto:Method: GET 2023-11-02T18:30:30.904 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.905 DEBUG:boto:Data: 2023-11-02T18:30:30.905 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.906 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.907 DEBUG:boto:Params: {} 2023-11-02T18:30:30.908 DEBUG:boto:Token: None 2023-11-02T18:30:30.908 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000021 2023-11-02T18:30:30.909 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/ni6yHh05l2jZyBJij/6MgNGKMQ= 2023-11-02T18:30:30.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/ni6yHh05l2jZyBJij/6MgNGKMQ=', 'Content-Length': '0'} 2023-11-02T18:30:30.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000227842ac80e23204-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.913 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.914 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.915 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000021 2023-11-02T18:30:30.915 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000021 2023-11-02T18:30:30.916 DEBUG:boto:Method: GET 2023-11-02T18:30:30.916 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000021 2023-11-02T18:30:30.917 DEBUG:boto:Data: 2023-11-02T18:30:30.918 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.919 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.919 DEBUG:boto:Params: {} 2023-11-02T18:30:30.921 DEBUG:boto:Token: None 2023-11-02T18:30:30.921 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4803355a7564d23-006543eac6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.928 DEBUG:boto:b'ibtljn-58obj000000000000000000211000falseobj000000000000000000212023-11-02T18:30:17.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.929 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:30.930 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:30.930 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000021 2023-11-02T18:30:30.931 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000021 2023-11-02T18:30:30.931 DEBUG:boto:Method: GET 2023-11-02T18:30:30.931 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000021 2023-11-02T18:30:30.932 DEBUG:boto:Data: 2023-11-02T18:30:30.932 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.933 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.933 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.933 DEBUG:boto:Params: {} 2023-11-02T18:30:30.934 DEBUG:boto:Token: None 2023-11-02T18:30:30.934 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/ 2023-11-02T18:30:30.935 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU= 2023-11-02T18:30:30.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1A4jk5QPR2wCwUf99zVddjp0DOU=', 'Content-Length': '0'} 2023-11-02T18:30:30.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078323ee4f4b720ba-006543eac6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.946 DEBUG:boto:b'ibtljn-58obj000000000000000000211000falseobj000000000000000000212023-11-02T18:30:17.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:30.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-11-02T18:30:30.947 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:30.948 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:30.948 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:30.949 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:30.949 DEBUG:boto:Method: GET 2023-11-02T18:30:30.950 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:30.950 DEBUG:boto:Data: 2023-11-02T18:30:30.950 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.951 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:30.951 DEBUG:boto:Port: 8001 2023-11-02T18:30:30.952 DEBUG:boto:Params: {} 2023-11-02T18:30:30.952 DEBUG:boto:Token: None 2023-11-02T18:30:30.953 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:30.953 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zz6hq8twIQWvfCZ+j5rsnMRr1v8= 2023-11-02T18:30:30.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zz6hq8twIQWvfCZ+j5rsnMRr1v8=', 'Content-Length': '0'} 2023-11-02T18:30:30.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fe0cb53ff078719-006543eac6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:30.957 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:30.958 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:30.958 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:30.959 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:30.959 DEBUG:boto:Method: GET 2023-11-02T18:30:30.959 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:30.960 DEBUG:boto:Data: 2023-11-02T18:30:30.960 DEBUG:boto:Headers: {} 2023-11-02T18:30:30.961 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:30.961 DEBUG:boto:Port: 8000 2023-11-02T18:30:30.962 DEBUG:boto:Params: {} 2023-11-02T18:30:30.962 DEBUG:boto:Token: None 2023-11-02T18:30:30.963 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:30 GMT /ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:30.963 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zz6hq8twIQWvfCZ+j5rsnMRr1v8= 2023-11-02T18:30:30.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zz6hq8twIQWvfCZ+j5rsnMRr1v8=', 'Content-Length': '0'} 2023-11-02T18:30:30.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cca00e980dbf6251-006543eac6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.006 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.007 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.007 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.008 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.008 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.009 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.009 DEBUG:boto:Data: 2023-11-02T18:30:31.010 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.010 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.010 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.011 DEBUG:boto:Params: {} 2023-11-02T18:30:31.011 DEBUG:boto:Token: None 2023-11-02T18:30:31.012 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eamfcNjQUwxUlEDV+Uxd0R2C94Y= 2023-11-02T18:30:31.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eamfcNjQUwxUlEDV+Uxd0R2C94Y=', 'Content-Length': '0'} 2023-11-02T18:30:31.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9d8bcf1a15a465e-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.016 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.017 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.017 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.018 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.018 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.019 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.019 DEBUG:boto:Data: 2023-11-02T18:30:31.019 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.020 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.020 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.021 DEBUG:boto:Params: {} 2023-11-02T18:30:31.021 DEBUG:boto:Token: None 2023-11-02T18:30:31.022 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.022 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eamfcNjQUwxUlEDV+Uxd0R2C94Y= 2023-11-02T18:30:31.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eamfcNjQUwxUlEDV+Uxd0R2C94Y=', 'Content-Length': '0'} 2023-11-02T18:30:31.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df5634832bcdee16-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-11-02T18:30:31.026 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.027 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.027 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.027 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.028 DEBUG:boto:Method: GET 2023-11-02T18:30:31.028 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.029 DEBUG:boto:Data: 2023-11-02T18:30:31.029 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.030 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.031 DEBUG:boto:Params: {} 2023-11-02T18:30:31.031 DEBUG:boto:Token: None 2023-11-02T18:30:31.032 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gBpajqaL3lpXvsb7Fsg6FmvFFWI= 2023-11-02T18:30:31.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gBpajqaL3lpXvsb7Fsg6FmvFFWI=', 'Content-Length': '0'} 2023-11-02T18:30:31.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3538d9177c399b-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.078 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.079 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.079 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.080 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.080 DEBUG:boto:Method: GET 2023-11-02T18:30:31.081 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.081 DEBUG:boto:Data: 2023-11-02T18:30:31.082 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.082 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.082 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.083 DEBUG:boto:Params: {} 2023-11-02T18:30:31.083 DEBUG:boto:Token: None 2023-11-02T18:30:31.084 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000022?versionId=null 2023-11-02T18:30:31.084 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gBpajqaL3lpXvsb7Fsg6FmvFFWI= 2023-11-02T18:30:31.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gBpajqaL3lpXvsb7Fsg6FmvFFWI=', 'Content-Length': '0'} 2023-11-02T18:30:31.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ebc8f061fb26f07-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.088 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.089 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.089 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000022 2023-11-02T18:30:31.090 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000022 2023-11-02T18:30:31.090 DEBUG:boto:Method: GET 2023-11-02T18:30:31.091 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000022 2023-11-02T18:30:31.091 DEBUG:boto:Data: 2023-11-02T18:30:31.091 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.092 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.092 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.093 DEBUG:boto:Params: {} 2023-11-02T18:30:31.093 DEBUG:boto:Token: None 2023-11-02T18:30:31.094 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daccdc901418e28b-006543eac7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.100 DEBUG:boto:b'ibtljn-58obj000000000000000000221000falseobj000000000000000000222023-11-02T18:30:17.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.101 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.101 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.102 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000022 2023-11-02T18:30:31.102 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000022 2023-11-02T18:30:31.103 DEBUG:boto:Method: GET 2023-11-02T18:30:31.103 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000022 2023-11-02T18:30:31.103 DEBUG:boto:Data: 2023-11-02T18:30:31.104 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.104 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.105 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.105 DEBUG:boto:Params: {} 2023-11-02T18:30:31.105 DEBUG:boto:Token: None 2023-11-02T18:30:31.106 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.106 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fc4c5cea01be623-006543eac7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.117 DEBUG:boto:b'ibtljn-58obj000000000000000000221000falseobj000000000000000000222023-11-02T18:30:17.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.117 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.118 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.118 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.119 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.119 DEBUG:boto:Data: 2023-11-02T18:30:31.119 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.120 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.120 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.121 DEBUG:boto:Params: {} 2023-11-02T18:30:31.121 DEBUG:boto:Token: None 2023-11-02T18:30:31.122 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.122 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IHM5to4X+sQGt8l8ObWAFeauVhQ= 2023-11-02T18:30:31.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IHM5to4X+sQGt8l8ObWAFeauVhQ=', 'Content-Length': '0'} 2023-11-02T18:30:31.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b7c82cfe91a98a-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.126 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.126 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.126 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.127 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.127 DEBUG:boto:Data: 2023-11-02T18:30:31.128 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.128 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.129 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.129 DEBUG:boto:Params: {} 2023-11-02T18:30:31.129 DEBUG:boto:Token: None 2023-11-02T18:30:31.130 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IHM5to4X+sQGt8l8ObWAFeauVhQ= 2023-11-02T18:30:31.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IHM5to4X+sQGt8l8ObWAFeauVhQ=', 'Content-Length': '0'} 2023-11-02T18:30:31.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d169caad2b4e844c-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-11-02T18:30:31.134 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.134 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.135 DEBUG:boto:Method: GET 2023-11-02T18:30:31.135 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.135 DEBUG:boto:Data: 2023-11-02T18:30:31.136 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.136 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.137 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.137 DEBUG:boto:Params: {} 2023-11-02T18:30:31.138 DEBUG:boto:Token: None 2023-11-02T18:30:31.138 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.139 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o1/VNBQMcmSCRdM0a1MG47dVJgA= 2023-11-02T18:30:31.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o1/VNBQMcmSCRdM0a1MG47dVJgA=', 'Content-Length': '0'} 2023-11-02T18:30:31.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2bc547ee9afdc2d-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.142 DEBUG:boto:path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.143 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.143 DEBUG:boto:Method: GET 2023-11-02T18:30:31.144 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.144 DEBUG:boto:Data: 2023-11-02T18:30:31.144 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.145 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.145 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.146 DEBUG:boto:Params: {} 2023-11-02T18:30:31.146 DEBUG:boto:Token: None 2023-11-02T18:30:31.147 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000022 2023-11-02T18:30:31.147 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o1/VNBQMcmSCRdM0a1MG47dVJgA= 2023-11-02T18:30:31.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o1/VNBQMcmSCRdM0a1MG47dVJgA=', 'Content-Length': '0'} 2023-11-02T18:30:31.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004de1adf57f423ff2-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.194 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.195 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.195 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000022 2023-11-02T18:30:31.196 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000022 2023-11-02T18:30:31.196 DEBUG:boto:Method: GET 2023-11-02T18:30:31.197 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000022 2023-11-02T18:30:31.197 DEBUG:boto:Data: 2023-11-02T18:30:31.197 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.198 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.198 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.199 DEBUG:boto:Params: {} 2023-11-02T18:30:31.199 DEBUG:boto:Token: None 2023-11-02T18:30:31.200 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.200 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1794f7420bcbae2-006543eac7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.207 DEBUG:boto:b'ibtljn-58obj000000000000000000221000falseobj000000000000000000222023-11-02T18:30:17.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.207 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.208 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.208 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000022 2023-11-02T18:30:31.209 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000022 2023-11-02T18:30:31.209 DEBUG:boto:Method: GET 2023-11-02T18:30:31.210 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000022 2023-11-02T18:30:31.210 DEBUG:boto:Data: 2023-11-02T18:30:31.211 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.211 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.211 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.212 DEBUG:boto:Params: {} 2023-11-02T18:30:31.212 DEBUG:boto:Token: None 2023-11-02T18:30:31.213 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.213 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9a50c711f177f84-006543eac7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.224 DEBUG:boto:b'ibtljn-58obj000000000000000000221000falseobj000000000000000000222023-11-02T18:30:17.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-11-02T18:30:31.225 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.225 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.226 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.226 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.227 DEBUG:boto:Method: GET 2023-11-02T18:30:31.227 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.228 DEBUG:boto:Data: 2023-11-02T18:30:31.228 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.228 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.229 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.229 DEBUG:boto:Params: {} 2023-11-02T18:30:31.230 DEBUG:boto:Token: None 2023-11-02T18:30:31.230 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.231 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6rHsjwnQkXfXJZf5G7Nii+gdPeU= 2023-11-02T18:30:31.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6rHsjwnQkXfXJZf5G7Nii+gdPeU=', 'Content-Length': '0'} 2023-11-02T18:30:31.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099960010a76720ff-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.274 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.275 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.275 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.276 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.276 DEBUG:boto:Method: GET 2023-11-02T18:30:31.277 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.277 DEBUG:boto:Data: 2023-11-02T18:30:31.278 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.278 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.279 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.279 DEBUG:boto:Params: {} 2023-11-02T18:30:31.280 DEBUG:boto:Token: None 2023-11-02T18:30:31.281 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.281 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6rHsjwnQkXfXJZf5G7Nii+gdPeU= 2023-11-02T18:30:31.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6rHsjwnQkXfXJZf5G7Nii+gdPeU=', 'Content-Length': '0'} 2023-11-02T18:30:31.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004494ecf4084fae1-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.285 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.286 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.286 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.287 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.287 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.288 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.288 DEBUG:boto:Data: 2023-11-02T18:30:31.289 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.289 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.289 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.290 DEBUG:boto:Params: {} 2023-11-02T18:30:31.290 DEBUG:boto:Token: None 2023-11-02T18:30:31.291 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.291 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n7oLIckOuqSDbfU4g7dkGDCGm14= 2023-11-02T18:30:31.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n7oLIckOuqSDbfU4g7dkGDCGm14=', 'Content-Length': '0'} 2023-11-02T18:30:31.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050ae183668b83c9b-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.295 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.296 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.296 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.297 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.297 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.298 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.298 DEBUG:boto:Data: 2023-11-02T18:30:31.299 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.299 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.299 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.300 DEBUG:boto:Params: {} 2023-11-02T18:30:31.300 DEBUG:boto:Token: None 2023-11-02T18:30:31.301 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.301 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n7oLIckOuqSDbfU4g7dkGDCGm14= 2023-11-02T18:30:31.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n7oLIckOuqSDbfU4g7dkGDCGm14=', 'Content-Length': '0'} 2023-11-02T18:30:31.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000475d46195bd65e05-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-11-02T18:30:31.306 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.306 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.307 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.307 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.308 DEBUG:boto:Method: GET 2023-11-02T18:30:31.308 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.308 DEBUG:boto:Data: 2023-11-02T18:30:31.309 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.309 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.310 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.310 DEBUG:boto:Params: {} 2023-11-02T18:30:31.311 DEBUG:boto:Token: None 2023-11-02T18:30:31.311 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.312 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6rHsjwnQkXfXJZf5G7Nii+gdPeU= 2023-11-02T18:30:31.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6rHsjwnQkXfXJZf5G7Nii+gdPeU=', 'Content-Length': '0'} 2023-11-02T18:30:31.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c8c478329381e3-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.316 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.316 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.317 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.317 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.318 DEBUG:boto:Method: GET 2023-11-02T18:30:31.318 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.318 DEBUG:boto:Data: 2023-11-02T18:30:31.319 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.319 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.320 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.320 DEBUG:boto:Params: {} 2023-11-02T18:30:31.321 DEBUG:boto:Token: None 2023-11-02T18:30:31.321 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000023?versionId=null 2023-11-02T18:30:31.322 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6rHsjwnQkXfXJZf5G7Nii+gdPeU= 2023-11-02T18:30:31.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6rHsjwnQkXfXJZf5G7Nii+gdPeU=', 'Content-Length': '0'} 2023-11-02T18:30:31.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d40c599c9d7066bb-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.326 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.327 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.327 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000023 2023-11-02T18:30:31.328 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000023 2023-11-02T18:30:31.328 DEBUG:boto:Method: GET 2023-11-02T18:30:31.329 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000023 2023-11-02T18:30:31.329 DEBUG:boto:Data: 2023-11-02T18:30:31.330 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.330 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.330 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.331 DEBUG:boto:Params: {} 2023-11-02T18:30:31.331 DEBUG:boto:Token: None 2023-11-02T18:30:31.332 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.332 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085b69f419a4ae221-006543eac7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.378 DEBUG:boto:b'ibtljn-58obj000000000000000000231000falseobj000000000000000000232023-11-02T18:30:18.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.379 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.380 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.380 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000023 2023-11-02T18:30:31.381 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000023 2023-11-02T18:30:31.381 DEBUG:boto:Method: GET 2023-11-02T18:30:31.382 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000023 2023-11-02T18:30:31.382 DEBUG:boto:Data: 2023-11-02T18:30:31.383 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.383 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.384 DEBUG:boto:Params: {} 2023-11-02T18:30:31.385 DEBUG:boto:Token: None 2023-11-02T18:30:31.385 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.386 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a57e4c46f78d1877-006543eac7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.397 DEBUG:boto:b'ibtljn-58obj000000000000000000231000falseobj000000000000000000232023-11-02T18:30:18.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.397 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.398 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.399 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.399 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.399 DEBUG:boto:Data: 2023-11-02T18:30:31.400 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.400 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.401 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.401 DEBUG:boto:Params: {} 2023-11-02T18:30:31.402 DEBUG:boto:Token: None 2023-11-02T18:30:31.402 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.403 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9s8hNh1uG8TqwjXm1P6Z1f6a6kI= 2023-11-02T18:30:31.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9s8hNh1uG8TqwjXm1P6Z1f6a6kI=', 'Content-Length': '0'} 2023-11-02T18:30:31.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065d05d2d8deae854-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.406 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.406 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.407 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.407 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.408 DEBUG:boto:Data: 2023-11-02T18:30:31.408 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.409 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.409 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.410 DEBUG:boto:Params: {} 2023-11-02T18:30:31.410 DEBUG:boto:Token: None 2023-11-02T18:30:31.410 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.411 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9s8hNh1uG8TqwjXm1P6Z1f6a6kI= 2023-11-02T18:30:31.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9s8hNh1uG8TqwjXm1P6Z1f6a6kI=', 'Content-Length': '0'} 2023-11-02T18:30:31.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8f4b38f8ea21711-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-11-02T18:30:31.415 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.416 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.417 DEBUG:boto:Method: GET 2023-11-02T18:30:31.417 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.418 DEBUG:boto:Data: 2023-11-02T18:30:31.418 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.419 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.419 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.420 DEBUG:boto:Params: {} 2023-11-02T18:30:31.420 DEBUG:boto:Token: None 2023-11-02T18:30:31.421 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.422 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PAo9F6OtMiE1sJF5nezCM6JlkrI= 2023-11-02T18:30:31.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PAo9F6OtMiE1sJF5nezCM6JlkrI=', 'Content-Length': '0'} 2023-11-02T18:30:31.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ef8d1402704a7e-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.426 DEBUG:boto:path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.427 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.427 DEBUG:boto:Method: GET 2023-11-02T18:30:31.428 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.428 DEBUG:boto:Data: 2023-11-02T18:30:31.429 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.429 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.429 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.430 DEBUG:boto:Params: {} 2023-11-02T18:30:31.430 DEBUG:boto:Token: None 2023-11-02T18:30:31.431 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000023 2023-11-02T18:30:31.431 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PAo9F6OtMiE1sJF5nezCM6JlkrI= 2023-11-02T18:30:31.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PAo9F6OtMiE1sJF5nezCM6JlkrI=', 'Content-Length': '0'} 2023-11-02T18:30:31.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000203a17360aabfbf6-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.435 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.436 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.436 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000023 2023-11-02T18:30:31.437 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000023 2023-11-02T18:30:31.437 DEBUG:boto:Method: GET 2023-11-02T18:30:31.438 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000023 2023-11-02T18:30:31.438 DEBUG:boto:Data: 2023-11-02T18:30:31.439 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.439 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.439 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.440 DEBUG:boto:Params: {} 2023-11-02T18:30:31.440 DEBUG:boto:Token: None 2023-11-02T18:30:31.441 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.441 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b05e4b2b3bcc6a5-006543eac7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.448 DEBUG:boto:b'ibtljn-58obj000000000000000000231000falseobj000000000000000000232023-11-02T18:30:18.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.448 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.449 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.449 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000023 2023-11-02T18:30:31.450 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000023 2023-11-02T18:30:31.450 DEBUG:boto:Method: GET 2023-11-02T18:30:31.451 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000023 2023-11-02T18:30:31.451 DEBUG:boto:Data: 2023-11-02T18:30:31.452 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.452 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.453 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.453 DEBUG:boto:Params: {} 2023-11-02T18:30:31.454 DEBUG:boto:Token: None 2023-11-02T18:30:31.454 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.455 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ce92e55ae1eccc2-006543eac7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.506 DEBUG:boto:b'ibtljn-58obj000000000000000000231000falseobj000000000000000000232023-11-02T18:30:18.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-11-02T18:30:31.508 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.508 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.509 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.509 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.510 DEBUG:boto:Method: GET 2023-11-02T18:30:31.510 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.510 DEBUG:boto:Data: 2023-11-02T18:30:31.511 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.511 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.512 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.512 DEBUG:boto:Params: {} 2023-11-02T18:30:31.513 DEBUG:boto:Token: None 2023-11-02T18:30:31.513 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.514 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JSHt65gm6fN9njCwnhXLWSY9ogw= 2023-11-02T18:30:31.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JSHt65gm6fN9njCwnhXLWSY9ogw=', 'Content-Length': '0'} 2023-11-02T18:30:31.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fc900dc6b8c6dac-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.519 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.519 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.520 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.521 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.521 DEBUG:boto:Method: GET 2023-11-02T18:30:31.522 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.522 DEBUG:boto:Data: 2023-11-02T18:30:31.523 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.526 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.526 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.527 DEBUG:boto:Params: {} 2023-11-02T18:30:31.527 DEBUG:boto:Token: None 2023-11-02T18:30:31.528 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.528 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JSHt65gm6fN9njCwnhXLWSY9ogw= 2023-11-02T18:30:31.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JSHt65gm6fN9njCwnhXLWSY9ogw=', 'Content-Length': '0'} 2023-11-02T18:30:31.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099ebd1072a23ac18-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.532 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.533 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.533 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.534 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.534 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.535 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.535 DEBUG:boto:Data: 2023-11-02T18:30:31.535 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.536 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.536 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.537 DEBUG:boto:Params: {} 2023-11-02T18:30:31.537 DEBUG:boto:Token: None 2023-11-02T18:30:31.538 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.538 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CXl3oVEeraxrHqXd3fxYLyvTFzM= 2023-11-02T18:30:31.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CXl3oVEeraxrHqXd3fxYLyvTFzM=', 'Content-Length': '0'} 2023-11-02T18:30:31.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f497c0aedb7b0c9c-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.542 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.543 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.543 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.543 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.544 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.544 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.545 DEBUG:boto:Data: 2023-11-02T18:30:31.545 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.545 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.546 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.546 DEBUG:boto:Params: {} 2023-11-02T18:30:31.547 DEBUG:boto:Token: None 2023-11-02T18:30:31.548 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.548 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CXl3oVEeraxrHqXd3fxYLyvTFzM= 2023-11-02T18:30:31.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CXl3oVEeraxrHqXd3fxYLyvTFzM=', 'Content-Length': '0'} 2023-11-02T18:30:31.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4cde65000633ff4-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-11-02T18:30:31.553 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.553 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.554 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.554 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.555 DEBUG:boto:Method: GET 2023-11-02T18:30:31.555 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.556 DEBUG:boto:Data: 2023-11-02T18:30:31.556 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.557 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.557 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.558 DEBUG:boto:Params: {} 2023-11-02T18:30:31.558 DEBUG:boto:Token: None 2023-11-02T18:30:31.559 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.559 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JSHt65gm6fN9njCwnhXLWSY9ogw= 2023-11-02T18:30:31.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JSHt65gm6fN9njCwnhXLWSY9ogw=', 'Content-Length': '0'} 2023-11-02T18:30:31.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fcc268fc8bf89f3-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.564 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.564 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.565 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.565 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.565 DEBUG:boto:Method: GET 2023-11-02T18:30:31.566 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.566 DEBUG:boto:Data: 2023-11-02T18:30:31.567 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.567 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.568 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.568 DEBUG:boto:Params: {} 2023-11-02T18:30:31.569 DEBUG:boto:Token: None 2023-11-02T18:30:31.569 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000024?versionId=null 2023-11-02T18:30:31.570 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JSHt65gm6fN9njCwnhXLWSY9ogw= 2023-11-02T18:30:31.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JSHt65gm6fN9njCwnhXLWSY9ogw=', 'Content-Length': '0'} 2023-11-02T18:30:31.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000844de28c9973b55d-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.614 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.615 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.616 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000024 2023-11-02T18:30:31.616 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000024 2023-11-02T18:30:31.616 DEBUG:boto:Method: GET 2023-11-02T18:30:31.617 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000024 2023-11-02T18:30:31.617 DEBUG:boto:Data: 2023-11-02T18:30:31.618 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.618 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.619 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.619 DEBUG:boto:Params: {} 2023-11-02T18:30:31.620 DEBUG:boto:Token: None 2023-11-02T18:30:31.620 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0b471c0ea626dd3-006543eac7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.627 DEBUG:boto:b'ibtljn-58obj000000000000000000241000falseobj000000000000000000242023-11-02T18:30:19.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.627 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.628 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.628 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000024 2023-11-02T18:30:31.629 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000024 2023-11-02T18:30:31.629 DEBUG:boto:Method: GET 2023-11-02T18:30:31.630 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000024 2023-11-02T18:30:31.630 DEBUG:boto:Data: 2023-11-02T18:30:31.631 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.631 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.632 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.632 DEBUG:boto:Params: {} 2023-11-02T18:30:31.633 DEBUG:boto:Token: None 2023-11-02T18:30:31.633 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.634 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098be5dfb9121456a-006543eac7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.643 DEBUG:boto:b'ibtljn-58obj000000000000000000241000falseobj000000000000000000242023-11-02T18:30:19.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.644 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.644 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.645 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.645 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.646 DEBUG:boto:Data: 2023-11-02T18:30:31.646 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.646 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.647 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.647 DEBUG:boto:Params: {} 2023-11-02T18:30:31.648 DEBUG:boto:Token: None 2023-11-02T18:30:31.648 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.649 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f5uZlo0luT/Q/Ck+GqazcsK6mPw= 2023-11-02T18:30:31.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f5uZlo0luT/Q/Ck+GqazcsK6mPw=', 'Content-Length': '0'} 2023-11-02T18:30:31.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8b3a9dfd3df8565-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.652 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.653 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.653 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.654 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.654 DEBUG:boto:Data: 2023-11-02T18:30:31.654 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.655 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.655 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.656 DEBUG:boto:Params: {} 2023-11-02T18:30:31.656 DEBUG:boto:Token: None 2023-11-02T18:30:31.657 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.657 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f5uZlo0luT/Q/Ck+GqazcsK6mPw= 2023-11-02T18:30:31.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f5uZlo0luT/Q/Ck+GqazcsK6mPw=', 'Content-Length': '0'} 2023-11-02T18:30:31.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0e9d12b29a33044-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-11-02T18:30:31.662 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.662 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.663 DEBUG:boto:Method: GET 2023-11-02T18:30:31.663 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.664 DEBUG:boto:Data: 2023-11-02T18:30:31.664 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.665 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.665 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.666 DEBUG:boto:Params: {} 2023-11-02T18:30:31.666 DEBUG:boto:Token: None 2023-11-02T18:30:31.667 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.667 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8a1gXwscfpwWAiC/P5t0BylbkAk= 2023-11-02T18:30:31.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8a1gXwscfpwWAiC/P5t0BylbkAk=', 'Content-Length': '0'} 2023-11-02T18:30:31.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e8e11cd682d9e7b-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.718 DEBUG:boto:path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.719 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.719 DEBUG:boto:Method: GET 2023-11-02T18:30:31.720 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.720 DEBUG:boto:Data: 2023-11-02T18:30:31.721 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.721 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.722 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.722 DEBUG:boto:Params: {} 2023-11-02T18:30:31.723 DEBUG:boto:Token: None 2023-11-02T18:30:31.723 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000024 2023-11-02T18:30:31.724 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8a1gXwscfpwWAiC/P5t0BylbkAk= 2023-11-02T18:30:31.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8a1gXwscfpwWAiC/P5t0BylbkAk=', 'Content-Length': '0'} 2023-11-02T18:30:31.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c677590c712a4b4-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.729 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.729 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.730 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000024 2023-11-02T18:30:31.730 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000024 2023-11-02T18:30:31.730 DEBUG:boto:Method: GET 2023-11-02T18:30:31.731 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000024 2023-11-02T18:30:31.731 DEBUG:boto:Data: 2023-11-02T18:30:31.732 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.732 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.733 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.733 DEBUG:boto:Params: {} 2023-11-02T18:30:31.734 DEBUG:boto:Token: None 2023-11-02T18:30:31.734 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.735 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca13f0119d9fb953-006543eac7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.741 DEBUG:boto:b'ibtljn-58obj000000000000000000241000falseobj000000000000000000242023-11-02T18:30:19.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.742 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.742 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.742 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000024 2023-11-02T18:30:31.743 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000024 2023-11-02T18:30:31.743 DEBUG:boto:Method: GET 2023-11-02T18:30:31.744 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000024 2023-11-02T18:30:31.744 DEBUG:boto:Data: 2023-11-02T18:30:31.745 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.745 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.746 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.746 DEBUG:boto:Params: {} 2023-11-02T18:30:31.747 DEBUG:boto:Token: None 2023-11-02T18:30:31.747 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.748 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c41c0d860b0ad461-006543eac7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.798 DEBUG:boto:b'ibtljn-58obj000000000000000000241000falseobj000000000000000000242023-11-02T18:30:19.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-11-02T18:30:31.800 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.800 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.800 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.801 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.801 DEBUG:boto:Method: GET 2023-11-02T18:30:31.802 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.802 DEBUG:boto:Data: 2023-11-02T18:30:31.803 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.803 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.804 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.804 DEBUG:boto:Params: {} 2023-11-02T18:30:31.805 DEBUG:boto:Token: None 2023-11-02T18:30:31.805 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.806 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MQB5f0rOu85Q4hXLgxLXYxsZPxU= 2023-11-02T18:30:31.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MQB5f0rOu85Q4hXLgxLXYxsZPxU=', 'Content-Length': '0'} 2023-11-02T18:30:31.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3cba1ecccbc2d2f-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.810 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.811 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.811 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.811 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.812 DEBUG:boto:Method: GET 2023-11-02T18:30:31.812 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.813 DEBUG:boto:Data: 2023-11-02T18:30:31.813 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.814 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.814 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.815 DEBUG:boto:Params: {} 2023-11-02T18:30:31.815 DEBUG:boto:Token: None 2023-11-02T18:30:31.816 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.816 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MQB5f0rOu85Q4hXLgxLXYxsZPxU= 2023-11-02T18:30:31.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MQB5f0rOu85Q4hXLgxLXYxsZPxU=', 'Content-Length': '0'} 2023-11-02T18:30:31.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b1a0b44294db9f0-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.820 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.821 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.821 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.822 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.822 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.823 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.823 DEBUG:boto:Data: 2023-11-02T18:30:31.823 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.824 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.824 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.825 DEBUG:boto:Params: {} 2023-11-02T18:30:31.825 DEBUG:boto:Token: None 2023-11-02T18:30:31.826 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.826 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3gEwW83VRjfMK3F0eqZGrLTKD+Y= 2023-11-02T18:30:31.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3gEwW83VRjfMK3F0eqZGrLTKD+Y=', 'Content-Length': '0'} 2023-11-02T18:30:31.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed403882718a2b3-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.830 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.831 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.831 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.831 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.832 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.832 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.833 DEBUG:boto:Data: 2023-11-02T18:30:31.833 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.834 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.834 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.834 DEBUG:boto:Params: {} 2023-11-02T18:30:31.835 DEBUG:boto:Token: None 2023-11-02T18:30:31.835 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.836 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3gEwW83VRjfMK3F0eqZGrLTKD+Y= 2023-11-02T18:30:31.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3gEwW83VRjfMK3F0eqZGrLTKD+Y=', 'Content-Length': '0'} 2023-11-02T18:30:31.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b6b6861a3917c5-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-11-02T18:30:31.840 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.841 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.841 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.841 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.842 DEBUG:boto:Method: GET 2023-11-02T18:30:31.842 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.843 DEBUG:boto:Data: 2023-11-02T18:30:31.843 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.844 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.844 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.845 DEBUG:boto:Params: {} 2023-11-02T18:30:31.845 DEBUG:boto:Token: None 2023-11-02T18:30:31.846 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.846 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MQB5f0rOu85Q4hXLgxLXYxsZPxU= 2023-11-02T18:30:31.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MQB5f0rOu85Q4hXLgxLXYxsZPxU=', 'Content-Length': '0'} 2023-11-02T18:30:31.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0106918ca89f49b-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.851 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.851 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.852 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.852 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.853 DEBUG:boto:Method: GET 2023-11-02T18:30:31.853 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.854 DEBUG:boto:Data: 2023-11-02T18:30:31.854 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.855 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.855 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.856 DEBUG:boto:Params: {} 2023-11-02T18:30:31.856 DEBUG:boto:Token: None 2023-11-02T18:30:31.857 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000025?versionId=null 2023-11-02T18:30:31.857 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MQB5f0rOu85Q4hXLgxLXYxsZPxU= 2023-11-02T18:30:31.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MQB5f0rOu85Q4hXLgxLXYxsZPxU=', 'Content-Length': '0'} 2023-11-02T18:30:31.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c78cc208b6001bf-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.902 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.903 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.904 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000025 2023-11-02T18:30:31.904 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000025 2023-11-02T18:30:31.904 DEBUG:boto:Method: GET 2023-11-02T18:30:31.905 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000025 2023-11-02T18:30:31.905 DEBUG:boto:Data: 2023-11-02T18:30:31.906 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.907 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.907 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.908 DEBUG:boto:Params: {} 2023-11-02T18:30:31.908 DEBUG:boto:Token: None 2023-11-02T18:30:31.909 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.909 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000280312e2a38e094f-006543eac7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.915 DEBUG:boto:b'ibtljn-58obj000000000000000000251000falseobj000000000000000000252023-11-02T18:30:16.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.916 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:31.916 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:31.917 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000025 2023-11-02T18:30:31.917 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000025 2023-11-02T18:30:31.918 DEBUG:boto:Method: GET 2023-11-02T18:30:31.918 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000025 2023-11-02T18:30:31.919 DEBUG:boto:Data: 2023-11-02T18:30:31.919 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.920 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.920 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.921 DEBUG:boto:Params: {} 2023-11-02T18:30:31.921 DEBUG:boto:Token: None 2023-11-02T18:30:31.922 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/ 2023-11-02T18:30:31.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY= 2023-11-02T18:30:31.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rfvy5s070cG3viS64622JXTp2FY=', 'Content-Length': '0'} 2023-11-02T18:30:31.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a71088d0c0ccf652-006543eac7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.934 DEBUG:boto:b'ibtljn-58obj000000000000000000251000falseobj000000000000000000252023-11-02T18:30:16.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:31.935 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.935 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.936 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.936 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.937 DEBUG:boto:Data: 2023-11-02T18:30:31.937 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.937 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.938 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.938 DEBUG:boto:Params: {} 2023-11-02T18:30:31.939 DEBUG:boto:Token: None 2023-11-02T18:30:31.939 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.940 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J34RApgzrA5yDn/WC2bUzdIQYoQ= 2023-11-02T18:30:31.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J34RApgzrA5yDn/WC2bUzdIQYoQ=', 'Content-Length': '0'} 2023-11-02T18:30:31.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047e645361dfbf3f4-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.943 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.944 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.944 DEBUG:boto:Method: HEAD 2023-11-02T18:30:31.945 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.945 DEBUG:boto:Data: 2023-11-02T18:30:31.946 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.946 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:31.946 DEBUG:boto:Port: 8000 2023-11-02T18:30:31.947 DEBUG:boto:Params: {} 2023-11-02T18:30:31.947 DEBUG:boto:Token: None 2023-11-02T18:30:31.948 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.948 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J34RApgzrA5yDn/WC2bUzdIQYoQ= 2023-11-02T18:30:31.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J34RApgzrA5yDn/WC2bUzdIQYoQ=', 'Content-Length': '0'} 2023-11-02T18:30:31.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053ce67dad8d072dd-006543eac7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:31.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-11-02T18:30:31.952 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.953 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.953 DEBUG:boto:Method: GET 2023-11-02T18:30:31.954 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.954 DEBUG:boto:Data: 2023-11-02T18:30:31.955 DEBUG:boto:Headers: {} 2023-11-02T18:30:31.955 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:31.956 DEBUG:boto:Port: 8001 2023-11-02T18:30:31.956 DEBUG:boto:Params: {} 2023-11-02T18:30:31.957 DEBUG:boto:Token: None 2023-11-02T18:30:31.957 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:31 GMT /ibtljn-58/obj00000000000000000025 2023-11-02T18:30:31.958 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F3WoBzhmWeXp421mYyQodYEcSUg= 2023-11-02T18:30:31.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F3WoBzhmWeXp421mYyQodYEcSUg=', 'Content-Length': '0'} 2023-11-02T18:30:31.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025219e0aef97f990-006543eac7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.006 DEBUG:boto:path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:32.007 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000025 2023-11-02T18:30:32.007 DEBUG:boto:Method: GET 2023-11-02T18:30:32.008 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000025 2023-11-02T18:30:32.008 DEBUG:boto:Data: 2023-11-02T18:30:32.009 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.009 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.010 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.010 DEBUG:boto:Params: {} 2023-11-02T18:30:32.011 DEBUG:boto:Token: None 2023-11-02T18:30:32.011 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000025 2023-11-02T18:30:32.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yoHgodvRCRaEeWbSpUqrqQCCR7c= 2023-11-02T18:30:32.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yoHgodvRCRaEeWbSpUqrqQCCR7c=', 'Content-Length': '0'} 2023-11-02T18:30:32.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4b13cf84d713dce-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.016 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.017 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.017 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000025 2023-11-02T18:30:32.018 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000025 2023-11-02T18:30:32.018 DEBUG:boto:Method: GET 2023-11-02T18:30:32.019 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000025 2023-11-02T18:30:32.019 DEBUG:boto:Data: 2023-11-02T18:30:32.019 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.020 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.020 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.021 DEBUG:boto:Params: {} 2023-11-02T18:30:32.021 DEBUG:boto:Token: None 2023-11-02T18:30:32.022 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.022 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cc16e92cb80953b-006543eac8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.029 DEBUG:boto:b'ibtljn-58obj000000000000000000251000falseobj000000000000000000252023-11-02T18:30:16.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.030 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.031 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.031 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000025 2023-11-02T18:30:32.031 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000025 2023-11-02T18:30:32.032 DEBUG:boto:Method: GET 2023-11-02T18:30:32.032 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000025 2023-11-02T18:30:32.033 DEBUG:boto:Data: 2023-11-02T18:30:32.033 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.034 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.034 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.035 DEBUG:boto:Params: {} 2023-11-02T18:30:32.035 DEBUG:boto:Token: None 2023-11-02T18:30:32.036 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.036 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edb937e97f2a6bf9-006543eac8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.086 DEBUG:boto:b'ibtljn-58obj000000000000000000251000falseobj000000000000000000252023-11-02T18:30:16.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-11-02T18:30:32.087 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.088 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.088 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.089 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.089 DEBUG:boto:Method: GET 2023-11-02T18:30:32.090 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.090 DEBUG:boto:Data: 2023-11-02T18:30:32.091 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.091 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.092 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.092 DEBUG:boto:Params: {} 2023-11-02T18:30:32.093 DEBUG:boto:Token: None 2023-11-02T18:30:32.093 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QR/cc2XXcHYH6crVX/b4dx//uCc= 2023-11-02T18:30:32.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QR/cc2XXcHYH6crVX/b4dx//uCc=', 'Content-Length': '0'} 2023-11-02T18:30:32.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040107636714115ea-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.099 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.099 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.100 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.100 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.101 DEBUG:boto:Method: GET 2023-11-02T18:30:32.101 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.102 DEBUG:boto:Data: 2023-11-02T18:30:32.102 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.103 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.104 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.104 DEBUG:boto:Params: {} 2023-11-02T18:30:32.105 DEBUG:boto:Token: None 2023-11-02T18:30:32.105 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.106 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QR/cc2XXcHYH6crVX/b4dx//uCc= 2023-11-02T18:30:32.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QR/cc2XXcHYH6crVX/b4dx//uCc=', 'Content-Length': '0'} 2023-11-02T18:30:32.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091e25c6a00bb55d2-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.110 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.110 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.111 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.111 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.112 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.112 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.113 DEBUG:boto:Data: 2023-11-02T18:30:32.113 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.113 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.114 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.114 DEBUG:boto:Params: {} 2023-11-02T18:30:32.115 DEBUG:boto:Token: None 2023-11-02T18:30:32.115 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.116 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+rqKmXRYfnX1oXdOJTG768y36tw= 2023-11-02T18:30:32.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+rqKmXRYfnX1oXdOJTG768y36tw=', 'Content-Length': '0'} 2023-11-02T18:30:32.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcc30d26a0203535-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.119 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.120 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.120 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.121 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.121 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.122 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.122 DEBUG:boto:Data: 2023-11-02T18:30:32.123 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.123 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.124 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.124 DEBUG:boto:Params: {} 2023-11-02T18:30:32.125 DEBUG:boto:Token: None 2023-11-02T18:30:32.125 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.126 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+rqKmXRYfnX1oXdOJTG768y36tw= 2023-11-02T18:30:32.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+rqKmXRYfnX1oXdOJTG768y36tw=', 'Content-Length': '0'} 2023-11-02T18:30:32.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6ff1cc4717aadca-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-11-02T18:30:32.131 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.131 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.132 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.132 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.133 DEBUG:boto:Method: GET 2023-11-02T18:30:32.133 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.134 DEBUG:boto:Data: 2023-11-02T18:30:32.134 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.135 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.136 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.136 DEBUG:boto:Params: {} 2023-11-02T18:30:32.136 DEBUG:boto:Token: None 2023-11-02T18:30:32.137 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.138 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QR/cc2XXcHYH6crVX/b4dx//uCc= 2023-11-02T18:30:32.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QR/cc2XXcHYH6crVX/b4dx//uCc=', 'Content-Length': '0'} 2023-11-02T18:30:32.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dff918473573e38-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.182 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.183 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.183 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.184 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.185 DEBUG:boto:Method: GET 2023-11-02T18:30:32.186 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.186 DEBUG:boto:Data: 2023-11-02T18:30:32.187 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.187 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.188 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.189 DEBUG:boto:Params: {} 2023-11-02T18:30:32.189 DEBUG:boto:Token: None 2023-11-02T18:30:32.190 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000026?versionId=null 2023-11-02T18:30:32.190 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QR/cc2XXcHYH6crVX/b4dx//uCc= 2023-11-02T18:30:32.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QR/cc2XXcHYH6crVX/b4dx//uCc=', 'Content-Length': '0'} 2023-11-02T18:30:32.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040fea2e016f4fe03-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.195 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.195 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.196 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000026 2023-11-02T18:30:32.196 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000026 2023-11-02T18:30:32.197 DEBUG:boto:Method: GET 2023-11-02T18:30:32.197 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000026 2023-11-02T18:30:32.198 DEBUG:boto:Data: 2023-11-02T18:30:32.198 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.198 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.199 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.199 DEBUG:boto:Params: {} 2023-11-02T18:30:32.200 DEBUG:boto:Token: None 2023-11-02T18:30:32.200 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9ccb5da763e2c6e-006543eac8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.207 DEBUG:boto:b'ibtljn-58obj000000000000000000261000falseobj000000000000000000262023-11-02T18:30:19.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.208 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.209 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.209 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000026 2023-11-02T18:30:32.210 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000026 2023-11-02T18:30:32.210 DEBUG:boto:Method: GET 2023-11-02T18:30:32.211 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000026 2023-11-02T18:30:32.211 DEBUG:boto:Data: 2023-11-02T18:30:32.211 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.212 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.212 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.213 DEBUG:boto:Params: {} 2023-11-02T18:30:32.213 DEBUG:boto:Token: None 2023-11-02T18:30:32.214 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.214 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a83a78e28cc592b8-006543eac8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.225 DEBUG:boto:b'ibtljn-58obj000000000000000000261000falseobj000000000000000000262023-11-02T18:30:19.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.226 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.227 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.227 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.228 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.228 DEBUG:boto:Data: 2023-11-02T18:30:32.229 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.229 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.230 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.230 DEBUG:boto:Params: {} 2023-11-02T18:30:32.230 DEBUG:boto:Token: None 2023-11-02T18:30:32.231 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.231 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XuS933BpLSIuO3m4DpPjAb6PIUg= 2023-11-02T18:30:32.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XuS933BpLSIuO3m4DpPjAb6PIUg=', 'Content-Length': '0'} 2023-11-02T18:30:32.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c6f7e9126012481-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.235 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.235 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.236 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.237 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.237 DEBUG:boto:Data: 2023-11-02T18:30:32.237 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.238 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.238 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.239 DEBUG:boto:Params: {} 2023-11-02T18:30:32.239 DEBUG:boto:Token: None 2023-11-02T18:30:32.240 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.240 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XuS933BpLSIuO3m4DpPjAb6PIUg= 2023-11-02T18:30:32.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XuS933BpLSIuO3m4DpPjAb6PIUg=', 'Content-Length': '0'} 2023-11-02T18:30:32.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ef964e4f2bf96b-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-11-02T18:30:32.244 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.245 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.245 DEBUG:boto:Method: GET 2023-11-02T18:30:32.246 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.246 DEBUG:boto:Data: 2023-11-02T18:30:32.247 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.247 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.248 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.248 DEBUG:boto:Params: {} 2023-11-02T18:30:32.249 DEBUG:boto:Token: None 2023-11-02T18:30:32.249 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.250 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CupQLzzdW0CAhy47yCbVrYRDkyw= 2023-11-02T18:30:32.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CupQLzzdW0CAhy47yCbVrYRDkyw=', 'Content-Length': '0'} 2023-11-02T18:30:32.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077db28454af6f7c6-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.294 DEBUG:boto:path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.295 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.295 DEBUG:boto:Method: GET 2023-11-02T18:30:32.296 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.296 DEBUG:boto:Data: 2023-11-02T18:30:32.297 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.297 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.298 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.298 DEBUG:boto:Params: {} 2023-11-02T18:30:32.299 DEBUG:boto:Token: None 2023-11-02T18:30:32.299 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000026 2023-11-02T18:30:32.300 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CupQLzzdW0CAhy47yCbVrYRDkyw= 2023-11-02T18:30:32.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CupQLzzdW0CAhy47yCbVrYRDkyw=', 'Content-Length': '0'} 2023-11-02T18:30:32.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7a8336a68f1c9fb-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.303 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.304 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.304 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000026 2023-11-02T18:30:32.305 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000026 2023-11-02T18:30:32.305 DEBUG:boto:Method: GET 2023-11-02T18:30:32.305 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000026 2023-11-02T18:30:32.306 DEBUG:boto:Data: 2023-11-02T18:30:32.306 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.307 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.307 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.308 DEBUG:boto:Params: {} 2023-11-02T18:30:32.308 DEBUG:boto:Token: None 2023-11-02T18:30:32.309 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.309 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbf46b312784c2ea-006543eac8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.314 DEBUG:boto:b'ibtljn-58obj000000000000000000261000falseobj000000000000000000262023-11-02T18:30:19.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.315 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.316 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.316 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000026 2023-11-02T18:30:32.316 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000026 2023-11-02T18:30:32.317 DEBUG:boto:Method: GET 2023-11-02T18:30:32.317 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000026 2023-11-02T18:30:32.318 DEBUG:boto:Data: 2023-11-02T18:30:32.318 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.319 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.319 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.319 DEBUG:boto:Params: {} 2023-11-02T18:30:32.320 DEBUG:boto:Token: None 2023-11-02T18:30:32.320 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.321 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd9fd6819afc5dde-006543eac8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.374 DEBUG:boto:b'ibtljn-58obj000000000000000000261000falseobj000000000000000000262023-11-02T18:30:19.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-11-02T18:30:32.376 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.376 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.377 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.377 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.378 DEBUG:boto:Method: GET 2023-11-02T18:30:32.378 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.379 DEBUG:boto:Data: 2023-11-02T18:30:32.379 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.380 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.380 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.381 DEBUG:boto:Params: {} 2023-11-02T18:30:32.381 DEBUG:boto:Token: None 2023-11-02T18:30:32.382 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.382 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lPkjSEZAzi8G64wm42Ll9vpSCQE= 2023-11-02T18:30:32.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lPkjSEZAzi8G64wm42Ll9vpSCQE=', 'Content-Length': '0'} 2023-11-02T18:30:32.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4d1bae1974536f0-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.387 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.387 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.388 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.388 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.389 DEBUG:boto:Method: GET 2023-11-02T18:30:32.389 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.390 DEBUG:boto:Data: 2023-11-02T18:30:32.390 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.391 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.391 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.392 DEBUG:boto:Params: {} 2023-11-02T18:30:32.392 DEBUG:boto:Token: None 2023-11-02T18:30:32.393 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.393 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lPkjSEZAzi8G64wm42Ll9vpSCQE= 2023-11-02T18:30:32.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lPkjSEZAzi8G64wm42Ll9vpSCQE=', 'Content-Length': '0'} 2023-11-02T18:30:32.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca519f4e4020568f-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.398 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.398 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.399 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.399 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.399 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.400 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.400 DEBUG:boto:Data: 2023-11-02T18:30:32.401 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.401 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.402 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.402 DEBUG:boto:Params: {} 2023-11-02T18:30:32.403 DEBUG:boto:Token: None 2023-11-02T18:30:32.403 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.404 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sHVL1EvU4Nu6FwYgWEt+ICfdMyc= 2023-11-02T18:30:32.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sHVL1EvU4Nu6FwYgWEt+ICfdMyc=', 'Content-Length': '0'} 2023-11-02T18:30:32.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d235f520600e33d-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.408 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.408 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.408 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.409 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.409 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.410 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.410 DEBUG:boto:Data: 2023-11-02T18:30:32.411 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.411 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.412 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.412 DEBUG:boto:Params: {} 2023-11-02T18:30:32.412 DEBUG:boto:Token: None 2023-11-02T18:30:32.413 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.413 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sHVL1EvU4Nu6FwYgWEt+ICfdMyc= 2023-11-02T18:30:32.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sHVL1EvU4Nu6FwYgWEt+ICfdMyc=', 'Content-Length': '0'} 2023-11-02T18:30:32.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7d89c825cb78386-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-11-02T18:30:32.418 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.418 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.419 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.419 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.420 DEBUG:boto:Method: GET 2023-11-02T18:30:32.420 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.421 DEBUG:boto:Data: 2023-11-02T18:30:32.422 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.423 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.423 DEBUG:boto:Params: {} 2023-11-02T18:30:32.424 DEBUG:boto:Token: None 2023-11-02T18:30:32.424 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.425 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lPkjSEZAzi8G64wm42Ll9vpSCQE= 2023-11-02T18:30:32.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lPkjSEZAzi8G64wm42Ll9vpSCQE=', 'Content-Length': '0'} 2023-11-02T18:30:32.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef825dbff1ac214f-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.474 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.475 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.475 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.476 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.476 DEBUG:boto:Method: GET 2023-11-02T18:30:32.477 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.477 DEBUG:boto:Data: 2023-11-02T18:30:32.478 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.478 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.479 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.479 DEBUG:boto:Params: {} 2023-11-02T18:30:32.479 DEBUG:boto:Token: None 2023-11-02T18:30:32.480 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000027?versionId=null 2023-11-02T18:30:32.480 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lPkjSEZAzi8G64wm42Ll9vpSCQE= 2023-11-02T18:30:32.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lPkjSEZAzi8G64wm42Ll9vpSCQE=', 'Content-Length': '0'} 2023-11-02T18:30:32.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0f7d4fe79a48fc7-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.485 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.485 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.486 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000027 2023-11-02T18:30:32.486 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000027 2023-11-02T18:30:32.486 DEBUG:boto:Method: GET 2023-11-02T18:30:32.487 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000027 2023-11-02T18:30:32.487 DEBUG:boto:Data: 2023-11-02T18:30:32.488 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.488 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.489 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.489 DEBUG:boto:Params: {} 2023-11-02T18:30:32.490 DEBUG:boto:Token: None 2023-11-02T18:30:32.490 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.491 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027e3358f4c16548b-006543eac8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.496 DEBUG:boto:b'ibtljn-58obj000000000000000000271000falseobj000000000000000000272023-11-02T18:30:19.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.497 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.498 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.498 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000027 2023-11-02T18:30:32.499 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000027 2023-11-02T18:30:32.499 DEBUG:boto:Method: GET 2023-11-02T18:30:32.499 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000027 2023-11-02T18:30:32.500 DEBUG:boto:Data: 2023-11-02T18:30:32.500 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.501 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.501 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.501 DEBUG:boto:Params: {} 2023-11-02T18:30:32.502 DEBUG:boto:Token: None 2023-11-02T18:30:32.502 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.503 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006196fde6f9940478-006543eac8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.513 DEBUG:boto:b'ibtljn-58obj000000000000000000271000falseobj000000000000000000272023-11-02T18:30:19.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.514 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.514 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.515 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.515 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.515 DEBUG:boto:Data: 2023-11-02T18:30:32.516 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.516 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.517 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.517 DEBUG:boto:Params: {} 2023-11-02T18:30:32.518 DEBUG:boto:Token: None 2023-11-02T18:30:32.518 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.519 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I13aAKn2taU6zCIGqXE/yPVuTKA= 2023-11-02T18:30:32.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I13aAKn2taU6zCIGqXE/yPVuTKA=', 'Content-Length': '0'} 2023-11-02T18:30:32.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012ff23270cbd7368-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.523 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.523 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.523 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.524 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.524 DEBUG:boto:Data: 2023-11-02T18:30:32.525 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.525 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.525 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.526 DEBUG:boto:Params: {} 2023-11-02T18:30:32.526 DEBUG:boto:Token: None 2023-11-02T18:30:32.527 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.527 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I13aAKn2taU6zCIGqXE/yPVuTKA= 2023-11-02T18:30:32.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I13aAKn2taU6zCIGqXE/yPVuTKA=', 'Content-Length': '0'} 2023-11-02T18:30:32.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c47f87ae0fdcad-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-11-02T18:30:32.531 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.532 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.532 DEBUG:boto:Method: GET 2023-11-02T18:30:32.533 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.533 DEBUG:boto:Data: 2023-11-02T18:30:32.533 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.534 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.534 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.535 DEBUG:boto:Params: {} 2023-11-02T18:30:32.535 DEBUG:boto:Token: None 2023-11-02T18:30:32.536 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.536 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eUmZ1cVtjI0zjkfHgxlWPAM19Kk= 2023-11-02T18:30:32.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eUmZ1cVtjI0zjkfHgxlWPAM19Kk=', 'Content-Length': '0'} 2023-11-02T18:30:32.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bb21f56678819a0-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.540 DEBUG:boto:path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.540 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.541 DEBUG:boto:Method: GET 2023-11-02T18:30:32.541 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.542 DEBUG:boto:Data: 2023-11-02T18:30:32.542 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.543 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.544 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.544 DEBUG:boto:Params: {} 2023-11-02T18:30:32.545 DEBUG:boto:Token: None 2023-11-02T18:30:32.545 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000027 2023-11-02T18:30:32.546 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eUmZ1cVtjI0zjkfHgxlWPAM19Kk= 2023-11-02T18:30:32.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eUmZ1cVtjI0zjkfHgxlWPAM19Kk=', 'Content-Length': '0'} 2023-11-02T18:30:32.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000323f4d76eaf1003c-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.594 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.595 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.596 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000027 2023-11-02T18:30:32.596 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000027 2023-11-02T18:30:32.596 DEBUG:boto:Method: GET 2023-11-02T18:30:32.597 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000027 2023-11-02T18:30:32.597 DEBUG:boto:Data: 2023-11-02T18:30:32.598 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.598 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.599 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.599 DEBUG:boto:Params: {} 2023-11-02T18:30:32.600 DEBUG:boto:Token: None 2023-11-02T18:30:32.601 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a2618894d923330-006543eac8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.608 DEBUG:boto:b'ibtljn-58obj000000000000000000271000falseobj000000000000000000272023-11-02T18:30:19.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.609 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.610 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.610 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000027 2023-11-02T18:30:32.610 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000027 2023-11-02T18:30:32.611 DEBUG:boto:Method: GET 2023-11-02T18:30:32.611 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000027 2023-11-02T18:30:32.612 DEBUG:boto:Data: 2023-11-02T18:30:32.612 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.613 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.613 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.613 DEBUG:boto:Params: {} 2023-11-02T18:30:32.614 DEBUG:boto:Token: None 2023-11-02T18:30:32.614 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.615 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000757245626c8b36af-006543eac8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.627 DEBUG:boto:b'ibtljn-58obj000000000000000000271000falseobj000000000000000000272023-11-02T18:30:19.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-11-02T18:30:32.628 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.628 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.628 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.629 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.629 DEBUG:boto:Method: GET 2023-11-02T18:30:32.630 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.630 DEBUG:boto:Data: 2023-11-02T18:30:32.631 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.631 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.631 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.632 DEBUG:boto:Params: {} 2023-11-02T18:30:32.632 DEBUG:boto:Token: None 2023-11-02T18:30:32.633 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.633 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XWAu9nG7DxvZ6Rrt+Id+yTU9WVQ= 2023-11-02T18:30:32.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XWAu9nG7DxvZ6Rrt+Id+yTU9WVQ=', 'Content-Length': '0'} 2023-11-02T18:30:32.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc3b100b624bf836-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.678 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.679 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.679 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.680 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.680 DEBUG:boto:Method: GET 2023-11-02T18:30:32.681 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.681 DEBUG:boto:Data: 2023-11-02T18:30:32.681 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.682 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.682 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.683 DEBUG:boto:Params: {} 2023-11-02T18:30:32.683 DEBUG:boto:Token: None 2023-11-02T18:30:32.684 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.684 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XWAu9nG7DxvZ6Rrt+Id+yTU9WVQ= 2023-11-02T18:30:32.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XWAu9nG7DxvZ6Rrt+Id+yTU9WVQ=', 'Content-Length': '0'} 2023-11-02T18:30:32.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076b534448a21eb4e-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.688 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.689 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.689 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.689 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.690 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.690 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.691 DEBUG:boto:Data: 2023-11-02T18:30:32.691 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.691 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.692 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.692 DEBUG:boto:Params: {} 2023-11-02T18:30:32.693 DEBUG:boto:Token: None 2023-11-02T18:30:32.693 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.693 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VZZ+FY40lGknjjKc6h7XNc4PJus= 2023-11-02T18:30:32.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VZZ+FY40lGknjjKc6h7XNc4PJus=', 'Content-Length': '0'} 2023-11-02T18:30:32.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b84111f70f036095-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.697 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.698 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.698 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.698 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.699 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.699 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.700 DEBUG:boto:Data: 2023-11-02T18:30:32.700 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.701 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.701 DEBUG:boto:Params: {} 2023-11-02T18:30:32.702 DEBUG:boto:Token: None 2023-11-02T18:30:32.702 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.703 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VZZ+FY40lGknjjKc6h7XNc4PJus= 2023-11-02T18:30:32.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VZZ+FY40lGknjjKc6h7XNc4PJus=', 'Content-Length': '0'} 2023-11-02T18:30:32.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f97739f61f23b6c7-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-11-02T18:30:32.707 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.707 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.708 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.708 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.709 DEBUG:boto:Method: GET 2023-11-02T18:30:32.709 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.710 DEBUG:boto:Data: 2023-11-02T18:30:32.710 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.711 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.711 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.711 DEBUG:boto:Params: {} 2023-11-02T18:30:32.712 DEBUG:boto:Token: None 2023-11-02T18:30:32.712 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.713 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XWAu9nG7DxvZ6Rrt+Id+yTU9WVQ= 2023-11-02T18:30:32.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XWAu9nG7DxvZ6Rrt+Id+yTU9WVQ=', 'Content-Length': '0'} 2023-11-02T18:30:32.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cef1d28590ad898-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.717 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.717 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.718 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.718 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.719 DEBUG:boto:Method: GET 2023-11-02T18:30:32.719 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.720 DEBUG:boto:Data: 2023-11-02T18:30:32.720 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.720 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.721 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.721 DEBUG:boto:Params: {} 2023-11-02T18:30:32.722 DEBUG:boto:Token: None 2023-11-02T18:30:32.722 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000028?versionId=null 2023-11-02T18:30:32.723 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XWAu9nG7DxvZ6Rrt+Id+yTU9WVQ= 2023-11-02T18:30:32.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XWAu9nG7DxvZ6Rrt+Id+yTU9WVQ=', 'Content-Length': '0'} 2023-11-02T18:30:32.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029180468ab336d27-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.727 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.727 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.728 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000028 2023-11-02T18:30:32.728 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000028 2023-11-02T18:30:32.728 DEBUG:boto:Method: GET 2023-11-02T18:30:32.729 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000028 2023-11-02T18:30:32.729 DEBUG:boto:Data: 2023-11-02T18:30:32.730 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.730 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.731 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.731 DEBUG:boto:Params: {} 2023-11-02T18:30:32.732 DEBUG:boto:Token: None 2023-11-02T18:30:32.732 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.732 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000383ad96b5e45988c-006543eac8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.738 DEBUG:boto:b'ibtljn-58obj000000000000000000281000falseobj000000000000000000282023-11-02T18:30:12.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.739 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.739 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.740 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000028 2023-11-02T18:30:32.740 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000028 2023-11-02T18:30:32.741 DEBUG:boto:Method: GET 2023-11-02T18:30:32.741 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000028 2023-11-02T18:30:32.742 DEBUG:boto:Data: 2023-11-02T18:30:32.742 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.742 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.743 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.743 DEBUG:boto:Params: {} 2023-11-02T18:30:32.744 DEBUG:boto:Token: None 2023-11-02T18:30:32.744 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.745 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007247a70633301732-006543eac8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.798 DEBUG:boto:b'ibtljn-58obj000000000000000000281000falseobj000000000000000000282023-11-02T18:30:12.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.799 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.800 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.800 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.801 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.801 DEBUG:boto:Data: 2023-11-02T18:30:32.801 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.802 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.802 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.803 DEBUG:boto:Params: {} 2023-11-02T18:30:32.803 DEBUG:boto:Token: None 2023-11-02T18:30:32.804 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.804 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pFmPgIhAQyn+yf/OGcc057bz7kc= 2023-11-02T18:30:32.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pFmPgIhAQyn+yf/OGcc057bz7kc=', 'Content-Length': '0'} 2023-11-02T18:30:32.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071c9fcda40c8a39f-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.808 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.809 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.809 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.809 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.810 DEBUG:boto:Data: 2023-11-02T18:30:32.810 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.811 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.811 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.811 DEBUG:boto:Params: {} 2023-11-02T18:30:32.812 DEBUG:boto:Token: None 2023-11-02T18:30:32.812 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.813 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pFmPgIhAQyn+yf/OGcc057bz7kc= 2023-11-02T18:30:32.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pFmPgIhAQyn+yf/OGcc057bz7kc=', 'Content-Length': '0'} 2023-11-02T18:30:32.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086fa894a38aee362-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-11-02T18:30:32.817 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.818 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.818 DEBUG:boto:Method: GET 2023-11-02T18:30:32.818 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.819 DEBUG:boto:Data: 2023-11-02T18:30:32.819 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.820 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.820 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.821 DEBUG:boto:Params: {} 2023-11-02T18:30:32.821 DEBUG:boto:Token: None 2023-11-02T18:30:32.822 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.822 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cVTRt2D+j650WYN+XmC2syflVzs= 2023-11-02T18:30:32.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cVTRt2D+j650WYN+XmC2syflVzs=', 'Content-Length': '0'} 2023-11-02T18:30:32.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055948ebd41fd99a8-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.866 DEBUG:boto:path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.867 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.868 DEBUG:boto:Method: GET 2023-11-02T18:30:32.868 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.868 DEBUG:boto:Data: 2023-11-02T18:30:32.869 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.869 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.870 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.870 DEBUG:boto:Params: {} 2023-11-02T18:30:32.871 DEBUG:boto:Token: None 2023-11-02T18:30:32.871 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000028 2023-11-02T18:30:32.872 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cVTRt2D+j650WYN+XmC2syflVzs= 2023-11-02T18:30:32.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cVTRt2D+j650WYN+XmC2syflVzs=', 'Content-Length': '0'} 2023-11-02T18:30:32.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5149a9bc316008f-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.876 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.877 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.877 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000028 2023-11-02T18:30:32.877 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000028 2023-11-02T18:30:32.878 DEBUG:boto:Method: GET 2023-11-02T18:30:32.878 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000028 2023-11-02T18:30:32.879 DEBUG:boto:Data: 2023-11-02T18:30:32.879 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.880 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.880 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.880 DEBUG:boto:Params: {} 2023-11-02T18:30:32.881 DEBUG:boto:Token: None 2023-11-02T18:30:32.882 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.882 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b581731335510581-006543eac8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.888 DEBUG:boto:b'ibtljn-58obj000000000000000000281000falseobj000000000000000000282023-11-02T18:30:12.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.889 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:32.890 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:32.890 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000028 2023-11-02T18:30:32.890 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000028 2023-11-02T18:30:32.891 DEBUG:boto:Method: GET 2023-11-02T18:30:32.892 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000028 2023-11-02T18:30:32.893 DEBUG:boto:Data: 2023-11-02T18:30:32.893 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.894 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.894 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.895 DEBUG:boto:Params: {} 2023-11-02T18:30:32.895 DEBUG:boto:Token: None 2023-11-02T18:30:32.896 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/ 2023-11-02T18:30:32.896 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s= 2023-11-02T18:30:32.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5uzmULymbK9QsU3S4rT5iDT4W8s=', 'Content-Length': '0'} 2023-11-02T18:30:32.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b950b349dd157caa-006543eac8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.908 DEBUG:boto:b'ibtljn-58obj000000000000000000281000falseobj000000000000000000282023-11-02T18:30:12.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:32.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-11-02T18:30:32.909 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:32.910 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:32.911 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.911 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.912 DEBUG:boto:Method: GET 2023-11-02T18:30:32.912 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.913 DEBUG:boto:Data: 2023-11-02T18:30:32.913 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.913 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.914 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.914 DEBUG:boto:Params: {} 2023-11-02T18:30:32.915 DEBUG:boto:Token: None 2023-11-02T18:30:32.915 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.916 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZFdXdlNkjfyfb2oMX26yUCGJIbo= 2023-11-02T18:30:32.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZFdXdlNkjfyfb2oMX26yUCGJIbo=', 'Content-Length': '0'} 2023-11-02T18:30:32.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004928a9fef4bff259-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.921 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:32.921 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:32.922 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.922 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.923 DEBUG:boto:Method: GET 2023-11-02T18:30:32.923 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.923 DEBUG:boto:Data: 2023-11-02T18:30:32.924 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.924 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.925 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.925 DEBUG:boto:Params: {} 2023-11-02T18:30:32.926 DEBUG:boto:Token: None 2023-11-02T18:30:32.926 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.927 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZFdXdlNkjfyfb2oMX26yUCGJIbo= 2023-11-02T18:30:32.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZFdXdlNkjfyfb2oMX26yUCGJIbo=', 'Content-Length': '0'} 2023-11-02T18:30:32.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091812f0baa6cdc9d-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.970 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:32.971 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:32.971 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.972 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.972 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.973 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.973 DEBUG:boto:Data: 2023-11-02T18:30:32.973 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.974 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.974 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.975 DEBUG:boto:Params: {} 2023-11-02T18:30:32.975 DEBUG:boto:Token: None 2023-11-02T18:30:32.975 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.976 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d4JIpelUed6Vpfo/01l0qahIKAE= 2023-11-02T18:30:32.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d4JIpelUed6Vpfo/01l0qahIKAE=', 'Content-Length': '0'} 2023-11-02T18:30:32.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c212e3cd440b17d0-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.980 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:32.980 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:32.981 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.981 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.982 DEBUG:boto:Method: HEAD 2023-11-02T18:30:32.982 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.982 DEBUG:boto:Data: 2023-11-02T18:30:32.984 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.984 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:32.984 DEBUG:boto:Port: 8000 2023-11-02T18:30:32.985 DEBUG:boto:Params: {} 2023-11-02T18:30:32.985 DEBUG:boto:Token: None 2023-11-02T18:30:32.986 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.987 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d4JIpelUed6Vpfo/01l0qahIKAE= 2023-11-02T18:30:32.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d4JIpelUed6Vpfo/01l0qahIKAE=', 'Content-Length': '0'} 2023-11-02T18:30:32.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007132bb372e78ff40-006543eac8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:32.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-11-02T18:30:32.992 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:32.992 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:32.993 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.993 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.994 DEBUG:boto:Method: GET 2023-11-02T18:30:32.995 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.995 DEBUG:boto:Data: 2023-11-02T18:30:32.995 DEBUG:boto:Headers: {} 2023-11-02T18:30:32.996 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:32.996 DEBUG:boto:Port: 8001 2023-11-02T18:30:32.997 DEBUG:boto:Params: {} 2023-11-02T18:30:32.997 DEBUG:boto:Token: None 2023-11-02T18:30:32.998 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:32 GMT /ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:32.998 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZFdXdlNkjfyfb2oMX26yUCGJIbo= 2023-11-02T18:30:32.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZFdXdlNkjfyfb2oMX26yUCGJIbo=', 'Content-Length': '0'} 2023-11-02T18:30:33.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ca993ab0f96d7b8-006543eac8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.042 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.043 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.043 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:33.043 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:33.044 DEBUG:boto:Method: GET 2023-11-02T18:30:33.044 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:33.045 DEBUG:boto:Data: 2023-11-02T18:30:33.045 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.046 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.047 DEBUG:boto:Params: {} 2023-11-02T18:30:33.047 DEBUG:boto:Token: None 2023-11-02T18:30:33.048 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000029?versionId=null 2023-11-02T18:30:33.048 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QDas9ak7gpesVcIg8RsjQBN4cF8= 2023-11-02T18:30:33.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QDas9ak7gpesVcIg8RsjQBN4cF8=', 'Content-Length': '0'} 2023-11-02T18:30:33.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff6fb2bf7ec58200-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.052 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.053 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.053 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000029 2023-11-02T18:30:33.053 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000029 2023-11-02T18:30:33.054 DEBUG:boto:Method: GET 2023-11-02T18:30:33.054 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000029 2023-11-02T18:30:33.055 DEBUG:boto:Data: 2023-11-02T18:30:33.055 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.056 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.056 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.056 DEBUG:boto:Params: {} 2023-11-02T18:30:33.057 DEBUG:boto:Token: None 2023-11-02T18:30:33.057 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.058 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eab95028ad9aaeb-006543eac9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.064 DEBUG:boto:b'ibtljn-58obj000000000000000000291000falseobj000000000000000000292023-11-02T18:30:17.892Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.064 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.065 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.065 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000029 2023-11-02T18:30:33.066 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000029 2023-11-02T18:30:33.066 DEBUG:boto:Method: GET 2023-11-02T18:30:33.066 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000029 2023-11-02T18:30:33.067 DEBUG:boto:Data: 2023-11-02T18:30:33.067 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.068 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.068 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.069 DEBUG:boto:Params: {} 2023-11-02T18:30:33.069 DEBUG:boto:Token: None 2023-11-02T18:30:33.070 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.070 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067bdaf090d67e641-006543eac9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.080 DEBUG:boto:b'ibtljn-58obj000000000000000000291000falseobj000000000000000000292023-11-02T18:30:17.892Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.081 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.081 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.082 DEBUG:boto:Method: HEAD 2023-11-02T18:30:33.082 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.082 DEBUG:boto:Data: 2023-11-02T18:30:33.083 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.083 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.084 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.084 DEBUG:boto:Params: {} 2023-11-02T18:30:33.084 DEBUG:boto:Token: None 2023-11-02T18:30:33.085 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.085 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SuFuhpGNxsQuAJimzdXGI9BYUPk= 2023-11-02T18:30:33.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SuFuhpGNxsQuAJimzdXGI9BYUPk=', 'Content-Length': '0'} 2023-11-02T18:30:33.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003eebe334e57596c-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.089 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.089 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.090 DEBUG:boto:Method: HEAD 2023-11-02T18:30:33.090 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.091 DEBUG:boto:Data: 2023-11-02T18:30:33.091 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.091 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.092 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.092 DEBUG:boto:Params: {} 2023-11-02T18:30:33.093 DEBUG:boto:Token: None 2023-11-02T18:30:33.093 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SuFuhpGNxsQuAJimzdXGI9BYUPk= 2023-11-02T18:30:33.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SuFuhpGNxsQuAJimzdXGI9BYUPk=', 'Content-Length': '0'} 2023-11-02T18:30:33.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000684452b59a147c28-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-11-02T18:30:33.098 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.098 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.099 DEBUG:boto:Method: GET 2023-11-02T18:30:33.099 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.100 DEBUG:boto:Data: 2023-11-02T18:30:33.100 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.101 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.101 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.101 DEBUG:boto:Params: {} 2023-11-02T18:30:33.102 DEBUG:boto:Token: None 2023-11-02T18:30:33.102 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.103 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RXelQOEdfTZgUKq0Nola6dGuODY= 2023-11-02T18:30:33.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RXelQOEdfTZgUKq0Nola6dGuODY=', 'Content-Length': '0'} 2023-11-02T18:30:33.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077e0b00921098274-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.107 DEBUG:boto:path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.107 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.107 DEBUG:boto:Method: GET 2023-11-02T18:30:33.108 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.108 DEBUG:boto:Data: 2023-11-02T18:30:33.109 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.109 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.110 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.110 DEBUG:boto:Params: {} 2023-11-02T18:30:33.110 DEBUG:boto:Token: None 2023-11-02T18:30:33.111 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000029 2023-11-02T18:30:33.112 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RXelQOEdfTZgUKq0Nola6dGuODY= 2023-11-02T18:30:33.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RXelQOEdfTZgUKq0Nola6dGuODY=', 'Content-Length': '0'} 2023-11-02T18:30:33.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ad110e19c139905-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.158 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.159 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.159 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000029 2023-11-02T18:30:33.160 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000029 2023-11-02T18:30:33.160 DEBUG:boto:Method: GET 2023-11-02T18:30:33.161 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000029 2023-11-02T18:30:33.161 DEBUG:boto:Data: 2023-11-02T18:30:33.162 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.162 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.163 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.163 DEBUG:boto:Params: {} 2023-11-02T18:30:33.163 DEBUG:boto:Token: None 2023-11-02T18:30:33.164 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.165 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078b91b0b9e5ab98b-006543eac9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.172 DEBUG:boto:b'ibtljn-58obj000000000000000000291000falseobj000000000000000000292023-11-02T18:30:17.892Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.172 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.173 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.173 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000029 2023-11-02T18:30:33.174 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000029 2023-11-02T18:30:33.174 DEBUG:boto:Method: GET 2023-11-02T18:30:33.175 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000029 2023-11-02T18:30:33.175 DEBUG:boto:Data: 2023-11-02T18:30:33.175 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.176 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.176 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.177 DEBUG:boto:Params: {} 2023-11-02T18:30:33.177 DEBUG:boto:Token: None 2023-11-02T18:30:33.178 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.178 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ee893f2b2f0e111-006543eac9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.190 DEBUG:boto:b'ibtljn-58obj000000000000000000291000falseobj000000000000000000292023-11-02T18:30:17.892Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-11-02T18:30:33.191 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.191 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.192 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.192 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.193 DEBUG:boto:Method: GET 2023-11-02T18:30:33.193 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.194 DEBUG:boto:Data: 2023-11-02T18:30:33.194 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.194 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.195 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.195 DEBUG:boto:Params: {} 2023-11-02T18:30:33.196 DEBUG:boto:Token: None 2023-11-02T18:30:33.196 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.197 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UoXLyyj/kI4GUEE7cVoxF1oimEw= 2023-11-02T18:30:33.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UoXLyyj/kI4GUEE7cVoxF1oimEw=', 'Content-Length': '0'} 2023-11-02T18:30:33.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002df9265a48a6181d-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.242 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.243 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.243 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.244 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.244 DEBUG:boto:Method: GET 2023-11-02T18:30:33.245 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.245 DEBUG:boto:Data: 2023-11-02T18:30:33.246 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.247 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.247 DEBUG:boto:Params: {} 2023-11-02T18:30:33.247 DEBUG:boto:Token: None 2023-11-02T18:30:33.248 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.249 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UoXLyyj/kI4GUEE7cVoxF1oimEw= 2023-11-02T18:30:33.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UoXLyyj/kI4GUEE7cVoxF1oimEw=', 'Content-Length': '0'} 2023-11-02T18:30:33.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b9c31b98898fc26-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.253 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.253 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.254 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.254 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.255 DEBUG:boto:Method: HEAD 2023-11-02T18:30:33.255 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.255 DEBUG:boto:Data: 2023-11-02T18:30:33.256 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.256 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.257 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.257 DEBUG:boto:Params: {} 2023-11-02T18:30:33.258 DEBUG:boto:Token: None 2023-11-02T18:30:33.258 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.259 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xKTdGI6dkohDFdplpD1neM7BdQw= 2023-11-02T18:30:33.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xKTdGI6dkohDFdplpD1neM7BdQw=', 'Content-Length': '0'} 2023-11-02T18:30:33.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea992bd831cba8f-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.263 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.263 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.264 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.264 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.264 DEBUG:boto:Method: HEAD 2023-11-02T18:30:33.265 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.265 DEBUG:boto:Data: 2023-11-02T18:30:33.266 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.266 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.266 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.267 DEBUG:boto:Params: {} 2023-11-02T18:30:33.267 DEBUG:boto:Token: None 2023-11-02T18:30:33.268 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.268 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xKTdGI6dkohDFdplpD1neM7BdQw= 2023-11-02T18:30:33.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xKTdGI6dkohDFdplpD1neM7BdQw=', 'Content-Length': '0'} 2023-11-02T18:30:33.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f515e7bd8deec2ce-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-11-02T18:30:33.272 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.273 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.273 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.274 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.274 DEBUG:boto:Method: GET 2023-11-02T18:30:33.274 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.275 DEBUG:boto:Data: 2023-11-02T18:30:33.275 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.276 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.276 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.277 DEBUG:boto:Params: {} 2023-11-02T18:30:33.277 DEBUG:boto:Token: None 2023-11-02T18:30:33.278 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.278 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UoXLyyj/kI4GUEE7cVoxF1oimEw= 2023-11-02T18:30:33.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UoXLyyj/kI4GUEE7cVoxF1oimEw=', 'Content-Length': '0'} 2023-11-02T18:30:33.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000946f66db69d871bf-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.282 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.283 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.283 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.283 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.284 DEBUG:boto:Method: GET 2023-11-02T18:30:33.284 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.285 DEBUG:boto:Data: 2023-11-02T18:30:33.285 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.286 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.286 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.287 DEBUG:boto:Params: {} 2023-11-02T18:30:33.287 DEBUG:boto:Token: None 2023-11-02T18:30:33.288 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000030?versionId=null 2023-11-02T18:30:33.288 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UoXLyyj/kI4GUEE7cVoxF1oimEw= 2023-11-02T18:30:33.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UoXLyyj/kI4GUEE7cVoxF1oimEw=', 'Content-Length': '0'} 2023-11-02T18:30:33.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d6742edf719d4aa-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.334 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.335 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.335 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000030 2023-11-02T18:30:33.336 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000030 2023-11-02T18:30:33.336 DEBUG:boto:Method: GET 2023-11-02T18:30:33.336 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000030 2023-11-02T18:30:33.337 DEBUG:boto:Data: 2023-11-02T18:30:33.337 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.338 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.338 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.338 DEBUG:boto:Params: {} 2023-11-02T18:30:33.339 DEBUG:boto:Token: None 2023-11-02T18:30:33.339 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.340 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080bf6a05aa97c5b1-006543eac9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.346 DEBUG:boto:b'ibtljn-58obj000000000000000000301000falseobj000000000000000000302023-11-02T18:30:19.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.346 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.347 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.347 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000030 2023-11-02T18:30:33.348 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000030 2023-11-02T18:30:33.348 DEBUG:boto:Method: GET 2023-11-02T18:30:33.348 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000030 2023-11-02T18:30:33.349 DEBUG:boto:Data: 2023-11-02T18:30:33.349 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.350 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.350 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.350 DEBUG:boto:Params: {} 2023-11-02T18:30:33.351 DEBUG:boto:Token: None 2023-11-02T18:30:33.351 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.352 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024af9415738877a4-006543eac9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.363 DEBUG:boto:b'ibtljn-58obj000000000000000000301000falseobj000000000000000000302023-11-02T18:30:19.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.364 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.364 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.365 DEBUG:boto:Method: HEAD 2023-11-02T18:30:33.365 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.365 DEBUG:boto:Data: 2023-11-02T18:30:33.366 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.367 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.367 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.368 DEBUG:boto:Params: {} 2023-11-02T18:30:33.368 DEBUG:boto:Token: None 2023-11-02T18:30:33.369 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.369 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kidpQ6tSZThrR7lFQMNrTzh7Ank= 2023-11-02T18:30:33.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kidpQ6tSZThrR7lFQMNrTzh7Ank=', 'Content-Length': '0'} 2023-11-02T18:30:33.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049c34c1942661c7b-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.376 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.376 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.377 DEBUG:boto:Method: HEAD 2023-11-02T18:30:33.377 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.377 DEBUG:boto:Data: 2023-11-02T18:30:33.378 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.378 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.379 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.379 DEBUG:boto:Params: {} 2023-11-02T18:30:33.380 DEBUG:boto:Token: None 2023-11-02T18:30:33.380 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.381 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kidpQ6tSZThrR7lFQMNrTzh7Ank= 2023-11-02T18:30:33.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kidpQ6tSZThrR7lFQMNrTzh7Ank=', 'Content-Length': '0'} 2023-11-02T18:30:33.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d0df083e81c67ca-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-11-02T18:30:33.385 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.385 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.386 DEBUG:boto:Method: GET 2023-11-02T18:30:33.386 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.387 DEBUG:boto:Data: 2023-11-02T18:30:33.387 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.387 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.388 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.389 DEBUG:boto:Params: {} 2023-11-02T18:30:33.390 DEBUG:boto:Token: None 2023-11-02T18:30:33.390 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.393 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JcdrkZ/luzg+oiAaJSB567BDlb0= 2023-11-02T18:30:33.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JcdrkZ/luzg+oiAaJSB567BDlb0=', 'Content-Length': '0'} 2023-11-02T18:30:33.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009136efd3169cac7c-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.443 DEBUG:boto:path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.444 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.445 DEBUG:boto:Method: GET 2023-11-02T18:30:33.446 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.446 DEBUG:boto:Data: 2023-11-02T18:30:33.447 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.448 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.448 DEBUG:boto:Params: {} 2023-11-02T18:30:33.448 DEBUG:boto:Token: None 2023-11-02T18:30:33.449 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000030 2023-11-02T18:30:33.449 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JcdrkZ/luzg+oiAaJSB567BDlb0= 2023-11-02T18:30:33.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JcdrkZ/luzg+oiAaJSB567BDlb0=', 'Content-Length': '0'} 2023-11-02T18:30:33.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bff8501f8ed2901-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.457 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.458 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.458 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000030 2023-11-02T18:30:33.459 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000030 2023-11-02T18:30:33.459 DEBUG:boto:Method: GET 2023-11-02T18:30:33.459 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000030 2023-11-02T18:30:33.461 DEBUG:boto:Data: 2023-11-02T18:30:33.461 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.463 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.463 DEBUG:boto:Params: {} 2023-11-02T18:30:33.464 DEBUG:boto:Token: None 2023-11-02T18:30:33.464 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b00fdf7504073fc-006543eac9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.471 DEBUG:boto:b'ibtljn-58obj000000000000000000301000falseobj000000000000000000302023-11-02T18:30:19.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.474 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.475 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.475 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000030 2023-11-02T18:30:33.476 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000030 2023-11-02T18:30:33.477 DEBUG:boto:Method: GET 2023-11-02T18:30:33.477 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000030 2023-11-02T18:30:33.478 DEBUG:boto:Data: 2023-11-02T18:30:33.478 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.479 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.479 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.480 DEBUG:boto:Params: {} 2023-11-02T18:30:33.480 DEBUG:boto:Token: None 2023-11-02T18:30:33.481 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.481 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dd6c356a60a6cd1-006543eac9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.542 DEBUG:boto:b'ibtljn-58obj000000000000000000301000falseobj000000000000000000302023-11-02T18:30:19.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-11-02T18:30:33.543 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.544 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.544 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.545 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.545 DEBUG:boto:Method: GET 2023-11-02T18:30:33.545 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.546 DEBUG:boto:Data: 2023-11-02T18:30:33.546 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.547 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.547 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.548 DEBUG:boto:Params: {} 2023-11-02T18:30:33.548 DEBUG:boto:Token: None 2023-11-02T18:30:33.549 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.549 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D4fl755NnHw+QnD57y4IH4vTvJk= 2023-11-02T18:30:33.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D4fl755NnHw+QnD57y4IH4vTvJk=', 'Content-Length': '0'} 2023-11-02T18:30:33.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb29b11defb3606f-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.554 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.555 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.555 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.555 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.556 DEBUG:boto:Method: GET 2023-11-02T18:30:33.556 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.557 DEBUG:boto:Data: 2023-11-02T18:30:33.557 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.558 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.558 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.559 DEBUG:boto:Params: {} 2023-11-02T18:30:33.559 DEBUG:boto:Token: None 2023-11-02T18:30:33.560 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.560 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D4fl755NnHw+QnD57y4IH4vTvJk= 2023-11-02T18:30:33.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D4fl755NnHw+QnD57y4IH4vTvJk=', 'Content-Length': '0'} 2023-11-02T18:30:33.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ffba308e32b0da5-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.564 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.565 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.565 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.566 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.566 DEBUG:boto:Method: HEAD 2023-11-02T18:30:33.567 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.567 DEBUG:boto:Data: 2023-11-02T18:30:33.567 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.568 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.568 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.569 DEBUG:boto:Params: {} 2023-11-02T18:30:33.569 DEBUG:boto:Token: None 2023-11-02T18:30:33.570 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.570 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TWBOjmd7ySc+MWl5T4ZOd58XhzY= 2023-11-02T18:30:33.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TWBOjmd7ySc+MWl5T4ZOd58XhzY=', 'Content-Length': '0'} 2023-11-02T18:30:33.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ef2020ca9035b47-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.578 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.578 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.579 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.579 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.579 DEBUG:boto:Method: HEAD 2023-11-02T18:30:33.581 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.581 DEBUG:boto:Data: 2023-11-02T18:30:33.582 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.582 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.583 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.583 DEBUG:boto:Params: {} 2023-11-02T18:30:33.583 DEBUG:boto:Token: None 2023-11-02T18:30:33.584 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.584 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TWBOjmd7ySc+MWl5T4ZOd58XhzY= 2023-11-02T18:30:33.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TWBOjmd7ySc+MWl5T4ZOd58XhzY=', 'Content-Length': '0'} 2023-11-02T18:30:33.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abd4c4ce8c394ca2-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-11-02T18:30:33.589 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.589 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.590 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.590 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.591 DEBUG:boto:Method: GET 2023-11-02T18:30:33.591 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.592 DEBUG:boto:Data: 2023-11-02T18:30:33.592 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.593 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.593 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.594 DEBUG:boto:Params: {} 2023-11-02T18:30:33.594 DEBUG:boto:Token: None 2023-11-02T18:30:33.595 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.595 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D4fl755NnHw+QnD57y4IH4vTvJk= 2023-11-02T18:30:33.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D4fl755NnHw+QnD57y4IH4vTvJk=', 'Content-Length': '0'} 2023-11-02T18:30:33.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfa4efb6ec5b5560-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.599 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.600 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.600 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.600 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.601 DEBUG:boto:Method: GET 2023-11-02T18:30:33.601 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.602 DEBUG:boto:Data: 2023-11-02T18:30:33.602 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.603 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.603 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.603 DEBUG:boto:Params: {} 2023-11-02T18:30:33.604 DEBUG:boto:Token: None 2023-11-02T18:30:33.604 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000031?versionId=null 2023-11-02T18:30:33.605 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D4fl755NnHw+QnD57y4IH4vTvJk= 2023-11-02T18:30:33.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D4fl755NnHw+QnD57y4IH4vTvJk=', 'Content-Length': '0'} 2023-11-02T18:30:33.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033487aa0e9b48d70-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.651 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.651 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.652 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000031 2023-11-02T18:30:33.652 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000031 2023-11-02T18:30:33.653 DEBUG:boto:Method: GET 2023-11-02T18:30:33.653 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000031 2023-11-02T18:30:33.653 DEBUG:boto:Data: 2023-11-02T18:30:33.654 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.655 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.655 DEBUG:boto:Params: {} 2023-11-02T18:30:33.656 DEBUG:boto:Token: None 2023-11-02T18:30:33.656 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.657 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f56d6618ece21c1b-006543eac9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.663 DEBUG:boto:b'ibtljn-58obj000000000000000000311000falseobj000000000000000000312023-11-02T18:30:16.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.663 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.664 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.664 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000031 2023-11-02T18:30:33.665 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000031 2023-11-02T18:30:33.665 DEBUG:boto:Method: GET 2023-11-02T18:30:33.666 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000031 2023-11-02T18:30:33.666 DEBUG:boto:Data: 2023-11-02T18:30:33.666 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.667 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.667 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.668 DEBUG:boto:Params: {} 2023-11-02T18:30:33.668 DEBUG:boto:Token: None 2023-11-02T18:30:33.669 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.669 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b95dc608919701d-006543eac9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.680 DEBUG:boto:b'ibtljn-58obj000000000000000000311000falseobj000000000000000000312023-11-02T18:30:16.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.680 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.681 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.681 DEBUG:boto:Method: HEAD 2023-11-02T18:30:33.682 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.682 DEBUG:boto:Data: 2023-11-02T18:30:33.682 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.683 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.683 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.684 DEBUG:boto:Params: {} 2023-11-02T18:30:33.684 DEBUG:boto:Token: None 2023-11-02T18:30:33.685 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.685 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PSr8LLOBkXLzADBI8CoAmYoXV2A= 2023-11-02T18:30:33.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PSr8LLOBkXLzADBI8CoAmYoXV2A=', 'Content-Length': '0'} 2023-11-02T18:30:33.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089c37389d6a11b18-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.688 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.689 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.689 DEBUG:boto:Method: HEAD 2023-11-02T18:30:33.690 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.690 DEBUG:boto:Data: 2023-11-02T18:30:33.691 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.691 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.691 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.692 DEBUG:boto:Params: {} 2023-11-02T18:30:33.692 DEBUG:boto:Token: None 2023-11-02T18:30:33.693 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.693 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PSr8LLOBkXLzADBI8CoAmYoXV2A= 2023-11-02T18:30:33.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PSr8LLOBkXLzADBI8CoAmYoXV2A=', 'Content-Length': '0'} 2023-11-02T18:30:33.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc4d52537f7e5b91-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-11-02T18:30:33.697 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.697 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.698 DEBUG:boto:Method: GET 2023-11-02T18:30:33.698 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.699 DEBUG:boto:Data: 2023-11-02T18:30:33.699 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.700 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.701 DEBUG:boto:Params: {} 2023-11-02T18:30:33.701 DEBUG:boto:Token: None 2023-11-02T18:30:33.702 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.702 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:40Hy4Obxoqftifwwt9Xji8PY200= 2023-11-02T18:30:33.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:40Hy4Obxoqftifwwt9Xji8PY200=', 'Content-Length': '0'} 2023-11-02T18:30:33.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dd5ef1f9936ecb1-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.747 DEBUG:boto:path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.747 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.748 DEBUG:boto:Method: GET 2023-11-02T18:30:33.748 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.749 DEBUG:boto:Data: 2023-11-02T18:30:33.749 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.749 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.750 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.750 DEBUG:boto:Params: {} 2023-11-02T18:30:33.751 DEBUG:boto:Token: None 2023-11-02T18:30:33.751 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000031 2023-11-02T18:30:33.752 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:40Hy4Obxoqftifwwt9Xji8PY200= 2023-11-02T18:30:33.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:40Hy4Obxoqftifwwt9Xji8PY200=', 'Content-Length': '0'} 2023-11-02T18:30:33.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9fd28db4f18edea-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.756 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.757 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.757 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000031 2023-11-02T18:30:33.758 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000031 2023-11-02T18:30:33.758 DEBUG:boto:Method: GET 2023-11-02T18:30:33.759 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000031 2023-11-02T18:30:33.759 DEBUG:boto:Data: 2023-11-02T18:30:33.759 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.760 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.760 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.761 DEBUG:boto:Params: {} 2023-11-02T18:30:33.761 DEBUG:boto:Token: None 2023-11-02T18:30:33.762 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.762 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087cc967cf227cdf8-006543eac9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.768 DEBUG:boto:b'ibtljn-58obj000000000000000000311000falseobj000000000000000000312023-11-02T18:30:16.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.769 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.769 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.770 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000031 2023-11-02T18:30:33.770 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000031 2023-11-02T18:30:33.771 DEBUG:boto:Method: GET 2023-11-02T18:30:33.771 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000031 2023-11-02T18:30:33.772 DEBUG:boto:Data: 2023-11-02T18:30:33.772 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.773 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.773 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.773 DEBUG:boto:Params: {} 2023-11-02T18:30:33.774 DEBUG:boto:Token: None 2023-11-02T18:30:33.774 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.775 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068996b31be350457-006543eac9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.830 DEBUG:boto:b'ibtljn-58obj000000000000000000311000falseobj000000000000000000312023-11-02T18:30:16.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-11-02T18:30:33.832 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:33.832 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:33.833 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.833 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.834 DEBUG:boto:Method: GET 2023-11-02T18:30:33.834 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.835 DEBUG:boto:Data: 2023-11-02T18:30:33.835 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.836 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.836 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.836 DEBUG:boto:Params: {} 2023-11-02T18:30:33.837 DEBUG:boto:Token: None 2023-11-02T18:30:33.838 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.838 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:urQK5J3aMsQRn5uldwjlm8l51HQ= 2023-11-02T18:30:33.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:urQK5J3aMsQRn5uldwjlm8l51HQ=', 'Content-Length': '0'} 2023-11-02T18:30:33.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e855aa24fef247e-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.844 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:33.844 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:33.845 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.845 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.846 DEBUG:boto:Method: GET 2023-11-02T18:30:33.846 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.847 DEBUG:boto:Data: 2023-11-02T18:30:33.847 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.848 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.848 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.849 DEBUG:boto:Params: {} 2023-11-02T18:30:33.849 DEBUG:boto:Token: None 2023-11-02T18:30:33.850 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.850 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:urQK5J3aMsQRn5uldwjlm8l51HQ= 2023-11-02T18:30:33.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:urQK5J3aMsQRn5uldwjlm8l51HQ=', 'Content-Length': '0'} 2023-11-02T18:30:33.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef6eb3565bc8e23c-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.855 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:33.856 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:33.856 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.857 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.858 DEBUG:boto:Method: HEAD 2023-11-02T18:30:33.858 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.859 DEBUG:boto:Data: 2023-11-02T18:30:33.859 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.860 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.860 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.861 DEBUG:boto:Params: {} 2023-11-02T18:30:33.862 DEBUG:boto:Token: None 2023-11-02T18:30:33.862 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.864 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XA1f+jok/RPIAF8pKCAO9RTi44I= 2023-11-02T18:30:33.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XA1f+jok/RPIAF8pKCAO9RTi44I=', 'Content-Length': '0'} 2023-11-02T18:30:33.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e14c9d135489981-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.870 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:33.871 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:33.872 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.873 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.874 DEBUG:boto:Method: HEAD 2023-11-02T18:30:33.874 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.875 DEBUG:boto:Data: 2023-11-02T18:30:33.875 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.876 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.877 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.877 DEBUG:boto:Params: {} 2023-11-02T18:30:33.878 DEBUG:boto:Token: None 2023-11-02T18:30:33.879 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.880 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XA1f+jok/RPIAF8pKCAO9RTi44I= 2023-11-02T18:30:33.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XA1f+jok/RPIAF8pKCAO9RTi44I=', 'Content-Length': '0'} 2023-11-02T18:30:33.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063f677740460924f-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-11-02T18:30:33.886 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:33.887 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:33.887 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.887 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.888 DEBUG:boto:Method: GET 2023-11-02T18:30:33.888 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.889 DEBUG:boto:Data: 2023-11-02T18:30:33.889 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.890 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.890 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.891 DEBUG:boto:Params: {} 2023-11-02T18:30:33.891 DEBUG:boto:Token: None 2023-11-02T18:30:33.892 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.893 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:urQK5J3aMsQRn5uldwjlm8l51HQ= 2023-11-02T18:30:33.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:urQK5J3aMsQRn5uldwjlm8l51HQ=', 'Content-Length': '0'} 2023-11-02T18:30:33.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000470e534bdec505cd-006543eac9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.940 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:33.941 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:33.942 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.942 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.943 DEBUG:boto:Method: GET 2023-11-02T18:30:33.943 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.944 DEBUG:boto:Data: 2023-11-02T18:30:33.944 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.944 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.945 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.945 DEBUG:boto:Params: {} 2023-11-02T18:30:33.946 DEBUG:boto:Token: None 2023-11-02T18:30:33.947 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/obj00000000000000000032?versionId=null 2023-11-02T18:30:33.947 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:urQK5J3aMsQRn5uldwjlm8l51HQ= 2023-11-02T18:30:33.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:urQK5J3aMsQRn5uldwjlm8l51HQ=', 'Content-Length': '0'} 2023-11-02T18:30:33.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bd416c792f8529a-006543eac9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.952 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.952 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.953 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000032 2023-11-02T18:30:33.953 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000032 2023-11-02T18:30:33.954 DEBUG:boto:Method: GET 2023-11-02T18:30:33.954 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000032 2023-11-02T18:30:33.954 DEBUG:boto:Data: 2023-11-02T18:30:33.955 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.955 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:33.956 DEBUG:boto:Port: 8001 2023-11-02T18:30:33.956 DEBUG:boto:Params: {} 2023-11-02T18:30:33.957 DEBUG:boto:Token: None 2023-11-02T18:30:33.957 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.958 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8c6bf31dd4d494c-006543eac9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:33.965 DEBUG:boto:b'ibtljn-58obj000000000000000000321000falseobj000000000000000000322023-11-02T18:30:14.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:33.966 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:33.967 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:33.967 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000032 2023-11-02T18:30:33.968 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000032 2023-11-02T18:30:33.968 DEBUG:boto:Method: GET 2023-11-02T18:30:33.969 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000032 2023-11-02T18:30:33.969 DEBUG:boto:Data: 2023-11-02T18:30:33.969 DEBUG:boto:Headers: {} 2023-11-02T18:30:33.970 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:33.970 DEBUG:boto:Port: 8000 2023-11-02T18:30:33.971 DEBUG:boto:Params: {} 2023-11-02T18:30:33.971 DEBUG:boto:Token: None 2023-11-02T18:30:33.972 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:33 GMT /ibtljn-58/ 2023-11-02T18:30:33.972 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98= 2023-11-02T18:30:33.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gs5G2UQkIjCZC2k9r7w516ohj98=', 'Content-Length': '0'} 2023-11-02T18:30:33.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004343d1ae25e7f1e3-006543eac9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.029 DEBUG:boto:b'ibtljn-58obj000000000000000000321000falseobj000000000000000000322023-11-02T18:30:14.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.030 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.030 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.031 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.031 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.032 DEBUG:boto:Data: 2023-11-02T18:30:34.032 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.033 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.033 DEBUG:boto:Params: {} 2023-11-02T18:30:34.034 DEBUG:boto:Token: None 2023-11-02T18:30:34.034 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wS19poftjUCG6zrYz7wXyXEM1Hs= 2023-11-02T18:30:34.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wS19poftjUCG6zrYz7wXyXEM1Hs=', 'Content-Length': '0'} 2023-11-02T18:30:34.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c7101044c3160b0-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.039 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.039 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.040 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.040 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.041 DEBUG:boto:Data: 2023-11-02T18:30:34.041 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.041 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.042 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.042 DEBUG:boto:Params: {} 2023-11-02T18:30:34.043 DEBUG:boto:Token: None 2023-11-02T18:30:34.043 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.044 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wS19poftjUCG6zrYz7wXyXEM1Hs= 2023-11-02T18:30:34.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wS19poftjUCG6zrYz7wXyXEM1Hs=', 'Content-Length': '0'} 2023-11-02T18:30:34.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000106b658a34014a61-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-11-02T18:30:34.048 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.049 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.049 DEBUG:boto:Method: GET 2023-11-02T18:30:34.049 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.050 DEBUG:boto:Data: 2023-11-02T18:30:34.050 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.051 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.051 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.052 DEBUG:boto:Params: {} 2023-11-02T18:30:34.052 DEBUG:boto:Token: None 2023-11-02T18:30:34.053 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.053 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zdCIyzY1tq6/2vocoPqQgm0xbME= 2023-11-02T18:30:34.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zdCIyzY1tq6/2vocoPqQgm0xbME=', 'Content-Length': '0'} 2023-11-02T18:30:34.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f108f9bac466aa6e-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.059 DEBUG:boto:path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.059 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.060 DEBUG:boto:Method: GET 2023-11-02T18:30:34.060 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.061 DEBUG:boto:Data: 2023-11-02T18:30:34.061 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.061 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.062 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.063 DEBUG:boto:Params: {} 2023-11-02T18:30:34.063 DEBUG:boto:Token: None 2023-11-02T18:30:34.063 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000032 2023-11-02T18:30:34.064 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zdCIyzY1tq6/2vocoPqQgm0xbME= 2023-11-02T18:30:34.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zdCIyzY1tq6/2vocoPqQgm0xbME=', 'Content-Length': '0'} 2023-11-02T18:30:34.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b060c796821e771c-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.077 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.077 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.078 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000032 2023-11-02T18:30:34.078 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000032 2023-11-02T18:30:34.079 DEBUG:boto:Method: GET 2023-11-02T18:30:34.079 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000032 2023-11-02T18:30:34.080 DEBUG:boto:Data: 2023-11-02T18:30:34.081 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.081 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.082 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.082 DEBUG:boto:Params: {} 2023-11-02T18:30:34.083 DEBUG:boto:Token: None 2023-11-02T18:30:34.083 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.084 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f828e1d5276558e6-006543eaca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.091 DEBUG:boto:b'ibtljn-58obj000000000000000000321000falseobj000000000000000000322023-11-02T18:30:14.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.092 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.092 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.093 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000032 2023-11-02T18:30:34.093 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000032 2023-11-02T18:30:34.094 DEBUG:boto:Method: GET 2023-11-02T18:30:34.094 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000032 2023-11-02T18:30:34.095 DEBUG:boto:Data: 2023-11-02T18:30:34.095 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.096 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.096 DEBUG:boto:Params: {} 2023-11-02T18:30:34.097 DEBUG:boto:Token: None 2023-11-02T18:30:34.097 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009f9647700803b47-006543eaca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.154 DEBUG:boto:b'ibtljn-58obj000000000000000000321000falseobj000000000000000000322023-11-02T18:30:14.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-11-02T18:30:34.155 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.156 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.156 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.156 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.157 DEBUG:boto:Method: GET 2023-11-02T18:30:34.157 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.158 DEBUG:boto:Data: 2023-11-02T18:30:34.158 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.159 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.159 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.159 DEBUG:boto:Params: {} 2023-11-02T18:30:34.160 DEBUG:boto:Token: None 2023-11-02T18:30:34.160 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.161 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1pICWn0C3KWPZkjkOn1amxFMZk= 2023-11-02T18:30:34.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1pICWn0C3KWPZkjkOn1amxFMZk=', 'Content-Length': '0'} 2023-11-02T18:30:34.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cf6e982f8de4ab2-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.170 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.171 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.171 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.172 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.172 DEBUG:boto:Method: GET 2023-11-02T18:30:34.173 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.173 DEBUG:boto:Data: 2023-11-02T18:30:34.174 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.175 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.175 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.176 DEBUG:boto:Params: {} 2023-11-02T18:30:34.176 DEBUG:boto:Token: None 2023-11-02T18:30:34.177 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1pICWn0C3KWPZkjkOn1amxFMZk= 2023-11-02T18:30:34.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1pICWn0C3KWPZkjkOn1amxFMZk=', 'Content-Length': '0'} 2023-11-02T18:30:34.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e29fd20f4abd199-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.183 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.183 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.184 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.184 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.184 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.185 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.185 DEBUG:boto:Data: 2023-11-02T18:30:34.186 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.186 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.187 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.187 DEBUG:boto:Params: {} 2023-11-02T18:30:34.190 DEBUG:boto:Token: None 2023-11-02T18:30:34.190 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.191 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pKjB6GFsce0ijSwO2hDa35K42b0= 2023-11-02T18:30:34.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pKjB6GFsce0ijSwO2hDa35K42b0=', 'Content-Length': '0'} 2023-11-02T18:30:34.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089d2b5d729f7ee0f-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.195 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.196 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.196 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.197 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.197 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.197 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.198 DEBUG:boto:Data: 2023-11-02T18:30:34.198 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.199 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.199 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.200 DEBUG:boto:Params: {} 2023-11-02T18:30:34.200 DEBUG:boto:Token: None 2023-11-02T18:30:34.201 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pKjB6GFsce0ijSwO2hDa35K42b0= 2023-11-02T18:30:34.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pKjB6GFsce0ijSwO2hDa35K42b0=', 'Content-Length': '0'} 2023-11-02T18:30:34.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000488c7b18db9a6f97-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-11-02T18:30:34.206 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.206 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.207 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.207 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.207 DEBUG:boto:Method: GET 2023-11-02T18:30:34.208 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.208 DEBUG:boto:Data: 2023-11-02T18:30:34.209 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.209 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.210 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.210 DEBUG:boto:Params: {} 2023-11-02T18:30:34.211 DEBUG:boto:Token: None 2023-11-02T18:30:34.211 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.212 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1pICWn0C3KWPZkjkOn1amxFMZk= 2023-11-02T18:30:34.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1pICWn0C3KWPZkjkOn1amxFMZk=', 'Content-Length': '0'} 2023-11-02T18:30:34.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000648368524c5fcd0b-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.263 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.264 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.264 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.265 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.265 DEBUG:boto:Method: GET 2023-11-02T18:30:34.266 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.266 DEBUG:boto:Data: 2023-11-02T18:30:34.266 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.267 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.267 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.268 DEBUG:boto:Params: {} 2023-11-02T18:30:34.268 DEBUG:boto:Token: None 2023-11-02T18:30:34.269 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000033?versionId=null 2023-11-02T18:30:34.269 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J1pICWn0C3KWPZkjkOn1amxFMZk= 2023-11-02T18:30:34.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1pICWn0C3KWPZkjkOn1amxFMZk=', 'Content-Length': '0'} 2023-11-02T18:30:34.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003725a862b60e117b-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.274 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.274 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.275 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000033 2023-11-02T18:30:34.275 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000033 2023-11-02T18:30:34.276 DEBUG:boto:Method: GET 2023-11-02T18:30:34.276 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000033 2023-11-02T18:30:34.276 DEBUG:boto:Data: 2023-11-02T18:30:34.277 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.277 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.278 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.278 DEBUG:boto:Params: {} 2023-11-02T18:30:34.279 DEBUG:boto:Token: None 2023-11-02T18:30:34.279 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.280 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043c5a079b83fdbbb-006543eaca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.289 DEBUG:boto:b'ibtljn-58obj000000000000000000331000falseobj000000000000000000332023-11-02T18:30:19.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.290 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.290 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.291 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000033 2023-11-02T18:30:34.291 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000033 2023-11-02T18:30:34.291 DEBUG:boto:Method: GET 2023-11-02T18:30:34.292 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000033 2023-11-02T18:30:34.292 DEBUG:boto:Data: 2023-11-02T18:30:34.293 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.293 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.294 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.294 DEBUG:boto:Params: {} 2023-11-02T18:30:34.294 DEBUG:boto:Token: None 2023-11-02T18:30:34.295 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.295 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002289b03a70fb457c-006543eaca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.303 DEBUG:boto:b'ibtljn-58obj000000000000000000331000falseobj000000000000000000332023-11-02T18:30:19.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.304 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.304 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.304 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.305 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.305 DEBUG:boto:Data: 2023-11-02T18:30:34.306 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.306 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.307 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.307 DEBUG:boto:Params: {} 2023-11-02T18:30:34.307 DEBUG:boto:Token: None 2023-11-02T18:30:34.308 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.308 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6X2l5l3bDAZ/CSgoBWV7sSoWSNU= 2023-11-02T18:30:34.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6X2l5l3bDAZ/CSgoBWV7sSoWSNU=', 'Content-Length': '0'} 2023-11-02T18:30:34.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce773b6231ddb498-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.314 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.315 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.315 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.316 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.316 DEBUG:boto:Data: 2023-11-02T18:30:34.317 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.318 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.318 DEBUG:boto:Params: {} 2023-11-02T18:30:34.319 DEBUG:boto:Token: None 2023-11-02T18:30:34.319 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6X2l5l3bDAZ/CSgoBWV7sSoWSNU= 2023-11-02T18:30:34.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6X2l5l3bDAZ/CSgoBWV7sSoWSNU=', 'Content-Length': '0'} 2023-11-02T18:30:34.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d445bc920baaa26d-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-11-02T18:30:34.325 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.326 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.326 DEBUG:boto:Method: GET 2023-11-02T18:30:34.327 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.327 DEBUG:boto:Data: 2023-11-02T18:30:34.328 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.329 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.329 DEBUG:boto:Params: {} 2023-11-02T18:30:34.330 DEBUG:boto:Token: None 2023-11-02T18:30:34.331 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.331 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a3Ivzeh+PMrzKk2lz9+FKTlT4JQ= 2023-11-02T18:30:34.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a3Ivzeh+PMrzKk2lz9+FKTlT4JQ=', 'Content-Length': '0'} 2023-11-02T18:30:34.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d5572eab482df43-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.378 DEBUG:boto:path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.379 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.380 DEBUG:boto:Method: GET 2023-11-02T18:30:34.380 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.381 DEBUG:boto:Data: 2023-11-02T18:30:34.381 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.381 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.382 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.382 DEBUG:boto:Params: {} 2023-11-02T18:30:34.383 DEBUG:boto:Token: None 2023-11-02T18:30:34.383 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000033 2023-11-02T18:30:34.384 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a3Ivzeh+PMrzKk2lz9+FKTlT4JQ= 2023-11-02T18:30:34.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a3Ivzeh+PMrzKk2lz9+FKTlT4JQ=', 'Content-Length': '0'} 2023-11-02T18:30:34.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098da674ef201b22a-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.388 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.389 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.389 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000033 2023-11-02T18:30:34.390 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000033 2023-11-02T18:30:34.390 DEBUG:boto:Method: GET 2023-11-02T18:30:34.390 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000033 2023-11-02T18:30:34.391 DEBUG:boto:Data: 2023-11-02T18:30:34.395 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.395 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.396 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.396 DEBUG:boto:Params: {} 2023-11-02T18:30:34.396 DEBUG:boto:Token: None 2023-11-02T18:30:34.397 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.397 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9aa24891be365d2-006543eaca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.403 DEBUG:boto:b'ibtljn-58obj000000000000000000331000falseobj000000000000000000332023-11-02T18:30:19.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.404 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.404 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.405 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000033 2023-11-02T18:30:34.405 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000033 2023-11-02T18:30:34.406 DEBUG:boto:Method: GET 2023-11-02T18:30:34.406 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000033 2023-11-02T18:30:34.407 DEBUG:boto:Data: 2023-11-02T18:30:34.407 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.407 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.408 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.408 DEBUG:boto:Params: {} 2023-11-02T18:30:34.409 DEBUG:boto:Token: None 2023-11-02T18:30:34.409 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.410 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000105ff2dbd5031a0a-006543eaca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.421 DEBUG:boto:b'ibtljn-58obj000000000000000000331000falseobj000000000000000000332023-11-02T18:30:19.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-11-02T18:30:34.422 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.422 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.423 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.423 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.424 DEBUG:boto:Method: GET 2023-11-02T18:30:34.424 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.425 DEBUG:boto:Data: 2023-11-02T18:30:34.425 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.425 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.426 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.426 DEBUG:boto:Params: {} 2023-11-02T18:30:34.427 DEBUG:boto:Token: None 2023-11-02T18:30:34.427 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qt3Cwv+2eprThVHMLYrZA+jujaQ= 2023-11-02T18:30:34.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qt3Cwv+2eprThVHMLYrZA+jujaQ=', 'Content-Length': '0'} 2023-11-02T18:30:34.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d228809f798c003-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.432 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.433 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.433 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.434 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.434 DEBUG:boto:Method: GET 2023-11-02T18:30:34.434 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.435 DEBUG:boto:Data: 2023-11-02T18:30:34.435 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.436 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.436 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.437 DEBUG:boto:Params: {} 2023-11-02T18:30:34.437 DEBUG:boto:Token: None 2023-11-02T18:30:34.438 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.438 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qt3Cwv+2eprThVHMLYrZA+jujaQ= 2023-11-02T18:30:34.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qt3Cwv+2eprThVHMLYrZA+jujaQ=', 'Content-Length': '0'} 2023-11-02T18:30:34.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a9075edfa1d3a78-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.442 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.443 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.443 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.443 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.444 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.444 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.445 DEBUG:boto:Data: 2023-11-02T18:30:34.445 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.445 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.446 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.446 DEBUG:boto:Params: {} 2023-11-02T18:30:34.447 DEBUG:boto:Token: None 2023-11-02T18:30:34.447 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.448 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QDaLiTOpq3YlTFnIbG/MGmxCmh4= 2023-11-02T18:30:34.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QDaLiTOpq3YlTFnIbG/MGmxCmh4=', 'Content-Length': '0'} 2023-11-02T18:30:34.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052fdd2095fdaeef8-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.454 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.454 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.455 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.455 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.456 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.456 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.457 DEBUG:boto:Data: 2023-11-02T18:30:34.457 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.457 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.458 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.458 DEBUG:boto:Params: {} 2023-11-02T18:30:34.459 DEBUG:boto:Token: None 2023-11-02T18:30:34.459 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.460 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QDaLiTOpq3YlTFnIbG/MGmxCmh4= 2023-11-02T18:30:34.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QDaLiTOpq3YlTFnIbG/MGmxCmh4=', 'Content-Length': '0'} 2023-11-02T18:30:34.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2ac21c71b0d4c92-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-11-02T18:30:34.465 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.466 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.466 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.467 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.467 DEBUG:boto:Method: GET 2023-11-02T18:30:34.468 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.468 DEBUG:boto:Data: 2023-11-02T18:30:34.468 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.469 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.469 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.470 DEBUG:boto:Params: {} 2023-11-02T18:30:34.470 DEBUG:boto:Token: None 2023-11-02T18:30:34.471 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.471 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qt3Cwv+2eprThVHMLYrZA+jujaQ= 2023-11-02T18:30:34.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qt3Cwv+2eprThVHMLYrZA+jujaQ=', 'Content-Length': '0'} 2023-11-02T18:30:34.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004850fea7d9baa3b0-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.520 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.520 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.521 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.521 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.522 DEBUG:boto:Method: GET 2023-11-02T18:30:34.522 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.523 DEBUG:boto:Data: 2023-11-02T18:30:34.523 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.523 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.524 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.524 DEBUG:boto:Params: {} 2023-11-02T18:30:34.525 DEBUG:boto:Token: None 2023-11-02T18:30:34.525 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000036?versionId=null 2023-11-02T18:30:34.526 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qt3Cwv+2eprThVHMLYrZA+jujaQ= 2023-11-02T18:30:34.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qt3Cwv+2eprThVHMLYrZA+jujaQ=', 'Content-Length': '0'} 2023-11-02T18:30:34.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c583460746136c-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.531 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.531 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.531 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000036 2023-11-02T18:30:34.532 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000036 2023-11-02T18:30:34.532 DEBUG:boto:Method: GET 2023-11-02T18:30:34.533 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000036 2023-11-02T18:30:34.533 DEBUG:boto:Data: 2023-11-02T18:30:34.534 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.534 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.534 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.535 DEBUG:boto:Params: {} 2023-11-02T18:30:34.535 DEBUG:boto:Token: None 2023-11-02T18:30:34.536 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.536 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d01811fe782dd501-006543eaca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.543 DEBUG:boto:b'ibtljn-58obj000000000000000000361000falseobj000000000000000000362023-11-02T18:30:20.191Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.543 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.544 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.544 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000036 2023-11-02T18:30:34.545 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000036 2023-11-02T18:30:34.545 DEBUG:boto:Method: GET 2023-11-02T18:30:34.546 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000036 2023-11-02T18:30:34.546 DEBUG:boto:Data: 2023-11-02T18:30:34.547 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.547 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.548 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.548 DEBUG:boto:Params: {} 2023-11-02T18:30:34.549 DEBUG:boto:Token: None 2023-11-02T18:30:34.549 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.550 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6fc40f41a2aff20-006543eaca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.562 DEBUG:boto:b'ibtljn-58obj000000000000000000361000falseobj000000000000000000362023-11-02T18:30:20.191Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.563 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.563 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.564 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.564 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.565 DEBUG:boto:Data: 2023-11-02T18:30:34.565 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.566 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.567 DEBUG:boto:Params: {} 2023-11-02T18:30:34.567 DEBUG:boto:Token: None 2023-11-02T18:30:34.568 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/KKyDcC6mmKq98AgXfmnlrwdXKY= 2023-11-02T18:30:34.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/KKyDcC6mmKq98AgXfmnlrwdXKY=', 'Content-Length': '0'} 2023-11-02T18:30:34.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089b9fbf07bb13dd5-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.573 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.573 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.574 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.574 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.575 DEBUG:boto:Data: 2023-11-02T18:30:34.575 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.575 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.576 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.576 DEBUG:boto:Params: {} 2023-11-02T18:30:34.577 DEBUG:boto:Token: None 2023-11-02T18:30:34.577 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.578 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/KKyDcC6mmKq98AgXfmnlrwdXKY= 2023-11-02T18:30:34.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/KKyDcC6mmKq98AgXfmnlrwdXKY=', 'Content-Length': '0'} 2023-11-02T18:30:34.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f9d2469ed0f164e-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-11-02T18:30:34.582 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.582 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.583 DEBUG:boto:Method: GET 2023-11-02T18:30:34.583 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.584 DEBUG:boto:Data: 2023-11-02T18:30:34.584 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.585 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.585 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.586 DEBUG:boto:Params: {} 2023-11-02T18:30:34.586 DEBUG:boto:Token: None 2023-11-02T18:30:34.587 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.587 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:azJm05gJiHdjINm+KiKxZXO/XUo= 2023-11-02T18:30:34.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:azJm05gJiHdjINm+KiKxZXO/XUo=', 'Content-Length': '0'} 2023-11-02T18:30:34.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce31516ad6f4d57-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.635 DEBUG:boto:path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.636 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.637 DEBUG:boto:Method: GET 2023-11-02T18:30:34.637 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.638 DEBUG:boto:Data: 2023-11-02T18:30:34.638 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.639 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.640 DEBUG:boto:Params: {} 2023-11-02T18:30:34.640 DEBUG:boto:Token: None 2023-11-02T18:30:34.641 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000036 2023-11-02T18:30:34.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:azJm05gJiHdjINm+KiKxZXO/XUo= 2023-11-02T18:30:34.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:azJm05gJiHdjINm+KiKxZXO/XUo=', 'Content-Length': '0'} 2023-11-02T18:30:34.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007950f86d00895b30-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.646 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.647 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.647 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000036 2023-11-02T18:30:34.647 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000036 2023-11-02T18:30:34.648 DEBUG:boto:Method: GET 2023-11-02T18:30:34.648 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000036 2023-11-02T18:30:34.649 DEBUG:boto:Data: 2023-11-02T18:30:34.649 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.650 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.650 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.651 DEBUG:boto:Params: {} 2023-11-02T18:30:34.651 DEBUG:boto:Token: None 2023-11-02T18:30:34.652 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.652 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037876d0cd9f3e903-006543eaca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.659 DEBUG:boto:b'ibtljn-58obj000000000000000000361000falseobj000000000000000000362023-11-02T18:30:20.191Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.660 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.661 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.661 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000036 2023-11-02T18:30:34.662 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000036 2023-11-02T18:30:34.662 DEBUG:boto:Method: GET 2023-11-02T18:30:34.662 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000036 2023-11-02T18:30:34.663 DEBUG:boto:Data: 2023-11-02T18:30:34.663 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.664 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.664 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.665 DEBUG:boto:Params: {} 2023-11-02T18:30:34.665 DEBUG:boto:Token: None 2023-11-02T18:30:34.666 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.666 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aaf523d8dc4f3f3-006543eaca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.678 DEBUG:boto:b'ibtljn-58obj000000000000000000361000falseobj000000000000000000362023-11-02T18:30:20.191Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-11-02T18:30:34.680 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.681 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.682 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.682 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.683 DEBUG:boto:Method: GET 2023-11-02T18:30:34.683 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.683 DEBUG:boto:Data: 2023-11-02T18:30:34.684 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.684 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.685 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.685 DEBUG:boto:Params: {} 2023-11-02T18:30:34.686 DEBUG:boto:Token: None 2023-11-02T18:30:34.686 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.687 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:flyVOyX/Z8bn/QYQO/UjgrLq610= 2023-11-02T18:30:34.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:flyVOyX/Z8bn/QYQO/UjgrLq610=', 'Content-Length': '0'} 2023-11-02T18:30:34.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014725ccd6e8c5286-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.691 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.692 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.692 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.693 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.693 DEBUG:boto:Method: GET 2023-11-02T18:30:34.694 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.694 DEBUG:boto:Data: 2023-11-02T18:30:34.695 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.695 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.696 DEBUG:boto:Params: {} 2023-11-02T18:30:34.696 DEBUG:boto:Token: None 2023-11-02T18:30:34.697 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.697 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:flyVOyX/Z8bn/QYQO/UjgrLq610= 2023-11-02T18:30:34.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:flyVOyX/Z8bn/QYQO/UjgrLq610=', 'Content-Length': '0'} 2023-11-02T18:30:34.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000321ae0e13cd91c60-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.701 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.702 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.702 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.703 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.703 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.703 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.704 DEBUG:boto:Data: 2023-11-02T18:30:34.704 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.705 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.705 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.705 DEBUG:boto:Params: {} 2023-11-02T18:30:34.706 DEBUG:boto:Token: None 2023-11-02T18:30:34.706 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.707 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c479UYHSMJdWBNpVPPFi94KuEDY= 2023-11-02T18:30:34.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c479UYHSMJdWBNpVPPFi94KuEDY=', 'Content-Length': '0'} 2023-11-02T18:30:34.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093818606eb08b5de-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.712 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.712 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.713 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.713 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.714 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.714 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.715 DEBUG:boto:Data: 2023-11-02T18:30:34.716 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.716 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.717 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.718 DEBUG:boto:Params: {} 2023-11-02T18:30:34.718 DEBUG:boto:Token: None 2023-11-02T18:30:34.719 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.719 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c479UYHSMJdWBNpVPPFi94KuEDY= 2023-11-02T18:30:34.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c479UYHSMJdWBNpVPPFi94KuEDY=', 'Content-Length': '0'} 2023-11-02T18:30:34.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e9157d3b5fd750-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-11-02T18:30:34.724 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.725 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.725 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.726 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.726 DEBUG:boto:Method: GET 2023-11-02T18:30:34.727 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.727 DEBUG:boto:Data: 2023-11-02T18:30:34.728 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.728 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.728 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.729 DEBUG:boto:Params: {} 2023-11-02T18:30:34.729 DEBUG:boto:Token: None 2023-11-02T18:30:34.730 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.730 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:flyVOyX/Z8bn/QYQO/UjgrLq610= 2023-11-02T18:30:34.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:flyVOyX/Z8bn/QYQO/UjgrLq610=', 'Content-Length': '0'} 2023-11-02T18:30:34.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021fc96789403ce7d-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.775 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.776 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.776 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.776 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.777 DEBUG:boto:Method: GET 2023-11-02T18:30:34.777 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.778 DEBUG:boto:Data: 2023-11-02T18:30:34.778 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.779 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.779 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.779 DEBUG:boto:Params: {} 2023-11-02T18:30:34.780 DEBUG:boto:Token: None 2023-11-02T18:30:34.781 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000037?versionId=null 2023-11-02T18:30:34.781 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:flyVOyX/Z8bn/QYQO/UjgrLq610= 2023-11-02T18:30:34.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:flyVOyX/Z8bn/QYQO/UjgrLq610=', 'Content-Length': '0'} 2023-11-02T18:30:34.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b50e9dbcd0f8b43-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.785 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.786 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.786 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000037 2023-11-02T18:30:34.787 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000037 2023-11-02T18:30:34.787 DEBUG:boto:Method: GET 2023-11-02T18:30:34.788 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000037 2023-11-02T18:30:34.788 DEBUG:boto:Data: 2023-11-02T18:30:34.789 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.789 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.789 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.790 DEBUG:boto:Params: {} 2023-11-02T18:30:34.790 DEBUG:boto:Token: None 2023-11-02T18:30:34.791 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.791 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b350354fe09e050-006543eaca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.797 DEBUG:boto:b'ibtljn-58obj000000000000000000371000falseobj000000000000000000372023-11-02T18:30:16.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.798 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.798 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.799 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000037 2023-11-02T18:30:34.799 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000037 2023-11-02T18:30:34.800 DEBUG:boto:Method: GET 2023-11-02T18:30:34.800 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000037 2023-11-02T18:30:34.800 DEBUG:boto:Data: 2023-11-02T18:30:34.801 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.801 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.802 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.802 DEBUG:boto:Params: {} 2023-11-02T18:30:34.803 DEBUG:boto:Token: None 2023-11-02T18:30:34.803 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.804 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022e48b1f8941fcb1-006543eaca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.815 DEBUG:boto:b'ibtljn-58obj000000000000000000371000falseobj000000000000000000372023-11-02T18:30:16.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.817 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.818 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.818 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.819 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.819 DEBUG:boto:Data: 2023-11-02T18:30:34.819 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.820 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.820 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.821 DEBUG:boto:Params: {} 2023-11-02T18:30:34.821 DEBUG:boto:Token: None 2023-11-02T18:30:34.822 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.822 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y4JvdD0nm73o1Lwjfr/OPi1BlyM= 2023-11-02T18:30:34.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y4JvdD0nm73o1Lwjfr/OPi1BlyM=', 'Content-Length': '0'} 2023-11-02T18:30:34.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f1f0a84619468f-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.826 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.826 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.827 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.827 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.828 DEBUG:boto:Data: 2023-11-02T18:30:34.828 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.828 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.829 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.829 DEBUG:boto:Params: {} 2023-11-02T18:30:34.830 DEBUG:boto:Token: None 2023-11-02T18:30:34.830 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.831 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y4JvdD0nm73o1Lwjfr/OPi1BlyM= 2023-11-02T18:30:34.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y4JvdD0nm73o1Lwjfr/OPi1BlyM=', 'Content-Length': '0'} 2023-11-02T18:30:34.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bffc3d4112450b8d-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-11-02T18:30:34.835 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.835 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.836 DEBUG:boto:Method: GET 2023-11-02T18:30:34.836 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.837 DEBUG:boto:Data: 2023-11-02T18:30:34.837 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.838 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.838 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.838 DEBUG:boto:Params: {} 2023-11-02T18:30:34.839 DEBUG:boto:Token: None 2023-11-02T18:30:34.839 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.840 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AbhpCmZvK+ZNlO0wlmpeF3ICFQ0= 2023-11-02T18:30:34.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AbhpCmZvK+ZNlO0wlmpeF3ICFQ0=', 'Content-Length': '0'} 2023-11-02T18:30:34.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000541c571bd0c5def7-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.844 DEBUG:boto:path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.844 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.845 DEBUG:boto:Method: GET 2023-11-02T18:30:34.845 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.846 DEBUG:boto:Data: 2023-11-02T18:30:34.846 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.847 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.847 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.847 DEBUG:boto:Params: {} 2023-11-02T18:30:34.848 DEBUG:boto:Token: None 2023-11-02T18:30:34.848 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000037 2023-11-02T18:30:34.849 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AbhpCmZvK+ZNlO0wlmpeF3ICFQ0= 2023-11-02T18:30:34.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AbhpCmZvK+ZNlO0wlmpeF3ICFQ0=', 'Content-Length': '0'} 2023-11-02T18:30:34.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a7e9b84d24bcd6-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.853 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.854 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.854 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000037 2023-11-02T18:30:34.854 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000037 2023-11-02T18:30:34.855 DEBUG:boto:Method: GET 2023-11-02T18:30:34.856 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000037 2023-11-02T18:30:34.856 DEBUG:boto:Data: 2023-11-02T18:30:34.856 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.857 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.858 DEBUG:boto:Params: {} 2023-11-02T18:30:34.858 DEBUG:boto:Token: None 2023-11-02T18:30:34.859 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.859 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002be3de094a7d79ea-006543eaca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.907 DEBUG:boto:b'ibtljn-58obj000000000000000000371000falseobj000000000000000000372023-11-02T18:30:16.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.908 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:34.908 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:34.909 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000037 2023-11-02T18:30:34.909 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000037 2023-11-02T18:30:34.910 DEBUG:boto:Method: GET 2023-11-02T18:30:34.910 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000037 2023-11-02T18:30:34.911 DEBUG:boto:Data: 2023-11-02T18:30:34.911 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.912 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.912 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.912 DEBUG:boto:Params: {} 2023-11-02T18:30:34.913 DEBUG:boto:Token: None 2023-11-02T18:30:34.913 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/ 2023-11-02T18:30:34.914 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0= 2023-11-02T18:30:34.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ThjpQmRmI9rXm1vBCBuu1vmjnz0=', 'Content-Length': '0'} 2023-11-02T18:30:34.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbeb78d210e8166b-006543eaca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.927 DEBUG:boto:b'ibtljn-58obj000000000000000000371000falseobj000000000000000000372023-11-02T18:30:16.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:34.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-11-02T18:30:34.928 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:34.928 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:34.929 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:34.929 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:34.930 DEBUG:boto:Method: GET 2023-11-02T18:30:34.930 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:34.930 DEBUG:boto:Data: 2023-11-02T18:30:34.931 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.931 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.932 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.932 DEBUG:boto:Params: {} 2023-11-02T18:30:34.933 DEBUG:boto:Token: None 2023-11-02T18:30:34.933 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:34.934 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J5b1nZB14CHp5KThxC1Lc0pRhqs= 2023-11-02T18:30:34.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J5b1nZB14CHp5KThxC1Lc0pRhqs=', 'Content-Length': '0'} 2023-11-02T18:30:34.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc62d6f748963c97-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.938 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:34.938 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:34.939 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:34.939 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:34.939 DEBUG:boto:Method: GET 2023-11-02T18:30:34.940 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:34.940 DEBUG:boto:Data: 2023-11-02T18:30:34.941 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.941 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:34.942 DEBUG:boto:Port: 8000 2023-11-02T18:30:34.942 DEBUG:boto:Params: {} 2023-11-02T18:30:34.943 DEBUG:boto:Token: None 2023-11-02T18:30:34.943 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:34.944 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J5b1nZB14CHp5KThxC1Lc0pRhqs= 2023-11-02T18:30:34.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J5b1nZB14CHp5KThxC1Lc0pRhqs=', 'Content-Length': '0'} 2023-11-02T18:30:34.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028cc17cc3b2e7c0-006543eaca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:34.991 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:34.992 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:34.992 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:34.993 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:34.993 DEBUG:boto:Method: HEAD 2023-11-02T18:30:34.993 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:34.994 DEBUG:boto:Data: 2023-11-02T18:30:34.994 DEBUG:boto:Headers: {} 2023-11-02T18:30:34.995 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:34.995 DEBUG:boto:Port: 8001 2023-11-02T18:30:34.995 DEBUG:boto:Params: {} 2023-11-02T18:30:34.996 DEBUG:boto:Token: None 2023-11-02T18:30:34.996 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:34 GMT /ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:34.997 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:70QvEm7l8pxiNi5MEILMDQ3TTIo= 2023-11-02T18:30:34.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:70QvEm7l8pxiNi5MEILMDQ3TTIo=', 'Content-Length': '0'} 2023-11-02T18:30:35.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9d885fee52bfab6-006543eaca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.001 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.001 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.002 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:35.006 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:35.008 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.008 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:35.008 DEBUG:boto:Data: 2023-11-02T18:30:35.009 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.009 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.010 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.010 DEBUG:boto:Params: {} 2023-11-02T18:30:35.011 DEBUG:boto:Token: None 2023-11-02T18:30:35.011 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:35.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzfEz2WNqIXb9gOOi0iYUJgHgUE= 2023-11-02T18:30:35.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzfEz2WNqIXb9gOOi0iYUJgHgUE=', 'Content-Length': '0'} 2023-11-02T18:30:35.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d06214be6cfae0cd-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-11-02T18:30:35.016 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.016 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.017 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:35.017 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:35.018 DEBUG:boto:Method: GET 2023-11-02T18:30:35.018 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:35.019 DEBUG:boto:Data: 2023-11-02T18:30:35.019 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.020 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.020 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.021 DEBUG:boto:Params: {} 2023-11-02T18:30:35.021 DEBUG:boto:Token: None 2023-11-02T18:30:35.022 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:35.022 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7I8HatYCQ8/mBLSVdmpBwDUYPjs= 2023-11-02T18:30:35.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7I8HatYCQ8/mBLSVdmpBwDUYPjs=', 'Content-Length': '0'} 2023-11-02T18:30:35.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004459f9dfcce0e5cf-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.026 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.027 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.027 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:35.028 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:35.028 DEBUG:boto:Method: GET 2023-11-02T18:30:35.028 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:35.029 DEBUG:boto:Data: 2023-11-02T18:30:35.029 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.030 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.031 DEBUG:boto:Params: {} 2023-11-02T18:30:35.031 DEBUG:boto:Token: None 2023-11-02T18:30:35.032 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000039?versionId=null 2023-11-02T18:30:35.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7I8HatYCQ8/mBLSVdmpBwDUYPjs= 2023-11-02T18:30:35.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7I8HatYCQ8/mBLSVdmpBwDUYPjs=', 'Content-Length': '0'} 2023-11-02T18:30:35.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e04de326b254b6-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.036 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.037 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.037 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000039 2023-11-02T18:30:35.038 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000039 2023-11-02T18:30:35.038 DEBUG:boto:Method: GET 2023-11-02T18:30:35.039 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000039 2023-11-02T18:30:35.039 DEBUG:boto:Data: 2023-11-02T18:30:35.039 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.040 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.040 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.041 DEBUG:boto:Params: {} 2023-11-02T18:30:35.041 DEBUG:boto:Token: None 2023-11-02T18:30:35.042 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.042 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e26fc94ea8571697-006543eacb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.051 DEBUG:boto:b'ibtljn-58obj000000000000000000391000falseobj000000000000000000392023-11-02T18:30:20.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.051 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.052 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.052 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000039 2023-11-02T18:30:35.053 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000039 2023-11-02T18:30:35.053 DEBUG:boto:Method: GET 2023-11-02T18:30:35.053 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000039 2023-11-02T18:30:35.054 DEBUG:boto:Data: 2023-11-02T18:30:35.054 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.055 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.055 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.056 DEBUG:boto:Params: {} 2023-11-02T18:30:35.056 DEBUG:boto:Token: None 2023-11-02T18:30:35.056 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.057 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f7b86c1afab508e-006543eacb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.106 DEBUG:boto:b'ibtljn-58obj000000000000000000391000falseobj000000000000000000392023-11-02T18:30:20.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.107 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.108 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.108 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.109 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.109 DEBUG:boto:Data: 2023-11-02T18:30:35.109 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.110 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.110 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.111 DEBUG:boto:Params: {} 2023-11-02T18:30:35.111 DEBUG:boto:Token: None 2023-11-02T18:30:35.112 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.112 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:loG3EYRc1lnQjGUfSdMTd5OmEKU= 2023-11-02T18:30:35.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:loG3EYRc1lnQjGUfSdMTd5OmEKU=', 'Content-Length': '0'} 2023-11-02T18:30:35.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dda68ee4cd47f10a-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.116 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.117 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.117 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.117 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.118 DEBUG:boto:Data: 2023-11-02T18:30:35.118 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.119 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.119 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.120 DEBUG:boto:Params: {} 2023-11-02T18:30:35.120 DEBUG:boto:Token: None 2023-11-02T18:30:35.120 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.121 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:loG3EYRc1lnQjGUfSdMTd5OmEKU= 2023-11-02T18:30:35.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:loG3EYRc1lnQjGUfSdMTd5OmEKU=', 'Content-Length': '0'} 2023-11-02T18:30:35.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e92224b7dcadceb-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-11-02T18:30:35.125 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.126 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.126 DEBUG:boto:Method: GET 2023-11-02T18:30:35.127 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.127 DEBUG:boto:Data: 2023-11-02T18:30:35.128 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.128 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.128 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.129 DEBUG:boto:Params: {} 2023-11-02T18:30:35.129 DEBUG:boto:Token: None 2023-11-02T18:30:35.130 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DehNM3LTn2OSR6XmFfIpWFNpNRE= 2023-11-02T18:30:35.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DehNM3LTn2OSR6XmFfIpWFNpNRE=', 'Content-Length': '0'} 2023-11-02T18:30:35.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceaa9d060baeda35-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.135 DEBUG:boto:path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.135 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.136 DEBUG:boto:Method: GET 2023-11-02T18:30:35.136 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.137 DEBUG:boto:Data: 2023-11-02T18:30:35.137 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.137 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.138 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.138 DEBUG:boto:Params: {} 2023-11-02T18:30:35.139 DEBUG:boto:Token: None 2023-11-02T18:30:35.139 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000039 2023-11-02T18:30:35.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DehNM3LTn2OSR6XmFfIpWFNpNRE= 2023-11-02T18:30:35.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DehNM3LTn2OSR6XmFfIpWFNpNRE=', 'Content-Length': '0'} 2023-11-02T18:30:35.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8b2d91c28cfc681-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.144 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.144 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.145 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000039 2023-11-02T18:30:35.145 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000039 2023-11-02T18:30:35.146 DEBUG:boto:Method: GET 2023-11-02T18:30:35.146 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000039 2023-11-02T18:30:35.146 DEBUG:boto:Data: 2023-11-02T18:30:35.147 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.147 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.148 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.148 DEBUG:boto:Params: {} 2023-11-02T18:30:35.149 DEBUG:boto:Token: None 2023-11-02T18:30:35.149 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.150 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cebe9aac4c425feb-006543eacb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.202 DEBUG:boto:b'ibtljn-58obj000000000000000000391000falseobj000000000000000000392023-11-02T18:30:20.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.203 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.204 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.204 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000039 2023-11-02T18:30:35.205 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000039 2023-11-02T18:30:35.205 DEBUG:boto:Method: GET 2023-11-02T18:30:35.205 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000039 2023-11-02T18:30:35.206 DEBUG:boto:Data: 2023-11-02T18:30:35.206 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.207 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.207 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.208 DEBUG:boto:Params: {} 2023-11-02T18:30:35.208 DEBUG:boto:Token: None 2023-11-02T18:30:35.209 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.209 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039dfee36d1a1bf8f-006543eacb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.224 DEBUG:boto:b'ibtljn-58obj000000000000000000391000falseobj000000000000000000392023-11-02T18:30:20.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-11-02T18:30:35.226 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.226 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.227 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.227 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.227 DEBUG:boto:Method: GET 2023-11-02T18:30:35.228 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.228 DEBUG:boto:Data: 2023-11-02T18:30:35.229 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.229 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.230 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.230 DEBUG:boto:Params: {} 2023-11-02T18:30:35.231 DEBUG:boto:Token: None 2023-11-02T18:30:35.231 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.232 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4J5JZbmCXsGh7GYOUvTsdJHXxIc= 2023-11-02T18:30:35.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4J5JZbmCXsGh7GYOUvTsdJHXxIc=', 'Content-Length': '0'} 2023-11-02T18:30:35.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3f3eac3d5a6ee6e-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.235 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.236 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.236 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.237 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.237 DEBUG:boto:Method: GET 2023-11-02T18:30:35.238 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.238 DEBUG:boto:Data: 2023-11-02T18:30:35.239 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.239 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.239 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.240 DEBUG:boto:Params: {} 2023-11-02T18:30:35.240 DEBUG:boto:Token: None 2023-11-02T18:30:35.241 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.241 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4J5JZbmCXsGh7GYOUvTsdJHXxIc= 2023-11-02T18:30:35.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4J5JZbmCXsGh7GYOUvTsdJHXxIc=', 'Content-Length': '0'} 2023-11-02T18:30:35.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ba857dfa3a26b5d-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.245 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.245 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.246 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.246 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.246 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.247 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.247 DEBUG:boto:Data: 2023-11-02T18:30:35.248 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.248 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.249 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.249 DEBUG:boto:Params: {} 2023-11-02T18:30:35.249 DEBUG:boto:Token: None 2023-11-02T18:30:35.250 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.250 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hyi+ON+o3OwIbuyA8A6jxVyQYG0= 2023-11-02T18:30:35.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hyi+ON+o3OwIbuyA8A6jxVyQYG0=', 'Content-Length': '0'} 2023-11-02T18:30:35.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ed63a23c9af04c0-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.254 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.254 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.255 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.255 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.256 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.256 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.256 DEBUG:boto:Data: 2023-11-02T18:30:35.257 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.257 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.258 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.258 DEBUG:boto:Params: {} 2023-11-02T18:30:35.259 DEBUG:boto:Token: None 2023-11-02T18:30:35.259 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.260 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hyi+ON+o3OwIbuyA8A6jxVyQYG0= 2023-11-02T18:30:35.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hyi+ON+o3OwIbuyA8A6jxVyQYG0=', 'Content-Length': '0'} 2023-11-02T18:30:35.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ce026866c88f01-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-11-02T18:30:35.264 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.264 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.265 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.265 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.266 DEBUG:boto:Method: GET 2023-11-02T18:30:35.266 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.267 DEBUG:boto:Data: 2023-11-02T18:30:35.267 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.267 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.268 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.268 DEBUG:boto:Params: {} 2023-11-02T18:30:35.269 DEBUG:boto:Token: None 2023-11-02T18:30:35.269 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.270 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4J5JZbmCXsGh7GYOUvTsdJHXxIc= 2023-11-02T18:30:35.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4J5JZbmCXsGh7GYOUvTsdJHXxIc=', 'Content-Length': '0'} 2023-11-02T18:30:35.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f2e9c893d94b26a-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.274 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.274 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.275 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.275 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.275 DEBUG:boto:Method: GET 2023-11-02T18:30:35.276 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.276 DEBUG:boto:Data: 2023-11-02T18:30:35.277 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.277 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.278 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.278 DEBUG:boto:Params: {} 2023-11-02T18:30:35.279 DEBUG:boto:Token: None 2023-11-02T18:30:35.279 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000040?versionId=null 2023-11-02T18:30:35.280 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4J5JZbmCXsGh7GYOUvTsdJHXxIc= 2023-11-02T18:30:35.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4J5JZbmCXsGh7GYOUvTsdJHXxIc=', 'Content-Length': '0'} 2023-11-02T18:30:35.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ca156cd5636e1b-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.326 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.327 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.327 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000040 2023-11-02T18:30:35.328 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000040 2023-11-02T18:30:35.328 DEBUG:boto:Method: GET 2023-11-02T18:30:35.329 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000040 2023-11-02T18:30:35.329 DEBUG:boto:Data: 2023-11-02T18:30:35.330 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.330 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.330 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.331 DEBUG:boto:Params: {} 2023-11-02T18:30:35.331 DEBUG:boto:Token: None 2023-11-02T18:30:35.332 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.332 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b7714227fba295a-006543eacb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.338 DEBUG:boto:b'ibtljn-58obj000000000000000000401000falseobj000000000000000000402023-11-02T18:30:20.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.339 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.339 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.340 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000040 2023-11-02T18:30:35.340 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000040 2023-11-02T18:30:35.340 DEBUG:boto:Method: GET 2023-11-02T18:30:35.341 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000040 2023-11-02T18:30:35.341 DEBUG:boto:Data: 2023-11-02T18:30:35.342 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.342 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.343 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.343 DEBUG:boto:Params: {} 2023-11-02T18:30:35.344 DEBUG:boto:Token: None 2023-11-02T18:30:35.344 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.344 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9f241371768ba68-006543eacb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.355 DEBUG:boto:b'ibtljn-58obj000000000000000000401000falseobj000000000000000000402023-11-02T18:30:20.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.355 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.356 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.356 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.357 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.357 DEBUG:boto:Data: 2023-11-02T18:30:35.358 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.359 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.359 DEBUG:boto:Params: {} 2023-11-02T18:30:35.359 DEBUG:boto:Token: None 2023-11-02T18:30:35.360 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.360 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hmPy+e2vpWdFJv3ow6ywSasMnug= 2023-11-02T18:30:35.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hmPy+e2vpWdFJv3ow6ywSasMnug=', 'Content-Length': '0'} 2023-11-02T18:30:35.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcc3d6d5078e8b81-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.364 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.364 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.364 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.365 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.365 DEBUG:boto:Data: 2023-11-02T18:30:35.366 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.367 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.367 DEBUG:boto:Params: {} 2023-11-02T18:30:35.367 DEBUG:boto:Token: None 2023-11-02T18:30:35.368 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.368 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hmPy+e2vpWdFJv3ow6ywSasMnug= 2023-11-02T18:30:35.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hmPy+e2vpWdFJv3ow6ywSasMnug=', 'Content-Length': '0'} 2023-11-02T18:30:35.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000411c4cd44d81014-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-11-02T18:30:35.372 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.373 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.373 DEBUG:boto:Method: GET 2023-11-02T18:30:35.374 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.374 DEBUG:boto:Data: 2023-11-02T18:30:35.374 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.375 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.375 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.376 DEBUG:boto:Params: {} 2023-11-02T18:30:35.376 DEBUG:boto:Token: None 2023-11-02T18:30:35.377 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.377 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qt6GbWQnBuNnFFwu0PMehdPVOH4= 2023-11-02T18:30:35.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qt6GbWQnBuNnFFwu0PMehdPVOH4=', 'Content-Length': '0'} 2023-11-02T18:30:35.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e9c6c1724890f68-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.422 DEBUG:boto:path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.423 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.424 DEBUG:boto:Method: GET 2023-11-02T18:30:35.424 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.424 DEBUG:boto:Data: 2023-11-02T18:30:35.425 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.425 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.426 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.426 DEBUG:boto:Params: {} 2023-11-02T18:30:35.427 DEBUG:boto:Token: None 2023-11-02T18:30:35.427 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000040 2023-11-02T18:30:35.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qt6GbWQnBuNnFFwu0PMehdPVOH4= 2023-11-02T18:30:35.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qt6GbWQnBuNnFFwu0PMehdPVOH4=', 'Content-Length': '0'} 2023-11-02T18:30:35.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e51d86da56a4124-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.432 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.432 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.433 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000040 2023-11-02T18:30:35.433 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000040 2023-11-02T18:30:35.433 DEBUG:boto:Method: GET 2023-11-02T18:30:35.434 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000040 2023-11-02T18:30:35.434 DEBUG:boto:Data: 2023-11-02T18:30:35.435 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.436 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.436 DEBUG:boto:Params: {} 2023-11-02T18:30:35.437 DEBUG:boto:Token: None 2023-11-02T18:30:35.437 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.438 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000902b3059c63c6770-006543eacb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.443 DEBUG:boto:b'ibtljn-58obj000000000000000000401000falseobj000000000000000000402023-11-02T18:30:20.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.444 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.445 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.445 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000040 2023-11-02T18:30:35.446 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000040 2023-11-02T18:30:35.446 DEBUG:boto:Method: GET 2023-11-02T18:30:35.447 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000040 2023-11-02T18:30:35.447 DEBUG:boto:Data: 2023-11-02T18:30:35.447 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.448 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.448 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.449 DEBUG:boto:Params: {} 2023-11-02T18:30:35.449 DEBUG:boto:Token: None 2023-11-02T18:30:35.450 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.450 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9a459f0a38bd2cf-006543eacb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.502 DEBUG:boto:b'ibtljn-58obj000000000000000000401000falseobj000000000000000000402023-11-02T18:30:20.099Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-11-02T18:30:35.504 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.504 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.505 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.505 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.505 DEBUG:boto:Method: GET 2023-11-02T18:30:35.506 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.506 DEBUG:boto:Data: 2023-11-02T18:30:35.507 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.507 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.508 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.508 DEBUG:boto:Params: {} 2023-11-02T18:30:35.509 DEBUG:boto:Token: None 2023-11-02T18:30:35.509 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.510 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tvyeGlgJNW8lIAlASv+ZfefQuDM= 2023-11-02T18:30:35.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tvyeGlgJNW8lIAlASv+ZfefQuDM=', 'Content-Length': '0'} 2023-11-02T18:30:35.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ffeb6b1b71e5855-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.514 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.515 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.515 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.516 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.516 DEBUG:boto:Method: GET 2023-11-02T18:30:35.517 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.517 DEBUG:boto:Data: 2023-11-02T18:30:35.518 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.518 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.519 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.519 DEBUG:boto:Params: {} 2023-11-02T18:30:35.520 DEBUG:boto:Token: None 2023-11-02T18:30:35.520 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tvyeGlgJNW8lIAlASv+ZfefQuDM= 2023-11-02T18:30:35.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tvyeGlgJNW8lIAlASv+ZfefQuDM=', 'Content-Length': '0'} 2023-11-02T18:30:35.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a5640f0c02a7554-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.525 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.525 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.526 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.526 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.527 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.527 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.527 DEBUG:boto:Data: 2023-11-02T18:30:35.528 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.528 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.529 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.529 DEBUG:boto:Params: {} 2023-11-02T18:30:35.530 DEBUG:boto:Token: None 2023-11-02T18:30:35.530 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.530 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lhCBzsCl/cSWUPewYIITTtoi4IE= 2023-11-02T18:30:35.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lhCBzsCl/cSWUPewYIITTtoi4IE=', 'Content-Length': '0'} 2023-11-02T18:30:35.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ec2c4b30ef724c7-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.535 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.535 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.535 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.536 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.536 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.537 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.537 DEBUG:boto:Data: 2023-11-02T18:30:35.538 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.538 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.538 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.539 DEBUG:boto:Params: {} 2023-11-02T18:30:35.539 DEBUG:boto:Token: None 2023-11-02T18:30:35.540 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.540 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lhCBzsCl/cSWUPewYIITTtoi4IE= 2023-11-02T18:30:35.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lhCBzsCl/cSWUPewYIITTtoi4IE=', 'Content-Length': '0'} 2023-11-02T18:30:35.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eff62d825676b77f-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-11-02T18:30:35.545 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.545 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.546 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.546 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.546 DEBUG:boto:Method: GET 2023-11-02T18:30:35.547 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.547 DEBUG:boto:Data: 2023-11-02T18:30:35.548 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.548 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.549 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.549 DEBUG:boto:Params: {} 2023-11-02T18:30:35.550 DEBUG:boto:Token: None 2023-11-02T18:30:35.550 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.551 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tvyeGlgJNW8lIAlASv+ZfefQuDM= 2023-11-02T18:30:35.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tvyeGlgJNW8lIAlASv+ZfefQuDM=', 'Content-Length': '0'} 2023-11-02T18:30:35.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1092b67dde08d70-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.555 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.555 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.556 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.556 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.557 DEBUG:boto:Method: GET 2023-11-02T18:30:35.557 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.558 DEBUG:boto:Data: 2023-11-02T18:30:35.558 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.559 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.559 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.559 DEBUG:boto:Params: {} 2023-11-02T18:30:35.560 DEBUG:boto:Token: None 2023-11-02T18:30:35.560 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000042?versionId=null 2023-11-02T18:30:35.561 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tvyeGlgJNW8lIAlASv+ZfefQuDM= 2023-11-02T18:30:35.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tvyeGlgJNW8lIAlASv+ZfefQuDM=', 'Content-Length': '0'} 2023-11-02T18:30:35.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e4e517f9659d8cd-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.606 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.607 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.607 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000042 2023-11-02T18:30:35.608 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000042 2023-11-02T18:30:35.608 DEBUG:boto:Method: GET 2023-11-02T18:30:35.609 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000042 2023-11-02T18:30:35.609 DEBUG:boto:Data: 2023-11-02T18:30:35.610 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.610 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.610 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.611 DEBUG:boto:Params: {} 2023-11-02T18:30:35.611 DEBUG:boto:Token: None 2023-11-02T18:30:35.612 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060641beb973b3fb0-006543eacb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.619 DEBUG:boto:b'ibtljn-58obj000000000000000000421000falseobj000000000000000000422023-11-02T18:30:16.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.620 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.620 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.621 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000042 2023-11-02T18:30:35.621 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000042 2023-11-02T18:30:35.622 DEBUG:boto:Method: GET 2023-11-02T18:30:35.622 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000042 2023-11-02T18:30:35.623 DEBUG:boto:Data: 2023-11-02T18:30:35.623 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.624 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.624 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.624 DEBUG:boto:Params: {} 2023-11-02T18:30:35.625 DEBUG:boto:Token: None 2023-11-02T18:30:35.625 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da1f940e35e7a7eb-006543eacb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.636 DEBUG:boto:b'ibtljn-58obj000000000000000000421000falseobj000000000000000000422023-11-02T18:30:16.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.636 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.637 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.637 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.638 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.638 DEBUG:boto:Data: 2023-11-02T18:30:35.639 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.639 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.640 DEBUG:boto:Params: {} 2023-11-02T18:30:35.640 DEBUG:boto:Token: None 2023-11-02T18:30:35.641 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ctq++aZeSv+ig3rc9TpAtp/ZjsE= 2023-11-02T18:30:35.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ctq++aZeSv+ig3rc9TpAtp/ZjsE=', 'Content-Length': '0'} 2023-11-02T18:30:35.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c10e034179ba8a4c-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.645 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.645 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.646 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.646 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.647 DEBUG:boto:Data: 2023-11-02T18:30:35.647 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.648 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.648 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.648 DEBUG:boto:Params: {} 2023-11-02T18:30:35.649 DEBUG:boto:Token: None 2023-11-02T18:30:35.649 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.650 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ctq++aZeSv+ig3rc9TpAtp/ZjsE= 2023-11-02T18:30:35.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ctq++aZeSv+ig3rc9TpAtp/ZjsE=', 'Content-Length': '0'} 2023-11-02T18:30:35.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d02613e3a738762-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-11-02T18:30:35.654 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.654 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.655 DEBUG:boto:Method: GET 2023-11-02T18:30:35.655 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.656 DEBUG:boto:Data: 2023-11-02T18:30:35.656 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.657 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.657 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.658 DEBUG:boto:Params: {} 2023-11-02T18:30:35.658 DEBUG:boto:Token: None 2023-11-02T18:30:35.659 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.659 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8zNZH6os7iWIPG3MsOJYB8rayZw= 2023-11-02T18:30:35.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8zNZH6os7iWIPG3MsOJYB8rayZw=', 'Content-Length': '0'} 2023-11-02T18:30:35.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076aae8e2c2953c95-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.663 DEBUG:boto:path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.664 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.664 DEBUG:boto:Method: GET 2023-11-02T18:30:35.665 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.665 DEBUG:boto:Data: 2023-11-02T18:30:35.665 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.666 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.666 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.667 DEBUG:boto:Params: {} 2023-11-02T18:30:35.667 DEBUG:boto:Token: None 2023-11-02T18:30:35.668 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000042 2023-11-02T18:30:35.668 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8zNZH6os7iWIPG3MsOJYB8rayZw= 2023-11-02T18:30:35.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8zNZH6os7iWIPG3MsOJYB8rayZw=', 'Content-Length': '0'} 2023-11-02T18:30:35.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eef2d5c355a8827-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.672 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.673 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.673 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000042 2023-11-02T18:30:35.674 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000042 2023-11-02T18:30:35.674 DEBUG:boto:Method: GET 2023-11-02T18:30:35.674 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000042 2023-11-02T18:30:35.675 DEBUG:boto:Data: 2023-11-02T18:30:35.675 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.676 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.676 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.677 DEBUG:boto:Params: {} 2023-11-02T18:30:35.677 DEBUG:boto:Token: None 2023-11-02T18:30:35.678 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.678 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000103825dd6b9b9cd0-006543eacb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.730 DEBUG:boto:b'ibtljn-58obj000000000000000000421000falseobj000000000000000000422023-11-02T18:30:16.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.731 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.732 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.732 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000042 2023-11-02T18:30:35.732 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000042 2023-11-02T18:30:35.733 DEBUG:boto:Method: GET 2023-11-02T18:30:35.733 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000042 2023-11-02T18:30:35.734 DEBUG:boto:Data: 2023-11-02T18:30:35.734 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.735 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.735 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.736 DEBUG:boto:Params: {} 2023-11-02T18:30:35.736 DEBUG:boto:Token: None 2023-11-02T18:30:35.737 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.737 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a52aa7b892328e8-006543eacb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.747 DEBUG:boto:b'ibtljn-58obj000000000000000000421000falseobj000000000000000000422023-11-02T18:30:16.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-11-02T18:30:35.748 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.749 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.749 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.750 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.750 DEBUG:boto:Method: GET 2023-11-02T18:30:35.751 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.751 DEBUG:boto:Data: 2023-11-02T18:30:35.752 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.752 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.753 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.753 DEBUG:boto:Params: {} 2023-11-02T18:30:35.754 DEBUG:boto:Token: None 2023-11-02T18:30:35.754 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.755 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FXmiupGWh53b9nlmnPXuUl6CJJo= 2023-11-02T18:30:35.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FXmiupGWh53b9nlmnPXuUl6CJJo=', 'Content-Length': '0'} 2023-11-02T18:30:35.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fd21c299727bf8f-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.759 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.759 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.760 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.760 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.760 DEBUG:boto:Method: GET 2023-11-02T18:30:35.761 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.761 DEBUG:boto:Data: 2023-11-02T18:30:35.762 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.762 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.763 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.763 DEBUG:boto:Params: {} 2023-11-02T18:30:35.764 DEBUG:boto:Token: None 2023-11-02T18:30:35.764 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.765 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FXmiupGWh53b9nlmnPXuUl6CJJo= 2023-11-02T18:30:35.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FXmiupGWh53b9nlmnPXuUl6CJJo=', 'Content-Length': '0'} 2023-11-02T18:30:35.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014e77452fb6e7015-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.810 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.811 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.811 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.812 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.812 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.813 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.813 DEBUG:boto:Data: 2023-11-02T18:30:35.814 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.814 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.815 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.815 DEBUG:boto:Params: {} 2023-11-02T18:30:35.816 DEBUG:boto:Token: None 2023-11-02T18:30:35.816 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.817 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FbD4oUfiowUxPX4ytFQz1x5QN+M= 2023-11-02T18:30:35.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FbD4oUfiowUxPX4ytFQz1x5QN+M=', 'Content-Length': '0'} 2023-11-02T18:30:35.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004bbc66e74d22df2-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.821 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.821 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.822 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.822 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.823 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.823 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.824 DEBUG:boto:Data: 2023-11-02T18:30:35.824 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.824 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.825 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.825 DEBUG:boto:Params: {} 2023-11-02T18:30:35.826 DEBUG:boto:Token: None 2023-11-02T18:30:35.826 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.827 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FbD4oUfiowUxPX4ytFQz1x5QN+M= 2023-11-02T18:30:35.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FbD4oUfiowUxPX4ytFQz1x5QN+M=', 'Content-Length': '0'} 2023-11-02T18:30:35.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb955d96c8147250-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-11-02T18:30:35.831 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.832 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.832 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.833 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.833 DEBUG:boto:Method: GET 2023-11-02T18:30:35.834 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.834 DEBUG:boto:Data: 2023-11-02T18:30:35.835 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.835 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.836 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.836 DEBUG:boto:Params: {} 2023-11-02T18:30:35.837 DEBUG:boto:Token: None 2023-11-02T18:30:35.838 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.838 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FXmiupGWh53b9nlmnPXuUl6CJJo= 2023-11-02T18:30:35.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FXmiupGWh53b9nlmnPXuUl6CJJo=', 'Content-Length': '0'} 2023-11-02T18:30:35.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000872525cb9316657d-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.843 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.843 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.844 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.844 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.844 DEBUG:boto:Method: GET 2023-11-02T18:30:35.845 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.845 DEBUG:boto:Data: 2023-11-02T18:30:35.846 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.846 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.847 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.847 DEBUG:boto:Params: {} 2023-11-02T18:30:35.848 DEBUG:boto:Token: None 2023-11-02T18:30:35.848 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000045?versionId=null 2023-11-02T18:30:35.849 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FXmiupGWh53b9nlmnPXuUl6CJJo= 2023-11-02T18:30:35.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FXmiupGWh53b9nlmnPXuUl6CJJo=', 'Content-Length': '0'} 2023-11-02T18:30:35.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068b4c9f04799e304-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.853 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.854 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.854 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000045 2023-11-02T18:30:35.855 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000045 2023-11-02T18:30:35.855 DEBUG:boto:Method: GET 2023-11-02T18:30:35.856 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000045 2023-11-02T18:30:35.856 DEBUG:boto:Data: 2023-11-02T18:30:35.857 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.857 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.858 DEBUG:boto:Params: {} 2023-11-02T18:30:35.858 DEBUG:boto:Token: None 2023-11-02T18:30:35.859 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.859 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038f91e29f272f3e0-006543eacb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.865 DEBUG:boto:b'ibtljn-58obj000000000000000000451000falseobj000000000000000000452023-11-02T18:30:20.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.866 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.866 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.867 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000045 2023-11-02T18:30:35.867 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000045 2023-11-02T18:30:35.868 DEBUG:boto:Method: GET 2023-11-02T18:30:35.868 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000045 2023-11-02T18:30:35.869 DEBUG:boto:Data: 2023-11-02T18:30:35.869 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.870 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.871 DEBUG:boto:Params: {} 2023-11-02T18:30:35.871 DEBUG:boto:Token: None 2023-11-02T18:30:35.872 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.872 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000295bcc586e1b08cd-006543eacb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.884 DEBUG:boto:b'ibtljn-58obj000000000000000000451000falseobj000000000000000000452023-11-02T18:30:20.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.885 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.885 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.886 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.886 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.887 DEBUG:boto:Data: 2023-11-02T18:30:35.887 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.887 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.888 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.888 DEBUG:boto:Params: {} 2023-11-02T18:30:35.889 DEBUG:boto:Token: None 2023-11-02T18:30:35.889 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.890 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JDv7Rfi+3ndiMTpB3xP+qaTA5x4= 2023-11-02T18:30:35.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JDv7Rfi+3ndiMTpB3xP+qaTA5x4=', 'Content-Length': '0'} 2023-11-02T18:30:35.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000757318ac58e72814-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.893 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.894 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.894 DEBUG:boto:Method: HEAD 2023-11-02T18:30:35.895 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.895 DEBUG:boto:Data: 2023-11-02T18:30:35.896 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.896 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.897 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.897 DEBUG:boto:Params: {} 2023-11-02T18:30:35.897 DEBUG:boto:Token: None 2023-11-02T18:30:35.898 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.898 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JDv7Rfi+3ndiMTpB3xP+qaTA5x4= 2023-11-02T18:30:35.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JDv7Rfi+3ndiMTpB3xP+qaTA5x4=', 'Content-Length': '0'} 2023-11-02T18:30:35.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028797218d87e3164-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-11-02T18:30:35.902 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.903 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.903 DEBUG:boto:Method: GET 2023-11-02T18:30:35.904 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.904 DEBUG:boto:Data: 2023-11-02T18:30:35.905 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.905 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.905 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.906 DEBUG:boto:Params: {} 2023-11-02T18:30:35.907 DEBUG:boto:Token: None 2023-11-02T18:30:35.907 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.907 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lVeAo/JyXVgRGXIm8IP5h3g5nDQ= 2023-11-02T18:30:35.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lVeAo/JyXVgRGXIm8IP5h3g5nDQ=', 'Content-Length': '0'} 2023-11-02T18:30:35.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000345c63aa69b816a-006543eacb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.954 DEBUG:boto:path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.955 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.955 DEBUG:boto:Method: GET 2023-11-02T18:30:35.956 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.956 DEBUG:boto:Data: 2023-11-02T18:30:35.957 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.957 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.958 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.958 DEBUG:boto:Params: {} 2023-11-02T18:30:35.959 DEBUG:boto:Token: None 2023-11-02T18:30:35.959 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/obj00000000000000000045 2023-11-02T18:30:35.960 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lVeAo/JyXVgRGXIm8IP5h3g5nDQ= 2023-11-02T18:30:35.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lVeAo/JyXVgRGXIm8IP5h3g5nDQ=', 'Content-Length': '0'} 2023-11-02T18:30:35.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040ede813028817b0-006543eacb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.964 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.964 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.965 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000045 2023-11-02T18:30:35.965 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000045 2023-11-02T18:30:35.966 DEBUG:boto:Method: GET 2023-11-02T18:30:35.966 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000045 2023-11-02T18:30:35.967 DEBUG:boto:Data: 2023-11-02T18:30:35.967 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.968 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:35.968 DEBUG:boto:Port: 8001 2023-11-02T18:30:35.968 DEBUG:boto:Params: {} 2023-11-02T18:30:35.969 DEBUG:boto:Token: None 2023-11-02T18:30:35.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.970 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000922157404d4b9b94-006543eacb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:35.977 DEBUG:boto:b'ibtljn-58obj000000000000000000451000falseobj000000000000000000452023-11-02T18:30:20.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:35.977 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:35.978 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:35.978 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000045 2023-11-02T18:30:35.979 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000045 2023-11-02T18:30:35.979 DEBUG:boto:Method: GET 2023-11-02T18:30:35.980 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000045 2023-11-02T18:30:35.980 DEBUG:boto:Data: 2023-11-02T18:30:35.981 DEBUG:boto:Headers: {} 2023-11-02T18:30:35.981 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:35.981 DEBUG:boto:Port: 8000 2023-11-02T18:30:35.982 DEBUG:boto:Params: {} 2023-11-02T18:30:35.982 DEBUG:boto:Token: None 2023-11-02T18:30:35.983 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:35 GMT /ibtljn-58/ 2023-11-02T18:30:35.983 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g= 2023-11-02T18:30:35.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jYeX/Jrxc7IMomnGojWQzBzL29g=', 'Content-Length': '0'} 2023-11-02T18:30:35.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ec14f66cf9e7a5-006543eacb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.034 DEBUG:boto:b'ibtljn-58obj000000000000000000451000falseobj000000000000000000452023-11-02T18:30:20.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-11-02T18:30:36.036 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.036 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.037 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.037 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.038 DEBUG:boto:Method: GET 2023-11-02T18:30:36.038 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.039 DEBUG:boto:Data: 2023-11-02T18:30:36.039 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.039 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.040 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.040 DEBUG:boto:Params: {} 2023-11-02T18:30:36.041 DEBUG:boto:Token: None 2023-11-02T18:30:36.041 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.042 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pDL7P3i7EaUJ0oR+dFVhu1B2CRY= 2023-11-02T18:30:36.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pDL7P3i7EaUJ0oR+dFVhu1B2CRY=', 'Content-Length': '0'} 2023-11-02T18:30:36.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dacfec8fa63d75f-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.047 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.047 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.048 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.048 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.049 DEBUG:boto:Method: GET 2023-11-02T18:30:36.049 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.050 DEBUG:boto:Data: 2023-11-02T18:30:36.050 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.051 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.051 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.052 DEBUG:boto:Params: {} 2023-11-02T18:30:36.052 DEBUG:boto:Token: None 2023-11-02T18:30:36.053 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.053 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pDL7P3i7EaUJ0oR+dFVhu1B2CRY= 2023-11-02T18:30:36.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pDL7P3i7EaUJ0oR+dFVhu1B2CRY=', 'Content-Length': '0'} 2023-11-02T18:30:36.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd63618159a2e782-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.057 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.058 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.058 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.059 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.059 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.060 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.060 DEBUG:boto:Data: 2023-11-02T18:30:36.060 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.061 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.061 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.062 DEBUG:boto:Params: {} 2023-11-02T18:30:36.062 DEBUG:boto:Token: None 2023-11-02T18:30:36.063 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.063 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EN+duR/Vv3hXRCbBsnLTNdKQAok= 2023-11-02T18:30:36.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EN+duR/Vv3hXRCbBsnLTNdKQAok=', 'Content-Length': '0'} 2023-11-02T18:30:36.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd74f1d5db91dd81-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.067 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.068 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.068 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.069 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.069 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.070 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.070 DEBUG:boto:Data: 2023-11-02T18:30:36.070 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.071 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.071 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.072 DEBUG:boto:Params: {} 2023-11-02T18:30:36.072 DEBUG:boto:Token: None 2023-11-02T18:30:36.073 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.073 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EN+duR/Vv3hXRCbBsnLTNdKQAok= 2023-11-02T18:30:36.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EN+duR/Vv3hXRCbBsnLTNdKQAok=', 'Content-Length': '0'} 2023-11-02T18:30:36.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed5ac2340da084a1-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-11-02T18:30:36.078 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.078 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.079 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.079 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.080 DEBUG:boto:Method: GET 2023-11-02T18:30:36.080 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.081 DEBUG:boto:Data: 2023-11-02T18:30:36.081 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.082 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.082 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.083 DEBUG:boto:Params: {} 2023-11-02T18:30:36.083 DEBUG:boto:Token: None 2023-11-02T18:30:36.084 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.085 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pDL7P3i7EaUJ0oR+dFVhu1B2CRY= 2023-11-02T18:30:36.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pDL7P3i7EaUJ0oR+dFVhu1B2CRY=', 'Content-Length': '0'} 2023-11-02T18:30:36.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfad6a28e57720c6-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.089 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.090 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.090 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.091 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.091 DEBUG:boto:Method: GET 2023-11-02T18:30:36.092 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.093 DEBUG:boto:Data: 2023-11-02T18:30:36.093 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.094 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.094 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.095 DEBUG:boto:Params: {} 2023-11-02T18:30:36.095 DEBUG:boto:Token: None 2023-11-02T18:30:36.096 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000048?versionId=null 2023-11-02T18:30:36.096 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pDL7P3i7EaUJ0oR+dFVhu1B2CRY= 2023-11-02T18:30:36.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pDL7P3i7EaUJ0oR+dFVhu1B2CRY=', 'Content-Length': '0'} 2023-11-02T18:30:36.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec4600f19ebb9ecc-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.101 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.101 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.102 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000048 2023-11-02T18:30:36.102 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000048 2023-11-02T18:30:36.103 DEBUG:boto:Method: GET 2023-11-02T18:30:36.103 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000048 2023-11-02T18:30:36.104 DEBUG:boto:Data: 2023-11-02T18:30:36.104 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.104 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.105 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.105 DEBUG:boto:Params: {} 2023-11-02T18:30:36.106 DEBUG:boto:Token: None 2023-11-02T18:30:36.106 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.107 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdec2cecc1461a1a-006543eacc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.113 DEBUG:boto:b'ibtljn-58obj000000000000000000481000falseobj000000000000000000482023-11-02T18:30:19.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.114 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.114 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.115 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000048 2023-11-02T18:30:36.115 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000048 2023-11-02T18:30:36.115 DEBUG:boto:Method: GET 2023-11-02T18:30:36.116 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000048 2023-11-02T18:30:36.116 DEBUG:boto:Data: 2023-11-02T18:30:36.117 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.117 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.118 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.118 DEBUG:boto:Params: {} 2023-11-02T18:30:36.119 DEBUG:boto:Token: None 2023-11-02T18:30:36.119 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000103bf62de04f0599-006543eacc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.131 DEBUG:boto:b'ibtljn-58obj000000000000000000481000falseobj000000000000000000482023-11-02T18:30:19.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.132 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.132 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.133 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.133 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.134 DEBUG:boto:Data: 2023-11-02T18:30:36.134 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.134 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.135 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.135 DEBUG:boto:Params: {} 2023-11-02T18:30:36.136 DEBUG:boto:Token: None 2023-11-02T18:30:36.136 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.137 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fagydtpuM78tWKgY/XKg270eoek= 2023-11-02T18:30:36.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fagydtpuM78tWKgY/XKg270eoek=', 'Content-Length': '0'} 2023-11-02T18:30:36.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f472a2af5346c2-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.140 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.141 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.141 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.142 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.142 DEBUG:boto:Data: 2023-11-02T18:30:36.143 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.143 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.143 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.144 DEBUG:boto:Params: {} 2023-11-02T18:30:36.144 DEBUG:boto:Token: None 2023-11-02T18:30:36.145 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.145 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fagydtpuM78tWKgY/XKg270eoek= 2023-11-02T18:30:36.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fagydtpuM78tWKgY/XKg270eoek=', 'Content-Length': '0'} 2023-11-02T18:30:36.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fecd06ba91c4816-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-11-02T18:30:36.149 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.150 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.150 DEBUG:boto:Method: GET 2023-11-02T18:30:36.151 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.151 DEBUG:boto:Data: 2023-11-02T18:30:36.152 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.152 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.153 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.153 DEBUG:boto:Params: {} 2023-11-02T18:30:36.154 DEBUG:boto:Token: None 2023-11-02T18:30:36.154 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.155 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L1IKygqn/qyEGc3D+GUAZvPPxv0= 2023-11-02T18:30:36.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L1IKygqn/qyEGc3D+GUAZvPPxv0=', 'Content-Length': '0'} 2023-11-02T18:30:36.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc1323c148d77a9a-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.198 DEBUG:boto:path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.199 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.199 DEBUG:boto:Method: GET 2023-11-02T18:30:36.200 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.200 DEBUG:boto:Data: 2023-11-02T18:30:36.201 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.201 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.202 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.202 DEBUG:boto:Params: {} 2023-11-02T18:30:36.203 DEBUG:boto:Token: None 2023-11-02T18:30:36.204 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000048 2023-11-02T18:30:36.204 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L1IKygqn/qyEGc3D+GUAZvPPxv0= 2023-11-02T18:30:36.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L1IKygqn/qyEGc3D+GUAZvPPxv0=', 'Content-Length': '0'} 2023-11-02T18:30:36.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000317f174557a600a8-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.208 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.209 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.210 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000048 2023-11-02T18:30:36.210 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000048 2023-11-02T18:30:36.211 DEBUG:boto:Method: GET 2023-11-02T18:30:36.211 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000048 2023-11-02T18:30:36.212 DEBUG:boto:Data: 2023-11-02T18:30:36.212 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.213 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.213 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.214 DEBUG:boto:Params: {} 2023-11-02T18:30:36.215 DEBUG:boto:Token: None 2023-11-02T18:30:36.215 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.216 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1d6a9b820e8ea61-006543eacc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.222 DEBUG:boto:b'ibtljn-58obj000000000000000000481000falseobj000000000000000000482023-11-02T18:30:19.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.223 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.223 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.224 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000048 2023-11-02T18:30:36.224 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000048 2023-11-02T18:30:36.225 DEBUG:boto:Method: GET 2023-11-02T18:30:36.225 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000048 2023-11-02T18:30:36.226 DEBUG:boto:Data: 2023-11-02T18:30:36.226 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.227 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.229 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.229 DEBUG:boto:Params: {} 2023-11-02T18:30:36.230 DEBUG:boto:Token: None 2023-11-02T18:30:36.230 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.231 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012d575e4ad085684-006543eacc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.243 DEBUG:boto:b'ibtljn-58obj000000000000000000481000falseobj000000000000000000482023-11-02T18:30:19.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-11-02T18:30:36.244 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.245 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.245 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.246 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.246 DEBUG:boto:Method: GET 2023-11-02T18:30:36.247 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.247 DEBUG:boto:Data: 2023-11-02T18:30:36.248 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.248 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.249 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.249 DEBUG:boto:Params: {} 2023-11-02T18:30:36.250 DEBUG:boto:Token: None 2023-11-02T18:30:36.250 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.251 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9DDhdcLakgEHj0yhxrVwb2syQXo= 2023-11-02T18:30:36.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9DDhdcLakgEHj0yhxrVwb2syQXo=', 'Content-Length': '0'} 2023-11-02T18:30:36.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d01bd8db680867e7-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.255 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.255 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.256 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.256 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.257 DEBUG:boto:Method: GET 2023-11-02T18:30:36.257 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.258 DEBUG:boto:Data: 2023-11-02T18:30:36.259 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.260 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.260 DEBUG:boto:Params: {} 2023-11-02T18:30:36.261 DEBUG:boto:Token: None 2023-11-02T18:30:36.261 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.262 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9DDhdcLakgEHj0yhxrVwb2syQXo= 2023-11-02T18:30:36.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9DDhdcLakgEHj0yhxrVwb2syQXo=', 'Content-Length': '0'} 2023-11-02T18:30:36.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092383b630aa73524-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.266 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.266 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.267 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.267 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.268 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.268 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.268 DEBUG:boto:Data: 2023-11-02T18:30:36.269 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.269 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.270 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.270 DEBUG:boto:Params: {} 2023-11-02T18:30:36.271 DEBUG:boto:Token: None 2023-11-02T18:30:36.271 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.272 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GvyAGRjlnSHmMGNqncxN0B2IwYU= 2023-11-02T18:30:36.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GvyAGRjlnSHmMGNqncxN0B2IwYU=', 'Content-Length': '0'} 2023-11-02T18:30:36.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000645fe3ff9565558e-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.276 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.277 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.277 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.278 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.278 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.279 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.279 DEBUG:boto:Data: 2023-11-02T18:30:36.280 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.280 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.281 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.281 DEBUG:boto:Params: {} 2023-11-02T18:30:36.282 DEBUG:boto:Token: None 2023-11-02T18:30:36.282 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.283 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GvyAGRjlnSHmMGNqncxN0B2IwYU= 2023-11-02T18:30:36.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GvyAGRjlnSHmMGNqncxN0B2IwYU=', 'Content-Length': '0'} 2023-11-02T18:30:36.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005992bae4fa0e97ab-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-11-02T18:30:36.287 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.288 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.288 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.289 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.290 DEBUG:boto:Method: GET 2023-11-02T18:30:36.290 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.291 DEBUG:boto:Data: 2023-11-02T18:30:36.291 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.292 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.292 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.293 DEBUG:boto:Params: {} 2023-11-02T18:30:36.293 DEBUG:boto:Token: None 2023-11-02T18:30:36.294 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.294 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9DDhdcLakgEHj0yhxrVwb2syQXo= 2023-11-02T18:30:36.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9DDhdcLakgEHj0yhxrVwb2syQXo=', 'Content-Length': '0'} 2023-11-02T18:30:36.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f22774feb049d796-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.338 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.339 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.340 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.340 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.341 DEBUG:boto:Method: GET 2023-11-02T18:30:36.341 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.342 DEBUG:boto:Data: 2023-11-02T18:30:36.343 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.343 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.344 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.344 DEBUG:boto:Params: {} 2023-11-02T18:30:36.345 DEBUG:boto:Token: None 2023-11-02T18:30:36.345 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000049?versionId=null 2023-11-02T18:30:36.346 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9DDhdcLakgEHj0yhxrVwb2syQXo= 2023-11-02T18:30:36.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9DDhdcLakgEHj0yhxrVwb2syQXo=', 'Content-Length': '0'} 2023-11-02T18:30:36.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006debd78f757f8ac-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.351 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.352 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.352 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000049 2023-11-02T18:30:36.353 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000049 2023-11-02T18:30:36.353 DEBUG:boto:Method: GET 2023-11-02T18:30:36.354 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000049 2023-11-02T18:30:36.354 DEBUG:boto:Data: 2023-11-02T18:30:36.355 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.355 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.356 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.356 DEBUG:boto:Params: {} 2023-11-02T18:30:36.357 DEBUG:boto:Token: None 2023-11-02T18:30:36.357 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.358 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b5585f3ca4502a1-006543eacc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.365 DEBUG:boto:b'ibtljn-58obj000000000000000000491000falseobj000000000000000000492023-11-02T18:30:16.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.366 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.366 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.367 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000049 2023-11-02T18:30:36.367 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000049 2023-11-02T18:30:36.368 DEBUG:boto:Method: GET 2023-11-02T18:30:36.368 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000049 2023-11-02T18:30:36.369 DEBUG:boto:Data: 2023-11-02T18:30:36.369 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.370 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.371 DEBUG:boto:Params: {} 2023-11-02T18:30:36.371 DEBUG:boto:Token: None 2023-11-02T18:30:36.372 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076ba74bbbba3b45f-006543eacc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.426 DEBUG:boto:b'ibtljn-58obj000000000000000000491000falseobj000000000000000000492023-11-02T18:30:16.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.427 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.428 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.429 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.429 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.430 DEBUG:boto:Data: 2023-11-02T18:30:36.430 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.431 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.431 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.432 DEBUG:boto:Params: {} 2023-11-02T18:30:36.432 DEBUG:boto:Token: None 2023-11-02T18:30:36.433 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.433 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fV0rGwWJecmWB9RgRtuGuctCu00= 2023-11-02T18:30:36.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fV0rGwWJecmWB9RgRtuGuctCu00=', 'Content-Length': '0'} 2023-11-02T18:30:36.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b129a9ab87f13c2-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.438 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.439 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.439 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.440 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.440 DEBUG:boto:Data: 2023-11-02T18:30:36.441 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.441 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.442 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.442 DEBUG:boto:Params: {} 2023-11-02T18:30:36.443 DEBUG:boto:Token: None 2023-11-02T18:30:36.443 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fV0rGwWJecmWB9RgRtuGuctCu00= 2023-11-02T18:30:36.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fV0rGwWJecmWB9RgRtuGuctCu00=', 'Content-Length': '0'} 2023-11-02T18:30:36.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c996156f925d9ffe-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-11-02T18:30:36.448 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.449 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.449 DEBUG:boto:Method: GET 2023-11-02T18:30:36.450 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.450 DEBUG:boto:Data: 2023-11-02T18:30:36.451 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.451 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.451 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.452 DEBUG:boto:Params: {} 2023-11-02T18:30:36.452 DEBUG:boto:Token: None 2023-11-02T18:30:36.453 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.454 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k9ij6yf+Vvnp6nOKGX3g7CktZII= 2023-11-02T18:30:36.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k9ij6yf+Vvnp6nOKGX3g7CktZII=', 'Content-Length': '0'} 2023-11-02T18:30:36.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010c70990b36e0820-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.458 DEBUG:boto:path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.459 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.459 DEBUG:boto:Method: GET 2023-11-02T18:30:36.460 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.460 DEBUG:boto:Data: 2023-11-02T18:30:36.460 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.461 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.461 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.462 DEBUG:boto:Params: {} 2023-11-02T18:30:36.462 DEBUG:boto:Token: None 2023-11-02T18:30:36.463 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000049 2023-11-02T18:30:36.464 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:k9ij6yf+Vvnp6nOKGX3g7CktZII= 2023-11-02T18:30:36.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:k9ij6yf+Vvnp6nOKGX3g7CktZII=', 'Content-Length': '0'} 2023-11-02T18:30:36.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021281d2e9c87e03f-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.468 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.469 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.469 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000049 2023-11-02T18:30:36.470 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000049 2023-11-02T18:30:36.470 DEBUG:boto:Method: GET 2023-11-02T18:30:36.471 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000049 2023-11-02T18:30:36.471 DEBUG:boto:Data: 2023-11-02T18:30:36.472 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.472 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.473 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.473 DEBUG:boto:Params: {} 2023-11-02T18:30:36.474 DEBUG:boto:Token: None 2023-11-02T18:30:36.475 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.475 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b150359fcca32351-006543eacc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.482 DEBUG:boto:b'ibtljn-58obj000000000000000000491000falseobj000000000000000000492023-11-02T18:30:16.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.483 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.483 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.484 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000049 2023-11-02T18:30:36.484 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000049 2023-11-02T18:30:36.484 DEBUG:boto:Method: GET 2023-11-02T18:30:36.485 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000049 2023-11-02T18:30:36.485 DEBUG:boto:Data: 2023-11-02T18:30:36.486 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.486 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.487 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.487 DEBUG:boto:Params: {} 2023-11-02T18:30:36.488 DEBUG:boto:Token: None 2023-11-02T18:30:36.488 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.489 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000871293418cbb6b86-006543eacc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.542 DEBUG:boto:b'ibtljn-58obj000000000000000000491000falseobj000000000000000000492023-11-02T18:30:16.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-11-02T18:30:36.544 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.544 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.545 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.545 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.546 DEBUG:boto:Method: GET 2023-11-02T18:30:36.546 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.547 DEBUG:boto:Data: 2023-11-02T18:30:36.547 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.547 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.548 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.548 DEBUG:boto:Params: {} 2023-11-02T18:30:36.549 DEBUG:boto:Token: None 2023-11-02T18:30:36.549 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.550 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qrrq95TTbvMqy6wuDSOkVipv838= 2023-11-02T18:30:36.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qrrq95TTbvMqy6wuDSOkVipv838=', 'Content-Length': '0'} 2023-11-02T18:30:36.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9bea2dabe73f005-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.555 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.555 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.556 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.556 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.557 DEBUG:boto:Method: GET 2023-11-02T18:30:36.557 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.558 DEBUG:boto:Data: 2023-11-02T18:30:36.558 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.559 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.559 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.559 DEBUG:boto:Params: {} 2023-11-02T18:30:36.560 DEBUG:boto:Token: None 2023-11-02T18:30:36.561 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.561 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qrrq95TTbvMqy6wuDSOkVipv838= 2023-11-02T18:30:36.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qrrq95TTbvMqy6wuDSOkVipv838=', 'Content-Length': '0'} 2023-11-02T18:30:36.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e4c08e78a8a6053-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.565 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.566 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.566 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.567 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.567 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.568 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.568 DEBUG:boto:Data: 2023-11-02T18:30:36.569 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.569 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.570 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.570 DEBUG:boto:Params: {} 2023-11-02T18:30:36.571 DEBUG:boto:Token: None 2023-11-02T18:30:36.571 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.572 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2iwWGezB82A63IIYDneI7/fuM44= 2023-11-02T18:30:36.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2iwWGezB82A63IIYDneI7/fuM44=', 'Content-Length': '0'} 2023-11-02T18:30:36.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a643135f624fee30-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.576 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.576 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.576 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.577 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.577 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.578 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.578 DEBUG:boto:Data: 2023-11-02T18:30:36.579 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.579 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.579 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.580 DEBUG:boto:Params: {} 2023-11-02T18:30:36.580 DEBUG:boto:Token: None 2023-11-02T18:30:36.581 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.581 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2iwWGezB82A63IIYDneI7/fuM44= 2023-11-02T18:30:36.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2iwWGezB82A63IIYDneI7/fuM44=', 'Content-Length': '0'} 2023-11-02T18:30:36.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a3a4c1ce9ce3a87-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-11-02T18:30:36.587 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.588 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.588 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.589 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.589 DEBUG:boto:Method: GET 2023-11-02T18:30:36.590 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.591 DEBUG:boto:Data: 2023-11-02T18:30:36.591 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.592 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.592 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.592 DEBUG:boto:Params: {} 2023-11-02T18:30:36.593 DEBUG:boto:Token: None 2023-11-02T18:30:36.594 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.595 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qrrq95TTbvMqy6wuDSOkVipv838= 2023-11-02T18:30:36.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qrrq95TTbvMqy6wuDSOkVipv838=', 'Content-Length': '0'} 2023-11-02T18:30:36.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c70993b56f7231d-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.598 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.599 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.600 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.600 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.601 DEBUG:boto:Method: GET 2023-11-02T18:30:36.601 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.602 DEBUG:boto:Data: 2023-11-02T18:30:36.602 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.603 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.603 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.604 DEBUG:boto:Params: {} 2023-11-02T18:30:36.604 DEBUG:boto:Token: None 2023-11-02T18:30:36.605 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000050?versionId=null 2023-11-02T18:30:36.606 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qrrq95TTbvMqy6wuDSOkVipv838= 2023-11-02T18:30:36.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qrrq95TTbvMqy6wuDSOkVipv838=', 'Content-Length': '0'} 2023-11-02T18:30:36.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f70fb2c7730993b9-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.609 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.610 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.610 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000050 2023-11-02T18:30:36.611 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000050 2023-11-02T18:30:36.611 DEBUG:boto:Method: GET 2023-11-02T18:30:36.612 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000050 2023-11-02T18:30:36.612 DEBUG:boto:Data: 2023-11-02T18:30:36.612 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.613 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.613 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.614 DEBUG:boto:Params: {} 2023-11-02T18:30:36.614 DEBUG:boto:Token: None 2023-11-02T18:30:36.615 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.615 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a78790d5cc4fd73e-006543eacc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.621 DEBUG:boto:b'ibtljn-58obj000000000000000000501000falseobj000000000000000000502023-11-02T18:30:18.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.622 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.622 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.623 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000050 2023-11-02T18:30:36.623 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000050 2023-11-02T18:30:36.624 DEBUG:boto:Method: GET 2023-11-02T18:30:36.624 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000050 2023-11-02T18:30:36.625 DEBUG:boto:Data: 2023-11-02T18:30:36.625 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.626 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.627 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.627 DEBUG:boto:Params: {} 2023-11-02T18:30:36.628 DEBUG:boto:Token: None 2023-11-02T18:30:36.628 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.629 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bcbc5ef4f333114-006543eacc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.641 DEBUG:boto:b'ibtljn-58obj000000000000000000501000falseobj000000000000000000502023-11-02T18:30:18.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.641 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.642 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.642 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.643 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.643 DEBUG:boto:Data: 2023-11-02T18:30:36.643 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.644 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.644 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.645 DEBUG:boto:Params: {} 2023-11-02T18:30:36.645 DEBUG:boto:Token: None 2023-11-02T18:30:36.646 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.646 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s7lcSjwzGOWys6IsvljRhcma6VA= 2023-11-02T18:30:36.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s7lcSjwzGOWys6IsvljRhcma6VA=', 'Content-Length': '0'} 2023-11-02T18:30:36.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db783a8bf6ba0213-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.650 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.651 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.651 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.652 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.652 DEBUG:boto:Data: 2023-11-02T18:30:36.653 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.653 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.654 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.654 DEBUG:boto:Params: {} 2023-11-02T18:30:36.655 DEBUG:boto:Token: None 2023-11-02T18:30:36.655 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s7lcSjwzGOWys6IsvljRhcma6VA= 2023-11-02T18:30:36.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s7lcSjwzGOWys6IsvljRhcma6VA=', 'Content-Length': '0'} 2023-11-02T18:30:36.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005314a0f5125541a2-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-11-02T18:30:36.659 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.660 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.661 DEBUG:boto:Method: GET 2023-11-02T18:30:36.661 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.662 DEBUG:boto:Data: 2023-11-02T18:30:36.662 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.663 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.664 DEBUG:boto:Params: {} 2023-11-02T18:30:36.664 DEBUG:boto:Token: None 2023-11-02T18:30:36.665 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.665 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WIPBLnQQ+XbmLxfZ5C1WPrRbF70= 2023-11-02T18:30:36.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WIPBLnQQ+XbmLxfZ5C1WPrRbF70=', 'Content-Length': '0'} 2023-11-02T18:30:36.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa1ff1de26049884-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.710 DEBUG:boto:path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.711 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.712 DEBUG:boto:Method: GET 2023-11-02T18:30:36.712 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.712 DEBUG:boto:Data: 2023-11-02T18:30:36.713 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.714 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.714 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.715 DEBUG:boto:Params: {} 2023-11-02T18:30:36.715 DEBUG:boto:Token: None 2023-11-02T18:30:36.716 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000050 2023-11-02T18:30:36.716 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WIPBLnQQ+XbmLxfZ5C1WPrRbF70= 2023-11-02T18:30:36.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WIPBLnQQ+XbmLxfZ5C1WPrRbF70=', 'Content-Length': '0'} 2023-11-02T18:30:36.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005983df00db3d5bb4-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.720 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.721 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.721 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000050 2023-11-02T18:30:36.722 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000050 2023-11-02T18:30:36.722 DEBUG:boto:Method: GET 2023-11-02T18:30:36.723 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000050 2023-11-02T18:30:36.723 DEBUG:boto:Data: 2023-11-02T18:30:36.724 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.724 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.725 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.725 DEBUG:boto:Params: {} 2023-11-02T18:30:36.726 DEBUG:boto:Token: None 2023-11-02T18:30:36.726 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.727 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002274a3fd25a284c5-006543eacc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.733 DEBUG:boto:b'ibtljn-58obj000000000000000000501000falseobj000000000000000000502023-11-02T18:30:18.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.734 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.735 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.735 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000050 2023-11-02T18:30:36.736 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000050 2023-11-02T18:30:36.736 DEBUG:boto:Method: GET 2023-11-02T18:30:36.737 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000050 2023-11-02T18:30:36.737 DEBUG:boto:Data: 2023-11-02T18:30:36.737 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.738 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.739 DEBUG:boto:Params: {} 2023-11-02T18:30:36.739 DEBUG:boto:Token: None 2023-11-02T18:30:36.740 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000648e1eec4ae9639a-006543eacc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.749 DEBUG:boto:b'ibtljn-58obj000000000000000000501000falseobj000000000000000000502023-11-02T18:30:18.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-11-02T18:30:36.750 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.751 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.751 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.752 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.752 DEBUG:boto:Method: GET 2023-11-02T18:30:36.752 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.753 DEBUG:boto:Data: 2023-11-02T18:30:36.753 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.754 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.755 DEBUG:boto:Params: {} 2023-11-02T18:30:36.755 DEBUG:boto:Token: None 2023-11-02T18:30:36.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.756 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:89W5Vgh3ysXKk9alQecdISL/4+I= 2023-11-02T18:30:36.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:89W5Vgh3ysXKk9alQecdISL/4+I=', 'Content-Length': '0'} 2023-11-02T18:30:36.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee59e3b6e58910a-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.760 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.761 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.761 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.762 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.762 DEBUG:boto:Method: GET 2023-11-02T18:30:36.763 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.763 DEBUG:boto:Data: 2023-11-02T18:30:36.764 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.764 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.765 DEBUG:boto:Params: {} 2023-11-02T18:30:36.765 DEBUG:boto:Token: None 2023-11-02T18:30:36.766 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:89W5Vgh3ysXKk9alQecdISL/4+I= 2023-11-02T18:30:36.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:89W5Vgh3ysXKk9alQecdISL/4+I=', 'Content-Length': '0'} 2023-11-02T18:30:36.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007711be337f7717c3-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.771 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.771 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.772 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.772 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.772 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.773 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.773 DEBUG:boto:Data: 2023-11-02T18:30:36.774 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.774 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.775 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.775 DEBUG:boto:Params: {} 2023-11-02T18:30:36.775 DEBUG:boto:Token: None 2023-11-02T18:30:36.776 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.776 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:paDT6jirsdcvYJpxHhRtffu6LcY= 2023-11-02T18:30:36.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:paDT6jirsdcvYJpxHhRtffu6LcY=', 'Content-Length': '0'} 2023-11-02T18:30:36.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028123b4fe99fc186-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.780 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.781 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.781 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.781 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.782 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.782 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.783 DEBUG:boto:Data: 2023-11-02T18:30:36.783 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.784 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.784 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.784 DEBUG:boto:Params: {} 2023-11-02T18:30:36.785 DEBUG:boto:Token: None 2023-11-02T18:30:36.786 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.786 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:paDT6jirsdcvYJpxHhRtffu6LcY= 2023-11-02T18:30:36.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:paDT6jirsdcvYJpxHhRtffu6LcY=', 'Content-Length': '0'} 2023-11-02T18:30:36.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8df89c7b422ab2b-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-11-02T18:30:36.791 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.791 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.792 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.792 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.793 DEBUG:boto:Method: GET 2023-11-02T18:30:36.793 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.794 DEBUG:boto:Data: 2023-11-02T18:30:36.794 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.795 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.795 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.795 DEBUG:boto:Params: {} 2023-11-02T18:30:36.796 DEBUG:boto:Token: None 2023-11-02T18:30:36.796 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.797 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:89W5Vgh3ysXKk9alQecdISL/4+I= 2023-11-02T18:30:36.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:89W5Vgh3ysXKk9alQecdISL/4+I=', 'Content-Length': '0'} 2023-11-02T18:30:36.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c692469dc06e5e32-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.842 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.843 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.843 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.844 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.844 DEBUG:boto:Method: GET 2023-11-02T18:30:36.845 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.845 DEBUG:boto:Data: 2023-11-02T18:30:36.846 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.846 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.847 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.847 DEBUG:boto:Params: {} 2023-11-02T18:30:36.848 DEBUG:boto:Token: None 2023-11-02T18:30:36.849 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000053?versionId=null 2023-11-02T18:30:36.849 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:89W5Vgh3ysXKk9alQecdISL/4+I= 2023-11-02T18:30:36.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:89W5Vgh3ysXKk9alQecdISL/4+I=', 'Content-Length': '0'} 2023-11-02T18:30:36.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d21a27201daa0aa-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.854 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.855 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.855 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000053 2023-11-02T18:30:36.856 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000053 2023-11-02T18:30:36.856 DEBUG:boto:Method: GET 2023-11-02T18:30:36.856 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000053 2023-11-02T18:30:36.857 DEBUG:boto:Data: 2023-11-02T18:30:36.858 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.858 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.859 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.859 DEBUG:boto:Params: {} 2023-11-02T18:30:36.860 DEBUG:boto:Token: None 2023-11-02T18:30:36.860 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.861 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077c224e4f7019c47-006543eacc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.867 DEBUG:boto:b'ibtljn-58obj000000000000000000531000falseobj000000000000000000532023-11-02T18:30:19.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.868 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.869 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.869 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000053 2023-11-02T18:30:36.869 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000053 2023-11-02T18:30:36.870 DEBUG:boto:Method: GET 2023-11-02T18:30:36.871 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000053 2023-11-02T18:30:36.871 DEBUG:boto:Data: 2023-11-02T18:30:36.872 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.872 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.873 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.873 DEBUG:boto:Params: {} 2023-11-02T18:30:36.874 DEBUG:boto:Token: None 2023-11-02T18:30:36.875 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.875 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009791235b150ef54-006543eacc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.888 DEBUG:boto:b'ibtljn-58obj000000000000000000531000falseobj000000000000000000532023-11-02T18:30:19.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.889 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.889 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.890 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.890 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.891 DEBUG:boto:Data: 2023-11-02T18:30:36.891 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.892 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.892 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.893 DEBUG:boto:Params: {} 2023-11-02T18:30:36.893 DEBUG:boto:Token: None 2023-11-02T18:30:36.894 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.894 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MgiE0J4uLVEATIyWyQXx23Hh5KU= 2023-11-02T18:30:36.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MgiE0J4uLVEATIyWyQXx23Hh5KU=', 'Content-Length': '0'} 2023-11-02T18:30:36.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adb27e650c48f78d-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.899 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.900 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.901 DEBUG:boto:Method: HEAD 2023-11-02T18:30:36.902 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.903 DEBUG:boto:Data: 2023-11-02T18:30:36.904 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.905 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.906 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.907 DEBUG:boto:Params: {} 2023-11-02T18:30:36.908 DEBUG:boto:Token: None 2023-11-02T18:30:36.909 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.910 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MgiE0J4uLVEATIyWyQXx23Hh5KU= 2023-11-02T18:30:36.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MgiE0J4uLVEATIyWyQXx23Hh5KU=', 'Content-Length': '0'} 2023-11-02T18:30:36.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d71aa20d4ea32d88-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-11-02T18:30:36.914 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.914 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.915 DEBUG:boto:Method: GET 2023-11-02T18:30:36.915 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.916 DEBUG:boto:Data: 2023-11-02T18:30:36.916 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.917 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.917 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.918 DEBUG:boto:Params: {} 2023-11-02T18:30:36.918 DEBUG:boto:Token: None 2023-11-02T18:30:36.919 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.919 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pdbFKhfT7/BNzQZw3R2aK9xf+UA= 2023-11-02T18:30:36.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pdbFKhfT7/BNzQZw3R2aK9xf+UA=', 'Content-Length': '0'} 2023-11-02T18:30:36.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043bad03e2c9137fd-006543eacc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.966 DEBUG:boto:path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.967 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.968 DEBUG:boto:Method: GET 2023-11-02T18:30:36.968 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.968 DEBUG:boto:Data: 2023-11-02T18:30:36.969 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.969 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.970 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.970 DEBUG:boto:Params: {} 2023-11-02T18:30:36.971 DEBUG:boto:Token: None 2023-11-02T18:30:36.971 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/obj00000000000000000053 2023-11-02T18:30:36.972 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pdbFKhfT7/BNzQZw3R2aK9xf+UA= 2023-11-02T18:30:36.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pdbFKhfT7/BNzQZw3R2aK9xf+UA=', 'Content-Length': '0'} 2023-11-02T18:30:36.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7b913f9b0bc3271-006543eacc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.976 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.977 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.977 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000053 2023-11-02T18:30:36.977 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000053 2023-11-02T18:30:36.978 DEBUG:boto:Method: GET 2023-11-02T18:30:36.978 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000053 2023-11-02T18:30:36.979 DEBUG:boto:Data: 2023-11-02T18:30:36.979 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.980 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:36.980 DEBUG:boto:Port: 8001 2023-11-02T18:30:36.981 DEBUG:boto:Params: {} 2023-11-02T18:30:36.981 DEBUG:boto:Token: None 2023-11-02T18:30:36.982 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.982 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:36.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046171e3d972984c4-006543eacc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:36.988 DEBUG:boto:b'ibtljn-58obj000000000000000000531000falseobj000000000000000000532023-11-02T18:30:19.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:36.989 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:36.989 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:36.990 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000053 2023-11-02T18:30:36.990 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000053 2023-11-02T18:30:36.991 DEBUG:boto:Method: GET 2023-11-02T18:30:36.991 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000053 2023-11-02T18:30:36.992 DEBUG:boto:Data: 2023-11-02T18:30:36.992 DEBUG:boto:Headers: {} 2023-11-02T18:30:36.993 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:36.993 DEBUG:boto:Port: 8000 2023-11-02T18:30:36.994 DEBUG:boto:Params: {} 2023-11-02T18:30:36.994 DEBUG:boto:Token: None 2023-11-02T18:30:36.995 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:36 GMT /ibtljn-58/ 2023-11-02T18:30:36.995 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc= 2023-11-02T18:30:36.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xllbcT5+kyXHQcgfDahDekZGbnc=', 'Content-Length': '0'} 2023-11-02T18:30:37.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dce1e09a6f3d6f37-006543eacc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.007 DEBUG:boto:b'ibtljn-58obj000000000000000000531000falseobj000000000000000000532023-11-02T18:30:19.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-11-02T18:30:37.009 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.009 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.010 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.010 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.011 DEBUG:boto:Method: GET 2023-11-02T18:30:37.011 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.012 DEBUG:boto:Data: 2023-11-02T18:30:37.012 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.013 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.013 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.014 DEBUG:boto:Params: {} 2023-11-02T18:30:37.014 DEBUG:boto:Token: None 2023-11-02T18:30:37.015 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Um/rZVkDBm5LHwaXZJgPLlb30/k= 2023-11-02T18:30:37.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Um/rZVkDBm5LHwaXZJgPLlb30/k=', 'Content-Length': '0'} 2023-11-02T18:30:37.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d97b6f260792028d-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.019 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.020 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.020 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.021 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.021 DEBUG:boto:Method: GET 2023-11-02T18:30:37.022 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.022 DEBUG:boto:Data: 2023-11-02T18:30:37.022 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.023 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.023 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.024 DEBUG:boto:Params: {} 2023-11-02T18:30:37.024 DEBUG:boto:Token: None 2023-11-02T18:30:37.025 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.025 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Um/rZVkDBm5LHwaXZJgPLlb30/k= 2023-11-02T18:30:37.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Um/rZVkDBm5LHwaXZJgPLlb30/k=', 'Content-Length': '0'} 2023-11-02T18:30:37.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df5ede3d9d296d88-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.029 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.030 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.030 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.031 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.031 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.032 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.032 DEBUG:boto:Data: 2023-11-02T18:30:37.033 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.033 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.034 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.034 DEBUG:boto:Params: {} 2023-11-02T18:30:37.035 DEBUG:boto:Token: None 2023-11-02T18:30:37.035 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.036 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IQ/qZi9brThtoLK9wI8FyxVurOQ= 2023-11-02T18:30:37.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IQ/qZi9brThtoLK9wI8FyxVurOQ=', 'Content-Length': '0'} 2023-11-02T18:30:37.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd68cb35e04a2f15-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.040 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.040 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.041 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.041 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.042 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.042 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.043 DEBUG:boto:Data: 2023-11-02T18:30:37.043 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.044 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.045 DEBUG:boto:Params: {} 2023-11-02T18:30:37.045 DEBUG:boto:Token: None 2023-11-02T18:30:37.046 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.046 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IQ/qZi9brThtoLK9wI8FyxVurOQ= 2023-11-02T18:30:37.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IQ/qZi9brThtoLK9wI8FyxVurOQ=', 'Content-Length': '0'} 2023-11-02T18:30:37.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e04148963b3d0e-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-11-02T18:30:37.051 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.051 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.052 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.052 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.053 DEBUG:boto:Method: GET 2023-11-02T18:30:37.053 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.054 DEBUG:boto:Data: 2023-11-02T18:30:37.054 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.055 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.055 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.056 DEBUG:boto:Params: {} 2023-11-02T18:30:37.056 DEBUG:boto:Token: None 2023-11-02T18:30:37.057 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.057 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Um/rZVkDBm5LHwaXZJgPLlb30/k= 2023-11-02T18:30:37.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Um/rZVkDBm5LHwaXZJgPLlb30/k=', 'Content-Length': '0'} 2023-11-02T18:30:37.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000218976cb0988ea9-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.102 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.103 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.103 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.104 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.104 DEBUG:boto:Method: GET 2023-11-02T18:30:37.105 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.105 DEBUG:boto:Data: 2023-11-02T18:30:37.106 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.107 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.107 DEBUG:boto:Params: {} 2023-11-02T18:30:37.107 DEBUG:boto:Token: None 2023-11-02T18:30:37.108 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000055?versionId=null 2023-11-02T18:30:37.109 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Um/rZVkDBm5LHwaXZJgPLlb30/k= 2023-11-02T18:30:37.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Um/rZVkDBm5LHwaXZJgPLlb30/k=', 'Content-Length': '0'} 2023-11-02T18:30:37.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e8f58b36fd54872-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.113 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.113 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.114 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000055 2023-11-02T18:30:37.114 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000055 2023-11-02T18:30:37.115 DEBUG:boto:Method: GET 2023-11-02T18:30:37.115 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000055 2023-11-02T18:30:37.115 DEBUG:boto:Data: 2023-11-02T18:30:37.116 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.116 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.117 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.117 DEBUG:boto:Params: {} 2023-11-02T18:30:37.118 DEBUG:boto:Token: None 2023-11-02T18:30:37.118 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.119 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020268d1f055c4c3c-006543eacd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.125 DEBUG:boto:b'ibtljn-58obj000000000000000000551000falseobj000000000000000000552023-11-02T18:30:14.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.125 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.126 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.126 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000055 2023-11-02T18:30:37.127 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000055 2023-11-02T18:30:37.127 DEBUG:boto:Method: GET 2023-11-02T18:30:37.128 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000055 2023-11-02T18:30:37.128 DEBUG:boto:Data: 2023-11-02T18:30:37.129 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.129 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.129 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.130 DEBUG:boto:Params: {} 2023-11-02T18:30:37.130 DEBUG:boto:Token: None 2023-11-02T18:30:37.131 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.132 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb3120421fb8dc75-006543eacd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.186 DEBUG:boto:b'ibtljn-58obj000000000000000000551000falseobj000000000000000000552023-11-02T18:30:14.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.187 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.188 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.188 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.189 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.189 DEBUG:boto:Data: 2023-11-02T18:30:37.189 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.190 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.190 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.191 DEBUG:boto:Params: {} 2023-11-02T18:30:37.191 DEBUG:boto:Token: None 2023-11-02T18:30:37.192 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.192 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lYVZmA2uD0iL2nuOzeu586G7pkE= 2023-11-02T18:30:37.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lYVZmA2uD0iL2nuOzeu586G7pkE=', 'Content-Length': '0'} 2023-11-02T18:30:37.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088e095f17da84440-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.196 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.197 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.197 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.198 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.198 DEBUG:boto:Data: 2023-11-02T18:30:37.199 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.199 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.200 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.200 DEBUG:boto:Params: {} 2023-11-02T18:30:37.201 DEBUG:boto:Token: None 2023-11-02T18:30:37.201 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.202 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lYVZmA2uD0iL2nuOzeu586G7pkE= 2023-11-02T18:30:37.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lYVZmA2uD0iL2nuOzeu586G7pkE=', 'Content-Length': '0'} 2023-11-02T18:30:37.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f23b0e9fbd91e4da-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-11-02T18:30:37.206 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.207 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.207 DEBUG:boto:Method: GET 2023-11-02T18:30:37.208 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.208 DEBUG:boto:Data: 2023-11-02T18:30:37.209 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.209 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.210 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.210 DEBUG:boto:Params: {} 2023-11-02T18:30:37.211 DEBUG:boto:Token: None 2023-11-02T18:30:37.211 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.212 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ivz3pKwDwabbfUMRB3MCDrmGaYo= 2023-11-02T18:30:37.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ivz3pKwDwabbfUMRB3MCDrmGaYo=', 'Content-Length': '0'} 2023-11-02T18:30:37.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009be1f81231459849-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.216 DEBUG:boto:path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.216 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.217 DEBUG:boto:Method: GET 2023-11-02T18:30:37.217 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.218 DEBUG:boto:Data: 2023-11-02T18:30:37.218 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.219 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.219 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.220 DEBUG:boto:Params: {} 2023-11-02T18:30:37.221 DEBUG:boto:Token: None 2023-11-02T18:30:37.221 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000055 2023-11-02T18:30:37.222 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ivz3pKwDwabbfUMRB3MCDrmGaYo= 2023-11-02T18:30:37.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ivz3pKwDwabbfUMRB3MCDrmGaYo=', 'Content-Length': '0'} 2023-11-02T18:30:37.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072781c1b8d1bf01b-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.226 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.227 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.227 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000055 2023-11-02T18:30:37.228 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000055 2023-11-02T18:30:37.228 DEBUG:boto:Method: GET 2023-11-02T18:30:37.229 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000055 2023-11-02T18:30:37.229 DEBUG:boto:Data: 2023-11-02T18:30:37.229 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.230 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.230 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.231 DEBUG:boto:Params: {} 2023-11-02T18:30:37.231 DEBUG:boto:Token: None 2023-11-02T18:30:37.232 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.232 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000260a977fea96f551-006543eacd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.278 DEBUG:boto:b'ibtljn-58obj000000000000000000551000falseobj000000000000000000552023-11-02T18:30:14.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.279 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.280 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.280 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000055 2023-11-02T18:30:37.281 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000055 2023-11-02T18:30:37.282 DEBUG:boto:Method: GET 2023-11-02T18:30:37.282 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000055 2023-11-02T18:30:37.283 DEBUG:boto:Data: 2023-11-02T18:30:37.283 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.283 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.284 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.284 DEBUG:boto:Params: {} 2023-11-02T18:30:37.285 DEBUG:boto:Token: None 2023-11-02T18:30:37.286 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.286 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004443e038f75d4818-006543eacd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.298 DEBUG:boto:b'ibtljn-58obj000000000000000000551000falseobj000000000000000000552023-11-02T18:30:14.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-11-02T18:30:37.299 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.300 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.300 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.301 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.301 DEBUG:boto:Method: GET 2023-11-02T18:30:37.301 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.302 DEBUG:boto:Data: 2023-11-02T18:30:37.302 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.303 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.304 DEBUG:boto:Params: {} 2023-11-02T18:30:37.304 DEBUG:boto:Token: None 2023-11-02T18:30:37.305 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cdeB9zUrJyjoD5XyKb8WJHbTknI= 2023-11-02T18:30:37.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cdeB9zUrJyjoD5XyKb8WJHbTknI=', 'Content-Length': '0'} 2023-11-02T18:30:37.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077e0e68c3cfe4d52-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.309 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.310 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.310 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.311 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.311 DEBUG:boto:Method: GET 2023-11-02T18:30:37.312 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.312 DEBUG:boto:Data: 2023-11-02T18:30:37.313 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.313 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.314 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.314 DEBUG:boto:Params: {} 2023-11-02T18:30:37.315 DEBUG:boto:Token: None 2023-11-02T18:30:37.315 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.316 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cdeB9zUrJyjoD5XyKb8WJHbTknI= 2023-11-02T18:30:37.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cdeB9zUrJyjoD5XyKb8WJHbTknI=', 'Content-Length': '0'} 2023-11-02T18:30:37.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053bffb0372f2b603-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.320 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.321 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.321 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.322 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.322 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.323 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.323 DEBUG:boto:Data: 2023-11-02T18:30:37.324 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.324 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.324 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.325 DEBUG:boto:Params: {} 2023-11-02T18:30:37.325 DEBUG:boto:Token: None 2023-11-02T18:30:37.326 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.326 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bitpnUH8FvPajX96ZtbcLo71w84= 2023-11-02T18:30:37.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bitpnUH8FvPajX96ZtbcLo71w84=', 'Content-Length': '0'} 2023-11-02T18:30:37.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ddf1426c9ce35c0-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.330 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.331 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.331 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.331 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.332 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.332 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.333 DEBUG:boto:Data: 2023-11-02T18:30:37.333 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.334 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.334 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.334 DEBUG:boto:Params: {} 2023-11-02T18:30:37.335 DEBUG:boto:Token: None 2023-11-02T18:30:37.335 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.336 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bitpnUH8FvPajX96ZtbcLo71w84= 2023-11-02T18:30:37.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bitpnUH8FvPajX96ZtbcLo71w84=', 'Content-Length': '0'} 2023-11-02T18:30:37.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0342ee1431605ad-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-11-02T18:30:37.340 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.341 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.341 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.342 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.342 DEBUG:boto:Method: GET 2023-11-02T18:30:37.343 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.343 DEBUG:boto:Data: 2023-11-02T18:30:37.344 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.344 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.345 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.345 DEBUG:boto:Params: {} 2023-11-02T18:30:37.346 DEBUG:boto:Token: None 2023-11-02T18:30:37.347 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.347 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cdeB9zUrJyjoD5XyKb8WJHbTknI= 2023-11-02T18:30:37.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cdeB9zUrJyjoD5XyKb8WJHbTknI=', 'Content-Length': '0'} 2023-11-02T18:30:37.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000111293cd71013510-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.351 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.352 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.352 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.353 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.353 DEBUG:boto:Method: GET 2023-11-02T18:30:37.354 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.354 DEBUG:boto:Data: 2023-11-02T18:30:37.355 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.355 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.356 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.356 DEBUG:boto:Params: {} 2023-11-02T18:30:37.357 DEBUG:boto:Token: None 2023-11-02T18:30:37.357 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000056?versionId=null 2023-11-02T18:30:37.358 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cdeB9zUrJyjoD5XyKb8WJHbTknI= 2023-11-02T18:30:37.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cdeB9zUrJyjoD5XyKb8WJHbTknI=', 'Content-Length': '0'} 2023-11-02T18:30:37.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bef64ddf3d216a8-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.402 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.403 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.403 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000056 2023-11-02T18:30:37.404 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000056 2023-11-02T18:30:37.404 DEBUG:boto:Method: GET 2023-11-02T18:30:37.405 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000056 2023-11-02T18:30:37.405 DEBUG:boto:Data: 2023-11-02T18:30:37.406 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.406 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.407 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.407 DEBUG:boto:Params: {} 2023-11-02T18:30:37.408 DEBUG:boto:Token: None 2023-11-02T18:30:37.408 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.409 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c00053642ac79c56-006543eacd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.415 DEBUG:boto:b'ibtljn-58obj000000000000000000561000falseobj000000000000000000562023-11-02T18:30:18.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.415 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.416 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.416 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000056 2023-11-02T18:30:37.417 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000056 2023-11-02T18:30:37.417 DEBUG:boto:Method: GET 2023-11-02T18:30:37.417 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000056 2023-11-02T18:30:37.418 DEBUG:boto:Data: 2023-11-02T18:30:37.418 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.419 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.419 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.420 DEBUG:boto:Params: {} 2023-11-02T18:30:37.420 DEBUG:boto:Token: None 2023-11-02T18:30:37.421 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.421 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9fbfa31b3e1bb02-006543eacd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.433 DEBUG:boto:b'ibtljn-58obj000000000000000000561000falseobj000000000000000000562023-11-02T18:30:18.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.433 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.434 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.434 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.435 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.435 DEBUG:boto:Data: 2023-11-02T18:30:37.436 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.436 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.437 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.437 DEBUG:boto:Params: {} 2023-11-02T18:30:37.437 DEBUG:boto:Token: None 2023-11-02T18:30:37.438 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.438 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VJoKNAZ8kBUq33Ve4l3HGKmkN9g= 2023-11-02T18:30:37.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VJoKNAZ8kBUq33Ve4l3HGKmkN9g=', 'Content-Length': '0'} 2023-11-02T18:30:37.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007101f2bba9c7d8db-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.441 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.442 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.442 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.443 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.443 DEBUG:boto:Data: 2023-11-02T18:30:37.444 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.444 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.444 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.445 DEBUG:boto:Params: {} 2023-11-02T18:30:37.445 DEBUG:boto:Token: None 2023-11-02T18:30:37.446 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.446 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VJoKNAZ8kBUq33Ve4l3HGKmkN9g= 2023-11-02T18:30:37.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VJoKNAZ8kBUq33Ve4l3HGKmkN9g=', 'Content-Length': '0'} 2023-11-02T18:30:37.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000595fdf7a6513ab2e-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-11-02T18:30:37.450 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.450 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.451 DEBUG:boto:Method: GET 2023-11-02T18:30:37.453 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.453 DEBUG:boto:Data: 2023-11-02T18:30:37.454 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.455 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.455 DEBUG:boto:Params: {} 2023-11-02T18:30:37.455 DEBUG:boto:Token: None 2023-11-02T18:30:37.456 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.456 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b0ZbYtIm/8uftmQiAMsBRzwoDyk= 2023-11-02T18:30:37.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b0ZbYtIm/8uftmQiAMsBRzwoDyk=', 'Content-Length': '0'} 2023-11-02T18:30:37.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0256ec07280f4f4-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.460 DEBUG:boto:path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.461 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.461 DEBUG:boto:Method: GET 2023-11-02T18:30:37.462 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.462 DEBUG:boto:Data: 2023-11-02T18:30:37.463 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.463 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.464 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.464 DEBUG:boto:Params: {} 2023-11-02T18:30:37.465 DEBUG:boto:Token: None 2023-11-02T18:30:37.465 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000056 2023-11-02T18:30:37.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b0ZbYtIm/8uftmQiAMsBRzwoDyk= 2023-11-02T18:30:37.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b0ZbYtIm/8uftmQiAMsBRzwoDyk=', 'Content-Length': '0'} 2023-11-02T18:30:37.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000500f3f5271d0de10-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.469 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.470 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.470 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000056 2023-11-02T18:30:37.470 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000056 2023-11-02T18:30:37.471 DEBUG:boto:Method: GET 2023-11-02T18:30:37.471 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000056 2023-11-02T18:30:37.472 DEBUG:boto:Data: 2023-11-02T18:30:37.472 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.473 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.473 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.473 DEBUG:boto:Params: {} 2023-11-02T18:30:37.474 DEBUG:boto:Token: None 2023-11-02T18:30:37.474 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.475 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a7ea927f92dc0fb-006543eacd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.522 DEBUG:boto:b'ibtljn-58obj000000000000000000561000falseobj000000000000000000562023-11-02T18:30:18.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.523 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.524 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.524 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000056 2023-11-02T18:30:37.525 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000056 2023-11-02T18:30:37.525 DEBUG:boto:Method: GET 2023-11-02T18:30:37.525 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000056 2023-11-02T18:30:37.526 DEBUG:boto:Data: 2023-11-02T18:30:37.526 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.527 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.527 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.528 DEBUG:boto:Params: {} 2023-11-02T18:30:37.528 DEBUG:boto:Token: None 2023-11-02T18:30:37.529 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.529 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3260ff90330720c-006543eacd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.541 DEBUG:boto:b'ibtljn-58obj000000000000000000561000falseobj000000000000000000562023-11-02T18:30:18.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-11-02T18:30:37.542 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.542 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.543 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.543 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.544 DEBUG:boto:Method: GET 2023-11-02T18:30:37.544 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.544 DEBUG:boto:Data: 2023-11-02T18:30:37.545 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.545 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.546 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.546 DEBUG:boto:Params: {} 2023-11-02T18:30:37.547 DEBUG:boto:Token: None 2023-11-02T18:30:37.547 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.548 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ndYic3PNVVLUW3IZ8PCTMhA9MGU= 2023-11-02T18:30:37.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ndYic3PNVVLUW3IZ8PCTMhA9MGU=', 'Content-Length': '0'} 2023-11-02T18:30:37.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0ddc5dddc5f32e7-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.551 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.552 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.552 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.553 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.553 DEBUG:boto:Method: GET 2023-11-02T18:30:37.554 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.554 DEBUG:boto:Data: 2023-11-02T18:30:37.555 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.555 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.555 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.556 DEBUG:boto:Params: {} 2023-11-02T18:30:37.556 DEBUG:boto:Token: None 2023-11-02T18:30:37.557 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.557 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ndYic3PNVVLUW3IZ8PCTMhA9MGU= 2023-11-02T18:30:37.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ndYic3PNVVLUW3IZ8PCTMhA9MGU=', 'Content-Length': '0'} 2023-11-02T18:30:37.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007772ee3ed724ddfa-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.606 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.607 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.607 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.608 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.608 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.609 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.609 DEBUG:boto:Data: 2023-11-02T18:30:37.610 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.610 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.610 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.611 DEBUG:boto:Params: {} 2023-11-02T18:30:37.611 DEBUG:boto:Token: None 2023-11-02T18:30:37.612 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6flYV81wtPpVNG5wqXTs5lTDups= 2023-11-02T18:30:37.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6flYV81wtPpVNG5wqXTs5lTDups=', 'Content-Length': '0'} 2023-11-02T18:30:37.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007390e2bc8bea421-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.616 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.617 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.617 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.617 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.618 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.618 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.619 DEBUG:boto:Data: 2023-11-02T18:30:37.619 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.620 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.620 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.620 DEBUG:boto:Params: {} 2023-11-02T18:30:37.621 DEBUG:boto:Token: None 2023-11-02T18:30:37.621 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.622 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6flYV81wtPpVNG5wqXTs5lTDups= 2023-11-02T18:30:37.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6flYV81wtPpVNG5wqXTs5lTDups=', 'Content-Length': '0'} 2023-11-02T18:30:37.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdadf903dab9d6e8-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-11-02T18:30:37.626 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.627 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.627 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.627 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.628 DEBUG:boto:Method: GET 2023-11-02T18:30:37.628 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.629 DEBUG:boto:Data: 2023-11-02T18:30:37.629 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.630 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.630 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.630 DEBUG:boto:Params: {} 2023-11-02T18:30:37.631 DEBUG:boto:Token: None 2023-11-02T18:30:37.631 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.632 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ndYic3PNVVLUW3IZ8PCTMhA9MGU= 2023-11-02T18:30:37.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ndYic3PNVVLUW3IZ8PCTMhA9MGU=', 'Content-Length': '0'} 2023-11-02T18:30:37.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d7aab52ab3ab1a-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.636 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.636 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.637 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.637 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.638 DEBUG:boto:Method: GET 2023-11-02T18:30:37.638 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.639 DEBUG:boto:Data: 2023-11-02T18:30:37.639 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.640 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.640 DEBUG:boto:Params: {} 2023-11-02T18:30:37.641 DEBUG:boto:Token: None 2023-11-02T18:30:37.641 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000058?versionId=null 2023-11-02T18:30:37.642 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ndYic3PNVVLUW3IZ8PCTMhA9MGU= 2023-11-02T18:30:37.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ndYic3PNVVLUW3IZ8PCTMhA9MGU=', 'Content-Length': '0'} 2023-11-02T18:30:37.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b731217498551245-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.646 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.646 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.647 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000058 2023-11-02T18:30:37.647 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000058 2023-11-02T18:30:37.648 DEBUG:boto:Method: GET 2023-11-02T18:30:37.648 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000058 2023-11-02T18:30:37.648 DEBUG:boto:Data: 2023-11-02T18:30:37.649 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.649 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.650 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.650 DEBUG:boto:Params: {} 2023-11-02T18:30:37.651 DEBUG:boto:Token: None 2023-11-02T18:30:37.651 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.651 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023783c6ec7aeab1e-006543eacd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.702 DEBUG:boto:b'ibtljn-58obj000000000000000000581000falseobj000000000000000000582023-11-02T18:30:15.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.703 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.704 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.704 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000058 2023-11-02T18:30:37.705 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000058 2023-11-02T18:30:37.705 DEBUG:boto:Method: GET 2023-11-02T18:30:37.706 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000058 2023-11-02T18:30:37.706 DEBUG:boto:Data: 2023-11-02T18:30:37.706 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.707 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.708 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.708 DEBUG:boto:Params: {} 2023-11-02T18:30:37.709 DEBUG:boto:Token: None 2023-11-02T18:30:37.709 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.710 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002736d5cce5425fc7-006543eacd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.720 DEBUG:boto:b'ibtljn-58obj000000000000000000581000falseobj000000000000000000582023-11-02T18:30:15.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.720 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.721 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.721 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.722 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.722 DEBUG:boto:Data: 2023-11-02T18:30:37.722 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.723 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.723 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.724 DEBUG:boto:Params: {} 2023-11-02T18:30:37.724 DEBUG:boto:Token: None 2023-11-02T18:30:37.725 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.725 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EVWQN791OudLy8ddjA4KGzLV7ro= 2023-11-02T18:30:37.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EVWQN791OudLy8ddjA4KGzLV7ro=', 'Content-Length': '0'} 2023-11-02T18:30:37.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fd32acd61f342d9-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.729 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.730 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.730 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.730 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.731 DEBUG:boto:Data: 2023-11-02T18:30:37.731 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.732 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.732 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.732 DEBUG:boto:Params: {} 2023-11-02T18:30:37.733 DEBUG:boto:Token: None 2023-11-02T18:30:37.733 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.734 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EVWQN791OudLy8ddjA4KGzLV7ro= 2023-11-02T18:30:37.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EVWQN791OudLy8ddjA4KGzLV7ro=', 'Content-Length': '0'} 2023-11-02T18:30:37.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d9838bd930b17ef-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-11-02T18:30:37.738 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.738 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.739 DEBUG:boto:Method: GET 2023-11-02T18:30:37.739 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.740 DEBUG:boto:Data: 2023-11-02T18:30:37.740 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.741 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.741 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.742 DEBUG:boto:Params: {} 2023-11-02T18:30:37.742 DEBUG:boto:Token: None 2023-11-02T18:30:37.743 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.744 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RxwIJEVxeEz3jPX6c4PlfPicxNw= 2023-11-02T18:30:37.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RxwIJEVxeEz3jPX6c4PlfPicxNw=', 'Content-Length': '0'} 2023-11-02T18:30:37.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000680a1494bb73a362-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.748 DEBUG:boto:path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.748 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.749 DEBUG:boto:Method: GET 2023-11-02T18:30:37.749 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.749 DEBUG:boto:Data: 2023-11-02T18:30:37.750 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.750 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.751 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.751 DEBUG:boto:Params: {} 2023-11-02T18:30:37.752 DEBUG:boto:Token: None 2023-11-02T18:30:37.752 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000058 2023-11-02T18:30:37.753 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RxwIJEVxeEz3jPX6c4PlfPicxNw= 2023-11-02T18:30:37.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RxwIJEVxeEz3jPX6c4PlfPicxNw=', 'Content-Length': '0'} 2023-11-02T18:30:37.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000058a62c26a8a5b0e-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.757 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.757 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.758 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000058 2023-11-02T18:30:37.758 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000058 2023-11-02T18:30:37.759 DEBUG:boto:Method: GET 2023-11-02T18:30:37.760 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000058 2023-11-02T18:30:37.760 DEBUG:boto:Data: 2023-11-02T18:30:37.761 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.761 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.761 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.762 DEBUG:boto:Params: {} 2023-11-02T18:30:37.762 DEBUG:boto:Token: None 2023-11-02T18:30:37.763 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.763 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa8cbec8726a23a-006543eacd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.770 DEBUG:boto:b'ibtljn-58obj000000000000000000581000falseobj000000000000000000582023-11-02T18:30:15.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.771 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.771 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.771 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000058 2023-11-02T18:30:37.772 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000058 2023-11-02T18:30:37.772 DEBUG:boto:Method: GET 2023-11-02T18:30:37.773 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000058 2023-11-02T18:30:37.773 DEBUG:boto:Data: 2023-11-02T18:30:37.774 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.774 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.775 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.775 DEBUG:boto:Params: {} 2023-11-02T18:30:37.776 DEBUG:boto:Token: None 2023-11-02T18:30:37.776 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.777 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcb3d368912dacb2-006543eacd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.830 DEBUG:boto:b'ibtljn-58obj000000000000000000581000falseobj000000000000000000582023-11-02T18:30:15.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-11-02T18:30:37.832 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.832 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.833 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.833 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.834 DEBUG:boto:Method: GET 2023-11-02T18:30:37.834 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.835 DEBUG:boto:Data: 2023-11-02T18:30:37.835 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.835 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.836 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.836 DEBUG:boto:Params: {} 2023-11-02T18:30:37.837 DEBUG:boto:Token: None 2023-11-02T18:30:37.837 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.838 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W7iSG6aZO0L2UyPSlrABPmGFlhc= 2023-11-02T18:30:37.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W7iSG6aZO0L2UyPSlrABPmGFlhc=', 'Content-Length': '0'} 2023-11-02T18:30:37.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8cc109dd780726a-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.843 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.843 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.844 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.844 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.845 DEBUG:boto:Method: GET 2023-11-02T18:30:37.845 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.845 DEBUG:boto:Data: 2023-11-02T18:30:37.846 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.846 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.847 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.847 DEBUG:boto:Params: {} 2023-11-02T18:30:37.848 DEBUG:boto:Token: None 2023-11-02T18:30:37.848 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.849 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W7iSG6aZO0L2UyPSlrABPmGFlhc= 2023-11-02T18:30:37.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W7iSG6aZO0L2UyPSlrABPmGFlhc=', 'Content-Length': '0'} 2023-11-02T18:30:37.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e8aa288c56529e-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.854 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.854 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.854 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.855 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.855 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.856 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.856 DEBUG:boto:Data: 2023-11-02T18:30:37.857 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.858 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.858 DEBUG:boto:Params: {} 2023-11-02T18:30:37.859 DEBUG:boto:Token: None 2023-11-02T18:30:37.859 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.860 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t+njRvYhd9x91PLYIulWgwLuHyc= 2023-11-02T18:30:37.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t+njRvYhd9x91PLYIulWgwLuHyc=', 'Content-Length': '0'} 2023-11-02T18:30:37.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f84b02b128136ca1-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.864 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.864 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.865 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.865 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.866 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.866 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.866 DEBUG:boto:Data: 2023-11-02T18:30:37.867 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.867 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.868 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.868 DEBUG:boto:Params: {} 2023-11-02T18:30:37.869 DEBUG:boto:Token: None 2023-11-02T18:30:37.869 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.870 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t+njRvYhd9x91PLYIulWgwLuHyc= 2023-11-02T18:30:37.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t+njRvYhd9x91PLYIulWgwLuHyc=', 'Content-Length': '0'} 2023-11-02T18:30:37.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000560c967c50a69459-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-11-02T18:30:37.874 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.874 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.875 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.875 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.876 DEBUG:boto:Method: GET 2023-11-02T18:30:37.876 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.877 DEBUG:boto:Data: 2023-11-02T18:30:37.877 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.878 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.878 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.878 DEBUG:boto:Params: {} 2023-11-02T18:30:37.879 DEBUG:boto:Token: None 2023-11-02T18:30:37.880 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.880 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W7iSG6aZO0L2UyPSlrABPmGFlhc= 2023-11-02T18:30:37.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W7iSG6aZO0L2UyPSlrABPmGFlhc=', 'Content-Length': '0'} 2023-11-02T18:30:37.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049b95ec92b179351-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.926 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.927 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.928 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.928 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.928 DEBUG:boto:Method: GET 2023-11-02T18:30:37.929 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.929 DEBUG:boto:Data: 2023-11-02T18:30:37.930 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.931 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.931 DEBUG:boto:Params: {} 2023-11-02T18:30:37.932 DEBUG:boto:Token: None 2023-11-02T18:30:37.932 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000059?versionId=null 2023-11-02T18:30:37.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W7iSG6aZO0L2UyPSlrABPmGFlhc= 2023-11-02T18:30:37.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W7iSG6aZO0L2UyPSlrABPmGFlhc=', 'Content-Length': '0'} 2023-11-02T18:30:37.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f97e9303e3d55d35-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.937 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.938 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.938 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000059 2023-11-02T18:30:37.939 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000059 2023-11-02T18:30:37.939 DEBUG:boto:Method: GET 2023-11-02T18:30:37.940 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000059 2023-11-02T18:30:37.940 DEBUG:boto:Data: 2023-11-02T18:30:37.941 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.941 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.942 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.942 DEBUG:boto:Params: {} 2023-11-02T18:30:37.942 DEBUG:boto:Token: None 2023-11-02T18:30:37.943 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.944 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e166f878d8ed35ab-006543eacd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.950 DEBUG:boto:b'ibtljn-58obj000000000000000000591000falseobj000000000000000000592023-11-02T18:30:16.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.950 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:37.951 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:37.951 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000059 2023-11-02T18:30:37.952 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000059 2023-11-02T18:30:37.952 DEBUG:boto:Method: GET 2023-11-02T18:30:37.953 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000059 2023-11-02T18:30:37.953 DEBUG:boto:Data: 2023-11-02T18:30:37.954 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.954 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.955 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.955 DEBUG:boto:Params: {} 2023-11-02T18:30:37.956 DEBUG:boto:Token: None 2023-11-02T18:30:37.956 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/ 2023-11-02T18:30:37.957 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso= 2023-11-02T18:30:37.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QIBz52Rz9Axe7OLhMrPCExhkCso=', 'Content-Length': '0'} 2023-11-02T18:30:37.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073f99ba7c0cfb5c4-006543eacd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.969 DEBUG:boto:b'ibtljn-58obj000000000000000000591000falseobj000000000000000000592023-11-02T18:30:16.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:37.970 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.970 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.971 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.971 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.972 DEBUG:boto:Data: 2023-11-02T18:30:37.972 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.973 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.973 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.973 DEBUG:boto:Params: {} 2023-11-02T18:30:37.974 DEBUG:boto:Token: None 2023-11-02T18:30:37.974 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wUakUDr18414/164bG73MoqUuY4= 2023-11-02T18:30:37.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wUakUDr18414/164bG73MoqUuY4=', 'Content-Length': '0'} 2023-11-02T18:30:37.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c220462aa13f5f65-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.979 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.979 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.980 DEBUG:boto:Method: HEAD 2023-11-02T18:30:37.980 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.981 DEBUG:boto:Data: 2023-11-02T18:30:37.981 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.982 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:37.982 DEBUG:boto:Port: 8000 2023-11-02T18:30:37.982 DEBUG:boto:Params: {} 2023-11-02T18:30:37.983 DEBUG:boto:Token: None 2023-11-02T18:30:37.983 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.984 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wUakUDr18414/164bG73MoqUuY4= 2023-11-02T18:30:37.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wUakUDr18414/164bG73MoqUuY4=', 'Content-Length': '0'} 2023-11-02T18:30:37.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5e6e1493a088385-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-11-02T18:30:37.988 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.988 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.989 DEBUG:boto:Method: GET 2023-11-02T18:30:37.989 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.990 DEBUG:boto:Data: 2023-11-02T18:30:37.990 DEBUG:boto:Headers: {} 2023-11-02T18:30:37.991 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:37.991 DEBUG:boto:Port: 8001 2023-11-02T18:30:37.992 DEBUG:boto:Params: {} 2023-11-02T18:30:37.992 DEBUG:boto:Token: None 2023-11-02T18:30:37.993 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:37 GMT /ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GsQILlFKo+pG/IikjicI3TrdUhk= 2023-11-02T18:30:37.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GsQILlFKo+pG/IikjicI3TrdUhk=', 'Content-Length': '0'} 2023-11-02T18:30:37.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047acead03425a03d-006543eacd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:37.997 DEBUG:boto:path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.998 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.998 DEBUG:boto:Method: GET 2023-11-02T18:30:37.999 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000059 2023-11-02T18:30:37.999 DEBUG:boto:Data: 2023-11-02T18:30:38.000 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.000 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.001 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.001 DEBUG:boto:Params: {} 2023-11-02T18:30:38.002 DEBUG:boto:Token: None 2023-11-02T18:30:38.002 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000059 2023-11-02T18:30:38.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7zCD08oK2HWRkAoiTWrDB0wI/Bk= 2023-11-02T18:30:38.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7zCD08oK2HWRkAoiTWrDB0wI/Bk=', 'Content-Length': '0'} 2023-11-02T18:30:38.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a7da18491f82cae-006543eacd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.046 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.047 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.047 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000059 2023-11-02T18:30:38.048 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000059 2023-11-02T18:30:38.048 DEBUG:boto:Method: GET 2023-11-02T18:30:38.049 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000059 2023-11-02T18:30:38.049 DEBUG:boto:Data: 2023-11-02T18:30:38.050 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.051 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.051 DEBUG:boto:Params: {} 2023-11-02T18:30:38.051 DEBUG:boto:Token: None 2023-11-02T18:30:38.052 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.053 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab5b4319bbba2ed3-006543eace-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.060 DEBUG:boto:b'ibtljn-58obj000000000000000000591000falseobj000000000000000000592023-11-02T18:30:16.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.060 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.061 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.061 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000059 2023-11-02T18:30:38.062 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000059 2023-11-02T18:30:38.062 DEBUG:boto:Method: GET 2023-11-02T18:30:38.063 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000059 2023-11-02T18:30:38.064 DEBUG:boto:Data: 2023-11-02T18:30:38.064 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.065 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.065 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.065 DEBUG:boto:Params: {} 2023-11-02T18:30:38.066 DEBUG:boto:Token: None 2023-11-02T18:30:38.067 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.067 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7fa15bfa76f59ff-006543eace-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.078 DEBUG:boto:b'ibtljn-58obj000000000000000000591000falseobj000000000000000000592023-11-02T18:30:16.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-11-02T18:30:38.079 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.080 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.080 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.080 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.081 DEBUG:boto:Method: GET 2023-11-02T18:30:38.081 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.082 DEBUG:boto:Data: 2023-11-02T18:30:38.082 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.083 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.083 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.084 DEBUG:boto:Params: {} 2023-11-02T18:30:38.084 DEBUG:boto:Token: None 2023-11-02T18:30:38.085 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.085 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ghJgYPjt2J4KdYmcl26U0QsVuco= 2023-11-02T18:30:38.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ghJgYPjt2J4KdYmcl26U0QsVuco=', 'Content-Length': '0'} 2023-11-02T18:30:38.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009830b92d6cda374d-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.134 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.135 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.136 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.136 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.136 DEBUG:boto:Method: GET 2023-11-02T18:30:38.137 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.137 DEBUG:boto:Data: 2023-11-02T18:30:38.138 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.139 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.139 DEBUG:boto:Params: {} 2023-11-02T18:30:38.140 DEBUG:boto:Token: None 2023-11-02T18:30:38.140 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ghJgYPjt2J4KdYmcl26U0QsVuco= 2023-11-02T18:30:38.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ghJgYPjt2J4KdYmcl26U0QsVuco=', 'Content-Length': '0'} 2023-11-02T18:30:38.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f2773269155546a-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.145 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.146 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.146 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.146 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.147 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.147 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.148 DEBUG:boto:Data: 2023-11-02T18:30:38.148 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.149 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.149 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.149 DEBUG:boto:Params: {} 2023-11-02T18:30:38.150 DEBUG:boto:Token: None 2023-11-02T18:30:38.150 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:98W0FaJpRkz+yPvp2PvCGaTX89s= 2023-11-02T18:30:38.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:98W0FaJpRkz+yPvp2PvCGaTX89s=', 'Content-Length': '0'} 2023-11-02T18:30:38.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a21199665a983ce9-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.155 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.155 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.156 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.156 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.157 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.157 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.157 DEBUG:boto:Data: 2023-11-02T18:30:38.158 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.158 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.159 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.159 DEBUG:boto:Params: {} 2023-11-02T18:30:38.160 DEBUG:boto:Token: None 2023-11-02T18:30:38.160 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.161 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:98W0FaJpRkz+yPvp2PvCGaTX89s= 2023-11-02T18:30:38.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:98W0FaJpRkz+yPvp2PvCGaTX89s=', 'Content-Length': '0'} 2023-11-02T18:30:38.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000738270d22371d16d-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-11-02T18:30:38.165 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.165 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.166 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.166 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.167 DEBUG:boto:Method: GET 2023-11-02T18:30:38.167 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.168 DEBUG:boto:Data: 2023-11-02T18:30:38.168 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.169 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.169 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.170 DEBUG:boto:Params: {} 2023-11-02T18:30:38.170 DEBUG:boto:Token: None 2023-11-02T18:30:38.171 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.171 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ghJgYPjt2J4KdYmcl26U0QsVuco= 2023-11-02T18:30:38.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ghJgYPjt2J4KdYmcl26U0QsVuco=', 'Content-Length': '0'} 2023-11-02T18:30:38.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5dd8f43525e7048-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.175 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.176 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.176 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.177 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.177 DEBUG:boto:Method: GET 2023-11-02T18:30:38.177 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.178 DEBUG:boto:Data: 2023-11-02T18:30:38.178 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.179 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.180 DEBUG:boto:Params: {} 2023-11-02T18:30:38.180 DEBUG:boto:Token: None 2023-11-02T18:30:38.181 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000060?versionId=null 2023-11-02T18:30:38.181 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ghJgYPjt2J4KdYmcl26U0QsVuco= 2023-11-02T18:30:38.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ghJgYPjt2J4KdYmcl26U0QsVuco=', 'Content-Length': '0'} 2023-11-02T18:30:38.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0ce6e40de1a3666-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.226 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.227 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.228 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000060 2023-11-02T18:30:38.228 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000060 2023-11-02T18:30:38.229 DEBUG:boto:Method: GET 2023-11-02T18:30:38.229 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000060 2023-11-02T18:30:38.229 DEBUG:boto:Data: 2023-11-02T18:30:38.230 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.230 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.231 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.231 DEBUG:boto:Params: {} 2023-11-02T18:30:38.232 DEBUG:boto:Token: None 2023-11-02T18:30:38.232 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.233 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0d6dcedf12d1de9-006543eace-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.239 DEBUG:boto:b'ibtljn-58obj000000000000000000601000falseobj000000000000000000602023-11-02T18:30:20.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.240 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.240 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.241 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000060 2023-11-02T18:30:38.241 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000060 2023-11-02T18:30:38.242 DEBUG:boto:Method: GET 2023-11-02T18:30:38.242 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000060 2023-11-02T18:30:38.243 DEBUG:boto:Data: 2023-11-02T18:30:38.243 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.244 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.244 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.245 DEBUG:boto:Params: {} 2023-11-02T18:30:38.245 DEBUG:boto:Token: None 2023-11-02T18:30:38.246 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.246 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c68ae936799a894-006543eace-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.257 DEBUG:boto:b'ibtljn-58obj000000000000000000601000falseobj000000000000000000602023-11-02T18:30:20.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.258 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.258 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.259 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.259 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.260 DEBUG:boto:Data: 2023-11-02T18:30:38.260 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.261 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.261 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.262 DEBUG:boto:Params: {} 2023-11-02T18:30:38.262 DEBUG:boto:Token: None 2023-11-02T18:30:38.263 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.263 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tjvmsZ0U7Amyqal8I7auYGs9/7o= 2023-11-02T18:30:38.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tjvmsZ0U7Amyqal8I7auYGs9/7o=', 'Content-Length': '0'} 2023-11-02T18:30:38.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ae7073f5f7de3b0-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.267 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.267 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.268 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.268 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.269 DEBUG:boto:Data: 2023-11-02T18:30:38.269 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.270 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.270 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.270 DEBUG:boto:Params: {} 2023-11-02T18:30:38.271 DEBUG:boto:Token: None 2023-11-02T18:30:38.271 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.272 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tjvmsZ0U7Amyqal8I7auYGs9/7o= 2023-11-02T18:30:38.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tjvmsZ0U7Amyqal8I7auYGs9/7o=', 'Content-Length': '0'} 2023-11-02T18:30:38.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c5e6c7b64cfe321-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-11-02T18:30:38.276 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.276 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.277 DEBUG:boto:Method: GET 2023-11-02T18:30:38.277 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.277 DEBUG:boto:Data: 2023-11-02T18:30:38.278 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.278 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.279 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.279 DEBUG:boto:Params: {} 2023-11-02T18:30:38.280 DEBUG:boto:Token: None 2023-11-02T18:30:38.280 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.281 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6zjEfKDtjHxSwRmm2at8GrYWs5g= 2023-11-02T18:30:38.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6zjEfKDtjHxSwRmm2at8GrYWs5g=', 'Content-Length': '0'} 2023-11-02T18:30:38.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000124198d1d829f714-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.330 DEBUG:boto:path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.331 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.331 DEBUG:boto:Method: GET 2023-11-02T18:30:38.332 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.332 DEBUG:boto:Data: 2023-11-02T18:30:38.333 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.333 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.334 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.334 DEBUG:boto:Params: {} 2023-11-02T18:30:38.335 DEBUG:boto:Token: None 2023-11-02T18:30:38.335 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000060 2023-11-02T18:30:38.336 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6zjEfKDtjHxSwRmm2at8GrYWs5g= 2023-11-02T18:30:38.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6zjEfKDtjHxSwRmm2at8GrYWs5g=', 'Content-Length': '0'} 2023-11-02T18:30:38.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a247b08f98152ba0-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.340 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.341 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.341 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000060 2023-11-02T18:30:38.341 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000060 2023-11-02T18:30:38.342 DEBUG:boto:Method: GET 2023-11-02T18:30:38.342 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000060 2023-11-02T18:30:38.343 DEBUG:boto:Data: 2023-11-02T18:30:38.343 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.344 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.344 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.344 DEBUG:boto:Params: {} 2023-11-02T18:30:38.345 DEBUG:boto:Token: None 2023-11-02T18:30:38.345 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.346 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b016317a41d614b-006543eace-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.352 DEBUG:boto:b'ibtljn-58obj000000000000000000601000falseobj000000000000000000602023-11-02T18:30:20.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.353 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.353 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.354 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000060 2023-11-02T18:30:38.354 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000060 2023-11-02T18:30:38.355 DEBUG:boto:Method: GET 2023-11-02T18:30:38.355 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000060 2023-11-02T18:30:38.356 DEBUG:boto:Data: 2023-11-02T18:30:38.356 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.357 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.357 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.358 DEBUG:boto:Params: {} 2023-11-02T18:30:38.358 DEBUG:boto:Token: None 2023-11-02T18:30:38.359 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.359 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000335333c4e98b7de3-006543eace-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.410 DEBUG:boto:b'ibtljn-58obj000000000000000000601000falseobj000000000000000000602023-11-02T18:30:20.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-11-02T18:30:38.412 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.412 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.413 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.413 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.414 DEBUG:boto:Method: GET 2023-11-02T18:30:38.414 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.415 DEBUG:boto:Data: 2023-11-02T18:30:38.416 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.416 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.417 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.417 DEBUG:boto:Params: {} 2023-11-02T18:30:38.418 DEBUG:boto:Token: None 2023-11-02T18:30:38.418 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.419 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:07aNoiPO73VKCAoWCMlA4JL8Nzo= 2023-11-02T18:30:38.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:07aNoiPO73VKCAoWCMlA4JL8Nzo=', 'Content-Length': '0'} 2023-11-02T18:30:38.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7cf4115b039dc17-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.423 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.424 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.424 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.425 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.425 DEBUG:boto:Method: GET 2023-11-02T18:30:38.426 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.426 DEBUG:boto:Data: 2023-11-02T18:30:38.427 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.427 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.428 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.428 DEBUG:boto:Params: {} 2023-11-02T18:30:38.429 DEBUG:boto:Token: None 2023-11-02T18:30:38.429 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.430 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:07aNoiPO73VKCAoWCMlA4JL8Nzo= 2023-11-02T18:30:38.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:07aNoiPO73VKCAoWCMlA4JL8Nzo=', 'Content-Length': '0'} 2023-11-02T18:30:38.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014fd131411eb16f5-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.434 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.434 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.435 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.435 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.436 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.436 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.436 DEBUG:boto:Data: 2023-11-02T18:30:38.437 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.438 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.438 DEBUG:boto:Params: {} 2023-11-02T18:30:38.439 DEBUG:boto:Token: None 2023-11-02T18:30:38.439 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.440 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0tWLlkQBaWVvL/A5dbFPJA8ea1s= 2023-11-02T18:30:38.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0tWLlkQBaWVvL/A5dbFPJA8ea1s=', 'Content-Length': '0'} 2023-11-02T18:30:38.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000391858e3f6871bc4-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.443 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.444 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.444 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.445 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.445 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.446 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.446 DEBUG:boto:Data: 2023-11-02T18:30:38.446 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.447 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.448 DEBUG:boto:Params: {} 2023-11-02T18:30:38.448 DEBUG:boto:Token: None 2023-11-02T18:30:38.449 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.449 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0tWLlkQBaWVvL/A5dbFPJA8ea1s= 2023-11-02T18:30:38.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0tWLlkQBaWVvL/A5dbFPJA8ea1s=', 'Content-Length': '0'} 2023-11-02T18:30:38.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a7b2a694e4eac38-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-11-02T18:30:38.456 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.456 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.457 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.457 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.457 DEBUG:boto:Method: GET 2023-11-02T18:30:38.458 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.458 DEBUG:boto:Data: 2023-11-02T18:30:38.459 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.459 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.460 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.460 DEBUG:boto:Params: {} 2023-11-02T18:30:38.461 DEBUG:boto:Token: None 2023-11-02T18:30:38.461 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.462 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:07aNoiPO73VKCAoWCMlA4JL8Nzo= 2023-11-02T18:30:38.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:07aNoiPO73VKCAoWCMlA4JL8Nzo=', 'Content-Length': '0'} 2023-11-02T18:30:38.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ab3d3963d78ac7-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.506 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.507 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.507 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.508 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.508 DEBUG:boto:Method: GET 2023-11-02T18:30:38.509 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.509 DEBUG:boto:Data: 2023-11-02T18:30:38.510 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.511 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.511 DEBUG:boto:Params: {} 2023-11-02T18:30:38.512 DEBUG:boto:Token: None 2023-11-02T18:30:38.512 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000061?versionId=null 2023-11-02T18:30:38.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:07aNoiPO73VKCAoWCMlA4JL8Nzo= 2023-11-02T18:30:38.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:07aNoiPO73VKCAoWCMlA4JL8Nzo=', 'Content-Length': '0'} 2023-11-02T18:30:38.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bf44512f3f70b97-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.518 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.518 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.518 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000061 2023-11-02T18:30:38.519 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000061 2023-11-02T18:30:38.519 DEBUG:boto:Method: GET 2023-11-02T18:30:38.520 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000061 2023-11-02T18:30:38.520 DEBUG:boto:Data: 2023-11-02T18:30:38.521 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.521 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.522 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.522 DEBUG:boto:Params: {} 2023-11-02T18:30:38.523 DEBUG:boto:Token: None 2023-11-02T18:30:38.523 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091c63e6d95427538-006543eace-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.531 DEBUG:boto:b'ibtljn-58obj000000000000000000611000falseobj000000000000000000612023-11-02T18:30:18.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.533 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.533 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.534 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000061 2023-11-02T18:30:38.534 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000061 2023-11-02T18:30:38.535 DEBUG:boto:Method: GET 2023-11-02T18:30:38.535 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000061 2023-11-02T18:30:38.536 DEBUG:boto:Data: 2023-11-02T18:30:38.536 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.537 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.537 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.538 DEBUG:boto:Params: {} 2023-11-02T18:30:38.538 DEBUG:boto:Token: None 2023-11-02T18:30:38.539 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.539 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab02cd43dddbde42-006543eace-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.551 DEBUG:boto:b'ibtljn-58obj000000000000000000611000falseobj000000000000000000612023-11-02T18:30:18.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.552 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.552 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.553 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.553 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.554 DEBUG:boto:Data: 2023-11-02T18:30:38.554 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.555 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.555 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.555 DEBUG:boto:Params: {} 2023-11-02T18:30:38.556 DEBUG:boto:Token: None 2023-11-02T18:30:38.556 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.557 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DnzgRJIot4mp34f1kXfiZ5kPSaQ= 2023-11-02T18:30:38.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DnzgRJIot4mp34f1kXfiZ5kPSaQ=', 'Content-Length': '0'} 2023-11-02T18:30:38.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9937a249c25093d-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.560 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.561 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.561 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.562 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.562 DEBUG:boto:Data: 2023-11-02T18:30:38.563 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.563 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.563 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.564 DEBUG:boto:Params: {} 2023-11-02T18:30:38.564 DEBUG:boto:Token: None 2023-11-02T18:30:38.565 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.565 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DnzgRJIot4mp34f1kXfiZ5kPSaQ= 2023-11-02T18:30:38.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DnzgRJIot4mp34f1kXfiZ5kPSaQ=', 'Content-Length': '0'} 2023-11-02T18:30:38.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6e20e90cacfd5d1-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-11-02T18:30:38.569 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.569 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.570 DEBUG:boto:Method: GET 2023-11-02T18:30:38.570 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.571 DEBUG:boto:Data: 2023-11-02T18:30:38.571 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.572 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.572 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.573 DEBUG:boto:Params: {} 2023-11-02T18:30:38.573 DEBUG:boto:Token: None 2023-11-02T18:30:38.573 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.574 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rr1SbW/r6KXmJOnKNthYdXJ+T48= 2023-11-02T18:30:38.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rr1SbW/r6KXmJOnKNthYdXJ+T48=', 'Content-Length': '0'} 2023-11-02T18:30:38.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7cfcdd218675225-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.578 DEBUG:boto:path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.578 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.579 DEBUG:boto:Method: GET 2023-11-02T18:30:38.579 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.580 DEBUG:boto:Data: 2023-11-02T18:30:38.580 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.581 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.581 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.581 DEBUG:boto:Params: {} 2023-11-02T18:30:38.582 DEBUG:boto:Token: None 2023-11-02T18:30:38.582 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000061 2023-11-02T18:30:38.583 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rr1SbW/r6KXmJOnKNthYdXJ+T48= 2023-11-02T18:30:38.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rr1SbW/r6KXmJOnKNthYdXJ+T48=', 'Content-Length': '0'} 2023-11-02T18:30:38.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1631ca87e544bf1-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.589 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.589 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.589 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000061 2023-11-02T18:30:38.590 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000061 2023-11-02T18:30:38.590 DEBUG:boto:Method: GET 2023-11-02T18:30:38.591 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000061 2023-11-02T18:30:38.591 DEBUG:boto:Data: 2023-11-02T18:30:38.592 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.593 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.594 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.594 DEBUG:boto:Params: {} 2023-11-02T18:30:38.595 DEBUG:boto:Token: None 2023-11-02T18:30:38.595 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.596 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faf16bff1410883f-006543eace-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.601 DEBUG:boto:b'ibtljn-58obj000000000000000000611000falseobj000000000000000000612023-11-02T18:30:18.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.601 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.602 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.602 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000061 2023-11-02T18:30:38.603 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000061 2023-11-02T18:30:38.603 DEBUG:boto:Method: GET 2023-11-02T18:30:38.604 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000061 2023-11-02T18:30:38.604 DEBUG:boto:Data: 2023-11-02T18:30:38.604 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.605 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.605 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.606 DEBUG:boto:Params: {} 2023-11-02T18:30:38.606 DEBUG:boto:Token: None 2023-11-02T18:30:38.607 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.607 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb0eeea41672aaa6-006543eace-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.658 DEBUG:boto:b'ibtljn-58obj000000000000000000611000falseobj000000000000000000612023-11-02T18:30:18.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-11-02T18:30:38.660 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.660 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.661 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.661 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.662 DEBUG:boto:Method: GET 2023-11-02T18:30:38.662 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.662 DEBUG:boto:Data: 2023-11-02T18:30:38.663 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.664 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.664 DEBUG:boto:Params: {} 2023-11-02T18:30:38.665 DEBUG:boto:Token: None 2023-11-02T18:30:38.665 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.666 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EaFO1RjU/+ZQzhynXHF4rCteCmU= 2023-11-02T18:30:38.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EaFO1RjU/+ZQzhynXHF4rCteCmU=', 'Content-Length': '0'} 2023-11-02T18:30:38.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a32522892973832-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.670 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.670 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.671 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.671 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.672 DEBUG:boto:Method: GET 2023-11-02T18:30:38.672 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.673 DEBUG:boto:Data: 2023-11-02T18:30:38.673 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.674 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.674 DEBUG:boto:Params: {} 2023-11-02T18:30:38.675 DEBUG:boto:Token: None 2023-11-02T18:30:38.676 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EaFO1RjU/+ZQzhynXHF4rCteCmU= 2023-11-02T18:30:38.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EaFO1RjU/+ZQzhynXHF4rCteCmU=', 'Content-Length': '0'} 2023-11-02T18:30:38.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071479c92e3d437ee-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.680 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.681 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.681 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.682 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.682 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.683 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.683 DEBUG:boto:Data: 2023-11-02T18:30:38.684 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.684 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.685 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.685 DEBUG:boto:Params: {} 2023-11-02T18:30:38.686 DEBUG:boto:Token: None 2023-11-02T18:30:38.687 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.687 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oQ7lOcRVNEXUsUllqJMXcqr2Rqc= 2023-11-02T18:30:38.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oQ7lOcRVNEXUsUllqJMXcqr2Rqc=', 'Content-Length': '0'} 2023-11-02T18:30:38.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090c47f5265f66562-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.692 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.692 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.693 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.693 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.693 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.694 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.694 DEBUG:boto:Data: 2023-11-02T18:30:38.695 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.696 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.696 DEBUG:boto:Params: {} 2023-11-02T18:30:38.697 DEBUG:boto:Token: None 2023-11-02T18:30:38.697 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.698 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oQ7lOcRVNEXUsUllqJMXcqr2Rqc= 2023-11-02T18:30:38.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oQ7lOcRVNEXUsUllqJMXcqr2Rqc=', 'Content-Length': '0'} 2023-11-02T18:30:38.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f8bc1aaa280de70-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-11-02T18:30:38.702 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.703 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.703 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.704 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.704 DEBUG:boto:Method: GET 2023-11-02T18:30:38.705 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.705 DEBUG:boto:Data: 2023-11-02T18:30:38.706 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.706 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.707 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.707 DEBUG:boto:Params: {} 2023-11-02T18:30:38.708 DEBUG:boto:Token: None 2023-11-02T18:30:38.708 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.709 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EaFO1RjU/+ZQzhynXHF4rCteCmU= 2023-11-02T18:30:38.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EaFO1RjU/+ZQzhynXHF4rCteCmU=', 'Content-Length': '0'} 2023-11-02T18:30:38.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2e1aac52fa1f736-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.754 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.755 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.755 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.756 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.756 DEBUG:boto:Method: GET 2023-11-02T18:30:38.757 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.757 DEBUG:boto:Data: 2023-11-02T18:30:38.758 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.758 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.759 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.759 DEBUG:boto:Params: {} 2023-11-02T18:30:38.760 DEBUG:boto:Token: None 2023-11-02T18:30:38.760 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000062?versionId=null 2023-11-02T18:30:38.761 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EaFO1RjU/+ZQzhynXHF4rCteCmU= 2023-11-02T18:30:38.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EaFO1RjU/+ZQzhynXHF4rCteCmU=', 'Content-Length': '0'} 2023-11-02T18:30:38.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a12b974ae5078d8a-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.765 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.765 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.766 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000062 2023-11-02T18:30:38.766 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000062 2023-11-02T18:30:38.767 DEBUG:boto:Method: GET 2023-11-02T18:30:38.767 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000062 2023-11-02T18:30:38.768 DEBUG:boto:Data: 2023-11-02T18:30:38.768 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.769 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.769 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.769 DEBUG:boto:Params: {} 2023-11-02T18:30:38.770 DEBUG:boto:Token: None 2023-11-02T18:30:38.771 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.771 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5b88613f3f6cbf6-006543eace-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.777 DEBUG:boto:b'ibtljn-58obj000000000000000000621000falseobj000000000000000000622023-11-02T18:30:20.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.778 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.778 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.779 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000062 2023-11-02T18:30:38.779 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000062 2023-11-02T18:30:38.780 DEBUG:boto:Method: GET 2023-11-02T18:30:38.780 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000062 2023-11-02T18:30:38.781 DEBUG:boto:Data: 2023-11-02T18:30:38.781 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.782 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.782 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.782 DEBUG:boto:Params: {} 2023-11-02T18:30:38.783 DEBUG:boto:Token: None 2023-11-02T18:30:38.783 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.784 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fda035ab4bca1c5d-006543eace-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.838 DEBUG:boto:b'ibtljn-58obj000000000000000000621000falseobj000000000000000000622023-11-02T18:30:20.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.839 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.840 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.840 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.841 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.841 DEBUG:boto:Data: 2023-11-02T18:30:38.841 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.842 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.842 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.843 DEBUG:boto:Params: {} 2023-11-02T18:30:38.843 DEBUG:boto:Token: None 2023-11-02T18:30:38.844 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.844 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7QtH2khGYb1TMIhn9AmxxgwEIVM= 2023-11-02T18:30:38.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7QtH2khGYb1TMIhn9AmxxgwEIVM=', 'Content-Length': '0'} 2023-11-02T18:30:38.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c0b3ae21a62b474-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.849 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.850 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.850 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.851 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.851 DEBUG:boto:Data: 2023-11-02T18:30:38.851 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.852 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.853 DEBUG:boto:Params: {} 2023-11-02T18:30:38.853 DEBUG:boto:Token: None 2023-11-02T18:30:38.854 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.854 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7QtH2khGYb1TMIhn9AmxxgwEIVM= 2023-11-02T18:30:38.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7QtH2khGYb1TMIhn9AmxxgwEIVM=', 'Content-Length': '0'} 2023-11-02T18:30:38.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eecf1e04043e803f-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-11-02T18:30:38.859 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.859 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.860 DEBUG:boto:Method: GET 2023-11-02T18:30:38.860 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.860 DEBUG:boto:Data: 2023-11-02T18:30:38.861 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.861 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.862 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.862 DEBUG:boto:Params: {} 2023-11-02T18:30:38.863 DEBUG:boto:Token: None 2023-11-02T18:30:38.863 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.864 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Sdtztptb4Fo5Ug4RSTd9QA2Uqxo= 2023-11-02T18:30:38.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Sdtztptb4Fo5Ug4RSTd9QA2Uqxo=', 'Content-Length': '0'} 2023-11-02T18:30:38.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a8c8958f6e88b32-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.868 DEBUG:boto:path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.869 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.869 DEBUG:boto:Method: GET 2023-11-02T18:30:38.870 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.870 DEBUG:boto:Data: 2023-11-02T18:30:38.870 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.871 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.871 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.872 DEBUG:boto:Params: {} 2023-11-02T18:30:38.872 DEBUG:boto:Token: None 2023-11-02T18:30:38.873 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000062 2023-11-02T18:30:38.873 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Sdtztptb4Fo5Ug4RSTd9QA2Uqxo= 2023-11-02T18:30:38.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Sdtztptb4Fo5Ug4RSTd9QA2Uqxo=', 'Content-Length': '0'} 2023-11-02T18:30:38.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052cf848620ce3c45-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.877 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.878 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.881 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000062 2023-11-02T18:30:38.881 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000062 2023-11-02T18:30:38.882 DEBUG:boto:Method: GET 2023-11-02T18:30:38.882 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000062 2023-11-02T18:30:38.883 DEBUG:boto:Data: 2023-11-02T18:30:38.883 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.884 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.884 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.884 DEBUG:boto:Params: {} 2023-11-02T18:30:38.885 DEBUG:boto:Token: None 2023-11-02T18:30:38.885 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.886 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021744844ac9896e9-006543eace-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.934 DEBUG:boto:b'ibtljn-58obj000000000000000000621000falseobj000000000000000000622023-11-02T18:30:20.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.935 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:38.936 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:38.936 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000062 2023-11-02T18:30:38.937 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000062 2023-11-02T18:30:38.937 DEBUG:boto:Method: GET 2023-11-02T18:30:38.938 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000062 2023-11-02T18:30:38.938 DEBUG:boto:Data: 2023-11-02T18:30:38.939 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.939 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.939 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.940 DEBUG:boto:Params: {} 2023-11-02T18:30:38.940 DEBUG:boto:Token: None 2023-11-02T18:30:38.941 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/ 2023-11-02T18:30:38.941 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0= 2023-11-02T18:30:38.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Duo5Z+2VHRaur3ISF5FTkwuqtn0=', 'Content-Length': '0'} 2023-11-02T18:30:38.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d4af08f70c4b054-006543eace-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.953 DEBUG:boto:b'ibtljn-58obj000000000000000000621000falseobj000000000000000000622023-11-02T18:30:20.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:38.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-11-02T18:30:38.954 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:38.955 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:38.955 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.956 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.956 DEBUG:boto:Method: GET 2023-11-02T18:30:38.956 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.957 DEBUG:boto:Data: 2023-11-02T18:30:38.957 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.958 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.958 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.959 DEBUG:boto:Params: {} 2023-11-02T18:30:38.959 DEBUG:boto:Token: None 2023-11-02T18:30:38.960 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.960 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vLdyBIV8r5KBR839JVt3yZJDneQ= 2023-11-02T18:30:38.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vLdyBIV8r5KBR839JVt3yZJDneQ=', 'Content-Length': '0'} 2023-11-02T18:30:38.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000400f70cb6b2fc3fb-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.964 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:38.965 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:38.965 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.966 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.966 DEBUG:boto:Method: GET 2023-11-02T18:30:38.967 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.967 DEBUG:boto:Data: 2023-11-02T18:30:38.967 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.968 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.968 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.969 DEBUG:boto:Params: {} 2023-11-02T18:30:38.969 DEBUG:boto:Token: None 2023-11-02T18:30:38.970 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.970 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vLdyBIV8r5KBR839JVt3yZJDneQ= 2023-11-02T18:30:38.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vLdyBIV8r5KBR839JVt3yZJDneQ=', 'Content-Length': '0'} 2023-11-02T18:30:38.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075f9c3dbe96df2d9-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.974 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:38.974 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:38.975 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.975 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.976 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.976 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.976 DEBUG:boto:Data: 2023-11-02T18:30:38.977 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.977 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.978 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.978 DEBUG:boto:Params: {} 2023-11-02T18:30:38.979 DEBUG:boto:Token: None 2023-11-02T18:30:38.979 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.979 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mLMWIKk7u7e0MWsMy2HyVZ3AmsY= 2023-11-02T18:30:38.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mLMWIKk7u7e0MWsMy2HyVZ3AmsY=', 'Content-Length': '0'} 2023-11-02T18:30:38.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b59529c8d182b68c-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.983 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:38.984 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:38.984 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.984 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.985 DEBUG:boto:Method: HEAD 2023-11-02T18:30:38.985 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.986 DEBUG:boto:Data: 2023-11-02T18:30:38.986 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.987 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:38.987 DEBUG:boto:Port: 8000 2023-11-02T18:30:38.987 DEBUG:boto:Params: {} 2023-11-02T18:30:38.988 DEBUG:boto:Token: None 2023-11-02T18:30:38.988 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.989 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mLMWIKk7u7e0MWsMy2HyVZ3AmsY= 2023-11-02T18:30:38.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mLMWIKk7u7e0MWsMy2HyVZ3AmsY=', 'Content-Length': '0'} 2023-11-02T18:30:38.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c327bd493f5c6ac4-006543eace-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:38.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-11-02T18:30:38.993 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:38.994 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:38.994 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.995 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.995 DEBUG:boto:Method: GET 2023-11-02T18:30:38.995 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.996 DEBUG:boto:Data: 2023-11-02T18:30:38.996 DEBUG:boto:Headers: {} 2023-11-02T18:30:38.997 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:38.997 DEBUG:boto:Port: 8001 2023-11-02T18:30:38.998 DEBUG:boto:Params: {} 2023-11-02T18:30:38.998 DEBUG:boto:Token: None 2023-11-02T18:30:38.999 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:38 GMT /ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:38.999 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vLdyBIV8r5KBR839JVt3yZJDneQ= 2023-11-02T18:30:39.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vLdyBIV8r5KBR839JVt3yZJDneQ=', 'Content-Length': '0'} 2023-11-02T18:30:39.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb5df62b5e24cc2a-006543eace-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.046 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.047 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.047 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:39.048 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:39.048 DEBUG:boto:Method: GET 2023-11-02T18:30:39.049 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:39.049 DEBUG:boto:Data: 2023-11-02T18:30:39.050 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.050 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.051 DEBUG:boto:Params: {} 2023-11-02T18:30:39.051 DEBUG:boto:Token: None 2023-11-02T18:30:39.052 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000065?versionId=null 2023-11-02T18:30:39.052 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:56IXI7oWUPojZRvkLZD/Q5LxqhE= 2023-11-02T18:30:39.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:56IXI7oWUPojZRvkLZD/Q5LxqhE=', 'Content-Length': '0'} 2023-11-02T18:30:39.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e2c4c07e3c0b4d-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.057 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.057 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.058 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000065 2023-11-02T18:30:39.058 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000065 2023-11-02T18:30:39.059 DEBUG:boto:Method: GET 2023-11-02T18:30:39.059 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000065 2023-11-02T18:30:39.059 DEBUG:boto:Data: 2023-11-02T18:30:39.060 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.060 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.061 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.061 DEBUG:boto:Params: {} 2023-11-02T18:30:39.062 DEBUG:boto:Token: None 2023-11-02T18:30:39.062 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.063 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000643ee5c329c2e26f-006543eacf-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.069 DEBUG:boto:b'ibtljn-58obj000000000000000000651000falseobj000000000000000000652023-11-02T18:30:20.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.069 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.070 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.070 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000065 2023-11-02T18:30:39.071 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000065 2023-11-02T18:30:39.071 DEBUG:boto:Method: GET 2023-11-02T18:30:39.072 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000065 2023-11-02T18:30:39.072 DEBUG:boto:Data: 2023-11-02T18:30:39.072 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.073 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.073 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.074 DEBUG:boto:Params: {} 2023-11-02T18:30:39.074 DEBUG:boto:Token: None 2023-11-02T18:30:39.075 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.075 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f81966deb6ffaa15-006543eacf-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.126 DEBUG:boto:b'ibtljn-58obj000000000000000000651000falseobj000000000000000000652023-11-02T18:30:20.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.127 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.128 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.128 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.129 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.129 DEBUG:boto:Data: 2023-11-02T18:30:39.129 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.130 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.130 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.131 DEBUG:boto:Params: {} 2023-11-02T18:30:39.131 DEBUG:boto:Token: None 2023-11-02T18:30:39.132 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.132 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2YKbB6XhlNdD2AGcUOgN/qi/Jx0= 2023-11-02T18:30:39.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2YKbB6XhlNdD2AGcUOgN/qi/Jx0=', 'Content-Length': '0'} 2023-11-02T18:30:39.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000397093bd36256b7e-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.136 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.137 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.137 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.138 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.138 DEBUG:boto:Data: 2023-11-02T18:30:39.139 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.139 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.140 DEBUG:boto:Params: {} 2023-11-02T18:30:39.140 DEBUG:boto:Token: None 2023-11-02T18:30:39.141 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2YKbB6XhlNdD2AGcUOgN/qi/Jx0= 2023-11-02T18:30:39.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2YKbB6XhlNdD2AGcUOgN/qi/Jx0=', 'Content-Length': '0'} 2023-11-02T18:30:39.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076506d925b536573-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-11-02T18:30:39.145 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.146 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.146 DEBUG:boto:Method: GET 2023-11-02T18:30:39.147 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.147 DEBUG:boto:Data: 2023-11-02T18:30:39.148 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.149 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.149 DEBUG:boto:Params: {} 2023-11-02T18:30:39.150 DEBUG:boto:Token: None 2023-11-02T18:30:39.150 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ozTtfbYHvU7sZ3o/tSDzd3nxmR8= 2023-11-02T18:30:39.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ozTtfbYHvU7sZ3o/tSDzd3nxmR8=', 'Content-Length': '0'} 2023-11-02T18:30:39.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7862795d30c4edc-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.155 DEBUG:boto:path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.155 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.156 DEBUG:boto:Method: GET 2023-11-02T18:30:39.156 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.157 DEBUG:boto:Data: 2023-11-02T18:30:39.157 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.158 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.158 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.158 DEBUG:boto:Params: {} 2023-11-02T18:30:39.159 DEBUG:boto:Token: None 2023-11-02T18:30:39.159 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000065 2023-11-02T18:30:39.160 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ozTtfbYHvU7sZ3o/tSDzd3nxmR8= 2023-11-02T18:30:39.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ozTtfbYHvU7sZ3o/tSDzd3nxmR8=', 'Content-Length': '0'} 2023-11-02T18:30:39.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000155f6d9002a6b332-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.164 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.165 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.165 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000065 2023-11-02T18:30:39.166 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000065 2023-11-02T18:30:39.166 DEBUG:boto:Method: GET 2023-11-02T18:30:39.166 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000065 2023-11-02T18:30:39.167 DEBUG:boto:Data: 2023-11-02T18:30:39.167 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.168 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.168 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.169 DEBUG:boto:Params: {} 2023-11-02T18:30:39.169 DEBUG:boto:Token: None 2023-11-02T18:30:39.170 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.170 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b88e084d075097c0-006543eacf-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.218 DEBUG:boto:b'ibtljn-58obj000000000000000000651000falseobj000000000000000000652023-11-02T18:30:20.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.219 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.220 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.220 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000065 2023-11-02T18:30:39.221 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000065 2023-11-02T18:30:39.221 DEBUG:boto:Method: GET 2023-11-02T18:30:39.221 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000065 2023-11-02T18:30:39.222 DEBUG:boto:Data: 2023-11-02T18:30:39.222 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.223 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.224 DEBUG:boto:Params: {} 2023-11-02T18:30:39.224 DEBUG:boto:Token: None 2023-11-02T18:30:39.225 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc82e631b9a59321-006543eacf-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.236 DEBUG:boto:b'ibtljn-58obj000000000000000000651000falseobj000000000000000000652023-11-02T18:30:20.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-11-02T18:30:39.238 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.238 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.239 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.239 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.240 DEBUG:boto:Method: GET 2023-11-02T18:30:39.240 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.240 DEBUG:boto:Data: 2023-11-02T18:30:39.241 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.241 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.242 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.242 DEBUG:boto:Params: {} 2023-11-02T18:30:39.243 DEBUG:boto:Token: None 2023-11-02T18:30:39.243 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.244 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:55ZskOIuDnW3xNMI1+5tqHhcB1w= 2023-11-02T18:30:39.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:55ZskOIuDnW3xNMI1+5tqHhcB1w=', 'Content-Length': '0'} 2023-11-02T18:30:39.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000696a67c0ebdb7529-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.247 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.248 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.248 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.249 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.249 DEBUG:boto:Method: GET 2023-11-02T18:30:39.250 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.250 DEBUG:boto:Data: 2023-11-02T18:30:39.251 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.251 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.251 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.252 DEBUG:boto:Params: {} 2023-11-02T18:30:39.252 DEBUG:boto:Token: None 2023-11-02T18:30:39.253 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.253 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:55ZskOIuDnW3xNMI1+5tqHhcB1w= 2023-11-02T18:30:39.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:55ZskOIuDnW3xNMI1+5tqHhcB1w=', 'Content-Length': '0'} 2023-11-02T18:30:39.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c79b89b44a8b75f-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.257 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.258 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.258 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.259 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.259 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.259 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.260 DEBUG:boto:Data: 2023-11-02T18:30:39.260 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.261 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.261 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.262 DEBUG:boto:Params: {} 2023-11-02T18:30:39.262 DEBUG:boto:Token: None 2023-11-02T18:30:39.262 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.263 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V7trq8e1xc/Jq747HV6aG1m7CpU= 2023-11-02T18:30:39.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V7trq8e1xc/Jq747HV6aG1m7CpU=', 'Content-Length': '0'} 2023-11-02T18:30:39.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000705f3327b2b3bb6c-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.267 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.267 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.268 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.268 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.269 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.269 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.269 DEBUG:boto:Data: 2023-11-02T18:30:39.270 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.270 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.271 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.271 DEBUG:boto:Params: {} 2023-11-02T18:30:39.272 DEBUG:boto:Token: None 2023-11-02T18:30:39.272 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.272 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V7trq8e1xc/Jq747HV6aG1m7CpU= 2023-11-02T18:30:39.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V7trq8e1xc/Jq747HV6aG1m7CpU=', 'Content-Length': '0'} 2023-11-02T18:30:39.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000793e3d1965437a7a-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-11-02T18:30:39.277 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.277 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.278 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.278 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.279 DEBUG:boto:Method: GET 2023-11-02T18:30:39.279 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.280 DEBUG:boto:Data: 2023-11-02T18:30:39.280 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.280 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.281 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.281 DEBUG:boto:Params: {} 2023-11-02T18:30:39.282 DEBUG:boto:Token: None 2023-11-02T18:30:39.282 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.283 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:55ZskOIuDnW3xNMI1+5tqHhcB1w= 2023-11-02T18:30:39.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:55ZskOIuDnW3xNMI1+5tqHhcB1w=', 'Content-Length': '0'} 2023-11-02T18:30:39.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a957599b261cf2b1-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.330 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.331 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.331 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.332 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.332 DEBUG:boto:Method: GET 2023-11-02T18:30:39.333 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.333 DEBUG:boto:Data: 2023-11-02T18:30:39.334 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.334 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.335 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.335 DEBUG:boto:Params: {} 2023-11-02T18:30:39.336 DEBUG:boto:Token: None 2023-11-02T18:30:39.336 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000076?versionId=null 2023-11-02T18:30:39.337 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:55ZskOIuDnW3xNMI1+5tqHhcB1w= 2023-11-02T18:30:39.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:55ZskOIuDnW3xNMI1+5tqHhcB1w=', 'Content-Length': '0'} 2023-11-02T18:30:39.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a4e408425a3d707-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.342 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.342 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.343 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000076 2023-11-02T18:30:39.343 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000076 2023-11-02T18:30:39.344 DEBUG:boto:Method: GET 2023-11-02T18:30:39.344 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000076 2023-11-02T18:30:39.345 DEBUG:boto:Data: 2023-11-02T18:30:39.345 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.346 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.346 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.347 DEBUG:boto:Params: {} 2023-11-02T18:30:39.347 DEBUG:boto:Token: None 2023-11-02T18:30:39.348 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.348 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcb8e7aa6c02a9ad-006543eacf-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.354 DEBUG:boto:b'ibtljn-58obj000000000000000000761000falseobj000000000000000000762023-11-02T18:30:12.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.355 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.356 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.356 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000076 2023-11-02T18:30:39.357 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000076 2023-11-02T18:30:39.357 DEBUG:boto:Method: GET 2023-11-02T18:30:39.358 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000076 2023-11-02T18:30:39.358 DEBUG:boto:Data: 2023-11-02T18:30:39.359 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.359 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.360 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.360 DEBUG:boto:Params: {} 2023-11-02T18:30:39.361 DEBUG:boto:Token: None 2023-11-02T18:30:39.361 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.362 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a02be827c4800366-006543eacf-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.371 DEBUG:boto:b'ibtljn-58obj000000000000000000761000falseobj000000000000000000762023-11-02T18:30:12.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.372 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.373 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.373 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.374 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.374 DEBUG:boto:Data: 2023-11-02T18:30:39.374 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.375 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.375 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.376 DEBUG:boto:Params: {} 2023-11-02T18:30:39.376 DEBUG:boto:Token: None 2023-11-02T18:30:39.377 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.377 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:suWkPakkNWHepyZH2hzWtquU1y4= 2023-11-02T18:30:39.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:suWkPakkNWHepyZH2hzWtquU1y4=', 'Content-Length': '0'} 2023-11-02T18:30:39.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000107117481f9ae553-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.381 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.382 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.382 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.383 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.383 DEBUG:boto:Data: 2023-11-02T18:30:39.384 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.384 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.385 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.385 DEBUG:boto:Params: {} 2023-11-02T18:30:39.386 DEBUG:boto:Token: None 2023-11-02T18:30:39.386 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.387 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:suWkPakkNWHepyZH2hzWtquU1y4= 2023-11-02T18:30:39.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:suWkPakkNWHepyZH2hzWtquU1y4=', 'Content-Length': '0'} 2023-11-02T18:30:39.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000430e075b0fd2fcae-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-11-02T18:30:39.391 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.392 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.392 DEBUG:boto:Method: GET 2023-11-02T18:30:39.393 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.393 DEBUG:boto:Data: 2023-11-02T18:30:39.394 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.394 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.395 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.395 DEBUG:boto:Params: {} 2023-11-02T18:30:39.396 DEBUG:boto:Token: None 2023-11-02T18:30:39.397 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.397 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tr0gPGK5EJbwVOJ5HatrmQTcFQo= 2023-11-02T18:30:39.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tr0gPGK5EJbwVOJ5HatrmQTcFQo=', 'Content-Length': '0'} 2023-11-02T18:30:39.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eef2bb41481ca1f-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.402 DEBUG:boto:path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.402 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.403 DEBUG:boto:Method: GET 2023-11-02T18:30:39.403 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.404 DEBUG:boto:Data: 2023-11-02T18:30:39.404 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.405 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.405 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.406 DEBUG:boto:Params: {} 2023-11-02T18:30:39.406 DEBUG:boto:Token: None 2023-11-02T18:30:39.407 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000076 2023-11-02T18:30:39.408 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tr0gPGK5EJbwVOJ5HatrmQTcFQo= 2023-11-02T18:30:39.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tr0gPGK5EJbwVOJ5HatrmQTcFQo=', 'Content-Length': '0'} 2023-11-02T18:30:39.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d78592002655e334-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.454 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.455 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.455 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000076 2023-11-02T18:30:39.456 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000076 2023-11-02T18:30:39.456 DEBUG:boto:Method: GET 2023-11-02T18:30:39.457 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000076 2023-11-02T18:30:39.458 DEBUG:boto:Data: 2023-11-02T18:30:39.458 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.459 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.459 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.460 DEBUG:boto:Params: {} 2023-11-02T18:30:39.460 DEBUG:boto:Token: None 2023-11-02T18:30:39.461 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.461 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8e386f1bfd550bb-006543eacf-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.468 DEBUG:boto:b'ibtljn-58obj000000000000000000761000falseobj000000000000000000762023-11-02T18:30:12.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.469 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.469 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.470 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000076 2023-11-02T18:30:39.470 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000076 2023-11-02T18:30:39.471 DEBUG:boto:Method: GET 2023-11-02T18:30:39.471 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000076 2023-11-02T18:30:39.472 DEBUG:boto:Data: 2023-11-02T18:30:39.472 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.473 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.473 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.474 DEBUG:boto:Params: {} 2023-11-02T18:30:39.474 DEBUG:boto:Token: None 2023-11-02T18:30:39.475 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.475 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfc00d9a347cd009-006543eacf-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.487 DEBUG:boto:b'ibtljn-58obj000000000000000000761000falseobj000000000000000000762023-11-02T18:30:12.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-11-02T18:30:39.488 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.489 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.489 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.490 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.490 DEBUG:boto:Method: GET 2023-11-02T18:30:39.491 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.491 DEBUG:boto:Data: 2023-11-02T18:30:39.492 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.492 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.493 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.493 DEBUG:boto:Params: {} 2023-11-02T18:30:39.494 DEBUG:boto:Token: None 2023-11-02T18:30:39.495 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.495 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RZZZz9WNYQYibPnJznoWGMbjQqc= 2023-11-02T18:30:39.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RZZZz9WNYQYibPnJznoWGMbjQqc=', 'Content-Length': '0'} 2023-11-02T18:30:39.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000480b1e3783ae7309-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.546 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.547 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.548 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.548 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.548 DEBUG:boto:Method: GET 2023-11-02T18:30:39.549 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.549 DEBUG:boto:Data: 2023-11-02T18:30:39.550 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.550 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.551 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.551 DEBUG:boto:Params: {} 2023-11-02T18:30:39.552 DEBUG:boto:Token: None 2023-11-02T18:30:39.552 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.553 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RZZZz9WNYQYibPnJznoWGMbjQqc= 2023-11-02T18:30:39.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RZZZz9WNYQYibPnJznoWGMbjQqc=', 'Content-Length': '0'} 2023-11-02T18:30:39.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cce5e3f9de2036ed-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.558 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.559 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.559 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.560 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.560 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.560 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.561 DEBUG:boto:Data: 2023-11-02T18:30:39.561 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.562 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.562 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.563 DEBUG:boto:Params: {} 2023-11-02T18:30:39.563 DEBUG:boto:Token: None 2023-11-02T18:30:39.563 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.564 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xPmM42igYQuSqXzlZkXMn5NyKsE= 2023-11-02T18:30:39.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xPmM42igYQuSqXzlZkXMn5NyKsE=', 'Content-Length': '0'} 2023-11-02T18:30:39.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9d704437f5935f2-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.568 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.568 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.568 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.569 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.569 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.570 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.570 DEBUG:boto:Data: 2023-11-02T18:30:39.570 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.571 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.571 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.572 DEBUG:boto:Params: {} 2023-11-02T18:30:39.572 DEBUG:boto:Token: None 2023-11-02T18:30:39.573 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.573 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xPmM42igYQuSqXzlZkXMn5NyKsE= 2023-11-02T18:30:39.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xPmM42igYQuSqXzlZkXMn5NyKsE=', 'Content-Length': '0'} 2023-11-02T18:30:39.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edbd279883f44193-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-11-02T18:30:39.577 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.578 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.578 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.579 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.579 DEBUG:boto:Method: GET 2023-11-02T18:30:39.580 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.580 DEBUG:boto:Data: 2023-11-02T18:30:39.580 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.581 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.581 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.582 DEBUG:boto:Params: {} 2023-11-02T18:30:39.582 DEBUG:boto:Token: None 2023-11-02T18:30:39.583 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.583 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RZZZz9WNYQYibPnJznoWGMbjQqc= 2023-11-02T18:30:39.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RZZZz9WNYQYibPnJznoWGMbjQqc=', 'Content-Length': '0'} 2023-11-02T18:30:39.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000875decbcb14d3a61-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.587 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.588 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.588 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.588 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.589 DEBUG:boto:Method: GET 2023-11-02T18:30:39.589 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.590 DEBUG:boto:Data: 2023-11-02T18:30:39.590 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.591 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.591 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.591 DEBUG:boto:Params: {} 2023-11-02T18:30:39.592 DEBUG:boto:Token: None 2023-11-02T18:30:39.592 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000133?versionId=null 2023-11-02T18:30:39.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RZZZz9WNYQYibPnJznoWGMbjQqc= 2023-11-02T18:30:39.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RZZZz9WNYQYibPnJznoWGMbjQqc=', 'Content-Length': '0'} 2023-11-02T18:30:39.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2c3d9dac51a7ccd-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.597 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.597 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.598 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000133 2023-11-02T18:30:39.598 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000133 2023-11-02T18:30:39.599 DEBUG:boto:Method: GET 2023-11-02T18:30:39.599 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000133 2023-11-02T18:30:39.600 DEBUG:boto:Data: 2023-11-02T18:30:39.600 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.601 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.601 DEBUG:boto:Params: {} 2023-11-02T18:30:39.602 DEBUG:boto:Token: None 2023-11-02T18:30:39.602 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.603 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a4c01ab5ed59779-006543eacf-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.650 DEBUG:boto:b'ibtljn-58obj000000000000000001331000falseobj000000000000000001332023-11-02T18:30:18.467Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.651 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.652 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.652 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000133 2023-11-02T18:30:39.653 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000133 2023-11-02T18:30:39.653 DEBUG:boto:Method: GET 2023-11-02T18:30:39.653 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000133 2023-11-02T18:30:39.654 DEBUG:boto:Data: 2023-11-02T18:30:39.654 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.655 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.655 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.656 DEBUG:boto:Params: {} 2023-11-02T18:30:39.656 DEBUG:boto:Token: None 2023-11-02T18:30:39.657 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.657 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000296b7202f4672ba6-006543eacf-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.668 DEBUG:boto:b'ibtljn-58obj000000000000000001331000falseobj000000000000000001332023-11-02T18:30:18.467Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.668 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.669 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.669 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.670 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.670 DEBUG:boto:Data: 2023-11-02T18:30:39.671 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.671 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.672 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.672 DEBUG:boto:Params: {} 2023-11-02T18:30:39.673 DEBUG:boto:Token: None 2023-11-02T18:30:39.673 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.673 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PrshypVAKSJGonPz3DN7Q1HAAAE= 2023-11-02T18:30:39.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PrshypVAKSJGonPz3DN7Q1HAAAE=', 'Content-Length': '0'} 2023-11-02T18:30:39.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000514d289a7c162626-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.677 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.678 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.678 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.678 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.679 DEBUG:boto:Data: 2023-11-02T18:30:39.679 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.680 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.681 DEBUG:boto:Params: {} 2023-11-02T18:30:39.681 DEBUG:boto:Token: None 2023-11-02T18:30:39.681 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.682 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PrshypVAKSJGonPz3DN7Q1HAAAE= 2023-11-02T18:30:39.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PrshypVAKSJGonPz3DN7Q1HAAAE=', 'Content-Length': '0'} 2023-11-02T18:30:39.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f64b9387de7d275-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-11-02T18:30:39.686 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.686 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.687 DEBUG:boto:Method: GET 2023-11-02T18:30:39.687 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.688 DEBUG:boto:Data: 2023-11-02T18:30:39.688 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.689 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.689 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.690 DEBUG:boto:Params: {} 2023-11-02T18:30:39.690 DEBUG:boto:Token: None 2023-11-02T18:30:39.691 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.691 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/TlC/CEQTX30/cxg4E6EAfl27+g= 2023-11-02T18:30:39.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/TlC/CEQTX30/cxg4E6EAfl27+g=', 'Content-Length': '0'} 2023-11-02T18:30:39.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e0edc5d2b0f99f3-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.695 DEBUG:boto:path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.695 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.696 DEBUG:boto:Method: GET 2023-11-02T18:30:39.696 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.697 DEBUG:boto:Data: 2023-11-02T18:30:39.697 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.698 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.698 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.698 DEBUG:boto:Params: {} 2023-11-02T18:30:39.699 DEBUG:boto:Token: None 2023-11-02T18:30:39.699 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000133 2023-11-02T18:30:39.701 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/TlC/CEQTX30/cxg4E6EAfl27+g= 2023-11-02T18:30:39.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/TlC/CEQTX30/cxg4E6EAfl27+g=', 'Content-Length': '0'} 2023-11-02T18:30:39.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000992cd8203991838e-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.746 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.747 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.747 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000133 2023-11-02T18:30:39.748 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000133 2023-11-02T18:30:39.748 DEBUG:boto:Method: GET 2023-11-02T18:30:39.749 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000133 2023-11-02T18:30:39.749 DEBUG:boto:Data: 2023-11-02T18:30:39.750 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.750 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.751 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.751 DEBUG:boto:Params: {} 2023-11-02T18:30:39.752 DEBUG:boto:Token: None 2023-11-02T18:30:39.752 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.753 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009122d636c7b5437-006543eacf-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.760 DEBUG:boto:b'ibtljn-58obj000000000000000001331000falseobj000000000000000001332023-11-02T18:30:18.467Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.760 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.761 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.761 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000133 2023-11-02T18:30:39.762 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000133 2023-11-02T18:30:39.762 DEBUG:boto:Method: GET 2023-11-02T18:30:39.763 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000133 2023-11-02T18:30:39.763 DEBUG:boto:Data: 2023-11-02T18:30:39.763 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.764 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.765 DEBUG:boto:Params: {} 2023-11-02T18:30:39.765 DEBUG:boto:Token: None 2023-11-02T18:30:39.766 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddf99fb766e685b8-006543eacf-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.778 DEBUG:boto:b'ibtljn-58obj000000000000000001331000falseobj000000000000000001332023-11-02T18:30:18.467Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-11-02T18:30:39.779 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.780 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.780 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.781 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.781 DEBUG:boto:Method: GET 2023-11-02T18:30:39.781 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.782 DEBUG:boto:Data: 2023-11-02T18:30:39.782 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.783 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.783 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.784 DEBUG:boto:Params: {} 2023-11-02T18:30:39.784 DEBUG:boto:Token: None 2023-11-02T18:30:39.784 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HjUiFwYzudUAEdI0XStP6sWXCMY= 2023-11-02T18:30:39.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HjUiFwYzudUAEdI0XStP6sWXCMY=', 'Content-Length': '0'} 2023-11-02T18:30:39.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fca6ec5bf3373be2-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.789 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.790 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.790 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.791 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.791 DEBUG:boto:Method: GET 2023-11-02T18:30:39.792 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.793 DEBUG:boto:Data: 2023-11-02T18:30:39.793 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.794 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.794 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.795 DEBUG:boto:Params: {} 2023-11-02T18:30:39.795 DEBUG:boto:Token: None 2023-11-02T18:30:39.796 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.796 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HjUiFwYzudUAEdI0XStP6sWXCMY= 2023-11-02T18:30:39.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HjUiFwYzudUAEdI0XStP6sWXCMY=', 'Content-Length': '0'} 2023-11-02T18:30:39.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000702071a28388bbba-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.801 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.801 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.802 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.802 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.803 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.803 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.804 DEBUG:boto:Data: 2023-11-02T18:30:39.804 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.805 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.805 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.806 DEBUG:boto:Params: {} 2023-11-02T18:30:39.806 DEBUG:boto:Token: None 2023-11-02T18:30:39.807 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.807 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X3YQV7SK1B2opkt9JJ5w7L+g6XM= 2023-11-02T18:30:39.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X3YQV7SK1B2opkt9JJ5w7L+g6XM=', 'Content-Length': '0'} 2023-11-02T18:30:39.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000819eae1d2af97be1-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.811 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.812 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.812 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.812 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.813 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.813 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.814 DEBUG:boto:Data: 2023-11-02T18:30:39.814 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.815 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.815 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.815 DEBUG:boto:Params: {} 2023-11-02T18:30:39.816 DEBUG:boto:Token: None 2023-11-02T18:30:39.816 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.817 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X3YQV7SK1B2opkt9JJ5w7L+g6XM= 2023-11-02T18:30:39.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X3YQV7SK1B2opkt9JJ5w7L+g6XM=', 'Content-Length': '0'} 2023-11-02T18:30:39.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098eb2fda0d7acb69-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-11-02T18:30:39.821 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.822 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.822 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.823 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.823 DEBUG:boto:Method: GET 2023-11-02T18:30:39.823 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.824 DEBUG:boto:Data: 2023-11-02T18:30:39.824 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.825 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.825 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.826 DEBUG:boto:Params: {} 2023-11-02T18:30:39.827 DEBUG:boto:Token: None 2023-11-02T18:30:39.827 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.828 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HjUiFwYzudUAEdI0XStP6sWXCMY= 2023-11-02T18:30:39.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HjUiFwYzudUAEdI0XStP6sWXCMY=', 'Content-Length': '0'} 2023-11-02T18:30:39.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078254c3fd0ce8dfb-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.874 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.875 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.876 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.876 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.877 DEBUG:boto:Method: GET 2023-11-02T18:30:39.878 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.880 DEBUG:boto:Data: 2023-11-02T18:30:39.880 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.881 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.881 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.882 DEBUG:boto:Params: {} 2023-11-02T18:30:39.882 DEBUG:boto:Token: None 2023-11-02T18:30:39.883 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000162?versionId=null 2023-11-02T18:30:39.883 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HjUiFwYzudUAEdI0XStP6sWXCMY= 2023-11-02T18:30:39.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HjUiFwYzudUAEdI0XStP6sWXCMY=', 'Content-Length': '0'} 2023-11-02T18:30:39.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000147bfc615dfd2f58-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.888 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.888 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.889 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000162 2023-11-02T18:30:39.889 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000162 2023-11-02T18:30:39.890 DEBUG:boto:Method: GET 2023-11-02T18:30:39.890 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000162 2023-11-02T18:30:39.891 DEBUG:boto:Data: 2023-11-02T18:30:39.891 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.891 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.892 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.892 DEBUG:boto:Params: {} 2023-11-02T18:30:39.893 DEBUG:boto:Token: None 2023-11-02T18:30:39.894 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.894 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9103d1a6df997e3-006543eacf-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.901 DEBUG:boto:b'ibtljn-58obj000000000000000001621000falseobj000000000000000001622023-11-02T18:30:18.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.902 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.902 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.905 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000162 2023-11-02T18:30:39.906 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000162 2023-11-02T18:30:39.906 DEBUG:boto:Method: GET 2023-11-02T18:30:39.907 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000162 2023-11-02T18:30:39.907 DEBUG:boto:Data: 2023-11-02T18:30:39.908 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.908 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.908 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.909 DEBUG:boto:Params: {} 2023-11-02T18:30:39.909 DEBUG:boto:Token: None 2023-11-02T18:30:39.910 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.910 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f47b5ff04dc599ee-006543eacf-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.921 DEBUG:boto:b'ibtljn-58obj000000000000000001621000falseobj000000000000000001622023-11-02T18:30:18.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:39.921 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.922 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.922 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.923 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.923 DEBUG:boto:Data: 2023-11-02T18:30:39.924 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.924 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.924 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.925 DEBUG:boto:Params: {} 2023-11-02T18:30:39.925 DEBUG:boto:Token: None 2023-11-02T18:30:39.926 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.926 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OYloBr8s4wyAO4H75YK8gKgUgh8= 2023-11-02T18:30:39.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OYloBr8s4wyAO4H75YK8gKgUgh8=', 'Content-Length': '0'} 2023-11-02T18:30:39.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000511291adefeb800d-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.930 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.930 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.931 DEBUG:boto:Method: HEAD 2023-11-02T18:30:39.931 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.932 DEBUG:boto:Data: 2023-11-02T18:30:39.932 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.933 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.933 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.933 DEBUG:boto:Params: {} 2023-11-02T18:30:39.934 DEBUG:boto:Token: None 2023-11-02T18:30:39.934 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.935 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OYloBr8s4wyAO4H75YK8gKgUgh8= 2023-11-02T18:30:39.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OYloBr8s4wyAO4H75YK8gKgUgh8=', 'Content-Length': '0'} 2023-11-02T18:30:39.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d37875d7c52301d7-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-11-02T18:30:39.939 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.939 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.940 DEBUG:boto:Method: GET 2023-11-02T18:30:39.940 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.941 DEBUG:boto:Data: 2023-11-02T18:30:39.941 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.941 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.942 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.942 DEBUG:boto:Params: {} 2023-11-02T18:30:39.943 DEBUG:boto:Token: None 2023-11-02T18:30:39.943 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.944 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C2jA/rCb3irqPdnyuqzstmLsVmc= 2023-11-02T18:30:39.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C2jA/rCb3irqPdnyuqzstmLsVmc=', 'Content-Length': '0'} 2023-11-02T18:30:39.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009477acc2280199e8-006543eacf-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.948 DEBUG:boto:path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.948 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.949 DEBUG:boto:Method: GET 2023-11-02T18:30:39.949 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.949 DEBUG:boto:Data: 2023-11-02T18:30:39.950 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:39.951 DEBUG:boto:Port: 8000 2023-11-02T18:30:39.951 DEBUG:boto:Params: {} 2023-11-02T18:30:39.952 DEBUG:boto:Token: None 2023-11-02T18:30:39.952 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/obj00000000000000000162 2023-11-02T18:30:39.953 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C2jA/rCb3irqPdnyuqzstmLsVmc= 2023-11-02T18:30:39.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C2jA/rCb3irqPdnyuqzstmLsVmc=', 'Content-Length': '0'} 2023-11-02T18:30:39.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065abbc42c4fc2340-006543eacf-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:39.957 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:39.957 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:39.957 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000162 2023-11-02T18:30:39.958 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000162 2023-11-02T18:30:39.958 DEBUG:boto:Method: GET 2023-11-02T18:30:39.959 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000162 2023-11-02T18:30:39.959 DEBUG:boto:Data: 2023-11-02T18:30:39.960 DEBUG:boto:Headers: {} 2023-11-02T18:30:39.960 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:39.961 DEBUG:boto:Port: 8001 2023-11-02T18:30:39.961 DEBUG:boto:Params: {} 2023-11-02T18:30:39.962 DEBUG:boto:Token: None 2023-11-02T18:30:39.962 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:39 GMT /ibtljn-58/ 2023-11-02T18:30:39.963 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8= 2023-11-02T18:30:39.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H185hEwe8AvnbeDizVtWTd+7SS8=', 'Content-Length': '0'} 2023-11-02T18:30:39.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da2db841768437b7-006543eacf-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.010 DEBUG:boto:b'ibtljn-58obj000000000000000001621000falseobj000000000000000001622023-11-02T18:30:18.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.011 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.012 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.012 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000162 2023-11-02T18:30:40.013 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000162 2023-11-02T18:30:40.013 DEBUG:boto:Method: GET 2023-11-02T18:30:40.014 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000162 2023-11-02T18:30:40.014 DEBUG:boto:Data: 2023-11-02T18:30:40.014 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.015 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.015 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.016 DEBUG:boto:Params: {} 2023-11-02T18:30:40.016 DEBUG:boto:Token: None 2023-11-02T18:30:40.017 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.017 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef81b4161aa9182a-006543ead0-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.028 DEBUG:boto:b'ibtljn-58obj000000000000000001621000falseobj000000000000000001622023-11-02T18:30:18.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-11-02T18:30:40.029 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.030 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.030 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.031 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.031 DEBUG:boto:Method: GET 2023-11-02T18:30:40.032 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.032 DEBUG:boto:Data: 2023-11-02T18:30:40.032 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.033 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.033 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.034 DEBUG:boto:Params: {} 2023-11-02T18:30:40.034 DEBUG:boto:Token: None 2023-11-02T18:30:40.035 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LBaLPnjhUKKGTfK2yvsfN6guj/Q= 2023-11-02T18:30:40.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LBaLPnjhUKKGTfK2yvsfN6guj/Q=', 'Content-Length': '0'} 2023-11-02T18:30:40.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007353f1de4b0a845b-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.039 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.040 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.040 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.041 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.041 DEBUG:boto:Method: GET 2023-11-02T18:30:40.042 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.042 DEBUG:boto:Data: 2023-11-02T18:30:40.043 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.043 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.044 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.044 DEBUG:boto:Params: {} 2023-11-02T18:30:40.044 DEBUG:boto:Token: None 2023-11-02T18:30:40.045 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.046 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LBaLPnjhUKKGTfK2yvsfN6guj/Q= 2023-11-02T18:30:40.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LBaLPnjhUKKGTfK2yvsfN6guj/Q=', 'Content-Length': '0'} 2023-11-02T18:30:40.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022ffd0490588374c-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.090 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.091 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.091 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.092 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.092 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.093 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.093 DEBUG:boto:Data: 2023-11-02T18:30:40.094 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.094 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.095 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.095 DEBUG:boto:Params: {} 2023-11-02T18:30:40.096 DEBUG:boto:Token: None 2023-11-02T18:30:40.096 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.097 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tUKX2Bp6TPhvLEdQ2DG7EXycbDo= 2023-11-02T18:30:40.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tUKX2Bp6TPhvLEdQ2DG7EXycbDo=', 'Content-Length': '0'} 2023-11-02T18:30:40.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041badabd9df37d4b-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.101 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.101 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.102 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.102 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.103 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.103 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.104 DEBUG:boto:Data: 2023-11-02T18:30:40.104 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.105 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.105 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.105 DEBUG:boto:Params: {} 2023-11-02T18:30:40.106 DEBUG:boto:Token: None 2023-11-02T18:30:40.106 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.107 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tUKX2Bp6TPhvLEdQ2DG7EXycbDo= 2023-11-02T18:30:40.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tUKX2Bp6TPhvLEdQ2DG7EXycbDo=', 'Content-Length': '0'} 2023-11-02T18:30:40.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037f914f023b7dbbb-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-11-02T18:30:40.111 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.112 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.112 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.113 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.113 DEBUG:boto:Method: GET 2023-11-02T18:30:40.114 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.114 DEBUG:boto:Data: 2023-11-02T18:30:40.115 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.115 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.115 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.116 DEBUG:boto:Params: {} 2023-11-02T18:30:40.116 DEBUG:boto:Token: None 2023-11-02T18:30:40.117 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.117 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LBaLPnjhUKKGTfK2yvsfN6guj/Q= 2023-11-02T18:30:40.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LBaLPnjhUKKGTfK2yvsfN6guj/Q=', 'Content-Length': '0'} 2023-11-02T18:30:40.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f003b79948100b7-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.122 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.122 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.123 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.123 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.124 DEBUG:boto:Method: GET 2023-11-02T18:30:40.124 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.125 DEBUG:boto:Data: 2023-11-02T18:30:40.125 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.126 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.126 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.127 DEBUG:boto:Params: {} 2023-11-02T18:30:40.127 DEBUG:boto:Token: None 2023-11-02T18:30:40.128 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000271?versionId=null 2023-11-02T18:30:40.128 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LBaLPnjhUKKGTfK2yvsfN6guj/Q= 2023-11-02T18:30:40.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LBaLPnjhUKKGTfK2yvsfN6guj/Q=', 'Content-Length': '0'} 2023-11-02T18:30:40.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001794ca4ef48941ac-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.132 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.133 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.133 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000271 2023-11-02T18:30:40.134 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000271 2023-11-02T18:30:40.134 DEBUG:boto:Method: GET 2023-11-02T18:30:40.135 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000271 2023-11-02T18:30:40.135 DEBUG:boto:Data: 2023-11-02T18:30:40.136 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.136 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.136 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.137 DEBUG:boto:Params: {} 2023-11-02T18:30:40.137 DEBUG:boto:Token: None 2023-11-02T18:30:40.138 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.138 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c08e919c9dd24ed-006543ead0-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.145 DEBUG:boto:b'ibtljn-58obj000000000000000002711000falseobj000000000000000002712023-11-02T18:30:18.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.146 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.146 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.147 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000271 2023-11-02T18:30:40.147 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000271 2023-11-02T18:30:40.148 DEBUG:boto:Method: GET 2023-11-02T18:30:40.148 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000271 2023-11-02T18:30:40.149 DEBUG:boto:Data: 2023-11-02T18:30:40.149 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.150 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.150 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.151 DEBUG:boto:Params: {} 2023-11-02T18:30:40.151 DEBUG:boto:Token: None 2023-11-02T18:30:40.153 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.153 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c7bad8a85907596-006543ead0-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.206 DEBUG:boto:b'ibtljn-58obj000000000000000002711000falseobj000000000000000002712023-11-02T18:30:18.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.207 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.208 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.208 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.208 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.209 DEBUG:boto:Data: 2023-11-02T18:30:40.209 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.210 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.210 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.211 DEBUG:boto:Params: {} 2023-11-02T18:30:40.211 DEBUG:boto:Token: None 2023-11-02T18:30:40.212 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.212 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OjsEFhx5cVMh0mjzm9e3f877cnQ= 2023-11-02T18:30:40.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OjsEFhx5cVMh0mjzm9e3f877cnQ=', 'Content-Length': '0'} 2023-11-02T18:30:40.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014e8c76d2956c39a-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.216 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.217 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.217 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.218 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.218 DEBUG:boto:Data: 2023-11-02T18:30:40.218 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.219 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.219 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.220 DEBUG:boto:Params: {} 2023-11-02T18:30:40.220 DEBUG:boto:Token: None 2023-11-02T18:30:40.221 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.221 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OjsEFhx5cVMh0mjzm9e3f877cnQ= 2023-11-02T18:30:40.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OjsEFhx5cVMh0mjzm9e3f877cnQ=', 'Content-Length': '0'} 2023-11-02T18:30:40.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0aed7eccb0c180d-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-11-02T18:30:40.226 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.226 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.227 DEBUG:boto:Method: GET 2023-11-02T18:30:40.227 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.228 DEBUG:boto:Data: 2023-11-02T18:30:40.228 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.228 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.229 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.229 DEBUG:boto:Params: {} 2023-11-02T18:30:40.230 DEBUG:boto:Token: None 2023-11-02T18:30:40.230 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.231 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T8Q+9vaS8BtbXWzi65oFTX3YvDg= 2023-11-02T18:30:40.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T8Q+9vaS8BtbXWzi65oFTX3YvDg=', 'Content-Length': '0'} 2023-11-02T18:30:40.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fb2bbf57fda4dcf-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.278 DEBUG:boto:path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.279 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.279 DEBUG:boto:Method: GET 2023-11-02T18:30:40.280 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.280 DEBUG:boto:Data: 2023-11-02T18:30:40.281 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.281 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.282 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.282 DEBUG:boto:Params: {} 2023-11-02T18:30:40.283 DEBUG:boto:Token: None 2023-11-02T18:30:40.283 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000271 2023-11-02T18:30:40.284 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T8Q+9vaS8BtbXWzi65oFTX3YvDg= 2023-11-02T18:30:40.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T8Q+9vaS8BtbXWzi65oFTX3YvDg=', 'Content-Length': '0'} 2023-11-02T18:30:40.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cda49029275a0a46-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.288 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.288 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.289 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000271 2023-11-02T18:30:40.289 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000271 2023-11-02T18:30:40.289 DEBUG:boto:Method: GET 2023-11-02T18:30:40.290 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000271 2023-11-02T18:30:40.290 DEBUG:boto:Data: 2023-11-02T18:30:40.291 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.291 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.292 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.292 DEBUG:boto:Params: {} 2023-11-02T18:30:40.293 DEBUG:boto:Token: None 2023-11-02T18:30:40.293 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.294 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a2b34984c67aa7c-006543ead0-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.299 DEBUG:boto:b'ibtljn-58obj000000000000000002711000falseobj000000000000000002712023-11-02T18:30:18.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.300 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.300 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.301 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000271 2023-11-02T18:30:40.301 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000271 2023-11-02T18:30:40.302 DEBUG:boto:Method: GET 2023-11-02T18:30:40.302 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000271 2023-11-02T18:30:40.302 DEBUG:boto:Data: 2023-11-02T18:30:40.303 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.304 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.304 DEBUG:boto:Params: {} 2023-11-02T18:30:40.305 DEBUG:boto:Token: None 2023-11-02T18:30:40.305 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.306 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e21b6ee5e5b138e6-006543ead0-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.315 DEBUG:boto:b'ibtljn-58obj000000000000000002711000falseobj000000000000000002712023-11-02T18:30:18.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-11-02T18:30:40.316 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.317 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.317 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.318 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.318 DEBUG:boto:Method: GET 2023-11-02T18:30:40.319 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.319 DEBUG:boto:Data: 2023-11-02T18:30:40.319 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.320 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.320 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.321 DEBUG:boto:Params: {} 2023-11-02T18:30:40.321 DEBUG:boto:Token: None 2023-11-02T18:30:40.322 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.322 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eR4RshBjLu1pSUyVGienSmQpNQc= 2023-11-02T18:30:40.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eR4RshBjLu1pSUyVGienSmQpNQc=', 'Content-Length': '0'} 2023-11-02T18:30:40.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccb696bf9beef453-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.326 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.327 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.327 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.328 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.328 DEBUG:boto:Method: GET 2023-11-02T18:30:40.329 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.329 DEBUG:boto:Data: 2023-11-02T18:30:40.329 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.330 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.330 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.331 DEBUG:boto:Params: {} 2023-11-02T18:30:40.331 DEBUG:boto:Token: None 2023-11-02T18:30:40.332 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.332 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eR4RshBjLu1pSUyVGienSmQpNQc= 2023-11-02T18:30:40.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eR4RshBjLu1pSUyVGienSmQpNQc=', 'Content-Length': '0'} 2023-11-02T18:30:40.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e44f8b88b3a884-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.378 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.379 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.379 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.380 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.380 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.381 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.381 DEBUG:boto:Data: 2023-11-02T18:30:40.381 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.382 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.382 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.383 DEBUG:boto:Params: {} 2023-11-02T18:30:40.383 DEBUG:boto:Token: None 2023-11-02T18:30:40.384 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.384 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kPNDKDr1QNH15ybLIfHfAR/9NcI= 2023-11-02T18:30:40.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kPNDKDr1QNH15ybLIfHfAR/9NcI=', 'Content-Length': '0'} 2023-11-02T18:30:40.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007edf7c54893b39f4-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.388 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.389 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.389 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.390 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.390 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.390 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.391 DEBUG:boto:Data: 2023-11-02T18:30:40.391 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.392 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.392 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.392 DEBUG:boto:Params: {} 2023-11-02T18:30:40.393 DEBUG:boto:Token: None 2023-11-02T18:30:40.393 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kPNDKDr1QNH15ybLIfHfAR/9NcI= 2023-11-02T18:30:40.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kPNDKDr1QNH15ybLIfHfAR/9NcI=', 'Content-Length': '0'} 2023-11-02T18:30:40.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d8ad62375b86d7d-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-11-02T18:30:40.398 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.398 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.399 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.399 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.400 DEBUG:boto:Method: GET 2023-11-02T18:30:40.400 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.401 DEBUG:boto:Data: 2023-11-02T18:30:40.401 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.401 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.402 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.402 DEBUG:boto:Params: {} 2023-11-02T18:30:40.403 DEBUG:boto:Token: None 2023-11-02T18:30:40.403 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.404 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eR4RshBjLu1pSUyVGienSmQpNQc= 2023-11-02T18:30:40.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eR4RshBjLu1pSUyVGienSmQpNQc=', 'Content-Length': '0'} 2023-11-02T18:30:40.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009da733faa1f0e4cf-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.450 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.451 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.451 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.452 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.452 DEBUG:boto:Method: GET 2023-11-02T18:30:40.453 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.453 DEBUG:boto:Data: 2023-11-02T18:30:40.454 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.455 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.455 DEBUG:boto:Params: {} 2023-11-02T18:30:40.455 DEBUG:boto:Token: None 2023-11-02T18:30:40.456 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000283?versionId=null 2023-11-02T18:30:40.456 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eR4RshBjLu1pSUyVGienSmQpNQc= 2023-11-02T18:30:40.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eR4RshBjLu1pSUyVGienSmQpNQc=', 'Content-Length': '0'} 2023-11-02T18:30:40.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079ce9a79eec6da73-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.461 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.461 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.462 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000283 2023-11-02T18:30:40.462 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000283 2023-11-02T18:30:40.463 DEBUG:boto:Method: GET 2023-11-02T18:30:40.463 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000283 2023-11-02T18:30:40.464 DEBUG:boto:Data: 2023-11-02T18:30:40.464 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.464 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.465 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.465 DEBUG:boto:Params: {} 2023-11-02T18:30:40.466 DEBUG:boto:Token: None 2023-11-02T18:30:40.466 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.467 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddd293bfe07d6985-006543ead0-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.473 DEBUG:boto:b'ibtljn-58obj000000000000000002831000falseobj000000000000000002832023-11-02T18:30:17.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.473 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.474 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.474 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000283 2023-11-02T18:30:40.475 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000283 2023-11-02T18:30:40.475 DEBUG:boto:Method: GET 2023-11-02T18:30:40.476 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000283 2023-11-02T18:30:40.476 DEBUG:boto:Data: 2023-11-02T18:30:40.476 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.477 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.477 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.478 DEBUG:boto:Params: {} 2023-11-02T18:30:40.478 DEBUG:boto:Token: None 2023-11-02T18:30:40.479 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.479 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d970965af77f76d4-006543ead0-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.489 DEBUG:boto:b'ibtljn-58obj000000000000000002831000falseobj000000000000000002832023-11-02T18:30:17.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.490 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.490 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.490 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.491 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.491 DEBUG:boto:Data: 2023-11-02T18:30:40.492 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.492 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.493 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.493 DEBUG:boto:Params: {} 2023-11-02T18:30:40.493 DEBUG:boto:Token: None 2023-11-02T18:30:40.494 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.494 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bYn9qbMKf2uXE1QqwrlXKt8p97w= 2023-11-02T18:30:40.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bYn9qbMKf2uXE1QqwrlXKt8p97w=', 'Content-Length': '0'} 2023-11-02T18:30:40.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900c3ed6e7b48e72-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.498 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.498 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.498 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.499 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.499 DEBUG:boto:Data: 2023-11-02T18:30:40.500 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.500 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.501 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.501 DEBUG:boto:Params: {} 2023-11-02T18:30:40.501 DEBUG:boto:Token: None 2023-11-02T18:30:40.502 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.502 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bYn9qbMKf2uXE1QqwrlXKt8p97w= 2023-11-02T18:30:40.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bYn9qbMKf2uXE1QqwrlXKt8p97w=', 'Content-Length': '0'} 2023-11-02T18:30:40.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002537f613651983f6-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-11-02T18:30:40.506 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.507 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.507 DEBUG:boto:Method: GET 2023-11-02T18:30:40.508 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.508 DEBUG:boto:Data: 2023-11-02T18:30:40.509 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.509 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.509 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.510 DEBUG:boto:Params: {} 2023-11-02T18:30:40.510 DEBUG:boto:Token: None 2023-11-02T18:30:40.511 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.511 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KKZVun09roNOIjdBWSu7EFv4v1U= 2023-11-02T18:30:40.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KKZVun09roNOIjdBWSu7EFv4v1U=', 'Content-Length': '0'} 2023-11-02T18:30:40.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057b0fae1698b4a3a-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.515 DEBUG:boto:path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.516 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.516 DEBUG:boto:Method: GET 2023-11-02T18:30:40.517 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.517 DEBUG:boto:Data: 2023-11-02T18:30:40.518 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.518 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.522 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.522 DEBUG:boto:Params: {} 2023-11-02T18:30:40.523 DEBUG:boto:Token: None 2023-11-02T18:30:40.523 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000283 2023-11-02T18:30:40.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KKZVun09roNOIjdBWSu7EFv4v1U= 2023-11-02T18:30:40.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KKZVun09roNOIjdBWSu7EFv4v1U=', 'Content-Length': '0'} 2023-11-02T18:30:40.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8a7ec744e8cc1c6-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.570 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.571 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.571 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000283 2023-11-02T18:30:40.572 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000283 2023-11-02T18:30:40.572 DEBUG:boto:Method: GET 2023-11-02T18:30:40.573 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000283 2023-11-02T18:30:40.573 DEBUG:boto:Data: 2023-11-02T18:30:40.574 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.574 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.575 DEBUG:boto:Params: {} 2023-11-02T18:30:40.575 DEBUG:boto:Token: None 2023-11-02T18:30:40.576 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.576 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3287738deb81af8-006543ead0-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.582 DEBUG:boto:b'ibtljn-58obj000000000000000002831000falseobj000000000000000002832023-11-02T18:30:17.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.583 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.583 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.584 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000283 2023-11-02T18:30:40.584 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000283 2023-11-02T18:30:40.585 DEBUG:boto:Method: GET 2023-11-02T18:30:40.585 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000283 2023-11-02T18:30:40.585 DEBUG:boto:Data: 2023-11-02T18:30:40.586 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.586 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.587 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.587 DEBUG:boto:Params: {} 2023-11-02T18:30:40.588 DEBUG:boto:Token: None 2023-11-02T18:30:40.588 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.589 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff61f930f4d9f341-006543ead0-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.597 DEBUG:boto:b'ibtljn-58obj000000000000000002831000falseobj000000000000000002832023-11-02T18:30:17.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-11-02T18:30:40.598 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.598 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.599 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.599 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.600 DEBUG:boto:Method: GET 2023-11-02T18:30:40.600 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.601 DEBUG:boto:Data: 2023-11-02T18:30:40.601 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.602 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.602 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.603 DEBUG:boto:Params: {} 2023-11-02T18:30:40.603 DEBUG:boto:Token: None 2023-11-02T18:30:40.604 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.604 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2NneEGfT79nHO934FF0JmQgCyT4= 2023-11-02T18:30:40.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2NneEGfT79nHO934FF0JmQgCyT4=', 'Content-Length': '0'} 2023-11-02T18:30:40.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3fbfbf118c6802e-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.654 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.655 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.655 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.656 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.656 DEBUG:boto:Method: GET 2023-11-02T18:30:40.657 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.657 DEBUG:boto:Data: 2023-11-02T18:30:40.657 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.658 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.658 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.659 DEBUG:boto:Params: {} 2023-11-02T18:30:40.659 DEBUG:boto:Token: None 2023-11-02T18:30:40.660 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.660 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2NneEGfT79nHO934FF0JmQgCyT4= 2023-11-02T18:30:40.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2NneEGfT79nHO934FF0JmQgCyT4=', 'Content-Length': '0'} 2023-11-02T18:30:40.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000968a489e47247f46-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.665 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.665 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.665 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.666 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.666 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.667 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.667 DEBUG:boto:Data: 2023-11-02T18:30:40.668 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.668 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.668 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.669 DEBUG:boto:Params: {} 2023-11-02T18:30:40.669 DEBUG:boto:Token: None 2023-11-02T18:30:40.670 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.670 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UTckQxOMA+jLfjL25Bt8dgrWSGs= 2023-11-02T18:30:40.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UTckQxOMA+jLfjL25Bt8dgrWSGs=', 'Content-Length': '0'} 2023-11-02T18:30:40.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa29153a35bf8c7e-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.674 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.674 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.675 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.675 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.676 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.676 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.677 DEBUG:boto:Data: 2023-11-02T18:30:40.677 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.678 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.678 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.678 DEBUG:boto:Params: {} 2023-11-02T18:30:40.679 DEBUG:boto:Token: None 2023-11-02T18:30:40.679 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.680 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UTckQxOMA+jLfjL25Bt8dgrWSGs= 2023-11-02T18:30:40.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UTckQxOMA+jLfjL25Bt8dgrWSGs=', 'Content-Length': '0'} 2023-11-02T18:30:40.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000985ca02af0beb541-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-11-02T18:30:40.684 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.685 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.685 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.686 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.686 DEBUG:boto:Method: GET 2023-11-02T18:30:40.687 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.688 DEBUG:boto:Data: 2023-11-02T18:30:40.688 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.689 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.689 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.689 DEBUG:boto:Params: {} 2023-11-02T18:30:40.690 DEBUG:boto:Token: None 2023-11-02T18:30:40.690 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.691 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2NneEGfT79nHO934FF0JmQgCyT4= 2023-11-02T18:30:40.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2NneEGfT79nHO934FF0JmQgCyT4=', 'Content-Length': '0'} 2023-11-02T18:30:40.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f08c377c48f0885d-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.695 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.695 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.696 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.696 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.697 DEBUG:boto:Method: GET 2023-11-02T18:30:40.697 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.698 DEBUG:boto:Data: 2023-11-02T18:30:40.698 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.699 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.699 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.699 DEBUG:boto:Params: {} 2023-11-02T18:30:40.700 DEBUG:boto:Token: None 2023-11-02T18:30:40.700 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000317?versionId=null 2023-11-02T18:30:40.701 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2NneEGfT79nHO934FF0JmQgCyT4= 2023-11-02T18:30:40.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2NneEGfT79nHO934FF0JmQgCyT4=', 'Content-Length': '0'} 2023-11-02T18:30:40.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d86defef51925838-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.746 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.747 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.748 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000317 2023-11-02T18:30:40.748 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000317 2023-11-02T18:30:40.749 DEBUG:boto:Method: GET 2023-11-02T18:30:40.749 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000317 2023-11-02T18:30:40.750 DEBUG:boto:Data: 2023-11-02T18:30:40.750 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.751 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.751 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.751 DEBUG:boto:Params: {} 2023-11-02T18:30:40.752 DEBUG:boto:Token: None 2023-11-02T18:30:40.752 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.753 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000955c2cd357979462-006543ead0-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.759 DEBUG:boto:b'ibtljn-58obj000000000000000003171000falseobj000000000000000003172023-11-02T18:30:18.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.759 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.760 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.760 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000317 2023-11-02T18:30:40.761 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000317 2023-11-02T18:30:40.761 DEBUG:boto:Method: GET 2023-11-02T18:30:40.762 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000317 2023-11-02T18:30:40.762 DEBUG:boto:Data: 2023-11-02T18:30:40.763 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.764 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.765 DEBUG:boto:Params: {} 2023-11-02T18:30:40.766 DEBUG:boto:Token: None 2023-11-02T18:30:40.767 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054d8ff921182f3d4-006543ead0-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.779 DEBUG:boto:b'ibtljn-58obj000000000000000003171000falseobj000000000000000003172023-11-02T18:30:18.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.780 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.780 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.781 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.781 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.782 DEBUG:boto:Data: 2023-11-02T18:30:40.782 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.783 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.783 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.783 DEBUG:boto:Params: {} 2023-11-02T18:30:40.784 DEBUG:boto:Token: None 2023-11-02T18:30:40.785 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p81VUHsJX/Q2MSgliHs6FaM1R2Q= 2023-11-02T18:30:40.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p81VUHsJX/Q2MSgliHs6FaM1R2Q=', 'Content-Length': '0'} 2023-11-02T18:30:40.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e597f0b89e1dbc78-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.789 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.790 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.790 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.790 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.791 DEBUG:boto:Data: 2023-11-02T18:30:40.791 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.792 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.792 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.793 DEBUG:boto:Params: {} 2023-11-02T18:30:40.793 DEBUG:boto:Token: None 2023-11-02T18:30:40.794 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.794 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p81VUHsJX/Q2MSgliHs6FaM1R2Q= 2023-11-02T18:30:40.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p81VUHsJX/Q2MSgliHs6FaM1R2Q=', 'Content-Length': '0'} 2023-11-02T18:30:40.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef0c6973f3909ede-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-11-02T18:30:40.798 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.799 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.800 DEBUG:boto:Method: GET 2023-11-02T18:30:40.800 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.801 DEBUG:boto:Data: 2023-11-02T18:30:40.801 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.802 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.803 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.803 DEBUG:boto:Params: {} 2023-11-02T18:30:40.804 DEBUG:boto:Token: None 2023-11-02T18:30:40.805 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.806 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/RPN3QvNjQPCmnNaTxhu1umH3GI= 2023-11-02T18:30:40.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/RPN3QvNjQPCmnNaTxhu1umH3GI=', 'Content-Length': '0'} 2023-11-02T18:30:40.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e0fc349417d703b-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.810 DEBUG:boto:path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.810 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.811 DEBUG:boto:Method: GET 2023-11-02T18:30:40.811 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.811 DEBUG:boto:Data: 2023-11-02T18:30:40.812 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.812 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.813 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.813 DEBUG:boto:Params: {} 2023-11-02T18:30:40.814 DEBUG:boto:Token: None 2023-11-02T18:30:40.814 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000317 2023-11-02T18:30:40.815 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/RPN3QvNjQPCmnNaTxhu1umH3GI= 2023-11-02T18:30:40.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/RPN3QvNjQPCmnNaTxhu1umH3GI=', 'Content-Length': '0'} 2023-11-02T18:30:40.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea76edaca203bbff-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.818 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.819 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.819 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000317 2023-11-02T18:30:40.820 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000317 2023-11-02T18:30:40.820 DEBUG:boto:Method: GET 2023-11-02T18:30:40.821 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000317 2023-11-02T18:30:40.821 DEBUG:boto:Data: 2023-11-02T18:30:40.822 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.822 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.822 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.823 DEBUG:boto:Params: {} 2023-11-02T18:30:40.823 DEBUG:boto:Token: None 2023-11-02T18:30:40.824 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.824 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003bc97981616df6b-006543ead0-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.830 DEBUG:boto:b'ibtljn-58obj000000000000000003171000falseobj000000000000000003172023-11-02T18:30:18.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.831 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.831 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.832 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000317 2023-11-02T18:30:40.832 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000317 2023-11-02T18:30:40.833 DEBUG:boto:Method: GET 2023-11-02T18:30:40.833 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000317 2023-11-02T18:30:40.834 DEBUG:boto:Data: 2023-11-02T18:30:40.834 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.835 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.835 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.835 DEBUG:boto:Params: {} 2023-11-02T18:30:40.836 DEBUG:boto:Token: None 2023-11-02T18:30:40.836 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/ 2023-11-02T18:30:40.837 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y= 2023-11-02T18:30:40.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZotwTtNQSxiNBs5QEBXJUnZbx3Y=', 'Content-Length': '0'} 2023-11-02T18:30:40.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f274b24b4cfbf728-006543ead0-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.845 DEBUG:boto:b'ibtljn-58obj000000000000000003171000falseobj000000000000000003172023-11-02T18:30:18.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:40.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-11-02T18:30:40.848 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:40.848 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:40.848 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.849 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.849 DEBUG:boto:Method: GET 2023-11-02T18:30:40.850 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.850 DEBUG:boto:Data: 2023-11-02T18:30:40.850 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.851 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.851 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.852 DEBUG:boto:Params: {} 2023-11-02T18:30:40.852 DEBUG:boto:Token: None 2023-11-02T18:30:40.853 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.853 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s4z7pr5lYOnZo3ZVVM7VluqF8So= 2023-11-02T18:30:40.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s4z7pr5lYOnZo3ZVVM7VluqF8So=', 'Content-Length': '0'} 2023-11-02T18:30:40.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e3cd49a1e61474c-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.857 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:40.857 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:40.858 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.858 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.859 DEBUG:boto:Method: GET 2023-11-02T18:30:40.860 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.860 DEBUG:boto:Data: 2023-11-02T18:30:40.861 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.861 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.864 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.864 DEBUG:boto:Params: {} 2023-11-02T18:30:40.865 DEBUG:boto:Token: None 2023-11-02T18:30:40.866 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.866 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s4z7pr5lYOnZo3ZVVM7VluqF8So= 2023-11-02T18:30:40.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s4z7pr5lYOnZo3ZVVM7VluqF8So=', 'Content-Length': '0'} 2023-11-02T18:30:40.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbcbf30566538208-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.910 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:40.911 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:40.911 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.912 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.912 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.913 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.913 DEBUG:boto:Data: 2023-11-02T18:30:40.913 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.914 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.914 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.915 DEBUG:boto:Params: {} 2023-11-02T18:30:40.915 DEBUG:boto:Token: None 2023-11-02T18:30:40.916 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.916 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CQJAYom256/LgUVDBEzVe8/VrhE= 2023-11-02T18:30:40.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CQJAYom256/LgUVDBEzVe8/VrhE=', 'Content-Length': '0'} 2023-11-02T18:30:40.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085b6184455102459-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.920 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:40.921 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:40.921 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.921 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.922 DEBUG:boto:Method: HEAD 2023-11-02T18:30:40.922 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.923 DEBUG:boto:Data: 2023-11-02T18:30:40.923 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.923 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.924 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.924 DEBUG:boto:Params: {} 2023-11-02T18:30:40.925 DEBUG:boto:Token: None 2023-11-02T18:30:40.925 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.926 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CQJAYom256/LgUVDBEzVe8/VrhE= 2023-11-02T18:30:40.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CQJAYom256/LgUVDBEzVe8/VrhE=', 'Content-Length': '0'} 2023-11-02T18:30:40.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025bfab2dfe00af50-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-11-02T18:30:40.930 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:40.931 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:40.931 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.932 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.932 DEBUG:boto:Method: GET 2023-11-02T18:30:40.932 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.933 DEBUG:boto:Data: 2023-11-02T18:30:40.933 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.934 DEBUG:boto:Port: 8001 2023-11-02T18:30:40.935 DEBUG:boto:Params: {} 2023-11-02T18:30:40.935 DEBUG:boto:Token: None 2023-11-02T18:30:40.935 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.936 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s4z7pr5lYOnZo3ZVVM7VluqF8So= 2023-11-02T18:30:40.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s4z7pr5lYOnZo3ZVVM7VluqF8So=', 'Content-Length': '0'} 2023-11-02T18:30:40.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7fd5780d3051834-006543ead0-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.982 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:40.983 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:40.983 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.984 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.984 DEBUG:boto:Method: GET 2023-11-02T18:30:40.985 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.985 DEBUG:boto:Data: 2023-11-02T18:30:40.985 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.986 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:40.986 DEBUG:boto:Port: 8000 2023-11-02T18:30:40.987 DEBUG:boto:Params: {} 2023-11-02T18:30:40.987 DEBUG:boto:Token: None 2023-11-02T18:30:40.988 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:40 GMT /ibtljn-58/obj00000000000000000423?versionId=null 2023-11-02T18:30:40.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s4z7pr5lYOnZo3ZVVM7VluqF8So= 2023-11-02T18:30:40.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s4z7pr5lYOnZo3ZVVM7VluqF8So=', 'Content-Length': '0'} 2023-11-02T18:30:40.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31c6a28708c802e-006543ead0-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:40.994 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:40.995 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:40.995 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000423 2023-11-02T18:30:40.996 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000423 2023-11-02T18:30:40.996 DEBUG:boto:Method: GET 2023-11-02T18:30:40.997 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000423 2023-11-02T18:30:40.998 DEBUG:boto:Data: 2023-11-02T18:30:40.998 DEBUG:boto:Headers: {} 2023-11-02T18:30:40.999 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:40.999 DEBUG:boto:Port: 8001 2023-11-02T18:30:41.001 DEBUG:boto:Params: {} 2023-11-02T18:30:41.001 DEBUG:boto:Token: None 2023-11-02T18:30:41.002 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/ 2023-11-02T18:30:41.002 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4= 2023-11-02T18:30:41.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4=', 'Content-Length': '0'} 2023-11-02T18:30:41.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eafc06526419ce84-006543ead0-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.008 DEBUG:boto:b'ibtljn-58obj000000000000000004231000falseobj000000000000000004232023-11-02T18:30:18.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:41.009 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:41.010 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:41.010 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000423 2023-11-02T18:30:41.011 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000423 2023-11-02T18:30:41.012 DEBUG:boto:Method: GET 2023-11-02T18:30:41.012 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000423 2023-11-02T18:30:41.013 DEBUG:boto:Data: 2023-11-02T18:30:41.013 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.014 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:41.014 DEBUG:boto:Port: 8000 2023-11-02T18:30:41.015 DEBUG:boto:Params: {} 2023-11-02T18:30:41.015 DEBUG:boto:Token: None 2023-11-02T18:30:41.016 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/ 2023-11-02T18:30:41.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4= 2023-11-02T18:30:41.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4=', 'Content-Length': '0'} 2023-11-02T18:30:41.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f0985dcd84230f6-006543ead1-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.028 DEBUG:boto:b'ibtljn-58obj000000000000000004231000falseobj000000000000000004232023-11-02T18:30:18.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:41.028 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.029 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.029 DEBUG:boto:Method: HEAD 2023-11-02T18:30:41.030 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.030 DEBUG:boto:Data: 2023-11-02T18:30:41.030 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.031 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:41.031 DEBUG:boto:Port: 8001 2023-11-02T18:30:41.032 DEBUG:boto:Params: {} 2023-11-02T18:30:41.032 DEBUG:boto:Token: None 2023-11-02T18:30:41.033 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.033 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:65Rpnvt+9R2mnfZEp/OEgvATYqs= 2023-11-02T18:30:41.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:65Rpnvt+9R2mnfZEp/OEgvATYqs=', 'Content-Length': '0'} 2023-11-02T18:30:41.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016fca3c0ecd37204-006543ead1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.038 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.039 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.039 DEBUG:boto:Method: HEAD 2023-11-02T18:30:41.039 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.040 DEBUG:boto:Data: 2023-11-02T18:30:41.040 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.041 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:41.041 DEBUG:boto:Port: 8000 2023-11-02T18:30:41.041 DEBUG:boto:Params: {} 2023-11-02T18:30:41.042 DEBUG:boto:Token: None 2023-11-02T18:30:41.043 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.044 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:65Rpnvt+9R2mnfZEp/OEgvATYqs= 2023-11-02T18:30:41.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:65Rpnvt+9R2mnfZEp/OEgvATYqs=', 'Content-Length': '0'} 2023-11-02T18:30:41.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef306f116cd9d70a-006543ead1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-11-02T18:30:41.048 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.049 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.049 DEBUG:boto:Method: GET 2023-11-02T18:30:41.050 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.051 DEBUG:boto:Data: 2023-11-02T18:30:41.051 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.052 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:41.052 DEBUG:boto:Port: 8001 2023-11-02T18:30:41.053 DEBUG:boto:Params: {} 2023-11-02T18:30:41.054 DEBUG:boto:Token: None 2023-11-02T18:30:41.055 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.056 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fimpKEHWeA/U9WlV4YcuIFqfvwY= 2023-11-02T18:30:41.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fimpKEHWeA/U9WlV4YcuIFqfvwY=', 'Content-Length': '0'} 2023-11-02T18:30:41.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb25e1419b3b024c-006543ead1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.060 DEBUG:boto:path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.061 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.061 DEBUG:boto:Method: GET 2023-11-02T18:30:41.062 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.062 DEBUG:boto:Data: 2023-11-02T18:30:41.063 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.063 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:41.064 DEBUG:boto:Port: 8000 2023-11-02T18:30:41.064 DEBUG:boto:Params: {} 2023-11-02T18:30:41.065 DEBUG:boto:Token: None 2023-11-02T18:30:41.065 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000423 2023-11-02T18:30:41.065 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fimpKEHWeA/U9WlV4YcuIFqfvwY= 2023-11-02T18:30:41.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fimpKEHWeA/U9WlV4YcuIFqfvwY=', 'Content-Length': '0'} 2023-11-02T18:30:41.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000817701bdd24cc99d-006543ead1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.110 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:41.111 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:41.111 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000423 2023-11-02T18:30:41.112 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000423 2023-11-02T18:30:41.112 DEBUG:boto:Method: GET 2023-11-02T18:30:41.113 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000423 2023-11-02T18:30:41.113 DEBUG:boto:Data: 2023-11-02T18:30:41.114 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.114 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:41.115 DEBUG:boto:Port: 8001 2023-11-02T18:30:41.115 DEBUG:boto:Params: {} 2023-11-02T18:30:41.116 DEBUG:boto:Token: None 2023-11-02T18:30:41.116 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/ 2023-11-02T18:30:41.117 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4= 2023-11-02T18:30:41.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4=', 'Content-Length': '0'} 2023-11-02T18:30:41.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e086f227f4efee5b-006543ead1-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.122 DEBUG:boto:b'ibtljn-58obj000000000000000004231000falseobj000000000000000004232023-11-02T18:30:18.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:41.123 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:41.124 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:41.124 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000423 2023-11-02T18:30:41.124 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000423 2023-11-02T18:30:41.125 DEBUG:boto:Method: GET 2023-11-02T18:30:41.125 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000423 2023-11-02T18:30:41.126 DEBUG:boto:Data: 2023-11-02T18:30:41.126 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.127 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:41.127 DEBUG:boto:Port: 8000 2023-11-02T18:30:41.128 DEBUG:boto:Params: {} 2023-11-02T18:30:41.128 DEBUG:boto:Token: None 2023-11-02T18:30:41.129 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/ 2023-11-02T18:30:41.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4= 2023-11-02T18:30:41.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4=', 'Content-Length': '0'} 2023-11-02T18:30:41.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040030dc010ec81c1-006543ead1-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.141 DEBUG:boto:b'ibtljn-58obj000000000000000004231000falseobj000000000000000004232023-11-02T18:30:18.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:41.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-11-02T18:30:41.143 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.143 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.144 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.145 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.145 DEBUG:boto:Method: GET 2023-11-02T18:30:41.146 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.147 DEBUG:boto:Data: 2023-11-02T18:30:41.148 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:41.149 DEBUG:boto:Port: 8001 2023-11-02T18:30:41.150 DEBUG:boto:Params: {} 2023-11-02T18:30:41.150 DEBUG:boto:Token: None 2023-11-02T18:30:41.151 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.152 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X7k+4jzblaVTanQzgjSuEEGYACc= 2023-11-02T18:30:41.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X7k+4jzblaVTanQzgjSuEEGYACc=', 'Content-Length': '0'} 2023-11-02T18:30:41.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dc13335970ee6c9-006543ead1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.198 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.199 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.199 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.200 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.200 DEBUG:boto:Method: GET 2023-11-02T18:30:41.201 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.201 DEBUG:boto:Data: 2023-11-02T18:30:41.202 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:41.203 DEBUG:boto:Port: 8000 2023-11-02T18:30:41.204 DEBUG:boto:Params: {} 2023-11-02T18:30:41.205 DEBUG:boto:Token: None 2023-11-02T18:30:41.205 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.206 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X7k+4jzblaVTanQzgjSuEEGYACc= 2023-11-02T18:30:41.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X7k+4jzblaVTanQzgjSuEEGYACc=', 'Content-Length': '0'} 2023-11-02T18:30:41.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3705ab95ceaf1f6-006543ead1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.211 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.211 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.212 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.212 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.212 DEBUG:boto:Method: HEAD 2023-11-02T18:30:41.213 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.213 DEBUG:boto:Data: 2023-11-02T18:30:41.214 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.214 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:41.215 DEBUG:boto:Port: 8001 2023-11-02T18:30:41.215 DEBUG:boto:Params: {} 2023-11-02T18:30:41.216 DEBUG:boto:Token: None 2023-11-02T18:30:41.216 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TNkQ10wiA33Bw4ADh1nWqjpsAVE= 2023-11-02T18:30:41.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TNkQ10wiA33Bw4ADh1nWqjpsAVE=', 'Content-Length': '0'} 2023-11-02T18:30:41.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000828e3f1187668104-006543ead1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.221 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.222 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.223 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.223 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.224 DEBUG:boto:Method: HEAD 2023-11-02T18:30:41.224 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.225 DEBUG:boto:Data: 2023-11-02T18:30:41.225 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.226 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:41.227 DEBUG:boto:Port: 8000 2023-11-02T18:30:41.227 DEBUG:boto:Params: {} 2023-11-02T18:30:41.228 DEBUG:boto:Token: None 2023-11-02T18:30:41.229 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.229 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TNkQ10wiA33Bw4ADh1nWqjpsAVE= 2023-11-02T18:30:41.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TNkQ10wiA33Bw4ADh1nWqjpsAVE=', 'Content-Length': '0'} 2023-11-02T18:30:41.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000860c52bd0de8f6ac-006543ead1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-11-02T18:30:41.234 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.235 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.235 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.236 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.236 DEBUG:boto:Method: GET 2023-11-02T18:30:41.236 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.237 DEBUG:boto:Data: 2023-11-02T18:30:41.237 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.238 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:41.238 DEBUG:boto:Port: 8001 2023-11-02T18:30:41.239 DEBUG:boto:Params: {} 2023-11-02T18:30:41.239 DEBUG:boto:Token: None 2023-11-02T18:30:41.240 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.240 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X7k+4jzblaVTanQzgjSuEEGYACc= 2023-11-02T18:30:41.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X7k+4jzblaVTanQzgjSuEEGYACc=', 'Content-Length': '0'} 2023-11-02T18:30:41.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb9790dfc8f54e5c-006543ead1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.245 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.245 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.246 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.246 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.247 DEBUG:boto:Method: GET 2023-11-02T18:30:41.247 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.248 DEBUG:boto:Data: 2023-11-02T18:30:41.248 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.248 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:41.249 DEBUG:boto:Port: 8000 2023-11-02T18:30:41.249 DEBUG:boto:Params: {} 2023-11-02T18:30:41.250 DEBUG:boto:Token: None 2023-11-02T18:30:41.250 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000467?versionId=null 2023-11-02T18:30:41.251 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X7k+4jzblaVTanQzgjSuEEGYACc= 2023-11-02T18:30:41.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X7k+4jzblaVTanQzgjSuEEGYACc=', 'Content-Length': '0'} 2023-11-02T18:30:41.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adb4d58fbf90e958-006543ead1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.255 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:41.256 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:41.256 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000467 2023-11-02T18:30:41.257 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000467 2023-11-02T18:30:41.257 DEBUG:boto:Method: GET 2023-11-02T18:30:41.258 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000467 2023-11-02T18:30:41.258 DEBUG:boto:Data: 2023-11-02T18:30:41.259 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:41.259 DEBUG:boto:Port: 8001 2023-11-02T18:30:41.260 DEBUG:boto:Params: {} 2023-11-02T18:30:41.260 DEBUG:boto:Token: None 2023-11-02T18:30:41.261 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/ 2023-11-02T18:30:41.261 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4= 2023-11-02T18:30:41.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4=', 'Content-Length': '0'} 2023-11-02T18:30:41.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b1e35960c27ff4b-006543ead1-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.267 DEBUG:boto:b'ibtljn-58obj000000000000000004671000falseobj000000000000000004672023-11-02T18:30:17.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:41.268 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:41.268 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:41.269 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000467 2023-11-02T18:30:41.269 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000467 2023-11-02T18:30:41.269 DEBUG:boto:Method: GET 2023-11-02T18:30:41.270 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000467 2023-11-02T18:30:41.270 DEBUG:boto:Data: 2023-11-02T18:30:41.271 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.271 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:41.272 DEBUG:boto:Port: 8000 2023-11-02T18:30:41.272 DEBUG:boto:Params: {} 2023-11-02T18:30:41.273 DEBUG:boto:Token: None 2023-11-02T18:30:41.273 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/ 2023-11-02T18:30:41.274 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4= 2023-11-02T18:30:41.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4=', 'Content-Length': '0'} 2023-11-02T18:30:41.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007486e322353a0f3e-006543ead1-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.284 DEBUG:boto:b'ibtljn-58obj000000000000000004671000falseobj000000000000000004672023-11-02T18:30:17.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:41.285 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.285 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.286 DEBUG:boto:Method: HEAD 2023-11-02T18:30:41.286 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.287 DEBUG:boto:Data: 2023-11-02T18:30:41.287 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.288 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:41.288 DEBUG:boto:Port: 8001 2023-11-02T18:30:41.289 DEBUG:boto:Params: {} 2023-11-02T18:30:41.289 DEBUG:boto:Token: None 2023-11-02T18:30:41.290 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.290 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sGcVHHVGI6Ya6blP6afmpu6Wj0w= 2023-11-02T18:30:41.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sGcVHHVGI6Ya6blP6afmpu6Wj0w=', 'Content-Length': '0'} 2023-11-02T18:30:41.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078ff49ad1c80d902-006543ead1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.294 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.294 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.295 DEBUG:boto:Method: HEAD 2023-11-02T18:30:41.295 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.296 DEBUG:boto:Data: 2023-11-02T18:30:41.296 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.296 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:41.297 DEBUG:boto:Port: 8000 2023-11-02T18:30:41.297 DEBUG:boto:Params: {} 2023-11-02T18:30:41.298 DEBUG:boto:Token: None 2023-11-02T18:30:41.298 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.299 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sGcVHHVGI6Ya6blP6afmpu6Wj0w= 2023-11-02T18:30:41.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sGcVHHVGI6Ya6blP6afmpu6Wj0w=', 'Content-Length': '0'} 2023-11-02T18:30:41.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686441567a6bd9d3-006543ead1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-11-02T18:30:41.303 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.303 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.303 DEBUG:boto:Method: GET 2023-11-02T18:30:41.304 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.304 DEBUG:boto:Data: 2023-11-02T18:30:41.305 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.305 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:41.306 DEBUG:boto:Port: 8001 2023-11-02T18:30:41.306 DEBUG:boto:Params: {} 2023-11-02T18:30:41.307 DEBUG:boto:Token: None 2023-11-02T18:30:41.307 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.308 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sfwk/hqGnMM6W9tful5LgzpcoJs= 2023-11-02T18:30:41.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sfwk/hqGnMM6W9tful5LgzpcoJs=', 'Content-Length': '0'} 2023-11-02T18:30:41.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-58:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011d0c0bbf8f83e44-006543ead1-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.355 DEBUG:boto:path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.356 DEBUG:boto:auth_path=/ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.356 DEBUG:boto:Method: GET 2023-11-02T18:30:41.357 DEBUG:boto:Path: /ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.357 DEBUG:boto:Data: 2023-11-02T18:30:41.358 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:41.359 DEBUG:boto:Port: 8000 2023-11-02T18:30:41.359 DEBUG:boto:Params: {} 2023-11-02T18:30:41.360 DEBUG:boto:Token: None 2023-11-02T18:30:41.360 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/obj00000000000000000467 2023-11-02T18:30:41.361 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sfwk/hqGnMM6W9tful5LgzpcoJs= 2023-11-02T18:30:41.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sfwk/hqGnMM6W9tful5LgzpcoJs=', 'Content-Length': '0'} 2023-11-02T18:30:41.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:30:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006002a2fb9a2b79ec-006543ead1-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.365 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:41.366 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:41.366 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000467 2023-11-02T18:30:41.366 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000467 2023-11-02T18:30:41.367 DEBUG:boto:Method: GET 2023-11-02T18:30:41.367 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000467 2023-11-02T18:30:41.368 DEBUG:boto:Data: 2023-11-02T18:30:41.368 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.369 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:30:41.369 DEBUG:boto:Port: 8001 2023-11-02T18:30:41.370 DEBUG:boto:Params: {} 2023-11-02T18:30:41.370 DEBUG:boto:Token: None 2023-11-02T18:30:41.371 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/ 2023-11-02T18:30:41.371 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4= 2023-11-02T18:30:41.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4=', 'Content-Length': '0'} 2023-11-02T18:30:41.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad3af096bc294de1-006543ead1-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.378 DEBUG:boto:b'ibtljn-58obj000000000000000004671000falseobj000000000000000004672023-11-02T18:30:17.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:41.379 DEBUG:boto:path=/ibtljn-58/ 2023-11-02T18:30:41.379 DEBUG:boto:auth_path=/ibtljn-58/ 2023-11-02T18:30:41.380 DEBUG:boto:path=/ibtljn-58/?prefix=obj00000000000000000467 2023-11-02T18:30:41.380 DEBUG:boto:auth_path=/ibtljn-58/?prefix=obj00000000000000000467 2023-11-02T18:30:41.381 DEBUG:boto:Method: GET 2023-11-02T18:30:41.381 DEBUG:boto:Path: /ibtljn-58/?prefix=obj00000000000000000467 2023-11-02T18:30:41.382 DEBUG:boto:Data: 2023-11-02T18:30:41.382 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:41.383 DEBUG:boto:Port: 8000 2023-11-02T18:30:41.384 DEBUG:boto:Params: {} 2023-11-02T18:30:41.385 DEBUG:boto:Token: None 2023-11-02T18:30:41.385 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-58/ 2023-11-02T18:30:41.386 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4= 2023-11-02T18:30:41.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0GcKGZtWReyKE20uYpFur+i8pO4=', 'Content-Length': '0'} 2023-11-02T18:30:41.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d4f3618c1963701-006543ead1-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.397 DEBUG:boto:b'ibtljn-58obj000000000000000004671000falseobj000000000000000004672023-11-02T18:30:17.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:30:41.398 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-58 zones={a2, a1} 2023-11-02T18:30:41.399 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2023-11-02T18:30:41.401 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:30:41.401 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:30:41.402 DEBUG:boto:Method: PUT 2023-11-02T18:30:41.402 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:30:41.403 DEBUG:boto:Data: 2023-11-02T18:30:41.403 DEBUG:boto:Headers: {} 2023-11-02T18:30:41.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:30:41.404 DEBUG:boto:Port: 8000 2023-11-02T18:30:41.404 DEBUG:boto:Params: {} 2023-11-02T18:30:41.405 DEBUG:boto:Token: None 2023-11-02T18:30:41.405 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:30:41 GMT /ibtljn-59/ 2023-11-02T18:30:41.406 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVvKSnNxzoP+rkfAOlK4qncu3/w= 2023-11-02T18:30:41.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:30:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVvKSnNxzoP+rkfAOlK4qncu3/w=', 'Content-Length': '0'} 2023-11-02T18:30:41.435 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004b11d58bd36f706b-006543ead1-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:30:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:30:41.436 DEBUG:rgw_multi.tests:created bucket=ibtljn-59 2023-11-02T18:30:41.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:41.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:41.438 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:41.482 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:41.482 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:41.541 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:30:41.574 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:30:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:30:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:30:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:30:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "0.000000" 2023-11-02T18:30:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949841.430647_492.1", 2023-11-02T18:30:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:30:41.430647Z" 2023-11-02T18:30:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:30:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T18:30:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:41.578 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:30:41.579 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:30:41.579 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:30:41.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:41.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:41.581 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:41.627 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:41.627 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:41.675 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:41.667+0000 7efcfc186f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:30:41.676 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:41.667+0000 7efcfc186f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:30:41.704 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:30:41.705 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:30:41.705 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:30:41.705 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:30:41.706 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:30:41.706 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:30:41.706 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:41.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:41.707 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:30:41.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:41.707 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:30:41.708 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:41.708 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:41.708 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:30:41.708 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:41.709 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:30:41.709 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:41.709 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:30:41.709 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:41.710 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:41.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:41.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:41.710 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:30:41.711 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:41.711 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:41.711 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:30:41.712 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:41.712 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:30:41.712 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:41.712 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:30:41.713 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:41.713 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:41.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:41.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:41.714 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:30:41.714 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:41.714 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:41.714 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:30:41.715 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:41.715 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:30:41.715 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:41.716 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:30:41.716 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:41.716 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:41.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:41.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:41.717 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:30:41.717 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:41.717 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:41.718 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:30:41.718 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:41.718 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:30:41.719 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:41.719 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:30:41.719 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:41.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:41.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:41.720 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:30:41.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:41.721 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:30:41.721 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:30:41.721 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:30:41.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:41.722 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:30:41Z" 2023-11-02T18:30:41.722 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:30:41.723 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:30:41Z" } 2023-11-02T18:30:41.724 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:30:41.724 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:30:41.724 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:30:41.725 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:30:46.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:46.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:46.727 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:46.773 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:46.774 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:46.829 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:46.819+0000 7f6ed8996f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:30:46.830 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:46.823+0000 7f6ed8996f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:30:46.857 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:30:46.857 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:30:46.858 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:30:46.858 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:30:46.858 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:30:46.858 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:30:46.859 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:46.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:46.859 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:30:46.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:46.859 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:30:46.860 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:46.860 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:46.860 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:30:46.860 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:46.861 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:30:46.861 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:46.861 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:30:46.861 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:46.861 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:46.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:46.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:46.862 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:30:46.862 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:46.863 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:46.863 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:30:46.863 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:46.863 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:30:46.863 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:46.864 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:30:46.864 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:46.864 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:46.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:46.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:46.865 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:30:46.865 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:46.865 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:46.865 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:30:46.866 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:46.866 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:30:46.866 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:46.866 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:30:46.866 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:46.867 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:46.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:46.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:46.867 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:30:46.868 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:46.868 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:46.868 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:30:46.868 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:46.868 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:30:46.869 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:46.869 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:30:46.869 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:46.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:46.870 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:46.870 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:30:46.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:46.870 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:30:46.870 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:30:46.871 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:30:46.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:46.871 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:30:46Z" 2023-11-02T18:30:46.871 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:30:46.872 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:30:46Z" } 2023-11-02T18:30:46.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:30:46.874 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:30:46.874 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:30:46.874 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:30:51.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:51.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:51.877 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:51.922 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:51.923 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:51.984 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:51.975+0000 7f342b519f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:30:51.985 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:51.979+0000 7f342b519f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:30:52.014 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:30:52.014 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:30:52.015 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:30:52.015 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:30:52.015 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:30:52.015 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:30:52.015 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:52.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:52.016 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:30:52.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:52.016 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:30:52.017 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:52.017 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:52.017 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:30:52.017 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:52.017 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:30:52.018 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:52.018 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:30:52.018 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:52.018 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:52.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:52.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:52.019 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:30:52.019 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:52.020 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:52.020 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:30:52.020 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:52.020 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:30:52.020 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:52.021 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:30:52.021 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:52.021 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:52.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:52.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:52.022 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:30:52.023 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:52.023 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:52.023 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:30:52.024 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:52.024 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:30:52.024 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:52.024 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:30:52.024 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:52.025 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:52.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:52.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:52.025 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:30:52.025 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:52.026 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:52.026 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:30:52.026 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:52.027 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:30:52.027 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:52.027 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:30:52.027 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:52.027 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:52.028 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:52.028 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:30:52.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:52.028 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:30:52.029 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:30:52.029 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:30:52.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:52.029 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:30:51Z" 2023-11-02T18:30:52.029 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:30:52.031 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:30:51Z" } 2023-11-02T18:30:52.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:30:52.032 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:30:52.032 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:30:52.032 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:30:57.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:57.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:30:57.035 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:30:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:30:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:30:57.142 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:57.139+0000 7f83f50aff40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:30:57.144 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:30:57.139+0000 7f83f50aff40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:30:57.171 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:30:57.172 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:30:57.172 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:30:57.172 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:30:57.172 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:30:57.173 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:30:57.173 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:57.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:57.173 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:30:57.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:57.174 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:30:57.174 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:57.174 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:57.174 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:30:57.175 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:57.175 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:30:57.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:57.175 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:30:57.175 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:57.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:57.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:57.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:57.176 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:30:57.176 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:57.177 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:57.177 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:30:57.177 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:57.177 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:30:57.178 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:57.178 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:30:57.178 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:57.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:57.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:57.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:57.179 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:30:57.179 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:57.179 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:57.180 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:30:57.180 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:57.180 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:30:57.180 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:57.180 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:30:57.181 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:57.181 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:57.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:57.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:30:57.181 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:30:57.182 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:30:57.182 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:30:57.182 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:30:57.182 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:30:57.182 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:30:57.183 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:30:57.183 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:30:57.183 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:30:57.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:57.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:30:57.184 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:30:57.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:57.184 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:30:57.184 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:30:57.185 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:30:57.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:30:57.185 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:30:57Z" 2023-11-02T18:30:57.185 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:30:57.186 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:30:57Z" } 2023-11-02T18:30:57.187 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:30:57.187 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:30:57.188 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:30:57.188 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:31:02.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:02.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:02.190 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:31:02.237 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:31:02.237 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:31:02.297 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:02.291+0000 7f29d0e9af40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:02.299 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:02.295+0000 7f29d0e9af40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:02.326 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:31:02.326 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:31:02.326 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:31:02.327 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:31:02.327 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:31:02.327 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:31:02.327 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:02.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:02.328 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:31:02.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:02.328 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:31:02.329 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:02.329 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:02.329 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:31:02.329 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:02.329 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:31:02.330 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:02.330 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:31:02.330 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:02.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:02.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:02.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:02.331 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:31:02.331 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:02.332 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:02.332 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:31:02.332 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:02.332 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:02.333 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:02.333 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:31:02.333 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:02.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:02.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:02.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:02.334 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:31:02.334 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:02.334 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:02.335 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:31:02.335 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:02.335 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:02.335 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:02.336 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:31:02.336 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:02.336 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:02.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:02.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:02.337 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:31:02.337 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:02.337 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:02.337 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:31:02.338 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:02.338 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:02.338 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:02.338 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:31:02.339 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:02.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:02.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:02.339 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:31:02.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:02.340 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:31:02.340 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:31:02.340 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:31:02.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:02.341 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:31:02Z" 2023-11-02T18:31:02.341 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:31:02.342 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:31:02Z" } 2023-11-02T18:31:02.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:31:02.343 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:02.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:02.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:31:07.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:07.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:07.346 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:31:07.395 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:31:07.395 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:31:07.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:07.451+0000 7fe69a320f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:07.457 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:07.451+0000 7fe69a320f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:07.486 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:31:07.486 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:31:07.486 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:31:07.487 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:31:07.487 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:31:07.487 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:31:07.487 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:07.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:07.488 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:31:07.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:07.488 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:31:07.488 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:07.489 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:07.489 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:31:07.489 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:07.489 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:31:07.489 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:07.490 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:31:07.490 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:07.490 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:07.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:07.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:07.491 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:31:07.491 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:07.491 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:07.491 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:31:07.492 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:07.492 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:07.492 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:07.492 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:31:07.492 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:07.493 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:07.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:07.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:07.493 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:31:07.494 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:07.494 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:07.494 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:31:07.494 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:07.494 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:07.495 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:07.495 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:31:07.495 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:07.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:07.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:07.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:07.496 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:31:07.496 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:07.496 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:07.497 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:31:07.497 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:07.497 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:07.497 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:07.497 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:31:07.498 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:07.498 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:07.498 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:07.498 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:31:07.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:07.499 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:31:07.499 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:31:07.499 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:31:07.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:07.500 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:31:07Z" 2023-11-02T18:31:07.500 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:31:07.501 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:31:07Z" } 2023-11-02T18:31:07.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:31:07.502 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:07.502 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:07.503 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:31:12.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:12.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:12.505 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:31:12.552 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:31:12.552 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:31:12.613 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:12.607+0000 7fc6f1b8ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:12.614 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:12.611+0000 7fc6f1b8ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:12.641 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:31:12.641 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:31:12.642 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:31:12.642 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:31:12.642 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:31:12.642 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:31:12.643 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:12.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:12.643 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:31:12.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:12.643 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:31:12.644 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:12.644 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:12.644 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:31:12.644 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:12.644 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:31:12.645 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:12.645 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:31:12.645 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:12.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:12.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:12.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:12.646 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:31:12.646 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:12.646 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:12.647 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:31:12.647 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:12.647 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:12.647 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:12.647 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:31:12.648 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:12.648 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:12.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:12.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:12.649 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:31:12.649 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:12.649 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:12.649 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:31:12.649 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:12.650 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:12.650 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:12.650 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:31:12.650 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:12.650 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:12.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:12.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:12.651 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:31:12.651 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:12.652 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:12.652 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:31:12.652 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:12.652 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:12.652 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:12.653 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:31:12.653 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:12.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:12.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:12.653 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:31:12.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:12.654 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:31:12.654 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:31:12.654 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:31:12.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:12.655 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:31:12Z" 2023-11-02T18:31:12.655 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:31:12.656 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:31:12Z" } 2023-11-02T18:31:12.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:31:12.657 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:12.657 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:12.658 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:31:17.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:17.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:17.660 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:31:17.705 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:31:17.706 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:31:17.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:17.755+0000 7fc808605f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:17.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:17.759+0000 7fc808605f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:17.790 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:31:17.791 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:31:17.791 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:31:17.791 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:31:17.791 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:31:17.791 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:31:17.792 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:17.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:17.792 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:31:17.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:17.792 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:31:17.793 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:17.793 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:17.793 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:31:17.793 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:17.794 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:31:17.794 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:17.794 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:31:17.794 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:17.794 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:17.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:17.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:17.795 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:31:17.795 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:17.796 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:17.796 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:31:17.796 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:17.796 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:17.796 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:17.797 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:31:17.797 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:17.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:17.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:17.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:17.798 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:31:17.798 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:17.798 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:17.798 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:31:17.799 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:17.799 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:17.799 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:17.799 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:31:17.800 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:17.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:17.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:17.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:17.800 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:31:17.801 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:17.801 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:17.801 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:31:17.801 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:17.801 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:17.802 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:17.802 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:31:17.802 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:17.802 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:17.803 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:17.803 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:31:17.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:17.803 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:31:17.803 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:31:17.804 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:31:17.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:17.804 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:31:17Z" 2023-11-02T18:31:17.804 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:31:17.805 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:31:17Z" } 2023-11-02T18:31:17.806 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:31:17.806 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:17.807 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:17.807 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:31:22.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:22.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:22.809 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:31:22.856 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:31:22.857 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:31:22.917 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:22.911+0000 7ffa31f83f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:22.919 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:22.915+0000 7ffa31f83f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:22.947 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:31:22.947 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:31:22.948 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:31:22.948 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:31:22.948 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:31:22.948 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:31:22.949 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:22.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:22.949 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:31:22.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:22.949 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:31:22.950 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:22.950 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:22.950 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:31:22.950 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:22.951 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:31:22.951 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:22.951 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:31:22.951 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:22.951 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:22.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:22.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:22.952 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:31:22.952 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:22.952 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:22.953 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:31:22.953 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:22.953 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:22.953 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:22.954 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:31:22.954 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:22.954 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:22.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:22.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:22.955 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:31:22.955 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:22.955 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:22.955 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:31:22.956 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:22.956 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:22.956 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:22.956 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:31:22.956 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:22.957 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:22.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:22.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:22.957 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:31:22.958 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:22.958 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:22.958 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:31:22.958 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:22.959 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:22.959 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:22.959 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:31:22.959 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:22.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:22.960 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:22.960 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:31:22.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:22.960 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:31:22.960 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:31:22.961 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:31:22.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:22.961 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:31:22Z" 2023-11-02T18:31:22.961 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:31:22.963 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:31:22Z" } 2023-11-02T18:31:22.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:31:22.964 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:22.964 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:22.965 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:31:27.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:27.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:27.967 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:31:28.013 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:31:28.013 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:31:28.069 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:28.067+0000 7fad61722f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:28.071 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:28.067+0000 7fad61722f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:28.100 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:31:28.100 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:31:28.100 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:31:28.100 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:31:28.101 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:31:28.101 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:31:28.101 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:28.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:28.101 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:31:28.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:28.102 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:31:28.102 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:28.102 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:28.103 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:31:28.103 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:28.103 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:31:28.103 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:28.103 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:31:28.104 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:28.104 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:28.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:28.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:28.104 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:31:28.105 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:28.105 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:28.105 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:31:28.105 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:28.105 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:28.106 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:28.106 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:31:28.106 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:28.106 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:28.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:28.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:28.107 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:31:28.107 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:28.107 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:28.108 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:31:28.108 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:28.108 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:28.108 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:28.108 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:31:28.109 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:28.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:28.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:28.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:28.110 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:31:28.110 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:28.110 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:28.110 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:31:28.110 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:28.111 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:28.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:28.111 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:31:28.111 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:28.111 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:28.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:28.112 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:31:28.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:28.112 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:31:28.113 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:31:28.113 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:31:28.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:28.113 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:31:28Z" 2023-11-02T18:31:28.113 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:31:28.115 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:31:28Z" } 2023-11-02T18:31:28.115 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:31:28.116 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:28.116 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:28.116 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:31:33.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:33.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:33.119 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:31:33.165 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:31:33.165 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:31:33.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:33.219+0000 7f42b1abcf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:33.226 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:33.223+0000 7f42b1abcf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:33.252 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:31:33.253 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:31:33.253 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:31:33.253 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:31:33.254 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:31:33.254 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:31:33.254 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:33.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:33.255 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:31:33.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:33.255 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:31:33.255 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:33.255 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:33.256 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:31:33.256 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:33.256 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:31:33.256 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:33.256 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:31:33.257 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:33.257 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:33.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:33.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:33.258 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:31:33.258 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:33.258 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:33.258 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:31:33.259 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:33.259 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:33.259 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:33.259 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:31:33.259 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:33.260 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:33.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:33.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:33.260 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:31:33.261 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:33.261 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:33.261 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:31:33.261 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:33.261 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:33.262 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:33.262 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:31:33.262 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:33.262 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:33.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:33.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:33.263 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:31:33.263 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:33.263 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:33.264 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:31:33.264 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:33.264 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:33.264 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:33.265 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:31:33.265 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:33.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:33.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:33.266 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:31:33.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:33.266 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:31:33.266 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:31:33.266 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:31:33.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:33.267 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:31:33Z" 2023-11-02T18:31:33.267 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:31:33.268 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:31:33Z" } 2023-11-02T18:31:33.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:31:33.270 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:33.270 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:33.270 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:31:38.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:38.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:38.273 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:31:38.320 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:31:38.321 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:31:38.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:38.375+0000 7f6102947f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:38.382 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:38.379+0000 7f6102947f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:38.410 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:31:38.411 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:31:38.411 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:31:38.411 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:31:38.411 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:31:38.412 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:31:38.412 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:38.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:38.412 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:31:38.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:38.413 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:31:38.413 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:38.413 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:38.413 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:31:38.413 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:38.414 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:31:38.414 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:38.414 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:31:38.414 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:38.415 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:38.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:38.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:38.415 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:31:38.415 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:38.416 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:38.416 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:31:38.416 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:38.416 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:38.417 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:38.417 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:31:38.417 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:38.417 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:38.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:38.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:38.418 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:31:38.418 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:38.418 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:38.419 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:31:38.419 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:38.419 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:38.419 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:38.419 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:31:38.420 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:38.420 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:38.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:38.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:38.421 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:31:38.421 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:38.421 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:38.421 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:31:38.421 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:38.422 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:38.422 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:38.422 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:31:38.422 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:38.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:38.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:38.423 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:31:38.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:38.423 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:31:38.424 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:31:38.424 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:31:38.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:38.424 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:31:38Z" 2023-11-02T18:31:38.424 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:31:38.425 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:31:38Z" } 2023-11-02T18:31:38.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:31:38.427 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:38.427 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:38.427 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:31:43.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:43.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:43.429 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:31:43.476 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:31:43.476 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:31:43.537 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:43.531+0000 7fb929f99f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:43.539 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:43.535+0000 7fb929f99f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:43.569 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:31:43.569 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:31:43.570 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:31:43.570 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:31:43.570 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:31:43.570 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:31:43.571 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:43.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:43.571 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:31:43.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:43.571 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:31:43.572 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:43.572 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:43.572 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:31:43.572 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:43.573 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:31:43.573 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:43.573 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:31:43.574 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:43.574 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:43.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:43.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:43.575 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:31:43.575 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:43.575 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:43.575 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:31:43.576 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:43.576 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:43.576 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:43.577 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:31:43.577 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:43.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:43.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:43.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:43.578 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:31:43.578 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:43.578 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:43.579 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:31:43.579 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:43.579 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:43.579 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:43.580 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:31:43.580 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:43.580 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:43.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:43.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:43.581 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:31:43.581 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:43.582 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:43.582 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:31:43.582 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:43.582 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:43.583 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:43.583 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:31:43.583 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:43.583 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:43.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:43.584 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:31:43.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:43.585 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:31:43.585 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:31:43.585 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:31:43.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:43.585 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:31:43Z" 2023-11-02T18:31:43.586 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:31:43.587 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:31:43Z" } 2023-11-02T18:31:43.588 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:31:43.588 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:43.589 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:43.589 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:31:48.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:48.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:48.592 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:31:48.633 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:31:48.634 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:31:48.685 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:48.678+0000 7f12a52e0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:48.686 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:48.682+0000 7f12a52e0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:48.713 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:31:48.714 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:31:48.714 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:31:48.714 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:31:48.714 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:31:48.715 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:31:48.715 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:48.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:48.715 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:31:48.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:48.716 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:31:48.716 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:48.716 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:48.716 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:31:48.717 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:48.717 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:31:48.717 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:48.717 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:31:48.717 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:48.718 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:48.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:48.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:48.718 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:31:48.719 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:48.719 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:48.719 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:31:48.719 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:48.719 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:48.720 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:48.720 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:31:48.720 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:48.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:48.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:48.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:48.721 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:31:48.721 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:48.721 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:48.722 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:31:48.722 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:48.722 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:48.722 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:48.723 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:31:48.723 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:48.723 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:48.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:48.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:48.724 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:31:48.724 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:48.724 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:48.724 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:31:48.725 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:48.725 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:48.725 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:48.725 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:31:48.725 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:48.726 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:48.726 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:48.726 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:31:48.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:48.727 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:31:48.727 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:31:48.727 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:31:48.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:48.727 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:31:48Z" 2023-11-02T18:31:48.728 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:31:48.729 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:31:48Z" } 2023-11-02T18:31:48.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:31:48.730 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:48.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:48.731 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:31:53.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:53.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:53.733 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:31:53.780 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:31:53.780 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:31:53.841 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:53.834+0000 7f536338cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:53.842 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:53.838+0000 7f536338cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:53.868 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:31:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:31:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:31:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:31:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:31:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:31:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:31:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:31:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:31:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:31:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:31:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:31:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:31:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:31:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:31:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:31:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:31:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:31:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:31:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:31:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:31:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:31:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:31:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:31:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:31:53Z" 2023-11-02T18:31:53.883 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:31:53.884 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:31:53Z" } 2023-11-02T18:31:53.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:31:53.885 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:53.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:53.886 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:31:58.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:58.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:31:58.889 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:31:58.936 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:31:58.936 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:31:58.995 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:58.990+0000 7fd878590f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:58.997 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:31:58.990+0000 7fd878590f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:31:59.025 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:31:59.026 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:31:59.026 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:31:59.026 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:31:59.026 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:31:59.027 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:31:59.029 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:59.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:59.030 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:31:59.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:59.031 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:31:59.031 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:59.032 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:59.032 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:31:59.033 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:59.033 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:31:59.033 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:59.034 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:31:59.034 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:59.034 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:59.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:59.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:59.035 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:31:59.036 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:59.036 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:59.036 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:31:59.037 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:59.037 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:59.037 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:59.037 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:31:59.038 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:59.038 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:59.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:59.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:59.039 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:31:59.039 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:59.039 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:59.039 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:31:59.039 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:59.040 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:59.040 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:59.040 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:31:59.040 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:59.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:59.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:59.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:31:59.041 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:31:59.041 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:31:59.042 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:31:59.042 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:31:59.042 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:31:59.042 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:31:59.043 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:31:59.043 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:31:59.043 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:31:59.043 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:59.044 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:31:59.044 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:31:59.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:59.044 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:31:59.044 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:31:59.045 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:31:59.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:31:59.045 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:31:58Z" 2023-11-02T18:31:59.045 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:31:59.047 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:31:58Z" } 2023-11-02T18:31:59.047 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:31:59.048 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:59.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:31:59.048 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:32:04.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:32:04.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:32:04.051 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:32:04.098 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:32:04.098 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:32:04.156 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:04.150+0000 7f9135f58f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:32:04.158 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:04.154+0000 7f9135f58f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:32:04.186 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:32:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:32:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:32:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:32:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:32:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:32:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:32:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:32:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:32:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:32:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:32:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:32:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:32:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:32:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:32:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:32:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:32:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:32:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:32:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:32:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:32:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:32:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:32:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:32:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:32:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:32:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:32:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:32:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:32:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:32:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:32:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:32:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:32:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:32:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:32:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:32:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:32:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:32:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:32:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:32:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:32:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:32:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:32:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:32:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:32:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:32:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:32:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:32:04Z" 2023-11-02T18:32:04.201 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:32:04.202 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:32:04Z" } 2023-11-02T18:32:04.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:32:04.204 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:32:04.204 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:32:04.205 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:32:09.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:32:09.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:32:09.208 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:32:09.254 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:32:09.254 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:32:09.314 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:09.310+0000 7fb57cce4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:32:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:09.310+0000 7fb57cce4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:32:09.345 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:32:09.345 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:32:09.346 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:32:09.346 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:32:09.346 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:32:09.346 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:32:09.346 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:09.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:09.347 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:32:09.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:32:09.347 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:32:09.348 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:32:09.348 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:32:09.348 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:32:09.348 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:32:09.349 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:32:09.349 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:32:09.349 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:32:09.349 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:09.350 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:09.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:09.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:32:09.350 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:32:09.350 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:32:09.351 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:32:09.351 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:32:09.351 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:32:09.351 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:32:09.351 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:32:09.352 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:32:09.352 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:09.352 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:09.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:09.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:32:09.353 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:32:09.353 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:32:09.353 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:32:09.353 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949781.602188_453.1", 2023-11-02T18:32:09.354 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:32:09.354 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:32:09.354 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:32:09.354 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:29:41.602188Z", 2023-11-02T18:32:09.354 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:09.355 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:09.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:09.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:32:09.355 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:32:09.356 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:32:09.356 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:32:09.356 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:32:09.356 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:32:09.357 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:32:09.357 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:32:09.357 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:32:09.357 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:09.357 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:09.358 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:09.358 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:32:09.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:09.358 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:32:09.359 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:32:09.359 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:32:09.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:09.359 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:32:09Z" 2023-11-02T18:32:09.360 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:32:09.361 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949781.602188_453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:29:41.602188Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:32:09Z" } 2023-11-02T18:32:09.361 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949781.602188_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:29:41.602188Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:32:09.362 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:32:09.362 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949781.602188_453.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:32:09.363 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698949841.430647_492.1 target=1_1698949781.602188_453.1 2023-11-02T18:32:14.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:32:14.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:32:14.366 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:32:14.413 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:32:14.413 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:32:14.474 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:14.470+0000 7fe397c41f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:32:14.475 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:14.470+0000 7fe397c41f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:32:14.503 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:32:14.504 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:32:14.504 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:32:14.504 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:32:14.505 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:32:14.505 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:32:14.505 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:14.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:14.505 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:32:14.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:32:14.506 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:32:14.506 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:32:14.506 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:32:14.507 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:32:14.507 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:32:14.507 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:32:14.507 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:32:14.508 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:32:14.508 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:14.508 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:14.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:14.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:32:14.509 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:32:14.509 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:32:14.509 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:32:14.509 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:32:14.510 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:32:14.510 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:32:14.510 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:32:14.510 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:32:14.511 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:14.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:14.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:14.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:32:14.512 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:32:14.512 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:32:14.512 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:32:14.512 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698949841.430647_492.1", 2023-11-02T18:32:14.512 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:32:14.513 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:32:14.513 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:32:14.513 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:30:41.430647Z", 2023-11-02T18:32:14.513 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:14.514 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:14.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:14.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:32:14.514 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:32:14.515 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:32:14.515 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:32:14.515 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:32:14.515 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:32:14.516 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:32:14.516 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:32:14.516 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:32:14.516 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:32:14.516 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:14.517 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:32:14.517 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:32:14.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:14.518 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:32:14.518 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:32:14.518 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:32:14.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:32:14.519 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:32:14Z" 2023-11-02T18:32:14.519 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:32:14.520 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698949841.430647_492.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:30:41.430647Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:32:14Z" } 2023-11-02T18:32:14.521 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949841.430647_492.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:30:41.430647Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:32:14.521 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:32:14.522 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949841.430647_492.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:32:14.522 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:32:14.522 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:14.523 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:14.524 DEBUG:boto:Method: HEAD 2023-11-02T18:32:14.524 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:32:14.524 DEBUG:boto:Data: 2023-11-02T18:32:14.525 DEBUG:boto:Headers: {} 2023-11-02T18:32:14.525 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:14.526 DEBUG:boto:Port: 8000 2023-11-02T18:32:14.526 DEBUG:boto:Params: {} 2023-11-02T18:32:14.527 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:32:14.527 DEBUG:boto:Token: None 2023-11-02T18:32:14.528 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:14 GMT /ibtljn-59/ 2023-11-02T18:32:14.529 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vb1iOQQPNHKYEJg8y3bKAN8V22c= 2023-11-02T18:32:14.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vb1iOQQPNHKYEJg8y3bKAN8V22c=', 'Content-Length': '0'} 2023-11-02T18:32:14.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f9f624f93637857-006543eb2e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:14.536 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:32:14.537 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:32:14.537 DEBUG:boto:Method: PUT 2023-11-02T18:32:14.538 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:32:14.538 DEBUG:boto:Data: 2023-11-02T18:32:14.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:32:14.539 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:14.540 DEBUG:boto:Port: 8000 2023-11-02T18:32:14.540 DEBUG:boto:Params: {} 2023-11-02T18:32:14.540 DEBUG:boto:Token: None 2023-11-02T18:32:14.541 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:32:14 GMT /ibtljn-59/obj11 2023-11-02T18:32:14.541 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dUB44BoZfpgIy5PY5VH/GfRihLQ= 2023-11-02T18:32:14.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:32:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dUB44BoZfpgIy5PY5VH/GfRihLQ='} 2023-11-02T18:32:14.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003b6582661d13270-006543eb2e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:14.557 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:14.558 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:14.558 DEBUG:boto:Method: HEAD 2023-11-02T18:32:14.558 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:32:14.559 DEBUG:boto:Data: 2023-11-02T18:32:14.559 DEBUG:boto:Headers: {} 2023-11-02T18:32:14.560 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:14.560 DEBUG:boto:Port: 8000 2023-11-02T18:32:14.561 DEBUG:boto:Params: {} 2023-11-02T18:32:14.561 DEBUG:boto:Token: None 2023-11-02T18:32:14.562 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:14 GMT /ibtljn-59/ 2023-11-02T18:32:14.562 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vb1iOQQPNHKYEJg8y3bKAN8V22c= 2023-11-02T18:32:14.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vb1iOQQPNHKYEJg8y3bKAN8V22c=', 'Content-Length': '0'} 2023-11-02T18:32:14.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd8e589be280e4d7-006543eb2e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:14.566 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:32:14.567 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:32:14.567 DEBUG:boto:Method: PUT 2023-11-02T18:32:14.568 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:32:14.568 DEBUG:boto:Data: 2023-11-02T18:32:14.569 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:32:14.569 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:14.570 DEBUG:boto:Port: 8000 2023-11-02T18:32:14.570 DEBUG:boto:Params: {} 2023-11-02T18:32:14.570 DEBUG:boto:Token: None 2023-11-02T18:32:14.571 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:32:14 GMT /ibtljn-59/obj22 2023-11-02T18:32:14.571 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TToKBJ8T8Se1WAicxdgjHLoJ15k= 2023-11-02T18:32:14.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:32:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TToKBJ8T8Se1WAicxdgjHLoJ15k='} 2023-11-02T18:32:14.579 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3903d6a1722ba32-006543eb2e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:14.579 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:14.580 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:14.580 DEBUG:boto:Method: HEAD 2023-11-02T18:32:14.580 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:32:14.581 DEBUG:boto:Data: 2023-11-02T18:32:14.581 DEBUG:boto:Headers: {} 2023-11-02T18:32:14.582 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:14.582 DEBUG:boto:Port: 8000 2023-11-02T18:32:14.582 DEBUG:boto:Params: {} 2023-11-02T18:32:14.583 DEBUG:boto:Token: None 2023-11-02T18:32:14.583 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:14 GMT /ibtljn-59/ 2023-11-02T18:32:14.584 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vb1iOQQPNHKYEJg8y3bKAN8V22c= 2023-11-02T18:32:14.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vb1iOQQPNHKYEJg8y3bKAN8V22c=', 'Content-Length': '0'} 2023-11-02T18:32:14.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000644d4437083f8c06-006543eb2e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:14.589 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:32:14.590 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:32:14.590 DEBUG:boto:Method: PUT 2023-11-02T18:32:14.591 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:32:14.591 DEBUG:boto:Data: 2023-11-02T18:32:14.592 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:32:14.592 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:14.593 DEBUG:boto:Port: 8000 2023-11-02T18:32:14.593 DEBUG:boto:Params: {} 2023-11-02T18:32:14.594 DEBUG:boto:Token: None 2023-11-02T18:32:14.594 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:32:14 GMT /ibtljn-59/obj33 2023-11-02T18:32:14.595 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zaucH40hjhQKJ9W/H5V0PPiuADk= 2023-11-02T18:32:14.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:32:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zaucH40hjhQKJ9W/H5V0PPiuADk='} 2023-11-02T18:32:14.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006860a9eda2e75a75-006543eb2e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:14.639 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:14.640 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:14.640 DEBUG:boto:Method: HEAD 2023-11-02T18:32:14.640 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:32:14.641 DEBUG:boto:Data: 2023-11-02T18:32:14.641 DEBUG:boto:Headers: {} 2023-11-02T18:32:14.642 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:14.642 DEBUG:boto:Port: 8000 2023-11-02T18:32:14.643 DEBUG:boto:Params: {} 2023-11-02T18:32:14.643 DEBUG:boto:Token: None 2023-11-02T18:32:14.644 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:14 GMT /ibtljn-59/ 2023-11-02T18:32:14.644 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vb1iOQQPNHKYEJg8y3bKAN8V22c= 2023-11-02T18:32:14.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vb1iOQQPNHKYEJg8y3bKAN8V22c=', 'Content-Length': '0'} 2023-11-02T18:32:14.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013701019f6a24b1a-006543eb2e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:14.649 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:32:14.649 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:32:14.649 DEBUG:boto:Method: PUT 2023-11-02T18:32:14.650 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:32:14.650 DEBUG:boto:Data: 2023-11-02T18:32:14.651 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:32:14.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:14.652 DEBUG:boto:Port: 8000 2023-11-02T18:32:14.652 DEBUG:boto:Params: {} 2023-11-02T18:32:14.653 DEBUG:boto:Token: None 2023-11-02T18:32:14.653 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:32:14 GMT /ibtljn-59/obj44 2023-11-02T18:32:14.654 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0Amfop3cLf/NGQnp+d49pdVhPIo= 2023-11-02T18:32:14.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:32:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0Amfop3cLf/NGQnp+d49pdVhPIo='} 2023-11-02T18:32:14.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b4f96e13f38765a-006543eb2e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:14.665 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:14.666 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:14.666 DEBUG:boto:Method: HEAD 2023-11-02T18:32:14.667 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:32:14.667 DEBUG:boto:Data: 2023-11-02T18:32:14.667 DEBUG:boto:Headers: {} 2023-11-02T18:32:14.668 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:14.668 DEBUG:boto:Port: 8000 2023-11-02T18:32:14.669 DEBUG:boto:Params: {} 2023-11-02T18:32:14.669 DEBUG:boto:Token: None 2023-11-02T18:32:14.670 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:14 GMT /ibtljn-59/ 2023-11-02T18:32:14.670 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vb1iOQQPNHKYEJg8y3bKAN8V22c= 2023-11-02T18:32:14.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vb1iOQQPNHKYEJg8y3bKAN8V22c=', 'Content-Length': '0'} 2023-11-02T18:32:14.675 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005cb1c012c0aca42b-006543eb2e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:14.676 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:32:14.676 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:32:14.677 DEBUG:boto:Method: PUT 2023-11-02T18:32:14.677 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:32:14.677 DEBUG:boto:Data: 2023-11-02T18:32:14.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:32:14.678 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:14.679 DEBUG:boto:Port: 8000 2023-11-02T18:32:14.679 DEBUG:boto:Params: {} 2023-11-02T18:32:14.680 DEBUG:boto:Token: None 2023-11-02T18:32:14.680 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:32:14 GMT /ibtljn-59/obj55 2023-11-02T18:32:14.681 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rxEVw/Cfuxua7sCKJp8lgRCM97U= 2023-11-02T18:32:14.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:32:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rxEVw/Cfuxua7sCKJp8lgRCM97U='} 2023-11-02T18:32:14.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ad055b29c1524f6-006543eb2e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:14.727 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-59 2023-11-02T18:32:14.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:32:14.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:32:14.729 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:32:14.780 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:32:14.781 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:32:14.849 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:14.842+0000 7f924f308f40 1 waiting to reach incremental sync.. 2023-11-02T18:32:19.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:19.846+0000 7f924f308f40 1 waiting to reach incremental sync.. 2023-11-02T18:32:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:24.846+0000 7f924f308f40 1 bucket sync caught up with source: 2023-11-02T18:32:24.854 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.41.6, 00000000002.66.6, 00000000002.44.6] 2023-11-02T18:32:24.854 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000001.41.6, 00000000002.66.6, 00000000002.44.6] 2023-11-02T18:32:24.854 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:24.846+0000 7f924f308f40 0 bucket checkpoint complete 2023-11-02T18:32:24.885 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-59 2023-11-02T18:32:24.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:32:24.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:32:24.887 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:32:24.936 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:32:24.937 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:32:25.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:25.002+0000 7f2080262f40 1 waiting to reach incremental sync.. 2023-11-02T18:32:30.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:30.006+0000 7f2080262f40 1 waiting to reach incremental sync.. 2023-11-02T18:32:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:35.006+0000 7f2080262f40 1 waiting to reach incremental sync.. 2023-11-02T18:32:40.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:40.006+0000 7f2080262f40 1 waiting to reach incremental sync.. 2023-11-02T18:32:45.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:45.010+0000 7f2080262f40 1 bucket sync caught up with source: 2023-11-02T18:32:45.016 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.76.6, 00000000002.265.6, 00000000002.322.6] 2023-11-02T18:32:45.016 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000001.76.6, 00000000002.265.6, 00000000002.322.6] 2023-11-02T18:32:45.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:45.010+0000 7f2080262f40 0 bucket checkpoint complete 2023-11-02T18:32:45.047 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:32:45.048 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.049 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.049 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.050 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:32:45.050 DEBUG:boto:Data: 2023-11-02T18:32:45.050 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.051 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.051 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.052 DEBUG:boto:Params: {} 2023-11-02T18:32:45.052 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:32:45.053 DEBUG:boto:Token: None 2023-11-02T18:32:45.053 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.054 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B4BCyFi4miGBY26a6WdDpwzJYCw= 2023-11-02T18:32:45.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B4BCyFi4miGBY26a6WdDpwzJYCw=', 'Content-Length': '0'} 2023-11-02T18:32:45.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4f40a78afeabf5f-006543eb4d-1b8c-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.061 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.062 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.063 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.063 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:32:45.064 DEBUG:boto:Data: 2023-11-02T18:32:45.064 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.064 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.065 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.065 DEBUG:boto:Params: {} 2023-11-02T18:32:45.066 DEBUG:boto:Token: None 2023-11-02T18:32:45.066 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.067 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B4BCyFi4miGBY26a6WdDpwzJYCw= 2023-11-02T18:32:45.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B4BCyFi4miGBY26a6WdDpwzJYCw=', 'Content-Length': '0'} 2023-11-02T18:32:45.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccc988a853331cc1-006543eb4d-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.073 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:32:45.073 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.074 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.074 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:32:45.075 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:32:45.075 DEBUG:boto:Method: GET 2023-11-02T18:32:45.076 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:32:45.077 DEBUG:boto:Data: 2023-11-02T18:32:45.077 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.078 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.078 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.078 DEBUG:boto:Params: {} 2023-11-02T18:32:45.079 DEBUG:boto:Token: None 2023-11-02T18:32:45.079 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/?versions 2023-11-02T18:32:45.081 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Cqjzkt431UO7M48HqV/JlOo/+Y8= 2023-11-02T18:32:45.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Cqjzkt431UO7M48HqV/JlOo/+Y8=', 'Content-Length': '0'} 2023-11-02T18:32:45.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a550e0ffac867f1d-006543eb4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.087 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:32:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:32:45.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:32:45.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:32:45.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:32:45.090 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:32:45.090 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.091 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.091 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:32:45.092 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:32:45.092 DEBUG:boto:Method: GET 2023-11-02T18:32:45.092 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:32:45.093 DEBUG:boto:Data: 2023-11-02T18:32:45.093 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.094 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.094 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.094 DEBUG:boto:Params: {} 2023-11-02T18:32:45.095 DEBUG:boto:Token: None 2023-11-02T18:32:45.095 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/?versions 2023-11-02T18:32:45.096 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Cqjzkt431UO7M48HqV/JlOo/+Y8= 2023-11-02T18:32:45.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Cqjzkt431UO7M48HqV/JlOo/+Y8=', 'Content-Length': '0'} 2023-11-02T18:32:45.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087ecdaf87e86cc4c-006543eb4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.102 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:32:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:32:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:32:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:32:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:32:45.105 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.105 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.106 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:32:45.106 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:32:45.107 DEBUG:boto:Method: GET 2023-11-02T18:32:45.107 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:32:45.107 DEBUG:boto:Data: 2023-11-02T18:32:45.108 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.108 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.109 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.109 DEBUG:boto:Params: {} 2023-11-02T18:32:45.110 DEBUG:boto:Token: None 2023-11-02T18:32:45.110 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/?versions 2023-11-02T18:32:45.111 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Cqjzkt431UO7M48HqV/JlOo/+Y8= 2023-11-02T18:32:45.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Cqjzkt431UO7M48HqV/JlOo/+Y8=', 'Content-Length': '0'} 2023-11-02T18:32:45.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd4474f75da7d915-006543eb4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.116 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.117 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.118 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.118 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:32:45.118 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:32:45.119 DEBUG:boto:Method: GET 2023-11-02T18:32:45.119 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:32:45.120 DEBUG:boto:Data: 2023-11-02T18:32:45.120 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.121 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.121 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.121 DEBUG:boto:Params: {} 2023-11-02T18:32:45.122 DEBUG:boto:Token: None 2023-11-02T18:32:45.122 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/?versions 2023-11-02T18:32:45.123 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Cqjzkt431UO7M48HqV/JlOo/+Y8= 2023-11-02T18:32:45.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Cqjzkt431UO7M48HqV/JlOo/+Y8=', 'Content-Length': '0'} 2023-11-02T18:32:45.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056dca8fd0643afd3-006543eb4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.129 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:32:45.130 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:32:45.131 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:32:45.131 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.132 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.132 DEBUG:boto:Method: GET 2023-11-02T18:32:45.133 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.133 DEBUG:boto:Data: 2023-11-02T18:32:45.134 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.135 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.135 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.136 DEBUG:boto:Params: {} 2023-11-02T18:32:45.136 DEBUG:boto:Token: None 2023-11-02T18:32:45.137 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.137 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:77WaLOar14EoadxvAXb+pHaiHBo= 2023-11-02T18:32:45.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:77WaLOar14EoadxvAXb+pHaiHBo=', 'Content-Length': '0'} 2023-11-02T18:32:45.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004da8d9b5da3866dd-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.182 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:32:45.183 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:32:45.183 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.184 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.184 DEBUG:boto:Method: GET 2023-11-02T18:32:45.185 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.185 DEBUG:boto:Data: 2023-11-02T18:32:45.186 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.186 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.187 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.187 DEBUG:boto:Params: {} 2023-11-02T18:32:45.188 DEBUG:boto:Token: None 2023-11-02T18:32:45.188 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.189 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:77WaLOar14EoadxvAXb+pHaiHBo= 2023-11-02T18:32:45.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:77WaLOar14EoadxvAXb+pHaiHBo=', 'Content-Length': '0'} 2023-11-02T18:32:45.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f099610408fe1b2-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.193 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:32:45.194 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:32:45.194 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.194 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.195 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.195 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.196 DEBUG:boto:Data: 2023-11-02T18:32:45.196 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.197 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.197 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.197 DEBUG:boto:Params: {} 2023-11-02T18:32:45.198 DEBUG:boto:Token: None 2023-11-02T18:32:45.199 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.199 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g4VHMiEPFsOTROBcdgC0EAt30b0= 2023-11-02T18:32:45.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g4VHMiEPFsOTROBcdgC0EAt30b0=', 'Content-Length': '0'} 2023-11-02T18:32:45.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000fd5c01ef519f98-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.203 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:32:45.204 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:32:45.204 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.204 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.205 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.205 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.206 DEBUG:boto:Data: 2023-11-02T18:32:45.206 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.206 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.207 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.207 DEBUG:boto:Params: {} 2023-11-02T18:32:45.208 DEBUG:boto:Token: None 2023-11-02T18:32:45.208 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.209 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g4VHMiEPFsOTROBcdgC0EAt30b0= 2023-11-02T18:32:45.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g4VHMiEPFsOTROBcdgC0EAt30b0=', 'Content-Length': '0'} 2023-11-02T18:32:45.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044d82d9f85d71bd1-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:32:45.213 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:32:45.213 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:32:45.214 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.214 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.215 DEBUG:boto:Method: GET 2023-11-02T18:32:45.215 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.216 DEBUG:boto:Data: 2023-11-02T18:32:45.216 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.217 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.217 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.218 DEBUG:boto:Params: {} 2023-11-02T18:32:45.218 DEBUG:boto:Token: None 2023-11-02T18:32:45.219 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.219 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:77WaLOar14EoadxvAXb+pHaiHBo= 2023-11-02T18:32:45.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:77WaLOar14EoadxvAXb+pHaiHBo=', 'Content-Length': '0'} 2023-11-02T18:32:45.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f98dddeb3b957ab0-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.223 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:32:45.224 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:32:45.224 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.225 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.225 DEBUG:boto:Method: GET 2023-11-02T18:32:45.226 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.226 DEBUG:boto:Data: 2023-11-02T18:32:45.226 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.227 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.227 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.228 DEBUG:boto:Params: {} 2023-11-02T18:32:45.228 DEBUG:boto:Token: None 2023-11-02T18:32:45.229 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:32:45.229 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:77WaLOar14EoadxvAXb+pHaiHBo= 2023-11-02T18:32:45.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:77WaLOar14EoadxvAXb+pHaiHBo=', 'Content-Length': '0'} 2023-11-02T18:32:45.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6702915e1758ece-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.274 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.275 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.276 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:32:45.276 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:32:45.277 DEBUG:boto:Method: GET 2023-11-02T18:32:45.277 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:32:45.278 DEBUG:boto:Data: 2023-11-02T18:32:45.278 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.279 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.279 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.280 DEBUG:boto:Params: {} 2023-11-02T18:32:45.280 DEBUG:boto:Token: None 2023-11-02T18:32:45.281 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.281 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU= 2023-11-02T18:32:45.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU=', 'Content-Length': '0'} 2023-11-02T18:32:45.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021051256a3da35ab-006543eb4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.287 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.288 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.289 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.289 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:32:45.290 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:32:45.290 DEBUG:boto:Method: GET 2023-11-02T18:32:45.291 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:32:45.291 DEBUG:boto:Data: 2023-11-02T18:32:45.292 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.293 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.294 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.294 DEBUG:boto:Params: {} 2023-11-02T18:32:45.295 DEBUG:boto:Token: None 2023-11-02T18:32:45.296 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.297 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU= 2023-11-02T18:32:45.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU=', 'Content-Length': '0'} 2023-11-02T18:32:45.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f174c12e8ab1ea0-006543eb4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.303 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.304 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:32:45.305 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:32:45.305 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.306 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:32:45.306 DEBUG:boto:Data: 2023-11-02T18:32:45.307 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.307 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.308 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.308 DEBUG:boto:Params: {} 2023-11-02T18:32:45.309 DEBUG:boto:Token: None 2023-11-02T18:32:45.309 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj11 2023-11-02T18:32:45.310 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ku8DrHL83zqKYQMb5jZbX0B9fAE= 2023-11-02T18:32:45.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ku8DrHL83zqKYQMb5jZbX0B9fAE=', 'Content-Length': '0'} 2023-11-02T18:32:45.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec4733283bc6060e-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.314 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:32:45.314 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:32:45.315 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.315 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:32:45.315 DEBUG:boto:Data: 2023-11-02T18:32:45.316 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.316 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.317 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.317 DEBUG:boto:Params: {} 2023-11-02T18:32:45.318 DEBUG:boto:Token: None 2023-11-02T18:32:45.318 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj11 2023-11-02T18:32:45.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ku8DrHL83zqKYQMb5jZbX0B9fAE= 2023-11-02T18:32:45.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ku8DrHL83zqKYQMb5jZbX0B9fAE=', 'Content-Length': '0'} 2023-11-02T18:32:45.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3c37dc03fdf44d9-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:32:45.324 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:32:45.325 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:32:45.325 DEBUG:boto:Method: GET 2023-11-02T18:32:45.326 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:32:45.326 DEBUG:boto:Data: 2023-11-02T18:32:45.327 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.328 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.329 DEBUG:boto:Params: {} 2023-11-02T18:32:45.330 DEBUG:boto:Token: None 2023-11-02T18:32:45.330 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj11 2023-11-02T18:32:45.331 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JaG9IWCfFzr1adNz5cQwbIVdDpY= 2023-11-02T18:32:45.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JaG9IWCfFzr1adNz5cQwbIVdDpY=', 'Content-Length': '0'} 2023-11-02T18:32:45.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfca354d29bd86f1-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.382 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:32:45.383 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:32:45.384 DEBUG:boto:Method: GET 2023-11-02T18:32:45.384 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:32:45.385 DEBUG:boto:Data: 2023-11-02T18:32:45.385 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.386 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.386 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.387 DEBUG:boto:Params: {} 2023-11-02T18:32:45.387 DEBUG:boto:Token: None 2023-11-02T18:32:45.388 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj11 2023-11-02T18:32:45.389 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JaG9IWCfFzr1adNz5cQwbIVdDpY= 2023-11-02T18:32:45.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JaG9IWCfFzr1adNz5cQwbIVdDpY=', 'Content-Length': '0'} 2023-11-02T18:32:45.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055dc8c66ad9aa971-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.394 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.394 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.395 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:32:45.395 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:32:45.396 DEBUG:boto:Method: GET 2023-11-02T18:32:45.396 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:32:45.397 DEBUG:boto:Data: 2023-11-02T18:32:45.397 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.398 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.399 DEBUG:boto:Params: {} 2023-11-02T18:32:45.399 DEBUG:boto:Token: None 2023-11-02T18:32:45.400 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU= 2023-11-02T18:32:45.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU=', 'Content-Length': '0'} 2023-11-02T18:32:45.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000290b23e49098433-006543eb4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.406 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.407 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.408 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.408 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:32:45.409 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:32:45.409 DEBUG:boto:Method: GET 2023-11-02T18:32:45.409 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:32:45.410 DEBUG:boto:Data: 2023-11-02T18:32:45.411 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.411 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.411 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.412 DEBUG:boto:Params: {} 2023-11-02T18:32:45.412 DEBUG:boto:Token: None 2023-11-02T18:32:45.413 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.413 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU= 2023-11-02T18:32:45.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU=', 'Content-Length': '0'} 2023-11-02T18:32:45.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054e4a10d0881926e-006543eb4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.420 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:32:45.421 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:32:45.421 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:32:45.422 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.422 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.422 DEBUG:boto:Method: GET 2023-11-02T18:32:45.423 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.424 DEBUG:boto:Data: 2023-11-02T18:32:45.424 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.424 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.425 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.426 DEBUG:boto:Params: {} 2023-11-02T18:32:45.426 DEBUG:boto:Token: None 2023-11-02T18:32:45.427 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.427 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IvTI0/+2rwic6jLimfqkWOy0d3Y= 2023-11-02T18:32:45.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IvTI0/+2rwic6jLimfqkWOy0d3Y=', 'Content-Length': '0'} 2023-11-02T18:32:45.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc9c07ca5d1ab548-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.432 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:32:45.432 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:32:45.433 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.433 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.434 DEBUG:boto:Method: GET 2023-11-02T18:32:45.434 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.435 DEBUG:boto:Data: 2023-11-02T18:32:45.435 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.436 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.436 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.436 DEBUG:boto:Params: {} 2023-11-02T18:32:45.437 DEBUG:boto:Token: None 2023-11-02T18:32:45.437 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.438 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IvTI0/+2rwic6jLimfqkWOy0d3Y= 2023-11-02T18:32:45.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IvTI0/+2rwic6jLimfqkWOy0d3Y=', 'Content-Length': '0'} 2023-11-02T18:32:45.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf43f78531520088-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.482 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:32:45.483 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:32:45.484 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.484 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.485 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.485 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.485 DEBUG:boto:Data: 2023-11-02T18:32:45.486 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.486 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.487 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.487 DEBUG:boto:Params: {} 2023-11-02T18:32:45.488 DEBUG:boto:Token: None 2023-11-02T18:32:45.488 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.489 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vIaKF+AugrovGs0RjjeFSfXRXp8= 2023-11-02T18:32:45.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vIaKF+AugrovGs0RjjeFSfXRXp8=', 'Content-Length': '0'} 2023-11-02T18:32:45.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e51484b237d92e90-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.494 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:32:45.495 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:32:45.495 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.496 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.497 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.497 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.498 DEBUG:boto:Data: 2023-11-02T18:32:45.498 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.499 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.499 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.500 DEBUG:boto:Params: {} 2023-11-02T18:32:45.500 DEBUG:boto:Token: None 2023-11-02T18:32:45.501 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.502 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vIaKF+AugrovGs0RjjeFSfXRXp8= 2023-11-02T18:32:45.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vIaKF+AugrovGs0RjjeFSfXRXp8=', 'Content-Length': '0'} 2023-11-02T18:32:45.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf6c18bca5eaaf2c-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:32:45.506 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:32:45.507 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:32:45.507 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.508 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.508 DEBUG:boto:Method: GET 2023-11-02T18:32:45.509 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.509 DEBUG:boto:Data: 2023-11-02T18:32:45.510 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.511 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.511 DEBUG:boto:Params: {} 2023-11-02T18:32:45.512 DEBUG:boto:Token: None 2023-11-02T18:32:45.512 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IvTI0/+2rwic6jLimfqkWOy0d3Y= 2023-11-02T18:32:45.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IvTI0/+2rwic6jLimfqkWOy0d3Y=', 'Content-Length': '0'} 2023-11-02T18:32:45.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0a2c35ed909df6d-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.517 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:32:45.517 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:32:45.518 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.518 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.521 DEBUG:boto:Method: GET 2023-11-02T18:32:45.522 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.522 DEBUG:boto:Data: 2023-11-02T18:32:45.523 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.523 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.523 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.524 DEBUG:boto:Params: {} 2023-11-02T18:32:45.524 DEBUG:boto:Token: None 2023-11-02T18:32:45.525 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:32:45.525 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IvTI0/+2rwic6jLimfqkWOy0d3Y= 2023-11-02T18:32:45.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IvTI0/+2rwic6jLimfqkWOy0d3Y=', 'Content-Length': '0'} 2023-11-02T18:32:45.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130bc4ed872cca56-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.529 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.530 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.530 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:32:45.531 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:32:45.531 DEBUG:boto:Method: GET 2023-11-02T18:32:45.532 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:32:45.532 DEBUG:boto:Data: 2023-11-02T18:32:45.533 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.533 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.533 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.534 DEBUG:boto:Params: {} 2023-11-02T18:32:45.534 DEBUG:boto:Token: None 2023-11-02T18:32:45.535 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.535 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU= 2023-11-02T18:32:45.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU=', 'Content-Length': '0'} 2023-11-02T18:32:45.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000258c7bd5bdfab606-006543eb4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.541 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.542 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.542 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.543 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:32:45.543 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:32:45.544 DEBUG:boto:Method: GET 2023-11-02T18:32:45.544 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:32:45.544 DEBUG:boto:Data: 2023-11-02T18:32:45.545 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.545 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.546 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.546 DEBUG:boto:Params: {} 2023-11-02T18:32:45.547 DEBUG:boto:Token: None 2023-11-02T18:32:45.547 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.548 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU= 2023-11-02T18:32:45.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU=', 'Content-Length': '0'} 2023-11-02T18:32:45.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff2d189fd6a6d621-006543eb4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.554 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.555 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:32:45.555 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:32:45.555 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.556 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:32:45.556 DEBUG:boto:Data: 2023-11-02T18:32:45.557 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.557 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.558 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.558 DEBUG:boto:Params: {} 2023-11-02T18:32:45.559 DEBUG:boto:Token: None 2023-11-02T18:32:45.559 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj22 2023-11-02T18:32:45.559 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5RUPm2/VSNbzgNAhkBzp3yPBRPc= 2023-11-02T18:32:45.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5RUPm2/VSNbzgNAhkBzp3yPBRPc=', 'Content-Length': '0'} 2023-11-02T18:32:45.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004874069cd0ed4fa6-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.563 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:32:45.564 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:32:45.564 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.565 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:32:45.565 DEBUG:boto:Data: 2023-11-02T18:32:45.566 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.566 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.568 DEBUG:boto:Params: {} 2023-11-02T18:32:45.568 DEBUG:boto:Token: None 2023-11-02T18:32:45.569 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj22 2023-11-02T18:32:45.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5RUPm2/VSNbzgNAhkBzp3yPBRPc= 2023-11-02T18:32:45.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5RUPm2/VSNbzgNAhkBzp3yPBRPc=', 'Content-Length': '0'} 2023-11-02T18:32:45.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004b69609cb8e81ba-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:32:45.573 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:32:45.574 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:32:45.574 DEBUG:boto:Method: GET 2023-11-02T18:32:45.575 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:32:45.575 DEBUG:boto:Data: 2023-11-02T18:32:45.576 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.576 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.577 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.577 DEBUG:boto:Params: {} 2023-11-02T18:32:45.577 DEBUG:boto:Token: None 2023-11-02T18:32:45.578 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj22 2023-11-02T18:32:45.578 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hnWXu+88ec7bf2qs3bCPncWoMjM= 2023-11-02T18:32:45.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hnWXu+88ec7bf2qs3bCPncWoMjM=', 'Content-Length': '0'} 2023-11-02T18:32:45.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087abf207348fe1ba-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.622 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:32:45.623 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:32:45.623 DEBUG:boto:Method: GET 2023-11-02T18:32:45.624 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:32:45.624 DEBUG:boto:Data: 2023-11-02T18:32:45.624 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.625 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.625 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.625 DEBUG:boto:Params: {} 2023-11-02T18:32:45.626 DEBUG:boto:Token: None 2023-11-02T18:32:45.626 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj22 2023-11-02T18:32:45.628 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hnWXu+88ec7bf2qs3bCPncWoMjM= 2023-11-02T18:32:45.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hnWXu+88ec7bf2qs3bCPncWoMjM=', 'Content-Length': '0'} 2023-11-02T18:32:45.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3273ca1dd6131c8-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.633 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.633 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.634 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:32:45.634 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:32:45.634 DEBUG:boto:Method: GET 2023-11-02T18:32:45.635 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:32:45.635 DEBUG:boto:Data: 2023-11-02T18:32:45.636 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.636 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.637 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.637 DEBUG:boto:Params: {} 2023-11-02T18:32:45.637 DEBUG:boto:Token: None 2023-11-02T18:32:45.638 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.638 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU= 2023-11-02T18:32:45.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU=', 'Content-Length': '0'} 2023-11-02T18:32:45.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062a4ba59cefb57cb-006543eb4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.644 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.645 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.645 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.646 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:32:45.646 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:32:45.646 DEBUG:boto:Method: GET 2023-11-02T18:32:45.647 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:32:45.647 DEBUG:boto:Data: 2023-11-02T18:32:45.648 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.648 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.648 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.649 DEBUG:boto:Params: {} 2023-11-02T18:32:45.649 DEBUG:boto:Token: None 2023-11-02T18:32:45.650 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.650 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU= 2023-11-02T18:32:45.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU=', 'Content-Length': '0'} 2023-11-02T18:32:45.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c043b4d2392dfd3-006543eb4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.698 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:32:45.700 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:32:45.700 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:32:45.701 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.701 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.702 DEBUG:boto:Method: GET 2023-11-02T18:32:45.702 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.702 DEBUG:boto:Data: 2023-11-02T18:32:45.703 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.703 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.704 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.704 DEBUG:boto:Params: {} 2023-11-02T18:32:45.705 DEBUG:boto:Token: None 2023-11-02T18:32:45.705 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.706 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IyXJiKunnKzsHF1HsDb5F4yUkFU= 2023-11-02T18:32:45.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IyXJiKunnKzsHF1HsDb5F4yUkFU=', 'Content-Length': '0'} 2023-11-02T18:32:45.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089640ac2e86f3dd6-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.710 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:32:45.710 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:32:45.711 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.711 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.712 DEBUG:boto:Method: GET 2023-11-02T18:32:45.712 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.712 DEBUG:boto:Data: 2023-11-02T18:32:45.713 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.713 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.714 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.714 DEBUG:boto:Params: {} 2023-11-02T18:32:45.715 DEBUG:boto:Token: None 2023-11-02T18:32:45.715 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.716 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IyXJiKunnKzsHF1HsDb5F4yUkFU= 2023-11-02T18:32:45.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IyXJiKunnKzsHF1HsDb5F4yUkFU=', 'Content-Length': '0'} 2023-11-02T18:32:45.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000856ac2e9b29c2863-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.720 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:32:45.720 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:32:45.721 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.721 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.721 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.722 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.722 DEBUG:boto:Data: 2023-11-02T18:32:45.723 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.723 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.725 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.725 DEBUG:boto:Params: {} 2023-11-02T18:32:45.726 DEBUG:boto:Token: None 2023-11-02T18:32:45.726 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.727 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CYnVJKMhrupX9qKuhyzNLLO+Au0= 2023-11-02T18:32:45.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CYnVJKMhrupX9qKuhyzNLLO+Au0=', 'Content-Length': '0'} 2023-11-02T18:32:45.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bebde4d74f013da2-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.731 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:32:45.731 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:32:45.731 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.732 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.732 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.733 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.733 DEBUG:boto:Data: 2023-11-02T18:32:45.733 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.734 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.734 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.735 DEBUG:boto:Params: {} 2023-11-02T18:32:45.735 DEBUG:boto:Token: None 2023-11-02T18:32:45.735 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.736 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CYnVJKMhrupX9qKuhyzNLLO+Au0= 2023-11-02T18:32:45.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CYnVJKMhrupX9qKuhyzNLLO+Au0=', 'Content-Length': '0'} 2023-11-02T18:32:45.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c53a21e1c54e8e-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:32:45.740 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:32:45.741 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:32:45.741 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.741 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.742 DEBUG:boto:Method: GET 2023-11-02T18:32:45.742 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.743 DEBUG:boto:Data: 2023-11-02T18:32:45.743 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.743 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.744 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.744 DEBUG:boto:Params: {} 2023-11-02T18:32:45.745 DEBUG:boto:Token: None 2023-11-02T18:32:45.745 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.746 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IyXJiKunnKzsHF1HsDb5F4yUkFU= 2023-11-02T18:32:45.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IyXJiKunnKzsHF1HsDb5F4yUkFU=', 'Content-Length': '0'} 2023-11-02T18:32:45.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037b945e76a5467df-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.750 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:32:45.750 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:32:45.751 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.751 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.752 DEBUG:boto:Method: GET 2023-11-02T18:32:45.752 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.753 DEBUG:boto:Data: 2023-11-02T18:32:45.753 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.753 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.754 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.754 DEBUG:boto:Params: {} 2023-11-02T18:32:45.755 DEBUG:boto:Token: None 2023-11-02T18:32:45.755 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:32:45.756 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IyXJiKunnKzsHF1HsDb5F4yUkFU= 2023-11-02T18:32:45.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IyXJiKunnKzsHF1HsDb5F4yUkFU=', 'Content-Length': '0'} 2023-11-02T18:32:45.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b34014099ac96c4c-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.802 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.803 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.804 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:32:45.804 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:32:45.804 DEBUG:boto:Method: GET 2023-11-02T18:32:45.805 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:32:45.805 DEBUG:boto:Data: 2023-11-02T18:32:45.806 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.807 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.807 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.808 DEBUG:boto:Params: {} 2023-11-02T18:32:45.808 DEBUG:boto:Token: None 2023-11-02T18:32:45.809 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.810 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU= 2023-11-02T18:32:45.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU=', 'Content-Length': '0'} 2023-11-02T18:32:45.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db938b2af8e0ad25-006543eb4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.816 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.817 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.817 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.818 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:32:45.818 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:32:45.819 DEBUG:boto:Method: GET 2023-11-02T18:32:45.819 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:32:45.820 DEBUG:boto:Data: 2023-11-02T18:32:45.820 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.820 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.821 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.821 DEBUG:boto:Params: {} 2023-11-02T18:32:45.822 DEBUG:boto:Token: None 2023-11-02T18:32:45.822 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.823 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU= 2023-11-02T18:32:45.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU=', 'Content-Length': '0'} 2023-11-02T18:32:45.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095046bb2d6661f23-006543eb4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.829 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.829 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:32:45.830 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:32:45.830 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.831 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:32:45.831 DEBUG:boto:Data: 2023-11-02T18:32:45.832 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.832 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.833 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.833 DEBUG:boto:Params: {} 2023-11-02T18:32:45.834 DEBUG:boto:Token: None 2023-11-02T18:32:45.834 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj33 2023-11-02T18:32:45.835 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gyDCiB5q7oaX786odF4Y6eyDTds= 2023-11-02T18:32:45.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gyDCiB5q7oaX786odF4Y6eyDTds=', 'Content-Length': '0'} 2023-11-02T18:32:45.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef5941285c2856da-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.839 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:32:45.839 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:32:45.840 DEBUG:boto:Method: HEAD 2023-11-02T18:32:45.840 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:32:45.841 DEBUG:boto:Data: 2023-11-02T18:32:45.841 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.842 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.842 DEBUG:boto:Params: {} 2023-11-02T18:32:45.843 DEBUG:boto:Token: None 2023-11-02T18:32:45.843 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj33 2023-11-02T18:32:45.844 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gyDCiB5q7oaX786odF4Y6eyDTds= 2023-11-02T18:32:45.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gyDCiB5q7oaX786odF4Y6eyDTds=', 'Content-Length': '0'} 2023-11-02T18:32:45.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040caf5999659d3ed-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:32:45.848 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:32:45.849 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:32:45.849 DEBUG:boto:Method: GET 2023-11-02T18:32:45.850 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:32:45.850 DEBUG:boto:Data: 2023-11-02T18:32:45.850 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.851 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.851 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.852 DEBUG:boto:Params: {} 2023-11-02T18:32:45.852 DEBUG:boto:Token: None 2023-11-02T18:32:45.853 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj33 2023-11-02T18:32:45.853 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Das35ufeJD/6qBHbxNTrDtao1k0= 2023-11-02T18:32:45.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Das35ufeJD/6qBHbxNTrDtao1k0=', 'Content-Length': '0'} 2023-11-02T18:32:45.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfd2ea8ca10169d5-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.858 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:32:45.858 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:32:45.859 DEBUG:boto:Method: GET 2023-11-02T18:32:45.859 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:32:45.860 DEBUG:boto:Data: 2023-11-02T18:32:45.860 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.861 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.861 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.862 DEBUG:boto:Params: {} 2023-11-02T18:32:45.862 DEBUG:boto:Token: None 2023-11-02T18:32:45.863 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj33 2023-11-02T18:32:45.863 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Das35ufeJD/6qBHbxNTrDtao1k0= 2023-11-02T18:32:45.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Das35ufeJD/6qBHbxNTrDtao1k0=', 'Content-Length': '0'} 2023-11-02T18:32:45.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afd71bec677ad96b-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.910 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.911 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.911 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:32:45.912 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:32:45.912 DEBUG:boto:Method: GET 2023-11-02T18:32:45.913 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:32:45.913 DEBUG:boto:Data: 2023-11-02T18:32:45.914 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.914 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.915 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.915 DEBUG:boto:Params: {} 2023-11-02T18:32:45.916 DEBUG:boto:Token: None 2023-11-02T18:32:45.916 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.917 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU= 2023-11-02T18:32:45.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU=', 'Content-Length': '0'} 2023-11-02T18:32:45.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d3ba38bd7ef4049-006543eb4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.923 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.924 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:45.924 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:45.925 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:32:45.925 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:32:45.926 DEBUG:boto:Method: GET 2023-11-02T18:32:45.926 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:32:45.927 DEBUG:boto:Data: 2023-11-02T18:32:45.927 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.929 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.929 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.930 DEBUG:boto:Params: {} 2023-11-02T18:32:45.930 DEBUG:boto:Token: None 2023-11-02T18:32:45.931 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/ 2023-11-02T18:32:45.931 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU= 2023-11-02T18:32:45.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4MEnjr4BUH/s3ixW8khUNumA8kU=', 'Content-Length': '0'} 2023-11-02T18:32:45.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060c1a1b80dbb92ad-006543eb4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.938 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:45.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:32:45.939 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:32:45.939 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:32:45.940 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:32:45.940 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:32:45.941 DEBUG:boto:Method: GET 2023-11-02T18:32:45.941 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:32:45.942 DEBUG:boto:Data: 2023-11-02T18:32:45.942 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.943 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:45.943 DEBUG:boto:Port: 8001 2023-11-02T18:32:45.944 DEBUG:boto:Params: {} 2023-11-02T18:32:45.944 DEBUG:boto:Token: None 2023-11-02T18:32:45.945 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:32:45.945 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tXxLhgzmSv60MOoJIR9XeVnkcMc= 2023-11-02T18:32:45.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tXxLhgzmSv60MOoJIR9XeVnkcMc=', 'Content-Length': '0'} 2023-11-02T18:32:45.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cdd3a0a0ca78c63-006543eb4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:45.990 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:32:45.991 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:32:45.991 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:32:45.992 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:32:45.992 DEBUG:boto:Method: GET 2023-11-02T18:32:45.993 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:32:45.993 DEBUG:boto:Data: 2023-11-02T18:32:45.994 DEBUG:boto:Headers: {} 2023-11-02T18:32:45.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:45.994 DEBUG:boto:Port: 8000 2023-11-02T18:32:45.995 DEBUG:boto:Params: {} 2023-11-02T18:32:45.996 DEBUG:boto:Token: None 2023-11-02T18:32:45.996 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:45 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:32:45.997 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tXxLhgzmSv60MOoJIR9XeVnkcMc= 2023-11-02T18:32:45.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tXxLhgzmSv60MOoJIR9XeVnkcMc=', 'Content-Length': '0'} 2023-11-02T18:32:46.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0564002fbb38077-006543eb4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.001 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:32:46.002 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:32:46.002 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.003 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.003 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.004 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.004 DEBUG:boto:Data: 2023-11-02T18:32:46.004 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.005 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.005 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.006 DEBUG:boto:Params: {} 2023-11-02T18:32:46.006 DEBUG:boto:Token: None 2023-11-02T18:32:46.007 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.007 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uAvq+v6Is18rh3TiYj3z4HsfGv4= 2023-11-02T18:32:46.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uAvq+v6Is18rh3TiYj3z4HsfGv4=', 'Content-Length': '0'} 2023-11-02T18:32:46.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083951d51571d3eec-006543eb4e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.011 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:32:46.012 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:32:46.012 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.012 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.013 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.013 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.014 DEBUG:boto:Data: 2023-11-02T18:32:46.014 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.015 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.015 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.016 DEBUG:boto:Params: {} 2023-11-02T18:32:46.016 DEBUG:boto:Token: None 2023-11-02T18:32:46.017 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.017 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uAvq+v6Is18rh3TiYj3z4HsfGv4= 2023-11-02T18:32:46.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uAvq+v6Is18rh3TiYj3z4HsfGv4=', 'Content-Length': '0'} 2023-11-02T18:32:46.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058f839db3d5d6dce-006543eb4e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:32:46.022 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:32:46.022 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:32:46.023 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.023 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.024 DEBUG:boto:Method: GET 2023-11-02T18:32:46.024 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.025 DEBUG:boto:Data: 2023-11-02T18:32:46.025 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.026 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.026 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.027 DEBUG:boto:Params: {} 2023-11-02T18:32:46.027 DEBUG:boto:Token: None 2023-11-02T18:32:46.028 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.029 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nOcL6wW1cS/pSa9wtkz0QGEJYmU= 2023-11-02T18:32:46.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nOcL6wW1cS/pSa9wtkz0QGEJYmU=', 'Content-Length': '0'} 2023-11-02T18:32:46.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ce2a964363371c5-006543eb4e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.036 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:32:46.036 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:32:46.037 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.037 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.038 DEBUG:boto:Method: GET 2023-11-02T18:32:46.038 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.038 DEBUG:boto:Data: 2023-11-02T18:32:46.039 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.039 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.040 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.040 DEBUG:boto:Params: {} 2023-11-02T18:32:46.041 DEBUG:boto:Token: None 2023-11-02T18:32:46.041 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:32:46.042 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nOcL6wW1cS/pSa9wtkz0QGEJYmU= 2023-11-02T18:32:46.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nOcL6wW1cS/pSa9wtkz0QGEJYmU=', 'Content-Length': '0'} 2023-11-02T18:32:46.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008605f4aac9ee4dcc-006543eb4e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.046 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.047 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.047 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:32:46.048 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:32:46.048 DEBUG:boto:Method: GET 2023-11-02T18:32:46.049 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:32:46.049 DEBUG:boto:Data: 2023-11-02T18:32:46.049 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.050 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.051 DEBUG:boto:Params: {} 2023-11-02T18:32:46.051 DEBUG:boto:Token: None 2023-11-02T18:32:46.052 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.052 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE= 2023-11-02T18:32:46.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE=', 'Content-Length': '0'} 2023-11-02T18:32:46.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005380520acd163e5-006543eb4e-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.098 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:46.099 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.100 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.100 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:32:46.101 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:32:46.101 DEBUG:boto:Method: GET 2023-11-02T18:32:46.102 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:32:46.102 DEBUG:boto:Data: 2023-11-02T18:32:46.103 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.103 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.104 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.104 DEBUG:boto:Params: {} 2023-11-02T18:32:46.105 DEBUG:boto:Token: None 2023-11-02T18:32:46.105 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.106 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE= 2023-11-02T18:32:46.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE=', 'Content-Length': '0'} 2023-11-02T18:32:46.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a019a7efe85ccb23-006543eb4e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.112 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:46.113 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:32:46.113 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:32:46.114 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.114 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:32:46.115 DEBUG:boto:Data: 2023-11-02T18:32:46.115 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.115 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.116 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.116 DEBUG:boto:Params: {} 2023-11-02T18:32:46.117 DEBUG:boto:Token: None 2023-11-02T18:32:46.117 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj44 2023-11-02T18:32:46.118 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mOe5qT80ndWq/I2K8K5ulvVc6qY= 2023-11-02T18:32:46.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mOe5qT80ndWq/I2K8K5ulvVc6qY=', 'Content-Length': '0'} 2023-11-02T18:32:46.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc61903c61159d4-006543eb4e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.122 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:32:46.123 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:32:46.123 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.124 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:32:46.124 DEBUG:boto:Data: 2023-11-02T18:32:46.125 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.125 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.126 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.126 DEBUG:boto:Params: {} 2023-11-02T18:32:46.127 DEBUG:boto:Token: None 2023-11-02T18:32:46.127 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj44 2023-11-02T18:32:46.128 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mOe5qT80ndWq/I2K8K5ulvVc6qY= 2023-11-02T18:32:46.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mOe5qT80ndWq/I2K8K5ulvVc6qY=', 'Content-Length': '0'} 2023-11-02T18:32:46.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c9cf0ee34a92672-006543eb4e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:32:46.134 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:32:46.134 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:32:46.135 DEBUG:boto:Method: GET 2023-11-02T18:32:46.135 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:32:46.136 DEBUG:boto:Data: 2023-11-02T18:32:46.136 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.136 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.137 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.137 DEBUG:boto:Params: {} 2023-11-02T18:32:46.138 DEBUG:boto:Token: None 2023-11-02T18:32:46.138 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj44 2023-11-02T18:32:46.139 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rL4WM0tx6cNjZ/caw7X/aMXtLqw= 2023-11-02T18:32:46.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rL4WM0tx6cNjZ/caw7X/aMXtLqw=', 'Content-Length': '0'} 2023-11-02T18:32:46.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000509a8228dd45f420-006543eb4e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.143 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:32:46.144 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:32:46.144 DEBUG:boto:Method: GET 2023-11-02T18:32:46.145 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:32:46.145 DEBUG:boto:Data: 2023-11-02T18:32:46.145 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.146 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.146 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.147 DEBUG:boto:Params: {} 2023-11-02T18:32:46.148 DEBUG:boto:Token: None 2023-11-02T18:32:46.148 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj44 2023-11-02T18:32:46.149 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rL4WM0tx6cNjZ/caw7X/aMXtLqw= 2023-11-02T18:32:46.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rL4WM0tx6cNjZ/caw7X/aMXtLqw=', 'Content-Length': '0'} 2023-11-02T18:32:46.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c8252178daf78d6-006543eb4e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.194 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.195 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.196 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:32:46.196 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:32:46.197 DEBUG:boto:Method: GET 2023-11-02T18:32:46.197 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:32:46.197 DEBUG:boto:Data: 2023-11-02T18:32:46.198 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.198 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.200 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.200 DEBUG:boto:Params: {} 2023-11-02T18:32:46.201 DEBUG:boto:Token: None 2023-11-02T18:32:46.201 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.202 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE= 2023-11-02T18:32:46.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE=', 'Content-Length': '0'} 2023-11-02T18:32:46.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005763a996c298aae2-006543eb4e-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.207 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:46.208 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.209 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.209 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:32:46.210 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:32:46.210 DEBUG:boto:Method: GET 2023-11-02T18:32:46.211 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:32:46.211 DEBUG:boto:Data: 2023-11-02T18:32:46.211 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.212 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.212 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.213 DEBUG:boto:Params: {} 2023-11-02T18:32:46.213 DEBUG:boto:Token: None 2023-11-02T18:32:46.214 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.214 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE= 2023-11-02T18:32:46.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE=', 'Content-Length': '0'} 2023-11-02T18:32:46.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005601fc2b609e85e4-006543eb4e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.221 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:46.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:32:46.222 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:32:46.222 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:32:46.223 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.223 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.224 DEBUG:boto:Method: GET 2023-11-02T18:32:46.224 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.225 DEBUG:boto:Data: 2023-11-02T18:32:46.225 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.225 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.226 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.226 DEBUG:boto:Params: {} 2023-11-02T18:32:46.227 DEBUG:boto:Token: None 2023-11-02T18:32:46.227 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.228 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qKGiwu4QwLfQCRjdqLjAJxGns4A= 2023-11-02T18:32:46.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qKGiwu4QwLfQCRjdqLjAJxGns4A=', 'Content-Length': '0'} 2023-11-02T18:32:46.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a3e20d760c4215c-006543eb4e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.232 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:32:46.232 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:32:46.233 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.233 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.234 DEBUG:boto:Method: GET 2023-11-02T18:32:46.234 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.234 DEBUG:boto:Data: 2023-11-02T18:32:46.235 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.235 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.236 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.236 DEBUG:boto:Params: {} 2023-11-02T18:32:46.237 DEBUG:boto:Token: None 2023-11-02T18:32:46.237 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.238 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qKGiwu4QwLfQCRjdqLjAJxGns4A= 2023-11-02T18:32:46.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qKGiwu4QwLfQCRjdqLjAJxGns4A=', 'Content-Length': '0'} 2023-11-02T18:32:46.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5a57df5914ec728-006543eb4e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.242 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:32:46.242 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:32:46.243 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.243 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.244 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.244 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.245 DEBUG:boto:Data: 2023-11-02T18:32:46.245 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.245 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.246 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.246 DEBUG:boto:Params: {} 2023-11-02T18:32:46.247 DEBUG:boto:Token: None 2023-11-02T18:32:46.247 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.248 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XObQjNe42VrPtgPHEv0vLmkuyLI= 2023-11-02T18:32:46.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XObQjNe42VrPtgPHEv0vLmkuyLI=', 'Content-Length': '0'} 2023-11-02T18:32:46.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057832eb5ecd402f8-006543eb4e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.252 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:32:46.252 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:32:46.253 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.253 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.253 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.254 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.254 DEBUG:boto:Data: 2023-11-02T18:32:46.255 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.255 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.255 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.256 DEBUG:boto:Params: {} 2023-11-02T18:32:46.256 DEBUG:boto:Token: None 2023-11-02T18:32:46.257 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.257 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XObQjNe42VrPtgPHEv0vLmkuyLI= 2023-11-02T18:32:46.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XObQjNe42VrPtgPHEv0vLmkuyLI=', 'Content-Length': '0'} 2023-11-02T18:32:46.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcf5f8853d499a3a-006543eb4e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:32:46.262 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:32:46.262 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:32:46.263 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.263 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.264 DEBUG:boto:Method: GET 2023-11-02T18:32:46.264 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.264 DEBUG:boto:Data: 2023-11-02T18:32:46.265 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.265 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.266 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.266 DEBUG:boto:Params: {} 2023-11-02T18:32:46.267 DEBUG:boto:Token: None 2023-11-02T18:32:46.267 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.268 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qKGiwu4QwLfQCRjdqLjAJxGns4A= 2023-11-02T18:32:46.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qKGiwu4QwLfQCRjdqLjAJxGns4A=', 'Content-Length': '0'} 2023-11-02T18:32:46.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fde66ca84e60735-006543eb4e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.318 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:32:46.319 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:32:46.319 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.320 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.320 DEBUG:boto:Method: GET 2023-11-02T18:32:46.321 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.321 DEBUG:boto:Data: 2023-11-02T18:32:46.322 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.322 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.322 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.323 DEBUG:boto:Params: {} 2023-11-02T18:32:46.324 DEBUG:boto:Token: None 2023-11-02T18:32:46.324 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:32:46.325 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qKGiwu4QwLfQCRjdqLjAJxGns4A= 2023-11-02T18:32:46.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qKGiwu4QwLfQCRjdqLjAJxGns4A=', 'Content-Length': '0'} 2023-11-02T18:32:46.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d42f191d6f83bed-006543eb4e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.329 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.329 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.330 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:32:46.330 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:32:46.331 DEBUG:boto:Method: GET 2023-11-02T18:32:46.331 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:32:46.332 DEBUG:boto:Data: 2023-11-02T18:32:46.332 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.332 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.333 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.333 DEBUG:boto:Params: {} 2023-11-02T18:32:46.334 DEBUG:boto:Token: None 2023-11-02T18:32:46.334 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.337 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE= 2023-11-02T18:32:46.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE=', 'Content-Length': '0'} 2023-11-02T18:32:46.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c88433d29cbf1083-006543eb4e-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.344 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:46.344 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.345 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.345 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:32:46.346 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:32:46.346 DEBUG:boto:Method: GET 2023-11-02T18:32:46.347 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:32:46.347 DEBUG:boto:Data: 2023-11-02T18:32:46.348 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.348 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.348 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.349 DEBUG:boto:Params: {} 2023-11-02T18:32:46.349 DEBUG:boto:Token: None 2023-11-02T18:32:46.350 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.351 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE= 2023-11-02T18:32:46.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE=', 'Content-Length': '0'} 2023-11-02T18:32:46.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c84f3dc1f3857e08-006543eb4e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.398 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:46.399 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:32:46.400 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:32:46.400 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.401 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:32:46.401 DEBUG:boto:Data: 2023-11-02T18:32:46.402 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.402 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.403 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.403 DEBUG:boto:Params: {} 2023-11-02T18:32:46.403 DEBUG:boto:Token: None 2023-11-02T18:32:46.404 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj55 2023-11-02T18:32:46.404 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YLOMeu4Wj3MLpvkUbiP3QmLbpfw= 2023-11-02T18:32:46.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YLOMeu4Wj3MLpvkUbiP3QmLbpfw=', 'Content-Length': '0'} 2023-11-02T18:32:46.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e87ce4cd1d470d1-006543eb4e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.409 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:32:46.409 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:32:46.410 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.410 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:32:46.410 DEBUG:boto:Data: 2023-11-02T18:32:46.411 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.411 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.412 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.412 DEBUG:boto:Params: {} 2023-11-02T18:32:46.413 DEBUG:boto:Token: None 2023-11-02T18:32:46.413 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj55 2023-11-02T18:32:46.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YLOMeu4Wj3MLpvkUbiP3QmLbpfw= 2023-11-02T18:32:46.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YLOMeu4Wj3MLpvkUbiP3QmLbpfw=', 'Content-Length': '0'} 2023-11-02T18:32:46.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de1299c13611eab6-006543eb4e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:32:46.419 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:32:46.419 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:32:46.420 DEBUG:boto:Method: GET 2023-11-02T18:32:46.420 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:32:46.420 DEBUG:boto:Data: 2023-11-02T18:32:46.421 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.421 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.422 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.422 DEBUG:boto:Params: {} 2023-11-02T18:32:46.423 DEBUG:boto:Token: None 2023-11-02T18:32:46.423 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj55 2023-11-02T18:32:46.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QwJICRlBfSsbUGtCQ7b+vN1ILWg= 2023-11-02T18:32:46.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QwJICRlBfSsbUGtCQ7b+vN1ILWg=', 'Content-Length': '0'} 2023-11-02T18:32:46.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad723fb999b1a7f-006543eb4e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.428 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:32:46.428 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:32:46.429 DEBUG:boto:Method: GET 2023-11-02T18:32:46.429 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:32:46.430 DEBUG:boto:Data: 2023-11-02T18:32:46.430 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.431 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.431 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.432 DEBUG:boto:Params: {} 2023-11-02T18:32:46.432 DEBUG:boto:Token: None 2023-11-02T18:32:46.433 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj55 2023-11-02T18:32:46.433 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QwJICRlBfSsbUGtCQ7b+vN1ILWg= 2023-11-02T18:32:46.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QwJICRlBfSsbUGtCQ7b+vN1ILWg=', 'Content-Length': '0'} 2023-11-02T18:32:46.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bbdc2b0ff7d350b-006543eb4e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.438 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.438 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.439 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:32:46.439 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:32:46.440 DEBUG:boto:Method: GET 2023-11-02T18:32:46.440 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:32:46.441 DEBUG:boto:Data: 2023-11-02T18:32:46.441 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.442 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:32:46.442 DEBUG:boto:Port: 8001 2023-11-02T18:32:46.443 DEBUG:boto:Params: {} 2023-11-02T18:32:46.443 DEBUG:boto:Token: None 2023-11-02T18:32:46.444 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE= 2023-11-02T18:32:46.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE=', 'Content-Length': '0'} 2023-11-02T18:32:46.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b036b69cef45e252-006543eb4e-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.494 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:46.495 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.496 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.496 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:32:46.497 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:32:46.497 DEBUG:boto:Method: GET 2023-11-02T18:32:46.498 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:32:46.498 DEBUG:boto:Data: 2023-11-02T18:32:46.499 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.499 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.500 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.500 DEBUG:boto:Params: {} 2023-11-02T18:32:46.501 DEBUG:boto:Token: None 2023-11-02T18:32:46.501 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.502 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE= 2023-11-02T18:32:46.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dkfAjh9fhEuAXdj7rZL03WhLDUE=', 'Content-Length': '0'} 2023-11-02T18:32:46.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058eed29eead1a0c7-006543eb4e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.508 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:32:46.509 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:32:46.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:32:46.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:32:46.510 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:32:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:32:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:32:46.636 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:32:46.637 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-59 2023-11-02T18:32:46.645 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 5 2023-11-02T18:32:46.720 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.720 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.721 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.721 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:32:46.722 DEBUG:boto:Data: 2023-11-02T18:32:46.722 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.723 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.723 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.724 DEBUG:boto:Params: {} 2023-11-02T18:32:46.724 DEBUG:boto:Token: None 2023-11-02T18:32:46.725 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.725 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3yOqV5xuWRBJuoB7sYGeSbCGZys= 2023-11-02T18:32:46.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3yOqV5xuWRBJuoB7sYGeSbCGZys=', 'Content-Length': '0'} 2023-11-02T18:32:46.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000502e1afcc5249cf9-006543eb4e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.732 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:32:46.732 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:32:46.733 DEBUG:boto:Method: PUT 2023-11-02T18:32:46.733 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:32:46.734 DEBUG:boto:Data: 2023-11-02T18:32:46.734 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:32:46.735 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.735 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.736 DEBUG:boto:Params: {} 2023-11-02T18:32:46.736 DEBUG:boto:Token: None 2023-11-02T18:32:46.737 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj17 2023-11-02T18:32:46.737 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KG2eWm3fRd0YRayu2f9YgOr9RVQ= 2023-11-02T18:32:46.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KG2eWm3fRd0YRayu2f9YgOr9RVQ='} 2023-11-02T18:32:46.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a088577a2c4c21b-006543eb4e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.750 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.751 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.751 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.752 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:32:46.752 DEBUG:boto:Data: 2023-11-02T18:32:46.753 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.753 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.754 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.754 DEBUG:boto:Params: {} 2023-11-02T18:32:46.755 DEBUG:boto:Token: None 2023-11-02T18:32:46.755 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.756 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3yOqV5xuWRBJuoB7sYGeSbCGZys= 2023-11-02T18:32:46.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3yOqV5xuWRBJuoB7sYGeSbCGZys=', 'Content-Length': '0'} 2023-11-02T18:32:46.762 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1ffc7e1173b51ee-006543eb4e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.763 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:32:46.763 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:32:46.764 DEBUG:boto:Method: PUT 2023-11-02T18:32:46.764 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:32:46.764 DEBUG:boto:Data: 2023-11-02T18:32:46.765 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:32:46.765 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.766 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.766 DEBUG:boto:Params: {} 2023-11-02T18:32:46.767 DEBUG:boto:Token: None 2023-11-02T18:32:46.767 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj34 2023-11-02T18:32:46.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yZslI38d8fL/EGMylyCzb0WQ1bI= 2023-11-02T18:32:46.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yZslI38d8fL/EGMylyCzb0WQ1bI='} 2023-11-02T18:32:46.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3704d0a15e5cc42-006543eb4e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.811 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.811 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.812 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.812 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:32:46.813 DEBUG:boto:Data: 2023-11-02T18:32:46.813 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.814 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.814 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.815 DEBUG:boto:Params: {} 2023-11-02T18:32:46.815 DEBUG:boto:Token: None 2023-11-02T18:32:46.816 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.816 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3yOqV5xuWRBJuoB7sYGeSbCGZys= 2023-11-02T18:32:46.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3yOqV5xuWRBJuoB7sYGeSbCGZys=', 'Content-Length': '0'} 2023-11-02T18:32:46.822 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8e5650e35081a7a-006543eb4e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.823 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:32:46.823 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:32:46.824 DEBUG:boto:Method: PUT 2023-11-02T18:32:46.824 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:32:46.825 DEBUG:boto:Data: 2023-11-02T18:32:46.825 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:32:46.826 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.826 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.827 DEBUG:boto:Params: {} 2023-11-02T18:32:46.827 DEBUG:boto:Token: None 2023-11-02T18:32:46.828 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj51 2023-11-02T18:32:46.828 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GSm3SMcmpG78dBNCJcLRe8JLYm8= 2023-11-02T18:32:46.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GSm3SMcmpG78dBNCJcLRe8JLYm8='} 2023-11-02T18:32:46.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e84510b87b8f26c-006543eb4e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.842 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.842 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.843 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.843 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:32:46.844 DEBUG:boto:Data: 2023-11-02T18:32:46.844 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.845 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.845 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.846 DEBUG:boto:Params: {} 2023-11-02T18:32:46.846 DEBUG:boto:Token: None 2023-11-02T18:32:46.847 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.847 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3yOqV5xuWRBJuoB7sYGeSbCGZys= 2023-11-02T18:32:46.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3yOqV5xuWRBJuoB7sYGeSbCGZys=', 'Content-Length': '0'} 2023-11-02T18:32:46.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c6183bd7a9b6238-006543eb4e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.852 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:32:46.853 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:32:46.853 DEBUG:boto:Method: PUT 2023-11-02T18:32:46.854 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:32:46.854 DEBUG:boto:Data: 2023-11-02T18:32:46.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:32:46.855 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.856 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.856 DEBUG:boto:Params: {} 2023-11-02T18:32:46.857 DEBUG:boto:Token: None 2023-11-02T18:32:46.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj68 2023-11-02T18:32:46.858 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1VY61hZR2qkGn2FQOuB2bH+zRjQ= 2023-11-02T18:32:46.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1VY61hZR2qkGn2FQOuB2bH+zRjQ='} 2023-11-02T18:32:46.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051aea1bcf93fd203-006543eb4e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.903 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:32:46.904 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:32:46.904 DEBUG:boto:Method: HEAD 2023-11-02T18:32:46.904 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:32:46.905 DEBUG:boto:Data: 2023-11-02T18:32:46.905 DEBUG:boto:Headers: {} 2023-11-02T18:32:46.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.906 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.906 DEBUG:boto:Params: {} 2023-11-02T18:32:46.907 DEBUG:boto:Token: None 2023-11-02T18:32:46.907 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/ 2023-11-02T18:32:46.908 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3yOqV5xuWRBJuoB7sYGeSbCGZys= 2023-11-02T18:32:46.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3yOqV5xuWRBJuoB7sYGeSbCGZys=', 'Content-Length': '0'} 2023-11-02T18:32:46.913 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000607086f4f597cb56-006543eb4e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.914 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:32:46.914 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:32:46.915 DEBUG:boto:Method: PUT 2023-11-02T18:32:46.915 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:32:46.916 DEBUG:boto:Data: 2023-11-02T18:32:46.916 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:32:46.916 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:32:46.917 DEBUG:boto:Port: 8000 2023-11-02T18:32:46.917 DEBUG:boto:Params: {} 2023-11-02T18:32:46.918 DEBUG:boto:Token: None 2023-11-02T18:32:46.918 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:32:46 GMT /ibtljn-59/obj85 2023-11-02T18:32:46.919 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HzCv7iI8osGzmCFkbgb2eK6HWJM= 2023-11-02T18:32:46.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:32:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HzCv7iI8osGzmCFkbgb2eK6HWJM='} 2023-11-02T18:32:46.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000823fe290d769c069-006543eb4e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:32:46.932 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-59 2023-11-02T18:32:46.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:32:46.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:32:46.933 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:32:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:32:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:32:47.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:47.050+0000 7f7325c02f40 1 waiting to reach latest gen 1, on 0.. 2023-11-02T18:32:52.057 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:52.050+0000 7f7325c02f40 1 waiting to reach latest gen 1, on 0.. 2023-11-02T18:32:57.059 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:32:57.053+0000 7f7325c02f40 1 waiting to reach latest gen 1, on 0.. 2023-11-02T18:33:02.060 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:33:02.053+0000 7f7325c02f40 1 waiting to reach latest gen 1, on 0.. 2023-11-02T18:33:07.063 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:33:07.057+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:33:07.064 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-11-02T18:33:07.064 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:12.066 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:33:12.061+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:33:12.067 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-11-02T18:33:12.067 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:17.069 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:33:17.061+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:33:17.070 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:17.070 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:22.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:33:22.065+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:33:22.073 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:22.073 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:27.076 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:33:27.069+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:33:27.076 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:27.076 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:32.079 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:33:32.073+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:33:32.079 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:32.080 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:37.082 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:33:37.077+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:33:37.083 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:37.083 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:42.085 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:33:42.077+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:33:42.086 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:42.086 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:47.088 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:33:47.081+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:33:47.089 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:47.089 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:52.092 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:33:52.085+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:33:52.092 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:52.092 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:33:57.089+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:33:57.095 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:33:57.095 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:02.098 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:02.093+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:34:02.099 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:02.099 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:07.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:07.092+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:34:07.101 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:07.102 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:12.096+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:34:12.105 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:12.105 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:17.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:17.100+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:34:17.108 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:17.108 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:22.111 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:22.104+0000 7f7325c02f40 1 waiting for incremental sync to catch up: 2023-11-02T18:34:22.111 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , , , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:22.112 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:27.114 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:27.104+0000 7f7325c02f40 1 bucket sync caught up with source: 2023-11-02T18:34:27.115 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:27.115 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000001.81.6, , , , , , 00000000001.45.6, , 00000000001.59.6, , , , 00000000001.51.6, 00000000001.42.6, , , ] 2023-11-02T18:34:27.116 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:27.104+0000 7f7325c02f40 0 bucket checkpoint complete 2023-11-02T18:34:27.144 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-59 2023-11-02T18:34:27.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:34:27.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:34:27.147 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:34:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:34:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:34:27.270 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:27.264+0000 7fb07319cf40 1 waiting for incremental sync to catch up: 2023-11-02T18:34:27.271 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.76.6, 00000000001.223.6, , , 00000000001.272.6, , , 00000000001.76.6, 00000000003.269.6, 00000000002.322.6] 2023-11-02T18:34:27.271 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.76.6, 00000000001.223.6, , , 00000000001.272.6, , , 00000000001.76.6, 00000000003.269.6, 00000000003.326.6] 2023-11-02T18:34:32.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:32.264+0000 7fb07319cf40 1 waiting for incremental sync to catch up: 2023-11-02T18:34:32.274 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.76.6, 00000000001.223.6, , , 00000000001.272.6, , , 00000000001.76.6, 00000000003.269.6, 00000000002.322.6] 2023-11-02T18:34:32.274 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.76.6, 00000000001.223.6, , , 00000000001.272.6, , , 00000000001.76.6, 00000000003.269.6, 00000000003.326.6] 2023-11-02T18:34:37.276 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:37.268+0000 7fb07319cf40 1 waiting for incremental sync to catch up: 2023-11-02T18:34:37.276 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.76.6, 00000000001.223.6, , , 00000000001.272.6, , , 00000000001.76.6, 00000000003.269.6, 00000000002.322.6] 2023-11-02T18:34:37.276 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.76.6, 00000000001.223.6, , , 00000000001.272.6, , , 00000000001.76.6, 00000000003.269.6, 00000000003.326.6] 2023-11-02T18:34:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:42.272+0000 7fb07319cf40 1 waiting for incremental sync to catch up: 2023-11-02T18:34:42.279 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.76.6, 00000000001.223.6, , , 00000000001.272.6, , , 00000000001.76.6, 00000000003.269.6, 00000000002.322.6] 2023-11-02T18:34:42.279 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.76.6, 00000000001.223.6, , , 00000000001.272.6, , , 00000000001.76.6, 00000000003.269.6, 00000000003.326.6] 2023-11-02T18:34:47.281 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:47.272+0000 7fb07319cf40 1 bucket sync caught up with source: 2023-11-02T18:34:47.282 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000001.76.6, 00000000001.223.6, , , 00000000001.272.6, , , 00000000001.76.6, 00000000003.269.6, 00000000003.326.6] 2023-11-02T18:34:47.282 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000001.76.6, 00000000001.223.6, , , 00000000001.272.6, , , 00000000001.76.6, 00000000003.269.6, 00000000003.326.6] 2023-11-02T18:34:47.283 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:47.272+0000 7fb07319cf40 0 bucket checkpoint complete 2023-11-02T18:34:47.312 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:34:47.313 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.314 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.315 DEBUG:boto:Method: HEAD 2023-11-02T18:34:47.316 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:34:47.316 DEBUG:boto:Data: 2023-11-02T18:34:47.317 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.317 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.318 DEBUG:boto:Params: {} 2023-11-02T18:34:47.319 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:34:47.319 DEBUG:boto:Token: None 2023-11-02T18:34:47.320 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/ 2023-11-02T18:34:47.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MH43KV3bTLIU/tlc2uqQNNXBXhA= 2023-11-02T18:34:47.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MH43KV3bTLIU/tlc2uqQNNXBXhA=', 'Content-Length': '0'} 2023-11-02T18:34:47.327 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab2ffd850a9d143a-006543ebc7-1b8c-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.328 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.329 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.329 DEBUG:boto:Method: HEAD 2023-11-02T18:34:47.330 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:34:47.331 DEBUG:boto:Data: 2023-11-02T18:34:47.331 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.332 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.332 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.333 DEBUG:boto:Params: {} 2023-11-02T18:34:47.333 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:34:47.334 DEBUG:boto:Token: None 2023-11-02T18:34:47.334 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/ 2023-11-02T18:34:47.335 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MH43KV3bTLIU/tlc2uqQNNXBXhA= 2023-11-02T18:34:47.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MH43KV3bTLIU/tlc2uqQNNXBXhA=', 'Content-Length': '0'} 2023-11-02T18:34:47.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccfdaab40fb0d6c9-006543ebc7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.343 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:34:47.343 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.344 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.344 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:34:47.345 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:34:47.345 DEBUG:boto:Method: GET 2023-11-02T18:34:47.346 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:34:47.346 DEBUG:boto:Data: 2023-11-02T18:34:47.347 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.347 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.348 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.348 DEBUG:boto:Params: {} 2023-11-02T18:34:47.349 DEBUG:boto:Token: None 2023-11-02T18:34:47.349 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/?versions 2023-11-02T18:34:47.350 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NWqGYEm9MCBOTDEmGmca36yuJnQ= 2023-11-02T18:34:47.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NWqGYEm9MCBOTDEmGmca36yuJnQ=', 'Content-Length': '0'} 2023-11-02T18:34:47.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075e0b4158f1ccfde-006543ebc7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.356 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:47.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:34:47.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:34:47.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:34:47.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:34:47.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:34:47.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:34:47.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:34:47.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:34:47.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:34:47.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:34:47.362 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:34:47.362 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.362 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.363 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:34:47.363 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:34:47.364 DEBUG:boto:Method: GET 2023-11-02T18:34:47.364 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:34:47.365 DEBUG:boto:Data: 2023-11-02T18:34:47.365 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.365 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.366 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.366 DEBUG:boto:Params: {} 2023-11-02T18:34:47.367 DEBUG:boto:Token: None 2023-11-02T18:34:47.367 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/?versions 2023-11-02T18:34:47.368 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NWqGYEm9MCBOTDEmGmca36yuJnQ= 2023-11-02T18:34:47.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NWqGYEm9MCBOTDEmGmca36yuJnQ=', 'Content-Length': '0'} 2023-11-02T18:34:47.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019fdd141e5739f63-006543ebc7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.375 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:47.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:34:47.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:34:47.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:34:47.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:34:47.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:34:47.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:34:47.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:34:47.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:34:47.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:34:47.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:34:47.380 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.381 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.381 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:34:47.382 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:34:47.382 DEBUG:boto:Method: GET 2023-11-02T18:34:47.383 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:34:47.383 DEBUG:boto:Data: 2023-11-02T18:34:47.383 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.384 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.384 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.385 DEBUG:boto:Params: {} 2023-11-02T18:34:47.385 DEBUG:boto:Token: None 2023-11-02T18:34:47.386 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/?versions 2023-11-02T18:34:47.386 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NWqGYEm9MCBOTDEmGmca36yuJnQ= 2023-11-02T18:34:47.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NWqGYEm9MCBOTDEmGmca36yuJnQ=', 'Content-Length': '0'} 2023-11-02T18:34:47.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c7ce05222e36e38-006543ebc7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.392 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:47.393 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.394 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.394 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:34:47.394 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:34:47.395 DEBUG:boto:Method: GET 2023-11-02T18:34:47.395 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:34:47.396 DEBUG:boto:Data: 2023-11-02T18:34:47.396 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.397 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.397 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.397 DEBUG:boto:Params: {} 2023-11-02T18:34:47.398 DEBUG:boto:Token: None 2023-11-02T18:34:47.398 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/?versions 2023-11-02T18:34:47.399 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NWqGYEm9MCBOTDEmGmca36yuJnQ= 2023-11-02T18:34:47.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NWqGYEm9MCBOTDEmGmca36yuJnQ=', 'Content-Length': '0'} 2023-11-02T18:34:47.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003f7905b546e9105-006543ebc7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.406 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:47.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:34:47.407 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:34:47.408 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:34:47.408 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.409 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.409 DEBUG:boto:Method: GET 2023-11-02T18:34:47.410 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.410 DEBUG:boto:Data: 2023-11-02T18:34:47.410 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.411 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.411 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.412 DEBUG:boto:Params: {} 2023-11-02T18:34:47.412 DEBUG:boto:Token: None 2023-11-02T18:34:47.413 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.413 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cR/41ctyPKreBnKCD4z4Myl677c= 2023-11-02T18:34:47.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cR/41ctyPKreBnKCD4z4Myl677c=', 'Content-Length': '0'} 2023-11-02T18:34:47.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007435fb6537415b49-006543ebc7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.458 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:34:47.459 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:34:47.459 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.460 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.460 DEBUG:boto:Method: GET 2023-11-02T18:34:47.461 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.461 DEBUG:boto:Data: 2023-11-02T18:34:47.462 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.463 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.463 DEBUG:boto:Params: {} 2023-11-02T18:34:47.464 DEBUG:boto:Token: None 2023-11-02T18:34:47.464 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cR/41ctyPKreBnKCD4z4Myl677c= 2023-11-02T18:34:47.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cR/41ctyPKreBnKCD4z4Myl677c=', 'Content-Length': '0'} 2023-11-02T18:34:47.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2cc11ff5c365b11-006543ebc7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.469 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:34:47.470 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:34:47.471 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.471 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.471 DEBUG:boto:Method: HEAD 2023-11-02T18:34:47.472 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.472 DEBUG:boto:Data: 2023-11-02T18:34:47.473 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.473 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.474 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.474 DEBUG:boto:Params: {} 2023-11-02T18:34:47.474 DEBUG:boto:Token: None 2023-11-02T18:34:47.475 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.475 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hG0QSdELJWp9242A5z7TkqESrKY= 2023-11-02T18:34:47.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hG0QSdELJWp9242A5z7TkqESrKY=', 'Content-Length': '0'} 2023-11-02T18:34:47.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000253c39ea950c66c6-006543ebc7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.479 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:34:47.480 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:34:47.480 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.481 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.481 DEBUG:boto:Method: HEAD 2023-11-02T18:34:47.482 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.482 DEBUG:boto:Data: 2023-11-02T18:34:47.482 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.483 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.483 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.484 DEBUG:boto:Params: {} 2023-11-02T18:34:47.484 DEBUG:boto:Token: None 2023-11-02T18:34:47.485 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.485 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hG0QSdELJWp9242A5z7TkqESrKY= 2023-11-02T18:34:47.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hG0QSdELJWp9242A5z7TkqESrKY=', 'Content-Length': '0'} 2023-11-02T18:34:47.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff7154e5d53814a3-006543ebc7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:34:47.490 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:34:47.490 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:34:47.491 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.491 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.492 DEBUG:boto:Method: GET 2023-11-02T18:34:47.492 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.493 DEBUG:boto:Data: 2023-11-02T18:34:47.493 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.494 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.494 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.495 DEBUG:boto:Params: {} 2023-11-02T18:34:47.495 DEBUG:boto:Token: None 2023-11-02T18:34:47.496 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cR/41ctyPKreBnKCD4z4Myl677c= 2023-11-02T18:34:47.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cR/41ctyPKreBnKCD4z4Myl677c=', 'Content-Length': '0'} 2023-11-02T18:34:47.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5abb24ddfe5918-006543ebc7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.501 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:34:47.501 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:34:47.502 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.502 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.503 DEBUG:boto:Method: GET 2023-11-02T18:34:47.503 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.504 DEBUG:boto:Data: 2023-11-02T18:34:47.504 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.504 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.505 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.505 DEBUG:boto:Params: {} 2023-11-02T18:34:47.506 DEBUG:boto:Token: None 2023-11-02T18:34:47.506 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:34:47.507 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cR/41ctyPKreBnKCD4z4Myl677c= 2023-11-02T18:34:47.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cR/41ctyPKreBnKCD4z4Myl677c=', 'Content-Length': '0'} 2023-11-02T18:34:47.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af167c989e44ce5d-006543ebc7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.511 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.511 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.512 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:34:47.512 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:34:47.513 DEBUG:boto:Method: GET 2023-11-02T18:34:47.513 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:34:47.513 DEBUG:boto:Data: 2023-11-02T18:34:47.514 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.514 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.515 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.515 DEBUG:boto:Params: {} 2023-11-02T18:34:47.516 DEBUG:boto:Token: None 2023-11-02T18:34:47.516 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/ 2023-11-02T18:34:47.517 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM= 2023-11-02T18:34:47.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM=', 'Content-Length': '0'} 2023-11-02T18:34:47.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000effad3cbb8e6239d-006543ebc7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.566 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:47.567 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.568 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.568 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:34:47.569 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:34:47.569 DEBUG:boto:Method: GET 2023-11-02T18:34:47.570 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:34:47.570 DEBUG:boto:Data: 2023-11-02T18:34:47.571 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.571 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.571 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.572 DEBUG:boto:Params: {} 2023-11-02T18:34:47.572 DEBUG:boto:Token: None 2023-11-02T18:34:47.573 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/ 2023-11-02T18:34:47.573 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM= 2023-11-02T18:34:47.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM=', 'Content-Length': '0'} 2023-11-02T18:34:47.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0e6a1e627864120-006543ebc7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.579 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:47.580 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:34:47.580 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:34:47.581 DEBUG:boto:Method: HEAD 2023-11-02T18:34:47.581 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:34:47.581 DEBUG:boto:Data: 2023-11-02T18:34:47.582 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.582 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.583 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.583 DEBUG:boto:Params: {} 2023-11-02T18:34:47.583 DEBUG:boto:Token: None 2023-11-02T18:34:47.584 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj11 2023-11-02T18:34:47.584 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Mo5mKpHInsNjEWmUiKFFmxIdN24= 2023-11-02T18:34:47.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Mo5mKpHInsNjEWmUiKFFmxIdN24=', 'Content-Length': '0'} 2023-11-02T18:34:47.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084014f7f54114aa7-006543ebc7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.588 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:34:47.588 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:34:47.589 DEBUG:boto:Method: HEAD 2023-11-02T18:34:47.589 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:34:47.589 DEBUG:boto:Data: 2023-11-02T18:34:47.590 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.591 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.591 DEBUG:boto:Params: {} 2023-11-02T18:34:47.591 DEBUG:boto:Token: None 2023-11-02T18:34:47.592 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj11 2023-11-02T18:34:47.592 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Mo5mKpHInsNjEWmUiKFFmxIdN24= 2023-11-02T18:34:47.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Mo5mKpHInsNjEWmUiKFFmxIdN24=', 'Content-Length': '0'} 2023-11-02T18:34:47.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf9a6e58e91e4273-006543ebc7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:34:47.596 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:34:47.596 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:34:47.597 DEBUG:boto:Method: GET 2023-11-02T18:34:47.597 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:34:47.598 DEBUG:boto:Data: 2023-11-02T18:34:47.598 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.599 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.599 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.599 DEBUG:boto:Params: {} 2023-11-02T18:34:47.600 DEBUG:boto:Token: None 2023-11-02T18:34:47.600 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj11 2023-11-02T18:34:47.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Wn8z7T0ClMufQO7v30rP0ShtZ1o= 2023-11-02T18:34:47.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Wn8z7T0ClMufQO7v30rP0ShtZ1o=', 'Content-Length': '0'} 2023-11-02T18:34:47.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f637625f835fad4-006543ebc7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.605 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:34:47.605 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:34:47.606 DEBUG:boto:Method: GET 2023-11-02T18:34:47.606 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:34:47.607 DEBUG:boto:Data: 2023-11-02T18:34:47.607 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.608 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.608 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.608 DEBUG:boto:Params: {} 2023-11-02T18:34:47.609 DEBUG:boto:Token: None 2023-11-02T18:34:47.609 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj11 2023-11-02T18:34:47.610 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Wn8z7T0ClMufQO7v30rP0ShtZ1o= 2023-11-02T18:34:47.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Wn8z7T0ClMufQO7v30rP0ShtZ1o=', 'Content-Length': '0'} 2023-11-02T18:34:47.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1f3c4f66e447a99-006543ebc7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.654 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.655 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.655 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:34:47.656 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:34:47.656 DEBUG:boto:Method: GET 2023-11-02T18:34:47.657 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:34:47.657 DEBUG:boto:Data: 2023-11-02T18:34:47.657 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.658 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.658 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.659 DEBUG:boto:Params: {} 2023-11-02T18:34:47.659 DEBUG:boto:Token: None 2023-11-02T18:34:47.660 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/ 2023-11-02T18:34:47.660 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM= 2023-11-02T18:34:47.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM=', 'Content-Length': '0'} 2023-11-02T18:34:47.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ba588aa52c970da-006543ebc7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.667 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:47.668 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.668 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.669 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:34:47.669 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:34:47.670 DEBUG:boto:Method: GET 2023-11-02T18:34:47.670 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:34:47.670 DEBUG:boto:Data: 2023-11-02T18:34:47.673 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.674 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.674 DEBUG:boto:Params: {} 2023-11-02T18:34:47.675 DEBUG:boto:Token: None 2023-11-02T18:34:47.676 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/ 2023-11-02T18:34:47.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM= 2023-11-02T18:34:47.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM=', 'Content-Length': '0'} 2023-11-02T18:34:47.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000973a40ad85f1e150-006543ebc7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.683 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:47.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:34:47.684 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:34:47.685 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:34:47.685 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.686 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.686 DEBUG:boto:Method: GET 2023-11-02T18:34:47.687 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.687 DEBUG:boto:Data: 2023-11-02T18:34:47.688 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.688 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.688 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.689 DEBUG:boto:Params: {} 2023-11-02T18:34:47.689 DEBUG:boto:Token: None 2023-11-02T18:34:47.690 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.690 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qvCIdGkumgrMhUtkM+IOv38ro0I= 2023-11-02T18:34:47.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qvCIdGkumgrMhUtkM+IOv38ro0I=', 'Content-Length': '0'} 2023-11-02T18:34:47.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ebbe35167a4c802-006543ebc7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.738 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:34:47.739 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:34:47.739 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.740 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.740 DEBUG:boto:Method: GET 2023-11-02T18:34:47.741 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.741 DEBUG:boto:Data: 2023-11-02T18:34:47.742 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.742 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.743 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.743 DEBUG:boto:Params: {} 2023-11-02T18:34:47.743 DEBUG:boto:Token: None 2023-11-02T18:34:47.744 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.744 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qvCIdGkumgrMhUtkM+IOv38ro0I= 2023-11-02T18:34:47.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qvCIdGkumgrMhUtkM+IOv38ro0I=', 'Content-Length': '0'} 2023-11-02T18:34:47.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007283b997820de9ff-006543ebc7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.749 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:34:47.749 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:34:47.749 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.750 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.750 DEBUG:boto:Method: HEAD 2023-11-02T18:34:47.751 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.752 DEBUG:boto:Data: 2023-11-02T18:34:47.752 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.753 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.753 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.754 DEBUG:boto:Params: {} 2023-11-02T18:34:47.754 DEBUG:boto:Token: None 2023-11-02T18:34:47.755 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.755 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SNbmaW+a5p9PZ6gw10DY0DWdwlU= 2023-11-02T18:34:47.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SNbmaW+a5p9PZ6gw10DY0DWdwlU=', 'Content-Length': '0'} 2023-11-02T18:34:47.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa28156a328e2a5d-006543ebc7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.759 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:34:47.759 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:34:47.760 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.760 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.761 DEBUG:boto:Method: HEAD 2023-11-02T18:34:47.761 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.761 DEBUG:boto:Data: 2023-11-02T18:34:47.762 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.762 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.763 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.763 DEBUG:boto:Params: {} 2023-11-02T18:34:47.764 DEBUG:boto:Token: None 2023-11-02T18:34:47.764 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.765 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SNbmaW+a5p9PZ6gw10DY0DWdwlU= 2023-11-02T18:34:47.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SNbmaW+a5p9PZ6gw10DY0DWdwlU=', 'Content-Length': '0'} 2023-11-02T18:34:47.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022378e80d1fb9909-006543ebc7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:34:47.769 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:34:47.769 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:34:47.770 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.770 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.771 DEBUG:boto:Method: GET 2023-11-02T18:34:47.771 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.772 DEBUG:boto:Data: 2023-11-02T18:34:47.772 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.772 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.773 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.773 DEBUG:boto:Params: {} 2023-11-02T18:34:47.774 DEBUG:boto:Token: None 2023-11-02T18:34:47.774 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.775 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qvCIdGkumgrMhUtkM+IOv38ro0I= 2023-11-02T18:34:47.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qvCIdGkumgrMhUtkM+IOv38ro0I=', 'Content-Length': '0'} 2023-11-02T18:34:47.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f805d8bb2f9c693e-006543ebc7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.779 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:34:47.779 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:34:47.780 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.780 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.781 DEBUG:boto:Method: GET 2023-11-02T18:34:47.781 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.782 DEBUG:boto:Data: 2023-11-02T18:34:47.782 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.782 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.783 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.783 DEBUG:boto:Params: {} 2023-11-02T18:34:47.784 DEBUG:boto:Token: None 2023-11-02T18:34:47.784 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:34:47.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qvCIdGkumgrMhUtkM+IOv38ro0I= 2023-11-02T18:34:47.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qvCIdGkumgrMhUtkM+IOv38ro0I=', 'Content-Length': '0'} 2023-11-02T18:34:47.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000661ace163b368555-006543ebc7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.834 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.835 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.836 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:34:47.836 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:34:47.836 DEBUG:boto:Method: GET 2023-11-02T18:34:47.837 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:34:47.837 DEBUG:boto:Data: 2023-11-02T18:34:47.838 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.838 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.839 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.839 DEBUG:boto:Params: {} 2023-11-02T18:34:47.840 DEBUG:boto:Token: None 2023-11-02T18:34:47.840 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/ 2023-11-02T18:34:47.841 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM= 2023-11-02T18:34:47.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM=', 'Content-Length': '0'} 2023-11-02T18:34:47.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f7497c0c15c02b4-006543ebc7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.846 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:47.847 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.848 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.848 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:34:47.849 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:34:47.849 DEBUG:boto:Method: GET 2023-11-02T18:34:47.850 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:34:47.850 DEBUG:boto:Data: 2023-11-02T18:34:47.851 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.852 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.853 DEBUG:boto:Params: {} 2023-11-02T18:34:47.853 DEBUG:boto:Token: None 2023-11-02T18:34:47.854 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/ 2023-11-02T18:34:47.855 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM= 2023-11-02T18:34:47.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM=', 'Content-Length': '0'} 2023-11-02T18:34:47.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea41f44af716c11a-006543ebc7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.861 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:47.863 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:34:47.864 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:34:47.864 DEBUG:boto:Method: HEAD 2023-11-02T18:34:47.865 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:34:47.865 DEBUG:boto:Data: 2023-11-02T18:34:47.866 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.866 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.866 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.867 DEBUG:boto:Params: {} 2023-11-02T18:34:47.867 DEBUG:boto:Token: None 2023-11-02T18:34:47.868 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj17 2023-11-02T18:34:47.868 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:odqt3jnGF3ItgHPHg99tYWlQJ14= 2023-11-02T18:34:47.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:odqt3jnGF3ItgHPHg99tYWlQJ14=', 'Content-Length': '0'} 2023-11-02T18:34:47.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca6410b0b57a849b-006543ebc7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.873 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:34:47.873 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:34:47.874 DEBUG:boto:Method: HEAD 2023-11-02T18:34:47.874 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:34:47.874 DEBUG:boto:Data: 2023-11-02T18:34:47.875 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.875 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.877 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.877 DEBUG:boto:Params: {} 2023-11-02T18:34:47.878 DEBUG:boto:Token: None 2023-11-02T18:34:47.878 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj17 2023-11-02T18:34:47.879 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:odqt3jnGF3ItgHPHg99tYWlQJ14= 2023-11-02T18:34:47.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:odqt3jnGF3ItgHPHg99tYWlQJ14=', 'Content-Length': '0'} 2023-11-02T18:34:47.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a98026e092f2d972-006543ebc7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:34:47.883 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:34:47.884 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:34:47.885 DEBUG:boto:Method: GET 2023-11-02T18:34:47.885 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:34:47.886 DEBUG:boto:Data: 2023-11-02T18:34:47.886 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.887 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.887 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.888 DEBUG:boto:Params: {} 2023-11-02T18:34:47.888 DEBUG:boto:Token: None 2023-11-02T18:34:47.889 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj17 2023-11-02T18:34:47.889 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rqy62PSfIuXbOmfW+Yj0wU487hs= 2023-11-02T18:34:47.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rqy62PSfIuXbOmfW+Yj0wU487hs=', 'Content-Length': '0'} 2023-11-02T18:34:47.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed33e3939e780894-006543ebc7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.938 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:34:47.939 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:34:47.939 DEBUG:boto:Method: GET 2023-11-02T18:34:47.940 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:34:47.940 DEBUG:boto:Data: 2023-11-02T18:34:47.941 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.942 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.942 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.942 DEBUG:boto:Params: {} 2023-11-02T18:34:47.943 DEBUG:boto:Token: None 2023-11-02T18:34:47.943 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/obj17 2023-11-02T18:34:47.944 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rqy62PSfIuXbOmfW+Yj0wU487hs= 2023-11-02T18:34:47.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rqy62PSfIuXbOmfW+Yj0wU487hs=', 'Content-Length': '0'} 2023-11-02T18:34:47.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dd42e9b932f5b48-006543ebc7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.948 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.949 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.949 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:34:47.950 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:34:47.950 DEBUG:boto:Method: GET 2023-11-02T18:34:47.950 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:34:47.951 DEBUG:boto:Data: 2023-11-02T18:34:47.951 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.952 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:47.952 DEBUG:boto:Port: 8001 2023-11-02T18:34:47.953 DEBUG:boto:Params: {} 2023-11-02T18:34:47.953 DEBUG:boto:Token: None 2023-11-02T18:34:47.954 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/ 2023-11-02T18:34:47.954 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM= 2023-11-02T18:34:47.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM=', 'Content-Length': '0'} 2023-11-02T18:34:47.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3710024b1c083e4-006543ebc7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:47.962 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:47.962 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:47.963 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:47.963 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:34:47.964 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:34:47.964 DEBUG:boto:Method: GET 2023-11-02T18:34:47.965 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:34:47.965 DEBUG:boto:Data: 2023-11-02T18:34:47.966 DEBUG:boto:Headers: {} 2023-11-02T18:34:47.966 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:47.967 DEBUG:boto:Port: 8000 2023-11-02T18:34:47.967 DEBUG:boto:Params: {} 2023-11-02T18:34:47.967 DEBUG:boto:Token: None 2023-11-02T18:34:47.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:47 GMT /ibtljn-59/ 2023-11-02T18:34:47.970 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM= 2023-11-02T18:34:47.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzGOG9WyA46TNqu11MlU6BSvJrM=', 'Content-Length': '0'} 2023-11-02T18:34:47.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077edbb61d19d1396-006543ebc7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.022 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:34:48.024 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:34:48.024 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:34:48.025 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.025 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.025 DEBUG:boto:Method: GET 2023-11-02T18:34:48.026 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.026 DEBUG:boto:Data: 2023-11-02T18:34:48.027 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.027 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.028 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.028 DEBUG:boto:Params: {} 2023-11-02T18:34:48.029 DEBUG:boto:Token: None 2023-11-02T18:34:48.029 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.030 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3C+lRdLCP/fwuYzeXpdEGH2X9bQ= 2023-11-02T18:34:48.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3C+lRdLCP/fwuYzeXpdEGH2X9bQ=', 'Content-Length': '0'} 2023-11-02T18:34:48.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a960d7ed884739-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.034 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:34:48.035 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:34:48.035 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.036 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.036 DEBUG:boto:Method: GET 2023-11-02T18:34:48.037 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.037 DEBUG:boto:Data: 2023-11-02T18:34:48.038 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.039 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.039 DEBUG:boto:Params: {} 2023-11-02T18:34:48.040 DEBUG:boto:Token: None 2023-11-02T18:34:48.040 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.041 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3C+lRdLCP/fwuYzeXpdEGH2X9bQ= 2023-11-02T18:34:48.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3C+lRdLCP/fwuYzeXpdEGH2X9bQ=', 'Content-Length': '0'} 2023-11-02T18:34:48.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009df58e65d0dbc90d-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.045 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:34:48.045 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:34:48.046 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.046 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.047 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.047 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.048 DEBUG:boto:Data: 2023-11-02T18:34:48.048 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.048 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.049 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.049 DEBUG:boto:Params: {} 2023-11-02T18:34:48.050 DEBUG:boto:Token: None 2023-11-02T18:34:48.050 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sgyqRD4UQhhJWxJ2n8onU10LMSY= 2023-11-02T18:34:48.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sgyqRD4UQhhJWxJ2n8onU10LMSY=', 'Content-Length': '0'} 2023-11-02T18:34:48.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f863d81b64765b5e-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.055 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:34:48.055 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:34:48.056 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.056 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.057 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.057 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.057 DEBUG:boto:Data: 2023-11-02T18:34:48.058 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.058 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.059 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.059 DEBUG:boto:Params: {} 2023-11-02T18:34:48.060 DEBUG:boto:Token: None 2023-11-02T18:34:48.060 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.061 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sgyqRD4UQhhJWxJ2n8onU10LMSY= 2023-11-02T18:34:48.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sgyqRD4UQhhJWxJ2n8onU10LMSY=', 'Content-Length': '0'} 2023-11-02T18:34:48.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef8c5e9b130071a7-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:34:48.065 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:34:48.066 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:34:48.066 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.066 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.067 DEBUG:boto:Method: GET 2023-11-02T18:34:48.067 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.068 DEBUG:boto:Data: 2023-11-02T18:34:48.068 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.069 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.069 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.069 DEBUG:boto:Params: {} 2023-11-02T18:34:48.070 DEBUG:boto:Token: None 2023-11-02T18:34:48.070 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.071 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3C+lRdLCP/fwuYzeXpdEGH2X9bQ= 2023-11-02T18:34:48.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3C+lRdLCP/fwuYzeXpdEGH2X9bQ=', 'Content-Length': '0'} 2023-11-02T18:34:48.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b665616c1a2b357f-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.075 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:34:48.076 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:34:48.076 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.077 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.077 DEBUG:boto:Method: GET 2023-11-02T18:34:48.077 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.078 DEBUG:boto:Data: 2023-11-02T18:34:48.078 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.079 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.079 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.081 DEBUG:boto:Params: {} 2023-11-02T18:34:48.082 DEBUG:boto:Token: None 2023-11-02T18:34:48.083 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:34:48.083 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3C+lRdLCP/fwuYzeXpdEGH2X9bQ= 2023-11-02T18:34:48.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3C+lRdLCP/fwuYzeXpdEGH2X9bQ=', 'Content-Length': '0'} 2023-11-02T18:34:48.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000117ef0028382fbd1-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.088 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.088 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.089 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:34:48.089 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:34:48.090 DEBUG:boto:Method: GET 2023-11-02T18:34:48.090 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:34:48.090 DEBUG:boto:Data: 2023-11-02T18:34:48.091 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.091 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.092 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.092 DEBUG:boto:Params: {} 2023-11-02T18:34:48.093 DEBUG:boto:Token: None 2023-11-02T18:34:48.093 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000795a5119cb9d843c-006543ebc8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.142 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.143 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.144 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.144 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:34:48.145 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:34:48.145 DEBUG:boto:Method: GET 2023-11-02T18:34:48.145 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:34:48.146 DEBUG:boto:Data: 2023-11-02T18:34:48.146 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.147 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.147 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.148 DEBUG:boto:Params: {} 2023-11-02T18:34:48.148 DEBUG:boto:Token: None 2023-11-02T18:34:48.149 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.149 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecd65a6ba63eeaa4-006543ebc8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.157 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.158 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:34:48.159 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:34:48.160 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.161 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:34:48.161 DEBUG:boto:Data: 2023-11-02T18:34:48.162 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.162 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.163 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.163 DEBUG:boto:Params: {} 2023-11-02T18:34:48.165 DEBUG:boto:Token: None 2023-11-02T18:34:48.165 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj22 2023-11-02T18:34:48.166 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TLxG3w4eLYznhSzeUZat8daqln8= 2023-11-02T18:34:48.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TLxG3w4eLYznhSzeUZat8daqln8=', 'Content-Length': '0'} 2023-11-02T18:34:48.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e752c6683d40e3-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.170 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:34:48.170 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:34:48.171 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.171 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:34:48.171 DEBUG:boto:Data: 2023-11-02T18:34:48.172 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.172 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.173 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.173 DEBUG:boto:Params: {} 2023-11-02T18:34:48.174 DEBUG:boto:Token: None 2023-11-02T18:34:48.174 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj22 2023-11-02T18:34:48.175 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TLxG3w4eLYznhSzeUZat8daqln8= 2023-11-02T18:34:48.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TLxG3w4eLYznhSzeUZat8daqln8=', 'Content-Length': '0'} 2023-11-02T18:34:48.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb62007550930fdc-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:34:48.179 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:34:48.180 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:34:48.180 DEBUG:boto:Method: GET 2023-11-02T18:34:48.180 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:34:48.181 DEBUG:boto:Data: 2023-11-02T18:34:48.181 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.182 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.182 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.183 DEBUG:boto:Params: {} 2023-11-02T18:34:48.183 DEBUG:boto:Token: None 2023-11-02T18:34:48.184 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj22 2023-11-02T18:34:48.184 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zIaKbhENm/iyrfch1MjShhL3rWo= 2023-11-02T18:34:48.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zIaKbhENm/iyrfch1MjShhL3rWo=', 'Content-Length': '0'} 2023-11-02T18:34:48.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecbbe36313af2894-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.188 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:34:48.189 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:34:48.190 DEBUG:boto:Method: GET 2023-11-02T18:34:48.191 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:34:48.191 DEBUG:boto:Data: 2023-11-02T18:34:48.191 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.192 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.192 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.193 DEBUG:boto:Params: {} 2023-11-02T18:34:48.193 DEBUG:boto:Token: None 2023-11-02T18:34:48.194 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj22 2023-11-02T18:34:48.194 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zIaKbhENm/iyrfch1MjShhL3rWo= 2023-11-02T18:34:48.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zIaKbhENm/iyrfch1MjShhL3rWo=', 'Content-Length': '0'} 2023-11-02T18:34:48.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060a41e466af03d9a-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.239 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.239 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.240 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:34:48.241 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:34:48.242 DEBUG:boto:Method: GET 2023-11-02T18:34:48.242 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:34:48.243 DEBUG:boto:Data: 2023-11-02T18:34:48.243 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.243 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.244 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.244 DEBUG:boto:Params: {} 2023-11-02T18:34:48.245 DEBUG:boto:Token: None 2023-11-02T18:34:48.245 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.246 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f4c6da1e8325b0d-006543ebc8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.252 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.253 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.254 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.254 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:34:48.255 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:34:48.255 DEBUG:boto:Method: GET 2023-11-02T18:34:48.256 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:34:48.256 DEBUG:boto:Data: 2023-11-02T18:34:48.256 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.257 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.257 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.258 DEBUG:boto:Params: {} 2023-11-02T18:34:48.259 DEBUG:boto:Token: None 2023-11-02T18:34:48.260 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.260 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000222f70e8287e016c-006543ebc8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.268 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:34:48.269 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:34:48.270 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:34:48.270 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.271 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.272 DEBUG:boto:Method: GET 2023-11-02T18:34:48.272 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.273 DEBUG:boto:Data: 2023-11-02T18:34:48.273 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.274 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.274 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.275 DEBUG:boto:Params: {} 2023-11-02T18:34:48.275 DEBUG:boto:Token: None 2023-11-02T18:34:48.276 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.276 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ev7HfkDSlBO3vHerEgQTPWfg9Y8= 2023-11-02T18:34:48.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ev7HfkDSlBO3vHerEgQTPWfg9Y8=', 'Content-Length': '0'} 2023-11-02T18:34:48.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b807b79da533b7a1-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.280 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:34:48.281 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:34:48.281 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.282 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.282 DEBUG:boto:Method: GET 2023-11-02T18:34:48.283 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.283 DEBUG:boto:Data: 2023-11-02T18:34:48.284 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.284 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.285 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.285 DEBUG:boto:Params: {} 2023-11-02T18:34:48.286 DEBUG:boto:Token: None 2023-11-02T18:34:48.286 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.287 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ev7HfkDSlBO3vHerEgQTPWfg9Y8= 2023-11-02T18:34:48.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ev7HfkDSlBO3vHerEgQTPWfg9Y8=', 'Content-Length': '0'} 2023-11-02T18:34:48.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000437db79c574d32ea-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.292 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:34:48.292 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:34:48.293 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.293 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.294 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.294 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.295 DEBUG:boto:Data: 2023-11-02T18:34:48.295 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.295 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.296 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.296 DEBUG:boto:Params: {} 2023-11-02T18:34:48.297 DEBUG:boto:Token: None 2023-11-02T18:34:48.297 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.298 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vIb5tk9XYJLluHn7k0llI9HHiac= 2023-11-02T18:34:48.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vIb5tk9XYJLluHn7k0llI9HHiac=', 'Content-Length': '0'} 2023-11-02T18:34:48.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8d3550e71a0c2a5-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.302 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:34:48.302 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:34:48.303 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.303 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.304 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.304 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.304 DEBUG:boto:Data: 2023-11-02T18:34:48.305 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.305 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.306 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.306 DEBUG:boto:Params: {} 2023-11-02T18:34:48.307 DEBUG:boto:Token: None 2023-11-02T18:34:48.307 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.307 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vIb5tk9XYJLluHn7k0llI9HHiac= 2023-11-02T18:34:48.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vIb5tk9XYJLluHn7k0llI9HHiac=', 'Content-Length': '0'} 2023-11-02T18:34:48.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c4def79901305f3-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:34:48.312 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:34:48.312 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:34:48.313 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.313 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.314 DEBUG:boto:Method: GET 2023-11-02T18:34:48.314 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.314 DEBUG:boto:Data: 2023-11-02T18:34:48.315 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.316 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.316 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.317 DEBUG:boto:Params: {} 2023-11-02T18:34:48.317 DEBUG:boto:Token: None 2023-11-02T18:34:48.318 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ev7HfkDSlBO3vHerEgQTPWfg9Y8= 2023-11-02T18:34:48.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ev7HfkDSlBO3vHerEgQTPWfg9Y8=', 'Content-Length': '0'} 2023-11-02T18:34:48.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042903d1b8cf80752-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.366 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:34:48.367 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:34:48.367 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.368 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.368 DEBUG:boto:Method: GET 2023-11-02T18:34:48.369 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.369 DEBUG:boto:Data: 2023-11-02T18:34:48.370 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.370 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.371 DEBUG:boto:Params: {} 2023-11-02T18:34:48.371 DEBUG:boto:Token: None 2023-11-02T18:34:48.372 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:34:48.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ev7HfkDSlBO3vHerEgQTPWfg9Y8= 2023-11-02T18:34:48.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ev7HfkDSlBO3vHerEgQTPWfg9Y8=', 'Content-Length': '0'} 2023-11-02T18:34:48.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c9c3cd1784a911e-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.377 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.377 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.378 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:34:48.378 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:34:48.379 DEBUG:boto:Method: GET 2023-11-02T18:34:48.379 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:34:48.379 DEBUG:boto:Data: 2023-11-02T18:34:48.380 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.380 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.381 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.381 DEBUG:boto:Params: {} 2023-11-02T18:34:48.382 DEBUG:boto:Token: None 2023-11-02T18:34:48.382 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.383 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dceec2518a77532a-006543ebc8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.389 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.390 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.391 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.391 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:34:48.391 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:34:48.392 DEBUG:boto:Method: GET 2023-11-02T18:34:48.392 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:34:48.393 DEBUG:boto:Data: 2023-11-02T18:34:48.393 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.394 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.394 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.395 DEBUG:boto:Params: {} 2023-11-02T18:34:48.395 DEBUG:boto:Token: None 2023-11-02T18:34:48.396 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.396 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb2ff0aa04065284-006543ebc8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.404 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.405 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:34:48.405 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:34:48.406 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.406 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:34:48.407 DEBUG:boto:Data: 2023-11-02T18:34:48.407 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.407 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.408 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.408 DEBUG:boto:Params: {} 2023-11-02T18:34:48.409 DEBUG:boto:Token: None 2023-11-02T18:34:48.409 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj33 2023-11-02T18:34:48.410 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M/HYDMWvJfVbZqfLasq0Z1XBee0= 2023-11-02T18:34:48.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M/HYDMWvJfVbZqfLasq0Z1XBee0=', 'Content-Length': '0'} 2023-11-02T18:34:48.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c990c41a58355cc1-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.413 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:34:48.414 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:34:48.414 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.415 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:34:48.415 DEBUG:boto:Data: 2023-11-02T18:34:48.416 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.416 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.416 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.417 DEBUG:boto:Params: {} 2023-11-02T18:34:48.417 DEBUG:boto:Token: None 2023-11-02T18:34:48.418 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj33 2023-11-02T18:34:48.418 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M/HYDMWvJfVbZqfLasq0Z1XBee0= 2023-11-02T18:34:48.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M/HYDMWvJfVbZqfLasq0Z1XBee0=', 'Content-Length': '0'} 2023-11-02T18:34:48.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061acdd18311cdd5e-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:34:48.423 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:34:48.423 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:34:48.423 DEBUG:boto:Method: GET 2023-11-02T18:34:48.424 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:34:48.424 DEBUG:boto:Data: 2023-11-02T18:34:48.425 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.425 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.426 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.426 DEBUG:boto:Params: {} 2023-11-02T18:34:48.427 DEBUG:boto:Token: None 2023-11-02T18:34:48.427 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj33 2023-11-02T18:34:48.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jhVJgCWaUHymDawVCsTThSue1FY= 2023-11-02T18:34:48.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jhVJgCWaUHymDawVCsTThSue1FY=', 'Content-Length': '0'} 2023-11-02T18:34:48.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf36dbb1784abad8-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.432 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:34:48.432 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:34:48.433 DEBUG:boto:Method: GET 2023-11-02T18:34:48.433 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:34:48.433 DEBUG:boto:Data: 2023-11-02T18:34:48.434 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.434 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.435 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.435 DEBUG:boto:Params: {} 2023-11-02T18:34:48.436 DEBUG:boto:Token: None 2023-11-02T18:34:48.436 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj33 2023-11-02T18:34:48.437 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jhVJgCWaUHymDawVCsTThSue1FY= 2023-11-02T18:34:48.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jhVJgCWaUHymDawVCsTThSue1FY=', 'Content-Length': '0'} 2023-11-02T18:34:48.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000b528c4e83a98ac-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.441 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.441 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.442 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:34:48.442 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:34:48.442 DEBUG:boto:Method: GET 2023-11-02T18:34:48.443 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:34:48.443 DEBUG:boto:Data: 2023-11-02T18:34:48.444 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.444 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.445 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.445 DEBUG:boto:Params: {} 2023-11-02T18:34:48.446 DEBUG:boto:Token: None 2023-11-02T18:34:48.446 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.446 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4a38fba49d66fd4-006543ebc8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.498 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.499 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.500 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.500 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:34:48.501 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:34:48.501 DEBUG:boto:Method: GET 2023-11-02T18:34:48.502 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:34:48.502 DEBUG:boto:Data: 2023-11-02T18:34:48.503 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.503 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.504 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.504 DEBUG:boto:Params: {} 2023-11-02T18:34:48.504 DEBUG:boto:Token: None 2023-11-02T18:34:48.505 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.506 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d120a59c2837be5-006543ebc8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.513 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:34:48.514 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:34:48.514 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:34:48.515 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.515 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.516 DEBUG:boto:Method: GET 2023-11-02T18:34:48.516 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.516 DEBUG:boto:Data: 2023-11-02T18:34:48.517 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.517 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.518 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.518 DEBUG:boto:Params: {} 2023-11-02T18:34:48.519 DEBUG:boto:Token: None 2023-11-02T18:34:48.519 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.520 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xAEku2YNlWJUPVaoB6Tmtu5Q9/A= 2023-11-02T18:34:48.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xAEku2YNlWJUPVaoB6Tmtu5Q9/A=', 'Content-Length': '0'} 2023-11-02T18:34:48.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a11e15c0f7e8b2a-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.524 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:34:48.524 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:34:48.525 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.525 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.526 DEBUG:boto:Method: GET 2023-11-02T18:34:48.526 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.527 DEBUG:boto:Data: 2023-11-02T18:34:48.527 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.527 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.528 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.528 DEBUG:boto:Params: {} 2023-11-02T18:34:48.529 DEBUG:boto:Token: None 2023-11-02T18:34:48.529 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.530 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xAEku2YNlWJUPVaoB6Tmtu5Q9/A= 2023-11-02T18:34:48.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xAEku2YNlWJUPVaoB6Tmtu5Q9/A=', 'Content-Length': '0'} 2023-11-02T18:34:48.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba048690962da343-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.574 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:34:48.575 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:34:48.576 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.577 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.577 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.578 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.578 DEBUG:boto:Data: 2023-11-02T18:34:48.579 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.579 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.580 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.580 DEBUG:boto:Params: {} 2023-11-02T18:34:48.580 DEBUG:boto:Token: None 2023-11-02T18:34:48.581 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.581 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VQBotZmT7k4jf9m4/KUGxSZ8hrc= 2023-11-02T18:34:48.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VQBotZmT7k4jf9m4/KUGxSZ8hrc=', 'Content-Length': '0'} 2023-11-02T18:34:48.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b26ad2e83ce2e9f-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.585 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:34:48.586 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:34:48.586 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.587 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.587 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.588 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.588 DEBUG:boto:Data: 2023-11-02T18:34:48.588 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.589 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.589 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.590 DEBUG:boto:Params: {} 2023-11-02T18:34:48.590 DEBUG:boto:Token: None 2023-11-02T18:34:48.591 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.591 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VQBotZmT7k4jf9m4/KUGxSZ8hrc= 2023-11-02T18:34:48.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VQBotZmT7k4jf9m4/KUGxSZ8hrc=', 'Content-Length': '0'} 2023-11-02T18:34:48.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f69d35793bd270b8-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:34:48.596 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:34:48.596 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:34:48.597 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.597 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.598 DEBUG:boto:Method: GET 2023-11-02T18:34:48.598 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.598 DEBUG:boto:Data: 2023-11-02T18:34:48.599 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.599 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.600 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.600 DEBUG:boto:Params: {} 2023-11-02T18:34:48.601 DEBUG:boto:Token: None 2023-11-02T18:34:48.601 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.602 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xAEku2YNlWJUPVaoB6Tmtu5Q9/A= 2023-11-02T18:34:48.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xAEku2YNlWJUPVaoB6Tmtu5Q9/A=', 'Content-Length': '0'} 2023-11-02T18:34:48.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caef2416089b1101-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.606 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:34:48.606 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:34:48.607 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.607 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.607 DEBUG:boto:Method: GET 2023-11-02T18:34:48.608 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.608 DEBUG:boto:Data: 2023-11-02T18:34:48.609 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.609 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.610 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.610 DEBUG:boto:Params: {} 2023-11-02T18:34:48.610 DEBUG:boto:Token: None 2023-11-02T18:34:48.611 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:34:48.611 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xAEku2YNlWJUPVaoB6Tmtu5Q9/A= 2023-11-02T18:34:48.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xAEku2YNlWJUPVaoB6Tmtu5Q9/A=', 'Content-Length': '0'} 2023-11-02T18:34:48.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c74dc44267f288d-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.616 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.616 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.617 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:34:48.617 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:34:48.617 DEBUG:boto:Method: GET 2023-11-02T18:34:48.618 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:34:48.618 DEBUG:boto:Data: 2023-11-02T18:34:48.619 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.620 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.620 DEBUG:boto:Params: {} 2023-11-02T18:34:48.621 DEBUG:boto:Token: None 2023-11-02T18:34:48.621 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.622 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb55608ff71afaae-006543ebc8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.628 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.629 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.629 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.630 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:34:48.630 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:34:48.631 DEBUG:boto:Method: GET 2023-11-02T18:34:48.631 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:34:48.632 DEBUG:boto:Data: 2023-11-02T18:34:48.632 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.632 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.633 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.633 DEBUG:boto:Params: {} 2023-11-02T18:34:48.634 DEBUG:boto:Token: None 2023-11-02T18:34:48.634 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.635 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f78c853476dfac0-006543ebc8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.682 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.683 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:34:48.684 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:34:48.684 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.685 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:34:48.685 DEBUG:boto:Data: 2023-11-02T18:34:48.685 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.686 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.687 DEBUG:boto:Params: {} 2023-11-02T18:34:48.687 DEBUG:boto:Token: None 2023-11-02T18:34:48.688 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj34 2023-11-02T18:34:48.688 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NJP7j5HFxITaBSLqpYXCWFCxgUM= 2023-11-02T18:34:48.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NJP7j5HFxITaBSLqpYXCWFCxgUM=', 'Content-Length': '0'} 2023-11-02T18:34:48.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d91446059fb4bee4-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.692 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:34:48.693 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:34:48.693 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.694 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:34:48.694 DEBUG:boto:Data: 2023-11-02T18:34:48.695 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.696 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.696 DEBUG:boto:Params: {} 2023-11-02T18:34:48.696 DEBUG:boto:Token: None 2023-11-02T18:34:48.697 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj34 2023-11-02T18:34:48.697 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NJP7j5HFxITaBSLqpYXCWFCxgUM= 2023-11-02T18:34:48.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NJP7j5HFxITaBSLqpYXCWFCxgUM=', 'Content-Length': '0'} 2023-11-02T18:34:48.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dfde73658b67c23-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:34:48.702 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:34:48.702 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:34:48.706 DEBUG:boto:Method: GET 2023-11-02T18:34:48.707 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:34:48.707 DEBUG:boto:Data: 2023-11-02T18:34:48.708 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.708 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.709 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.709 DEBUG:boto:Params: {} 2023-11-02T18:34:48.710 DEBUG:boto:Token: None 2023-11-02T18:34:48.710 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj34 2023-11-02T18:34:48.711 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b1cG4AE5ARxmnimmRN2Sb5Ttf3c= 2023-11-02T18:34:48.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b1cG4AE5ARxmnimmRN2Sb5Ttf3c=', 'Content-Length': '0'} 2023-11-02T18:34:48.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094b20176923673d1-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.715 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:34:48.715 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:34:48.716 DEBUG:boto:Method: GET 2023-11-02T18:34:48.716 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:34:48.717 DEBUG:boto:Data: 2023-11-02T18:34:48.717 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.718 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.718 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.718 DEBUG:boto:Params: {} 2023-11-02T18:34:48.719 DEBUG:boto:Token: None 2023-11-02T18:34:48.720 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj34 2023-11-02T18:34:48.720 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b1cG4AE5ARxmnimmRN2Sb5Ttf3c= 2023-11-02T18:34:48.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b1cG4AE5ARxmnimmRN2Sb5Ttf3c=', 'Content-Length': '0'} 2023-11-02T18:34:48.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099aa6fa7a269a7ba-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.724 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.725 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.725 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:34:48.726 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:34:48.726 DEBUG:boto:Method: GET 2023-11-02T18:34:48.727 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:34:48.727 DEBUG:boto:Data: 2023-11-02T18:34:48.727 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.728 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.728 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.729 DEBUG:boto:Params: {} 2023-11-02T18:34:48.729 DEBUG:boto:Token: None 2023-11-02T18:34:48.730 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.730 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bb26148c959dec1-006543ebc8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.778 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.780 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.780 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.781 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:34:48.781 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:34:48.782 DEBUG:boto:Method: GET 2023-11-02T18:34:48.782 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:34:48.783 DEBUG:boto:Data: 2023-11-02T18:34:48.783 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.784 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.784 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.784 DEBUG:boto:Params: {} 2023-11-02T18:34:48.785 DEBUG:boto:Token: None 2023-11-02T18:34:48.786 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.786 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dccdfe0d3c8fcb61-006543ebc8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.794 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:34:48.795 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:34:48.796 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:34:48.796 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.797 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.797 DEBUG:boto:Method: GET 2023-11-02T18:34:48.798 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.798 DEBUG:boto:Data: 2023-11-02T18:34:48.799 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.800 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.800 DEBUG:boto:Params: {} 2023-11-02T18:34:48.801 DEBUG:boto:Token: None 2023-11-02T18:34:48.802 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.802 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9eNATqxzrkM6E5qMxKvecNB38b0= 2023-11-02T18:34:48.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9eNATqxzrkM6E5qMxKvecNB38b0=', 'Content-Length': '0'} 2023-11-02T18:34:48.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007608aace0dea8adc-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.807 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:34:48.808 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:34:48.808 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.809 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.809 DEBUG:boto:Method: GET 2023-11-02T18:34:48.810 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.810 DEBUG:boto:Data: 2023-11-02T18:34:48.811 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.811 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.812 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.812 DEBUG:boto:Params: {} 2023-11-02T18:34:48.813 DEBUG:boto:Token: None 2023-11-02T18:34:48.813 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.814 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9eNATqxzrkM6E5qMxKvecNB38b0= 2023-11-02T18:34:48.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9eNATqxzrkM6E5qMxKvecNB38b0=', 'Content-Length': '0'} 2023-11-02T18:34:48.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085ff6f60f1d72a07-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.819 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:34:48.819 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:34:48.819 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.820 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.820 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.821 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.821 DEBUG:boto:Data: 2023-11-02T18:34:48.822 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.822 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.822 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.823 DEBUG:boto:Params: {} 2023-11-02T18:34:48.823 DEBUG:boto:Token: None 2023-11-02T18:34:48.824 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.824 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rwcanv+IH7v20HgadUcD+aea8/c= 2023-11-02T18:34:48.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rwcanv+IH7v20HgadUcD+aea8/c=', 'Content-Length': '0'} 2023-11-02T18:34:48.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c4c40a513271e6-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.828 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:34:48.829 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:34:48.829 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.830 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.830 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.830 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.831 DEBUG:boto:Data: 2023-11-02T18:34:48.831 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.832 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.832 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.833 DEBUG:boto:Params: {} 2023-11-02T18:34:48.833 DEBUG:boto:Token: None 2023-11-02T18:34:48.834 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.834 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rwcanv+IH7v20HgadUcD+aea8/c= 2023-11-02T18:34:48.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rwcanv+IH7v20HgadUcD+aea8/c=', 'Content-Length': '0'} 2023-11-02T18:34:48.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004432c28c303b2d04-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:34:48.838 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:34:48.839 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:34:48.839 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.840 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.840 DEBUG:boto:Method: GET 2023-11-02T18:34:48.841 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.841 DEBUG:boto:Data: 2023-11-02T18:34:48.842 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.842 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.843 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.843 DEBUG:boto:Params: {} 2023-11-02T18:34:48.844 DEBUG:boto:Token: None 2023-11-02T18:34:48.844 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.845 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9eNATqxzrkM6E5qMxKvecNB38b0= 2023-11-02T18:34:48.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9eNATqxzrkM6E5qMxKvecNB38b0=', 'Content-Length': '0'} 2023-11-02T18:34:48.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d9284db0657dee1-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.849 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:34:48.849 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:34:48.850 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.850 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.851 DEBUG:boto:Method: GET 2023-11-02T18:34:48.851 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.852 DEBUG:boto:Data: 2023-11-02T18:34:48.852 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.853 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.853 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.854 DEBUG:boto:Params: {} 2023-11-02T18:34:48.854 DEBUG:boto:Token: None 2023-11-02T18:34:48.855 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:34:48.855 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9eNATqxzrkM6E5qMxKvecNB38b0= 2023-11-02T18:34:48.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9eNATqxzrkM6E5qMxKvecNB38b0=', 'Content-Length': '0'} 2023-11-02T18:34:48.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a587f9946e0797c-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.865 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.866 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.866 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:34:48.867 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:34:48.868 DEBUG:boto:Method: GET 2023-11-02T18:34:48.868 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:34:48.869 DEBUG:boto:Data: 2023-11-02T18:34:48.869 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.870 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.871 DEBUG:boto:Params: {} 2023-11-02T18:34:48.872 DEBUG:boto:Token: None 2023-11-02T18:34:48.872 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.873 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3f9f7b55005eeb4-006543ebc8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.926 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.927 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:48.928 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:48.928 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:34:48.929 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:34:48.929 DEBUG:boto:Method: GET 2023-11-02T18:34:48.929 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:34:48.930 DEBUG:boto:Data: 2023-11-02T18:34:48.930 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.931 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.931 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.932 DEBUG:boto:Params: {} 2023-11-02T18:34:48.932 DEBUG:boto:Token: None 2023-11-02T18:34:48.933 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/ 2023-11-02T18:34:48.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI= 2023-11-02T18:34:48.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BPgTQvnEr9FsQ3qBfclKVcIXRYI=', 'Content-Length': '0'} 2023-11-02T18:34:48.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec0e53cc1109613c-006543ebc8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.941 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:48.942 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:34:48.943 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:34:48.943 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.943 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:34:48.944 DEBUG:boto:Data: 2023-11-02T18:34:48.944 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.945 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.945 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.946 DEBUG:boto:Params: {} 2023-11-02T18:34:48.946 DEBUG:boto:Token: None 2023-11-02T18:34:48.947 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj44 2023-11-02T18:34:48.947 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KZrlUXgbucPk5qBdFdVQBw00JF0= 2023-11-02T18:34:48.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KZrlUXgbucPk5qBdFdVQBw00JF0=', 'Content-Length': '0'} 2023-11-02T18:34:48.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b439f09f3ee3f2a9-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.951 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:34:48.951 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:34:48.952 DEBUG:boto:Method: HEAD 2023-11-02T18:34:48.952 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:34:48.953 DEBUG:boto:Data: 2023-11-02T18:34:48.953 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.954 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.954 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.955 DEBUG:boto:Params: {} 2023-11-02T18:34:48.955 DEBUG:boto:Token: None 2023-11-02T18:34:48.956 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj44 2023-11-02T18:34:48.956 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KZrlUXgbucPk5qBdFdVQBw00JF0= 2023-11-02T18:34:48.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KZrlUXgbucPk5qBdFdVQBw00JF0=', 'Content-Length': '0'} 2023-11-02T18:34:48.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000173a30e97dd9f978-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:34:48.960 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:34:48.961 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:34:48.961 DEBUG:boto:Method: GET 2023-11-02T18:34:48.962 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:34:48.962 DEBUG:boto:Data: 2023-11-02T18:34:48.963 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.963 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:48.963 DEBUG:boto:Port: 8001 2023-11-02T18:34:48.964 DEBUG:boto:Params: {} 2023-11-02T18:34:48.964 DEBUG:boto:Token: None 2023-11-02T18:34:48.965 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj44 2023-11-02T18:34:48.965 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oTPel0SK0PvH4y7Bn9Y7MTWD+8g= 2023-11-02T18:34:48.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oTPel0SK0PvH4y7Bn9Y7MTWD+8g=', 'Content-Length': '0'} 2023-11-02T18:34:48.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b3c8f85fcf3be61-006543ebc8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:48.970 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:34:48.970 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:34:48.971 DEBUG:boto:Method: GET 2023-11-02T18:34:48.971 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:34:48.972 DEBUG:boto:Data: 2023-11-02T18:34:48.972 DEBUG:boto:Headers: {} 2023-11-02T18:34:48.973 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:48.973 DEBUG:boto:Port: 8000 2023-11-02T18:34:48.974 DEBUG:boto:Params: {} 2023-11-02T18:34:48.974 DEBUG:boto:Token: None 2023-11-02T18:34:48.975 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:48 GMT /ibtljn-59/obj44 2023-11-02T18:34:48.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oTPel0SK0PvH4y7Bn9Y7MTWD+8g= 2023-11-02T18:34:48.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oTPel0SK0PvH4y7Bn9Y7MTWD+8g=', 'Content-Length': '0'} 2023-11-02T18:34:48.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4992f4f5780c60a-006543ebc8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.026 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.027 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.027 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:34:49.028 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:34:49.028 DEBUG:boto:Method: GET 2023-11-02T18:34:49.029 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:34:49.029 DEBUG:boto:Data: 2023-11-02T18:34:49.030 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.030 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.031 DEBUG:boto:Params: {} 2023-11-02T18:34:49.031 DEBUG:boto:Token: None 2023-11-02T18:34:49.032 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cef088e6e7992c87-006543ebc9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.039 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.039 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.040 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.040 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:34:49.041 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:34:49.041 DEBUG:boto:Method: GET 2023-11-02T18:34:49.042 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:34:49.042 DEBUG:boto:Data: 2023-11-02T18:34:49.042 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.043 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.043 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.044 DEBUG:boto:Params: {} 2023-11-02T18:34:49.044 DEBUG:boto:Token: None 2023-11-02T18:34:49.045 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.045 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bb9d774bcfa9bea-006543ebc9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.054 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:34:49.055 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:34:49.055 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:34:49.056 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.056 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.057 DEBUG:boto:Method: GET 2023-11-02T18:34:49.057 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.058 DEBUG:boto:Data: 2023-11-02T18:34:49.058 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.059 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.059 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.059 DEBUG:boto:Params: {} 2023-11-02T18:34:49.060 DEBUG:boto:Token: None 2023-11-02T18:34:49.060 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.061 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IKipzlxGY6gxYn1TmmjYyyuScr4= 2023-11-02T18:34:49.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IKipzlxGY6gxYn1TmmjYyyuScr4=', 'Content-Length': '0'} 2023-11-02T18:34:49.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a8c62c743216385-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.106 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:34:49.107 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:34:49.108 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.108 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.109 DEBUG:boto:Method: GET 2023-11-02T18:34:49.109 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.110 DEBUG:boto:Data: 2023-11-02T18:34:49.110 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.111 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.111 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.111 DEBUG:boto:Params: {} 2023-11-02T18:34:49.113 DEBUG:boto:Token: None 2023-11-02T18:34:49.114 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.114 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IKipzlxGY6gxYn1TmmjYyyuScr4= 2023-11-02T18:34:49.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IKipzlxGY6gxYn1TmmjYyyuScr4=', 'Content-Length': '0'} 2023-11-02T18:34:49.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014cba7feb9d663b1-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.119 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:34:49.119 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:34:49.120 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.120 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.121 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.121 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.121 DEBUG:boto:Data: 2023-11-02T18:34:49.122 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.122 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.124 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.124 DEBUG:boto:Params: {} 2023-11-02T18:34:49.125 DEBUG:boto:Token: None 2023-11-02T18:34:49.126 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.127 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vGUJrOGOqF8SQV7JdEzBfho+8Hs= 2023-11-02T18:34:49.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vGUJrOGOqF8SQV7JdEzBfho+8Hs=', 'Content-Length': '0'} 2023-11-02T18:34:49.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000150742da4b4f79e1-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.132 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:34:49.132 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:34:49.133 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.133 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.134 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.134 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.134 DEBUG:boto:Data: 2023-11-02T18:34:49.135 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.135 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.136 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.136 DEBUG:boto:Params: {} 2023-11-02T18:34:49.137 DEBUG:boto:Token: None 2023-11-02T18:34:49.138 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.138 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vGUJrOGOqF8SQV7JdEzBfho+8Hs= 2023-11-02T18:34:49.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vGUJrOGOqF8SQV7JdEzBfho+8Hs=', 'Content-Length': '0'} 2023-11-02T18:34:49.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ba7a8454992929b-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:34:49.144 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:34:49.145 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:34:49.145 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.146 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.147 DEBUG:boto:Method: GET 2023-11-02T18:34:49.147 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.148 DEBUG:boto:Data: 2023-11-02T18:34:49.149 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.149 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.150 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.150 DEBUG:boto:Params: {} 2023-11-02T18:34:49.151 DEBUG:boto:Token: None 2023-11-02T18:34:49.151 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.152 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IKipzlxGY6gxYn1TmmjYyyuScr4= 2023-11-02T18:34:49.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IKipzlxGY6gxYn1TmmjYyyuScr4=', 'Content-Length': '0'} 2023-11-02T18:34:49.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008718d590c27b109-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.157 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:34:49.157 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:34:49.158 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.159 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.159 DEBUG:boto:Method: GET 2023-11-02T18:34:49.160 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.160 DEBUG:boto:Data: 2023-11-02T18:34:49.161 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.161 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.162 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.162 DEBUG:boto:Params: {} 2023-11-02T18:34:49.163 DEBUG:boto:Token: None 2023-11-02T18:34:49.163 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:34:49.164 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IKipzlxGY6gxYn1TmmjYyyuScr4= 2023-11-02T18:34:49.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IKipzlxGY6gxYn1TmmjYyyuScr4=', 'Content-Length': '0'} 2023-11-02T18:34:49.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6bb16558e84fdd6-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.168 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.170 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.170 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:34:49.171 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:34:49.171 DEBUG:boto:Method: GET 2023-11-02T18:34:49.172 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:34:49.172 DEBUG:boto:Data: 2023-11-02T18:34:49.172 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.173 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.173 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.174 DEBUG:boto:Params: {} 2023-11-02T18:34:49.175 DEBUG:boto:Token: None 2023-11-02T18:34:49.175 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.176 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d6d739783cb56b8-006543ebc9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.226 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.227 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.228 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.228 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:34:49.229 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:34:49.229 DEBUG:boto:Method: GET 2023-11-02T18:34:49.230 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:34:49.230 DEBUG:boto:Data: 2023-11-02T18:34:49.231 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.232 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.232 DEBUG:boto:Params: {} 2023-11-02T18:34:49.233 DEBUG:boto:Token: None 2023-11-02T18:34:49.233 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.234 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1e369c70ae0d04d-006543ebc9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.241 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.242 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:34:49.243 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:34:49.243 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.244 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:34:49.244 DEBUG:boto:Data: 2023-11-02T18:34:49.245 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.245 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.245 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.246 DEBUG:boto:Params: {} 2023-11-02T18:34:49.246 DEBUG:boto:Token: None 2023-11-02T18:34:49.247 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj51 2023-11-02T18:34:49.247 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a32v7OndJCQdL+qG7oNcDmH15WU= 2023-11-02T18:34:49.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a32v7OndJCQdL+qG7oNcDmH15WU=', 'Content-Length': '0'} 2023-11-02T18:34:49.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cb6f8c82e334508-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.252 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:34:49.252 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:34:49.253 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.253 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:34:49.254 DEBUG:boto:Data: 2023-11-02T18:34:49.254 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.255 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.255 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.256 DEBUG:boto:Params: {} 2023-11-02T18:34:49.256 DEBUG:boto:Token: None 2023-11-02T18:34:49.257 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj51 2023-11-02T18:34:49.257 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a32v7OndJCQdL+qG7oNcDmH15WU= 2023-11-02T18:34:49.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a32v7OndJCQdL+qG7oNcDmH15WU=', 'Content-Length': '0'} 2023-11-02T18:34:49.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca55b19756a8faad-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:34:49.262 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:34:49.262 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:34:49.263 DEBUG:boto:Method: GET 2023-11-02T18:34:49.263 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:34:49.264 DEBUG:boto:Data: 2023-11-02T18:34:49.264 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.264 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.265 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.265 DEBUG:boto:Params: {} 2023-11-02T18:34:49.266 DEBUG:boto:Token: None 2023-11-02T18:34:49.266 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj51 2023-11-02T18:34:49.267 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0b9MUAQReKudkLfMO/5f6PfW7E= 2023-11-02T18:34:49.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0b9MUAQReKudkLfMO/5f6PfW7E=', 'Content-Length': '0'} 2023-11-02T18:34:49.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076e5091f11e14228-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.271 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:34:49.272 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:34:49.272 DEBUG:boto:Method: GET 2023-11-02T18:34:49.273 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:34:49.273 DEBUG:boto:Data: 2023-11-02T18:34:49.274 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.274 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.275 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.276 DEBUG:boto:Params: {} 2023-11-02T18:34:49.276 DEBUG:boto:Token: None 2023-11-02T18:34:49.277 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj51 2023-11-02T18:34:49.277 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R0b9MUAQReKudkLfMO/5f6PfW7E= 2023-11-02T18:34:49.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R0b9MUAQReKudkLfMO/5f6PfW7E=', 'Content-Length': '0'} 2023-11-02T18:34:49.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a5a127cf5917e6-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.326 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.327 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.328 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:34:49.328 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:34:49.328 DEBUG:boto:Method: GET 2023-11-02T18:34:49.329 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:34:49.329 DEBUG:boto:Data: 2023-11-02T18:34:49.330 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.330 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.331 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.331 DEBUG:boto:Params: {} 2023-11-02T18:34:49.332 DEBUG:boto:Token: None 2023-11-02T18:34:49.332 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.333 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f45a488355380d4a-006543ebc9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.339 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.340 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.341 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.341 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:34:49.342 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:34:49.342 DEBUG:boto:Method: GET 2023-11-02T18:34:49.342 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:34:49.343 DEBUG:boto:Data: 2023-11-02T18:34:49.343 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.344 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.344 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.345 DEBUG:boto:Params: {} 2023-11-02T18:34:49.345 DEBUG:boto:Token: None 2023-11-02T18:34:49.346 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.346 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e0371e450472993-006543ebc9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.353 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:34:49.355 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:34:49.355 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:34:49.355 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.356 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.356 DEBUG:boto:Method: GET 2023-11-02T18:34:49.357 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.357 DEBUG:boto:Data: 2023-11-02T18:34:49.358 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.359 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.359 DEBUG:boto:Params: {} 2023-11-02T18:34:49.359 DEBUG:boto:Token: None 2023-11-02T18:34:49.360 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.361 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2lsLfI9gCI3aV32TdervGXK+c4Y= 2023-11-02T18:34:49.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2lsLfI9gCI3aV32TdervGXK+c4Y=', 'Content-Length': '0'} 2023-11-02T18:34:49.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c463c6f5d3aa7d65-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.406 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:34:49.407 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:34:49.407 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.408 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.408 DEBUG:boto:Method: GET 2023-11-02T18:34:49.409 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.409 DEBUG:boto:Data: 2023-11-02T18:34:49.410 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.410 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.410 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.411 DEBUG:boto:Params: {} 2023-11-02T18:34:49.411 DEBUG:boto:Token: None 2023-11-02T18:34:49.412 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2lsLfI9gCI3aV32TdervGXK+c4Y= 2023-11-02T18:34:49.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2lsLfI9gCI3aV32TdervGXK+c4Y=', 'Content-Length': '0'} 2023-11-02T18:34:49.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005337d67b20e64c82-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.417 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:34:49.418 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:34:49.418 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.419 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.419 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.419 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.420 DEBUG:boto:Data: 2023-11-02T18:34:49.420 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.421 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.421 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.422 DEBUG:boto:Params: {} 2023-11-02T18:34:49.422 DEBUG:boto:Token: None 2023-11-02T18:34:49.423 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.423 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cVqUfp++F4jrPvpEnLeoUa8Z8Xo= 2023-11-02T18:34:49.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cVqUfp++F4jrPvpEnLeoUa8Z8Xo=', 'Content-Length': '0'} 2023-11-02T18:34:49.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceb29631a27e2350-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.427 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:34:49.428 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:34:49.428 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.428 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.429 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.429 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.430 DEBUG:boto:Data: 2023-11-02T18:34:49.430 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.431 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.431 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.432 DEBUG:boto:Params: {} 2023-11-02T18:34:49.432 DEBUG:boto:Token: None 2023-11-02T18:34:49.433 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.433 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cVqUfp++F4jrPvpEnLeoUa8Z8Xo= 2023-11-02T18:34:49.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cVqUfp++F4jrPvpEnLeoUa8Z8Xo=', 'Content-Length': '0'} 2023-11-02T18:34:49.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001c336b8e76e7e49-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:34:49.438 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:34:49.438 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:34:49.439 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.439 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.440 DEBUG:boto:Method: GET 2023-11-02T18:34:49.440 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.440 DEBUG:boto:Data: 2023-11-02T18:34:49.441 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.441 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.442 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.442 DEBUG:boto:Params: {} 2023-11-02T18:34:49.443 DEBUG:boto:Token: None 2023-11-02T18:34:49.443 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2lsLfI9gCI3aV32TdervGXK+c4Y= 2023-11-02T18:34:49.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2lsLfI9gCI3aV32TdervGXK+c4Y=', 'Content-Length': '0'} 2023-11-02T18:34:49.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee57cc5f52f9b8b8-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.448 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:34:49.448 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:34:49.449 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.449 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.450 DEBUG:boto:Method: GET 2023-11-02T18:34:49.450 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.451 DEBUG:boto:Data: 2023-11-02T18:34:49.451 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.451 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.452 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.452 DEBUG:boto:Params: {} 2023-11-02T18:34:49.453 DEBUG:boto:Token: None 2023-11-02T18:34:49.453 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:34:49.454 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2lsLfI9gCI3aV32TdervGXK+c4Y= 2023-11-02T18:34:49.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2lsLfI9gCI3aV32TdervGXK+c4Y=', 'Content-Length': '0'} 2023-11-02T18:34:49.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007da566e17576eace-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.459 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.459 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.460 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:34:49.460 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:34:49.460 DEBUG:boto:Method: GET 2023-11-02T18:34:49.461 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:34:49.461 DEBUG:boto:Data: 2023-11-02T18:34:49.462 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.463 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.463 DEBUG:boto:Params: {} 2023-11-02T18:34:49.464 DEBUG:boto:Token: None 2023-11-02T18:34:49.464 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000775e3912b872fb2d-006543ebc9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.514 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.516 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.517 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.517 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:34:49.517 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:34:49.518 DEBUG:boto:Method: GET 2023-11-02T18:34:49.518 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:34:49.519 DEBUG:boto:Data: 2023-11-02T18:34:49.519 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.522 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.523 DEBUG:boto:Params: {} 2023-11-02T18:34:49.523 DEBUG:boto:Token: None 2023-11-02T18:34:49.524 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bbf00dbe2a88a5b-006543ebc9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.532 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.532 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:34:49.533 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:34:49.533 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.534 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:34:49.534 DEBUG:boto:Data: 2023-11-02T18:34:49.535 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.535 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.536 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.536 DEBUG:boto:Params: {} 2023-11-02T18:34:49.537 DEBUG:boto:Token: None 2023-11-02T18:34:49.537 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj55 2023-11-02T18:34:49.538 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9HJzNRWwlvY+gub7VAvtnsxOH4U= 2023-11-02T18:34:49.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9HJzNRWwlvY+gub7VAvtnsxOH4U=', 'Content-Length': '0'} 2023-11-02T18:34:49.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8c10e6ed2376b4-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.542 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:34:49.542 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:34:49.542 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.543 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:34:49.543 DEBUG:boto:Data: 2023-11-02T18:34:49.544 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.544 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.545 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.545 DEBUG:boto:Params: {} 2023-11-02T18:34:49.546 DEBUG:boto:Token: None 2023-11-02T18:34:49.546 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj55 2023-11-02T18:34:49.547 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9HJzNRWwlvY+gub7VAvtnsxOH4U= 2023-11-02T18:34:49.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9HJzNRWwlvY+gub7VAvtnsxOH4U=', 'Content-Length': '0'} 2023-11-02T18:34:49.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eadd25720e19ebff-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:34:49.551 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:34:49.551 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:34:49.552 DEBUG:boto:Method: GET 2023-11-02T18:34:49.552 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:34:49.553 DEBUG:boto:Data: 2023-11-02T18:34:49.553 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.553 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.554 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.554 DEBUG:boto:Params: {} 2023-11-02T18:34:49.555 DEBUG:boto:Token: None 2023-11-02T18:34:49.555 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj55 2023-11-02T18:34:49.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xd9N0qDjsoyJouEKLr2AKr+f42s= 2023-11-02T18:34:49.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xd9N0qDjsoyJouEKLr2AKr+f42s=', 'Content-Length': '0'} 2023-11-02T18:34:49.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b2720cbbf19f799-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.560 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:34:49.561 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:34:49.561 DEBUG:boto:Method: GET 2023-11-02T18:34:49.562 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:34:49.562 DEBUG:boto:Data: 2023-11-02T18:34:49.563 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.563 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.564 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.564 DEBUG:boto:Params: {} 2023-11-02T18:34:49.565 DEBUG:boto:Token: None 2023-11-02T18:34:49.565 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj55 2023-11-02T18:34:49.566 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xd9N0qDjsoyJouEKLr2AKr+f42s= 2023-11-02T18:34:49.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xd9N0qDjsoyJouEKLr2AKr+f42s=', 'Content-Length': '0'} 2023-11-02T18:34:49.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5b5181c71b2f96f-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.570 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.570 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.571 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:34:49.571 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:34:49.572 DEBUG:boto:Method: GET 2023-11-02T18:34:49.572 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:34:49.573 DEBUG:boto:Data: 2023-11-02T18:34:49.573 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.574 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.575 DEBUG:boto:Params: {} 2023-11-02T18:34:49.575 DEBUG:boto:Token: None 2023-11-02T18:34:49.576 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.576 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f796faf35412f4fc-006543ebc9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.582 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.583 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.583 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.584 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:34:49.584 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:34:49.585 DEBUG:boto:Method: GET 2023-11-02T18:34:49.585 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:34:49.586 DEBUG:boto:Data: 2023-11-02T18:34:49.586 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.587 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.587 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.588 DEBUG:boto:Params: {} 2023-11-02T18:34:49.588 DEBUG:boto:Token: None 2023-11-02T18:34:49.589 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.589 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3586b09965898fd-006543ebc9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.599 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:34:49.600 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:34:49.601 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:34:49.602 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.602 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.603 DEBUG:boto:Method: GET 2023-11-02T18:34:49.603 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.604 DEBUG:boto:Data: 2023-11-02T18:34:49.604 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.605 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.605 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.606 DEBUG:boto:Params: {} 2023-11-02T18:34:49.606 DEBUG:boto:Token: None 2023-11-02T18:34:49.607 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.607 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MRxq6F1IMB2CFHVY4Iua+ngjEb8= 2023-11-02T18:34:49.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MRxq6F1IMB2CFHVY4Iua+ngjEb8=', 'Content-Length': '0'} 2023-11-02T18:34:49.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000547c0b19e58ae0c5-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.658 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:34:49.659 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:34:49.659 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.660 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.660 DEBUG:boto:Method: GET 2023-11-02T18:34:49.661 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.661 DEBUG:boto:Data: 2023-11-02T18:34:49.662 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.662 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.663 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.663 DEBUG:boto:Params: {} 2023-11-02T18:34:49.663 DEBUG:boto:Token: None 2023-11-02T18:34:49.664 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.664 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MRxq6F1IMB2CFHVY4Iua+ngjEb8= 2023-11-02T18:34:49.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MRxq6F1IMB2CFHVY4Iua+ngjEb8=', 'Content-Length': '0'} 2023-11-02T18:34:49.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000140485e019d0a8ee-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.668 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:34:49.669 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:34:49.669 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.670 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.670 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.671 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.671 DEBUG:boto:Data: 2023-11-02T18:34:49.672 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.672 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.672 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.673 DEBUG:boto:Params: {} 2023-11-02T18:34:49.673 DEBUG:boto:Token: None 2023-11-02T18:34:49.674 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.674 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3fsL2UZpkNv9uEQqmVmjsXl2NWo= 2023-11-02T18:34:49.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3fsL2UZpkNv9uEQqmVmjsXl2NWo=', 'Content-Length': '0'} 2023-11-02T18:34:49.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b1d81406fabbb9b-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.678 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:34:49.678 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:34:49.679 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.679 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.680 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.680 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.681 DEBUG:boto:Data: 2023-11-02T18:34:49.681 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.681 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.682 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.682 DEBUG:boto:Params: {} 2023-11-02T18:34:49.683 DEBUG:boto:Token: None 2023-11-02T18:34:49.683 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.684 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3fsL2UZpkNv9uEQqmVmjsXl2NWo= 2023-11-02T18:34:49.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3fsL2UZpkNv9uEQqmVmjsXl2NWo=', 'Content-Length': '0'} 2023-11-02T18:34:49.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9d72c12f3c3d604-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:34:49.688 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:34:49.689 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:34:49.689 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.689 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.690 DEBUG:boto:Method: GET 2023-11-02T18:34:49.690 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.691 DEBUG:boto:Data: 2023-11-02T18:34:49.691 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.692 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.692 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.693 DEBUG:boto:Params: {} 2023-11-02T18:34:49.693 DEBUG:boto:Token: None 2023-11-02T18:34:49.694 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.694 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MRxq6F1IMB2CFHVY4Iua+ngjEb8= 2023-11-02T18:34:49.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MRxq6F1IMB2CFHVY4Iua+ngjEb8=', 'Content-Length': '0'} 2023-11-02T18:34:49.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c71d732ba0915a2-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.699 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:34:49.699 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:34:49.699 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.700 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.700 DEBUG:boto:Method: GET 2023-11-02T18:34:49.701 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.701 DEBUG:boto:Data: 2023-11-02T18:34:49.702 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.702 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.703 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.703 DEBUG:boto:Params: {} 2023-11-02T18:34:49.703 DEBUG:boto:Token: None 2023-11-02T18:34:49.704 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:34:49.705 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MRxq6F1IMB2CFHVY4Iua+ngjEb8= 2023-11-02T18:34:49.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MRxq6F1IMB2CFHVY4Iua+ngjEb8=', 'Content-Length': '0'} 2023-11-02T18:34:49.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b15150c5dde3b03-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.750 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.751 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.751 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:34:49.752 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:34:49.752 DEBUG:boto:Method: GET 2023-11-02T18:34:49.753 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:34:49.753 DEBUG:boto:Data: 2023-11-02T18:34:49.754 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.755 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.755 DEBUG:boto:Params: {} 2023-11-02T18:34:49.755 DEBUG:boto:Token: None 2023-11-02T18:34:49.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbdb67199331effe-006543ebc9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.763 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.764 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.764 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.764 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:34:49.765 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:34:49.765 DEBUG:boto:Method: GET 2023-11-02T18:34:49.766 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:34:49.766 DEBUG:boto:Data: 2023-11-02T18:34:49.767 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.767 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.768 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.768 DEBUG:boto:Params: {} 2023-11-02T18:34:49.769 DEBUG:boto:Token: None 2023-11-02T18:34:49.769 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.770 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5ce93d9e36264c1-006543ebc9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.777 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.778 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:34:49.778 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:34:49.779 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.779 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:34:49.779 DEBUG:boto:Data: 2023-11-02T18:34:49.780 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.780 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.781 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.781 DEBUG:boto:Params: {} 2023-11-02T18:34:49.782 DEBUG:boto:Token: None 2023-11-02T18:34:49.782 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj68 2023-11-02T18:34:49.783 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tpG4gssZyvvgraBPwvKPpAzrwoo= 2023-11-02T18:34:49.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tpG4gssZyvvgraBPwvKPpAzrwoo=', 'Content-Length': '0'} 2023-11-02T18:34:49.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000394a3b690bdca569-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.787 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:34:49.787 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:34:49.787 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.788 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:34:49.788 DEBUG:boto:Data: 2023-11-02T18:34:49.789 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.789 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.790 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.790 DEBUG:boto:Params: {} 2023-11-02T18:34:49.790 DEBUG:boto:Token: None 2023-11-02T18:34:49.791 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj68 2023-11-02T18:34:49.791 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tpG4gssZyvvgraBPwvKPpAzrwoo= 2023-11-02T18:34:49.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tpG4gssZyvvgraBPwvKPpAzrwoo=', 'Content-Length': '0'} 2023-11-02T18:34:49.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c807489d8c6a98d-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:34:49.796 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:34:49.796 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:34:49.796 DEBUG:boto:Method: GET 2023-11-02T18:34:49.797 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:34:49.797 DEBUG:boto:Data: 2023-11-02T18:34:49.798 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.798 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.799 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.799 DEBUG:boto:Params: {} 2023-11-02T18:34:49.800 DEBUG:boto:Token: None 2023-11-02T18:34:49.800 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj68 2023-11-02T18:34:49.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oAPZa7CTOTMRuK64Az3DDzfa+SU= 2023-11-02T18:34:49.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oAPZa7CTOTMRuK64Az3DDzfa+SU=', 'Content-Length': '0'} 2023-11-02T18:34:49.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaddc80de17dba80-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.805 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:34:49.806 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:34:49.806 DEBUG:boto:Method: GET 2023-11-02T18:34:49.806 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:34:49.807 DEBUG:boto:Data: 2023-11-02T18:34:49.807 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.808 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.808 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.809 DEBUG:boto:Params: {} 2023-11-02T18:34:49.809 DEBUG:boto:Token: None 2023-11-02T18:34:49.810 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj68 2023-11-02T18:34:49.810 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oAPZa7CTOTMRuK64Az3DDzfa+SU= 2023-11-02T18:34:49.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oAPZa7CTOTMRuK64Az3DDzfa+SU=', 'Content-Length': '0'} 2023-11-02T18:34:49.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7b0abc040f894e0-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.854 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.855 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.855 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:34:49.856 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:34:49.856 DEBUG:boto:Method: GET 2023-11-02T18:34:49.857 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:34:49.857 DEBUG:boto:Data: 2023-11-02T18:34:49.858 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.858 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.859 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.859 DEBUG:boto:Params: {} 2023-11-02T18:34:49.860 DEBUG:boto:Token: None 2023-11-02T18:34:49.860 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.861 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dfab2da33fe4588-006543ebc9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.866 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.867 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.867 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.868 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:34:49.868 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:34:49.869 DEBUG:boto:Method: GET 2023-11-02T18:34:49.869 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:34:49.870 DEBUG:boto:Data: 2023-11-02T18:34:49.870 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.871 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.871 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.871 DEBUG:boto:Params: {} 2023-11-02T18:34:49.872 DEBUG:boto:Token: None 2023-11-02T18:34:49.872 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.873 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb2034bdb24f981e-006543ebc9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.879 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:34:49.880 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:34:49.881 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:34:49.881 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.882 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.882 DEBUG:boto:Method: GET 2023-11-02T18:34:49.882 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.883 DEBUG:boto:Data: 2023-11-02T18:34:49.883 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.884 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.884 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.885 DEBUG:boto:Params: {} 2023-11-02T18:34:49.885 DEBUG:boto:Token: None 2023-11-02T18:34:49.886 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.886 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c/d7II3nPRoPehPy00wSDu6f15A= 2023-11-02T18:34:49.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c/d7II3nPRoPehPy00wSDu6f15A=', 'Content-Length': '0'} 2023-11-02T18:34:49.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000645ff0101b9543d3-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.930 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:34:49.931 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:34:49.932 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.932 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.932 DEBUG:boto:Method: GET 2023-11-02T18:34:49.933 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.933 DEBUG:boto:Data: 2023-11-02T18:34:49.934 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.935 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.935 DEBUG:boto:Params: {} 2023-11-02T18:34:49.936 DEBUG:boto:Token: None 2023-11-02T18:34:49.937 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.937 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c/d7II3nPRoPehPy00wSDu6f15A= 2023-11-02T18:34:49.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c/d7II3nPRoPehPy00wSDu6f15A=', 'Content-Length': '0'} 2023-11-02T18:34:49.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a9b26f435c8cd87-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.942 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:34:49.942 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:34:49.943 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.943 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.944 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.944 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.945 DEBUG:boto:Data: 2023-11-02T18:34:49.945 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.946 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.946 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.947 DEBUG:boto:Params: {} 2023-11-02T18:34:49.947 DEBUG:boto:Token: None 2023-11-02T18:34:49.948 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.948 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z6yIxlG3/Tw+I23XZqH77aywzuc= 2023-11-02T18:34:49.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z6yIxlG3/Tw+I23XZqH77aywzuc=', 'Content-Length': '0'} 2023-11-02T18:34:49.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8abc62e4813df8e-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.952 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:34:49.953 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:34:49.953 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.954 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.954 DEBUG:boto:Method: HEAD 2023-11-02T18:34:49.955 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.955 DEBUG:boto:Data: 2023-11-02T18:34:49.956 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.956 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.956 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.957 DEBUG:boto:Params: {} 2023-11-02T18:34:49.957 DEBUG:boto:Token: None 2023-11-02T18:34:49.958 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.958 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z6yIxlG3/Tw+I23XZqH77aywzuc= 2023-11-02T18:34:49.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z6yIxlG3/Tw+I23XZqH77aywzuc=', 'Content-Length': '0'} 2023-11-02T18:34:49.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083126f6c525acb34-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:34:49.963 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:34:49.964 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:34:49.964 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.964 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.965 DEBUG:boto:Method: GET 2023-11-02T18:34:49.965 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.966 DEBUG:boto:Data: 2023-11-02T18:34:49.966 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.967 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.967 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.968 DEBUG:boto:Params: {} 2023-11-02T18:34:49.968 DEBUG:boto:Token: None 2023-11-02T18:34:49.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c/d7II3nPRoPehPy00wSDu6f15A= 2023-11-02T18:34:49.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c/d7II3nPRoPehPy00wSDu6f15A=', 'Content-Length': '0'} 2023-11-02T18:34:49.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3c7e5540225b000-006543ebc9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.974 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:34:49.974 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:34:49.975 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.975 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.976 DEBUG:boto:Method: GET 2023-11-02T18:34:49.976 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.976 DEBUG:boto:Data: 2023-11-02T18:34:49.977 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.977 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:49.978 DEBUG:boto:Port: 8000 2023-11-02T18:34:49.978 DEBUG:boto:Params: {} 2023-11-02T18:34:49.979 DEBUG:boto:Token: None 2023-11-02T18:34:49.979 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:34:49.980 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c/d7II3nPRoPehPy00wSDu6f15A= 2023-11-02T18:34:49.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c/d7II3nPRoPehPy00wSDu6f15A=', 'Content-Length': '0'} 2023-11-02T18:34:49.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c322f51d4b90246-006543ebc9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.984 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.985 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.985 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:34:49.986 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:34:49.986 DEBUG:boto:Method: GET 2023-11-02T18:34:49.987 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:34:49.987 DEBUG:boto:Data: 2023-11-02T18:34:49.987 DEBUG:boto:Headers: {} 2023-11-02T18:34:49.988 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:49.988 DEBUG:boto:Port: 8001 2023-11-02T18:34:49.989 DEBUG:boto:Params: {} 2023-11-02T18:34:49.989 DEBUG:boto:Token: None 2023-11-02T18:34:49.990 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:49 GMT /ibtljn-59/ 2023-11-02T18:34:49.990 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM= 2023-11-02T18:34:49.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:49 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhImF99O9ebqjrul5UQnWB7AdkM=', 'Content-Length': '0'} 2023-11-02T18:34:49.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030cb9a22cabb2c77-006543ebc9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:49 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:49.997 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:49.997 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:49.998 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:49.998 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:34:49.999 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:34:49.999 DEBUG:boto:Method: GET 2023-11-02T18:34:50.000 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:34:50.000 DEBUG:boto:Data: 2023-11-02T18:34:50.000 DEBUG:boto:Headers: {} 2023-11-02T18:34:50.001 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.001 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.002 DEBUG:boto:Params: {} 2023-11-02T18:34:50.002 DEBUG:boto:Token: None 2023-11-02T18:34:50.003 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/ 2023-11-02T18:34:50.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BhlsAwdTD8SjPlaEoiiPZSRoLTY= 2023-11-02T18:34:50.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BhlsAwdTD8SjPlaEoiiPZSRoLTY=', 'Content-Length': '0'} 2023-11-02T18:34:50.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000356cac660da2c194-006543ebc9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.010 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:50.011 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:34:50.012 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:34:50.012 DEBUG:boto:Method: HEAD 2023-11-02T18:34:50.012 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:34:50.013 DEBUG:boto:Data: 2023-11-02T18:34:50.013 DEBUG:boto:Headers: {} 2023-11-02T18:34:50.014 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:50.014 DEBUG:boto:Port: 8001 2023-11-02T18:34:50.015 DEBUG:boto:Params: {} 2023-11-02T18:34:50.015 DEBUG:boto:Token: None 2023-11-02T18:34:50.015 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/obj85 2023-11-02T18:34:50.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4TBmAM/ceOyEJBJtemc34l0TFOs= 2023-11-02T18:34:50.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4TBmAM/ceOyEJBJtemc34l0TFOs=', 'Content-Length': '0'} 2023-11-02T18:34:50.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcad605c4271058e-006543ebca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.020 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:34:50.020 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:34:50.020 DEBUG:boto:Method: HEAD 2023-11-02T18:34:50.021 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:34:50.021 DEBUG:boto:Data: 2023-11-02T18:34:50.022 DEBUG:boto:Headers: {} 2023-11-02T18:34:50.022 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.022 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.023 DEBUG:boto:Params: {} 2023-11-02T18:34:50.023 DEBUG:boto:Token: None 2023-11-02T18:34:50.024 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/obj85 2023-11-02T18:34:50.024 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4TBmAM/ceOyEJBJtemc34l0TFOs= 2023-11-02T18:34:50.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4TBmAM/ceOyEJBJtemc34l0TFOs=', 'Content-Length': '0'} 2023-11-02T18:34:50.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006767987fe700197-006543ebca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:34:50.028 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:34:50.029 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:34:50.029 DEBUG:boto:Method: GET 2023-11-02T18:34:50.029 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:34:50.030 DEBUG:boto:Data: 2023-11-02T18:34:50.030 DEBUG:boto:Headers: {} 2023-11-02T18:34:50.031 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:50.031 DEBUG:boto:Port: 8001 2023-11-02T18:34:50.032 DEBUG:boto:Params: {} 2023-11-02T18:34:50.032 DEBUG:boto:Token: None 2023-11-02T18:34:50.033 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/obj85 2023-11-02T18:34:50.033 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KGnctYOq+ke399SX5Ee3iL7T0IY= 2023-11-02T18:34:50.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KGnctYOq+ke399SX5Ee3iL7T0IY=', 'Content-Length': '0'} 2023-11-02T18:34:50.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001abf0930fced183f-006543ebca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.078 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:34:50.079 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:34:50.079 DEBUG:boto:Method: GET 2023-11-02T18:34:50.080 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:34:50.080 DEBUG:boto:Data: 2023-11-02T18:34:50.081 DEBUG:boto:Headers: {} 2023-11-02T18:34:50.081 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.082 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.082 DEBUG:boto:Params: {} 2023-11-02T18:34:50.083 DEBUG:boto:Token: None 2023-11-02T18:34:50.083 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/obj85 2023-11-02T18:34:50.084 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KGnctYOq+ke399SX5Ee3iL7T0IY= 2023-11-02T18:34:50.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KGnctYOq+ke399SX5Ee3iL7T0IY=', 'Content-Length': '0'} 2023-11-02T18:34:50.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003941889815d801f6-006543ebca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.088 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:50.089 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:50.089 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:34:50.090 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:34:50.090 DEBUG:boto:Method: GET 2023-11-02T18:34:50.090 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:34:50.091 DEBUG:boto:Data: 2023-11-02T18:34:50.091 DEBUG:boto:Headers: {} 2023-11-02T18:34:50.092 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:34:50.092 DEBUG:boto:Port: 8001 2023-11-02T18:34:50.093 DEBUG:boto:Params: {} 2023-11-02T18:34:50.093 DEBUG:boto:Token: None 2023-11-02T18:34:50.094 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/ 2023-11-02T18:34:50.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BhlsAwdTD8SjPlaEoiiPZSRoLTY= 2023-11-02T18:34:50.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BhlsAwdTD8SjPlaEoiiPZSRoLTY=', 'Content-Length': '0'} 2023-11-02T18:34:50.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ecf37dc717e69ad-006543ebca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.099 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:50.100 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:50.100 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:50.101 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:34:50.101 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:34:50.102 DEBUG:boto:Method: GET 2023-11-02T18:34:50.102 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:34:50.103 DEBUG:boto:Data: 2023-11-02T18:34:50.103 DEBUG:boto:Headers: {} 2023-11-02T18:34:50.104 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.104 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.104 DEBUG:boto:Params: {} 2023-11-02T18:34:50.105 DEBUG:boto:Token: None 2023-11-02T18:34:50.105 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/ 2023-11-02T18:34:50.106 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BhlsAwdTD8SjPlaEoiiPZSRoLTY= 2023-11-02T18:34:50.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BhlsAwdTD8SjPlaEoiiPZSRoLTY=', 'Content-Length': '0'} 2023-11-02T18:34:50.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018c378582f166e22-006543ebca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.112 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:34:50.113 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:34:50.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:34:50.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:34:50.115 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:34:50.162 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:34:50.162 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:34:50.244 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:34:50.245 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-59 2023-11-02T18:34:50.259 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 10 2023-11-02T18:34:50.348 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:50.349 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:50.349 DEBUG:boto:Method: HEAD 2023-11-02T18:34:50.349 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:34:50.350 DEBUG:boto:Data: 2023-11-02T18:34:50.350 DEBUG:boto:Headers: {} 2023-11-02T18:34:50.351 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.351 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.351 DEBUG:boto:Params: {} 2023-11-02T18:34:50.352 DEBUG:boto:Token: None 2023-11-02T18:34:50.352 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/ 2023-11-02T18:34:50.353 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ee0A3giglTpIwtm2431z8BPO/pw= 2023-11-02T18:34:50.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ee0A3giglTpIwtm2431z8BPO/pw=', 'Content-Length': '0'} 2023-11-02T18:34:50.358 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb5abe08977d5d90-006543ebca-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.359 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:34:50.359 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:34:50.360 DEBUG:boto:Method: PUT 2023-11-02T18:34:50.360 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:34:50.360 DEBUG:boto:Data: 2023-11-02T18:34:50.361 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:34:50.361 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.362 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.362 DEBUG:boto:Params: {} 2023-11-02T18:34:50.363 DEBUG:boto:Token: None 2023-11-02T18:34:50.363 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/obj19 2023-11-02T18:34:50.363 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2WpC3Nvg2NxSat17zOhuS+A7YPw= 2023-11-02T18:34:50.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2WpC3Nvg2NxSat17zOhuS+A7YPw='} 2023-11-02T18:34:50.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077d13ba1b400e5f9-006543ebca-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.376 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:50.377 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:50.377 DEBUG:boto:Method: HEAD 2023-11-02T18:34:50.378 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:34:50.378 DEBUG:boto:Data: 2023-11-02T18:34:50.379 DEBUG:boto:Headers: {} 2023-11-02T18:34:50.379 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.379 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.380 DEBUG:boto:Params: {} 2023-11-02T18:34:50.380 DEBUG:boto:Token: None 2023-11-02T18:34:50.381 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/ 2023-11-02T18:34:50.381 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ee0A3giglTpIwtm2431z8BPO/pw= 2023-11-02T18:34:50.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ee0A3giglTpIwtm2431z8BPO/pw=', 'Content-Length': '0'} 2023-11-02T18:34:50.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027f8b7f45407b4d3-006543ebca-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.388 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:34:50.389 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:34:50.389 DEBUG:boto:Method: PUT 2023-11-02T18:34:50.390 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:34:50.390 DEBUG:boto:Data: 2023-11-02T18:34:50.391 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:34:50.391 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.391 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.392 DEBUG:boto:Params: {} 2023-11-02T18:34:50.392 DEBUG:boto:Token: None 2023-11-02T18:34:50.393 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/obj38 2023-11-02T18:34:50.393 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nGb0qkj2en5RZh0uHmH15w9Dy5M= 2023-11-02T18:34:50.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nGb0qkj2en5RZh0uHmH15w9Dy5M='} 2023-11-02T18:34:50.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bfca5350792118af-006543ebca-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.439 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:50.440 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:50.440 DEBUG:boto:Method: HEAD 2023-11-02T18:34:50.440 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:34:50.441 DEBUG:boto:Data: 2023-11-02T18:34:50.441 DEBUG:boto:Headers: {} 2023-11-02T18:34:50.442 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.442 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.442 DEBUG:boto:Params: {} 2023-11-02T18:34:50.443 DEBUG:boto:Token: None 2023-11-02T18:34:50.444 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/ 2023-11-02T18:34:50.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ee0A3giglTpIwtm2431z8BPO/pw= 2023-11-02T18:34:50.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ee0A3giglTpIwtm2431z8BPO/pw=', 'Content-Length': '0'} 2023-11-02T18:34:50.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000496277ec19d5c016-006543ebca-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.450 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:34:50.451 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:34:50.451 DEBUG:boto:Method: PUT 2023-11-02T18:34:50.452 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:34:50.452 DEBUG:boto:Data: 2023-11-02T18:34:50.452 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:34:50.453 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.453 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.454 DEBUG:boto:Params: {} 2023-11-02T18:34:50.454 DEBUG:boto:Token: None 2023-11-02T18:34:50.455 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/obj57 2023-11-02T18:34:50.455 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2Eavr2xX10B3p+Yq6MH9+7lQI3o= 2023-11-02T18:34:50.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2Eavr2xX10B3p+Yq6MH9+7lQI3o='} 2023-11-02T18:34:50.464 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004198f81c3d2f55f9-006543ebca-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.465 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:50.465 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:50.466 DEBUG:boto:Method: HEAD 2023-11-02T18:34:50.466 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:34:50.466 DEBUG:boto:Data: 2023-11-02T18:34:50.467 DEBUG:boto:Headers: {} 2023-11-02T18:34:50.467 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.468 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.468 DEBUG:boto:Params: {} 2023-11-02T18:34:50.469 DEBUG:boto:Token: None 2023-11-02T18:34:50.469 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/ 2023-11-02T18:34:50.470 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ee0A3giglTpIwtm2431z8BPO/pw= 2023-11-02T18:34:50.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ee0A3giglTpIwtm2431z8BPO/pw=', 'Content-Length': '0'} 2023-11-02T18:34:50.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000372c42c118f4caad-006543ebca-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.476 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:34:50.476 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:34:50.477 DEBUG:boto:Method: PUT 2023-11-02T18:34:50.477 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:34:50.478 DEBUG:boto:Data: 2023-11-02T18:34:50.478 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:34:50.479 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.479 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.480 DEBUG:boto:Params: {} 2023-11-02T18:34:50.480 DEBUG:boto:Token: None 2023-11-02T18:34:50.481 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/obj76 2023-11-02T18:34:50.481 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TpvhfE3VPzZKnv/jJHzhmMdTAvw= 2023-11-02T18:34:50.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TpvhfE3VPzZKnv/jJHzhmMdTAvw='} 2023-11-02T18:34:50.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001093608ab5011a2f-006543ebca-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.527 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:34:50.528 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:34:50.528 DEBUG:boto:Method: HEAD 2023-11-02T18:34:50.529 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:34:50.529 DEBUG:boto:Data: 2023-11-02T18:34:50.530 DEBUG:boto:Headers: {} 2023-11-02T18:34:50.530 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.531 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.531 DEBUG:boto:Params: {} 2023-11-02T18:34:50.532 DEBUG:boto:Token: None 2023-11-02T18:34:50.532 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/ 2023-11-02T18:34:50.533 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ee0A3giglTpIwtm2431z8BPO/pw= 2023-11-02T18:34:50.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ee0A3giglTpIwtm2431z8BPO/pw=', 'Content-Length': '0'} 2023-11-02T18:34:50.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000935c94fff90474b2-006543ebca-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.542 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:34:50.543 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:34:50.544 DEBUG:boto:Method: PUT 2023-11-02T18:34:50.545 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:34:50.545 DEBUG:boto:Data: 2023-11-02T18:34:50.546 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:34:50.547 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:34:50.547 DEBUG:boto:Port: 8000 2023-11-02T18:34:50.547 DEBUG:boto:Params: {} 2023-11-02T18:34:50.548 DEBUG:boto:Token: None 2023-11-02T18:34:50.548 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:34:50 GMT /ibtljn-59/obj95 2023-11-02T18:34:50.549 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EPjrOHWnznCMvSe8ebyBkdVoAGE= 2023-11-02T18:34:50.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:34:50 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EPjrOHWnznCMvSe8ebyBkdVoAGE='} 2023-11-02T18:34:50.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b401b83769c66c5f-006543ebca-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:34:50.560 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-59 2023-11-02T18:34:50.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:34:50.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:34:50.561 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:34:50.614 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:34:50.614 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:34:50.679 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:50.672+0000 7fc96d1e1f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T18:34:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:34:55.672+0000 7fc96d1e1f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T18:35:00.681 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:35:00.672+0000 7fc96d1e1f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T18:35:05.686 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:35:05.676+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:35:05.686 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-11-02T18:35:05.687 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:35:10.689 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:35:10.680+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:35:10.689 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-11-02T18:35:10.689 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:35:15.692 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:35:15.684+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:35:15.692 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-11-02T18:35:15.693 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:35:20.695 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:35:20.687+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:35:20.696 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-11-02T18:35:20.696 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:35:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:35:25.691+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:35:25.699 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-11-02T18:35:25.699 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:35:30.702 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:35:30.691+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:35:30.702 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-11-02T18:35:30.703 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:35:35.705 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:35:35.695+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:35:35.706 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-11-02T18:35:35.706 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:35:40.709 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:35:40.699+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:35:40.709 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-11-02T18:35:40.709 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:35:45.711 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:35:45.707+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:35:45.712 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , , , , , , , ] 2023-11-02T18:35:45.712 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:35:50.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:35:50.711+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:35:50.715 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , , , , , , , ] 2023-11-02T18:35:50.715 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:35:55.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:35:55.711+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:35:55.718 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , , , , , , , ] 2023-11-02T18:35:55.718 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:36:00.721 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:36:00.715+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:36:00.722 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , , , , , , , ] 2023-11-02T18:36:00.722 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:36:05.725 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:36:05.719+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:36:05.725 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , , , , , , , ] 2023-11-02T18:36:05.725 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:36:10.728 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:36:10.723+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:36:10.728 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , , , , , , , ] 2023-11-02T18:36:10.729 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:36:15.731 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:36:15.727+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:36:15.732 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , , , , , , , ] 2023-11-02T18:36:15.732 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:36:20.734 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:36:20.731+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:36:20.735 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , , , , , , , ] 2023-11-02T18:36:20.735 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:36:25.738 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:36:25.731+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:36:25.738 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:36:25.738 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:36:30.741 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:36:30.735+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:36:30.741 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:36:30.742 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:36:35.744 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:36:35.739+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:36:35.745 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:36:35.745 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:36:40.747 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:36:40.742+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:36:40.748 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:36:40.748 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:36:45.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:36:45.746+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:36:45.751 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:36:45.751 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:36:50.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:36:50.746+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:36:50.755 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:36:50.755 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:36:55.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:36:55.750+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:36:55.758 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:36:55.758 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:37:00.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:37:00.754+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:37:00.761 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:37:00.762 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:37:05.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:37:05.758+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:37:05.765 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:37:05.765 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:37:10.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:37:10.762+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:37:10.769 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:37:10.769 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:37:15.771 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:37:15.766+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:37:15.771 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:37:15.772 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:37:20.774 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:37:20.770+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:37:20.775 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:37:20.775 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:37:25.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:37:25.770+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:37:25.778 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:37:25.779 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:37:30.781 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:37:30.774+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:37:30.781 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:37:30.782 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:37:35.784 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:37:35.778+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:37:35.785 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:37:35.785 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:37:40.788 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:37:40.782+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:37:40.789 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000001.91.6, , , , , , , , , , , ] 2023-11-02T18:37:40.789 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:37:45.791 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:37:45.786+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:37:45.792 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:37:45.792 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:37:50.794 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:37:50.786+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:37:50.795 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:37:50.795 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:37:55.798 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:37:55.789+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:37:55.798 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:37:55.799 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:38:00.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:38:00.793+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:38:00.802 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:38:00.802 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:38:05.805 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:38:05.797+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:38:05.806 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:38:05.806 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:38:10.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:38:10.801+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:38:10.809 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:38:10.809 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:38:15.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:38:15.805+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:38:15.812 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:38:15.813 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:38:20.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:38:20.809+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:38:20.816 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:38:20.816 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:38:25.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:38:25.809+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:38:25.819 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:38:25.819 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:38:30.822 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:38:30.813+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:38:30.822 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:38:30.823 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:38:35.825 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:38:35.817+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:38:35.826 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:38:35.826 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:38:40.828 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:38:40.821+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:38:40.829 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:38:40.829 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:38:45.832 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:38:45.825+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:38:45.832 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:38:45.832 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:38:50.835 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:38:50.829+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:38:50.835 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:38:50.836 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:38:55.838 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:38:55.829+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:38:55.839 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:38:55.839 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:39:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:00.833+0000 7fc96d1e1f40 1 waiting for incremental sync to catch up: 2023-11-02T18:39:00.842 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000002.93.6, , , , , , , , , , , ] 2023-11-02T18:39:00.842 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:39:05.846 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:05.837+0000 7fc96d1e1f40 1 bucket sync caught up with source: 2023-11-02T18:39:05.846 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:39:05.847 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.116.6, 00000000004.97.6, , , , , , , , , , , ] 2023-11-02T18:39:05.847 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:05.837+0000 7fc96d1e1f40 0 bucket checkpoint complete 2023-11-02T18:39:05.877 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-59 2023-11-02T18:39:05.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:39:05.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:39:05.879 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:39:05.928 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:39:05.928 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:39:05.999 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:05.993+0000 7f797d017f40 1 bucket sync caught up with source: 2023-11-02T18:39:05.999 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.80.6, 00000000001.223.6, 00000000001.131.6, 00000000001.295.6, 00000000001.272.6, , , 00000000002.82.6, 00000000004.273.6, 00000000003.326.6] 2023-11-02T18:39:05.999 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.80.6, 00000000001.223.6, 00000000001.131.6, 00000000001.295.6, 00000000001.272.6, , , 00000000002.82.6, 00000000004.273.6, 00000000003.326.6] 2023-11-02T18:39:06.000 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:05.993+0000 7f797d017f40 0 bucket checkpoint complete 2023-11-02T18:39:06.026 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:39:06.027 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.028 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.029 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.029 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:39:06.030 DEBUG:boto:Data: 2023-11-02T18:39:06.030 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.031 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.031 DEBUG:boto:Params: {} 2023-11-02T18:39:06.032 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:39:06.032 DEBUG:boto:Token: None 2023-11-02T18:39:06.033 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.033 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P0ujU3HJ1Tca+dseoimk8ys330E= 2023-11-02T18:39:06.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P0ujU3HJ1Tca+dseoimk8ys330E=', 'Content-Length': '0'} 2023-11-02T18:39:06.041 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d24e0782b8699d1-006543ecca-1b8c-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.042 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.042 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.043 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.043 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:39:06.044 DEBUG:boto:Data: 2023-11-02T18:39:06.044 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.045 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.045 DEBUG:boto:Params: {} 2023-11-02T18:39:06.046 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:39:06.046 DEBUG:boto:Token: None 2023-11-02T18:39:06.047 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.047 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P0ujU3HJ1Tca+dseoimk8ys330E= 2023-11-02T18:39:06.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P0ujU3HJ1Tca+dseoimk8ys330E=', 'Content-Length': '0'} 2023-11-02T18:39:06.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000645edba00dd00b72-006543ecca-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.055 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:39:06.055 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.056 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.056 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:39:06.057 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:39:06.057 DEBUG:boto:Method: GET 2023-11-02T18:39:06.057 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:39:06.058 DEBUG:boto:Data: 2023-11-02T18:39:06.058 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.059 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.059 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.060 DEBUG:boto:Params: {} 2023-11-02T18:39:06.060 DEBUG:boto:Token: None 2023-11-02T18:39:06.061 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/?versions 2023-11-02T18:39:06.061 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sqC5si7OCPnx86uQ6FMqeHzJ+dY= 2023-11-02T18:39:06.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sqC5si7OCPnx86uQ6FMqeHzJ+dY=', 'Content-Length': '0'} 2023-11-02T18:39:06.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7da3cc7d232b396-006543ecca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.067 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:39:06.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:39:06.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:39:06.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:39:06.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:39:06.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:39:06.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:39:06.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:39:06.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:39:06.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:39:06.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:39:06.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:39:06.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:39:06.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:39:06.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:39:06.075 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:39:06.075 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.075 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.076 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:39:06.076 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:39:06.077 DEBUG:boto:Method: GET 2023-11-02T18:39:06.077 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:39:06.078 DEBUG:boto:Data: 2023-11-02T18:39:06.078 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.078 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.079 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.079 DEBUG:boto:Params: {} 2023-11-02T18:39:06.080 DEBUG:boto:Token: None 2023-11-02T18:39:06.080 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/?versions 2023-11-02T18:39:06.081 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sqC5si7OCPnx86uQ6FMqeHzJ+dY= 2023-11-02T18:39:06.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sqC5si7OCPnx86uQ6FMqeHzJ+dY=', 'Content-Length': '0'} 2023-11-02T18:39:06.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045dd69317ed60b11-006543ecca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.088 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:39:06.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:39:06.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:39:06.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:39:06.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:39:06.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:39:06.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:39:06.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:39:06.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:39:06.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:39:06.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:39:06.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:39:06.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:39:06.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:39:06.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:39:06.096 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.096 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.097 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:39:06.097 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:39:06.097 DEBUG:boto:Method: GET 2023-11-02T18:39:06.098 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:39:06.098 DEBUG:boto:Data: 2023-11-02T18:39:06.099 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.099 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.099 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.100 DEBUG:boto:Params: {} 2023-11-02T18:39:06.100 DEBUG:boto:Token: None 2023-11-02T18:39:06.101 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/?versions 2023-11-02T18:39:06.101 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sqC5si7OCPnx86uQ6FMqeHzJ+dY= 2023-11-02T18:39:06.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sqC5si7OCPnx86uQ6FMqeHzJ+dY=', 'Content-Length': '0'} 2023-11-02T18:39:06.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077d16ff1ced4f519-006543ecca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.107 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.109 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.109 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.110 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:39:06.110 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:39:06.111 DEBUG:boto:Method: GET 2023-11-02T18:39:06.111 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:39:06.111 DEBUG:boto:Data: 2023-11-02T18:39:06.112 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.112 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.113 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.113 DEBUG:boto:Params: {} 2023-11-02T18:39:06.113 DEBUG:boto:Token: None 2023-11-02T18:39:06.114 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/?versions 2023-11-02T18:39:06.114 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sqC5si7OCPnx86uQ6FMqeHzJ+dY= 2023-11-02T18:39:06.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sqC5si7OCPnx86uQ6FMqeHzJ+dY=', 'Content-Length': '0'} 2023-11-02T18:39:06.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3dc90f0dbf104f0-006543ecca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.121 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:39:06.123 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:39:06.123 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:39:06.124 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.124 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.125 DEBUG:boto:Method: GET 2023-11-02T18:39:06.125 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.126 DEBUG:boto:Data: 2023-11-02T18:39:06.126 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.126 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.127 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.127 DEBUG:boto:Params: {} 2023-11-02T18:39:06.128 DEBUG:boto:Token: None 2023-11-02T18:39:06.128 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Dof4sW+gBxYWlbbaAZ//0Tal1h8= 2023-11-02T18:39:06.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Dof4sW+gBxYWlbbaAZ//0Tal1h8=', 'Content-Length': '0'} 2023-11-02T18:39:06.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a21ca63385782d8-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.174 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:39:06.175 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:39:06.175 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.176 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.176 DEBUG:boto:Method: GET 2023-11-02T18:39:06.177 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.177 DEBUG:boto:Data: 2023-11-02T18:39:06.178 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.178 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.178 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.179 DEBUG:boto:Params: {} 2023-11-02T18:39:06.179 DEBUG:boto:Token: None 2023-11-02T18:39:06.180 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.180 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Dof4sW+gBxYWlbbaAZ//0Tal1h8= 2023-11-02T18:39:06.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Dof4sW+gBxYWlbbaAZ//0Tal1h8=', 'Content-Length': '0'} 2023-11-02T18:39:06.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ce1f299d8753d61-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.185 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:39:06.185 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:39:06.186 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.186 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.187 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.187 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.188 DEBUG:boto:Data: 2023-11-02T18:39:06.188 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.188 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.189 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.189 DEBUG:boto:Params: {} 2023-11-02T18:39:06.190 DEBUG:boto:Token: None 2023-11-02T18:39:06.190 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.191 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nnB2Gb+mQdxB6L3Qnp9U0/QBQKU= 2023-11-02T18:39:06.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nnB2Gb+mQdxB6L3Qnp9U0/QBQKU=', 'Content-Length': '0'} 2023-11-02T18:39:06.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024afcefc11bd71c5-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.195 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:39:06.196 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:39:06.196 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.197 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.197 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.197 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.198 DEBUG:boto:Data: 2023-11-02T18:39:06.198 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.199 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.199 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.199 DEBUG:boto:Params: {} 2023-11-02T18:39:06.200 DEBUG:boto:Token: None 2023-11-02T18:39:06.200 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nnB2Gb+mQdxB6L3Qnp9U0/QBQKU= 2023-11-02T18:39:06.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nnB2Gb+mQdxB6L3Qnp9U0/QBQKU=', 'Content-Length': '0'} 2023-11-02T18:39:06.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc375890da552ee0-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:39:06.205 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:39:06.206 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:39:06.206 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.207 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.207 DEBUG:boto:Method: GET 2023-11-02T18:39:06.208 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.208 DEBUG:boto:Data: 2023-11-02T18:39:06.208 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.209 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.209 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.210 DEBUG:boto:Params: {} 2023-11-02T18:39:06.210 DEBUG:boto:Token: None 2023-11-02T18:39:06.211 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.211 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Dof4sW+gBxYWlbbaAZ//0Tal1h8= 2023-11-02T18:39:06.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Dof4sW+gBxYWlbbaAZ//0Tal1h8=', 'Content-Length': '0'} 2023-11-02T18:39:06.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e26e750df9e1376-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.216 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:39:06.216 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:39:06.217 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.217 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.217 DEBUG:boto:Method: GET 2023-11-02T18:39:06.218 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.218 DEBUG:boto:Data: 2023-11-02T18:39:06.219 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.219 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.220 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.220 DEBUG:boto:Params: {} 2023-11-02T18:39:06.221 DEBUG:boto:Token: None 2023-11-02T18:39:06.221 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:39:06.222 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Dof4sW+gBxYWlbbaAZ//0Tal1h8= 2023-11-02T18:39:06.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Dof4sW+gBxYWlbbaAZ//0Tal1h8=', 'Content-Length': '0'} 2023-11-02T18:39:06.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000596057102267fe8b-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.266 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.267 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.267 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:39:06.268 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:39:06.268 DEBUG:boto:Method: GET 2023-11-02T18:39:06.269 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:39:06.269 DEBUG:boto:Data: 2023-11-02T18:39:06.270 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.270 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.271 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.271 DEBUG:boto:Params: {} 2023-11-02T18:39:06.272 DEBUG:boto:Token: None 2023-11-02T18:39:06.272 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.273 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw= 2023-11-02T18:39:06.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw=', 'Content-Length': '0'} 2023-11-02T18:39:06.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e1f7c3471a97748-006543ecca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.279 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.280 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.280 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.281 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:39:06.281 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:39:06.282 DEBUG:boto:Method: GET 2023-11-02T18:39:06.282 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:39:06.282 DEBUG:boto:Data: 2023-11-02T18:39:06.283 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.283 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.284 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.284 DEBUG:boto:Params: {} 2023-11-02T18:39:06.285 DEBUG:boto:Token: None 2023-11-02T18:39:06.285 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.286 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw= 2023-11-02T18:39:06.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw=', 'Content-Length': '0'} 2023-11-02T18:39:06.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000653d3f891b3928b0-006543ecca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.293 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.294 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:39:06.294 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:39:06.295 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.295 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:39:06.296 DEBUG:boto:Data: 2023-11-02T18:39:06.296 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.296 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.297 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.297 DEBUG:boto:Params: {} 2023-11-02T18:39:06.298 DEBUG:boto:Token: None 2023-11-02T18:39:06.298 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj11 2023-11-02T18:39:06.299 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BsYRd9tIbqG8V3AhMwMVn8Hlhmo= 2023-11-02T18:39:06.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BsYRd9tIbqG8V3AhMwMVn8Hlhmo=', 'Content-Length': '0'} 2023-11-02T18:39:06.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2e4da5eb1e82f17-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.303 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:39:06.303 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:39:06.304 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.304 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:39:06.304 DEBUG:boto:Data: 2023-11-02T18:39:06.305 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.305 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.306 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.306 DEBUG:boto:Params: {} 2023-11-02T18:39:06.306 DEBUG:boto:Token: None 2023-11-02T18:39:06.307 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj11 2023-11-02T18:39:06.307 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BsYRd9tIbqG8V3AhMwMVn8Hlhmo= 2023-11-02T18:39:06.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BsYRd9tIbqG8V3AhMwMVn8Hlhmo=', 'Content-Length': '0'} 2023-11-02T18:39:06.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1201e7c62e5ec7c-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:39:06.314 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:39:06.314 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:39:06.315 DEBUG:boto:Method: GET 2023-11-02T18:39:06.315 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:39:06.315 DEBUG:boto:Data: 2023-11-02T18:39:06.316 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.316 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.317 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.317 DEBUG:boto:Params: {} 2023-11-02T18:39:06.318 DEBUG:boto:Token: None 2023-11-02T18:39:06.318 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj11 2023-11-02T18:39:06.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:59RBU10kxWGqJHca7HItjWsPj1I= 2023-11-02T18:39:06.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:59RBU10kxWGqJHca7HItjWsPj1I=', 'Content-Length': '0'} 2023-11-02T18:39:06.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d24e577fe16f3335-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.366 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:39:06.367 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:39:06.367 DEBUG:boto:Method: GET 2023-11-02T18:39:06.368 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:39:06.368 DEBUG:boto:Data: 2023-11-02T18:39:06.369 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.369 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.370 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.370 DEBUG:boto:Params: {} 2023-11-02T18:39:06.371 DEBUG:boto:Token: None 2023-11-02T18:39:06.371 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj11 2023-11-02T18:39:06.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:59RBU10kxWGqJHca7HItjWsPj1I= 2023-11-02T18:39:06.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:59RBU10kxWGqJHca7HItjWsPj1I=', 'Content-Length': '0'} 2023-11-02T18:39:06.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe38c9860b8deb48-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.376 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.377 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.377 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:39:06.378 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:39:06.378 DEBUG:boto:Method: GET 2023-11-02T18:39:06.378 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:39:06.379 DEBUG:boto:Data: 2023-11-02T18:39:06.379 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.380 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.380 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.380 DEBUG:boto:Params: {} 2023-11-02T18:39:06.381 DEBUG:boto:Token: None 2023-11-02T18:39:06.381 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.382 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw= 2023-11-02T18:39:06.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw=', 'Content-Length': '0'} 2023-11-02T18:39:06.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008682cf63d1f2b61-006543ecca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.388 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.389 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.389 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.390 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:39:06.390 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:39:06.391 DEBUG:boto:Method: GET 2023-11-02T18:39:06.391 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:39:06.392 DEBUG:boto:Data: 2023-11-02T18:39:06.392 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.392 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.393 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.393 DEBUG:boto:Params: {} 2023-11-02T18:39:06.394 DEBUG:boto:Token: None 2023-11-02T18:39:06.394 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.395 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw= 2023-11-02T18:39:06.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw=', 'Content-Length': '0'} 2023-11-02T18:39:06.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcafa66d2cf39702-006543ecca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.442 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:39:06.444 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:39:06.444 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:39:06.445 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.446 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.447 DEBUG:boto:Method: GET 2023-11-02T18:39:06.447 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.447 DEBUG:boto:Data: 2023-11-02T18:39:06.448 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.448 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.449 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.449 DEBUG:boto:Params: {} 2023-11-02T18:39:06.450 DEBUG:boto:Token: None 2023-11-02T18:39:06.451 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.452 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zfe4WvQsxSmmL3/KgNsNzfDUim4= 2023-11-02T18:39:06.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zfe4WvQsxSmmL3/KgNsNzfDUim4=', 'Content-Length': '0'} 2023-11-02T18:39:06.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efa78ead8475ce9c-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.458 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:39:06.458 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:39:06.459 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.459 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.460 DEBUG:boto:Method: GET 2023-11-02T18:39:06.460 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.461 DEBUG:boto:Data: 2023-11-02T18:39:06.461 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.461 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.462 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.462 DEBUG:boto:Params: {} 2023-11-02T18:39:06.463 DEBUG:boto:Token: None 2023-11-02T18:39:06.463 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.464 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zfe4WvQsxSmmL3/KgNsNzfDUim4= 2023-11-02T18:39:06.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zfe4WvQsxSmmL3/KgNsNzfDUim4=', 'Content-Length': '0'} 2023-11-02T18:39:06.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c8267e058dc4410-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.468 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:39:06.469 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:39:06.469 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.469 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.470 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.470 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.471 DEBUG:boto:Data: 2023-11-02T18:39:06.471 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.471 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.472 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.472 DEBUG:boto:Params: {} 2023-11-02T18:39:06.473 DEBUG:boto:Token: None 2023-11-02T18:39:06.473 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.474 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JT860UM+sgRE/Ic4mHxhLCVwxA0= 2023-11-02T18:39:06.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JT860UM+sgRE/Ic4mHxhLCVwxA0=', 'Content-Length': '0'} 2023-11-02T18:39:06.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e836e8d3390cec3-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.478 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:39:06.478 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:39:06.478 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.479 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.479 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.480 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.480 DEBUG:boto:Data: 2023-11-02T18:39:06.481 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.481 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.481 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.482 DEBUG:boto:Params: {} 2023-11-02T18:39:06.482 DEBUG:boto:Token: None 2023-11-02T18:39:06.483 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.483 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JT860UM+sgRE/Ic4mHxhLCVwxA0= 2023-11-02T18:39:06.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JT860UM+sgRE/Ic4mHxhLCVwxA0=', 'Content-Length': '0'} 2023-11-02T18:39:06.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001062a36d362456f6-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:39:06.488 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:39:06.488 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:39:06.489 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.489 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.489 DEBUG:boto:Method: GET 2023-11-02T18:39:06.490 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.490 DEBUG:boto:Data: 2023-11-02T18:39:06.491 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.491 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.492 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.492 DEBUG:boto:Params: {} 2023-11-02T18:39:06.492 DEBUG:boto:Token: None 2023-11-02T18:39:06.493 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.493 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zfe4WvQsxSmmL3/KgNsNzfDUim4= 2023-11-02T18:39:06.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zfe4WvQsxSmmL3/KgNsNzfDUim4=', 'Content-Length': '0'} 2023-11-02T18:39:06.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f1e6be1d0019ff6-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.498 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:39:06.498 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:39:06.499 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.499 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.500 DEBUG:boto:Method: GET 2023-11-02T18:39:06.500 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.501 DEBUG:boto:Data: 2023-11-02T18:39:06.501 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.501 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.502 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.502 DEBUG:boto:Params: {} 2023-11-02T18:39:06.503 DEBUG:boto:Token: None 2023-11-02T18:39:06.503 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:39:06.504 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zfe4WvQsxSmmL3/KgNsNzfDUim4= 2023-11-02T18:39:06.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zfe4WvQsxSmmL3/KgNsNzfDUim4=', 'Content-Length': '0'} 2023-11-02T18:39:06.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001e56ccef2ff62c8-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.554 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.555 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.555 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:39:06.556 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:39:06.556 DEBUG:boto:Method: GET 2023-11-02T18:39:06.557 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:39:06.557 DEBUG:boto:Data: 2023-11-02T18:39:06.558 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.558 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.559 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.559 DEBUG:boto:Params: {} 2023-11-02T18:39:06.559 DEBUG:boto:Token: None 2023-11-02T18:39:06.560 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.561 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw= 2023-11-02T18:39:06.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw=', 'Content-Length': '0'} 2023-11-02T18:39:06.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f73e42aa542b9d0-006543ecca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.566 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.567 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.568 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.568 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:39:06.568 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:39:06.569 DEBUG:boto:Method: GET 2023-11-02T18:39:06.569 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:39:06.570 DEBUG:boto:Data: 2023-11-02T18:39:06.570 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.571 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.571 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.571 DEBUG:boto:Params: {} 2023-11-02T18:39:06.572 DEBUG:boto:Token: None 2023-11-02T18:39:06.572 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.573 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw= 2023-11-02T18:39:06.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw=', 'Content-Length': '0'} 2023-11-02T18:39:06.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029a3e88de962910e-006543ecca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.581 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.582 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:39:06.582 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:39:06.583 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.583 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:39:06.583 DEBUG:boto:Data: 2023-11-02T18:39:06.584 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.584 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.585 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.585 DEBUG:boto:Params: {} 2023-11-02T18:39:06.586 DEBUG:boto:Token: None 2023-11-02T18:39:06.586 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj17 2023-11-02T18:39:06.587 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o0LuHdYl1awBcyzOnzfWIvpzolU= 2023-11-02T18:39:06.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o0LuHdYl1awBcyzOnzfWIvpzolU=', 'Content-Length': '0'} 2023-11-02T18:39:06.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c35f68fb4c502271-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.590 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:39:06.591 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:39:06.591 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.592 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:39:06.592 DEBUG:boto:Data: 2023-11-02T18:39:06.592 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.593 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.593 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.594 DEBUG:boto:Params: {} 2023-11-02T18:39:06.594 DEBUG:boto:Token: None 2023-11-02T18:39:06.594 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj17 2023-11-02T18:39:06.595 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o0LuHdYl1awBcyzOnzfWIvpzolU= 2023-11-02T18:39:06.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o0LuHdYl1awBcyzOnzfWIvpzolU=', 'Content-Length': '0'} 2023-11-02T18:39:06.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003d3ca717bd0dee2-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:39:06.599 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:39:06.600 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:39:06.600 DEBUG:boto:Method: GET 2023-11-02T18:39:06.601 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:39:06.601 DEBUG:boto:Data: 2023-11-02T18:39:06.601 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.602 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.602 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.603 DEBUG:boto:Params: {} 2023-11-02T18:39:06.603 DEBUG:boto:Token: None 2023-11-02T18:39:06.604 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj17 2023-11-02T18:39:06.604 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LbquNENKtLFhDQlp1APaU0M8oYk= 2023-11-02T18:39:06.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LbquNENKtLFhDQlp1APaU0M8oYk=', 'Content-Length': '0'} 2023-11-02T18:39:06.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5acaa0dbef1ce1f-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.654 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:39:06.655 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:39:06.655 DEBUG:boto:Method: GET 2023-11-02T18:39:06.656 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:39:06.656 DEBUG:boto:Data: 2023-11-02T18:39:06.657 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.657 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.658 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.658 DEBUG:boto:Params: {} 2023-11-02T18:39:06.659 DEBUG:boto:Token: None 2023-11-02T18:39:06.659 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj17 2023-11-02T18:39:06.660 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LbquNENKtLFhDQlp1APaU0M8oYk= 2023-11-02T18:39:06.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LbquNENKtLFhDQlp1APaU0M8oYk=', 'Content-Length': '0'} 2023-11-02T18:39:06.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029b86ec3effefebf-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.664 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.664 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.665 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:39:06.665 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:39:06.666 DEBUG:boto:Method: GET 2023-11-02T18:39:06.666 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:39:06.667 DEBUG:boto:Data: 2023-11-02T18:39:06.667 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.667 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.668 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.668 DEBUG:boto:Params: {} 2023-11-02T18:39:06.669 DEBUG:boto:Token: None 2023-11-02T18:39:06.669 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.670 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw= 2023-11-02T18:39:06.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw=', 'Content-Length': '0'} 2023-11-02T18:39:06.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6daefeae41163ae-006543ecca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.677 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.677 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.678 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.678 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:39:06.679 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:39:06.679 DEBUG:boto:Method: GET 2023-11-02T18:39:06.680 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:39:06.680 DEBUG:boto:Data: 2023-11-02T18:39:06.681 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.681 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.681 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.682 DEBUG:boto:Params: {} 2023-11-02T18:39:06.682 DEBUG:boto:Token: None 2023-11-02T18:39:06.683 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.683 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw= 2023-11-02T18:39:06.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw=', 'Content-Length': '0'} 2023-11-02T18:39:06.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e6c2999048faedb-006543ecca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.738 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:39:06.740 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:39:06.740 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:39:06.741 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.741 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.741 DEBUG:boto:Method: GET 2023-11-02T18:39:06.742 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.742 DEBUG:boto:Data: 2023-11-02T18:39:06.743 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.743 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.744 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.744 DEBUG:boto:Params: {} 2023-11-02T18:39:06.745 DEBUG:boto:Token: None 2023-11-02T18:39:06.745 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.746 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IQvmYIg/R3v3kLPmsNT8/egw25I= 2023-11-02T18:39:06.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IQvmYIg/R3v3kLPmsNT8/egw25I=', 'Content-Length': '0'} 2023-11-02T18:39:06.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9684bcb56f28b3a-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.750 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:39:06.752 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:39:06.752 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.753 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.753 DEBUG:boto:Method: GET 2023-11-02T18:39:06.754 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.755 DEBUG:boto:Data: 2023-11-02T18:39:06.755 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.755 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.756 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.756 DEBUG:boto:Params: {} 2023-11-02T18:39:06.757 DEBUG:boto:Token: None 2023-11-02T18:39:06.757 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.758 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IQvmYIg/R3v3kLPmsNT8/egw25I= 2023-11-02T18:39:06.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IQvmYIg/R3v3kLPmsNT8/egw25I=', 'Content-Length': '0'} 2023-11-02T18:39:06.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006172d2abed005445-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.762 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:39:06.763 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:39:06.763 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.763 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.764 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.765 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.765 DEBUG:boto:Data: 2023-11-02T18:39:06.765 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.766 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.766 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.767 DEBUG:boto:Params: {} 2023-11-02T18:39:06.767 DEBUG:boto:Token: None 2023-11-02T18:39:06.768 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nla6tbQB3sl02rYGYbWVemgpwYI= 2023-11-02T18:39:06.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nla6tbQB3sl02rYGYbWVemgpwYI=', 'Content-Length': '0'} 2023-11-02T18:39:06.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bee27d41139afa3-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.772 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:39:06.773 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:39:06.773 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.774 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.774 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.775 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.775 DEBUG:boto:Data: 2023-11-02T18:39:06.776 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.776 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.777 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.777 DEBUG:boto:Params: {} 2023-11-02T18:39:06.778 DEBUG:boto:Token: None 2023-11-02T18:39:06.778 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.778 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nla6tbQB3sl02rYGYbWVemgpwYI= 2023-11-02T18:39:06.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nla6tbQB3sl02rYGYbWVemgpwYI=', 'Content-Length': '0'} 2023-11-02T18:39:06.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a86fcecbe726e4ee-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:39:06.783 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:39:06.783 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:39:06.784 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.784 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.785 DEBUG:boto:Method: GET 2023-11-02T18:39:06.785 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.786 DEBUG:boto:Data: 2023-11-02T18:39:06.786 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.787 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.787 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.788 DEBUG:boto:Params: {} 2023-11-02T18:39:06.788 DEBUG:boto:Token: None 2023-11-02T18:39:06.789 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IQvmYIg/R3v3kLPmsNT8/egw25I= 2023-11-02T18:39:06.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IQvmYIg/R3v3kLPmsNT8/egw25I=', 'Content-Length': '0'} 2023-11-02T18:39:06.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012aeb6ec4e1c3aad-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.793 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:39:06.794 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:39:06.794 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.795 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.795 DEBUG:boto:Method: GET 2023-11-02T18:39:06.796 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.796 DEBUG:boto:Data: 2023-11-02T18:39:06.797 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.797 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.798 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.798 DEBUG:boto:Params: {} 2023-11-02T18:39:06.799 DEBUG:boto:Token: None 2023-11-02T18:39:06.799 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:39:06.800 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IQvmYIg/R3v3kLPmsNT8/egw25I= 2023-11-02T18:39:06.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IQvmYIg/R3v3kLPmsNT8/egw25I=', 'Content-Length': '0'} 2023-11-02T18:39:06.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043401cafbf3764ce-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.804 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.804 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.805 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:39:06.805 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:39:06.806 DEBUG:boto:Method: GET 2023-11-02T18:39:06.806 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:39:06.807 DEBUG:boto:Data: 2023-11-02T18:39:06.807 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.808 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.808 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.808 DEBUG:boto:Params: {} 2023-11-02T18:39:06.809 DEBUG:boto:Token: None 2023-11-02T18:39:06.809 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.810 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw= 2023-11-02T18:39:06.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw=', 'Content-Length': '0'} 2023-11-02T18:39:06.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c2b4ed54dde75f2-006543ecca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.816 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.817 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.817 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.818 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:39:06.818 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:39:06.819 DEBUG:boto:Method: GET 2023-11-02T18:39:06.819 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:39:06.820 DEBUG:boto:Data: 2023-11-02T18:39:06.820 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.820 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.821 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.821 DEBUG:boto:Params: {} 2023-11-02T18:39:06.822 DEBUG:boto:Token: None 2023-11-02T18:39:06.822 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.823 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw= 2023-11-02T18:39:06.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw=', 'Content-Length': '0'} 2023-11-02T18:39:06.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d894619688f0878-006543ecca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.830 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.831 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:39:06.831 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:39:06.832 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.832 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:39:06.832 DEBUG:boto:Data: 2023-11-02T18:39:06.833 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.833 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.834 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.834 DEBUG:boto:Params: {} 2023-11-02T18:39:06.835 DEBUG:boto:Token: None 2023-11-02T18:39:06.835 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj19 2023-11-02T18:39:06.835 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WriGb+UOjMMyZuuCfwqDvPEOGz4= 2023-11-02T18:39:06.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WriGb+UOjMMyZuuCfwqDvPEOGz4=', 'Content-Length': '0'} 2023-11-02T18:39:06.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000032f375294f57a43-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.839 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:39:06.840 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:39:06.840 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.841 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:39:06.841 DEBUG:boto:Data: 2023-11-02T18:39:06.842 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.842 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.842 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.843 DEBUG:boto:Params: {} 2023-11-02T18:39:06.843 DEBUG:boto:Token: None 2023-11-02T18:39:06.844 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj19 2023-11-02T18:39:06.844 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WriGb+UOjMMyZuuCfwqDvPEOGz4= 2023-11-02T18:39:06.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WriGb+UOjMMyZuuCfwqDvPEOGz4=', 'Content-Length': '0'} 2023-11-02T18:39:06.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f848cbbf0575182-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:39:06.848 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:39:06.849 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:39:06.849 DEBUG:boto:Method: GET 2023-11-02T18:39:06.850 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:39:06.850 DEBUG:boto:Data: 2023-11-02T18:39:06.851 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.851 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.852 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.852 DEBUG:boto:Params: {} 2023-11-02T18:39:06.853 DEBUG:boto:Token: None 2023-11-02T18:39:06.853 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj19 2023-11-02T18:39:06.854 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lMCGCaMMCjP6pgXbmKRZNIElkAM= 2023-11-02T18:39:06.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lMCGCaMMCjP6pgXbmKRZNIElkAM=', 'Content-Length': '0'} 2023-11-02T18:39:06.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098857b8864cef6f2-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.898 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:39:06.899 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:39:06.899 DEBUG:boto:Method: GET 2023-11-02T18:39:06.900 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:39:06.900 DEBUG:boto:Data: 2023-11-02T18:39:06.901 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.901 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.902 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.902 DEBUG:boto:Params: {} 2023-11-02T18:39:06.903 DEBUG:boto:Token: None 2023-11-02T18:39:06.903 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj19 2023-11-02T18:39:06.904 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lMCGCaMMCjP6pgXbmKRZNIElkAM= 2023-11-02T18:39:06.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lMCGCaMMCjP6pgXbmKRZNIElkAM=', 'Content-Length': '0'} 2023-11-02T18:39:06.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000844465583ba3a6a6-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.908 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.909 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.909 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:39:06.909 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:39:06.910 DEBUG:boto:Method: GET 2023-11-02T18:39:06.910 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:39:06.911 DEBUG:boto:Data: 2023-11-02T18:39:06.911 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.912 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.912 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.913 DEBUG:boto:Params: {} 2023-11-02T18:39:06.913 DEBUG:boto:Token: None 2023-11-02T18:39:06.914 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.914 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw= 2023-11-02T18:39:06.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw=', 'Content-Length': '0'} 2023-11-02T18:39:06.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002829af5c70c9b2bb-006543ecca-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.920 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.921 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:06.922 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:06.922 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:39:06.923 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:39:06.923 DEBUG:boto:Method: GET 2023-11-02T18:39:06.924 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:39:06.924 DEBUG:boto:Data: 2023-11-02T18:39:06.924 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.925 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.925 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.926 DEBUG:boto:Params: {} 2023-11-02T18:39:06.926 DEBUG:boto:Token: None 2023-11-02T18:39:06.927 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/ 2023-11-02T18:39:06.927 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw= 2023-11-02T18:39:06.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OxvfbbSNDVa52PJ/Yxhr/YBm1Jw=', 'Content-Length': '0'} 2023-11-02T18:39:06.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000517c3c86a0e3ac8e-006543ecca-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.934 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:06.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:39:06.935 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:39:06.936 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:39:06.936 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.937 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.937 DEBUG:boto:Method: GET 2023-11-02T18:39:06.938 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.938 DEBUG:boto:Data: 2023-11-02T18:39:06.938 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.939 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.939 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.940 DEBUG:boto:Params: {} 2023-11-02T18:39:06.940 DEBUG:boto:Token: None 2023-11-02T18:39:06.941 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.941 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j8xJwPz0fJrO2vR1jHh6CFx71Ns= 2023-11-02T18:39:06.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j8xJwPz0fJrO2vR1jHh6CFx71Ns=', 'Content-Length': '0'} 2023-11-02T18:39:06.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f797bcea37ad3e6-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.946 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:39:06.946 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:39:06.947 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.947 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.948 DEBUG:boto:Method: GET 2023-11-02T18:39:06.948 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.949 DEBUG:boto:Data: 2023-11-02T18:39:06.949 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.949 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.950 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.950 DEBUG:boto:Params: {} 2023-11-02T18:39:06.951 DEBUG:boto:Token: None 2023-11-02T18:39:06.951 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.952 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j8xJwPz0fJrO2vR1jHh6CFx71Ns= 2023-11-02T18:39:06.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j8xJwPz0fJrO2vR1jHh6CFx71Ns=', 'Content-Length': '0'} 2023-11-02T18:39:06.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fa40d4db83b3510-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.956 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:39:06.957 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:39:06.957 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.958 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.958 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.958 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.959 DEBUG:boto:Data: 2023-11-02T18:39:06.959 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.960 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.960 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.960 DEBUG:boto:Params: {} 2023-11-02T18:39:06.961 DEBUG:boto:Token: None 2023-11-02T18:39:06.961 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.962 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:imQfkivTTwMqS7NbWaPBz2ofbAU= 2023-11-02T18:39:06.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:imQfkivTTwMqS7NbWaPBz2ofbAU=', 'Content-Length': '0'} 2023-11-02T18:39:06.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1f8bfd79c1674b7-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.966 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:39:06.966 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:39:06.967 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.967 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.967 DEBUG:boto:Method: HEAD 2023-11-02T18:39:06.968 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.968 DEBUG:boto:Data: 2023-11-02T18:39:06.969 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.969 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.970 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.970 DEBUG:boto:Params: {} 2023-11-02T18:39:06.970 DEBUG:boto:Token: None 2023-11-02T18:39:06.971 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.971 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:imQfkivTTwMqS7NbWaPBz2ofbAU= 2023-11-02T18:39:06.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:imQfkivTTwMqS7NbWaPBz2ofbAU=', 'Content-Length': '0'} 2023-11-02T18:39:06.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448c0b647a510575-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:39:06.976 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:39:06.976 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:39:06.977 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.977 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.977 DEBUG:boto:Method: GET 2023-11-02T18:39:06.978 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.978 DEBUG:boto:Data: 2023-11-02T18:39:06.979 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.979 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:06.980 DEBUG:boto:Port: 8001 2023-11-02T18:39:06.980 DEBUG:boto:Params: {} 2023-11-02T18:39:06.981 DEBUG:boto:Token: None 2023-11-02T18:39:06.981 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.982 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j8xJwPz0fJrO2vR1jHh6CFx71Ns= 2023-11-02T18:39:06.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j8xJwPz0fJrO2vR1jHh6CFx71Ns=', 'Content-Length': '0'} 2023-11-02T18:39:06.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f926fe60f97ae2a-006543ecca-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:06.986 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:39:06.987 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:39:06.987 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.987 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.988 DEBUG:boto:Method: GET 2023-11-02T18:39:06.988 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.989 DEBUG:boto:Data: 2023-11-02T18:39:06.989 DEBUG:boto:Headers: {} 2023-11-02T18:39:06.990 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:06.990 DEBUG:boto:Port: 8000 2023-11-02T18:39:06.991 DEBUG:boto:Params: {} 2023-11-02T18:39:06.991 DEBUG:boto:Token: None 2023-11-02T18:39:06.991 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:06 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:39:06.992 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j8xJwPz0fJrO2vR1jHh6CFx71Ns= 2023-11-02T18:39:06.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j8xJwPz0fJrO2vR1jHh6CFx71Ns=', 'Content-Length': '0'} 2023-11-02T18:39:06.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008df2b2efb9baa4df-006543ecca-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.038 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.039 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.039 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:39:07.040 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:39:07.040 DEBUG:boto:Method: GET 2023-11-02T18:39:07.041 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:39:07.041 DEBUG:boto:Data: 2023-11-02T18:39:07.041 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.042 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.042 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.043 DEBUG:boto:Params: {} 2023-11-02T18:39:07.043 DEBUG:boto:Token: None 2023-11-02T18:39:07.044 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.044 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c36deac2bceabcab-006543eccb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.050 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.051 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.052 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.052 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:39:07.052 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:39:07.053 DEBUG:boto:Method: GET 2023-11-02T18:39:07.053 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:39:07.054 DEBUG:boto:Data: 2023-11-02T18:39:07.054 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.055 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.055 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.056 DEBUG:boto:Params: {} 2023-11-02T18:39:07.056 DEBUG:boto:Token: None 2023-11-02T18:39:07.057 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.057 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b9cb7a06d44f656-006543eccb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.066 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.066 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:39:07.067 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:39:07.067 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.068 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:39:07.068 DEBUG:boto:Data: 2023-11-02T18:39:07.068 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.069 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.069 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.070 DEBUG:boto:Params: {} 2023-11-02T18:39:07.070 DEBUG:boto:Token: None 2023-11-02T18:39:07.071 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj22 2023-11-02T18:39:07.071 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:arGsrQgA4U1rtKmE/Ki77UI8zTQ= 2023-11-02T18:39:07.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:arGsrQgA4U1rtKmE/Ki77UI8zTQ=', 'Content-Length': '0'} 2023-11-02T18:39:07.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089db128af3d2d620-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.075 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:39:07.075 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:39:07.076 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.076 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:39:07.077 DEBUG:boto:Data: 2023-11-02T18:39:07.077 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.077 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.078 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.078 DEBUG:boto:Params: {} 2023-11-02T18:39:07.079 DEBUG:boto:Token: None 2023-11-02T18:39:07.079 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj22 2023-11-02T18:39:07.080 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:arGsrQgA4U1rtKmE/Ki77UI8zTQ= 2023-11-02T18:39:07.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:arGsrQgA4U1rtKmE/Ki77UI8zTQ=', 'Content-Length': '0'} 2023-11-02T18:39:07.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000092306d1109d366f-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:39:07.084 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:39:07.084 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:39:07.084 DEBUG:boto:Method: GET 2023-11-02T18:39:07.085 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:39:07.085 DEBUG:boto:Data: 2023-11-02T18:39:07.086 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.086 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.087 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.087 DEBUG:boto:Params: {} 2023-11-02T18:39:07.088 DEBUG:boto:Token: None 2023-11-02T18:39:07.088 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj22 2023-11-02T18:39:07.089 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMcgVCBa8eKH6/4YPkalXbqJbJA= 2023-11-02T18:39:07.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMcgVCBa8eKH6/4YPkalXbqJbJA=', 'Content-Length': '0'} 2023-11-02T18:39:07.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053e776e4b2c0e9db-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.134 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:39:07.135 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:39:07.135 DEBUG:boto:Method: GET 2023-11-02T18:39:07.136 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:39:07.136 DEBUG:boto:Data: 2023-11-02T18:39:07.137 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.137 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.138 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.138 DEBUG:boto:Params: {} 2023-11-02T18:39:07.139 DEBUG:boto:Token: None 2023-11-02T18:39:07.139 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj22 2023-11-02T18:39:07.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RMcgVCBa8eKH6/4YPkalXbqJbJA= 2023-11-02T18:39:07.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RMcgVCBa8eKH6/4YPkalXbqJbJA=', 'Content-Length': '0'} 2023-11-02T18:39:07.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b081279f1cb293c-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.144 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.145 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.145 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:39:07.145 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:39:07.146 DEBUG:boto:Method: GET 2023-11-02T18:39:07.146 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:39:07.147 DEBUG:boto:Data: 2023-11-02T18:39:07.147 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.148 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.148 DEBUG:boto:Params: {} 2023-11-02T18:39:07.149 DEBUG:boto:Token: None 2023-11-02T18:39:07.149 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.150 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e64351d1026da72-006543eccb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.156 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.157 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.158 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.158 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:39:07.159 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:39:07.159 DEBUG:boto:Method: GET 2023-11-02T18:39:07.159 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:39:07.160 DEBUG:boto:Data: 2023-11-02T18:39:07.160 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.161 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.161 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.162 DEBUG:boto:Params: {} 2023-11-02T18:39:07.162 DEBUG:boto:Token: None 2023-11-02T18:39:07.163 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.163 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4d8673b23646274-006543eccb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.214 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:39:07.216 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:39:07.216 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:39:07.217 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.217 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.217 DEBUG:boto:Method: GET 2023-11-02T18:39:07.218 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.218 DEBUG:boto:Data: 2023-11-02T18:39:07.219 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.219 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.220 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.220 DEBUG:boto:Params: {} 2023-11-02T18:39:07.221 DEBUG:boto:Token: None 2023-11-02T18:39:07.221 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.222 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+do99yh0cLOFXVuWgQBAcyeYBj0= 2023-11-02T18:39:07.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+do99yh0cLOFXVuWgQBAcyeYBj0=', 'Content-Length': '0'} 2023-11-02T18:39:07.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd67f74433c9727-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.226 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:39:07.227 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:39:07.227 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.227 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.228 DEBUG:boto:Method: GET 2023-11-02T18:39:07.228 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.229 DEBUG:boto:Data: 2023-11-02T18:39:07.229 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.230 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.230 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.230 DEBUG:boto:Params: {} 2023-11-02T18:39:07.231 DEBUG:boto:Token: None 2023-11-02T18:39:07.231 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.232 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+do99yh0cLOFXVuWgQBAcyeYBj0= 2023-11-02T18:39:07.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+do99yh0cLOFXVuWgQBAcyeYBj0=', 'Content-Length': '0'} 2023-11-02T18:39:07.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000417711a8da759df3-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.236 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:39:07.237 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:39:07.237 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.237 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.238 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.238 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.239 DEBUG:boto:Data: 2023-11-02T18:39:07.239 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.240 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.240 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.240 DEBUG:boto:Params: {} 2023-11-02T18:39:07.241 DEBUG:boto:Token: None 2023-11-02T18:39:07.241 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DFQsyjescrgB4cxk881qL2wXscA= 2023-11-02T18:39:07.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DFQsyjescrgB4cxk881qL2wXscA=', 'Content-Length': '0'} 2023-11-02T18:39:07.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bcbbb3d49b5fd08-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.246 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:39:07.246 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:39:07.247 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.247 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.248 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.248 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.248 DEBUG:boto:Data: 2023-11-02T18:39:07.249 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.249 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.250 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.250 DEBUG:boto:Params: {} 2023-11-02T18:39:07.251 DEBUG:boto:Token: None 2023-11-02T18:39:07.251 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.252 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DFQsyjescrgB4cxk881qL2wXscA= 2023-11-02T18:39:07.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DFQsyjescrgB4cxk881qL2wXscA=', 'Content-Length': '0'} 2023-11-02T18:39:07.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b56cec44ae66623b-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:39:07.256 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:39:07.256 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:39:07.257 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.257 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.258 DEBUG:boto:Method: GET 2023-11-02T18:39:07.258 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.259 DEBUG:boto:Data: 2023-11-02T18:39:07.259 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.260 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.260 DEBUG:boto:Params: {} 2023-11-02T18:39:07.261 DEBUG:boto:Token: None 2023-11-02T18:39:07.261 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.262 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+do99yh0cLOFXVuWgQBAcyeYBj0= 2023-11-02T18:39:07.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+do99yh0cLOFXVuWgQBAcyeYBj0=', 'Content-Length': '0'} 2023-11-02T18:39:07.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc684263901fcf7a-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.266 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:39:07.266 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:39:07.269 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.269 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.270 DEBUG:boto:Method: GET 2023-11-02T18:39:07.270 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.271 DEBUG:boto:Data: 2023-11-02T18:39:07.271 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.272 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.272 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.273 DEBUG:boto:Params: {} 2023-11-02T18:39:07.273 DEBUG:boto:Token: None 2023-11-02T18:39:07.274 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:39:07.274 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+do99yh0cLOFXVuWgQBAcyeYBj0= 2023-11-02T18:39:07.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+do99yh0cLOFXVuWgQBAcyeYBj0=', 'Content-Length': '0'} 2023-11-02T18:39:07.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f7ef482b1d2293c-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.318 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.319 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.319 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:39:07.320 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:39:07.320 DEBUG:boto:Method: GET 2023-11-02T18:39:07.321 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:39:07.321 DEBUG:boto:Data: 2023-11-02T18:39:07.322 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.322 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.322 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.323 DEBUG:boto:Params: {} 2023-11-02T18:39:07.323 DEBUG:boto:Token: None 2023-11-02T18:39:07.324 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.324 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010a5d6ad2ccae7bb-006543eccb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.330 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.331 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.332 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.332 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:39:07.333 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:39:07.333 DEBUG:boto:Method: GET 2023-11-02T18:39:07.334 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:39:07.334 DEBUG:boto:Data: 2023-11-02T18:39:07.335 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.335 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.335 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.336 DEBUG:boto:Params: {} 2023-11-02T18:39:07.336 DEBUG:boto:Token: None 2023-11-02T18:39:07.337 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.337 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7a0aa250b858b5c-006543eccb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.345 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.346 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:39:07.346 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:39:07.347 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.347 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:39:07.347 DEBUG:boto:Data: 2023-11-02T18:39:07.348 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.348 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.349 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.349 DEBUG:boto:Params: {} 2023-11-02T18:39:07.350 DEBUG:boto:Token: None 2023-11-02T18:39:07.350 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj33 2023-11-02T18:39:07.351 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B3hV9h2zFMfIn0JdZAsU9F2mCgU= 2023-11-02T18:39:07.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B3hV9h2zFMfIn0JdZAsU9F2mCgU=', 'Content-Length': '0'} 2023-11-02T18:39:07.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd9b160ccb206a25-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.354 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:39:07.355 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:39:07.355 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.356 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:39:07.356 DEBUG:boto:Data: 2023-11-02T18:39:07.357 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.357 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.357 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.358 DEBUG:boto:Params: {} 2023-11-02T18:39:07.358 DEBUG:boto:Token: None 2023-11-02T18:39:07.359 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj33 2023-11-02T18:39:07.359 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B3hV9h2zFMfIn0JdZAsU9F2mCgU= 2023-11-02T18:39:07.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B3hV9h2zFMfIn0JdZAsU9F2mCgU=', 'Content-Length': '0'} 2023-11-02T18:39:07.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b184d1684798bea-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:39:07.364 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:39:07.365 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:39:07.365 DEBUG:boto:Method: GET 2023-11-02T18:39:07.366 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:39:07.366 DEBUG:boto:Data: 2023-11-02T18:39:07.367 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.367 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.368 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.368 DEBUG:boto:Params: {} 2023-11-02T18:39:07.369 DEBUG:boto:Token: None 2023-11-02T18:39:07.369 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj33 2023-11-02T18:39:07.370 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QBAVUzf2kZPmbBnjK06gQHlnb0I= 2023-11-02T18:39:07.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QBAVUzf2kZPmbBnjK06gQHlnb0I=', 'Content-Length': '0'} 2023-11-02T18:39:07.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1914ad36d4d4316-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.374 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:39:07.374 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:39:07.375 DEBUG:boto:Method: GET 2023-11-02T18:39:07.375 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:39:07.376 DEBUG:boto:Data: 2023-11-02T18:39:07.376 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.377 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.377 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.378 DEBUG:boto:Params: {} 2023-11-02T18:39:07.378 DEBUG:boto:Token: None 2023-11-02T18:39:07.379 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj33 2023-11-02T18:39:07.379 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QBAVUzf2kZPmbBnjK06gQHlnb0I= 2023-11-02T18:39:07.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QBAVUzf2kZPmbBnjK06gQHlnb0I=', 'Content-Length': '0'} 2023-11-02T18:39:07.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036abc74634a16941-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.430 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.431 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.431 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:39:07.432 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:39:07.432 DEBUG:boto:Method: GET 2023-11-02T18:39:07.433 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:39:07.433 DEBUG:boto:Data: 2023-11-02T18:39:07.434 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.434 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.435 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.435 DEBUG:boto:Params: {} 2023-11-02T18:39:07.436 DEBUG:boto:Token: None 2023-11-02T18:39:07.436 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.437 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f1a9e06fc909ac2-006543eccb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.443 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.444 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.444 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.445 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:39:07.445 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:39:07.445 DEBUG:boto:Method: GET 2023-11-02T18:39:07.446 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:39:07.446 DEBUG:boto:Data: 2023-11-02T18:39:07.447 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.448 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.448 DEBUG:boto:Params: {} 2023-11-02T18:39:07.448 DEBUG:boto:Token: None 2023-11-02T18:39:07.449 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.449 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fcafb23ef326371-006543eccb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.458 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:39:07.459 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:39:07.459 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:39:07.460 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.460 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.461 DEBUG:boto:Method: GET 2023-11-02T18:39:07.461 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.462 DEBUG:boto:Data: 2023-11-02T18:39:07.462 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.463 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.463 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.463 DEBUG:boto:Params: {} 2023-11-02T18:39:07.464 DEBUG:boto:Token: None 2023-11-02T18:39:07.464 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2dwzSL/AyNDl+Y5gkTaCjzx+HBI= 2023-11-02T18:39:07.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2dwzSL/AyNDl+Y5gkTaCjzx+HBI=', 'Content-Length': '0'} 2023-11-02T18:39:07.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044ff8c9a36d2bfe2-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.510 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:39:07.511 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:39:07.511 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.512 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.512 DEBUG:boto:Method: GET 2023-11-02T18:39:07.513 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.513 DEBUG:boto:Data: 2023-11-02T18:39:07.514 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.514 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.514 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.515 DEBUG:boto:Params: {} 2023-11-02T18:39:07.516 DEBUG:boto:Token: None 2023-11-02T18:39:07.516 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.517 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2dwzSL/AyNDl+Y5gkTaCjzx+HBI= 2023-11-02T18:39:07.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2dwzSL/AyNDl+Y5gkTaCjzx+HBI=', 'Content-Length': '0'} 2023-11-02T18:39:07.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb74c4adf068b1c4-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.521 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:39:07.521 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:39:07.522 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.522 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.523 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.523 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.524 DEBUG:boto:Data: 2023-11-02T18:39:07.524 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.524 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.525 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.525 DEBUG:boto:Params: {} 2023-11-02T18:39:07.526 DEBUG:boto:Token: None 2023-11-02T18:39:07.526 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.527 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9v5Le5BcDkh2FYhxlwitgooiSKU= 2023-11-02T18:39:07.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9v5Le5BcDkh2FYhxlwitgooiSKU=', 'Content-Length': '0'} 2023-11-02T18:39:07.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd741796aba4ecc3-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.531 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:39:07.532 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:39:07.532 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.532 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.533 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.533 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.534 DEBUG:boto:Data: 2023-11-02T18:39:07.534 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.535 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.535 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.535 DEBUG:boto:Params: {} 2023-11-02T18:39:07.536 DEBUG:boto:Token: None 2023-11-02T18:39:07.537 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.537 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9v5Le5BcDkh2FYhxlwitgooiSKU= 2023-11-02T18:39:07.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9v5Le5BcDkh2FYhxlwitgooiSKU=', 'Content-Length': '0'} 2023-11-02T18:39:07.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074b18b9615c1db4c-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:39:07.541 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:39:07.542 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:39:07.542 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.543 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.543 DEBUG:boto:Method: GET 2023-11-02T18:39:07.544 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.544 DEBUG:boto:Data: 2023-11-02T18:39:07.544 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.545 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.545 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.546 DEBUG:boto:Params: {} 2023-11-02T18:39:07.546 DEBUG:boto:Token: None 2023-11-02T18:39:07.547 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.547 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2dwzSL/AyNDl+Y5gkTaCjzx+HBI= 2023-11-02T18:39:07.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2dwzSL/AyNDl+Y5gkTaCjzx+HBI=', 'Content-Length': '0'} 2023-11-02T18:39:07.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a779dc38c4ed4ace-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.551 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:39:07.552 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:39:07.552 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.553 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.553 DEBUG:boto:Method: GET 2023-11-02T18:39:07.553 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.554 DEBUG:boto:Data: 2023-11-02T18:39:07.554 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.555 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.555 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.556 DEBUG:boto:Params: {} 2023-11-02T18:39:07.556 DEBUG:boto:Token: None 2023-11-02T18:39:07.557 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:39:07.557 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2dwzSL/AyNDl+Y5gkTaCjzx+HBI= 2023-11-02T18:39:07.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2dwzSL/AyNDl+Y5gkTaCjzx+HBI=', 'Content-Length': '0'} 2023-11-02T18:39:07.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8b6e4247b9f6ab5-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.561 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.562 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.562 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:39:07.563 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:39:07.563 DEBUG:boto:Method: GET 2023-11-02T18:39:07.564 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:39:07.564 DEBUG:boto:Data: 2023-11-02T18:39:07.564 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.565 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.565 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.566 DEBUG:boto:Params: {} 2023-11-02T18:39:07.566 DEBUG:boto:Token: None 2023-11-02T18:39:07.567 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.567 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b449206e88beb18e-006543eccb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.614 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.615 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.616 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.616 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:39:07.617 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:39:07.617 DEBUG:boto:Method: GET 2023-11-02T18:39:07.617 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:39:07.618 DEBUG:boto:Data: 2023-11-02T18:39:07.618 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.619 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.620 DEBUG:boto:Params: {} 2023-11-02T18:39:07.620 DEBUG:boto:Token: None 2023-11-02T18:39:07.621 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003561df018368f4fc-006543eccb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.627 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.628 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:39:07.628 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:39:07.629 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.629 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:39:07.629 DEBUG:boto:Data: 2023-11-02T18:39:07.630 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.630 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.631 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.631 DEBUG:boto:Params: {} 2023-11-02T18:39:07.632 DEBUG:boto:Token: None 2023-11-02T18:39:07.632 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj34 2023-11-02T18:39:07.633 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2NI55yj8Tp0m15JS8gyPLI0TFhA= 2023-11-02T18:39:07.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2NI55yj8Tp0m15JS8gyPLI0TFhA=', 'Content-Length': '0'} 2023-11-02T18:39:07.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c2c8ef6f241bca-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.636 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:39:07.637 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:39:07.637 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.638 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:39:07.638 DEBUG:boto:Data: 2023-11-02T18:39:07.639 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.639 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.640 DEBUG:boto:Params: {} 2023-11-02T18:39:07.640 DEBUG:boto:Token: None 2023-11-02T18:39:07.641 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj34 2023-11-02T18:39:07.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2NI55yj8Tp0m15JS8gyPLI0TFhA= 2023-11-02T18:39:07.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2NI55yj8Tp0m15JS8gyPLI0TFhA=', 'Content-Length': '0'} 2023-11-02T18:39:07.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc5ab7c9de495360-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:39:07.645 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:39:07.645 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:39:07.646 DEBUG:boto:Method: GET 2023-11-02T18:39:07.646 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:39:07.647 DEBUG:boto:Data: 2023-11-02T18:39:07.647 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.648 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.648 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.649 DEBUG:boto:Params: {} 2023-11-02T18:39:07.649 DEBUG:boto:Token: None 2023-11-02T18:39:07.650 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj34 2023-11-02T18:39:07.650 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DdRJnxmJORcjRlJkN96oUXnGlTk= 2023-11-02T18:39:07.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DdRJnxmJORcjRlJkN96oUXnGlTk=', 'Content-Length': '0'} 2023-11-02T18:39:07.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000352c33207cf122bb-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.654 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:39:07.655 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:39:07.655 DEBUG:boto:Method: GET 2023-11-02T18:39:07.656 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:39:07.656 DEBUG:boto:Data: 2023-11-02T18:39:07.656 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.657 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.657 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.658 DEBUG:boto:Params: {} 2023-11-02T18:39:07.658 DEBUG:boto:Token: None 2023-11-02T18:39:07.659 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj34 2023-11-02T18:39:07.659 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DdRJnxmJORcjRlJkN96oUXnGlTk= 2023-11-02T18:39:07.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DdRJnxmJORcjRlJkN96oUXnGlTk=', 'Content-Length': '0'} 2023-11-02T18:39:07.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003255aa54e2ed727f-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.706 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.707 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.707 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:39:07.708 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:39:07.708 DEBUG:boto:Method: GET 2023-11-02T18:39:07.709 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:39:07.709 DEBUG:boto:Data: 2023-11-02T18:39:07.710 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.710 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.710 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.711 DEBUG:boto:Params: {} 2023-11-02T18:39:07.711 DEBUG:boto:Token: None 2023-11-02T18:39:07.712 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000006f1e807cba9317-006543eccb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.719 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.720 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.720 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.721 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:39:07.721 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:39:07.721 DEBUG:boto:Method: GET 2023-11-02T18:39:07.722 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:39:07.722 DEBUG:boto:Data: 2023-11-02T18:39:07.723 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.723 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.724 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.724 DEBUG:boto:Params: {} 2023-11-02T18:39:07.725 DEBUG:boto:Token: None 2023-11-02T18:39:07.725 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.726 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af47f80cfebf1bbd-006543eccb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.732 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:39:07.733 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:39:07.734 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:39:07.734 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.735 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.735 DEBUG:boto:Method: GET 2023-11-02T18:39:07.736 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.736 DEBUG:boto:Data: 2023-11-02T18:39:07.736 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.737 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.737 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.738 DEBUG:boto:Params: {} 2023-11-02T18:39:07.738 DEBUG:boto:Token: None 2023-11-02T18:39:07.739 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.739 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+5BX4UaAA+D0cKzrrsmeoN4DJaU= 2023-11-02T18:39:07.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+5BX4UaAA+D0cKzrrsmeoN4DJaU=', 'Content-Length': '0'} 2023-11-02T18:39:07.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6461b0ef0203dac-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.790 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:39:07.791 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:39:07.791 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.792 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.792 DEBUG:boto:Method: GET 2023-11-02T18:39:07.793 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.793 DEBUG:boto:Data: 2023-11-02T18:39:07.794 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.794 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.795 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.795 DEBUG:boto:Params: {} 2023-11-02T18:39:07.796 DEBUG:boto:Token: None 2023-11-02T18:39:07.796 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.797 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+5BX4UaAA+D0cKzrrsmeoN4DJaU= 2023-11-02T18:39:07.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+5BX4UaAA+D0cKzrrsmeoN4DJaU=', 'Content-Length': '0'} 2023-11-02T18:39:07.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006082df66de27b25d-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.801 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:39:07.802 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:39:07.802 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.802 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.803 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.803 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.804 DEBUG:boto:Data: 2023-11-02T18:39:07.804 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.805 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.805 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.805 DEBUG:boto:Params: {} 2023-11-02T18:39:07.806 DEBUG:boto:Token: None 2023-11-02T18:39:07.806 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.807 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eQjDUoAW5UlyZiNpcPT5HuJ8anY= 2023-11-02T18:39:07.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eQjDUoAW5UlyZiNpcPT5HuJ8anY=', 'Content-Length': '0'} 2023-11-02T18:39:07.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f96af14fd271f97-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.811 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:39:07.811 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:39:07.812 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.812 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.812 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.813 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.813 DEBUG:boto:Data: 2023-11-02T18:39:07.814 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.814 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.815 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.815 DEBUG:boto:Params: {} 2023-11-02T18:39:07.815 DEBUG:boto:Token: None 2023-11-02T18:39:07.816 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.816 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eQjDUoAW5UlyZiNpcPT5HuJ8anY= 2023-11-02T18:39:07.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eQjDUoAW5UlyZiNpcPT5HuJ8anY=', 'Content-Length': '0'} 2023-11-02T18:39:07.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017bf5441c56c7b16-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:39:07.821 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:39:07.821 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:39:07.822 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.822 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.823 DEBUG:boto:Method: GET 2023-11-02T18:39:07.823 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.824 DEBUG:boto:Data: 2023-11-02T18:39:07.824 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.824 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.825 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.825 DEBUG:boto:Params: {} 2023-11-02T18:39:07.826 DEBUG:boto:Token: None 2023-11-02T18:39:07.826 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.827 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+5BX4UaAA+D0cKzrrsmeoN4DJaU= 2023-11-02T18:39:07.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+5BX4UaAA+D0cKzrrsmeoN4DJaU=', 'Content-Length': '0'} 2023-11-02T18:39:07.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091c809a300c39331-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.831 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:39:07.831 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:39:07.832 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.832 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.833 DEBUG:boto:Method: GET 2023-11-02T18:39:07.833 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.834 DEBUG:boto:Data: 2023-11-02T18:39:07.834 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.835 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.835 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.836 DEBUG:boto:Params: {} 2023-11-02T18:39:07.836 DEBUG:boto:Token: None 2023-11-02T18:39:07.837 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:39:07.837 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+5BX4UaAA+D0cKzrrsmeoN4DJaU= 2023-11-02T18:39:07.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+5BX4UaAA+D0cKzrrsmeoN4DJaU=', 'Content-Length': '0'} 2023-11-02T18:39:07.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000724ee4f6e9fa51e4-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.887 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.887 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.888 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:39:07.888 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:39:07.889 DEBUG:boto:Method: GET 2023-11-02T18:39:07.889 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:39:07.890 DEBUG:boto:Data: 2023-11-02T18:39:07.890 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.891 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.891 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.891 DEBUG:boto:Params: {} 2023-11-02T18:39:07.892 DEBUG:boto:Token: None 2023-11-02T18:39:07.893 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.893 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000360bf0d83bb322d1-006543eccb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.900 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.901 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.901 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.902 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:39:07.902 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:39:07.903 DEBUG:boto:Method: GET 2023-11-02T18:39:07.903 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:39:07.904 DEBUG:boto:Data: 2023-11-02T18:39:07.904 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.904 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.905 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.905 DEBUG:boto:Params: {} 2023-11-02T18:39:07.906 DEBUG:boto:Token: None 2023-11-02T18:39:07.906 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.907 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:07.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000598d1cd3188bb071-006543eccb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.914 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:07.915 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:39:07.915 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:39:07.916 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.916 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:39:07.917 DEBUG:boto:Data: 2023-11-02T18:39:07.917 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.918 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.919 DEBUG:boto:Params: {} 2023-11-02T18:39:07.919 DEBUG:boto:Token: None 2023-11-02T18:39:07.920 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj38 2023-11-02T18:39:07.920 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gTqon42eiY2gu/HXkmaHNyMiZ3Q= 2023-11-02T18:39:07.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gTqon42eiY2gu/HXkmaHNyMiZ3Q=', 'Content-Length': '0'} 2023-11-02T18:39:07.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d77a5e2de79430cf-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.924 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:39:07.924 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:39:07.925 DEBUG:boto:Method: HEAD 2023-11-02T18:39:07.925 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:39:07.926 DEBUG:boto:Data: 2023-11-02T18:39:07.926 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.926 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.927 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.927 DEBUG:boto:Params: {} 2023-11-02T18:39:07.927 DEBUG:boto:Token: None 2023-11-02T18:39:07.928 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj38 2023-11-02T18:39:07.928 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gTqon42eiY2gu/HXkmaHNyMiZ3Q= 2023-11-02T18:39:07.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gTqon42eiY2gu/HXkmaHNyMiZ3Q=', 'Content-Length': '0'} 2023-11-02T18:39:07.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e8cbeb81a4954e2-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:39:07.932 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:39:07.933 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:39:07.933 DEBUG:boto:Method: GET 2023-11-02T18:39:07.934 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:39:07.934 DEBUG:boto:Data: 2023-11-02T18:39:07.935 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.935 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.935 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.936 DEBUG:boto:Params: {} 2023-11-02T18:39:07.936 DEBUG:boto:Token: None 2023-11-02T18:39:07.937 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj38 2023-11-02T18:39:07.937 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rtD9vU9VwSd8myuAJM+t7RrGP1w= 2023-11-02T18:39:07.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rtD9vU9VwSd8myuAJM+t7RrGP1w=', 'Content-Length': '0'} 2023-11-02T18:39:07.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003168285614c060d5-006543eccb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.982 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:39:07.983 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:39:07.983 DEBUG:boto:Method: GET 2023-11-02T18:39:07.984 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:39:07.984 DEBUG:boto:Data: 2023-11-02T18:39:07.985 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.985 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:07.986 DEBUG:boto:Port: 8000 2023-11-02T18:39:07.986 DEBUG:boto:Params: {} 2023-11-02T18:39:07.987 DEBUG:boto:Token: None 2023-11-02T18:39:07.987 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/obj38 2023-11-02T18:39:07.987 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rtD9vU9VwSd8myuAJM+t7RrGP1w= 2023-11-02T18:39:07.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rtD9vU9VwSd8myuAJM+t7RrGP1w=', 'Content-Length': '0'} 2023-11-02T18:39:07.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7f6ce9cb9157e65-006543eccb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:07.992 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:07.992 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:07.993 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:39:07.993 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:39:07.993 DEBUG:boto:Method: GET 2023-11-02T18:39:07.994 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:39:07.994 DEBUG:boto:Data: 2023-11-02T18:39:07.995 DEBUG:boto:Headers: {} 2023-11-02T18:39:07.995 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:07.996 DEBUG:boto:Port: 8001 2023-11-02T18:39:07.996 DEBUG:boto:Params: {} 2023-11-02T18:39:07.997 DEBUG:boto:Token: None 2023-11-02T18:39:07.997 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:07 GMT /ibtljn-59/ 2023-11-02T18:39:07.998 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0= 2023-11-02T18:39:07.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9Lb0AgWNippa75janNiCyfILQQ0=', 'Content-Length': '0'} 2023-11-02T18:39:08.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf9e47b5e275ae59-006543eccb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.003 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.004 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.005 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.005 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:39:08.006 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:39:08.006 DEBUG:boto:Method: GET 2023-11-02T18:39:08.006 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:39:08.007 DEBUG:boto:Data: 2023-11-02T18:39:08.007 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.008 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.008 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.009 DEBUG:boto:Params: {} 2023-11-02T18:39:08.009 DEBUG:boto:Token: None 2023-11-02T18:39:08.010 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.010 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bf28fc133affdf0-006543eccc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.058 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:39:08.060 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:39:08.060 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:39:08.060 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.061 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.061 DEBUG:boto:Method: GET 2023-11-02T18:39:08.062 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.062 DEBUG:boto:Data: 2023-11-02T18:39:08.063 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.063 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.064 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.064 DEBUG:boto:Params: {} 2023-11-02T18:39:08.064 DEBUG:boto:Token: None 2023-11-02T18:39:08.065 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.065 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VMUXHuAjvUU62kW3KTEjjwRPVJk= 2023-11-02T18:39:08.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VMUXHuAjvUU62kW3KTEjjwRPVJk=', 'Content-Length': '0'} 2023-11-02T18:39:08.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063c35d0d4f183b54-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.070 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:39:08.070 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:39:08.071 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.071 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.072 DEBUG:boto:Method: GET 2023-11-02T18:39:08.072 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.073 DEBUG:boto:Data: 2023-11-02T18:39:08.073 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.074 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.074 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.074 DEBUG:boto:Params: {} 2023-11-02T18:39:08.075 DEBUG:boto:Token: None 2023-11-02T18:39:08.075 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.076 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VMUXHuAjvUU62kW3KTEjjwRPVJk= 2023-11-02T18:39:08.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VMUXHuAjvUU62kW3KTEjjwRPVJk=', 'Content-Length': '0'} 2023-11-02T18:39:08.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000944c9d5812abe718-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.081 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:39:08.081 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:39:08.082 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.082 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.085 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.085 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.086 DEBUG:boto:Data: 2023-11-02T18:39:08.086 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.087 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.087 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.088 DEBUG:boto:Params: {} 2023-11-02T18:39:08.089 DEBUG:boto:Token: None 2023-11-02T18:39:08.089 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.090 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xzFp8MjQV+w8mR0mSDNr+u1xymY= 2023-11-02T18:39:08.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xzFp8MjQV+w8mR0mSDNr+u1xymY=', 'Content-Length': '0'} 2023-11-02T18:39:08.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e2c2275ee86611-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.094 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:39:08.094 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:39:08.095 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.095 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.096 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.096 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.096 DEBUG:boto:Data: 2023-11-02T18:39:08.097 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.097 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.098 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.098 DEBUG:boto:Params: {} 2023-11-02T18:39:08.099 DEBUG:boto:Token: None 2023-11-02T18:39:08.099 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.100 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xzFp8MjQV+w8mR0mSDNr+u1xymY= 2023-11-02T18:39:08.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xzFp8MjQV+w8mR0mSDNr+u1xymY=', 'Content-Length': '0'} 2023-11-02T18:39:08.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005096677a426ef22b-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:39:08.104 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:39:08.105 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:39:08.105 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.106 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.106 DEBUG:boto:Method: GET 2023-11-02T18:39:08.106 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.107 DEBUG:boto:Data: 2023-11-02T18:39:08.107 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.108 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.108 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.109 DEBUG:boto:Params: {} 2023-11-02T18:39:08.109 DEBUG:boto:Token: None 2023-11-02T18:39:08.110 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.110 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VMUXHuAjvUU62kW3KTEjjwRPVJk= 2023-11-02T18:39:08.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VMUXHuAjvUU62kW3KTEjjwRPVJk=', 'Content-Length': '0'} 2023-11-02T18:39:08.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c1c529c5e76b7b1-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.115 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:39:08.115 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:39:08.115 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.116 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.116 DEBUG:boto:Method: GET 2023-11-02T18:39:08.117 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.117 DEBUG:boto:Data: 2023-11-02T18:39:08.118 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.118 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.119 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.119 DEBUG:boto:Params: {} 2023-11-02T18:39:08.120 DEBUG:boto:Token: None 2023-11-02T18:39:08.120 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:39:08.121 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VMUXHuAjvUU62kW3KTEjjwRPVJk= 2023-11-02T18:39:08.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VMUXHuAjvUU62kW3KTEjjwRPVJk=', 'Content-Length': '0'} 2023-11-02T18:39:08.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002dfaf9fcb87a733-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.170 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.171 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.171 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:39:08.172 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:39:08.172 DEBUG:boto:Method: GET 2023-11-02T18:39:08.173 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:39:08.173 DEBUG:boto:Data: 2023-11-02T18:39:08.174 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.174 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.174 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.175 DEBUG:boto:Params: {} 2023-11-02T18:39:08.175 DEBUG:boto:Token: None 2023-11-02T18:39:08.176 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.176 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb9dc3db16d70fa6-006543eccc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.183 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.183 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.184 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.184 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:39:08.185 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:39:08.185 DEBUG:boto:Method: GET 2023-11-02T18:39:08.186 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:39:08.186 DEBUG:boto:Data: 2023-11-02T18:39:08.187 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.187 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.187 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.188 DEBUG:boto:Params: {} 2023-11-02T18:39:08.188 DEBUG:boto:Token: None 2023-11-02T18:39:08.189 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.189 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c761a2e39896dd43-006543eccc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.197 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.198 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:39:08.198 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:39:08.199 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.199 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:39:08.200 DEBUG:boto:Data: 2023-11-02T18:39:08.200 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.200 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.201 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.201 DEBUG:boto:Params: {} 2023-11-02T18:39:08.202 DEBUG:boto:Token: None 2023-11-02T18:39:08.202 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj44 2023-11-02T18:39:08.203 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qDfacF+6BXt9qc611ah2SoonsFY= 2023-11-02T18:39:08.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qDfacF+6BXt9qc611ah2SoonsFY=', 'Content-Length': '0'} 2023-11-02T18:39:08.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033e304e9ab23a083-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.206 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:39:08.207 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:39:08.207 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.208 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:39:08.208 DEBUG:boto:Data: 2023-11-02T18:39:08.209 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.209 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.209 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.210 DEBUG:boto:Params: {} 2023-11-02T18:39:08.210 DEBUG:boto:Token: None 2023-11-02T18:39:08.211 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj44 2023-11-02T18:39:08.211 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qDfacF+6BXt9qc611ah2SoonsFY= 2023-11-02T18:39:08.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qDfacF+6BXt9qc611ah2SoonsFY=', 'Content-Length': '0'} 2023-11-02T18:39:08.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9dfe42cac7e7c64-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:39:08.216 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:39:08.216 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:39:08.217 DEBUG:boto:Method: GET 2023-11-02T18:39:08.217 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:39:08.218 DEBUG:boto:Data: 2023-11-02T18:39:08.218 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.218 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.219 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.219 DEBUG:boto:Params: {} 2023-11-02T18:39:08.220 DEBUG:boto:Token: None 2023-11-02T18:39:08.220 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj44 2023-11-02T18:39:08.221 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sDi5dE336FOs0Z/kNJegrLcw//4= 2023-11-02T18:39:08.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sDi5dE336FOs0Z/kNJegrLcw//4=', 'Content-Length': '0'} 2023-11-02T18:39:08.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7d64e2badf69b03-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.225 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:39:08.225 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:39:08.226 DEBUG:boto:Method: GET 2023-11-02T18:39:08.226 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:39:08.227 DEBUG:boto:Data: 2023-11-02T18:39:08.227 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.228 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.228 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.229 DEBUG:boto:Params: {} 2023-11-02T18:39:08.229 DEBUG:boto:Token: None 2023-11-02T18:39:08.230 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj44 2023-11-02T18:39:08.230 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sDi5dE336FOs0Z/kNJegrLcw//4= 2023-11-02T18:39:08.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sDi5dE336FOs0Z/kNJegrLcw//4=', 'Content-Length': '0'} 2023-11-02T18:39:08.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cd3055910be20a7-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.278 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.279 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.279 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:39:08.280 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:39:08.280 DEBUG:boto:Method: GET 2023-11-02T18:39:08.281 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:39:08.281 DEBUG:boto:Data: 2023-11-02T18:39:08.282 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.282 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.283 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.283 DEBUG:boto:Params: {} 2023-11-02T18:39:08.284 DEBUG:boto:Token: None 2023-11-02T18:39:08.284 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.285 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edcc5cf5d0a8872a-006543eccc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.293 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.294 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.294 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.295 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:39:08.295 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:39:08.296 DEBUG:boto:Method: GET 2023-11-02T18:39:08.296 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:39:08.296 DEBUG:boto:Data: 2023-11-02T18:39:08.297 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.298 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.298 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.298 DEBUG:boto:Params: {} 2023-11-02T18:39:08.299 DEBUG:boto:Token: None 2023-11-02T18:39:08.299 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.300 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021da352e724a8f50-006543eccc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.308 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:39:08.309 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:39:08.309 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:39:08.310 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.310 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.311 DEBUG:boto:Method: GET 2023-11-02T18:39:08.311 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.312 DEBUG:boto:Data: 2023-11-02T18:39:08.312 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.313 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.313 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.314 DEBUG:boto:Params: {} 2023-11-02T18:39:08.314 DEBUG:boto:Token: None 2023-11-02T18:39:08.315 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.315 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fRfX33SNWTFphCssqZ7/gQRZeEc= 2023-11-02T18:39:08.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fRfX33SNWTFphCssqZ7/gQRZeEc=', 'Content-Length': '0'} 2023-11-02T18:39:08.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe207bf494d60aa0-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.366 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:39:08.367 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:39:08.367 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.368 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.368 DEBUG:boto:Method: GET 2023-11-02T18:39:08.369 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.369 DEBUG:boto:Data: 2023-11-02T18:39:08.370 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.371 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.372 DEBUG:boto:Params: {} 2023-11-02T18:39:08.373 DEBUG:boto:Token: None 2023-11-02T18:39:08.373 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.374 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fRfX33SNWTFphCssqZ7/gQRZeEc= 2023-11-02T18:39:08.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fRfX33SNWTFphCssqZ7/gQRZeEc=', 'Content-Length': '0'} 2023-11-02T18:39:08.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9519b48f4109d17-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.378 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:39:08.379 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:39:08.379 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.380 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.380 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.380 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.381 DEBUG:boto:Data: 2023-11-02T18:39:08.381 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.382 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.382 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.383 DEBUG:boto:Params: {} 2023-11-02T18:39:08.383 DEBUG:boto:Token: None 2023-11-02T18:39:08.384 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.384 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yPq/N4PZmJvkkDsuHaKWJJPFJ2U= 2023-11-02T18:39:08.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yPq/N4PZmJvkkDsuHaKWJJPFJ2U=', 'Content-Length': '0'} 2023-11-02T18:39:08.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052c100a8af21c5af-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.388 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:39:08.389 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:39:08.389 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.390 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.390 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.391 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.391 DEBUG:boto:Data: 2023-11-02T18:39:08.392 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.392 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.392 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.393 DEBUG:boto:Params: {} 2023-11-02T18:39:08.393 DEBUG:boto:Token: None 2023-11-02T18:39:08.394 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yPq/N4PZmJvkkDsuHaKWJJPFJ2U= 2023-11-02T18:39:08.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yPq/N4PZmJvkkDsuHaKWJJPFJ2U=', 'Content-Length': '0'} 2023-11-02T18:39:08.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9fe3f949e0b102a-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:39:08.399 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:39:08.399 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:39:08.400 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.400 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.401 DEBUG:boto:Method: GET 2023-11-02T18:39:08.401 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.402 DEBUG:boto:Data: 2023-11-02T18:39:08.402 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.403 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.404 DEBUG:boto:Params: {} 2023-11-02T18:39:08.404 DEBUG:boto:Token: None 2023-11-02T18:39:08.405 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fRfX33SNWTFphCssqZ7/gQRZeEc= 2023-11-02T18:39:08.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fRfX33SNWTFphCssqZ7/gQRZeEc=', 'Content-Length': '0'} 2023-11-02T18:39:08.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c8f3be0cd0c3374-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.410 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:39:08.410 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:39:08.411 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.411 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.411 DEBUG:boto:Method: GET 2023-11-02T18:39:08.412 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.412 DEBUG:boto:Data: 2023-11-02T18:39:08.413 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.413 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.414 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.414 DEBUG:boto:Params: {} 2023-11-02T18:39:08.415 DEBUG:boto:Token: None 2023-11-02T18:39:08.415 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:39:08.416 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fRfX33SNWTFphCssqZ7/gQRZeEc= 2023-11-02T18:39:08.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fRfX33SNWTFphCssqZ7/gQRZeEc=', 'Content-Length': '0'} 2023-11-02T18:39:08.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000079f1bb71f2cf15d-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.420 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.421 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.421 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:39:08.422 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:39:08.422 DEBUG:boto:Method: GET 2023-11-02T18:39:08.423 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:39:08.423 DEBUG:boto:Data: 2023-11-02T18:39:08.424 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.424 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.425 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.425 DEBUG:boto:Params: {} 2023-11-02T18:39:08.426 DEBUG:boto:Token: None 2023-11-02T18:39:08.426 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.427 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df886fae6e9a09b7-006543eccc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.435 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.436 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.436 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.437 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:39:08.438 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:39:08.438 DEBUG:boto:Method: GET 2023-11-02T18:39:08.439 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:39:08.439 DEBUG:boto:Data: 2023-11-02T18:39:08.440 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.440 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.441 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.442 DEBUG:boto:Params: {} 2023-11-02T18:39:08.442 DEBUG:boto:Token: None 2023-11-02T18:39:08.443 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef2403640e28ee56-006543eccc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.453 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.454 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:39:08.455 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:39:08.455 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.456 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:39:08.457 DEBUG:boto:Data: 2023-11-02T18:39:08.457 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.458 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.459 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.459 DEBUG:boto:Params: {} 2023-11-02T18:39:08.460 DEBUG:boto:Token: None 2023-11-02T18:39:08.460 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj51 2023-11-02T18:39:08.461 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QhCci/KtQcKkurVWQqJUVe1lHRI= 2023-11-02T18:39:08.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QhCci/KtQcKkurVWQqJUVe1lHRI=', 'Content-Length': '0'} 2023-11-02T18:39:08.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac19f2140662f380-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.465 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:39:08.466 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:39:08.466 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.467 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:39:08.467 DEBUG:boto:Data: 2023-11-02T18:39:08.468 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.469 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.469 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.469 DEBUG:boto:Params: {} 2023-11-02T18:39:08.470 DEBUG:boto:Token: None 2023-11-02T18:39:08.471 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj51 2023-11-02T18:39:08.471 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QhCci/KtQcKkurVWQqJUVe1lHRI= 2023-11-02T18:39:08.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QhCci/KtQcKkurVWQqJUVe1lHRI=', 'Content-Length': '0'} 2023-11-02T18:39:08.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eadd3ed29b1da364-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:39:08.476 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:39:08.476 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:39:08.477 DEBUG:boto:Method: GET 2023-11-02T18:39:08.478 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:39:08.478 DEBUG:boto:Data: 2023-11-02T18:39:08.479 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.479 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.480 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.480 DEBUG:boto:Params: {} 2023-11-02T18:39:08.481 DEBUG:boto:Token: None 2023-11-02T18:39:08.482 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj51 2023-11-02T18:39:08.482 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zKeU2Y4E7IP+4pvTNwY8IkaVtls= 2023-11-02T18:39:08.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zKeU2Y4E7IP+4pvTNwY8IkaVtls=', 'Content-Length': '0'} 2023-11-02T18:39:08.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f71fd48eb920131-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.526 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:39:08.527 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:39:08.527 DEBUG:boto:Method: GET 2023-11-02T18:39:08.528 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:39:08.528 DEBUG:boto:Data: 2023-11-02T18:39:08.529 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.529 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.530 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.530 DEBUG:boto:Params: {} 2023-11-02T18:39:08.531 DEBUG:boto:Token: None 2023-11-02T18:39:08.531 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj51 2023-11-02T18:39:08.532 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zKeU2Y4E7IP+4pvTNwY8IkaVtls= 2023-11-02T18:39:08.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zKeU2Y4E7IP+4pvTNwY8IkaVtls=', 'Content-Length': '0'} 2023-11-02T18:39:08.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000526c64cc4aeb23c1-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.536 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.537 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.537 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:39:08.538 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:39:08.538 DEBUG:boto:Method: GET 2023-11-02T18:39:08.538 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:39:08.539 DEBUG:boto:Data: 2023-11-02T18:39:08.539 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.540 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.540 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.541 DEBUG:boto:Params: {} 2023-11-02T18:39:08.541 DEBUG:boto:Token: None 2023-11-02T18:39:08.542 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.542 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa178734d50106fc-006543eccc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.549 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.550 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.550 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.551 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:39:08.551 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:39:08.552 DEBUG:boto:Method: GET 2023-11-02T18:39:08.552 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:39:08.553 DEBUG:boto:Data: 2023-11-02T18:39:08.553 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.554 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.554 DEBUG:boto:Params: {} 2023-11-02T18:39:08.555 DEBUG:boto:Token: None 2023-11-02T18:39:08.556 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009831cccafa337b55-006543eccc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.564 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:39:08.565 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:39:08.565 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:39:08.566 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.566 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.567 DEBUG:boto:Method: GET 2023-11-02T18:39:08.567 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.567 DEBUG:boto:Data: 2023-11-02T18:39:08.568 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.568 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.569 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.569 DEBUG:boto:Params: {} 2023-11-02T18:39:08.570 DEBUG:boto:Token: None 2023-11-02T18:39:08.570 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.571 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QA2YsOmQy7nEfPkCMnfQwmPhfx8= 2023-11-02T18:39:08.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QA2YsOmQy7nEfPkCMnfQwmPhfx8=', 'Content-Length': '0'} 2023-11-02T18:39:08.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075c949ed8d05a836-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.575 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:39:08.575 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:39:08.576 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.576 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.577 DEBUG:boto:Method: GET 2023-11-02T18:39:08.577 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.577 DEBUG:boto:Data: 2023-11-02T18:39:08.578 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.578 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.579 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.579 DEBUG:boto:Params: {} 2023-11-02T18:39:08.580 DEBUG:boto:Token: None 2023-11-02T18:39:08.580 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.581 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QA2YsOmQy7nEfPkCMnfQwmPhfx8= 2023-11-02T18:39:08.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QA2YsOmQy7nEfPkCMnfQwmPhfx8=', 'Content-Length': '0'} 2023-11-02T18:39:08.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff93a9e070ad23c-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.585 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:39:08.585 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:39:08.586 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.586 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.587 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.587 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.587 DEBUG:boto:Data: 2023-11-02T18:39:08.588 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.588 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.589 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.589 DEBUG:boto:Params: {} 2023-11-02T18:39:08.590 DEBUG:boto:Token: None 2023-11-02T18:39:08.590 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.591 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YEY/F7jC3EnZ1gpl2u/OlNmQNtU= 2023-11-02T18:39:08.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YEY/F7jC3EnZ1gpl2u/OlNmQNtU=', 'Content-Length': '0'} 2023-11-02T18:39:08.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081741e51d716c862-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.595 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:39:08.595 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:39:08.596 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.596 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.596 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.597 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.597 DEBUG:boto:Data: 2023-11-02T18:39:08.598 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.598 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.599 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.599 DEBUG:boto:Params: {} 2023-11-02T18:39:08.600 DEBUG:boto:Token: None 2023-11-02T18:39:08.600 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YEY/F7jC3EnZ1gpl2u/OlNmQNtU= 2023-11-02T18:39:08.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YEY/F7jC3EnZ1gpl2u/OlNmQNtU=', 'Content-Length': '0'} 2023-11-02T18:39:08.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b784eb8a10f67d52-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:39:08.605 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:39:08.605 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:39:08.606 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.606 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.607 DEBUG:boto:Method: GET 2023-11-02T18:39:08.607 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.608 DEBUG:boto:Data: 2023-11-02T18:39:08.608 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.609 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.609 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.610 DEBUG:boto:Params: {} 2023-11-02T18:39:08.610 DEBUG:boto:Token: None 2023-11-02T18:39:08.611 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.611 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QA2YsOmQy7nEfPkCMnfQwmPhfx8= 2023-11-02T18:39:08.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QA2YsOmQy7nEfPkCMnfQwmPhfx8=', 'Content-Length': '0'} 2023-11-02T18:39:08.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6baa076321282e9-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.658 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:39:08.659 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:39:08.660 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.660 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.660 DEBUG:boto:Method: GET 2023-11-02T18:39:08.661 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.661 DEBUG:boto:Data: 2023-11-02T18:39:08.662 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.662 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.663 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.663 DEBUG:boto:Params: {} 2023-11-02T18:39:08.664 DEBUG:boto:Token: None 2023-11-02T18:39:08.664 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:39:08.665 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QA2YsOmQy7nEfPkCMnfQwmPhfx8= 2023-11-02T18:39:08.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QA2YsOmQy7nEfPkCMnfQwmPhfx8=', 'Content-Length': '0'} 2023-11-02T18:39:08.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3d36f6bad24fd37-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.669 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.670 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.670 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:39:08.671 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:39:08.671 DEBUG:boto:Method: GET 2023-11-02T18:39:08.672 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:39:08.672 DEBUG:boto:Data: 2023-11-02T18:39:08.673 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.673 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.673 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.674 DEBUG:boto:Params: {} 2023-11-02T18:39:08.674 DEBUG:boto:Token: None 2023-11-02T18:39:08.675 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.675 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000069e20aed46f0660-006543eccc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.681 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.682 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.683 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.683 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:39:08.684 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:39:08.684 DEBUG:boto:Method: GET 2023-11-02T18:39:08.684 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:39:08.685 DEBUG:boto:Data: 2023-11-02T18:39:08.685 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.686 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.687 DEBUG:boto:Params: {} 2023-11-02T18:39:08.687 DEBUG:boto:Token: None 2023-11-02T18:39:08.688 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.688 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7219a7fa1c9b6fb-006543eccc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.697 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.698 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:39:08.699 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:39:08.699 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.699 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:39:08.700 DEBUG:boto:Data: 2023-11-02T18:39:08.700 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.701 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.701 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.702 DEBUG:boto:Params: {} 2023-11-02T18:39:08.702 DEBUG:boto:Token: None 2023-11-02T18:39:08.703 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj55 2023-11-02T18:39:08.703 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sofid4EfUcJ31MSa972twS/XLwQ= 2023-11-02T18:39:08.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sofid4EfUcJ31MSa972twS/XLwQ=', 'Content-Length': '0'} 2023-11-02T18:39:08.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a2e264e6ef303c2-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.707 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:39:08.707 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:39:08.707 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.708 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:39:08.708 DEBUG:boto:Data: 2023-11-02T18:39:08.709 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.709 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.710 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.710 DEBUG:boto:Params: {} 2023-11-02T18:39:08.711 DEBUG:boto:Token: None 2023-11-02T18:39:08.711 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj55 2023-11-02T18:39:08.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sofid4EfUcJ31MSa972twS/XLwQ= 2023-11-02T18:39:08.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sofid4EfUcJ31MSa972twS/XLwQ=', 'Content-Length': '0'} 2023-11-02T18:39:08.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b6fccb5241477f2-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:39:08.716 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:39:08.716 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:39:08.717 DEBUG:boto:Method: GET 2023-11-02T18:39:08.717 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:39:08.717 DEBUG:boto:Data: 2023-11-02T18:39:08.718 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.718 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.719 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.719 DEBUG:boto:Params: {} 2023-11-02T18:39:08.720 DEBUG:boto:Token: None 2023-11-02T18:39:08.720 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj55 2023-11-02T18:39:08.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KMyQ+cLwRDIhWEoF2hD/JidxH6k= 2023-11-02T18:39:08.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KMyQ+cLwRDIhWEoF2hD/JidxH6k=', 'Content-Length': '0'} 2023-11-02T18:39:08.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caf02e9a1149a72a-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.725 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:39:08.725 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:39:08.726 DEBUG:boto:Method: GET 2023-11-02T18:39:08.726 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:39:08.726 DEBUG:boto:Data: 2023-11-02T18:39:08.727 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.727 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.728 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.728 DEBUG:boto:Params: {} 2023-11-02T18:39:08.729 DEBUG:boto:Token: None 2023-11-02T18:39:08.729 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj55 2023-11-02T18:39:08.730 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KMyQ+cLwRDIhWEoF2hD/JidxH6k= 2023-11-02T18:39:08.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KMyQ+cLwRDIhWEoF2hD/JidxH6k=', 'Content-Length': '0'} 2023-11-02T18:39:08.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081eab8c9aa6ad78c-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.734 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.734 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.735 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:39:08.735 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:39:08.736 DEBUG:boto:Method: GET 2023-11-02T18:39:08.736 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:39:08.736 DEBUG:boto:Data: 2023-11-02T18:39:08.737 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.737 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.738 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.738 DEBUG:boto:Params: {} 2023-11-02T18:39:08.739 DEBUG:boto:Token: None 2023-11-02T18:39:08.739 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd2e2a004ed934e2-006543eccc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.746 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.747 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.747 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.748 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:39:08.748 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:39:08.749 DEBUG:boto:Method: GET 2023-11-02T18:39:08.749 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:39:08.750 DEBUG:boto:Data: 2023-11-02T18:39:08.750 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.751 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.751 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.752 DEBUG:boto:Params: {} 2023-11-02T18:39:08.752 DEBUG:boto:Token: None 2023-11-02T18:39:08.753 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.753 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c625783df35fd7a8-006543eccc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.761 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:39:08.762 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:39:08.763 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:39:08.763 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.763 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.764 DEBUG:boto:Method: GET 2023-11-02T18:39:08.764 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.765 DEBUG:boto:Data: 2023-11-02T18:39:08.765 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.765 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.766 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.766 DEBUG:boto:Params: {} 2023-11-02T18:39:08.767 DEBUG:boto:Token: None 2023-11-02T18:39:08.767 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QaT7Dnugh7TFIYNgqGRG5bgjw8Y= 2023-11-02T18:39:08.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QaT7Dnugh7TFIYNgqGRG5bgjw8Y=', 'Content-Length': '0'} 2023-11-02T18:39:08.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2b07e2017922d70-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.814 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:39:08.815 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:39:08.815 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.816 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.816 DEBUG:boto:Method: GET 2023-11-02T18:39:08.817 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.817 DEBUG:boto:Data: 2023-11-02T18:39:08.818 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.818 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.818 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.819 DEBUG:boto:Params: {} 2023-11-02T18:39:08.819 DEBUG:boto:Token: None 2023-11-02T18:39:08.820 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.820 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QaT7Dnugh7TFIYNgqGRG5bgjw8Y= 2023-11-02T18:39:08.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QaT7Dnugh7TFIYNgqGRG5bgjw8Y=', 'Content-Length': '0'} 2023-11-02T18:39:08.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a4d61cb9387f60-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.825 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:39:08.825 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:39:08.826 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.826 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.826 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.827 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.827 DEBUG:boto:Data: 2023-11-02T18:39:08.828 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.828 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.829 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.829 DEBUG:boto:Params: {} 2023-11-02T18:39:08.830 DEBUG:boto:Token: None 2023-11-02T18:39:08.831 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.832 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8fsyHeHbO4ktZiQuiUxXYqNfv2g= 2023-11-02T18:39:08.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8fsyHeHbO4ktZiQuiUxXYqNfv2g=', 'Content-Length': '0'} 2023-11-02T18:39:08.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003abefd38b6ca1d08-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.836 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:39:08.836 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:39:08.837 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.837 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.838 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.838 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.839 DEBUG:boto:Data: 2023-11-02T18:39:08.839 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.839 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.840 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.840 DEBUG:boto:Params: {} 2023-11-02T18:39:08.841 DEBUG:boto:Token: None 2023-11-02T18:39:08.841 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.842 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8fsyHeHbO4ktZiQuiUxXYqNfv2g= 2023-11-02T18:39:08.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8fsyHeHbO4ktZiQuiUxXYqNfv2g=', 'Content-Length': '0'} 2023-11-02T18:39:08.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e6f92ada45033a3-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:39:08.846 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:39:08.846 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:39:08.847 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.847 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.848 DEBUG:boto:Method: GET 2023-11-02T18:39:08.848 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.849 DEBUG:boto:Data: 2023-11-02T18:39:08.849 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.849 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.850 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.850 DEBUG:boto:Params: {} 2023-11-02T18:39:08.851 DEBUG:boto:Token: None 2023-11-02T18:39:08.852 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.852 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QaT7Dnugh7TFIYNgqGRG5bgjw8Y= 2023-11-02T18:39:08.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QaT7Dnugh7TFIYNgqGRG5bgjw8Y=', 'Content-Length': '0'} 2023-11-02T18:39:08.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000931f46f9085e516d-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.857 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:39:08.858 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:39:08.858 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.859 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.859 DEBUG:boto:Method: GET 2023-11-02T18:39:08.859 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.860 DEBUG:boto:Data: 2023-11-02T18:39:08.860 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.861 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.861 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.862 DEBUG:boto:Params: {} 2023-11-02T18:39:08.862 DEBUG:boto:Token: None 2023-11-02T18:39:08.863 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:39:08.863 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QaT7Dnugh7TFIYNgqGRG5bgjw8Y= 2023-11-02T18:39:08.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QaT7Dnugh7TFIYNgqGRG5bgjw8Y=', 'Content-Length': '0'} 2023-11-02T18:39:08.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003cb43241c1d19f2-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.910 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.911 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.911 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:39:08.912 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:39:08.912 DEBUG:boto:Method: GET 2023-11-02T18:39:08.913 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:39:08.913 DEBUG:boto:Data: 2023-11-02T18:39:08.914 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.914 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.915 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.915 DEBUG:boto:Params: {} 2023-11-02T18:39:08.916 DEBUG:boto:Token: None 2023-11-02T18:39:08.916 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.917 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58cea5f16ad23df-006543eccc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.923 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.924 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:08.924 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:08.924 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:39:08.925 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:39:08.925 DEBUG:boto:Method: GET 2023-11-02T18:39:08.926 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:39:08.926 DEBUG:boto:Data: 2023-11-02T18:39:08.927 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.927 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.928 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.928 DEBUG:boto:Params: {} 2023-11-02T18:39:08.928 DEBUG:boto:Token: None 2023-11-02T18:39:08.929 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/ 2023-11-02T18:39:08.929 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8= 2023-11-02T18:39:08.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kNWucP58uEFQzn3L8BwU84Bosh8=', 'Content-Length': '0'} 2023-11-02T18:39:08.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000589932a301336d97-006543eccc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.937 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:08.938 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:39:08.938 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:39:08.939 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.939 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:39:08.939 DEBUG:boto:Data: 2023-11-02T18:39:08.940 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.940 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.941 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.941 DEBUG:boto:Params: {} 2023-11-02T18:39:08.942 DEBUG:boto:Token: None 2023-11-02T18:39:08.942 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj57 2023-11-02T18:39:08.942 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DWIbYweEb9mAx1JV75D468vk2Co= 2023-11-02T18:39:08.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DWIbYweEb9mAx1JV75D468vk2Co=', 'Content-Length': '0'} 2023-11-02T18:39:08.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094eb5b127ba52f24-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.946 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:39:08.947 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:39:08.947 DEBUG:boto:Method: HEAD 2023-11-02T18:39:08.948 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:39:08.948 DEBUG:boto:Data: 2023-11-02T18:39:08.949 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.949 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:08.949 DEBUG:boto:Port: 8000 2023-11-02T18:39:08.950 DEBUG:boto:Params: {} 2023-11-02T18:39:08.950 DEBUG:boto:Token: None 2023-11-02T18:39:08.951 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj57 2023-11-02T18:39:08.951 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DWIbYweEb9mAx1JV75D468vk2Co= 2023-11-02T18:39:08.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DWIbYweEb9mAx1JV75D468vk2Co=', 'Content-Length': '0'} 2023-11-02T18:39:08.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f22e51bbb54a437-006543eccc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:08.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:39:08.956 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:39:08.956 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:39:08.956 DEBUG:boto:Method: GET 2023-11-02T18:39:08.957 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:39:08.957 DEBUG:boto:Data: 2023-11-02T18:39:08.958 DEBUG:boto:Headers: {} 2023-11-02T18:39:08.958 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:08.959 DEBUG:boto:Port: 8001 2023-11-02T18:39:08.959 DEBUG:boto:Params: {} 2023-11-02T18:39:08.960 DEBUG:boto:Token: None 2023-11-02T18:39:08.960 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:08 GMT /ibtljn-59/obj57 2023-11-02T18:39:08.961 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5qehjZ3DGpF2qgX4XcUo5Qa1z0Q= 2023-11-02T18:39:08.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5qehjZ3DGpF2qgX4XcUo5Qa1z0Q=', 'Content-Length': '0'} 2023-11-02T18:39:08.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000452e06380c7dc1be-006543eccc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.006 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:39:09.007 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:39:09.007 DEBUG:boto:Method: GET 2023-11-02T18:39:09.008 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:39:09.008 DEBUG:boto:Data: 2023-11-02T18:39:09.009 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.009 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.010 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.010 DEBUG:boto:Params: {} 2023-11-02T18:39:09.011 DEBUG:boto:Token: None 2023-11-02T18:39:09.011 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj57 2023-11-02T18:39:09.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+SANvzaHZWf86WQG21ZdeOCEL9g= 2023-11-02T18:39:09.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+SANvzaHZWf86WQG21ZdeOCEL9g=', 'Content-Length': '0'} 2023-11-02T18:39:09.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a61a781d8f3b75a-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.016 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.017 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.017 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:39:09.018 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:39:09.018 DEBUG:boto:Method: GET 2023-11-02T18:39:09.018 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:39:09.019 DEBUG:boto:Data: 2023-11-02T18:39:09.019 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.020 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.020 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.021 DEBUG:boto:Params: {} 2023-11-02T18:39:09.021 DEBUG:boto:Token: None 2023-11-02T18:39:09.022 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.022 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001890d8b838c569ae-006543eccd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.028 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.029 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.029 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.030 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:39:09.030 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:39:09.031 DEBUG:boto:Method: GET 2023-11-02T18:39:09.031 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:39:09.032 DEBUG:boto:Data: 2023-11-02T18:39:09.032 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.033 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.033 DEBUG:boto:Params: {} 2023-11-02T18:39:09.034 DEBUG:boto:Token: None 2023-11-02T18:39:09.034 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000682469a43c972878-006543eccd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.086 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:39:09.090 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:39:09.090 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:39:09.091 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.091 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.091 DEBUG:boto:Method: GET 2023-11-02T18:39:09.092 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.092 DEBUG:boto:Data: 2023-11-02T18:39:09.093 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.093 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.094 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.094 DEBUG:boto:Params: {} 2023-11-02T18:39:09.095 DEBUG:boto:Token: None 2023-11-02T18:39:09.095 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.096 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WpldC72j9JHAcS5KOSfRATYnI9E= 2023-11-02T18:39:09.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WpldC72j9JHAcS5KOSfRATYnI9E=', 'Content-Length': '0'} 2023-11-02T18:39:09.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcc0911219ea3c3e-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.100 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:39:09.101 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:39:09.101 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.102 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.102 DEBUG:boto:Method: GET 2023-11-02T18:39:09.102 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.103 DEBUG:boto:Data: 2023-11-02T18:39:09.103 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.104 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.104 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.105 DEBUG:boto:Params: {} 2023-11-02T18:39:09.105 DEBUG:boto:Token: None 2023-11-02T18:39:09.106 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.106 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WpldC72j9JHAcS5KOSfRATYnI9E= 2023-11-02T18:39:09.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WpldC72j9JHAcS5KOSfRATYnI9E=', 'Content-Length': '0'} 2023-11-02T18:39:09.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d71ba6275410a4b-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.111 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:39:09.111 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:39:09.112 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.114 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.114 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.115 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.115 DEBUG:boto:Data: 2023-11-02T18:39:09.116 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.116 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.117 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.117 DEBUG:boto:Params: {} 2023-11-02T18:39:09.118 DEBUG:boto:Token: None 2023-11-02T18:39:09.118 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.119 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9jKelZST8qEQ9T8t+dCam4ekZYs= 2023-11-02T18:39:09.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9jKelZST8qEQ9T8t+dCam4ekZYs=', 'Content-Length': '0'} 2023-11-02T18:39:09.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd375f5d87d23f65-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.123 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:39:09.123 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:39:09.124 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.124 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.125 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.125 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.126 DEBUG:boto:Data: 2023-11-02T18:39:09.126 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.126 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.127 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.127 DEBUG:boto:Params: {} 2023-11-02T18:39:09.128 DEBUG:boto:Token: None 2023-11-02T18:39:09.129 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9jKelZST8qEQ9T8t+dCam4ekZYs= 2023-11-02T18:39:09.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9jKelZST8qEQ9T8t+dCam4ekZYs=', 'Content-Length': '0'} 2023-11-02T18:39:09.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dcc5a59cf4ede4a-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:39:09.134 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:39:09.134 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:39:09.135 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.135 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.136 DEBUG:boto:Method: GET 2023-11-02T18:39:09.136 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.137 DEBUG:boto:Data: 2023-11-02T18:39:09.137 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.138 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.138 DEBUG:boto:Params: {} 2023-11-02T18:39:09.139 DEBUG:boto:Token: None 2023-11-02T18:39:09.139 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WpldC72j9JHAcS5KOSfRATYnI9E= 2023-11-02T18:39:09.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WpldC72j9JHAcS5KOSfRATYnI9E=', 'Content-Length': '0'} 2023-11-02T18:39:09.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd69a57c703beead-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.144 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:39:09.145 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:39:09.145 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.146 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.146 DEBUG:boto:Method: GET 2023-11-02T18:39:09.147 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.147 DEBUG:boto:Data: 2023-11-02T18:39:09.147 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.148 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.149 DEBUG:boto:Params: {} 2023-11-02T18:39:09.149 DEBUG:boto:Token: None 2023-11-02T18:39:09.150 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:39:09.150 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WpldC72j9JHAcS5KOSfRATYnI9E= 2023-11-02T18:39:09.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WpldC72j9JHAcS5KOSfRATYnI9E=', 'Content-Length': '0'} 2023-11-02T18:39:09.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053c8a749659c5448-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.155 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.155 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.155 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:39:09.156 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:39:09.156 DEBUG:boto:Method: GET 2023-11-02T18:39:09.157 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:39:09.157 DEBUG:boto:Data: 2023-11-02T18:39:09.158 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.158 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.159 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.159 DEBUG:boto:Params: {} 2023-11-02T18:39:09.160 DEBUG:boto:Token: None 2023-11-02T18:39:09.160 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.161 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036fe9b362f067cf6-006543eccd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.206 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.207 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.208 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.208 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:39:09.209 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:39:09.209 DEBUG:boto:Method: GET 2023-11-02T18:39:09.209 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:39:09.210 DEBUG:boto:Data: 2023-11-02T18:39:09.210 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.211 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.211 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.212 DEBUG:boto:Params: {} 2023-11-02T18:39:09.212 DEBUG:boto:Token: None 2023-11-02T18:39:09.213 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.213 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8d85656c2ba9701-006543eccd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.221 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.222 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:39:09.222 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:39:09.222 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.223 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:39:09.223 DEBUG:boto:Data: 2023-11-02T18:39:09.224 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.224 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.225 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.225 DEBUG:boto:Params: {} 2023-11-02T18:39:09.225 DEBUG:boto:Token: None 2023-11-02T18:39:09.226 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj68 2023-11-02T18:39:09.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zWpdCybalzDk6WVfDfY5OxHizdU= 2023-11-02T18:39:09.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zWpdCybalzDk6WVfDfY5OxHizdU=', 'Content-Length': '0'} 2023-11-02T18:39:09.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd26d8b4ce22b6a7-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.231 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:39:09.231 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:39:09.231 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.232 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:39:09.232 DEBUG:boto:Data: 2023-11-02T18:39:09.233 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.233 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.234 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.234 DEBUG:boto:Params: {} 2023-11-02T18:39:09.234 DEBUG:boto:Token: None 2023-11-02T18:39:09.235 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj68 2023-11-02T18:39:09.235 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zWpdCybalzDk6WVfDfY5OxHizdU= 2023-11-02T18:39:09.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zWpdCybalzDk6WVfDfY5OxHizdU=', 'Content-Length': '0'} 2023-11-02T18:39:09.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c605fd8e4e914aa-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:39:09.240 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:39:09.241 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:39:09.241 DEBUG:boto:Method: GET 2023-11-02T18:39:09.241 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:39:09.242 DEBUG:boto:Data: 2023-11-02T18:39:09.242 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.243 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.243 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.244 DEBUG:boto:Params: {} 2023-11-02T18:39:09.244 DEBUG:boto:Token: None 2023-11-02T18:39:09.245 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj68 2023-11-02T18:39:09.245 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y0nn1qDN6PIdMSrJnjTM+btUoWE= 2023-11-02T18:39:09.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y0nn1qDN6PIdMSrJnjTM+btUoWE=', 'Content-Length': '0'} 2023-11-02T18:39:09.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cec5e1ab87b162a8-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.249 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:39:09.250 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:39:09.250 DEBUG:boto:Method: GET 2023-11-02T18:39:09.251 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:39:09.251 DEBUG:boto:Data: 2023-11-02T18:39:09.252 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.252 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.252 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.253 DEBUG:boto:Params: {} 2023-11-02T18:39:09.253 DEBUG:boto:Token: None 2023-11-02T18:39:09.254 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj68 2023-11-02T18:39:09.254 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y0nn1qDN6PIdMSrJnjTM+btUoWE= 2023-11-02T18:39:09.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y0nn1qDN6PIdMSrJnjTM+btUoWE=', 'Content-Length': '0'} 2023-11-02T18:39:09.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf9d2cbcd155f41-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.298 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.299 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.299 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:39:09.300 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:39:09.300 DEBUG:boto:Method: GET 2023-11-02T18:39:09.301 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:39:09.301 DEBUG:boto:Data: 2023-11-02T18:39:09.302 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.302 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.303 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.303 DEBUG:boto:Params: {} 2023-11-02T18:39:09.303 DEBUG:boto:Token: None 2023-11-02T18:39:09.304 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f7ee50e29e5140f-006543eccd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.311 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.311 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.312 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.312 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:39:09.313 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:39:09.313 DEBUG:boto:Method: GET 2023-11-02T18:39:09.314 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:39:09.314 DEBUG:boto:Data: 2023-11-02T18:39:09.314 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.315 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.317 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.318 DEBUG:boto:Params: {} 2023-11-02T18:39:09.318 DEBUG:boto:Token: None 2023-11-02T18:39:09.319 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e2f18e4935c2adf-006543eccd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.327 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:39:09.328 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:39:09.328 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:39:09.329 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.329 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.329 DEBUG:boto:Method: GET 2023-11-02T18:39:09.330 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.330 DEBUG:boto:Data: 2023-11-02T18:39:09.331 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.331 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.331 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.332 DEBUG:boto:Params: {} 2023-11-02T18:39:09.332 DEBUG:boto:Token: None 2023-11-02T18:39:09.333 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.333 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:czDjk9q6J2+XkpxHpNdbrNhTlRU= 2023-11-02T18:39:09.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:czDjk9q6J2+XkpxHpNdbrNhTlRU=', 'Content-Length': '0'} 2023-11-02T18:39:09.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb1f5ff2a52ee8f7-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.338 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:39:09.338 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:39:09.339 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.339 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.339 DEBUG:boto:Method: GET 2023-11-02T18:39:09.340 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.340 DEBUG:boto:Data: 2023-11-02T18:39:09.341 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.341 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.342 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.342 DEBUG:boto:Params: {} 2023-11-02T18:39:09.343 DEBUG:boto:Token: None 2023-11-02T18:39:09.343 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.344 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:czDjk9q6J2+XkpxHpNdbrNhTlRU= 2023-11-02T18:39:09.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:czDjk9q6J2+XkpxHpNdbrNhTlRU=', 'Content-Length': '0'} 2023-11-02T18:39:09.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000765f5b01032392f3-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.348 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:39:09.348 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:39:09.348 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.349 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.349 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.350 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.350 DEBUG:boto:Data: 2023-11-02T18:39:09.351 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.351 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.351 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.352 DEBUG:boto:Params: {} 2023-11-02T18:39:09.352 DEBUG:boto:Token: None 2023-11-02T18:39:09.353 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.353 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dPRTixzxgqAE3HUNsocxl/851UM= 2023-11-02T18:39:09.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dPRTixzxgqAE3HUNsocxl/851UM=', 'Content-Length': '0'} 2023-11-02T18:39:09.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f99b4ddf9fc1f4eb-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.357 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:39:09.357 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:39:09.358 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.358 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.359 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.359 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.360 DEBUG:boto:Data: 2023-11-02T18:39:09.360 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.360 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.361 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.361 DEBUG:boto:Params: {} 2023-11-02T18:39:09.362 DEBUG:boto:Token: None 2023-11-02T18:39:09.362 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.363 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dPRTixzxgqAE3HUNsocxl/851UM= 2023-11-02T18:39:09.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dPRTixzxgqAE3HUNsocxl/851UM=', 'Content-Length': '0'} 2023-11-02T18:39:09.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cae58251980ac0aa-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:39:09.367 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:39:09.367 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:39:09.368 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.368 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.369 DEBUG:boto:Method: GET 2023-11-02T18:39:09.369 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.370 DEBUG:boto:Data: 2023-11-02T18:39:09.370 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.371 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.371 DEBUG:boto:Params: {} 2023-11-02T18:39:09.372 DEBUG:boto:Token: None 2023-11-02T18:39:09.372 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.373 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:czDjk9q6J2+XkpxHpNdbrNhTlRU= 2023-11-02T18:39:09.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:czDjk9q6J2+XkpxHpNdbrNhTlRU=', 'Content-Length': '0'} 2023-11-02T18:39:09.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096dd8006f48cfd50-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.377 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:39:09.377 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:39:09.378 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.378 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.379 DEBUG:boto:Method: GET 2023-11-02T18:39:09.379 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.379 DEBUG:boto:Data: 2023-11-02T18:39:09.381 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.381 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.382 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.383 DEBUG:boto:Params: {} 2023-11-02T18:39:09.383 DEBUG:boto:Token: None 2023-11-02T18:39:09.384 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:39:09.384 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:czDjk9q6J2+XkpxHpNdbrNhTlRU= 2023-11-02T18:39:09.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:czDjk9q6J2+XkpxHpNdbrNhTlRU=', 'Content-Length': '0'} 2023-11-02T18:39:09.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0499a641f4a8eeb-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.430 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.431 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.431 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:39:09.432 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:39:09.432 DEBUG:boto:Method: GET 2023-11-02T18:39:09.433 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:39:09.433 DEBUG:boto:Data: 2023-11-02T18:39:09.434 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.434 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.435 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.435 DEBUG:boto:Params: {} 2023-11-02T18:39:09.436 DEBUG:boto:Token: None 2023-11-02T18:39:09.436 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.437 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047c8bf6279f80a33-006543eccd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.443 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.444 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.444 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.444 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:39:09.445 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:39:09.445 DEBUG:boto:Method: GET 2023-11-02T18:39:09.446 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:39:09.446 DEBUG:boto:Data: 2023-11-02T18:39:09.447 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.447 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.448 DEBUG:boto:Params: {} 2023-11-02T18:39:09.448 DEBUG:boto:Token: None 2023-11-02T18:39:09.449 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.449 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000369fa14c79af80f0-006543eccd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.457 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.458 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:39:09.458 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:39:09.459 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.459 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:39:09.460 DEBUG:boto:Data: 2023-11-02T18:39:09.460 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.461 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.461 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.462 DEBUG:boto:Params: {} 2023-11-02T18:39:09.462 DEBUG:boto:Token: None 2023-11-02T18:39:09.463 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj76 2023-11-02T18:39:09.463 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l6bk2f+YFZfj6XzXJoYPUUByrxg= 2023-11-02T18:39:09.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l6bk2f+YFZfj6XzXJoYPUUByrxg=', 'Content-Length': '0'} 2023-11-02T18:39:09.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e44b2da5761a9ea5-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.467 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:39:09.467 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:39:09.468 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.468 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:39:09.468 DEBUG:boto:Data: 2023-11-02T18:39:09.469 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.469 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.470 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.470 DEBUG:boto:Params: {} 2023-11-02T18:39:09.471 DEBUG:boto:Token: None 2023-11-02T18:39:09.471 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj76 2023-11-02T18:39:09.471 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l6bk2f+YFZfj6XzXJoYPUUByrxg= 2023-11-02T18:39:09.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l6bk2f+YFZfj6XzXJoYPUUByrxg=', 'Content-Length': '0'} 2023-11-02T18:39:09.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0167128a267d523-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:39:09.475 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:39:09.476 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:39:09.476 DEBUG:boto:Method: GET 2023-11-02T18:39:09.477 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:39:09.477 DEBUG:boto:Data: 2023-11-02T18:39:09.478 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.478 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.478 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.479 DEBUG:boto:Params: {} 2023-11-02T18:39:09.479 DEBUG:boto:Token: None 2023-11-02T18:39:09.480 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj76 2023-11-02T18:39:09.480 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OYRIQm8KOAIVyq45x0Ug4lMsOkA= 2023-11-02T18:39:09.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OYRIQm8KOAIVyq45x0Ug4lMsOkA=', 'Content-Length': '0'} 2023-11-02T18:39:09.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d8a4eeb705f0b2d-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.530 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:39:09.532 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:39:09.532 DEBUG:boto:Method: GET 2023-11-02T18:39:09.533 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:39:09.533 DEBUG:boto:Data: 2023-11-02T18:39:09.534 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.534 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.535 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.535 DEBUG:boto:Params: {} 2023-11-02T18:39:09.536 DEBUG:boto:Token: None 2023-11-02T18:39:09.536 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj76 2023-11-02T18:39:09.537 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OYRIQm8KOAIVyq45x0Ug4lMsOkA= 2023-11-02T18:39:09.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OYRIQm8KOAIVyq45x0Ug4lMsOkA=', 'Content-Length': '0'} 2023-11-02T18:39:09.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006da0e2a9847ffa30-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.541 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.542 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.542 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:39:09.543 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:39:09.543 DEBUG:boto:Method: GET 2023-11-02T18:39:09.544 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:39:09.544 DEBUG:boto:Data: 2023-11-02T18:39:09.545 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.545 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.546 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.546 DEBUG:boto:Params: {} 2023-11-02T18:39:09.547 DEBUG:boto:Token: None 2023-11-02T18:39:09.548 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.549 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004469b2d25d107378-006543eccd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.555 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.556 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.556 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.557 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:39:09.557 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:39:09.557 DEBUG:boto:Method: GET 2023-11-02T18:39:09.558 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:39:09.559 DEBUG:boto:Data: 2023-11-02T18:39:09.559 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.559 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.560 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.560 DEBUG:boto:Params: {} 2023-11-02T18:39:09.561 DEBUG:boto:Token: None 2023-11-02T18:39:09.561 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.562 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000202cf1a24ca46320-006543eccd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.610 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:39:09.612 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:39:09.612 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:39:09.613 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.614 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.614 DEBUG:boto:Method: GET 2023-11-02T18:39:09.615 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.615 DEBUG:boto:Data: 2023-11-02T18:39:09.616 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.616 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.617 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.617 DEBUG:boto:Params: {} 2023-11-02T18:39:09.618 DEBUG:boto:Token: None 2023-11-02T18:39:09.618 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.619 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0tJPyO54A7iq1W0S9ZdZprnbDQI= 2023-11-02T18:39:09.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0tJPyO54A7iq1W0S9ZdZprnbDQI=', 'Content-Length': '0'} 2023-11-02T18:39:09.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094768eeb956a3666-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.622 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:39:09.623 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:39:09.623 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.624 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.624 DEBUG:boto:Method: GET 2023-11-02T18:39:09.625 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.625 DEBUG:boto:Data: 2023-11-02T18:39:09.626 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.626 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.627 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.627 DEBUG:boto:Params: {} 2023-11-02T18:39:09.628 DEBUG:boto:Token: None 2023-11-02T18:39:09.628 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.629 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0tJPyO54A7iq1W0S9ZdZprnbDQI= 2023-11-02T18:39:09.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0tJPyO54A7iq1W0S9ZdZprnbDQI=', 'Content-Length': '0'} 2023-11-02T18:39:09.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a8c51f97a84c8b0-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.633 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:39:09.633 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:39:09.634 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.634 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.634 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.635 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.635 DEBUG:boto:Data: 2023-11-02T18:39:09.636 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.636 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.636 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.637 DEBUG:boto:Params: {} 2023-11-02T18:39:09.637 DEBUG:boto:Token: None 2023-11-02T18:39:09.638 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.638 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5Uj3Tu7akMJyWP9D5QxveXvNUMs= 2023-11-02T18:39:09.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5Uj3Tu7akMJyWP9D5QxveXvNUMs=', 'Content-Length': '0'} 2023-11-02T18:39:09.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037eaa9a03ea2e68c-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.642 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:39:09.642 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:39:09.643 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.643 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.643 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.644 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.644 DEBUG:boto:Data: 2023-11-02T18:39:09.645 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.645 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.646 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.646 DEBUG:boto:Params: {} 2023-11-02T18:39:09.647 DEBUG:boto:Token: None 2023-11-02T18:39:09.647 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.648 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5Uj3Tu7akMJyWP9D5QxveXvNUMs= 2023-11-02T18:39:09.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5Uj3Tu7akMJyWP9D5QxveXvNUMs=', 'Content-Length': '0'} 2023-11-02T18:39:09.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008675048cbf298b9-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:39:09.652 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:39:09.653 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:39:09.653 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.654 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.654 DEBUG:boto:Method: GET 2023-11-02T18:39:09.655 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.655 DEBUG:boto:Data: 2023-11-02T18:39:09.656 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.656 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.657 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.657 DEBUG:boto:Params: {} 2023-11-02T18:39:09.657 DEBUG:boto:Token: None 2023-11-02T18:39:09.658 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.659 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0tJPyO54A7iq1W0S9ZdZprnbDQI= 2023-11-02T18:39:09.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0tJPyO54A7iq1W0S9ZdZprnbDQI=', 'Content-Length': '0'} 2023-11-02T18:39:09.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057a7caa93336b201-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.663 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:39:09.663 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:39:09.664 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.664 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.664 DEBUG:boto:Method: GET 2023-11-02T18:39:09.665 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.665 DEBUG:boto:Data: 2023-11-02T18:39:09.666 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.666 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.667 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.667 DEBUG:boto:Params: {} 2023-11-02T18:39:09.668 DEBUG:boto:Token: None 2023-11-02T18:39:09.668 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:39:09.669 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0tJPyO54A7iq1W0S9ZdZprnbDQI= 2023-11-02T18:39:09.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0tJPyO54A7iq1W0S9ZdZprnbDQI=', 'Content-Length': '0'} 2023-11-02T18:39:09.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd28c7d64c0ffad8-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.673 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.673 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.674 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:39:09.674 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:39:09.675 DEBUG:boto:Method: GET 2023-11-02T18:39:09.676 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:39:09.676 DEBUG:boto:Data: 2023-11-02T18:39:09.676 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.677 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.677 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.678 DEBUG:boto:Params: {} 2023-11-02T18:39:09.678 DEBUG:boto:Token: None 2023-11-02T18:39:09.679 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.679 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f14c7d056419bdbc-006543eccd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.686 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.686 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.687 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.687 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:39:09.688 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:39:09.688 DEBUG:boto:Method: GET 2023-11-02T18:39:09.689 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:39:09.689 DEBUG:boto:Data: 2023-11-02T18:39:09.689 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.690 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.691 DEBUG:boto:Params: {} 2023-11-02T18:39:09.691 DEBUG:boto:Token: None 2023-11-02T18:39:09.692 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.692 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e10525df2de1c868-006543eccd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.742 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.743 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:39:09.744 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:39:09.744 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.745 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:39:09.745 DEBUG:boto:Data: 2023-11-02T18:39:09.746 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.746 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.747 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.747 DEBUG:boto:Params: {} 2023-11-02T18:39:09.747 DEBUG:boto:Token: None 2023-11-02T18:39:09.748 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj85 2023-11-02T18:39:09.748 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wq9E8U07YmlR+Xa4fovW4LQtSV0= 2023-11-02T18:39:09.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wq9E8U07YmlR+Xa4fovW4LQtSV0=', 'Content-Length': '0'} 2023-11-02T18:39:09.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0f0d406e21fa1b5-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.752 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:39:09.753 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:39:09.753 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.754 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:39:09.754 DEBUG:boto:Data: 2023-11-02T18:39:09.755 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.755 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.755 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.756 DEBUG:boto:Params: {} 2023-11-02T18:39:09.756 DEBUG:boto:Token: None 2023-11-02T18:39:09.757 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj85 2023-11-02T18:39:09.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wq9E8U07YmlR+Xa4fovW4LQtSV0= 2023-11-02T18:39:09.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wq9E8U07YmlR+Xa4fovW4LQtSV0=', 'Content-Length': '0'} 2023-11-02T18:39:09.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000498096be8c50608e-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:39:09.762 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:39:09.762 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:39:09.763 DEBUG:boto:Method: GET 2023-11-02T18:39:09.763 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:39:09.763 DEBUG:boto:Data: 2023-11-02T18:39:09.764 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.765 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.765 DEBUG:boto:Params: {} 2023-11-02T18:39:09.766 DEBUG:boto:Token: None 2023-11-02T18:39:09.766 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj85 2023-11-02T18:39:09.767 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5HPy514xYzPPgRUQMv9OdViAfe8= 2023-11-02T18:39:09.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5HPy514xYzPPgRUQMv9OdViAfe8=', 'Content-Length': '0'} 2023-11-02T18:39:09.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044e53516346ce79f-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.771 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:39:09.772 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:39:09.772 DEBUG:boto:Method: GET 2023-11-02T18:39:09.773 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:39:09.773 DEBUG:boto:Data: 2023-11-02T18:39:09.774 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.774 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.775 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.776 DEBUG:boto:Params: {} 2023-11-02T18:39:09.777 DEBUG:boto:Token: None 2023-11-02T18:39:09.777 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj85 2023-11-02T18:39:09.779 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5HPy514xYzPPgRUQMv9OdViAfe8= 2023-11-02T18:39:09.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5HPy514xYzPPgRUQMv9OdViAfe8=', 'Content-Length': '0'} 2023-11-02T18:39:09.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061e52a6f90f6ff63-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.783 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.783 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.784 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:39:09.784 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:39:09.785 DEBUG:boto:Method: GET 2023-11-02T18:39:09.785 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:39:09.786 DEBUG:boto:Data: 2023-11-02T18:39:09.786 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.786 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.787 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.787 DEBUG:boto:Params: {} 2023-11-02T18:39:09.788 DEBUG:boto:Token: None 2023-11-02T18:39:09.788 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039136cf4705f5a04-006543eccd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.838 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.839 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.840 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.840 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:39:09.841 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:39:09.841 DEBUG:boto:Method: GET 2023-11-02T18:39:09.842 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:39:09.842 DEBUG:boto:Data: 2023-11-02T18:39:09.842 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.843 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.843 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.844 DEBUG:boto:Params: {} 2023-11-02T18:39:09.844 DEBUG:boto:Token: None 2023-11-02T18:39:09.845 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.845 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b8c25a84dc88a86-006543eccd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.853 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:39:09.854 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:39:09.855 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:39:09.855 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.855 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.856 DEBUG:boto:Method: GET 2023-11-02T18:39:09.856 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.857 DEBUG:boto:Data: 2023-11-02T18:39:09.857 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.858 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.858 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.858 DEBUG:boto:Params: {} 2023-11-02T18:39:09.859 DEBUG:boto:Token: None 2023-11-02T18:39:09.859 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.860 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+wbuth0nNV9V+Hd6Qp0dXQe9RDU= 2023-11-02T18:39:09.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+wbuth0nNV9V+Hd6Qp0dXQe9RDU=', 'Content-Length': '0'} 2023-11-02T18:39:09.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000191d828adc553897-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.864 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:39:09.865 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:39:09.865 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.865 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.866 DEBUG:boto:Method: GET 2023-11-02T18:39:09.866 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.867 DEBUG:boto:Data: 2023-11-02T18:39:09.867 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.868 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.868 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.869 DEBUG:boto:Params: {} 2023-11-02T18:39:09.869 DEBUG:boto:Token: None 2023-11-02T18:39:09.870 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.870 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+wbuth0nNV9V+Hd6Qp0dXQe9RDU= 2023-11-02T18:39:09.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+wbuth0nNV9V+Hd6Qp0dXQe9RDU=', 'Content-Length': '0'} 2023-11-02T18:39:09.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000086c1c9841e25639-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.874 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:39:09.875 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:39:09.875 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.875 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.876 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.876 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.877 DEBUG:boto:Data: 2023-11-02T18:39:09.877 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.877 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.878 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.878 DEBUG:boto:Params: {} 2023-11-02T18:39:09.879 DEBUG:boto:Token: None 2023-11-02T18:39:09.879 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.880 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fddv+hKYedaCXNRDLkX3vg3PFwQ= 2023-11-02T18:39:09.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fddv+hKYedaCXNRDLkX3vg3PFwQ=', 'Content-Length': '0'} 2023-11-02T18:39:09.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2be658b790270f2-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.884 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:39:09.884 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:39:09.885 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.885 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.885 DEBUG:boto:Method: HEAD 2023-11-02T18:39:09.886 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.886 DEBUG:boto:Data: 2023-11-02T18:39:09.887 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.887 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.888 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.888 DEBUG:boto:Params: {} 2023-11-02T18:39:09.889 DEBUG:boto:Token: None 2023-11-02T18:39:09.889 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.890 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fddv+hKYedaCXNRDLkX3vg3PFwQ= 2023-11-02T18:39:09.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fddv+hKYedaCXNRDLkX3vg3PFwQ=', 'Content-Length': '0'} 2023-11-02T18:39:09.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b0fd42e2c28e76a-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:39:09.894 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:39:09.894 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:39:09.895 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.895 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.896 DEBUG:boto:Method: GET 2023-11-02T18:39:09.896 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.897 DEBUG:boto:Data: 2023-11-02T18:39:09.897 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.898 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.898 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.899 DEBUG:boto:Params: {} 2023-11-02T18:39:09.899 DEBUG:boto:Token: None 2023-11-02T18:39:09.900 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.900 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+wbuth0nNV9V+Hd6Qp0dXQe9RDU= 2023-11-02T18:39:09.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+wbuth0nNV9V+Hd6Qp0dXQe9RDU=', 'Content-Length': '0'} 2023-11-02T18:39:09.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003683aa91a2587dc5-006543eccd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.946 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:39:09.947 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:39:09.947 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.948 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.948 DEBUG:boto:Method: GET 2023-11-02T18:39:09.949 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.949 DEBUG:boto:Data: 2023-11-02T18:39:09.950 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.951 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.951 DEBUG:boto:Params: {} 2023-11-02T18:39:09.952 DEBUG:boto:Token: None 2023-11-02T18:39:09.952 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:39:09.953 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+wbuth0nNV9V+Hd6Qp0dXQe9RDU= 2023-11-02T18:39:09.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+wbuth0nNV9V+Hd6Qp0dXQe9RDU=', 'Content-Length': '0'} 2023-11-02T18:39:09.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047db4adefcb511c3-006543eccd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.957 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.957 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.958 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:39:09.958 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:39:09.959 DEBUG:boto:Method: GET 2023-11-02T18:39:09.959 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:39:09.960 DEBUG:boto:Data: 2023-11-02T18:39:09.960 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.961 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:09.962 DEBUG:boto:Port: 8001 2023-11-02T18:39:09.962 DEBUG:boto:Params: {} 2023-11-02T18:39:09.963 DEBUG:boto:Token: None 2023-11-02T18:39:09.963 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.964 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a878966e67d8c170-006543eccd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:09.970 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:09.971 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:09.972 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:09.972 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:39:09.972 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:39:09.973 DEBUG:boto:Method: GET 2023-11-02T18:39:09.973 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:39:09.974 DEBUG:boto:Data: 2023-11-02T18:39:09.974 DEBUG:boto:Headers: {} 2023-11-02T18:39:09.975 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:09.975 DEBUG:boto:Port: 8000 2023-11-02T18:39:09.976 DEBUG:boto:Params: {} 2023-11-02T18:39:09.976 DEBUG:boto:Token: None 2023-11-02T18:39:09.977 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:09 GMT /ibtljn-59/ 2023-11-02T18:39:09.977 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY= 2023-11-02T18:39:09.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bywc+mEBjyqb6BkK0OVpEVf2ebY=', 'Content-Length': '0'} 2023-11-02T18:39:09.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068dbad2cb91f8829-006543eccd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.026 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:10.027 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:39:10.028 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:39:10.028 DEBUG:boto:Method: HEAD 2023-11-02T18:39:10.029 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:39:10.029 DEBUG:boto:Data: 2023-11-02T18:39:10.030 DEBUG:boto:Headers: {} 2023-11-02T18:39:10.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:10.031 DEBUG:boto:Port: 8001 2023-11-02T18:39:10.031 DEBUG:boto:Params: {} 2023-11-02T18:39:10.032 DEBUG:boto:Token: None 2023-11-02T18:39:10.032 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/obj95 2023-11-02T18:39:10.033 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y1a4EHNTyL0oNo6PftWibJYO67c= 2023-11-02T18:39:10.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y1a4EHNTyL0oNo6PftWibJYO67c=', 'Content-Length': '0'} 2023-11-02T18:39:10.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b10223a943080a7-006543ecce-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.037 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:39:10.038 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:39:10.039 DEBUG:boto:Method: HEAD 2023-11-02T18:39:10.039 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:39:10.040 DEBUG:boto:Data: 2023-11-02T18:39:10.040 DEBUG:boto:Headers: {} 2023-11-02T18:39:10.041 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.041 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.041 DEBUG:boto:Params: {} 2023-11-02T18:39:10.042 DEBUG:boto:Token: None 2023-11-02T18:39:10.042 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/obj95 2023-11-02T18:39:10.043 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y1a4EHNTyL0oNo6PftWibJYO67c= 2023-11-02T18:39:10.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y1a4EHNTyL0oNo6PftWibJYO67c=', 'Content-Length': '0'} 2023-11-02T18:39:10.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005765241e04b92ae9-006543ecce-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:39:10.048 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:39:10.048 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:39:10.049 DEBUG:boto:Method: GET 2023-11-02T18:39:10.049 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:39:10.050 DEBUG:boto:Data: 2023-11-02T18:39:10.050 DEBUG:boto:Headers: {} 2023-11-02T18:39:10.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:10.051 DEBUG:boto:Port: 8001 2023-11-02T18:39:10.051 DEBUG:boto:Params: {} 2023-11-02T18:39:10.052 DEBUG:boto:Token: None 2023-11-02T18:39:10.052 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/obj95 2023-11-02T18:39:10.053 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IO1AO/eD1GTvOo1cVaMiGE7THBg= 2023-11-02T18:39:10.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IO1AO/eD1GTvOo1cVaMiGE7THBg=', 'Content-Length': '0'} 2023-11-02T18:39:10.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000600f9073bb6d8d4-006543ecce-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.057 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:39:10.058 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:39:10.058 DEBUG:boto:Method: GET 2023-11-02T18:39:10.059 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:39:10.059 DEBUG:boto:Data: 2023-11-02T18:39:10.060 DEBUG:boto:Headers: {} 2023-11-02T18:39:10.060 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.061 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.061 DEBUG:boto:Params: {} 2023-11-02T18:39:10.061 DEBUG:boto:Token: None 2023-11-02T18:39:10.062 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/obj95 2023-11-02T18:39:10.062 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IO1AO/eD1GTvOo1cVaMiGE7THBg= 2023-11-02T18:39:10.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IO1AO/eD1GTvOo1cVaMiGE7THBg=', 'Content-Length': '0'} 2023-11-02T18:39:10.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a467e8ceb24816c-006543ecce-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.068 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:10.069 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:10.069 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:39:10.070 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:39:10.070 DEBUG:boto:Method: GET 2023-11-02T18:39:10.070 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:39:10.071 DEBUG:boto:Data: 2023-11-02T18:39:10.071 DEBUG:boto:Headers: {} 2023-11-02T18:39:10.072 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:39:10.072 DEBUG:boto:Port: 8001 2023-11-02T18:39:10.073 DEBUG:boto:Params: {} 2023-11-02T18:39:10.073 DEBUG:boto:Token: None 2023-11-02T18:39:10.074 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/ 2023-11-02T18:39:10.074 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z3FKVjqE7Y187vFoIOWpBFtQ/+Y= 2023-11-02T18:39:10.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z3FKVjqE7Y187vFoIOWpBFtQ/+Y=', 'Content-Length': '0'} 2023-11-02T18:39:10.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000570197fc773d8454-006543ecce-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.081 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:10.081 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:10.082 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:10.082 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:39:10.084 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:39:10.084 DEBUG:boto:Method: GET 2023-11-02T18:39:10.085 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:39:10.085 DEBUG:boto:Data: 2023-11-02T18:39:10.085 DEBUG:boto:Headers: {} 2023-11-02T18:39:10.086 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.086 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.087 DEBUG:boto:Params: {} 2023-11-02T18:39:10.087 DEBUG:boto:Token: None 2023-11-02T18:39:10.088 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/ 2023-11-02T18:39:10.088 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z3FKVjqE7Y187vFoIOWpBFtQ/+Y= 2023-11-02T18:39:10.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z3FKVjqE7Y187vFoIOWpBFtQ/+Y=', 'Content-Length': '0'} 2023-11-02T18:39:10.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cee7163e0f55f6b-006543ecce-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.142 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:39:10.143 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:39:10.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:39:10.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:39:10.145 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:39:10.193 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:39:10.194 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:39:10.280 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:39:10.280 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-59 2023-11-02T18:39:10.294 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 15 2023-11-02T18:39:10.391 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:10.392 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:10.393 DEBUG:boto:Method: HEAD 2023-11-02T18:39:10.393 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:39:10.394 DEBUG:boto:Data: 2023-11-02T18:39:10.395 DEBUG:boto:Headers: {} 2023-11-02T18:39:10.395 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.396 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.396 DEBUG:boto:Params: {} 2023-11-02T18:39:10.397 DEBUG:boto:Token: None 2023-11-02T18:39:10.397 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/ 2023-11-02T18:39:10.398 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+e64E12NCMguY+ZZW8F2sHzYV40= 2023-11-02T18:39:10.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+e64E12NCMguY+ZZW8F2sHzYV40=', 'Content-Length': '0'} 2023-11-02T18:39:10.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086e6834df3d53476-006543ecce-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.405 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:39:10.406 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:39:10.406 DEBUG:boto:Method: PUT 2023-11-02T18:39:10.407 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:39:10.407 DEBUG:boto:Data: 2023-11-02T18:39:10.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:39:10.408 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.408 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.409 DEBUG:boto:Params: {} 2023-11-02T18:39:10.409 DEBUG:boto:Token: None 2023-11-02T18:39:10.410 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/obj23 2023-11-02T18:39:10.410 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dM/7ml5DTAD9604tz0poZFz8rsY= 2023-11-02T18:39:10.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dM/7ml5DTAD9604tz0poZFz8rsY='} 2023-11-02T18:39:10.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000fa98d58ca57b871-006543ecce-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.423 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:10.423 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:10.424 DEBUG:boto:Method: HEAD 2023-11-02T18:39:10.424 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:39:10.424 DEBUG:boto:Data: 2023-11-02T18:39:10.425 DEBUG:boto:Headers: {} 2023-11-02T18:39:10.425 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.426 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.426 DEBUG:boto:Params: {} 2023-11-02T18:39:10.427 DEBUG:boto:Token: None 2023-11-02T18:39:10.427 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/ 2023-11-02T18:39:10.427 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+e64E12NCMguY+ZZW8F2sHzYV40= 2023-11-02T18:39:10.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+e64E12NCMguY+ZZW8F2sHzYV40=', 'Content-Length': '0'} 2023-11-02T18:39:10.434 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088739cee1abb7e4d-006543ecce-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.435 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:39:10.435 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:39:10.435 DEBUG:boto:Method: PUT 2023-11-02T18:39:10.436 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:39:10.436 DEBUG:boto:Data: 2023-11-02T18:39:10.437 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:39:10.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.438 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.438 DEBUG:boto:Params: {} 2023-11-02T18:39:10.439 DEBUG:boto:Token: None 2023-11-02T18:39:10.439 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/obj46 2023-11-02T18:39:10.440 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DIRSKraNYOoS/wSoeI+lqOpu6Sw= 2023-11-02T18:39:10.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DIRSKraNYOoS/wSoeI+lqOpu6Sw='} 2023-11-02T18:39:10.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c14db06721bd098a-006543ecce-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.483 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:10.483 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:10.484 DEBUG:boto:Method: HEAD 2023-11-02T18:39:10.484 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:39:10.485 DEBUG:boto:Data: 2023-11-02T18:39:10.485 DEBUG:boto:Headers: {} 2023-11-02T18:39:10.486 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.486 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.487 DEBUG:boto:Params: {} 2023-11-02T18:39:10.487 DEBUG:boto:Token: None 2023-11-02T18:39:10.487 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/ 2023-11-02T18:39:10.488 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+e64E12NCMguY+ZZW8F2sHzYV40= 2023-11-02T18:39:10.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+e64E12NCMguY+ZZW8F2sHzYV40=', 'Content-Length': '0'} 2023-11-02T18:39:10.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe5209257e68ac57-006543ecce-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.496 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:39:10.497 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:39:10.497 DEBUG:boto:Method: PUT 2023-11-02T18:39:10.498 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:39:10.498 DEBUG:boto:Data: 2023-11-02T18:39:10.499 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:39:10.499 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.499 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.500 DEBUG:boto:Params: {} 2023-11-02T18:39:10.500 DEBUG:boto:Token: None 2023-11-02T18:39:10.501 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/obj69 2023-11-02T18:39:10.501 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UTxK4affqiD1EDvPhJqWtG1k7x0= 2023-11-02T18:39:10.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UTxK4affqiD1EDvPhJqWtG1k7x0='} 2023-11-02T18:39:10.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a176289a2308d4f-006543ecce-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.512 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:10.512 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:10.513 DEBUG:boto:Method: HEAD 2023-11-02T18:39:10.513 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:39:10.513 DEBUG:boto:Data: 2023-11-02T18:39:10.514 DEBUG:boto:Headers: {} 2023-11-02T18:39:10.514 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.515 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.515 DEBUG:boto:Params: {} 2023-11-02T18:39:10.516 DEBUG:boto:Token: None 2023-11-02T18:39:10.516 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/ 2023-11-02T18:39:10.516 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+e64E12NCMguY+ZZW8F2sHzYV40= 2023-11-02T18:39:10.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+e64E12NCMguY+ZZW8F2sHzYV40=', 'Content-Length': '0'} 2023-11-02T18:39:10.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed7760abbafd79c9-006543ecce-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.523 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:39:10.523 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:39:10.524 DEBUG:boto:Method: PUT 2023-11-02T18:39:10.524 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:39:10.525 DEBUG:boto:Data: 2023-11-02T18:39:10.525 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:39:10.526 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.526 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.526 DEBUG:boto:Params: {} 2023-11-02T18:39:10.527 DEBUG:boto:Token: None 2023-11-02T18:39:10.527 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/obj92 2023-11-02T18:39:10.528 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hManZakWqeNYZLuODGr6VRpZ47s= 2023-11-02T18:39:10.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hManZakWqeNYZLuODGr6VRpZ47s='} 2023-11-02T18:39:10.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb49a90154ca72d3-006543ecce-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.572 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:39:10.572 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:39:10.573 DEBUG:boto:Method: HEAD 2023-11-02T18:39:10.573 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:39:10.574 DEBUG:boto:Data: 2023-11-02T18:39:10.574 DEBUG:boto:Headers: {} 2023-11-02T18:39:10.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.575 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.575 DEBUG:boto:Params: {} 2023-11-02T18:39:10.576 DEBUG:boto:Token: None 2023-11-02T18:39:10.576 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/ 2023-11-02T18:39:10.577 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+e64E12NCMguY+ZZW8F2sHzYV40= 2023-11-02T18:39:10.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+e64E12NCMguY+ZZW8F2sHzYV40=', 'Content-Length': '0'} 2023-11-02T18:39:10.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bcac7fb34f38931b-006543ecce-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.584 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:39:10.585 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:39:10.585 DEBUG:boto:Method: PUT 2023-11-02T18:39:10.586 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:39:10.586 DEBUG:boto:Data: 2023-11-02T18:39:10.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:39:10.587 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:39:10.587 DEBUG:boto:Port: 8000 2023-11-02T18:39:10.588 DEBUG:boto:Params: {} 2023-11-02T18:39:10.588 DEBUG:boto:Token: None 2023-11-02T18:39:10.589 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:39:10 GMT /ibtljn-59/obj115 2023-11-02T18:39:10.589 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qfuboYBewDomiL7ekKtYz4HCw1U= 2023-11-02T18:39:10.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:39:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qfuboYBewDomiL7ekKtYz4HCw1U='} 2023-11-02T18:39:10.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014b5be2c7cda1541-006543ecce-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:39:10.602 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-59 2023-11-02T18:39:10.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:39:10.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:39:10.603 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:39:10.658 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:39:10.658 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:39:10.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:10.709+0000 7fcf71160f40 1 waiting to reach latest gen 3, on 2.. 2023-11-02T18:39:15.719 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:15.713+0000 7fcf71160f40 1 waiting to reach latest gen 3, on 2.. 2023-11-02T18:39:20.720 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:20.713+0000 7fcf71160f40 1 waiting to reach latest gen 3, on 2.. 2023-11-02T18:39:25.725 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:25.717+0000 7fcf71160f40 1 waiting for incremental sync to catch up: 2023-11-02T18:39:25.726 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:39:25.726 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000002.74.6, , 00000000002.88.6, , , , , , 00000000001.39.6, , 00000000001.51.6, , , , ] 2023-11-02T18:39:30.728 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:30.720+0000 7fcf71160f40 1 waiting for incremental sync to catch up: 2023-11-02T18:39:30.729 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:39:30.729 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000002.74.6, , 00000000002.88.6, , , , , , 00000000001.39.6, , 00000000001.51.6, , , , ] 2023-11-02T18:39:35.732 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:35.724+0000 7fcf71160f40 1 waiting for incremental sync to catch up: 2023-11-02T18:39:35.733 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , , , , , , , , , , , , , ] 2023-11-02T18:39:35.733 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000002.74.6, , 00000000002.88.6, , , , , , 00000000001.39.6, , 00000000001.51.6, , , , ] 2023-11-02T18:39:40.736 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:40.728+0000 7fcf71160f40 1 waiting for incremental sync to catch up: 2023-11-02T18:39:40.737 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , , , , , , , , , , , , , ] 2023-11-02T18:39:40.737 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000002.74.6, , 00000000002.88.6, , , , , , 00000000001.39.6, , 00000000001.51.6, , , , ] 2023-11-02T18:39:45.740 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:45.732+0000 7fcf71160f40 1 waiting for incremental sync to catch up: 2023-11-02T18:39:45.741 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , 00000000002.88.6, , , , , , , , 00000000001.51.6, , , , ] 2023-11-02T18:39:45.741 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000002.74.6, , 00000000002.88.6, , , , , , 00000000001.39.6, , 00000000001.51.6, , , , ] 2023-11-02T18:39:50.744 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:50.736+0000 7fcf71160f40 1 waiting for incremental sync to catch up: 2023-11-02T18:39:50.745 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , 00000000002.88.6, , , , , , , , 00000000001.51.6, , , , ] 2023-11-02T18:39:50.745 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000002.74.6, , 00000000002.88.6, , , , , , 00000000001.39.6, , 00000000001.51.6, , , , ] 2023-11-02T18:39:55.748 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:39:55.740+0000 7fcf71160f40 1 waiting for incremental sync to catch up: 2023-11-02T18:39:55.749 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , 00000000002.88.6, , , , , , , , 00000000001.51.6, , , , ] 2023-11-02T18:39:55.749 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000002.74.6, , 00000000002.88.6, , , , , , 00000000001.39.6, , 00000000001.51.6, , , , ] 2023-11-02T18:40:00.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:00.744+0000 7fcf71160f40 1 waiting for incremental sync to catch up: 2023-11-02T18:40:00.753 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , 00000000002.88.6, , , , , , , , 00000000001.51.6, , , , ] 2023-11-02T18:40:00.753 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000002.74.6, , 00000000002.88.6, , , , , , 00000000001.39.6, , 00000000001.51.6, , , , ] 2023-11-02T18:40:05.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:05.748+0000 7fcf71160f40 1 waiting for incremental sync to catch up: 2023-11-02T18:40:05.757 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , 00000000002.88.6, , , , , , , , 00000000001.51.6, , , , ] 2023-11-02T18:40:05.757 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000002.74.6, , 00000000002.88.6, , , , , , 00000000001.39.6, , 00000000001.51.6, , , , ] 2023-11-02T18:40:10.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:10.752+0000 7fcf71160f40 1 waiting for incremental sync to catch up: 2023-11-02T18:40:10.760 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , 00000000002.88.6, , , , , , , , 00000000001.51.6, , , , ] 2023-11-02T18:40:10.761 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000002.74.6, , 00000000002.88.6, , , , , , 00000000001.39.6, , 00000000001.51.6, , , , ] 2023-11-02T18:40:15.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:15.756+0000 7fcf71160f40 1 bucket sync caught up with source: 2023-11-02T18:40:15.765 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , 00000000002.74.6, , 00000000002.88.6, , , , , , 00000000001.39.6, , 00000000001.51.6, , , , ] 2023-11-02T18:40:15.765 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000002.74.6, , 00000000002.88.6, , , , , , 00000000001.39.6, , 00000000001.51.6, , , , ] 2023-11-02T18:40:15.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:15.756+0000 7fcf71160f40 0 bucket checkpoint complete 2023-11-02T18:40:15.793 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-59 2023-11-02T18:40:15.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:40:15.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:40:15.795 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:40:15.847 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:40:15.848 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:40:15.919 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:15.912+0000 7f907eb16f40 1 waiting for incremental sync to catch up: 2023-11-02T18:40:15.920 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.80.6, 00000000001.223.6, 00000000001.131.6, 00000000002.299.6, 00000000001.272.6, , , 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:40:15.920 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000002.299.6, 00000000002.278.6, , , 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:40:20.922 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:20.912+0000 7f907eb16f40 1 waiting for incremental sync to catch up: 2023-11-02T18:40:20.922 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.80.6, 00000000001.223.6, 00000000001.131.6, 00000000002.299.6, 00000000001.272.6, , , 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:40:20.923 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000002.299.6, 00000000002.278.6, , , 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:40:25.924 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:25.916+0000 7f907eb16f40 1 bucket sync caught up with source: 2023-11-02T18:40:25.925 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000002.299.6, 00000000002.278.6, , , 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:40:25.925 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000002.299.6, 00000000002.278.6, , , 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:40:25.925 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:25.916+0000 7f907eb16f40 0 bucket checkpoint complete 2023-11-02T18:40:25.955 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:40:25.956 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:25.957 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:25.957 DEBUG:boto:Method: HEAD 2023-11-02T18:40:25.958 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:40:25.958 DEBUG:boto:Data: 2023-11-02T18:40:25.959 DEBUG:boto:Headers: {} 2023-11-02T18:40:25.959 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:25.959 DEBUG:boto:Port: 8001 2023-11-02T18:40:25.960 DEBUG:boto:Params: {} 2023-11-02T18:40:25.961 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:40:25.961 DEBUG:boto:Token: None 2023-11-02T18:40:25.962 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:25 GMT /ibtljn-59/ 2023-11-02T18:40:25.962 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXNEPlTK4UM4Ab+5Qbi85hguzYc= 2023-11-02T18:40:25.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXNEPlTK4UM4Ab+5Qbi85hguzYc=', 'Content-Length': '0'} 2023-11-02T18:40:25.968 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c58ef6714383dae-006543ed19-1b8c-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:40:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:25.969 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:25.969 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:25.970 DEBUG:boto:Method: HEAD 2023-11-02T18:40:25.970 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:40:25.971 DEBUG:boto:Data: 2023-11-02T18:40:25.971 DEBUG:boto:Headers: {} 2023-11-02T18:40:25.972 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:25.972 DEBUG:boto:Port: 8000 2023-11-02T18:40:25.972 DEBUG:boto:Params: {} 2023-11-02T18:40:25.973 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:40:25.973 DEBUG:boto:Token: None 2023-11-02T18:40:25.974 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:25 GMT /ibtljn-59/ 2023-11-02T18:40:25.974 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fXNEPlTK4UM4Ab+5Qbi85hguzYc= 2023-11-02T18:40:25.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fXNEPlTK4UM4Ab+5Qbi85hguzYc=', 'Content-Length': '0'} 2023-11-02T18:40:25.983 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025d16333944ab179-006543ed19-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:40:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:25.984 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:40:25.984 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:25.985 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:25.985 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:40:25.986 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:40:25.986 DEBUG:boto:Method: GET 2023-11-02T18:40:25.986 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:40:25.987 DEBUG:boto:Data: 2023-11-02T18:40:25.987 DEBUG:boto:Headers: {} 2023-11-02T18:40:25.988 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:25.989 DEBUG:boto:Port: 8001 2023-11-02T18:40:25.989 DEBUG:boto:Params: {} 2023-11-02T18:40:25.990 DEBUG:boto:Token: None 2023-11-02T18:40:25.990 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:25 GMT /ibtljn-59/?versions 2023-11-02T18:40:25.991 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ORl+CIZIVUI4ZUBRsmK71VO+17o= 2023-11-02T18:40:25.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ORl+CIZIVUI4ZUBRsmK71VO+17o=', 'Content-Length': '0'} 2023-11-02T18:40:25.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d220828a6036038f-006543ed19-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:25.998 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:40:26.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T18:40:26.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:40:26.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:40:26.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:40:26.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T18:40:26.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:40:26.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:40:26.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:40:26.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:40:26.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T18:40:26.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:40:26.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:40:26.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:40:26.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:40:26.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T18:40:26.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:40:26.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:40:26.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T18:40:26.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:40:26.009 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:40:26.010 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.010 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.011 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:40:26.011 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:40:26.012 DEBUG:boto:Method: GET 2023-11-02T18:40:26.012 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:40:26.012 DEBUG:boto:Data: 2023-11-02T18:40:26.013 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.013 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.014 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.014 DEBUG:boto:Params: {} 2023-11-02T18:40:26.015 DEBUG:boto:Token: None 2023-11-02T18:40:26.015 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/?versions 2023-11-02T18:40:26.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3Gc5XPwIpuPAo3H7DqHqXQCwhG8= 2023-11-02T18:40:26.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3Gc5XPwIpuPAo3H7DqHqXQCwhG8=', 'Content-Length': '0'} 2023-11-02T18:40:26.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000502d641fe75caa16-006543ed1a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.024 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:40:26.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T18:40:26.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:40:26.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:40:26.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:40:26.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T18:40:26.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:40:26.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:40:26.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:40:26.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:40:26.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T18:40:26.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:40:26.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:40:26.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:40:26.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:40:26.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T18:40:26.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:40:26.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:40:26.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T18:40:26.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:40:26.034 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.034 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.035 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:40:26.035 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:40:26.036 DEBUG:boto:Method: GET 2023-11-02T18:40:26.036 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:40:26.037 DEBUG:boto:Data: 2023-11-02T18:40:26.037 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.038 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.038 DEBUG:boto:Params: {} 2023-11-02T18:40:26.039 DEBUG:boto:Token: None 2023-11-02T18:40:26.039 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/?versions 2023-11-02T18:40:26.040 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3Gc5XPwIpuPAo3H7DqHqXQCwhG8= 2023-11-02T18:40:26.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3Gc5XPwIpuPAo3H7DqHqXQCwhG8=', 'Content-Length': '0'} 2023-11-02T18:40:26.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c62b1425d3c65f18-006543ed1a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.047 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.048 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.049 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.049 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:40:26.050 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:40:26.050 DEBUG:boto:Method: GET 2023-11-02T18:40:26.051 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:40:26.051 DEBUG:boto:Data: 2023-11-02T18:40:26.051 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.052 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.052 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.053 DEBUG:boto:Params: {} 2023-11-02T18:40:26.053 DEBUG:boto:Token: None 2023-11-02T18:40:26.054 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/?versions 2023-11-02T18:40:26.054 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3Gc5XPwIpuPAo3H7DqHqXQCwhG8= 2023-11-02T18:40:26.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3Gc5XPwIpuPAo3H7DqHqXQCwhG8=', 'Content-Length': '0'} 2023-11-02T18:40:26.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f51d1bd53fac19c6-006543ed1a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.062 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:40:26.064 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:40:26.065 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:40:26.065 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.065 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.066 DEBUG:boto:Method: GET 2023-11-02T18:40:26.066 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.067 DEBUG:boto:Data: 2023-11-02T18:40:26.067 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.068 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.068 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.069 DEBUG:boto:Params: {} 2023-11-02T18:40:26.069 DEBUG:boto:Token: None 2023-11-02T18:40:26.070 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.070 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fr7HgWF/i9N1zZ99N8pwnXJrj/Q= 2023-11-02T18:40:26.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fr7HgWF/i9N1zZ99N8pwnXJrj/Q=', 'Content-Length': '0'} 2023-11-02T18:40:26.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f654bed4923d330-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.074 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:40:26.074 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:40:26.075 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.075 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.076 DEBUG:boto:Method: GET 2023-11-02T18:40:26.076 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.077 DEBUG:boto:Data: 2023-11-02T18:40:26.077 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.078 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.078 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.079 DEBUG:boto:Params: {} 2023-11-02T18:40:26.079 DEBUG:boto:Token: None 2023-11-02T18:40:26.079 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.080 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fr7HgWF/i9N1zZ99N8pwnXJrj/Q= 2023-11-02T18:40:26.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fr7HgWF/i9N1zZ99N8pwnXJrj/Q=', 'Content-Length': '0'} 2023-11-02T18:40:26.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd90479f405e744d-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.126 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:40:26.127 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:40:26.127 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.128 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.128 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.129 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.129 DEBUG:boto:Data: 2023-11-02T18:40:26.130 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.130 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.130 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.131 DEBUG:boto:Params: {} 2023-11-02T18:40:26.131 DEBUG:boto:Token: None 2023-11-02T18:40:26.132 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.132 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i/NrTpX1LFE73ILCmpZoVozeJb4= 2023-11-02T18:40:26.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i/NrTpX1LFE73ILCmpZoVozeJb4=', 'Content-Length': '0'} 2023-11-02T18:40:26.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d205be592613f8a4-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.136 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:40:26.137 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:40:26.137 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.137 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.138 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.138 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.139 DEBUG:boto:Data: 2023-11-02T18:40:26.139 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.140 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.140 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.140 DEBUG:boto:Params: {} 2023-11-02T18:40:26.141 DEBUG:boto:Token: None 2023-11-02T18:40:26.141 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.142 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i/NrTpX1LFE73ILCmpZoVozeJb4= 2023-11-02T18:40:26.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i/NrTpX1LFE73ILCmpZoVozeJb4=', 'Content-Length': '0'} 2023-11-02T18:40:26.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6f302f19cc326be-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:40:26.147 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:40:26.147 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:40:26.148 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.148 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.148 DEBUG:boto:Method: GET 2023-11-02T18:40:26.149 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.149 DEBUG:boto:Data: 2023-11-02T18:40:26.150 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.150 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.151 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.151 DEBUG:boto:Params: {} 2023-11-02T18:40:26.151 DEBUG:boto:Token: None 2023-11-02T18:40:26.152 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.152 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fr7HgWF/i9N1zZ99N8pwnXJrj/Q= 2023-11-02T18:40:26.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fr7HgWF/i9N1zZ99N8pwnXJrj/Q=', 'Content-Length': '0'} 2023-11-02T18:40:26.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca52e29de5efbbb1-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.198 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:40:26.199 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:40:26.199 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.200 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.200 DEBUG:boto:Method: GET 2023-11-02T18:40:26.201 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.201 DEBUG:boto:Data: 2023-11-02T18:40:26.202 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.202 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.203 DEBUG:boto:Params: {} 2023-11-02T18:40:26.203 DEBUG:boto:Token: None 2023-11-02T18:40:26.204 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:40:26.204 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fr7HgWF/i9N1zZ99N8pwnXJrj/Q= 2023-11-02T18:40:26.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fr7HgWF/i9N1zZ99N8pwnXJrj/Q=', 'Content-Length': '0'} 2023-11-02T18:40:26.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b17013a32474d94f-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.209 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.209 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.209 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:40:26.210 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:40:26.210 DEBUG:boto:Method: GET 2023-11-02T18:40:26.211 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:40:26.211 DEBUG:boto:Data: 2023-11-02T18:40:26.212 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.212 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.212 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.213 DEBUG:boto:Params: {} 2023-11-02T18:40:26.213 DEBUG:boto:Token: None 2023-11-02T18:40:26.214 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/ 2023-11-02T18:40:26.214 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs= 2023-11-02T18:40:26.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs=', 'Content-Length': '0'} 2023-11-02T18:40:26.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b42a3e5a32211ba7-006543ed1a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.220 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.221 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.221 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.221 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:40:26.222 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:40:26.222 DEBUG:boto:Method: GET 2023-11-02T18:40:26.223 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:40:26.223 DEBUG:boto:Data: 2023-11-02T18:40:26.224 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.224 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.224 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.225 DEBUG:boto:Params: {} 2023-11-02T18:40:26.225 DEBUG:boto:Token: None 2023-11-02T18:40:26.226 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/ 2023-11-02T18:40:26.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs= 2023-11-02T18:40:26.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs=', 'Content-Length': '0'} 2023-11-02T18:40:26.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001480bf394f1209cd-006543ed1a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.234 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.235 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:40:26.235 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:40:26.236 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.236 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:40:26.236 DEBUG:boto:Data: 2023-11-02T18:40:26.237 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.237 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.238 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.238 DEBUG:boto:Params: {} 2023-11-02T18:40:26.238 DEBUG:boto:Token: None 2023-11-02T18:40:26.239 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj11 2023-11-02T18:40:26.239 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZOF+RhexiYWTetN0W2yL9cO75JU= 2023-11-02T18:40:26.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZOF+RhexiYWTetN0W2yL9cO75JU=', 'Content-Length': '0'} 2023-11-02T18:40:26.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edb10fadd038583e-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.243 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:40:26.243 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:40:26.244 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.244 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:40:26.245 DEBUG:boto:Data: 2023-11-02T18:40:26.245 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.246 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.246 DEBUG:boto:Params: {} 2023-11-02T18:40:26.247 DEBUG:boto:Token: None 2023-11-02T18:40:26.247 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj11 2023-11-02T18:40:26.248 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZOF+RhexiYWTetN0W2yL9cO75JU= 2023-11-02T18:40:26.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZOF+RhexiYWTetN0W2yL9cO75JU=', 'Content-Length': '0'} 2023-11-02T18:40:26.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065c61530481b6082-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:40:26.252 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:40:26.252 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:40:26.253 DEBUG:boto:Method: GET 2023-11-02T18:40:26.253 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:40:26.253 DEBUG:boto:Data: 2023-11-02T18:40:26.254 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.255 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.255 DEBUG:boto:Params: {} 2023-11-02T18:40:26.256 DEBUG:boto:Token: None 2023-11-02T18:40:26.256 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj11 2023-11-02T18:40:26.257 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QqRylPc6tcwx/hK5gaBOhs/c8Gs= 2023-11-02T18:40:26.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QqRylPc6tcwx/hK5gaBOhs/c8Gs=', 'Content-Length': '0'} 2023-11-02T18:40:26.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d267a8c05d54b174-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.306 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:40:26.307 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:40:26.307 DEBUG:boto:Method: GET 2023-11-02T18:40:26.308 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:40:26.308 DEBUG:boto:Data: 2023-11-02T18:40:26.309 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.309 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.310 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.311 DEBUG:boto:Params: {} 2023-11-02T18:40:26.312 DEBUG:boto:Token: None 2023-11-02T18:40:26.312 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj11 2023-11-02T18:40:26.313 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QqRylPc6tcwx/hK5gaBOhs/c8Gs= 2023-11-02T18:40:26.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QqRylPc6tcwx/hK5gaBOhs/c8Gs=', 'Content-Length': '0'} 2023-11-02T18:40:26.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc4b48f6941c9ccb-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.319 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.319 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.320 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:40:26.320 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:40:26.321 DEBUG:boto:Method: GET 2023-11-02T18:40:26.321 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:40:26.322 DEBUG:boto:Data: 2023-11-02T18:40:26.322 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.323 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.323 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.323 DEBUG:boto:Params: {} 2023-11-02T18:40:26.324 DEBUG:boto:Token: None 2023-11-02T18:40:26.324 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/ 2023-11-02T18:40:26.325 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs= 2023-11-02T18:40:26.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs=', 'Content-Length': '0'} 2023-11-02T18:40:26.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cf989871184843a-006543ed1a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.330 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.331 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.332 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.332 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:40:26.333 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:40:26.333 DEBUG:boto:Method: GET 2023-11-02T18:40:26.333 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:40:26.334 DEBUG:boto:Data: 2023-11-02T18:40:26.334 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.335 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.335 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.336 DEBUG:boto:Params: {} 2023-11-02T18:40:26.336 DEBUG:boto:Token: None 2023-11-02T18:40:26.337 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/ 2023-11-02T18:40:26.337 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs= 2023-11-02T18:40:26.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs=', 'Content-Length': '0'} 2023-11-02T18:40:26.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec1939ba27d8a213-006543ed1a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.345 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:40:26.346 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:40:26.346 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:40:26.347 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.347 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.348 DEBUG:boto:Method: GET 2023-11-02T18:40:26.348 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.349 DEBUG:boto:Data: 2023-11-02T18:40:26.349 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.349 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.350 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.350 DEBUG:boto:Params: {} 2023-11-02T18:40:26.351 DEBUG:boto:Token: None 2023-11-02T18:40:26.351 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.352 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LBA7EIYlWrUis/fUZYo9GlNrHrg= 2023-11-02T18:40:26.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LBA7EIYlWrUis/fUZYo9GlNrHrg=', 'Content-Length': '0'} 2023-11-02T18:40:26.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d34da2dfcecfae7-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.356 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:40:26.356 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:40:26.357 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.357 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.358 DEBUG:boto:Method: GET 2023-11-02T18:40:26.358 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.359 DEBUG:boto:Data: 2023-11-02T18:40:26.359 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.360 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.360 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.360 DEBUG:boto:Params: {} 2023-11-02T18:40:26.361 DEBUG:boto:Token: None 2023-11-02T18:40:26.361 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.362 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LBA7EIYlWrUis/fUZYo9GlNrHrg= 2023-11-02T18:40:26.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LBA7EIYlWrUis/fUZYo9GlNrHrg=', 'Content-Length': '0'} 2023-11-02T18:40:26.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d1fc52d0d17624e-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.366 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:40:26.367 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:40:26.367 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.367 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.368 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.368 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.369 DEBUG:boto:Data: 2023-11-02T18:40:26.369 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.370 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.370 DEBUG:boto:Params: {} 2023-11-02T18:40:26.371 DEBUG:boto:Token: None 2023-11-02T18:40:26.371 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dAgOaN1Il8/rxZ9604IPzlJRSlo= 2023-11-02T18:40:26.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dAgOaN1Il8/rxZ9604IPzlJRSlo=', 'Content-Length': '0'} 2023-11-02T18:40:26.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e7648737544bed7-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.376 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:40:26.377 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:40:26.377 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.377 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.378 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.378 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.379 DEBUG:boto:Data: 2023-11-02T18:40:26.379 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.379 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.380 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.380 DEBUG:boto:Params: {} 2023-11-02T18:40:26.381 DEBUG:boto:Token: None 2023-11-02T18:40:26.381 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.382 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dAgOaN1Il8/rxZ9604IPzlJRSlo= 2023-11-02T18:40:26.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dAgOaN1Il8/rxZ9604IPzlJRSlo=', 'Content-Length': '0'} 2023-11-02T18:40:26.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000773820c4af000a7a-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:40:26.386 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:40:26.387 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:40:26.387 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.388 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.388 DEBUG:boto:Method: GET 2023-11-02T18:40:26.388 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.389 DEBUG:boto:Data: 2023-11-02T18:40:26.389 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.390 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.390 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.391 DEBUG:boto:Params: {} 2023-11-02T18:40:26.391 DEBUG:boto:Token: None 2023-11-02T18:40:26.392 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.392 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LBA7EIYlWrUis/fUZYo9GlNrHrg= 2023-11-02T18:40:26.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LBA7EIYlWrUis/fUZYo9GlNrHrg=', 'Content-Length': '0'} 2023-11-02T18:40:26.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c46d713e4898cfe4-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.442 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:40:26.443 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:40:26.443 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.444 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.444 DEBUG:boto:Method: GET 2023-11-02T18:40:26.445 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.445 DEBUG:boto:Data: 2023-11-02T18:40:26.446 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.446 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.447 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.447 DEBUG:boto:Params: {} 2023-11-02T18:40:26.447 DEBUG:boto:Token: None 2023-11-02T18:40:26.448 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:40:26.449 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LBA7EIYlWrUis/fUZYo9GlNrHrg= 2023-11-02T18:40:26.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LBA7EIYlWrUis/fUZYo9GlNrHrg=', 'Content-Length': '0'} 2023-11-02T18:40:26.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009067e13466475657-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.453 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.453 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.454 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:40:26.454 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:40:26.455 DEBUG:boto:Method: GET 2023-11-02T18:40:26.455 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:40:26.455 DEBUG:boto:Data: 2023-11-02T18:40:26.456 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.456 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.457 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.457 DEBUG:boto:Params: {} 2023-11-02T18:40:26.458 DEBUG:boto:Token: None 2023-11-02T18:40:26.458 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/ 2023-11-02T18:40:26.459 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs= 2023-11-02T18:40:26.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs=', 'Content-Length': '0'} 2023-11-02T18:40:26.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083a5a4e4d144ea20-006543ed1a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.465 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.465 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.466 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.466 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:40:26.467 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:40:26.467 DEBUG:boto:Method: GET 2023-11-02T18:40:26.468 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:40:26.468 DEBUG:boto:Data: 2023-11-02T18:40:26.468 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.469 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.469 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.470 DEBUG:boto:Params: {} 2023-11-02T18:40:26.470 DEBUG:boto:Token: None 2023-11-02T18:40:26.471 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/ 2023-11-02T18:40:26.471 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs= 2023-11-02T18:40:26.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs=', 'Content-Length': '0'} 2023-11-02T18:40:26.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0eade11138a4ab2-006543ed1a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.522 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.523 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:40:26.524 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:40:26.524 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.525 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:40:26.525 DEBUG:boto:Data: 2023-11-02T18:40:26.525 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.526 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.526 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.527 DEBUG:boto:Params: {} 2023-11-02T18:40:26.527 DEBUG:boto:Token: None 2023-11-02T18:40:26.528 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj115 2023-11-02T18:40:26.528 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0mzpyPdefgu1V2REW71ZGvn1bjw= 2023-11-02T18:40:26.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0mzpyPdefgu1V2REW71ZGvn1bjw=', 'Content-Length': '0'} 2023-11-02T18:40:26.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082f3fa0fb37e88d8-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.533 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:40:26.534 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:40:26.534 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.535 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:40:26.535 DEBUG:boto:Data: 2023-11-02T18:40:26.536 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.536 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.537 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.538 DEBUG:boto:Params: {} 2023-11-02T18:40:26.538 DEBUG:boto:Token: None 2023-11-02T18:40:26.539 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj115 2023-11-02T18:40:26.539 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0mzpyPdefgu1V2REW71ZGvn1bjw= 2023-11-02T18:40:26.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0mzpyPdefgu1V2REW71ZGvn1bjw=', 'Content-Length': '0'} 2023-11-02T18:40:26.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006291a51585956610-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:40:26.544 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:40:26.544 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:40:26.545 DEBUG:boto:Method: GET 2023-11-02T18:40:26.545 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:40:26.545 DEBUG:boto:Data: 2023-11-02T18:40:26.546 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.546 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.547 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.548 DEBUG:boto:Params: {} 2023-11-02T18:40:26.548 DEBUG:boto:Token: None 2023-11-02T18:40:26.549 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj115 2023-11-02T18:40:26.550 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9H0K/S9Sx1xRsAI2XtDhSMhmnnM= 2023-11-02T18:40:26.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9H0K/S9Sx1xRsAI2XtDhSMhmnnM=', 'Content-Length': '0'} 2023-11-02T18:40:26.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6c7e45c9fa72781-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.554 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:40:26.554 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:40:26.555 DEBUG:boto:Method: GET 2023-11-02T18:40:26.555 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:40:26.556 DEBUG:boto:Data: 2023-11-02T18:40:26.556 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.556 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.557 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.557 DEBUG:boto:Params: {} 2023-11-02T18:40:26.558 DEBUG:boto:Token: None 2023-11-02T18:40:26.558 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj115 2023-11-02T18:40:26.559 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9H0K/S9Sx1xRsAI2XtDhSMhmnnM= 2023-11-02T18:40:26.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9H0K/S9Sx1xRsAI2XtDhSMhmnnM=', 'Content-Length': '0'} 2023-11-02T18:40:26.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1feff20704624a9-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.564 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.564 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.564 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:40:26.565 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:40:26.565 DEBUG:boto:Method: GET 2023-11-02T18:40:26.566 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:40:26.566 DEBUG:boto:Data: 2023-11-02T18:40:26.567 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.567 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.568 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.568 DEBUG:boto:Params: {} 2023-11-02T18:40:26.569 DEBUG:boto:Token: None 2023-11-02T18:40:26.569 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/ 2023-11-02T18:40:26.570 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs= 2023-11-02T18:40:26.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs=', 'Content-Length': '0'} 2023-11-02T18:40:26.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000981c2f69646b1890-006543ed1a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.576 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.577 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.577 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.578 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:40:26.578 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:40:26.579 DEBUG:boto:Method: GET 2023-11-02T18:40:26.579 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:40:26.580 DEBUG:boto:Data: 2023-11-02T18:40:26.580 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.580 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.581 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.581 DEBUG:boto:Params: {} 2023-11-02T18:40:26.582 DEBUG:boto:Token: None 2023-11-02T18:40:26.582 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/ 2023-11-02T18:40:26.583 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs= 2023-11-02T18:40:26.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs=', 'Content-Length': '0'} 2023-11-02T18:40:26.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000450f433cc5d2b26b-006543ed1a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.634 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:40:26.636 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:40:26.636 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:40:26.637 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.637 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.637 DEBUG:boto:Method: GET 2023-11-02T18:40:26.638 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.638 DEBUG:boto:Data: 2023-11-02T18:40:26.639 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.640 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.641 DEBUG:boto:Params: {} 2023-11-02T18:40:26.642 DEBUG:boto:Token: None 2023-11-02T18:40:26.642 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.643 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aXa1PKCQWS6eGE5N1PUXhET2qHg= 2023-11-02T18:40:26.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aXa1PKCQWS6eGE5N1PUXhET2qHg=', 'Content-Length': '0'} 2023-11-02T18:40:26.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000140555cd84734c16-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.647 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:40:26.648 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:40:26.648 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.649 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.649 DEBUG:boto:Method: GET 2023-11-02T18:40:26.650 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.650 DEBUG:boto:Data: 2023-11-02T18:40:26.651 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.652 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.652 DEBUG:boto:Params: {} 2023-11-02T18:40:26.653 DEBUG:boto:Token: None 2023-11-02T18:40:26.653 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.654 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aXa1PKCQWS6eGE5N1PUXhET2qHg= 2023-11-02T18:40:26.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aXa1PKCQWS6eGE5N1PUXhET2qHg=', 'Content-Length': '0'} 2023-11-02T18:40:26.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9ecd5326f924a97-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.658 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:40:26.658 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:40:26.659 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.659 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.659 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.660 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.660 DEBUG:boto:Data: 2023-11-02T18:40:26.661 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.661 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.662 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.662 DEBUG:boto:Params: {} 2023-11-02T18:40:26.662 DEBUG:boto:Token: None 2023-11-02T18:40:26.663 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.663 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ShW39DaFEN/ThnFN5jkWBQ4R3Gs= 2023-11-02T18:40:26.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ShW39DaFEN/ThnFN5jkWBQ4R3Gs=', 'Content-Length': '0'} 2023-11-02T18:40:26.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee436e6e61043df9-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.668 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:40:26.668 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:40:26.668 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.669 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.669 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.670 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.670 DEBUG:boto:Data: 2023-11-02T18:40:26.671 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.671 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.672 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.672 DEBUG:boto:Params: {} 2023-11-02T18:40:26.672 DEBUG:boto:Token: None 2023-11-02T18:40:26.673 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.673 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ShW39DaFEN/ThnFN5jkWBQ4R3Gs= 2023-11-02T18:40:26.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ShW39DaFEN/ThnFN5jkWBQ4R3Gs=', 'Content-Length': '0'} 2023-11-02T18:40:26.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007550cdb3c2a0529c-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:40:26.678 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:40:26.678 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:40:26.679 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.679 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.680 DEBUG:boto:Method: GET 2023-11-02T18:40:26.680 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.681 DEBUG:boto:Data: 2023-11-02T18:40:26.681 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.681 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.682 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.682 DEBUG:boto:Params: {} 2023-11-02T18:40:26.683 DEBUG:boto:Token: None 2023-11-02T18:40:26.683 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.684 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aXa1PKCQWS6eGE5N1PUXhET2qHg= 2023-11-02T18:40:26.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aXa1PKCQWS6eGE5N1PUXhET2qHg=', 'Content-Length': '0'} 2023-11-02T18:40:26.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6610b90c0fc07b6-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.730 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:40:26.731 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:40:26.731 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.732 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.732 DEBUG:boto:Method: GET 2023-11-02T18:40:26.733 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.733 DEBUG:boto:Data: 2023-11-02T18:40:26.734 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.734 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.735 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.735 DEBUG:boto:Params: {} 2023-11-02T18:40:26.735 DEBUG:boto:Token: None 2023-11-02T18:40:26.736 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:40:26.736 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aXa1PKCQWS6eGE5N1PUXhET2qHg= 2023-11-02T18:40:26.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aXa1PKCQWS6eGE5N1PUXhET2qHg=', 'Content-Length': '0'} 2023-11-02T18:40:26.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000546573df4fa03a45-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.741 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.741 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.741 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:40:26.742 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:40:26.742 DEBUG:boto:Method: GET 2023-11-02T18:40:26.743 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:40:26.743 DEBUG:boto:Data: 2023-11-02T18:40:26.744 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.744 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.745 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.745 DEBUG:boto:Params: {} 2023-11-02T18:40:26.745 DEBUG:boto:Token: None 2023-11-02T18:40:26.746 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/ 2023-11-02T18:40:26.746 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs= 2023-11-02T18:40:26.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs=', 'Content-Length': '0'} 2023-11-02T18:40:26.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fca8a7755699c7d-006543ed1a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.752 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.753 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.754 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.754 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:40:26.755 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:40:26.755 DEBUG:boto:Method: GET 2023-11-02T18:40:26.755 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:40:26.756 DEBUG:boto:Data: 2023-11-02T18:40:26.756 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.757 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.757 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.758 DEBUG:boto:Params: {} 2023-11-02T18:40:26.758 DEBUG:boto:Token: None 2023-11-02T18:40:26.759 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/ 2023-11-02T18:40:26.759 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs= 2023-11-02T18:40:26.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs=', 'Content-Length': '0'} 2023-11-02T18:40:26.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055be23839523ed0e-006543ed1a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.768 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.768 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:40:26.769 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:40:26.769 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.770 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:40:26.770 DEBUG:boto:Data: 2023-11-02T18:40:26.770 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.771 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.771 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.772 DEBUG:boto:Params: {} 2023-11-02T18:40:26.772 DEBUG:boto:Token: None 2023-11-02T18:40:26.773 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj17 2023-11-02T18:40:26.773 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Svtm3Vq2ynmWj25dQbG30v6bOD8= 2023-11-02T18:40:26.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Svtm3Vq2ynmWj25dQbG30v6bOD8=', 'Content-Length': '0'} 2023-11-02T18:40:26.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7159482173ed046-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.777 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:40:26.777 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:40:26.778 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.778 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:40:26.778 DEBUG:boto:Data: 2023-11-02T18:40:26.779 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.779 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.780 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.780 DEBUG:boto:Params: {} 2023-11-02T18:40:26.781 DEBUG:boto:Token: None 2023-11-02T18:40:26.781 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj17 2023-11-02T18:40:26.782 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Svtm3Vq2ynmWj25dQbG30v6bOD8= 2023-11-02T18:40:26.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Svtm3Vq2ynmWj25dQbG30v6bOD8=', 'Content-Length': '0'} 2023-11-02T18:40:26.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec265c3ce44da4ff-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:40:26.786 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:40:26.786 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:40:26.787 DEBUG:boto:Method: GET 2023-11-02T18:40:26.787 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:40:26.788 DEBUG:boto:Data: 2023-11-02T18:40:26.788 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.788 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.789 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.789 DEBUG:boto:Params: {} 2023-11-02T18:40:26.790 DEBUG:boto:Token: None 2023-11-02T18:40:26.790 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj17 2023-11-02T18:40:26.791 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z7ZEVFw2lgjkLICliZkeW/2oeeA= 2023-11-02T18:40:26.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z7ZEVFw2lgjkLICliZkeW/2oeeA=', 'Content-Length': '0'} 2023-11-02T18:40:26.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ab75b778ccffab0-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.795 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:40:26.795 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:40:26.796 DEBUG:boto:Method: GET 2023-11-02T18:40:26.796 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:40:26.797 DEBUG:boto:Data: 2023-11-02T18:40:26.797 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.798 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.798 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.799 DEBUG:boto:Params: {} 2023-11-02T18:40:26.799 DEBUG:boto:Token: None 2023-11-02T18:40:26.800 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj17 2023-11-02T18:40:26.800 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z7ZEVFw2lgjkLICliZkeW/2oeeA= 2023-11-02T18:40:26.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z7ZEVFw2lgjkLICliZkeW/2oeeA=', 'Content-Length': '0'} 2023-11-02T18:40:26.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be7566bfa186c40d-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.846 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.847 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.847 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:40:26.848 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:40:26.848 DEBUG:boto:Method: GET 2023-11-02T18:40:26.849 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:40:26.849 DEBUG:boto:Data: 2023-11-02T18:40:26.850 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.850 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.851 DEBUG:boto:Params: {} 2023-11-02T18:40:26.851 DEBUG:boto:Token: None 2023-11-02T18:40:26.852 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/ 2023-11-02T18:40:26.853 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs= 2023-11-02T18:40:26.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs=', 'Content-Length': '0'} 2023-11-02T18:40:26.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003adb0c279a8abb3b-006543ed1a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.859 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.860 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:26.860 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:26.861 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:40:26.861 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:40:26.861 DEBUG:boto:Method: GET 2023-11-02T18:40:26.862 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:40:26.862 DEBUG:boto:Data: 2023-11-02T18:40:26.863 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.863 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.864 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.864 DEBUG:boto:Params: {} 2023-11-02T18:40:26.865 DEBUG:boto:Token: None 2023-11-02T18:40:26.865 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/ 2023-11-02T18:40:26.866 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs= 2023-11-02T18:40:26.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dqOTEelpNv3jhOYmncr5zPEgEfs=', 'Content-Length': '0'} 2023-11-02T18:40:26.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fe9cec07e62f996-006543ed1a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.874 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:26.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:40:26.875 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:40:26.875 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:40:26.876 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.876 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.877 DEBUG:boto:Method: GET 2023-11-02T18:40:26.877 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.877 DEBUG:boto:Data: 2023-11-02T18:40:26.878 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.878 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.879 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.879 DEBUG:boto:Params: {} 2023-11-02T18:40:26.880 DEBUG:boto:Token: None 2023-11-02T18:40:26.880 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.881 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RL/893F72pWGHGryrxuvRoPox4c= 2023-11-02T18:40:26.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RL/893F72pWGHGryrxuvRoPox4c=', 'Content-Length': '0'} 2023-11-02T18:40:26.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea892313789ae0d8-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.930 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:40:26.931 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:40:26.931 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.932 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.932 DEBUG:boto:Method: GET 2023-11-02T18:40:26.933 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.933 DEBUG:boto:Data: 2023-11-02T18:40:26.933 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.934 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.935 DEBUG:boto:Params: {} 2023-11-02T18:40:26.935 DEBUG:boto:Token: None 2023-11-02T18:40:26.936 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.936 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RL/893F72pWGHGryrxuvRoPox4c= 2023-11-02T18:40:26.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RL/893F72pWGHGryrxuvRoPox4c=', 'Content-Length': '0'} 2023-11-02T18:40:26.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b34b57236161618-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.941 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:40:26.941 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:40:26.941 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.942 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.942 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.943 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.943 DEBUG:boto:Data: 2023-11-02T18:40:26.944 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.944 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.944 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.945 DEBUG:boto:Params: {} 2023-11-02T18:40:26.945 DEBUG:boto:Token: None 2023-11-02T18:40:26.946 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.946 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bSdLLD0hc5cPAtC/LmyjFfIAHNs= 2023-11-02T18:40:26.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bSdLLD0hc5cPAtC/LmyjFfIAHNs=', 'Content-Length': '0'} 2023-11-02T18:40:26.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d42413bbc88a526f-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.950 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:40:26.951 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:40:26.951 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.951 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.952 DEBUG:boto:Method: HEAD 2023-11-02T18:40:26.952 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.953 DEBUG:boto:Data: 2023-11-02T18:40:26.953 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.954 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.954 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.955 DEBUG:boto:Params: {} 2023-11-02T18:40:26.955 DEBUG:boto:Token: None 2023-11-02T18:40:26.955 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.956 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bSdLLD0hc5cPAtC/LmyjFfIAHNs= 2023-11-02T18:40:26.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bSdLLD0hc5cPAtC/LmyjFfIAHNs=', 'Content-Length': '0'} 2023-11-02T18:40:26.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6d3fcc113a13fa2-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:40:26.960 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:40:26.961 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:40:26.962 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.962 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.963 DEBUG:boto:Method: GET 2023-11-02T18:40:26.963 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.963 DEBUG:boto:Data: 2023-11-02T18:40:26.964 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.964 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:26.965 DEBUG:boto:Port: 8001 2023-11-02T18:40:26.965 DEBUG:boto:Params: {} 2023-11-02T18:40:26.966 DEBUG:boto:Token: None 2023-11-02T18:40:26.966 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.967 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RL/893F72pWGHGryrxuvRoPox4c= 2023-11-02T18:40:26.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RL/893F72pWGHGryrxuvRoPox4c=', 'Content-Length': '0'} 2023-11-02T18:40:26.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b53e9ea96f59f15-006543ed1a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:26.971 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:40:26.971 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:40:26.972 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.972 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.973 DEBUG:boto:Method: GET 2023-11-02T18:40:26.973 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.974 DEBUG:boto:Data: 2023-11-02T18:40:26.974 DEBUG:boto:Headers: {} 2023-11-02T18:40:26.974 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:26.975 DEBUG:boto:Port: 8000 2023-11-02T18:40:26.975 DEBUG:boto:Params: {} 2023-11-02T18:40:26.976 DEBUG:boto:Token: None 2023-11-02T18:40:26.976 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:26 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:40:26.977 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RL/893F72pWGHGryrxuvRoPox4c= 2023-11-02T18:40:26.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RL/893F72pWGHGryrxuvRoPox4c=', 'Content-Length': '0'} 2023-11-02T18:40:26.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb9a8825ce5aa133-006543ed1a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.022 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.023 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.024 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:40:27.024 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:40:27.024 DEBUG:boto:Method: GET 2023-11-02T18:40:27.025 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:40:27.025 DEBUG:boto:Data: 2023-11-02T18:40:27.026 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.026 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.027 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.027 DEBUG:boto:Params: {} 2023-11-02T18:40:27.028 DEBUG:boto:Token: None 2023-11-02T18:40:27.028 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.029 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b743433ee4dd7661-006543ed1b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.035 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.036 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.036 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.036 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:40:27.037 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:40:27.037 DEBUG:boto:Method: GET 2023-11-02T18:40:27.038 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:40:27.038 DEBUG:boto:Data: 2023-11-02T18:40:27.039 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.039 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.040 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.040 DEBUG:boto:Params: {} 2023-11-02T18:40:27.041 DEBUG:boto:Token: None 2023-11-02T18:40:27.041 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.042 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010d12685e32c07a2-006543ed1b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.050 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.051 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:40:27.051 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:40:27.051 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.052 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:40:27.052 DEBUG:boto:Data: 2023-11-02T18:40:27.053 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.053 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.054 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.054 DEBUG:boto:Params: {} 2023-11-02T18:40:27.055 DEBUG:boto:Token: None 2023-11-02T18:40:27.055 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj19 2023-11-02T18:40:27.056 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3q2FfBa7IRBCWKq7fiZI8gvUb2g= 2023-11-02T18:40:27.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3q2FfBa7IRBCWKq7fiZI8gvUb2g=', 'Content-Length': '0'} 2023-11-02T18:40:27.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c060c67122ea68-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.059 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:40:27.060 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:40:27.060 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.061 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:40:27.061 DEBUG:boto:Data: 2023-11-02T18:40:27.061 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.062 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.062 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.063 DEBUG:boto:Params: {} 2023-11-02T18:40:27.063 DEBUG:boto:Token: None 2023-11-02T18:40:27.064 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj19 2023-11-02T18:40:27.064 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3q2FfBa7IRBCWKq7fiZI8gvUb2g= 2023-11-02T18:40:27.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3q2FfBa7IRBCWKq7fiZI8gvUb2g=', 'Content-Length': '0'} 2023-11-02T18:40:27.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4ed88677979bee7-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:40:27.068 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:40:27.069 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:40:27.069 DEBUG:boto:Method: GET 2023-11-02T18:40:27.070 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:40:27.070 DEBUG:boto:Data: 2023-11-02T18:40:27.070 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.071 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.071 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.072 DEBUG:boto:Params: {} 2023-11-02T18:40:27.072 DEBUG:boto:Token: None 2023-11-02T18:40:27.073 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj19 2023-11-02T18:40:27.073 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QKtd/a0zABCWMZ1DJv3Vcw5EIrE= 2023-11-02T18:40:27.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QKtd/a0zABCWMZ1DJv3Vcw5EIrE=', 'Content-Length': '0'} 2023-11-02T18:40:27.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fceb808f9ee0af31-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.118 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:40:27.119 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:40:27.119 DEBUG:boto:Method: GET 2023-11-02T18:40:27.121 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:40:27.122 DEBUG:boto:Data: 2023-11-02T18:40:27.122 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.122 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.123 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.123 DEBUG:boto:Params: {} 2023-11-02T18:40:27.124 DEBUG:boto:Token: None 2023-11-02T18:40:27.125 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj19 2023-11-02T18:40:27.125 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QKtd/a0zABCWMZ1DJv3Vcw5EIrE= 2023-11-02T18:40:27.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QKtd/a0zABCWMZ1DJv3Vcw5EIrE=', 'Content-Length': '0'} 2023-11-02T18:40:27.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d3596c4bda4717f-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.130 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.130 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.131 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:40:27.131 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:40:27.132 DEBUG:boto:Method: GET 2023-11-02T18:40:27.132 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:40:27.133 DEBUG:boto:Data: 2023-11-02T18:40:27.133 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.134 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.134 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.135 DEBUG:boto:Params: {} 2023-11-02T18:40:27.135 DEBUG:boto:Token: None 2023-11-02T18:40:27.136 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.136 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c77fbdc60926b27-006543ed1b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.143 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.144 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.144 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.145 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:40:27.145 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:40:27.145 DEBUG:boto:Method: GET 2023-11-02T18:40:27.146 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:40:27.146 DEBUG:boto:Data: 2023-11-02T18:40:27.147 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.147 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.148 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.148 DEBUG:boto:Params: {} 2023-11-02T18:40:27.149 DEBUG:boto:Token: None 2023-11-02T18:40:27.149 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.150 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008db991daed499738-006543ed1b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.159 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:40:27.160 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:40:27.161 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:40:27.161 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.162 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.162 DEBUG:boto:Method: GET 2023-11-02T18:40:27.163 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.163 DEBUG:boto:Data: 2023-11-02T18:40:27.163 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.164 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.164 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.165 DEBUG:boto:Params: {} 2023-11-02T18:40:27.165 DEBUG:boto:Token: None 2023-11-02T18:40:27.166 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.166 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:46Y3YRK9KxpBuIz9mzibsZuBYek= 2023-11-02T18:40:27.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:46Y3YRK9KxpBuIz9mzibsZuBYek=', 'Content-Length': '0'} 2023-11-02T18:40:27.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002817582d08df306d-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.174 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:40:27.174 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:40:27.175 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.175 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.176 DEBUG:boto:Method: GET 2023-11-02T18:40:27.176 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.177 DEBUG:boto:Data: 2023-11-02T18:40:27.177 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.177 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.178 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.178 DEBUG:boto:Params: {} 2023-11-02T18:40:27.179 DEBUG:boto:Token: None 2023-11-02T18:40:27.179 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.180 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:46Y3YRK9KxpBuIz9mzibsZuBYek= 2023-11-02T18:40:27.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:46Y3YRK9KxpBuIz9mzibsZuBYek=', 'Content-Length': '0'} 2023-11-02T18:40:27.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deff3f769345b2a0-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.226 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:40:27.227 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:40:27.227 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.228 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.228 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.229 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.229 DEBUG:boto:Data: 2023-11-02T18:40:27.229 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.230 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.230 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.231 DEBUG:boto:Params: {} 2023-11-02T18:40:27.231 DEBUG:boto:Token: None 2023-11-02T18:40:27.232 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.232 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s2aKc9r1B0GohSSp1sspNFmoIDY= 2023-11-02T18:40:27.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s2aKc9r1B0GohSSp1sspNFmoIDY=', 'Content-Length': '0'} 2023-11-02T18:40:27.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb73736ee3748c43-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.236 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:40:27.237 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:40:27.237 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.238 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.238 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.239 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.239 DEBUG:boto:Data: 2023-11-02T18:40:27.239 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.240 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.240 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.241 DEBUG:boto:Params: {} 2023-11-02T18:40:27.241 DEBUG:boto:Token: None 2023-11-02T18:40:27.241 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s2aKc9r1B0GohSSp1sspNFmoIDY= 2023-11-02T18:40:27.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s2aKc9r1B0GohSSp1sspNFmoIDY=', 'Content-Length': '0'} 2023-11-02T18:40:27.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e57140c9476642-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:40:27.247 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:40:27.247 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:40:27.248 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.249 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.249 DEBUG:boto:Method: GET 2023-11-02T18:40:27.249 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.250 DEBUG:boto:Data: 2023-11-02T18:40:27.250 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.251 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.251 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.252 DEBUG:boto:Params: {} 2023-11-02T18:40:27.252 DEBUG:boto:Token: None 2023-11-02T18:40:27.253 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.253 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:46Y3YRK9KxpBuIz9mzibsZuBYek= 2023-11-02T18:40:27.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:46Y3YRK9KxpBuIz9mzibsZuBYek=', 'Content-Length': '0'} 2023-11-02T18:40:27.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fbf3ef08a7de6e1-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.302 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:40:27.303 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:40:27.303 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.304 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.304 DEBUG:boto:Method: GET 2023-11-02T18:40:27.305 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.305 DEBUG:boto:Data: 2023-11-02T18:40:27.306 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.306 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.307 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.307 DEBUG:boto:Params: {} 2023-11-02T18:40:27.307 DEBUG:boto:Token: None 2023-11-02T18:40:27.308 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:40:27.308 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:46Y3YRK9KxpBuIz9mzibsZuBYek= 2023-11-02T18:40:27.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:46Y3YRK9KxpBuIz9mzibsZuBYek=', 'Content-Length': '0'} 2023-11-02T18:40:27.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000826c8d153f94d864-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.313 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.313 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.314 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:40:27.314 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:40:27.315 DEBUG:boto:Method: GET 2023-11-02T18:40:27.315 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:40:27.315 DEBUG:boto:Data: 2023-11-02T18:40:27.316 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.316 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.317 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.317 DEBUG:boto:Params: {} 2023-11-02T18:40:27.318 DEBUG:boto:Token: None 2023-11-02T18:40:27.318 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deab597c8c564a06-006543ed1b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.327 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.327 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.328 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.328 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:40:27.329 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:40:27.329 DEBUG:boto:Method: GET 2023-11-02T18:40:27.330 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:40:27.330 DEBUG:boto:Data: 2023-11-02T18:40:27.330 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.331 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.331 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.332 DEBUG:boto:Params: {} 2023-11-02T18:40:27.332 DEBUG:boto:Token: None 2023-11-02T18:40:27.333 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.333 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089e458a2b55f1f74-006543ed1b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.341 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.342 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:40:27.342 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:40:27.343 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.343 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:40:27.344 DEBUG:boto:Data: 2023-11-02T18:40:27.344 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.345 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.345 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.345 DEBUG:boto:Params: {} 2023-11-02T18:40:27.346 DEBUG:boto:Token: None 2023-11-02T18:40:27.346 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj22 2023-11-02T18:40:27.347 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xgL3zNwy19GEinKCxf8KDfaBXzk= 2023-11-02T18:40:27.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xgL3zNwy19GEinKCxf8KDfaBXzk=', 'Content-Length': '0'} 2023-11-02T18:40:27.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c0eb6dd03d440e6-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.351 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:40:27.351 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:40:27.351 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.352 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:40:27.352 DEBUG:boto:Data: 2023-11-02T18:40:27.353 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.353 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.353 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.354 DEBUG:boto:Params: {} 2023-11-02T18:40:27.354 DEBUG:boto:Token: None 2023-11-02T18:40:27.355 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj22 2023-11-02T18:40:27.355 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xgL3zNwy19GEinKCxf8KDfaBXzk= 2023-11-02T18:40:27.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xgL3zNwy19GEinKCxf8KDfaBXzk=', 'Content-Length': '0'} 2023-11-02T18:40:27.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca337b7d4b160906-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:40:27.359 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:40:27.360 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:40:27.360 DEBUG:boto:Method: GET 2023-11-02T18:40:27.361 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:40:27.361 DEBUG:boto:Data: 2023-11-02T18:40:27.362 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.362 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.363 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.363 DEBUG:boto:Params: {} 2023-11-02T18:40:27.363 DEBUG:boto:Token: None 2023-11-02T18:40:27.364 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj22 2023-11-02T18:40:27.364 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fKDE/ksIEvx1Lr/4jqcMETbHSLI= 2023-11-02T18:40:27.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fKDE/ksIEvx1Lr/4jqcMETbHSLI=', 'Content-Length': '0'} 2023-11-02T18:40:27.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060984f7edcb66163-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.369 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:40:27.370 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:40:27.371 DEBUG:boto:Method: GET 2023-11-02T18:40:27.372 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:40:27.373 DEBUG:boto:Data: 2023-11-02T18:40:27.373 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.374 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.375 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.376 DEBUG:boto:Params: {} 2023-11-02T18:40:27.377 DEBUG:boto:Token: None 2023-11-02T18:40:27.378 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj22 2023-11-02T18:40:27.378 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fKDE/ksIEvx1Lr/4jqcMETbHSLI= 2023-11-02T18:40:27.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fKDE/ksIEvx1Lr/4jqcMETbHSLI=', 'Content-Length': '0'} 2023-11-02T18:40:27.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2e34a5d64c4d0be-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.426 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.428 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.428 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:40:27.429 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:40:27.429 DEBUG:boto:Method: GET 2023-11-02T18:40:27.430 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:40:27.430 DEBUG:boto:Data: 2023-11-02T18:40:27.431 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.431 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.431 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.432 DEBUG:boto:Params: {} 2023-11-02T18:40:27.432 DEBUG:boto:Token: None 2023-11-02T18:40:27.433 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.434 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bda3d746a8885f3c-006543ed1b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.440 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.441 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.442 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.442 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:40:27.442 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:40:27.443 DEBUG:boto:Method: GET 2023-11-02T18:40:27.443 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:40:27.444 DEBUG:boto:Data: 2023-11-02T18:40:27.444 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.445 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.446 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.446 DEBUG:boto:Params: {} 2023-11-02T18:40:27.447 DEBUG:boto:Token: None 2023-11-02T18:40:27.448 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.448 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd5b855a91b4ecba-006543ed1b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.457 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:40:27.458 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:40:27.459 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:40:27.459 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.460 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.460 DEBUG:boto:Method: GET 2023-11-02T18:40:27.461 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.461 DEBUG:boto:Data: 2023-11-02T18:40:27.462 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.463 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.463 DEBUG:boto:Params: {} 2023-11-02T18:40:27.464 DEBUG:boto:Token: None 2023-11-02T18:40:27.464 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y0PTjw8fcfiG4j8hKEDEB4uLDbo= 2023-11-02T18:40:27.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y0PTjw8fcfiG4j8hKEDEB4uLDbo=', 'Content-Length': '0'} 2023-11-02T18:40:27.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076cbe6ccae0d605b-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.510 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:40:27.511 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:40:27.511 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.512 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.512 DEBUG:boto:Method: GET 2023-11-02T18:40:27.513 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.513 DEBUG:boto:Data: 2023-11-02T18:40:27.514 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.514 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.514 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.515 DEBUG:boto:Params: {} 2023-11-02T18:40:27.515 DEBUG:boto:Token: None 2023-11-02T18:40:27.516 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.516 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y0PTjw8fcfiG4j8hKEDEB4uLDbo= 2023-11-02T18:40:27.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y0PTjw8fcfiG4j8hKEDEB4uLDbo=', 'Content-Length': '0'} 2023-11-02T18:40:27.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c5dbaf089c3768a-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.521 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:40:27.521 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:40:27.522 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.522 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.523 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.523 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.523 DEBUG:boto:Data: 2023-11-02T18:40:27.524 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.524 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.525 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.525 DEBUG:boto:Params: {} 2023-11-02T18:40:27.526 DEBUG:boto:Token: None 2023-11-02T18:40:27.526 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.527 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QX2NH0hETFFp4ZggCHezWfas+DQ= 2023-11-02T18:40:27.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QX2NH0hETFFp4ZggCHezWfas+DQ=', 'Content-Length': '0'} 2023-11-02T18:40:27.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081513f9475076cf7-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.531 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:40:27.531 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:40:27.532 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.532 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.532 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.533 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.533 DEBUG:boto:Data: 2023-11-02T18:40:27.534 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.534 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.534 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.535 DEBUG:boto:Params: {} 2023-11-02T18:40:27.535 DEBUG:boto:Token: None 2023-11-02T18:40:27.536 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.536 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QX2NH0hETFFp4ZggCHezWfas+DQ= 2023-11-02T18:40:27.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QX2NH0hETFFp4ZggCHezWfas+DQ=', 'Content-Length': '0'} 2023-11-02T18:40:27.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088254093ead2bdf9-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:40:27.541 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:40:27.541 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:40:27.542 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.542 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.543 DEBUG:boto:Method: GET 2023-11-02T18:40:27.543 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.544 DEBUG:boto:Data: 2023-11-02T18:40:27.544 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.544 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.545 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.545 DEBUG:boto:Params: {} 2023-11-02T18:40:27.546 DEBUG:boto:Token: None 2023-11-02T18:40:27.546 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.547 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y0PTjw8fcfiG4j8hKEDEB4uLDbo= 2023-11-02T18:40:27.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y0PTjw8fcfiG4j8hKEDEB4uLDbo=', 'Content-Length': '0'} 2023-11-02T18:40:27.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1939ca51dd71e27-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.551 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:40:27.551 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:40:27.552 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.552 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.553 DEBUG:boto:Method: GET 2023-11-02T18:40:27.553 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.553 DEBUG:boto:Data: 2023-11-02T18:40:27.554 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.555 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.555 DEBUG:boto:Params: {} 2023-11-02T18:40:27.556 DEBUG:boto:Token: None 2023-11-02T18:40:27.556 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:40:27.557 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y0PTjw8fcfiG4j8hKEDEB4uLDbo= 2023-11-02T18:40:27.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y0PTjw8fcfiG4j8hKEDEB4uLDbo=', 'Content-Length': '0'} 2023-11-02T18:40:27.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b14de2221b3d86e2-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.606 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.607 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.607 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:40:27.608 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:40:27.608 DEBUG:boto:Method: GET 2023-11-02T18:40:27.609 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:40:27.609 DEBUG:boto:Data: 2023-11-02T18:40:27.609 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.610 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.610 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.611 DEBUG:boto:Params: {} 2023-11-02T18:40:27.611 DEBUG:boto:Token: None 2023-11-02T18:40:27.612 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c05f3891557be2b9-006543ed1b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.618 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.620 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.620 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.621 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:40:27.622 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:40:27.622 DEBUG:boto:Method: GET 2023-11-02T18:40:27.623 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:40:27.623 DEBUG:boto:Data: 2023-11-02T18:40:27.624 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.624 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.625 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.625 DEBUG:boto:Params: {} 2023-11-02T18:40:27.626 DEBUG:boto:Token: None 2023-11-02T18:40:27.626 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.630 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ba853aca17c2237-006543ed1b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.638 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.639 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:40:27.640 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:40:27.640 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.641 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:40:27.641 DEBUG:boto:Data: 2023-11-02T18:40:27.642 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.642 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.643 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.643 DEBUG:boto:Params: {} 2023-11-02T18:40:27.644 DEBUG:boto:Token: None 2023-11-02T18:40:27.644 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj23 2023-11-02T18:40:27.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nUmPkUmV/mUnxjbsJ4dACvsKA4s= 2023-11-02T18:40:27.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nUmPkUmV/mUnxjbsJ4dACvsKA4s=', 'Content-Length': '0'} 2023-11-02T18:40:27.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bae6b188138ab636-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.648 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:40:27.649 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:40:27.649 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.650 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:40:27.650 DEBUG:boto:Data: 2023-11-02T18:40:27.650 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.651 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.652 DEBUG:boto:Params: {} 2023-11-02T18:40:27.652 DEBUG:boto:Token: None 2023-11-02T18:40:27.653 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj23 2023-11-02T18:40:27.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nUmPkUmV/mUnxjbsJ4dACvsKA4s= 2023-11-02T18:40:27.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nUmPkUmV/mUnxjbsJ4dACvsKA4s=', 'Content-Length': '0'} 2023-11-02T18:40:27.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000951cf379eaf833c2-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:40:27.657 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:40:27.658 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:40:27.658 DEBUG:boto:Method: GET 2023-11-02T18:40:27.659 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:40:27.659 DEBUG:boto:Data: 2023-11-02T18:40:27.659 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.660 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.660 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.661 DEBUG:boto:Params: {} 2023-11-02T18:40:27.661 DEBUG:boto:Token: None 2023-11-02T18:40:27.662 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj23 2023-11-02T18:40:27.662 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6FrOgRGPF0tJFlMNtR9+EVZcSqI= 2023-11-02T18:40:27.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6FrOgRGPF0tJFlMNtR9+EVZcSqI=', 'Content-Length': '0'} 2023-11-02T18:40:27.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc934b912242bf4d-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.706 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:40:27.707 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:40:27.707 DEBUG:boto:Method: GET 2023-11-02T18:40:27.708 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:40:27.708 DEBUG:boto:Data: 2023-11-02T18:40:27.709 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.709 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.709 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.710 DEBUG:boto:Params: {} 2023-11-02T18:40:27.710 DEBUG:boto:Token: None 2023-11-02T18:40:27.711 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj23 2023-11-02T18:40:27.711 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6FrOgRGPF0tJFlMNtR9+EVZcSqI= 2023-11-02T18:40:27.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6FrOgRGPF0tJFlMNtR9+EVZcSqI=', 'Content-Length': '0'} 2023-11-02T18:40:27.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003621cf7872f15d27-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.716 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.716 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.717 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:40:27.717 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:40:27.717 DEBUG:boto:Method: GET 2023-11-02T18:40:27.718 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:40:27.718 DEBUG:boto:Data: 2023-11-02T18:40:27.719 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.719 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.719 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.720 DEBUG:boto:Params: {} 2023-11-02T18:40:27.720 DEBUG:boto:Token: None 2023-11-02T18:40:27.721 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0716c008b114269-006543ed1b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.727 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.728 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.729 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.729 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:40:27.730 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:40:27.730 DEBUG:boto:Method: GET 2023-11-02T18:40:27.730 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:40:27.731 DEBUG:boto:Data: 2023-11-02T18:40:27.731 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.732 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.732 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.733 DEBUG:boto:Params: {} 2023-11-02T18:40:27.733 DEBUG:boto:Token: None 2023-11-02T18:40:27.734 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.734 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f12030812c23d1c1-006543ed1b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.782 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:40:27.784 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:40:27.784 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:40:27.785 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.785 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.785 DEBUG:boto:Method: GET 2023-11-02T18:40:27.786 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.786 DEBUG:boto:Data: 2023-11-02T18:40:27.787 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.787 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.788 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.788 DEBUG:boto:Params: {} 2023-11-02T18:40:27.788 DEBUG:boto:Token: None 2023-11-02T18:40:27.789 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.790 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UKRtlWaVI6IKaRRwz2973v/LBrA= 2023-11-02T18:40:27.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UKRtlWaVI6IKaRRwz2973v/LBrA=', 'Content-Length': '0'} 2023-11-02T18:40:27.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b038617dca4d15c-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.794 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:40:27.795 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:40:27.795 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.796 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.796 DEBUG:boto:Method: GET 2023-11-02T18:40:27.796 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.797 DEBUG:boto:Data: 2023-11-02T18:40:27.797 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.798 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.798 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.799 DEBUG:boto:Params: {} 2023-11-02T18:40:27.799 DEBUG:boto:Token: None 2023-11-02T18:40:27.800 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.800 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UKRtlWaVI6IKaRRwz2973v/LBrA= 2023-11-02T18:40:27.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UKRtlWaVI6IKaRRwz2973v/LBrA=', 'Content-Length': '0'} 2023-11-02T18:40:27.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf264d425a9e2dfb-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.804 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:40:27.805 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:40:27.805 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.806 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.806 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.806 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.807 DEBUG:boto:Data: 2023-11-02T18:40:27.807 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.808 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.808 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.808 DEBUG:boto:Params: {} 2023-11-02T18:40:27.809 DEBUG:boto:Token: None 2023-11-02T18:40:27.809 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.810 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uo5c6dYXzqTzmCaUYA1C+YxbKDU= 2023-11-02T18:40:27.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uo5c6dYXzqTzmCaUYA1C+YxbKDU=', 'Content-Length': '0'} 2023-11-02T18:40:27.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa76b18fa32fd6ac-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.814 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:40:27.814 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:40:27.815 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.815 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.816 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.816 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.816 DEBUG:boto:Data: 2023-11-02T18:40:27.817 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.817 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.818 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.818 DEBUG:boto:Params: {} 2023-11-02T18:40:27.818 DEBUG:boto:Token: None 2023-11-02T18:40:27.819 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.819 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uo5c6dYXzqTzmCaUYA1C+YxbKDU= 2023-11-02T18:40:27.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uo5c6dYXzqTzmCaUYA1C+YxbKDU=', 'Content-Length': '0'} 2023-11-02T18:40:27.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067cba82d37ef56b7-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:40:27.824 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:40:27.824 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:40:27.824 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.825 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.825 DEBUG:boto:Method: GET 2023-11-02T18:40:27.826 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.826 DEBUG:boto:Data: 2023-11-02T18:40:27.827 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.827 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.827 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.828 DEBUG:boto:Params: {} 2023-11-02T18:40:27.828 DEBUG:boto:Token: None 2023-11-02T18:40:27.829 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.829 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UKRtlWaVI6IKaRRwz2973v/LBrA= 2023-11-02T18:40:27.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UKRtlWaVI6IKaRRwz2973v/LBrA=', 'Content-Length': '0'} 2023-11-02T18:40:27.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084dea73c57f25f2c-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.834 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:40:27.835 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:40:27.835 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.835 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.836 DEBUG:boto:Method: GET 2023-11-02T18:40:27.836 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.837 DEBUG:boto:Data: 2023-11-02T18:40:27.837 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.838 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.838 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.838 DEBUG:boto:Params: {} 2023-11-02T18:40:27.839 DEBUG:boto:Token: None 2023-11-02T18:40:27.839 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:40:27.840 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UKRtlWaVI6IKaRRwz2973v/LBrA= 2023-11-02T18:40:27.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UKRtlWaVI6IKaRRwz2973v/LBrA=', 'Content-Length': '0'} 2023-11-02T18:40:27.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bac894aca2124dca-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.886 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.887 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.887 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:40:27.888 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:40:27.888 DEBUG:boto:Method: GET 2023-11-02T18:40:27.889 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:40:27.889 DEBUG:boto:Data: 2023-11-02T18:40:27.890 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.890 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.890 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.891 DEBUG:boto:Params: {} 2023-11-02T18:40:27.891 DEBUG:boto:Token: None 2023-11-02T18:40:27.892 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.892 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2e3d8dfcdda156f-006543ed1b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.898 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.899 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.899 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.900 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:40:27.900 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:40:27.901 DEBUG:boto:Method: GET 2023-11-02T18:40:27.901 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:40:27.901 DEBUG:boto:Data: 2023-11-02T18:40:27.902 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.902 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.903 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.903 DEBUG:boto:Params: {} 2023-11-02T18:40:27.904 DEBUG:boto:Token: None 2023-11-02T18:40:27.904 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/ 2023-11-02T18:40:27.905 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk= 2023-11-02T18:40:27.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FB2jRGEP1t68O7dcDioew2r9bhk=', 'Content-Length': '0'} 2023-11-02T18:40:27.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006710fa1c7ae693b3-006543ed1b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.915 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:27.915 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:40:27.916 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:40:27.916 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.917 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:40:27.917 DEBUG:boto:Data: 2023-11-02T18:40:27.917 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.918 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.919 DEBUG:boto:Params: {} 2023-11-02T18:40:27.919 DEBUG:boto:Token: None 2023-11-02T18:40:27.920 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj33 2023-11-02T18:40:27.920 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TLnwfRbnGKwISzMVfPVfRbtwI2U= 2023-11-02T18:40:27.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TLnwfRbnGKwISzMVfPVfRbtwI2U=', 'Content-Length': '0'} 2023-11-02T18:40:27.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003deff92f33e2190f-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.924 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:40:27.924 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:40:27.925 DEBUG:boto:Method: HEAD 2023-11-02T18:40:27.925 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:40:27.925 DEBUG:boto:Data: 2023-11-02T18:40:27.926 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.926 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.927 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.927 DEBUG:boto:Params: {} 2023-11-02T18:40:27.928 DEBUG:boto:Token: None 2023-11-02T18:40:27.928 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj33 2023-11-02T18:40:27.929 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TLnwfRbnGKwISzMVfPVfRbtwI2U= 2023-11-02T18:40:27.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TLnwfRbnGKwISzMVfPVfRbtwI2U=', 'Content-Length': '0'} 2023-11-02T18:40:27.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7a8891fb4e57ac4-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:40:27.933 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:40:27.933 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:40:27.934 DEBUG:boto:Method: GET 2023-11-02T18:40:27.934 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:40:27.935 DEBUG:boto:Data: 2023-11-02T18:40:27.935 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.935 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:27.936 DEBUG:boto:Port: 8001 2023-11-02T18:40:27.936 DEBUG:boto:Params: {} 2023-11-02T18:40:27.937 DEBUG:boto:Token: None 2023-11-02T18:40:27.937 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj33 2023-11-02T18:40:27.938 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EH5cnbXZkPpyxCBIh0JXedbpkps= 2023-11-02T18:40:27.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EH5cnbXZkPpyxCBIh0JXedbpkps=', 'Content-Length': '0'} 2023-11-02T18:40:27.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f82593f07594a5b0-006543ed1b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.986 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:40:27.987 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:40:27.987 DEBUG:boto:Method: GET 2023-11-02T18:40:27.988 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:40:27.988 DEBUG:boto:Data: 2023-11-02T18:40:27.989 DEBUG:boto:Headers: {} 2023-11-02T18:40:27.989 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:27.990 DEBUG:boto:Port: 8000 2023-11-02T18:40:27.990 DEBUG:boto:Params: {} 2023-11-02T18:40:27.991 DEBUG:boto:Token: None 2023-11-02T18:40:27.991 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:27 GMT /ibtljn-59/obj33 2023-11-02T18:40:27.991 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EH5cnbXZkPpyxCBIh0JXedbpkps= 2023-11-02T18:40:27.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EH5cnbXZkPpyxCBIh0JXedbpkps=', 'Content-Length': '0'} 2023-11-02T18:40:27.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d21c95ff5d08825d-006543ed1b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:27.996 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:27.997 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:27.997 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:40:27.998 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:40:27.998 DEBUG:boto:Method: GET 2023-11-02T18:40:27.999 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:40:27.999 DEBUG:boto:Data: 2023-11-02T18:40:28.000 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.000 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.001 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.001 DEBUG:boto:Params: {} 2023-11-02T18:40:28.001 DEBUG:boto:Token: None 2023-11-02T18:40:28.002 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d029aacd7e8f511-006543ed1b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.009 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.010 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.010 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.011 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:40:28.011 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:40:28.012 DEBUG:boto:Method: GET 2023-11-02T18:40:28.012 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:40:28.013 DEBUG:boto:Data: 2023-11-02T18:40:28.013 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.014 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.014 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.014 DEBUG:boto:Params: {} 2023-11-02T18:40:28.015 DEBUG:boto:Token: None 2023-11-02T18:40:28.016 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007571e6a43bf499c3-006543ed1c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.066 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:40:28.068 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:40:28.068 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:40:28.069 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.069 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.070 DEBUG:boto:Method: GET 2023-11-02T18:40:28.070 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.071 DEBUG:boto:Data: 2023-11-02T18:40:28.071 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.071 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.073 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.073 DEBUG:boto:Params: {} 2023-11-02T18:40:28.074 DEBUG:boto:Token: None 2023-11-02T18:40:28.074 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.075 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gB8OTg3Hhzgllhbsxn/m6VvhwLA= 2023-11-02T18:40:28.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gB8OTg3Hhzgllhbsxn/m6VvhwLA=', 'Content-Length': '0'} 2023-11-02T18:40:28.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5d244cb07fe7af5-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.079 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:40:28.080 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:40:28.080 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.081 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.081 DEBUG:boto:Method: GET 2023-11-02T18:40:28.082 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.082 DEBUG:boto:Data: 2023-11-02T18:40:28.083 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.083 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.084 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.084 DEBUG:boto:Params: {} 2023-11-02T18:40:28.084 DEBUG:boto:Token: None 2023-11-02T18:40:28.085 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.085 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gB8OTg3Hhzgllhbsxn/m6VvhwLA= 2023-11-02T18:40:28.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gB8OTg3Hhzgllhbsxn/m6VvhwLA=', 'Content-Length': '0'} 2023-11-02T18:40:28.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000842793ccf414f35b-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.090 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:40:28.090 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:40:28.091 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.091 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.092 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.092 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.092 DEBUG:boto:Data: 2023-11-02T18:40:28.093 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.093 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.094 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.094 DEBUG:boto:Params: {} 2023-11-02T18:40:28.095 DEBUG:boto:Token: None 2023-11-02T18:40:28.095 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.096 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I8I02CcHU0OJLTXR1X0x9rWN4Sk= 2023-11-02T18:40:28.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I8I02CcHU0OJLTXR1X0x9rWN4Sk=', 'Content-Length': '0'} 2023-11-02T18:40:28.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc7d7eb0b41e00b2-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.100 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:40:28.100 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:40:28.101 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.101 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.102 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.102 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.102 DEBUG:boto:Data: 2023-11-02T18:40:28.103 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.103 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.104 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.104 DEBUG:boto:Params: {} 2023-11-02T18:40:28.105 DEBUG:boto:Token: None 2023-11-02T18:40:28.105 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.106 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I8I02CcHU0OJLTXR1X0x9rWN4Sk= 2023-11-02T18:40:28.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I8I02CcHU0OJLTXR1X0x9rWN4Sk=', 'Content-Length': '0'} 2023-11-02T18:40:28.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000939cd1676a7ee15f-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:40:28.110 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:40:28.111 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:40:28.111 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.112 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.112 DEBUG:boto:Method: GET 2023-11-02T18:40:28.113 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.113 DEBUG:boto:Data: 2023-11-02T18:40:28.113 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.114 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.114 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.115 DEBUG:boto:Params: {} 2023-11-02T18:40:28.115 DEBUG:boto:Token: None 2023-11-02T18:40:28.116 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.116 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gB8OTg3Hhzgllhbsxn/m6VvhwLA= 2023-11-02T18:40:28.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gB8OTg3Hhzgllhbsxn/m6VvhwLA=', 'Content-Length': '0'} 2023-11-02T18:40:28.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e862b56db0d2f6-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.162 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:40:28.163 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:40:28.164 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.164 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.164 DEBUG:boto:Method: GET 2023-11-02T18:40:28.165 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.165 DEBUG:boto:Data: 2023-11-02T18:40:28.166 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.167 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.167 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.167 DEBUG:boto:Params: {} 2023-11-02T18:40:28.168 DEBUG:boto:Token: None 2023-11-02T18:40:28.168 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:40:28.169 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gB8OTg3Hhzgllhbsxn/m6VvhwLA= 2023-11-02T18:40:28.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gB8OTg3Hhzgllhbsxn/m6VvhwLA=', 'Content-Length': '0'} 2023-11-02T18:40:28.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3dc8fb17b4cb733-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.173 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.174 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.174 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:40:28.175 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:40:28.175 DEBUG:boto:Method: GET 2023-11-02T18:40:28.176 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:40:28.176 DEBUG:boto:Data: 2023-11-02T18:40:28.177 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.177 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.177 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.178 DEBUG:boto:Params: {} 2023-11-02T18:40:28.178 DEBUG:boto:Token: None 2023-11-02T18:40:28.179 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.179 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007af09d0529e8fa70-006543ed1c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.185 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.186 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.187 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.187 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:40:28.187 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:40:28.188 DEBUG:boto:Method: GET 2023-11-02T18:40:28.188 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:40:28.189 DEBUG:boto:Data: 2023-11-02T18:40:28.189 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.190 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.190 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.191 DEBUG:boto:Params: {} 2023-11-02T18:40:28.191 DEBUG:boto:Token: None 2023-11-02T18:40:28.192 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.192 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeede831ce997513-006543ed1c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.200 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.201 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:40:28.201 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:40:28.202 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.202 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:40:28.203 DEBUG:boto:Data: 2023-11-02T18:40:28.203 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.204 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.204 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.205 DEBUG:boto:Params: {} 2023-11-02T18:40:28.205 DEBUG:boto:Token: None 2023-11-02T18:40:28.206 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj34 2023-11-02T18:40:28.206 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ALZcXHPFPT46VXwzCOckqicBiTw= 2023-11-02T18:40:28.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ALZcXHPFPT46VXwzCOckqicBiTw=', 'Content-Length': '0'} 2023-11-02T18:40:28.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d3c87cc2cfda8b9-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.210 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:40:28.210 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:40:28.211 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.211 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:40:28.212 DEBUG:boto:Data: 2023-11-02T18:40:28.212 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.213 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.213 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.214 DEBUG:boto:Params: {} 2023-11-02T18:40:28.214 DEBUG:boto:Token: None 2023-11-02T18:40:28.215 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj34 2023-11-02T18:40:28.215 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ALZcXHPFPT46VXwzCOckqicBiTw= 2023-11-02T18:40:28.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ALZcXHPFPT46VXwzCOckqicBiTw=', 'Content-Length': '0'} 2023-11-02T18:40:28.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052973fd945661caf-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:40:28.220 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:40:28.220 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:40:28.221 DEBUG:boto:Method: GET 2023-11-02T18:40:28.221 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:40:28.222 DEBUG:boto:Data: 2023-11-02T18:40:28.222 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.223 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.224 DEBUG:boto:Params: {} 2023-11-02T18:40:28.224 DEBUG:boto:Token: None 2023-11-02T18:40:28.225 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj34 2023-11-02T18:40:28.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u7JqyD0qWLTQcrHq7G6xboR7yYk= 2023-11-02T18:40:28.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u7JqyD0qWLTQcrHq7G6xboR7yYk=', 'Content-Length': '0'} 2023-11-02T18:40:28.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000868518abfe1b7b3e-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.229 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:40:28.230 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:40:28.231 DEBUG:boto:Method: GET 2023-11-02T18:40:28.231 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:40:28.232 DEBUG:boto:Data: 2023-11-02T18:40:28.232 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.233 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.233 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.233 DEBUG:boto:Params: {} 2023-11-02T18:40:28.234 DEBUG:boto:Token: None 2023-11-02T18:40:28.235 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj34 2023-11-02T18:40:28.235 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u7JqyD0qWLTQcrHq7G6xboR7yYk= 2023-11-02T18:40:28.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u7JqyD0qWLTQcrHq7G6xboR7yYk=', 'Content-Length': '0'} 2023-11-02T18:40:28.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000459af086e7dfa85e-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.286 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.287 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.288 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:40:28.288 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:40:28.288 DEBUG:boto:Method: GET 2023-11-02T18:40:28.289 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:40:28.289 DEBUG:boto:Data: 2023-11-02T18:40:28.290 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.290 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.291 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.291 DEBUG:boto:Params: {} 2023-11-02T18:40:28.292 DEBUG:boto:Token: None 2023-11-02T18:40:28.292 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.293 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b35ba349bd0ed68e-006543ed1c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.299 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.299 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.300 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.300 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:40:28.301 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:40:28.301 DEBUG:boto:Method: GET 2023-11-02T18:40:28.302 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:40:28.302 DEBUG:boto:Data: 2023-11-02T18:40:28.303 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.304 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.304 DEBUG:boto:Params: {} 2023-11-02T18:40:28.304 DEBUG:boto:Token: None 2023-11-02T18:40:28.305 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7726d649510d5e8-006543ed1c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.313 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:40:28.314 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:40:28.315 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:40:28.315 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.316 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.316 DEBUG:boto:Method: GET 2023-11-02T18:40:28.317 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.317 DEBUG:boto:Data: 2023-11-02T18:40:28.318 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.318 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.319 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.319 DEBUG:boto:Params: {} 2023-11-02T18:40:28.320 DEBUG:boto:Token: None 2023-11-02T18:40:28.320 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.321 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FdGXOjBFMUJvQJ/EVyI36oC+FSA= 2023-11-02T18:40:28.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FdGXOjBFMUJvQJ/EVyI36oC+FSA=', 'Content-Length': '0'} 2023-11-02T18:40:28.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a55bc97c44a568a5-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.370 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:40:28.371 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:40:28.372 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.372 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.372 DEBUG:boto:Method: GET 2023-11-02T18:40:28.373 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.373 DEBUG:boto:Data: 2023-11-02T18:40:28.374 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.374 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.375 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.375 DEBUG:boto:Params: {} 2023-11-02T18:40:28.376 DEBUG:boto:Token: None 2023-11-02T18:40:28.376 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.377 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FdGXOjBFMUJvQJ/EVyI36oC+FSA= 2023-11-02T18:40:28.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FdGXOjBFMUJvQJ/EVyI36oC+FSA=', 'Content-Length': '0'} 2023-11-02T18:40:28.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042cfe3a8319811b0-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.381 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:40:28.382 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:40:28.382 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.382 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.383 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.383 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.384 DEBUG:boto:Data: 2023-11-02T18:40:28.384 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.385 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.385 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.385 DEBUG:boto:Params: {} 2023-11-02T18:40:28.386 DEBUG:boto:Token: None 2023-11-02T18:40:28.386 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.387 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n0G8GPZyQaeu5JtdeBChZUDZ2tM= 2023-11-02T18:40:28.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n0G8GPZyQaeu5JtdeBChZUDZ2tM=', 'Content-Length': '0'} 2023-11-02T18:40:28.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042ecdf030e984b1c-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.391 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:40:28.391 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:40:28.392 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.392 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.393 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.393 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.394 DEBUG:boto:Data: 2023-11-02T18:40:28.394 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.395 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.395 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.395 DEBUG:boto:Params: {} 2023-11-02T18:40:28.396 DEBUG:boto:Token: None 2023-11-02T18:40:28.396 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.397 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n0G8GPZyQaeu5JtdeBChZUDZ2tM= 2023-11-02T18:40:28.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n0G8GPZyQaeu5JtdeBChZUDZ2tM=', 'Content-Length': '0'} 2023-11-02T18:40:28.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d6688d397a98018-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:40:28.401 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:40:28.402 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:40:28.402 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.403 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.403 DEBUG:boto:Method: GET 2023-11-02T18:40:28.404 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.404 DEBUG:boto:Data: 2023-11-02T18:40:28.405 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.405 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.405 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.406 DEBUG:boto:Params: {} 2023-11-02T18:40:28.406 DEBUG:boto:Token: None 2023-11-02T18:40:28.407 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.408 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FdGXOjBFMUJvQJ/EVyI36oC+FSA= 2023-11-02T18:40:28.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FdGXOjBFMUJvQJ/EVyI36oC+FSA=', 'Content-Length': '0'} 2023-11-02T18:40:28.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f38ede9b033975c4-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.412 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:40:28.412 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:40:28.413 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.413 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.413 DEBUG:boto:Method: GET 2023-11-02T18:40:28.414 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.414 DEBUG:boto:Data: 2023-11-02T18:40:28.415 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.415 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.416 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.416 DEBUG:boto:Params: {} 2023-11-02T18:40:28.417 DEBUG:boto:Token: None 2023-11-02T18:40:28.417 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:40:28.418 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FdGXOjBFMUJvQJ/EVyI36oC+FSA= 2023-11-02T18:40:28.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FdGXOjBFMUJvQJ/EVyI36oC+FSA=', 'Content-Length': '0'} 2023-11-02T18:40:28.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000155c67a6f5c0c2d5-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.422 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.423 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.423 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:40:28.424 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:40:28.424 DEBUG:boto:Method: GET 2023-11-02T18:40:28.424 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:40:28.425 DEBUG:boto:Data: 2023-11-02T18:40:28.425 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.426 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.427 DEBUG:boto:Params: {} 2023-11-02T18:40:28.427 DEBUG:boto:Token: None 2023-11-02T18:40:28.428 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea2c2beecb4380c6-006543ed1c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.474 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.475 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.476 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.476 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:40:28.477 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:40:28.477 DEBUG:boto:Method: GET 2023-11-02T18:40:28.477 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:40:28.478 DEBUG:boto:Data: 2023-11-02T18:40:28.478 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.479 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.479 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.480 DEBUG:boto:Params: {} 2023-11-02T18:40:28.480 DEBUG:boto:Token: None 2023-11-02T18:40:28.481 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.481 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c86426632cb6d1a-006543ed1c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.490 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.491 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:40:28.491 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:40:28.492 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.492 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:40:28.492 DEBUG:boto:Data: 2023-11-02T18:40:28.493 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.494 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.494 DEBUG:boto:Params: {} 2023-11-02T18:40:28.495 DEBUG:boto:Token: None 2023-11-02T18:40:28.495 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj38 2023-11-02T18:40:28.495 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aojFv5/smFB/GMr3DLZv4vKpeb8= 2023-11-02T18:40:28.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aojFv5/smFB/GMr3DLZv4vKpeb8=', 'Content-Length': '0'} 2023-11-02T18:40:28.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dda157a9a0a41f4-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.500 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:40:28.500 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:40:28.501 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.501 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:40:28.502 DEBUG:boto:Data: 2023-11-02T18:40:28.502 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.502 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.503 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.503 DEBUG:boto:Params: {} 2023-11-02T18:40:28.504 DEBUG:boto:Token: None 2023-11-02T18:40:28.504 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj38 2023-11-02T18:40:28.505 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aojFv5/smFB/GMr3DLZv4vKpeb8= 2023-11-02T18:40:28.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aojFv5/smFB/GMr3DLZv4vKpeb8=', 'Content-Length': '0'} 2023-11-02T18:40:28.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004eeba3b164e4cc3-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:40:28.509 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:40:28.510 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:40:28.510 DEBUG:boto:Method: GET 2023-11-02T18:40:28.510 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:40:28.511 DEBUG:boto:Data: 2023-11-02T18:40:28.511 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.512 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.512 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.513 DEBUG:boto:Params: {} 2023-11-02T18:40:28.513 DEBUG:boto:Token: None 2023-11-02T18:40:28.514 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj38 2023-11-02T18:40:28.514 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DV8QSkXlDLb/20Og0gck3aOu+T0= 2023-11-02T18:40:28.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DV8QSkXlDLb/20Og0gck3aOu+T0=', 'Content-Length': '0'} 2023-11-02T18:40:28.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d17a19b308756e19-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.518 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:40:28.519 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:40:28.519 DEBUG:boto:Method: GET 2023-11-02T18:40:28.519 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:40:28.520 DEBUG:boto:Data: 2023-11-02T18:40:28.520 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.521 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.521 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.522 DEBUG:boto:Params: {} 2023-11-02T18:40:28.522 DEBUG:boto:Token: None 2023-11-02T18:40:28.523 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj38 2023-11-02T18:40:28.523 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DV8QSkXlDLb/20Og0gck3aOu+T0= 2023-11-02T18:40:28.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DV8QSkXlDLb/20Og0gck3aOu+T0=', 'Content-Length': '0'} 2023-11-02T18:40:28.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088adabb4e113990f-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.570 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.571 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.571 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:40:28.572 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:40:28.572 DEBUG:boto:Method: GET 2023-11-02T18:40:28.573 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:40:28.573 DEBUG:boto:Data: 2023-11-02T18:40:28.573 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.574 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.575 DEBUG:boto:Params: {} 2023-11-02T18:40:28.575 DEBUG:boto:Token: None 2023-11-02T18:40:28.576 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.576 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b74f7d32aa0e964-006543ed1c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.582 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.583 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.584 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.584 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:40:28.584 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:40:28.585 DEBUG:boto:Method: GET 2023-11-02T18:40:28.585 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:40:28.586 DEBUG:boto:Data: 2023-11-02T18:40:28.586 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.587 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.587 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.587 DEBUG:boto:Params: {} 2023-11-02T18:40:28.588 DEBUG:boto:Token: None 2023-11-02T18:40:28.588 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.589 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a55989ca9d672ad4-006543ed1c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.597 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:40:28.599 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:40:28.599 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:40:28.599 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.600 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.600 DEBUG:boto:Method: GET 2023-11-02T18:40:28.601 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.601 DEBUG:boto:Data: 2023-11-02T18:40:28.602 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.602 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.603 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.603 DEBUG:boto:Params: {} 2023-11-02T18:40:28.604 DEBUG:boto:Token: None 2023-11-02T18:40:28.604 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.605 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4sa15kmIoIx4mgzf/DyrL1zLUYw= 2023-11-02T18:40:28.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4sa15kmIoIx4mgzf/DyrL1zLUYw=', 'Content-Length': '0'} 2023-11-02T18:40:28.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078d822de0853725e-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.651 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:40:28.651 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:40:28.652 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.652 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.653 DEBUG:boto:Method: GET 2023-11-02T18:40:28.653 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.654 DEBUG:boto:Data: 2023-11-02T18:40:28.654 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.655 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.655 DEBUG:boto:Params: {} 2023-11-02T18:40:28.656 DEBUG:boto:Token: None 2023-11-02T18:40:28.656 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.657 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4sa15kmIoIx4mgzf/DyrL1zLUYw= 2023-11-02T18:40:28.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4sa15kmIoIx4mgzf/DyrL1zLUYw=', 'Content-Length': '0'} 2023-11-02T18:40:28.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000870546963aa77f65-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.661 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:40:28.662 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:40:28.662 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.666 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.668 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.668 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.669 DEBUG:boto:Data: 2023-11-02T18:40:28.669 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.670 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.670 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.670 DEBUG:boto:Params: {} 2023-11-02T18:40:28.671 DEBUG:boto:Token: None 2023-11-02T18:40:28.671 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.672 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0qJXuLpt1/zkQCkI3F5pUPzD17o= 2023-11-02T18:40:28.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0qJXuLpt1/zkQCkI3F5pUPzD17o=', 'Content-Length': '0'} 2023-11-02T18:40:28.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5283c9534a56ed9-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.677 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:40:28.677 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:40:28.678 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.678 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.678 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.679 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.679 DEBUG:boto:Data: 2023-11-02T18:40:28.680 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.681 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.681 DEBUG:boto:Params: {} 2023-11-02T18:40:28.682 DEBUG:boto:Token: None 2023-11-02T18:40:28.682 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.683 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0qJXuLpt1/zkQCkI3F5pUPzD17o= 2023-11-02T18:40:28.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0qJXuLpt1/zkQCkI3F5pUPzD17o=', 'Content-Length': '0'} 2023-11-02T18:40:28.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd3f21eec2a08f84-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:40:28.687 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:40:28.688 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:40:28.688 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.689 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.689 DEBUG:boto:Method: GET 2023-11-02T18:40:28.689 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.690 DEBUG:boto:Data: 2023-11-02T18:40:28.691 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.691 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.692 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.692 DEBUG:boto:Params: {} 2023-11-02T18:40:28.693 DEBUG:boto:Token: None 2023-11-02T18:40:28.693 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.694 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4sa15kmIoIx4mgzf/DyrL1zLUYw= 2023-11-02T18:40:28.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4sa15kmIoIx4mgzf/DyrL1zLUYw=', 'Content-Length': '0'} 2023-11-02T18:40:28.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008869580dd684b3dd-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.698 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:40:28.699 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:40:28.699 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.699 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.700 DEBUG:boto:Method: GET 2023-11-02T18:40:28.700 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.701 DEBUG:boto:Data: 2023-11-02T18:40:28.701 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.702 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.702 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.703 DEBUG:boto:Params: {} 2023-11-02T18:40:28.703 DEBUG:boto:Token: None 2023-11-02T18:40:28.703 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:40:28.704 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4sa15kmIoIx4mgzf/DyrL1zLUYw= 2023-11-02T18:40:28.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4sa15kmIoIx4mgzf/DyrL1zLUYw=', 'Content-Length': '0'} 2023-11-02T18:40:28.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb162a254bfce0f7-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.708 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.709 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.709 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:40:28.710 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:40:28.710 DEBUG:boto:Method: GET 2023-11-02T18:40:28.711 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:40:28.711 DEBUG:boto:Data: 2023-11-02T18:40:28.712 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.712 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.713 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.713 DEBUG:boto:Params: {} 2023-11-02T18:40:28.714 DEBUG:boto:Token: None 2023-11-02T18:40:28.714 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.715 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b10e3a98f61bfe07-006543ed1c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.721 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.723 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.723 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.724 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:40:28.724 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:40:28.725 DEBUG:boto:Method: GET 2023-11-02T18:40:28.725 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:40:28.726 DEBUG:boto:Data: 2023-11-02T18:40:28.726 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.726 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.727 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.727 DEBUG:boto:Params: {} 2023-11-02T18:40:28.728 DEBUG:boto:Token: None 2023-11-02T18:40:28.728 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.729 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cc219c103498386-006543ed1c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.736 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.737 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:40:28.738 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:40:28.738 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.739 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:40:28.739 DEBUG:boto:Data: 2023-11-02T18:40:28.740 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.740 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.740 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.741 DEBUG:boto:Params: {} 2023-11-02T18:40:28.741 DEBUG:boto:Token: None 2023-11-02T18:40:28.742 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj44 2023-11-02T18:40:28.742 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3WpPWp3ThGRDqSWAwYM16d2RtP8= 2023-11-02T18:40:28.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3WpPWp3ThGRDqSWAwYM16d2RtP8=', 'Content-Length': '0'} 2023-11-02T18:40:28.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a2b1b593f752146-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.746 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:40:28.747 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:40:28.748 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.748 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:40:28.750 DEBUG:boto:Data: 2023-11-02T18:40:28.751 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.751 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.752 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.752 DEBUG:boto:Params: {} 2023-11-02T18:40:28.753 DEBUG:boto:Token: None 2023-11-02T18:40:28.753 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj44 2023-11-02T18:40:28.754 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3WpPWp3ThGRDqSWAwYM16d2RtP8= 2023-11-02T18:40:28.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3WpPWp3ThGRDqSWAwYM16d2RtP8=', 'Content-Length': '0'} 2023-11-02T18:40:28.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a5dafcb100c1b8f-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:40:28.758 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:40:28.759 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:40:28.759 DEBUG:boto:Method: GET 2023-11-02T18:40:28.759 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:40:28.760 DEBUG:boto:Data: 2023-11-02T18:40:28.760 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.761 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.761 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.762 DEBUG:boto:Params: {} 2023-11-02T18:40:28.762 DEBUG:boto:Token: None 2023-11-02T18:40:28.763 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj44 2023-11-02T18:40:28.763 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KksDvV9DCu4B8HMKO8W92H9eLo0= 2023-11-02T18:40:28.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KksDvV9DCu4B8HMKO8W92H9eLo0=', 'Content-Length': '0'} 2023-11-02T18:40:28.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4b814bb4fd46533-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.810 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:40:28.811 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:40:28.811 DEBUG:boto:Method: GET 2023-11-02T18:40:28.812 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:40:28.812 DEBUG:boto:Data: 2023-11-02T18:40:28.813 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.813 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.814 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.814 DEBUG:boto:Params: {} 2023-11-02T18:40:28.815 DEBUG:boto:Token: None 2023-11-02T18:40:28.815 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj44 2023-11-02T18:40:28.816 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KksDvV9DCu4B8HMKO8W92H9eLo0= 2023-11-02T18:40:28.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KksDvV9DCu4B8HMKO8W92H9eLo0=', 'Content-Length': '0'} 2023-11-02T18:40:28.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008430d3a6b26db4c3-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.820 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.821 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.821 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:40:28.822 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:40:28.822 DEBUG:boto:Method: GET 2023-11-02T18:40:28.823 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:40:28.823 DEBUG:boto:Data: 2023-11-02T18:40:28.823 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.824 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.824 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.825 DEBUG:boto:Params: {} 2023-11-02T18:40:28.825 DEBUG:boto:Token: None 2023-11-02T18:40:28.826 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.826 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b77626027d9d0d98-006543ed1c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.833 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.834 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.834 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.835 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:40:28.835 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:40:28.835 DEBUG:boto:Method: GET 2023-11-02T18:40:28.836 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:40:28.836 DEBUG:boto:Data: 2023-11-02T18:40:28.837 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.837 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.837 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.838 DEBUG:boto:Params: {} 2023-11-02T18:40:28.838 DEBUG:boto:Token: None 2023-11-02T18:40:28.839 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.839 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8dd31dde078731b-006543ed1c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.848 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:40:28.849 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:40:28.850 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:40:28.850 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.850 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.851 DEBUG:boto:Method: GET 2023-11-02T18:40:28.851 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.852 DEBUG:boto:Data: 2023-11-02T18:40:28.852 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.853 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.853 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.854 DEBUG:boto:Params: {} 2023-11-02T18:40:28.854 DEBUG:boto:Token: None 2023-11-02T18:40:28.855 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.855 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:keTJMKliyUdTUpYIuulwMMaUhMw= 2023-11-02T18:40:28.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:keTJMKliyUdTUpYIuulwMMaUhMw=', 'Content-Length': '0'} 2023-11-02T18:40:28.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f2da48ede68268f-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.859 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:40:28.859 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:40:28.860 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.860 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.861 DEBUG:boto:Method: GET 2023-11-02T18:40:28.861 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.862 DEBUG:boto:Data: 2023-11-02T18:40:28.862 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.862 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.863 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.863 DEBUG:boto:Params: {} 2023-11-02T18:40:28.864 DEBUG:boto:Token: None 2023-11-02T18:40:28.864 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.865 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:keTJMKliyUdTUpYIuulwMMaUhMw= 2023-11-02T18:40:28.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:keTJMKliyUdTUpYIuulwMMaUhMw=', 'Content-Length': '0'} 2023-11-02T18:40:28.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e947c0d9268d021-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.869 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:40:28.869 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:40:28.870 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.870 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.873 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.873 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.874 DEBUG:boto:Data: 2023-11-02T18:40:28.874 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.875 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.875 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.876 DEBUG:boto:Params: {} 2023-11-02T18:40:28.876 DEBUG:boto:Token: None 2023-11-02T18:40:28.877 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3lQI5cm/CaNY4isCSbkx4ysBR5w= 2023-11-02T18:40:28.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3lQI5cm/CaNY4isCSbkx4ysBR5w=', 'Content-Length': '0'} 2023-11-02T18:40:28.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d9d4b592d080b67-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.881 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:40:28.881 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:40:28.882 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.882 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.883 DEBUG:boto:Method: HEAD 2023-11-02T18:40:28.883 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.884 DEBUG:boto:Data: 2023-11-02T18:40:28.884 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.884 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.885 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.885 DEBUG:boto:Params: {} 2023-11-02T18:40:28.886 DEBUG:boto:Token: None 2023-11-02T18:40:28.886 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.887 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3lQI5cm/CaNY4isCSbkx4ysBR5w= 2023-11-02T18:40:28.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3lQI5cm/CaNY4isCSbkx4ysBR5w=', 'Content-Length': '0'} 2023-11-02T18:40:28.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dd534ff99c166c9-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:40:28.891 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:40:28.892 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:40:28.892 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.892 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.893 DEBUG:boto:Method: GET 2023-11-02T18:40:28.893 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.894 DEBUG:boto:Data: 2023-11-02T18:40:28.894 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.895 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.895 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.895 DEBUG:boto:Params: {} 2023-11-02T18:40:28.896 DEBUG:boto:Token: None 2023-11-02T18:40:28.896 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.897 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:keTJMKliyUdTUpYIuulwMMaUhMw= 2023-11-02T18:40:28.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:keTJMKliyUdTUpYIuulwMMaUhMw=', 'Content-Length': '0'} 2023-11-02T18:40:28.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f277794b39c3edd-006543ed1c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.946 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:40:28.947 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:40:28.947 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.948 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.948 DEBUG:boto:Method: GET 2023-11-02T18:40:28.949 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.949 DEBUG:boto:Data: 2023-11-02T18:40:28.950 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.950 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.951 DEBUG:boto:Params: {} 2023-11-02T18:40:28.951 DEBUG:boto:Token: None 2023-11-02T18:40:28.952 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:40:28.952 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:keTJMKliyUdTUpYIuulwMMaUhMw= 2023-11-02T18:40:28.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:keTJMKliyUdTUpYIuulwMMaUhMw=', 'Content-Length': '0'} 2023-11-02T18:40:28.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002236f94b05761712-006543ed1c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.957 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.957 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.958 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:40:28.958 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:40:28.959 DEBUG:boto:Method: GET 2023-11-02T18:40:28.959 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:40:28.960 DEBUG:boto:Data: 2023-11-02T18:40:28.960 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.961 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:28.961 DEBUG:boto:Port: 8001 2023-11-02T18:40:28.961 DEBUG:boto:Params: {} 2023-11-02T18:40:28.962 DEBUG:boto:Token: None 2023-11-02T18:40:28.962 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.963 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004741e6a85215f43f-006543ed1c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:28.970 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:28.970 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:28.971 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:28.971 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:40:28.972 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:40:28.972 DEBUG:boto:Method: GET 2023-11-02T18:40:28.973 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:40:28.973 DEBUG:boto:Data: 2023-11-02T18:40:28.973 DEBUG:boto:Headers: {} 2023-11-02T18:40:28.974 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:28.974 DEBUG:boto:Port: 8000 2023-11-02T18:40:28.975 DEBUG:boto:Params: {} 2023-11-02T18:40:28.975 DEBUG:boto:Token: None 2023-11-02T18:40:28.976 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:28 GMT /ibtljn-59/ 2023-11-02T18:40:28.976 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w= 2023-11-02T18:40:28.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ws3EbX1Qo/qw+cCp5PefKl6U46w=', 'Content-Length': '0'} 2023-11-02T18:40:28.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000433affb89feaa063-006543ed1c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.027 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.028 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:40:29.029 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:40:29.030 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.030 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:40:29.030 DEBUG:boto:Data: 2023-11-02T18:40:29.031 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.031 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.032 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.032 DEBUG:boto:Params: {} 2023-11-02T18:40:29.033 DEBUG:boto:Token: None 2023-11-02T18:40:29.033 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj46 2023-11-02T18:40:29.034 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FAVgvKIzd1+gILWeZkCtGSYUpsE= 2023-11-02T18:40:29.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FAVgvKIzd1+gILWeZkCtGSYUpsE=', 'Content-Length': '0'} 2023-11-02T18:40:29.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbae1307d4116603-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.038 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:40:29.039 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:40:29.039 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.040 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:40:29.040 DEBUG:boto:Data: 2023-11-02T18:40:29.040 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.041 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.041 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.042 DEBUG:boto:Params: {} 2023-11-02T18:40:29.042 DEBUG:boto:Token: None 2023-11-02T18:40:29.043 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj46 2023-11-02T18:40:29.043 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FAVgvKIzd1+gILWeZkCtGSYUpsE= 2023-11-02T18:40:29.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FAVgvKIzd1+gILWeZkCtGSYUpsE=', 'Content-Length': '0'} 2023-11-02T18:40:29.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d82a2e2e11fabb9-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:40:29.048 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:40:29.048 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:40:29.049 DEBUG:boto:Method: GET 2023-11-02T18:40:29.049 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:40:29.050 DEBUG:boto:Data: 2023-11-02T18:40:29.051 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.051 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.052 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.053 DEBUG:boto:Params: {} 2023-11-02T18:40:29.053 DEBUG:boto:Token: None 2023-11-02T18:40:29.054 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj46 2023-11-02T18:40:29.054 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SEiqVhrtP207x0nkXsbN5LRxntc= 2023-11-02T18:40:29.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SEiqVhrtP207x0nkXsbN5LRxntc=', 'Content-Length': '0'} 2023-11-02T18:40:29.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8e1cf46de59f9f1-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.059 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:40:29.060 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:40:29.061 DEBUG:boto:Method: GET 2023-11-02T18:40:29.061 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:40:29.062 DEBUG:boto:Data: 2023-11-02T18:40:29.062 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.063 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.063 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.063 DEBUG:boto:Params: {} 2023-11-02T18:40:29.064 DEBUG:boto:Token: None 2023-11-02T18:40:29.064 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj46 2023-11-02T18:40:29.065 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SEiqVhrtP207x0nkXsbN5LRxntc= 2023-11-02T18:40:29.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SEiqVhrtP207x0nkXsbN5LRxntc=', 'Content-Length': '0'} 2023-11-02T18:40:29.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5575a29fe6644ed-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.069 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.070 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.070 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:40:29.071 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:40:29.071 DEBUG:boto:Method: GET 2023-11-02T18:40:29.072 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:40:29.072 DEBUG:boto:Data: 2023-11-02T18:40:29.072 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.073 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.073 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.074 DEBUG:boto:Params: {} 2023-11-02T18:40:29.074 DEBUG:boto:Token: None 2023-11-02T18:40:29.077 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.078 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016dee4a8ab70ecf6-006543ed1d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.084 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.085 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.085 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.085 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:40:29.086 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:40:29.086 DEBUG:boto:Method: GET 2023-11-02T18:40:29.087 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:40:29.087 DEBUG:boto:Data: 2023-11-02T18:40:29.088 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.088 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.088 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.089 DEBUG:boto:Params: {} 2023-11-02T18:40:29.089 DEBUG:boto:Token: None 2023-11-02T18:40:29.090 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.090 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3a44c0d206bfbec-006543ed1d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.099 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:40:29.101 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:40:29.101 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:40:29.101 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.102 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.102 DEBUG:boto:Method: GET 2023-11-02T18:40:29.103 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.103 DEBUG:boto:Data: 2023-11-02T18:40:29.104 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.104 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.105 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.105 DEBUG:boto:Params: {} 2023-11-02T18:40:29.105 DEBUG:boto:Token: None 2023-11-02T18:40:29.106 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.106 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gL+Bxak+Dm7Fhm/jsinmpLaYdSw= 2023-11-02T18:40:29.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gL+Bxak+Dm7Fhm/jsinmpLaYdSw=', 'Content-Length': '0'} 2023-11-02T18:40:29.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf2957c209df090f-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.150 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:40:29.151 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:40:29.151 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.152 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.152 DEBUG:boto:Method: GET 2023-11-02T18:40:29.153 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.153 DEBUG:boto:Data: 2023-11-02T18:40:29.154 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.154 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.154 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.155 DEBUG:boto:Params: {} 2023-11-02T18:40:29.155 DEBUG:boto:Token: None 2023-11-02T18:40:29.156 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.156 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gL+Bxak+Dm7Fhm/jsinmpLaYdSw= 2023-11-02T18:40:29.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gL+Bxak+Dm7Fhm/jsinmpLaYdSw=', 'Content-Length': '0'} 2023-11-02T18:40:29.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000062d3e1b6fdc4e6f-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.161 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:40:29.161 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:40:29.162 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.162 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.163 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.163 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.164 DEBUG:boto:Data: 2023-11-02T18:40:29.164 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.164 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.165 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.165 DEBUG:boto:Params: {} 2023-11-02T18:40:29.166 DEBUG:boto:Token: None 2023-11-02T18:40:29.166 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.167 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pfZE5wdNjYc45LMMhiT1eoqwCgs= 2023-11-02T18:40:29.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pfZE5wdNjYc45LMMhiT1eoqwCgs=', 'Content-Length': '0'} 2023-11-02T18:40:29.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b2d82f2652bb830-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.171 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:40:29.171 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:40:29.172 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.172 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.172 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.173 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.173 DEBUG:boto:Data: 2023-11-02T18:40:29.174 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.174 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.174 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.175 DEBUG:boto:Params: {} 2023-11-02T18:40:29.175 DEBUG:boto:Token: None 2023-11-02T18:40:29.176 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.176 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pfZE5wdNjYc45LMMhiT1eoqwCgs= 2023-11-02T18:40:29.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pfZE5wdNjYc45LMMhiT1eoqwCgs=', 'Content-Length': '0'} 2023-11-02T18:40:29.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048c35618fd86ff75-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:40:29.182 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:40:29.182 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:40:29.183 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.183 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.184 DEBUG:boto:Method: GET 2023-11-02T18:40:29.184 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.185 DEBUG:boto:Data: 2023-11-02T18:40:29.185 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.186 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.187 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.187 DEBUG:boto:Params: {} 2023-11-02T18:40:29.189 DEBUG:boto:Token: None 2023-11-02T18:40:29.190 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.191 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gL+Bxak+Dm7Fhm/jsinmpLaYdSw= 2023-11-02T18:40:29.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gL+Bxak+Dm7Fhm/jsinmpLaYdSw=', 'Content-Length': '0'} 2023-11-02T18:40:29.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad98ffe0792c97a6-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.195 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:40:29.196 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:40:29.196 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.197 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.197 DEBUG:boto:Method: GET 2023-11-02T18:40:29.198 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.198 DEBUG:boto:Data: 2023-11-02T18:40:29.199 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.199 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.200 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.200 DEBUG:boto:Params: {} 2023-11-02T18:40:29.200 DEBUG:boto:Token: None 2023-11-02T18:40:29.201 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:40:29.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gL+Bxak+Dm7Fhm/jsinmpLaYdSw= 2023-11-02T18:40:29.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gL+Bxak+Dm7Fhm/jsinmpLaYdSw=', 'Content-Length': '0'} 2023-11-02T18:40:29.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6b0dbb3cfc0e4c-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.250 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.251 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.251 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:40:29.252 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:40:29.252 DEBUG:boto:Method: GET 2023-11-02T18:40:29.253 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:40:29.253 DEBUG:boto:Data: 2023-11-02T18:40:29.254 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.255 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.255 DEBUG:boto:Params: {} 2023-11-02T18:40:29.256 DEBUG:boto:Token: None 2023-11-02T18:40:29.256 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.257 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a01f4ee84e734c4-006543ed1d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.263 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.264 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.264 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.265 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:40:29.265 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:40:29.265 DEBUG:boto:Method: GET 2023-11-02T18:40:29.266 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:40:29.266 DEBUG:boto:Data: 2023-11-02T18:40:29.267 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.267 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.268 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.268 DEBUG:boto:Params: {} 2023-11-02T18:40:29.269 DEBUG:boto:Token: None 2023-11-02T18:40:29.269 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.270 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6c22222eb9933e4-006543ed1d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.279 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.280 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:40:29.281 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:40:29.281 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.282 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:40:29.282 DEBUG:boto:Data: 2023-11-02T18:40:29.282 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.283 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.283 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.284 DEBUG:boto:Params: {} 2023-11-02T18:40:29.284 DEBUG:boto:Token: None 2023-11-02T18:40:29.285 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj51 2023-11-02T18:40:29.285 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3RFiyST3a2D88Qkocjv6PknRDDw= 2023-11-02T18:40:29.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3RFiyST3a2D88Qkocjv6PknRDDw=', 'Content-Length': '0'} 2023-11-02T18:40:29.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1091bd575e3610f-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.289 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:40:29.289 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:40:29.290 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.290 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:40:29.291 DEBUG:boto:Data: 2023-11-02T18:40:29.291 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.291 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.292 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.292 DEBUG:boto:Params: {} 2023-11-02T18:40:29.293 DEBUG:boto:Token: None 2023-11-02T18:40:29.293 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj51 2023-11-02T18:40:29.294 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3RFiyST3a2D88Qkocjv6PknRDDw= 2023-11-02T18:40:29.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3RFiyST3a2D88Qkocjv6PknRDDw=', 'Content-Length': '0'} 2023-11-02T18:40:29.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aef5f5ea70146de-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:40:29.298 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:40:29.299 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:40:29.299 DEBUG:boto:Method: GET 2023-11-02T18:40:29.300 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:40:29.300 DEBUG:boto:Data: 2023-11-02T18:40:29.301 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.301 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.301 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.302 DEBUG:boto:Params: {} 2023-11-02T18:40:29.302 DEBUG:boto:Token: None 2023-11-02T18:40:29.303 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj51 2023-11-02T18:40:29.303 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KUjnF7HscgjRi20uAkjV3CuA8hQ= 2023-11-02T18:40:29.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KUjnF7HscgjRi20uAkjV3CuA8hQ=', 'Content-Length': '0'} 2023-11-02T18:40:29.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002505de28b4619a7a-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.308 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:40:29.308 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:40:29.309 DEBUG:boto:Method: GET 2023-11-02T18:40:29.309 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:40:29.310 DEBUG:boto:Data: 2023-11-02T18:40:29.310 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.311 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.311 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.312 DEBUG:boto:Params: {} 2023-11-02T18:40:29.312 DEBUG:boto:Token: None 2023-11-02T18:40:29.313 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj51 2023-11-02T18:40:29.313 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KUjnF7HscgjRi20uAkjV3CuA8hQ= 2023-11-02T18:40:29.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KUjnF7HscgjRi20uAkjV3CuA8hQ=', 'Content-Length': '0'} 2023-11-02T18:40:29.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df93318972e92410-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.317 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.318 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.319 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:40:29.319 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:40:29.319 DEBUG:boto:Method: GET 2023-11-02T18:40:29.320 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:40:29.320 DEBUG:boto:Data: 2023-11-02T18:40:29.321 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.321 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.321 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.322 DEBUG:boto:Params: {} 2023-11-02T18:40:29.322 DEBUG:boto:Token: None 2023-11-02T18:40:29.323 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.324 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053d465c16b78cb04-006543ed1d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.374 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.375 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.376 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.376 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:40:29.377 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:40:29.377 DEBUG:boto:Method: GET 2023-11-02T18:40:29.377 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:40:29.378 DEBUG:boto:Data: 2023-11-02T18:40:29.378 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.379 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.380 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.381 DEBUG:boto:Params: {} 2023-11-02T18:40:29.381 DEBUG:boto:Token: None 2023-11-02T18:40:29.382 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.382 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070d5d9600c0eb1bb-006543ed1d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.390 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:40:29.391 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:40:29.392 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:40:29.392 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.393 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.393 DEBUG:boto:Method: GET 2023-11-02T18:40:29.394 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.394 DEBUG:boto:Data: 2023-11-02T18:40:29.394 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.395 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.395 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.396 DEBUG:boto:Params: {} 2023-11-02T18:40:29.396 DEBUG:boto:Token: None 2023-11-02T18:40:29.397 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.397 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J41xxmKVyHgiG4BMjStr47yWmb0= 2023-11-02T18:40:29.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J41xxmKVyHgiG4BMjStr47yWmb0=', 'Content-Length': '0'} 2023-11-02T18:40:29.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065987d8ee655e724-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.401 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:40:29.401 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:40:29.402 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.402 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.403 DEBUG:boto:Method: GET 2023-11-02T18:40:29.403 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.403 DEBUG:boto:Data: 2023-11-02T18:40:29.404 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.404 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.405 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.405 DEBUG:boto:Params: {} 2023-11-02T18:40:29.406 DEBUG:boto:Token: None 2023-11-02T18:40:29.406 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.407 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J41xxmKVyHgiG4BMjStr47yWmb0= 2023-11-02T18:40:29.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J41xxmKVyHgiG4BMjStr47yWmb0=', 'Content-Length': '0'} 2023-11-02T18:40:29.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3a6053e42aff990-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.450 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:40:29.451 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:40:29.451 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.452 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.452 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.453 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.453 DEBUG:boto:Data: 2023-11-02T18:40:29.454 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.455 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.455 DEBUG:boto:Params: {} 2023-11-02T18:40:29.456 DEBUG:boto:Token: None 2023-11-02T18:40:29.456 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.457 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ifoz6y3rEOMPEAqgH6M/1Zp90nE= 2023-11-02T18:40:29.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ifoz6y3rEOMPEAqgH6M/1Zp90nE=', 'Content-Length': '0'} 2023-11-02T18:40:29.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e70e692d2a0a2d6-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.461 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:40:29.461 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:40:29.462 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.462 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.463 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.463 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.464 DEBUG:boto:Data: 2023-11-02T18:40:29.464 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.464 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.465 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.465 DEBUG:boto:Params: {} 2023-11-02T18:40:29.466 DEBUG:boto:Token: None 2023-11-02T18:40:29.466 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.467 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ifoz6y3rEOMPEAqgH6M/1Zp90nE= 2023-11-02T18:40:29.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ifoz6y3rEOMPEAqgH6M/1Zp90nE=', 'Content-Length': '0'} 2023-11-02T18:40:29.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a89b4c1634d5fa7e-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:40:29.471 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:40:29.472 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:40:29.472 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.473 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.473 DEBUG:boto:Method: GET 2023-11-02T18:40:29.473 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.474 DEBUG:boto:Data: 2023-11-02T18:40:29.474 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.475 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.475 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.476 DEBUG:boto:Params: {} 2023-11-02T18:40:29.476 DEBUG:boto:Token: None 2023-11-02T18:40:29.477 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.477 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J41xxmKVyHgiG4BMjStr47yWmb0= 2023-11-02T18:40:29.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J41xxmKVyHgiG4BMjStr47yWmb0=', 'Content-Length': '0'} 2023-11-02T18:40:29.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e7e294f1f9c62e1-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.482 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:40:29.482 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:40:29.483 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.483 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.483 DEBUG:boto:Method: GET 2023-11-02T18:40:29.484 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.484 DEBUG:boto:Data: 2023-11-02T18:40:29.485 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.485 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.486 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.486 DEBUG:boto:Params: {} 2023-11-02T18:40:29.487 DEBUG:boto:Token: None 2023-11-02T18:40:29.487 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:40:29.491 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J41xxmKVyHgiG4BMjStr47yWmb0= 2023-11-02T18:40:29.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J41xxmKVyHgiG4BMjStr47yWmb0=', 'Content-Length': '0'} 2023-11-02T18:40:29.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ae47a9190c42f61-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.496 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.496 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.497 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:40:29.497 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:40:29.498 DEBUG:boto:Method: GET 2023-11-02T18:40:29.498 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:40:29.499 DEBUG:boto:Data: 2023-11-02T18:40:29.499 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.500 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.500 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.501 DEBUG:boto:Params: {} 2023-11-02T18:40:29.501 DEBUG:boto:Token: None 2023-11-02T18:40:29.502 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.502 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006ca11f4dd9996c9-006543ed1d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.509 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.509 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.510 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.510 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:40:29.511 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:40:29.511 DEBUG:boto:Method: GET 2023-11-02T18:40:29.512 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:40:29.512 DEBUG:boto:Data: 2023-11-02T18:40:29.512 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.513 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.513 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.514 DEBUG:boto:Params: {} 2023-11-02T18:40:29.514 DEBUG:boto:Token: None 2023-11-02T18:40:29.515 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.515 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0fe1996d08ebf73-006543ed1d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.524 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.525 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:40:29.525 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:40:29.526 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.526 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:40:29.527 DEBUG:boto:Data: 2023-11-02T18:40:29.527 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.528 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.528 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.529 DEBUG:boto:Params: {} 2023-11-02T18:40:29.529 DEBUG:boto:Token: None 2023-11-02T18:40:29.530 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj55 2023-11-02T18:40:29.530 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WcS2vJuIQ3uu3/lJPwXU0iKo2QQ= 2023-11-02T18:40:29.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WcS2vJuIQ3uu3/lJPwXU0iKo2QQ=', 'Content-Length': '0'} 2023-11-02T18:40:29.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc2ed2511b2bf266-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.534 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:40:29.534 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:40:29.535 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.535 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:40:29.536 DEBUG:boto:Data: 2023-11-02T18:40:29.536 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.537 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.537 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.537 DEBUG:boto:Params: {} 2023-11-02T18:40:29.538 DEBUG:boto:Token: None 2023-11-02T18:40:29.539 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj55 2023-11-02T18:40:29.539 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WcS2vJuIQ3uu3/lJPwXU0iKo2QQ= 2023-11-02T18:40:29.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WcS2vJuIQ3uu3/lJPwXU0iKo2QQ=', 'Content-Length': '0'} 2023-11-02T18:40:29.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f32ed412c212d212-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:40:29.544 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:40:29.544 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:40:29.545 DEBUG:boto:Method: GET 2023-11-02T18:40:29.545 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:40:29.545 DEBUG:boto:Data: 2023-11-02T18:40:29.546 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.546 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.547 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.547 DEBUG:boto:Params: {} 2023-11-02T18:40:29.548 DEBUG:boto:Token: None 2023-11-02T18:40:29.548 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj55 2023-11-02T18:40:29.549 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KQTJCXwBTgy43jfRyIaz5tiKLe4= 2023-11-02T18:40:29.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KQTJCXwBTgy43jfRyIaz5tiKLe4=', 'Content-Length': '0'} 2023-11-02T18:40:29.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db2540f0e90389f0-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.594 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:40:29.595 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:40:29.595 DEBUG:boto:Method: GET 2023-11-02T18:40:29.596 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:40:29.596 DEBUG:boto:Data: 2023-11-02T18:40:29.597 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.597 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.598 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.598 DEBUG:boto:Params: {} 2023-11-02T18:40:29.599 DEBUG:boto:Token: None 2023-11-02T18:40:29.599 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj55 2023-11-02T18:40:29.600 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KQTJCXwBTgy43jfRyIaz5tiKLe4= 2023-11-02T18:40:29.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KQTJCXwBTgy43jfRyIaz5tiKLe4=', 'Content-Length': '0'} 2023-11-02T18:40:29.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f04eb6d63e54247c-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.605 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.605 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.606 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:40:29.606 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:40:29.607 DEBUG:boto:Method: GET 2023-11-02T18:40:29.607 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:40:29.608 DEBUG:boto:Data: 2023-11-02T18:40:29.608 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.609 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.609 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.610 DEBUG:boto:Params: {} 2023-11-02T18:40:29.611 DEBUG:boto:Token: None 2023-11-02T18:40:29.611 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000772277f8935c99c6-006543ed1d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.618 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.619 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.620 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.620 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:40:29.621 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:40:29.621 DEBUG:boto:Method: GET 2023-11-02T18:40:29.622 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:40:29.622 DEBUG:boto:Data: 2023-11-02T18:40:29.623 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.623 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.624 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.624 DEBUG:boto:Params: {} 2023-11-02T18:40:29.625 DEBUG:boto:Token: None 2023-11-02T18:40:29.625 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef9394da45d89e8a-006543ed1d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.631 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:40:29.633 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:40:29.633 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:40:29.634 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.634 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.635 DEBUG:boto:Method: GET 2023-11-02T18:40:29.635 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.636 DEBUG:boto:Data: 2023-11-02T18:40:29.636 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.637 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.637 DEBUG:boto:Params: {} 2023-11-02T18:40:29.638 DEBUG:boto:Token: None 2023-11-02T18:40:29.639 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.639 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sN9QqIfZnKt6NKgjA1AUhUTcLWs= 2023-11-02T18:40:29.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sN9QqIfZnKt6NKgjA1AUhUTcLWs=', 'Content-Length': '0'} 2023-11-02T18:40:29.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3fe734cdc90d323-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.643 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:40:29.644 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:40:29.644 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.645 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.645 DEBUG:boto:Method: GET 2023-11-02T18:40:29.645 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.646 DEBUG:boto:Data: 2023-11-02T18:40:29.646 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.647 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.647 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.648 DEBUG:boto:Params: {} 2023-11-02T18:40:29.648 DEBUG:boto:Token: None 2023-11-02T18:40:29.649 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.649 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sN9QqIfZnKt6NKgjA1AUhUTcLWs= 2023-11-02T18:40:29.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sN9QqIfZnKt6NKgjA1AUhUTcLWs=', 'Content-Length': '0'} 2023-11-02T18:40:29.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecac339588302e9d-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.653 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:40:29.654 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:40:29.654 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.654 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.655 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.655 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.656 DEBUG:boto:Data: 2023-11-02T18:40:29.656 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.657 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.657 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.658 DEBUG:boto:Params: {} 2023-11-02T18:40:29.658 DEBUG:boto:Token: None 2023-11-02T18:40:29.659 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.659 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ncvl3fRTbD0X2ZR8mbK3wOMpWig= 2023-11-02T18:40:29.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ncvl3fRTbD0X2ZR8mbK3wOMpWig=', 'Content-Length': '0'} 2023-11-02T18:40:29.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000536cec4f5a6a59a6-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.663 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:40:29.664 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:40:29.664 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.665 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.665 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.665 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.666 DEBUG:boto:Data: 2023-11-02T18:40:29.666 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.667 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.667 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.668 DEBUG:boto:Params: {} 2023-11-02T18:40:29.668 DEBUG:boto:Token: None 2023-11-02T18:40:29.669 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.669 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ncvl3fRTbD0X2ZR8mbK3wOMpWig= 2023-11-02T18:40:29.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ncvl3fRTbD0X2ZR8mbK3wOMpWig=', 'Content-Length': '0'} 2023-11-02T18:40:29.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa2fcfac491b2020-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:40:29.674 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:40:29.674 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:40:29.675 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.675 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.676 DEBUG:boto:Method: GET 2023-11-02T18:40:29.676 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.677 DEBUG:boto:Data: 2023-11-02T18:40:29.677 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.678 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.678 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.679 DEBUG:boto:Params: {} 2023-11-02T18:40:29.679 DEBUG:boto:Token: None 2023-11-02T18:40:29.680 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.681 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sN9QqIfZnKt6NKgjA1AUhUTcLWs= 2023-11-02T18:40:29.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sN9QqIfZnKt6NKgjA1AUhUTcLWs=', 'Content-Length': '0'} 2023-11-02T18:40:29.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000120ee452d7271c0-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.726 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:40:29.727 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:40:29.728 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.728 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.728 DEBUG:boto:Method: GET 2023-11-02T18:40:29.729 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.729 DEBUG:boto:Data: 2023-11-02T18:40:29.730 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.730 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.731 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.731 DEBUG:boto:Params: {} 2023-11-02T18:40:29.732 DEBUG:boto:Token: None 2023-11-02T18:40:29.732 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:40:29.733 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sN9QqIfZnKt6NKgjA1AUhUTcLWs= 2023-11-02T18:40:29.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sN9QqIfZnKt6NKgjA1AUhUTcLWs=', 'Content-Length': '0'} 2023-11-02T18:40:29.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000525d9770b38db319-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.737 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.738 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.738 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:40:29.739 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:40:29.739 DEBUG:boto:Method: GET 2023-11-02T18:40:29.740 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:40:29.740 DEBUG:boto:Data: 2023-11-02T18:40:29.741 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.741 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.742 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.742 DEBUG:boto:Params: {} 2023-11-02T18:40:29.743 DEBUG:boto:Token: None 2023-11-02T18:40:29.743 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.744 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002336cfde3a375796-006543ed1d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.749 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.750 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.751 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.751 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:40:29.752 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:40:29.752 DEBUG:boto:Method: GET 2023-11-02T18:40:29.753 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:40:29.753 DEBUG:boto:Data: 2023-11-02T18:40:29.753 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.754 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.755 DEBUG:boto:Params: {} 2023-11-02T18:40:29.756 DEBUG:boto:Token: None 2023-11-02T18:40:29.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b2139a8b81c3145-006543ed1d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.764 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.765 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:40:29.765 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:40:29.766 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.766 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:40:29.767 DEBUG:boto:Data: 2023-11-02T18:40:29.767 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.768 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.768 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.769 DEBUG:boto:Params: {} 2023-11-02T18:40:29.769 DEBUG:boto:Token: None 2023-11-02T18:40:29.770 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj57 2023-11-02T18:40:29.770 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l2dROZ7MrOZQFxq09zXILDOrd8c= 2023-11-02T18:40:29.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l2dROZ7MrOZQFxq09zXILDOrd8c=', 'Content-Length': '0'} 2023-11-02T18:40:29.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5e4cbc3c86b4c7c-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.774 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:40:29.774 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:40:29.775 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.775 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:40:29.776 DEBUG:boto:Data: 2023-11-02T18:40:29.776 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.777 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.777 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.777 DEBUG:boto:Params: {} 2023-11-02T18:40:29.778 DEBUG:boto:Token: None 2023-11-02T18:40:29.778 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj57 2023-11-02T18:40:29.779 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l2dROZ7MrOZQFxq09zXILDOrd8c= 2023-11-02T18:40:29.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l2dROZ7MrOZQFxq09zXILDOrd8c=', 'Content-Length': '0'} 2023-11-02T18:40:29.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b8fff5d074dbbfc-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:40:29.783 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:40:29.783 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:40:29.784 DEBUG:boto:Method: GET 2023-11-02T18:40:29.784 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:40:29.785 DEBUG:boto:Data: 2023-11-02T18:40:29.785 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.786 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.786 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.787 DEBUG:boto:Params: {} 2023-11-02T18:40:29.787 DEBUG:boto:Token: None 2023-11-02T18:40:29.788 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj57 2023-11-02T18:40:29.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3s8YGCOT0W82f27Sx8+gQy11c+I= 2023-11-02T18:40:29.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3s8YGCOT0W82f27Sx8+gQy11c+I=', 'Content-Length': '0'} 2023-11-02T18:40:29.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abf286924efa59c3-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.793 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:40:29.793 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:40:29.794 DEBUG:boto:Method: GET 2023-11-02T18:40:29.794 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:40:29.795 DEBUG:boto:Data: 2023-11-02T18:40:29.795 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.796 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.796 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.796 DEBUG:boto:Params: {} 2023-11-02T18:40:29.797 DEBUG:boto:Token: None 2023-11-02T18:40:29.798 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj57 2023-11-02T18:40:29.798 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3s8YGCOT0W82f27Sx8+gQy11c+I= 2023-11-02T18:40:29.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3s8YGCOT0W82f27Sx8+gQy11c+I=', 'Content-Length': '0'} 2023-11-02T18:40:29.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feb23fca562ee76c-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.842 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.843 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.843 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:40:29.844 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:40:29.844 DEBUG:boto:Method: GET 2023-11-02T18:40:29.845 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:40:29.845 DEBUG:boto:Data: 2023-11-02T18:40:29.846 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.846 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.847 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.847 DEBUG:boto:Params: {} 2023-11-02T18:40:29.848 DEBUG:boto:Token: None 2023-11-02T18:40:29.848 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.849 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce824b245e20ca5d-006543ed1d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.856 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.857 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.858 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.858 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:40:29.859 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:40:29.859 DEBUG:boto:Method: GET 2023-11-02T18:40:29.859 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:40:29.860 DEBUG:boto:Data: 2023-11-02T18:40:29.860 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.861 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.861 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.862 DEBUG:boto:Params: {} 2023-11-02T18:40:29.862 DEBUG:boto:Token: None 2023-11-02T18:40:29.863 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.863 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6132ae4507fc971-006543ed1d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.871 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:29.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:40:29.873 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:40:29.873 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:40:29.874 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.874 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.875 DEBUG:boto:Method: GET 2023-11-02T18:40:29.875 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.875 DEBUG:boto:Data: 2023-11-02T18:40:29.876 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.876 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.877 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.877 DEBUG:boto:Params: {} 2023-11-02T18:40:29.878 DEBUG:boto:Token: None 2023-11-02T18:40:29.878 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.879 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5cwNiqdRiYATtmWEYpjGv2G333U= 2023-11-02T18:40:29.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5cwNiqdRiYATtmWEYpjGv2G333U=', 'Content-Length': '0'} 2023-11-02T18:40:29.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008496d71083a9efd0-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.926 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:40:29.927 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:40:29.928 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.928 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.928 DEBUG:boto:Method: GET 2023-11-02T18:40:29.929 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.929 DEBUG:boto:Data: 2023-11-02T18:40:29.930 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.931 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.931 DEBUG:boto:Params: {} 2023-11-02T18:40:29.932 DEBUG:boto:Token: None 2023-11-02T18:40:29.932 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5cwNiqdRiYATtmWEYpjGv2G333U= 2023-11-02T18:40:29.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5cwNiqdRiYATtmWEYpjGv2G333U=', 'Content-Length': '0'} 2023-11-02T18:40:29.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2ed17f90bd1244a-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.937 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:40:29.938 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:40:29.938 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.939 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.939 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.940 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.940 DEBUG:boto:Data: 2023-11-02T18:40:29.940 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.941 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.941 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.942 DEBUG:boto:Params: {} 2023-11-02T18:40:29.942 DEBUG:boto:Token: None 2023-11-02T18:40:29.943 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.943 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CVdwAew3aNKQUOPmvMrTntufscs= 2023-11-02T18:40:29.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CVdwAew3aNKQUOPmvMrTntufscs=', 'Content-Length': '0'} 2023-11-02T18:40:29.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5659b9e6e03de92-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.948 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:40:29.948 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:40:29.948 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.949 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.949 DEBUG:boto:Method: HEAD 2023-11-02T18:40:29.950 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.950 DEBUG:boto:Data: 2023-11-02T18:40:29.951 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.951 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.951 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.952 DEBUG:boto:Params: {} 2023-11-02T18:40:29.952 DEBUG:boto:Token: None 2023-11-02T18:40:29.953 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.953 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CVdwAew3aNKQUOPmvMrTntufscs= 2023-11-02T18:40:29.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CVdwAew3aNKQUOPmvMrTntufscs=', 'Content-Length': '0'} 2023-11-02T18:40:29.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000f5e5261b56f19e-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:40:29.958 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:40:29.959 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:40:29.959 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.960 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.960 DEBUG:boto:Method: GET 2023-11-02T18:40:29.961 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.961 DEBUG:boto:Data: 2023-11-02T18:40:29.962 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.962 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.963 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.963 DEBUG:boto:Params: {} 2023-11-02T18:40:29.963 DEBUG:boto:Token: None 2023-11-02T18:40:29.964 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.965 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5cwNiqdRiYATtmWEYpjGv2G333U= 2023-11-02T18:40:29.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5cwNiqdRiYATtmWEYpjGv2G333U=', 'Content-Length': '0'} 2023-11-02T18:40:29.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000358a1d93a70fc159-006543ed1d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.969 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:40:29.969 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:40:29.970 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.970 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.971 DEBUG:boto:Method: GET 2023-11-02T18:40:29.971 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.972 DEBUG:boto:Data: 2023-11-02T18:40:29.972 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.972 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:29.973 DEBUG:boto:Port: 8000 2023-11-02T18:40:29.973 DEBUG:boto:Params: {} 2023-11-02T18:40:29.974 DEBUG:boto:Token: None 2023-11-02T18:40:29.974 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:40:29.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5cwNiqdRiYATtmWEYpjGv2G333U= 2023-11-02T18:40:29.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5cwNiqdRiYATtmWEYpjGv2G333U=', 'Content-Length': '0'} 2023-11-02T18:40:29.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f797746d125f3e9b-006543ed1d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:29.979 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:29.980 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:29.980 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:40:29.980 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:40:29.981 DEBUG:boto:Method: GET 2023-11-02T18:40:29.981 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:40:29.982 DEBUG:boto:Data: 2023-11-02T18:40:29.982 DEBUG:boto:Headers: {} 2023-11-02T18:40:29.983 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:29.983 DEBUG:boto:Port: 8001 2023-11-02T18:40:29.984 DEBUG:boto:Params: {} 2023-11-02T18:40:29.984 DEBUG:boto:Token: None 2023-11-02T18:40:29.985 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:29 GMT /ibtljn-59/ 2023-11-02T18:40:29.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI= 2023-11-02T18:40:29.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eFL4MbRdPadc1rybNncmnt9KnMI=', 'Content-Length': '0'} 2023-11-02T18:40:29.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff3fd03970037513-006543ed1d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.034 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.035 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.036 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.036 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:40:30.037 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:40:30.037 DEBUG:boto:Method: GET 2023-11-02T18:40:30.038 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:40:30.038 DEBUG:boto:Data: 2023-11-02T18:40:30.039 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.039 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.040 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.040 DEBUG:boto:Params: {} 2023-11-02T18:40:30.041 DEBUG:boto:Token: None 2023-11-02T18:40:30.041 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.042 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000447139e4b95bf659-006543ed1e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.051 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.052 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:40:30.053 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:40:30.053 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.054 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:40:30.054 DEBUG:boto:Data: 2023-11-02T18:40:30.055 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.055 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.056 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.056 DEBUG:boto:Params: {} 2023-11-02T18:40:30.057 DEBUG:boto:Token: None 2023-11-02T18:40:30.057 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj68 2023-11-02T18:40:30.058 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tyh4mffO+ywmbr6NTUpRdZh0SE8= 2023-11-02T18:40:30.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tyh4mffO+ywmbr6NTUpRdZh0SE8=', 'Content-Length': '0'} 2023-11-02T18:40:30.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e53095161c618df-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.062 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:40:30.062 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:40:30.063 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.063 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:40:30.064 DEBUG:boto:Data: 2023-11-02T18:40:30.064 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.064 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.065 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.065 DEBUG:boto:Params: {} 2023-11-02T18:40:30.066 DEBUG:boto:Token: None 2023-11-02T18:40:30.066 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj68 2023-11-02T18:40:30.067 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tyh4mffO+ywmbr6NTUpRdZh0SE8= 2023-11-02T18:40:30.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tyh4mffO+ywmbr6NTUpRdZh0SE8=', 'Content-Length': '0'} 2023-11-02T18:40:30.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc06bbc46c0b3611-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:40:30.071 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:40:30.072 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:40:30.072 DEBUG:boto:Method: GET 2023-11-02T18:40:30.073 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:40:30.073 DEBUG:boto:Data: 2023-11-02T18:40:30.074 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.074 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.075 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.075 DEBUG:boto:Params: {} 2023-11-02T18:40:30.075 DEBUG:boto:Token: None 2023-11-02T18:40:30.076 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj68 2023-11-02T18:40:30.076 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MpkIwP+BrFrfPvYsw1TJ78cNmCY= 2023-11-02T18:40:30.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MpkIwP+BrFrfPvYsw1TJ78cNmCY=', 'Content-Length': '0'} 2023-11-02T18:40:30.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007098bf44b6091309-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.081 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:40:30.081 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:40:30.082 DEBUG:boto:Method: GET 2023-11-02T18:40:30.082 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:40:30.083 DEBUG:boto:Data: 2023-11-02T18:40:30.083 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.084 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.084 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.085 DEBUG:boto:Params: {} 2023-11-02T18:40:30.085 DEBUG:boto:Token: None 2023-11-02T18:40:30.086 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj68 2023-11-02T18:40:30.086 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MpkIwP+BrFrfPvYsw1TJ78cNmCY= 2023-11-02T18:40:30.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MpkIwP+BrFrfPvYsw1TJ78cNmCY=', 'Content-Length': '0'} 2023-11-02T18:40:30.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd5d89845a6c015a-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.090 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.091 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.091 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:40:30.092 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:40:30.092 DEBUG:boto:Method: GET 2023-11-02T18:40:30.093 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:40:30.093 DEBUG:boto:Data: 2023-11-02T18:40:30.094 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.094 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.095 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.095 DEBUG:boto:Params: {} 2023-11-02T18:40:30.096 DEBUG:boto:Token: None 2023-11-02T18:40:30.096 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.097 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027f3571011e506e6-006543ed1e-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.103 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.104 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.104 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.104 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:40:30.105 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:40:30.105 DEBUG:boto:Method: GET 2023-11-02T18:40:30.106 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:40:30.106 DEBUG:boto:Data: 2023-11-02T18:40:30.107 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.107 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.108 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.108 DEBUG:boto:Params: {} 2023-11-02T18:40:30.109 DEBUG:boto:Token: None 2023-11-02T18:40:30.109 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.110 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeb0fbde900e6a31-006543ed1e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.119 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:40:30.120 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:40:30.120 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:40:30.121 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.121 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.122 DEBUG:boto:Method: GET 2023-11-02T18:40:30.122 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.122 DEBUG:boto:Data: 2023-11-02T18:40:30.123 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.123 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.124 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.124 DEBUG:boto:Params: {} 2023-11-02T18:40:30.125 DEBUG:boto:Token: None 2023-11-02T18:40:30.126 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.126 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m5dKTk8NyvVGYprZwPmqNu3RPe8= 2023-11-02T18:40:30.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m5dKTk8NyvVGYprZwPmqNu3RPe8=', 'Content-Length': '0'} 2023-11-02T18:40:30.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000168683ad58443168-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.170 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:40:30.171 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:40:30.171 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.172 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.172 DEBUG:boto:Method: GET 2023-11-02T18:40:30.173 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.173 DEBUG:boto:Data: 2023-11-02T18:40:30.174 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.174 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.175 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.175 DEBUG:boto:Params: {} 2023-11-02T18:40:30.176 DEBUG:boto:Token: None 2023-11-02T18:40:30.176 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m5dKTk8NyvVGYprZwPmqNu3RPe8= 2023-11-02T18:40:30.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m5dKTk8NyvVGYprZwPmqNu3RPe8=', 'Content-Length': '0'} 2023-11-02T18:40:30.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b84b4e33d0b443-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.181 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:40:30.182 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:40:30.182 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.183 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.183 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.184 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.184 DEBUG:boto:Data: 2023-11-02T18:40:30.185 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.185 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.185 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.186 DEBUG:boto:Params: {} 2023-11-02T18:40:30.186 DEBUG:boto:Token: None 2023-11-02T18:40:30.187 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.187 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L/RZDtlaLt69vSAJJxvFTRVNhpQ= 2023-11-02T18:40:30.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L/RZDtlaLt69vSAJJxvFTRVNhpQ=', 'Content-Length': '0'} 2023-11-02T18:40:30.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe259af3352d3bb6-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.191 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:40:30.192 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:40:30.192 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.193 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.193 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.194 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.194 DEBUG:boto:Data: 2023-11-02T18:40:30.194 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.195 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.195 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.196 DEBUG:boto:Params: {} 2023-11-02T18:40:30.196 DEBUG:boto:Token: None 2023-11-02T18:40:30.197 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.197 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L/RZDtlaLt69vSAJJxvFTRVNhpQ= 2023-11-02T18:40:30.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L/RZDtlaLt69vSAJJxvFTRVNhpQ=', 'Content-Length': '0'} 2023-11-02T18:40:30.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b5f9e0d379655d-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:40:30.202 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:40:30.202 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:40:30.203 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.203 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.204 DEBUG:boto:Method: GET 2023-11-02T18:40:30.204 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.204 DEBUG:boto:Data: 2023-11-02T18:40:30.205 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.205 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.206 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.206 DEBUG:boto:Params: {} 2023-11-02T18:40:30.207 DEBUG:boto:Token: None 2023-11-02T18:40:30.207 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m5dKTk8NyvVGYprZwPmqNu3RPe8= 2023-11-02T18:40:30.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m5dKTk8NyvVGYprZwPmqNu3RPe8=', 'Content-Length': '0'} 2023-11-02T18:40:30.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000487fc3d15623de5f-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.212 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:40:30.213 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:40:30.213 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.214 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.214 DEBUG:boto:Method: GET 2023-11-02T18:40:30.215 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.215 DEBUG:boto:Data: 2023-11-02T18:40:30.216 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.216 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.217 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.217 DEBUG:boto:Params: {} 2023-11-02T18:40:30.218 DEBUG:boto:Token: None 2023-11-02T18:40:30.218 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:40:30.219 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m5dKTk8NyvVGYprZwPmqNu3RPe8= 2023-11-02T18:40:30.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m5dKTk8NyvVGYprZwPmqNu3RPe8=', 'Content-Length': '0'} 2023-11-02T18:40:30.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b21a05d2cc9a803-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.223 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.224 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.224 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:40:30.224 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:40:30.225 DEBUG:boto:Method: GET 2023-11-02T18:40:30.225 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:40:30.226 DEBUG:boto:Data: 2023-11-02T18:40:30.226 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.227 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.227 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.228 DEBUG:boto:Params: {} 2023-11-02T18:40:30.228 DEBUG:boto:Token: None 2023-11-02T18:40:30.228 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.229 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb1bc718cced2bef-006543ed1e-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.278 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.279 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.280 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.280 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:40:30.281 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:40:30.281 DEBUG:boto:Method: GET 2023-11-02T18:40:30.281 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:40:30.282 DEBUG:boto:Data: 2023-11-02T18:40:30.282 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.283 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.283 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.284 DEBUG:boto:Params: {} 2023-11-02T18:40:30.284 DEBUG:boto:Token: None 2023-11-02T18:40:30.285 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.285 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6e894f5c3ad420-006543ed1e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.294 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.295 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:40:30.296 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:40:30.296 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.297 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:40:30.297 DEBUG:boto:Data: 2023-11-02T18:40:30.298 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.298 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.298 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.299 DEBUG:boto:Params: {} 2023-11-02T18:40:30.299 DEBUG:boto:Token: None 2023-11-02T18:40:30.300 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj69 2023-11-02T18:40:30.300 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ldz91RiHH6+C6MNioA5KqxujLE0= 2023-11-02T18:40:30.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ldz91RiHH6+C6MNioA5KqxujLE0=', 'Content-Length': '0'} 2023-11-02T18:40:30.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc944d60e679cfa9-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.304 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:40:30.305 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:40:30.305 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.306 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:40:30.306 DEBUG:boto:Data: 2023-11-02T18:40:30.310 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.310 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.310 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.311 DEBUG:boto:Params: {} 2023-11-02T18:40:30.311 DEBUG:boto:Token: None 2023-11-02T18:40:30.312 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj69 2023-11-02T18:40:30.312 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ldz91RiHH6+C6MNioA5KqxujLE0= 2023-11-02T18:40:30.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ldz91RiHH6+C6MNioA5KqxujLE0=', 'Content-Length': '0'} 2023-11-02T18:40:30.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e7d89ec2df5ab3-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:40:30.317 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:40:30.317 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:40:30.318 DEBUG:boto:Method: GET 2023-11-02T18:40:30.318 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:40:30.318 DEBUG:boto:Data: 2023-11-02T18:40:30.319 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.319 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.320 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.320 DEBUG:boto:Params: {} 2023-11-02T18:40:30.321 DEBUG:boto:Token: None 2023-11-02T18:40:30.321 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj69 2023-11-02T18:40:30.322 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EcROlAkyjnNC8IPKUdMSiB0KNes= 2023-11-02T18:40:30.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EcROlAkyjnNC8IPKUdMSiB0KNes=', 'Content-Length': '0'} 2023-11-02T18:40:30.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000579efc062eed8dda-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.326 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:40:30.326 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:40:30.327 DEBUG:boto:Method: GET 2023-11-02T18:40:30.327 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:40:30.328 DEBUG:boto:Data: 2023-11-02T18:40:30.328 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.329 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.329 DEBUG:boto:Params: {} 2023-11-02T18:40:30.330 DEBUG:boto:Token: None 2023-11-02T18:40:30.330 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj69 2023-11-02T18:40:30.331 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EcROlAkyjnNC8IPKUdMSiB0KNes= 2023-11-02T18:40:30.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EcROlAkyjnNC8IPKUdMSiB0KNes=', 'Content-Length': '0'} 2023-11-02T18:40:30.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009612221328b0399d-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.378 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.379 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.379 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:40:30.380 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:40:30.380 DEBUG:boto:Method: GET 2023-11-02T18:40:30.381 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:40:30.381 DEBUG:boto:Data: 2023-11-02T18:40:30.382 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.382 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.382 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.383 DEBUG:boto:Params: {} 2023-11-02T18:40:30.383 DEBUG:boto:Token: None 2023-11-02T18:40:30.384 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.385 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac3930e00c8d7e98-006543ed1e-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.391 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.392 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.392 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.393 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:40:30.393 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:40:30.394 DEBUG:boto:Method: GET 2023-11-02T18:40:30.394 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:40:30.395 DEBUG:boto:Data: 2023-11-02T18:40:30.395 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.396 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.396 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.396 DEBUG:boto:Params: {} 2023-11-02T18:40:30.397 DEBUG:boto:Token: None 2023-11-02T18:40:30.397 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.398 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000898ebd3e4e1bc68f-006543ed1e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.406 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:40:30.407 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:40:30.408 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:40:30.408 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.409 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.409 DEBUG:boto:Method: GET 2023-11-02T18:40:30.409 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.410 DEBUG:boto:Data: 2023-11-02T18:40:30.410 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.411 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.411 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.412 DEBUG:boto:Params: {} 2023-11-02T18:40:30.412 DEBUG:boto:Token: None 2023-11-02T18:40:30.413 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.413 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3uBqTCet3NwGGef/1tamTQjrtfc= 2023-11-02T18:40:30.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3uBqTCet3NwGGef/1tamTQjrtfc=', 'Content-Length': '0'} 2023-11-02T18:40:30.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000104d5eb9cdd6e223-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.458 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:40:30.459 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:40:30.459 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.460 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.460 DEBUG:boto:Method: GET 2023-11-02T18:40:30.461 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.461 DEBUG:boto:Data: 2023-11-02T18:40:30.462 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.463 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.463 DEBUG:boto:Params: {} 2023-11-02T18:40:30.463 DEBUG:boto:Token: None 2023-11-02T18:40:30.464 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.464 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3uBqTCet3NwGGef/1tamTQjrtfc= 2023-11-02T18:40:30.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3uBqTCet3NwGGef/1tamTQjrtfc=', 'Content-Length': '0'} 2023-11-02T18:40:30.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e06849f63c14b3c-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.469 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:40:30.470 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:40:30.470 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.470 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.471 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.471 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.472 DEBUG:boto:Data: 2023-11-02T18:40:30.472 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.472 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.473 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.473 DEBUG:boto:Params: {} 2023-11-02T18:40:30.474 DEBUG:boto:Token: None 2023-11-02T18:40:30.474 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.475 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rcenzItGcFpT51mU1dniRZWY9zQ= 2023-11-02T18:40:30.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rcenzItGcFpT51mU1dniRZWY9zQ=', 'Content-Length': '0'} 2023-11-02T18:40:30.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe57f9830a20418a-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.479 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:40:30.479 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:40:30.480 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.480 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.481 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.481 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.481 DEBUG:boto:Data: 2023-11-02T18:40:30.482 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.482 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.483 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.483 DEBUG:boto:Params: {} 2023-11-02T18:40:30.484 DEBUG:boto:Token: None 2023-11-02T18:40:30.484 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.485 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rcenzItGcFpT51mU1dniRZWY9zQ= 2023-11-02T18:40:30.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rcenzItGcFpT51mU1dniRZWY9zQ=', 'Content-Length': '0'} 2023-11-02T18:40:30.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071090e4b4ef08083-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:40:30.489 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:40:30.490 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:40:30.490 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.491 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.491 DEBUG:boto:Method: GET 2023-11-02T18:40:30.492 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.492 DEBUG:boto:Data: 2023-11-02T18:40:30.493 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.493 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.494 DEBUG:boto:Params: {} 2023-11-02T18:40:30.495 DEBUG:boto:Token: None 2023-11-02T18:40:30.495 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3uBqTCet3NwGGef/1tamTQjrtfc= 2023-11-02T18:40:30.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3uBqTCet3NwGGef/1tamTQjrtfc=', 'Content-Length': '0'} 2023-11-02T18:40:30.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7fe7f5f2ac21e26-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.500 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:40:30.500 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:40:30.501 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.501 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.502 DEBUG:boto:Method: GET 2023-11-02T18:40:30.502 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.503 DEBUG:boto:Data: 2023-11-02T18:40:30.503 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.504 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.504 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.505 DEBUG:boto:Params: {} 2023-11-02T18:40:30.505 DEBUG:boto:Token: None 2023-11-02T18:40:30.506 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:40:30.506 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3uBqTCet3NwGGef/1tamTQjrtfc= 2023-11-02T18:40:30.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3uBqTCet3NwGGef/1tamTQjrtfc=', 'Content-Length': '0'} 2023-11-02T18:40:30.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce0838ec43849c0a-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.510 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.513 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.514 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:40:30.514 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:40:30.515 DEBUG:boto:Method: GET 2023-11-02T18:40:30.515 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:40:30.516 DEBUG:boto:Data: 2023-11-02T18:40:30.516 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.517 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.517 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.517 DEBUG:boto:Params: {} 2023-11-02T18:40:30.518 DEBUG:boto:Token: None 2023-11-02T18:40:30.518 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.519 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4df13db8fa86a60-006543ed1e-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.525 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.526 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.527 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.527 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:40:30.527 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:40:30.528 DEBUG:boto:Method: GET 2023-11-02T18:40:30.528 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:40:30.529 DEBUG:boto:Data: 2023-11-02T18:40:30.529 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.530 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.530 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.531 DEBUG:boto:Params: {} 2023-11-02T18:40:30.531 DEBUG:boto:Token: None 2023-11-02T18:40:30.532 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.532 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1722fad396a1235-006543ed1e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.541 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.542 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:40:30.542 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:40:30.543 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.543 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:40:30.544 DEBUG:boto:Data: 2023-11-02T18:40:30.544 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.544 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.545 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.545 DEBUG:boto:Params: {} 2023-11-02T18:40:30.546 DEBUG:boto:Token: None 2023-11-02T18:40:30.547 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj76 2023-11-02T18:40:30.547 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kFnWqpUn0r2w3/X7+n1ecydy+Ho= 2023-11-02T18:40:30.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kFnWqpUn0r2w3/X7+n1ecydy+Ho=', 'Content-Length': '0'} 2023-11-02T18:40:30.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041c8725d96783199-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.551 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:40:30.552 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:40:30.552 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.553 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:40:30.553 DEBUG:boto:Data: 2023-11-02T18:40:30.554 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.555 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.555 DEBUG:boto:Params: {} 2023-11-02T18:40:30.556 DEBUG:boto:Token: None 2023-11-02T18:40:30.556 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj76 2023-11-02T18:40:30.557 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kFnWqpUn0r2w3/X7+n1ecydy+Ho= 2023-11-02T18:40:30.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kFnWqpUn0r2w3/X7+n1ecydy+Ho=', 'Content-Length': '0'} 2023-11-02T18:40:30.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf278dc19efc6368-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:40:30.561 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:40:30.562 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:40:30.562 DEBUG:boto:Method: GET 2023-11-02T18:40:30.563 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:40:30.563 DEBUG:boto:Data: 2023-11-02T18:40:30.564 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.565 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.565 DEBUG:boto:Params: {} 2023-11-02T18:40:30.566 DEBUG:boto:Token: None 2023-11-02T18:40:30.566 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj76 2023-11-02T18:40:30.567 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PMLz9AJIDN/hIF2g7as4edxw5PY= 2023-11-02T18:40:30.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PMLz9AJIDN/hIF2g7as4edxw5PY=', 'Content-Length': '0'} 2023-11-02T18:40:30.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebdf4e8673bf4a75-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.614 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:40:30.615 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:40:30.615 DEBUG:boto:Method: GET 2023-11-02T18:40:30.616 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:40:30.616 DEBUG:boto:Data: 2023-11-02T18:40:30.617 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.617 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.618 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.618 DEBUG:boto:Params: {} 2023-11-02T18:40:30.619 DEBUG:boto:Token: None 2023-11-02T18:40:30.619 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj76 2023-11-02T18:40:30.620 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PMLz9AJIDN/hIF2g7as4edxw5PY= 2023-11-02T18:40:30.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PMLz9AJIDN/hIF2g7as4edxw5PY=', 'Content-Length': '0'} 2023-11-02T18:40:30.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7c2866f8909aa97-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.623 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.624 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.624 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:40:30.625 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:40:30.625 DEBUG:boto:Method: GET 2023-11-02T18:40:30.626 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:40:30.626 DEBUG:boto:Data: 2023-11-02T18:40:30.627 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.627 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.627 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.628 DEBUG:boto:Params: {} 2023-11-02T18:40:30.628 DEBUG:boto:Token: None 2023-11-02T18:40:30.629 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.629 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cee1287184cb2a00-006543ed1e-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.634 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.635 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.636 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.636 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:40:30.636 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:40:30.637 DEBUG:boto:Method: GET 2023-11-02T18:40:30.637 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:40:30.638 DEBUG:boto:Data: 2023-11-02T18:40:30.638 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.639 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.640 DEBUG:boto:Params: {} 2023-11-02T18:40:30.640 DEBUG:boto:Token: None 2023-11-02T18:40:30.641 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebe020e0dfa0681f-006543ed1e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.649 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:40:30.651 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:40:30.651 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:40:30.652 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.652 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.654 DEBUG:boto:Method: GET 2023-11-02T18:40:30.655 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.655 DEBUG:boto:Data: 2023-11-02T18:40:30.656 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.656 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.657 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.657 DEBUG:boto:Params: {} 2023-11-02T18:40:30.658 DEBUG:boto:Token: None 2023-11-02T18:40:30.658 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.659 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oj9NSmLRQZOjzhJySCrEFBbLzfk= 2023-11-02T18:40:30.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oj9NSmLRQZOjzhJySCrEFBbLzfk=', 'Content-Length': '0'} 2023-11-02T18:40:30.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f3f834ec88f2c9a-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.663 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:40:30.663 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:40:30.663 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.664 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.664 DEBUG:boto:Method: GET 2023-11-02T18:40:30.665 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.665 DEBUG:boto:Data: 2023-11-02T18:40:30.666 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.666 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.667 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.667 DEBUG:boto:Params: {} 2023-11-02T18:40:30.668 DEBUG:boto:Token: None 2023-11-02T18:40:30.668 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.669 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oj9NSmLRQZOjzhJySCrEFBbLzfk= 2023-11-02T18:40:30.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oj9NSmLRQZOjzhJySCrEFBbLzfk=', 'Content-Length': '0'} 2023-11-02T18:40:30.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e2e279e179f58fd-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.672 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:40:30.673 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:40:30.673 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.674 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.674 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.675 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.675 DEBUG:boto:Data: 2023-11-02T18:40:30.675 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.676 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.676 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.677 DEBUG:boto:Params: {} 2023-11-02T18:40:30.677 DEBUG:boto:Token: None 2023-11-02T18:40:30.678 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.678 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2H7hIihlVuaoam/9iU2kO/8/QPc= 2023-11-02T18:40:30.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2H7hIihlVuaoam/9iU2kO/8/QPc=', 'Content-Length': '0'} 2023-11-02T18:40:30.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a12e6fc15a9ec34e-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.682 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:40:30.683 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:40:30.683 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.683 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.684 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.684 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.685 DEBUG:boto:Data: 2023-11-02T18:40:30.685 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.686 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.687 DEBUG:boto:Params: {} 2023-11-02T18:40:30.687 DEBUG:boto:Token: None 2023-11-02T18:40:30.687 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.688 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2H7hIihlVuaoam/9iU2kO/8/QPc= 2023-11-02T18:40:30.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2H7hIihlVuaoam/9iU2kO/8/QPc=', 'Content-Length': '0'} 2023-11-02T18:40:30.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f2f4df6f892b281-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:40:30.692 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:40:30.693 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:40:30.693 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.694 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.694 DEBUG:boto:Method: GET 2023-11-02T18:40:30.695 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.695 DEBUG:boto:Data: 2023-11-02T18:40:30.696 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.697 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.697 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.698 DEBUG:boto:Params: {} 2023-11-02T18:40:30.699 DEBUG:boto:Token: None 2023-11-02T18:40:30.699 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.700 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oj9NSmLRQZOjzhJySCrEFBbLzfk= 2023-11-02T18:40:30.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oj9NSmLRQZOjzhJySCrEFBbLzfk=', 'Content-Length': '0'} 2023-11-02T18:40:30.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d38f871b99a96fd-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.705 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:40:30.706 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:40:30.706 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.707 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.707 DEBUG:boto:Method: GET 2023-11-02T18:40:30.708 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.708 DEBUG:boto:Data: 2023-11-02T18:40:30.709 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.709 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.710 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.710 DEBUG:boto:Params: {} 2023-11-02T18:40:30.711 DEBUG:boto:Token: None 2023-11-02T18:40:30.711 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:40:30.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oj9NSmLRQZOjzhJySCrEFBbLzfk= 2023-11-02T18:40:30.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oj9NSmLRQZOjzhJySCrEFBbLzfk=', 'Content-Length': '0'} 2023-11-02T18:40:30.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1144c997314283b-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.762 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.763 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.763 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:40:30.764 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:40:30.764 DEBUG:boto:Method: GET 2023-11-02T18:40:30.765 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:40:30.765 DEBUG:boto:Data: 2023-11-02T18:40:30.765 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.766 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.766 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.767 DEBUG:boto:Params: {} 2023-11-02T18:40:30.767 DEBUG:boto:Token: None 2023-11-02T18:40:30.768 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d496ab440d34b834-006543ed1e-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.774 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.775 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.775 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.776 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:40:30.776 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:40:30.776 DEBUG:boto:Method: GET 2023-11-02T18:40:30.777 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:40:30.777 DEBUG:boto:Data: 2023-11-02T18:40:30.778 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.778 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.779 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.779 DEBUG:boto:Params: {} 2023-11-02T18:40:30.780 DEBUG:boto:Token: None 2023-11-02T18:40:30.780 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.781 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006bb1861af8f393c-006543ed1e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.789 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.790 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:40:30.791 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:40:30.791 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.792 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:40:30.793 DEBUG:boto:Data: 2023-11-02T18:40:30.793 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.794 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.794 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.795 DEBUG:boto:Params: {} 2023-11-02T18:40:30.796 DEBUG:boto:Token: None 2023-11-02T18:40:30.796 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj85 2023-11-02T18:40:30.797 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9YSw05dvO/RG+uElq9QyUHc35Sg= 2023-11-02T18:40:30.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9YSw05dvO/RG+uElq9QyUHc35Sg=', 'Content-Length': '0'} 2023-11-02T18:40:30.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d29c8c36fd0c4a2-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.801 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:40:30.802 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:40:30.802 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.803 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:40:30.803 DEBUG:boto:Data: 2023-11-02T18:40:30.803 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.804 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.804 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.805 DEBUG:boto:Params: {} 2023-11-02T18:40:30.805 DEBUG:boto:Token: None 2023-11-02T18:40:30.806 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj85 2023-11-02T18:40:30.806 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9YSw05dvO/RG+uElq9QyUHc35Sg= 2023-11-02T18:40:30.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9YSw05dvO/RG+uElq9QyUHc35Sg=', 'Content-Length': '0'} 2023-11-02T18:40:30.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e9c9980bbd32c67-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:40:30.810 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:40:30.811 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:40:30.811 DEBUG:boto:Method: GET 2023-11-02T18:40:30.812 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:40:30.812 DEBUG:boto:Data: 2023-11-02T18:40:30.812 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.813 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.813 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.814 DEBUG:boto:Params: {} 2023-11-02T18:40:30.814 DEBUG:boto:Token: None 2023-11-02T18:40:30.815 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj85 2023-11-02T18:40:30.815 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2aK8SDJfOkl1eSTNZwYV4cSP6qE= 2023-11-02T18:40:30.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2aK8SDJfOkl1eSTNZwYV4cSP6qE=', 'Content-Length': '0'} 2023-11-02T18:40:30.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d06152950967844d-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.820 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:40:30.820 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:40:30.820 DEBUG:boto:Method: GET 2023-11-02T18:40:30.821 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:40:30.821 DEBUG:boto:Data: 2023-11-02T18:40:30.822 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.822 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.823 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.823 DEBUG:boto:Params: {} 2023-11-02T18:40:30.823 DEBUG:boto:Token: None 2023-11-02T18:40:30.824 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj85 2023-11-02T18:40:30.824 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2aK8SDJfOkl1eSTNZwYV4cSP6qE= 2023-11-02T18:40:30.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2aK8SDJfOkl1eSTNZwYV4cSP6qE=', 'Content-Length': '0'} 2023-11-02T18:40:30.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1c5a838e83a3cf4-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.829 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.829 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.830 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:40:30.831 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:40:30.831 DEBUG:boto:Method: GET 2023-11-02T18:40:30.832 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:40:30.833 DEBUG:boto:Data: 2023-11-02T18:40:30.833 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.833 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.834 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.835 DEBUG:boto:Params: {} 2023-11-02T18:40:30.835 DEBUG:boto:Token: None 2023-11-02T18:40:30.836 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.836 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009488cb4fef971906-006543ed1e-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.882 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.883 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:30.883 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:30.884 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:40:30.884 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:40:30.885 DEBUG:boto:Method: GET 2023-11-02T18:40:30.885 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:40:30.886 DEBUG:boto:Data: 2023-11-02T18:40:30.886 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.887 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.887 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.887 DEBUG:boto:Params: {} 2023-11-02T18:40:30.888 DEBUG:boto:Token: None 2023-11-02T18:40:30.888 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/ 2023-11-02T18:40:30.889 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg= 2023-11-02T18:40:30.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cj11SPTBhNT3RIztwfiTwLZzHfg=', 'Content-Length': '0'} 2023-11-02T18:40:30.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006d2852db301104c-006543ed1e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.898 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:30.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:40:30.900 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:40:30.900 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:40:30.901 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.902 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.903 DEBUG:boto:Method: GET 2023-11-02T18:40:30.903 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.904 DEBUG:boto:Data: 2023-11-02T18:40:30.905 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.906 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.907 DEBUG:boto:Params: {} 2023-11-02T18:40:30.908 DEBUG:boto:Token: None 2023-11-02T18:40:30.909 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.910 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6BwpW1o0VsqJzd5DzThs7U1KVRA= 2023-11-02T18:40:30.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6BwpW1o0VsqJzd5DzThs7U1KVRA=', 'Content-Length': '0'} 2023-11-02T18:40:30.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ee13a2e8de0edda-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.915 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:40:30.916 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:40:30.917 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.917 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.918 DEBUG:boto:Method: GET 2023-11-02T18:40:30.919 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.920 DEBUG:boto:Data: 2023-11-02T18:40:30.920 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.921 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.921 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.922 DEBUG:boto:Params: {} 2023-11-02T18:40:30.922 DEBUG:boto:Token: None 2023-11-02T18:40:30.923 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.923 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6BwpW1o0VsqJzd5DzThs7U1KVRA= 2023-11-02T18:40:30.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6BwpW1o0VsqJzd5DzThs7U1KVRA=', 'Content-Length': '0'} 2023-11-02T18:40:30.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042241563956163b3-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.974 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:40:30.975 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:40:30.975 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.976 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.976 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.977 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.977 DEBUG:boto:Data: 2023-11-02T18:40:30.978 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.978 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.979 DEBUG:boto:Port: 8001 2023-11-02T18:40:30.979 DEBUG:boto:Params: {} 2023-11-02T18:40:30.980 DEBUG:boto:Token: None 2023-11-02T18:40:30.980 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.981 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dRVykslwxp0ezhzczYFpCONaJjw= 2023-11-02T18:40:30.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dRVykslwxp0ezhzczYFpCONaJjw=', 'Content-Length': '0'} 2023-11-02T18:40:30.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000657a81ce05d79fe1-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.985 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:40:30.985 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:40:30.986 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.986 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.987 DEBUG:boto:Method: HEAD 2023-11-02T18:40:30.987 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.987 DEBUG:boto:Data: 2023-11-02T18:40:30.988 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.988 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:30.989 DEBUG:boto:Port: 8000 2023-11-02T18:40:30.989 DEBUG:boto:Params: {} 2023-11-02T18:40:30.990 DEBUG:boto:Token: None 2023-11-02T18:40:30.990 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:30 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.991 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dRVykslwxp0ezhzczYFpCONaJjw= 2023-11-02T18:40:30.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dRVykslwxp0ezhzczYFpCONaJjw=', 'Content-Length': '0'} 2023-11-02T18:40:30.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025703a2a4034f7d6-006543ed1e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:30.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:40:30.995 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:40:30.996 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:40:30.996 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.996 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.997 DEBUG:boto:Method: GET 2023-11-02T18:40:30.997 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:40:30.998 DEBUG:boto:Data: 2023-11-02T18:40:30.998 DEBUG:boto:Headers: {} 2023-11-02T18:40:30.999 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:30.999 DEBUG:boto:Port: 8001 2023-11-02T18:40:31.000 DEBUG:boto:Params: {} 2023-11-02T18:40:31.000 DEBUG:boto:Token: None 2023-11-02T18:40:31.001 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:40:31.001 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g7079GAQgASXx6hwJ+G4p1mQiSE= 2023-11-02T18:40:31.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g7079GAQgASXx6hwJ+G4p1mQiSE=', 'Content-Length': '0'} 2023-11-02T18:40:31.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9677149a6e2bad2-006543ed1e-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.005 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:40:31.006 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:40:31.006 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:40:31.007 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:40:31.007 DEBUG:boto:Method: GET 2023-11-02T18:40:31.008 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:40:31.008 DEBUG:boto:Data: 2023-11-02T18:40:31.009 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.009 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.010 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.010 DEBUG:boto:Params: {} 2023-11-02T18:40:31.011 DEBUG:boto:Token: None 2023-11-02T18:40:31.011 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:40:31.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g7079GAQgASXx6hwJ+G4p1mQiSE= 2023-11-02T18:40:31.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g7079GAQgASXx6hwJ+G4p1mQiSE=', 'Content-Length': '0'} 2023-11-02T18:40:31.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a20a1223fe49e84-006543ed1f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.016 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.017 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.017 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:40:31.017 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:40:31.018 DEBUG:boto:Method: GET 2023-11-02T18:40:31.018 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:40:31.019 DEBUG:boto:Data: 2023-11-02T18:40:31.019 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.020 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:31.020 DEBUG:boto:Port: 8001 2023-11-02T18:40:31.021 DEBUG:boto:Params: {} 2023-11-02T18:40:31.021 DEBUG:boto:Token: None 2023-11-02T18:40:31.022 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.022 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4= 2023-11-02T18:40:31.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4=', 'Content-Length': '0'} 2023-11-02T18:40:31.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ebce3478ae8f296-006543ed1f-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.028 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:31.029 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.029 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.030 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:40:31.030 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:40:31.031 DEBUG:boto:Method: GET 2023-11-02T18:40:31.031 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:40:31.032 DEBUG:boto:Data: 2023-11-02T18:40:31.032 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.033 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.033 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.033 DEBUG:boto:Params: {} 2023-11-02T18:40:31.034 DEBUG:boto:Token: None 2023-11-02T18:40:31.034 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4= 2023-11-02T18:40:31.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4=', 'Content-Length': '0'} 2023-11-02T18:40:31.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d19e813e33f77d6c-006543ed1f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.043 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:31.044 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:40:31.044 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:40:31.045 DEBUG:boto:Method: HEAD 2023-11-02T18:40:31.045 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:40:31.046 DEBUG:boto:Data: 2023-11-02T18:40:31.046 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.047 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:31.047 DEBUG:boto:Port: 8001 2023-11-02T18:40:31.047 DEBUG:boto:Params: {} 2023-11-02T18:40:31.048 DEBUG:boto:Token: None 2023-11-02T18:40:31.049 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj92 2023-11-02T18:40:31.049 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BSa3aCSNuDSZSqggg4th6vZSRPY= 2023-11-02T18:40:31.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BSa3aCSNuDSZSqggg4th6vZSRPY=', 'Content-Length': '0'} 2023-11-02T18:40:31.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000571b091a037a4f75-006543ed1f-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.053 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:40:31.053 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:40:31.054 DEBUG:boto:Method: HEAD 2023-11-02T18:40:31.054 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:40:31.055 DEBUG:boto:Data: 2023-11-02T18:40:31.055 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.056 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.056 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.056 DEBUG:boto:Params: {} 2023-11-02T18:40:31.057 DEBUG:boto:Token: None 2023-11-02T18:40:31.057 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj92 2023-11-02T18:40:31.058 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BSa3aCSNuDSZSqggg4th6vZSRPY= 2023-11-02T18:40:31.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BSa3aCSNuDSZSqggg4th6vZSRPY=', 'Content-Length': '0'} 2023-11-02T18:40:31.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f96ace14c058230b-006543ed1f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:40:31.062 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:40:31.062 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:40:31.063 DEBUG:boto:Method: GET 2023-11-02T18:40:31.063 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:40:31.064 DEBUG:boto:Data: 2023-11-02T18:40:31.064 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.065 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:31.065 DEBUG:boto:Port: 8001 2023-11-02T18:40:31.065 DEBUG:boto:Params: {} 2023-11-02T18:40:31.066 DEBUG:boto:Token: None 2023-11-02T18:40:31.066 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj92 2023-11-02T18:40:31.067 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JwEGsTCyp1G+AT0GN24/QGt26i4= 2023-11-02T18:40:31.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JwEGsTCyp1G+AT0GN24/QGt26i4=', 'Content-Length': '0'} 2023-11-02T18:40:31.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042c32a4aab0fa41c-006543ed1f-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.118 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:40:31.119 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:40:31.119 DEBUG:boto:Method: GET 2023-11-02T18:40:31.120 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:40:31.121 DEBUG:boto:Data: 2023-11-02T18:40:31.121 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.121 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.122 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.122 DEBUG:boto:Params: {} 2023-11-02T18:40:31.123 DEBUG:boto:Token: None 2023-11-02T18:40:31.123 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj92 2023-11-02T18:40:31.124 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JwEGsTCyp1G+AT0GN24/QGt26i4= 2023-11-02T18:40:31.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JwEGsTCyp1G+AT0GN24/QGt26i4=', 'Content-Length': '0'} 2023-11-02T18:40:31.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e07880828c9ba59b-006543ed1f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.128 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.129 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.129 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:40:31.130 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:40:31.130 DEBUG:boto:Method: GET 2023-11-02T18:40:31.131 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:40:31.133 DEBUG:boto:Data: 2023-11-02T18:40:31.134 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.134 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:31.135 DEBUG:boto:Port: 8001 2023-11-02T18:40:31.135 DEBUG:boto:Params: {} 2023-11-02T18:40:31.136 DEBUG:boto:Token: None 2023-11-02T18:40:31.137 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.137 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4= 2023-11-02T18:40:31.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4=', 'Content-Length': '0'} 2023-11-02T18:40:31.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c069745e150b1306-006543ed1f-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.144 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:31.144 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.145 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.146 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:40:31.146 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:40:31.147 DEBUG:boto:Method: GET 2023-11-02T18:40:31.147 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:40:31.147 DEBUG:boto:Data: 2023-11-02T18:40:31.148 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.149 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.149 DEBUG:boto:Params: {} 2023-11-02T18:40:31.150 DEBUG:boto:Token: None 2023-11-02T18:40:31.150 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4= 2023-11-02T18:40:31.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4=', 'Content-Length': '0'} 2023-11-02T18:40:31.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000788bc72b5f5498f7-006543ed1f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.160 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:31.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:40:31.161 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:40:31.162 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:40:31.162 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.163 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.163 DEBUG:boto:Method: GET 2023-11-02T18:40:31.164 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.164 DEBUG:boto:Data: 2023-11-02T18:40:31.164 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.165 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:31.165 DEBUG:boto:Port: 8001 2023-11-02T18:40:31.166 DEBUG:boto:Params: {} 2023-11-02T18:40:31.166 DEBUG:boto:Token: None 2023-11-02T18:40:31.167 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.167 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ihho3I3HcUloSoJfLJwx4Dt8GqU= 2023-11-02T18:40:31.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ihho3I3HcUloSoJfLJwx4Dt8GqU=', 'Content-Length': '0'} 2023-11-02T18:40:31.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d339ef54b43b16b9-006543ed1f-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.171 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:40:31.172 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:40:31.172 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.172 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.173 DEBUG:boto:Method: GET 2023-11-02T18:40:31.173 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.174 DEBUG:boto:Data: 2023-11-02T18:40:31.174 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.175 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.175 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.176 DEBUG:boto:Params: {} 2023-11-02T18:40:31.176 DEBUG:boto:Token: None 2023-11-02T18:40:31.177 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ihho3I3HcUloSoJfLJwx4Dt8GqU= 2023-11-02T18:40:31.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ihho3I3HcUloSoJfLJwx4Dt8GqU=', 'Content-Length': '0'} 2023-11-02T18:40:31.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d35b112db220a8ef-006543ed1f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.182 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:40:31.182 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:40:31.183 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.183 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.183 DEBUG:boto:Method: HEAD 2023-11-02T18:40:31.184 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.184 DEBUG:boto:Data: 2023-11-02T18:40:31.185 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.185 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:31.186 DEBUG:boto:Port: 8001 2023-11-02T18:40:31.186 DEBUG:boto:Params: {} 2023-11-02T18:40:31.187 DEBUG:boto:Token: None 2023-11-02T18:40:31.187 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.188 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6Y4Xy0Pv7HJd8hR4mMuaWcrcyEI= 2023-11-02T18:40:31.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6Y4Xy0Pv7HJd8hR4mMuaWcrcyEI=', 'Content-Length': '0'} 2023-11-02T18:40:31.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ecd57f0db5e1caf-006543ed1f-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.192 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:40:31.192 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:40:31.193 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.193 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.193 DEBUG:boto:Method: HEAD 2023-11-02T18:40:31.194 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.194 DEBUG:boto:Data: 2023-11-02T18:40:31.195 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.195 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.196 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.196 DEBUG:boto:Params: {} 2023-11-02T18:40:31.196 DEBUG:boto:Token: None 2023-11-02T18:40:31.197 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.197 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6Y4Xy0Pv7HJd8hR4mMuaWcrcyEI= 2023-11-02T18:40:31.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6Y4Xy0Pv7HJd8hR4mMuaWcrcyEI=', 'Content-Length': '0'} 2023-11-02T18:40:31.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000651edde30678fc95-006543ed1f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:40:31.202 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:40:31.202 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:40:31.203 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.203 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.204 DEBUG:boto:Method: GET 2023-11-02T18:40:31.204 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.205 DEBUG:boto:Data: 2023-11-02T18:40:31.205 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.206 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:31.206 DEBUG:boto:Port: 8001 2023-11-02T18:40:31.207 DEBUG:boto:Params: {} 2023-11-02T18:40:31.207 DEBUG:boto:Token: None 2023-11-02T18:40:31.208 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ihho3I3HcUloSoJfLJwx4Dt8GqU= 2023-11-02T18:40:31.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ihho3I3HcUloSoJfLJwx4Dt8GqU=', 'Content-Length': '0'} 2023-11-02T18:40:31.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae21d3e725393593-006543ed1f-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.254 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:40:31.255 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:40:31.256 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.256 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.257 DEBUG:boto:Method: GET 2023-11-02T18:40:31.257 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.258 DEBUG:boto:Data: 2023-11-02T18:40:31.258 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.259 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.259 DEBUG:boto:Params: {} 2023-11-02T18:40:31.260 DEBUG:boto:Token: None 2023-11-02T18:40:31.261 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:40:31.261 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ihho3I3HcUloSoJfLJwx4Dt8GqU= 2023-11-02T18:40:31.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ihho3I3HcUloSoJfLJwx4Dt8GqU=', 'Content-Length': '0'} 2023-11-02T18:40:31.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc1ee96ad2649bcb-006543ed1f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.265 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.266 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.266 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:40:31.267 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:40:31.267 DEBUG:boto:Method: GET 2023-11-02T18:40:31.268 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:40:31.268 DEBUG:boto:Data: 2023-11-02T18:40:31.269 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.269 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:31.270 DEBUG:boto:Port: 8001 2023-11-02T18:40:31.270 DEBUG:boto:Params: {} 2023-11-02T18:40:31.271 DEBUG:boto:Token: None 2023-11-02T18:40:31.271 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.272 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4= 2023-11-02T18:40:31.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4=', 'Content-Length': '0'} 2023-11-02T18:40:31.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f947d5d851356bea-006543ed1f-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.278 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:31.279 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.279 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.280 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:40:31.280 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:40:31.281 DEBUG:boto:Method: GET 2023-11-02T18:40:31.281 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:40:31.282 DEBUG:boto:Data: 2023-11-02T18:40:31.282 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.283 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.283 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.284 DEBUG:boto:Params: {} 2023-11-02T18:40:31.284 DEBUG:boto:Token: None 2023-11-02T18:40:31.285 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.285 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4= 2023-11-02T18:40:31.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4=', 'Content-Length': '0'} 2023-11-02T18:40:31.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004260d8a9bedea82d-006543ed1f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.294 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:31.297 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:40:31.297 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:40:31.298 DEBUG:boto:Method: HEAD 2023-11-02T18:40:31.298 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:40:31.299 DEBUG:boto:Data: 2023-11-02T18:40:31.299 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.300 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:31.300 DEBUG:boto:Port: 8001 2023-11-02T18:40:31.301 DEBUG:boto:Params: {} 2023-11-02T18:40:31.301 DEBUG:boto:Token: None 2023-11-02T18:40:31.301 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj95 2023-11-02T18:40:31.302 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w/qyNZ3RerGZAnvnNt+j5YsqYQk= 2023-11-02T18:40:31.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w/qyNZ3RerGZAnvnNt+j5YsqYQk=', 'Content-Length': '0'} 2023-11-02T18:40:31.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b851705af412806-006543ed1f-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.306 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:40:31.306 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:40:31.307 DEBUG:boto:Method: HEAD 2023-11-02T18:40:31.307 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:40:31.307 DEBUG:boto:Data: 2023-11-02T18:40:31.308 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.308 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.309 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.309 DEBUG:boto:Params: {} 2023-11-02T18:40:31.310 DEBUG:boto:Token: None 2023-11-02T18:40:31.310 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj95 2023-11-02T18:40:31.311 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w/qyNZ3RerGZAnvnNt+j5YsqYQk= 2023-11-02T18:40:31.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w/qyNZ3RerGZAnvnNt+j5YsqYQk=', 'Content-Length': '0'} 2023-11-02T18:40:31.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000221f5006e3011bac-006543ed1f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:40:31.315 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:40:31.315 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:40:31.316 DEBUG:boto:Method: GET 2023-11-02T18:40:31.316 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:40:31.317 DEBUG:boto:Data: 2023-11-02T18:40:31.317 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.318 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:31.318 DEBUG:boto:Port: 8001 2023-11-02T18:40:31.319 DEBUG:boto:Params: {} 2023-11-02T18:40:31.319 DEBUG:boto:Token: None 2023-11-02T18:40:31.320 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj95 2023-11-02T18:40:31.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aap6BSW5H6s4nyOY3dbeKamQxJM= 2023-11-02T18:40:31.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aap6BSW5H6s4nyOY3dbeKamQxJM=', 'Content-Length': '0'} 2023-11-02T18:40:31.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046e04b78c375c9a0-006543ed1f-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.325 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:40:31.325 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:40:31.326 DEBUG:boto:Method: GET 2023-11-02T18:40:31.326 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:40:31.327 DEBUG:boto:Data: 2023-11-02T18:40:31.327 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.328 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.329 DEBUG:boto:Params: {} 2023-11-02T18:40:31.329 DEBUG:boto:Token: None 2023-11-02T18:40:31.329 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj95 2023-11-02T18:40:31.330 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aap6BSW5H6s4nyOY3dbeKamQxJM= 2023-11-02T18:40:31.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aap6BSW5H6s4nyOY3dbeKamQxJM=', 'Content-Length': '0'} 2023-11-02T18:40:31.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c89aa910213b6ce1-006543ed1f-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.374 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.375 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.375 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:40:31.376 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:40:31.376 DEBUG:boto:Method: GET 2023-11-02T18:40:31.377 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:40:31.377 DEBUG:boto:Data: 2023-11-02T18:40:31.378 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.378 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:40:31.378 DEBUG:boto:Port: 8001 2023-11-02T18:40:31.379 DEBUG:boto:Params: {} 2023-11-02T18:40:31.379 DEBUG:boto:Token: None 2023-11-02T18:40:31.380 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.380 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4= 2023-11-02T18:40:31.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4=', 'Content-Length': '0'} 2023-11-02T18:40:31.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065e4f02f9b176cfd-006543ed1f-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.387 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:31.388 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.389 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.389 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:40:31.390 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:40:31.390 DEBUG:boto:Method: GET 2023-11-02T18:40:31.391 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:40:31.391 DEBUG:boto:Data: 2023-11-02T18:40:31.392 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.392 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.392 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.393 DEBUG:boto:Params: {} 2023-11-02T18:40:31.393 DEBUG:boto:Token: None 2023-11-02T18:40:31.394 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4= 2023-11-02T18:40:31.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fa1dmuHSe1sjJiTbEys5I6PWeU4=', 'Content-Length': '0'} 2023-11-02T18:40:31.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4b35f7785b68595-006543ed1f-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.404 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:40:31.405 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:40:31.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:40:31.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:40:31.406 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:40:31.454 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:40:31.454 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:40:31.538 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:40:31.538 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-59 2023-11-02T18:40:31.556 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 20 2023-11-02T18:40:31.656 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.656 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.657 DEBUG:boto:Method: HEAD 2023-11-02T18:40:31.657 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:40:31.658 DEBUG:boto:Data: 2023-11-02T18:40:31.658 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.659 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.659 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.659 DEBUG:boto:Params: {} 2023-11-02T18:40:31.660 DEBUG:boto:Token: None 2023-11-02T18:40:31.660 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.661 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MK8CYFLYnjnvOhU/11JNTcRenjg= 2023-11-02T18:40:31.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MK8CYFLYnjnvOhU/11JNTcRenjg=', 'Content-Length': '0'} 2023-11-02T18:40:31.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066b33934e61659e8-006543ed1f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.668 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:40:31.668 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:40:31.669 DEBUG:boto:Method: PUT 2023-11-02T18:40:31.669 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:40:31.669 DEBUG:boto:Data: 2023-11-02T18:40:31.670 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:40:31.670 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.671 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.672 DEBUG:boto:Params: {} 2023-11-02T18:40:31.672 DEBUG:boto:Token: None 2023-11-02T18:40:31.673 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj29 2023-11-02T18:40:31.673 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X7MXUbVULLsHbI7OV+Q6jYg1EDk= 2023-11-02T18:40:31.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X7MXUbVULLsHbI7OV+Q6jYg1EDk='} 2023-11-02T18:40:31.685 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093f49a37efa19f7e-006543ed1f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.685 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.686 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.686 DEBUG:boto:Method: HEAD 2023-11-02T18:40:31.687 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:40:31.687 DEBUG:boto:Data: 2023-11-02T18:40:31.687 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.688 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.688 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.689 DEBUG:boto:Params: {} 2023-11-02T18:40:31.689 DEBUG:boto:Token: None 2023-11-02T18:40:31.690 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.690 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MK8CYFLYnjnvOhU/11JNTcRenjg= 2023-11-02T18:40:31.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MK8CYFLYnjnvOhU/11JNTcRenjg=', 'Content-Length': '0'} 2023-11-02T18:40:31.697 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b70437738f9aeacd-006543ed1f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.697 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:40:31.698 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:40:31.698 DEBUG:boto:Method: PUT 2023-11-02T18:40:31.699 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:40:31.699 DEBUG:boto:Data: 2023-11-02T18:40:31.700 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:40:31.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.700 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.701 DEBUG:boto:Params: {} 2023-11-02T18:40:31.701 DEBUG:boto:Token: None 2023-11-02T18:40:31.701 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj58 2023-11-02T18:40:31.702 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OYtDo1Ca9oKJPzoCy9/F8HqIG44= 2023-11-02T18:40:31.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OYtDo1Ca9oKJPzoCy9/F8HqIG44='} 2023-11-02T18:40:31.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f58dfaa8a2690d4-006543ed1f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.747 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.747 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.749 DEBUG:boto:Method: HEAD 2023-11-02T18:40:31.749 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:40:31.750 DEBUG:boto:Data: 2023-11-02T18:40:31.750 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.751 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.751 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.751 DEBUG:boto:Params: {} 2023-11-02T18:40:31.752 DEBUG:boto:Token: None 2023-11-02T18:40:31.752 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.753 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MK8CYFLYnjnvOhU/11JNTcRenjg= 2023-11-02T18:40:31.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MK8CYFLYnjnvOhU/11JNTcRenjg=', 'Content-Length': '0'} 2023-11-02T18:40:31.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b38efc3592670d55-006543ed1f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.760 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:40:31.761 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:40:31.761 DEBUG:boto:Method: PUT 2023-11-02T18:40:31.762 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:40:31.762 DEBUG:boto:Data: 2023-11-02T18:40:31.763 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:40:31.763 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.763 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.764 DEBUG:boto:Params: {} 2023-11-02T18:40:31.764 DEBUG:boto:Token: None 2023-11-02T18:40:31.765 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj87 2023-11-02T18:40:31.765 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YaTWhOt6lvPgqN0s7K4KBQmDi2Q= 2023-11-02T18:40:31.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YaTWhOt6lvPgqN0s7K4KBQmDi2Q='} 2023-11-02T18:40:31.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f75eef7d16020101-006543ed1f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.777 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.778 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.778 DEBUG:boto:Method: HEAD 2023-11-02T18:40:31.778 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:40:31.779 DEBUG:boto:Data: 2023-11-02T18:40:31.779 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.780 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.780 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.781 DEBUG:boto:Params: {} 2023-11-02T18:40:31.781 DEBUG:boto:Token: None 2023-11-02T18:40:31.782 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.783 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MK8CYFLYnjnvOhU/11JNTcRenjg= 2023-11-02T18:40:31.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MK8CYFLYnjnvOhU/11JNTcRenjg=', 'Content-Length': '0'} 2023-11-02T18:40:31.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9d26e0b252721be-006543ed1f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.790 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:40:31.791 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:40:31.791 DEBUG:boto:Method: PUT 2023-11-02T18:40:31.792 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:40:31.792 DEBUG:boto:Data: 2023-11-02T18:40:31.792 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:40:31.793 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.793 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.794 DEBUG:boto:Params: {} 2023-11-02T18:40:31.794 DEBUG:boto:Token: None 2023-11-02T18:40:31.795 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj116 2023-11-02T18:40:31.795 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sGwyEHUWFGKXgmC4tLuLKZvdEAs= 2023-11-02T18:40:31.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sGwyEHUWFGKXgmC4tLuLKZvdEAs='} 2023-11-02T18:40:31.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fa0b88e42e79cf7-006543ed1f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.839 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:40:31.840 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:40:31.840 DEBUG:boto:Method: HEAD 2023-11-02T18:40:31.841 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:40:31.841 DEBUG:boto:Data: 2023-11-02T18:40:31.842 DEBUG:boto:Headers: {} 2023-11-02T18:40:31.842 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.843 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.843 DEBUG:boto:Params: {} 2023-11-02T18:40:31.844 DEBUG:boto:Token: None 2023-11-02T18:40:31.844 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/ 2023-11-02T18:40:31.845 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MK8CYFLYnjnvOhU/11JNTcRenjg= 2023-11-02T18:40:31.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MK8CYFLYnjnvOhU/11JNTcRenjg=', 'Content-Length': '0'} 2023-11-02T18:40:31.853 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000954f8af45d1bacdf-006543ed1f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.854 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:40:31.854 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:40:31.855 DEBUG:boto:Method: PUT 2023-11-02T18:40:31.855 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:40:31.855 DEBUG:boto:Data: 2023-11-02T18:40:31.856 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:40:31.856 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:40:31.857 DEBUG:boto:Port: 8000 2023-11-02T18:40:31.857 DEBUG:boto:Params: {} 2023-11-02T18:40:31.858 DEBUG:boto:Token: None 2023-11-02T18:40:31.858 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:40:31 GMT /ibtljn-59/obj145 2023-11-02T18:40:31.859 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ugDWmsoU/MTrDiu3p+GMZ/zz40c= 2023-11-02T18:40:31.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:40:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ugDWmsoU/MTrDiu3p+GMZ/zz40c='} 2023-11-02T18:40:31.869 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c87a0546e3fc39a6-006543ed1f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:40:31.870 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-59 2023-11-02T18:40:31.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:40:31.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:40:31.872 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:40:31.924 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:40:31.925 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:40:31.994 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:31.984+0000 7f947f571f40 1 waiting to reach latest gen 4, on 3.. 2023-11-02T18:40:36.996 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:36.988+0000 7f947f571f40 1 waiting to reach latest gen 4, on 3.. 2023-11-02T18:40:41.997 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:41.988+0000 7f947f571f40 1 waiting to reach latest gen 4, on 3.. 2023-11-02T18:40:47.002 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:46.992+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:40:47.003 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:40:47.003 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:40:52.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:52.000+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:40:52.007 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:40:52.007 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:40:57.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:40:57.003+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:40:57.012 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:40:57.012 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:02.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:41:02.007+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:41:02.016 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:02.017 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:07.020 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:41:07.011+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:41:07.021 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:07.021 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:12.025 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:41:12.015+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:41:12.025 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:12.026 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:17.030 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:41:17.019+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:41:17.030 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.67.6, , , , , , , , , , , , , , , , , , 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:17.030 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:22.034 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:41:22.023+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:41:22.035 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.67.6, , , , , , , , , , , , , , , , , , 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:22.035 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:41:27.035+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:41:27.040 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.67.6, , , , , , , , , , , , , , , , , , 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:27.040 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:32.044 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:41:32.039+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:41:32.044 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.67.6, , , , , , , , , , , , , , , , , , 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:32.045 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:41:37.043+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:41:37.049 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.67.6, 00000000001.48.6, , , , , , , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:37.049 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:42.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:41:42.047+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:41:42.054 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.67.6, 00000000001.48.6, , , , , , , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:42.054 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:47.059 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:41:47.051+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:41:47.059 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.67.6, 00000000001.48.6, , , , , , , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:47.060 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:52.063 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:41:52.059+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:41:52.063 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.67.6, 00000000001.48.6, , , , , , , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:52.063 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:57.068 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:41:57.063+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:41:57.068 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.67.6, 00000000001.48.6, , , , , , , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:41:57.068 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:42:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:02.067+0000 7f947f571f40 1 waiting for incremental sync to catch up: 2023-11-02T18:42:02.072 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.67.6, 00000000001.48.6, , , , , , , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:42:02.073 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:42:07.077 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:07.071+0000 7f947f571f40 1 bucket sync caught up with source: 2023-11-02T18:42:07.078 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:42:07.078 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [00000000002.67.6, 00000000001.48.6, , , , , 00000000001.78.6, , , , , , , , , , , 00000000002.53.6, 00000000001.92.6, , , , , , , , , , ] 2023-11-02T18:42:07.078 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:07.071+0000 7f947f571f40 0 bucket checkpoint complete 2023-11-02T18:42:07.105 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-59 2023-11-02T18:42:07.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:42:07.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:42:07.107 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:42:07.159 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:42:07.159 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:42:07.228 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:07.223+0000 7f71c267cf40 1 waiting for incremental sync to catch up: 2023-11-02T18:42:07.228 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000003.303.6, 00000000003.282.6, 00000000001.303.6, 00000000001.305.6, 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:42:07.229 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000003.303.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:42:12.230 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:12.223+0000 7f71c267cf40 1 waiting for incremental sync to catch up: 2023-11-02T18:42:12.231 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000003.303.6, 00000000003.282.6, 00000000001.303.6, 00000000001.305.6, 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:42:12.231 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000003.303.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:42:17.233 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:17.227+0000 7f71c267cf40 1 waiting for incremental sync to catch up: 2023-11-02T18:42:17.234 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000003.303.6, 00000000003.282.6, 00000000001.303.6, 00000000001.305.6, 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:42:17.234 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000003.303.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:42:22.236 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:22.230+0000 7f71c267cf40 1 waiting for incremental sync to catch up: 2023-11-02T18:42:22.237 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000003.303.6, 00000000003.282.6, 00000000001.303.6, 00000000001.305.6, 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:42:22.237 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000003.303.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:42:27.239 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:27.234+0000 7f71c267cf40 1 bucket sync caught up with source: 2023-11-02T18:42:27.239 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000003.303.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:42:27.240 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000002.80.6, 00000000002.229.6, 00000000001.131.6, 00000000003.303.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000002.82.6, 00000000005.277.6, 00000000004.332.6] 2023-11-02T18:42:27.240 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:27.234+0000 7f71c267cf40 0 bucket checkpoint complete 2023-11-02T18:42:27.270 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:42:27.271 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.271 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.272 DEBUG:boto:Method: HEAD 2023-11-02T18:42:27.272 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:42:27.272 DEBUG:boto:Data: 2023-11-02T18:42:27.273 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.273 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.274 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.275 DEBUG:boto:Params: {} 2023-11-02T18:42:27.275 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:42:27.276 DEBUG:boto:Token: None 2023-11-02T18:42:27.276 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/ 2023-11-02T18:42:27.277 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5XyrVwu1r8vVcqCQZjLiAzg78Xw= 2023-11-02T18:42:27.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5XyrVwu1r8vVcqCQZjLiAzg78Xw=', 'Content-Length': '0'} 2023-11-02T18:42:27.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000697cb06a4c52f115-006543ed93-1b8c-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.284 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.285 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.285 DEBUG:boto:Method: HEAD 2023-11-02T18:42:27.286 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:42:27.286 DEBUG:boto:Data: 2023-11-02T18:42:27.286 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.287 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.287 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.288 DEBUG:boto:Params: {} 2023-11-02T18:42:27.288 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:42:27.289 DEBUG:boto:Token: None 2023-11-02T18:42:27.289 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/ 2023-11-02T18:42:27.290 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5XyrVwu1r8vVcqCQZjLiAzg78Xw= 2023-11-02T18:42:27.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5XyrVwu1r8vVcqCQZjLiAzg78Xw=', 'Content-Length': '0'} 2023-11-02T18:42:27.302 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052aea9fb477a309b-006543ed93-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.303 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:42:27.304 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.304 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.305 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:42:27.305 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:42:27.306 DEBUG:boto:Method: GET 2023-11-02T18:42:27.306 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:42:27.307 DEBUG:boto:Data: 2023-11-02T18:42:27.308 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.308 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.309 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.309 DEBUG:boto:Params: {} 2023-11-02T18:42:27.310 DEBUG:boto:Token: None 2023-11-02T18:42:27.310 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/?versions 2023-11-02T18:42:27.311 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:05FwU6zuAMNpc5FMZdADCoHmgps= 2023-11-02T18:42:27.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:05FwU6zuAMNpc5FMZdADCoHmgps=', 'Content-Length': '0'} 2023-11-02T18:42:27.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d776b3be0667c48-006543ed93-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.317 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:27.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:42:27.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T18:42:27.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-11-02T18:42:27.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-11-02T18:42:27.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:42:27.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:42:27.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:42:27.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T18:42:27.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-11-02T18:42:27.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:42:27.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:42:27.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:42:27.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:42:27.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T18:42:27.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:42:27.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:42:27.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:42:27.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-11-02T18:42:27.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:42:27.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T18:42:27.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:42:27.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:42:27.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-11-02T18:42:27.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T18:42:27.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:42:27.330 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:42:27.331 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.331 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.332 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:42:27.333 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:42:27.333 DEBUG:boto:Method: GET 2023-11-02T18:42:27.334 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:42:27.334 DEBUG:boto:Data: 2023-11-02T18:42:27.335 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.335 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.336 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.337 DEBUG:boto:Params: {} 2023-11-02T18:42:27.337 DEBUG:boto:Token: None 2023-11-02T18:42:27.338 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/?versions 2023-11-02T18:42:27.338 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:05FwU6zuAMNpc5FMZdADCoHmgps= 2023-11-02T18:42:27.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:05FwU6zuAMNpc5FMZdADCoHmgps=', 'Content-Length': '0'} 2023-11-02T18:42:27.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008376d5884fa0a3b3-006543ed93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.347 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:27.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:42:27.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T18:42:27.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-11-02T18:42:27.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-11-02T18:42:27.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:42:27.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:42:27.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:42:27.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T18:42:27.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-11-02T18:42:27.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:42:27.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:42:27.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:42:27.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:42:27.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T18:42:27.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:42:27.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:42:27.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:42:27.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-11-02T18:42:27.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:42:27.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T18:42:27.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:42:27.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:42:27.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-11-02T18:42:27.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T18:42:27.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:42:27.358 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.359 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.359 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:42:27.360 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:42:27.360 DEBUG:boto:Method: GET 2023-11-02T18:42:27.361 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:42:27.361 DEBUG:boto:Data: 2023-11-02T18:42:27.361 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.362 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.362 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.363 DEBUG:boto:Params: {} 2023-11-02T18:42:27.365 DEBUG:boto:Token: None 2023-11-02T18:42:27.366 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/?versions 2023-11-02T18:42:27.366 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:05FwU6zuAMNpc5FMZdADCoHmgps= 2023-11-02T18:42:27.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:05FwU6zuAMNpc5FMZdADCoHmgps=', 'Content-Length': '0'} 2023-11-02T18:42:27.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8eca09faa9b6b30-006543ed93-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.373 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:27.376 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.377 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.377 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:42:27.378 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:42:27.379 DEBUG:boto:Method: GET 2023-11-02T18:42:27.380 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:42:27.380 DEBUG:boto:Data: 2023-11-02T18:42:27.381 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.381 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.382 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.382 DEBUG:boto:Params: {} 2023-11-02T18:42:27.383 DEBUG:boto:Token: None 2023-11-02T18:42:27.383 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/?versions 2023-11-02T18:42:27.384 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:05FwU6zuAMNpc5FMZdADCoHmgps= 2023-11-02T18:42:27.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:05FwU6zuAMNpc5FMZdADCoHmgps=', 'Content-Length': '0'} 2023-11-02T18:42:27.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c75d037a39b4771-006543ed93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.394 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:27.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:42:27.396 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:42:27.397 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:42:27.397 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.398 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.398 DEBUG:boto:Method: GET 2023-11-02T18:42:27.399 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.399 DEBUG:boto:Data: 2023-11-02T18:42:27.400 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.400 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.401 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.401 DEBUG:boto:Params: {} 2023-11-02T18:42:27.401 DEBUG:boto:Token: None 2023-11-02T18:42:27.402 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.403 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QLY6zE/FdWbMj1WTqOIR2nf8og0= 2023-11-02T18:42:27.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QLY6zE/FdWbMj1WTqOIR2nf8og0=', 'Content-Length': '0'} 2023-11-02T18:42:27.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c16548d0bbb0367-006543ed93-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.407 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:42:27.408 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:42:27.409 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.409 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.410 DEBUG:boto:Method: GET 2023-11-02T18:42:27.410 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.410 DEBUG:boto:Data: 2023-11-02T18:42:27.411 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.411 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.412 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.412 DEBUG:boto:Params: {} 2023-11-02T18:42:27.413 DEBUG:boto:Token: None 2023-11-02T18:42:27.413 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QLY6zE/FdWbMj1WTqOIR2nf8og0= 2023-11-02T18:42:27.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QLY6zE/FdWbMj1WTqOIR2nf8og0=', 'Content-Length': '0'} 2023-11-02T18:42:27.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079454c59041b615c-006543ed93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.418 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:42:27.419 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:42:27.419 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.420 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.420 DEBUG:boto:Method: HEAD 2023-11-02T18:42:27.421 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.421 DEBUG:boto:Data: 2023-11-02T18:42:27.421 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.422 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.423 DEBUG:boto:Params: {} 2023-11-02T18:42:27.423 DEBUG:boto:Token: None 2023-11-02T18:42:27.424 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FwYj1PM2W94LkMKyOPV1Xm7a9y0= 2023-11-02T18:42:27.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FwYj1PM2W94LkMKyOPV1Xm7a9y0=', 'Content-Length': '0'} 2023-11-02T18:42:27.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c612910ac199a831-006543ed93-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.428 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:42:27.428 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:42:27.429 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.429 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.430 DEBUG:boto:Method: HEAD 2023-11-02T18:42:27.430 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.430 DEBUG:boto:Data: 2023-11-02T18:42:27.431 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.431 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.432 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.432 DEBUG:boto:Params: {} 2023-11-02T18:42:27.432 DEBUG:boto:Token: None 2023-11-02T18:42:27.433 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.433 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FwYj1PM2W94LkMKyOPV1Xm7a9y0= 2023-11-02T18:42:27.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FwYj1PM2W94LkMKyOPV1Xm7a9y0=', 'Content-Length': '0'} 2023-11-02T18:42:27.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000330ea72fc1ca9b8b-006543ed93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:42:27.438 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:42:27.439 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:42:27.439 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.439 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.440 DEBUG:boto:Method: GET 2023-11-02T18:42:27.440 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.441 DEBUG:boto:Data: 2023-11-02T18:42:27.441 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.441 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.442 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.442 DEBUG:boto:Params: {} 2023-11-02T18:42:27.443 DEBUG:boto:Token: None 2023-11-02T18:42:27.444 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QLY6zE/FdWbMj1WTqOIR2nf8og0= 2023-11-02T18:42:27.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QLY6zE/FdWbMj1WTqOIR2nf8og0=', 'Content-Length': '0'} 2023-11-02T18:42:27.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032bfd499553eb839-006543ed93-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.490 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:42:27.491 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:42:27.491 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.492 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.492 DEBUG:boto:Method: GET 2023-11-02T18:42:27.493 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.493 DEBUG:boto:Data: 2023-11-02T18:42:27.494 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.494 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.494 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.495 DEBUG:boto:Params: {} 2023-11-02T18:42:27.495 DEBUG:boto:Token: None 2023-11-02T18:42:27.496 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:42:27.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QLY6zE/FdWbMj1WTqOIR2nf8og0= 2023-11-02T18:42:27.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QLY6zE/FdWbMj1WTqOIR2nf8og0=', 'Content-Length': '0'} 2023-11-02T18:42:27.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfed1d229076cb4c-006543ed93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.501 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.501 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.502 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:42:27.502 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:42:27.503 DEBUG:boto:Method: GET 2023-11-02T18:42:27.503 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:42:27.503 DEBUG:boto:Data: 2023-11-02T18:42:27.504 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.504 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.505 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.505 DEBUG:boto:Params: {} 2023-11-02T18:42:27.506 DEBUG:boto:Token: None 2023-11-02T18:42:27.506 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/ 2023-11-02T18:42:27.507 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38= 2023-11-02T18:42:27.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38=', 'Content-Length': '0'} 2023-11-02T18:42:27.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004decdd86bbd4f14-006543ed93-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.513 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:27.514 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.515 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.516 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:42:27.517 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:42:27.517 DEBUG:boto:Method: GET 2023-11-02T18:42:27.518 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:42:27.518 DEBUG:boto:Data: 2023-11-02T18:42:27.519 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.520 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.520 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.521 DEBUG:boto:Params: {} 2023-11-02T18:42:27.521 DEBUG:boto:Token: None 2023-11-02T18:42:27.522 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/ 2023-11-02T18:42:27.522 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38= 2023-11-02T18:42:27.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38=', 'Content-Length': '0'} 2023-11-02T18:42:27.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f15f4162840217b1-006543ed93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.574 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:27.576 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:42:27.577 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:42:27.577 DEBUG:boto:Method: HEAD 2023-11-02T18:42:27.578 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:42:27.578 DEBUG:boto:Data: 2023-11-02T18:42:27.579 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.580 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.580 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.580 DEBUG:boto:Params: {} 2023-11-02T18:42:27.581 DEBUG:boto:Token: None 2023-11-02T18:42:27.581 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj11 2023-11-02T18:42:27.582 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7GbTLbvbz7KBtqAzx4hqN0lPyNw= 2023-11-02T18:42:27.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7GbTLbvbz7KBtqAzx4hqN0lPyNw=', 'Content-Length': '0'} 2023-11-02T18:42:27.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe495fdc8d72357-006543ed93-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.586 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:42:27.587 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:42:27.587 DEBUG:boto:Method: HEAD 2023-11-02T18:42:27.588 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:42:27.588 DEBUG:boto:Data: 2023-11-02T18:42:27.588 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.589 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.589 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.590 DEBUG:boto:Params: {} 2023-11-02T18:42:27.590 DEBUG:boto:Token: None 2023-11-02T18:42:27.591 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj11 2023-11-02T18:42:27.591 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7GbTLbvbz7KBtqAzx4hqN0lPyNw= 2023-11-02T18:42:27.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7GbTLbvbz7KBtqAzx4hqN0lPyNw=', 'Content-Length': '0'} 2023-11-02T18:42:27.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009336a47f03faec41-006543ed93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:42:27.596 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:42:27.596 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:42:27.597 DEBUG:boto:Method: GET 2023-11-02T18:42:27.597 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:42:27.597 DEBUG:boto:Data: 2023-11-02T18:42:27.598 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.598 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.599 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.599 DEBUG:boto:Params: {} 2023-11-02T18:42:27.600 DEBUG:boto:Token: None 2023-11-02T18:42:27.600 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj11 2023-11-02T18:42:27.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2B30f5icsQCX5Dl7vhT4YDOaTrY= 2023-11-02T18:42:27.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2B30f5icsQCX5Dl7vhT4YDOaTrY=', 'Content-Length': '0'} 2023-11-02T18:42:27.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d237441d0be2316-006543ed93-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.605 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:42:27.605 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:42:27.606 DEBUG:boto:Method: GET 2023-11-02T18:42:27.606 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:42:27.607 DEBUG:boto:Data: 2023-11-02T18:42:27.607 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.608 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.608 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.609 DEBUG:boto:Params: {} 2023-11-02T18:42:27.609 DEBUG:boto:Token: None 2023-11-02T18:42:27.610 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj11 2023-11-02T18:42:27.610 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2B30f5icsQCX5Dl7vhT4YDOaTrY= 2023-11-02T18:42:27.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2B30f5icsQCX5Dl7vhT4YDOaTrY=', 'Content-Length': '0'} 2023-11-02T18:42:27.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7d4884d6532df3f-006543ed93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.618 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.618 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.619 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:42:27.619 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:42:27.620 DEBUG:boto:Method: GET 2023-11-02T18:42:27.621 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:42:27.622 DEBUG:boto:Data: 2023-11-02T18:42:27.622 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.623 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.624 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.624 DEBUG:boto:Params: {} 2023-11-02T18:42:27.625 DEBUG:boto:Token: None 2023-11-02T18:42:27.626 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/ 2023-11-02T18:42:27.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38= 2023-11-02T18:42:27.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38=', 'Content-Length': '0'} 2023-11-02T18:42:27.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031dce8dd75055ef8-006543ed93-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.632 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:27.632 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.633 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.634 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:42:27.634 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:42:27.635 DEBUG:boto:Method: GET 2023-11-02T18:42:27.636 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:42:27.637 DEBUG:boto:Data: 2023-11-02T18:42:27.637 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.638 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.639 DEBUG:boto:Params: {} 2023-11-02T18:42:27.639 DEBUG:boto:Token: None 2023-11-02T18:42:27.640 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/ 2023-11-02T18:42:27.640 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38= 2023-11-02T18:42:27.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38=', 'Content-Length': '0'} 2023-11-02T18:42:27.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9acbebe118d2890-006543ed93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.690 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:27.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:42:27.692 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:42:27.692 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:42:27.693 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.693 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.694 DEBUG:boto:Method: GET 2023-11-02T18:42:27.694 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.695 DEBUG:boto:Data: 2023-11-02T18:42:27.695 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.696 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.696 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.697 DEBUG:boto:Params: {} 2023-11-02T18:42:27.697 DEBUG:boto:Token: None 2023-11-02T18:42:27.698 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.699 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kBbBZSntweoooPFSF/7DVEs7bWw= 2023-11-02T18:42:27.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kBbBZSntweoooPFSF/7DVEs7bWw=', 'Content-Length': '0'} 2023-11-02T18:42:27.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ed6409ecd01d54e-006543ed93-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.703 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:42:27.704 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:42:27.704 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.704 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.705 DEBUG:boto:Method: GET 2023-11-02T18:42:27.705 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.706 DEBUG:boto:Data: 2023-11-02T18:42:27.706 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.707 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.707 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.708 DEBUG:boto:Params: {} 2023-11-02T18:42:27.708 DEBUG:boto:Token: None 2023-11-02T18:42:27.708 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.709 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kBbBZSntweoooPFSF/7DVEs7bWw= 2023-11-02T18:42:27.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kBbBZSntweoooPFSF/7DVEs7bWw=', 'Content-Length': '0'} 2023-11-02T18:42:27.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de5d9415a9d784f0-006543ed93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.713 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:42:27.714 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:42:27.714 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.715 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.715 DEBUG:boto:Method: HEAD 2023-11-02T18:42:27.715 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.716 DEBUG:boto:Data: 2023-11-02T18:42:27.716 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.717 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.717 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.718 DEBUG:boto:Params: {} 2023-11-02T18:42:27.718 DEBUG:boto:Token: None 2023-11-02T18:42:27.719 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.719 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rhu1kmO/P3qpT8ISZfzWLqkYyZo= 2023-11-02T18:42:27.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rhu1kmO/P3qpT8ISZfzWLqkYyZo=', 'Content-Length': '0'} 2023-11-02T18:42:27.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acd2ccb7d3b413a6-006543ed93-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.723 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:42:27.723 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:42:27.724 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.724 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.724 DEBUG:boto:Method: HEAD 2023-11-02T18:42:27.725 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.725 DEBUG:boto:Data: 2023-11-02T18:42:27.726 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.727 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.727 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.728 DEBUG:boto:Params: {} 2023-11-02T18:42:27.728 DEBUG:boto:Token: None 2023-11-02T18:42:27.729 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.730 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rhu1kmO/P3qpT8ISZfzWLqkYyZo= 2023-11-02T18:42:27.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rhu1kmO/P3qpT8ISZfzWLqkYyZo=', 'Content-Length': '0'} 2023-11-02T18:42:27.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003bae725384f052b-006543ed93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:42:27.735 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:42:27.736 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:42:27.736 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.736 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.737 DEBUG:boto:Method: GET 2023-11-02T18:42:27.737 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.738 DEBUG:boto:Data: 2023-11-02T18:42:27.738 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.739 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.739 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.740 DEBUG:boto:Params: {} 2023-11-02T18:42:27.740 DEBUG:boto:Token: None 2023-11-02T18:42:27.741 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.741 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kBbBZSntweoooPFSF/7DVEs7bWw= 2023-11-02T18:42:27.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kBbBZSntweoooPFSF/7DVEs7bWw=', 'Content-Length': '0'} 2023-11-02T18:42:27.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039897c36fc4ecb36-006543ed93-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.786 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:42:27.787 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:42:27.787 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.788 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.788 DEBUG:boto:Method: GET 2023-11-02T18:42:27.789 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.789 DEBUG:boto:Data: 2023-11-02T18:42:27.790 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.791 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.791 DEBUG:boto:Params: {} 2023-11-02T18:42:27.791 DEBUG:boto:Token: None 2023-11-02T18:42:27.792 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:42:27.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kBbBZSntweoooPFSF/7DVEs7bWw= 2023-11-02T18:42:27.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kBbBZSntweoooPFSF/7DVEs7bWw=', 'Content-Length': '0'} 2023-11-02T18:42:27.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e89c3c5d3014a35-006543ed93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.797 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.797 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.798 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:42:27.798 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:42:27.798 DEBUG:boto:Method: GET 2023-11-02T18:42:27.799 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:42:27.799 DEBUG:boto:Data: 2023-11-02T18:42:27.800 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.800 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.801 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.801 DEBUG:boto:Params: {} 2023-11-02T18:42:27.802 DEBUG:boto:Token: None 2023-11-02T18:42:27.802 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/ 2023-11-02T18:42:27.803 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38= 2023-11-02T18:42:27.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38=', 'Content-Length': '0'} 2023-11-02T18:42:27.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000024abd8ce89b7e0-006543ed93-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.809 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:27.809 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.810 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.810 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:42:27.811 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:42:27.811 DEBUG:boto:Method: GET 2023-11-02T18:42:27.812 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:42:27.812 DEBUG:boto:Data: 2023-11-02T18:42:27.813 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.813 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.813 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.814 DEBUG:boto:Params: {} 2023-11-02T18:42:27.814 DEBUG:boto:Token: None 2023-11-02T18:42:27.815 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/ 2023-11-02T18:42:27.815 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38= 2023-11-02T18:42:27.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38=', 'Content-Length': '0'} 2023-11-02T18:42:27.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000044de55b1ae2235e-006543ed93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.825 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:27.826 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:42:27.826 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:42:27.827 DEBUG:boto:Method: HEAD 2023-11-02T18:42:27.827 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:42:27.828 DEBUG:boto:Data: 2023-11-02T18:42:27.828 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.828 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.829 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.829 DEBUG:boto:Params: {} 2023-11-02T18:42:27.830 DEBUG:boto:Token: None 2023-11-02T18:42:27.830 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj115 2023-11-02T18:42:27.831 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p3pw6D835lujKO/9freOnLmV5NY= 2023-11-02T18:42:27.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p3pw6D835lujKO/9freOnLmV5NY=', 'Content-Length': '0'} 2023-11-02T18:42:27.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1ddd51d2e30d923-006543ed93-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.835 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:42:27.836 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:42:27.836 DEBUG:boto:Method: HEAD 2023-11-02T18:42:27.837 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:42:27.837 DEBUG:boto:Data: 2023-11-02T18:42:27.838 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.838 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.838 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.839 DEBUG:boto:Params: {} 2023-11-02T18:42:27.839 DEBUG:boto:Token: None 2023-11-02T18:42:27.840 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj115 2023-11-02T18:42:27.840 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p3pw6D835lujKO/9freOnLmV5NY= 2023-11-02T18:42:27.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p3pw6D835lujKO/9freOnLmV5NY=', 'Content-Length': '0'} 2023-11-02T18:42:27.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f38eefab8bf9d609-006543ed93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:42:27.845 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:42:27.845 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:42:27.845 DEBUG:boto:Method: GET 2023-11-02T18:42:27.846 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:42:27.846 DEBUG:boto:Data: 2023-11-02T18:42:27.847 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.847 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.848 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.848 DEBUG:boto:Params: {} 2023-11-02T18:42:27.849 DEBUG:boto:Token: None 2023-11-02T18:42:27.849 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj115 2023-11-02T18:42:27.850 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lCP5JUf9F0sungjUFpuWYbsBDKs= 2023-11-02T18:42:27.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lCP5JUf9F0sungjUFpuWYbsBDKs=', 'Content-Length': '0'} 2023-11-02T18:42:27.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e22adcc54f605a11-006543ed93-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.854 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:42:27.854 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:42:27.855 DEBUG:boto:Method: GET 2023-11-02T18:42:27.855 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:42:27.856 DEBUG:boto:Data: 2023-11-02T18:42:27.856 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.857 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.858 DEBUG:boto:Params: {} 2023-11-02T18:42:27.858 DEBUG:boto:Token: None 2023-11-02T18:42:27.859 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj115 2023-11-02T18:42:27.859 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lCP5JUf9F0sungjUFpuWYbsBDKs= 2023-11-02T18:42:27.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lCP5JUf9F0sungjUFpuWYbsBDKs=', 'Content-Length': '0'} 2023-11-02T18:42:27.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012de1db49d6848ad-006543ed93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.906 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.907 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.907 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:42:27.908 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:42:27.908 DEBUG:boto:Method: GET 2023-11-02T18:42:27.909 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:42:27.909 DEBUG:boto:Data: 2023-11-02T18:42:27.910 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.911 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.911 DEBUG:boto:Params: {} 2023-11-02T18:42:27.912 DEBUG:boto:Token: None 2023-11-02T18:42:27.912 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/ 2023-11-02T18:42:27.913 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38= 2023-11-02T18:42:27.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38=', 'Content-Length': '0'} 2023-11-02T18:42:27.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012dde6ee81512171-006543ed93-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.919 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:27.920 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:27.920 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:27.921 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:42:27.921 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:42:27.922 DEBUG:boto:Method: GET 2023-11-02T18:42:27.922 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:42:27.923 DEBUG:boto:Data: 2023-11-02T18:42:27.923 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.923 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.925 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.925 DEBUG:boto:Params: {} 2023-11-02T18:42:27.926 DEBUG:boto:Token: None 2023-11-02T18:42:27.926 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/ 2023-11-02T18:42:27.927 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38= 2023-11-02T18:42:27.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWuJAhitOtJurc73CPHPFoDmK38=', 'Content-Length': '0'} 2023-11-02T18:42:27.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029566c7e2c80841e-006543ed93-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.936 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:27.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:42:27.937 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:42:27.938 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:42:27.938 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:42:27.939 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:42:27.939 DEBUG:boto:Method: GET 2023-11-02T18:42:27.939 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:42:27.940 DEBUG:boto:Data: 2023-11-02T18:42:27.940 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.941 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:27.941 DEBUG:boto:Port: 8001 2023-11-02T18:42:27.942 DEBUG:boto:Params: {} 2023-11-02T18:42:27.942 DEBUG:boto:Token: None 2023-11-02T18:42:27.943 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:42:27.943 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x7khz1587KX3v8OpY3i6bsY2MNM= 2023-11-02T18:42:27.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x7khz1587KX3v8OpY3i6bsY2MNM=', 'Content-Length': '0'} 2023-11-02T18:42:27.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c785d91baacd3198-006543ed93-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:27.990 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:42:27.991 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:42:27.991 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:42:27.992 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:42:27.992 DEBUG:boto:Method: GET 2023-11-02T18:42:27.993 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:42:27.993 DEBUG:boto:Data: 2023-11-02T18:42:27.994 DEBUG:boto:Headers: {} 2023-11-02T18:42:27.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:27.995 DEBUG:boto:Port: 8000 2023-11-02T18:42:27.996 DEBUG:boto:Params: {} 2023-11-02T18:42:27.996 DEBUG:boto:Token: None 2023-11-02T18:42:27.997 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:27 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:42:27.997 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x7khz1587KX3v8OpY3i6bsY2MNM= 2023-11-02T18:42:27.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x7khz1587KX3v8OpY3i6bsY2MNM=', 'Content-Length': '0'} 2023-11-02T18:42:28.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092cbb40c2c069195-006543ed93-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.002 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:42:28.002 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:42:28.003 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.003 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.004 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.004 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.004 DEBUG:boto:Data: 2023-11-02T18:42:28.005 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.005 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.006 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.006 DEBUG:boto:Params: {} 2023-11-02T18:42:28.007 DEBUG:boto:Token: None 2023-11-02T18:42:28.007 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.008 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:imujAwgou7NLfOGuf9GG+iwPrhk= 2023-11-02T18:42:28.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:imujAwgou7NLfOGuf9GG+iwPrhk=', 'Content-Length': '0'} 2023-11-02T18:42:28.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005170e8ccd1c58c64-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.012 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:42:28.013 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:42:28.013 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.014 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.014 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.015 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.015 DEBUG:boto:Data: 2023-11-02T18:42:28.016 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.016 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.017 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.017 DEBUG:boto:Params: {} 2023-11-02T18:42:28.018 DEBUG:boto:Token: None 2023-11-02T18:42:28.018 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.019 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:imujAwgou7NLfOGuf9GG+iwPrhk= 2023-11-02T18:42:28.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:imujAwgou7NLfOGuf9GG+iwPrhk=', 'Content-Length': '0'} 2023-11-02T18:42:28.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053e79e2ec6e7233f-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:42:28.023 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:42:28.024 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:42:28.024 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.025 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.025 DEBUG:boto:Method: GET 2023-11-02T18:42:28.026 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.026 DEBUG:boto:Data: 2023-11-02T18:42:28.027 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.027 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.028 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.028 DEBUG:boto:Params: {} 2023-11-02T18:42:28.029 DEBUG:boto:Token: None 2023-11-02T18:42:28.029 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.030 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B3VpWZvaIGMrPreG7U71Avjy/GM= 2023-11-02T18:42:28.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B3VpWZvaIGMrPreG7U71Avjy/GM=', 'Content-Length': '0'} 2023-11-02T18:42:28.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c11b5ce552755fc2-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.034 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:42:28.035 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:42:28.035 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.036 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.036 DEBUG:boto:Method: GET 2023-11-02T18:42:28.037 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.037 DEBUG:boto:Data: 2023-11-02T18:42:28.038 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.039 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.039 DEBUG:boto:Params: {} 2023-11-02T18:42:28.040 DEBUG:boto:Token: None 2023-11-02T18:42:28.040 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:42:28.041 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B3VpWZvaIGMrPreG7U71Avjy/GM= 2023-11-02T18:42:28.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B3VpWZvaIGMrPreG7U71Avjy/GM=', 'Content-Length': '0'} 2023-11-02T18:42:28.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a20974d8ad89437a-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.045 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.045 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.046 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:42:28.046 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:42:28.047 DEBUG:boto:Method: GET 2023-11-02T18:42:28.047 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:42:28.048 DEBUG:boto:Data: 2023-11-02T18:42:28.048 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.049 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.049 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.050 DEBUG:boto:Params: {} 2023-11-02T18:42:28.050 DEBUG:boto:Token: None 2023-11-02T18:42:28.051 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe72b225be54c81e-006543ed94-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.058 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.059 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.060 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.060 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:42:28.061 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:42:28.061 DEBUG:boto:Method: GET 2023-11-02T18:42:28.062 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:42:28.062 DEBUG:boto:Data: 2023-11-02T18:42:28.063 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.063 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.063 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.064 DEBUG:boto:Params: {} 2023-11-02T18:42:28.064 DEBUG:boto:Token: None 2023-11-02T18:42:28.065 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.065 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed4ef0f67e23904e-006543ed94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.074 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.075 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:42:28.075 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:42:28.076 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.076 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:42:28.077 DEBUG:boto:Data: 2023-11-02T18:42:28.077 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.078 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.078 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.079 DEBUG:boto:Params: {} 2023-11-02T18:42:28.079 DEBUG:boto:Token: None 2023-11-02T18:42:28.080 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj116 2023-11-02T18:42:28.080 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ai9bXflFvuvdfOWgHECBJ8TfpTs= 2023-11-02T18:42:28.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ai9bXflFvuvdfOWgHECBJ8TfpTs=', 'Content-Length': '0'} 2023-11-02T18:42:28.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e33ad83a4babe9fa-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.084 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:42:28.084 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:42:28.085 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.085 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:42:28.086 DEBUG:boto:Data: 2023-11-02T18:42:28.086 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.087 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.087 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.087 DEBUG:boto:Params: {} 2023-11-02T18:42:28.088 DEBUG:boto:Token: None 2023-11-02T18:42:28.088 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj116 2023-11-02T18:42:28.089 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ai9bXflFvuvdfOWgHECBJ8TfpTs= 2023-11-02T18:42:28.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ai9bXflFvuvdfOWgHECBJ8TfpTs=', 'Content-Length': '0'} 2023-11-02T18:42:28.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aff47a477740b0d7-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:42:28.092 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:42:28.093 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:42:28.093 DEBUG:boto:Method: GET 2023-11-02T18:42:28.094 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:42:28.094 DEBUG:boto:Data: 2023-11-02T18:42:28.095 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.095 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.096 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.096 DEBUG:boto:Params: {} 2023-11-02T18:42:28.097 DEBUG:boto:Token: None 2023-11-02T18:42:28.097 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj116 2023-11-02T18:42:28.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gh1xGqMyqAokgFxiuCSp/k50V80= 2023-11-02T18:42:28.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gh1xGqMyqAokgFxiuCSp/k50V80=', 'Content-Length': '0'} 2023-11-02T18:42:28.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba751eafd491bb52-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.142 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:42:28.143 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:42:28.143 DEBUG:boto:Method: GET 2023-11-02T18:42:28.144 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:42:28.144 DEBUG:boto:Data: 2023-11-02T18:42:28.144 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.145 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.145 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.146 DEBUG:boto:Params: {} 2023-11-02T18:42:28.146 DEBUG:boto:Token: None 2023-11-02T18:42:28.147 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj116 2023-11-02T18:42:28.147 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gh1xGqMyqAokgFxiuCSp/k50V80= 2023-11-02T18:42:28.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gh1xGqMyqAokgFxiuCSp/k50V80=', 'Content-Length': '0'} 2023-11-02T18:42:28.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df47643e92f2acba-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.152 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.152 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.153 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:42:28.153 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:42:28.154 DEBUG:boto:Method: GET 2023-11-02T18:42:28.154 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:42:28.155 DEBUG:boto:Data: 2023-11-02T18:42:28.155 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.156 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.156 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.156 DEBUG:boto:Params: {} 2023-11-02T18:42:28.157 DEBUG:boto:Token: None 2023-11-02T18:42:28.157 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.158 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bd4e190baf82e2d-006543ed94-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.164 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.165 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.165 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.166 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:42:28.166 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:42:28.167 DEBUG:boto:Method: GET 2023-11-02T18:42:28.167 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:42:28.168 DEBUG:boto:Data: 2023-11-02T18:42:28.168 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.169 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.169 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.169 DEBUG:boto:Params: {} 2023-11-02T18:42:28.170 DEBUG:boto:Token: None 2023-11-02T18:42:28.170 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.171 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af3e696597fc28c8-006543ed94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.183 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:42:28.184 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:42:28.184 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:42:28.185 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.185 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.186 DEBUG:boto:Method: GET 2023-11-02T18:42:28.186 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.187 DEBUG:boto:Data: 2023-11-02T18:42:28.187 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.188 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.188 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.188 DEBUG:boto:Params: {} 2023-11-02T18:42:28.189 DEBUG:boto:Token: None 2023-11-02T18:42:28.189 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.190 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gm2Py6hxk4c7oM64WOFhHKXbi3M= 2023-11-02T18:42:28.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gm2Py6hxk4c7oM64WOFhHKXbi3M=', 'Content-Length': '0'} 2023-11-02T18:42:28.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e6439761261a72c-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.194 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:42:28.194 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:42:28.195 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.195 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.196 DEBUG:boto:Method: GET 2023-11-02T18:42:28.196 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.197 DEBUG:boto:Data: 2023-11-02T18:42:28.197 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.197 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.198 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.198 DEBUG:boto:Params: {} 2023-11-02T18:42:28.199 DEBUG:boto:Token: None 2023-11-02T18:42:28.199 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.200 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gm2Py6hxk4c7oM64WOFhHKXbi3M= 2023-11-02T18:42:28.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gm2Py6hxk4c7oM64WOFhHKXbi3M=', 'Content-Length': '0'} 2023-11-02T18:42:28.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee963bbad801bbeb-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.203 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:42:28.204 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:42:28.205 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.205 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.205 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.206 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.206 DEBUG:boto:Data: 2023-11-02T18:42:28.207 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.207 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.207 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.208 DEBUG:boto:Params: {} 2023-11-02T18:42:28.208 DEBUG:boto:Token: None 2023-11-02T18:42:28.209 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.209 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wSoD6Ou0jF4QgX8P7r8lSphvmGA= 2023-11-02T18:42:28.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wSoD6Ou0jF4QgX8P7r8lSphvmGA=', 'Content-Length': '0'} 2023-11-02T18:42:28.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1d8c164b3a107c0-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.213 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:42:28.214 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:42:28.214 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.214 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.215 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.215 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.216 DEBUG:boto:Data: 2023-11-02T18:42:28.216 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.216 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.217 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.217 DEBUG:boto:Params: {} 2023-11-02T18:42:28.218 DEBUG:boto:Token: None 2023-11-02T18:42:28.218 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.219 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wSoD6Ou0jF4QgX8P7r8lSphvmGA= 2023-11-02T18:42:28.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wSoD6Ou0jF4QgX8P7r8lSphvmGA=', 'Content-Length': '0'} 2023-11-02T18:42:28.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4178579f81cb8c5-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:42:28.223 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:42:28.224 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:42:28.224 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.224 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.225 DEBUG:boto:Method: GET 2023-11-02T18:42:28.225 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.226 DEBUG:boto:Data: 2023-11-02T18:42:28.226 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.227 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.227 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.227 DEBUG:boto:Params: {} 2023-11-02T18:42:28.228 DEBUG:boto:Token: None 2023-11-02T18:42:28.228 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.229 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gm2Py6hxk4c7oM64WOFhHKXbi3M= 2023-11-02T18:42:28.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gm2Py6hxk4c7oM64WOFhHKXbi3M=', 'Content-Length': '0'} 2023-11-02T18:42:28.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f887d9255b79c06-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.233 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:42:28.234 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:42:28.234 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.234 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.235 DEBUG:boto:Method: GET 2023-11-02T18:42:28.235 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.236 DEBUG:boto:Data: 2023-11-02T18:42:28.236 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.237 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.237 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.237 DEBUG:boto:Params: {} 2023-11-02T18:42:28.238 DEBUG:boto:Token: None 2023-11-02T18:42:28.238 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:42:28.239 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gm2Py6hxk4c7oM64WOFhHKXbi3M= 2023-11-02T18:42:28.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gm2Py6hxk4c7oM64WOFhHKXbi3M=', 'Content-Length': '0'} 2023-11-02T18:42:28.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aec3847c9169115-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.286 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.287 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.287 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:42:28.288 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:42:28.288 DEBUG:boto:Method: GET 2023-11-02T18:42:28.289 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:42:28.289 DEBUG:boto:Data: 2023-11-02T18:42:28.290 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.290 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.290 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.291 DEBUG:boto:Params: {} 2023-11-02T18:42:28.291 DEBUG:boto:Token: None 2023-11-02T18:42:28.292 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.292 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7f5b16be6d35db2-006543ed94-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.298 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.299 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.300 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.300 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:42:28.301 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:42:28.301 DEBUG:boto:Method: GET 2023-11-02T18:42:28.301 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:42:28.302 DEBUG:boto:Data: 2023-11-02T18:42:28.302 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.303 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.304 DEBUG:boto:Params: {} 2023-11-02T18:42:28.304 DEBUG:boto:Token: None 2023-11-02T18:42:28.305 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027677462d476b7af-006543ed94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.315 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.316 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:42:28.316 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:42:28.317 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.317 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:42:28.317 DEBUG:boto:Data: 2023-11-02T18:42:28.318 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.318 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.319 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.319 DEBUG:boto:Params: {} 2023-11-02T18:42:28.320 DEBUG:boto:Token: None 2023-11-02T18:42:28.320 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj145 2023-11-02T18:42:28.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:unrNLOvQlcqF+iJffztBN9yUZq8= 2023-11-02T18:42:28.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:unrNLOvQlcqF+iJffztBN9yUZq8=', 'Content-Length': '0'} 2023-11-02T18:42:28.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006496f35bd22bda5e-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.324 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:42:28.325 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:42:28.325 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.325 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:42:28.326 DEBUG:boto:Data: 2023-11-02T18:42:28.326 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.327 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.327 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.328 DEBUG:boto:Params: {} 2023-11-02T18:42:28.329 DEBUG:boto:Token: None 2023-11-02T18:42:28.329 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj145 2023-11-02T18:42:28.329 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:unrNLOvQlcqF+iJffztBN9yUZq8= 2023-11-02T18:42:28.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:unrNLOvQlcqF+iJffztBN9yUZq8=', 'Content-Length': '0'} 2023-11-02T18:42:28.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7707f41b7a6aa7e-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:42:28.335 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:42:28.335 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:42:28.336 DEBUG:boto:Method: GET 2023-11-02T18:42:28.336 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:42:28.337 DEBUG:boto:Data: 2023-11-02T18:42:28.337 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.338 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.338 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.338 DEBUG:boto:Params: {} 2023-11-02T18:42:28.339 DEBUG:boto:Token: None 2023-11-02T18:42:28.340 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj145 2023-11-02T18:42:28.340 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+JvCjI6gsGdN7H1usdGF40vnOgg= 2023-11-02T18:42:28.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+JvCjI6gsGdN7H1usdGF40vnOgg=', 'Content-Length': '0'} 2023-11-02T18:42:28.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ac50af191902202-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.386 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:42:28.387 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:42:28.387 DEBUG:boto:Method: GET 2023-11-02T18:42:28.388 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:42:28.388 DEBUG:boto:Data: 2023-11-02T18:42:28.389 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.389 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.390 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.390 DEBUG:boto:Params: {} 2023-11-02T18:42:28.391 DEBUG:boto:Token: None 2023-11-02T18:42:28.391 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj145 2023-11-02T18:42:28.392 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+JvCjI6gsGdN7H1usdGF40vnOgg= 2023-11-02T18:42:28.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+JvCjI6gsGdN7H1usdGF40vnOgg=', 'Content-Length': '0'} 2023-11-02T18:42:28.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059de0ceb1acda15a-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.398 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.398 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.399 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:42:28.399 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:42:28.400 DEBUG:boto:Method: GET 2023-11-02T18:42:28.400 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:42:28.401 DEBUG:boto:Data: 2023-11-02T18:42:28.401 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.401 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.402 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.402 DEBUG:boto:Params: {} 2023-11-02T18:42:28.403 DEBUG:boto:Token: None 2023-11-02T18:42:28.403 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.404 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b61e51c6289efee0-006543ed94-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.410 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.411 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.412 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.412 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:42:28.413 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:42:28.413 DEBUG:boto:Method: GET 2023-11-02T18:42:28.414 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:42:28.414 DEBUG:boto:Data: 2023-11-02T18:42:28.415 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.415 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.415 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.416 DEBUG:boto:Params: {} 2023-11-02T18:42:28.416 DEBUG:boto:Token: None 2023-11-02T18:42:28.417 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.417 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecbd1abcb35bf4f4-006543ed94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.470 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:42:28.472 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:42:28.472 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:42:28.473 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.473 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.473 DEBUG:boto:Method: GET 2023-11-02T18:42:28.474 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.474 DEBUG:boto:Data: 2023-11-02T18:42:28.475 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.475 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.476 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.476 DEBUG:boto:Params: {} 2023-11-02T18:42:28.477 DEBUG:boto:Token: None 2023-11-02T18:42:28.477 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.478 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZPThfOjsW3WHnm7MckdR1iUgRqk= 2023-11-02T18:42:28.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZPThfOjsW3WHnm7MckdR1iUgRqk=', 'Content-Length': '0'} 2023-11-02T18:42:28.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c53d42a55cdd4f6-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.482 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:42:28.483 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:42:28.484 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.484 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.484 DEBUG:boto:Method: GET 2023-11-02T18:42:28.485 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.485 DEBUG:boto:Data: 2023-11-02T18:42:28.486 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.486 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.487 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.487 DEBUG:boto:Params: {} 2023-11-02T18:42:28.487 DEBUG:boto:Token: None 2023-11-02T18:42:28.488 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.489 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZPThfOjsW3WHnm7MckdR1iUgRqk= 2023-11-02T18:42:28.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZPThfOjsW3WHnm7MckdR1iUgRqk=', 'Content-Length': '0'} 2023-11-02T18:42:28.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0691adca7234af7-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.493 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:42:28.493 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:42:28.494 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.494 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.495 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.495 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.497 DEBUG:boto:Data: 2023-11-02T18:42:28.497 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.497 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.498 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.498 DEBUG:boto:Params: {} 2023-11-02T18:42:28.499 DEBUG:boto:Token: None 2023-11-02T18:42:28.499 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.500 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t0zRCHYO7vHOW0/4XuZ6zHpDVrk= 2023-11-02T18:42:28.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t0zRCHYO7vHOW0/4XuZ6zHpDVrk=', 'Content-Length': '0'} 2023-11-02T18:42:28.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcdd6ffea8554fc0-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.504 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:42:28.506 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:42:28.506 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.506 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.507 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.507 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.508 DEBUG:boto:Data: 2023-11-02T18:42:28.508 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.509 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.509 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.510 DEBUG:boto:Params: {} 2023-11-02T18:42:28.510 DEBUG:boto:Token: None 2023-11-02T18:42:28.511 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.511 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t0zRCHYO7vHOW0/4XuZ6zHpDVrk= 2023-11-02T18:42:28.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t0zRCHYO7vHOW0/4XuZ6zHpDVrk=', 'Content-Length': '0'} 2023-11-02T18:42:28.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007def6b196b04cb4a-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:42:28.516 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:42:28.516 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:42:28.516 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.517 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.517 DEBUG:boto:Method: GET 2023-11-02T18:42:28.518 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.518 DEBUG:boto:Data: 2023-11-02T18:42:28.519 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.520 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.520 DEBUG:boto:Params: {} 2023-11-02T18:42:28.521 DEBUG:boto:Token: None 2023-11-02T18:42:28.522 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.522 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZPThfOjsW3WHnm7MckdR1iUgRqk= 2023-11-02T18:42:28.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZPThfOjsW3WHnm7MckdR1iUgRqk=', 'Content-Length': '0'} 2023-11-02T18:42:28.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa4d17d02d7b0dff-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.526 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:42:28.527 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:42:28.527 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.528 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.529 DEBUG:boto:Method: GET 2023-11-02T18:42:28.529 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.530 DEBUG:boto:Data: 2023-11-02T18:42:28.530 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.531 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.531 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.531 DEBUG:boto:Params: {} 2023-11-02T18:42:28.532 DEBUG:boto:Token: None 2023-11-02T18:42:28.532 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:42:28.533 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZPThfOjsW3WHnm7MckdR1iUgRqk= 2023-11-02T18:42:28.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZPThfOjsW3WHnm7MckdR1iUgRqk=', 'Content-Length': '0'} 2023-11-02T18:42:28.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001e0cd3a14004457-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.578 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.579 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.579 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:42:28.580 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:42:28.580 DEBUG:boto:Method: GET 2023-11-02T18:42:28.581 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:42:28.581 DEBUG:boto:Data: 2023-11-02T18:42:28.582 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.582 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.583 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.583 DEBUG:boto:Params: {} 2023-11-02T18:42:28.584 DEBUG:boto:Token: None 2023-11-02T18:42:28.584 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.585 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068863ac866e34aea-006543ed94-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.594 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.595 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.595 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.596 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:42:28.596 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:42:28.597 DEBUG:boto:Method: GET 2023-11-02T18:42:28.597 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:42:28.598 DEBUG:boto:Data: 2023-11-02T18:42:28.598 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.599 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.599 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.599 DEBUG:boto:Params: {} 2023-11-02T18:42:28.600 DEBUG:boto:Token: None 2023-11-02T18:42:28.600 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd26d2a88b08f402-006543ed94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.611 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.611 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:42:28.612 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:42:28.612 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.612 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:42:28.613 DEBUG:boto:Data: 2023-11-02T18:42:28.613 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.614 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.614 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.615 DEBUG:boto:Params: {} 2023-11-02T18:42:28.616 DEBUG:boto:Token: None 2023-11-02T18:42:28.616 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj17 2023-11-02T18:42:28.617 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TwZHN36Nc8zR9pUebd403wK7AK0= 2023-11-02T18:42:28.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TwZHN36Nc8zR9pUebd403wK7AK0=', 'Content-Length': '0'} 2023-11-02T18:42:28.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbea872fd4fb71d1-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.621 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:42:28.622 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:42:28.622 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.623 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:42:28.623 DEBUG:boto:Data: 2023-11-02T18:42:28.623 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.624 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.624 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.625 DEBUG:boto:Params: {} 2023-11-02T18:42:28.625 DEBUG:boto:Token: None 2023-11-02T18:42:28.626 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj17 2023-11-02T18:42:28.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TwZHN36Nc8zR9pUebd403wK7AK0= 2023-11-02T18:42:28.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TwZHN36Nc8zR9pUebd403wK7AK0=', 'Content-Length': '0'} 2023-11-02T18:42:28.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b40dc66f188b7f7-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:42:28.631 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:42:28.631 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:42:28.632 DEBUG:boto:Method: GET 2023-11-02T18:42:28.633 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:42:28.633 DEBUG:boto:Data: 2023-11-02T18:42:28.634 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.634 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.635 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.636 DEBUG:boto:Params: {} 2023-11-02T18:42:28.636 DEBUG:boto:Token: None 2023-11-02T18:42:28.637 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj17 2023-11-02T18:42:28.637 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtfeF+7eJpCXldi0buUWv2l8DM4= 2023-11-02T18:42:28.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtfeF+7eJpCXldi0buUWv2l8DM4=', 'Content-Length': '0'} 2023-11-02T18:42:28.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e702c855335526a9-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.682 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:42:28.683 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:42:28.683 DEBUG:boto:Method: GET 2023-11-02T18:42:28.684 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:42:28.684 DEBUG:boto:Data: 2023-11-02T18:42:28.685 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.685 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.686 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.686 DEBUG:boto:Params: {} 2023-11-02T18:42:28.687 DEBUG:boto:Token: None 2023-11-02T18:42:28.687 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj17 2023-11-02T18:42:28.688 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtfeF+7eJpCXldi0buUWv2l8DM4= 2023-11-02T18:42:28.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtfeF+7eJpCXldi0buUWv2l8DM4=', 'Content-Length': '0'} 2023-11-02T18:42:28.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000895a5dcbf220d966-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.692 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.693 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.693 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:42:28.694 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:42:28.694 DEBUG:boto:Method: GET 2023-11-02T18:42:28.695 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:42:28.695 DEBUG:boto:Data: 2023-11-02T18:42:28.696 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.696 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.696 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.697 DEBUG:boto:Params: {} 2023-11-02T18:42:28.697 DEBUG:boto:Token: None 2023-11-02T18:42:28.698 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.699 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e39e408146c463e6-006543ed94-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.704 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.705 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.705 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.706 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:42:28.706 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:42:28.706 DEBUG:boto:Method: GET 2023-11-02T18:42:28.707 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:42:28.707 DEBUG:boto:Data: 2023-11-02T18:42:28.708 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.708 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.709 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.709 DEBUG:boto:Params: {} 2023-11-02T18:42:28.709 DEBUG:boto:Token: None 2023-11-02T18:42:28.710 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.710 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d05bb0ad2611ec9-006543ed94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.762 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:42:28.764 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:42:28.764 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:42:28.765 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.765 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.765 DEBUG:boto:Method: GET 2023-11-02T18:42:28.766 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.766 DEBUG:boto:Data: 2023-11-02T18:42:28.767 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.767 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.768 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.768 DEBUG:boto:Params: {} 2023-11-02T18:42:28.769 DEBUG:boto:Token: None 2023-11-02T18:42:28.770 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.770 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ochYEuvHrZ9UCXI11C7TGDBvNJg= 2023-11-02T18:42:28.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ochYEuvHrZ9UCXI11C7TGDBvNJg=', 'Content-Length': '0'} 2023-11-02T18:42:28.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab5b60ac4ae4b2ac-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.775 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:42:28.775 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:42:28.776 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.776 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.776 DEBUG:boto:Method: GET 2023-11-02T18:42:28.777 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.777 DEBUG:boto:Data: 2023-11-02T18:42:28.778 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.778 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.779 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.779 DEBUG:boto:Params: {} 2023-11-02T18:42:28.780 DEBUG:boto:Token: None 2023-11-02T18:42:28.780 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.781 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ochYEuvHrZ9UCXI11C7TGDBvNJg= 2023-11-02T18:42:28.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ochYEuvHrZ9UCXI11C7TGDBvNJg=', 'Content-Length': '0'} 2023-11-02T18:42:28.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8658b8423c2a0ab-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.785 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:42:28.785 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:42:28.786 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.786 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.787 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.787 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.788 DEBUG:boto:Data: 2023-11-02T18:42:28.788 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.788 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.789 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.789 DEBUG:boto:Params: {} 2023-11-02T18:42:28.790 DEBUG:boto:Token: None 2023-11-02T18:42:28.790 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.791 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OqpmFIRY9TnlfwENo/4C55AAOgM= 2023-11-02T18:42:28.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OqpmFIRY9TnlfwENo/4C55AAOgM=', 'Content-Length': '0'} 2023-11-02T18:42:28.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a141687995611237-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.795 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:42:28.795 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:42:28.796 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.797 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.798 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.798 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.799 DEBUG:boto:Data: 2023-11-02T18:42:28.799 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.800 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.800 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.800 DEBUG:boto:Params: {} 2023-11-02T18:42:28.801 DEBUG:boto:Token: None 2023-11-02T18:42:28.802 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.802 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OqpmFIRY9TnlfwENo/4C55AAOgM= 2023-11-02T18:42:28.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OqpmFIRY9TnlfwENo/4C55AAOgM=', 'Content-Length': '0'} 2023-11-02T18:42:28.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001210d83f57875509-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:42:28.807 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:42:28.807 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:42:28.808 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.808 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.808 DEBUG:boto:Method: GET 2023-11-02T18:42:28.809 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.809 DEBUG:boto:Data: 2023-11-02T18:42:28.810 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.810 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.811 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.811 DEBUG:boto:Params: {} 2023-11-02T18:42:28.812 DEBUG:boto:Token: None 2023-11-02T18:42:28.812 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.813 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ochYEuvHrZ9UCXI11C7TGDBvNJg= 2023-11-02T18:42:28.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ochYEuvHrZ9UCXI11C7TGDBvNJg=', 'Content-Length': '0'} 2023-11-02T18:42:28.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db46a43b08791b76-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.817 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:42:28.817 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:42:28.818 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.818 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.819 DEBUG:boto:Method: GET 2023-11-02T18:42:28.819 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.820 DEBUG:boto:Data: 2023-11-02T18:42:28.820 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.821 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.821 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.822 DEBUG:boto:Params: {} 2023-11-02T18:42:28.822 DEBUG:boto:Token: None 2023-11-02T18:42:28.823 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:42:28.823 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ochYEuvHrZ9UCXI11C7TGDBvNJg= 2023-11-02T18:42:28.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ochYEuvHrZ9UCXI11C7TGDBvNJg=', 'Content-Length': '0'} 2023-11-02T18:42:28.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001954234ee063fd77-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.827 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.828 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.829 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:42:28.829 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:42:28.831 DEBUG:boto:Method: GET 2023-11-02T18:42:28.831 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:42:28.832 DEBUG:boto:Data: 2023-11-02T18:42:28.832 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.834 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.834 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.835 DEBUG:boto:Params: {} 2023-11-02T18:42:28.835 DEBUG:boto:Token: None 2023-11-02T18:42:28.836 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.836 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea6fe45886350db0-006543ed94-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.843 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.843 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.844 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.844 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:42:28.845 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:42:28.845 DEBUG:boto:Method: GET 2023-11-02T18:42:28.846 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:42:28.846 DEBUG:boto:Data: 2023-11-02T18:42:28.846 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.847 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.847 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.848 DEBUG:boto:Params: {} 2023-11-02T18:42:28.848 DEBUG:boto:Token: None 2023-11-02T18:42:28.849 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.849 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005173c45048ed4aea-006543ed94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.859 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.860 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:42:28.860 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:42:28.861 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.861 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:42:28.862 DEBUG:boto:Data: 2023-11-02T18:42:28.862 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.863 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.863 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.864 DEBUG:boto:Params: {} 2023-11-02T18:42:28.864 DEBUG:boto:Token: None 2023-11-02T18:42:28.865 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj19 2023-11-02T18:42:28.865 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DHrnpvPBq7mTELVnAXh5SpKvl3w= 2023-11-02T18:42:28.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DHrnpvPBq7mTELVnAXh5SpKvl3w=', 'Content-Length': '0'} 2023-11-02T18:42:28.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c790a482b4177e8-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.868 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:42:28.869 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:42:28.869 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.869 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:42:28.870 DEBUG:boto:Data: 2023-11-02T18:42:28.870 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.871 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.871 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.872 DEBUG:boto:Params: {} 2023-11-02T18:42:28.872 DEBUG:boto:Token: None 2023-11-02T18:42:28.872 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj19 2023-11-02T18:42:28.873 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DHrnpvPBq7mTELVnAXh5SpKvl3w= 2023-11-02T18:42:28.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DHrnpvPBq7mTELVnAXh5SpKvl3w=', 'Content-Length': '0'} 2023-11-02T18:42:28.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b00a0eb20886a84-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:42:28.877 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:42:28.877 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:42:28.878 DEBUG:boto:Method: GET 2023-11-02T18:42:28.878 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:42:28.879 DEBUG:boto:Data: 2023-11-02T18:42:28.879 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.880 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.880 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.881 DEBUG:boto:Params: {} 2023-11-02T18:42:28.881 DEBUG:boto:Token: None 2023-11-02T18:42:28.882 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj19 2023-11-02T18:42:28.882 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oIswcvblKurpcsHHLsKNaVQtXDc= 2023-11-02T18:42:28.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oIswcvblKurpcsHHLsKNaVQtXDc=', 'Content-Length': '0'} 2023-11-02T18:42:28.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001879fcbe7f98e552-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.926 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:42:28.927 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:42:28.927 DEBUG:boto:Method: GET 2023-11-02T18:42:28.928 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:42:28.928 DEBUG:boto:Data: 2023-11-02T18:42:28.929 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.929 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.930 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.930 DEBUG:boto:Params: {} 2023-11-02T18:42:28.931 DEBUG:boto:Token: None 2023-11-02T18:42:28.931 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj19 2023-11-02T18:42:28.932 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oIswcvblKurpcsHHLsKNaVQtXDc= 2023-11-02T18:42:28.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oIswcvblKurpcsHHLsKNaVQtXDc=', 'Content-Length': '0'} 2023-11-02T18:42:28.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf26005033281a07-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.936 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.937 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.937 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:42:28.938 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:42:28.938 DEBUG:boto:Method: GET 2023-11-02T18:42:28.939 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:42:28.939 DEBUG:boto:Data: 2023-11-02T18:42:28.939 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.940 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.940 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.941 DEBUG:boto:Params: {} 2023-11-02T18:42:28.941 DEBUG:boto:Token: None 2023-11-02T18:42:28.942 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.943 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000805a8d1bf0642ba5-006543ed94-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.949 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.950 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:28.950 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:28.951 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:42:28.951 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:42:28.951 DEBUG:boto:Method: GET 2023-11-02T18:42:28.952 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:42:28.952 DEBUG:boto:Data: 2023-11-02T18:42:28.953 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.953 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.954 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.954 DEBUG:boto:Params: {} 2023-11-02T18:42:28.955 DEBUG:boto:Token: None 2023-11-02T18:42:28.955 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/ 2023-11-02T18:42:28.956 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw= 2023-11-02T18:42:28.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s5tLoZGNFMtrK1drj/e3/FxXRhw=', 'Content-Length': '0'} 2023-11-02T18:42:28.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c005d7beff1a2e04-006543ed94-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.965 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:28.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:42:28.967 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:42:28.967 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:42:28.967 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.968 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.968 DEBUG:boto:Method: GET 2023-11-02T18:42:28.969 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.969 DEBUG:boto:Data: 2023-11-02T18:42:28.970 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.970 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.970 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.971 DEBUG:boto:Params: {} 2023-11-02T18:42:28.971 DEBUG:boto:Token: None 2023-11-02T18:42:28.972 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.972 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7kJbjAFOehoTFYeUtlyHXbk0lT8= 2023-11-02T18:42:28.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7kJbjAFOehoTFYeUtlyHXbk0lT8=', 'Content-Length': '0'} 2023-11-02T18:42:28.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fb7ef640aad6bcd-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.977 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:42:28.978 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:42:28.978 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.978 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.979 DEBUG:boto:Method: GET 2023-11-02T18:42:28.979 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.980 DEBUG:boto:Data: 2023-11-02T18:42:28.980 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.981 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:28.981 DEBUG:boto:Port: 8000 2023-11-02T18:42:28.981 DEBUG:boto:Params: {} 2023-11-02T18:42:28.982 DEBUG:boto:Token: None 2023-11-02T18:42:28.982 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.983 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7kJbjAFOehoTFYeUtlyHXbk0lT8= 2023-11-02T18:42:28.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7kJbjAFOehoTFYeUtlyHXbk0lT8=', 'Content-Length': '0'} 2023-11-02T18:42:28.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eea3abf5bc521c61-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.987 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:42:28.987 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:42:28.988 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.988 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.989 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.989 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.990 DEBUG:boto:Data: 2023-11-02T18:42:28.990 DEBUG:boto:Headers: {} 2023-11-02T18:42:28.990 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:28.991 DEBUG:boto:Port: 8001 2023-11-02T18:42:28.991 DEBUG:boto:Params: {} 2023-11-02T18:42:28.992 DEBUG:boto:Token: None 2023-11-02T18:42:28.992 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:28 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9SHSVKiQAB7tZP4Vo1XvtupubiY= 2023-11-02T18:42:28.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9SHSVKiQAB7tZP4Vo1XvtupubiY=', 'Content-Length': '0'} 2023-11-02T18:42:28.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000617063abad7f3e43-006543ed94-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:28.997 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:42:28.997 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:42:28.997 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.998 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.998 DEBUG:boto:Method: HEAD 2023-11-02T18:42:28.999 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:42:28.999 DEBUG:boto:Data: 2023-11-02T18:42:29.000 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.000 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.000 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.001 DEBUG:boto:Params: {} 2023-11-02T18:42:29.001 DEBUG:boto:Token: None 2023-11-02T18:42:29.002 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:42:29.002 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pFnhAc//x0BGG5e+MWR6/Swgeho= 2023-11-02T18:42:29.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pFnhAc//x0BGG5e+MWR6/Swgeho=', 'Content-Length': '0'} 2023-11-02T18:42:29.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bafabc9b5b00e46e-006543ed94-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:42:29.007 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:42:29.007 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:42:29.008 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:42:29.008 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:42:29.008 DEBUG:boto:Method: GET 2023-11-02T18:42:29.009 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:42:29.009 DEBUG:boto:Data: 2023-11-02T18:42:29.010 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.010 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.011 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.011 DEBUG:boto:Params: {} 2023-11-02T18:42:29.012 DEBUG:boto:Token: None 2023-11-02T18:42:29.012 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:42:29.013 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y5mZl89AUyi26AYx64WRR2sshsM= 2023-11-02T18:42:29.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y5mZl89AUyi26AYx64WRR2sshsM=', 'Content-Length': '0'} 2023-11-02T18:42:29.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dd2ebabed927a24-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.058 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:42:29.059 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:42:29.059 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:42:29.061 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:42:29.062 DEBUG:boto:Method: GET 2023-11-02T18:42:29.062 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:42:29.063 DEBUG:boto:Data: 2023-11-02T18:42:29.064 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.064 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.064 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.065 DEBUG:boto:Params: {} 2023-11-02T18:42:29.066 DEBUG:boto:Token: None 2023-11-02T18:42:29.066 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:42:29.067 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y5mZl89AUyi26AYx64WRR2sshsM= 2023-11-02T18:42:29.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y5mZl89AUyi26AYx64WRR2sshsM=', 'Content-Length': '0'} 2023-11-02T18:42:29.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003e2f91ec516e3f3-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.071 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.072 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.072 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:42:29.073 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:42:29.073 DEBUG:boto:Method: GET 2023-11-02T18:42:29.074 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:42:29.074 DEBUG:boto:Data: 2023-11-02T18:42:29.075 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.075 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.076 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.076 DEBUG:boto:Params: {} 2023-11-02T18:42:29.077 DEBUG:boto:Token: None 2023-11-02T18:42:29.077 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.078 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000873fd1c0c0ca796a-006543ed95-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.084 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.085 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.085 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.086 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:42:29.086 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:42:29.087 DEBUG:boto:Method: GET 2023-11-02T18:42:29.087 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:42:29.087 DEBUG:boto:Data: 2023-11-02T18:42:29.088 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.088 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.089 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.089 DEBUG:boto:Params: {} 2023-11-02T18:42:29.090 DEBUG:boto:Token: None 2023-11-02T18:42:29.090 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.091 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b371f81cb0eb12fd-006543ed95-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.100 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.100 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:42:29.101 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:42:29.101 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.102 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:42:29.102 DEBUG:boto:Data: 2023-11-02T18:42:29.103 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.103 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.103 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.104 DEBUG:boto:Params: {} 2023-11-02T18:42:29.104 DEBUG:boto:Token: None 2023-11-02T18:42:29.105 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj22 2023-11-02T18:42:29.105 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:luR3FM8L4A+Qt5kITZfLB1LJdA8= 2023-11-02T18:42:29.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:luR3FM8L4A+Qt5kITZfLB1LJdA8=', 'Content-Length': '0'} 2023-11-02T18:42:29.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9f1ccb71baa3187-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.109 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:42:29.109 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:42:29.110 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.110 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:42:29.111 DEBUG:boto:Data: 2023-11-02T18:42:29.111 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.112 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.112 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.113 DEBUG:boto:Params: {} 2023-11-02T18:42:29.113 DEBUG:boto:Token: None 2023-11-02T18:42:29.114 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj22 2023-11-02T18:42:29.114 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:luR3FM8L4A+Qt5kITZfLB1LJdA8= 2023-11-02T18:42:29.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:luR3FM8L4A+Qt5kITZfLB1LJdA8=', 'Content-Length': '0'} 2023-11-02T18:42:29.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069182e3c40a7c562-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:42:29.119 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:42:29.119 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:42:29.120 DEBUG:boto:Method: GET 2023-11-02T18:42:29.120 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:42:29.121 DEBUG:boto:Data: 2023-11-02T18:42:29.121 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.122 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.122 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.122 DEBUG:boto:Params: {} 2023-11-02T18:42:29.123 DEBUG:boto:Token: None 2023-11-02T18:42:29.124 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj22 2023-11-02T18:42:29.124 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6/KWSTlXzMY9TRB8z1rxh3ygVe8= 2023-11-02T18:42:29.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6/KWSTlXzMY9TRB8z1rxh3ygVe8=', 'Content-Length': '0'} 2023-11-02T18:42:29.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a07ca6d72d6af0f4-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.128 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:42:29.128 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:42:29.129 DEBUG:boto:Method: GET 2023-11-02T18:42:29.129 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:42:29.130 DEBUG:boto:Data: 2023-11-02T18:42:29.130 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.131 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.131 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.132 DEBUG:boto:Params: {} 2023-11-02T18:42:29.132 DEBUG:boto:Token: None 2023-11-02T18:42:29.133 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj22 2023-11-02T18:42:29.133 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6/KWSTlXzMY9TRB8z1rxh3ygVe8= 2023-11-02T18:42:29.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6/KWSTlXzMY9TRB8z1rxh3ygVe8=', 'Content-Length': '0'} 2023-11-02T18:42:29.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7b308fa575be397-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.137 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.138 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.138 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:42:29.139 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:42:29.139 DEBUG:boto:Method: GET 2023-11-02T18:42:29.139 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:42:29.140 DEBUG:boto:Data: 2023-11-02T18:42:29.140 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.141 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.141 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.142 DEBUG:boto:Params: {} 2023-11-02T18:42:29.142 DEBUG:boto:Token: None 2023-11-02T18:42:29.143 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.143 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1054808ff66c414-006543ed95-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.190 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.191 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.191 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.192 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:42:29.192 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:42:29.193 DEBUG:boto:Method: GET 2023-11-02T18:42:29.193 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:42:29.194 DEBUG:boto:Data: 2023-11-02T18:42:29.194 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.194 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.195 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.195 DEBUG:boto:Params: {} 2023-11-02T18:42:29.196 DEBUG:boto:Token: None 2023-11-02T18:42:29.196 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.197 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd55a5080e40121-006543ed95-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.206 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:42:29.208 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:42:29.208 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:42:29.209 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.209 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.209 DEBUG:boto:Method: GET 2023-11-02T18:42:29.210 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.210 DEBUG:boto:Data: 2023-11-02T18:42:29.211 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.211 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.212 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.212 DEBUG:boto:Params: {} 2023-11-02T18:42:29.213 DEBUG:boto:Token: None 2023-11-02T18:42:29.213 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.214 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PyZ1niOGhT2Bqz8+lQzjZetvdC8= 2023-11-02T18:42:29.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PyZ1niOGhT2Bqz8+lQzjZetvdC8=', 'Content-Length': '0'} 2023-11-02T18:42:29.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0d6d3a80c6ae924-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.217 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:42:29.218 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:42:29.218 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.219 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.219 DEBUG:boto:Method: GET 2023-11-02T18:42:29.220 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.220 DEBUG:boto:Data: 2023-11-02T18:42:29.221 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.221 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.221 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.222 DEBUG:boto:Params: {} 2023-11-02T18:42:29.222 DEBUG:boto:Token: None 2023-11-02T18:42:29.223 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.223 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PyZ1niOGhT2Bqz8+lQzjZetvdC8= 2023-11-02T18:42:29.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PyZ1niOGhT2Bqz8+lQzjZetvdC8=', 'Content-Length': '0'} 2023-11-02T18:42:29.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000518e375f59733a5b-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.270 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:42:29.271 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:42:29.271 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.272 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.272 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.273 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.273 DEBUG:boto:Data: 2023-11-02T18:42:29.273 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.274 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.274 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.275 DEBUG:boto:Params: {} 2023-11-02T18:42:29.275 DEBUG:boto:Token: None 2023-11-02T18:42:29.276 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.276 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2y9jD2tokzUk89bS7ZiHOHVOgJQ= 2023-11-02T18:42:29.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2y9jD2tokzUk89bS7ZiHOHVOgJQ=', 'Content-Length': '0'} 2023-11-02T18:42:29.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000862bba952920833c-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.280 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:42:29.281 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:42:29.281 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.282 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.282 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.283 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.283 DEBUG:boto:Data: 2023-11-02T18:42:29.283 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.284 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.284 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.285 DEBUG:boto:Params: {} 2023-11-02T18:42:29.285 DEBUG:boto:Token: None 2023-11-02T18:42:29.286 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.286 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2y9jD2tokzUk89bS7ZiHOHVOgJQ= 2023-11-02T18:42:29.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2y9jD2tokzUk89bS7ZiHOHVOgJQ=', 'Content-Length': '0'} 2023-11-02T18:42:29.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040d44d2d60dd9f19-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:42:29.291 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:42:29.291 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:42:29.291 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.292 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.292 DEBUG:boto:Method: GET 2023-11-02T18:42:29.293 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.293 DEBUG:boto:Data: 2023-11-02T18:42:29.294 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.294 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.295 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.295 DEBUG:boto:Params: {} 2023-11-02T18:42:29.295 DEBUG:boto:Token: None 2023-11-02T18:42:29.296 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.296 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PyZ1niOGhT2Bqz8+lQzjZetvdC8= 2023-11-02T18:42:29.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PyZ1niOGhT2Bqz8+lQzjZetvdC8=', 'Content-Length': '0'} 2023-11-02T18:42:29.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c4f2bb972435147-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.301 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:42:29.301 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:42:29.301 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.302 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.302 DEBUG:boto:Method: GET 2023-11-02T18:42:29.303 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.303 DEBUG:boto:Data: 2023-11-02T18:42:29.304 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.304 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.305 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.305 DEBUG:boto:Params: {} 2023-11-02T18:42:29.305 DEBUG:boto:Token: None 2023-11-02T18:42:29.306 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:42:29.306 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PyZ1niOGhT2Bqz8+lQzjZetvdC8= 2023-11-02T18:42:29.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PyZ1niOGhT2Bqz8+lQzjZetvdC8=', 'Content-Length': '0'} 2023-11-02T18:42:29.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c9209337cee6f87-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.311 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.311 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.312 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:42:29.312 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:42:29.313 DEBUG:boto:Method: GET 2023-11-02T18:42:29.313 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:42:29.313 DEBUG:boto:Data: 2023-11-02T18:42:29.314 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.314 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.315 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.315 DEBUG:boto:Params: {} 2023-11-02T18:42:29.316 DEBUG:boto:Token: None 2023-11-02T18:42:29.316 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.317 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e8d4909c138b2e2-006543ed95-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.366 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.367 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.368 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.368 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:42:29.369 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:42:29.369 DEBUG:boto:Method: GET 2023-11-02T18:42:29.369 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:42:29.370 DEBUG:boto:Data: 2023-11-02T18:42:29.370 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.371 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.371 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.372 DEBUG:boto:Params: {} 2023-11-02T18:42:29.373 DEBUG:boto:Token: None 2023-11-02T18:42:29.373 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.374 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd56074be09d6314-006543ed95-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.384 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.384 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:42:29.385 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:42:29.385 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.386 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:42:29.386 DEBUG:boto:Data: 2023-11-02T18:42:29.387 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.387 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.387 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.388 DEBUG:boto:Params: {} 2023-11-02T18:42:29.388 DEBUG:boto:Token: None 2023-11-02T18:42:29.389 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj23 2023-11-02T18:42:29.389 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Nm1Dt555kX6/aa6gSb6CCDMm38k= 2023-11-02T18:42:29.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Nm1Dt555kX6/aa6gSb6CCDMm38k=', 'Content-Length': '0'} 2023-11-02T18:42:29.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae4e99929b511bd4-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.393 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:42:29.393 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:42:29.394 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.394 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:42:29.395 DEBUG:boto:Data: 2023-11-02T18:42:29.395 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.395 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.396 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.396 DEBUG:boto:Params: {} 2023-11-02T18:42:29.397 DEBUG:boto:Token: None 2023-11-02T18:42:29.397 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj23 2023-11-02T18:42:29.398 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Nm1Dt555kX6/aa6gSb6CCDMm38k= 2023-11-02T18:42:29.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Nm1Dt555kX6/aa6gSb6CCDMm38k=', 'Content-Length': '0'} 2023-11-02T18:42:29.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a3ef1c30be96a68-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:42:29.402 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:42:29.402 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:42:29.403 DEBUG:boto:Method: GET 2023-11-02T18:42:29.403 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:42:29.404 DEBUG:boto:Data: 2023-11-02T18:42:29.404 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.404 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.405 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.405 DEBUG:boto:Params: {} 2023-11-02T18:42:29.406 DEBUG:boto:Token: None 2023-11-02T18:42:29.409 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj23 2023-11-02T18:42:29.410 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F8OB59SqsL1JQC9v2M4bK+Wmu+U= 2023-11-02T18:42:29.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F8OB59SqsL1JQC9v2M4bK+Wmu+U=', 'Content-Length': '0'} 2023-11-02T18:42:29.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b4d0a7e50e12a0-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.413 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:42:29.414 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:42:29.414 DEBUG:boto:Method: GET 2023-11-02T18:42:29.415 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:42:29.415 DEBUG:boto:Data: 2023-11-02T18:42:29.416 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.416 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.417 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.417 DEBUG:boto:Params: {} 2023-11-02T18:42:29.417 DEBUG:boto:Token: None 2023-11-02T18:42:29.418 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj23 2023-11-02T18:42:29.419 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F8OB59SqsL1JQC9v2M4bK+Wmu+U= 2023-11-02T18:42:29.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F8OB59SqsL1JQC9v2M4bK+Wmu+U=', 'Content-Length': '0'} 2023-11-02T18:42:29.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2aac67868f08152-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.423 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.423 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.424 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:42:29.424 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:42:29.425 DEBUG:boto:Method: GET 2023-11-02T18:42:29.425 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:42:29.425 DEBUG:boto:Data: 2023-11-02T18:42:29.426 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.427 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.427 DEBUG:boto:Params: {} 2023-11-02T18:42:29.428 DEBUG:boto:Token: None 2023-11-02T18:42:29.428 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.429 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001399826a381c880a-006543ed95-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.435 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.436 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.436 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.437 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:42:29.437 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:42:29.438 DEBUG:boto:Method: GET 2023-11-02T18:42:29.438 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:42:29.438 DEBUG:boto:Data: 2023-11-02T18:42:29.439 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.439 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.440 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.441 DEBUG:boto:Params: {} 2023-11-02T18:42:29.441 DEBUG:boto:Token: None 2023-11-02T18:42:29.443 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.443 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f6eacc23867e6cd-006543ed95-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.494 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:42:29.496 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:42:29.497 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:42:29.497 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.498 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.498 DEBUG:boto:Method: GET 2023-11-02T18:42:29.499 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.499 DEBUG:boto:Data: 2023-11-02T18:42:29.500 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.500 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.500 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.501 DEBUG:boto:Params: {} 2023-11-02T18:42:29.501 DEBUG:boto:Token: None 2023-11-02T18:42:29.502 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.502 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rJnUhiFhzPwWxpc2BMoqlmDHIcA= 2023-11-02T18:42:29.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rJnUhiFhzPwWxpc2BMoqlmDHIcA=', 'Content-Length': '0'} 2023-11-02T18:42:29.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daf7fcaab1fdfb86-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.507 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:42:29.507 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:42:29.508 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.508 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.509 DEBUG:boto:Method: GET 2023-11-02T18:42:29.509 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.510 DEBUG:boto:Data: 2023-11-02T18:42:29.510 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.511 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.511 DEBUG:boto:Params: {} 2023-11-02T18:42:29.512 DEBUG:boto:Token: None 2023-11-02T18:42:29.512 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rJnUhiFhzPwWxpc2BMoqlmDHIcA= 2023-11-02T18:42:29.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rJnUhiFhzPwWxpc2BMoqlmDHIcA=', 'Content-Length': '0'} 2023-11-02T18:42:29.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081cf0cd8f96f126b-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.517 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:42:29.518 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:42:29.518 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.519 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.519 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.520 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.520 DEBUG:boto:Data: 2023-11-02T18:42:29.520 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.521 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.521 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.522 DEBUG:boto:Params: {} 2023-11-02T18:42:29.522 DEBUG:boto:Token: None 2023-11-02T18:42:29.523 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.523 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ue8V/R7JGzC60o6Y2d/WXX8q8K4= 2023-11-02T18:42:29.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ue8V/R7JGzC60o6Y2d/WXX8q8K4=', 'Content-Length': '0'} 2023-11-02T18:42:29.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b565183baaa27c99-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.527 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:42:29.528 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:42:29.528 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.529 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.529 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.529 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.530 DEBUG:boto:Data: 2023-11-02T18:42:29.530 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.531 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.531 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.531 DEBUG:boto:Params: {} 2023-11-02T18:42:29.532 DEBUG:boto:Token: None 2023-11-02T18:42:29.532 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.533 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ue8V/R7JGzC60o6Y2d/WXX8q8K4= 2023-11-02T18:42:29.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ue8V/R7JGzC60o6Y2d/WXX8q8K4=', 'Content-Length': '0'} 2023-11-02T18:42:29.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c06a0c8fe33c0af-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:42:29.537 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:42:29.538 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:42:29.538 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.539 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.539 DEBUG:boto:Method: GET 2023-11-02T18:42:29.539 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.540 DEBUG:boto:Data: 2023-11-02T18:42:29.540 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.541 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.542 DEBUG:boto:Params: {} 2023-11-02T18:42:29.542 DEBUG:boto:Token: None 2023-11-02T18:42:29.543 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rJnUhiFhzPwWxpc2BMoqlmDHIcA= 2023-11-02T18:42:29.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rJnUhiFhzPwWxpc2BMoqlmDHIcA=', 'Content-Length': '0'} 2023-11-02T18:42:29.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000185bd484084b54ca-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.594 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:42:29.595 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:42:29.595 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.596 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.596 DEBUG:boto:Method: GET 2023-11-02T18:42:29.597 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.598 DEBUG:boto:Data: 2023-11-02T18:42:29.599 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.599 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.600 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.600 DEBUG:boto:Params: {} 2023-11-02T18:42:29.601 DEBUG:boto:Token: None 2023-11-02T18:42:29.601 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:42:29.602 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rJnUhiFhzPwWxpc2BMoqlmDHIcA= 2023-11-02T18:42:29.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rJnUhiFhzPwWxpc2BMoqlmDHIcA=', 'Content-Length': '0'} 2023-11-02T18:42:29.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b2991837f7aa79c-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.606 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.607 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.607 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:42:29.608 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:42:29.608 DEBUG:boto:Method: GET 2023-11-02T18:42:29.609 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:42:29.609 DEBUG:boto:Data: 2023-11-02T18:42:29.609 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.610 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.610 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.611 DEBUG:boto:Params: {} 2023-11-02T18:42:29.611 DEBUG:boto:Token: None 2023-11-02T18:42:29.612 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d92f115bdd958f0-006543ed95-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.623 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.624 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.624 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.625 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:42:29.625 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:42:29.625 DEBUG:boto:Method: GET 2023-11-02T18:42:29.626 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:42:29.626 DEBUG:boto:Data: 2023-11-02T18:42:29.627 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.627 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.628 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.628 DEBUG:boto:Params: {} 2023-11-02T18:42:29.629 DEBUG:boto:Token: None 2023-11-02T18:42:29.629 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.630 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011200aa858b7a16c-006543ed95-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.635 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.636 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:42:29.637 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:42:29.637 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.637 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:42:29.638 DEBUG:boto:Data: 2023-11-02T18:42:29.638 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.640 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.640 DEBUG:boto:Params: {} 2023-11-02T18:42:29.641 DEBUG:boto:Token: None 2023-11-02T18:42:29.641 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj29 2023-11-02T18:42:29.642 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e6BDByP+9lGq2l8IAyffQHwep4k= 2023-11-02T18:42:29.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e6BDByP+9lGq2l8IAyffQHwep4k=', 'Content-Length': '0'} 2023-11-02T18:42:29.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059254c25dac0d705-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.647 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:42:29.647 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:42:29.648 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.649 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:42:29.649 DEBUG:boto:Data: 2023-11-02T18:42:29.650 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.650 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.651 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.651 DEBUG:boto:Params: {} 2023-11-02T18:42:29.652 DEBUG:boto:Token: None 2023-11-02T18:42:29.653 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj29 2023-11-02T18:42:29.654 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e6BDByP+9lGq2l8IAyffQHwep4k= 2023-11-02T18:42:29.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e6BDByP+9lGq2l8IAyffQHwep4k=', 'Content-Length': '0'} 2023-11-02T18:42:29.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c54396ec9bcc5cb-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:42:29.659 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:42:29.660 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:42:29.661 DEBUG:boto:Method: GET 2023-11-02T18:42:29.661 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:42:29.662 DEBUG:boto:Data: 2023-11-02T18:42:29.662 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.663 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.664 DEBUG:boto:Params: {} 2023-11-02T18:42:29.664 DEBUG:boto:Token: None 2023-11-02T18:42:29.664 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj29 2023-11-02T18:42:29.665 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IDJcL5zF2nfQxf+wOCPfkTMateQ= 2023-11-02T18:42:29.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IDJcL5zF2nfQxf+wOCPfkTMateQ=', 'Content-Length': '0'} 2023-11-02T18:42:29.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9a56db3d0f8a43d-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.670 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:42:29.671 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:42:29.671 DEBUG:boto:Method: GET 2023-11-02T18:42:29.672 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:42:29.673 DEBUG:boto:Data: 2023-11-02T18:42:29.674 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.675 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.675 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.676 DEBUG:boto:Params: {} 2023-11-02T18:42:29.677 DEBUG:boto:Token: None 2023-11-02T18:42:29.678 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj29 2023-11-02T18:42:29.679 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IDJcL5zF2nfQxf+wOCPfkTMateQ= 2023-11-02T18:42:29.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IDJcL5zF2nfQxf+wOCPfkTMateQ=', 'Content-Length': '0'} 2023-11-02T18:42:29.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c0b58f6a9cedd49-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.684 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.685 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.686 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:42:29.686 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:42:29.687 DEBUG:boto:Method: GET 2023-11-02T18:42:29.688 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:42:29.689 DEBUG:boto:Data: 2023-11-02T18:42:29.690 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.691 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.692 DEBUG:boto:Params: {} 2023-11-02T18:42:29.693 DEBUG:boto:Token: None 2023-11-02T18:42:29.694 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.695 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fae8fb7a400c346a-006543ed95-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.701 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.702 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.703 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.703 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:42:29.704 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:42:29.705 DEBUG:boto:Method: GET 2023-11-02T18:42:29.705 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:42:29.706 DEBUG:boto:Data: 2023-11-02T18:42:29.706 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.706 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.707 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.707 DEBUG:boto:Params: {} 2023-11-02T18:42:29.708 DEBUG:boto:Token: None 2023-11-02T18:42:29.708 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.709 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d20e9b8431f658c-006543ed95-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.758 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:42:29.760 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:42:29.760 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:42:29.760 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.761 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.761 DEBUG:boto:Method: GET 2023-11-02T18:42:29.762 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.762 DEBUG:boto:Data: 2023-11-02T18:42:29.763 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.763 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.764 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.764 DEBUG:boto:Params: {} 2023-11-02T18:42:29.764 DEBUG:boto:Token: None 2023-11-02T18:42:29.765 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C+p7+3qMOtqpm7eWnpjU8ADtR1w= 2023-11-02T18:42:29.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C+p7+3qMOtqpm7eWnpjU8ADtR1w=', 'Content-Length': '0'} 2023-11-02T18:42:29.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba0952d88c3c26a-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.771 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:42:29.772 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:42:29.773 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.774 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.774 DEBUG:boto:Method: GET 2023-11-02T18:42:29.775 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.776 DEBUG:boto:Data: 2023-11-02T18:42:29.777 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.777 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.777 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.778 DEBUG:boto:Params: {} 2023-11-02T18:42:29.778 DEBUG:boto:Token: None 2023-11-02T18:42:29.779 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.779 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C+p7+3qMOtqpm7eWnpjU8ADtR1w= 2023-11-02T18:42:29.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C+p7+3qMOtqpm7eWnpjU8ADtR1w=', 'Content-Length': '0'} 2023-11-02T18:42:29.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233d3e4186902ae0-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.784 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:42:29.785 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:42:29.785 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.786 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.786 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.787 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.787 DEBUG:boto:Data: 2023-11-02T18:42:29.788 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.788 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.788 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.789 DEBUG:boto:Params: {} 2023-11-02T18:42:29.789 DEBUG:boto:Token: None 2023-11-02T18:42:29.790 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.790 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xuCd0gYe36YSmHXL1EbSqXpqmkk= 2023-11-02T18:42:29.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xuCd0gYe36YSmHXL1EbSqXpqmkk=', 'Content-Length': '0'} 2023-11-02T18:42:29.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001709f99fc39f6188-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.794 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:42:29.795 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:42:29.795 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.796 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.796 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.796 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.797 DEBUG:boto:Data: 2023-11-02T18:42:29.797 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.797 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.798 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.798 DEBUG:boto:Params: {} 2023-11-02T18:42:29.799 DEBUG:boto:Token: None 2023-11-02T18:42:29.799 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.800 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xuCd0gYe36YSmHXL1EbSqXpqmkk= 2023-11-02T18:42:29.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xuCd0gYe36YSmHXL1EbSqXpqmkk=', 'Content-Length': '0'} 2023-11-02T18:42:29.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cf6b668bea9ebb7-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:42:29.805 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:42:29.806 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:42:29.807 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.807 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.808 DEBUG:boto:Method: GET 2023-11-02T18:42:29.808 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.808 DEBUG:boto:Data: 2023-11-02T18:42:29.809 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.809 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.810 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.810 DEBUG:boto:Params: {} 2023-11-02T18:42:29.811 DEBUG:boto:Token: None 2023-11-02T18:42:29.811 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.812 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C+p7+3qMOtqpm7eWnpjU8ADtR1w= 2023-11-02T18:42:29.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C+p7+3qMOtqpm7eWnpjU8ADtR1w=', 'Content-Length': '0'} 2023-11-02T18:42:29.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbf3a0289f6466e5-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.862 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:42:29.863 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:42:29.864 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.864 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.865 DEBUG:boto:Method: GET 2023-11-02T18:42:29.865 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.866 DEBUG:boto:Data: 2023-11-02T18:42:29.866 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.867 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.867 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.868 DEBUG:boto:Params: {} 2023-11-02T18:42:29.868 DEBUG:boto:Token: None 2023-11-02T18:42:29.869 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:42:29.869 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C+p7+3qMOtqpm7eWnpjU8ADtR1w= 2023-11-02T18:42:29.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C+p7+3qMOtqpm7eWnpjU8ADtR1w=', 'Content-Length': '0'} 2023-11-02T18:42:29.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da52aa6d691ebc95-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.874 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.874 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.875 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:42:29.875 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:42:29.876 DEBUG:boto:Method: GET 2023-11-02T18:42:29.876 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:42:29.877 DEBUG:boto:Data: 2023-11-02T18:42:29.877 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.878 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.878 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.879 DEBUG:boto:Params: {} 2023-11-02T18:42:29.879 DEBUG:boto:Token: None 2023-11-02T18:42:29.880 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.880 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfd6bc1ad65702c5-006543ed95-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.887 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.888 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.888 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.889 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:42:29.889 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:42:29.890 DEBUG:boto:Method: GET 2023-11-02T18:42:29.890 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:42:29.891 DEBUG:boto:Data: 2023-11-02T18:42:29.891 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.892 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.892 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.894 DEBUG:boto:Params: {} 2023-11-02T18:42:29.894 DEBUG:boto:Token: None 2023-11-02T18:42:29.895 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.896 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:29.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdbe35aeae77e7ff-006543ed95-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.906 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:29.907 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:42:29.908 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:42:29.908 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.908 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:42:29.909 DEBUG:boto:Data: 2023-11-02T18:42:29.909 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.910 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.911 DEBUG:boto:Params: {} 2023-11-02T18:42:29.911 DEBUG:boto:Token: None 2023-11-02T18:42:29.912 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj33 2023-11-02T18:42:29.912 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TncYCJgyoxjnN8FiuYBO/jvom2w= 2023-11-02T18:42:29.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TncYCJgyoxjnN8FiuYBO/jvom2w=', 'Content-Length': '0'} 2023-11-02T18:42:29.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f485d3b72a1bd9e-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.916 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:42:29.916 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:42:29.917 DEBUG:boto:Method: HEAD 2023-11-02T18:42:29.917 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:42:29.918 DEBUG:boto:Data: 2023-11-02T18:42:29.918 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.919 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.919 DEBUG:boto:Params: {} 2023-11-02T18:42:29.920 DEBUG:boto:Token: None 2023-11-02T18:42:29.920 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj33 2023-11-02T18:42:29.921 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TncYCJgyoxjnN8FiuYBO/jvom2w= 2023-11-02T18:42:29.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TncYCJgyoxjnN8FiuYBO/jvom2w=', 'Content-Length': '0'} 2023-11-02T18:42:29.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae16d2ff80947c1d-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:42:29.925 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:42:29.926 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:42:29.926 DEBUG:boto:Method: GET 2023-11-02T18:42:29.927 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:42:29.928 DEBUG:boto:Data: 2023-11-02T18:42:29.929 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.929 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.929 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.930 DEBUG:boto:Params: {} 2023-11-02T18:42:29.930 DEBUG:boto:Token: None 2023-11-02T18:42:29.931 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj33 2023-11-02T18:42:29.931 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WWiKscrYJEJK2oSeDiIsb0o3XHg= 2023-11-02T18:42:29.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WWiKscrYJEJK2oSeDiIsb0o3XHg=', 'Content-Length': '0'} 2023-11-02T18:42:29.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001df7ef83893c288c-006543ed95-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.982 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:42:29.983 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:42:29.983 DEBUG:boto:Method: GET 2023-11-02T18:42:29.984 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:42:29.984 DEBUG:boto:Data: 2023-11-02T18:42:29.985 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.985 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:29.986 DEBUG:boto:Port: 8000 2023-11-02T18:42:29.986 DEBUG:boto:Params: {} 2023-11-02T18:42:29.987 DEBUG:boto:Token: None 2023-11-02T18:42:29.987 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/obj33 2023-11-02T18:42:29.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WWiKscrYJEJK2oSeDiIsb0o3XHg= 2023-11-02T18:42:29.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WWiKscrYJEJK2oSeDiIsb0o3XHg=', 'Content-Length': '0'} 2023-11-02T18:42:29.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dc5ce003397dbc6-006543ed95-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:29.992 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:29.992 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:29.993 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:42:29.993 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:42:29.994 DEBUG:boto:Method: GET 2023-11-02T18:42:29.994 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:42:29.995 DEBUG:boto:Data: 2023-11-02T18:42:29.995 DEBUG:boto:Headers: {} 2023-11-02T18:42:29.996 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:29.996 DEBUG:boto:Port: 8001 2023-11-02T18:42:29.996 DEBUG:boto:Params: {} 2023-11-02T18:42:29.997 DEBUG:boto:Token: None 2023-11-02T18:42:29.997 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:29 GMT /ibtljn-59/ 2023-11-02T18:42:29.998 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc= 2023-11-02T18:42:29.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5eRDwa4azAyWZZ6uOmrfoeQKNnc=', 'Content-Length': '0'} 2023-11-02T18:42:30.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8def0fa25c4f4cb-006543ed95-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.005 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.006 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.007 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.007 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:42:30.008 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:42:30.008 DEBUG:boto:Method: GET 2023-11-02T18:42:30.009 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:42:30.009 DEBUG:boto:Data: 2023-11-02T18:42:30.010 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.010 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.010 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.011 DEBUG:boto:Params: {} 2023-11-02T18:42:30.011 DEBUG:boto:Token: None 2023-11-02T18:42:30.012 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3153b4fa17f985c-006543ed96-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.023 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:42:30.024 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:42:30.025 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:42:30.025 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.026 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.027 DEBUG:boto:Method: GET 2023-11-02T18:42:30.027 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.028 DEBUG:boto:Data: 2023-11-02T18:42:30.028 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.029 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.029 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.030 DEBUG:boto:Params: {} 2023-11-02T18:42:30.030 DEBUG:boto:Token: None 2023-11-02T18:42:30.031 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5zeAd4Fq3Tn/ujNNOtTFsTUvsWQ= 2023-11-02T18:42:30.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5zeAd4Fq3Tn/ujNNOtTFsTUvsWQ=', 'Content-Length': '0'} 2023-11-02T18:42:30.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083a27a6ab19e34d9-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.036 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:42:30.036 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:42:30.037 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.037 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.038 DEBUG:boto:Method: GET 2023-11-02T18:42:30.038 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.039 DEBUG:boto:Data: 2023-11-02T18:42:30.039 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.040 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.040 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.041 DEBUG:boto:Params: {} 2023-11-02T18:42:30.041 DEBUG:boto:Token: None 2023-11-02T18:42:30.042 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.042 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5zeAd4Fq3Tn/ujNNOtTFsTUvsWQ= 2023-11-02T18:42:30.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5zeAd4Fq3Tn/ujNNOtTFsTUvsWQ=', 'Content-Length': '0'} 2023-11-02T18:42:30.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e19109766bda8ff-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.046 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:42:30.047 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:42:30.047 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.048 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.048 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.049 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.049 DEBUG:boto:Data: 2023-11-02T18:42:30.049 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.050 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.051 DEBUG:boto:Params: {} 2023-11-02T18:42:30.051 DEBUG:boto:Token: None 2023-11-02T18:42:30.052 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.052 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FUzpKcxZEjL7B7JyEzV3OX6yLZo= 2023-11-02T18:42:30.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FUzpKcxZEjL7B7JyEzV3OX6yLZo=', 'Content-Length': '0'} 2023-11-02T18:42:30.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aeea4a2673d28ef-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.056 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:42:30.058 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:42:30.058 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.059 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.059 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.059 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.060 DEBUG:boto:Data: 2023-11-02T18:42:30.060 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.061 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.061 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.062 DEBUG:boto:Params: {} 2023-11-02T18:42:30.062 DEBUG:boto:Token: None 2023-11-02T18:42:30.063 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.063 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FUzpKcxZEjL7B7JyEzV3OX6yLZo= 2023-11-02T18:42:30.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FUzpKcxZEjL7B7JyEzV3OX6yLZo=', 'Content-Length': '0'} 2023-11-02T18:42:30.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004821a61e11bcaf4-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:42:30.067 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:42:30.068 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:42:30.068 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.069 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.069 DEBUG:boto:Method: GET 2023-11-02T18:42:30.070 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.070 DEBUG:boto:Data: 2023-11-02T18:42:30.071 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.071 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.072 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.072 DEBUG:boto:Params: {} 2023-11-02T18:42:30.073 DEBUG:boto:Token: None 2023-11-02T18:42:30.073 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.074 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5zeAd4Fq3Tn/ujNNOtTFsTUvsWQ= 2023-11-02T18:42:30.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5zeAd4Fq3Tn/ujNNOtTFsTUvsWQ=', 'Content-Length': '0'} 2023-11-02T18:42:30.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8dc0504262dddac-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.122 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:42:30.123 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:42:30.123 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.124 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.124 DEBUG:boto:Method: GET 2023-11-02T18:42:30.125 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.125 DEBUG:boto:Data: 2023-11-02T18:42:30.126 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.126 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.127 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.127 DEBUG:boto:Params: {} 2023-11-02T18:42:30.128 DEBUG:boto:Token: None 2023-11-02T18:42:30.128 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:42:30.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5zeAd4Fq3Tn/ujNNOtTFsTUvsWQ= 2023-11-02T18:42:30.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5zeAd4Fq3Tn/ujNNOtTFsTUvsWQ=', 'Content-Length': '0'} 2023-11-02T18:42:30.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000465f9912c2f805a0-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.133 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.134 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.134 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:42:30.134 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:42:30.135 DEBUG:boto:Method: GET 2023-11-02T18:42:30.135 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:42:30.136 DEBUG:boto:Data: 2023-11-02T18:42:30.136 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.137 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.137 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.138 DEBUG:boto:Params: {} 2023-11-02T18:42:30.138 DEBUG:boto:Token: None 2023-11-02T18:42:30.139 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.139 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042cb52bd5a1decbd-006543ed96-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.145 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.146 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.147 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.147 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:42:30.147 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:42:30.148 DEBUG:boto:Method: GET 2023-11-02T18:42:30.148 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:42:30.149 DEBUG:boto:Data: 2023-11-02T18:42:30.149 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.150 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.150 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.151 DEBUG:boto:Params: {} 2023-11-02T18:42:30.151 DEBUG:boto:Token: None 2023-11-02T18:42:30.151 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.152 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dd5908dc8b5c97b-006543ed96-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.206 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.207 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:42:30.208 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:42:30.208 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.209 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:42:30.209 DEBUG:boto:Data: 2023-11-02T18:42:30.210 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.210 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.211 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.211 DEBUG:boto:Params: {} 2023-11-02T18:42:30.212 DEBUG:boto:Token: None 2023-11-02T18:42:30.213 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj34 2023-11-02T18:42:30.213 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t1Z+xVGjYRyquD0t4tANVQh6n+E= 2023-11-02T18:42:30.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t1Z+xVGjYRyquD0t4tANVQh6n+E=', 'Content-Length': '0'} 2023-11-02T18:42:30.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c66e5cbbd57eae1-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.217 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:42:30.218 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:42:30.218 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.219 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:42:30.219 DEBUG:boto:Data: 2023-11-02T18:42:30.220 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.220 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.220 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.221 DEBUG:boto:Params: {} 2023-11-02T18:42:30.221 DEBUG:boto:Token: None 2023-11-02T18:42:30.222 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj34 2023-11-02T18:42:30.222 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t1Z+xVGjYRyquD0t4tANVQh6n+E= 2023-11-02T18:42:30.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t1Z+xVGjYRyquD0t4tANVQh6n+E=', 'Content-Length': '0'} 2023-11-02T18:42:30.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002edd774ee0129d34-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:42:30.227 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:42:30.227 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:42:30.228 DEBUG:boto:Method: GET 2023-11-02T18:42:30.228 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:42:30.229 DEBUG:boto:Data: 2023-11-02T18:42:30.229 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.230 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.230 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.231 DEBUG:boto:Params: {} 2023-11-02T18:42:30.231 DEBUG:boto:Token: None 2023-11-02T18:42:30.232 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj34 2023-11-02T18:42:30.232 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:homIPKUUIlO2bzWssoDHw9ysgQ0= 2023-11-02T18:42:30.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:homIPKUUIlO2bzWssoDHw9ysgQ0=', 'Content-Length': '0'} 2023-11-02T18:42:30.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b55d32d11dfd22a-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.238 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:42:30.238 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:42:30.239 DEBUG:boto:Method: GET 2023-11-02T18:42:30.239 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:42:30.240 DEBUG:boto:Data: 2023-11-02T18:42:30.240 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.241 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.241 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.241 DEBUG:boto:Params: {} 2023-11-02T18:42:30.242 DEBUG:boto:Token: None 2023-11-02T18:42:30.243 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj34 2023-11-02T18:42:30.243 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:homIPKUUIlO2bzWssoDHw9ysgQ0= 2023-11-02T18:42:30.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:homIPKUUIlO2bzWssoDHw9ysgQ0=', 'Content-Length': '0'} 2023-11-02T18:42:30.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0d68464c3c56b4f-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.248 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.248 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.249 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:42:30.249 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:42:30.250 DEBUG:boto:Method: GET 2023-11-02T18:42:30.250 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:42:30.251 DEBUG:boto:Data: 2023-11-02T18:42:30.251 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.252 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.252 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.252 DEBUG:boto:Params: {} 2023-11-02T18:42:30.253 DEBUG:boto:Token: None 2023-11-02T18:42:30.254 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.254 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000733411af721ba9a4-006543ed96-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.260 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.261 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.262 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.262 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:42:30.263 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:42:30.263 DEBUG:boto:Method: GET 2023-11-02T18:42:30.264 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:42:30.264 DEBUG:boto:Data: 2023-11-02T18:42:30.265 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.265 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.266 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.266 DEBUG:boto:Params: {} 2023-11-02T18:42:30.267 DEBUG:boto:Token: None 2023-11-02T18:42:30.267 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.268 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072f77a0ddf003037-006543ed96-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.277 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:42:30.279 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:42:30.279 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:42:30.279 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.280 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.280 DEBUG:boto:Method: GET 2023-11-02T18:42:30.281 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.281 DEBUG:boto:Data: 2023-11-02T18:42:30.282 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.282 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.283 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.283 DEBUG:boto:Params: {} 2023-11-02T18:42:30.284 DEBUG:boto:Token: None 2023-11-02T18:42:30.284 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.285 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SbhIWrS8qF20Gb38zkc4nnPL4MU= 2023-11-02T18:42:30.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SbhIWrS8qF20Gb38zkc4nnPL4MU=', 'Content-Length': '0'} 2023-11-02T18:42:30.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dc482e3a9c9e8af-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.330 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:42:30.331 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:42:30.331 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.332 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.332 DEBUG:boto:Method: GET 2023-11-02T18:42:30.333 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.333 DEBUG:boto:Data: 2023-11-02T18:42:30.334 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.334 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.335 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.335 DEBUG:boto:Params: {} 2023-11-02T18:42:30.336 DEBUG:boto:Token: None 2023-11-02T18:42:30.336 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.337 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SbhIWrS8qF20Gb38zkc4nnPL4MU= 2023-11-02T18:42:30.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SbhIWrS8qF20Gb38zkc4nnPL4MU=', 'Content-Length': '0'} 2023-11-02T18:42:30.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c13d91f25d8f497-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.341 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:42:30.342 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:42:30.342 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.343 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.343 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.344 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.344 DEBUG:boto:Data: 2023-11-02T18:42:30.345 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.345 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.346 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.346 DEBUG:boto:Params: {} 2023-11-02T18:42:30.347 DEBUG:boto:Token: None 2023-11-02T18:42:30.347 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.348 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y7aCpPyjA4f4YS9lGybSajGBzxw= 2023-11-02T18:42:30.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y7aCpPyjA4f4YS9lGybSajGBzxw=', 'Content-Length': '0'} 2023-11-02T18:42:30.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003666e2b0dfbed543-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.352 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:42:30.352 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:42:30.353 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.353 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.353 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.355 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.355 DEBUG:boto:Data: 2023-11-02T18:42:30.356 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.356 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.357 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.358 DEBUG:boto:Params: {} 2023-11-02T18:42:30.358 DEBUG:boto:Token: None 2023-11-02T18:42:30.359 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.359 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y7aCpPyjA4f4YS9lGybSajGBzxw= 2023-11-02T18:42:30.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y7aCpPyjA4f4YS9lGybSajGBzxw=', 'Content-Length': '0'} 2023-11-02T18:42:30.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d7d332b778ec6e-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:42:30.364 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:42:30.364 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:42:30.365 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.366 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.366 DEBUG:boto:Method: GET 2023-11-02T18:42:30.367 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.368 DEBUG:boto:Data: 2023-11-02T18:42:30.369 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.369 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.370 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.371 DEBUG:boto:Params: {} 2023-11-02T18:42:30.371 DEBUG:boto:Token: None 2023-11-02T18:42:30.372 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SbhIWrS8qF20Gb38zkc4nnPL4MU= 2023-11-02T18:42:30.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SbhIWrS8qF20Gb38zkc4nnPL4MU=', 'Content-Length': '0'} 2023-11-02T18:42:30.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c48f75cc3acd904-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.376 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:42:30.377 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:42:30.377 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.377 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.378 DEBUG:boto:Method: GET 2023-11-02T18:42:30.378 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.379 DEBUG:boto:Data: 2023-11-02T18:42:30.379 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.380 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.380 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.381 DEBUG:boto:Params: {} 2023-11-02T18:42:30.381 DEBUG:boto:Token: None 2023-11-02T18:42:30.382 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:42:30.383 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SbhIWrS8qF20Gb38zkc4nnPL4MU= 2023-11-02T18:42:30.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SbhIWrS8qF20Gb38zkc4nnPL4MU=', 'Content-Length': '0'} 2023-11-02T18:42:30.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1095eb738306ccd-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.430 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.431 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.431 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:42:30.432 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:42:30.433 DEBUG:boto:Method: GET 2023-11-02T18:42:30.433 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:42:30.434 DEBUG:boto:Data: 2023-11-02T18:42:30.434 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.436 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.437 DEBUG:boto:Params: {} 2023-11-02T18:42:30.437 DEBUG:boto:Token: None 2023-11-02T18:42:30.438 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.439 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066a69e0c6f0bc6a6-006543ed96-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.446 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.446 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.447 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.447 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:42:30.448 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:42:30.448 DEBUG:boto:Method: GET 2023-11-02T18:42:30.449 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:42:30.449 DEBUG:boto:Data: 2023-11-02T18:42:30.449 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.450 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.450 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.451 DEBUG:boto:Params: {} 2023-11-02T18:42:30.451 DEBUG:boto:Token: None 2023-11-02T18:42:30.451 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.452 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000537d4268bb324996-006543ed96-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.462 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.463 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:42:30.463 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:42:30.464 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.465 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:42:30.465 DEBUG:boto:Data: 2023-11-02T18:42:30.466 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.466 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.467 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.467 DEBUG:boto:Params: {} 2023-11-02T18:42:30.468 DEBUG:boto:Token: None 2023-11-02T18:42:30.468 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj38 2023-11-02T18:42:30.469 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hB8zfpdwXCwN9zWHnGlPvhkTrTw= 2023-11-02T18:42:30.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hB8zfpdwXCwN9zWHnGlPvhkTrTw=', 'Content-Length': '0'} 2023-11-02T18:42:30.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b0432b8352bd2aa-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.472 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:42:30.472 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:42:30.473 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.473 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:42:30.474 DEBUG:boto:Data: 2023-11-02T18:42:30.474 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.475 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.475 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.476 DEBUG:boto:Params: {} 2023-11-02T18:42:30.476 DEBUG:boto:Token: None 2023-11-02T18:42:30.477 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj38 2023-11-02T18:42:30.477 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hB8zfpdwXCwN9zWHnGlPvhkTrTw= 2023-11-02T18:42:30.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hB8zfpdwXCwN9zWHnGlPvhkTrTw=', 'Content-Length': '0'} 2023-11-02T18:42:30.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043d8a945fb410c6c-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:42:30.481 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:42:30.482 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:42:30.482 DEBUG:boto:Method: GET 2023-11-02T18:42:30.483 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:42:30.483 DEBUG:boto:Data: 2023-11-02T18:42:30.484 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.484 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.484 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.485 DEBUG:boto:Params: {} 2023-11-02T18:42:30.485 DEBUG:boto:Token: None 2023-11-02T18:42:30.486 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj38 2023-11-02T18:42:30.486 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fCIjXlwONFEL6xXsujAAlWF6bqo= 2023-11-02T18:42:30.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fCIjXlwONFEL6xXsujAAlWF6bqo=', 'Content-Length': '0'} 2023-11-02T18:42:30.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064abdc878b88c52c-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.530 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:42:30.531 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:42:30.531 DEBUG:boto:Method: GET 2023-11-02T18:42:30.532 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:42:30.533 DEBUG:boto:Data: 2023-11-02T18:42:30.533 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.534 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.534 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.535 DEBUG:boto:Params: {} 2023-11-02T18:42:30.535 DEBUG:boto:Token: None 2023-11-02T18:42:30.536 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj38 2023-11-02T18:42:30.536 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fCIjXlwONFEL6xXsujAAlWF6bqo= 2023-11-02T18:42:30.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fCIjXlwONFEL6xXsujAAlWF6bqo=', 'Content-Length': '0'} 2023-11-02T18:42:30.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d55edb4337a2575d-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.540 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.541 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.541 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:42:30.542 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:42:30.542 DEBUG:boto:Method: GET 2023-11-02T18:42:30.543 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:42:30.543 DEBUG:boto:Data: 2023-11-02T18:42:30.543 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.544 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.544 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.545 DEBUG:boto:Params: {} 2023-11-02T18:42:30.545 DEBUG:boto:Token: None 2023-11-02T18:42:30.546 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.546 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c83da2966c0760e3-006543ed96-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.553 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.554 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.555 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.555 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:42:30.556 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:42:30.556 DEBUG:boto:Method: GET 2023-11-02T18:42:30.557 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:42:30.557 DEBUG:boto:Data: 2023-11-02T18:42:30.558 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.559 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.559 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.560 DEBUG:boto:Params: {} 2023-11-02T18:42:30.560 DEBUG:boto:Token: None 2023-11-02T18:42:30.561 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.561 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052063f29e8221adf-006543ed96-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.610 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:42:30.612 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:42:30.612 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:42:30.613 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.613 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.613 DEBUG:boto:Method: GET 2023-11-02T18:42:30.614 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.614 DEBUG:boto:Data: 2023-11-02T18:42:30.615 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.615 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.616 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.616 DEBUG:boto:Params: {} 2023-11-02T18:42:30.617 DEBUG:boto:Token: None 2023-11-02T18:42:30.617 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.618 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+inVm3Ah06AqmQKdEK/G3tnt3VE= 2023-11-02T18:42:30.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+inVm3Ah06AqmQKdEK/G3tnt3VE=', 'Content-Length': '0'} 2023-11-02T18:42:30.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073ba72c25ca072ac-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.623 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:42:30.623 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:42:30.624 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.624 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.624 DEBUG:boto:Method: GET 2023-11-02T18:42:30.625 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.625 DEBUG:boto:Data: 2023-11-02T18:42:30.626 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.626 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.627 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.627 DEBUG:boto:Params: {} 2023-11-02T18:42:30.628 DEBUG:boto:Token: None 2023-11-02T18:42:30.628 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.629 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+inVm3Ah06AqmQKdEK/G3tnt3VE= 2023-11-02T18:42:30.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+inVm3Ah06AqmQKdEK/G3tnt3VE=', 'Content-Length': '0'} 2023-11-02T18:42:30.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b62f6c56ee3b983-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.632 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:42:30.633 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:42:30.633 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.633 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.634 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.634 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.635 DEBUG:boto:Data: 2023-11-02T18:42:30.635 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.635 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.636 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.636 DEBUG:boto:Params: {} 2023-11-02T18:42:30.637 DEBUG:boto:Token: None 2023-11-02T18:42:30.637 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.638 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z9s3gtHjJ6ILUe6oWHAt0Hliqvk= 2023-11-02T18:42:30.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z9s3gtHjJ6ILUe6oWHAt0Hliqvk=', 'Content-Length': '0'} 2023-11-02T18:42:30.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008106c8fac4d097cc-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.641 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:42:30.641 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:42:30.642 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.642 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.643 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.643 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.644 DEBUG:boto:Data: 2023-11-02T18:42:30.644 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.644 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.645 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.645 DEBUG:boto:Params: {} 2023-11-02T18:42:30.646 DEBUG:boto:Token: None 2023-11-02T18:42:30.646 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.647 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z9s3gtHjJ6ILUe6oWHAt0Hliqvk= 2023-11-02T18:42:30.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z9s3gtHjJ6ILUe6oWHAt0Hliqvk=', 'Content-Length': '0'} 2023-11-02T18:42:30.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a38b8128296d8281-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:42:30.650 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:42:30.651 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:42:30.651 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.652 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.652 DEBUG:boto:Method: GET 2023-11-02T18:42:30.653 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.653 DEBUG:boto:Data: 2023-11-02T18:42:30.654 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.654 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.655 DEBUG:boto:Params: {} 2023-11-02T18:42:30.655 DEBUG:boto:Token: None 2023-11-02T18:42:30.656 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+inVm3Ah06AqmQKdEK/G3tnt3VE= 2023-11-02T18:42:30.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+inVm3Ah06AqmQKdEK/G3tnt3VE=', 'Content-Length': '0'} 2023-11-02T18:42:30.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcdcebdfe49341c9-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.660 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:42:30.660 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:42:30.661 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.661 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.662 DEBUG:boto:Method: GET 2023-11-02T18:42:30.662 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.663 DEBUG:boto:Data: 2023-11-02T18:42:30.663 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.664 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.664 DEBUG:boto:Params: {} 2023-11-02T18:42:30.665 DEBUG:boto:Token: None 2023-11-02T18:42:30.665 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:42:30.666 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+inVm3Ah06AqmQKdEK/G3tnt3VE= 2023-11-02T18:42:30.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+inVm3Ah06AqmQKdEK/G3tnt3VE=', 'Content-Length': '0'} 2023-11-02T18:42:30.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008679e65d4020835c-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.669 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.670 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.670 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:42:30.671 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:42:30.671 DEBUG:boto:Method: GET 2023-11-02T18:42:30.672 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:42:30.672 DEBUG:boto:Data: 2023-11-02T18:42:30.672 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.673 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.673 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.674 DEBUG:boto:Params: {} 2023-11-02T18:42:30.674 DEBUG:boto:Token: None 2023-11-02T18:42:30.675 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.675 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b59c06d5d69c4355-006543ed96-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.681 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.681 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.682 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.682 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:42:30.683 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:42:30.683 DEBUG:boto:Method: GET 2023-11-02T18:42:30.684 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:42:30.684 DEBUG:boto:Data: 2023-11-02T18:42:30.685 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.685 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.686 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.687 DEBUG:boto:Params: {} 2023-11-02T18:42:30.687 DEBUG:boto:Token: None 2023-11-02T18:42:30.688 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.688 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d4b7ee919c698e2-006543ed96-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.696 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.697 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:42:30.697 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:42:30.698 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.698 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:42:30.699 DEBUG:boto:Data: 2023-11-02T18:42:30.699 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.699 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.700 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.700 DEBUG:boto:Params: {} 2023-11-02T18:42:30.701 DEBUG:boto:Token: None 2023-11-02T18:42:30.701 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj44 2023-11-02T18:42:30.702 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PSnxcjaQU8JRIYcc7yt5a4kjnzM= 2023-11-02T18:42:30.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PSnxcjaQU8JRIYcc7yt5a4kjnzM=', 'Content-Length': '0'} 2023-11-02T18:42:30.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e39b285cb084bbd6-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.705 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:42:30.706 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:42:30.706 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.707 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:42:30.707 DEBUG:boto:Data: 2023-11-02T18:42:30.707 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.708 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.708 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.709 DEBUG:boto:Params: {} 2023-11-02T18:42:30.709 DEBUG:boto:Token: None 2023-11-02T18:42:30.710 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj44 2023-11-02T18:42:30.710 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PSnxcjaQU8JRIYcc7yt5a4kjnzM= 2023-11-02T18:42:30.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PSnxcjaQU8JRIYcc7yt5a4kjnzM=', 'Content-Length': '0'} 2023-11-02T18:42:30.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8043275f70fecd-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:42:30.714 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:42:30.715 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:42:30.715 DEBUG:boto:Method: GET 2023-11-02T18:42:30.716 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:42:30.716 DEBUG:boto:Data: 2023-11-02T18:42:30.717 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.717 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.717 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.718 DEBUG:boto:Params: {} 2023-11-02T18:42:30.718 DEBUG:boto:Token: None 2023-11-02T18:42:30.719 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj44 2023-11-02T18:42:30.719 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+YEV6ZNv06iLOUU+CFgRlSfH1+I= 2023-11-02T18:42:30.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+YEV6ZNv06iLOUU+CFgRlSfH1+I=', 'Content-Length': '0'} 2023-11-02T18:42:30.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac784025639cd57-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.770 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:42:30.771 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:42:30.771 DEBUG:boto:Method: GET 2023-11-02T18:42:30.772 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:42:30.772 DEBUG:boto:Data: 2023-11-02T18:42:30.773 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.773 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.774 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.775 DEBUG:boto:Params: {} 2023-11-02T18:42:30.776 DEBUG:boto:Token: None 2023-11-02T18:42:30.777 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj44 2023-11-02T18:42:30.778 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+YEV6ZNv06iLOUU+CFgRlSfH1+I= 2023-11-02T18:42:30.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+YEV6ZNv06iLOUU+CFgRlSfH1+I=', 'Content-Length': '0'} 2023-11-02T18:42:30.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c3d9c24f0b6d2c8-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.783 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.783 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.784 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:42:30.784 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:42:30.784 DEBUG:boto:Method: GET 2023-11-02T18:42:30.785 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:42:30.785 DEBUG:boto:Data: 2023-11-02T18:42:30.786 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.786 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.787 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.788 DEBUG:boto:Params: {} 2023-11-02T18:42:30.788 DEBUG:boto:Token: None 2023-11-02T18:42:30.789 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.790 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002099ab680f25ffe-006543ed96-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.797 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.799 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.799 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.800 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:42:30.801 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:42:30.801 DEBUG:boto:Method: GET 2023-11-02T18:42:30.802 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:42:30.802 DEBUG:boto:Data: 2023-11-02T18:42:30.803 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.804 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.804 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.805 DEBUG:boto:Params: {} 2023-11-02T18:42:30.805 DEBUG:boto:Token: None 2023-11-02T18:42:30.806 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.806 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c91f82af42feb25f-006543ed96-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.816 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:42:30.817 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:42:30.818 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:42:30.818 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.819 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.819 DEBUG:boto:Method: GET 2023-11-02T18:42:30.819 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.820 DEBUG:boto:Data: 2023-11-02T18:42:30.820 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.821 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.821 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.822 DEBUG:boto:Params: {} 2023-11-02T18:42:30.822 DEBUG:boto:Token: None 2023-11-02T18:42:30.823 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.823 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hhcd5XFjvlExe/vTn8QTjYVOego= 2023-11-02T18:42:30.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hhcd5XFjvlExe/vTn8QTjYVOego=', 'Content-Length': '0'} 2023-11-02T18:42:30.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075a0f5bbde47fd3e-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.827 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:42:30.828 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:42:30.828 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.829 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.829 DEBUG:boto:Method: GET 2023-11-02T18:42:30.830 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.830 DEBUG:boto:Data: 2023-11-02T18:42:30.831 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.831 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.832 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.832 DEBUG:boto:Params: {} 2023-11-02T18:42:30.833 DEBUG:boto:Token: None 2023-11-02T18:42:30.833 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.834 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hhcd5XFjvlExe/vTn8QTjYVOego= 2023-11-02T18:42:30.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hhcd5XFjvlExe/vTn8QTjYVOego=', 'Content-Length': '0'} 2023-11-02T18:42:30.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007703be7bd190462a-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.838 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:42:30.838 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:42:30.839 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.839 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.840 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.840 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.840 DEBUG:boto:Data: 2023-11-02T18:42:30.841 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.842 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.842 DEBUG:boto:Params: {} 2023-11-02T18:42:30.843 DEBUG:boto:Token: None 2023-11-02T18:42:30.843 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.844 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:82c3UrCbMpjTL90+Cr6C04Iiwi4= 2023-11-02T18:42:30.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:82c3UrCbMpjTL90+Cr6C04Iiwi4=', 'Content-Length': '0'} 2023-11-02T18:42:30.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069955bb2f31acf49-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.849 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:42:30.849 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:42:30.850 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.850 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.851 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.851 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.851 DEBUG:boto:Data: 2023-11-02T18:42:30.852 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.853 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.853 DEBUG:boto:Params: {} 2023-11-02T18:42:30.854 DEBUG:boto:Token: None 2023-11-02T18:42:30.854 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.855 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:82c3UrCbMpjTL90+Cr6C04Iiwi4= 2023-11-02T18:42:30.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:82c3UrCbMpjTL90+Cr6C04Iiwi4=', 'Content-Length': '0'} 2023-11-02T18:42:30.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003608de94a87750cb-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:42:30.859 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:42:30.860 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:42:30.860 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.861 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.861 DEBUG:boto:Method: GET 2023-11-02T18:42:30.861 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.862 DEBUG:boto:Data: 2023-11-02T18:42:30.863 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.863 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.863 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.864 DEBUG:boto:Params: {} 2023-11-02T18:42:30.864 DEBUG:boto:Token: None 2023-11-02T18:42:30.865 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.865 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hhcd5XFjvlExe/vTn8QTjYVOego= 2023-11-02T18:42:30.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hhcd5XFjvlExe/vTn8QTjYVOego=', 'Content-Length': '0'} 2023-11-02T18:42:30.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb039f3b098a98c8-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.911 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:42:30.912 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:42:30.913 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.913 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.914 DEBUG:boto:Method: GET 2023-11-02T18:42:30.914 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.914 DEBUG:boto:Data: 2023-11-02T18:42:30.915 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.915 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.916 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.916 DEBUG:boto:Params: {} 2023-11-02T18:42:30.917 DEBUG:boto:Token: None 2023-11-02T18:42:30.917 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:42:30.918 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hhcd5XFjvlExe/vTn8QTjYVOego= 2023-11-02T18:42:30.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hhcd5XFjvlExe/vTn8QTjYVOego=', 'Content-Length': '0'} 2023-11-02T18:42:30.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002735d3a1e0db7cb2-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.922 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.923 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.923 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:42:30.924 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:42:30.924 DEBUG:boto:Method: GET 2023-11-02T18:42:30.925 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:42:30.925 DEBUG:boto:Data: 2023-11-02T18:42:30.925 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.926 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.926 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.927 DEBUG:boto:Params: {} 2023-11-02T18:42:30.927 DEBUG:boto:Token: None 2023-11-02T18:42:30.928 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.929 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c8df3db64d3357b-006543ed96-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.934 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.935 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:30.936 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:30.936 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:42:30.936 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:42:30.937 DEBUG:boto:Method: GET 2023-11-02T18:42:30.937 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:42:30.938 DEBUG:boto:Data: 2023-11-02T18:42:30.938 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.939 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.939 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.940 DEBUG:boto:Params: {} 2023-11-02T18:42:30.940 DEBUG:boto:Token: None 2023-11-02T18:42:30.941 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/ 2023-11-02T18:42:30.941 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M= 2023-11-02T18:42:30.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zGqcVWIxzJjT5Y0mDF4Kv7eSl2M=', 'Content-Length': '0'} 2023-11-02T18:42:30.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000540455d6812eaea-006543ed96-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.950 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:30.951 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:42:30.951 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:42:30.952 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.952 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:42:30.952 DEBUG:boto:Data: 2023-11-02T18:42:30.953 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.953 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.954 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.954 DEBUG:boto:Params: {} 2023-11-02T18:42:30.955 DEBUG:boto:Token: None 2023-11-02T18:42:30.955 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj46 2023-11-02T18:42:30.956 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0lehJx+OA7mL7MjiTpWChyWx40c= 2023-11-02T18:42:30.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0lehJx+OA7mL7MjiTpWChyWx40c=', 'Content-Length': '0'} 2023-11-02T18:42:30.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d719e6866e90c9f-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.959 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:42:30.960 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:42:30.960 DEBUG:boto:Method: HEAD 2023-11-02T18:42:30.961 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:42:30.961 DEBUG:boto:Data: 2023-11-02T18:42:30.961 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.962 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:30.962 DEBUG:boto:Port: 8000 2023-11-02T18:42:30.963 DEBUG:boto:Params: {} 2023-11-02T18:42:30.963 DEBUG:boto:Token: None 2023-11-02T18:42:30.964 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj46 2023-11-02T18:42:30.964 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0lehJx+OA7mL7MjiTpWChyWx40c= 2023-11-02T18:42:30.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0lehJx+OA7mL7MjiTpWChyWx40c=', 'Content-Length': '0'} 2023-11-02T18:42:30.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec473812be4b7157-006543ed96-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:30.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:42:30.968 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:42:30.969 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:42:30.969 DEBUG:boto:Method: GET 2023-11-02T18:42:30.970 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:42:30.970 DEBUG:boto:Data: 2023-11-02T18:42:30.970 DEBUG:boto:Headers: {} 2023-11-02T18:42:30.971 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:30.971 DEBUG:boto:Port: 8001 2023-11-02T18:42:30.972 DEBUG:boto:Params: {} 2023-11-02T18:42:30.972 DEBUG:boto:Token: None 2023-11-02T18:42:30.973 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:30 GMT /ibtljn-59/obj46 2023-11-02T18:42:30.973 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IvXJrke8jO9rNGaf72E0L/R7yBI= 2023-11-02T18:42:30.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IvXJrke8jO9rNGaf72E0L/R7yBI=', 'Content-Length': '0'} 2023-11-02T18:42:30.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b413ccab2f777054-006543ed96-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.018 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:42:31.019 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:42:31.019 DEBUG:boto:Method: GET 2023-11-02T18:42:31.020 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:42:31.020 DEBUG:boto:Data: 2023-11-02T18:42:31.021 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.021 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.022 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.022 DEBUG:boto:Params: {} 2023-11-02T18:42:31.023 DEBUG:boto:Token: None 2023-11-02T18:42:31.023 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj46 2023-11-02T18:42:31.024 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V14TaqwAULzOGsb2iXSiY/0hpC0= 2023-11-02T18:42:31.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V14TaqwAULzOGsb2iXSiY/0hpC0=', 'Content-Length': '0'} 2023-11-02T18:42:31.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c49453faba0305a-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.028 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.028 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.029 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:42:31.029 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:42:31.030 DEBUG:boto:Method: GET 2023-11-02T18:42:31.030 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:42:31.031 DEBUG:boto:Data: 2023-11-02T18:42:31.031 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.031 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.032 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.032 DEBUG:boto:Params: {} 2023-11-02T18:42:31.033 DEBUG:boto:Token: None 2023-11-02T18:42:31.033 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.034 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c86c55175edefd5d-006543ed97-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.040 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.041 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.041 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.041 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:42:31.042 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:42:31.042 DEBUG:boto:Method: GET 2023-11-02T18:42:31.043 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:42:31.043 DEBUG:boto:Data: 2023-11-02T18:42:31.043 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.044 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.045 DEBUG:boto:Params: {} 2023-11-02T18:42:31.045 DEBUG:boto:Token: None 2023-11-02T18:42:31.046 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.046 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c62852fa6a2187a8-006543ed97-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.102 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:42:31.104 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:42:31.104 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:42:31.105 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.105 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.105 DEBUG:boto:Method: GET 2023-11-02T18:42:31.106 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.106 DEBUG:boto:Data: 2023-11-02T18:42:31.107 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.107 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.108 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.108 DEBUG:boto:Params: {} 2023-11-02T18:42:31.109 DEBUG:boto:Token: None 2023-11-02T18:42:31.109 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.110 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WBk2KZBwj7+tRFHRxuh/DqQr2bc= 2023-11-02T18:42:31.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WBk2KZBwj7+tRFHRxuh/DqQr2bc=', 'Content-Length': '0'} 2023-11-02T18:42:31.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc1b717bab04b41b-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.114 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:42:31.115 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:42:31.115 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.116 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.116 DEBUG:boto:Method: GET 2023-11-02T18:42:31.117 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.117 DEBUG:boto:Data: 2023-11-02T18:42:31.118 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.118 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.119 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.119 DEBUG:boto:Params: {} 2023-11-02T18:42:31.119 DEBUG:boto:Token: None 2023-11-02T18:42:31.120 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WBk2KZBwj7+tRFHRxuh/DqQr2bc= 2023-11-02T18:42:31.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WBk2KZBwj7+tRFHRxuh/DqQr2bc=', 'Content-Length': '0'} 2023-11-02T18:42:31.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030b17d4566697bd5-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.125 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:42:31.125 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:42:31.125 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.126 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.126 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.127 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.127 DEBUG:boto:Data: 2023-11-02T18:42:31.128 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.128 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.129 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.129 DEBUG:boto:Params: {} 2023-11-02T18:42:31.129 DEBUG:boto:Token: None 2023-11-02T18:42:31.130 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B2L/82h2KfFjDLu8D2ru4zdJrcE= 2023-11-02T18:42:31.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B2L/82h2KfFjDLu8D2ru4zdJrcE=', 'Content-Length': '0'} 2023-11-02T18:42:31.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033d95f9e38c477cc-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.134 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:42:31.135 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:42:31.135 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.136 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.136 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.137 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.137 DEBUG:boto:Data: 2023-11-02T18:42:31.137 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.138 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.139 DEBUG:boto:Params: {} 2023-11-02T18:42:31.139 DEBUG:boto:Token: None 2023-11-02T18:42:31.140 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B2L/82h2KfFjDLu8D2ru4zdJrcE= 2023-11-02T18:42:31.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B2L/82h2KfFjDLu8D2ru4zdJrcE=', 'Content-Length': '0'} 2023-11-02T18:42:31.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c82106e0c3ed37-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:42:31.144 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:42:31.145 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:42:31.145 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.146 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.146 DEBUG:boto:Method: GET 2023-11-02T18:42:31.147 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.147 DEBUG:boto:Data: 2023-11-02T18:42:31.148 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.149 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.149 DEBUG:boto:Params: {} 2023-11-02T18:42:31.150 DEBUG:boto:Token: None 2023-11-02T18:42:31.150 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WBk2KZBwj7+tRFHRxuh/DqQr2bc= 2023-11-02T18:42:31.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WBk2KZBwj7+tRFHRxuh/DqQr2bc=', 'Content-Length': '0'} 2023-11-02T18:42:31.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025b8f4460c771839-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.155 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:42:31.156 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:42:31.156 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.157 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.157 DEBUG:boto:Method: GET 2023-11-02T18:42:31.158 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.158 DEBUG:boto:Data: 2023-11-02T18:42:31.159 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.159 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.160 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.160 DEBUG:boto:Params: {} 2023-11-02T18:42:31.161 DEBUG:boto:Token: None 2023-11-02T18:42:31.161 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:42:31.162 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WBk2KZBwj7+tRFHRxuh/DqQr2bc= 2023-11-02T18:42:31.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WBk2KZBwj7+tRFHRxuh/DqQr2bc=', 'Content-Length': '0'} 2023-11-02T18:42:31.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063ff2274f66383a2-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.206 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.207 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.207 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:42:31.208 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:42:31.208 DEBUG:boto:Method: GET 2023-11-02T18:42:31.209 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:42:31.209 DEBUG:boto:Data: 2023-11-02T18:42:31.210 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.210 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.211 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.211 DEBUG:boto:Params: {} 2023-11-02T18:42:31.211 DEBUG:boto:Token: None 2023-11-02T18:42:31.212 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.213 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b84e990a7f0b66c6-006543ed97-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.219 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.220 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.220 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.221 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:42:31.221 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:42:31.221 DEBUG:boto:Method: GET 2023-11-02T18:42:31.222 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:42:31.222 DEBUG:boto:Data: 2023-11-02T18:42:31.223 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.224 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.224 DEBUG:boto:Params: {} 2023-11-02T18:42:31.225 DEBUG:boto:Token: None 2023-11-02T18:42:31.225 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020309bcb2dc68dfe-006543ed97-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.235 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.236 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:42:31.236 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:42:31.237 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.237 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:42:31.237 DEBUG:boto:Data: 2023-11-02T18:42:31.238 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.238 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.239 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.239 DEBUG:boto:Params: {} 2023-11-02T18:42:31.240 DEBUG:boto:Token: None 2023-11-02T18:42:31.240 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj51 2023-11-02T18:42:31.241 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/uP5J93wvi4JQbxYy1GTi3B69u4= 2023-11-02T18:42:31.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/uP5J93wvi4JQbxYy1GTi3B69u4=', 'Content-Length': '0'} 2023-11-02T18:42:31.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003feab5f694f6c342-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.244 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:42:31.245 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:42:31.245 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.246 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:42:31.246 DEBUG:boto:Data: 2023-11-02T18:42:31.247 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.247 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.248 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.248 DEBUG:boto:Params: {} 2023-11-02T18:42:31.249 DEBUG:boto:Token: None 2023-11-02T18:42:31.249 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj51 2023-11-02T18:42:31.250 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/uP5J93wvi4JQbxYy1GTi3B69u4= 2023-11-02T18:42:31.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/uP5J93wvi4JQbxYy1GTi3B69u4=', 'Content-Length': '0'} 2023-11-02T18:42:31.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005203a79693d1cfb9-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:42:31.257 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:42:31.258 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:42:31.258 DEBUG:boto:Method: GET 2023-11-02T18:42:31.259 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:42:31.259 DEBUG:boto:Data: 2023-11-02T18:42:31.260 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.260 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.260 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.261 DEBUG:boto:Params: {} 2023-11-02T18:42:31.261 DEBUG:boto:Token: None 2023-11-02T18:42:31.262 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj51 2023-11-02T18:42:31.262 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I7/QTUwpK5CN4A/U91bwEF08nSc= 2023-11-02T18:42:31.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I7/QTUwpK5CN4A/U91bwEF08nSc=', 'Content-Length': '0'} 2023-11-02T18:42:31.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa15785dc6d0b7d4-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.307 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:42:31.308 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:42:31.309 DEBUG:boto:Method: GET 2023-11-02T18:42:31.309 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:42:31.310 DEBUG:boto:Data: 2023-11-02T18:42:31.310 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.311 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.311 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.311 DEBUG:boto:Params: {} 2023-11-02T18:42:31.312 DEBUG:boto:Token: None 2023-11-02T18:42:31.312 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj51 2023-11-02T18:42:31.313 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I7/QTUwpK5CN4A/U91bwEF08nSc= 2023-11-02T18:42:31.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I7/QTUwpK5CN4A/U91bwEF08nSc=', 'Content-Length': '0'} 2023-11-02T18:42:31.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000288a71d0904344a6-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.318 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.319 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.320 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:42:31.320 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:42:31.321 DEBUG:boto:Method: GET 2023-11-02T18:42:31.321 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:42:31.322 DEBUG:boto:Data: 2023-11-02T18:42:31.322 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.323 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.323 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.323 DEBUG:boto:Params: {} 2023-11-02T18:42:31.324 DEBUG:boto:Token: None 2023-11-02T18:42:31.324 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.325 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e88722d05df5f28-006543ed97-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.332 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.333 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.333 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.334 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:42:31.334 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:42:31.335 DEBUG:boto:Method: GET 2023-11-02T18:42:31.335 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:42:31.336 DEBUG:boto:Data: 2023-11-02T18:42:31.336 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.337 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.337 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.338 DEBUG:boto:Params: {} 2023-11-02T18:42:31.338 DEBUG:boto:Token: None 2023-11-02T18:42:31.339 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.339 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003354637b854dac25-006543ed97-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.349 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:42:31.351 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:42:31.351 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:42:31.351 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.352 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.352 DEBUG:boto:Method: GET 2023-11-02T18:42:31.353 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.354 DEBUG:boto:Data: 2023-11-02T18:42:31.354 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.355 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.355 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.356 DEBUG:boto:Params: {} 2023-11-02T18:42:31.356 DEBUG:boto:Token: None 2023-11-02T18:42:31.357 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.357 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jdox+Ej1EyPAbRr5FIAVsQlo2Ms= 2023-11-02T18:42:31.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jdox+Ej1EyPAbRr5FIAVsQlo2Ms=', 'Content-Length': '0'} 2023-11-02T18:42:31.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ea2f775dfbea3d6-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.361 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:42:31.362 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:42:31.362 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.363 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.363 DEBUG:boto:Method: GET 2023-11-02T18:42:31.364 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.364 DEBUG:boto:Data: 2023-11-02T18:42:31.365 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.365 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.366 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.366 DEBUG:boto:Params: {} 2023-11-02T18:42:31.367 DEBUG:boto:Token: None 2023-11-02T18:42:31.367 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.368 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jdox+Ej1EyPAbRr5FIAVsQlo2Ms= 2023-11-02T18:42:31.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jdox+Ej1EyPAbRr5FIAVsQlo2Ms=', 'Content-Length': '0'} 2023-11-02T18:42:31.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010e87434d6a90950-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.414 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:42:31.415 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:42:31.415 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.416 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.416 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.417 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.417 DEBUG:boto:Data: 2023-11-02T18:42:31.417 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.418 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.418 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.419 DEBUG:boto:Params: {} 2023-11-02T18:42:31.419 DEBUG:boto:Token: None 2023-11-02T18:42:31.420 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.420 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5YGhPs5J0tbZDFa/RSD1Xp08GEY= 2023-11-02T18:42:31.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5YGhPs5J0tbZDFa/RSD1Xp08GEY=', 'Content-Length': '0'} 2023-11-02T18:42:31.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff2f72228b22bf02-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.424 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:42:31.425 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:42:31.425 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.426 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.426 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.426 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.427 DEBUG:boto:Data: 2023-11-02T18:42:31.427 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.428 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.428 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.429 DEBUG:boto:Params: {} 2023-11-02T18:42:31.429 DEBUG:boto:Token: None 2023-11-02T18:42:31.430 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.430 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5YGhPs5J0tbZDFa/RSD1Xp08GEY= 2023-11-02T18:42:31.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5YGhPs5J0tbZDFa/RSD1Xp08GEY=', 'Content-Length': '0'} 2023-11-02T18:42:31.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135a0e6c2d69c4b7-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:42:31.434 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:42:31.435 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:42:31.435 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.436 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.436 DEBUG:boto:Method: GET 2023-11-02T18:42:31.437 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.437 DEBUG:boto:Data: 2023-11-02T18:42:31.438 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.438 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.439 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.439 DEBUG:boto:Params: {} 2023-11-02T18:42:31.440 DEBUG:boto:Token: None 2023-11-02T18:42:31.440 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.441 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jdox+Ej1EyPAbRr5FIAVsQlo2Ms= 2023-11-02T18:42:31.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jdox+Ej1EyPAbRr5FIAVsQlo2Ms=', 'Content-Length': '0'} 2023-11-02T18:42:31.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087ab3e44585f225b-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.486 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:42:31.487 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:42:31.487 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.488 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.488 DEBUG:boto:Method: GET 2023-11-02T18:42:31.489 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.489 DEBUG:boto:Data: 2023-11-02T18:42:31.489 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.490 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.490 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.491 DEBUG:boto:Params: {} 2023-11-02T18:42:31.491 DEBUG:boto:Token: None 2023-11-02T18:42:31.492 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:42:31.492 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jdox+Ej1EyPAbRr5FIAVsQlo2Ms= 2023-11-02T18:42:31.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jdox+Ej1EyPAbRr5FIAVsQlo2Ms=', 'Content-Length': '0'} 2023-11-02T18:42:31.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016e192778b27d629-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.496 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.497 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.497 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:42:31.498 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:42:31.498 DEBUG:boto:Method: GET 2023-11-02T18:42:31.499 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:42:31.499 DEBUG:boto:Data: 2023-11-02T18:42:31.499 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.500 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.500 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.501 DEBUG:boto:Params: {} 2023-11-02T18:42:31.501 DEBUG:boto:Token: None 2023-11-02T18:42:31.502 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.502 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a84053298dc632-006543ed97-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.509 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.510 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.510 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.511 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:42:31.511 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:42:31.512 DEBUG:boto:Method: GET 2023-11-02T18:42:31.512 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:42:31.513 DEBUG:boto:Data: 2023-11-02T18:42:31.513 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.514 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.514 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.514 DEBUG:boto:Params: {} 2023-11-02T18:42:31.515 DEBUG:boto:Token: None 2023-11-02T18:42:31.515 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.516 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d36e580494f32829-006543ed97-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.525 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.526 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:42:31.526 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:42:31.527 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.527 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:42:31.528 DEBUG:boto:Data: 2023-11-02T18:42:31.528 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.529 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.529 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.530 DEBUG:boto:Params: {} 2023-11-02T18:42:31.530 DEBUG:boto:Token: None 2023-11-02T18:42:31.530 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj55 2023-11-02T18:42:31.531 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e/5lsVezHpeoz3hb51+khFTS0x0= 2023-11-02T18:42:31.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e/5lsVezHpeoz3hb51+khFTS0x0=', 'Content-Length': '0'} 2023-11-02T18:42:31.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e607caf8fb21d510-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.535 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:42:31.535 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:42:31.535 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.536 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:42:31.536 DEBUG:boto:Data: 2023-11-02T18:42:31.537 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.537 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.538 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.538 DEBUG:boto:Params: {} 2023-11-02T18:42:31.539 DEBUG:boto:Token: None 2023-11-02T18:42:31.539 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj55 2023-11-02T18:42:31.540 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e/5lsVezHpeoz3hb51+khFTS0x0= 2023-11-02T18:42:31.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e/5lsVezHpeoz3hb51+khFTS0x0=', 'Content-Length': '0'} 2023-11-02T18:42:31.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b731ec9b8085023-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:42:31.544 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:42:31.544 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:42:31.545 DEBUG:boto:Method: GET 2023-11-02T18:42:31.545 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:42:31.546 DEBUG:boto:Data: 2023-11-02T18:42:31.546 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.546 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.547 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.547 DEBUG:boto:Params: {} 2023-11-02T18:42:31.548 DEBUG:boto:Token: None 2023-11-02T18:42:31.548 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj55 2023-11-02T18:42:31.549 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yAOg+lY+ivtH515YjO6+buPHOjQ= 2023-11-02T18:42:31.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yAOg+lY+ivtH515YjO6+buPHOjQ=', 'Content-Length': '0'} 2023-11-02T18:42:31.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040ed1a3918f608ab-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.553 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:42:31.553 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:42:31.554 DEBUG:boto:Method: GET 2023-11-02T18:42:31.554 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:42:31.555 DEBUG:boto:Data: 2023-11-02T18:42:31.555 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.556 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.556 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.557 DEBUG:boto:Params: {} 2023-11-02T18:42:31.557 DEBUG:boto:Token: None 2023-11-02T18:42:31.558 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj55 2023-11-02T18:42:31.558 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yAOg+lY+ivtH515YjO6+buPHOjQ= 2023-11-02T18:42:31.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yAOg+lY+ivtH515YjO6+buPHOjQ=', 'Content-Length': '0'} 2023-11-02T18:42:31.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db38dabaf6605c68-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.602 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.603 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.604 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:42:31.604 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:42:31.604 DEBUG:boto:Method: GET 2023-11-02T18:42:31.605 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:42:31.605 DEBUG:boto:Data: 2023-11-02T18:42:31.606 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.606 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.607 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.607 DEBUG:boto:Params: {} 2023-11-02T18:42:31.608 DEBUG:boto:Token: None 2023-11-02T18:42:31.608 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.609 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024047785327443af-006543ed97-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.615 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.616 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.616 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.617 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:42:31.617 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:42:31.618 DEBUG:boto:Method: GET 2023-11-02T18:42:31.618 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:42:31.619 DEBUG:boto:Data: 2023-11-02T18:42:31.619 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.620 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.620 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.621 DEBUG:boto:Params: {} 2023-11-02T18:42:31.621 DEBUG:boto:Token: None 2023-11-02T18:42:31.622 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.622 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fc871fac245cc66-006543ed97-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.631 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:42:31.632 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:42:31.633 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:42:31.633 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.634 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.634 DEBUG:boto:Method: GET 2023-11-02T18:42:31.635 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.635 DEBUG:boto:Data: 2023-11-02T18:42:31.636 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.636 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.637 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.637 DEBUG:boto:Params: {} 2023-11-02T18:42:31.638 DEBUG:boto:Token: None 2023-11-02T18:42:31.638 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.639 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vO0P4m2+4TCJ5hzo0ok5adt/ZLg= 2023-11-02T18:42:31.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vO0P4m2+4TCJ5hzo0ok5adt/ZLg=', 'Content-Length': '0'} 2023-11-02T18:42:31.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000da098851e787512-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.682 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:42:31.683 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:42:31.683 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.684 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.684 DEBUG:boto:Method: GET 2023-11-02T18:42:31.685 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.685 DEBUG:boto:Data: 2023-11-02T18:42:31.685 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.686 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.687 DEBUG:boto:Params: {} 2023-11-02T18:42:31.687 DEBUG:boto:Token: None 2023-11-02T18:42:31.688 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.688 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vO0P4m2+4TCJ5hzo0ok5adt/ZLg= 2023-11-02T18:42:31.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vO0P4m2+4TCJ5hzo0ok5adt/ZLg=', 'Content-Length': '0'} 2023-11-02T18:42:31.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d516a75231addd-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.693 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:42:31.693 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:42:31.693 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.694 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.694 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.695 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.695 DEBUG:boto:Data: 2023-11-02T18:42:31.696 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.696 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.697 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.697 DEBUG:boto:Params: {} 2023-11-02T18:42:31.697 DEBUG:boto:Token: None 2023-11-02T18:42:31.698 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.698 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gH6V8DR7AFl4yxbk1P82MPEKq6I= 2023-11-02T18:42:31.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gH6V8DR7AFl4yxbk1P82MPEKq6I=', 'Content-Length': '0'} 2023-11-02T18:42:31.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa7381dd00db73ea-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.702 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:42:31.703 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:42:31.703 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.704 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.704 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.705 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.705 DEBUG:boto:Data: 2023-11-02T18:42:31.705 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.706 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.706 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.707 DEBUG:boto:Params: {} 2023-11-02T18:42:31.707 DEBUG:boto:Token: None 2023-11-02T18:42:31.708 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.708 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gH6V8DR7AFl4yxbk1P82MPEKq6I= 2023-11-02T18:42:31.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gH6V8DR7AFl4yxbk1P82MPEKq6I=', 'Content-Length': '0'} 2023-11-02T18:42:31.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007b7958d2c98ab52-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:42:31.713 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:42:31.713 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:42:31.714 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.714 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.715 DEBUG:boto:Method: GET 2023-11-02T18:42:31.715 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.716 DEBUG:boto:Data: 2023-11-02T18:42:31.716 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.717 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.717 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.717 DEBUG:boto:Params: {} 2023-11-02T18:42:31.718 DEBUG:boto:Token: None 2023-11-02T18:42:31.718 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.719 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vO0P4m2+4TCJ5hzo0ok5adt/ZLg= 2023-11-02T18:42:31.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vO0P4m2+4TCJ5hzo0ok5adt/ZLg=', 'Content-Length': '0'} 2023-11-02T18:42:31.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6b4153bd6ef5479-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.724 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:42:31.724 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:42:31.724 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.725 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.725 DEBUG:boto:Method: GET 2023-11-02T18:42:31.726 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.726 DEBUG:boto:Data: 2023-11-02T18:42:31.727 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.727 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.728 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.728 DEBUG:boto:Params: {} 2023-11-02T18:42:31.729 DEBUG:boto:Token: None 2023-11-02T18:42:31.729 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:42:31.730 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vO0P4m2+4TCJ5hzo0ok5adt/ZLg= 2023-11-02T18:42:31.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vO0P4m2+4TCJ5hzo0ok5adt/ZLg=', 'Content-Length': '0'} 2023-11-02T18:42:31.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008de14faecee9f2a9-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.774 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.775 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.775 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:42:31.776 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:42:31.776 DEBUG:boto:Method: GET 2023-11-02T18:42:31.777 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:42:31.777 DEBUG:boto:Data: 2023-11-02T18:42:31.778 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.778 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.779 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.779 DEBUG:boto:Params: {} 2023-11-02T18:42:31.780 DEBUG:boto:Token: None 2023-11-02T18:42:31.780 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.781 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012ad8d7db592b53b-006543ed97-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.787 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.788 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.789 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.789 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:42:31.789 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:42:31.790 DEBUG:boto:Method: GET 2023-11-02T18:42:31.790 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:42:31.791 DEBUG:boto:Data: 2023-11-02T18:42:31.791 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.792 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.792 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.792 DEBUG:boto:Params: {} 2023-11-02T18:42:31.793 DEBUG:boto:Token: None 2023-11-02T18:42:31.793 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.794 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2fca441e518ba65-006543ed97-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.804 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.804 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:42:31.805 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:42:31.805 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.806 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:42:31.806 DEBUG:boto:Data: 2023-11-02T18:42:31.807 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.807 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.807 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.808 DEBUG:boto:Params: {} 2023-11-02T18:42:31.808 DEBUG:boto:Token: None 2023-11-02T18:42:31.809 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj57 2023-11-02T18:42:31.809 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrJnUV5K5iISsIVrJZLngruCO1A= 2023-11-02T18:42:31.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrJnUV5K5iISsIVrJZLngruCO1A=', 'Content-Length': '0'} 2023-11-02T18:42:31.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093668c8915911d56-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.813 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:42:31.813 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:42:31.814 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.814 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:42:31.814 DEBUG:boto:Data: 2023-11-02T18:42:31.815 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.815 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.816 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.816 DEBUG:boto:Params: {} 2023-11-02T18:42:31.817 DEBUG:boto:Token: None 2023-11-02T18:42:31.817 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj57 2023-11-02T18:42:31.818 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GrJnUV5K5iISsIVrJZLngruCO1A= 2023-11-02T18:42:31.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GrJnUV5K5iISsIVrJZLngruCO1A=', 'Content-Length': '0'} 2023-11-02T18:42:31.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c35bc33a6d7619e-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:42:31.822 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:42:31.822 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:42:31.823 DEBUG:boto:Method: GET 2023-11-02T18:42:31.823 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:42:31.824 DEBUG:boto:Data: 2023-11-02T18:42:31.824 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.825 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.825 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.826 DEBUG:boto:Params: {} 2023-11-02T18:42:31.826 DEBUG:boto:Token: None 2023-11-02T18:42:31.827 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj57 2023-11-02T18:42:31.827 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bdDtq44Th8cRsVd5qS1PYEMdFM0= 2023-11-02T18:42:31.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bdDtq44Th8cRsVd5qS1PYEMdFM0=', 'Content-Length': '0'} 2023-11-02T18:42:31.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000df7f8afedf042da-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.831 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:42:31.832 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:42:31.832 DEBUG:boto:Method: GET 2023-11-02T18:42:31.833 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:42:31.833 DEBUG:boto:Data: 2023-11-02T18:42:31.834 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.834 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.835 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.835 DEBUG:boto:Params: {} 2023-11-02T18:42:31.835 DEBUG:boto:Token: None 2023-11-02T18:42:31.836 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj57 2023-11-02T18:42:31.837 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bdDtq44Th8cRsVd5qS1PYEMdFM0= 2023-11-02T18:42:31.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bdDtq44Th8cRsVd5qS1PYEMdFM0=', 'Content-Length': '0'} 2023-11-02T18:42:31.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005deea682c63291d0-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.841 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.842 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.842 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:42:31.842 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:42:31.843 DEBUG:boto:Method: GET 2023-11-02T18:42:31.843 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:42:31.844 DEBUG:boto:Data: 2023-11-02T18:42:31.844 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.845 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.845 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.846 DEBUG:boto:Params: {} 2023-11-02T18:42:31.846 DEBUG:boto:Token: None 2023-11-02T18:42:31.847 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.847 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f939c2069908767-006543ed97-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.894 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.895 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:31.896 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:31.896 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:42:31.897 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:42:31.897 DEBUG:boto:Method: GET 2023-11-02T18:42:31.897 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:42:31.898 DEBUG:boto:Data: 2023-11-02T18:42:31.898 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.899 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.899 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.900 DEBUG:boto:Params: {} 2023-11-02T18:42:31.900 DEBUG:boto:Token: None 2023-11-02T18:42:31.901 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/ 2023-11-02T18:42:31.901 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U= 2023-11-02T18:42:31.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2rNdJMfSyVsf4BFAKs7tmAY6p/U=', 'Content-Length': '0'} 2023-11-02T18:42:31.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e07e520d888458cf-006543ed97-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.911 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:31.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:42:31.912 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:42:31.913 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:42:31.913 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.914 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.914 DEBUG:boto:Method: GET 2023-11-02T18:42:31.915 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.915 DEBUG:boto:Data: 2023-11-02T18:42:31.916 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.916 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.917 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.917 DEBUG:boto:Params: {} 2023-11-02T18:42:31.918 DEBUG:boto:Token: None 2023-11-02T18:42:31.918 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.919 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:juNlY9aEiFDC9Ik1yNxtPLFgrZE= 2023-11-02T18:42:31.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:juNlY9aEiFDC9Ik1yNxtPLFgrZE=', 'Content-Length': '0'} 2023-11-02T18:42:31.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d1eb039df0e6759-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.923 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:42:31.923 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:42:31.924 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.924 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.925 DEBUG:boto:Method: GET 2023-11-02T18:42:31.925 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.926 DEBUG:boto:Data: 2023-11-02T18:42:31.927 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.927 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.927 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.928 DEBUG:boto:Params: {} 2023-11-02T18:42:31.928 DEBUG:boto:Token: None 2023-11-02T18:42:31.929 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.929 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:juNlY9aEiFDC9Ik1yNxtPLFgrZE= 2023-11-02T18:42:31.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:juNlY9aEiFDC9Ik1yNxtPLFgrZE=', 'Content-Length': '0'} 2023-11-02T18:42:31.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfeb689c3097c8e9-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.974 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:42:31.975 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:42:31.975 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.976 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.976 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.976 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.977 DEBUG:boto:Data: 2023-11-02T18:42:31.977 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.978 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.978 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.979 DEBUG:boto:Params: {} 2023-11-02T18:42:31.979 DEBUG:boto:Token: None 2023-11-02T18:42:31.980 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.980 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JbUxIQl8wkERoLo+ZnVq8mPt9f4= 2023-11-02T18:42:31.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JbUxIQl8wkERoLo+ZnVq8mPt9f4=', 'Content-Length': '0'} 2023-11-02T18:42:31.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1616d840136e6e1-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.985 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:42:31.985 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:42:31.985 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.986 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.986 DEBUG:boto:Method: HEAD 2023-11-02T18:42:31.987 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.987 DEBUG:boto:Data: 2023-11-02T18:42:31.988 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.988 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:31.989 DEBUG:boto:Port: 8000 2023-11-02T18:42:31.989 DEBUG:boto:Params: {} 2023-11-02T18:42:31.989 DEBUG:boto:Token: None 2023-11-02T18:42:31.990 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:31 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.990 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JbUxIQl8wkERoLo+ZnVq8mPt9f4= 2023-11-02T18:42:31.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JbUxIQl8wkERoLo+ZnVq8mPt9f4=', 'Content-Length': '0'} 2023-11-02T18:42:31.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b52ae332d0ab8cc3-006543ed97-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:31.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:42:31.995 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:42:31.995 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:42:31.996 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.996 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.996 DEBUG:boto:Method: GET 2023-11-02T18:42:31.997 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:42:31.997 DEBUG:boto:Data: 2023-11-02T18:42:31.998 DEBUG:boto:Headers: {} 2023-11-02T18:42:31.998 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:31.999 DEBUG:boto:Port: 8001 2023-11-02T18:42:31.999 DEBUG:boto:Params: {} 2023-11-02T18:42:32.000 DEBUG:boto:Token: None 2023-11-02T18:42:32.000 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:42:32.001 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H6/LXNPiv5lh/qXN/Og7LQDEC9E= 2023-11-02T18:42:32.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H6/LXNPiv5lh/qXN/Og7LQDEC9E=', 'Content-Length': '0'} 2023-11-02T18:42:32.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040084cf141c6d8f0-006543ed97-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.005 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:42:32.005 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:42:32.006 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:42:32.006 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:42:32.007 DEBUG:boto:Method: GET 2023-11-02T18:42:32.007 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:42:32.008 DEBUG:boto:Data: 2023-11-02T18:42:32.008 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.009 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.009 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.009 DEBUG:boto:Params: {} 2023-11-02T18:42:32.010 DEBUG:boto:Token: None 2023-11-02T18:42:32.010 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:42:32.011 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H6/LXNPiv5lh/qXN/Og7LQDEC9E= 2023-11-02T18:42:32.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H6/LXNPiv5lh/qXN/Og7LQDEC9E=', 'Content-Length': '0'} 2023-11-02T18:42:32.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d72935a6486122d1-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.015 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.016 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.016 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:42:32.016 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:42:32.017 DEBUG:boto:Method: GET 2023-11-02T18:42:32.017 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:42:32.018 DEBUG:boto:Data: 2023-11-02T18:42:32.018 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.019 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.019 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.020 DEBUG:boto:Params: {} 2023-11-02T18:42:32.020 DEBUG:boto:Token: None 2023-11-02T18:42:32.021 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.021 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3b06a6c1776bc4f-006543ed98-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.074 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.081 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.081 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.082 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:42:32.082 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:42:32.083 DEBUG:boto:Method: GET 2023-11-02T18:42:32.083 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:42:32.084 DEBUG:boto:Data: 2023-11-02T18:42:32.084 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.084 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.085 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.085 DEBUG:boto:Params: {} 2023-11-02T18:42:32.086 DEBUG:boto:Token: None 2023-11-02T18:42:32.086 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.087 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064d6f5ff26064863-006543ed98-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.096 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.097 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:42:32.097 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:42:32.098 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.098 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:42:32.099 DEBUG:boto:Data: 2023-11-02T18:42:32.099 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.100 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.100 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.100 DEBUG:boto:Params: {} 2023-11-02T18:42:32.101 DEBUG:boto:Token: None 2023-11-02T18:42:32.101 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj58 2023-11-02T18:42:32.102 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hgPOeFbeJNpcSKKvCnlHjD/VI2s= 2023-11-02T18:42:32.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hgPOeFbeJNpcSKKvCnlHjD/VI2s=', 'Content-Length': '0'} 2023-11-02T18:42:32.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094edade0acb0ca58-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.106 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:42:32.106 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:42:32.107 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.107 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:42:32.108 DEBUG:boto:Data: 2023-11-02T18:42:32.108 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.108 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.109 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.109 DEBUG:boto:Params: {} 2023-11-02T18:42:32.110 DEBUG:boto:Token: None 2023-11-02T18:42:32.110 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj58 2023-11-02T18:42:32.111 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hgPOeFbeJNpcSKKvCnlHjD/VI2s= 2023-11-02T18:42:32.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hgPOeFbeJNpcSKKvCnlHjD/VI2s=', 'Content-Length': '0'} 2023-11-02T18:42:32.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023a8473feb1abeb9-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:42:32.115 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:42:32.116 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:42:32.116 DEBUG:boto:Method: GET 2023-11-02T18:42:32.117 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:42:32.117 DEBUG:boto:Data: 2023-11-02T18:42:32.118 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.118 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.119 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.119 DEBUG:boto:Params: {} 2023-11-02T18:42:32.120 DEBUG:boto:Token: None 2023-11-02T18:42:32.120 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj58 2023-11-02T18:42:32.121 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hlVDu/rUTdNNizEvOnS3+Ed7REQ= 2023-11-02T18:42:32.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hlVDu/rUTdNNizEvOnS3+Ed7REQ=', 'Content-Length': '0'} 2023-11-02T18:42:32.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057612d49fc45559d-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.125 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:42:32.125 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:42:32.126 DEBUG:boto:Method: GET 2023-11-02T18:42:32.126 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:42:32.127 DEBUG:boto:Data: 2023-11-02T18:42:32.127 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.128 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.128 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.129 DEBUG:boto:Params: {} 2023-11-02T18:42:32.129 DEBUG:boto:Token: None 2023-11-02T18:42:32.130 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj58 2023-11-02T18:42:32.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hlVDu/rUTdNNizEvOnS3+Ed7REQ= 2023-11-02T18:42:32.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hlVDu/rUTdNNizEvOnS3+Ed7REQ=', 'Content-Length': '0'} 2023-11-02T18:42:32.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6db3a460a3372fc-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.134 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.135 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.135 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:42:32.136 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:42:32.136 DEBUG:boto:Method: GET 2023-11-02T18:42:32.136 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:42:32.137 DEBUG:boto:Data: 2023-11-02T18:42:32.137 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.138 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.139 DEBUG:boto:Params: {} 2023-11-02T18:42:32.139 DEBUG:boto:Token: None 2023-11-02T18:42:32.140 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b703e64fe616ea4-006543ed98-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.146 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.147 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.147 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.148 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:42:32.148 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:42:32.149 DEBUG:boto:Method: GET 2023-11-02T18:42:32.149 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:42:32.150 DEBUG:boto:Data: 2023-11-02T18:42:32.150 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.151 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.151 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.152 DEBUG:boto:Params: {} 2023-11-02T18:42:32.152 DEBUG:boto:Token: None 2023-11-02T18:42:32.153 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.153 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5fe60fd072cec1f-006543ed98-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.162 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:42:32.164 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:42:32.164 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:42:32.164 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.165 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.165 DEBUG:boto:Method: GET 2023-11-02T18:42:32.166 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.166 DEBUG:boto:Data: 2023-11-02T18:42:32.167 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.167 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.168 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.168 DEBUG:boto:Params: {} 2023-11-02T18:42:32.169 DEBUG:boto:Token: None 2023-11-02T18:42:32.169 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.170 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DrhY5QBxJ0E5bgDvReFTIO7bOW4= 2023-11-02T18:42:32.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DrhY5QBxJ0E5bgDvReFTIO7bOW4=', 'Content-Length': '0'} 2023-11-02T18:42:32.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a3a0b8368ad53b7-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.173 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:42:32.174 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:42:32.174 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.175 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.175 DEBUG:boto:Method: GET 2023-11-02T18:42:32.176 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.176 DEBUG:boto:Data: 2023-11-02T18:42:32.176 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.177 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.177 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.178 DEBUG:boto:Params: {} 2023-11-02T18:42:32.178 DEBUG:boto:Token: None 2023-11-02T18:42:32.179 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.179 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DrhY5QBxJ0E5bgDvReFTIO7bOW4= 2023-11-02T18:42:32.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DrhY5QBxJ0E5bgDvReFTIO7bOW4=', 'Content-Length': '0'} 2023-11-02T18:42:32.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b01ba18b0d03c6-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.226 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:42:32.227 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:42:32.227 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.228 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.228 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.229 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.229 DEBUG:boto:Data: 2023-11-02T18:42:32.230 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.230 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.231 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.231 DEBUG:boto:Params: {} 2023-11-02T18:42:32.232 DEBUG:boto:Token: None 2023-11-02T18:42:32.232 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.233 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccS+Q4sm7qLja7+DsC8aw4eQbp4= 2023-11-02T18:42:32.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccS+Q4sm7qLja7+DsC8aw4eQbp4=', 'Content-Length': '0'} 2023-11-02T18:42:32.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d18f1858ccd5fc2b-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.237 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:42:32.237 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:42:32.238 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.238 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.239 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.239 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.239 DEBUG:boto:Data: 2023-11-02T18:42:32.240 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.240 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.241 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.241 DEBUG:boto:Params: {} 2023-11-02T18:42:32.241 DEBUG:boto:Token: None 2023-11-02T18:42:32.242 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccS+Q4sm7qLja7+DsC8aw4eQbp4= 2023-11-02T18:42:32.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccS+Q4sm7qLja7+DsC8aw4eQbp4=', 'Content-Length': '0'} 2023-11-02T18:42:32.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082f48eb97fbc7ffd-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:42:32.247 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:42:32.247 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:42:32.248 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.248 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.249 DEBUG:boto:Method: GET 2023-11-02T18:42:32.249 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.250 DEBUG:boto:Data: 2023-11-02T18:42:32.253 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.254 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.255 DEBUG:boto:Params: {} 2023-11-02T18:42:32.255 DEBUG:boto:Token: None 2023-11-02T18:42:32.256 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.256 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DrhY5QBxJ0E5bgDvReFTIO7bOW4= 2023-11-02T18:42:32.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DrhY5QBxJ0E5bgDvReFTIO7bOW4=', 'Content-Length': '0'} 2023-11-02T18:42:32.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d959d325cb21128-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.260 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:42:32.261 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:42:32.261 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.262 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.262 DEBUG:boto:Method: GET 2023-11-02T18:42:32.263 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.263 DEBUG:boto:Data: 2023-11-02T18:42:32.263 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.264 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.264 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.265 DEBUG:boto:Params: {} 2023-11-02T18:42:32.265 DEBUG:boto:Token: None 2023-11-02T18:42:32.266 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:42:32.266 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DrhY5QBxJ0E5bgDvReFTIO7bOW4= 2023-11-02T18:42:32.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DrhY5QBxJ0E5bgDvReFTIO7bOW4=', 'Content-Length': '0'} 2023-11-02T18:42:32.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005042fc197a6193a-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.271 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.271 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.272 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:42:32.272 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:42:32.272 DEBUG:boto:Method: GET 2023-11-02T18:42:32.273 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:42:32.273 DEBUG:boto:Data: 2023-11-02T18:42:32.274 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.274 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.275 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.275 DEBUG:boto:Params: {} 2023-11-02T18:42:32.276 DEBUG:boto:Token: None 2023-11-02T18:42:32.276 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.277 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000803048ac432754bf-006543ed98-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.326 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.327 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.328 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.328 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:42:32.329 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:42:32.329 DEBUG:boto:Method: GET 2023-11-02T18:42:32.330 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:42:32.330 DEBUG:boto:Data: 2023-11-02T18:42:32.331 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.331 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.332 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.332 DEBUG:boto:Params: {} 2023-11-02T18:42:32.333 DEBUG:boto:Token: None 2023-11-02T18:42:32.333 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.334 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e08b28681dea9e85-006543ed98-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.342 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.342 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:42:32.343 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:42:32.343 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.344 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:42:32.344 DEBUG:boto:Data: 2023-11-02T18:42:32.345 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.345 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.346 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.346 DEBUG:boto:Params: {} 2023-11-02T18:42:32.347 DEBUG:boto:Token: None 2023-11-02T18:42:32.347 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj68 2023-11-02T18:42:32.348 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gMSsIQonSmH75SuOEY9xIT7u5D8= 2023-11-02T18:42:32.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gMSsIQonSmH75SuOEY9xIT7u5D8=', 'Content-Length': '0'} 2023-11-02T18:42:32.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089fdd53cc77ff212-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.351 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:42:32.352 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:42:32.352 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.353 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:42:32.353 DEBUG:boto:Data: 2023-11-02T18:42:32.354 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.354 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.354 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.355 DEBUG:boto:Params: {} 2023-11-02T18:42:32.355 DEBUG:boto:Token: None 2023-11-02T18:42:32.356 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj68 2023-11-02T18:42:32.356 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gMSsIQonSmH75SuOEY9xIT7u5D8= 2023-11-02T18:42:32.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gMSsIQonSmH75SuOEY9xIT7u5D8=', 'Content-Length': '0'} 2023-11-02T18:42:32.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27590429ed17525-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:42:32.361 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:42:32.361 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:42:32.362 DEBUG:boto:Method: GET 2023-11-02T18:42:32.362 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:42:32.363 DEBUG:boto:Data: 2023-11-02T18:42:32.363 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.364 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.364 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.365 DEBUG:boto:Params: {} 2023-11-02T18:42:32.365 DEBUG:boto:Token: None 2023-11-02T18:42:32.365 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj68 2023-11-02T18:42:32.366 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:954ePpy+w55phx8ILFt2+WpKxDo= 2023-11-02T18:42:32.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:954ePpy+w55phx8ILFt2+WpKxDo=', 'Content-Length': '0'} 2023-11-02T18:42:32.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2d92b8dd41305b0-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.370 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:42:32.371 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:42:32.371 DEBUG:boto:Method: GET 2023-11-02T18:42:32.371 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:42:32.372 DEBUG:boto:Data: 2023-11-02T18:42:32.372 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.373 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.373 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.374 DEBUG:boto:Params: {} 2023-11-02T18:42:32.374 DEBUG:boto:Token: None 2023-11-02T18:42:32.375 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj68 2023-11-02T18:42:32.375 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:954ePpy+w55phx8ILFt2+WpKxDo= 2023-11-02T18:42:32.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:954ePpy+w55phx8ILFt2+WpKxDo=', 'Content-Length': '0'} 2023-11-02T18:42:32.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000424db6ee6f13f566-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.422 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.423 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.423 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:42:32.424 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:42:32.424 DEBUG:boto:Method: GET 2023-11-02T18:42:32.425 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:42:32.425 DEBUG:boto:Data: 2023-11-02T18:42:32.426 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.426 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.427 DEBUG:boto:Params: {} 2023-11-02T18:42:32.427 DEBUG:boto:Token: None 2023-11-02T18:42:32.428 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.429 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c03b4242e40c8463-006543ed98-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.435 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.436 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.436 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.436 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:42:32.437 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:42:32.437 DEBUG:boto:Method: GET 2023-11-02T18:42:32.438 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:42:32.438 DEBUG:boto:Data: 2023-11-02T18:42:32.439 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.439 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.440 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.440 DEBUG:boto:Params: {} 2023-11-02T18:42:32.441 DEBUG:boto:Token: None 2023-11-02T18:42:32.441 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.442 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c4296259e15ec28-006543ed98-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.451 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:42:32.452 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:42:32.453 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:42:32.453 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.454 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.454 DEBUG:boto:Method: GET 2023-11-02T18:42:32.454 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.455 DEBUG:boto:Data: 2023-11-02T18:42:32.455 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.456 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.456 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.457 DEBUG:boto:Params: {} 2023-11-02T18:42:32.457 DEBUG:boto:Token: None 2023-11-02T18:42:32.458 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.458 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oBlEGvgUkc7VMu1V21UiwjUeefc= 2023-11-02T18:42:32.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oBlEGvgUkc7VMu1V21UiwjUeefc=', 'Content-Length': '0'} 2023-11-02T18:42:32.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9ebfb85dff008d0-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.506 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:42:32.507 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:42:32.507 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.508 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.508 DEBUG:boto:Method: GET 2023-11-02T18:42:32.509 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.509 DEBUG:boto:Data: 2023-11-02T18:42:32.510 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.510 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.511 DEBUG:boto:Params: {} 2023-11-02T18:42:32.511 DEBUG:boto:Token: None 2023-11-02T18:42:32.512 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oBlEGvgUkc7VMu1V21UiwjUeefc= 2023-11-02T18:42:32.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oBlEGvgUkc7VMu1V21UiwjUeefc=', 'Content-Length': '0'} 2023-11-02T18:42:32.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f46a7a6f238b611-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.517 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:42:32.517 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:42:32.518 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.518 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.519 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.519 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.520 DEBUG:boto:Data: 2023-11-02T18:42:32.520 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.521 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.521 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.522 DEBUG:boto:Params: {} 2023-11-02T18:42:32.522 DEBUG:boto:Token: None 2023-11-02T18:42:32.523 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.523 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Hc2ItT2LVjrbO2QB3mlu0nwBgic= 2023-11-02T18:42:32.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Hc2ItT2LVjrbO2QB3mlu0nwBgic=', 'Content-Length': '0'} 2023-11-02T18:42:32.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1d282e6dfd9737c-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.527 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:42:32.528 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:42:32.528 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.528 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.529 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.529 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.530 DEBUG:boto:Data: 2023-11-02T18:42:32.530 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.531 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.531 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.532 DEBUG:boto:Params: {} 2023-11-02T18:42:32.532 DEBUG:boto:Token: None 2023-11-02T18:42:32.533 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.533 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Hc2ItT2LVjrbO2QB3mlu0nwBgic= 2023-11-02T18:42:32.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Hc2ItT2LVjrbO2QB3mlu0nwBgic=', 'Content-Length': '0'} 2023-11-02T18:42:32.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000491fa5ba54f5675d-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:42:32.537 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:42:32.538 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:42:32.538 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.539 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.539 DEBUG:boto:Method: GET 2023-11-02T18:42:32.540 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.540 DEBUG:boto:Data: 2023-11-02T18:42:32.541 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.542 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.542 DEBUG:boto:Params: {} 2023-11-02T18:42:32.542 DEBUG:boto:Token: None 2023-11-02T18:42:32.543 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.543 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oBlEGvgUkc7VMu1V21UiwjUeefc= 2023-11-02T18:42:32.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oBlEGvgUkc7VMu1V21UiwjUeefc=', 'Content-Length': '0'} 2023-11-02T18:42:32.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdfb242b8e1e1ad8-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.548 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:42:32.548 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:42:32.548 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.549 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.549 DEBUG:boto:Method: GET 2023-11-02T18:42:32.550 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.550 DEBUG:boto:Data: 2023-11-02T18:42:32.551 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.551 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.552 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.552 DEBUG:boto:Params: {} 2023-11-02T18:42:32.553 DEBUG:boto:Token: None 2023-11-02T18:42:32.553 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:42:32.554 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oBlEGvgUkc7VMu1V21UiwjUeefc= 2023-11-02T18:42:32.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oBlEGvgUkc7VMu1V21UiwjUeefc=', 'Content-Length': '0'} 2023-11-02T18:42:32.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9d8dba6911973a1-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.558 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.559 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.559 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:42:32.560 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:42:32.560 DEBUG:boto:Method: GET 2023-11-02T18:42:32.561 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:42:32.561 DEBUG:boto:Data: 2023-11-02T18:42:32.561 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.562 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.563 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.563 DEBUG:boto:Params: {} 2023-11-02T18:42:32.563 DEBUG:boto:Token: None 2023-11-02T18:42:32.564 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.564 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff302d11eb011518-006543ed98-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.614 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.615 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.616 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.616 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:42:32.616 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:42:32.617 DEBUG:boto:Method: GET 2023-11-02T18:42:32.617 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:42:32.618 DEBUG:boto:Data: 2023-11-02T18:42:32.618 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.619 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.620 DEBUG:boto:Params: {} 2023-11-02T18:42:32.620 DEBUG:boto:Token: None 2023-11-02T18:42:32.621 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d966d17b74ee2faf-006543ed98-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.631 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.632 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:42:32.632 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:42:32.633 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.633 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:42:32.634 DEBUG:boto:Data: 2023-11-02T18:42:32.634 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.635 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.635 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.636 DEBUG:boto:Params: {} 2023-11-02T18:42:32.636 DEBUG:boto:Token: None 2023-11-02T18:42:32.637 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj69 2023-11-02T18:42:32.637 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2SGzh2efIUiIHsaA8+HBrDKAMjI= 2023-11-02T18:42:32.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2SGzh2efIUiIHsaA8+HBrDKAMjI=', 'Content-Length': '0'} 2023-11-02T18:42:32.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007beb5bed4214e28-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.641 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:42:32.641 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:42:32.642 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.642 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:42:32.643 DEBUG:boto:Data: 2023-11-02T18:42:32.643 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.643 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.644 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.644 DEBUG:boto:Params: {} 2023-11-02T18:42:32.645 DEBUG:boto:Token: None 2023-11-02T18:42:32.645 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj69 2023-11-02T18:42:32.646 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2SGzh2efIUiIHsaA8+HBrDKAMjI= 2023-11-02T18:42:32.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2SGzh2efIUiIHsaA8+HBrDKAMjI=', 'Content-Length': '0'} 2023-11-02T18:42:32.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019b0643230432a19-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:42:32.650 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:42:32.650 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:42:32.651 DEBUG:boto:Method: GET 2023-11-02T18:42:32.651 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:42:32.652 DEBUG:boto:Data: 2023-11-02T18:42:32.652 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.652 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.653 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.653 DEBUG:boto:Params: {} 2023-11-02T18:42:32.654 DEBUG:boto:Token: None 2023-11-02T18:42:32.654 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj69 2023-11-02T18:42:32.655 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fgdTZxt30GIBD/WdpwLkOq3iJMk= 2023-11-02T18:42:32.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fgdTZxt30GIBD/WdpwLkOq3iJMk=', 'Content-Length': '0'} 2023-11-02T18:42:32.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f53705fc89fbbe26-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.658 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:42:32.659 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:42:32.659 DEBUG:boto:Method: GET 2023-11-02T18:42:32.660 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:42:32.660 DEBUG:boto:Data: 2023-11-02T18:42:32.661 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.661 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.662 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.662 DEBUG:boto:Params: {} 2023-11-02T18:42:32.663 DEBUG:boto:Token: None 2023-11-02T18:42:32.663 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj69 2023-11-02T18:42:32.664 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fgdTZxt30GIBD/WdpwLkOq3iJMk= 2023-11-02T18:42:32.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fgdTZxt30GIBD/WdpwLkOq3iJMk=', 'Content-Length': '0'} 2023-11-02T18:42:32.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050e4cb6a763ba5db-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.714 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.715 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.715 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:42:32.716 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:42:32.716 DEBUG:boto:Method: GET 2023-11-02T18:42:32.717 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:42:32.717 DEBUG:boto:Data: 2023-11-02T18:42:32.718 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.718 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.719 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.719 DEBUG:boto:Params: {} 2023-11-02T18:42:32.720 DEBUG:boto:Token: None 2023-11-02T18:42:32.720 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092467d825a9c70ce-006543ed98-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.727 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.728 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.729 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.729 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:42:32.729 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:42:32.730 DEBUG:boto:Method: GET 2023-11-02T18:42:32.730 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:42:32.731 DEBUG:boto:Data: 2023-11-02T18:42:32.731 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.732 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.732 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.733 DEBUG:boto:Params: {} 2023-11-02T18:42:32.733 DEBUG:boto:Token: None 2023-11-02T18:42:32.734 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.734 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006776538974f5349d-006543ed98-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.743 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:42:32.744 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:42:32.745 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:42:32.745 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.746 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.746 DEBUG:boto:Method: GET 2023-11-02T18:42:32.747 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.747 DEBUG:boto:Data: 2023-11-02T18:42:32.748 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.748 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.748 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.749 DEBUG:boto:Params: {} 2023-11-02T18:42:32.749 DEBUG:boto:Token: None 2023-11-02T18:42:32.750 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.750 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b/pc6lpXxFhtRmEUnRlJpSpEOTI= 2023-11-02T18:42:32.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b/pc6lpXxFhtRmEUnRlJpSpEOTI=', 'Content-Length': '0'} 2023-11-02T18:42:32.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef720cf5139ce590-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.754 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:42:32.755 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:42:32.755 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.756 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.756 DEBUG:boto:Method: GET 2023-11-02T18:42:32.757 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.757 DEBUG:boto:Data: 2023-11-02T18:42:32.758 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.758 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.758 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.759 DEBUG:boto:Params: {} 2023-11-02T18:42:32.760 DEBUG:boto:Token: None 2023-11-02T18:42:32.760 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.761 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b/pc6lpXxFhtRmEUnRlJpSpEOTI= 2023-11-02T18:42:32.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b/pc6lpXxFhtRmEUnRlJpSpEOTI=', 'Content-Length': '0'} 2023-11-02T18:42:32.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077f9a5c76e00c579-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.765 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:42:32.766 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:42:32.766 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.767 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.767 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.768 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.768 DEBUG:boto:Data: 2023-11-02T18:42:32.769 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.769 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.770 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.770 DEBUG:boto:Params: {} 2023-11-02T18:42:32.771 DEBUG:boto:Token: None 2023-11-02T18:42:32.771 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.772 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cd6LKsxP36vrEKvsBdLZ7ftFJOk= 2023-11-02T18:42:32.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cd6LKsxP36vrEKvsBdLZ7ftFJOk=', 'Content-Length': '0'} 2023-11-02T18:42:32.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000290565589e6c4274-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.776 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:42:32.776 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:42:32.777 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.777 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.777 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.778 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.778 DEBUG:boto:Data: 2023-11-02T18:42:32.779 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.779 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.780 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.780 DEBUG:boto:Params: {} 2023-11-02T18:42:32.781 DEBUG:boto:Token: None 2023-11-02T18:42:32.781 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.782 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cd6LKsxP36vrEKvsBdLZ7ftFJOk= 2023-11-02T18:42:32.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cd6LKsxP36vrEKvsBdLZ7ftFJOk=', 'Content-Length': '0'} 2023-11-02T18:42:32.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009043bf36de70ca6b-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:42:32.786 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:42:32.787 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:42:32.787 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.788 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.788 DEBUG:boto:Method: GET 2023-11-02T18:42:32.789 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.789 DEBUG:boto:Data: 2023-11-02T18:42:32.790 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.791 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.791 DEBUG:boto:Params: {} 2023-11-02T18:42:32.792 DEBUG:boto:Token: None 2023-11-02T18:42:32.792 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b/pc6lpXxFhtRmEUnRlJpSpEOTI= 2023-11-02T18:42:32.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b/pc6lpXxFhtRmEUnRlJpSpEOTI=', 'Content-Length': '0'} 2023-11-02T18:42:32.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060f5b5d143122d1c-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.838 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:42:32.839 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:42:32.840 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.840 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.841 DEBUG:boto:Method: GET 2023-11-02T18:42:32.842 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.842 DEBUG:boto:Data: 2023-11-02T18:42:32.843 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.843 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.844 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.844 DEBUG:boto:Params: {} 2023-11-02T18:42:32.845 DEBUG:boto:Token: None 2023-11-02T18:42:32.845 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:42:32.846 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b/pc6lpXxFhtRmEUnRlJpSpEOTI= 2023-11-02T18:42:32.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b/pc6lpXxFhtRmEUnRlJpSpEOTI=', 'Content-Length': '0'} 2023-11-02T18:42:32.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069b9ccc3237eae0e-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.851 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.851 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.852 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:42:32.852 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:42:32.853 DEBUG:boto:Method: GET 2023-11-02T18:42:32.853 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:42:32.853 DEBUG:boto:Data: 2023-11-02T18:42:32.854 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.854 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.856 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.856 DEBUG:boto:Params: {} 2023-11-02T18:42:32.856 DEBUG:boto:Token: None 2023-11-02T18:42:32.857 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.857 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eb5afb3d96f4dc6-006543ed98-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.864 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.864 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.865 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.866 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:42:32.867 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:42:32.867 DEBUG:boto:Method: GET 2023-11-02T18:42:32.868 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:42:32.868 DEBUG:boto:Data: 2023-11-02T18:42:32.868 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.869 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.869 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.870 DEBUG:boto:Params: {} 2023-11-02T18:42:32.870 DEBUG:boto:Token: None 2023-11-02T18:42:32.871 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.871 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000488eefa59df4475c-006543ed98-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.882 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.883 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:42:32.883 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:42:32.884 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.884 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:42:32.885 DEBUG:boto:Data: 2023-11-02T18:42:32.885 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.885 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.886 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.886 DEBUG:boto:Params: {} 2023-11-02T18:42:32.887 DEBUG:boto:Token: None 2023-11-02T18:42:32.887 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj76 2023-11-02T18:42:32.889 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TWVFyj+lJZPdmxWgx1mRiH+dsmM= 2023-11-02T18:42:32.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TWVFyj+lJZPdmxWgx1mRiH+dsmM=', 'Content-Length': '0'} 2023-11-02T18:42:32.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fa147cd8c5f48c5-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.893 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:42:32.893 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:42:32.894 DEBUG:boto:Method: HEAD 2023-11-02T18:42:32.894 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:42:32.895 DEBUG:boto:Data: 2023-11-02T18:42:32.895 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.896 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.896 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.896 DEBUG:boto:Params: {} 2023-11-02T18:42:32.897 DEBUG:boto:Token: None 2023-11-02T18:42:32.898 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj76 2023-11-02T18:42:32.898 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TWVFyj+lJZPdmxWgx1mRiH+dsmM= 2023-11-02T18:42:32.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TWVFyj+lJZPdmxWgx1mRiH+dsmM=', 'Content-Length': '0'} 2023-11-02T18:42:32.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ee4d75a6862155-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:42:32.902 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:42:32.903 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:42:32.903 DEBUG:boto:Method: GET 2023-11-02T18:42:32.904 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:42:32.905 DEBUG:boto:Data: 2023-11-02T18:42:32.905 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.906 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.907 DEBUG:boto:Params: {} 2023-11-02T18:42:32.907 DEBUG:boto:Token: None 2023-11-02T18:42:32.908 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj76 2023-11-02T18:42:32.908 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CqoUK8yOn/yMBTIiufACbWucv3s= 2023-11-02T18:42:32.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CqoUK8yOn/yMBTIiufACbWucv3s=', 'Content-Length': '0'} 2023-11-02T18:42:32.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059289cbf8df4afe7-006543ed98-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.913 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:42:32.913 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:42:32.914 DEBUG:boto:Method: GET 2023-11-02T18:42:32.914 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:42:32.915 DEBUG:boto:Data: 2023-11-02T18:42:32.915 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.915 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.916 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.916 DEBUG:boto:Params: {} 2023-11-02T18:42:32.917 DEBUG:boto:Token: None 2023-11-02T18:42:32.917 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/obj76 2023-11-02T18:42:32.918 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CqoUK8yOn/yMBTIiufACbWucv3s= 2023-11-02T18:42:32.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CqoUK8yOn/yMBTIiufACbWucv3s=', 'Content-Length': '0'} 2023-11-02T18:42:32.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df0a4906bc5b264b-006543ed98-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.922 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.922 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.923 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:42:32.923 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:42:32.924 DEBUG:boto:Method: GET 2023-11-02T18:42:32.924 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:42:32.925 DEBUG:boto:Data: 2023-11-02T18:42:32.925 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.926 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:32.926 DEBUG:boto:Port: 8001 2023-11-02T18:42:32.927 DEBUG:boto:Params: {} 2023-11-02T18:42:32.927 DEBUG:boto:Token: None 2023-11-02T18:42:32.927 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.928 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c42aa7948818c821-006543ed98-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.978 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.979 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:32.980 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:32.980 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:42:32.981 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:42:32.981 DEBUG:boto:Method: GET 2023-11-02T18:42:32.982 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:42:32.982 DEBUG:boto:Data: 2023-11-02T18:42:32.982 DEBUG:boto:Headers: {} 2023-11-02T18:42:32.983 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:32.983 DEBUG:boto:Port: 8000 2023-11-02T18:42:32.984 DEBUG:boto:Params: {} 2023-11-02T18:42:32.984 DEBUG:boto:Token: None 2023-11-02T18:42:32.985 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:32 GMT /ibtljn-59/ 2023-11-02T18:42:32.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc= 2023-11-02T18:42:32.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8UUACi+u8DlO7pre8JQkrQyaokc=', 'Content-Length': '0'} 2023-11-02T18:42:32.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e25e3eb9a9606cb-006543ed98-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:32.995 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:32.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:42:32.997 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:42:32.998 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:42:32.999 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.000 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.000 DEBUG:boto:Method: GET 2023-11-02T18:42:33.000 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.001 DEBUG:boto:Data: 2023-11-02T18:42:33.001 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.002 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.002 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.003 DEBUG:boto:Params: {} 2023-11-02T18:42:33.003 DEBUG:boto:Token: None 2023-11-02T18:42:33.004 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.005 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HmzmKVt0rsGM+Cz1u5VSiNZdizE= 2023-11-02T18:42:33.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HmzmKVt0rsGM+Cz1u5VSiNZdizE=', 'Content-Length': '0'} 2023-11-02T18:42:33.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008056c72e01046c7b-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.009 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:42:33.009 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:42:33.010 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.010 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.011 DEBUG:boto:Method: GET 2023-11-02T18:42:33.011 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.012 DEBUG:boto:Data: 2023-11-02T18:42:33.012 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.012 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.013 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.013 DEBUG:boto:Params: {} 2023-11-02T18:42:33.014 DEBUG:boto:Token: None 2023-11-02T18:42:33.014 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.015 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HmzmKVt0rsGM+Cz1u5VSiNZdizE= 2023-11-02T18:42:33.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HmzmKVt0rsGM+Cz1u5VSiNZdizE=', 'Content-Length': '0'} 2023-11-02T18:42:33.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003199abb8f94c04a6-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.019 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:42:33.019 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:42:33.020 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.020 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.021 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.021 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.022 DEBUG:boto:Data: 2023-11-02T18:42:33.022 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.022 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.023 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.023 DEBUG:boto:Params: {} 2023-11-02T18:42:33.024 DEBUG:boto:Token: None 2023-11-02T18:42:33.024 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.025 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gh5lnxBYLGuDRLzZsoGfYpSqIno= 2023-11-02T18:42:33.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gh5lnxBYLGuDRLzZsoGfYpSqIno=', 'Content-Length': '0'} 2023-11-02T18:42:33.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000396879630e517dcc-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.028 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:42:33.029 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:42:33.029 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.030 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.030 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.031 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.031 DEBUG:boto:Data: 2023-11-02T18:42:33.031 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.032 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.033 DEBUG:boto:Params: {} 2023-11-02T18:42:33.033 DEBUG:boto:Token: None 2023-11-02T18:42:33.034 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.034 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gh5lnxBYLGuDRLzZsoGfYpSqIno= 2023-11-02T18:42:33.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gh5lnxBYLGuDRLzZsoGfYpSqIno=', 'Content-Length': '0'} 2023-11-02T18:42:33.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acdc26cd1f80dcc1-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:42:33.038 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:42:33.039 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:42:33.039 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.040 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.040 DEBUG:boto:Method: GET 2023-11-02T18:42:33.040 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.041 DEBUG:boto:Data: 2023-11-02T18:42:33.041 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.042 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.042 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.043 DEBUG:boto:Params: {} 2023-11-02T18:42:33.043 DEBUG:boto:Token: None 2023-11-02T18:42:33.044 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.044 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HmzmKVt0rsGM+Cz1u5VSiNZdizE= 2023-11-02T18:42:33.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HmzmKVt0rsGM+Cz1u5VSiNZdizE=', 'Content-Length': '0'} 2023-11-02T18:42:33.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014f7c2ac583ce285-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.095 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:42:33.096 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:42:33.096 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.097 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.097 DEBUG:boto:Method: GET 2023-11-02T18:42:33.098 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.098 DEBUG:boto:Data: 2023-11-02T18:42:33.099 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.099 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.100 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.100 DEBUG:boto:Params: {} 2023-11-02T18:42:33.101 DEBUG:boto:Token: None 2023-11-02T18:42:33.101 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:42:33.102 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HmzmKVt0rsGM+Cz1u5VSiNZdizE= 2023-11-02T18:42:33.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HmzmKVt0rsGM+Cz1u5VSiNZdizE=', 'Content-Length': '0'} 2023-11-02T18:42:33.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4c28e9e2f3ac9dc-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.106 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.107 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.107 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:42:33.108 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:42:33.108 DEBUG:boto:Method: GET 2023-11-02T18:42:33.108 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:42:33.109 DEBUG:boto:Data: 2023-11-02T18:42:33.109 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.110 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.110 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.111 DEBUG:boto:Params: {} 2023-11-02T18:42:33.111 DEBUG:boto:Token: None 2023-11-02T18:42:33.112 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.112 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da909d7fae6977ae-006543ed99-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.119 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.119 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.120 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.120 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:42:33.121 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:42:33.121 DEBUG:boto:Method: GET 2023-11-02T18:42:33.122 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:42:33.122 DEBUG:boto:Data: 2023-11-02T18:42:33.123 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.123 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.123 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.124 DEBUG:boto:Params: {} 2023-11-02T18:42:33.124 DEBUG:boto:Token: None 2023-11-02T18:42:33.125 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.125 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013aa4704d1906c44-006543ed99-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.136 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.137 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:42:33.137 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:42:33.138 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.138 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:42:33.139 DEBUG:boto:Data: 2023-11-02T18:42:33.139 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.140 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.140 DEBUG:boto:Params: {} 2023-11-02T18:42:33.141 DEBUG:boto:Token: None 2023-11-02T18:42:33.141 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj85 2023-11-02T18:42:33.142 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bQeoZXj2W8NiNadzLsenjvriRd8= 2023-11-02T18:42:33.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bQeoZXj2W8NiNadzLsenjvriRd8=', 'Content-Length': '0'} 2023-11-02T18:42:33.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049565a28e31a5cac-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.145 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:42:33.146 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:42:33.146 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.147 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:42:33.147 DEBUG:boto:Data: 2023-11-02T18:42:33.148 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.149 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.149 DEBUG:boto:Params: {} 2023-11-02T18:42:33.150 DEBUG:boto:Token: None 2023-11-02T18:42:33.150 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj85 2023-11-02T18:42:33.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bQeoZXj2W8NiNadzLsenjvriRd8= 2023-11-02T18:42:33.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bQeoZXj2W8NiNadzLsenjvriRd8=', 'Content-Length': '0'} 2023-11-02T18:42:33.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026a05873379d1528-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:42:33.155 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:42:33.155 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:42:33.156 DEBUG:boto:Method: GET 2023-11-02T18:42:33.156 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:42:33.156 DEBUG:boto:Data: 2023-11-02T18:42:33.157 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.157 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.158 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.158 DEBUG:boto:Params: {} 2023-11-02T18:42:33.159 DEBUG:boto:Token: None 2023-11-02T18:42:33.159 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj85 2023-11-02T18:42:33.160 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hAxpMpmFUY1RbuNU0//q22MUNb8= 2023-11-02T18:42:33.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hAxpMpmFUY1RbuNU0//q22MUNb8=', 'Content-Length': '0'} 2023-11-02T18:42:33.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053bc52b812c56dac-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.164 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:42:33.164 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:42:33.165 DEBUG:boto:Method: GET 2023-11-02T18:42:33.165 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:42:33.165 DEBUG:boto:Data: 2023-11-02T18:42:33.166 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.166 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.167 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.167 DEBUG:boto:Params: {} 2023-11-02T18:42:33.168 DEBUG:boto:Token: None 2023-11-02T18:42:33.168 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj85 2023-11-02T18:42:33.169 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hAxpMpmFUY1RbuNU0//q22MUNb8= 2023-11-02T18:42:33.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hAxpMpmFUY1RbuNU0//q22MUNb8=', 'Content-Length': '0'} 2023-11-02T18:42:33.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c857e4aae3751ff-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.173 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.173 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.174 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:42:33.174 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:42:33.175 DEBUG:boto:Method: GET 2023-11-02T18:42:33.175 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:42:33.176 DEBUG:boto:Data: 2023-11-02T18:42:33.176 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.177 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.177 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.177 DEBUG:boto:Params: {} 2023-11-02T18:42:33.178 DEBUG:boto:Token: None 2023-11-02T18:42:33.179 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.179 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002063378ea21c9f35-006543ed99-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.226 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.227 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.228 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.228 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:42:33.229 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:42:33.229 DEBUG:boto:Method: GET 2023-11-02T18:42:33.229 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:42:33.231 DEBUG:boto:Data: 2023-11-02T18:42:33.231 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.232 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.232 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.232 DEBUG:boto:Params: {} 2023-11-02T18:42:33.233 DEBUG:boto:Token: None 2023-11-02T18:42:33.234 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.234 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fd0a33895d6d2c4-006543ed99-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.244 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:42:33.245 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:42:33.246 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:42:33.246 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.247 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.247 DEBUG:boto:Method: GET 2023-11-02T18:42:33.248 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.248 DEBUG:boto:Data: 2023-11-02T18:42:33.249 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.249 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.250 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.250 DEBUG:boto:Params: {} 2023-11-02T18:42:33.251 DEBUG:boto:Token: None 2023-11-02T18:42:33.251 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.252 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pan8fIJ0UU60pV/pkv40cLS16r8= 2023-11-02T18:42:33.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pan8fIJ0UU60pV/pkv40cLS16r8=', 'Content-Length': '0'} 2023-11-02T18:42:33.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037f8e6190bb20aef-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.256 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:42:33.256 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:42:33.257 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.257 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.258 DEBUG:boto:Method: GET 2023-11-02T18:42:33.258 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.259 DEBUG:boto:Data: 2023-11-02T18:42:33.259 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.260 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.260 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.261 DEBUG:boto:Params: {} 2023-11-02T18:42:33.261 DEBUG:boto:Token: None 2023-11-02T18:42:33.262 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.262 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pan8fIJ0UU60pV/pkv40cLS16r8= 2023-11-02T18:42:33.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pan8fIJ0UU60pV/pkv40cLS16r8=', 'Content-Length': '0'} 2023-11-02T18:42:33.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d105fd5cfc653ac6-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.266 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:42:33.267 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:42:33.267 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.268 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.268 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.268 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.269 DEBUG:boto:Data: 2023-11-02T18:42:33.269 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.270 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.270 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.270 DEBUG:boto:Params: {} 2023-11-02T18:42:33.271 DEBUG:boto:Token: None 2023-11-02T18:42:33.271 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.272 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8IJ2Jhw0RGwVcWG1/nNkrvmLCIk= 2023-11-02T18:42:33.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8IJ2Jhw0RGwVcWG1/nNkrvmLCIk=', 'Content-Length': '0'} 2023-11-02T18:42:33.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d2c9d01a33e7cb0-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.276 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:42:33.276 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:42:33.277 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.277 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.278 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.278 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.279 DEBUG:boto:Data: 2023-11-02T18:42:33.279 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.280 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.280 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.281 DEBUG:boto:Params: {} 2023-11-02T18:42:33.281 DEBUG:boto:Token: None 2023-11-02T18:42:33.282 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.282 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8IJ2Jhw0RGwVcWG1/nNkrvmLCIk= 2023-11-02T18:42:33.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8IJ2Jhw0RGwVcWG1/nNkrvmLCIk=', 'Content-Length': '0'} 2023-11-02T18:42:33.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053b4fba1cf15ff49-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:42:33.287 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:42:33.288 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:42:33.288 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.288 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.289 DEBUG:boto:Method: GET 2023-11-02T18:42:33.289 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.290 DEBUG:boto:Data: 2023-11-02T18:42:33.290 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.291 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.291 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.292 DEBUG:boto:Params: {} 2023-11-02T18:42:33.292 DEBUG:boto:Token: None 2023-11-02T18:42:33.293 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.293 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pan8fIJ0UU60pV/pkv40cLS16r8= 2023-11-02T18:42:33.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pan8fIJ0UU60pV/pkv40cLS16r8=', 'Content-Length': '0'} 2023-11-02T18:42:33.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e9f346351fcf4c-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.338 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:42:33.339 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:42:33.339 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.340 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.340 DEBUG:boto:Method: GET 2023-11-02T18:42:33.341 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.341 DEBUG:boto:Data: 2023-11-02T18:42:33.342 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.342 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.343 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.343 DEBUG:boto:Params: {} 2023-11-02T18:42:33.344 DEBUG:boto:Token: None 2023-11-02T18:42:33.344 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:42:33.345 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pan8fIJ0UU60pV/pkv40cLS16r8= 2023-11-02T18:42:33.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pan8fIJ0UU60pV/pkv40cLS16r8=', 'Content-Length': '0'} 2023-11-02T18:42:33.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050dcca735b6cf96d-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.349 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.350 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.350 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:42:33.351 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:42:33.351 DEBUG:boto:Method: GET 2023-11-02T18:42:33.352 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:42:33.352 DEBUG:boto:Data: 2023-11-02T18:42:33.353 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.353 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.353 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.354 DEBUG:boto:Params: {} 2023-11-02T18:42:33.354 DEBUG:boto:Token: None 2023-11-02T18:42:33.355 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.356 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e459a927e395f33-006543ed99-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.362 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.362 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.363 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.363 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:42:33.364 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:42:33.364 DEBUG:boto:Method: GET 2023-11-02T18:42:33.365 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:42:33.365 DEBUG:boto:Data: 2023-11-02T18:42:33.366 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.367 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.367 DEBUG:boto:Params: {} 2023-11-02T18:42:33.368 DEBUG:boto:Token: None 2023-11-02T18:42:33.368 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.369 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a17145412130fa93-006543ed99-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.377 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.378 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:42:33.379 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:42:33.379 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.380 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:42:33.380 DEBUG:boto:Data: 2023-11-02T18:42:33.381 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.381 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.381 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.382 DEBUG:boto:Params: {} 2023-11-02T18:42:33.382 DEBUG:boto:Token: None 2023-11-02T18:42:33.383 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj87 2023-11-02T18:42:33.383 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BZqsDcGLWzg6IKg3cBcFZaLyWyo= 2023-11-02T18:42:33.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BZqsDcGLWzg6IKg3cBcFZaLyWyo=', 'Content-Length': '0'} 2023-11-02T18:42:33.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f1ce373e6bfd5bf-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.387 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:42:33.388 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:42:33.388 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.388 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:42:33.389 DEBUG:boto:Data: 2023-11-02T18:42:33.389 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.390 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.390 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.391 DEBUG:boto:Params: {} 2023-11-02T18:42:33.391 DEBUG:boto:Token: None 2023-11-02T18:42:33.392 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj87 2023-11-02T18:42:33.392 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BZqsDcGLWzg6IKg3cBcFZaLyWyo= 2023-11-02T18:42:33.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BZqsDcGLWzg6IKg3cBcFZaLyWyo=', 'Content-Length': '0'} 2023-11-02T18:42:33.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000010e3c2a96e9f70-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:42:33.396 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:42:33.397 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:42:33.397 DEBUG:boto:Method: GET 2023-11-02T18:42:33.398 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:42:33.399 DEBUG:boto:Data: 2023-11-02T18:42:33.399 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.400 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.400 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.400 DEBUG:boto:Params: {} 2023-11-02T18:42:33.401 DEBUG:boto:Token: None 2023-11-02T18:42:33.401 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj87 2023-11-02T18:42:33.402 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TMJq6lEzmz9iuq/uUx8DcY3bVH0= 2023-11-02T18:42:33.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TMJq6lEzmz9iuq/uUx8DcY3bVH0=', 'Content-Length': '0'} 2023-11-02T18:42:33.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000359358d9e31287c-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.450 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:42:33.451 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:42:33.451 DEBUG:boto:Method: GET 2023-11-02T18:42:33.452 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:42:33.452 DEBUG:boto:Data: 2023-11-02T18:42:33.453 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.453 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.454 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.454 DEBUG:boto:Params: {} 2023-11-02T18:42:33.455 DEBUG:boto:Token: None 2023-11-02T18:42:33.455 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj87 2023-11-02T18:42:33.456 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TMJq6lEzmz9iuq/uUx8DcY3bVH0= 2023-11-02T18:42:33.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TMJq6lEzmz9iuq/uUx8DcY3bVH0=', 'Content-Length': '0'} 2023-11-02T18:42:33.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000922cbb7b24e08408-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.460 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.460 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.461 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:42:33.461 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:42:33.462 DEBUG:boto:Method: GET 2023-11-02T18:42:33.462 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:42:33.463 DEBUG:boto:Data: 2023-11-02T18:42:33.463 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.464 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.464 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.464 DEBUG:boto:Params: {} 2023-11-02T18:42:33.465 DEBUG:boto:Token: None 2023-11-02T18:42:33.465 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.466 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d086ae5732318bd-006543ed99-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.472 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.472 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.473 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.473 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:42:33.474 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:42:33.474 DEBUG:boto:Method: GET 2023-11-02T18:42:33.475 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:42:33.475 DEBUG:boto:Data: 2023-11-02T18:42:33.475 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.476 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.476 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.477 DEBUG:boto:Params: {} 2023-11-02T18:42:33.477 DEBUG:boto:Token: None 2023-11-02T18:42:33.478 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.478 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001faa592452dbd42e-006543ed99-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.488 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:42:33.489 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:42:33.490 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:42:33.490 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.490 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.491 DEBUG:boto:Method: GET 2023-11-02T18:42:33.491 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.492 DEBUG:boto:Data: 2023-11-02T18:42:33.492 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.493 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.494 DEBUG:boto:Params: {} 2023-11-02T18:42:33.494 DEBUG:boto:Token: None 2023-11-02T18:42:33.495 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.495 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kjKRrErTCpka6VO9medr6k+jzKY= 2023-11-02T18:42:33.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kjKRrErTCpka6VO9medr6k+jzKY=', 'Content-Length': '0'} 2023-11-02T18:42:33.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003e51200586d79a4-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.499 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:42:33.500 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:42:33.500 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.501 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.501 DEBUG:boto:Method: GET 2023-11-02T18:42:33.501 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.502 DEBUG:boto:Data: 2023-11-02T18:42:33.502 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.503 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.503 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.504 DEBUG:boto:Params: {} 2023-11-02T18:42:33.506 DEBUG:boto:Token: None 2023-11-02T18:42:33.507 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.507 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kjKRrErTCpka6VO9medr6k+jzKY= 2023-11-02T18:42:33.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kjKRrErTCpka6VO9medr6k+jzKY=', 'Content-Length': '0'} 2023-11-02T18:42:33.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0512a95804391c2-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.511 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:42:33.512 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:42:33.512 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.513 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.513 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.513 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.514 DEBUG:boto:Data: 2023-11-02T18:42:33.514 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.515 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.515 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.516 DEBUG:boto:Params: {} 2023-11-02T18:42:33.516 DEBUG:boto:Token: None 2023-11-02T18:42:33.517 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.517 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VpnrnwSq6gRXVFsLqNs4U+2ww2Q= 2023-11-02T18:42:33.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VpnrnwSq6gRXVFsLqNs4U+2ww2Q=', 'Content-Length': '0'} 2023-11-02T18:42:33.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3d8c160279f6f89-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.522 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:42:33.522 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:42:33.523 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.523 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.524 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.524 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.524 DEBUG:boto:Data: 2023-11-02T18:42:33.525 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.525 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.526 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.526 DEBUG:boto:Params: {} 2023-11-02T18:42:33.527 DEBUG:boto:Token: None 2023-11-02T18:42:33.527 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.528 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VpnrnwSq6gRXVFsLqNs4U+2ww2Q= 2023-11-02T18:42:33.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VpnrnwSq6gRXVFsLqNs4U+2ww2Q=', 'Content-Length': '0'} 2023-11-02T18:42:33.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000343f2c53edf2852b-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:42:33.532 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:42:33.532 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:42:33.533 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.533 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.534 DEBUG:boto:Method: GET 2023-11-02T18:42:33.534 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.535 DEBUG:boto:Data: 2023-11-02T18:42:33.535 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.535 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.536 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.536 DEBUG:boto:Params: {} 2023-11-02T18:42:33.537 DEBUG:boto:Token: None 2023-11-02T18:42:33.537 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.538 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kjKRrErTCpka6VO9medr6k+jzKY= 2023-11-02T18:42:33.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kjKRrErTCpka6VO9medr6k+jzKY=', 'Content-Length': '0'} 2023-11-02T18:42:33.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcaffef81a5bac11-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.586 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:42:33.587 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:42:33.587 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.588 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.588 DEBUG:boto:Method: GET 2023-11-02T18:42:33.589 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.589 DEBUG:boto:Data: 2023-11-02T18:42:33.590 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.590 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.591 DEBUG:boto:Params: {} 2023-11-02T18:42:33.591 DEBUG:boto:Token: None 2023-11-02T18:42:33.592 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:42:33.592 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kjKRrErTCpka6VO9medr6k+jzKY= 2023-11-02T18:42:33.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kjKRrErTCpka6VO9medr6k+jzKY=', 'Content-Length': '0'} 2023-11-02T18:42:33.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003a81a389402c33b-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.597 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.597 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.598 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:42:33.598 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:42:33.599 DEBUG:boto:Method: GET 2023-11-02T18:42:33.599 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:42:33.600 DEBUG:boto:Data: 2023-11-02T18:42:33.600 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.601 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.601 DEBUG:boto:Params: {} 2023-11-02T18:42:33.602 DEBUG:boto:Token: None 2023-11-02T18:42:33.602 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.603 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009314514ae97b4316-006543ed99-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.610 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.611 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.611 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.611 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:42:33.612 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:42:33.612 DEBUG:boto:Method: GET 2023-11-02T18:42:33.613 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:42:33.613 DEBUG:boto:Data: 2023-11-02T18:42:33.614 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.614 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.615 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.615 DEBUG:boto:Params: {} 2023-11-02T18:42:33.615 DEBUG:boto:Token: None 2023-11-02T18:42:33.616 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.617 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7c924c04e0f6ed-006543ed99-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.625 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.626 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:42:33.626 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:42:33.627 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.627 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:42:33.627 DEBUG:boto:Data: 2023-11-02T18:42:33.628 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.628 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.629 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.629 DEBUG:boto:Params: {} 2023-11-02T18:42:33.630 DEBUG:boto:Token: None 2023-11-02T18:42:33.630 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj92 2023-11-02T18:42:33.631 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I8vv6x0DRY0ETbyqeAL4lD3rSu0= 2023-11-02T18:42:33.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I8vv6x0DRY0ETbyqeAL4lD3rSu0=', 'Content-Length': '0'} 2023-11-02T18:42:33.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001771e26b0dc88da0-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.634 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:42:33.635 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:42:33.635 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.636 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:42:33.636 DEBUG:boto:Data: 2023-11-02T18:42:33.637 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.638 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.638 DEBUG:boto:Params: {} 2023-11-02T18:42:33.638 DEBUG:boto:Token: None 2023-11-02T18:42:33.639 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj92 2023-11-02T18:42:33.639 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I8vv6x0DRY0ETbyqeAL4lD3rSu0= 2023-11-02T18:42:33.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I8vv6x0DRY0ETbyqeAL4lD3rSu0=', 'Content-Length': '0'} 2023-11-02T18:42:33.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000910ee3880cbebda5-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:42:33.643 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:42:33.644 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:42:33.644 DEBUG:boto:Method: GET 2023-11-02T18:42:33.645 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:42:33.645 DEBUG:boto:Data: 2023-11-02T18:42:33.646 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.646 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.646 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.647 DEBUG:boto:Params: {} 2023-11-02T18:42:33.647 DEBUG:boto:Token: None 2023-11-02T18:42:33.648 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj92 2023-11-02T18:42:33.648 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:scL+n8MCeEFRPwGrFT/1daNKsEk= 2023-11-02T18:42:33.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:scL+n8MCeEFRPwGrFT/1daNKsEk=', 'Content-Length': '0'} 2023-11-02T18:42:33.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6be13c0170fbe22-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.653 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:42:33.653 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:42:33.654 DEBUG:boto:Method: GET 2023-11-02T18:42:33.654 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:42:33.655 DEBUG:boto:Data: 2023-11-02T18:42:33.655 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.656 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.656 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.657 DEBUG:boto:Params: {} 2023-11-02T18:42:33.657 DEBUG:boto:Token: None 2023-11-02T18:42:33.658 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj92 2023-11-02T18:42:33.658 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:scL+n8MCeEFRPwGrFT/1daNKsEk= 2023-11-02T18:42:33.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:scL+n8MCeEFRPwGrFT/1daNKsEk=', 'Content-Length': '0'} 2023-11-02T18:42:33.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b14c18fa30c9e950-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.702 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.703 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.703 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:42:33.704 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:42:33.704 DEBUG:boto:Method: GET 2023-11-02T18:42:33.705 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:42:33.705 DEBUG:boto:Data: 2023-11-02T18:42:33.706 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.706 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.706 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.707 DEBUG:boto:Params: {} 2023-11-02T18:42:33.707 DEBUG:boto:Token: None 2023-11-02T18:42:33.709 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.710 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dba92cb310c9f68b-006543ed99-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.716 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.717 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.717 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.718 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:42:33.718 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:42:33.719 DEBUG:boto:Method: GET 2023-11-02T18:42:33.719 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:42:33.720 DEBUG:boto:Data: 2023-11-02T18:42:33.720 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.720 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.721 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.721 DEBUG:boto:Params: {} 2023-11-02T18:42:33.722 DEBUG:boto:Token: None 2023-11-02T18:42:33.722 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.723 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069b1af1ff9fbb3e8-006543ed99-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.733 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:42:33.734 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:42:33.735 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:42:33.735 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.735 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.736 DEBUG:boto:Method: GET 2023-11-02T18:42:33.736 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.737 DEBUG:boto:Data: 2023-11-02T18:42:33.737 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.738 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.738 DEBUG:boto:Params: {} 2023-11-02T18:42:33.739 DEBUG:boto:Token: None 2023-11-02T18:42:33.739 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sNxyx50QbfgtpRVMJgG+EDP5EgQ= 2023-11-02T18:42:33.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sNxyx50QbfgtpRVMJgG+EDP5EgQ=', 'Content-Length': '0'} 2023-11-02T18:42:33.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000483ca5e66a6e38d2-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.787 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:42:33.787 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:42:33.788 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.788 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.789 DEBUG:boto:Method: GET 2023-11-02T18:42:33.789 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.790 DEBUG:boto:Data: 2023-11-02T18:42:33.790 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.791 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.791 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.792 DEBUG:boto:Params: {} 2023-11-02T18:42:33.792 DEBUG:boto:Token: None 2023-11-02T18:42:33.793 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sNxyx50QbfgtpRVMJgG+EDP5EgQ= 2023-11-02T18:42:33.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sNxyx50QbfgtpRVMJgG+EDP5EgQ=', 'Content-Length': '0'} 2023-11-02T18:42:33.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017927efc47544451-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.798 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:42:33.799 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:42:33.799 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.800 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.800 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.800 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.801 DEBUG:boto:Data: 2023-11-02T18:42:33.801 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.802 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.802 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.803 DEBUG:boto:Params: {} 2023-11-02T18:42:33.803 DEBUG:boto:Token: None 2023-11-02T18:42:33.804 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.804 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GvII1UcpEu5HVQEanhF0VhbSm8Q= 2023-11-02T18:42:33.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GvII1UcpEu5HVQEanhF0VhbSm8Q=', 'Content-Length': '0'} 2023-11-02T18:42:33.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a31c14f404d99f07-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.808 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:42:33.809 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:42:33.809 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.810 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.810 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.811 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.811 DEBUG:boto:Data: 2023-11-02T18:42:33.812 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.812 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.812 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.813 DEBUG:boto:Params: {} 2023-11-02T18:42:33.813 DEBUG:boto:Token: None 2023-11-02T18:42:33.814 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.814 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GvII1UcpEu5HVQEanhF0VhbSm8Q= 2023-11-02T18:42:33.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GvII1UcpEu5HVQEanhF0VhbSm8Q=', 'Content-Length': '0'} 2023-11-02T18:42:33.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efec050c20d261b3-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:42:33.819 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:42:33.819 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:42:33.820 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.820 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.821 DEBUG:boto:Method: GET 2023-11-02T18:42:33.821 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.822 DEBUG:boto:Data: 2023-11-02T18:42:33.822 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.823 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.823 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.824 DEBUG:boto:Params: {} 2023-11-02T18:42:33.824 DEBUG:boto:Token: None 2023-11-02T18:42:33.825 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.825 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sNxyx50QbfgtpRVMJgG+EDP5EgQ= 2023-11-02T18:42:33.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sNxyx50QbfgtpRVMJgG+EDP5EgQ=', 'Content-Length': '0'} 2023-11-02T18:42:33.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000393d4a1668a1ac03-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.829 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:42:33.830 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:42:33.830 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.831 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.831 DEBUG:boto:Method: GET 2023-11-02T18:42:33.832 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.832 DEBUG:boto:Data: 2023-11-02T18:42:33.832 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.833 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.833 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.834 DEBUG:boto:Params: {} 2023-11-02T18:42:33.834 DEBUG:boto:Token: None 2023-11-02T18:42:33.835 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:42:33.835 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sNxyx50QbfgtpRVMJgG+EDP5EgQ= 2023-11-02T18:42:33.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sNxyx50QbfgtpRVMJgG+EDP5EgQ=', 'Content-Length': '0'} 2023-11-02T18:42:33.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7ed6e15c9fa3046-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.840 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.840 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.840 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:42:33.841 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:42:33.841 DEBUG:boto:Method: GET 2023-11-02T18:42:33.842 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:42:33.842 DEBUG:boto:Data: 2023-11-02T18:42:33.843 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.843 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.843 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.844 DEBUG:boto:Params: {} 2023-11-02T18:42:33.844 DEBUG:boto:Token: None 2023-11-02T18:42:33.845 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.845 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e8a45e615f4ab8-006543ed99-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.852 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.853 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.853 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.853 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:42:33.854 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:42:33.854 DEBUG:boto:Method: GET 2023-11-02T18:42:33.855 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:42:33.855 DEBUG:boto:Data: 2023-11-02T18:42:33.856 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.856 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.857 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.857 DEBUG:boto:Params: {} 2023-11-02T18:42:33.857 DEBUG:boto:Token: None 2023-11-02T18:42:33.858 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.858 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e78d384232b594d-006543ed99-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.867 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.867 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:42:33.868 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:42:33.868 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.869 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:42:33.869 DEBUG:boto:Data: 2023-11-02T18:42:33.870 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.870 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.871 DEBUG:boto:Params: {} 2023-11-02T18:42:33.871 DEBUG:boto:Token: None 2023-11-02T18:42:33.872 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj95 2023-11-02T18:42:33.872 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T6XxzVw8HGlVAI0ele4iR8oatsc= 2023-11-02T18:42:33.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T6XxzVw8HGlVAI0ele4iR8oatsc=', 'Content-Length': '0'} 2023-11-02T18:42:33.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f887f0596753c24-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.876 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:42:33.876 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:42:33.877 DEBUG:boto:Method: HEAD 2023-11-02T18:42:33.877 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:42:33.877 DEBUG:boto:Data: 2023-11-02T18:42:33.878 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.878 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.879 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.879 DEBUG:boto:Params: {} 2023-11-02T18:42:33.880 DEBUG:boto:Token: None 2023-11-02T18:42:33.880 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj95 2023-11-02T18:42:33.880 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T6XxzVw8HGlVAI0ele4iR8oatsc= 2023-11-02T18:42:33.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T6XxzVw8HGlVAI0ele4iR8oatsc=', 'Content-Length': '0'} 2023-11-02T18:42:33.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fa7d648c92d1749-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:42:33.884 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:42:33.885 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:42:33.885 DEBUG:boto:Method: GET 2023-11-02T18:42:33.886 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:42:33.886 DEBUG:boto:Data: 2023-11-02T18:42:33.886 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.887 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.887 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.888 DEBUG:boto:Params: {} 2023-11-02T18:42:33.888 DEBUG:boto:Token: None 2023-11-02T18:42:33.889 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj95 2023-11-02T18:42:33.889 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9wTarbrVdlf/e8Ow1kJlYWo6XiI= 2023-11-02T18:42:33.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9wTarbrVdlf/e8Ow1kJlYWo6XiI=', 'Content-Length': '0'} 2023-11-02T18:42:33.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed97df5ae535e726-006543ed99-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.934 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:42:33.935 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:42:33.936 DEBUG:boto:Method: GET 2023-11-02T18:42:33.936 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:42:33.937 DEBUG:boto:Data: 2023-11-02T18:42:33.937 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.938 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.938 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.939 DEBUG:boto:Params: {} 2023-11-02T18:42:33.939 DEBUG:boto:Token: None 2023-11-02T18:42:33.940 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/obj95 2023-11-02T18:42:33.941 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9wTarbrVdlf/e8Ow1kJlYWo6XiI= 2023-11-02T18:42:33.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9wTarbrVdlf/e8Ow1kJlYWo6XiI=', 'Content-Length': '0'} 2023-11-02T18:42:33.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045079cfed98aea77-006543ed99-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.945 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.946 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.946 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:42:33.947 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:42:33.947 DEBUG:boto:Method: GET 2023-11-02T18:42:33.948 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:42:33.948 DEBUG:boto:Data: 2023-11-02T18:42:33.949 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.949 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:42:33.949 DEBUG:boto:Port: 8001 2023-11-02T18:42:33.950 DEBUG:boto:Params: {} 2023-11-02T18:42:33.950 DEBUG:boto:Token: None 2023-11-02T18:42:33.951 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.951 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af1eaf33d28901d9-006543ed99-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.958 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.959 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:33.959 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:33.960 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:42:33.960 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:42:33.960 DEBUG:boto:Method: GET 2023-11-02T18:42:33.961 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:42:33.962 DEBUG:boto:Data: 2023-11-02T18:42:33.962 DEBUG:boto:Headers: {} 2023-11-02T18:42:33.963 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:33.963 DEBUG:boto:Port: 8000 2023-11-02T18:42:33.963 DEBUG:boto:Params: {} 2023-11-02T18:42:33.964 DEBUG:boto:Token: None 2023-11-02T18:42:33.964 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:42:33 GMT /ibtljn-59/ 2023-11-02T18:42:33.965 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg= 2023-11-02T18:42:33.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9RFsUz5TQQz1U3XZEJ6fUy1Islg=', 'Content-Length': '0'} 2023-11-02T18:42:33.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f84a8e47ec7281f-006543ed99-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:42:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:33.975 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:42:33.976 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:42:33.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:42:33.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:42:33.978 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:42:34.022 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:42:34.022 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:42:34.110 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:42:34.110 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-59 2023-11-02T18:42:34.136 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 25 2023-11-02T18:42:34.259 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:34.259 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:34.260 DEBUG:boto:Method: HEAD 2023-11-02T18:42:34.260 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:42:34.261 DEBUG:boto:Data: 2023-11-02T18:42:34.261 DEBUG:boto:Headers: {} 2023-11-02T18:42:34.262 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:34.262 DEBUG:boto:Port: 8000 2023-11-02T18:42:34.262 DEBUG:boto:Params: {} 2023-11-02T18:42:34.263 DEBUG:boto:Token: None 2023-11-02T18:42:34.263 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:34 GMT /ibtljn-59/ 2023-11-02T18:42:34.265 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mbXyvNVMLXMfCj3YmQfavFHh3cM= 2023-11-02T18:42:34.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mbXyvNVMLXMfCj3YmQfavFHh3cM=', 'Content-Length': '0'} 2023-11-02T18:42:34.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcde3fca92e6c4b9-006543ed9a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:34.273 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:42:34.274 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:42:34.275 DEBUG:boto:Method: PUT 2023-11-02T18:42:34.275 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:42:34.275 DEBUG:boto:Data: 2023-11-02T18:42:34.276 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:42:34.276 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:34.277 DEBUG:boto:Port: 8000 2023-11-02T18:42:34.277 DEBUG:boto:Params: {} 2023-11-02T18:42:34.278 DEBUG:boto:Token: None 2023-11-02T18:42:34.278 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:42:34 GMT /ibtljn-59/obj31 2023-11-02T18:42:34.279 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5yWgB152WwTtWabVDKfBX4Rd+AU= 2023-11-02T18:42:34.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:42:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5yWgB152WwTtWabVDKfBX4Rd+AU='} 2023-11-02T18:42:34.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e17626b421f9056e-006543ed9a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:34.291 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:34.291 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:34.292 DEBUG:boto:Method: HEAD 2023-11-02T18:42:34.292 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:42:34.293 DEBUG:boto:Data: 2023-11-02T18:42:34.293 DEBUG:boto:Headers: {} 2023-11-02T18:42:34.293 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:34.294 DEBUG:boto:Port: 8000 2023-11-02T18:42:34.294 DEBUG:boto:Params: {} 2023-11-02T18:42:34.295 DEBUG:boto:Token: None 2023-11-02T18:42:34.295 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:34 GMT /ibtljn-59/ 2023-11-02T18:42:34.296 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mbXyvNVMLXMfCj3YmQfavFHh3cM= 2023-11-02T18:42:34.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mbXyvNVMLXMfCj3YmQfavFHh3cM=', 'Content-Length': '0'} 2023-11-02T18:42:34.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000301d2f9bd7cd4bbe-006543ed9a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:34.304 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:42:34.304 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:42:34.304 DEBUG:boto:Method: PUT 2023-11-02T18:42:34.305 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:42:34.305 DEBUG:boto:Data: 2023-11-02T18:42:34.306 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:42:34.306 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:34.307 DEBUG:boto:Port: 8000 2023-11-02T18:42:34.307 DEBUG:boto:Params: {} 2023-11-02T18:42:34.308 DEBUG:boto:Token: None 2023-11-02T18:42:34.308 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:42:34 GMT /ibtljn-59/obj62 2023-11-02T18:42:34.309 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rnCfhAKIHwneZdQY9hbBU81t0a4= 2023-11-02T18:42:34.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:42:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rnCfhAKIHwneZdQY9hbBU81t0a4='} 2023-11-02T18:42:34.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022a7a225cc6eb62e-006543ed9a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:34.359 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:34.360 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:34.360 DEBUG:boto:Method: HEAD 2023-11-02T18:42:34.361 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:42:34.361 DEBUG:boto:Data: 2023-11-02T18:42:34.361 DEBUG:boto:Headers: {} 2023-11-02T18:42:34.362 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:34.362 DEBUG:boto:Port: 8000 2023-11-02T18:42:34.363 DEBUG:boto:Params: {} 2023-11-02T18:42:34.363 DEBUG:boto:Token: None 2023-11-02T18:42:34.364 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:34 GMT /ibtljn-59/ 2023-11-02T18:42:34.364 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mbXyvNVMLXMfCj3YmQfavFHh3cM= 2023-11-02T18:42:34.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mbXyvNVMLXMfCj3YmQfavFHh3cM=', 'Content-Length': '0'} 2023-11-02T18:42:34.372 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d703096db4b770d5-006543ed9a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:34.372 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:42:34.373 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:42:34.373 DEBUG:boto:Method: PUT 2023-11-02T18:42:34.374 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:42:34.374 DEBUG:boto:Data: 2023-11-02T18:42:34.375 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:42:34.375 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:34.375 DEBUG:boto:Port: 8000 2023-11-02T18:42:34.376 DEBUG:boto:Params: {} 2023-11-02T18:42:34.376 DEBUG:boto:Token: None 2023-11-02T18:42:34.377 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:42:34 GMT /ibtljn-59/obj93 2023-11-02T18:42:34.377 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rYP0qUYyc1j/6Nv6UNG6igkkHUE= 2023-11-02T18:42:34.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:42:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rYP0qUYyc1j/6Nv6UNG6igkkHUE='} 2023-11-02T18:42:34.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5e0214aa3d5765e-006543ed9a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:34.389 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:34.389 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:34.390 DEBUG:boto:Method: HEAD 2023-11-02T18:42:34.390 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:42:34.391 DEBUG:boto:Data: 2023-11-02T18:42:34.391 DEBUG:boto:Headers: {} 2023-11-02T18:42:34.391 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:34.392 DEBUG:boto:Port: 8000 2023-11-02T18:42:34.392 DEBUG:boto:Params: {} 2023-11-02T18:42:34.393 DEBUG:boto:Token: None 2023-11-02T18:42:34.393 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:34 GMT /ibtljn-59/ 2023-11-02T18:42:34.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mbXyvNVMLXMfCj3YmQfavFHh3cM= 2023-11-02T18:42:34.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mbXyvNVMLXMfCj3YmQfavFHh3cM=', 'Content-Length': '0'} 2023-11-02T18:42:34.401 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1bfa6de53cc8945-006543ed9a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:34.401 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:42:34.402 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:42:34.402 DEBUG:boto:Method: PUT 2023-11-02T18:42:34.403 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:42:34.403 DEBUG:boto:Data: 2023-11-02T18:42:34.404 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:42:34.404 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:34.404 DEBUG:boto:Port: 8000 2023-11-02T18:42:34.405 DEBUG:boto:Params: {} 2023-11-02T18:42:34.405 DEBUG:boto:Token: None 2023-11-02T18:42:34.406 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:42:34 GMT /ibtljn-59/obj124 2023-11-02T18:42:34.406 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jolw8L9rVY6Hkwkx4Wo0mrvT230= 2023-11-02T18:42:34.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:42:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jolw8L9rVY6Hkwkx4Wo0mrvT230='} 2023-11-02T18:42:34.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efebfe4d1d5cb17d-006543ed9a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:34.451 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:42:34.452 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:42:34.452 DEBUG:boto:Method: HEAD 2023-11-02T18:42:34.453 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:42:34.453 DEBUG:boto:Data: 2023-11-02T18:42:34.454 DEBUG:boto:Headers: {} 2023-11-02T18:42:34.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:34.455 DEBUG:boto:Port: 8000 2023-11-02T18:42:34.455 DEBUG:boto:Params: {} 2023-11-02T18:42:34.456 DEBUG:boto:Token: None 2023-11-02T18:42:34.456 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:42:34 GMT /ibtljn-59/ 2023-11-02T18:42:34.457 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mbXyvNVMLXMfCj3YmQfavFHh3cM= 2023-11-02T18:42:34.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:42:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mbXyvNVMLXMfCj3YmQfavFHh3cM=', 'Content-Length': '0'} 2023-11-02T18:42:34.464 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4a174a5f560042a-006543ed9a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:34.465 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:42:34.465 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:42:34.466 DEBUG:boto:Method: PUT 2023-11-02T18:42:34.466 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:42:34.466 DEBUG:boto:Data: 2023-11-02T18:42:34.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:42:34.467 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:42:34.468 DEBUG:boto:Port: 8000 2023-11-02T18:42:34.468 DEBUG:boto:Params: {} 2023-11-02T18:42:34.469 DEBUG:boto:Token: None 2023-11-02T18:42:34.469 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:42:34 GMT /ibtljn-59/obj155 2023-11-02T18:42:34.470 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3NvzQTeH0Ee+hnc4H6migq5Nd/0= 2023-11-02T18:42:34.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:42:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3NvzQTeH0Ee+hnc4H6migq5Nd/0='} 2023-11-02T18:42:34.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000802a9ad3598d08ff-006543ed9a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:42:34.481 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-59 2023-11-02T18:42:34.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:42:34.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:42:34.482 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:42:34.536 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:42:34.536 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:42:34.606 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:34.598+0000 7f1d6ac14f40 1 waiting to reach latest gen 5, on 4.. 2023-11-02T18:42:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:39.602+0000 7f1d6ac14f40 1 waiting to reach latest gen 5, on 4.. 2023-11-02T18:42:44.608 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:44.602+0000 7f1d6ac14f40 1 waiting to reach latest gen 5, on 4.. 2023-11-02T18:42:49.615 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:49.610+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:42:49.615 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:42:49.615 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:42:54.621 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:54.614+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:42:54.622 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:42:54.622 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:42:59.625 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:42:59.618+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:42:59.625 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:42:59.626 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:04.630 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:04.622+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:43:04.630 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:43:04.631 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:09.633 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:09.626+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:43:09.633 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:43:09.633 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:14.638 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:14.630+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:43:14.639 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:43:14.639 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:19.646 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:19.638+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:43:19.646 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:43:19.647 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:24.650 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:24.642+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:43:24.650 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:43:24.650 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:29.657 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:29.650+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:43:29.657 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , 00000000002.54.6, , , , 00000000001.64.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:29.658 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:34.661 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:34.654+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:43:34.661 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , 00000000002.54.6, , , , 00000000001.64.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:34.662 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:39.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:39.658+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:43:39.666 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , 00000000002.54.6, , , , 00000000001.64.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:39.666 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:44.671 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:44.665+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:43:44.672 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , 00000000002.54.6, , , , 00000000001.64.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:44.672 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:49.676 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:49.669+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:43:49.676 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , 00000000002.54.6, , , , 00000000001.64.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:49.676 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:54.681 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:54.673+0000 7f1d6ac14f40 1 waiting for incremental sync to catch up: 2023-11-02T18:43:54.681 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , 00000000002.54.6, , , , 00000000001.64.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:54.682 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:59.685 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:59.677+0000 7f1d6ac14f40 1 bucket sync caught up with source: 2023-11-02T18:43:59.685 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:59.685 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , 00000000002.54.6, , 00000000001.57.6, , 00000000001.64.6, , , , , , , , , , 00000000001.63.6, , , , , , , , , , , , , , 00000000001.63.6] 2023-11-02T18:43:59.686 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:59.677+0000 7f1d6ac14f40 0 bucket checkpoint complete 2023-11-02T18:43:59.718 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-59 2023-11-02T18:43:59.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:43:59.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:43:59.720 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:43:59.773 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:43:59.773 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:43:59.846 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:43:59.837+0000 7fafbc8f8f40 1 waiting for incremental sync to catch up: 2023-11-02T18:43:59.846 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000003.91.6, 00000000002.229.6, 00000000001.131.6, 00000000004.310.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000002.82.6, 00000000006.286.6, 00000000004.332.6] 2023-11-02T18:43:59.846 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000003.91.6, 00000000002.229.6, 00000000002.142.6, 00000000004.310.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000003.93.6, 00000000006.286.6, 00000000004.332.6] 2023-11-02T18:44:04.848 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:44:04.841+0000 7fafbc8f8f40 1 bucket sync caught up with source: 2023-11-02T18:44:04.849 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000003.91.6, 00000000002.229.6, 00000000002.142.6, 00000000004.310.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000003.93.6, 00000000006.286.6, 00000000004.332.6] 2023-11-02T18:44:04.849 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000003.91.6, 00000000002.229.6, 00000000002.142.6, 00000000004.310.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000003.93.6, 00000000006.286.6, 00000000004.332.6] 2023-11-02T18:44:04.849 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:44:04.841+0000 7fafbc8f8f40 0 bucket checkpoint complete 2023-11-02T18:44:04.878 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:44:04.879 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:04.880 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:04.880 DEBUG:boto:Method: HEAD 2023-11-02T18:44:04.880 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:44:04.881 DEBUG:boto:Data: 2023-11-02T18:44:04.881 DEBUG:boto:Headers: {} 2023-11-02T18:44:04.882 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:04.882 DEBUG:boto:Port: 8001 2023-11-02T18:44:04.883 DEBUG:boto:Params: {} 2023-11-02T18:44:04.883 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:44:04.884 DEBUG:boto:Token: None 2023-11-02T18:44:04.884 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:04 GMT /ibtljn-59/ 2023-11-02T18:44:04.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WOhx2ja8+IGPVhe93jlgBUxHJl4= 2023-11-02T18:44:04.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WOhx2ja8+IGPVhe93jlgBUxHJl4=', 'Content-Length': '0'} 2023-11-02T18:44:04.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055557b8be8efb898-006543edf4-1b8c-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:44:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:04.892 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:04.892 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:04.892 DEBUG:boto:Method: HEAD 2023-11-02T18:44:04.893 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:44:04.893 DEBUG:boto:Data: 2023-11-02T18:44:04.894 DEBUG:boto:Headers: {} 2023-11-02T18:44:04.894 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:04.894 DEBUG:boto:Port: 8000 2023-11-02T18:44:04.895 DEBUG:boto:Params: {} 2023-11-02T18:44:04.895 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:44:04.896 DEBUG:boto:Token: None 2023-11-02T18:44:04.896 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:04 GMT /ibtljn-59/ 2023-11-02T18:44:04.897 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WOhx2ja8+IGPVhe93jlgBUxHJl4= 2023-11-02T18:44:04.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WOhx2ja8+IGPVhe93jlgBUxHJl4=', 'Content-Length': '0'} 2023-11-02T18:44:04.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1034b56ba84c371-006543edf4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:44:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:04.906 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:44:04.907 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:04.907 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:04.908 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:44:04.908 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:44:04.908 DEBUG:boto:Method: GET 2023-11-02T18:44:04.909 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:44:04.909 DEBUG:boto:Data: 2023-11-02T18:44:04.910 DEBUG:boto:Headers: {} 2023-11-02T18:44:04.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:04.911 DEBUG:boto:Port: 8001 2023-11-02T18:44:04.911 DEBUG:boto:Params: {} 2023-11-02T18:44:04.912 DEBUG:boto:Token: None 2023-11-02T18:44:04.912 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:04 GMT /ibtljn-59/?versions 2023-11-02T18:44:04.913 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cs67BVVvGHaaaxyKtRmPoG3GeGM= 2023-11-02T18:44:04.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cs67BVVvGHaaaxyKtRmPoG3GeGM=', 'Content-Length': '0'} 2023-11-02T18:44:04.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfd3ea8c844b17c7-006543edf4-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:04.918 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:04.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:44:04.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T18:44:04.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-11-02T18:44:04.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-11-02T18:44:04.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-11-02T18:44:04.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-11-02T18:44:04.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:44:04.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:44:04.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:44:04.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T18:44:04.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-11-02T18:44:04.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-11-02T18:44:04.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:44:04.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:44:04.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:44:04.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:44:04.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T18:44:04.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:44:04.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:44:04.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:44:04.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-11-02T18:44:04.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-11-02T18:44:04.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:44:04.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T18:44:04.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:44:04.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:44:04.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-11-02T18:44:04.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T18:44:04.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-11-02T18:44:04.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:44:04.933 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:44:04.933 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:04.933 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:04.934 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:44:04.934 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:44:04.935 DEBUG:boto:Method: GET 2023-11-02T18:44:04.935 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:44:04.936 DEBUG:boto:Data: 2023-11-02T18:44:04.936 DEBUG:boto:Headers: {} 2023-11-02T18:44:04.937 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:04.937 DEBUG:boto:Port: 8000 2023-11-02T18:44:04.937 DEBUG:boto:Params: {} 2023-11-02T18:44:04.938 DEBUG:boto:Token: None 2023-11-02T18:44:04.938 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:04 GMT /ibtljn-59/?versions 2023-11-02T18:44:04.939 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cs67BVVvGHaaaxyKtRmPoG3GeGM= 2023-11-02T18:44:04.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cs67BVVvGHaaaxyKtRmPoG3GeGM=', 'Content-Length': '0'} 2023-11-02T18:44:04.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f6a048974f966c5-006543edf4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:04.948 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:04.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:44:04.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T18:44:04.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-11-02T18:44:04.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-11-02T18:44:04.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-11-02T18:44:04.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-11-02T18:44:04.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:44:04.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:44:04.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:44:04.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T18:44:04.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-11-02T18:44:04.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-11-02T18:44:04.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:44:04.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:44:04.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:44:04.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:44:04.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T18:44:04.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:44:04.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:44:04.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:44:04.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-11-02T18:44:04.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-11-02T18:44:04.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:44:04.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T18:44:04.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:44:04.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:44:04.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-11-02T18:44:04.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T18:44:04.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-11-02T18:44:04.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:44:04.962 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:04.962 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:04.963 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:44:04.963 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:44:04.964 DEBUG:boto:Method: GET 2023-11-02T18:44:04.964 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:44:04.965 DEBUG:boto:Data: 2023-11-02T18:44:04.965 DEBUG:boto:Headers: {} 2023-11-02T18:44:04.965 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:04.966 DEBUG:boto:Port: 8001 2023-11-02T18:44:04.966 DEBUG:boto:Params: {} 2023-11-02T18:44:04.967 DEBUG:boto:Token: None 2023-11-02T18:44:04.967 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:04 GMT /ibtljn-59/?versions 2023-11-02T18:44:04.968 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cs67BVVvGHaaaxyKtRmPoG3GeGM= 2023-11-02T18:44:04.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cs67BVVvGHaaaxyKtRmPoG3GeGM=', 'Content-Length': '0'} 2023-11-02T18:44:04.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ce39cfef322900d-006543edf4-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:04.974 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:04.976 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:04.976 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:04.977 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:44:04.977 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:44:04.977 DEBUG:boto:Method: GET 2023-11-02T18:44:04.978 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:44:04.978 DEBUG:boto:Data: 2023-11-02T18:44:04.979 DEBUG:boto:Headers: {} 2023-11-02T18:44:04.979 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:04.980 DEBUG:boto:Port: 8000 2023-11-02T18:44:04.980 DEBUG:boto:Params: {} 2023-11-02T18:44:04.980 DEBUG:boto:Token: None 2023-11-02T18:44:04.981 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:04 GMT /ibtljn-59/?versions 2023-11-02T18:44:04.981 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cs67BVVvGHaaaxyKtRmPoG3GeGM= 2023-11-02T18:44:04.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cs67BVVvGHaaaxyKtRmPoG3GeGM=', 'Content-Length': '0'} 2023-11-02T18:44:04.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000745a6e2be886e14-006543edf4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:04.993 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:04.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:44:04.995 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:44:04.996 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:44:04.996 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:44:04.997 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:44:04.997 DEBUG:boto:Method: GET 2023-11-02T18:44:04.998 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:44:04.998 DEBUG:boto:Data: 2023-11-02T18:44:04.999 DEBUG:boto:Headers: {} 2023-11-02T18:44:04.999 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:04.999 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.000 DEBUG:boto:Params: {} 2023-11-02T18:44:05.001 DEBUG:boto:Token: None 2023-11-02T18:44:05.001 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.002 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KFr8sAagglj7nbuWj91hALOscbI= 2023-11-02T18:44:05.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KFr8sAagglj7nbuWj91hALOscbI=', 'Content-Length': '0'} 2023-11-02T18:44:05.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035fb0963105f1a48-006543edf4-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.006 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:44:05.006 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:44:05.007 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.007 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.008 DEBUG:boto:Method: GET 2023-11-02T18:44:05.008 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.008 DEBUG:boto:Data: 2023-11-02T18:44:05.009 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.009 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.010 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.010 DEBUG:boto:Params: {} 2023-11-02T18:44:05.011 DEBUG:boto:Token: None 2023-11-02T18:44:05.011 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.012 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KFr8sAagglj7nbuWj91hALOscbI= 2023-11-02T18:44:05.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KFr8sAagglj7nbuWj91hALOscbI=', 'Content-Length': '0'} 2023-11-02T18:44:05.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be327f89d1df4c2c-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.058 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:44:05.059 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:44:05.059 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.060 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.060 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.061 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.061 DEBUG:boto:Data: 2023-11-02T18:44:05.061 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.062 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.062 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.063 DEBUG:boto:Params: {} 2023-11-02T18:44:05.063 DEBUG:boto:Token: None 2023-11-02T18:44:05.064 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.064 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZG33aHbqmZRMTUY2WqI3MHT1V5A= 2023-11-02T18:44:05.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZG33aHbqmZRMTUY2WqI3MHT1V5A=', 'Content-Length': '0'} 2023-11-02T18:44:05.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd8dffbf15277ec-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.069 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:44:05.069 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:44:05.069 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.070 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.070 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.071 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.071 DEBUG:boto:Data: 2023-11-02T18:44:05.071 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.072 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.072 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.073 DEBUG:boto:Params: {} 2023-11-02T18:44:05.073 DEBUG:boto:Token: None 2023-11-02T18:44:05.074 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.074 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZG33aHbqmZRMTUY2WqI3MHT1V5A= 2023-11-02T18:44:05.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZG33aHbqmZRMTUY2WqI3MHT1V5A=', 'Content-Length': '0'} 2023-11-02T18:44:05.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004eb386349d93c4f-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:44:05.079 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:44:05.079 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:44:05.080 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.080 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.081 DEBUG:boto:Method: GET 2023-11-02T18:44:05.081 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.081 DEBUG:boto:Data: 2023-11-02T18:44:05.082 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.082 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.083 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.083 DEBUG:boto:Params: {} 2023-11-02T18:44:05.084 DEBUG:boto:Token: None 2023-11-02T18:44:05.084 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.085 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KFr8sAagglj7nbuWj91hALOscbI= 2023-11-02T18:44:05.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KFr8sAagglj7nbuWj91hALOscbI=', 'Content-Length': '0'} 2023-11-02T18:44:05.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f558857db7e9de38-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.130 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:44:05.131 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:44:05.131 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.132 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.132 DEBUG:boto:Method: GET 2023-11-02T18:44:05.133 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.133 DEBUG:boto:Data: 2023-11-02T18:44:05.133 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.134 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.134 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.135 DEBUG:boto:Params: {} 2023-11-02T18:44:05.135 DEBUG:boto:Token: None 2023-11-02T18:44:05.136 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:44:05.136 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KFr8sAagglj7nbuWj91hALOscbI= 2023-11-02T18:44:05.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KFr8sAagglj7nbuWj91hALOscbI=', 'Content-Length': '0'} 2023-11-02T18:44:05.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000da0560ffc636040-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.141 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.141 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.141 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:44:05.142 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:44:05.142 DEBUG:boto:Method: GET 2023-11-02T18:44:05.143 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:44:05.143 DEBUG:boto:Data: 2023-11-02T18:44:05.144 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.144 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.145 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.145 DEBUG:boto:Params: {} 2023-11-02T18:44:05.146 DEBUG:boto:Token: None 2023-11-02T18:44:05.146 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.147 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003301e981b90bbd32-006543edf5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.153 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.154 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.154 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.155 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:44:05.155 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:44:05.155 DEBUG:boto:Method: GET 2023-11-02T18:44:05.156 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:44:05.156 DEBUG:boto:Data: 2023-11-02T18:44:05.157 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.157 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.158 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.158 DEBUG:boto:Params: {} 2023-11-02T18:44:05.159 DEBUG:boto:Token: None 2023-11-02T18:44:05.159 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.160 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8b90e5b17d981f8-006543edf5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.169 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.170 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:44:05.170 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:44:05.171 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.171 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:44:05.172 DEBUG:boto:Data: 2023-11-02T18:44:05.172 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.172 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.173 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.173 DEBUG:boto:Params: {} 2023-11-02T18:44:05.174 DEBUG:boto:Token: None 2023-11-02T18:44:05.174 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj11 2023-11-02T18:44:05.175 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ATQ2TS1F30rYlazqBQ0OXEwolA8= 2023-11-02T18:44:05.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ATQ2TS1F30rYlazqBQ0OXEwolA8=', 'Content-Length': '0'} 2023-11-02T18:44:05.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003095d4c3372ffe7f-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.179 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:44:05.179 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:44:05.179 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.180 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:44:05.180 DEBUG:boto:Data: 2023-11-02T18:44:05.181 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.181 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.182 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.182 DEBUG:boto:Params: {} 2023-11-02T18:44:05.183 DEBUG:boto:Token: None 2023-11-02T18:44:05.183 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj11 2023-11-02T18:44:05.183 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ATQ2TS1F30rYlazqBQ0OXEwolA8= 2023-11-02T18:44:05.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ATQ2TS1F30rYlazqBQ0OXEwolA8=', 'Content-Length': '0'} 2023-11-02T18:44:05.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000490dcd7cae256d04-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:44:05.188 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:44:05.188 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:44:05.188 DEBUG:boto:Method: GET 2023-11-02T18:44:05.189 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:44:05.189 DEBUG:boto:Data: 2023-11-02T18:44:05.190 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.190 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.191 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.191 DEBUG:boto:Params: {} 2023-11-02T18:44:05.192 DEBUG:boto:Token: None 2023-11-02T18:44:05.192 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj11 2023-11-02T18:44:05.192 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MwPHcS/m1iuBQosFaCgoi6A/4+M= 2023-11-02T18:44:05.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MwPHcS/m1iuBQosFaCgoi6A/4+M=', 'Content-Length': '0'} 2023-11-02T18:44:05.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f73618442411e103-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.196 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:44:05.197 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:44:05.197 DEBUG:boto:Method: GET 2023-11-02T18:44:05.198 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:44:05.198 DEBUG:boto:Data: 2023-11-02T18:44:05.199 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.199 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.199 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.200 DEBUG:boto:Params: {} 2023-11-02T18:44:05.200 DEBUG:boto:Token: None 2023-11-02T18:44:05.201 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj11 2023-11-02T18:44:05.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MwPHcS/m1iuBQosFaCgoi6A/4+M= 2023-11-02T18:44:05.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MwPHcS/m1iuBQosFaCgoi6A/4+M=', 'Content-Length': '0'} 2023-11-02T18:44:05.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000523a2e7b8a9d456-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.246 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.247 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.247 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:44:05.248 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:44:05.248 DEBUG:boto:Method: GET 2023-11-02T18:44:05.249 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:44:05.249 DEBUG:boto:Data: 2023-11-02T18:44:05.250 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.250 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.251 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.251 DEBUG:boto:Params: {} 2023-11-02T18:44:05.252 DEBUG:boto:Token: None 2023-11-02T18:44:05.252 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.253 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001119742e77cfb421-006543edf5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.259 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.260 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.261 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.261 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:44:05.262 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:44:05.262 DEBUG:boto:Method: GET 2023-11-02T18:44:05.263 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:44:05.263 DEBUG:boto:Data: 2023-11-02T18:44:05.264 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.264 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.264 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.265 DEBUG:boto:Params: {} 2023-11-02T18:44:05.265 DEBUG:boto:Token: None 2023-11-02T18:44:05.266 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.266 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092b1c25d5dda698a-006543edf5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.276 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:44:05.277 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:44:05.278 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:44:05.278 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.278 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.279 DEBUG:boto:Method: GET 2023-11-02T18:44:05.279 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.280 DEBUG:boto:Data: 2023-11-02T18:44:05.280 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.281 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.281 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.281 DEBUG:boto:Params: {} 2023-11-02T18:44:05.282 DEBUG:boto:Token: None 2023-11-02T18:44:05.283 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.283 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g9u0pA/bjIx6uTfwOW6X3xx2DVw= 2023-11-02T18:44:05.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g9u0pA/bjIx6uTfwOW6X3xx2DVw=', 'Content-Length': '0'} 2023-11-02T18:44:05.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fdb583fb391074c-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.330 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:44:05.331 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:44:05.331 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.332 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.332 DEBUG:boto:Method: GET 2023-11-02T18:44:05.333 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.333 DEBUG:boto:Data: 2023-11-02T18:44:05.334 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.334 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.334 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.335 DEBUG:boto:Params: {} 2023-11-02T18:44:05.335 DEBUG:boto:Token: None 2023-11-02T18:44:05.336 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.336 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g9u0pA/bjIx6uTfwOW6X3xx2DVw= 2023-11-02T18:44:05.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g9u0pA/bjIx6uTfwOW6X3xx2DVw=', 'Content-Length': '0'} 2023-11-02T18:44:05.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a591f6a50d6a71b7-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.341 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:44:05.342 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:44:05.342 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.343 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.343 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.343 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.344 DEBUG:boto:Data: 2023-11-02T18:44:05.344 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.345 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.345 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.346 DEBUG:boto:Params: {} 2023-11-02T18:44:05.346 DEBUG:boto:Token: None 2023-11-02T18:44:05.347 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.347 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:se7YGKiw82rX9fhkC8VG7QE3mYI= 2023-11-02T18:44:05.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:se7YGKiw82rX9fhkC8VG7QE3mYI=', 'Content-Length': '0'} 2023-11-02T18:44:05.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003903ffe5871beb9-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.351 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:44:05.352 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:44:05.352 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.353 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.353 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.353 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.354 DEBUG:boto:Data: 2023-11-02T18:44:05.354 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.355 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.355 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.356 DEBUG:boto:Params: {} 2023-11-02T18:44:05.356 DEBUG:boto:Token: None 2023-11-02T18:44:05.357 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.357 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:se7YGKiw82rX9fhkC8VG7QE3mYI= 2023-11-02T18:44:05.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:se7YGKiw82rX9fhkC8VG7QE3mYI=', 'Content-Length': '0'} 2023-11-02T18:44:05.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004552d6286d89eee9-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:44:05.361 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:44:05.362 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:44:05.362 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.363 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.363 DEBUG:boto:Method: GET 2023-11-02T18:44:05.364 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.364 DEBUG:boto:Data: 2023-11-02T18:44:05.364 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.365 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.365 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.366 DEBUG:boto:Params: {} 2023-11-02T18:44:05.366 DEBUG:boto:Token: None 2023-11-02T18:44:05.367 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.367 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g9u0pA/bjIx6uTfwOW6X3xx2DVw= 2023-11-02T18:44:05.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g9u0pA/bjIx6uTfwOW6X3xx2DVw=', 'Content-Length': '0'} 2023-11-02T18:44:05.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc8aa34211c8d73-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.371 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:44:05.372 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:44:05.372 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.373 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.373 DEBUG:boto:Method: GET 2023-11-02T18:44:05.374 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.374 DEBUG:boto:Data: 2023-11-02T18:44:05.375 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.375 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.375 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.376 DEBUG:boto:Params: {} 2023-11-02T18:44:05.376 DEBUG:boto:Token: None 2023-11-02T18:44:05.377 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:44:05.377 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g9u0pA/bjIx6uTfwOW6X3xx2DVw= 2023-11-02T18:44:05.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g9u0pA/bjIx6uTfwOW6X3xx2DVw=', 'Content-Length': '0'} 2023-11-02T18:44:05.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000250ff95f633a5bc5-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.382 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.382 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.383 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:44:05.383 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:44:05.384 DEBUG:boto:Method: GET 2023-11-02T18:44:05.384 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:44:05.385 DEBUG:boto:Data: 2023-11-02T18:44:05.385 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.385 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.386 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.386 DEBUG:boto:Params: {} 2023-11-02T18:44:05.387 DEBUG:boto:Token: None 2023-11-02T18:44:05.387 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.388 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034b9e82adf8bd6cf-006543edf5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.434 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.435 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.436 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.436 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:44:05.437 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:44:05.437 DEBUG:boto:Method: GET 2023-11-02T18:44:05.437 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:44:05.438 DEBUG:boto:Data: 2023-11-02T18:44:05.438 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.439 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.439 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.440 DEBUG:boto:Params: {} 2023-11-02T18:44:05.440 DEBUG:boto:Token: None 2023-11-02T18:44:05.441 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.441 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad8b6112fcf987b1-006543edf5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.450 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.451 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:44:05.452 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:44:05.452 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.452 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:44:05.453 DEBUG:boto:Data: 2023-11-02T18:44:05.453 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.454 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.455 DEBUG:boto:Params: {} 2023-11-02T18:44:05.455 DEBUG:boto:Token: None 2023-11-02T18:44:05.456 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj115 2023-11-02T18:44:05.456 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RIKu2uCl/KR6eULsaWjHlL/cZRk= 2023-11-02T18:44:05.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RIKu2uCl/KR6eULsaWjHlL/cZRk=', 'Content-Length': '0'} 2023-11-02T18:44:05.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4b39153ff48b469-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.460 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:44:05.460 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:44:05.461 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.461 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:44:05.462 DEBUG:boto:Data: 2023-11-02T18:44:05.462 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.463 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.463 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.463 DEBUG:boto:Params: {} 2023-11-02T18:44:05.464 DEBUG:boto:Token: None 2023-11-02T18:44:05.464 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj115 2023-11-02T18:44:05.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RIKu2uCl/KR6eULsaWjHlL/cZRk= 2023-11-02T18:44:05.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RIKu2uCl/KR6eULsaWjHlL/cZRk=', 'Content-Length': '0'} 2023-11-02T18:44:05.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f35dc6f755eff5cf-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:44:05.469 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:44:05.469 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:44:05.470 DEBUG:boto:Method: GET 2023-11-02T18:44:05.470 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:44:05.471 DEBUG:boto:Data: 2023-11-02T18:44:05.471 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.472 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.472 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.472 DEBUG:boto:Params: {} 2023-11-02T18:44:05.473 DEBUG:boto:Token: None 2023-11-02T18:44:05.473 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj115 2023-11-02T18:44:05.474 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UZAysQyMJWKKr7VvMDSPiPQsxoI= 2023-11-02T18:44:05.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UZAysQyMJWKKr7VvMDSPiPQsxoI=', 'Content-Length': '0'} 2023-11-02T18:44:05.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdf90b557acd6afa-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.478 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:44:05.478 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:44:05.478 DEBUG:boto:Method: GET 2023-11-02T18:44:05.479 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:44:05.479 DEBUG:boto:Data: 2023-11-02T18:44:05.480 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.480 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.481 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.481 DEBUG:boto:Params: {} 2023-11-02T18:44:05.482 DEBUG:boto:Token: None 2023-11-02T18:44:05.482 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj115 2023-11-02T18:44:05.483 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UZAysQyMJWKKr7VvMDSPiPQsxoI= 2023-11-02T18:44:05.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UZAysQyMJWKKr7VvMDSPiPQsxoI=', 'Content-Length': '0'} 2023-11-02T18:44:05.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7e9f9a864662e0b-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.530 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.531 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.531 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:44:05.532 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:44:05.532 DEBUG:boto:Method: GET 2023-11-02T18:44:05.533 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:44:05.533 DEBUG:boto:Data: 2023-11-02T18:44:05.534 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.534 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.535 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.535 DEBUG:boto:Params: {} 2023-11-02T18:44:05.536 DEBUG:boto:Token: None 2023-11-02T18:44:05.536 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.537 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c451bc9c9327f499-006543edf5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.543 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.544 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.545 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.545 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:44:05.546 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:44:05.546 DEBUG:boto:Method: GET 2023-11-02T18:44:05.546 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:44:05.547 DEBUG:boto:Data: 2023-11-02T18:44:05.547 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.548 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.548 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.549 DEBUG:boto:Params: {} 2023-11-02T18:44:05.549 DEBUG:boto:Token: None 2023-11-02T18:44:05.550 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.550 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000243996e3f0de7c4-006543edf5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.561 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:44:05.562 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:44:05.562 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:44:05.563 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.563 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.564 DEBUG:boto:Method: GET 2023-11-02T18:44:05.564 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.565 DEBUG:boto:Data: 2023-11-02T18:44:05.565 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.566 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.567 DEBUG:boto:Params: {} 2023-11-02T18:44:05.567 DEBUG:boto:Token: None 2023-11-02T18:44:05.568 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.568 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s7n7qwKyty946Och+raqISFRGHw= 2023-11-02T18:44:05.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s7n7qwKyty946Och+raqISFRGHw=', 'Content-Length': '0'} 2023-11-02T18:44:05.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf2b9d6e142ddeb9-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.618 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:44:05.619 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:44:05.619 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.620 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.620 DEBUG:boto:Method: GET 2023-11-02T18:44:05.621 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.621 DEBUG:boto:Data: 2023-11-02T18:44:05.621 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.622 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.622 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.623 DEBUG:boto:Params: {} 2023-11-02T18:44:05.623 DEBUG:boto:Token: None 2023-11-02T18:44:05.624 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s7n7qwKyty946Och+raqISFRGHw= 2023-11-02T18:44:05.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s7n7qwKyty946Och+raqISFRGHw=', 'Content-Length': '0'} 2023-11-02T18:44:05.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ae744eddf217d1b-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.630 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:44:05.630 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:44:05.631 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.631 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.631 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.632 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.633 DEBUG:boto:Data: 2023-11-02T18:44:05.634 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.634 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.634 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.635 DEBUG:boto:Params: {} 2023-11-02T18:44:05.635 DEBUG:boto:Token: None 2023-11-02T18:44:05.636 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.636 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WfzWtglNEmfjs2C0uZeMnmZeFjQ= 2023-11-02T18:44:05.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WfzWtglNEmfjs2C0uZeMnmZeFjQ=', 'Content-Length': '0'} 2023-11-02T18:44:05.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ddc8269987b67ed-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.639 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:44:05.640 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:44:05.640 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.641 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.641 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.642 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.642 DEBUG:boto:Data: 2023-11-02T18:44:05.642 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.643 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.643 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.644 DEBUG:boto:Params: {} 2023-11-02T18:44:05.644 DEBUG:boto:Token: None 2023-11-02T18:44:05.645 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WfzWtglNEmfjs2C0uZeMnmZeFjQ= 2023-11-02T18:44:05.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WfzWtglNEmfjs2C0uZeMnmZeFjQ=', 'Content-Length': '0'} 2023-11-02T18:44:05.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071cc3a8466fc74c5-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:44:05.649 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:44:05.650 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:44:05.650 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.651 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.651 DEBUG:boto:Method: GET 2023-11-02T18:44:05.651 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.652 DEBUG:boto:Data: 2023-11-02T18:44:05.652 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.653 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.653 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.654 DEBUG:boto:Params: {} 2023-11-02T18:44:05.654 DEBUG:boto:Token: None 2023-11-02T18:44:05.655 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.655 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s7n7qwKyty946Och+raqISFRGHw= 2023-11-02T18:44:05.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s7n7qwKyty946Och+raqISFRGHw=', 'Content-Length': '0'} 2023-11-02T18:44:05.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ec300920da930fe-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.659 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:44:05.659 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:44:05.660 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.660 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.661 DEBUG:boto:Method: GET 2023-11-02T18:44:05.662 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.662 DEBUG:boto:Data: 2023-11-02T18:44:05.663 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.664 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.664 DEBUG:boto:Params: {} 2023-11-02T18:44:05.665 DEBUG:boto:Token: None 2023-11-02T18:44:05.665 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:44:05.666 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s7n7qwKyty946Och+raqISFRGHw= 2023-11-02T18:44:05.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s7n7qwKyty946Och+raqISFRGHw=', 'Content-Length': '0'} 2023-11-02T18:44:05.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2fbd3af7a20b399-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.670 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.670 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.671 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:44:05.671 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:44:05.671 DEBUG:boto:Method: GET 2023-11-02T18:44:05.672 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:44:05.672 DEBUG:boto:Data: 2023-11-02T18:44:05.673 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.673 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.674 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.674 DEBUG:boto:Params: {} 2023-11-02T18:44:05.674 DEBUG:boto:Token: None 2023-11-02T18:44:05.675 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.675 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094167436cbf82057-006543edf5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.726 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.727 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.728 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.728 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:44:05.729 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:44:05.729 DEBUG:boto:Method: GET 2023-11-02T18:44:05.730 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:44:05.730 DEBUG:boto:Data: 2023-11-02T18:44:05.731 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.731 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.731 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.732 DEBUG:boto:Params: {} 2023-11-02T18:44:05.732 DEBUG:boto:Token: None 2023-11-02T18:44:05.733 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.733 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009712302a79ea1249-006543edf5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.743 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.744 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:44:05.744 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:44:05.745 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.745 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:44:05.746 DEBUG:boto:Data: 2023-11-02T18:44:05.746 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.747 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.747 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.747 DEBUG:boto:Params: {} 2023-11-02T18:44:05.748 DEBUG:boto:Token: None 2023-11-02T18:44:05.748 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj116 2023-11-02T18:44:05.749 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+h+lQnUuHEqzN4n4Y+3k9aB3IHk= 2023-11-02T18:44:05.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+h+lQnUuHEqzN4n4Y+3k9aB3IHk=', 'Content-Length': '0'} 2023-11-02T18:44:05.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027831b2b82cdc8e6-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.752 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:44:05.753 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:44:05.753 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.754 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:44:05.754 DEBUG:boto:Data: 2023-11-02T18:44:05.755 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.755 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.755 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.756 DEBUG:boto:Params: {} 2023-11-02T18:44:05.756 DEBUG:boto:Token: None 2023-11-02T18:44:05.757 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj116 2023-11-02T18:44:05.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+h+lQnUuHEqzN4n4Y+3k9aB3IHk= 2023-11-02T18:44:05.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+h+lQnUuHEqzN4n4Y+3k9aB3IHk=', 'Content-Length': '0'} 2023-11-02T18:44:05.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f5dc5cc6d553136-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:44:05.761 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:44:05.762 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:44:05.762 DEBUG:boto:Method: GET 2023-11-02T18:44:05.763 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:44:05.763 DEBUG:boto:Data: 2023-11-02T18:44:05.764 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.764 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.765 DEBUG:boto:Params: {} 2023-11-02T18:44:05.765 DEBUG:boto:Token: None 2023-11-02T18:44:05.766 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj116 2023-11-02T18:44:05.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+y5todRiN78MxQGMvj/MMye9yI0= 2023-11-02T18:44:05.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+y5todRiN78MxQGMvj/MMye9yI0=', 'Content-Length': '0'} 2023-11-02T18:44:05.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068fafe9a48d81a70-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.770 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:44:05.771 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:44:05.771 DEBUG:boto:Method: GET 2023-11-02T18:44:05.772 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:44:05.772 DEBUG:boto:Data: 2023-11-02T18:44:05.773 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.773 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.774 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.774 DEBUG:boto:Params: {} 2023-11-02T18:44:05.774 DEBUG:boto:Token: None 2023-11-02T18:44:05.775 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj116 2023-11-02T18:44:05.775 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+y5todRiN78MxQGMvj/MMye9yI0= 2023-11-02T18:44:05.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+y5todRiN78MxQGMvj/MMye9yI0=', 'Content-Length': '0'} 2023-11-02T18:44:05.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a94751a5ed6846cf-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.822 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.823 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.823 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:44:05.824 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:44:05.824 DEBUG:boto:Method: GET 2023-11-02T18:44:05.825 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:44:05.825 DEBUG:boto:Data: 2023-11-02T18:44:05.826 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.826 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.827 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.827 DEBUG:boto:Params: {} 2023-11-02T18:44:05.828 DEBUG:boto:Token: None 2023-11-02T18:44:05.828 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.829 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b190484a28cc17c-006543edf5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.835 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.835 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.836 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.836 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:44:05.837 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:44:05.837 DEBUG:boto:Method: GET 2023-11-02T18:44:05.838 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:44:05.838 DEBUG:boto:Data: 2023-11-02T18:44:05.839 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.839 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.840 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.840 DEBUG:boto:Params: {} 2023-11-02T18:44:05.841 DEBUG:boto:Token: None 2023-11-02T18:44:05.841 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.842 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091df6d76ff9bea91-006543edf5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.853 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T18:44:05.854 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:44:05.854 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:44:05.855 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.855 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.856 DEBUG:boto:Method: GET 2023-11-02T18:44:05.856 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.857 DEBUG:boto:Data: 2023-11-02T18:44:05.857 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.858 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.858 DEBUG:boto:Params: {} 2023-11-02T18:44:05.859 DEBUG:boto:Token: None 2023-11-02T18:44:05.859 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.860 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5hygrYr0+a9e4rDXDxgsiRJ7Wy0= 2023-11-02T18:44:05.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5hygrYr0+a9e4rDXDxgsiRJ7Wy0=', 'Content-Length': '0'} 2023-11-02T18:44:05.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cf63e0780ed7cad-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.863 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:44:05.864 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:44:05.864 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.865 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.865 DEBUG:boto:Method: GET 2023-11-02T18:44:05.866 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.866 DEBUG:boto:Data: 2023-11-02T18:44:05.867 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.867 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.867 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.868 DEBUG:boto:Params: {} 2023-11-02T18:44:05.868 DEBUG:boto:Token: None 2023-11-02T18:44:05.869 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.869 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5hygrYr0+a9e4rDXDxgsiRJ7Wy0= 2023-11-02T18:44:05.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5hygrYr0+a9e4rDXDxgsiRJ7Wy0=', 'Content-Length': '0'} 2023-11-02T18:44:05.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006952bcd7c88e9111-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.873 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:44:05.874 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:44:05.874 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.875 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.875 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.875 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.876 DEBUG:boto:Data: 2023-11-02T18:44:05.876 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.877 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.877 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.878 DEBUG:boto:Params: {} 2023-11-02T18:44:05.878 DEBUG:boto:Token: None 2023-11-02T18:44:05.878 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.879 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9/kKEA7D4JbHGdNleqEj7pjDvuo= 2023-11-02T18:44:05.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9/kKEA7D4JbHGdNleqEj7pjDvuo=', 'Content-Length': '0'} 2023-11-02T18:44:05.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f38ebbbc71dbd8-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.883 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:44:05.883 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:44:05.884 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.884 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.884 DEBUG:boto:Method: HEAD 2023-11-02T18:44:05.885 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.885 DEBUG:boto:Data: 2023-11-02T18:44:05.886 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.886 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.887 DEBUG:boto:Params: {} 2023-11-02T18:44:05.887 DEBUG:boto:Token: None 2023-11-02T18:44:05.888 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.888 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9/kKEA7D4JbHGdNleqEj7pjDvuo= 2023-11-02T18:44:05.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9/kKEA7D4JbHGdNleqEj7pjDvuo=', 'Content-Length': '0'} 2023-11-02T18:44:05.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086fac1bdd5a24f45-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T18:44:05.893 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:44:05.893 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:44:05.894 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.894 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.894 DEBUG:boto:Method: GET 2023-11-02T18:44:05.895 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.895 DEBUG:boto:Data: 2023-11-02T18:44:05.896 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.896 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.897 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.897 DEBUG:boto:Params: {} 2023-11-02T18:44:05.897 DEBUG:boto:Token: None 2023-11-02T18:44:05.898 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.899 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5hygrYr0+a9e4rDXDxgsiRJ7Wy0= 2023-11-02T18:44:05.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5hygrYr0+a9e4rDXDxgsiRJ7Wy0=', 'Content-Length': '0'} 2023-11-02T18:44:05.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdbe6c174a5e151c-006543edf5-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.946 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:44:05.947 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:44:05.947 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.948 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.948 DEBUG:boto:Method: GET 2023-11-02T18:44:05.949 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.949 DEBUG:boto:Data: 2023-11-02T18:44:05.950 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.951 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.951 DEBUG:boto:Params: {} 2023-11-02T18:44:05.952 DEBUG:boto:Token: None 2023-11-02T18:44:05.952 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:44:05.953 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5hygrYr0+a9e4rDXDxgsiRJ7Wy0= 2023-11-02T18:44:05.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5hygrYr0+a9e4rDXDxgsiRJ7Wy0=', 'Content-Length': '0'} 2023-11-02T18:44:05.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052fc0b15f688d213-006543edf5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.957 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.958 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.958 DEBUG:boto:path=/ibtljn-59/?prefix=obj124 2023-11-02T18:44:05.958 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj124 2023-11-02T18:44:05.959 DEBUG:boto:Method: GET 2023-11-02T18:44:05.959 DEBUG:boto:Path: /ibtljn-59/?prefix=obj124 2023-11-02T18:44:05.960 DEBUG:boto:Data: 2023-11-02T18:44:05.960 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.960 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:05.961 DEBUG:boto:Port: 8001 2023-11-02T18:44:05.961 DEBUG:boto:Params: {} 2023-11-02T18:44:05.962 DEBUG:boto:Token: None 2023-11-02T18:44:05.962 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.963 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e2d567d0c5eb063-006543edf5-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:05.969 DEBUG:boto:b'ibtljn-59obj1241000falseobj1242023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:05.970 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:05.970 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:05.971 DEBUG:boto:path=/ibtljn-59/?prefix=obj124 2023-11-02T18:44:05.971 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj124 2023-11-02T18:44:05.972 DEBUG:boto:Method: GET 2023-11-02T18:44:05.972 DEBUG:boto:Path: /ibtljn-59/?prefix=obj124 2023-11-02T18:44:05.972 DEBUG:boto:Data: 2023-11-02T18:44:05.973 DEBUG:boto:Headers: {} 2023-11-02T18:44:05.973 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:05.974 DEBUG:boto:Port: 8000 2023-11-02T18:44:05.974 DEBUG:boto:Params: {} 2023-11-02T18:44:05.975 DEBUG:boto:Token: None 2023-11-02T18:44:05.975 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:05 GMT /ibtljn-59/ 2023-11-02T18:44:05.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4= 2023-11-02T18:44:05.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DZ0xHN0HEq11Cq9qgc8wI4t7gY4=', 'Content-Length': '0'} 2023-11-02T18:44:05.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0d3923ec100f3b8-006543edf5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.026 DEBUG:boto:b'ibtljn-59obj1241000falseobj1242023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.027 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:44:06.028 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:44:06.028 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.028 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:44:06.029 DEBUG:boto:Data: 2023-11-02T18:44:06.029 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.030 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.030 DEBUG:boto:Params: {} 2023-11-02T18:44:06.031 DEBUG:boto:Token: None 2023-11-02T18:44:06.031 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj124 2023-11-02T18:44:06.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:U5SMtDU72gLLaESAnkeNB2wFuwk= 2023-11-02T18:44:06.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:U5SMtDU72gLLaESAnkeNB2wFuwk=', 'Content-Length': '0'} 2023-11-02T18:44:06.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3d1d0f35896cd1e-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.036 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:44:06.037 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:44:06.037 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.038 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:44:06.038 DEBUG:boto:Data: 2023-11-02T18:44:06.038 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.039 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.039 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.040 DEBUG:boto:Params: {} 2023-11-02T18:44:06.040 DEBUG:boto:Token: None 2023-11-02T18:44:06.041 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj124 2023-11-02T18:44:06.041 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:U5SMtDU72gLLaESAnkeNB2wFuwk= 2023-11-02T18:44:06.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:U5SMtDU72gLLaESAnkeNB2wFuwk=', 'Content-Length': '0'} 2023-11-02T18:44:06.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049cb24be48872b35-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T18:44:06.046 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:44:06.046 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:44:06.047 DEBUG:boto:Method: GET 2023-11-02T18:44:06.047 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:44:06.048 DEBUG:boto:Data: 2023-11-02T18:44:06.048 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.049 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.049 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.049 DEBUG:boto:Params: {} 2023-11-02T18:44:06.050 DEBUG:boto:Token: None 2023-11-02T18:44:06.050 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj124 2023-11-02T18:44:06.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uXpolTe/qG4B2RNpGSbA+1YJGQU= 2023-11-02T18:44:06.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uXpolTe/qG4B2RNpGSbA+1YJGQU=', 'Content-Length': '0'} 2023-11-02T18:44:06.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc8198dd0eb2fb33-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.055 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:44:06.055 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:44:06.056 DEBUG:boto:Method: GET 2023-11-02T18:44:06.056 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:44:06.057 DEBUG:boto:Data: 2023-11-02T18:44:06.057 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.058 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.058 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.058 DEBUG:boto:Params: {} 2023-11-02T18:44:06.059 DEBUG:boto:Token: None 2023-11-02T18:44:06.059 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj124 2023-11-02T18:44:06.060 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uXpolTe/qG4B2RNpGSbA+1YJGQU= 2023-11-02T18:44:06.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uXpolTe/qG4B2RNpGSbA+1YJGQU=', 'Content-Length': '0'} 2023-11-02T18:44:06.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bce782a612c9c696-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.064 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.065 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.065 DEBUG:boto:path=/ibtljn-59/?prefix=obj124 2023-11-02T18:44:06.066 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj124 2023-11-02T18:44:06.066 DEBUG:boto:Method: GET 2023-11-02T18:44:06.067 DEBUG:boto:Path: /ibtljn-59/?prefix=obj124 2023-11-02T18:44:06.067 DEBUG:boto:Data: 2023-11-02T18:44:06.067 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.068 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.068 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.069 DEBUG:boto:Params: {} 2023-11-02T18:44:06.069 DEBUG:boto:Token: None 2023-11-02T18:44:06.070 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.070 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d21e2dea30556e-006543edf6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.118 DEBUG:boto:b'ibtljn-59obj1241000falseobj1242023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.119 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.120 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.120 DEBUG:boto:path=/ibtljn-59/?prefix=obj124 2023-11-02T18:44:06.120 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj124 2023-11-02T18:44:06.121 DEBUG:boto:Method: GET 2023-11-02T18:44:06.121 DEBUG:boto:Path: /ibtljn-59/?prefix=obj124 2023-11-02T18:44:06.122 DEBUG:boto:Data: 2023-11-02T18:44:06.122 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.123 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.123 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.124 DEBUG:boto:Params: {} 2023-11-02T18:44:06.124 DEBUG:boto:Token: None 2023-11-02T18:44:06.125 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.125 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8c6ae3b3499ec7c-006543edf6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.136 DEBUG:boto:b'ibtljn-59obj1241000falseobj1242023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:44:06.137 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:44:06.137 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:44:06.138 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.138 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.139 DEBUG:boto:Method: GET 2023-11-02T18:44:06.139 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.140 DEBUG:boto:Data: 2023-11-02T18:44:06.140 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.141 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.141 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.142 DEBUG:boto:Params: {} 2023-11-02T18:44:06.142 DEBUG:boto:Token: None 2023-11-02T18:44:06.143 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.143 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mH6N5ppuLWmpclIj/ZxC7StIvhs= 2023-11-02T18:44:06.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mH6N5ppuLWmpclIj/ZxC7StIvhs=', 'Content-Length': '0'} 2023-11-02T18:44:06.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001073134ca4240c22-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.147 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:44:06.148 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:44:06.148 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.149 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.149 DEBUG:boto:Method: GET 2023-11-02T18:44:06.150 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.150 DEBUG:boto:Data: 2023-11-02T18:44:06.150 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.151 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.151 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.152 DEBUG:boto:Params: {} 2023-11-02T18:44:06.152 DEBUG:boto:Token: None 2023-11-02T18:44:06.153 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.153 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mH6N5ppuLWmpclIj/ZxC7StIvhs= 2023-11-02T18:44:06.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mH6N5ppuLWmpclIj/ZxC7StIvhs=', 'Content-Length': '0'} 2023-11-02T18:44:06.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c79f5e1e44f75c-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.157 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:44:06.158 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:44:06.158 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.159 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.159 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.160 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.160 DEBUG:boto:Data: 2023-11-02T18:44:06.160 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.161 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.161 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.162 DEBUG:boto:Params: {} 2023-11-02T18:44:06.162 DEBUG:boto:Token: None 2023-11-02T18:44:06.163 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.163 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ulfxBQ04VrJJg/4ENf95ngpPCHk= 2023-11-02T18:44:06.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ulfxBQ04VrJJg/4ENf95ngpPCHk=', 'Content-Length': '0'} 2023-11-02T18:44:06.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcba64af6f69b801-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.167 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:44:06.168 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:44:06.168 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.169 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.169 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.169 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.170 DEBUG:boto:Data: 2023-11-02T18:44:06.170 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.171 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.171 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.172 DEBUG:boto:Params: {} 2023-11-02T18:44:06.172 DEBUG:boto:Token: None 2023-11-02T18:44:06.172 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.173 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ulfxBQ04VrJJg/4ENf95ngpPCHk= 2023-11-02T18:44:06.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ulfxBQ04VrJJg/4ENf95ngpPCHk=', 'Content-Length': '0'} 2023-11-02T18:44:06.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001bf8ead07620873-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:44:06.178 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:44:06.178 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:44:06.179 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.179 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.180 DEBUG:boto:Method: GET 2023-11-02T18:44:06.180 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.181 DEBUG:boto:Data: 2023-11-02T18:44:06.181 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.182 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.182 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.182 DEBUG:boto:Params: {} 2023-11-02T18:44:06.183 DEBUG:boto:Token: None 2023-11-02T18:44:06.183 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.184 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mH6N5ppuLWmpclIj/ZxC7StIvhs= 2023-11-02T18:44:06.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mH6N5ppuLWmpclIj/ZxC7StIvhs=', 'Content-Length': '0'} 2023-11-02T18:44:06.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fee45c8789602ee7-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.234 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:44:06.235 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:44:06.235 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.236 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.236 DEBUG:boto:Method: GET 2023-11-02T18:44:06.237 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.237 DEBUG:boto:Data: 2023-11-02T18:44:06.238 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.238 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.239 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.239 DEBUG:boto:Params: {} 2023-11-02T18:44:06.239 DEBUG:boto:Token: None 2023-11-02T18:44:06.240 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:44:06.240 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mH6N5ppuLWmpclIj/ZxC7StIvhs= 2023-11-02T18:44:06.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mH6N5ppuLWmpclIj/ZxC7StIvhs=', 'Content-Length': '0'} 2023-11-02T18:44:06.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc14e165ad6379a9-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.245 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.245 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.246 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:44:06.246 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:44:06.247 DEBUG:boto:Method: GET 2023-11-02T18:44:06.247 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:44:06.248 DEBUG:boto:Data: 2023-11-02T18:44:06.248 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.248 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.249 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.249 DEBUG:boto:Params: {} 2023-11-02T18:44:06.250 DEBUG:boto:Token: None 2023-11-02T18:44:06.250 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.251 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3d7d142ec7b929f-006543edf6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.257 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.258 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.259 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.259 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:44:06.259 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:44:06.260 DEBUG:boto:Method: GET 2023-11-02T18:44:06.260 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:44:06.261 DEBUG:boto:Data: 2023-11-02T18:44:06.261 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.261 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.262 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.262 DEBUG:boto:Params: {} 2023-11-02T18:44:06.263 DEBUG:boto:Token: None 2023-11-02T18:44:06.263 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.264 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7ae106acaba9b97-006543edf6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.314 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.315 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:44:06.317 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:44:06.317 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.318 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:44:06.318 DEBUG:boto:Data: 2023-11-02T18:44:06.319 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.319 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.319 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.320 DEBUG:boto:Params: {} 2023-11-02T18:44:06.320 DEBUG:boto:Token: None 2023-11-02T18:44:06.321 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj145 2023-11-02T18:44:06.321 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fx4tQe1S1JDTPv5rTTUnl3dyWmk= 2023-11-02T18:44:06.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fx4tQe1S1JDTPv5rTTUnl3dyWmk=', 'Content-Length': '0'} 2023-11-02T18:44:06.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000913f23e04b9ea0ec-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.326 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:44:06.326 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:44:06.326 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.327 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:44:06.327 DEBUG:boto:Data: 2023-11-02T18:44:06.328 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.329 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.329 DEBUG:boto:Params: {} 2023-11-02T18:44:06.330 DEBUG:boto:Token: None 2023-11-02T18:44:06.330 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj145 2023-11-02T18:44:06.331 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fx4tQe1S1JDTPv5rTTUnl3dyWmk= 2023-11-02T18:44:06.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fx4tQe1S1JDTPv5rTTUnl3dyWmk=', 'Content-Length': '0'} 2023-11-02T18:44:06.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ec307389112509f-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:44:06.335 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:44:06.336 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:44:06.337 DEBUG:boto:Method: GET 2023-11-02T18:44:06.338 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:44:06.338 DEBUG:boto:Data: 2023-11-02T18:44:06.339 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.339 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.340 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.340 DEBUG:boto:Params: {} 2023-11-02T18:44:06.341 DEBUG:boto:Token: None 2023-11-02T18:44:06.341 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj145 2023-11-02T18:44:06.342 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8dHZJ5axIHg6qAaRmpDZMGcxkY0= 2023-11-02T18:44:06.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8dHZJ5axIHg6qAaRmpDZMGcxkY0=', 'Content-Length': '0'} 2023-11-02T18:44:06.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001afd42aa6652e21e-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.346 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:44:06.347 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:44:06.347 DEBUG:boto:Method: GET 2023-11-02T18:44:06.347 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:44:06.348 DEBUG:boto:Data: 2023-11-02T18:44:06.348 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.349 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.349 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.350 DEBUG:boto:Params: {} 2023-11-02T18:44:06.350 DEBUG:boto:Token: None 2023-11-02T18:44:06.351 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj145 2023-11-02T18:44:06.352 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8dHZJ5axIHg6qAaRmpDZMGcxkY0= 2023-11-02T18:44:06.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8dHZJ5axIHg6qAaRmpDZMGcxkY0=', 'Content-Length': '0'} 2023-11-02T18:44:06.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cfa72c369219329-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.356 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.356 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.357 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:44:06.357 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:44:06.358 DEBUG:boto:Method: GET 2023-11-02T18:44:06.358 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:44:06.359 DEBUG:boto:Data: 2023-11-02T18:44:06.359 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.360 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.360 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.361 DEBUG:boto:Params: {} 2023-11-02T18:44:06.361 DEBUG:boto:Token: None 2023-11-02T18:44:06.362 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.362 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfa5785ad26ed132-006543edf6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.369 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.369 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.370 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.370 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:44:06.371 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:44:06.371 DEBUG:boto:Method: GET 2023-11-02T18:44:06.372 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:44:06.372 DEBUG:boto:Data: 2023-11-02T18:44:06.372 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.373 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.373 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.374 DEBUG:boto:Params: {} 2023-11-02T18:44:06.374 DEBUG:boto:Token: None 2023-11-02T18:44:06.375 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.375 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054076bff966a5324-006543edf6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.386 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T18:44:06.387 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:44:06.387 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:44:06.388 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.388 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.389 DEBUG:boto:Method: GET 2023-11-02T18:44:06.389 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.390 DEBUG:boto:Data: 2023-11-02T18:44:06.390 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.391 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.391 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.391 DEBUG:boto:Params: {} 2023-11-02T18:44:06.392 DEBUG:boto:Token: None 2023-11-02T18:44:06.392 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.393 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v5b25GIY9Pz+51SiHa8q2YPQ7C0= 2023-11-02T18:44:06.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v5b25GIY9Pz+51SiHa8q2YPQ7C0=', 'Content-Length': '0'} 2023-11-02T18:44:06.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062e376fe7ba64ebe-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.397 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:44:06.397 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:44:06.398 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.398 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.399 DEBUG:boto:Method: GET 2023-11-02T18:44:06.399 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.400 DEBUG:boto:Data: 2023-11-02T18:44:06.400 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.400 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.401 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.401 DEBUG:boto:Params: {} 2023-11-02T18:44:06.402 DEBUG:boto:Token: None 2023-11-02T18:44:06.402 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.403 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v5b25GIY9Pz+51SiHa8q2YPQ7C0= 2023-11-02T18:44:06.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v5b25GIY9Pz+51SiHa8q2YPQ7C0=', 'Content-Length': '0'} 2023-11-02T18:44:06.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005401e58d410561d-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.446 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:44:06.447 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:44:06.447 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.448 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.448 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.449 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.449 DEBUG:boto:Data: 2023-11-02T18:44:06.449 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.450 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.450 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.451 DEBUG:boto:Params: {} 2023-11-02T18:44:06.451 DEBUG:boto:Token: None 2023-11-02T18:44:06.452 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.452 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m0iZpFev0h+BNhmYA1CQEicaIBA= 2023-11-02T18:44:06.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m0iZpFev0h+BNhmYA1CQEicaIBA=', 'Content-Length': '0'} 2023-11-02T18:44:06.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007877e3b1ac2dab44-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.456 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:44:06.457 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:44:06.457 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.458 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.458 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.459 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.459 DEBUG:boto:Data: 2023-11-02T18:44:06.459 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.460 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.460 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.461 DEBUG:boto:Params: {} 2023-11-02T18:44:06.461 DEBUG:boto:Token: None 2023-11-02T18:44:06.462 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.462 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m0iZpFev0h+BNhmYA1CQEicaIBA= 2023-11-02T18:44:06.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m0iZpFev0h+BNhmYA1CQEicaIBA=', 'Content-Length': '0'} 2023-11-02T18:44:06.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcd46d0d8938e167-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T18:44:06.467 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:44:06.467 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:44:06.467 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.468 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.468 DEBUG:boto:Method: GET 2023-11-02T18:44:06.469 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.469 DEBUG:boto:Data: 2023-11-02T18:44:06.469 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.470 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.470 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.471 DEBUG:boto:Params: {} 2023-11-02T18:44:06.471 DEBUG:boto:Token: None 2023-11-02T18:44:06.472 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.472 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v5b25GIY9Pz+51SiHa8q2YPQ7C0= 2023-11-02T18:44:06.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v5b25GIY9Pz+51SiHa8q2YPQ7C0=', 'Content-Length': '0'} 2023-11-02T18:44:06.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a53eb4dadac6235-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.518 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:44:06.519 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:44:06.519 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.520 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.520 DEBUG:boto:Method: GET 2023-11-02T18:44:06.521 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.521 DEBUG:boto:Data: 2023-11-02T18:44:06.522 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.523 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.523 DEBUG:boto:Params: {} 2023-11-02T18:44:06.523 DEBUG:boto:Token: None 2023-11-02T18:44:06.524 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:44:06.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v5b25GIY9Pz+51SiHa8q2YPQ7C0= 2023-11-02T18:44:06.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v5b25GIY9Pz+51SiHa8q2YPQ7C0=', 'Content-Length': '0'} 2023-11-02T18:44:06.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024e1b5aaa9356be9-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.529 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.529 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.530 DEBUG:boto:path=/ibtljn-59/?prefix=obj155 2023-11-02T18:44:06.530 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj155 2023-11-02T18:44:06.531 DEBUG:boto:Method: GET 2023-11-02T18:44:06.531 DEBUG:boto:Path: /ibtljn-59/?prefix=obj155 2023-11-02T18:44:06.532 DEBUG:boto:Data: 2023-11-02T18:44:06.532 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.533 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.533 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.533 DEBUG:boto:Params: {} 2023-11-02T18:44:06.534 DEBUG:boto:Token: None 2023-11-02T18:44:06.534 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.535 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077eadfd5da70f91b-006543edf6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.543 DEBUG:boto:b'ibtljn-59obj1551000falseobj1552023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.544 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.544 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.544 DEBUG:boto:path=/ibtljn-59/?prefix=obj155 2023-11-02T18:44:06.545 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj155 2023-11-02T18:44:06.545 DEBUG:boto:Method: GET 2023-11-02T18:44:06.546 DEBUG:boto:Path: /ibtljn-59/?prefix=obj155 2023-11-02T18:44:06.546 DEBUG:boto:Data: 2023-11-02T18:44:06.547 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.547 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.548 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.548 DEBUG:boto:Params: {} 2023-11-02T18:44:06.549 DEBUG:boto:Token: None 2023-11-02T18:44:06.549 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.550 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e8854229678a946-006543edf6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.560 DEBUG:boto:b'ibtljn-59obj1551000falseobj1552023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.561 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:44:06.562 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:44:06.562 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.563 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:44:06.563 DEBUG:boto:Data: 2023-11-02T18:44:06.564 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.565 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.565 DEBUG:boto:Params: {} 2023-11-02T18:44:06.565 DEBUG:boto:Token: None 2023-11-02T18:44:06.566 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj155 2023-11-02T18:44:06.566 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6+0bmUQ3RPuWp7DsfiybegWzv90= 2023-11-02T18:44:06.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6+0bmUQ3RPuWp7DsfiybegWzv90=', 'Content-Length': '0'} 2023-11-02T18:44:06.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eac77145ecaad02-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.570 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:44:06.571 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:44:06.571 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.572 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:44:06.572 DEBUG:boto:Data: 2023-11-02T18:44:06.572 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.573 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.573 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.574 DEBUG:boto:Params: {} 2023-11-02T18:44:06.574 DEBUG:boto:Token: None 2023-11-02T18:44:06.575 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj155 2023-11-02T18:44:06.575 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6+0bmUQ3RPuWp7DsfiybegWzv90= 2023-11-02T18:44:06.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6+0bmUQ3RPuWp7DsfiybegWzv90=', 'Content-Length': '0'} 2023-11-02T18:44:06.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000727bb03662f2280-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T18:44:06.579 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:44:06.580 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:44:06.580 DEBUG:boto:Method: GET 2023-11-02T18:44:06.581 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:44:06.581 DEBUG:boto:Data: 2023-11-02T18:44:06.581 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.582 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.582 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.583 DEBUG:boto:Params: {} 2023-11-02T18:44:06.583 DEBUG:boto:Token: None 2023-11-02T18:44:06.584 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj155 2023-11-02T18:44:06.584 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Lj2awT/9hZaenaEytesB/FBLVxs= 2023-11-02T18:44:06.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Lj2awT/9hZaenaEytesB/FBLVxs=', 'Content-Length': '0'} 2023-11-02T18:44:06.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e7b9d45113874b2-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.588 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:44:06.589 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:44:06.589 DEBUG:boto:Method: GET 2023-11-02T18:44:06.590 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:44:06.590 DEBUG:boto:Data: 2023-11-02T18:44:06.590 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.591 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.591 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.592 DEBUG:boto:Params: {} 2023-11-02T18:44:06.592 DEBUG:boto:Token: None 2023-11-02T18:44:06.593 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj155 2023-11-02T18:44:06.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Lj2awT/9hZaenaEytesB/FBLVxs= 2023-11-02T18:44:06.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Lj2awT/9hZaenaEytesB/FBLVxs=', 'Content-Length': '0'} 2023-11-02T18:44:06.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004adf49dbe75840fd-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.638 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.639 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.639 DEBUG:boto:path=/ibtljn-59/?prefix=obj155 2023-11-02T18:44:06.640 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj155 2023-11-02T18:44:06.640 DEBUG:boto:Method: GET 2023-11-02T18:44:06.640 DEBUG:boto:Path: /ibtljn-59/?prefix=obj155 2023-11-02T18:44:06.641 DEBUG:boto:Data: 2023-11-02T18:44:06.641 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.642 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.642 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.642 DEBUG:boto:Params: {} 2023-11-02T18:44:06.643 DEBUG:boto:Token: None 2023-11-02T18:44:06.644 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee0a73fea864895e-006543edf6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.651 DEBUG:boto:b'ibtljn-59obj1551000falseobj1552023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.652 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.652 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.652 DEBUG:boto:path=/ibtljn-59/?prefix=obj155 2023-11-02T18:44:06.653 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj155 2023-11-02T18:44:06.653 DEBUG:boto:Method: GET 2023-11-02T18:44:06.654 DEBUG:boto:Path: /ibtljn-59/?prefix=obj155 2023-11-02T18:44:06.654 DEBUG:boto:Data: 2023-11-02T18:44:06.655 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.655 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.656 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.656 DEBUG:boto:Params: {} 2023-11-02T18:44:06.656 DEBUG:boto:Token: None 2023-11-02T18:44:06.657 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.657 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7504ec235876702-006543edf6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.670 DEBUG:boto:b'ibtljn-59obj1551000falseobj1552023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:44:06.671 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:44:06.671 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:44:06.672 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.672 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.673 DEBUG:boto:Method: GET 2023-11-02T18:44:06.673 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.673 DEBUG:boto:Data: 2023-11-02T18:44:06.674 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.675 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.675 DEBUG:boto:Params: {} 2023-11-02T18:44:06.675 DEBUG:boto:Token: None 2023-11-02T18:44:06.676 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gJMpnO+C539ol35yDrH8olTY89Q= 2023-11-02T18:44:06.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gJMpnO+C539ol35yDrH8olTY89Q=', 'Content-Length': '0'} 2023-11-02T18:44:06.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006435d226821a21a9-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.726 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:44:06.727 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:44:06.727 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.728 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.728 DEBUG:boto:Method: GET 2023-11-02T18:44:06.728 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.729 DEBUG:boto:Data: 2023-11-02T18:44:06.729 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.730 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.730 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.731 DEBUG:boto:Params: {} 2023-11-02T18:44:06.731 DEBUG:boto:Token: None 2023-11-02T18:44:06.732 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.732 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gJMpnO+C539ol35yDrH8olTY89Q= 2023-11-02T18:44:06.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gJMpnO+C539ol35yDrH8olTY89Q=', 'Content-Length': '0'} 2023-11-02T18:44:06.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc91a2d2886d96f5-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.736 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:44:06.737 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:44:06.738 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.738 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.739 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.739 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.740 DEBUG:boto:Data: 2023-11-02T18:44:06.740 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.740 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.741 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.741 DEBUG:boto:Params: {} 2023-11-02T18:44:06.741 DEBUG:boto:Token: None 2023-11-02T18:44:06.742 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.742 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3cDZJTL+4av7jlbs+EunIWmVatg= 2023-11-02T18:44:06.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3cDZJTL+4av7jlbs+EunIWmVatg=', 'Content-Length': '0'} 2023-11-02T18:44:06.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0b194a172676435-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.749 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:44:06.749 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:44:06.750 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.751 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.751 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.751 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.752 DEBUG:boto:Data: 2023-11-02T18:44:06.752 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.753 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.753 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.754 DEBUG:boto:Params: {} 2023-11-02T18:44:06.754 DEBUG:boto:Token: None 2023-11-02T18:44:06.755 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.755 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3cDZJTL+4av7jlbs+EunIWmVatg= 2023-11-02T18:44:06.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3cDZJTL+4av7jlbs+EunIWmVatg=', 'Content-Length': '0'} 2023-11-02T18:44:06.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fc3ac32afadefea-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:44:06.760 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:44:06.761 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:44:06.761 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.762 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.762 DEBUG:boto:Method: GET 2023-11-02T18:44:06.763 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.763 DEBUG:boto:Data: 2023-11-02T18:44:06.764 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.764 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.765 DEBUG:boto:Params: {} 2023-11-02T18:44:06.765 DEBUG:boto:Token: None 2023-11-02T18:44:06.766 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gJMpnO+C539ol35yDrH8olTY89Q= 2023-11-02T18:44:06.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gJMpnO+C539ol35yDrH8olTY89Q=', 'Content-Length': '0'} 2023-11-02T18:44:06.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000054fff44f8c0600b-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.770 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:44:06.771 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:44:06.771 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.772 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.772 DEBUG:boto:Method: GET 2023-11-02T18:44:06.773 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.773 DEBUG:boto:Data: 2023-11-02T18:44:06.773 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.774 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.774 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.775 DEBUG:boto:Params: {} 2023-11-02T18:44:06.775 DEBUG:boto:Token: None 2023-11-02T18:44:06.776 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:44:06.776 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gJMpnO+C539ol35yDrH8olTY89Q= 2023-11-02T18:44:06.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gJMpnO+C539ol35yDrH8olTY89Q=', 'Content-Length': '0'} 2023-11-02T18:44:06.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000181d9f9021376ba8-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.822 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.823 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.823 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:44:06.824 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:44:06.824 DEBUG:boto:Method: GET 2023-11-02T18:44:06.825 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:44:06.825 DEBUG:boto:Data: 2023-11-02T18:44:06.826 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.826 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.827 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.827 DEBUG:boto:Params: {} 2023-11-02T18:44:06.828 DEBUG:boto:Token: None 2023-11-02T18:44:06.828 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.829 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000089373e0eeebec7c-006543edf6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.835 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.836 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.836 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.837 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:44:06.837 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:44:06.838 DEBUG:boto:Method: GET 2023-11-02T18:44:06.838 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:44:06.839 DEBUG:boto:Data: 2023-11-02T18:44:06.839 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.840 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.840 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.840 DEBUG:boto:Params: {} 2023-11-02T18:44:06.841 DEBUG:boto:Token: None 2023-11-02T18:44:06.841 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.842 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000106731918c352ed-006543edf6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.853 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.853 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:44:06.854 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:44:06.854 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.855 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:44:06.855 DEBUG:boto:Data: 2023-11-02T18:44:06.856 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.856 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.857 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.857 DEBUG:boto:Params: {} 2023-11-02T18:44:06.857 DEBUG:boto:Token: None 2023-11-02T18:44:06.858 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj17 2023-11-02T18:44:06.858 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZDHUrD9fbPZpD+8iBLc7mzS3Q+0= 2023-11-02T18:44:06.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZDHUrD9fbPZpD+8iBLc7mzS3Q+0=', 'Content-Length': '0'} 2023-11-02T18:44:06.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e71a843f5f5d088-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.862 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:44:06.863 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:44:06.863 DEBUG:boto:Method: HEAD 2023-11-02T18:44:06.863 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:44:06.864 DEBUG:boto:Data: 2023-11-02T18:44:06.864 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.865 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.865 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.865 DEBUG:boto:Params: {} 2023-11-02T18:44:06.866 DEBUG:boto:Token: None 2023-11-02T18:44:06.866 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj17 2023-11-02T18:44:06.867 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZDHUrD9fbPZpD+8iBLc7mzS3Q+0= 2023-11-02T18:44:06.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZDHUrD9fbPZpD+8iBLc7mzS3Q+0=', 'Content-Length': '0'} 2023-11-02T18:44:06.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087fa9c0e8c4edeb6-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:44:06.871 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:44:06.871 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:44:06.872 DEBUG:boto:Method: GET 2023-11-02T18:44:06.872 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:44:06.873 DEBUG:boto:Data: 2023-11-02T18:44:06.873 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.873 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.874 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.874 DEBUG:boto:Params: {} 2023-11-02T18:44:06.875 DEBUG:boto:Token: None 2023-11-02T18:44:06.875 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj17 2023-11-02T18:44:06.876 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tpY4tEngSQ/Inaux3wg/UiQq58A= 2023-11-02T18:44:06.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tpY4tEngSQ/Inaux3wg/UiQq58A=', 'Content-Length': '0'} 2023-11-02T18:44:06.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000299eeb3dda5167e9-006543edf6-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.922 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:44:06.923 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:44:06.924 DEBUG:boto:Method: GET 2023-11-02T18:44:06.924 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:44:06.925 DEBUG:boto:Data: 2023-11-02T18:44:06.925 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.926 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.926 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.927 DEBUG:boto:Params: {} 2023-11-02T18:44:06.927 DEBUG:boto:Token: None 2023-11-02T18:44:06.928 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/obj17 2023-11-02T18:44:06.928 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tpY4tEngSQ/Inaux3wg/UiQq58A= 2023-11-02T18:44:06.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tpY4tEngSQ/Inaux3wg/UiQq58A=', 'Content-Length': '0'} 2023-11-02T18:44:06.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cf8e5215d33119c-006543edf6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.933 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.933 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.934 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:44:06.934 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:44:06.935 DEBUG:boto:Method: GET 2023-11-02T18:44:06.935 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:44:06.936 DEBUG:boto:Data: 2023-11-02T18:44:06.936 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.937 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:06.937 DEBUG:boto:Port: 8001 2023-11-02T18:44:06.937 DEBUG:boto:Params: {} 2023-11-02T18:44:06.938 DEBUG:boto:Token: None 2023-11-02T18:44:06.938 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.939 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002190325afd73c507-006543edf6-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:06.945 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:06.946 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:06.947 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:06.947 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:44:06.948 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:44:06.948 DEBUG:boto:Method: GET 2023-11-02T18:44:06.949 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:44:06.949 DEBUG:boto:Data: 2023-11-02T18:44:06.950 DEBUG:boto:Headers: {} 2023-11-02T18:44:06.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:06.951 DEBUG:boto:Port: 8000 2023-11-02T18:44:06.951 DEBUG:boto:Params: {} 2023-11-02T18:44:06.952 DEBUG:boto:Token: None 2023-11-02T18:44:06.952 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:06 GMT /ibtljn-59/ 2023-11-02T18:44:06.953 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc= 2023-11-02T18:44:06.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rdYsQkKvHhfASl1suoHeNEvZLEc=', 'Content-Length': '0'} 2023-11-02T18:44:06.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097017febeae724a7-006543edf6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.006 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:44:07.008 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:44:07.008 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:44:07.009 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.009 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.010 DEBUG:boto:Method: GET 2023-11-02T18:44:07.010 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.011 DEBUG:boto:Data: 2023-11-02T18:44:07.011 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.012 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.012 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.013 DEBUG:boto:Params: {} 2023-11-02T18:44:07.013 DEBUG:boto:Token: None 2023-11-02T18:44:07.014 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.015 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JX3Hi6xWFjOrV7Zj6Scdn7uOBn0= 2023-11-02T18:44:07.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JX3Hi6xWFjOrV7Zj6Scdn7uOBn0=', 'Content-Length': '0'} 2023-11-02T18:44:07.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f013cd0987bef76-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.019 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:44:07.020 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:44:07.020 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.021 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.021 DEBUG:boto:Method: GET 2023-11-02T18:44:07.022 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.022 DEBUG:boto:Data: 2023-11-02T18:44:07.023 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.023 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.024 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.024 DEBUG:boto:Params: {} 2023-11-02T18:44:07.025 DEBUG:boto:Token: None 2023-11-02T18:44:07.025 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.026 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JX3Hi6xWFjOrV7Zj6Scdn7uOBn0= 2023-11-02T18:44:07.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JX3Hi6xWFjOrV7Zj6Scdn7uOBn0=', 'Content-Length': '0'} 2023-11-02T18:44:07.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000261d23c66dbb9545-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.030 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:44:07.031 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:44:07.031 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.032 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.032 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.033 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.033 DEBUG:boto:Data: 2023-11-02T18:44:07.033 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.034 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.034 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.035 DEBUG:boto:Params: {} 2023-11-02T18:44:07.035 DEBUG:boto:Token: None 2023-11-02T18:44:07.036 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.036 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O9HHGbU4+3CsDPQZeRmIOKddW5s= 2023-11-02T18:44:07.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O9HHGbU4+3CsDPQZeRmIOKddW5s=', 'Content-Length': '0'} 2023-11-02T18:44:07.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffea4eede022736f-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.041 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:44:07.041 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:44:07.041 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.042 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.042 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.043 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.043 DEBUG:boto:Data: 2023-11-02T18:44:07.044 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.044 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.045 DEBUG:boto:Params: {} 2023-11-02T18:44:07.045 DEBUG:boto:Token: None 2023-11-02T18:44:07.046 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.046 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O9HHGbU4+3CsDPQZeRmIOKddW5s= 2023-11-02T18:44:07.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O9HHGbU4+3CsDPQZeRmIOKddW5s=', 'Content-Length': '0'} 2023-11-02T18:44:07.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fffc94e9bf3aa25a-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:44:07.051 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:44:07.051 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:44:07.052 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.052 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.053 DEBUG:boto:Method: GET 2023-11-02T18:44:07.053 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.054 DEBUG:boto:Data: 2023-11-02T18:44:07.054 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.055 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.055 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.056 DEBUG:boto:Params: {} 2023-11-02T18:44:07.056 DEBUG:boto:Token: None 2023-11-02T18:44:07.057 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.057 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JX3Hi6xWFjOrV7Zj6Scdn7uOBn0= 2023-11-02T18:44:07.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JX3Hi6xWFjOrV7Zj6Scdn7uOBn0=', 'Content-Length': '0'} 2023-11-02T18:44:07.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047e066541b3be09a-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.102 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:44:07.103 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:44:07.103 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.104 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.104 DEBUG:boto:Method: GET 2023-11-02T18:44:07.105 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.106 DEBUG:boto:Data: 2023-11-02T18:44:07.106 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.107 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.107 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.107 DEBUG:boto:Params: {} 2023-11-02T18:44:07.108 DEBUG:boto:Token: None 2023-11-02T18:44:07.109 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:44:07.109 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JX3Hi6xWFjOrV7Zj6Scdn7uOBn0= 2023-11-02T18:44:07.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JX3Hi6xWFjOrV7Zj6Scdn7uOBn0=', 'Content-Length': '0'} 2023-11-02T18:44:07.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046589b23ad015183-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.113 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.114 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.115 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:44:07.115 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:44:07.116 DEBUG:boto:Method: GET 2023-11-02T18:44:07.116 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:44:07.117 DEBUG:boto:Data: 2023-11-02T18:44:07.117 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.118 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.118 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.119 DEBUG:boto:Params: {} 2023-11-02T18:44:07.119 DEBUG:boto:Token: None 2023-11-02T18:44:07.120 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096c1c63e3c77afbf-006543edf7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.126 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.127 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.128 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.128 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:44:07.129 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:44:07.129 DEBUG:boto:Method: GET 2023-11-02T18:44:07.130 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:44:07.130 DEBUG:boto:Data: 2023-11-02T18:44:07.130 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.131 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.131 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.132 DEBUG:boto:Params: {} 2023-11-02T18:44:07.132 DEBUG:boto:Token: None 2023-11-02T18:44:07.133 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.134 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013598389662fb7be-006543edf7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.143 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.144 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:44:07.144 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:44:07.145 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.145 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:44:07.146 DEBUG:boto:Data: 2023-11-02T18:44:07.146 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.147 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.147 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.148 DEBUG:boto:Params: {} 2023-11-02T18:44:07.148 DEBUG:boto:Token: None 2023-11-02T18:44:07.149 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj19 2023-11-02T18:44:07.150 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y0PK5NRQfQ61RF9vTQaHHkIj+PM= 2023-11-02T18:44:07.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y0PK5NRQfQ61RF9vTQaHHkIj+PM=', 'Content-Length': '0'} 2023-11-02T18:44:07.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4b5a1b84e454172-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.153 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:44:07.154 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:44:07.157 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.158 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:44:07.158 DEBUG:boto:Data: 2023-11-02T18:44:07.158 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.159 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.159 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.160 DEBUG:boto:Params: {} 2023-11-02T18:44:07.160 DEBUG:boto:Token: None 2023-11-02T18:44:07.161 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj19 2023-11-02T18:44:07.161 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y0PK5NRQfQ61RF9vTQaHHkIj+PM= 2023-11-02T18:44:07.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y0PK5NRQfQ61RF9vTQaHHkIj+PM=', 'Content-Length': '0'} 2023-11-02T18:44:07.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432b55af17609d14-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:44:07.165 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:44:07.166 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:44:07.166 DEBUG:boto:Method: GET 2023-11-02T18:44:07.167 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:44:07.167 DEBUG:boto:Data: 2023-11-02T18:44:07.168 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.168 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.169 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.169 DEBUG:boto:Params: {} 2023-11-02T18:44:07.170 DEBUG:boto:Token: None 2023-11-02T18:44:07.170 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj19 2023-11-02T18:44:07.171 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P2rSYTVD71ZnFWVtCZA/o/v7ijg= 2023-11-02T18:44:07.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P2rSYTVD71ZnFWVtCZA/o/v7ijg=', 'Content-Length': '0'} 2023-11-02T18:44:07.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e34d80045d11f77-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.175 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:44:07.175 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:44:07.176 DEBUG:boto:Method: GET 2023-11-02T18:44:07.176 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:44:07.177 DEBUG:boto:Data: 2023-11-02T18:44:07.177 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.178 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.178 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.179 DEBUG:boto:Params: {} 2023-11-02T18:44:07.179 DEBUG:boto:Token: None 2023-11-02T18:44:07.180 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj19 2023-11-02T18:44:07.180 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P2rSYTVD71ZnFWVtCZA/o/v7ijg= 2023-11-02T18:44:07.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P2rSYTVD71ZnFWVtCZA/o/v7ijg=', 'Content-Length': '0'} 2023-11-02T18:44:07.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008af350eb2cbffeff-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.185 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.185 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.186 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:44:07.186 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:44:07.187 DEBUG:boto:Method: GET 2023-11-02T18:44:07.187 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:44:07.188 DEBUG:boto:Data: 2023-11-02T18:44:07.188 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.189 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.189 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.190 DEBUG:boto:Params: {} 2023-11-02T18:44:07.190 DEBUG:boto:Token: None 2023-11-02T18:44:07.191 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.191 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011f010f00810096c-006543edf7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.242 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.243 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.244 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.244 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:44:07.245 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:44:07.245 DEBUG:boto:Method: GET 2023-11-02T18:44:07.246 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:44:07.246 DEBUG:boto:Data: 2023-11-02T18:44:07.247 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.247 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.248 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.248 DEBUG:boto:Params: {} 2023-11-02T18:44:07.249 DEBUG:boto:Token: None 2023-11-02T18:44:07.250 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.250 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005931e4e1508cefc5-006543edf7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.261 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:44:07.262 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:44:07.263 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:44:07.263 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.264 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.264 DEBUG:boto:Method: GET 2023-11-02T18:44:07.265 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.265 DEBUG:boto:Data: 2023-11-02T18:44:07.266 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.266 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.267 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.267 DEBUG:boto:Params: {} 2023-11-02T18:44:07.268 DEBUG:boto:Token: None 2023-11-02T18:44:07.269 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.269 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:StXGvuYnrWvBp6h1FaavAmLj6cE= 2023-11-02T18:44:07.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:StXGvuYnrWvBp6h1FaavAmLj6cE=', 'Content-Length': '0'} 2023-11-02T18:44:07.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f409993175bdf1-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.273 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:44:07.274 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:44:07.274 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.275 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.275 DEBUG:boto:Method: GET 2023-11-02T18:44:07.276 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.276 DEBUG:boto:Data: 2023-11-02T18:44:07.277 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.277 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.278 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.278 DEBUG:boto:Params: {} 2023-11-02T18:44:07.279 DEBUG:boto:Token: None 2023-11-02T18:44:07.279 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.280 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:StXGvuYnrWvBp6h1FaavAmLj6cE= 2023-11-02T18:44:07.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:StXGvuYnrWvBp6h1FaavAmLj6cE=', 'Content-Length': '0'} 2023-11-02T18:44:07.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab311a999cdd8a6-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.330 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:44:07.331 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:44:07.331 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.332 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.332 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.333 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.333 DEBUG:boto:Data: 2023-11-02T18:44:07.333 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.334 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.334 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.335 DEBUG:boto:Params: {} 2023-11-02T18:44:07.335 DEBUG:boto:Token: None 2023-11-02T18:44:07.336 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.336 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EVW6xAYTDuhDA24/W56OYbCLLEU= 2023-11-02T18:44:07.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EVW6xAYTDuhDA24/W56OYbCLLEU=', 'Content-Length': '0'} 2023-11-02T18:44:07.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048da6975d31aedc9-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.340 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:44:07.341 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:44:07.341 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.342 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.342 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.342 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.343 DEBUG:boto:Data: 2023-11-02T18:44:07.343 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.344 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.344 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.345 DEBUG:boto:Params: {} 2023-11-02T18:44:07.345 DEBUG:boto:Token: None 2023-11-02T18:44:07.346 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.346 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EVW6xAYTDuhDA24/W56OYbCLLEU= 2023-11-02T18:44:07.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EVW6xAYTDuhDA24/W56OYbCLLEU=', 'Content-Length': '0'} 2023-11-02T18:44:07.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7274200282270c4-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:44:07.351 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:44:07.351 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:44:07.352 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.352 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.353 DEBUG:boto:Method: GET 2023-11-02T18:44:07.353 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.353 DEBUG:boto:Data: 2023-11-02T18:44:07.354 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.354 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.355 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.355 DEBUG:boto:Params: {} 2023-11-02T18:44:07.356 DEBUG:boto:Token: None 2023-11-02T18:44:07.357 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.357 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:StXGvuYnrWvBp6h1FaavAmLj6cE= 2023-11-02T18:44:07.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:StXGvuYnrWvBp6h1FaavAmLj6cE=', 'Content-Length': '0'} 2023-11-02T18:44:07.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000206c8be76af84a53-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.362 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:44:07.362 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:44:07.363 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.363 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.364 DEBUG:boto:Method: GET 2023-11-02T18:44:07.364 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.365 DEBUG:boto:Data: 2023-11-02T18:44:07.365 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.366 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.367 DEBUG:boto:Params: {} 2023-11-02T18:44:07.367 DEBUG:boto:Token: None 2023-11-02T18:44:07.368 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:44:07.368 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:StXGvuYnrWvBp6h1FaavAmLj6cE= 2023-11-02T18:44:07.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:StXGvuYnrWvBp6h1FaavAmLj6cE=', 'Content-Length': '0'} 2023-11-02T18:44:07.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af1ceea06c9f81d-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.372 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.373 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.373 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:44:07.374 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:44:07.374 DEBUG:boto:Method: GET 2023-11-02T18:44:07.375 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:44:07.375 DEBUG:boto:Data: 2023-11-02T18:44:07.376 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.376 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.376 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.377 DEBUG:boto:Params: {} 2023-11-02T18:44:07.377 DEBUG:boto:Token: None 2023-11-02T18:44:07.378 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.378 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e94c4d1a7b869a99-006543edf7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.384 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.385 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.386 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.386 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:44:07.387 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:44:07.387 DEBUG:boto:Method: GET 2023-11-02T18:44:07.387 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:44:07.388 DEBUG:boto:Data: 2023-11-02T18:44:07.388 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.389 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.389 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.390 DEBUG:boto:Params: {} 2023-11-02T18:44:07.390 DEBUG:boto:Token: None 2023-11-02T18:44:07.391 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.391 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a288aabd6ae2c589-006543edf7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.401 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.402 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:44:07.402 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:44:07.403 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.403 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:44:07.403 DEBUG:boto:Data: 2023-11-02T18:44:07.404 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.404 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.405 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.405 DEBUG:boto:Params: {} 2023-11-02T18:44:07.406 DEBUG:boto:Token: None 2023-11-02T18:44:07.406 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj22 2023-11-02T18:44:07.406 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AdF3SNmuK1SUgiSUehaxdoTdLQo= 2023-11-02T18:44:07.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AdF3SNmuK1SUgiSUehaxdoTdLQo=', 'Content-Length': '0'} 2023-11-02T18:44:07.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f30cafc4bfa06963-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.410 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:44:07.411 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:44:07.411 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.412 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:44:07.412 DEBUG:boto:Data: 2023-11-02T18:44:07.413 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.413 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.414 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.414 DEBUG:boto:Params: {} 2023-11-02T18:44:07.415 DEBUG:boto:Token: None 2023-11-02T18:44:07.415 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj22 2023-11-02T18:44:07.416 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AdF3SNmuK1SUgiSUehaxdoTdLQo= 2023-11-02T18:44:07.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AdF3SNmuK1SUgiSUehaxdoTdLQo=', 'Content-Length': '0'} 2023-11-02T18:44:07.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cb3408f85db7da9-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:44:07.419 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:44:07.420 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:44:07.420 DEBUG:boto:Method: GET 2023-11-02T18:44:07.421 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:44:07.421 DEBUG:boto:Data: 2023-11-02T18:44:07.422 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.423 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.423 DEBUG:boto:Params: {} 2023-11-02T18:44:07.424 DEBUG:boto:Token: None 2023-11-02T18:44:07.424 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj22 2023-11-02T18:44:07.425 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TzwEvfxzg4sxkCgGHhSCQLZRpmw= 2023-11-02T18:44:07.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TzwEvfxzg4sxkCgGHhSCQLZRpmw=', 'Content-Length': '0'} 2023-11-02T18:44:07.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db0cbeff576f6588-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.474 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:44:07.475 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:44:07.475 DEBUG:boto:Method: GET 2023-11-02T18:44:07.476 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:44:07.476 DEBUG:boto:Data: 2023-11-02T18:44:07.477 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.477 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.478 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.478 DEBUG:boto:Params: {} 2023-11-02T18:44:07.479 DEBUG:boto:Token: None 2023-11-02T18:44:07.479 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj22 2023-11-02T18:44:07.480 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TzwEvfxzg4sxkCgGHhSCQLZRpmw= 2023-11-02T18:44:07.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TzwEvfxzg4sxkCgGHhSCQLZRpmw=', 'Content-Length': '0'} 2023-11-02T18:44:07.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e01f2260b7bce30-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.486 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.486 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.487 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:44:07.487 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:44:07.487 DEBUG:boto:Method: GET 2023-11-02T18:44:07.488 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:44:07.488 DEBUG:boto:Data: 2023-11-02T18:44:07.489 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.489 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.490 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.490 DEBUG:boto:Params: {} 2023-11-02T18:44:07.491 DEBUG:boto:Token: None 2023-11-02T18:44:07.491 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.492 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2b0b0ef11a03690-006543edf7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.498 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.499 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.499 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.499 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:44:07.500 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:44:07.500 DEBUG:boto:Method: GET 2023-11-02T18:44:07.501 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:44:07.501 DEBUG:boto:Data: 2023-11-02T18:44:07.501 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.502 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.502 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.503 DEBUG:boto:Params: {} 2023-11-02T18:44:07.503 DEBUG:boto:Token: None 2023-11-02T18:44:07.504 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.504 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cf79b8374299aa6-006543edf7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.515 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:44:07.516 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:44:07.516 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:44:07.517 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.517 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.517 DEBUG:boto:Method: GET 2023-11-02T18:44:07.518 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.518 DEBUG:boto:Data: 2023-11-02T18:44:07.519 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.519 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.520 DEBUG:boto:Params: {} 2023-11-02T18:44:07.520 DEBUG:boto:Token: None 2023-11-02T18:44:07.521 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZWdt2LDDsDsiKKXp/maJ9J0jP8c= 2023-11-02T18:44:07.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZWdt2LDDsDsiKKXp/maJ9J0jP8c=', 'Content-Length': '0'} 2023-11-02T18:44:07.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007664635517daf740-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.525 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:44:07.526 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:44:07.526 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.527 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.527 DEBUG:boto:Method: GET 2023-11-02T18:44:07.528 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.528 DEBUG:boto:Data: 2023-11-02T18:44:07.528 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.529 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.529 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.530 DEBUG:boto:Params: {} 2023-11-02T18:44:07.530 DEBUG:boto:Token: None 2023-11-02T18:44:07.531 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.531 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZWdt2LDDsDsiKKXp/maJ9J0jP8c= 2023-11-02T18:44:07.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZWdt2LDDsDsiKKXp/maJ9J0jP8c=', 'Content-Length': '0'} 2023-11-02T18:44:07.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d37be9f18a928a3-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.535 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:44:07.535 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:44:07.536 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.536 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.538 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.538 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.538 DEBUG:boto:Data: 2023-11-02T18:44:07.539 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.539 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.540 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.540 DEBUG:boto:Params: {} 2023-11-02T18:44:07.541 DEBUG:boto:Token: None 2023-11-02T18:44:07.541 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.542 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MhSPmteZcMXr4EybKBGpb1VqI14= 2023-11-02T18:44:07.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MhSPmteZcMXr4EybKBGpb1VqI14=', 'Content-Length': '0'} 2023-11-02T18:44:07.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015493ff622e7b697-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.546 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:44:07.546 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:44:07.547 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.547 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.547 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.548 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.548 DEBUG:boto:Data: 2023-11-02T18:44:07.549 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.549 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.550 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.550 DEBUG:boto:Params: {} 2023-11-02T18:44:07.550 DEBUG:boto:Token: None 2023-11-02T18:44:07.551 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.551 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MhSPmteZcMXr4EybKBGpb1VqI14= 2023-11-02T18:44:07.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MhSPmteZcMXr4EybKBGpb1VqI14=', 'Content-Length': '0'} 2023-11-02T18:44:07.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001487257017397914-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:44:07.556 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:44:07.556 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:44:07.556 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.557 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.557 DEBUG:boto:Method: GET 2023-11-02T18:44:07.558 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.558 DEBUG:boto:Data: 2023-11-02T18:44:07.559 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.559 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.560 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.560 DEBUG:boto:Params: {} 2023-11-02T18:44:07.560 DEBUG:boto:Token: None 2023-11-02T18:44:07.561 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.561 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZWdt2LDDsDsiKKXp/maJ9J0jP8c= 2023-11-02T18:44:07.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZWdt2LDDsDsiKKXp/maJ9J0jP8c=', 'Content-Length': '0'} 2023-11-02T18:44:07.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e896069600c06a11-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.566 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:44:07.567 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:44:07.567 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.567 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.568 DEBUG:boto:Method: GET 2023-11-02T18:44:07.568 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.569 DEBUG:boto:Data: 2023-11-02T18:44:07.569 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.570 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.570 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.571 DEBUG:boto:Params: {} 2023-11-02T18:44:07.571 DEBUG:boto:Token: None 2023-11-02T18:44:07.572 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:44:07.572 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZWdt2LDDsDsiKKXp/maJ9J0jP8c= 2023-11-02T18:44:07.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZWdt2LDDsDsiKKXp/maJ9J0jP8c=', 'Content-Length': '0'} 2023-11-02T18:44:07.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b7b33115cc1122c-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.622 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.623 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.623 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:44:07.624 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:44:07.624 DEBUG:boto:Method: GET 2023-11-02T18:44:07.625 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:44:07.625 DEBUG:boto:Data: 2023-11-02T18:44:07.626 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.626 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.626 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.627 DEBUG:boto:Params: {} 2023-11-02T18:44:07.627 DEBUG:boto:Token: None 2023-11-02T18:44:07.628 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.628 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053193cbe6f626666-006543edf7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.634 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.635 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.636 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.636 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:44:07.637 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:44:07.637 DEBUG:boto:Method: GET 2023-11-02T18:44:07.637 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:44:07.638 DEBUG:boto:Data: 2023-11-02T18:44:07.638 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.639 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.640 DEBUG:boto:Params: {} 2023-11-02T18:44:07.640 DEBUG:boto:Token: None 2023-11-02T18:44:07.641 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0bca9e6a17dd932-006543edf7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.648 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.649 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:44:07.649 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:44:07.649 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.650 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:44:07.650 DEBUG:boto:Data: 2023-11-02T18:44:07.651 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.652 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.652 DEBUG:boto:Params: {} 2023-11-02T18:44:07.652 DEBUG:boto:Token: None 2023-11-02T18:44:07.653 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj23 2023-11-02T18:44:07.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TVyF9glnJHdQKEWmYXhaZZhOi+o= 2023-11-02T18:44:07.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TVyF9glnJHdQKEWmYXhaZZhOi+o=', 'Content-Length': '0'} 2023-11-02T18:44:07.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f57fe560695c7d59-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.656 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:44:07.657 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:44:07.657 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.658 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:44:07.658 DEBUG:boto:Data: 2023-11-02T18:44:07.659 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.659 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.659 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.660 DEBUG:boto:Params: {} 2023-11-02T18:44:07.660 DEBUG:boto:Token: None 2023-11-02T18:44:07.661 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj23 2023-11-02T18:44:07.661 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TVyF9glnJHdQKEWmYXhaZZhOi+o= 2023-11-02T18:44:07.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TVyF9glnJHdQKEWmYXhaZZhOi+o=', 'Content-Length': '0'} 2023-11-02T18:44:07.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b15b4554f8dec96-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:44:07.665 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:44:07.666 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:44:07.666 DEBUG:boto:Method: GET 2023-11-02T18:44:07.667 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:44:07.667 DEBUG:boto:Data: 2023-11-02T18:44:07.668 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.668 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.668 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.669 DEBUG:boto:Params: {} 2023-11-02T18:44:07.669 DEBUG:boto:Token: None 2023-11-02T18:44:07.670 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj23 2023-11-02T18:44:07.670 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z5a9E8UnUScNxk+tUVDKMO1A+t8= 2023-11-02T18:44:07.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z5a9E8UnUScNxk+tUVDKMO1A+t8=', 'Content-Length': '0'} 2023-11-02T18:44:07.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdb98f7cec13d4a4-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.714 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:44:07.715 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:44:07.716 DEBUG:boto:Method: GET 2023-11-02T18:44:07.716 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:44:07.716 DEBUG:boto:Data: 2023-11-02T18:44:07.717 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.717 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.718 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.718 DEBUG:boto:Params: {} 2023-11-02T18:44:07.719 DEBUG:boto:Token: None 2023-11-02T18:44:07.719 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj23 2023-11-02T18:44:07.720 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z5a9E8UnUScNxk+tUVDKMO1A+t8= 2023-11-02T18:44:07.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z5a9E8UnUScNxk+tUVDKMO1A+t8=', 'Content-Length': '0'} 2023-11-02T18:44:07.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041eb998d1daed3f8-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.724 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.725 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.725 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:44:07.726 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:44:07.726 DEBUG:boto:Method: GET 2023-11-02T18:44:07.726 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:44:07.727 DEBUG:boto:Data: 2023-11-02T18:44:07.727 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.728 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.728 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.729 DEBUG:boto:Params: {} 2023-11-02T18:44:07.729 DEBUG:boto:Token: None 2023-11-02T18:44:07.730 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.730 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d724f94c1c50265-006543edf7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.737 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.738 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.738 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.739 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:44:07.739 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:44:07.739 DEBUG:boto:Method: GET 2023-11-02T18:44:07.740 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:44:07.740 DEBUG:boto:Data: 2023-11-02T18:44:07.741 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.741 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.742 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.742 DEBUG:boto:Params: {} 2023-11-02T18:44:07.743 DEBUG:boto:Token: None 2023-11-02T18:44:07.743 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.744 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d276ec06b12af5bf-006543edf7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.794 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:44:07.796 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:44:07.796 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:44:07.797 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.797 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.798 DEBUG:boto:Method: GET 2023-11-02T18:44:07.798 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.799 DEBUG:boto:Data: 2023-11-02T18:44:07.799 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.800 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.800 DEBUG:boto:Params: {} 2023-11-02T18:44:07.801 DEBUG:boto:Token: None 2023-11-02T18:44:07.801 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.802 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AI+ae1UvuDlB96zyfPLGXKp1jJE= 2023-11-02T18:44:07.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AI+ae1UvuDlB96zyfPLGXKp1jJE=', 'Content-Length': '0'} 2023-11-02T18:44:07.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000436733074af2a0c1-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.807 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:44:07.807 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:44:07.808 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.808 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.808 DEBUG:boto:Method: GET 2023-11-02T18:44:07.809 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.809 DEBUG:boto:Data: 2023-11-02T18:44:07.810 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.810 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.811 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.811 DEBUG:boto:Params: {} 2023-11-02T18:44:07.812 DEBUG:boto:Token: None 2023-11-02T18:44:07.812 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.812 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AI+ae1UvuDlB96zyfPLGXKp1jJE= 2023-11-02T18:44:07.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AI+ae1UvuDlB96zyfPLGXKp1jJE=', 'Content-Length': '0'} 2023-11-02T18:44:07.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c8c73a51fc98a76-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.817 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:44:07.817 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:44:07.818 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.818 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.819 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.819 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.820 DEBUG:boto:Data: 2023-11-02T18:44:07.820 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.820 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.821 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.821 DEBUG:boto:Params: {} 2023-11-02T18:44:07.822 DEBUG:boto:Token: None 2023-11-02T18:44:07.822 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.823 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d6D9R2Er6JQB8UI8TKH0WhWXZ8Y= 2023-11-02T18:44:07.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d6D9R2Er6JQB8UI8TKH0WhWXZ8Y=', 'Content-Length': '0'} 2023-11-02T18:44:07.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaf03fc05a15b244-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.827 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:44:07.827 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:44:07.828 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.828 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.829 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.829 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.830 DEBUG:boto:Data: 2023-11-02T18:44:07.830 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.830 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.831 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.831 DEBUG:boto:Params: {} 2023-11-02T18:44:07.832 DEBUG:boto:Token: None 2023-11-02T18:44:07.832 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.833 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d6D9R2Er6JQB8UI8TKH0WhWXZ8Y= 2023-11-02T18:44:07.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d6D9R2Er6JQB8UI8TKH0WhWXZ8Y=', 'Content-Length': '0'} 2023-11-02T18:44:07.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022b609b32f960da4-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:44:07.837 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:44:07.838 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:44:07.838 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.839 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.839 DEBUG:boto:Method: GET 2023-11-02T18:44:07.839 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.840 DEBUG:boto:Data: 2023-11-02T18:44:07.840 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.841 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.842 DEBUG:boto:Params: {} 2023-11-02T18:44:07.842 DEBUG:boto:Token: None 2023-11-02T18:44:07.843 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.843 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AI+ae1UvuDlB96zyfPLGXKp1jJE= 2023-11-02T18:44:07.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AI+ae1UvuDlB96zyfPLGXKp1jJE=', 'Content-Length': '0'} 2023-11-02T18:44:07.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007631155d4820c701-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.848 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:44:07.848 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:44:07.849 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.849 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.849 DEBUG:boto:Method: GET 2023-11-02T18:44:07.850 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.850 DEBUG:boto:Data: 2023-11-02T18:44:07.851 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.851 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.852 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.852 DEBUG:boto:Params: {} 2023-11-02T18:44:07.853 DEBUG:boto:Token: None 2023-11-02T18:44:07.853 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:44:07.854 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AI+ae1UvuDlB96zyfPLGXKp1jJE= 2023-11-02T18:44:07.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AI+ae1UvuDlB96zyfPLGXKp1jJE=', 'Content-Length': '0'} 2023-11-02T18:44:07.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a764c41df37ec390-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.898 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.899 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.899 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:44:07.900 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:44:07.900 DEBUG:boto:Method: GET 2023-11-02T18:44:07.901 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:44:07.901 DEBUG:boto:Data: 2023-11-02T18:44:07.902 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.902 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.903 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.903 DEBUG:boto:Params: {} 2023-11-02T18:44:07.904 DEBUG:boto:Token: None 2023-11-02T18:44:07.904 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.905 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036799cf4fa7978fe-006543edf7-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.911 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.911 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:07.912 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:07.912 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:44:07.913 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:44:07.913 DEBUG:boto:Method: GET 2023-11-02T18:44:07.914 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:44:07.914 DEBUG:boto:Data: 2023-11-02T18:44:07.915 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.915 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.915 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.916 DEBUG:boto:Params: {} 2023-11-02T18:44:07.916 DEBUG:boto:Token: None 2023-11-02T18:44:07.917 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/ 2023-11-02T18:44:07.917 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA= 2023-11-02T18:44:07.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NVLzHf2aq89HeKAxd2Cu4UobFqA=', 'Content-Length': '0'} 2023-11-02T18:44:07.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066a4cd2a00a17dab-006543edf7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.928 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:07.928 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:44:07.929 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:44:07.929 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.930 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:44:07.930 DEBUG:boto:Data: 2023-11-02T18:44:07.931 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.931 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.932 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.932 DEBUG:boto:Params: {} 2023-11-02T18:44:07.932 DEBUG:boto:Token: None 2023-11-02T18:44:07.933 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj29 2023-11-02T18:44:07.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p0LUJ8FhKt6gmJ2TpmypL/iLrzs= 2023-11-02T18:44:07.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p0LUJ8FhKt6gmJ2TpmypL/iLrzs=', 'Content-Length': '0'} 2023-11-02T18:44:07.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea0bb26c23b2996c-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.937 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:44:07.937 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:44:07.938 DEBUG:boto:Method: HEAD 2023-11-02T18:44:07.938 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:44:07.939 DEBUG:boto:Data: 2023-11-02T18:44:07.939 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.939 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:07.940 DEBUG:boto:Port: 8000 2023-11-02T18:44:07.940 DEBUG:boto:Params: {} 2023-11-02T18:44:07.941 DEBUG:boto:Token: None 2023-11-02T18:44:07.941 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj29 2023-11-02T18:44:07.942 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p0LUJ8FhKt6gmJ2TpmypL/iLrzs= 2023-11-02T18:44:07.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p0LUJ8FhKt6gmJ2TpmypL/iLrzs=', 'Content-Length': '0'} 2023-11-02T18:44:07.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000737d3c327802fccd-006543edf7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:07.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:44:07.946 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:44:07.946 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:44:07.947 DEBUG:boto:Method: GET 2023-11-02T18:44:07.947 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:44:07.948 DEBUG:boto:Data: 2023-11-02T18:44:07.948 DEBUG:boto:Headers: {} 2023-11-02T18:44:07.949 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:07.949 DEBUG:boto:Port: 8001 2023-11-02T18:44:07.949 DEBUG:boto:Params: {} 2023-11-02T18:44:07.950 DEBUG:boto:Token: None 2023-11-02T18:44:07.950 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:07 GMT /ibtljn-59/obj29 2023-11-02T18:44:07.951 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:txNRywsB0rArMM3PgBm755gZXf4= 2023-11-02T18:44:07.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:txNRywsB0rArMM3PgBm755gZXf4=', 'Content-Length': '0'} 2023-11-02T18:44:07.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d849afcfba47b078-006543edf7-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.002 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:44:08.003 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:44:08.004 DEBUG:boto:Method: GET 2023-11-02T18:44:08.004 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:44:08.005 DEBUG:boto:Data: 2023-11-02T18:44:08.005 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.005 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.006 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.006 DEBUG:boto:Params: {} 2023-11-02T18:44:08.007 DEBUG:boto:Token: None 2023-11-02T18:44:08.008 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj29 2023-11-02T18:44:08.008 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GGajN1bC0RfU7VV+8+kHJ/+nNGQ= 2023-11-02T18:44:08.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GGajN1bC0RfU7VV+8+kHJ/+nNGQ=', 'Content-Length': '0'} 2023-11-02T18:44:08.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9bd54bd531ae71c-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.013 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.013 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.014 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:44:08.014 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:44:08.015 DEBUG:boto:Method: GET 2023-11-02T18:44:08.015 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:44:08.016 DEBUG:boto:Data: 2023-11-02T18:44:08.016 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.017 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.017 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.017 DEBUG:boto:Params: {} 2023-11-02T18:44:08.018 DEBUG:boto:Token: None 2023-11-02T18:44:08.018 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.019 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077a788b6d164ece0-006543edf8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.025 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.026 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.027 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.027 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:44:08.028 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:44:08.028 DEBUG:boto:Method: GET 2023-11-02T18:44:08.029 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:44:08.029 DEBUG:boto:Data: 2023-11-02T18:44:08.030 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.031 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.031 DEBUG:boto:Params: {} 2023-11-02T18:44:08.032 DEBUG:boto:Token: None 2023-11-02T18:44:08.032 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.033 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d825d255d59cf51-006543edf8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.086 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T18:44:08.088 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:44:08.089 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:44:08.089 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.090 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.090 DEBUG:boto:Method: GET 2023-11-02T18:44:08.091 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.091 DEBUG:boto:Data: 2023-11-02T18:44:08.091 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.092 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.092 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.093 DEBUG:boto:Params: {} 2023-11-02T18:44:08.093 DEBUG:boto:Token: None 2023-11-02T18:44:08.094 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.095 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kOsEp1A7OwQt1/iva9P3DH9rmcQ= 2023-11-02T18:44:08.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kOsEp1A7OwQt1/iva9P3DH9rmcQ=', 'Content-Length': '0'} 2023-11-02T18:44:08.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecbfd9b31bdf4bb8-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.099 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:44:08.100 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:44:08.100 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.100 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.101 DEBUG:boto:Method: GET 2023-11-02T18:44:08.101 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.102 DEBUG:boto:Data: 2023-11-02T18:44:08.102 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.103 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.103 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.103 DEBUG:boto:Params: {} 2023-11-02T18:44:08.104 DEBUG:boto:Token: None 2023-11-02T18:44:08.104 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.105 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kOsEp1A7OwQt1/iva9P3DH9rmcQ= 2023-11-02T18:44:08.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kOsEp1A7OwQt1/iva9P3DH9rmcQ=', 'Content-Length': '0'} 2023-11-02T18:44:08.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a3c67935e76b749-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.109 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:44:08.110 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:44:08.110 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.110 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.111 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.111 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.112 DEBUG:boto:Data: 2023-11-02T18:44:08.112 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.113 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.113 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.113 DEBUG:boto:Params: {} 2023-11-02T18:44:08.114 DEBUG:boto:Token: None 2023-11-02T18:44:08.114 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.115 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tlDQa719VN98J9j1FGY7rkX55KI= 2023-11-02T18:44:08.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tlDQa719VN98J9j1FGY7rkX55KI=', 'Content-Length': '0'} 2023-11-02T18:44:08.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001147256a66b6a3a8-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.119 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:44:08.119 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:44:08.120 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.120 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.120 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.121 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.121 DEBUG:boto:Data: 2023-11-02T18:44:08.122 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.122 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.123 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.123 DEBUG:boto:Params: {} 2023-11-02T18:44:08.124 DEBUG:boto:Token: None 2023-11-02T18:44:08.124 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.124 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tlDQa719VN98J9j1FGY7rkX55KI= 2023-11-02T18:44:08.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tlDQa719VN98J9j1FGY7rkX55KI=', 'Content-Length': '0'} 2023-11-02T18:44:08.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e63b1ac0cbe6b40-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T18:44:08.129 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:44:08.129 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:44:08.130 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.130 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.131 DEBUG:boto:Method: GET 2023-11-02T18:44:08.131 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.132 DEBUG:boto:Data: 2023-11-02T18:44:08.132 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.133 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.133 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.133 DEBUG:boto:Params: {} 2023-11-02T18:44:08.134 DEBUG:boto:Token: None 2023-11-02T18:44:08.134 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.135 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kOsEp1A7OwQt1/iva9P3DH9rmcQ= 2023-11-02T18:44:08.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kOsEp1A7OwQt1/iva9P3DH9rmcQ=', 'Content-Length': '0'} 2023-11-02T18:44:08.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039506498b4b81d50-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.139 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:44:08.140 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:44:08.140 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.140 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.141 DEBUG:boto:Method: GET 2023-11-02T18:44:08.141 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.142 DEBUG:boto:Data: 2023-11-02T18:44:08.142 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.143 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.143 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.144 DEBUG:boto:Params: {} 2023-11-02T18:44:08.144 DEBUG:boto:Token: None 2023-11-02T18:44:08.145 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:44:08.145 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kOsEp1A7OwQt1/iva9P3DH9rmcQ= 2023-11-02T18:44:08.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kOsEp1A7OwQt1/iva9P3DH9rmcQ=', 'Content-Length': '0'} 2023-11-02T18:44:08.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043f958a1be6f8f80-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.149 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.150 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.150 DEBUG:boto:path=/ibtljn-59/?prefix=obj31 2023-11-02T18:44:08.151 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj31 2023-11-02T18:44:08.151 DEBUG:boto:Method: GET 2023-11-02T18:44:08.152 DEBUG:boto:Path: /ibtljn-59/?prefix=obj31 2023-11-02T18:44:08.152 DEBUG:boto:Data: 2023-11-02T18:44:08.152 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.153 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.153 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.154 DEBUG:boto:Params: {} 2023-11-02T18:44:08.154 DEBUG:boto:Token: None 2023-11-02T18:44:08.155 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.155 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3cd67c1757ab7a4-006543edf8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.202 DEBUG:boto:b'ibtljn-59obj311000falseobj312023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.203 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.204 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.204 DEBUG:boto:path=/ibtljn-59/?prefix=obj31 2023-11-02T18:44:08.205 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj31 2023-11-02T18:44:08.205 DEBUG:boto:Method: GET 2023-11-02T18:44:08.206 DEBUG:boto:Path: /ibtljn-59/?prefix=obj31 2023-11-02T18:44:08.206 DEBUG:boto:Data: 2023-11-02T18:44:08.207 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.207 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.207 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.208 DEBUG:boto:Params: {} 2023-11-02T18:44:08.208 DEBUG:boto:Token: None 2023-11-02T18:44:08.209 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.209 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ace6558b585ddb5d-006543edf8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.219 DEBUG:boto:b'ibtljn-59obj311000falseobj312023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.219 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:44:08.220 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:44:08.220 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.221 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:44:08.221 DEBUG:boto:Data: 2023-11-02T18:44:08.222 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.222 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.223 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.223 DEBUG:boto:Params: {} 2023-11-02T18:44:08.224 DEBUG:boto:Token: None 2023-11-02T18:44:08.224 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj31 2023-11-02T18:44:08.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gk9268jL7lFOW0U4iOnqVZQ3H7g= 2023-11-02T18:44:08.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gk9268jL7lFOW0U4iOnqVZQ3H7g=', 'Content-Length': '0'} 2023-11-02T18:44:08.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c9d7d73fccb85a-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.228 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:44:08.229 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:44:08.229 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.230 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:44:08.230 DEBUG:boto:Data: 2023-11-02T18:44:08.231 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.232 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.232 DEBUG:boto:Params: {} 2023-11-02T18:44:08.232 DEBUG:boto:Token: None 2023-11-02T18:44:08.233 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj31 2023-11-02T18:44:08.233 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gk9268jL7lFOW0U4iOnqVZQ3H7g= 2023-11-02T18:44:08.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gk9268jL7lFOW0U4iOnqVZQ3H7g=', 'Content-Length': '0'} 2023-11-02T18:44:08.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055468dfd410f21d8-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T18:44:08.238 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:44:08.238 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:44:08.239 DEBUG:boto:Method: GET 2023-11-02T18:44:08.239 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:44:08.240 DEBUG:boto:Data: 2023-11-02T18:44:08.240 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.241 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.241 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.242 DEBUG:boto:Params: {} 2023-11-02T18:44:08.242 DEBUG:boto:Token: None 2023-11-02T18:44:08.243 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj31 2023-11-02T18:44:08.243 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yc1963kuO69dI5D1cMf79it7lbs= 2023-11-02T18:44:08.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yc1963kuO69dI5D1cMf79it7lbs=', 'Content-Length': '0'} 2023-11-02T18:44:08.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8bf04d6a6b435d7-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.247 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:44:08.248 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:44:08.248 DEBUG:boto:Method: GET 2023-11-02T18:44:08.249 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:44:08.249 DEBUG:boto:Data: 2023-11-02T18:44:08.250 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.250 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.250 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.251 DEBUG:boto:Params: {} 2023-11-02T18:44:08.251 DEBUG:boto:Token: None 2023-11-02T18:44:08.252 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj31 2023-11-02T18:44:08.252 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yc1963kuO69dI5D1cMf79it7lbs= 2023-11-02T18:44:08.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yc1963kuO69dI5D1cMf79it7lbs=', 'Content-Length': '0'} 2023-11-02T18:44:08.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000711b8e884be8c938-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.302 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.303 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.303 DEBUG:boto:path=/ibtljn-59/?prefix=obj31 2023-11-02T18:44:08.304 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj31 2023-11-02T18:44:08.304 DEBUG:boto:Method: GET 2023-11-02T18:44:08.305 DEBUG:boto:Path: /ibtljn-59/?prefix=obj31 2023-11-02T18:44:08.305 DEBUG:boto:Data: 2023-11-02T18:44:08.306 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.306 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.307 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.308 DEBUG:boto:Params: {} 2023-11-02T18:44:08.309 DEBUG:boto:Token: None 2023-11-02T18:44:08.309 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.310 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d92dec234724bb5-006543edf8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.316 DEBUG:boto:b'ibtljn-59obj311000falseobj312023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.317 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.317 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.318 DEBUG:boto:path=/ibtljn-59/?prefix=obj31 2023-11-02T18:44:08.318 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj31 2023-11-02T18:44:08.318 DEBUG:boto:Method: GET 2023-11-02T18:44:08.319 DEBUG:boto:Path: /ibtljn-59/?prefix=obj31 2023-11-02T18:44:08.319 DEBUG:boto:Data: 2023-11-02T18:44:08.320 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.320 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.321 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.321 DEBUG:boto:Params: {} 2023-11-02T18:44:08.322 DEBUG:boto:Token: None 2023-11-02T18:44:08.322 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.323 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000118d09e3bed4db8f-006543edf8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.332 DEBUG:boto:b'ibtljn-59obj311000falseobj312023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:44:08.333 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:44:08.333 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:44:08.334 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.334 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.334 DEBUG:boto:Method: GET 2023-11-02T18:44:08.335 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.335 DEBUG:boto:Data: 2023-11-02T18:44:08.336 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.336 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.337 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.337 DEBUG:boto:Params: {} 2023-11-02T18:44:08.338 DEBUG:boto:Token: None 2023-11-02T18:44:08.338 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.339 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zsYucxySFeKmkkIMPDEC3IEFiwM= 2023-11-02T18:44:08.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zsYucxySFeKmkkIMPDEC3IEFiwM=', 'Content-Length': '0'} 2023-11-02T18:44:08.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beb19a038129204d-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.382 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:44:08.383 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:44:08.383 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.384 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.384 DEBUG:boto:Method: GET 2023-11-02T18:44:08.385 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.385 DEBUG:boto:Data: 2023-11-02T18:44:08.386 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.386 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.387 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.387 DEBUG:boto:Params: {} 2023-11-02T18:44:08.388 DEBUG:boto:Token: None 2023-11-02T18:44:08.388 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.389 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zsYucxySFeKmkkIMPDEC3IEFiwM= 2023-11-02T18:44:08.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zsYucxySFeKmkkIMPDEC3IEFiwM=', 'Content-Length': '0'} 2023-11-02T18:44:08.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a1a1437e3e4e9cc-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.394 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:44:08.395 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:44:08.395 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.395 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.396 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.396 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.397 DEBUG:boto:Data: 2023-11-02T18:44:08.397 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.398 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.398 DEBUG:boto:Params: {} 2023-11-02T18:44:08.399 DEBUG:boto:Token: None 2023-11-02T18:44:08.399 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:su2WrqRW5HtHA+nmUFy0MQa8rcA= 2023-11-02T18:44:08.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:su2WrqRW5HtHA+nmUFy0MQa8rcA=', 'Content-Length': '0'} 2023-11-02T18:44:08.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d87b933bd013b1e3-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.404 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:44:08.404 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:44:08.405 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.405 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.406 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.406 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.406 DEBUG:boto:Data: 2023-11-02T18:44:08.407 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.407 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.408 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.408 DEBUG:boto:Params: {} 2023-11-02T18:44:08.409 DEBUG:boto:Token: None 2023-11-02T18:44:08.409 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.410 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:su2WrqRW5HtHA+nmUFy0MQa8rcA= 2023-11-02T18:44:08.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:su2WrqRW5HtHA+nmUFy0MQa8rcA=', 'Content-Length': '0'} 2023-11-02T18:44:08.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f05bc32c6f8ddfd-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:44:08.414 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:44:08.414 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:44:08.415 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.415 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.416 DEBUG:boto:Method: GET 2023-11-02T18:44:08.416 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.417 DEBUG:boto:Data: 2023-11-02T18:44:08.417 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.417 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.418 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.418 DEBUG:boto:Params: {} 2023-11-02T18:44:08.419 DEBUG:boto:Token: None 2023-11-02T18:44:08.419 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.420 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zsYucxySFeKmkkIMPDEC3IEFiwM= 2023-11-02T18:44:08.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zsYucxySFeKmkkIMPDEC3IEFiwM=', 'Content-Length': '0'} 2023-11-02T18:44:08.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051e17fc18f904d04-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.424 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:44:08.425 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:44:08.425 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.425 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.426 DEBUG:boto:Method: GET 2023-11-02T18:44:08.426 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.427 DEBUG:boto:Data: 2023-11-02T18:44:08.427 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.427 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.428 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.428 DEBUG:boto:Params: {} 2023-11-02T18:44:08.429 DEBUG:boto:Token: None 2023-11-02T18:44:08.429 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:44:08.430 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zsYucxySFeKmkkIMPDEC3IEFiwM= 2023-11-02T18:44:08.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zsYucxySFeKmkkIMPDEC3IEFiwM=', 'Content-Length': '0'} 2023-11-02T18:44:08.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e1268d89d1f2ae-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.434 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.435 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.435 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:44:08.435 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:44:08.436 DEBUG:boto:Method: GET 2023-11-02T18:44:08.436 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:44:08.437 DEBUG:boto:Data: 2023-11-02T18:44:08.437 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.438 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.438 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.439 DEBUG:boto:Params: {} 2023-11-02T18:44:08.439 DEBUG:boto:Token: None 2023-11-02T18:44:08.439 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.440 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000917d167210a733ba-006543edf8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.486 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.487 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.488 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.488 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:44:08.489 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:44:08.489 DEBUG:boto:Method: GET 2023-11-02T18:44:08.490 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:44:08.490 DEBUG:boto:Data: 2023-11-02T18:44:08.490 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.491 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.491 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.492 DEBUG:boto:Params: {} 2023-11-02T18:44:08.492 DEBUG:boto:Token: None 2023-11-02T18:44:08.493 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.493 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002decdddd7c01f1e-006543edf8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.503 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.504 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:44:08.504 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:44:08.504 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.505 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:44:08.505 DEBUG:boto:Data: 2023-11-02T18:44:08.506 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.507 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.507 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.508 DEBUG:boto:Params: {} 2023-11-02T18:44:08.508 DEBUG:boto:Token: None 2023-11-02T18:44:08.509 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj33 2023-11-02T18:44:08.509 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JcCajdywNC1jbdXBrXnpShmrqCc= 2023-11-02T18:44:08.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JcCajdywNC1jbdXBrXnpShmrqCc=', 'Content-Length': '0'} 2023-11-02T18:44:08.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041d9daa283dba49d-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.513 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:44:08.513 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:44:08.514 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.514 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:44:08.514 DEBUG:boto:Data: 2023-11-02T18:44:08.515 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.515 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.516 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.516 DEBUG:boto:Params: {} 2023-11-02T18:44:08.516 DEBUG:boto:Token: None 2023-11-02T18:44:08.517 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj33 2023-11-02T18:44:08.517 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JcCajdywNC1jbdXBrXnpShmrqCc= 2023-11-02T18:44:08.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JcCajdywNC1jbdXBrXnpShmrqCc=', 'Content-Length': '0'} 2023-11-02T18:44:08.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034f5f340aa930bc3-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:44:08.521 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:44:08.522 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:44:08.522 DEBUG:boto:Method: GET 2023-11-02T18:44:08.523 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:44:08.523 DEBUG:boto:Data: 2023-11-02T18:44:08.523 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.524 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.524 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.525 DEBUG:boto:Params: {} 2023-11-02T18:44:08.525 DEBUG:boto:Token: None 2023-11-02T18:44:08.526 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj33 2023-11-02T18:44:08.526 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KbnemCd6j2vXYsdltKb7wXSnMPc= 2023-11-02T18:44:08.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KbnemCd6j2vXYsdltKb7wXSnMPc=', 'Content-Length': '0'} 2023-11-02T18:44:08.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e448d9ef44980861-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.530 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:44:08.531 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:44:08.531 DEBUG:boto:Method: GET 2023-11-02T18:44:08.531 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:44:08.532 DEBUG:boto:Data: 2023-11-02T18:44:08.532 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.533 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.533 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.533 DEBUG:boto:Params: {} 2023-11-02T18:44:08.534 DEBUG:boto:Token: None 2023-11-02T18:44:08.534 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj33 2023-11-02T18:44:08.535 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KbnemCd6j2vXYsdltKb7wXSnMPc= 2023-11-02T18:44:08.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KbnemCd6j2vXYsdltKb7wXSnMPc=', 'Content-Length': '0'} 2023-11-02T18:44:08.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c0ab37714f072a5-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.583 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.583 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.583 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:44:08.584 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:44:08.584 DEBUG:boto:Method: GET 2023-11-02T18:44:08.585 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:44:08.585 DEBUG:boto:Data: 2023-11-02T18:44:08.585 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.586 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.586 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.587 DEBUG:boto:Params: {} 2023-11-02T18:44:08.587 DEBUG:boto:Token: None 2023-11-02T18:44:08.588 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.588 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf72cd21962a8365-006543edf8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.594 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.595 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.595 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.595 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:44:08.596 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:44:08.596 DEBUG:boto:Method: GET 2023-11-02T18:44:08.598 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:44:08.598 DEBUG:boto:Data: 2023-11-02T18:44:08.598 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.599 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.599 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.600 DEBUG:boto:Params: {} 2023-11-02T18:44:08.600 DEBUG:boto:Token: None 2023-11-02T18:44:08.601 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003250c67c60ebcc65-006543edf8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.612 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:44:08.613 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:44:08.613 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:44:08.614 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.614 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.615 DEBUG:boto:Method: GET 2023-11-02T18:44:08.615 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.615 DEBUG:boto:Data: 2023-11-02T18:44:08.616 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.616 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.616 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.617 DEBUG:boto:Params: {} 2023-11-02T18:44:08.617 DEBUG:boto:Token: None 2023-11-02T18:44:08.618 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.619 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cCu/bB3aebT0mriHP8HChZrZrL8= 2023-11-02T18:44:08.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cCu/bB3aebT0mriHP8HChZrZrL8=', 'Content-Length': '0'} 2023-11-02T18:44:08.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029eaf692da192dc9-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.662 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:44:08.663 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:44:08.663 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.664 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.664 DEBUG:boto:Method: GET 2023-11-02T18:44:08.665 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.665 DEBUG:boto:Data: 2023-11-02T18:44:08.666 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.666 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.666 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.667 DEBUG:boto:Params: {} 2023-11-02T18:44:08.667 DEBUG:boto:Token: None 2023-11-02T18:44:08.668 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.668 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cCu/bB3aebT0mriHP8HChZrZrL8= 2023-11-02T18:44:08.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cCu/bB3aebT0mriHP8HChZrZrL8=', 'Content-Length': '0'} 2023-11-02T18:44:08.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c12fb040befd4aca-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.673 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:44:08.673 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:44:08.674 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.674 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.675 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.675 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.676 DEBUG:boto:Data: 2023-11-02T18:44:08.676 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.676 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.677 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.677 DEBUG:boto:Params: {} 2023-11-02T18:44:08.678 DEBUG:boto:Token: None 2023-11-02T18:44:08.678 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.679 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3bHkzloSq6AjUVPvAJqgc+VbQnM= 2023-11-02T18:44:08.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3bHkzloSq6AjUVPvAJqgc+VbQnM=', 'Content-Length': '0'} 2023-11-02T18:44:08.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f2ced5fa090a02-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.683 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:44:08.683 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:44:08.683 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.684 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.684 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.685 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.685 DEBUG:boto:Data: 2023-11-02T18:44:08.686 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.686 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.687 DEBUG:boto:Params: {} 2023-11-02T18:44:08.687 DEBUG:boto:Token: None 2023-11-02T18:44:08.688 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.688 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3bHkzloSq6AjUVPvAJqgc+VbQnM= 2023-11-02T18:44:08.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3bHkzloSq6AjUVPvAJqgc+VbQnM=', 'Content-Length': '0'} 2023-11-02T18:44:08.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1e7ad9cbad2ee2e-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:44:08.693 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:44:08.693 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:44:08.694 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.694 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.695 DEBUG:boto:Method: GET 2023-11-02T18:44:08.695 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.696 DEBUG:boto:Data: 2023-11-02T18:44:08.696 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.696 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.697 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.697 DEBUG:boto:Params: {} 2023-11-02T18:44:08.698 DEBUG:boto:Token: None 2023-11-02T18:44:08.698 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.699 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cCu/bB3aebT0mriHP8HChZrZrL8= 2023-11-02T18:44:08.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cCu/bB3aebT0mriHP8HChZrZrL8=', 'Content-Length': '0'} 2023-11-02T18:44:08.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e963ec8618eeb244-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.703 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:44:08.704 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:44:08.704 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.704 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.705 DEBUG:boto:Method: GET 2023-11-02T18:44:08.705 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.706 DEBUG:boto:Data: 2023-11-02T18:44:08.706 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.707 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.707 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.707 DEBUG:boto:Params: {} 2023-11-02T18:44:08.708 DEBUG:boto:Token: None 2023-11-02T18:44:08.708 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:44:08.709 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cCu/bB3aebT0mriHP8HChZrZrL8= 2023-11-02T18:44:08.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cCu/bB3aebT0mriHP8HChZrZrL8=', 'Content-Length': '0'} 2023-11-02T18:44:08.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dac1b6d05c9042e-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.754 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.755 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.756 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:44:08.756 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:44:08.756 DEBUG:boto:Method: GET 2023-11-02T18:44:08.757 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:44:08.757 DEBUG:boto:Data: 2023-11-02T18:44:08.758 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.758 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.759 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.759 DEBUG:boto:Params: {} 2023-11-02T18:44:08.760 DEBUG:boto:Token: None 2023-11-02T18:44:08.760 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.761 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000787f5d69019f7f31-006543edf8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.766 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.767 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.767 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.768 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:44:08.768 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:44:08.769 DEBUG:boto:Method: GET 2023-11-02T18:44:08.769 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:44:08.770 DEBUG:boto:Data: 2023-11-02T18:44:08.770 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.771 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.771 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.771 DEBUG:boto:Params: {} 2023-11-02T18:44:08.772 DEBUG:boto:Token: None 2023-11-02T18:44:08.772 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.773 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce1183cb07845d8a-006543edf8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.783 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.783 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:44:08.784 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:44:08.784 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.784 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:44:08.785 DEBUG:boto:Data: 2023-11-02T18:44:08.785 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.786 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.786 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.786 DEBUG:boto:Params: {} 2023-11-02T18:44:08.787 DEBUG:boto:Token: None 2023-11-02T18:44:08.787 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj34 2023-11-02T18:44:08.788 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HsB/GSltWEj0KQMWqPlLpMFqVuU= 2023-11-02T18:44:08.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HsB/GSltWEj0KQMWqPlLpMFqVuU=', 'Content-Length': '0'} 2023-11-02T18:44:08.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c721867bb8e3fc-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.791 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:44:08.791 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:44:08.792 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.792 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:44:08.793 DEBUG:boto:Data: 2023-11-02T18:44:08.793 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.793 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.794 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.794 DEBUG:boto:Params: {} 2023-11-02T18:44:08.795 DEBUG:boto:Token: None 2023-11-02T18:44:08.795 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj34 2023-11-02T18:44:08.796 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HsB/GSltWEj0KQMWqPlLpMFqVuU= 2023-11-02T18:44:08.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HsB/GSltWEj0KQMWqPlLpMFqVuU=', 'Content-Length': '0'} 2023-11-02T18:44:08.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd9bb5587cd28b5a-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:44:08.800 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:44:08.800 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:44:08.801 DEBUG:boto:Method: GET 2023-11-02T18:44:08.801 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:44:08.802 DEBUG:boto:Data: 2023-11-02T18:44:08.802 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.802 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.803 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.803 DEBUG:boto:Params: {} 2023-11-02T18:44:08.804 DEBUG:boto:Token: None 2023-11-02T18:44:08.804 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj34 2023-11-02T18:44:08.805 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eNOdY1ne+7VZBoA97Ze3yaFESog= 2023-11-02T18:44:08.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eNOdY1ne+7VZBoA97Ze3yaFESog=', 'Content-Length': '0'} 2023-11-02T18:44:08.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ddc92fb40f9f1f3-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.850 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:44:08.851 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:44:08.851 DEBUG:boto:Method: GET 2023-11-02T18:44:08.852 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:44:08.852 DEBUG:boto:Data: 2023-11-02T18:44:08.853 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.853 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.853 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.854 DEBUG:boto:Params: {} 2023-11-02T18:44:08.854 DEBUG:boto:Token: None 2023-11-02T18:44:08.855 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj34 2023-11-02T18:44:08.855 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eNOdY1ne+7VZBoA97Ze3yaFESog= 2023-11-02T18:44:08.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eNOdY1ne+7VZBoA97Ze3yaFESog=', 'Content-Length': '0'} 2023-11-02T18:44:08.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000020f08cf49dabc98-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.860 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.860 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.861 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:44:08.861 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:44:08.861 DEBUG:boto:Method: GET 2023-11-02T18:44:08.862 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:44:08.862 DEBUG:boto:Data: 2023-11-02T18:44:08.863 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.863 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.864 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.864 DEBUG:boto:Params: {} 2023-11-02T18:44:08.865 DEBUG:boto:Token: None 2023-11-02T18:44:08.865 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.866 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000391f6d0217de4677-006543edf8-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.871 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.872 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:08.872 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:08.873 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:44:08.873 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:44:08.874 DEBUG:boto:Method: GET 2023-11-02T18:44:08.874 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:44:08.875 DEBUG:boto:Data: 2023-11-02T18:44:08.875 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.875 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.876 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.876 DEBUG:boto:Params: {} 2023-11-02T18:44:08.877 DEBUG:boto:Token: None 2023-11-02T18:44:08.877 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/ 2023-11-02T18:44:08.878 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs= 2023-11-02T18:44:08.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WqlwyfiKFgAwwPzyu2Fc+MNncJs=', 'Content-Length': '0'} 2023-11-02T18:44:08.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1653008eed7f549-006543edf8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.926 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:08.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:44:08.928 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:44:08.928 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:44:08.929 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.929 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.930 DEBUG:boto:Method: GET 2023-11-02T18:44:08.930 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.930 DEBUG:boto:Data: 2023-11-02T18:44:08.931 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.931 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.932 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.932 DEBUG:boto:Params: {} 2023-11-02T18:44:08.933 DEBUG:boto:Token: None 2023-11-02T18:44:08.933 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.934 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cgRkbWw+/bs42k2ZwFzjGKYk0y4= 2023-11-02T18:44:08.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cgRkbWw+/bs42k2ZwFzjGKYk0y4=', 'Content-Length': '0'} 2023-11-02T18:44:08.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d2d6406294e9cf5-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.938 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:44:08.939 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:44:08.939 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.940 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.940 DEBUG:boto:Method: GET 2023-11-02T18:44:08.941 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.941 DEBUG:boto:Data: 2023-11-02T18:44:08.941 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.942 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.942 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.943 DEBUG:boto:Params: {} 2023-11-02T18:44:08.943 DEBUG:boto:Token: None 2023-11-02T18:44:08.944 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.944 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cgRkbWw+/bs42k2ZwFzjGKYk0y4= 2023-11-02T18:44:08.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cgRkbWw+/bs42k2ZwFzjGKYk0y4=', 'Content-Length': '0'} 2023-11-02T18:44:08.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000254e1a50c00b7654-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.949 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:44:08.949 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:44:08.950 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.950 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.951 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.951 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.952 DEBUG:boto:Data: 2023-11-02T18:44:08.952 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.952 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.953 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.953 DEBUG:boto:Params: {} 2023-11-02T18:44:08.954 DEBUG:boto:Token: None 2023-11-02T18:44:08.954 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.955 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QqV3+NDBYfpVmT6ciB3HJIOCe+0= 2023-11-02T18:44:08.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QqV3+NDBYfpVmT6ciB3HJIOCe+0=', 'Content-Length': '0'} 2023-11-02T18:44:08.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a25fa165df520a8-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.959 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:44:08.959 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:44:08.960 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.960 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.960 DEBUG:boto:Method: HEAD 2023-11-02T18:44:08.961 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.961 DEBUG:boto:Data: 2023-11-02T18:44:08.962 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.962 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.963 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.963 DEBUG:boto:Params: {} 2023-11-02T18:44:08.963 DEBUG:boto:Token: None 2023-11-02T18:44:08.964 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.964 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QqV3+NDBYfpVmT6ciB3HJIOCe+0= 2023-11-02T18:44:08.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QqV3+NDBYfpVmT6ciB3HJIOCe+0=', 'Content-Length': '0'} 2023-11-02T18:44:08.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b7ad96d2ecc572-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:44:08.969 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:44:08.969 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:44:08.970 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.970 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.971 DEBUG:boto:Method: GET 2023-11-02T18:44:08.971 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.971 DEBUG:boto:Data: 2023-11-02T18:44:08.972 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.972 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:08.973 DEBUG:boto:Port: 8001 2023-11-02T18:44:08.973 DEBUG:boto:Params: {} 2023-11-02T18:44:08.974 DEBUG:boto:Token: None 2023-11-02T18:44:08.974 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cgRkbWw+/bs42k2ZwFzjGKYk0y4= 2023-11-02T18:44:08.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cgRkbWw+/bs42k2ZwFzjGKYk0y4=', 'Content-Length': '0'} 2023-11-02T18:44:08.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c14b74e57cbdf4c-006543edf8-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:08.979 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:44:08.979 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:44:08.980 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.980 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.981 DEBUG:boto:Method: GET 2023-11-02T18:44:08.981 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.982 DEBUG:boto:Data: 2023-11-02T18:44:08.982 DEBUG:boto:Headers: {} 2023-11-02T18:44:08.982 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:08.983 DEBUG:boto:Port: 8000 2023-11-02T18:44:08.983 DEBUG:boto:Params: {} 2023-11-02T18:44:08.984 DEBUG:boto:Token: None 2023-11-02T18:44:08.984 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:08 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:44:08.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cgRkbWw+/bs42k2ZwFzjGKYk0y4= 2023-11-02T18:44:08.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cgRkbWw+/bs42k2ZwFzjGKYk0y4=', 'Content-Length': '0'} 2023-11-02T18:44:08.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bccc390c143cb1f6-006543edf8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.030 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.031 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.032 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:44:09.032 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:44:09.032 DEBUG:boto:Method: GET 2023-11-02T18:44:09.033 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:44:09.033 DEBUG:boto:Data: 2023-11-02T18:44:09.034 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.034 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.035 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.035 DEBUG:boto:Params: {} 2023-11-02T18:44:09.036 DEBUG:boto:Token: None 2023-11-02T18:44:09.036 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.037 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ca7446e405f6704-006543edf9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.043 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.043 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.044 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.044 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:44:09.045 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:44:09.045 DEBUG:boto:Method: GET 2023-11-02T18:44:09.045 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:44:09.046 DEBUG:boto:Data: 2023-11-02T18:44:09.046 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.047 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.047 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.048 DEBUG:boto:Params: {} 2023-11-02T18:44:09.048 DEBUG:boto:Token: None 2023-11-02T18:44:09.049 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.049 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eda2b724f4df0f5-006543edf9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.059 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.060 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:44:09.060 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:44:09.061 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.061 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:44:09.061 DEBUG:boto:Data: 2023-11-02T18:44:09.062 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.062 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.063 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.063 DEBUG:boto:Params: {} 2023-11-02T18:44:09.063 DEBUG:boto:Token: None 2023-11-02T18:44:09.064 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj38 2023-11-02T18:44:09.064 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1/bbZcMW4v4v+VCwG9eUvZIy1D4= 2023-11-02T18:44:09.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1/bbZcMW4v4v+VCwG9eUvZIy1D4=', 'Content-Length': '0'} 2023-11-02T18:44:09.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002310acbaab3604ac-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.068 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:44:09.068 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:44:09.068 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.069 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:44:09.069 DEBUG:boto:Data: 2023-11-02T18:44:09.070 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.071 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.071 DEBUG:boto:Params: {} 2023-11-02T18:44:09.071 DEBUG:boto:Token: None 2023-11-02T18:44:09.072 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj38 2023-11-02T18:44:09.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1/bbZcMW4v4v+VCwG9eUvZIy1D4= 2023-11-02T18:44:09.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1/bbZcMW4v4v+VCwG9eUvZIy1D4=', 'Content-Length': '0'} 2023-11-02T18:44:09.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd34f37789e7fa6e-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:44:09.076 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:44:09.077 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:44:09.077 DEBUG:boto:Method: GET 2023-11-02T18:44:09.078 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:44:09.078 DEBUG:boto:Data: 2023-11-02T18:44:09.079 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.079 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.079 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.080 DEBUG:boto:Params: {} 2023-11-02T18:44:09.080 DEBUG:boto:Token: None 2023-11-02T18:44:09.081 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj38 2023-11-02T18:44:09.081 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Mi87YU1mTVzh26W9Q1dlvlwIXM= 2023-11-02T18:44:09.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Mi87YU1mTVzh26W9Q1dlvlwIXM=', 'Content-Length': '0'} 2023-11-02T18:44:09.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000455a678ccf2021d8-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.130 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:44:09.131 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:44:09.131 DEBUG:boto:Method: GET 2023-11-02T18:44:09.132 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:44:09.132 DEBUG:boto:Data: 2023-11-02T18:44:09.133 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.133 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.134 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.134 DEBUG:boto:Params: {} 2023-11-02T18:44:09.134 DEBUG:boto:Token: None 2023-11-02T18:44:09.135 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj38 2023-11-02T18:44:09.136 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7Mi87YU1mTVzh26W9Q1dlvlwIXM= 2023-11-02T18:44:09.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7Mi87YU1mTVzh26W9Q1dlvlwIXM=', 'Content-Length': '0'} 2023-11-02T18:44:09.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a2a4c0cf52561b7-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.139 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.140 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.140 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:44:09.141 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:44:09.141 DEBUG:boto:Method: GET 2023-11-02T18:44:09.142 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:44:09.142 DEBUG:boto:Data: 2023-11-02T18:44:09.143 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.143 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.143 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.144 DEBUG:boto:Params: {} 2023-11-02T18:44:09.144 DEBUG:boto:Token: None 2023-11-02T18:44:09.145 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.145 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003476ba0deb87ab28-006543edf9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.153 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.153 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.154 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.154 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:44:09.155 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:44:09.155 DEBUG:boto:Method: GET 2023-11-02T18:44:09.155 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:44:09.156 DEBUG:boto:Data: 2023-11-02T18:44:09.156 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.157 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.157 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.158 DEBUG:boto:Params: {} 2023-11-02T18:44:09.158 DEBUG:boto:Token: None 2023-11-02T18:44:09.159 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.159 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b2f00f5e9158d3c-006543edf9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.210 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:44:09.212 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:44:09.212 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:44:09.213 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.214 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.214 DEBUG:boto:Method: GET 2023-11-02T18:44:09.215 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.215 DEBUG:boto:Data: 2023-11-02T18:44:09.216 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.216 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.216 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.217 DEBUG:boto:Params: {} 2023-11-02T18:44:09.217 DEBUG:boto:Token: None 2023-11-02T18:44:09.218 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.218 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rsoOhBfyYgEVC6Qnu6pcAPxulKM= 2023-11-02T18:44:09.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rsoOhBfyYgEVC6Qnu6pcAPxulKM=', 'Content-Length': '0'} 2023-11-02T18:44:09.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035fcbdcdf0300238-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.223 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:44:09.223 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:44:09.224 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.224 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.224 DEBUG:boto:Method: GET 2023-11-02T18:44:09.225 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.225 DEBUG:boto:Data: 2023-11-02T18:44:09.226 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.226 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.227 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.227 DEBUG:boto:Params: {} 2023-11-02T18:44:09.227 DEBUG:boto:Token: None 2023-11-02T18:44:09.228 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.229 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rsoOhBfyYgEVC6Qnu6pcAPxulKM= 2023-11-02T18:44:09.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rsoOhBfyYgEVC6Qnu6pcAPxulKM=', 'Content-Length': '0'} 2023-11-02T18:44:09.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e8bf71973e8e172-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.233 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:44:09.233 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:44:09.234 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.234 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.235 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.235 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.235 DEBUG:boto:Data: 2023-11-02T18:44:09.236 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.236 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.237 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.237 DEBUG:boto:Params: {} 2023-11-02T18:44:09.238 DEBUG:boto:Token: None 2023-11-02T18:44:09.238 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.239 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+YDKDpMhPZWeuPgWDa1t56+NLKI= 2023-11-02T18:44:09.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+YDKDpMhPZWeuPgWDa1t56+NLKI=', 'Content-Length': '0'} 2023-11-02T18:44:09.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062e5f38e421bbebb-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.243 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:44:09.243 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:44:09.244 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.244 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.245 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.245 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.245 DEBUG:boto:Data: 2023-11-02T18:44:09.246 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.247 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.247 DEBUG:boto:Params: {} 2023-11-02T18:44:09.248 DEBUG:boto:Token: None 2023-11-02T18:44:09.248 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.249 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+YDKDpMhPZWeuPgWDa1t56+NLKI= 2023-11-02T18:44:09.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+YDKDpMhPZWeuPgWDa1t56+NLKI=', 'Content-Length': '0'} 2023-11-02T18:44:09.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fe16c56f19a2816-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:44:09.253 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:44:09.254 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:44:09.254 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.254 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.255 DEBUG:boto:Method: GET 2023-11-02T18:44:09.255 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.256 DEBUG:boto:Data: 2023-11-02T18:44:09.256 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.257 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.257 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.257 DEBUG:boto:Params: {} 2023-11-02T18:44:09.258 DEBUG:boto:Token: None 2023-11-02T18:44:09.259 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.259 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rsoOhBfyYgEVC6Qnu6pcAPxulKM= 2023-11-02T18:44:09.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rsoOhBfyYgEVC6Qnu6pcAPxulKM=', 'Content-Length': '0'} 2023-11-02T18:44:09.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7b00c832f4eb8b6-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.263 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:44:09.264 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:44:09.264 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.265 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.265 DEBUG:boto:Method: GET 2023-11-02T18:44:09.266 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.266 DEBUG:boto:Data: 2023-11-02T18:44:09.267 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.268 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.269 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.269 DEBUG:boto:Params: {} 2023-11-02T18:44:09.269 DEBUG:boto:Token: None 2023-11-02T18:44:09.270 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:44:09.270 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rsoOhBfyYgEVC6Qnu6pcAPxulKM= 2023-11-02T18:44:09.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rsoOhBfyYgEVC6Qnu6pcAPxulKM=', 'Content-Length': '0'} 2023-11-02T18:44:09.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd65ce7b184b50a-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.318 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.319 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.320 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:44:09.320 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:44:09.320 DEBUG:boto:Method: GET 2023-11-02T18:44:09.321 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:44:09.321 DEBUG:boto:Data: 2023-11-02T18:44:09.322 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.322 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.322 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.323 DEBUG:boto:Params: {} 2023-11-02T18:44:09.323 DEBUG:boto:Token: None 2023-11-02T18:44:09.324 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.324 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028a4610f5d2f6851-006543edf9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.330 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.332 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.332 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.332 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:44:09.333 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:44:09.334 DEBUG:boto:Method: GET 2023-11-02T18:44:09.334 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:44:09.335 DEBUG:boto:Data: 2023-11-02T18:44:09.335 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.335 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.336 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.336 DEBUG:boto:Params: {} 2023-11-02T18:44:09.337 DEBUG:boto:Token: None 2023-11-02T18:44:09.337 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.338 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea286d155f854b09-006543edf9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.348 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.349 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:44:09.349 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:44:09.349 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.350 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:44:09.351 DEBUG:boto:Data: 2023-11-02T18:44:09.351 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.351 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.352 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.352 DEBUG:boto:Params: {} 2023-11-02T18:44:09.353 DEBUG:boto:Token: None 2023-11-02T18:44:09.353 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj44 2023-11-02T18:44:09.354 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HVgPnskG+6kOQ7I2ZjVm4nCK8wg= 2023-11-02T18:44:09.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HVgPnskG+6kOQ7I2ZjVm4nCK8wg=', 'Content-Length': '0'} 2023-11-02T18:44:09.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6bc0377e82220e7-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.357 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:44:09.358 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:44:09.358 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.359 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:44:09.359 DEBUG:boto:Data: 2023-11-02T18:44:09.359 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.360 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.360 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.361 DEBUG:boto:Params: {} 2023-11-02T18:44:09.361 DEBUG:boto:Token: None 2023-11-02T18:44:09.362 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj44 2023-11-02T18:44:09.362 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HVgPnskG+6kOQ7I2ZjVm4nCK8wg= 2023-11-02T18:44:09.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HVgPnskG+6kOQ7I2ZjVm4nCK8wg=', 'Content-Length': '0'} 2023-11-02T18:44:09.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f27f597d43b1dba-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:44:09.366 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:44:09.367 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:44:09.367 DEBUG:boto:Method: GET 2023-11-02T18:44:09.368 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:44:09.368 DEBUG:boto:Data: 2023-11-02T18:44:09.369 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.369 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.370 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.370 DEBUG:boto:Params: {} 2023-11-02T18:44:09.371 DEBUG:boto:Token: None 2023-11-02T18:44:09.371 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj44 2023-11-02T18:44:09.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ERzXUR68f8oYCYbV9Lw+2ubguR0= 2023-11-02T18:44:09.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ERzXUR68f8oYCYbV9Lw+2ubguR0=', 'Content-Length': '0'} 2023-11-02T18:44:09.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5c9cb894f80007f-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.376 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:44:09.376 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:44:09.377 DEBUG:boto:Method: GET 2023-11-02T18:44:09.377 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:44:09.378 DEBUG:boto:Data: 2023-11-02T18:44:09.378 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.378 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.379 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.379 DEBUG:boto:Params: {} 2023-11-02T18:44:09.380 DEBUG:boto:Token: None 2023-11-02T18:44:09.380 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj44 2023-11-02T18:44:09.381 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ERzXUR68f8oYCYbV9Lw+2ubguR0= 2023-11-02T18:44:09.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ERzXUR68f8oYCYbV9Lw+2ubguR0=', 'Content-Length': '0'} 2023-11-02T18:44:09.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5be07062689d786-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.385 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.386 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.386 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:44:09.386 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:44:09.387 DEBUG:boto:Method: GET 2023-11-02T18:44:09.387 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:44:09.388 DEBUG:boto:Data: 2023-11-02T18:44:09.388 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.389 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.389 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.389 DEBUG:boto:Params: {} 2023-11-02T18:44:09.390 DEBUG:boto:Token: None 2023-11-02T18:44:09.390 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.391 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce45ef35e9a32bae-006543edf9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.438 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.439 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.439 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.440 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:44:09.440 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:44:09.441 DEBUG:boto:Method: GET 2023-11-02T18:44:09.441 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:44:09.441 DEBUG:boto:Data: 2023-11-02T18:44:09.442 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.442 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.443 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.443 DEBUG:boto:Params: {} 2023-11-02T18:44:09.444 DEBUG:boto:Token: None 2023-11-02T18:44:09.444 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.445 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038aabfe2418d1c65-006543edf9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.455 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:44:09.457 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:44:09.457 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:44:09.457 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.458 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.458 DEBUG:boto:Method: GET 2023-11-02T18:44:09.459 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.459 DEBUG:boto:Data: 2023-11-02T18:44:09.460 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.460 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.460 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.461 DEBUG:boto:Params: {} 2023-11-02T18:44:09.461 DEBUG:boto:Token: None 2023-11-02T18:44:09.462 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.462 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/bcIxxVjr8kz9neUb3O1Ss2D9QA= 2023-11-02T18:44:09.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/bcIxxVjr8kz9neUb3O1Ss2D9QA=', 'Content-Length': '0'} 2023-11-02T18:44:09.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fd79094434c8d35-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.466 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:44:09.467 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:44:09.467 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.467 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.468 DEBUG:boto:Method: GET 2023-11-02T18:44:09.468 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.469 DEBUG:boto:Data: 2023-11-02T18:44:09.469 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.470 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.470 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.471 DEBUG:boto:Params: {} 2023-11-02T18:44:09.471 DEBUG:boto:Token: None 2023-11-02T18:44:09.472 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.472 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/bcIxxVjr8kz9neUb3O1Ss2D9QA= 2023-11-02T18:44:09.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/bcIxxVjr8kz9neUb3O1Ss2D9QA=', 'Content-Length': '0'} 2023-11-02T18:44:09.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e3ba367e1dd2d60-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.518 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:44:09.519 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:44:09.519 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.520 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.520 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.521 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.521 DEBUG:boto:Data: 2023-11-02T18:44:09.521 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.522 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.523 DEBUG:boto:Params: {} 2023-11-02T18:44:09.523 DEBUG:boto:Token: None 2023-11-02T18:44:09.523 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D9Wlt2hco2+bqQmXKygrOBGFHLs= 2023-11-02T18:44:09.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D9Wlt2hco2+bqQmXKygrOBGFHLs=', 'Content-Length': '0'} 2023-11-02T18:44:09.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da10af6c929aab07-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.528 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:44:09.528 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:44:09.529 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.529 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.529 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.530 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.530 DEBUG:boto:Data: 2023-11-02T18:44:09.531 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.531 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.531 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.532 DEBUG:boto:Params: {} 2023-11-02T18:44:09.532 DEBUG:boto:Token: None 2023-11-02T18:44:09.533 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.533 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D9Wlt2hco2+bqQmXKygrOBGFHLs= 2023-11-02T18:44:09.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D9Wlt2hco2+bqQmXKygrOBGFHLs=', 'Content-Length': '0'} 2023-11-02T18:44:09.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014dc9133660faa6f-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:44:09.538 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:44:09.538 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:44:09.539 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.539 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.540 DEBUG:boto:Method: GET 2023-11-02T18:44:09.540 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.540 DEBUG:boto:Data: 2023-11-02T18:44:09.541 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.542 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.542 DEBUG:boto:Params: {} 2023-11-02T18:44:09.543 DEBUG:boto:Token: None 2023-11-02T18:44:09.543 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/bcIxxVjr8kz9neUb3O1Ss2D9QA= 2023-11-02T18:44:09.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/bcIxxVjr8kz9neUb3O1Ss2D9QA=', 'Content-Length': '0'} 2023-11-02T18:44:09.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008463f07568f1b428-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.548 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:44:09.548 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:44:09.549 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.549 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.549 DEBUG:boto:Method: GET 2023-11-02T18:44:09.550 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.550 DEBUG:boto:Data: 2023-11-02T18:44:09.551 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.551 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.552 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.552 DEBUG:boto:Params: {} 2023-11-02T18:44:09.553 DEBUG:boto:Token: None 2023-11-02T18:44:09.553 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:44:09.554 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/bcIxxVjr8kz9neUb3O1Ss2D9QA= 2023-11-02T18:44:09.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/bcIxxVjr8kz9neUb3O1Ss2D9QA=', 'Content-Length': '0'} 2023-11-02T18:44:09.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4b6b6dd3c6801ad-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.558 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.558 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.559 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:44:09.559 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:44:09.560 DEBUG:boto:Method: GET 2023-11-02T18:44:09.560 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:44:09.560 DEBUG:boto:Data: 2023-11-02T18:44:09.561 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.561 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.562 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.562 DEBUG:boto:Params: {} 2023-11-02T18:44:09.563 DEBUG:boto:Token: None 2023-11-02T18:44:09.563 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.564 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015d0a6d293ae485f-006543edf9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.614 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.615 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.616 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.616 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:44:09.617 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:44:09.617 DEBUG:boto:Method: GET 2023-11-02T18:44:09.617 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:44:09.618 DEBUG:boto:Data: 2023-11-02T18:44:09.618 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.619 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.620 DEBUG:boto:Params: {} 2023-11-02T18:44:09.620 DEBUG:boto:Token: None 2023-11-02T18:44:09.621 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004189102814e46d61-006543edf9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.631 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.631 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:44:09.632 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:44:09.632 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.633 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:44:09.633 DEBUG:boto:Data: 2023-11-02T18:44:09.634 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.634 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.634 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.635 DEBUG:boto:Params: {} 2023-11-02T18:44:09.635 DEBUG:boto:Token: None 2023-11-02T18:44:09.636 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj46 2023-11-02T18:44:09.636 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xz0wlJ+rADy9gvEJc9rqtrWuchE= 2023-11-02T18:44:09.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xz0wlJ+rADy9gvEJc9rqtrWuchE=', 'Content-Length': '0'} 2023-11-02T18:44:09.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e32aada98137aaa5-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.639 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:44:09.640 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:44:09.640 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.641 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:44:09.641 DEBUG:boto:Data: 2023-11-02T18:44:09.641 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.642 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.642 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.643 DEBUG:boto:Params: {} 2023-11-02T18:44:09.643 DEBUG:boto:Token: None 2023-11-02T18:44:09.643 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj46 2023-11-02T18:44:09.644 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xz0wlJ+rADy9gvEJc9rqtrWuchE= 2023-11-02T18:44:09.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xz0wlJ+rADy9gvEJc9rqtrWuchE=', 'Content-Length': '0'} 2023-11-02T18:44:09.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063ea34d03abb62da-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:44:09.648 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:44:09.648 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:44:09.648 DEBUG:boto:Method: GET 2023-11-02T18:44:09.649 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:44:09.649 DEBUG:boto:Data: 2023-11-02T18:44:09.650 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.650 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.651 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.651 DEBUG:boto:Params: {} 2023-11-02T18:44:09.651 DEBUG:boto:Token: None 2023-11-02T18:44:09.652 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj46 2023-11-02T18:44:09.652 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UWD51GrHcN49ZC3fNCCmupsuP8U= 2023-11-02T18:44:09.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UWD51GrHcN49ZC3fNCCmupsuP8U=', 'Content-Length': '0'} 2023-11-02T18:44:09.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ae6082cdcaeb76e-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.656 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:44:09.657 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:44:09.657 DEBUG:boto:Method: GET 2023-11-02T18:44:09.657 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:44:09.658 DEBUG:boto:Data: 2023-11-02T18:44:09.658 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.659 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.659 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.660 DEBUG:boto:Params: {} 2023-11-02T18:44:09.660 DEBUG:boto:Token: None 2023-11-02T18:44:09.661 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj46 2023-11-02T18:44:09.661 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UWD51GrHcN49ZC3fNCCmupsuP8U= 2023-11-02T18:44:09.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UWD51GrHcN49ZC3fNCCmupsuP8U=', 'Content-Length': '0'} 2023-11-02T18:44:09.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045b303f440c64e9e-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.706 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.707 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.708 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:44:09.708 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:44:09.708 DEBUG:boto:Method: GET 2023-11-02T18:44:09.709 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:44:09.709 DEBUG:boto:Data: 2023-11-02T18:44:09.710 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.710 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.711 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.711 DEBUG:boto:Params: {} 2023-11-02T18:44:09.712 DEBUG:boto:Token: None 2023-11-02T18:44:09.712 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.713 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087fd083d4a5defa9-006543edf9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.719 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.720 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.720 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.721 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:44:09.721 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:44:09.722 DEBUG:boto:Method: GET 2023-11-02T18:44:09.722 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:44:09.722 DEBUG:boto:Data: 2023-11-02T18:44:09.723 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.723 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.724 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.724 DEBUG:boto:Params: {} 2023-11-02T18:44:09.725 DEBUG:boto:Token: None 2023-11-02T18:44:09.725 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.726 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1dd94bced2ccb54-006543edf9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.735 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:44:09.736 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:44:09.737 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:44:09.737 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.738 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.738 DEBUG:boto:Method: GET 2023-11-02T18:44:09.739 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.739 DEBUG:boto:Data: 2023-11-02T18:44:09.739 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.740 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.740 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.741 DEBUG:boto:Params: {} 2023-11-02T18:44:09.741 DEBUG:boto:Token: None 2023-11-02T18:44:09.742 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.742 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WL+a/7QdGgExUilA+AkwfvieksA= 2023-11-02T18:44:09.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WL+a/7QdGgExUilA+AkwfvieksA=', 'Content-Length': '0'} 2023-11-02T18:44:09.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e6fc8093fae792e-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.786 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:44:09.787 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:44:09.788 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.788 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.788 DEBUG:boto:Method: GET 2023-11-02T18:44:09.789 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.789 DEBUG:boto:Data: 2023-11-02T18:44:09.790 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.791 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.791 DEBUG:boto:Params: {} 2023-11-02T18:44:09.792 DEBUG:boto:Token: None 2023-11-02T18:44:09.792 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WL+a/7QdGgExUilA+AkwfvieksA= 2023-11-02T18:44:09.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WL+a/7QdGgExUilA+AkwfvieksA=', 'Content-Length': '0'} 2023-11-02T18:44:09.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a49efb06f06bdb12-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.797 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:44:09.797 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:44:09.798 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.798 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.799 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.799 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.799 DEBUG:boto:Data: 2023-11-02T18:44:09.800 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.800 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.801 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.801 DEBUG:boto:Params: {} 2023-11-02T18:44:09.802 DEBUG:boto:Token: None 2023-11-02T18:44:09.802 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.803 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o/XX8KbQcwutolAN3d4SDqDDcQc= 2023-11-02T18:44:09.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o/XX8KbQcwutolAN3d4SDqDDcQc=', 'Content-Length': '0'} 2023-11-02T18:44:09.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b5514304fecb325-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.806 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:44:09.807 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:44:09.807 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.808 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.808 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.808 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.809 DEBUG:boto:Data: 2023-11-02T18:44:09.809 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.810 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.810 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.810 DEBUG:boto:Params: {} 2023-11-02T18:44:09.811 DEBUG:boto:Token: None 2023-11-02T18:44:09.811 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.812 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o/XX8KbQcwutolAN3d4SDqDDcQc= 2023-11-02T18:44:09.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o/XX8KbQcwutolAN3d4SDqDDcQc=', 'Content-Length': '0'} 2023-11-02T18:44:09.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000017a75ed97bee251-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:44:09.816 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:44:09.817 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:44:09.817 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.817 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.818 DEBUG:boto:Method: GET 2023-11-02T18:44:09.822 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.822 DEBUG:boto:Data: 2023-11-02T18:44:09.822 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.823 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.823 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.824 DEBUG:boto:Params: {} 2023-11-02T18:44:09.824 DEBUG:boto:Token: None 2023-11-02T18:44:09.825 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.825 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WL+a/7QdGgExUilA+AkwfvieksA= 2023-11-02T18:44:09.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WL+a/7QdGgExUilA+AkwfvieksA=', 'Content-Length': '0'} 2023-11-02T18:44:09.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000779011b1c8b4f656-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.829 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:44:09.830 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:44:09.830 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.831 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.831 DEBUG:boto:Method: GET 2023-11-02T18:44:09.831 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.832 DEBUG:boto:Data: 2023-11-02T18:44:09.832 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.833 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.833 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.833 DEBUG:boto:Params: {} 2023-11-02T18:44:09.834 DEBUG:boto:Token: None 2023-11-02T18:44:09.834 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:44:09.835 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WL+a/7QdGgExUilA+AkwfvieksA= 2023-11-02T18:44:09.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WL+a/7QdGgExUilA+AkwfvieksA=', 'Content-Length': '0'} 2023-11-02T18:44:09.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000673949a82741f4af-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.882 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.883 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.883 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:44:09.884 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:44:09.884 DEBUG:boto:Method: GET 2023-11-02T18:44:09.885 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:44:09.885 DEBUG:boto:Data: 2023-11-02T18:44:09.886 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.887 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.887 DEBUG:boto:Params: {} 2023-11-02T18:44:09.888 DEBUG:boto:Token: None 2023-11-02T18:44:09.888 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.889 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6987cdb4ffb326f-006543edf9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.894 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.895 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.895 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.895 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:44:09.896 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:44:09.896 DEBUG:boto:Method: GET 2023-11-02T18:44:09.897 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:44:09.897 DEBUG:boto:Data: 2023-11-02T18:44:09.897 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.898 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.898 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.899 DEBUG:boto:Params: {} 2023-11-02T18:44:09.899 DEBUG:boto:Token: None 2023-11-02T18:44:09.900 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.900 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066bb0a7de871340a-006543edf9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.909 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:09.910 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:44:09.910 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:44:09.911 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.911 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:44:09.912 DEBUG:boto:Data: 2023-11-02T18:44:09.912 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.912 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.913 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.913 DEBUG:boto:Params: {} 2023-11-02T18:44:09.914 DEBUG:boto:Token: None 2023-11-02T18:44:09.914 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj51 2023-11-02T18:44:09.914 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KU7HgCZ8nQWhopIqWYSXGBVdmtk= 2023-11-02T18:44:09.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KU7HgCZ8nQWhopIqWYSXGBVdmtk=', 'Content-Length': '0'} 2023-11-02T18:44:09.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d90be9f5d62df23b-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.918 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:44:09.918 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:44:09.919 DEBUG:boto:Method: HEAD 2023-11-02T18:44:09.919 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:44:09.920 DEBUG:boto:Data: 2023-11-02T18:44:09.920 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.920 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.921 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.921 DEBUG:boto:Params: {} 2023-11-02T18:44:09.922 DEBUG:boto:Token: None 2023-11-02T18:44:09.922 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj51 2023-11-02T18:44:09.923 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KU7HgCZ8nQWhopIqWYSXGBVdmtk= 2023-11-02T18:44:09.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KU7HgCZ8nQWhopIqWYSXGBVdmtk=', 'Content-Length': '0'} 2023-11-02T18:44:09.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006229e0154dc73200-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:44:09.926 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:44:09.927 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:44:09.927 DEBUG:boto:Method: GET 2023-11-02T18:44:09.928 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:44:09.928 DEBUG:boto:Data: 2023-11-02T18:44:09.929 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.929 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.930 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.930 DEBUG:boto:Params: {} 2023-11-02T18:44:09.931 DEBUG:boto:Token: None 2023-11-02T18:44:09.931 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj51 2023-11-02T18:44:09.931 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ED0g4tuXP8dMUDkr5C+Vl2HqAOo= 2023-11-02T18:44:09.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ED0g4tuXP8dMUDkr5C+Vl2HqAOo=', 'Content-Length': '0'} 2023-11-02T18:44:09.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000ecf5b10638dac3-006543edf9-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.935 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:44:09.936 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:44:09.936 DEBUG:boto:Method: GET 2023-11-02T18:44:09.937 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:44:09.937 DEBUG:boto:Data: 2023-11-02T18:44:09.937 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.938 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:09.938 DEBUG:boto:Port: 8000 2023-11-02T18:44:09.939 DEBUG:boto:Params: {} 2023-11-02T18:44:09.939 DEBUG:boto:Token: None 2023-11-02T18:44:09.940 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/obj51 2023-11-02T18:44:09.940 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ED0g4tuXP8dMUDkr5C+Vl2HqAOo= 2023-11-02T18:44:09.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ED0g4tuXP8dMUDkr5C+Vl2HqAOo=', 'Content-Length': '0'} 2023-11-02T18:44:09.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df36c1330ab4a8d6-006543edf9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:09.944 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:09.945 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:09.945 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:44:09.945 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:44:09.946 DEBUG:boto:Method: GET 2023-11-02T18:44:09.946 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:44:09.947 DEBUG:boto:Data: 2023-11-02T18:44:09.947 DEBUG:boto:Headers: {} 2023-11-02T18:44:09.948 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:09.948 DEBUG:boto:Port: 8001 2023-11-02T18:44:09.948 DEBUG:boto:Params: {} 2023-11-02T18:44:09.949 DEBUG:boto:Token: None 2023-11-02T18:44:09.949 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:09 GMT /ibtljn-59/ 2023-11-02T18:44:09.950 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA= 2023-11-02T18:44:09.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xtggcpq6cmz9K8kc9lNm/FrVoeA=', 'Content-Length': '0'} 2023-11-02T18:44:09.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f9201c1d755e39-006543edf9-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.002 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.003 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.004 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.004 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:44:10.004 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:44:10.005 DEBUG:boto:Method: GET 2023-11-02T18:44:10.005 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:44:10.006 DEBUG:boto:Data: 2023-11-02T18:44:10.006 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.007 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.007 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.007 DEBUG:boto:Params: {} 2023-11-02T18:44:10.008 DEBUG:boto:Token: None 2023-11-02T18:44:10.008 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.009 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013a50d2fa3757407-006543edfa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.019 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:44:10.020 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:44:10.020 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:44:10.021 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.021 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.022 DEBUG:boto:Method: GET 2023-11-02T18:44:10.022 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.023 DEBUG:boto:Data: 2023-11-02T18:44:10.023 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.024 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.024 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.024 DEBUG:boto:Params: {} 2023-11-02T18:44:10.025 DEBUG:boto:Token: None 2023-11-02T18:44:10.025 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.026 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2H0yv43KEvlmo4PFctN8nIJ0NyY= 2023-11-02T18:44:10.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2H0yv43KEvlmo4PFctN8nIJ0NyY=', 'Content-Length': '0'} 2023-11-02T18:44:10.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ba1d6ad0db4c59-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.035 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:44:10.035 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:44:10.036 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.036 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.036 DEBUG:boto:Method: GET 2023-11-02T18:44:10.037 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.037 DEBUG:boto:Data: 2023-11-02T18:44:10.038 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.038 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.039 DEBUG:boto:Params: {} 2023-11-02T18:44:10.039 DEBUG:boto:Token: None 2023-11-02T18:44:10.040 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.040 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2H0yv43KEvlmo4PFctN8nIJ0NyY= 2023-11-02T18:44:10.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2H0yv43KEvlmo4PFctN8nIJ0NyY=', 'Content-Length': '0'} 2023-11-02T18:44:10.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a12a52f43eb18908-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.086 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:44:10.087 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:44:10.087 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.088 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.088 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.088 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.089 DEBUG:boto:Data: 2023-11-02T18:44:10.089 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.090 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.090 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.090 DEBUG:boto:Params: {} 2023-11-02T18:44:10.091 DEBUG:boto:Token: None 2023-11-02T18:44:10.091 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.092 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Hu6NsoOcP2FDeq62swPw++xvUYk= 2023-11-02T18:44:10.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Hu6NsoOcP2FDeq62swPw++xvUYk=', 'Content-Length': '0'} 2023-11-02T18:44:10.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b24607d95e4cc53-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.096 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:44:10.096 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:44:10.096 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.097 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.097 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.098 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.098 DEBUG:boto:Data: 2023-11-02T18:44:10.098 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.099 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.099 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.099 DEBUG:boto:Params: {} 2023-11-02T18:44:10.100 DEBUG:boto:Token: None 2023-11-02T18:44:10.100 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.101 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Hu6NsoOcP2FDeq62swPw++xvUYk= 2023-11-02T18:44:10.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Hu6NsoOcP2FDeq62swPw++xvUYk=', 'Content-Length': '0'} 2023-11-02T18:44:10.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001743d90e71af8334-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:44:10.105 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:44:10.106 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:44:10.106 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.107 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.107 DEBUG:boto:Method: GET 2023-11-02T18:44:10.107 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.108 DEBUG:boto:Data: 2023-11-02T18:44:10.108 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.109 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.109 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.109 DEBUG:boto:Params: {} 2023-11-02T18:44:10.110 DEBUG:boto:Token: None 2023-11-02T18:44:10.111 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.111 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2H0yv43KEvlmo4PFctN8nIJ0NyY= 2023-11-02T18:44:10.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2H0yv43KEvlmo4PFctN8nIJ0NyY=', 'Content-Length': '0'} 2023-11-02T18:44:10.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa32f49f65ec2761-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.115 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:44:10.116 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:44:10.116 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.116 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.117 DEBUG:boto:Method: GET 2023-11-02T18:44:10.117 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.118 DEBUG:boto:Data: 2023-11-02T18:44:10.118 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.118 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.119 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.119 DEBUG:boto:Params: {} 2023-11-02T18:44:10.120 DEBUG:boto:Token: None 2023-11-02T18:44:10.120 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:44:10.121 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2H0yv43KEvlmo4PFctN8nIJ0NyY= 2023-11-02T18:44:10.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2H0yv43KEvlmo4PFctN8nIJ0NyY=', 'Content-Length': '0'} 2023-11-02T18:44:10.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae115d7433da1f44-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.125 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.126 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.126 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:44:10.126 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:44:10.127 DEBUG:boto:Method: GET 2023-11-02T18:44:10.127 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:44:10.128 DEBUG:boto:Data: 2023-11-02T18:44:10.128 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.128 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.129 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.129 DEBUG:boto:Params: {} 2023-11-02T18:44:10.130 DEBUG:boto:Token: None 2023-11-02T18:44:10.130 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.131 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072dc4779ec41d553-006543edfa-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.182 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.183 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.184 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.184 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:44:10.185 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:44:10.185 DEBUG:boto:Method: GET 2023-11-02T18:44:10.186 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:44:10.186 DEBUG:boto:Data: 2023-11-02T18:44:10.187 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.187 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.187 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.188 DEBUG:boto:Params: {} 2023-11-02T18:44:10.188 DEBUG:boto:Token: None 2023-11-02T18:44:10.189 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.189 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf55a931a45a4fe7-006543edfa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.199 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.200 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:44:10.200 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:44:10.201 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.201 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:44:10.201 DEBUG:boto:Data: 2023-11-02T18:44:10.202 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.203 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.203 DEBUG:boto:Params: {} 2023-11-02T18:44:10.203 DEBUG:boto:Token: None 2023-11-02T18:44:10.204 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj55 2023-11-02T18:44:10.204 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YTpsUrs35B4mZv4dY9lwVHEOS4U= 2023-11-02T18:44:10.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YTpsUrs35B4mZv4dY9lwVHEOS4U=', 'Content-Length': '0'} 2023-11-02T18:44:10.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0397ffb3d2027af-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.208 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:44:10.208 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:44:10.209 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.209 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:44:10.209 DEBUG:boto:Data: 2023-11-02T18:44:10.210 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.210 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.211 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.211 DEBUG:boto:Params: {} 2023-11-02T18:44:10.211 DEBUG:boto:Token: None 2023-11-02T18:44:10.212 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj55 2023-11-02T18:44:10.212 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YTpsUrs35B4mZv4dY9lwVHEOS4U= 2023-11-02T18:44:10.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YTpsUrs35B4mZv4dY9lwVHEOS4U=', 'Content-Length': '0'} 2023-11-02T18:44:10.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6b902e85dfa4743-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:44:10.216 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:44:10.217 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:44:10.217 DEBUG:boto:Method: GET 2023-11-02T18:44:10.217 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:44:10.218 DEBUG:boto:Data: 2023-11-02T18:44:10.218 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.219 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.219 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.220 DEBUG:boto:Params: {} 2023-11-02T18:44:10.220 DEBUG:boto:Token: None 2023-11-02T18:44:10.221 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj55 2023-11-02T18:44:10.221 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iHHf3ffUrcfGeGeC9GvQUZ9N1bo= 2023-11-02T18:44:10.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iHHf3ffUrcfGeGeC9GvQUZ9N1bo=', 'Content-Length': '0'} 2023-11-02T18:44:10.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7c8df8eea7a9ca2-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.224 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:44:10.225 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:44:10.225 DEBUG:boto:Method: GET 2023-11-02T18:44:10.226 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:44:10.226 DEBUG:boto:Data: 2023-11-02T18:44:10.227 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.227 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.228 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.228 DEBUG:boto:Params: {} 2023-11-02T18:44:10.228 DEBUG:boto:Token: None 2023-11-02T18:44:10.229 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj55 2023-11-02T18:44:10.229 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iHHf3ffUrcfGeGeC9GvQUZ9N1bo= 2023-11-02T18:44:10.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iHHf3ffUrcfGeGeC9GvQUZ9N1bo=', 'Content-Length': '0'} 2023-11-02T18:44:10.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b5442e7649a27e0-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.274 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.275 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.275 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:44:10.276 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:44:10.276 DEBUG:boto:Method: GET 2023-11-02T18:44:10.277 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:44:10.277 DEBUG:boto:Data: 2023-11-02T18:44:10.278 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.278 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.278 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.279 DEBUG:boto:Params: {} 2023-11-02T18:44:10.279 DEBUG:boto:Token: None 2023-11-02T18:44:10.280 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.280 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d54cbbdfec6d74f-006543edfa-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.286 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.287 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.287 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.288 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:44:10.288 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:44:10.288 DEBUG:boto:Method: GET 2023-11-02T18:44:10.289 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:44:10.289 DEBUG:boto:Data: 2023-11-02T18:44:10.290 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.290 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.290 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.291 DEBUG:boto:Params: {} 2023-11-02T18:44:10.291 DEBUG:boto:Token: None 2023-11-02T18:44:10.292 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.292 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000412ccf1f44f20568-006543edfa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.302 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:44:10.303 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:44:10.304 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:44:10.304 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.305 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.305 DEBUG:boto:Method: GET 2023-11-02T18:44:10.305 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.306 DEBUG:boto:Data: 2023-11-02T18:44:10.306 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.307 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.307 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.308 DEBUG:boto:Params: {} 2023-11-02T18:44:10.308 DEBUG:boto:Token: None 2023-11-02T18:44:10.309 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.309 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3vgAVaDfrCuvnjTX43ZPZG7kxhA= 2023-11-02T18:44:10.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3vgAVaDfrCuvnjTX43ZPZG7kxhA=', 'Content-Length': '0'} 2023-11-02T18:44:10.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc23ce69f4fd5aef-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.358 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:44:10.359 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:44:10.359 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.360 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.360 DEBUG:boto:Method: GET 2023-11-02T18:44:10.360 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.361 DEBUG:boto:Data: 2023-11-02T18:44:10.361 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.362 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.362 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.362 DEBUG:boto:Params: {} 2023-11-02T18:44:10.363 DEBUG:boto:Token: None 2023-11-02T18:44:10.363 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.364 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3vgAVaDfrCuvnjTX43ZPZG7kxhA= 2023-11-02T18:44:10.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3vgAVaDfrCuvnjTX43ZPZG7kxhA=', 'Content-Length': '0'} 2023-11-02T18:44:10.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e2af73cd4f167e1-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.368 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:44:10.368 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:44:10.369 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.369 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.369 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.370 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.370 DEBUG:boto:Data: 2023-11-02T18:44:10.371 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.371 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.371 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.372 DEBUG:boto:Params: {} 2023-11-02T18:44:10.372 DEBUG:boto:Token: None 2023-11-02T18:44:10.373 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.373 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p5Ppa7lb0XlEEMcFD6m2MVVdPIQ= 2023-11-02T18:44:10.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p5Ppa7lb0XlEEMcFD6m2MVVdPIQ=', 'Content-Length': '0'} 2023-11-02T18:44:10.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b5fe298daad1f23-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.377 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:44:10.378 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:44:10.378 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.378 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.379 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.379 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.380 DEBUG:boto:Data: 2023-11-02T18:44:10.380 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.381 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.381 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.381 DEBUG:boto:Params: {} 2023-11-02T18:44:10.382 DEBUG:boto:Token: None 2023-11-02T18:44:10.382 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.383 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p5Ppa7lb0XlEEMcFD6m2MVVdPIQ= 2023-11-02T18:44:10.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p5Ppa7lb0XlEEMcFD6m2MVVdPIQ=', 'Content-Length': '0'} 2023-11-02T18:44:10.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bff491d9a37962b-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:44:10.389 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:44:10.390 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:44:10.390 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.391 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.391 DEBUG:boto:Method: GET 2023-11-02T18:44:10.392 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.392 DEBUG:boto:Data: 2023-11-02T18:44:10.393 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.393 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.393 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.394 DEBUG:boto:Params: {} 2023-11-02T18:44:10.394 DEBUG:boto:Token: None 2023-11-02T18:44:10.395 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.395 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3vgAVaDfrCuvnjTX43ZPZG7kxhA= 2023-11-02T18:44:10.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3vgAVaDfrCuvnjTX43ZPZG7kxhA=', 'Content-Length': '0'} 2023-11-02T18:44:10.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067168e315749df25-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.400 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:44:10.400 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:44:10.400 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.401 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.401 DEBUG:boto:Method: GET 2023-11-02T18:44:10.402 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.402 DEBUG:boto:Data: 2023-11-02T18:44:10.403 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.404 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.404 DEBUG:boto:Params: {} 2023-11-02T18:44:10.405 DEBUG:boto:Token: None 2023-11-02T18:44:10.405 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:44:10.406 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3vgAVaDfrCuvnjTX43ZPZG7kxhA= 2023-11-02T18:44:10.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3vgAVaDfrCuvnjTX43ZPZG7kxhA=', 'Content-Length': '0'} 2023-11-02T18:44:10.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d314a49d14375562-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.410 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.410 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.411 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:44:10.411 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:44:10.412 DEBUG:boto:Method: GET 2023-11-02T18:44:10.412 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:44:10.412 DEBUG:boto:Data: 2023-11-02T18:44:10.413 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.413 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.414 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.414 DEBUG:boto:Params: {} 2023-11-02T18:44:10.415 DEBUG:boto:Token: None 2023-11-02T18:44:10.415 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.416 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c386216ca7e387e8-006543edfa-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.421 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.422 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.422 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.423 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:44:10.423 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:44:10.424 DEBUG:boto:Method: GET 2023-11-02T18:44:10.424 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:44:10.425 DEBUG:boto:Data: 2023-11-02T18:44:10.425 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.426 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.426 DEBUG:boto:Params: {} 2023-11-02T18:44:10.427 DEBUG:boto:Token: None 2023-11-02T18:44:10.427 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000916a1e535633df56-006543edfa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.442 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.443 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:44:10.443 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:44:10.443 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.444 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:44:10.444 DEBUG:boto:Data: 2023-11-02T18:44:10.445 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.445 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.446 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.446 DEBUG:boto:Params: {} 2023-11-02T18:44:10.446 DEBUG:boto:Token: None 2023-11-02T18:44:10.447 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj57 2023-11-02T18:44:10.447 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KSNJ+Y48aqjvBoam8UeGiZdqPyM= 2023-11-02T18:44:10.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KSNJ+Y48aqjvBoam8UeGiZdqPyM=', 'Content-Length': '0'} 2023-11-02T18:44:10.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004805dbc451282320-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.451 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:44:10.451 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:44:10.452 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.452 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:44:10.453 DEBUG:boto:Data: 2023-11-02T18:44:10.453 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.454 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.455 DEBUG:boto:Params: {} 2023-11-02T18:44:10.455 DEBUG:boto:Token: None 2023-11-02T18:44:10.456 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj57 2023-11-02T18:44:10.456 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KSNJ+Y48aqjvBoam8UeGiZdqPyM= 2023-11-02T18:44:10.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KSNJ+Y48aqjvBoam8UeGiZdqPyM=', 'Content-Length': '0'} 2023-11-02T18:44:10.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027df32b0bd75a4b6-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:44:10.460 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:44:10.461 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:44:10.461 DEBUG:boto:Method: GET 2023-11-02T18:44:10.462 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:44:10.462 DEBUG:boto:Data: 2023-11-02T18:44:10.463 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.463 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.464 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.464 DEBUG:boto:Params: {} 2023-11-02T18:44:10.465 DEBUG:boto:Token: None 2023-11-02T18:44:10.465 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj57 2023-11-02T18:44:10.466 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3EkTEPVWS/PEnlrSSZYxTvzx9a4= 2023-11-02T18:44:10.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3EkTEPVWS/PEnlrSSZYxTvzx9a4=', 'Content-Length': '0'} 2023-11-02T18:44:10.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056dfe2ca2dc68dbe-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.515 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:44:10.604 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:44:10.605 DEBUG:boto:Method: GET 2023-11-02T18:44:10.605 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:44:10.606 DEBUG:boto:Data: 2023-11-02T18:44:10.606 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.607 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.607 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.608 DEBUG:boto:Params: {} 2023-11-02T18:44:10.608 DEBUG:boto:Token: None 2023-11-02T18:44:10.609 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj57 2023-11-02T18:44:10.609 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3EkTEPVWS/PEnlrSSZYxTvzx9a4= 2023-11-02T18:44:10.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3EkTEPVWS/PEnlrSSZYxTvzx9a4=', 'Content-Length': '0'} 2023-11-02T18:44:10.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e64decfa72ce6e47-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.614 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.614 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.615 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:44:10.615 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:44:10.616 DEBUG:boto:Method: GET 2023-11-02T18:44:10.616 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:44:10.616 DEBUG:boto:Data: 2023-11-02T18:44:10.617 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.617 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.618 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.618 DEBUG:boto:Params: {} 2023-11-02T18:44:10.619 DEBUG:boto:Token: None 2023-11-02T18:44:10.619 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.620 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f631b8b21be6481e-006543edfa-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.626 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.627 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.627 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.628 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:44:10.628 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:44:10.629 DEBUG:boto:Method: GET 2023-11-02T18:44:10.629 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:44:10.630 DEBUG:boto:Data: 2023-11-02T18:44:10.630 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.631 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.631 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.632 DEBUG:boto:Params: {} 2023-11-02T18:44:10.632 DEBUG:boto:Token: None 2023-11-02T18:44:10.633 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.634 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061861eb9d27f7a65-006543edfa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.643 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:44:10.644 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:44:10.645 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:44:10.645 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.646 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.646 DEBUG:boto:Method: GET 2023-11-02T18:44:10.647 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.647 DEBUG:boto:Data: 2023-11-02T18:44:10.648 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.648 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.649 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.649 DEBUG:boto:Params: {} 2023-11-02T18:44:10.650 DEBUG:boto:Token: None 2023-11-02T18:44:10.651 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.651 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:15DRBk5gdDeyMwTBejM25i3LuUc= 2023-11-02T18:44:10.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:15DRBk5gdDeyMwTBejM25i3LuUc=', 'Content-Length': '0'} 2023-11-02T18:44:10.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009879cd3167380cc0-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.655 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:44:10.656 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:44:10.656 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.657 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.657 DEBUG:boto:Method: GET 2023-11-02T18:44:10.658 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.658 DEBUG:boto:Data: 2023-11-02T18:44:10.659 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.659 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.660 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.660 DEBUG:boto:Params: {} 2023-11-02T18:44:10.661 DEBUG:boto:Token: None 2023-11-02T18:44:10.661 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.662 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:15DRBk5gdDeyMwTBejM25i3LuUc= 2023-11-02T18:44:10.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:15DRBk5gdDeyMwTBejM25i3LuUc=', 'Content-Length': '0'} 2023-11-02T18:44:10.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f68c9cbdb8de57f4-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.666 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:44:10.667 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:44:10.667 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.668 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.668 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.669 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.669 DEBUG:boto:Data: 2023-11-02T18:44:10.670 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.670 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.671 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.671 DEBUG:boto:Params: {} 2023-11-02T18:44:10.672 DEBUG:boto:Token: None 2023-11-02T18:44:10.672 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.673 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PZ44N/0pJLWfNSkGgkxabwHxh8o= 2023-11-02T18:44:10.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PZ44N/0pJLWfNSkGgkxabwHxh8o=', 'Content-Length': '0'} 2023-11-02T18:44:10.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000748273ed6f09cae7-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.677 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:44:10.677 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:44:10.678 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.678 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.679 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.679 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.679 DEBUG:boto:Data: 2023-11-02T18:44:10.680 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.681 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.681 DEBUG:boto:Params: {} 2023-11-02T18:44:10.682 DEBUG:boto:Token: None 2023-11-02T18:44:10.682 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.683 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PZ44N/0pJLWfNSkGgkxabwHxh8o= 2023-11-02T18:44:10.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PZ44N/0pJLWfNSkGgkxabwHxh8o=', 'Content-Length': '0'} 2023-11-02T18:44:10.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3967fa90cd7baf6-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:44:10.687 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:44:10.688 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:44:10.688 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.689 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.689 DEBUG:boto:Method: GET 2023-11-02T18:44:10.690 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.690 DEBUG:boto:Data: 2023-11-02T18:44:10.691 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.691 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.692 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.692 DEBUG:boto:Params: {} 2023-11-02T18:44:10.693 DEBUG:boto:Token: None 2023-11-02T18:44:10.693 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.694 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:15DRBk5gdDeyMwTBejM25i3LuUc= 2023-11-02T18:44:10.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:15DRBk5gdDeyMwTBejM25i3LuUc=', 'Content-Length': '0'} 2023-11-02T18:44:10.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e409bd7cea831e-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.738 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:44:10.739 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:44:10.739 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.740 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.740 DEBUG:boto:Method: GET 2023-11-02T18:44:10.741 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.741 DEBUG:boto:Data: 2023-11-02T18:44:10.742 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.742 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.743 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.743 DEBUG:boto:Params: {} 2023-11-02T18:44:10.744 DEBUG:boto:Token: None 2023-11-02T18:44:10.744 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:44:10.745 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:15DRBk5gdDeyMwTBejM25i3LuUc= 2023-11-02T18:44:10.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:15DRBk5gdDeyMwTBejM25i3LuUc=', 'Content-Length': '0'} 2023-11-02T18:44:10.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bfab59741651095-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.750 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.750 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.751 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:44:10.751 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:44:10.752 DEBUG:boto:Method: GET 2023-11-02T18:44:10.752 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:44:10.752 DEBUG:boto:Data: 2023-11-02T18:44:10.753 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.753 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.753 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.754 DEBUG:boto:Params: {} 2023-11-02T18:44:10.754 DEBUG:boto:Token: None 2023-11-02T18:44:10.755 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.755 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e5472943b0ce640-006543edfa-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.761 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.762 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.762 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.762 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:44:10.763 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:44:10.763 DEBUG:boto:Method: GET 2023-11-02T18:44:10.764 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:44:10.764 DEBUG:boto:Data: 2023-11-02T18:44:10.764 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.765 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.765 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.766 DEBUG:boto:Params: {} 2023-11-02T18:44:10.766 DEBUG:boto:Token: None 2023-11-02T18:44:10.767 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.767 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b169303f9c1b3405-006543edfa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.777 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.778 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:44:10.778 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:44:10.778 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.779 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:44:10.779 DEBUG:boto:Data: 2023-11-02T18:44:10.780 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.780 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.781 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.781 DEBUG:boto:Params: {} 2023-11-02T18:44:10.781 DEBUG:boto:Token: None 2023-11-02T18:44:10.782 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj58 2023-11-02T18:44:10.782 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2xaCKrGEBoIEXMuGi9+MlU7/Blw= 2023-11-02T18:44:10.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2xaCKrGEBoIEXMuGi9+MlU7/Blw=', 'Content-Length': '0'} 2023-11-02T18:44:10.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f5103c1c25926ea-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.786 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:44:10.787 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:44:10.787 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.788 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:44:10.788 DEBUG:boto:Data: 2023-11-02T18:44:10.789 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.789 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.790 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.790 DEBUG:boto:Params: {} 2023-11-02T18:44:10.791 DEBUG:boto:Token: None 2023-11-02T18:44:10.791 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj58 2023-11-02T18:44:10.792 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2xaCKrGEBoIEXMuGi9+MlU7/Blw= 2023-11-02T18:44:10.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2xaCKrGEBoIEXMuGi9+MlU7/Blw=', 'Content-Length': '0'} 2023-11-02T18:44:10.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006524562a549a1620-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:44:10.796 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:44:10.796 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:44:10.797 DEBUG:boto:Method: GET 2023-11-02T18:44:10.797 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:44:10.797 DEBUG:boto:Data: 2023-11-02T18:44:10.798 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.798 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.799 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.799 DEBUG:boto:Params: {} 2023-11-02T18:44:10.800 DEBUG:boto:Token: None 2023-11-02T18:44:10.800 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj58 2023-11-02T18:44:10.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jAbKuQaYo+vktx5sMDcaj1gmJy4= 2023-11-02T18:44:10.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jAbKuQaYo+vktx5sMDcaj1gmJy4=', 'Content-Length': '0'} 2023-11-02T18:44:10.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f4bcab8b4c0c48-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.804 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:44:10.805 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:44:10.805 DEBUG:boto:Method: GET 2023-11-02T18:44:10.805 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:44:10.806 DEBUG:boto:Data: 2023-11-02T18:44:10.806 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.807 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.807 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.808 DEBUG:boto:Params: {} 2023-11-02T18:44:10.808 DEBUG:boto:Token: None 2023-11-02T18:44:10.809 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj58 2023-11-02T18:44:10.809 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jAbKuQaYo+vktx5sMDcaj1gmJy4= 2023-11-02T18:44:10.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jAbKuQaYo+vktx5sMDcaj1gmJy4=', 'Content-Length': '0'} 2023-11-02T18:44:10.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f1092ead5fb1738-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.858 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.859 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.860 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:44:10.860 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:44:10.860 DEBUG:boto:Method: GET 2023-11-02T18:44:10.861 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:44:10.861 DEBUG:boto:Data: 2023-11-02T18:44:10.862 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.862 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.863 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.863 DEBUG:boto:Params: {} 2023-11-02T18:44:10.864 DEBUG:boto:Token: None 2023-11-02T18:44:10.864 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.865 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e19b8637cca6ec17-006543edfa-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.871 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.872 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.872 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.873 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:44:10.873 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:44:10.874 DEBUG:boto:Method: GET 2023-11-02T18:44:10.874 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:44:10.875 DEBUG:boto:Data: 2023-11-02T18:44:10.875 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.876 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.876 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.876 DEBUG:boto:Params: {} 2023-11-02T18:44:10.877 DEBUG:boto:Token: None 2023-11-02T18:44:10.877 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/ 2023-11-02T18:44:10.878 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc= 2023-11-02T18:44:10.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ww+idFIIg2GaPnfjTVfQ6aQyXAc=', 'Content-Length': '0'} 2023-11-02T18:44:10.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f61d6415aeb9fef2-006543edfa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.887 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:10.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T18:44:10.888 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:44:10.889 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:44:10.889 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.890 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.890 DEBUG:boto:Method: GET 2023-11-02T18:44:10.891 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.891 DEBUG:boto:Data: 2023-11-02T18:44:10.892 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.892 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.893 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.893 DEBUG:boto:Params: {} 2023-11-02T18:44:10.894 DEBUG:boto:Token: None 2023-11-02T18:44:10.894 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.895 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I8KgHx3lgkvUI4Swg6YPPE3Ufz4= 2023-11-02T18:44:10.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I8KgHx3lgkvUI4Swg6YPPE3Ufz4=', 'Content-Length': '0'} 2023-11-02T18:44:10.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f0f85de8bfe4162-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.942 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:44:10.943 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:44:10.943 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.944 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.944 DEBUG:boto:Method: GET 2023-11-02T18:44:10.945 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.945 DEBUG:boto:Data: 2023-11-02T18:44:10.946 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.946 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.947 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.947 DEBUG:boto:Params: {} 2023-11-02T18:44:10.948 DEBUG:boto:Token: None 2023-11-02T18:44:10.948 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.949 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I8KgHx3lgkvUI4Swg6YPPE3Ufz4= 2023-11-02T18:44:10.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I8KgHx3lgkvUI4Swg6YPPE3Ufz4=', 'Content-Length': '0'} 2023-11-02T18:44:10.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005266866e486ec28f-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.954 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:44:10.955 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:44:10.955 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.956 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.956 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.956 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.957 DEBUG:boto:Data: 2023-11-02T18:44:10.957 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.958 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.958 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.958 DEBUG:boto:Params: {} 2023-11-02T18:44:10.959 DEBUG:boto:Token: None 2023-11-02T18:44:10.959 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.960 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7MBsEILMHDy6TVcSnqE5kYebzuc= 2023-11-02T18:44:10.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7MBsEILMHDy6TVcSnqE5kYebzuc=', 'Content-Length': '0'} 2023-11-02T18:44:10.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1d04941bbb55ac8-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.964 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:44:10.964 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:44:10.965 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.965 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.965 DEBUG:boto:Method: HEAD 2023-11-02T18:44:10.966 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.966 DEBUG:boto:Data: 2023-11-02T18:44:10.967 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.967 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.968 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.968 DEBUG:boto:Params: {} 2023-11-02T18:44:10.968 DEBUG:boto:Token: None 2023-11-02T18:44:10.969 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7MBsEILMHDy6TVcSnqE5kYebzuc= 2023-11-02T18:44:10.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7MBsEILMHDy6TVcSnqE5kYebzuc=', 'Content-Length': '0'} 2023-11-02T18:44:10.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036a4a35f8b31d373-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T18:44:10.974 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:44:10.974 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:44:10.975 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.975 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.976 DEBUG:boto:Method: GET 2023-11-02T18:44:10.976 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.976 DEBUG:boto:Data: 2023-11-02T18:44:10.977 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.977 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.978 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.978 DEBUG:boto:Params: {} 2023-11-02T18:44:10.979 DEBUG:boto:Token: None 2023-11-02T18:44:10.979 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.980 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I8KgHx3lgkvUI4Swg6YPPE3Ufz4= 2023-11-02T18:44:10.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I8KgHx3lgkvUI4Swg6YPPE3Ufz4=', 'Content-Length': '0'} 2023-11-02T18:44:10.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b9bb1e4270abe1d-006543edfa-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.984 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:44:10.984 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:44:10.985 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.985 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.986 DEBUG:boto:Method: GET 2023-11-02T18:44:10.986 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.987 DEBUG:boto:Data: 2023-11-02T18:44:10.987 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.987 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:10.988 DEBUG:boto:Port: 8000 2023-11-02T18:44:10.988 DEBUG:boto:Params: {} 2023-11-02T18:44:10.989 DEBUG:boto:Token: None 2023-11-02T18:44:10.989 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:10 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:44:10.990 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I8KgHx3lgkvUI4Swg6YPPE3Ufz4= 2023-11-02T18:44:10.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:10 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I8KgHx3lgkvUI4Swg6YPPE3Ufz4=', 'Content-Length': '0'} 2023-11-02T18:44:10.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed6fbcea4ffc60f8-006543edfa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:10.994 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:10.995 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:10.995 DEBUG:boto:path=/ibtljn-59/?prefix=obj62 2023-11-02T18:44:10.995 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj62 2023-11-02T18:44:10.996 DEBUG:boto:Method: GET 2023-11-02T18:44:10.996 DEBUG:boto:Path: /ibtljn-59/?prefix=obj62 2023-11-02T18:44:10.997 DEBUG:boto:Data: 2023-11-02T18:44:10.997 DEBUG:boto:Headers: {} 2023-11-02T18:44:10.998 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:10.998 DEBUG:boto:Port: 8001 2023-11-02T18:44:10.999 DEBUG:boto:Params: {} 2023-11-02T18:44:10.999 DEBUG:boto:Token: None 2023-11-02T18:44:11.000 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.000 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfe0541381e1ffe9-006543edfa-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:10 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.046 DEBUG:boto:b'ibtljn-59obj621000falseobj622023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.047 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.048 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.048 DEBUG:boto:path=/ibtljn-59/?prefix=obj62 2023-11-02T18:44:11.049 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj62 2023-11-02T18:44:11.049 DEBUG:boto:Method: GET 2023-11-02T18:44:11.049 DEBUG:boto:Path: /ibtljn-59/?prefix=obj62 2023-11-02T18:44:11.050 DEBUG:boto:Data: 2023-11-02T18:44:11.050 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.051 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.051 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.052 DEBUG:boto:Params: {} 2023-11-02T18:44:11.052 DEBUG:boto:Token: None 2023-11-02T18:44:11.053 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.053 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6c284fcbe4a5eba-006543edfb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.063 DEBUG:boto:b'ibtljn-59obj621000falseobj622023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.063 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:44:11.064 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:44:11.064 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.065 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:44:11.065 DEBUG:boto:Data: 2023-11-02T18:44:11.066 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.066 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.066 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.067 DEBUG:boto:Params: {} 2023-11-02T18:44:11.067 DEBUG:boto:Token: None 2023-11-02T18:44:11.068 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj62 2023-11-02T18:44:11.068 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m7ip/yOdR5yEEnaWIbO8YPUFFF4= 2023-11-02T18:44:11.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m7ip/yOdR5yEEnaWIbO8YPUFFF4=', 'Content-Length': '0'} 2023-11-02T18:44:11.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004819d98c11ccd705-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.072 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:44:11.072 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:44:11.073 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.073 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:44:11.073 DEBUG:boto:Data: 2023-11-02T18:44:11.074 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.074 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.074 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.075 DEBUG:boto:Params: {} 2023-11-02T18:44:11.075 DEBUG:boto:Token: None 2023-11-02T18:44:11.076 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj62 2023-11-02T18:44:11.076 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m7ip/yOdR5yEEnaWIbO8YPUFFF4= 2023-11-02T18:44:11.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m7ip/yOdR5yEEnaWIbO8YPUFFF4=', 'Content-Length': '0'} 2023-11-02T18:44:11.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d0f24c686338a84-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T18:44:11.080 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:44:11.080 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:44:11.081 DEBUG:boto:Method: GET 2023-11-02T18:44:11.081 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:44:11.082 DEBUG:boto:Data: 2023-11-02T18:44:11.082 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.082 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.083 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.083 DEBUG:boto:Params: {} 2023-11-02T18:44:11.084 DEBUG:boto:Token: None 2023-11-02T18:44:11.084 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj62 2023-11-02T18:44:11.084 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:toudB9gT/29kjC7gA7Uu6JyfLbY= 2023-11-02T18:44:11.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:toudB9gT/29kjC7gA7Uu6JyfLbY=', 'Content-Length': '0'} 2023-11-02T18:44:11.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d0be6d05c8fc5fd-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.089 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:44:11.089 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:44:11.090 DEBUG:boto:Method: GET 2023-11-02T18:44:11.090 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:44:11.091 DEBUG:boto:Data: 2023-11-02T18:44:11.091 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.092 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.092 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.092 DEBUG:boto:Params: {} 2023-11-02T18:44:11.093 DEBUG:boto:Token: None 2023-11-02T18:44:11.093 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj62 2023-11-02T18:44:11.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:toudB9gT/29kjC7gA7Uu6JyfLbY= 2023-11-02T18:44:11.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:toudB9gT/29kjC7gA7Uu6JyfLbY=', 'Content-Length': '0'} 2023-11-02T18:44:11.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e56664e1b54f45-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.138 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.139 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.139 DEBUG:boto:path=/ibtljn-59/?prefix=obj62 2023-11-02T18:44:11.140 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj62 2023-11-02T18:44:11.140 DEBUG:boto:Method: GET 2023-11-02T18:44:11.141 DEBUG:boto:Path: /ibtljn-59/?prefix=obj62 2023-11-02T18:44:11.141 DEBUG:boto:Data: 2023-11-02T18:44:11.141 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.142 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.142 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.143 DEBUG:boto:Params: {} 2023-11-02T18:44:11.143 DEBUG:boto:Token: None 2023-11-02T18:44:11.144 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.144 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036f206f2bb8516e6-006543edfb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.150 DEBUG:boto:b'ibtljn-59obj621000falseobj622023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.151 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.151 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.151 DEBUG:boto:path=/ibtljn-59/?prefix=obj62 2023-11-02T18:44:11.152 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj62 2023-11-02T18:44:11.152 DEBUG:boto:Method: GET 2023-11-02T18:44:11.153 DEBUG:boto:Path: /ibtljn-59/?prefix=obj62 2023-11-02T18:44:11.153 DEBUG:boto:Data: 2023-11-02T18:44:11.153 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.154 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.154 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.155 DEBUG:boto:Params: {} 2023-11-02T18:44:11.155 DEBUG:boto:Token: None 2023-11-02T18:44:11.155 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.156 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afddc5c5d2523ed1-006543edfb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.165 DEBUG:boto:b'ibtljn-59obj621000falseobj622023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:44:11.166 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:44:11.166 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:44:11.167 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.167 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.168 DEBUG:boto:Method: GET 2023-11-02T18:44:11.168 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.169 DEBUG:boto:Data: 2023-11-02T18:44:11.169 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.169 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.170 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.170 DEBUG:boto:Params: {} 2023-11-02T18:44:11.171 DEBUG:boto:Token: None 2023-11-02T18:44:11.171 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.172 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QynPb8Kp8LmIFvs2ikpXv3GkJsM= 2023-11-02T18:44:11.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QynPb8Kp8LmIFvs2ikpXv3GkJsM=', 'Content-Length': '0'} 2023-11-02T18:44:11.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fa9c06f048b10dc-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.218 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:44:11.219 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:44:11.219 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.220 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.220 DEBUG:boto:Method: GET 2023-11-02T18:44:11.221 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.221 DEBUG:boto:Data: 2023-11-02T18:44:11.222 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.222 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.223 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.223 DEBUG:boto:Params: {} 2023-11-02T18:44:11.224 DEBUG:boto:Token: None 2023-11-02T18:44:11.224 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QynPb8Kp8LmIFvs2ikpXv3GkJsM= 2023-11-02T18:44:11.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QynPb8Kp8LmIFvs2ikpXv3GkJsM=', 'Content-Length': '0'} 2023-11-02T18:44:11.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cac9afbe533e676a-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.229 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:44:11.229 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:44:11.230 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.230 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.231 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.231 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.231 DEBUG:boto:Data: 2023-11-02T18:44:11.232 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.232 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.233 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.233 DEBUG:boto:Params: {} 2023-11-02T18:44:11.234 DEBUG:boto:Token: None 2023-11-02T18:44:11.234 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.235 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c8wyMXDm7GVBP5MC/rbzZq7Dg2U= 2023-11-02T18:44:11.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c8wyMXDm7GVBP5MC/rbzZq7Dg2U=', 'Content-Length': '0'} 2023-11-02T18:44:11.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000971739f38f60c168-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.239 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:44:11.239 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:44:11.240 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.240 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.240 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.241 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.241 DEBUG:boto:Data: 2023-11-02T18:44:11.242 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.242 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.242 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.243 DEBUG:boto:Params: {} 2023-11-02T18:44:11.243 DEBUG:boto:Token: None 2023-11-02T18:44:11.244 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.244 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c8wyMXDm7GVBP5MC/rbzZq7Dg2U= 2023-11-02T18:44:11.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c8wyMXDm7GVBP5MC/rbzZq7Dg2U=', 'Content-Length': '0'} 2023-11-02T18:44:11.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c75617d0ad1e671-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:44:11.248 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:44:11.249 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:44:11.249 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.250 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.250 DEBUG:boto:Method: GET 2023-11-02T18:44:11.251 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.251 DEBUG:boto:Data: 2023-11-02T18:44:11.252 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.252 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.252 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.253 DEBUG:boto:Params: {} 2023-11-02T18:44:11.253 DEBUG:boto:Token: None 2023-11-02T18:44:11.254 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.254 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QynPb8Kp8LmIFvs2ikpXv3GkJsM= 2023-11-02T18:44:11.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QynPb8Kp8LmIFvs2ikpXv3GkJsM=', 'Content-Length': '0'} 2023-11-02T18:44:11.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1e3dd7646350f89-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.259 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:44:11.259 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:44:11.259 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.260 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.260 DEBUG:boto:Method: GET 2023-11-02T18:44:11.261 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.261 DEBUG:boto:Data: 2023-11-02T18:44:11.262 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.262 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.262 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.263 DEBUG:boto:Params: {} 2023-11-02T18:44:11.263 DEBUG:boto:Token: None 2023-11-02T18:44:11.264 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:44:11.264 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QynPb8Kp8LmIFvs2ikpXv3GkJsM= 2023-11-02T18:44:11.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QynPb8Kp8LmIFvs2ikpXv3GkJsM=', 'Content-Length': '0'} 2023-11-02T18:44:11.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec429ba860f84bc-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.310 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.311 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.311 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:44:11.312 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:44:11.312 DEBUG:boto:Method: GET 2023-11-02T18:44:11.313 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:44:11.313 DEBUG:boto:Data: 2023-11-02T18:44:11.314 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.314 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.315 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.315 DEBUG:boto:Params: {} 2023-11-02T18:44:11.316 DEBUG:boto:Token: None 2023-11-02T18:44:11.316 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.317 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c18f7ff7e1f9fb12-006543edfb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.323 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.324 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.324 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.325 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:44:11.325 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:44:11.325 DEBUG:boto:Method: GET 2023-11-02T18:44:11.326 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:44:11.326 DEBUG:boto:Data: 2023-11-02T18:44:11.327 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.327 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.328 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.328 DEBUG:boto:Params: {} 2023-11-02T18:44:11.329 DEBUG:boto:Token: None 2023-11-02T18:44:11.329 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.330 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b5bdba4e9f34a68-006543edfb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.339 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.340 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:44:11.340 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:44:11.341 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.341 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:44:11.341 DEBUG:boto:Data: 2023-11-02T18:44:11.342 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.342 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.343 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.343 DEBUG:boto:Params: {} 2023-11-02T18:44:11.344 DEBUG:boto:Token: None 2023-11-02T18:44:11.344 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj68 2023-11-02T18:44:11.344 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d7MpGGjR26wXMb/ExeaNSQUjK/Q= 2023-11-02T18:44:11.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d7MpGGjR26wXMb/ExeaNSQUjK/Q=', 'Content-Length': '0'} 2023-11-02T18:44:11.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008afc774e5e0a024c-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.348 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:44:11.348 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:44:11.349 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.349 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:44:11.350 DEBUG:boto:Data: 2023-11-02T18:44:11.350 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.351 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.351 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.351 DEBUG:boto:Params: {} 2023-11-02T18:44:11.352 DEBUG:boto:Token: None 2023-11-02T18:44:11.352 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj68 2023-11-02T18:44:11.353 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d7MpGGjR26wXMb/ExeaNSQUjK/Q= 2023-11-02T18:44:11.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d7MpGGjR26wXMb/ExeaNSQUjK/Q=', 'Content-Length': '0'} 2023-11-02T18:44:11.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd5764de73e3d338-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:44:11.356 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:44:11.357 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:44:11.357 DEBUG:boto:Method: GET 2023-11-02T18:44:11.358 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:44:11.358 DEBUG:boto:Data: 2023-11-02T18:44:11.359 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.359 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.359 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.360 DEBUG:boto:Params: {} 2023-11-02T18:44:11.360 DEBUG:boto:Token: None 2023-11-02T18:44:11.361 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj68 2023-11-02T18:44:11.361 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KMg5uQOaDmFDKC/UqVQI2QcMqlM= 2023-11-02T18:44:11.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KMg5uQOaDmFDKC/UqVQI2QcMqlM=', 'Content-Length': '0'} 2023-11-02T18:44:11.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066925fe58d63da4b-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.406 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:44:11.407 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:44:11.407 DEBUG:boto:Method: GET 2023-11-02T18:44:11.408 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:44:11.408 DEBUG:boto:Data: 2023-11-02T18:44:11.409 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.409 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.410 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.410 DEBUG:boto:Params: {} 2023-11-02T18:44:11.411 DEBUG:boto:Token: None 2023-11-02T18:44:11.411 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj68 2023-11-02T18:44:11.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KMg5uQOaDmFDKC/UqVQI2QcMqlM= 2023-11-02T18:44:11.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KMg5uQOaDmFDKC/UqVQI2QcMqlM=', 'Content-Length': '0'} 2023-11-02T18:44:11.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c876fb735513e65a-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.416 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.417 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.417 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:44:11.417 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:44:11.418 DEBUG:boto:Method: GET 2023-11-02T18:44:11.418 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:44:11.419 DEBUG:boto:Data: 2023-11-02T18:44:11.419 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.420 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.420 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.421 DEBUG:boto:Params: {} 2023-11-02T18:44:11.421 DEBUG:boto:Token: None 2023-11-02T18:44:11.421 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.422 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd37f3a353dceda2-006543edfb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.428 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.428 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.429 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.429 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:44:11.430 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:44:11.430 DEBUG:boto:Method: GET 2023-11-02T18:44:11.430 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:44:11.431 DEBUG:boto:Data: 2023-11-02T18:44:11.431 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.432 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.432 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.433 DEBUG:boto:Params: {} 2023-11-02T18:44:11.433 DEBUG:boto:Token: None 2023-11-02T18:44:11.434 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.434 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dce3b76660b9672-006543edfb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.490 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:44:11.492 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:44:11.493 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:44:11.494 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.494 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.495 DEBUG:boto:Method: GET 2023-11-02T18:44:11.496 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.497 DEBUG:boto:Data: 2023-11-02T18:44:11.497 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.498 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.499 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.499 DEBUG:boto:Params: {} 2023-11-02T18:44:11.500 DEBUG:boto:Token: None 2023-11-02T18:44:11.501 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.502 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9a1xGIv7Cp86pB9CUvQ8OChgiNY= 2023-11-02T18:44:11.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9a1xGIv7Cp86pB9CUvQ8OChgiNY=', 'Content-Length': '0'} 2023-11-02T18:44:11.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000312d1d6e4d1ac32e-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.507 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:44:11.507 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:44:11.508 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.508 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.509 DEBUG:boto:Method: GET 2023-11-02T18:44:11.510 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.510 DEBUG:boto:Data: 2023-11-02T18:44:11.511 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.511 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.512 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.512 DEBUG:boto:Params: {} 2023-11-02T18:44:11.513 DEBUG:boto:Token: None 2023-11-02T18:44:11.513 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.514 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9a1xGIv7Cp86pB9CUvQ8OChgiNY= 2023-11-02T18:44:11.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9a1xGIv7Cp86pB9CUvQ8OChgiNY=', 'Content-Length': '0'} 2023-11-02T18:44:11.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c199b96ba06a3b7-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.518 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:44:11.518 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:44:11.519 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.519 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.520 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.520 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.520 DEBUG:boto:Data: 2023-11-02T18:44:11.521 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.521 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.522 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.522 DEBUG:boto:Params: {} 2023-11-02T18:44:11.523 DEBUG:boto:Token: None 2023-11-02T18:44:11.523 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MAkJB2YvxxHDg2VIas+wWTWUEwc= 2023-11-02T18:44:11.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MAkJB2YvxxHDg2VIas+wWTWUEwc=', 'Content-Length': '0'} 2023-11-02T18:44:11.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001707ecdeb680eebc-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.527 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:44:11.528 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:44:11.528 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.529 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.529 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.530 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.530 DEBUG:boto:Data: 2023-11-02T18:44:11.530 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.531 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.531 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.532 DEBUG:boto:Params: {} 2023-11-02T18:44:11.532 DEBUG:boto:Token: None 2023-11-02T18:44:11.533 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.533 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MAkJB2YvxxHDg2VIas+wWTWUEwc= 2023-11-02T18:44:11.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MAkJB2YvxxHDg2VIas+wWTWUEwc=', 'Content-Length': '0'} 2023-11-02T18:44:11.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005649d64139fcd28a-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:44:11.537 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:44:11.538 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:44:11.538 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.539 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.539 DEBUG:boto:Method: GET 2023-11-02T18:44:11.539 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.540 DEBUG:boto:Data: 2023-11-02T18:44:11.540 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.541 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.542 DEBUG:boto:Params: {} 2023-11-02T18:44:11.542 DEBUG:boto:Token: None 2023-11-02T18:44:11.543 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.543 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9a1xGIv7Cp86pB9CUvQ8OChgiNY= 2023-11-02T18:44:11.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9a1xGIv7Cp86pB9CUvQ8OChgiNY=', 'Content-Length': '0'} 2023-11-02T18:44:11.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d135b8f0f74ae85-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.547 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:44:11.548 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:44:11.548 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.549 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.549 DEBUG:boto:Method: GET 2023-11-02T18:44:11.550 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.550 DEBUG:boto:Data: 2023-11-02T18:44:11.550 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.551 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.551 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.552 DEBUG:boto:Params: {} 2023-11-02T18:44:11.552 DEBUG:boto:Token: None 2023-11-02T18:44:11.553 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:44:11.553 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9a1xGIv7Cp86pB9CUvQ8OChgiNY= 2023-11-02T18:44:11.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9a1xGIv7Cp86pB9CUvQ8OChgiNY=', 'Content-Length': '0'} 2023-11-02T18:44:11.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a64b421c952dea-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.557 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.558 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.558 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:44:11.559 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:44:11.559 DEBUG:boto:Method: GET 2023-11-02T18:44:11.560 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:44:11.560 DEBUG:boto:Data: 2023-11-02T18:44:11.560 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.561 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.561 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.562 DEBUG:boto:Params: {} 2023-11-02T18:44:11.562 DEBUG:boto:Token: None 2023-11-02T18:44:11.563 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.563 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001047f7a688881bfa-006543edfb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.610 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.611 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.612 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.612 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:44:11.612 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:44:11.613 DEBUG:boto:Method: GET 2023-11-02T18:44:11.613 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:44:11.614 DEBUG:boto:Data: 2023-11-02T18:44:11.614 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.615 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.615 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.616 DEBUG:boto:Params: {} 2023-11-02T18:44:11.616 DEBUG:boto:Token: None 2023-11-02T18:44:11.617 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.617 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cf44b98c14d3284-006543edfb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.630 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.630 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:44:11.631 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:44:11.631 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.632 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:44:11.632 DEBUG:boto:Data: 2023-11-02T18:44:11.633 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.633 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.633 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.634 DEBUG:boto:Params: {} 2023-11-02T18:44:11.634 DEBUG:boto:Token: None 2023-11-02T18:44:11.635 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj69 2023-11-02T18:44:11.635 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8WGuaPuqfv/7991VYkh/cBMET5o= 2023-11-02T18:44:11.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8WGuaPuqfv/7991VYkh/cBMET5o=', 'Content-Length': '0'} 2023-11-02T18:44:11.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a745e6bd1f4a2266-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.638 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:44:11.639 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:44:11.639 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.640 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:44:11.640 DEBUG:boto:Data: 2023-11-02T18:44:11.640 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.641 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.641 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.642 DEBUG:boto:Params: {} 2023-11-02T18:44:11.642 DEBUG:boto:Token: None 2023-11-02T18:44:11.643 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj69 2023-11-02T18:44:11.643 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8WGuaPuqfv/7991VYkh/cBMET5o= 2023-11-02T18:44:11.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8WGuaPuqfv/7991VYkh/cBMET5o=', 'Content-Length': '0'} 2023-11-02T18:44:11.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084f8dce648fbe06b-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:44:11.648 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:44:11.648 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:44:11.648 DEBUG:boto:Method: GET 2023-11-02T18:44:11.649 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:44:11.649 DEBUG:boto:Data: 2023-11-02T18:44:11.650 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.650 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.651 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.651 DEBUG:boto:Params: {} 2023-11-02T18:44:11.651 DEBUG:boto:Token: None 2023-11-02T18:44:11.652 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj69 2023-11-02T18:44:11.652 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iw6ZbrRDiZV9dVdxSrivfOBsVH4= 2023-11-02T18:44:11.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iw6ZbrRDiZV9dVdxSrivfOBsVH4=', 'Content-Length': '0'} 2023-11-02T18:44:11.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fce99e8ca7d1767-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.656 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:44:11.656 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:44:11.657 DEBUG:boto:Method: GET 2023-11-02T18:44:11.657 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:44:11.658 DEBUG:boto:Data: 2023-11-02T18:44:11.658 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.659 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.659 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.659 DEBUG:boto:Params: {} 2023-11-02T18:44:11.660 DEBUG:boto:Token: None 2023-11-02T18:44:11.660 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj69 2023-11-02T18:44:11.661 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iw6ZbrRDiZV9dVdxSrivfOBsVH4= 2023-11-02T18:44:11.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iw6ZbrRDiZV9dVdxSrivfOBsVH4=', 'Content-Length': '0'} 2023-11-02T18:44:11.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d462a0bddccab25-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.706 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.707 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.707 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:44:11.708 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:44:11.708 DEBUG:boto:Method: GET 2023-11-02T18:44:11.709 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:44:11.709 DEBUG:boto:Data: 2023-11-02T18:44:11.709 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.710 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.710 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.711 DEBUG:boto:Params: {} 2023-11-02T18:44:11.711 DEBUG:boto:Token: None 2023-11-02T18:44:11.712 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4207df031da4591-006543edfb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.718 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.719 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.720 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.720 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:44:11.720 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:44:11.721 DEBUG:boto:Method: GET 2023-11-02T18:44:11.721 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:44:11.722 DEBUG:boto:Data: 2023-11-02T18:44:11.722 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.723 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.723 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.724 DEBUG:boto:Params: {} 2023-11-02T18:44:11.724 DEBUG:boto:Token: None 2023-11-02T18:44:11.725 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.725 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bef530d8aae1faca-006543edfb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.735 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:44:11.736 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:44:11.736 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:44:11.737 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.737 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.738 DEBUG:boto:Method: GET 2023-11-02T18:44:11.738 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.739 DEBUG:boto:Data: 2023-11-02T18:44:11.739 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.739 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.740 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.740 DEBUG:boto:Params: {} 2023-11-02T18:44:11.741 DEBUG:boto:Token: None 2023-11-02T18:44:11.741 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.742 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+HUqVZaf1hRsVlEt8WtJH7zWzWs= 2023-11-02T18:44:11.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+HUqVZaf1hRsVlEt8WtJH7zWzWs=', 'Content-Length': '0'} 2023-11-02T18:44:11.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2e0c48b798f9b9e-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.745 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:44:11.746 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:44:11.746 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.747 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.747 DEBUG:boto:Method: GET 2023-11-02T18:44:11.748 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.748 DEBUG:boto:Data: 2023-11-02T18:44:11.748 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.749 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.749 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.750 DEBUG:boto:Params: {} 2023-11-02T18:44:11.750 DEBUG:boto:Token: None 2023-11-02T18:44:11.751 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.751 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+HUqVZaf1hRsVlEt8WtJH7zWzWs= 2023-11-02T18:44:11.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+HUqVZaf1hRsVlEt8WtJH7zWzWs=', 'Content-Length': '0'} 2023-11-02T18:44:11.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6441f744bcd2172-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.755 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:44:11.755 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:44:11.756 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.756 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.756 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.757 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.757 DEBUG:boto:Data: 2023-11-02T18:44:11.758 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.758 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.758 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.759 DEBUG:boto:Params: {} 2023-11-02T18:44:11.759 DEBUG:boto:Token: None 2023-11-02T18:44:11.760 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.760 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9W4qnW9qKdRxjWYzI6JV47t4RI8= 2023-11-02T18:44:11.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9W4qnW9qKdRxjWYzI6JV47t4RI8=', 'Content-Length': '0'} 2023-11-02T18:44:11.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d174658d9ba02abc-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.764 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:44:11.764 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:44:11.765 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.765 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.766 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.766 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.766 DEBUG:boto:Data: 2023-11-02T18:44:11.767 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.767 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.768 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.768 DEBUG:boto:Params: {} 2023-11-02T18:44:11.768 DEBUG:boto:Token: None 2023-11-02T18:44:11.769 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.769 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9W4qnW9qKdRxjWYzI6JV47t4RI8= 2023-11-02T18:44:11.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9W4qnW9qKdRxjWYzI6JV47t4RI8=', 'Content-Length': '0'} 2023-11-02T18:44:11.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad0f1cd5c22254ea-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:44:11.773 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:44:11.774 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:44:11.774 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.775 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.775 DEBUG:boto:Method: GET 2023-11-02T18:44:11.776 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.776 DEBUG:boto:Data: 2023-11-02T18:44:11.777 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.777 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.778 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.778 DEBUG:boto:Params: {} 2023-11-02T18:44:11.779 DEBUG:boto:Token: None 2023-11-02T18:44:11.779 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.780 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+HUqVZaf1hRsVlEt8WtJH7zWzWs= 2023-11-02T18:44:11.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+HUqVZaf1hRsVlEt8WtJH7zWzWs=', 'Content-Length': '0'} 2023-11-02T18:44:11.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005417a8b5ea2d4376-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.826 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:44:11.827 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:44:11.827 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.828 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.828 DEBUG:boto:Method: GET 2023-11-02T18:44:11.829 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.829 DEBUG:boto:Data: 2023-11-02T18:44:11.830 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.830 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.831 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.831 DEBUG:boto:Params: {} 2023-11-02T18:44:11.832 DEBUG:boto:Token: None 2023-11-02T18:44:11.832 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:44:11.833 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+HUqVZaf1hRsVlEt8WtJH7zWzWs= 2023-11-02T18:44:11.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+HUqVZaf1hRsVlEt8WtJH7zWzWs=', 'Content-Length': '0'} 2023-11-02T18:44:11.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008de018b12f6c133b-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.837 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.837 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.838 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:44:11.838 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:44:11.839 DEBUG:boto:Method: GET 2023-11-02T18:44:11.839 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:44:11.840 DEBUG:boto:Data: 2023-11-02T18:44:11.840 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.840 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.841 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.841 DEBUG:boto:Params: {} 2023-11-02T18:44:11.842 DEBUG:boto:Token: None 2023-11-02T18:44:11.842 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.843 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036334464e6d22f16-006543edfb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.849 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.849 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.850 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.850 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:44:11.851 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:44:11.851 DEBUG:boto:Method: GET 2023-11-02T18:44:11.852 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:44:11.852 DEBUG:boto:Data: 2023-11-02T18:44:11.852 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.853 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.853 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.854 DEBUG:boto:Params: {} 2023-11-02T18:44:11.854 DEBUG:boto:Token: None 2023-11-02T18:44:11.855 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.855 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c65d212e7ee9bee6-006543edfb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.865 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.866 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:44:11.866 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:44:11.867 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.867 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:44:11.868 DEBUG:boto:Data: 2023-11-02T18:44:11.868 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.868 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.869 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.869 DEBUG:boto:Params: {} 2023-11-02T18:44:11.870 DEBUG:boto:Token: None 2023-11-02T18:44:11.870 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj76 2023-11-02T18:44:11.871 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9TZqjsBsuw6CFbc2V7OTVIpaXKI= 2023-11-02T18:44:11.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9TZqjsBsuw6CFbc2V7OTVIpaXKI=', 'Content-Length': '0'} 2023-11-02T18:44:11.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6b6820b35514906-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.875 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:44:11.875 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:44:11.875 DEBUG:boto:Method: HEAD 2023-11-02T18:44:11.876 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:44:11.876 DEBUG:boto:Data: 2023-11-02T18:44:11.877 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.877 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.877 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.878 DEBUG:boto:Params: {} 2023-11-02T18:44:11.878 DEBUG:boto:Token: None 2023-11-02T18:44:11.879 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj76 2023-11-02T18:44:11.879 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9TZqjsBsuw6CFbc2V7OTVIpaXKI= 2023-11-02T18:44:11.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9TZqjsBsuw6CFbc2V7OTVIpaXKI=', 'Content-Length': '0'} 2023-11-02T18:44:11.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d0ae1fa56cade20-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:44:11.884 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:44:11.884 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:44:11.885 DEBUG:boto:Method: GET 2023-11-02T18:44:11.885 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:44:11.886 DEBUG:boto:Data: 2023-11-02T18:44:11.886 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.887 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.887 DEBUG:boto:Params: {} 2023-11-02T18:44:11.888 DEBUG:boto:Token: None 2023-11-02T18:44:11.888 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj76 2023-11-02T18:44:11.889 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2af7nj+7XScBUaQ9dvnMpLHvtWk= 2023-11-02T18:44:11.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2af7nj+7XScBUaQ9dvnMpLHvtWk=', 'Content-Length': '0'} 2023-11-02T18:44:11.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f86334b890b7c6d-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.893 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:44:11.893 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:44:11.894 DEBUG:boto:Method: GET 2023-11-02T18:44:11.894 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:44:11.895 DEBUG:boto:Data: 2023-11-02T18:44:11.895 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.896 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.896 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.897 DEBUG:boto:Params: {} 2023-11-02T18:44:11.897 DEBUG:boto:Token: None 2023-11-02T18:44:11.898 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj76 2023-11-02T18:44:11.898 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2af7nj+7XScBUaQ9dvnMpLHvtWk= 2023-11-02T18:44:11.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2af7nj+7XScBUaQ9dvnMpLHvtWk=', 'Content-Length': '0'} 2023-11-02T18:44:11.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcf8019cc8874187-006543edfb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.942 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.943 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.943 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:44:11.944 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:44:11.944 DEBUG:boto:Method: GET 2023-11-02T18:44:11.945 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:44:11.945 DEBUG:boto:Data: 2023-11-02T18:44:11.945 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.946 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.946 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.947 DEBUG:boto:Params: {} 2023-11-02T18:44:11.947 DEBUG:boto:Token: None 2023-11-02T18:44:11.948 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.948 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd89c629ca3d95bf-006543edfb-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.954 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.955 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:11.955 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:11.956 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:44:11.956 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:44:11.957 DEBUG:boto:Method: GET 2023-11-02T18:44:11.957 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:44:11.958 DEBUG:boto:Data: 2023-11-02T18:44:11.958 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.958 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:11.959 DEBUG:boto:Port: 8000 2023-11-02T18:44:11.959 DEBUG:boto:Params: {} 2023-11-02T18:44:11.960 DEBUG:boto:Token: None 2023-11-02T18:44:11.960 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/ 2023-11-02T18:44:11.961 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE= 2023-11-02T18:44:11.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EjFe2sAsScXBpHeIX/8Fvt+OHHE=', 'Content-Length': '0'} 2023-11-02T18:44:11.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075ccc642d6a75971-006543edfb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:11.972 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:11.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:44:11.973 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:44:11.973 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:44:11.974 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:44:11.975 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:44:11.975 DEBUG:boto:Method: GET 2023-11-02T18:44:11.975 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:44:11.976 DEBUG:boto:Data: 2023-11-02T18:44:11.976 DEBUG:boto:Headers: {} 2023-11-02T18:44:11.977 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:11.977 DEBUG:boto:Port: 8001 2023-11-02T18:44:11.978 DEBUG:boto:Params: {} 2023-11-02T18:44:11.978 DEBUG:boto:Token: None 2023-11-02T18:44:11.979 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:11 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:44:11.979 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:05wbCwc9pqeanrw/pPCw9RRGxw4= 2023-11-02T18:44:11.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:05wbCwc9pqeanrw/pPCw9RRGxw4=', 'Content-Length': '0'} 2023-11-02T18:44:11.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000521516e604d1ab46-006543edfb-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.026 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:44:12.027 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:44:12.027 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.028 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.028 DEBUG:boto:Method: GET 2023-11-02T18:44:12.029 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.029 DEBUG:boto:Data: 2023-11-02T18:44:12.029 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.030 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.031 DEBUG:boto:Params: {} 2023-11-02T18:44:12.031 DEBUG:boto:Token: None 2023-11-02T18:44:12.032 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WQizJ772N7e0mnM8v/h2nKXkNB4= 2023-11-02T18:44:12.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WQizJ772N7e0mnM8v/h2nKXkNB4=', 'Content-Length': '0'} 2023-11-02T18:44:12.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a5bcfbed9da8ff6-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.037 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:44:12.037 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:44:12.037 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.038 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.038 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.039 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.039 DEBUG:boto:Data: 2023-11-02T18:44:12.040 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.040 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.040 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.041 DEBUG:boto:Params: {} 2023-11-02T18:44:12.041 DEBUG:boto:Token: None 2023-11-02T18:44:12.042 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.042 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eEPEeLF65bQ2aLLNWs/Qxvyl+Lk= 2023-11-02T18:44:12.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eEPEeLF65bQ2aLLNWs/Qxvyl+Lk=', 'Content-Length': '0'} 2023-11-02T18:44:12.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c86376c4d2897b6-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.047 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:44:12.047 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:44:12.047 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.048 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.048 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.049 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.049 DEBUG:boto:Data: 2023-11-02T18:44:12.050 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.050 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.051 DEBUG:boto:Params: {} 2023-11-02T18:44:12.051 DEBUG:boto:Token: None 2023-11-02T18:44:12.052 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.052 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eEPEeLF65bQ2aLLNWs/Qxvyl+Lk= 2023-11-02T18:44:12.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eEPEeLF65bQ2aLLNWs/Qxvyl+Lk=', 'Content-Length': '0'} 2023-11-02T18:44:12.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e683d5381a6a54-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:44:12.057 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:44:12.057 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:44:12.057 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.058 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.058 DEBUG:boto:Method: GET 2023-11-02T18:44:12.059 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.059 DEBUG:boto:Data: 2023-11-02T18:44:12.060 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.060 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.061 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.061 DEBUG:boto:Params: {} 2023-11-02T18:44:12.061 DEBUG:boto:Token: None 2023-11-02T18:44:12.062 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.062 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WQizJ772N7e0mnM8v/h2nKXkNB4= 2023-11-02T18:44:12.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WQizJ772N7e0mnM8v/h2nKXkNB4=', 'Content-Length': '0'} 2023-11-02T18:44:12.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000338103e2e84900be-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.066 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:44:12.067 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:44:12.067 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.068 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.068 DEBUG:boto:Method: GET 2023-11-02T18:44:12.069 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.069 DEBUG:boto:Data: 2023-11-02T18:44:12.070 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.071 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.071 DEBUG:boto:Params: {} 2023-11-02T18:44:12.071 DEBUG:boto:Token: None 2023-11-02T18:44:12.072 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:44:12.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WQizJ772N7e0mnM8v/h2nKXkNB4= 2023-11-02T18:44:12.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WQizJ772N7e0mnM8v/h2nKXkNB4=', 'Content-Length': '0'} 2023-11-02T18:44:12.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bebffd306211d25e-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.122 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.123 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.123 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:44:12.124 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:44:12.124 DEBUG:boto:Method: GET 2023-11-02T18:44:12.125 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:44:12.125 DEBUG:boto:Data: 2023-11-02T18:44:12.126 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.126 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.127 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.127 DEBUG:boto:Params: {} 2023-11-02T18:44:12.127 DEBUG:boto:Token: None 2023-11-02T18:44:12.128 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.128 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000576f0bdf992e2d62-006543edfc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.135 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.136 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.136 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.137 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:44:12.137 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:44:12.138 DEBUG:boto:Method: GET 2023-11-02T18:44:12.138 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:44:12.138 DEBUG:boto:Data: 2023-11-02T18:44:12.139 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.140 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.140 DEBUG:boto:Params: {} 2023-11-02T18:44:12.141 DEBUG:boto:Token: None 2023-11-02T18:44:12.141 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.142 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cf75c27b3aae3fd-006543edfc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.152 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.152 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:44:12.153 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:44:12.153 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.154 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:44:12.154 DEBUG:boto:Data: 2023-11-02T18:44:12.154 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.155 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.155 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.156 DEBUG:boto:Params: {} 2023-11-02T18:44:12.156 DEBUG:boto:Token: None 2023-11-02T18:44:12.157 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj85 2023-11-02T18:44:12.157 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CWqWBefQ1v7hKZh+7yKxLDMMJSU= 2023-11-02T18:44:12.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CWqWBefQ1v7hKZh+7yKxLDMMJSU=', 'Content-Length': '0'} 2023-11-02T18:44:12.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e76be2491a04698-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.161 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:44:12.161 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:44:12.162 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.162 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:44:12.162 DEBUG:boto:Data: 2023-11-02T18:44:12.163 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.163 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.164 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.164 DEBUG:boto:Params: {} 2023-11-02T18:44:12.165 DEBUG:boto:Token: None 2023-11-02T18:44:12.165 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj85 2023-11-02T18:44:12.166 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CWqWBefQ1v7hKZh+7yKxLDMMJSU= 2023-11-02T18:44:12.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CWqWBefQ1v7hKZh+7yKxLDMMJSU=', 'Content-Length': '0'} 2023-11-02T18:44:12.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000810a7aada9d08e8e-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:44:12.170 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:44:12.170 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:44:12.171 DEBUG:boto:Method: GET 2023-11-02T18:44:12.171 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:44:12.172 DEBUG:boto:Data: 2023-11-02T18:44:12.172 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.172 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.173 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.173 DEBUG:boto:Params: {} 2023-11-02T18:44:12.174 DEBUG:boto:Token: None 2023-11-02T18:44:12.174 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj85 2023-11-02T18:44:12.175 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6c9wpqhx8F1Pa9+/KE9Gg3C9YOI= 2023-11-02T18:44:12.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6c9wpqhx8F1Pa9+/KE9Gg3C9YOI=', 'Content-Length': '0'} 2023-11-02T18:44:12.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a635b82d88dcfc4-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.179 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:44:12.179 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:44:12.180 DEBUG:boto:Method: GET 2023-11-02T18:44:12.180 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:44:12.180 DEBUG:boto:Data: 2023-11-02T18:44:12.181 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.181 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.182 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.182 DEBUG:boto:Params: {} 2023-11-02T18:44:12.183 DEBUG:boto:Token: None 2023-11-02T18:44:12.183 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj85 2023-11-02T18:44:12.184 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6c9wpqhx8F1Pa9+/KE9Gg3C9YOI= 2023-11-02T18:44:12.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6c9wpqhx8F1Pa9+/KE9Gg3C9YOI=', 'Content-Length': '0'} 2023-11-02T18:44:12.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db55b3c641ec24c-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.188 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.188 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.189 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:44:12.189 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:44:12.190 DEBUG:boto:Method: GET 2023-11-02T18:44:12.190 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:44:12.190 DEBUG:boto:Data: 2023-11-02T18:44:12.191 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.191 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.192 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.192 DEBUG:boto:Params: {} 2023-11-02T18:44:12.193 DEBUG:boto:Token: None 2023-11-02T18:44:12.193 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.194 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7a592022f96fa52-006543edfc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.200 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.201 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.201 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.202 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:44:12.202 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:44:12.202 DEBUG:boto:Method: GET 2023-11-02T18:44:12.203 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:44:12.203 DEBUG:boto:Data: 2023-11-02T18:44:12.204 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.204 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.205 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.205 DEBUG:boto:Params: {} 2023-11-02T18:44:12.206 DEBUG:boto:Token: None 2023-11-02T18:44:12.206 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.207 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c96ae5d46c858c04-006543edfc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.258 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:44:12.260 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:44:12.260 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:44:12.260 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.261 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.261 DEBUG:boto:Method: GET 2023-11-02T18:44:12.262 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.262 DEBUG:boto:Data: 2023-11-02T18:44:12.263 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.264 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.264 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.265 DEBUG:boto:Params: {} 2023-11-02T18:44:12.265 DEBUG:boto:Token: None 2023-11-02T18:44:12.266 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.266 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sYVUrB0ksvHmbSttOIcoBiHP2RY= 2023-11-02T18:44:12.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sYVUrB0ksvHmbSttOIcoBiHP2RY=', 'Content-Length': '0'} 2023-11-02T18:44:12.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6af8582a01127cc-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.271 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:44:12.271 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:44:12.271 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.272 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.272 DEBUG:boto:Method: GET 2023-11-02T18:44:12.273 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.273 DEBUG:boto:Data: 2023-11-02T18:44:12.274 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.274 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.275 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.275 DEBUG:boto:Params: {} 2023-11-02T18:44:12.276 DEBUG:boto:Token: None 2023-11-02T18:44:12.276 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.277 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sYVUrB0ksvHmbSttOIcoBiHP2RY= 2023-11-02T18:44:12.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sYVUrB0ksvHmbSttOIcoBiHP2RY=', 'Content-Length': '0'} 2023-11-02T18:44:12.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d555acd60238e32c-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.281 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:44:12.281 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:44:12.282 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.282 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.283 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.283 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.283 DEBUG:boto:Data: 2023-11-02T18:44:12.284 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.284 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.285 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.285 DEBUG:boto:Params: {} 2023-11-02T18:44:12.286 DEBUG:boto:Token: None 2023-11-02T18:44:12.286 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.287 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0X4LjTLWQYp8l0Gk0gxUOm8nzcM= 2023-11-02T18:44:12.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0X4LjTLWQYp8l0Gk0gxUOm8nzcM=', 'Content-Length': '0'} 2023-11-02T18:44:12.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000681555b401079509-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.291 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:44:12.291 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:44:12.292 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.292 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.292 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.293 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.293 DEBUG:boto:Data: 2023-11-02T18:44:12.294 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.294 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.295 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.295 DEBUG:boto:Params: {} 2023-11-02T18:44:12.296 DEBUG:boto:Token: None 2023-11-02T18:44:12.296 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.297 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0X4LjTLWQYp8l0Gk0gxUOm8nzcM= 2023-11-02T18:44:12.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0X4LjTLWQYp8l0Gk0gxUOm8nzcM=', 'Content-Length': '0'} 2023-11-02T18:44:12.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e74d08404780e1e3-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:44:12.301 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:44:12.302 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:44:12.302 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.302 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.303 DEBUG:boto:Method: GET 2023-11-02T18:44:12.303 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.304 DEBUG:boto:Data: 2023-11-02T18:44:12.304 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.305 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.305 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.305 DEBUG:boto:Params: {} 2023-11-02T18:44:12.306 DEBUG:boto:Token: None 2023-11-02T18:44:12.307 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.307 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sYVUrB0ksvHmbSttOIcoBiHP2RY= 2023-11-02T18:44:12.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sYVUrB0ksvHmbSttOIcoBiHP2RY=', 'Content-Length': '0'} 2023-11-02T18:44:12.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a038b565d53a115f-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.354 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:44:12.355 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:44:12.355 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.356 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.356 DEBUG:boto:Method: GET 2023-11-02T18:44:12.357 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.357 DEBUG:boto:Data: 2023-11-02T18:44:12.358 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.359 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.359 DEBUG:boto:Params: {} 2023-11-02T18:44:12.359 DEBUG:boto:Token: None 2023-11-02T18:44:12.360 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:44:12.360 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sYVUrB0ksvHmbSttOIcoBiHP2RY= 2023-11-02T18:44:12.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sYVUrB0ksvHmbSttOIcoBiHP2RY=', 'Content-Length': '0'} 2023-11-02T18:44:12.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006351c02a8af8f9e7-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.365 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.365 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.366 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:44:12.366 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:44:12.367 DEBUG:boto:Method: GET 2023-11-02T18:44:12.367 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:44:12.367 DEBUG:boto:Data: 2023-11-02T18:44:12.368 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.368 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.369 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.369 DEBUG:boto:Params: {} 2023-11-02T18:44:12.369 DEBUG:boto:Token: None 2023-11-02T18:44:12.370 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.370 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c17fc2c64fc1604-006543edfc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.376 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.377 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.377 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.378 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:44:12.378 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:44:12.379 DEBUG:boto:Method: GET 2023-11-02T18:44:12.379 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:44:12.379 DEBUG:boto:Data: 2023-11-02T18:44:12.380 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.380 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.381 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.381 DEBUG:boto:Params: {} 2023-11-02T18:44:12.382 DEBUG:boto:Token: None 2023-11-02T18:44:12.382 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.383 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007083c892f5bcef25-006543edfc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.434 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.435 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:44:12.436 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:44:12.436 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.437 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:44:12.437 DEBUG:boto:Data: 2023-11-02T18:44:12.437 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.438 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.438 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.439 DEBUG:boto:Params: {} 2023-11-02T18:44:12.439 DEBUG:boto:Token: None 2023-11-02T18:44:12.440 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj87 2023-11-02T18:44:12.441 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dwLMoiig4wMNV5M8uQ7aj42pICg= 2023-11-02T18:44:12.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dwLMoiig4wMNV5M8uQ7aj42pICg=', 'Content-Length': '0'} 2023-11-02T18:44:12.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fd9cdfae95ec6f0-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.446 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:44:12.446 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:44:12.447 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.447 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:44:12.447 DEBUG:boto:Data: 2023-11-02T18:44:12.448 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.448 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.449 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.449 DEBUG:boto:Params: {} 2023-11-02T18:44:12.450 DEBUG:boto:Token: None 2023-11-02T18:44:12.450 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj87 2023-11-02T18:44:12.451 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dwLMoiig4wMNV5M8uQ7aj42pICg= 2023-11-02T18:44:12.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dwLMoiig4wMNV5M8uQ7aj42pICg=', 'Content-Length': '0'} 2023-11-02T18:44:12.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008748abb34ae0910c-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:44:12.455 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:44:12.456 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:44:12.456 DEBUG:boto:Method: GET 2023-11-02T18:44:12.457 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:44:12.457 DEBUG:boto:Data: 2023-11-02T18:44:12.457 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.458 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.458 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.459 DEBUG:boto:Params: {} 2023-11-02T18:44:12.459 DEBUG:boto:Token: None 2023-11-02T18:44:12.460 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj87 2023-11-02T18:44:12.460 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8uCy9dn5RiE7mXO5aM1jbo/oFm0= 2023-11-02T18:44:12.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8uCy9dn5RiE7mXO5aM1jbo/oFm0=', 'Content-Length': '0'} 2023-11-02T18:44:12.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ea4bca5e8283d89-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.465 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:44:12.466 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:44:12.466 DEBUG:boto:Method: GET 2023-11-02T18:44:12.466 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:44:12.467 DEBUG:boto:Data: 2023-11-02T18:44:12.467 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.468 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.468 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.469 DEBUG:boto:Params: {} 2023-11-02T18:44:12.469 DEBUG:boto:Token: None 2023-11-02T18:44:12.470 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj87 2023-11-02T18:44:12.470 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8uCy9dn5RiE7mXO5aM1jbo/oFm0= 2023-11-02T18:44:12.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8uCy9dn5RiE7mXO5aM1jbo/oFm0=', 'Content-Length': '0'} 2023-11-02T18:44:12.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000279004ffa230dc3b-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.475 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.475 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.476 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:44:12.476 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:44:12.477 DEBUG:boto:Method: GET 2023-11-02T18:44:12.477 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:44:12.477 DEBUG:boto:Data: 2023-11-02T18:44:12.478 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.478 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.479 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.479 DEBUG:boto:Params: {} 2023-11-02T18:44:12.480 DEBUG:boto:Token: None 2023-11-02T18:44:12.481 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.481 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000457a73423e6164f0-006543edfc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.487 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.488 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.488 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.489 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:44:12.489 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:44:12.490 DEBUG:boto:Method: GET 2023-11-02T18:44:12.490 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:44:12.490 DEBUG:boto:Data: 2023-11-02T18:44:12.491 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.491 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.492 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.492 DEBUG:boto:Params: {} 2023-11-02T18:44:12.493 DEBUG:boto:Token: None 2023-11-02T18:44:12.493 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.494 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bf007a3e74d8ce6-006543edfc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.550 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:44:12.552 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:44:12.552 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:44:12.553 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.553 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.554 DEBUG:boto:Method: GET 2023-11-02T18:44:12.554 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.555 DEBUG:boto:Data: 2023-11-02T18:44:12.555 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.556 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.556 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.556 DEBUG:boto:Params: {} 2023-11-02T18:44:12.557 DEBUG:boto:Token: None 2023-11-02T18:44:12.558 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.558 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a9tB+rIfSIyxBSXL4utcUDfjntM= 2023-11-02T18:44:12.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a9tB+rIfSIyxBSXL4utcUDfjntM=', 'Content-Length': '0'} 2023-11-02T18:44:12.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fb75215f9ff5e8a-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.563 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:44:12.563 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:44:12.564 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.564 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.565 DEBUG:boto:Method: GET 2023-11-02T18:44:12.565 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.565 DEBUG:boto:Data: 2023-11-02T18:44:12.566 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.567 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.567 DEBUG:boto:Params: {} 2023-11-02T18:44:12.568 DEBUG:boto:Token: None 2023-11-02T18:44:12.568 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a9tB+rIfSIyxBSXL4utcUDfjntM= 2023-11-02T18:44:12.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a9tB+rIfSIyxBSXL4utcUDfjntM=', 'Content-Length': '0'} 2023-11-02T18:44:12.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b7f67d8e4f8d00d-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.573 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:44:12.574 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:44:12.574 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.574 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.575 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.575 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.576 DEBUG:boto:Data: 2023-11-02T18:44:12.576 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.577 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.577 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.577 DEBUG:boto:Params: {} 2023-11-02T18:44:12.578 DEBUG:boto:Token: None 2023-11-02T18:44:12.578 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.579 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+5Xw94nWqs1q+aGikY5CjSws2NY= 2023-11-02T18:44:12.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+5Xw94nWqs1q+aGikY5CjSws2NY=', 'Content-Length': '0'} 2023-11-02T18:44:12.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1325f013f30a201-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.583 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:44:12.583 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:44:12.584 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.584 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.584 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.585 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.585 DEBUG:boto:Data: 2023-11-02T18:44:12.586 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.586 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.587 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.587 DEBUG:boto:Params: {} 2023-11-02T18:44:12.588 DEBUG:boto:Token: None 2023-11-02T18:44:12.588 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.589 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+5Xw94nWqs1q+aGikY5CjSws2NY= 2023-11-02T18:44:12.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+5Xw94nWqs1q+aGikY5CjSws2NY=', 'Content-Length': '0'} 2023-11-02T18:44:12.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096c87ec6aed6e0bb-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:44:12.593 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:44:12.593 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:44:12.594 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.594 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.595 DEBUG:boto:Method: GET 2023-11-02T18:44:12.595 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.596 DEBUG:boto:Data: 2023-11-02T18:44:12.596 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.596 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.597 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.597 DEBUG:boto:Params: {} 2023-11-02T18:44:12.598 DEBUG:boto:Token: None 2023-11-02T18:44:12.598 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.599 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a9tB+rIfSIyxBSXL4utcUDfjntM= 2023-11-02T18:44:12.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a9tB+rIfSIyxBSXL4utcUDfjntM=', 'Content-Length': '0'} 2023-11-02T18:44:12.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016fbc085784a5050-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.646 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:44:12.647 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:44:12.647 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.648 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.649 DEBUG:boto:Method: GET 2023-11-02T18:44:12.649 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.650 DEBUG:boto:Data: 2023-11-02T18:44:12.650 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.650 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.651 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.651 DEBUG:boto:Params: {} 2023-11-02T18:44:12.652 DEBUG:boto:Token: None 2023-11-02T18:44:12.652 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:44:12.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a9tB+rIfSIyxBSXL4utcUDfjntM= 2023-11-02T18:44:12.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a9tB+rIfSIyxBSXL4utcUDfjntM=', 'Content-Length': '0'} 2023-11-02T18:44:12.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823782e40c08d304-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.657 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.657 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.658 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:44:12.658 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:44:12.659 DEBUG:boto:Method: GET 2023-11-02T18:44:12.659 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:44:12.659 DEBUG:boto:Data: 2023-11-02T18:44:12.660 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.660 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.661 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.661 DEBUG:boto:Params: {} 2023-11-02T18:44:12.662 DEBUG:boto:Token: None 2023-11-02T18:44:12.662 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.663 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e7ed4fc97a92431-006543edfc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.670 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.671 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.671 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.672 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:44:12.672 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:44:12.673 DEBUG:boto:Method: GET 2023-11-02T18:44:12.673 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:44:12.674 DEBUG:boto:Data: 2023-11-02T18:44:12.674 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.675 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.675 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.676 DEBUG:boto:Params: {} 2023-11-02T18:44:12.676 DEBUG:boto:Token: None 2023-11-02T18:44:12.677 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.677 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c03ea551a0bdb9a-006543edfc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.687 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.687 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:44:12.688 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:44:12.688 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.689 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:44:12.689 DEBUG:boto:Data: 2023-11-02T18:44:12.689 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.690 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.691 DEBUG:boto:Params: {} 2023-11-02T18:44:12.691 DEBUG:boto:Token: None 2023-11-02T18:44:12.693 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj92 2023-11-02T18:44:12.694 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W3oUmSLbgq5c9SgpS6OG2un76uQ= 2023-11-02T18:44:12.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W3oUmSLbgq5c9SgpS6OG2un76uQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c105a62db3a6ec-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.698 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:44:12.698 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:44:12.698 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.699 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:44:12.699 DEBUG:boto:Data: 2023-11-02T18:44:12.700 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.701 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.701 DEBUG:boto:Params: {} 2023-11-02T18:44:12.701 DEBUG:boto:Token: None 2023-11-02T18:44:12.702 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj92 2023-11-02T18:44:12.702 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W3oUmSLbgq5c9SgpS6OG2un76uQ= 2023-11-02T18:44:12.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W3oUmSLbgq5c9SgpS6OG2un76uQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a7dec60985c53dd-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:44:12.707 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:44:12.707 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:44:12.708 DEBUG:boto:Method: GET 2023-11-02T18:44:12.708 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:44:12.709 DEBUG:boto:Data: 2023-11-02T18:44:12.709 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.710 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.710 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.711 DEBUG:boto:Params: {} 2023-11-02T18:44:12.711 DEBUG:boto:Token: None 2023-11-02T18:44:12.712 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj92 2023-11-02T18:44:12.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gcHiDuQT8ppZuXnQp6bXl7kecBQ= 2023-11-02T18:44:12.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gcHiDuQT8ppZuXnQp6bXl7kecBQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f256d7fb7da3ebe9-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.716 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:44:12.716 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:44:12.717 DEBUG:boto:Method: GET 2023-11-02T18:44:12.717 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:44:12.718 DEBUG:boto:Data: 2023-11-02T18:44:12.718 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.719 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.719 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.719 DEBUG:boto:Params: {} 2023-11-02T18:44:12.720 DEBUG:boto:Token: None 2023-11-02T18:44:12.720 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj92 2023-11-02T18:44:12.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gcHiDuQT8ppZuXnQp6bXl7kecBQ= 2023-11-02T18:44:12.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gcHiDuQT8ppZuXnQp6bXl7kecBQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eafeb1fc42d7381-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.766 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.767 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.768 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:44:12.768 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:44:12.768 DEBUG:boto:Method: GET 2023-11-02T18:44:12.769 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:44:12.769 DEBUG:boto:Data: 2023-11-02T18:44:12.770 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.770 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.771 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.771 DEBUG:boto:Params: {} 2023-11-02T18:44:12.772 DEBUG:boto:Token: None 2023-11-02T18:44:12.772 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.773 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8a17dc5820d3629-006543edfc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.779 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.779 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.780 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.780 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:44:12.781 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:44:12.781 DEBUG:boto:Method: GET 2023-11-02T18:44:12.782 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:44:12.782 DEBUG:boto:Data: 2023-11-02T18:44:12.782 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.783 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.783 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.784 DEBUG:boto:Params: {} 2023-11-02T18:44:12.784 DEBUG:boto:Token: None 2023-11-02T18:44:12.785 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a899ac97621149-006543edfc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.794 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T18:44:12.796 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:44:12.796 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:44:12.797 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.797 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.797 DEBUG:boto:Method: GET 2023-11-02T18:44:12.798 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.798 DEBUG:boto:Data: 2023-11-02T18:44:12.799 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.800 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.800 DEBUG:boto:Params: {} 2023-11-02T18:44:12.801 DEBUG:boto:Token: None 2023-11-02T18:44:12.801 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.802 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8nL1cPHM4+sHJfPT8af4A3G9t1o= 2023-11-02T18:44:12.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8nL1cPHM4+sHJfPT8af4A3G9t1o=', 'Content-Length': '0'} 2023-11-02T18:44:12.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c4aba6c2b0cfc6-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.850 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:44:12.851 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:44:12.852 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.852 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.853 DEBUG:boto:Method: GET 2023-11-02T18:44:12.853 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.853 DEBUG:boto:Data: 2023-11-02T18:44:12.854 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.854 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.855 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.855 DEBUG:boto:Params: {} 2023-11-02T18:44:12.856 DEBUG:boto:Token: None 2023-11-02T18:44:12.857 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.857 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8nL1cPHM4+sHJfPT8af4A3G9t1o= 2023-11-02T18:44:12.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8nL1cPHM4+sHJfPT8af4A3G9t1o=', 'Content-Length': '0'} 2023-11-02T18:44:12.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008302b645421050de-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.862 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:44:12.862 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:44:12.863 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.863 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.864 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.864 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.864 DEBUG:boto:Data: 2023-11-02T18:44:12.865 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.865 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.866 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.866 DEBUG:boto:Params: {} 2023-11-02T18:44:12.867 DEBUG:boto:Token: None 2023-11-02T18:44:12.867 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.868 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ti22UO2DilxnvFYvVSgHIwgVChU= 2023-11-02T18:44:12.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ti22UO2DilxnvFYvVSgHIwgVChU=', 'Content-Length': '0'} 2023-11-02T18:44:12.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b992a1a3ad3d43f-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.872 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:44:12.872 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:44:12.873 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.873 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.874 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.874 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.874 DEBUG:boto:Data: 2023-11-02T18:44:12.875 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.875 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.876 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.876 DEBUG:boto:Params: {} 2023-11-02T18:44:12.877 DEBUG:boto:Token: None 2023-11-02T18:44:12.877 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.878 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ti22UO2DilxnvFYvVSgHIwgVChU= 2023-11-02T18:44:12.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ti22UO2DilxnvFYvVSgHIwgVChU=', 'Content-Length': '0'} 2023-11-02T18:44:12.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc291119e7a20db5-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T18:44:12.882 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:44:12.882 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:44:12.883 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.883 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.884 DEBUG:boto:Method: GET 2023-11-02T18:44:12.884 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.885 DEBUG:boto:Data: 2023-11-02T18:44:12.885 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.885 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.886 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.886 DEBUG:boto:Params: {} 2023-11-02T18:44:12.887 DEBUG:boto:Token: None 2023-11-02T18:44:12.887 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.888 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8nL1cPHM4+sHJfPT8af4A3G9t1o= 2023-11-02T18:44:12.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8nL1cPHM4+sHJfPT8af4A3G9t1o=', 'Content-Length': '0'} 2023-11-02T18:44:12.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a11c50370f3e0758-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.892 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:44:12.893 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:44:12.893 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.894 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.894 DEBUG:boto:Method: GET 2023-11-02T18:44:12.894 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.895 DEBUG:boto:Data: 2023-11-02T18:44:12.895 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.896 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.896 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.897 DEBUG:boto:Params: {} 2023-11-02T18:44:12.897 DEBUG:boto:Token: None 2023-11-02T18:44:12.898 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:44:12.898 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8nL1cPHM4+sHJfPT8af4A3G9t1o= 2023-11-02T18:44:12.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8nL1cPHM4+sHJfPT8af4A3G9t1o=', 'Content-Length': '0'} 2023-11-02T18:44:12.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c02b8a785474c44a-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.903 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.903 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.904 DEBUG:boto:path=/ibtljn-59/?prefix=obj93 2023-11-02T18:44:12.904 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj93 2023-11-02T18:44:12.905 DEBUG:boto:Method: GET 2023-11-02T18:44:12.905 DEBUG:boto:Path: /ibtljn-59/?prefix=obj93 2023-11-02T18:44:12.906 DEBUG:boto:Data: 2023-11-02T18:44:12.906 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.907 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.907 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.908 DEBUG:boto:Params: {} 2023-11-02T18:44:12.908 DEBUG:boto:Token: None 2023-11-02T18:44:12.909 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.909 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097be5dc9c707e195-006543edfc-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.962 DEBUG:boto:b'ibtljn-59obj931000falseobj932023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.963 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:12.964 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:12.964 DEBUG:boto:path=/ibtljn-59/?prefix=obj93 2023-11-02T18:44:12.965 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj93 2023-11-02T18:44:12.965 DEBUG:boto:Method: GET 2023-11-02T18:44:12.966 DEBUG:boto:Path: /ibtljn-59/?prefix=obj93 2023-11-02T18:44:12.966 DEBUG:boto:Data: 2023-11-02T18:44:12.966 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.967 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.967 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.968 DEBUG:boto:Params: {} 2023-11-02T18:44:12.968 DEBUG:boto:Token: None 2023-11-02T18:44:12.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/ 2023-11-02T18:44:12.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ= 2023-11-02T18:44:12.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uwi0Pb73GYt5QMHAq1x0GrEBitQ=', 'Content-Length': '0'} 2023-11-02T18:44:12.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009df8dafa81ffe97d-006543edfc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.980 DEBUG:boto:b'ibtljn-59obj931000falseobj932023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:12.980 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:44:12.981 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:44:12.981 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.982 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:44:12.982 DEBUG:boto:Data: 2023-11-02T18:44:12.983 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.983 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:12.983 DEBUG:boto:Port: 8001 2023-11-02T18:44:12.984 DEBUG:boto:Params: {} 2023-11-02T18:44:12.984 DEBUG:boto:Token: None 2023-11-02T18:44:12.985 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj93 2023-11-02T18:44:12.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cnoLefHN+fv/rgNiMt76F5DqQXg= 2023-11-02T18:44:12.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cnoLefHN+fv/rgNiMt76F5DqQXg=', 'Content-Length': '0'} 2023-11-02T18:44:12.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000590106b7c0bd7b14-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.989 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:44:12.990 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:44:12.990 DEBUG:boto:Method: HEAD 2023-11-02T18:44:12.991 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:44:12.991 DEBUG:boto:Data: 2023-11-02T18:44:12.992 DEBUG:boto:Headers: {} 2023-11-02T18:44:12.992 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:12.993 DEBUG:boto:Port: 8000 2023-11-02T18:44:12.993 DEBUG:boto:Params: {} 2023-11-02T18:44:12.993 DEBUG:boto:Token: None 2023-11-02T18:44:12.994 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:12 GMT /ibtljn-59/obj93 2023-11-02T18:44:12.994 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cnoLefHN+fv/rgNiMt76F5DqQXg= 2023-11-02T18:44:12.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:12 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cnoLefHN+fv/rgNiMt76F5DqQXg=', 'Content-Length': '0'} 2023-11-02T18:44:12.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008e826eb5f060d5c-006543edfc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:12.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T18:44:12.999 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:44:12.999 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:44:13.000 DEBUG:boto:Method: GET 2023-11-02T18:44:13.000 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:44:13.001 DEBUG:boto:Data: 2023-11-02T18:44:13.001 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.002 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:13.002 DEBUG:boto:Port: 8001 2023-11-02T18:44:13.003 DEBUG:boto:Params: {} 2023-11-02T18:44:13.003 DEBUG:boto:Token: None 2023-11-02T18:44:13.004 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj93 2023-11-02T18:44:13.004 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5ac04/hPfBeH0JFMY1IHOcCJ8gw= 2023-11-02T18:44:13.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5ac04/hPfBeH0JFMY1IHOcCJ8gw=', 'Content-Length': '0'} 2023-11-02T18:44:13.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5d25d8914aedbe1-006543edfc-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.008 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:44:13.009 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:44:13.009 DEBUG:boto:Method: GET 2023-11-02T18:44:13.010 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:44:13.010 DEBUG:boto:Data: 2023-11-02T18:44:13.011 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.011 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.012 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.012 DEBUG:boto:Params: {} 2023-11-02T18:44:13.013 DEBUG:boto:Token: None 2023-11-02T18:44:13.013 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj93 2023-11-02T18:44:13.014 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5ac04/hPfBeH0JFMY1IHOcCJ8gw= 2023-11-02T18:44:13.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5ac04/hPfBeH0JFMY1IHOcCJ8gw=', 'Content-Length': '0'} 2023-11-02T18:44:13.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005348986c38fcf0e3-006543edfd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.018 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:13.018 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:13.019 DEBUG:boto:path=/ibtljn-59/?prefix=obj93 2023-11-02T18:44:13.019 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj93 2023-11-02T18:44:13.020 DEBUG:boto:Method: GET 2023-11-02T18:44:13.020 DEBUG:boto:Path: /ibtljn-59/?prefix=obj93 2023-11-02T18:44:13.021 DEBUG:boto:Data: 2023-11-02T18:44:13.021 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.022 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:13.022 DEBUG:boto:Port: 8001 2023-11-02T18:44:13.023 DEBUG:boto:Params: {} 2023-11-02T18:44:13.023 DEBUG:boto:Token: None 2023-11-02T18:44:13.024 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/ 2023-11-02T18:44:13.025 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IO5SyFII+bwMxHtiNuy0Z0lW768= 2023-11-02T18:44:13.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IO5SyFII+bwMxHtiNuy0Z0lW768=', 'Content-Length': '0'} 2023-11-02T18:44:13.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0947ae485d23c75-006543edfd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.031 DEBUG:boto:b'ibtljn-59obj931000falseobj932023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:13.032 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:13.032 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:13.033 DEBUG:boto:path=/ibtljn-59/?prefix=obj93 2023-11-02T18:44:13.034 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj93 2023-11-02T18:44:13.034 DEBUG:boto:Method: GET 2023-11-02T18:44:13.035 DEBUG:boto:Path: /ibtljn-59/?prefix=obj93 2023-11-02T18:44:13.036 DEBUG:boto:Data: 2023-11-02T18:44:13.037 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.038 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.039 DEBUG:boto:Params: {} 2023-11-02T18:44:13.040 DEBUG:boto:Token: None 2023-11-02T18:44:13.040 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/ 2023-11-02T18:44:13.041 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IO5SyFII+bwMxHtiNuy0Z0lW768= 2023-11-02T18:44:13.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IO5SyFII+bwMxHtiNuy0Z0lW768=', 'Content-Length': '0'} 2023-11-02T18:44:13.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2877ee9911f8405-006543edfd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.051 DEBUG:boto:b'ibtljn-59obj931000falseobj932023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:13.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:44:13.052 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:44:13.053 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:44:13.053 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.054 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.054 DEBUG:boto:Method: GET 2023-11-02T18:44:13.055 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.055 DEBUG:boto:Data: 2023-11-02T18:44:13.055 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.056 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:13.056 DEBUG:boto:Port: 8001 2023-11-02T18:44:13.057 DEBUG:boto:Params: {} 2023-11-02T18:44:13.057 DEBUG:boto:Token: None 2023-11-02T18:44:13.058 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.058 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uyvXzzUFPUIbVdBuoYuc5s3rlwc= 2023-11-02T18:44:13.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uyvXzzUFPUIbVdBuoYuc5s3rlwc=', 'Content-Length': '0'} 2023-11-02T18:44:13.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b74a17c96599c5f1-006543edfd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.102 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:44:13.103 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:44:13.104 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.104 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.104 DEBUG:boto:Method: GET 2023-11-02T18:44:13.105 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.105 DEBUG:boto:Data: 2023-11-02T18:44:13.106 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.107 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.107 DEBUG:boto:Params: {} 2023-11-02T18:44:13.108 DEBUG:boto:Token: None 2023-11-02T18:44:13.109 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.109 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uyvXzzUFPUIbVdBuoYuc5s3rlwc= 2023-11-02T18:44:13.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uyvXzzUFPUIbVdBuoYuc5s3rlwc=', 'Content-Length': '0'} 2023-11-02T18:44:13.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046f38d7823cc9c0f-006543edfd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.114 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:44:13.114 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:44:13.115 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.115 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.116 DEBUG:boto:Method: HEAD 2023-11-02T18:44:13.116 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.116 DEBUG:boto:Data: 2023-11-02T18:44:13.117 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.117 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:13.118 DEBUG:boto:Port: 8001 2023-11-02T18:44:13.118 DEBUG:boto:Params: {} 2023-11-02T18:44:13.119 DEBUG:boto:Token: None 2023-11-02T18:44:13.119 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ASBtYPJjBFdJ9431OC+EuiPLRHo= 2023-11-02T18:44:13.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ASBtYPJjBFdJ9431OC+EuiPLRHo=', 'Content-Length': '0'} 2023-11-02T18:44:13.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d19e3b529021277-006543edfd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.124 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:44:13.124 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:44:13.124 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.125 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.125 DEBUG:boto:Method: HEAD 2023-11-02T18:44:13.126 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.126 DEBUG:boto:Data: 2023-11-02T18:44:13.127 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.127 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.128 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.128 DEBUG:boto:Params: {} 2023-11-02T18:44:13.129 DEBUG:boto:Token: None 2023-11-02T18:44:13.129 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ASBtYPJjBFdJ9431OC+EuiPLRHo= 2023-11-02T18:44:13.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ASBtYPJjBFdJ9431OC+EuiPLRHo=', 'Content-Length': '0'} 2023-11-02T18:44:13.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000491b187cff5e5cb7-006543edfd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:44:13.134 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:44:13.134 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:44:13.135 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.135 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.136 DEBUG:boto:Method: GET 2023-11-02T18:44:13.136 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.137 DEBUG:boto:Data: 2023-11-02T18:44:13.137 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:13.138 DEBUG:boto:Port: 8001 2023-11-02T18:44:13.139 DEBUG:boto:Params: {} 2023-11-02T18:44:13.139 DEBUG:boto:Token: None 2023-11-02T18:44:13.139 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uyvXzzUFPUIbVdBuoYuc5s3rlwc= 2023-11-02T18:44:13.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uyvXzzUFPUIbVdBuoYuc5s3rlwc=', 'Content-Length': '0'} 2023-11-02T18:44:13.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9aef85f75b019e1-006543edfd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.145 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:44:13.145 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:44:13.146 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.146 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.147 DEBUG:boto:Method: GET 2023-11-02T18:44:13.147 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.147 DEBUG:boto:Data: 2023-11-02T18:44:13.148 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.149 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.149 DEBUG:boto:Params: {} 2023-11-02T18:44:13.150 DEBUG:boto:Token: None 2023-11-02T18:44:13.150 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:44:13.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uyvXzzUFPUIbVdBuoYuc5s3rlwc= 2023-11-02T18:44:13.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uyvXzzUFPUIbVdBuoYuc5s3rlwc=', 'Content-Length': '0'} 2023-11-02T18:44:13.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055f6958f87d108c0-006543edfd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.155 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:13.155 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:13.156 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:44:13.156 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:44:13.157 DEBUG:boto:Method: GET 2023-11-02T18:44:13.157 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:44:13.158 DEBUG:boto:Data: 2023-11-02T18:44:13.158 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.159 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:13.159 DEBUG:boto:Port: 8001 2023-11-02T18:44:13.159 DEBUG:boto:Params: {} 2023-11-02T18:44:13.160 DEBUG:boto:Token: None 2023-11-02T18:44:13.160 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/ 2023-11-02T18:44:13.161 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IO5SyFII+bwMxHtiNuy0Z0lW768= 2023-11-02T18:44:13.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IO5SyFII+bwMxHtiNuy0Z0lW768=', 'Content-Length': '0'} 2023-11-02T18:44:13.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c42461cad0923db6-006543edfd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.210 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:13.212 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:13.212 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:13.213 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:44:13.213 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:44:13.214 DEBUG:boto:Method: GET 2023-11-02T18:44:13.214 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:44:13.215 DEBUG:boto:Data: 2023-11-02T18:44:13.215 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.216 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.216 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.216 DEBUG:boto:Params: {} 2023-11-02T18:44:13.217 DEBUG:boto:Token: None 2023-11-02T18:44:13.217 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/ 2023-11-02T18:44:13.218 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IO5SyFII+bwMxHtiNuy0Z0lW768= 2023-11-02T18:44:13.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IO5SyFII+bwMxHtiNuy0Z0lW768=', 'Content-Length': '0'} 2023-11-02T18:44:13.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dd55443a9c935d9-006543edfd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.228 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:13.229 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:44:13.230 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:44:13.230 DEBUG:boto:Method: HEAD 2023-11-02T18:44:13.230 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:44:13.231 DEBUG:boto:Data: 2023-11-02T18:44:13.231 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.232 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:13.232 DEBUG:boto:Port: 8001 2023-11-02T18:44:13.233 DEBUG:boto:Params: {} 2023-11-02T18:44:13.233 DEBUG:boto:Token: None 2023-11-02T18:44:13.234 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj95 2023-11-02T18:44:13.234 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NajXkmWamPeE3EG5Y9Fn8KcFk8c= 2023-11-02T18:44:13.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NajXkmWamPeE3EG5Y9Fn8KcFk8c=', 'Content-Length': '0'} 2023-11-02T18:44:13.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009550c1e27c9157ac-006543edfd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.238 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:44:13.238 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:44:13.239 DEBUG:boto:Method: HEAD 2023-11-02T18:44:13.239 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:44:13.239 DEBUG:boto:Data: 2023-11-02T18:44:13.240 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.240 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.241 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.241 DEBUG:boto:Params: {} 2023-11-02T18:44:13.242 DEBUG:boto:Token: None 2023-11-02T18:44:13.242 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj95 2023-11-02T18:44:13.243 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NajXkmWamPeE3EG5Y9Fn8KcFk8c= 2023-11-02T18:44:13.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NajXkmWamPeE3EG5Y9Fn8KcFk8c=', 'Content-Length': '0'} 2023-11-02T18:44:13.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6617bc4da870255-006543edfd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:44:13.246 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:44:13.247 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:44:13.247 DEBUG:boto:Method: GET 2023-11-02T18:44:13.248 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:44:13.248 DEBUG:boto:Data: 2023-11-02T18:44:13.249 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.249 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:13.250 DEBUG:boto:Port: 8001 2023-11-02T18:44:13.250 DEBUG:boto:Params: {} 2023-11-02T18:44:13.251 DEBUG:boto:Token: None 2023-11-02T18:44:13.251 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj95 2023-11-02T18:44:13.251 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HVOfaTZTbWAig8bSkDOXcCCSb6Y= 2023-11-02T18:44:13.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HVOfaTZTbWAig8bSkDOXcCCSb6Y=', 'Content-Length': '0'} 2023-11-02T18:44:13.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d26c0e3c9e887b50-006543edfd-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.255 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:44:13.256 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:44:13.256 DEBUG:boto:Method: GET 2023-11-02T18:44:13.256 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:44:13.257 DEBUG:boto:Data: 2023-11-02T18:44:13.257 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.258 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.258 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.259 DEBUG:boto:Params: {} 2023-11-02T18:44:13.259 DEBUG:boto:Token: None 2023-11-02T18:44:13.260 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj95 2023-11-02T18:44:13.260 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HVOfaTZTbWAig8bSkDOXcCCSb6Y= 2023-11-02T18:44:13.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HVOfaTZTbWAig8bSkDOXcCCSb6Y=', 'Content-Length': '0'} 2023-11-02T18:44:13.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077642a47c498d385-006543edfd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.311 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:13.312 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:13.312 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:44:13.312 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:44:13.313 DEBUG:boto:Method: GET 2023-11-02T18:44:13.313 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:44:13.314 DEBUG:boto:Data: 2023-11-02T18:44:13.314 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.315 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:44:13.315 DEBUG:boto:Port: 8001 2023-11-02T18:44:13.315 DEBUG:boto:Params: {} 2023-11-02T18:44:13.316 DEBUG:boto:Token: None 2023-11-02T18:44:13.316 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/ 2023-11-02T18:44:13.317 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IO5SyFII+bwMxHtiNuy0Z0lW768= 2023-11-02T18:44:13.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IO5SyFII+bwMxHtiNuy0Z0lW768=', 'Content-Length': '0'} 2023-11-02T18:44:13.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fd923f1d22ba975-006543edfd-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.323 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:13.324 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:13.324 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:13.324 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:44:13.325 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:44:13.325 DEBUG:boto:Method: GET 2023-11-02T18:44:13.326 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:44:13.326 DEBUG:boto:Data: 2023-11-02T18:44:13.326 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.327 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.327 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.328 DEBUG:boto:Params: {} 2023-11-02T18:44:13.328 DEBUG:boto:Token: None 2023-11-02T18:44:13.329 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/ 2023-11-02T18:44:13.329 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IO5SyFII+bwMxHtiNuy0Z0lW768= 2023-11-02T18:44:13.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IO5SyFII+bwMxHtiNuy0Z0lW768=', 'Content-Length': '0'} 2023-11-02T18:44:13.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000186a39416a016c20-006543edfd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.339 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:44:13.339 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:44:13.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:44:13.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:44:13.341 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:44:13.385 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:44:13.385 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:44:13.477 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:44:13.478 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-59 2023-11-02T18:44:13.502 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 30 2023-11-02T18:44:13.636 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:13.637 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:13.637 DEBUG:boto:Method: HEAD 2023-11-02T18:44:13.638 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:44:13.638 DEBUG:boto:Data: 2023-11-02T18:44:13.639 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.640 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.640 DEBUG:boto:Params: {} 2023-11-02T18:44:13.640 DEBUG:boto:Token: None 2023-11-02T18:44:13.641 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/ 2023-11-02T18:44:13.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+T/I1DikhMOspOLMFYzwK7fYg+U= 2023-11-02T18:44:13.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+T/I1DikhMOspOLMFYzwK7fYg+U=', 'Content-Length': '0'} 2023-11-02T18:44:13.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d5fdf826d17b92f-006543edfd-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.649 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:44:13.649 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:44:13.649 DEBUG:boto:Method: PUT 2023-11-02T18:44:13.650 DEBUG:boto:Path: /ibtljn-59/obj37 2023-11-02T18:44:13.650 DEBUG:boto:Data: 2023-11-02T18:44:13.651 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:44:13.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.652 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.652 DEBUG:boto:Params: {} 2023-11-02T18:44:13.652 DEBUG:boto:Token: None 2023-11-02T18:44:13.653 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj37 2023-11-02T18:44:13.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:q0PEW8pQBpUu0I58QD1QYINbq2w= 2023-11-02T18:44:13.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:q0PEW8pQBpUu0I58QD1QYINbq2w='} 2023-11-02T18:44:13.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032cd5f71c735a45c-006543edfd-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.663 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:13.664 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:13.664 DEBUG:boto:Method: HEAD 2023-11-02T18:44:13.665 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:44:13.665 DEBUG:boto:Data: 2023-11-02T18:44:13.666 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.666 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.667 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.667 DEBUG:boto:Params: {} 2023-11-02T18:44:13.668 DEBUG:boto:Token: None 2023-11-02T18:44:13.668 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/ 2023-11-02T18:44:13.669 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+T/I1DikhMOspOLMFYzwK7fYg+U= 2023-11-02T18:44:13.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+T/I1DikhMOspOLMFYzwK7fYg+U=', 'Content-Length': '0'} 2023-11-02T18:44:13.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0ac50f71f4184fd-006543edfd-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.677 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:44:13.678 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:44:13.678 DEBUG:boto:Method: PUT 2023-11-02T18:44:13.678 DEBUG:boto:Path: /ibtljn-59/obj74 2023-11-02T18:44:13.679 DEBUG:boto:Data: 2023-11-02T18:44:13.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:44:13.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.680 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.680 DEBUG:boto:Params: {} 2023-11-02T18:44:13.681 DEBUG:boto:Token: None 2023-11-02T18:44:13.681 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj74 2023-11-02T18:44:13.682 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rvF71IZ+KgfErGnqMQ/CbRJc6n4= 2023-11-02T18:44:13.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rvF71IZ+KgfErGnqMQ/CbRJc6n4='} 2023-11-02T18:44:13.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b514224fa566b5d-006543edfd-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.731 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:13.732 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:13.732 DEBUG:boto:Method: HEAD 2023-11-02T18:44:13.733 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:44:13.733 DEBUG:boto:Data: 2023-11-02T18:44:13.733 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.734 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.734 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.735 DEBUG:boto:Params: {} 2023-11-02T18:44:13.735 DEBUG:boto:Token: None 2023-11-02T18:44:13.736 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/ 2023-11-02T18:44:13.736 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+T/I1DikhMOspOLMFYzwK7fYg+U= 2023-11-02T18:44:13.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+T/I1DikhMOspOLMFYzwK7fYg+U=', 'Content-Length': '0'} 2023-11-02T18:44:13.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9cf213513e073b0-006543edfd-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.745 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:44:13.745 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:44:13.746 DEBUG:boto:Method: PUT 2023-11-02T18:44:13.746 DEBUG:boto:Path: /ibtljn-59/obj111 2023-11-02T18:44:13.747 DEBUG:boto:Data: 2023-11-02T18:44:13.747 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:44:13.748 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.748 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.749 DEBUG:boto:Params: {} 2023-11-02T18:44:13.749 DEBUG:boto:Token: None 2023-11-02T18:44:13.750 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj111 2023-11-02T18:44:13.750 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E6adSywQDtcSClprBO3bWHjBlxI= 2023-11-02T18:44:13.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E6adSywQDtcSClprBO3bWHjBlxI='} 2023-11-02T18:44:13.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f50feb48fc042b9b-006543edfd-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.761 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:13.761 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:13.761 DEBUG:boto:Method: HEAD 2023-11-02T18:44:13.762 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:44:13.762 DEBUG:boto:Data: 2023-11-02T18:44:13.763 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.763 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.764 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.764 DEBUG:boto:Params: {} 2023-11-02T18:44:13.764 DEBUG:boto:Token: None 2023-11-02T18:44:13.765 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/ 2023-11-02T18:44:13.765 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+T/I1DikhMOspOLMFYzwK7fYg+U= 2023-11-02T18:44:13.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+T/I1DikhMOspOLMFYzwK7fYg+U=', 'Content-Length': '0'} 2023-11-02T18:44:13.773 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031e076d8c7b41c2f-006543edfd-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.774 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:44:13.774 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:44:13.775 DEBUG:boto:Method: PUT 2023-11-02T18:44:13.775 DEBUG:boto:Path: /ibtljn-59/obj148 2023-11-02T18:44:13.775 DEBUG:boto:Data: 2023-11-02T18:44:13.776 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:44:13.776 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.777 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.777 DEBUG:boto:Params: {} 2023-11-02T18:44:13.778 DEBUG:boto:Token: None 2023-11-02T18:44:13.778 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj148 2023-11-02T18:44:13.779 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zgyPPmPNjf9/BTCMEn0nTGCtFpg= 2023-11-02T18:44:13.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zgyPPmPNjf9/BTCMEn0nTGCtFpg='} 2023-11-02T18:44:13.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006745320a429d470f-006543edfd-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.823 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:44:13.824 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:44:13.824 DEBUG:boto:Method: HEAD 2023-11-02T18:44:13.825 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:44:13.825 DEBUG:boto:Data: 2023-11-02T18:44:13.825 DEBUG:boto:Headers: {} 2023-11-02T18:44:13.826 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.826 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.827 DEBUG:boto:Params: {} 2023-11-02T18:44:13.827 DEBUG:boto:Token: None 2023-11-02T18:44:13.828 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/ 2023-11-02T18:44:13.828 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+T/I1DikhMOspOLMFYzwK7fYg+U= 2023-11-02T18:44:13.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+T/I1DikhMOspOLMFYzwK7fYg+U=', 'Content-Length': '0'} 2023-11-02T18:44:13.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e22d0e6eef6c7192-006543edfd-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.837 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:44:13.838 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:44:13.838 DEBUG:boto:Method: PUT 2023-11-02T18:44:13.839 DEBUG:boto:Path: /ibtljn-59/obj185 2023-11-02T18:44:13.839 DEBUG:boto:Data: 2023-11-02T18:44:13.840 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:44:13.840 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:44:13.840 DEBUG:boto:Port: 8000 2023-11-02T18:44:13.841 DEBUG:boto:Params: {} 2023-11-02T18:44:13.841 DEBUG:boto:Token: None 2023-11-02T18:44:13.842 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:44:13 GMT /ibtljn-59/obj185 2023-11-02T18:44:13.842 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQuWBtOO8QNT///B0fyUQNCXFz4= 2023-11-02T18:44:13.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:44:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQuWBtOO8QNT///B0fyUQNCXFz4='} 2023-11-02T18:44:13.853 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b296a440b59e12a-006543edfd-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:44:13.854 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-59 2023-11-02T18:44:13.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:44:13.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:44:13.855 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:44:13.907 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:44:13.908 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:44:13.978 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:44:13.969+0000 7fa83abc8f40 1 waiting to reach latest gen 6, on 5.. 2023-11-02T18:44:18.979 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:44:18.969+0000 7fa83abc8f40 1 waiting to reach latest gen 6, on 5.. 2023-11-02T18:44:23.981 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:44:23.973+0000 7fa83abc8f40 1 waiting to reach latest gen 6, on 5.. 2023-11-02T18:44:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:44:28.981+0000 7fa83abc8f40 1 waiting for incremental sync to catch up: 2023-11-02T18:44:28.987 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.66.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:44:28.987 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:44:33.992 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:44:33.985+0000 7fa83abc8f40 1 waiting for incremental sync to catch up: 2023-11-02T18:44:33.993 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.66.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:44:33.993 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:44:38.998 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:44:38.989+0000 7fa83abc8f40 1 waiting for incremental sync to catch up: 2023-11-02T18:44:38.998 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.66.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:44:38.999 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:44:44.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:44:43.997+0000 7fa83abc8f40 1 waiting for incremental sync to catch up: 2023-11-02T18:44:44.004 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.66.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:44:44.005 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:44:49.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:44:49.001+0000 7fa83abc8f40 1 waiting for incremental sync to catch up: 2023-11-02T18:44:49.013 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.66.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:44:49.014 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:44:54.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:44:54.009+0000 7fa83abc8f40 1 waiting for incremental sync to catch up: 2023-11-02T18:44:54.017 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.66.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:44:54.017 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:44:59.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:44:59.012+0000 7fa83abc8f40 1 waiting for incremental sync to catch up: 2023-11-02T18:44:59.022 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.66.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:44:59.023 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:45:04.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:04.020+0000 7fa83abc8f40 1 waiting for incremental sync to catch up: 2023-11-02T18:45:04.028 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.66.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:45:04.028 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:45:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:09.024+0000 7fa83abc8f40 1 waiting for incremental sync to catch up: 2023-11-02T18:45:09.034 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.66.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:45:09.034 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:45:14.038 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:14.028+0000 7fa83abc8f40 1 waiting for incremental sync to catch up: 2023-11-02T18:45:14.039 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.66.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-11-02T18:45:14.039 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:45:19.045 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:19.036+0000 7fa83abc8f40 1 bucket sync caught up with source: 2023-11-02T18:45:19.046 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:45:19.046 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:45:19.046 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:19.036+0000 7fa83abc8f40 0 bucket checkpoint complete 2023-11-02T18:45:19.072 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-59 2023-11-02T18:45:19.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:45:19.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:45:19.074 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:45:19.125 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:19.126 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:19.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:19.188+0000 7f8df146df40 1 waiting for incremental sync to catch up: 2023-11-02T18:45:19.198 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000003.91.6, 00000000002.229.6, 00000000002.142.6, 00000000004.310.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000003.93.6, 00000000006.286.6, 00000000005.344.6] 2023-11-02T18:45:19.198 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000003.91.6, 00000000002.229.6, 00000000002.142.6, 00000000005.315.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000004.98.6, 00000000007.291.6, 00000000006.346.6] 2023-11-02T18:45:24.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:24.192+0000 7f8df146df40 1 bucket sync caught up with source: 2023-11-02T18:45:24.200 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000003.91.6, 00000000002.229.6, 00000000002.142.6, 00000000005.315.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000004.98.6, 00000000007.291.6, 00000000006.346.6] 2023-11-02T18:45:24.201 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000003.91.6, 00000000002.229.6, 00000000002.142.6, 00000000005.315.6, 00000000004.284.6, 00000000001.303.6, 00000000001.305.6, 00000000004.98.6, 00000000007.291.6, 00000000006.346.6] 2023-11-02T18:45:24.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:24.192+0000 7f8df146df40 0 bucket checkpoint complete 2023-11-02T18:45:24.231 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:45:24.231 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.232 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.232 DEBUG:boto:Method: HEAD 2023-11-02T18:45:24.233 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:45:24.233 DEBUG:boto:Data: 2023-11-02T18:45:24.234 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.234 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.235 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.235 DEBUG:boto:Params: {} 2023-11-02T18:45:24.236 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:45:24.237 DEBUG:boto:Token: None 2023-11-02T18:45:24.237 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/ 2023-11-02T18:45:24.238 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HU4pT152fqwa0hmMUOYaVevWQ7c= 2023-11-02T18:45:24.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HU4pT152fqwa0hmMUOYaVevWQ7c=', 'Content-Length': '0'} 2023-11-02T18:45:24.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000454b70b5227c80fe-006543ee44-1b8c-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.245 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.246 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.246 DEBUG:boto:Method: HEAD 2023-11-02T18:45:24.247 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:45:24.247 DEBUG:boto:Data: 2023-11-02T18:45:24.248 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.248 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.249 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.249 DEBUG:boto:Params: {} 2023-11-02T18:45:24.250 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:45:24.250 DEBUG:boto:Token: None 2023-11-02T18:45:24.251 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/ 2023-11-02T18:45:24.251 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HU4pT152fqwa0hmMUOYaVevWQ7c= 2023-11-02T18:45:24.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HU4pT152fqwa0hmMUOYaVevWQ7c=', 'Content-Length': '0'} 2023-11-02T18:45:24.261 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f041eec33056e55-006543ee44-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.262 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:45:24.262 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.263 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.263 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:24.264 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:24.264 DEBUG:boto:Method: GET 2023-11-02T18:45:24.265 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:24.265 DEBUG:boto:Data: 2023-11-02T18:45:24.266 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.266 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.267 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.267 DEBUG:boto:Params: {} 2023-11-02T18:45:24.268 DEBUG:boto:Token: None 2023-11-02T18:45:24.268 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/?versions 2023-11-02T18:45:24.269 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kZX8LryORuFQLsNq7JeYZJu+i7A= 2023-11-02T18:45:24.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kZX8LryORuFQLsNq7JeYZJu+i7A=', 'Content-Length': '0'} 2023-11-02T18:45:24.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4ea1b8217c24056-006543ee44-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.277 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:45:24.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-11-02T18:45:24.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T18:45:24.281 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-11-02T18:45:24.281 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-11-02T18:45:24.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-11-02T18:45:24.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-11-02T18:45:24.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-11-02T18:45:24.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:45:24.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-11-02T18:45:24.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:45:24.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:45:24.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T18:45:24.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-11-02T18:45:24.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-11-02T18:45:24.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:45:24.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:45:24.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-11-02T18:45:24.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:45:24.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:45:24.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T18:45:24.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:45:24.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:45:24.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:45:24.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-11-02T18:45:24.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-11-02T18:45:24.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:45:24.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T18:45:24.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-11-02T18:45:24.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:45:24.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:45:24.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-11-02T18:45:24.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T18:45:24.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-11-02T18:45:24.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:45:24.294 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:45:24.295 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.295 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.296 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:24.296 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:24.297 DEBUG:boto:Method: GET 2023-11-02T18:45:24.297 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:24.298 DEBUG:boto:Data: 2023-11-02T18:45:24.298 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.299 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.299 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.299 DEBUG:boto:Params: {} 2023-11-02T18:45:24.300 DEBUG:boto:Token: None 2023-11-02T18:45:24.301 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/?versions 2023-11-02T18:45:24.301 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kZX8LryORuFQLsNq7JeYZJu+i7A= 2023-11-02T18:45:24.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kZX8LryORuFQLsNq7JeYZJu+i7A=', 'Content-Length': '0'} 2023-11-02T18:45:24.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7761b17d8049ad8-006543ee44-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.311 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:45:24.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-11-02T18:45:24.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T18:45:24.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-11-02T18:45:24.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-11-02T18:45:24.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-11-02T18:45:24.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-11-02T18:45:24.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-11-02T18:45:24.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:45:24.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-11-02T18:45:24.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:45:24.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:45:24.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T18:45:24.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-11-02T18:45:24.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-11-02T18:45:24.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:45:24.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:45:24.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-11-02T18:45:24.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:45:24.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:45:24.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T18:45:24.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:45:24.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:45:24.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:45:24.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-11-02T18:45:24.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-11-02T18:45:24.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:45:24.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T18:45:24.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-11-02T18:45:24.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:45:24.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:45:24.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-11-02T18:45:24.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T18:45:24.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-11-02T18:45:24.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:45:24.329 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.329 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.330 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:24.330 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:24.331 DEBUG:boto:Method: GET 2023-11-02T18:45:24.331 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:24.332 DEBUG:boto:Data: 2023-11-02T18:45:24.332 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.333 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.333 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.333 DEBUG:boto:Params: {} 2023-11-02T18:45:24.334 DEBUG:boto:Token: None 2023-11-02T18:45:24.335 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/?versions 2023-11-02T18:45:24.335 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kZX8LryORuFQLsNq7JeYZJu+i7A= 2023-11-02T18:45:24.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kZX8LryORuFQLsNq7JeYZJu+i7A=', 'Content-Length': '0'} 2023-11-02T18:45:24.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef43759f2eb42484-006543ee44-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.342 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.344 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.344 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.345 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:24.345 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:24.346 DEBUG:boto:Method: GET 2023-11-02T18:45:24.346 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:24.346 DEBUG:boto:Data: 2023-11-02T18:45:24.347 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.347 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.348 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.348 DEBUG:boto:Params: {} 2023-11-02T18:45:24.349 DEBUG:boto:Token: None 2023-11-02T18:45:24.349 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/?versions 2023-11-02T18:45:24.350 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kZX8LryORuFQLsNq7JeYZJu+i7A= 2023-11-02T18:45:24.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kZX8LryORuFQLsNq7JeYZJu+i7A=', 'Content-Length': '0'} 2023-11-02T18:45:24.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000835924859577315c-006543ee44-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.358 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:45:24.361 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:24.362 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:24.362 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.363 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.363 DEBUG:boto:Method: GET 2023-11-02T18:45:24.363 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.364 DEBUG:boto:Data: 2023-11-02T18:45:24.364 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.365 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.365 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.366 DEBUG:boto:Params: {} 2023-11-02T18:45:24.366 DEBUG:boto:Token: None 2023-11-02T18:45:24.367 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.367 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ruw1zOxhlfhzsSaOAYzSn7YRux0= 2023-11-02T18:45:24.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ruw1zOxhlfhzsSaOAYzSn7YRux0=', 'Content-Length': '0'} 2023-11-02T18:45:24.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c52368081a94a68-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.372 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:24.372 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:24.372 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.373 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.373 DEBUG:boto:Method: GET 2023-11-02T18:45:24.374 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.374 DEBUG:boto:Data: 2023-11-02T18:45:24.375 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.375 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.376 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.376 DEBUG:boto:Params: {} 2023-11-02T18:45:24.377 DEBUG:boto:Token: None 2023-11-02T18:45:24.377 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.378 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ruw1zOxhlfhzsSaOAYzSn7YRux0= 2023-11-02T18:45:24.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ruw1zOxhlfhzsSaOAYzSn7YRux0=', 'Content-Length': '0'} 2023-11-02T18:45:24.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a297c90c2bc3f56f-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.382 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:24.382 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:24.383 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.383 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.383 DEBUG:boto:Method: HEAD 2023-11-02T18:45:24.384 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.384 DEBUG:boto:Data: 2023-11-02T18:45:24.385 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.385 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.386 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.386 DEBUG:boto:Params: {} 2023-11-02T18:45:24.387 DEBUG:boto:Token: None 2023-11-02T18:45:24.387 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.388 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LBtPpycEkPltKLav248Gwvu1YS0= 2023-11-02T18:45:24.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LBtPpycEkPltKLav248Gwvu1YS0=', 'Content-Length': '0'} 2023-11-02T18:45:24.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000327a585bb2951405-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.391 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:24.392 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:24.392 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.393 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.393 DEBUG:boto:Method: HEAD 2023-11-02T18:45:24.394 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.394 DEBUG:boto:Data: 2023-11-02T18:45:24.394 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.395 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.395 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.396 DEBUG:boto:Params: {} 2023-11-02T18:45:24.396 DEBUG:boto:Token: None 2023-11-02T18:45:24.397 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.397 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LBtPpycEkPltKLav248Gwvu1YS0= 2023-11-02T18:45:24.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LBtPpycEkPltKLav248Gwvu1YS0=', 'Content-Length': '0'} 2023-11-02T18:45:24.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c30e0b1a11c0dc14-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:45:24.402 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:24.402 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:24.403 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.403 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.404 DEBUG:boto:Method: GET 2023-11-02T18:45:24.404 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.405 DEBUG:boto:Data: 2023-11-02T18:45:24.405 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.405 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.406 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.406 DEBUG:boto:Params: {} 2023-11-02T18:45:24.407 DEBUG:boto:Token: None 2023-11-02T18:45:24.407 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.408 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ruw1zOxhlfhzsSaOAYzSn7YRux0= 2023-11-02T18:45:24.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ruw1zOxhlfhzsSaOAYzSn7YRux0=', 'Content-Length': '0'} 2023-11-02T18:45:24.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e30b3da261f3a77-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.458 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:24.459 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:24.460 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.460 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.460 DEBUG:boto:Method: GET 2023-11-02T18:45:24.461 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.461 DEBUG:boto:Data: 2023-11-02T18:45:24.462 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.463 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.463 DEBUG:boto:Params: {} 2023-11-02T18:45:24.464 DEBUG:boto:Token: None 2023-11-02T18:45:24.464 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:24.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ruw1zOxhlfhzsSaOAYzSn7YRux0= 2023-11-02T18:45:24.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ruw1zOxhlfhzsSaOAYzSn7YRux0=', 'Content-Length': '0'} 2023-11-02T18:45:24.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e9c5b57dbf1d714-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.469 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.469 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.470 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:24.471 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:24.471 DEBUG:boto:Method: GET 2023-11-02T18:45:24.472 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:45:24.472 DEBUG:boto:Data: 2023-11-02T18:45:24.473 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.473 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.474 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.474 DEBUG:boto:Params: {} 2023-11-02T18:45:24.475 DEBUG:boto:Token: None 2023-11-02T18:45:24.475 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/ 2023-11-02T18:45:24.476 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk= 2023-11-02T18:45:24.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk=', 'Content-Length': '0'} 2023-11-02T18:45:24.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000932f1464a30db510-006543ee44-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.482 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.483 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.484 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.484 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:24.484 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:24.485 DEBUG:boto:Method: GET 2023-11-02T18:45:24.485 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:45:24.486 DEBUG:boto:Data: 2023-11-02T18:45:24.486 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.487 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.487 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.487 DEBUG:boto:Params: {} 2023-11-02T18:45:24.488 DEBUG:boto:Token: None 2023-11-02T18:45:24.488 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/ 2023-11-02T18:45:24.489 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk= 2023-11-02T18:45:24.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk=', 'Content-Length': '0'} 2023-11-02T18:45:24.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006edec7570c68de1d-006543ee44-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.499 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.500 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:24.501 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:24.501 DEBUG:boto:Method: HEAD 2023-11-02T18:45:24.501 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:45:24.503 DEBUG:boto:Data: 2023-11-02T18:45:24.503 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.504 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.504 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.505 DEBUG:boto:Params: {} 2023-11-02T18:45:24.505 DEBUG:boto:Token: None 2023-11-02T18:45:24.506 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj11 2023-11-02T18:45:24.506 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KSezDuSqVtBLu6LnAkvYWagyuWo= 2023-11-02T18:45:24.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KSezDuSqVtBLu6LnAkvYWagyuWo=', 'Content-Length': '0'} 2023-11-02T18:45:24.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc69c95998a11710-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.510 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:24.510 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:24.511 DEBUG:boto:Method: HEAD 2023-11-02T18:45:24.511 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:45:24.512 DEBUG:boto:Data: 2023-11-02T18:45:24.512 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.513 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.513 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.514 DEBUG:boto:Params: {} 2023-11-02T18:45:24.514 DEBUG:boto:Token: None 2023-11-02T18:45:24.514 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj11 2023-11-02T18:45:24.515 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KSezDuSqVtBLu6LnAkvYWagyuWo= 2023-11-02T18:45:24.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KSezDuSqVtBLu6LnAkvYWagyuWo=', 'Content-Length': '0'} 2023-11-02T18:45:24.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099a259adfcd45f6d-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:45:24.519 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:24.520 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:24.520 DEBUG:boto:Method: GET 2023-11-02T18:45:24.521 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:45:24.521 DEBUG:boto:Data: 2023-11-02T18:45:24.522 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.523 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.523 DEBUG:boto:Params: {} 2023-11-02T18:45:24.524 DEBUG:boto:Token: None 2023-11-02T18:45:24.524 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj11 2023-11-02T18:45:24.525 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dBDwTo1oqRvN+v7iAtk9hQ1Ziy4= 2023-11-02T18:45:24.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dBDwTo1oqRvN+v7iAtk9hQ1Ziy4=', 'Content-Length': '0'} 2023-11-02T18:45:24.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002edbec2a156c7580-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.529 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:24.529 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:24.530 DEBUG:boto:Method: GET 2023-11-02T18:45:24.530 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:45:24.531 DEBUG:boto:Data: 2023-11-02T18:45:24.531 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.531 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.532 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.532 DEBUG:boto:Params: {} 2023-11-02T18:45:24.533 DEBUG:boto:Token: None 2023-11-02T18:45:24.533 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj11 2023-11-02T18:45:24.534 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dBDwTo1oqRvN+v7iAtk9hQ1Ziy4= 2023-11-02T18:45:24.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dBDwTo1oqRvN+v7iAtk9hQ1Ziy4=', 'Content-Length': '0'} 2023-11-02T18:45:24.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2da633d93ec80a2-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.582 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.583 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.583 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:24.584 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:24.584 DEBUG:boto:Method: GET 2023-11-02T18:45:24.585 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:45:24.585 DEBUG:boto:Data: 2023-11-02T18:45:24.586 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.586 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.587 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.587 DEBUG:boto:Params: {} 2023-11-02T18:45:24.588 DEBUG:boto:Token: None 2023-11-02T18:45:24.588 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/ 2023-11-02T18:45:24.589 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk= 2023-11-02T18:45:24.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk=', 'Content-Length': '0'} 2023-11-02T18:45:24.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000a6828be4ef6123-006543ee44-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.595 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.596 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.596 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.597 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:24.597 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:24.598 DEBUG:boto:Method: GET 2023-11-02T18:45:24.598 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:45:24.599 DEBUG:boto:Data: 2023-11-02T18:45:24.599 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.600 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.601 DEBUG:boto:Params: {} 2023-11-02T18:45:24.601 DEBUG:boto:Token: None 2023-11-02T18:45:24.602 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/ 2023-11-02T18:45:24.602 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk= 2023-11-02T18:45:24.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk=', 'Content-Length': '0'} 2023-11-02T18:45:24.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bf5a410c7f67e3e-006543ee44-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.614 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-11-02T18:45:24.615 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:24.616 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:24.616 DEBUG:boto:path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.617 DEBUG:boto:auth_path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.617 DEBUG:boto:Method: GET 2023-11-02T18:45:24.618 DEBUG:boto:Path: /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.618 DEBUG:boto:Data: 2023-11-02T18:45:24.619 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.620 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.620 DEBUG:boto:Params: {} 2023-11-02T18:45:24.621 DEBUG:boto:Token: None 2023-11-02T18:45:24.622 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.622 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V3fmDPvp2D/l2fcALuA1Mni2Mp0= 2023-11-02T18:45:24.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V3fmDPvp2D/l2fcALuA1Mni2Mp0=', 'Content-Length': '0'} 2023-11-02T18:45:24.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fac0d430a8563c82-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.670 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:24.671 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:24.672 DEBUG:boto:path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.672 DEBUG:boto:auth_path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.673 DEBUG:boto:Method: GET 2023-11-02T18:45:24.673 DEBUG:boto:Path: /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.674 DEBUG:boto:Data: 2023-11-02T18:45:24.674 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.675 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.675 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.676 DEBUG:boto:Params: {} 2023-11-02T18:45:24.676 DEBUG:boto:Token: None 2023-11-02T18:45:24.677 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.677 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V3fmDPvp2D/l2fcALuA1Mni2Mp0= 2023-11-02T18:45:24.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V3fmDPvp2D/l2fcALuA1Mni2Mp0=', 'Content-Length': '0'} 2023-11-02T18:45:24.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006ef853612676c5f-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.682 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:24.682 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:24.683 DEBUG:boto:path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.686 DEBUG:boto:auth_path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.686 DEBUG:boto:Method: HEAD 2023-11-02T18:45:24.687 DEBUG:boto:Path: /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.687 DEBUG:boto:Data: 2023-11-02T18:45:24.688 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.688 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.688 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.689 DEBUG:boto:Params: {} 2023-11-02T18:45:24.689 DEBUG:boto:Token: None 2023-11-02T18:45:24.690 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.690 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QS2yt5Y+dCV9SDMHtyDSPmM694M= 2023-11-02T18:45:24.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QS2yt5Y+dCV9SDMHtyDSPmM694M=', 'Content-Length': '0'} 2023-11-02T18:45:24.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbdf3558c6d568bf-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.694 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:24.695 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:24.695 DEBUG:boto:path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.696 DEBUG:boto:auth_path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.696 DEBUG:boto:Method: HEAD 2023-11-02T18:45:24.696 DEBUG:boto:Path: /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.697 DEBUG:boto:Data: 2023-11-02T18:45:24.697 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.698 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.698 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.699 DEBUG:boto:Params: {} 2023-11-02T18:45:24.699 DEBUG:boto:Token: None 2023-11-02T18:45:24.700 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.700 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QS2yt5Y+dCV9SDMHtyDSPmM694M= 2023-11-02T18:45:24.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QS2yt5Y+dCV9SDMHtyDSPmM694M=', 'Content-Length': '0'} 2023-11-02T18:45:24.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061755614b6f90ba2-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-11-02T18:45:24.705 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:24.705 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:24.706 DEBUG:boto:path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.706 DEBUG:boto:auth_path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.707 DEBUG:boto:Method: GET 2023-11-02T18:45:24.707 DEBUG:boto:Path: /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.707 DEBUG:boto:Data: 2023-11-02T18:45:24.708 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.708 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.709 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.709 DEBUG:boto:Params: {} 2023-11-02T18:45:24.710 DEBUG:boto:Token: None 2023-11-02T18:45:24.710 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.711 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V3fmDPvp2D/l2fcALuA1Mni2Mp0= 2023-11-02T18:45:24.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V3fmDPvp2D/l2fcALuA1Mni2Mp0=', 'Content-Length': '0'} 2023-11-02T18:45:24.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a342c1585a3ea016-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.715 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:24.716 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:24.716 DEBUG:boto:path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.716 DEBUG:boto:auth_path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.717 DEBUG:boto:Method: GET 2023-11-02T18:45:24.717 DEBUG:boto:Path: /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.718 DEBUG:boto:Data: 2023-11-02T18:45:24.718 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.719 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.719 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.720 DEBUG:boto:Params: {} 2023-11-02T18:45:24.720 DEBUG:boto:Token: None 2023-11-02T18:45:24.721 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:24.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V3fmDPvp2D/l2fcALuA1Mni2Mp0= 2023-11-02T18:45:24.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V3fmDPvp2D/l2fcALuA1Mni2Mp0=', 'Content-Length': '0'} 2023-11-02T18:45:24.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cab0d84995cf8224-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.725 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.726 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.726 DEBUG:boto:path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:24.727 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:24.727 DEBUG:boto:Method: GET 2023-11-02T18:45:24.728 DEBUG:boto:Path: /ibtljn-59/?prefix=obj111 2023-11-02T18:45:24.728 DEBUG:boto:Data: 2023-11-02T18:45:24.729 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.729 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.729 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.730 DEBUG:boto:Params: {} 2023-11-02T18:45:24.730 DEBUG:boto:Token: None 2023-11-02T18:45:24.731 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/ 2023-11-02T18:45:24.731 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk= 2023-11-02T18:45:24.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk=', 'Content-Length': '0'} 2023-11-02T18:45:24.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a527cf1d1ca5dca-006543ee44-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.738 DEBUG:boto:b'ibtljn-59obj1111000falseobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.739 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.739 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.740 DEBUG:boto:path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:24.740 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:24.740 DEBUG:boto:Method: GET 2023-11-02T18:45:24.741 DEBUG:boto:Path: /ibtljn-59/?prefix=obj111 2023-11-02T18:45:24.741 DEBUG:boto:Data: 2023-11-02T18:45:24.742 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.742 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.743 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.743 DEBUG:boto:Params: {} 2023-11-02T18:45:24.744 DEBUG:boto:Token: None 2023-11-02T18:45:24.744 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/ 2023-11-02T18:45:24.745 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk= 2023-11-02T18:45:24.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk=', 'Content-Length': '0'} 2023-11-02T18:45:24.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6d7214ff03f6074-006543ee44-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.755 DEBUG:boto:b'ibtljn-59obj1111000falseobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.756 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:24.756 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:24.757 DEBUG:boto:Method: HEAD 2023-11-02T18:45:24.757 DEBUG:boto:Path: /ibtljn-59/obj111 2023-11-02T18:45:24.758 DEBUG:boto:Data: 2023-11-02T18:45:24.758 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.759 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.759 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.759 DEBUG:boto:Params: {} 2023-11-02T18:45:24.760 DEBUG:boto:Token: None 2023-11-02T18:45:24.760 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj111 2023-11-02T18:45:24.761 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aCpW6L1Z4WIL7QN3WbGxRFEVVCI= 2023-11-02T18:45:24.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aCpW6L1Z4WIL7QN3WbGxRFEVVCI=', 'Content-Length': '0'} 2023-11-02T18:45:24.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359ec4a43e85bf59-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.765 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:24.765 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:24.766 DEBUG:boto:Method: HEAD 2023-11-02T18:45:24.766 DEBUG:boto:Path: /ibtljn-59/obj111 2023-11-02T18:45:24.767 DEBUG:boto:Data: 2023-11-02T18:45:24.767 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.768 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.768 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.768 DEBUG:boto:Params: {} 2023-11-02T18:45:24.769 DEBUG:boto:Token: None 2023-11-02T18:45:24.769 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj111 2023-11-02T18:45:24.770 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aCpW6L1Z4WIL7QN3WbGxRFEVVCI= 2023-11-02T18:45:24.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aCpW6L1Z4WIL7QN3WbGxRFEVVCI=', 'Content-Length': '0'} 2023-11-02T18:45:24.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024ae4af64abe3607-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-11-02T18:45:24.774 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:24.775 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:24.775 DEBUG:boto:Method: GET 2023-11-02T18:45:24.775 DEBUG:boto:Path: /ibtljn-59/obj111 2023-11-02T18:45:24.776 DEBUG:boto:Data: 2023-11-02T18:45:24.776 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.777 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.777 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.778 DEBUG:boto:Params: {} 2023-11-02T18:45:24.778 DEBUG:boto:Token: None 2023-11-02T18:45:24.779 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj111 2023-11-02T18:45:24.779 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1pDU9kZhqPy/sLnMOhpuGEPxIUA= 2023-11-02T18:45:24.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1pDU9kZhqPy/sLnMOhpuGEPxIUA=', 'Content-Length': '0'} 2023-11-02T18:45:24.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e1be18458764dec-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.826 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:24.827 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:24.827 DEBUG:boto:Method: GET 2023-11-02T18:45:24.828 DEBUG:boto:Path: /ibtljn-59/obj111 2023-11-02T18:45:24.828 DEBUG:boto:Data: 2023-11-02T18:45:24.829 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.829 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.830 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.830 DEBUG:boto:Params: {} 2023-11-02T18:45:24.831 DEBUG:boto:Token: None 2023-11-02T18:45:24.831 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj111 2023-11-02T18:45:24.832 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1pDU9kZhqPy/sLnMOhpuGEPxIUA= 2023-11-02T18:45:24.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1pDU9kZhqPy/sLnMOhpuGEPxIUA=', 'Content-Length': '0'} 2023-11-02T18:45:24.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1541f1eae415394-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.836 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.836 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.837 DEBUG:boto:path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:24.837 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:24.838 DEBUG:boto:Method: GET 2023-11-02T18:45:24.838 DEBUG:boto:Path: /ibtljn-59/?prefix=obj111 2023-11-02T18:45:24.839 DEBUG:boto:Data: 2023-11-02T18:45:24.839 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.839 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.840 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.840 DEBUG:boto:Params: {} 2023-11-02T18:45:24.841 DEBUG:boto:Token: None 2023-11-02T18:45:24.841 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/ 2023-11-02T18:45:24.842 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk= 2023-11-02T18:45:24.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk=', 'Content-Length': '0'} 2023-11-02T18:45:24.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7a473f65ce3c460-006543ee44-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.848 DEBUG:boto:b'ibtljn-59obj1111000falseobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.849 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.849 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.850 DEBUG:boto:path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:24.850 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:24.851 DEBUG:boto:Method: GET 2023-11-02T18:45:24.851 DEBUG:boto:Path: /ibtljn-59/?prefix=obj111 2023-11-02T18:45:24.851 DEBUG:boto:Data: 2023-11-02T18:45:24.852 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.853 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.853 DEBUG:boto:Params: {} 2023-11-02T18:45:24.854 DEBUG:boto:Token: None 2023-11-02T18:45:24.854 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/ 2023-11-02T18:45:24.855 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk= 2023-11-02T18:45:24.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk=', 'Content-Length': '0'} 2023-11-02T18:45:24.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003caa9f8558f42e9d-006543ee44-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.865 DEBUG:boto:b'ibtljn-59obj1111000falseobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:45:24.867 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:24.867 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:24.868 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.868 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.868 DEBUG:boto:Method: GET 2023-11-02T18:45:24.869 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.869 DEBUG:boto:Data: 2023-11-02T18:45:24.870 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.871 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.871 DEBUG:boto:Params: {} 2023-11-02T18:45:24.872 DEBUG:boto:Token: None 2023-11-02T18:45:24.872 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.873 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zx2p3lmLDDllNd4lB9Nri9uJ3bk= 2023-11-02T18:45:24.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zx2p3lmLDDllNd4lB9Nri9uJ3bk=', 'Content-Length': '0'} 2023-11-02T18:45:24.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcd1330a219af5ff-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.876 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:24.877 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:24.877 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.878 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.878 DEBUG:boto:Method: GET 2023-11-02T18:45:24.879 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.879 DEBUG:boto:Data: 2023-11-02T18:45:24.879 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.880 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.880 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.881 DEBUG:boto:Params: {} 2023-11-02T18:45:24.881 DEBUG:boto:Token: None 2023-11-02T18:45:24.882 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.882 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zx2p3lmLDDllNd4lB9Nri9uJ3bk= 2023-11-02T18:45:24.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zx2p3lmLDDllNd4lB9Nri9uJ3bk=', 'Content-Length': '0'} 2023-11-02T18:45:24.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006685dad3a1ad2d0a-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.886 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:24.886 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:24.889 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.889 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.890 DEBUG:boto:Method: HEAD 2023-11-02T18:45:24.890 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.891 DEBUG:boto:Data: 2023-11-02T18:45:24.891 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.891 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.892 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.892 DEBUG:boto:Params: {} 2023-11-02T18:45:24.893 DEBUG:boto:Token: None 2023-11-02T18:45:24.893 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.894 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:11yJRjAXCLEIhOLmeR5yYv6TxrQ= 2023-11-02T18:45:24.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:11yJRjAXCLEIhOLmeR5yYv6TxrQ=', 'Content-Length': '0'} 2023-11-02T18:45:24.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a39dffcd25948d46-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.898 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:24.898 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:24.899 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.899 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.899 DEBUG:boto:Method: HEAD 2023-11-02T18:45:24.900 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.900 DEBUG:boto:Data: 2023-11-02T18:45:24.901 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.901 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.902 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.902 DEBUG:boto:Params: {} 2023-11-02T18:45:24.902 DEBUG:boto:Token: None 2023-11-02T18:45:24.903 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.903 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:11yJRjAXCLEIhOLmeR5yYv6TxrQ= 2023-11-02T18:45:24.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:11yJRjAXCLEIhOLmeR5yYv6TxrQ=', 'Content-Length': '0'} 2023-11-02T18:45:24.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3930a26a6b56803-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:45:24.908 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:24.908 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:24.909 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.909 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.910 DEBUG:boto:Method: GET 2023-11-02T18:45:24.910 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.911 DEBUG:boto:Data: 2023-11-02T18:45:24.911 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.912 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.912 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.912 DEBUG:boto:Params: {} 2023-11-02T18:45:24.913 DEBUG:boto:Token: None 2023-11-02T18:45:24.913 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.914 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zx2p3lmLDDllNd4lB9Nri9uJ3bk= 2023-11-02T18:45:24.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zx2p3lmLDDllNd4lB9Nri9uJ3bk=', 'Content-Length': '0'} 2023-11-02T18:45:24.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e166755c95b936e-006543ee44-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.958 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:24.959 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:24.959 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.960 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.960 DEBUG:boto:Method: GET 2023-11-02T18:45:24.961 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.961 DEBUG:boto:Data: 2023-11-02T18:45:24.962 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.962 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.963 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.963 DEBUG:boto:Params: {} 2023-11-02T18:45:24.964 DEBUG:boto:Token: None 2023-11-02T18:45:24.964 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:24.965 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zx2p3lmLDDllNd4lB9Nri9uJ3bk= 2023-11-02T18:45:24.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zx2p3lmLDDllNd4lB9Nri9uJ3bk=', 'Content-Length': '0'} 2023-11-02T18:45:24.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2ff674eafb4711b-006543ee44-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.969 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.969 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.970 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:24.970 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:24.971 DEBUG:boto:Method: GET 2023-11-02T18:45:24.971 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:45:24.972 DEBUG:boto:Data: 2023-11-02T18:45:24.972 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.972 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:24.973 DEBUG:boto:Port: 8001 2023-11-02T18:45:24.973 DEBUG:boto:Params: {} 2023-11-02T18:45:24.974 DEBUG:boto:Token: None 2023-11-02T18:45:24.974 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/ 2023-11-02T18:45:24.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk= 2023-11-02T18:45:24.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk=', 'Content-Length': '0'} 2023-11-02T18:45:24.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a726c6e32924d68-006543ee44-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:24.981 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:24.982 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:24.982 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:24.983 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:24.983 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:24.983 DEBUG:boto:Method: GET 2023-11-02T18:45:24.984 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:45:24.984 DEBUG:boto:Data: 2023-11-02T18:45:24.985 DEBUG:boto:Headers: {} 2023-11-02T18:45:24.985 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:24.986 DEBUG:boto:Port: 8000 2023-11-02T18:45:24.986 DEBUG:boto:Params: {} 2023-11-02T18:45:24.987 DEBUG:boto:Token: None 2023-11-02T18:45:24.987 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:24 GMT /ibtljn-59/ 2023-11-02T18:45:24.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk= 2023-11-02T18:45:24.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tKZC8OF+t25R9XV59S8QnzF9vrk=', 'Content-Length': '0'} 2023-11-02T18:45:24.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f6e37657c3762ea-006543ee44-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.038 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.039 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:25.040 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:25.040 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.040 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:45:25.041 DEBUG:boto:Data: 2023-11-02T18:45:25.041 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.042 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.042 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.042 DEBUG:boto:Params: {} 2023-11-02T18:45:25.043 DEBUG:boto:Token: None 2023-11-02T18:45:25.043 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj115 2023-11-02T18:45:25.044 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WuUm96mhpyMwbkfURt8ma3efXxo= 2023-11-02T18:45:25.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WuUm96mhpyMwbkfURt8ma3efXxo=', 'Content-Length': '0'} 2023-11-02T18:45:25.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d4b46a7dcebba71-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.048 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:25.048 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:25.049 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.049 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:45:25.050 DEBUG:boto:Data: 2023-11-02T18:45:25.050 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.051 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.051 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.051 DEBUG:boto:Params: {} 2023-11-02T18:45:25.052 DEBUG:boto:Token: None 2023-11-02T18:45:25.052 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj115 2023-11-02T18:45:25.053 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WuUm96mhpyMwbkfURt8ma3efXxo= 2023-11-02T18:45:25.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WuUm96mhpyMwbkfURt8ma3efXxo=', 'Content-Length': '0'} 2023-11-02T18:45:25.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5dcb6caa359d17a-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:45:25.057 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:25.058 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:25.059 DEBUG:boto:Method: GET 2023-11-02T18:45:25.059 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:45:25.059 DEBUG:boto:Data: 2023-11-02T18:45:25.060 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.060 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.061 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.061 DEBUG:boto:Params: {} 2023-11-02T18:45:25.062 DEBUG:boto:Token: None 2023-11-02T18:45:25.062 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj115 2023-11-02T18:45:25.063 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J431IZ3nTRlPVbmDDvu5TMttQjk= 2023-11-02T18:45:25.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J431IZ3nTRlPVbmDDvu5TMttQjk=', 'Content-Length': '0'} 2023-11-02T18:45:25.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098e4e5ac53b411bc-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.067 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:25.067 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:25.068 DEBUG:boto:Method: GET 2023-11-02T18:45:25.068 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:45:25.069 DEBUG:boto:Data: 2023-11-02T18:45:25.069 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.070 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.070 DEBUG:boto:Params: {} 2023-11-02T18:45:25.071 DEBUG:boto:Token: None 2023-11-02T18:45:25.071 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj115 2023-11-02T18:45:25.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J431IZ3nTRlPVbmDDvu5TMttQjk= 2023-11-02T18:45:25.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J431IZ3nTRlPVbmDDvu5TMttQjk=', 'Content-Length': '0'} 2023-11-02T18:45:25.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a43c2eaa455aa0e-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.076 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.076 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.077 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:25.077 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:25.078 DEBUG:boto:Method: GET 2023-11-02T18:45:25.078 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:45:25.079 DEBUG:boto:Data: 2023-11-02T18:45:25.079 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.079 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.080 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.080 DEBUG:boto:Params: {} 2023-11-02T18:45:25.081 DEBUG:boto:Token: None 2023-11-02T18:45:25.081 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.082 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000308511c09e8d5835-006543ee45-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.134 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.135 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.136 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.136 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:25.137 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:25.137 DEBUG:boto:Method: GET 2023-11-02T18:45:25.137 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:45:25.138 DEBUG:boto:Data: 2023-11-02T18:45:25.138 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.139 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.140 DEBUG:boto:Params: {} 2023-11-02T18:45:25.140 DEBUG:boto:Token: None 2023-11-02T18:45:25.141 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043909bd7078d5d0c-006543ee45-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.153 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:45:25.154 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:25.154 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:25.155 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.155 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.156 DEBUG:boto:Method: GET 2023-11-02T18:45:25.156 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.157 DEBUG:boto:Data: 2023-11-02T18:45:25.157 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.158 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.158 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.158 DEBUG:boto:Params: {} 2023-11-02T18:45:25.159 DEBUG:boto:Token: None 2023-11-02T18:45:25.160 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.160 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pdmx/XMvjXW2cLqJ/8wSL6z6JIE= 2023-11-02T18:45:25.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pdmx/XMvjXW2cLqJ/8wSL6z6JIE=', 'Content-Length': '0'} 2023-11-02T18:45:25.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c49e6129e19b02a-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.164 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:25.165 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:25.165 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.165 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.166 DEBUG:boto:Method: GET 2023-11-02T18:45:25.166 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.167 DEBUG:boto:Data: 2023-11-02T18:45:25.167 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.168 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.168 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.169 DEBUG:boto:Params: {} 2023-11-02T18:45:25.169 DEBUG:boto:Token: None 2023-11-02T18:45:25.170 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.170 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pdmx/XMvjXW2cLqJ/8wSL6z6JIE= 2023-11-02T18:45:25.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pdmx/XMvjXW2cLqJ/8wSL6z6JIE=', 'Content-Length': '0'} 2023-11-02T18:45:25.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb360c0a45ccae59-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.218 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:25.219 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:25.219 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.220 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.220 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.221 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.222 DEBUG:boto:Data: 2023-11-02T18:45:25.222 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.223 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.223 DEBUG:boto:Params: {} 2023-11-02T18:45:25.224 DEBUG:boto:Token: None 2023-11-02T18:45:25.224 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wdZTrIP/CBAQYGvnANKUtPv6M1A= 2023-11-02T18:45:25.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wdZTrIP/CBAQYGvnANKUtPv6M1A=', 'Content-Length': '0'} 2023-11-02T18:45:25.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d97fa486b01a139c-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.229 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:25.230 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:25.230 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.231 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.231 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.231 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.232 DEBUG:boto:Data: 2023-11-02T18:45:25.232 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.232 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.233 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.233 DEBUG:boto:Params: {} 2023-11-02T18:45:25.234 DEBUG:boto:Token: None 2023-11-02T18:45:25.234 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.235 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wdZTrIP/CBAQYGvnANKUtPv6M1A= 2023-11-02T18:45:25.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wdZTrIP/CBAQYGvnANKUtPv6M1A=', 'Content-Length': '0'} 2023-11-02T18:45:25.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f3165c22b5b07e7-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:45:25.239 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:25.240 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:25.240 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.240 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.241 DEBUG:boto:Method: GET 2023-11-02T18:45:25.241 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.242 DEBUG:boto:Data: 2023-11-02T18:45:25.242 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.243 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.243 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.244 DEBUG:boto:Params: {} 2023-11-02T18:45:25.244 DEBUG:boto:Token: None 2023-11-02T18:45:25.245 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.245 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pdmx/XMvjXW2cLqJ/8wSL6z6JIE= 2023-11-02T18:45:25.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pdmx/XMvjXW2cLqJ/8wSL6z6JIE=', 'Content-Length': '0'} 2023-11-02T18:45:25.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022bae4cc2d1b2de3-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.249 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:25.250 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:25.250 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.251 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.252 DEBUG:boto:Method: GET 2023-11-02T18:45:25.252 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.252 DEBUG:boto:Data: 2023-11-02T18:45:25.253 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.253 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.254 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.254 DEBUG:boto:Params: {} 2023-11-02T18:45:25.255 DEBUG:boto:Token: None 2023-11-02T18:45:25.255 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:25.256 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pdmx/XMvjXW2cLqJ/8wSL6z6JIE= 2023-11-02T18:45:25.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pdmx/XMvjXW2cLqJ/8wSL6z6JIE=', 'Content-Length': '0'} 2023-11-02T18:45:25.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8c13cb2a9ed9d62-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.260 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.261 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.261 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:25.261 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:25.262 DEBUG:boto:Method: GET 2023-11-02T18:45:25.262 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:45:25.263 DEBUG:boto:Data: 2023-11-02T18:45:25.263 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.264 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.264 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.265 DEBUG:boto:Params: {} 2023-11-02T18:45:25.265 DEBUG:boto:Token: None 2023-11-02T18:45:25.266 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.266 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aae5bbf6759a353d-006543ee45-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.271 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.272 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.273 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.273 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:25.274 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:25.274 DEBUG:boto:Method: GET 2023-11-02T18:45:25.274 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:45:25.275 DEBUG:boto:Data: 2023-11-02T18:45:25.276 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.276 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.277 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.277 DEBUG:boto:Params: {} 2023-11-02T18:45:25.277 DEBUG:boto:Token: None 2023-11-02T18:45:25.278 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.278 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6a55e4eab131c7f-006543ee45-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.289 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.289 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:25.290 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:25.290 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.291 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:45:25.291 DEBUG:boto:Data: 2023-11-02T18:45:25.291 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.292 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.292 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.293 DEBUG:boto:Params: {} 2023-11-02T18:45:25.293 DEBUG:boto:Token: None 2023-11-02T18:45:25.294 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj116 2023-11-02T18:45:25.294 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IMT2qc03g4sZ5swT+EdgxWRtW30= 2023-11-02T18:45:25.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IMT2qc03g4sZ5swT+EdgxWRtW30=', 'Content-Length': '0'} 2023-11-02T18:45:25.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f364e8c8fe59fe5-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.300 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:25.301 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:25.301 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.302 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:45:25.302 DEBUG:boto:Data: 2023-11-02T18:45:25.302 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.303 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.304 DEBUG:boto:Params: {} 2023-11-02T18:45:25.304 DEBUG:boto:Token: None 2023-11-02T18:45:25.305 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj116 2023-11-02T18:45:25.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IMT2qc03g4sZ5swT+EdgxWRtW30= 2023-11-02T18:45:25.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IMT2qc03g4sZ5swT+EdgxWRtW30=', 'Content-Length': '0'} 2023-11-02T18:45:25.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000949dc3f1b0e1ddf0-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:45:25.309 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:25.310 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:25.310 DEBUG:boto:Method: GET 2023-11-02T18:45:25.311 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:45:25.311 DEBUG:boto:Data: 2023-11-02T18:45:25.312 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.312 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.312 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.313 DEBUG:boto:Params: {} 2023-11-02T18:45:25.313 DEBUG:boto:Token: None 2023-11-02T18:45:25.314 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj116 2023-11-02T18:45:25.314 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SdjlHvUVDSftjWLuri3WsdP0XpI= 2023-11-02T18:45:25.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SdjlHvUVDSftjWLuri3WsdP0XpI=', 'Content-Length': '0'} 2023-11-02T18:45:25.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be01b7a84d7d204b-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.358 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:25.359 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:25.359 DEBUG:boto:Method: GET 2023-11-02T18:45:25.360 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:45:25.360 DEBUG:boto:Data: 2023-11-02T18:45:25.361 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.361 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.362 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.362 DEBUG:boto:Params: {} 2023-11-02T18:45:25.363 DEBUG:boto:Token: None 2023-11-02T18:45:25.363 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj116 2023-11-02T18:45:25.364 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SdjlHvUVDSftjWLuri3WsdP0XpI= 2023-11-02T18:45:25.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SdjlHvUVDSftjWLuri3WsdP0XpI=', 'Content-Length': '0'} 2023-11-02T18:45:25.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058ed9b684b36c785-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.368 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.368 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.369 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:25.369 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:25.370 DEBUG:boto:Method: GET 2023-11-02T18:45:25.370 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:45:25.370 DEBUG:boto:Data: 2023-11-02T18:45:25.371 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.371 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.372 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.372 DEBUG:boto:Params: {} 2023-11-02T18:45:25.373 DEBUG:boto:Token: None 2023-11-02T18:45:25.373 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.374 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eb9d189d7f1873d-006543ee45-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.379 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.380 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.381 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.381 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:25.382 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:25.382 DEBUG:boto:Method: GET 2023-11-02T18:45:25.382 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:45:25.384 DEBUG:boto:Data: 2023-11-02T18:45:25.384 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.385 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.385 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.386 DEBUG:boto:Params: {} 2023-11-02T18:45:25.386 DEBUG:boto:Token: None 2023-11-02T18:45:25.387 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.387 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086d358261f3542d2-006543ee45-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.398 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T18:45:25.400 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:25.400 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:25.400 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.401 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.401 DEBUG:boto:Method: GET 2023-11-02T18:45:25.402 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.402 DEBUG:boto:Data: 2023-11-02T18:45:25.403 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.403 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.404 DEBUG:boto:Params: {} 2023-11-02T18:45:25.404 DEBUG:boto:Token: None 2023-11-02T18:45:25.405 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+VRnW5r6s3jhYrYwo5CTEfDhB+8= 2023-11-02T18:45:25.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+VRnW5r6s3jhYrYwo5CTEfDhB+8=', 'Content-Length': '0'} 2023-11-02T18:45:25.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023b0d3464d3a8c23-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.409 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:25.409 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:25.410 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.410 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.411 DEBUG:boto:Method: GET 2023-11-02T18:45:25.411 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.412 DEBUG:boto:Data: 2023-11-02T18:45:25.412 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.413 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.413 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.413 DEBUG:boto:Params: {} 2023-11-02T18:45:25.414 DEBUG:boto:Token: None 2023-11-02T18:45:25.414 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.415 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+VRnW5r6s3jhYrYwo5CTEfDhB+8= 2023-11-02T18:45:25.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+VRnW5r6s3jhYrYwo5CTEfDhB+8=', 'Content-Length': '0'} 2023-11-02T18:45:25.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd86fb875bbddb43-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.419 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:25.419 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:25.420 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.420 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.420 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.421 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.421 DEBUG:boto:Data: 2023-11-02T18:45:25.422 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.422 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.423 DEBUG:boto:Params: {} 2023-11-02T18:45:25.423 DEBUG:boto:Token: None 2023-11-02T18:45:25.424 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7M54vMve3hJdawlBNMA4l4yfq9M= 2023-11-02T18:45:25.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7M54vMve3hJdawlBNMA4l4yfq9M=', 'Content-Length': '0'} 2023-11-02T18:45:25.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f697e94d044fa655-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.428 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:25.428 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:25.429 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.429 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.429 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.430 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.430 DEBUG:boto:Data: 2023-11-02T18:45:25.431 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.431 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.431 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.432 DEBUG:boto:Params: {} 2023-11-02T18:45:25.432 DEBUG:boto:Token: None 2023-11-02T18:45:25.433 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.433 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7M54vMve3hJdawlBNMA4l4yfq9M= 2023-11-02T18:45:25.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7M54vMve3hJdawlBNMA4l4yfq9M=', 'Content-Length': '0'} 2023-11-02T18:45:25.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072a5222ea43b6ac7-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T18:45:25.437 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:25.438 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:25.438 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.439 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.439 DEBUG:boto:Method: GET 2023-11-02T18:45:25.440 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.440 DEBUG:boto:Data: 2023-11-02T18:45:25.441 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.441 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.441 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.442 DEBUG:boto:Params: {} 2023-11-02T18:45:25.442 DEBUG:boto:Token: None 2023-11-02T18:45:25.443 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.443 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+VRnW5r6s3jhYrYwo5CTEfDhB+8= 2023-11-02T18:45:25.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+VRnW5r6s3jhYrYwo5CTEfDhB+8=', 'Content-Length': '0'} 2023-11-02T18:45:25.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005feb5b3529b626f3-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.490 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:25.491 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:25.491 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.492 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.492 DEBUG:boto:Method: GET 2023-11-02T18:45:25.493 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.493 DEBUG:boto:Data: 2023-11-02T18:45:25.494 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.494 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.495 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.495 DEBUG:boto:Params: {} 2023-11-02T18:45:25.495 DEBUG:boto:Token: None 2023-11-02T18:45:25.496 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:25.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+VRnW5r6s3jhYrYwo5CTEfDhB+8= 2023-11-02T18:45:25.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+VRnW5r6s3jhYrYwo5CTEfDhB+8=', 'Content-Length': '0'} 2023-11-02T18:45:25.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3fcc2983ed006b-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.503 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.504 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.505 DEBUG:boto:path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:25.505 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:25.506 DEBUG:boto:Method: GET 2023-11-02T18:45:25.506 DEBUG:boto:Path: /ibtljn-59/?prefix=obj124 2023-11-02T18:45:25.506 DEBUG:boto:Data: 2023-11-02T18:45:25.507 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.507 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.508 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.508 DEBUG:boto:Params: {} 2023-11-02T18:45:25.509 DEBUG:boto:Token: None 2023-11-02T18:45:25.509 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.510 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086917caad8db9e5b-006543ee45-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.516 DEBUG:boto:b'ibtljn-59obj1241000falseobj1242023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.517 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.517 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.518 DEBUG:boto:path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:25.518 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:25.518 DEBUG:boto:Method: GET 2023-11-02T18:45:25.519 DEBUG:boto:Path: /ibtljn-59/?prefix=obj124 2023-11-02T18:45:25.519 DEBUG:boto:Data: 2023-11-02T18:45:25.520 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.520 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.521 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.521 DEBUG:boto:Params: {} 2023-11-02T18:45:25.522 DEBUG:boto:Token: None 2023-11-02T18:45:25.522 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.523 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ad25179807d68d7-006543ee45-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.533 DEBUG:boto:b'ibtljn-59obj1241000falseobj1242023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.534 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:25.535 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:25.535 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.535 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:45:25.536 DEBUG:boto:Data: 2023-11-02T18:45:25.536 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.537 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.537 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.537 DEBUG:boto:Params: {} 2023-11-02T18:45:25.538 DEBUG:boto:Token: None 2023-11-02T18:45:25.538 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj124 2023-11-02T18:45:25.539 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R8bGmifcJ0VqYX9bbP9wV1+GgvY= 2023-11-02T18:45:25.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R8bGmifcJ0VqYX9bbP9wV1+GgvY=', 'Content-Length': '0'} 2023-11-02T18:45:25.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae30cf365427891e-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.542 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:25.543 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:25.543 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.544 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:45:25.544 DEBUG:boto:Data: 2023-11-02T18:45:25.544 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.545 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.545 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.546 DEBUG:boto:Params: {} 2023-11-02T18:45:25.546 DEBUG:boto:Token: None 2023-11-02T18:45:25.547 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj124 2023-11-02T18:45:25.547 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R8bGmifcJ0VqYX9bbP9wV1+GgvY= 2023-11-02T18:45:25.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R8bGmifcJ0VqYX9bbP9wV1+GgvY=', 'Content-Length': '0'} 2023-11-02T18:45:25.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b30e8714b1554e47-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T18:45:25.551 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:25.552 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:25.552 DEBUG:boto:Method: GET 2023-11-02T18:45:25.553 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:45:25.553 DEBUG:boto:Data: 2023-11-02T18:45:25.554 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.554 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.555 DEBUG:boto:Params: {} 2023-11-02T18:45:25.555 DEBUG:boto:Token: None 2023-11-02T18:45:25.556 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj124 2023-11-02T18:45:25.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NYM2bcJEnHfiHVk00zTCw2cE19U= 2023-11-02T18:45:25.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NYM2bcJEnHfiHVk00zTCw2cE19U=', 'Content-Length': '0'} 2023-11-02T18:45:25.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3f1599b84a13f36-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.560 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:25.561 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:25.561 DEBUG:boto:Method: GET 2023-11-02T18:45:25.561 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:45:25.562 DEBUG:boto:Data: 2023-11-02T18:45:25.562 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.563 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.563 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.564 DEBUG:boto:Params: {} 2023-11-02T18:45:25.564 DEBUG:boto:Token: None 2023-11-02T18:45:25.565 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj124 2023-11-02T18:45:25.565 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NYM2bcJEnHfiHVk00zTCw2cE19U= 2023-11-02T18:45:25.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NYM2bcJEnHfiHVk00zTCw2cE19U=', 'Content-Length': '0'} 2023-11-02T18:45:25.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d68d6a7de262aefc-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.610 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.611 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.611 DEBUG:boto:path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:25.612 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:25.612 DEBUG:boto:Method: GET 2023-11-02T18:45:25.613 DEBUG:boto:Path: /ibtljn-59/?prefix=obj124 2023-11-02T18:45:25.613 DEBUG:boto:Data: 2023-11-02T18:45:25.614 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.614 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.614 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.615 DEBUG:boto:Params: {} 2023-11-02T18:45:25.615 DEBUG:boto:Token: None 2023-11-02T18:45:25.616 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.616 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000419dfd33d03dd549-006543ee45-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.623 DEBUG:boto:b'ibtljn-59obj1241000falseobj1242023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.623 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.624 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.624 DEBUG:boto:path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:25.625 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:25.625 DEBUG:boto:Method: GET 2023-11-02T18:45:25.626 DEBUG:boto:Path: /ibtljn-59/?prefix=obj124 2023-11-02T18:45:25.626 DEBUG:boto:Data: 2023-11-02T18:45:25.626 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.627 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.629 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.630 DEBUG:boto:Params: {} 2023-11-02T18:45:25.630 DEBUG:boto:Token: None 2023-11-02T18:45:25.631 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.631 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000624c2a8163c5f95b-006543ee45-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.638 DEBUG:boto:b'ibtljn-59obj1241000falseobj1242023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:45:25.639 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:25.640 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:25.640 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.640 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.641 DEBUG:boto:Method: GET 2023-11-02T18:45:25.641 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.642 DEBUG:boto:Data: 2023-11-02T18:45:25.642 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.643 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.643 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.643 DEBUG:boto:Params: {} 2023-11-02T18:45:25.644 DEBUG:boto:Token: None 2023-11-02T18:45:25.644 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qGuYlpR/viK1Vt2plDImBQi4LKo= 2023-11-02T18:45:25.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qGuYlpR/viK1Vt2plDImBQi4LKo=', 'Content-Length': '0'} 2023-11-02T18:45:25.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066871ac6c5c30249-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.690 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:25.691 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:25.691 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.692 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.692 DEBUG:boto:Method: GET 2023-11-02T18:45:25.693 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.693 DEBUG:boto:Data: 2023-11-02T18:45:25.694 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.694 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.694 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.695 DEBUG:boto:Params: {} 2023-11-02T18:45:25.695 DEBUG:boto:Token: None 2023-11-02T18:45:25.696 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.696 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qGuYlpR/viK1Vt2plDImBQi4LKo= 2023-11-02T18:45:25.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qGuYlpR/viK1Vt2plDImBQi4LKo=', 'Content-Length': '0'} 2023-11-02T18:45:25.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab2d9e4b5332df73-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.701 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:25.701 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:25.701 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.702 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.702 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.703 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.703 DEBUG:boto:Data: 2023-11-02T18:45:25.704 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.704 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.704 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.705 DEBUG:boto:Params: {} 2023-11-02T18:45:25.705 DEBUG:boto:Token: None 2023-11-02T18:45:25.706 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.706 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d2vUGuhts10GSYF1y+udKlSl2M4= 2023-11-02T18:45:25.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d2vUGuhts10GSYF1y+udKlSl2M4=', 'Content-Length': '0'} 2023-11-02T18:45:25.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa2186a255cff09-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.710 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:25.711 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:25.711 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.711 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.712 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.713 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.713 DEBUG:boto:Data: 2023-11-02T18:45:25.714 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.714 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.714 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.715 DEBUG:boto:Params: {} 2023-11-02T18:45:25.715 DEBUG:boto:Token: None 2023-11-02T18:45:25.716 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.716 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d2vUGuhts10GSYF1y+udKlSl2M4= 2023-11-02T18:45:25.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d2vUGuhts10GSYF1y+udKlSl2M4=', 'Content-Length': '0'} 2023-11-02T18:45:25.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9fcfaee778c0487-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:45:25.721 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:25.721 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:25.722 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.722 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.722 DEBUG:boto:Method: GET 2023-11-02T18:45:25.723 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.723 DEBUG:boto:Data: 2023-11-02T18:45:25.724 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.724 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.725 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.725 DEBUG:boto:Params: {} 2023-11-02T18:45:25.725 DEBUG:boto:Token: None 2023-11-02T18:45:25.726 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.726 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qGuYlpR/viK1Vt2plDImBQi4LKo= 2023-11-02T18:45:25.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qGuYlpR/viK1Vt2plDImBQi4LKo=', 'Content-Length': '0'} 2023-11-02T18:45:25.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6fdfd7cde250606-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.731 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:25.731 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:25.732 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.732 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.733 DEBUG:boto:Method: GET 2023-11-02T18:45:25.733 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.733 DEBUG:boto:Data: 2023-11-02T18:45:25.734 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.734 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.735 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.735 DEBUG:boto:Params: {} 2023-11-02T18:45:25.736 DEBUG:boto:Token: None 2023-11-02T18:45:25.736 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:25.737 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qGuYlpR/viK1Vt2plDImBQi4LKo= 2023-11-02T18:45:25.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qGuYlpR/viK1Vt2plDImBQi4LKo=', 'Content-Length': '0'} 2023-11-02T18:45:25.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4b6aa8d5616ce05-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.782 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.783 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.784 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:25.784 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:25.784 DEBUG:boto:Method: GET 2023-11-02T18:45:25.785 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:45:25.785 DEBUG:boto:Data: 2023-11-02T18:45:25.786 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.786 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.787 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.787 DEBUG:boto:Params: {} 2023-11-02T18:45:25.788 DEBUG:boto:Token: None 2023-11-02T18:45:25.788 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cd4227042f299c6-006543ee45-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.796 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.797 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.797 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.797 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:25.798 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:25.798 DEBUG:boto:Method: GET 2023-11-02T18:45:25.799 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:45:25.799 DEBUG:boto:Data: 2023-11-02T18:45:25.800 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.800 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.800 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.801 DEBUG:boto:Params: {} 2023-11-02T18:45:25.801 DEBUG:boto:Token: None 2023-11-02T18:45:25.802 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.802 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081df7035e10420b8-006543ee45-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.813 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.813 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:25.814 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:25.814 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.815 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:45:25.815 DEBUG:boto:Data: 2023-11-02T18:45:25.815 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.816 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.816 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.817 DEBUG:boto:Params: {} 2023-11-02T18:45:25.817 DEBUG:boto:Token: None 2023-11-02T18:45:25.818 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj145 2023-11-02T18:45:25.818 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wyjlb2drUcWvX0RqvNHZci3OWPM= 2023-11-02T18:45:25.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wyjlb2drUcWvX0RqvNHZci3OWPM=', 'Content-Length': '0'} 2023-11-02T18:45:25.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8de7464c5e01f03-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.821 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:25.823 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:25.823 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.824 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:45:25.824 DEBUG:boto:Data: 2023-11-02T18:45:25.825 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.825 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.825 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.826 DEBUG:boto:Params: {} 2023-11-02T18:45:25.826 DEBUG:boto:Token: None 2023-11-02T18:45:25.827 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj145 2023-11-02T18:45:25.827 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wyjlb2drUcWvX0RqvNHZci3OWPM= 2023-11-02T18:45:25.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wyjlb2drUcWvX0RqvNHZci3OWPM=', 'Content-Length': '0'} 2023-11-02T18:45:25.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e6abc3fc9308188-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:45:25.831 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:25.832 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:25.832 DEBUG:boto:Method: GET 2023-11-02T18:45:25.833 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:45:25.833 DEBUG:boto:Data: 2023-11-02T18:45:25.834 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.834 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.834 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.835 DEBUG:boto:Params: {} 2023-11-02T18:45:25.835 DEBUG:boto:Token: None 2023-11-02T18:45:25.836 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj145 2023-11-02T18:45:25.837 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nE3VgMBoRVZJcR4+qJ54BD4pkjg= 2023-11-02T18:45:25.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nE3VgMBoRVZJcR4+qJ54BD4pkjg=', 'Content-Length': '0'} 2023-11-02T18:45:25.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000196b845a5dceed62-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.840 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:25.841 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:25.841 DEBUG:boto:Method: GET 2023-11-02T18:45:25.842 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:45:25.842 DEBUG:boto:Data: 2023-11-02T18:45:25.843 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.843 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.843 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.844 DEBUG:boto:Params: {} 2023-11-02T18:45:25.844 DEBUG:boto:Token: None 2023-11-02T18:45:25.845 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj145 2023-11-02T18:45:25.845 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nE3VgMBoRVZJcR4+qJ54BD4pkjg= 2023-11-02T18:45:25.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nE3VgMBoRVZJcR4+qJ54BD4pkjg=', 'Content-Length': '0'} 2023-11-02T18:45:25.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc8f8b6cf5d9e31c-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.894 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.895 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.896 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:25.896 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:25.896 DEBUG:boto:Method: GET 2023-11-02T18:45:25.897 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:45:25.897 DEBUG:boto:Data: 2023-11-02T18:45:25.898 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.898 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.899 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.899 DEBUG:boto:Params: {} 2023-11-02T18:45:25.900 DEBUG:boto:Token: None 2023-11-02T18:45:25.900 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.901 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dc3d6c331822d22-006543ee45-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.909 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.910 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:25.911 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:25.911 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:25.912 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:25.912 DEBUG:boto:Method: GET 2023-11-02T18:45:25.913 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:45:25.913 DEBUG:boto:Data: 2023-11-02T18:45:25.914 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.914 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.915 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.915 DEBUG:boto:Params: {} 2023-11-02T18:45:25.916 DEBUG:boto:Token: None 2023-11-02T18:45:25.916 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/ 2023-11-02T18:45:25.917 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A= 2023-11-02T18:45:25.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ccbUrb9YhyGQB5COR/sSr8spN8A=', 'Content-Length': '0'} 2023-11-02T18:45:25.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007beed4de89f92b2e-006543ee45-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.927 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:25.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-11-02T18:45:25.929 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:25.929 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:25.930 DEBUG:boto:path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:25.930 DEBUG:boto:auth_path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:25.931 DEBUG:boto:Method: GET 2023-11-02T18:45:25.931 DEBUG:boto:Path: /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:25.931 DEBUG:boto:Data: 2023-11-02T18:45:25.932 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.932 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.933 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.933 DEBUG:boto:Params: {} 2023-11-02T18:45:25.934 DEBUG:boto:Token: None 2023-11-02T18:45:25.934 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:25.935 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iCVZBtgG8TY1n8h6GeMb9pmjssk= 2023-11-02T18:45:25.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iCVZBtgG8TY1n8h6GeMb9pmjssk=', 'Content-Length': '0'} 2023-11-02T18:45:25.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000153c1f56676aa9ef-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.982 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:25.983 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:25.983 DEBUG:boto:path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:25.984 DEBUG:boto:auth_path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:25.984 DEBUG:boto:Method: GET 2023-11-02T18:45:25.985 DEBUG:boto:Path: /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:25.985 DEBUG:boto:Data: 2023-11-02T18:45:25.986 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.986 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:25.987 DEBUG:boto:Port: 8000 2023-11-02T18:45:25.987 DEBUG:boto:Params: {} 2023-11-02T18:45:25.988 DEBUG:boto:Token: None 2023-11-02T18:45:25.988 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:25.989 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iCVZBtgG8TY1n8h6GeMb9pmjssk= 2023-11-02T18:45:25.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iCVZBtgG8TY1n8h6GeMb9pmjssk=', 'Content-Length': '0'} 2023-11-02T18:45:25.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d67d8f04408373e-006543ee45-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:25.993 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:25.994 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:25.994 DEBUG:boto:path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:25.995 DEBUG:boto:auth_path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:25.995 DEBUG:boto:Method: HEAD 2023-11-02T18:45:25.995 DEBUG:boto:Path: /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:25.996 DEBUG:boto:Data: 2023-11-02T18:45:25.996 DEBUG:boto:Headers: {} 2023-11-02T18:45:25.997 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:25.997 DEBUG:boto:Port: 8001 2023-11-02T18:45:25.998 DEBUG:boto:Params: {} 2023-11-02T18:45:25.998 DEBUG:boto:Token: None 2023-11-02T18:45:25.999 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:25 GMT /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:25.999 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:udkFQLtuC0tpDMx4XdIzRlUp268= 2023-11-02T18:45:25.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:udkFQLtuC0tpDMx4XdIzRlUp268=', 'Content-Length': '0'} 2023-11-02T18:45:26.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a591a97365d38b42-006543ee45-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.003 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:26.003 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:26.004 DEBUG:boto:path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:26.004 DEBUG:boto:auth_path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:26.004 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.005 DEBUG:boto:Path: /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:26.005 DEBUG:boto:Data: 2023-11-02T18:45:26.006 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.006 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.007 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.007 DEBUG:boto:Params: {} 2023-11-02T18:45:26.007 DEBUG:boto:Token: None 2023-11-02T18:45:26.008 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:26.008 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OtJ4q+kqzHe9Terhv73myYQ1JyM= 2023-11-02T18:45:26.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OtJ4q+kqzHe9Terhv73myYQ1JyM=', 'Content-Length': '0'} 2023-11-02T18:45:26.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c0bb63ff9a5bc81-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-11-02T18:45:26.014 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:26.014 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:26.015 DEBUG:boto:path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:26.015 DEBUG:boto:auth_path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:26.016 DEBUG:boto:Method: GET 2023-11-02T18:45:26.016 DEBUG:boto:Path: /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:26.017 DEBUG:boto:Data: 2023-11-02T18:45:26.017 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.017 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.018 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.018 DEBUG:boto:Params: {} 2023-11-02T18:45:26.019 DEBUG:boto:Token: None 2023-11-02T18:45:26.019 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:26.020 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1ktYkVVRutCNlXwkRQHgzpxeKUw= 2023-11-02T18:45:26.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1ktYkVVRutCNlXwkRQHgzpxeKUw=', 'Content-Length': '0'} 2023-11-02T18:45:26.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017a6e018bae9ec23-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.025 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:26.025 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:26.025 DEBUG:boto:path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:26.026 DEBUG:boto:auth_path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:26.026 DEBUG:boto:Method: GET 2023-11-02T18:45:26.027 DEBUG:boto:Path: /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:26.027 DEBUG:boto:Data: 2023-11-02T18:45:26.028 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.028 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.029 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.029 DEBUG:boto:Params: {} 2023-11-02T18:45:26.029 DEBUG:boto:Token: None 2023-11-02T18:45:26.030 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:26.031 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1ktYkVVRutCNlXwkRQHgzpxeKUw= 2023-11-02T18:45:26.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1ktYkVVRutCNlXwkRQHgzpxeKUw=', 'Content-Length': '0'} 2023-11-02T18:45:26.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a92f9376403cd73-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.074 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.075 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.075 DEBUG:boto:path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:26.076 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:26.076 DEBUG:boto:Method: GET 2023-11-02T18:45:26.077 DEBUG:boto:Path: /ibtljn-59/?prefix=obj148 2023-11-02T18:45:26.077 DEBUG:boto:Data: 2023-11-02T18:45:26.078 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.078 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.079 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.079 DEBUG:boto:Params: {} 2023-11-02T18:45:26.080 DEBUG:boto:Token: None 2023-11-02T18:45:26.080 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.081 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a097b92b8f9f7afd-006543ee46-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.086 DEBUG:boto:b'ibtljn-59obj1481000falseobj1482023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.087 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.088 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.088 DEBUG:boto:path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:26.088 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:26.089 DEBUG:boto:Method: GET 2023-11-02T18:45:26.089 DEBUG:boto:Path: /ibtljn-59/?prefix=obj148 2023-11-02T18:45:26.090 DEBUG:boto:Data: 2023-11-02T18:45:26.090 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.091 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.091 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.091 DEBUG:boto:Params: {} 2023-11-02T18:45:26.092 DEBUG:boto:Token: None 2023-11-02T18:45:26.093 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.093 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af1f26ae8fcf3ac2-006543ee46-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.103 DEBUG:boto:b'ibtljn-59obj1481000falseobj1482023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.104 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:26.105 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:26.105 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.106 DEBUG:boto:Path: /ibtljn-59/obj148 2023-11-02T18:45:26.106 DEBUG:boto:Data: 2023-11-02T18:45:26.107 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.107 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.108 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.108 DEBUG:boto:Params: {} 2023-11-02T18:45:26.108 DEBUG:boto:Token: None 2023-11-02T18:45:26.109 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj148 2023-11-02T18:45:26.109 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:omgtJ7qFx11aj7hNn0S7E6m1swE= 2023-11-02T18:45:26.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:omgtJ7qFx11aj7hNn0S7E6m1swE=', 'Content-Length': '0'} 2023-11-02T18:45:26.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047038d71d86946be-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.116 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:26.116 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:26.116 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.117 DEBUG:boto:Path: /ibtljn-59/obj148 2023-11-02T18:45:26.117 DEBUG:boto:Data: 2023-11-02T18:45:26.118 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.118 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.119 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.119 DEBUG:boto:Params: {} 2023-11-02T18:45:26.120 DEBUG:boto:Token: None 2023-11-02T18:45:26.120 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj148 2023-11-02T18:45:26.121 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:omgtJ7qFx11aj7hNn0S7E6m1swE= 2023-11-02T18:45:26.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:omgtJ7qFx11aj7hNn0S7E6m1swE=', 'Content-Length': '0'} 2023-11-02T18:45:26.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d87ca94e5ed2ac9-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-11-02T18:45:26.125 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:26.125 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:26.126 DEBUG:boto:Method: GET 2023-11-02T18:45:26.126 DEBUG:boto:Path: /ibtljn-59/obj148 2023-11-02T18:45:26.127 DEBUG:boto:Data: 2023-11-02T18:45:26.127 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.128 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.128 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.129 DEBUG:boto:Params: {} 2023-11-02T18:45:26.129 DEBUG:boto:Token: None 2023-11-02T18:45:26.129 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj148 2023-11-02T18:45:26.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RKVcKBiObkuhM6zKqmYXw0337kc= 2023-11-02T18:45:26.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RKVcKBiObkuhM6zKqmYXw0337kc=', 'Content-Length': '0'} 2023-11-02T18:45:26.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ebe63c83023c91-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.174 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:26.175 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:26.175 DEBUG:boto:Method: GET 2023-11-02T18:45:26.176 DEBUG:boto:Path: /ibtljn-59/obj148 2023-11-02T18:45:26.176 DEBUG:boto:Data: 2023-11-02T18:45:26.177 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.177 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.178 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.178 DEBUG:boto:Params: {} 2023-11-02T18:45:26.179 DEBUG:boto:Token: None 2023-11-02T18:45:26.180 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj148 2023-11-02T18:45:26.180 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RKVcKBiObkuhM6zKqmYXw0337kc= 2023-11-02T18:45:26.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RKVcKBiObkuhM6zKqmYXw0337kc=', 'Content-Length': '0'} 2023-11-02T18:45:26.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6ab9009984f47ce-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.184 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.185 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.185 DEBUG:boto:path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:26.186 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:26.186 DEBUG:boto:Method: GET 2023-11-02T18:45:26.187 DEBUG:boto:Path: /ibtljn-59/?prefix=obj148 2023-11-02T18:45:26.187 DEBUG:boto:Data: 2023-11-02T18:45:26.187 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.188 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.188 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.189 DEBUG:boto:Params: {} 2023-11-02T18:45:26.189 DEBUG:boto:Token: None 2023-11-02T18:45:26.190 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.190 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004845e8a02a406efa-006543ee46-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.197 DEBUG:boto:b'ibtljn-59obj1481000falseobj1482023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.197 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.198 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.198 DEBUG:boto:path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:26.199 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:26.199 DEBUG:boto:Method: GET 2023-11-02T18:45:26.200 DEBUG:boto:Path: /ibtljn-59/?prefix=obj148 2023-11-02T18:45:26.200 DEBUG:boto:Data: 2023-11-02T18:45:26.201 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.201 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.202 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.202 DEBUG:boto:Params: {} 2023-11-02T18:45:26.202 DEBUG:boto:Token: None 2023-11-02T18:45:26.203 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.203 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d93b726236e11ce-006543ee46-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.258 DEBUG:boto:b'ibtljn-59obj1481000falseobj1482023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T18:45:26.260 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:26.260 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:26.260 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.261 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.261 DEBUG:boto:Method: GET 2023-11-02T18:45:26.262 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.262 DEBUG:boto:Data: 2023-11-02T18:45:26.263 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.263 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.264 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.264 DEBUG:boto:Params: {} 2023-11-02T18:45:26.265 DEBUG:boto:Token: None 2023-11-02T18:45:26.265 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.266 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VTti9qa7zXz9kQX5cNlPq7CSoFg= 2023-11-02T18:45:26.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VTti9qa7zXz9kQX5cNlPq7CSoFg=', 'Content-Length': '0'} 2023-11-02T18:45:26.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010e7909f61f1bf98-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.270 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:26.271 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:26.271 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.272 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.272 DEBUG:boto:Method: GET 2023-11-02T18:45:26.272 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.273 DEBUG:boto:Data: 2023-11-02T18:45:26.273 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.274 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.274 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.275 DEBUG:boto:Params: {} 2023-11-02T18:45:26.275 DEBUG:boto:Token: None 2023-11-02T18:45:26.276 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.276 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VTti9qa7zXz9kQX5cNlPq7CSoFg= 2023-11-02T18:45:26.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VTti9qa7zXz9kQX5cNlPq7CSoFg=', 'Content-Length': '0'} 2023-11-02T18:45:26.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039688e1318ed88ac-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.280 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:26.281 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:26.281 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.282 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.282 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.282 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.283 DEBUG:boto:Data: 2023-11-02T18:45:26.283 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.284 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.284 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.284 DEBUG:boto:Params: {} 2023-11-02T18:45:26.285 DEBUG:boto:Token: None 2023-11-02T18:45:26.285 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.286 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/B9gr26K0RBYyqlVjhkoqAkLECs= 2023-11-02T18:45:26.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/B9gr26K0RBYyqlVjhkoqAkLECs=', 'Content-Length': '0'} 2023-11-02T18:45:26.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000579df74da937c704-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.290 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:26.290 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:26.290 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.291 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.291 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.292 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.292 DEBUG:boto:Data: 2023-11-02T18:45:26.292 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.293 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.293 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.294 DEBUG:boto:Params: {} 2023-11-02T18:45:26.294 DEBUG:boto:Token: None 2023-11-02T18:45:26.295 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.295 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/B9gr26K0RBYyqlVjhkoqAkLECs= 2023-11-02T18:45:26.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/B9gr26K0RBYyqlVjhkoqAkLECs=', 'Content-Length': '0'} 2023-11-02T18:45:26.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000116e8c721bd3427b-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T18:45:26.299 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:26.300 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:26.300 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.301 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.301 DEBUG:boto:Method: GET 2023-11-02T18:45:26.301 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.302 DEBUG:boto:Data: 2023-11-02T18:45:26.302 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.303 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.304 DEBUG:boto:Params: {} 2023-11-02T18:45:26.304 DEBUG:boto:Token: None 2023-11-02T18:45:26.305 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VTti9qa7zXz9kQX5cNlPq7CSoFg= 2023-11-02T18:45:26.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VTti9qa7zXz9kQX5cNlPq7CSoFg=', 'Content-Length': '0'} 2023-11-02T18:45:26.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adc716601c025730-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.310 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:26.310 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:26.311 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.311 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.311 DEBUG:boto:Method: GET 2023-11-02T18:45:26.312 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.312 DEBUG:boto:Data: 2023-11-02T18:45:26.313 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.313 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.314 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.314 DEBUG:boto:Params: {} 2023-11-02T18:45:26.315 DEBUG:boto:Token: None 2023-11-02T18:45:26.315 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:26.316 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VTti9qa7zXz9kQX5cNlPq7CSoFg= 2023-11-02T18:45:26.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VTti9qa7zXz9kQX5cNlPq7CSoFg=', 'Content-Length': '0'} 2023-11-02T18:45:26.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028b1e6b94c7e0983-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.366 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.367 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.367 DEBUG:boto:path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:26.368 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:26.368 DEBUG:boto:Method: GET 2023-11-02T18:45:26.369 DEBUG:boto:Path: /ibtljn-59/?prefix=obj155 2023-11-02T18:45:26.369 DEBUG:boto:Data: 2023-11-02T18:45:26.370 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.371 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.371 DEBUG:boto:Params: {} 2023-11-02T18:45:26.371 DEBUG:boto:Token: None 2023-11-02T18:45:26.372 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f99618c674148f06-006543ee46-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.378 DEBUG:boto:b'ibtljn-59obj1551000falseobj1552023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.379 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.380 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.380 DEBUG:boto:path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:26.381 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:26.381 DEBUG:boto:Method: GET 2023-11-02T18:45:26.381 DEBUG:boto:Path: /ibtljn-59/?prefix=obj155 2023-11-02T18:45:26.382 DEBUG:boto:Data: 2023-11-02T18:45:26.382 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.383 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.384 DEBUG:boto:Params: {} 2023-11-02T18:45:26.385 DEBUG:boto:Token: None 2023-11-02T18:45:26.385 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.386 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dbba0e5db93fdb2-006543ee46-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.396 DEBUG:boto:b'ibtljn-59obj1551000falseobj1552023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.397 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:26.397 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:26.397 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.398 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:45:26.398 DEBUG:boto:Data: 2023-11-02T18:45:26.399 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.399 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.400 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.400 DEBUG:boto:Params: {} 2023-11-02T18:45:26.401 DEBUG:boto:Token: None 2023-11-02T18:45:26.401 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj155 2023-11-02T18:45:26.401 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LWQVFTzXt2aG2IUhqDsdPtTxTFg= 2023-11-02T18:45:26.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LWQVFTzXt2aG2IUhqDsdPtTxTFg=', 'Content-Length': '0'} 2023-11-02T18:45:26.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004928ab7875818bf8-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.405 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:26.405 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:26.406 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.406 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:45:26.407 DEBUG:boto:Data: 2023-11-02T18:45:26.407 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.408 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.408 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.408 DEBUG:boto:Params: {} 2023-11-02T18:45:26.409 DEBUG:boto:Token: None 2023-11-02T18:45:26.409 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj155 2023-11-02T18:45:26.410 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LWQVFTzXt2aG2IUhqDsdPtTxTFg= 2023-11-02T18:45:26.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LWQVFTzXt2aG2IUhqDsdPtTxTFg=', 'Content-Length': '0'} 2023-11-02T18:45:26.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002662476be2f5523d-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T18:45:26.414 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:26.414 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:26.415 DEBUG:boto:Method: GET 2023-11-02T18:45:26.415 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:45:26.416 DEBUG:boto:Data: 2023-11-02T18:45:26.416 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.417 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.417 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.417 DEBUG:boto:Params: {} 2023-11-02T18:45:26.418 DEBUG:boto:Token: None 2023-11-02T18:45:26.418 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj155 2023-11-02T18:45:26.419 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2UufCOzuDFmoPh2fMzvjZiLjcQk= 2023-11-02T18:45:26.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2UufCOzuDFmoPh2fMzvjZiLjcQk=', 'Content-Length': '0'} 2023-11-02T18:45:26.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053faae230d939bda-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.423 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:26.423 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:26.424 DEBUG:boto:Method: GET 2023-11-02T18:45:26.424 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:45:26.424 DEBUG:boto:Data: 2023-11-02T18:45:26.425 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.425 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.426 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.426 DEBUG:boto:Params: {} 2023-11-02T18:45:26.427 DEBUG:boto:Token: None 2023-11-02T18:45:26.427 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj155 2023-11-02T18:45:26.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2UufCOzuDFmoPh2fMzvjZiLjcQk= 2023-11-02T18:45:26.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2UufCOzuDFmoPh2fMzvjZiLjcQk=', 'Content-Length': '0'} 2023-11-02T18:45:26.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1108fa96850aa6e-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.432 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.432 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.432 DEBUG:boto:path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:26.433 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:26.433 DEBUG:boto:Method: GET 2023-11-02T18:45:26.434 DEBUG:boto:Path: /ibtljn-59/?prefix=obj155 2023-11-02T18:45:26.434 DEBUG:boto:Data: 2023-11-02T18:45:26.435 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.435 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.436 DEBUG:boto:Params: {} 2023-11-02T18:45:26.436 DEBUG:boto:Token: None 2023-11-02T18:45:26.437 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.437 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000814a8231002b0553-006543ee46-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.486 DEBUG:boto:b'ibtljn-59obj1551000falseobj1552023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.487 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.488 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.488 DEBUG:boto:path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:26.489 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:26.489 DEBUG:boto:Method: GET 2023-11-02T18:45:26.490 DEBUG:boto:Path: /ibtljn-59/?prefix=obj155 2023-11-02T18:45:26.490 DEBUG:boto:Data: 2023-11-02T18:45:26.490 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.491 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.491 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.492 DEBUG:boto:Params: {} 2023-11-02T18:45:26.492 DEBUG:boto:Token: None 2023-11-02T18:45:26.493 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.493 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fc00d9334772bd3-006543ee46-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.504 DEBUG:boto:b'ibtljn-59obj1551000falseobj1552023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:45:26.505 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:26.506 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:26.506 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.507 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.507 DEBUG:boto:Method: GET 2023-11-02T18:45:26.508 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.508 DEBUG:boto:Data: 2023-11-02T18:45:26.508 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.509 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.509 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.510 DEBUG:boto:Params: {} 2023-11-02T18:45:26.510 DEBUG:boto:Token: None 2023-11-02T18:45:26.511 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.511 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J+NhnTYKo6662rMCE63/VI7R87c= 2023-11-02T18:45:26.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J+NhnTYKo6662rMCE63/VI7R87c=', 'Content-Length': '0'} 2023-11-02T18:45:26.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b6841cf45fc2a7-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.515 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:26.516 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:26.516 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.517 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.517 DEBUG:boto:Method: GET 2023-11-02T18:45:26.517 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.518 DEBUG:boto:Data: 2023-11-02T18:45:26.518 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.519 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.520 DEBUG:boto:Params: {} 2023-11-02T18:45:26.520 DEBUG:boto:Token: None 2023-11-02T18:45:26.521 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J+NhnTYKo6662rMCE63/VI7R87c= 2023-11-02T18:45:26.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J+NhnTYKo6662rMCE63/VI7R87c=', 'Content-Length': '0'} 2023-11-02T18:45:26.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f997c20d4e2ed28e-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.566 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:26.567 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:26.567 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.568 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.568 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.569 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.569 DEBUG:boto:Data: 2023-11-02T18:45:26.570 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.570 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.570 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.571 DEBUG:boto:Params: {} 2023-11-02T18:45:26.572 DEBUG:boto:Token: None 2023-11-02T18:45:26.572 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.573 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T3m1WNKoIEVhrm4p86hnwK8UxGU= 2023-11-02T18:45:26.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T3m1WNKoIEVhrm4p86hnwK8UxGU=', 'Content-Length': '0'} 2023-11-02T18:45:26.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8872b9302619bf5-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.577 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:26.577 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:26.577 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.578 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.578 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.579 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.579 DEBUG:boto:Data: 2023-11-02T18:45:26.580 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.580 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.580 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.581 DEBUG:boto:Params: {} 2023-11-02T18:45:26.581 DEBUG:boto:Token: None 2023-11-02T18:45:26.582 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.582 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T3m1WNKoIEVhrm4p86hnwK8UxGU= 2023-11-02T18:45:26.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T3m1WNKoIEVhrm4p86hnwK8UxGU=', 'Content-Length': '0'} 2023-11-02T18:45:26.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005809f8988d19467b-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:45:26.587 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:26.587 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:26.588 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.588 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.589 DEBUG:boto:Method: GET 2023-11-02T18:45:26.589 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.589 DEBUG:boto:Data: 2023-11-02T18:45:26.590 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.591 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.591 DEBUG:boto:Params: {} 2023-11-02T18:45:26.592 DEBUG:boto:Token: None 2023-11-02T18:45:26.592 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J+NhnTYKo6662rMCE63/VI7R87c= 2023-11-02T18:45:26.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J+NhnTYKo6662rMCE63/VI7R87c=', 'Content-Length': '0'} 2023-11-02T18:45:26.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000137e728a572a512e-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.597 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:26.598 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:26.598 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.599 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.599 DEBUG:boto:Method: GET 2023-11-02T18:45:26.599 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.600 DEBUG:boto:Data: 2023-11-02T18:45:26.600 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.601 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.601 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.602 DEBUG:boto:Params: {} 2023-11-02T18:45:26.602 DEBUG:boto:Token: None 2023-11-02T18:45:26.603 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:26.603 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J+NhnTYKo6662rMCE63/VI7R87c= 2023-11-02T18:45:26.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J+NhnTYKo6662rMCE63/VI7R87c=', 'Content-Length': '0'} 2023-11-02T18:45:26.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004afc47004542d982-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.607 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.608 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.608 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:26.609 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:26.609 DEBUG:boto:Method: GET 2023-11-02T18:45:26.610 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:45:26.610 DEBUG:boto:Data: 2023-11-02T18:45:26.611 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.611 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.611 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.612 DEBUG:boto:Params: {} 2023-11-02T18:45:26.612 DEBUG:boto:Token: None 2023-11-02T18:45:26.613 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.613 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000794df85af6a21ff9-006543ee46-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.619 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.620 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.621 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.621 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:26.622 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:26.622 DEBUG:boto:Method: GET 2023-11-02T18:45:26.623 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:45:26.623 DEBUG:boto:Data: 2023-11-02T18:45:26.623 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.624 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.624 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.625 DEBUG:boto:Params: {} 2023-11-02T18:45:26.625 DEBUG:boto:Token: None 2023-11-02T18:45:26.626 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c9ed9402f173d90-006543ee46-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.636 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.637 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:26.638 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:26.638 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.638 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:45:26.639 DEBUG:boto:Data: 2023-11-02T18:45:26.639 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.640 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.640 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.640 DEBUG:boto:Params: {} 2023-11-02T18:45:26.641 DEBUG:boto:Token: None 2023-11-02T18:45:26.641 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj17 2023-11-02T18:45:26.642 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XbqlFhdEInwemmnHt+qg7oB4+yM= 2023-11-02T18:45:26.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XbqlFhdEInwemmnHt+qg7oB4+yM=', 'Content-Length': '0'} 2023-11-02T18:45:26.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037500a4af26816b6-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.645 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:26.646 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:26.646 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.647 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:45:26.647 DEBUG:boto:Data: 2023-11-02T18:45:26.647 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.648 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.648 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.649 DEBUG:boto:Params: {} 2023-11-02T18:45:26.649 DEBUG:boto:Token: None 2023-11-02T18:45:26.650 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj17 2023-11-02T18:45:26.650 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XbqlFhdEInwemmnHt+qg7oB4+yM= 2023-11-02T18:45:26.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XbqlFhdEInwemmnHt+qg7oB4+yM=', 'Content-Length': '0'} 2023-11-02T18:45:26.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3ee960af9c2bc1d-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:45:26.654 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:26.655 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:26.655 DEBUG:boto:Method: GET 2023-11-02T18:45:26.655 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:45:26.656 DEBUG:boto:Data: 2023-11-02T18:45:26.656 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.657 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.657 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.658 DEBUG:boto:Params: {} 2023-11-02T18:45:26.658 DEBUG:boto:Token: None 2023-11-02T18:45:26.659 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj17 2023-11-02T18:45:26.659 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Cx2Q1OuwLx7/bSHljDbgi6XuW38= 2023-11-02T18:45:26.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Cx2Q1OuwLx7/bSHljDbgi6XuW38=', 'Content-Length': '0'} 2023-11-02T18:45:26.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e360e1ba4e0dca-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.702 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:26.703 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:26.704 DEBUG:boto:Method: GET 2023-11-02T18:45:26.704 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:45:26.704 DEBUG:boto:Data: 2023-11-02T18:45:26.705 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.705 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.706 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.706 DEBUG:boto:Params: {} 2023-11-02T18:45:26.707 DEBUG:boto:Token: None 2023-11-02T18:45:26.707 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj17 2023-11-02T18:45:26.708 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Cx2Q1OuwLx7/bSHljDbgi6XuW38= 2023-11-02T18:45:26.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Cx2Q1OuwLx7/bSHljDbgi6XuW38=', 'Content-Length': '0'} 2023-11-02T18:45:26.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000644c182eee60780f-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.712 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.713 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.713 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:26.714 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:26.714 DEBUG:boto:Method: GET 2023-11-02T18:45:26.714 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:45:26.715 DEBUG:boto:Data: 2023-11-02T18:45:26.715 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.716 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.716 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.717 DEBUG:boto:Params: {} 2023-11-02T18:45:26.717 DEBUG:boto:Token: None 2023-11-02T18:45:26.718 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.718 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000576366b3a04aea9e-006543ee46-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.724 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.725 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.725 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.726 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:26.726 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:26.727 DEBUG:boto:Method: GET 2023-11-02T18:45:26.730 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:45:26.730 DEBUG:boto:Data: 2023-11-02T18:45:26.731 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.732 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.732 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.733 DEBUG:boto:Params: {} 2023-11-02T18:45:26.733 DEBUG:boto:Token: None 2023-11-02T18:45:26.734 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.734 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000829eec4914dbf9a0-006543ee46-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.745 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-11-02T18:45:26.746 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:26.747 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:26.747 DEBUG:boto:path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.748 DEBUG:boto:auth_path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.748 DEBUG:boto:Method: GET 2023-11-02T18:45:26.748 DEBUG:boto:Path: /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.749 DEBUG:boto:Data: 2023-11-02T18:45:26.749 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.750 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.750 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.751 DEBUG:boto:Params: {} 2023-11-02T18:45:26.751 DEBUG:boto:Token: None 2023-11-02T18:45:26.752 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.752 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hxzAfAOllUCUnAfgFyAj9v6juP0= 2023-11-02T18:45:26.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hxzAfAOllUCUnAfgFyAj9v6juP0=', 'Content-Length': '0'} 2023-11-02T18:45:26.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b742aaf896c55e7f-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.756 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:26.757 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:26.757 DEBUG:boto:path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.758 DEBUG:boto:auth_path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.758 DEBUG:boto:Method: GET 2023-11-02T18:45:26.759 DEBUG:boto:Path: /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.759 DEBUG:boto:Data: 2023-11-02T18:45:26.759 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.760 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.760 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.761 DEBUG:boto:Params: {} 2023-11-02T18:45:26.761 DEBUG:boto:Token: None 2023-11-02T18:45:26.762 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.762 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hxzAfAOllUCUnAfgFyAj9v6juP0= 2023-11-02T18:45:26.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hxzAfAOllUCUnAfgFyAj9v6juP0=', 'Content-Length': '0'} 2023-11-02T18:45:26.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000399403d262be90b4-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.766 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:26.767 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:26.767 DEBUG:boto:path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.768 DEBUG:boto:auth_path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.768 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.769 DEBUG:boto:Path: /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.769 DEBUG:boto:Data: 2023-11-02T18:45:26.769 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.770 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.770 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.771 DEBUG:boto:Params: {} 2023-11-02T18:45:26.771 DEBUG:boto:Token: None 2023-11-02T18:45:26.772 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.772 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PTTL1tDHn1NpCWnmNJNelXeLxtw= 2023-11-02T18:45:26.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PTTL1tDHn1NpCWnmNJNelXeLxtw=', 'Content-Length': '0'} 2023-11-02T18:45:26.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003258f8ae14f9a33c-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.776 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:26.776 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:26.777 DEBUG:boto:path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.777 DEBUG:boto:auth_path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.778 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.778 DEBUG:boto:Path: /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.779 DEBUG:boto:Data: 2023-11-02T18:45:26.779 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.779 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.780 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.780 DEBUG:boto:Params: {} 2023-11-02T18:45:26.781 DEBUG:boto:Token: None 2023-11-02T18:45:26.781 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.782 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PTTL1tDHn1NpCWnmNJNelXeLxtw= 2023-11-02T18:45:26.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PTTL1tDHn1NpCWnmNJNelXeLxtw=', 'Content-Length': '0'} 2023-11-02T18:45:26.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de2bfd163b3d2062-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-11-02T18:45:26.786 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:26.786 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:26.787 DEBUG:boto:path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.787 DEBUG:boto:auth_path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.788 DEBUG:boto:Method: GET 2023-11-02T18:45:26.788 DEBUG:boto:Path: /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.789 DEBUG:boto:Data: 2023-11-02T18:45:26.789 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.790 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.790 DEBUG:boto:Params: {} 2023-11-02T18:45:26.791 DEBUG:boto:Token: None 2023-11-02T18:45:26.791 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.792 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hxzAfAOllUCUnAfgFyAj9v6juP0= 2023-11-02T18:45:26.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hxzAfAOllUCUnAfgFyAj9v6juP0=', 'Content-Length': '0'} 2023-11-02T18:45:26.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c08ded8b4998212d-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.838 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:26.839 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:26.840 DEBUG:boto:path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.840 DEBUG:boto:auth_path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.840 DEBUG:boto:Method: GET 2023-11-02T18:45:26.841 DEBUG:boto:Path: /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.841 DEBUG:boto:Data: 2023-11-02T18:45:26.842 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.842 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.843 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.843 DEBUG:boto:Params: {} 2023-11-02T18:45:26.844 DEBUG:boto:Token: None 2023-11-02T18:45:26.844 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:26.845 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hxzAfAOllUCUnAfgFyAj9v6juP0= 2023-11-02T18:45:26.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hxzAfAOllUCUnAfgFyAj9v6juP0=', 'Content-Length': '0'} 2023-11-02T18:45:26.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000062d5e2f804fd037-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.849 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.850 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.850 DEBUG:boto:path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:26.851 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:26.851 DEBUG:boto:Method: GET 2023-11-02T18:45:26.852 DEBUG:boto:Path: /ibtljn-59/?prefix=obj185 2023-11-02T18:45:26.852 DEBUG:boto:Data: 2023-11-02T18:45:26.853 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.853 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.854 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.854 DEBUG:boto:Params: {} 2023-11-02T18:45:26.855 DEBUG:boto:Token: None 2023-11-02T18:45:26.855 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.856 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bab92dc0bbfc818-006543ee46-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.862 DEBUG:boto:b'ibtljn-59obj1851000falseobj1852023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.862 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.863 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.863 DEBUG:boto:path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:26.864 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:26.864 DEBUG:boto:Method: GET 2023-11-02T18:45:26.865 DEBUG:boto:Path: /ibtljn-59/?prefix=obj185 2023-11-02T18:45:26.865 DEBUG:boto:Data: 2023-11-02T18:45:26.866 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.866 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.867 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.867 DEBUG:boto:Params: {} 2023-11-02T18:45:26.867 DEBUG:boto:Token: None 2023-11-02T18:45:26.868 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.868 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c45736d46247af29-006543ee46-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.880 DEBUG:boto:b'ibtljn-59obj1851000falseobj1852023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.880 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:26.881 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:26.881 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.882 DEBUG:boto:Path: /ibtljn-59/obj185 2023-11-02T18:45:26.882 DEBUG:boto:Data: 2023-11-02T18:45:26.882 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.883 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.883 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.884 DEBUG:boto:Params: {} 2023-11-02T18:45:26.884 DEBUG:boto:Token: None 2023-11-02T18:45:26.885 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj185 2023-11-02T18:45:26.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jXzWXpqk4oXq1bLnoIkg1Ygm/3c= 2023-11-02T18:45:26.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jXzWXpqk4oXq1bLnoIkg1Ygm/3c=', 'Content-Length': '0'} 2023-11-02T18:45:26.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1480b3edc97e45e-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.888 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:26.889 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:26.889 DEBUG:boto:Method: HEAD 2023-11-02T18:45:26.890 DEBUG:boto:Path: /ibtljn-59/obj185 2023-11-02T18:45:26.890 DEBUG:boto:Data: 2023-11-02T18:45:26.891 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.891 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.891 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.892 DEBUG:boto:Params: {} 2023-11-02T18:45:26.892 DEBUG:boto:Token: None 2023-11-02T18:45:26.893 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj185 2023-11-02T18:45:26.893 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jXzWXpqk4oXq1bLnoIkg1Ygm/3c= 2023-11-02T18:45:26.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jXzWXpqk4oXq1bLnoIkg1Ygm/3c=', 'Content-Length': '0'} 2023-11-02T18:45:26.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d5b4570b0ec4c05-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-11-02T18:45:26.897 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:26.897 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:26.898 DEBUG:boto:Method: GET 2023-11-02T18:45:26.898 DEBUG:boto:Path: /ibtljn-59/obj185 2023-11-02T18:45:26.899 DEBUG:boto:Data: 2023-11-02T18:45:26.899 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.900 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.900 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.901 DEBUG:boto:Params: {} 2023-11-02T18:45:26.901 DEBUG:boto:Token: None 2023-11-02T18:45:26.902 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj185 2023-11-02T18:45:26.902 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C71yfnmMyxTKYgwGCk28kk3Eykg= 2023-11-02T18:45:26.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C71yfnmMyxTKYgwGCk28kk3Eykg=', 'Content-Length': '0'} 2023-11-02T18:45:26.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a09acbdda3174de-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.906 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:26.906 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:26.907 DEBUG:boto:Method: GET 2023-11-02T18:45:26.907 DEBUG:boto:Path: /ibtljn-59/obj185 2023-11-02T18:45:26.908 DEBUG:boto:Data: 2023-11-02T18:45:26.908 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.909 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.909 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.909 DEBUG:boto:Params: {} 2023-11-02T18:45:26.910 DEBUG:boto:Token: None 2023-11-02T18:45:26.910 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/obj185 2023-11-02T18:45:26.911 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C71yfnmMyxTKYgwGCk28kk3Eykg= 2023-11-02T18:45:26.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C71yfnmMyxTKYgwGCk28kk3Eykg=', 'Content-Length': '0'} 2023-11-02T18:45:26.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000598ae468f50923cb-006543ee46-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.962 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.963 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.964 DEBUG:boto:path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:26.964 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:26.964 DEBUG:boto:Method: GET 2023-11-02T18:45:26.965 DEBUG:boto:Path: /ibtljn-59/?prefix=obj185 2023-11-02T18:45:26.965 DEBUG:boto:Data: 2023-11-02T18:45:26.966 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.966 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.967 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.967 DEBUG:boto:Params: {} 2023-11-02T18:45:26.968 DEBUG:boto:Token: None 2023-11-02T18:45:26.968 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002bd3c47a08c47a0-006543ee46-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.975 DEBUG:boto:b'ibtljn-59obj1851000falseobj1852023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.976 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:26.977 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:26.977 DEBUG:boto:path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:26.978 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:26.978 DEBUG:boto:Method: GET 2023-11-02T18:45:26.978 DEBUG:boto:Path: /ibtljn-59/?prefix=obj185 2023-11-02T18:45:26.979 DEBUG:boto:Data: 2023-11-02T18:45:26.979 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.980 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:26.980 DEBUG:boto:Port: 8000 2023-11-02T18:45:26.981 DEBUG:boto:Params: {} 2023-11-02T18:45:26.981 DEBUG:boto:Token: None 2023-11-02T18:45:26.982 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:26 GMT /ibtljn-59/ 2023-11-02T18:45:26.982 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU= 2023-11-02T18:45:26.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lZbMro396RmkhhUPNsJTpXS4bnU=', 'Content-Length': '0'} 2023-11-02T18:45:26.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2279b9a84167236-006543ee46-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:26.993 DEBUG:boto:b'ibtljn-59obj1851000falseobj1852023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:26.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:45:26.994 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:26.995 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:26.995 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:26.995 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:26.996 DEBUG:boto:Method: GET 2023-11-02T18:45:26.996 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:26.997 DEBUG:boto:Data: 2023-11-02T18:45:26.997 DEBUG:boto:Headers: {} 2023-11-02T18:45:26.998 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:26.998 DEBUG:boto:Port: 8001 2023-11-02T18:45:26.999 DEBUG:boto:Params: {} 2023-11-02T18:45:26.999 DEBUG:boto:Token: None 2023-11-02T18:45:27.000 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.000 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FZFBUv0aYe8bgmLmPCgaAVBvyLE= 2023-11-02T18:45:27.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FZFBUv0aYe8bgmLmPCgaAVBvyLE=', 'Content-Length': '0'} 2023-11-02T18:45:27.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cf60edfa8bf073c-006543ee46-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.050 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:27.051 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:27.051 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.052 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.052 DEBUG:boto:Method: GET 2023-11-02T18:45:27.053 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.053 DEBUG:boto:Data: 2023-11-02T18:45:27.054 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.054 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.055 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.055 DEBUG:boto:Params: {} 2023-11-02T18:45:27.056 DEBUG:boto:Token: None 2023-11-02T18:45:27.056 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.057 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FZFBUv0aYe8bgmLmPCgaAVBvyLE= 2023-11-02T18:45:27.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FZFBUv0aYe8bgmLmPCgaAVBvyLE=', 'Content-Length': '0'} 2023-11-02T18:45:27.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3ec177f724862cf-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.061 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:27.062 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:27.062 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.062 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.063 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.063 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.064 DEBUG:boto:Data: 2023-11-02T18:45:27.064 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.065 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.065 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.065 DEBUG:boto:Params: {} 2023-11-02T18:45:27.066 DEBUG:boto:Token: None 2023-11-02T18:45:27.066 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.067 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KjZjA1fibj6uNw/7753H18/0Guo= 2023-11-02T18:45:27.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KjZjA1fibj6uNw/7753H18/0Guo=', 'Content-Length': '0'} 2023-11-02T18:45:27.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007074e87754c214a3-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.071 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:27.071 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:27.072 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.072 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.073 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.073 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.074 DEBUG:boto:Data: 2023-11-02T18:45:27.074 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.075 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.075 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.075 DEBUG:boto:Params: {} 2023-11-02T18:45:27.076 DEBUG:boto:Token: None 2023-11-02T18:45:27.076 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.077 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KjZjA1fibj6uNw/7753H18/0Guo= 2023-11-02T18:45:27.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KjZjA1fibj6uNw/7753H18/0Guo=', 'Content-Length': '0'} 2023-11-02T18:45:27.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b301fcf24305fa67-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:45:27.082 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:27.082 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:27.083 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.083 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.084 DEBUG:boto:Method: GET 2023-11-02T18:45:27.084 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.085 DEBUG:boto:Data: 2023-11-02T18:45:27.085 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.085 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.086 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.086 DEBUG:boto:Params: {} 2023-11-02T18:45:27.087 DEBUG:boto:Token: None 2023-11-02T18:45:27.087 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.088 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FZFBUv0aYe8bgmLmPCgaAVBvyLE= 2023-11-02T18:45:27.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FZFBUv0aYe8bgmLmPCgaAVBvyLE=', 'Content-Length': '0'} 2023-11-02T18:45:27.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afa6c7f19e8921ab-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.092 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:27.092 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:27.093 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.093 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.094 DEBUG:boto:Method: GET 2023-11-02T18:45:27.094 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.095 DEBUG:boto:Data: 2023-11-02T18:45:27.095 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.096 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.097 DEBUG:boto:Params: {} 2023-11-02T18:45:27.097 DEBUG:boto:Token: None 2023-11-02T18:45:27.098 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:27.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FZFBUv0aYe8bgmLmPCgaAVBvyLE= 2023-11-02T18:45:27.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FZFBUv0aYe8bgmLmPCgaAVBvyLE=', 'Content-Length': '0'} 2023-11-02T18:45:27.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d64fd1fbcb8a1329-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.102 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.103 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.103 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:27.104 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:27.104 DEBUG:boto:Method: GET 2023-11-02T18:45:27.105 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:45:27.105 DEBUG:boto:Data: 2023-11-02T18:45:27.105 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.106 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.107 DEBUG:boto:Params: {} 2023-11-02T18:45:27.107 DEBUG:boto:Token: None 2023-11-02T18:45:27.108 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.108 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:27.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093b865ae0f57d15c-006543ee47-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.158 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:27.159 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.159 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.160 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:27.160 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:27.161 DEBUG:boto:Method: GET 2023-11-02T18:45:27.161 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:45:27.162 DEBUG:boto:Data: 2023-11-02T18:45:27.162 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.163 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.163 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.163 DEBUG:boto:Params: {} 2023-11-02T18:45:27.164 DEBUG:boto:Token: None 2023-11-02T18:45:27.165 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.165 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:27.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007db988ff48be218f-006543ee47-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.177 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:27.177 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:27.178 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:27.178 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.179 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:45:27.179 DEBUG:boto:Data: 2023-11-02T18:45:27.180 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.180 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.180 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.181 DEBUG:boto:Params: {} 2023-11-02T18:45:27.181 DEBUG:boto:Token: None 2023-11-02T18:45:27.182 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj19 2023-11-02T18:45:27.182 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2cVKf5juBfuSSU+xDQ4d4jR5X8M= 2023-11-02T18:45:27.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2cVKf5juBfuSSU+xDQ4d4jR5X8M=', 'Content-Length': '0'} 2023-11-02T18:45:27.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de7cf0fcbf7294d6-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.186 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:27.186 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:27.187 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.187 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:45:27.187 DEBUG:boto:Data: 2023-11-02T18:45:27.188 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.188 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.189 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.189 DEBUG:boto:Params: {} 2023-11-02T18:45:27.190 DEBUG:boto:Token: None 2023-11-02T18:45:27.190 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj19 2023-11-02T18:45:27.191 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2cVKf5juBfuSSU+xDQ4d4jR5X8M= 2023-11-02T18:45:27.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2cVKf5juBfuSSU+xDQ4d4jR5X8M=', 'Content-Length': '0'} 2023-11-02T18:45:27.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097aada01c4f471b1-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:45:27.194 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:27.195 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:27.195 DEBUG:boto:Method: GET 2023-11-02T18:45:27.196 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:45:27.196 DEBUG:boto:Data: 2023-11-02T18:45:27.197 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.197 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.198 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.198 DEBUG:boto:Params: {} 2023-11-02T18:45:27.199 DEBUG:boto:Token: None 2023-11-02T18:45:27.199 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj19 2023-11-02T18:45:27.199 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aOwN9IBWWFrnPsbQypcBuY+Trws= 2023-11-02T18:45:27.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aOwN9IBWWFrnPsbQypcBuY+Trws=', 'Content-Length': '0'} 2023-11-02T18:45:27.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014a88e7c39a33c54-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.203 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:27.203 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:27.204 DEBUG:boto:Method: GET 2023-11-02T18:45:27.204 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:45:27.205 DEBUG:boto:Data: 2023-11-02T18:45:27.205 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.206 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.206 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.207 DEBUG:boto:Params: {} 2023-11-02T18:45:27.207 DEBUG:boto:Token: None 2023-11-02T18:45:27.208 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj19 2023-11-02T18:45:27.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aOwN9IBWWFrnPsbQypcBuY+Trws= 2023-11-02T18:45:27.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aOwN9IBWWFrnPsbQypcBuY+Trws=', 'Content-Length': '0'} 2023-11-02T18:45:27.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079cc6d7f5121aa88-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.258 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.259 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.259 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:27.260 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:27.260 DEBUG:boto:Method: GET 2023-11-02T18:45:27.261 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:45:27.261 DEBUG:boto:Data: 2023-11-02T18:45:27.261 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.262 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.262 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.263 DEBUG:boto:Params: {} 2023-11-02T18:45:27.263 DEBUG:boto:Token: None 2023-11-02T18:45:27.264 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.264 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:27.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcf04473107db2e1-006543ee47-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.271 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:27.272 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.272 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.273 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:27.273 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:27.274 DEBUG:boto:Method: GET 2023-11-02T18:45:27.274 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:45:27.274 DEBUG:boto:Data: 2023-11-02T18:45:27.275 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.275 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.276 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.276 DEBUG:boto:Params: {} 2023-11-02T18:45:27.277 DEBUG:boto:Token: None 2023-11-02T18:45:27.277 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.278 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:27.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079b38c818ecea702-006543ee47-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.289 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:27.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:45:27.290 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:27.290 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:27.291 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.291 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.292 DEBUG:boto:Method: GET 2023-11-02T18:45:27.292 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.292 DEBUG:boto:Data: 2023-11-02T18:45:27.293 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.293 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.294 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.294 DEBUG:boto:Params: {} 2023-11-02T18:45:27.295 DEBUG:boto:Token: None 2023-11-02T18:45:27.295 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.296 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PBsvT4lkSaVSgCyxv9J24H03Ov0= 2023-11-02T18:45:27.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PBsvT4lkSaVSgCyxv9J24H03Ov0=', 'Content-Length': '0'} 2023-11-02T18:45:27.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cc15bc9a1df00aa-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.300 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:27.300 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:27.301 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.301 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.301 DEBUG:boto:Method: GET 2023-11-02T18:45:27.302 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.302 DEBUG:boto:Data: 2023-11-02T18:45:27.303 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.304 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.304 DEBUG:boto:Params: {} 2023-11-02T18:45:27.305 DEBUG:boto:Token: None 2023-11-02T18:45:27.305 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.306 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PBsvT4lkSaVSgCyxv9J24H03Ov0= 2023-11-02T18:45:27.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PBsvT4lkSaVSgCyxv9J24H03Ov0=', 'Content-Length': '0'} 2023-11-02T18:45:27.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000564669967863e818-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.310 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:27.310 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:27.311 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.311 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.312 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.312 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.312 DEBUG:boto:Data: 2023-11-02T18:45:27.313 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.313 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.314 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.314 DEBUG:boto:Params: {} 2023-11-02T18:45:27.315 DEBUG:boto:Token: None 2023-11-02T18:45:27.315 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.316 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NCRZWfEYX+DuYW8vC/MgajeQDew= 2023-11-02T18:45:27.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NCRZWfEYX+DuYW8vC/MgajeQDew=', 'Content-Length': '0'} 2023-11-02T18:45:27.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e99f58f8929aa31a-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.319 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:27.320 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:27.320 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.320 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.321 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.321 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.322 DEBUG:boto:Data: 2023-11-02T18:45:27.322 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.322 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.323 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.323 DEBUG:boto:Params: {} 2023-11-02T18:45:27.324 DEBUG:boto:Token: None 2023-11-02T18:45:27.324 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.325 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NCRZWfEYX+DuYW8vC/MgajeQDew= 2023-11-02T18:45:27.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NCRZWfEYX+DuYW8vC/MgajeQDew=', 'Content-Length': '0'} 2023-11-02T18:45:27.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db4bb1871fdfb7a-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:45:27.329 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:27.329 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:27.330 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.330 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.331 DEBUG:boto:Method: GET 2023-11-02T18:45:27.331 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.332 DEBUG:boto:Data: 2023-11-02T18:45:27.332 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.333 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.333 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.333 DEBUG:boto:Params: {} 2023-11-02T18:45:27.334 DEBUG:boto:Token: None 2023-11-02T18:45:27.334 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.335 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PBsvT4lkSaVSgCyxv9J24H03Ov0= 2023-11-02T18:45:27.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PBsvT4lkSaVSgCyxv9J24H03Ov0=', 'Content-Length': '0'} 2023-11-02T18:45:27.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b37e97c72a4579d3-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.382 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:27.383 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:27.383 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.384 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.384 DEBUG:boto:Method: GET 2023-11-02T18:45:27.385 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.385 DEBUG:boto:Data: 2023-11-02T18:45:27.386 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.386 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.387 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.387 DEBUG:boto:Params: {} 2023-11-02T18:45:27.388 DEBUG:boto:Token: None 2023-11-02T18:45:27.388 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:27.389 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PBsvT4lkSaVSgCyxv9J24H03Ov0= 2023-11-02T18:45:27.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PBsvT4lkSaVSgCyxv9J24H03Ov0=', 'Content-Length': '0'} 2023-11-02T18:45:27.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000696739fd4a84ee7d-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.394 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.394 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.395 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:27.395 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:27.396 DEBUG:boto:Method: GET 2023-11-02T18:45:27.396 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:45:27.397 DEBUG:boto:Data: 2023-11-02T18:45:27.397 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.397 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.398 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.398 DEBUG:boto:Params: {} 2023-11-02T18:45:27.399 DEBUG:boto:Token: None 2023-11-02T18:45:27.399 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:27.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093f7f12526140703-006543ee47-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.406 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:27.408 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.408 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.409 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:27.409 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:27.410 DEBUG:boto:Method: GET 2023-11-02T18:45:27.410 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:45:27.411 DEBUG:boto:Data: 2023-11-02T18:45:27.411 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.412 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.412 DEBUG:boto:Params: {} 2023-11-02T18:45:27.413 DEBUG:boto:Token: None 2023-11-02T18:45:27.413 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:27.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58ed955148418fd-006543ee47-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.425 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:27.425 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:27.426 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:27.426 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.427 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:45:27.427 DEBUG:boto:Data: 2023-11-02T18:45:27.428 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.428 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.429 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.429 DEBUG:boto:Params: {} 2023-11-02T18:45:27.429 DEBUG:boto:Token: None 2023-11-02T18:45:27.430 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj22 2023-11-02T18:45:27.430 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2s67Q3RJvSUFEXY682121bojy7w= 2023-11-02T18:45:27.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2s67Q3RJvSUFEXY682121bojy7w=', 'Content-Length': '0'} 2023-11-02T18:45:27.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aa97cc2afd698d6-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.434 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:27.434 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:27.435 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.435 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:45:27.436 DEBUG:boto:Data: 2023-11-02T18:45:27.436 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.437 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.437 DEBUG:boto:Params: {} 2023-11-02T18:45:27.438 DEBUG:boto:Token: None 2023-11-02T18:45:27.438 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj22 2023-11-02T18:45:27.439 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2s67Q3RJvSUFEXY682121bojy7w= 2023-11-02T18:45:27.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2s67Q3RJvSUFEXY682121bojy7w=', 'Content-Length': '0'} 2023-11-02T18:45:27.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc71f475c90bda98-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:45:27.443 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:27.443 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:27.444 DEBUG:boto:Method: GET 2023-11-02T18:45:27.444 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:45:27.445 DEBUG:boto:Data: 2023-11-02T18:45:27.445 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.445 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.446 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.446 DEBUG:boto:Params: {} 2023-11-02T18:45:27.447 DEBUG:boto:Token: None 2023-11-02T18:45:27.447 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj22 2023-11-02T18:45:27.448 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:db9YLrTHawC5HSedQYU0HJf/khA= 2023-11-02T18:45:27.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:db9YLrTHawC5HSedQYU0HJf/khA=', 'Content-Length': '0'} 2023-11-02T18:45:27.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097c68147f38b9e09-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.451 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:27.452 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:27.452 DEBUG:boto:Method: GET 2023-11-02T18:45:27.453 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:45:27.453 DEBUG:boto:Data: 2023-11-02T18:45:27.454 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.455 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.455 DEBUG:boto:Params: {} 2023-11-02T18:45:27.456 DEBUG:boto:Token: None 2023-11-02T18:45:27.456 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj22 2023-11-02T18:45:27.457 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:db9YLrTHawC5HSedQYU0HJf/khA= 2023-11-02T18:45:27.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:db9YLrTHawC5HSedQYU0HJf/khA=', 'Content-Length': '0'} 2023-11-02T18:45:27.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000752350ea6e9f23ae-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.506 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.507 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.507 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:27.508 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:27.508 DEBUG:boto:Method: GET 2023-11-02T18:45:27.509 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:45:27.509 DEBUG:boto:Data: 2023-11-02T18:45:27.510 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.511 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.511 DEBUG:boto:Params: {} 2023-11-02T18:45:27.512 DEBUG:boto:Token: None 2023-11-02T18:45:27.512 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:27.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000531237da55da5077-006543ee47-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.519 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:27.519 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.520 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.520 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:27.521 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:27.521 DEBUG:boto:Method: GET 2023-11-02T18:45:27.522 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:45:27.522 DEBUG:boto:Data: 2023-11-02T18:45:27.523 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.523 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.523 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.524 DEBUG:boto:Params: {} 2023-11-02T18:45:27.524 DEBUG:boto:Token: None 2023-11-02T18:45:27.525 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.526 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:27.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009739cff7cc246c59-006543ee47-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.536 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:27.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:45:27.538 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:27.539 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:27.539 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.540 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.540 DEBUG:boto:Method: GET 2023-11-02T18:45:27.540 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.541 DEBUG:boto:Data: 2023-11-02T18:45:27.541 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.542 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.542 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.543 DEBUG:boto:Params: {} 2023-11-02T18:45:27.544 DEBUG:boto:Token: None 2023-11-02T18:45:27.544 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.545 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CRMR3eVNB4W6iQ92jPb5bqeFKBo= 2023-11-02T18:45:27.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CRMR3eVNB4W6iQ92jPb5bqeFKBo=', 'Content-Length': '0'} 2023-11-02T18:45:27.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e6d9d88e3785c0-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.594 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:27.595 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:27.596 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.597 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.597 DEBUG:boto:Method: GET 2023-11-02T18:45:27.598 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.599 DEBUG:boto:Data: 2023-11-02T18:45:27.599 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.600 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.600 DEBUG:boto:Params: {} 2023-11-02T18:45:27.601 DEBUG:boto:Token: None 2023-11-02T18:45:27.602 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.602 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CRMR3eVNB4W6iQ92jPb5bqeFKBo= 2023-11-02T18:45:27.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CRMR3eVNB4W6iQ92jPb5bqeFKBo=', 'Content-Length': '0'} 2023-11-02T18:45:27.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1212c7bad5dd7ab-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.607 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:27.607 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:27.607 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.608 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.608 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.609 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.609 DEBUG:boto:Data: 2023-11-02T18:45:27.610 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.610 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.611 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.611 DEBUG:boto:Params: {} 2023-11-02T18:45:27.611 DEBUG:boto:Token: None 2023-11-02T18:45:27.612 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qSuXN0/pNdM+KrCbnktJB6ixe08= 2023-11-02T18:45:27.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qSuXN0/pNdM+KrCbnktJB6ixe08=', 'Content-Length': '0'} 2023-11-02T18:45:27.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eac5a3ee80083164-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.617 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:27.617 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:27.618 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.619 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.620 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.620 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.621 DEBUG:boto:Data: 2023-11-02T18:45:27.622 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.622 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.622 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.623 DEBUG:boto:Params: {} 2023-11-02T18:45:27.623 DEBUG:boto:Token: None 2023-11-02T18:45:27.624 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qSuXN0/pNdM+KrCbnktJB6ixe08= 2023-11-02T18:45:27.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qSuXN0/pNdM+KrCbnktJB6ixe08=', 'Content-Length': '0'} 2023-11-02T18:45:27.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000921f0e55a15246a3-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:45:27.630 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:27.630 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:27.631 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.631 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.632 DEBUG:boto:Method: GET 2023-11-02T18:45:27.632 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.633 DEBUG:boto:Data: 2023-11-02T18:45:27.633 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.634 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.634 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.634 DEBUG:boto:Params: {} 2023-11-02T18:45:27.635 DEBUG:boto:Token: None 2023-11-02T18:45:27.635 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.636 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CRMR3eVNB4W6iQ92jPb5bqeFKBo= 2023-11-02T18:45:27.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CRMR3eVNB4W6iQ92jPb5bqeFKBo=', 'Content-Length': '0'} 2023-11-02T18:45:27.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b192763f46e4ed2-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.640 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:27.641 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:27.642 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.645 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.646 DEBUG:boto:Method: GET 2023-11-02T18:45:27.646 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.647 DEBUG:boto:Data: 2023-11-02T18:45:27.647 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.648 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.648 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.649 DEBUG:boto:Params: {} 2023-11-02T18:45:27.649 DEBUG:boto:Token: None 2023-11-02T18:45:27.650 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:27.651 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CRMR3eVNB4W6iQ92jPb5bqeFKBo= 2023-11-02T18:45:27.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CRMR3eVNB4W6iQ92jPb5bqeFKBo=', 'Content-Length': '0'} 2023-11-02T18:45:27.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d282d3e74c441fd6-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.656 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.656 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.657 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:27.657 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:27.657 DEBUG:boto:Method: GET 2023-11-02T18:45:27.658 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:45:27.658 DEBUG:boto:Data: 2023-11-02T18:45:27.659 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.659 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.660 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.660 DEBUG:boto:Params: {} 2023-11-02T18:45:27.661 DEBUG:boto:Token: None 2023-11-02T18:45:27.661 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.662 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:27.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f73875ab5aa19cf-006543ee47-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.706 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:27.707 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.708 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.708 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:27.709 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:27.709 DEBUG:boto:Method: GET 2023-11-02T18:45:27.710 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:45:27.710 DEBUG:boto:Data: 2023-11-02T18:45:27.711 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.711 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.712 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.712 DEBUG:boto:Params: {} 2023-11-02T18:45:27.713 DEBUG:boto:Token: None 2023-11-02T18:45:27.713 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.714 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:27.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d7974578ba5bddb-006543ee47-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.725 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:27.726 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:27.727 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:27.727 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.728 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:45:27.728 DEBUG:boto:Data: 2023-11-02T18:45:27.729 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.729 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.730 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.730 DEBUG:boto:Params: {} 2023-11-02T18:45:27.731 DEBUG:boto:Token: None 2023-11-02T18:45:27.732 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj23 2023-11-02T18:45:27.732 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+BlF78AcDfHvF+hQb1MnfXRgSrg= 2023-11-02T18:45:27.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+BlF78AcDfHvF+hQb1MnfXRgSrg=', 'Content-Length': '0'} 2023-11-02T18:45:27.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000209aac3692d64ffe-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.736 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:27.736 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:27.737 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.737 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:45:27.738 DEBUG:boto:Data: 2023-11-02T18:45:27.738 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.739 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.739 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.740 DEBUG:boto:Params: {} 2023-11-02T18:45:27.740 DEBUG:boto:Token: None 2023-11-02T18:45:27.741 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj23 2023-11-02T18:45:27.741 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+BlF78AcDfHvF+hQb1MnfXRgSrg= 2023-11-02T18:45:27.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+BlF78AcDfHvF+hQb1MnfXRgSrg=', 'Content-Length': '0'} 2023-11-02T18:45:27.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d342328571005446-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:45:27.746 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:27.746 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:27.747 DEBUG:boto:Method: GET 2023-11-02T18:45:27.747 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:45:27.748 DEBUG:boto:Data: 2023-11-02T18:45:27.748 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.749 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.749 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.750 DEBUG:boto:Params: {} 2023-11-02T18:45:27.753 DEBUG:boto:Token: None 2023-11-02T18:45:27.754 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj23 2023-11-02T18:45:27.754 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2M1KY79RPof8qcT/5CKA8s3h5NE= 2023-11-02T18:45:27.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2M1KY79RPof8qcT/5CKA8s3h5NE=', 'Content-Length': '0'} 2023-11-02T18:45:27.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006babb149fb5bd413-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.758 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:27.759 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:27.759 DEBUG:boto:Method: GET 2023-11-02T18:45:27.760 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:45:27.760 DEBUG:boto:Data: 2023-11-02T18:45:27.761 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.761 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.762 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.762 DEBUG:boto:Params: {} 2023-11-02T18:45:27.763 DEBUG:boto:Token: None 2023-11-02T18:45:27.763 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj23 2023-11-02T18:45:27.764 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2M1KY79RPof8qcT/5CKA8s3h5NE= 2023-11-02T18:45:27.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2M1KY79RPof8qcT/5CKA8s3h5NE=', 'Content-Length': '0'} 2023-11-02T18:45:27.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014bcef1aacf80414-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.810 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.811 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.812 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:27.812 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:27.812 DEBUG:boto:Method: GET 2023-11-02T18:45:27.813 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:45:27.814 DEBUG:boto:Data: 2023-11-02T18:45:27.814 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.815 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.815 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.816 DEBUG:boto:Params: {} 2023-11-02T18:45:27.816 DEBUG:boto:Token: None 2023-11-02T18:45:27.817 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.817 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:27.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000001be062cbe185bb-006543ee47-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.824 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:27.824 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.825 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.825 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:27.826 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:27.826 DEBUG:boto:Method: GET 2023-11-02T18:45:27.827 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:45:27.827 DEBUG:boto:Data: 2023-11-02T18:45:27.828 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.828 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.829 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.829 DEBUG:boto:Params: {} 2023-11-02T18:45:27.830 DEBUG:boto:Token: None 2023-11-02T18:45:27.830 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.831 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:27.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c1c794eb24fe2ea-006543ee47-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.842 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:27.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:45:27.843 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:27.844 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:27.844 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.845 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.845 DEBUG:boto:Method: GET 2023-11-02T18:45:27.846 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.846 DEBUG:boto:Data: 2023-11-02T18:45:27.847 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.847 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.847 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.848 DEBUG:boto:Params: {} 2023-11-02T18:45:27.848 DEBUG:boto:Token: None 2023-11-02T18:45:27.849 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.850 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WIsN+M3D+CSUnlfF/ENG4MolkdU= 2023-11-02T18:45:27.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WIsN+M3D+CSUnlfF/ENG4MolkdU=', 'Content-Length': '0'} 2023-11-02T18:45:27.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bee114945abb52c-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.894 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:27.895 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:27.895 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.896 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.896 DEBUG:boto:Method: GET 2023-11-02T18:45:27.897 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.897 DEBUG:boto:Data: 2023-11-02T18:45:27.898 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.898 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.899 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.899 DEBUG:boto:Params: {} 2023-11-02T18:45:27.900 DEBUG:boto:Token: None 2023-11-02T18:45:27.900 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.901 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WIsN+M3D+CSUnlfF/ENG4MolkdU= 2023-11-02T18:45:27.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WIsN+M3D+CSUnlfF/ENG4MolkdU=', 'Content-Length': '0'} 2023-11-02T18:45:27.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000122aa5f2133f223c-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.907 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:27.907 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:27.908 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.909 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.909 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.910 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.910 DEBUG:boto:Data: 2023-11-02T18:45:27.911 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.911 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.912 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.912 DEBUG:boto:Params: {} 2023-11-02T18:45:27.912 DEBUG:boto:Token: None 2023-11-02T18:45:27.913 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.913 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XAjM8dArMmlgygP25XUtkqEEdJQ= 2023-11-02T18:45:27.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XAjM8dArMmlgygP25XUtkqEEdJQ=', 'Content-Length': '0'} 2023-11-02T18:45:27.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd44d8d130dd5483-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.918 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:27.918 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:27.919 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.919 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.919 DEBUG:boto:Method: HEAD 2023-11-02T18:45:27.920 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.920 DEBUG:boto:Data: 2023-11-02T18:45:27.921 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.921 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.921 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.922 DEBUG:boto:Params: {} 2023-11-02T18:45:27.922 DEBUG:boto:Token: None 2023-11-02T18:45:27.923 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.923 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XAjM8dArMmlgygP25XUtkqEEdJQ= 2023-11-02T18:45:27.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XAjM8dArMmlgygP25XUtkqEEdJQ=', 'Content-Length': '0'} 2023-11-02T18:45:27.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fd79251e430939a-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:45:27.928 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:27.928 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:27.929 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.929 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.930 DEBUG:boto:Method: GET 2023-11-02T18:45:27.930 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.931 DEBUG:boto:Data: 2023-11-02T18:45:27.931 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.932 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.932 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.932 DEBUG:boto:Params: {} 2023-11-02T18:45:27.933 DEBUG:boto:Token: None 2023-11-02T18:45:27.933 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.934 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WIsN+M3D+CSUnlfF/ENG4MolkdU= 2023-11-02T18:45:27.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WIsN+M3D+CSUnlfF/ENG4MolkdU=', 'Content-Length': '0'} 2023-11-02T18:45:27.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003df92d556c3a9e14-006543ee47-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.938 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:27.939 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:27.939 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.940 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.940 DEBUG:boto:Method: GET 2023-11-02T18:45:27.941 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.941 DEBUG:boto:Data: 2023-11-02T18:45:27.941 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.942 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:27.942 DEBUG:boto:Port: 8000 2023-11-02T18:45:27.943 DEBUG:boto:Params: {} 2023-11-02T18:45:27.943 DEBUG:boto:Token: None 2023-11-02T18:45:27.944 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:27.944 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WIsN+M3D+CSUnlfF/ENG4MolkdU= 2023-11-02T18:45:27.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WIsN+M3D+CSUnlfF/ENG4MolkdU=', 'Content-Length': '0'} 2023-11-02T18:45:27.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ff254a61d6a0b1-006543ee47-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:27.990 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:27.991 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:27.991 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:27.992 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:27.992 DEBUG:boto:Method: GET 2023-11-02T18:45:27.993 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:45:27.993 DEBUG:boto:Data: 2023-11-02T18:45:27.994 DEBUG:boto:Headers: {} 2023-11-02T18:45:27.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:27.995 DEBUG:boto:Port: 8001 2023-11-02T18:45:27.995 DEBUG:boto:Params: {} 2023-11-02T18:45:27.996 DEBUG:boto:Token: None 2023-11-02T18:45:27.996 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:27 GMT /ibtljn-59/ 2023-11-02T18:45:27.997 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk= 2023-11-02T18:45:27.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:27 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oklKpdYcZj1KwDp100YTeOx68pk=', 'Content-Length': '0'} 2023-11-02T18:45:28.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9e4582b15234fe3-006543ee47-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.003 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.003 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.004 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.004 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:28.005 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:28.005 DEBUG:boto:Method: GET 2023-11-02T18:45:28.006 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:45:28.006 DEBUG:boto:Data: 2023-11-02T18:45:28.006 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.007 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.007 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.008 DEBUG:boto:Params: {} 2023-11-02T18:45:28.008 DEBUG:boto:Token: None 2023-11-02T18:45:28.009 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.009 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000361e820bc15e7647-006543ee48-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.021 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.022 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:28.022 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:28.023 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.023 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:45:28.024 DEBUG:boto:Data: 2023-11-02T18:45:28.024 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.024 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.025 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.025 DEBUG:boto:Params: {} 2023-11-02T18:45:28.026 DEBUG:boto:Token: None 2023-11-02T18:45:28.026 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj29 2023-11-02T18:45:28.027 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gaUDx1pfQS6fr5KK/milLn0jJWI= 2023-11-02T18:45:28.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gaUDx1pfQS6fr5KK/milLn0jJWI=', 'Content-Length': '0'} 2023-11-02T18:45:28.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000272f8db1a941754c-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.030 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:28.031 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:28.031 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.032 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:45:28.032 DEBUG:boto:Data: 2023-11-02T18:45:28.033 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.033 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.033 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.034 DEBUG:boto:Params: {} 2023-11-02T18:45:28.034 DEBUG:boto:Token: None 2023-11-02T18:45:28.035 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj29 2023-11-02T18:45:28.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gaUDx1pfQS6fr5KK/milLn0jJWI= 2023-11-02T18:45:28.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gaUDx1pfQS6fr5KK/milLn0jJWI=', 'Content-Length': '0'} 2023-11-02T18:45:28.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4b9ea146089f496-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:45:28.040 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:28.040 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:28.040 DEBUG:boto:Method: GET 2023-11-02T18:45:28.041 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:45:28.041 DEBUG:boto:Data: 2023-11-02T18:45:28.042 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.042 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.043 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.043 DEBUG:boto:Params: {} 2023-11-02T18:45:28.044 DEBUG:boto:Token: None 2023-11-02T18:45:28.044 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj29 2023-11-02T18:45:28.045 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rZZa/e1NINjyTmjgv+8v1gVKrhI= 2023-11-02T18:45:28.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rZZa/e1NINjyTmjgv+8v1gVKrhI=', 'Content-Length': '0'} 2023-11-02T18:45:28.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd38c4343ed263f5-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.049 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:28.049 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:28.051 DEBUG:boto:Method: GET 2023-11-02T18:45:28.051 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:45:28.052 DEBUG:boto:Data: 2023-11-02T18:45:28.052 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.053 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.053 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.054 DEBUG:boto:Params: {} 2023-11-02T18:45:28.054 DEBUG:boto:Token: None 2023-11-02T18:45:28.055 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj29 2023-11-02T18:45:28.055 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rZZa/e1NINjyTmjgv+8v1gVKrhI= 2023-11-02T18:45:28.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rZZa/e1NINjyTmjgv+8v1gVKrhI=', 'Content-Length': '0'} 2023-11-02T18:45:28.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008571228c77643688-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.108 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.108 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.109 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:28.109 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:28.110 DEBUG:boto:Method: GET 2023-11-02T18:45:28.110 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:45:28.111 DEBUG:boto:Data: 2023-11-02T18:45:28.111 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.112 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.112 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.113 DEBUG:boto:Params: {} 2023-11-02T18:45:28.113 DEBUG:boto:Token: None 2023-11-02T18:45:28.114 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.114 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6a5786ef8796c37-006543ee48-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.120 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.121 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.121 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.122 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:28.122 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:28.123 DEBUG:boto:Method: GET 2023-11-02T18:45:28.123 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:45:28.124 DEBUG:boto:Data: 2023-11-02T18:45:28.124 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.125 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.125 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.126 DEBUG:boto:Params: {} 2023-11-02T18:45:28.126 DEBUG:boto:Token: None 2023-11-02T18:45:28.127 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.127 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004b97ffc4a1d3956-006543ee48-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.138 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T18:45:28.140 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:28.141 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:28.141 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.142 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.142 DEBUG:boto:Method: GET 2023-11-02T18:45:28.143 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.143 DEBUG:boto:Data: 2023-11-02T18:45:28.143 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.144 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.144 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.145 DEBUG:boto:Params: {} 2023-11-02T18:45:28.145 DEBUG:boto:Token: None 2023-11-02T18:45:28.146 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.146 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NpPZbFtWfGtxjI037kDJ6OknmmE= 2023-11-02T18:45:28.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NpPZbFtWfGtxjI037kDJ6OknmmE=', 'Content-Length': '0'} 2023-11-02T18:45:28.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc62038b1c28373f-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.190 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:28.191 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:28.191 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.192 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.192 DEBUG:boto:Method: GET 2023-11-02T18:45:28.193 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.193 DEBUG:boto:Data: 2023-11-02T18:45:28.194 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.194 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.195 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.195 DEBUG:boto:Params: {} 2023-11-02T18:45:28.196 DEBUG:boto:Token: None 2023-11-02T18:45:28.196 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.197 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NpPZbFtWfGtxjI037kDJ6OknmmE= 2023-11-02T18:45:28.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NpPZbFtWfGtxjI037kDJ6OknmmE=', 'Content-Length': '0'} 2023-11-02T18:45:28.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093246020a5b6cd5d-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.201 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:28.202 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:28.203 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.203 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.204 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.204 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.205 DEBUG:boto:Data: 2023-11-02T18:45:28.205 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.206 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.206 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.207 DEBUG:boto:Params: {} 2023-11-02T18:45:28.207 DEBUG:boto:Token: None 2023-11-02T18:45:28.208 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JOKTMfpvcWiYXhW9JHuRqnyDp7c= 2023-11-02T18:45:28.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JOKTMfpvcWiYXhW9JHuRqnyDp7c=', 'Content-Length': '0'} 2023-11-02T18:45:28.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a952e6455dcedb76-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.212 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:28.213 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:28.213 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.214 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.214 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.214 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.215 DEBUG:boto:Data: 2023-11-02T18:45:28.215 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.216 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.216 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.217 DEBUG:boto:Params: {} 2023-11-02T18:45:28.217 DEBUG:boto:Token: None 2023-11-02T18:45:28.218 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.218 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JOKTMfpvcWiYXhW9JHuRqnyDp7c= 2023-11-02T18:45:28.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JOKTMfpvcWiYXhW9JHuRqnyDp7c=', 'Content-Length': '0'} 2023-11-02T18:45:28.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ff4b62ef5f92633-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T18:45:28.223 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:28.223 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:28.224 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.224 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.225 DEBUG:boto:Method: GET 2023-11-02T18:45:28.225 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.225 DEBUG:boto:Data: 2023-11-02T18:45:28.226 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.226 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.227 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.227 DEBUG:boto:Params: {} 2023-11-02T18:45:28.228 DEBUG:boto:Token: None 2023-11-02T18:45:28.228 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.229 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NpPZbFtWfGtxjI037kDJ6OknmmE= 2023-11-02T18:45:28.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NpPZbFtWfGtxjI037kDJ6OknmmE=', 'Content-Length': '0'} 2023-11-02T18:45:28.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d077a5cd7c7a75e-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.233 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:28.233 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:28.234 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.235 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.235 DEBUG:boto:Method: GET 2023-11-02T18:45:28.236 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.236 DEBUG:boto:Data: 2023-11-02T18:45:28.236 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.237 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.237 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.238 DEBUG:boto:Params: {} 2023-11-02T18:45:28.238 DEBUG:boto:Token: None 2023-11-02T18:45:28.239 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:28.239 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NpPZbFtWfGtxjI037kDJ6OknmmE= 2023-11-02T18:45:28.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NpPZbFtWfGtxjI037kDJ6OknmmE=', 'Content-Length': '0'} 2023-11-02T18:45:28.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed9d19be9399b26e-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.244 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.244 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.245 DEBUG:boto:path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:28.245 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:28.245 DEBUG:boto:Method: GET 2023-11-02T18:45:28.246 DEBUG:boto:Path: /ibtljn-59/?prefix=obj31 2023-11-02T18:45:28.246 DEBUG:boto:Data: 2023-11-02T18:45:28.247 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.247 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.248 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.248 DEBUG:boto:Params: {} 2023-11-02T18:45:28.249 DEBUG:boto:Token: None 2023-11-02T18:45:28.249 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.250 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adf40b66aa637ebf-006543ee48-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.256 DEBUG:boto:b'ibtljn-59obj311000falseobj312023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.256 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.257 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.257 DEBUG:boto:path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:28.258 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:28.258 DEBUG:boto:Method: GET 2023-11-02T18:45:28.259 DEBUG:boto:Path: /ibtljn-59/?prefix=obj31 2023-11-02T18:45:28.259 DEBUG:boto:Data: 2023-11-02T18:45:28.259 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.260 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.260 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.261 DEBUG:boto:Params: {} 2023-11-02T18:45:28.261 DEBUG:boto:Token: None 2023-11-02T18:45:28.262 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.262 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc72d8873e03c2c0-006543ee48-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.314 DEBUG:boto:b'ibtljn-59obj311000falseobj312023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.315 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:28.316 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:28.316 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.317 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:45:28.317 DEBUG:boto:Data: 2023-11-02T18:45:28.317 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.318 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.318 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.319 DEBUG:boto:Params: {} 2023-11-02T18:45:28.319 DEBUG:boto:Token: None 2023-11-02T18:45:28.320 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj31 2023-11-02T18:45:28.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N94FG8ZN0WpYjai274PAIp2RKto= 2023-11-02T18:45:28.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N94FG8ZN0WpYjai274PAIp2RKto=', 'Content-Length': '0'} 2023-11-02T18:45:28.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000503c921db048adec-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.324 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:28.325 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:28.325 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.325 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:45:28.326 DEBUG:boto:Data: 2023-11-02T18:45:28.326 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.327 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.327 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.327 DEBUG:boto:Params: {} 2023-11-02T18:45:28.328 DEBUG:boto:Token: None 2023-11-02T18:45:28.328 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj31 2023-11-02T18:45:28.329 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N94FG8ZN0WpYjai274PAIp2RKto= 2023-11-02T18:45:28.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N94FG8ZN0WpYjai274PAIp2RKto=', 'Content-Length': '0'} 2023-11-02T18:45:28.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6c49e7551ee6511-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T18:45:28.334 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:28.334 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:28.335 DEBUG:boto:Method: GET 2023-11-02T18:45:28.335 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:45:28.336 DEBUG:boto:Data: 2023-11-02T18:45:28.336 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.336 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.337 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.337 DEBUG:boto:Params: {} 2023-11-02T18:45:28.338 DEBUG:boto:Token: None 2023-11-02T18:45:28.339 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj31 2023-11-02T18:45:28.340 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qL1Eeur5uqtNFEoowD4ygDMpTgI= 2023-11-02T18:45:28.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qL1Eeur5uqtNFEoowD4ygDMpTgI=', 'Content-Length': '0'} 2023-11-02T18:45:28.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6576ffb5396875d-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.386 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:28.387 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:28.387 DEBUG:boto:Method: GET 2023-11-02T18:45:28.388 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:45:28.388 DEBUG:boto:Data: 2023-11-02T18:45:28.389 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.389 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.390 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.390 DEBUG:boto:Params: {} 2023-11-02T18:45:28.390 DEBUG:boto:Token: None 2023-11-02T18:45:28.391 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj31 2023-11-02T18:45:28.391 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qL1Eeur5uqtNFEoowD4ygDMpTgI= 2023-11-02T18:45:28.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qL1Eeur5uqtNFEoowD4ygDMpTgI=', 'Content-Length': '0'} 2023-11-02T18:45:28.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054a959d021eac38f-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.396 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.396 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.396 DEBUG:boto:path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:28.397 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:28.397 DEBUG:boto:Method: GET 2023-11-02T18:45:28.398 DEBUG:boto:Path: /ibtljn-59/?prefix=obj31 2023-11-02T18:45:28.398 DEBUG:boto:Data: 2023-11-02T18:45:28.399 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.399 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.400 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.400 DEBUG:boto:Params: {} 2023-11-02T18:45:28.401 DEBUG:boto:Token: None 2023-11-02T18:45:28.401 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.402 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b61d090d801d0d5-006543ee48-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.408 DEBUG:boto:b'ibtljn-59obj311000falseobj312023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.408 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.409 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.409 DEBUG:boto:path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:28.410 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:28.410 DEBUG:boto:Method: GET 2023-11-02T18:45:28.410 DEBUG:boto:Path: /ibtljn-59/?prefix=obj31 2023-11-02T18:45:28.411 DEBUG:boto:Data: 2023-11-02T18:45:28.411 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.412 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.413 DEBUG:boto:Params: {} 2023-11-02T18:45:28.413 DEBUG:boto:Token: None 2023-11-02T18:45:28.414 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006105c7b380de4026-006543ee48-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.425 DEBUG:boto:b'ibtljn-59obj311000falseobj312023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:45:28.426 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:28.427 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:28.427 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.427 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.428 DEBUG:boto:Method: GET 2023-11-02T18:45:28.428 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.429 DEBUG:boto:Data: 2023-11-02T18:45:28.429 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.430 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.430 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.431 DEBUG:boto:Params: {} 2023-11-02T18:45:28.431 DEBUG:boto:Token: None 2023-11-02T18:45:28.432 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.432 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ObF+/460TxRF9pPBvRx/n3rj04I= 2023-11-02T18:45:28.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ObF+/460TxRF9pPBvRx/n3rj04I=', 'Content-Length': '0'} 2023-11-02T18:45:28.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d62d901b39f30c8-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.436 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:28.436 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:28.437 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.437 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.438 DEBUG:boto:Method: GET 2023-11-02T18:45:28.438 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.438 DEBUG:boto:Data: 2023-11-02T18:45:28.439 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.439 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.440 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.440 DEBUG:boto:Params: {} 2023-11-02T18:45:28.441 DEBUG:boto:Token: None 2023-11-02T18:45:28.441 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.442 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ObF+/460TxRF9pPBvRx/n3rj04I= 2023-11-02T18:45:28.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ObF+/460TxRF9pPBvRx/n3rj04I=', 'Content-Length': '0'} 2023-11-02T18:45:28.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024fdfd9e2a33d2e9-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.486 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:28.487 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:28.488 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.488 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.488 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.489 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.489 DEBUG:boto:Data: 2023-11-02T18:45:28.490 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.490 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.491 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.491 DEBUG:boto:Params: {} 2023-11-02T18:45:28.492 DEBUG:boto:Token: None 2023-11-02T18:45:28.492 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.493 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A5EtokwMeXI3MAF0aAagi8vyihI= 2023-11-02T18:45:28.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A5EtokwMeXI3MAF0aAagi8vyihI=', 'Content-Length': '0'} 2023-11-02T18:45:28.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051d13f1d39c9b18c-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.497 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:28.498 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:28.498 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.499 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.499 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.499 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.500 DEBUG:boto:Data: 2023-11-02T18:45:28.500 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.501 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.501 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.501 DEBUG:boto:Params: {} 2023-11-02T18:45:28.502 DEBUG:boto:Token: None 2023-11-02T18:45:28.502 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.503 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A5EtokwMeXI3MAF0aAagi8vyihI= 2023-11-02T18:45:28.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A5EtokwMeXI3MAF0aAagi8vyihI=', 'Content-Length': '0'} 2023-11-02T18:45:28.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ba77ba5d2b166a7-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:45:28.508 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:28.508 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:28.508 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.509 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.509 DEBUG:boto:Method: GET 2023-11-02T18:45:28.510 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.510 DEBUG:boto:Data: 2023-11-02T18:45:28.511 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.511 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.512 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.512 DEBUG:boto:Params: {} 2023-11-02T18:45:28.513 DEBUG:boto:Token: None 2023-11-02T18:45:28.513 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.514 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ObF+/460TxRF9pPBvRx/n3rj04I= 2023-11-02T18:45:28.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ObF+/460TxRF9pPBvRx/n3rj04I=', 'Content-Length': '0'} 2023-11-02T18:45:28.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000943dc4dc759a20d6-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.518 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:28.518 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:28.519 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.519 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.520 DEBUG:boto:Method: GET 2023-11-02T18:45:28.520 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.521 DEBUG:boto:Data: 2023-11-02T18:45:28.521 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.522 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.523 DEBUG:boto:Params: {} 2023-11-02T18:45:28.523 DEBUG:boto:Token: None 2023-11-02T18:45:28.524 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:28.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ObF+/460TxRF9pPBvRx/n3rj04I= 2023-11-02T18:45:28.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ObF+/460TxRF9pPBvRx/n3rj04I=', 'Content-Length': '0'} 2023-11-02T18:45:28.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008254e7001ec2173a-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.528 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.529 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.529 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:28.530 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:28.530 DEBUG:boto:Method: GET 2023-11-02T18:45:28.531 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:45:28.531 DEBUG:boto:Data: 2023-11-02T18:45:28.532 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.532 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.533 DEBUG:boto:Params: {} 2023-11-02T18:45:28.533 DEBUG:boto:Token: None 2023-11-02T18:45:28.534 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.534 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d75d0f08df9b60ff-006543ee48-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.540 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.541 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.541 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.542 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:28.543 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:28.543 DEBUG:boto:Method: GET 2023-11-02T18:45:28.544 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:45:28.544 DEBUG:boto:Data: 2023-11-02T18:45:28.545 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.545 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.546 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.546 DEBUG:boto:Params: {} 2023-11-02T18:45:28.547 DEBUG:boto:Token: None 2023-11-02T18:45:28.547 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.548 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007743f5f340827577-006543ee48-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.602 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.603 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:28.604 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:28.604 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.605 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:45:28.605 DEBUG:boto:Data: 2023-11-02T18:45:28.606 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.606 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.607 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.607 DEBUG:boto:Params: {} 2023-11-02T18:45:28.607 DEBUG:boto:Token: None 2023-11-02T18:45:28.608 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj33 2023-11-02T18:45:28.609 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rbiqvErkYTgTe0AR6GVwbJwgiHM= 2023-11-02T18:45:28.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rbiqvErkYTgTe0AR6GVwbJwgiHM=', 'Content-Length': '0'} 2023-11-02T18:45:28.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee50799d118244f7-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.613 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:28.613 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:28.613 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.614 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:45:28.614 DEBUG:boto:Data: 2023-11-02T18:45:28.615 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.615 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.616 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.616 DEBUG:boto:Params: {} 2023-11-02T18:45:28.617 DEBUG:boto:Token: None 2023-11-02T18:45:28.617 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj33 2023-11-02T18:45:28.618 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rbiqvErkYTgTe0AR6GVwbJwgiHM= 2023-11-02T18:45:28.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rbiqvErkYTgTe0AR6GVwbJwgiHM=', 'Content-Length': '0'} 2023-11-02T18:45:28.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd8e173080f45bbc-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:45:28.622 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:28.623 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:28.623 DEBUG:boto:Method: GET 2023-11-02T18:45:28.624 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:45:28.624 DEBUG:boto:Data: 2023-11-02T18:45:28.625 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.625 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.626 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.626 DEBUG:boto:Params: {} 2023-11-02T18:45:28.627 DEBUG:boto:Token: None 2023-11-02T18:45:28.627 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj33 2023-11-02T18:45:28.628 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lWLrCL4yPLt1MPAvzg327Pmd9sg= 2023-11-02T18:45:28.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lWLrCL4yPLt1MPAvzg327Pmd9sg=', 'Content-Length': '0'} 2023-11-02T18:45:28.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffa45a037f95a7de-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.678 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:28.679 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:28.679 DEBUG:boto:Method: GET 2023-11-02T18:45:28.680 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:45:28.680 DEBUG:boto:Data: 2023-11-02T18:45:28.681 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.681 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.682 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.682 DEBUG:boto:Params: {} 2023-11-02T18:45:28.683 DEBUG:boto:Token: None 2023-11-02T18:45:28.683 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj33 2023-11-02T18:45:28.684 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lWLrCL4yPLt1MPAvzg327Pmd9sg= 2023-11-02T18:45:28.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lWLrCL4yPLt1MPAvzg327Pmd9sg=', 'Content-Length': '0'} 2023-11-02T18:45:28.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000659f9d6c0db375b3-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.688 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.688 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.689 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:28.689 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:28.690 DEBUG:boto:Method: GET 2023-11-02T18:45:28.690 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:45:28.690 DEBUG:boto:Data: 2023-11-02T18:45:28.691 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.691 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.692 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.692 DEBUG:boto:Params: {} 2023-11-02T18:45:28.693 DEBUG:boto:Token: None 2023-11-02T18:45:28.693 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.694 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004746a93d43087256-006543ee48-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.700 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.701 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.702 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.702 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:28.702 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:28.703 DEBUG:boto:Method: GET 2023-11-02T18:45:28.703 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:45:28.704 DEBUG:boto:Data: 2023-11-02T18:45:28.704 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.705 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.705 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.706 DEBUG:boto:Params: {} 2023-11-02T18:45:28.706 DEBUG:boto:Token: None 2023-11-02T18:45:28.707 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.707 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000074af46d76bfffe-006543ee48-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.717 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:45:28.718 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:28.718 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:28.719 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.719 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.720 DEBUG:boto:Method: GET 2023-11-02T18:45:28.720 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.720 DEBUG:boto:Data: 2023-11-02T18:45:28.721 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.721 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.722 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.722 DEBUG:boto:Params: {} 2023-11-02T18:45:28.724 DEBUG:boto:Token: None 2023-11-02T18:45:28.724 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.725 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JYOTVU2ZC0UQYdLkd9Tz03AY56s= 2023-11-02T18:45:28.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JYOTVU2ZC0UQYdLkd9Tz03AY56s=', 'Content-Length': '0'} 2023-11-02T18:45:28.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c2025b8b4990163-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.729 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:28.729 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:28.730 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.730 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.731 DEBUG:boto:Method: GET 2023-11-02T18:45:28.731 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.732 DEBUG:boto:Data: 2023-11-02T18:45:28.732 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.733 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.733 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.734 DEBUG:boto:Params: {} 2023-11-02T18:45:28.734 DEBUG:boto:Token: None 2023-11-02T18:45:28.735 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.735 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JYOTVU2ZC0UQYdLkd9Tz03AY56s= 2023-11-02T18:45:28.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JYOTVU2ZC0UQYdLkd9Tz03AY56s=', 'Content-Length': '0'} 2023-11-02T18:45:28.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b31807bb1110293-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.782 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:28.783 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:28.784 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.784 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.785 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.785 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.785 DEBUG:boto:Data: 2023-11-02T18:45:28.786 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.786 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.787 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.787 DEBUG:boto:Params: {} 2023-11-02T18:45:28.788 DEBUG:boto:Token: None 2023-11-02T18:45:28.788 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzB+X3O89PpBxim+0YizqLUQ/sg= 2023-11-02T18:45:28.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzB+X3O89PpBxim+0YizqLUQ/sg=', 'Content-Length': '0'} 2023-11-02T18:45:28.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7211e21c2019965-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.793 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:28.793 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:28.794 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.794 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.795 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.795 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.795 DEBUG:boto:Data: 2023-11-02T18:45:28.796 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.797 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.797 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.798 DEBUG:boto:Params: {} 2023-11-02T18:45:28.798 DEBUG:boto:Token: None 2023-11-02T18:45:28.799 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.799 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vzB+X3O89PpBxim+0YizqLUQ/sg= 2023-11-02T18:45:28.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vzB+X3O89PpBxim+0YizqLUQ/sg=', 'Content-Length': '0'} 2023-11-02T18:45:28.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c01df4f0dd26566-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:45:28.804 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:28.804 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:28.804 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.805 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.805 DEBUG:boto:Method: GET 2023-11-02T18:45:28.806 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.806 DEBUG:boto:Data: 2023-11-02T18:45:28.807 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.807 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.808 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.808 DEBUG:boto:Params: {} 2023-11-02T18:45:28.809 DEBUG:boto:Token: None 2023-11-02T18:45:28.809 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.810 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JYOTVU2ZC0UQYdLkd9Tz03AY56s= 2023-11-02T18:45:28.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JYOTVU2ZC0UQYdLkd9Tz03AY56s=', 'Content-Length': '0'} 2023-11-02T18:45:28.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2be07fd89086005-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.858 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:28.859 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:28.859 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.860 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.860 DEBUG:boto:Method: GET 2023-11-02T18:45:28.861 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.861 DEBUG:boto:Data: 2023-11-02T18:45:28.862 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.862 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.863 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.863 DEBUG:boto:Params: {} 2023-11-02T18:45:28.864 DEBUG:boto:Token: None 2023-11-02T18:45:28.864 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:28.865 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JYOTVU2ZC0UQYdLkd9Tz03AY56s= 2023-11-02T18:45:28.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JYOTVU2ZC0UQYdLkd9Tz03AY56s=', 'Content-Length': '0'} 2023-11-02T18:45:28.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d47be16dd02dde3d-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.869 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.870 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.870 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:28.871 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:28.871 DEBUG:boto:Method: GET 2023-11-02T18:45:28.871 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:45:28.872 DEBUG:boto:Data: 2023-11-02T18:45:28.872 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.873 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.873 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.874 DEBUG:boto:Params: {} 2023-11-02T18:45:28.874 DEBUG:boto:Token: None 2023-11-02T18:45:28.875 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.875 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd5af3b147021bd2-006543ee48-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.881 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.882 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.882 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.883 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:28.883 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:28.883 DEBUG:boto:Method: GET 2023-11-02T18:45:28.884 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:45:28.884 DEBUG:boto:Data: 2023-11-02T18:45:28.885 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.885 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.886 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.886 DEBUG:boto:Params: {} 2023-11-02T18:45:28.886 DEBUG:boto:Token: None 2023-11-02T18:45:28.887 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.888 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fc22a1b49ad0325-006543ee48-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.898 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.899 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:28.900 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:28.900 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.901 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:45:28.901 DEBUG:boto:Data: 2023-11-02T18:45:28.901 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.902 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.902 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.903 DEBUG:boto:Params: {} 2023-11-02T18:45:28.903 DEBUG:boto:Token: None 2023-11-02T18:45:28.904 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj34 2023-11-02T18:45:28.904 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pRrf2UEeGrlI97wPC8NOuLD3i9I= 2023-11-02T18:45:28.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pRrf2UEeGrlI97wPC8NOuLD3i9I=', 'Content-Length': '0'} 2023-11-02T18:45:28.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072e7ecf9e5502c5f-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.908 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:28.908 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:28.909 DEBUG:boto:Method: HEAD 2023-11-02T18:45:28.909 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:45:28.909 DEBUG:boto:Data: 2023-11-02T18:45:28.910 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.911 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.911 DEBUG:boto:Params: {} 2023-11-02T18:45:28.912 DEBUG:boto:Token: None 2023-11-02T18:45:28.912 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj34 2023-11-02T18:45:28.912 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pRrf2UEeGrlI97wPC8NOuLD3i9I= 2023-11-02T18:45:28.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pRrf2UEeGrlI97wPC8NOuLD3i9I=', 'Content-Length': '0'} 2023-11-02T18:45:28.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000037da859cf78bb99-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:45:28.916 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:28.917 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:28.917 DEBUG:boto:Method: GET 2023-11-02T18:45:28.918 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:45:28.918 DEBUG:boto:Data: 2023-11-02T18:45:28.918 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.919 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.919 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.920 DEBUG:boto:Params: {} 2023-11-02T18:45:28.920 DEBUG:boto:Token: None 2023-11-02T18:45:28.921 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj34 2023-11-02T18:45:28.921 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u/hKnzoww8dcin1pekMqUDkCvwM= 2023-11-02T18:45:28.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u/hKnzoww8dcin1pekMqUDkCvwM=', 'Content-Length': '0'} 2023-11-02T18:45:28.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e5fe92a5c0b8d2f-006543ee48-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.970 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:28.971 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:28.971 DEBUG:boto:Method: GET 2023-11-02T18:45:28.973 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:45:28.974 DEBUG:boto:Data: 2023-11-02T18:45:28.974 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.975 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.975 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.975 DEBUG:boto:Params: {} 2023-11-02T18:45:28.976 DEBUG:boto:Token: None 2023-11-02T18:45:28.976 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/obj34 2023-11-02T18:45:28.977 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u/hKnzoww8dcin1pekMqUDkCvwM= 2023-11-02T18:45:28.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u/hKnzoww8dcin1pekMqUDkCvwM=', 'Content-Length': '0'} 2023-11-02T18:45:28.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029f20e74fd5ebb24-006543ee48-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.981 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.982 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.982 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:28.983 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:28.983 DEBUG:boto:Method: GET 2023-11-02T18:45:28.984 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:45:28.984 DEBUG:boto:Data: 2023-11-02T18:45:28.984 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.985 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:28.985 DEBUG:boto:Port: 8001 2023-11-02T18:45:28.986 DEBUG:boto:Params: {} 2023-11-02T18:45:28.986 DEBUG:boto:Token: None 2023-11-02T18:45:28.987 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.987 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:28.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:28.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004697650fe07b73fb-006543ee48-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:28.993 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:28.993 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:28.994 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:28.994 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:28.995 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:28.995 DEBUG:boto:Method: GET 2023-11-02T18:45:28.996 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:45:28.996 DEBUG:boto:Data: 2023-11-02T18:45:28.997 DEBUG:boto:Headers: {} 2023-11-02T18:45:28.997 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:28.997 DEBUG:boto:Port: 8000 2023-11-02T18:45:28.998 DEBUG:boto:Params: {} 2023-11-02T18:45:28.998 DEBUG:boto:Token: None 2023-11-02T18:45:28.999 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:28 GMT /ibtljn-59/ 2023-11-02T18:45:28.999 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA= 2023-11-02T18:45:29.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:28 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VlO5PvCRLCk04o3852pzrDQcpZA=', 'Content-Length': '0'} 2023-11-02T18:45:29.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc317d88598e84e5-006543ee48-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.008 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-11-02T18:45:29.010 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:29.010 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:29.010 DEBUG:boto:path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.011 DEBUG:boto:auth_path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.011 DEBUG:boto:Method: GET 2023-11-02T18:45:29.012 DEBUG:boto:Path: /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.012 DEBUG:boto:Data: 2023-11-02T18:45:29.013 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.013 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.013 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.014 DEBUG:boto:Params: {} 2023-11-02T18:45:29.014 DEBUG:boto:Token: None 2023-11-02T18:45:29.015 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.015 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pAA7pqLoOSgJizBrPPeT3Iv8lx0= 2023-11-02T18:45:29.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pAA7pqLoOSgJizBrPPeT3Iv8lx0=', 'Content-Length': '0'} 2023-11-02T18:45:29.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d469fd59a35158a-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.019 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:29.020 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:29.020 DEBUG:boto:path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.021 DEBUG:boto:auth_path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.021 DEBUG:boto:Method: GET 2023-11-02T18:45:29.022 DEBUG:boto:Path: /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.022 DEBUG:boto:Data: 2023-11-02T18:45:29.022 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.023 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.023 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.024 DEBUG:boto:Params: {} 2023-11-02T18:45:29.024 DEBUG:boto:Token: None 2023-11-02T18:45:29.025 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.025 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pAA7pqLoOSgJizBrPPeT3Iv8lx0= 2023-11-02T18:45:29.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pAA7pqLoOSgJizBrPPeT3Iv8lx0=', 'Content-Length': '0'} 2023-11-02T18:45:29.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fadd7309d5e53259-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.029 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:29.030 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:29.030 DEBUG:boto:path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.030 DEBUG:boto:auth_path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.031 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.031 DEBUG:boto:Path: /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.032 DEBUG:boto:Data: 2023-11-02T18:45:29.032 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.033 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.033 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.033 DEBUG:boto:Params: {} 2023-11-02T18:45:29.034 DEBUG:boto:Token: None 2023-11-02T18:45:29.034 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bJijgOkYsHP7+wSIO22Zv7FvJGI= 2023-11-02T18:45:29.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bJijgOkYsHP7+wSIO22Zv7FvJGI=', 'Content-Length': '0'} 2023-11-02T18:45:29.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5add6c57929ed22-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.039 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:29.039 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:29.039 DEBUG:boto:path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.040 DEBUG:boto:auth_path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.040 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.041 DEBUG:boto:Path: /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.041 DEBUG:boto:Data: 2023-11-02T18:45:29.041 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.042 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.042 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.043 DEBUG:boto:Params: {} 2023-11-02T18:45:29.043 DEBUG:boto:Token: None 2023-11-02T18:45:29.044 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.044 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bJijgOkYsHP7+wSIO22Zv7FvJGI= 2023-11-02T18:45:29.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bJijgOkYsHP7+wSIO22Zv7FvJGI=', 'Content-Length': '0'} 2023-11-02T18:45:29.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fa1b52ec477cb75-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-11-02T18:45:29.048 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:29.049 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:29.049 DEBUG:boto:path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.050 DEBUG:boto:auth_path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.050 DEBUG:boto:Method: GET 2023-11-02T18:45:29.051 DEBUG:boto:Path: /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.051 DEBUG:boto:Data: 2023-11-02T18:45:29.051 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.052 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.052 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.053 DEBUG:boto:Params: {} 2023-11-02T18:45:29.053 DEBUG:boto:Token: None 2023-11-02T18:45:29.054 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.054 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pAA7pqLoOSgJizBrPPeT3Iv8lx0= 2023-11-02T18:45:29.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pAA7pqLoOSgJizBrPPeT3Iv8lx0=', 'Content-Length': '0'} 2023-11-02T18:45:29.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f889485c17d4bcb-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.098 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:29.099 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:29.100 DEBUG:boto:path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.100 DEBUG:boto:auth_path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.100 DEBUG:boto:Method: GET 2023-11-02T18:45:29.101 DEBUG:boto:Path: /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.101 DEBUG:boto:Data: 2023-11-02T18:45:29.102 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.102 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.103 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.103 DEBUG:boto:Params: {} 2023-11-02T18:45:29.104 DEBUG:boto:Token: None 2023-11-02T18:45:29.104 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:29.105 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pAA7pqLoOSgJizBrPPeT3Iv8lx0= 2023-11-02T18:45:29.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pAA7pqLoOSgJizBrPPeT3Iv8lx0=', 'Content-Length': '0'} 2023-11-02T18:45:29.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048fe85cf580cec4d-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.109 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.109 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.110 DEBUG:boto:path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:29.110 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:29.111 DEBUG:boto:Method: GET 2023-11-02T18:45:29.111 DEBUG:boto:Path: /ibtljn-59/?prefix=obj37 2023-11-02T18:45:29.111 DEBUG:boto:Data: 2023-11-02T18:45:29.112 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.112 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.113 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.113 DEBUG:boto:Params: {} 2023-11-02T18:45:29.114 DEBUG:boto:Token: None 2023-11-02T18:45:29.114 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.115 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004685e6b867c669c4-006543ee49-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.120 DEBUG:boto:b'ibtljn-59obj371000falseobj372023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.121 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.121 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.122 DEBUG:boto:path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:29.122 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:29.123 DEBUG:boto:Method: GET 2023-11-02T18:45:29.123 DEBUG:boto:Path: /ibtljn-59/?prefix=obj37 2023-11-02T18:45:29.124 DEBUG:boto:Data: 2023-11-02T18:45:29.124 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.124 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.125 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.125 DEBUG:boto:Params: {} 2023-11-02T18:45:29.126 DEBUG:boto:Token: None 2023-11-02T18:45:29.126 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.127 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bd91e10872676aa-006543ee49-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.137 DEBUG:boto:b'ibtljn-59obj371000falseobj372023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.138 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:29.138 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:29.139 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.139 DEBUG:boto:Path: /ibtljn-59/obj37 2023-11-02T18:45:29.140 DEBUG:boto:Data: 2023-11-02T18:45:29.140 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.141 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.141 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.142 DEBUG:boto:Params: {} 2023-11-02T18:45:29.142 DEBUG:boto:Token: None 2023-11-02T18:45:29.142 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj37 2023-11-02T18:45:29.143 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l17Og0PmUcJmhn9YYy2RV7S8+DE= 2023-11-02T18:45:29.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l17Og0PmUcJmhn9YYy2RV7S8+DE=', 'Content-Length': '0'} 2023-11-02T18:45:29.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e293c2601b310a4b-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.146 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:29.147 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:29.147 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.148 DEBUG:boto:Path: /ibtljn-59/obj37 2023-11-02T18:45:29.148 DEBUG:boto:Data: 2023-11-02T18:45:29.149 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.149 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.149 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.150 DEBUG:boto:Params: {} 2023-11-02T18:45:29.150 DEBUG:boto:Token: None 2023-11-02T18:45:29.151 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj37 2023-11-02T18:45:29.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l17Og0PmUcJmhn9YYy2RV7S8+DE= 2023-11-02T18:45:29.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l17Og0PmUcJmhn9YYy2RV7S8+DE=', 'Content-Length': '0'} 2023-11-02T18:45:29.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088c0fef9ca45b86f-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-11-02T18:45:29.155 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:29.156 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:29.156 DEBUG:boto:Method: GET 2023-11-02T18:45:29.157 DEBUG:boto:Path: /ibtljn-59/obj37 2023-11-02T18:45:29.157 DEBUG:boto:Data: 2023-11-02T18:45:29.158 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.158 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.158 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.159 DEBUG:boto:Params: {} 2023-11-02T18:45:29.159 DEBUG:boto:Token: None 2023-11-02T18:45:29.160 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj37 2023-11-02T18:45:29.160 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o+0cN6kCw+ChJEEq+pl62tkjqb8= 2023-11-02T18:45:29.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o+0cN6kCw+ChJEEq+pl62tkjqb8=', 'Content-Length': '0'} 2023-11-02T18:45:29.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b6a4fa4d37dd90f-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.164 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:29.165 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:29.165 DEBUG:boto:Method: GET 2023-11-02T18:45:29.165 DEBUG:boto:Path: /ibtljn-59/obj37 2023-11-02T18:45:29.166 DEBUG:boto:Data: 2023-11-02T18:45:29.166 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.167 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.167 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.168 DEBUG:boto:Params: {} 2023-11-02T18:45:29.168 DEBUG:boto:Token: None 2023-11-02T18:45:29.169 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj37 2023-11-02T18:45:29.169 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o+0cN6kCw+ChJEEq+pl62tkjqb8= 2023-11-02T18:45:29.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o+0cN6kCw+ChJEEq+pl62tkjqb8=', 'Content-Length': '0'} 2023-11-02T18:45:29.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062720f1352b0c1b6-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.219 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.219 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.220 DEBUG:boto:path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:29.220 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:29.221 DEBUG:boto:Method: GET 2023-11-02T18:45:29.221 DEBUG:boto:Path: /ibtljn-59/?prefix=obj37 2023-11-02T18:45:29.221 DEBUG:boto:Data: 2023-11-02T18:45:29.222 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.222 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.223 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.223 DEBUG:boto:Params: {} 2023-11-02T18:45:29.224 DEBUG:boto:Token: None 2023-11-02T18:45:29.224 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000390c59574f4a579b-006543ee49-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.231 DEBUG:boto:b'ibtljn-59obj371000falseobj372023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.232 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.232 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.233 DEBUG:boto:path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:29.233 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:29.234 DEBUG:boto:Method: GET 2023-11-02T18:45:29.234 DEBUG:boto:Path: /ibtljn-59/?prefix=obj37 2023-11-02T18:45:29.235 DEBUG:boto:Data: 2023-11-02T18:45:29.235 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.236 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.236 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.236 DEBUG:boto:Params: {} 2023-11-02T18:45:29.237 DEBUG:boto:Token: None 2023-11-02T18:45:29.237 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.238 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8422ec441132e8e-006543ee49-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.248 DEBUG:boto:b'ibtljn-59obj371000falseobj372023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:45:29.250 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:29.250 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:29.251 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.251 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.252 DEBUG:boto:Method: GET 2023-11-02T18:45:29.252 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.253 DEBUG:boto:Data: 2023-11-02T18:45:29.253 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.253 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.254 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.254 DEBUG:boto:Params: {} 2023-11-02T18:45:29.255 DEBUG:boto:Token: None 2023-11-02T18:45:29.255 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.256 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nYaQVBIoqczAF4eDoXKsPqfk3HY= 2023-11-02T18:45:29.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nYaQVBIoqczAF4eDoXKsPqfk3HY=', 'Content-Length': '0'} 2023-11-02T18:45:29.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cb4351643429d91-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.302 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:29.303 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:29.303 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.304 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.304 DEBUG:boto:Method: GET 2023-11-02T18:45:29.305 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.305 DEBUG:boto:Data: 2023-11-02T18:45:29.306 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.306 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.307 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.307 DEBUG:boto:Params: {} 2023-11-02T18:45:29.308 DEBUG:boto:Token: None 2023-11-02T18:45:29.308 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.309 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nYaQVBIoqczAF4eDoXKsPqfk3HY= 2023-11-02T18:45:29.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nYaQVBIoqczAF4eDoXKsPqfk3HY=', 'Content-Length': '0'} 2023-11-02T18:45:29.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024e53b5ef7a56f9a-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.313 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:29.314 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:29.314 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.315 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.315 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.316 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.316 DEBUG:boto:Data: 2023-11-02T18:45:29.316 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.317 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.318 DEBUG:boto:Params: {} 2023-11-02T18:45:29.318 DEBUG:boto:Token: None 2023-11-02T18:45:29.319 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u8MR+IythnbwMchWRWFA+Ycd77Y= 2023-11-02T18:45:29.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u8MR+IythnbwMchWRWFA+Ycd77Y=', 'Content-Length': '0'} 2023-11-02T18:45:29.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbf8333ed7fadb35-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.323 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:29.324 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:29.324 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.325 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.325 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.325 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.326 DEBUG:boto:Data: 2023-11-02T18:45:29.326 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.327 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.327 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.327 DEBUG:boto:Params: {} 2023-11-02T18:45:29.328 DEBUG:boto:Token: None 2023-11-02T18:45:29.328 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.329 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u8MR+IythnbwMchWRWFA+Ycd77Y= 2023-11-02T18:45:29.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u8MR+IythnbwMchWRWFA+Ycd77Y=', 'Content-Length': '0'} 2023-11-02T18:45:29.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d56f49bfa4ebbb45-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:45:29.334 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:29.334 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:29.335 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.335 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.336 DEBUG:boto:Method: GET 2023-11-02T18:45:29.336 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.336 DEBUG:boto:Data: 2023-11-02T18:45:29.337 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.337 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.338 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.338 DEBUG:boto:Params: {} 2023-11-02T18:45:29.339 DEBUG:boto:Token: None 2023-11-02T18:45:29.339 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.340 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nYaQVBIoqczAF4eDoXKsPqfk3HY= 2023-11-02T18:45:29.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nYaQVBIoqczAF4eDoXKsPqfk3HY=', 'Content-Length': '0'} 2023-11-02T18:45:29.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006a4c0caba47a27f-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.344 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:29.344 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:29.345 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.345 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.346 DEBUG:boto:Method: GET 2023-11-02T18:45:29.346 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.347 DEBUG:boto:Data: 2023-11-02T18:45:29.347 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.347 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.348 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.348 DEBUG:boto:Params: {} 2023-11-02T18:45:29.349 DEBUG:boto:Token: None 2023-11-02T18:45:29.349 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:29.350 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nYaQVBIoqczAF4eDoXKsPqfk3HY= 2023-11-02T18:45:29.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nYaQVBIoqczAF4eDoXKsPqfk3HY=', 'Content-Length': '0'} 2023-11-02T18:45:29.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d923b5748a308064-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.354 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.355 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.355 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:29.356 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:29.356 DEBUG:boto:Method: GET 2023-11-02T18:45:29.356 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:45:29.358 DEBUG:boto:Data: 2023-11-02T18:45:29.358 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.359 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.359 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.359 DEBUG:boto:Params: {} 2023-11-02T18:45:29.360 DEBUG:boto:Token: None 2023-11-02T18:45:29.360 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.361 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef58dd3928e7e51e-006543ee49-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.414 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.415 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.416 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.416 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:29.417 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:29.417 DEBUG:boto:Method: GET 2023-11-02T18:45:29.417 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:45:29.418 DEBUG:boto:Data: 2023-11-02T18:45:29.418 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.419 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.419 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.420 DEBUG:boto:Params: {} 2023-11-02T18:45:29.420 DEBUG:boto:Token: None 2023-11-02T18:45:29.421 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.421 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055a64d0ddb81ce7f-006543ee49-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.432 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.432 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:29.433 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:29.433 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.434 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:45:29.434 DEBUG:boto:Data: 2023-11-02T18:45:29.435 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.435 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.436 DEBUG:boto:Params: {} 2023-11-02T18:45:29.436 DEBUG:boto:Token: None 2023-11-02T18:45:29.437 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj38 2023-11-02T18:45:29.437 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dC6y3JYQn5MMYjM+MUjTgNpipj8= 2023-11-02T18:45:29.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dC6y3JYQn5MMYjM+MUjTgNpipj8=', 'Content-Length': '0'} 2023-11-02T18:45:29.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d939494b9b1441a0-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.441 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:29.441 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:29.442 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.442 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:45:29.442 DEBUG:boto:Data: 2023-11-02T18:45:29.443 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.443 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.444 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.444 DEBUG:boto:Params: {} 2023-11-02T18:45:29.445 DEBUG:boto:Token: None 2023-11-02T18:45:29.445 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj38 2023-11-02T18:45:29.446 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dC6y3JYQn5MMYjM+MUjTgNpipj8= 2023-11-02T18:45:29.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dC6y3JYQn5MMYjM+MUjTgNpipj8=', 'Content-Length': '0'} 2023-11-02T18:45:29.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018646cbf70adc886-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:45:29.449 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:29.450 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:29.450 DEBUG:boto:Method: GET 2023-11-02T18:45:29.451 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:45:29.451 DEBUG:boto:Data: 2023-11-02T18:45:29.452 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.452 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.453 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.453 DEBUG:boto:Params: {} 2023-11-02T18:45:29.453 DEBUG:boto:Token: None 2023-11-02T18:45:29.454 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj38 2023-11-02T18:45:29.454 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BUQwpeoOhEcUdFfFL2cttUuyBuA= 2023-11-02T18:45:29.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BUQwpeoOhEcUdFfFL2cttUuyBuA=', 'Content-Length': '0'} 2023-11-02T18:45:29.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f4a26801d2354cd-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.459 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:29.459 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:29.460 DEBUG:boto:Method: GET 2023-11-02T18:45:29.460 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:45:29.461 DEBUG:boto:Data: 2023-11-02T18:45:29.461 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.461 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.462 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.462 DEBUG:boto:Params: {} 2023-11-02T18:45:29.463 DEBUG:boto:Token: None 2023-11-02T18:45:29.464 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj38 2023-11-02T18:45:29.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BUQwpeoOhEcUdFfFL2cttUuyBuA= 2023-11-02T18:45:29.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BUQwpeoOhEcUdFfFL2cttUuyBuA=', 'Content-Length': '0'} 2023-11-02T18:45:29.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050339e6998fd1fdb-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.514 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.515 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.515 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:29.516 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:29.516 DEBUG:boto:Method: GET 2023-11-02T18:45:29.517 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:45:29.517 DEBUG:boto:Data: 2023-11-02T18:45:29.518 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.519 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.519 DEBUG:boto:Params: {} 2023-11-02T18:45:29.520 DEBUG:boto:Token: None 2023-11-02T18:45:29.520 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067a76d97a37be6db-006543ee49-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.527 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.529 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.529 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.530 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:29.530 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:29.531 DEBUG:boto:Method: GET 2023-11-02T18:45:29.532 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:45:29.533 DEBUG:boto:Data: 2023-11-02T18:45:29.533 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.534 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.535 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.536 DEBUG:boto:Params: {} 2023-11-02T18:45:29.537 DEBUG:boto:Token: None 2023-11-02T18:45:29.537 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.538 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000930c16ecb13c083c-006543ee49-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.549 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:45:29.550 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:29.550 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:29.551 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.551 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.552 DEBUG:boto:Method: GET 2023-11-02T18:45:29.552 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.553 DEBUG:boto:Data: 2023-11-02T18:45:29.553 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.554 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.554 DEBUG:boto:Params: {} 2023-11-02T18:45:29.555 DEBUG:boto:Token: None 2023-11-02T18:45:29.556 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YaDhr8JoCy2mtbBsLgYgsHJRil0= 2023-11-02T18:45:29.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YaDhr8JoCy2mtbBsLgYgsHJRil0=', 'Content-Length': '0'} 2023-11-02T18:45:29.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd56bdba9e2d5f0d-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.560 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:29.561 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:29.561 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.561 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.562 DEBUG:boto:Method: GET 2023-11-02T18:45:29.562 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.563 DEBUG:boto:Data: 2023-11-02T18:45:29.563 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.564 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.564 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.565 DEBUG:boto:Params: {} 2023-11-02T18:45:29.565 DEBUG:boto:Token: None 2023-11-02T18:45:29.565 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.566 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YaDhr8JoCy2mtbBsLgYgsHJRil0= 2023-11-02T18:45:29.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YaDhr8JoCy2mtbBsLgYgsHJRil0=', 'Content-Length': '0'} 2023-11-02T18:45:29.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4cdf3b2a73f8fbe-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.570 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:29.570 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:29.571 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.571 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.572 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.572 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.573 DEBUG:boto:Data: 2023-11-02T18:45:29.573 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.573 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.574 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.574 DEBUG:boto:Params: {} 2023-11-02T18:45:29.575 DEBUG:boto:Token: None 2023-11-02T18:45:29.575 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.576 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hg0jPtEdO7kTnTMSJpKtXvI87tQ= 2023-11-02T18:45:29.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hg0jPtEdO7kTnTMSJpKtXvI87tQ=', 'Content-Length': '0'} 2023-11-02T18:45:29.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f84dfe25af9cb9d1-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.580 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:29.580 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:29.581 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.581 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.582 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.582 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.583 DEBUG:boto:Data: 2023-11-02T18:45:29.585 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.586 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.586 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.586 DEBUG:boto:Params: {} 2023-11-02T18:45:29.587 DEBUG:boto:Token: None 2023-11-02T18:45:29.587 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.588 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hg0jPtEdO7kTnTMSJpKtXvI87tQ= 2023-11-02T18:45:29.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hg0jPtEdO7kTnTMSJpKtXvI87tQ=', 'Content-Length': '0'} 2023-11-02T18:45:29.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000076831e5f5a767bb-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:45:29.592 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:29.593 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:29.593 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.594 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.594 DEBUG:boto:Method: GET 2023-11-02T18:45:29.594 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.595 DEBUG:boto:Data: 2023-11-02T18:45:29.595 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.596 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.596 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.597 DEBUG:boto:Params: {} 2023-11-02T18:45:29.597 DEBUG:boto:Token: None 2023-11-02T18:45:29.598 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.598 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YaDhr8JoCy2mtbBsLgYgsHJRil0= 2023-11-02T18:45:29.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YaDhr8JoCy2mtbBsLgYgsHJRil0=', 'Content-Length': '0'} 2023-11-02T18:45:29.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93dc82ee94f59e8-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.642 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:29.643 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:29.643 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.644 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.644 DEBUG:boto:Method: GET 2023-11-02T18:45:29.645 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.645 DEBUG:boto:Data: 2023-11-02T18:45:29.646 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.646 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.647 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.647 DEBUG:boto:Params: {} 2023-11-02T18:45:29.648 DEBUG:boto:Token: None 2023-11-02T18:45:29.648 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:29.649 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YaDhr8JoCy2mtbBsLgYgsHJRil0= 2023-11-02T18:45:29.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YaDhr8JoCy2mtbBsLgYgsHJRil0=', 'Content-Length': '0'} 2023-11-02T18:45:29.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d9bc273a389acae-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.653 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.653 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.654 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:29.654 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:29.654 DEBUG:boto:Method: GET 2023-11-02T18:45:29.655 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:45:29.655 DEBUG:boto:Data: 2023-11-02T18:45:29.656 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.656 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.657 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.657 DEBUG:boto:Params: {} 2023-11-02T18:45:29.658 DEBUG:boto:Token: None 2023-11-02T18:45:29.658 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.659 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9fceab98f3b81ac-006543ee49-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.666 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.666 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.667 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.667 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:29.668 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:29.668 DEBUG:boto:Method: GET 2023-11-02T18:45:29.669 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:45:29.669 DEBUG:boto:Data: 2023-11-02T18:45:29.669 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.670 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.670 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.671 DEBUG:boto:Params: {} 2023-11-02T18:45:29.671 DEBUG:boto:Token: None 2023-11-02T18:45:29.672 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.672 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb9267e623b9ffe-006543ee49-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.722 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.723 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:29.724 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:29.724 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.725 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:45:29.725 DEBUG:boto:Data: 2023-11-02T18:45:29.725 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.726 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.726 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.727 DEBUG:boto:Params: {} 2023-11-02T18:45:29.727 DEBUG:boto:Token: None 2023-11-02T18:45:29.728 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj44 2023-11-02T18:45:29.728 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gG2BI9nfloMb4WzSRpn4HhGksSk= 2023-11-02T18:45:29.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gG2BI9nfloMb4WzSRpn4HhGksSk=', 'Content-Length': '0'} 2023-11-02T18:45:29.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005550c5a57e7e3072-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.733 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:29.733 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:29.734 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.734 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:45:29.735 DEBUG:boto:Data: 2023-11-02T18:45:29.735 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.736 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.736 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.736 DEBUG:boto:Params: {} 2023-11-02T18:45:29.737 DEBUG:boto:Token: None 2023-11-02T18:45:29.737 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj44 2023-11-02T18:45:29.738 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gG2BI9nfloMb4WzSRpn4HhGksSk= 2023-11-02T18:45:29.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gG2BI9nfloMb4WzSRpn4HhGksSk=', 'Content-Length': '0'} 2023-11-02T18:45:29.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e1fdc5fa2fc4b4e-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:45:29.742 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:29.743 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:29.743 DEBUG:boto:Method: GET 2023-11-02T18:45:29.744 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:45:29.744 DEBUG:boto:Data: 2023-11-02T18:45:29.744 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.745 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.745 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.746 DEBUG:boto:Params: {} 2023-11-02T18:45:29.746 DEBUG:boto:Token: None 2023-11-02T18:45:29.747 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj44 2023-11-02T18:45:29.747 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:etRtLDNSJBkWzhm4PIjWIAUF9zU= 2023-11-02T18:45:29.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:etRtLDNSJBkWzhm4PIjWIAUF9zU=', 'Content-Length': '0'} 2023-11-02T18:45:29.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000516ab263222193b9-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.751 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:29.752 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:29.752 DEBUG:boto:Method: GET 2023-11-02T18:45:29.753 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:45:29.753 DEBUG:boto:Data: 2023-11-02T18:45:29.754 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.755 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.755 DEBUG:boto:Params: {} 2023-11-02T18:45:29.756 DEBUG:boto:Token: None 2023-11-02T18:45:29.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj44 2023-11-02T18:45:29.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:etRtLDNSJBkWzhm4PIjWIAUF9zU= 2023-11-02T18:45:29.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:etRtLDNSJBkWzhm4PIjWIAUF9zU=', 'Content-Length': '0'} 2023-11-02T18:45:29.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014f1d21ccc6bea84-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.761 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.761 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.762 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:29.762 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:29.763 DEBUG:boto:Method: GET 2023-11-02T18:45:29.763 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:45:29.763 DEBUG:boto:Data: 2023-11-02T18:45:29.764 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.765 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.765 DEBUG:boto:Params: {} 2023-11-02T18:45:29.766 DEBUG:boto:Token: None 2023-11-02T18:45:29.766 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.767 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000234625e697ac8f9b-006543ee49-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.814 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.815 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.816 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.816 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:29.817 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:29.817 DEBUG:boto:Method: GET 2023-11-02T18:45:29.818 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:45:29.818 DEBUG:boto:Data: 2023-11-02T18:45:29.818 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.819 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.819 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.820 DEBUG:boto:Params: {} 2023-11-02T18:45:29.820 DEBUG:boto:Token: None 2023-11-02T18:45:29.821 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.821 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7396e14d9f260cb-006543ee49-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.832 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:45:29.833 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:29.834 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:29.835 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.835 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.836 DEBUG:boto:Method: GET 2023-11-02T18:45:29.836 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.837 DEBUG:boto:Data: 2023-11-02T18:45:29.837 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.837 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.838 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.838 DEBUG:boto:Params: {} 2023-11-02T18:45:29.839 DEBUG:boto:Token: None 2023-11-02T18:45:29.840 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.840 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ajZy3VDrSpx+K5nF9EYiWuLme4w= 2023-11-02T18:45:29.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ajZy3VDrSpx+K5nF9EYiWuLme4w=', 'Content-Length': '0'} 2023-11-02T18:45:29.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1af7ab3c886a98b-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.844 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:29.844 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:29.845 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.845 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.846 DEBUG:boto:Method: GET 2023-11-02T18:45:29.846 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.847 DEBUG:boto:Data: 2023-11-02T18:45:29.847 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.848 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.848 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.848 DEBUG:boto:Params: {} 2023-11-02T18:45:29.849 DEBUG:boto:Token: None 2023-11-02T18:45:29.849 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.850 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ajZy3VDrSpx+K5nF9EYiWuLme4w= 2023-11-02T18:45:29.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ajZy3VDrSpx+K5nF9EYiWuLme4w=', 'Content-Length': '0'} 2023-11-02T18:45:29.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000956caf69cf305774-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.854 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:29.854 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:29.855 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.855 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.856 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.856 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.856 DEBUG:boto:Data: 2023-11-02T18:45:29.857 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.858 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.858 DEBUG:boto:Params: {} 2023-11-02T18:45:29.859 DEBUG:boto:Token: None 2023-11-02T18:45:29.859 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.860 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NejjcQ6DGsNWtLq2CJea4dMRric= 2023-11-02T18:45:29.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NejjcQ6DGsNWtLq2CJea4dMRric=', 'Content-Length': '0'} 2023-11-02T18:45:29.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af70d6a47476fa32-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.864 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:29.864 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:29.864 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.865 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.865 DEBUG:boto:Method: HEAD 2023-11-02T18:45:29.866 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.866 DEBUG:boto:Data: 2023-11-02T18:45:29.867 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.867 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.867 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.868 DEBUG:boto:Params: {} 2023-11-02T18:45:29.868 DEBUG:boto:Token: None 2023-11-02T18:45:29.869 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.869 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NejjcQ6DGsNWtLq2CJea4dMRric= 2023-11-02T18:45:29.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NejjcQ6DGsNWtLq2CJea4dMRric=', 'Content-Length': '0'} 2023-11-02T18:45:29.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1ddb15e1bdbc6ed-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:45:29.873 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:29.873 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:29.874 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.874 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.875 DEBUG:boto:Method: GET 2023-11-02T18:45:29.875 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.875 DEBUG:boto:Data: 2023-11-02T18:45:29.876 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.876 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.877 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.877 DEBUG:boto:Params: {} 2023-11-02T18:45:29.878 DEBUG:boto:Token: None 2023-11-02T18:45:29.878 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.879 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ajZy3VDrSpx+K5nF9EYiWuLme4w= 2023-11-02T18:45:29.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ajZy3VDrSpx+K5nF9EYiWuLme4w=', 'Content-Length': '0'} 2023-11-02T18:45:29.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050e4378eaa675442-006543ee49-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.922 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:29.923 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:29.923 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.924 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.924 DEBUG:boto:Method: GET 2023-11-02T18:45:29.925 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.925 DEBUG:boto:Data: 2023-11-02T18:45:29.926 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.926 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.926 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.927 DEBUG:boto:Params: {} 2023-11-02T18:45:29.927 DEBUG:boto:Token: None 2023-11-02T18:45:29.928 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:29.928 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ajZy3VDrSpx+K5nF9EYiWuLme4w= 2023-11-02T18:45:29.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ajZy3VDrSpx+K5nF9EYiWuLme4w=', 'Content-Length': '0'} 2023-11-02T18:45:29.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0e2acdd5ffeb747-006543ee49-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.932 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.933 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.933 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:29.934 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:29.934 DEBUG:boto:Method: GET 2023-11-02T18:45:29.935 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:45:29.935 DEBUG:boto:Data: 2023-11-02T18:45:29.936 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.936 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:29.936 DEBUG:boto:Port: 8001 2023-11-02T18:45:29.937 DEBUG:boto:Params: {} 2023-11-02T18:45:29.937 DEBUG:boto:Token: None 2023-11-02T18:45:29.938 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.938 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004d6e8bbdf377f4f-006543ee49-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:29.944 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:29.945 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:29.945 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:29.946 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:29.946 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:29.947 DEBUG:boto:Method: GET 2023-11-02T18:45:29.947 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:45:29.947 DEBUG:boto:Data: 2023-11-02T18:45:29.948 DEBUG:boto:Headers: {} 2023-11-02T18:45:29.948 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:29.949 DEBUG:boto:Port: 8000 2023-11-02T18:45:29.949 DEBUG:boto:Params: {} 2023-11-02T18:45:29.950 DEBUG:boto:Token: None 2023-11-02T18:45:29.950 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:29 GMT /ibtljn-59/ 2023-11-02T18:45:29.951 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8= 2023-11-02T18:45:29.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4Uwj5kkrWvDdqtCB1vEQX+21C8=', 'Content-Length': '0'} 2023-11-02T18:45:29.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0353fa60efa1019-006543ee49-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.002 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.003 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:30.004 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:30.004 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.005 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:45:30.005 DEBUG:boto:Data: 2023-11-02T18:45:30.005 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.006 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.006 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.007 DEBUG:boto:Params: {} 2023-11-02T18:45:30.007 DEBUG:boto:Token: None 2023-11-02T18:45:30.007 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj46 2023-11-02T18:45:30.008 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g7ulu4eVadb9pR5RUCYxA7IBGe4= 2023-11-02T18:45:30.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g7ulu4eVadb9pR5RUCYxA7IBGe4=', 'Content-Length': '0'} 2023-11-02T18:45:30.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001b33c0db58a87cb-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.013 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:30.013 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:30.014 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.014 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:45:30.014 DEBUG:boto:Data: 2023-11-02T18:45:30.015 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.015 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.016 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.016 DEBUG:boto:Params: {} 2023-11-02T18:45:30.017 DEBUG:boto:Token: None 2023-11-02T18:45:30.017 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj46 2023-11-02T18:45:30.018 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g7ulu4eVadb9pR5RUCYxA7IBGe4= 2023-11-02T18:45:30.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g7ulu4eVadb9pR5RUCYxA7IBGe4=', 'Content-Length': '0'} 2023-11-02T18:45:30.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002359cfc7ad49cdb9-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:45:30.022 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:30.023 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:30.023 DEBUG:boto:Method: GET 2023-11-02T18:45:30.024 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:45:30.025 DEBUG:boto:Data: 2023-11-02T18:45:30.025 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.026 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.026 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.027 DEBUG:boto:Params: {} 2023-11-02T18:45:30.027 DEBUG:boto:Token: None 2023-11-02T18:45:30.028 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj46 2023-11-02T18:45:30.028 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:alP+oKo+sQ1heaMQ5l3AxIr+Qsw= 2023-11-02T18:45:30.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:alP+oKo+sQ1heaMQ5l3AxIr+Qsw=', 'Content-Length': '0'} 2023-11-02T18:45:30.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a9d172ef4460d1f-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.033 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:30.033 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:30.034 DEBUG:boto:Method: GET 2023-11-02T18:45:30.034 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:45:30.034 DEBUG:boto:Data: 2023-11-02T18:45:30.035 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.035 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.036 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.036 DEBUG:boto:Params: {} 2023-11-02T18:45:30.037 DEBUG:boto:Token: None 2023-11-02T18:45:30.037 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj46 2023-11-02T18:45:30.038 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:alP+oKo+sQ1heaMQ5l3AxIr+Qsw= 2023-11-02T18:45:30.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:alP+oKo+sQ1heaMQ5l3AxIr+Qsw=', 'Content-Length': '0'} 2023-11-02T18:45:30.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa6a9d8cd1732c1a-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.042 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.043 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.043 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:30.044 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:30.044 DEBUG:boto:Method: GET 2023-11-02T18:45:30.045 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:45:30.045 DEBUG:boto:Data: 2023-11-02T18:45:30.045 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.046 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.047 DEBUG:boto:Params: {} 2023-11-02T18:45:30.047 DEBUG:boto:Token: None 2023-11-02T18:45:30.048 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.048 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037810e0b157e6c68-006543ee4a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.094 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.095 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.096 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.096 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:30.097 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:30.097 DEBUG:boto:Method: GET 2023-11-02T18:45:30.097 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:45:30.098 DEBUG:boto:Data: 2023-11-02T18:45:30.098 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.099 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.099 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.100 DEBUG:boto:Params: {} 2023-11-02T18:45:30.100 DEBUG:boto:Token: None 2023-11-02T18:45:30.101 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.101 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b91ac54234b7c44-006543ee4a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.112 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:45:30.113 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:30.114 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:30.114 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.115 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.115 DEBUG:boto:Method: GET 2023-11-02T18:45:30.116 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.116 DEBUG:boto:Data: 2023-11-02T18:45:30.117 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.117 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.117 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.118 DEBUG:boto:Params: {} 2023-11-02T18:45:30.118 DEBUG:boto:Token: None 2023-11-02T18:45:30.119 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.119 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Nbm/HMKrrl1H77D5QQtVU+M6UfA= 2023-11-02T18:45:30.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Nbm/HMKrrl1H77D5QQtVU+M6UfA=', 'Content-Length': '0'} 2023-11-02T18:45:30.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a785cce8e48b2f60-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.123 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:30.123 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:30.124 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.124 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.125 DEBUG:boto:Method: GET 2023-11-02T18:45:30.125 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.126 DEBUG:boto:Data: 2023-11-02T18:45:30.126 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.126 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.127 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.127 DEBUG:boto:Params: {} 2023-11-02T18:45:30.128 DEBUG:boto:Token: None 2023-11-02T18:45:30.128 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Nbm/HMKrrl1H77D5QQtVU+M6UfA= 2023-11-02T18:45:30.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Nbm/HMKrrl1H77D5QQtVU+M6UfA=', 'Content-Length': '0'} 2023-11-02T18:45:30.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be08bcbbe71d9275-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.133 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:30.133 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:30.133 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.134 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.134 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.135 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.135 DEBUG:boto:Data: 2023-11-02T18:45:30.136 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.136 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.137 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.137 DEBUG:boto:Params: {} 2023-11-02T18:45:30.137 DEBUG:boto:Token: None 2023-11-02T18:45:30.138 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.138 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Gs7TB2shqkrtn6HV9RmDUAzdSJg= 2023-11-02T18:45:30.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Gs7TB2shqkrtn6HV9RmDUAzdSJg=', 'Content-Length': '0'} 2023-11-02T18:45:30.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eadeecd5f7b979d-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.142 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:30.143 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:30.143 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.144 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.144 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.144 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.145 DEBUG:boto:Data: 2023-11-02T18:45:30.145 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.146 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.146 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.146 DEBUG:boto:Params: {} 2023-11-02T18:45:30.147 DEBUG:boto:Token: None 2023-11-02T18:45:30.147 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.148 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Gs7TB2shqkrtn6HV9RmDUAzdSJg= 2023-11-02T18:45:30.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Gs7TB2shqkrtn6HV9RmDUAzdSJg=', 'Content-Length': '0'} 2023-11-02T18:45:30.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a1fb85e2883b134-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:45:30.152 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:30.152 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:30.153 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.153 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.154 DEBUG:boto:Method: GET 2023-11-02T18:45:30.154 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.155 DEBUG:boto:Data: 2023-11-02T18:45:30.155 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.155 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.156 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.156 DEBUG:boto:Params: {} 2023-11-02T18:45:30.157 DEBUG:boto:Token: None 2023-11-02T18:45:30.157 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.158 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Nbm/HMKrrl1H77D5QQtVU+M6UfA= 2023-11-02T18:45:30.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Nbm/HMKrrl1H77D5QQtVU+M6UfA=', 'Content-Length': '0'} 2023-11-02T18:45:30.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094113df41fb227f3-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.162 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:30.163 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:30.163 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.163 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.164 DEBUG:boto:Method: GET 2023-11-02T18:45:30.165 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.165 DEBUG:boto:Data: 2023-11-02T18:45:30.165 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.166 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.166 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.167 DEBUG:boto:Params: {} 2023-11-02T18:45:30.167 DEBUG:boto:Token: None 2023-11-02T18:45:30.168 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:30.168 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Nbm/HMKrrl1H77D5QQtVU+M6UfA= 2023-11-02T18:45:30.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Nbm/HMKrrl1H77D5QQtVU+M6UfA=', 'Content-Length': '0'} 2023-11-02T18:45:30.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f40783e062b1f46-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.218 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.219 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.220 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:30.220 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:30.220 DEBUG:boto:Method: GET 2023-11-02T18:45:30.221 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:45:30.221 DEBUG:boto:Data: 2023-11-02T18:45:30.222 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.222 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.223 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.223 DEBUG:boto:Params: {} 2023-11-02T18:45:30.224 DEBUG:boto:Token: None 2023-11-02T18:45:30.224 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6532929021d4d2e-006543ee4a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.231 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.232 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.232 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.232 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:30.233 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:30.233 DEBUG:boto:Method: GET 2023-11-02T18:45:30.234 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:45:30.234 DEBUG:boto:Data: 2023-11-02T18:45:30.235 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.235 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.236 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.236 DEBUG:boto:Params: {} 2023-11-02T18:45:30.236 DEBUG:boto:Token: None 2023-11-02T18:45:30.237 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.237 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079423abdf135f98b-006543ee4a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.248 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.249 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:30.249 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:30.249 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.250 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:45:30.250 DEBUG:boto:Data: 2023-11-02T18:45:30.251 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.251 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.252 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.252 DEBUG:boto:Params: {} 2023-11-02T18:45:30.253 DEBUG:boto:Token: None 2023-11-02T18:45:30.253 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj51 2023-11-02T18:45:30.253 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:snHK06Z0tAH1w27ES+QJKrR+KnM= 2023-11-02T18:45:30.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:snHK06Z0tAH1w27ES+QJKrR+KnM=', 'Content-Length': '0'} 2023-11-02T18:45:30.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000413f5de849602695-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.257 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:30.257 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:30.258 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.258 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:45:30.258 DEBUG:boto:Data: 2023-11-02T18:45:30.259 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.260 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.260 DEBUG:boto:Params: {} 2023-11-02T18:45:30.261 DEBUG:boto:Token: None 2023-11-02T18:45:30.261 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj51 2023-11-02T18:45:30.261 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:snHK06Z0tAH1w27ES+QJKrR+KnM= 2023-11-02T18:45:30.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:snHK06Z0tAH1w27ES+QJKrR+KnM=', 'Content-Length': '0'} 2023-11-02T18:45:30.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b6b489ef32d0acb-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:45:30.265 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:30.266 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:30.266 DEBUG:boto:Method: GET 2023-11-02T18:45:30.267 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:45:30.267 DEBUG:boto:Data: 2023-11-02T18:45:30.267 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.268 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.268 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.269 DEBUG:boto:Params: {} 2023-11-02T18:45:30.269 DEBUG:boto:Token: None 2023-11-02T18:45:30.270 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj51 2023-11-02T18:45:30.270 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VskOe+hwifMkMG94wWxRkHRAwVA= 2023-11-02T18:45:30.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VskOe+hwifMkMG94wWxRkHRAwVA=', 'Content-Length': '0'} 2023-11-02T18:45:30.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046cd8d7826d09be8-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.314 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:30.315 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:30.316 DEBUG:boto:Method: GET 2023-11-02T18:45:30.316 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:45:30.316 DEBUG:boto:Data: 2023-11-02T18:45:30.317 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.318 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.318 DEBUG:boto:Params: {} 2023-11-02T18:45:30.319 DEBUG:boto:Token: None 2023-11-02T18:45:30.319 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj51 2023-11-02T18:45:30.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VskOe+hwifMkMG94wWxRkHRAwVA= 2023-11-02T18:45:30.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VskOe+hwifMkMG94wWxRkHRAwVA=', 'Content-Length': '0'} 2023-11-02T18:45:30.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9a656e628909248-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.324 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.325 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.325 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:30.325 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:30.326 DEBUG:boto:Method: GET 2023-11-02T18:45:30.326 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:45:30.327 DEBUG:boto:Data: 2023-11-02T18:45:30.327 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.328 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.329 DEBUG:boto:Params: {} 2023-11-02T18:45:30.329 DEBUG:boto:Token: None 2023-11-02T18:45:30.330 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.330 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f391a1ba6ddefb62-006543ee4a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.335 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.335 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.336 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.336 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:30.337 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:30.337 DEBUG:boto:Method: GET 2023-11-02T18:45:30.338 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:45:30.338 DEBUG:boto:Data: 2023-11-02T18:45:30.339 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.339 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.339 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.340 DEBUG:boto:Params: {} 2023-11-02T18:45:30.340 DEBUG:boto:Token: None 2023-11-02T18:45:30.341 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.341 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbc3c073f214d1e9-006543ee4a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.394 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:45:30.397 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:30.397 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:30.398 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.398 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.399 DEBUG:boto:Method: GET 2023-11-02T18:45:30.399 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.400 DEBUG:boto:Data: 2023-11-02T18:45:30.400 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.401 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.401 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.402 DEBUG:boto:Params: {} 2023-11-02T18:45:30.402 DEBUG:boto:Token: None 2023-11-02T18:45:30.403 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.403 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V/zM8nDHOGxWqUxvegRQ/ksMhkA= 2023-11-02T18:45:30.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V/zM8nDHOGxWqUxvegRQ/ksMhkA=', 'Content-Length': '0'} 2023-11-02T18:45:30.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032e76e46245469be-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.410 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:30.411 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:30.411 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.411 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.412 DEBUG:boto:Method: GET 2023-11-02T18:45:30.412 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.413 DEBUG:boto:Data: 2023-11-02T18:45:30.413 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.414 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.414 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.415 DEBUG:boto:Params: {} 2023-11-02T18:45:30.415 DEBUG:boto:Token: None 2023-11-02T18:45:30.416 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.416 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V/zM8nDHOGxWqUxvegRQ/ksMhkA= 2023-11-02T18:45:30.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V/zM8nDHOGxWqUxvegRQ/ksMhkA=', 'Content-Length': '0'} 2023-11-02T18:45:30.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b0074c9c9e6e6cd-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.420 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:30.421 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:30.421 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.422 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.422 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.423 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.423 DEBUG:boto:Data: 2023-11-02T18:45:30.423 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.424 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.425 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.425 DEBUG:boto:Params: {} 2023-11-02T18:45:30.426 DEBUG:boto:Token: None 2023-11-02T18:45:30.426 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.427 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u3K2TKSz85Hd0ESfENkT3k/FJLM= 2023-11-02T18:45:30.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u3K2TKSz85Hd0ESfENkT3k/FJLM=', 'Content-Length': '0'} 2023-11-02T18:45:30.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c8dfd4149a5d380-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.431 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:30.431 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:30.432 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.432 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.432 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.433 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.433 DEBUG:boto:Data: 2023-11-02T18:45:30.434 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.434 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.435 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.435 DEBUG:boto:Params: {} 2023-11-02T18:45:30.435 DEBUG:boto:Token: None 2023-11-02T18:45:30.436 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.436 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u3K2TKSz85Hd0ESfENkT3k/FJLM= 2023-11-02T18:45:30.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u3K2TKSz85Hd0ESfENkT3k/FJLM=', 'Content-Length': '0'} 2023-11-02T18:45:30.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ae599e924220bc5-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:45:30.441 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:30.441 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:30.442 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.442 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.443 DEBUG:boto:Method: GET 2023-11-02T18:45:30.443 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.444 DEBUG:boto:Data: 2023-11-02T18:45:30.444 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.445 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.445 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.445 DEBUG:boto:Params: {} 2023-11-02T18:45:30.446 DEBUG:boto:Token: None 2023-11-02T18:45:30.446 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.447 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V/zM8nDHOGxWqUxvegRQ/ksMhkA= 2023-11-02T18:45:30.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V/zM8nDHOGxWqUxvegRQ/ksMhkA=', 'Content-Length': '0'} 2023-11-02T18:45:30.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bf3c00c6fbda016-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.451 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:30.452 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:30.452 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.452 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.453 DEBUG:boto:Method: GET 2023-11-02T18:45:30.453 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.454 DEBUG:boto:Data: 2023-11-02T18:45:30.454 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.455 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.455 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.456 DEBUG:boto:Params: {} 2023-11-02T18:45:30.456 DEBUG:boto:Token: None 2023-11-02T18:45:30.457 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:30.457 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V/zM8nDHOGxWqUxvegRQ/ksMhkA= 2023-11-02T18:45:30.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V/zM8nDHOGxWqUxvegRQ/ksMhkA=', 'Content-Length': '0'} 2023-11-02T18:45:30.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074c57210ca736bef-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.502 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.503 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.503 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:30.504 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:30.504 DEBUG:boto:Method: GET 2023-11-02T18:45:30.505 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:45:30.505 DEBUG:boto:Data: 2023-11-02T18:45:30.506 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.506 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.507 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.507 DEBUG:boto:Params: {} 2023-11-02T18:45:30.508 DEBUG:boto:Token: None 2023-11-02T18:45:30.508 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.509 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003509348c0a8d8e2a-006543ee4a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.514 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.515 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.516 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.516 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:30.516 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:30.517 DEBUG:boto:Method: GET 2023-11-02T18:45:30.517 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:45:30.518 DEBUG:boto:Data: 2023-11-02T18:45:30.518 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.519 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.519 DEBUG:boto:Params: {} 2023-11-02T18:45:30.520 DEBUG:boto:Token: None 2023-11-02T18:45:30.520 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b57809dde1c23b80-006543ee4a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.531 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.532 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:30.532 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:30.533 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.533 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:45:30.534 DEBUG:boto:Data: 2023-11-02T18:45:30.534 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.535 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.535 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.536 DEBUG:boto:Params: {} 2023-11-02T18:45:30.536 DEBUG:boto:Token: None 2023-11-02T18:45:30.537 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj55 2023-11-02T18:45:30.537 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7+iFvPv/a9h4vcNOJLwLzf3mvbw= 2023-11-02T18:45:30.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7+iFvPv/a9h4vcNOJLwLzf3mvbw=', 'Content-Length': '0'} 2023-11-02T18:45:30.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a4239ba7ab08296-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.541 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:30.542 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:30.542 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.542 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:45:30.543 DEBUG:boto:Data: 2023-11-02T18:45:30.543 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.544 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.544 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.545 DEBUG:boto:Params: {} 2023-11-02T18:45:30.545 DEBUG:boto:Token: None 2023-11-02T18:45:30.546 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj55 2023-11-02T18:45:30.546 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7+iFvPv/a9h4vcNOJLwLzf3mvbw= 2023-11-02T18:45:30.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7+iFvPv/a9h4vcNOJLwLzf3mvbw=', 'Content-Length': '0'} 2023-11-02T18:45:30.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000278bff0a11fa82c0-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:45:30.550 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:30.551 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:30.551 DEBUG:boto:Method: GET 2023-11-02T18:45:30.552 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:45:30.552 DEBUG:boto:Data: 2023-11-02T18:45:30.553 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.553 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.554 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.554 DEBUG:boto:Params: {} 2023-11-02T18:45:30.555 DEBUG:boto:Token: None 2023-11-02T18:45:30.555 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj55 2023-11-02T18:45:30.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QmcrgJ038Ns5mJkTJIomTv1XksQ= 2023-11-02T18:45:30.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QmcrgJ038Ns5mJkTJIomTv1XksQ=', 'Content-Length': '0'} 2023-11-02T18:45:30.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be9dd0b25b6c7bcf-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.606 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:30.607 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:30.608 DEBUG:boto:Method: GET 2023-11-02T18:45:30.608 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:45:30.608 DEBUG:boto:Data: 2023-11-02T18:45:30.609 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.609 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.610 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.610 DEBUG:boto:Params: {} 2023-11-02T18:45:30.611 DEBUG:boto:Token: None 2023-11-02T18:45:30.611 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj55 2023-11-02T18:45:30.612 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QmcrgJ038Ns5mJkTJIomTv1XksQ= 2023-11-02T18:45:30.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QmcrgJ038Ns5mJkTJIomTv1XksQ=', 'Content-Length': '0'} 2023-11-02T18:45:30.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000089a4414ed6eb78b-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.617 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.618 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.618 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:30.619 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:30.620 DEBUG:boto:Method: GET 2023-11-02T18:45:30.620 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:45:30.621 DEBUG:boto:Data: 2023-11-02T18:45:30.621 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.622 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.622 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.623 DEBUG:boto:Params: {} 2023-11-02T18:45:30.623 DEBUG:boto:Token: None 2023-11-02T18:45:30.624 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040369bfb019bd004-006543ee4a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.630 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.631 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.632 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.632 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:30.633 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:30.633 DEBUG:boto:Method: GET 2023-11-02T18:45:30.634 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:45:30.634 DEBUG:boto:Data: 2023-11-02T18:45:30.635 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.635 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.636 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.636 DEBUG:boto:Params: {} 2023-11-02T18:45:30.637 DEBUG:boto:Token: None 2023-11-02T18:45:30.637 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.638 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf64dcb6701c01c3-006543ee4a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.648 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:45:30.650 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:30.650 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:30.651 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.651 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.652 DEBUG:boto:Method: GET 2023-11-02T18:45:30.652 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.653 DEBUG:boto:Data: 2023-11-02T18:45:30.653 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.654 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.654 DEBUG:boto:Params: {} 2023-11-02T18:45:30.655 DEBUG:boto:Token: None 2023-11-02T18:45:30.656 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sE6YH1c0zgrPyxddhinrco3RGAk= 2023-11-02T18:45:30.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sE6YH1c0zgrPyxddhinrco3RGAk=', 'Content-Length': '0'} 2023-11-02T18:45:30.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000044d9ce4f5b8cf42-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.660 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:30.660 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:30.661 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.661 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.662 DEBUG:boto:Method: GET 2023-11-02T18:45:30.662 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.662 DEBUG:boto:Data: 2023-11-02T18:45:30.663 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.664 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.665 DEBUG:boto:Params: {} 2023-11-02T18:45:30.665 DEBUG:boto:Token: None 2023-11-02T18:45:30.666 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.667 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sE6YH1c0zgrPyxddhinrco3RGAk= 2023-11-02T18:45:30.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sE6YH1c0zgrPyxddhinrco3RGAk=', 'Content-Length': '0'} 2023-11-02T18:45:30.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089c00a7816e0c786-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.718 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:30.719 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:30.720 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.720 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.720 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.721 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.721 DEBUG:boto:Data: 2023-11-02T18:45:30.722 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.722 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.722 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.723 DEBUG:boto:Params: {} 2023-11-02T18:45:30.723 DEBUG:boto:Token: None 2023-11-02T18:45:30.724 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.724 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jb+vJRXVV8OmDyAPSb16xwDEgBA= 2023-11-02T18:45:30.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jb+vJRXVV8OmDyAPSb16xwDEgBA=', 'Content-Length': '0'} 2023-11-02T18:45:30.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093fe32dcdb842e3a-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.728 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:30.729 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:30.729 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.730 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.730 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.731 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.731 DEBUG:boto:Data: 2023-11-02T18:45:30.731 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.732 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.732 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.733 DEBUG:boto:Params: {} 2023-11-02T18:45:30.733 DEBUG:boto:Token: None 2023-11-02T18:45:30.734 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.734 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Jb+vJRXVV8OmDyAPSb16xwDEgBA= 2023-11-02T18:45:30.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jb+vJRXVV8OmDyAPSb16xwDEgBA=', 'Content-Length': '0'} 2023-11-02T18:45:30.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087be760f337a241d-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:45:30.739 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:30.740 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:30.740 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.740 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.741 DEBUG:boto:Method: GET 2023-11-02T18:45:30.741 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.742 DEBUG:boto:Data: 2023-11-02T18:45:30.742 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.743 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.743 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.743 DEBUG:boto:Params: {} 2023-11-02T18:45:30.744 DEBUG:boto:Token: None 2023-11-02T18:45:30.744 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.745 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sE6YH1c0zgrPyxddhinrco3RGAk= 2023-11-02T18:45:30.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sE6YH1c0zgrPyxddhinrco3RGAk=', 'Content-Length': '0'} 2023-11-02T18:45:30.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8b294c4c5c29a96-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.790 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:30.791 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:30.791 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.792 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.792 DEBUG:boto:Method: GET 2023-11-02T18:45:30.793 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.794 DEBUG:boto:Data: 2023-11-02T18:45:30.795 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.795 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.796 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.796 DEBUG:boto:Params: {} 2023-11-02T18:45:30.797 DEBUG:boto:Token: None 2023-11-02T18:45:30.797 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:30.798 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sE6YH1c0zgrPyxddhinrco3RGAk= 2023-11-02T18:45:30.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sE6YH1c0zgrPyxddhinrco3RGAk=', 'Content-Length': '0'} 2023-11-02T18:45:30.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002258f349f2359645-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.802 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.804 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.804 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:30.805 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:30.805 DEBUG:boto:Method: GET 2023-11-02T18:45:30.806 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:45:30.806 DEBUG:boto:Data: 2023-11-02T18:45:30.806 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.807 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.808 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.808 DEBUG:boto:Params: {} 2023-11-02T18:45:30.809 DEBUG:boto:Token: None 2023-11-02T18:45:30.809 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.810 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ba771f2edb81052-006543ee4a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.816 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.817 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.817 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.819 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:30.819 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:30.819 DEBUG:boto:Method: GET 2023-11-02T18:45:30.820 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:45:30.821 DEBUG:boto:Data: 2023-11-02T18:45:30.822 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.822 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.823 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.823 DEBUG:boto:Params: {} 2023-11-02T18:45:30.824 DEBUG:boto:Token: None 2023-11-02T18:45:30.824 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.825 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2ee3663feb39938-006543ee4a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.836 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.837 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:30.838 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:30.839 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.839 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:45:30.840 DEBUG:boto:Data: 2023-11-02T18:45:30.840 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.841 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.841 DEBUG:boto:Params: {} 2023-11-02T18:45:30.842 DEBUG:boto:Token: None 2023-11-02T18:45:30.843 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj57 2023-11-02T18:45:30.843 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dnDsWTLp8Q4payOPS3Px5OCVBcE= 2023-11-02T18:45:30.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dnDsWTLp8Q4payOPS3Px5OCVBcE=', 'Content-Length': '0'} 2023-11-02T18:45:30.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d875b4554d514fbc-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.847 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:30.847 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:30.848 DEBUG:boto:Method: HEAD 2023-11-02T18:45:30.848 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:45:30.849 DEBUG:boto:Data: 2023-11-02T18:45:30.849 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.850 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.851 DEBUG:boto:Params: {} 2023-11-02T18:45:30.851 DEBUG:boto:Token: None 2023-11-02T18:45:30.852 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj57 2023-11-02T18:45:30.852 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dnDsWTLp8Q4payOPS3Px5OCVBcE= 2023-11-02T18:45:30.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dnDsWTLp8Q4payOPS3Px5OCVBcE=', 'Content-Length': '0'} 2023-11-02T18:45:30.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034a0e00e8928e8f8-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:45:30.856 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:30.857 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:30.857 DEBUG:boto:Method: GET 2023-11-02T18:45:30.858 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:45:30.858 DEBUG:boto:Data: 2023-11-02T18:45:30.859 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.859 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.860 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.860 DEBUG:boto:Params: {} 2023-11-02T18:45:30.861 DEBUG:boto:Token: None 2023-11-02T18:45:30.861 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj57 2023-11-02T18:45:30.862 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3WOrMJcYnhqdKEY1UnhWUzX+HpI= 2023-11-02T18:45:30.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3WOrMJcYnhqdKEY1UnhWUzX+HpI=', 'Content-Length': '0'} 2023-11-02T18:45:30.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007887501e151e468b-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.866 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:30.866 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:30.867 DEBUG:boto:Method: GET 2023-11-02T18:45:30.867 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:45:30.868 DEBUG:boto:Data: 2023-11-02T18:45:30.868 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.868 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.869 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.869 DEBUG:boto:Params: {} 2023-11-02T18:45:30.870 DEBUG:boto:Token: None 2023-11-02T18:45:30.870 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj57 2023-11-02T18:45:30.871 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3WOrMJcYnhqdKEY1UnhWUzX+HpI= 2023-11-02T18:45:30.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3WOrMJcYnhqdKEY1UnhWUzX+HpI=', 'Content-Length': '0'} 2023-11-02T18:45:30.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d81235afc6d1f66-006543ee4a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.922 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.923 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.924 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:30.924 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:30.925 DEBUG:boto:Method: GET 2023-11-02T18:45:30.925 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:45:30.926 DEBUG:boto:Data: 2023-11-02T18:45:30.926 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.926 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.927 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.927 DEBUG:boto:Params: {} 2023-11-02T18:45:30.928 DEBUG:boto:Token: None 2023-11-02T18:45:30.929 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.929 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cc96cb064da28e2-006543ee4a-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.935 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.936 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:30.936 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:30.936 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:30.937 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:30.937 DEBUG:boto:Method: GET 2023-11-02T18:45:30.938 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:45:30.938 DEBUG:boto:Data: 2023-11-02T18:45:30.939 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.939 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:30.939 DEBUG:boto:Port: 8000 2023-11-02T18:45:30.940 DEBUG:boto:Params: {} 2023-11-02T18:45:30.940 DEBUG:boto:Token: None 2023-11-02T18:45:30.941 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/ 2023-11-02T18:45:30.941 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik= 2023-11-02T18:45:30.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FX+ebMvzKluknQLm/p8uq+jNsik=', 'Content-Length': '0'} 2023-11-02T18:45:30.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033e2220db6cbba90-006543ee4a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:30.953 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:30.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:45:30.954 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:30.954 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:30.955 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:30.955 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:30.956 DEBUG:boto:Method: GET 2023-11-02T18:45:30.956 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:30.957 DEBUG:boto:Data: 2023-11-02T18:45:30.957 DEBUG:boto:Headers: {} 2023-11-02T18:45:30.958 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:30.958 DEBUG:boto:Port: 8001 2023-11-02T18:45:30.958 DEBUG:boto:Params: {} 2023-11-02T18:45:30.959 DEBUG:boto:Token: None 2023-11-02T18:45:30.959 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:30 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:30.960 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ErgCUj8zex85k0Ozjr5Pjc1R4/8= 2023-11-02T18:45:30.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:30 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ErgCUj8zex85k0Ozjr5Pjc1R4/8=', 'Content-Length': '0'} 2023-11-02T18:45:30.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a21fe7073077a79-006543ee4a-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:30 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.006 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:31.007 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:31.007 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.008 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.008 DEBUG:boto:Method: GET 2023-11-02T18:45:31.009 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.009 DEBUG:boto:Data: 2023-11-02T18:45:31.010 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.010 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.010 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.011 DEBUG:boto:Params: {} 2023-11-02T18:45:31.012 DEBUG:boto:Token: None 2023-11-02T18:45:31.012 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.013 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ce3SifGOuIJAeXevDVeJE3utu1U= 2023-11-02T18:45:31.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ce3SifGOuIJAeXevDVeJE3utu1U=', 'Content-Length': '0'} 2023-11-02T18:45:31.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fda2ca301e8919d2-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.017 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:31.017 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:31.018 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.018 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.018 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.022 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.023 DEBUG:boto:Data: 2023-11-02T18:45:31.023 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.023 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.024 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.024 DEBUG:boto:Params: {} 2023-11-02T18:45:31.025 DEBUG:boto:Token: None 2023-11-02T18:45:31.025 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.026 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aBtMeWdgIzpDmBgMH9g40LGxtAk= 2023-11-02T18:45:31.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aBtMeWdgIzpDmBgMH9g40LGxtAk=', 'Content-Length': '0'} 2023-11-02T18:45:31.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a06fee73a76922d-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.030 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:31.031 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:31.031 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.032 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.032 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.032 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.033 DEBUG:boto:Data: 2023-11-02T18:45:31.033 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.034 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.034 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.035 DEBUG:boto:Params: {} 2023-11-02T18:45:31.035 DEBUG:boto:Token: None 2023-11-02T18:45:31.036 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.036 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aBtMeWdgIzpDmBgMH9g40LGxtAk= 2023-11-02T18:45:31.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aBtMeWdgIzpDmBgMH9g40LGxtAk=', 'Content-Length': '0'} 2023-11-02T18:45:31.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ac8b450f25ef6b-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:45:31.041 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:31.041 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:31.042 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.042 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.042 DEBUG:boto:Method: GET 2023-11-02T18:45:31.043 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.043 DEBUG:boto:Data: 2023-11-02T18:45:31.044 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.045 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.045 DEBUG:boto:Params: {} 2023-11-02T18:45:31.046 DEBUG:boto:Token: None 2023-11-02T18:45:31.046 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.047 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ce3SifGOuIJAeXevDVeJE3utu1U= 2023-11-02T18:45:31.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ce3SifGOuIJAeXevDVeJE3utu1U=', 'Content-Length': '0'} 2023-11-02T18:45:31.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dd25de384db50f8-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.051 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:31.052 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:31.052 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.053 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.053 DEBUG:boto:Method: GET 2023-11-02T18:45:31.053 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.054 DEBUG:boto:Data: 2023-11-02T18:45:31.054 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.055 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.055 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.056 DEBUG:boto:Params: {} 2023-11-02T18:45:31.056 DEBUG:boto:Token: None 2023-11-02T18:45:31.057 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:31.057 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ce3SifGOuIJAeXevDVeJE3utu1U= 2023-11-02T18:45:31.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ce3SifGOuIJAeXevDVeJE3utu1U=', 'Content-Length': '0'} 2023-11-02T18:45:31.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003431e7d4a010c20-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.102 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.103 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.104 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:31.104 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:31.105 DEBUG:boto:Method: GET 2023-11-02T18:45:31.105 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:45:31.105 DEBUG:boto:Data: 2023-11-02T18:45:31.106 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.107 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.107 DEBUG:boto:Params: {} 2023-11-02T18:45:31.108 DEBUG:boto:Token: None 2023-11-02T18:45:31.108 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.109 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003064acb98180613-006543ee4b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.115 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.116 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.116 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.117 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:31.117 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:31.118 DEBUG:boto:Method: GET 2023-11-02T18:45:31.118 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:45:31.119 DEBUG:boto:Data: 2023-11-02T18:45:31.119 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.120 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.120 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.121 DEBUG:boto:Params: {} 2023-11-02T18:45:31.121 DEBUG:boto:Token: None 2023-11-02T18:45:31.122 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.123 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df709da6373d3b4f-006543ee4b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.133 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.134 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:31.135 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:31.136 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.137 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:45:31.137 DEBUG:boto:Data: 2023-11-02T18:45:31.138 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.139 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.139 DEBUG:boto:Params: {} 2023-11-02T18:45:31.140 DEBUG:boto:Token: None 2023-11-02T18:45:31.140 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj58 2023-11-02T18:45:31.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C3zYx95PN0wUTrDQS1saQ9GgbUg= 2023-11-02T18:45:31.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C3zYx95PN0wUTrDQS1saQ9GgbUg=', 'Content-Length': '0'} 2023-11-02T18:45:31.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005034d783b98fef60-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.144 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:31.145 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:31.145 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.146 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:45:31.146 DEBUG:boto:Data: 2023-11-02T18:45:31.147 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.147 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.148 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.148 DEBUG:boto:Params: {} 2023-11-02T18:45:31.148 DEBUG:boto:Token: None 2023-11-02T18:45:31.149 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj58 2023-11-02T18:45:31.149 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C3zYx95PN0wUTrDQS1saQ9GgbUg= 2023-11-02T18:45:31.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C3zYx95PN0wUTrDQS1saQ9GgbUg=', 'Content-Length': '0'} 2023-11-02T18:45:31.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ca4856037d0b77d-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:45:31.154 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:31.154 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:31.155 DEBUG:boto:Method: GET 2023-11-02T18:45:31.155 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:45:31.156 DEBUG:boto:Data: 2023-11-02T18:45:31.157 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.157 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.158 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.158 DEBUG:boto:Params: {} 2023-11-02T18:45:31.159 DEBUG:boto:Token: None 2023-11-02T18:45:31.159 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj58 2023-11-02T18:45:31.160 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+7FU5dCMcRgOSxlYvdCISSJ9R3U= 2023-11-02T18:45:31.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+7FU5dCMcRgOSxlYvdCISSJ9R3U=', 'Content-Length': '0'} 2023-11-02T18:45:31.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055682116f909b3d9-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.164 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:31.164 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:31.165 DEBUG:boto:Method: GET 2023-11-02T18:45:31.165 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:45:31.166 DEBUG:boto:Data: 2023-11-02T18:45:31.166 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.167 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.167 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.168 DEBUG:boto:Params: {} 2023-11-02T18:45:31.169 DEBUG:boto:Token: None 2023-11-02T18:45:31.169 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj58 2023-11-02T18:45:31.170 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+7FU5dCMcRgOSxlYvdCISSJ9R3U= 2023-11-02T18:45:31.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+7FU5dCMcRgOSxlYvdCISSJ9R3U=', 'Content-Length': '0'} 2023-11-02T18:45:31.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000764d2e7f311c70f2-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.175 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.176 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.176 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:31.177 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:31.178 DEBUG:boto:Method: GET 2023-11-02T18:45:31.178 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:45:31.179 DEBUG:boto:Data: 2023-11-02T18:45:31.179 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.180 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.181 DEBUG:boto:Params: {} 2023-11-02T18:45:31.182 DEBUG:boto:Token: None 2023-11-02T18:45:31.182 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.183 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000219c1f1c1b0cf7dd-006543ee4b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.230 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.232 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.233 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.233 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:31.234 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:31.235 DEBUG:boto:Method: GET 2023-11-02T18:45:31.236 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:45:31.237 DEBUG:boto:Data: 2023-11-02T18:45:31.238 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.239 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.239 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.240 DEBUG:boto:Params: {} 2023-11-02T18:45:31.240 DEBUG:boto:Token: None 2023-11-02T18:45:31.241 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.241 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069b123acb2774a04-006543ee4b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.251 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T18:45:31.253 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:31.253 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:31.253 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.254 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.254 DEBUG:boto:Method: GET 2023-11-02T18:45:31.255 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.256 DEBUG:boto:Data: 2023-11-02T18:45:31.256 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.257 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.257 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.258 DEBUG:boto:Params: {} 2023-11-02T18:45:31.259 DEBUG:boto:Token: None 2023-11-02T18:45:31.259 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.260 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kjS/Scw7V0CAfi+fWB5hC9w44U4= 2023-11-02T18:45:31.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kjS/Scw7V0CAfi+fWB5hC9w44U4=', 'Content-Length': '0'} 2023-11-02T18:45:31.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000611b7cc822ce15a9-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.265 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:31.266 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:31.266 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.267 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.268 DEBUG:boto:Method: GET 2023-11-02T18:45:31.269 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.269 DEBUG:boto:Data: 2023-11-02T18:45:31.270 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.271 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.271 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.272 DEBUG:boto:Params: {} 2023-11-02T18:45:31.272 DEBUG:boto:Token: None 2023-11-02T18:45:31.273 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.274 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kjS/Scw7V0CAfi+fWB5hC9w44U4= 2023-11-02T18:45:31.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kjS/Scw7V0CAfi+fWB5hC9w44U4=', 'Content-Length': '0'} 2023-11-02T18:45:31.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c2aa833d86fbde-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.322 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:31.323 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:31.323 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.324 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.324 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.325 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.325 DEBUG:boto:Data: 2023-11-02T18:45:31.325 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.326 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.326 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.327 DEBUG:boto:Params: {} 2023-11-02T18:45:31.327 DEBUG:boto:Token: None 2023-11-02T18:45:31.328 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.328 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UFVOtCmyxvw8Pt+H74gVJwcQ72U= 2023-11-02T18:45:31.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UFVOtCmyxvw8Pt+H74gVJwcQ72U=', 'Content-Length': '0'} 2023-11-02T18:45:31.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e37e6415efc3aa32-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.332 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:31.333 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:31.333 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.334 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.334 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.335 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.335 DEBUG:boto:Data: 2023-11-02T18:45:31.335 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.336 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.336 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.337 DEBUG:boto:Params: {} 2023-11-02T18:45:31.337 DEBUG:boto:Token: None 2023-11-02T18:45:31.338 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.338 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UFVOtCmyxvw8Pt+H74gVJwcQ72U= 2023-11-02T18:45:31.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UFVOtCmyxvw8Pt+H74gVJwcQ72U=', 'Content-Length': '0'} 2023-11-02T18:45:31.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf70c0c0893d4c59-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T18:45:31.342 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:31.343 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:31.343 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.344 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.344 DEBUG:boto:Method: GET 2023-11-02T18:45:31.344 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.345 DEBUG:boto:Data: 2023-11-02T18:45:31.345 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.346 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.346 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.347 DEBUG:boto:Params: {} 2023-11-02T18:45:31.347 DEBUG:boto:Token: None 2023-11-02T18:45:31.348 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.348 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kjS/Scw7V0CAfi+fWB5hC9w44U4= 2023-11-02T18:45:31.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kjS/Scw7V0CAfi+fWB5hC9w44U4=', 'Content-Length': '0'} 2023-11-02T18:45:31.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ae098553317cc2-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.353 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:31.354 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:31.354 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.355 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.355 DEBUG:boto:Method: GET 2023-11-02T18:45:31.356 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.356 DEBUG:boto:Data: 2023-11-02T18:45:31.357 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.357 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.358 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.358 DEBUG:boto:Params: {} 2023-11-02T18:45:31.358 DEBUG:boto:Token: None 2023-11-02T18:45:31.359 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:31.360 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kjS/Scw7V0CAfi+fWB5hC9w44U4= 2023-11-02T18:45:31.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kjS/Scw7V0CAfi+fWB5hC9w44U4=', 'Content-Length': '0'} 2023-11-02T18:45:31.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000507a0953b60dc5c7-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.364 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.364 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.365 DEBUG:boto:path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:31.365 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:31.366 DEBUG:boto:Method: GET 2023-11-02T18:45:31.366 DEBUG:boto:Path: /ibtljn-59/?prefix=obj62 2023-11-02T18:45:31.366 DEBUG:boto:Data: 2023-11-02T18:45:31.367 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.367 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.368 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.368 DEBUG:boto:Params: {} 2023-11-02T18:45:31.369 DEBUG:boto:Token: None 2023-11-02T18:45:31.369 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.370 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a89ad9847a1467ef-006543ee4b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.418 DEBUG:boto:b'ibtljn-59obj621000falseobj622023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.419 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.420 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.420 DEBUG:boto:path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:31.420 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:31.421 DEBUG:boto:Method: GET 2023-11-02T18:45:31.421 DEBUG:boto:Path: /ibtljn-59/?prefix=obj62 2023-11-02T18:45:31.422 DEBUG:boto:Data: 2023-11-02T18:45:31.423 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.423 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.424 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.424 DEBUG:boto:Params: {} 2023-11-02T18:45:31.424 DEBUG:boto:Token: None 2023-11-02T18:45:31.425 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.425 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004adce135ab18cd59-006543ee4b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.436 DEBUG:boto:b'ibtljn-59obj621000falseobj622023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.437 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:31.437 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:31.438 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.438 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:45:31.438 DEBUG:boto:Data: 2023-11-02T18:45:31.439 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.439 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.440 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.440 DEBUG:boto:Params: {} 2023-11-02T18:45:31.441 DEBUG:boto:Token: None 2023-11-02T18:45:31.441 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj62 2023-11-02T18:45:31.442 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gdIyjusWqM8hiSqWi/pIHi67GwY= 2023-11-02T18:45:31.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gdIyjusWqM8hiSqWi/pIHi67GwY=', 'Content-Length': '0'} 2023-11-02T18:45:31.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090e98b7165dfc37e-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.445 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:31.446 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:31.446 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.447 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:45:31.447 DEBUG:boto:Data: 2023-11-02T18:45:31.447 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.448 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.448 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.449 DEBUG:boto:Params: {} 2023-11-02T18:45:31.449 DEBUG:boto:Token: None 2023-11-02T18:45:31.449 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj62 2023-11-02T18:45:31.450 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gdIyjusWqM8hiSqWi/pIHi67GwY= 2023-11-02T18:45:31.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gdIyjusWqM8hiSqWi/pIHi67GwY=', 'Content-Length': '0'} 2023-11-02T18:45:31.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000721cec6249547944-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T18:45:31.454 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:31.454 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:31.455 DEBUG:boto:Method: GET 2023-11-02T18:45:31.455 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:45:31.456 DEBUG:boto:Data: 2023-11-02T18:45:31.456 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.456 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.457 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.457 DEBUG:boto:Params: {} 2023-11-02T18:45:31.458 DEBUG:boto:Token: None 2023-11-02T18:45:31.458 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj62 2023-11-02T18:45:31.459 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UegnSkuk7ulxOwRv7vy2zB3lsOU= 2023-11-02T18:45:31.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UegnSkuk7ulxOwRv7vy2zB3lsOU=', 'Content-Length': '0'} 2023-11-02T18:45:31.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e973cb423c7f48ac-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.462 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:31.463 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:31.463 DEBUG:boto:Method: GET 2023-11-02T18:45:31.464 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:45:31.464 DEBUG:boto:Data: 2023-11-02T18:45:31.465 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.465 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.466 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.466 DEBUG:boto:Params: {} 2023-11-02T18:45:31.466 DEBUG:boto:Token: None 2023-11-02T18:45:31.467 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj62 2023-11-02T18:45:31.467 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UegnSkuk7ulxOwRv7vy2zB3lsOU= 2023-11-02T18:45:31.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UegnSkuk7ulxOwRv7vy2zB3lsOU=', 'Content-Length': '0'} 2023-11-02T18:45:31.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005a9b1e57916244c-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.471 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.472 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.472 DEBUG:boto:path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:31.473 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:31.473 DEBUG:boto:Method: GET 2023-11-02T18:45:31.474 DEBUG:boto:Path: /ibtljn-59/?prefix=obj62 2023-11-02T18:45:31.474 DEBUG:boto:Data: 2023-11-02T18:45:31.475 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.475 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.475 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.476 DEBUG:boto:Params: {} 2023-11-02T18:45:31.476 DEBUG:boto:Token: None 2023-11-02T18:45:31.477 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.477 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000289097443734328f-006543ee4b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.483 DEBUG:boto:b'ibtljn-59obj621000falseobj622023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.483 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.484 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.484 DEBUG:boto:path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:31.485 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:31.486 DEBUG:boto:Method: GET 2023-11-02T18:45:31.486 DEBUG:boto:Path: /ibtljn-59/?prefix=obj62 2023-11-02T18:45:31.487 DEBUG:boto:Data: 2023-11-02T18:45:31.487 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.487 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.488 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.488 DEBUG:boto:Params: {} 2023-11-02T18:45:31.489 DEBUG:boto:Token: None 2023-11-02T18:45:31.489 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.490 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eea60963f2f0b7e-006543ee4b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.542 DEBUG:boto:b'ibtljn-59obj621000falseobj622023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:45:31.544 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:31.544 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:31.545 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.545 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.546 DEBUG:boto:Method: GET 2023-11-02T18:45:31.548 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.548 DEBUG:boto:Data: 2023-11-02T18:45:31.549 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.550 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.550 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.551 DEBUG:boto:Params: {} 2023-11-02T18:45:31.551 DEBUG:boto:Token: None 2023-11-02T18:45:31.552 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.552 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Da3w6Te3wBShgFccz3xkQVQZ1kQ= 2023-11-02T18:45:31.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Da3w6Te3wBShgFccz3xkQVQZ1kQ=', 'Content-Length': '0'} 2023-11-02T18:45:31.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4644eb5d9495d5e-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.557 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:31.557 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:31.558 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.558 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.559 DEBUG:boto:Method: GET 2023-11-02T18:45:31.559 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.560 DEBUG:boto:Data: 2023-11-02T18:45:31.560 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.561 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.561 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.561 DEBUG:boto:Params: {} 2023-11-02T18:45:31.562 DEBUG:boto:Token: None 2023-11-02T18:45:31.562 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.563 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Da3w6Te3wBShgFccz3xkQVQZ1kQ= 2023-11-02T18:45:31.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Da3w6Te3wBShgFccz3xkQVQZ1kQ=', 'Content-Length': '0'} 2023-11-02T18:45:31.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000972cb48a1351347f-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.567 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:31.567 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:31.568 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.568 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.569 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.569 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.570 DEBUG:boto:Data: 2023-11-02T18:45:31.570 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.570 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.571 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.571 DEBUG:boto:Params: {} 2023-11-02T18:45:31.572 DEBUG:boto:Token: None 2023-11-02T18:45:31.572 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.573 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Evi3b+o9mUi4U3wYaCA0mOArG2c= 2023-11-02T18:45:31.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Evi3b+o9mUi4U3wYaCA0mOArG2c=', 'Content-Length': '0'} 2023-11-02T18:45:31.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001597d8db71dd59bf-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.576 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:31.577 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:31.577 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.578 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.578 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.578 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.579 DEBUG:boto:Data: 2023-11-02T18:45:31.579 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.580 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.580 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.581 DEBUG:boto:Params: {} 2023-11-02T18:45:31.581 DEBUG:boto:Token: None 2023-11-02T18:45:31.582 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.582 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Evi3b+o9mUi4U3wYaCA0mOArG2c= 2023-11-02T18:45:31.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Evi3b+o9mUi4U3wYaCA0mOArG2c=', 'Content-Length': '0'} 2023-11-02T18:45:31.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000106b084b042f192b-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:45:31.587 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:31.587 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:31.588 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.588 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.589 DEBUG:boto:Method: GET 2023-11-02T18:45:31.589 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.589 DEBUG:boto:Data: 2023-11-02T18:45:31.590 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.591 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.591 DEBUG:boto:Params: {} 2023-11-02T18:45:31.592 DEBUG:boto:Token: None 2023-11-02T18:45:31.592 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Da3w6Te3wBShgFccz3xkQVQZ1kQ= 2023-11-02T18:45:31.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Da3w6Te3wBShgFccz3xkQVQZ1kQ=', 'Content-Length': '0'} 2023-11-02T18:45:31.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000749a285a9737b981-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.638 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:31.639 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:31.639 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.640 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.640 DEBUG:boto:Method: GET 2023-11-02T18:45:31.641 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.641 DEBUG:boto:Data: 2023-11-02T18:45:31.642 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.642 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.643 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.643 DEBUG:boto:Params: {} 2023-11-02T18:45:31.643 DEBUG:boto:Token: None 2023-11-02T18:45:31.644 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:31.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Da3w6Te3wBShgFccz3xkQVQZ1kQ= 2023-11-02T18:45:31.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Da3w6Te3wBShgFccz3xkQVQZ1kQ=', 'Content-Length': '0'} 2023-11-02T18:45:31.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060cb2719efbd0565-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.649 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.649 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.650 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:31.650 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:31.650 DEBUG:boto:Method: GET 2023-11-02T18:45:31.651 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:45:31.651 DEBUG:boto:Data: 2023-11-02T18:45:31.652 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.652 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.653 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.653 DEBUG:boto:Params: {} 2023-11-02T18:45:31.653 DEBUG:boto:Token: None 2023-11-02T18:45:31.654 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.654 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecece717b6bf9873-006543ee4b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.659 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.660 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.660 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.661 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:31.661 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:31.662 DEBUG:boto:Method: GET 2023-11-02T18:45:31.662 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:45:31.663 DEBUG:boto:Data: 2023-11-02T18:45:31.663 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.664 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.664 DEBUG:boto:Params: {} 2023-11-02T18:45:31.665 DEBUG:boto:Token: None 2023-11-02T18:45:31.665 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.666 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aa457e32c7add2e-006543ee4b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.718 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.719 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:31.720 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:31.720 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.721 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:45:31.721 DEBUG:boto:Data: 2023-11-02T18:45:31.721 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.722 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.722 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.723 DEBUG:boto:Params: {} 2023-11-02T18:45:31.723 DEBUG:boto:Token: None 2023-11-02T18:45:31.724 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj68 2023-11-02T18:45:31.724 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0M9kZNuKduLfXbfU9+ssvYQrEgI= 2023-11-02T18:45:31.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0M9kZNuKduLfXbfU9+ssvYQrEgI=', 'Content-Length': '0'} 2023-11-02T18:45:31.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8976ba8b7df1253-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.728 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:31.729 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:31.729 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.730 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:45:31.730 DEBUG:boto:Data: 2023-11-02T18:45:31.730 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.731 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.731 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.732 DEBUG:boto:Params: {} 2023-11-02T18:45:31.732 DEBUG:boto:Token: None 2023-11-02T18:45:31.733 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj68 2023-11-02T18:45:31.733 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0M9kZNuKduLfXbfU9+ssvYQrEgI= 2023-11-02T18:45:31.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0M9kZNuKduLfXbfU9+ssvYQrEgI=', 'Content-Length': '0'} 2023-11-02T18:45:31.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acaea62ec9889341-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:45:31.737 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:31.738 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:31.738 DEBUG:boto:Method: GET 2023-11-02T18:45:31.739 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:45:31.739 DEBUG:boto:Data: 2023-11-02T18:45:31.740 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.740 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.740 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.741 DEBUG:boto:Params: {} 2023-11-02T18:45:31.741 DEBUG:boto:Token: None 2023-11-02T18:45:31.742 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj68 2023-11-02T18:45:31.742 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qBcSDtzCgOsQDoJwfd7fbsU1dKk= 2023-11-02T18:45:31.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qBcSDtzCgOsQDoJwfd7fbsU1dKk=', 'Content-Length': '0'} 2023-11-02T18:45:31.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6062636c358ff64-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.747 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:31.747 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:31.747 DEBUG:boto:Method: GET 2023-11-02T18:45:31.748 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:45:31.748 DEBUG:boto:Data: 2023-11-02T18:45:31.749 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.749 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.750 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.750 DEBUG:boto:Params: {} 2023-11-02T18:45:31.751 DEBUG:boto:Token: None 2023-11-02T18:45:31.751 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj68 2023-11-02T18:45:31.752 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qBcSDtzCgOsQDoJwfd7fbsU1dKk= 2023-11-02T18:45:31.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qBcSDtzCgOsQDoJwfd7fbsU1dKk=', 'Content-Length': '0'} 2023-11-02T18:45:31.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6c4d8708faaeaae-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.756 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.756 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.757 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:31.757 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:31.758 DEBUG:boto:Method: GET 2023-11-02T18:45:31.758 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:45:31.758 DEBUG:boto:Data: 2023-11-02T18:45:31.759 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.759 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.760 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.760 DEBUG:boto:Params: {} 2023-11-02T18:45:31.761 DEBUG:boto:Token: None 2023-11-02T18:45:31.761 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.762 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d88e67617814354d-006543ee4b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.767 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.768 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.768 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.769 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:31.769 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:31.770 DEBUG:boto:Method: GET 2023-11-02T18:45:31.770 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:45:31.770 DEBUG:boto:Data: 2023-11-02T18:45:31.771 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.771 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.772 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.772 DEBUG:boto:Params: {} 2023-11-02T18:45:31.773 DEBUG:boto:Token: None 2023-11-02T18:45:31.773 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.774 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3a082d70f5da32e-006543ee4b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.783 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:45:31.784 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:31.785 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:31.785 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.786 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.786 DEBUG:boto:Method: GET 2023-11-02T18:45:31.787 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.787 DEBUG:boto:Data: 2023-11-02T18:45:31.787 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.788 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.788 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.789 DEBUG:boto:Params: {} 2023-11-02T18:45:31.789 DEBUG:boto:Token: None 2023-11-02T18:45:31.790 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.790 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8akpDsKU2t5tFDn+k4cdLNZfWmA= 2023-11-02T18:45:31.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8akpDsKU2t5tFDn+k4cdLNZfWmA=', 'Content-Length': '0'} 2023-11-02T18:45:31.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aceb5fe86e44166-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.835 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:31.836 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:31.836 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.837 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.837 DEBUG:boto:Method: GET 2023-11-02T18:45:31.837 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.839 DEBUG:boto:Data: 2023-11-02T18:45:31.839 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.840 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.840 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.841 DEBUG:boto:Params: {} 2023-11-02T18:45:31.841 DEBUG:boto:Token: None 2023-11-02T18:45:31.842 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.842 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8akpDsKU2t5tFDn+k4cdLNZfWmA= 2023-11-02T18:45:31.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8akpDsKU2t5tFDn+k4cdLNZfWmA=', 'Content-Length': '0'} 2023-11-02T18:45:31.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5c9d945f68afb9a-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.847 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:31.847 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:31.848 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.848 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.849 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.849 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.850 DEBUG:boto:Data: 2023-11-02T18:45:31.850 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.851 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.851 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.852 DEBUG:boto:Params: {} 2023-11-02T18:45:31.852 DEBUG:boto:Token: None 2023-11-02T18:45:31.853 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.853 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:y/FqyLohPtvuvJglLlka+1nREsY= 2023-11-02T18:45:31.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:y/FqyLohPtvuvJglLlka+1nREsY=', 'Content-Length': '0'} 2023-11-02T18:45:31.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003afc0179819a596b-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.857 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:31.857 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:31.858 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.858 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.859 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.859 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.860 DEBUG:boto:Data: 2023-11-02T18:45:31.860 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.860 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.861 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.861 DEBUG:boto:Params: {} 2023-11-02T18:45:31.862 DEBUG:boto:Token: None 2023-11-02T18:45:31.862 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.863 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:y/FqyLohPtvuvJglLlka+1nREsY= 2023-11-02T18:45:31.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:y/FqyLohPtvuvJglLlka+1nREsY=', 'Content-Length': '0'} 2023-11-02T18:45:31.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c7e20556701ca26-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:45:31.868 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:31.868 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:31.869 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.869 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.869 DEBUG:boto:Method: GET 2023-11-02T18:45:31.870 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.870 DEBUG:boto:Data: 2023-11-02T18:45:31.871 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.871 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.872 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.872 DEBUG:boto:Params: {} 2023-11-02T18:45:31.873 DEBUG:boto:Token: None 2023-11-02T18:45:31.873 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.874 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8akpDsKU2t5tFDn+k4cdLNZfWmA= 2023-11-02T18:45:31.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8akpDsKU2t5tFDn+k4cdLNZfWmA=', 'Content-Length': '0'} 2023-11-02T18:45:31.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a05e6393283fb32d-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.878 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:31.878 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:31.879 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.879 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.879 DEBUG:boto:Method: GET 2023-11-02T18:45:31.880 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.880 DEBUG:boto:Data: 2023-11-02T18:45:31.881 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.881 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.882 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.882 DEBUG:boto:Params: {} 2023-11-02T18:45:31.883 DEBUG:boto:Token: None 2023-11-02T18:45:31.883 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:31.884 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8akpDsKU2t5tFDn+k4cdLNZfWmA= 2023-11-02T18:45:31.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8akpDsKU2t5tFDn+k4cdLNZfWmA=', 'Content-Length': '0'} 2023-11-02T18:45:31.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036f789928ef47786-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.930 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.931 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.932 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:45:31.932 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:45:31.932 DEBUG:boto:Method: GET 2023-11-02T18:45:31.933 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:45:31.933 DEBUG:boto:Data: 2023-11-02T18:45:31.934 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.935 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.935 DEBUG:boto:Params: {} 2023-11-02T18:45:31.936 DEBUG:boto:Token: None 2023-11-02T18:45:31.936 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.937 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000052d5d0a35b949d9-006543ee4b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.943 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.944 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.944 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.945 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:45:31.945 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:45:31.945 DEBUG:boto:Method: GET 2023-11-02T18:45:31.946 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:45:31.947 DEBUG:boto:Data: 2023-11-02T18:45:31.947 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.947 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.948 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.948 DEBUG:boto:Params: {} 2023-11-02T18:45:31.949 DEBUG:boto:Token: None 2023-11-02T18:45:31.949 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/ 2023-11-02T18:45:31.950 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY= 2023-11-02T18:45:31.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pcmmIcxu4gwVhhWehBbmAz8T/JY=', 'Content-Length': '0'} 2023-11-02T18:45:31.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015adf696c978e489-006543ee4b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.960 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:31.961 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:31.961 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:31.962 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.962 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:45:31.963 DEBUG:boto:Data: 2023-11-02T18:45:31.963 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.963 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.964 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.964 DEBUG:boto:Params: {} 2023-11-02T18:45:31.965 DEBUG:boto:Token: None 2023-11-02T18:45:31.965 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj69 2023-11-02T18:45:31.966 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PpBYL+DlQqPkyYtVxvy6/bNDRTg= 2023-11-02T18:45:31.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PpBYL+DlQqPkyYtVxvy6/bNDRTg=', 'Content-Length': '0'} 2023-11-02T18:45:31.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d0e69964633ef0d-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.969 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:31.970 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:31.970 DEBUG:boto:Method: HEAD 2023-11-02T18:45:31.970 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:45:31.971 DEBUG:boto:Data: 2023-11-02T18:45:31.971 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.972 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.972 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.973 DEBUG:boto:Params: {} 2023-11-02T18:45:31.973 DEBUG:boto:Token: None 2023-11-02T18:45:31.974 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj69 2023-11-02T18:45:31.974 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PpBYL+DlQqPkyYtVxvy6/bNDRTg= 2023-11-02T18:45:31.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PpBYL+DlQqPkyYtVxvy6/bNDRTg=', 'Content-Length': '0'} 2023-11-02T18:45:31.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003a5184c09282f5b-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:45:31.978 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:31.978 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:31.979 DEBUG:boto:Method: GET 2023-11-02T18:45:31.979 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:45:31.980 DEBUG:boto:Data: 2023-11-02T18:45:31.980 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.981 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:31.981 DEBUG:boto:Port: 8001 2023-11-02T18:45:31.982 DEBUG:boto:Params: {} 2023-11-02T18:45:31.982 DEBUG:boto:Token: None 2023-11-02T18:45:31.983 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj69 2023-11-02T18:45:31.983 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z2pG5fFb6NJkwXUsxBEwved+uVs= 2023-11-02T18:45:31.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z2pG5fFb6NJkwXUsxBEwved+uVs=', 'Content-Length': '0'} 2023-11-02T18:45:31.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035c0903b9378de21-006543ee4b-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.988 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:31.988 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:31.989 DEBUG:boto:Method: GET 2023-11-02T18:45:31.989 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:45:31.990 DEBUG:boto:Data: 2023-11-02T18:45:31.990 DEBUG:boto:Headers: {} 2023-11-02T18:45:31.991 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:31.991 DEBUG:boto:Port: 8000 2023-11-02T18:45:31.992 DEBUG:boto:Params: {} 2023-11-02T18:45:31.992 DEBUG:boto:Token: None 2023-11-02T18:45:31.993 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:31 GMT /ibtljn-59/obj69 2023-11-02T18:45:31.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z2pG5fFb6NJkwXUsxBEwved+uVs= 2023-11-02T18:45:31.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z2pG5fFb6NJkwXUsxBEwved+uVs=', 'Content-Length': '0'} 2023-11-02T18:45:31.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7752dfb5e697482-006543ee4b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:31.997 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:31.998 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:31.998 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:45:31.998 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:45:31.999 DEBUG:boto:Method: GET 2023-11-02T18:45:31.999 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:45:32.000 DEBUG:boto:Data: 2023-11-02T18:45:32.000 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.000 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.001 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.001 DEBUG:boto:Params: {} 2023-11-02T18:45:32.002 DEBUG:boto:Token: None 2023-11-02T18:45:32.002 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d775cd3d9031a38-006543ee4b-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.050 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.051 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.052 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.052 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:45:32.053 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:45:32.053 DEBUG:boto:Method: GET 2023-11-02T18:45:32.054 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:45:32.054 DEBUG:boto:Data: 2023-11-02T18:45:32.054 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.055 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.055 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.056 DEBUG:boto:Params: {} 2023-11-02T18:45:32.056 DEBUG:boto:Token: None 2023-11-02T18:45:32.057 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.057 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c3a5db846d541e8-006543ee4c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.069 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-11-02T18:45:32.070 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:45:32.070 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:45:32.071 DEBUG:boto:path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.071 DEBUG:boto:auth_path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.071 DEBUG:boto:Method: GET 2023-11-02T18:45:32.072 DEBUG:boto:Path: /ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.072 DEBUG:boto:Data: 2023-11-02T18:45:32.073 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.073 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.074 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.074 DEBUG:boto:Params: {} 2023-11-02T18:45:32.075 DEBUG:boto:Token: None 2023-11-02T18:45:32.075 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.076 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:88pajX3tCz4b7FBY9vxR+32fRTI= 2023-11-02T18:45:32.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:88pajX3tCz4b7FBY9vxR+32fRTI=', 'Content-Length': '0'} 2023-11-02T18:45:32.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079c47a293d5051a4-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.080 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:45:32.081 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:45:32.081 DEBUG:boto:path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.082 DEBUG:boto:auth_path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.082 DEBUG:boto:Method: GET 2023-11-02T18:45:32.083 DEBUG:boto:Path: /ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.083 DEBUG:boto:Data: 2023-11-02T18:45:32.083 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.084 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.084 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.085 DEBUG:boto:Params: {} 2023-11-02T18:45:32.085 DEBUG:boto:Token: None 2023-11-02T18:45:32.086 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.086 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:88pajX3tCz4b7FBY9vxR+32fRTI= 2023-11-02T18:45:32.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:88pajX3tCz4b7FBY9vxR+32fRTI=', 'Content-Length': '0'} 2023-11-02T18:45:32.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001697da47bc9849a3-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.135 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:45:32.135 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:45:32.136 DEBUG:boto:path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.136 DEBUG:boto:auth_path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.137 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.137 DEBUG:boto:Path: /ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.138 DEBUG:boto:Data: 2023-11-02T18:45:32.138 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.139 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.139 DEBUG:boto:Params: {} 2023-11-02T18:45:32.140 DEBUG:boto:Token: None 2023-11-02T18:45:32.141 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G22MHahJARnuly4QxG9GfepMGHA= 2023-11-02T18:45:32.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G22MHahJARnuly4QxG9GfepMGHA=', 'Content-Length': '0'} 2023-11-02T18:45:32.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b2c1cd68d82c915-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.146 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:45:32.146 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:45:32.147 DEBUG:boto:path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.147 DEBUG:boto:auth_path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.148 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.148 DEBUG:boto:Path: /ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.149 DEBUG:boto:Data: 2023-11-02T18:45:32.149 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.150 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.150 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.151 DEBUG:boto:Params: {} 2023-11-02T18:45:32.152 DEBUG:boto:Token: None 2023-11-02T18:45:32.152 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.153 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G22MHahJARnuly4QxG9GfepMGHA= 2023-11-02T18:45:32.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G22MHahJARnuly4QxG9GfepMGHA=', 'Content-Length': '0'} 2023-11-02T18:45:32.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b716c5eb93eb0bb-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-11-02T18:45:32.157 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:45:32.158 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:45:32.158 DEBUG:boto:path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.159 DEBUG:boto:auth_path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.159 DEBUG:boto:Method: GET 2023-11-02T18:45:32.160 DEBUG:boto:Path: /ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.160 DEBUG:boto:Data: 2023-11-02T18:45:32.161 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.161 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.162 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.162 DEBUG:boto:Params: {} 2023-11-02T18:45:32.163 DEBUG:boto:Token: None 2023-11-02T18:45:32.164 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.164 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:88pajX3tCz4b7FBY9vxR+32fRTI= 2023-11-02T18:45:32.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:88pajX3tCz4b7FBY9vxR+32fRTI=', 'Content-Length': '0'} 2023-11-02T18:45:32.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdb179d746a9a09f-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.169 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:45:32.169 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:45:32.170 DEBUG:boto:path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.170 DEBUG:boto:auth_path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.171 DEBUG:boto:Method: GET 2023-11-02T18:45:32.171 DEBUG:boto:Path: /ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.172 DEBUG:boto:Data: 2023-11-02T18:45:32.172 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.173 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.173 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.174 DEBUG:boto:Params: {} 2023-11-02T18:45:32.174 DEBUG:boto:Token: None 2023-11-02T18:45:32.175 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj74?versionId=null 2023-11-02T18:45:32.175 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:88pajX3tCz4b7FBY9vxR+32fRTI= 2023-11-02T18:45:32.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:88pajX3tCz4b7FBY9vxR+32fRTI=', 'Content-Length': '0'} 2023-11-02T18:45:32.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000446310b6f631c6a4-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.179 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.180 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.180 DEBUG:boto:path=/ibtljn-59/?prefix=obj74 2023-11-02T18:45:32.181 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj74 2023-11-02T18:45:32.182 DEBUG:boto:Method: GET 2023-11-02T18:45:32.182 DEBUG:boto:Path: /ibtljn-59/?prefix=obj74 2023-11-02T18:45:32.183 DEBUG:boto:Data: 2023-11-02T18:45:32.183 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.184 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.184 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.185 DEBUG:boto:Params: {} 2023-11-02T18:45:32.185 DEBUG:boto:Token: None 2023-11-02T18:45:32.186 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.186 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c67775327faa49c-006543ee4c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.193 DEBUG:boto:b'ibtljn-59obj741000falseobj742023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.194 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.195 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.195 DEBUG:boto:path=/ibtljn-59/?prefix=obj74 2023-11-02T18:45:32.196 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj74 2023-11-02T18:45:32.197 DEBUG:boto:Method: GET 2023-11-02T18:45:32.197 DEBUG:boto:Path: /ibtljn-59/?prefix=obj74 2023-11-02T18:45:32.198 DEBUG:boto:Data: 2023-11-02T18:45:32.199 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.199 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.200 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.200 DEBUG:boto:Params: {} 2023-11-02T18:45:32.201 DEBUG:boto:Token: None 2023-11-02T18:45:32.202 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.203 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000823cea775198fc87-006543ee4c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.254 DEBUG:boto:b'ibtljn-59obj741000falseobj742023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.256 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:45:32.256 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:45:32.257 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.257 DEBUG:boto:Path: /ibtljn-59/obj74 2023-11-02T18:45:32.258 DEBUG:boto:Data: 2023-11-02T18:45:32.258 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.259 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.260 DEBUG:boto:Params: {} 2023-11-02T18:45:32.260 DEBUG:boto:Token: None 2023-11-02T18:45:32.261 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj74 2023-11-02T18:45:32.262 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xebfAI3+5tcxrQEOQ993f9Ke0WU= 2023-11-02T18:45:32.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xebfAI3+5tcxrQEOQ993f9Ke0WU=', 'Content-Length': '0'} 2023-11-02T18:45:32.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f335701325cfea18-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.267 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:45:32.268 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:45:32.269 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.269 DEBUG:boto:Path: /ibtljn-59/obj74 2023-11-02T18:45:32.270 DEBUG:boto:Data: 2023-11-02T18:45:32.271 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.271 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.272 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.273 DEBUG:boto:Params: {} 2023-11-02T18:45:32.274 DEBUG:boto:Token: None 2023-11-02T18:45:32.274 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj74 2023-11-02T18:45:32.275 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xebfAI3+5tcxrQEOQ993f9Ke0WU= 2023-11-02T18:45:32.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xebfAI3+5tcxrQEOQ993f9Ke0WU=', 'Content-Length': '0'} 2023-11-02T18:45:32.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a89b961cdaccb3df-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-11-02T18:45:32.279 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:45:32.280 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:45:32.281 DEBUG:boto:Method: GET 2023-11-02T18:45:32.281 DEBUG:boto:Path: /ibtljn-59/obj74 2023-11-02T18:45:32.282 DEBUG:boto:Data: 2023-11-02T18:45:32.282 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.283 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.284 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.285 DEBUG:boto:Params: {} 2023-11-02T18:45:32.285 DEBUG:boto:Token: None 2023-11-02T18:45:32.286 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj74 2023-11-02T18:45:32.286 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SsWb/HdSQByED8rC7o3k9kOVf6s= 2023-11-02T18:45:32.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SsWb/HdSQByED8rC7o3k9kOVf6s=', 'Content-Length': '0'} 2023-11-02T18:45:32.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005072547a42487406-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.292 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:45:32.293 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:45:32.293 DEBUG:boto:Method: GET 2023-11-02T18:45:32.294 DEBUG:boto:Path: /ibtljn-59/obj74 2023-11-02T18:45:32.295 DEBUG:boto:Data: 2023-11-02T18:45:32.295 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.296 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.297 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.297 DEBUG:boto:Params: {} 2023-11-02T18:45:32.298 DEBUG:boto:Token: None 2023-11-02T18:45:32.299 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj74 2023-11-02T18:45:32.299 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SsWb/HdSQByED8rC7o3k9kOVf6s= 2023-11-02T18:45:32.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SsWb/HdSQByED8rC7o3k9kOVf6s=', 'Content-Length': '0'} 2023-11-02T18:45:32.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f562d07f7f455131-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.305 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.305 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.305 DEBUG:boto:path=/ibtljn-59/?prefix=obj74 2023-11-02T18:45:32.306 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj74 2023-11-02T18:45:32.306 DEBUG:boto:Method: GET 2023-11-02T18:45:32.307 DEBUG:boto:Path: /ibtljn-59/?prefix=obj74 2023-11-02T18:45:32.308 DEBUG:boto:Data: 2023-11-02T18:45:32.308 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.308 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.309 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.309 DEBUG:boto:Params: {} 2023-11-02T18:45:32.310 DEBUG:boto:Token: None 2023-11-02T18:45:32.311 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.311 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccb6cd0437c3b1a0-006543ee4c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.358 DEBUG:boto:b'ibtljn-59obj741000falseobj742023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.359 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.360 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.360 DEBUG:boto:path=/ibtljn-59/?prefix=obj74 2023-11-02T18:45:32.361 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj74 2023-11-02T18:45:32.361 DEBUG:boto:Method: GET 2023-11-02T18:45:32.362 DEBUG:boto:Path: /ibtljn-59/?prefix=obj74 2023-11-02T18:45:32.362 DEBUG:boto:Data: 2023-11-02T18:45:32.362 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.363 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.363 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.364 DEBUG:boto:Params: {} 2023-11-02T18:45:32.364 DEBUG:boto:Token: None 2023-11-02T18:45:32.365 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.365 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3e44fa194c21304-006543ee4c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.377 DEBUG:boto:b'ibtljn-59obj741000falseobj742023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:45:32.378 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:45:32.379 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:45:32.379 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.379 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.380 DEBUG:boto:Method: GET 2023-11-02T18:45:32.380 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.382 DEBUG:boto:Data: 2023-11-02T18:45:32.382 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.382 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.383 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.383 DEBUG:boto:Params: {} 2023-11-02T18:45:32.384 DEBUG:boto:Token: None 2023-11-02T18:45:32.384 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.385 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lzKgZ69zIV/N9NZGufhYei/r1AI= 2023-11-02T18:45:32.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lzKgZ69zIV/N9NZGufhYei/r1AI=', 'Content-Length': '0'} 2023-11-02T18:45:32.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012a46699840689a4-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.388 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:45:32.389 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:45:32.389 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.390 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.390 DEBUG:boto:Method: GET 2023-11-02T18:45:32.391 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.391 DEBUG:boto:Data: 2023-11-02T18:45:32.391 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.392 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.392 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.393 DEBUG:boto:Params: {} 2023-11-02T18:45:32.393 DEBUG:boto:Token: None 2023-11-02T18:45:32.394 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lzKgZ69zIV/N9NZGufhYei/r1AI= 2023-11-02T18:45:32.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lzKgZ69zIV/N9NZGufhYei/r1AI=', 'Content-Length': '0'} 2023-11-02T18:45:32.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162cafd246e3e2be-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.438 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:45:32.439 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:45:32.439 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.440 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.440 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.441 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.441 DEBUG:boto:Data: 2023-11-02T18:45:32.442 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.442 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.443 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.443 DEBUG:boto:Params: {} 2023-11-02T18:45:32.444 DEBUG:boto:Token: None 2023-11-02T18:45:32.444 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.445 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/q0+XyAr7YjXT85crqOIHIg7W5Y= 2023-11-02T18:45:32.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/q0+XyAr7YjXT85crqOIHIg7W5Y=', 'Content-Length': '0'} 2023-11-02T18:45:32.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000efe07db5373e8fa-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.449 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:45:32.449 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:45:32.450 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.450 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.451 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.451 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.453 DEBUG:boto:Data: 2023-11-02T18:45:32.454 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.455 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.455 DEBUG:boto:Params: {} 2023-11-02T18:45:32.456 DEBUG:boto:Token: None 2023-11-02T18:45:32.456 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.457 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/q0+XyAr7YjXT85crqOIHIg7W5Y= 2023-11-02T18:45:32.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/q0+XyAr7YjXT85crqOIHIg7W5Y=', 'Content-Length': '0'} 2023-11-02T18:45:32.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070e23c596648fa67-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:45:32.461 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:45:32.462 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:45:32.462 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.463 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.463 DEBUG:boto:Method: GET 2023-11-02T18:45:32.464 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.464 DEBUG:boto:Data: 2023-11-02T18:45:32.464 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.465 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.465 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.466 DEBUG:boto:Params: {} 2023-11-02T18:45:32.466 DEBUG:boto:Token: None 2023-11-02T18:45:32.467 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.467 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lzKgZ69zIV/N9NZGufhYei/r1AI= 2023-11-02T18:45:32.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lzKgZ69zIV/N9NZGufhYei/r1AI=', 'Content-Length': '0'} 2023-11-02T18:45:32.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071784f38dd52ce1c-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.472 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:45:32.472 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:45:32.473 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.473 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.474 DEBUG:boto:Method: GET 2023-11-02T18:45:32.474 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.475 DEBUG:boto:Data: 2023-11-02T18:45:32.475 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.476 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.476 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.476 DEBUG:boto:Params: {} 2023-11-02T18:45:32.477 DEBUG:boto:Token: None 2023-11-02T18:45:32.477 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:45:32.478 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lzKgZ69zIV/N9NZGufhYei/r1AI= 2023-11-02T18:45:32.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lzKgZ69zIV/N9NZGufhYei/r1AI=', 'Content-Length': '0'} 2023-11-02T18:45:32.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087b44582f9b0779b-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.482 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.482 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.483 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:45:32.483 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:45:32.484 DEBUG:boto:Method: GET 2023-11-02T18:45:32.484 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:45:32.485 DEBUG:boto:Data: 2023-11-02T18:45:32.485 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.486 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.486 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.486 DEBUG:boto:Params: {} 2023-11-02T18:45:32.487 DEBUG:boto:Token: None 2023-11-02T18:45:32.487 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.488 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b85dd33dce3a9439-006543ee4c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.542 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.543 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.544 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.544 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:45:32.545 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:45:32.545 DEBUG:boto:Method: GET 2023-11-02T18:45:32.545 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:45:32.546 DEBUG:boto:Data: 2023-11-02T18:45:32.546 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.547 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.547 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.548 DEBUG:boto:Params: {} 2023-11-02T18:45:32.548 DEBUG:boto:Token: None 2023-11-02T18:45:32.549 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.549 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f4066773d0e8fd0-006543ee4c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.560 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.560 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:45:32.561 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:45:32.561 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.562 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:45:32.562 DEBUG:boto:Data: 2023-11-02T18:45:32.563 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.563 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.563 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.564 DEBUG:boto:Params: {} 2023-11-02T18:45:32.564 DEBUG:boto:Token: None 2023-11-02T18:45:32.565 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj76 2023-11-02T18:45:32.565 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FnmbhfkmHuM1kwVC8Tc9o/iqdZw= 2023-11-02T18:45:32.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FnmbhfkmHuM1kwVC8Tc9o/iqdZw=', 'Content-Length': '0'} 2023-11-02T18:45:32.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce3d7d1b14ce5ed7-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.569 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:45:32.570 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:45:32.570 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.570 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:45:32.571 DEBUG:boto:Data: 2023-11-02T18:45:32.571 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.572 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.572 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.573 DEBUG:boto:Params: {} 2023-11-02T18:45:32.573 DEBUG:boto:Token: None 2023-11-02T18:45:32.574 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj76 2023-11-02T18:45:32.575 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FnmbhfkmHuM1kwVC8Tc9o/iqdZw= 2023-11-02T18:45:32.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FnmbhfkmHuM1kwVC8Tc9o/iqdZw=', 'Content-Length': '0'} 2023-11-02T18:45:32.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000908ffb3df731b9bf-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:45:32.579 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:45:32.579 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:45:32.580 DEBUG:boto:Method: GET 2023-11-02T18:45:32.580 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:45:32.581 DEBUG:boto:Data: 2023-11-02T18:45:32.581 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.582 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.582 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.583 DEBUG:boto:Params: {} 2023-11-02T18:45:32.583 DEBUG:boto:Token: None 2023-11-02T18:45:32.584 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj76 2023-11-02T18:45:32.584 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fxKgzp96FABuMT97FDnmg6TdVgM= 2023-11-02T18:45:32.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fxKgzp96FABuMT97FDnmg6TdVgM=', 'Content-Length': '0'} 2023-11-02T18:45:32.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0cd17844e550077-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.588 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:45:32.589 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:45:32.589 DEBUG:boto:Method: GET 2023-11-02T18:45:32.590 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:45:32.590 DEBUG:boto:Data: 2023-11-02T18:45:32.591 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.591 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.591 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.592 DEBUG:boto:Params: {} 2023-11-02T18:45:32.592 DEBUG:boto:Token: None 2023-11-02T18:45:32.593 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj76 2023-11-02T18:45:32.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fxKgzp96FABuMT97FDnmg6TdVgM= 2023-11-02T18:45:32.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fxKgzp96FABuMT97FDnmg6TdVgM=', 'Content-Length': '0'} 2023-11-02T18:45:32.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7ecb2d629f1ed4b-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.638 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.639 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.639 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:45:32.640 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:45:32.640 DEBUG:boto:Method: GET 2023-11-02T18:45:32.641 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:45:32.641 DEBUG:boto:Data: 2023-11-02T18:45:32.642 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.642 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.643 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.643 DEBUG:boto:Params: {} 2023-11-02T18:45:32.643 DEBUG:boto:Token: None 2023-11-02T18:45:32.644 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a801fbaf946048c0-006543ee4c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.650 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.650 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.651 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.651 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:45:32.652 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:45:32.652 DEBUG:boto:Method: GET 2023-11-02T18:45:32.653 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:45:32.653 DEBUG:boto:Data: 2023-11-02T18:45:32.654 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.654 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.655 DEBUG:boto:Params: {} 2023-11-02T18:45:32.655 DEBUG:boto:Token: None 2023-11-02T18:45:32.656 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cd463c3a472924a-006543ee4c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.666 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:45:32.667 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:45:32.667 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:45:32.668 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.668 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.669 DEBUG:boto:Method: GET 2023-11-02T18:45:32.669 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.669 DEBUG:boto:Data: 2023-11-02T18:45:32.670 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.670 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.671 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.671 DEBUG:boto:Params: {} 2023-11-02T18:45:32.672 DEBUG:boto:Token: None 2023-11-02T18:45:32.673 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.673 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RqM+QERdGAkNLp1zoFmqceGiHhg= 2023-11-02T18:45:32.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RqM+QERdGAkNLp1zoFmqceGiHhg=', 'Content-Length': '0'} 2023-11-02T18:45:32.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce061a2f2d2d0266-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.677 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:45:32.677 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:45:32.678 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.678 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.679 DEBUG:boto:Method: GET 2023-11-02T18:45:32.679 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.680 DEBUG:boto:Data: 2023-11-02T18:45:32.680 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.681 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.681 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.681 DEBUG:boto:Params: {} 2023-11-02T18:45:32.682 DEBUG:boto:Token: None 2023-11-02T18:45:32.682 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.683 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RqM+QERdGAkNLp1zoFmqceGiHhg= 2023-11-02T18:45:32.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RqM+QERdGAkNLp1zoFmqceGiHhg=', 'Content-Length': '0'} 2023-11-02T18:45:32.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028a0f7451d1075ea-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.687 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:45:32.687 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:45:32.688 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.688 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.688 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.689 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.689 DEBUG:boto:Data: 2023-11-02T18:45:32.690 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.691 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.691 DEBUG:boto:Params: {} 2023-11-02T18:45:32.692 DEBUG:boto:Token: None 2023-11-02T18:45:32.692 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.693 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EkkSZTmAyYjWgUIzTtcxbKdTkKQ= 2023-11-02T18:45:32.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EkkSZTmAyYjWgUIzTtcxbKdTkKQ=', 'Content-Length': '0'} 2023-11-02T18:45:32.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d327b06ed8c5362a-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.696 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:45:32.697 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:45:32.697 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.698 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.698 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.699 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.699 DEBUG:boto:Data: 2023-11-02T18:45:32.700 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.700 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.701 DEBUG:boto:Params: {} 2023-11-02T18:45:32.701 DEBUG:boto:Token: None 2023-11-02T18:45:32.702 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.702 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EkkSZTmAyYjWgUIzTtcxbKdTkKQ= 2023-11-02T18:45:32.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EkkSZTmAyYjWgUIzTtcxbKdTkKQ=', 'Content-Length': '0'} 2023-11-02T18:45:32.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000101aeaa76ddbdd16-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:45:32.707 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:45:32.708 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:45:32.708 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.708 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.709 DEBUG:boto:Method: GET 2023-11-02T18:45:32.709 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.710 DEBUG:boto:Data: 2023-11-02T18:45:32.710 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.711 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.711 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.712 DEBUG:boto:Params: {} 2023-11-02T18:45:32.712 DEBUG:boto:Token: None 2023-11-02T18:45:32.713 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.713 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RqM+QERdGAkNLp1zoFmqceGiHhg= 2023-11-02T18:45:32.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RqM+QERdGAkNLp1zoFmqceGiHhg=', 'Content-Length': '0'} 2023-11-02T18:45:32.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003697cfdef741eec1-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.758 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:45:32.759 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:45:32.760 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.760 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.761 DEBUG:boto:Method: GET 2023-11-02T18:45:32.761 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.761 DEBUG:boto:Data: 2023-11-02T18:45:32.762 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.762 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.763 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.763 DEBUG:boto:Params: {} 2023-11-02T18:45:32.764 DEBUG:boto:Token: None 2023-11-02T18:45:32.764 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:45:32.765 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RqM+QERdGAkNLp1zoFmqceGiHhg= 2023-11-02T18:45:32.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RqM+QERdGAkNLp1zoFmqceGiHhg=', 'Content-Length': '0'} 2023-11-02T18:45:32.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d647f9fac5c8685-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.769 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.770 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.770 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:45:32.771 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:45:32.771 DEBUG:boto:Method: GET 2023-11-02T18:45:32.772 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:45:32.772 DEBUG:boto:Data: 2023-11-02T18:45:32.773 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.773 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.774 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.774 DEBUG:boto:Params: {} 2023-11-02T18:45:32.775 DEBUG:boto:Token: None 2023-11-02T18:45:32.775 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.776 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035ef76aca0f34dfb-006543ee4c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.781 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.782 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.783 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.783 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:45:32.783 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:45:32.784 DEBUG:boto:Method: GET 2023-11-02T18:45:32.784 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:45:32.785 DEBUG:boto:Data: 2023-11-02T18:45:32.785 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.786 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.786 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.787 DEBUG:boto:Params: {} 2023-11-02T18:45:32.787 DEBUG:boto:Token: None 2023-11-02T18:45:32.787 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.788 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005335e1d820262c62-006543ee4c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.799 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.799 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:45:32.800 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:45:32.800 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.801 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:45:32.801 DEBUG:boto:Data: 2023-11-02T18:45:32.801 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.802 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.802 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.803 DEBUG:boto:Params: {} 2023-11-02T18:45:32.803 DEBUG:boto:Token: None 2023-11-02T18:45:32.804 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj85 2023-11-02T18:45:32.804 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vAuvE/kkw9IJEkVxOKpQgKWFfDU= 2023-11-02T18:45:32.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vAuvE/kkw9IJEkVxOKpQgKWFfDU=', 'Content-Length': '0'} 2023-11-02T18:45:32.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d122caf94200ee1-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.808 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:45:32.808 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:45:32.809 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.809 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:45:32.809 DEBUG:boto:Data: 2023-11-02T18:45:32.810 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.810 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.811 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.811 DEBUG:boto:Params: {} 2023-11-02T18:45:32.812 DEBUG:boto:Token: None 2023-11-02T18:45:32.812 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj85 2023-11-02T18:45:32.813 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vAuvE/kkw9IJEkVxOKpQgKWFfDU= 2023-11-02T18:45:32.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vAuvE/kkw9IJEkVxOKpQgKWFfDU=', 'Content-Length': '0'} 2023-11-02T18:45:32.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acebaf1396ba54b2-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:45:32.816 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:45:32.817 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:45:32.817 DEBUG:boto:Method: GET 2023-11-02T18:45:32.818 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:45:32.818 DEBUG:boto:Data: 2023-11-02T18:45:32.819 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.819 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.819 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.820 DEBUG:boto:Params: {} 2023-11-02T18:45:32.820 DEBUG:boto:Token: None 2023-11-02T18:45:32.821 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj85 2023-11-02T18:45:32.821 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y6SDehZhacodvMohbS6l03vI+Ck= 2023-11-02T18:45:32.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y6SDehZhacodvMohbS6l03vI+Ck=', 'Content-Length': '0'} 2023-11-02T18:45:32.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e51211907646117-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.866 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:45:32.867 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:45:32.868 DEBUG:boto:Method: GET 2023-11-02T18:45:32.868 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:45:32.868 DEBUG:boto:Data: 2023-11-02T18:45:32.869 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.869 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.870 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.870 DEBUG:boto:Params: {} 2023-11-02T18:45:32.871 DEBUG:boto:Token: None 2023-11-02T18:45:32.871 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj85 2023-11-02T18:45:32.872 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y6SDehZhacodvMohbS6l03vI+Ck= 2023-11-02T18:45:32.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y6SDehZhacodvMohbS6l03vI+Ck=', 'Content-Length': '0'} 2023-11-02T18:45:32.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdb431350b4769be-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.876 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.876 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.877 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:45:32.877 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:45:32.878 DEBUG:boto:Method: GET 2023-11-02T18:45:32.878 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:45:32.878 DEBUG:boto:Data: 2023-11-02T18:45:32.879 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.879 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.880 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.880 DEBUG:boto:Params: {} 2023-11-02T18:45:32.881 DEBUG:boto:Token: None 2023-11-02T18:45:32.881 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.882 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000605c39a30c28bd33-006543ee4c-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.888 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.889 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:32.890 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:32.890 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:45:32.890 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:45:32.891 DEBUG:boto:Method: GET 2023-11-02T18:45:32.891 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:45:32.892 DEBUG:boto:Data: 2023-11-02T18:45:32.892 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.893 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.893 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.893 DEBUG:boto:Params: {} 2023-11-02T18:45:32.894 DEBUG:boto:Token: None 2023-11-02T18:45:32.894 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/ 2023-11-02T18:45:32.895 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY= 2023-11-02T18:45:32.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8tjYjY5UzQfaz1wsXWgky4YmBY=', 'Content-Length': '0'} 2023-11-02T18:45:32.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000855f81716cdad3a8-006543ee4c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.905 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:32.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:45:32.906 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:45:32.907 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:45:32.907 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.908 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.908 DEBUG:boto:Method: GET 2023-11-02T18:45:32.909 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.909 DEBUG:boto:Data: 2023-11-02T18:45:32.910 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.911 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.911 DEBUG:boto:Params: {} 2023-11-02T18:45:32.912 DEBUG:boto:Token: None 2023-11-02T18:45:32.912 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.913 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/eosdOuXFitG1DfQWS2ji95hjck= 2023-11-02T18:45:32.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/eosdOuXFitG1DfQWS2ji95hjck=', 'Content-Length': '0'} 2023-11-02T18:45:32.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d43f2ee42f45e5c0-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.917 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:45:32.917 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:45:32.917 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.918 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.918 DEBUG:boto:Method: GET 2023-11-02T18:45:32.919 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.919 DEBUG:boto:Data: 2023-11-02T18:45:32.920 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.920 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.920 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.921 DEBUG:boto:Params: {} 2023-11-02T18:45:32.921 DEBUG:boto:Token: None 2023-11-02T18:45:32.922 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/eosdOuXFitG1DfQWS2ji95hjck= 2023-11-02T18:45:32.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/eosdOuXFitG1DfQWS2ji95hjck=', 'Content-Length': '0'} 2023-11-02T18:45:32.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d53e357a79681a4-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.966 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:45:32.967 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:45:32.967 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.968 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.968 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.969 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.969 DEBUG:boto:Data: 2023-11-02T18:45:32.969 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.970 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.970 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.971 DEBUG:boto:Params: {} 2023-11-02T18:45:32.971 DEBUG:boto:Token: None 2023-11-02T18:45:32.972 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.972 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lUvR64nbSUw5+yNYRx/GcTbxdVA= 2023-11-02T18:45:32.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lUvR64nbSUw5+yNYRx/GcTbxdVA=', 'Content-Length': '0'} 2023-11-02T18:45:32.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a88c8f78a621117-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.976 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:45:32.977 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:45:32.977 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.978 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.978 DEBUG:boto:Method: HEAD 2023-11-02T18:45:32.978 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.979 DEBUG:boto:Data: 2023-11-02T18:45:32.979 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.980 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:32.980 DEBUG:boto:Port: 8000 2023-11-02T18:45:32.980 DEBUG:boto:Params: {} 2023-11-02T18:45:32.981 DEBUG:boto:Token: None 2023-11-02T18:45:32.981 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.982 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lUvR64nbSUw5+yNYRx/GcTbxdVA= 2023-11-02T18:45:32.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lUvR64nbSUw5+yNYRx/GcTbxdVA=', 'Content-Length': '0'} 2023-11-02T18:45:32.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f9cddbe48e735a-006543ee4c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:32.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:45:32.987 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:45:32.987 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:45:32.987 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.988 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.988 DEBUG:boto:Method: GET 2023-11-02T18:45:32.989 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.989 DEBUG:boto:Data: 2023-11-02T18:45:32.990 DEBUG:boto:Headers: {} 2023-11-02T18:45:32.990 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:32.990 DEBUG:boto:Port: 8001 2023-11-02T18:45:32.991 DEBUG:boto:Params: {} 2023-11-02T18:45:32.991 DEBUG:boto:Token: None 2023-11-02T18:45:32.992 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:32 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:45:32.992 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/eosdOuXFitG1DfQWS2ji95hjck= 2023-11-02T18:45:32.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:32 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/eosdOuXFitG1DfQWS2ji95hjck=', 'Content-Length': '0'} 2023-11-02T18:45:32.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1eb033f30a4f12-006543ee4c-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:32 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.042 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:45:33.043 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:45:33.043 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:45:33.044 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:45:33.044 DEBUG:boto:Method: GET 2023-11-02T18:45:33.045 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:45:33.045 DEBUG:boto:Data: 2023-11-02T18:45:33.046 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.046 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.047 DEBUG:boto:Params: {} 2023-11-02T18:45:33.047 DEBUG:boto:Token: None 2023-11-02T18:45:33.048 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:45:33.048 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1v4S8KoZMr8wCMOwNdBPpQvv2I= 2023-11-02T18:45:33.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1v4S8KoZMr8wCMOwNdBPpQvv2I=', 'Content-Length': '0'} 2023-11-02T18:45:33.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad082b18b26cbd41-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.053 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.054 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.054 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:45:33.054 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:45:33.055 DEBUG:boto:Method: GET 2023-11-02T18:45:33.055 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:45:33.056 DEBUG:boto:Data: 2023-11-02T18:45:33.056 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.057 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.057 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.058 DEBUG:boto:Params: {} 2023-11-02T18:45:33.058 DEBUG:boto:Token: None 2023-11-02T18:45:33.059 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.059 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041f7af1350309ec0-006543ee4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.065 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.066 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.067 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.067 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:45:33.067 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:45:33.069 DEBUG:boto:Method: GET 2023-11-02T18:45:33.070 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:45:33.070 DEBUG:boto:Data: 2023-11-02T18:45:33.071 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.071 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.072 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.072 DEBUG:boto:Params: {} 2023-11-02T18:45:33.073 DEBUG:boto:Token: None 2023-11-02T18:45:33.073 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.074 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000844f2e8c43b76bb9-006543ee4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.084 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.085 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:45:33.085 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:45:33.086 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.086 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:45:33.087 DEBUG:boto:Data: 2023-11-02T18:45:33.087 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.087 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.088 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.088 DEBUG:boto:Params: {} 2023-11-02T18:45:33.089 DEBUG:boto:Token: None 2023-11-02T18:45:33.089 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj87 2023-11-02T18:45:33.090 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f2YcSUNlesAdj4VdAcMIlk7J5dE= 2023-11-02T18:45:33.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f2YcSUNlesAdj4VdAcMIlk7J5dE=', 'Content-Length': '0'} 2023-11-02T18:45:33.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b5b25538c72e258-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.093 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:45:33.093 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:45:33.094 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.094 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:45:33.095 DEBUG:boto:Data: 2023-11-02T18:45:33.095 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.096 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.096 DEBUG:boto:Params: {} 2023-11-02T18:45:33.097 DEBUG:boto:Token: None 2023-11-02T18:45:33.097 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj87 2023-11-02T18:45:33.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f2YcSUNlesAdj4VdAcMIlk7J5dE= 2023-11-02T18:45:33.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f2YcSUNlesAdj4VdAcMIlk7J5dE=', 'Content-Length': '0'} 2023-11-02T18:45:33.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bdcd86790952ae3-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:45:33.102 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:45:33.102 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:45:33.103 DEBUG:boto:Method: GET 2023-11-02T18:45:33.103 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:45:33.104 DEBUG:boto:Data: 2023-11-02T18:45:33.104 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.105 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.105 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.106 DEBUG:boto:Params: {} 2023-11-02T18:45:33.106 DEBUG:boto:Token: None 2023-11-02T18:45:33.107 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj87 2023-11-02T18:45:33.107 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+mrS/iJHSIBDHXQRf9gB5T0GKHA= 2023-11-02T18:45:33.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+mrS/iJHSIBDHXQRf9gB5T0GKHA=', 'Content-Length': '0'} 2023-11-02T18:45:33.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa2caf818ebcbbad-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.111 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:45:33.112 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:45:33.112 DEBUG:boto:Method: GET 2023-11-02T18:45:33.113 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:45:33.113 DEBUG:boto:Data: 2023-11-02T18:45:33.113 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.114 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.114 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.115 DEBUG:boto:Params: {} 2023-11-02T18:45:33.115 DEBUG:boto:Token: None 2023-11-02T18:45:33.116 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj87 2023-11-02T18:45:33.116 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+mrS/iJHSIBDHXQRf9gB5T0GKHA= 2023-11-02T18:45:33.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+mrS/iJHSIBDHXQRf9gB5T0GKHA=', 'Content-Length': '0'} 2023-11-02T18:45:33.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075dc7ea7668e140e-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.166 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.167 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.168 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:45:33.168 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:45:33.168 DEBUG:boto:Method: GET 2023-11-02T18:45:33.169 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:45:33.169 DEBUG:boto:Data: 2023-11-02T18:45:33.170 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.170 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.171 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.171 DEBUG:boto:Params: {} 2023-11-02T18:45:33.172 DEBUG:boto:Token: None 2023-11-02T18:45:33.172 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.173 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000020b034ab9de56c7-006543ee4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.179 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.179 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.180 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.180 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:45:33.181 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:45:33.181 DEBUG:boto:Method: GET 2023-11-02T18:45:33.181 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:45:33.182 DEBUG:boto:Data: 2023-11-02T18:45:33.182 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.183 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.183 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.184 DEBUG:boto:Params: {} 2023-11-02T18:45:33.184 DEBUG:boto:Token: None 2023-11-02T18:45:33.185 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.185 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1584231ed657b8d-006543ee4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.196 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:45:33.197 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:45:33.198 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:45:33.198 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.199 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.199 DEBUG:boto:Method: GET 2023-11-02T18:45:33.200 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.200 DEBUG:boto:Data: 2023-11-02T18:45:33.200 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.201 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.201 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.202 DEBUG:boto:Params: {} 2023-11-02T18:45:33.202 DEBUG:boto:Token: None 2023-11-02T18:45:33.203 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.204 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7cCKcfoluEPw5YF1VKGLMGVKMs8= 2023-11-02T18:45:33.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7cCKcfoluEPw5YF1VKGLMGVKMs8=', 'Content-Length': '0'} 2023-11-02T18:45:33.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7c716d5003f68dc-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.254 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:45:33.255 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:45:33.255 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.256 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.256 DEBUG:boto:Method: GET 2023-11-02T18:45:33.257 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.257 DEBUG:boto:Data: 2023-11-02T18:45:33.258 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.258 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.259 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.259 DEBUG:boto:Params: {} 2023-11-02T18:45:33.260 DEBUG:boto:Token: None 2023-11-02T18:45:33.260 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.261 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7cCKcfoluEPw5YF1VKGLMGVKMs8= 2023-11-02T18:45:33.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7cCKcfoluEPw5YF1VKGLMGVKMs8=', 'Content-Length': '0'} 2023-11-02T18:45:33.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e83e4db283c88d74-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.265 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:45:33.265 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:45:33.266 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.266 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.267 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.267 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.268 DEBUG:boto:Data: 2023-11-02T18:45:33.268 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.268 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.269 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.269 DEBUG:boto:Params: {} 2023-11-02T18:45:33.270 DEBUG:boto:Token: None 2023-11-02T18:45:33.270 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.271 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QabuDOfITouG2ezi9hO4ARtD1SI= 2023-11-02T18:45:33.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QabuDOfITouG2ezi9hO4ARtD1SI=', 'Content-Length': '0'} 2023-11-02T18:45:33.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005360b5e6170aa29a-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.275 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:45:33.275 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:45:33.276 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.276 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.276 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.277 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.277 DEBUG:boto:Data: 2023-11-02T18:45:33.278 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.278 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.279 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.279 DEBUG:boto:Params: {} 2023-11-02T18:45:33.279 DEBUG:boto:Token: None 2023-11-02T18:45:33.280 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.280 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QabuDOfITouG2ezi9hO4ARtD1SI= 2023-11-02T18:45:33.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QabuDOfITouG2ezi9hO4ARtD1SI=', 'Content-Length': '0'} 2023-11-02T18:45:33.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000456776fd01cd29eb-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:45:33.285 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:45:33.285 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:45:33.286 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.286 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.287 DEBUG:boto:Method: GET 2023-11-02T18:45:33.287 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.288 DEBUG:boto:Data: 2023-11-02T18:45:33.288 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.288 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.289 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.289 DEBUG:boto:Params: {} 2023-11-02T18:45:33.290 DEBUG:boto:Token: None 2023-11-02T18:45:33.290 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.291 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7cCKcfoluEPw5YF1VKGLMGVKMs8= 2023-11-02T18:45:33.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7cCKcfoluEPw5YF1VKGLMGVKMs8=', 'Content-Length': '0'} 2023-11-02T18:45:33.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039ff8ac187dfdf38-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.295 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:45:33.296 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:45:33.296 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.297 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.297 DEBUG:boto:Method: GET 2023-11-02T18:45:33.298 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.298 DEBUG:boto:Data: 2023-11-02T18:45:33.299 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.299 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.300 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.300 DEBUG:boto:Params: {} 2023-11-02T18:45:33.301 DEBUG:boto:Token: None 2023-11-02T18:45:33.301 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:45:33.302 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7cCKcfoluEPw5YF1VKGLMGVKMs8= 2023-11-02T18:45:33.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7cCKcfoluEPw5YF1VKGLMGVKMs8=', 'Content-Length': '0'} 2023-11-02T18:45:33.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eafd715b9e58405-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.350 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.351 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.352 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:45:33.352 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:45:33.352 DEBUG:boto:Method: GET 2023-11-02T18:45:33.353 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:45:33.353 DEBUG:boto:Data: 2023-11-02T18:45:33.354 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.355 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.355 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.356 DEBUG:boto:Params: {} 2023-11-02T18:45:33.356 DEBUG:boto:Token: None 2023-11-02T18:45:33.357 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.357 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d18283fc5947c3e1-006543ee4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.363 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.364 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.365 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.365 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:45:33.366 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:45:33.366 DEBUG:boto:Method: GET 2023-11-02T18:45:33.367 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:45:33.367 DEBUG:boto:Data: 2023-11-02T18:45:33.368 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.369 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.369 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.370 DEBUG:boto:Params: {} 2023-11-02T18:45:33.370 DEBUG:boto:Token: None 2023-11-02T18:45:33.371 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.371 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086a3d111bf58b934-006543ee4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.382 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.382 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:45:33.383 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:45:33.383 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.384 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:45:33.384 DEBUG:boto:Data: 2023-11-02T18:45:33.385 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.385 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.385 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.386 DEBUG:boto:Params: {} 2023-11-02T18:45:33.386 DEBUG:boto:Token: None 2023-11-02T18:45:33.387 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj92 2023-11-02T18:45:33.387 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TN+ihjXQj1CKnaQkTYNsGKjxmJ0= 2023-11-02T18:45:33.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TN+ihjXQj1CKnaQkTYNsGKjxmJ0=', 'Content-Length': '0'} 2023-11-02T18:45:33.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098743a238b5b60ae-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.391 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:45:33.391 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:45:33.392 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.392 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:45:33.392 DEBUG:boto:Data: 2023-11-02T18:45:33.393 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.393 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.394 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.394 DEBUG:boto:Params: {} 2023-11-02T18:45:33.395 DEBUG:boto:Token: None 2023-11-02T18:45:33.395 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj92 2023-11-02T18:45:33.396 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TN+ihjXQj1CKnaQkTYNsGKjxmJ0= 2023-11-02T18:45:33.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TN+ihjXQj1CKnaQkTYNsGKjxmJ0=', 'Content-Length': '0'} 2023-11-02T18:45:33.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e94ca63029365b50-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:45:33.400 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:45:33.400 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:45:33.401 DEBUG:boto:Method: GET 2023-11-02T18:45:33.401 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:45:33.401 DEBUG:boto:Data: 2023-11-02T18:45:33.402 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.402 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.403 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.403 DEBUG:boto:Params: {} 2023-11-02T18:45:33.404 DEBUG:boto:Token: None 2023-11-02T18:45:33.404 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj92 2023-11-02T18:45:33.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P1xE8uUsLMiWzCM5fBJJjH64w5M= 2023-11-02T18:45:33.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P1xE8uUsLMiWzCM5fBJJjH64w5M=', 'Content-Length': '0'} 2023-11-02T18:45:33.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cf5e9b435ad4b08-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.450 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:45:33.451 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:45:33.451 DEBUG:boto:Method: GET 2023-11-02T18:45:33.452 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:45:33.452 DEBUG:boto:Data: 2023-11-02T18:45:33.453 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.453 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.454 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.454 DEBUG:boto:Params: {} 2023-11-02T18:45:33.455 DEBUG:boto:Token: None 2023-11-02T18:45:33.455 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj92 2023-11-02T18:45:33.456 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P1xE8uUsLMiWzCM5fBJJjH64w5M= 2023-11-02T18:45:33.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P1xE8uUsLMiWzCM5fBJJjH64w5M=', 'Content-Length': '0'} 2023-11-02T18:45:33.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dad4ab0b9504066c-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.460 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.460 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.461 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:45:33.461 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:45:33.462 DEBUG:boto:Method: GET 2023-11-02T18:45:33.462 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:45:33.463 DEBUG:boto:Data: 2023-11-02T18:45:33.463 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.464 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.464 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.464 DEBUG:boto:Params: {} 2023-11-02T18:45:33.465 DEBUG:boto:Token: None 2023-11-02T18:45:33.465 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.466 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bc7b1f037c7df2a-006543ee4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.472 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.472 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.473 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.473 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:45:33.474 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:45:33.474 DEBUG:boto:Method: GET 2023-11-02T18:45:33.475 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:45:33.475 DEBUG:boto:Data: 2023-11-02T18:45:33.475 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.476 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.476 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.477 DEBUG:boto:Params: {} 2023-11-02T18:45:33.477 DEBUG:boto:Token: None 2023-11-02T18:45:33.478 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.482 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000088ae36b8a8609fc-006543ee4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.534 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T18:45:33.536 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:45:33.536 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:45:33.537 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.537 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.538 DEBUG:boto:Method: GET 2023-11-02T18:45:33.538 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.538 DEBUG:boto:Data: 2023-11-02T18:45:33.539 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.539 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.540 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.540 DEBUG:boto:Params: {} 2023-11-02T18:45:33.541 DEBUG:boto:Token: None 2023-11-02T18:45:33.541 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.542 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:14dXz0zalkNlH12iKRmSEKUhxdg= 2023-11-02T18:45:33.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:14dXz0zalkNlH12iKRmSEKUhxdg=', 'Content-Length': '0'} 2023-11-02T18:45:33.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1e3adc42dbeee6f-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.546 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:45:33.546 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:45:33.547 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.547 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.548 DEBUG:boto:Method: GET 2023-11-02T18:45:33.548 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.549 DEBUG:boto:Data: 2023-11-02T18:45:33.549 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.549 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.550 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.550 DEBUG:boto:Params: {} 2023-11-02T18:45:33.551 DEBUG:boto:Token: None 2023-11-02T18:45:33.551 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.552 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:14dXz0zalkNlH12iKRmSEKUhxdg= 2023-11-02T18:45:33.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:14dXz0zalkNlH12iKRmSEKUhxdg=', 'Content-Length': '0'} 2023-11-02T18:45:33.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005581f680b8b6a298-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.556 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:45:33.556 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:45:33.557 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.557 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.558 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.558 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.559 DEBUG:boto:Data: 2023-11-02T18:45:33.559 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.559 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.560 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.560 DEBUG:boto:Params: {} 2023-11-02T18:45:33.561 DEBUG:boto:Token: None 2023-11-02T18:45:33.561 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.562 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VIHkT408ku2lJW9gZ+gKzqyc/6U= 2023-11-02T18:45:33.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VIHkT408ku2lJW9gZ+gKzqyc/6U=', 'Content-Length': '0'} 2023-11-02T18:45:33.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fba42a47bf3e06e-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.565 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:45:33.566 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:45:33.566 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.567 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.567 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.568 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.568 DEBUG:boto:Data: 2023-11-02T18:45:33.568 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.569 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.569 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.570 DEBUG:boto:Params: {} 2023-11-02T18:45:33.570 DEBUG:boto:Token: None 2023-11-02T18:45:33.571 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.571 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VIHkT408ku2lJW9gZ+gKzqyc/6U= 2023-11-02T18:45:33.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VIHkT408ku2lJW9gZ+gKzqyc/6U=', 'Content-Length': '0'} 2023-11-02T18:45:33.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046270957411186b3-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T18:45:33.575 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:45:33.576 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:45:33.576 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.577 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.577 DEBUG:boto:Method: GET 2023-11-02T18:45:33.578 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.578 DEBUG:boto:Data: 2023-11-02T18:45:33.579 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.579 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.579 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.580 DEBUG:boto:Params: {} 2023-11-02T18:45:33.580 DEBUG:boto:Token: None 2023-11-02T18:45:33.581 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.581 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:14dXz0zalkNlH12iKRmSEKUhxdg= 2023-11-02T18:45:33.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:14dXz0zalkNlH12iKRmSEKUhxdg=', 'Content-Length': '0'} 2023-11-02T18:45:33.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ee4b1d6351cef50-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.585 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:45:33.586 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:45:33.586 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.587 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.587 DEBUG:boto:Method: GET 2023-11-02T18:45:33.588 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.588 DEBUG:boto:Data: 2023-11-02T18:45:33.588 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.589 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.589 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.590 DEBUG:boto:Params: {} 2023-11-02T18:45:33.590 DEBUG:boto:Token: None 2023-11-02T18:45:33.591 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:45:33.591 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:14dXz0zalkNlH12iKRmSEKUhxdg= 2023-11-02T18:45:33.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:14dXz0zalkNlH12iKRmSEKUhxdg=', 'Content-Length': '0'} 2023-11-02T18:45:33.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8bb7a7cd5e33067-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.638 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.639 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.639 DEBUG:boto:path=/ibtljn-59/?prefix=obj93 2023-11-02T18:45:33.640 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj93 2023-11-02T18:45:33.640 DEBUG:boto:Method: GET 2023-11-02T18:45:33.641 DEBUG:boto:Path: /ibtljn-59/?prefix=obj93 2023-11-02T18:45:33.641 DEBUG:boto:Data: 2023-11-02T18:45:33.642 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.642 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.642 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.643 DEBUG:boto:Params: {} 2023-11-02T18:45:33.643 DEBUG:boto:Token: None 2023-11-02T18:45:33.644 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.644 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052d68cd93c195ef0-006543ee4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.649 DEBUG:boto:b'ibtljn-59obj931000falseobj932023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.650 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.651 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.651 DEBUG:boto:path=/ibtljn-59/?prefix=obj93 2023-11-02T18:45:33.651 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj93 2023-11-02T18:45:33.652 DEBUG:boto:Method: GET 2023-11-02T18:45:33.653 DEBUG:boto:Path: /ibtljn-59/?prefix=obj93 2023-11-02T18:45:33.653 DEBUG:boto:Data: 2023-11-02T18:45:33.654 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.655 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.655 DEBUG:boto:Params: {} 2023-11-02T18:45:33.656 DEBUG:boto:Token: None 2023-11-02T18:45:33.656 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.657 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a4f63109aa56716-006543ee4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.665 DEBUG:boto:b'ibtljn-59obj931000falseobj932023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.666 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:45:33.666 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:45:33.667 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.667 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:45:33.668 DEBUG:boto:Data: 2023-11-02T18:45:33.668 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.668 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.669 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.669 DEBUG:boto:Params: {} 2023-11-02T18:45:33.670 DEBUG:boto:Token: None 2023-11-02T18:45:33.670 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj93 2023-11-02T18:45:33.671 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gvBEfWifzSItybHxnhOlJmTuqPw= 2023-11-02T18:45:33.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gvBEfWifzSItybHxnhOlJmTuqPw=', 'Content-Length': '0'} 2023-11-02T18:45:33.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af556f5d66d6f99-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.674 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:45:33.675 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:45:33.675 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.676 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:45:33.677 DEBUG:boto:Data: 2023-11-02T18:45:33.677 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.677 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.678 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.678 DEBUG:boto:Params: {} 2023-11-02T18:45:33.679 DEBUG:boto:Token: None 2023-11-02T18:45:33.679 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj93 2023-11-02T18:45:33.680 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gvBEfWifzSItybHxnhOlJmTuqPw= 2023-11-02T18:45:33.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gvBEfWifzSItybHxnhOlJmTuqPw=', 'Content-Length': '0'} 2023-11-02T18:45:33.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b0d89349f20da41-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T18:45:33.687 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:45:33.687 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:45:33.688 DEBUG:boto:Method: GET 2023-11-02T18:45:33.688 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:45:33.689 DEBUG:boto:Data: 2023-11-02T18:45:33.689 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.690 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.691 DEBUG:boto:Params: {} 2023-11-02T18:45:33.691 DEBUG:boto:Token: None 2023-11-02T18:45:33.692 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj93 2023-11-02T18:45:33.692 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uvJIhsD/2Q3lkiQuwW34Nymz3XE= 2023-11-02T18:45:33.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uvJIhsD/2Q3lkiQuwW34Nymz3XE=', 'Content-Length': '0'} 2023-11-02T18:45:33.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093a5ba777c557008-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.696 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:45:33.696 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:45:33.697 DEBUG:boto:Method: GET 2023-11-02T18:45:33.697 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:45:33.698 DEBUG:boto:Data: 2023-11-02T18:45:33.698 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.699 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.699 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.700 DEBUG:boto:Params: {} 2023-11-02T18:45:33.700 DEBUG:boto:Token: None 2023-11-02T18:45:33.701 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj93 2023-11-02T18:45:33.701 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uvJIhsD/2Q3lkiQuwW34Nymz3XE= 2023-11-02T18:45:33.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uvJIhsD/2Q3lkiQuwW34Nymz3XE=', 'Content-Length': '0'} 2023-11-02T18:45:33.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c31a302ecc60336-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.750 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.751 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.752 DEBUG:boto:path=/ibtljn-59/?prefix=obj93 2023-11-02T18:45:33.752 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj93 2023-11-02T18:45:33.753 DEBUG:boto:Method: GET 2023-11-02T18:45:33.753 DEBUG:boto:Path: /ibtljn-59/?prefix=obj93 2023-11-02T18:45:33.753 DEBUG:boto:Data: 2023-11-02T18:45:33.754 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.755 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.755 DEBUG:boto:Params: {} 2023-11-02T18:45:33.756 DEBUG:boto:Token: None 2023-11-02T18:45:33.757 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a0a74ffe0e52e5c-006543ee4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.763 DEBUG:boto:b'ibtljn-59obj931000falseobj932023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.764 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.764 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.765 DEBUG:boto:path=/ibtljn-59/?prefix=obj93 2023-11-02T18:45:33.766 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj93 2023-11-02T18:45:33.766 DEBUG:boto:Method: GET 2023-11-02T18:45:33.767 DEBUG:boto:Path: /ibtljn-59/?prefix=obj93 2023-11-02T18:45:33.767 DEBUG:boto:Data: 2023-11-02T18:45:33.767 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.768 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.768 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.769 DEBUG:boto:Params: {} 2023-11-02T18:45:33.769 DEBUG:boto:Token: None 2023-11-02T18:45:33.770 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.770 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b34a701feae78e58-006543ee4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.782 DEBUG:boto:b'ibtljn-59obj931000falseobj932023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:45:33.783 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:45:33.783 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:45:33.784 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.784 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.785 DEBUG:boto:Method: GET 2023-11-02T18:45:33.785 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.786 DEBUG:boto:Data: 2023-11-02T18:45:33.786 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.787 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.787 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.788 DEBUG:boto:Params: {} 2023-11-02T18:45:33.788 DEBUG:boto:Token: None 2023-11-02T18:45:33.789 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eu0ffMILNTApnccCCISDi0v+gVE= 2023-11-02T18:45:33.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eu0ffMILNTApnccCCISDi0v+gVE=', 'Content-Length': '0'} 2023-11-02T18:45:33.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056367d164dc40b08-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.834 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:45:33.835 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:45:33.836 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.836 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.836 DEBUG:boto:Method: GET 2023-11-02T18:45:33.837 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.837 DEBUG:boto:Data: 2023-11-02T18:45:33.838 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.838 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.839 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.839 DEBUG:boto:Params: {} 2023-11-02T18:45:33.840 DEBUG:boto:Token: None 2023-11-02T18:45:33.840 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.841 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eu0ffMILNTApnccCCISDi0v+gVE= 2023-11-02T18:45:33.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eu0ffMILNTApnccCCISDi0v+gVE=', 'Content-Length': '0'} 2023-11-02T18:45:33.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4b58bffe321e2ac-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.845 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:45:33.846 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:45:33.846 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.847 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.847 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.848 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.848 DEBUG:boto:Data: 2023-11-02T18:45:33.848 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.849 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.849 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.850 DEBUG:boto:Params: {} 2023-11-02T18:45:33.850 DEBUG:boto:Token: None 2023-11-02T18:45:33.851 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.851 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b3i2CpeGVrcveeEcw5UJ4yHHGj4= 2023-11-02T18:45:33.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b3i2CpeGVrcveeEcw5UJ4yHHGj4=', 'Content-Length': '0'} 2023-11-02T18:45:33.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ff1f8172de6301f-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.855 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:45:33.856 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:45:33.856 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.857 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.857 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.857 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.858 DEBUG:boto:Data: 2023-11-02T18:45:33.858 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.859 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.859 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.860 DEBUG:boto:Params: {} 2023-11-02T18:45:33.860 DEBUG:boto:Token: None 2023-11-02T18:45:33.861 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.861 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b3i2CpeGVrcveeEcw5UJ4yHHGj4= 2023-11-02T18:45:33.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b3i2CpeGVrcveeEcw5UJ4yHHGj4=', 'Content-Length': '0'} 2023-11-02T18:45:33.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e2e224c34e52b59-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:45:33.866 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:45:33.867 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:45:33.867 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.867 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.868 DEBUG:boto:Method: GET 2023-11-02T18:45:33.868 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.869 DEBUG:boto:Data: 2023-11-02T18:45:33.869 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.870 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.871 DEBUG:boto:Params: {} 2023-11-02T18:45:33.871 DEBUG:boto:Token: None 2023-11-02T18:45:33.872 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.872 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eu0ffMILNTApnccCCISDi0v+gVE= 2023-11-02T18:45:33.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eu0ffMILNTApnccCCISDi0v+gVE=', 'Content-Length': '0'} 2023-11-02T18:45:33.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffb9b8f675597332-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.877 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:45:33.877 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:45:33.878 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.878 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.879 DEBUG:boto:Method: GET 2023-11-02T18:45:33.879 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.880 DEBUG:boto:Data: 2023-11-02T18:45:33.880 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.880 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.881 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.881 DEBUG:boto:Params: {} 2023-11-02T18:45:33.882 DEBUG:boto:Token: None 2023-11-02T18:45:33.882 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:45:33.883 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eu0ffMILNTApnccCCISDi0v+gVE= 2023-11-02T18:45:33.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eu0ffMILNTApnccCCISDi0v+gVE=', 'Content-Length': '0'} 2023-11-02T18:45:33.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000941d70586102268d-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.887 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.889 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.890 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:45:33.890 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:45:33.891 DEBUG:boto:Method: GET 2023-11-02T18:45:33.891 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:45:33.892 DEBUG:boto:Data: 2023-11-02T18:45:33.892 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.892 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.893 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.893 DEBUG:boto:Params: {} 2023-11-02T18:45:33.894 DEBUG:boto:Token: None 2023-11-02T18:45:33.894 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.895 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069f77941449704b9-006543ee4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.942 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.943 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.944 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.944 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:45:33.945 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:45:33.945 DEBUG:boto:Method: GET 2023-11-02T18:45:33.946 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:45:33.946 DEBUG:boto:Data: 2023-11-02T18:45:33.947 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.947 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.948 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.948 DEBUG:boto:Params: {} 2023-11-02T18:45:33.949 DEBUG:boto:Token: None 2023-11-02T18:45:33.950 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/ 2023-11-02T18:45:33.950 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug= 2023-11-02T18:45:33.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CZGGitgA1uQRui53DmTQMSIlfug=', 'Content-Length': '0'} 2023-11-02T18:45:33.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4592b79a30979fb-006543ee4d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.960 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:33.961 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:45:33.962 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:45:33.962 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.963 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:45:33.963 DEBUG:boto:Data: 2023-11-02T18:45:33.963 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.964 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.964 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.965 DEBUG:boto:Params: {} 2023-11-02T18:45:33.965 DEBUG:boto:Token: None 2023-11-02T18:45:33.966 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj95 2023-11-02T18:45:33.966 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MFa5JvTYgsmymoP/V8DU49Zg8yw= 2023-11-02T18:45:33.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MFa5JvTYgsmymoP/V8DU49Zg8yw=', 'Content-Length': '0'} 2023-11-02T18:45:33.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028cb64a65f4cf8c5-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.970 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:45:33.970 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:45:33.971 DEBUG:boto:Method: HEAD 2023-11-02T18:45:33.971 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:45:33.972 DEBUG:boto:Data: 2023-11-02T18:45:33.972 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.972 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.973 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.973 DEBUG:boto:Params: {} 2023-11-02T18:45:33.974 DEBUG:boto:Token: None 2023-11-02T18:45:33.974 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj95 2023-11-02T18:45:33.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MFa5JvTYgsmymoP/V8DU49Zg8yw= 2023-11-02T18:45:33.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MFa5JvTYgsmymoP/V8DU49Zg8yw=', 'Content-Length': '0'} 2023-11-02T18:45:33.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df9e0902024e5816-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:45:33.979 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:45:33.979 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:45:33.980 DEBUG:boto:Method: GET 2023-11-02T18:45:33.980 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:45:33.981 DEBUG:boto:Data: 2023-11-02T18:45:33.981 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.982 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:33.982 DEBUG:boto:Port: 8001 2023-11-02T18:45:33.983 DEBUG:boto:Params: {} 2023-11-02T18:45:33.983 DEBUG:boto:Token: None 2023-11-02T18:45:33.984 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj95 2023-11-02T18:45:33.984 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V7bnYvO8+q7tK2xUjbNPSiPeNcY= 2023-11-02T18:45:33.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V7bnYvO8+q7tK2xUjbNPSiPeNcY=', 'Content-Length': '0'} 2023-11-02T18:45:33.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f74359f47083edf0-006543ee4d-1b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.988 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:45:33.989 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:45:33.989 DEBUG:boto:Method: GET 2023-11-02T18:45:33.989 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:45:33.990 DEBUG:boto:Data: 2023-11-02T18:45:33.990 DEBUG:boto:Headers: {} 2023-11-02T18:45:33.991 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:33.991 DEBUG:boto:Port: 8000 2023-11-02T18:45:33.992 DEBUG:boto:Params: {} 2023-11-02T18:45:33.992 DEBUG:boto:Token: None 2023-11-02T18:45:33.993 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:33 GMT /ibtljn-59/obj95 2023-11-02T18:45:33.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V7bnYvO8+q7tK2xUjbNPSiPeNcY= 2023-11-02T18:45:33.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V7bnYvO8+q7tK2xUjbNPSiPeNcY=', 'Content-Length': '0'} 2023-11-02T18:45:33.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c848e28131136b4-006543ee4d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:33.998 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:33.998 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:33.999 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:45:33.999 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:45:33.999 DEBUG:boto:Method: GET 2023-11-02T18:45:34.000 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:45:34.001 DEBUG:boto:Data: 2023-11-02T18:45:34.001 DEBUG:boto:Headers: {} 2023-11-02T18:45:34.001 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:34.002 DEBUG:boto:Port: 8001 2023-11-02T18:45:34.002 DEBUG:boto:Params: {} 2023-11-02T18:45:34.003 DEBUG:boto:Token: None 2023-11-02T18:45:34.004 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:34 GMT /ibtljn-59/ 2023-11-02T18:45:34.004 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pfk7rl4SdNnbWi4vxje3HbemF2U= 2023-11-02T18:45:34.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pfk7rl4SdNnbWi4vxje3HbemF2U=', 'Content-Length': '0'} 2023-11-02T18:45:34.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070b163ce595b4107-006543ee4d-1b8c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:34.009 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:34.010 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:34.010 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:34.011 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:45:34.011 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:45:34.012 DEBUG:boto:Method: GET 2023-11-02T18:45:34.012 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:45:34.013 DEBUG:boto:Data: 2023-11-02T18:45:34.013 DEBUG:boto:Headers: {} 2023-11-02T18:45:34.014 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:34.014 DEBUG:boto:Port: 8000 2023-11-02T18:45:34.015 DEBUG:boto:Params: {} 2023-11-02T18:45:34.015 DEBUG:boto:Token: None 2023-11-02T18:45:34.016 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:34 GMT /ibtljn-59/ 2023-11-02T18:45:34.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pfk7rl4SdNnbWi4vxje3HbemF2U= 2023-11-02T18:45:34.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pfk7rl4SdNnbWi4vxje3HbemF2U=', 'Content-Length': '0'} 2023-11-02T18:45:34.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d65d13f39826b12-006543ee4e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:34.026 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:34.027 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:45:34.028 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-11-02T18:45:34.028 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T18:45:34.029 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:45:34.020+0000 7f3642e02700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 66152) UID: 0 2023-11-02T18:45:34.030 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:45:34.020+0000 7f3646661f80 -1 shutting down 2023-11-02T18:45:37.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-11-02T18:45:40.126 INFO:tasks.rgw.c1.client.1:Stopped 2023-11-02T18:45:40.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:45:40.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:45:40.128 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:45:40.179 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:40.179 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:40.244 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:45:40.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.245 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj11", 2023-11-02T18:45:40.245 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.246 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.246 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.246 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:45:40.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.247 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.247 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.247 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.247 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.247 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:40.248 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.543369Z", 2023-11-02T18:45:40.248 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.248 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.248 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.249 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.250 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.250 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.251 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.251 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.251 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_imR4WJm5sHt1F2sbUmYYVZCJ4-vAjxR", 2023-11-02T18:45:40.252 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.252 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.252 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.253 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj111", 2023-11-02T18:45:40.253 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.253 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.253 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.254 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:45:40.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.254 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.254 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.255 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.255 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.255 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:40.255 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.749440Z", 2023-11-02T18:45:40.256 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.256 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.256 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.256 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.256 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.257 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.257 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.257 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.258 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_L5NRT39xkBB_jBoTmWoj-IJIdoorM71", 2023-11-02T18:45:40.258 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.258 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.258 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.259 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj115", 2023-11-02T18:45:40.259 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.259 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.260 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.260 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1 2023-11-02T18:45:40.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.260 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.261 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.261 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.261 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.261 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:40.262 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.589046Z", 2023-11-02T18:45:40.262 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.262 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.262 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.263 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.263 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.263 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.263 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.263 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.264 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_lw_2200xXcIc-bK3fgvOY31heH5O9yY", 2023-11-02T18:45:40.264 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.264 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.265 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.265 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj116", 2023-11-02T18:45:40.265 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.266 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.266 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.266 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:45:40.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.267 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.267 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.267 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.267 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.267 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:40.268 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.795424Z", 2023-11-02T18:45:40.268 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.268 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.268 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.269 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.269 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.269 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.269 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.270 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.270 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_ecTtabyH2Sk1O33WafGvCJz_oJAH1Io", 2023-11-02T18:45:40.270 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.271 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.271 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.271 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj124", 2023-11-02T18:45:40.272 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.272 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.272 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.272 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:45:40.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.273 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.273 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.273 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.273 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.274 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:40.274 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.406701Z", 2023-11-02T18:45:40.274 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.274 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.274 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.275 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.275 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.275 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.275 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.276 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.276 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_-TARRhBACahUcvfW9yZnfL5bujSy_0q", 2023-11-02T18:45:40.276 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.276 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.277 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.277 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj145", 2023-11-02T18:45:40.277 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.278 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.278 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.278 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:40.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.279 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.279 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.279 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.279 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.280 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:40.280 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.858962Z", 2023-11-02T18:45:40.280 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.280 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.280 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.281 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.281 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.281 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.281 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.281 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.282 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_nXDhJEFLKEzFrkobxUqsHqqzaWCeCgl", 2023-11-02T18:45:40.282 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.282 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.283 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.283 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj148", 2023-11-02T18:45:40.283 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.284 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.284 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.284 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:45:40.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.285 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.285 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.285 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.286 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.286 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:40.286 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.778084Z", 2023-11-02T18:45:40.286 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.287 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.287 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.287 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.287 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.287 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.288 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.288 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.288 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_luqnqBT1GP428wwFmyey9wiDM1LK-FF", 2023-11-02T18:45:40.289 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.289 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.289 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.290 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj155", 2023-11-02T18:45:40.290 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.290 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.290 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.291 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:40.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.291 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.291 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.292 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.292 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.292 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:40.292 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.469524Z", 2023-11-02T18:45:40.292 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.293 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.293 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.293 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.293 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.294 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.294 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.294 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.295 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_IUQoLktzZvdqE68Bqu607GpPPfIjYQA", 2023-11-02T18:45:40.296 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.296 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.296 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.297 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj17", 2023-11-02T18:45:40.297 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.297 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.297 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.298 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:45:40.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.298 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.298 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.298 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.299 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.299 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:40.299 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.738565Z", 2023-11-02T18:45:40.299 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.300 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.300 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.300 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.300 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.301 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.301 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.301 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.301 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_D2vS0zXSU39_7sDVN7Ev8QTY7u0gFht", 2023-11-02T18:45:40.302 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.302 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.302 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.303 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj185", 2023-11-02T18:45:40.303 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.303 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.304 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.304 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1 2023-11-02T18:45:40.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.304 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.304 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.305 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.305 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.305 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:40.305 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.842657Z", 2023-11-02T18:45:40.306 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.306 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.306 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.306 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.307 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.307 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.307 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.307 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.308 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_gKYQh9kqPAqejQGH8Ev5KvnSntpYOl9", 2023-11-02T18:45:40.308 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.308 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.308 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.309 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj19", 2023-11-02T18:45:40.309 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.309 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.310 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.310 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:40.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.310 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.310 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.311 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.311 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.311 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:40.311 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.364071Z", 2023-11-02T18:45:40.312 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.312 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.312 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.312 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.312 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.313 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.313 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.313 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.314 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_kQ47up49HdyN9-jrEbqVqP_17QA3Ohc", 2023-11-02T18:45:40.314 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.314 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.314 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.315 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj22", 2023-11-02T18:45:40.315 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.315 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.316 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.316 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:45:40.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.316 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.317 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.317 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.317 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.317 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:40.317 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.572127Z", 2023-11-02T18:45:40.318 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.318 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.318 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.318 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.319 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.319 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.319 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.319 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.320 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_znlcwXM8ilS-HE0A_cRM0TrHcoCJ5Kg", 2023-11-02T18:45:40.320 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.320 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.320 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.321 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj23", 2023-11-02T18:45:40.321 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.322 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.322 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.322 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:40.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.322 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.323 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.323 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.323 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.323 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:40.324 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.410749Z", 2023-11-02T18:45:40.324 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.324 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.324 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.324 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.325 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.325 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.325 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.325 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.326 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_dE7305B7RTuyM6AnK7kpyDHQY5ScQ1K", 2023-11-02T18:45:40.326 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.326 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.326 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.327 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj29", 2023-11-02T18:45:40.327 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.327 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.328 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.328 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:40.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.328 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.329 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.329 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.329 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.329 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:40.329 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.673282Z", 2023-11-02T18:45:40.330 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.330 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.330 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.331 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.331 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.331 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.331 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.332 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.332 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_uz2NTbovoLhBcbatkDy1y1l3oaCsZUY", 2023-11-02T18:45:40.332 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.332 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.333 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.333 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj31", 2023-11-02T18:45:40.333 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.334 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.334 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.334 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:45:40.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.335 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.335 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.335 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.335 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.336 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:40.336 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.279072Z", 2023-11-02T18:45:40.336 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.336 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.337 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.337 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.337 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.337 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.337 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.338 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.338 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "__rMjch3ijGL5RYsakLA8-FU4ameucQh", 2023-11-02T18:45:40.338 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.339 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.339 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.340 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj33", 2023-11-02T18:45:40.340 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.340 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.340 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.341 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:45:40.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.341 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.341 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.341 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.342 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.342 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:40.342 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.595835Z", 2023-11-02T18:45:40.342 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.343 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.343 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.343 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.343 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.343 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.344 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.344 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.344 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_pNTtyArjWPUtWt4vqbLrh9c7VEOezsx", 2023-11-02T18:45:40.345 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.345 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.345 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.346 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj34", 2023-11-02T18:45:40.346 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.346 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.346 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.347 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:40.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.347 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.347 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.348 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.348 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.348 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:40.348 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.768510Z", 2023-11-02T18:45:40.348 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.349 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.349 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.349 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.350 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.350 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.350 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.350 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.351 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_TIfNlm1gEDOW-4TgdSvyntG2hqeBabv", 2023-11-02T18:45:40.351 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.351 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.351 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.352 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj37", 2023-11-02T18:45:40.352 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.352 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.353 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.353 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:45:40.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.353 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.353 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.354 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.354 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.354 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:40.354 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.652984Z", 2023-11-02T18:45:40.355 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.355 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.355 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.355 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.355 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.356 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.356 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.356 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.356 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_37z1IrMq4azS5lv1clokSukI7V1vRnE", 2023-11-02T18:45:40.357 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.357 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.357 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.358 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj38", 2023-11-02T18:45:40.358 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.358 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.358 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.359 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:40.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.359 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.359 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.360 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.360 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.360 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:40.360 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.393766Z", 2023-11-02T18:45:40.361 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.361 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.361 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.361 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.362 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.362 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.362 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.362 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.363 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_F3ZwwZVE4uUuGHtUEMwUHkUqoIJku_4", 2023-11-02T18:45:40.363 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.363 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.363 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.364 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj44", 2023-11-02T18:45:40.364 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.365 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.365 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.365 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:40.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.366 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.366 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.366 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.366 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.367 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:40.367 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.654344Z", 2023-11-02T18:45:40.367 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.367 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.368 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.368 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.368 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.368 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.368 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.369 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.369 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_RlCzKRueK6j1bxLBWZm_Snp6MYUZbuN", 2023-11-02T18:45:40.369 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.370 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.370 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.371 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj46", 2023-11-02T18:45:40.371 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.371 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.371 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.372 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:45:40.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.372 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.372 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.373 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.373 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.373 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:40.373 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.439748Z", 2023-11-02T18:45:40.373 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.374 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.374 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.374 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.374 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.375 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.375 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.375 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.375 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_ur4e_TTX5zisIPj8ReJwSvZWiUa-3aA", 2023-11-02T18:45:40.376 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.376 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.376 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.377 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj51", 2023-11-02T18:45:40.377 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.377 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.378 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.378 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:45:40.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.378 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.378 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.379 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.379 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.379 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:40.379 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.828934Z", 2023-11-02T18:45:40.380 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.380 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.380 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.380 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.381 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.381 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.381 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.381 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.382 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_JisX64DlJxWLCeUSf1w3YstImpMUUrI", 2023-11-02T18:45:40.382 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.382 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.382 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.383 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj55", 2023-11-02T18:45:40.383 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.384 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.384 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.384 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:45:40.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.385 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.385 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.385 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.385 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.386 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:40.386 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.681652Z", 2023-11-02T18:45:40.386 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.386 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.387 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.387 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.387 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.387 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.387 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.388 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.388 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_Rdw3GC0aot1wMStqvdhrM7ooEhFXpa6", 2023-11-02T18:45:40.388 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.388 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.389 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.389 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj57", 2023-11-02T18:45:40.390 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.390 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.390 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.390 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:45:40.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.391 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.391 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.391 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.391 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.392 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:40.392 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.455583Z", 2023-11-02T18:45:40.392 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.392 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.393 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.393 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.393 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.393 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.393 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.394 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.394 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_9nm83_JsbFP2i6-BiwLmyg88DpzEBf2", 2023-11-02T18:45:40.394 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.395 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.395 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.395 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj58", 2023-11-02T18:45:40.396 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.396 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.396 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.396 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 14 2023-11-02T18:45:40.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.397 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.397 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.397 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.397 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.398 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:40.398 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.702322Z", 2023-11-02T18:45:40.398 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.398 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.399 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.399 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.399 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.399 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.399 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.400 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.400 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_keXnH4D77oxKMj8fh1T9TXEk3LpDVpC", 2023-11-02T18:45:40.400 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.401 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.401 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.401 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj62", 2023-11-02T18:45:40.402 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.402 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.402 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.402 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:45:40.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.403 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.403 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.403 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.403 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.404 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:40.404 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.308477Z", 2023-11-02T18:45:40.404 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.404 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.404 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.405 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.405 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.405 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.405 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.406 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.406 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_2Pv8SEODmdRrVX4b92mZTac6yy3rS3L", 2023-11-02T18:45:40.406 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.406 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.407 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.407 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj68", 2023-11-02T18:45:40.408 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.408 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.408 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.408 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:45:40.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.409 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.409 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.409 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.409 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.409 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:40.410 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.858441Z", 2023-11-02T18:45:40.410 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.410 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.410 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.410 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.411 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.411 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.411 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.411 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.412 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_tg9yO0jDV6Fc3g1-HbGlpa3r34jHnAc", 2023-11-02T18:45:40.412 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.412 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.412 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.413 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj69", 2023-11-02T18:45:40.413 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.414 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.414 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.414 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:40.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.414 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.415 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.415 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.415 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.415 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:40.416 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.501361Z", 2023-11-02T18:45:40.416 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.416 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.416 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.416 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.417 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.417 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.417 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.417 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.418 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_--IMB-OEA9DnjQ0tVcVBhhov7T9Ra80", 2023-11-02T18:45:40.418 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.418 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.419 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.419 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj74", 2023-11-02T18:45:40.419 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.420 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.420 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.420 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 12 2023-11-02T18:45:40.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.421 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.421 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.421 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.421 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.421 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:40.422 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.681498Z", 2023-11-02T18:45:40.422 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.422 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.422 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.423 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.423 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.423 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.423 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.424 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.424 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_f_cZ9wQe3CArlZvmy6U0gu3MBJh76vL", 2023-11-02T18:45:40.424 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.425 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.425 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.425 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj76", 2023-11-02T18:45:40.426 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.426 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.426 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.426 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:45:40.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.427 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.427 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.427 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.427 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.428 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:40.428 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.481539Z", 2023-11-02T18:45:40.428 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.428 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.428 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.429 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.429 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.429 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.429 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.430 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.430 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.430 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_M8pMh3a6Y0GJw5t6SZyrkG3hCxDsa3j", 2023-11-02T18:45:40.430 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.431 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.431 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.431 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.432 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj85", 2023-11-02T18:45:40.432 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.432 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.432 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.432 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:40.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.433 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.433 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.433 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.434 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.434 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:40.434 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.919466Z", 2023-11-02T18:45:40.434 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.434 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.435 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.435 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.435 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.435 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.436 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.436 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.436 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_v381IEh0vevjDRX-5Dg2fOcSv5ORvIV", 2023-11-02T18:45:40.436 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.437 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.437 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.438 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj87", 2023-11-02T18:45:40.438 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.438 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.438 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.438 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 42 2023-11-02T18:45:40.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.439 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.439 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.442 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.442 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.442 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:40.443 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.765417Z", 2023-11-02T18:45:40.443 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.443 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.443 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.444 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.444 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.444 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.444 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.445 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.445 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_krHrQ5Q55NLCWof93_UtqbiWgSeRb38", 2023-11-02T18:45:40.445 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.445 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.446 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.446 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj92", 2023-11-02T18:45:40.447 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.447 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.447 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.447 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 11 2023-11-02T18:45:40.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.448 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.448 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.448 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.448 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.449 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:40.449 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.528009Z", 2023-11-02T18:45:40.449 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.449 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.450 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.450 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.450 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.450 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.451 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.451 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.451 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_AvcDb72kLUvYIU1vPfhF2mTAkdDuhTi", 2023-11-02T18:45:40.452 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.452 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.452 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.453 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj93", 2023-11-02T18:45:40.453 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.453 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.454 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.454 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:45:40.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.454 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.455 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.455 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.455 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.455 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:40.456 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.377248Z", 2023-11-02T18:45:40.456 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.456 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.456 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.457 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.457 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.457 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.457 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.458 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.458 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_zPtrdCEjTlFZrmC2QrN9T7gY04_lGuK", 2023-11-02T18:45:40.458 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.458 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.459 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:40.459 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj95", 2023-11-02T18:45:40.460 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:40.460 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:40.460 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:40.460 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:40.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.461 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:40.461 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:40.461 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:40.461 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:40.462 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:40.462 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.549728Z", 2023-11-02T18:45:40.462 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:40.462 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:40.462 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:40.463 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:40.463 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:40.463 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:40.463 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:40.463 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:40.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:40.464 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_TEYuRtZDD8RHj0n4su0Dy_YB2uv9b9Q", 2023-11-02T18:45:40.464 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:40.464 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:40.464 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:40.465 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:45:40.465 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:45:40.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:40.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:40.467 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj11 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:40.517 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:40.518 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:40.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:40.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:40.701 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj111 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:40.750 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:40.750 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:40.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:40.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:40.941 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj115 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:41.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:41.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:41.180 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj116 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:41.229 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:41.230 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:41.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:41.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:41.427 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj124 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:41.469 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:41.470 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:41.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:41.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:41.659 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj145 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:41.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:41.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:41.897 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj148 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:41.987 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:41.987 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:42.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:42.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:42.182 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj155 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:42.232 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:42.232 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:42.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:42.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:42.428 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj17 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:42.476 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:42.477 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:42.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-11-02T18:45:42.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:42.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:42.679 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj185 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:42.767 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:42.767 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:42.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:42.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:42.957 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj19 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:43.005 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:43.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:43.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:43.196 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj22 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:43.241 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:43.242 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:43.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:43.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:43.431 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj23 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:43.481 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:43.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:43.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:43.669 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj29 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:43.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:43.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:43.904 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj31 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:43.947 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:43.948 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:44.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:44.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:44.139 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj33 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:44.225 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:44.225 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:44.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:44.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:44.411 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj34 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:44.459 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:44.460 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:44.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:44.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:44.657 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj37 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:44.702 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:44.702 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:44.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:44.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:44.891 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj38 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:44.978 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:44.978 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:45.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:45.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:45.178 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj44 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:45.227 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:45.228 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:45.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:45.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:45.421 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj46 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:45.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:45.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:45.658 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj51 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:45.705 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:45.705 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:45.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:45.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:45.895 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:45.944 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:45.944 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:46.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:46.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:46.143 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj57 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:46.192 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:46.193 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:46.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:46.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:46.384 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj58 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:46.430 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:46.430 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:46.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:46.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:46.624 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj62 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:46.673 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:46.674 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:46.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:46.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:46.862 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:46.909 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:46.910 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:47.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:47.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:47.103 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj69 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:47.149 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:47.150 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:47.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:47.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:47.347 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:47.395 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:47.395 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:47.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:47.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:47.586 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:47.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:47.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:47.824 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj85 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:47.874 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:47.874 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:48.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:48.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:48.071 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:48.121 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:48.121 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:48.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:48.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:48.319 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj92 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:48.369 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:48.369 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:48.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:48.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:48.558 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj93 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:48.604 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:48.604 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:48.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:48.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:48.788 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --object obj95 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:48.833 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:48.833 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:49.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:49.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:49.018 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog trim --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:49.063 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:49.064 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:49.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:49.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:49.162 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:49.209 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:49.210 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:49.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:49.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:45:49.305 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:45:49.354 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:49.355 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:49.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.496+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2023-11-02T18:45:49.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.496+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: waiting on bucket lease 2023-11-02T18:45:49.517 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.508+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: status after acquiring the lock is: init 2023-11-02T18:45:49.535 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.528+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: listing bucket for full sync 2023-11-02T18:45:49.544 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj11 2023-11-02T18:45:49.544 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj11 op=0 op_state=1 2023-11-02T18:45:49.545 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj11[0] 2023-11-02T18:45:49.545 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj111 2023-11-02T18:45:49.545 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj111 op=0 op_state=1 2023-11-02T18:45:49.546 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-11-02T18:45:49.546 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.546 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj111 2023-11-02T18:45:49.546 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj115 2023-11-02T18:45:49.547 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj115 op=0 op_state=1 2023-11-02T18:45:49.547 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj115[0] 2023-11-02T18:45:49.547 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj116 2023-11-02T18:45:49.548 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj116 op=0 op_state=1 2023-11-02T18:45:49.548 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj116[0] 2023-11-02T18:45:49.548 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj124 2023-11-02T18:45:49.548 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj124 op=0 op_state=1 2023-11-02T18:45:49.549 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj124[0] 2023-11-02T18:45:49.549 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj145 2023-11-02T18:45:49.549 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj145 op=0 op_state=1 2023-11-02T18:45:49.549 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj145[0] 2023-11-02T18:45:49.549 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj148 2023-11-02T18:45:49.550 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj148 op=0 op_state=1 2023-11-02T18:45:49.550 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj148[0] 2023-11-02T18:45:49.550 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj155 2023-11-02T18:45:49.550 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj155 op=0 op_state=1 2023-11-02T18:45:49.551 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-11-02T18:45:49.551 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.551 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj155 2023-11-02T18:45:49.551 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj17 2023-11-02T18:45:49.551 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj17 op=0 op_state=1 2023-11-02T18:45:49.552 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj17[0] 2023-11-02T18:45:49.552 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj185 2023-11-02T18:45:49.552 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj185 op=0 op_state=1 2023-11-02T18:45:49.553 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj185[0] 2023-11-02T18:45:49.553 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj19 2023-11-02T18:45:49.553 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj19 op=0 op_state=1 2023-11-02T18:45:49.554 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj19[0] 2023-11-02T18:45:49.554 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj22 2023-11-02T18:45:49.554 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.532+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj22 op=0 op_state=1 2023-11-02T18:45:49.555 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: : injecting data sync error on key=obj22 2023-11-02T18:45:49.555 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.555 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj22 2023-11-02T18:45:49.555 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj23 2023-11-02T18:45:49.555 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj23 op=0 op_state=1 2023-11-02T18:45:49.556 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj23[0] 2023-11-02T18:45:49.556 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj29 2023-11-02T18:45:49.556 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj29 op=0 op_state=1 2023-11-02T18:45:49.556 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-11-02T18:45:49.557 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.557 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj29 2023-11-02T18:45:49.559 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj31 2023-11-02T18:45:49.559 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj31 op=0 op_state=1 2023-11-02T18:45:49.560 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.536+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj31[0] 2023-11-02T18:45:49.560 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj33 2023-11-02T18:45:49.560 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj33 op=0 op_state=1 2023-11-02T18:45:49.561 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj33[0] 2023-11-02T18:45:49.561 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj34 2023-11-02T18:45:49.562 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj34 op=0 op_state=1 2023-11-02T18:45:49.562 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: : injecting data sync error on key=obj34 2023-11-02T18:45:49.562 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.563 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj34 2023-11-02T18:45:49.563 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj37 2023-11-02T18:45:49.564 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj37 op=0 op_state=1 2023-11-02T18:45:49.564 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj37[0] 2023-11-02T18:45:49.564 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj38 2023-11-02T18:45:49.565 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj38 op=0 op_state=1 2023-11-02T18:45:49.565 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj38[0] 2023-11-02T18:45:49.565 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj44 2023-11-02T18:45:49.566 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj44 op=0 op_state=1 2023-11-02T18:45:49.566 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj44[0] 2023-11-02T18:45:49.566 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj46 2023-11-02T18:45:49.567 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj46 op=0 op_state=1 2023-11-02T18:45:49.567 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj46[0] 2023-11-02T18:45:49.567 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.567 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj51 2023-11-02T18:45:49.567 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj51 op=0 op_state=1 2023-11-02T18:45:49.568 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj51[0] 2023-11-02T18:45:49.568 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.568 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj55 2023-11-02T18:45:49.568 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj55 op=0 op_state=1 2023-11-02T18:45:49.569 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj55[0] 2023-11-02T18:45:49.569 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.569 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.569 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj57 2023-11-02T18:45:49.570 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj57 op=0 op_state=1 2023-11-02T18:45:49.570 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj57[0] 2023-11-02T18:45:49.570 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj58 2023-11-02T18:45:49.570 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj58 op=0 op_state=1 2023-11-02T18:45:49.570 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.540+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj58[0] 2023-11-02T18:45:49.571 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.544+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.571 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.544+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj62 2023-11-02T18:45:49.571 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.544+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj62 op=0 op_state=1 2023-11-02T18:45:49.571 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.544+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj62[0] 2023-11-02T18:45:49.578 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: success 2023-11-02T18:45:49.578 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj68 2023-11-02T18:45:49.578 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj68 op=0 op_state=1 2023-11-02T18:45:49.579 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj68[0] 2023-11-02T18:45:49.579 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: success 2023-11-02T18:45:49.579 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: success 2023-11-02T18:45:49.579 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj69 2023-11-02T18:45:49.580 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj69 op=0 op_state=1 2023-11-02T18:45:49.580 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj69[0] 2023-11-02T18:45:49.580 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj74 2023-11-02T18:45:49.580 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj74 op=0 op_state=1 2023-11-02T18:45:49.580 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj74[0] 2023-11-02T18:45:49.581 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: success 2023-11-02T18:45:49.582 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj76 2023-11-02T18:45:49.582 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj76 op=0 op_state=1 2023-11-02T18:45:49.582 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj76[0] 2023-11-02T18:45:49.582 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: success 2023-11-02T18:45:49.583 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: success 2023-11-02T18:45:49.583 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: success 2023-11-02T18:45:49.583 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj85 2023-11-02T18:45:49.583 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj85 op=0 op_state=1 2023-11-02T18:45:49.584 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj85[0] 2023-11-02T18:45:49.584 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj87 2023-11-02T18:45:49.584 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj87 op=0 op_state=1 2023-11-02T18:45:49.584 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: success 2023-11-02T18:45:49.584 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj87[0] 2023-11-02T18:45:49.585 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj92 2023-11-02T18:45:49.585 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj92 op=0 op_state=1 2023-11-02T18:45:49.585 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj92[0] 2023-11-02T18:45:49.585 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj93 2023-11-02T18:45:49.586 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.568+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj93 op=0 op_state=1 2023-11-02T18:45:49.586 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.572+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj93[0] 2023-11-02T18:45:49.586 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.572+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: success 2023-11-02T18:45:49.587 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.572+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: success 2023-11-02T18:45:49.587 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.572+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: success 2023-11-02T18:45:49.587 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.572+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: success 2023-11-02T18:45:49.588 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.572+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj95 2023-11-02T18:45:49.588 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.572+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj95 op=0 op_state=1 2023-11-02T18:45:49.589 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.572+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj95[0] 2023-11-02T18:45:49.590 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.572+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: success 2023-11-02T18:45:49.590 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.576+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: success 2023-11-02T18:45:49.590 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.580+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: success 2023-11-02T18:45:49.591 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.580+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: success 2023-11-02T18:45:49.591 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.580+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: success 2023-11-02T18:45:49.591 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.580+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: success 2023-11-02T18:45:49.591 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.580+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: updating marker oid=bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24 marker=obj11 2023-11-02T18:45:49.592 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.580+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: success 2023-11-02T18:45:49.593 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.580+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: success 2023-11-02T18:45:49.593 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.584+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: success 2023-11-02T18:45:49.593 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.584+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: success 2023-11-02T18:45:49.595 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.588+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: success 2023-11-02T18:45:49.596 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.588+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: success 2023-11-02T18:45:49.596 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.588+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: success 2023-11-02T18:45:49.597 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.588+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: success 2023-11-02T18:45:49.597 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.588+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: success 2023-11-02T18:45:49.598 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.588+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: success 2023-11-02T18:45:49.599 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.592+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: success 2023-11-02T18:45:49.600 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.592+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: success 2023-11-02T18:45:49.600 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.592+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: backing out with sync_status=-5 2023-11-02T18:45:49.600 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.592+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: ERROR: full sync failed. error: -5 2023-11-02T18:45:49.602 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.592+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-11-02T18:45:49.603 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.592+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: waiting on bucket lease 2023-11-02T18:45:49.607 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.600+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: status after acquiring the lock is: full 2023-11-02T18:45:49.621 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.612+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: listing bucket for full sync 2023-11-02T18:45:49.630 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj11 2023-11-02T18:45:49.631 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj11 op=0 op_state=1 2023-11-02T18:45:49.631 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj11[0] 2023-11-02T18:45:49.631 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj111 2023-11-02T18:45:49.631 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj111 op=0 op_state=1 2023-11-02T18:45:49.632 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj111[0] 2023-11-02T18:45:49.632 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj115 2023-11-02T18:45:49.632 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj115 op=0 op_state=1 2023-11-02T18:45:49.633 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj115[0] 2023-11-02T18:45:49.633 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj116 2023-11-02T18:45:49.633 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj116 op=0 op_state=1 2023-11-02T18:45:49.633 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj116[0] 2023-11-02T18:45:49.633 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj124 2023-11-02T18:45:49.634 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj124 op=0 op_state=1 2023-11-02T18:45:49.634 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj124[0] 2023-11-02T18:45:49.634 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj145 2023-11-02T18:45:49.634 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj145 op=0 op_state=1 2023-11-02T18:45:49.635 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj145[0] 2023-11-02T18:45:49.635 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj148 2023-11-02T18:45:49.635 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj148 op=0 op_state=1 2023-11-02T18:45:49.635 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj148[0] 2023-11-02T18:45:49.636 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj155 2023-11-02T18:45:49.636 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj155 op=0 op_state=1 2023-11-02T18:45:49.636 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj155[0] 2023-11-02T18:45:49.636 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj17 2023-11-02T18:45:49.637 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj17 op=0 op_state=1 2023-11-02T18:45:49.637 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj17[0] 2023-11-02T18:45:49.637 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj185 2023-11-02T18:45:49.638 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj185 op=0 op_state=1 2023-11-02T18:45:49.638 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj185[0] 2023-11-02T18:45:49.638 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj19 2023-11-02T18:45:49.639 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj19 op=0 op_state=1 2023-11-02T18:45:49.639 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj19[0] 2023-11-02T18:45:49.639 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj22 2023-11-02T18:45:49.639 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj22 op=0 op_state=1 2023-11-02T18:45:49.640 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj22[0] 2023-11-02T18:45:49.640 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj23 2023-11-02T18:45:49.640 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj23 op=0 op_state=1 2023-11-02T18:45:49.640 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj23[0] 2023-11-02T18:45:49.641 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj29 2023-11-02T18:45:49.641 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj29 op=0 op_state=1 2023-11-02T18:45:49.642 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj29[0] 2023-11-02T18:45:49.642 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj31 2023-11-02T18:45:49.643 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj31 op=0 op_state=1 2023-11-02T18:45:49.643 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj31[0] 2023-11-02T18:45:49.643 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj33 2023-11-02T18:45:49.644 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj33 op=0 op_state=1 2023-11-02T18:45:49.644 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj33[0] 2023-11-02T18:45:49.644 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj34 2023-11-02T18:45:49.644 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj34 op=0 op_state=1 2023-11-02T18:45:49.645 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj34[0] 2023-11-02T18:45:49.645 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj37 2023-11-02T18:45:49.646 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj37 op=0 op_state=1 2023-11-02T18:45:49.646 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj37[0] 2023-11-02T18:45:49.646 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj38 2023-11-02T18:45:49.646 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj38 op=0 op_state=1 2023-11-02T18:45:49.647 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj38[0] 2023-11-02T18:45:49.647 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj44 2023-11-02T18:45:49.647 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj44 op=0 op_state=1 2023-11-02T18:45:49.648 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj44[0] 2023-11-02T18:45:49.648 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj46 2023-11-02T18:45:49.648 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj46 op=0 op_state=1 2023-11-02T18:45:49.649 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.620+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj46[0] 2023-11-02T18:45:49.655 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.624+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: success 2023-11-02T18:45:49.656 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.624+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj51 2023-11-02T18:45:49.656 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.624+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj51 op=0 op_state=1 2023-11-02T18:45:49.657 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.624+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj51[0] 2023-11-02T18:45:49.657 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: success 2023-11-02T18:45:49.658 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: success 2023-11-02T18:45:49.658 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj55 2023-11-02T18:45:49.658 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj55 op=0 op_state=1 2023-11-02T18:45:49.659 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj55[0] 2023-11-02T18:45:49.659 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj57 2023-11-02T18:45:49.659 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj57 op=0 op_state=1 2023-11-02T18:45:49.660 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj57[0] 2023-11-02T18:45:49.660 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: success 2023-11-02T18:45:49.660 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj58 2023-11-02T18:45:49.660 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj58 op=0 op_state=1 2023-11-02T18:45:49.661 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj58[0] 2023-11-02T18:45:49.661 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: success 2023-11-02T18:45:49.661 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj62 2023-11-02T18:45:49.662 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj62 op=0 op_state=1 2023-11-02T18:45:49.662 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj62[0] 2023-11-02T18:45:49.662 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: success 2023-11-02T18:45:49.662 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj68 2023-11-02T18:45:49.663 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj68 op=0 op_state=1 2023-11-02T18:45:49.663 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: : injecting data sync error on key=obj68 2023-11-02T18:45:49.663 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.663 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.628+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj68 2023-11-02T18:45:49.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: success 2023-11-02T18:45:49.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj69 2023-11-02T18:45:49.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj69 op=0 op_state=1 2023-11-02T18:45:49.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj69[0] 2023-11-02T18:45:49.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: success 2023-11-02T18:45:49.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj74 2023-11-02T18:45:49.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj74 op=0 op_state=1 2023-11-02T18:45:49.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj74[0] 2023-11-02T18:45:49.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: success 2023-11-02T18:45:49.666 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj76 2023-11-02T18:45:49.666 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj76 op=0 op_state=1 2023-11-02T18:45:49.666 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: : injecting data sync error on key=obj76 2023-11-02T18:45:49.666 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.667 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj76 2023-11-02T18:45:49.667 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: success 2023-11-02T18:45:49.667 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: success 2023-11-02T18:45:49.667 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj85 2023-11-02T18:45:49.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj85 op=0 op_state=1 2023-11-02T18:45:49.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj85[0] 2023-11-02T18:45:49.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj87 2023-11-02T18:45:49.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj87 op=0 op_state=1 2023-11-02T18:45:49.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-11-02T18:45:49.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj87 2023-11-02T18:45:49.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: success 2023-11-02T18:45:49.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: updating marker oid=bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24 marker=obj11 2023-11-02T18:45:49.670 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: success 2023-11-02T18:45:49.670 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj92 2023-11-02T18:45:49.670 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj92 op=0 op_state=1 2023-11-02T18:45:49.670 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj92[0] 2023-11-02T18:45:49.670 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.671 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj93 2023-11-02T18:45:49.671 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj93 op=0 op_state=1 2023-11-02T18:45:49.671 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.632+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj93[0] 2023-11-02T18:45:49.671 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: success 2023-11-02T18:45:49.672 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj95 2023-11-02T18:45:49.672 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj95 op=0 op_state=1 2023-11-02T18:45:49.672 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj95[0] 2023-11-02T18:45:49.672 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: success 2023-11-02T18:45:49.672 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: success 2023-11-02T18:45:49.673 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: success 2023-11-02T18:45:49.673 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: success 2023-11-02T18:45:49.673 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: success 2023-11-02T18:45:49.673 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.674 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.674 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: success 2023-11-02T18:45:49.674 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: success 2023-11-02T18:45:49.674 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: success 2023-11-02T18:45:49.674 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: success 2023-11-02T18:45:49.675 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.636+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: success 2023-11-02T18:45:49.675 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.640+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: success 2023-11-02T18:45:49.675 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.640+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: success 2023-11-02T18:45:49.675 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.640+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: success 2023-11-02T18:45:49.675 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.640+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: success 2023-11-02T18:45:49.676 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.640+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: success 2023-11-02T18:45:49.676 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.640+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: success 2023-11-02T18:45:49.676 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.644+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: success 2023-11-02T18:45:49.676 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.644+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: success 2023-11-02T18:45:49.677 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.644+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: updating marker oid=bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24 marker=obj62 2023-11-02T18:45:49.677 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.644+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: backing out with sync_status=-5 2023-11-02T18:45:49.677 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.644+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: ERROR: full sync failed. error: -5 2023-11-02T18:45:49.677 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.648+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-11-02T18:45:49.677 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.648+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: waiting on bucket lease 2023-11-02T18:45:49.678 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.652+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: status after acquiring the lock is: full 2023-11-02T18:45:49.678 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.664+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: listing bucket for full sync 2023-11-02T18:45:49.723 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj11 2023-11-02T18:45:49.723 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj11 op=0 op_state=1 2023-11-02T18:45:49.724 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj11[0] 2023-11-02T18:45:49.724 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj111 2023-11-02T18:45:49.725 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj111 op=0 op_state=1 2023-11-02T18:45:49.725 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj111[0] 2023-11-02T18:45:49.725 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj115 2023-11-02T18:45:49.726 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj115 op=0 op_state=1 2023-11-02T18:45:49.726 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj115[0] 2023-11-02T18:45:49.727 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj116 2023-11-02T18:45:49.727 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj116 op=0 op_state=1 2023-11-02T18:45:49.728 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: : injecting data sync error on key=obj116 2023-11-02T18:45:49.728 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.729 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj116 2023-11-02T18:45:49.729 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj124 2023-11-02T18:45:49.729 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj124 op=0 op_state=1 2023-11-02T18:45:49.730 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj124[0] 2023-11-02T18:45:49.730 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj145 2023-11-02T18:45:49.731 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj145 op=0 op_state=1 2023-11-02T18:45:49.731 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj145[0] 2023-11-02T18:45:49.732 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj148 2023-11-02T18:45:49.732 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj148 op=0 op_state=1 2023-11-02T18:45:49.733 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj148[0] 2023-11-02T18:45:49.733 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj155 2023-11-02T18:45:49.733 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj155 op=0 op_state=1 2023-11-02T18:45:49.734 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.712+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-11-02T18:45:49.747 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.747 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj155 2023-11-02T18:45:49.748 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj17 2023-11-02T18:45:49.748 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj17 op=0 op_state=1 2023-11-02T18:45:49.749 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj17[0] 2023-11-02T18:45:49.749 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj185 2023-11-02T18:45:49.749 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj185 op=0 op_state=1 2023-11-02T18:45:49.750 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj185[0] 2023-11-02T18:45:49.750 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj19 2023-11-02T18:45:49.750 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj19 op=0 op_state=1 2023-11-02T18:45:49.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj19[0] 2023-11-02T18:45:49.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj22 2023-11-02T18:45:49.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj22 op=0 op_state=1 2023-11-02T18:45:49.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj22[0] 2023-11-02T18:45:49.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj23 2023-11-02T18:45:49.753 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj23 op=0 op_state=1 2023-11-02T18:45:49.753 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj23[0] 2023-11-02T18:45:49.753 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj29 2023-11-02T18:45:49.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj29 op=0 op_state=1 2023-11-02T18:45:49.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj29[0] 2023-11-02T18:45:49.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj31 2023-11-02T18:45:49.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj31 op=0 op_state=1 2023-11-02T18:45:49.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj31[0] 2023-11-02T18:45:49.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj33 2023-11-02T18:45:49.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj33 op=0 op_state=1 2023-11-02T18:45:49.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj33[0] 2023-11-02T18:45:49.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj34 2023-11-02T18:45:49.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj34 op=0 op_state=1 2023-11-02T18:45:49.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj34[0] 2023-11-02T18:45:49.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj37 2023-11-02T18:45:49.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj37 op=0 op_state=1 2023-11-02T18:45:49.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj37[0] 2023-11-02T18:45:49.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj38 2023-11-02T18:45:49.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj38 op=0 op_state=1 2023-11-02T18:45:49.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj38[0] 2023-11-02T18:45:49.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj44 2023-11-02T18:45:49.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj44 op=0 op_state=1 2023-11-02T18:45:49.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj44[0] 2023-11-02T18:45:49.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj46 2023-11-02T18:45:49.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj46 op=0 op_state=1 2023-11-02T18:45:49.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.716+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj46[0] 2023-11-02T18:45:49.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.720+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: success 2023-11-02T18:45:49.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.720+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj51 2023-11-02T18:45:49.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.720+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj51 op=0 op_state=1 2023-11-02T18:45:49.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.720+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj51[0] 2023-11-02T18:45:49.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: success 2023-11-02T18:45:49.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj55 2023-11-02T18:45:49.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj55 op=0 op_state=1 2023-11-02T18:45:49.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj55[0] 2023-11-02T18:45:49.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: success 2023-11-02T18:45:49.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj57 2023-11-02T18:45:49.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj57 op=0 op_state=1 2023-11-02T18:45:49.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj57[0] 2023-11-02T18:45:49.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: success 2023-11-02T18:45:49.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj58 2023-11-02T18:45:49.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj58 op=0 op_state=1 2023-11-02T18:45:49.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj58[0] 2023-11-02T18:45:49.766 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: success 2023-11-02T18:45:49.766 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: success 2023-11-02T18:45:49.766 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj62 2023-11-02T18:45:49.766 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj62 op=0 op_state=1 2023-11-02T18:45:49.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: : injecting data sync error on key=obj62 2023-11-02T18:45:49.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj62 2023-11-02T18:45:49.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj68 2023-11-02T18:45:49.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj68 op=0 op_state=1 2023-11-02T18:45:49.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.724+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj68[0] 2023-11-02T18:45:49.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj69 2023-11-02T18:45:49.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj69 op=0 op_state=1 2023-11-02T18:45:49.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj69[0] 2023-11-02T18:45:49.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: success 2023-11-02T18:45:49.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj74 2023-11-02T18:45:49.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj74 op=0 op_state=1 2023-11-02T18:45:49.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj74[0] 2023-11-02T18:45:49.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.771 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: success 2023-11-02T18:45:49.771 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj76 2023-11-02T18:45:49.771 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj76 op=0 op_state=1 2023-11-02T18:45:49.771 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj76[0] 2023-11-02T18:45:49.771 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: success 2023-11-02T18:45:49.772 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: success 2023-11-02T18:45:49.772 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: success 2023-11-02T18:45:49.772 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: success 2023-11-02T18:45:49.772 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj85 2023-11-02T18:45:49.773 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj85 op=0 op_state=1 2023-11-02T18:45:49.773 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: success 2023-11-02T18:45:49.773 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: success 2023-11-02T18:45:49.773 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: success 2023-11-02T18:45:49.773 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: success 2023-11-02T18:45:49.774 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: success 2023-11-02T18:45:49.774 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: success 2023-11-02T18:45:49.774 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-11-02T18:45:49.774 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj85 2023-11-02T18:45:49.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: success 2023-11-02T18:45:49.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj87 2023-11-02T18:45:49.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj87 op=0 op_state=1 2023-11-02T18:45:49.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-11-02T18:45:49.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj87 2023-11-02T18:45:49.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj92 2023-11-02T18:45:49.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj92 op=0 op_state=1 2023-11-02T18:45:49.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: success 2023-11-02T18:45:49.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj92[0] 2023-11-02T18:45:49.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: success 2023-11-02T18:45:49.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: success 2023-11-02T18:45:49.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj93 2023-11-02T18:45:49.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj93 op=0 op_state=1 2023-11-02T18:45:49.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.728+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: : injecting data sync error on key=obj93 2023-11-02T18:45:49.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj93 2023-11-02T18:45:49.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj95 2023-11-02T18:45:49.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj95 op=0 op_state=1 2023-11-02T18:45:49.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj95[0] 2023-11-02T18:45:49.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: success 2023-11-02T18:45:49.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.780 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: success 2023-11-02T18:45:49.780 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: success 2023-11-02T18:45:49.780 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: success 2023-11-02T18:45:49.780 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: success 2023-11-02T18:45:49.780 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.781 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.781 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: success 2023-11-02T18:45:49.781 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: success 2023-11-02T18:45:49.781 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.782 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.732+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: updating marker oid=bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24 marker=obj115 2023-11-02T18:45:49.782 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.736+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: backing out with sync_status=-5 2023-11-02T18:45:49.782 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.736+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: ERROR: full sync failed. error: -5 2023-11-02T18:45:49.782 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.736+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-11-02T18:45:49.783 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.736+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: waiting on bucket lease 2023-11-02T18:45:49.798 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.740+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: status after acquiring the lock is: full 2023-11-02T18:45:49.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.756+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: listing bucket for full sync 2023-11-02T18:45:49.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj11 2023-11-02T18:45:49.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj11 op=0 op_state=1 2023-11-02T18:45:49.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj11[0] 2023-11-02T18:45:49.801 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj111 2023-11-02T18:45:49.801 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj111 op=0 op_state=1 2023-11-02T18:45:49.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-11-02T18:45:49.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj111 2023-11-02T18:45:49.803 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj115 2023-11-02T18:45:49.803 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj115 op=0 op_state=1 2023-11-02T18:45:49.803 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj115[0] 2023-11-02T18:45:49.803 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj116 2023-11-02T18:45:49.804 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj116 op=0 op_state=1 2023-11-02T18:45:49.804 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj116[0] 2023-11-02T18:45:49.804 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj124 2023-11-02T18:45:49.804 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj124 op=0 op_state=1 2023-11-02T18:45:49.805 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: : injecting data sync error on key=obj124 2023-11-02T18:45:49.805 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.805 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj124 2023-11-02T18:45:49.805 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj145 2023-11-02T18:45:49.805 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj145 op=0 op_state=1 2023-11-02T18:45:49.806 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj145[0] 2023-11-02T18:45:49.806 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj148 2023-11-02T18:45:49.806 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj148 op=0 op_state=1 2023-11-02T18:45:49.806 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: : injecting data sync error on key=obj148 2023-11-02T18:45:49.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj148 2023-11-02T18:45:49.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj155 2023-11-02T18:45:49.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj155 op=0 op_state=1 2023-11-02T18:45:49.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj155[0] 2023-11-02T18:45:49.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj17 2023-11-02T18:45:49.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj17 op=0 op_state=1 2023-11-02T18:45:49.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: : injecting data sync error on key=obj17 2023-11-02T18:45:49.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj17 2023-11-02T18:45:49.809 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj185 2023-11-02T18:45:49.809 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj185 op=0 op_state=1 2023-11-02T18:45:49.809 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj185[0] 2023-11-02T18:45:49.809 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj19 2023-11-02T18:45:49.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj19 op=0 op_state=1 2023-11-02T18:45:49.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj19[0] 2023-11-02T18:45:49.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj22 2023-11-02T18:45:49.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj22 op=0 op_state=1 2023-11-02T18:45:49.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj22[0] 2023-11-02T18:45:49.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj23 2023-11-02T18:45:49.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj23 op=0 op_state=1 2023-11-02T18:45:49.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj23[0] 2023-11-02T18:45:49.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj29 2023-11-02T18:45:49.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj29 op=0 op_state=1 2023-11-02T18:45:49.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.764+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj29[0] 2023-11-02T18:45:49.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.768+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj31 2023-11-02T18:45:49.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.768+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj31 op=0 op_state=1 2023-11-02T18:45:49.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.768+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj31[0] 2023-11-02T18:45:49.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.768+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj33 2023-11-02T18:45:49.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.768+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj33 op=0 op_state=1 2023-11-02T18:45:49.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.768+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj33[0] 2023-11-02T18:45:49.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.768+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj34 2023-11-02T18:45:49.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.768+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj34 op=0 op_state=1 2023-11-02T18:45:49.814 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.768+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj34[0] 2023-11-02T18:45:49.814 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.768+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj37 2023-11-02T18:45:49.814 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj37 op=0 op_state=1 2023-11-02T18:45:49.814 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj37[0] 2023-11-02T18:45:49.814 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj38 2023-11-02T18:45:49.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj38 op=0 op_state=1 2023-11-02T18:45:49.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj38[0] 2023-11-02T18:45:49.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj44 2023-11-02T18:45:49.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj44 op=0 op_state=1 2023-11-02T18:45:49.816 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: success 2023-11-02T18:45:49.816 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: success 2023-11-02T18:45:49.816 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj44[0] 2023-11-02T18:45:49.816 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: success 2023-11-02T18:45:49.816 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: success 2023-11-02T18:45:49.817 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj46 2023-11-02T18:45:49.817 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj46 op=0 op_state=1 2023-11-02T18:45:49.817 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj46[0] 2023-11-02T18:45:49.817 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: success 2023-11-02T18:45:49.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: success 2023-11-02T18:45:49.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: success 2023-11-02T18:45:49.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: success 2023-11-02T18:45:49.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.819 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.819 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj51 2023-11-02T18:45:49.819 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj51 op=0 op_state=1 2023-11-02T18:45:49.819 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj51[0] 2023-11-02T18:45:49.819 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj55 2023-11-02T18:45:49.820 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj55 op=0 op_state=1 2023-11-02T18:45:49.820 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: success 2023-11-02T18:45:49.820 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: success 2023-11-02T18:45:49.820 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: : injecting data sync error on key=obj55 2023-11-02T18:45:49.821 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: success 2023-11-02T18:45:49.821 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.821 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj55 2023-11-02T18:45:49.821 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj57 2023-11-02T18:45:49.821 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj57 op=0 op_state=1 2023-11-02T18:45:49.822 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: : injecting data sync error on key=obj57 2023-11-02T18:45:49.822 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.822 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj57 2023-11-02T18:45:49.822 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj58 2023-11-02T18:45:49.822 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj58 op=0 op_state=1 2023-11-02T18:45:49.823 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.772+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj58[0] 2023-11-02T18:45:49.823 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj62 2023-11-02T18:45:49.823 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj62 op=0 op_state=1 2023-11-02T18:45:49.823 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj62[0] 2023-11-02T18:45:49.823 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: success 2023-11-02T18:45:49.825 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj68 2023-11-02T18:45:49.825 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj68 op=0 op_state=1 2023-11-02T18:45:49.825 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj68[0] 2023-11-02T18:45:49.826 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj69 2023-11-02T18:45:49.826 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj69 op=0 op_state=1 2023-11-02T18:45:49.826 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj69[0] 2023-11-02T18:45:49.826 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: success 2023-11-02T18:45:49.827 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: success 2023-11-02T18:45:49.827 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.827 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj74 2023-11-02T18:45:49.827 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj74 op=0 op_state=1 2023-11-02T18:45:49.828 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj74[0] 2023-11-02T18:45:49.828 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj76 2023-11-02T18:45:49.828 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj76 op=0 op_state=1 2023-11-02T18:45:49.828 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj76[0] 2023-11-02T18:45:49.828 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: success 2023-11-02T18:45:49.829 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: success 2023-11-02T18:45:49.829 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: success 2023-11-02T18:45:49.829 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj85 2023-11-02T18:45:49.829 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj85 op=0 op_state=1 2023-11-02T18:45:49.830 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: success 2023-11-02T18:45:49.830 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj85[0] 2023-11-02T18:45:49.830 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj87 2023-11-02T18:45:49.830 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj87 op=0 op_state=1 2023-11-02T18:45:49.830 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj87[0] 2023-11-02T18:45:49.831 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj92 2023-11-02T18:45:49.831 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj92 op=0 op_state=1 2023-11-02T18:45:49.831 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj92[0] 2023-11-02T18:45:49.831 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj93 2023-11-02T18:45:49.831 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj93 op=0 op_state=1 2023-11-02T18:45:49.832 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj93[0] 2023-11-02T18:45:49.832 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj95 2023-11-02T18:45:49.832 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj95 op=0 op_state=1 2023-11-02T18:45:49.832 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj95[0] 2023-11-02T18:45:49.833 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.833 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.833 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.776+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: success 2023-11-02T18:45:49.833 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: success 2023-11-02T18:45:49.833 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: success 2023-11-02T18:45:49.834 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: success 2023-11-02T18:45:49.834 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: success 2023-11-02T18:45:49.834 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: success 2023-11-02T18:45:49.834 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: success 2023-11-02T18:45:49.834 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: success 2023-11-02T18:45:49.835 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: success 2023-11-02T18:45:49.835 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: success 2023-11-02T18:45:49.835 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: success 2023-11-02T18:45:49.835 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: updating marker oid=bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24 marker=obj11 2023-11-02T18:45:49.836 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: backing out with sync_status=-5 2023-11-02T18:45:49.836 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.780+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: ERROR: full sync failed. error: -5 2023-11-02T18:45:49.836 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.784+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-11-02T18:45:49.836 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.784+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: waiting on bucket lease 2023-11-02T18:45:49.844 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.836+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: status after acquiring the lock is: full 2023-11-02T18:45:49.864 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.856+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: listing bucket for full sync 2023-11-02T18:45:49.875 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj11 2023-11-02T18:45:49.875 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj11 op=0 op_state=1 2023-11-02T18:45:49.876 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj11[0] 2023-11-02T18:45:49.876 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj111 2023-11-02T18:45:49.876 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj111 op=0 op_state=1 2023-11-02T18:45:49.876 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj111[0] 2023-11-02T18:45:49.877 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj115 2023-11-02T18:45:49.877 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj115 op=0 op_state=1 2023-11-02T18:45:49.877 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj115[0] 2023-11-02T18:45:49.877 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj116 2023-11-02T18:45:49.877 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj116 op=0 op_state=1 2023-11-02T18:45:49.878 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj116[0] 2023-11-02T18:45:49.878 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj124 2023-11-02T18:45:49.878 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj124 op=0 op_state=1 2023-11-02T18:45:49.878 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj124[0] 2023-11-02T18:45:49.879 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj145 2023-11-02T18:45:49.879 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj145 op=0 op_state=1 2023-11-02T18:45:49.879 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: : injecting data sync error on key=obj145 2023-11-02T18:45:49.879 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.879 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj145 2023-11-02T18:45:49.880 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj148 2023-11-02T18:45:49.880 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj148 op=0 op_state=1 2023-11-02T18:45:49.880 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj148[0] 2023-11-02T18:45:49.880 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj155 2023-11-02T18:45:49.881 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj155 op=0 op_state=1 2023-11-02T18:45:49.881 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj155[0] 2023-11-02T18:45:49.881 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj17 2023-11-02T18:45:49.881 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj17 op=0 op_state=1 2023-11-02T18:45:49.881 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj17[0] 2023-11-02T18:45:49.882 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj185 2023-11-02T18:45:49.882 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj185 op=0 op_state=1 2023-11-02T18:45:49.883 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj185[0] 2023-11-02T18:45:49.883 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj19 2023-11-02T18:45:49.883 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj19 op=0 op_state=1 2023-11-02T18:45:49.883 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj19[0] 2023-11-02T18:45:49.883 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj22 2023-11-02T18:45:49.884 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj22 op=0 op_state=1 2023-11-02T18:45:49.884 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj22[0] 2023-11-02T18:45:49.884 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj23 2023-11-02T18:45:49.884 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj23 op=0 op_state=1 2023-11-02T18:45:49.885 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj23[0] 2023-11-02T18:45:49.886 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj29 2023-11-02T18:45:49.886 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj29 op=0 op_state=1 2023-11-02T18:45:49.886 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-11-02T18:45:49.886 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.887 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj29 2023-11-02T18:45:49.887 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj31 2023-11-02T18:45:49.887 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj31 op=0 op_state=1 2023-11-02T18:45:49.888 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj31[0] 2023-11-02T18:45:49.888 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj33 2023-11-02T18:45:49.888 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj33 op=0 op_state=1 2023-11-02T18:45:49.888 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj33[0] 2023-11-02T18:45:49.889 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj34 2023-11-02T18:45:49.889 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj34 op=0 op_state=1 2023-11-02T18:45:49.889 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj34[0] 2023-11-02T18:45:49.889 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj37 2023-11-02T18:45:49.890 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj37 op=0 op_state=1 2023-11-02T18:45:49.890 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj37[0] 2023-11-02T18:45:49.890 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj38 2023-11-02T18:45:49.890 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj38 op=0 op_state=1 2023-11-02T18:45:49.891 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj38[0] 2023-11-02T18:45:49.891 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj44 2023-11-02T18:45:49.891 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj44 op=0 op_state=1 2023-11-02T18:45:49.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj44[0] 2023-11-02T18:45:49.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj46 2023-11-02T18:45:49.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj46 op=0 op_state=1 2023-11-02T18:45:49.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.864+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj46[0] 2023-11-02T18:45:49.900 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: success 2023-11-02T18:45:49.901 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: success 2023-11-02T18:45:49.901 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: success 2023-11-02T18:45:49.901 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: success 2023-11-02T18:45:49.902 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj51 2023-11-02T18:45:49.902 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj51 op=0 op_state=1 2023-11-02T18:45:49.902 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj51[0] 2023-11-02T18:45:49.903 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj55 2023-11-02T18:45:49.903 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj55 op=0 op_state=1 2023-11-02T18:45:49.903 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj55[0] 2023-11-02T18:45:49.903 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj57 2023-11-02T18:45:49.903 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj57 op=0 op_state=1 2023-11-02T18:45:49.904 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj57[0] 2023-11-02T18:45:49.904 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj58 2023-11-02T18:45:49.904 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj58 op=0 op_state=1 2023-11-02T18:45:49.904 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj58[0] 2023-11-02T18:45:49.905 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: success 2023-11-02T18:45:49.905 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.905 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj62 2023-11-02T18:45:49.905 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj62 op=0 op_state=1 2023-11-02T18:45:49.905 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj62[0] 2023-11-02T18:45:49.906 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj68 2023-11-02T18:45:49.906 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj68 op=0 op_state=1 2023-11-02T18:45:49.906 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.868+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj68[0] 2023-11-02T18:45:49.906 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.907 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: success 2023-11-02T18:45:49.907 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: success 2023-11-02T18:45:49.907 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: success 2023-11-02T18:45:49.907 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: success 2023-11-02T18:45:49.907 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: success 2023-11-02T18:45:49.908 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj69 2023-11-02T18:45:49.908 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj69 op=0 op_state=1 2023-11-02T18:45:49.908 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj69[0] 2023-11-02T18:45:49.908 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: success 2023-11-02T18:45:49.909 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: success 2023-11-02T18:45:49.909 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj74 2023-11-02T18:45:49.909 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj74 op=0 op_state=1 2023-11-02T18:45:49.909 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj74[0] 2023-11-02T18:45:49.909 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj76 2023-11-02T18:45:49.910 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj76 op=0 op_state=1 2023-11-02T18:45:49.910 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj76[0] 2023-11-02T18:45:49.910 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj85 2023-11-02T18:45:49.910 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj85 op=0 op_state=1 2023-11-02T18:45:49.910 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-11-02T18:45:49.911 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-11-02T18:45:49.911 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 0 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: ERROR: failed to sync object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj85 2023-11-02T18:45:49.911 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj87 2023-11-02T18:45:49.911 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj87 op=0 op_state=1 2023-11-02T18:45:49.912 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj87[0] 2023-11-02T18:45:49.912 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj92 2023-11-02T18:45:49.912 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj92 op=0 op_state=1 2023-11-02T18:45:49.913 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj92[0] 2023-11-02T18:45:49.913 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj93 2023-11-02T18:45:49.913 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj93 op=0 op_state=1 2023-11-02T18:45:49.913 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj93[0] 2023-11-02T18:45:49.914 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj95 2023-11-02T18:45:49.914 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj95 op=0 op_state=1 2023-11-02T18:45:49.914 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj95[0] 2023-11-02T18:45:49.914 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: success 2023-11-02T18:45:49.914 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: success 2023-11-02T18:45:49.915 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: success 2023-11-02T18:45:49.915 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: success 2023-11-02T18:45:49.915 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: success 2023-11-02T18:45:49.915 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.872+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: success 2023-11-02T18:45:49.916 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: success 2023-11-02T18:45:49.916 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: updating marker oid=bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24 marker=obj124 2023-11-02T18:45:49.916 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: success 2023-11-02T18:45:49.916 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: a sync operation returned error 2023-11-02T18:45:49.916 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: success 2023-11-02T18:45:49.917 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: success 2023-11-02T18:45:49.917 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: success 2023-11-02T18:45:49.917 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: success 2023-11-02T18:45:49.917 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: success 2023-11-02T18:45:49.918 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: success 2023-11-02T18:45:49.918 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: success 2023-11-02T18:45:49.918 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: success 2023-11-02T18:45:49.918 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: success 2023-11-02T18:45:49.918 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: success 2023-11-02T18:45:49.919 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: success 2023-11-02T18:45:49.919 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: success 2023-11-02T18:45:49.919 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: backing out with sync_status=-5 2023-11-02T18:45:49.919 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.876+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: ERROR: full sync failed. error: -5 2023-11-02T18:45:49.919 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.880+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-11-02T18:45:49.920 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.880+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: waiting on bucket lease 2023-11-02T18:45:49.920 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.884+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: status after acquiring the lock is: full 2023-11-02T18:45:49.920 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.900+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: listing bucket for full sync 2023-11-02T18:45:49.960 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj11 2023-11-02T18:45:49.960 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj11 op=0 op_state=1 2023-11-02T18:45:49.960 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj11[0] 2023-11-02T18:45:49.961 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj111 2023-11-02T18:45:49.961 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj111 op=0 op_state=1 2023-11-02T18:45:49.961 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj111[0] 2023-11-02T18:45:49.961 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj115 2023-11-02T18:45:49.962 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj115 op=0 op_state=1 2023-11-02T18:45:49.962 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj115[0] 2023-11-02T18:45:49.962 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj116 2023-11-02T18:45:49.962 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj116 op=0 op_state=1 2023-11-02T18:45:49.962 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj116[0] 2023-11-02T18:45:49.963 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj124 2023-11-02T18:45:49.963 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj124 op=0 op_state=1 2023-11-02T18:45:49.963 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj124[0] 2023-11-02T18:45:49.963 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj145 2023-11-02T18:45:49.964 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj145 op=0 op_state=1 2023-11-02T18:45:49.964 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj145[0] 2023-11-02T18:45:49.964 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj148 2023-11-02T18:45:49.964 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj148 op=0 op_state=1 2023-11-02T18:45:49.964 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj148[0] 2023-11-02T18:45:49.965 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj155 2023-11-02T18:45:49.965 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj155 op=0 op_state=1 2023-11-02T18:45:49.965 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.948+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj155[0] 2023-11-02T18:45:49.965 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj17 2023-11-02T18:45:49.966 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj17 op=0 op_state=1 2023-11-02T18:45:49.966 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj17[0] 2023-11-02T18:45:49.966 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj185 2023-11-02T18:45:49.966 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj185 op=0 op_state=1 2023-11-02T18:45:49.967 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj185[0] 2023-11-02T18:45:49.967 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj19 2023-11-02T18:45:49.967 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj19 op=0 op_state=1 2023-11-02T18:45:49.967 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj19[0] 2023-11-02T18:45:49.967 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj22 2023-11-02T18:45:49.968 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj22 op=0 op_state=1 2023-11-02T18:45:49.968 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj22[0] 2023-11-02T18:45:49.968 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj23 2023-11-02T18:45:49.968 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj23 op=0 op_state=1 2023-11-02T18:45:49.972 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj23[0] 2023-11-02T18:45:49.972 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj29 2023-11-02T18:45:49.972 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj29 op=0 op_state=1 2023-11-02T18:45:49.972 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj29[0] 2023-11-02T18:45:49.973 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj31 2023-11-02T18:45:49.973 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj31 op=0 op_state=1 2023-11-02T18:45:49.973 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj31[0] 2023-11-02T18:45:49.973 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj33 2023-11-02T18:45:49.973 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj33 op=0 op_state=1 2023-11-02T18:45:49.974 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.952+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj33[0] 2023-11-02T18:45:49.974 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj34 2023-11-02T18:45:49.974 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj34 op=0 op_state=1 2023-11-02T18:45:49.974 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj34[0] 2023-11-02T18:45:49.975 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj37 2023-11-02T18:45:49.975 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj37 op=0 op_state=1 2023-11-02T18:45:49.975 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj37[0] 2023-11-02T18:45:49.975 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj38 2023-11-02T18:45:49.975 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj38 op=0 op_state=1 2023-11-02T18:45:49.976 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj11]: success 2023-11-02T18:45:49.976 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj38[0] 2023-11-02T18:45:49.976 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj44 2023-11-02T18:45:49.976 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj44 op=0 op_state=1 2023-11-02T18:45:49.977 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj115]: success 2023-11-02T18:45:49.977 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.956+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj44[0] 2023-11-02T18:45:49.977 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj46 2023-11-02T18:45:49.977 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj46 op=0 op_state=1 2023-11-02T18:45:49.977 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj46[0] 2023-11-02T18:45:49.978 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj51 2023-11-02T18:45:49.978 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj51 op=0 op_state=1 2023-11-02T18:45:49.978 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj51[0] 2023-11-02T18:45:49.978 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj55 2023-11-02T18:45:49.979 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj55 op=0 op_state=1 2023-11-02T18:45:49.979 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj55[0] 2023-11-02T18:45:49.979 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj111]: success 2023-11-02T18:45:49.979 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj57 2023-11-02T18:45:49.980 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj57 op=0 op_state=1 2023-11-02T18:45:49.980 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj57[0] 2023-11-02T18:45:49.980 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj124]: success 2023-11-02T18:45:49.980 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj58 2023-11-02T18:45:49.980 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj58 op=0 op_state=1 2023-11-02T18:45:49.981 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj58[0] 2023-11-02T18:45:49.981 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj116]: success 2023-11-02T18:45:49.981 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj145]: success 2023-11-02T18:45:49.981 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj62 2023-11-02T18:45:49.982 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj62 op=0 op_state=1 2023-11-02T18:45:49.982 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj62[0] 2023-11-02T18:45:49.982 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj148]: success 2023-11-02T18:45:49.982 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj68 2023-11-02T18:45:49.983 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj68 op=0 op_state=1 2023-11-02T18:45:49.983 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj68[0] 2023-11-02T18:45:49.983 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj69 2023-11-02T18:45:49.983 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj69 op=0 op_state=1 2023-11-02T18:45:49.983 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj69[0] 2023-11-02T18:45:49.984 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj155]: success 2023-11-02T18:45:49.984 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj74 2023-11-02T18:45:49.984 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj74 op=0 op_state=1 2023-11-02T18:45:49.984 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.960+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj74[0] 2023-11-02T18:45:49.987 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj22]: success 2023-11-02T18:45:49.987 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj23]: success 2023-11-02T18:45:49.987 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj29]: success 2023-11-02T18:45:49.987 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj17]: success 2023-11-02T18:45:49.988 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: updating marker oid=bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24 marker=obj17 2023-11-02T18:45:49.988 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj46]: success 2023-11-02T18:45:49.988 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj185]: success 2023-11-02T18:45:49.988 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj55]: success 2023-11-02T18:45:49.989 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj44]: success 2023-11-02T18:45:49.989 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj34]: success 2023-11-02T18:45:49.989 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj76 2023-11-02T18:45:49.989 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj76 op=0 op_state=1 2023-11-02T18:45:49.989 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj76[0] 2023-11-02T18:45:49.990 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj85 2023-11-02T18:45:49.990 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj85 op=0 op_state=1 2023-11-02T18:45:49.990 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj85[0] 2023-11-02T18:45:49.990 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj87 2023-11-02T18:45:49.991 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj87 op=0 op_state=1 2023-11-02T18:45:49.991 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj87[0] 2023-11-02T18:45:49.991 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj19]: success 2023-11-02T18:45:49.991 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj92 2023-11-02T18:45:49.991 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj92 op=0 op_state=1 2023-11-02T18:45:49.992 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj51]: success 2023-11-02T18:45:49.992 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj92[0] 2023-11-02T18:45:49.992 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj93 2023-11-02T18:45:49.992 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj93 op=0 op_state=1 2023-11-02T18:45:49.993 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj93[0] 2023-11-02T18:45:49.993 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: [full sync] syncing object: ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0/obj95 2023-11-02T18:45:49.993 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b=bucket sync single entry (source_zone=d61b2bea-59a8-4454-8491-ac3afbd04937) b= log_entry=obj95 op=0 op_state=1 2023-11-02T18:45:49.993 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: bucket sync: sync obj: d61b2bea-59a8-4454-8491-ac3afbd04937/:ibtljn-59[d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24])/obj95[0] 2023-11-02T18:45:49.993 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj31]: success 2023-11-02T18:45:49.994 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj37]: success 2023-11-02T18:45:49.994 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj68]: success 2023-11-02T18:45:49.994 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.964+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj33]: success 2023-11-02T18:45:49.994 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: updating marker oid=bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24 marker=obj37 2023-11-02T18:45:49.995 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj62]: success 2023-11-02T18:45:49.995 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj69]: success 2023-11-02T18:45:49.995 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj57]: success 2023-11-02T18:45:49.995 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj38]: success 2023-11-02T18:45:49.995 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj58]: success 2023-11-02T18:45:49.996 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj92]: success 2023-11-02T18:45:49.996 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj76]: success 2023-11-02T18:45:49.996 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj87]: success 2023-11-02T18:45:49.996 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj74]: success 2023-11-02T18:45:49.996 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj85]: success 2023-11-02T18:45:49.997 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: updating marker oid=bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24 marker=obj92 2023-11-02T18:45:49.997 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.968+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj95]: success 2023-11-02T18:45:49.997 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.972+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:entry[obj93]: success 2023-11-02T18:45:49.997 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.972+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: updating marker oid=bucket.full-sync-status.d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24 marker=obj95 2023-11-02T18:45:49.998 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.972+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: set bucket state=incremental 2023-11-02T18:45:49.998 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.976+0000 7f014aaa1f40 5 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:full_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: bucket status objv={r=_iMKgozrI6RUp6puY3mCXvNj:151,w=:0} 2023-11-02T18:45:49.998 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.976+0000 7f014aaa1f40 10 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: WARNING: requested sync of past generation 0 < 6, returning success 2023-11-02T18:45:50.003 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.003 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:2]: sync status for source bucket shard: 2 2023-11-02T18:45:50.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:2]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:2]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:3]: sync status for source bucket shard: 2 2023-11-02T18:45:50.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:3]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:3]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: sync status for source bucket shard: 2 2023-11-02T18:45:50.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:1]: sync status for source bucket shard: 2 2023-11-02T18:45:50.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:4]: sync status for source bucket shard: 2 2023-11-02T18:45:50.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:0]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:1]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:1]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:4]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:4]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:6]: sync status for source bucket shard: 2 2023-11-02T18:45:50.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:6]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:6]: listing bilog for incremental sync; position=00000000002.66.6 2023-11-02T18:45:50.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:5]: sync status for source bucket shard: 2 2023-11-02T18:45:50.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:7]: sync status for source bucket shard: 2 2023-11-02T18:45:50.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:5]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:5]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.980+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:7]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:7]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:8]: sync status for source bucket shard: 2 2023-11-02T18:45:50.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:9]: sync status for source bucket shard: 2 2023-11-02T18:45:50.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:8]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:8]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:9]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:9]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:11]: sync status for source bucket shard: 2 2023-11-02T18:45:50.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:10]: sync status for source bucket shard: 2 2023-11-02T18:45:50.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:12]: sync status for source bucket shard: 2 2023-11-02T18:45:50.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:14]: sync status for source bucket shard: 2 2023-11-02T18:45:50.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:13]: sync status for source bucket shard: 2 2023-11-02T18:45:50.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:17]: sync status for source bucket shard: 2 2023-11-02T18:45:50.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:11]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:11]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:10]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:10]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:12]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:12]: listing bilog for incremental sync; position=00000000002.70.6 2023-11-02T18:45:50.015 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:14]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:14]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.015 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:13]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:13]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.015 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.984+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:17]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:17]: listing bilog for incremental sync; position=00000000002.51.6 2023-11-02T18:45:50.015 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:22]: sync status for source bucket shard: 2 2023-11-02T18:45:50.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:19]: sync status for source bucket shard: 2 2023-11-02T18:45:50.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:20]: sync status for source bucket shard: 2 2023-11-02T18:45:50.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:21]: sync status for source bucket shard: 2 2023-11-02T18:45:50.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:22]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:22]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:19]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:19]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.018 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:20]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:20]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.018 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:21]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:21]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.018 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.018 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.018 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.019 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.019 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:23]: sync status for source bucket shard: 2 2023-11-02T18:45:50.019 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:24]: sync status for source bucket shard: 2 2023-11-02T18:45:50.019 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:26]: sync status for source bucket shard: 2 2023-11-02T18:45:50.020 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:25]: sync status for source bucket shard: 2 2023-11-02T18:45:50.020 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:23]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:23]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.020 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:24]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:24]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.020 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:26]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:26]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.020 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:25]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:25]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.988+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:27]: sync status for source bucket shard: 2 2023-11-02T18:45:50.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:28]: sync status for source bucket shard: 2 2023-11-02T18:45:50.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:27]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:27]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:28]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:28]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:29]: sync status for source bucket shard: 2 2023-11-02T18:45:50.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:29]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:29]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:30]: sync status for source bucket shard: 2 2023-11-02T18:45:50.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:30]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:30]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.024 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:31]: sync status for source bucket shard: 2 2023-11-02T18:45:50.024 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:32]: sync status for source bucket shard: 2 2023-11-02T18:45:50.024 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:34]: sync status for source bucket shard: 2 2023-11-02T18:45:50.024 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:31]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:31]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.025 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:32]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:32]: listing bilog for incremental sync; position=00000000001.63.6 2023-11-02T18:45:50.025 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:34]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:34]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.025 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:33]: sync status for source bucket shard: 2 2023-11-02T18:45:50.025 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:33]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:33]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.025 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-11-02T18:45:50.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:15]: sync status for source bucket shard: 2 2023-11-02T18:45:50.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:35]: sync status for source bucket shard: 2 2023-11-02T18:45:50.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:16]: sync status for source bucket shard: 2 2023-11-02T18:45:50.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:15]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:15]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:35]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:35]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.992+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:16]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:16]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.996+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:18]: sync status for source bucket shard: 2 2023-11-02T18:45:50.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.996+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:18]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:18]: listing bilog for incremental sync; position= 2023-11-02T18:45:50.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.996+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:36]: sync status for source bucket shard: 2 2023-11-02T18:45:50.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:49.996+0000 7f014aaa1f40 20 RGW-SYNC:bucket[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24<-ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:36]:inc_sync[ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24:36]: listing bilog for incremental sync; position=00000000001.134.6 2023-11-02T18:45:50.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:45:50.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:45:50.085 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:45:50.174 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:50.174 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:50.239 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:45:50.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.240 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj11", 2023-11-02T18:45:50.240 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.240 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.240 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.241 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:45:50.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.241 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.241 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.242 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.242 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.242 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.242 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.543369Z", 2023-11-02T18:45:50.243 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.243 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.244 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.244 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.244 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.244 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.244 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.245 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.245 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.267498149748224624", 2023-11-02T18:45:50.245 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.246 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.246 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.247 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj111", 2023-11-02T18:45:50.247 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.247 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.248 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.248 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:50.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.248 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.249 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.249 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.249 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.249 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:50.249 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.749440Z", 2023-11-02T18:45:50.250 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.250 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.250 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.250 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.251 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.251 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.251 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.251 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.252 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.17658591362641308571", 2023-11-02T18:45:50.252 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.252 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.252 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.253 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj115", 2023-11-02T18:45:50.253 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.253 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.254 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.254 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1 2023-11-02T18:45:50.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.254 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.254 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.255 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.255 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.255 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.255 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.589046Z", 2023-11-02T18:45:50.256 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.256 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.256 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.256 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.256 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.1492307949825692993", 2023-11-02T18:45:50.258 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.258 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.258 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.259 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj116", 2023-11-02T18:45:50.259 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.259 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.259 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 11 2023-11-02T18:45:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.795424Z", 2023-11-02T18:45:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.262 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.262 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.262 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.262 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.5737788846417353975", 2023-11-02T18:45:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj124", 2023-11-02T18:45:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:45:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.406701Z", 2023-11-02T18:45:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.269 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.269 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.269 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.269 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.17288191201750331773", 2023-11-02T18:45:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj145", 2023-11-02T18:45:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.858962Z", 2023-11-02T18:45:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.14445864556775094694", 2023-11-02T18:45:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.278 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.278 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj148", 2023-11-02T18:45:50.280 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.280 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 15 2023-11-02T18:45:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.282 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.282 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.284 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.778084Z", 2023-11-02T18:45:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.287 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.287 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.287 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.289 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.7441408978948278031", 2023-11-02T18:45:50.289 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.289 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.289 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.291 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj155", 2023-11-02T18:45:50.291 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.291 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.292 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.292 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:50.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.293 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.293 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.293 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.293 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.469524Z", 2023-11-02T18:45:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.9235364218407815423", 2023-11-02T18:45:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj17", 2023-11-02T18:45:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:45:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.738565Z", 2023-11-02T18:45:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.8793425021319299611", 2023-11-02T18:45:50.303 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.303 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.303 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj185", 2023-11-02T18:45:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1 2023-11-02T18:45:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.842657Z", 2023-11-02T18:45:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.1957212353549951274", 2023-11-02T18:45:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj19", 2023-11-02T18:45:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 12 2023-11-02T18:45:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.312 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.312 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.312 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.312 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.313 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.364071Z", 2023-11-02T18:45:50.313 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.313 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.313 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.313 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.315 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.8633747526341682681", 2023-11-02T18:45:50.315 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.315 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.315 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj22", 2023-11-02T18:45:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:45:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.572127Z", 2023-11-02T18:45:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.16397673769119955506", 2023-11-02T18:45:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj23", 2023-11-02T18:45:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.410749Z", 2023-11-02T18:45:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.1128588594124404849", 2023-11-02T18:45:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj29", 2023-11-02T18:45:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.673282Z", 2023-11-02T18:45:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.10661315782927032190", 2023-11-02T18:45:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj31", 2023-11-02T18:45:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 17 2023-11-02T18:45:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.279072Z", 2023-11-02T18:45:50.338 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.338 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.338 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.338 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.338 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.16246215260891579758", 2023-11-02T18:45:50.340 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.340 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.340 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj33", 2023-11-02T18:45:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.342 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:45:50.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.342 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.342 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.342 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.595835Z", 2023-11-02T18:45:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.7521198215814142581", 2023-11-02T18:45:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj34", 2023-11-02T18:45:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 14 2023-11-02T18:45:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.349 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.349 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.768510Z", 2023-11-02T18:45:50.349 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.349 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.350 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.350 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.350 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.350 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.351 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.351 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.351 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.15235762245106781250", 2023-11-02T18:45:50.351 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.352 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.352 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.353 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj37", 2023-11-02T18:45:50.353 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.353 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.652984Z", 2023-11-02T18:45:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.3660687014400205916", 2023-11-02T18:45:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj38", 2023-11-02T18:45:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.393766Z", 2023-11-02T18:45:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.13819950011634489519", 2023-11-02T18:45:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.365 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.365 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj44", 2023-11-02T18:45:50.366 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.366 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.366 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.366 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.654344Z", 2023-11-02T18:45:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.7732564979390903898", 2023-11-02T18:45:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj46", 2023-11-02T18:45:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.439748Z", 2023-11-02T18:45:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.375 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.375 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.375 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.375 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.13928983179520969098", 2023-11-02T18:45:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj51", 2023-11-02T18:45:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.828934Z", 2023-11-02T18:45:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.2198971628027179628", 2023-11-02T18:45:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj55", 2023-11-02T18:45:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:45:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.386 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.681652Z", 2023-11-02T18:45:50.386 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.386 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.386 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.386 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.387 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.387 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.387 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.387 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.388 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.8849667493394195702", 2023-11-02T18:45:50.388 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.388 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.388 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.389 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj57", 2023-11-02T18:45:50.389 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.390 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.390 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.390 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.391 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.391 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.391 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.391 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.391 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.392 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.455583Z", 2023-11-02T18:45:50.392 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.392 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.392 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.4726800609094620665", 2023-11-02T18:45:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.395 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.395 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj58", 2023-11-02T18:45:50.395 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.396 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.396 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.396 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:50.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.397 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.397 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.397 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.397 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.397 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.398 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.702322Z", 2023-11-02T18:45:50.398 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.398 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.398 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.399 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.399 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.399 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.399 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.399 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.400 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.6870487155108909268", 2023-11-02T18:45:50.400 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.400 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.401 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.401 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj62", 2023-11-02T18:45:50.401 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.402 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.402 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.402 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 2 2023-11-02T18:45:50.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.403 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.403 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.403 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.403 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.404 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.404 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.308477Z", 2023-11-02T18:45:50.404 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.404 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.404 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.405 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.405 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.405 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.405 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.406 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.406 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.2497142801627526702", 2023-11-02T18:45:50.406 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.406 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.407 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.407 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj68", 2023-11-02T18:45:50.407 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.408 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.408 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.408 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:45:50.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.408 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.409 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.409 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.409 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.409 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.410 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.858441Z", 2023-11-02T18:45:50.410 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.410 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.410 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.411 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.411 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.411 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.411 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.411 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.412 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.5885819596027449859", 2023-11-02T18:45:50.412 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.412 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.413 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.413 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj69", 2023-11-02T18:45:50.413 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.414 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.414 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.414 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 13 2023-11-02T18:45:50.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.414 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.415 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.415 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.415 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.415 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.416 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.501361Z", 2023-11-02T18:45:50.416 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.416 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.416 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.417 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.417 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.417 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.417 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.417 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.418 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.7644687235755838799", 2023-11-02T18:45:50.418 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.418 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.419 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.419 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj74", 2023-11-02T18:45:50.419 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.420 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.420 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.420 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10 2023-11-02T18:45:50.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.420 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.421 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.421 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.421 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.421 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:50.421 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.681498Z", 2023-11-02T18:45:50.422 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.422 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.422 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.422 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.422 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.1968427238723840861", 2023-11-02T18:45:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj76", 2023-11-02T18:45:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:45:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.481539Z", 2023-11-02T18:45:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.1194404475041815087", 2023-11-02T18:45:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj85", 2023-11-02T18:45:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.919466Z", 2023-11-02T18:45:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.9385469260180602985", 2023-11-02T18:45:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj87", 2023-11-02T18:45:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 49 2023-11-02T18:45:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.765417Z", 2023-11-02T18:45:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.17824947695698182401", 2023-11-02T18:45:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj92", 2023-11-02T18:45:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 9 2023-11-02T18:45:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.528009Z", 2023-11-02T18:45:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.18107189600002405075", 2023-11-02T18:45:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj93", 2023-11-02T18:45:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.452 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.452 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:50.452 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.377248Z", 2023-11-02T18:45:50.452 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.452 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.453 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.453 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.453 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.453 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.454 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.454 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.454 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.11952590015501268574", 2023-11-02T18:45:50.454 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.455 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.455 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.455 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj95", 2023-11-02T18:45:50.456 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.456 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.456 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 9, 2023-11-02T18:45:50.456 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:50.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.457 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.457 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.457 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.458 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.458 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.458 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.549728Z", 2023-11-02T18:45:50.458 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.458 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.459 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.459 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.459 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.459 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.460 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.460 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.460 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.12970850949941062751", 2023-11-02T18:45:50.460 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.461 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.461 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.461 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:45:50.461 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:45:50.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:45:50.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:45:50.463 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:45:50.514 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:50.514 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:50.576 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:45:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj11", 2023-11-02T18:45:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.578 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 8 2023-11-02T18:45:50.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.578 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.579 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.579 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.579 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.579 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.580 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.543369Z", 2023-11-02T18:45:50.580 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.580 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.580 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.580 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.581 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.581 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.581 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.581 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.582 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_BA8aYYpWMNYgnhjjhP3pVlq2-_NkBh8", 2023-11-02T18:45:50.582 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.582 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.582 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.584 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj111", 2023-11-02T18:45:50.584 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.584 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.584 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.585 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:50.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.585 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.585 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.585 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.586 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.586 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.586 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.749440Z", 2023-11-02T18:45:50.586 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.587 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.587 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.587 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.587 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.587 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.588 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.589 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.589 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_43gRZIS6BDLHlTGQr3Txkfl4Crg_Hya", 2023-11-02T18:45:50.589 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.589 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.590 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.590 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj115", 2023-11-02T18:45:50.591 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.591 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.591 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.591 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:45:50.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.592 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.592 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.592 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.592 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.592 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:50.593 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.589046Z", 2023-11-02T18:45:50.593 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.594 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.594 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.594 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.595 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.595 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.595 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.595 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.596 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_ywA3HQX9u876JG8PB_ed2Nj3u_rvB71", 2023-11-02T18:45:50.597 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.597 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.597 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.598 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj116", 2023-11-02T18:45:50.598 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.598 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.599 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.599 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 8 2023-11-02T18:45:50.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.599 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.599 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.600 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.600 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.600 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.600 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.795424Z", 2023-11-02T18:45:50.601 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.601 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.601 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.601 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.601 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.602 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.602 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.602 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_Fu88JDsvDxaBFFE_MY5zY_Xl787-XDJ", 2023-11-02T18:45:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj124", 2023-11-02T18:45:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.406701Z", 2023-11-02T18:45:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_NQQz5fcBaulMUChoE9dT9LYRDnGa32t", 2023-11-02T18:45:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj145", 2023-11-02T18:45:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:45:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.858962Z", 2023-11-02T18:45:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_nZOMT-xkTf6GyJd9D6Bt9Uj9MpLHMpH", 2023-11-02T18:45:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj148", 2023-11-02T18:45:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 8 2023-11-02T18:45:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.778084Z", 2023-11-02T18:45:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_OydkpStIu8NQNZbnyVoKKS8hbopSbaq", 2023-11-02T18:45:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj155", 2023-11-02T18:45:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 13 2023-11-02T18:45:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.469524Z", 2023-11-02T18:45:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_xsTCZMCizrjoBcxcxvMPz14VFLaS86U", 2023-11-02T18:45:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj17", 2023-11-02T18:45:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.738565Z", 2023-11-02T18:45:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_PtyOaGKEUEGEyanVdYjgDjE50YNyl0V", 2023-11-02T18:45:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj185", 2023-11-02T18:45:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 3 2023-11-02T18:45:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.842657Z", 2023-11-02T18:45:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_ZVvlcCyFnguWyX0c1rSWcPzxMrWHIKx", 2023-11-02T18:45:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj19", 2023-11-02T18:45:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 8 2023-11-02T18:45:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.364071Z", 2023-11-02T18:45:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_vaY_jceJ5HxnRnFbGhXgX8ub_ZnJOYG", 2023-11-02T18:45:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj22", 2023-11-02T18:45:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 12 2023-11-02T18:45:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.572127Z", 2023-11-02T18:45:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_VMa_l4pbgWbvMmCcYnlhsnSfG8xSMtW", 2023-11-02T18:45:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj23", 2023-11-02T18:45:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:45:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.410749Z", 2023-11-02T18:45:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_rUebh45RA_g-pl6HvWE7ZIK0HRp0ujd", 2023-11-02T18:45:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj29", 2023-11-02T18:45:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 8 2023-11-02T18:45:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.673282Z", 2023-11-02T18:45:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_EtAW74IcFSd4dSiCgiWqIqUyzB2eXQL", 2023-11-02T18:45:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj31", 2023-11-02T18:45:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 8 2023-11-02T18:45:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.279072Z", 2023-11-02T18:45:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_z2SHYfz7QOhbzmijZo2tkvxQq-oe-Al", 2023-11-02T18:45:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj33", 2023-11-02T18:45:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.595835Z", 2023-11-02T18:45:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_QZM4GHUYQOmuGbz1sZ_AGu11HYNj1a0", 2023-11-02T18:45:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj34", 2023-11-02T18:45:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:45:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.768510Z", 2023-11-02T18:45:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_3NRaWPT31__w_4dggS2rmElvrvy-7Y-", 2023-11-02T18:45:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj37", 2023-11-02T18:45:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.652984Z", 2023-11-02T18:45:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_5MVgGy28gslqQccmDPIok9NBiRCXMpQ", 2023-11-02T18:45:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj38", 2023-11-02T18:45:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:45:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.393766Z", 2023-11-02T18:45:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_sEHHsI84KRmT9-zuhKXvk10yVM3A81j", 2023-11-02T18:45:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj44", 2023-11-02T18:45:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 6 2023-11-02T18:45:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.654344Z", 2023-11-02T18:45:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_LgaFFDB0tSKET3e5sMiIBOKuz6jynhF", 2023-11-02T18:45:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj46", 2023-11-02T18:45:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10 2023-11-02T18:45:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.439748Z", 2023-11-02T18:45:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_8aUUxoniWTwKmzNlzvsjUt_hxUE7v7M", 2023-11-02T18:45:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj51", 2023-11-02T18:45:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.712 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.712 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.712 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.712 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.713 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.713 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.713 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.713 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.714 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.828934Z", 2023-11-02T18:45:50.714 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.714 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.714 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.714 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.715 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.715 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.715 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.715 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.716 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_HlVf0ymnRMf4a6NdDuIgVngwvE3400O", 2023-11-02T18:45:50.716 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.716 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.716 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.717 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj55", 2023-11-02T18:45:50.717 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.717 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.718 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.718 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 5 2023-11-02T18:45:50.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.718 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.719 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.719 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.719 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.719 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.719 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:14.681652Z", 2023-11-02T18:45:50.720 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.720 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.720 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.720 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.720 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.721 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.721 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.721 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.722 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_tcZ3r9kySGsdbOjNkbtylWx0ZS0K03o", 2023-11-02T18:45:50.722 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.722 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.722 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.723 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj57", 2023-11-02T18:45:50.723 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.723 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.723 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.724 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.724 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.724 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.725 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.725 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.725 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.725 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.455583Z", 2023-11-02T18:45:50.725 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.726 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.726 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.726 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.726 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.727 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.727 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.727 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.728 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_sAYyCNlMsyurIoXYelM5M65uPbIubAN", 2023-11-02T18:45:50.728 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.728 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.728 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.729 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj58", 2023-11-02T18:45:50.729 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.729 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.730 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.730 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 16 2023-11-02T18:45:50.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.730 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.730 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.731 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.731 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.731 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.731 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.702322Z", 2023-11-02T18:45:50.731 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.732 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.732 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.732 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.732 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.733 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.733 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.733 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.733 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_P_f_-qNIAOj55Jlnx9k9kZ4OAnbBJEo", 2023-11-02T18:45:50.734 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.734 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.734 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.735 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj62", 2023-11-02T18:45:50.735 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.735 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.735 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.736 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 4 2023-11-02T18:45:50.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.736 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.736 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.737 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.737 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.737 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.737 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.308477Z", 2023-11-02T18:45:50.737 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.738 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.738 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.738 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.738 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.739 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.739 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.739 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.740 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_tQGQE78zJCerjAZ08RI1ST9g3u4Wr18", 2023-11-02T18:45:50.740 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.740 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.740 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.741 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj68", 2023-11-02T18:45:50.741 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.741 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.742 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.742 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 10 2023-11-02T18:45:50.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.742 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.742 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.743 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.743 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.743 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.743 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.858441Z", 2023-11-02T18:45:50.743 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.744 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.744 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.744 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.744 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.744 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.745 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.745 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.745 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_qOdiSN3d3HkOgftjxkqFqmWpdMS3lsN", 2023-11-02T18:45:50.746 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.746 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.746 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.747 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj69", 2023-11-02T18:45:50.747 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.747 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.747 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.747 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 9 2023-11-02T18:45:50.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.748 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.748 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.748 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.749 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.749 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.749 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.501361Z", 2023-11-02T18:45:50.749 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.749 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.750 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.750 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.750 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.750 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.750 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.751 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.751 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_8dZ1J2xBuvG6gD0D15Ptu5UIXz7ytfv", 2023-11-02T18:45:50.751 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.752 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.752 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.752 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj74", 2023-11-02T18:45:50.753 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.753 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.753 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.753 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 15 2023-11-02T18:45:50.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.754 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.754 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.754 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.754 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.755 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.755 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:44:13.681498Z", 2023-11-02T18:45:50.755 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.755 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.756 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.756 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.756 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.756 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.757 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.757 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.757 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_SZVZfSBUFWyA7ASFU39HGYXMqYU8oDs", 2023-11-02T18:45:50.757 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.758 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.758 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.759 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj76", 2023-11-02T18:45:50.759 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.759 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.759 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.759 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:45:50.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.760 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.760 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.760 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.760 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.761 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 5, 2023-11-02T18:45:50.761 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.481539Z", 2023-11-02T18:45:50.761 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.761 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.761 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.762 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.762 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.762 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.762 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.763 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.763 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_V2kiqCzDDP7cW3fBjXgbDz5-1qjYzQs", 2023-11-02T18:45:50.763 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.763 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.764 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.764 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj85", 2023-11-02T18:45:50.764 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.765 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.765 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.765 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 11 2023-11-02T18:45:50.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.765 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.766 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.766 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.766 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.766 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.766 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:32:46.919466Z", 2023-11-02T18:45:50.767 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.767 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.767 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.767 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.768 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.768 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.768 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.768 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.769 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_y1oC6UuZqVcnMXsOHPXFh81ZMvq2Kve", 2023-11-02T18:45:50.769 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.769 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.769 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.770 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj87", 2023-11-02T18:45:50.770 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.770 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.771 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.771 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 44 2023-11-02T18:45:50.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.771 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.771 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.773 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.773 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.773 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 16, 2023-11-02T18:45:50.773 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:40:31.765417Z", 2023-11-02T18:45:50.774 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.774 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.774 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.774 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.774 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.775 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.775 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.775 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.776 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_ezWrBUqvOfd6h4mwk2WY5xuK1rDI3-f", 2023-11-02T18:45:50.776 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.776 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.776 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.777 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj92", 2023-11-02T18:45:50.777 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.777 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.778 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.778 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 16 2023-11-02T18:45:50.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.778 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.778 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.779 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.779 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.779 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.779 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:39:10.528009Z", 2023-11-02T18:45:50.780 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.780 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.780 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.780 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.780 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.781 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.781 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.781 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.781 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_VN4U7sA67pwL0wkeyYOKBqH26bdrYpz", 2023-11-02T18:45:50.782 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.782 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.782 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.783 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj93", 2023-11-02T18:45:50.783 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.783 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.783 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.784 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 11 2023-11-02T18:45:50.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.784 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.784 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.784 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.785 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.785 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 6, 2023-11-02T18:45:50.785 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:42:34.377248Z", 2023-11-02T18:45:50.785 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.785 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.786 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.786 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.786 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.786 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.787 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.787 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.787 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_ex-IMbsMvq5S3YGStmxYi6anIxgWVPU", 2023-11-02T18:45:50.787 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.788 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.788 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:45:50.788 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "obj95", 2023-11-02T18:45:50.789 INFO:teuthology.orchestra.run.smithi017.stdout: "instance": "", 2023-11-02T18:45:50.789 INFO:teuthology.orchestra.run.smithi017.stdout: "ver": { 2023-11-02T18:45:50.789 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 13, 2023-11-02T18:45:50.789 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 7 2023-11-02T18:45:50.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.790 INFO:teuthology.orchestra.run.smithi017.stdout: "locator": "", 2023-11-02T18:45:50.790 INFO:teuthology.orchestra.run.smithi017.stdout: "exists": true, 2023-11-02T18:45:50.790 INFO:teuthology.orchestra.run.smithi017.stdout: "meta": { 2023-11-02T18:45:50.790 INFO:teuthology.orchestra.run.smithi017.stdout: "category": 1, 2023-11-02T18:45:50.791 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2023-11-02T18:45:50.791 INFO:teuthology.orchestra.run.smithi017.stdout: "mtime": "2023-11-02T18:34:50.549728Z", 2023-11-02T18:45:50.791 INFO:teuthology.orchestra.run.smithi017.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-11-02T18:45:50.791 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_class": "", 2023-11-02T18:45:50.791 INFO:teuthology.orchestra.run.smithi017.stdout: "owner": "rgw-multisite-test-user", 2023-11-02T18:45:50.792 INFO:teuthology.orchestra.run.smithi017.stdout: "owner_display_name": "Multisite Test User", 2023-11-02T18:45:50.792 INFO:teuthology.orchestra.run.smithi017.stdout: "content_type": "application/octet-stream", 2023-11-02T18:45:50.792 INFO:teuthology.orchestra.run.smithi017.stdout: "accounted_size": 3, 2023-11-02T18:45:50.792 INFO:teuthology.orchestra.run.smithi017.stdout: "user_data": "", 2023-11-02T18:45:50.792 INFO:teuthology.orchestra.run.smithi017.stdout: "appendable": false 2023-11-02T18:45:50.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:45:50.793 INFO:teuthology.orchestra.run.smithi017.stdout: "tag": "_LzUGJ8KqzhghmBRAIJTsMBCewhnY9yc", 2023-11-02T18:45:50.793 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 0, 2023-11-02T18:45:50.793 INFO:teuthology.orchestra.run.smithi017.stdout: "pending_map": [], 2023-11-02T18:45:50.793 INFO:teuthology.orchestra.run.smithi017.stdout: "versioned_epoch": 0 2023-11-02T18:45:50.794 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:45:50.794 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:45:50.795 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-11-02T18:45:50.796 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-11-02T18:45:50.800 INFO:tasks.rgw.c1.client.1:Started 2023-11-02T18:45:50.801 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:50.821 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T18:45:50.821 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T18:45:50.821 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-11-02T18:45:50.821 INFO:teuthology.orchestra.run.smithi017.stderr:curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8001: Connection refused 2023-11-02T18:45:50.822 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-11-02T18:45:51.823 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:51.840 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T18:45:51.840 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T18:45:51.842 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2023-11-02T18:45:51.843 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2023-11-02T18:45:51.844 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-59 2023-11-02T18:45:51.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:45:51.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:45:51.845 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:45:51.897 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:51.897 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:51.963 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:51.952+0000 7f232fb5bf40 1 bucket sync caught up with source: 2023-11-02T18:45:51.963 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:45:51.964 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.66.6, , , , , , 00000000002.70.6, , , , , 00000000002.51.6, , , , , , , , , , , , , , , 00000000001.63.6, , , , 00000000001.134.6] 2023-11-02T18:45:51.964 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:51.952+0000 7f232fb5bf40 0 bucket checkpoint complete 2023-11-02T18:45:51.986 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-59 2023-11-02T18:45:51.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:45:51.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:45:51.988 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:45:52.039 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:45:52.039 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:45:52.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:52.096+0000 7f610d2d6f40 1 bucket sync caught up with source: 2023-11-02T18:45:52.104 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000009.108.6, 00000000006.249.6, 00000000006.155.6, 00000000015.337.6, 00000000012.310.6, 00000000003.331.6, 00000000003.333.6, 00000000012.116.6, 00000000021.321.6, 00000000018.372.6] 2023-11-02T18:45:52.104 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000009.108.6, 00000000006.249.6, 00000000006.155.6, 00000000015.337.6, 00000000012.310.6, 00000000003.331.6, 00000000003.333.6, 00000000012.116.6, 00000000021.321.6, 00000000018.372.6] 2023-11-02T18:45:52.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:45:52.096+0000 7f610d2d6f40 0 bucket checkpoint complete 2023-11-02T18:45:52.134 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:45:52.134 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:52.135 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:52.135 DEBUG:boto:Method: HEAD 2023-11-02T18:45:52.136 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:45:52.136 DEBUG:boto:Data: 2023-11-02T18:45:52.137 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.137 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.137 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.138 DEBUG:boto:Params: {} 2023-11-02T18:45:52.138 DEBUG:boto:Token: None 2023-11-02T18:45:52.139 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/ 2023-11-02T18:45:52.140 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ynvR13ds5R/Anelad3bTDM+zoIA= 2023-11-02T18:45:52.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ynvR13ds5R/Anelad3bTDM+zoIA=', 'Content-Length': '0'} 2023-11-02T18:45:52.141 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-11-02T18:45:52.142 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:45:52.365 DEBUG:boto:Token: None 2023-11-02T18:45:52.366 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/ 2023-11-02T18:45:52.366 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ynvR13ds5R/Anelad3bTDM+zoIA= 2023-11-02T18:45:52.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ynvR13ds5R/Anelad3bTDM+zoIA=', 'Content-Length': '0'} 2023-11-02T18:45:52.377 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009dd14fd7ba6158b-006543ee60-1e2a-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.377 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:52.378 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:52.379 DEBUG:boto:Method: HEAD 2023-11-02T18:45:52.380 DEBUG:boto:Path: /ibtljn-59/ 2023-11-02T18:45:52.380 DEBUG:boto:Data: 2023-11-02T18:45:52.381 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.381 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.381 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.387 DEBUG:boto:Params: {} 2023-11-02T18:45:52.388 DEBUG:boto:Token: None 2023-11-02T18:45:52.388 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/ 2023-11-02T18:45:52.389 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ynvR13ds5R/Anelad3bTDM+zoIA= 2023-11-02T18:45:52.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ynvR13ds5R/Anelad3bTDM+zoIA=', 'Content-Length': '0'} 2023-11-02T18:45:52.397 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c564f99c3aee560e-006543ee60-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.398 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:45:52.398 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:52.399 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:52.399 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:52.400 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:52.400 DEBUG:boto:Method: GET 2023-11-02T18:45:52.401 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:52.401 DEBUG:boto:Data: 2023-11-02T18:45:52.402 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.402 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.405 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.406 DEBUG:boto:Params: {} 2023-11-02T18:45:52.408 DEBUG:boto:Token: None 2023-11-02T18:45:52.408 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/?versions 2023-11-02T18:45:52.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQvxACS9tU9GT6g2oCaM3kEqtGA= 2023-11-02T18:45:52.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQvxACS9tU9GT6g2oCaM3kEqtGA=', 'Content-Length': '0'} 2023-11-02T18:45:52.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd9fb6e70ac6251d-006543ee60-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.424 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:52.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:45:52.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-11-02T18:45:52.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T18:45:52.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-11-02T18:45:52.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-11-02T18:45:52.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-11-02T18:45:52.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-11-02T18:45:52.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-11-02T18:45:52.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:45:52.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-11-02T18:45:52.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:45:52.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:45:52.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T18:45:52.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-11-02T18:45:52.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-11-02T18:45:52.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:45:52.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:45:52.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-11-02T18:45:52.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:45:52.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:45:52.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T18:45:52.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:45:52.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:45:52.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:45:52.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-11-02T18:45:52.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-11-02T18:45:52.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:45:52.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T18:45:52.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-11-02T18:45:52.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:45:52.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:45:52.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-11-02T18:45:52.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T18:45:52.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-11-02T18:45:52.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:45:52.444 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:45:52.445 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:52.447 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:52.447 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:52.448 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:52.448 DEBUG:boto:Method: GET 2023-11-02T18:45:52.449 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:52.449 DEBUG:boto:Data: 2023-11-02T18:45:52.450 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.450 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.451 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.451 DEBUG:boto:Params: {} 2023-11-02T18:45:52.452 DEBUG:boto:Token: None 2023-11-02T18:45:52.464 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/?versions 2023-11-02T18:45:52.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQvxACS9tU9GT6g2oCaM3kEqtGA= 2023-11-02T18:45:52.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQvxACS9tU9GT6g2oCaM3kEqtGA=', 'Content-Length': '0'} 2023-11-02T18:45:52.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7f3a12d17ea0c01-006543ee60-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.477 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:52.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:45:52.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-11-02T18:45:52.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T18:45:52.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-11-02T18:45:52.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-11-02T18:45:52.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-11-02T18:45:52.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-11-02T18:45:52.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-11-02T18:45:52.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:45:52.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-11-02T18:45:52.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:45:52.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:45:52.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T18:45:52.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-11-02T18:45:52.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-11-02T18:45:52.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:45:52.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:45:52.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-11-02T18:45:52.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:45:52.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:45:52.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T18:45:52.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:45:52.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:45:52.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:45:52.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-11-02T18:45:52.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-11-02T18:45:52.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:45:52.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T18:45:52.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-11-02T18:45:52.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:45:52.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:45:52.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-11-02T18:45:52.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T18:45:52.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-11-02T18:45:52.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:45:52.493 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:52.493 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:52.494 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:52.494 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:52.495 DEBUG:boto:Method: GET 2023-11-02T18:45:52.495 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:52.496 DEBUG:boto:Data: 2023-11-02T18:45:52.496 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.497 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.497 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.497 DEBUG:boto:Params: {} 2023-11-02T18:45:52.498 DEBUG:boto:Token: None 2023-11-02T18:45:52.499 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/?versions 2023-11-02T18:45:52.499 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQvxACS9tU9GT6g2oCaM3kEqtGA= 2023-11-02T18:45:52.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQvxACS9tU9GT6g2oCaM3kEqtGA=', 'Content-Length': '0'} 2023-11-02T18:45:52.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e5bfd14d91a4e74-006543ee60-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.508 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:52.511 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:52.511 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:52.512 DEBUG:boto:path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:52.512 DEBUG:boto:auth_path=/ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:52.513 DEBUG:boto:Method: GET 2023-11-02T18:45:52.513 DEBUG:boto:Path: /ibtljn-59/?versions&max-keys=999 2023-11-02T18:45:52.513 DEBUG:boto:Data: 2023-11-02T18:45:52.514 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.514 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.515 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.515 DEBUG:boto:Params: {} 2023-11-02T18:45:52.516 DEBUG:boto:Token: None 2023-11-02T18:45:52.516 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/?versions 2023-11-02T18:45:52.517 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BQvxACS9tU9GT6g2oCaM3kEqtGA= 2023-11-02T18:45:52.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BQvxACS9tU9GT6g2oCaM3kEqtGA=', 'Content-Length': '0'} 2023-11-02T18:45:52.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000070d91d27b8b273b-006543ee60-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.527 DEBUG:boto:b'ibtljn-59999falseobj11nulltrue2023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:52.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:45:52.531 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:52.532 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:52.532 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.533 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.533 DEBUG:boto:Method: GET 2023-11-02T18:45:52.534 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.534 DEBUG:boto:Data: 2023-11-02T18:45:52.538 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.540 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.540 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.540 DEBUG:boto:Params: {} 2023-11-02T18:45:52.541 DEBUG:boto:Token: None 2023-11-02T18:45:52.542 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.542 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ecmVAUWB8wNQnzeHNRX8z3V3Wd8= 2023-11-02T18:45:52.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ecmVAUWB8wNQnzeHNRX8z3V3Wd8=', 'Content-Length': '0'} 2023-11-02T18:45:52.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001da0d446dd1e1190-006543ee60-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.550 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:52.551 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:52.551 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.551 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.552 DEBUG:boto:Method: GET 2023-11-02T18:45:52.552 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.553 DEBUG:boto:Data: 2023-11-02T18:45:52.553 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.554 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.554 DEBUG:boto:Params: {} 2023-11-02T18:45:52.555 DEBUG:boto:Token: None 2023-11-02T18:45:52.555 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ecmVAUWB8wNQnzeHNRX8z3V3Wd8= 2023-11-02T18:45:52.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ecmVAUWB8wNQnzeHNRX8z3V3Wd8=', 'Content-Length': '0'} 2023-11-02T18:45:52.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088745bf37cb28665-006543ee60-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.602 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:52.603 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:52.603 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.604 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.604 DEBUG:boto:Method: HEAD 2023-11-02T18:45:52.605 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.605 DEBUG:boto:Data: 2023-11-02T18:45:52.605 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.606 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.607 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.607 DEBUG:boto:Params: {} 2023-11-02T18:45:52.608 DEBUG:boto:Token: None 2023-11-02T18:45:52.609 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.610 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T4aTdrKhcjOSQQimMFMUWmO5B8U= 2023-11-02T18:45:52.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T4aTdrKhcjOSQQimMFMUWmO5B8U=', 'Content-Length': '0'} 2023-11-02T18:45:52.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000552ee5e11f31f402-006543ee60-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.617 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:52.618 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:52.619 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.619 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.619 DEBUG:boto:Method: HEAD 2023-11-02T18:45:52.620 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.620 DEBUG:boto:Data: 2023-11-02T18:45:52.621 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.621 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.622 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.622 DEBUG:boto:Params: {} 2023-11-02T18:45:52.622 DEBUG:boto:Token: None 2023-11-02T18:45:52.623 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.623 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T4aTdrKhcjOSQQimMFMUWmO5B8U= 2023-11-02T18:45:52.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T4aTdrKhcjOSQQimMFMUWmO5B8U=', 'Content-Length': '0'} 2023-11-02T18:45:52.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039275ec8aae2ff06-006543ee60-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:45:52.632 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:52.633 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:52.634 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.634 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.635 DEBUG:boto:Method: GET 2023-11-02T18:45:52.635 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.636 DEBUG:boto:Data: 2023-11-02T18:45:52.636 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.636 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.637 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.640 DEBUG:boto:Params: {} 2023-11-02T18:45:52.640 DEBUG:boto:Token: None 2023-11-02T18:45:52.641 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ecmVAUWB8wNQnzeHNRX8z3V3Wd8= 2023-11-02T18:45:52.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ecmVAUWB8wNQnzeHNRX8z3V3Wd8=', 'Content-Length': '0'} 2023-11-02T18:45:52.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3c9c38117603b7b-006543ee60-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.686 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:52.687 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:52.687 DEBUG:boto:path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.688 DEBUG:boto:auth_path=/ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.688 DEBUG:boto:Method: GET 2023-11-02T18:45:52.689 DEBUG:boto:Path: /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.689 DEBUG:boto:Data: 2023-11-02T18:45:52.690 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.691 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.691 DEBUG:boto:Params: {} 2023-11-02T18:45:52.692 DEBUG:boto:Token: None 2023-11-02T18:45:52.692 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj11?versionId=null 2023-11-02T18:45:52.693 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ecmVAUWB8wNQnzeHNRX8z3V3Wd8= 2023-11-02T18:45:52.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ecmVAUWB8wNQnzeHNRX8z3V3Wd8=', 'Content-Length': '0'} 2023-11-02T18:45:52.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d7bc47f3b562660-006543ee60-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.697 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:52.697 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:52.698 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:52.698 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:52.699 DEBUG:boto:Method: GET 2023-11-02T18:45:52.699 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:45:52.699 DEBUG:boto:Data: 2023-11-02T18:45:52.700 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.701 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.701 DEBUG:boto:Params: {} 2023-11-02T18:45:52.702 DEBUG:boto:Token: None 2023-11-02T18:45:52.702 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/ 2023-11-02T18:45:52.703 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u+Va51RfSw7TdyxKazhULuGRKGE= 2023-11-02T18:45:52.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u+Va51RfSw7TdyxKazhULuGRKGE=', 'Content-Length': '0'} 2023-11-02T18:45:52.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a9fc80b55900f8f-006543ee60-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.709 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:52.710 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:52.711 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:52.711 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:52.711 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:52.712 DEBUG:boto:Method: GET 2023-11-02T18:45:52.712 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:45:52.713 DEBUG:boto:Data: 2023-11-02T18:45:52.713 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.714 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.714 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.714 DEBUG:boto:Params: {} 2023-11-02T18:45:52.715 DEBUG:boto:Token: None 2023-11-02T18:45:52.715 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/ 2023-11-02T18:45:52.716 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u+Va51RfSw7TdyxKazhULuGRKGE= 2023-11-02T18:45:52.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u+Va51RfSw7TdyxKazhULuGRKGE=', 'Content-Length': '0'} 2023-11-02T18:45:52.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020c8d97ac15931d5-006543ee60-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.727 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:52.728 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:52.728 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:52.729 DEBUG:boto:Method: HEAD 2023-11-02T18:45:52.729 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:45:52.730 DEBUG:boto:Data: 2023-11-02T18:45:52.730 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.730 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.731 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.731 DEBUG:boto:Params: {} 2023-11-02T18:45:52.732 DEBUG:boto:Token: None 2023-11-02T18:45:52.732 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj11 2023-11-02T18:45:52.733 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TM2o9dojQ3d7qwyUhWIIlZ43dzY= 2023-11-02T18:45:52.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TM2o9dojQ3d7qwyUhWIIlZ43dzY=', 'Content-Length': '0'} 2023-11-02T18:45:52.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f9ff00abc11c8bd-006543ee60-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.736 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:52.737 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:52.737 DEBUG:boto:Method: HEAD 2023-11-02T18:45:52.737 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:45:52.738 DEBUG:boto:Data: 2023-11-02T18:45:52.738 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.739 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.742 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.743 DEBUG:boto:Params: {} 2023-11-02T18:45:52.743 DEBUG:boto:Token: None 2023-11-02T18:45:52.744 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj11 2023-11-02T18:45:52.744 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TM2o9dojQ3d7qwyUhWIIlZ43dzY= 2023-11-02T18:45:52.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TM2o9dojQ3d7qwyUhWIIlZ43dzY=', 'Content-Length': '0'} 2023-11-02T18:45:52.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1460c1f3676b4b2-006543ee60-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:45:52.748 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:52.749 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:52.749 DEBUG:boto:Method: GET 2023-11-02T18:45:52.750 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:45:52.750 DEBUG:boto:Data: 2023-11-02T18:45:52.751 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.751 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.752 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.752 DEBUG:boto:Params: {} 2023-11-02T18:45:52.752 DEBUG:boto:Token: None 2023-11-02T18:45:52.753 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj11 2023-11-02T18:45:52.753 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+NAyTbmznRaiBUePhxuz2dUswF8= 2023-11-02T18:45:52.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+NAyTbmznRaiBUePhxuz2dUswF8=', 'Content-Length': '0'} 2023-11-02T18:45:52.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a61e00075ebc0d3-006543ee60-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.798 DEBUG:boto:path=/ibtljn-59/obj11 2023-11-02T18:45:52.799 DEBUG:boto:auth_path=/ibtljn-59/obj11 2023-11-02T18:45:52.800 DEBUG:boto:Method: GET 2023-11-02T18:45:52.800 DEBUG:boto:Path: /ibtljn-59/obj11 2023-11-02T18:45:52.801 DEBUG:boto:Data: 2023-11-02T18:45:52.801 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.802 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.802 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.803 DEBUG:boto:Params: {} 2023-11-02T18:45:52.803 DEBUG:boto:Token: None 2023-11-02T18:45:52.804 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj11 2023-11-02T18:45:52.804 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+NAyTbmznRaiBUePhxuz2dUswF8= 2023-11-02T18:45:52.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+NAyTbmznRaiBUePhxuz2dUswF8=', 'Content-Length': '0'} 2023-11-02T18:45:52.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d75fbdedbfe11e34-006543ee60-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.809 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:52.810 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:52.810 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:52.811 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:52.811 DEBUG:boto:Method: GET 2023-11-02T18:45:52.812 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:45:52.812 DEBUG:boto:Data: 2023-11-02T18:45:52.812 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.813 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.813 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.814 DEBUG:boto:Params: {} 2023-11-02T18:45:52.814 DEBUG:boto:Token: None 2023-11-02T18:45:52.815 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/ 2023-11-02T18:45:52.815 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u+Va51RfSw7TdyxKazhULuGRKGE= 2023-11-02T18:45:52.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u+Va51RfSw7TdyxKazhULuGRKGE=', 'Content-Length': '0'} 2023-11-02T18:45:52.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000569ca319b0bed2f4-006543ee60-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.823 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:52.823 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:52.824 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:52.825 DEBUG:boto:path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:52.826 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj11 2023-11-02T18:45:52.827 DEBUG:boto:Method: GET 2023-11-02T18:45:52.827 DEBUG:boto:Path: /ibtljn-59/?prefix=obj11 2023-11-02T18:45:52.828 DEBUG:boto:Data: 2023-11-02T18:45:52.829 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.829 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.830 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.830 DEBUG:boto:Params: {} 2023-11-02T18:45:52.831 DEBUG:boto:Token: None 2023-11-02T18:45:52.831 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/ 2023-11-02T18:45:52.832 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u+Va51RfSw7TdyxKazhULuGRKGE= 2023-11-02T18:45:52.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u+Va51RfSw7TdyxKazhULuGRKGE=', 'Content-Length': '0'} 2023-11-02T18:45:52.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fee9badb4a4213c-006543ee60-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.844 DEBUG:boto:b'ibtljn-59obj111000falseobj112023-11-02T18:32:14.543Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:52.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-11-02T18:45:52.846 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:52.846 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:52.847 DEBUG:boto:path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.847 DEBUG:boto:auth_path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.847 DEBUG:boto:Method: GET 2023-11-02T18:45:52.848 DEBUG:boto:Path: /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.848 DEBUG:boto:Data: 2023-11-02T18:45:52.849 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.850 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.851 DEBUG:boto:Params: {} 2023-11-02T18:45:52.851 DEBUG:boto:Token: None 2023-11-02T18:45:52.852 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.852 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+PAZfg2IG8egzIIOmXTT2augRn4= 2023-11-02T18:45:52.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+PAZfg2IG8egzIIOmXTT2augRn4=', 'Content-Length': '0'} 2023-11-02T18:45:52.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbb1ed7eedfa0523-006543ee60-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.859 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:52.860 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:52.860 DEBUG:boto:path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.860 DEBUG:boto:auth_path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.861 DEBUG:boto:Method: GET 2023-11-02T18:45:52.861 DEBUG:boto:Path: /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.862 DEBUG:boto:Data: 2023-11-02T18:45:52.862 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.863 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.863 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.863 DEBUG:boto:Params: {} 2023-11-02T18:45:52.864 DEBUG:boto:Token: None 2023-11-02T18:45:52.864 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.865 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+PAZfg2IG8egzIIOmXTT2augRn4= 2023-11-02T18:45:52.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+PAZfg2IG8egzIIOmXTT2augRn4=', 'Content-Length': '0'} 2023-11-02T18:45:52.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d3f87e016e69f0d-006543ee60-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.869 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:52.870 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:52.870 DEBUG:boto:path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.870 DEBUG:boto:auth_path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.871 DEBUG:boto:Method: HEAD 2023-11-02T18:45:52.871 DEBUG:boto:Path: /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.872 DEBUG:boto:Data: 2023-11-02T18:45:52.872 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.872 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.873 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.873 DEBUG:boto:Params: {} 2023-11-02T18:45:52.874 DEBUG:boto:Token: None 2023-11-02T18:45:52.874 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.875 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pat9L239aiodcq9dAhw/fXfvdW4= 2023-11-02T18:45:52.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pat9L239aiodcq9dAhw/fXfvdW4=', 'Content-Length': '0'} 2023-11-02T18:45:52.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b4f9332bd5413d1-006543ee60-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.879 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:52.880 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:52.880 DEBUG:boto:path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.880 DEBUG:boto:auth_path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.881 DEBUG:boto:Method: HEAD 2023-11-02T18:45:52.881 DEBUG:boto:Path: /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.882 DEBUG:boto:Data: 2023-11-02T18:45:52.882 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.882 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.883 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.883 DEBUG:boto:Params: {} 2023-11-02T18:45:52.884 DEBUG:boto:Token: None 2023-11-02T18:45:52.885 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pat9L239aiodcq9dAhw/fXfvdW4= 2023-11-02T18:45:52.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pat9L239aiodcq9dAhw/fXfvdW4=', 'Content-Length': '0'} 2023-11-02T18:45:52.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fef7acaa70f7007-006543ee60-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-11-02T18:45:52.890 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:52.890 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:52.891 DEBUG:boto:path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.891 DEBUG:boto:auth_path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.891 DEBUG:boto:Method: GET 2023-11-02T18:45:52.892 DEBUG:boto:Path: /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.893 DEBUG:boto:Data: 2023-11-02T18:45:52.893 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.894 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.894 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.895 DEBUG:boto:Params: {} 2023-11-02T18:45:52.895 DEBUG:boto:Token: None 2023-11-02T18:45:52.896 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.896 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+PAZfg2IG8egzIIOmXTT2augRn4= 2023-11-02T18:45:52.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+PAZfg2IG8egzIIOmXTT2augRn4=', 'Content-Length': '0'} 2023-11-02T18:45:52.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e18742c9686b7a8-006543ee60-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.942 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:52.946 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:52.947 DEBUG:boto:path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.947 DEBUG:boto:auth_path=/ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.948 DEBUG:boto:Method: GET 2023-11-02T18:45:52.948 DEBUG:boto:Path: /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.948 DEBUG:boto:Data: 2023-11-02T18:45:52.949 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.949 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.950 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.950 DEBUG:boto:Params: {} 2023-11-02T18:45:52.951 DEBUG:boto:Token: None 2023-11-02T18:45:52.951 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/obj111?versionId=null 2023-11-02T18:45:52.952 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+PAZfg2IG8egzIIOmXTT2augRn4= 2023-11-02T18:45:52.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+PAZfg2IG8egzIIOmXTT2augRn4=', 'Content-Length': '0'} 2023-11-02T18:45:52.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004444add0e0cd3c63-006543ee60-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.956 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:52.957 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:52.957 DEBUG:boto:path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:52.958 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:52.958 DEBUG:boto:Method: GET 2023-11-02T18:45:52.958 DEBUG:boto:Path: /ibtljn-59/?prefix=obj111 2023-11-02T18:45:52.959 DEBUG:boto:Data: 2023-11-02T18:45:52.959 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.960 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:52.960 DEBUG:boto:Port: 8001 2023-11-02T18:45:52.961 DEBUG:boto:Params: {} 2023-11-02T18:45:52.961 DEBUG:boto:Token: None 2023-11-02T18:45:52.962 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/ 2023-11-02T18:45:52.962 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u+Va51RfSw7TdyxKazhULuGRKGE= 2023-11-02T18:45:52.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u+Va51RfSw7TdyxKazhULuGRKGE=', 'Content-Length': '0'} 2023-11-02T18:45:52.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f8d5f0ed6837f88-006543ee60-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:52.969 DEBUG:boto:b'ibtljn-59obj1111000falseobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:52.970 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:52.970 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:52.971 DEBUG:boto:path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:52.971 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:52.972 DEBUG:boto:Method: GET 2023-11-02T18:45:52.972 DEBUG:boto:Path: /ibtljn-59/?prefix=obj111 2023-11-02T18:45:52.972 DEBUG:boto:Data: 2023-11-02T18:45:52.973 DEBUG:boto:Headers: {} 2023-11-02T18:45:52.974 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:52.974 DEBUG:boto:Port: 8000 2023-11-02T18:45:52.975 DEBUG:boto:Params: {} 2023-11-02T18:45:52.975 DEBUG:boto:Token: None 2023-11-02T18:45:52.976 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:52 GMT /ibtljn-59/ 2023-11-02T18:45:52.976 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u+Va51RfSw7TdyxKazhULuGRKGE= 2023-11-02T18:45:52.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u+Va51RfSw7TdyxKazhULuGRKGE=', 'Content-Length': '0'} 2023-11-02T18:45:52.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8d1e553350983f0-006543ee60-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.030 DEBUG:boto:b'ibtljn-59obj1111000falseobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.032 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:53.032 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:53.033 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.033 DEBUG:boto:Path: /ibtljn-59/obj111 2023-11-02T18:45:53.034 DEBUG:boto:Data: 2023-11-02T18:45:53.034 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.034 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.035 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.035 DEBUG:boto:Params: {} 2023-11-02T18:45:53.036 DEBUG:boto:Token: None 2023-11-02T18:45:53.036 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj111 2023-11-02T18:45:53.037 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L+KF/VeoIw58sPLSMnJchTf8XiU= 2023-11-02T18:45:53.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L+KF/VeoIw58sPLSMnJchTf8XiU=', 'Content-Length': '0'} 2023-11-02T18:45:53.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9bc7c64242734da-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.041 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:53.042 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:53.042 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.043 DEBUG:boto:Path: /ibtljn-59/obj111 2023-11-02T18:45:53.043 DEBUG:boto:Data: 2023-11-02T18:45:53.044 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.045 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.045 DEBUG:boto:Params: {} 2023-11-02T18:45:53.045 DEBUG:boto:Token: None 2023-11-02T18:45:53.046 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj111 2023-11-02T18:45:53.046 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L+KF/VeoIw58sPLSMnJchTf8XiU= 2023-11-02T18:45:53.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L+KF/VeoIw58sPLSMnJchTf8XiU=', 'Content-Length': '0'} 2023-11-02T18:45:53.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e60b90ee6a9e4cc-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-11-02T18:45:53.051 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:53.051 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:53.052 DEBUG:boto:Method: GET 2023-11-02T18:45:53.052 DEBUG:boto:Path: /ibtljn-59/obj111 2023-11-02T18:45:53.052 DEBUG:boto:Data: 2023-11-02T18:45:53.053 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.053 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.054 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.054 DEBUG:boto:Params: {} 2023-11-02T18:45:53.055 DEBUG:boto:Token: None 2023-11-02T18:45:53.055 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj111 2023-11-02T18:45:53.056 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5iABsQfpGfDHAqIt3hxn/481PhM= 2023-11-02T18:45:53.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5iABsQfpGfDHAqIt3hxn/481PhM=', 'Content-Length': '0'} 2023-11-02T18:45:53.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000808fb79db482fb70-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.060 DEBUG:boto:path=/ibtljn-59/obj111 2023-11-02T18:45:53.060 DEBUG:boto:auth_path=/ibtljn-59/obj111 2023-11-02T18:45:53.061 DEBUG:boto:Method: GET 2023-11-02T18:45:53.061 DEBUG:boto:Path: /ibtljn-59/obj111 2023-11-02T18:45:53.062 DEBUG:boto:Data: 2023-11-02T18:45:53.062 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.062 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.063 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.063 DEBUG:boto:Params: {} 2023-11-02T18:45:53.064 DEBUG:boto:Token: None 2023-11-02T18:45:53.064 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj111 2023-11-02T18:45:53.065 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5iABsQfpGfDHAqIt3hxn/481PhM= 2023-11-02T18:45:53.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5iABsQfpGfDHAqIt3hxn/481PhM=', 'Content-Length': '0'} 2023-11-02T18:45:53.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000871a27616fba26b-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.069 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.069 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.069 DEBUG:boto:path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:53.070 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:53.070 DEBUG:boto:Method: GET 2023-11-02T18:45:53.071 DEBUG:boto:Path: /ibtljn-59/?prefix=obj111 2023-11-02T18:45:53.071 DEBUG:boto:Data: 2023-11-02T18:45:53.072 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.072 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.072 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.073 DEBUG:boto:Params: {} 2023-11-02T18:45:53.073 DEBUG:boto:Token: None 2023-11-02T18:45:53.074 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.074 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfc9132454ee07f6-006543ee61-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.122 DEBUG:boto:b'ibtljn-59obj1111000falseobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.123 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.124 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.124 DEBUG:boto:path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:53.125 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj111 2023-11-02T18:45:53.125 DEBUG:boto:Method: GET 2023-11-02T18:45:53.126 DEBUG:boto:Path: /ibtljn-59/?prefix=obj111 2023-11-02T18:45:53.126 DEBUG:boto:Data: 2023-11-02T18:45:53.126 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.127 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.127 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.128 DEBUG:boto:Params: {} 2023-11-02T18:45:53.128 DEBUG:boto:Token: None 2023-11-02T18:45:53.129 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bfbe0633aa28816-006543ee61-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.140 DEBUG:boto:b'ibtljn-59obj1111000falseobj1112023-11-02T18:44:13.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:45:53.142 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:53.142 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:53.143 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.143 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.143 DEBUG:boto:Method: GET 2023-11-02T18:45:53.144 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.144 DEBUG:boto:Data: 2023-11-02T18:45:53.145 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.145 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.146 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.146 DEBUG:boto:Params: {} 2023-11-02T18:45:53.151 DEBUG:boto:Token: None 2023-11-02T18:45:53.151 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.152 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1MSKrfLWjBsHDdfcBwxnrtU1AeQ= 2023-11-02T18:45:53.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1MSKrfLWjBsHDdfcBwxnrtU1AeQ=', 'Content-Length': '0'} 2023-11-02T18:45:53.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081a7bdca509d8aec-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.156 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:53.157 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:53.158 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.158 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.159 DEBUG:boto:Method: GET 2023-11-02T18:45:53.159 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.159 DEBUG:boto:Data: 2023-11-02T18:45:53.160 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.160 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.161 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.161 DEBUG:boto:Params: {} 2023-11-02T18:45:53.162 DEBUG:boto:Token: None 2023-11-02T18:45:53.162 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.163 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1MSKrfLWjBsHDdfcBwxnrtU1AeQ= 2023-11-02T18:45:53.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1MSKrfLWjBsHDdfcBwxnrtU1AeQ=', 'Content-Length': '0'} 2023-11-02T18:45:53.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a6d756de6bbfb06-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.167 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:53.167 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:53.168 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.168 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.168 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.169 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.169 DEBUG:boto:Data: 2023-11-02T18:45:53.170 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.170 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.170 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.171 DEBUG:boto:Params: {} 2023-11-02T18:45:53.171 DEBUG:boto:Token: None 2023-11-02T18:45:53.172 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.172 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KSegkSCQQWRGfhwUIyOkW4/pSyg= 2023-11-02T18:45:53.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KSegkSCQQWRGfhwUIyOkW4/pSyg=', 'Content-Length': '0'} 2023-11-02T18:45:53.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000322d1a2d956f906a-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.176 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:53.176 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:53.177 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.177 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.178 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.178 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.179 DEBUG:boto:Data: 2023-11-02T18:45:53.179 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.180 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.180 DEBUG:boto:Params: {} 2023-11-02T18:45:53.181 DEBUG:boto:Token: None 2023-11-02T18:45:53.181 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.182 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KSegkSCQQWRGfhwUIyOkW4/pSyg= 2023-11-02T18:45:53.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KSegkSCQQWRGfhwUIyOkW4/pSyg=', 'Content-Length': '0'} 2023-11-02T18:45:53.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000189f93b446b66a95-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:45:53.186 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:53.186 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:53.187 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.187 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.188 DEBUG:boto:Method: GET 2023-11-02T18:45:53.188 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.188 DEBUG:boto:Data: 2023-11-02T18:45:53.189 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.189 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.190 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.190 DEBUG:boto:Params: {} 2023-11-02T18:45:53.191 DEBUG:boto:Token: None 2023-11-02T18:45:53.191 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.192 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1MSKrfLWjBsHDdfcBwxnrtU1AeQ= 2023-11-02T18:45:53.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1MSKrfLWjBsHDdfcBwxnrtU1AeQ=', 'Content-Length': '0'} 2023-11-02T18:45:53.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081d59ae6d6050ad8-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.238 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:53.239 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:53.240 DEBUG:boto:path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.240 DEBUG:boto:auth_path=/ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.241 DEBUG:boto:Method: GET 2023-11-02T18:45:53.241 DEBUG:boto:Path: /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.241 DEBUG:boto:Data: 2023-11-02T18:45:53.242 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.242 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.243 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.244 DEBUG:boto:Params: {} 2023-11-02T18:45:53.244 DEBUG:boto:Token: None 2023-11-02T18:45:53.245 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj115?versionId=null 2023-11-02T18:45:53.245 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1MSKrfLWjBsHDdfcBwxnrtU1AeQ= 2023-11-02T18:45:53.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1MSKrfLWjBsHDdfcBwxnrtU1AeQ=', 'Content-Length': '0'} 2023-11-02T18:45:53.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eee8c04029cf3b8-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.250 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.250 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.251 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:53.251 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:53.251 DEBUG:boto:Method: GET 2023-11-02T18:45:53.253 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:45:53.254 DEBUG:boto:Data: 2023-11-02T18:45:53.254 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.255 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.255 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.256 DEBUG:boto:Params: {} 2023-11-02T18:45:53.256 DEBUG:boto:Token: None 2023-11-02T18:45:53.257 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.258 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000368e43d311d54f3f-006543ee61-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.263 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.265 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.265 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.265 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:53.266 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:53.266 DEBUG:boto:Method: GET 2023-11-02T18:45:53.267 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:45:53.267 DEBUG:boto:Data: 2023-11-02T18:45:53.268 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.268 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.269 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.269 DEBUG:boto:Params: {} 2023-11-02T18:45:53.270 DEBUG:boto:Token: None 2023-11-02T18:45:53.270 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.271 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072641db7cc277cc4-006543ee61-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.281 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.282 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:53.282 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:53.283 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.283 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:45:53.284 DEBUG:boto:Data: 2023-11-02T18:45:53.284 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.284 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.285 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.285 DEBUG:boto:Params: {} 2023-11-02T18:45:53.286 DEBUG:boto:Token: None 2023-11-02T18:45:53.286 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj115 2023-11-02T18:45:53.287 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2pIy+J4qQJbG0Oq0WO64SZSXjhQ= 2023-11-02T18:45:53.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2pIy+J4qQJbG0Oq0WO64SZSXjhQ=', 'Content-Length': '0'} 2023-11-02T18:45:53.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f98fb7672e68cc8a-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.291 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:53.291 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:53.291 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.292 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:45:53.292 DEBUG:boto:Data: 2023-11-02T18:45:53.293 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.293 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.294 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.294 DEBUG:boto:Params: {} 2023-11-02T18:45:53.294 DEBUG:boto:Token: None 2023-11-02T18:45:53.295 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj115 2023-11-02T18:45:53.295 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2pIy+J4qQJbG0Oq0WO64SZSXjhQ= 2023-11-02T18:45:53.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2pIy+J4qQJbG0Oq0WO64SZSXjhQ=', 'Content-Length': '0'} 2023-11-02T18:45:53.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008591a5d3a3f77e2f-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:45:53.299 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:53.300 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:53.300 DEBUG:boto:Method: GET 2023-11-02T18:45:53.301 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:45:53.301 DEBUG:boto:Data: 2023-11-02T18:45:53.301 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.302 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.303 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.303 DEBUG:boto:Params: {} 2023-11-02T18:45:53.303 DEBUG:boto:Token: None 2023-11-02T18:45:53.304 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj115 2023-11-02T18:45:53.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l1Scmr5F8+qd+LsJVd4eXdIVNCY= 2023-11-02T18:45:53.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l1Scmr5F8+qd+LsJVd4eXdIVNCY=', 'Content-Length': '0'} 2023-11-02T18:45:53.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bacfe4fde248f83b-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.309 DEBUG:boto:path=/ibtljn-59/obj115 2023-11-02T18:45:53.309 DEBUG:boto:auth_path=/ibtljn-59/obj115 2023-11-02T18:45:53.310 DEBUG:boto:Method: GET 2023-11-02T18:45:53.310 DEBUG:boto:Path: /ibtljn-59/obj115 2023-11-02T18:45:53.311 DEBUG:boto:Data: 2023-11-02T18:45:53.311 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.312 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.312 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.313 DEBUG:boto:Params: {} 2023-11-02T18:45:53.313 DEBUG:boto:Token: None 2023-11-02T18:45:53.314 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj115 2023-11-02T18:45:53.314 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l1Scmr5F8+qd+LsJVd4eXdIVNCY= 2023-11-02T18:45:53.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l1Scmr5F8+qd+LsJVd4eXdIVNCY=', 'Content-Length': '0'} 2023-11-02T18:45:53.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a05ccc6dd771d1d4-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.358 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.359 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.360 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:53.360 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:53.361 DEBUG:boto:Method: GET 2023-11-02T18:45:53.361 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:45:53.361 DEBUG:boto:Data: 2023-11-02T18:45:53.362 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.363 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.363 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.364 DEBUG:boto:Params: {} 2023-11-02T18:45:53.364 DEBUG:boto:Token: None 2023-11-02T18:45:53.365 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.365 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a9ead8e24afdcb6-006543ee61-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.371 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.372 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.373 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.373 DEBUG:boto:path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:53.374 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj115 2023-11-02T18:45:53.374 DEBUG:boto:Method: GET 2023-11-02T18:45:53.375 DEBUG:boto:Path: /ibtljn-59/?prefix=obj115 2023-11-02T18:45:53.375 DEBUG:boto:Data: 2023-11-02T18:45:53.376 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.377 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.377 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.378 DEBUG:boto:Params: {} 2023-11-02T18:45:53.378 DEBUG:boto:Token: None 2023-11-02T18:45:53.379 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.379 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9d7d9668ca80d20-006543ee61-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.390 DEBUG:boto:b'ibtljn-59obj1151000falseobj1152023-11-02T18:39:10.589Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:45:53.391 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:53.392 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:53.392 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.392 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.393 DEBUG:boto:Method: GET 2023-11-02T18:45:53.393 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.394 DEBUG:boto:Data: 2023-11-02T18:45:53.395 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.395 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.396 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.396 DEBUG:boto:Params: {} 2023-11-02T18:45:53.396 DEBUG:boto:Token: None 2023-11-02T18:45:53.397 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.397 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+Bdk/3qHJ9MZQl0jFexg1WM6pco= 2023-11-02T18:45:53.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+Bdk/3qHJ9MZQl0jFexg1WM6pco=', 'Content-Length': '0'} 2023-11-02T18:45:53.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d7380af3e3d7180-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.450 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:53.451 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:53.451 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.452 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.452 DEBUG:boto:Method: GET 2023-11-02T18:45:53.453 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.453 DEBUG:boto:Data: 2023-11-02T18:45:53.454 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.455 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.455 DEBUG:boto:Params: {} 2023-11-02T18:45:53.456 DEBUG:boto:Token: None 2023-11-02T18:45:53.456 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.457 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+Bdk/3qHJ9MZQl0jFexg1WM6pco= 2023-11-02T18:45:53.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+Bdk/3qHJ9MZQl0jFexg1WM6pco=', 'Content-Length': '0'} 2023-11-02T18:45:53.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008de68dfeff50e5ce-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.461 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:53.462 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:53.462 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.463 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.463 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.464 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.464 DEBUG:boto:Data: 2023-11-02T18:45:53.465 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.465 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.465 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.466 DEBUG:boto:Params: {} 2023-11-02T18:45:53.466 DEBUG:boto:Token: None 2023-11-02T18:45:53.467 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.467 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tGbxznCneq/669mqCLsKjF+oLl8= 2023-11-02T18:45:53.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tGbxznCneq/669mqCLsKjF+oLl8=', 'Content-Length': '0'} 2023-11-02T18:45:53.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be5eaa3cf87d3d17-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.472 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:53.472 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:53.473 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.473 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.474 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.474 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.475 DEBUG:boto:Data: 2023-11-02T18:45:53.475 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.475 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.476 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.476 DEBUG:boto:Params: {} 2023-11-02T18:45:53.477 DEBUG:boto:Token: None 2023-11-02T18:45:53.477 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.478 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tGbxznCneq/669mqCLsKjF+oLl8= 2023-11-02T18:45:53.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tGbxznCneq/669mqCLsKjF+oLl8=', 'Content-Length': '0'} 2023-11-02T18:45:53.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000251a1c4a1606b4be-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:45:53.482 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:53.483 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:53.483 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.484 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.484 DEBUG:boto:Method: GET 2023-11-02T18:45:53.485 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.485 DEBUG:boto:Data: 2023-11-02T18:45:53.486 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.486 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.486 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.487 DEBUG:boto:Params: {} 2023-11-02T18:45:53.487 DEBUG:boto:Token: None 2023-11-02T18:45:53.488 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.488 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+Bdk/3qHJ9MZQl0jFexg1WM6pco= 2023-11-02T18:45:53.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+Bdk/3qHJ9MZQl0jFexg1WM6pco=', 'Content-Length': '0'} 2023-11-02T18:45:53.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2c480675a852b0a-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.494 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:53.495 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:53.495 DEBUG:boto:path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.496 DEBUG:boto:auth_path=/ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.496 DEBUG:boto:Method: GET 2023-11-02T18:45:53.497 DEBUG:boto:Path: /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.497 DEBUG:boto:Data: 2023-11-02T18:45:53.498 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.498 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.499 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.499 DEBUG:boto:Params: {} 2023-11-02T18:45:53.500 DEBUG:boto:Token: None 2023-11-02T18:45:53.500 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj116?versionId=null 2023-11-02T18:45:53.501 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+Bdk/3qHJ9MZQl0jFexg1WM6pco= 2023-11-02T18:45:53.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+Bdk/3qHJ9MZQl0jFexg1WM6pco=', 'Content-Length': '0'} 2023-11-02T18:45:53.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb1594541f27f19c-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.505 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.505 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.506 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:53.506 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:53.507 DEBUG:boto:Method: GET 2023-11-02T18:45:53.507 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:45:53.508 DEBUG:boto:Data: 2023-11-02T18:45:53.508 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.509 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.509 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.509 DEBUG:boto:Params: {} 2023-11-02T18:45:53.510 DEBUG:boto:Token: None 2023-11-02T18:45:53.510 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.511 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c45cc5e5bc18febe-006543ee61-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.562 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.563 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.564 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.564 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:53.565 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:53.565 DEBUG:boto:Method: GET 2023-11-02T18:45:53.566 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:45:53.566 DEBUG:boto:Data: 2023-11-02T18:45:53.567 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.567 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.568 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.568 DEBUG:boto:Params: {} 2023-11-02T18:45:53.569 DEBUG:boto:Token: None 2023-11-02T18:45:53.570 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.570 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ea333b39ea2d194-006543ee61-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.582 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.583 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:53.583 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:53.584 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.584 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:45:53.585 DEBUG:boto:Data: 2023-11-02T18:45:53.585 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.586 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.586 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.587 DEBUG:boto:Params: {} 2023-11-02T18:45:53.587 DEBUG:boto:Token: None 2023-11-02T18:45:53.588 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj116 2023-11-02T18:45:53.588 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UHz87iQWJ9LAkFsnSPiI7YAVBOU= 2023-11-02T18:45:53.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UHz87iQWJ9LAkFsnSPiI7YAVBOU=', 'Content-Length': '0'} 2023-11-02T18:45:53.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003395e0591b211cef-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.592 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:53.592 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:53.593 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.593 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:45:53.594 DEBUG:boto:Data: 2023-11-02T18:45:53.594 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.595 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.595 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.596 DEBUG:boto:Params: {} 2023-11-02T18:45:53.596 DEBUG:boto:Token: None 2023-11-02T18:45:53.597 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj116 2023-11-02T18:45:53.597 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UHz87iQWJ9LAkFsnSPiI7YAVBOU= 2023-11-02T18:45:53.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UHz87iQWJ9LAkFsnSPiI7YAVBOU=', 'Content-Length': '0'} 2023-11-02T18:45:53.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb71f6538000bc18-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:45:53.601 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:53.601 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:53.602 DEBUG:boto:Method: GET 2023-11-02T18:45:53.602 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:45:53.603 DEBUG:boto:Data: 2023-11-02T18:45:53.603 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.604 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.604 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.605 DEBUG:boto:Params: {} 2023-11-02T18:45:53.605 DEBUG:boto:Token: None 2023-11-02T18:45:53.606 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj116 2023-11-02T18:45:53.606 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8RW/bjflyHCyXZmFTooLgRWtczg= 2023-11-02T18:45:53.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8RW/bjflyHCyXZmFTooLgRWtczg=', 'Content-Length': '0'} 2023-11-02T18:45:53.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a435270b2a5a9fcd-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.610 DEBUG:boto:path=/ibtljn-59/obj116 2023-11-02T18:45:53.611 DEBUG:boto:auth_path=/ibtljn-59/obj116 2023-11-02T18:45:53.611 DEBUG:boto:Method: GET 2023-11-02T18:45:53.612 DEBUG:boto:Path: /ibtljn-59/obj116 2023-11-02T18:45:53.612 DEBUG:boto:Data: 2023-11-02T18:45:53.613 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.613 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.614 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.614 DEBUG:boto:Params: {} 2023-11-02T18:45:53.615 DEBUG:boto:Token: None 2023-11-02T18:45:53.615 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj116 2023-11-02T18:45:53.616 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8RW/bjflyHCyXZmFTooLgRWtczg= 2023-11-02T18:45:53.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8RW/bjflyHCyXZmFTooLgRWtczg=', 'Content-Length': '0'} 2023-11-02T18:45:53.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000185555a3b37e0804-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.620 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.621 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.621 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:53.621 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:53.622 DEBUG:boto:Method: GET 2023-11-02T18:45:53.622 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:45:53.623 DEBUG:boto:Data: 2023-11-02T18:45:53.623 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.624 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.624 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.625 DEBUG:boto:Params: {} 2023-11-02T18:45:53.625 DEBUG:boto:Token: None 2023-11-02T18:45:53.626 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009002e2081a81cd92-006543ee61-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.634 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.635 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.635 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.636 DEBUG:boto:path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:53.636 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj116 2023-11-02T18:45:53.637 DEBUG:boto:Method: GET 2023-11-02T18:45:53.637 DEBUG:boto:Path: /ibtljn-59/?prefix=obj116 2023-11-02T18:45:53.638 DEBUG:boto:Data: 2023-11-02T18:45:53.638 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.639 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.640 DEBUG:boto:Params: {} 2023-11-02T18:45:53.640 DEBUG:boto:Token: None 2023-11-02T18:45:53.641 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1233d5edc9a6f69-006543ee61-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.690 DEBUG:boto:b'ibtljn-59obj1161000falseobj1162023-11-02T18:40:31.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T18:45:53.692 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:53.692 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:53.693 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.693 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.693 DEBUG:boto:Method: GET 2023-11-02T18:45:53.694 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.694 DEBUG:boto:Data: 2023-11-02T18:45:53.695 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.696 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.696 DEBUG:boto:Params: {} 2023-11-02T18:45:53.697 DEBUG:boto:Token: None 2023-11-02T18:45:53.697 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.698 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3gC0pLpOcULGKs5II2iAUYTm7Hg= 2023-11-02T18:45:53.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3gC0pLpOcULGKs5II2iAUYTm7Hg=', 'Content-Length': '0'} 2023-11-02T18:45:53.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cd9c195a502cb23-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.702 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:53.703 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:53.703 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.704 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.704 DEBUG:boto:Method: GET 2023-11-02T18:45:53.705 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.705 DEBUG:boto:Data: 2023-11-02T18:45:53.705 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.706 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.706 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.707 DEBUG:boto:Params: {} 2023-11-02T18:45:53.707 DEBUG:boto:Token: None 2023-11-02T18:45:53.708 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.708 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3gC0pLpOcULGKs5II2iAUYTm7Hg= 2023-11-02T18:45:53.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3gC0pLpOcULGKs5II2iAUYTm7Hg=', 'Content-Length': '0'} 2023-11-02T18:45:53.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009596645ab2d5f849-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.713 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:53.714 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:53.714 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.714 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.715 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.715 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.716 DEBUG:boto:Data: 2023-11-02T18:45:53.716 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.716 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.717 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.717 DEBUG:boto:Params: {} 2023-11-02T18:45:53.718 DEBUG:boto:Token: None 2023-11-02T18:45:53.719 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.719 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XdPE/p9r94rLf/bz+xo7jErSYrc= 2023-11-02T18:45:53.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XdPE/p9r94rLf/bz+xo7jErSYrc=', 'Content-Length': '0'} 2023-11-02T18:45:53.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f67d01c93831c60-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.723 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:53.723 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:53.724 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.724 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.725 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.725 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.726 DEBUG:boto:Data: 2023-11-02T18:45:53.726 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.727 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.727 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.728 DEBUG:boto:Params: {} 2023-11-02T18:45:53.728 DEBUG:boto:Token: None 2023-11-02T18:45:53.729 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.729 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XdPE/p9r94rLf/bz+xo7jErSYrc= 2023-11-02T18:45:53.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XdPE/p9r94rLf/bz+xo7jErSYrc=', 'Content-Length': '0'} 2023-11-02T18:45:53.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cfdaa6aaf8a7081-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T18:45:53.735 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:53.736 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:53.736 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.736 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.737 DEBUG:boto:Method: GET 2023-11-02T18:45:53.737 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.738 DEBUG:boto:Data: 2023-11-02T18:45:53.738 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.739 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.739 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.739 DEBUG:boto:Params: {} 2023-11-02T18:45:53.740 DEBUG:boto:Token: None 2023-11-02T18:45:53.740 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.741 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3gC0pLpOcULGKs5II2iAUYTm7Hg= 2023-11-02T18:45:53.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3gC0pLpOcULGKs5II2iAUYTm7Hg=', 'Content-Length': '0'} 2023-11-02T18:45:53.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a4dd666af387478-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.790 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:53.791 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:53.791 DEBUG:boto:path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.792 DEBUG:boto:auth_path=/ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.792 DEBUG:boto:Method: GET 2023-11-02T18:45:53.793 DEBUG:boto:Path: /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.793 DEBUG:boto:Data: 2023-11-02T18:45:53.794 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.794 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.795 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.795 DEBUG:boto:Params: {} 2023-11-02T18:45:53.796 DEBUG:boto:Token: None 2023-11-02T18:45:53.796 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj124?versionId=null 2023-11-02T18:45:53.797 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3gC0pLpOcULGKs5II2iAUYTm7Hg= 2023-11-02T18:45:53.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3gC0pLpOcULGKs5II2iAUYTm7Hg=', 'Content-Length': '0'} 2023-11-02T18:45:53.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e98f0356320fba90-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.801 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.802 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.802 DEBUG:boto:path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:53.802 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:53.803 DEBUG:boto:Method: GET 2023-11-02T18:45:53.803 DEBUG:boto:Path: /ibtljn-59/?prefix=obj124 2023-11-02T18:45:53.804 DEBUG:boto:Data: 2023-11-02T18:45:53.804 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.805 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.805 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.805 DEBUG:boto:Params: {} 2023-11-02T18:45:53.806 DEBUG:boto:Token: None 2023-11-02T18:45:53.806 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.807 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a141020ae0fa411-006543ee61-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.813 DEBUG:boto:b'ibtljn-59obj1241000falseobj1242023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.814 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.815 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.815 DEBUG:boto:path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:53.815 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:53.816 DEBUG:boto:Method: GET 2023-11-02T18:45:53.816 DEBUG:boto:Path: /ibtljn-59/?prefix=obj124 2023-11-02T18:45:53.817 DEBUG:boto:Data: 2023-11-02T18:45:53.817 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.818 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.818 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.819 DEBUG:boto:Params: {} 2023-11-02T18:45:53.819 DEBUG:boto:Token: None 2023-11-02T18:45:53.820 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.820 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8a137cb9e2af3aa-006543ee61-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.879 DEBUG:boto:b'ibtljn-59obj1241000falseobj1242023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.880 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:53.881 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:53.881 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.881 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:45:53.882 DEBUG:boto:Data: 2023-11-02T18:45:53.882 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.883 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.883 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.883 DEBUG:boto:Params: {} 2023-11-02T18:45:53.884 DEBUG:boto:Token: None 2023-11-02T18:45:53.885 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj124 2023-11-02T18:45:53.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H3+i6Z6poa8b/80rALzWOv9+ctc= 2023-11-02T18:45:53.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H3+i6Z6poa8b/80rALzWOv9+ctc=', 'Content-Length': '0'} 2023-11-02T18:45:53.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a95f1f56a29f4fa-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.890 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:53.891 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:53.891 DEBUG:boto:Method: HEAD 2023-11-02T18:45:53.891 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:45:53.892 DEBUG:boto:Data: 2023-11-02T18:45:53.892 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.893 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.893 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.893 DEBUG:boto:Params: {} 2023-11-02T18:45:53.894 DEBUG:boto:Token: None 2023-11-02T18:45:53.894 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj124 2023-11-02T18:45:53.895 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H3+i6Z6poa8b/80rALzWOv9+ctc= 2023-11-02T18:45:53.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H3+i6Z6poa8b/80rALzWOv9+ctc=', 'Content-Length': '0'} 2023-11-02T18:45:53.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000946a2c51bb61297b-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T18:45:53.899 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:53.899 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:53.900 DEBUG:boto:Method: GET 2023-11-02T18:45:53.900 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:45:53.900 DEBUG:boto:Data: 2023-11-02T18:45:53.901 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.901 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.902 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.902 DEBUG:boto:Params: {} 2023-11-02T18:45:53.903 DEBUG:boto:Token: None 2023-11-02T18:45:53.903 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj124 2023-11-02T18:45:53.904 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MuMfJ2Oeus/0ShYswVp3gqgA4R0= 2023-11-02T18:45:53.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MuMfJ2Oeus/0ShYswVp3gqgA4R0=', 'Content-Length': '0'} 2023-11-02T18:45:53.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001663099226871476-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.908 DEBUG:boto:path=/ibtljn-59/obj124 2023-11-02T18:45:53.908 DEBUG:boto:auth_path=/ibtljn-59/obj124 2023-11-02T18:45:53.908 DEBUG:boto:Method: GET 2023-11-02T18:45:53.909 DEBUG:boto:Path: /ibtljn-59/obj124 2023-11-02T18:45:53.909 DEBUG:boto:Data: 2023-11-02T18:45:53.910 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.911 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.911 DEBUG:boto:Params: {} 2023-11-02T18:45:53.912 DEBUG:boto:Token: None 2023-11-02T18:45:53.912 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj124 2023-11-02T18:45:53.913 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MuMfJ2Oeus/0ShYswVp3gqgA4R0= 2023-11-02T18:45:53.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MuMfJ2Oeus/0ShYswVp3gqgA4R0=', 'Content-Length': '0'} 2023-11-02T18:45:53.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000532b6ccac33aaf96-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.916 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.917 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.917 DEBUG:boto:path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:53.918 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:53.918 DEBUG:boto:Method: GET 2023-11-02T18:45:53.919 DEBUG:boto:Path: /ibtljn-59/?prefix=obj124 2023-11-02T18:45:53.919 DEBUG:boto:Data: 2023-11-02T18:45:53.920 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.920 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.920 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.921 DEBUG:boto:Params: {} 2023-11-02T18:45:53.921 DEBUG:boto:Token: None 2023-11-02T18:45:53.922 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ef8af8715b73037-006543ee61-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.928 DEBUG:boto:b'ibtljn-59obj1241000falseobj1242023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.929 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:53.929 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:53.930 DEBUG:boto:path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:53.930 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj124 2023-11-02T18:45:53.930 DEBUG:boto:Method: GET 2023-11-02T18:45:53.931 DEBUG:boto:Path: /ibtljn-59/?prefix=obj124 2023-11-02T18:45:53.932 DEBUG:boto:Data: 2023-11-02T18:45:53.932 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.932 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:53.933 DEBUG:boto:Port: 8000 2023-11-02T18:45:53.933 DEBUG:boto:Params: {} 2023-11-02T18:45:53.934 DEBUG:boto:Token: None 2023-11-02T18:45:53.934 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/ 2023-11-02T18:45:53.935 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE= 2023-11-02T18:45:53.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/1WQC0dqSiopKMFNBuaWbfbn6iE=', 'Content-Length': '0'} 2023-11-02T18:45:53.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5ecbb229a3376db-006543ee61-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.946 DEBUG:boto:b'ibtljn-59obj1241000falseobj1242023-11-02T18:42:34.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:53.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:45:53.947 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:53.947 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:53.948 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:53.948 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:53.949 DEBUG:boto:Method: GET 2023-11-02T18:45:53.949 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:53.949 DEBUG:boto:Data: 2023-11-02T18:45:53.950 DEBUG:boto:Headers: {} 2023-11-02T18:45:53.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:53.951 DEBUG:boto:Port: 8001 2023-11-02T18:45:53.951 DEBUG:boto:Params: {} 2023-11-02T18:45:53.952 DEBUG:boto:Token: None 2023-11-02T18:45:53.952 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:53 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:53.953 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A0rEJmO/0BAFCAmWnInWuLzcySk= 2023-11-02T18:45:53.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A0rEJmO/0BAFCAmWnInWuLzcySk=', 'Content-Length': '0'} 2023-11-02T18:45:53.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cb5f67ac9bb4430-006543ee61-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:53.998 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:53.999 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:53.999 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.000 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.000 DEBUG:boto:Method: GET 2023-11-02T18:45:54.001 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.001 DEBUG:boto:Data: 2023-11-02T18:45:54.002 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.002 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.003 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.003 DEBUG:boto:Params: {} 2023-11-02T18:45:54.003 DEBUG:boto:Token: None 2023-11-02T18:45:54.004 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.004 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KYtw+PxQ+xkBb/T1eyZXg4Co11U= 2023-11-02T18:45:54.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KYtw+PxQ+xkBb/T1eyZXg4Co11U=', 'Content-Length': '0'} 2023-11-02T18:45:54.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008906d4789ab56c6d-006543ee61-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.009 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:54.009 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:54.010 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.010 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.011 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.011 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.011 DEBUG:boto:Data: 2023-11-02T18:45:54.013 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.013 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.014 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.014 DEBUG:boto:Params: {} 2023-11-02T18:45:54.014 DEBUG:boto:Token: None 2023-11-02T18:45:54.015 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.015 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RdOWwxeuDpGhRmdWhuU9Wang8LM= 2023-11-02T18:45:54.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RdOWwxeuDpGhRmdWhuU9Wang8LM=', 'Content-Length': '0'} 2023-11-02T18:45:54.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b707c86763cfad59-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.020 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:54.020 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:54.021 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.021 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.022 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.022 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.022 DEBUG:boto:Data: 2023-11-02T18:45:54.023 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.023 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.024 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.024 DEBUG:boto:Params: {} 2023-11-02T18:45:54.024 DEBUG:boto:Token: None 2023-11-02T18:45:54.025 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.025 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RdOWwxeuDpGhRmdWhuU9Wang8LM= 2023-11-02T18:45:54.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RdOWwxeuDpGhRmdWhuU9Wang8LM=', 'Content-Length': '0'} 2023-11-02T18:45:54.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d732e74641135b1b-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:45:54.030 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:54.030 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:54.031 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.031 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.032 DEBUG:boto:Method: GET 2023-11-02T18:45:54.032 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.032 DEBUG:boto:Data: 2023-11-02T18:45:54.033 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.033 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.034 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.034 DEBUG:boto:Params: {} 2023-11-02T18:45:54.035 DEBUG:boto:Token: None 2023-11-02T18:45:54.035 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.036 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KYtw+PxQ+xkBb/T1eyZXg4Co11U= 2023-11-02T18:45:54.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KYtw+PxQ+xkBb/T1eyZXg4Co11U=', 'Content-Length': '0'} 2023-11-02T18:45:54.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088fedeae05a1c05c-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.040 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:54.041 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:54.041 DEBUG:boto:path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.042 DEBUG:boto:auth_path=/ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.042 DEBUG:boto:Method: GET 2023-11-02T18:45:54.043 DEBUG:boto:Path: /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.043 DEBUG:boto:Data: 2023-11-02T18:45:54.044 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.045 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.045 DEBUG:boto:Params: {} 2023-11-02T18:45:54.046 DEBUG:boto:Token: None 2023-11-02T18:45:54.046 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj145?versionId=null 2023-11-02T18:45:54.047 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KYtw+PxQ+xkBb/T1eyZXg4Co11U= 2023-11-02T18:45:54.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KYtw+PxQ+xkBb/T1eyZXg4Co11U=', 'Content-Length': '0'} 2023-11-02T18:45:54.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df6c833a9c8895b8-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.094 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.095 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.096 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:54.096 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:54.096 DEBUG:boto:Method: GET 2023-11-02T18:45:54.097 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:45:54.097 DEBUG:boto:Data: 2023-11-02T18:45:54.098 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.099 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.099 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.099 DEBUG:boto:Params: {} 2023-11-02T18:45:54.100 DEBUG:boto:Token: None 2023-11-02T18:45:54.101 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.101 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e94134abcad661b-006543ee62-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.107 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.108 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.109 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.109 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:54.110 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:54.110 DEBUG:boto:Method: GET 2023-11-02T18:45:54.111 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:45:54.111 DEBUG:boto:Data: 2023-11-02T18:45:54.112 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.112 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.113 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.113 DEBUG:boto:Params: {} 2023-11-02T18:45:54.114 DEBUG:boto:Token: None 2023-11-02T18:45:54.114 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.115 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7c1aa0918018175-006543ee62-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.126 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.127 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:54.128 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:54.128 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.129 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:45:54.129 DEBUG:boto:Data: 2023-11-02T18:45:54.130 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.130 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.131 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.131 DEBUG:boto:Params: {} 2023-11-02T18:45:54.132 DEBUG:boto:Token: None 2023-11-02T18:45:54.132 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj145 2023-11-02T18:45:54.133 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jkgtO/DUI0iCXgY+LGYwq7iKo4o= 2023-11-02T18:45:54.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jkgtO/DUI0iCXgY+LGYwq7iKo4o=', 'Content-Length': '0'} 2023-11-02T18:45:54.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007edcf0dabb1f7aad-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.137 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:54.137 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:54.137 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.138 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:45:54.138 DEBUG:boto:Data: 2023-11-02T18:45:54.139 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.140 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.140 DEBUG:boto:Params: {} 2023-11-02T18:45:54.141 DEBUG:boto:Token: None 2023-11-02T18:45:54.141 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj145 2023-11-02T18:45:54.142 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jkgtO/DUI0iCXgY+LGYwq7iKo4o= 2023-11-02T18:45:54.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jkgtO/DUI0iCXgY+LGYwq7iKo4o=', 'Content-Length': '0'} 2023-11-02T18:45:54.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f019db06a2aa0a0-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:45:54.146 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:54.147 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:54.147 DEBUG:boto:Method: GET 2023-11-02T18:45:54.148 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:45:54.148 DEBUG:boto:Data: 2023-11-02T18:45:54.149 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.149 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.150 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.150 DEBUG:boto:Params: {} 2023-11-02T18:45:54.151 DEBUG:boto:Token: None 2023-11-02T18:45:54.151 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj145 2023-11-02T18:45:54.152 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nTotliVlfVOEDsavI1sgn6cVTzs= 2023-11-02T18:45:54.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nTotliVlfVOEDsavI1sgn6cVTzs=', 'Content-Length': '0'} 2023-11-02T18:45:54.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5879629fb0c413e-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.198 DEBUG:boto:path=/ibtljn-59/obj145 2023-11-02T18:45:54.199 DEBUG:boto:auth_path=/ibtljn-59/obj145 2023-11-02T18:45:54.200 DEBUG:boto:Method: GET 2023-11-02T18:45:54.200 DEBUG:boto:Path: /ibtljn-59/obj145 2023-11-02T18:45:54.201 DEBUG:boto:Data: 2023-11-02T18:45:54.201 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.202 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.203 DEBUG:boto:Params: {} 2023-11-02T18:45:54.203 DEBUG:boto:Token: None 2023-11-02T18:45:54.204 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj145 2023-11-02T18:45:54.204 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nTotliVlfVOEDsavI1sgn6cVTzs= 2023-11-02T18:45:54.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nTotliVlfVOEDsavI1sgn6cVTzs=', 'Content-Length': '0'} 2023-11-02T18:45:54.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016ccd06b65db9a9c-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.209 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.209 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.210 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:54.210 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:54.211 DEBUG:boto:Method: GET 2023-11-02T18:45:54.211 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:45:54.212 DEBUG:boto:Data: 2023-11-02T18:45:54.212 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.212 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.213 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.213 DEBUG:boto:Params: {} 2023-11-02T18:45:54.214 DEBUG:boto:Token: None 2023-11-02T18:45:54.214 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.215 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a293c8c49e774a40-006543ee62-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.221 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.223 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.223 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.223 DEBUG:boto:path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:54.224 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj145 2023-11-02T18:45:54.224 DEBUG:boto:Method: GET 2023-11-02T18:45:54.225 DEBUG:boto:Path: /ibtljn-59/?prefix=obj145 2023-11-02T18:45:54.226 DEBUG:boto:Data: 2023-11-02T18:45:54.226 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.227 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.227 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.228 DEBUG:boto:Params: {} 2023-11-02T18:45:54.228 DEBUG:boto:Token: None 2023-11-02T18:45:54.229 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.229 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8639e2e26bffd35-006543ee62-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.282 DEBUG:boto:b'ibtljn-59obj1451000falseobj1452023-11-02T18:40:31.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-11-02T18:45:54.284 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:54.284 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:54.285 DEBUG:boto:path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.285 DEBUG:boto:auth_path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.285 DEBUG:boto:Method: GET 2023-11-02T18:45:54.286 DEBUG:boto:Path: /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.286 DEBUG:boto:Data: 2023-11-02T18:45:54.287 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.287 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.288 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.288 DEBUG:boto:Params: {} 2023-11-02T18:45:54.289 DEBUG:boto:Token: None 2023-11-02T18:45:54.289 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.290 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1RmRhkVpV2mkT2OHq0XLY2YIFjY= 2023-11-02T18:45:54.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1RmRhkVpV2mkT2OHq0XLY2YIFjY=', 'Content-Length': '0'} 2023-11-02T18:45:54.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b158827d911044-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.294 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:54.295 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:54.295 DEBUG:boto:path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.296 DEBUG:boto:auth_path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.296 DEBUG:boto:Method: GET 2023-11-02T18:45:54.297 DEBUG:boto:Path: /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.297 DEBUG:boto:Data: 2023-11-02T18:45:54.298 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.298 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.298 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.299 DEBUG:boto:Params: {} 2023-11-02T18:45:54.299 DEBUG:boto:Token: None 2023-11-02T18:45:54.300 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.300 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1RmRhkVpV2mkT2OHq0XLY2YIFjY= 2023-11-02T18:45:54.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1RmRhkVpV2mkT2OHq0XLY2YIFjY=', 'Content-Length': '0'} 2023-11-02T18:45:54.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b22de2c47c50de41-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.305 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:54.305 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:54.305 DEBUG:boto:path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.306 DEBUG:boto:auth_path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.306 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.307 DEBUG:boto:Path: /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.307 DEBUG:boto:Data: 2023-11-02T18:45:54.308 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.308 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.308 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.309 DEBUG:boto:Params: {} 2023-11-02T18:45:54.309 DEBUG:boto:Token: None 2023-11-02T18:45:54.310 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.310 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LgYd7p+jLvx7RBRnWf/J41xJTO8= 2023-11-02T18:45:54.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LgYd7p+jLvx7RBRnWf/J41xJTO8=', 'Content-Length': '0'} 2023-11-02T18:45:54.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000044d6c195cb25049-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.315 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:54.315 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:54.315 DEBUG:boto:path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.316 DEBUG:boto:auth_path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.316 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.317 DEBUG:boto:Path: /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.317 DEBUG:boto:Data: 2023-11-02T18:45:54.318 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.318 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.318 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.319 DEBUG:boto:Params: {} 2023-11-02T18:45:54.319 DEBUG:boto:Token: None 2023-11-02T18:45:54.320 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LgYd7p+jLvx7RBRnWf/J41xJTO8= 2023-11-02T18:45:54.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LgYd7p+jLvx7RBRnWf/J41xJTO8=', 'Content-Length': '0'} 2023-11-02T18:45:54.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000795bb76b03608aa7-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-11-02T18:45:54.325 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:54.325 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:54.326 DEBUG:boto:path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.326 DEBUG:boto:auth_path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.326 DEBUG:boto:Method: GET 2023-11-02T18:45:54.327 DEBUG:boto:Path: /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.327 DEBUG:boto:Data: 2023-11-02T18:45:54.328 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.329 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.329 DEBUG:boto:Params: {} 2023-11-02T18:45:54.330 DEBUG:boto:Token: None 2023-11-02T18:45:54.330 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.331 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1RmRhkVpV2mkT2OHq0XLY2YIFjY= 2023-11-02T18:45:54.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1RmRhkVpV2mkT2OHq0XLY2YIFjY=', 'Content-Length': '0'} 2023-11-02T18:45:54.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dba636118bd15d5-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.335 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:54.336 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:54.336 DEBUG:boto:path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.337 DEBUG:boto:auth_path=/ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.337 DEBUG:boto:Method: GET 2023-11-02T18:45:54.338 DEBUG:boto:Path: /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.338 DEBUG:boto:Data: 2023-11-02T18:45:54.339 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.339 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.340 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.340 DEBUG:boto:Params: {} 2023-11-02T18:45:54.341 DEBUG:boto:Token: None 2023-11-02T18:45:54.341 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj148?versionId=null 2023-11-02T18:45:54.342 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1RmRhkVpV2mkT2OHq0XLY2YIFjY= 2023-11-02T18:45:54.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1RmRhkVpV2mkT2OHq0XLY2YIFjY=', 'Content-Length': '0'} 2023-11-02T18:45:54.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000800045efa3e01491-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.386 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.387 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.388 DEBUG:boto:path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:54.388 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:54.388 DEBUG:boto:Method: GET 2023-11-02T18:45:54.389 DEBUG:boto:Path: /ibtljn-59/?prefix=obj148 2023-11-02T18:45:54.389 DEBUG:boto:Data: 2023-11-02T18:45:54.390 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.390 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.391 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.391 DEBUG:boto:Params: {} 2023-11-02T18:45:54.392 DEBUG:boto:Token: None 2023-11-02T18:45:54.392 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.393 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9db1ed0ba1f5bc7-006543ee62-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.399 DEBUG:boto:b'ibtljn-59obj1481000falseobj1482023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.400 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.400 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.400 DEBUG:boto:path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:54.401 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:54.401 DEBUG:boto:Method: GET 2023-11-02T18:45:54.402 DEBUG:boto:Path: /ibtljn-59/?prefix=obj148 2023-11-02T18:45:54.402 DEBUG:boto:Data: 2023-11-02T18:45:54.403 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.403 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.404 DEBUG:boto:Params: {} 2023-11-02T18:45:54.404 DEBUG:boto:Token: None 2023-11-02T18:45:54.405 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.406 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081e942e06aa8458b-006543ee62-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.416 DEBUG:boto:b'ibtljn-59obj1481000falseobj1482023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.417 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:54.418 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:54.418 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.419 DEBUG:boto:Path: /ibtljn-59/obj148 2023-11-02T18:45:54.419 DEBUG:boto:Data: 2023-11-02T18:45:54.419 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.420 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.420 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.421 DEBUG:boto:Params: {} 2023-11-02T18:45:54.421 DEBUG:boto:Token: None 2023-11-02T18:45:54.422 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj148 2023-11-02T18:45:54.422 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zI4PJiBIXA5ZSCUDwoWd9iYrjHE= 2023-11-02T18:45:54.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zI4PJiBIXA5ZSCUDwoWd9iYrjHE=', 'Content-Length': '0'} 2023-11-02T18:45:54.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f038a8d66ee2f84-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.426 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:54.426 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:54.426 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.427 DEBUG:boto:Path: /ibtljn-59/obj148 2023-11-02T18:45:54.427 DEBUG:boto:Data: 2023-11-02T18:45:54.428 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.428 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.429 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.429 DEBUG:boto:Params: {} 2023-11-02T18:45:54.429 DEBUG:boto:Token: None 2023-11-02T18:45:54.430 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj148 2023-11-02T18:45:54.431 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zI4PJiBIXA5ZSCUDwoWd9iYrjHE= 2023-11-02T18:45:54.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zI4PJiBIXA5ZSCUDwoWd9iYrjHE=', 'Content-Length': '0'} 2023-11-02T18:45:54.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d95c73a123e880a1-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-11-02T18:45:54.435 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:54.435 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:54.436 DEBUG:boto:Method: GET 2023-11-02T18:45:54.436 DEBUG:boto:Path: /ibtljn-59/obj148 2023-11-02T18:45:54.436 DEBUG:boto:Data: 2023-11-02T18:45:54.437 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.438 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.438 DEBUG:boto:Params: {} 2023-11-02T18:45:54.439 DEBUG:boto:Token: None 2023-11-02T18:45:54.440 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj148 2023-11-02T18:45:54.441 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z1pulSubjrH2I3cUhHxXB6ZZ/os= 2023-11-02T18:45:54.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z1pulSubjrH2I3cUhHxXB6ZZ/os=', 'Content-Length': '0'} 2023-11-02T18:45:54.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a59b93507b5d7dd4-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.445 DEBUG:boto:path=/ibtljn-59/obj148 2023-11-02T18:45:54.445 DEBUG:boto:auth_path=/ibtljn-59/obj148 2023-11-02T18:45:54.446 DEBUG:boto:Method: GET 2023-11-02T18:45:54.446 DEBUG:boto:Path: /ibtljn-59/obj148 2023-11-02T18:45:54.446 DEBUG:boto:Data: 2023-11-02T18:45:54.447 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.448 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.448 DEBUG:boto:Params: {} 2023-11-02T18:45:54.449 DEBUG:boto:Token: None 2023-11-02T18:45:54.449 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj148 2023-11-02T18:45:54.450 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z1pulSubjrH2I3cUhHxXB6ZZ/os= 2023-11-02T18:45:54.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z1pulSubjrH2I3cUhHxXB6ZZ/os=', 'Content-Length': '0'} 2023-11-02T18:45:54.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c096c0efd051a9c3-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.453 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.454 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.454 DEBUG:boto:path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:54.455 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:54.455 DEBUG:boto:Method: GET 2023-11-02T18:45:54.456 DEBUG:boto:Path: /ibtljn-59/?prefix=obj148 2023-11-02T18:45:54.456 DEBUG:boto:Data: 2023-11-02T18:45:54.456 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.457 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.457 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.458 DEBUG:boto:Params: {} 2023-11-02T18:45:54.458 DEBUG:boto:Token: None 2023-11-02T18:45:54.459 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.459 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7f72acaa85c102a-006543ee62-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.510 DEBUG:boto:b'ibtljn-59obj1481000falseobj1482023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.511 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.512 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.512 DEBUG:boto:path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:54.513 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj148 2023-11-02T18:45:54.513 DEBUG:boto:Method: GET 2023-11-02T18:45:54.514 DEBUG:boto:Path: /ibtljn-59/?prefix=obj148 2023-11-02T18:45:54.514 DEBUG:boto:Data: 2023-11-02T18:45:54.515 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.515 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.516 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.516 DEBUG:boto:Params: {} 2023-11-02T18:45:54.517 DEBUG:boto:Token: None 2023-11-02T18:45:54.517 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.518 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e225caf4b5f06fa-006543ee62-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.528 DEBUG:boto:b'ibtljn-59obj1481000falseobj1482023-11-02T18:44:13.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T18:45:54.529 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:54.530 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:54.530 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.531 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.531 DEBUG:boto:Method: GET 2023-11-02T18:45:54.532 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.532 DEBUG:boto:Data: 2023-11-02T18:45:54.532 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.533 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.533 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.534 DEBUG:boto:Params: {} 2023-11-02T18:45:54.534 DEBUG:boto:Token: None 2023-11-02T18:45:54.535 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.535 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CyqoLv32VB3rUvx1qSw6WjY18o8= 2023-11-02T18:45:54.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CyqoLv32VB3rUvx1qSw6WjY18o8=', 'Content-Length': '0'} 2023-11-02T18:45:54.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006558c9883acc8982-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.541 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:54.541 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:54.542 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.542 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.543 DEBUG:boto:Method: GET 2023-11-02T18:45:54.543 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.544 DEBUG:boto:Data: 2023-11-02T18:45:54.544 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.545 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.545 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.546 DEBUG:boto:Params: {} 2023-11-02T18:45:54.546 DEBUG:boto:Token: None 2023-11-02T18:45:54.547 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.547 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CyqoLv32VB3rUvx1qSw6WjY18o8= 2023-11-02T18:45:54.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CyqoLv32VB3rUvx1qSw6WjY18o8=', 'Content-Length': '0'} 2023-11-02T18:45:54.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c3c3f0fddb1c603-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.594 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:54.595 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:54.595 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.596 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.596 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.597 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.597 DEBUG:boto:Data: 2023-11-02T18:45:54.598 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.598 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.598 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.599 DEBUG:boto:Params: {} 2023-11-02T18:45:54.599 DEBUG:boto:Token: None 2023-11-02T18:45:54.600 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.600 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XR1iUxfU+DWl7dY6jpL8OUfWSrg= 2023-11-02T18:45:54.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XR1iUxfU+DWl7dY6jpL8OUfWSrg=', 'Content-Length': '0'} 2023-11-02T18:45:54.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088e45bc4ecfaade7-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.604 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:54.605 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:54.605 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.606 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.606 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.606 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.607 DEBUG:boto:Data: 2023-11-02T18:45:54.607 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.608 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.608 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.608 DEBUG:boto:Params: {} 2023-11-02T18:45:54.609 DEBUG:boto:Token: None 2023-11-02T18:45:54.609 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.610 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XR1iUxfU+DWl7dY6jpL8OUfWSrg= 2023-11-02T18:45:54.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XR1iUxfU+DWl7dY6jpL8OUfWSrg=', 'Content-Length': '0'} 2023-11-02T18:45:54.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f392328638296f1-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T18:45:54.615 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:54.615 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:54.616 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.616 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.616 DEBUG:boto:Method: GET 2023-11-02T18:45:54.617 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.617 DEBUG:boto:Data: 2023-11-02T18:45:54.618 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.618 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.619 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.619 DEBUG:boto:Params: {} 2023-11-02T18:45:54.620 DEBUG:boto:Token: None 2023-11-02T18:45:54.620 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CyqoLv32VB3rUvx1qSw6WjY18o8= 2023-11-02T18:45:54.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CyqoLv32VB3rUvx1qSw6WjY18o8=', 'Content-Length': '0'} 2023-11-02T18:45:54.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045daeb6e37c2a60b-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.625 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:54.625 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:54.626 DEBUG:boto:path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.626 DEBUG:boto:auth_path=/ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.626 DEBUG:boto:Method: GET 2023-11-02T18:45:54.627 DEBUG:boto:Path: /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.627 DEBUG:boto:Data: 2023-11-02T18:45:54.628 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.628 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.629 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.629 DEBUG:boto:Params: {} 2023-11-02T18:45:54.629 DEBUG:boto:Token: None 2023-11-02T18:45:54.630 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj155?versionId=null 2023-11-02T18:45:54.630 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CyqoLv32VB3rUvx1qSw6WjY18o8= 2023-11-02T18:45:54.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CyqoLv32VB3rUvx1qSw6WjY18o8=', 'Content-Length': '0'} 2023-11-02T18:45:54.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c758e6a41dcbda9-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.634 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.635 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.635 DEBUG:boto:path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:54.636 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:54.636 DEBUG:boto:Method: GET 2023-11-02T18:45:54.637 DEBUG:boto:Path: /ibtljn-59/?prefix=obj155 2023-11-02T18:45:54.637 DEBUG:boto:Data: 2023-11-02T18:45:54.638 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.638 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.638 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.639 DEBUG:boto:Params: {} 2023-11-02T18:45:54.639 DEBUG:boto:Token: None 2023-11-02T18:45:54.640 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.640 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f2c63fb39fefad3-006543ee62-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.686 DEBUG:boto:b'ibtljn-59obj1551000falseobj1552023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.687 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.688 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.688 DEBUG:boto:path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:54.689 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:54.689 DEBUG:boto:Method: GET 2023-11-02T18:45:54.689 DEBUG:boto:Path: /ibtljn-59/?prefix=obj155 2023-11-02T18:45:54.690 DEBUG:boto:Data: 2023-11-02T18:45:54.690 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.691 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.691 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.692 DEBUG:boto:Params: {} 2023-11-02T18:45:54.692 DEBUG:boto:Token: None 2023-11-02T18:45:54.693 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.693 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ba6f40ced279df5-006543ee62-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.705 DEBUG:boto:b'ibtljn-59obj1551000falseobj1552023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.705 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:54.706 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:54.706 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.707 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:45:54.707 DEBUG:boto:Data: 2023-11-02T18:45:54.708 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.708 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.709 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.709 DEBUG:boto:Params: {} 2023-11-02T18:45:54.709 DEBUG:boto:Token: None 2023-11-02T18:45:54.710 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj155 2023-11-02T18:45:54.710 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EKLSsmx6rrLQRHyBoV9KLFpPDU4= 2023-11-02T18:45:54.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EKLSsmx6rrLQRHyBoV9KLFpPDU4=', 'Content-Length': '0'} 2023-11-02T18:45:54.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004195700ded8a9644-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.714 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:54.715 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:54.715 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.716 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:45:54.716 DEBUG:boto:Data: 2023-11-02T18:45:54.717 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.717 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.718 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.718 DEBUG:boto:Params: {} 2023-11-02T18:45:54.719 DEBUG:boto:Token: None 2023-11-02T18:45:54.719 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj155 2023-11-02T18:45:54.720 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EKLSsmx6rrLQRHyBoV9KLFpPDU4= 2023-11-02T18:45:54.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EKLSsmx6rrLQRHyBoV9KLFpPDU4=', 'Content-Length': '0'} 2023-11-02T18:45:54.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067eea6f8aa6f202e-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T18:45:54.724 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:54.724 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:54.725 DEBUG:boto:Method: GET 2023-11-02T18:45:54.725 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:45:54.726 DEBUG:boto:Data: 2023-11-02T18:45:54.726 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.726 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.727 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.727 DEBUG:boto:Params: {} 2023-11-02T18:45:54.728 DEBUG:boto:Token: None 2023-11-02T18:45:54.728 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj155 2023-11-02T18:45:54.729 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c0SZpvMWmGxL6tO7id7IDGtJYXE= 2023-11-02T18:45:54.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c0SZpvMWmGxL6tO7id7IDGtJYXE=', 'Content-Length': '0'} 2023-11-02T18:45:54.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fec9a6ccd6acce7-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.733 DEBUG:boto:path=/ibtljn-59/obj155 2023-11-02T18:45:54.733 DEBUG:boto:auth_path=/ibtljn-59/obj155 2023-11-02T18:45:54.734 DEBUG:boto:Method: GET 2023-11-02T18:45:54.734 DEBUG:boto:Path: /ibtljn-59/obj155 2023-11-02T18:45:54.735 DEBUG:boto:Data: 2023-11-02T18:45:54.735 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.735 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.736 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.736 DEBUG:boto:Params: {} 2023-11-02T18:45:54.737 DEBUG:boto:Token: None 2023-11-02T18:45:54.738 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj155 2023-11-02T18:45:54.738 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c0SZpvMWmGxL6tO7id7IDGtJYXE= 2023-11-02T18:45:54.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c0SZpvMWmGxL6tO7id7IDGtJYXE=', 'Content-Length': '0'} 2023-11-02T18:45:54.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b40fca7aa12ffccb-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.786 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.787 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.788 DEBUG:boto:path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:54.788 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:54.789 DEBUG:boto:Method: GET 2023-11-02T18:45:54.789 DEBUG:boto:Path: /ibtljn-59/?prefix=obj155 2023-11-02T18:45:54.789 DEBUG:boto:Data: 2023-11-02T18:45:54.790 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.791 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.791 DEBUG:boto:Params: {} 2023-11-02T18:45:54.792 DEBUG:boto:Token: None 2023-11-02T18:45:54.792 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005508cec4165b6e79-006543ee62-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.799 DEBUG:boto:b'ibtljn-59obj1551000falseobj1552023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.800 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.801 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.801 DEBUG:boto:path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:54.802 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj155 2023-11-02T18:45:54.802 DEBUG:boto:Method: GET 2023-11-02T18:45:54.803 DEBUG:boto:Path: /ibtljn-59/?prefix=obj155 2023-11-02T18:45:54.803 DEBUG:boto:Data: 2023-11-02T18:45:54.804 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.804 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.804 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.805 DEBUG:boto:Params: {} 2023-11-02T18:45:54.805 DEBUG:boto:Token: None 2023-11-02T18:45:54.806 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.807 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000469031192ecf4b33-006543ee62-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.819 DEBUG:boto:b'ibtljn-59obj1551000falseobj1552023-11-02T18:42:34.469Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:45:54.820 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:54.820 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:54.821 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.821 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.822 DEBUG:boto:Method: GET 2023-11-02T18:45:54.822 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.823 DEBUG:boto:Data: 2023-11-02T18:45:54.823 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.823 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.824 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.824 DEBUG:boto:Params: {} 2023-11-02T18:45:54.825 DEBUG:boto:Token: None 2023-11-02T18:45:54.825 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.826 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZtRgQp/iXGnr+/UKWOsKGvEQIUU= 2023-11-02T18:45:54.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZtRgQp/iXGnr+/UKWOsKGvEQIUU=', 'Content-Length': '0'} 2023-11-02T18:45:54.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000096cccffe65a8d51-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.830 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:54.831 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:54.831 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.832 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.832 DEBUG:boto:Method: GET 2023-11-02T18:45:54.832 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.833 DEBUG:boto:Data: 2023-11-02T18:45:54.833 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.834 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.834 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.835 DEBUG:boto:Params: {} 2023-11-02T18:45:54.835 DEBUG:boto:Token: None 2023-11-02T18:45:54.836 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.836 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZtRgQp/iXGnr+/UKWOsKGvEQIUU= 2023-11-02T18:45:54.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZtRgQp/iXGnr+/UKWOsKGvEQIUU=', 'Content-Length': '0'} 2023-11-02T18:45:54.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070cdb4bcdef4b195-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.840 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:54.841 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:54.841 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.842 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.842 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.843 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.843 DEBUG:boto:Data: 2023-11-02T18:45:54.843 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.844 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.844 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.845 DEBUG:boto:Params: {} 2023-11-02T18:45:54.845 DEBUG:boto:Token: None 2023-11-02T18:45:54.846 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.846 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d6vvThH3M/Q8w0HdUAhBc5Ep3pA= 2023-11-02T18:45:54.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d6vvThH3M/Q8w0HdUAhBc5Ep3pA=', 'Content-Length': '0'} 2023-11-02T18:45:54.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f415841210d12fb4-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.850 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:54.851 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:54.851 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.851 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.852 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.852 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.853 DEBUG:boto:Data: 2023-11-02T18:45:54.853 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.853 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.854 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.854 DEBUG:boto:Params: {} 2023-11-02T18:45:54.855 DEBUG:boto:Token: None 2023-11-02T18:45:54.855 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.856 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d6vvThH3M/Q8w0HdUAhBc5Ep3pA= 2023-11-02T18:45:54.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d6vvThH3M/Q8w0HdUAhBc5Ep3pA=', 'Content-Length': '0'} 2023-11-02T18:45:54.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8468cbfde6ec04d-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:45:54.861 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:54.861 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:54.862 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.862 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.862 DEBUG:boto:Method: GET 2023-11-02T18:45:54.863 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.863 DEBUG:boto:Data: 2023-11-02T18:45:54.864 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.864 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.865 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.865 DEBUG:boto:Params: {} 2023-11-02T18:45:54.866 DEBUG:boto:Token: None 2023-11-02T18:45:54.866 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.867 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZtRgQp/iXGnr+/UKWOsKGvEQIUU= 2023-11-02T18:45:54.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZtRgQp/iXGnr+/UKWOsKGvEQIUU=', 'Content-Length': '0'} 2023-11-02T18:45:54.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028c6f6019969603-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.871 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:54.872 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:54.872 DEBUG:boto:path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.873 DEBUG:boto:auth_path=/ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.873 DEBUG:boto:Method: GET 2023-11-02T18:45:54.873 DEBUG:boto:Path: /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.874 DEBUG:boto:Data: 2023-11-02T18:45:54.874 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.875 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.875 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.876 DEBUG:boto:Params: {} 2023-11-02T18:45:54.876 DEBUG:boto:Token: None 2023-11-02T18:45:54.877 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj17?versionId=null 2023-11-02T18:45:54.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZtRgQp/iXGnr+/UKWOsKGvEQIUU= 2023-11-02T18:45:54.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZtRgQp/iXGnr+/UKWOsKGvEQIUU=', 'Content-Length': '0'} 2023-11-02T18:45:54.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f3c08e7ffea0f38-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.922 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.923 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.923 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:54.924 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:54.924 DEBUG:boto:Method: GET 2023-11-02T18:45:54.925 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:45:54.926 DEBUG:boto:Data: 2023-11-02T18:45:54.926 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.927 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.927 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.927 DEBUG:boto:Params: {} 2023-11-02T18:45:54.928 DEBUG:boto:Token: None 2023-11-02T18:45:54.928 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.929 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000703afad99f6b8abb-006543ee62-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.936 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.937 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:54.937 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:54.938 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:54.938 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:54.939 DEBUG:boto:Method: GET 2023-11-02T18:45:54.939 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:45:54.940 DEBUG:boto:Data: 2023-11-02T18:45:54.940 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.941 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.941 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.942 DEBUG:boto:Params: {} 2023-11-02T18:45:54.942 DEBUG:boto:Token: None 2023-11-02T18:45:54.943 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/ 2023-11-02T18:45:54.943 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI= 2023-11-02T18:45:54.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTiY5zsvHCh/bpJk5cnLKcx2cPI=', 'Content-Length': '0'} 2023-11-02T18:45:54.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007260b3718bc8b0df-006543ee62-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.955 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:54.955 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:54.956 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:54.956 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.957 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:45:54.957 DEBUG:boto:Data: 2023-11-02T18:45:54.958 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.958 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.958 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.959 DEBUG:boto:Params: {} 2023-11-02T18:45:54.959 DEBUG:boto:Token: None 2023-11-02T18:45:54.960 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj17 2023-11-02T18:45:54.960 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OMfxMhOyaATMx5NgxkHVL7P4ooM= 2023-11-02T18:45:54.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OMfxMhOyaATMx5NgxkHVL7P4ooM=', 'Content-Length': '0'} 2023-11-02T18:45:54.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009fbc3a8b7e61418-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.964 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:54.964 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:54.965 DEBUG:boto:Method: HEAD 2023-11-02T18:45:54.965 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:45:54.965 DEBUG:boto:Data: 2023-11-02T18:45:54.966 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.966 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:54.967 DEBUG:boto:Port: 8000 2023-11-02T18:45:54.967 DEBUG:boto:Params: {} 2023-11-02T18:45:54.968 DEBUG:boto:Token: None 2023-11-02T18:45:54.968 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj17 2023-11-02T18:45:54.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OMfxMhOyaATMx5NgxkHVL7P4ooM= 2023-11-02T18:45:54.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OMfxMhOyaATMx5NgxkHVL7P4ooM=', 'Content-Length': '0'} 2023-11-02T18:45:54.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3c5532450982994-006543ee62-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:54.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:45:54.973 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:54.973 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:54.974 DEBUG:boto:Method: GET 2023-11-02T18:45:54.974 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:45:54.974 DEBUG:boto:Data: 2023-11-02T18:45:54.975 DEBUG:boto:Headers: {} 2023-11-02T18:45:54.975 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:54.976 DEBUG:boto:Port: 8001 2023-11-02T18:45:54.976 DEBUG:boto:Params: {} 2023-11-02T18:45:54.977 DEBUG:boto:Token: None 2023-11-02T18:45:54.977 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:54 GMT /ibtljn-59/obj17 2023-11-02T18:45:54.978 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y+x3SI2wJt9v4kqf2K86n4QWYz4= 2023-11-02T18:45:54.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y+x3SI2wJt9v4kqf2K86n4QWYz4=', 'Content-Length': '0'} 2023-11-02T18:45:54.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ac26787372da75c-006543ee62-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.022 DEBUG:boto:path=/ibtljn-59/obj17 2023-11-02T18:45:55.023 DEBUG:boto:auth_path=/ibtljn-59/obj17 2023-11-02T18:45:55.023 DEBUG:boto:Method: GET 2023-11-02T18:45:55.024 DEBUG:boto:Path: /ibtljn-59/obj17 2023-11-02T18:45:55.024 DEBUG:boto:Data: 2023-11-02T18:45:55.025 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.025 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.026 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.026 DEBUG:boto:Params: {} 2023-11-02T18:45:55.027 DEBUG:boto:Token: None 2023-11-02T18:45:55.027 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj17 2023-11-02T18:45:55.028 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zk3/kAZkvyzrpgvvgo1HLItmzNE= 2023-11-02T18:45:55.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zk3/kAZkvyzrpgvvgo1HLItmzNE=', 'Content-Length': '0'} 2023-11-02T18:45:55.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000193a26858c85ce9d-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.032 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.033 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.033 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:55.034 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:55.035 DEBUG:boto:Method: GET 2023-11-02T18:45:55.035 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:45:55.036 DEBUG:boto:Data: 2023-11-02T18:45:55.036 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.036 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.037 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.037 DEBUG:boto:Params: {} 2023-11-02T18:45:55.038 DEBUG:boto:Token: None 2023-11-02T18:45:55.038 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.039 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba94506d540a14b9-006543ee63-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.046 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.047 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.047 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.047 DEBUG:boto:path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:55.048 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj17 2023-11-02T18:45:55.048 DEBUG:boto:Method: GET 2023-11-02T18:45:55.049 DEBUG:boto:Path: /ibtljn-59/?prefix=obj17 2023-11-02T18:45:55.049 DEBUG:boto:Data: 2023-11-02T18:45:55.050 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.051 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.051 DEBUG:boto:Params: {} 2023-11-02T18:45:55.051 DEBUG:boto:Token: None 2023-11-02T18:45:55.052 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.052 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb5b285eee05a98a-006543ee63-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.106 DEBUG:boto:b'ibtljn-59obj171000falseobj172023-11-02T18:32:46.738Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-11-02T18:45:55.108 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:55.108 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:55.108 DEBUG:boto:path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.109 DEBUG:boto:auth_path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.109 DEBUG:boto:Method: GET 2023-11-02T18:45:55.110 DEBUG:boto:Path: /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.110 DEBUG:boto:Data: 2023-11-02T18:45:55.111 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.111 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.112 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.112 DEBUG:boto:Params: {} 2023-11-02T18:45:55.112 DEBUG:boto:Token: None 2023-11-02T18:45:55.113 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.113 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/SfAAQNiLBCneUPZ6D6QirRtqVQ= 2023-11-02T18:45:55.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/SfAAQNiLBCneUPZ6D6QirRtqVQ=', 'Content-Length': '0'} 2023-11-02T18:45:55.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f9c349eb21b6ea-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.118 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:55.119 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:55.119 DEBUG:boto:path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.120 DEBUG:boto:auth_path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.120 DEBUG:boto:Method: GET 2023-11-02T18:45:55.121 DEBUG:boto:Path: /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.121 DEBUG:boto:Data: 2023-11-02T18:45:55.121 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.122 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.122 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.123 DEBUG:boto:Params: {} 2023-11-02T18:45:55.123 DEBUG:boto:Token: None 2023-11-02T18:45:55.124 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.124 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/SfAAQNiLBCneUPZ6D6QirRtqVQ= 2023-11-02T18:45:55.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/SfAAQNiLBCneUPZ6D6QirRtqVQ=', 'Content-Length': '0'} 2023-11-02T18:45:55.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1400e0763b55c34-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.128 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:55.129 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:55.129 DEBUG:boto:path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.130 DEBUG:boto:auth_path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.130 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.130 DEBUG:boto:Path: /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.131 DEBUG:boto:Data: 2023-11-02T18:45:55.131 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.132 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.132 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.132 DEBUG:boto:Params: {} 2023-11-02T18:45:55.133 DEBUG:boto:Token: None 2023-11-02T18:45:55.133 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.134 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ao8O9DnSJ6nDDP0tTGSD8nsYXXo= 2023-11-02T18:45:55.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ao8O9DnSJ6nDDP0tTGSD8nsYXXo=', 'Content-Length': '0'} 2023-11-02T18:45:55.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000245c3133afd43bfa-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.138 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:55.138 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:55.139 DEBUG:boto:path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.139 DEBUG:boto:auth_path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.139 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.140 DEBUG:boto:Path: /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.140 DEBUG:boto:Data: 2023-11-02T18:45:55.141 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.141 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.141 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.142 DEBUG:boto:Params: {} 2023-11-02T18:45:55.142 DEBUG:boto:Token: None 2023-11-02T18:45:55.143 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.143 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ao8O9DnSJ6nDDP0tTGSD8nsYXXo= 2023-11-02T18:45:55.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ao8O9DnSJ6nDDP0tTGSD8nsYXXo=', 'Content-Length': '0'} 2023-11-02T18:45:55.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2992386fd762500-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-11-02T18:45:55.147 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:55.148 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:55.148 DEBUG:boto:path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.149 DEBUG:boto:auth_path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.149 DEBUG:boto:Method: GET 2023-11-02T18:45:55.149 DEBUG:boto:Path: /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.150 DEBUG:boto:Data: 2023-11-02T18:45:55.150 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.151 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.151 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.152 DEBUG:boto:Params: {} 2023-11-02T18:45:55.152 DEBUG:boto:Token: None 2023-11-02T18:45:55.153 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.153 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/SfAAQNiLBCneUPZ6D6QirRtqVQ= 2023-11-02T18:45:55.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/SfAAQNiLBCneUPZ6D6QirRtqVQ=', 'Content-Length': '0'} 2023-11-02T18:45:55.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000127de6554542a9d5-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.158 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:55.158 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:55.158 DEBUG:boto:path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.159 DEBUG:boto:auth_path=/ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.159 DEBUG:boto:Method: GET 2023-11-02T18:45:55.160 DEBUG:boto:Path: /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.160 DEBUG:boto:Data: 2023-11-02T18:45:55.161 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.161 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.162 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.162 DEBUG:boto:Params: {} 2023-11-02T18:45:55.162 DEBUG:boto:Token: None 2023-11-02T18:45:55.163 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj185?versionId=null 2023-11-02T18:45:55.163 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/SfAAQNiLBCneUPZ6D6QirRtqVQ= 2023-11-02T18:45:55.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/SfAAQNiLBCneUPZ6D6QirRtqVQ=', 'Content-Length': '0'} 2023-11-02T18:45:55.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086c1891efd1ed5ba-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.210 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.211 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.211 DEBUG:boto:path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:55.212 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:55.212 DEBUG:boto:Method: GET 2023-11-02T18:45:55.213 DEBUG:boto:Path: /ibtljn-59/?prefix=obj185 2023-11-02T18:45:55.213 DEBUG:boto:Data: 2023-11-02T18:45:55.214 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.214 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.215 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.215 DEBUG:boto:Params: {} 2023-11-02T18:45:55.215 DEBUG:boto:Token: None 2023-11-02T18:45:55.216 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.216 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e5858eabb0ab82d-006543ee63-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.222 DEBUG:boto:b'ibtljn-59obj1851000falseobj1852023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.223 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.224 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.224 DEBUG:boto:path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:55.224 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:55.225 DEBUG:boto:Method: GET 2023-11-02T18:45:55.225 DEBUG:boto:Path: /ibtljn-59/?prefix=obj185 2023-11-02T18:45:55.226 DEBUG:boto:Data: 2023-11-02T18:45:55.226 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.227 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.227 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.227 DEBUG:boto:Params: {} 2023-11-02T18:45:55.228 DEBUG:boto:Token: None 2023-11-02T18:45:55.228 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.229 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d29eca1a2355220b-006543ee63-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.240 DEBUG:boto:b'ibtljn-59obj1851000falseobj1852023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.241 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:55.241 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:55.242 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.242 DEBUG:boto:Path: /ibtljn-59/obj185 2023-11-02T18:45:55.243 DEBUG:boto:Data: 2023-11-02T18:45:55.243 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.243 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.244 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.245 DEBUG:boto:Params: {} 2023-11-02T18:45:55.245 DEBUG:boto:Token: None 2023-11-02T18:45:55.245 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj185 2023-11-02T18:45:55.246 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CFpTliFFRS3diwC6m2I0bH0FDwU= 2023-11-02T18:45:55.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CFpTliFFRS3diwC6m2I0bH0FDwU=', 'Content-Length': '0'} 2023-11-02T18:45:55.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d11a62bcf23e25e-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.249 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:55.250 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:55.250 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.251 DEBUG:boto:Path: /ibtljn-59/obj185 2023-11-02T18:45:55.251 DEBUG:boto:Data: 2023-11-02T18:45:55.252 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.252 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.252 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.253 DEBUG:boto:Params: {} 2023-11-02T18:45:55.253 DEBUG:boto:Token: None 2023-11-02T18:45:55.254 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj185 2023-11-02T18:45:55.254 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CFpTliFFRS3diwC6m2I0bH0FDwU= 2023-11-02T18:45:55.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CFpTliFFRS3diwC6m2I0bH0FDwU=', 'Content-Length': '0'} 2023-11-02T18:45:55.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006907c5b8d66e44ce-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-11-02T18:45:55.258 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:55.259 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:55.259 DEBUG:boto:Method: GET 2023-11-02T18:45:55.260 DEBUG:boto:Path: /ibtljn-59/obj185 2023-11-02T18:45:55.260 DEBUG:boto:Data: 2023-11-02T18:45:55.261 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.261 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.262 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.262 DEBUG:boto:Params: {} 2023-11-02T18:45:55.263 DEBUG:boto:Token: None 2023-11-02T18:45:55.263 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj185 2023-11-02T18:45:55.264 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cwHbGJAJ4HBoOZsTBX+1uBrrXhs= 2023-11-02T18:45:55.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cwHbGJAJ4HBoOZsTBX+1uBrrXhs=', 'Content-Length': '0'} 2023-11-02T18:45:55.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db5fc4cdeb67bf0e-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.268 DEBUG:boto:path=/ibtljn-59/obj185 2023-11-02T18:45:55.268 DEBUG:boto:auth_path=/ibtljn-59/obj185 2023-11-02T18:45:55.268 DEBUG:boto:Method: GET 2023-11-02T18:45:55.269 DEBUG:boto:Path: /ibtljn-59/obj185 2023-11-02T18:45:55.269 DEBUG:boto:Data: 2023-11-02T18:45:55.270 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.270 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.271 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.271 DEBUG:boto:Params: {} 2023-11-02T18:45:55.272 DEBUG:boto:Token: None 2023-11-02T18:45:55.272 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj185 2023-11-02T18:45:55.273 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cwHbGJAJ4HBoOZsTBX+1uBrrXhs= 2023-11-02T18:45:55.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cwHbGJAJ4HBoOZsTBX+1uBrrXhs=', 'Content-Length': '0'} 2023-11-02T18:45:55.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000403b4dd9e9fc15d9-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.276 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.277 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.277 DEBUG:boto:path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:55.278 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:55.278 DEBUG:boto:Method: GET 2023-11-02T18:45:55.278 DEBUG:boto:Path: /ibtljn-59/?prefix=obj185 2023-11-02T18:45:55.279 DEBUG:boto:Data: 2023-11-02T18:45:55.279 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.280 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.280 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.281 DEBUG:boto:Params: {} 2023-11-02T18:45:55.281 DEBUG:boto:Token: None 2023-11-02T18:45:55.282 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.282 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb657eeebb4e1d8f-006543ee63-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.330 DEBUG:boto:b'ibtljn-59obj1851000falseobj1852023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.331 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.332 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.332 DEBUG:boto:path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:55.332 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj185 2023-11-02T18:45:55.333 DEBUG:boto:Method: GET 2023-11-02T18:45:55.333 DEBUG:boto:Path: /ibtljn-59/?prefix=obj185 2023-11-02T18:45:55.334 DEBUG:boto:Data: 2023-11-02T18:45:55.334 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.335 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.335 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.335 DEBUG:boto:Params: {} 2023-11-02T18:45:55.336 DEBUG:boto:Token: None 2023-11-02T18:45:55.336 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.337 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002423754b1745f10b-006543ee63-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.348 DEBUG:boto:b'ibtljn-59obj1851000falseobj1852023-11-02T18:44:13.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:45:55.349 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:55.350 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:55.350 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.350 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.351 DEBUG:boto:Method: GET 2023-11-02T18:45:55.351 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.352 DEBUG:boto:Data: 2023-11-02T18:45:55.352 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.353 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.353 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.354 DEBUG:boto:Params: {} 2023-11-02T18:45:55.354 DEBUG:boto:Token: None 2023-11-02T18:45:55.355 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.355 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zG+dZQs7Xj65CVT5zQ4skiVoeew= 2023-11-02T18:45:55.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zG+dZQs7Xj65CVT5zQ4skiVoeew=', 'Content-Length': '0'} 2023-11-02T18:45:55.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009559778374073b3f-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.359 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:55.359 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:55.360 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.360 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.361 DEBUG:boto:Method: GET 2023-11-02T18:45:55.361 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.362 DEBUG:boto:Data: 2023-11-02T18:45:55.362 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.362 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.363 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.363 DEBUG:boto:Params: {} 2023-11-02T18:45:55.364 DEBUG:boto:Token: None 2023-11-02T18:45:55.364 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.365 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zG+dZQs7Xj65CVT5zQ4skiVoeew= 2023-11-02T18:45:55.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zG+dZQs7Xj65CVT5zQ4skiVoeew=', 'Content-Length': '0'} 2023-11-02T18:45:55.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055415fa798d9baf6-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.410 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:55.411 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:55.411 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.413 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.413 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.414 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.414 DEBUG:boto:Data: 2023-11-02T18:45:55.415 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.415 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.415 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.416 DEBUG:boto:Params: {} 2023-11-02T18:45:55.416 DEBUG:boto:Token: None 2023-11-02T18:45:55.417 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.417 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wnrV7IcFfYVKX1EvaBO6vmUppOE= 2023-11-02T18:45:55.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wnrV7IcFfYVKX1EvaBO6vmUppOE=', 'Content-Length': '0'} 2023-11-02T18:45:55.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f84e2e3f4847adb0-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.422 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:55.422 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:55.423 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.423 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.423 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.424 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.424 DEBUG:boto:Data: 2023-11-02T18:45:55.425 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.425 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.426 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.426 DEBUG:boto:Params: {} 2023-11-02T18:45:55.427 DEBUG:boto:Token: None 2023-11-02T18:45:55.427 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.427 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wnrV7IcFfYVKX1EvaBO6vmUppOE= 2023-11-02T18:45:55.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wnrV7IcFfYVKX1EvaBO6vmUppOE=', 'Content-Length': '0'} 2023-11-02T18:45:55.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0da8982f38c8b76-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:45:55.432 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:55.432 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:55.433 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.433 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.434 DEBUG:boto:Method: GET 2023-11-02T18:45:55.434 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.435 DEBUG:boto:Data: 2023-11-02T18:45:55.435 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.436 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.436 DEBUG:boto:Params: {} 2023-11-02T18:45:55.437 DEBUG:boto:Token: None 2023-11-02T18:45:55.437 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.438 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zG+dZQs7Xj65CVT5zQ4skiVoeew= 2023-11-02T18:45:55.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zG+dZQs7Xj65CVT5zQ4skiVoeew=', 'Content-Length': '0'} 2023-11-02T18:45:55.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009855c377ebd3507c-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.442 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:55.443 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:55.443 DEBUG:boto:path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.444 DEBUG:boto:auth_path=/ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.444 DEBUG:boto:Method: GET 2023-11-02T18:45:55.444 DEBUG:boto:Path: /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.445 DEBUG:boto:Data: 2023-11-02T18:45:55.445 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.446 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.446 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.447 DEBUG:boto:Params: {} 2023-11-02T18:45:55.447 DEBUG:boto:Token: None 2023-11-02T18:45:55.448 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj19?versionId=null 2023-11-02T18:45:55.448 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zG+dZQs7Xj65CVT5zQ4skiVoeew= 2023-11-02T18:45:55.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zG+dZQs7Xj65CVT5zQ4skiVoeew=', 'Content-Length': '0'} 2023-11-02T18:45:55.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e676c20be3a3b85a-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.452 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.453 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.453 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:55.454 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:55.454 DEBUG:boto:Method: GET 2023-11-02T18:45:55.455 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:45:55.455 DEBUG:boto:Data: 2023-11-02T18:45:55.455 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.456 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.456 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.457 DEBUG:boto:Params: {} 2023-11-02T18:45:55.457 DEBUG:boto:Token: None 2023-11-02T18:45:55.458 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.458 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1777fe4f300f103-006543ee63-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.464 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.465 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.465 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.466 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:55.466 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:55.467 DEBUG:boto:Method: GET 2023-11-02T18:45:55.467 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:45:55.467 DEBUG:boto:Data: 2023-11-02T18:45:55.468 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.468 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.469 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.469 DEBUG:boto:Params: {} 2023-11-02T18:45:55.470 DEBUG:boto:Token: None 2023-11-02T18:45:55.470 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.471 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4683bb5dcb1d8ca-006543ee63-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.526 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.527 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:55.528 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:55.528 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.529 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:45:55.529 DEBUG:boto:Data: 2023-11-02T18:45:55.529 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.530 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.530 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.531 DEBUG:boto:Params: {} 2023-11-02T18:45:55.531 DEBUG:boto:Token: None 2023-11-02T18:45:55.532 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj19 2023-11-02T18:45:55.532 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b+EHFFH1s/kNb4l2dk+U+5rKCKU= 2023-11-02T18:45:55.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b+EHFFH1s/kNb4l2dk+U+5rKCKU=', 'Content-Length': '0'} 2023-11-02T18:45:55.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028554c664ec0f426-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.536 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:55.537 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:55.537 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.538 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:45:55.538 DEBUG:boto:Data: 2023-11-02T18:45:55.539 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.539 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.539 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.540 DEBUG:boto:Params: {} 2023-11-02T18:45:55.540 DEBUG:boto:Token: None 2023-11-02T18:45:55.541 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj19 2023-11-02T18:45:55.541 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b+EHFFH1s/kNb4l2dk+U+5rKCKU= 2023-11-02T18:45:55.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b+EHFFH1s/kNb4l2dk+U+5rKCKU=', 'Content-Length': '0'} 2023-11-02T18:45:55.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000422c2c6a3b805f1c-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:45:55.546 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:55.546 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:55.546 DEBUG:boto:Method: GET 2023-11-02T18:45:55.547 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:45:55.547 DEBUG:boto:Data: 2023-11-02T18:45:55.548 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.548 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.549 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.549 DEBUG:boto:Params: {} 2023-11-02T18:45:55.549 DEBUG:boto:Token: None 2023-11-02T18:45:55.550 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj19 2023-11-02T18:45:55.550 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aYPHouUR4BF3qu4OLy9e7qVjIgQ= 2023-11-02T18:45:55.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aYPHouUR4BF3qu4OLy9e7qVjIgQ=', 'Content-Length': '0'} 2023-11-02T18:45:55.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c06a63260f8070d7-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.554 DEBUG:boto:path=/ibtljn-59/obj19 2023-11-02T18:45:55.555 DEBUG:boto:auth_path=/ibtljn-59/obj19 2023-11-02T18:45:55.555 DEBUG:boto:Method: GET 2023-11-02T18:45:55.556 DEBUG:boto:Path: /ibtljn-59/obj19 2023-11-02T18:45:55.556 DEBUG:boto:Data: 2023-11-02T18:45:55.557 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.557 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.557 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.558 DEBUG:boto:Params: {} 2023-11-02T18:45:55.559 DEBUG:boto:Token: None 2023-11-02T18:45:55.559 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj19 2023-11-02T18:45:55.559 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aYPHouUR4BF3qu4OLy9e7qVjIgQ= 2023-11-02T18:45:55.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aYPHouUR4BF3qu4OLy9e7qVjIgQ=', 'Content-Length': '0'} 2023-11-02T18:45:55.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a6ad3c4cd406949-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.564 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.564 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.565 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:55.565 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:55.565 DEBUG:boto:Method: GET 2023-11-02T18:45:55.566 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:45:55.566 DEBUG:boto:Data: 2023-11-02T18:45:55.567 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.567 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.568 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.568 DEBUG:boto:Params: {} 2023-11-02T18:45:55.569 DEBUG:boto:Token: None 2023-11-02T18:45:55.569 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b88963f825e1d9cf-006543ee63-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.575 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.576 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.577 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.577 DEBUG:boto:path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:55.577 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj19 2023-11-02T18:45:55.578 DEBUG:boto:Method: GET 2023-11-02T18:45:55.578 DEBUG:boto:Path: /ibtljn-59/?prefix=obj19 2023-11-02T18:45:55.579 DEBUG:boto:Data: 2023-11-02T18:45:55.579 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.580 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.580 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.580 DEBUG:boto:Params: {} 2023-11-02T18:45:55.581 DEBUG:boto:Token: None 2023-11-02T18:45:55.581 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.582 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d551a1ad6f57eff3-006543ee63-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.592 DEBUG:boto:b'ibtljn-59obj191000falseobj192023-11-02T18:34:50.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:45:55.594 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:55.594 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:55.594 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.595 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.595 DEBUG:boto:Method: GET 2023-11-02T18:45:55.596 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.596 DEBUG:boto:Data: 2023-11-02T18:45:55.597 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.597 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.597 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.598 DEBUG:boto:Params: {} 2023-11-02T18:45:55.598 DEBUG:boto:Token: None 2023-11-02T18:45:55.599 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.599 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A8roZUX9XYb6LIT3jaBUtTvVPWI= 2023-11-02T18:45:55.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A8roZUX9XYb6LIT3jaBUtTvVPWI=', 'Content-Length': '0'} 2023-11-02T18:45:55.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075a0bf81427860d5-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.646 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:55.647 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:55.647 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.648 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.648 DEBUG:boto:Method: GET 2023-11-02T18:45:55.649 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.649 DEBUG:boto:Data: 2023-11-02T18:45:55.649 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.650 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.650 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.651 DEBUG:boto:Params: {} 2023-11-02T18:45:55.652 DEBUG:boto:Token: None 2023-11-02T18:45:55.653 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A8roZUX9XYb6LIT3jaBUtTvVPWI= 2023-11-02T18:45:55.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A8roZUX9XYb6LIT3jaBUtTvVPWI=', 'Content-Length': '0'} 2023-11-02T18:45:55.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004768b372b818c670-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.657 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:55.658 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:55.658 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.658 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.659 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.659 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.660 DEBUG:boto:Data: 2023-11-02T18:45:55.660 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.660 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.661 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.661 DEBUG:boto:Params: {} 2023-11-02T18:45:55.662 DEBUG:boto:Token: None 2023-11-02T18:45:55.662 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.663 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CExBSr0ivAQhJYQ5GK3Nz1KI87Q= 2023-11-02T18:45:55.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CExBSr0ivAQhJYQ5GK3Nz1KI87Q=', 'Content-Length': '0'} 2023-11-02T18:45:55.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e51e7346d3663d-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.667 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:55.667 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:55.667 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.668 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.668 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.669 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.669 DEBUG:boto:Data: 2023-11-02T18:45:55.669 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.670 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.670 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.671 DEBUG:boto:Params: {} 2023-11-02T18:45:55.671 DEBUG:boto:Token: None 2023-11-02T18:45:55.672 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.672 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CExBSr0ivAQhJYQ5GK3Nz1KI87Q= 2023-11-02T18:45:55.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CExBSr0ivAQhJYQ5GK3Nz1KI87Q=', 'Content-Length': '0'} 2023-11-02T18:45:55.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6f41c4acd0e5e20-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:45:55.676 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:55.677 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:55.677 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.678 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.678 DEBUG:boto:Method: GET 2023-11-02T18:45:55.679 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.679 DEBUG:boto:Data: 2023-11-02T18:45:55.679 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.680 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.681 DEBUG:boto:Params: {} 2023-11-02T18:45:55.681 DEBUG:boto:Token: None 2023-11-02T18:45:55.682 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.682 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A8roZUX9XYb6LIT3jaBUtTvVPWI= 2023-11-02T18:45:55.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A8roZUX9XYb6LIT3jaBUtTvVPWI=', 'Content-Length': '0'} 2023-11-02T18:45:55.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f25899d1662ab7-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.687 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:55.687 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:55.687 DEBUG:boto:path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.688 DEBUG:boto:auth_path=/ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.688 DEBUG:boto:Method: GET 2023-11-02T18:45:55.689 DEBUG:boto:Path: /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.689 DEBUG:boto:Data: 2023-11-02T18:45:55.690 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.690 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.691 DEBUG:boto:Params: {} 2023-11-02T18:45:55.691 DEBUG:boto:Token: None 2023-11-02T18:45:55.692 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj22?versionId=null 2023-11-02T18:45:55.692 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A8roZUX9XYb6LIT3jaBUtTvVPWI= 2023-11-02T18:45:55.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A8roZUX9XYb6LIT3jaBUtTvVPWI=', 'Content-Length': '0'} 2023-11-02T18:45:55.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e58ff08b03b666c9-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.742 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.743 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.743 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:55.744 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:55.744 DEBUG:boto:Method: GET 2023-11-02T18:45:55.745 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:45:55.745 DEBUG:boto:Data: 2023-11-02T18:45:55.746 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.746 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.746 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.747 DEBUG:boto:Params: {} 2023-11-02T18:45:55.747 DEBUG:boto:Token: None 2023-11-02T18:45:55.748 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.748 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044e8f758a1eb165c-006543ee63-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.755 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.755 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.756 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.756 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:55.757 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:55.757 DEBUG:boto:Method: GET 2023-11-02T18:45:55.758 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:45:55.758 DEBUG:boto:Data: 2023-11-02T18:45:55.758 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.759 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.759 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.760 DEBUG:boto:Params: {} 2023-11-02T18:45:55.760 DEBUG:boto:Token: None 2023-11-02T18:45:55.761 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.761 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a05742630055e19-006543ee63-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.770 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.770 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:55.771 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:55.771 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.772 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:45:55.772 DEBUG:boto:Data: 2023-11-02T18:45:55.773 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.773 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.773 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.774 DEBUG:boto:Params: {} 2023-11-02T18:45:55.774 DEBUG:boto:Token: None 2023-11-02T18:45:55.775 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj22 2023-11-02T18:45:55.775 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FPd3moOgKL2IrJGojGYKBz1Wrqw= 2023-11-02T18:45:55.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FPd3moOgKL2IrJGojGYKBz1Wrqw=', 'Content-Length': '0'} 2023-11-02T18:45:55.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071fd3c9a09380fae-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.779 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:55.779 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:55.780 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.780 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:45:55.781 DEBUG:boto:Data: 2023-11-02T18:45:55.781 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.781 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.782 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.782 DEBUG:boto:Params: {} 2023-11-02T18:45:55.783 DEBUG:boto:Token: None 2023-11-02T18:45:55.783 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj22 2023-11-02T18:45:55.784 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FPd3moOgKL2IrJGojGYKBz1Wrqw= 2023-11-02T18:45:55.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FPd3moOgKL2IrJGojGYKBz1Wrqw=', 'Content-Length': '0'} 2023-11-02T18:45:55.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe1633b8cf0d2e2-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:45:55.788 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:55.788 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:55.789 DEBUG:boto:Method: GET 2023-11-02T18:45:55.789 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:45:55.789 DEBUG:boto:Data: 2023-11-02T18:45:55.790 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.791 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.791 DEBUG:boto:Params: {} 2023-11-02T18:45:55.792 DEBUG:boto:Token: None 2023-11-02T18:45:55.792 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj22 2023-11-02T18:45:55.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QaqyB/BjMGvISRTwiWBQSRaffjg= 2023-11-02T18:45:55.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QaqyB/BjMGvISRTwiWBQSRaffjg=', 'Content-Length': '0'} 2023-11-02T18:45:55.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064cfa22063fcece5-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.842 DEBUG:boto:path=/ibtljn-59/obj22 2023-11-02T18:45:55.843 DEBUG:boto:auth_path=/ibtljn-59/obj22 2023-11-02T18:45:55.844 DEBUG:boto:Method: GET 2023-11-02T18:45:55.844 DEBUG:boto:Path: /ibtljn-59/obj22 2023-11-02T18:45:55.844 DEBUG:boto:Data: 2023-11-02T18:45:55.845 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.845 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.846 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.846 DEBUG:boto:Params: {} 2023-11-02T18:45:55.847 DEBUG:boto:Token: None 2023-11-02T18:45:55.848 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj22 2023-11-02T18:45:55.848 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QaqyB/BjMGvISRTwiWBQSRaffjg= 2023-11-02T18:45:55.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QaqyB/BjMGvISRTwiWBQSRaffjg=', 'Content-Length': '0'} 2023-11-02T18:45:55.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eadd9d1ccb78c2c8-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.853 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.853 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.854 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:55.854 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:55.855 DEBUG:boto:Method: GET 2023-11-02T18:45:55.855 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:45:55.856 DEBUG:boto:Data: 2023-11-02T18:45:55.856 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.857 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.858 DEBUG:boto:Params: {} 2023-11-02T18:45:55.858 DEBUG:boto:Token: None 2023-11-02T18:45:55.859 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.859 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4bc4249d45a7217-006543ee63-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.865 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.866 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:55.867 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:55.867 DEBUG:boto:path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:55.868 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj22 2023-11-02T18:45:55.868 DEBUG:boto:Method: GET 2023-11-02T18:45:55.869 DEBUG:boto:Path: /ibtljn-59/?prefix=obj22 2023-11-02T18:45:55.869 DEBUG:boto:Data: 2023-11-02T18:45:55.870 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.871 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.871 DEBUG:boto:Params: {} 2023-11-02T18:45:55.872 DEBUG:boto:Token: None 2023-11-02T18:45:55.872 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/ 2023-11-02T18:45:55.873 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU= 2023-11-02T18:45:55.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3orcD+2e8VomBRqJOP4twUg6faU=', 'Content-Length': '0'} 2023-11-02T18:45:55.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036fa3fe373089df3-006543ee63-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.926 DEBUG:boto:b'ibtljn-59obj221000falseobj222023-11-02T18:32:14.572Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:55.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:45:55.928 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:55.928 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:55.929 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.929 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.929 DEBUG:boto:Method: GET 2023-11-02T18:45:55.930 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.930 DEBUG:boto:Data: 2023-11-02T18:45:55.931 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.931 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.931 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.932 DEBUG:boto:Params: {} 2023-11-02T18:45:55.932 DEBUG:boto:Token: None 2023-11-02T18:45:55.933 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Flwdo1bNL6lW3ORZsAo72NX4NIQ= 2023-11-02T18:45:55.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Flwdo1bNL6lW3ORZsAo72NX4NIQ=', 'Content-Length': '0'} 2023-11-02T18:45:55.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036a57a10794c759e-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.938 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:55.938 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:55.939 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.939 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.940 DEBUG:boto:Method: GET 2023-11-02T18:45:55.940 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.940 DEBUG:boto:Data: 2023-11-02T18:45:55.941 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.941 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.942 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.942 DEBUG:boto:Params: {} 2023-11-02T18:45:55.943 DEBUG:boto:Token: None 2023-11-02T18:45:55.943 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.944 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Flwdo1bNL6lW3ORZsAo72NX4NIQ= 2023-11-02T18:45:55.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Flwdo1bNL6lW3ORZsAo72NX4NIQ=', 'Content-Length': '0'} 2023-11-02T18:45:55.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000127773d2f6ccc895-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.948 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:55.949 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:55.949 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.949 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.950 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.950 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.951 DEBUG:boto:Data: 2023-11-02T18:45:55.951 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.952 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.952 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.952 DEBUG:boto:Params: {} 2023-11-02T18:45:55.953 DEBUG:boto:Token: None 2023-11-02T18:45:55.953 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.954 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vf89xAGDpD1S4Q/RKdm1AYnuPA0= 2023-11-02T18:45:55.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vf89xAGDpD1S4Q/RKdm1AYnuPA0=', 'Content-Length': '0'} 2023-11-02T18:45:55.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f95323299d0a7a05-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.958 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:55.958 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:55.959 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.959 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.959 DEBUG:boto:Method: HEAD 2023-11-02T18:45:55.960 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.960 DEBUG:boto:Data: 2023-11-02T18:45:55.961 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.961 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.961 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.962 DEBUG:boto:Params: {} 2023-11-02T18:45:55.962 DEBUG:boto:Token: None 2023-11-02T18:45:55.963 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.963 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vf89xAGDpD1S4Q/RKdm1AYnuPA0= 2023-11-02T18:45:55.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vf89xAGDpD1S4Q/RKdm1AYnuPA0=', 'Content-Length': '0'} 2023-11-02T18:45:55.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2d0090184c219b8-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:45:55.968 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:55.968 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:55.969 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.969 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.969 DEBUG:boto:Method: GET 2023-11-02T18:45:55.970 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.970 DEBUG:boto:Data: 2023-11-02T18:45:55.971 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.971 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:55.971 DEBUG:boto:Port: 8001 2023-11-02T18:45:55.972 DEBUG:boto:Params: {} 2023-11-02T18:45:55.972 DEBUG:boto:Token: None 2023-11-02T18:45:55.973 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.973 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Flwdo1bNL6lW3ORZsAo72NX4NIQ= 2023-11-02T18:45:55.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Flwdo1bNL6lW3ORZsAo72NX4NIQ=', 'Content-Length': '0'} 2023-11-02T18:45:55.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d55d8d2e0c4cc4a4-006543ee63-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:55.977 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:55.978 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:55.978 DEBUG:boto:path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.978 DEBUG:boto:auth_path=/ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.979 DEBUG:boto:Method: GET 2023-11-02T18:45:55.979 DEBUG:boto:Path: /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.980 DEBUG:boto:Data: 2023-11-02T18:45:55.980 DEBUG:boto:Headers: {} 2023-11-02T18:45:55.980 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:55.981 DEBUG:boto:Port: 8000 2023-11-02T18:45:55.981 DEBUG:boto:Params: {} 2023-11-02T18:45:55.982 DEBUG:boto:Token: None 2023-11-02T18:45:55.982 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:55 GMT /ibtljn-59/obj23?versionId=null 2023-11-02T18:45:55.983 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Flwdo1bNL6lW3ORZsAo72NX4NIQ= 2023-11-02T18:45:55.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Flwdo1bNL6lW3ORZsAo72NX4NIQ=', 'Content-Length': '0'} 2023-11-02T18:45:55.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b1c1be39865c74e-006543ee63-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.026 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.027 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.027 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:56.030 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:56.030 DEBUG:boto:Method: GET 2023-11-02T18:45:56.031 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:45:56.031 DEBUG:boto:Data: 2023-11-02T18:45:56.032 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.032 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.033 DEBUG:boto:Params: {} 2023-11-02T18:45:56.033 DEBUG:boto:Token: None 2023-11-02T18:45:56.034 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.034 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019b005107f794444-006543ee64-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.040 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.041 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.041 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.042 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:56.042 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:56.042 DEBUG:boto:Method: GET 2023-11-02T18:45:56.043 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:45:56.043 DEBUG:boto:Data: 2023-11-02T18:45:56.044 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.045 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.045 DEBUG:boto:Params: {} 2023-11-02T18:45:56.046 DEBUG:boto:Token: None 2023-11-02T18:45:56.046 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.047 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000800fb9a0108ad726-006543ee64-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.057 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.058 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:56.058 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:56.059 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.059 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:45:56.060 DEBUG:boto:Data: 2023-11-02T18:45:56.060 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.060 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.061 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.061 DEBUG:boto:Params: {} 2023-11-02T18:45:56.062 DEBUG:boto:Token: None 2023-11-02T18:45:56.062 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj23 2023-11-02T18:45:56.063 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HFfKDmIIMJWnAfZl2epX2IkTI0U= 2023-11-02T18:45:56.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HFfKDmIIMJWnAfZl2epX2IkTI0U=', 'Content-Length': '0'} 2023-11-02T18:45:56.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096173ae9c5df2b23-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.066 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:56.067 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:56.067 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.067 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:45:56.068 DEBUG:boto:Data: 2023-11-02T18:45:56.068 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.069 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.069 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.069 DEBUG:boto:Params: {} 2023-11-02T18:45:56.070 DEBUG:boto:Token: None 2023-11-02T18:45:56.070 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj23 2023-11-02T18:45:56.071 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HFfKDmIIMJWnAfZl2epX2IkTI0U= 2023-11-02T18:45:56.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HFfKDmIIMJWnAfZl2epX2IkTI0U=', 'Content-Length': '0'} 2023-11-02T18:45:56.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078491652ab8eb09e-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:45:56.074 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:56.075 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:56.075 DEBUG:boto:Method: GET 2023-11-02T18:45:56.076 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:45:56.076 DEBUG:boto:Data: 2023-11-02T18:45:56.076 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.077 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.077 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.078 DEBUG:boto:Params: {} 2023-11-02T18:45:56.078 DEBUG:boto:Token: None 2023-11-02T18:45:56.079 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj23 2023-11-02T18:45:56.079 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TlpcO/UzJmQPBfDxwzscOlZ0jYQ= 2023-11-02T18:45:56.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TlpcO/UzJmQPBfDxwzscOlZ0jYQ=', 'Content-Length': '0'} 2023-11-02T18:45:56.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a920f0483e7d077b-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.083 DEBUG:boto:path=/ibtljn-59/obj23 2023-11-02T18:45:56.083 DEBUG:boto:auth_path=/ibtljn-59/obj23 2023-11-02T18:45:56.084 DEBUG:boto:Method: GET 2023-11-02T18:45:56.084 DEBUG:boto:Path: /ibtljn-59/obj23 2023-11-02T18:45:56.084 DEBUG:boto:Data: 2023-11-02T18:45:56.085 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.085 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.086 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.086 DEBUG:boto:Params: {} 2023-11-02T18:45:56.087 DEBUG:boto:Token: None 2023-11-02T18:45:56.087 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj23 2023-11-02T18:45:56.087 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TlpcO/UzJmQPBfDxwzscOlZ0jYQ= 2023-11-02T18:45:56.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TlpcO/UzJmQPBfDxwzscOlZ0jYQ=', 'Content-Length': '0'} 2023-11-02T18:45:56.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d2fe3abf161266-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.091 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.092 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.092 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:56.093 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:56.093 DEBUG:boto:Method: GET 2023-11-02T18:45:56.093 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:45:56.094 DEBUG:boto:Data: 2023-11-02T18:45:56.094 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.095 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.095 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.096 DEBUG:boto:Params: {} 2023-11-02T18:45:56.096 DEBUG:boto:Token: None 2023-11-02T18:45:56.097 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.097 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f876a6f089d4bba-006543ee64-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.146 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.147 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.148 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.148 DEBUG:boto:path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:56.149 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj23 2023-11-02T18:45:56.149 DEBUG:boto:Method: GET 2023-11-02T18:45:56.149 DEBUG:boto:Path: /ibtljn-59/?prefix=obj23 2023-11-02T18:45:56.150 DEBUG:boto:Data: 2023-11-02T18:45:56.150 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.151 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.151 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.152 DEBUG:boto:Params: {} 2023-11-02T18:45:56.152 DEBUG:boto:Token: None 2023-11-02T18:45:56.153 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.153 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001943bf817894fb7f-006543ee64-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.166 DEBUG:boto:b'ibtljn-59obj231000falseobj232023-11-02T18:39:10.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:45:56.167 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:56.167 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:56.168 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.168 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.169 DEBUG:boto:Method: GET 2023-11-02T18:45:56.169 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.169 DEBUG:boto:Data: 2023-11-02T18:45:56.170 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.170 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.171 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.171 DEBUG:boto:Params: {} 2023-11-02T18:45:56.172 DEBUG:boto:Token: None 2023-11-02T18:45:56.172 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.173 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JlFwsT9J02YAWLcaOXTckfoMYCY= 2023-11-02T18:45:56.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JlFwsT9J02YAWLcaOXTckfoMYCY=', 'Content-Length': '0'} 2023-11-02T18:45:56.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb08c9daafd88968-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.177 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:56.177 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:56.178 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.178 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.179 DEBUG:boto:Method: GET 2023-11-02T18:45:56.179 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.179 DEBUG:boto:Data: 2023-11-02T18:45:56.180 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.180 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.181 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.181 DEBUG:boto:Params: {} 2023-11-02T18:45:56.182 DEBUG:boto:Token: None 2023-11-02T18:45:56.182 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.183 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JlFwsT9J02YAWLcaOXTckfoMYCY= 2023-11-02T18:45:56.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JlFwsT9J02YAWLcaOXTckfoMYCY=', 'Content-Length': '0'} 2023-11-02T18:45:56.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fff2f65f5051eec7-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.230 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:56.231 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:56.231 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.232 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.232 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.233 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.233 DEBUG:boto:Data: 2023-11-02T18:45:56.234 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.234 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.234 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.235 DEBUG:boto:Params: {} 2023-11-02T18:45:56.235 DEBUG:boto:Token: None 2023-11-02T18:45:56.236 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.237 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9P7m957Iq4RhNN3e7d5+gPm8ABQ= 2023-11-02T18:45:56.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9P7m957Iq4RhNN3e7d5+gPm8ABQ=', 'Content-Length': '0'} 2023-11-02T18:45:56.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac4e7b9b77f30b61-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.242 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:56.242 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:56.243 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.244 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.244 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.244 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.245 DEBUG:boto:Data: 2023-11-02T18:45:56.245 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.246 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.247 DEBUG:boto:Params: {} 2023-11-02T18:45:56.247 DEBUG:boto:Token: None 2023-11-02T18:45:56.248 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.248 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9P7m957Iq4RhNN3e7d5+gPm8ABQ= 2023-11-02T18:45:56.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9P7m957Iq4RhNN3e7d5+gPm8ABQ=', 'Content-Length': '0'} 2023-11-02T18:45:56.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ac418f47576ae0e-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:45:56.253 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:56.254 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:56.254 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.254 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.255 DEBUG:boto:Method: GET 2023-11-02T18:45:56.255 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.256 DEBUG:boto:Data: 2023-11-02T18:45:56.256 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.257 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.257 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.258 DEBUG:boto:Params: {} 2023-11-02T18:45:56.258 DEBUG:boto:Token: None 2023-11-02T18:45:56.259 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.259 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JlFwsT9J02YAWLcaOXTckfoMYCY= 2023-11-02T18:45:56.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JlFwsT9J02YAWLcaOXTckfoMYCY=', 'Content-Length': '0'} 2023-11-02T18:45:56.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6667b280aa8d007-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.264 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:56.264 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:56.265 DEBUG:boto:path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.265 DEBUG:boto:auth_path=/ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.265 DEBUG:boto:Method: GET 2023-11-02T18:45:56.266 DEBUG:boto:Path: /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.266 DEBUG:boto:Data: 2023-11-02T18:45:56.267 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.267 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.268 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.268 DEBUG:boto:Params: {} 2023-11-02T18:45:56.269 DEBUG:boto:Token: None 2023-11-02T18:45:56.269 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj29?versionId=null 2023-11-02T18:45:56.270 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JlFwsT9J02YAWLcaOXTckfoMYCY= 2023-11-02T18:45:56.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JlFwsT9J02YAWLcaOXTckfoMYCY=', 'Content-Length': '0'} 2023-11-02T18:45:56.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e0530c9157b0d7-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.274 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.274 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.275 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:56.275 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:56.276 DEBUG:boto:Method: GET 2023-11-02T18:45:56.276 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:45:56.277 DEBUG:boto:Data: 2023-11-02T18:45:56.277 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.277 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.278 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.278 DEBUG:boto:Params: {} 2023-11-02T18:45:56.279 DEBUG:boto:Token: None 2023-11-02T18:45:56.280 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.280 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd960e6fd6c6f027-006543ee64-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.326 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.327 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.328 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.328 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:56.329 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:56.329 DEBUG:boto:Method: GET 2023-11-02T18:45:56.330 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:45:56.330 DEBUG:boto:Data: 2023-11-02T18:45:56.331 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.331 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.331 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.332 DEBUG:boto:Params: {} 2023-11-02T18:45:56.332 DEBUG:boto:Token: None 2023-11-02T18:45:56.333 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.333 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd4e480ec784574c-006543ee64-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.344 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.345 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:56.345 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:56.346 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.346 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:45:56.347 DEBUG:boto:Data: 2023-11-02T18:45:56.347 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.348 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.348 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.348 DEBUG:boto:Params: {} 2023-11-02T18:45:56.349 DEBUG:boto:Token: None 2023-11-02T18:45:56.349 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj29 2023-11-02T18:45:56.350 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l3iKF5TYAnS78PACXDxNncRonOE= 2023-11-02T18:45:56.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l3iKF5TYAnS78PACXDxNncRonOE=', 'Content-Length': '0'} 2023-11-02T18:45:56.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028fa3d6965c36a53-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.354 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:56.355 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:56.355 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.356 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:45:56.356 DEBUG:boto:Data: 2023-11-02T18:45:56.356 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.357 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.357 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.358 DEBUG:boto:Params: {} 2023-11-02T18:45:56.358 DEBUG:boto:Token: None 2023-11-02T18:45:56.359 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj29 2023-11-02T18:45:56.359 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l3iKF5TYAnS78PACXDxNncRonOE= 2023-11-02T18:45:56.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l3iKF5TYAnS78PACXDxNncRonOE=', 'Content-Length': '0'} 2023-11-02T18:45:56.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a3bbf53682ece42-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:45:56.363 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:56.364 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:56.364 DEBUG:boto:Method: GET 2023-11-02T18:45:56.365 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:45:56.365 DEBUG:boto:Data: 2023-11-02T18:45:56.365 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.367 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.367 DEBUG:boto:Params: {} 2023-11-02T18:45:56.367 DEBUG:boto:Token: None 2023-11-02T18:45:56.368 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj29 2023-11-02T18:45:56.369 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jpg/iQyYv5S/7dJDngc0QKtM+KU= 2023-11-02T18:45:56.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jpg/iQyYv5S/7dJDngc0QKtM+KU=', 'Content-Length': '0'} 2023-11-02T18:45:56.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c141b67cfd640dda-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.373 DEBUG:boto:path=/ibtljn-59/obj29 2023-11-02T18:45:56.373 DEBUG:boto:auth_path=/ibtljn-59/obj29 2023-11-02T18:45:56.374 DEBUG:boto:Method: GET 2023-11-02T18:45:56.374 DEBUG:boto:Path: /ibtljn-59/obj29 2023-11-02T18:45:56.374 DEBUG:boto:Data: 2023-11-02T18:45:56.375 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.375 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.376 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.376 DEBUG:boto:Params: {} 2023-11-02T18:45:56.377 DEBUG:boto:Token: None 2023-11-02T18:45:56.377 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj29 2023-11-02T18:45:56.378 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jpg/iQyYv5S/7dJDngc0QKtM+KU= 2023-11-02T18:45:56.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jpg/iQyYv5S/7dJDngc0QKtM+KU=', 'Content-Length': '0'} 2023-11-02T18:45:56.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6283b333df04604-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.423 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.423 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.424 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:56.424 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:56.425 DEBUG:boto:Method: GET 2023-11-02T18:45:56.425 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:45:56.426 DEBUG:boto:Data: 2023-11-02T18:45:56.426 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.427 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.427 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.428 DEBUG:boto:Params: {} 2023-11-02T18:45:56.428 DEBUG:boto:Token: None 2023-11-02T18:45:56.429 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.429 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc2808539fd8d698-006543ee64-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.436 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.437 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.437 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.437 DEBUG:boto:path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:56.438 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj29 2023-11-02T18:45:56.438 DEBUG:boto:Method: GET 2023-11-02T18:45:56.441 DEBUG:boto:Path: /ibtljn-59/?prefix=obj29 2023-11-02T18:45:56.441 DEBUG:boto:Data: 2023-11-02T18:45:56.442 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.442 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.443 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.443 DEBUG:boto:Params: {} 2023-11-02T18:45:56.443 DEBUG:boto:Token: None 2023-11-02T18:45:56.444 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e85fdd5e3a00d7a9-006543ee64-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.455 DEBUG:boto:b'ibtljn-59obj291000falseobj292023-11-02T18:40:31.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T18:45:56.456 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:56.456 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:56.457 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.457 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.458 DEBUG:boto:Method: GET 2023-11-02T18:45:56.458 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.459 DEBUG:boto:Data: 2023-11-02T18:45:56.459 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.460 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.460 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.461 DEBUG:boto:Params: {} 2023-11-02T18:45:56.461 DEBUG:boto:Token: None 2023-11-02T18:45:56.462 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.462 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4e3hSahqIoZGitmZKihK6sFPp6E= 2023-11-02T18:45:56.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4e3hSahqIoZGitmZKihK6sFPp6E=', 'Content-Length': '0'} 2023-11-02T18:45:56.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed200de07261a9c2-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.466 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:56.466 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:56.467 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.467 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.468 DEBUG:boto:Method: GET 2023-11-02T18:45:56.468 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.469 DEBUG:boto:Data: 2023-11-02T18:45:56.469 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.470 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.470 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.470 DEBUG:boto:Params: {} 2023-11-02T18:45:56.471 DEBUG:boto:Token: None 2023-11-02T18:45:56.471 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.472 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4e3hSahqIoZGitmZKihK6sFPp6E= 2023-11-02T18:45:56.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4e3hSahqIoZGitmZKihK6sFPp6E=', 'Content-Length': '0'} 2023-11-02T18:45:56.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bd7e59ba6e7a1b7-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.476 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:56.476 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:56.477 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.477 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.478 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.478 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.479 DEBUG:boto:Data: 2023-11-02T18:45:56.479 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.479 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.480 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.480 DEBUG:boto:Params: {} 2023-11-02T18:45:56.481 DEBUG:boto:Token: None 2023-11-02T18:45:56.481 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.482 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rRMSj5AhYv0HczomxOceBbAdKZg= 2023-11-02T18:45:56.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rRMSj5AhYv0HczomxOceBbAdKZg=', 'Content-Length': '0'} 2023-11-02T18:45:56.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085923c77f139373d-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.486 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:56.486 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:56.487 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.487 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.488 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.488 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.489 DEBUG:boto:Data: 2023-11-02T18:45:56.489 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.489 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.490 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.490 DEBUG:boto:Params: {} 2023-11-02T18:45:56.491 DEBUG:boto:Token: None 2023-11-02T18:45:56.491 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.492 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rRMSj5AhYv0HczomxOceBbAdKZg= 2023-11-02T18:45:56.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rRMSj5AhYv0HczomxOceBbAdKZg=', 'Content-Length': '0'} 2023-11-02T18:45:56.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5eb5ccf068988a0-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T18:45:56.496 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:56.497 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:56.497 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.498 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.498 DEBUG:boto:Method: GET 2023-11-02T18:45:56.499 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.499 DEBUG:boto:Data: 2023-11-02T18:45:56.499 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.500 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.500 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.501 DEBUG:boto:Params: {} 2023-11-02T18:45:56.501 DEBUG:boto:Token: None 2023-11-02T18:45:56.502 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.502 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4e3hSahqIoZGitmZKihK6sFPp6E= 2023-11-02T18:45:56.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4e3hSahqIoZGitmZKihK6sFPp6E=', 'Content-Length': '0'} 2023-11-02T18:45:56.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005792ff8589408d83-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.507 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:56.507 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:56.508 DEBUG:boto:path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.508 DEBUG:boto:auth_path=/ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.508 DEBUG:boto:Method: GET 2023-11-02T18:45:56.509 DEBUG:boto:Path: /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.509 DEBUG:boto:Data: 2023-11-02T18:45:56.510 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.510 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.511 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.511 DEBUG:boto:Params: {} 2023-11-02T18:45:56.512 DEBUG:boto:Token: None 2023-11-02T18:45:56.512 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj31?versionId=null 2023-11-02T18:45:56.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4e3hSahqIoZGitmZKihK6sFPp6E= 2023-11-02T18:45:56.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4e3hSahqIoZGitmZKihK6sFPp6E=', 'Content-Length': '0'} 2023-11-02T18:45:56.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000987db7dee970ab6b-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.558 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.559 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.560 DEBUG:boto:path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:56.560 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:56.560 DEBUG:boto:Method: GET 2023-11-02T18:45:56.561 DEBUG:boto:Path: /ibtljn-59/?prefix=obj31 2023-11-02T18:45:56.561 DEBUG:boto:Data: 2023-11-02T18:45:56.562 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.562 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.563 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.563 DEBUG:boto:Params: {} 2023-11-02T18:45:56.564 DEBUG:boto:Token: None 2023-11-02T18:45:56.564 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.565 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093b63bfff363bb0e-006543ee64-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.571 DEBUG:boto:b'ibtljn-59obj311000falseobj312023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.572 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.572 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.573 DEBUG:boto:path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:56.573 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:56.574 DEBUG:boto:Method: GET 2023-11-02T18:45:56.574 DEBUG:boto:Path: /ibtljn-59/?prefix=obj31 2023-11-02T18:45:56.575 DEBUG:boto:Data: 2023-11-02T18:45:56.575 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.576 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.576 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.576 DEBUG:boto:Params: {} 2023-11-02T18:45:56.577 DEBUG:boto:Token: None 2023-11-02T18:45:56.577 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.578 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b33d348a21a2e20-006543ee64-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.589 DEBUG:boto:b'ibtljn-59obj311000falseobj312023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.590 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:56.590 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:56.591 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.591 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:45:56.592 DEBUG:boto:Data: 2023-11-02T18:45:56.592 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.593 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.593 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.593 DEBUG:boto:Params: {} 2023-11-02T18:45:56.594 DEBUG:boto:Token: None 2023-11-02T18:45:56.594 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj31 2023-11-02T18:45:56.595 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aM7oGuj8p6FJGEK0VcRCDSTbNXI= 2023-11-02T18:45:56.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aM7oGuj8p6FJGEK0VcRCDSTbNXI=', 'Content-Length': '0'} 2023-11-02T18:45:56.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0c9552c20cbebd6-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.599 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:56.599 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:56.599 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.600 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:45:56.601 DEBUG:boto:Data: 2023-11-02T18:45:56.601 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.602 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.602 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.603 DEBUG:boto:Params: {} 2023-11-02T18:45:56.603 DEBUG:boto:Token: None 2023-11-02T18:45:56.603 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj31 2023-11-02T18:45:56.604 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aM7oGuj8p6FJGEK0VcRCDSTbNXI= 2023-11-02T18:45:56.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aM7oGuj8p6FJGEK0VcRCDSTbNXI=', 'Content-Length': '0'} 2023-11-02T18:45:56.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de1ce7377eafb0dc-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T18:45:56.608 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:56.608 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:56.609 DEBUG:boto:Method: GET 2023-11-02T18:45:56.609 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:45:56.610 DEBUG:boto:Data: 2023-11-02T18:45:56.610 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.611 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.612 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.612 DEBUG:boto:Params: {} 2023-11-02T18:45:56.614 DEBUG:boto:Token: None 2023-11-02T18:45:56.614 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj31 2023-11-02T18:45:56.615 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hdAaPdsIGC/JeehJuFA0NqCHU7I= 2023-11-02T18:45:56.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hdAaPdsIGC/JeehJuFA0NqCHU7I=', 'Content-Length': '0'} 2023-11-02T18:45:56.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eafa36b7f6709c3-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.658 DEBUG:boto:path=/ibtljn-59/obj31 2023-11-02T18:45:56.659 DEBUG:boto:auth_path=/ibtljn-59/obj31 2023-11-02T18:45:56.660 DEBUG:boto:Method: GET 2023-11-02T18:45:56.660 DEBUG:boto:Path: /ibtljn-59/obj31 2023-11-02T18:45:56.660 DEBUG:boto:Data: 2023-11-02T18:45:56.661 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.661 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.662 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.662 DEBUG:boto:Params: {} 2023-11-02T18:45:56.663 DEBUG:boto:Token: None 2023-11-02T18:45:56.664 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj31 2023-11-02T18:45:56.664 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hdAaPdsIGC/JeehJuFA0NqCHU7I= 2023-11-02T18:45:56.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hdAaPdsIGC/JeehJuFA0NqCHU7I=', 'Content-Length': '0'} 2023-11-02T18:45:56.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011579478d44b296a-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.669 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.669 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.670 DEBUG:boto:path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:56.670 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:56.670 DEBUG:boto:Method: GET 2023-11-02T18:45:56.671 DEBUG:boto:Path: /ibtljn-59/?prefix=obj31 2023-11-02T18:45:56.671 DEBUG:boto:Data: 2023-11-02T18:45:56.672 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.672 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.673 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.673 DEBUG:boto:Params: {} 2023-11-02T18:45:56.674 DEBUG:boto:Token: None 2023-11-02T18:45:56.674 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.675 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057a62f71e0526d74-006543ee64-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.680 DEBUG:boto:b'ibtljn-59obj311000falseobj312023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.681 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.681 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.682 DEBUG:boto:path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:56.682 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj31 2023-11-02T18:45:56.683 DEBUG:boto:Method: GET 2023-11-02T18:45:56.683 DEBUG:boto:Path: /ibtljn-59/?prefix=obj31 2023-11-02T18:45:56.684 DEBUG:boto:Data: 2023-11-02T18:45:56.684 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.684 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.685 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.685 DEBUG:boto:Params: {} 2023-11-02T18:45:56.686 DEBUG:boto:Token: None 2023-11-02T18:45:56.686 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.687 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a276e82bcf56488-006543ee64-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.738 DEBUG:boto:b'ibtljn-59obj311000falseobj312023-11-02T18:42:34.279Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:45:56.740 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:56.740 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:56.741 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.741 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.742 DEBUG:boto:Method: GET 2023-11-02T18:45:56.742 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.743 DEBUG:boto:Data: 2023-11-02T18:45:56.743 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.743 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.744 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.744 DEBUG:boto:Params: {} 2023-11-02T18:45:56.745 DEBUG:boto:Token: None 2023-11-02T18:45:56.745 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.746 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vtKzlz9GUbafZetR/AuJrouhjj4= 2023-11-02T18:45:56.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vtKzlz9GUbafZetR/AuJrouhjj4=', 'Content-Length': '0'} 2023-11-02T18:45:56.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c7ba886c322815-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.751 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:56.751 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:56.751 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.752 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.752 DEBUG:boto:Method: GET 2023-11-02T18:45:56.753 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.753 DEBUG:boto:Data: 2023-11-02T18:45:56.754 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.755 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.755 DEBUG:boto:Params: {} 2023-11-02T18:45:56.756 DEBUG:boto:Token: None 2023-11-02T18:45:56.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vtKzlz9GUbafZetR/AuJrouhjj4= 2023-11-02T18:45:56.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vtKzlz9GUbafZetR/AuJrouhjj4=', 'Content-Length': '0'} 2023-11-02T18:45:56.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d22b572aee3b25f-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.761 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:56.761 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:56.762 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.762 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.763 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.763 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.764 DEBUG:boto:Data: 2023-11-02T18:45:56.764 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.765 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.765 DEBUG:boto:Params: {} 2023-11-02T18:45:56.766 DEBUG:boto:Token: None 2023-11-02T18:45:56.766 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.767 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sEWpYNSRHjbYc2lCoWTsicCZ/Y4= 2023-11-02T18:45:56.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sEWpYNSRHjbYc2lCoWTsicCZ/Y4=', 'Content-Length': '0'} 2023-11-02T18:45:56.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b618e1edded0e0c5-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.771 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:56.771 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:56.772 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.772 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.773 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.773 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.774 DEBUG:boto:Data: 2023-11-02T18:45:56.774 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.775 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.775 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.775 DEBUG:boto:Params: {} 2023-11-02T18:45:56.776 DEBUG:boto:Token: None 2023-11-02T18:45:56.776 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.777 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sEWpYNSRHjbYc2lCoWTsicCZ/Y4= 2023-11-02T18:45:56.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sEWpYNSRHjbYc2lCoWTsicCZ/Y4=', 'Content-Length': '0'} 2023-11-02T18:45:56.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf839311d04e1e8c-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:45:56.781 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:56.782 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:56.782 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.783 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.783 DEBUG:boto:Method: GET 2023-11-02T18:45:56.783 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.784 DEBUG:boto:Data: 2023-11-02T18:45:56.784 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.785 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.785 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.786 DEBUG:boto:Params: {} 2023-11-02T18:45:56.786 DEBUG:boto:Token: None 2023-11-02T18:45:56.787 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.787 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vtKzlz9GUbafZetR/AuJrouhjj4= 2023-11-02T18:45:56.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vtKzlz9GUbafZetR/AuJrouhjj4=', 'Content-Length': '0'} 2023-11-02T18:45:56.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b686b21bb14e6d-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.792 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:56.792 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:56.793 DEBUG:boto:path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.793 DEBUG:boto:auth_path=/ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.794 DEBUG:boto:Method: GET 2023-11-02T18:45:56.794 DEBUG:boto:Path: /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.795 DEBUG:boto:Data: 2023-11-02T18:45:56.795 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.795 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.796 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.796 DEBUG:boto:Params: {} 2023-11-02T18:45:56.797 DEBUG:boto:Token: None 2023-11-02T18:45:56.797 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj33?versionId=null 2023-11-02T18:45:56.798 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vtKzlz9GUbafZetR/AuJrouhjj4= 2023-11-02T18:45:56.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vtKzlz9GUbafZetR/AuJrouhjj4=', 'Content-Length': '0'} 2023-11-02T18:45:56.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050d6a6c23562d397-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.803 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.803 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.804 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:56.804 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:56.805 DEBUG:boto:Method: GET 2023-11-02T18:45:56.805 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:45:56.805 DEBUG:boto:Data: 2023-11-02T18:45:56.806 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.806 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.807 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.807 DEBUG:boto:Params: {} 2023-11-02T18:45:56.808 DEBUG:boto:Token: None 2023-11-02T18:45:56.808 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.809 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071860b3d688174f7-006543ee64-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.858 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.859 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.860 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.860 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:56.861 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:56.861 DEBUG:boto:Method: GET 2023-11-02T18:45:56.862 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:45:56.862 DEBUG:boto:Data: 2023-11-02T18:45:56.863 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.863 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.864 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.864 DEBUG:boto:Params: {} 2023-11-02T18:45:56.865 DEBUG:boto:Token: None 2023-11-02T18:45:56.865 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.866 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd3b30f79d2e310b-006543ee64-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.876 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.877 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:56.878 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:56.878 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.879 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:45:56.879 DEBUG:boto:Data: 2023-11-02T18:45:56.879 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.880 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.880 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.881 DEBUG:boto:Params: {} 2023-11-02T18:45:56.881 DEBUG:boto:Token: None 2023-11-02T18:45:56.882 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj33 2023-11-02T18:45:56.882 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PkJGRi7fA21dS29GFDWLcDhSTlU= 2023-11-02T18:45:56.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PkJGRi7fA21dS29GFDWLcDhSTlU=', 'Content-Length': '0'} 2023-11-02T18:45:56.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007b63831796ddc5e-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.887 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:56.887 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:56.888 DEBUG:boto:Method: HEAD 2023-11-02T18:45:56.888 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:45:56.889 DEBUG:boto:Data: 2023-11-02T18:45:56.889 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.889 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.890 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.890 DEBUG:boto:Params: {} 2023-11-02T18:45:56.891 DEBUG:boto:Token: None 2023-11-02T18:45:56.891 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj33 2023-11-02T18:45:56.892 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PkJGRi7fA21dS29GFDWLcDhSTlU= 2023-11-02T18:45:56.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PkJGRi7fA21dS29GFDWLcDhSTlU=', 'Content-Length': '0'} 2023-11-02T18:45:56.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000808f9dac2734e721-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:45:56.896 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:56.897 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:56.897 DEBUG:boto:Method: GET 2023-11-02T18:45:56.898 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:45:56.898 DEBUG:boto:Data: 2023-11-02T18:45:56.899 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.899 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.900 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.900 DEBUG:boto:Params: {} 2023-11-02T18:45:56.901 DEBUG:boto:Token: None 2023-11-02T18:45:56.901 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj33 2023-11-02T18:45:56.902 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1KcR2Z8xae6p8AO4/5hnG4U7/Y8= 2023-11-02T18:45:56.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1KcR2Z8xae6p8AO4/5hnG4U7/Y8=', 'Content-Length': '0'} 2023-11-02T18:45:56.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af042d0aba44253e-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.906 DEBUG:boto:path=/ibtljn-59/obj33 2023-11-02T18:45:56.907 DEBUG:boto:auth_path=/ibtljn-59/obj33 2023-11-02T18:45:56.907 DEBUG:boto:Method: GET 2023-11-02T18:45:56.908 DEBUG:boto:Path: /ibtljn-59/obj33 2023-11-02T18:45:56.908 DEBUG:boto:Data: 2023-11-02T18:45:56.908 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.909 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.909 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.910 DEBUG:boto:Params: {} 2023-11-02T18:45:56.911 DEBUG:boto:Token: None 2023-11-02T18:45:56.911 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/obj33 2023-11-02T18:45:56.912 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1KcR2Z8xae6p8AO4/5hnG4U7/Y8= 2023-11-02T18:45:56.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1KcR2Z8xae6p8AO4/5hnG4U7/Y8=', 'Content-Length': '0'} 2023-11-02T18:45:56.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098057334bb89d3cc-006543ee64-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.962 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.963 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.963 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:56.964 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:56.964 DEBUG:boto:Method: GET 2023-11-02T18:45:56.965 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:45:56.965 DEBUG:boto:Data: 2023-11-02T18:45:56.966 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.966 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.967 DEBUG:boto:Port: 8001 2023-11-02T18:45:56.967 DEBUG:boto:Params: {} 2023-11-02T18:45:56.968 DEBUG:boto:Token: None 2023-11-02T18:45:56.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079eeeaa08121b935-006543ee64-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.975 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.976 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:56.976 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:56.977 DEBUG:boto:path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:56.977 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj33 2023-11-02T18:45:56.978 DEBUG:boto:Method: GET 2023-11-02T18:45:56.978 DEBUG:boto:Path: /ibtljn-59/?prefix=obj33 2023-11-02T18:45:56.979 DEBUG:boto:Data: 2023-11-02T18:45:56.979 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.980 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:56.980 DEBUG:boto:Port: 8000 2023-11-02T18:45:56.981 DEBUG:boto:Params: {} 2023-11-02T18:45:56.981 DEBUG:boto:Token: None 2023-11-02T18:45:56.982 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:56 GMT /ibtljn-59/ 2023-11-02T18:45:56.982 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs= 2023-11-02T18:45:56.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AtmgLk21TF2Kwb/MLm+K8Dm0ELs=', 'Content-Length': '0'} 2023-11-02T18:45:56.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000562738e87976cbac-006543ee64-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:56.993 DEBUG:boto:b'ibtljn-59obj331000falseobj332023-11-02T18:32:14.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:56.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:45:56.995 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:56.995 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:56.996 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:56.996 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:56.997 DEBUG:boto:Method: GET 2023-11-02T18:45:56.997 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:56.997 DEBUG:boto:Data: 2023-11-02T18:45:56.998 DEBUG:boto:Headers: {} 2023-11-02T18:45:56.998 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:56.999 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.000 DEBUG:boto:Params: {} 2023-11-02T18:45:57.000 DEBUG:boto:Token: None 2023-11-02T18:45:57.001 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.001 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WKZRhrrFl5lnwlrmRmXbkVFQHQE= 2023-11-02T18:45:57.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WKZRhrrFl5lnwlrmRmXbkVFQHQE=', 'Content-Length': '0'} 2023-11-02T18:45:57.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007471a939527516ce-006543ee64-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.005 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:57.006 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:57.006 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.007 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.007 DEBUG:boto:Method: GET 2023-11-02T18:45:57.008 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.008 DEBUG:boto:Data: 2023-11-02T18:45:57.009 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.009 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.009 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.010 DEBUG:boto:Params: {} 2023-11-02T18:45:57.010 DEBUG:boto:Token: None 2023-11-02T18:45:57.011 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.011 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WKZRhrrFl5lnwlrmRmXbkVFQHQE= 2023-11-02T18:45:57.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WKZRhrrFl5lnwlrmRmXbkVFQHQE=', 'Content-Length': '0'} 2023-11-02T18:45:57.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbfda1aba10f9455-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.015 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:57.015 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:57.016 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.016 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.017 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.017 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.018 DEBUG:boto:Data: 2023-11-02T18:45:57.018 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.018 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.019 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.019 DEBUG:boto:Params: {} 2023-11-02T18:45:57.020 DEBUG:boto:Token: None 2023-11-02T18:45:57.020 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.021 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VTAMLBgooKZ5Dt2KOGg10cGiZMc= 2023-11-02T18:45:57.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VTAMLBgooKZ5Dt2KOGg10cGiZMc=', 'Content-Length': '0'} 2023-11-02T18:45:57.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c6c8282be292edc-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.024 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:57.025 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:57.025 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.026 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.026 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.027 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.027 DEBUG:boto:Data: 2023-11-02T18:45:57.027 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.028 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.028 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.029 DEBUG:boto:Params: {} 2023-11-02T18:45:57.029 DEBUG:boto:Token: None 2023-11-02T18:45:57.030 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.030 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VTAMLBgooKZ5Dt2KOGg10cGiZMc= 2023-11-02T18:45:57.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VTAMLBgooKZ5Dt2KOGg10cGiZMc=', 'Content-Length': '0'} 2023-11-02T18:45:57.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad56de538ee3bda0-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:45:57.034 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:57.035 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:57.035 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.036 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.036 DEBUG:boto:Method: GET 2023-11-02T18:45:57.037 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.037 DEBUG:boto:Data: 2023-11-02T18:45:57.037 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.038 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.039 DEBUG:boto:Params: {} 2023-11-02T18:45:57.039 DEBUG:boto:Token: None 2023-11-02T18:45:57.040 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.040 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WKZRhrrFl5lnwlrmRmXbkVFQHQE= 2023-11-02T18:45:57.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WKZRhrrFl5lnwlrmRmXbkVFQHQE=', 'Content-Length': '0'} 2023-11-02T18:45:57.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041862e341a6f40d6-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.086 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:57.087 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:57.087 DEBUG:boto:path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.088 DEBUG:boto:auth_path=/ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.088 DEBUG:boto:Method: GET 2023-11-02T18:45:57.089 DEBUG:boto:Path: /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.089 DEBUG:boto:Data: 2023-11-02T18:45:57.090 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.090 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.091 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.091 DEBUG:boto:Params: {} 2023-11-02T18:45:57.092 DEBUG:boto:Token: None 2023-11-02T18:45:57.092 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj34?versionId=null 2023-11-02T18:45:57.093 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WKZRhrrFl5lnwlrmRmXbkVFQHQE= 2023-11-02T18:45:57.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WKZRhrrFl5lnwlrmRmXbkVFQHQE=', 'Content-Length': '0'} 2023-11-02T18:45:57.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1b50ab4a2d2ccdb-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.098 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.098 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.099 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:57.099 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:57.100 DEBUG:boto:Method: GET 2023-11-02T18:45:57.100 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:45:57.101 DEBUG:boto:Data: 2023-11-02T18:45:57.101 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.102 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.102 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.103 DEBUG:boto:Params: {} 2023-11-02T18:45:57.103 DEBUG:boto:Token: None 2023-11-02T18:45:57.104 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a813db223ee2776b-006543ee65-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.111 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.111 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.112 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.112 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:57.113 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:57.113 DEBUG:boto:Method: GET 2023-11-02T18:45:57.114 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:45:57.114 DEBUG:boto:Data: 2023-11-02T18:45:57.115 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.115 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.115 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.116 DEBUG:boto:Params: {} 2023-11-02T18:45:57.116 DEBUG:boto:Token: None 2023-11-02T18:45:57.117 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.117 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000263bb2fd19d9dea5-006543ee65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.128 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.129 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:57.129 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:57.130 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.130 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:45:57.130 DEBUG:boto:Data: 2023-11-02T18:45:57.131 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.131 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.132 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.132 DEBUG:boto:Params: {} 2023-11-02T18:45:57.133 DEBUG:boto:Token: None 2023-11-02T18:45:57.133 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj34 2023-11-02T18:45:57.134 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H7Fy2pzj2ZlUa1//x3GE0m1BDi4= 2023-11-02T18:45:57.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H7Fy2pzj2ZlUa1//x3GE0m1BDi4=', 'Content-Length': '0'} 2023-11-02T18:45:57.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011bf0fc8664e7d4f-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.137 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:57.138 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:57.138 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.138 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:45:57.139 DEBUG:boto:Data: 2023-11-02T18:45:57.139 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.140 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.140 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.140 DEBUG:boto:Params: {} 2023-11-02T18:45:57.141 DEBUG:boto:Token: None 2023-11-02T18:45:57.141 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj34 2023-11-02T18:45:57.142 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H7Fy2pzj2ZlUa1//x3GE0m1BDi4= 2023-11-02T18:45:57.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H7Fy2pzj2ZlUa1//x3GE0m1BDi4=', 'Content-Length': '0'} 2023-11-02T18:45:57.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000945921ce7e12d4e6-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:45:57.146 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:57.146 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:57.147 DEBUG:boto:Method: GET 2023-11-02T18:45:57.147 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:45:57.148 DEBUG:boto:Data: 2023-11-02T18:45:57.148 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.148 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.149 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.149 DEBUG:boto:Params: {} 2023-11-02T18:45:57.150 DEBUG:boto:Token: None 2023-11-02T18:45:57.150 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj34 2023-11-02T18:45:57.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BsMifThKdIDiloByWavu2HrtVr8= 2023-11-02T18:45:57.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BsMifThKdIDiloByWavu2HrtVr8=', 'Content-Length': '0'} 2023-11-02T18:45:57.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e94091b3773cb9c7-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.155 DEBUG:boto:path=/ibtljn-59/obj34 2023-11-02T18:45:57.155 DEBUG:boto:auth_path=/ibtljn-59/obj34 2023-11-02T18:45:57.156 DEBUG:boto:Method: GET 2023-11-02T18:45:57.156 DEBUG:boto:Path: /ibtljn-59/obj34 2023-11-02T18:45:57.156 DEBUG:boto:Data: 2023-11-02T18:45:57.157 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.157 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.158 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.158 DEBUG:boto:Params: {} 2023-11-02T18:45:57.159 DEBUG:boto:Token: None 2023-11-02T18:45:57.159 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj34 2023-11-02T18:45:57.160 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BsMifThKdIDiloByWavu2HrtVr8= 2023-11-02T18:45:57.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BsMifThKdIDiloByWavu2HrtVr8=', 'Content-Length': '0'} 2023-11-02T18:45:57.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efe37969bee5dd31-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.164 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.164 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.165 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:57.165 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:57.165 DEBUG:boto:Method: GET 2023-11-02T18:45:57.166 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:45:57.166 DEBUG:boto:Data: 2023-11-02T18:45:57.167 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.167 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.168 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.168 DEBUG:boto:Params: {} 2023-11-02T18:45:57.169 DEBUG:boto:Token: None 2023-11-02T18:45:57.169 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.170 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000006d189ff322feb0-006543ee65-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.218 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.219 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.220 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.220 DEBUG:boto:path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:57.221 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj34 2023-11-02T18:45:57.221 DEBUG:boto:Method: GET 2023-11-02T18:45:57.222 DEBUG:boto:Path: /ibtljn-59/?prefix=obj34 2023-11-02T18:45:57.222 DEBUG:boto:Data: 2023-11-02T18:45:57.222 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.223 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.224 DEBUG:boto:Params: {} 2023-11-02T18:45:57.224 DEBUG:boto:Token: None 2023-11-02T18:45:57.225 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d1c4d4f65c0742d-006543ee65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.237 DEBUG:boto:b'ibtljn-59obj341000falseobj342023-11-02T18:32:46.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-11-02T18:45:57.238 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:57.238 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:57.239 DEBUG:boto:path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.239 DEBUG:boto:auth_path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.240 DEBUG:boto:Method: GET 2023-11-02T18:45:57.240 DEBUG:boto:Path: /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.241 DEBUG:boto:Data: 2023-11-02T18:45:57.241 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.241 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.242 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.242 DEBUG:boto:Params: {} 2023-11-02T18:45:57.243 DEBUG:boto:Token: None 2023-11-02T18:45:57.243 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.244 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OafIwBS3KopSKEa8zn3DWobQf/Q= 2023-11-02T18:45:57.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OafIwBS3KopSKEa8zn3DWobQf/Q=', 'Content-Length': '0'} 2023-11-02T18:45:57.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4206fae0a8a0e6e-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.248 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:57.248 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:57.249 DEBUG:boto:path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.249 DEBUG:boto:auth_path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.250 DEBUG:boto:Method: GET 2023-11-02T18:45:57.250 DEBUG:boto:Path: /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.251 DEBUG:boto:Data: 2023-11-02T18:45:57.251 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.251 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.252 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.252 DEBUG:boto:Params: {} 2023-11-02T18:45:57.253 DEBUG:boto:Token: None 2023-11-02T18:45:57.253 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.254 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OafIwBS3KopSKEa8zn3DWobQf/Q= 2023-11-02T18:45:57.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OafIwBS3KopSKEa8zn3DWobQf/Q=', 'Content-Length': '0'} 2023-11-02T18:45:57.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a24a1064cff4908-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.258 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:57.258 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:57.259 DEBUG:boto:path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.259 DEBUG:boto:auth_path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.260 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.260 DEBUG:boto:Path: /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.260 DEBUG:boto:Data: 2023-11-02T18:45:57.261 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.261 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.262 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.262 DEBUG:boto:Params: {} 2023-11-02T18:45:57.262 DEBUG:boto:Token: None 2023-11-02T18:45:57.263 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.263 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zY/YdDiTBulMSyR8hqcChWJy9Pk= 2023-11-02T18:45:57.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zY/YdDiTBulMSyR8hqcChWJy9Pk=', 'Content-Length': '0'} 2023-11-02T18:45:57.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cff9f9f8092f8f81-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.267 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:57.268 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:57.268 DEBUG:boto:path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.268 DEBUG:boto:auth_path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.269 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.269 DEBUG:boto:Path: /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.270 DEBUG:boto:Data: 2023-11-02T18:45:57.270 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.270 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.271 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.271 DEBUG:boto:Params: {} 2023-11-02T18:45:57.272 DEBUG:boto:Token: None 2023-11-02T18:45:57.272 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.273 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zY/YdDiTBulMSyR8hqcChWJy9Pk= 2023-11-02T18:45:57.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zY/YdDiTBulMSyR8hqcChWJy9Pk=', 'Content-Length': '0'} 2023-11-02T18:45:57.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d425c9b7a9289fa3-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-11-02T18:45:57.277 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:57.278 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:57.278 DEBUG:boto:path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.278 DEBUG:boto:auth_path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.279 DEBUG:boto:Method: GET 2023-11-02T18:45:57.279 DEBUG:boto:Path: /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.280 DEBUG:boto:Data: 2023-11-02T18:45:57.280 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.281 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.281 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.282 DEBUG:boto:Params: {} 2023-11-02T18:45:57.282 DEBUG:boto:Token: None 2023-11-02T18:45:57.283 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.283 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OafIwBS3KopSKEa8zn3DWobQf/Q= 2023-11-02T18:45:57.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OafIwBS3KopSKEa8zn3DWobQf/Q=', 'Content-Length': '0'} 2023-11-02T18:45:57.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042108f48921cced1-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.334 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:57.335 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:57.335 DEBUG:boto:path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.336 DEBUG:boto:auth_path=/ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.336 DEBUG:boto:Method: GET 2023-11-02T18:45:57.337 DEBUG:boto:Path: /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.337 DEBUG:boto:Data: 2023-11-02T18:45:57.338 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.338 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.339 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.339 DEBUG:boto:Params: {} 2023-11-02T18:45:57.340 DEBUG:boto:Token: None 2023-11-02T18:45:57.340 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj37?versionId=null 2023-11-02T18:45:57.341 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OafIwBS3KopSKEa8zn3DWobQf/Q= 2023-11-02T18:45:57.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OafIwBS3KopSKEa8zn3DWobQf/Q=', 'Content-Length': '0'} 2023-11-02T18:45:57.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092a6d18cc1761bba-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.345 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.346 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.346 DEBUG:boto:path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:57.347 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:57.347 DEBUG:boto:Method: GET 2023-11-02T18:45:57.347 DEBUG:boto:Path: /ibtljn-59/?prefix=obj37 2023-11-02T18:45:57.348 DEBUG:boto:Data: 2023-11-02T18:45:57.348 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.349 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.349 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.350 DEBUG:boto:Params: {} 2023-11-02T18:45:57.350 DEBUG:boto:Token: None 2023-11-02T18:45:57.351 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.351 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe6c6aa885eee550-006543ee65-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.358 DEBUG:boto:b'ibtljn-59obj371000falseobj372023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.358 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.359 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.359 DEBUG:boto:path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:57.360 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:57.360 DEBUG:boto:Method: GET 2023-11-02T18:45:57.361 DEBUG:boto:Path: /ibtljn-59/?prefix=obj37 2023-11-02T18:45:57.361 DEBUG:boto:Data: 2023-11-02T18:45:57.362 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.362 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.362 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.363 DEBUG:boto:Params: {} 2023-11-02T18:45:57.364 DEBUG:boto:Token: None 2023-11-02T18:45:57.364 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.364 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e61708684bf96435-006543ee65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.418 DEBUG:boto:b'ibtljn-59obj371000falseobj372023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.419 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:57.420 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:57.420 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.421 DEBUG:boto:Path: /ibtljn-59/obj37 2023-11-02T18:45:57.421 DEBUG:boto:Data: 2023-11-02T18:45:57.421 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.422 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.423 DEBUG:boto:Params: {} 2023-11-02T18:45:57.423 DEBUG:boto:Token: None 2023-11-02T18:45:57.424 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj37 2023-11-02T18:45:57.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Nia3djKH/Q+0ScoeN+C80d4MVj0= 2023-11-02T18:45:57.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Nia3djKH/Q+0ScoeN+C80d4MVj0=', 'Content-Length': '0'} 2023-11-02T18:45:57.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f45bd26f1f3d6dde-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.428 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:57.429 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:57.429 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.430 DEBUG:boto:Path: /ibtljn-59/obj37 2023-11-02T18:45:57.430 DEBUG:boto:Data: 2023-11-02T18:45:57.431 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.431 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.432 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.432 DEBUG:boto:Params: {} 2023-11-02T18:45:57.432 DEBUG:boto:Token: None 2023-11-02T18:45:57.433 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj37 2023-11-02T18:45:57.433 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Nia3djKH/Q+0ScoeN+C80d4MVj0= 2023-11-02T18:45:57.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Nia3djKH/Q+0ScoeN+C80d4MVj0=', 'Content-Length': '0'} 2023-11-02T18:45:57.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087f7f374af43542f-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-11-02T18:45:57.438 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:57.438 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:57.439 DEBUG:boto:Method: GET 2023-11-02T18:45:57.439 DEBUG:boto:Path: /ibtljn-59/obj37 2023-11-02T18:45:57.440 DEBUG:boto:Data: 2023-11-02T18:45:57.440 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.440 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.441 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.441 DEBUG:boto:Params: {} 2023-11-02T18:45:57.442 DEBUG:boto:Token: None 2023-11-02T18:45:57.442 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj37 2023-11-02T18:45:57.443 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TOhKkFXPOCRiXg5sWBvXK846DyA= 2023-11-02T18:45:57.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TOhKkFXPOCRiXg5sWBvXK846DyA=', 'Content-Length': '0'} 2023-11-02T18:45:57.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c8c2df69c47e46-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.447 DEBUG:boto:path=/ibtljn-59/obj37 2023-11-02T18:45:57.448 DEBUG:boto:auth_path=/ibtljn-59/obj37 2023-11-02T18:45:57.448 DEBUG:boto:Method: GET 2023-11-02T18:45:57.449 DEBUG:boto:Path: /ibtljn-59/obj37 2023-11-02T18:45:57.449 DEBUG:boto:Data: 2023-11-02T18:45:57.450 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.450 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.451 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.451 DEBUG:boto:Params: {} 2023-11-02T18:45:57.451 DEBUG:boto:Token: None 2023-11-02T18:45:57.452 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj37 2023-11-02T18:45:57.452 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TOhKkFXPOCRiXg5sWBvXK846DyA= 2023-11-02T18:45:57.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TOhKkFXPOCRiXg5sWBvXK846DyA=', 'Content-Length': '0'} 2023-11-02T18:45:57.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087175f0a0265ed00-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.456 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.457 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.457 DEBUG:boto:path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:57.458 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:57.458 DEBUG:boto:Method: GET 2023-11-02T18:45:57.459 DEBUG:boto:Path: /ibtljn-59/?prefix=obj37 2023-11-02T18:45:57.459 DEBUG:boto:Data: 2023-11-02T18:45:57.461 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.461 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.462 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.462 DEBUG:boto:Params: {} 2023-11-02T18:45:57.463 DEBUG:boto:Token: None 2023-11-02T18:45:57.463 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.464 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc2d0eba339323be-006543ee65-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.514 DEBUG:boto:b'ibtljn-59obj371000falseobj372023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.515 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.516 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.516 DEBUG:boto:path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:57.516 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj37 2023-11-02T18:45:57.517 DEBUG:boto:Method: GET 2023-11-02T18:45:57.517 DEBUG:boto:Path: /ibtljn-59/?prefix=obj37 2023-11-02T18:45:57.518 DEBUG:boto:Data: 2023-11-02T18:45:57.518 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.519 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.519 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.520 DEBUG:boto:Params: {} 2023-11-02T18:45:57.520 DEBUG:boto:Token: None 2023-11-02T18:45:57.521 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.521 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052659d1350c1bd40-006543ee65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.531 DEBUG:boto:b'ibtljn-59obj371000falseobj372023-11-02T18:44:13.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:45:57.532 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:57.532 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:57.533 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.533 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.534 DEBUG:boto:Method: GET 2023-11-02T18:45:57.534 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.535 DEBUG:boto:Data: 2023-11-02T18:45:57.535 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.536 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.536 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.536 DEBUG:boto:Params: {} 2023-11-02T18:45:57.537 DEBUG:boto:Token: None 2023-11-02T18:45:57.537 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.538 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zr4aUN2nW9uyfoCyG9oDhpq35gs= 2023-11-02T18:45:57.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zr4aUN2nW9uyfoCyG9oDhpq35gs=', 'Content-Length': '0'} 2023-11-02T18:45:57.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d533557e6599dd3e-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.542 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:57.543 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:57.543 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.544 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.544 DEBUG:boto:Method: GET 2023-11-02T18:45:57.545 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.545 DEBUG:boto:Data: 2023-11-02T18:45:57.545 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.546 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.546 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.547 DEBUG:boto:Params: {} 2023-11-02T18:45:57.547 DEBUG:boto:Token: None 2023-11-02T18:45:57.548 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.548 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zr4aUN2nW9uyfoCyG9oDhpq35gs= 2023-11-02T18:45:57.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zr4aUN2nW9uyfoCyG9oDhpq35gs=', 'Content-Length': '0'} 2023-11-02T18:45:57.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f885ea0a5d98899-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.552 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:57.553 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:57.553 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.554 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.554 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.555 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.555 DEBUG:boto:Data: 2023-11-02T18:45:57.555 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.556 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.556 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.557 DEBUG:boto:Params: {} 2023-11-02T18:45:57.557 DEBUG:boto:Token: None 2023-11-02T18:45:57.557 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.558 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PZ8Q+byVJkGoS8cXhOmdj1Mjkfs= 2023-11-02T18:45:57.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PZ8Q+byVJkGoS8cXhOmdj1Mjkfs=', 'Content-Length': '0'} 2023-11-02T18:45:57.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a56d9555651a6071-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.562 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:57.562 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:57.563 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.563 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.563 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.564 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.564 DEBUG:boto:Data: 2023-11-02T18:45:57.565 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.565 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.566 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.566 DEBUG:boto:Params: {} 2023-11-02T18:45:57.566 DEBUG:boto:Token: None 2023-11-02T18:45:57.567 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.567 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PZ8Q+byVJkGoS8cXhOmdj1Mjkfs= 2023-11-02T18:45:57.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PZ8Q+byVJkGoS8cXhOmdj1Mjkfs=', 'Content-Length': '0'} 2023-11-02T18:45:57.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007df0ed0ab0f2ef9b-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:45:57.572 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:57.572 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:57.573 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.573 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.574 DEBUG:boto:Method: GET 2023-11-02T18:45:57.574 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.574 DEBUG:boto:Data: 2023-11-02T18:45:57.575 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.575 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.576 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.576 DEBUG:boto:Params: {} 2023-11-02T18:45:57.577 DEBUG:boto:Token: None 2023-11-02T18:45:57.577 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.578 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zr4aUN2nW9uyfoCyG9oDhpq35gs= 2023-11-02T18:45:57.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zr4aUN2nW9uyfoCyG9oDhpq35gs=', 'Content-Length': '0'} 2023-11-02T18:45:57.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de628611b3196bcd-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.582 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:57.582 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:57.583 DEBUG:boto:path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.583 DEBUG:boto:auth_path=/ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.584 DEBUG:boto:Method: GET 2023-11-02T18:45:57.584 DEBUG:boto:Path: /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.584 DEBUG:boto:Data: 2023-11-02T18:45:57.585 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.585 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.586 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.586 DEBUG:boto:Params: {} 2023-11-02T18:45:57.587 DEBUG:boto:Token: None 2023-11-02T18:45:57.587 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj38?versionId=null 2023-11-02T18:45:57.588 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Zr4aUN2nW9uyfoCyG9oDhpq35gs= 2023-11-02T18:45:57.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Zr4aUN2nW9uyfoCyG9oDhpq35gs=', 'Content-Length': '0'} 2023-11-02T18:45:57.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000116e83ee17754053-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.634 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.635 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.635 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:57.636 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:57.636 DEBUG:boto:Method: GET 2023-11-02T18:45:57.637 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:45:57.637 DEBUG:boto:Data: 2023-11-02T18:45:57.638 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.638 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.638 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.639 DEBUG:boto:Params: {} 2023-11-02T18:45:57.639 DEBUG:boto:Token: None 2023-11-02T18:45:57.640 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.640 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7c57fcdef80c95c-006543ee65-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.644 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.645 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.646 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.646 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:57.647 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:57.647 DEBUG:boto:Method: GET 2023-11-02T18:45:57.647 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:45:57.648 DEBUG:boto:Data: 2023-11-02T18:45:57.648 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.649 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.649 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.650 DEBUG:boto:Params: {} 2023-11-02T18:45:57.650 DEBUG:boto:Token: None 2023-11-02T18:45:57.651 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.651 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0954ecef27f7727-006543ee65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.659 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.660 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:57.661 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:57.661 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.661 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:45:57.662 DEBUG:boto:Data: 2023-11-02T18:45:57.662 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.665 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.666 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.666 DEBUG:boto:Params: {} 2023-11-02T18:45:57.667 DEBUG:boto:Token: None 2023-11-02T18:45:57.667 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj38 2023-11-02T18:45:57.668 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5aEFoKQ/2TyZqlRA8bJYHZLXcqc= 2023-11-02T18:45:57.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5aEFoKQ/2TyZqlRA8bJYHZLXcqc=', 'Content-Length': '0'} 2023-11-02T18:45:57.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048535c38904ae29d-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.671 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:57.672 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:57.672 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.673 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:45:57.673 DEBUG:boto:Data: 2023-11-02T18:45:57.674 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.674 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.675 DEBUG:boto:Params: {} 2023-11-02T18:45:57.675 DEBUG:boto:Token: None 2023-11-02T18:45:57.676 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj38 2023-11-02T18:45:57.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5aEFoKQ/2TyZqlRA8bJYHZLXcqc= 2023-11-02T18:45:57.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5aEFoKQ/2TyZqlRA8bJYHZLXcqc=', 'Content-Length': '0'} 2023-11-02T18:45:57.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ceef4e66480aea-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:45:57.681 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:57.681 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:57.682 DEBUG:boto:Method: GET 2023-11-02T18:45:57.682 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:45:57.682 DEBUG:boto:Data: 2023-11-02T18:45:57.683 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.683 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.684 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.684 DEBUG:boto:Params: {} 2023-11-02T18:45:57.685 DEBUG:boto:Token: None 2023-11-02T18:45:57.685 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj38 2023-11-02T18:45:57.686 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:82MMx2jXbtiN7H8qQiGCAObCVk0= 2023-11-02T18:45:57.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:82MMx2jXbtiN7H8qQiGCAObCVk0=', 'Content-Length': '0'} 2023-11-02T18:45:57.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cee200d173c2d5c1-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.730 DEBUG:boto:path=/ibtljn-59/obj38 2023-11-02T18:45:57.731 DEBUG:boto:auth_path=/ibtljn-59/obj38 2023-11-02T18:45:57.731 DEBUG:boto:Method: GET 2023-11-02T18:45:57.732 DEBUG:boto:Path: /ibtljn-59/obj38 2023-11-02T18:45:57.732 DEBUG:boto:Data: 2023-11-02T18:45:57.733 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.733 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.733 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.734 DEBUG:boto:Params: {} 2023-11-02T18:45:57.734 DEBUG:boto:Token: None 2023-11-02T18:45:57.735 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj38 2023-11-02T18:45:57.735 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:82MMx2jXbtiN7H8qQiGCAObCVk0= 2023-11-02T18:45:57.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:82MMx2jXbtiN7H8qQiGCAObCVk0=', 'Content-Length': '0'} 2023-11-02T18:45:57.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008f57c65359def08-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.740 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.740 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.741 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:57.741 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:57.742 DEBUG:boto:Method: GET 2023-11-02T18:45:57.742 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:45:57.743 DEBUG:boto:Data: 2023-11-02T18:45:57.743 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.743 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.744 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.744 DEBUG:boto:Params: {} 2023-11-02T18:45:57.745 DEBUG:boto:Token: None 2023-11-02T18:45:57.745 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.746 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000720e38d93f35789b-006543ee65-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.752 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.752 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.753 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.753 DEBUG:boto:path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:57.754 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj38 2023-11-02T18:45:57.754 DEBUG:boto:Method: GET 2023-11-02T18:45:57.755 DEBUG:boto:Path: /ibtljn-59/?prefix=obj38 2023-11-02T18:45:57.755 DEBUG:boto:Data: 2023-11-02T18:45:57.756 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.756 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.757 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.757 DEBUG:boto:Params: {} 2023-11-02T18:45:57.757 DEBUG:boto:Token: None 2023-11-02T18:45:57.758 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.759 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067970b2c3c627137-006543ee65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.810 DEBUG:boto:b'ibtljn-59obj381000falseobj382023-11-02T18:34:50.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:45:57.812 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:57.812 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:57.812 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.813 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.813 DEBUG:boto:Method: GET 2023-11-02T18:45:57.814 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.814 DEBUG:boto:Data: 2023-11-02T18:45:57.815 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.815 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.816 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.816 DEBUG:boto:Params: {} 2023-11-02T18:45:57.817 DEBUG:boto:Token: None 2023-11-02T18:45:57.817 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.818 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CMgEeZ5zcyAY6Coe8Bl7n1VlFA8= 2023-11-02T18:45:57.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CMgEeZ5zcyAY6Coe8Bl7n1VlFA8=', 'Content-Length': '0'} 2023-11-02T18:45:57.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012385ff49d662e83-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.822 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:57.823 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:57.823 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.824 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.824 DEBUG:boto:Method: GET 2023-11-02T18:45:57.825 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.825 DEBUG:boto:Data: 2023-11-02T18:45:57.825 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.826 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.826 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.827 DEBUG:boto:Params: {} 2023-11-02T18:45:57.827 DEBUG:boto:Token: None 2023-11-02T18:45:57.828 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.828 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CMgEeZ5zcyAY6Coe8Bl7n1VlFA8= 2023-11-02T18:45:57.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CMgEeZ5zcyAY6Coe8Bl7n1VlFA8=', 'Content-Length': '0'} 2023-11-02T18:45:57.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dda565b334ddb43a-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.833 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:57.833 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:57.834 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.834 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.834 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.835 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.835 DEBUG:boto:Data: 2023-11-02T18:45:57.836 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.836 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.837 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.837 DEBUG:boto:Params: {} 2023-11-02T18:45:57.837 DEBUG:boto:Token: None 2023-11-02T18:45:57.838 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.838 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tFPUsBzsNXKcNRSp4Og1eiNfoBY= 2023-11-02T18:45:57.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tFPUsBzsNXKcNRSp4Og1eiNfoBY=', 'Content-Length': '0'} 2023-11-02T18:45:57.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffab92f2b3ceaba1-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.842 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:57.843 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:57.843 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.844 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.844 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.844 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.845 DEBUG:boto:Data: 2023-11-02T18:45:57.845 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.846 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.846 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.847 DEBUG:boto:Params: {} 2023-11-02T18:45:57.847 DEBUG:boto:Token: None 2023-11-02T18:45:57.848 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.848 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tFPUsBzsNXKcNRSp4Og1eiNfoBY= 2023-11-02T18:45:57.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tFPUsBzsNXKcNRSp4Og1eiNfoBY=', 'Content-Length': '0'} 2023-11-02T18:45:57.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fb70aa0f55b305f-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:45:57.853 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:57.853 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:57.854 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.854 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.854 DEBUG:boto:Method: GET 2023-11-02T18:45:57.855 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.855 DEBUG:boto:Data: 2023-11-02T18:45:57.856 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.856 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.857 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.857 DEBUG:boto:Params: {} 2023-11-02T18:45:57.858 DEBUG:boto:Token: None 2023-11-02T18:45:57.858 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.859 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CMgEeZ5zcyAY6Coe8Bl7n1VlFA8= 2023-11-02T18:45:57.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CMgEeZ5zcyAY6Coe8Bl7n1VlFA8=', 'Content-Length': '0'} 2023-11-02T18:45:57.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce15a22ba44086f3-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.863 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:57.863 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:57.864 DEBUG:boto:path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.864 DEBUG:boto:auth_path=/ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.865 DEBUG:boto:Method: GET 2023-11-02T18:45:57.865 DEBUG:boto:Path: /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.865 DEBUG:boto:Data: 2023-11-02T18:45:57.866 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.866 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.867 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.867 DEBUG:boto:Params: {} 2023-11-02T18:45:57.869 DEBUG:boto:Token: None 2023-11-02T18:45:57.870 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj44?versionId=null 2023-11-02T18:45:57.870 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CMgEeZ5zcyAY6Coe8Bl7n1VlFA8= 2023-11-02T18:45:57.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CMgEeZ5zcyAY6Coe8Bl7n1VlFA8=', 'Content-Length': '0'} 2023-11-02T18:45:57.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8426228bb6f2ba3-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.918 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.919 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.919 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:57.920 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:57.920 DEBUG:boto:Method: GET 2023-11-02T18:45:57.921 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:45:57.921 DEBUG:boto:Data: 2023-11-02T18:45:57.922 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.922 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.923 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.923 DEBUG:boto:Params: {} 2023-11-02T18:45:57.923 DEBUG:boto:Token: None 2023-11-02T18:45:57.924 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.924 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b5fab03be2b8d6b-006543ee65-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.931 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.931 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.932 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.932 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:57.933 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:57.933 DEBUG:boto:Method: GET 2023-11-02T18:45:57.933 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:45:57.934 DEBUG:boto:Data: 2023-11-02T18:45:57.934 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.935 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.935 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.936 DEBUG:boto:Params: {} 2023-11-02T18:45:57.936 DEBUG:boto:Token: None 2023-11-02T18:45:57.937 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.937 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004441273911eb5e9-006543ee65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.948 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.948 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:57.949 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:57.949 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.950 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:45:57.950 DEBUG:boto:Data: 2023-11-02T18:45:57.950 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.951 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.951 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.952 DEBUG:boto:Params: {} 2023-11-02T18:45:57.952 DEBUG:boto:Token: None 2023-11-02T18:45:57.953 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj44 2023-11-02T18:45:57.953 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Uuw4pG2LgR6Pq2bnWyj6gdk4oRw= 2023-11-02T18:45:57.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Uuw4pG2LgR6Pq2bnWyj6gdk4oRw=', 'Content-Length': '0'} 2023-11-02T18:45:57.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acced8844c98e099-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.956 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:57.957 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:57.957 DEBUG:boto:Method: HEAD 2023-11-02T18:45:57.958 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:45:57.958 DEBUG:boto:Data: 2023-11-02T18:45:57.959 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.959 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.959 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.960 DEBUG:boto:Params: {} 2023-11-02T18:45:57.960 DEBUG:boto:Token: None 2023-11-02T18:45:57.961 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj44 2023-11-02T18:45:57.961 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Uuw4pG2LgR6Pq2bnWyj6gdk4oRw= 2023-11-02T18:45:57.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Uuw4pG2LgR6Pq2bnWyj6gdk4oRw=', 'Content-Length': '0'} 2023-11-02T18:45:57.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd3a1ea2f07d1367-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:45:57.965 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:57.965 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:57.966 DEBUG:boto:Method: GET 2023-11-02T18:45:57.966 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:45:57.967 DEBUG:boto:Data: 2023-11-02T18:45:57.967 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.968 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.968 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.968 DEBUG:boto:Params: {} 2023-11-02T18:45:57.969 DEBUG:boto:Token: None 2023-11-02T18:45:57.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj44 2023-11-02T18:45:57.970 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AKQ8F2ICQnQTsjwnLjXzP1pZOzA= 2023-11-02T18:45:57.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AKQ8F2ICQnQTsjwnLjXzP1pZOzA=', 'Content-Length': '0'} 2023-11-02T18:45:57.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035771978e677021d-006543ee65-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.974 DEBUG:boto:path=/ibtljn-59/obj44 2023-11-02T18:45:57.974 DEBUG:boto:auth_path=/ibtljn-59/obj44 2023-11-02T18:45:57.975 DEBUG:boto:Method: GET 2023-11-02T18:45:57.975 DEBUG:boto:Path: /ibtljn-59/obj44 2023-11-02T18:45:57.976 DEBUG:boto:Data: 2023-11-02T18:45:57.976 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.976 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.977 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.977 DEBUG:boto:Params: {} 2023-11-02T18:45:57.978 DEBUG:boto:Token: None 2023-11-02T18:45:57.978 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/obj44 2023-11-02T18:45:57.979 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AKQ8F2ICQnQTsjwnLjXzP1pZOzA= 2023-11-02T18:45:57.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AKQ8F2ICQnQTsjwnLjXzP1pZOzA=', 'Content-Length': '0'} 2023-11-02T18:45:57.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000395d45aa4f9d20ec-006543ee65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.983 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.983 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.984 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:57.984 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:57.985 DEBUG:boto:Method: GET 2023-11-02T18:45:57.985 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:45:57.986 DEBUG:boto:Data: 2023-11-02T18:45:57.986 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.987 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:57.987 DEBUG:boto:Port: 8001 2023-11-02T18:45:57.987 DEBUG:boto:Params: {} 2023-11-02T18:45:57.988 DEBUG:boto:Token: None 2023-11-02T18:45:57.988 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:57 GMT /ibtljn-59/ 2023-11-02T18:45:57.989 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo= 2023-11-02T18:45:57.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTeqF0RCbFmhSFZMNH9SrVNxAHo=', 'Content-Length': '0'} 2023-11-02T18:45:57.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ef48f3b37fe521b-006543ee65-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:57.994 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:57.995 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:57.995 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:57.996 DEBUG:boto:path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:57.996 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj44 2023-11-02T18:45:57.997 DEBUG:boto:Method: GET 2023-11-02T18:45:57.997 DEBUG:boto:Path: /ibtljn-59/?prefix=obj44 2023-11-02T18:45:57.998 DEBUG:boto:Data: 2023-11-02T18:45:57.998 DEBUG:boto:Headers: {} 2023-11-02T18:45:57.998 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:57.999 DEBUG:boto:Port: 8000 2023-11-02T18:45:57.999 DEBUG:boto:Params: {} 2023-11-02T18:45:58.000 DEBUG:boto:Token: None 2023-11-02T18:45:58.000 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.001 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087888a390c35760a-006543ee65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.054 DEBUG:boto:b'ibtljn-59obj441000falseobj442023-11-02T18:32:14.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:45:58.056 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:58.056 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:58.057 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.057 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.057 DEBUG:boto:Method: GET 2023-11-02T18:45:58.058 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.058 DEBUG:boto:Data: 2023-11-02T18:45:58.059 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.059 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.060 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.060 DEBUG:boto:Params: {} 2023-11-02T18:45:58.061 DEBUG:boto:Token: None 2023-11-02T18:45:58.061 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.062 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wIBlQQZSksG5SG8TEGGNRQU9LXk= 2023-11-02T18:45:58.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wIBlQQZSksG5SG8TEGGNRQU9LXk=', 'Content-Length': '0'} 2023-11-02T18:45:58.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000333ef59807e07dcc-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.067 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:58.067 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:58.068 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.068 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.069 DEBUG:boto:Method: GET 2023-11-02T18:45:58.069 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.069 DEBUG:boto:Data: 2023-11-02T18:45:58.070 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.071 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.071 DEBUG:boto:Params: {} 2023-11-02T18:45:58.072 DEBUG:boto:Token: None 2023-11-02T18:45:58.072 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.073 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wIBlQQZSksG5SG8TEGGNRQU9LXk= 2023-11-02T18:45:58.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wIBlQQZSksG5SG8TEGGNRQU9LXk=', 'Content-Length': '0'} 2023-11-02T18:45:58.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe7634b80b4b579-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.076 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:58.077 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:58.077 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.077 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.078 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.078 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.079 DEBUG:boto:Data: 2023-11-02T18:45:58.079 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.080 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.080 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.080 DEBUG:boto:Params: {} 2023-11-02T18:45:58.081 DEBUG:boto:Token: None 2023-11-02T18:45:58.081 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.082 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/iXNhWHp8aT8836VIBWH70k6JUA= 2023-11-02T18:45:58.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/iXNhWHp8aT8836VIBWH70k6JUA=', 'Content-Length': '0'} 2023-11-02T18:45:58.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6c233fa34c00528-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.085 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:58.086 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:58.086 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.086 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.087 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.087 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.088 DEBUG:boto:Data: 2023-11-02T18:45:58.088 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.089 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.089 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.089 DEBUG:boto:Params: {} 2023-11-02T18:45:58.090 DEBUG:boto:Token: None 2023-11-02T18:45:58.090 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.091 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/iXNhWHp8aT8836VIBWH70k6JUA= 2023-11-02T18:45:58.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/iXNhWHp8aT8836VIBWH70k6JUA=', 'Content-Length': '0'} 2023-11-02T18:45:58.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000399a55b120775831-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:45:58.095 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:58.095 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:58.096 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.096 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.097 DEBUG:boto:Method: GET 2023-11-02T18:45:58.097 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.097 DEBUG:boto:Data: 2023-11-02T18:45:58.098 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.098 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.099 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.099 DEBUG:boto:Params: {} 2023-11-02T18:45:58.100 DEBUG:boto:Token: None 2023-11-02T18:45:58.100 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.101 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wIBlQQZSksG5SG8TEGGNRQU9LXk= 2023-11-02T18:45:58.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wIBlQQZSksG5SG8TEGGNRQU9LXk=', 'Content-Length': '0'} 2023-11-02T18:45:58.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a570a3d7766959ae-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.150 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:58.151 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:58.151 DEBUG:boto:path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.152 DEBUG:boto:auth_path=/ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.153 DEBUG:boto:Method: GET 2023-11-02T18:45:58.154 DEBUG:boto:Path: /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.154 DEBUG:boto:Data: 2023-11-02T18:45:58.155 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.155 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.155 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.156 DEBUG:boto:Params: {} 2023-11-02T18:45:58.156 DEBUG:boto:Token: None 2023-11-02T18:45:58.157 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj46?versionId=null 2023-11-02T18:45:58.158 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wIBlQQZSksG5SG8TEGGNRQU9LXk= 2023-11-02T18:45:58.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wIBlQQZSksG5SG8TEGGNRQU9LXk=', 'Content-Length': '0'} 2023-11-02T18:45:58.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d696c336ad7cff80-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.162 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.162 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.163 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:58.163 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:58.164 DEBUG:boto:Method: GET 2023-11-02T18:45:58.164 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:45:58.165 DEBUG:boto:Data: 2023-11-02T18:45:58.165 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.166 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.166 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.166 DEBUG:boto:Params: {} 2023-11-02T18:45:58.167 DEBUG:boto:Token: None 2023-11-02T18:45:58.167 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.168 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070d1b8a7ee1f1487-006543ee66-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.175 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.175 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.176 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.176 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:58.177 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:58.177 DEBUG:boto:Method: GET 2023-11-02T18:45:58.178 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:45:58.178 DEBUG:boto:Data: 2023-11-02T18:45:58.179 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.180 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.180 DEBUG:boto:Params: {} 2023-11-02T18:45:58.181 DEBUG:boto:Token: None 2023-11-02T18:45:58.181 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.182 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f1159f57c0e5ba2-006543ee66-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.192 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.193 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:58.193 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:58.194 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.194 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:45:58.195 DEBUG:boto:Data: 2023-11-02T18:45:58.195 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.196 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.196 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.197 DEBUG:boto:Params: {} 2023-11-02T18:45:58.197 DEBUG:boto:Token: None 2023-11-02T18:45:58.198 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj46 2023-11-02T18:45:58.198 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LFf2cNh2XFF1vaH5t/vFX1o76cA= 2023-11-02T18:45:58.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LFf2cNh2XFF1vaH5t/vFX1o76cA=', 'Content-Length': '0'} 2023-11-02T18:45:58.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000044e412549f509c2-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.202 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:58.202 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:58.203 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.203 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:45:58.203 DEBUG:boto:Data: 2023-11-02T18:45:58.204 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.204 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.205 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.205 DEBUG:boto:Params: {} 2023-11-02T18:45:58.206 DEBUG:boto:Token: None 2023-11-02T18:45:58.206 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj46 2023-11-02T18:45:58.207 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LFf2cNh2XFF1vaH5t/vFX1o76cA= 2023-11-02T18:45:58.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LFf2cNh2XFF1vaH5t/vFX1o76cA=', 'Content-Length': '0'} 2023-11-02T18:45:58.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a78bdfe70218a76-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:45:58.211 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:58.211 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:58.212 DEBUG:boto:Method: GET 2023-11-02T18:45:58.213 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:45:58.213 DEBUG:boto:Data: 2023-11-02T18:45:58.214 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.214 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.215 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.215 DEBUG:boto:Params: {} 2023-11-02T18:45:58.216 DEBUG:boto:Token: None 2023-11-02T18:45:58.216 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj46 2023-11-02T18:45:58.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KreKOcx0tinQEzvF2OqiRHbBAWU= 2023-11-02T18:45:58.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KreKOcx0tinQEzvF2OqiRHbBAWU=', 'Content-Length': '0'} 2023-11-02T18:45:58.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083b48186f0779703-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.221 DEBUG:boto:path=/ibtljn-59/obj46 2023-11-02T18:45:58.221 DEBUG:boto:auth_path=/ibtljn-59/obj46 2023-11-02T18:45:58.222 DEBUG:boto:Method: GET 2023-11-02T18:45:58.222 DEBUG:boto:Path: /ibtljn-59/obj46 2023-11-02T18:45:58.223 DEBUG:boto:Data: 2023-11-02T18:45:58.223 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.224 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.224 DEBUG:boto:Params: {} 2023-11-02T18:45:58.225 DEBUG:boto:Token: None 2023-11-02T18:45:58.225 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj46 2023-11-02T18:45:58.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KreKOcx0tinQEzvF2OqiRHbBAWU= 2023-11-02T18:45:58.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KreKOcx0tinQEzvF2OqiRHbBAWU=', 'Content-Length': '0'} 2023-11-02T18:45:58.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d02e0e883ec33156-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.270 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.271 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.272 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:58.272 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:58.272 DEBUG:boto:Method: GET 2023-11-02T18:45:58.273 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:45:58.273 DEBUG:boto:Data: 2023-11-02T18:45:58.274 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.275 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.275 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.275 DEBUG:boto:Params: {} 2023-11-02T18:45:58.276 DEBUG:boto:Token: None 2023-11-02T18:45:58.277 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.277 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007ef46e03c6f6689-006543ee66-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.283 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.284 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.285 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.285 DEBUG:boto:path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:58.286 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj46 2023-11-02T18:45:58.286 DEBUG:boto:Method: GET 2023-11-02T18:45:58.287 DEBUG:boto:Path: /ibtljn-59/?prefix=obj46 2023-11-02T18:45:58.287 DEBUG:boto:Data: 2023-11-02T18:45:58.288 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.288 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.289 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.289 DEBUG:boto:Params: {} 2023-11-02T18:45:58.290 DEBUG:boto:Token: None 2023-11-02T18:45:58.290 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.291 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003785a43a6fac24a8-006543ee66-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.302 DEBUG:boto:b'ibtljn-59obj461000falseobj462023-11-02T18:39:10.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:45:58.303 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:58.304 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:58.304 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.305 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.305 DEBUG:boto:Method: GET 2023-11-02T18:45:58.306 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.306 DEBUG:boto:Data: 2023-11-02T18:45:58.307 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.307 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.308 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.308 DEBUG:boto:Params: {} 2023-11-02T18:45:58.309 DEBUG:boto:Token: None 2023-11-02T18:45:58.309 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.310 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vnqGetGQ6uYB/IyugV1A6RVJ2a0= 2023-11-02T18:45:58.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vnqGetGQ6uYB/IyugV1A6RVJ2a0=', 'Content-Length': '0'} 2023-11-02T18:45:58.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d886152d339bd63d-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.354 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:58.355 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:58.356 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.356 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.357 DEBUG:boto:Method: GET 2023-11-02T18:45:58.358 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.358 DEBUG:boto:Data: 2023-11-02T18:45:58.359 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.359 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.360 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.360 DEBUG:boto:Params: {} 2023-11-02T18:45:58.361 DEBUG:boto:Token: None 2023-11-02T18:45:58.361 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.362 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vnqGetGQ6uYB/IyugV1A6RVJ2a0= 2023-11-02T18:45:58.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vnqGetGQ6uYB/IyugV1A6RVJ2a0=', 'Content-Length': '0'} 2023-11-02T18:45:58.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b65095df6fff794d-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.366 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:58.367 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:58.367 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.368 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.368 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.368 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.369 DEBUG:boto:Data: 2023-11-02T18:45:58.369 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.370 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.371 DEBUG:boto:Params: {} 2023-11-02T18:45:58.371 DEBUG:boto:Token: None 2023-11-02T18:45:58.372 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SaGBOJ6iqfMJDYtepCp7hI6cIJA= 2023-11-02T18:45:58.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SaGBOJ6iqfMJDYtepCp7hI6cIJA=', 'Content-Length': '0'} 2023-11-02T18:45:58.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc175e807420747a-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.376 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:58.377 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:58.377 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.377 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.378 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.378 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.379 DEBUG:boto:Data: 2023-11-02T18:45:58.379 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.379 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.380 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.380 DEBUG:boto:Params: {} 2023-11-02T18:45:58.381 DEBUG:boto:Token: None 2023-11-02T18:45:58.381 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.382 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SaGBOJ6iqfMJDYtepCp7hI6cIJA= 2023-11-02T18:45:58.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SaGBOJ6iqfMJDYtepCp7hI6cIJA=', 'Content-Length': '0'} 2023-11-02T18:45:58.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fbd350b7ed69a07-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:45:58.386 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:58.387 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:58.387 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.388 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.388 DEBUG:boto:Method: GET 2023-11-02T18:45:58.388 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.389 DEBUG:boto:Data: 2023-11-02T18:45:58.389 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.390 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.390 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.391 DEBUG:boto:Params: {} 2023-11-02T18:45:58.391 DEBUG:boto:Token: None 2023-11-02T18:45:58.392 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.392 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vnqGetGQ6uYB/IyugV1A6RVJ2a0= 2023-11-02T18:45:58.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vnqGetGQ6uYB/IyugV1A6RVJ2a0=', 'Content-Length': '0'} 2023-11-02T18:45:58.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055d8859f1cf85b9c-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.396 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:58.397 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:58.397 DEBUG:boto:path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.397 DEBUG:boto:auth_path=/ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.398 DEBUG:boto:Method: GET 2023-11-02T18:45:58.398 DEBUG:boto:Path: /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.399 DEBUG:boto:Data: 2023-11-02T18:45:58.399 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.400 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.400 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.400 DEBUG:boto:Params: {} 2023-11-02T18:45:58.401 DEBUG:boto:Token: None 2023-11-02T18:45:58.401 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj51?versionId=null 2023-11-02T18:45:58.402 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vnqGetGQ6uYB/IyugV1A6RVJ2a0= 2023-11-02T18:45:58.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vnqGetGQ6uYB/IyugV1A6RVJ2a0=', 'Content-Length': '0'} 2023-11-02T18:45:58.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002db32a1fe45cd68-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.407 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.407 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.407 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:58.408 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:58.408 DEBUG:boto:Method: GET 2023-11-02T18:45:58.409 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:45:58.409 DEBUG:boto:Data: 2023-11-02T18:45:58.410 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.410 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.410 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.411 DEBUG:boto:Params: {} 2023-11-02T18:45:58.411 DEBUG:boto:Token: None 2023-11-02T18:45:58.412 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f741c698a1a74156-006543ee66-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.462 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.463 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.464 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.464 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:58.465 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:58.465 DEBUG:boto:Method: GET 2023-11-02T18:45:58.465 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:45:58.466 DEBUG:boto:Data: 2023-11-02T18:45:58.466 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.467 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.467 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.468 DEBUG:boto:Params: {} 2023-11-02T18:45:58.468 DEBUG:boto:Token: None 2023-11-02T18:45:58.469 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.469 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008426f404a22329e5-006543ee66-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.481 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.481 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:58.482 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:58.482 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.483 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:45:58.483 DEBUG:boto:Data: 2023-11-02T18:45:58.483 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.485 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.486 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.486 DEBUG:boto:Params: {} 2023-11-02T18:45:58.487 DEBUG:boto:Token: None 2023-11-02T18:45:58.487 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj51 2023-11-02T18:45:58.487 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VSAU5j2Kn6fhHMc+VDizQGzjhHk= 2023-11-02T18:45:58.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VSAU5j2Kn6fhHMc+VDizQGzjhHk=', 'Content-Length': '0'} 2023-11-02T18:45:58.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d27c82408ec0f06-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.491 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:58.491 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:58.492 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.492 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:45:58.492 DEBUG:boto:Data: 2023-11-02T18:45:58.493 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.494 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.494 DEBUG:boto:Params: {} 2023-11-02T18:45:58.495 DEBUG:boto:Token: None 2023-11-02T18:45:58.495 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj51 2023-11-02T18:45:58.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VSAU5j2Kn6fhHMc+VDizQGzjhHk= 2023-11-02T18:45:58.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VSAU5j2Kn6fhHMc+VDizQGzjhHk=', 'Content-Length': '0'} 2023-11-02T18:45:58.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b98cae3e1fd72515-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:45:58.500 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:58.500 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:58.500 DEBUG:boto:Method: GET 2023-11-02T18:45:58.501 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:45:58.501 DEBUG:boto:Data: 2023-11-02T18:45:58.502 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.502 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.503 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.503 DEBUG:boto:Params: {} 2023-11-02T18:45:58.504 DEBUG:boto:Token: None 2023-11-02T18:45:58.504 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj51 2023-11-02T18:45:58.505 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+l9Gp4k4e5CrQ3SKGpduaPWs3M= 2023-11-02T18:45:58.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+l9Gp4k4e5CrQ3SKGpduaPWs3M=', 'Content-Length': '0'} 2023-11-02T18:45:58.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e438afacfa4c0b3d-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.508 DEBUG:boto:path=/ibtljn-59/obj51 2023-11-02T18:45:58.509 DEBUG:boto:auth_path=/ibtljn-59/obj51 2023-11-02T18:45:58.509 DEBUG:boto:Method: GET 2023-11-02T18:45:58.510 DEBUG:boto:Path: /ibtljn-59/obj51 2023-11-02T18:45:58.510 DEBUG:boto:Data: 2023-11-02T18:45:58.510 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.511 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.511 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.512 DEBUG:boto:Params: {} 2023-11-02T18:45:58.512 DEBUG:boto:Token: None 2023-11-02T18:45:58.513 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj51 2023-11-02T18:45:58.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+l9Gp4k4e5CrQ3SKGpduaPWs3M= 2023-11-02T18:45:58.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+l9Gp4k4e5CrQ3SKGpduaPWs3M=', 'Content-Length': '0'} 2023-11-02T18:45:58.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcaa97c4d4fc7ca8-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.562 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.563 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.563 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:58.564 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:58.564 DEBUG:boto:Method: GET 2023-11-02T18:45:58.565 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:45:58.565 DEBUG:boto:Data: 2023-11-02T18:45:58.566 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.566 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.567 DEBUG:boto:Params: {} 2023-11-02T18:45:58.567 DEBUG:boto:Token: None 2023-11-02T18:45:58.568 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.568 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035f3c34ed5689c79-006543ee66-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.575 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.576 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.576 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.577 DEBUG:boto:path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:58.577 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj51 2023-11-02T18:45:58.578 DEBUG:boto:Method: GET 2023-11-02T18:45:58.578 DEBUG:boto:Path: /ibtljn-59/?prefix=obj51 2023-11-02T18:45:58.578 DEBUG:boto:Data: 2023-11-02T18:45:58.579 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.579 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.580 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.580 DEBUG:boto:Params: {} 2023-11-02T18:45:58.581 DEBUG:boto:Token: None 2023-11-02T18:45:58.581 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.582 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7bf089cbea45295-006543ee66-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.593 DEBUG:boto:b'ibtljn-59obj511000falseobj512023-11-02T18:32:46.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:45:58.595 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:58.595 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:58.595 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.596 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.596 DEBUG:boto:Method: GET 2023-11-02T18:45:58.597 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.597 DEBUG:boto:Data: 2023-11-02T18:45:58.598 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.598 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.599 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.599 DEBUG:boto:Params: {} 2023-11-02T18:45:58.600 DEBUG:boto:Token: None 2023-11-02T18:45:58.601 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.601 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MoJL+CORF/ZSHSvaf5U++FtxdJ0= 2023-11-02T18:45:58.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MoJL+CORF/ZSHSvaf5U++FtxdJ0=', 'Content-Length': '0'} 2023-11-02T18:45:58.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d99b9ae80742eec2-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.646 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:58.647 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:58.648 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.648 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.649 DEBUG:boto:Method: GET 2023-11-02T18:45:58.649 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.649 DEBUG:boto:Data: 2023-11-02T18:45:58.651 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.652 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.652 DEBUG:boto:Params: {} 2023-11-02T18:45:58.653 DEBUG:boto:Token: None 2023-11-02T18:45:58.654 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.654 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MoJL+CORF/ZSHSvaf5U++FtxdJ0= 2023-11-02T18:45:58.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MoJL+CORF/ZSHSvaf5U++FtxdJ0=', 'Content-Length': '0'} 2023-11-02T18:45:58.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ce88dac1bba01bb-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.659 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:58.659 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:58.660 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.660 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.660 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.661 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.661 DEBUG:boto:Data: 2023-11-02T18:45:58.662 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.662 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.663 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.663 DEBUG:boto:Params: {} 2023-11-02T18:45:58.664 DEBUG:boto:Token: None 2023-11-02T18:45:58.664 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.665 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lnwOuUWLigMkZpEr8iA8PAzZMwM= 2023-11-02T18:45:58.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lnwOuUWLigMkZpEr8iA8PAzZMwM=', 'Content-Length': '0'} 2023-11-02T18:45:58.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e6b6d50c3fe6d44-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.669 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:58.669 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:58.669 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.670 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.670 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.671 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.671 DEBUG:boto:Data: 2023-11-02T18:45:58.671 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.672 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.672 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.673 DEBUG:boto:Params: {} 2023-11-02T18:45:58.673 DEBUG:boto:Token: None 2023-11-02T18:45:58.674 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.674 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lnwOuUWLigMkZpEr8iA8PAzZMwM= 2023-11-02T18:45:58.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lnwOuUWLigMkZpEr8iA8PAzZMwM=', 'Content-Length': '0'} 2023-11-02T18:45:58.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c879cabac5f56788-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:45:58.678 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:58.679 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:58.679 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.680 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.680 DEBUG:boto:Method: GET 2023-11-02T18:45:58.681 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.681 DEBUG:boto:Data: 2023-11-02T18:45:58.682 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.682 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.683 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.683 DEBUG:boto:Params: {} 2023-11-02T18:45:58.683 DEBUG:boto:Token: None 2023-11-02T18:45:58.684 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.684 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MoJL+CORF/ZSHSvaf5U++FtxdJ0= 2023-11-02T18:45:58.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MoJL+CORF/ZSHSvaf5U++FtxdJ0=', 'Content-Length': '0'} 2023-11-02T18:45:58.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000926bb56070f53120-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.689 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:58.689 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:58.690 DEBUG:boto:path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.690 DEBUG:boto:auth_path=/ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.691 DEBUG:boto:Method: GET 2023-11-02T18:45:58.693 DEBUG:boto:Path: /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.694 DEBUG:boto:Data: 2023-11-02T18:45:58.694 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.695 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.696 DEBUG:boto:Params: {} 2023-11-02T18:45:58.696 DEBUG:boto:Token: None 2023-11-02T18:45:58.697 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj55?versionId=null 2023-11-02T18:45:58.697 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MoJL+CORF/ZSHSvaf5U++FtxdJ0= 2023-11-02T18:45:58.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MoJL+CORF/ZSHSvaf5U++FtxdJ0=', 'Content-Length': '0'} 2023-11-02T18:45:58.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067c13c555fbca23c-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.701 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.702 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.702 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:58.702 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:58.703 DEBUG:boto:Method: GET 2023-11-02T18:45:58.703 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:45:58.704 DEBUG:boto:Data: 2023-11-02T18:45:58.704 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.705 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.705 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.706 DEBUG:boto:Params: {} 2023-11-02T18:45:58.706 DEBUG:boto:Token: None 2023-11-02T18:45:58.707 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.707 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057d1905e9f75777d-006543ee66-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.754 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.755 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.756 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.756 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:58.757 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:58.757 DEBUG:boto:Method: GET 2023-11-02T18:45:58.757 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:45:58.758 DEBUG:boto:Data: 2023-11-02T18:45:58.758 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.759 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.759 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.760 DEBUG:boto:Params: {} 2023-11-02T18:45:58.760 DEBUG:boto:Token: None 2023-11-02T18:45:58.761 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.761 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014a858409dd2dafb-006543ee66-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.772 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.773 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:58.773 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:58.773 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.774 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:45:58.774 DEBUG:boto:Data: 2023-11-02T18:45:58.775 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.775 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.776 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.776 DEBUG:boto:Params: {} 2023-11-02T18:45:58.777 DEBUG:boto:Token: None 2023-11-02T18:45:58.777 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj55 2023-11-02T18:45:58.778 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:q5YtOIJD47VNDEHBhJY9h1hnzmQ= 2023-11-02T18:45:58.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:q5YtOIJD47VNDEHBhJY9h1hnzmQ=', 'Content-Length': '0'} 2023-11-02T18:45:58.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4045a3e4373de00-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.781 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:58.781 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:58.782 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.782 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:45:58.783 DEBUG:boto:Data: 2023-11-02T18:45:58.783 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.784 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.784 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.784 DEBUG:boto:Params: {} 2023-11-02T18:45:58.785 DEBUG:boto:Token: None 2023-11-02T18:45:58.785 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj55 2023-11-02T18:45:58.786 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:q5YtOIJD47VNDEHBhJY9h1hnzmQ= 2023-11-02T18:45:58.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:q5YtOIJD47VNDEHBhJY9h1hnzmQ=', 'Content-Length': '0'} 2023-11-02T18:45:58.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031b577d73ac8a114-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:45:58.790 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:58.790 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:58.791 DEBUG:boto:Method: GET 2023-11-02T18:45:58.791 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:45:58.791 DEBUG:boto:Data: 2023-11-02T18:45:58.792 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.792 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.793 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.793 DEBUG:boto:Params: {} 2023-11-02T18:45:58.794 DEBUG:boto:Token: None 2023-11-02T18:45:58.794 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj55 2023-11-02T18:45:58.795 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lXZwxpUTOZCynu3b5mYf6g+3sSY= 2023-11-02T18:45:58.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lXZwxpUTOZCynu3b5mYf6g+3sSY=', 'Content-Length': '0'} 2023-11-02T18:45:58.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061e01334749ad5aa-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.799 DEBUG:boto:path=/ibtljn-59/obj55 2023-11-02T18:45:58.799 DEBUG:boto:auth_path=/ibtljn-59/obj55 2023-11-02T18:45:58.800 DEBUG:boto:Method: GET 2023-11-02T18:45:58.800 DEBUG:boto:Path: /ibtljn-59/obj55 2023-11-02T18:45:58.801 DEBUG:boto:Data: 2023-11-02T18:45:58.801 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.802 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.802 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.802 DEBUG:boto:Params: {} 2023-11-02T18:45:58.803 DEBUG:boto:Token: None 2023-11-02T18:45:58.803 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj55 2023-11-02T18:45:58.804 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lXZwxpUTOZCynu3b5mYf6g+3sSY= 2023-11-02T18:45:58.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lXZwxpUTOZCynu3b5mYf6g+3sSY=', 'Content-Length': '0'} 2023-11-02T18:45:58.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071663c4a98694998-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.808 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.809 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.809 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:58.810 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:58.810 DEBUG:boto:Method: GET 2023-11-02T18:45:58.811 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:45:58.811 DEBUG:boto:Data: 2023-11-02T18:45:58.812 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.812 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.813 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.813 DEBUG:boto:Params: {} 2023-11-02T18:45:58.814 DEBUG:boto:Token: None 2023-11-02T18:45:58.814 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.815 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e12d4de3c5f7a75e-006543ee66-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.820 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.821 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.821 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.822 DEBUG:boto:path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:58.822 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj55 2023-11-02T18:45:58.823 DEBUG:boto:Method: GET 2023-11-02T18:45:58.823 DEBUG:boto:Path: /ibtljn-59/?prefix=obj55 2023-11-02T18:45:58.824 DEBUG:boto:Data: 2023-11-02T18:45:58.824 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.825 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.825 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.826 DEBUG:boto:Params: {} 2023-11-02T18:45:58.826 DEBUG:boto:Token: None 2023-11-02T18:45:58.827 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.827 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:58.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d58b2f57f380d46-006543ee66-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.882 DEBUG:boto:b'ibtljn-59obj551000falseobj552023-11-02T18:32:14.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:58.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:45:58.884 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:58.884 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:58.885 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.885 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.886 DEBUG:boto:Method: GET 2023-11-02T18:45:58.886 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.887 DEBUG:boto:Data: 2023-11-02T18:45:58.887 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.887 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.888 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.888 DEBUG:boto:Params: {} 2023-11-02T18:45:58.889 DEBUG:boto:Token: None 2023-11-02T18:45:58.890 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.890 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XEhMOC/lUoW5+ifLYSjxa6E/n7A= 2023-11-02T18:45:58.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XEhMOC/lUoW5+ifLYSjxa6E/n7A=', 'Content-Length': '0'} 2023-11-02T18:45:58.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f3847cc1304315a-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.895 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:58.896 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:58.896 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.897 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.897 DEBUG:boto:Method: GET 2023-11-02T18:45:58.897 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.898 DEBUG:boto:Data: 2023-11-02T18:45:58.899 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.899 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.899 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.900 DEBUG:boto:Params: {} 2023-11-02T18:45:58.900 DEBUG:boto:Token: None 2023-11-02T18:45:58.901 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.902 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XEhMOC/lUoW5+ifLYSjxa6E/n7A= 2023-11-02T18:45:58.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XEhMOC/lUoW5+ifLYSjxa6E/n7A=', 'Content-Length': '0'} 2023-11-02T18:45:58.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0c5474f6c87c286-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.906 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:58.907 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:58.907 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.908 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.908 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.909 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.909 DEBUG:boto:Data: 2023-11-02T18:45:58.910 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.910 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.911 DEBUG:boto:Params: {} 2023-11-02T18:45:58.912 DEBUG:boto:Token: None 2023-11-02T18:45:58.912 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.913 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GOnd7jAGMDOj/jdBmgASgh9hFNA= 2023-11-02T18:45:58.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GOnd7jAGMDOj/jdBmgASgh9hFNA=', 'Content-Length': '0'} 2023-11-02T18:45:58.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052b7fe3c9049ce6e-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.917 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:58.917 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:58.918 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.918 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.919 DEBUG:boto:Method: HEAD 2023-11-02T18:45:58.919 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.919 DEBUG:boto:Data: 2023-11-02T18:45:58.920 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.920 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.921 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.921 DEBUG:boto:Params: {} 2023-11-02T18:45:58.922 DEBUG:boto:Token: None 2023-11-02T18:45:58.922 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.923 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GOnd7jAGMDOj/jdBmgASgh9hFNA= 2023-11-02T18:45:58.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GOnd7jAGMDOj/jdBmgASgh9hFNA=', 'Content-Length': '0'} 2023-11-02T18:45:58.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fb19245872f82fd-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:45:58.927 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:58.927 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:58.928 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.928 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.929 DEBUG:boto:Method: GET 2023-11-02T18:45:58.929 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.930 DEBUG:boto:Data: 2023-11-02T18:45:58.930 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.931 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.931 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.931 DEBUG:boto:Params: {} 2023-11-02T18:45:58.932 DEBUG:boto:Token: None 2023-11-02T18:45:58.932 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XEhMOC/lUoW5+ifLYSjxa6E/n7A= 2023-11-02T18:45:58.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XEhMOC/lUoW5+ifLYSjxa6E/n7A=', 'Content-Length': '0'} 2023-11-02T18:45:58.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000704d8643221ed3cd-006543ee66-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.978 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:58.979 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:58.979 DEBUG:boto:path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.980 DEBUG:boto:auth_path=/ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.980 DEBUG:boto:Method: GET 2023-11-02T18:45:58.981 DEBUG:boto:Path: /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.981 DEBUG:boto:Data: 2023-11-02T18:45:58.982 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.982 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:58.982 DEBUG:boto:Port: 8000 2023-11-02T18:45:58.983 DEBUG:boto:Params: {} 2023-11-02T18:45:58.983 DEBUG:boto:Token: None 2023-11-02T18:45:58.984 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/obj57?versionId=null 2023-11-02T18:45:58.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XEhMOC/lUoW5+ifLYSjxa6E/n7A= 2023-11-02T18:45:58.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XEhMOC/lUoW5+ifLYSjxa6E/n7A=', 'Content-Length': '0'} 2023-11-02T18:45:58.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078afcaf3d7ad4a91-006543ee66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:58.989 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:58.989 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:58.990 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:58.990 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:58.991 DEBUG:boto:Method: GET 2023-11-02T18:45:58.991 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:45:58.991 DEBUG:boto:Data: 2023-11-02T18:45:58.992 DEBUG:boto:Headers: {} 2023-11-02T18:45:58.992 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:58.993 DEBUG:boto:Port: 8001 2023-11-02T18:45:58.993 DEBUG:boto:Params: {} 2023-11-02T18:45:58.994 DEBUG:boto:Token: None 2023-11-02T18:45:58.994 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:58 GMT /ibtljn-59/ 2023-11-02T18:45:58.995 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y= 2023-11-02T18:45:58.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O1mulQV0kOYbDv0r63xXoR3x42Y=', 'Content-Length': '0'} 2023-11-02T18:45:59.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee927bf9ebcc8b1d-006543ee66-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.001 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.002 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.002 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.003 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:59.003 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:59.004 DEBUG:boto:Method: GET 2023-11-02T18:45:59.004 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:45:59.005 DEBUG:boto:Data: 2023-11-02T18:45:59.005 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.006 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.006 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.007 DEBUG:boto:Params: {} 2023-11-02T18:45:59.007 DEBUG:boto:Token: None 2023-11-02T18:45:59.008 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.008 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044e6dcac8da95ca2-006543ee67-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.019 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.020 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:59.021 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:59.021 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.021 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:45:59.022 DEBUG:boto:Data: 2023-11-02T18:45:59.022 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.023 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.023 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.023 DEBUG:boto:Params: {} 2023-11-02T18:45:59.024 DEBUG:boto:Token: None 2023-11-02T18:45:59.024 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj57 2023-11-02T18:45:59.025 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3K77dp0NVbpJGVLbTz3KBACsgLU= 2023-11-02T18:45:59.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3K77dp0NVbpJGVLbTz3KBACsgLU=', 'Content-Length': '0'} 2023-11-02T18:45:59.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef93df77ec1cc891-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.028 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:59.029 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:59.029 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.030 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:45:59.030 DEBUG:boto:Data: 2023-11-02T18:45:59.030 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.031 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.031 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.032 DEBUG:boto:Params: {} 2023-11-02T18:45:59.032 DEBUG:boto:Token: None 2023-11-02T18:45:59.033 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj57 2023-11-02T18:45:59.033 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3K77dp0NVbpJGVLbTz3KBACsgLU= 2023-11-02T18:45:59.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3K77dp0NVbpJGVLbTz3KBACsgLU=', 'Content-Length': '0'} 2023-11-02T18:45:59.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e8ee275d33028b1-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:45:59.037 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:59.038 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:59.038 DEBUG:boto:Method: GET 2023-11-02T18:45:59.039 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:45:59.039 DEBUG:boto:Data: 2023-11-02T18:45:59.039 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.040 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.040 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.041 DEBUG:boto:Params: {} 2023-11-02T18:45:59.041 DEBUG:boto:Token: None 2023-11-02T18:45:59.042 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj57 2023-11-02T18:45:59.042 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p1DDREHLRVzvGYmCZJEf1YpWXjI= 2023-11-02T18:45:59.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p1DDREHLRVzvGYmCZJEf1YpWXjI=', 'Content-Length': '0'} 2023-11-02T18:45:59.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3d7c7fa67b7c392-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.046 DEBUG:boto:path=/ibtljn-59/obj57 2023-11-02T18:45:59.047 DEBUG:boto:auth_path=/ibtljn-59/obj57 2023-11-02T18:45:59.047 DEBUG:boto:Method: GET 2023-11-02T18:45:59.048 DEBUG:boto:Path: /ibtljn-59/obj57 2023-11-02T18:45:59.048 DEBUG:boto:Data: 2023-11-02T18:45:59.049 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.049 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.050 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.050 DEBUG:boto:Params: {} 2023-11-02T18:45:59.051 DEBUG:boto:Token: None 2023-11-02T18:45:59.051 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj57 2023-11-02T18:45:59.052 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p1DDREHLRVzvGYmCZJEf1YpWXjI= 2023-11-02T18:45:59.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p1DDREHLRVzvGYmCZJEf1YpWXjI=', 'Content-Length': '0'} 2023-11-02T18:45:59.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df1e2c6def789725-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.102 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.103 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.104 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:59.104 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:59.105 DEBUG:boto:Method: GET 2023-11-02T18:45:59.105 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:45:59.106 DEBUG:boto:Data: 2023-11-02T18:45:59.106 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.107 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.107 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.108 DEBUG:boto:Params: {} 2023-11-02T18:45:59.108 DEBUG:boto:Token: None 2023-11-02T18:45:59.109 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.109 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f07c3c7a96dff3b8-006543ee67-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.115 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.116 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.117 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.117 DEBUG:boto:path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:59.118 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj57 2023-11-02T18:45:59.118 DEBUG:boto:Method: GET 2023-11-02T18:45:59.118 DEBUG:boto:Path: /ibtljn-59/?prefix=obj57 2023-11-02T18:45:59.119 DEBUG:boto:Data: 2023-11-02T18:45:59.120 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.120 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.120 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.121 DEBUG:boto:Params: {} 2023-11-02T18:45:59.122 DEBUG:boto:Token: None 2023-11-02T18:45:59.122 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.123 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2ea408c16c152d-006543ee67-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.135 DEBUG:boto:b'ibtljn-59obj571000falseobj572023-11-02T18:34:50.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:45:59.136 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:59.137 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:59.137 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.138 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.138 DEBUG:boto:Method: GET 2023-11-02T18:45:59.138 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.139 DEBUG:boto:Data: 2023-11-02T18:45:59.139 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.140 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.140 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.141 DEBUG:boto:Params: {} 2023-11-02T18:45:59.141 DEBUG:boto:Token: None 2023-11-02T18:45:59.142 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.142 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sxkPx/WbJjscInCowHe09RDcO5Q= 2023-11-02T18:45:59.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sxkPx/WbJjscInCowHe09RDcO5Q=', 'Content-Length': '0'} 2023-11-02T18:45:59.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a87032881a56289-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.190 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:59.191 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:59.191 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.192 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.192 DEBUG:boto:Method: GET 2023-11-02T18:45:59.193 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.193 DEBUG:boto:Data: 2023-11-02T18:45:59.194 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.194 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.195 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.195 DEBUG:boto:Params: {} 2023-11-02T18:45:59.195 DEBUG:boto:Token: None 2023-11-02T18:45:59.196 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.196 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sxkPx/WbJjscInCowHe09RDcO5Q= 2023-11-02T18:45:59.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sxkPx/WbJjscInCowHe09RDcO5Q=', 'Content-Length': '0'} 2023-11-02T18:45:59.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067f1e621d4b15d0b-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.201 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:59.201 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:59.202 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.202 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.202 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.203 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.203 DEBUG:boto:Data: 2023-11-02T18:45:59.204 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.204 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.205 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.205 DEBUG:boto:Params: {} 2023-11-02T18:45:59.206 DEBUG:boto:Token: None 2023-11-02T18:45:59.206 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.207 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZqXOR8fkEYVTZ8UdH5LY3jhXgo8= 2023-11-02T18:45:59.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZqXOR8fkEYVTZ8UdH5LY3jhXgo8=', 'Content-Length': '0'} 2023-11-02T18:45:59.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008936fb4e4c7fe6d5-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.211 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:59.211 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:59.212 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.212 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.213 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.213 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.213 DEBUG:boto:Data: 2023-11-02T18:45:59.214 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.214 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.215 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.215 DEBUG:boto:Params: {} 2023-11-02T18:45:59.216 DEBUG:boto:Token: None 2023-11-02T18:45:59.216 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZqXOR8fkEYVTZ8UdH5LY3jhXgo8= 2023-11-02T18:45:59.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZqXOR8fkEYVTZ8UdH5LY3jhXgo8=', 'Content-Length': '0'} 2023-11-02T18:45:59.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab484e3682ef3c22-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:45:59.221 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:59.222 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:59.222 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.223 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.223 DEBUG:boto:Method: GET 2023-11-02T18:45:59.223 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.224 DEBUG:boto:Data: 2023-11-02T18:45:59.224 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.225 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.225 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.226 DEBUG:boto:Params: {} 2023-11-02T18:45:59.226 DEBUG:boto:Token: None 2023-11-02T18:45:59.227 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.227 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sxkPx/WbJjscInCowHe09RDcO5Q= 2023-11-02T18:45:59.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sxkPx/WbJjscInCowHe09RDcO5Q=', 'Content-Length': '0'} 2023-11-02T18:45:59.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006b3493eae4e7016-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.232 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:59.232 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:59.233 DEBUG:boto:path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.233 DEBUG:boto:auth_path=/ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.233 DEBUG:boto:Method: GET 2023-11-02T18:45:59.234 DEBUG:boto:Path: /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.234 DEBUG:boto:Data: 2023-11-02T18:45:59.235 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.235 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.236 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.236 DEBUG:boto:Params: {} 2023-11-02T18:45:59.237 DEBUG:boto:Token: None 2023-11-02T18:45:59.237 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj58?versionId=null 2023-11-02T18:45:59.238 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sxkPx/WbJjscInCowHe09RDcO5Q= 2023-11-02T18:45:59.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sxkPx/WbJjscInCowHe09RDcO5Q=', 'Content-Length': '0'} 2023-11-02T18:45:59.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad9b7761bd0a25f2-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.282 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.283 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.283 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:59.284 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:59.284 DEBUG:boto:Method: GET 2023-11-02T18:45:59.285 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:45:59.285 DEBUG:boto:Data: 2023-11-02T18:45:59.286 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.286 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.287 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.287 DEBUG:boto:Params: {} 2023-11-02T18:45:59.288 DEBUG:boto:Token: None 2023-11-02T18:45:59.288 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.289 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a23780d34b2060d-006543ee67-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.295 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.296 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.296 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.297 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:59.297 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:59.298 DEBUG:boto:Method: GET 2023-11-02T18:45:59.298 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:45:59.299 DEBUG:boto:Data: 2023-11-02T18:45:59.299 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.300 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.300 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.301 DEBUG:boto:Params: {} 2023-11-02T18:45:59.301 DEBUG:boto:Token: None 2023-11-02T18:45:59.302 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.302 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000674a60a16af44560-006543ee67-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.313 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.314 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:59.314 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:59.315 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.315 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:45:59.316 DEBUG:boto:Data: 2023-11-02T18:45:59.316 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.317 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.318 DEBUG:boto:Params: {} 2023-11-02T18:45:59.318 DEBUG:boto:Token: None 2023-11-02T18:45:59.319 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj58 2023-11-02T18:45:59.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sMsGv7mjzb1eRNQ7muReir2X4Mk= 2023-11-02T18:45:59.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sMsGv7mjzb1eRNQ7muReir2X4Mk=', 'Content-Length': '0'} 2023-11-02T18:45:59.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0c2f1d29a1c04f8-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.323 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:59.323 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:59.323 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.324 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:45:59.324 DEBUG:boto:Data: 2023-11-02T18:45:59.325 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.325 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.326 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.326 DEBUG:boto:Params: {} 2023-11-02T18:45:59.327 DEBUG:boto:Token: None 2023-11-02T18:45:59.327 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj58 2023-11-02T18:45:59.328 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sMsGv7mjzb1eRNQ7muReir2X4Mk= 2023-11-02T18:45:59.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sMsGv7mjzb1eRNQ7muReir2X4Mk=', 'Content-Length': '0'} 2023-11-02T18:45:59.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076cc1c804997c0be-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:45:59.332 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:59.333 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:59.333 DEBUG:boto:Method: GET 2023-11-02T18:45:59.334 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:45:59.334 DEBUG:boto:Data: 2023-11-02T18:45:59.334 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.335 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.335 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.336 DEBUG:boto:Params: {} 2023-11-02T18:45:59.336 DEBUG:boto:Token: None 2023-11-02T18:45:59.337 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj58 2023-11-02T18:45:59.338 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QS0nBCuvWDUaU+3Jc9GiHlX2qpI= 2023-11-02T18:45:59.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QS0nBCuvWDUaU+3Jc9GiHlX2qpI=', 'Content-Length': '0'} 2023-11-02T18:45:59.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f9097213d22469f-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.342 DEBUG:boto:path=/ibtljn-59/obj58 2023-11-02T18:45:59.342 DEBUG:boto:auth_path=/ibtljn-59/obj58 2023-11-02T18:45:59.343 DEBUG:boto:Method: GET 2023-11-02T18:45:59.343 DEBUG:boto:Path: /ibtljn-59/obj58 2023-11-02T18:45:59.343 DEBUG:boto:Data: 2023-11-02T18:45:59.344 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.344 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.345 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.345 DEBUG:boto:Params: {} 2023-11-02T18:45:59.346 DEBUG:boto:Token: None 2023-11-02T18:45:59.346 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj58 2023-11-02T18:45:59.347 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QS0nBCuvWDUaU+3Jc9GiHlX2qpI= 2023-11-02T18:45:59.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QS0nBCuvWDUaU+3Jc9GiHlX2qpI=', 'Content-Length': '0'} 2023-11-02T18:45:59.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e039ca40635a107-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.394 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.395 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.396 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:59.396 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:59.396 DEBUG:boto:Method: GET 2023-11-02T18:45:59.397 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:45:59.397 DEBUG:boto:Data: 2023-11-02T18:45:59.398 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.399 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.399 DEBUG:boto:Params: {} 2023-11-02T18:45:59.400 DEBUG:boto:Token: None 2023-11-02T18:45:59.400 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.401 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077bc3a466e3caef0-006543ee67-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.407 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.408 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.409 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.410 DEBUG:boto:path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:59.410 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj58 2023-11-02T18:45:59.410 DEBUG:boto:Method: GET 2023-11-02T18:45:59.411 DEBUG:boto:Path: /ibtljn-59/?prefix=obj58 2023-11-02T18:45:59.411 DEBUG:boto:Data: 2023-11-02T18:45:59.412 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.413 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.413 DEBUG:boto:Params: {} 2023-11-02T18:45:59.414 DEBUG:boto:Token: None 2023-11-02T18:45:59.414 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.415 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039a801ea238c4bb9-006543ee67-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.425 DEBUG:boto:b'ibtljn-59obj581000falseobj582023-11-02T18:40:31.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T18:45:59.426 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:59.427 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:59.427 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.428 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.428 DEBUG:boto:Method: GET 2023-11-02T18:45:59.429 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.429 DEBUG:boto:Data: 2023-11-02T18:45:59.430 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.430 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.431 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.431 DEBUG:boto:Params: {} 2023-11-02T18:45:59.432 DEBUG:boto:Token: None 2023-11-02T18:45:59.432 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.433 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QOryMAUtAvUAEzEbDqjpTwfEQwE= 2023-11-02T18:45:59.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QOryMAUtAvUAEzEbDqjpTwfEQwE=', 'Content-Length': '0'} 2023-11-02T18:45:59.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b758bf1a6974966a-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.478 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:59.479 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:59.479 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.480 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.480 DEBUG:boto:Method: GET 2023-11-02T18:45:59.481 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.481 DEBUG:boto:Data: 2023-11-02T18:45:59.482 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.482 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.483 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.483 DEBUG:boto:Params: {} 2023-11-02T18:45:59.484 DEBUG:boto:Token: None 2023-11-02T18:45:59.484 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.485 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QOryMAUtAvUAEzEbDqjpTwfEQwE= 2023-11-02T18:45:59.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QOryMAUtAvUAEzEbDqjpTwfEQwE=', 'Content-Length': '0'} 2023-11-02T18:45:59.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031103d6b7e207932-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.489 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:59.490 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:59.490 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.491 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.491 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.492 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.492 DEBUG:boto:Data: 2023-11-02T18:45:59.493 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.493 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.494 DEBUG:boto:Params: {} 2023-11-02T18:45:59.494 DEBUG:boto:Token: None 2023-11-02T18:45:59.495 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.495 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bw6LnPqRiprYdOJ6ABl7U1x9CnM= 2023-11-02T18:45:59.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bw6LnPqRiprYdOJ6ABl7U1x9CnM=', 'Content-Length': '0'} 2023-11-02T18:45:59.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cedae56fceb2070b-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.500 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:59.500 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:59.500 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.501 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.501 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.502 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.502 DEBUG:boto:Data: 2023-11-02T18:45:59.503 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.503 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.504 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.504 DEBUG:boto:Params: {} 2023-11-02T18:45:59.504 DEBUG:boto:Token: None 2023-11-02T18:45:59.505 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.505 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bw6LnPqRiprYdOJ6ABl7U1x9CnM= 2023-11-02T18:45:59.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bw6LnPqRiprYdOJ6ABl7U1x9CnM=', 'Content-Length': '0'} 2023-11-02T18:45:59.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b4df055a6848d0d-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T18:45:59.510 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:59.510 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:59.511 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.511 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.512 DEBUG:boto:Method: GET 2023-11-02T18:45:59.513 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.514 DEBUG:boto:Data: 2023-11-02T18:45:59.514 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.515 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.515 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.516 DEBUG:boto:Params: {} 2023-11-02T18:45:59.516 DEBUG:boto:Token: None 2023-11-02T18:45:59.517 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.517 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QOryMAUtAvUAEzEbDqjpTwfEQwE= 2023-11-02T18:45:59.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QOryMAUtAvUAEzEbDqjpTwfEQwE=', 'Content-Length': '0'} 2023-11-02T18:45:59.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b82694a1f938a6e8-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.522 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:59.522 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:59.523 DEBUG:boto:path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.523 DEBUG:boto:auth_path=/ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.524 DEBUG:boto:Method: GET 2023-11-02T18:45:59.524 DEBUG:boto:Path: /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.525 DEBUG:boto:Data: 2023-11-02T18:45:59.525 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.525 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.526 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.526 DEBUG:boto:Params: {} 2023-11-02T18:45:59.527 DEBUG:boto:Token: None 2023-11-02T18:45:59.528 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj62?versionId=null 2023-11-02T18:45:59.528 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QOryMAUtAvUAEzEbDqjpTwfEQwE= 2023-11-02T18:45:59.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QOryMAUtAvUAEzEbDqjpTwfEQwE=', 'Content-Length': '0'} 2023-11-02T18:45:59.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000070f26a24b2659b7-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.533 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.533 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.534 DEBUG:boto:path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:59.534 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:59.534 DEBUG:boto:Method: GET 2023-11-02T18:45:59.535 DEBUG:boto:Path: /ibtljn-59/?prefix=obj62 2023-11-02T18:45:59.535 DEBUG:boto:Data: 2023-11-02T18:45:59.536 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.536 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.537 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.537 DEBUG:boto:Params: {} 2023-11-02T18:45:59.538 DEBUG:boto:Token: None 2023-11-02T18:45:59.538 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.539 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005348faee6b81aed-006543ee67-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.545 DEBUG:boto:b'ibtljn-59obj621000falseobj622023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.546 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.547 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.547 DEBUG:boto:path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:59.547 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:59.548 DEBUG:boto:Method: GET 2023-11-02T18:45:59.548 DEBUG:boto:Path: /ibtljn-59/?prefix=obj62 2023-11-02T18:45:59.549 DEBUG:boto:Data: 2023-11-02T18:45:59.549 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.550 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.550 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.551 DEBUG:boto:Params: {} 2023-11-02T18:45:59.551 DEBUG:boto:Token: None 2023-11-02T18:45:59.552 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.552 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daa676f162ff6713-006543ee67-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.563 DEBUG:boto:b'ibtljn-59obj621000falseobj622023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.564 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:59.564 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:59.565 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.565 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:45:59.566 DEBUG:boto:Data: 2023-11-02T18:45:59.566 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.567 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.567 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.568 DEBUG:boto:Params: {} 2023-11-02T18:45:59.568 DEBUG:boto:Token: None 2023-11-02T18:45:59.569 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj62 2023-11-02T18:45:59.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+7LU/fW8SQUvtj6MjUdVaO02NZw= 2023-11-02T18:45:59.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+7LU/fW8SQUvtj6MjUdVaO02NZw=', 'Content-Length': '0'} 2023-11-02T18:45:59.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a9171a1db317e33-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.573 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:59.573 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:59.574 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.574 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:45:59.574 DEBUG:boto:Data: 2023-11-02T18:45:59.575 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.575 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.576 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.576 DEBUG:boto:Params: {} 2023-11-02T18:45:59.577 DEBUG:boto:Token: None 2023-11-02T18:45:59.577 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj62 2023-11-02T18:45:59.578 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+7LU/fW8SQUvtj6MjUdVaO02NZw= 2023-11-02T18:45:59.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+7LU/fW8SQUvtj6MjUdVaO02NZw=', 'Content-Length': '0'} 2023-11-02T18:45:59.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a81dbdc850b8ed71-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T18:45:59.582 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:59.583 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:59.583 DEBUG:boto:Method: GET 2023-11-02T18:45:59.584 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:45:59.584 DEBUG:boto:Data: 2023-11-02T18:45:59.585 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.585 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.586 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.586 DEBUG:boto:Params: {} 2023-11-02T18:45:59.587 DEBUG:boto:Token: None 2023-11-02T18:45:59.587 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj62 2023-11-02T18:45:59.588 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ncqeOXe1oXq7bneOKW0OwY4CTRA= 2023-11-02T18:45:59.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ncqeOXe1oXq7bneOKW0OwY4CTRA=', 'Content-Length': '0'} 2023-11-02T18:45:59.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5e1eb4d0ebcfb5-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.638 DEBUG:boto:path=/ibtljn-59/obj62 2023-11-02T18:45:59.639 DEBUG:boto:auth_path=/ibtljn-59/obj62 2023-11-02T18:45:59.640 DEBUG:boto:Method: GET 2023-11-02T18:45:59.640 DEBUG:boto:Path: /ibtljn-59/obj62 2023-11-02T18:45:59.640 DEBUG:boto:Data: 2023-11-02T18:45:59.641 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.641 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.642 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.642 DEBUG:boto:Params: {} 2023-11-02T18:45:59.643 DEBUG:boto:Token: None 2023-11-02T18:45:59.643 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj62 2023-11-02T18:45:59.644 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ncqeOXe1oXq7bneOKW0OwY4CTRA= 2023-11-02T18:45:59.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ncqeOXe1oXq7bneOKW0OwY4CTRA=', 'Content-Length': '0'} 2023-11-02T18:45:59.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002321fcb00c21c235-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.649 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.650 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.650 DEBUG:boto:path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:59.651 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:59.651 DEBUG:boto:Method: GET 2023-11-02T18:45:59.651 DEBUG:boto:Path: /ibtljn-59/?prefix=obj62 2023-11-02T18:45:59.652 DEBUG:boto:Data: 2023-11-02T18:45:59.652 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.653 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.653 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.654 DEBUG:boto:Params: {} 2023-11-02T18:45:59.654 DEBUG:boto:Token: None 2023-11-02T18:45:59.655 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020b73bfc38a9efb2-006543ee67-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.660 DEBUG:boto:b'ibtljn-59obj621000falseobj622023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.661 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.661 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.661 DEBUG:boto:path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:59.662 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj62 2023-11-02T18:45:59.662 DEBUG:boto:Method: GET 2023-11-02T18:45:59.663 DEBUG:boto:Path: /ibtljn-59/?prefix=obj62 2023-11-02T18:45:59.663 DEBUG:boto:Data: 2023-11-02T18:45:59.664 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.664 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.665 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.665 DEBUG:boto:Params: {} 2023-11-02T18:45:59.666 DEBUG:boto:Token: None 2023-11-02T18:45:59.666 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.667 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040f72467bf12c5a6-006543ee67-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.674 DEBUG:boto:b'ibtljn-59obj621000falseobj622023-11-02T18:42:34.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:45:59.675 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:59.676 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:59.676 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.677 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.678 DEBUG:boto:Method: GET 2023-11-02T18:45:59.678 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.679 DEBUG:boto:Data: 2023-11-02T18:45:59.679 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.680 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.681 DEBUG:boto:Params: {} 2023-11-02T18:45:59.681 DEBUG:boto:Token: None 2023-11-02T18:45:59.682 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.683 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IRHl11N8Hq+fxpj0icu0rw2E120= 2023-11-02T18:45:59.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IRHl11N8Hq+fxpj0icu0rw2E120=', 'Content-Length': '0'} 2023-11-02T18:45:59.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b5963e206070e4-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.688 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:59.689 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:59.689 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.690 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.690 DEBUG:boto:Method: GET 2023-11-02T18:45:59.691 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.692 DEBUG:boto:Data: 2023-11-02T18:45:59.693 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.693 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.694 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.695 DEBUG:boto:Params: {} 2023-11-02T18:45:59.695 DEBUG:boto:Token: None 2023-11-02T18:45:59.696 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.696 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IRHl11N8Hq+fxpj0icu0rw2E120= 2023-11-02T18:45:59.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IRHl11N8Hq+fxpj0icu0rw2E120=', 'Content-Length': '0'} 2023-11-02T18:45:59.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085d691667c6fcff4-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.701 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:59.701 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:59.702 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.702 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.703 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.703 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.704 DEBUG:boto:Data: 2023-11-02T18:45:59.704 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.704 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.705 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.705 DEBUG:boto:Params: {} 2023-11-02T18:45:59.706 DEBUG:boto:Token: None 2023-11-02T18:45:59.706 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.707 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UfLm8Ws6O9LAkCVzSzKiMtf6ACw= 2023-11-02T18:45:59.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UfLm8Ws6O9LAkCVzSzKiMtf6ACw=', 'Content-Length': '0'} 2023-11-02T18:45:59.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000809c06bf048fffac-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.711 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:59.711 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:59.712 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.712 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.713 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.713 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.714 DEBUG:boto:Data: 2023-11-02T18:45:59.714 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.715 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.715 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.716 DEBUG:boto:Params: {} 2023-11-02T18:45:59.716 DEBUG:boto:Token: None 2023-11-02T18:45:59.717 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.717 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UfLm8Ws6O9LAkCVzSzKiMtf6ACw= 2023-11-02T18:45:59.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UfLm8Ws6O9LAkCVzSzKiMtf6ACw=', 'Content-Length': '0'} 2023-11-02T18:45:59.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b55546ca0de29b94-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:45:59.722 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:59.722 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:59.723 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.723 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.724 DEBUG:boto:Method: GET 2023-11-02T18:45:59.724 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.725 DEBUG:boto:Data: 2023-11-02T18:45:59.725 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.726 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.726 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.727 DEBUG:boto:Params: {} 2023-11-02T18:45:59.727 DEBUG:boto:Token: None 2023-11-02T18:45:59.728 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.729 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IRHl11N8Hq+fxpj0icu0rw2E120= 2023-11-02T18:45:59.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IRHl11N8Hq+fxpj0icu0rw2E120=', 'Content-Length': '0'} 2023-11-02T18:45:59.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3a8fcf7884f1f3a-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.778 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:59.779 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:59.779 DEBUG:boto:path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.780 DEBUG:boto:auth_path=/ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.780 DEBUG:boto:Method: GET 2023-11-02T18:45:59.781 DEBUG:boto:Path: /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.781 DEBUG:boto:Data: 2023-11-02T18:45:59.782 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.782 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.782 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.783 DEBUG:boto:Params: {} 2023-11-02T18:45:59.783 DEBUG:boto:Token: None 2023-11-02T18:45:59.784 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj68?versionId=null 2023-11-02T18:45:59.784 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IRHl11N8Hq+fxpj0icu0rw2E120= 2023-11-02T18:45:59.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IRHl11N8Hq+fxpj0icu0rw2E120=', 'Content-Length': '0'} 2023-11-02T18:45:59.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000680fbe2a22de291f-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.789 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.790 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.790 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:59.790 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:59.791 DEBUG:boto:Method: GET 2023-11-02T18:45:59.791 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:45:59.792 DEBUG:boto:Data: 2023-11-02T18:45:59.792 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.793 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.793 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.794 DEBUG:boto:Params: {} 2023-11-02T18:45:59.794 DEBUG:boto:Token: None 2023-11-02T18:45:59.795 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.795 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000944647add30125ee-006543ee67-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.801 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.802 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.802 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.803 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:59.803 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:59.804 DEBUG:boto:Method: GET 2023-11-02T18:45:59.804 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:45:59.805 DEBUG:boto:Data: 2023-11-02T18:45:59.805 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.805 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.806 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.806 DEBUG:boto:Params: {} 2023-11-02T18:45:59.807 DEBUG:boto:Token: None 2023-11-02T18:45:59.807 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.808 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099c3ae8314dfe421-006543ee67-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.819 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.819 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:59.820 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:59.820 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.821 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:45:59.821 DEBUG:boto:Data: 2023-11-02T18:45:59.821 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.822 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.822 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.823 DEBUG:boto:Params: {} 2023-11-02T18:45:59.823 DEBUG:boto:Token: None 2023-11-02T18:45:59.824 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj68 2023-11-02T18:45:59.824 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KU3zfZtWC/pefznJntvUF7pDY1c= 2023-11-02T18:45:59.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KU3zfZtWC/pefznJntvUF7pDY1c=', 'Content-Length': '0'} 2023-11-02T18:45:59.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f53b2b522a653d0-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.827 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:59.828 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:59.828 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.829 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:45:59.829 DEBUG:boto:Data: 2023-11-02T18:45:59.830 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.830 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.831 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.831 DEBUG:boto:Params: {} 2023-11-02T18:45:59.831 DEBUG:boto:Token: None 2023-11-02T18:45:59.832 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj68 2023-11-02T18:45:59.832 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KU3zfZtWC/pefznJntvUF7pDY1c= 2023-11-02T18:45:59.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KU3zfZtWC/pefznJntvUF7pDY1c=', 'Content-Length': '0'} 2023-11-02T18:45:59.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000818ec74b03434d1d-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:45:59.836 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:59.837 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:59.837 DEBUG:boto:Method: GET 2023-11-02T18:45:59.838 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:45:59.838 DEBUG:boto:Data: 2023-11-02T18:45:59.839 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.839 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.840 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.840 DEBUG:boto:Params: {} 2023-11-02T18:45:59.840 DEBUG:boto:Token: None 2023-11-02T18:45:59.841 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj68 2023-11-02T18:45:59.841 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:21od3UCxRX/SfNOKls+2MtHBslo= 2023-11-02T18:45:59.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:21od3UCxRX/SfNOKls+2MtHBslo=', 'Content-Length': '0'} 2023-11-02T18:45:59.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c81cac64849a6c6-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.886 DEBUG:boto:path=/ibtljn-59/obj68 2023-11-02T18:45:59.887 DEBUG:boto:auth_path=/ibtljn-59/obj68 2023-11-02T18:45:59.887 DEBUG:boto:Method: GET 2023-11-02T18:45:59.888 DEBUG:boto:Path: /ibtljn-59/obj68 2023-11-02T18:45:59.888 DEBUG:boto:Data: 2023-11-02T18:45:59.889 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.889 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.890 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.890 DEBUG:boto:Params: {} 2023-11-02T18:45:59.891 DEBUG:boto:Token: None 2023-11-02T18:45:59.891 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj68 2023-11-02T18:45:59.892 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:21od3UCxRX/SfNOKls+2MtHBslo= 2023-11-02T18:45:59.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:21od3UCxRX/SfNOKls+2MtHBslo=', 'Content-Length': '0'} 2023-11-02T18:45:59.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000325c477b47419327-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.896 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.897 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.897 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:59.898 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:59.898 DEBUG:boto:Method: GET 2023-11-02T18:45:59.899 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:45:59.899 DEBUG:boto:Data: 2023-11-02T18:45:59.901 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.902 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.902 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.903 DEBUG:boto:Params: {} 2023-11-02T18:45:59.903 DEBUG:boto:Token: None 2023-11-02T18:45:59.904 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.904 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ae128d47cfcf555-006543ee67-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.910 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.911 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:45:59.912 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:45:59.912 DEBUG:boto:path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:59.912 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj68 2023-11-02T18:45:59.913 DEBUG:boto:Method: GET 2023-11-02T18:45:59.913 DEBUG:boto:Path: /ibtljn-59/?prefix=obj68 2023-11-02T18:45:59.914 DEBUG:boto:Data: 2023-11-02T18:45:59.914 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.915 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.915 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.916 DEBUG:boto:Params: {} 2023-11-02T18:45:59.916 DEBUG:boto:Token: None 2023-11-02T18:45:59.917 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/ 2023-11-02T18:45:59.918 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E= 2023-11-02T18:45:59.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A+q0FM2ZlbK1DA5HcnNRb6xga8E=', 'Content-Length': '0'} 2023-11-02T18:45:59.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003be2cef35f772d9c-006543ee67-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.974 DEBUG:boto:b'ibtljn-59obj681000falseobj682023-11-02T18:32:46.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:45:59.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:45:59.976 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:59.976 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:59.976 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:59.977 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:59.977 DEBUG:boto:Method: GET 2023-11-02T18:45:59.978 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:59.978 DEBUG:boto:Data: 2023-11-02T18:45:59.979 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.979 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:45:59.979 DEBUG:boto:Port: 8001 2023-11-02T18:45:59.980 DEBUG:boto:Params: {} 2023-11-02T18:45:59.980 DEBUG:boto:Token: None 2023-11-02T18:45:59.981 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:59.982 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LA5C4VUEx4ErNMFt6AhwPHxTlwc= 2023-11-02T18:45:59.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LA5C4VUEx4ErNMFt6AhwPHxTlwc=', 'Content-Length': '0'} 2023-11-02T18:45:59.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003164eadeeb5cb76c-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.986 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:59.987 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:59.987 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:59.988 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:59.988 DEBUG:boto:Method: GET 2023-11-02T18:45:59.989 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:59.989 DEBUG:boto:Data: 2023-11-02T18:45:59.990 DEBUG:boto:Headers: {} 2023-11-02T18:45:59.990 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:45:59.991 DEBUG:boto:Port: 8000 2023-11-02T18:45:59.991 DEBUG:boto:Params: {} 2023-11-02T18:45:59.991 DEBUG:boto:Token: None 2023-11-02T18:45:59.992 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:45:59 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:59.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LA5C4VUEx4ErNMFt6AhwPHxTlwc= 2023-11-02T18:45:59.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:45:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LA5C4VUEx4ErNMFt6AhwPHxTlwc=', 'Content-Length': '0'} 2023-11-02T18:45:59.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d22bf1bb3d7f839-006543ee67-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:45:59.997 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:45:59.997 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:45:59.998 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:59.998 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:45:59.998 DEBUG:boto:Method: HEAD 2023-11-02T18:45:59.999 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:45:59.999 DEBUG:boto:Data: 2023-11-02T18:46:00.000 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.000 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.001 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.001 DEBUG:boto:Params: {} 2023-11-02T18:46:00.001 DEBUG:boto:Token: None 2023-11-02T18:46:00.002 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.002 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qf5rBGT8pJ5B/9O4Pv2dpxmiGvg= 2023-11-02T18:46:00.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qf5rBGT8pJ5B/9O4Pv2dpxmiGvg=', 'Content-Length': '0'} 2023-11-02T18:46:00.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039e30da46b9dddb9-006543ee67-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:45:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.006 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:46:00.007 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:46:00.007 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.008 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.008 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.009 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.009 DEBUG:boto:Data: 2023-11-02T18:46:00.010 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.010 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.011 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.011 DEBUG:boto:Params: {} 2023-11-02T18:46:00.012 DEBUG:boto:Token: None 2023-11-02T18:46:00.012 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.013 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Qf5rBGT8pJ5B/9O4Pv2dpxmiGvg= 2023-11-02T18:46:00.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Qf5rBGT8pJ5B/9O4Pv2dpxmiGvg=', 'Content-Length': '0'} 2023-11-02T18:46:00.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008af38ef5cc914d88-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:46:00.017 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:46:00.017 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:46:00.018 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.018 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.019 DEBUG:boto:Method: GET 2023-11-02T18:46:00.019 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.020 DEBUG:boto:Data: 2023-11-02T18:46:00.020 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.021 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.021 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.022 DEBUG:boto:Params: {} 2023-11-02T18:46:00.022 DEBUG:boto:Token: None 2023-11-02T18:46:00.023 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.023 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kTRsbMX5Fg2cMMO1V92o3K0m+ig= 2023-11-02T18:46:00.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kTRsbMX5Fg2cMMO1V92o3K0m+ig=', 'Content-Length': '0'} 2023-11-02T18:46:00.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e00c67ff1cd144a9-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.028 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:46:00.028 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:46:00.029 DEBUG:boto:path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.029 DEBUG:boto:auth_path=/ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.030 DEBUG:boto:Method: GET 2023-11-02T18:46:00.030 DEBUG:boto:Path: /ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.031 DEBUG:boto:Data: 2023-11-02T18:46:00.031 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.031 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.032 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.032 DEBUG:boto:Params: {} 2023-11-02T18:46:00.033 DEBUG:boto:Token: None 2023-11-02T18:46:00.033 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj69?versionId=null 2023-11-02T18:46:00.034 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kTRsbMX5Fg2cMMO1V92o3K0m+ig= 2023-11-02T18:46:00.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kTRsbMX5Fg2cMMO1V92o3K0m+ig=', 'Content-Length': '0'} 2023-11-02T18:46:00.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0a6cd704b5a6459-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.078 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.079 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.080 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:46:00.080 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:46:00.080 DEBUG:boto:Method: GET 2023-11-02T18:46:00.081 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:46:00.081 DEBUG:boto:Data: 2023-11-02T18:46:00.082 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.082 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.083 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.083 DEBUG:boto:Params: {} 2023-11-02T18:46:00.084 DEBUG:boto:Token: None 2023-11-02T18:46:00.084 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.085 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057e4b54c70e3ee5d-006543ee68-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.091 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.092 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.092 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.093 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:46:00.093 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:46:00.094 DEBUG:boto:Method: GET 2023-11-02T18:46:00.094 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:46:00.095 DEBUG:boto:Data: 2023-11-02T18:46:00.095 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.096 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.096 DEBUG:boto:Params: {} 2023-11-02T18:46:00.097 DEBUG:boto:Token: None 2023-11-02T18:46:00.097 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d04f556bcfa85b61-006543ee68-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.109 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.110 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:46:00.110 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:46:00.111 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.111 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:46:00.112 DEBUG:boto:Data: 2023-11-02T18:46:00.112 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.113 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.113 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.113 DEBUG:boto:Params: {} 2023-11-02T18:46:00.114 DEBUG:boto:Token: None 2023-11-02T18:46:00.114 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj69 2023-11-02T18:46:00.115 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:noKD9Pvi/qMKhKae2OdstLODq9Y= 2023-11-02T18:46:00.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:noKD9Pvi/qMKhKae2OdstLODq9Y=', 'Content-Length': '0'} 2023-11-02T18:46:00.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a256666b236ab3e-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.118 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:46:00.119 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:46:00.119 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.120 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:46:00.120 DEBUG:boto:Data: 2023-11-02T18:46:00.120 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.121 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.121 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.122 DEBUG:boto:Params: {} 2023-11-02T18:46:00.122 DEBUG:boto:Token: None 2023-11-02T18:46:00.123 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj69 2023-11-02T18:46:00.123 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:noKD9Pvi/qMKhKae2OdstLODq9Y= 2023-11-02T18:46:00.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:noKD9Pvi/qMKhKae2OdstLODq9Y=', 'Content-Length': '0'} 2023-11-02T18:46:00.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a957a4e8ccddfe2-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:46:00.127 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:46:00.129 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:46:00.129 DEBUG:boto:Method: GET 2023-11-02T18:46:00.130 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:46:00.130 DEBUG:boto:Data: 2023-11-02T18:46:00.131 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.131 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.132 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.132 DEBUG:boto:Params: {} 2023-11-02T18:46:00.133 DEBUG:boto:Token: None 2023-11-02T18:46:00.133 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj69 2023-11-02T18:46:00.134 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CKqudKzM3lTgAQ9xYbGPGsjwfdY= 2023-11-02T18:46:00.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CKqudKzM3lTgAQ9xYbGPGsjwfdY=', 'Content-Length': '0'} 2023-11-02T18:46:00.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab6372f96e9f156-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.138 DEBUG:boto:path=/ibtljn-59/obj69 2023-11-02T18:46:00.139 DEBUG:boto:auth_path=/ibtljn-59/obj69 2023-11-02T18:46:00.139 DEBUG:boto:Method: GET 2023-11-02T18:46:00.140 DEBUG:boto:Path: /ibtljn-59/obj69 2023-11-02T18:46:00.140 DEBUG:boto:Data: 2023-11-02T18:46:00.140 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.141 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.141 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.142 DEBUG:boto:Params: {} 2023-11-02T18:46:00.142 DEBUG:boto:Token: None 2023-11-02T18:46:00.143 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj69 2023-11-02T18:46:00.143 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CKqudKzM3lTgAQ9xYbGPGsjwfdY= 2023-11-02T18:46:00.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CKqudKzM3lTgAQ9xYbGPGsjwfdY=', 'Content-Length': '0'} 2023-11-02T18:46:00.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acd7052cfdd0447c-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.147 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.148 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.148 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:46:00.149 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:46:00.149 DEBUG:boto:Method: GET 2023-11-02T18:46:00.150 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:46:00.150 DEBUG:boto:Data: 2023-11-02T18:46:00.151 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.151 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.151 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.152 DEBUG:boto:Params: {} 2023-11-02T18:46:00.152 DEBUG:boto:Token: None 2023-11-02T18:46:00.153 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.153 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040d667df75226c8f-006543ee68-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.159 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.160 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.161 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.161 DEBUG:boto:path=/ibtljn-59/?prefix=obj69 2023-11-02T18:46:00.162 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj69 2023-11-02T18:46:00.162 DEBUG:boto:Method: GET 2023-11-02T18:46:00.162 DEBUG:boto:Path: /ibtljn-59/?prefix=obj69 2023-11-02T18:46:00.163 DEBUG:boto:Data: 2023-11-02T18:46:00.163 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.164 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.164 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.165 DEBUG:boto:Params: {} 2023-11-02T18:46:00.165 DEBUG:boto:Token: None 2023-11-02T18:46:00.166 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.166 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b1a15502e74c568-006543ee68-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.218 DEBUG:boto:b'ibtljn-59obj691000falseobj692023-11-02T18:39:10.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-11-02T18:46:00.220 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:46:00.220 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:46:00.221 DEBUG:boto:path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.221 DEBUG:boto:auth_path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.222 DEBUG:boto:Method: GET 2023-11-02T18:46:00.222 DEBUG:boto:Path: /ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.223 DEBUG:boto:Data: 2023-11-02T18:46:00.223 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.224 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.224 DEBUG:boto:Params: {} 2023-11-02T18:46:00.225 DEBUG:boto:Token: None 2023-11-02T18:46:00.225 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FXea45zCncbuEXgO1VpG8wI67Tw= 2023-11-02T18:46:00.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FXea45zCncbuEXgO1VpG8wI67Tw=', 'Content-Length': '0'} 2023-11-02T18:46:00.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000203928000ffe3459-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.231 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:46:00.231 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:46:00.232 DEBUG:boto:path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.232 DEBUG:boto:auth_path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.232 DEBUG:boto:Method: GET 2023-11-02T18:46:00.233 DEBUG:boto:Path: /ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.233 DEBUG:boto:Data: 2023-11-02T18:46:00.234 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.234 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.235 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.235 DEBUG:boto:Params: {} 2023-11-02T18:46:00.236 DEBUG:boto:Token: None 2023-11-02T18:46:00.236 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.237 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FXea45zCncbuEXgO1VpG8wI67Tw= 2023-11-02T18:46:00.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FXea45zCncbuEXgO1VpG8wI67Tw=', 'Content-Length': '0'} 2023-11-02T18:46:00.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e8629bbfa24abe5-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.241 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:46:00.241 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:46:00.242 DEBUG:boto:path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.242 DEBUG:boto:auth_path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.243 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.243 DEBUG:boto:Path: /ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.243 DEBUG:boto:Data: 2023-11-02T18:46:00.244 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.244 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.245 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.245 DEBUG:boto:Params: {} 2023-11-02T18:46:00.246 DEBUG:boto:Token: None 2023-11-02T18:46:00.246 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.247 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTpg2cOYimnr2ssENAlx5m05Rms= 2023-11-02T18:46:00.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTpg2cOYimnr2ssENAlx5m05Rms=', 'Content-Length': '0'} 2023-11-02T18:46:00.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000624292a3f849fc70-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.251 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:46:00.251 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:46:00.252 DEBUG:boto:path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.252 DEBUG:boto:auth_path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.252 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.253 DEBUG:boto:Path: /ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.253 DEBUG:boto:Data: 2023-11-02T18:46:00.254 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.254 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.255 DEBUG:boto:Params: {} 2023-11-02T18:46:00.255 DEBUG:boto:Token: None 2023-11-02T18:46:00.256 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.256 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TTpg2cOYimnr2ssENAlx5m05Rms= 2023-11-02T18:46:00.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TTpg2cOYimnr2ssENAlx5m05Rms=', 'Content-Length': '0'} 2023-11-02T18:46:00.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078004eccd86b7197-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-11-02T18:46:00.261 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:46:00.261 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:46:00.262 DEBUG:boto:path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.262 DEBUG:boto:auth_path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.262 DEBUG:boto:Method: GET 2023-11-02T18:46:00.263 DEBUG:boto:Path: /ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.263 DEBUG:boto:Data: 2023-11-02T18:46:00.264 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.264 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.265 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.265 DEBUG:boto:Params: {} 2023-11-02T18:46:00.266 DEBUG:boto:Token: None 2023-11-02T18:46:00.266 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.267 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FXea45zCncbuEXgO1VpG8wI67Tw= 2023-11-02T18:46:00.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FXea45zCncbuEXgO1VpG8wI67Tw=', 'Content-Length': '0'} 2023-11-02T18:46:00.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c95ec90db79eaa-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.271 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:46:00.272 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:46:00.272 DEBUG:boto:path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.273 DEBUG:boto:auth_path=/ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.273 DEBUG:boto:Method: GET 2023-11-02T18:46:00.274 DEBUG:boto:Path: /ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.274 DEBUG:boto:Data: 2023-11-02T18:46:00.275 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.275 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.276 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.276 DEBUG:boto:Params: {} 2023-11-02T18:46:00.277 DEBUG:boto:Token: None 2023-11-02T18:46:00.277 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj74?versionId=null 2023-11-02T18:46:00.278 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FXea45zCncbuEXgO1VpG8wI67Tw= 2023-11-02T18:46:00.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FXea45zCncbuEXgO1VpG8wI67Tw=', 'Content-Length': '0'} 2023-11-02T18:46:00.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c186ab83dab53581-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.322 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.323 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.323 DEBUG:boto:path=/ibtljn-59/?prefix=obj74 2023-11-02T18:46:00.324 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj74 2023-11-02T18:46:00.324 DEBUG:boto:Method: GET 2023-11-02T18:46:00.325 DEBUG:boto:Path: /ibtljn-59/?prefix=obj74 2023-11-02T18:46:00.325 DEBUG:boto:Data: 2023-11-02T18:46:00.326 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.326 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.327 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.327 DEBUG:boto:Params: {} 2023-11-02T18:46:00.328 DEBUG:boto:Token: None 2023-11-02T18:46:00.328 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.329 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079340055a535df6d-006543ee68-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.337 DEBUG:boto:b'ibtljn-59obj741000falseobj742023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.338 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.339 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.339 DEBUG:boto:path=/ibtljn-59/?prefix=obj74 2023-11-02T18:46:00.340 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj74 2023-11-02T18:46:00.340 DEBUG:boto:Method: GET 2023-11-02T18:46:00.340 DEBUG:boto:Path: /ibtljn-59/?prefix=obj74 2023-11-02T18:46:00.341 DEBUG:boto:Data: 2023-11-02T18:46:00.341 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.342 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.342 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.343 DEBUG:boto:Params: {} 2023-11-02T18:46:00.343 DEBUG:boto:Token: None 2023-11-02T18:46:00.343 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.344 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6929b44cbd7f968-006543ee68-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.352 DEBUG:boto:b'ibtljn-59obj741000falseobj742023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.353 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:46:00.353 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:46:00.354 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.354 DEBUG:boto:Path: /ibtljn-59/obj74 2023-11-02T18:46:00.355 DEBUG:boto:Data: 2023-11-02T18:46:00.355 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.356 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.356 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.356 DEBUG:boto:Params: {} 2023-11-02T18:46:00.357 DEBUG:boto:Token: None 2023-11-02T18:46:00.357 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj74 2023-11-02T18:46:00.358 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MfQb/DpD9hPCpEWJH/y/msqrG9M= 2023-11-02T18:46:00.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MfQb/DpD9hPCpEWJH/y/msqrG9M=', 'Content-Length': '0'} 2023-11-02T18:46:00.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5bf58a533afbc14-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.363 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:46:00.364 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:46:00.364 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.364 DEBUG:boto:Path: /ibtljn-59/obj74 2023-11-02T18:46:00.365 DEBUG:boto:Data: 2023-11-02T18:46:00.365 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.366 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.367 DEBUG:boto:Params: {} 2023-11-02T18:46:00.367 DEBUG:boto:Token: None 2023-11-02T18:46:00.368 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj74 2023-11-02T18:46:00.368 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MfQb/DpD9hPCpEWJH/y/msqrG9M= 2023-11-02T18:46:00.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MfQb/DpD9hPCpEWJH/y/msqrG9M=', 'Content-Length': '0'} 2023-11-02T18:46:00.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f8ba531f3e6a80c-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-11-02T18:46:00.372 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:46:00.373 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:46:00.373 DEBUG:boto:Method: GET 2023-11-02T18:46:00.374 DEBUG:boto:Path: /ibtljn-59/obj74 2023-11-02T18:46:00.374 DEBUG:boto:Data: 2023-11-02T18:46:00.374 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.375 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.375 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.376 DEBUG:boto:Params: {} 2023-11-02T18:46:00.376 DEBUG:boto:Token: None 2023-11-02T18:46:00.377 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj74 2023-11-02T18:46:00.377 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuUC8BR0b/Q3YMpyT2PnvCARrOg= 2023-11-02T18:46:00.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuUC8BR0b/Q3YMpyT2PnvCARrOg=', 'Content-Length': '0'} 2023-11-02T18:46:00.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000476a028d63f47540-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.422 DEBUG:boto:path=/ibtljn-59/obj74 2023-11-02T18:46:00.423 DEBUG:boto:auth_path=/ibtljn-59/obj74 2023-11-02T18:46:00.423 DEBUG:boto:Method: GET 2023-11-02T18:46:00.424 DEBUG:boto:Path: /ibtljn-59/obj74 2023-11-02T18:46:00.424 DEBUG:boto:Data: 2023-11-02T18:46:00.425 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.425 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.426 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.426 DEBUG:boto:Params: {} 2023-11-02T18:46:00.427 DEBUG:boto:Token: None 2023-11-02T18:46:00.427 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj74 2023-11-02T18:46:00.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KuUC8BR0b/Q3YMpyT2PnvCARrOg= 2023-11-02T18:46:00.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KuUC8BR0b/Q3YMpyT2PnvCARrOg=', 'Content-Length': '0'} 2023-11-02T18:46:00.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000471f8e99b519146b-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.432 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.432 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.433 DEBUG:boto:path=/ibtljn-59/?prefix=obj74 2023-11-02T18:46:00.433 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj74 2023-11-02T18:46:00.434 DEBUG:boto:Method: GET 2023-11-02T18:46:00.434 DEBUG:boto:Path: /ibtljn-59/?prefix=obj74 2023-11-02T18:46:00.434 DEBUG:boto:Data: 2023-11-02T18:46:00.435 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.436 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.436 DEBUG:boto:Params: {} 2023-11-02T18:46:00.437 DEBUG:boto:Token: None 2023-11-02T18:46:00.437 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.438 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f355ec74ceda048e-006543ee68-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.444 DEBUG:boto:b'ibtljn-59obj741000falseobj742023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.444 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.445 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.445 DEBUG:boto:path=/ibtljn-59/?prefix=obj74 2023-11-02T18:46:00.446 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj74 2023-11-02T18:46:00.446 DEBUG:boto:Method: GET 2023-11-02T18:46:00.447 DEBUG:boto:Path: /ibtljn-59/?prefix=obj74 2023-11-02T18:46:00.447 DEBUG:boto:Data: 2023-11-02T18:46:00.448 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.448 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.448 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.449 DEBUG:boto:Params: {} 2023-11-02T18:46:00.449 DEBUG:boto:Token: None 2023-11-02T18:46:00.450 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.450 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000387f8464159c2e27-006543ee68-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.502 DEBUG:boto:b'ibtljn-59obj741000falseobj742023-11-02T18:44:13.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:46:00.504 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:46:00.504 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:46:00.505 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.505 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.505 DEBUG:boto:Method: GET 2023-11-02T18:46:00.506 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.506 DEBUG:boto:Data: 2023-11-02T18:46:00.507 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.507 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.508 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.508 DEBUG:boto:Params: {} 2023-11-02T18:46:00.508 DEBUG:boto:Token: None 2023-11-02T18:46:00.509 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.509 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+nryOI8ihzh6Xm1BeuVHlApi0LE= 2023-11-02T18:46:00.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+nryOI8ihzh6Xm1BeuVHlApi0LE=', 'Content-Length': '0'} 2023-11-02T18:46:00.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000728e04f14996c00e-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.514 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:46:00.515 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:46:00.515 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.516 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.516 DEBUG:boto:Method: GET 2023-11-02T18:46:00.516 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.517 DEBUG:boto:Data: 2023-11-02T18:46:00.517 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.518 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.518 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.519 DEBUG:boto:Params: {} 2023-11-02T18:46:00.519 DEBUG:boto:Token: None 2023-11-02T18:46:00.520 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.520 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+nryOI8ihzh6Xm1BeuVHlApi0LE= 2023-11-02T18:46:00.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+nryOI8ihzh6Xm1BeuVHlApi0LE=', 'Content-Length': '0'} 2023-11-02T18:46:00.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e82df68ca276ea67-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.524 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:46:00.525 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:46:00.525 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.526 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.526 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.527 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.527 DEBUG:boto:Data: 2023-11-02T18:46:00.528 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.528 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.528 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.529 DEBUG:boto:Params: {} 2023-11-02T18:46:00.529 DEBUG:boto:Token: None 2023-11-02T18:46:00.530 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.530 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3GcWAdicA/YYc39fxlgn9IgateI= 2023-11-02T18:46:00.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3GcWAdicA/YYc39fxlgn9IgateI=', 'Content-Length': '0'} 2023-11-02T18:46:00.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003539f14c093be7c-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.534 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:46:00.535 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:46:00.535 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.536 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.536 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.537 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.537 DEBUG:boto:Data: 2023-11-02T18:46:00.537 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.538 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.538 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.539 DEBUG:boto:Params: {} 2023-11-02T18:46:00.539 DEBUG:boto:Token: None 2023-11-02T18:46:00.540 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.540 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3GcWAdicA/YYc39fxlgn9IgateI= 2023-11-02T18:46:00.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3GcWAdicA/YYc39fxlgn9IgateI=', 'Content-Length': '0'} 2023-11-02T18:46:00.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d104594d24f79e24-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:46:00.545 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:46:00.545 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:46:00.546 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.546 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.547 DEBUG:boto:Method: GET 2023-11-02T18:46:00.547 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.547 DEBUG:boto:Data: 2023-11-02T18:46:00.548 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.548 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.549 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.549 DEBUG:boto:Params: {} 2023-11-02T18:46:00.550 DEBUG:boto:Token: None 2023-11-02T18:46:00.550 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.551 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+nryOI8ihzh6Xm1BeuVHlApi0LE= 2023-11-02T18:46:00.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+nryOI8ihzh6Xm1BeuVHlApi0LE=', 'Content-Length': '0'} 2023-11-02T18:46:00.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4ba98e93a327691-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.555 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:46:00.555 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:46:00.556 DEBUG:boto:path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.556 DEBUG:boto:auth_path=/ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.557 DEBUG:boto:Method: GET 2023-11-02T18:46:00.557 DEBUG:boto:Path: /ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.558 DEBUG:boto:Data: 2023-11-02T18:46:00.558 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.559 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.559 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.560 DEBUG:boto:Params: {} 2023-11-02T18:46:00.560 DEBUG:boto:Token: None 2023-11-02T18:46:00.560 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj76?versionId=null 2023-11-02T18:46:00.561 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+nryOI8ihzh6Xm1BeuVHlApi0LE= 2023-11-02T18:46:00.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+nryOI8ihzh6Xm1BeuVHlApi0LE=', 'Content-Length': '0'} 2023-11-02T18:46:00.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b03ca848fba8c210-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.610 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.611 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.611 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:46:00.612 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:46:00.612 DEBUG:boto:Method: GET 2023-11-02T18:46:00.612 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:46:00.613 DEBUG:boto:Data: 2023-11-02T18:46:00.613 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.614 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.614 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.615 DEBUG:boto:Params: {} 2023-11-02T18:46:00.615 DEBUG:boto:Token: None 2023-11-02T18:46:00.615 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.616 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abe61cbaf13abc91-006543ee68-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.622 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.623 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.623 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.624 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:46:00.624 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:46:00.625 DEBUG:boto:Method: GET 2023-11-02T18:46:00.625 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:46:00.626 DEBUG:boto:Data: 2023-11-02T18:46:00.626 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.627 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.627 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.628 DEBUG:boto:Params: {} 2023-11-02T18:46:00.628 DEBUG:boto:Token: None 2023-11-02T18:46:00.629 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.629 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072e2b47537e2088d-006543ee68-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.639 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.640 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:46:00.641 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:46:00.641 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.641 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:46:00.642 DEBUG:boto:Data: 2023-11-02T18:46:00.642 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.643 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.643 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.644 DEBUG:boto:Params: {} 2023-11-02T18:46:00.644 DEBUG:boto:Token: None 2023-11-02T18:46:00.645 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj76 2023-11-02T18:46:00.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eMwYrqQoQYpZFD60SdaLpHT0sVg= 2023-11-02T18:46:00.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eMwYrqQoQYpZFD60SdaLpHT0sVg=', 'Content-Length': '0'} 2023-11-02T18:46:00.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9cde63c92584ec3-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.649 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:46:00.649 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:46:00.650 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.650 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:46:00.650 DEBUG:boto:Data: 2023-11-02T18:46:00.651 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.651 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.652 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.652 DEBUG:boto:Params: {} 2023-11-02T18:46:00.653 DEBUG:boto:Token: None 2023-11-02T18:46:00.653 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj76 2023-11-02T18:46:00.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eMwYrqQoQYpZFD60SdaLpHT0sVg= 2023-11-02T18:46:00.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eMwYrqQoQYpZFD60SdaLpHT0sVg=', 'Content-Length': '0'} 2023-11-02T18:46:00.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000150c1fef49d920f8-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:46:00.658 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:46:00.658 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:46:00.659 DEBUG:boto:Method: GET 2023-11-02T18:46:00.659 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:46:00.660 DEBUG:boto:Data: 2023-11-02T18:46:00.660 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.660 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.661 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.661 DEBUG:boto:Params: {} 2023-11-02T18:46:00.662 DEBUG:boto:Token: None 2023-11-02T18:46:00.662 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj76 2023-11-02T18:46:00.663 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NBgzD8C/MyTf4ELJrRYzeLVtHEc= 2023-11-02T18:46:00.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NBgzD8C/MyTf4ELJrRYzeLVtHEc=', 'Content-Length': '0'} 2023-11-02T18:46:00.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097c2c47688d7d4f6-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.667 DEBUG:boto:path=/ibtljn-59/obj76 2023-11-02T18:46:00.668 DEBUG:boto:auth_path=/ibtljn-59/obj76 2023-11-02T18:46:00.668 DEBUG:boto:Method: GET 2023-11-02T18:46:00.668 DEBUG:boto:Path: /ibtljn-59/obj76 2023-11-02T18:46:00.669 DEBUG:boto:Data: 2023-11-02T18:46:00.669 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.670 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.670 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.671 DEBUG:boto:Params: {} 2023-11-02T18:46:00.671 DEBUG:boto:Token: None 2023-11-02T18:46:00.672 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj76 2023-11-02T18:46:00.672 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NBgzD8C/MyTf4ELJrRYzeLVtHEc= 2023-11-02T18:46:00.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NBgzD8C/MyTf4ELJrRYzeLVtHEc=', 'Content-Length': '0'} 2023-11-02T18:46:00.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b6bfbf2af25919d-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.677 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.677 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.678 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:46:00.678 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:46:00.679 DEBUG:boto:Method: GET 2023-11-02T18:46:00.679 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:46:00.680 DEBUG:boto:Data: 2023-11-02T18:46:00.680 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.681 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.682 DEBUG:boto:Params: {} 2023-11-02T18:46:00.682 DEBUG:boto:Token: None 2023-11-02T18:46:00.683 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.683 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceb1bd37b3377d0b-006543ee68-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.730 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.731 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.732 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.733 DEBUG:boto:path=/ibtljn-59/?prefix=obj76 2023-11-02T18:46:00.733 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj76 2023-11-02T18:46:00.734 DEBUG:boto:Method: GET 2023-11-02T18:46:00.734 DEBUG:boto:Path: /ibtljn-59/?prefix=obj76 2023-11-02T18:46:00.735 DEBUG:boto:Data: 2023-11-02T18:46:00.736 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.736 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.736 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.737 DEBUG:boto:Params: {} 2023-11-02T18:46:00.737 DEBUG:boto:Token: None 2023-11-02T18:46:00.738 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.739 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a00596e09dd52fb9-006543ee68-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.750 DEBUG:boto:b'ibtljn-59obj761000falseobj762023-11-02T18:34:50.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:46:00.751 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:46:00.751 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:46:00.752 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.752 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.753 DEBUG:boto:Method: GET 2023-11-02T18:46:00.753 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.754 DEBUG:boto:Data: 2023-11-02T18:46:00.754 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.755 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.755 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.756 DEBUG:boto:Params: {} 2023-11-02T18:46:00.756 DEBUG:boto:Token: None 2023-11-02T18:46:00.757 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.758 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:80ANXpfkRcONaTlLRhOh+HhtpUg= 2023-11-02T18:46:00.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:80ANXpfkRcONaTlLRhOh+HhtpUg=', 'Content-Length': '0'} 2023-11-02T18:46:00.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8774e13bf744cb8-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.762 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:46:00.763 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:46:00.763 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.764 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.764 DEBUG:boto:Method: GET 2023-11-02T18:46:00.765 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.765 DEBUG:boto:Data: 2023-11-02T18:46:00.765 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.766 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.766 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.767 DEBUG:boto:Params: {} 2023-11-02T18:46:00.767 DEBUG:boto:Token: None 2023-11-02T18:46:00.768 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:80ANXpfkRcONaTlLRhOh+HhtpUg= 2023-11-02T18:46:00.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:80ANXpfkRcONaTlLRhOh+HhtpUg=', 'Content-Length': '0'} 2023-11-02T18:46:00.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c1964e59dfb4825-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.814 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:46:00.815 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:46:00.815 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.816 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.816 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.817 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.817 DEBUG:boto:Data: 2023-11-02T18:46:00.818 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.818 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.818 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.819 DEBUG:boto:Params: {} 2023-11-02T18:46:00.819 DEBUG:boto:Token: None 2023-11-02T18:46:00.820 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.820 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dOgCzGYgwyTzulFhtkbqqPiQdzA= 2023-11-02T18:46:00.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dOgCzGYgwyTzulFhtkbqqPiQdzA=', 'Content-Length': '0'} 2023-11-02T18:46:00.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021e03d618a3b48ba-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.825 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:46:00.825 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:46:00.825 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.826 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.826 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.827 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.827 DEBUG:boto:Data: 2023-11-02T18:46:00.828 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.828 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.828 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.829 DEBUG:boto:Params: {} 2023-11-02T18:46:00.829 DEBUG:boto:Token: None 2023-11-02T18:46:00.830 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.830 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dOgCzGYgwyTzulFhtkbqqPiQdzA= 2023-11-02T18:46:00.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dOgCzGYgwyTzulFhtkbqqPiQdzA=', 'Content-Length': '0'} 2023-11-02T18:46:00.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f66858f89fcb414-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:46:00.835 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:46:00.835 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:46:00.836 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.836 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.837 DEBUG:boto:Method: GET 2023-11-02T18:46:00.837 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.837 DEBUG:boto:Data: 2023-11-02T18:46:00.838 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.838 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.839 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.839 DEBUG:boto:Params: {} 2023-11-02T18:46:00.840 DEBUG:boto:Token: None 2023-11-02T18:46:00.840 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.841 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:80ANXpfkRcONaTlLRhOh+HhtpUg= 2023-11-02T18:46:00.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:80ANXpfkRcONaTlLRhOh+HhtpUg=', 'Content-Length': '0'} 2023-11-02T18:46:00.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1d882d9b38ee993-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.845 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:46:00.846 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:46:00.846 DEBUG:boto:path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.847 DEBUG:boto:auth_path=/ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.847 DEBUG:boto:Method: GET 2023-11-02T18:46:00.847 DEBUG:boto:Path: /ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.848 DEBUG:boto:Data: 2023-11-02T18:46:00.848 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.849 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.849 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.849 DEBUG:boto:Params: {} 2023-11-02T18:46:00.850 DEBUG:boto:Token: None 2023-11-02T18:46:00.850 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj85?versionId=null 2023-11-02T18:46:00.851 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:80ANXpfkRcONaTlLRhOh+HhtpUg= 2023-11-02T18:46:00.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:80ANXpfkRcONaTlLRhOh+HhtpUg=', 'Content-Length': '0'} 2023-11-02T18:46:00.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000408cbf982ebb72be-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.855 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.855 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.856 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:46:00.856 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:46:00.857 DEBUG:boto:Method: GET 2023-11-02T18:46:00.857 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:46:00.858 DEBUG:boto:Data: 2023-11-02T18:46:00.858 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.859 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.859 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.859 DEBUG:boto:Params: {} 2023-11-02T18:46:00.860 DEBUG:boto:Token: None 2023-11-02T18:46:00.860 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.861 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0fb4aa01fd62490-006543ee68-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.910 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.911 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.912 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.912 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:46:00.913 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:46:00.913 DEBUG:boto:Method: GET 2023-11-02T18:46:00.914 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:46:00.914 DEBUG:boto:Data: 2023-11-02T18:46:00.915 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.915 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.916 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.916 DEBUG:boto:Params: {} 2023-11-02T18:46:00.917 DEBUG:boto:Token: None 2023-11-02T18:46:00.917 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.918 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030ed06da6f068024-006543ee68-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.927 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.928 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:46:00.929 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:46:00.929 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.929 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:46:00.930 DEBUG:boto:Data: 2023-11-02T18:46:00.930 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.931 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.931 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.932 DEBUG:boto:Params: {} 2023-11-02T18:46:00.932 DEBUG:boto:Token: None 2023-11-02T18:46:00.933 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj85 2023-11-02T18:46:00.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4dAItFPms2qoNDdEm42HvNKL6lY= 2023-11-02T18:46:00.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4dAItFPms2qoNDdEm42HvNKL6lY=', 'Content-Length': '0'} 2023-11-02T18:46:00.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000120a8e3a9dd56090-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.936 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:46:00.937 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:46:00.937 DEBUG:boto:Method: HEAD 2023-11-02T18:46:00.938 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:46:00.938 DEBUG:boto:Data: 2023-11-02T18:46:00.939 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.939 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.940 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.940 DEBUG:boto:Params: {} 2023-11-02T18:46:00.940 DEBUG:boto:Token: None 2023-11-02T18:46:00.941 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj85 2023-11-02T18:46:00.941 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4dAItFPms2qoNDdEm42HvNKL6lY= 2023-11-02T18:46:00.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4dAItFPms2qoNDdEm42HvNKL6lY=', 'Content-Length': '0'} 2023-11-02T18:46:00.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089bc9f465609350e-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:46:00.945 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:46:00.946 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:46:00.946 DEBUG:boto:Method: GET 2023-11-02T18:46:00.947 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:46:00.947 DEBUG:boto:Data: 2023-11-02T18:46:00.947 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.948 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.948 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.949 DEBUG:boto:Params: {} 2023-11-02T18:46:00.949 DEBUG:boto:Token: None 2023-11-02T18:46:00.950 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj85 2023-11-02T18:46:00.950 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5mk1vH9kc66c4EKKdzkW7jU9AkQ= 2023-11-02T18:46:00.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5mk1vH9kc66c4EKKdzkW7jU9AkQ=', 'Content-Length': '0'} 2023-11-02T18:46:00.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed26add25824bbbd-006543ee68-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.954 DEBUG:boto:path=/ibtljn-59/obj85 2023-11-02T18:46:00.955 DEBUG:boto:auth_path=/ibtljn-59/obj85 2023-11-02T18:46:00.955 DEBUG:boto:Method: GET 2023-11-02T18:46:00.956 DEBUG:boto:Path: /ibtljn-59/obj85 2023-11-02T18:46:00.956 DEBUG:boto:Data: 2023-11-02T18:46:00.956 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.957 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.957 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.959 DEBUG:boto:Params: {} 2023-11-02T18:46:00.959 DEBUG:boto:Token: None 2023-11-02T18:46:00.960 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/obj85 2023-11-02T18:46:00.960 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5mk1vH9kc66c4EKKdzkW7jU9AkQ= 2023-11-02T18:46:00.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5mk1vH9kc66c4EKKdzkW7jU9AkQ=', 'Content-Length': '0'} 2023-11-02T18:46:00.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:32:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc83e984fea35e00-006543ee68-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.964 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.965 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.965 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:46:00.966 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:46:00.966 DEBUG:boto:Method: GET 2023-11-02T18:46:00.967 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:46:00.967 DEBUG:boto:Data: 2023-11-02T18:46:00.967 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.968 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:00.968 DEBUG:boto:Port: 8001 2023-11-02T18:46:00.969 DEBUG:boto:Params: {} 2023-11-02T18:46:00.969 DEBUG:boto:Token: None 2023-11-02T18:46:00.970 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.970 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000524d889af75228d2-006543ee68-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:00.975 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:00.976 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:00.977 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:00.977 DEBUG:boto:path=/ibtljn-59/?prefix=obj85 2023-11-02T18:46:00.977 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj85 2023-11-02T18:46:00.978 DEBUG:boto:Method: GET 2023-11-02T18:46:00.978 DEBUG:boto:Path: /ibtljn-59/?prefix=obj85 2023-11-02T18:46:00.979 DEBUG:boto:Data: 2023-11-02T18:46:00.979 DEBUG:boto:Headers: {} 2023-11-02T18:46:00.980 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:00.980 DEBUG:boto:Port: 8000 2023-11-02T18:46:00.981 DEBUG:boto:Params: {} 2023-11-02T18:46:00.981 DEBUG:boto:Token: None 2023-11-02T18:46:00.982 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:00 GMT /ibtljn-59/ 2023-11-02T18:46:00.982 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w= 2023-11-02T18:46:00.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRKtDy7D1DjRdcaYaxZV311Se/w=', 'Content-Length': '0'} 2023-11-02T18:46:00.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061724801a5e34121-006543ee68-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.034 DEBUG:boto:b'ibtljn-59obj851000falseobj852023-11-02T18:32:46.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:46:01.036 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:46:01.036 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:46:01.037 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.037 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.037 DEBUG:boto:Method: GET 2023-11-02T18:46:01.038 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.038 DEBUG:boto:Data: 2023-11-02T18:46:01.039 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.039 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.040 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.040 DEBUG:boto:Params: {} 2023-11-02T18:46:01.041 DEBUG:boto:Token: None 2023-11-02T18:46:01.041 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.042 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Lp0PUs35l2egKdwo+EOS+hRcmq0= 2023-11-02T18:46:01.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Lp0PUs35l2egKdwo+EOS+hRcmq0=', 'Content-Length': '0'} 2023-11-02T18:46:01.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d4ed832b16ca565-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.046 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:46:01.047 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:46:01.047 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.048 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.048 DEBUG:boto:Method: GET 2023-11-02T18:46:01.049 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.049 DEBUG:boto:Data: 2023-11-02T18:46:01.049 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.050 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.051 DEBUG:boto:Params: {} 2023-11-02T18:46:01.051 DEBUG:boto:Token: None 2023-11-02T18:46:01.052 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.052 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Lp0PUs35l2egKdwo+EOS+hRcmq0= 2023-11-02T18:46:01.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Lp0PUs35l2egKdwo+EOS+hRcmq0=', 'Content-Length': '0'} 2023-11-02T18:46:01.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049d3c4ff3506c108-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.056 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:46:01.057 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:46:01.057 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.058 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.058 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.058 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.059 DEBUG:boto:Data: 2023-11-02T18:46:01.059 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.060 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.060 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.061 DEBUG:boto:Params: {} 2023-11-02T18:46:01.061 DEBUG:boto:Token: None 2023-11-02T18:46:01.062 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.062 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5M9eM6bQYy2APMN1s6O3rE7BlMI= 2023-11-02T18:46:01.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5M9eM6bQYy2APMN1s6O3rE7BlMI=', 'Content-Length': '0'} 2023-11-02T18:46:01.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2d6235b4601d62c-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.066 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:46:01.067 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:46:01.067 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.067 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.068 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.068 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.069 DEBUG:boto:Data: 2023-11-02T18:46:01.069 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.070 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.070 DEBUG:boto:Params: {} 2023-11-02T18:46:01.071 DEBUG:boto:Token: None 2023-11-02T18:46:01.071 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5M9eM6bQYy2APMN1s6O3rE7BlMI= 2023-11-02T18:46:01.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5M9eM6bQYy2APMN1s6O3rE7BlMI=', 'Content-Length': '0'} 2023-11-02T18:46:01.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073cb8ecc22eb69fc-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:46:01.076 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:46:01.077 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:46:01.077 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.078 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.078 DEBUG:boto:Method: GET 2023-11-02T18:46:01.079 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.079 DEBUG:boto:Data: 2023-11-02T18:46:01.080 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.080 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.080 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.081 DEBUG:boto:Params: {} 2023-11-02T18:46:01.081 DEBUG:boto:Token: None 2023-11-02T18:46:01.082 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.082 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Lp0PUs35l2egKdwo+EOS+hRcmq0= 2023-11-02T18:46:01.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Lp0PUs35l2egKdwo+EOS+hRcmq0=', 'Content-Length': '0'} 2023-11-02T18:46:01.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014cfb03d4b38c146-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.087 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:46:01.087 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:46:01.088 DEBUG:boto:path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.088 DEBUG:boto:auth_path=/ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.088 DEBUG:boto:Method: GET 2023-11-02T18:46:01.089 DEBUG:boto:Path: /ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.089 DEBUG:boto:Data: 2023-11-02T18:46:01.090 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.090 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.091 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.091 DEBUG:boto:Params: {} 2023-11-02T18:46:01.092 DEBUG:boto:Token: None 2023-11-02T18:46:01.092 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj87?versionId=null 2023-11-02T18:46:01.093 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Lp0PUs35l2egKdwo+EOS+hRcmq0= 2023-11-02T18:46:01.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Lp0PUs35l2egKdwo+EOS+hRcmq0=', 'Content-Length': '0'} 2023-11-02T18:46:01.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4ebea88653819aa-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.142 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.143 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.143 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:46:01.144 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:46:01.144 DEBUG:boto:Method: GET 2023-11-02T18:46:01.145 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:46:01.145 DEBUG:boto:Data: 2023-11-02T18:46:01.146 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.146 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.147 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.147 DEBUG:boto:Params: {} 2023-11-02T18:46:01.148 DEBUG:boto:Token: None 2023-11-02T18:46:01.148 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.149 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea727358561f220d-006543ee69-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.155 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.157 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.158 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.158 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:46:01.159 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:46:01.159 DEBUG:boto:Method: GET 2023-11-02T18:46:01.159 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:46:01.160 DEBUG:boto:Data: 2023-11-02T18:46:01.160 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.161 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.161 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.162 DEBUG:boto:Params: {} 2023-11-02T18:46:01.162 DEBUG:boto:Token: None 2023-11-02T18:46:01.163 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.163 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dde0563def06d40-006543ee69-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.174 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.175 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:46:01.176 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:46:01.176 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.177 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:46:01.177 DEBUG:boto:Data: 2023-11-02T18:46:01.177 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.178 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.178 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.179 DEBUG:boto:Params: {} 2023-11-02T18:46:01.179 DEBUG:boto:Token: None 2023-11-02T18:46:01.180 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj87 2023-11-02T18:46:01.180 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rg2aXmZCsOknA21IK5gKyPt8oZU= 2023-11-02T18:46:01.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rg2aXmZCsOknA21IK5gKyPt8oZU=', 'Content-Length': '0'} 2023-11-02T18:46:01.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe82f6669618648-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.183 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:46:01.184 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:46:01.185 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.185 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:46:01.186 DEBUG:boto:Data: 2023-11-02T18:46:01.187 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.187 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.187 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.188 DEBUG:boto:Params: {} 2023-11-02T18:46:01.188 DEBUG:boto:Token: None 2023-11-02T18:46:01.189 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj87 2023-11-02T18:46:01.190 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rg2aXmZCsOknA21IK5gKyPt8oZU= 2023-11-02T18:46:01.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rg2aXmZCsOknA21IK5gKyPt8oZU=', 'Content-Length': '0'} 2023-11-02T18:46:01.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa62760fc2e42397-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:46:01.194 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:46:01.195 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:46:01.195 DEBUG:boto:Method: GET 2023-11-02T18:46:01.196 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:46:01.196 DEBUG:boto:Data: 2023-11-02T18:46:01.196 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.197 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.197 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.198 DEBUG:boto:Params: {} 2023-11-02T18:46:01.198 DEBUG:boto:Token: None 2023-11-02T18:46:01.199 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj87 2023-11-02T18:46:01.199 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z6UhmyPf9eM29ZewYpIlea2SUSk= 2023-11-02T18:46:01.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z6UhmyPf9eM29ZewYpIlea2SUSk=', 'Content-Length': '0'} 2023-11-02T18:46:01.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea06d3272409c075-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.246 DEBUG:boto:path=/ibtljn-59/obj87 2023-11-02T18:46:01.247 DEBUG:boto:auth_path=/ibtljn-59/obj87 2023-11-02T18:46:01.247 DEBUG:boto:Method: GET 2023-11-02T18:46:01.248 DEBUG:boto:Path: /ibtljn-59/obj87 2023-11-02T18:46:01.248 DEBUG:boto:Data: 2023-11-02T18:46:01.249 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.249 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.250 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.250 DEBUG:boto:Params: {} 2023-11-02T18:46:01.251 DEBUG:boto:Token: None 2023-11-02T18:46:01.251 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj87 2023-11-02T18:46:01.252 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:z6UhmyPf9eM29ZewYpIlea2SUSk= 2023-11-02T18:46:01.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:z6UhmyPf9eM29ZewYpIlea2SUSk=', 'Content-Length': '0'} 2023-11-02T18:46:01.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:40:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005282c649454648fa-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.256 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.256 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.257 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:46:01.257 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:46:01.258 DEBUG:boto:Method: GET 2023-11-02T18:46:01.258 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:46:01.259 DEBUG:boto:Data: 2023-11-02T18:46:01.259 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.260 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.260 DEBUG:boto:Params: {} 2023-11-02T18:46:01.261 DEBUG:boto:Token: None 2023-11-02T18:46:01.261 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.262 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c8713705dff9586-006543ee69-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.268 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.269 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.269 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.270 DEBUG:boto:path=/ibtljn-59/?prefix=obj87 2023-11-02T18:46:01.270 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj87 2023-11-02T18:46:01.270 DEBUG:boto:Method: GET 2023-11-02T18:46:01.271 DEBUG:boto:Path: /ibtljn-59/?prefix=obj87 2023-11-02T18:46:01.271 DEBUG:boto:Data: 2023-11-02T18:46:01.272 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.272 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.273 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.273 DEBUG:boto:Params: {} 2023-11-02T18:46:01.273 DEBUG:boto:Token: None 2023-11-02T18:46:01.274 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.274 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002618c2412e901429-006543ee69-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.330 DEBUG:boto:b'ibtljn-59obj871000falseobj872023-11-02T18:40:31.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:46:01.332 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:46:01.332 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:46:01.333 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.333 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.333 DEBUG:boto:Method: GET 2023-11-02T18:46:01.334 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.334 DEBUG:boto:Data: 2023-11-02T18:46:01.335 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.335 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.336 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.336 DEBUG:boto:Params: {} 2023-11-02T18:46:01.337 DEBUG:boto:Token: None 2023-11-02T18:46:01.337 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.338 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N32vxi/jBD0idsMtR0K4k1BK7II= 2023-11-02T18:46:01.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N32vxi/jBD0idsMtR0K4k1BK7II=', 'Content-Length': '0'} 2023-11-02T18:46:01.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c45ee89f549fdba-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.343 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:46:01.343 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:46:01.344 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.344 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.345 DEBUG:boto:Method: GET 2023-11-02T18:46:01.345 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.345 DEBUG:boto:Data: 2023-11-02T18:46:01.346 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.346 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.347 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.347 DEBUG:boto:Params: {} 2023-11-02T18:46:01.348 DEBUG:boto:Token: None 2023-11-02T18:46:01.348 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.349 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N32vxi/jBD0idsMtR0K4k1BK7II= 2023-11-02T18:46:01.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N32vxi/jBD0idsMtR0K4k1BK7II=', 'Content-Length': '0'} 2023-11-02T18:46:01.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000955ef2a2c39eb5d-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.353 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:46:01.354 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:46:01.354 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.354 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.355 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.355 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.356 DEBUG:boto:Data: 2023-11-02T18:46:01.356 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.357 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.357 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.357 DEBUG:boto:Params: {} 2023-11-02T18:46:01.358 DEBUG:boto:Token: None 2023-11-02T18:46:01.358 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.359 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j8qCDHSgfsi6J6WhmNOoqAxqElk= 2023-11-02T18:46:01.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j8qCDHSgfsi6J6WhmNOoqAxqElk=', 'Content-Length': '0'} 2023-11-02T18:46:01.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000992536314f7f6770-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.364 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:46:01.365 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:46:01.365 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.365 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.366 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.366 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.367 DEBUG:boto:Data: 2023-11-02T18:46:01.367 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.367 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.368 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.368 DEBUG:boto:Params: {} 2023-11-02T18:46:01.369 DEBUG:boto:Token: None 2023-11-02T18:46:01.369 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.370 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j8qCDHSgfsi6J6WhmNOoqAxqElk= 2023-11-02T18:46:01.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j8qCDHSgfsi6J6WhmNOoqAxqElk=', 'Content-Length': '0'} 2023-11-02T18:46:01.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8b5026f64947cf8-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:46:01.374 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:46:01.375 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:46:01.375 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.375 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.376 DEBUG:boto:Method: GET 2023-11-02T18:46:01.376 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.377 DEBUG:boto:Data: 2023-11-02T18:46:01.377 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.378 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.378 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.379 DEBUG:boto:Params: {} 2023-11-02T18:46:01.379 DEBUG:boto:Token: None 2023-11-02T18:46:01.380 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.380 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N32vxi/jBD0idsMtR0K4k1BK7II= 2023-11-02T18:46:01.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N32vxi/jBD0idsMtR0K4k1BK7II=', 'Content-Length': '0'} 2023-11-02T18:46:01.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000847e261c59f9ab89-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.430 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:46:01.431 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:46:01.431 DEBUG:boto:path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.432 DEBUG:boto:auth_path=/ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.432 DEBUG:boto:Method: GET 2023-11-02T18:46:01.433 DEBUG:boto:Path: /ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.433 DEBUG:boto:Data: 2023-11-02T18:46:01.433 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.434 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.434 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.435 DEBUG:boto:Params: {} 2023-11-02T18:46:01.435 DEBUG:boto:Token: None 2023-11-02T18:46:01.436 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj92?versionId=null 2023-11-02T18:46:01.436 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N32vxi/jBD0idsMtR0K4k1BK7II= 2023-11-02T18:46:01.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N32vxi/jBD0idsMtR0K4k1BK7II=', 'Content-Length': '0'} 2023-11-02T18:46:01.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e0d69c08cffbbdc-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.441 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.441 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.442 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:46:01.442 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:46:01.443 DEBUG:boto:Method: GET 2023-11-02T18:46:01.443 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:46:01.444 DEBUG:boto:Data: 2023-11-02T18:46:01.444 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.444 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.445 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.445 DEBUG:boto:Params: {} 2023-11-02T18:46:01.446 DEBUG:boto:Token: None 2023-11-02T18:46:01.446 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.447 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043bc812e0a992d5f-006543ee69-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.453 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.454 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.454 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.455 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:46:01.455 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:46:01.455 DEBUG:boto:Method: GET 2023-11-02T18:46:01.456 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:46:01.456 DEBUG:boto:Data: 2023-11-02T18:46:01.457 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.457 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.458 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.458 DEBUG:boto:Params: {} 2023-11-02T18:46:01.459 DEBUG:boto:Token: None 2023-11-02T18:46:01.459 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.459 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3b75c37a733c3a9-006543ee69-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.510 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.511 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:46:01.512 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:46:01.512 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.513 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:46:01.513 DEBUG:boto:Data: 2023-11-02T18:46:01.514 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.514 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.514 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.515 DEBUG:boto:Params: {} 2023-11-02T18:46:01.515 DEBUG:boto:Token: None 2023-11-02T18:46:01.516 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj92 2023-11-02T18:46:01.516 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OUD7ycXSXkZsQbWSZSiwyHZur80= 2023-11-02T18:46:01.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OUD7ycXSXkZsQbWSZSiwyHZur80=', 'Content-Length': '0'} 2023-11-02T18:46:01.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000277d63e6945e7fee-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.520 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:46:01.521 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:46:01.521 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.521 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:46:01.522 DEBUG:boto:Data: 2023-11-02T18:46:01.522 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.523 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.523 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.524 DEBUG:boto:Params: {} 2023-11-02T18:46:01.524 DEBUG:boto:Token: None 2023-11-02T18:46:01.525 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj92 2023-11-02T18:46:01.525 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OUD7ycXSXkZsQbWSZSiwyHZur80= 2023-11-02T18:46:01.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OUD7ycXSXkZsQbWSZSiwyHZur80=', 'Content-Length': '0'} 2023-11-02T18:46:01.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083cdc557e5792487-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:46:01.530 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:46:01.530 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:46:01.531 DEBUG:boto:Method: GET 2023-11-02T18:46:01.531 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:46:01.531 DEBUG:boto:Data: 2023-11-02T18:46:01.532 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.533 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.533 DEBUG:boto:Params: {} 2023-11-02T18:46:01.534 DEBUG:boto:Token: None 2023-11-02T18:46:01.534 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj92 2023-11-02T18:46:01.535 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:h6XOW7lTUbhvMpfQSSqOKQ47qvo= 2023-11-02T18:46:01.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h6XOW7lTUbhvMpfQSSqOKQ47qvo=', 'Content-Length': '0'} 2023-11-02T18:46:01.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcddbea450aabc17-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.539 DEBUG:boto:path=/ibtljn-59/obj92 2023-11-02T18:46:01.539 DEBUG:boto:auth_path=/ibtljn-59/obj92 2023-11-02T18:46:01.540 DEBUG:boto:Method: GET 2023-11-02T18:46:01.540 DEBUG:boto:Path: /ibtljn-59/obj92 2023-11-02T18:46:01.541 DEBUG:boto:Data: 2023-11-02T18:46:01.541 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.542 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.542 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.542 DEBUG:boto:Params: {} 2023-11-02T18:46:01.543 DEBUG:boto:Token: None 2023-11-02T18:46:01.543 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj92 2023-11-02T18:46:01.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:h6XOW7lTUbhvMpfQSSqOKQ47qvo= 2023-11-02T18:46:01.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h6XOW7lTUbhvMpfQSSqOKQ47qvo=', 'Content-Length': '0'} 2023-11-02T18:46:01.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:39:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f2cc1ce6a1d12d1-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.548 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.549 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.549 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:46:01.550 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:46:01.550 DEBUG:boto:Method: GET 2023-11-02T18:46:01.550 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:46:01.551 DEBUG:boto:Data: 2023-11-02T18:46:01.551 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.552 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.552 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.552 DEBUG:boto:Params: {} 2023-11-02T18:46:01.553 DEBUG:boto:Token: None 2023-11-02T18:46:01.553 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.554 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002657e0088d0bc695-006543ee69-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.606 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.607 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.608 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.608 DEBUG:boto:path=/ibtljn-59/?prefix=obj92 2023-11-02T18:46:01.609 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj92 2023-11-02T18:46:01.609 DEBUG:boto:Method: GET 2023-11-02T18:46:01.610 DEBUG:boto:Path: /ibtljn-59/?prefix=obj92 2023-11-02T18:46:01.610 DEBUG:boto:Data: 2023-11-02T18:46:01.610 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.611 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.611 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.612 DEBUG:boto:Params: {} 2023-11-02T18:46:01.612 DEBUG:boto:Token: None 2023-11-02T18:46:01.613 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.613 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089b238db009ed916-006543ee69-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.624 DEBUG:boto:b'ibtljn-59obj921000falseobj922023-11-02T18:39:10.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T18:46:01.625 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:46:01.625 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:46:01.626 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.626 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.627 DEBUG:boto:Method: GET 2023-11-02T18:46:01.627 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.628 DEBUG:boto:Data: 2023-11-02T18:46:01.628 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.628 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.629 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.629 DEBUG:boto:Params: {} 2023-11-02T18:46:01.630 DEBUG:boto:Token: None 2023-11-02T18:46:01.630 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.631 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XMkXyjlIL6ziu6lR2dX1StV8rfc= 2023-11-02T18:46:01.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XMkXyjlIL6ziu6lR2dX1StV8rfc=', 'Content-Length': '0'} 2023-11-02T18:46:01.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000577c139147e27ce4-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.634 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:46:01.635 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:46:01.635 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.636 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.636 DEBUG:boto:Method: GET 2023-11-02T18:46:01.636 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.637 DEBUG:boto:Data: 2023-11-02T18:46:01.637 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.638 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.638 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.639 DEBUG:boto:Params: {} 2023-11-02T18:46:01.639 DEBUG:boto:Token: None 2023-11-02T18:46:01.640 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.640 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XMkXyjlIL6ziu6lR2dX1StV8rfc= 2023-11-02T18:46:01.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XMkXyjlIL6ziu6lR2dX1StV8rfc=', 'Content-Length': '0'} 2023-11-02T18:46:01.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b4adf65f3f78c31-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.644 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:46:01.644 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:46:01.645 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.645 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.645 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.646 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.646 DEBUG:boto:Data: 2023-11-02T18:46:01.647 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.647 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.648 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.648 DEBUG:boto:Params: {} 2023-11-02T18:46:01.648 DEBUG:boto:Token: None 2023-11-02T18:46:01.649 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.649 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s7IQGJVuzPesa93s774Gx7OSXO8= 2023-11-02T18:46:01.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s7IQGJVuzPesa93s774Gx7OSXO8=', 'Content-Length': '0'} 2023-11-02T18:46:01.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003049bb6c3039260b-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.653 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:46:01.653 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:46:01.654 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.654 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.655 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.655 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.655 DEBUG:boto:Data: 2023-11-02T18:46:01.656 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.656 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.657 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.657 DEBUG:boto:Params: {} 2023-11-02T18:46:01.658 DEBUG:boto:Token: None 2023-11-02T18:46:01.658 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.658 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s7IQGJVuzPesa93s774Gx7OSXO8= 2023-11-02T18:46:01.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s7IQGJVuzPesa93s774Gx7OSXO8=', 'Content-Length': '0'} 2023-11-02T18:46:01.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4b918a17551502-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T18:46:01.663 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:46:01.664 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:46:01.664 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.665 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.665 DEBUG:boto:Method: GET 2023-11-02T18:46:01.665 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.666 DEBUG:boto:Data: 2023-11-02T18:46:01.666 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.667 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.667 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.668 DEBUG:boto:Params: {} 2023-11-02T18:46:01.668 DEBUG:boto:Token: None 2023-11-02T18:46:01.669 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.669 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XMkXyjlIL6ziu6lR2dX1StV8rfc= 2023-11-02T18:46:01.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XMkXyjlIL6ziu6lR2dX1StV8rfc=', 'Content-Length': '0'} 2023-11-02T18:46:01.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000379ba49a3839607f-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.673 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:46:01.673 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:46:01.673 DEBUG:boto:path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.674 DEBUG:boto:auth_path=/ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.674 DEBUG:boto:Method: GET 2023-11-02T18:46:01.675 DEBUG:boto:Path: /ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.675 DEBUG:boto:Data: 2023-11-02T18:46:01.676 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.676 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.677 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.677 DEBUG:boto:Params: {} 2023-11-02T18:46:01.677 DEBUG:boto:Token: None 2023-11-02T18:46:01.678 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj93?versionId=null 2023-11-02T18:46:01.679 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XMkXyjlIL6ziu6lR2dX1StV8rfc= 2023-11-02T18:46:01.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XMkXyjlIL6ziu6lR2dX1StV8rfc=', 'Content-Length': '0'} 2023-11-02T18:46:01.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a8e42d17950e1f2-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.722 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.723 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.724 DEBUG:boto:path=/ibtljn-59/?prefix=obj93 2023-11-02T18:46:01.724 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj93 2023-11-02T18:46:01.725 DEBUG:boto:Method: GET 2023-11-02T18:46:01.725 DEBUG:boto:Path: /ibtljn-59/?prefix=obj93 2023-11-02T18:46:01.725 DEBUG:boto:Data: 2023-11-02T18:46:01.726 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.726 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.727 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.727 DEBUG:boto:Params: {} 2023-11-02T18:46:01.728 DEBUG:boto:Token: None 2023-11-02T18:46:01.728 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.729 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd83d5abbe75974d-006543ee69-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.735 DEBUG:boto:b'ibtljn-59obj931000falseobj932023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.735 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.736 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.736 DEBUG:boto:path=/ibtljn-59/?prefix=obj93 2023-11-02T18:46:01.737 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj93 2023-11-02T18:46:01.737 DEBUG:boto:Method: GET 2023-11-02T18:46:01.738 DEBUG:boto:Path: /ibtljn-59/?prefix=obj93 2023-11-02T18:46:01.738 DEBUG:boto:Data: 2023-11-02T18:46:01.739 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.739 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.739 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.740 DEBUG:boto:Params: {} 2023-11-02T18:46:01.740 DEBUG:boto:Token: None 2023-11-02T18:46:01.741 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.741 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dd7c54dca56b547-006543ee69-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.752 DEBUG:boto:b'ibtljn-59obj931000falseobj932023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.753 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:46:01.754 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:46:01.754 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.755 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:46:01.755 DEBUG:boto:Data: 2023-11-02T18:46:01.755 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.756 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.756 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.757 DEBUG:boto:Params: {} 2023-11-02T18:46:01.757 DEBUG:boto:Token: None 2023-11-02T18:46:01.758 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj93 2023-11-02T18:46:01.758 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iJEWO3r8A40vdyLBNAJkzEfGuqE= 2023-11-02T18:46:01.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iJEWO3r8A40vdyLBNAJkzEfGuqE=', 'Content-Length': '0'} 2023-11-02T18:46:01.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000200ab130bbc3adae-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.761 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:46:01.762 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:46:01.762 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.763 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:46:01.763 DEBUG:boto:Data: 2023-11-02T18:46:01.764 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.764 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.765 DEBUG:boto:Params: {} 2023-11-02T18:46:01.765 DEBUG:boto:Token: None 2023-11-02T18:46:01.766 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj93 2023-11-02T18:46:01.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iJEWO3r8A40vdyLBNAJkzEfGuqE= 2023-11-02T18:46:01.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iJEWO3r8A40vdyLBNAJkzEfGuqE=', 'Content-Length': '0'} 2023-11-02T18:46:01.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bca783c840de8028-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T18:46:01.771 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:46:01.771 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:46:01.772 DEBUG:boto:Method: GET 2023-11-02T18:46:01.772 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:46:01.772 DEBUG:boto:Data: 2023-11-02T18:46:01.773 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.773 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.774 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.774 DEBUG:boto:Params: {} 2023-11-02T18:46:01.775 DEBUG:boto:Token: None 2023-11-02T18:46:01.775 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj93 2023-11-02T18:46:01.776 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zh11tn4qoForzdqV8uxKyF2Pw9Q= 2023-11-02T18:46:01.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zh11tn4qoForzdqV8uxKyF2Pw9Q=', 'Content-Length': '0'} 2023-11-02T18:46:01.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006539c5245296187a-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.779 DEBUG:boto:path=/ibtljn-59/obj93 2023-11-02T18:46:01.780 DEBUG:boto:auth_path=/ibtljn-59/obj93 2023-11-02T18:46:01.780 DEBUG:boto:Method: GET 2023-11-02T18:46:01.781 DEBUG:boto:Path: /ibtljn-59/obj93 2023-11-02T18:46:01.781 DEBUG:boto:Data: 2023-11-02T18:46:01.782 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.782 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.782 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.783 DEBUG:boto:Params: {} 2023-11-02T18:46:01.783 DEBUG:boto:Token: None 2023-11-02T18:46:01.784 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj93 2023-11-02T18:46:01.784 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zh11tn4qoForzdqV8uxKyF2Pw9Q= 2023-11-02T18:46:01.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zh11tn4qoForzdqV8uxKyF2Pw9Q=', 'Content-Length': '0'} 2023-11-02T18:46:01.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fbdd4ead0d42f23-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.788 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.789 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.789 DEBUG:boto:path=/ibtljn-59/?prefix=obj93 2023-11-02T18:46:01.790 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj93 2023-11-02T18:46:01.790 DEBUG:boto:Method: GET 2023-11-02T18:46:01.791 DEBUG:boto:Path: /ibtljn-59/?prefix=obj93 2023-11-02T18:46:01.791 DEBUG:boto:Data: 2023-11-02T18:46:01.792 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.792 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.792 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.793 DEBUG:boto:Params: {} 2023-11-02T18:46:01.793 DEBUG:boto:Token: None 2023-11-02T18:46:01.794 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.794 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea0b9ab9978b6853-006543ee69-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.842 DEBUG:boto:b'ibtljn-59obj931000falseobj932023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.843 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.844 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.844 DEBUG:boto:path=/ibtljn-59/?prefix=obj93 2023-11-02T18:46:01.844 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj93 2023-11-02T18:46:01.845 DEBUG:boto:Method: GET 2023-11-02T18:46:01.845 DEBUG:boto:Path: /ibtljn-59/?prefix=obj93 2023-11-02T18:46:01.846 DEBUG:boto:Data: 2023-11-02T18:46:01.846 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.847 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.847 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.848 DEBUG:boto:Params: {} 2023-11-02T18:46:01.848 DEBUG:boto:Token: None 2023-11-02T18:46:01.849 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.849 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3c8190b9cab898b-006543ee69-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.859 DEBUG:boto:b'ibtljn-59obj931000falseobj932023-11-02T18:42:34.377Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:46:01.861 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:46:01.861 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:46:01.861 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.862 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.862 DEBUG:boto:Method: GET 2023-11-02T18:46:01.863 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.863 DEBUG:boto:Data: 2023-11-02T18:46:01.864 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.864 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.865 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.865 DEBUG:boto:Params: {} 2023-11-02T18:46:01.866 DEBUG:boto:Token: None 2023-11-02T18:46:01.866 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.867 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l00X8pY23BzAy96PmSdSk2jaIus= 2023-11-02T18:46:01.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l00X8pY23BzAy96PmSdSk2jaIus=', 'Content-Length': '0'} 2023-11-02T18:46:01.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e23f69495938844e-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.871 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:46:01.871 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:46:01.872 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.872 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.873 DEBUG:boto:Method: GET 2023-11-02T18:46:01.873 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.873 DEBUG:boto:Data: 2023-11-02T18:46:01.874 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.874 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.875 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.875 DEBUG:boto:Params: {} 2023-11-02T18:46:01.876 DEBUG:boto:Token: None 2023-11-02T18:46:01.876 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l00X8pY23BzAy96PmSdSk2jaIus= 2023-11-02T18:46:01.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l00X8pY23BzAy96PmSdSk2jaIus=', 'Content-Length': '0'} 2023-11-02T18:46:01.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079d361df62316e4f-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.922 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:46:01.923 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:46:01.923 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.924 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.924 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.925 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.925 DEBUG:boto:Data: 2023-11-02T18:46:01.926 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.926 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.927 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.927 DEBUG:boto:Params: {} 2023-11-02T18:46:01.928 DEBUG:boto:Token: None 2023-11-02T18:46:01.928 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.929 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mIa6oPhS/QG3hfBrTXEKjYkPz3s= 2023-11-02T18:46:01.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mIa6oPhS/QG3hfBrTXEKjYkPz3s=', 'Content-Length': '0'} 2023-11-02T18:46:01.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000371590aaf5440e3a-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.933 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:46:01.933 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:46:01.934 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.934 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.935 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.935 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.936 DEBUG:boto:Data: 2023-11-02T18:46:01.936 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.936 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.937 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.937 DEBUG:boto:Params: {} 2023-11-02T18:46:01.938 DEBUG:boto:Token: None 2023-11-02T18:46:01.938 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.939 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mIa6oPhS/QG3hfBrTXEKjYkPz3s= 2023-11-02T18:46:01.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mIa6oPhS/QG3hfBrTXEKjYkPz3s=', 'Content-Length': '0'} 2023-11-02T18:46:01.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b191b912e5fd731-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:46:01.943 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:46:01.944 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:46:01.944 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.944 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.945 DEBUG:boto:Method: GET 2023-11-02T18:46:01.945 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.946 DEBUG:boto:Data: 2023-11-02T18:46:01.946 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.947 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.947 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.948 DEBUG:boto:Params: {} 2023-11-02T18:46:01.948 DEBUG:boto:Token: None 2023-11-02T18:46:01.949 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.949 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l00X8pY23BzAy96PmSdSk2jaIus= 2023-11-02T18:46:01.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l00X8pY23BzAy96PmSdSk2jaIus=', 'Content-Length': '0'} 2023-11-02T18:46:01.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc5ba5c9d863a98-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.953 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:46:01.954 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:46:01.954 DEBUG:boto:path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.955 DEBUG:boto:auth_path=/ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.955 DEBUG:boto:Method: GET 2023-11-02T18:46:01.956 DEBUG:boto:Path: /ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.956 DEBUG:boto:Data: 2023-11-02T18:46:01.956 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.957 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.957 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.958 DEBUG:boto:Params: {} 2023-11-02T18:46:01.958 DEBUG:boto:Token: None 2023-11-02T18:46:01.959 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj95?versionId=null 2023-11-02T18:46:01.959 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l00X8pY23BzAy96PmSdSk2jaIus= 2023-11-02T18:46:01.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l00X8pY23BzAy96PmSdSk2jaIus=', 'Content-Length': '0'} 2023-11-02T18:46:01.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098d35ca6d584a593-006543ee69-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.964 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.964 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.965 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:46:01.965 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:46:01.966 DEBUG:boto:Method: GET 2023-11-02T18:46:01.966 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:46:01.967 DEBUG:boto:Data: 2023-11-02T18:46:01.967 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.968 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.968 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.969 DEBUG:boto:Params: {} 2023-11-02T18:46:01.969 DEBUG:boto:Token: None 2023-11-02T18:46:01.970 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.970 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cdeb6720fbe1655-006543ee69-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.977 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.977 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:01.978 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:01.978 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:46:01.979 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:46:01.979 DEBUG:boto:Method: GET 2023-11-02T18:46:01.980 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:46:01.980 DEBUG:boto:Data: 2023-11-02T18:46:01.981 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.981 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:01.981 DEBUG:boto:Port: 8000 2023-11-02T18:46:01.982 DEBUG:boto:Params: {} 2023-11-02T18:46:01.982 DEBUG:boto:Token: None 2023-11-02T18:46:01.983 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/ 2023-11-02T18:46:01.983 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY= 2023-11-02T18:46:01.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XKSRgN0LOIIX+0riYxWWKhjueAY=', 'Content-Length': '0'} 2023-11-02T18:46:01.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000448883d72aee6e0a-006543ee69-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:01.995 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:01.995 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:46:01.996 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:46:01.996 DEBUG:boto:Method: HEAD 2023-11-02T18:46:01.997 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:46:01.997 DEBUG:boto:Data: 2023-11-02T18:46:01.997 DEBUG:boto:Headers: {} 2023-11-02T18:46:01.998 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:01.998 DEBUG:boto:Port: 8001 2023-11-02T18:46:01.999 DEBUG:boto:Params: {} 2023-11-02T18:46:01.999 DEBUG:boto:Token: None 2023-11-02T18:46:01.999 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:01 GMT /ibtljn-59/obj95 2023-11-02T18:46:02.000 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yTKq0bl2JflmTrLC4m8cFgkmr2I= 2023-11-02T18:46:02.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yTKq0bl2JflmTrLC4m8cFgkmr2I=', 'Content-Length': '0'} 2023-11-02T18:46:02.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d97c3fbd72a5380-006543ee69-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:02.004 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:46:02.004 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:46:02.005 DEBUG:boto:Method: HEAD 2023-11-02T18:46:02.005 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:46:02.006 DEBUG:boto:Data: 2023-11-02T18:46:02.006 DEBUG:boto:Headers: {} 2023-11-02T18:46:02.006 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:02.007 DEBUG:boto:Port: 8000 2023-11-02T18:46:02.007 DEBUG:boto:Params: {} 2023-11-02T18:46:02.008 DEBUG:boto:Token: None 2023-11-02T18:46:02.008 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:46:02 GMT /ibtljn-59/obj95 2023-11-02T18:46:02.009 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fv2YlVWxK5K36X68OvhEJnn1kSs= 2023-11-02T18:46:02.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fv2YlVWxK5K36X68OvhEJnn1kSs=', 'Content-Length': '0'} 2023-11-02T18:46:02.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028c6460abf284ba7-006543ee6a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:02.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:46:02.013 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:46:02.013 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:46:02.014 DEBUG:boto:Method: GET 2023-11-02T18:46:02.014 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:46:02.015 DEBUG:boto:Data: 2023-11-02T18:46:02.015 DEBUG:boto:Headers: {} 2023-11-02T18:46:02.016 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:02.016 DEBUG:boto:Port: 8001 2023-11-02T18:46:02.017 DEBUG:boto:Params: {} 2023-11-02T18:46:02.017 DEBUG:boto:Token: None 2023-11-02T18:46:02.017 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:02 GMT /ibtljn-59/obj95 2023-11-02T18:46:02.018 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nSaBYhrE/RJkJ7jzpcSPi+Q9WBE= 2023-11-02T18:46:02.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nSaBYhrE/RJkJ7jzpcSPi+Q9WBE=', 'Content-Length': '0'} 2023-11-02T18:46:02.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-59:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086e92c6c51247124-006543ee6a-1e2a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:02.066 DEBUG:boto:path=/ibtljn-59/obj95 2023-11-02T18:46:02.067 DEBUG:boto:auth_path=/ibtljn-59/obj95 2023-11-02T18:46:02.067 DEBUG:boto:Method: GET 2023-11-02T18:46:02.068 DEBUG:boto:Path: /ibtljn-59/obj95 2023-11-02T18:46:02.068 DEBUG:boto:Data: 2023-11-02T18:46:02.069 DEBUG:boto:Headers: {} 2023-11-02T18:46:02.069 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:02.070 DEBUG:boto:Port: 8000 2023-11-02T18:46:02.070 DEBUG:boto:Params: {} 2023-11-02T18:46:02.071 DEBUG:boto:Token: None 2023-11-02T18:46:02.071 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:02 GMT /ibtljn-59/obj95 2023-11-02T18:46:02.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nSaBYhrE/RJkJ7jzpcSPi+Q9WBE= 2023-11-02T18:46:02.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nSaBYhrE/RJkJ7jzpcSPi+Q9WBE=', 'Content-Length': '0'} 2023-11-02T18:46:02.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:34:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fc60f526918fdc5-006543ee6a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:46:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:02.076 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:02.076 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:02.077 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:46:02.077 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:46:02.077 DEBUG:boto:Method: GET 2023-11-02T18:46:02.078 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:46:02.078 DEBUG:boto:Data: 2023-11-02T18:46:02.079 DEBUG:boto:Headers: {} 2023-11-02T18:46:02.079 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:46:02.080 DEBUG:boto:Port: 8001 2023-11-02T18:46:02.080 DEBUG:boto:Params: {} 2023-11-02T18:46:02.080 DEBUG:boto:Token: None 2023-11-02T18:46:02.081 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:02 GMT /ibtljn-59/ 2023-11-02T18:46:02.081 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x0FsW/LcEZeFNJHwBqXqtnbcCP4= 2023-11-02T18:46:02.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x0FsW/LcEZeFNJHwBqXqtnbcCP4=', 'Content-Length': '0'} 2023-11-02T18:46:02.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000585b2a94b125ef4c-006543ee6a-1e2a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:02.087 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:02.088 DEBUG:boto:path=/ibtljn-59/ 2023-11-02T18:46:02.088 DEBUG:boto:auth_path=/ibtljn-59/ 2023-11-02T18:46:02.089 DEBUG:boto:path=/ibtljn-59/?prefix=obj95 2023-11-02T18:46:02.089 DEBUG:boto:auth_path=/ibtljn-59/?prefix=obj95 2023-11-02T18:46:02.090 DEBUG:boto:Method: GET 2023-11-02T18:46:02.090 DEBUG:boto:Path: /ibtljn-59/?prefix=obj95 2023-11-02T18:46:02.091 DEBUG:boto:Data: 2023-11-02T18:46:02.091 DEBUG:boto:Headers: {} 2023-11-02T18:46:02.091 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:46:02.092 DEBUG:boto:Port: 8000 2023-11-02T18:46:02.092 DEBUG:boto:Params: {} 2023-11-02T18:46:02.093 DEBUG:boto:Token: None 2023-11-02T18:46:02.093 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:46:02 GMT /ibtljn-59/ 2023-11-02T18:46:02.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x0FsW/LcEZeFNJHwBqXqtnbcCP4= 2023-11-02T18:46:02.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:46:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x0FsW/LcEZeFNJHwBqXqtnbcCP4=', 'Content-Length': '0'} 2023-11-02T18:46:02.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039456bb2942e3723-006543ee6a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:46:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:02.104 DEBUG:boto:b'ibtljn-59obj951000falseobj952023-11-02T18:34:50.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:46:02.105 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-59 zones={a2, a1} 2023-11-02T18:46:02.106 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2023-11-02T18:46:02.110 INFO:rgw_multi.tests:create role zone=a1 name=roles-ibtljn-1 2023-11-02T18:46:02.110 DEBUG:boto:Method: POST 2023-11-02T18:46:02.111 DEBUG:boto:Path: / 2023-11-02T18:46:02.111 DEBUG:boto:Data: 2023-11-02T18:46:02.112 DEBUG:boto:Headers: {} 2023-11-02T18:46:02.112 DEBUG:boto:Host: smithi017.front.sepia.ceph.com 2023-11-02T18:46:02.113 DEBUG:boto:Port: 8000 2023-11-02T18:46:02.113 DEBUG:boto:Params: {'RoleName': 'roles-ibtljn-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-11-02T18:46:02.113 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:46:02.114 DEBUG:boto:Token: None 2023-11-02T18:46:02.115 DEBUG:boto:CanonicalRequest: POST / host:smithi017.front.sepia.ceph.com:8000 x-amz-date:20231102T184602Z host;x-amz-date 6bb1ffec827951183d107732cd1b52209fb6d218b1f801f6f84ed548dab87b88 2023-11-02T18:46:02.115 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231102T184602Z 20231102/front/smithi017/aws4_request 8b7be19ed8664cc2c52cb610bc30fe87ba77aa202707cee819ac97973f2f61cb 2023-11-02T18:46:02.116 DEBUG:boto:Signature: 3f6926a58fcb77cae19f2ac361b5706d80370ede05ebd657e12114cc28c4305f 2023-11-02T18:46:02.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231102T184602Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=7LD6AK3P9XRGER738BKH/20231102/front/smithi017/aws4_request,SignedHeaders=host;x-amz-date,Signature=3f6926a58fcb77cae19f2ac361b5706d80370ede05ebd657e12114cc28c4305f', 'Host': 'smithi017.front.sepia.ceph.com:8000'} 2023-11-02T18:46:02.137 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a0ffc85a6d78aa58-006543ee6a-1669-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Thu, 02 Nov 2023 18:46:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:02.138 DEBUG:boto:b'98418f25-57f5-4c56-8c79-f4a3e933a813roles-ibtljn-1/arn:aws:iam:::role/roles-ibtljn-12023-11-02T18:46:02.117Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000a0ffc85a6d78aa58-006543ee6a-1669-a1' 2023-11-02T18:46:02.139 INFO:rgw_multi.tests:create role zone=a2 name=roles-ibtljn-2 2023-11-02T18:46:02.139 DEBUG:boto:Method: POST 2023-11-02T18:46:02.139 DEBUG:boto:Path: / 2023-11-02T18:46:02.140 DEBUG:boto:Data: 2023-11-02T18:46:02.140 DEBUG:boto:Headers: {} 2023-11-02T18:46:02.141 DEBUG:boto:Host: smithi017.front.sepia.ceph.com 2023-11-02T18:46:02.141 DEBUG:boto:Port: 8001 2023-11-02T18:46:02.141 DEBUG:boto:Params: {'RoleName': 'roles-ibtljn-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-11-02T18:46:02.142 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:46:02.142 DEBUG:boto:Token: None 2023-11-02T18:46:02.143 DEBUG:boto:CanonicalRequest: POST / host:smithi017.front.sepia.ceph.com:8001 x-amz-date:20231102T184602Z host;x-amz-date f51c08e18d7eef9ca70186d7798206ff481c97ba73fa042478d0accbdadf8de6 2023-11-02T18:46:02.143 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231102T184602Z 20231102/front/smithi017/aws4_request 5bc736bc19e413879d6db754632612dc32a25766038122743d2ecf98ad5f8eed 2023-11-02T18:46:02.144 DEBUG:boto:Signature: 7f24ef4e2137427806c2d68a4e79cefd8d1fbb24a23c2accc4c80e72e5b0736d 2023-11-02T18:46:02.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231102T184602Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=7LD6AK3P9XRGER738BKH/20231102/front/smithi017/aws4_request,SignedHeaders=host;x-amz-date,Signature=7f24ef4e2137427806c2d68a4e79cefd8d1fbb24a23c2accc4c80e72e5b0736d', 'Host': 'smithi017.front.sepia.ceph.com:8001'} 2023-11-02T18:46:02.183 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002f8ef36565dfffeb-006543ee6a-1e2a-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Thu, 02 Nov 2023 18:46:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:46:02.184 DEBUG:boto:b'5d74ae1f-8b80-46a7-a19d-7633b6ba658eroles-ibtljn-2/arn:aws:iam:::role/roles-ibtljn-22023-11-02T18:46:02.167Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000002f8ef36565dfffeb-006543ee6a-1e2a-a2' 2023-11-02T18:46:02.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:02.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:02.185 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:02.233 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:02.234 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:02.296 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:46:02.326 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:46:02.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:02.326 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:02.327 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:46:02.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:02.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:02.327 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:46:02.327 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T18:46:02.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:02.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:02.328 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:02.329 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:44:13.513960Z" 2023-11-02T18:46:02.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:02.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:02.329 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:46:02.329 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T18:46:02.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:02.330 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:46:02.331 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:02.331 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:46:02.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:02.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:02.333 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:02.374 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:02.374 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:02.430 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:02.420+0000 7f99dfd12f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:02.432 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:02.424+0000 7f99dfd12f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:02.474 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:46:02.475 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:46:02.475 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:46:02.475 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:46:02.476 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:46:02.476 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:46:02.477 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:02.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:02.478 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:46:02.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:02.478 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:46:02.479 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:02.479 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:02.479 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:02.480 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:02.480 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:46:02.480 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:02.480 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:46:02.480 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:02.481 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:02.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:02.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:02.481 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:46:02.482 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:02.482 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:02.482 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:46:02.482 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:02.483 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:02.483 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:02.483 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:46:02.483 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:02.483 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:02.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:02.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:02.484 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:46:02.484 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:02.485 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:02.485 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:02.485 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:02.485 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:02.486 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:02.486 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:46:02.486 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:02.486 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:02.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:02.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:02.487 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:46:02.487 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:02.488 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:02.488 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:46:02.488 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:02.488 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:02.488 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:02.489 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:46:02.489 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:02.489 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:02.489 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:02.490 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:46:02.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:02.490 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:46:02.490 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:46:02.491 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:46:02.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:02.491 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:46:02Z" 2023-11-02T18:46:02.491 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:46:02.493 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:46:02Z" } 2023-11-02T18:46:02.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:46:02.494 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:02.494 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:46:02.494 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:46:07.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:07.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:07.497 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:07.546 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:07.547 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:07.609 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:07.600+0000 7f05ca292f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:07.610 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:07.600+0000 7f05ca292f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:07.633 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:46:07.634 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:46:07.634 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:46:07.634 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:46:07.634 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:46:07.635 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:46:07.635 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:07.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:07.635 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:46:07.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:07.636 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:46:07.636 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:07.636 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:07.636 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:07.637 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:07.637 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:46:07.637 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:07.637 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:46:07.638 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:07.638 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:07.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:07.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:07.638 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:46:07.639 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:07.639 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:07.639 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:46:07.639 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:07.639 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:07.640 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:07.640 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:46:07.640 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:07.640 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:07.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:07.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:07.641 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:46:07.641 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:07.641 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:07.642 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:07.642 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:07.642 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:07.642 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:07.643 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:46:07.643 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:07.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:07.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:07.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:07.644 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:46:07.644 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:07.644 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:07.644 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:46:07.645 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:07.645 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:07.645 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:07.645 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:46:07.645 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:07.646 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:07.646 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:07.646 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:46:07.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:07.647 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:46:07.647 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:46:07.647 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:46:07.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:07.647 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:46:07Z" 2023-11-02T18:46:07.648 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:46:07.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:46:07Z" } 2023-11-02T18:46:07.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:46:07.650 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:07.650 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:46:07.651 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:46:12.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:12.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:12.653 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:12.700 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:12.701 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:12.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:12.748+0000 7fd53ca84f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:12.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:12.752+0000 7fd53ca84f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:12.788 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:46:12.788 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:46:12.788 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:46:12.788 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:46:12.789 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:46:12.789 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:46:12.789 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:12.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:12.790 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:46:12.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:12.790 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:46:12.790 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:12.791 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:12.791 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:12.791 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:12.791 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:46:12.791 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:12.792 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:46:12.792 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:12.792 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:12.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:12.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:12.793 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:46:12.793 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:12.793 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:12.793 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:46:12.794 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:12.794 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:12.794 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:12.794 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:46:12.795 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:12.795 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:12.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:12.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:12.795 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:46:12.796 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:12.796 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:12.796 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:12.796 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:12.797 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:12.797 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:12.797 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:46:12.797 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:12.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:12.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:12.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:12.798 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:46:12.798 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:12.798 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:12.799 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:46:12.799 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:12.799 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:12.799 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:12.800 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:46:12.800 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:12.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:12.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:12.800 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:46:12.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:12.801 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:46:12.801 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:46:12.801 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:46:12.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:12.802 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:46:12Z" 2023-11-02T18:46:12.802 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:46:12.803 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:46:12Z" } 2023-11-02T18:46:12.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:46:12.804 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:12.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:46:12.805 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:46:17.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:17.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:17.807 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:17.852 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:17.852 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:17.913 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:17.903+0000 7f3b33a04f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:17.915 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:17.907+0000 7f3b33a04f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:17.941 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:46:17.942 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:46:17.942 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:46:17.942 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:46:17.942 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:46:17.943 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:46:17.943 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:17.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:17.943 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:46:17.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:17.944 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:46:17.944 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:17.944 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:17.944 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:17.944 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:17.945 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:46:17.945 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:17.945 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:46:17.945 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:17.945 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:17.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:17.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:17.946 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:46:17.946 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:17.947 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:17.947 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:46:17.947 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:17.947 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:17.947 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:17.948 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:46:17.948 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:17.948 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:17.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:17.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:17.949 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:46:17.949 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:17.949 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:17.949 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:17.950 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:17.950 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:17.950 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:17.950 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:46:17.950 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:17.951 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:17.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:17.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:17.951 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:46:17.951 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:17.952 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:17.952 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:46:17.952 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:17.952 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:17.953 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:17.953 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:46:17.953 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:17.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:17.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:17.954 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:46:17.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:17.954 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:46:17.954 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:46:17.954 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:46:17.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:17.955 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:46:17Z" 2023-11-02T18:46:17.955 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:46:17.957 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:46:17Z" } 2023-11-02T18:46:17.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:46:17.958 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:17.959 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:46:17.959 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:46:22.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:22.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:22.962 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:23.009 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:23.010 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:23.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:23.063+0000 7fc9240bdf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:23.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:23.063+0000 7fc9240bdf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:23.101 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:46:23.101 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:46:23.102 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:46:23.102 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:46:23.102 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:46:23.102 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:46:23.103 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:23.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:23.103 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:46:23.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:46:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:23.105 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:23.105 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:46:23.105 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:23.105 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:46:23.105 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:23.106 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:23.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:23.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:23.106 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:46:23.106 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:23.107 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:23.107 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:46:23.107 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:23.107 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:23.108 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:23.108 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:46:23.108 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:23.108 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:23.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:23.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:23.109 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:46:23.109 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:23.110 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:23.110 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:23.110 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:23.111 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:23.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:23.111 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:46:23.111 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:23.111 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:23.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:23.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:23.112 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:46:23.112 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:23.113 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:23.113 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:46:23.113 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:23.113 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:23.114 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:23.114 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:46:23.114 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:23.114 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:23.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:23.115 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:46:23.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:23.115 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:46:23.116 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:46:23.116 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:46:23.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:23.116 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:46:23Z" 2023-11-02T18:46:23.116 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:46:23.118 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:46:23Z" } 2023-11-02T18:46:23.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:46:23.119 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:23.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:46:23.120 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:46:28.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:28.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:28.125 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:28.172 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:28.173 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:28.233 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:28.223+0000 7fcc55d1bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:28.234 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:28.223+0000 7fcc55d1bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:28.259 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:46:28.260 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:46:28.260 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:46:28.260 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:46:28.260 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:46:28.261 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:46:28.261 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:28.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:28.261 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:46:28.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:28.262 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:46:28.262 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:28.262 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:28.262 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:28.263 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:28.263 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:46:28.263 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:28.263 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:46:28.264 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:28.264 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:28.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:28.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:28.264 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:46:28.265 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:28.265 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:28.265 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:46:28.265 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:28.266 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:28.266 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:28.266 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:46:28.266 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:28.267 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:28.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:28.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:28.267 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:46:28.267 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:28.268 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:28.268 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:28.268 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:28.268 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:28.269 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:28.269 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:46:28.269 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:28.269 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:28.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:28.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:28.270 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:46:28.270 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:28.270 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:28.271 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:46:28.271 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:28.271 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:28.271 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:28.271 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:46:28.272 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:28.272 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:28.272 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:28.272 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:46:28.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:28.273 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:46:28.273 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:46:28.273 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:46:28.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:28.274 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:46:28Z" 2023-11-02T18:46:28.274 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:46:28.275 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:46:28Z" } 2023-11-02T18:46:28.276 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:46:28.276 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:28.277 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:46:28.277 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:46:33.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:33.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:33.280 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:33.383 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:33.371+0000 7f73ba6a2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:33.384 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:33.375+0000 7f73ba6a2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:33.414 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:46:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:46:33.415 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:46:33.415 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:46:33.415 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:46:33.415 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:46:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:46:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:46:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:46:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:46:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:46:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:46:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:46:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:46:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:46:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:46:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:46:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:46:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:46:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:46:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:46:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:46:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:46:33Z" 2023-11-02T18:46:33.429 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:46:33.430 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:46:33Z" } 2023-11-02T18:46:33.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:46:33.431 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:33.431 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:46:33.432 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:46:38.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:38.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:38.434 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:38.483 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:38.483 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:38.542 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:38.535+0000 7f39bdd33f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:38.544 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:38.539+0000 7f39bdd33f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:38.572 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:46:38.572 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:46:38.573 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:46:38.573 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:46:38.573 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:46:38.573 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:46:38.574 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:38.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:38.574 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:46:38.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:38.575 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:46:38.575 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:38.575 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:38.575 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:38.575 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:38.576 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:46:38.576 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:38.576 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:46:38.576 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:38.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:38.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:38.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:38.577 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:46:38.577 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:38.578 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:38.578 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:46:38.578 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:38.578 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:38.579 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:38.579 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:46:38.579 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:38.579 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:38.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:38.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:38.580 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:46:38.580 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:38.580 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:38.581 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:38.581 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:38.581 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:38.581 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:38.581 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:46:38.582 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:38.582 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:38.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:38.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:38.583 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:46:38.583 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:38.583 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:38.583 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:46:38.583 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:38.584 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:38.584 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:38.584 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:46:38.584 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:38.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:38.585 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:38.585 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:46:38.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:38.585 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:46:38.586 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:46:38.586 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:46:38.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:38.586 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:46:38Z" 2023-11-02T18:46:38.586 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:46:38.588 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:46:38Z" } 2023-11-02T18:46:38.588 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:46:38.589 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:38.589 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:46:38.589 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:46:43.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:43.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:43.592 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:43.694 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:43.687+0000 7fd9d3a2ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:43.696 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:43.691+0000 7fd9d3a2ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:43.723 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:46:43.724 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:46:43.724 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:46:43.724 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:46:43.724 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:46:43.724 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:46:43.725 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:43.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:43.726 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:46:43.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:43.726 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:46:43.726 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:43.727 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:43.727 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:43.727 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:43.727 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:46:43.727 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:43.728 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:46:43.728 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:43.728 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:43.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:43.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:43.729 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:46:43.729 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:43.729 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:43.730 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:46:43.730 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:43.730 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:43.730 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:43.731 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:46:43.731 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:43.731 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:43.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:43.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:43.732 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:46:43.732 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:43.732 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:43.732 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:43.732 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:46:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:46:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:46:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:43.736 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:46:43.736 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:43.736 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:43.736 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:43.736 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:46:43.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:43.737 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:46:43.737 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:46:43.737 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:46:43.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:43.738 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:46:43Z" 2023-11-02T18:46:43.738 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:46:43.739 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:46:43Z" } 2023-11-02T18:46:43.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:46:43.740 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:43.741 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:46:43.741 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:46:48.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:48.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:48.744 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:48.790 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:48.791 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:48.854 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:48.847+0000 7f0bbe204f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:48.856 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:48.851+0000 7f0bbe204f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:48.886 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:46:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:46:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:46:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:46:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:46:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:46:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:46:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:46:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:48.891 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:48.891 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:48.891 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:46:48.891 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:46:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:46:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:46:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:46:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:46:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:46:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:46:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:46:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:46:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:46:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:46:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:46:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:46:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:46:48Z" 2023-11-02T18:46:48.903 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:46:48.905 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:46:48Z" } 2023-11-02T18:46:48.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:46:48.906 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:48.907 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:46:48.907 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:46:53.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:53.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:53.910 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:53.955 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:53.956 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:54.011+0000 7fe97b561f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:54.020 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:54.015+0000 7fe97b561f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:54.048 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:46:54.049 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:46:54.049 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:46:54.049 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:46:54.050 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:46:54.050 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:46:54.050 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:54.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:54.050 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:46:54.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:54.051 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:46:54.051 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:54.051 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:54.052 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:54.052 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:54.052 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:46:54.052 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:54.052 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:46:54.053 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:54.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:54.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:54.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:54.054 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:46:54.054 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:54.054 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:54.054 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:46:54.054 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:54.055 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:54.055 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:54.055 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:46:54.055 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:54.056 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:54.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:54.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:54.057 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:46:54.057 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:54.057 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:54.057 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:54.058 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:54.058 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:54.058 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:54.058 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:46:54.059 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:54.059 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:54.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:54.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:54.059 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:46:54.060 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:54.060 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:54.060 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:46:54.060 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:54.060 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:54.061 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:54.061 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:46:54.061 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:54.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:54.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:54.062 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:46:54.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:54.062 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:46:54.062 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:46:54.063 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:46:54.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:54.063 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:46:54Z" 2023-11-02T18:46:54.063 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:46:54.064 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:46:54Z" } 2023-11-02T18:46:54.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:46:54.065 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:54.066 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:46:54.067 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:46:59.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:59.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:46:59.069 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:46:59.116 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:46:59.116 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:46:59.172 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:59.167+0000 7fa80ddd8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:59.174 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:46:59.167+0000 7fa80ddd8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:46:59.202 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:46:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:46:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:46:59.203 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:46:59.203 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:46:59.203 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:46:59.203 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:59.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:59.204 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:46:59.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:59.204 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:46:59.204 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:46:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:46:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:46:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:46:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:46:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:46:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:46:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:46:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:46:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:46:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:46:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:46:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:46:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:46:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:46:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:46:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:46:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:46:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:46:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:46:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:46:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:46:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:46:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:46:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:46:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:46:59Z" 2023-11-02T18:46:59.216 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:46:59.217 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:46:59Z" } 2023-11-02T18:46:59.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:46:59.218 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:46:59.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:46:59.219 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:47:04.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:04.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:04.222 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:47:04.267 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:47:04.267 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:47:04.327 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:04.323+0000 7f6643a0cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:47:04.329 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:04.323+0000 7f6643a0cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:47:04.355 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:47:04.356 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:47:04.356 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:47:04.356 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:47:04.356 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:47:04.357 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:47:04.357 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:04.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:04.357 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:47:04.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:04.358 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:47:04.358 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:04.359 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:04.359 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:47:04.359 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:04.359 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:47:04.359 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:04.360 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:47:04.361 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:04.361 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:04.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:04.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:04.362 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:47:04.362 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:04.362 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:04.362 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "", 2023-11-02T18:47:04.362 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:04.363 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:04.363 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:04.363 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "0.000000", 2023-11-02T18:47:04.363 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:04.364 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:04.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:04.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:04.364 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:47:04.364 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:04.365 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:04.365 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:47:04.365 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:04.365 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:04.365 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:04.366 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:47:04.367 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:04.367 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:04.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:04.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:04.368 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:47:04.368 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:04.369 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:04.369 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947722.231870_98.1", 2023-11-02T18:47:04.369 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:04.369 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:04.370 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:04.370 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T18:47:04.370 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:04.371 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:04.371 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:04.372 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:47:04.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:04.373 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:47:04.373 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:47:04.373 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:47:04.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:04.374 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:47:04Z" 2023-11-02T18:47:04.374 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:47:04.375 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698947722.231870_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T17:55:22.231870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:47:04Z" } 2023-11-02T18:47:04.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T18:47:04.376 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:47:04.377 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698950653.513960_526.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T18:47:04.377 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1698950762.156551_196.1 target=, shard=3 master=1_1698950762.124040_294.1 target=1_1698947722.231870_98.1 2023-11-02T18:47:09.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:09.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:09.380 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:47:09.429 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:47:09.429 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:47:09.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:09.483+0000 7f20699c1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:47:09.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:09.483+0000 7f20699c1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:47:09.521 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:47:09.521 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:47:09.522 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:47:09.522 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:47:09.522 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:47:09.522 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:47:09.522 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:09.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.523 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:47:09.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:09.523 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:47:09.523 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:09.524 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:09.524 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:47:09.524 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:09.524 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:47:09.524 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:09.525 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:47:09.525 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:09.525 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:09.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:09.526 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:47:09.526 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:09.526 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:09.526 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:47:09.527 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:09.527 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:09.527 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:09.527 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T18:47:09.527 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:09.528 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:09.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:09.529 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:47:09.529 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:09.529 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:09.529 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:47:09.529 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:09.530 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:09.530 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:09.530 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:47:09.530 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:09.530 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:09.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:09.531 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:47:09.531 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:09.532 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:09.532 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:47:09.532 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:09.532 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:09.532 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:09.533 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T18:47:09.533 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:09.533 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:09.533 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:09.533 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:47:09.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.534 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:47:09.534 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:47:09.534 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:47:09.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.535 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:47:09Z" 2023-11-02T18:47:09.535 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:47:09.536 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:47:09Z" } 2023-11-02T18:47:09.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T18:47:09.537 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:47:09.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:47:09.538 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:47:09.539 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-ibtljn-1 zones={a2, a1} 2023-11-02T18:47:09.539 DEBUG:boto:Method: POST 2023-11-02T18:47:09.540 DEBUG:boto:Path: / 2023-11-02T18:47:09.540 DEBUG:boto:Data: 2023-11-02T18:47:09.540 DEBUG:boto:Headers: {} 2023-11-02T18:47:09.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com 2023-11-02T18:47:09.542 DEBUG:boto:Port: 8001 2023-11-02T18:47:09.542 DEBUG:boto:Params: {'RoleName': 'roles-ibtljn-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-11-02T18:47:09.543 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:47:09.543 DEBUG:boto:Token: None 2023-11-02T18:47:09.544 DEBUG:boto:CanonicalRequest: POST / host:smithi017.front.sepia.ceph.com:8001 x-amz-date:20231102T184709Z host;x-amz-date db11cd4131fcf08f6a3566c030ac3b10918577c652580f3ec975468371f77cf8 2023-11-02T18:47:09.545 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231102T184709Z 20231102/front/smithi017/aws4_request 484fda996641cd0d77ee7beca8ba391b39a2239b91b4fca39f0f79fca625ee47 2023-11-02T18:47:09.545 DEBUG:boto:Signature: 896a46e307fca5b05675e9ee2906fb5d1109bd6a26be5399b75f6ac6841d2a44 2023-11-02T18:47:09.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231102T184709Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=7LD6AK3P9XRGER738BKH/20231102/front/smithi017/aws4_request,SignedHeaders=host;x-amz-date,Signature=896a46e307fca5b05675e9ee2906fb5d1109bd6a26be5399b75f6ac6841d2a44', 'Host': 'smithi017.front.sepia.ceph.com:8001'} 2023-11-02T18:47:09.551 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000040f62190ba53ee7f-006543eead-1e2a-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Thu, 02 Nov 2023 18:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:09.551 DEBUG:boto:b'tx0000040f62190ba53ee7f-006543eead-1e2a-a298418f25-57f5-4c56-8c79-f4a3e933a813roles-ibtljn-1/arn:aws:iam:::role/roles-ibtljn-12023-11-02T18:47:07.106Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-11-02T18:47:09.553 DEBUG:boto:Method: POST 2023-11-02T18:47:09.553 DEBUG:boto:Path: / 2023-11-02T18:47:09.553 DEBUG:boto:Data: 2023-11-02T18:47:09.554 DEBUG:boto:Headers: {} 2023-11-02T18:47:09.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com 2023-11-02T18:47:09.555 DEBUG:boto:Port: 8000 2023-11-02T18:47:09.555 DEBUG:boto:Params: {'RoleName': 'roles-ibtljn-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-11-02T18:47:09.556 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:47:09.556 DEBUG:boto:Token: None 2023-11-02T18:47:09.557 DEBUG:boto:CanonicalRequest: POST / host:smithi017.front.sepia.ceph.com:8000 x-amz-date:20231102T184709Z host;x-amz-date db11cd4131fcf08f6a3566c030ac3b10918577c652580f3ec975468371f77cf8 2023-11-02T18:47:09.557 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231102T184709Z 20231102/front/smithi017/aws4_request a298a2f38085acd9e08382251d29e06a5d3a9b018359951c9e47c58bc8a5e1d8 2023-11-02T18:47:09.557 DEBUG:boto:Signature: 96554be81b88af02c136c654620f182bbcbe1c88300699f4a1946a1a284f927d 2023-11-02T18:47:09.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231102T184709Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=7LD6AK3P9XRGER738BKH/20231102/front/smithi017/aws4_request,SignedHeaders=host;x-amz-date,Signature=96554be81b88af02c136c654620f182bbcbe1c88300699f4a1946a1a284f927d', 'Host': 'smithi017.front.sepia.ceph.com:8000'} 2023-11-02T18:47:09.561 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000038f1efb7439aef40-006543eead-1669-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Thu, 02 Nov 2023 18:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:09.562 DEBUG:boto:b'tx0000038f1efb7439aef40-006543eead-1669-a198418f25-57f5-4c56-8c79-f4a3e933a813roles-ibtljn-1/arn:aws:iam:::role/roles-ibtljn-12023-11-02T18:46:02.117Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-11-02T18:47:09.563 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-ibtljn-1 2023-11-02T18:47:09.563 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-ibtljn-1 zones={a2, a1} 2023-11-02T18:47:09.564 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-ibtljn-2 zones={a1, a2} 2023-11-02T18:47:09.564 DEBUG:boto:Method: POST 2023-11-02T18:47:09.565 DEBUG:boto:Path: / 2023-11-02T18:47:09.565 DEBUG:boto:Data: 2023-11-02T18:47:09.565 DEBUG:boto:Headers: {} 2023-11-02T18:47:09.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com 2023-11-02T18:47:09.566 DEBUG:boto:Port: 8000 2023-11-02T18:47:09.567 DEBUG:boto:Params: {'RoleName': 'roles-ibtljn-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-11-02T18:47:09.567 DEBUG:boto:Token: None 2023-11-02T18:47:09.568 DEBUG:boto:CanonicalRequest: POST / host:smithi017.front.sepia.ceph.com:8000 x-amz-date:20231102T184709Z host;x-amz-date fdbb0ffca7f04b93864050eae4f114ca849881285c8f84db0cdf42337034dc18 2023-11-02T18:47:09.568 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231102T184709Z 20231102/front/smithi017/aws4_request e48c3666e1f4ae16f76afa516187a81c9fe70030b30360877b6f7608d9c66412 2023-11-02T18:47:09.569 DEBUG:boto:Signature: f21513a7d9cd9ecd2b48c2881f83b25505cb0e0077fdfa2bd3cbe462c3b478e0 2023-11-02T18:47:09.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231102T184709Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=7LD6AK3P9XRGER738BKH/20231102/front/smithi017/aws4_request,SignedHeaders=host;x-amz-date,Signature=f21513a7d9cd9ecd2b48c2881f83b25505cb0e0077fdfa2bd3cbe462c3b478e0', 'Host': 'smithi017.front.sepia.ceph.com:8000'} 2023-11-02T18:47:09.571 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000047a495aeee2d21ab-006543eead-1669-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Thu, 02 Nov 2023 18:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:09.572 DEBUG:boto:b'tx0000047a495aeee2d21ab-006543eead-1669-a15d74ae1f-8b80-46a7-a19d-7633b6ba658eroles-ibtljn-2/arn:aws:iam:::role/roles-ibtljn-22023-11-02T18:46:02.149Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-11-02T18:47:09.573 DEBUG:boto:Method: POST 2023-11-02T18:47:09.573 DEBUG:boto:Path: / 2023-11-02T18:47:09.574 DEBUG:boto:Data: 2023-11-02T18:47:09.574 DEBUG:boto:Headers: {} 2023-11-02T18:47:09.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com 2023-11-02T18:47:09.575 DEBUG:boto:Port: 8001 2023-11-02T18:47:09.575 DEBUG:boto:Params: {'RoleName': 'roles-ibtljn-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-11-02T18:47:09.576 DEBUG:boto:Token: None 2023-11-02T18:47:09.576 DEBUG:boto:CanonicalRequest: POST / host:smithi017.front.sepia.ceph.com:8001 x-amz-date:20231102T184709Z host;x-amz-date fdbb0ffca7f04b93864050eae4f114ca849881285c8f84db0cdf42337034dc18 2023-11-02T18:47:09.577 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231102T184709Z 20231102/front/smithi017/aws4_request 8b1965215c367ae89b217a9b1bf658044f269195565c47cf04fa9127fe9125cc 2023-11-02T18:47:09.577 DEBUG:boto:Signature: 4497d46129becc9bee12a67f4fa839fc16df7ad058e0867d70973dcc45bcfeb8 2023-11-02T18:47:09.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231102T184709Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=7LD6AK3P9XRGER738BKH/20231102/front/smithi017/aws4_request,SignedHeaders=host;x-amz-date,Signature=4497d46129becc9bee12a67f4fa839fc16df7ad058e0867d70973dcc45bcfeb8', 'Host': 'smithi017.front.sepia.ceph.com:8001'} 2023-11-02T18:47:09.581 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000690be3b372f66061-006543eead-1e2a-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Thu, 02 Nov 2023 18:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:09.582 DEBUG:boto:b'tx00000690be3b372f66061-006543eead-1e2a-a25d74ae1f-8b80-46a7-a19d-7633b6ba658eroles-ibtljn-2/arn:aws:iam:::role/roles-ibtljn-22023-11-02T18:46:02.149Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-11-02T18:47:09.583 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-ibtljn-2 2023-11-02T18:47:09.583 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-ibtljn-2 zones={a1, a2} 2023-11-02T18:47:09.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2023-11-02T18:47:09.585 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:47:09.585 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:47:09.586 DEBUG:boto:Method: PUT 2023-11-02T18:47:09.586 DEBUG:boto:Path: /ibtljn-60/ 2023-11-02T18:47:09.587 DEBUG:boto:Data: 2023-11-02T18:47:09.587 DEBUG:boto:Headers: {} 2023-11-02T18:47:09.587 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:47:09.588 DEBUG:boto:Port: 8000 2023-11-02T18:47:09.588 DEBUG:boto:Params: {} 2023-11-02T18:47:09.589 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:47:09.589 DEBUG:boto:Token: None 2023-11-02T18:47:09.590 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:47:09 GMT /ibtljn-60/ 2023-11-02T18:47:09.590 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AkT1lCJbEUwY6TtnoMsrlR1sQAY= 2023-11-02T18:47:09.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:47:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AkT1lCJbEUwY6TtnoMsrlR1sQAY=', 'Content-Length': '0'} 2023-11-02T18:47:09.627 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005fac71c0e6100d85-006543eead-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:47:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:09.628 DEBUG:rgw_multi.tests:created bucket=ibtljn-60 2023-11-02T18:47:09.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:09.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:09.629 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:47:09.670 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:47:09.670 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:47:09.727 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:47:09.757 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:47:09.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:09.758 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:47:09.758 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:47:09.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:09.758 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:47:09.759 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T18:47:09.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:09.759 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950829.620500_530.1", 2023-11-02T18:47:09.760 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:47:09.620500Z" 2023-11-02T18:47:09.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:09.760 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:47:09.760 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T18:47:09.761 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:09.761 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:47:09.762 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950829.620500_530.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:47:09.763 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:47:09.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:09.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:09.764 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:47:09.850 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:47:09.850 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:47:09.906 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:09.899+0000 7fac74d09f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:47:09.907 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:09.903+0000 7fac74d09f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:47:09.934 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:47:09.934 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:47:09.935 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:47:09.935 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:47:09.935 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:47:09.935 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:47:09.935 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:09.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.936 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:47:09.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:09.936 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:47:09.936 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:09.937 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:09.937 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:47:09.937 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:09.937 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:47:09.937 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:09.938 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:47:09.938 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:09.938 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:09.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:09.939 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:47:09.939 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:09.939 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:09.939 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:47:09.940 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:09.940 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:09.940 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:09.940 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T18:47:09.940 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:09.941 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:09.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:09.941 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:47:09.941 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:09.942 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:09.942 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:47:09.942 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:09.942 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:09.943 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:09.943 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:47:09.943 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:09.943 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:09.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:09.944 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:47:09.944 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:09.944 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:09.944 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:47:09.945 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:09.945 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:09.945 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:09.945 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T18:47:09.945 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:09.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:09.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:09.946 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:47:09.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.947 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:47:09.947 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:47:09.947 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:47:09.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:09.947 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:47:09Z" 2023-11-02T18:47:09.948 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:47:09.949 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:47:09Z" } 2023-11-02T18:47:09.949 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T18:47:09.950 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950829.620500_530.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:47:09.950 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:47:09.950 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698950829.620500_530.1 target=1_1698950653.513960_526.1 2023-11-02T18:47:14.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:14.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:14.953 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:47:15.001 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:47:15.001 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:47:15.057 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:15.051+0000 7f1286a86f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:47:15.059 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:15.055+0000 7f1286a86f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:47:15.087 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:47:15.087 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:47:15.087 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:47:15.088 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:47:15.088 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:47:15.088 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:47:15.088 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:15.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:15.089 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:47:15.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:15.089 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:47:15.090 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:15.090 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:15.090 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:47:15.090 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:15.090 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:47:15.091 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:15.091 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:47:15.091 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:15.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:15.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:15.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:15.092 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:47:15.092 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:15.092 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:15.093 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:47:15.093 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:15.093 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:15.093 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:15.093 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T18:47:15.094 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:15.094 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:15.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:15.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:15.095 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:47:15.095 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:15.095 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:15.095 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:47:15.095 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:15.096 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:15.096 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:15.097 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:47:15.097 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:15.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:15.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:15.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:15.098 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:47:15.099 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:15.099 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:15.099 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:47:15.099 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:15.100 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:15.100 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:15.100 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T18:47:15.100 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:15.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:15.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:15.101 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:47:15.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:15.102 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:47:15.102 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:47:15.102 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:47:15.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:15.103 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:47:15Z" 2023-11-02T18:47:15.103 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:47:15.104 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:47:15Z" } 2023-11-02T18:47:15.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T18:47:15.105 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950829.620500_530.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:47:15.105 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:47:15.106 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698950829.620500_530.1 target=1_1698950653.513960_526.1 2023-11-02T18:47:20.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:20.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:20.108 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:47:20.154 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:47:20.154 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:47:20.216 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:20.211+0000 7f2a8f320f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:47:20.218 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:20.211+0000 7f2a8f320f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:47:20.245 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:47:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:47:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:47:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:47:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:47:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:47:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:47:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:47:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:47:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:47:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:47:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:47:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:47:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:20.252 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:20.252 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T18:47:20.252 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:20.252 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:47:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950653.513960_526.1", 2023-11-02T18:47:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.513960Z", 2023-11-02T18:47:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:47:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:47:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T18:47:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:47:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:47:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:47:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:47:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:47:20Z" 2023-11-02T18:47:20.260 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:47:20.261 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950653.513960_526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:44:13.513960Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:47:20Z" } 2023-11-02T18:47:20.261 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950653.513960_526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.513960Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T18:47:20.262 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950829.620500_530.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:47:20.262 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950653.513960_526.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:47:20.263 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698950829.620500_530.1 target=1_1698950653.513960_526.1 2023-11-02T18:47:25.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:25.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:47:25.265 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:47:25.310 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:47:25.311 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:47:25.371 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:25.363+0000 7f3c3f501f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:47:25.372 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:25.367+0000 7f3c3f501f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:47:25.402 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:47:25.402 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:47:25.402 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:47:25.403 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:47:25.403 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:47:25.403 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:47:25.403 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:25.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:25.404 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:47:25.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:25.404 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:47:25.405 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:25.405 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:25.405 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:47:25.405 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:25.406 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:47:25.406 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:25.406 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:47:25.406 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:25.407 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:25.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:25.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:25.407 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:47:25.408 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:25.408 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:25.408 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:47:25.408 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:25.409 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:25.409 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:25.409 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T18:47:25.409 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:25.410 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:25.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:25.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:25.411 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:47:25.411 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:25.411 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:25.411 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950829.620500_530.1", 2023-11-02T18:47:25.412 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:25.413 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:25.413 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:25.413 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:47:09.620500Z", 2023-11-02T18:47:25.414 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:25.414 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:25.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:25.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:47:25.415 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:47:25.415 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:47:25.415 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:47:25.416 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:47:25.416 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:47:25.416 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:47:25.416 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:47:25.416 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T18:47:25.417 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:47:25.417 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:25.417 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:47:25.418 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:47:25.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:25.418 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:47:25.418 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:47:25.419 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:47:25.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:47:25.419 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:47:25Z" 2023-11-02T18:47:25.419 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:47:25.421 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698950829.620500_530.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:47:09.620500Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:47:25Z" } 2023-11-02T18:47:25.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698950829.620500_530.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:47:09.620500Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T18:47:25.422 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950829.620500_530.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:47:25.422 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698950829.620500_530.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:47:25.423 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:47:25.423 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-11-02T18:47:25.424 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T18:47:25.425 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:47:25.419+0000 7f71ce812700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 73886) UID: 0 2023-11-02T18:47:25.425 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:47:25.419+0000 7f71d2071f80 -1 shutting down 2023-11-02T18:47:30.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-11-02T18:47:31.526 INFO:tasks.rgw.c1.client.1:Stopped 2023-11-02T18:47:31.527 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:47:31.527 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:47:31.528 DEBUG:boto:Method: HEAD 2023-11-02T18:47:31.528 DEBUG:boto:Path: /ibtljn-60/ 2023-11-02T18:47:31.529 DEBUG:boto:Data: 2023-11-02T18:47:31.529 DEBUG:boto:Headers: {} 2023-11-02T18:47:31.529 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:47:31.530 DEBUG:boto:Port: 8000 2023-11-02T18:47:31.530 DEBUG:boto:Params: {} 2023-11-02T18:47:31.531 DEBUG:boto:Token: None 2023-11-02T18:47:31.531 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:47:31 GMT /ibtljn-60/ 2023-11-02T18:47:31.532 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fbXb6AXu0Oz2GxI2BJadw/2epaE= 2023-11-02T18:47:31.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:47:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fbXb6AXu0Oz2GxI2BJadw/2epaE=', 'Content-Length': '0'} 2023-11-02T18:47:31.537 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a406eb13744da393-006543eec3-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:31.538 DEBUG:boto:path=/ibtljn-60/obj11 2023-11-02T18:47:31.538 DEBUG:boto:auth_path=/ibtljn-60/obj11 2023-11-02T18:47:31.539 DEBUG:boto:Method: PUT 2023-11-02T18:47:31.539 DEBUG:boto:Path: /ibtljn-60/obj11 2023-11-02T18:47:31.540 DEBUG:boto:Data: 2023-11-02T18:47:31.540 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:47:31.540 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:47:31.541 DEBUG:boto:Port: 8000 2023-11-02T18:47:31.541 DEBUG:boto:Params: {} 2023-11-02T18:47:31.542 DEBUG:boto:Token: None 2023-11-02T18:47:31.542 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:47:31 GMT /ibtljn-60/obj11 2023-11-02T18:47:31.543 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HtnygpskKTrZw4aU057loQPP+T0= 2023-11-02T18:47:31.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:47:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HtnygpskKTrZw4aU057loQPP+T0='} 2023-11-02T18:47:31.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f617375f840faeaa-006543eec3-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:31.560 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:47:31.561 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:47:31.561 DEBUG:boto:Method: HEAD 2023-11-02T18:47:31.562 DEBUG:boto:Path: /ibtljn-60/ 2023-11-02T18:47:31.562 DEBUG:boto:Data: 2023-11-02T18:47:31.562 DEBUG:boto:Headers: {} 2023-11-02T18:47:31.563 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:47:31.563 DEBUG:boto:Port: 8000 2023-11-02T18:47:31.564 DEBUG:boto:Params: {} 2023-11-02T18:47:31.564 DEBUG:boto:Token: None 2023-11-02T18:47:31.565 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:47:31 GMT /ibtljn-60/ 2023-11-02T18:47:31.565 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fbXb6AXu0Oz2GxI2BJadw/2epaE= 2023-11-02T18:47:31.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:47:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fbXb6AXu0Oz2GxI2BJadw/2epaE=', 'Content-Length': '0'} 2023-11-02T18:47:31.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000644e5ac017182aa1-006543eec3-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:31.571 DEBUG:boto:path=/ibtljn-60/obj22 2023-11-02T18:47:31.572 DEBUG:boto:auth_path=/ibtljn-60/obj22 2023-11-02T18:47:31.573 DEBUG:boto:Method: PUT 2023-11-02T18:47:31.574 DEBUG:boto:Path: /ibtljn-60/obj22 2023-11-02T18:47:31.574 DEBUG:boto:Data: 2023-11-02T18:47:31.575 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:47:31.575 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:47:31.576 DEBUG:boto:Port: 8000 2023-11-02T18:47:31.576 DEBUG:boto:Params: {} 2023-11-02T18:47:31.577 DEBUG:boto:Token: None 2023-11-02T18:47:31.577 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:47:31 GMT /ibtljn-60/obj22 2023-11-02T18:47:31.578 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fvKvne/JwweOvQTFgH/0V4P+La8= 2023-11-02T18:47:31.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:47:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fvKvne/JwweOvQTFgH/0V4P+La8='} 2023-11-02T18:47:31.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb428ed7b059a64d-006543eec3-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:31.587 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:47:31.588 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:47:31.588 DEBUG:boto:Method: HEAD 2023-11-02T18:47:31.588 DEBUG:boto:Path: /ibtljn-60/ 2023-11-02T18:47:31.589 DEBUG:boto:Data: 2023-11-02T18:47:31.589 DEBUG:boto:Headers: {} 2023-11-02T18:47:31.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:47:31.590 DEBUG:boto:Port: 8000 2023-11-02T18:47:31.590 DEBUG:boto:Params: {} 2023-11-02T18:47:31.591 DEBUG:boto:Token: None 2023-11-02T18:47:31.591 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:47:31 GMT /ibtljn-60/ 2023-11-02T18:47:31.592 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fbXb6AXu0Oz2GxI2BJadw/2epaE= 2023-11-02T18:47:31.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:47:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fbXb6AXu0Oz2GxI2BJadw/2epaE=', 'Content-Length': '0'} 2023-11-02T18:47:31.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002da3bfd6fc812b42-006543eec3-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:31.597 DEBUG:boto:path=/ibtljn-60/obj33 2023-11-02T18:47:31.598 DEBUG:boto:auth_path=/ibtljn-60/obj33 2023-11-02T18:47:31.598 DEBUG:boto:Method: PUT 2023-11-02T18:47:31.599 DEBUG:boto:Path: /ibtljn-60/obj33 2023-11-02T18:47:31.599 DEBUG:boto:Data: 2023-11-02T18:47:31.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:47:31.600 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:47:31.600 DEBUG:boto:Port: 8000 2023-11-02T18:47:31.601 DEBUG:boto:Params: {} 2023-11-02T18:47:31.601 DEBUG:boto:Token: None 2023-11-02T18:47:31.602 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:47:31 GMT /ibtljn-60/obj33 2023-11-02T18:47:31.602 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t5yCeGO1D/jPM8Vz9MXLMbNShgI= 2023-11-02T18:47:31.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:47:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t5yCeGO1D/jPM8Vz9MXLMbNShgI='} 2023-11-02T18:47:31.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c167cdfb866c9412-006543eec3-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:31.651 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:47:31.652 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:47:31.652 DEBUG:boto:Method: HEAD 2023-11-02T18:47:31.652 DEBUG:boto:Path: /ibtljn-60/ 2023-11-02T18:47:31.653 DEBUG:boto:Data: 2023-11-02T18:47:31.653 DEBUG:boto:Headers: {} 2023-11-02T18:47:31.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:47:31.654 DEBUG:boto:Port: 8000 2023-11-02T18:47:31.654 DEBUG:boto:Params: {} 2023-11-02T18:47:31.655 DEBUG:boto:Token: None 2023-11-02T18:47:31.655 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:47:31 GMT /ibtljn-60/ 2023-11-02T18:47:31.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fbXb6AXu0Oz2GxI2BJadw/2epaE= 2023-11-02T18:47:31.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:47:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fbXb6AXu0Oz2GxI2BJadw/2epaE=', 'Content-Length': '0'} 2023-11-02T18:47:31.660 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b47b7563d8c4e139-006543eec3-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:31.660 DEBUG:boto:path=/ibtljn-60/obj44 2023-11-02T18:47:31.661 DEBUG:boto:auth_path=/ibtljn-60/obj44 2023-11-02T18:47:31.661 DEBUG:boto:Method: PUT 2023-11-02T18:47:31.661 DEBUG:boto:Path: /ibtljn-60/obj44 2023-11-02T18:47:31.662 DEBUG:boto:Data: 2023-11-02T18:47:31.662 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:47:31.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:47:31.663 DEBUG:boto:Port: 8000 2023-11-02T18:47:31.664 DEBUG:boto:Params: {} 2023-11-02T18:47:31.664 DEBUG:boto:Token: None 2023-11-02T18:47:31.665 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:47:31 GMT /ibtljn-60/obj44 2023-11-02T18:47:31.665 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VPtwyKb2o9evyAL+DSM23wAbpPc= 2023-11-02T18:47:31.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:47:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VPtwyKb2o9evyAL+DSM23wAbpPc='} 2023-11-02T18:47:31.676 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d184289731049cc-006543eec3-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:31.676 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:47:31.677 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:47:31.677 DEBUG:boto:Method: HEAD 2023-11-02T18:47:31.677 DEBUG:boto:Path: /ibtljn-60/ 2023-11-02T18:47:31.678 DEBUG:boto:Data: 2023-11-02T18:47:31.678 DEBUG:boto:Headers: {} 2023-11-02T18:47:31.679 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:47:31.679 DEBUG:boto:Port: 8000 2023-11-02T18:47:31.680 DEBUG:boto:Params: {} 2023-11-02T18:47:31.680 DEBUG:boto:Token: None 2023-11-02T18:47:31.681 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:47:31 GMT /ibtljn-60/ 2023-11-02T18:47:31.681 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fbXb6AXu0Oz2GxI2BJadw/2epaE= 2023-11-02T18:47:31.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:47:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fbXb6AXu0Oz2GxI2BJadw/2epaE=', 'Content-Length': '0'} 2023-11-02T18:47:31.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080c68d1bd39c2c14-006543eec3-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:31.686 DEBUG:boto:path=/ibtljn-60/obj55 2023-11-02T18:47:31.687 DEBUG:boto:auth_path=/ibtljn-60/obj55 2023-11-02T18:47:31.687 DEBUG:boto:Method: PUT 2023-11-02T18:47:31.688 DEBUG:boto:Path: /ibtljn-60/obj55 2023-11-02T18:47:31.688 DEBUG:boto:Data: 2023-11-02T18:47:31.689 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:47:31.689 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:47:31.689 DEBUG:boto:Port: 8000 2023-11-02T18:47:31.690 DEBUG:boto:Params: {} 2023-11-02T18:47:31.690 DEBUG:boto:Token: None 2023-11-02T18:47:31.691 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:47:31 GMT /ibtljn-60/obj55 2023-11-02T18:47:31.691 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:y/EQLQCypcY7E+DUy1HyPi7U3IY= 2023-11-02T18:47:31.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:47:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:y/EQLQCypcY7E+DUy1HyPi7U3IY='} 2023-11-02T18:47:31.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec471db5b0c63b75-006543eec3-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:47:31.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:47:31.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:47:31.740 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:47:31.788 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:47:31.789 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:47:31.849 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:31.843+0000 7fbd99a5cf40 5 RGW-SYNC:data: waiting on data sync status lease 2023-11-02T18:47:31.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:31.847+0000 7fbd99a5cf40 5 RGW-SYNC:data: acquired data sync status lease 2023-11-02T18:47:31.891 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-11-02T18:47:31.892 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-11-02T18:47:31.894 INFO:tasks.rgw.c1.client.1:Started 2023-11-02T18:47:31.895 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T18:47:31.910 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T18:47:31.910 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T18:47:31.910 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-11-02T18:47:31.911 INFO:teuthology.orchestra.run.smithi017.stderr:curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8001: Connection refused 2023-11-02T18:47:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-11-02T18:47:32.912 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T18:47:32.930 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T18:47:32.930 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T18:47:32.931 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-11-02T18:47:32.932 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2023-11-02T18:47:32.932 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-60 2023-11-02T18:47:32.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:47:32.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:47:32.934 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:47:32.984 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:47:32.984 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:47:33.050 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:33.043+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:47:36.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-11-02T18:47:38.051 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:38.043+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:47:43.052 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:43.047+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:47:48.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:48.047+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:47:53.054 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:53.047+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:47:58.054 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:47:58.047+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:48:03.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:48:03.051+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:48:08.057 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:48:08.051+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:48:13.058 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:48:13.050+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:48:18.059 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:48:18.054+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:48:23.060 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:48:23.054+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:48:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:48:28.054+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:48:33.062 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:48:33.054+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:48:38.063 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:48:38.058+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:48:43.064 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:48:43.058+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:48:48.065 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:48:48.058+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:48:53.067 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:48:53.058+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:48:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:48:58.062+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:49:03.068 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:49:03.062+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:49:03.816 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:49:03.810+0000 7f1029cb4700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-11-02T18:49:08.070 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:49:08.062+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:49:13.071 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:49:13.066+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:49:18.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:49:18.066+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:49:23.073 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:49:23.066+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:49:28.074 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:49:28.066+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:49:33.075 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:49:33.070+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:49:38.077 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:49:38.070+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:49:43.078 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:49:43.070+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:49:48.079 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:49:48.074+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:49:53.080 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:49:53.074+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:49:58.081 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:49:58.074+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:50:03.082 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:03.078+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:50:08.083 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:08.078+0000 7f3ee67e6f40 1 waiting to reach incremental sync.. 2023-11-02T18:50:13.087 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:13.082+0000 7f3ee67e6f40 1 bucket sync caught up with source: 2023-11-02T18:50:13.087 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.63.6, 00000000002.63.6, 00000000002.56.6] 2023-11-02T18:50:13.088 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000001.63.6, 00000000002.63.6, 00000000002.56.6] 2023-11-02T18:50:13.088 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:13.082+0000 7f3ee67e6f40 0 bucket checkpoint complete 2023-11-02T18:50:13.118 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-60 2023-11-02T18:50:13.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:50:13.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:50:13.120 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:50:13.170 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:13.171 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:13.240 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:13.234+0000 7fd99c0f4f40 1 bucket sync caught up with source: 2023-11-02T18:50:13.240 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.108.6, 00000000002.122.6, 00000000002.83.6] 2023-11-02T18:50:13.241 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000001.108.6, 00000000002.122.6, 00000000002.83.6] 2023-11-02T18:50:13.241 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:13.234+0000 7fd99c0f4f40 0 bucket checkpoint complete 2023-11-02T18:50:13.268 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-60 zones={a2, a1} 2023-11-02T18:50:13.269 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.269 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.270 DEBUG:boto:Method: HEAD 2023-11-02T18:50:13.270 DEBUG:boto:Path: /ibtljn-60/ 2023-11-02T18:50:13.271 DEBUG:boto:Data: 2023-11-02T18:50:13.271 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.272 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.272 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.273 DEBUG:boto:Params: {} 2023-11-02T18:50:13.273 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:50:13.274 DEBUG:boto:Token: None 2023-11-02T18:50:13.274 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/ 2023-11-02T18:50:13.275 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fx5MCpl19R49lyuTe5L5nAwmpdY= 2023-11-02T18:50:13.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fx5MCpl19R49lyuTe5L5nAwmpdY=', 'Content-Length': '0'} 2023-11-02T18:50:13.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e51f22b529c66479-006543ef65-1eea-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.285 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.285 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.286 DEBUG:boto:Method: HEAD 2023-11-02T18:50:13.286 DEBUG:boto:Path: /ibtljn-60/ 2023-11-02T18:50:13.287 DEBUG:boto:Data: 2023-11-02T18:50:13.287 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.287 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.288 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.288 DEBUG:boto:Params: {} 2023-11-02T18:50:13.289 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:50:13.289 DEBUG:boto:Token: None 2023-11-02T18:50:13.290 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/ 2023-11-02T18:50:13.290 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fx5MCpl19R49lyuTe5L5nAwmpdY= 2023-11-02T18:50:13.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fx5MCpl19R49lyuTe5L5nAwmpdY=', 'Content-Length': '0'} 2023-11-02T18:50:13.296 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003641218702fc80f8-006543ef65-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.296 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:50:13.298 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.299 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.299 DEBUG:boto:path=/ibtljn-60/?versions&max-keys=999 2023-11-02T18:50:13.300 DEBUG:boto:auth_path=/ibtljn-60/?versions&max-keys=999 2023-11-02T18:50:13.300 DEBUG:boto:Method: GET 2023-11-02T18:50:13.301 DEBUG:boto:Path: /ibtljn-60/?versions&max-keys=999 2023-11-02T18:50:13.301 DEBUG:boto:Data: 2023-11-02T18:50:13.301 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.302 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.302 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.303 DEBUG:boto:Params: {} 2023-11-02T18:50:13.303 DEBUG:boto:Token: None 2023-11-02T18:50:13.304 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/?versions 2023-11-02T18:50:13.304 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mV3RkCgEE8fVrLR6k3G7kgbhboc= 2023-11-02T18:50:13.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mV3RkCgEE8fVrLR6k3G7kgbhboc=', 'Content-Length': '0'} 2023-11-02T18:50:13.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000848fab6b5641a171-006543ef65-1eea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.310 DEBUG:boto:b'ibtljn-60999falseobj11nulltrue2023-11-02T18:47:31.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:47:31.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:47:31.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:47:31.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:47:31.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:13.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:50:13.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:50:13.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:50:13.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:50:13.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:50:13.313 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:50:13.314 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.314 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.315 DEBUG:boto:path=/ibtljn-60/?versions&max-keys=999 2023-11-02T18:50:13.315 DEBUG:boto:auth_path=/ibtljn-60/?versions&max-keys=999 2023-11-02T18:50:13.316 DEBUG:boto:Method: GET 2023-11-02T18:50:13.316 DEBUG:boto:Path: /ibtljn-60/?versions&max-keys=999 2023-11-02T18:50:13.316 DEBUG:boto:Data: 2023-11-02T18:50:13.317 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.318 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.318 DEBUG:boto:Params: {} 2023-11-02T18:50:13.319 DEBUG:boto:Token: None 2023-11-02T18:50:13.320 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/?versions 2023-11-02T18:50:13.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mV3RkCgEE8fVrLR6k3G7kgbhboc= 2023-11-02T18:50:13.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mV3RkCgEE8fVrLR6k3G7kgbhboc=', 'Content-Length': '0'} 2023-11-02T18:50:13.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cd865741278506a-006543ef65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.326 DEBUG:boto:b'ibtljn-60999falseobj11nulltrue2023-11-02T18:47:31.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:47:31.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:47:31.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:47:31.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:47:31.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:13.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:50:13.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:50:13.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:50:13.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:50:13.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:50:13.329 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.329 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.330 DEBUG:boto:path=/ibtljn-60/?versions&max-keys=999 2023-11-02T18:50:13.330 DEBUG:boto:auth_path=/ibtljn-60/?versions&max-keys=999 2023-11-02T18:50:13.331 DEBUG:boto:Method: GET 2023-11-02T18:50:13.331 DEBUG:boto:Path: /ibtljn-60/?versions&max-keys=999 2023-11-02T18:50:13.332 DEBUG:boto:Data: 2023-11-02T18:50:13.332 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.333 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.333 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.333 DEBUG:boto:Params: {} 2023-11-02T18:50:13.334 DEBUG:boto:Token: None 2023-11-02T18:50:13.334 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/?versions 2023-11-02T18:50:13.335 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mV3RkCgEE8fVrLR6k3G7kgbhboc= 2023-11-02T18:50:13.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mV3RkCgEE8fVrLR6k3G7kgbhboc=', 'Content-Length': '0'} 2023-11-02T18:50:13.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc1907f5ae5303d5-006543ef65-1eea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.341 DEBUG:boto:b'ibtljn-60999falseobj11nulltrue2023-11-02T18:47:31.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:47:31.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:47:31.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:47:31.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:47:31.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:13.341 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.342 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.342 DEBUG:boto:path=/ibtljn-60/?versions&max-keys=999 2023-11-02T18:50:13.343 DEBUG:boto:auth_path=/ibtljn-60/?versions&max-keys=999 2023-11-02T18:50:13.343 DEBUG:boto:Method: GET 2023-11-02T18:50:13.344 DEBUG:boto:Path: /ibtljn-60/?versions&max-keys=999 2023-11-02T18:50:13.344 DEBUG:boto:Data: 2023-11-02T18:50:13.344 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.345 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.345 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.346 DEBUG:boto:Params: {} 2023-11-02T18:50:13.346 DEBUG:boto:Token: None 2023-11-02T18:50:13.347 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/?versions 2023-11-02T18:50:13.347 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mV3RkCgEE8fVrLR6k3G7kgbhboc= 2023-11-02T18:50:13.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mV3RkCgEE8fVrLR6k3G7kgbhboc=', 'Content-Length': '0'} 2023-11-02T18:50:13.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d57cd978997acb1-006543ef65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.353 DEBUG:boto:b'ibtljn-60999falseobj11nulltrue2023-11-02T18:47:31.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:47:31.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:47:31.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:47:31.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:47:31.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:13.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:50:13.354 DEBUG:boto:path=/ibtljn-60/obj11 2023-11-02T18:50:13.355 DEBUG:boto:auth_path=/ibtljn-60/obj11 2023-11-02T18:50:13.355 DEBUG:boto:path=/ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.356 DEBUG:boto:auth_path=/ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.356 DEBUG:boto:Method: GET 2023-11-02T18:50:13.356 DEBUG:boto:Path: /ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.357 DEBUG:boto:Data: 2023-11-02T18:50:13.357 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.358 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.359 DEBUG:boto:Params: {} 2023-11-02T18:50:13.359 DEBUG:boto:Token: None 2023-11-02T18:50:13.360 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.360 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6zSn6h/g1Rwo+r67IusWQ94+Pdc= 2023-11-02T18:50:13.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6zSn6h/g1Rwo+r67IusWQ94+Pdc=', 'Content-Length': '0'} 2023-11-02T18:50:13.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fb98e1eff5d43bd-006543ef65-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.406 DEBUG:boto:path=/ibtljn-60/obj11 2023-11-02T18:50:13.407 DEBUG:boto:auth_path=/ibtljn-60/obj11 2023-11-02T18:50:13.407 DEBUG:boto:path=/ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.408 DEBUG:boto:auth_path=/ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.408 DEBUG:boto:Method: GET 2023-11-02T18:50:13.409 DEBUG:boto:Path: /ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.409 DEBUG:boto:Data: 2023-11-02T18:50:13.410 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.410 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.410 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.411 DEBUG:boto:Params: {} 2023-11-02T18:50:13.411 DEBUG:boto:Token: None 2023-11-02T18:50:13.412 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6zSn6h/g1Rwo+r67IusWQ94+Pdc= 2023-11-02T18:50:13.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6zSn6h/g1Rwo+r67IusWQ94+Pdc=', 'Content-Length': '0'} 2023-11-02T18:50:13.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069f68d324b871b62-006543ef65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.417 DEBUG:boto:path=/ibtljn-60/obj11 2023-11-02T18:50:13.417 DEBUG:boto:auth_path=/ibtljn-60/obj11 2023-11-02T18:50:13.418 DEBUG:boto:path=/ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.418 DEBUG:boto:auth_path=/ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.419 DEBUG:boto:Method: HEAD 2023-11-02T18:50:13.419 DEBUG:boto:Path: /ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.419 DEBUG:boto:Data: 2023-11-02T18:50:13.420 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.420 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.421 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.421 DEBUG:boto:Params: {} 2023-11-02T18:50:13.421 DEBUG:boto:Token: None 2023-11-02T18:50:13.422 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.422 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:myC1c1ZoOhJVx1Ia6bRqvaPY8Xc= 2023-11-02T18:50:13.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:myC1c1ZoOhJVx1Ia6bRqvaPY8Xc=', 'Content-Length': '0'} 2023-11-02T18:50:13.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9189f1a3a8baa46-006543ef65-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.426 DEBUG:boto:path=/ibtljn-60/obj11 2023-11-02T18:50:13.427 DEBUG:boto:auth_path=/ibtljn-60/obj11 2023-11-02T18:50:13.427 DEBUG:boto:path=/ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.427 DEBUG:boto:auth_path=/ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.428 DEBUG:boto:Method: HEAD 2023-11-02T18:50:13.428 DEBUG:boto:Path: /ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.429 DEBUG:boto:Data: 2023-11-02T18:50:13.429 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.429 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.430 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.430 DEBUG:boto:Params: {} 2023-11-02T18:50:13.431 DEBUG:boto:Token: None 2023-11-02T18:50:13.431 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.432 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:myC1c1ZoOhJVx1Ia6bRqvaPY8Xc= 2023-11-02T18:50:13.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:myC1c1ZoOhJVx1Ia6bRqvaPY8Xc=', 'Content-Length': '0'} 2023-11-02T18:50:13.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6822c7fbe69d9ee-006543ef65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:50:13.436 DEBUG:boto:path=/ibtljn-60/obj11 2023-11-02T18:50:13.437 DEBUG:boto:auth_path=/ibtljn-60/obj11 2023-11-02T18:50:13.437 DEBUG:boto:path=/ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.437 DEBUG:boto:auth_path=/ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.438 DEBUG:boto:Method: GET 2023-11-02T18:50:13.438 DEBUG:boto:Path: /ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.439 DEBUG:boto:Data: 2023-11-02T18:50:13.439 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.440 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.440 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.440 DEBUG:boto:Params: {} 2023-11-02T18:50:13.441 DEBUG:boto:Token: None 2023-11-02T18:50:13.441 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.442 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6zSn6h/g1Rwo+r67IusWQ94+Pdc= 2023-11-02T18:50:13.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6zSn6h/g1Rwo+r67IusWQ94+Pdc=', 'Content-Length': '0'} 2023-11-02T18:50:13.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010ab39bb7009f572-006543ef65-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.446 DEBUG:boto:path=/ibtljn-60/obj11 2023-11-02T18:50:13.447 DEBUG:boto:auth_path=/ibtljn-60/obj11 2023-11-02T18:50:13.447 DEBUG:boto:path=/ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.448 DEBUG:boto:auth_path=/ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.448 DEBUG:boto:Method: GET 2023-11-02T18:50:13.448 DEBUG:boto:Path: /ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.449 DEBUG:boto:Data: 2023-11-02T18:50:13.449 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.450 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.450 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.451 DEBUG:boto:Params: {} 2023-11-02T18:50:13.451 DEBUG:boto:Token: None 2023-11-02T18:50:13.452 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj11?versionId=null 2023-11-02T18:50:13.452 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6zSn6h/g1Rwo+r67IusWQ94+Pdc= 2023-11-02T18:50:13.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6zSn6h/g1Rwo+r67IusWQ94+Pdc=', 'Content-Length': '0'} 2023-11-02T18:50:13.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096cce3906faf1f7e-006543ef65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.503 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.503 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.504 DEBUG:boto:path=/ibtljn-60/?prefix=obj11 2023-11-02T18:50:13.504 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj11 2023-11-02T18:50:13.505 DEBUG:boto:Method: GET 2023-11-02T18:50:13.505 DEBUG:boto:Path: /ibtljn-60/?prefix=obj11 2023-11-02T18:50:13.506 DEBUG:boto:Data: 2023-11-02T18:50:13.506 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.506 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.507 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.507 DEBUG:boto:Params: {} 2023-11-02T18:50:13.508 DEBUG:boto:Token: None 2023-11-02T18:50:13.509 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/ 2023-11-02T18:50:13.510 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE= 2023-11-02T18:50:13.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE=', 'Content-Length': '0'} 2023-11-02T18:50:13.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba35068175aaf21f-006543ef65-1eea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.516 DEBUG:boto:b'ibtljn-60obj111000falseobj112023-11-02T18:47:31.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:13.517 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.517 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.517 DEBUG:boto:path=/ibtljn-60/?prefix=obj11 2023-11-02T18:50:13.518 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj11 2023-11-02T18:50:13.518 DEBUG:boto:Method: GET 2023-11-02T18:50:13.519 DEBUG:boto:Path: /ibtljn-60/?prefix=obj11 2023-11-02T18:50:13.519 DEBUG:boto:Data: 2023-11-02T18:50:13.520 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.520 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.521 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.521 DEBUG:boto:Params: {} 2023-11-02T18:50:13.521 DEBUG:boto:Token: None 2023-11-02T18:50:13.522 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/ 2023-11-02T18:50:13.522 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE= 2023-11-02T18:50:13.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE=', 'Content-Length': '0'} 2023-11-02T18:50:13.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000102ba1c200635a50-006543ef65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.529 DEBUG:boto:b'ibtljn-60obj111000falseobj112023-11-02T18:47:31.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:13.529 DEBUG:boto:path=/ibtljn-60/obj11 2023-11-02T18:50:13.530 DEBUG:boto:auth_path=/ibtljn-60/obj11 2023-11-02T18:50:13.530 DEBUG:boto:Method: HEAD 2023-11-02T18:50:13.531 DEBUG:boto:Path: /ibtljn-60/obj11 2023-11-02T18:50:13.531 DEBUG:boto:Data: 2023-11-02T18:50:13.531 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.532 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.533 DEBUG:boto:Params: {} 2023-11-02T18:50:13.533 DEBUG:boto:Token: None 2023-11-02T18:50:13.534 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj11 2023-11-02T18:50:13.534 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BC04c22Ch2EkBzjIem9+MG81gGA= 2023-11-02T18:50:13.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BC04c22Ch2EkBzjIem9+MG81gGA=', 'Content-Length': '0'} 2023-11-02T18:50:13.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c120275d47734e77-006543ef65-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.538 DEBUG:boto:path=/ibtljn-60/obj11 2023-11-02T18:50:13.539 DEBUG:boto:auth_path=/ibtljn-60/obj11 2023-11-02T18:50:13.539 DEBUG:boto:Method: HEAD 2023-11-02T18:50:13.540 DEBUG:boto:Path: /ibtljn-60/obj11 2023-11-02T18:50:13.540 DEBUG:boto:Data: 2023-11-02T18:50:13.541 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.541 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.542 DEBUG:boto:Params: {} 2023-11-02T18:50:13.542 DEBUG:boto:Token: None 2023-11-02T18:50:13.543 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj11 2023-11-02T18:50:13.543 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BC04c22Ch2EkBzjIem9+MG81gGA= 2023-11-02T18:50:13.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BC04c22Ch2EkBzjIem9+MG81gGA=', 'Content-Length': '0'} 2023-11-02T18:50:13.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b2128b1cf6b441e-006543ef65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:50:13.548 DEBUG:boto:path=/ibtljn-60/obj11 2023-11-02T18:50:13.548 DEBUG:boto:auth_path=/ibtljn-60/obj11 2023-11-02T18:50:13.549 DEBUG:boto:Method: GET 2023-11-02T18:50:13.549 DEBUG:boto:Path: /ibtljn-60/obj11 2023-11-02T18:50:13.550 DEBUG:boto:Data: 2023-11-02T18:50:13.550 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.550 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.551 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.551 DEBUG:boto:Params: {} 2023-11-02T18:50:13.552 DEBUG:boto:Token: None 2023-11-02T18:50:13.552 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj11 2023-11-02T18:50:13.553 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:69jJgQEie7Njv9bAsko1ofXTebM= 2023-11-02T18:50:13.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:69jJgQEie7Njv9bAsko1ofXTebM=', 'Content-Length': '0'} 2023-11-02T18:50:13.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003669476e19fcaae7-006543ef65-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.557 DEBUG:boto:path=/ibtljn-60/obj11 2023-11-02T18:50:13.557 DEBUG:boto:auth_path=/ibtljn-60/obj11 2023-11-02T18:50:13.558 DEBUG:boto:Method: GET 2023-11-02T18:50:13.558 DEBUG:boto:Path: /ibtljn-60/obj11 2023-11-02T18:50:13.559 DEBUG:boto:Data: 2023-11-02T18:50:13.559 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.560 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.560 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.560 DEBUG:boto:Params: {} 2023-11-02T18:50:13.561 DEBUG:boto:Token: None 2023-11-02T18:50:13.561 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj11 2023-11-02T18:50:13.562 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:69jJgQEie7Njv9bAsko1ofXTebM= 2023-11-02T18:50:13.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:69jJgQEie7Njv9bAsko1ofXTebM=', 'Content-Length': '0'} 2023-11-02T18:50:13.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba4fe4f43d441879-006543ef65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.606 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.607 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.607 DEBUG:boto:path=/ibtljn-60/?prefix=obj11 2023-11-02T18:50:13.608 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj11 2023-11-02T18:50:13.608 DEBUG:boto:Method: GET 2023-11-02T18:50:13.609 DEBUG:boto:Path: /ibtljn-60/?prefix=obj11 2023-11-02T18:50:13.609 DEBUG:boto:Data: 2023-11-02T18:50:13.610 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.610 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.611 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.611 DEBUG:boto:Params: {} 2023-11-02T18:50:13.612 DEBUG:boto:Token: None 2023-11-02T18:50:13.612 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/ 2023-11-02T18:50:13.613 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE= 2023-11-02T18:50:13.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE=', 'Content-Length': '0'} 2023-11-02T18:50:13.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f87a3f2a61b2fa5-006543ef65-1eea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.619 DEBUG:boto:b'ibtljn-60obj111000falseobj112023-11-02T18:47:31.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:13.620 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.620 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.621 DEBUG:boto:path=/ibtljn-60/?prefix=obj11 2023-11-02T18:50:13.621 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj11 2023-11-02T18:50:13.621 DEBUG:boto:Method: GET 2023-11-02T18:50:13.622 DEBUG:boto:Path: /ibtljn-60/?prefix=obj11 2023-11-02T18:50:13.622 DEBUG:boto:Data: 2023-11-02T18:50:13.623 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.623 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.624 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.624 DEBUG:boto:Params: {} 2023-11-02T18:50:13.625 DEBUG:boto:Token: None 2023-11-02T18:50:13.625 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/ 2023-11-02T18:50:13.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE= 2023-11-02T18:50:13.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE=', 'Content-Length': '0'} 2023-11-02T18:50:13.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c08bd206b4e75f5e-006543ef65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.631 DEBUG:boto:b'ibtljn-60obj111000falseobj112023-11-02T18:47:31.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:13.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:50:13.633 DEBUG:boto:path=/ibtljn-60/obj22 2023-11-02T18:50:13.633 DEBUG:boto:auth_path=/ibtljn-60/obj22 2023-11-02T18:50:13.634 DEBUG:boto:path=/ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.634 DEBUG:boto:auth_path=/ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.635 DEBUG:boto:Method: GET 2023-11-02T18:50:13.635 DEBUG:boto:Path: /ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.635 DEBUG:boto:Data: 2023-11-02T18:50:13.636 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.636 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.637 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.637 DEBUG:boto:Params: {} 2023-11-02T18:50:13.638 DEBUG:boto:Token: None 2023-11-02T18:50:13.638 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.639 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2iZBgtOV/JPzCF56bbOIuAksDlk= 2023-11-02T18:50:13.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2iZBgtOV/JPzCF56bbOIuAksDlk=', 'Content-Length': '0'} 2023-11-02T18:50:13.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016445cb06b317502-006543ef65-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.686 DEBUG:boto:path=/ibtljn-60/obj22 2023-11-02T18:50:13.687 DEBUG:boto:auth_path=/ibtljn-60/obj22 2023-11-02T18:50:13.687 DEBUG:boto:path=/ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.688 DEBUG:boto:auth_path=/ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.688 DEBUG:boto:Method: GET 2023-11-02T18:50:13.689 DEBUG:boto:Path: /ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.689 DEBUG:boto:Data: 2023-11-02T18:50:13.690 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.691 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.691 DEBUG:boto:Params: {} 2023-11-02T18:50:13.692 DEBUG:boto:Token: None 2023-11-02T18:50:13.692 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.693 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2iZBgtOV/JPzCF56bbOIuAksDlk= 2023-11-02T18:50:13.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2iZBgtOV/JPzCF56bbOIuAksDlk=', 'Content-Length': '0'} 2023-11-02T18:50:13.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d6e88e191ff972b-006543ef65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.697 DEBUG:boto:path=/ibtljn-60/obj22 2023-11-02T18:50:13.697 DEBUG:boto:auth_path=/ibtljn-60/obj22 2023-11-02T18:50:13.698 DEBUG:boto:path=/ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.698 DEBUG:boto:auth_path=/ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.699 DEBUG:boto:Method: HEAD 2023-11-02T18:50:13.699 DEBUG:boto:Path: /ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.700 DEBUG:boto:Data: 2023-11-02T18:50:13.700 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.701 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.701 DEBUG:boto:Params: {} 2023-11-02T18:50:13.702 DEBUG:boto:Token: None 2023-11-02T18:50:13.702 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.703 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Kdbw5nIF+avp0/fcKq09RM49OC4= 2023-11-02T18:50:13.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Kdbw5nIF+avp0/fcKq09RM49OC4=', 'Content-Length': '0'} 2023-11-02T18:50:13.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000089f55b406ea2c86-006543ef65-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.707 DEBUG:boto:path=/ibtljn-60/obj22 2023-11-02T18:50:13.707 DEBUG:boto:auth_path=/ibtljn-60/obj22 2023-11-02T18:50:13.708 DEBUG:boto:path=/ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.708 DEBUG:boto:auth_path=/ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.709 DEBUG:boto:Method: HEAD 2023-11-02T18:50:13.709 DEBUG:boto:Path: /ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.709 DEBUG:boto:Data: 2023-11-02T18:50:13.710 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.710 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.711 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.711 DEBUG:boto:Params: {} 2023-11-02T18:50:13.712 DEBUG:boto:Token: None 2023-11-02T18:50:13.712 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.713 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Kdbw5nIF+avp0/fcKq09RM49OC4= 2023-11-02T18:50:13.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Kdbw5nIF+avp0/fcKq09RM49OC4=', 'Content-Length': '0'} 2023-11-02T18:50:13.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039af1d08c87abe11-006543ef65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:50:13.718 DEBUG:boto:path=/ibtljn-60/obj22 2023-11-02T18:50:13.719 DEBUG:boto:auth_path=/ibtljn-60/obj22 2023-11-02T18:50:13.719 DEBUG:boto:path=/ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.720 DEBUG:boto:auth_path=/ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.720 DEBUG:boto:Method: GET 2023-11-02T18:50:13.721 DEBUG:boto:Path: /ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.721 DEBUG:boto:Data: 2023-11-02T18:50:13.722 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.722 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.723 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.723 DEBUG:boto:Params: {} 2023-11-02T18:50:13.724 DEBUG:boto:Token: None 2023-11-02T18:50:13.724 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.725 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2iZBgtOV/JPzCF56bbOIuAksDlk= 2023-11-02T18:50:13.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2iZBgtOV/JPzCF56bbOIuAksDlk=', 'Content-Length': '0'} 2023-11-02T18:50:13.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdd6f2ba5e5a6953-006543ef65-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.729 DEBUG:boto:path=/ibtljn-60/obj22 2023-11-02T18:50:13.729 DEBUG:boto:auth_path=/ibtljn-60/obj22 2023-11-02T18:50:13.730 DEBUG:boto:path=/ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.730 DEBUG:boto:auth_path=/ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.731 DEBUG:boto:Method: GET 2023-11-02T18:50:13.731 DEBUG:boto:Path: /ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.732 DEBUG:boto:Data: 2023-11-02T18:50:13.732 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.733 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.733 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.734 DEBUG:boto:Params: {} 2023-11-02T18:50:13.734 DEBUG:boto:Token: None 2023-11-02T18:50:13.735 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj22?versionId=null 2023-11-02T18:50:13.735 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2iZBgtOV/JPzCF56bbOIuAksDlk= 2023-11-02T18:50:13.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2iZBgtOV/JPzCF56bbOIuAksDlk=', 'Content-Length': '0'} 2023-11-02T18:50:13.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023be2f64a73ac882-006543ef65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.782 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.783 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.783 DEBUG:boto:path=/ibtljn-60/?prefix=obj22 2023-11-02T18:50:13.784 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj22 2023-11-02T18:50:13.784 DEBUG:boto:Method: GET 2023-11-02T18:50:13.785 DEBUG:boto:Path: /ibtljn-60/?prefix=obj22 2023-11-02T18:50:13.786 DEBUG:boto:Data: 2023-11-02T18:50:13.787 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.787 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.787 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.788 DEBUG:boto:Params: {} 2023-11-02T18:50:13.791 DEBUG:boto:Token: None 2023-11-02T18:50:13.792 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/ 2023-11-02T18:50:13.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE= 2023-11-02T18:50:13.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE=', 'Content-Length': '0'} 2023-11-02T18:50:13.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090669310baea53ed-006543ef65-1eea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.799 DEBUG:boto:b'ibtljn-60obj221000falseobj222023-11-02T18:47:31.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:13.800 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.800 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.800 DEBUG:boto:path=/ibtljn-60/?prefix=obj22 2023-11-02T18:50:13.802 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj22 2023-11-02T18:50:13.802 DEBUG:boto:Method: GET 2023-11-02T18:50:13.803 DEBUG:boto:Path: /ibtljn-60/?prefix=obj22 2023-11-02T18:50:13.803 DEBUG:boto:Data: 2023-11-02T18:50:13.804 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.804 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.805 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.805 DEBUG:boto:Params: {} 2023-11-02T18:50:13.805 DEBUG:boto:Token: None 2023-11-02T18:50:13.806 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/ 2023-11-02T18:50:13.806 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE= 2023-11-02T18:50:13.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE=', 'Content-Length': '0'} 2023-11-02T18:50:13.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d25c10cfeb53d5e-006543ef65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.813 DEBUG:boto:b'ibtljn-60obj221000falseobj222023-11-02T18:47:31.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:13.813 DEBUG:boto:path=/ibtljn-60/obj22 2023-11-02T18:50:13.814 DEBUG:boto:auth_path=/ibtljn-60/obj22 2023-11-02T18:50:13.815 DEBUG:boto:Method: HEAD 2023-11-02T18:50:13.816 DEBUG:boto:Path: /ibtljn-60/obj22 2023-11-02T18:50:13.817 DEBUG:boto:Data: 2023-11-02T18:50:13.818 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.818 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.818 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.819 DEBUG:boto:Params: {} 2023-11-02T18:50:13.819 DEBUG:boto:Token: None 2023-11-02T18:50:13.820 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj22 2023-11-02T18:50:13.820 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bLcLWuPv5cyy1xjzr0PYR0r9a5o= 2023-11-02T18:50:13.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bLcLWuPv5cyy1xjzr0PYR0r9a5o=', 'Content-Length': '0'} 2023-11-02T18:50:13.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039b2c42aa80f73e9-006543ef65-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.824 DEBUG:boto:path=/ibtljn-60/obj22 2023-11-02T18:50:13.825 DEBUG:boto:auth_path=/ibtljn-60/obj22 2023-11-02T18:50:13.825 DEBUG:boto:Method: HEAD 2023-11-02T18:50:13.826 DEBUG:boto:Path: /ibtljn-60/obj22 2023-11-02T18:50:13.826 DEBUG:boto:Data: 2023-11-02T18:50:13.826 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.827 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.827 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.828 DEBUG:boto:Params: {} 2023-11-02T18:50:13.828 DEBUG:boto:Token: None 2023-11-02T18:50:13.829 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj22 2023-11-02T18:50:13.829 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bLcLWuPv5cyy1xjzr0PYR0r9a5o= 2023-11-02T18:50:13.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bLcLWuPv5cyy1xjzr0PYR0r9a5o=', 'Content-Length': '0'} 2023-11-02T18:50:13.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074626c5901e69c59-006543ef65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:50:13.834 DEBUG:boto:path=/ibtljn-60/obj22 2023-11-02T18:50:13.834 DEBUG:boto:auth_path=/ibtljn-60/obj22 2023-11-02T18:50:13.835 DEBUG:boto:Method: GET 2023-11-02T18:50:13.835 DEBUG:boto:Path: /ibtljn-60/obj22 2023-11-02T18:50:13.836 DEBUG:boto:Data: 2023-11-02T18:50:13.836 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.836 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.837 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.837 DEBUG:boto:Params: {} 2023-11-02T18:50:13.838 DEBUG:boto:Token: None 2023-11-02T18:50:13.838 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj22 2023-11-02T18:50:13.839 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+mhFY1OJeBxuP/JytrKus3Is5HQ= 2023-11-02T18:50:13.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+mhFY1OJeBxuP/JytrKus3Is5HQ=', 'Content-Length': '0'} 2023-11-02T18:50:13.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000011db06583d930a2-006543ef65-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.886 DEBUG:boto:path=/ibtljn-60/obj22 2023-11-02T18:50:13.887 DEBUG:boto:auth_path=/ibtljn-60/obj22 2023-11-02T18:50:13.888 DEBUG:boto:Method: GET 2023-11-02T18:50:13.888 DEBUG:boto:Path: /ibtljn-60/obj22 2023-11-02T18:50:13.889 DEBUG:boto:Data: 2023-11-02T18:50:13.890 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.890 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.891 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.892 DEBUG:boto:Params: {} 2023-11-02T18:50:13.893 DEBUG:boto:Token: None 2023-11-02T18:50:13.894 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj22 2023-11-02T18:50:13.894 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+mhFY1OJeBxuP/JytrKus3Is5HQ= 2023-11-02T18:50:13.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+mhFY1OJeBxuP/JytrKus3Is5HQ=', 'Content-Length': '0'} 2023-11-02T18:50:13.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000607fc9654900916b-006543ef65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.899 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.900 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.900 DEBUG:boto:path=/ibtljn-60/?prefix=obj22 2023-11-02T18:50:13.901 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj22 2023-11-02T18:50:13.901 DEBUG:boto:Method: GET 2023-11-02T18:50:13.901 DEBUG:boto:Path: /ibtljn-60/?prefix=obj22 2023-11-02T18:50:13.902 DEBUG:boto:Data: 2023-11-02T18:50:13.902 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.903 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.903 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.904 DEBUG:boto:Params: {} 2023-11-02T18:50:13.904 DEBUG:boto:Token: None 2023-11-02T18:50:13.905 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/ 2023-11-02T18:50:13.905 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE= 2023-11-02T18:50:13.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE=', 'Content-Length': '0'} 2023-11-02T18:50:13.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4e2ba192ec74c8b-006543ef65-1eea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.911 DEBUG:boto:b'ibtljn-60obj221000falseobj222023-11-02T18:47:31.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:13.912 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:13.912 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:13.913 DEBUG:boto:path=/ibtljn-60/?prefix=obj22 2023-11-02T18:50:13.913 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj22 2023-11-02T18:50:13.914 DEBUG:boto:Method: GET 2023-11-02T18:50:13.914 DEBUG:boto:Path: /ibtljn-60/?prefix=obj22 2023-11-02T18:50:13.917 DEBUG:boto:Data: 2023-11-02T18:50:13.918 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.919 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.919 DEBUG:boto:Params: {} 2023-11-02T18:50:13.919 DEBUG:boto:Token: None 2023-11-02T18:50:13.920 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/ 2023-11-02T18:50:13.921 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE= 2023-11-02T18:50:13.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DhzICL2pFe59IpQ+RJT0j0260kE=', 'Content-Length': '0'} 2023-11-02T18:50:13.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bffabfcacebf591-006543ef65-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.970 DEBUG:boto:b'ibtljn-60obj221000falseobj222023-11-02T18:47:31.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:13.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:50:13.972 DEBUG:boto:path=/ibtljn-60/obj33 2023-11-02T18:50:13.972 DEBUG:boto:auth_path=/ibtljn-60/obj33 2023-11-02T18:50:13.973 DEBUG:boto:path=/ibtljn-60/obj33?versionId=null 2023-11-02T18:50:13.973 DEBUG:boto:auth_path=/ibtljn-60/obj33?versionId=null 2023-11-02T18:50:13.974 DEBUG:boto:Method: GET 2023-11-02T18:50:13.974 DEBUG:boto:Path: /ibtljn-60/obj33?versionId=null 2023-11-02T18:50:13.975 DEBUG:boto:Data: 2023-11-02T18:50:13.976 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.976 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:13.977 DEBUG:boto:Port: 8001 2023-11-02T18:50:13.978 DEBUG:boto:Params: {} 2023-11-02T18:50:13.979 DEBUG:boto:Token: None 2023-11-02T18:50:13.980 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj33?versionId=null 2023-11-02T18:50:13.981 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6UzvH3ddQBiGd2e/X9jlLUZ1vek= 2023-11-02T18:50:13.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6UzvH3ddQBiGd2e/X9jlLUZ1vek=', 'Content-Length': '0'} 2023-11-02T18:50:13.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d65b08db8bdc4e4f-006543ef65-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.986 DEBUG:boto:path=/ibtljn-60/obj33 2023-11-02T18:50:13.986 DEBUG:boto:auth_path=/ibtljn-60/obj33 2023-11-02T18:50:13.986 DEBUG:boto:path=/ibtljn-60/obj33?versionId=null 2023-11-02T18:50:13.987 DEBUG:boto:auth_path=/ibtljn-60/obj33?versionId=null 2023-11-02T18:50:13.988 DEBUG:boto:Method: GET 2023-11-02T18:50:13.988 DEBUG:boto:Path: /ibtljn-60/obj33?versionId=null 2023-11-02T18:50:13.989 DEBUG:boto:Data: 2023-11-02T18:50:13.989 DEBUG:boto:Headers: {} 2023-11-02T18:50:13.990 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:13.990 DEBUG:boto:Port: 8000 2023-11-02T18:50:13.991 DEBUG:boto:Params: {} 2023-11-02T18:50:13.991 DEBUG:boto:Token: None 2023-11-02T18:50:13.992 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:13 GMT /ibtljn-60/obj33?versionId=null 2023-11-02T18:50:13.992 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6UzvH3ddQBiGd2e/X9jlLUZ1vek= 2023-11-02T18:50:13.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6UzvH3ddQBiGd2e/X9jlLUZ1vek=', 'Content-Length': '0'} 2023-11-02T18:50:13.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8adfdd34b311233-006543ef65-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:13.997 DEBUG:boto:path=/ibtljn-60/obj33 2023-11-02T18:50:13.997 DEBUG:boto:auth_path=/ibtljn-60/obj33 2023-11-02T18:50:13.998 DEBUG:boto:path=/ibtljn-60/obj33?versionId=null 2023-11-02T18:50:13.998 DEBUG:boto:auth_path=/ibtljn-60/obj33?versionId=null 2023-11-02T18:50:13.999 DEBUG:boto:Method: HEAD 2023-11-02T18:50:13.999 DEBUG:boto:Path: /ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.000 DEBUG:boto:Data: 2023-11-02T18:50:14.000 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.000 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.001 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.001 DEBUG:boto:Params: {} 2023-11-02T18:50:14.002 DEBUG:boto:Token: None 2023-11-02T18:50:14.002 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYkUyEMdNdU+TDErfRk5NTkQ9FQ= 2023-11-02T18:50:14.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYkUyEMdNdU+TDErfRk5NTkQ9FQ=', 'Content-Length': '0'} 2023-11-02T18:50:14.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b9dc477f725f9ba-006543ef65-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.007 DEBUG:boto:path=/ibtljn-60/obj33 2023-11-02T18:50:14.008 DEBUG:boto:auth_path=/ibtljn-60/obj33 2023-11-02T18:50:14.008 DEBUG:boto:path=/ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.009 DEBUG:boto:auth_path=/ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.010 DEBUG:boto:Method: HEAD 2023-11-02T18:50:14.010 DEBUG:boto:Path: /ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.011 DEBUG:boto:Data: 2023-11-02T18:50:14.012 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.012 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.013 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.014 DEBUG:boto:Params: {} 2023-11-02T18:50:14.014 DEBUG:boto:Token: None 2023-11-02T18:50:14.015 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYkUyEMdNdU+TDErfRk5NTkQ9FQ= 2023-11-02T18:50:14.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYkUyEMdNdU+TDErfRk5NTkQ9FQ=', 'Content-Length': '0'} 2023-11-02T18:50:14.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d18ee16db0ab9997-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:50:14.021 DEBUG:boto:path=/ibtljn-60/obj33 2023-11-02T18:50:14.022 DEBUG:boto:auth_path=/ibtljn-60/obj33 2023-11-02T18:50:14.023 DEBUG:boto:path=/ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.024 DEBUG:boto:auth_path=/ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.024 DEBUG:boto:Method: GET 2023-11-02T18:50:14.025 DEBUG:boto:Path: /ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.026 DEBUG:boto:Data: 2023-11-02T18:50:14.027 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.027 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.028 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.029 DEBUG:boto:Params: {} 2023-11-02T18:50:14.030 DEBUG:boto:Token: None 2023-11-02T18:50:14.031 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f4dibImXvSfNtpPClBgz+BzrMAg= 2023-11-02T18:50:14.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f4dibImXvSfNtpPClBgz+BzrMAg=', 'Content-Length': '0'} 2023-11-02T18:50:14.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f77179961808b74e-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.038 DEBUG:boto:path=/ibtljn-60/obj33 2023-11-02T18:50:14.038 DEBUG:boto:auth_path=/ibtljn-60/obj33 2023-11-02T18:50:14.039 DEBUG:boto:path=/ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.040 DEBUG:boto:auth_path=/ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.041 DEBUG:boto:Method: GET 2023-11-02T18:50:14.042 DEBUG:boto:Path: /ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.042 DEBUG:boto:Data: 2023-11-02T18:50:14.043 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.044 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.045 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.045 DEBUG:boto:Params: {} 2023-11-02T18:50:14.046 DEBUG:boto:Token: None 2023-11-02T18:50:14.047 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj33?versionId=null 2023-11-02T18:50:14.048 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f4dibImXvSfNtpPClBgz+BzrMAg= 2023-11-02T18:50:14.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f4dibImXvSfNtpPClBgz+BzrMAg=', 'Content-Length': '0'} 2023-11-02T18:50:14.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053e0fc26e01234e8-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.094 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:14.095 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:14.095 DEBUG:boto:path=/ibtljn-60/?prefix=obj33 2023-11-02T18:50:14.096 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj33 2023-11-02T18:50:14.096 DEBUG:boto:Method: GET 2023-11-02T18:50:14.097 DEBUG:boto:Path: /ibtljn-60/?prefix=obj33 2023-11-02T18:50:14.097 DEBUG:boto:Data: 2023-11-02T18:50:14.098 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.098 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.098 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.099 DEBUG:boto:Params: {} 2023-11-02T18:50:14.099 DEBUG:boto:Token: None 2023-11-02T18:50:14.100 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/ 2023-11-02T18:50:14.100 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc= 2023-11-02T18:50:14.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc=', 'Content-Length': '0'} 2023-11-02T18:50:14.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ca4f323b01c20a0-006543ef66-1eea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.107 DEBUG:boto:b'ibtljn-60obj331000falseobj332023-11-02T18:47:31.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:14.108 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:14.108 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:14.109 DEBUG:boto:path=/ibtljn-60/?prefix=obj33 2023-11-02T18:50:14.109 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj33 2023-11-02T18:50:14.109 DEBUG:boto:Method: GET 2023-11-02T18:50:14.110 DEBUG:boto:Path: /ibtljn-60/?prefix=obj33 2023-11-02T18:50:14.110 DEBUG:boto:Data: 2023-11-02T18:50:14.111 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.111 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.112 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.112 DEBUG:boto:Params: {} 2023-11-02T18:50:14.113 DEBUG:boto:Token: None 2023-11-02T18:50:14.113 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/ 2023-11-02T18:50:14.114 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc= 2023-11-02T18:50:14.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc=', 'Content-Length': '0'} 2023-11-02T18:50:14.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acc7bdc9a7ddb729-006543ef66-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.120 DEBUG:boto:b'ibtljn-60obj331000falseobj332023-11-02T18:47:31.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:14.120 DEBUG:boto:path=/ibtljn-60/obj33 2023-11-02T18:50:14.121 DEBUG:boto:auth_path=/ibtljn-60/obj33 2023-11-02T18:50:14.121 DEBUG:boto:Method: HEAD 2023-11-02T18:50:14.122 DEBUG:boto:Path: /ibtljn-60/obj33 2023-11-02T18:50:14.122 DEBUG:boto:Data: 2023-11-02T18:50:14.122 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.123 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.123 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.124 DEBUG:boto:Params: {} 2023-11-02T18:50:14.124 DEBUG:boto:Token: None 2023-11-02T18:50:14.125 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj33 2023-11-02T18:50:14.125 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OpHUmgGykA94PrQEi4Pz2SnPWO4= 2023-11-02T18:50:14.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OpHUmgGykA94PrQEi4Pz2SnPWO4=', 'Content-Length': '0'} 2023-11-02T18:50:14.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060b2577db6542e34-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.129 DEBUG:boto:path=/ibtljn-60/obj33 2023-11-02T18:50:14.130 DEBUG:boto:auth_path=/ibtljn-60/obj33 2023-11-02T18:50:14.130 DEBUG:boto:Method: HEAD 2023-11-02T18:50:14.131 DEBUG:boto:Path: /ibtljn-60/obj33 2023-11-02T18:50:14.131 DEBUG:boto:Data: 2023-11-02T18:50:14.131 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.132 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.132 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.133 DEBUG:boto:Params: {} 2023-11-02T18:50:14.133 DEBUG:boto:Token: None 2023-11-02T18:50:14.134 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj33 2023-11-02T18:50:14.134 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OpHUmgGykA94PrQEi4Pz2SnPWO4= 2023-11-02T18:50:14.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OpHUmgGykA94PrQEi4Pz2SnPWO4=', 'Content-Length': '0'} 2023-11-02T18:50:14.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009960ab91cdbfb712-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:50:14.139 DEBUG:boto:path=/ibtljn-60/obj33 2023-11-02T18:50:14.139 DEBUG:boto:auth_path=/ibtljn-60/obj33 2023-11-02T18:50:14.140 DEBUG:boto:Method: GET 2023-11-02T18:50:14.140 DEBUG:boto:Path: /ibtljn-60/obj33 2023-11-02T18:50:14.141 DEBUG:boto:Data: 2023-11-02T18:50:14.141 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.141 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.142 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.142 DEBUG:boto:Params: {} 2023-11-02T18:50:14.143 DEBUG:boto:Token: None 2023-11-02T18:50:14.143 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj33 2023-11-02T18:50:14.144 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yhc0ktIqBBrPbdoTfoo7iwerx3o= 2023-11-02T18:50:14.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yhc0ktIqBBrPbdoTfoo7iwerx3o=', 'Content-Length': '0'} 2023-11-02T18:50:14.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b08da1a7839239-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.194 DEBUG:boto:path=/ibtljn-60/obj33 2023-11-02T18:50:14.195 DEBUG:boto:auth_path=/ibtljn-60/obj33 2023-11-02T18:50:14.195 DEBUG:boto:Method: GET 2023-11-02T18:50:14.196 DEBUG:boto:Path: /ibtljn-60/obj33 2023-11-02T18:50:14.197 DEBUG:boto:Data: 2023-11-02T18:50:14.198 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.198 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.198 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.199 DEBUG:boto:Params: {} 2023-11-02T18:50:14.200 DEBUG:boto:Token: None 2023-11-02T18:50:14.200 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj33 2023-11-02T18:50:14.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yhc0ktIqBBrPbdoTfoo7iwerx3o= 2023-11-02T18:50:14.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yhc0ktIqBBrPbdoTfoo7iwerx3o=', 'Content-Length': '0'} 2023-11-02T18:50:14.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ead1764f103e07f-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.206 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:14.207 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:14.208 DEBUG:boto:path=/ibtljn-60/?prefix=obj33 2023-11-02T18:50:14.209 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj33 2023-11-02T18:50:14.209 DEBUG:boto:Method: GET 2023-11-02T18:50:14.210 DEBUG:boto:Path: /ibtljn-60/?prefix=obj33 2023-11-02T18:50:14.211 DEBUG:boto:Data: 2023-11-02T18:50:14.212 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.212 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.213 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.214 DEBUG:boto:Params: {} 2023-11-02T18:50:14.215 DEBUG:boto:Token: None 2023-11-02T18:50:14.216 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/ 2023-11-02T18:50:14.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc= 2023-11-02T18:50:14.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc=', 'Content-Length': '0'} 2023-11-02T18:50:14.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036daa0873b58793b-006543ef66-1eea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.225 DEBUG:boto:b'ibtljn-60obj331000falseobj332023-11-02T18:47:31.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:14.226 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:14.227 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:14.227 DEBUG:boto:path=/ibtljn-60/?prefix=obj33 2023-11-02T18:50:14.228 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj33 2023-11-02T18:50:14.228 DEBUG:boto:Method: GET 2023-11-02T18:50:14.229 DEBUG:boto:Path: /ibtljn-60/?prefix=obj33 2023-11-02T18:50:14.229 DEBUG:boto:Data: 2023-11-02T18:50:14.230 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.231 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.232 DEBUG:boto:Params: {} 2023-11-02T18:50:14.232 DEBUG:boto:Token: None 2023-11-02T18:50:14.233 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/ 2023-11-02T18:50:14.233 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc= 2023-11-02T18:50:14.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc=', 'Content-Length': '0'} 2023-11-02T18:50:14.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2ee709c3c8f6777-006543ef66-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.240 DEBUG:boto:b'ibtljn-60obj331000falseobj332023-11-02T18:47:31.601Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:14.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:50:14.241 DEBUG:boto:path=/ibtljn-60/obj44 2023-11-02T18:50:14.242 DEBUG:boto:auth_path=/ibtljn-60/obj44 2023-11-02T18:50:14.243 DEBUG:boto:path=/ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.243 DEBUG:boto:auth_path=/ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.244 DEBUG:boto:Method: GET 2023-11-02T18:50:14.244 DEBUG:boto:Path: /ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.246 DEBUG:boto:Data: 2023-11-02T18:50:14.246 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.247 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.248 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.249 DEBUG:boto:Params: {} 2023-11-02T18:50:14.250 DEBUG:boto:Token: None 2023-11-02T18:50:14.250 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.251 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BNZpga/Dh1s72YqxsyqsrnYU3ZE= 2023-11-02T18:50:14.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BNZpga/Dh1s72YqxsyqsrnYU3ZE=', 'Content-Length': '0'} 2023-11-02T18:50:14.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090de92cc457774c1-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.257 DEBUG:boto:path=/ibtljn-60/obj44 2023-11-02T18:50:14.258 DEBUG:boto:auth_path=/ibtljn-60/obj44 2023-11-02T18:50:14.259 DEBUG:boto:path=/ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.259 DEBUG:boto:auth_path=/ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.260 DEBUG:boto:Method: GET 2023-11-02T18:50:14.261 DEBUG:boto:Path: /ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.261 DEBUG:boto:Data: 2023-11-02T18:50:14.262 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.263 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.263 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.264 DEBUG:boto:Params: {} 2023-11-02T18:50:14.265 DEBUG:boto:Token: None 2023-11-02T18:50:14.266 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.267 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BNZpga/Dh1s72YqxsyqsrnYU3ZE= 2023-11-02T18:50:14.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BNZpga/Dh1s72YqxsyqsrnYU3ZE=', 'Content-Length': '0'} 2023-11-02T18:50:14.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dbdfcd5f14e50ab-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.314 DEBUG:boto:path=/ibtljn-60/obj44 2023-11-02T18:50:14.315 DEBUG:boto:auth_path=/ibtljn-60/obj44 2023-11-02T18:50:14.315 DEBUG:boto:path=/ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.316 DEBUG:boto:auth_path=/ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.316 DEBUG:boto:Method: HEAD 2023-11-02T18:50:14.317 DEBUG:boto:Path: /ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.317 DEBUG:boto:Data: 2023-11-02T18:50:14.318 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.318 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.318 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.319 DEBUG:boto:Params: {} 2023-11-02T18:50:14.319 DEBUG:boto:Token: None 2023-11-02T18:50:14.320 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zHX/0IDmNV1J7GJwdXy7bAgSI7Q= 2023-11-02T18:50:14.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zHX/0IDmNV1J7GJwdXy7bAgSI7Q=', 'Content-Length': '0'} 2023-11-02T18:50:14.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6fae44ac75c2e27-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.325 DEBUG:boto:path=/ibtljn-60/obj44 2023-11-02T18:50:14.325 DEBUG:boto:auth_path=/ibtljn-60/obj44 2023-11-02T18:50:14.325 DEBUG:boto:path=/ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.326 DEBUG:boto:auth_path=/ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.326 DEBUG:boto:Method: HEAD 2023-11-02T18:50:14.327 DEBUG:boto:Path: /ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.327 DEBUG:boto:Data: 2023-11-02T18:50:14.328 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.328 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.329 DEBUG:boto:Params: {} 2023-11-02T18:50:14.329 DEBUG:boto:Token: None 2023-11-02T18:50:14.330 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.333 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zHX/0IDmNV1J7GJwdXy7bAgSI7Q= 2023-11-02T18:50:14.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zHX/0IDmNV1J7GJwdXy7bAgSI7Q=', 'Content-Length': '0'} 2023-11-02T18:50:14.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc1a407f0e9b13f7-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:50:14.338 DEBUG:boto:path=/ibtljn-60/obj44 2023-11-02T18:50:14.338 DEBUG:boto:auth_path=/ibtljn-60/obj44 2023-11-02T18:50:14.339 DEBUG:boto:path=/ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.339 DEBUG:boto:auth_path=/ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.340 DEBUG:boto:Method: GET 2023-11-02T18:50:14.340 DEBUG:boto:Path: /ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.341 DEBUG:boto:Data: 2023-11-02T18:50:14.341 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.342 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.342 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.343 DEBUG:boto:Params: {} 2023-11-02T18:50:14.343 DEBUG:boto:Token: None 2023-11-02T18:50:14.344 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.344 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BNZpga/Dh1s72YqxsyqsrnYU3ZE= 2023-11-02T18:50:14.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BNZpga/Dh1s72YqxsyqsrnYU3ZE=', 'Content-Length': '0'} 2023-11-02T18:50:14.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c3b60432b6dd3a0-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.390 DEBUG:boto:path=/ibtljn-60/obj44 2023-11-02T18:50:14.391 DEBUG:boto:auth_path=/ibtljn-60/obj44 2023-11-02T18:50:14.392 DEBUG:boto:path=/ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.392 DEBUG:boto:auth_path=/ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.393 DEBUG:boto:Method: GET 2023-11-02T18:50:14.393 DEBUG:boto:Path: /ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.394 DEBUG:boto:Data: 2023-11-02T18:50:14.394 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.395 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.395 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.396 DEBUG:boto:Params: {} 2023-11-02T18:50:14.396 DEBUG:boto:Token: None 2023-11-02T18:50:14.397 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj44?versionId=null 2023-11-02T18:50:14.397 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BNZpga/Dh1s72YqxsyqsrnYU3ZE= 2023-11-02T18:50:14.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BNZpga/Dh1s72YqxsyqsrnYU3ZE=', 'Content-Length': '0'} 2023-11-02T18:50:14.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e958e231ef74b9f1-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.403 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:14.403 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:14.404 DEBUG:boto:path=/ibtljn-60/?prefix=obj44 2023-11-02T18:50:14.404 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj44 2023-11-02T18:50:14.405 DEBUG:boto:Method: GET 2023-11-02T18:50:14.405 DEBUG:boto:Path: /ibtljn-60/?prefix=obj44 2023-11-02T18:50:14.406 DEBUG:boto:Data: 2023-11-02T18:50:14.406 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.407 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.407 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.407 DEBUG:boto:Params: {} 2023-11-02T18:50:14.408 DEBUG:boto:Token: None 2023-11-02T18:50:14.409 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/ 2023-11-02T18:50:14.409 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc= 2023-11-02T18:50:14.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc=', 'Content-Length': '0'} 2023-11-02T18:50:14.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000979c1d4b36208161-006543ef66-1eea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.416 DEBUG:boto:b'ibtljn-60obj441000falseobj442023-11-02T18:47:31.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:14.417 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:14.417 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:14.418 DEBUG:boto:path=/ibtljn-60/?prefix=obj44 2023-11-02T18:50:14.418 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj44 2023-11-02T18:50:14.419 DEBUG:boto:Method: GET 2023-11-02T18:50:14.419 DEBUG:boto:Path: /ibtljn-60/?prefix=obj44 2023-11-02T18:50:14.420 DEBUG:boto:Data: 2023-11-02T18:50:14.421 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.421 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.422 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.422 DEBUG:boto:Params: {} 2023-11-02T18:50:14.423 DEBUG:boto:Token: None 2023-11-02T18:50:14.423 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/ 2023-11-02T18:50:14.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc= 2023-11-02T18:50:14.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc=', 'Content-Length': '0'} 2023-11-02T18:50:14.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c86611af289107a1-006543ef66-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.430 DEBUG:boto:b'ibtljn-60obj441000falseobj442023-11-02T18:47:31.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:14.431 DEBUG:boto:path=/ibtljn-60/obj44 2023-11-02T18:50:14.431 DEBUG:boto:auth_path=/ibtljn-60/obj44 2023-11-02T18:50:14.432 DEBUG:boto:Method: HEAD 2023-11-02T18:50:14.432 DEBUG:boto:Path: /ibtljn-60/obj44 2023-11-02T18:50:14.432 DEBUG:boto:Data: 2023-11-02T18:50:14.433 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.433 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.434 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.434 DEBUG:boto:Params: {} 2023-11-02T18:50:14.435 DEBUG:boto:Token: None 2023-11-02T18:50:14.435 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj44 2023-11-02T18:50:14.436 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:es5+rLzF5YVYO0XazvJ7QajUpe8= 2023-11-02T18:50:14.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:es5+rLzF5YVYO0XazvJ7QajUpe8=', 'Content-Length': '0'} 2023-11-02T18:50:14.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e350b5380213c57d-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.440 DEBUG:boto:path=/ibtljn-60/obj44 2023-11-02T18:50:14.441 DEBUG:boto:auth_path=/ibtljn-60/obj44 2023-11-02T18:50:14.442 DEBUG:boto:Method: HEAD 2023-11-02T18:50:14.442 DEBUG:boto:Path: /ibtljn-60/obj44 2023-11-02T18:50:14.443 DEBUG:boto:Data: 2023-11-02T18:50:14.443 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.444 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.445 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.445 DEBUG:boto:Params: {} 2023-11-02T18:50:14.446 DEBUG:boto:Token: None 2023-11-02T18:50:14.446 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj44 2023-11-02T18:50:14.447 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:es5+rLzF5YVYO0XazvJ7QajUpe8= 2023-11-02T18:50:14.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:es5+rLzF5YVYO0XazvJ7QajUpe8=', 'Content-Length': '0'} 2023-11-02T18:50:14.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acb9354033cbe2db-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:50:14.453 DEBUG:boto:path=/ibtljn-60/obj44 2023-11-02T18:50:14.453 DEBUG:boto:auth_path=/ibtljn-60/obj44 2023-11-02T18:50:14.454 DEBUG:boto:Method: GET 2023-11-02T18:50:14.454 DEBUG:boto:Path: /ibtljn-60/obj44 2023-11-02T18:50:14.455 DEBUG:boto:Data: 2023-11-02T18:50:14.455 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.456 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.456 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.457 DEBUG:boto:Params: {} 2023-11-02T18:50:14.457 DEBUG:boto:Token: None 2023-11-02T18:50:14.458 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj44 2023-11-02T18:50:14.458 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sgqgSD/ZF6jXwlEV9LEwuzw9lPI= 2023-11-02T18:50:14.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sgqgSD/ZF6jXwlEV9LEwuzw9lPI=', 'Content-Length': '0'} 2023-11-02T18:50:14.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000569a236d68b0a594-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.502 DEBUG:boto:path=/ibtljn-60/obj44 2023-11-02T18:50:14.503 DEBUG:boto:auth_path=/ibtljn-60/obj44 2023-11-02T18:50:14.504 DEBUG:boto:Method: GET 2023-11-02T18:50:14.505 DEBUG:boto:Path: /ibtljn-60/obj44 2023-11-02T18:50:14.506 DEBUG:boto:Data: 2023-11-02T18:50:14.507 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.508 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.508 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.509 DEBUG:boto:Params: {} 2023-11-02T18:50:14.510 DEBUG:boto:Token: None 2023-11-02T18:50:14.511 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj44 2023-11-02T18:50:14.512 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sgqgSD/ZF6jXwlEV9LEwuzw9lPI= 2023-11-02T18:50:14.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sgqgSD/ZF6jXwlEV9LEwuzw9lPI=', 'Content-Length': '0'} 2023-11-02T18:50:14.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000639ce3241396c2d3-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.518 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:14.518 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:14.519 DEBUG:boto:path=/ibtljn-60/?prefix=obj44 2023-11-02T18:50:14.519 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj44 2023-11-02T18:50:14.519 DEBUG:boto:Method: GET 2023-11-02T18:50:14.520 DEBUG:boto:Path: /ibtljn-60/?prefix=obj44 2023-11-02T18:50:14.520 DEBUG:boto:Data: 2023-11-02T18:50:14.521 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.522 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.523 DEBUG:boto:Params: {} 2023-11-02T18:50:14.524 DEBUG:boto:Token: None 2023-11-02T18:50:14.524 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/ 2023-11-02T18:50:14.525 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc= 2023-11-02T18:50:14.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc=', 'Content-Length': '0'} 2023-11-02T18:50:14.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000188f138c505ce6f4-006543ef66-1eea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.532 DEBUG:boto:b'ibtljn-60obj441000falseobj442023-11-02T18:47:31.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:14.532 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:14.533 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:14.533 DEBUG:boto:path=/ibtljn-60/?prefix=obj44 2023-11-02T18:50:14.534 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj44 2023-11-02T18:50:14.534 DEBUG:boto:Method: GET 2023-11-02T18:50:14.535 DEBUG:boto:Path: /ibtljn-60/?prefix=obj44 2023-11-02T18:50:14.535 DEBUG:boto:Data: 2023-11-02T18:50:14.535 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.536 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.536 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.537 DEBUG:boto:Params: {} 2023-11-02T18:50:14.537 DEBUG:boto:Token: None 2023-11-02T18:50:14.538 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/ 2023-11-02T18:50:14.538 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc= 2023-11-02T18:50:14.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc=', 'Content-Length': '0'} 2023-11-02T18:50:14.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1731a5611c3583f-006543ef66-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.544 DEBUG:boto:b'ibtljn-60obj441000falseobj442023-11-02T18:47:31.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:14.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:50:14.546 DEBUG:boto:path=/ibtljn-60/obj55 2023-11-02T18:50:14.546 DEBUG:boto:auth_path=/ibtljn-60/obj55 2023-11-02T18:50:14.547 DEBUG:boto:path=/ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.547 DEBUG:boto:auth_path=/ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.547 DEBUG:boto:Method: GET 2023-11-02T18:50:14.548 DEBUG:boto:Path: /ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.548 DEBUG:boto:Data: 2023-11-02T18:50:14.549 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.549 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.550 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.550 DEBUG:boto:Params: {} 2023-11-02T18:50:14.551 DEBUG:boto:Token: None 2023-11-02T18:50:14.551 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.552 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sqa2b4gGxndvrtx+TDZ24rcFE1w= 2023-11-02T18:50:14.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sqa2b4gGxndvrtx+TDZ24rcFE1w=', 'Content-Length': '0'} 2023-11-02T18:50:14.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064fb9179ec2b2b17-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.556 DEBUG:boto:path=/ibtljn-60/obj55 2023-11-02T18:50:14.557 DEBUG:boto:auth_path=/ibtljn-60/obj55 2023-11-02T18:50:14.557 DEBUG:boto:path=/ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.558 DEBUG:boto:auth_path=/ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.559 DEBUG:boto:Method: GET 2023-11-02T18:50:14.560 DEBUG:boto:Path: /ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.560 DEBUG:boto:Data: 2023-11-02T18:50:14.561 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.562 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.562 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.563 DEBUG:boto:Params: {} 2023-11-02T18:50:14.564 DEBUG:boto:Token: None 2023-11-02T18:50:14.564 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.565 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sqa2b4gGxndvrtx+TDZ24rcFE1w= 2023-11-02T18:50:14.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sqa2b4gGxndvrtx+TDZ24rcFE1w=', 'Content-Length': '0'} 2023-11-02T18:50:14.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c2a71923c976a4-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.610 DEBUG:boto:path=/ibtljn-60/obj55 2023-11-02T18:50:14.611 DEBUG:boto:auth_path=/ibtljn-60/obj55 2023-11-02T18:50:14.612 DEBUG:boto:path=/ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.612 DEBUG:boto:auth_path=/ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.613 DEBUG:boto:Method: HEAD 2023-11-02T18:50:14.614 DEBUG:boto:Path: /ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.614 DEBUG:boto:Data: 2023-11-02T18:50:14.615 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.615 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.616 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.616 DEBUG:boto:Params: {} 2023-11-02T18:50:14.617 DEBUG:boto:Token: None 2023-11-02T18:50:14.617 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.618 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W+DW5zUyarsJnhXiRp7pkR3WbYQ= 2023-11-02T18:50:14.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W+DW5zUyarsJnhXiRp7pkR3WbYQ=', 'Content-Length': '0'} 2023-11-02T18:50:14.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b304ee03c53b9df9-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.622 DEBUG:boto:path=/ibtljn-60/obj55 2023-11-02T18:50:14.623 DEBUG:boto:auth_path=/ibtljn-60/obj55 2023-11-02T18:50:14.623 DEBUG:boto:path=/ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.624 DEBUG:boto:auth_path=/ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.624 DEBUG:boto:Method: HEAD 2023-11-02T18:50:14.624 DEBUG:boto:Path: /ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.625 DEBUG:boto:Data: 2023-11-02T18:50:14.625 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.626 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.626 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.627 DEBUG:boto:Params: {} 2023-11-02T18:50:14.627 DEBUG:boto:Token: None 2023-11-02T18:50:14.628 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.628 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W+DW5zUyarsJnhXiRp7pkR3WbYQ= 2023-11-02T18:50:14.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W+DW5zUyarsJnhXiRp7pkR3WbYQ=', 'Content-Length': '0'} 2023-11-02T18:50:14.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ede63aef94b95de3-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:50:14.633 DEBUG:boto:path=/ibtljn-60/obj55 2023-11-02T18:50:14.634 DEBUG:boto:auth_path=/ibtljn-60/obj55 2023-11-02T18:50:14.634 DEBUG:boto:path=/ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.635 DEBUG:boto:auth_path=/ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.635 DEBUG:boto:Method: GET 2023-11-02T18:50:14.636 DEBUG:boto:Path: /ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.636 DEBUG:boto:Data: 2023-11-02T18:50:14.637 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.638 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.638 DEBUG:boto:Params: {} 2023-11-02T18:50:14.639 DEBUG:boto:Token: None 2023-11-02T18:50:14.639 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.640 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sqa2b4gGxndvrtx+TDZ24rcFE1w= 2023-11-02T18:50:14.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sqa2b4gGxndvrtx+TDZ24rcFE1w=', 'Content-Length': '0'} 2023-11-02T18:50:14.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060ea95298e250626-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.645 DEBUG:boto:path=/ibtljn-60/obj55 2023-11-02T18:50:14.646 DEBUG:boto:auth_path=/ibtljn-60/obj55 2023-11-02T18:50:14.646 DEBUG:boto:path=/ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.647 DEBUG:boto:auth_path=/ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.647 DEBUG:boto:Method: GET 2023-11-02T18:50:14.648 DEBUG:boto:Path: /ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.648 DEBUG:boto:Data: 2023-11-02T18:50:14.649 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.649 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.650 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.650 DEBUG:boto:Params: {} 2023-11-02T18:50:14.651 DEBUG:boto:Token: None 2023-11-02T18:50:14.651 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj55?versionId=null 2023-11-02T18:50:14.652 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sqa2b4gGxndvrtx+TDZ24rcFE1w= 2023-11-02T18:50:14.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sqa2b4gGxndvrtx+TDZ24rcFE1w=', 'Content-Length': '0'} 2023-11-02T18:50:14.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be5265a72de15fb-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.656 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:14.656 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:14.657 DEBUG:boto:path=/ibtljn-60/?prefix=obj55 2023-11-02T18:50:14.657 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj55 2023-11-02T18:50:14.658 DEBUG:boto:Method: GET 2023-11-02T18:50:14.659 DEBUG:boto:Path: /ibtljn-60/?prefix=obj55 2023-11-02T18:50:14.659 DEBUG:boto:Data: 2023-11-02T18:50:14.660 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.661 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.662 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.663 DEBUG:boto:Params: {} 2023-11-02T18:50:14.663 DEBUG:boto:Token: None 2023-11-02T18:50:14.664 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/ 2023-11-02T18:50:14.665 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc= 2023-11-02T18:50:14.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc=', 'Content-Length': '0'} 2023-11-02T18:50:14.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e696b153c4762bf-006543ef66-1eea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.714 DEBUG:boto:b'ibtljn-60obj551000falseobj552023-11-02T18:47:31.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:14.715 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:14.716 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:14.716 DEBUG:boto:path=/ibtljn-60/?prefix=obj55 2023-11-02T18:50:14.717 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj55 2023-11-02T18:50:14.717 DEBUG:boto:Method: GET 2023-11-02T18:50:14.718 DEBUG:boto:Path: /ibtljn-60/?prefix=obj55 2023-11-02T18:50:14.718 DEBUG:boto:Data: 2023-11-02T18:50:14.719 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.719 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.719 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.720 DEBUG:boto:Params: {} 2023-11-02T18:50:14.720 DEBUG:boto:Token: None 2023-11-02T18:50:14.721 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/ 2023-11-02T18:50:14.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc= 2023-11-02T18:50:14.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc=', 'Content-Length': '0'} 2023-11-02T18:50:14.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f1e7f1ef9dffa4e-006543ef66-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.728 DEBUG:boto:b'ibtljn-60obj551000falseobj552023-11-02T18:47:31.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:14.729 DEBUG:boto:path=/ibtljn-60/obj55 2023-11-02T18:50:14.729 DEBUG:boto:auth_path=/ibtljn-60/obj55 2023-11-02T18:50:14.730 DEBUG:boto:Method: HEAD 2023-11-02T18:50:14.730 DEBUG:boto:Path: /ibtljn-60/obj55 2023-11-02T18:50:14.730 DEBUG:boto:Data: 2023-11-02T18:50:14.731 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.731 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.732 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.732 DEBUG:boto:Params: {} 2023-11-02T18:50:14.733 DEBUG:boto:Token: None 2023-11-02T18:50:14.733 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj55 2023-11-02T18:50:14.734 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WATd+3jTHRMuhtPekyhtJ1IJgy0= 2023-11-02T18:50:14.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WATd+3jTHRMuhtPekyhtJ1IJgy0=', 'Content-Length': '0'} 2023-11-02T18:50:14.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dceddd95152900d2-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.738 DEBUG:boto:path=/ibtljn-60/obj55 2023-11-02T18:50:14.738 DEBUG:boto:auth_path=/ibtljn-60/obj55 2023-11-02T18:50:14.739 DEBUG:boto:Method: HEAD 2023-11-02T18:50:14.740 DEBUG:boto:Path: /ibtljn-60/obj55 2023-11-02T18:50:14.740 DEBUG:boto:Data: 2023-11-02T18:50:14.741 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.742 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.743 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.743 DEBUG:boto:Params: {} 2023-11-02T18:50:14.743 DEBUG:boto:Token: None 2023-11-02T18:50:14.746 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj55 2023-11-02T18:50:14.746 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WATd+3jTHRMuhtPekyhtJ1IJgy0= 2023-11-02T18:50:14.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WATd+3jTHRMuhtPekyhtJ1IJgy0=', 'Content-Length': '0'} 2023-11-02T18:50:14.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de60c3f74c43c0a2-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:50:14.751 DEBUG:boto:path=/ibtljn-60/obj55 2023-11-02T18:50:14.751 DEBUG:boto:auth_path=/ibtljn-60/obj55 2023-11-02T18:50:14.752 DEBUG:boto:Method: GET 2023-11-02T18:50:14.752 DEBUG:boto:Path: /ibtljn-60/obj55 2023-11-02T18:50:14.753 DEBUG:boto:Data: 2023-11-02T18:50:14.753 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.754 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.755 DEBUG:boto:Params: {} 2023-11-02T18:50:14.755 DEBUG:boto:Token: None 2023-11-02T18:50:14.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj55 2023-11-02T18:50:14.756 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FyCyhdRmUw7scZKp2QHsbCGYsQY= 2023-11-02T18:50:14.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FyCyhdRmUw7scZKp2QHsbCGYsQY=', 'Content-Length': '0'} 2023-11-02T18:50:14.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-60:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af0421db34879b03-006543ef66-1eea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.760 DEBUG:boto:path=/ibtljn-60/obj55 2023-11-02T18:50:14.761 DEBUG:boto:auth_path=/ibtljn-60/obj55 2023-11-02T18:50:14.761 DEBUG:boto:Method: GET 2023-11-02T18:50:14.762 DEBUG:boto:Path: /ibtljn-60/obj55 2023-11-02T18:50:14.762 DEBUG:boto:Data: 2023-11-02T18:50:14.763 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.763 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.764 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.764 DEBUG:boto:Params: {} 2023-11-02T18:50:14.765 DEBUG:boto:Token: None 2023-11-02T18:50:14.765 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/obj55 2023-11-02T18:50:14.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FyCyhdRmUw7scZKp2QHsbCGYsQY= 2023-11-02T18:50:14.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FyCyhdRmUw7scZKp2QHsbCGYsQY=', 'Content-Length': '0'} 2023-11-02T18:50:14.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:47:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b43ae1343a612d1-006543ef66-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.810 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:14.811 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:14.811 DEBUG:boto:path=/ibtljn-60/?prefix=obj55 2023-11-02T18:50:14.812 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj55 2023-11-02T18:50:14.813 DEBUG:boto:Method: GET 2023-11-02T18:50:14.813 DEBUG:boto:Path: /ibtljn-60/?prefix=obj55 2023-11-02T18:50:14.814 DEBUG:boto:Data: 2023-11-02T18:50:14.814 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.815 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:14.815 DEBUG:boto:Port: 8001 2023-11-02T18:50:14.816 DEBUG:boto:Params: {} 2023-11-02T18:50:14.816 DEBUG:boto:Token: None 2023-11-02T18:50:14.817 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/ 2023-11-02T18:50:14.817 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc= 2023-11-02T18:50:14.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc=', 'Content-Length': '0'} 2023-11-02T18:50:14.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeae13361229f691-006543ef66-1eea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.824 DEBUG:boto:b'ibtljn-60obj551000falseobj552023-11-02T18:47:31.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:14.825 DEBUG:boto:path=/ibtljn-60/ 2023-11-02T18:50:14.826 DEBUG:boto:auth_path=/ibtljn-60/ 2023-11-02T18:50:14.826 DEBUG:boto:path=/ibtljn-60/?prefix=obj55 2023-11-02T18:50:14.826 DEBUG:boto:auth_path=/ibtljn-60/?prefix=obj55 2023-11-02T18:50:14.827 DEBUG:boto:Method: GET 2023-11-02T18:50:14.827 DEBUG:boto:Path: /ibtljn-60/?prefix=obj55 2023-11-02T18:50:14.828 DEBUG:boto:Data: 2023-11-02T18:50:14.828 DEBUG:boto:Headers: {} 2023-11-02T18:50:14.829 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:14.829 DEBUG:boto:Port: 8000 2023-11-02T18:50:14.829 DEBUG:boto:Params: {} 2023-11-02T18:50:14.830 DEBUG:boto:Token: None 2023-11-02T18:50:14.830 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:50:14 GMT /ibtljn-60/ 2023-11-02T18:50:14.831 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc= 2023-11-02T18:50:14.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:14 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RDw8qkon5mDW/FTBdxNSq6jiCTc=', 'Content-Length': '0'} 2023-11-02T18:50:14.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c3ab33df2b25197-006543ef66-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:50:14 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:14.837 DEBUG:boto:b'ibtljn-60obj551000falseobj552023-11-02T18:47:31.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:50:14.838 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-60 zones={a2, a1} 2023-11-02T18:50:14.838 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-11-02T18:50:14.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:50:14.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:50:14.840 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:50:14.885 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:14.885 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:14.991 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:50:14.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:14.992 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000018893", 2023-11-02T18:50:14.992 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:44:13.588442Z" 2023-11-02T18:50:14.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:14.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:14.994 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000030455", 2023-11-02T18:50:14.995 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:47:31.670443Z" 2023-11-02T18:50:14.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:14.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:14.995 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000027209", 2023-11-02T18:50:14.995 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:47:41.383045Z" 2023-11-02T18:50:14.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:14.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:14.996 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000025974", 2023-11-02T18:50:14.996 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:47:41.386293Z" 2023-11-02T18:50:14.997 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:14.997 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:50:14.998 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000018893', 1: '00000000000000000000:00000000000000030455', 2: '00000000000000000000:00000000000000027209', 3: '00000000000000000000:00000000000000025974'} 2023-11-02T18:50:14.999 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:50:14.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:50:14.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:50:15.000 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:50:15.088 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:15.088 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:15.179 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:50:15.179 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:50:15.179 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:50:15.180 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:50:15.180 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:50:15.180 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 4196603556495824236 2023-11-02T18:50:15.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.181 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:50:15.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.181 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:50:15.181 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:50:15.181 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:50:15.182 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000018893", 2023-11-02T18:50:15.182 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:50:15.182 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T18:50:15.182 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:50:15.183 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:44:13.588442Z" 2023-11-02T18:50:15.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.183 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:50:15.184 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:50:15.184 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:50:15.184 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000030455", 2023-11-02T18:50:15.184 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:50:15.184 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 51, 2023-11-02T18:50:15.185 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:50:15.185 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:47:31.670443Z" 2023-11-02T18:50:15.185 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.186 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:50:15.186 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:50:15.186 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:50:15.186 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000027209", 2023-11-02T18:50:15.187 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:50:15.187 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:50:15.187 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:50:15.187 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:47:41.383045Z" 2023-11-02T18:50:15.187 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.188 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:50:15.188 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:50:15.188 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:50:15.189 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000025974", 2023-11-02T18:50:15.189 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:50:15.189 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:50:15.189 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:50:15.190 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:47:41.386293Z" 2023-11-02T18:50:15.190 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.190 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.190 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:50:15.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.191 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:50:15.191 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 60, 2023-11-02T18:50:15.191 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 60 2023-11-02T18:50:15.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.192 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:50:15Z" 2023-11-02T18:50:15.192 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:50:15.193 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 4196603556495824236 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018893", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T18:44:13.588442Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000030455", "next_step_marker": "", "total_entries": 51, "pos": 0, "timestamp": "2023-11-02T18:47:31.670443Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027209", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:47:41.383045Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025974", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:47:41.386293Z" } } ] }, "full_sync": { "total": 60, "complete": 60 }, "current_time": "2023-11-02T18:50:15Z" } 2023-11-02T18:50:15.194 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018893', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T18:44:13.588442Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000030455', 'next_step_marker': '', 'total_entries': 51, 'pos': 0, 'timestamp': '2023-11-02T18:47:31.670443Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027209', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:47:41.383045Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025974', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:47:41.386293Z'}}] 2023-11-02T18:50:15.194 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000018893', 1: '00000000000000000000:00000000000000030455', 2: '00000000000000000000:00000000000000027209', 3: '00000000000000000000:00000000000000025974'} 2023-11-02T18:50:15.194 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000018893', 1: '00000000000000000000:00000000000000030455', 2: '00000000000000000000:00000000000000027209', 3: '00000000000000000000:00000000000000025974'} 2023-11-02T18:50:15.195 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:50:15.195 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-11-02T18:50:15.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:50:15.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:50:15.197 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:50:15.242 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:15.242 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:15.347 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:50:15.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.348 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000005300", 2023-11-02T18:50:15.348 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:45:50.047006Z" 2023-11-02T18:50:15.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.349 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000014319", 2023-11-02T18:50:15.349 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:47:52.859584Z" 2023-11-02T18:50:15.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.350 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000012916", 2023-11-02T18:50:15.350 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:47:54.025029Z" 2023-11-02T18:50:15.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.350 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000010546", 2023-11-02T18:50:15.351 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:48:52.810247Z" 2023-11-02T18:50:15.351 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.351 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:50:15.352 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005300', 1: '00000000000000000000:00000000000000014319', 2: '00000000000000000000:00000000000000012916', 3: '00000000000000000000:00000000000000010546'} 2023-11-02T18:50:15.353 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:50:15.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:50:15.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:50:15.354 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:50:15.440 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:15.441 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:15.528 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:50:15.529 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:50:15.529 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:50:15.529 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:50:15.529 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:50:15.530 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:50:15.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.530 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:50:15.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.530 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:50:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:50:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:50:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000005300", 2023-11-02T18:50:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:50:15.532 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:50:15.532 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:50:15.532 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:45:50.047006Z" 2023-11-02T18:50:15.532 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.533 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:50:15.533 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:50:15.533 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:50:15.534 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000014319", 2023-11-02T18:50:15.534 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:50:15.534 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:50:15.534 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:50:15.535 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:47:52.859584Z" 2023-11-02T18:50:15.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.536 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:50:15.536 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:50:15.536 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:50:15.536 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000012916", 2023-11-02T18:50:15.537 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:50:15.537 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:50:15.537 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:50:15.537 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:47:54.025029Z" 2023-11-02T18:50:15.537 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.538 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:50:15.538 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:50:15.539 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:50:15.539 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000010546", 2023-11-02T18:50:15.539 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:50:15.539 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:50:15.539 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:50:15.540 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:48:52.810247Z" 2023-11-02T18:50:15.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.541 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:50:15.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.542 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:50:15.542 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:50:15.542 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:50:15.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.543 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:50:15Z" 2023-11-02T18:50:15.543 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:50:15.544 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005300", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:45:50.047006Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:47:52.859584Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012916", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:47:54.025029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010546", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:48:52.810247Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:50:15Z" } 2023-11-02T18:50:15.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005300', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:45:50.047006Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:47:52.859584Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012916', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:47:54.025029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010546', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:48:52.810247Z'}}] 2023-11-02T18:50:15.545 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005300', 1: '00000000000000000000:00000000000000014319', 2: '00000000000000000000:00000000000000012916', 3: '00000000000000000000:00000000000000010546'} 2023-11-02T18:50:15.546 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005300', 1: '00000000000000000000:00000000000000014319', 2: '00000000000000000000:00000000000000012916', 3: '00000000000000000000:00000000000000010546'} 2023-11-02T18:50:15.546 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:50:15.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2023-11-02T18:50:15.548 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:15.549 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:15.549 DEBUG:boto:Method: PUT 2023-11-02T18:50:15.549 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:15.550 DEBUG:boto:Data: 2023-11-02T18:50:15.550 DEBUG:boto:Headers: {} 2023-11-02T18:50:15.551 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:15.551 DEBUG:boto:Port: 8000 2023-11-02T18:50:15.552 DEBUG:boto:Params: {} 2023-11-02T18:50:15.552 DEBUG:boto:Token: None 2023-11-02T18:50:15.553 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:50:15 GMT /ibtljn-61/ 2023-11-02T18:50:15.553 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qFBN4f9iqrhQCjofVJGbDKizWlo= 2023-11-02T18:50:15.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:15 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qFBN4f9iqrhQCjofVJGbDKizWlo=', 'Content-Length': '0'} 2023-11-02T18:50:15.586 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004d32bef2c8629f60-006543ef67-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:15 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:15.587 DEBUG:rgw_multi.tests:created bucket=ibtljn-61 2023-11-02T18:50:15.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:50:15.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:50:15.589 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:50:15.632 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:15.632 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:15.689 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:50:15.718 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:50:15.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.718 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:50:15.719 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:50:15.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.719 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:50:15.720 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T18:50:15.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.720 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951015.580599_534.1", 2023-11-02T18:50:15.721 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:50:15.580599Z" 2023-11-02T18:50:15.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.721 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:50:15.722 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T18:50:15.722 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.722 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:50:15.723 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951015.580599_534.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:50:15.724 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:50:15.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:50:15.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:50:15.725 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:50:15.775 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:15.775 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:15.828 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:15.822+0000 7fa983972f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:50:15.829 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:15.822+0000 7fa983972f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:50:15.857 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:50:15.857 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:50:15.858 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:50:15.858 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:50:15.858 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:50:15.858 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:50:15.859 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:50:15.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.859 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:50:15.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.860 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:50:15.860 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:50:15.860 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:50:15.860 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:50:15.860 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:50:15.861 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:50:15.861 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:50:15.861 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:50:15.861 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:50:15.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.862 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:50:15.862 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:50:15.863 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:50:15.863 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:50:15.863 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:50:15.863 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:50:15.864 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:50:15.864 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T18:50:15.864 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:50:15.864 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.865 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:50:15.865 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:50:15.865 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:50:15.866 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951015.580599_534.1", 2023-11-02T18:50:15.866 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:50:15.866 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:50:15.866 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:50:15.867 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:50:15.580599Z", 2023-11-02T18:50:15.867 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:50:15.867 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:50:15.868 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:50:15.868 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:50:15.868 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:50:15.869 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:50:15.869 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:50:15.869 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:50:15.869 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:50:15.870 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T18:50:15.870 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:50:15.870 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.870 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:50:15.870 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:50:15.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.871 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:50:15.871 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:50:15.872 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:50:15.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:50:15.872 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:50:15Z" 2023-11-02T18:50:15.872 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:50:15.874 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951015.580599_534.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:50:15.580599Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:50:15Z" } 2023-11-02T18:50:15.874 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951015.580599_534.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:50:15.580599Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T18:50:15.875 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951015.580599_534.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:50:15.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951015.580599_534.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:50:15.876 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:50:15.876 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-11-02T18:50:15.877 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T18:50:15.878 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:50:15.870+0000 7f10324c5700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 76526) UID: 0 2023-11-02T18:50:15.878 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:50:15.870+0000 7f1035d24f80 -1 shutting down 2023-11-02T18:50:16.345 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-11-02T18:50:21.950 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2023-11-02T18:50:21.978 INFO:tasks.rgw.c1.client.1:Stopped 2023-11-02T18:50:21.979 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:21.980 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:21.980 DEBUG:boto:Method: HEAD 2023-11-02T18:50:21.981 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:21.981 DEBUG:boto:Data: 2023-11-02T18:50:21.981 DEBUG:boto:Headers: {} 2023-11-02T18:50:21.982 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:21.982 DEBUG:boto:Port: 8000 2023-11-02T18:50:21.983 DEBUG:boto:Params: {} 2023-11-02T18:50:21.983 DEBUG:boto:Token: None 2023-11-02T18:50:21.984 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:21 GMT /ibtljn-61/ 2023-11-02T18:50:21.984 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CnBhbAFXh/pZpXyBGr7JV3GQRLQ= 2023-11-02T18:50:21.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CnBhbAFXh/pZpXyBGr7JV3GQRLQ=', 'Content-Length': '0'} 2023-11-02T18:50:21.990 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a660727f727b0f34-006543ef6d-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:21.991 DEBUG:boto:path=/ibtljn-61/obj11 2023-11-02T18:50:21.991 DEBUG:boto:auth_path=/ibtljn-61/obj11 2023-11-02T18:50:21.991 DEBUG:boto:Method: PUT 2023-11-02T18:50:21.992 DEBUG:boto:Path: /ibtljn-61/obj11 2023-11-02T18:50:21.992 DEBUG:boto:Data: 2023-11-02T18:50:21.993 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:21.993 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:21.993 DEBUG:boto:Port: 8000 2023-11-02T18:50:21.994 DEBUG:boto:Params: {} 2023-11-02T18:50:21.994 DEBUG:boto:Token: None 2023-11-02T18:50:21.995 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:21 GMT /ibtljn-61/obj11 2023-11-02T18:50:21.995 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1JHmlfMnAQyz7aqqWVtVKI1aXbU= 2023-11-02T18:50:21.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1JHmlfMnAQyz7aqqWVtVKI1aXbU='} 2023-11-02T18:50:22.009 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfa224107d5643bc-006543ef6d-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.009 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:22.010 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:22.010 DEBUG:boto:Method: HEAD 2023-11-02T18:50:22.011 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:22.011 DEBUG:boto:Data: 2023-11-02T18:50:22.012 DEBUG:boto:Headers: {} 2023-11-02T18:50:22.012 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.013 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.013 DEBUG:boto:Params: {} 2023-11-02T18:50:22.013 DEBUG:boto:Token: None 2023-11-02T18:50:22.014 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/ 2023-11-02T18:50:22.014 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I= 2023-11-02T18:50:22.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I=', 'Content-Length': '0'} 2023-11-02T18:50:22.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007511e97101a40dfd-006543ef6e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.019 DEBUG:boto:path=/ibtljn-61/obj22 2023-11-02T18:50:22.020 DEBUG:boto:auth_path=/ibtljn-61/obj22 2023-11-02T18:50:22.020 DEBUG:boto:Method: PUT 2023-11-02T18:50:22.021 DEBUG:boto:Path: /ibtljn-61/obj22 2023-11-02T18:50:22.021 DEBUG:boto:Data: 2023-11-02T18:50:22.022 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:22.022 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.023 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.023 DEBUG:boto:Params: {} 2023-11-02T18:50:22.023 DEBUG:boto:Token: None 2023-11-02T18:50:22.024 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/obj22 2023-11-02T18:50:22.024 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qNlBj0zyaRhNxwIDFzKvns7sQ5Y= 2023-11-02T18:50:22.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qNlBj0zyaRhNxwIDFzKvns7sQ5Y='} 2023-11-02T18:50:22.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000815e4f07a7b5fee8-006543ef6e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.071 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:22.072 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:22.072 DEBUG:boto:Method: HEAD 2023-11-02T18:50:22.073 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:22.073 DEBUG:boto:Data: 2023-11-02T18:50:22.074 DEBUG:boto:Headers: {} 2023-11-02T18:50:22.074 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.075 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.075 DEBUG:boto:Params: {} 2023-11-02T18:50:22.076 DEBUG:boto:Token: None 2023-11-02T18:50:22.076 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/ 2023-11-02T18:50:22.077 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I= 2023-11-02T18:50:22.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I=', 'Content-Length': '0'} 2023-11-02T18:50:22.082 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070aa9db5e007746b-006543ef6e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.083 DEBUG:boto:path=/ibtljn-61/obj33 2023-11-02T18:50:22.083 DEBUG:boto:auth_path=/ibtljn-61/obj33 2023-11-02T18:50:22.084 DEBUG:boto:Method: PUT 2023-11-02T18:50:22.084 DEBUG:boto:Path: /ibtljn-61/obj33 2023-11-02T18:50:22.085 DEBUG:boto:Data: 2023-11-02T18:50:22.085 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:22.086 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.086 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.087 DEBUG:boto:Params: {} 2023-11-02T18:50:22.087 DEBUG:boto:Token: None 2023-11-02T18:50:22.088 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/obj33 2023-11-02T18:50:22.088 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RErbS8H8qEiQXHyzzBCG35Oe3Qw= 2023-11-02T18:50:22.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RErbS8H8qEiQXHyzzBCG35Oe3Qw='} 2023-11-02T18:50:22.101 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e9cef5d87da5a2a-006543ef6e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.102 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:22.102 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:22.103 DEBUG:boto:Method: HEAD 2023-11-02T18:50:22.103 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:22.103 DEBUG:boto:Data: 2023-11-02T18:50:22.104 DEBUG:boto:Headers: {} 2023-11-02T18:50:22.104 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.105 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.105 DEBUG:boto:Params: {} 2023-11-02T18:50:22.106 DEBUG:boto:Token: None 2023-11-02T18:50:22.106 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/ 2023-11-02T18:50:22.107 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I= 2023-11-02T18:50:22.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I=', 'Content-Length': '0'} 2023-11-02T18:50:22.112 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f5a6f2e770aa967-006543ef6e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.112 DEBUG:boto:path=/ibtljn-61/obj44 2023-11-02T18:50:22.113 DEBUG:boto:auth_path=/ibtljn-61/obj44 2023-11-02T18:50:22.113 DEBUG:boto:Method: PUT 2023-11-02T18:50:22.114 DEBUG:boto:Path: /ibtljn-61/obj44 2023-11-02T18:50:22.114 DEBUG:boto:Data: 2023-11-02T18:50:22.115 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:22.115 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.116 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.116 DEBUG:boto:Params: {} 2023-11-02T18:50:22.117 DEBUG:boto:Token: None 2023-11-02T18:50:22.117 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/obj44 2023-11-02T18:50:22.118 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UQSWJ9QAZ0JEGii0NtPQvik0wV4= 2023-11-02T18:50:22.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UQSWJ9QAZ0JEGii0NtPQvik0wV4='} 2023-11-02T18:50:22.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056d0d86bf71f32dd-006543ef6e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.167 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:22.168 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:22.168 DEBUG:boto:Method: HEAD 2023-11-02T18:50:22.168 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:22.169 DEBUG:boto:Data: 2023-11-02T18:50:22.169 DEBUG:boto:Headers: {} 2023-11-02T18:50:22.170 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.170 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.171 DEBUG:boto:Params: {} 2023-11-02T18:50:22.171 DEBUG:boto:Token: None 2023-11-02T18:50:22.171 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/ 2023-11-02T18:50:22.172 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I= 2023-11-02T18:50:22.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I=', 'Content-Length': '0'} 2023-11-02T18:50:22.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040bc5767972c836a-006543ef6e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.178 DEBUG:boto:path=/ibtljn-61/obj55 2023-11-02T18:50:22.179 DEBUG:boto:auth_path=/ibtljn-61/obj55 2023-11-02T18:50:22.180 DEBUG:boto:Method: PUT 2023-11-02T18:50:22.180 DEBUG:boto:Path: /ibtljn-61/obj55 2023-11-02T18:50:22.181 DEBUG:boto:Data: 2023-11-02T18:50:22.182 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:22.183 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.183 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.184 DEBUG:boto:Params: {} 2023-11-02T18:50:22.185 DEBUG:boto:Token: None 2023-11-02T18:50:22.186 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/obj55 2023-11-02T18:50:22.186 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RQwIlXqIow1kqqYaAERdBAkE9RY= 2023-11-02T18:50:22.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RQwIlXqIow1kqqYaAERdBAkE9RY='} 2023-11-02T18:50:22.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075ee0b41e693c849-006543ef6e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:22.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:22.197 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:50:22.247 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:22.247 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:22.328 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:50:22.328 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-61 2023-11-02T18:50:22.337 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 5 2023-11-02T18:50:22.417 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:22.418 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:22.418 DEBUG:boto:Method: HEAD 2023-11-02T18:50:22.419 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:22.419 DEBUG:boto:Data: 2023-11-02T18:50:22.419 DEBUG:boto:Headers: {} 2023-11-02T18:50:22.420 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.420 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.421 DEBUG:boto:Params: {} 2023-11-02T18:50:22.421 DEBUG:boto:Token: None 2023-11-02T18:50:22.422 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/ 2023-11-02T18:50:22.422 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I= 2023-11-02T18:50:22.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I=', 'Content-Length': '0'} 2023-11-02T18:50:22.428 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000aa044709559a691-006543ef6e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.429 DEBUG:boto:path=/ibtljn-61/obj17 2023-11-02T18:50:22.429 DEBUG:boto:auth_path=/ibtljn-61/obj17 2023-11-02T18:50:22.430 DEBUG:boto:Method: PUT 2023-11-02T18:50:22.430 DEBUG:boto:Path: /ibtljn-61/obj17 2023-11-02T18:50:22.430 DEBUG:boto:Data: 2023-11-02T18:50:22.431 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:22.431 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.432 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.432 DEBUG:boto:Params: {} 2023-11-02T18:50:22.433 DEBUG:boto:Token: None 2023-11-02T18:50:22.433 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/obj17 2023-11-02T18:50:22.433 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kt2PLIPDZuZ8N/Mb5hBiDmw/DGA= 2023-11-02T18:50:22.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kt2PLIPDZuZ8N/Mb5hBiDmw/DGA='} 2023-11-02T18:50:22.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b30a904828e551a6-006543ef6e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.479 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:22.480 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:22.480 DEBUG:boto:Method: HEAD 2023-11-02T18:50:22.480 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:22.481 DEBUG:boto:Data: 2023-11-02T18:50:22.481 DEBUG:boto:Headers: {} 2023-11-02T18:50:22.482 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.482 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.483 DEBUG:boto:Params: {} 2023-11-02T18:50:22.483 DEBUG:boto:Token: None 2023-11-02T18:50:22.484 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/ 2023-11-02T18:50:22.484 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I= 2023-11-02T18:50:22.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I=', 'Content-Length': '0'} 2023-11-02T18:50:22.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3734b4a1ede0ac5-006543ef6e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.490 DEBUG:boto:path=/ibtljn-61/obj34 2023-11-02T18:50:22.491 DEBUG:boto:auth_path=/ibtljn-61/obj34 2023-11-02T18:50:22.491 DEBUG:boto:Method: PUT 2023-11-02T18:50:22.492 DEBUG:boto:Path: /ibtljn-61/obj34 2023-11-02T18:50:22.492 DEBUG:boto:Data: 2023-11-02T18:50:22.493 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:22.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.493 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.494 DEBUG:boto:Params: {} 2023-11-02T18:50:22.494 DEBUG:boto:Token: None 2023-11-02T18:50:22.495 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/obj34 2023-11-02T18:50:22.495 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x4MwcP3Oyjoq96j5IxOZa/Hl3uE= 2023-11-02T18:50:22.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x4MwcP3Oyjoq96j5IxOZa/Hl3uE='} 2023-11-02T18:50:22.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008275a16ce79d8038-006543ef6e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.508 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:22.509 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:22.509 DEBUG:boto:Method: HEAD 2023-11-02T18:50:22.509 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:22.510 DEBUG:boto:Data: 2023-11-02T18:50:22.510 DEBUG:boto:Headers: {} 2023-11-02T18:50:22.511 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.511 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.512 DEBUG:boto:Params: {} 2023-11-02T18:50:22.512 DEBUG:boto:Token: None 2023-11-02T18:50:22.512 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/ 2023-11-02T18:50:22.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I= 2023-11-02T18:50:22.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I=', 'Content-Length': '0'} 2023-11-02T18:50:22.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa6c3e9fa758749a-006543ef6e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.519 DEBUG:boto:path=/ibtljn-61/obj51 2023-11-02T18:50:22.519 DEBUG:boto:auth_path=/ibtljn-61/obj51 2023-11-02T18:50:22.519 DEBUG:boto:Method: PUT 2023-11-02T18:50:22.520 DEBUG:boto:Path: /ibtljn-61/obj51 2023-11-02T18:50:22.520 DEBUG:boto:Data: 2023-11-02T18:50:22.521 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:22.521 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.522 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.522 DEBUG:boto:Params: {} 2023-11-02T18:50:22.523 DEBUG:boto:Token: None 2023-11-02T18:50:22.523 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/obj51 2023-11-02T18:50:22.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PP44XQxdgJNRQVQOBsZTZvc4oz0= 2023-11-02T18:50:22.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PP44XQxdgJNRQVQOBsZTZvc4oz0='} 2023-11-02T18:50:22.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004d05fff855bc4f1-006543ef6e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.571 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:22.572 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:22.572 DEBUG:boto:Method: HEAD 2023-11-02T18:50:22.572 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:22.573 DEBUG:boto:Data: 2023-11-02T18:50:22.573 DEBUG:boto:Headers: {} 2023-11-02T18:50:22.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.574 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.574 DEBUG:boto:Params: {} 2023-11-02T18:50:22.575 DEBUG:boto:Token: None 2023-11-02T18:50:22.575 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/ 2023-11-02T18:50:22.576 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I= 2023-11-02T18:50:22.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I=', 'Content-Length': '0'} 2023-11-02T18:50:22.581 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070d1a5b26465f910-006543ef6e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.582 DEBUG:boto:path=/ibtljn-61/obj68 2023-11-02T18:50:22.582 DEBUG:boto:auth_path=/ibtljn-61/obj68 2023-11-02T18:50:22.583 DEBUG:boto:Method: PUT 2023-11-02T18:50:22.583 DEBUG:boto:Path: /ibtljn-61/obj68 2023-11-02T18:50:22.584 DEBUG:boto:Data: 2023-11-02T18:50:22.584 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:22.585 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.585 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.585 DEBUG:boto:Params: {} 2023-11-02T18:50:22.586 DEBUG:boto:Token: None 2023-11-02T18:50:22.586 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/obj68 2023-11-02T18:50:22.587 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:r1LFDmfIJybtgnZ1iAVfDD1jhNE= 2023-11-02T18:50:22.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:r1LFDmfIJybtgnZ1iAVfDD1jhNE='} 2023-11-02T18:50:22.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f70f77ac54dd93b-006543ef6e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.600 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:22.600 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:22.601 DEBUG:boto:Method: HEAD 2023-11-02T18:50:22.601 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:22.602 DEBUG:boto:Data: 2023-11-02T18:50:22.602 DEBUG:boto:Headers: {} 2023-11-02T18:50:22.603 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.603 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.603 DEBUG:boto:Params: {} 2023-11-02T18:50:22.604 DEBUG:boto:Token: None 2023-11-02T18:50:22.604 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/ 2023-11-02T18:50:22.605 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I= 2023-11-02T18:50:22.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I=', 'Content-Length': '0'} 2023-11-02T18:50:22.610 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f62e2a2248eba420-006543ef6e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.611 DEBUG:boto:path=/ibtljn-61/obj85 2023-11-02T18:50:22.612 DEBUG:boto:auth_path=/ibtljn-61/obj85 2023-11-02T18:50:22.612 DEBUG:boto:Method: PUT 2023-11-02T18:50:22.612 DEBUG:boto:Path: /ibtljn-61/obj85 2023-11-02T18:50:22.613 DEBUG:boto:Data: 2023-11-02T18:50:22.613 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:22.614 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.614 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.614 DEBUG:boto:Params: {} 2023-11-02T18:50:22.615 DEBUG:boto:Token: None 2023-11-02T18:50:22.615 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/obj85 2023-11-02T18:50:22.616 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4hQLhtBLWLHDCAqEcInS9PsJtdM= 2023-11-02T18:50:22.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4hQLhtBLWLHDCAqEcInS9PsJtdM='} 2023-11-02T18:50:22.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005d6fa6f324b78b9-006543ef6e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:22.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:22.664 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:50:22.716 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:22.716 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:22.798 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:50:22.798 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-61 2023-11-02T18:50:22.812 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 10 2023-11-02T18:50:22.909 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:22.909 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:22.910 DEBUG:boto:Method: HEAD 2023-11-02T18:50:22.910 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:22.910 DEBUG:boto:Data: 2023-11-02T18:50:22.911 DEBUG:boto:Headers: {} 2023-11-02T18:50:22.911 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.912 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.912 DEBUG:boto:Params: {} 2023-11-02T18:50:22.913 DEBUG:boto:Token: None 2023-11-02T18:50:22.913 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/ 2023-11-02T18:50:22.914 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I= 2023-11-02T18:50:22.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I=', 'Content-Length': '0'} 2023-11-02T18:50:22.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063d164ca011c2f05-006543ef6e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.920 DEBUG:boto:path=/ibtljn-61/obj19 2023-11-02T18:50:22.920 DEBUG:boto:auth_path=/ibtljn-61/obj19 2023-11-02T18:50:22.921 DEBUG:boto:Method: PUT 2023-11-02T18:50:22.921 DEBUG:boto:Path: /ibtljn-61/obj19 2023-11-02T18:50:22.922 DEBUG:boto:Data: 2023-11-02T18:50:22.922 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:22.922 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.923 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.923 DEBUG:boto:Params: {} 2023-11-02T18:50:22.924 DEBUG:boto:Token: None 2023-11-02T18:50:22.924 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/obj19 2023-11-02T18:50:22.925 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i32ZumILyrvTm69qWQ9SZkr9Maw= 2023-11-02T18:50:22.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i32ZumILyrvTm69qWQ9SZkr9Maw='} 2023-11-02T18:50:22.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b6925979aa1b1a5-006543ef6e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.938 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:22.938 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:22.939 DEBUG:boto:Method: HEAD 2023-11-02T18:50:22.940 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:22.941 DEBUG:boto:Data: 2023-11-02T18:50:22.941 DEBUG:boto:Headers: {} 2023-11-02T18:50:22.942 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.942 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.942 DEBUG:boto:Params: {} 2023-11-02T18:50:22.943 DEBUG:boto:Token: None 2023-11-02T18:50:22.943 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/ 2023-11-02T18:50:22.944 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I= 2023-11-02T18:50:22.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+d6jBq1Ltl87tW4QLDRPVM/Hn0I=', 'Content-Length': '0'} 2023-11-02T18:50:22.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c9e069ee3130f07-006543ef6e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.949 DEBUG:boto:path=/ibtljn-61/obj38 2023-11-02T18:50:22.950 DEBUG:boto:auth_path=/ibtljn-61/obj38 2023-11-02T18:50:22.951 DEBUG:boto:Method: PUT 2023-11-02T18:50:22.952 DEBUG:boto:Path: /ibtljn-61/obj38 2023-11-02T18:50:22.952 DEBUG:boto:Data: 2023-11-02T18:50:22.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:22.953 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:22.953 DEBUG:boto:Port: 8000 2023-11-02T18:50:22.954 DEBUG:boto:Params: {} 2023-11-02T18:50:22.954 DEBUG:boto:Token: None 2023-11-02T18:50:22.955 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:22 GMT /ibtljn-61/obj38 2023-11-02T18:50:22.955 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6ClsWVVOEtVl+GECKLOR0BNqt9I= 2023-11-02T18:50:22.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6ClsWVVOEtVl+GECKLOR0BNqt9I='} 2023-11-02T18:50:22.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f1e714f12cade3d-006543ef6e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:22.999 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:23.000 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:23.000 DEBUG:boto:Method: HEAD 2023-11-02T18:50:23.000 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:23.001 DEBUG:boto:Data: 2023-11-02T18:50:23.001 DEBUG:boto:Headers: {} 2023-11-02T18:50:23.002 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.002 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.003 DEBUG:boto:Params: {} 2023-11-02T18:50:23.003 DEBUG:boto:Token: None 2023-11-02T18:50:23.004 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/ 2023-11-02T18:50:23.004 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM= 2023-11-02T18:50:23.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM=', 'Content-Length': '0'} 2023-11-02T18:50:23.010 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006af8603c050c3b4f-006543ef6f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.011 DEBUG:boto:path=/ibtljn-61/obj57 2023-11-02T18:50:23.011 DEBUG:boto:auth_path=/ibtljn-61/obj57 2023-11-02T18:50:23.012 DEBUG:boto:Method: PUT 2023-11-02T18:50:23.012 DEBUG:boto:Path: /ibtljn-61/obj57 2023-11-02T18:50:23.013 DEBUG:boto:Data: 2023-11-02T18:50:23.014 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:23.014 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.014 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.015 DEBUG:boto:Params: {} 2023-11-02T18:50:23.015 DEBUG:boto:Token: None 2023-11-02T18:50:23.016 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/obj57 2023-11-02T18:50:23.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fvblJWsLr6ompC2NUuwnCO9OErU= 2023-11-02T18:50:23.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fvblJWsLr6ompC2NUuwnCO9OErU='} 2023-11-02T18:50:23.027 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019abcdc355f4e0fe-006543ef6f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.028 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:23.028 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:23.029 DEBUG:boto:Method: HEAD 2023-11-02T18:50:23.029 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:23.030 DEBUG:boto:Data: 2023-11-02T18:50:23.030 DEBUG:boto:Headers: {} 2023-11-02T18:50:23.031 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.031 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.032 DEBUG:boto:Params: {} 2023-11-02T18:50:23.032 DEBUG:boto:Token: None 2023-11-02T18:50:23.033 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/ 2023-11-02T18:50:23.033 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM= 2023-11-02T18:50:23.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM=', 'Content-Length': '0'} 2023-11-02T18:50:23.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070273ada7e768df1-006543ef6f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.039 DEBUG:boto:path=/ibtljn-61/obj76 2023-11-02T18:50:23.040 DEBUG:boto:auth_path=/ibtljn-61/obj76 2023-11-02T18:50:23.040 DEBUG:boto:Method: PUT 2023-11-02T18:50:23.041 DEBUG:boto:Path: /ibtljn-61/obj76 2023-11-02T18:50:23.041 DEBUG:boto:Data: 2023-11-02T18:50:23.041 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:23.042 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.042 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.043 DEBUG:boto:Params: {} 2023-11-02T18:50:23.043 DEBUG:boto:Token: None 2023-11-02T18:50:23.044 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/obj76 2023-11-02T18:50:23.044 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rfEjhpqbBZ9Ssgg3Wgtl0R3JO4I= 2023-11-02T18:50:23.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rfEjhpqbBZ9Ssgg3Wgtl0R3JO4I='} 2023-11-02T18:50:23.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000135605781baa522b-006543ef6f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.091 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:23.092 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:23.092 DEBUG:boto:Method: HEAD 2023-11-02T18:50:23.093 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:23.093 DEBUG:boto:Data: 2023-11-02T18:50:23.093 DEBUG:boto:Headers: {} 2023-11-02T18:50:23.094 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.094 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.095 DEBUG:boto:Params: {} 2023-11-02T18:50:23.095 DEBUG:boto:Token: None 2023-11-02T18:50:23.096 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/ 2023-11-02T18:50:23.096 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM= 2023-11-02T18:50:23.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM=', 'Content-Length': '0'} 2023-11-02T18:50:23.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7fce50fd854d4a9-006543ef6f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.102 DEBUG:boto:path=/ibtljn-61/obj95 2023-11-02T18:50:23.103 DEBUG:boto:auth_path=/ibtljn-61/obj95 2023-11-02T18:50:23.103 DEBUG:boto:Method: PUT 2023-11-02T18:50:23.104 DEBUG:boto:Path: /ibtljn-61/obj95 2023-11-02T18:50:23.104 DEBUG:boto:Data: 2023-11-02T18:50:23.105 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:23.105 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.105 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.106 DEBUG:boto:Params: {} 2023-11-02T18:50:23.106 DEBUG:boto:Token: None 2023-11-02T18:50:23.107 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/obj95 2023-11-02T18:50:23.107 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Q40I4Z57ZuZn//XmReUTcp8EMHQ= 2023-11-02T18:50:23.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Q40I4Z57ZuZn//XmReUTcp8EMHQ='} 2023-11-02T18:50:23.117 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072f4498e3c42ecd9-006543ef6f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:23.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:23.119 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:50:23.167 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:23.167 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:23.252 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:50:23.252 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-61 2023-11-02T18:50:23.266 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 15 2023-11-02T18:50:23.370 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:23.371 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:23.371 DEBUG:boto:Method: HEAD 2023-11-02T18:50:23.372 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:23.373 DEBUG:boto:Data: 2023-11-02T18:50:23.373 DEBUG:boto:Headers: {} 2023-11-02T18:50:23.374 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.374 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.375 DEBUG:boto:Params: {} 2023-11-02T18:50:23.375 DEBUG:boto:Token: None 2023-11-02T18:50:23.376 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/ 2023-11-02T18:50:23.376 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM= 2023-11-02T18:50:23.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM=', 'Content-Length': '0'} 2023-11-02T18:50:23.383 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050697b97b8bd995e-006543ef6f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.384 DEBUG:boto:path=/ibtljn-61/obj23 2023-11-02T18:50:23.384 DEBUG:boto:auth_path=/ibtljn-61/obj23 2023-11-02T18:50:23.385 DEBUG:boto:Method: PUT 2023-11-02T18:50:23.385 DEBUG:boto:Path: /ibtljn-61/obj23 2023-11-02T18:50:23.385 DEBUG:boto:Data: 2023-11-02T18:50:23.386 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:23.386 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.387 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.387 DEBUG:boto:Params: {} 2023-11-02T18:50:23.388 DEBUG:boto:Token: None 2023-11-02T18:50:23.389 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/obj23 2023-11-02T18:50:23.389 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7jTOXuM33sv65mh2GfWx9X9vCqE= 2023-11-02T18:50:23.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7jTOXuM33sv65mh2GfWx9X9vCqE='} 2023-11-02T18:50:23.435 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c53911c536be338-006543ef6f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.435 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:23.436 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:23.436 DEBUG:boto:Method: HEAD 2023-11-02T18:50:23.436 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:23.437 DEBUG:boto:Data: 2023-11-02T18:50:23.437 DEBUG:boto:Headers: {} 2023-11-02T18:50:23.438 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.438 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.438 DEBUG:boto:Params: {} 2023-11-02T18:50:23.439 DEBUG:boto:Token: None 2023-11-02T18:50:23.439 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/ 2023-11-02T18:50:23.440 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM= 2023-11-02T18:50:23.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM=', 'Content-Length': '0'} 2023-11-02T18:50:23.447 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3591e455243ca47-006543ef6f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.448 DEBUG:boto:path=/ibtljn-61/obj46 2023-11-02T18:50:23.448 DEBUG:boto:auth_path=/ibtljn-61/obj46 2023-11-02T18:50:23.448 DEBUG:boto:Method: PUT 2023-11-02T18:50:23.449 DEBUG:boto:Path: /ibtljn-61/obj46 2023-11-02T18:50:23.449 DEBUG:boto:Data: 2023-11-02T18:50:23.450 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:23.450 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.450 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.451 DEBUG:boto:Params: {} 2023-11-02T18:50:23.451 DEBUG:boto:Token: None 2023-11-02T18:50:23.452 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/obj46 2023-11-02T18:50:23.452 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a0LJAUGGPrGFe58YNBvhLzcpjwk= 2023-11-02T18:50:23.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a0LJAUGGPrGFe58YNBvhLzcpjwk='} 2023-11-02T18:50:23.464 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000574f2f2e4209cf18-006543ef6f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.465 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:23.465 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:23.466 DEBUG:boto:Method: HEAD 2023-11-02T18:50:23.466 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:23.467 DEBUG:boto:Data: 2023-11-02T18:50:23.467 DEBUG:boto:Headers: {} 2023-11-02T18:50:23.468 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.468 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.468 DEBUG:boto:Params: {} 2023-11-02T18:50:23.469 DEBUG:boto:Token: None 2023-11-02T18:50:23.469 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/ 2023-11-02T18:50:23.470 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM= 2023-11-02T18:50:23.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM=', 'Content-Length': '0'} 2023-11-02T18:50:23.476 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a43ed629a1e0d46-006543ef6f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.476 DEBUG:boto:path=/ibtljn-61/obj69 2023-11-02T18:50:23.477 DEBUG:boto:auth_path=/ibtljn-61/obj69 2023-11-02T18:50:23.477 DEBUG:boto:Method: PUT 2023-11-02T18:50:23.477 DEBUG:boto:Path: /ibtljn-61/obj69 2023-11-02T18:50:23.478 DEBUG:boto:Data: 2023-11-02T18:50:23.478 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:23.479 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.479 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.480 DEBUG:boto:Params: {} 2023-11-02T18:50:23.480 DEBUG:boto:Token: None 2023-11-02T18:50:23.480 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/obj69 2023-11-02T18:50:23.481 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HwRgClPt+3S5Pl1QVy6346yitWQ= 2023-11-02T18:50:23.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HwRgClPt+3S5Pl1QVy6346yitWQ='} 2023-11-02T18:50:23.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f709c688696f0c7d-006543ef6f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.531 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:23.531 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:23.532 DEBUG:boto:Method: HEAD 2023-11-02T18:50:23.532 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:23.533 DEBUG:boto:Data: 2023-11-02T18:50:23.533 DEBUG:boto:Headers: {} 2023-11-02T18:50:23.534 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.535 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.535 DEBUG:boto:Params: {} 2023-11-02T18:50:23.536 DEBUG:boto:Token: None 2023-11-02T18:50:23.536 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/ 2023-11-02T18:50:23.537 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM= 2023-11-02T18:50:23.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM=', 'Content-Length': '0'} 2023-11-02T18:50:23.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033da9fd2dcfbe47a-006543ef6f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.545 DEBUG:boto:path=/ibtljn-61/obj92 2023-11-02T18:50:23.546 DEBUG:boto:auth_path=/ibtljn-61/obj92 2023-11-02T18:50:23.546 DEBUG:boto:Method: PUT 2023-11-02T18:50:23.547 DEBUG:boto:Path: /ibtljn-61/obj92 2023-11-02T18:50:23.547 DEBUG:boto:Data: 2023-11-02T18:50:23.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:23.548 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.548 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.549 DEBUG:boto:Params: {} 2023-11-02T18:50:23.549 DEBUG:boto:Token: None 2023-11-02T18:50:23.550 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/obj92 2023-11-02T18:50:23.550 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ua9PKsKQf+dUOPgqIq56Vctsxc8= 2023-11-02T18:50:23.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ua9PKsKQf+dUOPgqIq56Vctsxc8='} 2023-11-02T18:50:23.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba852bec8ffb845f-006543ef6f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.563 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:23.563 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:23.563 DEBUG:boto:Method: HEAD 2023-11-02T18:50:23.564 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:23.564 DEBUG:boto:Data: 2023-11-02T18:50:23.565 DEBUG:boto:Headers: {} 2023-11-02T18:50:23.565 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.565 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.566 DEBUG:boto:Params: {} 2023-11-02T18:50:23.566 DEBUG:boto:Token: None 2023-11-02T18:50:23.567 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/ 2023-11-02T18:50:23.567 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM= 2023-11-02T18:50:23.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM=', 'Content-Length': '0'} 2023-11-02T18:50:23.574 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091aade6b8b37ae88-006543ef6f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.575 DEBUG:boto:path=/ibtljn-61/obj115 2023-11-02T18:50:23.575 DEBUG:boto:auth_path=/ibtljn-61/obj115 2023-11-02T18:50:23.576 DEBUG:boto:Method: PUT 2023-11-02T18:50:23.577 DEBUG:boto:Path: /ibtljn-61/obj115 2023-11-02T18:50:23.577 DEBUG:boto:Data: 2023-11-02T18:50:23.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:23.578 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.579 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.579 DEBUG:boto:Params: {} 2023-11-02T18:50:23.580 DEBUG:boto:Token: None 2023-11-02T18:50:23.580 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/obj115 2023-11-02T18:50:23.581 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:u0V1S79Rly6lBnqLhZJuOE+wd6w= 2023-11-02T18:50:23.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:u0V1S79Rly6lBnqLhZJuOE+wd6w='} 2023-11-02T18:50:23.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001965ff4f09219589-006543ef6f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:23.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:23.628 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:50:23.680 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:23.681 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:23.762 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:50:23.763 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-61 2023-11-02T18:50:23.780 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 20 2023-11-02T18:50:23.891 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:23.892 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:23.892 DEBUG:boto:Method: HEAD 2023-11-02T18:50:23.893 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:23.893 DEBUG:boto:Data: 2023-11-02T18:50:23.893 DEBUG:boto:Headers: {} 2023-11-02T18:50:23.894 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.894 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.895 DEBUG:boto:Params: {} 2023-11-02T18:50:23.895 DEBUG:boto:Token: None 2023-11-02T18:50:23.896 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/ 2023-11-02T18:50:23.896 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM= 2023-11-02T18:50:23.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM=', 'Content-Length': '0'} 2023-11-02T18:50:23.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077138104e6aa9cd4-006543ef6f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.904 DEBUG:boto:path=/ibtljn-61/obj29 2023-11-02T18:50:23.904 DEBUG:boto:auth_path=/ibtljn-61/obj29 2023-11-02T18:50:23.905 DEBUG:boto:Method: PUT 2023-11-02T18:50:23.905 DEBUG:boto:Path: /ibtljn-61/obj29 2023-11-02T18:50:23.905 DEBUG:boto:Data: 2023-11-02T18:50:23.906 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:23.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.907 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.907 DEBUG:boto:Params: {} 2023-11-02T18:50:23.908 DEBUG:boto:Token: None 2023-11-02T18:50:23.908 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/obj29 2023-11-02T18:50:23.909 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p+Ovwvs22m581WMYw5HrnVj/jIE= 2023-11-02T18:50:23.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p+Ovwvs22m581WMYw5HrnVj/jIE='} 2023-11-02T18:50:23.921 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3614731018f5bf0-006543ef6f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.921 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:23.922 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:23.922 DEBUG:boto:Method: HEAD 2023-11-02T18:50:23.922 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:23.923 DEBUG:boto:Data: 2023-11-02T18:50:23.923 DEBUG:boto:Headers: {} 2023-11-02T18:50:23.924 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.924 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.924 DEBUG:boto:Params: {} 2023-11-02T18:50:23.925 DEBUG:boto:Token: None 2023-11-02T18:50:23.925 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/ 2023-11-02T18:50:23.926 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM= 2023-11-02T18:50:23.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM=', 'Content-Length': '0'} 2023-11-02T18:50:23.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d65fedbf0e66e297-006543ef6f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.933 DEBUG:boto:path=/ibtljn-61/obj58 2023-11-02T18:50:23.934 DEBUG:boto:auth_path=/ibtljn-61/obj58 2023-11-02T18:50:23.934 DEBUG:boto:Method: PUT 2023-11-02T18:50:23.934 DEBUG:boto:Path: /ibtljn-61/obj58 2023-11-02T18:50:23.935 DEBUG:boto:Data: 2023-11-02T18:50:23.935 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:23.936 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.936 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.937 DEBUG:boto:Params: {} 2023-11-02T18:50:23.937 DEBUG:boto:Token: None 2023-11-02T18:50:23.938 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/obj58 2023-11-02T18:50:23.938 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sAfGLaJKcxZbTa4qSQ6MNMre41k= 2023-11-02T18:50:23.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sAfGLaJKcxZbTa4qSQ6MNMre41k='} 2023-11-02T18:50:23.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000811d2b913fbc1ae4-006543ef6f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.983 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:23.984 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:23.984 DEBUG:boto:Method: HEAD 2023-11-02T18:50:23.985 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:23.985 DEBUG:boto:Data: 2023-11-02T18:50:23.985 DEBUG:boto:Headers: {} 2023-11-02T18:50:23.986 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:23.986 DEBUG:boto:Port: 8000 2023-11-02T18:50:23.987 DEBUG:boto:Params: {} 2023-11-02T18:50:23.987 DEBUG:boto:Token: None 2023-11-02T18:50:23.988 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:23 GMT /ibtljn-61/ 2023-11-02T18:50:23.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM= 2023-11-02T18:50:23.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:23 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H++fyNQRiLQSmZlvS+UrS9qtpBM=', 'Content-Length': '0'} 2023-11-02T18:50:23.998 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000630f3adb6d86c227-006543ef6f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:23.999 DEBUG:boto:path=/ibtljn-61/obj87 2023-11-02T18:50:23.999 DEBUG:boto:auth_path=/ibtljn-61/obj87 2023-11-02T18:50:23.999 DEBUG:boto:Method: PUT 2023-11-02T18:50:24.000 DEBUG:boto:Path: /ibtljn-61/obj87 2023-11-02T18:50:24.000 DEBUG:boto:Data: 2023-11-02T18:50:24.001 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:24.001 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.001 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.002 DEBUG:boto:Params: {} 2023-11-02T18:50:24.002 DEBUG:boto:Token: None 2023-11-02T18:50:24.003 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/obj87 2023-11-02T18:50:24.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mOGUCAVYUutoq2aRdUcL7dguVfc= 2023-11-02T18:50:24.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mOGUCAVYUutoq2aRdUcL7dguVfc='} 2023-11-02T18:50:24.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef98971ed77453dc-006543ef6f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.017 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:24.017 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:24.018 DEBUG:boto:Method: HEAD 2023-11-02T18:50:24.018 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:24.018 DEBUG:boto:Data: 2023-11-02T18:50:24.019 DEBUG:boto:Headers: {} 2023-11-02T18:50:24.020 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.020 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.020 DEBUG:boto:Params: {} 2023-11-02T18:50:24.021 DEBUG:boto:Token: None 2023-11-02T18:50:24.021 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/ 2023-11-02T18:50:24.022 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg= 2023-11-02T18:50:24.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg=', 'Content-Length': '0'} 2023-11-02T18:50:24.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051077495bcd1cf12-006543ef70-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.030 DEBUG:boto:path=/ibtljn-61/obj116 2023-11-02T18:50:24.030 DEBUG:boto:auth_path=/ibtljn-61/obj116 2023-11-02T18:50:24.031 DEBUG:boto:Method: PUT 2023-11-02T18:50:24.031 DEBUG:boto:Path: /ibtljn-61/obj116 2023-11-02T18:50:24.031 DEBUG:boto:Data: 2023-11-02T18:50:24.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:24.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.033 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.033 DEBUG:boto:Params: {} 2023-11-02T18:50:24.034 DEBUG:boto:Token: None 2023-11-02T18:50:24.034 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/obj116 2023-11-02T18:50:24.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lCfVbdvV5lSlVMyqbUH2/UmH75Q= 2023-11-02T18:50:24.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lCfVbdvV5lSlVMyqbUH2/UmH75Q='} 2023-11-02T18:50:24.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084427cebc126f1c7-006543ef70-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.083 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:24.084 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:24.084 DEBUG:boto:Method: HEAD 2023-11-02T18:50:24.084 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:24.085 DEBUG:boto:Data: 2023-11-02T18:50:24.085 DEBUG:boto:Headers: {} 2023-11-02T18:50:24.086 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.086 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.087 DEBUG:boto:Params: {} 2023-11-02T18:50:24.087 DEBUG:boto:Token: None 2023-11-02T18:50:24.088 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/ 2023-11-02T18:50:24.088 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg= 2023-11-02T18:50:24.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg=', 'Content-Length': '0'} 2023-11-02T18:50:24.096 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad9faea466896855-006543ef70-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.097 DEBUG:boto:path=/ibtljn-61/obj145 2023-11-02T18:50:24.097 DEBUG:boto:auth_path=/ibtljn-61/obj145 2023-11-02T18:50:24.097 DEBUG:boto:Method: PUT 2023-11-02T18:50:24.098 DEBUG:boto:Path: /ibtljn-61/obj145 2023-11-02T18:50:24.098 DEBUG:boto:Data: 2023-11-02T18:50:24.099 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:24.099 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.100 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.100 DEBUG:boto:Params: {} 2023-11-02T18:50:24.100 DEBUG:boto:Token: None 2023-11-02T18:50:24.101 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/obj145 2023-11-02T18:50:24.101 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SrRX6v8+BL3AmenSQAGBKNqd0VQ= 2023-11-02T18:50:24.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SrRX6v8+BL3AmenSQAGBKNqd0VQ='} 2023-11-02T18:50:24.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000726a1554db96672c-006543ef70-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:24.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:24.114 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:50:24.163 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:24.163 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:24.252 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:50:24.253 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-61 2023-11-02T18:50:24.275 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 25 2023-11-02T18:50:24.397 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:24.398 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:24.398 DEBUG:boto:Method: HEAD 2023-11-02T18:50:24.398 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:24.399 DEBUG:boto:Data: 2023-11-02T18:50:24.399 DEBUG:boto:Headers: {} 2023-11-02T18:50:24.400 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.400 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.400 DEBUG:boto:Params: {} 2023-11-02T18:50:24.401 DEBUG:boto:Token: None 2023-11-02T18:50:24.401 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/ 2023-11-02T18:50:24.402 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg= 2023-11-02T18:50:24.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg=', 'Content-Length': '0'} 2023-11-02T18:50:24.408 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045ff4dbae09435cd-006543ef70-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.409 DEBUG:boto:path=/ibtljn-61/obj31 2023-11-02T18:50:24.409 DEBUG:boto:auth_path=/ibtljn-61/obj31 2023-11-02T18:50:24.410 DEBUG:boto:Method: PUT 2023-11-02T18:50:24.410 DEBUG:boto:Path: /ibtljn-61/obj31 2023-11-02T18:50:24.411 DEBUG:boto:Data: 2023-11-02T18:50:24.411 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:24.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.412 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.413 DEBUG:boto:Params: {} 2023-11-02T18:50:24.413 DEBUG:boto:Token: None 2023-11-02T18:50:24.414 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/obj31 2023-11-02T18:50:24.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V8hGEUiqgpI1dpJOd/DvapbhtGE= 2023-11-02T18:50:24.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V8hGEUiqgpI1dpJOd/DvapbhtGE='} 2023-11-02T18:50:24.459 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000528727f60485ca5b-006543ef70-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.460 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:24.460 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:24.461 DEBUG:boto:Method: HEAD 2023-11-02T18:50:24.461 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:24.462 DEBUG:boto:Data: 2023-11-02T18:50:24.462 DEBUG:boto:Headers: {} 2023-11-02T18:50:24.463 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.463 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.464 DEBUG:boto:Params: {} 2023-11-02T18:50:24.464 DEBUG:boto:Token: None 2023-11-02T18:50:24.465 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/ 2023-11-02T18:50:24.465 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg= 2023-11-02T18:50:24.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg=', 'Content-Length': '0'} 2023-11-02T18:50:24.473 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c012ff233e151f9-006543ef70-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.474 DEBUG:boto:path=/ibtljn-61/obj62 2023-11-02T18:50:24.474 DEBUG:boto:auth_path=/ibtljn-61/obj62 2023-11-02T18:50:24.475 DEBUG:boto:Method: PUT 2023-11-02T18:50:24.475 DEBUG:boto:Path: /ibtljn-61/obj62 2023-11-02T18:50:24.476 DEBUG:boto:Data: 2023-11-02T18:50:24.476 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:24.476 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.477 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.477 DEBUG:boto:Params: {} 2023-11-02T18:50:24.478 DEBUG:boto:Token: None 2023-11-02T18:50:24.479 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/obj62 2023-11-02T18:50:24.479 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X50FTB9wzBGbvbJ+3ZKJDjueelU= 2023-11-02T18:50:24.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X50FTB9wzBGbvbJ+3ZKJDjueelU='} 2023-11-02T18:50:24.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049d566878b284bff-006543ef70-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.491 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:24.491 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:24.492 DEBUG:boto:Method: HEAD 2023-11-02T18:50:24.492 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:24.492 DEBUG:boto:Data: 2023-11-02T18:50:24.493 DEBUG:boto:Headers: {} 2023-11-02T18:50:24.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.494 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.494 DEBUG:boto:Params: {} 2023-11-02T18:50:24.495 DEBUG:boto:Token: None 2023-11-02T18:50:24.495 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/ 2023-11-02T18:50:24.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg= 2023-11-02T18:50:24.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg=', 'Content-Length': '0'} 2023-11-02T18:50:24.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b25c437d45377eeb-006543ef70-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.503 DEBUG:boto:path=/ibtljn-61/obj93 2023-11-02T18:50:24.504 DEBUG:boto:auth_path=/ibtljn-61/obj93 2023-11-02T18:50:24.504 DEBUG:boto:Method: PUT 2023-11-02T18:50:24.505 DEBUG:boto:Path: /ibtljn-61/obj93 2023-11-02T18:50:24.505 DEBUG:boto:Data: 2023-11-02T18:50:24.506 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:24.506 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.506 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.507 DEBUG:boto:Params: {} 2023-11-02T18:50:24.507 DEBUG:boto:Token: None 2023-11-02T18:50:24.508 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/obj93 2023-11-02T18:50:24.508 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OlKe5QXeNwrak4lvwDbdgZuKMag= 2023-11-02T18:50:24.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OlKe5QXeNwrak4lvwDbdgZuKMag='} 2023-11-02T18:50:24.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d8bb78d9950baa2b-006543ef70-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.555 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:24.556 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:24.556 DEBUG:boto:Method: HEAD 2023-11-02T18:50:24.556 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:24.557 DEBUG:boto:Data: 2023-11-02T18:50:24.557 DEBUG:boto:Headers: {} 2023-11-02T18:50:24.558 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.558 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.559 DEBUG:boto:Params: {} 2023-11-02T18:50:24.559 DEBUG:boto:Token: None 2023-11-02T18:50:24.560 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/ 2023-11-02T18:50:24.560 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg= 2023-11-02T18:50:24.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg=', 'Content-Length': '0'} 2023-11-02T18:50:24.569 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a65c8acf71e4ecc-006543ef70-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.570 DEBUG:boto:path=/ibtljn-61/obj124 2023-11-02T18:50:24.570 DEBUG:boto:auth_path=/ibtljn-61/obj124 2023-11-02T18:50:24.571 DEBUG:boto:Method: PUT 2023-11-02T18:50:24.571 DEBUG:boto:Path: /ibtljn-61/obj124 2023-11-02T18:50:24.571 DEBUG:boto:Data: 2023-11-02T18:50:24.572 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:24.573 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.573 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.574 DEBUG:boto:Params: {} 2023-11-02T18:50:24.574 DEBUG:boto:Token: None 2023-11-02T18:50:24.575 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/obj124 2023-11-02T18:50:24.575 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZakXIyEweiwxLwUXjfLCdwuWbMw= 2023-11-02T18:50:24.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZakXIyEweiwxLwUXjfLCdwuWbMw='} 2023-11-02T18:50:24.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009470deae2a95cc19-006543ef70-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.588 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:24.588 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:24.589 DEBUG:boto:Method: HEAD 2023-11-02T18:50:24.589 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:24.590 DEBUG:boto:Data: 2023-11-02T18:50:24.590 DEBUG:boto:Headers: {} 2023-11-02T18:50:24.591 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.591 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.591 DEBUG:boto:Params: {} 2023-11-02T18:50:24.592 DEBUG:boto:Token: None 2023-11-02T18:50:24.593 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/ 2023-11-02T18:50:24.594 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg= 2023-11-02T18:50:24.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg=', 'Content-Length': '0'} 2023-11-02T18:50:24.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000227eefbbed5ee560-006543ef70-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.602 DEBUG:boto:path=/ibtljn-61/obj155 2023-11-02T18:50:24.603 DEBUG:boto:auth_path=/ibtljn-61/obj155 2023-11-02T18:50:24.603 DEBUG:boto:Method: PUT 2023-11-02T18:50:24.604 DEBUG:boto:Path: /ibtljn-61/obj155 2023-11-02T18:50:24.604 DEBUG:boto:Data: 2023-11-02T18:50:24.604 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:24.605 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.605 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.606 DEBUG:boto:Params: {} 2023-11-02T18:50:24.606 DEBUG:boto:Token: None 2023-11-02T18:50:24.606 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/obj155 2023-11-02T18:50:24.607 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Asw9/XA5m6x03514QRni1r+IjZI= 2023-11-02T18:50:24.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Asw9/XA5m6x03514QRni1r+IjZI='} 2023-11-02T18:50:24.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089b5979e3b160677-006543ef70-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:24.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:24.652 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:50:24.701 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:24.701 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:24.796 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:50:24.796 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-61 2023-11-02T18:50:24.822 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 30 2023-11-02T18:50:24.953 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:24.953 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:24.954 DEBUG:boto:Method: HEAD 2023-11-02T18:50:24.954 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:24.955 DEBUG:boto:Data: 2023-11-02T18:50:24.955 DEBUG:boto:Headers: {} 2023-11-02T18:50:24.956 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.956 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.956 DEBUG:boto:Params: {} 2023-11-02T18:50:24.957 DEBUG:boto:Token: None 2023-11-02T18:50:24.957 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/ 2023-11-02T18:50:24.958 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg= 2023-11-02T18:50:24.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg=', 'Content-Length': '0'} 2023-11-02T18:50:24.965 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c49a0155a34abf9-006543ef70-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.966 DEBUG:boto:path=/ibtljn-61/obj37 2023-11-02T18:50:24.966 DEBUG:boto:auth_path=/ibtljn-61/obj37 2023-11-02T18:50:24.967 DEBUG:boto:Method: PUT 2023-11-02T18:50:24.967 DEBUG:boto:Path: /ibtljn-61/obj37 2023-11-02T18:50:24.968 DEBUG:boto:Data: 2023-11-02T18:50:24.968 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:24.969 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.969 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.970 DEBUG:boto:Params: {} 2023-11-02T18:50:24.970 DEBUG:boto:Token: None 2023-11-02T18:50:24.971 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/obj37 2023-11-02T18:50:24.971 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RQtbEq4xwG5T3D3B2InNku5YNkc= 2023-11-02T18:50:24.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RQtbEq4xwG5T3D3B2InNku5YNkc='} 2023-11-02T18:50:24.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b38877e83ad1680d-006543ef70-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.983 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:24.983 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:24.986 DEBUG:boto:Method: HEAD 2023-11-02T18:50:24.986 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:24.987 DEBUG:boto:Data: 2023-11-02T18:50:24.987 DEBUG:boto:Headers: {} 2023-11-02T18:50:24.988 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:24.988 DEBUG:boto:Port: 8000 2023-11-02T18:50:24.989 DEBUG:boto:Params: {} 2023-11-02T18:50:24.989 DEBUG:boto:Token: None 2023-11-02T18:50:24.990 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:24 GMT /ibtljn-61/ 2023-11-02T18:50:24.990 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg= 2023-11-02T18:50:24.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GxZfKzHK06hDglvqPDLqHForjyg=', 'Content-Length': '0'} 2023-11-02T18:50:24.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007391e774119f0251-006543ef70-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:24.998 DEBUG:boto:path=/ibtljn-61/obj74 2023-11-02T18:50:24.999 DEBUG:boto:auth_path=/ibtljn-61/obj74 2023-11-02T18:50:24.999 DEBUG:boto:Method: PUT 2023-11-02T18:50:25.000 DEBUG:boto:Path: /ibtljn-61/obj74 2023-11-02T18:50:25.000 DEBUG:boto:Data: 2023-11-02T18:50:25.000 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:25.001 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:25.001 DEBUG:boto:Port: 8000 2023-11-02T18:50:25.002 DEBUG:boto:Params: {} 2023-11-02T18:50:25.002 DEBUG:boto:Token: None 2023-11-02T18:50:25.003 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:25 GMT /ibtljn-61/obj74 2023-11-02T18:50:25.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QK57YgRuvDSiI51QMYRTlCSu7+g= 2023-11-02T18:50:25.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QK57YgRuvDSiI51QMYRTlCSu7+g='} 2023-11-02T18:50:25.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001dd032a32818adfe-006543ef70-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:25.051 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:25.052 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:25.052 DEBUG:boto:Method: HEAD 2023-11-02T18:50:25.053 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:25.053 DEBUG:boto:Data: 2023-11-02T18:50:25.053 DEBUG:boto:Headers: {} 2023-11-02T18:50:25.054 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:25.054 DEBUG:boto:Port: 8000 2023-11-02T18:50:25.055 DEBUG:boto:Params: {} 2023-11-02T18:50:25.055 DEBUG:boto:Token: None 2023-11-02T18:50:25.056 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:25 GMT /ibtljn-61/ 2023-11-02T18:50:25.056 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6YVbqujf3cvsDBferQT/tZlUrDQ= 2023-11-02T18:50:25.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6YVbqujf3cvsDBferQT/tZlUrDQ=', 'Content-Length': '0'} 2023-11-02T18:50:25.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000faae717aba8291d3-006543ef71-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:25.065 DEBUG:boto:path=/ibtljn-61/obj111 2023-11-02T18:50:25.065 DEBUG:boto:auth_path=/ibtljn-61/obj111 2023-11-02T18:50:25.066 DEBUG:boto:Method: PUT 2023-11-02T18:50:25.066 DEBUG:boto:Path: /ibtljn-61/obj111 2023-11-02T18:50:25.066 DEBUG:boto:Data: 2023-11-02T18:50:25.067 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:25.067 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:25.068 DEBUG:boto:Port: 8000 2023-11-02T18:50:25.068 DEBUG:boto:Params: {} 2023-11-02T18:50:25.069 DEBUG:boto:Token: None 2023-11-02T18:50:25.069 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:25 GMT /ibtljn-61/obj111 2023-11-02T18:50:25.069 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:96YdABJtsbfaO607g9T+afj1D30= 2023-11-02T18:50:25.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:96YdABJtsbfaO607g9T+afj1D30='} 2023-11-02T18:50:25.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006568d618f5de5e7d-006543ef71-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:25.081 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:25.081 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:25.082 DEBUG:boto:Method: HEAD 2023-11-02T18:50:25.082 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:25.083 DEBUG:boto:Data: 2023-11-02T18:50:25.083 DEBUG:boto:Headers: {} 2023-11-02T18:50:25.083 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:25.084 DEBUG:boto:Port: 8000 2023-11-02T18:50:25.084 DEBUG:boto:Params: {} 2023-11-02T18:50:25.085 DEBUG:boto:Token: None 2023-11-02T18:50:25.085 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:25 GMT /ibtljn-61/ 2023-11-02T18:50:25.086 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6YVbqujf3cvsDBferQT/tZlUrDQ= 2023-11-02T18:50:25.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6YVbqujf3cvsDBferQT/tZlUrDQ=', 'Content-Length': '0'} 2023-11-02T18:50:25.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b2a6e20909fdb00-006543ef71-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:25.094 DEBUG:boto:path=/ibtljn-61/obj148 2023-11-02T18:50:25.094 DEBUG:boto:auth_path=/ibtljn-61/obj148 2023-11-02T18:50:25.095 DEBUG:boto:Method: PUT 2023-11-02T18:50:25.095 DEBUG:boto:Path: /ibtljn-61/obj148 2023-11-02T18:50:25.095 DEBUG:boto:Data: 2023-11-02T18:50:25.096 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:25.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:25.097 DEBUG:boto:Port: 8000 2023-11-02T18:50:25.097 DEBUG:boto:Params: {} 2023-11-02T18:50:25.098 DEBUG:boto:Token: None 2023-11-02T18:50:25.098 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:25 GMT /ibtljn-61/obj148 2023-11-02T18:50:25.099 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LRuG99NJDcLETTN95MTkIxM2YRc= 2023-11-02T18:50:25.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LRuG99NJDcLETTN95MTkIxM2YRc='} 2023-11-02T18:50:25.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044a4629675f8d002-006543ef71-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:25.143 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:50:25.144 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:50:25.144 DEBUG:boto:Method: HEAD 2023-11-02T18:50:25.145 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:50:25.145 DEBUG:boto:Data: 2023-11-02T18:50:25.146 DEBUG:boto:Headers: {} 2023-11-02T18:50:25.146 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:25.146 DEBUG:boto:Port: 8000 2023-11-02T18:50:25.147 DEBUG:boto:Params: {} 2023-11-02T18:50:25.147 DEBUG:boto:Token: None 2023-11-02T18:50:25.148 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:50:25 GMT /ibtljn-61/ 2023-11-02T18:50:25.148 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6YVbqujf3cvsDBferQT/tZlUrDQ= 2023-11-02T18:50:25.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:50:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6YVbqujf3cvsDBferQT/tZlUrDQ=', 'Content-Length': '0'} 2023-11-02T18:50:25.157 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000785fd75bd6eadd4-006543ef71-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:25.157 DEBUG:boto:path=/ibtljn-61/obj185 2023-11-02T18:50:25.158 DEBUG:boto:auth_path=/ibtljn-61/obj185 2023-11-02T18:50:25.158 DEBUG:boto:Method: PUT 2023-11-02T18:50:25.159 DEBUG:boto:Path: /ibtljn-61/obj185 2023-11-02T18:50:25.159 DEBUG:boto:Data: 2023-11-02T18:50:25.160 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:50:25.160 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:50:25.160 DEBUG:boto:Port: 8000 2023-11-02T18:50:25.161 DEBUG:boto:Params: {} 2023-11-02T18:50:25.161 DEBUG:boto:Token: None 2023-11-02T18:50:25.162 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:50:25 GMT /ibtljn-61/obj185 2023-11-02T18:50:25.162 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kd6SLwWhE22IZnmM7+Cs1Hild/U= 2023-11-02T18:50:25.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:50:25 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kd6SLwWhE22IZnmM7+Cs1Hild/U='} 2023-11-02T18:50:25.173 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9d7c8e5a43196e5-006543ef71-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:50:25.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:25.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:50:25.175 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:50:25.220 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:25.220 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:25.280 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:25.273+0000 7f8e89e86f40 5 RGW-SYNC:data: waiting on data sync status lease 2023-11-02T18:50:25.282 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:25.277+0000 7f8e89e86f40 5 RGW-SYNC:data: acquired data sync status lease 2023-11-02T18:50:25.322 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-11-02T18:50:25.323 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-11-02T18:50:25.325 INFO:tasks.rgw.c1.client.1:Started 2023-11-02T18:50:25.326 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:25.341 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T18:50:25.342 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T18:50:25.342 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-11-02T18:50:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-11-02T18:50:25.344 INFO:teuthology.orchestra.run.smithi017.stderr:curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8001: Connection refused 2023-11-02T18:50:26.345 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T18:50:26.360 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T18:50:26.361 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T18:50:26.363 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-11-02T18:50:26.364 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2023-11-02T18:50:26.365 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-61 2023-11-02T18:50:26.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:50:26.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:50:26.367 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:50:26.461 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:50:26.462 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:50:26.534 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:26.525+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:50:27.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-11-02T18:50:31.535 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:31.529+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:50:36.536 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:36.529+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:50:41.537 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:41.529+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:50:46.538 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:46.529+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:50:51.539 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:51.533+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:50:56.540 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:50:56.533+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:51:01.541 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:51:01.533+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:51:06.542 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:51:06.533+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:51:11.542 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:51:11.537+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:51:16.544 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:51:16.537+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:51:21.544 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:51:21.537+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:51:26.545 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:51:26.537+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:51:31.546 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:51:31.541+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:51:36.547 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:51:36.541+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:51:41.548 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:51:41.541+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:51:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:51:46.541+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:51:51.550 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:51:51.541+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:51:56.552 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:51:56.545+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:52:01.553 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:01.544+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:52:06.554 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:06.544+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:52:11.555 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:11.548+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:52:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:16.548+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:52:21.557 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:21.552+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:52:26.558 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:26.548+0000 7fef439f9f40 1 waiting to reach incremental sync.. 2023-11-02T18:52:31.566 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:31.556+0000 7fef439f9f40 1 bucket sync caught up with source: 2023-11-02T18:52:31.566 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.118.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.74.6, , , , 00000000001.68.6] 2023-11-02T18:52:31.567 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.118.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.74.6, , , , 00000000001.68.6] 2023-11-02T18:52:31.567 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:31.556+0000 7fef439f9f40 0 bucket checkpoint complete 2023-11-02T18:52:31.592 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-61 2023-11-02T18:52:31.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:52:31.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:52:31.594 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:52:31.647 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:52:31.648 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:52:31.708 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:31.700+0000 7f95b6186f40 1 waiting to reach incremental sync.. 2023-11-02T18:52:36.711 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:36.704+0000 7f95b6186f40 1 bucket sync caught up with source: 2023-11-02T18:52:36.711 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000003.253.6, 00000000002.301.6, 00000000002.162.6, 00000000005.300.6, 00000000004.296.6, 00000000001.345.6, 00000000001.119.6, 00000000004.261.6, 00000000007.251.6, 00000000006.305.6] 2023-11-02T18:52:36.712 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000003.253.6, 00000000002.301.6, 00000000002.162.6, 00000000005.300.6, 00000000004.296.6, 00000000001.345.6, 00000000001.119.6, 00000000004.261.6, 00000000007.251.6, 00000000006.305.6] 2023-11-02T18:52:36.712 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:36.704+0000 7f95b6186f40 0 bucket checkpoint complete 2023-11-02T18:52:36.743 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-61 zones={a2, a1} 2023-11-02T18:52:36.743 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:36.744 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:36.745 DEBUG:boto:Method: HEAD 2023-11-02T18:52:36.745 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:52:36.746 DEBUG:boto:Data: 2023-11-02T18:52:36.746 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.747 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:36.749 DEBUG:boto:Port: 8001 2023-11-02T18:52:36.750 DEBUG:boto:Params: {} 2023-11-02T18:52:36.751 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:52:36.752 DEBUG:boto:Token: None 2023-11-02T18:52:36.753 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/ 2023-11-02T18:52:36.753 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dfffCV0ED5xA9R8Xk0CH1iHZSuc= 2023-11-02T18:52:36.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dfffCV0ED5xA9R8Xk0CH1iHZSuc=', 'Content-Length': '0'} 2023-11-02T18:52:36.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000437199e4a097b553-006543eff4-1f6a-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.765 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:36.766 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:36.766 DEBUG:boto:Method: HEAD 2023-11-02T18:52:36.767 DEBUG:boto:Path: /ibtljn-61/ 2023-11-02T18:52:36.767 DEBUG:boto:Data: 2023-11-02T18:52:36.768 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.768 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:36.769 DEBUG:boto:Port: 8000 2023-11-02T18:52:36.769 DEBUG:boto:Params: {} 2023-11-02T18:52:36.770 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:52:36.771 DEBUG:boto:Token: None 2023-11-02T18:52:36.771 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/ 2023-11-02T18:52:36.772 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dfffCV0ED5xA9R8Xk0CH1iHZSuc= 2023-11-02T18:52:36.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dfffCV0ED5xA9R8Xk0CH1iHZSuc=', 'Content-Length': '0'} 2023-11-02T18:52:36.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053ad4809dbf54d51-006543eff4-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.786 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:52:36.786 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:36.787 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:36.787 DEBUG:boto:path=/ibtljn-61/?versions&max-keys=999 2023-11-02T18:52:36.788 DEBUG:boto:auth_path=/ibtljn-61/?versions&max-keys=999 2023-11-02T18:52:36.788 DEBUG:boto:Method: GET 2023-11-02T18:52:36.788 DEBUG:boto:Path: /ibtljn-61/?versions&max-keys=999 2023-11-02T18:52:36.789 DEBUG:boto:Data: 2023-11-02T18:52:36.789 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:36.790 DEBUG:boto:Port: 8001 2023-11-02T18:52:36.791 DEBUG:boto:Params: {} 2023-11-02T18:52:36.791 DEBUG:boto:Token: None 2023-11-02T18:52:36.792 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/?versions 2023-11-02T18:52:36.792 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9YArSFwTXmssIhNagVU8v4m0LOM= 2023-11-02T18:52:36.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9YArSFwTXmssIhNagVU8v4m0LOM=', 'Content-Length': '0'} 2023-11-02T18:52:36.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0b351e076bea0e5-006543eff4-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.798 DEBUG:boto:b'ibtljn-61999falseobj11nulltrue2023-11-02T18:50:21.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T18:50:25.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:50:23.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:50:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:50:24.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:50:24.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T18:50:25.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:50:24.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:50:22.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T18:50:25.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:50:22.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:50:22.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:50:23.389Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:50:23.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:50:24.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:50:22.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:50:22.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T18:50:24.971Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:50:22.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:50:22.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:50:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:50:22.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:50:22.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:50:23.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:50:23.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:50:24.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:50:22.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:50:23.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T18:50:25.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:50:23.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:50:22.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:50:24.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:50:23.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:50:24.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:50:23.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:36.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:52:36.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-11-02T18:52:36.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T18:52:36.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-11-02T18:52:36.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-11-02T18:52:36.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-11-02T18:52:36.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-11-02T18:52:36.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-11-02T18:52:36.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:52:36.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-11-02T18:52:36.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:52:36.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:52:36.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T18:52:36.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-11-02T18:52:36.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-11-02T18:52:36.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:52:36.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:52:36.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-11-02T18:52:36.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:52:36.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:52:36.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T18:52:36.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:52:36.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:52:36.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:52:36.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-11-02T18:52:36.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-11-02T18:52:36.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:52:36.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T18:52:36.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-11-02T18:52:36.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:52:36.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:52:36.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-11-02T18:52:36.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T18:52:36.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-11-02T18:52:36.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:52:36.815 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:52:36.815 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:36.816 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:36.816 DEBUG:boto:path=/ibtljn-61/?versions&max-keys=999 2023-11-02T18:52:36.817 DEBUG:boto:auth_path=/ibtljn-61/?versions&max-keys=999 2023-11-02T18:52:36.817 DEBUG:boto:Method: GET 2023-11-02T18:52:36.817 DEBUG:boto:Path: /ibtljn-61/?versions&max-keys=999 2023-11-02T18:52:36.818 DEBUG:boto:Data: 2023-11-02T18:52:36.818 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.819 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:36.819 DEBUG:boto:Port: 8000 2023-11-02T18:52:36.820 DEBUG:boto:Params: {} 2023-11-02T18:52:36.820 DEBUG:boto:Token: None 2023-11-02T18:52:36.821 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/?versions 2023-11-02T18:52:36.821 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9YArSFwTXmssIhNagVU8v4m0LOM= 2023-11-02T18:52:36.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9YArSFwTXmssIhNagVU8v4m0LOM=', 'Content-Length': '0'} 2023-11-02T18:52:36.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a739f2817ddb16a7-006543eff4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.832 DEBUG:boto:b'ibtljn-61999falseobj11nulltrue2023-11-02T18:50:21.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T18:50:25.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:50:23.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:50:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:50:24.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:50:24.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T18:50:25.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:50:24.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:50:22.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T18:50:25.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:50:22.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:50:22.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:50:23.389Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:50:23.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:50:24.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:50:22.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:50:22.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T18:50:24.971Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:50:22.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:50:22.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:50:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:50:22.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:50:22.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:50:23.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:50:23.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:50:24.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:50:22.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:50:23.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T18:50:25.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:50:23.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:50:22.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:50:24.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:50:23.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:50:24.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:50:23.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:36.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:52:36.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-11-02T18:52:36.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T18:52:36.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-11-02T18:52:36.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-11-02T18:52:36.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-11-02T18:52:36.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-11-02T18:52:36.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-11-02T18:52:36.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T18:52:36.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-11-02T18:52:36.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T18:52:36.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:52:36.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T18:52:36.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-11-02T18:52:36.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-11-02T18:52:36.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:52:36.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T18:52:36.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-11-02T18:52:36.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T18:52:36.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:52:36.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T18:52:36.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T18:52:36.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:52:36.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T18:52:36.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-11-02T18:52:36.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-11-02T18:52:36.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T18:52:36.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T18:52:36.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-11-02T18:52:36.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T18:52:36.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T18:52:36.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-11-02T18:52:36.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T18:52:36.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-11-02T18:52:36.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T18:52:36.849 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:36.849 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:36.849 DEBUG:boto:path=/ibtljn-61/?versions&max-keys=999 2023-11-02T18:52:36.850 DEBUG:boto:auth_path=/ibtljn-61/?versions&max-keys=999 2023-11-02T18:52:36.850 DEBUG:boto:Method: GET 2023-11-02T18:52:36.851 DEBUG:boto:Path: /ibtljn-61/?versions&max-keys=999 2023-11-02T18:52:36.851 DEBUG:boto:Data: 2023-11-02T18:52:36.852 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:36.852 DEBUG:boto:Port: 8001 2023-11-02T18:52:36.853 DEBUG:boto:Params: {} 2023-11-02T18:52:36.853 DEBUG:boto:Token: None 2023-11-02T18:52:36.854 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/?versions 2023-11-02T18:52:36.854 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9YArSFwTXmssIhNagVU8v4m0LOM= 2023-11-02T18:52:36.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9YArSFwTXmssIhNagVU8v4m0LOM=', 'Content-Length': '0'} 2023-11-02T18:52:36.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2f76ebcb93432ee-006543eff4-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.861 DEBUG:boto:b'ibtljn-61999falseobj11nulltrue2023-11-02T18:50:21.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T18:50:25.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:50:23.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:50:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:50:24.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:50:24.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T18:50:25.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:50:24.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:50:22.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T18:50:25.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:50:22.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:50:22.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:50:23.389Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:50:23.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:50:24.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:50:22.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:50:22.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T18:50:24.971Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:50:22.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:50:22.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:50:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:50:22.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:50:22.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:50:23.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:50:23.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:50:24.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:50:22.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:50:23.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T18:50:25.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:50:23.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:50:22.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:50:24.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:50:23.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:50:24.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:50:23.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:36.863 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:36.864 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:36.864 DEBUG:boto:path=/ibtljn-61/?versions&max-keys=999 2023-11-02T18:52:36.865 DEBUG:boto:auth_path=/ibtljn-61/?versions&max-keys=999 2023-11-02T18:52:36.865 DEBUG:boto:Method: GET 2023-11-02T18:52:36.866 DEBUG:boto:Path: /ibtljn-61/?versions&max-keys=999 2023-11-02T18:52:36.866 DEBUG:boto:Data: 2023-11-02T18:52:36.867 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.867 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:36.867 DEBUG:boto:Port: 8000 2023-11-02T18:52:36.868 DEBUG:boto:Params: {} 2023-11-02T18:52:36.868 DEBUG:boto:Token: None 2023-11-02T18:52:36.869 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/?versions 2023-11-02T18:52:36.870 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9YArSFwTXmssIhNagVU8v4m0LOM= 2023-11-02T18:52:36.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9YArSFwTXmssIhNagVU8v4m0LOM=', 'Content-Length': '0'} 2023-11-02T18:52:36.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ab6e65db0c2c95a-006543eff4-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.880 DEBUG:boto:b'ibtljn-61999falseobj11nulltrue2023-11-02T18:50:21.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T18:50:25.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T18:50:23.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T18:50:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T18:50:24.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T18:50:24.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T18:50:25.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T18:50:24.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T18:50:22.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T18:50:25.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T18:50:22.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:50:22.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T18:50:23.389Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T18:50:23.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T18:50:24.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:50:22.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T18:50:22.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T18:50:24.971Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T18:50:22.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:50:22.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T18:50:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T18:50:22.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:50:22.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T18:50:23.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T18:50:23.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T18:50:24.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T18:50:22.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T18:50:23.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T18:50:25.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T18:50:23.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T18:50:22.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T18:50:24.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T18:50:23.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T18:50:24.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T18:50:23.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:36.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:52:36.883 DEBUG:boto:path=/ibtljn-61/obj11 2023-11-02T18:52:36.884 DEBUG:boto:auth_path=/ibtljn-61/obj11 2023-11-02T18:52:36.884 DEBUG:boto:path=/ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.885 DEBUG:boto:auth_path=/ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.885 DEBUG:boto:Method: GET 2023-11-02T18:52:36.886 DEBUG:boto:Path: /ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.886 DEBUG:boto:Data: 2023-11-02T18:52:36.887 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.887 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:36.888 DEBUG:boto:Port: 8001 2023-11-02T18:52:36.888 DEBUG:boto:Params: {} 2023-11-02T18:52:36.889 DEBUG:boto:Token: None 2023-11-02T18:52:36.889 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.890 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rck7aXKFXyCDD3z9qVI9Yx8MG2A= 2023-11-02T18:52:36.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rck7aXKFXyCDD3z9qVI9Yx8MG2A=', 'Content-Length': '0'} 2023-11-02T18:52:36.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084ba02fa0d4d1df3-006543eff4-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.894 DEBUG:boto:path=/ibtljn-61/obj11 2023-11-02T18:52:36.895 DEBUG:boto:auth_path=/ibtljn-61/obj11 2023-11-02T18:52:36.895 DEBUG:boto:path=/ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.896 DEBUG:boto:auth_path=/ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.896 DEBUG:boto:Method: GET 2023-11-02T18:52:36.897 DEBUG:boto:Path: /ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.897 DEBUG:boto:Data: 2023-11-02T18:52:36.898 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.898 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:36.899 DEBUG:boto:Port: 8000 2023-11-02T18:52:36.899 DEBUG:boto:Params: {} 2023-11-02T18:52:36.900 DEBUG:boto:Token: None 2023-11-02T18:52:36.900 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.901 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rck7aXKFXyCDD3z9qVI9Yx8MG2A= 2023-11-02T18:52:36.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rck7aXKFXyCDD3z9qVI9Yx8MG2A=', 'Content-Length': '0'} 2023-11-02T18:52:36.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018b6c6122c3113d9-006543eff4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.904 DEBUG:boto:path=/ibtljn-61/obj11 2023-11-02T18:52:36.905 DEBUG:boto:auth_path=/ibtljn-61/obj11 2023-11-02T18:52:36.905 DEBUG:boto:path=/ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.905 DEBUG:boto:auth_path=/ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.906 DEBUG:boto:Method: HEAD 2023-11-02T18:52:36.906 DEBUG:boto:Path: /ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.907 DEBUG:boto:Data: 2023-11-02T18:52:36.907 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.908 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:36.908 DEBUG:boto:Port: 8001 2023-11-02T18:52:36.908 DEBUG:boto:Params: {} 2023-11-02T18:52:36.909 DEBUG:boto:Token: None 2023-11-02T18:52:36.909 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.910 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ar4Iua0BX7FFK8lCFTf/85/hMOs= 2023-11-02T18:52:36.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ar4Iua0BX7FFK8lCFTf/85/hMOs=', 'Content-Length': '0'} 2023-11-02T18:52:36.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a6914d208082980-006543eff4-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.914 DEBUG:boto:path=/ibtljn-61/obj11 2023-11-02T18:52:36.914 DEBUG:boto:auth_path=/ibtljn-61/obj11 2023-11-02T18:52:36.915 DEBUG:boto:path=/ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.915 DEBUG:boto:auth_path=/ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.916 DEBUG:boto:Method: HEAD 2023-11-02T18:52:36.916 DEBUG:boto:Path: /ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.917 DEBUG:boto:Data: 2023-11-02T18:52:36.917 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:36.918 DEBUG:boto:Port: 8000 2023-11-02T18:52:36.918 DEBUG:boto:Params: {} 2023-11-02T18:52:36.919 DEBUG:boto:Token: None 2023-11-02T18:52:36.920 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.920 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ar4Iua0BX7FFK8lCFTf/85/hMOs= 2023-11-02T18:52:36.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ar4Iua0BX7FFK8lCFTf/85/hMOs=', 'Content-Length': '0'} 2023-11-02T18:52:36.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000235d2f9468492459-006543eff4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:52:36.925 DEBUG:boto:path=/ibtljn-61/obj11 2023-11-02T18:52:36.925 DEBUG:boto:auth_path=/ibtljn-61/obj11 2023-11-02T18:52:36.925 DEBUG:boto:path=/ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.926 DEBUG:boto:auth_path=/ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.926 DEBUG:boto:Method: GET 2023-11-02T18:52:36.927 DEBUG:boto:Path: /ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.927 DEBUG:boto:Data: 2023-11-02T18:52:36.928 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.928 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:36.929 DEBUG:boto:Port: 8001 2023-11-02T18:52:36.929 DEBUG:boto:Params: {} 2023-11-02T18:52:36.930 DEBUG:boto:Token: None 2023-11-02T18:52:36.930 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.931 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rck7aXKFXyCDD3z9qVI9Yx8MG2A= 2023-11-02T18:52:36.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rck7aXKFXyCDD3z9qVI9Yx8MG2A=', 'Content-Length': '0'} 2023-11-02T18:52:36.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000591df2bd15e043a3-006543eff4-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.935 DEBUG:boto:path=/ibtljn-61/obj11 2023-11-02T18:52:36.935 DEBUG:boto:auth_path=/ibtljn-61/obj11 2023-11-02T18:52:36.936 DEBUG:boto:path=/ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.937 DEBUG:boto:auth_path=/ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.937 DEBUG:boto:Method: GET 2023-11-02T18:52:36.937 DEBUG:boto:Path: /ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.938 DEBUG:boto:Data: 2023-11-02T18:52:36.938 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.939 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:36.939 DEBUG:boto:Port: 8000 2023-11-02T18:52:36.940 DEBUG:boto:Params: {} 2023-11-02T18:52:36.940 DEBUG:boto:Token: None 2023-11-02T18:52:36.941 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/obj11?versionId=null 2023-11-02T18:52:36.941 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rck7aXKFXyCDD3z9qVI9Yx8MG2A= 2023-11-02T18:52:36.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rck7aXKFXyCDD3z9qVI9Yx8MG2A=', 'Content-Length': '0'} 2023-11-02T18:52:36.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005421d312a0cad7bd-006543eff4-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.986 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:36.987 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:36.988 DEBUG:boto:path=/ibtljn-61/?prefix=obj11 2023-11-02T18:52:36.988 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj11 2023-11-02T18:52:36.988 DEBUG:boto:Method: GET 2023-11-02T18:52:36.989 DEBUG:boto:Path: /ibtljn-61/?prefix=obj11 2023-11-02T18:52:36.989 DEBUG:boto:Data: 2023-11-02T18:52:36.990 DEBUG:boto:Headers: {} 2023-11-02T18:52:36.990 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:36.991 DEBUG:boto:Port: 8001 2023-11-02T18:52:36.991 DEBUG:boto:Params: {} 2023-11-02T18:52:36.992 DEBUG:boto:Token: None 2023-11-02T18:52:36.992 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:36 GMT /ibtljn-61/ 2023-11-02T18:52:36.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:skxl2tnKH70rA/3mazgeHPqvrfI= 2023-11-02T18:52:36.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:skxl2tnKH70rA/3mazgeHPqvrfI=', 'Content-Length': '0'} 2023-11-02T18:52:36.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056c760ea9c3b62e4-006543eff4-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:36.999 DEBUG:boto:b'ibtljn-61obj111000falseobj112023-11-02T18:50:21.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T18:50:25.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:50:23.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:50:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.000 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.000 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.001 DEBUG:boto:path=/ibtljn-61/?prefix=obj11 2023-11-02T18:52:37.001 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj11 2023-11-02T18:52:37.002 DEBUG:boto:Method: GET 2023-11-02T18:52:37.002 DEBUG:boto:Path: /ibtljn-61/?prefix=obj11 2023-11-02T18:52:37.003 DEBUG:boto:Data: 2023-11-02T18:52:37.003 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.003 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.004 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.004 DEBUG:boto:Params: {} 2023-11-02T18:52:37.005 DEBUG:boto:Token: None 2023-11-02T18:52:37.005 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.006 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d67a5cfe9d0b41e-006543eff5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.016 DEBUG:boto:b'ibtljn-61obj111000falseobj112023-11-02T18:50:21.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T18:50:25.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:50:23.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:50:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.017 DEBUG:boto:path=/ibtljn-61/obj11 2023-11-02T18:52:37.018 DEBUG:boto:auth_path=/ibtljn-61/obj11 2023-11-02T18:52:37.018 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.019 DEBUG:boto:Path: /ibtljn-61/obj11 2023-11-02T18:52:37.019 DEBUG:boto:Data: 2023-11-02T18:52:37.019 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.020 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.020 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.021 DEBUG:boto:Params: {} 2023-11-02T18:52:37.021 DEBUG:boto:Token: None 2023-11-02T18:52:37.022 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj11 2023-11-02T18:52:37.022 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VWpxfkZDfUK2vudug5pO8yQHjjo= 2023-11-02T18:52:37.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VWpxfkZDfUK2vudug5pO8yQHjjo=', 'Content-Length': '0'} 2023-11-02T18:52:37.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085637d7df7fc14c9-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.026 DEBUG:boto:path=/ibtljn-61/obj11 2023-11-02T18:52:37.026 DEBUG:boto:auth_path=/ibtljn-61/obj11 2023-11-02T18:52:37.026 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.027 DEBUG:boto:Path: /ibtljn-61/obj11 2023-11-02T18:52:37.027 DEBUG:boto:Data: 2023-11-02T18:52:37.028 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.028 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.029 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.029 DEBUG:boto:Params: {} 2023-11-02T18:52:37.030 DEBUG:boto:Token: None 2023-11-02T18:52:37.030 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj11 2023-11-02T18:52:37.031 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VWpxfkZDfUK2vudug5pO8yQHjjo= 2023-11-02T18:52:37.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VWpxfkZDfUK2vudug5pO8yQHjjo=', 'Content-Length': '0'} 2023-11-02T18:52:37.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003130040449f02c8d-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:52:37.035 DEBUG:boto:path=/ibtljn-61/obj11 2023-11-02T18:52:37.035 DEBUG:boto:auth_path=/ibtljn-61/obj11 2023-11-02T18:52:37.036 DEBUG:boto:Method: GET 2023-11-02T18:52:37.036 DEBUG:boto:Path: /ibtljn-61/obj11 2023-11-02T18:52:37.036 DEBUG:boto:Data: 2023-11-02T18:52:37.037 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.037 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.038 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.038 DEBUG:boto:Params: {} 2023-11-02T18:52:37.039 DEBUG:boto:Token: None 2023-11-02T18:52:37.039 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj11 2023-11-02T18:52:37.040 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VAzazSoWM7kf5Szd3DHfsuzA4DE= 2023-11-02T18:52:37.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VAzazSoWM7kf5Szd3DHfsuzA4DE=', 'Content-Length': '0'} 2023-11-02T18:52:37.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ebb041315268fe7-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.090 DEBUG:boto:path=/ibtljn-61/obj11 2023-11-02T18:52:37.091 DEBUG:boto:auth_path=/ibtljn-61/obj11 2023-11-02T18:52:37.092 DEBUG:boto:Method: GET 2023-11-02T18:52:37.092 DEBUG:boto:Path: /ibtljn-61/obj11 2023-11-02T18:52:37.093 DEBUG:boto:Data: 2023-11-02T18:52:37.093 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.094 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.094 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.095 DEBUG:boto:Params: {} 2023-11-02T18:52:37.095 DEBUG:boto:Token: None 2023-11-02T18:52:37.096 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj11 2023-11-02T18:52:37.097 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VAzazSoWM7kf5Szd3DHfsuzA4DE= 2023-11-02T18:52:37.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VAzazSoWM7kf5Szd3DHfsuzA4DE=', 'Content-Length': '0'} 2023-11-02T18:52:37.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e3241491213d39-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.102 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.102 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.103 DEBUG:boto:path=/ibtljn-61/?prefix=obj11 2023-11-02T18:52:37.103 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj11 2023-11-02T18:52:37.104 DEBUG:boto:Method: GET 2023-11-02T18:52:37.104 DEBUG:boto:Path: /ibtljn-61/?prefix=obj11 2023-11-02T18:52:37.105 DEBUG:boto:Data: 2023-11-02T18:52:37.105 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.106 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.107 DEBUG:boto:Params: {} 2023-11-02T18:52:37.107 DEBUG:boto:Token: None 2023-11-02T18:52:37.108 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.109 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c71667ac03afc91-006543eff5-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.116 DEBUG:boto:b'ibtljn-61obj111000falseobj112023-11-02T18:50:21.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T18:50:25.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:50:23.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:50:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.117 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.117 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.118 DEBUG:boto:path=/ibtljn-61/?prefix=obj11 2023-11-02T18:52:37.119 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj11 2023-11-02T18:52:37.119 DEBUG:boto:Method: GET 2023-11-02T18:52:37.120 DEBUG:boto:Path: /ibtljn-61/?prefix=obj11 2023-11-02T18:52:37.120 DEBUG:boto:Data: 2023-11-02T18:52:37.121 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.121 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.121 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.122 DEBUG:boto:Params: {} 2023-11-02T18:52:37.123 DEBUG:boto:Token: None 2023-11-02T18:52:37.123 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.124 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000712d513667aa1996-006543eff5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.134 DEBUG:boto:b'ibtljn-61obj111000falseobj112023-11-02T18:50:21.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T18:50:25.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T18:50:23.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T18:50:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-11-02T18:52:37.135 DEBUG:boto:path=/ibtljn-61/obj111 2023-11-02T18:52:37.136 DEBUG:boto:auth_path=/ibtljn-61/obj111 2023-11-02T18:52:37.136 DEBUG:boto:path=/ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.137 DEBUG:boto:auth_path=/ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.137 DEBUG:boto:Method: GET 2023-11-02T18:52:37.138 DEBUG:boto:Path: /ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.138 DEBUG:boto:Data: 2023-11-02T18:52:37.139 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.140 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.140 DEBUG:boto:Params: {} 2023-11-02T18:52:37.141 DEBUG:boto:Token: None 2023-11-02T18:52:37.141 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.142 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hKTUulJQe8OaYDUIXzgQrhdfFcs= 2023-11-02T18:52:37.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hKTUulJQe8OaYDUIXzgQrhdfFcs=', 'Content-Length': '0'} 2023-11-02T18:52:37.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021589dbbfc5571d8-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.146 DEBUG:boto:path=/ibtljn-61/obj111 2023-11-02T18:52:37.146 DEBUG:boto:auth_path=/ibtljn-61/obj111 2023-11-02T18:52:37.147 DEBUG:boto:path=/ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.147 DEBUG:boto:auth_path=/ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.147 DEBUG:boto:Method: GET 2023-11-02T18:52:37.148 DEBUG:boto:Path: /ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.148 DEBUG:boto:Data: 2023-11-02T18:52:37.149 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.149 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.150 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.150 DEBUG:boto:Params: {} 2023-11-02T18:52:37.151 DEBUG:boto:Token: None 2023-11-02T18:52:37.152 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.152 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hKTUulJQe8OaYDUIXzgQrhdfFcs= 2023-11-02T18:52:37.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hKTUulJQe8OaYDUIXzgQrhdfFcs=', 'Content-Length': '0'} 2023-11-02T18:52:37.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f91f9335ab790148-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.198 DEBUG:boto:path=/ibtljn-61/obj111 2023-11-02T18:52:37.199 DEBUG:boto:auth_path=/ibtljn-61/obj111 2023-11-02T18:52:37.199 DEBUG:boto:path=/ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.200 DEBUG:boto:auth_path=/ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.200 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.200 DEBUG:boto:Path: /ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.201 DEBUG:boto:Data: 2023-11-02T18:52:37.201 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.202 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.203 DEBUG:boto:Params: {} 2023-11-02T18:52:37.203 DEBUG:boto:Token: None 2023-11-02T18:52:37.204 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.204 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W9SrYnvLux0iHuBCdEwl+TlQlN4= 2023-11-02T18:52:37.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W9SrYnvLux0iHuBCdEwl+TlQlN4=', 'Content-Length': '0'} 2023-11-02T18:52:37.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9edbafb456a3fa3-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.208 DEBUG:boto:path=/ibtljn-61/obj111 2023-11-02T18:52:37.209 DEBUG:boto:auth_path=/ibtljn-61/obj111 2023-11-02T18:52:37.209 DEBUG:boto:path=/ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.210 DEBUG:boto:auth_path=/ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.210 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.211 DEBUG:boto:Path: /ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.211 DEBUG:boto:Data: 2023-11-02T18:52:37.211 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.212 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.212 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.213 DEBUG:boto:Params: {} 2023-11-02T18:52:37.213 DEBUG:boto:Token: None 2023-11-02T18:52:37.214 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.214 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W9SrYnvLux0iHuBCdEwl+TlQlN4= 2023-11-02T18:52:37.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W9SrYnvLux0iHuBCdEwl+TlQlN4=', 'Content-Length': '0'} 2023-11-02T18:52:37.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eae09bed54bb5e36-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-11-02T18:52:37.219 DEBUG:boto:path=/ibtljn-61/obj111 2023-11-02T18:52:37.220 DEBUG:boto:auth_path=/ibtljn-61/obj111 2023-11-02T18:52:37.220 DEBUG:boto:path=/ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.221 DEBUG:boto:auth_path=/ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.221 DEBUG:boto:Method: GET 2023-11-02T18:52:37.222 DEBUG:boto:Path: /ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.222 DEBUG:boto:Data: 2023-11-02T18:52:37.223 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.224 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.224 DEBUG:boto:Params: {} 2023-11-02T18:52:37.224 DEBUG:boto:Token: None 2023-11-02T18:52:37.225 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hKTUulJQe8OaYDUIXzgQrhdfFcs= 2023-11-02T18:52:37.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hKTUulJQe8OaYDUIXzgQrhdfFcs=', 'Content-Length': '0'} 2023-11-02T18:52:37.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053dca3f9a5b16f3c-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.274 DEBUG:boto:path=/ibtljn-61/obj111 2023-11-02T18:52:37.275 DEBUG:boto:auth_path=/ibtljn-61/obj111 2023-11-02T18:52:37.276 DEBUG:boto:path=/ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.276 DEBUG:boto:auth_path=/ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.277 DEBUG:boto:Method: GET 2023-11-02T18:52:37.277 DEBUG:boto:Path: /ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.278 DEBUG:boto:Data: 2023-11-02T18:52:37.278 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.279 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.279 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.280 DEBUG:boto:Params: {} 2023-11-02T18:52:37.280 DEBUG:boto:Token: None 2023-11-02T18:52:37.281 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj111?versionId=null 2023-11-02T18:52:37.281 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hKTUulJQe8OaYDUIXzgQrhdfFcs= 2023-11-02T18:52:37.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hKTUulJQe8OaYDUIXzgQrhdfFcs=', 'Content-Length': '0'} 2023-11-02T18:52:37.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025cddfab46f7d280-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.285 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.286 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.286 DEBUG:boto:path=/ibtljn-61/?prefix=obj111 2023-11-02T18:52:37.287 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj111 2023-11-02T18:52:37.287 DEBUG:boto:Method: GET 2023-11-02T18:52:37.288 DEBUG:boto:Path: /ibtljn-61/?prefix=obj111 2023-11-02T18:52:37.288 DEBUG:boto:Data: 2023-11-02T18:52:37.289 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.289 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.290 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.290 DEBUG:boto:Params: {} 2023-11-02T18:52:37.291 DEBUG:boto:Token: None 2023-11-02T18:52:37.291 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.292 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038d895ee4355200c-006543eff5-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.300 DEBUG:boto:b'ibtljn-61obj1111000falseobj1112023-11-02T18:50:25.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.300 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.301 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.301 DEBUG:boto:path=/ibtljn-61/?prefix=obj111 2023-11-02T18:52:37.302 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj111 2023-11-02T18:52:37.303 DEBUG:boto:Method: GET 2023-11-02T18:52:37.303 DEBUG:boto:Path: /ibtljn-61/?prefix=obj111 2023-11-02T18:52:37.304 DEBUG:boto:Data: 2023-11-02T18:52:37.304 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.305 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.305 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.306 DEBUG:boto:Params: {} 2023-11-02T18:52:37.306 DEBUG:boto:Token: None 2023-11-02T18:52:37.307 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.307 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000669982e1365aa985-006543eff5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.318 DEBUG:boto:b'ibtljn-61obj1111000falseobj1112023-11-02T18:50:25.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.319 DEBUG:boto:path=/ibtljn-61/obj111 2023-11-02T18:52:37.319 DEBUG:boto:auth_path=/ibtljn-61/obj111 2023-11-02T18:52:37.320 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.320 DEBUG:boto:Path: /ibtljn-61/obj111 2023-11-02T18:52:37.321 DEBUG:boto:Data: 2023-11-02T18:52:37.321 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.322 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.322 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.323 DEBUG:boto:Params: {} 2023-11-02T18:52:37.323 DEBUG:boto:Token: None 2023-11-02T18:52:37.324 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj111 2023-11-02T18:52:37.324 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rfSMCugb2MtaXfI8Psrm8RN0E1s= 2023-11-02T18:52:37.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rfSMCugb2MtaXfI8Psrm8RN0E1s=', 'Content-Length': '0'} 2023-11-02T18:52:37.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb0631819b235724-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.328 DEBUG:boto:path=/ibtljn-61/obj111 2023-11-02T18:52:37.328 DEBUG:boto:auth_path=/ibtljn-61/obj111 2023-11-02T18:52:37.329 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.329 DEBUG:boto:Path: /ibtljn-61/obj111 2023-11-02T18:52:37.330 DEBUG:boto:Data: 2023-11-02T18:52:37.330 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.331 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.331 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.332 DEBUG:boto:Params: {} 2023-11-02T18:52:37.332 DEBUG:boto:Token: None 2023-11-02T18:52:37.333 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj111 2023-11-02T18:52:37.333 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rfSMCugb2MtaXfI8Psrm8RN0E1s= 2023-11-02T18:52:37.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rfSMCugb2MtaXfI8Psrm8RN0E1s=', 'Content-Length': '0'} 2023-11-02T18:52:37.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7b9717280ac32d5-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-11-02T18:52:37.338 DEBUG:boto:path=/ibtljn-61/obj111 2023-11-02T18:52:37.338 DEBUG:boto:auth_path=/ibtljn-61/obj111 2023-11-02T18:52:37.339 DEBUG:boto:Method: GET 2023-11-02T18:52:37.339 DEBUG:boto:Path: /ibtljn-61/obj111 2023-11-02T18:52:37.340 DEBUG:boto:Data: 2023-11-02T18:52:37.340 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.341 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.341 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.342 DEBUG:boto:Params: {} 2023-11-02T18:52:37.342 DEBUG:boto:Token: None 2023-11-02T18:52:37.343 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj111 2023-11-02T18:52:37.343 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8Ov0h+/kIajAB8HSOCDXz3fCBLA= 2023-11-02T18:52:37.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8Ov0h+/kIajAB8HSOCDXz3fCBLA=', 'Content-Length': '0'} 2023-11-02T18:52:37.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007da14b1f5a129eb6-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.394 DEBUG:boto:path=/ibtljn-61/obj111 2023-11-02T18:52:37.395 DEBUG:boto:auth_path=/ibtljn-61/obj111 2023-11-02T18:52:37.395 DEBUG:boto:Method: GET 2023-11-02T18:52:37.396 DEBUG:boto:Path: /ibtljn-61/obj111 2023-11-02T18:52:37.396 DEBUG:boto:Data: 2023-11-02T18:52:37.397 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.397 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.398 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.398 DEBUG:boto:Params: {} 2023-11-02T18:52:37.399 DEBUG:boto:Token: None 2023-11-02T18:52:37.399 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj111 2023-11-02T18:52:37.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8Ov0h+/kIajAB8HSOCDXz3fCBLA= 2023-11-02T18:52:37.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8Ov0h+/kIajAB8HSOCDXz3fCBLA=', 'Content-Length': '0'} 2023-11-02T18:52:37.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c87cb4cec4954d53-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.404 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.405 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.405 DEBUG:boto:path=/ibtljn-61/?prefix=obj111 2023-11-02T18:52:37.406 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj111 2023-11-02T18:52:37.406 DEBUG:boto:Method: GET 2023-11-02T18:52:37.406 DEBUG:boto:Path: /ibtljn-61/?prefix=obj111 2023-11-02T18:52:37.407 DEBUG:boto:Data: 2023-11-02T18:52:37.407 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.408 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.409 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.409 DEBUG:boto:Params: {} 2023-11-02T18:52:37.410 DEBUG:boto:Token: None 2023-11-02T18:52:37.410 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.411 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d684e77104dd2631-006543eff5-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.418 DEBUG:boto:b'ibtljn-61obj1111000falseobj1112023-11-02T18:50:25.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.418 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.419 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.419 DEBUG:boto:path=/ibtljn-61/?prefix=obj111 2023-11-02T18:52:37.420 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj111 2023-11-02T18:52:37.420 DEBUG:boto:Method: GET 2023-11-02T18:52:37.421 DEBUG:boto:Path: /ibtljn-61/?prefix=obj111 2023-11-02T18:52:37.421 DEBUG:boto:Data: 2023-11-02T18:52:37.421 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.422 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.423 DEBUG:boto:Params: {} 2023-11-02T18:52:37.423 DEBUG:boto:Token: None 2023-11-02T18:52:37.424 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.424 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c826d8bb3c25b4d8-006543eff5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.478 DEBUG:boto:b'ibtljn-61obj1111000falseobj1112023-11-02T18:50:25.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:52:37.479 DEBUG:boto:path=/ibtljn-61/obj115 2023-11-02T18:52:37.480 DEBUG:boto:auth_path=/ibtljn-61/obj115 2023-11-02T18:52:37.480 DEBUG:boto:path=/ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.481 DEBUG:boto:auth_path=/ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.481 DEBUG:boto:Method: GET 2023-11-02T18:52:37.481 DEBUG:boto:Path: /ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.482 DEBUG:boto:Data: 2023-11-02T18:52:37.482 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.483 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.483 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.484 DEBUG:boto:Params: {} 2023-11-02T18:52:37.484 DEBUG:boto:Token: None 2023-11-02T18:52:37.485 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.485 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e/nr/9BvYj1WUtfc2ZtP+6ZRzTA= 2023-11-02T18:52:37.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e/nr/9BvYj1WUtfc2ZtP+6ZRzTA=', 'Content-Length': '0'} 2023-11-02T18:52:37.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa417f81db323469-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.489 DEBUG:boto:path=/ibtljn-61/obj115 2023-11-02T18:52:37.490 DEBUG:boto:auth_path=/ibtljn-61/obj115 2023-11-02T18:52:37.490 DEBUG:boto:path=/ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.491 DEBUG:boto:auth_path=/ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.491 DEBUG:boto:Method: GET 2023-11-02T18:52:37.492 DEBUG:boto:Path: /ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.492 DEBUG:boto:Data: 2023-11-02T18:52:37.492 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.493 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.494 DEBUG:boto:Params: {} 2023-11-02T18:52:37.494 DEBUG:boto:Token: None 2023-11-02T18:52:37.495 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.495 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e/nr/9BvYj1WUtfc2ZtP+6ZRzTA= 2023-11-02T18:52:37.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e/nr/9BvYj1WUtfc2ZtP+6ZRzTA=', 'Content-Length': '0'} 2023-11-02T18:52:37.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e3b793ecc3cfbee-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.500 DEBUG:boto:path=/ibtljn-61/obj115 2023-11-02T18:52:37.500 DEBUG:boto:auth_path=/ibtljn-61/obj115 2023-11-02T18:52:37.501 DEBUG:boto:path=/ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.501 DEBUG:boto:auth_path=/ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.501 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.502 DEBUG:boto:Path: /ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.502 DEBUG:boto:Data: 2023-11-02T18:52:37.503 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.503 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.504 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.504 DEBUG:boto:Params: {} 2023-11-02T18:52:37.504 DEBUG:boto:Token: None 2023-11-02T18:52:37.505 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.505 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V82JmQ4u7FwfUBx5+aavcBDS6+c= 2023-11-02T18:52:37.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V82JmQ4u7FwfUBx5+aavcBDS6+c=', 'Content-Length': '0'} 2023-11-02T18:52:37.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f055feb17a5592c-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.509 DEBUG:boto:path=/ibtljn-61/obj115 2023-11-02T18:52:37.510 DEBUG:boto:auth_path=/ibtljn-61/obj115 2023-11-02T18:52:37.510 DEBUG:boto:path=/ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.511 DEBUG:boto:auth_path=/ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.511 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.511 DEBUG:boto:Path: /ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.512 DEBUG:boto:Data: 2023-11-02T18:52:37.512 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.513 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.513 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.513 DEBUG:boto:Params: {} 2023-11-02T18:52:37.514 DEBUG:boto:Token: None 2023-11-02T18:52:37.514 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.515 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V82JmQ4u7FwfUBx5+aavcBDS6+c= 2023-11-02T18:52:37.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V82JmQ4u7FwfUBx5+aavcBDS6+c=', 'Content-Length': '0'} 2023-11-02T18:52:37.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdeee426a5d0b849-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:52:37.519 DEBUG:boto:path=/ibtljn-61/obj115 2023-11-02T18:52:37.520 DEBUG:boto:auth_path=/ibtljn-61/obj115 2023-11-02T18:52:37.520 DEBUG:boto:path=/ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.520 DEBUG:boto:auth_path=/ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.521 DEBUG:boto:Method: GET 2023-11-02T18:52:37.521 DEBUG:boto:Path: /ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.522 DEBUG:boto:Data: 2023-11-02T18:52:37.522 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.523 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.523 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.523 DEBUG:boto:Params: {} 2023-11-02T18:52:37.524 DEBUG:boto:Token: None 2023-11-02T18:52:37.524 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.525 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e/nr/9BvYj1WUtfc2ZtP+6ZRzTA= 2023-11-02T18:52:37.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e/nr/9BvYj1WUtfc2ZtP+6ZRzTA=', 'Content-Length': '0'} 2023-11-02T18:52:37.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbdcbaff4182a0d6-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.529 DEBUG:boto:path=/ibtljn-61/obj115 2023-11-02T18:52:37.529 DEBUG:boto:auth_path=/ibtljn-61/obj115 2023-11-02T18:52:37.530 DEBUG:boto:path=/ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.530 DEBUG:boto:auth_path=/ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.531 DEBUG:boto:Method: GET 2023-11-02T18:52:37.531 DEBUG:boto:Path: /ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.531 DEBUG:boto:Data: 2023-11-02T18:52:37.532 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.533 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.533 DEBUG:boto:Params: {} 2023-11-02T18:52:37.534 DEBUG:boto:Token: None 2023-11-02T18:52:37.534 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj115?versionId=null 2023-11-02T18:52:37.535 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e/nr/9BvYj1WUtfc2ZtP+6ZRzTA= 2023-11-02T18:52:37.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e/nr/9BvYj1WUtfc2ZtP+6ZRzTA=', 'Content-Length': '0'} 2023-11-02T18:52:37.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047fbf4dc3d22ac06-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.539 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.539 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.540 DEBUG:boto:path=/ibtljn-61/?prefix=obj115 2023-11-02T18:52:37.540 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj115 2023-11-02T18:52:37.541 DEBUG:boto:Method: GET 2023-11-02T18:52:37.541 DEBUG:boto:Path: /ibtljn-61/?prefix=obj115 2023-11-02T18:52:37.542 DEBUG:boto:Data: 2023-11-02T18:52:37.542 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.542 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.543 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.543 DEBUG:boto:Params: {} 2023-11-02T18:52:37.544 DEBUG:boto:Token: None 2023-11-02T18:52:37.544 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.545 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dfbdc944cae834d-006543eff5-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.551 DEBUG:boto:b'ibtljn-61obj1151000falseobj1152023-11-02T18:50:23.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.552 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.553 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.553 DEBUG:boto:path=/ibtljn-61/?prefix=obj115 2023-11-02T18:52:37.553 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj115 2023-11-02T18:52:37.554 DEBUG:boto:Method: GET 2023-11-02T18:52:37.555 DEBUG:boto:Path: /ibtljn-61/?prefix=obj115 2023-11-02T18:52:37.555 DEBUG:boto:Data: 2023-11-02T18:52:37.556 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.556 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.557 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.557 DEBUG:boto:Params: {} 2023-11-02T18:52:37.557 DEBUG:boto:Token: None 2023-11-02T18:52:37.558 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.559 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065eac55298d7f221-006543eff5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.610 DEBUG:boto:b'ibtljn-61obj1151000falseobj1152023-11-02T18:50:23.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.611 DEBUG:boto:path=/ibtljn-61/obj115 2023-11-02T18:52:37.612 DEBUG:boto:auth_path=/ibtljn-61/obj115 2023-11-02T18:52:37.612 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.613 DEBUG:boto:Path: /ibtljn-61/obj115 2023-11-02T18:52:37.613 DEBUG:boto:Data: 2023-11-02T18:52:37.613 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.614 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.614 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.615 DEBUG:boto:Params: {} 2023-11-02T18:52:37.615 DEBUG:boto:Token: None 2023-11-02T18:52:37.616 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj115 2023-11-02T18:52:37.616 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CuFKAAlQn4zay8Frg4jm0IuwGVg= 2023-11-02T18:52:37.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CuFKAAlQn4zay8Frg4jm0IuwGVg=', 'Content-Length': '0'} 2023-11-02T18:52:37.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b68ba05237d2bd-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.620 DEBUG:boto:path=/ibtljn-61/obj115 2023-11-02T18:52:37.621 DEBUG:boto:auth_path=/ibtljn-61/obj115 2023-11-02T18:52:37.621 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.622 DEBUG:boto:Path: /ibtljn-61/obj115 2023-11-02T18:52:37.622 DEBUG:boto:Data: 2023-11-02T18:52:37.622 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.623 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.623 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.624 DEBUG:boto:Params: {} 2023-11-02T18:52:37.624 DEBUG:boto:Token: None 2023-11-02T18:52:37.625 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj115 2023-11-02T18:52:37.625 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CuFKAAlQn4zay8Frg4jm0IuwGVg= 2023-11-02T18:52:37.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CuFKAAlQn4zay8Frg4jm0IuwGVg=', 'Content-Length': '0'} 2023-11-02T18:52:37.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f0f391244d06d9-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T18:52:37.630 DEBUG:boto:path=/ibtljn-61/obj115 2023-11-02T18:52:37.630 DEBUG:boto:auth_path=/ibtljn-61/obj115 2023-11-02T18:52:37.631 DEBUG:boto:Method: GET 2023-11-02T18:52:37.631 DEBUG:boto:Path: /ibtljn-61/obj115 2023-11-02T18:52:37.631 DEBUG:boto:Data: 2023-11-02T18:52:37.632 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.632 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.633 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.633 DEBUG:boto:Params: {} 2023-11-02T18:52:37.634 DEBUG:boto:Token: None 2023-11-02T18:52:37.634 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj115 2023-11-02T18:52:37.635 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yi1e7hn7EmcLIDYCDWoma40OsEY= 2023-11-02T18:52:37.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yi1e7hn7EmcLIDYCDWoma40OsEY=', 'Content-Length': '0'} 2023-11-02T18:52:37.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3eff23d4c554aa1-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.682 DEBUG:boto:path=/ibtljn-61/obj115 2023-11-02T18:52:37.683 DEBUG:boto:auth_path=/ibtljn-61/obj115 2023-11-02T18:52:37.684 DEBUG:boto:Method: GET 2023-11-02T18:52:37.684 DEBUG:boto:Path: /ibtljn-61/obj115 2023-11-02T18:52:37.685 DEBUG:boto:Data: 2023-11-02T18:52:37.685 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.686 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.686 DEBUG:boto:Params: {} 2023-11-02T18:52:37.687 DEBUG:boto:Token: None 2023-11-02T18:52:37.687 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj115 2023-11-02T18:52:37.688 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yi1e7hn7EmcLIDYCDWoma40OsEY= 2023-11-02T18:52:37.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yi1e7hn7EmcLIDYCDWoma40OsEY=', 'Content-Length': '0'} 2023-11-02T18:52:37.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b5a0a96bb8e01e-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.693 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.693 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.694 DEBUG:boto:path=/ibtljn-61/?prefix=obj115 2023-11-02T18:52:37.694 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj115 2023-11-02T18:52:37.694 DEBUG:boto:Method: GET 2023-11-02T18:52:37.695 DEBUG:boto:Path: /ibtljn-61/?prefix=obj115 2023-11-02T18:52:37.695 DEBUG:boto:Data: 2023-11-02T18:52:37.696 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.696 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.697 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.697 DEBUG:boto:Params: {} 2023-11-02T18:52:37.698 DEBUG:boto:Token: None 2023-11-02T18:52:37.698 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.700 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003047ddc0bb404ff6-006543eff5-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.707 DEBUG:boto:b'ibtljn-61obj1151000falseobj1152023-11-02T18:50:23.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.707 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.708 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.708 DEBUG:boto:path=/ibtljn-61/?prefix=obj115 2023-11-02T18:52:37.709 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj115 2023-11-02T18:52:37.709 DEBUG:boto:Method: GET 2023-11-02T18:52:37.710 DEBUG:boto:Path: /ibtljn-61/?prefix=obj115 2023-11-02T18:52:37.710 DEBUG:boto:Data: 2023-11-02T18:52:37.711 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.711 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.712 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.712 DEBUG:boto:Params: {} 2023-11-02T18:52:37.713 DEBUG:boto:Token: None 2023-11-02T18:52:37.713 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.714 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fd52e192a387d4d-006543eff5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.725 DEBUG:boto:b'ibtljn-61obj1151000falseobj1152023-11-02T18:50:23.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:52:37.726 DEBUG:boto:path=/ibtljn-61/obj116 2023-11-02T18:52:37.727 DEBUG:boto:auth_path=/ibtljn-61/obj116 2023-11-02T18:52:37.727 DEBUG:boto:path=/ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.727 DEBUG:boto:auth_path=/ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.728 DEBUG:boto:Method: GET 2023-11-02T18:52:37.729 DEBUG:boto:Path: /ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.729 DEBUG:boto:Data: 2023-11-02T18:52:37.729 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.730 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.730 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.731 DEBUG:boto:Params: {} 2023-11-02T18:52:37.732 DEBUG:boto:Token: None 2023-11-02T18:52:37.732 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.733 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p9/7vkqmU1/mHrB7sQ13Na7Rg9A= 2023-11-02T18:52:37.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p9/7vkqmU1/mHrB7sQ13Na7Rg9A=', 'Content-Length': '0'} 2023-11-02T18:52:37.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1536b98922f7a5f-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.736 DEBUG:boto:path=/ibtljn-61/obj116 2023-11-02T18:52:37.737 DEBUG:boto:auth_path=/ibtljn-61/obj116 2023-11-02T18:52:37.737 DEBUG:boto:path=/ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.738 DEBUG:boto:auth_path=/ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.738 DEBUG:boto:Method: GET 2023-11-02T18:52:37.739 DEBUG:boto:Path: /ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.739 DEBUG:boto:Data: 2023-11-02T18:52:37.739 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.740 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.740 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.741 DEBUG:boto:Params: {} 2023-11-02T18:52:37.741 DEBUG:boto:Token: None 2023-11-02T18:52:37.742 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.742 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p9/7vkqmU1/mHrB7sQ13Na7Rg9A= 2023-11-02T18:52:37.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p9/7vkqmU1/mHrB7sQ13Na7Rg9A=', 'Content-Length': '0'} 2023-11-02T18:52:37.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000082855da7f97eaad-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.786 DEBUG:boto:path=/ibtljn-61/obj116 2023-11-02T18:52:37.788 DEBUG:boto:auth_path=/ibtljn-61/obj116 2023-11-02T18:52:37.788 DEBUG:boto:path=/ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.789 DEBUG:boto:auth_path=/ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.789 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.789 DEBUG:boto:Path: /ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.790 DEBUG:boto:Data: 2023-11-02T18:52:37.790 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.791 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.791 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.791 DEBUG:boto:Params: {} 2023-11-02T18:52:37.792 DEBUG:boto:Token: None 2023-11-02T18:52:37.792 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FDFcZlkQX7T3LCRD/8fwOznNMlw= 2023-11-02T18:52:37.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FDFcZlkQX7T3LCRD/8fwOznNMlw=', 'Content-Length': '0'} 2023-11-02T18:52:37.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b83809646d03eeb3-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.797 DEBUG:boto:path=/ibtljn-61/obj116 2023-11-02T18:52:37.798 DEBUG:boto:auth_path=/ibtljn-61/obj116 2023-11-02T18:52:37.798 DEBUG:boto:path=/ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.799 DEBUG:boto:auth_path=/ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.799 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.799 DEBUG:boto:Path: /ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.800 DEBUG:boto:Data: 2023-11-02T18:52:37.800 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.801 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.801 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.802 DEBUG:boto:Params: {} 2023-11-02T18:52:37.802 DEBUG:boto:Token: None 2023-11-02T18:52:37.803 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.803 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FDFcZlkQX7T3LCRD/8fwOznNMlw= 2023-11-02T18:52:37.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FDFcZlkQX7T3LCRD/8fwOznNMlw=', 'Content-Length': '0'} 2023-11-02T18:52:37.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041a9102f7e28462d-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:52:37.807 DEBUG:boto:path=/ibtljn-61/obj116 2023-11-02T18:52:37.808 DEBUG:boto:auth_path=/ibtljn-61/obj116 2023-11-02T18:52:37.808 DEBUG:boto:path=/ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.809 DEBUG:boto:auth_path=/ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.809 DEBUG:boto:Method: GET 2023-11-02T18:52:37.810 DEBUG:boto:Path: /ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.810 DEBUG:boto:Data: 2023-11-02T18:52:37.810 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.811 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.811 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.812 DEBUG:boto:Params: {} 2023-11-02T18:52:37.812 DEBUG:boto:Token: None 2023-11-02T18:52:37.813 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.813 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p9/7vkqmU1/mHrB7sQ13Na7Rg9A= 2023-11-02T18:52:37.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p9/7vkqmU1/mHrB7sQ13Na7Rg9A=', 'Content-Length': '0'} 2023-11-02T18:52:37.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000109dbc1a04da0035-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.818 DEBUG:boto:path=/ibtljn-61/obj116 2023-11-02T18:52:37.818 DEBUG:boto:auth_path=/ibtljn-61/obj116 2023-11-02T18:52:37.819 DEBUG:boto:path=/ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.819 DEBUG:boto:auth_path=/ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.820 DEBUG:boto:Method: GET 2023-11-02T18:52:37.820 DEBUG:boto:Path: /ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.820 DEBUG:boto:Data: 2023-11-02T18:52:37.821 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.821 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.822 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.822 DEBUG:boto:Params: {} 2023-11-02T18:52:37.823 DEBUG:boto:Token: None 2023-11-02T18:52:37.823 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj116?versionId=null 2023-11-02T18:52:37.824 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p9/7vkqmU1/mHrB7sQ13Na7Rg9A= 2023-11-02T18:52:37.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p9/7vkqmU1/mHrB7sQ13Na7Rg9A=', 'Content-Length': '0'} 2023-11-02T18:52:37.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d0d7e7e6da1fd93-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.828 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.828 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.829 DEBUG:boto:path=/ibtljn-61/?prefix=obj116 2023-11-02T18:52:37.829 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj116 2023-11-02T18:52:37.830 DEBUG:boto:Method: GET 2023-11-02T18:52:37.830 DEBUG:boto:Path: /ibtljn-61/?prefix=obj116 2023-11-02T18:52:37.831 DEBUG:boto:Data: 2023-11-02T18:52:37.831 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.832 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.832 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.832 DEBUG:boto:Params: {} 2023-11-02T18:52:37.833 DEBUG:boto:Token: None 2023-11-02T18:52:37.833 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.834 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074d487a606dae810-006543eff5-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.841 DEBUG:boto:b'ibtljn-61obj1161000falseobj1162023-11-02T18:50:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.842 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.842 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.842 DEBUG:boto:path=/ibtljn-61/?prefix=obj116 2023-11-02T18:52:37.843 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj116 2023-11-02T18:52:37.843 DEBUG:boto:Method: GET 2023-11-02T18:52:37.844 DEBUG:boto:Path: /ibtljn-61/?prefix=obj116 2023-11-02T18:52:37.844 DEBUG:boto:Data: 2023-11-02T18:52:37.845 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.845 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.846 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.846 DEBUG:boto:Params: {} 2023-11-02T18:52:37.847 DEBUG:boto:Token: None 2023-11-02T18:52:37.847 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.848 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d701128db2d17413-006543eff5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.902 DEBUG:boto:b'ibtljn-61obj1161000falseobj1162023-11-02T18:50:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.903 DEBUG:boto:path=/ibtljn-61/obj116 2023-11-02T18:52:37.904 DEBUG:boto:auth_path=/ibtljn-61/obj116 2023-11-02T18:52:37.904 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.905 DEBUG:boto:Path: /ibtljn-61/obj116 2023-11-02T18:52:37.905 DEBUG:boto:Data: 2023-11-02T18:52:37.905 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.906 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.907 DEBUG:boto:Params: {} 2023-11-02T18:52:37.907 DEBUG:boto:Token: None 2023-11-02T18:52:37.908 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj116 2023-11-02T18:52:37.909 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0n8842VRMVDMNx8zf3JfRfzSpkY= 2023-11-02T18:52:37.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0n8842VRMVDMNx8zf3JfRfzSpkY=', 'Content-Length': '0'} 2023-11-02T18:52:37.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b3188b1956eb34-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.913 DEBUG:boto:path=/ibtljn-61/obj116 2023-11-02T18:52:37.914 DEBUG:boto:auth_path=/ibtljn-61/obj116 2023-11-02T18:52:37.914 DEBUG:boto:Method: HEAD 2023-11-02T18:52:37.914 DEBUG:boto:Path: /ibtljn-61/obj116 2023-11-02T18:52:37.915 DEBUG:boto:Data: 2023-11-02T18:52:37.916 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.916 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.917 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.917 DEBUG:boto:Params: {} 2023-11-02T18:52:37.918 DEBUG:boto:Token: None 2023-11-02T18:52:37.918 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj116 2023-11-02T18:52:37.919 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0n8842VRMVDMNx8zf3JfRfzSpkY= 2023-11-02T18:52:37.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0n8842VRMVDMNx8zf3JfRfzSpkY=', 'Content-Length': '0'} 2023-11-02T18:52:37.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037f3fccf86e5ff2b-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T18:52:37.923 DEBUG:boto:path=/ibtljn-61/obj116 2023-11-02T18:52:37.924 DEBUG:boto:auth_path=/ibtljn-61/obj116 2023-11-02T18:52:37.924 DEBUG:boto:Method: GET 2023-11-02T18:52:37.925 DEBUG:boto:Path: /ibtljn-61/obj116 2023-11-02T18:52:37.925 DEBUG:boto:Data: 2023-11-02T18:52:37.925 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.926 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.926 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.927 DEBUG:boto:Params: {} 2023-11-02T18:52:37.927 DEBUG:boto:Token: None 2023-11-02T18:52:37.928 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj116 2023-11-02T18:52:37.928 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aeExBvrPRZ1Qz+mrx1bSwFMG0Us= 2023-11-02T18:52:37.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aeExBvrPRZ1Qz+mrx1bSwFMG0Us=', 'Content-Length': '0'} 2023-11-02T18:52:37.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c44fd7699ba9fa-006543eff5-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.933 DEBUG:boto:path=/ibtljn-61/obj116 2023-11-02T18:52:37.933 DEBUG:boto:auth_path=/ibtljn-61/obj116 2023-11-02T18:52:37.934 DEBUG:boto:Method: GET 2023-11-02T18:52:37.934 DEBUG:boto:Path: /ibtljn-61/obj116 2023-11-02T18:52:37.934 DEBUG:boto:Data: 2023-11-02T18:52:37.935 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.935 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:37.936 DEBUG:boto:Port: 8000 2023-11-02T18:52:37.936 DEBUG:boto:Params: {} 2023-11-02T18:52:37.937 DEBUG:boto:Token: None 2023-11-02T18:52:37.937 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/obj116 2023-11-02T18:52:37.938 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aeExBvrPRZ1Qz+mrx1bSwFMG0Us= 2023-11-02T18:52:37.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aeExBvrPRZ1Qz+mrx1bSwFMG0Us=', 'Content-Length': '0'} 2023-11-02T18:52:37.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f33fbb4e6d491bd6-006543eff5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.942 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:37.942 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:37.943 DEBUG:boto:path=/ibtljn-61/?prefix=obj116 2023-11-02T18:52:37.943 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj116 2023-11-02T18:52:37.944 DEBUG:boto:Method: GET 2023-11-02T18:52:37.944 DEBUG:boto:Path: /ibtljn-61/?prefix=obj116 2023-11-02T18:52:37.945 DEBUG:boto:Data: 2023-11-02T18:52:37.945 DEBUG:boto:Headers: {} 2023-11-02T18:52:37.945 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:37.946 DEBUG:boto:Port: 8001 2023-11-02T18:52:37.946 DEBUG:boto:Params: {} 2023-11-02T18:52:37.947 DEBUG:boto:Token: None 2023-11-02T18:52:37.947 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:37 GMT /ibtljn-61/ 2023-11-02T18:52:37.948 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI= 2023-11-02T18:52:37.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bvrT+k5uhxVH8av53eE5SrIH4eI=', 'Content-Length': '0'} 2023-11-02T18:52:37.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffe29b32ecefce59-006543eff5-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:37.998 DEBUG:boto:b'ibtljn-61obj1161000falseobj1162023-11-02T18:50:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:37.999 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.000 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.000 DEBUG:boto:path=/ibtljn-61/?prefix=obj116 2023-11-02T18:52:38.001 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj116 2023-11-02T18:52:38.001 DEBUG:boto:Method: GET 2023-11-02T18:52:38.002 DEBUG:boto:Path: /ibtljn-61/?prefix=obj116 2023-11-02T18:52:38.002 DEBUG:boto:Data: 2023-11-02T18:52:38.002 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.003 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.003 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.004 DEBUG:boto:Params: {} 2023-11-02T18:52:38.004 DEBUG:boto:Token: None 2023-11-02T18:52:38.005 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.005 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f4b3495e7a853e2-006543eff6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.017 DEBUG:boto:b'ibtljn-61obj1161000falseobj1162023-11-02T18:50:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T18:52:38.018 DEBUG:boto:path=/ibtljn-61/obj124 2023-11-02T18:52:38.018 DEBUG:boto:auth_path=/ibtljn-61/obj124 2023-11-02T18:52:38.019 DEBUG:boto:path=/ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.019 DEBUG:boto:auth_path=/ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.020 DEBUG:boto:Method: GET 2023-11-02T18:52:38.020 DEBUG:boto:Path: /ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.020 DEBUG:boto:Data: 2023-11-02T18:52:38.021 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.021 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.022 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.022 DEBUG:boto:Params: {} 2023-11-02T18:52:38.023 DEBUG:boto:Token: None 2023-11-02T18:52:38.023 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.024 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:avyS+pybJxeDcD0+wORh04TGGXc= 2023-11-02T18:52:38.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:avyS+pybJxeDcD0+wORh04TGGXc=', 'Content-Length': '0'} 2023-11-02T18:52:38.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000791946f870a171ca-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.027 DEBUG:boto:path=/ibtljn-61/obj124 2023-11-02T18:52:38.028 DEBUG:boto:auth_path=/ibtljn-61/obj124 2023-11-02T18:52:38.028 DEBUG:boto:path=/ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.028 DEBUG:boto:auth_path=/ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.029 DEBUG:boto:Method: GET 2023-11-02T18:52:38.029 DEBUG:boto:Path: /ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.030 DEBUG:boto:Data: 2023-11-02T18:52:38.030 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.031 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.031 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.032 DEBUG:boto:Params: {} 2023-11-02T18:52:38.032 DEBUG:boto:Token: None 2023-11-02T18:52:38.033 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.033 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:avyS+pybJxeDcD0+wORh04TGGXc= 2023-11-02T18:52:38.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:avyS+pybJxeDcD0+wORh04TGGXc=', 'Content-Length': '0'} 2023-11-02T18:52:38.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fd305168e9a4de2-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.037 DEBUG:boto:path=/ibtljn-61/obj124 2023-11-02T18:52:38.037 DEBUG:boto:auth_path=/ibtljn-61/obj124 2023-11-02T18:52:38.038 DEBUG:boto:path=/ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.038 DEBUG:boto:auth_path=/ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.039 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.039 DEBUG:boto:Path: /ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.040 DEBUG:boto:Data: 2023-11-02T18:52:38.040 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.040 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.041 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.041 DEBUG:boto:Params: {} 2023-11-02T18:52:38.042 DEBUG:boto:Token: None 2023-11-02T18:52:38.042 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.043 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5sDpeVaI/Gd7oep0qkilTwjphIQ= 2023-11-02T18:52:38.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5sDpeVaI/Gd7oep0qkilTwjphIQ=', 'Content-Length': '0'} 2023-11-02T18:52:38.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007cb77025eb6b638-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.046 DEBUG:boto:path=/ibtljn-61/obj124 2023-11-02T18:52:38.047 DEBUG:boto:auth_path=/ibtljn-61/obj124 2023-11-02T18:52:38.047 DEBUG:boto:path=/ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.048 DEBUG:boto:auth_path=/ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.048 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.049 DEBUG:boto:Path: /ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.049 DEBUG:boto:Data: 2023-11-02T18:52:38.049 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.050 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.051 DEBUG:boto:Params: {} 2023-11-02T18:52:38.051 DEBUG:boto:Token: None 2023-11-02T18:52:38.051 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.052 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5sDpeVaI/Gd7oep0qkilTwjphIQ= 2023-11-02T18:52:38.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5sDpeVaI/Gd7oep0qkilTwjphIQ=', 'Content-Length': '0'} 2023-11-02T18:52:38.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003438430c62937ff9-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T18:52:38.056 DEBUG:boto:path=/ibtljn-61/obj124 2023-11-02T18:52:38.057 DEBUG:boto:auth_path=/ibtljn-61/obj124 2023-11-02T18:52:38.057 DEBUG:boto:path=/ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.058 DEBUG:boto:auth_path=/ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.058 DEBUG:boto:Method: GET 2023-11-02T18:52:38.059 DEBUG:boto:Path: /ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.059 DEBUG:boto:Data: 2023-11-02T18:52:38.059 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.060 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.060 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.061 DEBUG:boto:Params: {} 2023-11-02T18:52:38.061 DEBUG:boto:Token: None 2023-11-02T18:52:38.062 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.062 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:avyS+pybJxeDcD0+wORh04TGGXc= 2023-11-02T18:52:38.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:avyS+pybJxeDcD0+wORh04TGGXc=', 'Content-Length': '0'} 2023-11-02T18:52:38.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007d95cf1ff64fda7-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.066 DEBUG:boto:path=/ibtljn-61/obj124 2023-11-02T18:52:38.067 DEBUG:boto:auth_path=/ibtljn-61/obj124 2023-11-02T18:52:38.067 DEBUG:boto:path=/ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.068 DEBUG:boto:auth_path=/ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.068 DEBUG:boto:Method: GET 2023-11-02T18:52:38.069 DEBUG:boto:Path: /ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.069 DEBUG:boto:Data: 2023-11-02T18:52:38.070 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.071 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.071 DEBUG:boto:Params: {} 2023-11-02T18:52:38.072 DEBUG:boto:Token: None 2023-11-02T18:52:38.072 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj124?versionId=null 2023-11-02T18:52:38.073 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:avyS+pybJxeDcD0+wORh04TGGXc= 2023-11-02T18:52:38.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:avyS+pybJxeDcD0+wORh04TGGXc=', 'Content-Length': '0'} 2023-11-02T18:52:38.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000476f21038405a561-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.122 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.123 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.123 DEBUG:boto:path=/ibtljn-61/?prefix=obj124 2023-11-02T18:52:38.124 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj124 2023-11-02T18:52:38.124 DEBUG:boto:Method: GET 2023-11-02T18:52:38.125 DEBUG:boto:Path: /ibtljn-61/?prefix=obj124 2023-11-02T18:52:38.125 DEBUG:boto:Data: 2023-11-02T18:52:38.126 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.126 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.127 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.127 DEBUG:boto:Params: {} 2023-11-02T18:52:38.128 DEBUG:boto:Token: None 2023-11-02T18:52:38.128 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005739338ffbecf7b0-006543eff6-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.135 DEBUG:boto:b'ibtljn-61obj1241000falseobj1242023-11-02T18:50:24.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.136 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.136 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.137 DEBUG:boto:path=/ibtljn-61/?prefix=obj124 2023-11-02T18:52:38.137 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj124 2023-11-02T18:52:38.138 DEBUG:boto:Method: GET 2023-11-02T18:52:38.138 DEBUG:boto:Path: /ibtljn-61/?prefix=obj124 2023-11-02T18:52:38.139 DEBUG:boto:Data: 2023-11-02T18:52:38.139 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.140 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.140 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.140 DEBUG:boto:Params: {} 2023-11-02T18:52:38.141 DEBUG:boto:Token: None 2023-11-02T18:52:38.141 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.142 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002609352634dc794b-006543eff6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.152 DEBUG:boto:b'ibtljn-61obj1241000falseobj1242023-11-02T18:50:24.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.153 DEBUG:boto:path=/ibtljn-61/obj124 2023-11-02T18:52:38.153 DEBUG:boto:auth_path=/ibtljn-61/obj124 2023-11-02T18:52:38.154 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.154 DEBUG:boto:Path: /ibtljn-61/obj124 2023-11-02T18:52:38.155 DEBUG:boto:Data: 2023-11-02T18:52:38.155 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.156 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.156 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.156 DEBUG:boto:Params: {} 2023-11-02T18:52:38.157 DEBUG:boto:Token: None 2023-11-02T18:52:38.157 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj124 2023-11-02T18:52:38.158 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ia5JoI0lvTaZCYtKyt6nzL+23VQ= 2023-11-02T18:52:38.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ia5JoI0lvTaZCYtKyt6nzL+23VQ=', 'Content-Length': '0'} 2023-11-02T18:52:38.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aec35f767e7065f-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.161 DEBUG:boto:path=/ibtljn-61/obj124 2023-11-02T18:52:38.162 DEBUG:boto:auth_path=/ibtljn-61/obj124 2023-11-02T18:52:38.162 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.162 DEBUG:boto:Path: /ibtljn-61/obj124 2023-11-02T18:52:38.163 DEBUG:boto:Data: 2023-11-02T18:52:38.163 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.164 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.164 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.165 DEBUG:boto:Params: {} 2023-11-02T18:52:38.165 DEBUG:boto:Token: None 2023-11-02T18:52:38.166 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj124 2023-11-02T18:52:38.166 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ia5JoI0lvTaZCYtKyt6nzL+23VQ= 2023-11-02T18:52:38.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ia5JoI0lvTaZCYtKyt6nzL+23VQ=', 'Content-Length': '0'} 2023-11-02T18:52:38.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000814c2c2c5aa198e2-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T18:52:38.170 DEBUG:boto:path=/ibtljn-61/obj124 2023-11-02T18:52:38.170 DEBUG:boto:auth_path=/ibtljn-61/obj124 2023-11-02T18:52:38.171 DEBUG:boto:Method: GET 2023-11-02T18:52:38.171 DEBUG:boto:Path: /ibtljn-61/obj124 2023-11-02T18:52:38.172 DEBUG:boto:Data: 2023-11-02T18:52:38.172 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.173 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.173 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.173 DEBUG:boto:Params: {} 2023-11-02T18:52:38.174 DEBUG:boto:Token: None 2023-11-02T18:52:38.174 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj124 2023-11-02T18:52:38.175 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dS4XHfP0C2O8VCdmjeXXKG6L4cY= 2023-11-02T18:52:38.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dS4XHfP0C2O8VCdmjeXXKG6L4cY=', 'Content-Length': '0'} 2023-11-02T18:52:38.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfb8bb3b1338b59c-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.218 DEBUG:boto:path=/ibtljn-61/obj124 2023-11-02T18:52:38.219 DEBUG:boto:auth_path=/ibtljn-61/obj124 2023-11-02T18:52:38.219 DEBUG:boto:Method: GET 2023-11-02T18:52:38.220 DEBUG:boto:Path: /ibtljn-61/obj124 2023-11-02T18:52:38.220 DEBUG:boto:Data: 2023-11-02T18:52:38.221 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.221 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.222 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.222 DEBUG:boto:Params: {} 2023-11-02T18:52:38.223 DEBUG:boto:Token: None 2023-11-02T18:52:38.223 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj124 2023-11-02T18:52:38.224 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dS4XHfP0C2O8VCdmjeXXKG6L4cY= 2023-11-02T18:52:38.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dS4XHfP0C2O8VCdmjeXXKG6L4cY=', 'Content-Length': '0'} 2023-11-02T18:52:38.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002573145136a9fc1f-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.228 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.228 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.229 DEBUG:boto:path=/ibtljn-61/?prefix=obj124 2023-11-02T18:52:38.229 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj124 2023-11-02T18:52:38.229 DEBUG:boto:Method: GET 2023-11-02T18:52:38.230 DEBUG:boto:Path: /ibtljn-61/?prefix=obj124 2023-11-02T18:52:38.230 DEBUG:boto:Data: 2023-11-02T18:52:38.231 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.231 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.232 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.232 DEBUG:boto:Params: {} 2023-11-02T18:52:38.233 DEBUG:boto:Token: None 2023-11-02T18:52:38.233 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.234 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f740218ec9b9b016-006543eff6-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.241 DEBUG:boto:b'ibtljn-61obj1241000falseobj1242023-11-02T18:50:24.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.242 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.242 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.243 DEBUG:boto:path=/ibtljn-61/?prefix=obj124 2023-11-02T18:52:38.243 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj124 2023-11-02T18:52:38.244 DEBUG:boto:Method: GET 2023-11-02T18:52:38.244 DEBUG:boto:Path: /ibtljn-61/?prefix=obj124 2023-11-02T18:52:38.245 DEBUG:boto:Data: 2023-11-02T18:52:38.245 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.245 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.246 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.246 DEBUG:boto:Params: {} 2023-11-02T18:52:38.247 DEBUG:boto:Token: None 2023-11-02T18:52:38.247 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.248 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005f3fe39939f30e0-006543eff6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.302 DEBUG:boto:b'ibtljn-61obj1241000falseobj1242023-11-02T18:50:24.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:52:38.304 DEBUG:boto:path=/ibtljn-61/obj145 2023-11-02T18:52:38.305 DEBUG:boto:auth_path=/ibtljn-61/obj145 2023-11-02T18:52:38.305 DEBUG:boto:path=/ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.305 DEBUG:boto:auth_path=/ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.306 DEBUG:boto:Method: GET 2023-11-02T18:52:38.306 DEBUG:boto:Path: /ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.307 DEBUG:boto:Data: 2023-11-02T18:52:38.307 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.308 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.308 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.309 DEBUG:boto:Params: {} 2023-11-02T18:52:38.309 DEBUG:boto:Token: None 2023-11-02T18:52:38.310 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.310 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5OrFpncy56fhKljuLCfnNyKtQPk= 2023-11-02T18:52:38.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5OrFpncy56fhKljuLCfnNyKtQPk=', 'Content-Length': '0'} 2023-11-02T18:52:38.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002eb9df769f1530a-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.315 DEBUG:boto:path=/ibtljn-61/obj145 2023-11-02T18:52:38.316 DEBUG:boto:auth_path=/ibtljn-61/obj145 2023-11-02T18:52:38.316 DEBUG:boto:path=/ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.316 DEBUG:boto:auth_path=/ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.317 DEBUG:boto:Method: GET 2023-11-02T18:52:38.317 DEBUG:boto:Path: /ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.318 DEBUG:boto:Data: 2023-11-02T18:52:38.318 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.319 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.319 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.320 DEBUG:boto:Params: {} 2023-11-02T18:52:38.320 DEBUG:boto:Token: None 2023-11-02T18:52:38.321 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.321 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5OrFpncy56fhKljuLCfnNyKtQPk= 2023-11-02T18:52:38.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5OrFpncy56fhKljuLCfnNyKtQPk=', 'Content-Length': '0'} 2023-11-02T18:52:38.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000311b3927201e2ff-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.326 DEBUG:boto:path=/ibtljn-61/obj145 2023-11-02T18:52:38.326 DEBUG:boto:auth_path=/ibtljn-61/obj145 2023-11-02T18:52:38.327 DEBUG:boto:path=/ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.327 DEBUG:boto:auth_path=/ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.328 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.328 DEBUG:boto:Path: /ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.328 DEBUG:boto:Data: 2023-11-02T18:52:38.329 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.329 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.330 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.330 DEBUG:boto:Params: {} 2023-11-02T18:52:38.331 DEBUG:boto:Token: None 2023-11-02T18:52:38.331 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.332 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g19U+s40S0pmQkqnXZeusuYwCCE= 2023-11-02T18:52:38.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g19U+s40S0pmQkqnXZeusuYwCCE=', 'Content-Length': '0'} 2023-11-02T18:52:38.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052f4370ff2c6764f-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.336 DEBUG:boto:path=/ibtljn-61/obj145 2023-11-02T18:52:38.337 DEBUG:boto:auth_path=/ibtljn-61/obj145 2023-11-02T18:52:38.337 DEBUG:boto:path=/ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.337 DEBUG:boto:auth_path=/ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.338 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.338 DEBUG:boto:Path: /ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.339 DEBUG:boto:Data: 2023-11-02T18:52:38.339 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.340 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.340 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.340 DEBUG:boto:Params: {} 2023-11-02T18:52:38.341 DEBUG:boto:Token: None 2023-11-02T18:52:38.342 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.342 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g19U+s40S0pmQkqnXZeusuYwCCE= 2023-11-02T18:52:38.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g19U+s40S0pmQkqnXZeusuYwCCE=', 'Content-Length': '0'} 2023-11-02T18:52:38.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e96a4427d99bab79-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:52:38.347 DEBUG:boto:path=/ibtljn-61/obj145 2023-11-02T18:52:38.347 DEBUG:boto:auth_path=/ibtljn-61/obj145 2023-11-02T18:52:38.347 DEBUG:boto:path=/ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.348 DEBUG:boto:auth_path=/ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.348 DEBUG:boto:Method: GET 2023-11-02T18:52:38.349 DEBUG:boto:Path: /ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.349 DEBUG:boto:Data: 2023-11-02T18:52:38.350 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.350 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.351 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.351 DEBUG:boto:Params: {} 2023-11-02T18:52:38.352 DEBUG:boto:Token: None 2023-11-02T18:52:38.352 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.353 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5OrFpncy56fhKljuLCfnNyKtQPk= 2023-11-02T18:52:38.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5OrFpncy56fhKljuLCfnNyKtQPk=', 'Content-Length': '0'} 2023-11-02T18:52:38.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000651f03c999e8e4b6-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.398 DEBUG:boto:path=/ibtljn-61/obj145 2023-11-02T18:52:38.399 DEBUG:boto:auth_path=/ibtljn-61/obj145 2023-11-02T18:52:38.400 DEBUG:boto:path=/ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.400 DEBUG:boto:auth_path=/ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.400 DEBUG:boto:Method: GET 2023-11-02T18:52:38.401 DEBUG:boto:Path: /ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.401 DEBUG:boto:Data: 2023-11-02T18:52:38.402 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.403 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.404 DEBUG:boto:Params: {} 2023-11-02T18:52:38.404 DEBUG:boto:Token: None 2023-11-02T18:52:38.405 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj145?versionId=null 2023-11-02T18:52:38.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5OrFpncy56fhKljuLCfnNyKtQPk= 2023-11-02T18:52:38.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5OrFpncy56fhKljuLCfnNyKtQPk=', 'Content-Length': '0'} 2023-11-02T18:52:38.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f99085be983af6b0-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.411 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.411 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.411 DEBUG:boto:path=/ibtljn-61/?prefix=obj145 2023-11-02T18:52:38.412 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj145 2023-11-02T18:52:38.412 DEBUG:boto:Method: GET 2023-11-02T18:52:38.413 DEBUG:boto:Path: /ibtljn-61/?prefix=obj145 2023-11-02T18:52:38.413 DEBUG:boto:Data: 2023-11-02T18:52:38.414 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.414 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.414 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.415 DEBUG:boto:Params: {} 2023-11-02T18:52:38.415 DEBUG:boto:Token: None 2023-11-02T18:52:38.416 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.416 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da86cceb25d72f37-006543eff6-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.423 DEBUG:boto:b'ibtljn-61obj1451000falseobj1452023-11-02T18:50:24.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.424 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.424 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.425 DEBUG:boto:path=/ibtljn-61/?prefix=obj145 2023-11-02T18:52:38.425 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj145 2023-11-02T18:52:38.426 DEBUG:boto:Method: GET 2023-11-02T18:52:38.426 DEBUG:boto:Path: /ibtljn-61/?prefix=obj145 2023-11-02T18:52:38.427 DEBUG:boto:Data: 2023-11-02T18:52:38.427 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.428 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.428 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.428 DEBUG:boto:Params: {} 2023-11-02T18:52:38.429 DEBUG:boto:Token: None 2023-11-02T18:52:38.430 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.430 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa1b2f82952e3ee5-006543eff6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.441 DEBUG:boto:b'ibtljn-61obj1451000falseobj1452023-11-02T18:50:24.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.442 DEBUG:boto:path=/ibtljn-61/obj145 2023-11-02T18:52:38.443 DEBUG:boto:auth_path=/ibtljn-61/obj145 2023-11-02T18:52:38.443 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.443 DEBUG:boto:Path: /ibtljn-61/obj145 2023-11-02T18:52:38.444 DEBUG:boto:Data: 2023-11-02T18:52:38.444 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.445 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.445 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.445 DEBUG:boto:Params: {} 2023-11-02T18:52:38.446 DEBUG:boto:Token: None 2023-11-02T18:52:38.446 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj145 2023-11-02T18:52:38.447 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JZYCnP5yNplNwdb80HJrobXMvz8= 2023-11-02T18:52:38.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JZYCnP5yNplNwdb80HJrobXMvz8=', 'Content-Length': '0'} 2023-11-02T18:52:38.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c943967fc391e9d-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.450 DEBUG:boto:path=/ibtljn-61/obj145 2023-11-02T18:52:38.451 DEBUG:boto:auth_path=/ibtljn-61/obj145 2023-11-02T18:52:38.451 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.452 DEBUG:boto:Path: /ibtljn-61/obj145 2023-11-02T18:52:38.452 DEBUG:boto:Data: 2023-11-02T18:52:38.453 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.453 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.453 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.454 DEBUG:boto:Params: {} 2023-11-02T18:52:38.454 DEBUG:boto:Token: None 2023-11-02T18:52:38.455 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj145 2023-11-02T18:52:38.455 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JZYCnP5yNplNwdb80HJrobXMvz8= 2023-11-02T18:52:38.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JZYCnP5yNplNwdb80HJrobXMvz8=', 'Content-Length': '0'} 2023-11-02T18:52:38.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018283bf792515678-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T18:52:38.459 DEBUG:boto:path=/ibtljn-61/obj145 2023-11-02T18:52:38.460 DEBUG:boto:auth_path=/ibtljn-61/obj145 2023-11-02T18:52:38.460 DEBUG:boto:Method: GET 2023-11-02T18:52:38.460 DEBUG:boto:Path: /ibtljn-61/obj145 2023-11-02T18:52:38.461 DEBUG:boto:Data: 2023-11-02T18:52:38.461 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.462 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.463 DEBUG:boto:Params: {} 2023-11-02T18:52:38.463 DEBUG:boto:Token: None 2023-11-02T18:52:38.464 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj145 2023-11-02T18:52:38.464 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:h9nYvOp25xMAntoEPP/NSDh9fRs= 2023-11-02T18:52:38.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h9nYvOp25xMAntoEPP/NSDh9fRs=', 'Content-Length': '0'} 2023-11-02T18:52:38.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028de3944878b4884-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.468 DEBUG:boto:path=/ibtljn-61/obj145 2023-11-02T18:52:38.468 DEBUG:boto:auth_path=/ibtljn-61/obj145 2023-11-02T18:52:38.469 DEBUG:boto:Method: GET 2023-11-02T18:52:38.469 DEBUG:boto:Path: /ibtljn-61/obj145 2023-11-02T18:52:38.470 DEBUG:boto:Data: 2023-11-02T18:52:38.470 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.471 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.471 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.471 DEBUG:boto:Params: {} 2023-11-02T18:52:38.472 DEBUG:boto:Token: None 2023-11-02T18:52:38.472 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj145 2023-11-02T18:52:38.473 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:h9nYvOp25xMAntoEPP/NSDh9fRs= 2023-11-02T18:52:38.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h9nYvOp25xMAntoEPP/NSDh9fRs=', 'Content-Length': '0'} 2023-11-02T18:52:38.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce3b1097d69ba96-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.477 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.477 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.478 DEBUG:boto:path=/ibtljn-61/?prefix=obj145 2023-11-02T18:52:38.478 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj145 2023-11-02T18:52:38.479 DEBUG:boto:Method: GET 2023-11-02T18:52:38.479 DEBUG:boto:Path: /ibtljn-61/?prefix=obj145 2023-11-02T18:52:38.480 DEBUG:boto:Data: 2023-11-02T18:52:38.480 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.481 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.481 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.481 DEBUG:boto:Params: {} 2023-11-02T18:52:38.482 DEBUG:boto:Token: None 2023-11-02T18:52:38.482 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.483 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ef32bfbc1cdc0c6-006543eff6-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.530 DEBUG:boto:b'ibtljn-61obj1451000falseobj1452023-11-02T18:50:24.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.531 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.532 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.532 DEBUG:boto:path=/ibtljn-61/?prefix=obj145 2023-11-02T18:52:38.533 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj145 2023-11-02T18:52:38.533 DEBUG:boto:Method: GET 2023-11-02T18:52:38.533 DEBUG:boto:Path: /ibtljn-61/?prefix=obj145 2023-11-02T18:52:38.534 DEBUG:boto:Data: 2023-11-02T18:52:38.534 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.535 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.535 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.536 DEBUG:boto:Params: {} 2023-11-02T18:52:38.536 DEBUG:boto:Token: None 2023-11-02T18:52:38.537 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.537 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d9c40f52469de7e-006543eff6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.549 DEBUG:boto:b'ibtljn-61obj1451000falseobj1452023-11-02T18:50:24.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-11-02T18:52:38.550 DEBUG:boto:path=/ibtljn-61/obj148 2023-11-02T18:52:38.550 DEBUG:boto:auth_path=/ibtljn-61/obj148 2023-11-02T18:52:38.551 DEBUG:boto:path=/ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.551 DEBUG:boto:auth_path=/ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.552 DEBUG:boto:Method: GET 2023-11-02T18:52:38.552 DEBUG:boto:Path: /ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.552 DEBUG:boto:Data: 2023-11-02T18:52:38.553 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.554 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.554 DEBUG:boto:Params: {} 2023-11-02T18:52:38.555 DEBUG:boto:Token: None 2023-11-02T18:52:38.555 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SY3dFxtq5zCbseTUVXA8RCSVOVM= 2023-11-02T18:52:38.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SY3dFxtq5zCbseTUVXA8RCSVOVM=', 'Content-Length': '0'} 2023-11-02T18:52:38.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ce36820f805be4a-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.560 DEBUG:boto:path=/ibtljn-61/obj148 2023-11-02T18:52:38.560 DEBUG:boto:auth_path=/ibtljn-61/obj148 2023-11-02T18:52:38.561 DEBUG:boto:path=/ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.561 DEBUG:boto:auth_path=/ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.562 DEBUG:boto:Method: GET 2023-11-02T18:52:38.562 DEBUG:boto:Path: /ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.563 DEBUG:boto:Data: 2023-11-02T18:52:38.563 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.563 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.564 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.564 DEBUG:boto:Params: {} 2023-11-02T18:52:38.565 DEBUG:boto:Token: None 2023-11-02T18:52:38.565 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.566 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SY3dFxtq5zCbseTUVXA8RCSVOVM= 2023-11-02T18:52:38.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SY3dFxtq5zCbseTUVXA8RCSVOVM=', 'Content-Length': '0'} 2023-11-02T18:52:38.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d26fd3098d544596-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.614 DEBUG:boto:path=/ibtljn-61/obj148 2023-11-02T18:52:38.615 DEBUG:boto:auth_path=/ibtljn-61/obj148 2023-11-02T18:52:38.615 DEBUG:boto:path=/ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.616 DEBUG:boto:auth_path=/ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.616 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.617 DEBUG:boto:Path: /ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.617 DEBUG:boto:Data: 2023-11-02T18:52:38.617 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.618 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.618 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.619 DEBUG:boto:Params: {} 2023-11-02T18:52:38.619 DEBUG:boto:Token: None 2023-11-02T18:52:38.620 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.620 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+I4dL1V5nEUtEwJW01AKxZqpkQ= 2023-11-02T18:52:38.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+I4dL1V5nEUtEwJW01AKxZqpkQ=', 'Content-Length': '0'} 2023-11-02T18:52:38.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000656dcd4981cba407-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.625 DEBUG:boto:path=/ibtljn-61/obj148 2023-11-02T18:52:38.625 DEBUG:boto:auth_path=/ibtljn-61/obj148 2023-11-02T18:52:38.625 DEBUG:boto:path=/ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.626 DEBUG:boto:auth_path=/ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.626 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.627 DEBUG:boto:Path: /ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.627 DEBUG:boto:Data: 2023-11-02T18:52:38.627 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.628 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.628 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.629 DEBUG:boto:Params: {} 2023-11-02T18:52:38.629 DEBUG:boto:Token: None 2023-11-02T18:52:38.630 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.630 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+I4dL1V5nEUtEwJW01AKxZqpkQ= 2023-11-02T18:52:38.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+I4dL1V5nEUtEwJW01AKxZqpkQ=', 'Content-Length': '0'} 2023-11-02T18:52:38.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6085f82274e8d9b-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-11-02T18:52:38.635 DEBUG:boto:path=/ibtljn-61/obj148 2023-11-02T18:52:38.635 DEBUG:boto:auth_path=/ibtljn-61/obj148 2023-11-02T18:52:38.636 DEBUG:boto:path=/ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.636 DEBUG:boto:auth_path=/ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.637 DEBUG:boto:Method: GET 2023-11-02T18:52:38.637 DEBUG:boto:Path: /ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.637 DEBUG:boto:Data: 2023-11-02T18:52:38.638 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.638 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.639 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.639 DEBUG:boto:Params: {} 2023-11-02T18:52:38.640 DEBUG:boto:Token: None 2023-11-02T18:52:38.640 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SY3dFxtq5zCbseTUVXA8RCSVOVM= 2023-11-02T18:52:38.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SY3dFxtq5zCbseTUVXA8RCSVOVM=', 'Content-Length': '0'} 2023-11-02T18:52:38.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fccbe3e768b6905-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.649 DEBUG:boto:path=/ibtljn-61/obj148 2023-11-02T18:52:38.649 DEBUG:boto:auth_path=/ibtljn-61/obj148 2023-11-02T18:52:38.650 DEBUG:boto:path=/ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.650 DEBUG:boto:auth_path=/ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.651 DEBUG:boto:Method: GET 2023-11-02T18:52:38.651 DEBUG:boto:Path: /ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.652 DEBUG:boto:Data: 2023-11-02T18:52:38.652 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.652 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.653 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.653 DEBUG:boto:Params: {} 2023-11-02T18:52:38.654 DEBUG:boto:Token: None 2023-11-02T18:52:38.654 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj148?versionId=null 2023-11-02T18:52:38.655 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SY3dFxtq5zCbseTUVXA8RCSVOVM= 2023-11-02T18:52:38.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SY3dFxtq5zCbseTUVXA8RCSVOVM=', 'Content-Length': '0'} 2023-11-02T18:52:38.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005adc4188f9357065-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.658 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.658 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.659 DEBUG:boto:path=/ibtljn-61/?prefix=obj148 2023-11-02T18:52:38.659 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj148 2023-11-02T18:52:38.660 DEBUG:boto:Method: GET 2023-11-02T18:52:38.660 DEBUG:boto:Path: /ibtljn-61/?prefix=obj148 2023-11-02T18:52:38.661 DEBUG:boto:Data: 2023-11-02T18:52:38.661 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.661 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.662 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.662 DEBUG:boto:Params: {} 2023-11-02T18:52:38.663 DEBUG:boto:Token: None 2023-11-02T18:52:38.663 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.664 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000921f45993daa0c4c-006543eff6-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.714 DEBUG:boto:b'ibtljn-61obj1481000falseobj1482023-11-02T18:50:25.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.715 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.716 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.716 DEBUG:boto:path=/ibtljn-61/?prefix=obj148 2023-11-02T18:52:38.717 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj148 2023-11-02T18:52:38.717 DEBUG:boto:Method: GET 2023-11-02T18:52:38.717 DEBUG:boto:Path: /ibtljn-61/?prefix=obj148 2023-11-02T18:52:38.718 DEBUG:boto:Data: 2023-11-02T18:52:38.718 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.719 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.719 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.720 DEBUG:boto:Params: {} 2023-11-02T18:52:38.720 DEBUG:boto:Token: None 2023-11-02T18:52:38.721 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098094c17b4a2ae6e-006543eff6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.732 DEBUG:boto:b'ibtljn-61obj1481000falseobj1482023-11-02T18:50:25.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.733 DEBUG:boto:path=/ibtljn-61/obj148 2023-11-02T18:52:38.733 DEBUG:boto:auth_path=/ibtljn-61/obj148 2023-11-02T18:52:38.734 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.735 DEBUG:boto:Path: /ibtljn-61/obj148 2023-11-02T18:52:38.735 DEBUG:boto:Data: 2023-11-02T18:52:38.736 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.736 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.737 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.737 DEBUG:boto:Params: {} 2023-11-02T18:52:38.738 DEBUG:boto:Token: None 2023-11-02T18:52:38.738 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj148 2023-11-02T18:52:38.739 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sVks6ROyDymQ7uyt5afM1kL+txg= 2023-11-02T18:52:38.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sVks6ROyDymQ7uyt5afM1kL+txg=', 'Content-Length': '0'} 2023-11-02T18:52:38.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93f3c2928140fd3-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.743 DEBUG:boto:path=/ibtljn-61/obj148 2023-11-02T18:52:38.743 DEBUG:boto:auth_path=/ibtljn-61/obj148 2023-11-02T18:52:38.744 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.744 DEBUG:boto:Path: /ibtljn-61/obj148 2023-11-02T18:52:38.744 DEBUG:boto:Data: 2023-11-02T18:52:38.745 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.745 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.746 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.746 DEBUG:boto:Params: {} 2023-11-02T18:52:38.747 DEBUG:boto:Token: None 2023-11-02T18:52:38.747 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj148 2023-11-02T18:52:38.748 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sVks6ROyDymQ7uyt5afM1kL+txg= 2023-11-02T18:52:38.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sVks6ROyDymQ7uyt5afM1kL+txg=', 'Content-Length': '0'} 2023-11-02T18:52:38.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8a45c937ae60e50-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-11-02T18:52:38.752 DEBUG:boto:path=/ibtljn-61/obj148 2023-11-02T18:52:38.752 DEBUG:boto:auth_path=/ibtljn-61/obj148 2023-11-02T18:52:38.752 DEBUG:boto:Method: GET 2023-11-02T18:52:38.753 DEBUG:boto:Path: /ibtljn-61/obj148 2023-11-02T18:52:38.753 DEBUG:boto:Data: 2023-11-02T18:52:38.754 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.755 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.755 DEBUG:boto:Params: {} 2023-11-02T18:52:38.756 DEBUG:boto:Token: None 2023-11-02T18:52:38.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj148 2023-11-02T18:52:38.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IIgPzsOuMOgWJCEJmPK4Ld3jo84= 2023-11-02T18:52:38.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IIgPzsOuMOgWJCEJmPK4Ld3jo84=', 'Content-Length': '0'} 2023-11-02T18:52:38.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a09b58f66b704b09-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.761 DEBUG:boto:path=/ibtljn-61/obj148 2023-11-02T18:52:38.761 DEBUG:boto:auth_path=/ibtljn-61/obj148 2023-11-02T18:52:38.761 DEBUG:boto:Method: GET 2023-11-02T18:52:38.762 DEBUG:boto:Path: /ibtljn-61/obj148 2023-11-02T18:52:38.762 DEBUG:boto:Data: 2023-11-02T18:52:38.763 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.763 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.764 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.764 DEBUG:boto:Params: {} 2023-11-02T18:52:38.765 DEBUG:boto:Token: None 2023-11-02T18:52:38.765 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj148 2023-11-02T18:52:38.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IIgPzsOuMOgWJCEJmPK4Ld3jo84= 2023-11-02T18:52:38.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IIgPzsOuMOgWJCEJmPK4Ld3jo84=', 'Content-Length': '0'} 2023-11-02T18:52:38.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082c951b953059c2b-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.810 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.811 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.811 DEBUG:boto:path=/ibtljn-61/?prefix=obj148 2023-11-02T18:52:38.812 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj148 2023-11-02T18:52:38.813 DEBUG:boto:Method: GET 2023-11-02T18:52:38.813 DEBUG:boto:Path: /ibtljn-61/?prefix=obj148 2023-11-02T18:52:38.814 DEBUG:boto:Data: 2023-11-02T18:52:38.814 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.815 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.815 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.815 DEBUG:boto:Params: {} 2023-11-02T18:52:38.816 DEBUG:boto:Token: None 2023-11-02T18:52:38.816 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.817 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a20b4a741e5f94ef-006543eff6-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.824 DEBUG:boto:b'ibtljn-61obj1481000falseobj1482023-11-02T18:50:25.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.825 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.825 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.826 DEBUG:boto:path=/ibtljn-61/?prefix=obj148 2023-11-02T18:52:38.826 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj148 2023-11-02T18:52:38.826 DEBUG:boto:Method: GET 2023-11-02T18:52:38.827 DEBUG:boto:Path: /ibtljn-61/?prefix=obj148 2023-11-02T18:52:38.827 DEBUG:boto:Data: 2023-11-02T18:52:38.828 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.828 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.829 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.829 DEBUG:boto:Params: {} 2023-11-02T18:52:38.830 DEBUG:boto:Token: None 2023-11-02T18:52:38.830 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.831 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016fb212fa6938119-006543eff6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.843 DEBUG:boto:b'ibtljn-61obj1481000falseobj1482023-11-02T18:50:25.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T18:52:38.845 DEBUG:boto:path=/ibtljn-61/obj155 2023-11-02T18:52:38.845 DEBUG:boto:auth_path=/ibtljn-61/obj155 2023-11-02T18:52:38.846 DEBUG:boto:path=/ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.846 DEBUG:boto:auth_path=/ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.847 DEBUG:boto:Method: GET 2023-11-02T18:52:38.847 DEBUG:boto:Path: /ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.848 DEBUG:boto:Data: 2023-11-02T18:52:38.848 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.849 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.849 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.850 DEBUG:boto:Params: {} 2023-11-02T18:52:38.850 DEBUG:boto:Token: None 2023-11-02T18:52:38.851 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.851 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hwnTuYCsB4RyBQ1QlufDGqfLX7U= 2023-11-02T18:52:38.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hwnTuYCsB4RyBQ1QlufDGqfLX7U=', 'Content-Length': '0'} 2023-11-02T18:52:38.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000281d2032ffe6142f-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.855 DEBUG:boto:path=/ibtljn-61/obj155 2023-11-02T18:52:38.856 DEBUG:boto:auth_path=/ibtljn-61/obj155 2023-11-02T18:52:38.856 DEBUG:boto:path=/ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.857 DEBUG:boto:auth_path=/ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.857 DEBUG:boto:Method: GET 2023-11-02T18:52:38.858 DEBUG:boto:Path: /ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.858 DEBUG:boto:Data: 2023-11-02T18:52:38.859 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.859 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.860 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.860 DEBUG:boto:Params: {} 2023-11-02T18:52:38.860 DEBUG:boto:Token: None 2023-11-02T18:52:38.861 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.861 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hwnTuYCsB4RyBQ1QlufDGqfLX7U= 2023-11-02T18:52:38.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hwnTuYCsB4RyBQ1QlufDGqfLX7U=', 'Content-Length': '0'} 2023-11-02T18:52:38.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c83063aaa11117c-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.865 DEBUG:boto:path=/ibtljn-61/obj155 2023-11-02T18:52:38.866 DEBUG:boto:auth_path=/ibtljn-61/obj155 2023-11-02T18:52:38.866 DEBUG:boto:path=/ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.867 DEBUG:boto:auth_path=/ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.867 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.868 DEBUG:boto:Path: /ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.868 DEBUG:boto:Data: 2023-11-02T18:52:38.869 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.869 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.869 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.870 DEBUG:boto:Params: {} 2023-11-02T18:52:38.870 DEBUG:boto:Token: None 2023-11-02T18:52:38.871 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.871 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WArs3+F6dyhFXLMVlBNdLZIwJAk= 2023-11-02T18:52:38.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WArs3+F6dyhFXLMVlBNdLZIwJAk=', 'Content-Length': '0'} 2023-11-02T18:52:38.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ce72e64279135d9-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.875 DEBUG:boto:path=/ibtljn-61/obj155 2023-11-02T18:52:38.876 DEBUG:boto:auth_path=/ibtljn-61/obj155 2023-11-02T18:52:38.876 DEBUG:boto:path=/ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.877 DEBUG:boto:auth_path=/ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.877 DEBUG:boto:Method: HEAD 2023-11-02T18:52:38.877 DEBUG:boto:Path: /ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.878 DEBUG:boto:Data: 2023-11-02T18:52:38.878 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.879 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.879 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.880 DEBUG:boto:Params: {} 2023-11-02T18:52:38.880 DEBUG:boto:Token: None 2023-11-02T18:52:38.881 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.881 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WArs3+F6dyhFXLMVlBNdLZIwJAk= 2023-11-02T18:52:38.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WArs3+F6dyhFXLMVlBNdLZIwJAk=', 'Content-Length': '0'} 2023-11-02T18:52:38.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000767f6de9761603c9-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T18:52:38.888 DEBUG:boto:path=/ibtljn-61/obj155 2023-11-02T18:52:38.888 DEBUG:boto:auth_path=/ibtljn-61/obj155 2023-11-02T18:52:38.888 DEBUG:boto:path=/ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.889 DEBUG:boto:auth_path=/ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.890 DEBUG:boto:Method: GET 2023-11-02T18:52:38.890 DEBUG:boto:Path: /ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.891 DEBUG:boto:Data: 2023-11-02T18:52:38.891 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.891 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.892 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.892 DEBUG:boto:Params: {} 2023-11-02T18:52:38.893 DEBUG:boto:Token: None 2023-11-02T18:52:38.893 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.894 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hwnTuYCsB4RyBQ1QlufDGqfLX7U= 2023-11-02T18:52:38.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hwnTuYCsB4RyBQ1QlufDGqfLX7U=', 'Content-Length': '0'} 2023-11-02T18:52:38.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003be68bad4a1ee964-006543eff6-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.938 DEBUG:boto:path=/ibtljn-61/obj155 2023-11-02T18:52:38.939 DEBUG:boto:auth_path=/ibtljn-61/obj155 2023-11-02T18:52:38.939 DEBUG:boto:path=/ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.940 DEBUG:boto:auth_path=/ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.940 DEBUG:boto:Method: GET 2023-11-02T18:52:38.941 DEBUG:boto:Path: /ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.941 DEBUG:boto:Data: 2023-11-02T18:52:38.942 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.942 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.943 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.943 DEBUG:boto:Params: {} 2023-11-02T18:52:38.943 DEBUG:boto:Token: None 2023-11-02T18:52:38.944 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/obj155?versionId=null 2023-11-02T18:52:38.944 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hwnTuYCsB4RyBQ1QlufDGqfLX7U= 2023-11-02T18:52:38.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hwnTuYCsB4RyBQ1QlufDGqfLX7U=', 'Content-Length': '0'} 2023-11-02T18:52:38.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e63f1190d43bc393-006543eff6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.949 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.949 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.950 DEBUG:boto:path=/ibtljn-61/?prefix=obj155 2023-11-02T18:52:38.950 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj155 2023-11-02T18:52:38.951 DEBUG:boto:Method: GET 2023-11-02T18:52:38.951 DEBUG:boto:Path: /ibtljn-61/?prefix=obj155 2023-11-02T18:52:38.951 DEBUG:boto:Data: 2023-11-02T18:52:38.952 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.952 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:38.953 DEBUG:boto:Port: 8001 2023-11-02T18:52:38.953 DEBUG:boto:Params: {} 2023-11-02T18:52:38.954 DEBUG:boto:Token: None 2023-11-02T18:52:38.954 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.955 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a618d94ebcdb49-006543eff6-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:38.962 DEBUG:boto:b'ibtljn-61obj1551000falseobj1552023-11-02T18:50:24.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:38.963 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:38.963 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:38.964 DEBUG:boto:path=/ibtljn-61/?prefix=obj155 2023-11-02T18:52:38.964 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj155 2023-11-02T18:52:38.965 DEBUG:boto:Method: GET 2023-11-02T18:52:38.965 DEBUG:boto:Path: /ibtljn-61/?prefix=obj155 2023-11-02T18:52:38.966 DEBUG:boto:Data: 2023-11-02T18:52:38.966 DEBUG:boto:Headers: {} 2023-11-02T18:52:38.967 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:38.967 DEBUG:boto:Port: 8000 2023-11-02T18:52:38.968 DEBUG:boto:Params: {} 2023-11-02T18:52:38.968 DEBUG:boto:Token: None 2023-11-02T18:52:38.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:38 GMT /ibtljn-61/ 2023-11-02T18:52:38.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8= 2023-11-02T18:52:38.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sALjPytKf+zCqhYDgnF6gyP7Fm8=', 'Content-Length': '0'} 2023-11-02T18:52:38.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d07b3278ce73e850-006543eff6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.024 DEBUG:boto:b'ibtljn-61obj1551000falseobj1552023-11-02T18:50:24.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.025 DEBUG:boto:path=/ibtljn-61/obj155 2023-11-02T18:52:39.026 DEBUG:boto:auth_path=/ibtljn-61/obj155 2023-11-02T18:52:39.026 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.027 DEBUG:boto:Path: /ibtljn-61/obj155 2023-11-02T18:52:39.027 DEBUG:boto:Data: 2023-11-02T18:52:39.028 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.028 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.028 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.029 DEBUG:boto:Params: {} 2023-11-02T18:52:39.030 DEBUG:boto:Token: None 2023-11-02T18:52:39.030 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj155 2023-11-02T18:52:39.031 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wRdEFYeo27VeFSHBb9wAdQjHW2I= 2023-11-02T18:52:39.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wRdEFYeo27VeFSHBb9wAdQjHW2I=', 'Content-Length': '0'} 2023-11-02T18:52:39.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052455370c1be4ca6-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.035 DEBUG:boto:path=/ibtljn-61/obj155 2023-11-02T18:52:39.035 DEBUG:boto:auth_path=/ibtljn-61/obj155 2023-11-02T18:52:39.036 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.036 DEBUG:boto:Path: /ibtljn-61/obj155 2023-11-02T18:52:39.037 DEBUG:boto:Data: 2023-11-02T18:52:39.037 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.038 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.039 DEBUG:boto:Params: {} 2023-11-02T18:52:39.039 DEBUG:boto:Token: None 2023-11-02T18:52:39.040 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj155 2023-11-02T18:52:39.040 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wRdEFYeo27VeFSHBb9wAdQjHW2I= 2023-11-02T18:52:39.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wRdEFYeo27VeFSHBb9wAdQjHW2I=', 'Content-Length': '0'} 2023-11-02T18:52:39.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015663795fdfe2af5-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T18:52:39.045 DEBUG:boto:path=/ibtljn-61/obj155 2023-11-02T18:52:39.045 DEBUG:boto:auth_path=/ibtljn-61/obj155 2023-11-02T18:52:39.046 DEBUG:boto:Method: GET 2023-11-02T18:52:39.046 DEBUG:boto:Path: /ibtljn-61/obj155 2023-11-02T18:52:39.046 DEBUG:boto:Data: 2023-11-02T18:52:39.047 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.047 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.048 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.048 DEBUG:boto:Params: {} 2023-11-02T18:52:39.049 DEBUG:boto:Token: None 2023-11-02T18:52:39.049 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj155 2023-11-02T18:52:39.050 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwJOOKJ6UEricYe1nG+4IXLayF0= 2023-11-02T18:52:39.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwJOOKJ6UEricYe1nG+4IXLayF0=', 'Content-Length': '0'} 2023-11-02T18:52:39.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000347e6d4a5e79702d-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.054 DEBUG:boto:path=/ibtljn-61/obj155 2023-11-02T18:52:39.054 DEBUG:boto:auth_path=/ibtljn-61/obj155 2023-11-02T18:52:39.055 DEBUG:boto:Method: GET 2023-11-02T18:52:39.055 DEBUG:boto:Path: /ibtljn-61/obj155 2023-11-02T18:52:39.056 DEBUG:boto:Data: 2023-11-02T18:52:39.056 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.056 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.057 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.057 DEBUG:boto:Params: {} 2023-11-02T18:52:39.058 DEBUG:boto:Token: None 2023-11-02T18:52:39.059 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj155 2023-11-02T18:52:39.059 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwJOOKJ6UEricYe1nG+4IXLayF0= 2023-11-02T18:52:39.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwJOOKJ6UEricYe1nG+4IXLayF0=', 'Content-Length': '0'} 2023-11-02T18:52:39.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b0b7625cec70d61-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.064 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.064 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.064 DEBUG:boto:path=/ibtljn-61/?prefix=obj155 2023-11-02T18:52:39.065 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj155 2023-11-02T18:52:39.065 DEBUG:boto:Method: GET 2023-11-02T18:52:39.066 DEBUG:boto:Path: /ibtljn-61/?prefix=obj155 2023-11-02T18:52:39.066 DEBUG:boto:Data: 2023-11-02T18:52:39.067 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.067 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.068 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.068 DEBUG:boto:Params: {} 2023-11-02T18:52:39.069 DEBUG:boto:Token: None 2023-11-02T18:52:39.069 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.069 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000678edf9c8bbb6b34-006543eff7-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.076 DEBUG:boto:b'ibtljn-61obj1551000falseobj1552023-11-02T18:50:24.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.077 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.078 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.078 DEBUG:boto:path=/ibtljn-61/?prefix=obj155 2023-11-02T18:52:39.079 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj155 2023-11-02T18:52:39.079 DEBUG:boto:Method: GET 2023-11-02T18:52:39.080 DEBUG:boto:Path: /ibtljn-61/?prefix=obj155 2023-11-02T18:52:39.080 DEBUG:boto:Data: 2023-11-02T18:52:39.081 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.081 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.082 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.082 DEBUG:boto:Params: {} 2023-11-02T18:52:39.083 DEBUG:boto:Token: None 2023-11-02T18:52:39.083 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.084 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba9ca1d49819932d-006543eff7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.138 DEBUG:boto:b'ibtljn-61obj1551000falseobj1552023-11-02T18:50:24.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:52:39.140 DEBUG:boto:path=/ibtljn-61/obj17 2023-11-02T18:52:39.140 DEBUG:boto:auth_path=/ibtljn-61/obj17 2023-11-02T18:52:39.141 DEBUG:boto:path=/ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.141 DEBUG:boto:auth_path=/ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.142 DEBUG:boto:Method: GET 2023-11-02T18:52:39.142 DEBUG:boto:Path: /ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.143 DEBUG:boto:Data: 2023-11-02T18:52:39.143 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.143 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.144 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.144 DEBUG:boto:Params: {} 2023-11-02T18:52:39.145 DEBUG:boto:Token: None 2023-11-02T18:52:39.145 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.146 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tQDC4D744Bar42G31cDPTGE6Hf8= 2023-11-02T18:52:39.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tQDC4D744Bar42G31cDPTGE6Hf8=', 'Content-Length': '0'} 2023-11-02T18:52:39.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069e1dee5305b520b-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.151 DEBUG:boto:path=/ibtljn-61/obj17 2023-11-02T18:52:39.151 DEBUG:boto:auth_path=/ibtljn-61/obj17 2023-11-02T18:52:39.151 DEBUG:boto:path=/ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.152 DEBUG:boto:auth_path=/ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.152 DEBUG:boto:Method: GET 2023-11-02T18:52:39.153 DEBUG:boto:Path: /ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.153 DEBUG:boto:Data: 2023-11-02T18:52:39.154 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.154 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.155 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.155 DEBUG:boto:Params: {} 2023-11-02T18:52:39.156 DEBUG:boto:Token: None 2023-11-02T18:52:39.156 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.157 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tQDC4D744Bar42G31cDPTGE6Hf8= 2023-11-02T18:52:39.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tQDC4D744Bar42G31cDPTGE6Hf8=', 'Content-Length': '0'} 2023-11-02T18:52:39.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000134d75c937aa7e32-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.161 DEBUG:boto:path=/ibtljn-61/obj17 2023-11-02T18:52:39.162 DEBUG:boto:auth_path=/ibtljn-61/obj17 2023-11-02T18:52:39.162 DEBUG:boto:path=/ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.163 DEBUG:boto:auth_path=/ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.163 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.164 DEBUG:boto:Path: /ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.164 DEBUG:boto:Data: 2023-11-02T18:52:39.164 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.165 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.165 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.166 DEBUG:boto:Params: {} 2023-11-02T18:52:39.166 DEBUG:boto:Token: None 2023-11-02T18:52:39.167 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.167 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uk505S1vcAgBdS18HScAZ86ig4M= 2023-11-02T18:52:39.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uk505S1vcAgBdS18HScAZ86ig4M=', 'Content-Length': '0'} 2023-11-02T18:52:39.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a9d94ca54b30c2c-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.171 DEBUG:boto:path=/ibtljn-61/obj17 2023-11-02T18:52:39.172 DEBUG:boto:auth_path=/ibtljn-61/obj17 2023-11-02T18:52:39.173 DEBUG:boto:path=/ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.173 DEBUG:boto:auth_path=/ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.174 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.175 DEBUG:boto:Path: /ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.175 DEBUG:boto:Data: 2023-11-02T18:52:39.175 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.176 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.176 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.177 DEBUG:boto:Params: {} 2023-11-02T18:52:39.177 DEBUG:boto:Token: None 2023-11-02T18:52:39.178 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.178 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uk505S1vcAgBdS18HScAZ86ig4M= 2023-11-02T18:52:39.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uk505S1vcAgBdS18HScAZ86ig4M=', 'Content-Length': '0'} 2023-11-02T18:52:39.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000542606c148745558-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:52:39.183 DEBUG:boto:path=/ibtljn-61/obj17 2023-11-02T18:52:39.184 DEBUG:boto:auth_path=/ibtljn-61/obj17 2023-11-02T18:52:39.184 DEBUG:boto:path=/ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.185 DEBUG:boto:auth_path=/ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.185 DEBUG:boto:Method: GET 2023-11-02T18:52:39.186 DEBUG:boto:Path: /ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.186 DEBUG:boto:Data: 2023-11-02T18:52:39.187 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.187 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.188 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.188 DEBUG:boto:Params: {} 2023-11-02T18:52:39.189 DEBUG:boto:Token: None 2023-11-02T18:52:39.189 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.190 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tQDC4D744Bar42G31cDPTGE6Hf8= 2023-11-02T18:52:39.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tQDC4D744Bar42G31cDPTGE6Hf8=', 'Content-Length': '0'} 2023-11-02T18:52:39.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f8c624f14e4927f-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.194 DEBUG:boto:path=/ibtljn-61/obj17 2023-11-02T18:52:39.195 DEBUG:boto:auth_path=/ibtljn-61/obj17 2023-11-02T18:52:39.195 DEBUG:boto:path=/ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.195 DEBUG:boto:auth_path=/ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.196 DEBUG:boto:Method: GET 2023-11-02T18:52:39.196 DEBUG:boto:Path: /ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.197 DEBUG:boto:Data: 2023-11-02T18:52:39.198 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.198 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.199 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.199 DEBUG:boto:Params: {} 2023-11-02T18:52:39.200 DEBUG:boto:Token: None 2023-11-02T18:52:39.200 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj17?versionId=null 2023-11-02T18:52:39.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tQDC4D744Bar42G31cDPTGE6Hf8= 2023-11-02T18:52:39.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tQDC4D744Bar42G31cDPTGE6Hf8=', 'Content-Length': '0'} 2023-11-02T18:52:39.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4fa5db88e57db5e-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.205 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.206 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.206 DEBUG:boto:path=/ibtljn-61/?prefix=obj17 2023-11-02T18:52:39.207 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj17 2023-11-02T18:52:39.207 DEBUG:boto:Method: GET 2023-11-02T18:52:39.208 DEBUG:boto:Path: /ibtljn-61/?prefix=obj17 2023-11-02T18:52:39.208 DEBUG:boto:Data: 2023-11-02T18:52:39.209 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.209 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.210 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.210 DEBUG:boto:Params: {} 2023-11-02T18:52:39.211 DEBUG:boto:Token: None 2023-11-02T18:52:39.211 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.212 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a083036ef0a33b45-006543eff7-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.218 DEBUG:boto:b'ibtljn-61obj171000falseobj172023-11-02T18:50:22.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.219 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.221 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.222 DEBUG:boto:path=/ibtljn-61/?prefix=obj17 2023-11-02T18:52:39.222 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj17 2023-11-02T18:52:39.223 DEBUG:boto:Method: GET 2023-11-02T18:52:39.223 DEBUG:boto:Path: /ibtljn-61/?prefix=obj17 2023-11-02T18:52:39.224 DEBUG:boto:Data: 2023-11-02T18:52:39.224 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.225 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.225 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.225 DEBUG:boto:Params: {} 2023-11-02T18:52:39.226 DEBUG:boto:Token: None 2023-11-02T18:52:39.227 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.227 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa9bee023c08e1f-006543eff7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.238 DEBUG:boto:b'ibtljn-61obj171000falseobj172023-11-02T18:50:22.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.239 DEBUG:boto:path=/ibtljn-61/obj17 2023-11-02T18:52:39.239 DEBUG:boto:auth_path=/ibtljn-61/obj17 2023-11-02T18:52:39.240 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.241 DEBUG:boto:Path: /ibtljn-61/obj17 2023-11-02T18:52:39.241 DEBUG:boto:Data: 2023-11-02T18:52:39.241 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.242 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.242 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.243 DEBUG:boto:Params: {} 2023-11-02T18:52:39.243 DEBUG:boto:Token: None 2023-11-02T18:52:39.244 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj17 2023-11-02T18:52:39.244 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eYJjQLpy3RtGBaNdYrGQcowZaXQ= 2023-11-02T18:52:39.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eYJjQLpy3RtGBaNdYrGQcowZaXQ=', 'Content-Length': '0'} 2023-11-02T18:52:39.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3c7a42780b85f1a-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.247 DEBUG:boto:path=/ibtljn-61/obj17 2023-11-02T18:52:39.248 DEBUG:boto:auth_path=/ibtljn-61/obj17 2023-11-02T18:52:39.248 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.249 DEBUG:boto:Path: /ibtljn-61/obj17 2023-11-02T18:52:39.249 DEBUG:boto:Data: 2023-11-02T18:52:39.250 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.250 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.250 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.251 DEBUG:boto:Params: {} 2023-11-02T18:52:39.251 DEBUG:boto:Token: None 2023-11-02T18:52:39.252 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj17 2023-11-02T18:52:39.253 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eYJjQLpy3RtGBaNdYrGQcowZaXQ= 2023-11-02T18:52:39.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eYJjQLpy3RtGBaNdYrGQcowZaXQ=', 'Content-Length': '0'} 2023-11-02T18:52:39.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f50100b5d5ab240e-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T18:52:39.257 DEBUG:boto:path=/ibtljn-61/obj17 2023-11-02T18:52:39.258 DEBUG:boto:auth_path=/ibtljn-61/obj17 2023-11-02T18:52:39.258 DEBUG:boto:Method: GET 2023-11-02T18:52:39.259 DEBUG:boto:Path: /ibtljn-61/obj17 2023-11-02T18:52:39.259 DEBUG:boto:Data: 2023-11-02T18:52:39.259 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.260 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.260 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.261 DEBUG:boto:Params: {} 2023-11-02T18:52:39.261 DEBUG:boto:Token: None 2023-11-02T18:52:39.263 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj17 2023-11-02T18:52:39.264 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4mUPGULkmAWuiwB7iJ2x+vCiprE= 2023-11-02T18:52:39.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4mUPGULkmAWuiwB7iJ2x+vCiprE=', 'Content-Length': '0'} 2023-11-02T18:52:39.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e8fa448b696e70a-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.314 DEBUG:boto:path=/ibtljn-61/obj17 2023-11-02T18:52:39.315 DEBUG:boto:auth_path=/ibtljn-61/obj17 2023-11-02T18:52:39.315 DEBUG:boto:Method: GET 2023-11-02T18:52:39.316 DEBUG:boto:Path: /ibtljn-61/obj17 2023-11-02T18:52:39.316 DEBUG:boto:Data: 2023-11-02T18:52:39.317 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.318 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.318 DEBUG:boto:Params: {} 2023-11-02T18:52:39.319 DEBUG:boto:Token: None 2023-11-02T18:52:39.319 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj17 2023-11-02T18:52:39.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4mUPGULkmAWuiwB7iJ2x+vCiprE= 2023-11-02T18:52:39.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4mUPGULkmAWuiwB7iJ2x+vCiprE=', 'Content-Length': '0'} 2023-11-02T18:52:39.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056ec819c26b1ec7f-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.325 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.325 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.326 DEBUG:boto:path=/ibtljn-61/?prefix=obj17 2023-11-02T18:52:39.326 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj17 2023-11-02T18:52:39.327 DEBUG:boto:Method: GET 2023-11-02T18:52:39.327 DEBUG:boto:Path: /ibtljn-61/?prefix=obj17 2023-11-02T18:52:39.328 DEBUG:boto:Data: 2023-11-02T18:52:39.328 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.329 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.329 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.330 DEBUG:boto:Params: {} 2023-11-02T18:52:39.331 DEBUG:boto:Token: None 2023-11-02T18:52:39.331 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.332 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6d126d45f57c63b-006543eff7-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.338 DEBUG:boto:b'ibtljn-61obj171000falseobj172023-11-02T18:50:22.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.339 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.340 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.340 DEBUG:boto:path=/ibtljn-61/?prefix=obj17 2023-11-02T18:52:39.341 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj17 2023-11-02T18:52:39.341 DEBUG:boto:Method: GET 2023-11-02T18:52:39.341 DEBUG:boto:Path: /ibtljn-61/?prefix=obj17 2023-11-02T18:52:39.342 DEBUG:boto:Data: 2023-11-02T18:52:39.343 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.344 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.344 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.345 DEBUG:boto:Params: {} 2023-11-02T18:52:39.345 DEBUG:boto:Token: None 2023-11-02T18:52:39.346 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.346 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e37c482d5adf9f7b-006543eff7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.358 DEBUG:boto:b'ibtljn-61obj171000falseobj172023-11-02T18:50:22.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-11-02T18:52:39.359 DEBUG:boto:path=/ibtljn-61/obj185 2023-11-02T18:52:39.360 DEBUG:boto:auth_path=/ibtljn-61/obj185 2023-11-02T18:52:39.360 DEBUG:boto:path=/ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.361 DEBUG:boto:auth_path=/ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.361 DEBUG:boto:Method: GET 2023-11-02T18:52:39.362 DEBUG:boto:Path: /ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.362 DEBUG:boto:Data: 2023-11-02T18:52:39.363 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.363 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.363 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.364 DEBUG:boto:Params: {} 2023-11-02T18:52:39.364 DEBUG:boto:Token: None 2023-11-02T18:52:39.365 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.365 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wLfUvF0Bnzj4336/uiO+nLfF2+I= 2023-11-02T18:52:39.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wLfUvF0Bnzj4336/uiO+nLfF2+I=', 'Content-Length': '0'} 2023-11-02T18:52:39.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c267a0553917b35-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.369 DEBUG:boto:path=/ibtljn-61/obj185 2023-11-02T18:52:39.370 DEBUG:boto:auth_path=/ibtljn-61/obj185 2023-11-02T18:52:39.370 DEBUG:boto:path=/ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.371 DEBUG:boto:auth_path=/ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.371 DEBUG:boto:Method: GET 2023-11-02T18:52:39.372 DEBUG:boto:Path: /ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.372 DEBUG:boto:Data: 2023-11-02T18:52:39.372 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.373 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.373 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.374 DEBUG:boto:Params: {} 2023-11-02T18:52:39.374 DEBUG:boto:Token: None 2023-11-02T18:52:39.375 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.375 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wLfUvF0Bnzj4336/uiO+nLfF2+I= 2023-11-02T18:52:39.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wLfUvF0Bnzj4336/uiO+nLfF2+I=', 'Content-Length': '0'} 2023-11-02T18:52:39.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e3d32c68ed35a7-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.379 DEBUG:boto:path=/ibtljn-61/obj185 2023-11-02T18:52:39.380 DEBUG:boto:auth_path=/ibtljn-61/obj185 2023-11-02T18:52:39.380 DEBUG:boto:path=/ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.381 DEBUG:boto:auth_path=/ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.381 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.382 DEBUG:boto:Path: /ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.382 DEBUG:boto:Data: 2023-11-02T18:52:39.382 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.383 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.384 DEBUG:boto:Params: {} 2023-11-02T18:52:39.384 DEBUG:boto:Token: None 2023-11-02T18:52:39.385 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.385 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Gt0ZxNsttMKKpa1EXYdlFBndUHo= 2023-11-02T18:52:39.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Gt0ZxNsttMKKpa1EXYdlFBndUHo=', 'Content-Length': '0'} 2023-11-02T18:52:39.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000185326a614d07895-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.389 DEBUG:boto:path=/ibtljn-61/obj185 2023-11-02T18:52:39.390 DEBUG:boto:auth_path=/ibtljn-61/obj185 2023-11-02T18:52:39.390 DEBUG:boto:path=/ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.390 DEBUG:boto:auth_path=/ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.391 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.391 DEBUG:boto:Path: /ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.392 DEBUG:boto:Data: 2023-11-02T18:52:39.392 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.393 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.393 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.393 DEBUG:boto:Params: {} 2023-11-02T18:52:39.394 DEBUG:boto:Token: None 2023-11-02T18:52:39.394 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.395 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Gt0ZxNsttMKKpa1EXYdlFBndUHo= 2023-11-02T18:52:39.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Gt0ZxNsttMKKpa1EXYdlFBndUHo=', 'Content-Length': '0'} 2023-11-02T18:52:39.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089868f501c3db5bb-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-11-02T18:52:39.399 DEBUG:boto:path=/ibtljn-61/obj185 2023-11-02T18:52:39.400 DEBUG:boto:auth_path=/ibtljn-61/obj185 2023-11-02T18:52:39.400 DEBUG:boto:path=/ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.400 DEBUG:boto:auth_path=/ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.401 DEBUG:boto:Method: GET 2023-11-02T18:52:39.401 DEBUG:boto:Path: /ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.402 DEBUG:boto:Data: 2023-11-02T18:52:39.402 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.403 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.404 DEBUG:boto:Params: {} 2023-11-02T18:52:39.404 DEBUG:boto:Token: None 2023-11-02T18:52:39.405 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wLfUvF0Bnzj4336/uiO+nLfF2+I= 2023-11-02T18:52:39.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wLfUvF0Bnzj4336/uiO+nLfF2+I=', 'Content-Length': '0'} 2023-11-02T18:52:39.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043605ef0ca71ceb9-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.409 DEBUG:boto:path=/ibtljn-61/obj185 2023-11-02T18:52:39.410 DEBUG:boto:auth_path=/ibtljn-61/obj185 2023-11-02T18:52:39.411 DEBUG:boto:path=/ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.411 DEBUG:boto:auth_path=/ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.412 DEBUG:boto:Method: GET 2023-11-02T18:52:39.412 DEBUG:boto:Path: /ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.413 DEBUG:boto:Data: 2023-11-02T18:52:39.413 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.414 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.414 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.415 DEBUG:boto:Params: {} 2023-11-02T18:52:39.415 DEBUG:boto:Token: None 2023-11-02T18:52:39.416 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj185?versionId=null 2023-11-02T18:52:39.416 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wLfUvF0Bnzj4336/uiO+nLfF2+I= 2023-11-02T18:52:39.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wLfUvF0Bnzj4336/uiO+nLfF2+I=', 'Content-Length': '0'} 2023-11-02T18:52:39.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000819997340ec7a182-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.462 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.463 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.464 DEBUG:boto:path=/ibtljn-61/?prefix=obj185 2023-11-02T18:52:39.464 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj185 2023-11-02T18:52:39.464 DEBUG:boto:Method: GET 2023-11-02T18:52:39.465 DEBUG:boto:Path: /ibtljn-61/?prefix=obj185 2023-11-02T18:52:39.465 DEBUG:boto:Data: 2023-11-02T18:52:39.466 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.466 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.467 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.467 DEBUG:boto:Params: {} 2023-11-02T18:52:39.468 DEBUG:boto:Token: None 2023-11-02T18:52:39.468 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.469 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090e4ca32f59cebe8-006543eff7-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.476 DEBUG:boto:b'ibtljn-61obj1851000falseobj1852023-11-02T18:50:25.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.477 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.478 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.478 DEBUG:boto:path=/ibtljn-61/?prefix=obj185 2023-11-02T18:52:39.478 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj185 2023-11-02T18:52:39.479 DEBUG:boto:Method: GET 2023-11-02T18:52:39.479 DEBUG:boto:Path: /ibtljn-61/?prefix=obj185 2023-11-02T18:52:39.480 DEBUG:boto:Data: 2023-11-02T18:52:39.480 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.481 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.481 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.481 DEBUG:boto:Params: {} 2023-11-02T18:52:39.482 DEBUG:boto:Token: None 2023-11-02T18:52:39.483 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.483 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090f93452edd1f93d-006543eff7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.493 DEBUG:boto:b'ibtljn-61obj1851000falseobj1852023-11-02T18:50:25.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.494 DEBUG:boto:path=/ibtljn-61/obj185 2023-11-02T18:52:39.494 DEBUG:boto:auth_path=/ibtljn-61/obj185 2023-11-02T18:52:39.495 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.495 DEBUG:boto:Path: /ibtljn-61/obj185 2023-11-02T18:52:39.496 DEBUG:boto:Data: 2023-11-02T18:52:39.496 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.497 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.497 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.497 DEBUG:boto:Params: {} 2023-11-02T18:52:39.498 DEBUG:boto:Token: None 2023-11-02T18:52:39.498 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj185 2023-11-02T18:52:39.499 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:05CZSYgoo4yJr/sCecehMHgHk5I= 2023-11-02T18:52:39.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:05CZSYgoo4yJr/sCecehMHgHk5I=', 'Content-Length': '0'} 2023-11-02T18:52:39.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007729f61ceb13bddb-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.503 DEBUG:boto:path=/ibtljn-61/obj185 2023-11-02T18:52:39.503 DEBUG:boto:auth_path=/ibtljn-61/obj185 2023-11-02T18:52:39.504 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.504 DEBUG:boto:Path: /ibtljn-61/obj185 2023-11-02T18:52:39.505 DEBUG:boto:Data: 2023-11-02T18:52:39.505 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.506 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.506 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.506 DEBUG:boto:Params: {} 2023-11-02T18:52:39.507 DEBUG:boto:Token: None 2023-11-02T18:52:39.507 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj185 2023-11-02T18:52:39.508 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:05CZSYgoo4yJr/sCecehMHgHk5I= 2023-11-02T18:52:39.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:05CZSYgoo4yJr/sCecehMHgHk5I=', 'Content-Length': '0'} 2023-11-02T18:52:39.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d57a011d23ddaf3-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-11-02T18:52:39.512 DEBUG:boto:path=/ibtljn-61/obj185 2023-11-02T18:52:39.512 DEBUG:boto:auth_path=/ibtljn-61/obj185 2023-11-02T18:52:39.513 DEBUG:boto:Method: GET 2023-11-02T18:52:39.513 DEBUG:boto:Path: /ibtljn-61/obj185 2023-11-02T18:52:39.514 DEBUG:boto:Data: 2023-11-02T18:52:39.514 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.515 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.515 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.515 DEBUG:boto:Params: {} 2023-11-02T18:52:39.516 DEBUG:boto:Token: None 2023-11-02T18:52:39.516 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj185 2023-11-02T18:52:39.517 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cDlkxB3lPcyU3Buw9S/AFpZkxZw= 2023-11-02T18:52:39.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cDlkxB3lPcyU3Buw9S/AFpZkxZw=', 'Content-Length': '0'} 2023-11-02T18:52:39.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004781da669e4ce512-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.566 DEBUG:boto:path=/ibtljn-61/obj185 2023-11-02T18:52:39.567 DEBUG:boto:auth_path=/ibtljn-61/obj185 2023-11-02T18:52:39.567 DEBUG:boto:Method: GET 2023-11-02T18:52:39.568 DEBUG:boto:Path: /ibtljn-61/obj185 2023-11-02T18:52:39.568 DEBUG:boto:Data: 2023-11-02T18:52:39.569 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.569 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.570 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.570 DEBUG:boto:Params: {} 2023-11-02T18:52:39.571 DEBUG:boto:Token: None 2023-11-02T18:52:39.571 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj185 2023-11-02T18:52:39.572 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cDlkxB3lPcyU3Buw9S/AFpZkxZw= 2023-11-02T18:52:39.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cDlkxB3lPcyU3Buw9S/AFpZkxZw=', 'Content-Length': '0'} 2023-11-02T18:52:39.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b1f3ece142d99ba-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.576 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.577 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.577 DEBUG:boto:path=/ibtljn-61/?prefix=obj185 2023-11-02T18:52:39.577 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj185 2023-11-02T18:52:39.578 DEBUG:boto:Method: GET 2023-11-02T18:52:39.578 DEBUG:boto:Path: /ibtljn-61/?prefix=obj185 2023-11-02T18:52:39.579 DEBUG:boto:Data: 2023-11-02T18:52:39.579 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.580 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.580 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.580 DEBUG:boto:Params: {} 2023-11-02T18:52:39.581 DEBUG:boto:Token: None 2023-11-02T18:52:39.581 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.582 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbb726663b9afefd-006543eff7-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.590 DEBUG:boto:b'ibtljn-61obj1851000falseobj1852023-11-02T18:50:25.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.591 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.592 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.592 DEBUG:boto:path=/ibtljn-61/?prefix=obj185 2023-11-02T18:52:39.593 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj185 2023-11-02T18:52:39.593 DEBUG:boto:Method: GET 2023-11-02T18:52:39.594 DEBUG:boto:Path: /ibtljn-61/?prefix=obj185 2023-11-02T18:52:39.594 DEBUG:boto:Data: 2023-11-02T18:52:39.595 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.595 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.596 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.596 DEBUG:boto:Params: {} 2023-11-02T18:52:39.597 DEBUG:boto:Token: None 2023-11-02T18:52:39.598 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.598 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f921e704dd082820-006543eff7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.650 DEBUG:boto:b'ibtljn-61obj1851000falseobj1852023-11-02T18:50:25.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:52:39.652 DEBUG:boto:path=/ibtljn-61/obj19 2023-11-02T18:52:39.653 DEBUG:boto:auth_path=/ibtljn-61/obj19 2023-11-02T18:52:39.653 DEBUG:boto:path=/ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.654 DEBUG:boto:auth_path=/ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.654 DEBUG:boto:Method: GET 2023-11-02T18:52:39.655 DEBUG:boto:Path: /ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.655 DEBUG:boto:Data: 2023-11-02T18:52:39.656 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.656 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.657 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.657 DEBUG:boto:Params: {} 2023-11-02T18:52:39.658 DEBUG:boto:Token: None 2023-11-02T18:52:39.659 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.659 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Xy6N6Kj1Vj1BGkhjx1ivA8d5IfA= 2023-11-02T18:52:39.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Xy6N6Kj1Vj1BGkhjx1ivA8d5IfA=', 'Content-Length': '0'} 2023-11-02T18:52:39.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069c80214ece30dd1-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.666 DEBUG:boto:path=/ibtljn-61/obj19 2023-11-02T18:52:39.666 DEBUG:boto:auth_path=/ibtljn-61/obj19 2023-11-02T18:52:39.667 DEBUG:boto:path=/ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.667 DEBUG:boto:auth_path=/ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.668 DEBUG:boto:Method: GET 2023-11-02T18:52:39.668 DEBUG:boto:Path: /ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.669 DEBUG:boto:Data: 2023-11-02T18:52:39.669 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.669 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.670 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.670 DEBUG:boto:Params: {} 2023-11-02T18:52:39.671 DEBUG:boto:Token: None 2023-11-02T18:52:39.672 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.672 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Xy6N6Kj1Vj1BGkhjx1ivA8d5IfA= 2023-11-02T18:52:39.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Xy6N6Kj1Vj1BGkhjx1ivA8d5IfA=', 'Content-Length': '0'} 2023-11-02T18:52:39.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015b1911a2294976a-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.676 DEBUG:boto:path=/ibtljn-61/obj19 2023-11-02T18:52:39.676 DEBUG:boto:auth_path=/ibtljn-61/obj19 2023-11-02T18:52:39.677 DEBUG:boto:path=/ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.677 DEBUG:boto:auth_path=/ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.678 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.678 DEBUG:boto:Path: /ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.679 DEBUG:boto:Data: 2023-11-02T18:52:39.679 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.680 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.680 DEBUG:boto:Params: {} 2023-11-02T18:52:39.681 DEBUG:boto:Token: None 2023-11-02T18:52:39.681 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.682 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zCufwq+H8AoY74UB4+GOyrIxCyc= 2023-11-02T18:52:39.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zCufwq+H8AoY74UB4+GOyrIxCyc=', 'Content-Length': '0'} 2023-11-02T18:52:39.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5522b17b5d1283d-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.685 DEBUG:boto:path=/ibtljn-61/obj19 2023-11-02T18:52:39.686 DEBUG:boto:auth_path=/ibtljn-61/obj19 2023-11-02T18:52:39.686 DEBUG:boto:path=/ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.687 DEBUG:boto:auth_path=/ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.687 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.688 DEBUG:boto:Path: /ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.688 DEBUG:boto:Data: 2023-11-02T18:52:39.688 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.689 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.689 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.690 DEBUG:boto:Params: {} 2023-11-02T18:52:39.690 DEBUG:boto:Token: None 2023-11-02T18:52:39.691 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.691 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zCufwq+H8AoY74UB4+GOyrIxCyc= 2023-11-02T18:52:39.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zCufwq+H8AoY74UB4+GOyrIxCyc=', 'Content-Length': '0'} 2023-11-02T18:52:39.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f245574a03b9e408-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:52:39.695 DEBUG:boto:path=/ibtljn-61/obj19 2023-11-02T18:52:39.695 DEBUG:boto:auth_path=/ibtljn-61/obj19 2023-11-02T18:52:39.696 DEBUG:boto:path=/ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.696 DEBUG:boto:auth_path=/ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.697 DEBUG:boto:Method: GET 2023-11-02T18:52:39.697 DEBUG:boto:Path: /ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.698 DEBUG:boto:Data: 2023-11-02T18:52:39.698 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.699 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.699 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.700 DEBUG:boto:Params: {} 2023-11-02T18:52:39.700 DEBUG:boto:Token: None 2023-11-02T18:52:39.701 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.701 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Xy6N6Kj1Vj1BGkhjx1ivA8d5IfA= 2023-11-02T18:52:39.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Xy6N6Kj1Vj1BGkhjx1ivA8d5IfA=', 'Content-Length': '0'} 2023-11-02T18:52:39.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9cc3aae7713b1d5-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.705 DEBUG:boto:path=/ibtljn-61/obj19 2023-11-02T18:52:39.706 DEBUG:boto:auth_path=/ibtljn-61/obj19 2023-11-02T18:52:39.706 DEBUG:boto:path=/ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.707 DEBUG:boto:auth_path=/ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.707 DEBUG:boto:Method: GET 2023-11-02T18:52:39.707 DEBUG:boto:Path: /ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.708 DEBUG:boto:Data: 2023-11-02T18:52:39.708 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.709 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.709 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.710 DEBUG:boto:Params: {} 2023-11-02T18:52:39.710 DEBUG:boto:Token: None 2023-11-02T18:52:39.711 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj19?versionId=null 2023-11-02T18:52:39.711 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Xy6N6Kj1Vj1BGkhjx1ivA8d5IfA= 2023-11-02T18:52:39.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Xy6N6Kj1Vj1BGkhjx1ivA8d5IfA=', 'Content-Length': '0'} 2023-11-02T18:52:39.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1958b329b787b90-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.716 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.716 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.717 DEBUG:boto:path=/ibtljn-61/?prefix=obj19 2023-11-02T18:52:39.717 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj19 2023-11-02T18:52:39.718 DEBUG:boto:Method: GET 2023-11-02T18:52:39.718 DEBUG:boto:Path: /ibtljn-61/?prefix=obj19 2023-11-02T18:52:39.719 DEBUG:boto:Data: 2023-11-02T18:52:39.719 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.720 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.720 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.721 DEBUG:boto:Params: {} 2023-11-02T18:52:39.721 DEBUG:boto:Token: None 2023-11-02T18:52:39.722 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.722 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9f6c26886d37e3d-006543eff7-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.728 DEBUG:boto:b'ibtljn-61obj191000falseobj192023-11-02T18:50:22.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.729 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.729 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.730 DEBUG:boto:path=/ibtljn-61/?prefix=obj19 2023-11-02T18:52:39.730 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj19 2023-11-02T18:52:39.731 DEBUG:boto:Method: GET 2023-11-02T18:52:39.731 DEBUG:boto:Path: /ibtljn-61/?prefix=obj19 2023-11-02T18:52:39.732 DEBUG:boto:Data: 2023-11-02T18:52:39.732 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.732 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.733 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.733 DEBUG:boto:Params: {} 2023-11-02T18:52:39.734 DEBUG:boto:Token: None 2023-11-02T18:52:39.734 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.735 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fdb17cc5284f620-006543eff7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.790 DEBUG:boto:b'ibtljn-61obj191000falseobj192023-11-02T18:50:22.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.791 DEBUG:boto:path=/ibtljn-61/obj19 2023-11-02T18:52:39.792 DEBUG:boto:auth_path=/ibtljn-61/obj19 2023-11-02T18:52:39.792 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.793 DEBUG:boto:Path: /ibtljn-61/obj19 2023-11-02T18:52:39.793 DEBUG:boto:Data: 2023-11-02T18:52:39.794 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.794 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.795 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.795 DEBUG:boto:Params: {} 2023-11-02T18:52:39.796 DEBUG:boto:Token: None 2023-11-02T18:52:39.796 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj19 2023-11-02T18:52:39.797 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4iYAAl3xPekWeZdI1XebBp5GB7w= 2023-11-02T18:52:39.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4iYAAl3xPekWeZdI1XebBp5GB7w=', 'Content-Length': '0'} 2023-11-02T18:52:39.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0deaa7a319d0b96-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.801 DEBUG:boto:path=/ibtljn-61/obj19 2023-11-02T18:52:39.801 DEBUG:boto:auth_path=/ibtljn-61/obj19 2023-11-02T18:52:39.802 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.802 DEBUG:boto:Path: /ibtljn-61/obj19 2023-11-02T18:52:39.802 DEBUG:boto:Data: 2023-11-02T18:52:39.803 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.803 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.804 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.804 DEBUG:boto:Params: {} 2023-11-02T18:52:39.805 DEBUG:boto:Token: None 2023-11-02T18:52:39.805 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj19 2023-11-02T18:52:39.806 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4iYAAl3xPekWeZdI1XebBp5GB7w= 2023-11-02T18:52:39.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4iYAAl3xPekWeZdI1XebBp5GB7w=', 'Content-Length': '0'} 2023-11-02T18:52:39.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b81081b3a743160-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T18:52:39.810 DEBUG:boto:path=/ibtljn-61/obj19 2023-11-02T18:52:39.811 DEBUG:boto:auth_path=/ibtljn-61/obj19 2023-11-02T18:52:39.811 DEBUG:boto:Method: GET 2023-11-02T18:52:39.811 DEBUG:boto:Path: /ibtljn-61/obj19 2023-11-02T18:52:39.812 DEBUG:boto:Data: 2023-11-02T18:52:39.812 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.813 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.814 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.814 DEBUG:boto:Params: {} 2023-11-02T18:52:39.815 DEBUG:boto:Token: None 2023-11-02T18:52:39.815 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj19 2023-11-02T18:52:39.816 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rqC0kzVBdwIFLDk1UEW2m7QS39U= 2023-11-02T18:52:39.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rqC0kzVBdwIFLDk1UEW2m7QS39U=', 'Content-Length': '0'} 2023-11-02T18:52:39.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001874d7ad5c247b63-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.862 DEBUG:boto:path=/ibtljn-61/obj19 2023-11-02T18:52:39.863 DEBUG:boto:auth_path=/ibtljn-61/obj19 2023-11-02T18:52:39.864 DEBUG:boto:Method: GET 2023-11-02T18:52:39.864 DEBUG:boto:Path: /ibtljn-61/obj19 2023-11-02T18:52:39.865 DEBUG:boto:Data: 2023-11-02T18:52:39.865 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.866 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.866 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.867 DEBUG:boto:Params: {} 2023-11-02T18:52:39.867 DEBUG:boto:Token: None 2023-11-02T18:52:39.868 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj19 2023-11-02T18:52:39.869 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rqC0kzVBdwIFLDk1UEW2m7QS39U= 2023-11-02T18:52:39.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rqC0kzVBdwIFLDk1UEW2m7QS39U=', 'Content-Length': '0'} 2023-11-02T18:52:39.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076d989ac2c56d384-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.873 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.874 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.874 DEBUG:boto:path=/ibtljn-61/?prefix=obj19 2023-11-02T18:52:39.875 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj19 2023-11-02T18:52:39.875 DEBUG:boto:Method: GET 2023-11-02T18:52:39.876 DEBUG:boto:Path: /ibtljn-61/?prefix=obj19 2023-11-02T18:52:39.876 DEBUG:boto:Data: 2023-11-02T18:52:39.877 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.877 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.878 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.878 DEBUG:boto:Params: {} 2023-11-02T18:52:39.879 DEBUG:boto:Token: None 2023-11-02T18:52:39.879 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.880 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d20ab5c83243a8a-006543eff7-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.886 DEBUG:boto:b'ibtljn-61obj191000falseobj192023-11-02T18:50:22.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.887 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:39.888 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:39.888 DEBUG:boto:path=/ibtljn-61/?prefix=obj19 2023-11-02T18:52:39.889 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj19 2023-11-02T18:52:39.889 DEBUG:boto:Method: GET 2023-11-02T18:52:39.889 DEBUG:boto:Path: /ibtljn-61/?prefix=obj19 2023-11-02T18:52:39.890 DEBUG:boto:Data: 2023-11-02T18:52:39.890 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.891 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.891 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.892 DEBUG:boto:Params: {} 2023-11-02T18:52:39.892 DEBUG:boto:Token: None 2023-11-02T18:52:39.893 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/ 2023-11-02T18:52:39.893 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8= 2023-11-02T18:52:39.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iZgcA5QqrcHErq9RUZjYq1BcCP8=', 'Content-Length': '0'} 2023-11-02T18:52:39.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c6581ac86b51cea-006543eff7-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.905 DEBUG:boto:b'ibtljn-61obj191000falseobj192023-11-02T18:50:22.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:39.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:52:39.906 DEBUG:boto:path=/ibtljn-61/obj22 2023-11-02T18:52:39.907 DEBUG:boto:auth_path=/ibtljn-61/obj22 2023-11-02T18:52:39.907 DEBUG:boto:path=/ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.908 DEBUG:boto:auth_path=/ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.908 DEBUG:boto:Method: GET 2023-11-02T18:52:39.909 DEBUG:boto:Path: /ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.909 DEBUG:boto:Data: 2023-11-02T18:52:39.910 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.911 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.911 DEBUG:boto:Params: {} 2023-11-02T18:52:39.912 DEBUG:boto:Token: None 2023-11-02T18:52:39.912 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.913 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yaU6pdFnYFqslWKYPJNUssdsRo4= 2023-11-02T18:52:39.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yaU6pdFnYFqslWKYPJNUssdsRo4=', 'Content-Length': '0'} 2023-11-02T18:52:39.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f8e512af2ade00a-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.917 DEBUG:boto:path=/ibtljn-61/obj22 2023-11-02T18:52:39.918 DEBUG:boto:auth_path=/ibtljn-61/obj22 2023-11-02T18:52:39.918 DEBUG:boto:path=/ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.919 DEBUG:boto:auth_path=/ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.919 DEBUG:boto:Method: GET 2023-11-02T18:52:39.920 DEBUG:boto:Path: /ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.920 DEBUG:boto:Data: 2023-11-02T18:52:39.920 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.921 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.921 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.922 DEBUG:boto:Params: {} 2023-11-02T18:52:39.922 DEBUG:boto:Token: None 2023-11-02T18:52:39.923 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.923 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yaU6pdFnYFqslWKYPJNUssdsRo4= 2023-11-02T18:52:39.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yaU6pdFnYFqslWKYPJNUssdsRo4=', 'Content-Length': '0'} 2023-11-02T18:52:39.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041c70752dc489f69-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.970 DEBUG:boto:path=/ibtljn-61/obj22 2023-11-02T18:52:39.971 DEBUG:boto:auth_path=/ibtljn-61/obj22 2023-11-02T18:52:39.971 DEBUG:boto:path=/ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.972 DEBUG:boto:auth_path=/ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.972 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.973 DEBUG:boto:Path: /ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.973 DEBUG:boto:Data: 2023-11-02T18:52:39.974 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.974 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.974 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.975 DEBUG:boto:Params: {} 2023-11-02T18:52:39.975 DEBUG:boto:Token: None 2023-11-02T18:52:39.976 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.976 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ht9zmY7DHXefxRMt5cL/46nfhCw= 2023-11-02T18:52:39.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ht9zmY7DHXefxRMt5cL/46nfhCw=', 'Content-Length': '0'} 2023-11-02T18:52:39.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081fb24ae165ff52a-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.980 DEBUG:boto:path=/ibtljn-61/obj22 2023-11-02T18:52:39.981 DEBUG:boto:auth_path=/ibtljn-61/obj22 2023-11-02T18:52:39.981 DEBUG:boto:path=/ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.982 DEBUG:boto:auth_path=/ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.982 DEBUG:boto:Method: HEAD 2023-11-02T18:52:39.983 DEBUG:boto:Path: /ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.983 DEBUG:boto:Data: 2023-11-02T18:52:39.983 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.984 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:39.984 DEBUG:boto:Port: 8000 2023-11-02T18:52:39.985 DEBUG:boto:Params: {} 2023-11-02T18:52:39.985 DEBUG:boto:Token: None 2023-11-02T18:52:39.986 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.986 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ht9zmY7DHXefxRMt5cL/46nfhCw= 2023-11-02T18:52:39.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ht9zmY7DHXefxRMt5cL/46nfhCw=', 'Content-Length': '0'} 2023-11-02T18:52:39.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014d350b4a76208af-006543eff7-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:39.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:52:39.991 DEBUG:boto:path=/ibtljn-61/obj22 2023-11-02T18:52:39.991 DEBUG:boto:auth_path=/ibtljn-61/obj22 2023-11-02T18:52:39.992 DEBUG:boto:path=/ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.992 DEBUG:boto:auth_path=/ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.992 DEBUG:boto:Method: GET 2023-11-02T18:52:39.993 DEBUG:boto:Path: /ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.993 DEBUG:boto:Data: 2023-11-02T18:52:39.994 DEBUG:boto:Headers: {} 2023-11-02T18:52:39.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:39.994 DEBUG:boto:Port: 8001 2023-11-02T18:52:39.995 DEBUG:boto:Params: {} 2023-11-02T18:52:39.995 DEBUG:boto:Token: None 2023-11-02T18:52:39.996 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:39 GMT /ibtljn-61/obj22?versionId=null 2023-11-02T18:52:39.996 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yaU6pdFnYFqslWKYPJNUssdsRo4= 2023-11-02T18:52:39.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:39 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yaU6pdFnYFqslWKYPJNUssdsRo4=', 'Content-Length': '0'} 2023-11-02T18:52:40.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000212d4cf9c57c91f2-006543eff7-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.042 DEBUG:boto:path=/ibtljn-61/obj22 2023-11-02T18:52:40.043 DEBUG:boto:auth_path=/ibtljn-61/obj22 2023-11-02T18:52:40.044 DEBUG:boto:path=/ibtljn-61/obj22?versionId=null 2023-11-02T18:52:40.044 DEBUG:boto:auth_path=/ibtljn-61/obj22?versionId=null 2023-11-02T18:52:40.044 DEBUG:boto:Method: GET 2023-11-02T18:52:40.045 DEBUG:boto:Path: /ibtljn-61/obj22?versionId=null 2023-11-02T18:52:40.045 DEBUG:boto:Data: 2023-11-02T18:52:40.046 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.047 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.047 DEBUG:boto:Params: {} 2023-11-02T18:52:40.048 DEBUG:boto:Token: None 2023-11-02T18:52:40.048 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj22?versionId=null 2023-11-02T18:52:40.049 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:h357qGb8hn57Q82nJE9ltxeFois= 2023-11-02T18:52:40.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h357qGb8hn57Q82nJE9ltxeFois=', 'Content-Length': '0'} 2023-11-02T18:52:40.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6b832a5132bdb4e-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.053 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.053 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.054 DEBUG:boto:path=/ibtljn-61/?prefix=obj22 2023-11-02T18:52:40.054 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj22 2023-11-02T18:52:40.055 DEBUG:boto:Method: GET 2023-11-02T18:52:40.055 DEBUG:boto:Path: /ibtljn-61/?prefix=obj22 2023-11-02T18:52:40.055 DEBUG:boto:Data: 2023-11-02T18:52:40.056 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.056 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.057 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.057 DEBUG:boto:Params: {} 2023-11-02T18:52:40.058 DEBUG:boto:Token: None 2023-11-02T18:52:40.058 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.059 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b570dd2d33d0fef4-006543eff8-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.066 DEBUG:boto:b'ibtljn-61obj221000falseobj222023-11-02T18:50:22.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.067 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.067 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.067 DEBUG:boto:path=/ibtljn-61/?prefix=obj22 2023-11-02T18:52:40.068 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj22 2023-11-02T18:52:40.068 DEBUG:boto:Method: GET 2023-11-02T18:52:40.069 DEBUG:boto:Path: /ibtljn-61/?prefix=obj22 2023-11-02T18:52:40.069 DEBUG:boto:Data: 2023-11-02T18:52:40.070 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.071 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.071 DEBUG:boto:Params: {} 2023-11-02T18:52:40.072 DEBUG:boto:Token: None 2023-11-02T18:52:40.072 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.073 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaabd4c74391195e-006543eff8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.083 DEBUG:boto:b'ibtljn-61obj221000falseobj222023-11-02T18:50:22.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.084 DEBUG:boto:path=/ibtljn-61/obj22 2023-11-02T18:52:40.085 DEBUG:boto:auth_path=/ibtljn-61/obj22 2023-11-02T18:52:40.085 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.086 DEBUG:boto:Path: /ibtljn-61/obj22 2023-11-02T18:52:40.086 DEBUG:boto:Data: 2023-11-02T18:52:40.087 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.087 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.087 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.088 DEBUG:boto:Params: {} 2023-11-02T18:52:40.088 DEBUG:boto:Token: None 2023-11-02T18:52:40.089 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj22 2023-11-02T18:52:40.089 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n39XtjR/AaykFWfKEtBZUvIl/9g= 2023-11-02T18:52:40.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n39XtjR/AaykFWfKEtBZUvIl/9g=', 'Content-Length': '0'} 2023-11-02T18:52:40.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd9fd76fdb760a31-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.093 DEBUG:boto:path=/ibtljn-61/obj22 2023-11-02T18:52:40.093 DEBUG:boto:auth_path=/ibtljn-61/obj22 2023-11-02T18:52:40.094 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.094 DEBUG:boto:Path: /ibtljn-61/obj22 2023-11-02T18:52:40.095 DEBUG:boto:Data: 2023-11-02T18:52:40.095 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.096 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.096 DEBUG:boto:Params: {} 2023-11-02T18:52:40.097 DEBUG:boto:Token: None 2023-11-02T18:52:40.097 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj22 2023-11-02T18:52:40.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n39XtjR/AaykFWfKEtBZUvIl/9g= 2023-11-02T18:52:40.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n39XtjR/AaykFWfKEtBZUvIl/9g=', 'Content-Length': '0'} 2023-11-02T18:52:40.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e35d328769cf6b5f-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:52:40.102 DEBUG:boto:path=/ibtljn-61/obj22 2023-11-02T18:52:40.103 DEBUG:boto:auth_path=/ibtljn-61/obj22 2023-11-02T18:52:40.104 DEBUG:boto:Method: GET 2023-11-02T18:52:40.104 DEBUG:boto:Path: /ibtljn-61/obj22 2023-11-02T18:52:40.105 DEBUG:boto:Data: 2023-11-02T18:52:40.106 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.108 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.108 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.109 DEBUG:boto:Params: {} 2023-11-02T18:52:40.109 DEBUG:boto:Token: None 2023-11-02T18:52:40.110 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj22 2023-11-02T18:52:40.111 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxIFNkA7fFGyDsJQDgbSbF/fmUM= 2023-11-02T18:52:40.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxIFNkA7fFGyDsJQDgbSbF/fmUM=', 'Content-Length': '0'} 2023-11-02T18:52:40.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7266c861aa7c061-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.114 DEBUG:boto:path=/ibtljn-61/obj22 2023-11-02T18:52:40.115 DEBUG:boto:auth_path=/ibtljn-61/obj22 2023-11-02T18:52:40.115 DEBUG:boto:Method: GET 2023-11-02T18:52:40.116 DEBUG:boto:Path: /ibtljn-61/obj22 2023-11-02T18:52:40.116 DEBUG:boto:Data: 2023-11-02T18:52:40.117 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.117 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.118 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.118 DEBUG:boto:Params: {} 2023-11-02T18:52:40.119 DEBUG:boto:Token: None 2023-11-02T18:52:40.119 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj22 2023-11-02T18:52:40.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BxIFNkA7fFGyDsJQDgbSbF/fmUM= 2023-11-02T18:52:40.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BxIFNkA7fFGyDsJQDgbSbF/fmUM=', 'Content-Length': '0'} 2023-11-02T18:52:40.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3fb79a175f2371c-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.166 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.167 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.168 DEBUG:boto:path=/ibtljn-61/?prefix=obj22 2023-11-02T18:52:40.168 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj22 2023-11-02T18:52:40.168 DEBUG:boto:Method: GET 2023-11-02T18:52:40.169 DEBUG:boto:Path: /ibtljn-61/?prefix=obj22 2023-11-02T18:52:40.169 DEBUG:boto:Data: 2023-11-02T18:52:40.170 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.170 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.171 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.171 DEBUG:boto:Params: {} 2023-11-02T18:52:40.172 DEBUG:boto:Token: None 2023-11-02T18:52:40.172 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.173 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065dffe1d91ca3dc4-006543eff8-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.179 DEBUG:boto:b'ibtljn-61obj221000falseobj222023-11-02T18:50:22.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.180 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.180 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.181 DEBUG:boto:path=/ibtljn-61/?prefix=obj22 2023-11-02T18:52:40.181 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj22 2023-11-02T18:52:40.182 DEBUG:boto:Method: GET 2023-11-02T18:52:40.182 DEBUG:boto:Path: /ibtljn-61/?prefix=obj22 2023-11-02T18:52:40.183 DEBUG:boto:Data: 2023-11-02T18:52:40.183 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.184 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.184 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.185 DEBUG:boto:Params: {} 2023-11-02T18:52:40.185 DEBUG:boto:Token: None 2023-11-02T18:52:40.186 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.186 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f03118900dc69222-006543eff8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.196 DEBUG:boto:b'ibtljn-61obj221000falseobj222023-11-02T18:50:22.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:52:40.197 DEBUG:boto:path=/ibtljn-61/obj23 2023-11-02T18:52:40.198 DEBUG:boto:auth_path=/ibtljn-61/obj23 2023-11-02T18:52:40.198 DEBUG:boto:path=/ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.199 DEBUG:boto:auth_path=/ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.199 DEBUG:boto:Method: GET 2023-11-02T18:52:40.200 DEBUG:boto:Path: /ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.200 DEBUG:boto:Data: 2023-11-02T18:52:40.201 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.201 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.201 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.202 DEBUG:boto:Params: {} 2023-11-02T18:52:40.202 DEBUG:boto:Token: None 2023-11-02T18:52:40.203 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.203 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s8Hc6MpVOevQSOqDcwCZjk5UBZU= 2023-11-02T18:52:40.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s8Hc6MpVOevQSOqDcwCZjk5UBZU=', 'Content-Length': '0'} 2023-11-02T18:52:40.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c99c954567abe80-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.250 DEBUG:boto:path=/ibtljn-61/obj23 2023-11-02T18:52:40.251 DEBUG:boto:auth_path=/ibtljn-61/obj23 2023-11-02T18:52:40.251 DEBUG:boto:path=/ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.252 DEBUG:boto:auth_path=/ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.252 DEBUG:boto:Method: GET 2023-11-02T18:52:40.253 DEBUG:boto:Path: /ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.253 DEBUG:boto:Data: 2023-11-02T18:52:40.254 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.254 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.255 DEBUG:boto:Params: {} 2023-11-02T18:52:40.255 DEBUG:boto:Token: None 2023-11-02T18:52:40.256 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.257 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s8Hc6MpVOevQSOqDcwCZjk5UBZU= 2023-11-02T18:52:40.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s8Hc6MpVOevQSOqDcwCZjk5UBZU=', 'Content-Length': '0'} 2023-11-02T18:52:40.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a182aee68f5d773-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.261 DEBUG:boto:path=/ibtljn-61/obj23 2023-11-02T18:52:40.262 DEBUG:boto:auth_path=/ibtljn-61/obj23 2023-11-02T18:52:40.262 DEBUG:boto:path=/ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.263 DEBUG:boto:auth_path=/ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.263 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.264 DEBUG:boto:Path: /ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.264 DEBUG:boto:Data: 2023-11-02T18:52:40.264 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.265 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.265 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.266 DEBUG:boto:Params: {} 2023-11-02T18:52:40.266 DEBUG:boto:Token: None 2023-11-02T18:52:40.267 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.267 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LVbd5JKrk5VOv5o78yKYB+zyCdE= 2023-11-02T18:52:40.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LVbd5JKrk5VOv5o78yKYB+zyCdE=', 'Content-Length': '0'} 2023-11-02T18:52:40.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004032bacf844376ed-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.271 DEBUG:boto:path=/ibtljn-61/obj23 2023-11-02T18:52:40.272 DEBUG:boto:auth_path=/ibtljn-61/obj23 2023-11-02T18:52:40.272 DEBUG:boto:path=/ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.272 DEBUG:boto:auth_path=/ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.273 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.273 DEBUG:boto:Path: /ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.274 DEBUG:boto:Data: 2023-11-02T18:52:40.274 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.275 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.275 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.276 DEBUG:boto:Params: {} 2023-11-02T18:52:40.276 DEBUG:boto:Token: None 2023-11-02T18:52:40.277 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.277 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LVbd5JKrk5VOv5o78yKYB+zyCdE= 2023-11-02T18:52:40.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LVbd5JKrk5VOv5o78yKYB+zyCdE=', 'Content-Length': '0'} 2023-11-02T18:52:40.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc510adb87fed803-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:52:40.282 DEBUG:boto:path=/ibtljn-61/obj23 2023-11-02T18:52:40.282 DEBUG:boto:auth_path=/ibtljn-61/obj23 2023-11-02T18:52:40.283 DEBUG:boto:path=/ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.283 DEBUG:boto:auth_path=/ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.283 DEBUG:boto:Method: GET 2023-11-02T18:52:40.284 DEBUG:boto:Path: /ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.284 DEBUG:boto:Data: 2023-11-02T18:52:40.285 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.285 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.286 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.286 DEBUG:boto:Params: {} 2023-11-02T18:52:40.287 DEBUG:boto:Token: None 2023-11-02T18:52:40.287 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.288 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s8Hc6MpVOevQSOqDcwCZjk5UBZU= 2023-11-02T18:52:40.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s8Hc6MpVOevQSOqDcwCZjk5UBZU=', 'Content-Length': '0'} 2023-11-02T18:52:40.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052690dc1146917ea-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.292 DEBUG:boto:path=/ibtljn-61/obj23 2023-11-02T18:52:40.292 DEBUG:boto:auth_path=/ibtljn-61/obj23 2023-11-02T18:52:40.293 DEBUG:boto:path=/ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.293 DEBUG:boto:auth_path=/ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.294 DEBUG:boto:Method: GET 2023-11-02T18:52:40.294 DEBUG:boto:Path: /ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.294 DEBUG:boto:Data: 2023-11-02T18:52:40.295 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.295 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.296 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.296 DEBUG:boto:Params: {} 2023-11-02T18:52:40.297 DEBUG:boto:Token: None 2023-11-02T18:52:40.297 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj23?versionId=null 2023-11-02T18:52:40.298 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s8Hc6MpVOevQSOqDcwCZjk5UBZU= 2023-11-02T18:52:40.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s8Hc6MpVOevQSOqDcwCZjk5UBZU=', 'Content-Length': '0'} 2023-11-02T18:52:40.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a64a74022b0c40e-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.302 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.302 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.303 DEBUG:boto:path=/ibtljn-61/?prefix=obj23 2023-11-02T18:52:40.303 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj23 2023-11-02T18:52:40.304 DEBUG:boto:Method: GET 2023-11-02T18:52:40.304 DEBUG:boto:Path: /ibtljn-61/?prefix=obj23 2023-11-02T18:52:40.304 DEBUG:boto:Data: 2023-11-02T18:52:40.305 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.305 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.306 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.306 DEBUG:boto:Params: {} 2023-11-02T18:52:40.307 DEBUG:boto:Token: None 2023-11-02T18:52:40.307 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.308 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072734f4396e631a6-006543eff8-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.354 DEBUG:boto:b'ibtljn-61obj231000falseobj232023-11-02T18:50:23.389Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.355 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.356 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.356 DEBUG:boto:path=/ibtljn-61/?prefix=obj23 2023-11-02T18:52:40.357 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj23 2023-11-02T18:52:40.357 DEBUG:boto:Method: GET 2023-11-02T18:52:40.358 DEBUG:boto:Path: /ibtljn-61/?prefix=obj23 2023-11-02T18:52:40.358 DEBUG:boto:Data: 2023-11-02T18:52:40.358 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.359 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.359 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.360 DEBUG:boto:Params: {} 2023-11-02T18:52:40.360 DEBUG:boto:Token: None 2023-11-02T18:52:40.361 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.361 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000460d7b08a5dc3868-006543eff8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.371 DEBUG:boto:b'ibtljn-61obj231000falseobj232023-11-02T18:50:23.389Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.371 DEBUG:boto:path=/ibtljn-61/obj23 2023-11-02T18:52:40.372 DEBUG:boto:auth_path=/ibtljn-61/obj23 2023-11-02T18:52:40.372 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.373 DEBUG:boto:Path: /ibtljn-61/obj23 2023-11-02T18:52:40.373 DEBUG:boto:Data: 2023-11-02T18:52:40.373 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.374 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.374 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.375 DEBUG:boto:Params: {} 2023-11-02T18:52:40.375 DEBUG:boto:Token: None 2023-11-02T18:52:40.376 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj23 2023-11-02T18:52:40.376 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nm5XA8ERlJ5uBdJl95Wc4GpsVJQ= 2023-11-02T18:52:40.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nm5XA8ERlJ5uBdJl95Wc4GpsVJQ=', 'Content-Length': '0'} 2023-11-02T18:52:40.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000904b5363417464d0-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.380 DEBUG:boto:path=/ibtljn-61/obj23 2023-11-02T18:52:40.380 DEBUG:boto:auth_path=/ibtljn-61/obj23 2023-11-02T18:52:40.381 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.381 DEBUG:boto:Path: /ibtljn-61/obj23 2023-11-02T18:52:40.382 DEBUG:boto:Data: 2023-11-02T18:52:40.382 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.383 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.383 DEBUG:boto:Params: {} 2023-11-02T18:52:40.384 DEBUG:boto:Token: None 2023-11-02T18:52:40.384 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj23 2023-11-02T18:52:40.385 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nm5XA8ERlJ5uBdJl95Wc4GpsVJQ= 2023-11-02T18:52:40.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nm5XA8ERlJ5uBdJl95Wc4GpsVJQ=', 'Content-Length': '0'} 2023-11-02T18:52:40.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008f5df71e86a37cb-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T18:52:40.389 DEBUG:boto:path=/ibtljn-61/obj23 2023-11-02T18:52:40.389 DEBUG:boto:auth_path=/ibtljn-61/obj23 2023-11-02T18:52:40.390 DEBUG:boto:Method: GET 2023-11-02T18:52:40.390 DEBUG:boto:Path: /ibtljn-61/obj23 2023-11-02T18:52:40.391 DEBUG:boto:Data: 2023-11-02T18:52:40.391 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.392 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.392 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.393 DEBUG:boto:Params: {} 2023-11-02T18:52:40.393 DEBUG:boto:Token: None 2023-11-02T18:52:40.394 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj23 2023-11-02T18:52:40.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c++h9pEKAP3aWZ+zpjQmwSZFG8Q= 2023-11-02T18:52:40.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c++h9pEKAP3aWZ+zpjQmwSZFG8Q=', 'Content-Length': '0'} 2023-11-02T18:52:40.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064452ac0e0e08c3c-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.398 DEBUG:boto:path=/ibtljn-61/obj23 2023-11-02T18:52:40.398 DEBUG:boto:auth_path=/ibtljn-61/obj23 2023-11-02T18:52:40.399 DEBUG:boto:Method: GET 2023-11-02T18:52:40.399 DEBUG:boto:Path: /ibtljn-61/obj23 2023-11-02T18:52:40.400 DEBUG:boto:Data: 2023-11-02T18:52:40.400 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.400 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.401 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.401 DEBUG:boto:Params: {} 2023-11-02T18:52:40.402 DEBUG:boto:Token: None 2023-11-02T18:52:40.402 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj23 2023-11-02T18:52:40.403 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c++h9pEKAP3aWZ+zpjQmwSZFG8Q= 2023-11-02T18:52:40.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c++h9pEKAP3aWZ+zpjQmwSZFG8Q=', 'Content-Length': '0'} 2023-11-02T18:52:40.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000424aace1e00ea846-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.450 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.451 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.451 DEBUG:boto:path=/ibtljn-61/?prefix=obj23 2023-11-02T18:52:40.452 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj23 2023-11-02T18:52:40.452 DEBUG:boto:Method: GET 2023-11-02T18:52:40.453 DEBUG:boto:Path: /ibtljn-61/?prefix=obj23 2023-11-02T18:52:40.453 DEBUG:boto:Data: 2023-11-02T18:52:40.454 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.455 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.455 DEBUG:boto:Params: {} 2023-11-02T18:52:40.456 DEBUG:boto:Token: None 2023-11-02T18:52:40.456 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.457 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b258d6b1038d664a-006543eff8-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.464 DEBUG:boto:b'ibtljn-61obj231000falseobj232023-11-02T18:50:23.389Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.465 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.466 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.466 DEBUG:boto:path=/ibtljn-61/?prefix=obj23 2023-11-02T18:52:40.466 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj23 2023-11-02T18:52:40.467 DEBUG:boto:Method: GET 2023-11-02T18:52:40.467 DEBUG:boto:Path: /ibtljn-61/?prefix=obj23 2023-11-02T18:52:40.468 DEBUG:boto:Data: 2023-11-02T18:52:40.468 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.469 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.469 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.470 DEBUG:boto:Params: {} 2023-11-02T18:52:40.470 DEBUG:boto:Token: None 2023-11-02T18:52:40.471 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.471 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a5389807b832f5b-006543eff8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.482 DEBUG:boto:b'ibtljn-61obj231000falseobj232023-11-02T18:50:23.389Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:52:40.484 DEBUG:boto:path=/ibtljn-61/obj29 2023-11-02T18:52:40.484 DEBUG:boto:auth_path=/ibtljn-61/obj29 2023-11-02T18:52:40.485 DEBUG:boto:path=/ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.485 DEBUG:boto:auth_path=/ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.485 DEBUG:boto:Method: GET 2023-11-02T18:52:40.486 DEBUG:boto:Path: /ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.486 DEBUG:boto:Data: 2023-11-02T18:52:40.487 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.487 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.488 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.488 DEBUG:boto:Params: {} 2023-11-02T18:52:40.489 DEBUG:boto:Token: None 2023-11-02T18:52:40.489 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.490 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2EJk53IpLEfsnq2xFsqt6tHmP4g= 2023-11-02T18:52:40.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2EJk53IpLEfsnq2xFsqt6tHmP4g=', 'Content-Length': '0'} 2023-11-02T18:52:40.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7e1e92741a06c51-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.535 DEBUG:boto:path=/ibtljn-61/obj29 2023-11-02T18:52:40.535 DEBUG:boto:auth_path=/ibtljn-61/obj29 2023-11-02T18:52:40.536 DEBUG:boto:path=/ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.536 DEBUG:boto:auth_path=/ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.537 DEBUG:boto:Method: GET 2023-11-02T18:52:40.537 DEBUG:boto:Path: /ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.538 DEBUG:boto:Data: 2023-11-02T18:52:40.538 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.539 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.539 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.540 DEBUG:boto:Params: {} 2023-11-02T18:52:40.540 DEBUG:boto:Token: None 2023-11-02T18:52:40.541 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.541 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2EJk53IpLEfsnq2xFsqt6tHmP4g= 2023-11-02T18:52:40.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2EJk53IpLEfsnq2xFsqt6tHmP4g=', 'Content-Length': '0'} 2023-11-02T18:52:40.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000288e01c663f5c3c6-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.545 DEBUG:boto:path=/ibtljn-61/obj29 2023-11-02T18:52:40.546 DEBUG:boto:auth_path=/ibtljn-61/obj29 2023-11-02T18:52:40.546 DEBUG:boto:path=/ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.546 DEBUG:boto:auth_path=/ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.547 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.547 DEBUG:boto:Path: /ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.548 DEBUG:boto:Data: 2023-11-02T18:52:40.548 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.549 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.549 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.549 DEBUG:boto:Params: {} 2023-11-02T18:52:40.550 DEBUG:boto:Token: None 2023-11-02T18:52:40.551 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.551 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uevmZ2lqzy6p1iztmM+JI3QVMdU= 2023-11-02T18:52:40.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uevmZ2lqzy6p1iztmM+JI3QVMdU=', 'Content-Length': '0'} 2023-11-02T18:52:40.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000132f524ac2ca8002-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.555 DEBUG:boto:path=/ibtljn-61/obj29 2023-11-02T18:52:40.555 DEBUG:boto:auth_path=/ibtljn-61/obj29 2023-11-02T18:52:40.556 DEBUG:boto:path=/ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.556 DEBUG:boto:auth_path=/ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.557 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.557 DEBUG:boto:Path: /ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.558 DEBUG:boto:Data: 2023-11-02T18:52:40.558 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.558 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.559 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.559 DEBUG:boto:Params: {} 2023-11-02T18:52:40.560 DEBUG:boto:Token: None 2023-11-02T18:52:40.560 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.561 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uevmZ2lqzy6p1iztmM+JI3QVMdU= 2023-11-02T18:52:40.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uevmZ2lqzy6p1iztmM+JI3QVMdU=', 'Content-Length': '0'} 2023-11-02T18:52:40.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000571946ac9c7a134c-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:52:40.565 DEBUG:boto:path=/ibtljn-61/obj29 2023-11-02T18:52:40.566 DEBUG:boto:auth_path=/ibtljn-61/obj29 2023-11-02T18:52:40.566 DEBUG:boto:path=/ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.566 DEBUG:boto:auth_path=/ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.567 DEBUG:boto:Method: GET 2023-11-02T18:52:40.567 DEBUG:boto:Path: /ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.568 DEBUG:boto:Data: 2023-11-02T18:52:40.568 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.569 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.569 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.569 DEBUG:boto:Params: {} 2023-11-02T18:52:40.570 DEBUG:boto:Token: None 2023-11-02T18:52:40.570 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.571 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2EJk53IpLEfsnq2xFsqt6tHmP4g= 2023-11-02T18:52:40.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2EJk53IpLEfsnq2xFsqt6tHmP4g=', 'Content-Length': '0'} 2023-11-02T18:52:40.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000535a6f1c5f010fd5-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.575 DEBUG:boto:path=/ibtljn-61/obj29 2023-11-02T18:52:40.576 DEBUG:boto:auth_path=/ibtljn-61/obj29 2023-11-02T18:52:40.576 DEBUG:boto:path=/ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.577 DEBUG:boto:auth_path=/ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.577 DEBUG:boto:Method: GET 2023-11-02T18:52:40.578 DEBUG:boto:Path: /ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.578 DEBUG:boto:Data: 2023-11-02T18:52:40.578 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.579 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.579 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.580 DEBUG:boto:Params: {} 2023-11-02T18:52:40.580 DEBUG:boto:Token: None 2023-11-02T18:52:40.581 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj29?versionId=null 2023-11-02T18:52:40.581 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2EJk53IpLEfsnq2xFsqt6tHmP4g= 2023-11-02T18:52:40.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2EJk53IpLEfsnq2xFsqt6tHmP4g=', 'Content-Length': '0'} 2023-11-02T18:52:40.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8c15e6f3b3bd9f7-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.586 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.586 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.586 DEBUG:boto:path=/ibtljn-61/?prefix=obj29 2023-11-02T18:52:40.587 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj29 2023-11-02T18:52:40.587 DEBUG:boto:Method: GET 2023-11-02T18:52:40.588 DEBUG:boto:Path: /ibtljn-61/?prefix=obj29 2023-11-02T18:52:40.588 DEBUG:boto:Data: 2023-11-02T18:52:40.589 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.589 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.589 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.590 DEBUG:boto:Params: {} 2023-11-02T18:52:40.590 DEBUG:boto:Token: None 2023-11-02T18:52:40.591 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.591 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8cf3db5d87f77e6-006543eff8-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.638 DEBUG:boto:b'ibtljn-61obj291000falseobj292023-11-02T18:50:23.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.640 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.640 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.640 DEBUG:boto:path=/ibtljn-61/?prefix=obj29 2023-11-02T18:52:40.641 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj29 2023-11-02T18:52:40.641 DEBUG:boto:Method: GET 2023-11-02T18:52:40.642 DEBUG:boto:Path: /ibtljn-61/?prefix=obj29 2023-11-02T18:52:40.642 DEBUG:boto:Data: 2023-11-02T18:52:40.643 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.643 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.644 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.644 DEBUG:boto:Params: {} 2023-11-02T18:52:40.644 DEBUG:boto:Token: None 2023-11-02T18:52:40.645 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c4e7c33f83f33c6-006543eff8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.657 DEBUG:boto:b'ibtljn-61obj291000falseobj292023-11-02T18:50:23.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.658 DEBUG:boto:path=/ibtljn-61/obj29 2023-11-02T18:52:40.658 DEBUG:boto:auth_path=/ibtljn-61/obj29 2023-11-02T18:52:40.659 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.660 DEBUG:boto:Path: /ibtljn-61/obj29 2023-11-02T18:52:40.660 DEBUG:boto:Data: 2023-11-02T18:52:40.660 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.661 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.661 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.662 DEBUG:boto:Params: {} 2023-11-02T18:52:40.662 DEBUG:boto:Token: None 2023-11-02T18:52:40.663 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj29 2023-11-02T18:52:40.664 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tHr0WxsFcoSJTE5/9qt+W5mcazY= 2023-11-02T18:52:40.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tHr0WxsFcoSJTE5/9qt+W5mcazY=', 'Content-Length': '0'} 2023-11-02T18:52:40.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062946b8df1f22cc2-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.668 DEBUG:boto:path=/ibtljn-61/obj29 2023-11-02T18:52:40.668 DEBUG:boto:auth_path=/ibtljn-61/obj29 2023-11-02T18:52:40.668 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.669 DEBUG:boto:Path: /ibtljn-61/obj29 2023-11-02T18:52:40.669 DEBUG:boto:Data: 2023-11-02T18:52:40.670 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.670 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.671 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.671 DEBUG:boto:Params: {} 2023-11-02T18:52:40.672 DEBUG:boto:Token: None 2023-11-02T18:52:40.672 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj29 2023-11-02T18:52:40.673 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tHr0WxsFcoSJTE5/9qt+W5mcazY= 2023-11-02T18:52:40.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tHr0WxsFcoSJTE5/9qt+W5mcazY=', 'Content-Length': '0'} 2023-11-02T18:52:40.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071c8e84c32f3c6ec-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T18:52:40.677 DEBUG:boto:path=/ibtljn-61/obj29 2023-11-02T18:52:40.678 DEBUG:boto:auth_path=/ibtljn-61/obj29 2023-11-02T18:52:40.678 DEBUG:boto:Method: GET 2023-11-02T18:52:40.679 DEBUG:boto:Path: /ibtljn-61/obj29 2023-11-02T18:52:40.679 DEBUG:boto:Data: 2023-11-02T18:52:40.680 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.680 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.681 DEBUG:boto:Params: {} 2023-11-02T18:52:40.681 DEBUG:boto:Token: None 2023-11-02T18:52:40.682 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj29 2023-11-02T18:52:40.682 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2e5EO9DnXXAKTfDgq0bLfHcpc4U= 2023-11-02T18:52:40.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2e5EO9DnXXAKTfDgq0bLfHcpc4U=', 'Content-Length': '0'} 2023-11-02T18:52:40.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b05b82b7ce74581-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.687 DEBUG:boto:path=/ibtljn-61/obj29 2023-11-02T18:52:40.687 DEBUG:boto:auth_path=/ibtljn-61/obj29 2023-11-02T18:52:40.688 DEBUG:boto:Method: GET 2023-11-02T18:52:40.688 DEBUG:boto:Path: /ibtljn-61/obj29 2023-11-02T18:52:40.689 DEBUG:boto:Data: 2023-11-02T18:52:40.689 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.690 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.691 DEBUG:boto:Params: {} 2023-11-02T18:52:40.691 DEBUG:boto:Token: None 2023-11-02T18:52:40.692 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj29 2023-11-02T18:52:40.692 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2e5EO9DnXXAKTfDgq0bLfHcpc4U= 2023-11-02T18:52:40.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2e5EO9DnXXAKTfDgq0bLfHcpc4U=', 'Content-Length': '0'} 2023-11-02T18:52:40.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002285128a92f4e23d-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.742 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.743 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.743 DEBUG:boto:path=/ibtljn-61/?prefix=obj29 2023-11-02T18:52:40.744 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj29 2023-11-02T18:52:40.744 DEBUG:boto:Method: GET 2023-11-02T18:52:40.745 DEBUG:boto:Path: /ibtljn-61/?prefix=obj29 2023-11-02T18:52:40.745 DEBUG:boto:Data: 2023-11-02T18:52:40.746 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.746 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.747 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.747 DEBUG:boto:Params: {} 2023-11-02T18:52:40.748 DEBUG:boto:Token: None 2023-11-02T18:52:40.748 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.749 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd2503941d8ad5a1-006543eff8-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.755 DEBUG:boto:b'ibtljn-61obj291000falseobj292023-11-02T18:50:23.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.756 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.757 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.758 DEBUG:boto:path=/ibtljn-61/?prefix=obj29 2023-11-02T18:52:40.758 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj29 2023-11-02T18:52:40.759 DEBUG:boto:Method: GET 2023-11-02T18:52:40.759 DEBUG:boto:Path: /ibtljn-61/?prefix=obj29 2023-11-02T18:52:40.760 DEBUG:boto:Data: 2023-11-02T18:52:40.760 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.761 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.761 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.762 DEBUG:boto:Params: {} 2023-11-02T18:52:40.762 DEBUG:boto:Token: None 2023-11-02T18:52:40.763 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.763 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026de42d5f8737f06-006543eff8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.775 DEBUG:boto:b'ibtljn-61obj291000falseobj292023-11-02T18:50:23.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T18:52:40.776 DEBUG:boto:path=/ibtljn-61/obj31 2023-11-02T18:52:40.776 DEBUG:boto:auth_path=/ibtljn-61/obj31 2023-11-02T18:52:40.777 DEBUG:boto:path=/ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.777 DEBUG:boto:auth_path=/ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.778 DEBUG:boto:Method: GET 2023-11-02T18:52:40.778 DEBUG:boto:Path: /ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.778 DEBUG:boto:Data: 2023-11-02T18:52:40.779 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.779 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.780 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.780 DEBUG:boto:Params: {} 2023-11-02T18:52:40.781 DEBUG:boto:Token: None 2023-11-02T18:52:40.781 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.782 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wiS0C63drPC+pAb8/BdD73J9JLI= 2023-11-02T18:52:40.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wiS0C63drPC+pAb8/BdD73J9JLI=', 'Content-Length': '0'} 2023-11-02T18:52:40.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c899847d8b2fe307-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.826 DEBUG:boto:path=/ibtljn-61/obj31 2023-11-02T18:52:40.827 DEBUG:boto:auth_path=/ibtljn-61/obj31 2023-11-02T18:52:40.827 DEBUG:boto:path=/ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.828 DEBUG:boto:auth_path=/ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.828 DEBUG:boto:Method: GET 2023-11-02T18:52:40.829 DEBUG:boto:Path: /ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.829 DEBUG:boto:Data: 2023-11-02T18:52:40.830 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.830 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.831 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.831 DEBUG:boto:Params: {} 2023-11-02T18:52:40.832 DEBUG:boto:Token: None 2023-11-02T18:52:40.832 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.833 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wiS0C63drPC+pAb8/BdD73J9JLI= 2023-11-02T18:52:40.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wiS0C63drPC+pAb8/BdD73J9JLI=', 'Content-Length': '0'} 2023-11-02T18:52:40.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000092432fb2a8d7894-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.837 DEBUG:boto:path=/ibtljn-61/obj31 2023-11-02T18:52:40.837 DEBUG:boto:auth_path=/ibtljn-61/obj31 2023-11-02T18:52:40.838 DEBUG:boto:path=/ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.838 DEBUG:boto:auth_path=/ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.838 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.839 DEBUG:boto:Path: /ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.840 DEBUG:boto:Data: 2023-11-02T18:52:40.840 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.840 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.841 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.841 DEBUG:boto:Params: {} 2023-11-02T18:52:40.842 DEBUG:boto:Token: None 2023-11-02T18:52:40.842 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.843 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BFs0l0oEGRQ02Sbsth2U3JZ8mok= 2023-11-02T18:52:40.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BFs0l0oEGRQ02Sbsth2U3JZ8mok=', 'Content-Length': '0'} 2023-11-02T18:52:40.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000727d44f4b280c9da-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.847 DEBUG:boto:path=/ibtljn-61/obj31 2023-11-02T18:52:40.847 DEBUG:boto:auth_path=/ibtljn-61/obj31 2023-11-02T18:52:40.848 DEBUG:boto:path=/ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.848 DEBUG:boto:auth_path=/ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.849 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.849 DEBUG:boto:Path: /ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.850 DEBUG:boto:Data: 2023-11-02T18:52:40.850 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.851 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.851 DEBUG:boto:Params: {} 2023-11-02T18:52:40.852 DEBUG:boto:Token: None 2023-11-02T18:52:40.852 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.853 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BFs0l0oEGRQ02Sbsth2U3JZ8mok= 2023-11-02T18:52:40.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BFs0l0oEGRQ02Sbsth2U3JZ8mok=', 'Content-Length': '0'} 2023-11-02T18:52:40.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e2a31e14fe3c1e-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T18:52:40.859 DEBUG:boto:path=/ibtljn-61/obj31 2023-11-02T18:52:40.859 DEBUG:boto:auth_path=/ibtljn-61/obj31 2023-11-02T18:52:40.859 DEBUG:boto:path=/ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.860 DEBUG:boto:auth_path=/ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.860 DEBUG:boto:Method: GET 2023-11-02T18:52:40.861 DEBUG:boto:Path: /ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.861 DEBUG:boto:Data: 2023-11-02T18:52:40.862 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.862 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.862 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.863 DEBUG:boto:Params: {} 2023-11-02T18:52:40.864 DEBUG:boto:Token: None 2023-11-02T18:52:40.864 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.865 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wiS0C63drPC+pAb8/BdD73J9JLI= 2023-11-02T18:52:40.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wiS0C63drPC+pAb8/BdD73J9JLI=', 'Content-Length': '0'} 2023-11-02T18:52:40.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000623c27ac2db228b9-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.869 DEBUG:boto:path=/ibtljn-61/obj31 2023-11-02T18:52:40.870 DEBUG:boto:auth_path=/ibtljn-61/obj31 2023-11-02T18:52:40.870 DEBUG:boto:path=/ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.871 DEBUG:boto:auth_path=/ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.872 DEBUG:boto:Method: GET 2023-11-02T18:52:40.872 DEBUG:boto:Path: /ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.873 DEBUG:boto:Data: 2023-11-02T18:52:40.874 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.874 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.875 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.876 DEBUG:boto:Params: {} 2023-11-02T18:52:40.876 DEBUG:boto:Token: None 2023-11-02T18:52:40.877 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj31?versionId=null 2023-11-02T18:52:40.877 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wiS0C63drPC+pAb8/BdD73J9JLI= 2023-11-02T18:52:40.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wiS0C63drPC+pAb8/BdD73J9JLI=', 'Content-Length': '0'} 2023-11-02T18:52:40.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014bbb11f09e466e0-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.882 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.883 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.883 DEBUG:boto:path=/ibtljn-61/?prefix=obj31 2023-11-02T18:52:40.884 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj31 2023-11-02T18:52:40.884 DEBUG:boto:Method: GET 2023-11-02T18:52:40.885 DEBUG:boto:Path: /ibtljn-61/?prefix=obj31 2023-11-02T18:52:40.885 DEBUG:boto:Data: 2023-11-02T18:52:40.885 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.886 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.887 DEBUG:boto:Params: {} 2023-11-02T18:52:40.887 DEBUG:boto:Token: None 2023-11-02T18:52:40.888 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.888 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036545bb05f4e8d8b-006543eff8-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.938 DEBUG:boto:b'ibtljn-61obj311000falseobj312023-11-02T18:50:24.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.939 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:40.940 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:40.940 DEBUG:boto:path=/ibtljn-61/?prefix=obj31 2023-11-02T18:52:40.941 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj31 2023-11-02T18:52:40.941 DEBUG:boto:Method: GET 2023-11-02T18:52:40.941 DEBUG:boto:Path: /ibtljn-61/?prefix=obj31 2023-11-02T18:52:40.942 DEBUG:boto:Data: 2023-11-02T18:52:40.943 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.943 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.943 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.944 DEBUG:boto:Params: {} 2023-11-02T18:52:40.944 DEBUG:boto:Token: None 2023-11-02T18:52:40.945 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/ 2023-11-02T18:52:40.945 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co= 2023-11-02T18:52:40.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EtuD7bfdFUXAh3f2nN4AwA2i3Co=', 'Content-Length': '0'} 2023-11-02T18:52:40.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069c85b5d9aacf646-006543eff8-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.957 DEBUG:boto:b'ibtljn-61obj311000falseobj312023-11-02T18:50:24.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:40.958 DEBUG:boto:path=/ibtljn-61/obj31 2023-11-02T18:52:40.958 DEBUG:boto:auth_path=/ibtljn-61/obj31 2023-11-02T18:52:40.959 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.959 DEBUG:boto:Path: /ibtljn-61/obj31 2023-11-02T18:52:40.959 DEBUG:boto:Data: 2023-11-02T18:52:40.960 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.960 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.961 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.961 DEBUG:boto:Params: {} 2023-11-02T18:52:40.961 DEBUG:boto:Token: None 2023-11-02T18:52:40.962 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj31 2023-11-02T18:52:40.962 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uLoso1GLuSLMuCdVfeyRu8rMoas= 2023-11-02T18:52:40.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uLoso1GLuSLMuCdVfeyRu8rMoas=', 'Content-Length': '0'} 2023-11-02T18:52:40.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005884e5f41817b5ec-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.966 DEBUG:boto:path=/ibtljn-61/obj31 2023-11-02T18:52:40.967 DEBUG:boto:auth_path=/ibtljn-61/obj31 2023-11-02T18:52:40.967 DEBUG:boto:Method: HEAD 2023-11-02T18:52:40.968 DEBUG:boto:Path: /ibtljn-61/obj31 2023-11-02T18:52:40.968 DEBUG:boto:Data: 2023-11-02T18:52:40.968 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.969 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.969 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.970 DEBUG:boto:Params: {} 2023-11-02T18:52:40.970 DEBUG:boto:Token: None 2023-11-02T18:52:40.971 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj31 2023-11-02T18:52:40.971 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uLoso1GLuSLMuCdVfeyRu8rMoas= 2023-11-02T18:52:40.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uLoso1GLuSLMuCdVfeyRu8rMoas=', 'Content-Length': '0'} 2023-11-02T18:52:40.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005065928f375ce365-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T18:52:40.975 DEBUG:boto:path=/ibtljn-61/obj31 2023-11-02T18:52:40.976 DEBUG:boto:auth_path=/ibtljn-61/obj31 2023-11-02T18:52:40.976 DEBUG:boto:Method: GET 2023-11-02T18:52:40.977 DEBUG:boto:Path: /ibtljn-61/obj31 2023-11-02T18:52:40.977 DEBUG:boto:Data: 2023-11-02T18:52:40.978 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.978 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:40.978 DEBUG:boto:Port: 8001 2023-11-02T18:52:40.979 DEBUG:boto:Params: {} 2023-11-02T18:52:40.979 DEBUG:boto:Token: None 2023-11-02T18:52:40.980 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj31 2023-11-02T18:52:40.980 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pnmWy5A327Spyg8Ot5dhY/olezY= 2023-11-02T18:52:40.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pnmWy5A327Spyg8Ot5dhY/olezY=', 'Content-Length': '0'} 2023-11-02T18:52:40.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000120979468fcc7607-006543eff8-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:40.984 DEBUG:boto:path=/ibtljn-61/obj31 2023-11-02T18:52:40.985 DEBUG:boto:auth_path=/ibtljn-61/obj31 2023-11-02T18:52:40.985 DEBUG:boto:Method: GET 2023-11-02T18:52:40.986 DEBUG:boto:Path: /ibtljn-61/obj31 2023-11-02T18:52:40.986 DEBUG:boto:Data: 2023-11-02T18:52:40.987 DEBUG:boto:Headers: {} 2023-11-02T18:52:40.987 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:40.988 DEBUG:boto:Port: 8000 2023-11-02T18:52:40.988 DEBUG:boto:Params: {} 2023-11-02T18:52:40.988 DEBUG:boto:Token: None 2023-11-02T18:52:40.989 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:40 GMT /ibtljn-61/obj31 2023-11-02T18:52:40.989 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pnmWy5A327Spyg8Ot5dhY/olezY= 2023-11-02T18:52:40.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:40 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pnmWy5A327Spyg8Ot5dhY/olezY=', 'Content-Length': '0'} 2023-11-02T18:52:40.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ec493173c53cd6-006543eff8-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.034 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.035 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.036 DEBUG:boto:path=/ibtljn-61/?prefix=obj31 2023-11-02T18:52:41.036 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj31 2023-11-02T18:52:41.037 DEBUG:boto:Method: GET 2023-11-02T18:52:41.038 DEBUG:boto:Path: /ibtljn-61/?prefix=obj31 2023-11-02T18:52:41.038 DEBUG:boto:Data: 2023-11-02T18:52:41.039 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.040 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.041 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.041 DEBUG:boto:Params: {} 2023-11-02T18:52:41.042 DEBUG:boto:Token: None 2023-11-02T18:52:41.042 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.043 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083a39849c439f450-006543eff9-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.050 DEBUG:boto:b'ibtljn-61obj311000falseobj312023-11-02T18:50:24.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.051 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.051 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.052 DEBUG:boto:path=/ibtljn-61/?prefix=obj31 2023-11-02T18:52:41.053 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj31 2023-11-02T18:52:41.054 DEBUG:boto:Method: GET 2023-11-02T18:52:41.054 DEBUG:boto:Path: /ibtljn-61/?prefix=obj31 2023-11-02T18:52:41.057 DEBUG:boto:Data: 2023-11-02T18:52:41.058 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.058 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.059 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.060 DEBUG:boto:Params: {} 2023-11-02T18:52:41.060 DEBUG:boto:Token: None 2023-11-02T18:52:41.061 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.061 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3d946f50c219dab-006543eff9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.072 DEBUG:boto:b'ibtljn-61obj311000falseobj312023-11-02T18:50:24.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:52:41.073 DEBUG:boto:path=/ibtljn-61/obj33 2023-11-02T18:52:41.073 DEBUG:boto:auth_path=/ibtljn-61/obj33 2023-11-02T18:52:41.074 DEBUG:boto:path=/ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.074 DEBUG:boto:auth_path=/ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.075 DEBUG:boto:Method: GET 2023-11-02T18:52:41.075 DEBUG:boto:Path: /ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.076 DEBUG:boto:Data: 2023-11-02T18:52:41.076 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.077 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.077 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.077 DEBUG:boto:Params: {} 2023-11-02T18:52:41.078 DEBUG:boto:Token: None 2023-11-02T18:52:41.079 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.079 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O2/eUrAOYLloy728CBsOWQOVhqk= 2023-11-02T18:52:41.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O2/eUrAOYLloy728CBsOWQOVhqk=', 'Content-Length': '0'} 2023-11-02T18:52:41.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aeb70a4c2b35bf5-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.083 DEBUG:boto:path=/ibtljn-61/obj33 2023-11-02T18:52:41.083 DEBUG:boto:auth_path=/ibtljn-61/obj33 2023-11-02T18:52:41.084 DEBUG:boto:path=/ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.084 DEBUG:boto:auth_path=/ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.084 DEBUG:boto:Method: GET 2023-11-02T18:52:41.085 DEBUG:boto:Path: /ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.086 DEBUG:boto:Data: 2023-11-02T18:52:41.086 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.087 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.087 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.088 DEBUG:boto:Params: {} 2023-11-02T18:52:41.088 DEBUG:boto:Token: None 2023-11-02T18:52:41.089 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.090 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O2/eUrAOYLloy728CBsOWQOVhqk= 2023-11-02T18:52:41.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O2/eUrAOYLloy728CBsOWQOVhqk=', 'Content-Length': '0'} 2023-11-02T18:52:41.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b8726c9ddfe906-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.094 DEBUG:boto:path=/ibtljn-61/obj33 2023-11-02T18:52:41.094 DEBUG:boto:auth_path=/ibtljn-61/obj33 2023-11-02T18:52:41.094 DEBUG:boto:path=/ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.095 DEBUG:boto:auth_path=/ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.095 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.096 DEBUG:boto:Path: /ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.096 DEBUG:boto:Data: 2023-11-02T18:52:41.096 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.097 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.097 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.098 DEBUG:boto:Params: {} 2023-11-02T18:52:41.098 DEBUG:boto:Token: None 2023-11-02T18:52:41.099 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.099 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+pstp6J8HP1dfkI7N2GObjnkc0= 2023-11-02T18:52:41.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+pstp6J8HP1dfkI7N2GObjnkc0=', 'Content-Length': '0'} 2023-11-02T18:52:41.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aa3d92c8b80f686-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.103 DEBUG:boto:path=/ibtljn-61/obj33 2023-11-02T18:52:41.103 DEBUG:boto:auth_path=/ibtljn-61/obj33 2023-11-02T18:52:41.104 DEBUG:boto:path=/ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.104 DEBUG:boto:auth_path=/ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.104 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.105 DEBUG:boto:Path: /ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.105 DEBUG:boto:Data: 2023-11-02T18:52:41.106 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.106 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.107 DEBUG:boto:Params: {} 2023-11-02T18:52:41.107 DEBUG:boto:Token: None 2023-11-02T18:52:41.108 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.108 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+pstp6J8HP1dfkI7N2GObjnkc0= 2023-11-02T18:52:41.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+pstp6J8HP1dfkI7N2GObjnkc0=', 'Content-Length': '0'} 2023-11-02T18:52:41.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b26a0c0656d19193-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:52:41.113 DEBUG:boto:path=/ibtljn-61/obj33 2023-11-02T18:52:41.113 DEBUG:boto:auth_path=/ibtljn-61/obj33 2023-11-02T18:52:41.114 DEBUG:boto:path=/ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.114 DEBUG:boto:auth_path=/ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.115 DEBUG:boto:Method: GET 2023-11-02T18:52:41.115 DEBUG:boto:Path: /ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.116 DEBUG:boto:Data: 2023-11-02T18:52:41.116 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.117 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.117 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.117 DEBUG:boto:Params: {} 2023-11-02T18:52:41.118 DEBUG:boto:Token: None 2023-11-02T18:52:41.118 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.119 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O2/eUrAOYLloy728CBsOWQOVhqk= 2023-11-02T18:52:41.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O2/eUrAOYLloy728CBsOWQOVhqk=', 'Content-Length': '0'} 2023-11-02T18:52:41.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a555004149b60c90-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.171 DEBUG:boto:path=/ibtljn-61/obj33 2023-11-02T18:52:41.171 DEBUG:boto:auth_path=/ibtljn-61/obj33 2023-11-02T18:52:41.172 DEBUG:boto:path=/ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.172 DEBUG:boto:auth_path=/ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.173 DEBUG:boto:Method: GET 2023-11-02T18:52:41.173 DEBUG:boto:Path: /ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.174 DEBUG:boto:Data: 2023-11-02T18:52:41.174 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.174 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.175 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.175 DEBUG:boto:Params: {} 2023-11-02T18:52:41.176 DEBUG:boto:Token: None 2023-11-02T18:52:41.176 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj33?versionId=null 2023-11-02T18:52:41.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O2/eUrAOYLloy728CBsOWQOVhqk= 2023-11-02T18:52:41.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O2/eUrAOYLloy728CBsOWQOVhqk=', 'Content-Length': '0'} 2023-11-02T18:52:41.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d51387bb161d581a-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.181 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.182 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.182 DEBUG:boto:path=/ibtljn-61/?prefix=obj33 2023-11-02T18:52:41.183 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj33 2023-11-02T18:52:41.183 DEBUG:boto:Method: GET 2023-11-02T18:52:41.184 DEBUG:boto:Path: /ibtljn-61/?prefix=obj33 2023-11-02T18:52:41.184 DEBUG:boto:Data: 2023-11-02T18:52:41.184 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.185 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.185 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.186 DEBUG:boto:Params: {} 2023-11-02T18:52:41.186 DEBUG:boto:Token: None 2023-11-02T18:52:41.187 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.187 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a7e0f25aa333cff-006543eff9-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.194 DEBUG:boto:b'ibtljn-61obj331000falseobj332023-11-02T18:50:22.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.195 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.195 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.196 DEBUG:boto:path=/ibtljn-61/?prefix=obj33 2023-11-02T18:52:41.196 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj33 2023-11-02T18:52:41.196 DEBUG:boto:Method: GET 2023-11-02T18:52:41.197 DEBUG:boto:Path: /ibtljn-61/?prefix=obj33 2023-11-02T18:52:41.197 DEBUG:boto:Data: 2023-11-02T18:52:41.198 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.198 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.199 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.199 DEBUG:boto:Params: {} 2023-11-02T18:52:41.200 DEBUG:boto:Token: None 2023-11-02T18:52:41.200 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a744aea57f3be51a-006543eff9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.212 DEBUG:boto:b'ibtljn-61obj331000falseobj332023-11-02T18:50:22.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.213 DEBUG:boto:path=/ibtljn-61/obj33 2023-11-02T18:52:41.213 DEBUG:boto:auth_path=/ibtljn-61/obj33 2023-11-02T18:52:41.213 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.214 DEBUG:boto:Path: /ibtljn-61/obj33 2023-11-02T18:52:41.214 DEBUG:boto:Data: 2023-11-02T18:52:41.215 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.215 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.216 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.216 DEBUG:boto:Params: {} 2023-11-02T18:52:41.217 DEBUG:boto:Token: None 2023-11-02T18:52:41.217 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj33 2023-11-02T18:52:41.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4eatDTw7RczGgQ+9qelRrogeTwk= 2023-11-02T18:52:41.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4eatDTw7RczGgQ+9qelRrogeTwk=', 'Content-Length': '0'} 2023-11-02T18:52:41.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae6905f0287182c-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.221 DEBUG:boto:path=/ibtljn-61/obj33 2023-11-02T18:52:41.221 DEBUG:boto:auth_path=/ibtljn-61/obj33 2023-11-02T18:52:41.222 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.222 DEBUG:boto:Path: /ibtljn-61/obj33 2023-11-02T18:52:41.223 DEBUG:boto:Data: 2023-11-02T18:52:41.223 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.224 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.224 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.224 DEBUG:boto:Params: {} 2023-11-02T18:52:41.225 DEBUG:boto:Token: None 2023-11-02T18:52:41.225 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj33 2023-11-02T18:52:41.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4eatDTw7RczGgQ+9qelRrogeTwk= 2023-11-02T18:52:41.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4eatDTw7RczGgQ+9qelRrogeTwk=', 'Content-Length': '0'} 2023-11-02T18:52:41.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6a2f2bd8d53db64-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:52:41.230 DEBUG:boto:path=/ibtljn-61/obj33 2023-11-02T18:52:41.231 DEBUG:boto:auth_path=/ibtljn-61/obj33 2023-11-02T18:52:41.231 DEBUG:boto:Method: GET 2023-11-02T18:52:41.231 DEBUG:boto:Path: /ibtljn-61/obj33 2023-11-02T18:52:41.232 DEBUG:boto:Data: 2023-11-02T18:52:41.232 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.233 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.233 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.234 DEBUG:boto:Params: {} 2023-11-02T18:52:41.234 DEBUG:boto:Token: None 2023-11-02T18:52:41.235 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj33 2023-11-02T18:52:41.235 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wiy5AFVwP7ExRZdI6YVIp93lka0= 2023-11-02T18:52:41.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wiy5AFVwP7ExRZdI6YVIp93lka0=', 'Content-Length': '0'} 2023-11-02T18:52:41.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad85d32123bba9b7-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.239 DEBUG:boto:path=/ibtljn-61/obj33 2023-11-02T18:52:41.239 DEBUG:boto:auth_path=/ibtljn-61/obj33 2023-11-02T18:52:41.240 DEBUG:boto:Method: GET 2023-11-02T18:52:41.240 DEBUG:boto:Path: /ibtljn-61/obj33 2023-11-02T18:52:41.241 DEBUG:boto:Data: 2023-11-02T18:52:41.241 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.242 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.242 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.243 DEBUG:boto:Params: {} 2023-11-02T18:52:41.243 DEBUG:boto:Token: None 2023-11-02T18:52:41.244 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj33 2023-11-02T18:52:41.244 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wiy5AFVwP7ExRZdI6YVIp93lka0= 2023-11-02T18:52:41.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wiy5AFVwP7ExRZdI6YVIp93lka0=', 'Content-Length': '0'} 2023-11-02T18:52:41.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000501c199f093d1dba-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.290 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.291 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.292 DEBUG:boto:path=/ibtljn-61/?prefix=obj33 2023-11-02T18:52:41.293 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj33 2023-11-02T18:52:41.294 DEBUG:boto:Method: GET 2023-11-02T18:52:41.294 DEBUG:boto:Path: /ibtljn-61/?prefix=obj33 2023-11-02T18:52:41.295 DEBUG:boto:Data: 2023-11-02T18:52:41.295 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.295 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.296 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.296 DEBUG:boto:Params: {} 2023-11-02T18:52:41.297 DEBUG:boto:Token: None 2023-11-02T18:52:41.298 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.298 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005a922cfafb485ef-006543eff9-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.305 DEBUG:boto:b'ibtljn-61obj331000falseobj332023-11-02T18:50:22.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.306 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.306 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.307 DEBUG:boto:path=/ibtljn-61/?prefix=obj33 2023-11-02T18:52:41.307 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj33 2023-11-02T18:52:41.308 DEBUG:boto:Method: GET 2023-11-02T18:52:41.308 DEBUG:boto:Path: /ibtljn-61/?prefix=obj33 2023-11-02T18:52:41.308 DEBUG:boto:Data: 2023-11-02T18:52:41.309 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.309 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.310 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.310 DEBUG:boto:Params: {} 2023-11-02T18:52:41.311 DEBUG:boto:Token: None 2023-11-02T18:52:41.311 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.312 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceca16ae3008fce5-006543eff9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.323 DEBUG:boto:b'ibtljn-61obj331000falseobj332023-11-02T18:50:22.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:52:41.324 DEBUG:boto:path=/ibtljn-61/obj34 2023-11-02T18:52:41.324 DEBUG:boto:auth_path=/ibtljn-61/obj34 2023-11-02T18:52:41.325 DEBUG:boto:path=/ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.325 DEBUG:boto:auth_path=/ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.326 DEBUG:boto:Method: GET 2023-11-02T18:52:41.326 DEBUG:boto:Path: /ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.327 DEBUG:boto:Data: 2023-11-02T18:52:41.327 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.328 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.328 DEBUG:boto:Params: {} 2023-11-02T18:52:41.329 DEBUG:boto:Token: None 2023-11-02T18:52:41.329 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.330 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sQ0kDCvCN1wCzoz19TPoEwny3fo= 2023-11-02T18:52:41.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sQ0kDCvCN1wCzoz19TPoEwny3fo=', 'Content-Length': '0'} 2023-11-02T18:52:41.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7f65ee3bfc9ea63-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.374 DEBUG:boto:path=/ibtljn-61/obj34 2023-11-02T18:52:41.375 DEBUG:boto:auth_path=/ibtljn-61/obj34 2023-11-02T18:52:41.376 DEBUG:boto:path=/ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.376 DEBUG:boto:auth_path=/ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.376 DEBUG:boto:Method: GET 2023-11-02T18:52:41.377 DEBUG:boto:Path: /ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.377 DEBUG:boto:Data: 2023-11-02T18:52:41.378 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.378 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.379 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.379 DEBUG:boto:Params: {} 2023-11-02T18:52:41.380 DEBUG:boto:Token: None 2023-11-02T18:52:41.380 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.381 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sQ0kDCvCN1wCzoz19TPoEwny3fo= 2023-11-02T18:52:41.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sQ0kDCvCN1wCzoz19TPoEwny3fo=', 'Content-Length': '0'} 2023-11-02T18:52:41.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a68456fb5ac64d8b-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.385 DEBUG:boto:path=/ibtljn-61/obj34 2023-11-02T18:52:41.385 DEBUG:boto:auth_path=/ibtljn-61/obj34 2023-11-02T18:52:41.386 DEBUG:boto:path=/ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.386 DEBUG:boto:auth_path=/ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.387 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.387 DEBUG:boto:Path: /ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.388 DEBUG:boto:Data: 2023-11-02T18:52:41.388 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.388 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.389 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.389 DEBUG:boto:Params: {} 2023-11-02T18:52:41.390 DEBUG:boto:Token: None 2023-11-02T18:52:41.390 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.391 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L1w38wwhhF1NXBSpNa8XIgYxDMk= 2023-11-02T18:52:41.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L1w38wwhhF1NXBSpNa8XIgYxDMk=', 'Content-Length': '0'} 2023-11-02T18:52:41.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e135cc93a156640-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.395 DEBUG:boto:path=/ibtljn-61/obj34 2023-11-02T18:52:41.395 DEBUG:boto:auth_path=/ibtljn-61/obj34 2023-11-02T18:52:41.396 DEBUG:boto:path=/ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.396 DEBUG:boto:auth_path=/ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.396 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.397 DEBUG:boto:Path: /ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.397 DEBUG:boto:Data: 2023-11-02T18:52:41.398 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.399 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.399 DEBUG:boto:Params: {} 2023-11-02T18:52:41.400 DEBUG:boto:Token: None 2023-11-02T18:52:41.400 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L1w38wwhhF1NXBSpNa8XIgYxDMk= 2023-11-02T18:52:41.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L1w38wwhhF1NXBSpNa8XIgYxDMk=', 'Content-Length': '0'} 2023-11-02T18:52:41.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d294aa92f245effb-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:52:41.405 DEBUG:boto:path=/ibtljn-61/obj34 2023-11-02T18:52:41.405 DEBUG:boto:auth_path=/ibtljn-61/obj34 2023-11-02T18:52:41.406 DEBUG:boto:path=/ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.406 DEBUG:boto:auth_path=/ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.406 DEBUG:boto:Method: GET 2023-11-02T18:52:41.407 DEBUG:boto:Path: /ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.407 DEBUG:boto:Data: 2023-11-02T18:52:41.408 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.408 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.409 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.409 DEBUG:boto:Params: {} 2023-11-02T18:52:41.410 DEBUG:boto:Token: None 2023-11-02T18:52:41.410 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.411 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sQ0kDCvCN1wCzoz19TPoEwny3fo= 2023-11-02T18:52:41.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sQ0kDCvCN1wCzoz19TPoEwny3fo=', 'Content-Length': '0'} 2023-11-02T18:52:41.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000773f3a85d33b3f4b-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.415 DEBUG:boto:path=/ibtljn-61/obj34 2023-11-02T18:52:41.415 DEBUG:boto:auth_path=/ibtljn-61/obj34 2023-11-02T18:52:41.416 DEBUG:boto:path=/ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.416 DEBUG:boto:auth_path=/ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.417 DEBUG:boto:Method: GET 2023-11-02T18:52:41.417 DEBUG:boto:Path: /ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.417 DEBUG:boto:Data: 2023-11-02T18:52:41.418 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.418 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.419 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.419 DEBUG:boto:Params: {} 2023-11-02T18:52:41.420 DEBUG:boto:Token: None 2023-11-02T18:52:41.420 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj34?versionId=null 2023-11-02T18:52:41.421 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sQ0kDCvCN1wCzoz19TPoEwny3fo= 2023-11-02T18:52:41.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sQ0kDCvCN1wCzoz19TPoEwny3fo=', 'Content-Length': '0'} 2023-11-02T18:52:41.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee1aca110928e248-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.466 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.467 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.468 DEBUG:boto:path=/ibtljn-61/?prefix=obj34 2023-11-02T18:52:41.468 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj34 2023-11-02T18:52:41.468 DEBUG:boto:Method: GET 2023-11-02T18:52:41.469 DEBUG:boto:Path: /ibtljn-61/?prefix=obj34 2023-11-02T18:52:41.469 DEBUG:boto:Data: 2023-11-02T18:52:41.470 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.470 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.473 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.473 DEBUG:boto:Params: {} 2023-11-02T18:52:41.474 DEBUG:boto:Token: None 2023-11-02T18:52:41.475 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.475 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005503c0e47ecc0d59-006543eff9-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.482 DEBUG:boto:b'ibtljn-61obj341000falseobj342023-11-02T18:50:22.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.483 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.483 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.484 DEBUG:boto:path=/ibtljn-61/?prefix=obj34 2023-11-02T18:52:41.484 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj34 2023-11-02T18:52:41.485 DEBUG:boto:Method: GET 2023-11-02T18:52:41.485 DEBUG:boto:Path: /ibtljn-61/?prefix=obj34 2023-11-02T18:52:41.486 DEBUG:boto:Data: 2023-11-02T18:52:41.486 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.487 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.487 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.488 DEBUG:boto:Params: {} 2023-11-02T18:52:41.488 DEBUG:boto:Token: None 2023-11-02T18:52:41.489 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.489 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009826abadc50a11e8-006543eff9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.499 DEBUG:boto:b'ibtljn-61obj341000falseobj342023-11-02T18:50:22.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.500 DEBUG:boto:path=/ibtljn-61/obj34 2023-11-02T18:52:41.500 DEBUG:boto:auth_path=/ibtljn-61/obj34 2023-11-02T18:52:41.501 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.501 DEBUG:boto:Path: /ibtljn-61/obj34 2023-11-02T18:52:41.502 DEBUG:boto:Data: 2023-11-02T18:52:41.502 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.502 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.503 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.503 DEBUG:boto:Params: {} 2023-11-02T18:52:41.504 DEBUG:boto:Token: None 2023-11-02T18:52:41.504 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj34 2023-11-02T18:52:41.505 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YuB63xYseC44Bs8owVQ/Aq/bvCg= 2023-11-02T18:52:41.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YuB63xYseC44Bs8owVQ/Aq/bvCg=', 'Content-Length': '0'} 2023-11-02T18:52:41.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fbeaeb120857126-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.508 DEBUG:boto:path=/ibtljn-61/obj34 2023-11-02T18:52:41.509 DEBUG:boto:auth_path=/ibtljn-61/obj34 2023-11-02T18:52:41.509 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.510 DEBUG:boto:Path: /ibtljn-61/obj34 2023-11-02T18:52:41.510 DEBUG:boto:Data: 2023-11-02T18:52:41.511 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.511 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.511 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.512 DEBUG:boto:Params: {} 2023-11-02T18:52:41.512 DEBUG:boto:Token: None 2023-11-02T18:52:41.513 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj34 2023-11-02T18:52:41.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YuB63xYseC44Bs8owVQ/Aq/bvCg= 2023-11-02T18:52:41.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YuB63xYseC44Bs8owVQ/Aq/bvCg=', 'Content-Length': '0'} 2023-11-02T18:52:41.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b986b4200f1f3cbd-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T18:52:41.518 DEBUG:boto:path=/ibtljn-61/obj34 2023-11-02T18:52:41.518 DEBUG:boto:auth_path=/ibtljn-61/obj34 2023-11-02T18:52:41.519 DEBUG:boto:Method: GET 2023-11-02T18:52:41.519 DEBUG:boto:Path: /ibtljn-61/obj34 2023-11-02T18:52:41.520 DEBUG:boto:Data: 2023-11-02T18:52:41.520 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.520 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.521 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.521 DEBUG:boto:Params: {} 2023-11-02T18:52:41.522 DEBUG:boto:Token: None 2023-11-02T18:52:41.522 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj34 2023-11-02T18:52:41.523 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y65nkbZnby2BnKmwwyzrH4ZQCg0= 2023-11-02T18:52:41.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y65nkbZnby2BnKmwwyzrH4ZQCg0=', 'Content-Length': '0'} 2023-11-02T18:52:41.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff5f1b5a7c810fbd-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.566 DEBUG:boto:path=/ibtljn-61/obj34 2023-11-02T18:52:41.567 DEBUG:boto:auth_path=/ibtljn-61/obj34 2023-11-02T18:52:41.568 DEBUG:boto:Method: GET 2023-11-02T18:52:41.568 DEBUG:boto:Path: /ibtljn-61/obj34 2023-11-02T18:52:41.568 DEBUG:boto:Data: 2023-11-02T18:52:41.569 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.569 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.570 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.570 DEBUG:boto:Params: {} 2023-11-02T18:52:41.571 DEBUG:boto:Token: None 2023-11-02T18:52:41.571 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj34 2023-11-02T18:52:41.572 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y65nkbZnby2BnKmwwyzrH4ZQCg0= 2023-11-02T18:52:41.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y65nkbZnby2BnKmwwyzrH4ZQCg0=', 'Content-Length': '0'} 2023-11-02T18:52:41.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e32af5ad26264628-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.576 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.577 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.577 DEBUG:boto:path=/ibtljn-61/?prefix=obj34 2023-11-02T18:52:41.578 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj34 2023-11-02T18:52:41.578 DEBUG:boto:Method: GET 2023-11-02T18:52:41.578 DEBUG:boto:Path: /ibtljn-61/?prefix=obj34 2023-11-02T18:52:41.579 DEBUG:boto:Data: 2023-11-02T18:52:41.579 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.580 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.580 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.581 DEBUG:boto:Params: {} 2023-11-02T18:52:41.581 DEBUG:boto:Token: None 2023-11-02T18:52:41.582 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.582 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007465b23daf9cff8f-006543eff9-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.589 DEBUG:boto:b'ibtljn-61obj341000falseobj342023-11-02T18:50:22.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.590 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.591 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.591 DEBUG:boto:path=/ibtljn-61/?prefix=obj34 2023-11-02T18:52:41.591 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj34 2023-11-02T18:52:41.592 DEBUG:boto:Method: GET 2023-11-02T18:52:41.592 DEBUG:boto:Path: /ibtljn-61/?prefix=obj34 2023-11-02T18:52:41.593 DEBUG:boto:Data: 2023-11-02T18:52:41.593 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.594 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.594 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.595 DEBUG:boto:Params: {} 2023-11-02T18:52:41.595 DEBUG:boto:Token: None 2023-11-02T18:52:41.596 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.596 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008abc27c8e2c1b4f3-006543eff9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.650 DEBUG:boto:b'ibtljn-61obj341000falseobj342023-11-02T18:50:22.495Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-11-02T18:52:41.652 DEBUG:boto:path=/ibtljn-61/obj37 2023-11-02T18:52:41.652 DEBUG:boto:auth_path=/ibtljn-61/obj37 2023-11-02T18:52:41.652 DEBUG:boto:path=/ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.653 DEBUG:boto:auth_path=/ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.653 DEBUG:boto:Method: GET 2023-11-02T18:52:41.654 DEBUG:boto:Path: /ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.654 DEBUG:boto:Data: 2023-11-02T18:52:41.655 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.655 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.656 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.656 DEBUG:boto:Params: {} 2023-11-02T18:52:41.656 DEBUG:boto:Token: None 2023-11-02T18:52:41.657 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.658 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o9JE3rE5T8QOSx7GQjmWfQ/KKdM= 2023-11-02T18:52:41.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o9JE3rE5T8QOSx7GQjmWfQ/KKdM=', 'Content-Length': '0'} 2023-11-02T18:52:41.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2fe7dd697f87c01-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.661 DEBUG:boto:path=/ibtljn-61/obj37 2023-11-02T18:52:41.662 DEBUG:boto:auth_path=/ibtljn-61/obj37 2023-11-02T18:52:41.662 DEBUG:boto:path=/ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.663 DEBUG:boto:auth_path=/ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.663 DEBUG:boto:Method: GET 2023-11-02T18:52:41.664 DEBUG:boto:Path: /ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.664 DEBUG:boto:Data: 2023-11-02T18:52:41.665 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.665 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.665 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.666 DEBUG:boto:Params: {} 2023-11-02T18:52:41.666 DEBUG:boto:Token: None 2023-11-02T18:52:41.667 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.667 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o9JE3rE5T8QOSx7GQjmWfQ/KKdM= 2023-11-02T18:52:41.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o9JE3rE5T8QOSx7GQjmWfQ/KKdM=', 'Content-Length': '0'} 2023-11-02T18:52:41.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004f760a605f728ba-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.673 DEBUG:boto:path=/ibtljn-61/obj37 2023-11-02T18:52:41.674 DEBUG:boto:auth_path=/ibtljn-61/obj37 2023-11-02T18:52:41.674 DEBUG:boto:path=/ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.674 DEBUG:boto:auth_path=/ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.675 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.676 DEBUG:boto:Path: /ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.676 DEBUG:boto:Data: 2023-11-02T18:52:41.676 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.677 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.677 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.678 DEBUG:boto:Params: {} 2023-11-02T18:52:41.678 DEBUG:boto:Token: None 2023-11-02T18:52:41.679 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.679 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KkMJvB9/Fh/SZo/51iH5zBqOwiY= 2023-11-02T18:52:41.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KkMJvB9/Fh/SZo/51iH5zBqOwiY=', 'Content-Length': '0'} 2023-11-02T18:52:41.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000814b6f75f9d9a8a0-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.683 DEBUG:boto:path=/ibtljn-61/obj37 2023-11-02T18:52:41.684 DEBUG:boto:auth_path=/ibtljn-61/obj37 2023-11-02T18:52:41.684 DEBUG:boto:path=/ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.685 DEBUG:boto:auth_path=/ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.685 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.685 DEBUG:boto:Path: /ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.686 DEBUG:boto:Data: 2023-11-02T18:52:41.686 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.687 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.687 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.688 DEBUG:boto:Params: {} 2023-11-02T18:52:41.688 DEBUG:boto:Token: None 2023-11-02T18:52:41.689 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.689 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KkMJvB9/Fh/SZo/51iH5zBqOwiY= 2023-11-02T18:52:41.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KkMJvB9/Fh/SZo/51iH5zBqOwiY=', 'Content-Length': '0'} 2023-11-02T18:52:41.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fee89242b236f95-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-11-02T18:52:41.693 DEBUG:boto:path=/ibtljn-61/obj37 2023-11-02T18:52:41.693 DEBUG:boto:auth_path=/ibtljn-61/obj37 2023-11-02T18:52:41.694 DEBUG:boto:path=/ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.694 DEBUG:boto:auth_path=/ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.695 DEBUG:boto:Method: GET 2023-11-02T18:52:41.695 DEBUG:boto:Path: /ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.696 DEBUG:boto:Data: 2023-11-02T18:52:41.696 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.696 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.697 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.697 DEBUG:boto:Params: {} 2023-11-02T18:52:41.698 DEBUG:boto:Token: None 2023-11-02T18:52:41.698 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.699 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o9JE3rE5T8QOSx7GQjmWfQ/KKdM= 2023-11-02T18:52:41.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o9JE3rE5T8QOSx7GQjmWfQ/KKdM=', 'Content-Length': '0'} 2023-11-02T18:52:41.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a1976c99326d5c2-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.703 DEBUG:boto:path=/ibtljn-61/obj37 2023-11-02T18:52:41.703 DEBUG:boto:auth_path=/ibtljn-61/obj37 2023-11-02T18:52:41.703 DEBUG:boto:path=/ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.704 DEBUG:boto:auth_path=/ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.704 DEBUG:boto:Method: GET 2023-11-02T18:52:41.705 DEBUG:boto:Path: /ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.705 DEBUG:boto:Data: 2023-11-02T18:52:41.706 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.706 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.707 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.707 DEBUG:boto:Params: {} 2023-11-02T18:52:41.708 DEBUG:boto:Token: None 2023-11-02T18:52:41.708 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj37?versionId=null 2023-11-02T18:52:41.709 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o9JE3rE5T8QOSx7GQjmWfQ/KKdM= 2023-11-02T18:52:41.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o9JE3rE5T8QOSx7GQjmWfQ/KKdM=', 'Content-Length': '0'} 2023-11-02T18:52:41.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f33a745a3873bd49-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.713 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.713 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.714 DEBUG:boto:path=/ibtljn-61/?prefix=obj37 2023-11-02T18:52:41.714 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj37 2023-11-02T18:52:41.715 DEBUG:boto:Method: GET 2023-11-02T18:52:41.715 DEBUG:boto:Path: /ibtljn-61/?prefix=obj37 2023-11-02T18:52:41.715 DEBUG:boto:Data: 2023-11-02T18:52:41.716 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.716 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.717 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.717 DEBUG:boto:Params: {} 2023-11-02T18:52:41.718 DEBUG:boto:Token: None 2023-11-02T18:52:41.718 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.719 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3421f1ba70ad582-006543eff9-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.725 DEBUG:boto:b'ibtljn-61obj371000falseobj372023-11-02T18:50:24.971Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.726 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.726 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.727 DEBUG:boto:path=/ibtljn-61/?prefix=obj37 2023-11-02T18:52:41.727 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj37 2023-11-02T18:52:41.728 DEBUG:boto:Method: GET 2023-11-02T18:52:41.728 DEBUG:boto:Path: /ibtljn-61/?prefix=obj37 2023-11-02T18:52:41.729 DEBUG:boto:Data: 2023-11-02T18:52:41.729 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.730 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.730 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.730 DEBUG:boto:Params: {} 2023-11-02T18:52:41.731 DEBUG:boto:Token: None 2023-11-02T18:52:41.731 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.732 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ca34d4397bee327-006543eff9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.786 DEBUG:boto:b'ibtljn-61obj371000falseobj372023-11-02T18:50:24.971Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.787 DEBUG:boto:path=/ibtljn-61/obj37 2023-11-02T18:52:41.788 DEBUG:boto:auth_path=/ibtljn-61/obj37 2023-11-02T18:52:41.788 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.789 DEBUG:boto:Path: /ibtljn-61/obj37 2023-11-02T18:52:41.789 DEBUG:boto:Data: 2023-11-02T18:52:41.789 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.790 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.791 DEBUG:boto:Params: {} 2023-11-02T18:52:41.791 DEBUG:boto:Token: None 2023-11-02T18:52:41.792 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj37 2023-11-02T18:52:41.792 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2chd9LAy4VLkba7qtcj3LVMBdHY= 2023-11-02T18:52:41.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2chd9LAy4VLkba7qtcj3LVMBdHY=', 'Content-Length': '0'} 2023-11-02T18:52:41.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc3383b0c34a7266-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.796 DEBUG:boto:path=/ibtljn-61/obj37 2023-11-02T18:52:41.797 DEBUG:boto:auth_path=/ibtljn-61/obj37 2023-11-02T18:52:41.797 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.798 DEBUG:boto:Path: /ibtljn-61/obj37 2023-11-02T18:52:41.798 DEBUG:boto:Data: 2023-11-02T18:52:41.798 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.799 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.800 DEBUG:boto:Params: {} 2023-11-02T18:52:41.800 DEBUG:boto:Token: None 2023-11-02T18:52:41.801 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj37 2023-11-02T18:52:41.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2chd9LAy4VLkba7qtcj3LVMBdHY= 2023-11-02T18:52:41.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2chd9LAy4VLkba7qtcj3LVMBdHY=', 'Content-Length': '0'} 2023-11-02T18:52:41.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbb89fbadd1e3e82-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-11-02T18:52:41.806 DEBUG:boto:path=/ibtljn-61/obj37 2023-11-02T18:52:41.806 DEBUG:boto:auth_path=/ibtljn-61/obj37 2023-11-02T18:52:41.807 DEBUG:boto:Method: GET 2023-11-02T18:52:41.807 DEBUG:boto:Path: /ibtljn-61/obj37 2023-11-02T18:52:41.808 DEBUG:boto:Data: 2023-11-02T18:52:41.808 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.809 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.810 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.810 DEBUG:boto:Params: {} 2023-11-02T18:52:41.811 DEBUG:boto:Token: None 2023-11-02T18:52:41.811 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj37 2023-11-02T18:52:41.812 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fbE1HkaeGgnZ2SqjaI2WqJ0qmq0= 2023-11-02T18:52:41.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fbE1HkaeGgnZ2SqjaI2WqJ0qmq0=', 'Content-Length': '0'} 2023-11-02T18:52:41.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c76b67019e1d28-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.858 DEBUG:boto:path=/ibtljn-61/obj37 2023-11-02T18:52:41.859 DEBUG:boto:auth_path=/ibtljn-61/obj37 2023-11-02T18:52:41.859 DEBUG:boto:Method: GET 2023-11-02T18:52:41.860 DEBUG:boto:Path: /ibtljn-61/obj37 2023-11-02T18:52:41.860 DEBUG:boto:Data: 2023-11-02T18:52:41.861 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.861 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.862 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.862 DEBUG:boto:Params: {} 2023-11-02T18:52:41.863 DEBUG:boto:Token: None 2023-11-02T18:52:41.863 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj37 2023-11-02T18:52:41.864 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fbE1HkaeGgnZ2SqjaI2WqJ0qmq0= 2023-11-02T18:52:41.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fbE1HkaeGgnZ2SqjaI2WqJ0qmq0=', 'Content-Length': '0'} 2023-11-02T18:52:41.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f24eaa324bf486c-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.868 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.870 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.870 DEBUG:boto:path=/ibtljn-61/?prefix=obj37 2023-11-02T18:52:41.871 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj37 2023-11-02T18:52:41.871 DEBUG:boto:Method: GET 2023-11-02T18:52:41.872 DEBUG:boto:Path: /ibtljn-61/?prefix=obj37 2023-11-02T18:52:41.872 DEBUG:boto:Data: 2023-11-02T18:52:41.872 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.873 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.873 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.874 DEBUG:boto:Params: {} 2023-11-02T18:52:41.874 DEBUG:boto:Token: None 2023-11-02T18:52:41.875 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.875 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065bb24a3abd9f25a-006543eff9-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.882 DEBUG:boto:b'ibtljn-61obj371000falseobj372023-11-02T18:50:24.971Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.883 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:41.883 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:41.884 DEBUG:boto:path=/ibtljn-61/?prefix=obj37 2023-11-02T18:52:41.884 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj37 2023-11-02T18:52:41.885 DEBUG:boto:Method: GET 2023-11-02T18:52:41.885 DEBUG:boto:Path: /ibtljn-61/?prefix=obj37 2023-11-02T18:52:41.886 DEBUG:boto:Data: 2023-11-02T18:52:41.886 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.887 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.887 DEBUG:boto:Params: {} 2023-11-02T18:52:41.888 DEBUG:boto:Token: None 2023-11-02T18:52:41.888 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/ 2023-11-02T18:52:41.889 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw= 2023-11-02T18:52:41.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BwkVYWyOucthZYg17HkJjj8BuHw=', 'Content-Length': '0'} 2023-11-02T18:52:41.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000213510f7c10a0c2-006543eff9-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.900 DEBUG:boto:b'ibtljn-61obj371000falseobj372023-11-02T18:50:24.971Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:41.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:52:41.901 DEBUG:boto:path=/ibtljn-61/obj38 2023-11-02T18:52:41.902 DEBUG:boto:auth_path=/ibtljn-61/obj38 2023-11-02T18:52:41.902 DEBUG:boto:path=/ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.903 DEBUG:boto:auth_path=/ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.903 DEBUG:boto:Method: GET 2023-11-02T18:52:41.903 DEBUG:boto:Path: /ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.904 DEBUG:boto:Data: 2023-11-02T18:52:41.904 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.905 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.905 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.906 DEBUG:boto:Params: {} 2023-11-02T18:52:41.907 DEBUG:boto:Token: None 2023-11-02T18:52:41.908 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.908 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1Ovl0IUcQeYid5bUIyyvD6hgicg= 2023-11-02T18:52:41.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1Ovl0IUcQeYid5bUIyyvD6hgicg=', 'Content-Length': '0'} 2023-11-02T18:52:41.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b60008e1a090bd8e-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.912 DEBUG:boto:path=/ibtljn-61/obj38 2023-11-02T18:52:41.912 DEBUG:boto:auth_path=/ibtljn-61/obj38 2023-11-02T18:52:41.913 DEBUG:boto:path=/ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.913 DEBUG:boto:auth_path=/ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.914 DEBUG:boto:Method: GET 2023-11-02T18:52:41.914 DEBUG:boto:Path: /ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.914 DEBUG:boto:Data: 2023-11-02T18:52:41.915 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.915 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.916 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.916 DEBUG:boto:Params: {} 2023-11-02T18:52:41.917 DEBUG:boto:Token: None 2023-11-02T18:52:41.917 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.918 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1Ovl0IUcQeYid5bUIyyvD6hgicg= 2023-11-02T18:52:41.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1Ovl0IUcQeYid5bUIyyvD6hgicg=', 'Content-Length': '0'} 2023-11-02T18:52:41.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000776f2b40e946868a-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.966 DEBUG:boto:path=/ibtljn-61/obj38 2023-11-02T18:52:41.967 DEBUG:boto:auth_path=/ibtljn-61/obj38 2023-11-02T18:52:41.968 DEBUG:boto:path=/ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.968 DEBUG:boto:auth_path=/ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.968 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.969 DEBUG:boto:Path: /ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.969 DEBUG:boto:Data: 2023-11-02T18:52:41.970 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.970 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.971 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.971 DEBUG:boto:Params: {} 2023-11-02T18:52:41.972 DEBUG:boto:Token: None 2023-11-02T18:52:41.972 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.973 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Xd1Yz7hBnr3TuuYcwFxJuDlqxWU= 2023-11-02T18:52:41.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Xd1Yz7hBnr3TuuYcwFxJuDlqxWU=', 'Content-Length': '0'} 2023-11-02T18:52:41.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f61c9320438ff77a-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.977 DEBUG:boto:path=/ibtljn-61/obj38 2023-11-02T18:52:41.977 DEBUG:boto:auth_path=/ibtljn-61/obj38 2023-11-02T18:52:41.978 DEBUG:boto:path=/ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.978 DEBUG:boto:auth_path=/ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.979 DEBUG:boto:Method: HEAD 2023-11-02T18:52:41.979 DEBUG:boto:Path: /ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.979 DEBUG:boto:Data: 2023-11-02T18:52:41.980 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.980 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:41.981 DEBUG:boto:Port: 8000 2023-11-02T18:52:41.981 DEBUG:boto:Params: {} 2023-11-02T18:52:41.982 DEBUG:boto:Token: None 2023-11-02T18:52:41.982 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.983 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Xd1Yz7hBnr3TuuYcwFxJuDlqxWU= 2023-11-02T18:52:41.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Xd1Yz7hBnr3TuuYcwFxJuDlqxWU=', 'Content-Length': '0'} 2023-11-02T18:52:41.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002abe077d98c411ea-006543eff9-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:41.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:52:41.987 DEBUG:boto:path=/ibtljn-61/obj38 2023-11-02T18:52:41.987 DEBUG:boto:auth_path=/ibtljn-61/obj38 2023-11-02T18:52:41.988 DEBUG:boto:path=/ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.988 DEBUG:boto:auth_path=/ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.989 DEBUG:boto:Method: GET 2023-11-02T18:52:41.989 DEBUG:boto:Path: /ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.990 DEBUG:boto:Data: 2023-11-02T18:52:41.990 DEBUG:boto:Headers: {} 2023-11-02T18:52:41.991 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:41.991 DEBUG:boto:Port: 8001 2023-11-02T18:52:41.992 DEBUG:boto:Params: {} 2023-11-02T18:52:41.992 DEBUG:boto:Token: None 2023-11-02T18:52:41.993 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:41 GMT /ibtljn-61/obj38?versionId=null 2023-11-02T18:52:41.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1Ovl0IUcQeYid5bUIyyvD6hgicg= 2023-11-02T18:52:41.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1Ovl0IUcQeYid5bUIyyvD6hgicg=', 'Content-Length': '0'} 2023-11-02T18:52:41.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019fa5d1c2004dc09-006543eff9-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.042 DEBUG:boto:path=/ibtljn-61/obj38 2023-11-02T18:52:42.043 DEBUG:boto:auth_path=/ibtljn-61/obj38 2023-11-02T18:52:42.043 DEBUG:boto:path=/ibtljn-61/obj38?versionId=null 2023-11-02T18:52:42.044 DEBUG:boto:auth_path=/ibtljn-61/obj38?versionId=null 2023-11-02T18:52:42.044 DEBUG:boto:Method: GET 2023-11-02T18:52:42.045 DEBUG:boto:Path: /ibtljn-61/obj38?versionId=null 2023-11-02T18:52:42.045 DEBUG:boto:Data: 2023-11-02T18:52:42.046 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.047 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.047 DEBUG:boto:Params: {} 2023-11-02T18:52:42.047 DEBUG:boto:Token: None 2023-11-02T18:52:42.048 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj38?versionId=null 2023-11-02T18:52:42.049 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:re3oJ1LkU585LDbCtREbzvjQvZM= 2023-11-02T18:52:42.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:re3oJ1LkU585LDbCtREbzvjQvZM=', 'Content-Length': '0'} 2023-11-02T18:52:42.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000658010d75fa49380-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.053 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.053 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.054 DEBUG:boto:path=/ibtljn-61/?prefix=obj38 2023-11-02T18:52:42.055 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj38 2023-11-02T18:52:42.055 DEBUG:boto:Method: GET 2023-11-02T18:52:42.055 DEBUG:boto:Path: /ibtljn-61/?prefix=obj38 2023-11-02T18:52:42.056 DEBUG:boto:Data: 2023-11-02T18:52:42.056 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.057 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.057 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.057 DEBUG:boto:Params: {} 2023-11-02T18:52:42.058 DEBUG:boto:Token: None 2023-11-02T18:52:42.059 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.059 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aba255d3aef6ba8-006543effa-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.066 DEBUG:boto:b'ibtljn-61obj381000falseobj382023-11-02T18:50:22.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.067 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.067 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.068 DEBUG:boto:path=/ibtljn-61/?prefix=obj38 2023-11-02T18:52:42.068 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj38 2023-11-02T18:52:42.069 DEBUG:boto:Method: GET 2023-11-02T18:52:42.069 DEBUG:boto:Path: /ibtljn-61/?prefix=obj38 2023-11-02T18:52:42.070 DEBUG:boto:Data: 2023-11-02T18:52:42.070 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.071 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.071 DEBUG:boto:Params: {} 2023-11-02T18:52:42.072 DEBUG:boto:Token: None 2023-11-02T18:52:42.072 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.073 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000143113f77367c267-006543effa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.086 DEBUG:boto:b'ibtljn-61obj381000falseobj382023-11-02T18:50:22.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.087 DEBUG:boto:path=/ibtljn-61/obj38 2023-11-02T18:52:42.087 DEBUG:boto:auth_path=/ibtljn-61/obj38 2023-11-02T18:52:42.087 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.088 DEBUG:boto:Path: /ibtljn-61/obj38 2023-11-02T18:52:42.088 DEBUG:boto:Data: 2023-11-02T18:52:42.089 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.089 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.090 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.090 DEBUG:boto:Params: {} 2023-11-02T18:52:42.091 DEBUG:boto:Token: None 2023-11-02T18:52:42.091 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj38 2023-11-02T18:52:42.092 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zkxIw0t8mcm7oTauQrow1/GxtvU= 2023-11-02T18:52:42.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zkxIw0t8mcm7oTauQrow1/GxtvU=', 'Content-Length': '0'} 2023-11-02T18:52:42.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033207bc0e6b84087-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.095 DEBUG:boto:path=/ibtljn-61/obj38 2023-11-02T18:52:42.095 DEBUG:boto:auth_path=/ibtljn-61/obj38 2023-11-02T18:52:42.096 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.096 DEBUG:boto:Path: /ibtljn-61/obj38 2023-11-02T18:52:42.097 DEBUG:boto:Data: 2023-11-02T18:52:42.097 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.098 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.098 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.099 DEBUG:boto:Params: {} 2023-11-02T18:52:42.099 DEBUG:boto:Token: None 2023-11-02T18:52:42.100 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj38 2023-11-02T18:52:42.100 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zkxIw0t8mcm7oTauQrow1/GxtvU= 2023-11-02T18:52:42.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zkxIw0t8mcm7oTauQrow1/GxtvU=', 'Content-Length': '0'} 2023-11-02T18:52:42.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000555cd5842e19e3f5-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T18:52:42.104 DEBUG:boto:path=/ibtljn-61/obj38 2023-11-02T18:52:42.105 DEBUG:boto:auth_path=/ibtljn-61/obj38 2023-11-02T18:52:42.105 DEBUG:boto:Method: GET 2023-11-02T18:52:42.106 DEBUG:boto:Path: /ibtljn-61/obj38 2023-11-02T18:52:42.106 DEBUG:boto:Data: 2023-11-02T18:52:42.107 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.107 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.107 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.108 DEBUG:boto:Params: {} 2023-11-02T18:52:42.108 DEBUG:boto:Token: None 2023-11-02T18:52:42.109 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj38 2023-11-02T18:52:42.109 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HKlQTnGRRY3go09Ndbw8L3BVVYU= 2023-11-02T18:52:42.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HKlQTnGRRY3go09Ndbw8L3BVVYU=', 'Content-Length': '0'} 2023-11-02T18:52:42.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dea6340168b127b-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.114 DEBUG:boto:path=/ibtljn-61/obj38 2023-11-02T18:52:42.114 DEBUG:boto:auth_path=/ibtljn-61/obj38 2023-11-02T18:52:42.115 DEBUG:boto:Method: GET 2023-11-02T18:52:42.115 DEBUG:boto:Path: /ibtljn-61/obj38 2023-11-02T18:52:42.116 DEBUG:boto:Data: 2023-11-02T18:52:42.116 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.116 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.117 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.117 DEBUG:boto:Params: {} 2023-11-02T18:52:42.118 DEBUG:boto:Token: None 2023-11-02T18:52:42.118 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj38 2023-11-02T18:52:42.119 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HKlQTnGRRY3go09Ndbw8L3BVVYU= 2023-11-02T18:52:42.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HKlQTnGRRY3go09Ndbw8L3BVVYU=', 'Content-Length': '0'} 2023-11-02T18:52:42.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d7c6e697278a95c-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.166 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.167 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.167 DEBUG:boto:path=/ibtljn-61/?prefix=obj38 2023-11-02T18:52:42.168 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj38 2023-11-02T18:52:42.168 DEBUG:boto:Method: GET 2023-11-02T18:52:42.169 DEBUG:boto:Path: /ibtljn-61/?prefix=obj38 2023-11-02T18:52:42.169 DEBUG:boto:Data: 2023-11-02T18:52:42.170 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.170 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.171 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.171 DEBUG:boto:Params: {} 2023-11-02T18:52:42.172 DEBUG:boto:Token: None 2023-11-02T18:52:42.172 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.173 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8caf2084488214a-006543effa-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.179 DEBUG:boto:b'ibtljn-61obj381000falseobj382023-11-02T18:50:22.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.180 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.180 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.181 DEBUG:boto:path=/ibtljn-61/?prefix=obj38 2023-11-02T18:52:42.181 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj38 2023-11-02T18:52:42.182 DEBUG:boto:Method: GET 2023-11-02T18:52:42.182 DEBUG:boto:Path: /ibtljn-61/?prefix=obj38 2023-11-02T18:52:42.183 DEBUG:boto:Data: 2023-11-02T18:52:42.183 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.183 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.184 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.184 DEBUG:boto:Params: {} 2023-11-02T18:52:42.185 DEBUG:boto:Token: None 2023-11-02T18:52:42.186 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.186 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c6c095480025552-006543effa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.196 DEBUG:boto:b'ibtljn-61obj381000falseobj382023-11-02T18:50:22.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:52:42.197 DEBUG:boto:path=/ibtljn-61/obj44 2023-11-02T18:52:42.198 DEBUG:boto:auth_path=/ibtljn-61/obj44 2023-11-02T18:52:42.198 DEBUG:boto:path=/ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.199 DEBUG:boto:auth_path=/ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.199 DEBUG:boto:Method: GET 2023-11-02T18:52:42.200 DEBUG:boto:Path: /ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.200 DEBUG:boto:Data: 2023-11-02T18:52:42.201 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.201 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.202 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.202 DEBUG:boto:Params: {} 2023-11-02T18:52:42.203 DEBUG:boto:Token: None 2023-11-02T18:52:42.204 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.204 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ubt7sPu60/kZQjo5y6tyJ+Da19s= 2023-11-02T18:52:42.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ubt7sPu60/kZQjo5y6tyJ+Da19s=', 'Content-Length': '0'} 2023-11-02T18:52:42.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec1fe8629527584f-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.250 DEBUG:boto:path=/ibtljn-61/obj44 2023-11-02T18:52:42.251 DEBUG:boto:auth_path=/ibtljn-61/obj44 2023-11-02T18:52:42.252 DEBUG:boto:path=/ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.252 DEBUG:boto:auth_path=/ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.253 DEBUG:boto:Method: GET 2023-11-02T18:52:42.253 DEBUG:boto:Path: /ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.254 DEBUG:boto:Data: 2023-11-02T18:52:42.254 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.254 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.255 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.255 DEBUG:boto:Params: {} 2023-11-02T18:52:42.256 DEBUG:boto:Token: None 2023-11-02T18:52:42.257 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.257 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ubt7sPu60/kZQjo5y6tyJ+Da19s= 2023-11-02T18:52:42.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ubt7sPu60/kZQjo5y6tyJ+Da19s=', 'Content-Length': '0'} 2023-11-02T18:52:42.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000c13ef49b139299-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.262 DEBUG:boto:path=/ibtljn-61/obj44 2023-11-02T18:52:42.262 DEBUG:boto:auth_path=/ibtljn-61/obj44 2023-11-02T18:52:42.263 DEBUG:boto:path=/ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.263 DEBUG:boto:auth_path=/ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.263 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.264 DEBUG:boto:Path: /ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.264 DEBUG:boto:Data: 2023-11-02T18:52:42.265 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.265 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.266 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.266 DEBUG:boto:Params: {} 2023-11-02T18:52:42.267 DEBUG:boto:Token: None 2023-11-02T18:52:42.267 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.268 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KdpXNLoqQEKh8/RdTumMgD8jqgE= 2023-11-02T18:52:42.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KdpXNLoqQEKh8/RdTumMgD8jqgE=', 'Content-Length': '0'} 2023-11-02T18:52:42.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000978b73632a5dac40-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.272 DEBUG:boto:path=/ibtljn-61/obj44 2023-11-02T18:52:42.272 DEBUG:boto:auth_path=/ibtljn-61/obj44 2023-11-02T18:52:42.273 DEBUG:boto:path=/ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.273 DEBUG:boto:auth_path=/ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.273 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.274 DEBUG:boto:Path: /ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.274 DEBUG:boto:Data: 2023-11-02T18:52:42.275 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.275 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.276 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.276 DEBUG:boto:Params: {} 2023-11-02T18:52:42.276 DEBUG:boto:Token: None 2023-11-02T18:52:42.277 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.277 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KdpXNLoqQEKh8/RdTumMgD8jqgE= 2023-11-02T18:52:42.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KdpXNLoqQEKh8/RdTumMgD8jqgE=', 'Content-Length': '0'} 2023-11-02T18:52:42.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c43c17cef991289c-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:52:42.282 DEBUG:boto:path=/ibtljn-61/obj44 2023-11-02T18:52:42.283 DEBUG:boto:auth_path=/ibtljn-61/obj44 2023-11-02T18:52:42.283 DEBUG:boto:path=/ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.284 DEBUG:boto:auth_path=/ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.284 DEBUG:boto:Method: GET 2023-11-02T18:52:42.285 DEBUG:boto:Path: /ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.285 DEBUG:boto:Data: 2023-11-02T18:52:42.286 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.286 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.287 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.289 DEBUG:boto:Params: {} 2023-11-02T18:52:42.290 DEBUG:boto:Token: None 2023-11-02T18:52:42.291 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.291 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ubt7sPu60/kZQjo5y6tyJ+Da19s= 2023-11-02T18:52:42.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ubt7sPu60/kZQjo5y6tyJ+Da19s=', 'Content-Length': '0'} 2023-11-02T18:52:42.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f93f4376ebd33732-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.296 DEBUG:boto:path=/ibtljn-61/obj44 2023-11-02T18:52:42.296 DEBUG:boto:auth_path=/ibtljn-61/obj44 2023-11-02T18:52:42.297 DEBUG:boto:path=/ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.297 DEBUG:boto:auth_path=/ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.298 DEBUG:boto:Method: GET 2023-11-02T18:52:42.298 DEBUG:boto:Path: /ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.299 DEBUG:boto:Data: 2023-11-02T18:52:42.299 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.300 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.300 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.301 DEBUG:boto:Params: {} 2023-11-02T18:52:42.301 DEBUG:boto:Token: None 2023-11-02T18:52:42.302 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj44?versionId=null 2023-11-02T18:52:42.302 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ubt7sPu60/kZQjo5y6tyJ+Da19s= 2023-11-02T18:52:42.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ubt7sPu60/kZQjo5y6tyJ+Da19s=', 'Content-Length': '0'} 2023-11-02T18:52:42.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070348b0f525adc51-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.307 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.308 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.308 DEBUG:boto:path=/ibtljn-61/?prefix=obj44 2023-11-02T18:52:42.309 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj44 2023-11-02T18:52:42.309 DEBUG:boto:Method: GET 2023-11-02T18:52:42.310 DEBUG:boto:Path: /ibtljn-61/?prefix=obj44 2023-11-02T18:52:42.310 DEBUG:boto:Data: 2023-11-02T18:52:42.311 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.311 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.312 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.312 DEBUG:boto:Params: {} 2023-11-02T18:52:42.312 DEBUG:boto:Token: None 2023-11-02T18:52:42.313 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.313 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5db32711c2f1efa-006543effa-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.362 DEBUG:boto:b'ibtljn-61obj441000falseobj442023-11-02T18:50:22.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.363 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.364 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.364 DEBUG:boto:path=/ibtljn-61/?prefix=obj44 2023-11-02T18:52:42.365 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj44 2023-11-02T18:52:42.365 DEBUG:boto:Method: GET 2023-11-02T18:52:42.366 DEBUG:boto:Path: /ibtljn-61/?prefix=obj44 2023-11-02T18:52:42.366 DEBUG:boto:Data: 2023-11-02T18:52:42.367 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.367 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.368 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.368 DEBUG:boto:Params: {} 2023-11-02T18:52:42.369 DEBUG:boto:Token: None 2023-11-02T18:52:42.369 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.370 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e14aaa32d4b0c352-006543effa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.379 DEBUG:boto:b'ibtljn-61obj441000falseobj442023-11-02T18:50:22.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.380 DEBUG:boto:path=/ibtljn-61/obj44 2023-11-02T18:52:42.381 DEBUG:boto:auth_path=/ibtljn-61/obj44 2023-11-02T18:52:42.381 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.381 DEBUG:boto:Path: /ibtljn-61/obj44 2023-11-02T18:52:42.382 DEBUG:boto:Data: 2023-11-02T18:52:42.383 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.383 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.384 DEBUG:boto:Params: {} 2023-11-02T18:52:42.384 DEBUG:boto:Token: None 2023-11-02T18:52:42.385 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj44 2023-11-02T18:52:42.385 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kBakkgL6Ir8BvCTvslhJjQdkXzw= 2023-11-02T18:52:42.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kBakkgL6Ir8BvCTvslhJjQdkXzw=', 'Content-Length': '0'} 2023-11-02T18:52:42.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6e313df30d237f4-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.389 DEBUG:boto:path=/ibtljn-61/obj44 2023-11-02T18:52:42.390 DEBUG:boto:auth_path=/ibtljn-61/obj44 2023-11-02T18:52:42.390 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.391 DEBUG:boto:Path: /ibtljn-61/obj44 2023-11-02T18:52:42.391 DEBUG:boto:Data: 2023-11-02T18:52:42.392 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.392 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.393 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.393 DEBUG:boto:Params: {} 2023-11-02T18:52:42.394 DEBUG:boto:Token: None 2023-11-02T18:52:42.394 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj44 2023-11-02T18:52:42.395 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kBakkgL6Ir8BvCTvslhJjQdkXzw= 2023-11-02T18:52:42.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kBakkgL6Ir8BvCTvslhJjQdkXzw=', 'Content-Length': '0'} 2023-11-02T18:52:42.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd59f436e3fc2006-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:52:42.399 DEBUG:boto:path=/ibtljn-61/obj44 2023-11-02T18:52:42.400 DEBUG:boto:auth_path=/ibtljn-61/obj44 2023-11-02T18:52:42.400 DEBUG:boto:Method: GET 2023-11-02T18:52:42.401 DEBUG:boto:Path: /ibtljn-61/obj44 2023-11-02T18:52:42.401 DEBUG:boto:Data: 2023-11-02T18:52:42.402 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.402 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.403 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.403 DEBUG:boto:Params: {} 2023-11-02T18:52:42.404 DEBUG:boto:Token: None 2023-11-02T18:52:42.404 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj44 2023-11-02T18:52:42.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:U5rpeOd6U4HwxYW7GwfqqGLtXlM= 2023-11-02T18:52:42.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:U5rpeOd6U4HwxYW7GwfqqGLtXlM=', 'Content-Length': '0'} 2023-11-02T18:52:42.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008199e0af3479a23-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.409 DEBUG:boto:path=/ibtljn-61/obj44 2023-11-02T18:52:42.410 DEBUG:boto:auth_path=/ibtljn-61/obj44 2023-11-02T18:52:42.410 DEBUG:boto:Method: GET 2023-11-02T18:52:42.411 DEBUG:boto:Path: /ibtljn-61/obj44 2023-11-02T18:52:42.411 DEBUG:boto:Data: 2023-11-02T18:52:42.412 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.413 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.413 DEBUG:boto:Params: {} 2023-11-02T18:52:42.414 DEBUG:boto:Token: None 2023-11-02T18:52:42.415 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj44 2023-11-02T18:52:42.415 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:U5rpeOd6U4HwxYW7GwfqqGLtXlM= 2023-11-02T18:52:42.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:U5rpeOd6U4HwxYW7GwfqqGLtXlM=', 'Content-Length': '0'} 2023-11-02T18:52:42.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000365b030848dcf502-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.419 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.420 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.420 DEBUG:boto:path=/ibtljn-61/?prefix=obj44 2023-11-02T18:52:42.421 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj44 2023-11-02T18:52:42.421 DEBUG:boto:Method: GET 2023-11-02T18:52:42.422 DEBUG:boto:Path: /ibtljn-61/?prefix=obj44 2023-11-02T18:52:42.422 DEBUG:boto:Data: 2023-11-02T18:52:42.423 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.423 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.424 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.424 DEBUG:boto:Params: {} 2023-11-02T18:52:42.424 DEBUG:boto:Token: None 2023-11-02T18:52:42.425 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.426 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2d021439131dc34-006543effa-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.432 DEBUG:boto:b'ibtljn-61obj441000falseobj442023-11-02T18:50:22.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.433 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.433 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.434 DEBUG:boto:path=/ibtljn-61/?prefix=obj44 2023-11-02T18:52:42.434 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj44 2023-11-02T18:52:42.435 DEBUG:boto:Method: GET 2023-11-02T18:52:42.435 DEBUG:boto:Path: /ibtljn-61/?prefix=obj44 2023-11-02T18:52:42.436 DEBUG:boto:Data: 2023-11-02T18:52:42.436 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.437 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.437 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.438 DEBUG:boto:Params: {} 2023-11-02T18:52:42.439 DEBUG:boto:Token: None 2023-11-02T18:52:42.439 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.440 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eef801b6fc94eb27-006543effa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.490 DEBUG:boto:b'ibtljn-61obj441000falseobj442023-11-02T18:50:22.118Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:52:42.493 DEBUG:boto:path=/ibtljn-61/obj46 2023-11-02T18:52:42.494 DEBUG:boto:auth_path=/ibtljn-61/obj46 2023-11-02T18:52:42.494 DEBUG:boto:path=/ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.495 DEBUG:boto:auth_path=/ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.495 DEBUG:boto:Method: GET 2023-11-02T18:52:42.496 DEBUG:boto:Path: /ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.496 DEBUG:boto:Data: 2023-11-02T18:52:42.497 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.497 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.498 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.498 DEBUG:boto:Params: {} 2023-11-02T18:52:42.499 DEBUG:boto:Token: None 2023-11-02T18:52:42.499 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.500 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dFBRbL2tEgchJoQ142oi5mS2x0A= 2023-11-02T18:52:42.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dFBRbL2tEgchJoQ142oi5mS2x0A=', 'Content-Length': '0'} 2023-11-02T18:52:42.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b680b3079cd35f0e-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.504 DEBUG:boto:path=/ibtljn-61/obj46 2023-11-02T18:52:42.505 DEBUG:boto:auth_path=/ibtljn-61/obj46 2023-11-02T18:52:42.505 DEBUG:boto:path=/ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.506 DEBUG:boto:auth_path=/ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.506 DEBUG:boto:Method: GET 2023-11-02T18:52:42.506 DEBUG:boto:Path: /ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.507 DEBUG:boto:Data: 2023-11-02T18:52:42.507 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.508 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.508 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.509 DEBUG:boto:Params: {} 2023-11-02T18:52:42.509 DEBUG:boto:Token: None 2023-11-02T18:52:42.510 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.510 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dFBRbL2tEgchJoQ142oi5mS2x0A= 2023-11-02T18:52:42.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dFBRbL2tEgchJoQ142oi5mS2x0A=', 'Content-Length': '0'} 2023-11-02T18:52:42.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e5b5fb0089c981-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.515 DEBUG:boto:path=/ibtljn-61/obj46 2023-11-02T18:52:42.515 DEBUG:boto:auth_path=/ibtljn-61/obj46 2023-11-02T18:52:42.515 DEBUG:boto:path=/ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.516 DEBUG:boto:auth_path=/ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.516 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.517 DEBUG:boto:Path: /ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.517 DEBUG:boto:Data: 2023-11-02T18:52:42.518 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.518 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.518 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.519 DEBUG:boto:Params: {} 2023-11-02T18:52:42.519 DEBUG:boto:Token: None 2023-11-02T18:52:42.520 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.520 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7z/wHH87YEo5/6JEJuFjeoGBWDo= 2023-11-02T18:52:42.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7z/wHH87YEo5/6JEJuFjeoGBWDo=', 'Content-Length': '0'} 2023-11-02T18:52:42.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018a8a65e6de2cdc2-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.524 DEBUG:boto:path=/ibtljn-61/obj46 2023-11-02T18:52:42.525 DEBUG:boto:auth_path=/ibtljn-61/obj46 2023-11-02T18:52:42.525 DEBUG:boto:path=/ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.525 DEBUG:boto:auth_path=/ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.526 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.526 DEBUG:boto:Path: /ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.527 DEBUG:boto:Data: 2023-11-02T18:52:42.527 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.528 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.528 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.528 DEBUG:boto:Params: {} 2023-11-02T18:52:42.529 DEBUG:boto:Token: None 2023-11-02T18:52:42.529 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.530 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7z/wHH87YEo5/6JEJuFjeoGBWDo= 2023-11-02T18:52:42.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7z/wHH87YEo5/6JEJuFjeoGBWDo=', 'Content-Length': '0'} 2023-11-02T18:52:42.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000071ee4ff5e1cf7a9-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:52:42.534 DEBUG:boto:path=/ibtljn-61/obj46 2023-11-02T18:52:42.534 DEBUG:boto:auth_path=/ibtljn-61/obj46 2023-11-02T18:52:42.535 DEBUG:boto:path=/ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.535 DEBUG:boto:auth_path=/ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.536 DEBUG:boto:Method: GET 2023-11-02T18:52:42.536 DEBUG:boto:Path: /ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.537 DEBUG:boto:Data: 2023-11-02T18:52:42.537 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.538 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.538 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.539 DEBUG:boto:Params: {} 2023-11-02T18:52:42.539 DEBUG:boto:Token: None 2023-11-02T18:52:42.540 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.540 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dFBRbL2tEgchJoQ142oi5mS2x0A= 2023-11-02T18:52:42.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dFBRbL2tEgchJoQ142oi5mS2x0A=', 'Content-Length': '0'} 2023-11-02T18:52:42.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003be3d12be3e4bb72-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.586 DEBUG:boto:path=/ibtljn-61/obj46 2023-11-02T18:52:42.587 DEBUG:boto:auth_path=/ibtljn-61/obj46 2023-11-02T18:52:42.587 DEBUG:boto:path=/ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.588 DEBUG:boto:auth_path=/ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.588 DEBUG:boto:Method: GET 2023-11-02T18:52:42.589 DEBUG:boto:Path: /ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.589 DEBUG:boto:Data: 2023-11-02T18:52:42.590 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.591 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.591 DEBUG:boto:Params: {} 2023-11-02T18:52:42.591 DEBUG:boto:Token: None 2023-11-02T18:52:42.592 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj46?versionId=null 2023-11-02T18:52:42.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dFBRbL2tEgchJoQ142oi5mS2x0A= 2023-11-02T18:52:42.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dFBRbL2tEgchJoQ142oi5mS2x0A=', 'Content-Length': '0'} 2023-11-02T18:52:42.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7f8a3edc9c4a4a2-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.597 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.597 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.598 DEBUG:boto:path=/ibtljn-61/?prefix=obj46 2023-11-02T18:52:42.598 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj46 2023-11-02T18:52:42.599 DEBUG:boto:Method: GET 2023-11-02T18:52:42.599 DEBUG:boto:Path: /ibtljn-61/?prefix=obj46 2023-11-02T18:52:42.600 DEBUG:boto:Data: 2023-11-02T18:52:42.600 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.601 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.601 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.602 DEBUG:boto:Params: {} 2023-11-02T18:52:42.602 DEBUG:boto:Token: None 2023-11-02T18:52:42.603 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.603 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000487f8ff13b893f9f-006543effa-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.610 DEBUG:boto:b'ibtljn-61obj461000falseobj462023-11-02T18:50:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.610 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.611 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.611 DEBUG:boto:path=/ibtljn-61/?prefix=obj46 2023-11-02T18:52:42.612 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj46 2023-11-02T18:52:42.612 DEBUG:boto:Method: GET 2023-11-02T18:52:42.612 DEBUG:boto:Path: /ibtljn-61/?prefix=obj46 2023-11-02T18:52:42.613 DEBUG:boto:Data: 2023-11-02T18:52:42.613 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.614 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.614 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.615 DEBUG:boto:Params: {} 2023-11-02T18:52:42.615 DEBUG:boto:Token: None 2023-11-02T18:52:42.616 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.616 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034b828c47a190fac-006543effa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.627 DEBUG:boto:b'ibtljn-61obj461000falseobj462023-11-02T18:50:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.628 DEBUG:boto:path=/ibtljn-61/obj46 2023-11-02T18:52:42.629 DEBUG:boto:auth_path=/ibtljn-61/obj46 2023-11-02T18:52:42.629 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.629 DEBUG:boto:Path: /ibtljn-61/obj46 2023-11-02T18:52:42.630 DEBUG:boto:Data: 2023-11-02T18:52:42.630 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.631 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.631 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.632 DEBUG:boto:Params: {} 2023-11-02T18:52:42.632 DEBUG:boto:Token: None 2023-11-02T18:52:42.633 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj46 2023-11-02T18:52:42.633 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:49qVzdbNiWbRzJFdk/5m6eEqa3U= 2023-11-02T18:52:42.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:49qVzdbNiWbRzJFdk/5m6eEqa3U=', 'Content-Length': '0'} 2023-11-02T18:52:42.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096340d430dd783fa-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.637 DEBUG:boto:path=/ibtljn-61/obj46 2023-11-02T18:52:42.637 DEBUG:boto:auth_path=/ibtljn-61/obj46 2023-11-02T18:52:42.637 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.638 DEBUG:boto:Path: /ibtljn-61/obj46 2023-11-02T18:52:42.638 DEBUG:boto:Data: 2023-11-02T18:52:42.639 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.640 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.640 DEBUG:boto:Params: {} 2023-11-02T18:52:42.640 DEBUG:boto:Token: None 2023-11-02T18:52:42.641 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj46 2023-11-02T18:52:42.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:49qVzdbNiWbRzJFdk/5m6eEqa3U= 2023-11-02T18:52:42.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:49qVzdbNiWbRzJFdk/5m6eEqa3U=', 'Content-Length': '0'} 2023-11-02T18:52:42.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee5efe8d03bb8266-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T18:52:42.645 DEBUG:boto:path=/ibtljn-61/obj46 2023-11-02T18:52:42.646 DEBUG:boto:auth_path=/ibtljn-61/obj46 2023-11-02T18:52:42.646 DEBUG:boto:Method: GET 2023-11-02T18:52:42.647 DEBUG:boto:Path: /ibtljn-61/obj46 2023-11-02T18:52:42.647 DEBUG:boto:Data: 2023-11-02T18:52:42.648 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.648 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.649 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.649 DEBUG:boto:Params: {} 2023-11-02T18:52:42.649 DEBUG:boto:Token: None 2023-11-02T18:52:42.650 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj46 2023-11-02T18:52:42.650 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c9YR86qZQx3qnB97nUZpRiqC0oY= 2023-11-02T18:52:42.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c9YR86qZQx3qnB97nUZpRiqC0oY=', 'Content-Length': '0'} 2023-11-02T18:52:42.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c41efc378c08a833-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.655 DEBUG:boto:path=/ibtljn-61/obj46 2023-11-02T18:52:42.655 DEBUG:boto:auth_path=/ibtljn-61/obj46 2023-11-02T18:52:42.656 DEBUG:boto:Method: GET 2023-11-02T18:52:42.656 DEBUG:boto:Path: /ibtljn-61/obj46 2023-11-02T18:52:42.657 DEBUG:boto:Data: 2023-11-02T18:52:42.657 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.658 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.658 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.659 DEBUG:boto:Params: {} 2023-11-02T18:52:42.659 DEBUG:boto:Token: None 2023-11-02T18:52:42.660 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj46 2023-11-02T18:52:42.660 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c9YR86qZQx3qnB97nUZpRiqC0oY= 2023-11-02T18:52:42.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c9YR86qZQx3qnB97nUZpRiqC0oY=', 'Content-Length': '0'} 2023-11-02T18:52:42.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9c8042c6df7bc4c-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.710 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.711 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.712 DEBUG:boto:path=/ibtljn-61/?prefix=obj46 2023-11-02T18:52:42.712 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj46 2023-11-02T18:52:42.713 DEBUG:boto:Method: GET 2023-11-02T18:52:42.713 DEBUG:boto:Path: /ibtljn-61/?prefix=obj46 2023-11-02T18:52:42.714 DEBUG:boto:Data: 2023-11-02T18:52:42.714 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.715 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.715 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.716 DEBUG:boto:Params: {} 2023-11-02T18:52:42.716 DEBUG:boto:Token: None 2023-11-02T18:52:42.717 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.717 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007509bb174f8edb96-006543effa-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.724 DEBUG:boto:b'ibtljn-61obj461000falseobj462023-11-02T18:50:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.725 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.726 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.726 DEBUG:boto:path=/ibtljn-61/?prefix=obj46 2023-11-02T18:52:42.727 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj46 2023-11-02T18:52:42.727 DEBUG:boto:Method: GET 2023-11-02T18:52:42.728 DEBUG:boto:Path: /ibtljn-61/?prefix=obj46 2023-11-02T18:52:42.728 DEBUG:boto:Data: 2023-11-02T18:52:42.729 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.729 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.730 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.730 DEBUG:boto:Params: {} 2023-11-02T18:52:42.731 DEBUG:boto:Token: None 2023-11-02T18:52:42.731 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.732 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006da337f3b6d17f96-006543effa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.742 DEBUG:boto:b'ibtljn-61obj461000falseobj462023-11-02T18:50:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:52:42.743 DEBUG:boto:path=/ibtljn-61/obj51 2023-11-02T18:52:42.744 DEBUG:boto:auth_path=/ibtljn-61/obj51 2023-11-02T18:52:42.744 DEBUG:boto:path=/ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.745 DEBUG:boto:auth_path=/ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.745 DEBUG:boto:Method: GET 2023-11-02T18:52:42.746 DEBUG:boto:Path: /ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.746 DEBUG:boto:Data: 2023-11-02T18:52:42.747 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.747 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.748 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.748 DEBUG:boto:Params: {} 2023-11-02T18:52:42.749 DEBUG:boto:Token: None 2023-11-02T18:52:42.750 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.750 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rl1a2MG8Vp3zdI+cVkvqiaCP8ts= 2023-11-02T18:52:42.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rl1a2MG8Vp3zdI+cVkvqiaCP8ts=', 'Content-Length': '0'} 2023-11-02T18:52:42.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bdb29d233c2332a-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.794 DEBUG:boto:path=/ibtljn-61/obj51 2023-11-02T18:52:42.795 DEBUG:boto:auth_path=/ibtljn-61/obj51 2023-11-02T18:52:42.796 DEBUG:boto:path=/ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.796 DEBUG:boto:auth_path=/ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.797 DEBUG:boto:Method: GET 2023-11-02T18:52:42.797 DEBUG:boto:Path: /ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.798 DEBUG:boto:Data: 2023-11-02T18:52:42.798 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.799 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.800 DEBUG:boto:Params: {} 2023-11-02T18:52:42.800 DEBUG:boto:Token: None 2023-11-02T18:52:42.801 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.802 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rl1a2MG8Vp3zdI+cVkvqiaCP8ts= 2023-11-02T18:52:42.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rl1a2MG8Vp3zdI+cVkvqiaCP8ts=', 'Content-Length': '0'} 2023-11-02T18:52:42.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094691f8631875427-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.806 DEBUG:boto:path=/ibtljn-61/obj51 2023-11-02T18:52:42.806 DEBUG:boto:auth_path=/ibtljn-61/obj51 2023-11-02T18:52:42.807 DEBUG:boto:path=/ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.807 DEBUG:boto:auth_path=/ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.808 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.808 DEBUG:boto:Path: /ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.809 DEBUG:boto:Data: 2023-11-02T18:52:42.809 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.810 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.810 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.811 DEBUG:boto:Params: {} 2023-11-02T18:52:42.811 DEBUG:boto:Token: None 2023-11-02T18:52:42.812 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.812 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cnYbBsC7ffQWVSV/o8/xw0QKkTM= 2023-11-02T18:52:42.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cnYbBsC7ffQWVSV/o8/xw0QKkTM=', 'Content-Length': '0'} 2023-11-02T18:52:42.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da02d79a79303b7-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.816 DEBUG:boto:path=/ibtljn-61/obj51 2023-11-02T18:52:42.817 DEBUG:boto:auth_path=/ibtljn-61/obj51 2023-11-02T18:52:42.818 DEBUG:boto:path=/ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.818 DEBUG:boto:auth_path=/ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.819 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.820 DEBUG:boto:Path: /ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.820 DEBUG:boto:Data: 2023-11-02T18:52:42.820 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.821 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.821 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.822 DEBUG:boto:Params: {} 2023-11-02T18:52:42.822 DEBUG:boto:Token: None 2023-11-02T18:52:42.823 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.823 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cnYbBsC7ffQWVSV/o8/xw0QKkTM= 2023-11-02T18:52:42.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cnYbBsC7ffQWVSV/o8/xw0QKkTM=', 'Content-Length': '0'} 2023-11-02T18:52:42.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a2e1b5cc954eccc-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:52:42.827 DEBUG:boto:path=/ibtljn-61/obj51 2023-11-02T18:52:42.828 DEBUG:boto:auth_path=/ibtljn-61/obj51 2023-11-02T18:52:42.828 DEBUG:boto:path=/ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.829 DEBUG:boto:auth_path=/ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.829 DEBUG:boto:Method: GET 2023-11-02T18:52:42.830 DEBUG:boto:Path: /ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.830 DEBUG:boto:Data: 2023-11-02T18:52:42.831 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.831 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.832 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.832 DEBUG:boto:Params: {} 2023-11-02T18:52:42.832 DEBUG:boto:Token: None 2023-11-02T18:52:42.833 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.834 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rl1a2MG8Vp3zdI+cVkvqiaCP8ts= 2023-11-02T18:52:42.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rl1a2MG8Vp3zdI+cVkvqiaCP8ts=', 'Content-Length': '0'} 2023-11-02T18:52:42.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b899ee81a18e421f-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.838 DEBUG:boto:path=/ibtljn-61/obj51 2023-11-02T18:52:42.838 DEBUG:boto:auth_path=/ibtljn-61/obj51 2023-11-02T18:52:42.839 DEBUG:boto:path=/ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.839 DEBUG:boto:auth_path=/ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.840 DEBUG:boto:Method: GET 2023-11-02T18:52:42.840 DEBUG:boto:Path: /ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.841 DEBUG:boto:Data: 2023-11-02T18:52:42.841 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.842 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.842 DEBUG:boto:Params: {} 2023-11-02T18:52:42.843 DEBUG:boto:Token: None 2023-11-02T18:52:42.843 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj51?versionId=null 2023-11-02T18:52:42.844 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rl1a2MG8Vp3zdI+cVkvqiaCP8ts= 2023-11-02T18:52:42.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rl1a2MG8Vp3zdI+cVkvqiaCP8ts=', 'Content-Length': '0'} 2023-11-02T18:52:42.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f98ac9da277d644-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.848 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.848 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.849 DEBUG:boto:path=/ibtljn-61/?prefix=obj51 2023-11-02T18:52:42.849 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj51 2023-11-02T18:52:42.850 DEBUG:boto:Method: GET 2023-11-02T18:52:42.850 DEBUG:boto:Path: /ibtljn-61/?prefix=obj51 2023-11-02T18:52:42.851 DEBUG:boto:Data: 2023-11-02T18:52:42.851 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.852 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.852 DEBUG:boto:Params: {} 2023-11-02T18:52:42.853 DEBUG:boto:Token: None 2023-11-02T18:52:42.853 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.854 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ade8081381370fb5-006543effa-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.861 DEBUG:boto:b'ibtljn-61obj511000falseobj512023-11-02T18:50:22.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.861 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:42.862 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:42.862 DEBUG:boto:path=/ibtljn-61/?prefix=obj51 2023-11-02T18:52:42.863 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj51 2023-11-02T18:52:42.863 DEBUG:boto:Method: GET 2023-11-02T18:52:42.864 DEBUG:boto:Path: /ibtljn-61/?prefix=obj51 2023-11-02T18:52:42.864 DEBUG:boto:Data: 2023-11-02T18:52:42.865 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.865 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.866 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.866 DEBUG:boto:Params: {} 2023-11-02T18:52:42.867 DEBUG:boto:Token: None 2023-11-02T18:52:42.867 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/ 2023-11-02T18:52:42.868 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018= 2023-11-02T18:52:42.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQackfjlzczSg8aaZaPHsaH7018=', 'Content-Length': '0'} 2023-11-02T18:52:42.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e4c50772f3d7e4e-006543effa-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.926 DEBUG:boto:b'ibtljn-61obj511000falseobj512023-11-02T18:50:22.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:42.927 DEBUG:boto:path=/ibtljn-61/obj51 2023-11-02T18:52:42.928 DEBUG:boto:auth_path=/ibtljn-61/obj51 2023-11-02T18:52:42.928 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.929 DEBUG:boto:Path: /ibtljn-61/obj51 2023-11-02T18:52:42.929 DEBUG:boto:Data: 2023-11-02T18:52:42.930 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.931 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.931 DEBUG:boto:Params: {} 2023-11-02T18:52:42.931 DEBUG:boto:Token: None 2023-11-02T18:52:42.932 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj51 2023-11-02T18:52:42.932 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5bBVkXhgGjqOSETKCHsaNUHwkT8= 2023-11-02T18:52:42.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5bBVkXhgGjqOSETKCHsaNUHwkT8=', 'Content-Length': '0'} 2023-11-02T18:52:42.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a69c9440493bf76-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.937 DEBUG:boto:path=/ibtljn-61/obj51 2023-11-02T18:52:42.937 DEBUG:boto:auth_path=/ibtljn-61/obj51 2023-11-02T18:52:42.938 DEBUG:boto:Method: HEAD 2023-11-02T18:52:42.938 DEBUG:boto:Path: /ibtljn-61/obj51 2023-11-02T18:52:42.939 DEBUG:boto:Data: 2023-11-02T18:52:42.939 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.939 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:42.940 DEBUG:boto:Port: 8000 2023-11-02T18:52:42.940 DEBUG:boto:Params: {} 2023-11-02T18:52:42.941 DEBUG:boto:Token: None 2023-11-02T18:52:42.942 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj51 2023-11-02T18:52:42.942 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5bBVkXhgGjqOSETKCHsaNUHwkT8= 2023-11-02T18:52:42.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5bBVkXhgGjqOSETKCHsaNUHwkT8=', 'Content-Length': '0'} 2023-11-02T18:52:42.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fae99a8a11b5f01-006543effa-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T18:52:42.948 DEBUG:boto:path=/ibtljn-61/obj51 2023-11-02T18:52:42.949 DEBUG:boto:auth_path=/ibtljn-61/obj51 2023-11-02T18:52:42.949 DEBUG:boto:Method: GET 2023-11-02T18:52:42.950 DEBUG:boto:Path: /ibtljn-61/obj51 2023-11-02T18:52:42.950 DEBUG:boto:Data: 2023-11-02T18:52:42.951 DEBUG:boto:Headers: {} 2023-11-02T18:52:42.951 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:42.952 DEBUG:boto:Port: 8001 2023-11-02T18:52:42.952 DEBUG:boto:Params: {} 2023-11-02T18:52:42.953 DEBUG:boto:Token: None 2023-11-02T18:52:42.953 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:42 GMT /ibtljn-61/obj51 2023-11-02T18:52:42.954 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:X1lqW1/HSsGE6xV1wqLUFGi0a+c= 2023-11-02T18:52:42.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:X1lqW1/HSsGE6xV1wqLUFGi0a+c=', 'Content-Length': '0'} 2023-11-02T18:52:42.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bf7e4dc0fc56bfd-006543effa-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:42.998 DEBUG:boto:path=/ibtljn-61/obj51 2023-11-02T18:52:42.999 DEBUG:boto:auth_path=/ibtljn-61/obj51 2023-11-02T18:52:43.000 DEBUG:boto:Method: GET 2023-11-02T18:52:43.000 DEBUG:boto:Path: /ibtljn-61/obj51 2023-11-02T18:52:43.001 DEBUG:boto:Data: 2023-11-02T18:52:43.001 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.002 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.002 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.003 DEBUG:boto:Params: {} 2023-11-02T18:52:43.004 DEBUG:boto:Token: None 2023-11-02T18:52:43.004 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj51 2023-11-02T18:52:43.005 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Spl2nFB93gd/wBcDppAWmX1BngY= 2023-11-02T18:52:43.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Spl2nFB93gd/wBcDppAWmX1BngY=', 'Content-Length': '0'} 2023-11-02T18:52:43.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfc2e59749d649e4-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.009 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.010 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.010 DEBUG:boto:path=/ibtljn-61/?prefix=obj51 2023-11-02T18:52:43.011 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj51 2023-11-02T18:52:43.011 DEBUG:boto:Method: GET 2023-11-02T18:52:43.012 DEBUG:boto:Path: /ibtljn-61/?prefix=obj51 2023-11-02T18:52:43.012 DEBUG:boto:Data: 2023-11-02T18:52:43.013 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.013 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.014 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.014 DEBUG:boto:Params: {} 2023-11-02T18:52:43.015 DEBUG:boto:Token: None 2023-11-02T18:52:43.016 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000243dbdae11ea0a1-006543effb-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.023 DEBUG:boto:b'ibtljn-61obj511000falseobj512023-11-02T18:50:22.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.024 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.025 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.025 DEBUG:boto:path=/ibtljn-61/?prefix=obj51 2023-11-02T18:52:43.026 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj51 2023-11-02T18:52:43.026 DEBUG:boto:Method: GET 2023-11-02T18:52:43.027 DEBUG:boto:Path: /ibtljn-61/?prefix=obj51 2023-11-02T18:52:43.027 DEBUG:boto:Data: 2023-11-02T18:52:43.028 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.028 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.029 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.029 DEBUG:boto:Params: {} 2023-11-02T18:52:43.030 DEBUG:boto:Token: None 2023-11-02T18:52:43.030 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.031 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033dd9c1ce4f2afe1-006543effb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.041 DEBUG:boto:b'ibtljn-61obj511000falseobj512023-11-02T18:50:22.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:52:43.043 DEBUG:boto:path=/ibtljn-61/obj55 2023-11-02T18:52:43.043 DEBUG:boto:auth_path=/ibtljn-61/obj55 2023-11-02T18:52:43.043 DEBUG:boto:path=/ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.044 DEBUG:boto:auth_path=/ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.045 DEBUG:boto:Method: GET 2023-11-02T18:52:43.045 DEBUG:boto:Path: /ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.046 DEBUG:boto:Data: 2023-11-02T18:52:43.046 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.047 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.047 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.048 DEBUG:boto:Params: {} 2023-11-02T18:52:43.048 DEBUG:boto:Token: None 2023-11-02T18:52:43.049 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.049 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nsmpod95tC+QVQ9cNFuBEJ3nwYw= 2023-11-02T18:52:43.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nsmpod95tC+QVQ9cNFuBEJ3nwYw=', 'Content-Length': '0'} 2023-11-02T18:52:43.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000713fd917e88dc558-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.053 DEBUG:boto:path=/ibtljn-61/obj55 2023-11-02T18:52:43.053 DEBUG:boto:auth_path=/ibtljn-61/obj55 2023-11-02T18:52:43.054 DEBUG:boto:path=/ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.054 DEBUG:boto:auth_path=/ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.055 DEBUG:boto:Method: GET 2023-11-02T18:52:43.055 DEBUG:boto:Path: /ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.056 DEBUG:boto:Data: 2023-11-02T18:52:43.056 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.057 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.057 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.058 DEBUG:boto:Params: {} 2023-11-02T18:52:43.058 DEBUG:boto:Token: None 2023-11-02T18:52:43.059 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.060 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nsmpod95tC+QVQ9cNFuBEJ3nwYw= 2023-11-02T18:52:43.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nsmpod95tC+QVQ9cNFuBEJ3nwYw=', 'Content-Length': '0'} 2023-11-02T18:52:43.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007924227802172330-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.063 DEBUG:boto:path=/ibtljn-61/obj55 2023-11-02T18:52:43.064 DEBUG:boto:auth_path=/ibtljn-61/obj55 2023-11-02T18:52:43.064 DEBUG:boto:path=/ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.065 DEBUG:boto:auth_path=/ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.065 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.065 DEBUG:boto:Path: /ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.066 DEBUG:boto:Data: 2023-11-02T18:52:43.066 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.067 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.067 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.068 DEBUG:boto:Params: {} 2023-11-02T18:52:43.068 DEBUG:boto:Token: None 2023-11-02T18:52:43.069 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.070 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZjRD56DHdof2uKznHJUZc819y9w= 2023-11-02T18:52:43.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZjRD56DHdof2uKznHJUZc819y9w=', 'Content-Length': '0'} 2023-11-02T18:52:43.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5dfe4e384db4b98-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.073 DEBUG:boto:path=/ibtljn-61/obj55 2023-11-02T18:52:43.074 DEBUG:boto:auth_path=/ibtljn-61/obj55 2023-11-02T18:52:43.075 DEBUG:boto:path=/ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.075 DEBUG:boto:auth_path=/ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.076 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.076 DEBUG:boto:Path: /ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.077 DEBUG:boto:Data: 2023-11-02T18:52:43.077 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.078 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.078 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.078 DEBUG:boto:Params: {} 2023-11-02T18:52:43.079 DEBUG:boto:Token: None 2023-11-02T18:52:43.080 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.080 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZjRD56DHdof2uKznHJUZc819y9w= 2023-11-02T18:52:43.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZjRD56DHdof2uKznHJUZc819y9w=', 'Content-Length': '0'} 2023-11-02T18:52:43.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077a8ed090821e295-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:52:43.085 DEBUG:boto:path=/ibtljn-61/obj55 2023-11-02T18:52:43.085 DEBUG:boto:auth_path=/ibtljn-61/obj55 2023-11-02T18:52:43.086 DEBUG:boto:path=/ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.086 DEBUG:boto:auth_path=/ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.087 DEBUG:boto:Method: GET 2023-11-02T18:52:43.087 DEBUG:boto:Path: /ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.088 DEBUG:boto:Data: 2023-11-02T18:52:43.088 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.089 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.089 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.090 DEBUG:boto:Params: {} 2023-11-02T18:52:43.090 DEBUG:boto:Token: None 2023-11-02T18:52:43.091 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.091 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nsmpod95tC+QVQ9cNFuBEJ3nwYw= 2023-11-02T18:52:43.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nsmpod95tC+QVQ9cNFuBEJ3nwYw=', 'Content-Length': '0'} 2023-11-02T18:52:43.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd706eed0b470cc8-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.095 DEBUG:boto:path=/ibtljn-61/obj55 2023-11-02T18:52:43.096 DEBUG:boto:auth_path=/ibtljn-61/obj55 2023-11-02T18:52:43.096 DEBUG:boto:path=/ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.097 DEBUG:boto:auth_path=/ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.098 DEBUG:boto:Method: GET 2023-11-02T18:52:43.098 DEBUG:boto:Path: /ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.099 DEBUG:boto:Data: 2023-11-02T18:52:43.099 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.101 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.101 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.102 DEBUG:boto:Params: {} 2023-11-02T18:52:43.102 DEBUG:boto:Token: None 2023-11-02T18:52:43.103 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj55?versionId=null 2023-11-02T18:52:43.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nsmpod95tC+QVQ9cNFuBEJ3nwYw= 2023-11-02T18:52:43.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nsmpod95tC+QVQ9cNFuBEJ3nwYw=', 'Content-Length': '0'} 2023-11-02T18:52:43.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000180f24f1b448de48-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.150 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.151 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.151 DEBUG:boto:path=/ibtljn-61/?prefix=obj55 2023-11-02T18:52:43.152 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj55 2023-11-02T18:52:43.152 DEBUG:boto:Method: GET 2023-11-02T18:52:43.153 DEBUG:boto:Path: /ibtljn-61/?prefix=obj55 2023-11-02T18:52:43.153 DEBUG:boto:Data: 2023-11-02T18:52:43.154 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.154 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.154 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.155 DEBUG:boto:Params: {} 2023-11-02T18:52:43.155 DEBUG:boto:Token: None 2023-11-02T18:52:43.156 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.157 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016b93bdfdb2f47b5-006543effb-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.163 DEBUG:boto:b'ibtljn-61obj551000falseobj552023-11-02T18:50:22.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.164 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.164 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.165 DEBUG:boto:path=/ibtljn-61/?prefix=obj55 2023-11-02T18:52:43.165 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj55 2023-11-02T18:52:43.166 DEBUG:boto:Method: GET 2023-11-02T18:52:43.166 DEBUG:boto:Path: /ibtljn-61/?prefix=obj55 2023-11-02T18:52:43.167 DEBUG:boto:Data: 2023-11-02T18:52:43.167 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.167 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.168 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.168 DEBUG:boto:Params: {} 2023-11-02T18:52:43.169 DEBUG:boto:Token: None 2023-11-02T18:52:43.169 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.170 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ab103e28c28c678-006543effb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.180 DEBUG:boto:b'ibtljn-61obj551000falseobj552023-11-02T18:50:22.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.181 DEBUG:boto:path=/ibtljn-61/obj55 2023-11-02T18:52:43.181 DEBUG:boto:auth_path=/ibtljn-61/obj55 2023-11-02T18:52:43.182 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.182 DEBUG:boto:Path: /ibtljn-61/obj55 2023-11-02T18:52:43.183 DEBUG:boto:Data: 2023-11-02T18:52:43.183 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.183 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.184 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.184 DEBUG:boto:Params: {} 2023-11-02T18:52:43.185 DEBUG:boto:Token: None 2023-11-02T18:52:43.185 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj55 2023-11-02T18:52:43.186 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ffXAwcOONCNXLOMytcnkyIP9aFA= 2023-11-02T18:52:43.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ffXAwcOONCNXLOMytcnkyIP9aFA=', 'Content-Length': '0'} 2023-11-02T18:52:43.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003655c18496d178d4-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.190 DEBUG:boto:path=/ibtljn-61/obj55 2023-11-02T18:52:43.191 DEBUG:boto:auth_path=/ibtljn-61/obj55 2023-11-02T18:52:43.191 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.191 DEBUG:boto:Path: /ibtljn-61/obj55 2023-11-02T18:52:43.192 DEBUG:boto:Data: 2023-11-02T18:52:43.193 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.194 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.194 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.195 DEBUG:boto:Params: {} 2023-11-02T18:52:43.195 DEBUG:boto:Token: None 2023-11-02T18:52:43.196 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj55 2023-11-02T18:52:43.196 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ffXAwcOONCNXLOMytcnkyIP9aFA= 2023-11-02T18:52:43.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ffXAwcOONCNXLOMytcnkyIP9aFA=', 'Content-Length': '0'} 2023-11-02T18:52:43.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054173e93b3bf2fa2-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:52:43.201 DEBUG:boto:path=/ibtljn-61/obj55 2023-11-02T18:52:43.201 DEBUG:boto:auth_path=/ibtljn-61/obj55 2023-11-02T18:52:43.202 DEBUG:boto:Method: GET 2023-11-02T18:52:43.202 DEBUG:boto:Path: /ibtljn-61/obj55 2023-11-02T18:52:43.203 DEBUG:boto:Data: 2023-11-02T18:52:43.203 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.204 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.204 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.205 DEBUG:boto:Params: {} 2023-11-02T18:52:43.205 DEBUG:boto:Token: None 2023-11-02T18:52:43.206 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj55 2023-11-02T18:52:43.207 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i3nd/hfjnFSK5F0jTKAx2jjLuZ8= 2023-11-02T18:52:43.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i3nd/hfjnFSK5F0jTKAx2jjLuZ8=', 'Content-Length': '0'} 2023-11-02T18:52:43.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0a790540f738b7-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.254 DEBUG:boto:path=/ibtljn-61/obj55 2023-11-02T18:52:43.255 DEBUG:boto:auth_path=/ibtljn-61/obj55 2023-11-02T18:52:43.256 DEBUG:boto:Method: GET 2023-11-02T18:52:43.256 DEBUG:boto:Path: /ibtljn-61/obj55 2023-11-02T18:52:43.257 DEBUG:boto:Data: 2023-11-02T18:52:43.258 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.258 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.259 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.259 DEBUG:boto:Params: {} 2023-11-02T18:52:43.260 DEBUG:boto:Token: None 2023-11-02T18:52:43.261 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj55 2023-11-02T18:52:43.261 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i3nd/hfjnFSK5F0jTKAx2jjLuZ8= 2023-11-02T18:52:43.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i3nd/hfjnFSK5F0jTKAx2jjLuZ8=', 'Content-Length': '0'} 2023-11-02T18:52:43.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085b9f6f15121095f-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.266 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.267 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.268 DEBUG:boto:path=/ibtljn-61/?prefix=obj55 2023-11-02T18:52:43.268 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj55 2023-11-02T18:52:43.269 DEBUG:boto:Method: GET 2023-11-02T18:52:43.270 DEBUG:boto:Path: /ibtljn-61/?prefix=obj55 2023-11-02T18:52:43.271 DEBUG:boto:Data: 2023-11-02T18:52:43.272 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.272 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.273 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.273 DEBUG:boto:Params: {} 2023-11-02T18:52:43.274 DEBUG:boto:Token: None 2023-11-02T18:52:43.274 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.275 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076ebb731fee34cad-006543effb-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.282 DEBUG:boto:b'ibtljn-61obj551000falseobj552023-11-02T18:50:22.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.283 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.284 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.284 DEBUG:boto:path=/ibtljn-61/?prefix=obj55 2023-11-02T18:52:43.285 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj55 2023-11-02T18:52:43.286 DEBUG:boto:Method: GET 2023-11-02T18:52:43.286 DEBUG:boto:Path: /ibtljn-61/?prefix=obj55 2023-11-02T18:52:43.287 DEBUG:boto:Data: 2023-11-02T18:52:43.287 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.288 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.288 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.289 DEBUG:boto:Params: {} 2023-11-02T18:52:43.289 DEBUG:boto:Token: None 2023-11-02T18:52:43.290 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.290 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000286e7eb23ac66238-006543effb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.301 DEBUG:boto:b'ibtljn-61obj551000falseobj552023-11-02T18:50:22.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:52:43.302 DEBUG:boto:path=/ibtljn-61/obj57 2023-11-02T18:52:43.303 DEBUG:boto:auth_path=/ibtljn-61/obj57 2023-11-02T18:52:43.303 DEBUG:boto:path=/ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.304 DEBUG:boto:auth_path=/ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.304 DEBUG:boto:Method: GET 2023-11-02T18:52:43.305 DEBUG:boto:Path: /ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.305 DEBUG:boto:Data: 2023-11-02T18:52:43.305 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.306 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.306 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.307 DEBUG:boto:Params: {} 2023-11-02T18:52:43.307 DEBUG:boto:Token: None 2023-11-02T18:52:43.308 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.308 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j2FJ9zraFYCh5neIWzp/q69t9wM= 2023-11-02T18:52:43.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j2FJ9zraFYCh5neIWzp/q69t9wM=', 'Content-Length': '0'} 2023-11-02T18:52:43.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcdb5e09188b7e47-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.313 DEBUG:boto:path=/ibtljn-61/obj57 2023-11-02T18:52:43.314 DEBUG:boto:auth_path=/ibtljn-61/obj57 2023-11-02T18:52:43.314 DEBUG:boto:path=/ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.315 DEBUG:boto:auth_path=/ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.315 DEBUG:boto:Method: GET 2023-11-02T18:52:43.316 DEBUG:boto:Path: /ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.316 DEBUG:boto:Data: 2023-11-02T18:52:43.316 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.317 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.318 DEBUG:boto:Params: {} 2023-11-02T18:52:43.318 DEBUG:boto:Token: None 2023-11-02T18:52:43.319 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j2FJ9zraFYCh5neIWzp/q69t9wM= 2023-11-02T18:52:43.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j2FJ9zraFYCh5neIWzp/q69t9wM=', 'Content-Length': '0'} 2023-11-02T18:52:43.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019523dd0e7682d09-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.366 DEBUG:boto:path=/ibtljn-61/obj57 2023-11-02T18:52:43.367 DEBUG:boto:auth_path=/ibtljn-61/obj57 2023-11-02T18:52:43.367 DEBUG:boto:path=/ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.368 DEBUG:boto:auth_path=/ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.368 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.369 DEBUG:boto:Path: /ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.369 DEBUG:boto:Data: 2023-11-02T18:52:43.370 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.370 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.371 DEBUG:boto:Params: {} 2023-11-02T18:52:43.371 DEBUG:boto:Token: None 2023-11-02T18:52:43.372 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.373 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f6NRZP58gCubCuf7GAdaSRsq78I= 2023-11-02T18:52:43.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f6NRZP58gCubCuf7GAdaSRsq78I=', 'Content-Length': '0'} 2023-11-02T18:52:43.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb8a9d25a47eac8b-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.377 DEBUG:boto:path=/ibtljn-61/obj57 2023-11-02T18:52:43.377 DEBUG:boto:auth_path=/ibtljn-61/obj57 2023-11-02T18:52:43.378 DEBUG:boto:path=/ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.378 DEBUG:boto:auth_path=/ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.378 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.379 DEBUG:boto:Path: /ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.379 DEBUG:boto:Data: 2023-11-02T18:52:43.380 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.380 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.381 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.381 DEBUG:boto:Params: {} 2023-11-02T18:52:43.382 DEBUG:boto:Token: None 2023-11-02T18:52:43.382 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.383 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f6NRZP58gCubCuf7GAdaSRsq78I= 2023-11-02T18:52:43.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f6NRZP58gCubCuf7GAdaSRsq78I=', 'Content-Length': '0'} 2023-11-02T18:52:43.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001610218921d324e1-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:52:43.387 DEBUG:boto:path=/ibtljn-61/obj57 2023-11-02T18:52:43.388 DEBUG:boto:auth_path=/ibtljn-61/obj57 2023-11-02T18:52:43.388 DEBUG:boto:path=/ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.389 DEBUG:boto:auth_path=/ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.389 DEBUG:boto:Method: GET 2023-11-02T18:52:43.390 DEBUG:boto:Path: /ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.390 DEBUG:boto:Data: 2023-11-02T18:52:43.391 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.391 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.392 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.392 DEBUG:boto:Params: {} 2023-11-02T18:52:43.393 DEBUG:boto:Token: None 2023-11-02T18:52:43.393 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.394 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j2FJ9zraFYCh5neIWzp/q69t9wM= 2023-11-02T18:52:43.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j2FJ9zraFYCh5neIWzp/q69t9wM=', 'Content-Length': '0'} 2023-11-02T18:52:43.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e575de81a4f2536f-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.442 DEBUG:boto:path=/ibtljn-61/obj57 2023-11-02T18:52:43.443 DEBUG:boto:auth_path=/ibtljn-61/obj57 2023-11-02T18:52:43.443 DEBUG:boto:path=/ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.444 DEBUG:boto:auth_path=/ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.444 DEBUG:boto:Method: GET 2023-11-02T18:52:43.445 DEBUG:boto:Path: /ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.445 DEBUG:boto:Data: 2023-11-02T18:52:43.446 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.446 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.447 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.447 DEBUG:boto:Params: {} 2023-11-02T18:52:43.448 DEBUG:boto:Token: None 2023-11-02T18:52:43.448 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj57?versionId=null 2023-11-02T18:52:43.449 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j2FJ9zraFYCh5neIWzp/q69t9wM= 2023-11-02T18:52:43.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j2FJ9zraFYCh5neIWzp/q69t9wM=', 'Content-Length': '0'} 2023-11-02T18:52:43.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000479c9e2bd8374b1f-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.453 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.453 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.454 DEBUG:boto:path=/ibtljn-61/?prefix=obj57 2023-11-02T18:52:43.454 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj57 2023-11-02T18:52:43.455 DEBUG:boto:Method: GET 2023-11-02T18:52:43.455 DEBUG:boto:Path: /ibtljn-61/?prefix=obj57 2023-11-02T18:52:43.456 DEBUG:boto:Data: 2023-11-02T18:52:43.456 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.456 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.457 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.457 DEBUG:boto:Params: {} 2023-11-02T18:52:43.458 DEBUG:boto:Token: None 2023-11-02T18:52:43.458 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.459 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069d09c4e564acd53-006543effb-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.466 DEBUG:boto:b'ibtljn-61obj571000falseobj572023-11-02T18:50:23.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.466 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.467 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.467 DEBUG:boto:path=/ibtljn-61/?prefix=obj57 2023-11-02T18:52:43.468 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj57 2023-11-02T18:52:43.468 DEBUG:boto:Method: GET 2023-11-02T18:52:43.469 DEBUG:boto:Path: /ibtljn-61/?prefix=obj57 2023-11-02T18:52:43.469 DEBUG:boto:Data: 2023-11-02T18:52:43.470 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.470 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.471 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.471 DEBUG:boto:Params: {} 2023-11-02T18:52:43.472 DEBUG:boto:Token: None 2023-11-02T18:52:43.473 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.473 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000903ed486d8ece386-006543effb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.485 DEBUG:boto:b'ibtljn-61obj571000falseobj572023-11-02T18:50:23.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.486 DEBUG:boto:path=/ibtljn-61/obj57 2023-11-02T18:52:43.487 DEBUG:boto:auth_path=/ibtljn-61/obj57 2023-11-02T18:52:43.487 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.487 DEBUG:boto:Path: /ibtljn-61/obj57 2023-11-02T18:52:43.488 DEBUG:boto:Data: 2023-11-02T18:52:43.488 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.489 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.489 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.490 DEBUG:boto:Params: {} 2023-11-02T18:52:43.490 DEBUG:boto:Token: None 2023-11-02T18:52:43.490 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj57 2023-11-02T18:52:43.491 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CVuXOmBOJebOIQ5hkiXkmolO0gw= 2023-11-02T18:52:43.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CVuXOmBOJebOIQ5hkiXkmolO0gw=', 'Content-Length': '0'} 2023-11-02T18:52:43.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5940c61bf36338-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.494 DEBUG:boto:path=/ibtljn-61/obj57 2023-11-02T18:52:43.495 DEBUG:boto:auth_path=/ibtljn-61/obj57 2023-11-02T18:52:43.495 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.496 DEBUG:boto:Path: /ibtljn-61/obj57 2023-11-02T18:52:43.496 DEBUG:boto:Data: 2023-11-02T18:52:43.497 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.497 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.497 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.498 DEBUG:boto:Params: {} 2023-11-02T18:52:43.498 DEBUG:boto:Token: None 2023-11-02T18:52:43.499 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj57 2023-11-02T18:52:43.499 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CVuXOmBOJebOIQ5hkiXkmolO0gw= 2023-11-02T18:52:43.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CVuXOmBOJebOIQ5hkiXkmolO0gw=', 'Content-Length': '0'} 2023-11-02T18:52:43.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a05b45bb0f2fc00-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T18:52:43.504 DEBUG:boto:path=/ibtljn-61/obj57 2023-11-02T18:52:43.504 DEBUG:boto:auth_path=/ibtljn-61/obj57 2023-11-02T18:52:43.504 DEBUG:boto:Method: GET 2023-11-02T18:52:43.505 DEBUG:boto:Path: /ibtljn-61/obj57 2023-11-02T18:52:43.505 DEBUG:boto:Data: 2023-11-02T18:52:43.506 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.506 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.507 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.507 DEBUG:boto:Params: {} 2023-11-02T18:52:43.508 DEBUG:boto:Token: None 2023-11-02T18:52:43.508 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj57 2023-11-02T18:52:43.509 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L/8Tp5shiqpLRsj5TYGnmlSavOU= 2023-11-02T18:52:43.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L/8Tp5shiqpLRsj5TYGnmlSavOU=', 'Content-Length': '0'} 2023-11-02T18:52:43.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c91aec5eebe89b-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.513 DEBUG:boto:path=/ibtljn-61/obj57 2023-11-02T18:52:43.513 DEBUG:boto:auth_path=/ibtljn-61/obj57 2023-11-02T18:52:43.514 DEBUG:boto:Method: GET 2023-11-02T18:52:43.514 DEBUG:boto:Path: /ibtljn-61/obj57 2023-11-02T18:52:43.515 DEBUG:boto:Data: 2023-11-02T18:52:43.515 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.516 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.516 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.517 DEBUG:boto:Params: {} 2023-11-02T18:52:43.517 DEBUG:boto:Token: None 2023-11-02T18:52:43.518 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj57 2023-11-02T18:52:43.518 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:L/8Tp5shiqpLRsj5TYGnmlSavOU= 2023-11-02T18:52:43.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L/8Tp5shiqpLRsj5TYGnmlSavOU=', 'Content-Length': '0'} 2023-11-02T18:52:43.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de31b016a77c200d-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.566 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.567 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.567 DEBUG:boto:path=/ibtljn-61/?prefix=obj57 2023-11-02T18:52:43.568 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj57 2023-11-02T18:52:43.568 DEBUG:boto:Method: GET 2023-11-02T18:52:43.569 DEBUG:boto:Path: /ibtljn-61/?prefix=obj57 2023-11-02T18:52:43.569 DEBUG:boto:Data: 2023-11-02T18:52:43.570 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.570 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.571 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.571 DEBUG:boto:Params: {} 2023-11-02T18:52:43.572 DEBUG:boto:Token: None 2023-11-02T18:52:43.572 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.573 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca9acd8c47f16ec6-006543effb-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.579 DEBUG:boto:b'ibtljn-61obj571000falseobj572023-11-02T18:50:23.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.580 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.581 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.581 DEBUG:boto:path=/ibtljn-61/?prefix=obj57 2023-11-02T18:52:43.581 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj57 2023-11-02T18:52:43.582 DEBUG:boto:Method: GET 2023-11-02T18:52:43.582 DEBUG:boto:Path: /ibtljn-61/?prefix=obj57 2023-11-02T18:52:43.583 DEBUG:boto:Data: 2023-11-02T18:52:43.583 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.584 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.584 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.584 DEBUG:boto:Params: {} 2023-11-02T18:52:43.585 DEBUG:boto:Token: None 2023-11-02T18:52:43.585 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.586 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007baae76033cf0a7d-006543effb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.596 DEBUG:boto:b'ibtljn-61obj571000falseobj572023-11-02T18:50:23.017Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:52:43.597 DEBUG:boto:path=/ibtljn-61/obj58 2023-11-02T18:52:43.598 DEBUG:boto:auth_path=/ibtljn-61/obj58 2023-11-02T18:52:43.598 DEBUG:boto:path=/ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.599 DEBUG:boto:auth_path=/ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.599 DEBUG:boto:Method: GET 2023-11-02T18:52:43.599 DEBUG:boto:Path: /ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.600 DEBUG:boto:Data: 2023-11-02T18:52:43.600 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.601 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.601 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.602 DEBUG:boto:Params: {} 2023-11-02T18:52:43.602 DEBUG:boto:Token: None 2023-11-02T18:52:43.603 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.603 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z78umW7UhO5Hp7hFwEY0w6JRpAM= 2023-11-02T18:52:43.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z78umW7UhO5Hp7hFwEY0w6JRpAM=', 'Content-Length': '0'} 2023-11-02T18:52:43.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000191545182281ddf6-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.650 DEBUG:boto:path=/ibtljn-61/obj58 2023-11-02T18:52:43.651 DEBUG:boto:auth_path=/ibtljn-61/obj58 2023-11-02T18:52:43.651 DEBUG:boto:path=/ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.652 DEBUG:boto:auth_path=/ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.652 DEBUG:boto:Method: GET 2023-11-02T18:52:43.652 DEBUG:boto:Path: /ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.653 DEBUG:boto:Data: 2023-11-02T18:52:43.653 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.654 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.655 DEBUG:boto:Params: {} 2023-11-02T18:52:43.655 DEBUG:boto:Token: None 2023-11-02T18:52:43.656 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z78umW7UhO5Hp7hFwEY0w6JRpAM= 2023-11-02T18:52:43.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z78umW7UhO5Hp7hFwEY0w6JRpAM=', 'Content-Length': '0'} 2023-11-02T18:52:43.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f708e1cc7420ba24-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.660 DEBUG:boto:path=/ibtljn-61/obj58 2023-11-02T18:52:43.660 DEBUG:boto:auth_path=/ibtljn-61/obj58 2023-11-02T18:52:43.661 DEBUG:boto:path=/ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.661 DEBUG:boto:auth_path=/ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.661 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.662 DEBUG:boto:Path: /ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.662 DEBUG:boto:Data: 2023-11-02T18:52:43.663 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.664 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.664 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.665 DEBUG:boto:Params: {} 2023-11-02T18:52:43.665 DEBUG:boto:Token: None 2023-11-02T18:52:43.666 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.667 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hBuz3Ty37zNLADZfXvZS5H2slr4= 2023-11-02T18:52:43.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hBuz3Ty37zNLADZfXvZS5H2slr4=', 'Content-Length': '0'} 2023-11-02T18:52:43.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e510fb41677ea0e2-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.671 DEBUG:boto:path=/ibtljn-61/obj58 2023-11-02T18:52:43.671 DEBUG:boto:auth_path=/ibtljn-61/obj58 2023-11-02T18:52:43.672 DEBUG:boto:path=/ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.672 DEBUG:boto:auth_path=/ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.672 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.673 DEBUG:boto:Path: /ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.673 DEBUG:boto:Data: 2023-11-02T18:52:43.674 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.675 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.675 DEBUG:boto:Params: {} 2023-11-02T18:52:43.675 DEBUG:boto:Token: None 2023-11-02T18:52:43.676 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.676 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hBuz3Ty37zNLADZfXvZS5H2slr4= 2023-11-02T18:52:43.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hBuz3Ty37zNLADZfXvZS5H2slr4=', 'Content-Length': '0'} 2023-11-02T18:52:43.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0ae912466bf6b10-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:52:43.680 DEBUG:boto:path=/ibtljn-61/obj58 2023-11-02T18:52:43.681 DEBUG:boto:auth_path=/ibtljn-61/obj58 2023-11-02T18:52:43.681 DEBUG:boto:path=/ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.682 DEBUG:boto:auth_path=/ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.682 DEBUG:boto:Method: GET 2023-11-02T18:52:43.683 DEBUG:boto:Path: /ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.683 DEBUG:boto:Data: 2023-11-02T18:52:43.684 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.684 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.685 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.685 DEBUG:boto:Params: {} 2023-11-02T18:52:43.686 DEBUG:boto:Token: None 2023-11-02T18:52:43.686 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.687 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z78umW7UhO5Hp7hFwEY0w6JRpAM= 2023-11-02T18:52:43.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z78umW7UhO5Hp7hFwEY0w6JRpAM=', 'Content-Length': '0'} 2023-11-02T18:52:43.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6fbb200b0dbc6c4-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.691 DEBUG:boto:path=/ibtljn-61/obj58 2023-11-02T18:52:43.691 DEBUG:boto:auth_path=/ibtljn-61/obj58 2023-11-02T18:52:43.692 DEBUG:boto:path=/ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.692 DEBUG:boto:auth_path=/ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.692 DEBUG:boto:Method: GET 2023-11-02T18:52:43.693 DEBUG:boto:Path: /ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.693 DEBUG:boto:Data: 2023-11-02T18:52:43.694 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.694 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.695 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.695 DEBUG:boto:Params: {} 2023-11-02T18:52:43.696 DEBUG:boto:Token: None 2023-11-02T18:52:43.696 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj58?versionId=null 2023-11-02T18:52:43.697 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z78umW7UhO5Hp7hFwEY0w6JRpAM= 2023-11-02T18:52:43.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z78umW7UhO5Hp7hFwEY0w6JRpAM=', 'Content-Length': '0'} 2023-11-02T18:52:43.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e38e4fb79a09c80-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.742 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.743 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.744 DEBUG:boto:path=/ibtljn-61/?prefix=obj58 2023-11-02T18:52:43.744 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj58 2023-11-02T18:52:43.744 DEBUG:boto:Method: GET 2023-11-02T18:52:43.745 DEBUG:boto:Path: /ibtljn-61/?prefix=obj58 2023-11-02T18:52:43.745 DEBUG:boto:Data: 2023-11-02T18:52:43.746 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.746 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.747 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.747 DEBUG:boto:Params: {} 2023-11-02T18:52:43.748 DEBUG:boto:Token: None 2023-11-02T18:52:43.748 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.749 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007be2a0f647e27212-006543effb-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.756 DEBUG:boto:b'ibtljn-61obj581000falseobj582023-11-02T18:50:23.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.757 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.757 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.757 DEBUG:boto:path=/ibtljn-61/?prefix=obj58 2023-11-02T18:52:43.758 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj58 2023-11-02T18:52:43.758 DEBUG:boto:Method: GET 2023-11-02T18:52:43.759 DEBUG:boto:Path: /ibtljn-61/?prefix=obj58 2023-11-02T18:52:43.759 DEBUG:boto:Data: 2023-11-02T18:52:43.760 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.760 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.761 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.761 DEBUG:boto:Params: {} 2023-11-02T18:52:43.762 DEBUG:boto:Token: None 2023-11-02T18:52:43.762 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.763 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000042d25d1cf6768fd-006543effb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.774 DEBUG:boto:b'ibtljn-61obj581000falseobj582023-11-02T18:50:23.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.775 DEBUG:boto:path=/ibtljn-61/obj58 2023-11-02T18:52:43.775 DEBUG:boto:auth_path=/ibtljn-61/obj58 2023-11-02T18:52:43.775 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.776 DEBUG:boto:Path: /ibtljn-61/obj58 2023-11-02T18:52:43.776 DEBUG:boto:Data: 2023-11-02T18:52:43.777 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.777 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.777 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.778 DEBUG:boto:Params: {} 2023-11-02T18:52:43.778 DEBUG:boto:Token: None 2023-11-02T18:52:43.780 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj58 2023-11-02T18:52:43.781 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d2+e11Dmr5LKfezIBPBTFd53wYM= 2023-11-02T18:52:43.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d2+e11Dmr5LKfezIBPBTFd53wYM=', 'Content-Length': '0'} 2023-11-02T18:52:43.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c51ff535d5ba9ca6-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.785 DEBUG:boto:path=/ibtljn-61/obj58 2023-11-02T18:52:43.785 DEBUG:boto:auth_path=/ibtljn-61/obj58 2023-11-02T18:52:43.785 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.786 DEBUG:boto:Path: /ibtljn-61/obj58 2023-11-02T18:52:43.786 DEBUG:boto:Data: 2023-11-02T18:52:43.787 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.787 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.788 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.788 DEBUG:boto:Params: {} 2023-11-02T18:52:43.788 DEBUG:boto:Token: None 2023-11-02T18:52:43.789 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj58 2023-11-02T18:52:43.789 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d2+e11Dmr5LKfezIBPBTFd53wYM= 2023-11-02T18:52:43.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d2+e11Dmr5LKfezIBPBTFd53wYM=', 'Content-Length': '0'} 2023-11-02T18:52:43.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8d855e6a281db29-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T18:52:43.794 DEBUG:boto:path=/ibtljn-61/obj58 2023-11-02T18:52:43.794 DEBUG:boto:auth_path=/ibtljn-61/obj58 2023-11-02T18:52:43.795 DEBUG:boto:Method: GET 2023-11-02T18:52:43.795 DEBUG:boto:Path: /ibtljn-61/obj58 2023-11-02T18:52:43.796 DEBUG:boto:Data: 2023-11-02T18:52:43.796 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.796 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.797 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.797 DEBUG:boto:Params: {} 2023-11-02T18:52:43.798 DEBUG:boto:Token: None 2023-11-02T18:52:43.799 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj58 2023-11-02T18:52:43.799 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s855s4MdYDaDgiT7w/rWYtdl+50= 2023-11-02T18:52:43.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s855s4MdYDaDgiT7w/rWYtdl+50=', 'Content-Length': '0'} 2023-11-02T18:52:43.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e33257f16f9179d7-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.846 DEBUG:boto:path=/ibtljn-61/obj58 2023-11-02T18:52:43.847 DEBUG:boto:auth_path=/ibtljn-61/obj58 2023-11-02T18:52:43.847 DEBUG:boto:Method: GET 2023-11-02T18:52:43.848 DEBUG:boto:Path: /ibtljn-61/obj58 2023-11-02T18:52:43.848 DEBUG:boto:Data: 2023-11-02T18:52:43.849 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.849 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.850 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.851 DEBUG:boto:Params: {} 2023-11-02T18:52:43.851 DEBUG:boto:Token: None 2023-11-02T18:52:43.852 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj58 2023-11-02T18:52:43.853 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s855s4MdYDaDgiT7w/rWYtdl+50= 2023-11-02T18:52:43.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s855s4MdYDaDgiT7w/rWYtdl+50=', 'Content-Length': '0'} 2023-11-02T18:52:43.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001882161326e6b6ed-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.857 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.858 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.858 DEBUG:boto:path=/ibtljn-61/?prefix=obj58 2023-11-02T18:52:43.859 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj58 2023-11-02T18:52:43.859 DEBUG:boto:Method: GET 2023-11-02T18:52:43.860 DEBUG:boto:Path: /ibtljn-61/?prefix=obj58 2023-11-02T18:52:43.860 DEBUG:boto:Data: 2023-11-02T18:52:43.861 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.861 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.862 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.862 DEBUG:boto:Params: {} 2023-11-02T18:52:43.863 DEBUG:boto:Token: None 2023-11-02T18:52:43.863 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.864 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa4a8cd3df9fed25-006543effb-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.871 DEBUG:boto:b'ibtljn-61obj581000falseobj582023-11-02T18:50:23.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.872 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:43.872 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:43.873 DEBUG:boto:path=/ibtljn-61/?prefix=obj58 2023-11-02T18:52:43.874 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj58 2023-11-02T18:52:43.874 DEBUG:boto:Method: GET 2023-11-02T18:52:43.875 DEBUG:boto:Path: /ibtljn-61/?prefix=obj58 2023-11-02T18:52:43.875 DEBUG:boto:Data: 2023-11-02T18:52:43.876 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.876 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.877 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.877 DEBUG:boto:Params: {} 2023-11-02T18:52:43.878 DEBUG:boto:Token: None 2023-11-02T18:52:43.878 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/ 2023-11-02T18:52:43.879 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc= 2023-11-02T18:52:43.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cYsNvM9wCWMbBNLAnn88q94HJEc=', 'Content-Length': '0'} 2023-11-02T18:52:43.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002139d380b1f28a05-006543effb-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.930 DEBUG:boto:b'ibtljn-61obj581000falseobj582023-11-02T18:50:23.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:43.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T18:52:43.932 DEBUG:boto:path=/ibtljn-61/obj62 2023-11-02T18:52:43.932 DEBUG:boto:auth_path=/ibtljn-61/obj62 2023-11-02T18:52:43.933 DEBUG:boto:path=/ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.933 DEBUG:boto:auth_path=/ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.934 DEBUG:boto:Method: GET 2023-11-02T18:52:43.934 DEBUG:boto:Path: /ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.934 DEBUG:boto:Data: 2023-11-02T18:52:43.935 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.935 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.936 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.936 DEBUG:boto:Params: {} 2023-11-02T18:52:43.937 DEBUG:boto:Token: None 2023-11-02T18:52:43.937 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.938 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bNHGjokF8UWgWo1XNYEyA+R2eck= 2023-11-02T18:52:43.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bNHGjokF8UWgWo1XNYEyA+R2eck=', 'Content-Length': '0'} 2023-11-02T18:52:43.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e930ebf37247f6-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.942 DEBUG:boto:path=/ibtljn-61/obj62 2023-11-02T18:52:43.943 DEBUG:boto:auth_path=/ibtljn-61/obj62 2023-11-02T18:52:43.943 DEBUG:boto:path=/ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.944 DEBUG:boto:auth_path=/ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.944 DEBUG:boto:Method: GET 2023-11-02T18:52:43.944 DEBUG:boto:Path: /ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.945 DEBUG:boto:Data: 2023-11-02T18:52:43.945 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.946 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.946 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.947 DEBUG:boto:Params: {} 2023-11-02T18:52:43.947 DEBUG:boto:Token: None 2023-11-02T18:52:43.948 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.948 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bNHGjokF8UWgWo1XNYEyA+R2eck= 2023-11-02T18:52:43.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bNHGjokF8UWgWo1XNYEyA+R2eck=', 'Content-Length': '0'} 2023-11-02T18:52:43.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009887c0d18eece2f-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.953 DEBUG:boto:path=/ibtljn-61/obj62 2023-11-02T18:52:43.953 DEBUG:boto:auth_path=/ibtljn-61/obj62 2023-11-02T18:52:43.954 DEBUG:boto:path=/ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.954 DEBUG:boto:auth_path=/ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.955 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.955 DEBUG:boto:Path: /ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.955 DEBUG:boto:Data: 2023-11-02T18:52:43.956 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.956 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.957 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.957 DEBUG:boto:Params: {} 2023-11-02T18:52:43.957 DEBUG:boto:Token: None 2023-11-02T18:52:43.958 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.958 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6colEZqgTd7ZG1awkih5gKJrCKw= 2023-11-02T18:52:43.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6colEZqgTd7ZG1awkih5gKJrCKw=', 'Content-Length': '0'} 2023-11-02T18:52:43.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac59b4cfb9773074-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.963 DEBUG:boto:path=/ibtljn-61/obj62 2023-11-02T18:52:43.963 DEBUG:boto:auth_path=/ibtljn-61/obj62 2023-11-02T18:52:43.964 DEBUG:boto:path=/ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.964 DEBUG:boto:auth_path=/ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.965 DEBUG:boto:Method: HEAD 2023-11-02T18:52:43.965 DEBUG:boto:Path: /ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.965 DEBUG:boto:Data: 2023-11-02T18:52:43.966 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.966 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:43.967 DEBUG:boto:Port: 8000 2023-11-02T18:52:43.967 DEBUG:boto:Params: {} 2023-11-02T18:52:43.967 DEBUG:boto:Token: None 2023-11-02T18:52:43.968 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.968 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6colEZqgTd7ZG1awkih5gKJrCKw= 2023-11-02T18:52:43.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6colEZqgTd7ZG1awkih5gKJrCKw=', 'Content-Length': '0'} 2023-11-02T18:52:43.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfec3f0d1e919894-006543effb-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T18:52:43.975 DEBUG:boto:path=/ibtljn-61/obj62 2023-11-02T18:52:43.976 DEBUG:boto:auth_path=/ibtljn-61/obj62 2023-11-02T18:52:43.977 DEBUG:boto:path=/ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.978 DEBUG:boto:auth_path=/ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.979 DEBUG:boto:Method: GET 2023-11-02T18:52:43.980 DEBUG:boto:Path: /ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.981 DEBUG:boto:Data: 2023-11-02T18:52:43.982 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.983 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:43.984 DEBUG:boto:Port: 8001 2023-11-02T18:52:43.984 DEBUG:boto:Params: {} 2023-11-02T18:52:43.985 DEBUG:boto:Token: None 2023-11-02T18:52:43.986 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:43 GMT /ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.987 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bNHGjokF8UWgWo1XNYEyA+R2eck= 2023-11-02T18:52:43.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bNHGjokF8UWgWo1XNYEyA+R2eck=', 'Content-Length': '0'} 2023-11-02T18:52:43.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007de69ea967de701-006543effb-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:43.993 DEBUG:boto:path=/ibtljn-61/obj62 2023-11-02T18:52:43.994 DEBUG:boto:auth_path=/ibtljn-61/obj62 2023-11-02T18:52:43.995 DEBUG:boto:path=/ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.996 DEBUG:boto:auth_path=/ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.997 DEBUG:boto:Method: GET 2023-11-02T18:52:43.997 DEBUG:boto:Path: /ibtljn-61/obj62?versionId=null 2023-11-02T18:52:43.998 DEBUG:boto:Data: 2023-11-02T18:52:43.999 DEBUG:boto:Headers: {} 2023-11-02T18:52:43.999 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.000 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.001 DEBUG:boto:Params: {} 2023-11-02T18:52:44.002 DEBUG:boto:Token: None 2023-11-02T18:52:44.003 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj62?versionId=null 2023-11-02T18:52:44.004 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FVGz1xTcKb4lOrN5E1HbSu0BSwQ= 2023-11-02T18:52:44.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FVGz1xTcKb4lOrN5E1HbSu0BSwQ=', 'Content-Length': '0'} 2023-11-02T18:52:44.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c7da48c47aebbe-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.010 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.011 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.011 DEBUG:boto:path=/ibtljn-61/?prefix=obj62 2023-11-02T18:52:44.011 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj62 2023-11-02T18:52:44.012 DEBUG:boto:Method: GET 2023-11-02T18:52:44.012 DEBUG:boto:Path: /ibtljn-61/?prefix=obj62 2023-11-02T18:52:44.013 DEBUG:boto:Data: 2023-11-02T18:52:44.013 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.014 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.014 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.014 DEBUG:boto:Params: {} 2023-11-02T18:52:44.015 DEBUG:boto:Token: None 2023-11-02T18:52:44.016 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ee96987c07e499-006543effc-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.066 DEBUG:boto:b'ibtljn-61obj621000falseobj622023-11-02T18:50:24.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.067 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.068 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.068 DEBUG:boto:path=/ibtljn-61/?prefix=obj62 2023-11-02T18:52:44.069 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj62 2023-11-02T18:52:44.069 DEBUG:boto:Method: GET 2023-11-02T18:52:44.069 DEBUG:boto:Path: /ibtljn-61/?prefix=obj62 2023-11-02T18:52:44.070 DEBUG:boto:Data: 2023-11-02T18:52:44.070 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.071 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.071 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.072 DEBUG:boto:Params: {} 2023-11-02T18:52:44.073 DEBUG:boto:Token: None 2023-11-02T18:52:44.073 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.074 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083df7610f80526b9-006543effc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.085 DEBUG:boto:b'ibtljn-61obj621000falseobj622023-11-02T18:50:24.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.086 DEBUG:boto:path=/ibtljn-61/obj62 2023-11-02T18:52:44.087 DEBUG:boto:auth_path=/ibtljn-61/obj62 2023-11-02T18:52:44.087 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.088 DEBUG:boto:Path: /ibtljn-61/obj62 2023-11-02T18:52:44.088 DEBUG:boto:Data: 2023-11-02T18:52:44.088 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.089 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.089 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.090 DEBUG:boto:Params: {} 2023-11-02T18:52:44.090 DEBUG:boto:Token: None 2023-11-02T18:52:44.091 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj62 2023-11-02T18:52:44.091 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7bggh0phvbxmO1gzZJ36GbRa5Uw= 2023-11-02T18:52:44.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7bggh0phvbxmO1gzZJ36GbRa5Uw=', 'Content-Length': '0'} 2023-11-02T18:52:44.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000146e51705b9072a0-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.095 DEBUG:boto:path=/ibtljn-61/obj62 2023-11-02T18:52:44.095 DEBUG:boto:auth_path=/ibtljn-61/obj62 2023-11-02T18:52:44.096 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.096 DEBUG:boto:Path: /ibtljn-61/obj62 2023-11-02T18:52:44.096 DEBUG:boto:Data: 2023-11-02T18:52:44.097 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.097 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.098 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.098 DEBUG:boto:Params: {} 2023-11-02T18:52:44.099 DEBUG:boto:Token: None 2023-11-02T18:52:44.099 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj62 2023-11-02T18:52:44.100 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7bggh0phvbxmO1gzZJ36GbRa5Uw= 2023-11-02T18:52:44.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7bggh0phvbxmO1gzZJ36GbRa5Uw=', 'Content-Length': '0'} 2023-11-02T18:52:44.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a0f589482c54d75-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T18:52:44.103 DEBUG:boto:path=/ibtljn-61/obj62 2023-11-02T18:52:44.104 DEBUG:boto:auth_path=/ibtljn-61/obj62 2023-11-02T18:52:44.104 DEBUG:boto:Method: GET 2023-11-02T18:52:44.105 DEBUG:boto:Path: /ibtljn-61/obj62 2023-11-02T18:52:44.105 DEBUG:boto:Data: 2023-11-02T18:52:44.105 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.106 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.106 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.107 DEBUG:boto:Params: {} 2023-11-02T18:52:44.107 DEBUG:boto:Token: None 2023-11-02T18:52:44.108 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj62 2023-11-02T18:52:44.108 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BLF4GfC88o5m7GJKdqdYqvvlgrY= 2023-11-02T18:52:44.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BLF4GfC88o5m7GJKdqdYqvvlgrY=', 'Content-Length': '0'} 2023-11-02T18:52:44.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057483f53334b68c3-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.113 DEBUG:boto:path=/ibtljn-61/obj62 2023-11-02T18:52:44.113 DEBUG:boto:auth_path=/ibtljn-61/obj62 2023-11-02T18:52:44.113 DEBUG:boto:Method: GET 2023-11-02T18:52:44.114 DEBUG:boto:Path: /ibtljn-61/obj62 2023-11-02T18:52:44.114 DEBUG:boto:Data: 2023-11-02T18:52:44.115 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.115 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.116 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.116 DEBUG:boto:Params: {} 2023-11-02T18:52:44.117 DEBUG:boto:Token: None 2023-11-02T18:52:44.117 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj62 2023-11-02T18:52:44.118 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BLF4GfC88o5m7GJKdqdYqvvlgrY= 2023-11-02T18:52:44.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BLF4GfC88o5m7GJKdqdYqvvlgrY=', 'Content-Length': '0'} 2023-11-02T18:52:44.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a826cba677ba45-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.162 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.163 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.164 DEBUG:boto:path=/ibtljn-61/?prefix=obj62 2023-11-02T18:52:44.164 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj62 2023-11-02T18:52:44.164 DEBUG:boto:Method: GET 2023-11-02T18:52:44.165 DEBUG:boto:Path: /ibtljn-61/?prefix=obj62 2023-11-02T18:52:44.165 DEBUG:boto:Data: 2023-11-02T18:52:44.166 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.166 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.167 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.167 DEBUG:boto:Params: {} 2023-11-02T18:52:44.168 DEBUG:boto:Token: None 2023-11-02T18:52:44.168 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.169 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab8a07ddb7ef5bab-006543effc-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.175 DEBUG:boto:b'ibtljn-61obj621000falseobj622023-11-02T18:50:24.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.176 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.176 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.177 DEBUG:boto:path=/ibtljn-61/?prefix=obj62 2023-11-02T18:52:44.177 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj62 2023-11-02T18:52:44.177 DEBUG:boto:Method: GET 2023-11-02T18:52:44.178 DEBUG:boto:Path: /ibtljn-61/?prefix=obj62 2023-11-02T18:52:44.178 DEBUG:boto:Data: 2023-11-02T18:52:44.179 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.180 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.180 DEBUG:boto:Params: {} 2023-11-02T18:52:44.181 DEBUG:boto:Token: None 2023-11-02T18:52:44.181 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.182 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d26ddfa50d3b976-006543effc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.192 DEBUG:boto:b'ibtljn-61obj621000falseobj622023-11-02T18:50:24.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:52:44.193 DEBUG:boto:path=/ibtljn-61/obj68 2023-11-02T18:52:44.194 DEBUG:boto:auth_path=/ibtljn-61/obj68 2023-11-02T18:52:44.194 DEBUG:boto:path=/ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.194 DEBUG:boto:auth_path=/ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.195 DEBUG:boto:Method: GET 2023-11-02T18:52:44.195 DEBUG:boto:Path: /ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.196 DEBUG:boto:Data: 2023-11-02T18:52:44.196 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.197 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.197 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.198 DEBUG:boto:Params: {} 2023-11-02T18:52:44.198 DEBUG:boto:Token: None 2023-11-02T18:52:44.200 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.201 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:05BkFEBEixrEUNySXu+TovvptS8= 2023-11-02T18:52:44.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:05BkFEBEixrEUNySXu+TovvptS8=', 'Content-Length': '0'} 2023-11-02T18:52:44.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b5853451a2606e2-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.206 DEBUG:boto:path=/ibtljn-61/obj68 2023-11-02T18:52:44.206 DEBUG:boto:auth_path=/ibtljn-61/obj68 2023-11-02T18:52:44.207 DEBUG:boto:path=/ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.208 DEBUG:boto:auth_path=/ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.208 DEBUG:boto:Method: GET 2023-11-02T18:52:44.209 DEBUG:boto:Path: /ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.209 DEBUG:boto:Data: 2023-11-02T18:52:44.210 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.210 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.211 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.212 DEBUG:boto:Params: {} 2023-11-02T18:52:44.212 DEBUG:boto:Token: None 2023-11-02T18:52:44.213 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.214 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:05BkFEBEixrEUNySXu+TovvptS8= 2023-11-02T18:52:44.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:05BkFEBEixrEUNySXu+TovvptS8=', 'Content-Length': '0'} 2023-11-02T18:52:44.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb1b06660a196c4e-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.219 DEBUG:boto:path=/ibtljn-61/obj68 2023-11-02T18:52:44.220 DEBUG:boto:auth_path=/ibtljn-61/obj68 2023-11-02T18:52:44.220 DEBUG:boto:path=/ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.220 DEBUG:boto:auth_path=/ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.221 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.221 DEBUG:boto:Path: /ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.222 DEBUG:boto:Data: 2023-11-02T18:52:44.222 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.223 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.223 DEBUG:boto:Params: {} 2023-11-02T18:52:44.224 DEBUG:boto:Token: None 2023-11-02T18:52:44.224 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N2Ay5K6eMu0MfpfYxntVNRXLPi8= 2023-11-02T18:52:44.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N2Ay5K6eMu0MfpfYxntVNRXLPi8=', 'Content-Length': '0'} 2023-11-02T18:52:44.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c43935caff13c771-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.229 DEBUG:boto:path=/ibtljn-61/obj68 2023-11-02T18:52:44.230 DEBUG:boto:auth_path=/ibtljn-61/obj68 2023-11-02T18:52:44.230 DEBUG:boto:path=/ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.231 DEBUG:boto:auth_path=/ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.231 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.232 DEBUG:boto:Path: /ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.232 DEBUG:boto:Data: 2023-11-02T18:52:44.232 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.233 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.233 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.234 DEBUG:boto:Params: {} 2023-11-02T18:52:44.234 DEBUG:boto:Token: None 2023-11-02T18:52:44.235 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.235 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N2Ay5K6eMu0MfpfYxntVNRXLPi8= 2023-11-02T18:52:44.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N2Ay5K6eMu0MfpfYxntVNRXLPi8=', 'Content-Length': '0'} 2023-11-02T18:52:44.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a64f9703cd4b294a-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:52:44.240 DEBUG:boto:path=/ibtljn-61/obj68 2023-11-02T18:52:44.240 DEBUG:boto:auth_path=/ibtljn-61/obj68 2023-11-02T18:52:44.241 DEBUG:boto:path=/ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.241 DEBUG:boto:auth_path=/ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.241 DEBUG:boto:Method: GET 2023-11-02T18:52:44.242 DEBUG:boto:Path: /ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.242 DEBUG:boto:Data: 2023-11-02T18:52:44.243 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.243 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.244 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.244 DEBUG:boto:Params: {} 2023-11-02T18:52:44.245 DEBUG:boto:Token: None 2023-11-02T18:52:44.247 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.247 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:05BkFEBEixrEUNySXu+TovvptS8= 2023-11-02T18:52:44.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:05BkFEBEixrEUNySXu+TovvptS8=', 'Content-Length': '0'} 2023-11-02T18:52:44.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000290f20eb9c38b3fc-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.252 DEBUG:boto:path=/ibtljn-61/obj68 2023-11-02T18:52:44.253 DEBUG:boto:auth_path=/ibtljn-61/obj68 2023-11-02T18:52:44.253 DEBUG:boto:path=/ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.254 DEBUG:boto:auth_path=/ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.254 DEBUG:boto:Method: GET 2023-11-02T18:52:44.255 DEBUG:boto:Path: /ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.256 DEBUG:boto:Data: 2023-11-02T18:52:44.256 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.257 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.258 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.259 DEBUG:boto:Params: {} 2023-11-02T18:52:44.259 DEBUG:boto:Token: None 2023-11-02T18:52:44.260 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj68?versionId=null 2023-11-02T18:52:44.261 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:05BkFEBEixrEUNySXu+TovvptS8= 2023-11-02T18:52:44.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:05BkFEBEixrEUNySXu+TovvptS8=', 'Content-Length': '0'} 2023-11-02T18:52:44.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c38b9a8ef67e6eb-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.310 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.311 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.311 DEBUG:boto:path=/ibtljn-61/?prefix=obj68 2023-11-02T18:52:44.312 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj68 2023-11-02T18:52:44.312 DEBUG:boto:Method: GET 2023-11-02T18:52:44.313 DEBUG:boto:Path: /ibtljn-61/?prefix=obj68 2023-11-02T18:52:44.314 DEBUG:boto:Data: 2023-11-02T18:52:44.314 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.315 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.315 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.316 DEBUG:boto:Params: {} 2023-11-02T18:52:44.317 DEBUG:boto:Token: None 2023-11-02T18:52:44.318 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b57a627215c3924a-006543effc-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.326 DEBUG:boto:b'ibtljn-61obj681000falseobj682023-11-02T18:50:22.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.327 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.328 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.329 DEBUG:boto:path=/ibtljn-61/?prefix=obj68 2023-11-02T18:52:44.330 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj68 2023-11-02T18:52:44.330 DEBUG:boto:Method: GET 2023-11-02T18:52:44.331 DEBUG:boto:Path: /ibtljn-61/?prefix=obj68 2023-11-02T18:52:44.332 DEBUG:boto:Data: 2023-11-02T18:52:44.333 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.333 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.334 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.335 DEBUG:boto:Params: {} 2023-11-02T18:52:44.337 DEBUG:boto:Token: None 2023-11-02T18:52:44.338 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.338 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095bd04e88228c9d7-006543effc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.349 DEBUG:boto:b'ibtljn-61obj681000falseobj682023-11-02T18:50:22.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.350 DEBUG:boto:path=/ibtljn-61/obj68 2023-11-02T18:52:44.351 DEBUG:boto:auth_path=/ibtljn-61/obj68 2023-11-02T18:52:44.352 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.352 DEBUG:boto:Path: /ibtljn-61/obj68 2023-11-02T18:52:44.353 DEBUG:boto:Data: 2023-11-02T18:52:44.354 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.355 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.355 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.356 DEBUG:boto:Params: {} 2023-11-02T18:52:44.357 DEBUG:boto:Token: None 2023-11-02T18:52:44.358 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj68 2023-11-02T18:52:44.359 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HfeS1UOe/e4+5dZt65cHHpcDrw8= 2023-11-02T18:52:44.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HfeS1UOe/e4+5dZt65cHHpcDrw8=', 'Content-Length': '0'} 2023-11-02T18:52:44.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095fb2b80e1dffbac-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.363 DEBUG:boto:path=/ibtljn-61/obj68 2023-11-02T18:52:44.363 DEBUG:boto:auth_path=/ibtljn-61/obj68 2023-11-02T18:52:44.364 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.365 DEBUG:boto:Path: /ibtljn-61/obj68 2023-11-02T18:52:44.365 DEBUG:boto:Data: 2023-11-02T18:52:44.366 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.367 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.367 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.368 DEBUG:boto:Params: {} 2023-11-02T18:52:44.369 DEBUG:boto:Token: None 2023-11-02T18:52:44.369 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj68 2023-11-02T18:52:44.370 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HfeS1UOe/e4+5dZt65cHHpcDrw8= 2023-11-02T18:52:44.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HfeS1UOe/e4+5dZt65cHHpcDrw8=', 'Content-Length': '0'} 2023-11-02T18:52:44.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ae20daef25aebff-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T18:52:44.374 DEBUG:boto:path=/ibtljn-61/obj68 2023-11-02T18:52:44.375 DEBUG:boto:auth_path=/ibtljn-61/obj68 2023-11-02T18:52:44.376 DEBUG:boto:Method: GET 2023-11-02T18:52:44.376 DEBUG:boto:Path: /ibtljn-61/obj68 2023-11-02T18:52:44.376 DEBUG:boto:Data: 2023-11-02T18:52:44.377 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.378 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.378 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.379 DEBUG:boto:Params: {} 2023-11-02T18:52:44.379 DEBUG:boto:Token: None 2023-11-02T18:52:44.380 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj68 2023-11-02T18:52:44.381 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1PAxUd8i2XQSrYcX7tF/t+emfPw= 2023-11-02T18:52:44.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1PAxUd8i2XQSrYcX7tF/t+emfPw=', 'Content-Length': '0'} 2023-11-02T18:52:44.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f37c075e59a348b4-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.430 DEBUG:boto:path=/ibtljn-61/obj68 2023-11-02T18:52:44.431 DEBUG:boto:auth_path=/ibtljn-61/obj68 2023-11-02T18:52:44.432 DEBUG:boto:Method: GET 2023-11-02T18:52:44.433 DEBUG:boto:Path: /ibtljn-61/obj68 2023-11-02T18:52:44.434 DEBUG:boto:Data: 2023-11-02T18:52:44.434 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.435 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.436 DEBUG:boto:Params: {} 2023-11-02T18:52:44.436 DEBUG:boto:Token: None 2023-11-02T18:52:44.437 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj68 2023-11-02T18:52:44.437 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1PAxUd8i2XQSrYcX7tF/t+emfPw= 2023-11-02T18:52:44.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1PAxUd8i2XQSrYcX7tF/t+emfPw=', 'Content-Length': '0'} 2023-11-02T18:52:44.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec8da92cdec2e8f-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.442 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.443 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.444 DEBUG:boto:path=/ibtljn-61/?prefix=obj68 2023-11-02T18:52:44.444 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj68 2023-11-02T18:52:44.445 DEBUG:boto:Method: GET 2023-11-02T18:52:44.446 DEBUG:boto:Path: /ibtljn-61/?prefix=obj68 2023-11-02T18:52:44.446 DEBUG:boto:Data: 2023-11-02T18:52:44.447 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.448 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.449 DEBUG:boto:Params: {} 2023-11-02T18:52:44.449 DEBUG:boto:Token: None 2023-11-02T18:52:44.450 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.451 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000911f7b1a0f1a43ea-006543effc-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.458 DEBUG:boto:b'ibtljn-61obj681000falseobj682023-11-02T18:50:22.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.458 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.459 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.460 DEBUG:boto:path=/ibtljn-61/?prefix=obj68 2023-11-02T18:52:44.460 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj68 2023-11-02T18:52:44.460 DEBUG:boto:Method: GET 2023-11-02T18:52:44.461 DEBUG:boto:Path: /ibtljn-61/?prefix=obj68 2023-11-02T18:52:44.461 DEBUG:boto:Data: 2023-11-02T18:52:44.462 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.463 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.464 DEBUG:boto:Params: {} 2023-11-02T18:52:44.465 DEBUG:boto:Token: None 2023-11-02T18:52:44.466 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.467 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000902a02312b461494-006543effc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.479 DEBUG:boto:b'ibtljn-61obj681000falseobj682023-11-02T18:50:22.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:52:44.481 DEBUG:boto:path=/ibtljn-61/obj69 2023-11-02T18:52:44.481 DEBUG:boto:auth_path=/ibtljn-61/obj69 2023-11-02T18:52:44.482 DEBUG:boto:path=/ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.482 DEBUG:boto:auth_path=/ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.483 DEBUG:boto:Method: GET 2023-11-02T18:52:44.483 DEBUG:boto:Path: /ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.484 DEBUG:boto:Data: 2023-11-02T18:52:44.484 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.485 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.485 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.486 DEBUG:boto:Params: {} 2023-11-02T18:52:44.486 DEBUG:boto:Token: None 2023-11-02T18:52:44.487 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.487 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IsRsltkf+cHM2Tydp2xNiY34jiw= 2023-11-02T18:52:44.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IsRsltkf+cHM2Tydp2xNiY34jiw=', 'Content-Length': '0'} 2023-11-02T18:52:44.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005653d065d7dfcea-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.491 DEBUG:boto:path=/ibtljn-61/obj69 2023-11-02T18:52:44.491 DEBUG:boto:auth_path=/ibtljn-61/obj69 2023-11-02T18:52:44.492 DEBUG:boto:path=/ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.492 DEBUG:boto:auth_path=/ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.493 DEBUG:boto:Method: GET 2023-11-02T18:52:44.493 DEBUG:boto:Path: /ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.494 DEBUG:boto:Data: 2023-11-02T18:52:44.494 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.495 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.495 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.495 DEBUG:boto:Params: {} 2023-11-02T18:52:44.496 DEBUG:boto:Token: None 2023-11-02T18:52:44.497 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.497 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IsRsltkf+cHM2Tydp2xNiY34jiw= 2023-11-02T18:52:44.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IsRsltkf+cHM2Tydp2xNiY34jiw=', 'Content-Length': '0'} 2023-11-02T18:52:44.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000006704d1553e9979-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.542 DEBUG:boto:path=/ibtljn-61/obj69 2023-11-02T18:52:44.543 DEBUG:boto:auth_path=/ibtljn-61/obj69 2023-11-02T18:52:44.543 DEBUG:boto:path=/ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.544 DEBUG:boto:auth_path=/ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.544 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.545 DEBUG:boto:Path: /ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.545 DEBUG:boto:Data: 2023-11-02T18:52:44.545 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.546 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.546 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.547 DEBUG:boto:Params: {} 2023-11-02T18:52:44.547 DEBUG:boto:Token: None 2023-11-02T18:52:44.548 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.548 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x6gRCglH7veCpxi640z+56uhUp8= 2023-11-02T18:52:44.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x6gRCglH7veCpxi640z+56uhUp8=', 'Content-Length': '0'} 2023-11-02T18:52:44.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea7ed829314b3984-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.552 DEBUG:boto:path=/ibtljn-61/obj69 2023-11-02T18:52:44.553 DEBUG:boto:auth_path=/ibtljn-61/obj69 2023-11-02T18:52:44.553 DEBUG:boto:path=/ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.554 DEBUG:boto:auth_path=/ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.554 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.555 DEBUG:boto:Path: /ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.555 DEBUG:boto:Data: 2023-11-02T18:52:44.555 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.556 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.556 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.557 DEBUG:boto:Params: {} 2023-11-02T18:52:44.557 DEBUG:boto:Token: None 2023-11-02T18:52:44.558 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.558 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x6gRCglH7veCpxi640z+56uhUp8= 2023-11-02T18:52:44.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x6gRCglH7veCpxi640z+56uhUp8=', 'Content-Length': '0'} 2023-11-02T18:52:44.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000408e654106fad09a-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:52:44.562 DEBUG:boto:path=/ibtljn-61/obj69 2023-11-02T18:52:44.563 DEBUG:boto:auth_path=/ibtljn-61/obj69 2023-11-02T18:52:44.563 DEBUG:boto:path=/ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.564 DEBUG:boto:auth_path=/ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.564 DEBUG:boto:Method: GET 2023-11-02T18:52:44.565 DEBUG:boto:Path: /ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.565 DEBUG:boto:Data: 2023-11-02T18:52:44.565 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.566 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.567 DEBUG:boto:Params: {} 2023-11-02T18:52:44.567 DEBUG:boto:Token: None 2023-11-02T18:52:44.568 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.568 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IsRsltkf+cHM2Tydp2xNiY34jiw= 2023-11-02T18:52:44.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IsRsltkf+cHM2Tydp2xNiY34jiw=', 'Content-Length': '0'} 2023-11-02T18:52:44.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7acb130f69a14c1-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.618 DEBUG:boto:path=/ibtljn-61/obj69 2023-11-02T18:52:44.619 DEBUG:boto:auth_path=/ibtljn-61/obj69 2023-11-02T18:52:44.619 DEBUG:boto:path=/ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.620 DEBUG:boto:auth_path=/ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.620 DEBUG:boto:Method: GET 2023-11-02T18:52:44.621 DEBUG:boto:Path: /ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.621 DEBUG:boto:Data: 2023-11-02T18:52:44.622 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.622 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.622 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.623 DEBUG:boto:Params: {} 2023-11-02T18:52:44.623 DEBUG:boto:Token: None 2023-11-02T18:52:44.624 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj69?versionId=null 2023-11-02T18:52:44.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IsRsltkf+cHM2Tydp2xNiY34jiw= 2023-11-02T18:52:44.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IsRsltkf+cHM2Tydp2xNiY34jiw=', 'Content-Length': '0'} 2023-11-02T18:52:44.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ef9365d9adc4301-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.629 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.629 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.629 DEBUG:boto:path=/ibtljn-61/?prefix=obj69 2023-11-02T18:52:44.630 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj69 2023-11-02T18:52:44.630 DEBUG:boto:Method: GET 2023-11-02T18:52:44.631 DEBUG:boto:Path: /ibtljn-61/?prefix=obj69 2023-11-02T18:52:44.631 DEBUG:boto:Data: 2023-11-02T18:52:44.632 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.632 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.633 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.633 DEBUG:boto:Params: {} 2023-11-02T18:52:44.634 DEBUG:boto:Token: None 2023-11-02T18:52:44.634 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.635 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f43239944a0282d-006543effc-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.642 DEBUG:boto:b'ibtljn-61obj691000falseobj692023-11-02T18:50:23.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.642 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.643 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.643 DEBUG:boto:path=/ibtljn-61/?prefix=obj69 2023-11-02T18:52:44.644 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj69 2023-11-02T18:52:44.644 DEBUG:boto:Method: GET 2023-11-02T18:52:44.645 DEBUG:boto:Path: /ibtljn-61/?prefix=obj69 2023-11-02T18:52:44.645 DEBUG:boto:Data: 2023-11-02T18:52:44.646 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.646 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.647 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.647 DEBUG:boto:Params: {} 2023-11-02T18:52:44.648 DEBUG:boto:Token: None 2023-11-02T18:52:44.648 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.649 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a1642530e615a0d-006543effc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.655 DEBUG:boto:b'ibtljn-61obj691000falseobj692023-11-02T18:50:23.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.656 DEBUG:boto:path=/ibtljn-61/obj69 2023-11-02T18:52:44.656 DEBUG:boto:auth_path=/ibtljn-61/obj69 2023-11-02T18:52:44.657 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.657 DEBUG:boto:Path: /ibtljn-61/obj69 2023-11-02T18:52:44.657 DEBUG:boto:Data: 2023-11-02T18:52:44.658 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.658 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.659 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.659 DEBUG:boto:Params: {} 2023-11-02T18:52:44.660 DEBUG:boto:Token: None 2023-11-02T18:52:44.660 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj69 2023-11-02T18:52:44.661 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YcWDOb0U/mYveqNPzRjXTCNqvE0= 2023-11-02T18:52:44.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YcWDOb0U/mYveqNPzRjXTCNqvE0=', 'Content-Length': '0'} 2023-11-02T18:52:44.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000304e6f06997d2039-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.664 DEBUG:boto:path=/ibtljn-61/obj69 2023-11-02T18:52:44.665 DEBUG:boto:auth_path=/ibtljn-61/obj69 2023-11-02T18:52:44.665 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.665 DEBUG:boto:Path: /ibtljn-61/obj69 2023-11-02T18:52:44.666 DEBUG:boto:Data: 2023-11-02T18:52:44.666 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.667 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.667 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.668 DEBUG:boto:Params: {} 2023-11-02T18:52:44.668 DEBUG:boto:Token: None 2023-11-02T18:52:44.669 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj69 2023-11-02T18:52:44.669 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YcWDOb0U/mYveqNPzRjXTCNqvE0= 2023-11-02T18:52:44.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YcWDOb0U/mYveqNPzRjXTCNqvE0=', 'Content-Length': '0'} 2023-11-02T18:52:44.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb2d9fa2f3e35b4d-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T18:52:44.673 DEBUG:boto:path=/ibtljn-61/obj69 2023-11-02T18:52:44.673 DEBUG:boto:auth_path=/ibtljn-61/obj69 2023-11-02T18:52:44.674 DEBUG:boto:Method: GET 2023-11-02T18:52:44.674 DEBUG:boto:Path: /ibtljn-61/obj69 2023-11-02T18:52:44.674 DEBUG:boto:Data: 2023-11-02T18:52:44.675 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.675 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.676 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.676 DEBUG:boto:Params: {} 2023-11-02T18:52:44.677 DEBUG:boto:Token: None 2023-11-02T18:52:44.677 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj69 2023-11-02T18:52:44.678 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1vByE4arcOQvOxzXDhePONXROfI= 2023-11-02T18:52:44.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1vByE4arcOQvOxzXDhePONXROfI=', 'Content-Length': '0'} 2023-11-02T18:52:44.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035a4d480da642a9c-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.722 DEBUG:boto:path=/ibtljn-61/obj69 2023-11-02T18:52:44.723 DEBUG:boto:auth_path=/ibtljn-61/obj69 2023-11-02T18:52:44.724 DEBUG:boto:Method: GET 2023-11-02T18:52:44.724 DEBUG:boto:Path: /ibtljn-61/obj69 2023-11-02T18:52:44.725 DEBUG:boto:Data: 2023-11-02T18:52:44.725 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.726 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.726 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.727 DEBUG:boto:Params: {} 2023-11-02T18:52:44.727 DEBUG:boto:Token: None 2023-11-02T18:52:44.728 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj69 2023-11-02T18:52:44.728 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1vByE4arcOQvOxzXDhePONXROfI= 2023-11-02T18:52:44.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1vByE4arcOQvOxzXDhePONXROfI=', 'Content-Length': '0'} 2023-11-02T18:52:44.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fb3b3083e8f8c89-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.733 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.734 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.734 DEBUG:boto:path=/ibtljn-61/?prefix=obj69 2023-11-02T18:52:44.735 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj69 2023-11-02T18:52:44.735 DEBUG:boto:Method: GET 2023-11-02T18:52:44.736 DEBUG:boto:Path: /ibtljn-61/?prefix=obj69 2023-11-02T18:52:44.736 DEBUG:boto:Data: 2023-11-02T18:52:44.737 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.737 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.738 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.738 DEBUG:boto:Params: {} 2023-11-02T18:52:44.739 DEBUG:boto:Token: None 2023-11-02T18:52:44.739 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008193f8ec2d294a5-006543effc-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.747 DEBUG:boto:b'ibtljn-61obj691000falseobj692023-11-02T18:50:23.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.749 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.749 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.750 DEBUG:boto:path=/ibtljn-61/?prefix=obj69 2023-11-02T18:52:44.750 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj69 2023-11-02T18:52:44.751 DEBUG:boto:Method: GET 2023-11-02T18:52:44.751 DEBUG:boto:Path: /ibtljn-61/?prefix=obj69 2023-11-02T18:52:44.752 DEBUG:boto:Data: 2023-11-02T18:52:44.752 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.753 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.753 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.754 DEBUG:boto:Params: {} 2023-11-02T18:52:44.754 DEBUG:boto:Token: None 2023-11-02T18:52:44.755 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.755 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e87e4c73d5a4713-006543effc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.766 DEBUG:boto:b'ibtljn-61obj691000falseobj692023-11-02T18:50:23.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-11-02T18:52:44.767 DEBUG:boto:path=/ibtljn-61/obj74 2023-11-02T18:52:44.767 DEBUG:boto:auth_path=/ibtljn-61/obj74 2023-11-02T18:52:44.768 DEBUG:boto:path=/ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.768 DEBUG:boto:auth_path=/ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.769 DEBUG:boto:Method: GET 2023-11-02T18:52:44.769 DEBUG:boto:Path: /ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.770 DEBUG:boto:Data: 2023-11-02T18:52:44.770 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.771 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.771 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.772 DEBUG:boto:Params: {} 2023-11-02T18:52:44.772 DEBUG:boto:Token: None 2023-11-02T18:52:44.773 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.773 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kVQ3kBfD5RoycAI8c0onTQhTA50= 2023-11-02T18:52:44.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kVQ3kBfD5RoycAI8c0onTQhTA50=', 'Content-Length': '0'} 2023-11-02T18:52:44.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5efbfa426620f65-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.777 DEBUG:boto:path=/ibtljn-61/obj74 2023-11-02T18:52:44.777 DEBUG:boto:auth_path=/ibtljn-61/obj74 2023-11-02T18:52:44.778 DEBUG:boto:path=/ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.778 DEBUG:boto:auth_path=/ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.779 DEBUG:boto:Method: GET 2023-11-02T18:52:44.779 DEBUG:boto:Path: /ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.780 DEBUG:boto:Data: 2023-11-02T18:52:44.780 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.780 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.781 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.781 DEBUG:boto:Params: {} 2023-11-02T18:52:44.782 DEBUG:boto:Token: None 2023-11-02T18:52:44.783 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.783 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kVQ3kBfD5RoycAI8c0onTQhTA50= 2023-11-02T18:52:44.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kVQ3kBfD5RoycAI8c0onTQhTA50=', 'Content-Length': '0'} 2023-11-02T18:52:44.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8037933d8091ce9-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.787 DEBUG:boto:path=/ibtljn-61/obj74 2023-11-02T18:52:44.787 DEBUG:boto:auth_path=/ibtljn-61/obj74 2023-11-02T18:52:44.788 DEBUG:boto:path=/ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.788 DEBUG:boto:auth_path=/ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.789 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.789 DEBUG:boto:Path: /ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.789 DEBUG:boto:Data: 2023-11-02T18:52:44.790 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.790 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.791 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.791 DEBUG:boto:Params: {} 2023-11-02T18:52:44.792 DEBUG:boto:Token: None 2023-11-02T18:52:44.792 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QRaqfXmNa+/GhmKke5NgYV5+Vvo= 2023-11-02T18:52:44.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QRaqfXmNa+/GhmKke5NgYV5+Vvo=', 'Content-Length': '0'} 2023-11-02T18:52:44.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c70e0d85bb223ae-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.796 DEBUG:boto:path=/ibtljn-61/obj74 2023-11-02T18:52:44.797 DEBUG:boto:auth_path=/ibtljn-61/obj74 2023-11-02T18:52:44.797 DEBUG:boto:path=/ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.798 DEBUG:boto:auth_path=/ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.798 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.799 DEBUG:boto:Path: /ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.799 DEBUG:boto:Data: 2023-11-02T18:52:44.800 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.800 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.800 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.801 DEBUG:boto:Params: {} 2023-11-02T18:52:44.801 DEBUG:boto:Token: None 2023-11-02T18:52:44.802 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.802 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QRaqfXmNa+/GhmKke5NgYV5+Vvo= 2023-11-02T18:52:44.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QRaqfXmNa+/GhmKke5NgYV5+Vvo=', 'Content-Length': '0'} 2023-11-02T18:52:44.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079355e39d8a6af56-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-11-02T18:52:44.806 DEBUG:boto:path=/ibtljn-61/obj74 2023-11-02T18:52:44.807 DEBUG:boto:auth_path=/ibtljn-61/obj74 2023-11-02T18:52:44.807 DEBUG:boto:path=/ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.808 DEBUG:boto:auth_path=/ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.808 DEBUG:boto:Method: GET 2023-11-02T18:52:44.809 DEBUG:boto:Path: /ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.809 DEBUG:boto:Data: 2023-11-02T18:52:44.810 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.810 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.811 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.811 DEBUG:boto:Params: {} 2023-11-02T18:52:44.812 DEBUG:boto:Token: None 2023-11-02T18:52:44.812 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.813 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kVQ3kBfD5RoycAI8c0onTQhTA50= 2023-11-02T18:52:44.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kVQ3kBfD5RoycAI8c0onTQhTA50=', 'Content-Length': '0'} 2023-11-02T18:52:44.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2eef5a720cacfb6-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.858 DEBUG:boto:path=/ibtljn-61/obj74 2023-11-02T18:52:44.859 DEBUG:boto:auth_path=/ibtljn-61/obj74 2023-11-02T18:52:44.859 DEBUG:boto:path=/ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.860 DEBUG:boto:auth_path=/ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.860 DEBUG:boto:Method: GET 2023-11-02T18:52:44.861 DEBUG:boto:Path: /ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.861 DEBUG:boto:Data: 2023-11-02T18:52:44.862 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.862 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.863 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.863 DEBUG:boto:Params: {} 2023-11-02T18:52:44.864 DEBUG:boto:Token: None 2023-11-02T18:52:44.864 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj74?versionId=null 2023-11-02T18:52:44.865 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kVQ3kBfD5RoycAI8c0onTQhTA50= 2023-11-02T18:52:44.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kVQ3kBfD5RoycAI8c0onTQhTA50=', 'Content-Length': '0'} 2023-11-02T18:52:44.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e86a12f979cd5e-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.869 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.870 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.870 DEBUG:boto:path=/ibtljn-61/?prefix=obj74 2023-11-02T18:52:44.871 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj74 2023-11-02T18:52:44.871 DEBUG:boto:Method: GET 2023-11-02T18:52:44.872 DEBUG:boto:Path: /ibtljn-61/?prefix=obj74 2023-11-02T18:52:44.872 DEBUG:boto:Data: 2023-11-02T18:52:44.873 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.873 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.873 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.874 DEBUG:boto:Params: {} 2023-11-02T18:52:44.874 DEBUG:boto:Token: None 2023-11-02T18:52:44.875 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.875 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042717864456943a7-006543effc-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.882 DEBUG:boto:b'ibtljn-61obj741000falseobj742023-11-02T18:50:25.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.882 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.883 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.883 DEBUG:boto:path=/ibtljn-61/?prefix=obj74 2023-11-02T18:52:44.884 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj74 2023-11-02T18:52:44.884 DEBUG:boto:Method: GET 2023-11-02T18:52:44.885 DEBUG:boto:Path: /ibtljn-61/?prefix=obj74 2023-11-02T18:52:44.885 DEBUG:boto:Data: 2023-11-02T18:52:44.885 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.886 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.886 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.887 DEBUG:boto:Params: {} 2023-11-02T18:52:44.887 DEBUG:boto:Token: None 2023-11-02T18:52:44.888 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.888 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000801fc730cc1fb47b-006543effc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.899 DEBUG:boto:b'ibtljn-61obj741000falseobj742023-11-02T18:50:25.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.900 DEBUG:boto:path=/ibtljn-61/obj74 2023-11-02T18:52:44.900 DEBUG:boto:auth_path=/ibtljn-61/obj74 2023-11-02T18:52:44.901 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.901 DEBUG:boto:Path: /ibtljn-61/obj74 2023-11-02T18:52:44.901 DEBUG:boto:Data: 2023-11-02T18:52:44.902 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.902 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.903 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.903 DEBUG:boto:Params: {} 2023-11-02T18:52:44.904 DEBUG:boto:Token: None 2023-11-02T18:52:44.904 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj74 2023-11-02T18:52:44.904 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vanE9C9tLMyPlqEO/kkH2YGxlpY= 2023-11-02T18:52:44.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vanE9C9tLMyPlqEO/kkH2YGxlpY=', 'Content-Length': '0'} 2023-11-02T18:52:44.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000c74a7234b41d66-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.908 DEBUG:boto:path=/ibtljn-61/obj74 2023-11-02T18:52:44.908 DEBUG:boto:auth_path=/ibtljn-61/obj74 2023-11-02T18:52:44.909 DEBUG:boto:Method: HEAD 2023-11-02T18:52:44.909 DEBUG:boto:Path: /ibtljn-61/obj74 2023-11-02T18:52:44.910 DEBUG:boto:Data: 2023-11-02T18:52:44.910 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.911 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.911 DEBUG:boto:Params: {} 2023-11-02T18:52:44.912 DEBUG:boto:Token: None 2023-11-02T18:52:44.912 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj74 2023-11-02T18:52:44.913 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vanE9C9tLMyPlqEO/kkH2YGxlpY= 2023-11-02T18:52:44.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vanE9C9tLMyPlqEO/kkH2YGxlpY=', 'Content-Length': '0'} 2023-11-02T18:52:44.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab579b6ddbdef4bd-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-11-02T18:52:44.917 DEBUG:boto:path=/ibtljn-61/obj74 2023-11-02T18:52:44.917 DEBUG:boto:auth_path=/ibtljn-61/obj74 2023-11-02T18:52:44.918 DEBUG:boto:Method: GET 2023-11-02T18:52:44.918 DEBUG:boto:Path: /ibtljn-61/obj74 2023-11-02T18:52:44.919 DEBUG:boto:Data: 2023-11-02T18:52:44.919 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.919 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.920 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.920 DEBUG:boto:Params: {} 2023-11-02T18:52:44.921 DEBUG:boto:Token: None 2023-11-02T18:52:44.921 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj74 2023-11-02T18:52:44.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mIFnHBkEQXEfDIdRDh8T/SAUvsI= 2023-11-02T18:52:44.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mIFnHBkEQXEfDIdRDh8T/SAUvsI=', 'Content-Length': '0'} 2023-11-02T18:52:44.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bff2fd60d3ea3d0-006543effc-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.970 DEBUG:boto:path=/ibtljn-61/obj74 2023-11-02T18:52:44.971 DEBUG:boto:auth_path=/ibtljn-61/obj74 2023-11-02T18:52:44.972 DEBUG:boto:Method: GET 2023-11-02T18:52:44.972 DEBUG:boto:Path: /ibtljn-61/obj74 2023-11-02T18:52:44.973 DEBUG:boto:Data: 2023-11-02T18:52:44.973 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.974 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.974 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.974 DEBUG:boto:Params: {} 2023-11-02T18:52:44.975 DEBUG:boto:Token: None 2023-11-02T18:52:44.976 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/obj74 2023-11-02T18:52:44.976 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mIFnHBkEQXEfDIdRDh8T/SAUvsI= 2023-11-02T18:52:44.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mIFnHBkEQXEfDIdRDh8T/SAUvsI=', 'Content-Length': '0'} 2023-11-02T18:52:44.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039ead1eee74ebaca-006543effc-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.980 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.981 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.981 DEBUG:boto:path=/ibtljn-61/?prefix=obj74 2023-11-02T18:52:44.982 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj74 2023-11-02T18:52:44.982 DEBUG:boto:Method: GET 2023-11-02T18:52:44.983 DEBUG:boto:Path: /ibtljn-61/?prefix=obj74 2023-11-02T18:52:44.983 DEBUG:boto:Data: 2023-11-02T18:52:44.983 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.984 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:44.984 DEBUG:boto:Port: 8001 2023-11-02T18:52:44.985 DEBUG:boto:Params: {} 2023-11-02T18:52:44.985 DEBUG:boto:Token: None 2023-11-02T18:52:44.986 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:44 GMT /ibtljn-61/ 2023-11-02T18:52:44.986 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs= 2023-11-02T18:52:44.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oDK3L3Stx7f2xjZBhdGt8uSFBGs=', 'Content-Length': '0'} 2023-11-02T18:52:44.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005be26e2e6dd709c9-006543effc-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:44.993 DEBUG:boto:b'ibtljn-61obj741000falseobj742023-11-02T18:50:25.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:44.994 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:44.995 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:44.995 DEBUG:boto:path=/ibtljn-61/?prefix=obj74 2023-11-02T18:52:44.996 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj74 2023-11-02T18:52:44.996 DEBUG:boto:Method: GET 2023-11-02T18:52:44.996 DEBUG:boto:Path: /ibtljn-61/?prefix=obj74 2023-11-02T18:52:44.997 DEBUG:boto:Data: 2023-11-02T18:52:44.997 DEBUG:boto:Headers: {} 2023-11-02T18:52:44.998 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:44.998 DEBUG:boto:Port: 8000 2023-11-02T18:52:44.999 DEBUG:boto:Params: {} 2023-11-02T18:52:44.999 DEBUG:boto:Token: None 2023-11-02T18:52:45.000 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.000 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079e9091b7d0779bd-006543effc-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.011 DEBUG:boto:b'ibtljn-61obj741000falseobj742023-11-02T18:50:25.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:52:45.013 DEBUG:boto:path=/ibtljn-61/obj76 2023-11-02T18:52:45.013 DEBUG:boto:auth_path=/ibtljn-61/obj76 2023-11-02T18:52:45.014 DEBUG:boto:path=/ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.014 DEBUG:boto:auth_path=/ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.015 DEBUG:boto:Method: GET 2023-11-02T18:52:45.015 DEBUG:boto:Path: /ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.015 DEBUG:boto:Data: 2023-11-02T18:52:45.016 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.016 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.017 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.017 DEBUG:boto:Params: {} 2023-11-02T18:52:45.018 DEBUG:boto:Token: None 2023-11-02T18:52:45.018 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.019 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aSX8K4+jTGFqswni/CmSUjIF6X4= 2023-11-02T18:52:45.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aSX8K4+jTGFqswni/CmSUjIF6X4=', 'Content-Length': '0'} 2023-11-02T18:52:45.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090d6fd1ae1ac5aea-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.023 DEBUG:boto:path=/ibtljn-61/obj76 2023-11-02T18:52:45.023 DEBUG:boto:auth_path=/ibtljn-61/obj76 2023-11-02T18:52:45.023 DEBUG:boto:path=/ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.024 DEBUG:boto:auth_path=/ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.024 DEBUG:boto:Method: GET 2023-11-02T18:52:45.025 DEBUG:boto:Path: /ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.025 DEBUG:boto:Data: 2023-11-02T18:52:45.026 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.026 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.027 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.027 DEBUG:boto:Params: {} 2023-11-02T18:52:45.028 DEBUG:boto:Token: None 2023-11-02T18:52:45.028 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.029 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aSX8K4+jTGFqswni/CmSUjIF6X4= 2023-11-02T18:52:45.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aSX8K4+jTGFqswni/CmSUjIF6X4=', 'Content-Length': '0'} 2023-11-02T18:52:45.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e505529decd7ce2-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.074 DEBUG:boto:path=/ibtljn-61/obj76 2023-11-02T18:52:45.075 DEBUG:boto:auth_path=/ibtljn-61/obj76 2023-11-02T18:52:45.075 DEBUG:boto:path=/ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.076 DEBUG:boto:auth_path=/ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.076 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.077 DEBUG:boto:Path: /ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.077 DEBUG:boto:Data: 2023-11-02T18:52:45.078 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.078 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.079 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.079 DEBUG:boto:Params: {} 2023-11-02T18:52:45.080 DEBUG:boto:Token: None 2023-11-02T18:52:45.080 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.080 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KI7VX7J5FY9q5+i6XqR5s6TVWHI= 2023-11-02T18:52:45.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KI7VX7J5FY9q5+i6XqR5s6TVWHI=', 'Content-Length': '0'} 2023-11-02T18:52:45.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006acbb0379a0f0c54-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.086 DEBUG:boto:path=/ibtljn-61/obj76 2023-11-02T18:52:45.086 DEBUG:boto:auth_path=/ibtljn-61/obj76 2023-11-02T18:52:45.086 DEBUG:boto:path=/ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.087 DEBUG:boto:auth_path=/ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.087 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.088 DEBUG:boto:Path: /ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.088 DEBUG:boto:Data: 2023-11-02T18:52:45.089 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.089 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.089 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.090 DEBUG:boto:Params: {} 2023-11-02T18:52:45.090 DEBUG:boto:Token: None 2023-11-02T18:52:45.091 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.091 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KI7VX7J5FY9q5+i6XqR5s6TVWHI= 2023-11-02T18:52:45.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KI7VX7J5FY9q5+i6XqR5s6TVWHI=', 'Content-Length': '0'} 2023-11-02T18:52:45.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085a9d2fb8a1003d8-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:52:45.096 DEBUG:boto:path=/ibtljn-61/obj76 2023-11-02T18:52:45.097 DEBUG:boto:auth_path=/ibtljn-61/obj76 2023-11-02T18:52:45.097 DEBUG:boto:path=/ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.098 DEBUG:boto:auth_path=/ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.098 DEBUG:boto:Method: GET 2023-11-02T18:52:45.098 DEBUG:boto:Path: /ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.099 DEBUG:boto:Data: 2023-11-02T18:52:45.099 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.100 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.100 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.101 DEBUG:boto:Params: {} 2023-11-02T18:52:45.101 DEBUG:boto:Token: None 2023-11-02T18:52:45.102 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.102 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aSX8K4+jTGFqswni/CmSUjIF6X4= 2023-11-02T18:52:45.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aSX8K4+jTGFqswni/CmSUjIF6X4=', 'Content-Length': '0'} 2023-11-02T18:52:45.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaabdc41c766f247-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.146 DEBUG:boto:path=/ibtljn-61/obj76 2023-11-02T18:52:45.147 DEBUG:boto:auth_path=/ibtljn-61/obj76 2023-11-02T18:52:45.148 DEBUG:boto:path=/ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.148 DEBUG:boto:auth_path=/ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.149 DEBUG:boto:Method: GET 2023-11-02T18:52:45.149 DEBUG:boto:Path: /ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.150 DEBUG:boto:Data: 2023-11-02T18:52:45.150 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.151 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.151 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.152 DEBUG:boto:Params: {} 2023-11-02T18:52:45.152 DEBUG:boto:Token: None 2023-11-02T18:52:45.153 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj76?versionId=null 2023-11-02T18:52:45.153 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aSX8K4+jTGFqswni/CmSUjIF6X4= 2023-11-02T18:52:45.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aSX8K4+jTGFqswni/CmSUjIF6X4=', 'Content-Length': '0'} 2023-11-02T18:52:45.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002abde4c286f456dc-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.158 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.159 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.159 DEBUG:boto:path=/ibtljn-61/?prefix=obj76 2023-11-02T18:52:45.160 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj76 2023-11-02T18:52:45.160 DEBUG:boto:Method: GET 2023-11-02T18:52:45.161 DEBUG:boto:Path: /ibtljn-61/?prefix=obj76 2023-11-02T18:52:45.161 DEBUG:boto:Data: 2023-11-02T18:52:45.162 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.162 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.163 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.163 DEBUG:boto:Params: {} 2023-11-02T18:52:45.164 DEBUG:boto:Token: None 2023-11-02T18:52:45.164 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.165 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005530521e59feb64b-006543effd-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.172 DEBUG:boto:b'ibtljn-61obj761000falseobj762023-11-02T18:50:23.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.172 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.173 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.173 DEBUG:boto:path=/ibtljn-61/?prefix=obj76 2023-11-02T18:52:45.174 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj76 2023-11-02T18:52:45.174 DEBUG:boto:Method: GET 2023-11-02T18:52:45.175 DEBUG:boto:Path: /ibtljn-61/?prefix=obj76 2023-11-02T18:52:45.175 DEBUG:boto:Data: 2023-11-02T18:52:45.176 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.176 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.176 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.177 DEBUG:boto:Params: {} 2023-11-02T18:52:45.177 DEBUG:boto:Token: None 2023-11-02T18:52:45.178 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.178 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c97515bb260fe2a-006543effd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.189 DEBUG:boto:b'ibtljn-61obj761000falseobj762023-11-02T18:50:23.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.190 DEBUG:boto:path=/ibtljn-61/obj76 2023-11-02T18:52:45.191 DEBUG:boto:auth_path=/ibtljn-61/obj76 2023-11-02T18:52:45.191 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.191 DEBUG:boto:Path: /ibtljn-61/obj76 2023-11-02T18:52:45.192 DEBUG:boto:Data: 2023-11-02T18:52:45.192 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.193 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.193 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.194 DEBUG:boto:Params: {} 2023-11-02T18:52:45.194 DEBUG:boto:Token: None 2023-11-02T18:52:45.195 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj76 2023-11-02T18:52:45.195 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4dKpJzkrFs/5MidRugOYW9PdnD8= 2023-11-02T18:52:45.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4dKpJzkrFs/5MidRugOYW9PdnD8=', 'Content-Length': '0'} 2023-11-02T18:52:45.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0249a61a5984d24-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.198 DEBUG:boto:path=/ibtljn-61/obj76 2023-11-02T18:52:45.199 DEBUG:boto:auth_path=/ibtljn-61/obj76 2023-11-02T18:52:45.199 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.200 DEBUG:boto:Path: /ibtljn-61/obj76 2023-11-02T18:52:45.200 DEBUG:boto:Data: 2023-11-02T18:52:45.200 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.201 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.201 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.202 DEBUG:boto:Params: {} 2023-11-02T18:52:45.202 DEBUG:boto:Token: None 2023-11-02T18:52:45.203 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj76 2023-11-02T18:52:45.203 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4dKpJzkrFs/5MidRugOYW9PdnD8= 2023-11-02T18:52:45.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4dKpJzkrFs/5MidRugOYW9PdnD8=', 'Content-Length': '0'} 2023-11-02T18:52:45.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f566f3f91e9ebed4-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T18:52:45.207 DEBUG:boto:path=/ibtljn-61/obj76 2023-11-02T18:52:45.207 DEBUG:boto:auth_path=/ibtljn-61/obj76 2023-11-02T18:52:45.208 DEBUG:boto:Method: GET 2023-11-02T18:52:45.208 DEBUG:boto:Path: /ibtljn-61/obj76 2023-11-02T18:52:45.209 DEBUG:boto:Data: 2023-11-02T18:52:45.209 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.210 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.210 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.210 DEBUG:boto:Params: {} 2023-11-02T18:52:45.211 DEBUG:boto:Token: None 2023-11-02T18:52:45.211 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj76 2023-11-02T18:52:45.212 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ONQ+qf2HTmuCAXQ6Z1SEjsf5CtQ= 2023-11-02T18:52:45.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ONQ+qf2HTmuCAXQ6Z1SEjsf5CtQ=', 'Content-Length': '0'} 2023-11-02T18:52:45.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ff35cefc4e80fcc-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.216 DEBUG:boto:path=/ibtljn-61/obj76 2023-11-02T18:52:45.216 DEBUG:boto:auth_path=/ibtljn-61/obj76 2023-11-02T18:52:45.217 DEBUG:boto:Method: GET 2023-11-02T18:52:45.217 DEBUG:boto:Path: /ibtljn-61/obj76 2023-11-02T18:52:45.218 DEBUG:boto:Data: 2023-11-02T18:52:45.218 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.219 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.219 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.219 DEBUG:boto:Params: {} 2023-11-02T18:52:45.220 DEBUG:boto:Token: None 2023-11-02T18:52:45.220 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj76 2023-11-02T18:52:45.221 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ONQ+qf2HTmuCAXQ6Z1SEjsf5CtQ= 2023-11-02T18:52:45.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ONQ+qf2HTmuCAXQ6Z1SEjsf5CtQ=', 'Content-Length': '0'} 2023-11-02T18:52:45.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006ab5fd19f922f8c-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.225 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.226 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.226 DEBUG:boto:path=/ibtljn-61/?prefix=obj76 2023-11-02T18:52:45.227 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj76 2023-11-02T18:52:45.227 DEBUG:boto:Method: GET 2023-11-02T18:52:45.227 DEBUG:boto:Path: /ibtljn-61/?prefix=obj76 2023-11-02T18:52:45.228 DEBUG:boto:Data: 2023-11-02T18:52:45.228 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.229 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.229 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.230 DEBUG:boto:Params: {} 2023-11-02T18:52:45.230 DEBUG:boto:Token: None 2023-11-02T18:52:45.231 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.231 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2e9594fa02a442f-006543effd-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.278 DEBUG:boto:b'ibtljn-61obj761000falseobj762023-11-02T18:50:23.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.279 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.280 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.280 DEBUG:boto:path=/ibtljn-61/?prefix=obj76 2023-11-02T18:52:45.281 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj76 2023-11-02T18:52:45.281 DEBUG:boto:Method: GET 2023-11-02T18:52:45.282 DEBUG:boto:Path: /ibtljn-61/?prefix=obj76 2023-11-02T18:52:45.282 DEBUG:boto:Data: 2023-11-02T18:52:45.283 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.283 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.284 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.284 DEBUG:boto:Params: {} 2023-11-02T18:52:45.285 DEBUG:boto:Token: None 2023-11-02T18:52:45.285 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.286 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc43d0681f428602-006543effd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.297 DEBUG:boto:b'ibtljn-61obj761000falseobj762023-11-02T18:50:23.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:52:45.299 DEBUG:boto:path=/ibtljn-61/obj85 2023-11-02T18:52:45.299 DEBUG:boto:auth_path=/ibtljn-61/obj85 2023-11-02T18:52:45.300 DEBUG:boto:path=/ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.300 DEBUG:boto:auth_path=/ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.301 DEBUG:boto:Method: GET 2023-11-02T18:52:45.301 DEBUG:boto:Path: /ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.301 DEBUG:boto:Data: 2023-11-02T18:52:45.302 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.302 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.303 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.303 DEBUG:boto:Params: {} 2023-11-02T18:52:45.304 DEBUG:boto:Token: None 2023-11-02T18:52:45.304 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ehQx2AuK2gBdQhlUXsdlmZfvvFc= 2023-11-02T18:52:45.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ehQx2AuK2gBdQhlUXsdlmZfvvFc=', 'Content-Length': '0'} 2023-11-02T18:52:45.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce831ca0c9900ddb-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.308 DEBUG:boto:path=/ibtljn-61/obj85 2023-11-02T18:52:45.309 DEBUG:boto:auth_path=/ibtljn-61/obj85 2023-11-02T18:52:45.309 DEBUG:boto:path=/ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.310 DEBUG:boto:auth_path=/ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.310 DEBUG:boto:Method: GET 2023-11-02T18:52:45.311 DEBUG:boto:Path: /ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.311 DEBUG:boto:Data: 2023-11-02T18:52:45.312 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.312 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.313 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.313 DEBUG:boto:Params: {} 2023-11-02T18:52:45.313 DEBUG:boto:Token: None 2023-11-02T18:52:45.314 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.315 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ehQx2AuK2gBdQhlUXsdlmZfvvFc= 2023-11-02T18:52:45.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ehQx2AuK2gBdQhlUXsdlmZfvvFc=', 'Content-Length': '0'} 2023-11-02T18:52:45.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3fc370df716c971-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.362 DEBUG:boto:path=/ibtljn-61/obj85 2023-11-02T18:52:45.364 DEBUG:boto:auth_path=/ibtljn-61/obj85 2023-11-02T18:52:45.364 DEBUG:boto:path=/ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.364 DEBUG:boto:auth_path=/ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.365 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.365 DEBUG:boto:Path: /ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.366 DEBUG:boto:Data: 2023-11-02T18:52:45.366 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.367 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.367 DEBUG:boto:Params: {} 2023-11-02T18:52:45.368 DEBUG:boto:Token: None 2023-11-02T18:52:45.368 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.369 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G95BdmboQTaRc9/94ljMuika9VA= 2023-11-02T18:52:45.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G95BdmboQTaRc9/94ljMuika9VA=', 'Content-Length': '0'} 2023-11-02T18:52:45.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a090e7849064519-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.373 DEBUG:boto:path=/ibtljn-61/obj85 2023-11-02T18:52:45.373 DEBUG:boto:auth_path=/ibtljn-61/obj85 2023-11-02T18:52:45.374 DEBUG:boto:path=/ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.374 DEBUG:boto:auth_path=/ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.375 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.375 DEBUG:boto:Path: /ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.376 DEBUG:boto:Data: 2023-11-02T18:52:45.376 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.376 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.377 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.377 DEBUG:boto:Params: {} 2023-11-02T18:52:45.378 DEBUG:boto:Token: None 2023-11-02T18:52:45.378 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.379 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G95BdmboQTaRc9/94ljMuika9VA= 2023-11-02T18:52:45.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G95BdmboQTaRc9/94ljMuika9VA=', 'Content-Length': '0'} 2023-11-02T18:52:45.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007684c8bf351dc7ab-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:52:45.383 DEBUG:boto:path=/ibtljn-61/obj85 2023-11-02T18:52:45.384 DEBUG:boto:auth_path=/ibtljn-61/obj85 2023-11-02T18:52:45.384 DEBUG:boto:path=/ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.385 DEBUG:boto:auth_path=/ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.385 DEBUG:boto:Method: GET 2023-11-02T18:52:45.385 DEBUG:boto:Path: /ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.386 DEBUG:boto:Data: 2023-11-02T18:52:45.386 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.387 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.387 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.388 DEBUG:boto:Params: {} 2023-11-02T18:52:45.388 DEBUG:boto:Token: None 2023-11-02T18:52:45.389 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.389 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ehQx2AuK2gBdQhlUXsdlmZfvvFc= 2023-11-02T18:52:45.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ehQx2AuK2gBdQhlUXsdlmZfvvFc=', 'Content-Length': '0'} 2023-11-02T18:52:45.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009d49a7ea283afcc-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.394 DEBUG:boto:path=/ibtljn-61/obj85 2023-11-02T18:52:45.394 DEBUG:boto:auth_path=/ibtljn-61/obj85 2023-11-02T18:52:45.395 DEBUG:boto:path=/ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.395 DEBUG:boto:auth_path=/ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.395 DEBUG:boto:Method: GET 2023-11-02T18:52:45.396 DEBUG:boto:Path: /ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.396 DEBUG:boto:Data: 2023-11-02T18:52:45.397 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.397 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.398 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.398 DEBUG:boto:Params: {} 2023-11-02T18:52:45.399 DEBUG:boto:Token: None 2023-11-02T18:52:45.399 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj85?versionId=null 2023-11-02T18:52:45.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ehQx2AuK2gBdQhlUXsdlmZfvvFc= 2023-11-02T18:52:45.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ehQx2AuK2gBdQhlUXsdlmZfvvFc=', 'Content-Length': '0'} 2023-11-02T18:52:45.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001e1aac1859ca403-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.405 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.405 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.405 DEBUG:boto:path=/ibtljn-61/?prefix=obj85 2023-11-02T18:52:45.406 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj85 2023-11-02T18:52:45.406 DEBUG:boto:Method: GET 2023-11-02T18:52:45.407 DEBUG:boto:Path: /ibtljn-61/?prefix=obj85 2023-11-02T18:52:45.407 DEBUG:boto:Data: 2023-11-02T18:52:45.408 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.408 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.409 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.409 DEBUG:boto:Params: {} 2023-11-02T18:52:45.410 DEBUG:boto:Token: None 2023-11-02T18:52:45.410 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.411 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079072bb5a77288ff-006543effd-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.417 DEBUG:boto:b'ibtljn-61obj851000falseobj852023-11-02T18:50:22.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.418 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.418 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.419 DEBUG:boto:path=/ibtljn-61/?prefix=obj85 2023-11-02T18:52:45.419 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj85 2023-11-02T18:52:45.419 DEBUG:boto:Method: GET 2023-11-02T18:52:45.420 DEBUG:boto:Path: /ibtljn-61/?prefix=obj85 2023-11-02T18:52:45.420 DEBUG:boto:Data: 2023-11-02T18:52:45.421 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.421 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.421 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.422 DEBUG:boto:Params: {} 2023-11-02T18:52:45.422 DEBUG:boto:Token: None 2023-11-02T18:52:45.423 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.423 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e1a8826c757b0cd-006543effd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.482 DEBUG:boto:b'ibtljn-61obj851000falseobj852023-11-02T18:50:22.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.483 DEBUG:boto:path=/ibtljn-61/obj85 2023-11-02T18:52:45.484 DEBUG:boto:auth_path=/ibtljn-61/obj85 2023-11-02T18:52:45.484 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.484 DEBUG:boto:Path: /ibtljn-61/obj85 2023-11-02T18:52:45.485 DEBUG:boto:Data: 2023-11-02T18:52:45.485 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.486 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.486 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.487 DEBUG:boto:Params: {} 2023-11-02T18:52:45.487 DEBUG:boto:Token: None 2023-11-02T18:52:45.488 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj85 2023-11-02T18:52:45.488 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lfbpjZzgSwE79hCKlGUTr8zSXAc= 2023-11-02T18:52:45.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lfbpjZzgSwE79hCKlGUTr8zSXAc=', 'Content-Length': '0'} 2023-11-02T18:52:45.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b1380fceea67635-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.492 DEBUG:boto:path=/ibtljn-61/obj85 2023-11-02T18:52:45.493 DEBUG:boto:auth_path=/ibtljn-61/obj85 2023-11-02T18:52:45.493 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.494 DEBUG:boto:Path: /ibtljn-61/obj85 2023-11-02T18:52:45.494 DEBUG:boto:Data: 2023-11-02T18:52:45.495 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.495 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.495 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.496 DEBUG:boto:Params: {} 2023-11-02T18:52:45.496 DEBUG:boto:Token: None 2023-11-02T18:52:45.497 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj85 2023-11-02T18:52:45.497 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lfbpjZzgSwE79hCKlGUTr8zSXAc= 2023-11-02T18:52:45.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lfbpjZzgSwE79hCKlGUTr8zSXAc=', 'Content-Length': '0'} 2023-11-02T18:52:45.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969f6aa21da81ede-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T18:52:45.504 DEBUG:boto:path=/ibtljn-61/obj85 2023-11-02T18:52:45.504 DEBUG:boto:auth_path=/ibtljn-61/obj85 2023-11-02T18:52:45.505 DEBUG:boto:Method: GET 2023-11-02T18:52:45.505 DEBUG:boto:Path: /ibtljn-61/obj85 2023-11-02T18:52:45.506 DEBUG:boto:Data: 2023-11-02T18:52:45.506 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.506 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.507 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.507 DEBUG:boto:Params: {} 2023-11-02T18:52:45.508 DEBUG:boto:Token: None 2023-11-02T18:52:45.508 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj85 2023-11-02T18:52:45.509 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/aZngzrV9YOylOMgLi5FRMwyrN4= 2023-11-02T18:52:45.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/aZngzrV9YOylOMgLi5FRMwyrN4=', 'Content-Length': '0'} 2023-11-02T18:52:45.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5fc980bfcd5730a-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.513 DEBUG:boto:path=/ibtljn-61/obj85 2023-11-02T18:52:45.514 DEBUG:boto:auth_path=/ibtljn-61/obj85 2023-11-02T18:52:45.514 DEBUG:boto:Method: GET 2023-11-02T18:52:45.515 DEBUG:boto:Path: /ibtljn-61/obj85 2023-11-02T18:52:45.515 DEBUG:boto:Data: 2023-11-02T18:52:45.515 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.516 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.516 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.517 DEBUG:boto:Params: {} 2023-11-02T18:52:45.517 DEBUG:boto:Token: None 2023-11-02T18:52:45.518 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj85 2023-11-02T18:52:45.518 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/aZngzrV9YOylOMgLi5FRMwyrN4= 2023-11-02T18:52:45.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/aZngzrV9YOylOMgLi5FRMwyrN4=', 'Content-Length': '0'} 2023-11-02T18:52:45.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007668c926593a9a54-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.523 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.523 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.524 DEBUG:boto:path=/ibtljn-61/?prefix=obj85 2023-11-02T18:52:45.524 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj85 2023-11-02T18:52:45.525 DEBUG:boto:Method: GET 2023-11-02T18:52:45.525 DEBUG:boto:Path: /ibtljn-61/?prefix=obj85 2023-11-02T18:52:45.525 DEBUG:boto:Data: 2023-11-02T18:52:45.526 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.526 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.527 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.527 DEBUG:boto:Params: {} 2023-11-02T18:52:45.528 DEBUG:boto:Token: None 2023-11-02T18:52:45.528 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.529 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b56cc20118b95e92-006543effd-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.536 DEBUG:boto:b'ibtljn-61obj851000falseobj852023-11-02T18:50:22.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.537 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.538 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.538 DEBUG:boto:path=/ibtljn-61/?prefix=obj85 2023-11-02T18:52:45.539 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj85 2023-11-02T18:52:45.539 DEBUG:boto:Method: GET 2023-11-02T18:52:45.540 DEBUG:boto:Path: /ibtljn-61/?prefix=obj85 2023-11-02T18:52:45.540 DEBUG:boto:Data: 2023-11-02T18:52:45.541 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.542 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.542 DEBUG:boto:Params: {} 2023-11-02T18:52:45.543 DEBUG:boto:Token: None 2023-11-02T18:52:45.543 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000154e447d35ad62e-006543effd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.555 DEBUG:boto:b'ibtljn-61obj851000falseobj852023-11-02T18:50:22.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:52:45.556 DEBUG:boto:path=/ibtljn-61/obj87 2023-11-02T18:52:45.556 DEBUG:boto:auth_path=/ibtljn-61/obj87 2023-11-02T18:52:45.557 DEBUG:boto:path=/ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.557 DEBUG:boto:auth_path=/ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.558 DEBUG:boto:Method: GET 2023-11-02T18:52:45.558 DEBUG:boto:Path: /ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.559 DEBUG:boto:Data: 2023-11-02T18:52:45.559 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.560 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.560 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.561 DEBUG:boto:Params: {} 2023-11-02T18:52:45.561 DEBUG:boto:Token: None 2023-11-02T18:52:45.562 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.562 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eDJh2IenVV+5PkwSgoyuWguU8xI= 2023-11-02T18:52:45.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eDJh2IenVV+5PkwSgoyuWguU8xI=', 'Content-Length': '0'} 2023-11-02T18:52:45.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb754c53b5f9d9c1-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.606 DEBUG:boto:path=/ibtljn-61/obj87 2023-11-02T18:52:45.607 DEBUG:boto:auth_path=/ibtljn-61/obj87 2023-11-02T18:52:45.608 DEBUG:boto:path=/ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.608 DEBUG:boto:auth_path=/ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.609 DEBUG:boto:Method: GET 2023-11-02T18:52:45.609 DEBUG:boto:Path: /ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.609 DEBUG:boto:Data: 2023-11-02T18:52:45.610 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.610 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.611 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.611 DEBUG:boto:Params: {} 2023-11-02T18:52:45.612 DEBUG:boto:Token: None 2023-11-02T18:52:45.612 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.613 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eDJh2IenVV+5PkwSgoyuWguU8xI= 2023-11-02T18:52:45.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eDJh2IenVV+5PkwSgoyuWguU8xI=', 'Content-Length': '0'} 2023-11-02T18:52:45.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebf401881154cd79-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.618 DEBUG:boto:path=/ibtljn-61/obj87 2023-11-02T18:52:45.618 DEBUG:boto:auth_path=/ibtljn-61/obj87 2023-11-02T18:52:45.619 DEBUG:boto:path=/ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.619 DEBUG:boto:auth_path=/ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.620 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.620 DEBUG:boto:Path: /ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.621 DEBUG:boto:Data: 2023-11-02T18:52:45.621 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.621 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.622 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.622 DEBUG:boto:Params: {} 2023-11-02T18:52:45.623 DEBUG:boto:Token: None 2023-11-02T18:52:45.624 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dwWqQc+fNW9JZnQBUTdxzxRAjXI= 2023-11-02T18:52:45.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dwWqQc+fNW9JZnQBUTdxzxRAjXI=', 'Content-Length': '0'} 2023-11-02T18:52:45.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057bad5bd776321dc-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.628 DEBUG:boto:path=/ibtljn-61/obj87 2023-11-02T18:52:45.629 DEBUG:boto:auth_path=/ibtljn-61/obj87 2023-11-02T18:52:45.629 DEBUG:boto:path=/ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.630 DEBUG:boto:auth_path=/ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.630 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.631 DEBUG:boto:Path: /ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.631 DEBUG:boto:Data: 2023-11-02T18:52:45.632 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.632 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.633 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.633 DEBUG:boto:Params: {} 2023-11-02T18:52:45.634 DEBUG:boto:Token: None 2023-11-02T18:52:45.634 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.635 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dwWqQc+fNW9JZnQBUTdxzxRAjXI= 2023-11-02T18:52:45.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dwWqQc+fNW9JZnQBUTdxzxRAjXI=', 'Content-Length': '0'} 2023-11-02T18:52:45.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045f88279e49b31fc-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:52:45.639 DEBUG:boto:path=/ibtljn-61/obj87 2023-11-02T18:52:45.640 DEBUG:boto:auth_path=/ibtljn-61/obj87 2023-11-02T18:52:45.640 DEBUG:boto:path=/ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.641 DEBUG:boto:auth_path=/ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.641 DEBUG:boto:Method: GET 2023-11-02T18:52:45.642 DEBUG:boto:Path: /ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.642 DEBUG:boto:Data: 2023-11-02T18:52:45.643 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.643 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.644 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.644 DEBUG:boto:Params: {} 2023-11-02T18:52:45.645 DEBUG:boto:Token: None 2023-11-02T18:52:45.645 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.646 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eDJh2IenVV+5PkwSgoyuWguU8xI= 2023-11-02T18:52:45.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eDJh2IenVV+5PkwSgoyuWguU8xI=', 'Content-Length': '0'} 2023-11-02T18:52:45.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083069d294257baca-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.654 DEBUG:boto:path=/ibtljn-61/obj87 2023-11-02T18:52:45.654 DEBUG:boto:auth_path=/ibtljn-61/obj87 2023-11-02T18:52:45.655 DEBUG:boto:path=/ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.655 DEBUG:boto:auth_path=/ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.656 DEBUG:boto:Method: GET 2023-11-02T18:52:45.656 DEBUG:boto:Path: /ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.657 DEBUG:boto:Data: 2023-11-02T18:52:45.657 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.658 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.658 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.659 DEBUG:boto:Params: {} 2023-11-02T18:52:45.659 DEBUG:boto:Token: None 2023-11-02T18:52:45.660 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj87?versionId=null 2023-11-02T18:52:45.660 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eDJh2IenVV+5PkwSgoyuWguU8xI= 2023-11-02T18:52:45.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eDJh2IenVV+5PkwSgoyuWguU8xI=', 'Content-Length': '0'} 2023-11-02T18:52:45.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e61f6c9219a7d662-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.664 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.664 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.665 DEBUG:boto:path=/ibtljn-61/?prefix=obj87 2023-11-02T18:52:45.666 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj87 2023-11-02T18:52:45.666 DEBUG:boto:Method: GET 2023-11-02T18:52:45.667 DEBUG:boto:Path: /ibtljn-61/?prefix=obj87 2023-11-02T18:52:45.667 DEBUG:boto:Data: 2023-11-02T18:52:45.668 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.668 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.669 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.669 DEBUG:boto:Params: {} 2023-11-02T18:52:45.670 DEBUG:boto:Token: None 2023-11-02T18:52:45.670 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.671 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8639084ffbf43e4-006543effd-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.722 DEBUG:boto:b'ibtljn-61obj871000falseobj872023-11-02T18:50:24.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.723 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.724 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.724 DEBUG:boto:path=/ibtljn-61/?prefix=obj87 2023-11-02T18:52:45.725 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj87 2023-11-02T18:52:45.725 DEBUG:boto:Method: GET 2023-11-02T18:52:45.726 DEBUG:boto:Path: /ibtljn-61/?prefix=obj87 2023-11-02T18:52:45.726 DEBUG:boto:Data: 2023-11-02T18:52:45.727 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.727 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.727 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.728 DEBUG:boto:Params: {} 2023-11-02T18:52:45.728 DEBUG:boto:Token: None 2023-11-02T18:52:45.729 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.730 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078a206c848505370-006543effd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.740 DEBUG:boto:b'ibtljn-61obj871000falseobj872023-11-02T18:50:24.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.740 DEBUG:boto:path=/ibtljn-61/obj87 2023-11-02T18:52:45.741 DEBUG:boto:auth_path=/ibtljn-61/obj87 2023-11-02T18:52:45.741 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.742 DEBUG:boto:Path: /ibtljn-61/obj87 2023-11-02T18:52:45.742 DEBUG:boto:Data: 2023-11-02T18:52:45.743 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.743 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.743 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.744 DEBUG:boto:Params: {} 2023-11-02T18:52:45.744 DEBUG:boto:Token: None 2023-11-02T18:52:45.745 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj87 2023-11-02T18:52:45.745 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QKP9+2w8/o+zQXnEgKFmq9yjwcM= 2023-11-02T18:52:45.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QKP9+2w8/o+zQXnEgKFmq9yjwcM=', 'Content-Length': '0'} 2023-11-02T18:52:45.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f65825a350c15a95-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.749 DEBUG:boto:path=/ibtljn-61/obj87 2023-11-02T18:52:45.749 DEBUG:boto:auth_path=/ibtljn-61/obj87 2023-11-02T18:52:45.750 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.750 DEBUG:boto:Path: /ibtljn-61/obj87 2023-11-02T18:52:45.750 DEBUG:boto:Data: 2023-11-02T18:52:45.751 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.751 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.753 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.753 DEBUG:boto:Params: {} 2023-11-02T18:52:45.754 DEBUG:boto:Token: None 2023-11-02T18:52:45.754 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj87 2023-11-02T18:52:45.755 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QKP9+2w8/o+zQXnEgKFmq9yjwcM= 2023-11-02T18:52:45.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QKP9+2w8/o+zQXnEgKFmq9yjwcM=', 'Content-Length': '0'} 2023-11-02T18:52:45.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab9a41f58a74552-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T18:52:45.759 DEBUG:boto:path=/ibtljn-61/obj87 2023-11-02T18:52:45.759 DEBUG:boto:auth_path=/ibtljn-61/obj87 2023-11-02T18:52:45.760 DEBUG:boto:Method: GET 2023-11-02T18:52:45.760 DEBUG:boto:Path: /ibtljn-61/obj87 2023-11-02T18:52:45.760 DEBUG:boto:Data: 2023-11-02T18:52:45.761 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.761 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.762 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.762 DEBUG:boto:Params: {} 2023-11-02T18:52:45.763 DEBUG:boto:Token: None 2023-11-02T18:52:45.763 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj87 2023-11-02T18:52:45.764 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:caZ3S9IBS27pOJanA1NbEN233j4= 2023-11-02T18:52:45.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:caZ3S9IBS27pOJanA1NbEN233j4=', 'Content-Length': '0'} 2023-11-02T18:52:45.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c52a50c9a5ad277-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.768 DEBUG:boto:path=/ibtljn-61/obj87 2023-11-02T18:52:45.768 DEBUG:boto:auth_path=/ibtljn-61/obj87 2023-11-02T18:52:45.769 DEBUG:boto:Method: GET 2023-11-02T18:52:45.769 DEBUG:boto:Path: /ibtljn-61/obj87 2023-11-02T18:52:45.770 DEBUG:boto:Data: 2023-11-02T18:52:45.770 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.770 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.771 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.771 DEBUG:boto:Params: {} 2023-11-02T18:52:45.774 DEBUG:boto:Token: None 2023-11-02T18:52:45.775 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj87 2023-11-02T18:52:45.775 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:caZ3S9IBS27pOJanA1NbEN233j4= 2023-11-02T18:52:45.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:caZ3S9IBS27pOJanA1NbEN233j4=', 'Content-Length': '0'} 2023-11-02T18:52:45.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0563d5b9a57579b-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.826 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.827 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.828 DEBUG:boto:path=/ibtljn-61/?prefix=obj87 2023-11-02T18:52:45.828 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj87 2023-11-02T18:52:45.829 DEBUG:boto:Method: GET 2023-11-02T18:52:45.829 DEBUG:boto:Path: /ibtljn-61/?prefix=obj87 2023-11-02T18:52:45.830 DEBUG:boto:Data: 2023-11-02T18:52:45.830 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.831 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.831 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.832 DEBUG:boto:Params: {} 2023-11-02T18:52:45.833 DEBUG:boto:Token: None 2023-11-02T18:52:45.833 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.834 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006667ce4838922a18-006543effd-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.841 DEBUG:boto:b'ibtljn-61obj871000falseobj872023-11-02T18:50:24.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.842 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.843 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.843 DEBUG:boto:path=/ibtljn-61/?prefix=obj87 2023-11-02T18:52:45.845 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj87 2023-11-02T18:52:45.845 DEBUG:boto:Method: GET 2023-11-02T18:52:45.846 DEBUG:boto:Path: /ibtljn-61/?prefix=obj87 2023-11-02T18:52:45.846 DEBUG:boto:Data: 2023-11-02T18:52:45.847 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.847 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.847 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.848 DEBUG:boto:Params: {} 2023-11-02T18:52:45.848 DEBUG:boto:Token: None 2023-11-02T18:52:45.849 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.849 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053567e30d994e93e-006543effd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.861 DEBUG:boto:b'ibtljn-61obj871000falseobj872023-11-02T18:50:24.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:52:45.864 DEBUG:boto:path=/ibtljn-61/obj92 2023-11-02T18:52:45.864 DEBUG:boto:auth_path=/ibtljn-61/obj92 2023-11-02T18:52:45.865 DEBUG:boto:path=/ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.865 DEBUG:boto:auth_path=/ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.866 DEBUG:boto:Method: GET 2023-11-02T18:52:45.866 DEBUG:boto:Path: /ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.867 DEBUG:boto:Data: 2023-11-02T18:52:45.867 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.868 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.868 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.869 DEBUG:boto:Params: {} 2023-11-02T18:52:45.869 DEBUG:boto:Token: None 2023-11-02T18:52:45.870 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.870 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t8OLUjXiq6YF/cW2d+s6pYVYL48= 2023-11-02T18:52:45.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t8OLUjXiq6YF/cW2d+s6pYVYL48=', 'Content-Length': '0'} 2023-11-02T18:52:45.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f2c66583b374de5-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.874 DEBUG:boto:path=/ibtljn-61/obj92 2023-11-02T18:52:45.875 DEBUG:boto:auth_path=/ibtljn-61/obj92 2023-11-02T18:52:45.875 DEBUG:boto:path=/ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.876 DEBUG:boto:auth_path=/ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.876 DEBUG:boto:Method: GET 2023-11-02T18:52:45.877 DEBUG:boto:Path: /ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.877 DEBUG:boto:Data: 2023-11-02T18:52:45.878 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.878 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.879 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.879 DEBUG:boto:Params: {} 2023-11-02T18:52:45.880 DEBUG:boto:Token: None 2023-11-02T18:52:45.880 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.881 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t8OLUjXiq6YF/cW2d+s6pYVYL48= 2023-11-02T18:52:45.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t8OLUjXiq6YF/cW2d+s6pYVYL48=', 'Content-Length': '0'} 2023-11-02T18:52:45.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2f8b1147e8478c9-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.885 DEBUG:boto:path=/ibtljn-61/obj92 2023-11-02T18:52:45.885 DEBUG:boto:auth_path=/ibtljn-61/obj92 2023-11-02T18:52:45.886 DEBUG:boto:path=/ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.886 DEBUG:boto:auth_path=/ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.887 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.887 DEBUG:boto:Path: /ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.887 DEBUG:boto:Data: 2023-11-02T18:52:45.888 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.888 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.889 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.889 DEBUG:boto:Params: {} 2023-11-02T18:52:45.890 DEBUG:boto:Token: None 2023-11-02T18:52:45.890 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.891 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZCMIoJu2D6qGidgF4HuPGLwmIDo= 2023-11-02T18:52:45.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZCMIoJu2D6qGidgF4HuPGLwmIDo=', 'Content-Length': '0'} 2023-11-02T18:52:45.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dff09d7a5c26ab1a-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.894 DEBUG:boto:path=/ibtljn-61/obj92 2023-11-02T18:52:45.895 DEBUG:boto:auth_path=/ibtljn-61/obj92 2023-11-02T18:52:45.895 DEBUG:boto:path=/ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.896 DEBUG:boto:auth_path=/ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.896 DEBUG:boto:Method: HEAD 2023-11-02T18:52:45.897 DEBUG:boto:Path: /ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.897 DEBUG:boto:Data: 2023-11-02T18:52:45.898 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.898 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.898 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.899 DEBUG:boto:Params: {} 2023-11-02T18:52:45.899 DEBUG:boto:Token: None 2023-11-02T18:52:45.900 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.900 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZCMIoJu2D6qGidgF4HuPGLwmIDo= 2023-11-02T18:52:45.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZCMIoJu2D6qGidgF4HuPGLwmIDo=', 'Content-Length': '0'} 2023-11-02T18:52:45.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d55856185f9ad7e8-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:52:45.905 DEBUG:boto:path=/ibtljn-61/obj92 2023-11-02T18:52:45.905 DEBUG:boto:auth_path=/ibtljn-61/obj92 2023-11-02T18:52:45.906 DEBUG:boto:path=/ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.906 DEBUG:boto:auth_path=/ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.906 DEBUG:boto:Method: GET 2023-11-02T18:52:45.907 DEBUG:boto:Path: /ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.907 DEBUG:boto:Data: 2023-11-02T18:52:45.908 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.908 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.909 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.909 DEBUG:boto:Params: {} 2023-11-02T18:52:45.910 DEBUG:boto:Token: None 2023-11-02T18:52:45.910 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.911 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t8OLUjXiq6YF/cW2d+s6pYVYL48= 2023-11-02T18:52:45.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t8OLUjXiq6YF/cW2d+s6pYVYL48=', 'Content-Length': '0'} 2023-11-02T18:52:45.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a71e2c4a3f6563ff-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.916 DEBUG:boto:path=/ibtljn-61/obj92 2023-11-02T18:52:45.916 DEBUG:boto:auth_path=/ibtljn-61/obj92 2023-11-02T18:52:45.917 DEBUG:boto:path=/ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.917 DEBUG:boto:auth_path=/ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.918 DEBUG:boto:Method: GET 2023-11-02T18:52:45.918 DEBUG:boto:Path: /ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.919 DEBUG:boto:Data: 2023-11-02T18:52:45.919 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.920 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.920 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.920 DEBUG:boto:Params: {} 2023-11-02T18:52:45.921 DEBUG:boto:Token: None 2023-11-02T18:52:45.921 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/obj92?versionId=null 2023-11-02T18:52:45.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t8OLUjXiq6YF/cW2d+s6pYVYL48= 2023-11-02T18:52:45.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t8OLUjXiq6YF/cW2d+s6pYVYL48=', 'Content-Length': '0'} 2023-11-02T18:52:45.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005791da88f301378-006543effd-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.966 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.967 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.967 DEBUG:boto:path=/ibtljn-61/?prefix=obj92 2023-11-02T18:52:45.968 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj92 2023-11-02T18:52:45.968 DEBUG:boto:Method: GET 2023-11-02T18:52:45.969 DEBUG:boto:Path: /ibtljn-61/?prefix=obj92 2023-11-02T18:52:45.969 DEBUG:boto:Data: 2023-11-02T18:52:45.970 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.970 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:45.971 DEBUG:boto:Port: 8001 2023-11-02T18:52:45.971 DEBUG:boto:Params: {} 2023-11-02T18:52:45.972 DEBUG:boto:Token: None 2023-11-02T18:52:45.972 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.973 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ba9c65b3f96421a-006543effd-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.981 DEBUG:boto:b'ibtljn-61obj921000falseobj922023-11-02T18:50:23.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.981 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:45.982 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:45.982 DEBUG:boto:path=/ibtljn-61/?prefix=obj92 2023-11-02T18:52:45.983 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj92 2023-11-02T18:52:45.983 DEBUG:boto:Method: GET 2023-11-02T18:52:45.984 DEBUG:boto:Path: /ibtljn-61/?prefix=obj92 2023-11-02T18:52:45.984 DEBUG:boto:Data: 2023-11-02T18:52:45.985 DEBUG:boto:Headers: {} 2023-11-02T18:52:45.985 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:45.986 DEBUG:boto:Port: 8000 2023-11-02T18:52:45.986 DEBUG:boto:Params: {} 2023-11-02T18:52:45.987 DEBUG:boto:Token: None 2023-11-02T18:52:45.987 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:45 GMT /ibtljn-61/ 2023-11-02T18:52:45.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4= 2023-11-02T18:52:45.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wKSAB9Vtz91gBnpCDrJAeSI0Og4=', 'Content-Length': '0'} 2023-11-02T18:52:45.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3c3a59e14507f40-006543effd-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:45.997 DEBUG:boto:b'ibtljn-61obj921000falseobj922023-11-02T18:50:23.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:45.998 DEBUG:boto:path=/ibtljn-61/obj92 2023-11-02T18:52:45.999 DEBUG:boto:auth_path=/ibtljn-61/obj92 2023-11-02T18:52:45.999 DEBUG:boto:Method: HEAD 2023-11-02T18:52:46.000 DEBUG:boto:Path: /ibtljn-61/obj92 2023-11-02T18:52:46.000 DEBUG:boto:Data: 2023-11-02T18:52:46.000 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.001 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.001 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.002 DEBUG:boto:Params: {} 2023-11-02T18:52:46.002 DEBUG:boto:Token: None 2023-11-02T18:52:46.003 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj92 2023-11-02T18:52:46.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G0txVdfjR67mfSnyYmlnGxqaTT0= 2023-11-02T18:52:46.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G0txVdfjR67mfSnyYmlnGxqaTT0=', 'Content-Length': '0'} 2023-11-02T18:52:46.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000191c5d0a45cf9ff1-006543effd-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.006 DEBUG:boto:path=/ibtljn-61/obj92 2023-11-02T18:52:46.007 DEBUG:boto:auth_path=/ibtljn-61/obj92 2023-11-02T18:52:46.007 DEBUG:boto:Method: HEAD 2023-11-02T18:52:46.008 DEBUG:boto:Path: /ibtljn-61/obj92 2023-11-02T18:52:46.008 DEBUG:boto:Data: 2023-11-02T18:52:46.009 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.009 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.009 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.010 DEBUG:boto:Params: {} 2023-11-02T18:52:46.010 DEBUG:boto:Token: None 2023-11-02T18:52:46.011 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj92 2023-11-02T18:52:46.011 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G0txVdfjR67mfSnyYmlnGxqaTT0= 2023-11-02T18:52:46.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G0txVdfjR67mfSnyYmlnGxqaTT0=', 'Content-Length': '0'} 2023-11-02T18:52:46.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900a3826ddb28f90-006543effe-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T18:52:46.015 DEBUG:boto:path=/ibtljn-61/obj92 2023-11-02T18:52:46.016 DEBUG:boto:auth_path=/ibtljn-61/obj92 2023-11-02T18:52:46.016 DEBUG:boto:Method: GET 2023-11-02T18:52:46.017 DEBUG:boto:Path: /ibtljn-61/obj92 2023-11-02T18:52:46.017 DEBUG:boto:Data: 2023-11-02T18:52:46.018 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.018 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.018 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.019 DEBUG:boto:Params: {} 2023-11-02T18:52:46.019 DEBUG:boto:Token: None 2023-11-02T18:52:46.020 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj92 2023-11-02T18:52:46.020 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OWIsp1tGr2bwRPnSTwfKN7sscfE= 2023-11-02T18:52:46.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OWIsp1tGr2bwRPnSTwfKN7sscfE=', 'Content-Length': '0'} 2023-11-02T18:52:46.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d007923682058d20-006543effe-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.066 DEBUG:boto:path=/ibtljn-61/obj92 2023-11-02T18:52:46.067 DEBUG:boto:auth_path=/ibtljn-61/obj92 2023-11-02T18:52:46.067 DEBUG:boto:Method: GET 2023-11-02T18:52:46.068 DEBUG:boto:Path: /ibtljn-61/obj92 2023-11-02T18:52:46.068 DEBUG:boto:Data: 2023-11-02T18:52:46.069 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.069 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.070 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.070 DEBUG:boto:Params: {} 2023-11-02T18:52:46.071 DEBUG:boto:Token: None 2023-11-02T18:52:46.071 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj92 2023-11-02T18:52:46.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OWIsp1tGr2bwRPnSTwfKN7sscfE= 2023-11-02T18:52:46.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OWIsp1tGr2bwRPnSTwfKN7sscfE=', 'Content-Length': '0'} 2023-11-02T18:52:46.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e1149c9a9c2234b-006543effe-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.076 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:46.077 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:46.077 DEBUG:boto:path=/ibtljn-61/?prefix=obj92 2023-11-02T18:52:46.078 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj92 2023-11-02T18:52:46.078 DEBUG:boto:Method: GET 2023-11-02T18:52:46.078 DEBUG:boto:Path: /ibtljn-61/?prefix=obj92 2023-11-02T18:52:46.079 DEBUG:boto:Data: 2023-11-02T18:52:46.080 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.080 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.080 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.081 DEBUG:boto:Params: {} 2023-11-02T18:52:46.081 DEBUG:boto:Token: None 2023-11-02T18:52:46.083 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/ 2023-11-02T18:52:46.083 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM= 2023-11-02T18:52:46.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM=', 'Content-Length': '0'} 2023-11-02T18:52:46.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d349a6fdc3e612b-006543effe-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.090 DEBUG:boto:b'ibtljn-61obj921000falseobj922023-11-02T18:50:23.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:46.091 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:46.092 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:46.092 DEBUG:boto:path=/ibtljn-61/?prefix=obj92 2023-11-02T18:52:46.093 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj92 2023-11-02T18:52:46.093 DEBUG:boto:Method: GET 2023-11-02T18:52:46.094 DEBUG:boto:Path: /ibtljn-61/?prefix=obj92 2023-11-02T18:52:46.094 DEBUG:boto:Data: 2023-11-02T18:52:46.095 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.095 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.095 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.096 DEBUG:boto:Params: {} 2023-11-02T18:52:46.096 DEBUG:boto:Token: None 2023-11-02T18:52:46.097 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/ 2023-11-02T18:52:46.098 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM= 2023-11-02T18:52:46.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM=', 'Content-Length': '0'} 2023-11-02T18:52:46.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000926073aa2b13ced3-006543effe-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.154 DEBUG:boto:b'ibtljn-61obj921000falseobj922023-11-02T18:50:23.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:46.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T18:52:46.156 DEBUG:boto:path=/ibtljn-61/obj93 2023-11-02T18:52:46.156 DEBUG:boto:auth_path=/ibtljn-61/obj93 2023-11-02T18:52:46.157 DEBUG:boto:path=/ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.157 DEBUG:boto:auth_path=/ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.158 DEBUG:boto:Method: GET 2023-11-02T18:52:46.158 DEBUG:boto:Path: /ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.159 DEBUG:boto:Data: 2023-11-02T18:52:46.159 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.159 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.160 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.160 DEBUG:boto:Params: {} 2023-11-02T18:52:46.161 DEBUG:boto:Token: None 2023-11-02T18:52:46.161 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.162 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QvJfhipXb+JcLBk8FnqIk6V6pKg= 2023-11-02T18:52:46.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QvJfhipXb+JcLBk8FnqIk6V6pKg=', 'Content-Length': '0'} 2023-11-02T18:52:46.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f32c0bed54d9c6b8-006543effe-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.166 DEBUG:boto:path=/ibtljn-61/obj93 2023-11-02T18:52:46.167 DEBUG:boto:auth_path=/ibtljn-61/obj93 2023-11-02T18:52:46.167 DEBUG:boto:path=/ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.168 DEBUG:boto:auth_path=/ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.168 DEBUG:boto:Method: GET 2023-11-02T18:52:46.168 DEBUG:boto:Path: /ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.169 DEBUG:boto:Data: 2023-11-02T18:52:46.169 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.170 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.170 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.171 DEBUG:boto:Params: {} 2023-11-02T18:52:46.171 DEBUG:boto:Token: None 2023-11-02T18:52:46.172 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.172 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QvJfhipXb+JcLBk8FnqIk6V6pKg= 2023-11-02T18:52:46.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QvJfhipXb+JcLBk8FnqIk6V6pKg=', 'Content-Length': '0'} 2023-11-02T18:52:46.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b0b1321dbfe3ace-006543effe-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.176 DEBUG:boto:path=/ibtljn-61/obj93 2023-11-02T18:52:46.177 DEBUG:boto:auth_path=/ibtljn-61/obj93 2023-11-02T18:52:46.177 DEBUG:boto:path=/ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.178 DEBUG:boto:auth_path=/ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.178 DEBUG:boto:Method: HEAD 2023-11-02T18:52:46.179 DEBUG:boto:Path: /ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.179 DEBUG:boto:Data: 2023-11-02T18:52:46.180 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.182 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.182 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.182 DEBUG:boto:Params: {} 2023-11-02T18:52:46.183 DEBUG:boto:Token: None 2023-11-02T18:52:46.184 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.184 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xT9Cb3VPVngAPSy89KnbZQwAIBw= 2023-11-02T18:52:46.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xT9Cb3VPVngAPSy89KnbZQwAIBw=', 'Content-Length': '0'} 2023-11-02T18:52:46.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000137d28b5ae02fa9a-006543effe-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.188 DEBUG:boto:path=/ibtljn-61/obj93 2023-11-02T18:52:46.189 DEBUG:boto:auth_path=/ibtljn-61/obj93 2023-11-02T18:52:46.189 DEBUG:boto:path=/ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.189 DEBUG:boto:auth_path=/ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.190 DEBUG:boto:Method: HEAD 2023-11-02T18:52:46.190 DEBUG:boto:Path: /ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.191 DEBUG:boto:Data: 2023-11-02T18:52:46.191 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.192 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.192 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.193 DEBUG:boto:Params: {} 2023-11-02T18:52:46.193 DEBUG:boto:Token: None 2023-11-02T18:52:46.193 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.194 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xT9Cb3VPVngAPSy89KnbZQwAIBw= 2023-11-02T18:52:46.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xT9Cb3VPVngAPSy89KnbZQwAIBw=', 'Content-Length': '0'} 2023-11-02T18:52:46.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd8d04ddb44475ea-006543effe-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T18:52:46.199 DEBUG:boto:path=/ibtljn-61/obj93 2023-11-02T18:52:46.199 DEBUG:boto:auth_path=/ibtljn-61/obj93 2023-11-02T18:52:46.200 DEBUG:boto:path=/ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.200 DEBUG:boto:auth_path=/ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.201 DEBUG:boto:Method: GET 2023-11-02T18:52:46.201 DEBUG:boto:Path: /ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.202 DEBUG:boto:Data: 2023-11-02T18:52:46.202 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.203 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.203 DEBUG:boto:Params: {} 2023-11-02T18:52:46.204 DEBUG:boto:Token: None 2023-11-02T18:52:46.204 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.205 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QvJfhipXb+JcLBk8FnqIk6V6pKg= 2023-11-02T18:52:46.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QvJfhipXb+JcLBk8FnqIk6V6pKg=', 'Content-Length': '0'} 2023-11-02T18:52:46.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca9b6435153576a1-006543effe-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.209 DEBUG:boto:path=/ibtljn-61/obj93 2023-11-02T18:52:46.209 DEBUG:boto:auth_path=/ibtljn-61/obj93 2023-11-02T18:52:46.210 DEBUG:boto:path=/ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.210 DEBUG:boto:auth_path=/ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.211 DEBUG:boto:Method: GET 2023-11-02T18:52:46.211 DEBUG:boto:Path: /ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.212 DEBUG:boto:Data: 2023-11-02T18:52:46.212 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.213 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.213 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.213 DEBUG:boto:Params: {} 2023-11-02T18:52:46.214 DEBUG:boto:Token: None 2023-11-02T18:52:46.214 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj93?versionId=null 2023-11-02T18:52:46.215 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QvJfhipXb+JcLBk8FnqIk6V6pKg= 2023-11-02T18:52:46.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QvJfhipXb+JcLBk8FnqIk6V6pKg=', 'Content-Length': '0'} 2023-11-02T18:52:46.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdc0a1286c2a25a0-006543effe-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.219 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:46.220 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:46.221 DEBUG:boto:path=/ibtljn-61/?prefix=obj93 2023-11-02T18:52:46.221 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj93 2023-11-02T18:52:46.221 DEBUG:boto:Method: GET 2023-11-02T18:52:46.222 DEBUG:boto:Path: /ibtljn-61/?prefix=obj93 2023-11-02T18:52:46.222 DEBUG:boto:Data: 2023-11-02T18:52:46.223 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.224 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.224 DEBUG:boto:Params: {} 2023-11-02T18:52:46.225 DEBUG:boto:Token: None 2023-11-02T18:52:46.225 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/ 2023-11-02T18:52:46.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM= 2023-11-02T18:52:46.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM=', 'Content-Length': '0'} 2023-11-02T18:52:46.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090f09eac06c241ee-006543effe-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.274 DEBUG:boto:b'ibtljn-61obj931000falseobj932023-11-02T18:50:24.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:46.275 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:46.276 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:46.276 DEBUG:boto:path=/ibtljn-61/?prefix=obj93 2023-11-02T18:52:46.277 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj93 2023-11-02T18:52:46.277 DEBUG:boto:Method: GET 2023-11-02T18:52:46.278 DEBUG:boto:Path: /ibtljn-61/?prefix=obj93 2023-11-02T18:52:46.278 DEBUG:boto:Data: 2023-11-02T18:52:46.278 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.279 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.279 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.280 DEBUG:boto:Params: {} 2023-11-02T18:52:46.280 DEBUG:boto:Token: None 2023-11-02T18:52:46.281 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/ 2023-11-02T18:52:46.281 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM= 2023-11-02T18:52:46.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM=', 'Content-Length': '0'} 2023-11-02T18:52:46.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cf7d8bc738ee46a-006543effe-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.293 DEBUG:boto:b'ibtljn-61obj931000falseobj932023-11-02T18:50:24.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:46.294 DEBUG:boto:path=/ibtljn-61/obj93 2023-11-02T18:52:46.294 DEBUG:boto:auth_path=/ibtljn-61/obj93 2023-11-02T18:52:46.295 DEBUG:boto:Method: HEAD 2023-11-02T18:52:46.295 DEBUG:boto:Path: /ibtljn-61/obj93 2023-11-02T18:52:46.295 DEBUG:boto:Data: 2023-11-02T18:52:46.296 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.296 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.297 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.297 DEBUG:boto:Params: {} 2023-11-02T18:52:46.298 DEBUG:boto:Token: None 2023-11-02T18:52:46.298 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj93 2023-11-02T18:52:46.299 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hrF6gHKocYWp+SuSZA+/FZ7cULk= 2023-11-02T18:52:46.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hrF6gHKocYWp+SuSZA+/FZ7cULk=', 'Content-Length': '0'} 2023-11-02T18:52:46.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007998a6c8437a946d-006543effe-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.302 DEBUG:boto:path=/ibtljn-61/obj93 2023-11-02T18:52:46.303 DEBUG:boto:auth_path=/ibtljn-61/obj93 2023-11-02T18:52:46.303 DEBUG:boto:Method: HEAD 2023-11-02T18:52:46.304 DEBUG:boto:Path: /ibtljn-61/obj93 2023-11-02T18:52:46.304 DEBUG:boto:Data: 2023-11-02T18:52:46.304 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.305 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.305 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.306 DEBUG:boto:Params: {} 2023-11-02T18:52:46.306 DEBUG:boto:Token: None 2023-11-02T18:52:46.307 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj93 2023-11-02T18:52:46.307 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hrF6gHKocYWp+SuSZA+/FZ7cULk= 2023-11-02T18:52:46.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hrF6gHKocYWp+SuSZA+/FZ7cULk=', 'Content-Length': '0'} 2023-11-02T18:52:46.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069806bc236452089-006543effe-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T18:52:46.312 DEBUG:boto:path=/ibtljn-61/obj93 2023-11-02T18:52:46.312 DEBUG:boto:auth_path=/ibtljn-61/obj93 2023-11-02T18:52:46.313 DEBUG:boto:Method: GET 2023-11-02T18:52:46.313 DEBUG:boto:Path: /ibtljn-61/obj93 2023-11-02T18:52:46.314 DEBUG:boto:Data: 2023-11-02T18:52:46.314 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.315 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.315 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.316 DEBUG:boto:Params: {} 2023-11-02T18:52:46.316 DEBUG:boto:Token: None 2023-11-02T18:52:46.317 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj93 2023-11-02T18:52:46.317 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:utxUtoOhMjXaYZzYDOhMVl/5nAM= 2023-11-02T18:52:46.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:utxUtoOhMjXaYZzYDOhMVl/5nAM=', 'Content-Length': '0'} 2023-11-02T18:52:46.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b90b377c3ab0915-006543effe-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.321 DEBUG:boto:path=/ibtljn-61/obj93 2023-11-02T18:52:46.322 DEBUG:boto:auth_path=/ibtljn-61/obj93 2023-11-02T18:52:46.323 DEBUG:boto:Method: GET 2023-11-02T18:52:46.324 DEBUG:boto:Path: /ibtljn-61/obj93 2023-11-02T18:52:46.325 DEBUG:boto:Data: 2023-11-02T18:52:46.325 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.326 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.326 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.327 DEBUG:boto:Params: {} 2023-11-02T18:52:46.327 DEBUG:boto:Token: None 2023-11-02T18:52:46.328 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj93 2023-11-02T18:52:46.328 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:utxUtoOhMjXaYZzYDOhMVl/5nAM= 2023-11-02T18:52:46.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:utxUtoOhMjXaYZzYDOhMVl/5nAM=', 'Content-Length': '0'} 2023-11-02T18:52:46.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef8122544fe8f1bb-006543effe-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.378 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:46.379 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:46.380 DEBUG:boto:path=/ibtljn-61/?prefix=obj93 2023-11-02T18:52:46.380 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj93 2023-11-02T18:52:46.380 DEBUG:boto:Method: GET 2023-11-02T18:52:46.381 DEBUG:boto:Path: /ibtljn-61/?prefix=obj93 2023-11-02T18:52:46.381 DEBUG:boto:Data: 2023-11-02T18:52:46.382 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.382 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.383 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.383 DEBUG:boto:Params: {} 2023-11-02T18:52:46.384 DEBUG:boto:Token: None 2023-11-02T18:52:46.384 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/ 2023-11-02T18:52:46.385 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM= 2023-11-02T18:52:46.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM=', 'Content-Length': '0'} 2023-11-02T18:52:46.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbc34cb776b98573-006543effe-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.391 DEBUG:boto:b'ibtljn-61obj931000falseobj932023-11-02T18:50:24.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:46.392 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:46.393 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:46.393 DEBUG:boto:path=/ibtljn-61/?prefix=obj93 2023-11-02T18:52:46.394 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj93 2023-11-02T18:52:46.394 DEBUG:boto:Method: GET 2023-11-02T18:52:46.395 DEBUG:boto:Path: /ibtljn-61/?prefix=obj93 2023-11-02T18:52:46.395 DEBUG:boto:Data: 2023-11-02T18:52:46.396 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.396 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.397 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.397 DEBUG:boto:Params: {} 2023-11-02T18:52:46.398 DEBUG:boto:Token: None 2023-11-02T18:52:46.398 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/ 2023-11-02T18:52:46.399 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM= 2023-11-02T18:52:46.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM=', 'Content-Length': '0'} 2023-11-02T18:52:46.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccd845cee2407d71-006543effe-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.410 DEBUG:boto:b'ibtljn-61obj931000falseobj932023-11-02T18:50:24.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:46.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:52:46.411 DEBUG:boto:path=/ibtljn-61/obj95 2023-11-02T18:52:46.411 DEBUG:boto:auth_path=/ibtljn-61/obj95 2023-11-02T18:52:46.412 DEBUG:boto:path=/ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.412 DEBUG:boto:auth_path=/ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.413 DEBUG:boto:Method: GET 2023-11-02T18:52:46.413 DEBUG:boto:Path: /ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.414 DEBUG:boto:Data: 2023-11-02T18:52:46.414 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.415 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.415 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.416 DEBUG:boto:Params: {} 2023-11-02T18:52:46.416 DEBUG:boto:Token: None 2023-11-02T18:52:46.417 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.417 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RIbWFMWRMPG2QE2YnaV7v3dPfZg= 2023-11-02T18:52:46.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RIbWFMWRMPG2QE2YnaV7v3dPfZg=', 'Content-Length': '0'} 2023-11-02T18:52:46.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e52f3dc26c461ef1-006543effe-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.421 DEBUG:boto:path=/ibtljn-61/obj95 2023-11-02T18:52:46.422 DEBUG:boto:auth_path=/ibtljn-61/obj95 2023-11-02T18:52:46.422 DEBUG:boto:path=/ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.423 DEBUG:boto:auth_path=/ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.423 DEBUG:boto:Method: GET 2023-11-02T18:52:46.424 DEBUG:boto:Path: /ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.424 DEBUG:boto:Data: 2023-11-02T18:52:46.424 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.425 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.425 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.426 DEBUG:boto:Params: {} 2023-11-02T18:52:46.426 DEBUG:boto:Token: None 2023-11-02T18:52:46.427 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.427 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RIbWFMWRMPG2QE2YnaV7v3dPfZg= 2023-11-02T18:52:46.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RIbWFMWRMPG2QE2YnaV7v3dPfZg=', 'Content-Length': '0'} 2023-11-02T18:52:46.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc401b4b3882549c-006543effe-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.431 DEBUG:boto:path=/ibtljn-61/obj95 2023-11-02T18:52:46.431 DEBUG:boto:auth_path=/ibtljn-61/obj95 2023-11-02T18:52:46.432 DEBUG:boto:path=/ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.432 DEBUG:boto:auth_path=/ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.433 DEBUG:boto:Method: HEAD 2023-11-02T18:52:46.433 DEBUG:boto:Path: /ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.434 DEBUG:boto:Data: 2023-11-02T18:52:46.434 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.435 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.435 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.436 DEBUG:boto:Params: {} 2023-11-02T18:52:46.436 DEBUG:boto:Token: None 2023-11-02T18:52:46.437 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.437 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lJuYORkV3qaDD/uEBOmBqRdZwHQ= 2023-11-02T18:52:46.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lJuYORkV3qaDD/uEBOmBqRdZwHQ=', 'Content-Length': '0'} 2023-11-02T18:52:46.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077f1cbfbe748ae1e-006543effe-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.441 DEBUG:boto:path=/ibtljn-61/obj95 2023-11-02T18:52:46.442 DEBUG:boto:auth_path=/ibtljn-61/obj95 2023-11-02T18:52:46.442 DEBUG:boto:path=/ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.443 DEBUG:boto:auth_path=/ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.443 DEBUG:boto:Method: HEAD 2023-11-02T18:52:46.443 DEBUG:boto:Path: /ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.444 DEBUG:boto:Data: 2023-11-02T18:52:46.444 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.445 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.445 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.446 DEBUG:boto:Params: {} 2023-11-02T18:52:46.446 DEBUG:boto:Token: None 2023-11-02T18:52:46.447 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.447 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lJuYORkV3qaDD/uEBOmBqRdZwHQ= 2023-11-02T18:52:46.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lJuYORkV3qaDD/uEBOmBqRdZwHQ=', 'Content-Length': '0'} 2023-11-02T18:52:46.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baceb51ec67204c4-006543effe-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:52:46.452 DEBUG:boto:path=/ibtljn-61/obj95 2023-11-02T18:52:46.452 DEBUG:boto:auth_path=/ibtljn-61/obj95 2023-11-02T18:52:46.453 DEBUG:boto:path=/ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.453 DEBUG:boto:auth_path=/ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.454 DEBUG:boto:Method: GET 2023-11-02T18:52:46.454 DEBUG:boto:Path: /ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.455 DEBUG:boto:Data: 2023-11-02T18:52:46.455 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.456 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.456 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.457 DEBUG:boto:Params: {} 2023-11-02T18:52:46.457 DEBUG:boto:Token: None 2023-11-02T18:52:46.458 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.458 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RIbWFMWRMPG2QE2YnaV7v3dPfZg= 2023-11-02T18:52:46.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RIbWFMWRMPG2QE2YnaV7v3dPfZg=', 'Content-Length': '0'} 2023-11-02T18:52:46.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002378dc8f99cdd7a9-006543effe-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.463 DEBUG:boto:path=/ibtljn-61/obj95 2023-11-02T18:52:46.463 DEBUG:boto:auth_path=/ibtljn-61/obj95 2023-11-02T18:52:46.464 DEBUG:boto:path=/ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.464 DEBUG:boto:auth_path=/ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.465 DEBUG:boto:Method: GET 2023-11-02T18:52:46.465 DEBUG:boto:Path: /ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.466 DEBUG:boto:Data: 2023-11-02T18:52:46.466 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.467 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.467 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.468 DEBUG:boto:Params: {} 2023-11-02T18:52:46.468 DEBUG:boto:Token: None 2023-11-02T18:52:46.469 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj95?versionId=null 2023-11-02T18:52:46.469 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RIbWFMWRMPG2QE2YnaV7v3dPfZg= 2023-11-02T18:52:46.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RIbWFMWRMPG2QE2YnaV7v3dPfZg=', 'Content-Length': '0'} 2023-11-02T18:52:46.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000956ffa537f797b76-006543effe-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.518 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:46.519 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:46.519 DEBUG:boto:path=/ibtljn-61/?prefix=obj95 2023-11-02T18:52:46.520 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj95 2023-11-02T18:52:46.520 DEBUG:boto:Method: GET 2023-11-02T18:52:46.521 DEBUG:boto:Path: /ibtljn-61/?prefix=obj95 2023-11-02T18:52:46.521 DEBUG:boto:Data: 2023-11-02T18:52:46.522 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.523 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.523 DEBUG:boto:Params: {} 2023-11-02T18:52:46.524 DEBUG:boto:Token: None 2023-11-02T18:52:46.524 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/ 2023-11-02T18:52:46.525 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM= 2023-11-02T18:52:46.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM=', 'Content-Length': '0'} 2023-11-02T18:52:46.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f552ee16d9d2656c-006543effe-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.532 DEBUG:boto:b'ibtljn-61obj951000falseobj952023-11-02T18:50:23.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:46.533 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:46.533 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:46.534 DEBUG:boto:path=/ibtljn-61/?prefix=obj95 2023-11-02T18:52:46.534 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj95 2023-11-02T18:52:46.534 DEBUG:boto:Method: GET 2023-11-02T18:52:46.535 DEBUG:boto:Path: /ibtljn-61/?prefix=obj95 2023-11-02T18:52:46.535 DEBUG:boto:Data: 2023-11-02T18:52:46.536 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.536 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.537 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.537 DEBUG:boto:Params: {} 2023-11-02T18:52:46.538 DEBUG:boto:Token: None 2023-11-02T18:52:46.539 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/ 2023-11-02T18:52:46.539 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM= 2023-11-02T18:52:46.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM=', 'Content-Length': '0'} 2023-11-02T18:52:46.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e55950dd00509e0-006543effe-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.550 DEBUG:boto:b'ibtljn-61obj951000falseobj952023-11-02T18:50:23.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:46.550 DEBUG:boto:path=/ibtljn-61/obj95 2023-11-02T18:52:46.551 DEBUG:boto:auth_path=/ibtljn-61/obj95 2023-11-02T18:52:46.552 DEBUG:boto:Method: HEAD 2023-11-02T18:52:46.552 DEBUG:boto:Path: /ibtljn-61/obj95 2023-11-02T18:52:46.552 DEBUG:boto:Data: 2023-11-02T18:52:46.553 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.553 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.554 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.554 DEBUG:boto:Params: {} 2023-11-02T18:52:46.555 DEBUG:boto:Token: None 2023-11-02T18:52:46.555 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj95 2023-11-02T18:52:46.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Iru+pHd/kyEOjYq1FqaL5zo1BRI= 2023-11-02T18:52:46.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Iru+pHd/kyEOjYq1FqaL5zo1BRI=', 'Content-Length': '0'} 2023-11-02T18:52:46.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f7b957a3736f4cb-006543effe-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.559 DEBUG:boto:path=/ibtljn-61/obj95 2023-11-02T18:52:46.560 DEBUG:boto:auth_path=/ibtljn-61/obj95 2023-11-02T18:52:46.560 DEBUG:boto:Method: HEAD 2023-11-02T18:52:46.561 DEBUG:boto:Path: /ibtljn-61/obj95 2023-11-02T18:52:46.561 DEBUG:boto:Data: 2023-11-02T18:52:46.562 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.562 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.563 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.563 DEBUG:boto:Params: {} 2023-11-02T18:52:46.564 DEBUG:boto:Token: None 2023-11-02T18:52:46.564 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj95 2023-11-02T18:52:46.565 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Iru+pHd/kyEOjYq1FqaL5zo1BRI= 2023-11-02T18:52:46.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Iru+pHd/kyEOjYq1FqaL5zo1BRI=', 'Content-Length': '0'} 2023-11-02T18:52:46.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091759c0f3add6bba-006543effe-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T18:52:46.569 DEBUG:boto:path=/ibtljn-61/obj95 2023-11-02T18:52:46.569 DEBUG:boto:auth_path=/ibtljn-61/obj95 2023-11-02T18:52:46.570 DEBUG:boto:Method: GET 2023-11-02T18:52:46.570 DEBUG:boto:Path: /ibtljn-61/obj95 2023-11-02T18:52:46.571 DEBUG:boto:Data: 2023-11-02T18:52:46.571 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.572 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.572 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.573 DEBUG:boto:Params: {} 2023-11-02T18:52:46.574 DEBUG:boto:Token: None 2023-11-02T18:52:46.574 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj95 2023-11-02T18:52:46.575 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XedIWVluvDEHt2JopfQTPoePwzY= 2023-11-02T18:52:46.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XedIWVluvDEHt2JopfQTPoePwzY=', 'Content-Length': '0'} 2023-11-02T18:52:46.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-61:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b08b07e72d3e400c-006543effe-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.622 DEBUG:boto:path=/ibtljn-61/obj95 2023-11-02T18:52:46.623 DEBUG:boto:auth_path=/ibtljn-61/obj95 2023-11-02T18:52:46.624 DEBUG:boto:Method: GET 2023-11-02T18:52:46.624 DEBUG:boto:Path: /ibtljn-61/obj95 2023-11-02T18:52:46.624 DEBUG:boto:Data: 2023-11-02T18:52:46.625 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.625 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.626 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.626 DEBUG:boto:Params: {} 2023-11-02T18:52:46.627 DEBUG:boto:Token: None 2023-11-02T18:52:46.627 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/obj95 2023-11-02T18:52:46.628 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XedIWVluvDEHt2JopfQTPoePwzY= 2023-11-02T18:52:46.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XedIWVluvDEHt2JopfQTPoePwzY=', 'Content-Length': '0'} 2023-11-02T18:52:46.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:50:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de5e43d283f7c6ed-006543effe-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.633 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:46.633 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:46.634 DEBUG:boto:path=/ibtljn-61/?prefix=obj95 2023-11-02T18:52:46.634 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj95 2023-11-02T18:52:46.635 DEBUG:boto:Method: GET 2023-11-02T18:52:46.635 DEBUG:boto:Path: /ibtljn-61/?prefix=obj95 2023-11-02T18:52:46.636 DEBUG:boto:Data: 2023-11-02T18:52:46.636 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:52:46.637 DEBUG:boto:Port: 8001 2023-11-02T18:52:46.638 DEBUG:boto:Params: {} 2023-11-02T18:52:46.638 DEBUG:boto:Token: None 2023-11-02T18:52:46.639 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/ 2023-11-02T18:52:46.639 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM= 2023-11-02T18:52:46.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM=', 'Content-Length': '0'} 2023-11-02T18:52:46.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021730d2b8affb6ae-006543effe-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.653 DEBUG:boto:b'ibtljn-61obj951000falseobj952023-11-02T18:50:23.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:46.654 DEBUG:boto:path=/ibtljn-61/ 2023-11-02T18:52:46.655 DEBUG:boto:auth_path=/ibtljn-61/ 2023-11-02T18:52:46.655 DEBUG:boto:path=/ibtljn-61/?prefix=obj95 2023-11-02T18:52:46.655 DEBUG:boto:auth_path=/ibtljn-61/?prefix=obj95 2023-11-02T18:52:46.656 DEBUG:boto:Method: GET 2023-11-02T18:52:46.656 DEBUG:boto:Path: /ibtljn-61/?prefix=obj95 2023-11-02T18:52:46.657 DEBUG:boto:Data: 2023-11-02T18:52:46.657 DEBUG:boto:Headers: {} 2023-11-02T18:52:46.658 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:46.658 DEBUG:boto:Port: 8000 2023-11-02T18:52:46.659 DEBUG:boto:Params: {} 2023-11-02T18:52:46.659 DEBUG:boto:Token: None 2023-11-02T18:52:46.660 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:52:46 GMT /ibtljn-61/ 2023-11-02T18:52:46.660 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM= 2023-11-02T18:52:46.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JxNb/742erU/aLKR8EmJA6LFUGM=', 'Content-Length': '0'} 2023-11-02T18:52:46.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fad81f0a1a5f2bf9-006543effe-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:46.710 DEBUG:boto:b'ibtljn-61obj951000falseobj952023-11-02T18:50:23.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:52:46.711 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-61 zones={a2, a1} 2023-11-02T18:52:46.712 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-11-02T18:52:46.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:52:46.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:52:46.713 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:52:46.759 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:52:46.760 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:52:46.863 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:52:46.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:46.864 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000023282", 2023-11-02T18:52:46.864 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:50:37.390664Z" 2023-11-02T18:52:46.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:46.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:46.865 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000037104", 2023-11-02T18:52:46.865 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:50:25.168287Z" 2023-11-02T18:52:46.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:46.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:46.866 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032867", 2023-11-02T18:52:46.866 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:50:37.394250Z" 2023-11-02T18:52:46.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:46.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:46.867 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032346", 2023-11-02T18:52:46.867 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:50:37.397815Z" 2023-11-02T18:52:46.867 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:46.868 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:52:46.869 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000023282', 1: '00000000000000000000:00000000000000037104', 2: '00000000000000000000:00000000000000032867', 3: '00000000000000000000:00000000000000032346'} 2023-11-02T18:52:46.869 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:52:46.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:52:46.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:52:46.871 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:52:46.919 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:52:46.920 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:52:47.009 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:52:47.009 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:52:47.009 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:52:47.009 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:52:47.010 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:52:47.010 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 10720254151908276686 2023-11-02T18:52:47.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.010 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:52:47.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.011 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:52:47.011 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:52:47.011 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:52:47.012 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000023282", 2023-11-02T18:52:47.012 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:52:47.012 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T18:52:47.012 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:52:47.012 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:50:37.390664Z" 2023-11-02T18:52:47.013 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.013 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:52:47.013 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:52:47.014 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:52:47.014 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000037104", 2023-11-02T18:52:47.014 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:52:47.015 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 52, 2023-11-02T18:52:47.015 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:52:47.015 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:50:25.168287Z" 2023-11-02T18:52:47.015 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.016 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:52:47.016 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:52:47.016 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:52:47.017 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032867", 2023-11-02T18:52:47.017 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:52:47.017 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:52:47.017 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:52:47.017 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:50:37.394250Z" 2023-11-02T18:52:47.018 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.019 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:52:47.020 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:52:47.020 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:52:47.020 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032346", 2023-11-02T18:52:47.020 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:52:47.021 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:52:47.021 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:52:47.021 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:50:37.397815Z" 2023-11-02T18:52:47.021 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.021 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.022 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:52:47.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.022 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:52:47.022 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 61, 2023-11-02T18:52:47.023 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 61 2023-11-02T18:52:47.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.023 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:52:46Z" 2023-11-02T18:52:47.023 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:52:47.024 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10720254151908276686 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023282", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T18:50:37.390664Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037104", "next_step_marker": "", "total_entries": 52, "pos": 0, "timestamp": "2023-11-02T18:50:25.168287Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032867", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:50:37.394250Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032346", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:50:37.397815Z" } } ] }, "full_sync": { "total": 61, "complete": 61 }, "current_time": "2023-11-02T18:52:46Z" } 2023-11-02T18:52:47.025 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023282', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T18:50:37.390664Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037104', 'next_step_marker': '', 'total_entries': 52, 'pos': 0, 'timestamp': '2023-11-02T18:50:25.168287Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032867', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:50:37.394250Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032346', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:50:37.397815Z'}}] 2023-11-02T18:52:47.025 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000023282', 1: '00000000000000000000:00000000000000037104', 2: '00000000000000000000:00000000000000032867', 3: '00000000000000000000:00000000000000032346'} 2023-11-02T18:52:47.026 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000023282', 1: '00000000000000000000:00000000000000037104', 2: '00000000000000000000:00000000000000032867', 3: '00000000000000000000:00000000000000032346'} 2023-11-02T18:52:47.026 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:52:47.026 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-11-02T18:52:47.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:52:47.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:52:47.028 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:52:47.074 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:52:47.075 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:52:47.175 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:52:47.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.176 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000005723", 2023-11-02T18:52:47.176 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:52:15.451244Z" 2023-11-02T18:52:47.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.177 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000014883", 2023-11-02T18:52:47.177 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:52:15.454995Z" 2023-11-02T18:52:47.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.178 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000013762", 2023-11-02T18:52:47.178 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:52:15.458545Z" 2023-11-02T18:52:47.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.179 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000011253", 2023-11-02T18:52:47.179 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:52:15.461992Z" 2023-11-02T18:52:47.179 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.179 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:52:47.181 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000014883', 2: '00000000000000000000:00000000000000013762', 3: '00000000000000000000:00000000000000011253'} 2023-11-02T18:52:47.181 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:52:47.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:52:47.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:52:47.183 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:52:47.232 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:52:47.232 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:52:47.322 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:52:47.322 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:52:47.323 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:52:47.323 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:52:47.323 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:52:47.323 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:52:47.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.324 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:52:47.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.324 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:52:47.325 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:52:47.325 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:52:47.325 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000005723", 2023-11-02T18:52:47.325 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:52:47.326 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:52:47.326 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:52:47.326 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:52:15.451244Z" 2023-11-02T18:52:47.326 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.327 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:52:47.327 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:52:47.327 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:52:47.328 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000014883", 2023-11-02T18:52:47.328 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:52:47.328 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:52:47.328 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:52:47.329 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:52:15.454995Z" 2023-11-02T18:52:47.329 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.330 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:52:47.330 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:52:47.330 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:52:47.330 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000013762", 2023-11-02T18:52:47.331 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:52:47.331 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:52:47.331 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:52:47.331 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:52:15.458545Z" 2023-11-02T18:52:47.332 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.332 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:52:47.333 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:52:47.333 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:52:47.333 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000011253", 2023-11-02T18:52:47.333 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:52:47.334 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:52:47.334 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:52:47.334 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:52:15.461992Z" 2023-11-02T18:52:47.334 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.335 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.335 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:52:47.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.335 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:52:47.336 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:52:47.336 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:52:47.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.336 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:52:47Z" 2023-11-02T18:52:47.337 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:52:47.339 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:52:15.451244Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014883", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:52:15.454995Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013762", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:52:15.458545Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011253", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:52:15.461992Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:52:47Z" } 2023-11-02T18:52:47.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:52:15.451244Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014883', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:52:15.454995Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013762', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:52:15.458545Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011253', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:52:15.461992Z'}}] 2023-11-02T18:52:47.340 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000014883', 2: '00000000000000000000:00000000000000013762', 3: '00000000000000000000:00000000000000011253'} 2023-11-02T18:52:47.340 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000014883', 2: '00000000000000000000:00000000000000013762', 3: '00000000000000000000:00000000000000011253'} 2023-11-02T18:52:47.340 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:52:47.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ok 2023-11-02T18:52:47.343 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:52:47.343 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:52:47.344 DEBUG:boto:Method: PUT 2023-11-02T18:52:47.344 DEBUG:boto:Path: /ibtljn-62/ 2023-11-02T18:52:47.345 DEBUG:boto:Data: 2023-11-02T18:52:47.345 DEBUG:boto:Headers: {} 2023-11-02T18:52:47.345 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:47.346 DEBUG:boto:Port: 8000 2023-11-02T18:52:47.346 DEBUG:boto:Params: {} 2023-11-02T18:52:47.347 DEBUG:boto:Token: None 2023-11-02T18:52:47.347 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:52:47 GMT /ibtljn-62/ 2023-11-02T18:52:47.348 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:U8VxtZKPEp7VPMH6FSJFVg7YDSk= 2023-11-02T18:52:47.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:U8VxtZKPEp7VPMH6FSJFVg7YDSk=', 'Content-Length': '0'} 2023-11-02T18:52:47.382 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007bde15fdf3751cb8-006543efff-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:47.382 DEBUG:rgw_multi.tests:created bucket=ibtljn-62 2023-11-02T18:52:47.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:52:47.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:52:47.384 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:52:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:52:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:52:47.478 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:52:47.506 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:52:47.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.506 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:52:47.507 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:52:47.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.507 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:52:47.508 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T18:52:47.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.508 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951167.375952_572.1", 2023-11-02T18:52:47.509 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:52:47.375952Z" 2023-11-02T18:52:47.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.509 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:52:47.509 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T18:52:47.510 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.510 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:52:47.511 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951167.375952_572.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:52:47.512 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:52:47.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:52:47.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:52:47.513 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:52:47.558 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:52:47.559 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:52:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:47.608+0000 7fda988fff40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:52:47.623 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:47.616+0000 7fda988fff40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:52:47.648 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:52:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:52:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:52:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:52:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:52:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:52:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:52:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:52:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:52:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:52:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:52:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:52:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:52:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:52:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:52:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:52:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:52:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:52:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:52:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:52:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:52:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:52:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:52:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:52:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T18:52:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:52:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:52:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:52:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:52:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951167.375952_572.1", 2023-11-02T18:52:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:52:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:52:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:52:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:52:47.375952Z", 2023-11-02T18:52:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:52:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:52:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:52:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:52:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:52:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:52:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:52:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:52:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:52:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T18:52:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:52:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:52:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:52:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:52:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:52:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:52:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:52:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:52:47Z" 2023-11-02T18:52:47.663 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:52:47.664 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951167.375952_572.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:52:47.375952Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:52:47Z" } 2023-11-02T18:52:47.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951167.375952_572.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:52:47.375952Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T18:52:47.665 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951167.375952_572.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:52:47.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951167.375952_572.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:52:47.666 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:52:47.667 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:52:47.667 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:52:47.667 DEBUG:boto:Method: HEAD 2023-11-02T18:52:47.668 DEBUG:boto:Path: /ibtljn-62/ 2023-11-02T18:52:47.668 DEBUG:boto:Data: 2023-11-02T18:52:47.669 DEBUG:boto:Headers: {} 2023-11-02T18:52:47.669 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:47.670 DEBUG:boto:Port: 8000 2023-11-02T18:52:47.670 DEBUG:boto:Params: {} 2023-11-02T18:52:47.671 DEBUG:boto:Token: None 2023-11-02T18:52:47.671 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:52:47 GMT /ibtljn-62/ 2023-11-02T18:52:47.671 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z5uCPlp5baGDya2zHGhCT+OZLsM= 2023-11-02T18:52:47.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:52:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z5uCPlp5baGDya2zHGhCT+OZLsM=', 'Content-Length': '0'} 2023-11-02T18:52:47.675 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfdf4d85bb3accbc-006543efff-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:47.676 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:52:47.676 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:52:47.677 DEBUG:boto:Method: PUT 2023-11-02T18:52:47.677 DEBUG:boto:Path: /ibtljn-62/dummy 2023-11-02T18:52:47.677 DEBUG:boto:Data: 2023-11-02T18:52:47.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:52:47.678 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:52:47.679 DEBUG:boto:Port: 8000 2023-11-02T18:52:47.679 DEBUG:boto:Params: {} 2023-11-02T18:52:47.680 DEBUG:boto:Token: None 2023-11-02T18:52:47.680 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:52:47 GMT /ibtljn-62/dummy 2023-11-02T18:52:47.681 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3b5co/FPhRArpgTI2dXwSmBbiGc= 2023-11-02T18:52:47.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:52:47 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3b5co/FPhRArpgTI2dXwSmBbiGc='} 2023-11-02T18:52:47.692 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000741b47f905b8f850-006543efff-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:52:47.693 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-62 2023-11-02T18:52:47.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:52:47.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:52:47.694 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:52:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:52:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:52:47.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:47.800+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:52:52.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:52.804+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:52:57.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:52:57.804+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:53:02.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:53:02.804+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:53:07.814 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:53:07.804+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:53:12.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:53:12.808+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:53:17.816 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:53:17.808+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:53:22.817 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:53:22.808+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:53:27.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:53:27.812+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:53:32.819 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:53:32.811+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:53:37.820 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:53:37.811+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:53:42.821 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:53:42.811+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:53:47.822 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:53:47.815+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:53:52.823 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:53:52.815+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:53:57.824 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:53:57.815+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:54:02.826 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:54:02.819+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:54:07.826 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:54:07.823+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:54:12.827 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:54:12.823+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:54:17.828 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:54:17.823+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:54:22.829 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:54:22.823+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:54:27.830 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:54:27.827+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:54:32.831 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:54:32.827+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:54:37.833 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:54:37.827+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:54:42.834 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:54:42.827+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:54:47.831+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:54:52.836 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:54:52.831+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:54:57.837 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:54:57.831+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:55:02.838 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:02.835+0000 7ff3d1509f40 1 waiting to reach incremental sync.. 2023-11-02T18:55:07.842 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:07.835+0000 7ff3d1509f40 1 bucket sync caught up with source: 2023-11-02T18:55:07.842 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:55:07.843 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:55:07.843 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:07.835+0000 7ff3d1509f40 0 bucket checkpoint complete 2023-11-02T18:55:07.873 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-62 2023-11-02T18:55:07.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:55:07.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:55:07.875 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:55:07.925 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:55:07.925 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:55:07.993 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:07.987+0000 7f74cb6acf40 1 waiting to reach incremental sync.. 2023-11-02T18:55:12.994 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:12.991+0000 7f74cb6acf40 1 waiting to reach incremental sync.. 2023-11-02T18:55:17.997 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:17.991+0000 7f74cb6acf40 1 bucket sync caught up with source: 2023-11-02T18:55:17.998 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.88.6, , , , , ] 2023-11-02T18:55:17.998 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.88.6, , , , , ] 2023-11-02T18:55:17.998 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:17.991+0000 7f74cb6acf40 0 bucket checkpoint complete 2023-11-02T18:55:18.028 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-62 zones={a2, a1} 2023-11-02T18:55:18.029 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:18.029 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:18.030 DEBUG:boto:Method: HEAD 2023-11-02T18:55:18.030 DEBUG:boto:Path: /ibtljn-62/ 2023-11-02T18:55:18.031 DEBUG:boto:Data: 2023-11-02T18:55:18.031 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:55:18.032 DEBUG:boto:Port: 8001 2023-11-02T18:55:18.033 DEBUG:boto:Params: {} 2023-11-02T18:55:18.033 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:55:18.034 DEBUG:boto:Token: None 2023-11-02T18:55:18.035 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/ 2023-11-02T18:55:18.035 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mtaJz4kxA++Ez00bN+3g2oII5Xg= 2023-11-02T18:55:18.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mtaJz4kxA++Ez00bN+3g2oII5Xg=', 'Content-Length': '0'} 2023-11-02T18:55:18.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a249a26a9779546-006543f096-1f6a-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.043 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:18.043 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:18.043 DEBUG:boto:Method: HEAD 2023-11-02T18:55:18.044 DEBUG:boto:Path: /ibtljn-62/ 2023-11-02T18:55:18.044 DEBUG:boto:Data: 2023-11-02T18:55:18.045 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.045 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:18.046 DEBUG:boto:Port: 8000 2023-11-02T18:55:18.046 DEBUG:boto:Params: {} 2023-11-02T18:55:18.047 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:55:18.047 DEBUG:boto:Token: None 2023-11-02T18:55:18.047 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/ 2023-11-02T18:55:18.048 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mtaJz4kxA++Ez00bN+3g2oII5Xg= 2023-11-02T18:55:18.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mtaJz4kxA++Ez00bN+3g2oII5Xg=', 'Content-Length': '0'} 2023-11-02T18:55:18.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000710bffb851551108-006543f096-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.055 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:55:18.055 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:18.056 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:18.056 DEBUG:boto:path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:55:18.056 DEBUG:boto:auth_path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:55:18.057 DEBUG:boto:Method: GET 2023-11-02T18:55:18.057 DEBUG:boto:Path: /ibtljn-62/?versions&max-keys=999 2023-11-02T18:55:18.058 DEBUG:boto:Data: 2023-11-02T18:55:18.058 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.059 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:55:18.059 DEBUG:boto:Port: 8001 2023-11-02T18:55:18.060 DEBUG:boto:Params: {} 2023-11-02T18:55:18.060 DEBUG:boto:Token: None 2023-11-02T18:55:18.060 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/?versions 2023-11-02T18:55:18.061 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9iyt+TJOVxQE0RIEHpb5+dkwiQM= 2023-11-02T18:55:18.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9iyt+TJOVxQE0RIEHpb5+dkwiQM=', 'Content-Length': '0'} 2023-11-02T18:55:18.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f698615cc1e28af6-006543f096-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.067 DEBUG:boto:b'ibtljn-62999falsedummynulltrue2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:55:18.069 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T18:55:18.069 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:55:18.069 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:18.070 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:18.070 DEBUG:boto:path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:55:18.071 DEBUG:boto:auth_path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:55:18.071 DEBUG:boto:Method: GET 2023-11-02T18:55:18.072 DEBUG:boto:Path: /ibtljn-62/?versions&max-keys=999 2023-11-02T18:55:18.072 DEBUG:boto:Data: 2023-11-02T18:55:18.072 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.073 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:18.073 DEBUG:boto:Port: 8000 2023-11-02T18:55:18.074 DEBUG:boto:Params: {} 2023-11-02T18:55:18.074 DEBUG:boto:Token: None 2023-11-02T18:55:18.075 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/?versions 2023-11-02T18:55:18.075 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9iyt+TJOVxQE0RIEHpb5+dkwiQM= 2023-11-02T18:55:18.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9iyt+TJOVxQE0RIEHpb5+dkwiQM=', 'Content-Length': '0'} 2023-11-02T18:55:18.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b1990beca77acc9-006543f096-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.081 DEBUG:boto:b'ibtljn-62999falsedummynulltrue2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:55:18.082 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T18:55:18.082 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:18.083 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:18.083 DEBUG:boto:path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:55:18.084 DEBUG:boto:auth_path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:55:18.084 DEBUG:boto:Method: GET 2023-11-02T18:55:18.084 DEBUG:boto:Path: /ibtljn-62/?versions&max-keys=999 2023-11-02T18:55:18.085 DEBUG:boto:Data: 2023-11-02T18:55:18.085 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.086 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:55:18.086 DEBUG:boto:Port: 8001 2023-11-02T18:55:18.087 DEBUG:boto:Params: {} 2023-11-02T18:55:18.087 DEBUG:boto:Token: None 2023-11-02T18:55:18.088 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/?versions 2023-11-02T18:55:18.088 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9iyt+TJOVxQE0RIEHpb5+dkwiQM= 2023-11-02T18:55:18.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9iyt+TJOVxQE0RIEHpb5+dkwiQM=', 'Content-Length': '0'} 2023-11-02T18:55:18.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000574d0ff7ab73e5d5-006543f096-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.094 DEBUG:boto:b'ibtljn-62999falsedummynulltrue2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:55:18.094 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:18.095 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:18.095 DEBUG:boto:path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:55:18.096 DEBUG:boto:auth_path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:55:18.096 DEBUG:boto:Method: GET 2023-11-02T18:55:18.097 DEBUG:boto:Path: /ibtljn-62/?versions&max-keys=999 2023-11-02T18:55:18.097 DEBUG:boto:Data: 2023-11-02T18:55:18.097 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.098 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:18.098 DEBUG:boto:Port: 8000 2023-11-02T18:55:18.099 DEBUG:boto:Params: {} 2023-11-02T18:55:18.099 DEBUG:boto:Token: None 2023-11-02T18:55:18.100 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/?versions 2023-11-02T18:55:18.100 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9iyt+TJOVxQE0RIEHpb5+dkwiQM= 2023-11-02T18:55:18.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9iyt+TJOVxQE0RIEHpb5+dkwiQM=', 'Content-Length': '0'} 2023-11-02T18:55:18.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee7a480b823a7c93-006543f096-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.106 DEBUG:boto:b'ibtljn-62999falsedummynulltrue2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:55:18.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T18:55:18.107 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:55:18.107 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:55:18.108 DEBUG:boto:path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.108 DEBUG:boto:auth_path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.109 DEBUG:boto:Method: GET 2023-11-02T18:55:18.109 DEBUG:boto:Path: /ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.109 DEBUG:boto:Data: 2023-11-02T18:55:18.110 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.110 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:55:18.111 DEBUG:boto:Port: 8001 2023-11-02T18:55:18.111 DEBUG:boto:Params: {} 2023-11-02T18:55:18.112 DEBUG:boto:Token: None 2023-11-02T18:55:18.112 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.113 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:riykyKEmeuXJ76hKpO+d7mxFCKk= 2023-11-02T18:55:18.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:riykyKEmeuXJ76hKpO+d7mxFCKk=', 'Content-Length': '0'} 2023-11-02T18:55:18.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000714be44115befa3e-006543f096-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.117 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:55:18.118 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:55:18.118 DEBUG:boto:path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.119 DEBUG:boto:auth_path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.119 DEBUG:boto:Method: GET 2023-11-02T18:55:18.119 DEBUG:boto:Path: /ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.120 DEBUG:boto:Data: 2023-11-02T18:55:18.120 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.121 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:18.121 DEBUG:boto:Port: 8000 2023-11-02T18:55:18.121 DEBUG:boto:Params: {} 2023-11-02T18:55:18.122 DEBUG:boto:Token: None 2023-11-02T18:55:18.122 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.123 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:riykyKEmeuXJ76hKpO+d7mxFCKk= 2023-11-02T18:55:18.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:riykyKEmeuXJ76hKpO+d7mxFCKk=', 'Content-Length': '0'} 2023-11-02T18:55:18.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002a0e34d96d2a13f-006543f096-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.127 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:55:18.128 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:55:18.128 DEBUG:boto:path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.128 DEBUG:boto:auth_path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.129 DEBUG:boto:Method: HEAD 2023-11-02T18:55:18.129 DEBUG:boto:Path: /ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.130 DEBUG:boto:Data: 2023-11-02T18:55:18.130 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.131 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:55:18.131 DEBUG:boto:Port: 8001 2023-11-02T18:55:18.131 DEBUG:boto:Params: {} 2023-11-02T18:55:18.132 DEBUG:boto:Token: None 2023-11-02T18:55:18.132 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.133 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fQ29JplaAqW15rhgbDxpgvkKKF8= 2023-11-02T18:55:18.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fQ29JplaAqW15rhgbDxpgvkKKF8=', 'Content-Length': '0'} 2023-11-02T18:55:18.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d078d4d217e94dd-006543f096-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.137 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:55:18.137 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:55:18.138 DEBUG:boto:path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.138 DEBUG:boto:auth_path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.138 DEBUG:boto:Method: HEAD 2023-11-02T18:55:18.139 DEBUG:boto:Path: /ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.139 DEBUG:boto:Data: 2023-11-02T18:55:18.140 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.140 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:18.141 DEBUG:boto:Port: 8000 2023-11-02T18:55:18.141 DEBUG:boto:Params: {} 2023-11-02T18:55:18.141 DEBUG:boto:Token: None 2023-11-02T18:55:18.142 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.142 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fQ29JplaAqW15rhgbDxpgvkKKF8= 2023-11-02T18:55:18.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fQ29JplaAqW15rhgbDxpgvkKKF8=', 'Content-Length': '0'} 2023-11-02T18:55:18.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083f903d312f266e7-006543f096-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T18:55:18.147 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:55:18.147 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:55:18.148 DEBUG:boto:path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.148 DEBUG:boto:auth_path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.149 DEBUG:boto:Method: GET 2023-11-02T18:55:18.149 DEBUG:boto:Path: /ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.149 DEBUG:boto:Data: 2023-11-02T18:55:18.150 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.150 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:55:18.151 DEBUG:boto:Port: 8001 2023-11-02T18:55:18.151 DEBUG:boto:Params: {} 2023-11-02T18:55:18.152 DEBUG:boto:Token: None 2023-11-02T18:55:18.152 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.153 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:riykyKEmeuXJ76hKpO+d7mxFCKk= 2023-11-02T18:55:18.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:riykyKEmeuXJ76hKpO+d7mxFCKk=', 'Content-Length': '0'} 2023-11-02T18:55:18.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9692144446d2b35-006543f096-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.198 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:55:18.199 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:55:18.199 DEBUG:boto:path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.200 DEBUG:boto:auth_path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.200 DEBUG:boto:Method: GET 2023-11-02T18:55:18.201 DEBUG:boto:Path: /ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.201 DEBUG:boto:Data: 2023-11-02T18:55:18.202 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:18.203 DEBUG:boto:Port: 8000 2023-11-02T18:55:18.203 DEBUG:boto:Params: {} 2023-11-02T18:55:18.204 DEBUG:boto:Token: None 2023-11-02T18:55:18.204 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/dummy?versionId=null 2023-11-02T18:55:18.205 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:riykyKEmeuXJ76hKpO+d7mxFCKk= 2023-11-02T18:55:18.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:riykyKEmeuXJ76hKpO+d7mxFCKk=', 'Content-Length': '0'} 2023-11-02T18:55:18.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a6178e9210de3b6-006543f096-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.209 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:18.209 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:18.210 DEBUG:boto:path=/ibtljn-62/?prefix=dummy 2023-11-02T18:55:18.210 DEBUG:boto:auth_path=/ibtljn-62/?prefix=dummy 2023-11-02T18:55:18.211 DEBUG:boto:Method: GET 2023-11-02T18:55:18.211 DEBUG:boto:Path: /ibtljn-62/?prefix=dummy 2023-11-02T18:55:18.212 DEBUG:boto:Data: 2023-11-02T18:55:18.212 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.212 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:55:18.213 DEBUG:boto:Port: 8001 2023-11-02T18:55:18.213 DEBUG:boto:Params: {} 2023-11-02T18:55:18.214 DEBUG:boto:Token: None 2023-11-02T18:55:18.214 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/ 2023-11-02T18:55:18.215 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRqxu+DwffVN8UUWu4lkM2clapM= 2023-11-02T18:55:18.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRqxu+DwffVN8UUWu4lkM2clapM=', 'Content-Length': '0'} 2023-11-02T18:55:18.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b19a5d02d43a2f3b-006543f096-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.221 DEBUG:boto:b'ibtljn-62dummy1000falsedummy2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:55:18.221 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:18.222 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:18.222 DEBUG:boto:path=/ibtljn-62/?prefix=dummy 2023-11-02T18:55:18.223 DEBUG:boto:auth_path=/ibtljn-62/?prefix=dummy 2023-11-02T18:55:18.223 DEBUG:boto:Method: GET 2023-11-02T18:55:18.224 DEBUG:boto:Path: /ibtljn-62/?prefix=dummy 2023-11-02T18:55:18.224 DEBUG:boto:Data: 2023-11-02T18:55:18.225 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.225 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:18.225 DEBUG:boto:Port: 8000 2023-11-02T18:55:18.226 DEBUG:boto:Params: {} 2023-11-02T18:55:18.226 DEBUG:boto:Token: None 2023-11-02T18:55:18.227 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/ 2023-11-02T18:55:18.227 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRqxu+DwffVN8UUWu4lkM2clapM= 2023-11-02T18:55:18.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRqxu+DwffVN8UUWu4lkM2clapM=', 'Content-Length': '0'} 2023-11-02T18:55:18.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4c7d75e64fc1161-006543f096-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.233 DEBUG:boto:b'ibtljn-62dummy1000falsedummy2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:55:18.234 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:55:18.234 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:55:18.235 DEBUG:boto:Method: HEAD 2023-11-02T18:55:18.235 DEBUG:boto:Path: /ibtljn-62/dummy 2023-11-02T18:55:18.236 DEBUG:boto:Data: 2023-11-02T18:55:18.236 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.237 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:55:18.237 DEBUG:boto:Port: 8001 2023-11-02T18:55:18.237 DEBUG:boto:Params: {} 2023-11-02T18:55:18.238 DEBUG:boto:Token: None 2023-11-02T18:55:18.238 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/dummy 2023-11-02T18:55:18.239 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ToeYQyK7pnHoLr+ze/MAJLtL8/o= 2023-11-02T18:55:18.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ToeYQyK7pnHoLr+ze/MAJLtL8/o=', 'Content-Length': '0'} 2023-11-02T18:55:18.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009923eaa025745c85-006543f096-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.243 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:55:18.244 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:55:18.244 DEBUG:boto:Method: HEAD 2023-11-02T18:55:18.245 DEBUG:boto:Path: /ibtljn-62/dummy 2023-11-02T18:55:18.245 DEBUG:boto:Data: 2023-11-02T18:55:18.245 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:18.246 DEBUG:boto:Port: 8000 2023-11-02T18:55:18.247 DEBUG:boto:Params: {} 2023-11-02T18:55:18.247 DEBUG:boto:Token: None 2023-11-02T18:55:18.248 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/dummy 2023-11-02T18:55:18.248 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ToeYQyK7pnHoLr+ze/MAJLtL8/o= 2023-11-02T18:55:18.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ToeYQyK7pnHoLr+ze/MAJLtL8/o=', 'Content-Length': '0'} 2023-11-02T18:55:18.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bd564e79e52f5cc-006543f096-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T18:55:18.253 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:55:18.253 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:55:18.253 DEBUG:boto:Method: GET 2023-11-02T18:55:18.254 DEBUG:boto:Path: /ibtljn-62/dummy 2023-11-02T18:55:18.254 DEBUG:boto:Data: 2023-11-02T18:55:18.255 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.255 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:55:18.256 DEBUG:boto:Port: 8001 2023-11-02T18:55:18.256 DEBUG:boto:Params: {} 2023-11-02T18:55:18.257 DEBUG:boto:Token: None 2023-11-02T18:55:18.257 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/dummy 2023-11-02T18:55:18.258 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xqX4zrHHkgqNcrn7IBC3Soxo68A= 2023-11-02T18:55:18.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xqX4zrHHkgqNcrn7IBC3Soxo68A=', 'Content-Length': '0'} 2023-11-02T18:55:18.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d48807bb0b60e89-006543f096-1f6a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.302 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:55:18.303 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:55:18.304 DEBUG:boto:Method: GET 2023-11-02T18:55:18.304 DEBUG:boto:Path: /ibtljn-62/dummy 2023-11-02T18:55:18.304 DEBUG:boto:Data: 2023-11-02T18:55:18.305 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.305 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:18.306 DEBUG:boto:Port: 8000 2023-11-02T18:55:18.306 DEBUG:boto:Params: {} 2023-11-02T18:55:18.307 DEBUG:boto:Token: None 2023-11-02T18:55:18.307 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/dummy 2023-11-02T18:55:18.308 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xqX4zrHHkgqNcrn7IBC3Soxo68A= 2023-11-02T18:55:18.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xqX4zrHHkgqNcrn7IBC3Soxo68A=', 'Content-Length': '0'} 2023-11-02T18:55:18.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000749d89e15319ff8b-006543f096-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.312 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:18.313 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:18.313 DEBUG:boto:path=/ibtljn-62/?prefix=dummy 2023-11-02T18:55:18.314 DEBUG:boto:auth_path=/ibtljn-62/?prefix=dummy 2023-11-02T18:55:18.314 DEBUG:boto:Method: GET 2023-11-02T18:55:18.315 DEBUG:boto:Path: /ibtljn-62/?prefix=dummy 2023-11-02T18:55:18.315 DEBUG:boto:Data: 2023-11-02T18:55:18.315 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.316 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:55:18.316 DEBUG:boto:Port: 8001 2023-11-02T18:55:18.317 DEBUG:boto:Params: {} 2023-11-02T18:55:18.317 DEBUG:boto:Token: None 2023-11-02T18:55:18.318 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/ 2023-11-02T18:55:18.318 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRqxu+DwffVN8UUWu4lkM2clapM= 2023-11-02T18:55:18.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRqxu+DwffVN8UUWu4lkM2clapM=', 'Content-Length': '0'} 2023-11-02T18:55:18.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da88c36ea78dbbbc-006543f096-1f6a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.324 DEBUG:boto:b'ibtljn-62dummy1000falsedummy2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:55:18.325 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:18.326 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:18.326 DEBUG:boto:path=/ibtljn-62/?prefix=dummy 2023-11-02T18:55:18.326 DEBUG:boto:auth_path=/ibtljn-62/?prefix=dummy 2023-11-02T18:55:18.327 DEBUG:boto:Method: GET 2023-11-02T18:55:18.327 DEBUG:boto:Path: /ibtljn-62/?prefix=dummy 2023-11-02T18:55:18.328 DEBUG:boto:Data: 2023-11-02T18:55:18.328 DEBUG:boto:Headers: {} 2023-11-02T18:55:18.329 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:18.329 DEBUG:boto:Port: 8000 2023-11-02T18:55:18.330 DEBUG:boto:Params: {} 2023-11-02T18:55:18.330 DEBUG:boto:Token: None 2023-11-02T18:55:18.331 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:55:18 GMT /ibtljn-62/ 2023-11-02T18:55:18.331 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gRqxu+DwffVN8UUWu4lkM2clapM= 2023-11-02T18:55:18.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gRqxu+DwffVN8UUWu4lkM2clapM=', 'Content-Length': '0'} 2023-11-02T18:55:18.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000050bd5560396f3af-006543f096-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:18.382 DEBUG:boto:b'ibtljn-62dummy1000falsedummy2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:55:18.383 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-62 zones={a2, a1} 2023-11-02T18:55:18.384 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-11-02T18:55:18.384 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T18:55:18.385 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:55:18.379+0000 7f62151f7700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 78450) UID: 0 2023-11-02T18:55:18.386 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T18:55:18.379+0000 7f6218a56f80 -1 shutting down 2023-11-02T18:55:24.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-11-02T18:55:24.483 INFO:tasks.rgw.c1.client.1:Stopped 2023-11-02T18:55:24.484 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:24.485 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:24.486 DEBUG:boto:Method: HEAD 2023-11-02T18:55:24.486 DEBUG:boto:Path: /ibtljn-62/ 2023-11-02T18:55:24.487 DEBUG:boto:Data: 2023-11-02T18:55:24.488 DEBUG:boto:Headers: {} 2023-11-02T18:55:24.488 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:24.490 DEBUG:boto:Port: 8000 2023-11-02T18:55:24.491 DEBUG:boto:Params: {} 2023-11-02T18:55:24.492 DEBUG:boto:Token: None 2023-11-02T18:55:24.493 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:55:24 GMT /ibtljn-62/ 2023-11-02T18:55:24.494 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TpmibQrIn66g+H1XqTsMW3wHuxc= 2023-11-02T18:55:24.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TpmibQrIn66g+H1XqTsMW3wHuxc=', 'Content-Length': '0'} 2023-11-02T18:55:24.505 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002723c8aba0086f84-006543f09c-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:24.506 DEBUG:boto:path=/ibtljn-62/obj11 2023-11-02T18:55:24.507 DEBUG:boto:auth_path=/ibtljn-62/obj11 2023-11-02T18:55:24.507 DEBUG:boto:Method: PUT 2023-11-02T18:55:24.508 DEBUG:boto:Path: /ibtljn-62/obj11 2023-11-02T18:55:24.508 DEBUG:boto:Data: 2023-11-02T18:55:24.521 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:55:24.521 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:24.522 DEBUG:boto:Port: 8000 2023-11-02T18:55:24.522 DEBUG:boto:Params: {} 2023-11-02T18:55:24.523 DEBUG:boto:Token: None 2023-11-02T18:55:24.523 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:55:24 GMT /ibtljn-62/obj11 2023-11-02T18:55:24.524 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UIwlq5gma4fjSkTieGbN3gKnDSs= 2023-11-02T18:55:24.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:55:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UIwlq5gma4fjSkTieGbN3gKnDSs='} 2023-11-02T18:55:24.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c517fcb9bf4554c6-006543f09c-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:24.539 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:24.539 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:24.539 DEBUG:boto:Method: HEAD 2023-11-02T18:55:24.540 DEBUG:boto:Path: /ibtljn-62/ 2023-11-02T18:55:24.540 DEBUG:boto:Data: 2023-11-02T18:55:24.541 DEBUG:boto:Headers: {} 2023-11-02T18:55:24.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:24.542 DEBUG:boto:Port: 8000 2023-11-02T18:55:24.542 DEBUG:boto:Params: {} 2023-11-02T18:55:24.542 DEBUG:boto:Token: None 2023-11-02T18:55:24.543 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:55:24 GMT /ibtljn-62/ 2023-11-02T18:55:24.543 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TpmibQrIn66g+H1XqTsMW3wHuxc= 2023-11-02T18:55:24.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TpmibQrIn66g+H1XqTsMW3wHuxc=', 'Content-Length': '0'} 2023-11-02T18:55:24.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef15bc14bb8aa2cc-006543f09c-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:24.553 DEBUG:boto:path=/ibtljn-62/obj22 2023-11-02T18:55:24.554 DEBUG:boto:auth_path=/ibtljn-62/obj22 2023-11-02T18:55:24.554 DEBUG:boto:Method: PUT 2023-11-02T18:55:24.555 DEBUG:boto:Path: /ibtljn-62/obj22 2023-11-02T18:55:24.555 DEBUG:boto:Data: 2023-11-02T18:55:24.572 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:55:24.573 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:24.573 DEBUG:boto:Port: 8000 2023-11-02T18:55:24.574 DEBUG:boto:Params: {} 2023-11-02T18:55:24.575 DEBUG:boto:Token: None 2023-11-02T18:55:24.577 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:55:24 GMT /ibtljn-62/obj22 2023-11-02T18:55:24.577 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Eaxr+1Bul7ORa+nEmaLWlI3UOrg= 2023-11-02T18:55:24.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:55:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Eaxr+1Bul7ORa+nEmaLWlI3UOrg='} 2023-11-02T18:55:24.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000687fb2bd8d6256b5-006543f09c-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:24.624 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:24.624 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:24.625 DEBUG:boto:Method: HEAD 2023-11-02T18:55:24.625 DEBUG:boto:Path: /ibtljn-62/ 2023-11-02T18:55:24.625 DEBUG:boto:Data: 2023-11-02T18:55:24.626 DEBUG:boto:Headers: {} 2023-11-02T18:55:24.626 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:24.626 DEBUG:boto:Port: 8000 2023-11-02T18:55:24.627 DEBUG:boto:Params: {} 2023-11-02T18:55:24.627 DEBUG:boto:Token: None 2023-11-02T18:55:24.628 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:55:24 GMT /ibtljn-62/ 2023-11-02T18:55:24.628 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TpmibQrIn66g+H1XqTsMW3wHuxc= 2023-11-02T18:55:24.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TpmibQrIn66g+H1XqTsMW3wHuxc=', 'Content-Length': '0'} 2023-11-02T18:55:24.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069ae51ae40315af0-006543f09c-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:24.636 DEBUG:boto:path=/ibtljn-62/obj33 2023-11-02T18:55:24.636 DEBUG:boto:auth_path=/ibtljn-62/obj33 2023-11-02T18:55:24.637 DEBUG:boto:Method: PUT 2023-11-02T18:55:24.637 DEBUG:boto:Path: /ibtljn-62/obj33 2023-11-02T18:55:24.637 DEBUG:boto:Data: 2023-11-02T18:55:24.638 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:55:24.638 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:24.639 DEBUG:boto:Port: 8000 2023-11-02T18:55:24.639 DEBUG:boto:Params: {} 2023-11-02T18:55:24.640 DEBUG:boto:Token: None 2023-11-02T18:55:24.641 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:55:24 GMT /ibtljn-62/obj33 2023-11-02T18:55:24.641 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WirfMmAGSzW/q6NDRrOAr+QoOXw= 2023-11-02T18:55:24.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:55:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WirfMmAGSzW/q6NDRrOAr+QoOXw='} 2023-11-02T18:55:24.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066a76c5258d4f73d-006543f09c-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:24.668 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:24.669 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:24.669 DEBUG:boto:Method: HEAD 2023-11-02T18:55:24.669 DEBUG:boto:Path: /ibtljn-62/ 2023-11-02T18:55:24.671 DEBUG:boto:Data: 2023-11-02T18:55:24.671 DEBUG:boto:Headers: {} 2023-11-02T18:55:24.673 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:24.674 DEBUG:boto:Port: 8000 2023-11-02T18:55:24.674 DEBUG:boto:Params: {} 2023-11-02T18:55:24.676 DEBUG:boto:Token: None 2023-11-02T18:55:24.676 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:55:24 GMT /ibtljn-62/ 2023-11-02T18:55:24.677 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TpmibQrIn66g+H1XqTsMW3wHuxc= 2023-11-02T18:55:24.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TpmibQrIn66g+H1XqTsMW3wHuxc=', 'Content-Length': '0'} 2023-11-02T18:55:24.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079ebf0a27a587bc6-006543f09c-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:24.702 DEBUG:boto:path=/ibtljn-62/obj44 2023-11-02T18:55:24.702 DEBUG:boto:auth_path=/ibtljn-62/obj44 2023-11-02T18:55:24.703 DEBUG:boto:Method: PUT 2023-11-02T18:55:24.703 DEBUG:boto:Path: /ibtljn-62/obj44 2023-11-02T18:55:24.703 DEBUG:boto:Data: 2023-11-02T18:55:24.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:55:24.716 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:24.716 DEBUG:boto:Port: 8000 2023-11-02T18:55:24.717 DEBUG:boto:Params: {} 2023-11-02T18:55:24.717 DEBUG:boto:Token: None 2023-11-02T18:55:24.718 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:55:24 GMT /ibtljn-62/obj44 2023-11-02T18:55:24.724 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8//o3sQCbpXYZCh6ODcTT4B8Wu0= 2023-11-02T18:55:24.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:55:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8//o3sQCbpXYZCh6ODcTT4B8Wu0='} 2023-11-02T18:55:24.744 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001851b5115746a627-006543f09c-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:24.744 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:55:24.745 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:55:24.745 DEBUG:boto:Method: HEAD 2023-11-02T18:55:24.745 DEBUG:boto:Path: /ibtljn-62/ 2023-11-02T18:55:24.746 DEBUG:boto:Data: 2023-11-02T18:55:24.749 DEBUG:boto:Headers: {} 2023-11-02T18:55:24.760 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:24.761 DEBUG:boto:Port: 8000 2023-11-02T18:55:24.762 DEBUG:boto:Params: {} 2023-11-02T18:55:24.762 DEBUG:boto:Token: None 2023-11-02T18:55:24.763 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:55:24 GMT /ibtljn-62/ 2023-11-02T18:55:24.763 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TpmibQrIn66g+H1XqTsMW3wHuxc= 2023-11-02T18:55:24.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:55:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TpmibQrIn66g+H1XqTsMW3wHuxc=', 'Content-Length': '0'} 2023-11-02T18:55:24.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dfc7781dc247f2c-006543f09c-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:24.773 DEBUG:boto:path=/ibtljn-62/obj55 2023-11-02T18:55:24.774 DEBUG:boto:auth_path=/ibtljn-62/obj55 2023-11-02T18:55:24.775 DEBUG:boto:Method: PUT 2023-11-02T18:55:24.776 DEBUG:boto:Path: /ibtljn-62/obj55 2023-11-02T18:55:24.777 DEBUG:boto:Data: 2023-11-02T18:55:24.777 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:55:24.778 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:55:24.778 DEBUG:boto:Port: 8000 2023-11-02T18:55:24.779 DEBUG:boto:Params: {} 2023-11-02T18:55:24.779 DEBUG:boto:Token: None 2023-11-02T18:55:24.780 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:55:24 GMT /ibtljn-62/obj55 2023-11-02T18:55:24.780 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kUyrac7AG53nyRetKJBF7KArpok= 2023-11-02T18:55:24.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:55:24 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kUyrac7AG53nyRetKJBF7KArpok='} 2023-11-02T18:55:24.790 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000547a5e8f09c3df95-006543f09c-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:55:24.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:55:24.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:55:24.793 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:55:24.862 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:55:24.862 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:55:24.923 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:24.919+0000 7f909aab9f40 5 RGW-SYNC:data: waiting on data sync status lease 2023-11-02T18:55:24.928 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:24.919+0000 7f909aab9f40 5 RGW-SYNC:data: acquired data sync status lease 2023-11-02T18:55:24.968 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-11-02T18:55:24.968 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-11-02T18:55:25.012 INFO:tasks.rgw.c1.client.1:Started 2023-11-02T18:55:25.013 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T18:55:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-11-02T18:55:25.032 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T18:55:25.032 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T18:55:25.032 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-11-02T18:55:25.033 INFO:teuthology.orchestra.run.smithi017.stderr:curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8001: Connection refused 2023-11-02T18:55:26.033 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T18:55:26.052 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T18:55:26.052 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T18:55:26.055 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-11-02T18:55:26.056 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2023-11-02T18:55:26.056 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-62 2023-11-02T18:55:26.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:55:26.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:55:26.058 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:55:26.107 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:55:26.107 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:55:26.173 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:26.167+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:55:26.173 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:55:26.174 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:55:30.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-11-02T18:55:31.176 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:31.170+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:55:31.176 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:55:31.176 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:55:36.178 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:36.174+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:55:36.178 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:55:36.179 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:55:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:41.174+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:55:41.181 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:55:41.182 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:55:46.183 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:46.178+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:55:46.183 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:55:46.184 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:55:51.185 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:51.178+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:55:51.186 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:55:51.186 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:55:56.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:55:56.182+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:55:56.188 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:55:56.189 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:56:01.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:56:01.186+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:56:01.191 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:56:01.191 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:56:06.193 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:56:06.186+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:56:06.193 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:56:06.194 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:56:11.195 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:56:11.190+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:56:11.196 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:56:11.196 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:56:16.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:56:16.190+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:56:16.198 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:56:16.198 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:56:21.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:56:21.194+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:56:21.201 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:56:21.201 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:56:26.205 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:56:26.198+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:56:26.206 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:56:26.206 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:56:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:56:31.198+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:56:31.205 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:56:31.206 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:56:36.208 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:56:36.202+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:56:36.208 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:56:36.209 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:56:41.210 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:56:41.202+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:56:41.210 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:56:41.210 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:56:46.212 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:56:46.206+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:56:46.213 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:56:46.213 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:56:51.217 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:56:51.210+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:56:51.218 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:56:51.218 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:56:56.217 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:56:56.210+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:56:56.217 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , , ] 2023-11-02T18:56:56.218 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:57:01.220 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:01.214+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:57:01.220 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , 00000000001.71.6, ] 2023-11-02T18:57:01.220 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:57:06.222 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:06.218+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:57:06.223 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , 00000000001.71.6, ] 2023-11-02T18:57:06.223 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:57:11.225 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:11.218+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:57:11.225 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , 00000000001.71.6, ] 2023-11-02T18:57:11.226 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:57:16.227 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:16.222+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:57:16.228 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , 00000000001.71.6, ] 2023-11-02T18:57:16.228 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:57:21.229 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:21.222+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:57:21.230 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , 00000000001.71.6, ] 2023-11-02T18:57:21.230 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:57:26.232 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:26.226+0000 7f4081a7df40 1 waiting for incremental sync to catch up: 2023-11-02T18:57:26.232 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , , 00000000001.71.6, ] 2023-11-02T18:57:26.233 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:57:31.234 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:31.230+0000 7f4081a7df40 1 bucket sync caught up with source: 2023-11-02T18:57:31.235 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:57:31.235 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.84.6, , , 00000000001.78.6, 00000000002.73.6, 00000000002.76.6] 2023-11-02T18:57:31.235 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:31.230+0000 7f4081a7df40 0 bucket checkpoint complete 2023-11-02T18:57:31.269 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-62 2023-11-02T18:57:31.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:57:31.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:57:31.271 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:57:31.324 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:57:31.324 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:57:31.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:31.386+0000 7f18af01cf40 1 waiting for incremental sync to catch up: 2023-11-02T18:57:31.394 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.88.6, , , , 00000000001.86.6, ] 2023-11-02T18:57:31.394 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.88.6, , , 00000000001.356.6, 00000000002.88.6, 00000000002.82.6] 2023-11-02T18:57:36.396 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:36.390+0000 7f18af01cf40 1 bucket sync caught up with source: 2023-11-02T18:57:36.396 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , 00000000001.88.6, , , 00000000001.356.6, 00000000002.88.6, 00000000002.82.6] 2023-11-02T18:57:36.396 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , 00000000001.88.6, , , 00000000001.356.6, 00000000002.88.6, 00000000002.82.6] 2023-11-02T18:57:36.397 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:36.390+0000 7f18af01cf40 0 bucket checkpoint complete 2023-11-02T18:57:36.426 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-62 zones={a2, a1} 2023-11-02T18:57:36.427 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:36.428 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:36.428 DEBUG:boto:Method: HEAD 2023-11-02T18:57:36.428 DEBUG:boto:Path: /ibtljn-62/ 2023-11-02T18:57:36.429 DEBUG:boto:Data: 2023-11-02T18:57:36.429 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.430 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.430 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.431 DEBUG:boto:Params: {} 2023-11-02T18:57:36.431 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T18:57:36.432 DEBUG:boto:Token: None 2023-11-02T18:57:36.433 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/ 2023-11-02T18:57:36.433 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZTYGDuMsFdP+4cEhvT4iqg5gqoQ= 2023-11-02T18:57:36.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZTYGDuMsFdP+4cEhvT4iqg5gqoQ=', 'Content-Length': '0'} 2023-11-02T18:57:36.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e3cc695372ff652-006543f120-1fca-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.445 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:36.445 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:36.446 DEBUG:boto:Method: HEAD 2023-11-02T18:57:36.446 DEBUG:boto:Path: /ibtljn-62/ 2023-11-02T18:57:36.446 DEBUG:boto:Data: 2023-11-02T18:57:36.447 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.447 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.447 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.448 DEBUG:boto:Params: {} 2023-11-02T18:57:36.448 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T18:57:36.449 DEBUG:boto:Token: None 2023-11-02T18:57:36.449 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/ 2023-11-02T18:57:36.450 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZTYGDuMsFdP+4cEhvT4iqg5gqoQ= 2023-11-02T18:57:36.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZTYGDuMsFdP+4cEhvT4iqg5gqoQ=', 'Content-Length': '0'} 2023-11-02T18:57:36.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000acbd16feb172719-006543f120-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.460 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T18:57:36.460 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:36.461 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:36.461 DEBUG:boto:path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:57:36.462 DEBUG:boto:auth_path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:57:36.462 DEBUG:boto:Method: GET 2023-11-02T18:57:36.462 DEBUG:boto:Path: /ibtljn-62/?versions&max-keys=999 2023-11-02T18:57:36.463 DEBUG:boto:Data: 2023-11-02T18:57:36.463 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.464 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.464 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.464 DEBUG:boto:Params: {} 2023-11-02T18:57:36.465 DEBUG:boto:Token: None 2023-11-02T18:57:36.465 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/?versions 2023-11-02T18:57:36.466 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vM6X3Kod1AUifV0H+YRIOnhztz0= 2023-11-02T18:57:36.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vM6X3Kod1AUifV0H+YRIOnhztz0=', 'Content-Length': '0'} 2023-11-02T18:57:36.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d588262ce4e77b86-006543f120-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.472 DEBUG:boto:b'ibtljn-62999falsedummynulltrue2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-11-02T18:55:24.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:55:24.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:55:24.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:55:24.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:55:24.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:36.473 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T18:57:36.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:57:36.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:57:36.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:57:36.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:57:36.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:57:36.475 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T18:57:36.476 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:36.476 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:36.476 DEBUG:boto:path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:57:36.477 DEBUG:boto:auth_path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:57:36.477 DEBUG:boto:Method: GET 2023-11-02T18:57:36.478 DEBUG:boto:Path: /ibtljn-62/?versions&max-keys=999 2023-11-02T18:57:36.478 DEBUG:boto:Data: 2023-11-02T18:57:36.479 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.479 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.479 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.480 DEBUG:boto:Params: {} 2023-11-02T18:57:36.480 DEBUG:boto:Token: None 2023-11-02T18:57:36.481 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/?versions 2023-11-02T18:57:36.481 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vM6X3Kod1AUifV0H+YRIOnhztz0= 2023-11-02T18:57:36.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vM6X3Kod1AUifV0H+YRIOnhztz0=', 'Content-Length': '0'} 2023-11-02T18:57:36.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b1e62f6036e333a-006543f120-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.487 DEBUG:boto:b'ibtljn-62999falsedummynulltrue2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-11-02T18:55:24.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:55:24.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:55:24.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:55:24.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:55:24.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:36.488 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-11-02T18:57:36.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T18:57:36.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T18:57:36.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T18:57:36.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T18:57:36.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T18:57:36.490 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:36.491 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:36.491 DEBUG:boto:path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:57:36.491 DEBUG:boto:auth_path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:57:36.492 DEBUG:boto:Method: GET 2023-11-02T18:57:36.492 DEBUG:boto:Path: /ibtljn-62/?versions&max-keys=999 2023-11-02T18:57:36.493 DEBUG:boto:Data: 2023-11-02T18:57:36.493 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.493 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.494 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.494 DEBUG:boto:Params: {} 2023-11-02T18:57:36.495 DEBUG:boto:Token: None 2023-11-02T18:57:36.495 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/?versions 2023-11-02T18:57:36.496 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vM6X3Kod1AUifV0H+YRIOnhztz0= 2023-11-02T18:57:36.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vM6X3Kod1AUifV0H+YRIOnhztz0=', 'Content-Length': '0'} 2023-11-02T18:57:36.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bee299642201958a-006543f120-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.502 DEBUG:boto:b'ibtljn-62999falsedummynulltrue2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-11-02T18:55:24.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:55:24.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:55:24.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:55:24.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:55:24.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:36.503 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:36.504 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:36.504 DEBUG:boto:path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:57:36.504 DEBUG:boto:auth_path=/ibtljn-62/?versions&max-keys=999 2023-11-02T18:57:36.505 DEBUG:boto:Method: GET 2023-11-02T18:57:36.505 DEBUG:boto:Path: /ibtljn-62/?versions&max-keys=999 2023-11-02T18:57:36.506 DEBUG:boto:Data: 2023-11-02T18:57:36.506 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.506 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.507 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.507 DEBUG:boto:Params: {} 2023-11-02T18:57:36.508 DEBUG:boto:Token: None 2023-11-02T18:57:36.508 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/?versions 2023-11-02T18:57:36.509 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vM6X3Kod1AUifV0H+YRIOnhztz0= 2023-11-02T18:57:36.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vM6X3Kod1AUifV0H+YRIOnhztz0=', 'Content-Length': '0'} 2023-11-02T18:57:36.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9ac2153a0d31e75-006543f120-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.515 DEBUG:boto:b'ibtljn-62999falsedummynulltrue2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-11-02T18:55:24.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T18:55:24.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T18:55:24.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T18:55:24.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T18:55:24.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:36.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T18:57:36.517 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:57:36.517 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:57:36.518 DEBUG:boto:path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.518 DEBUG:boto:auth_path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.518 DEBUG:boto:Method: GET 2023-11-02T18:57:36.519 DEBUG:boto:Path: /ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.519 DEBUG:boto:Data: 2023-11-02T18:57:36.520 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.520 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.521 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.521 DEBUG:boto:Params: {} 2023-11-02T18:57:36.522 DEBUG:boto:Token: None 2023-11-02T18:57:36.522 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.523 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R7SwQAt6zXlXLsiSZ83Ga3LUfwc= 2023-11-02T18:57:36.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R7SwQAt6zXlXLsiSZ83Ga3LUfwc=', 'Content-Length': '0'} 2023-11-02T18:57:36.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066ae54c063c47110-006543f120-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.570 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:57:36.571 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:57:36.571 DEBUG:boto:path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.572 DEBUG:boto:auth_path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.572 DEBUG:boto:Method: GET 2023-11-02T18:57:36.573 DEBUG:boto:Path: /ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.573 DEBUG:boto:Data: 2023-11-02T18:57:36.574 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.575 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.575 DEBUG:boto:Params: {} 2023-11-02T18:57:36.575 DEBUG:boto:Token: None 2023-11-02T18:57:36.576 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.576 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R7SwQAt6zXlXLsiSZ83Ga3LUfwc= 2023-11-02T18:57:36.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R7SwQAt6zXlXLsiSZ83Ga3LUfwc=', 'Content-Length': '0'} 2023-11-02T18:57:36.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6b88a34e0142cff-006543f120-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.581 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:57:36.581 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:57:36.581 DEBUG:boto:path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.582 DEBUG:boto:auth_path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.582 DEBUG:boto:Method: HEAD 2023-11-02T18:57:36.583 DEBUG:boto:Path: /ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.583 DEBUG:boto:Data: 2023-11-02T18:57:36.584 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.584 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.584 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.585 DEBUG:boto:Params: {} 2023-11-02T18:57:36.585 DEBUG:boto:Token: None 2023-11-02T18:57:36.586 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.586 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JWo4Eus3oI56eGxa6cMvBVUCTKY= 2023-11-02T18:57:36.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JWo4Eus3oI56eGxa6cMvBVUCTKY=', 'Content-Length': '0'} 2023-11-02T18:57:36.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a5e33e693cbb9e3-006543f120-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.590 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:57:36.591 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:57:36.591 DEBUG:boto:path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.592 DEBUG:boto:auth_path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.592 DEBUG:boto:Method: HEAD 2023-11-02T18:57:36.592 DEBUG:boto:Path: /ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.593 DEBUG:boto:Data: 2023-11-02T18:57:36.593 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.594 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.594 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.595 DEBUG:boto:Params: {} 2023-11-02T18:57:36.595 DEBUG:boto:Token: None 2023-11-02T18:57:36.595 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.596 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JWo4Eus3oI56eGxa6cMvBVUCTKY= 2023-11-02T18:57:36.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JWo4Eus3oI56eGxa6cMvBVUCTKY=', 'Content-Length': '0'} 2023-11-02T18:57:36.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009267e3297b484836-006543f120-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T18:57:36.600 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:57:36.601 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:57:36.601 DEBUG:boto:path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.602 DEBUG:boto:auth_path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.602 DEBUG:boto:Method: GET 2023-11-02T18:57:36.602 DEBUG:boto:Path: /ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.603 DEBUG:boto:Data: 2023-11-02T18:57:36.603 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.604 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.604 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.605 DEBUG:boto:Params: {} 2023-11-02T18:57:36.605 DEBUG:boto:Token: None 2023-11-02T18:57:36.606 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.606 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R7SwQAt6zXlXLsiSZ83Ga3LUfwc= 2023-11-02T18:57:36.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R7SwQAt6zXlXLsiSZ83Ga3LUfwc=', 'Content-Length': '0'} 2023-11-02T18:57:36.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc13ff85398c4d43-006543f120-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.611 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:57:36.611 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:57:36.612 DEBUG:boto:path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.612 DEBUG:boto:auth_path=/ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.612 DEBUG:boto:Method: GET 2023-11-02T18:57:36.613 DEBUG:boto:Path: /ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.613 DEBUG:boto:Data: 2023-11-02T18:57:36.614 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.614 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.615 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.615 DEBUG:boto:Params: {} 2023-11-02T18:57:36.615 DEBUG:boto:Token: None 2023-11-02T18:57:36.616 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/dummy?versionId=null 2023-11-02T18:57:36.617 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R7SwQAt6zXlXLsiSZ83Ga3LUfwc= 2023-11-02T18:57:36.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R7SwQAt6zXlXLsiSZ83Ga3LUfwc=', 'Content-Length': '0'} 2023-11-02T18:57:36.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3d5561906bc6377-006543f120-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.662 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:36.663 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:36.663 DEBUG:boto:path=/ibtljn-62/?prefix=dummy 2023-11-02T18:57:36.664 DEBUG:boto:auth_path=/ibtljn-62/?prefix=dummy 2023-11-02T18:57:36.664 DEBUG:boto:Method: GET 2023-11-02T18:57:36.665 DEBUG:boto:Path: /ibtljn-62/?prefix=dummy 2023-11-02T18:57:36.666 DEBUG:boto:Data: 2023-11-02T18:57:36.666 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.667 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.667 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.667 DEBUG:boto:Params: {} 2023-11-02T18:57:36.668 DEBUG:boto:Token: None 2023-11-02T18:57:36.669 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/ 2023-11-02T18:57:36.669 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LCncexKasPp2DRcd5Q31pJ1Bt60= 2023-11-02T18:57:36.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LCncexKasPp2DRcd5Q31pJ1Bt60=', 'Content-Length': '0'} 2023-11-02T18:57:36.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db5e65fbb77ce059-006543f120-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.675 DEBUG:boto:b'ibtljn-62dummy1000falsedummy2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:36.676 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:36.677 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:36.677 DEBUG:boto:path=/ibtljn-62/?prefix=dummy 2023-11-02T18:57:36.678 DEBUG:boto:auth_path=/ibtljn-62/?prefix=dummy 2023-11-02T18:57:36.678 DEBUG:boto:Method: GET 2023-11-02T18:57:36.679 DEBUG:boto:Path: /ibtljn-62/?prefix=dummy 2023-11-02T18:57:36.679 DEBUG:boto:Data: 2023-11-02T18:57:36.679 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.680 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.681 DEBUG:boto:Params: {} 2023-11-02T18:57:36.681 DEBUG:boto:Token: None 2023-11-02T18:57:36.682 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/ 2023-11-02T18:57:36.682 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LCncexKasPp2DRcd5Q31pJ1Bt60= 2023-11-02T18:57:36.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LCncexKasPp2DRcd5Q31pJ1Bt60=', 'Content-Length': '0'} 2023-11-02T18:57:36.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d0c9d279a11a77b-006543f120-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.690 DEBUG:boto:b'ibtljn-62dummy1000falsedummy2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:36.691 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:57:36.691 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:57:36.691 DEBUG:boto:Method: HEAD 2023-11-02T18:57:36.692 DEBUG:boto:Path: /ibtljn-62/dummy 2023-11-02T18:57:36.692 DEBUG:boto:Data: 2023-11-02T18:57:36.693 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.693 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.693 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.694 DEBUG:boto:Params: {} 2023-11-02T18:57:36.694 DEBUG:boto:Token: None 2023-11-02T18:57:36.695 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/dummy 2023-11-02T18:57:36.695 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y0Fv3ol3QfCLHw5nIhjclpntRE4= 2023-11-02T18:57:36.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y0Fv3ol3QfCLHw5nIhjclpntRE4=', 'Content-Length': '0'} 2023-11-02T18:57:36.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001af69957fbc0b8be-006543f120-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.699 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:57:36.700 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:57:36.700 DEBUG:boto:Method: HEAD 2023-11-02T18:57:36.701 DEBUG:boto:Path: /ibtljn-62/dummy 2023-11-02T18:57:36.701 DEBUG:boto:Data: 2023-11-02T18:57:36.701 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.702 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.702 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.703 DEBUG:boto:Params: {} 2023-11-02T18:57:36.703 DEBUG:boto:Token: None 2023-11-02T18:57:36.703 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/dummy 2023-11-02T18:57:36.704 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y0Fv3ol3QfCLHw5nIhjclpntRE4= 2023-11-02T18:57:36.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y0Fv3ol3QfCLHw5nIhjclpntRE4=', 'Content-Length': '0'} 2023-11-02T18:57:36.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a90a86973ff6a5d-006543f120-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-11-02T18:57:36.708 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:57:36.708 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:57:36.709 DEBUG:boto:Method: GET 2023-11-02T18:57:36.709 DEBUG:boto:Path: /ibtljn-62/dummy 2023-11-02T18:57:36.710 DEBUG:boto:Data: 2023-11-02T18:57:36.710 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.711 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.711 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.711 DEBUG:boto:Params: {} 2023-11-02T18:57:36.712 DEBUG:boto:Token: None 2023-11-02T18:57:36.712 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/dummy 2023-11-02T18:57:36.713 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jBma/2yX+heBJ2ZwbxS9P9/XP18= 2023-11-02T18:57:36.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jBma/2yX+heBJ2ZwbxS9P9/XP18=', 'Content-Length': '0'} 2023-11-02T18:57:36.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1dc71ba97ef0938-006543f120-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.758 DEBUG:boto:path=/ibtljn-62/dummy 2023-11-02T18:57:36.759 DEBUG:boto:auth_path=/ibtljn-62/dummy 2023-11-02T18:57:36.759 DEBUG:boto:Method: GET 2023-11-02T18:57:36.760 DEBUG:boto:Path: /ibtljn-62/dummy 2023-11-02T18:57:36.760 DEBUG:boto:Data: 2023-11-02T18:57:36.760 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.761 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.761 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.762 DEBUG:boto:Params: {} 2023-11-02T18:57:36.762 DEBUG:boto:Token: None 2023-11-02T18:57:36.763 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/dummy 2023-11-02T18:57:36.763 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jBma/2yX+heBJ2ZwbxS9P9/XP18= 2023-11-02T18:57:36.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jBma/2yX+heBJ2ZwbxS9P9/XP18=', 'Content-Length': '0'} 2023-11-02T18:57:36.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:52:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f652d5ca975c3ad5-006543f120-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.768 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:36.768 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:36.769 DEBUG:boto:path=/ibtljn-62/?prefix=dummy 2023-11-02T18:57:36.770 DEBUG:boto:auth_path=/ibtljn-62/?prefix=dummy 2023-11-02T18:57:36.771 DEBUG:boto:Method: GET 2023-11-02T18:57:36.772 DEBUG:boto:Path: /ibtljn-62/?prefix=dummy 2023-11-02T18:57:36.772 DEBUG:boto:Data: 2023-11-02T18:57:36.773 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.773 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.774 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.775 DEBUG:boto:Params: {} 2023-11-02T18:57:36.775 DEBUG:boto:Token: None 2023-11-02T18:57:36.776 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/ 2023-11-02T18:57:36.776 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LCncexKasPp2DRcd5Q31pJ1Bt60= 2023-11-02T18:57:36.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LCncexKasPp2DRcd5Q31pJ1Bt60=', 'Content-Length': '0'} 2023-11-02T18:57:36.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079c119ee82184f92-006543f120-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.783 DEBUG:boto:b'ibtljn-62dummy1000falsedummy2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:36.784 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:36.785 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:36.786 DEBUG:boto:path=/ibtljn-62/?prefix=dummy 2023-11-02T18:57:36.787 DEBUG:boto:auth_path=/ibtljn-62/?prefix=dummy 2023-11-02T18:57:36.788 DEBUG:boto:Method: GET 2023-11-02T18:57:36.788 DEBUG:boto:Path: /ibtljn-62/?prefix=dummy 2023-11-02T18:57:36.789 DEBUG:boto:Data: 2023-11-02T18:57:36.790 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.791 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.792 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.792 DEBUG:boto:Params: {} 2023-11-02T18:57:36.793 DEBUG:boto:Token: None 2023-11-02T18:57:36.794 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/ 2023-11-02T18:57:36.795 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LCncexKasPp2DRcd5Q31pJ1Bt60= 2023-11-02T18:57:36.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LCncexKasPp2DRcd5Q31pJ1Bt60=', 'Content-Length': '0'} 2023-11-02T18:57:36.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f00cfb53085fe75b-006543f120-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.846 DEBUG:boto:b'ibtljn-62dummy1000falsedummy2023-11-02T18:52:47.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:36.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:57:36.848 DEBUG:boto:path=/ibtljn-62/obj11 2023-11-02T18:57:36.848 DEBUG:boto:auth_path=/ibtljn-62/obj11 2023-11-02T18:57:36.848 DEBUG:boto:path=/ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.849 DEBUG:boto:auth_path=/ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.849 DEBUG:boto:Method: GET 2023-11-02T18:57:36.850 DEBUG:boto:Path: /ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.850 DEBUG:boto:Data: 2023-11-02T18:57:36.851 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.851 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.851 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.852 DEBUG:boto:Params: {} 2023-11-02T18:57:36.852 DEBUG:boto:Token: None 2023-11-02T18:57:36.853 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.853 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T/bx6yadH19oLgu95eP7t/4QqDw= 2023-11-02T18:57:36.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T/bx6yadH19oLgu95eP7t/4QqDw=', 'Content-Length': '0'} 2023-11-02T18:57:36.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4e65e87acf9b20f-006543f120-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.858 DEBUG:boto:path=/ibtljn-62/obj11 2023-11-02T18:57:36.859 DEBUG:boto:auth_path=/ibtljn-62/obj11 2023-11-02T18:57:36.859 DEBUG:boto:path=/ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.860 DEBUG:boto:auth_path=/ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.860 DEBUG:boto:Method: GET 2023-11-02T18:57:36.861 DEBUG:boto:Path: /ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.861 DEBUG:boto:Data: 2023-11-02T18:57:36.861 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.862 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.862 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.863 DEBUG:boto:Params: {} 2023-11-02T18:57:36.863 DEBUG:boto:Token: None 2023-11-02T18:57:36.864 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.864 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T/bx6yadH19oLgu95eP7t/4QqDw= 2023-11-02T18:57:36.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T/bx6yadH19oLgu95eP7t/4QqDw=', 'Content-Length': '0'} 2023-11-02T18:57:36.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d22b92730c91b74-006543f120-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.869 DEBUG:boto:path=/ibtljn-62/obj11 2023-11-02T18:57:36.869 DEBUG:boto:auth_path=/ibtljn-62/obj11 2023-11-02T18:57:36.869 DEBUG:boto:path=/ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.870 DEBUG:boto:auth_path=/ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.870 DEBUG:boto:Method: HEAD 2023-11-02T18:57:36.871 DEBUG:boto:Path: /ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.871 DEBUG:boto:Data: 2023-11-02T18:57:36.872 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.872 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.873 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.873 DEBUG:boto:Params: {} 2023-11-02T18:57:36.874 DEBUG:boto:Token: None 2023-11-02T18:57:36.874 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.874 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/q0CaTF/rCHPl1N3VvDJ94gt5T0= 2023-11-02T18:57:36.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/q0CaTF/rCHPl1N3VvDJ94gt5T0=', 'Content-Length': '0'} 2023-11-02T18:57:36.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4985b5fcacfa23f-006543f120-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.878 DEBUG:boto:path=/ibtljn-62/obj11 2023-11-02T18:57:36.879 DEBUG:boto:auth_path=/ibtljn-62/obj11 2023-11-02T18:57:36.879 DEBUG:boto:path=/ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.880 DEBUG:boto:auth_path=/ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.880 DEBUG:boto:Method: HEAD 2023-11-02T18:57:36.881 DEBUG:boto:Path: /ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.881 DEBUG:boto:Data: 2023-11-02T18:57:36.881 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.882 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.882 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.883 DEBUG:boto:Params: {} 2023-11-02T18:57:36.883 DEBUG:boto:Token: None 2023-11-02T18:57:36.884 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.884 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/q0CaTF/rCHPl1N3VvDJ94gt5T0= 2023-11-02T18:57:36.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/q0CaTF/rCHPl1N3VvDJ94gt5T0=', 'Content-Length': '0'} 2023-11-02T18:57:36.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bd74331139cd38d-006543f120-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:57:36.889 DEBUG:boto:path=/ibtljn-62/obj11 2023-11-02T18:57:36.889 DEBUG:boto:auth_path=/ibtljn-62/obj11 2023-11-02T18:57:36.889 DEBUG:boto:path=/ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.890 DEBUG:boto:auth_path=/ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.890 DEBUG:boto:Method: GET 2023-11-02T18:57:36.891 DEBUG:boto:Path: /ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.891 DEBUG:boto:Data: 2023-11-02T18:57:36.892 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.892 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.893 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.893 DEBUG:boto:Params: {} 2023-11-02T18:57:36.893 DEBUG:boto:Token: None 2023-11-02T18:57:36.894 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.895 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T/bx6yadH19oLgu95eP7t/4QqDw= 2023-11-02T18:57:36.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T/bx6yadH19oLgu95eP7t/4QqDw=', 'Content-Length': '0'} 2023-11-02T18:57:36.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b744082ccbc806-006543f120-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.899 DEBUG:boto:path=/ibtljn-62/obj11 2023-11-02T18:57:36.899 DEBUG:boto:auth_path=/ibtljn-62/obj11 2023-11-02T18:57:36.900 DEBUG:boto:path=/ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.900 DEBUG:boto:auth_path=/ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.901 DEBUG:boto:Method: GET 2023-11-02T18:57:36.901 DEBUG:boto:Path: /ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.901 DEBUG:boto:Data: 2023-11-02T18:57:36.902 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.902 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.903 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.903 DEBUG:boto:Params: {} 2023-11-02T18:57:36.904 DEBUG:boto:Token: None 2023-11-02T18:57:36.904 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/obj11?versionId=null 2023-11-02T18:57:36.905 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T/bx6yadH19oLgu95eP7t/4QqDw= 2023-11-02T18:57:36.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T/bx6yadH19oLgu95eP7t/4QqDw=', 'Content-Length': '0'} 2023-11-02T18:57:36.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000819c63b57e3d0374-006543f120-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.954 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:36.955 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:36.955 DEBUG:boto:path=/ibtljn-62/?prefix=obj11 2023-11-02T18:57:36.956 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj11 2023-11-02T18:57:36.956 DEBUG:boto:Method: GET 2023-11-02T18:57:36.957 DEBUG:boto:Path: /ibtljn-62/?prefix=obj11 2023-11-02T18:57:36.957 DEBUG:boto:Data: 2023-11-02T18:57:36.958 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.958 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.959 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.959 DEBUG:boto:Params: {} 2023-11-02T18:57:36.959 DEBUG:boto:Token: None 2023-11-02T18:57:36.960 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/ 2023-11-02T18:57:36.960 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LCncexKasPp2DRcd5Q31pJ1Bt60= 2023-11-02T18:57:36.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LCncexKasPp2DRcd5Q31pJ1Bt60=', 'Content-Length': '0'} 2023-11-02T18:57:36.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc3daeb77a3f35d1-006543f120-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.967 DEBUG:boto:b'ibtljn-62obj111000falseobj112023-11-02T18:55:24.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:36.967 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:36.968 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:36.968 DEBUG:boto:path=/ibtljn-62/?prefix=obj11 2023-11-02T18:57:36.969 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj11 2023-11-02T18:57:36.969 DEBUG:boto:Method: GET 2023-11-02T18:57:36.970 DEBUG:boto:Path: /ibtljn-62/?prefix=obj11 2023-11-02T18:57:36.970 DEBUG:boto:Data: 2023-11-02T18:57:36.971 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.971 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.971 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.972 DEBUG:boto:Params: {} 2023-11-02T18:57:36.972 DEBUG:boto:Token: None 2023-11-02T18:57:36.973 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/ 2023-11-02T18:57:36.973 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LCncexKasPp2DRcd5Q31pJ1Bt60= 2023-11-02T18:57:36.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LCncexKasPp2DRcd5Q31pJ1Bt60=', 'Content-Length': '0'} 2023-11-02T18:57:36.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001228e17eab353e57-006543f120-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.979 DEBUG:boto:b'ibtljn-62obj111000falseobj112023-11-02T18:55:24.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:36.980 DEBUG:boto:path=/ibtljn-62/obj11 2023-11-02T18:57:36.980 DEBUG:boto:auth_path=/ibtljn-62/obj11 2023-11-02T18:57:36.981 DEBUG:boto:Method: HEAD 2023-11-02T18:57:36.981 DEBUG:boto:Path: /ibtljn-62/obj11 2023-11-02T18:57:36.981 DEBUG:boto:Data: 2023-11-02T18:57:36.982 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.982 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:36.983 DEBUG:boto:Port: 8001 2023-11-02T18:57:36.983 DEBUG:boto:Params: {} 2023-11-02T18:57:36.984 DEBUG:boto:Token: None 2023-11-02T18:57:36.984 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/obj11 2023-11-02T18:57:36.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+Jto7cZxvvvA28Neu8ja0I34qi0= 2023-11-02T18:57:36.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+Jto7cZxvvvA28Neu8ja0I34qi0=', 'Content-Length': '0'} 2023-11-02T18:57:36.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bfade3eb63f3c08-006543f120-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.989 DEBUG:boto:path=/ibtljn-62/obj11 2023-11-02T18:57:36.989 DEBUG:boto:auth_path=/ibtljn-62/obj11 2023-11-02T18:57:36.990 DEBUG:boto:Method: HEAD 2023-11-02T18:57:36.990 DEBUG:boto:Path: /ibtljn-62/obj11 2023-11-02T18:57:36.991 DEBUG:boto:Data: 2023-11-02T18:57:36.991 DEBUG:boto:Headers: {} 2023-11-02T18:57:36.991 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:36.992 DEBUG:boto:Port: 8000 2023-11-02T18:57:36.992 DEBUG:boto:Params: {} 2023-11-02T18:57:36.993 DEBUG:boto:Token: None 2023-11-02T18:57:36.993 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:36 GMT /ibtljn-62/obj11 2023-11-02T18:57:36.993 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+Jto7cZxvvvA28Neu8ja0I34qi0= 2023-11-02T18:57:36.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+Jto7cZxvvvA28Neu8ja0I34qi0=', 'Content-Length': '0'} 2023-11-02T18:57:36.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062cdadbb4f5c96a4-006543f120-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:36.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T18:57:36.998 DEBUG:boto:path=/ibtljn-62/obj11 2023-11-02T18:57:36.999 DEBUG:boto:auth_path=/ibtljn-62/obj11 2023-11-02T18:57:36.999 DEBUG:boto:Method: GET 2023-11-02T18:57:36.999 DEBUG:boto:Path: /ibtljn-62/obj11 2023-11-02T18:57:37.000 DEBUG:boto:Data: 2023-11-02T18:57:37.000 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.001 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.001 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.002 DEBUG:boto:Params: {} 2023-11-02T18:57:37.002 DEBUG:boto:Token: None 2023-11-02T18:57:37.003 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj11 2023-11-02T18:57:37.003 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3rRBl814woZ11Wsuvf+AkW2khR8= 2023-11-02T18:57:37.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3rRBl814woZ11Wsuvf+AkW2khR8=', 'Content-Length': '0'} 2023-11-02T18:57:37.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000279c8cb88f47183-006543f120-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.054 DEBUG:boto:path=/ibtljn-62/obj11 2023-11-02T18:57:37.055 DEBUG:boto:auth_path=/ibtljn-62/obj11 2023-11-02T18:57:37.055 DEBUG:boto:Method: GET 2023-11-02T18:57:37.056 DEBUG:boto:Path: /ibtljn-62/obj11 2023-11-02T18:57:37.056 DEBUG:boto:Data: 2023-11-02T18:57:37.057 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.057 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.058 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.058 DEBUG:boto:Params: {} 2023-11-02T18:57:37.059 DEBUG:boto:Token: None 2023-11-02T18:57:37.059 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj11 2023-11-02T18:57:37.060 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3rRBl814woZ11Wsuvf+AkW2khR8= 2023-11-02T18:57:37.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3rRBl814woZ11Wsuvf+AkW2khR8=', 'Content-Length': '0'} 2023-11-02T18:57:37.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000514542b86f2b0cca-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.064 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.065 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.065 DEBUG:boto:path=/ibtljn-62/?prefix=obj11 2023-11-02T18:57:37.065 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj11 2023-11-02T18:57:37.066 DEBUG:boto:Method: GET 2023-11-02T18:57:37.066 DEBUG:boto:Path: /ibtljn-62/?prefix=obj11 2023-11-02T18:57:37.067 DEBUG:boto:Data: 2023-11-02T18:57:37.067 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.068 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.068 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.069 DEBUG:boto:Params: {} 2023-11-02T18:57:37.069 DEBUG:boto:Token: None 2023-11-02T18:57:37.069 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.070 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008035dd479fcfdc38-006543f121-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.076 DEBUG:boto:b'ibtljn-62obj111000falseobj112023-11-02T18:55:24.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.077 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.078 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.078 DEBUG:boto:path=/ibtljn-62/?prefix=obj11 2023-11-02T18:57:37.079 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj11 2023-11-02T18:57:37.079 DEBUG:boto:Method: GET 2023-11-02T18:57:37.079 DEBUG:boto:Path: /ibtljn-62/?prefix=obj11 2023-11-02T18:57:37.080 DEBUG:boto:Data: 2023-11-02T18:57:37.080 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.081 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.081 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.082 DEBUG:boto:Params: {} 2023-11-02T18:57:37.082 DEBUG:boto:Token: None 2023-11-02T18:57:37.083 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.083 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000de506943691f2ae-006543f121-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.130 DEBUG:boto:b'ibtljn-62obj111000falseobj112023-11-02T18:55:24.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:57:37.132 DEBUG:boto:path=/ibtljn-62/obj22 2023-11-02T18:57:37.132 DEBUG:boto:auth_path=/ibtljn-62/obj22 2023-11-02T18:57:37.133 DEBUG:boto:path=/ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.133 DEBUG:boto:auth_path=/ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.133 DEBUG:boto:Method: GET 2023-11-02T18:57:37.134 DEBUG:boto:Path: /ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.134 DEBUG:boto:Data: 2023-11-02T18:57:37.135 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.135 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.136 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.136 DEBUG:boto:Params: {} 2023-11-02T18:57:37.136 DEBUG:boto:Token: None 2023-11-02T18:57:37.137 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.137 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2JIDGdU3rymYTD9eob96kxvV4H4= 2023-11-02T18:57:37.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2JIDGdU3rymYTD9eob96kxvV4H4=', 'Content-Length': '0'} 2023-11-02T18:57:37.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000338f8632d9299bed-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.142 DEBUG:boto:path=/ibtljn-62/obj22 2023-11-02T18:57:37.143 DEBUG:boto:auth_path=/ibtljn-62/obj22 2023-11-02T18:57:37.143 DEBUG:boto:path=/ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.143 DEBUG:boto:auth_path=/ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.144 DEBUG:boto:Method: GET 2023-11-02T18:57:37.144 DEBUG:boto:Path: /ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.144 DEBUG:boto:Data: 2023-11-02T18:57:37.145 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.145 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.146 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.146 DEBUG:boto:Params: {} 2023-11-02T18:57:37.147 DEBUG:boto:Token: None 2023-11-02T18:57:37.147 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.148 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2JIDGdU3rymYTD9eob96kxvV4H4= 2023-11-02T18:57:37.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2JIDGdU3rymYTD9eob96kxvV4H4=', 'Content-Length': '0'} 2023-11-02T18:57:37.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009f69d8993e7321d-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.152 DEBUG:boto:path=/ibtljn-62/obj22 2023-11-02T18:57:37.152 DEBUG:boto:auth_path=/ibtljn-62/obj22 2023-11-02T18:57:37.153 DEBUG:boto:path=/ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.153 DEBUG:boto:auth_path=/ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.154 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.154 DEBUG:boto:Path: /ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.155 DEBUG:boto:Data: 2023-11-02T18:57:37.155 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.155 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.156 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.156 DEBUG:boto:Params: {} 2023-11-02T18:57:37.157 DEBUG:boto:Token: None 2023-11-02T18:57:37.157 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.158 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JNx0+oDd1bEZ2klmIMq/9pYIJ5A= 2023-11-02T18:57:37.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JNx0+oDd1bEZ2klmIMq/9pYIJ5A=', 'Content-Length': '0'} 2023-11-02T18:57:37.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005698197add02adf9-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.162 DEBUG:boto:path=/ibtljn-62/obj22 2023-11-02T18:57:37.162 DEBUG:boto:auth_path=/ibtljn-62/obj22 2023-11-02T18:57:37.163 DEBUG:boto:path=/ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.163 DEBUG:boto:auth_path=/ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.164 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.164 DEBUG:boto:Path: /ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.164 DEBUG:boto:Data: 2023-11-02T18:57:37.165 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.165 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.166 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.166 DEBUG:boto:Params: {} 2023-11-02T18:57:37.166 DEBUG:boto:Token: None 2023-11-02T18:57:37.167 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.167 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JNx0+oDd1bEZ2klmIMq/9pYIJ5A= 2023-11-02T18:57:37.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JNx0+oDd1bEZ2klmIMq/9pYIJ5A=', 'Content-Length': '0'} 2023-11-02T18:57:37.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edb028cd59d0b9d8-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:57:37.172 DEBUG:boto:path=/ibtljn-62/obj22 2023-11-02T18:57:37.173 DEBUG:boto:auth_path=/ibtljn-62/obj22 2023-11-02T18:57:37.173 DEBUG:boto:path=/ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.173 DEBUG:boto:auth_path=/ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.174 DEBUG:boto:Method: GET 2023-11-02T18:57:37.174 DEBUG:boto:Path: /ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.175 DEBUG:boto:Data: 2023-11-02T18:57:37.175 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.176 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.176 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.177 DEBUG:boto:Params: {} 2023-11-02T18:57:37.177 DEBUG:boto:Token: None 2023-11-02T18:57:37.178 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.178 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2JIDGdU3rymYTD9eob96kxvV4H4= 2023-11-02T18:57:37.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2JIDGdU3rymYTD9eob96kxvV4H4=', 'Content-Length': '0'} 2023-11-02T18:57:37.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd5ab5a1472d46aa-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.182 DEBUG:boto:path=/ibtljn-62/obj22 2023-11-02T18:57:37.183 DEBUG:boto:auth_path=/ibtljn-62/obj22 2023-11-02T18:57:37.183 DEBUG:boto:path=/ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.184 DEBUG:boto:auth_path=/ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.184 DEBUG:boto:Method: GET 2023-11-02T18:57:37.185 DEBUG:boto:Path: /ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.185 DEBUG:boto:Data: 2023-11-02T18:57:37.185 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.186 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.186 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.187 DEBUG:boto:Params: {} 2023-11-02T18:57:37.187 DEBUG:boto:Token: None 2023-11-02T18:57:37.188 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj22?versionId=null 2023-11-02T18:57:37.188 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2JIDGdU3rymYTD9eob96kxvV4H4= 2023-11-02T18:57:37.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2JIDGdU3rymYTD9eob96kxvV4H4=', 'Content-Length': '0'} 2023-11-02T18:57:37.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005689926169afe2e-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.234 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.235 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.235 DEBUG:boto:path=/ibtljn-62/?prefix=obj22 2023-11-02T18:57:37.236 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj22 2023-11-02T18:57:37.236 DEBUG:boto:Method: GET 2023-11-02T18:57:37.237 DEBUG:boto:Path: /ibtljn-62/?prefix=obj22 2023-11-02T18:57:37.237 DEBUG:boto:Data: 2023-11-02T18:57:37.237 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.238 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.238 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.239 DEBUG:boto:Params: {} 2023-11-02T18:57:37.239 DEBUG:boto:Token: None 2023-11-02T18:57:37.240 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.240 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000921c3c15aeeac020-006543f121-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.247 DEBUG:boto:b'ibtljn-62obj221000falseobj222023-11-02T18:55:24.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.247 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.248 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.248 DEBUG:boto:path=/ibtljn-62/?prefix=obj22 2023-11-02T18:57:37.249 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj22 2023-11-02T18:57:37.249 DEBUG:boto:Method: GET 2023-11-02T18:57:37.249 DEBUG:boto:Path: /ibtljn-62/?prefix=obj22 2023-11-02T18:57:37.250 DEBUG:boto:Data: 2023-11-02T18:57:37.250 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.251 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.251 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.252 DEBUG:boto:Params: {} 2023-11-02T18:57:37.252 DEBUG:boto:Token: None 2023-11-02T18:57:37.252 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.253 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5aaa70e76d410ef-006543f121-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.259 DEBUG:boto:b'ibtljn-62obj221000falseobj222023-11-02T18:55:24.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.260 DEBUG:boto:path=/ibtljn-62/obj22 2023-11-02T18:57:37.260 DEBUG:boto:auth_path=/ibtljn-62/obj22 2023-11-02T18:57:37.261 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.261 DEBUG:boto:Path: /ibtljn-62/obj22 2023-11-02T18:57:37.261 DEBUG:boto:Data: 2023-11-02T18:57:37.262 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.262 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.263 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.263 DEBUG:boto:Params: {} 2023-11-02T18:57:37.264 DEBUG:boto:Token: None 2023-11-02T18:57:37.264 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj22 2023-11-02T18:57:37.265 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Kilcbc+KRE0yZLSBeqfcf0QY9NQ= 2023-11-02T18:57:37.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Kilcbc+KRE0yZLSBeqfcf0QY9NQ=', 'Content-Length': '0'} 2023-11-02T18:57:37.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e47eedff00903a4-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.269 DEBUG:boto:path=/ibtljn-62/obj22 2023-11-02T18:57:37.269 DEBUG:boto:auth_path=/ibtljn-62/obj22 2023-11-02T18:57:37.269 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.270 DEBUG:boto:Path: /ibtljn-62/obj22 2023-11-02T18:57:37.270 DEBUG:boto:Data: 2023-11-02T18:57:37.271 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.271 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.272 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.272 DEBUG:boto:Params: {} 2023-11-02T18:57:37.272 DEBUG:boto:Token: None 2023-11-02T18:57:37.273 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj22 2023-11-02T18:57:37.273 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Kilcbc+KRE0yZLSBeqfcf0QY9NQ= 2023-11-02T18:57:37.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Kilcbc+KRE0yZLSBeqfcf0QY9NQ=', 'Content-Length': '0'} 2023-11-02T18:57:37.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003ec351d0175770e-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T18:57:37.278 DEBUG:boto:path=/ibtljn-62/obj22 2023-11-02T18:57:37.278 DEBUG:boto:auth_path=/ibtljn-62/obj22 2023-11-02T18:57:37.279 DEBUG:boto:Method: GET 2023-11-02T18:57:37.279 DEBUG:boto:Path: /ibtljn-62/obj22 2023-11-02T18:57:37.280 DEBUG:boto:Data: 2023-11-02T18:57:37.280 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.280 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.281 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.281 DEBUG:boto:Params: {} 2023-11-02T18:57:37.282 DEBUG:boto:Token: None 2023-11-02T18:57:37.282 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj22 2023-11-02T18:57:37.283 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sBqKRVVx1zwyi945J/XjxSGvrEE= 2023-11-02T18:57:37.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sBqKRVVx1zwyi945J/XjxSGvrEE=', 'Content-Length': '0'} 2023-11-02T18:57:37.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0af3aac3daffdc9-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.330 DEBUG:boto:path=/ibtljn-62/obj22 2023-11-02T18:57:37.331 DEBUG:boto:auth_path=/ibtljn-62/obj22 2023-11-02T18:57:37.331 DEBUG:boto:Method: GET 2023-11-02T18:57:37.332 DEBUG:boto:Path: /ibtljn-62/obj22 2023-11-02T18:57:37.333 DEBUG:boto:Data: 2023-11-02T18:57:37.333 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.334 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.334 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.335 DEBUG:boto:Params: {} 2023-11-02T18:57:37.335 DEBUG:boto:Token: None 2023-11-02T18:57:37.336 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj22 2023-11-02T18:57:37.336 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sBqKRVVx1zwyi945J/XjxSGvrEE= 2023-11-02T18:57:37.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sBqKRVVx1zwyi945J/XjxSGvrEE=', 'Content-Length': '0'} 2023-11-02T18:57:37.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003061133e28e8fed7-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.341 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.341 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.342 DEBUG:boto:path=/ibtljn-62/?prefix=obj22 2023-11-02T18:57:37.342 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj22 2023-11-02T18:57:37.343 DEBUG:boto:Method: GET 2023-11-02T18:57:37.343 DEBUG:boto:Path: /ibtljn-62/?prefix=obj22 2023-11-02T18:57:37.344 DEBUG:boto:Data: 2023-11-02T18:57:37.344 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.344 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.345 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.345 DEBUG:boto:Params: {} 2023-11-02T18:57:37.346 DEBUG:boto:Token: None 2023-11-02T18:57:37.346 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.347 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0f48b5c0a03c36a-006543f121-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.355 DEBUG:boto:b'ibtljn-62obj221000falseobj222023-11-02T18:55:24.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.355 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.356 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.357 DEBUG:boto:path=/ibtljn-62/?prefix=obj22 2023-11-02T18:57:37.357 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj22 2023-11-02T18:57:37.358 DEBUG:boto:Method: GET 2023-11-02T18:57:37.358 DEBUG:boto:Path: /ibtljn-62/?prefix=obj22 2023-11-02T18:57:37.359 DEBUG:boto:Data: 2023-11-02T18:57:37.359 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.359 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.360 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.360 DEBUG:boto:Params: {} 2023-11-02T18:57:37.361 DEBUG:boto:Token: None 2023-11-02T18:57:37.361 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.362 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc7eea58ee9e1743-006543f121-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.410 DEBUG:boto:b'ibtljn-62obj221000falseobj222023-11-02T18:55:24.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:57:37.412 DEBUG:boto:path=/ibtljn-62/obj33 2023-11-02T18:57:37.412 DEBUG:boto:auth_path=/ibtljn-62/obj33 2023-11-02T18:57:37.412 DEBUG:boto:path=/ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.413 DEBUG:boto:auth_path=/ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.413 DEBUG:boto:Method: GET 2023-11-02T18:57:37.414 DEBUG:boto:Path: /ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.414 DEBUG:boto:Data: 2023-11-02T18:57:37.415 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.415 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.416 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.416 DEBUG:boto:Params: {} 2023-11-02T18:57:37.417 DEBUG:boto:Token: None 2023-11-02T18:57:37.417 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.418 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J0LClvf91b+v9zNyfNkqnfcGJzk= 2023-11-02T18:57:37.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J0LClvf91b+v9zNyfNkqnfcGJzk=', 'Content-Length': '0'} 2023-11-02T18:57:37.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e245a0d50995e28-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.423 DEBUG:boto:path=/ibtljn-62/obj33 2023-11-02T18:57:37.424 DEBUG:boto:auth_path=/ibtljn-62/obj33 2023-11-02T18:57:37.425 DEBUG:boto:path=/ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.425 DEBUG:boto:auth_path=/ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.426 DEBUG:boto:Method: GET 2023-11-02T18:57:37.427 DEBUG:boto:Path: /ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.428 DEBUG:boto:Data: 2023-11-02T18:57:37.428 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.429 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.430 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.431 DEBUG:boto:Params: {} 2023-11-02T18:57:37.431 DEBUG:boto:Token: None 2023-11-02T18:57:37.432 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.432 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J0LClvf91b+v9zNyfNkqnfcGJzk= 2023-11-02T18:57:37.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J0LClvf91b+v9zNyfNkqnfcGJzk=', 'Content-Length': '0'} 2023-11-02T18:57:37.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff9ecb7e2615a1b-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.438 DEBUG:boto:path=/ibtljn-62/obj33 2023-11-02T18:57:37.438 DEBUG:boto:auth_path=/ibtljn-62/obj33 2023-11-02T18:57:37.439 DEBUG:boto:path=/ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.439 DEBUG:boto:auth_path=/ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.439 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.440 DEBUG:boto:Path: /ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.440 DEBUG:boto:Data: 2023-11-02T18:57:37.441 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.441 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.441 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.442 DEBUG:boto:Params: {} 2023-11-02T18:57:37.442 DEBUG:boto:Token: None 2023-11-02T18:57:37.443 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.443 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mEca/iU7NE7+E1jEWdeboOFc2pA= 2023-11-02T18:57:37.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mEca/iU7NE7+E1jEWdeboOFc2pA=', 'Content-Length': '0'} 2023-11-02T18:57:37.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9127ed9d645b893-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.447 DEBUG:boto:path=/ibtljn-62/obj33 2023-11-02T18:57:37.448 DEBUG:boto:auth_path=/ibtljn-62/obj33 2023-11-02T18:57:37.448 DEBUG:boto:path=/ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.449 DEBUG:boto:auth_path=/ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.449 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.449 DEBUG:boto:Path: /ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.450 DEBUG:boto:Data: 2023-11-02T18:57:37.451 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.451 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.452 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.452 DEBUG:boto:Params: {} 2023-11-02T18:57:37.452 DEBUG:boto:Token: None 2023-11-02T18:57:37.453 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.453 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mEca/iU7NE7+E1jEWdeboOFc2pA= 2023-11-02T18:57:37.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mEca/iU7NE7+E1jEWdeboOFc2pA=', 'Content-Length': '0'} 2023-11-02T18:57:37.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe73fd34681ce5d0-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:57:37.458 DEBUG:boto:path=/ibtljn-62/obj33 2023-11-02T18:57:37.458 DEBUG:boto:auth_path=/ibtljn-62/obj33 2023-11-02T18:57:37.459 DEBUG:boto:path=/ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.459 DEBUG:boto:auth_path=/ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.459 DEBUG:boto:Method: GET 2023-11-02T18:57:37.460 DEBUG:boto:Path: /ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.460 DEBUG:boto:Data: 2023-11-02T18:57:37.461 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.461 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.461 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.462 DEBUG:boto:Params: {} 2023-11-02T18:57:37.462 DEBUG:boto:Token: None 2023-11-02T18:57:37.463 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.463 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J0LClvf91b+v9zNyfNkqnfcGJzk= 2023-11-02T18:57:37.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J0LClvf91b+v9zNyfNkqnfcGJzk=', 'Content-Length': '0'} 2023-11-02T18:57:37.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c05df3efaf22061-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.510 DEBUG:boto:path=/ibtljn-62/obj33 2023-11-02T18:57:37.511 DEBUG:boto:auth_path=/ibtljn-62/obj33 2023-11-02T18:57:37.511 DEBUG:boto:path=/ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.512 DEBUG:boto:auth_path=/ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.512 DEBUG:boto:Method: GET 2023-11-02T18:57:37.513 DEBUG:boto:Path: /ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.513 DEBUG:boto:Data: 2023-11-02T18:57:37.514 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.514 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.514 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.515 DEBUG:boto:Params: {} 2023-11-02T18:57:37.515 DEBUG:boto:Token: None 2023-11-02T18:57:37.516 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj33?versionId=null 2023-11-02T18:57:37.516 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J0LClvf91b+v9zNyfNkqnfcGJzk= 2023-11-02T18:57:37.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J0LClvf91b+v9zNyfNkqnfcGJzk=', 'Content-Length': '0'} 2023-11-02T18:57:37.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da6a882fa6870fd-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.521 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.522 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.522 DEBUG:boto:path=/ibtljn-62/?prefix=obj33 2023-11-02T18:57:37.523 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj33 2023-11-02T18:57:37.523 DEBUG:boto:Method: GET 2023-11-02T18:57:37.524 DEBUG:boto:Path: /ibtljn-62/?prefix=obj33 2023-11-02T18:57:37.524 DEBUG:boto:Data: 2023-11-02T18:57:37.525 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.525 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.525 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.526 DEBUG:boto:Params: {} 2023-11-02T18:57:37.526 DEBUG:boto:Token: None 2023-11-02T18:57:37.527 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.527 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f470cb47e0a467ae-006543f121-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.533 DEBUG:boto:b'ibtljn-62obj331000falseobj332023-11-02T18:55:24.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.534 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.535 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.535 DEBUG:boto:path=/ibtljn-62/?prefix=obj33 2023-11-02T18:57:37.536 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj33 2023-11-02T18:57:37.536 DEBUG:boto:Method: GET 2023-11-02T18:57:37.537 DEBUG:boto:Path: /ibtljn-62/?prefix=obj33 2023-11-02T18:57:37.537 DEBUG:boto:Data: 2023-11-02T18:57:37.537 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.538 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.538 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.539 DEBUG:boto:Params: {} 2023-11-02T18:57:37.540 DEBUG:boto:Token: None 2023-11-02T18:57:37.541 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.541 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e16a8abcadd263b-006543f121-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.590 DEBUG:boto:b'ibtljn-62obj331000falseobj332023-11-02T18:55:24.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.591 DEBUG:boto:path=/ibtljn-62/obj33 2023-11-02T18:57:37.592 DEBUG:boto:auth_path=/ibtljn-62/obj33 2023-11-02T18:57:37.592 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.592 DEBUG:boto:Path: /ibtljn-62/obj33 2023-11-02T18:57:37.593 DEBUG:boto:Data: 2023-11-02T18:57:37.593 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.594 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.594 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.594 DEBUG:boto:Params: {} 2023-11-02T18:57:37.595 DEBUG:boto:Token: None 2023-11-02T18:57:37.595 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj33 2023-11-02T18:57:37.596 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OUftY/Wiq6Y+lSUz4z6b/NYr8UU= 2023-11-02T18:57:37.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OUftY/Wiq6Y+lSUz4z6b/NYr8UU=', 'Content-Length': '0'} 2023-11-02T18:57:37.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da451fcd7744ca47-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.600 DEBUG:boto:path=/ibtljn-62/obj33 2023-11-02T18:57:37.601 DEBUG:boto:auth_path=/ibtljn-62/obj33 2023-11-02T18:57:37.601 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.601 DEBUG:boto:Path: /ibtljn-62/obj33 2023-11-02T18:57:37.602 DEBUG:boto:Data: 2023-11-02T18:57:37.602 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.603 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.603 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.603 DEBUG:boto:Params: {} 2023-11-02T18:57:37.604 DEBUG:boto:Token: None 2023-11-02T18:57:37.604 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj33 2023-11-02T18:57:37.605 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OUftY/Wiq6Y+lSUz4z6b/NYr8UU= 2023-11-02T18:57:37.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OUftY/Wiq6Y+lSUz4z6b/NYr8UU=', 'Content-Length': '0'} 2023-11-02T18:57:37.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a120e02a663a033-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T18:57:37.609 DEBUG:boto:path=/ibtljn-62/obj33 2023-11-02T18:57:37.609 DEBUG:boto:auth_path=/ibtljn-62/obj33 2023-11-02T18:57:37.609 DEBUG:boto:Method: GET 2023-11-02T18:57:37.610 DEBUG:boto:Path: /ibtljn-62/obj33 2023-11-02T18:57:37.610 DEBUG:boto:Data: 2023-11-02T18:57:37.611 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.611 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.612 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.612 DEBUG:boto:Params: {} 2023-11-02T18:57:37.612 DEBUG:boto:Token: None 2023-11-02T18:57:37.613 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj33 2023-11-02T18:57:37.613 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z4O7OG4LErMEYZ4Qtg35BQ0h9LU= 2023-11-02T18:57:37.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z4O7OG4LErMEYZ4Qtg35BQ0h9LU=', 'Content-Length': '0'} 2023-11-02T18:57:37.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc7f5560a583262c-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.618 DEBUG:boto:path=/ibtljn-62/obj33 2023-11-02T18:57:37.618 DEBUG:boto:auth_path=/ibtljn-62/obj33 2023-11-02T18:57:37.619 DEBUG:boto:Method: GET 2023-11-02T18:57:37.619 DEBUG:boto:Path: /ibtljn-62/obj33 2023-11-02T18:57:37.620 DEBUG:boto:Data: 2023-11-02T18:57:37.620 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.621 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.621 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.622 DEBUG:boto:Params: {} 2023-11-02T18:57:37.622 DEBUG:boto:Token: None 2023-11-02T18:57:37.623 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj33 2023-11-02T18:57:37.623 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Z4O7OG4LErMEYZ4Qtg35BQ0h9LU= 2023-11-02T18:57:37.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z4O7OG4LErMEYZ4Qtg35BQ0h9LU=', 'Content-Length': '0'} 2023-11-02T18:57:37.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce62d1ec39ffd667-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.628 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.628 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.629 DEBUG:boto:path=/ibtljn-62/?prefix=obj33 2023-11-02T18:57:37.629 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj33 2023-11-02T18:57:37.630 DEBUG:boto:Method: GET 2023-11-02T18:57:37.630 DEBUG:boto:Path: /ibtljn-62/?prefix=obj33 2023-11-02T18:57:37.631 DEBUG:boto:Data: 2023-11-02T18:57:37.631 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.632 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.632 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.632 DEBUG:boto:Params: {} 2023-11-02T18:57:37.633 DEBUG:boto:Token: None 2023-11-02T18:57:37.633 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.634 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8983f6464761ecc-006543f121-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.641 DEBUG:boto:b'ibtljn-62obj331000falseobj332023-11-02T18:55:24.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.642 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.642 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.643 DEBUG:boto:path=/ibtljn-62/?prefix=obj33 2023-11-02T18:57:37.643 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj33 2023-11-02T18:57:37.644 DEBUG:boto:Method: GET 2023-11-02T18:57:37.645 DEBUG:boto:Path: /ibtljn-62/?prefix=obj33 2023-11-02T18:57:37.646 DEBUG:boto:Data: 2023-11-02T18:57:37.647 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.647 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.647 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.648 DEBUG:boto:Params: {} 2023-11-02T18:57:37.649 DEBUG:boto:Token: None 2023-11-02T18:57:37.649 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.650 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093ff556df0885f25-006543f121-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.702 DEBUG:boto:b'ibtljn-62obj331000falseobj332023-11-02T18:55:24.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:57:37.704 DEBUG:boto:path=/ibtljn-62/obj44 2023-11-02T18:57:37.704 DEBUG:boto:auth_path=/ibtljn-62/obj44 2023-11-02T18:57:37.704 DEBUG:boto:path=/ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.705 DEBUG:boto:auth_path=/ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.705 DEBUG:boto:Method: GET 2023-11-02T18:57:37.706 DEBUG:boto:Path: /ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.706 DEBUG:boto:Data: 2023-11-02T18:57:37.707 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.707 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.708 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.708 DEBUG:boto:Params: {} 2023-11-02T18:57:37.708 DEBUG:boto:Token: None 2023-11-02T18:57:37.709 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.710 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3V0+hb5SnWiR7HTnfPMLWCe2B+4= 2023-11-02T18:57:37.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3V0+hb5SnWiR7HTnfPMLWCe2B+4=', 'Content-Length': '0'} 2023-11-02T18:57:37.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900f9d8a22238733-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.715 DEBUG:boto:path=/ibtljn-62/obj44 2023-11-02T18:57:37.715 DEBUG:boto:auth_path=/ibtljn-62/obj44 2023-11-02T18:57:37.716 DEBUG:boto:path=/ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.716 DEBUG:boto:auth_path=/ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.717 DEBUG:boto:Method: GET 2023-11-02T18:57:37.717 DEBUG:boto:Path: /ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.718 DEBUG:boto:Data: 2023-11-02T18:57:37.718 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.719 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.719 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.720 DEBUG:boto:Params: {} 2023-11-02T18:57:37.720 DEBUG:boto:Token: None 2023-11-02T18:57:37.721 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3V0+hb5SnWiR7HTnfPMLWCe2B+4= 2023-11-02T18:57:37.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3V0+hb5SnWiR7HTnfPMLWCe2B+4=', 'Content-Length': '0'} 2023-11-02T18:57:37.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000400949c5c370abde-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.726 DEBUG:boto:path=/ibtljn-62/obj44 2023-11-02T18:57:37.726 DEBUG:boto:auth_path=/ibtljn-62/obj44 2023-11-02T18:57:37.727 DEBUG:boto:path=/ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.727 DEBUG:boto:auth_path=/ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.728 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.728 DEBUG:boto:Path: /ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.729 DEBUG:boto:Data: 2023-11-02T18:57:37.729 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.730 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.730 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.730 DEBUG:boto:Params: {} 2023-11-02T18:57:37.731 DEBUG:boto:Token: None 2023-11-02T18:57:37.731 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.732 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0DPA5zxl1QPtEOHZyjutJvdsyow= 2023-11-02T18:57:37.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0DPA5zxl1QPtEOHZyjutJvdsyow=', 'Content-Length': '0'} 2023-11-02T18:57:37.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000316cb35b693460f1-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.736 DEBUG:boto:path=/ibtljn-62/obj44 2023-11-02T18:57:37.736 DEBUG:boto:auth_path=/ibtljn-62/obj44 2023-11-02T18:57:37.737 DEBUG:boto:path=/ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.737 DEBUG:boto:auth_path=/ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.738 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.738 DEBUG:boto:Path: /ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.739 DEBUG:boto:Data: 2023-11-02T18:57:37.739 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.739 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.740 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.740 DEBUG:boto:Params: {} 2023-11-02T18:57:37.741 DEBUG:boto:Token: None 2023-11-02T18:57:37.741 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.742 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0DPA5zxl1QPtEOHZyjutJvdsyow= 2023-11-02T18:57:37.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0DPA5zxl1QPtEOHZyjutJvdsyow=', 'Content-Length': '0'} 2023-11-02T18:57:37.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2d2f83106010f4-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:57:37.746 DEBUG:boto:path=/ibtljn-62/obj44 2023-11-02T18:57:37.747 DEBUG:boto:auth_path=/ibtljn-62/obj44 2023-11-02T18:57:37.748 DEBUG:boto:path=/ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.748 DEBUG:boto:auth_path=/ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.748 DEBUG:boto:Method: GET 2023-11-02T18:57:37.749 DEBUG:boto:Path: /ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.749 DEBUG:boto:Data: 2023-11-02T18:57:37.750 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.750 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.751 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.751 DEBUG:boto:Params: {} 2023-11-02T18:57:37.752 DEBUG:boto:Token: None 2023-11-02T18:57:37.752 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.753 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3V0+hb5SnWiR7HTnfPMLWCe2B+4= 2023-11-02T18:57:37.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3V0+hb5SnWiR7HTnfPMLWCe2B+4=', 'Content-Length': '0'} 2023-11-02T18:57:37.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e0e0f5001e81641-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.798 DEBUG:boto:path=/ibtljn-62/obj44 2023-11-02T18:57:37.799 DEBUG:boto:auth_path=/ibtljn-62/obj44 2023-11-02T18:57:37.799 DEBUG:boto:path=/ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.800 DEBUG:boto:auth_path=/ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.800 DEBUG:boto:Method: GET 2023-11-02T18:57:37.801 DEBUG:boto:Path: /ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.801 DEBUG:boto:Data: 2023-11-02T18:57:37.801 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.802 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.802 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.803 DEBUG:boto:Params: {} 2023-11-02T18:57:37.803 DEBUG:boto:Token: None 2023-11-02T18:57:37.804 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj44?versionId=null 2023-11-02T18:57:37.804 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3V0+hb5SnWiR7HTnfPMLWCe2B+4= 2023-11-02T18:57:37.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3V0+hb5SnWiR7HTnfPMLWCe2B+4=', 'Content-Length': '0'} 2023-11-02T18:57:37.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f15d03b85dce468b-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.809 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.809 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.810 DEBUG:boto:path=/ibtljn-62/?prefix=obj44 2023-11-02T18:57:37.810 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj44 2023-11-02T18:57:37.811 DEBUG:boto:Method: GET 2023-11-02T18:57:37.811 DEBUG:boto:Path: /ibtljn-62/?prefix=obj44 2023-11-02T18:57:37.811 DEBUG:boto:Data: 2023-11-02T18:57:37.812 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.812 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.813 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.813 DEBUG:boto:Params: {} 2023-11-02T18:57:37.814 DEBUG:boto:Token: None 2023-11-02T18:57:37.814 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.815 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001065295601654753-006543f121-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.822 DEBUG:boto:b'ibtljn-62obj441000falseobj442023-11-02T18:55:24.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.822 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.823 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.823 DEBUG:boto:path=/ibtljn-62/?prefix=obj44 2023-11-02T18:57:37.824 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj44 2023-11-02T18:57:37.824 DEBUG:boto:Method: GET 2023-11-02T18:57:37.825 DEBUG:boto:Path: /ibtljn-62/?prefix=obj44 2023-11-02T18:57:37.825 DEBUG:boto:Data: 2023-11-02T18:57:37.825 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.826 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.826 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.827 DEBUG:boto:Params: {} 2023-11-02T18:57:37.827 DEBUG:boto:Token: None 2023-11-02T18:57:37.828 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.828 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6d41100fe03c9b9-006543f121-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.834 DEBUG:boto:b'ibtljn-62obj441000falseobj442023-11-02T18:55:24.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.835 DEBUG:boto:path=/ibtljn-62/obj44 2023-11-02T18:57:37.835 DEBUG:boto:auth_path=/ibtljn-62/obj44 2023-11-02T18:57:37.836 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.836 DEBUG:boto:Path: /ibtljn-62/obj44 2023-11-02T18:57:37.837 DEBUG:boto:Data: 2023-11-02T18:57:37.837 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.837 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.838 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.838 DEBUG:boto:Params: {} 2023-11-02T18:57:37.839 DEBUG:boto:Token: None 2023-11-02T18:57:37.839 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj44 2023-11-02T18:57:37.840 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pj/JxWVlVu8r/U0HBUAHOpP93Ek= 2023-11-02T18:57:37.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pj/JxWVlVu8r/U0HBUAHOpP93Ek=', 'Content-Length': '0'} 2023-11-02T18:57:37.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088398d3053e4b725-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.843 DEBUG:boto:path=/ibtljn-62/obj44 2023-11-02T18:57:37.844 DEBUG:boto:auth_path=/ibtljn-62/obj44 2023-11-02T18:57:37.844 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.845 DEBUG:boto:Path: /ibtljn-62/obj44 2023-11-02T18:57:37.845 DEBUG:boto:Data: 2023-11-02T18:57:37.845 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.846 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.846 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.847 DEBUG:boto:Params: {} 2023-11-02T18:57:37.847 DEBUG:boto:Token: None 2023-11-02T18:57:37.848 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj44 2023-11-02T18:57:37.848 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Pj/JxWVlVu8r/U0HBUAHOpP93Ek= 2023-11-02T18:57:37.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Pj/JxWVlVu8r/U0HBUAHOpP93Ek=', 'Content-Length': '0'} 2023-11-02T18:57:37.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044d82019fee7e2ba-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T18:57:37.853 DEBUG:boto:path=/ibtljn-62/obj44 2023-11-02T18:57:37.853 DEBUG:boto:auth_path=/ibtljn-62/obj44 2023-11-02T18:57:37.854 DEBUG:boto:Method: GET 2023-11-02T18:57:37.854 DEBUG:boto:Path: /ibtljn-62/obj44 2023-11-02T18:57:37.855 DEBUG:boto:Data: 2023-11-02T18:57:37.855 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.856 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.856 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.857 DEBUG:boto:Params: {} 2023-11-02T18:57:37.857 DEBUG:boto:Token: None 2023-11-02T18:57:37.858 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj44 2023-11-02T18:57:37.858 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hqLmkpe/gptBrJkSmmUD0zbquD8= 2023-11-02T18:57:37.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hqLmkpe/gptBrJkSmmUD0zbquD8=', 'Content-Length': '0'} 2023-11-02T18:57:37.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c11f0bb2b609bad-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.906 DEBUG:boto:path=/ibtljn-62/obj44 2023-11-02T18:57:37.907 DEBUG:boto:auth_path=/ibtljn-62/obj44 2023-11-02T18:57:37.907 DEBUG:boto:Method: GET 2023-11-02T18:57:37.908 DEBUG:boto:Path: /ibtljn-62/obj44 2023-11-02T18:57:37.908 DEBUG:boto:Data: 2023-11-02T18:57:37.909 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.909 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.910 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.910 DEBUG:boto:Params: {} 2023-11-02T18:57:37.911 DEBUG:boto:Token: None 2023-11-02T18:57:37.911 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj44 2023-11-02T18:57:37.912 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hqLmkpe/gptBrJkSmmUD0zbquD8= 2023-11-02T18:57:37.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hqLmkpe/gptBrJkSmmUD0zbquD8=', 'Content-Length': '0'} 2023-11-02T18:57:37.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a91785936a29d1c5-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.916 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.917 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.917 DEBUG:boto:path=/ibtljn-62/?prefix=obj44 2023-11-02T18:57:37.917 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj44 2023-11-02T18:57:37.918 DEBUG:boto:Method: GET 2023-11-02T18:57:37.918 DEBUG:boto:Path: /ibtljn-62/?prefix=obj44 2023-11-02T18:57:37.919 DEBUG:boto:Data: 2023-11-02T18:57:37.919 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.920 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.920 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.921 DEBUG:boto:Params: {} 2023-11-02T18:57:37.921 DEBUG:boto:Token: None 2023-11-02T18:57:37.922 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062abe7857d2bea8b-006543f121-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.928 DEBUG:boto:b'ibtljn-62obj441000falseobj442023-11-02T18:55:24.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.929 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:37.929 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:37.930 DEBUG:boto:path=/ibtljn-62/?prefix=obj44 2023-11-02T18:57:37.930 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj44 2023-11-02T18:57:37.931 DEBUG:boto:Method: GET 2023-11-02T18:57:37.931 DEBUG:boto:Path: /ibtljn-62/?prefix=obj44 2023-11-02T18:57:37.931 DEBUG:boto:Data: 2023-11-02T18:57:37.932 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.932 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.933 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.933 DEBUG:boto:Params: {} 2023-11-02T18:57:37.934 DEBUG:boto:Token: None 2023-11-02T18:57:37.934 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/ 2023-11-02T18:57:37.935 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4= 2023-11-02T18:57:37.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSOJL4tPcy6muO6JdTdtHZO4dC4=', 'Content-Length': '0'} 2023-11-02T18:57:37.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de87b8d446d39fac-006543f121-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.940 DEBUG:boto:b'ibtljn-62obj441000falseobj442023-11-02T18:55:24.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:37.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:57:37.941 DEBUG:boto:path=/ibtljn-62/obj55 2023-11-02T18:57:37.942 DEBUG:boto:auth_path=/ibtljn-62/obj55 2023-11-02T18:57:37.942 DEBUG:boto:path=/ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.943 DEBUG:boto:auth_path=/ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.943 DEBUG:boto:Method: GET 2023-11-02T18:57:37.944 DEBUG:boto:Path: /ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.944 DEBUG:boto:Data: 2023-11-02T18:57:37.945 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.945 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.945 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.946 DEBUG:boto:Params: {} 2023-11-02T18:57:37.947 DEBUG:boto:Token: None 2023-11-02T18:57:37.947 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.948 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CaB7N1OjRSwdmYbolijAC2ecmHk= 2023-11-02T18:57:37.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CaB7N1OjRSwdmYbolijAC2ecmHk=', 'Content-Length': '0'} 2023-11-02T18:57:37.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8618104b137a4c3-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.952 DEBUG:boto:path=/ibtljn-62/obj55 2023-11-02T18:57:37.952 DEBUG:boto:auth_path=/ibtljn-62/obj55 2023-11-02T18:57:37.953 DEBUG:boto:path=/ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.953 DEBUG:boto:auth_path=/ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.954 DEBUG:boto:Method: GET 2023-11-02T18:57:37.954 DEBUG:boto:Path: /ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.955 DEBUG:boto:Data: 2023-11-02T18:57:37.955 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.955 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.956 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.956 DEBUG:boto:Params: {} 2023-11-02T18:57:37.957 DEBUG:boto:Token: None 2023-11-02T18:57:37.957 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.958 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CaB7N1OjRSwdmYbolijAC2ecmHk= 2023-11-02T18:57:37.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CaB7N1OjRSwdmYbolijAC2ecmHk=', 'Content-Length': '0'} 2023-11-02T18:57:37.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024db733ff369b99a-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.962 DEBUG:boto:path=/ibtljn-62/obj55 2023-11-02T18:57:37.963 DEBUG:boto:auth_path=/ibtljn-62/obj55 2023-11-02T18:57:37.963 DEBUG:boto:path=/ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.964 DEBUG:boto:auth_path=/ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.964 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.964 DEBUG:boto:Path: /ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.965 DEBUG:boto:Data: 2023-11-02T18:57:37.965 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.966 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.966 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.967 DEBUG:boto:Params: {} 2023-11-02T18:57:37.967 DEBUG:boto:Token: None 2023-11-02T18:57:37.967 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.968 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LWSe+LUm/BGFy8U5UoV3GTXYAgY= 2023-11-02T18:57:37.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LWSe+LUm/BGFy8U5UoV3GTXYAgY=', 'Content-Length': '0'} 2023-11-02T18:57:37.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070c7d351dae7c7a1-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.972 DEBUG:boto:path=/ibtljn-62/obj55 2023-11-02T18:57:37.972 DEBUG:boto:auth_path=/ibtljn-62/obj55 2023-11-02T18:57:37.973 DEBUG:boto:path=/ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.973 DEBUG:boto:auth_path=/ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.974 DEBUG:boto:Method: HEAD 2023-11-02T18:57:37.974 DEBUG:boto:Path: /ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.975 DEBUG:boto:Data: 2023-11-02T18:57:37.975 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.975 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:37.976 DEBUG:boto:Port: 8000 2023-11-02T18:57:37.976 DEBUG:boto:Params: {} 2023-11-02T18:57:37.977 DEBUG:boto:Token: None 2023-11-02T18:57:37.977 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.978 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LWSe+LUm/BGFy8U5UoV3GTXYAgY= 2023-11-02T18:57:37.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LWSe+LUm/BGFy8U5UoV3GTXYAgY=', 'Content-Length': '0'} 2023-11-02T18:57:37.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000009ed0aff2b26383-006543f121-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:37.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:57:37.982 DEBUG:boto:path=/ibtljn-62/obj55 2023-11-02T18:57:37.982 DEBUG:boto:auth_path=/ibtljn-62/obj55 2023-11-02T18:57:37.983 DEBUG:boto:path=/ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.983 DEBUG:boto:auth_path=/ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.984 DEBUG:boto:Method: GET 2023-11-02T18:57:37.984 DEBUG:boto:Path: /ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.985 DEBUG:boto:Data: 2023-11-02T18:57:37.985 DEBUG:boto:Headers: {} 2023-11-02T18:57:37.986 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:37.986 DEBUG:boto:Port: 8001 2023-11-02T18:57:37.986 DEBUG:boto:Params: {} 2023-11-02T18:57:37.987 DEBUG:boto:Token: None 2023-11-02T18:57:37.987 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:37 GMT /ibtljn-62/obj55?versionId=null 2023-11-02T18:57:37.988 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CaB7N1OjRSwdmYbolijAC2ecmHk= 2023-11-02T18:57:37.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CaB7N1OjRSwdmYbolijAC2ecmHk=', 'Content-Length': '0'} 2023-11-02T18:57:37.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3592ea8daaa6af2-006543f121-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:38.034 DEBUG:boto:path=/ibtljn-62/obj55 2023-11-02T18:57:38.035 DEBUG:boto:auth_path=/ibtljn-62/obj55 2023-11-02T18:57:38.035 DEBUG:boto:path=/ibtljn-62/obj55?versionId=null 2023-11-02T18:57:38.036 DEBUG:boto:auth_path=/ibtljn-62/obj55?versionId=null 2023-11-02T18:57:38.036 DEBUG:boto:Method: GET 2023-11-02T18:57:38.037 DEBUG:boto:Path: /ibtljn-62/obj55?versionId=null 2023-11-02T18:57:38.037 DEBUG:boto:Data: 2023-11-02T18:57:38.038 DEBUG:boto:Headers: {} 2023-11-02T18:57:38.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:38.039 DEBUG:boto:Port: 8000 2023-11-02T18:57:38.039 DEBUG:boto:Params: {} 2023-11-02T18:57:38.040 DEBUG:boto:Token: None 2023-11-02T18:57:38.040 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:38 GMT /ibtljn-62/obj55?versionId=null 2023-11-02T18:57:38.041 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sD6jLMDDCgZAXqtOKpf3uT2WxdY= 2023-11-02T18:57:38.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sD6jLMDDCgZAXqtOKpf3uT2WxdY=', 'Content-Length': '0'} 2023-11-02T18:57:38.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5abbfcf46d01c8d-006543f122-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:38.045 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:38.046 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:38.046 DEBUG:boto:path=/ibtljn-62/?prefix=obj55 2023-11-02T18:57:38.047 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj55 2023-11-02T18:57:38.047 DEBUG:boto:Method: GET 2023-11-02T18:57:38.047 DEBUG:boto:Path: /ibtljn-62/?prefix=obj55 2023-11-02T18:57:38.048 DEBUG:boto:Data: 2023-11-02T18:57:38.048 DEBUG:boto:Headers: {} 2023-11-02T18:57:38.049 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:38.049 DEBUG:boto:Port: 8001 2023-11-02T18:57:38.049 DEBUG:boto:Params: {} 2023-11-02T18:57:38.050 DEBUG:boto:Token: None 2023-11-02T18:57:38.050 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:38 GMT /ibtljn-62/ 2023-11-02T18:57:38.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zQ6RXKfs64Los04ICjdar1liUMU= 2023-11-02T18:57:38.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zQ6RXKfs64Los04ICjdar1liUMU=', 'Content-Length': '0'} 2023-11-02T18:57:38.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e4312b5d1f74dc5-006543f122-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:38.057 DEBUG:boto:b'ibtljn-62obj551000falseobj552023-11-02T18:55:24.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:38.057 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:38.058 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:38.058 DEBUG:boto:path=/ibtljn-62/?prefix=obj55 2023-11-02T18:57:38.059 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj55 2023-11-02T18:57:38.059 DEBUG:boto:Method: GET 2023-11-02T18:57:38.060 DEBUG:boto:Path: /ibtljn-62/?prefix=obj55 2023-11-02T18:57:38.060 DEBUG:boto:Data: 2023-11-02T18:57:38.060 DEBUG:boto:Headers: {} 2023-11-02T18:57:38.061 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:38.061 DEBUG:boto:Port: 8000 2023-11-02T18:57:38.062 DEBUG:boto:Params: {} 2023-11-02T18:57:38.062 DEBUG:boto:Token: None 2023-11-02T18:57:38.063 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:38 GMT /ibtljn-62/ 2023-11-02T18:57:38.063 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zQ6RXKfs64Los04ICjdar1liUMU= 2023-11-02T18:57:38.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zQ6RXKfs64Los04ICjdar1liUMU=', 'Content-Length': '0'} 2023-11-02T18:57:38.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb0793cb70d4a85b-006543f122-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:38.069 DEBUG:boto:b'ibtljn-62obj551000falseobj552023-11-02T18:55:24.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:38.070 DEBUG:boto:path=/ibtljn-62/obj55 2023-11-02T18:57:38.071 DEBUG:boto:auth_path=/ibtljn-62/obj55 2023-11-02T18:57:38.071 DEBUG:boto:Method: HEAD 2023-11-02T18:57:38.071 DEBUG:boto:Path: /ibtljn-62/obj55 2023-11-02T18:57:38.072 DEBUG:boto:Data: 2023-11-02T18:57:38.072 DEBUG:boto:Headers: {} 2023-11-02T18:57:38.073 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:38.073 DEBUG:boto:Port: 8001 2023-11-02T18:57:38.074 DEBUG:boto:Params: {} 2023-11-02T18:57:38.074 DEBUG:boto:Token: None 2023-11-02T18:57:38.075 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:38 GMT /ibtljn-62/obj55 2023-11-02T18:57:38.075 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xp05nGpZZGdfyvApjvVEMxZerw4= 2023-11-02T18:57:38.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xp05nGpZZGdfyvApjvVEMxZerw4=', 'Content-Length': '0'} 2023-11-02T18:57:38.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbaeb137253274d7-006543f122-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:38.079 DEBUG:boto:path=/ibtljn-62/obj55 2023-11-02T18:57:38.080 DEBUG:boto:auth_path=/ibtljn-62/obj55 2023-11-02T18:57:38.080 DEBUG:boto:Method: HEAD 2023-11-02T18:57:38.080 DEBUG:boto:Path: /ibtljn-62/obj55 2023-11-02T18:57:38.081 DEBUG:boto:Data: 2023-11-02T18:57:38.081 DEBUG:boto:Headers: {} 2023-11-02T18:57:38.082 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:38.082 DEBUG:boto:Port: 8000 2023-11-02T18:57:38.083 DEBUG:boto:Params: {} 2023-11-02T18:57:38.083 DEBUG:boto:Token: None 2023-11-02T18:57:38.084 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:38 GMT /ibtljn-62/obj55 2023-11-02T18:57:38.084 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xp05nGpZZGdfyvApjvVEMxZerw4= 2023-11-02T18:57:38.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xp05nGpZZGdfyvApjvVEMxZerw4=', 'Content-Length': '0'} 2023-11-02T18:57:38.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9e84e9d1d652481-006543f122-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:38.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T18:57:38.089 DEBUG:boto:path=/ibtljn-62/obj55 2023-11-02T18:57:38.089 DEBUG:boto:auth_path=/ibtljn-62/obj55 2023-11-02T18:57:38.090 DEBUG:boto:Method: GET 2023-11-02T18:57:38.090 DEBUG:boto:Path: /ibtljn-62/obj55 2023-11-02T18:57:38.091 DEBUG:boto:Data: 2023-11-02T18:57:38.091 DEBUG:boto:Headers: {} 2023-11-02T18:57:38.092 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:38.092 DEBUG:boto:Port: 8001 2023-11-02T18:57:38.093 DEBUG:boto:Params: {} 2023-11-02T18:57:38.093 DEBUG:boto:Token: None 2023-11-02T18:57:38.093 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:38 GMT /ibtljn-62/obj55 2023-11-02T18:57:38.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VUPx6nqeQq5UdnmUHAzS8mhDcSo= 2023-11-02T18:57:38.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VUPx6nqeQq5UdnmUHAzS8mhDcSo=', 'Content-Length': '0'} 2023-11-02T18:57:38.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-62:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf9e3d1c7d06a530-006543f122-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:38.138 DEBUG:boto:path=/ibtljn-62/obj55 2023-11-02T18:57:38.139 DEBUG:boto:auth_path=/ibtljn-62/obj55 2023-11-02T18:57:38.139 DEBUG:boto:Method: GET 2023-11-02T18:57:38.140 DEBUG:boto:Path: /ibtljn-62/obj55 2023-11-02T18:57:38.140 DEBUG:boto:Data: 2023-11-02T18:57:38.141 DEBUG:boto:Headers: {} 2023-11-02T18:57:38.141 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:38.142 DEBUG:boto:Port: 8000 2023-11-02T18:57:38.142 DEBUG:boto:Params: {} 2023-11-02T18:57:38.143 DEBUG:boto:Token: None 2023-11-02T18:57:38.143 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:38 GMT /ibtljn-62/obj55 2023-11-02T18:57:38.144 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VUPx6nqeQq5UdnmUHAzS8mhDcSo= 2023-11-02T18:57:38.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VUPx6nqeQq5UdnmUHAzS8mhDcSo=', 'Content-Length': '0'} 2023-11-02T18:57:38.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:55:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca67b11c119de632-006543f122-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:38.148 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:38.149 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:38.149 DEBUG:boto:path=/ibtljn-62/?prefix=obj55 2023-11-02T18:57:38.149 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj55 2023-11-02T18:57:38.150 DEBUG:boto:Method: GET 2023-11-02T18:57:38.150 DEBUG:boto:Path: /ibtljn-62/?prefix=obj55 2023-11-02T18:57:38.151 DEBUG:boto:Data: 2023-11-02T18:57:38.151 DEBUG:boto:Headers: {} 2023-11-02T18:57:38.152 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T18:57:38.152 DEBUG:boto:Port: 8001 2023-11-02T18:57:38.153 DEBUG:boto:Params: {} 2023-11-02T18:57:38.153 DEBUG:boto:Token: None 2023-11-02T18:57:38.154 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:38 GMT /ibtljn-62/ 2023-11-02T18:57:38.154 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zQ6RXKfs64Los04ICjdar1liUMU= 2023-11-02T18:57:38.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zQ6RXKfs64Los04ICjdar1liUMU=', 'Content-Length': '0'} 2023-11-02T18:57:38.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000822a6f30572c04ee-006543f122-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:38.161 DEBUG:boto:b'ibtljn-62obj551000falseobj552023-11-02T18:55:24.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:38.161 DEBUG:boto:path=/ibtljn-62/ 2023-11-02T18:57:38.162 DEBUG:boto:auth_path=/ibtljn-62/ 2023-11-02T18:57:38.162 DEBUG:boto:path=/ibtljn-62/?prefix=obj55 2023-11-02T18:57:38.163 DEBUG:boto:auth_path=/ibtljn-62/?prefix=obj55 2023-11-02T18:57:38.163 DEBUG:boto:Method: GET 2023-11-02T18:57:38.164 DEBUG:boto:Path: /ibtljn-62/?prefix=obj55 2023-11-02T18:57:38.164 DEBUG:boto:Data: 2023-11-02T18:57:38.165 DEBUG:boto:Headers: {} 2023-11-02T18:57:38.165 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:38.166 DEBUG:boto:Port: 8000 2023-11-02T18:57:38.166 DEBUG:boto:Params: {} 2023-11-02T18:57:38.167 DEBUG:boto:Token: None 2023-11-02T18:57:38.167 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 18:57:38 GMT /ibtljn-62/ 2023-11-02T18:57:38.168 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zQ6RXKfs64Los04ICjdar1liUMU= 2023-11-02T18:57:38.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zQ6RXKfs64Los04ICjdar1liUMU=', 'Content-Length': '0'} 2023-11-02T18:57:38.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002043d7c08b048389-006543f122-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:38.173 DEBUG:boto:b'ibtljn-62obj551000falseobj552023-11-02T18:55:24.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T18:57:38.174 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-62 zones={a2, a1} 2023-11-02T18:57:38.175 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-11-02T18:57:38.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:38.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:38.176 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:57:38.220 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:57:38.221 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:57:38.322 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:57:38.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.323 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000023282", 2023-11-02T18:57:38.323 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:50:37.390664Z" 2023-11-02T18:57:38.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.324 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000037246", 2023-11-02T18:57:38.325 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:55:24.738633Z" 2023-11-02T18:57:38.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.326 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000033290", 2023-11-02T18:57:38.326 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:55:45.402985Z" 2023-11-02T18:57:38.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.327 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032630", 2023-11-02T18:57:38.327 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:55:45.406590Z" 2023-11-02T18:57:38.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:38.328 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:57:38.329 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000023282', 1: '00000000000000000000:00000000000000037246', 2: '00000000000000000000:00000000000000033290', 3: '00000000000000000000:00000000000000032630'} 2023-11-02T18:57:38.329 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:57:38.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:38.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:38.331 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:57:38.380 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:57:38.380 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:57:38.483 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:57:38.484 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:57:38.484 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:57:38.484 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:57:38.485 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:57:38.485 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 5409087981735102677 2023-11-02T18:57:38.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.485 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:57:38.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.486 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:57:38.486 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:38.486 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:38.486 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000023282", 2023-11-02T18:57:38.487 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:38.487 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T18:57:38.487 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:38.487 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:50:37.390664Z" 2023-11-02T18:57:38.488 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:38.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.488 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:57:38.488 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:38.489 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:38.489 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000037246", 2023-11-02T18:57:38.489 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:38.489 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 53, 2023-11-02T18:57:38.490 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:38.490 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:55:24.738633Z" 2023-11-02T18:57:38.490 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:38.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.491 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:57:38.491 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:38.491 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:38.492 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000033290", 2023-11-02T18:57:38.492 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:38.492 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:38.492 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:38.492 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:55:45.402985Z" 2023-11-02T18:57:38.493 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:38.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.493 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:57:38.494 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:38.494 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:38.494 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032630", 2023-11-02T18:57:38.494 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:38.495 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:38.495 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:38.495 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:55:45.406590Z" 2023-11-02T18:57:38.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:38.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:38.496 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:57:38.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.496 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:57:38.496 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 62, 2023-11-02T18:57:38.496 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 62 2023-11-02T18:57:38.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.497 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:57:38Z" 2023-11-02T18:57:38.497 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:57:38.498 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5409087981735102677 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023282", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T18:50:37.390664Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037246", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-11-02T18:55:24.738633Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033290", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:55:45.402985Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032630", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:55:45.406590Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2023-11-02T18:57:38Z" } 2023-11-02T18:57:38.499 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023282', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T18:50:37.390664Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037246', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-11-02T18:55:24.738633Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033290', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:55:45.402985Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032630', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:55:45.406590Z'}}] 2023-11-02T18:57:38.499 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000023282', 1: '00000000000000000000:00000000000000037246', 2: '00000000000000000000:00000000000000033290', 3: '00000000000000000000:00000000000000032630'} 2023-11-02T18:57:38.500 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000023282', 1: '00000000000000000000:00000000000000037246', 2: '00000000000000000000:00000000000000033290', 3: '00000000000000000000:00000000000000032630'} 2023-11-02T18:57:38.500 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T18:57:38.500 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-11-02T18:57:38.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:38.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:38.502 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:57:38.584 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:57:38.584 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:57:38.689 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:57:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000005723", 2023-11-02T18:57:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:52:15.451244Z" 2023-11-02T18:57:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000015024", 2023-11-02T18:57:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:57:27.104161Z" 2023-11-02T18:57:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000014185", 2023-11-02T18:57:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:57:27.064473Z" 2023-11-02T18:57:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000011537", 2023-11-02T18:57:38.694 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:57:37.136835Z" 2023-11-02T18:57:38.694 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:38.694 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:57:38.695 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015024', 2: '00000000000000000000:00000000000000014185', 3: '00000000000000000000:00000000000000011537'} 2023-11-02T18:57:38.696 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:57:38.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:38.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:38.697 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:57:38.748 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:57:38.749 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:57:38.835 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:57:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:57:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:57:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:57:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:57:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:57:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:57:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:57:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000005723", 2023-11-02T18:57:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:52:15.451244Z" 2023-11-02T18:57:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:57:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000015024", 2023-11-02T18:57:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:27.104161Z" 2023-11-02T18:57:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:57:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000014185", 2023-11-02T18:57:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:27.064473Z" 2023-11-02T18:57:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:57:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000011395", 2023-11-02T18:57:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:26.643717Z" 2023-11-02T18:57:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:57:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:57:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:57:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:57:38.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:38.848 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:57:38Z" 2023-11-02T18:57:38.848 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:57:38.849 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:52:15.451244Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015024", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:27.104161Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014185", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:27.064473Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011395", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:26.643717Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:57:38Z" } 2023-11-02T18:57:38.850 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:52:15.451244Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015024', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:27.104161Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014185', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:27.064473Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011395', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:26.643717Z'}}] 2023-11-02T18:57:38.850 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015024', 2: '00000000000000000000:00000000000000014185', 3: '00000000000000000000:00000000000000011537'} 2023-11-02T18:57:38.851 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015024', 2: '00000000000000000000:00000000000000014185', 3: '00000000000000000000:00000000000000011395'} 2023-11-02T18:57:38.851 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=3 master=00000000000000000000:00000000000000011537 target=00000000000000000000:00000000000000011395 2023-11-02T18:57:43.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:43.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:43.853 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:57:43.902 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:57:43.902 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:57:43.989 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:57:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:57:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:57:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:57:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:57:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:57:43.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:43.991 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:57:43.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:43.991 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:57:43.992 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:43.992 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:43.993 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000005723", 2023-11-02T18:57:43.993 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:43.993 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:43.994 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:43.994 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:52:15.451244Z" 2023-11-02T18:57:43.995 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:43.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:43.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:43.996 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:57:43.996 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:43.996 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:43.997 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000015024", 2023-11-02T18:57:43.997 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:43.997 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:43.997 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:43.997 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:27.104161Z" 2023-11-02T18:57:43.998 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:43.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:43.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:43.999 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:57:43.999 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:43.999 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:44.000 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000014185", 2023-11-02T18:57:44.000 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:44.001 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:44.001 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:44.001 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:27.064473Z" 2023-11-02T18:57:44.002 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:44.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:44.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:44.002 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:57:44.003 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:44.003 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:44.003 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000011395", 2023-11-02T18:57:44.004 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:44.004 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:44.004 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:44.004 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:26.643717Z" 2023-11-02T18:57:44.005 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:44.005 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:44.005 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:57:44.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:44.006 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:57:44.006 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:57:44.006 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:57:44.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:44.006 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:57:43Z" 2023-11-02T18:57:44.007 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:57:44.008 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:52:15.451244Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015024", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:27.104161Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014185", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:27.064473Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011395", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:26.643717Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:57:43Z" } 2023-11-02T18:57:44.009 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:52:15.451244Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015024', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:27.104161Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014185', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:27.064473Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011395', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:26.643717Z'}}] 2023-11-02T18:57:44.010 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015024', 2: '00000000000000000000:00000000000000014185', 3: '00000000000000000000:00000000000000011537'} 2023-11-02T18:57:44.011 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015024', 2: '00000000000000000000:00000000000000014185', 3: '00000000000000000000:00000000000000011395'} 2023-11-02T18:57:44.011 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=3 master=00000000000000000000:00000000000000011537 target=00000000000000000000:00000000000000011395 2023-11-02T18:57:49.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:49.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:49.014 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:57:49.062 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:57:49.063 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:57:49.155 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:57:49.155 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:57:49.155 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:57:49.155 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:57:49.156 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:57:49.156 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:57:49.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:49.157 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:57:49.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:49.157 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:57:49.157 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:49.158 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:49.158 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000005723", 2023-11-02T18:57:49.158 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:49.158 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:49.159 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:49.159 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:52:15.451244Z" 2023-11-02T18:57:49.159 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:49.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:49.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:49.160 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:57:49.160 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:49.160 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:49.161 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000015024", 2023-11-02T18:57:49.161 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:49.161 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:49.161 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:49.162 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:27.104161Z" 2023-11-02T18:57:49.162 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:49.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:49.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:49.163 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:57:49.163 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:49.163 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:49.163 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000014185", 2023-11-02T18:57:49.163 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:49.164 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:49.164 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:49.164 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:27.064473Z" 2023-11-02T18:57:49.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:49.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:49.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:49.165 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:57:49.165 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:49.166 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:49.166 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000011395", 2023-11-02T18:57:49.166 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:49.166 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:49.167 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:49.167 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:26.643717Z" 2023-11-02T18:57:49.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:49.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:49.167 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:57:49.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:49.168 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:57:49.168 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:57:49.168 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:57:49.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:49.169 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:57:49Z" 2023-11-02T18:57:49.169 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:57:49.170 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:52:15.451244Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015024", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:27.104161Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014185", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:27.064473Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011395", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:26.643717Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:57:49Z" } 2023-11-02T18:57:49.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:52:15.451244Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015024', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:27.104161Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014185', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:27.064473Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011395', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:26.643717Z'}}] 2023-11-02T18:57:49.171 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015024', 2: '00000000000000000000:00000000000000014185', 3: '00000000000000000000:00000000000000011537'} 2023-11-02T18:57:49.172 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015024', 2: '00000000000000000000:00000000000000014185', 3: '00000000000000000000:00000000000000011395'} 2023-11-02T18:57:49.172 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=3 master=00000000000000000000:00000000000000011537 target=00000000000000000000:00000000000000011395 2023-11-02T18:57:54.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:54.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:54.174 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:57:54.222 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:57:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:57:54.314 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:57:54.314 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:57:54.314 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:57:54.315 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:57:54.315 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:57:54.315 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T18:57:54.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.315 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:57:54.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:54.316 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:57:54.316 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:54.316 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:54.317 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000005723", 2023-11-02T18:57:54.317 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:54.317 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:54.317 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:54.318 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:52:15.451244Z" 2023-11-02T18:57:54.318 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:54.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:54.319 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:57:54.319 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:54.319 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:54.319 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000015024", 2023-11-02T18:57:54.320 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:54.320 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:54.320 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:54.320 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:27.104161Z" 2023-11-02T18:57:54.320 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:54.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:54.321 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:57:54.321 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:54.322 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:54.322 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000014185", 2023-11-02T18:57:54.322 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:54.322 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:54.323 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:54.323 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:27.064473Z" 2023-11-02T18:57:54.323 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:54.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:54.324 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:57:54.324 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:54.324 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T18:57:54.324 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000011537", 2023-11-02T18:57:54.324 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:54.325 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:54.325 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:54.325 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:37.136835Z" 2023-11-02T18:57:54.325 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:54.326 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:54.326 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:57:54.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.326 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:57:54.327 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T18:57:54.327 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T18:57:54.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.327 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:57:54Z" 2023-11-02T18:57:54.327 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:57:54.328 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:52:15.451244Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015024", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:27.104161Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014185", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:27.064473Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011537", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:37.136835Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T18:57:54Z" } 2023-11-02T18:57:54.329 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:52:15.451244Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015024', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:27.104161Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014185', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:27.064473Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011537', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:37.136835Z'}}] 2023-11-02T18:57:54.329 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015024', 2: '00000000000000000000:00000000000000014185', 3: '00000000000000000000:00000000000000011537'} 2023-11-02T18:57:54.330 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015024', 2: '00000000000000000000:00000000000000014185', 3: '00000000000000000000:00000000000000011537'} 2023-11-02T18:57:54.330 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T18:57:54.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ok 2023-11-02T18:57:54.335 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:57:54.335 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:57:54.336 DEBUG:boto:Method: PUT 2023-11-02T18:57:54.336 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:57:54.337 DEBUG:boto:Data: 2023-11-02T18:57:54.337 DEBUG:boto:Headers: {} 2023-11-02T18:57:54.338 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:54.338 DEBUG:boto:Port: 8000 2023-11-02T18:57:54.338 DEBUG:boto:Params: {} 2023-11-02T18:57:54.339 DEBUG:boto:Token: None 2023-11-02T18:57:54.340 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 18:57:54 GMT /ibtljn-63/ 2023-11-02T18:57:54.340 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WmCbU+e/xPXfoJLNIbat4jj3Lxc= 2023-11-02T18:57:54.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WmCbU+e/xPXfoJLNIbat4jj3Lxc=', 'Content-Length': '0'} 2023-11-02T18:57:54.374 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000141d599255523221-006543f132-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:57:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:54.375 DEBUG:rgw_multi.tests:created bucket=ibtljn-63 2023-11-02T18:57:54.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:54.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:54.376 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:57:54.421 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:57:54.421 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:57:54.481 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T18:57:54.511 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T18:57:54.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:54.512 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:57:54.512 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T18:57:54.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:54.513 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:57:54.513 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T18:57:54.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:54.514 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951474.369610_580.1", 2023-11-02T18:57:54.514 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:57:54.369610Z" 2023-11-02T18:57:54.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:54.515 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:57:54.515 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T18:57:54.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:54.516 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T18:57:54.517 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951474.369610_580.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:57:54.517 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T18:57:54.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:54.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:54.519 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:57:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:57:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:57:54.666 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:54.661+0000 7f06b38c7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:57:54.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:54.661+0000 7f06b38c7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:57:54.694 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:57:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:57:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:57:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:57:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:57:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:57:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:57:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:57:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:57:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:57:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:57:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:57:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:57:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:57:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:57:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:57:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:57:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T18:57:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:57:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:57:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:57:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951474.358322_577.1", 2023-11-02T18:57:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:54.358322Z", 2023-11-02T18:57:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:57:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:57:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:57:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:57:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T18:57:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:57:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:57:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:57:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:57:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:57:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:57:54Z" 2023-11-02T18:57:54.708 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:57:54.709 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951474.358322_577.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:54.358322Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:57:54Z" } 2023-11-02T18:57:54.709 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951474.358322_577.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:54.358322Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T18:57:54.710 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951474.369610_580.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:57:54.710 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951474.358322_577.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:57:54.711 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951474.369610_580.1 target=1_1698951474.358322_577.1 2023-11-02T18:57:59.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:59.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T18:57:59.713 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T18:57:59.759 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:57:59.759 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:57:59.818 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:59.813+0000 7f851d3fff40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:57:59.820 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:57:59.813+0000 7f851d3fff40 20 RGW-SYNC:meta: read sync status 2023-11-02T18:57:59.849 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T18:57:59.849 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T18:57:59.849 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T18:57:59.849 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T18:57:59.850 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T18:57:59.850 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T18:57:59.850 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:57:59.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:59.851 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T18:57:59.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:59.851 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T18:57:59.851 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:59.851 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:57:59.852 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T18:57:59.852 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:59.852 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T18:57:59.852 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:59.852 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T18:57:59.853 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:57:59.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:59.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:59.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:59.853 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T18:57:59.854 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:59.854 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:57:59.854 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T18:57:59.854 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:59.855 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:59.855 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:59.855 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T18:57:59.855 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:57:59.855 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:59.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:59.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:59.856 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T18:57:59.858 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:59.858 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:57:59.858 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951474.369610_580.1", 2023-11-02T18:57:59.859 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:59.859 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:59.859 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:59.859 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:57:54.369610Z", 2023-11-02T18:57:59.859 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:57:59.860 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:59.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:59.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T18:57:59.860 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T18:57:59.861 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T18:57:59.861 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T18:57:59.861 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T18:57:59.861 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T18:57:59.861 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T18:57:59.862 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T18:57:59.862 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T18:57:59.862 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T18:57:59.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:59.863 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T18:57:59.863 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T18:57:59.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:59.863 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T18:57:59.863 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T18:57:59.864 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T18:57:59.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T18:57:59.865 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T18:57:59Z" 2023-11-02T18:57:59.865 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T18:57:59.866 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951474.369610_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:57:54.369610Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T18:57:59Z" } 2023-11-02T18:57:59.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951474.369610_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:54.369610Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T18:57:59.867 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951474.369610_580.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:57:59.868 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951474.369610_580.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T18:57:59.868 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T18:57:59.869 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:57:59.869 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:57:59.869 DEBUG:boto:Method: HEAD 2023-11-02T18:57:59.870 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:57:59.870 DEBUG:boto:Data: 2023-11-02T18:57:59.871 DEBUG:boto:Headers: {} 2023-11-02T18:57:59.871 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:59.872 DEBUG:boto:Port: 8000 2023-11-02T18:57:59.872 DEBUG:boto:Params: {} 2023-11-02T18:57:59.873 DEBUG:boto:Token: None 2023-11-02T18:57:59.873 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:59 GMT /ibtljn-63/ 2023-11-02T18:57:59.873 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE= 2023-11-02T18:57:59.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE=', 'Content-Length': '0'} 2023-11-02T18:57:59.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf85ebf584783ad1-006543f137-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:59.880 DEBUG:boto:path=/ibtljn-63/obj11 2023-11-02T18:57:59.880 DEBUG:boto:auth_path=/ibtljn-63/obj11 2023-11-02T18:57:59.881 DEBUG:boto:Method: PUT 2023-11-02T18:57:59.881 DEBUG:boto:Path: /ibtljn-63/obj11 2023-11-02T18:57:59.882 DEBUG:boto:Data: 2023-11-02T18:57:59.882 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:57:59.883 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:59.883 DEBUG:boto:Port: 8000 2023-11-02T18:57:59.883 DEBUG:boto:Params: {} 2023-11-02T18:57:59.884 DEBUG:boto:Token: None 2023-11-02T18:57:59.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:57:59 GMT /ibtljn-63/obj11 2023-11-02T18:57:59.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VY5LUhHg3nx363zhtOjm56h843w= 2023-11-02T18:57:59.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VY5LUhHg3nx363zhtOjm56h843w='} 2023-11-02T18:57:59.901 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c8d91b6527e39d8-006543f137-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:59.902 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:57:59.903 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:57:59.904 DEBUG:boto:Method: HEAD 2023-11-02T18:57:59.904 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:57:59.905 DEBUG:boto:Data: 2023-11-02T18:57:59.905 DEBUG:boto:Headers: {} 2023-11-02T18:57:59.906 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:59.906 DEBUG:boto:Port: 8000 2023-11-02T18:57:59.907 DEBUG:boto:Params: {} 2023-11-02T18:57:59.907 DEBUG:boto:Token: None 2023-11-02T18:57:59.908 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:59 GMT /ibtljn-63/ 2023-11-02T18:57:59.908 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE= 2023-11-02T18:57:59.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE=', 'Content-Length': '0'} 2023-11-02T18:57:59.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b6cb921502a686f-006543f137-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:59.913 DEBUG:boto:path=/ibtljn-63/obj22 2023-11-02T18:57:59.913 DEBUG:boto:auth_path=/ibtljn-63/obj22 2023-11-02T18:57:59.914 DEBUG:boto:Method: PUT 2023-11-02T18:57:59.914 DEBUG:boto:Path: /ibtljn-63/obj22 2023-11-02T18:57:59.915 DEBUG:boto:Data: 2023-11-02T18:57:59.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:57:59.916 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:59.916 DEBUG:boto:Port: 8000 2023-11-02T18:57:59.917 DEBUG:boto:Params: {} 2023-11-02T18:57:59.918 DEBUG:boto:Token: None 2023-11-02T18:57:59.918 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:57:59 GMT /ibtljn-63/obj22 2023-11-02T18:57:59.919 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y7x3pQcT13BUTT2aL5hmt1YyiEQ= 2023-11-02T18:57:59.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y7x3pQcT13BUTT2aL5hmt1YyiEQ='} 2023-11-02T18:57:59.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007432a0e8fefb7dc8-006543f137-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:59.963 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:57:59.964 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:57:59.964 DEBUG:boto:Method: HEAD 2023-11-02T18:57:59.964 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:57:59.965 DEBUG:boto:Data: 2023-11-02T18:57:59.965 DEBUG:boto:Headers: {} 2023-11-02T18:57:59.966 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:59.966 DEBUG:boto:Port: 8000 2023-11-02T18:57:59.967 DEBUG:boto:Params: {} 2023-11-02T18:57:59.967 DEBUG:boto:Token: None 2023-11-02T18:57:59.968 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:59 GMT /ibtljn-63/ 2023-11-02T18:57:59.968 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE= 2023-11-02T18:57:59.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE=', 'Content-Length': '0'} 2023-11-02T18:57:59.973 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000474aa5c9ce6b5877-006543f137-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:59.973 DEBUG:boto:path=/ibtljn-63/obj33 2023-11-02T18:57:59.974 DEBUG:boto:auth_path=/ibtljn-63/obj33 2023-11-02T18:57:59.974 DEBUG:boto:Method: PUT 2023-11-02T18:57:59.975 DEBUG:boto:Path: /ibtljn-63/obj33 2023-11-02T18:57:59.975 DEBUG:boto:Data: 2023-11-02T18:57:59.976 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:57:59.976 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:59.976 DEBUG:boto:Port: 8000 2023-11-02T18:57:59.977 DEBUG:boto:Params: {} 2023-11-02T18:57:59.977 DEBUG:boto:Token: None 2023-11-02T18:57:59.978 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:57:59 GMT /ibtljn-63/obj33 2023-11-02T18:57:59.978 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:czlt7VanvPNrm34+ZKqUbnP/tj4= 2023-11-02T18:57:59.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:czlt7VanvPNrm34+ZKqUbnP/tj4='} 2023-11-02T18:57:59.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f2d6bece15d33ca-006543f137-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:57:59.991 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:57:59.992 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:57:59.992 DEBUG:boto:Method: HEAD 2023-11-02T18:57:59.992 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:57:59.993 DEBUG:boto:Data: 2023-11-02T18:57:59.993 DEBUG:boto:Headers: {} 2023-11-02T18:57:59.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:57:59.994 DEBUG:boto:Port: 8000 2023-11-02T18:57:59.995 DEBUG:boto:Params: {} 2023-11-02T18:57:59.996 DEBUG:boto:Token: None 2023-11-02T18:57:59.996 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:57:59 GMT /ibtljn-63/ 2023-11-02T18:57:59.996 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE= 2023-11-02T18:57:59.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE=', 'Content-Length': '0'} 2023-11-02T18:58:00.001 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094ffb2f26a7e35f9-006543f137-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.002 DEBUG:boto:path=/ibtljn-63/obj44 2023-11-02T18:58:00.003 DEBUG:boto:auth_path=/ibtljn-63/obj44 2023-11-02T18:58:00.003 DEBUG:boto:Method: PUT 2023-11-02T18:58:00.004 DEBUG:boto:Path: /ibtljn-63/obj44 2023-11-02T18:58:00.004 DEBUG:boto:Data: 2023-11-02T18:58:00.004 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:00.005 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.005 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.006 DEBUG:boto:Params: {} 2023-11-02T18:58:00.006 DEBUG:boto:Token: None 2023-11-02T18:58:00.007 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/obj44 2023-11-02T18:58:00.007 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vinwwq86xjYYeLuu8r2FPiJ0d5Y= 2023-11-02T18:58:00.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vinwwq86xjYYeLuu8r2FPiJ0d5Y='} 2023-11-02T18:58:00.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2b9191d215e8204-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.051 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:00.052 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:00.052 DEBUG:boto:Method: HEAD 2023-11-02T18:58:00.053 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:00.053 DEBUG:boto:Data: 2023-11-02T18:58:00.053 DEBUG:boto:Headers: {} 2023-11-02T18:58:00.054 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.054 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.055 DEBUG:boto:Params: {} 2023-11-02T18:58:00.055 DEBUG:boto:Token: None 2023-11-02T18:58:00.056 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/ 2023-11-02T18:58:00.056 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T18:58:00.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T18:58:00.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000660b26b6b40a452a-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.062 DEBUG:boto:path=/ibtljn-63/obj55 2023-11-02T18:58:00.062 DEBUG:boto:auth_path=/ibtljn-63/obj55 2023-11-02T18:58:00.063 DEBUG:boto:Method: PUT 2023-11-02T18:58:00.063 DEBUG:boto:Path: /ibtljn-63/obj55 2023-11-02T18:58:00.064 DEBUG:boto:Data: 2023-11-02T18:58:00.064 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:00.065 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.065 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.066 DEBUG:boto:Params: {} 2023-11-02T18:58:00.066 DEBUG:boto:Token: None 2023-11-02T18:58:00.067 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/obj55 2023-11-02T18:58:00.067 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SPnClwpAaby1yEAsouobAvdM1rg= 2023-11-02T18:58:00.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SPnClwpAaby1yEAsouobAvdM1rg='} 2023-11-02T18:58:00.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e73933bd1c093d1-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:58:00.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:58:00.078 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:58:00.127 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:58:00.128 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:58:00.211 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:58:00.211 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-63 2023-11-02T18:58:00.220 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 5 2023-11-02T18:58:00.306 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:00.306 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:00.307 DEBUG:boto:Method: HEAD 2023-11-02T18:58:00.307 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:00.308 DEBUG:boto:Data: 2023-11-02T18:58:00.308 DEBUG:boto:Headers: {} 2023-11-02T18:58:00.308 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.309 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.309 DEBUG:boto:Params: {} 2023-11-02T18:58:00.310 DEBUG:boto:Token: None 2023-11-02T18:58:00.310 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/ 2023-11-02T18:58:00.311 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T18:58:00.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T18:58:00.317 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dec164db5a86eeed-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.317 DEBUG:boto:path=/ibtljn-63/obj17 2023-11-02T18:58:00.318 DEBUG:boto:auth_path=/ibtljn-63/obj17 2023-11-02T18:58:00.318 DEBUG:boto:Method: PUT 2023-11-02T18:58:00.319 DEBUG:boto:Path: /ibtljn-63/obj17 2023-11-02T18:58:00.319 DEBUG:boto:Data: 2023-11-02T18:58:00.320 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:00.320 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.321 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.321 DEBUG:boto:Params: {} 2023-11-02T18:58:00.322 DEBUG:boto:Token: None 2023-11-02T18:58:00.322 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/obj17 2023-11-02T18:58:00.323 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:HF5Ac4/n28U5nHDB+ir0G266mRk= 2023-11-02T18:58:00.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HF5Ac4/n28U5nHDB+ir0G266mRk='} 2023-11-02T18:58:00.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014ce0e30f4d38427-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.367 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:00.368 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:00.368 DEBUG:boto:Method: HEAD 2023-11-02T18:58:00.369 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:00.369 DEBUG:boto:Data: 2023-11-02T18:58:00.369 DEBUG:boto:Headers: {} 2023-11-02T18:58:00.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.370 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.371 DEBUG:boto:Params: {} 2023-11-02T18:58:00.371 DEBUG:boto:Token: None 2023-11-02T18:58:00.372 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/ 2023-11-02T18:58:00.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T18:58:00.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T18:58:00.378 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000aea9e71b6e62e27-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.378 DEBUG:boto:path=/ibtljn-63/obj34 2023-11-02T18:58:00.379 DEBUG:boto:auth_path=/ibtljn-63/obj34 2023-11-02T18:58:00.379 DEBUG:boto:Method: PUT 2023-11-02T18:58:00.380 DEBUG:boto:Path: /ibtljn-63/obj34 2023-11-02T18:58:00.380 DEBUG:boto:Data: 2023-11-02T18:58:00.381 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:00.381 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.381 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.382 DEBUG:boto:Params: {} 2023-11-02T18:58:00.382 DEBUG:boto:Token: None 2023-11-02T18:58:00.383 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/obj34 2023-11-02T18:58:00.383 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vtmtirGUHwcyx2uqexTTJYT4vBQ= 2023-11-02T18:58:00.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vtmtirGUHwcyx2uqexTTJYT4vBQ='} 2023-11-02T18:58:00.395 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096a4c9acded83912-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.396 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:00.396 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:00.397 DEBUG:boto:Method: HEAD 2023-11-02T18:58:00.397 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:00.398 DEBUG:boto:Data: 2023-11-02T18:58:00.398 DEBUG:boto:Headers: {} 2023-11-02T18:58:00.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.399 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.399 DEBUG:boto:Params: {} 2023-11-02T18:58:00.400 DEBUG:boto:Token: None 2023-11-02T18:58:00.400 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/ 2023-11-02T18:58:00.401 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T18:58:00.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T18:58:00.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa2980caece9d769-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.407 DEBUG:boto:path=/ibtljn-63/obj51 2023-11-02T18:58:00.407 DEBUG:boto:auth_path=/ibtljn-63/obj51 2023-11-02T18:58:00.408 DEBUG:boto:Method: PUT 2023-11-02T18:58:00.408 DEBUG:boto:Path: /ibtljn-63/obj51 2023-11-02T18:58:00.408 DEBUG:boto:Data: 2023-11-02T18:58:00.409 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:00.409 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.410 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.410 DEBUG:boto:Params: {} 2023-11-02T18:58:00.411 DEBUG:boto:Token: None 2023-11-02T18:58:00.411 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/obj51 2023-11-02T18:58:00.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DBAzwEYRpiASOabBuOoRrHnGCfs= 2023-11-02T18:58:00.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DBAzwEYRpiASOabBuOoRrHnGCfs='} 2023-11-02T18:58:00.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000918070cec8dfbbc-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.459 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:00.460 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:00.460 DEBUG:boto:Method: HEAD 2023-11-02T18:58:00.461 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:00.461 DEBUG:boto:Data: 2023-11-02T18:58:00.461 DEBUG:boto:Headers: {} 2023-11-02T18:58:00.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.462 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.463 DEBUG:boto:Params: {} 2023-11-02T18:58:00.463 DEBUG:boto:Token: None 2023-11-02T18:58:00.464 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/ 2023-11-02T18:58:00.464 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T18:58:00.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T18:58:00.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007029686c1c692e82-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.471 DEBUG:boto:path=/ibtljn-63/obj68 2023-11-02T18:58:00.471 DEBUG:boto:auth_path=/ibtljn-63/obj68 2023-11-02T18:58:00.472 DEBUG:boto:Method: PUT 2023-11-02T18:58:00.472 DEBUG:boto:Path: /ibtljn-63/obj68 2023-11-02T18:58:00.473 DEBUG:boto:Data: 2023-11-02T18:58:00.473 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:00.473 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.474 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.474 DEBUG:boto:Params: {} 2023-11-02T18:58:00.475 DEBUG:boto:Token: None 2023-11-02T18:58:00.475 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/obj68 2023-11-02T18:58:00.477 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UuNFSdLnvgEY1g0rCJHDu4C6/b8= 2023-11-02T18:58:00.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UuNFSdLnvgEY1g0rCJHDu4C6/b8='} 2023-11-02T18:58:00.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004907d37e1d7deded-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.492 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:00.492 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:00.492 DEBUG:boto:Method: HEAD 2023-11-02T18:58:00.493 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:00.493 DEBUG:boto:Data: 2023-11-02T18:58:00.494 DEBUG:boto:Headers: {} 2023-11-02T18:58:00.494 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.495 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.495 DEBUG:boto:Params: {} 2023-11-02T18:58:00.496 DEBUG:boto:Token: None 2023-11-02T18:58:00.496 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/ 2023-11-02T18:58:00.497 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T18:58:00.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T18:58:00.502 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022a4f4946cd7159c-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.503 DEBUG:boto:path=/ibtljn-63/obj85 2023-11-02T18:58:00.503 DEBUG:boto:auth_path=/ibtljn-63/obj85 2023-11-02T18:58:00.504 DEBUG:boto:Method: PUT 2023-11-02T18:58:00.504 DEBUG:boto:Path: /ibtljn-63/obj85 2023-11-02T18:58:00.505 DEBUG:boto:Data: 2023-11-02T18:58:00.505 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:00.505 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.506 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.506 DEBUG:boto:Params: {} 2023-11-02T18:58:00.507 DEBUG:boto:Token: None 2023-11-02T18:58:00.507 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/obj85 2023-11-02T18:58:00.508 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j6tpHHsMvN2podymvT5y/Zbzi7A= 2023-11-02T18:58:00.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j6tpHHsMvN2podymvT5y/Zbzi7A='} 2023-11-02T18:58:00.550 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000339ea9f31cd5317c-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:58:00.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:58:00.552 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:58:00.600 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:58:00.600 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:58:00.684 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:58:00.684 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-63 2023-11-02T18:58:00.698 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 10 2023-11-02T18:58:00.799 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:00.799 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:00.800 DEBUG:boto:Method: HEAD 2023-11-02T18:58:00.800 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:00.801 DEBUG:boto:Data: 2023-11-02T18:58:00.801 DEBUG:boto:Headers: {} 2023-11-02T18:58:00.801 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.802 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.802 DEBUG:boto:Params: {} 2023-11-02T18:58:00.803 DEBUG:boto:Token: None 2023-11-02T18:58:00.803 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/ 2023-11-02T18:58:00.804 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T18:58:00.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T18:58:00.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000994fd4da2b01bf76-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.810 DEBUG:boto:path=/ibtljn-63/obj19 2023-11-02T18:58:00.811 DEBUG:boto:auth_path=/ibtljn-63/obj19 2023-11-02T18:58:00.811 DEBUG:boto:Method: PUT 2023-11-02T18:58:00.812 DEBUG:boto:Path: /ibtljn-63/obj19 2023-11-02T18:58:00.812 DEBUG:boto:Data: 2023-11-02T18:58:00.812 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:00.813 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.813 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.814 DEBUG:boto:Params: {} 2023-11-02T18:58:00.814 DEBUG:boto:Token: None 2023-11-02T18:58:00.815 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/obj19 2023-11-02T18:58:00.815 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2bMbGTpeObSfnAPILFVWb8FqONg= 2023-11-02T18:58:00.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2bMbGTpeObSfnAPILFVWb8FqONg='} 2023-11-02T18:58:00.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5b89cf6dfb239ed-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.829 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:00.829 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:00.829 DEBUG:boto:Method: HEAD 2023-11-02T18:58:00.830 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:00.830 DEBUG:boto:Data: 2023-11-02T18:58:00.831 DEBUG:boto:Headers: {} 2023-11-02T18:58:00.831 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.832 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.832 DEBUG:boto:Params: {} 2023-11-02T18:58:00.832 DEBUG:boto:Token: None 2023-11-02T18:58:00.833 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/ 2023-11-02T18:58:00.833 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T18:58:00.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T18:58:00.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e78ec5ccfe89cd9-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.839 DEBUG:boto:path=/ibtljn-63/obj38 2023-11-02T18:58:00.839 DEBUG:boto:auth_path=/ibtljn-63/obj38 2023-11-02T18:58:00.840 DEBUG:boto:Method: PUT 2023-11-02T18:58:00.840 DEBUG:boto:Path: /ibtljn-63/obj38 2023-11-02T18:58:00.841 DEBUG:boto:Data: 2023-11-02T18:58:00.841 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:00.841 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.842 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.842 DEBUG:boto:Params: {} 2023-11-02T18:58:00.843 DEBUG:boto:Token: None 2023-11-02T18:58:00.843 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/obj38 2023-11-02T18:58:00.844 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hsm2mlFB643Pcs4a8LKGRccW4Oo= 2023-11-02T18:58:00.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hsm2mlFB643Pcs4a8LKGRccW4Oo='} 2023-11-02T18:58:00.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e86c23d3059b1488-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.887 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:00.889 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:00.889 DEBUG:boto:Method: HEAD 2023-11-02T18:58:00.890 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:00.890 DEBUG:boto:Data: 2023-11-02T18:58:00.891 DEBUG:boto:Headers: {} 2023-11-02T18:58:00.891 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.891 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.892 DEBUG:boto:Params: {} 2023-11-02T18:58:00.892 DEBUG:boto:Token: None 2023-11-02T18:58:00.893 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/ 2023-11-02T18:58:00.893 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T18:58:00.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T18:58:00.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003065be418dc5385f-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.901 DEBUG:boto:path=/ibtljn-63/obj57 2023-11-02T18:58:00.901 DEBUG:boto:auth_path=/ibtljn-63/obj57 2023-11-02T18:58:00.902 DEBUG:boto:Method: PUT 2023-11-02T18:58:00.902 DEBUG:boto:Path: /ibtljn-63/obj57 2023-11-02T18:58:00.903 DEBUG:boto:Data: 2023-11-02T18:58:00.903 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:00.903 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.904 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.904 DEBUG:boto:Params: {} 2023-11-02T18:58:00.905 DEBUG:boto:Token: None 2023-11-02T18:58:00.905 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/obj57 2023-11-02T18:58:00.906 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BpPIhsUn6yPe3RcNqnMsWEBYK5I= 2023-11-02T18:58:00.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BpPIhsUn6yPe3RcNqnMsWEBYK5I='} 2023-11-02T18:58:00.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d545f036f0270026-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.916 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:00.916 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:00.916 DEBUG:boto:Method: HEAD 2023-11-02T18:58:00.917 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:00.917 DEBUG:boto:Data: 2023-11-02T18:58:00.918 DEBUG:boto:Headers: {} 2023-11-02T18:58:00.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.919 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.919 DEBUG:boto:Params: {} 2023-11-02T18:58:00.920 DEBUG:boto:Token: None 2023-11-02T18:58:00.920 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/ 2023-11-02T18:58:00.921 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T18:58:00.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T18:58:00.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3590b4ce8897701-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.927 DEBUG:boto:path=/ibtljn-63/obj76 2023-11-02T18:58:00.928 DEBUG:boto:auth_path=/ibtljn-63/obj76 2023-11-02T18:58:00.928 DEBUG:boto:Method: PUT 2023-11-02T18:58:00.928 DEBUG:boto:Path: /ibtljn-63/obj76 2023-11-02T18:58:00.929 DEBUG:boto:Data: 2023-11-02T18:58:00.929 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:00.930 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.930 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.931 DEBUG:boto:Params: {} 2023-11-02T18:58:00.931 DEBUG:boto:Token: None 2023-11-02T18:58:00.932 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/obj76 2023-11-02T18:58:00.932 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5fdzGslu5QGrFAgnd3wwqoCEaZM= 2023-11-02T18:58:00.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5fdzGslu5QGrFAgnd3wwqoCEaZM='} 2023-11-02T18:58:00.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cd76f879459348c-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.979 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:00.980 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:00.980 DEBUG:boto:Method: HEAD 2023-11-02T18:58:00.981 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:00.981 DEBUG:boto:Data: 2023-11-02T18:58:00.982 DEBUG:boto:Headers: {} 2023-11-02T18:58:00.982 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.983 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.983 DEBUG:boto:Params: {} 2023-11-02T18:58:00.983 DEBUG:boto:Token: None 2023-11-02T18:58:00.984 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/ 2023-11-02T18:58:00.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T18:58:00.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T18:58:00.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a5e6a348fa43f6e-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:00.992 DEBUG:boto:path=/ibtljn-63/obj95 2023-11-02T18:58:00.992 DEBUG:boto:auth_path=/ibtljn-63/obj95 2023-11-02T18:58:00.993 DEBUG:boto:Method: PUT 2023-11-02T18:58:00.993 DEBUG:boto:Path: /ibtljn-63/obj95 2023-11-02T18:58:00.994 DEBUG:boto:Data: 2023-11-02T18:58:00.994 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:00.994 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:00.995 DEBUG:boto:Port: 8000 2023-11-02T18:58:00.995 DEBUG:boto:Params: {} 2023-11-02T18:58:00.995 DEBUG:boto:Token: None 2023-11-02T18:58:00.996 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:00 GMT /ibtljn-63/obj95 2023-11-02T18:58:00.996 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:INSngDeiqoVHTu2xtecIMp9sHj0= 2023-11-02T18:58:00.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:INSngDeiqoVHTu2xtecIMp9sHj0='} 2023-11-02T18:58:01.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080037615466f2feb-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:58:01.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:58:01.007 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:58:01.056 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:58:01.056 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:58:01.144 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:58:01.145 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-63 2023-11-02T18:58:01.159 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 15 2023-11-02T18:58:01.257 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:01.257 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:01.258 DEBUG:boto:Method: HEAD 2023-11-02T18:58:01.258 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:01.259 DEBUG:boto:Data: 2023-11-02T18:58:01.259 DEBUG:boto:Headers: {} 2023-11-02T18:58:01.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.260 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.260 DEBUG:boto:Params: {} 2023-11-02T18:58:01.261 DEBUG:boto:Token: None 2023-11-02T18:58:01.261 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/ 2023-11-02T18:58:01.262 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T18:58:01.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T18:58:01.267 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d94a42fbcfc7647-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.268 DEBUG:boto:path=/ibtljn-63/obj23 2023-11-02T18:58:01.269 DEBUG:boto:auth_path=/ibtljn-63/obj23 2023-11-02T18:58:01.269 DEBUG:boto:Method: PUT 2023-11-02T18:58:01.270 DEBUG:boto:Path: /ibtljn-63/obj23 2023-11-02T18:58:01.270 DEBUG:boto:Data: 2023-11-02T18:58:01.271 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:01.271 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.271 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.272 DEBUG:boto:Params: {} 2023-11-02T18:58:01.272 DEBUG:boto:Token: None 2023-11-02T18:58:01.273 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/obj23 2023-11-02T18:58:01.274 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b36FpwxY61j0nTbV9L2ZpPEhPuQ= 2023-11-02T18:58:01.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b36FpwxY61j0nTbV9L2ZpPEhPuQ='} 2023-11-02T18:58:01.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b85bacc533ebcbe2-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.319 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:01.320 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:01.320 DEBUG:boto:Method: HEAD 2023-11-02T18:58:01.321 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:01.321 DEBUG:boto:Data: 2023-11-02T18:58:01.321 DEBUG:boto:Headers: {} 2023-11-02T18:58:01.322 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.322 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.323 DEBUG:boto:Params: {} 2023-11-02T18:58:01.323 DEBUG:boto:Token: None 2023-11-02T18:58:01.324 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/ 2023-11-02T18:58:01.324 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T18:58:01.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T18:58:01.331 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a860776008fb5ba6-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.332 DEBUG:boto:path=/ibtljn-63/obj46 2023-11-02T18:58:01.332 DEBUG:boto:auth_path=/ibtljn-63/obj46 2023-11-02T18:58:01.332 DEBUG:boto:Method: PUT 2023-11-02T18:58:01.333 DEBUG:boto:Path: /ibtljn-63/obj46 2023-11-02T18:58:01.333 DEBUG:boto:Data: 2023-11-02T18:58:01.334 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:01.334 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.335 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.335 DEBUG:boto:Params: {} 2023-11-02T18:58:01.336 DEBUG:boto:Token: None 2023-11-02T18:58:01.336 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/obj46 2023-11-02T18:58:01.337 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/xLyVjOQgsLBiZbxePaI0iPkXos= 2023-11-02T18:58:01.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/xLyVjOQgsLBiZbxePaI0iPkXos='} 2023-11-02T18:58:01.349 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd1c37641a833fb8-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.350 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:01.350 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:01.351 DEBUG:boto:Method: HEAD 2023-11-02T18:58:01.351 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:01.352 DEBUG:boto:Data: 2023-11-02T18:58:01.352 DEBUG:boto:Headers: {} 2023-11-02T18:58:01.353 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.353 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.353 DEBUG:boto:Params: {} 2023-11-02T18:58:01.354 DEBUG:boto:Token: None 2023-11-02T18:58:01.354 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/ 2023-11-02T18:58:01.355 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T18:58:01.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T18:58:01.360 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071844c61035573b8-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.361 DEBUG:boto:path=/ibtljn-63/obj69 2023-11-02T18:58:01.362 DEBUG:boto:auth_path=/ibtljn-63/obj69 2023-11-02T18:58:01.362 DEBUG:boto:Method: PUT 2023-11-02T18:58:01.362 DEBUG:boto:Path: /ibtljn-63/obj69 2023-11-02T18:58:01.363 DEBUG:boto:Data: 2023-11-02T18:58:01.363 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:01.364 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.364 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.365 DEBUG:boto:Params: {} 2023-11-02T18:58:01.365 DEBUG:boto:Token: None 2023-11-02T18:58:01.365 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/obj69 2023-11-02T18:58:01.366 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d8ZmfTg6EOhSzh0h22SQW0Z6PAU= 2023-11-02T18:58:01.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d8ZmfTg6EOhSzh0h22SQW0Z6PAU='} 2023-11-02T18:58:01.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a90dd9d7747cf963-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.411 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:01.412 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:01.412 DEBUG:boto:Method: HEAD 2023-11-02T18:58:01.412 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:01.413 DEBUG:boto:Data: 2023-11-02T18:58:01.413 DEBUG:boto:Headers: {} 2023-11-02T18:58:01.414 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.414 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.414 DEBUG:boto:Params: {} 2023-11-02T18:58:01.415 DEBUG:boto:Token: None 2023-11-02T18:58:01.415 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/ 2023-11-02T18:58:01.416 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T18:58:01.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T18:58:01.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005ae94bb7a93382d-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.423 DEBUG:boto:path=/ibtljn-63/obj92 2023-11-02T18:58:01.423 DEBUG:boto:auth_path=/ibtljn-63/obj92 2023-11-02T18:58:01.424 DEBUG:boto:Method: PUT 2023-11-02T18:58:01.424 DEBUG:boto:Path: /ibtljn-63/obj92 2023-11-02T18:58:01.425 DEBUG:boto:Data: 2023-11-02T18:58:01.425 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:01.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.426 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.426 DEBUG:boto:Params: {} 2023-11-02T18:58:01.427 DEBUG:boto:Token: None 2023-11-02T18:58:01.427 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/obj92 2023-11-02T18:58:01.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3f/j928m1BTYdMVhEAzvVxiocbo= 2023-11-02T18:58:01.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3f/j928m1BTYdMVhEAzvVxiocbo='} 2023-11-02T18:58:01.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b88a4dca9dcbe683-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.440 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:01.440 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:01.440 DEBUG:boto:Method: HEAD 2023-11-02T18:58:01.441 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:01.441 DEBUG:boto:Data: 2023-11-02T18:58:01.442 DEBUG:boto:Headers: {} 2023-11-02T18:58:01.442 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.442 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.443 DEBUG:boto:Params: {} 2023-11-02T18:58:01.443 DEBUG:boto:Token: None 2023-11-02T18:58:01.444 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/ 2023-11-02T18:58:01.444 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T18:58:01.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T18:58:01.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004242f9626cfa86e3-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.451 DEBUG:boto:path=/ibtljn-63/obj115 2023-11-02T18:58:01.451 DEBUG:boto:auth_path=/ibtljn-63/obj115 2023-11-02T18:58:01.451 DEBUG:boto:Method: PUT 2023-11-02T18:58:01.452 DEBUG:boto:Path: /ibtljn-63/obj115 2023-11-02T18:58:01.452 DEBUG:boto:Data: 2023-11-02T18:58:01.453 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:01.453 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.454 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.454 DEBUG:boto:Params: {} 2023-11-02T18:58:01.455 DEBUG:boto:Token: None 2023-11-02T18:58:01.455 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/obj115 2023-11-02T18:58:01.456 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:J3FabMctedHifIr8TYgNSKjSmxs= 2023-11-02T18:58:01.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J3FabMctedHifIr8TYgNSKjSmxs='} 2023-11-02T18:58:01.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb494ffd4722f723-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:58:01.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:58:01.500 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:58:01.550 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:58:01.551 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:58:01.636 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:58:01.637 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-63 2023-11-02T18:58:01.655 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 20 2023-11-02T18:58:01.757 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:01.758 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:01.758 DEBUG:boto:Method: HEAD 2023-11-02T18:58:01.759 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:01.759 DEBUG:boto:Data: 2023-11-02T18:58:01.759 DEBUG:boto:Headers: {} 2023-11-02T18:58:01.760 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.760 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.761 DEBUG:boto:Params: {} 2023-11-02T18:58:01.761 DEBUG:boto:Token: None 2023-11-02T18:58:01.762 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/ 2023-11-02T18:58:01.762 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T18:58:01.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T18:58:01.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001865950e88a3dc5d-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.770 DEBUG:boto:path=/ibtljn-63/obj29 2023-11-02T18:58:01.770 DEBUG:boto:auth_path=/ibtljn-63/obj29 2023-11-02T18:58:01.771 DEBUG:boto:Method: PUT 2023-11-02T18:58:01.771 DEBUG:boto:Path: /ibtljn-63/obj29 2023-11-02T18:58:01.771 DEBUG:boto:Data: 2023-11-02T18:58:01.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:01.772 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.773 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.773 DEBUG:boto:Params: {} 2023-11-02T18:58:01.774 DEBUG:boto:Token: None 2023-11-02T18:58:01.775 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/obj29 2023-11-02T18:58:01.775 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dFGg5gjTfvj9n5rVIZx0H64JVkM= 2023-11-02T18:58:01.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dFGg5gjTfvj9n5rVIZx0H64JVkM='} 2023-11-02T18:58:01.788 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f90dafb1ff01c5b-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.789 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:01.789 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:01.789 DEBUG:boto:Method: HEAD 2023-11-02T18:58:01.790 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:01.790 DEBUG:boto:Data: 2023-11-02T18:58:01.791 DEBUG:boto:Headers: {} 2023-11-02T18:58:01.791 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.791 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.792 DEBUG:boto:Params: {} 2023-11-02T18:58:01.792 DEBUG:boto:Token: None 2023-11-02T18:58:01.793 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/ 2023-11-02T18:58:01.793 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T18:58:01.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T18:58:01.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063a7c9e4aad1f803-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.800 DEBUG:boto:path=/ibtljn-63/obj58 2023-11-02T18:58:01.801 DEBUG:boto:auth_path=/ibtljn-63/obj58 2023-11-02T18:58:01.801 DEBUG:boto:Method: PUT 2023-11-02T18:58:01.802 DEBUG:boto:Path: /ibtljn-63/obj58 2023-11-02T18:58:01.802 DEBUG:boto:Data: 2023-11-02T18:58:01.803 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:01.803 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.803 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.804 DEBUG:boto:Params: {} 2023-11-02T18:58:01.804 DEBUG:boto:Token: None 2023-11-02T18:58:01.805 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/obj58 2023-11-02T18:58:01.805 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rYdGhcqgBakArbK4eQmOM5YlKII= 2023-11-02T18:58:01.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rYdGhcqgBakArbK4eQmOM5YlKII='} 2023-11-02T18:58:01.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fed7e4dfa48aa04a-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.851 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:01.852 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:01.852 DEBUG:boto:Method: HEAD 2023-11-02T18:58:01.853 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:01.853 DEBUG:boto:Data: 2023-11-02T18:58:01.854 DEBUG:boto:Headers: {} 2023-11-02T18:58:01.854 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.854 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.855 DEBUG:boto:Params: {} 2023-11-02T18:58:01.855 DEBUG:boto:Token: None 2023-11-02T18:58:01.856 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/ 2023-11-02T18:58:01.856 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T18:58:01.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T18:58:01.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd44eee09c2d516f-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.864 DEBUG:boto:path=/ibtljn-63/obj87 2023-11-02T18:58:01.865 DEBUG:boto:auth_path=/ibtljn-63/obj87 2023-11-02T18:58:01.865 DEBUG:boto:Method: PUT 2023-11-02T18:58:01.866 DEBUG:boto:Path: /ibtljn-63/obj87 2023-11-02T18:58:01.866 DEBUG:boto:Data: 2023-11-02T18:58:01.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:01.867 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.868 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.868 DEBUG:boto:Params: {} 2023-11-02T18:58:01.869 DEBUG:boto:Token: None 2023-11-02T18:58:01.869 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/obj87 2023-11-02T18:58:01.869 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TSU1MKpZuejo8MHA0nDLnszk1r8= 2023-11-02T18:58:01.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSU1MKpZuejo8MHA0nDLnszk1r8='} 2023-11-02T18:58:01.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005bb560cf52f98f25-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.882 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:01.882 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:01.883 DEBUG:boto:Method: HEAD 2023-11-02T18:58:01.883 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:01.884 DEBUG:boto:Data: 2023-11-02T18:58:01.884 DEBUG:boto:Headers: {} 2023-11-02T18:58:01.884 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.885 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.885 DEBUG:boto:Params: {} 2023-11-02T18:58:01.886 DEBUG:boto:Token: None 2023-11-02T18:58:01.886 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/ 2023-11-02T18:58:01.887 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T18:58:01.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T18:58:01.893 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053ad6928ecf1bd47-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.894 DEBUG:boto:path=/ibtljn-63/obj116 2023-11-02T18:58:01.894 DEBUG:boto:auth_path=/ibtljn-63/obj116 2023-11-02T18:58:01.895 DEBUG:boto:Method: PUT 2023-11-02T18:58:01.895 DEBUG:boto:Path: /ibtljn-63/obj116 2023-11-02T18:58:01.896 DEBUG:boto:Data: 2023-11-02T18:58:01.896 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:01.897 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.897 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.898 DEBUG:boto:Params: {} 2023-11-02T18:58:01.898 DEBUG:boto:Token: None 2023-11-02T18:58:01.899 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/obj116 2023-11-02T18:58:01.899 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c/O10xEaST8VrT0SqYqFGtTlTXA= 2023-11-02T18:58:01.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c/O10xEaST8VrT0SqYqFGtTlTXA='} 2023-11-02T18:58:01.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc875cdfc3ab4dea-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.947 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:01.948 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:01.948 DEBUG:boto:Method: HEAD 2023-11-02T18:58:01.949 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:01.949 DEBUG:boto:Data: 2023-11-02T18:58:01.949 DEBUG:boto:Headers: {} 2023-11-02T18:58:01.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.950 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.951 DEBUG:boto:Params: {} 2023-11-02T18:58:01.951 DEBUG:boto:Token: None 2023-11-02T18:58:01.952 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/ 2023-11-02T18:58:01.952 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T18:58:01.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T18:58:01.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaf8fa01d3eacc16-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.960 DEBUG:boto:path=/ibtljn-63/obj145 2023-11-02T18:58:01.960 DEBUG:boto:auth_path=/ibtljn-63/obj145 2023-11-02T18:58:01.961 DEBUG:boto:Method: PUT 2023-11-02T18:58:01.961 DEBUG:boto:Path: /ibtljn-63/obj145 2023-11-02T18:58:01.962 DEBUG:boto:Data: 2023-11-02T18:58:01.962 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:01.963 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:01.963 DEBUG:boto:Port: 8000 2023-11-02T18:58:01.963 DEBUG:boto:Params: {} 2023-11-02T18:58:01.964 DEBUG:boto:Token: None 2023-11-02T18:58:01.964 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:01 GMT /ibtljn-63/obj145 2023-11-02T18:58:01.965 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sATZYa3juQ4lsr6sXN+QwJd18LQ= 2023-11-02T18:58:01.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sATZYa3juQ4lsr6sXN+QwJd18LQ='} 2023-11-02T18:58:01.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b85047405bd1098-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:01.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:58:01.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:58:01.977 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:58:02.026 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:58:02.027 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:58:02.114 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:58:02.115 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-63 2023-11-02T18:58:02.139 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 25 2023-11-02T18:58:02.266 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:02.267 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:02.267 DEBUG:boto:Method: HEAD 2023-11-02T18:58:02.268 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:02.268 DEBUG:boto:Data: 2023-11-02T18:58:02.268 DEBUG:boto:Headers: {} 2023-11-02T18:58:02.269 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.269 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.270 DEBUG:boto:Params: {} 2023-11-02T18:58:02.270 DEBUG:boto:Token: None 2023-11-02T18:58:02.271 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/ 2023-11-02T18:58:02.271 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T18:58:02.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T18:58:02.278 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000839a9073a29b2e6b-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.279 DEBUG:boto:path=/ibtljn-63/obj31 2023-11-02T18:58:02.279 DEBUG:boto:auth_path=/ibtljn-63/obj31 2023-11-02T18:58:02.280 DEBUG:boto:Method: PUT 2023-11-02T18:58:02.280 DEBUG:boto:Path: /ibtljn-63/obj31 2023-11-02T18:58:02.280 DEBUG:boto:Data: 2023-11-02T18:58:02.281 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:02.281 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.282 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.282 DEBUG:boto:Params: {} 2023-11-02T18:58:02.283 DEBUG:boto:Token: None 2023-11-02T18:58:02.283 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/obj31 2023-11-02T18:58:02.284 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Bk2Jg/pKnDCSbFHP59KJ5pyJQsE= 2023-11-02T18:58:02.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bk2Jg/pKnDCSbFHP59KJ5pyJQsE='} 2023-11-02T18:58:02.326 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f74ff07f65438d2-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.327 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:02.328 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:02.328 DEBUG:boto:Method: HEAD 2023-11-02T18:58:02.329 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:02.329 DEBUG:boto:Data: 2023-11-02T18:58:02.330 DEBUG:boto:Headers: {} 2023-11-02T18:58:02.330 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.331 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.331 DEBUG:boto:Params: {} 2023-11-02T18:58:02.332 DEBUG:boto:Token: None 2023-11-02T18:58:02.332 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/ 2023-11-02T18:58:02.334 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T18:58:02.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T18:58:02.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f5ff619acaee089-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.342 DEBUG:boto:path=/ibtljn-63/obj62 2023-11-02T18:58:02.342 DEBUG:boto:auth_path=/ibtljn-63/obj62 2023-11-02T18:58:02.343 DEBUG:boto:Method: PUT 2023-11-02T18:58:02.343 DEBUG:boto:Path: /ibtljn-63/obj62 2023-11-02T18:58:02.344 DEBUG:boto:Data: 2023-11-02T18:58:02.344 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:02.344 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.345 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.345 DEBUG:boto:Params: {} 2023-11-02T18:58:02.346 DEBUG:boto:Token: None 2023-11-02T18:58:02.346 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/obj62 2023-11-02T18:58:02.347 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0OmLa8OcO6YRjVeUAuyOnXe4deY= 2023-11-02T18:58:02.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0OmLa8OcO6YRjVeUAuyOnXe4deY='} 2023-11-02T18:58:02.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ab0364662b620e7-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.358 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:02.359 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:02.359 DEBUG:boto:Method: HEAD 2023-11-02T18:58:02.360 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:02.360 DEBUG:boto:Data: 2023-11-02T18:58:02.361 DEBUG:boto:Headers: {} 2023-11-02T18:58:02.361 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.361 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.362 DEBUG:boto:Params: {} 2023-11-02T18:58:02.362 DEBUG:boto:Token: None 2023-11-02T18:58:02.363 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/ 2023-11-02T18:58:02.363 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T18:58:02.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T18:58:02.370 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b974a0e74a0fc4d6-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.371 DEBUG:boto:path=/ibtljn-63/obj93 2023-11-02T18:58:02.371 DEBUG:boto:auth_path=/ibtljn-63/obj93 2023-11-02T18:58:02.372 DEBUG:boto:Method: PUT 2023-11-02T18:58:02.372 DEBUG:boto:Path: /ibtljn-63/obj93 2023-11-02T18:58:02.373 DEBUG:boto:Data: 2023-11-02T18:58:02.373 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:02.373 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.374 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.374 DEBUG:boto:Params: {} 2023-11-02T18:58:02.375 DEBUG:boto:Token: None 2023-11-02T18:58:02.375 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/obj93 2023-11-02T18:58:02.376 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RpHE0ZxOC4JrLb2hOe+o92owwBY= 2023-11-02T18:58:02.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RpHE0ZxOC4JrLb2hOe+o92owwBY='} 2023-11-02T18:58:02.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbc476bccf90de04-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.423 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:02.424 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:02.424 DEBUG:boto:Method: HEAD 2023-11-02T18:58:02.424 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:02.425 DEBUG:boto:Data: 2023-11-02T18:58:02.425 DEBUG:boto:Headers: {} 2023-11-02T18:58:02.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.426 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.427 DEBUG:boto:Params: {} 2023-11-02T18:58:02.427 DEBUG:boto:Token: None 2023-11-02T18:58:02.428 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/ 2023-11-02T18:58:02.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T18:58:02.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T18:58:02.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6ea3a3b3b841083-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.437 DEBUG:boto:path=/ibtljn-63/obj124 2023-11-02T18:58:02.437 DEBUG:boto:auth_path=/ibtljn-63/obj124 2023-11-02T18:58:02.438 DEBUG:boto:Method: PUT 2023-11-02T18:58:02.438 DEBUG:boto:Path: /ibtljn-63/obj124 2023-11-02T18:58:02.438 DEBUG:boto:Data: 2023-11-02T18:58:02.439 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:02.439 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.440 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.440 DEBUG:boto:Params: {} 2023-11-02T18:58:02.441 DEBUG:boto:Token: None 2023-11-02T18:58:02.441 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/obj124 2023-11-02T18:58:02.442 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0r8TiryND2XSk18xD5a2PxEgzp4= 2023-11-02T18:58:02.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0r8TiryND2XSk18xD5a2PxEgzp4='} 2023-11-02T18:58:02.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e7c772a80f79fe4-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.453 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:02.454 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:02.454 DEBUG:boto:Method: HEAD 2023-11-02T18:58:02.454 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:02.455 DEBUG:boto:Data: 2023-11-02T18:58:02.455 DEBUG:boto:Headers: {} 2023-11-02T18:58:02.456 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.456 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.457 DEBUG:boto:Params: {} 2023-11-02T18:58:02.457 DEBUG:boto:Token: None 2023-11-02T18:58:02.457 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/ 2023-11-02T18:58:02.458 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T18:58:02.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T18:58:02.465 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3a5832f1ffbbe0b-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.465 DEBUG:boto:path=/ibtljn-63/obj155 2023-11-02T18:58:02.466 DEBUG:boto:auth_path=/ibtljn-63/obj155 2023-11-02T18:58:02.466 DEBUG:boto:Method: PUT 2023-11-02T18:58:02.467 DEBUG:boto:Path: /ibtljn-63/obj155 2023-11-02T18:58:02.467 DEBUG:boto:Data: 2023-11-02T18:58:02.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:02.468 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.468 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.468 DEBUG:boto:Params: {} 2023-11-02T18:58:02.469 DEBUG:boto:Token: None 2023-11-02T18:58:02.469 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/obj155 2023-11-02T18:58:02.470 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XcCyHSdrNqGKLAz0CV4yCfHZ3D8= 2023-11-02T18:58:02.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XcCyHSdrNqGKLAz0CV4yCfHZ3D8='} 2023-11-02T18:58:02.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4f0c77483ad0bf6-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:58:02.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T18:58:02.516 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T18:58:02.566 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:58:02.566 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:58:02.658 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T18:58:02.658 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-63 2023-11-02T18:58:02.684 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 30 2023-11-02T18:58:02.816 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:02.816 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:02.817 DEBUG:boto:Method: HEAD 2023-11-02T18:58:02.817 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:02.818 DEBUG:boto:Data: 2023-11-02T18:58:02.818 DEBUG:boto:Headers: {} 2023-11-02T18:58:02.818 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.819 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.819 DEBUG:boto:Params: {} 2023-11-02T18:58:02.820 DEBUG:boto:Token: None 2023-11-02T18:58:02.820 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/ 2023-11-02T18:58:02.821 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T18:58:02.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T18:58:02.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c10ac17c4e37c61-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.828 DEBUG:boto:path=/ibtljn-63/obj37 2023-11-02T18:58:02.829 DEBUG:boto:auth_path=/ibtljn-63/obj37 2023-11-02T18:58:02.829 DEBUG:boto:Method: PUT 2023-11-02T18:58:02.830 DEBUG:boto:Path: /ibtljn-63/obj37 2023-11-02T18:58:02.830 DEBUG:boto:Data: 2023-11-02T18:58:02.830 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:02.831 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.831 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.832 DEBUG:boto:Params: {} 2023-11-02T18:58:02.832 DEBUG:boto:Token: None 2023-11-02T18:58:02.833 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/obj37 2023-11-02T18:58:02.833 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jnDXNCgJvYoUhU1No/HgAPmCAck= 2023-11-02T18:58:02.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jnDXNCgJvYoUhU1No/HgAPmCAck='} 2023-11-02T18:58:02.844 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000851fc34cbc20edb4-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.844 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:02.845 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:02.845 DEBUG:boto:Method: HEAD 2023-11-02T18:58:02.846 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:02.846 DEBUG:boto:Data: 2023-11-02T18:58:02.846 DEBUG:boto:Headers: {} 2023-11-02T18:58:02.847 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.847 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.848 DEBUG:boto:Params: {} 2023-11-02T18:58:02.848 DEBUG:boto:Token: None 2023-11-02T18:58:02.849 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/ 2023-11-02T18:58:02.849 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T18:58:02.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T18:58:02.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b29bde3be3c9b5bd-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.857 DEBUG:boto:path=/ibtljn-63/obj74 2023-11-02T18:58:02.857 DEBUG:boto:auth_path=/ibtljn-63/obj74 2023-11-02T18:58:02.858 DEBUG:boto:Method: PUT 2023-11-02T18:58:02.858 DEBUG:boto:Path: /ibtljn-63/obj74 2023-11-02T18:58:02.859 DEBUG:boto:Data: 2023-11-02T18:58:02.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:02.859 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.860 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.860 DEBUG:boto:Params: {} 2023-11-02T18:58:02.860 DEBUG:boto:Token: None 2023-11-02T18:58:02.861 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/obj74 2023-11-02T18:58:02.861 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8rrAn2ebV86nm5P8Y2DFrcXFc4E= 2023-11-02T18:58:02.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8rrAn2ebV86nm5P8Y2DFrcXFc4E='} 2023-11-02T18:58:02.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009db2f2a5a7e78349-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.911 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:02.911 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:02.912 DEBUG:boto:Method: HEAD 2023-11-02T18:58:02.912 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:02.912 DEBUG:boto:Data: 2023-11-02T18:58:02.913 DEBUG:boto:Headers: {} 2023-11-02T18:58:02.913 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.914 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.914 DEBUG:boto:Params: {} 2023-11-02T18:58:02.914 DEBUG:boto:Token: None 2023-11-02T18:58:02.915 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/ 2023-11-02T18:58:02.915 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T18:58:02.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T18:58:02.924 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f21b4b03f34a85a1-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.924 DEBUG:boto:path=/ibtljn-63/obj111 2023-11-02T18:58:02.925 DEBUG:boto:auth_path=/ibtljn-63/obj111 2023-11-02T18:58:02.925 DEBUG:boto:Method: PUT 2023-11-02T18:58:02.926 DEBUG:boto:Path: /ibtljn-63/obj111 2023-11-02T18:58:02.926 DEBUG:boto:Data: 2023-11-02T18:58:02.927 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:02.927 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.928 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.928 DEBUG:boto:Params: {} 2023-11-02T18:58:02.928 DEBUG:boto:Token: None 2023-11-02T18:58:02.929 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/obj111 2023-11-02T18:58:02.929 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:zkxXDRZ9NDBwwR0hsuMMkLNxwuU= 2023-11-02T18:58:02.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zkxXDRZ9NDBwwR0hsuMMkLNxwuU='} 2023-11-02T18:58:02.939 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063a53687d8aac81e-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.940 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:02.940 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:02.941 DEBUG:boto:Method: HEAD 2023-11-02T18:58:02.941 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:02.942 DEBUG:boto:Data: 2023-11-02T18:58:02.942 DEBUG:boto:Headers: {} 2023-11-02T18:58:02.943 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.943 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.944 DEBUG:boto:Params: {} 2023-11-02T18:58:02.944 DEBUG:boto:Token: None 2023-11-02T18:58:02.945 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/ 2023-11-02T18:58:02.946 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T18:58:02.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T18:58:02.953 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b08a3cadc50e91e-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:02.954 DEBUG:boto:path=/ibtljn-63/obj148 2023-11-02T18:58:02.954 DEBUG:boto:auth_path=/ibtljn-63/obj148 2023-11-02T18:58:02.954 DEBUG:boto:Method: PUT 2023-11-02T18:58:02.955 DEBUG:boto:Path: /ibtljn-63/obj148 2023-11-02T18:58:02.955 DEBUG:boto:Data: 2023-11-02T18:58:02.956 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:02.956 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:02.957 DEBUG:boto:Port: 8000 2023-11-02T18:58:02.957 DEBUG:boto:Params: {} 2023-11-02T18:58:02.957 DEBUG:boto:Token: None 2023-11-02T18:58:02.958 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:02 GMT /ibtljn-63/obj148 2023-11-02T18:58:02.959 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XQxGr2z5Ov3dSGhv54i0saLdeKE= 2023-11-02T18:58:02.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XQxGr2z5Ov3dSGhv54i0saLdeKE='} 2023-11-02T18:58:03.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a4de0d57491a170-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:03.003 DEBUG:boto:path=/ibtljn-63/ 2023-11-02T18:58:03.004 DEBUG:boto:auth_path=/ibtljn-63/ 2023-11-02T18:58:03.004 DEBUG:boto:Method: HEAD 2023-11-02T18:58:03.005 DEBUG:boto:Path: /ibtljn-63/ 2023-11-02T18:58:03.005 DEBUG:boto:Data: 2023-11-02T18:58:03.006 DEBUG:boto:Headers: {} 2023-11-02T18:58:03.006 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:03.006 DEBUG:boto:Port: 8000 2023-11-02T18:58:03.007 DEBUG:boto:Params: {} 2023-11-02T18:58:03.008 DEBUG:boto:Token: None 2023-11-02T18:58:03.008 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 18:58:03 GMT /ibtljn-63/ 2023-11-02T18:58:03.009 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:t4Mmfrq8g0lZrG9AOlcKBI4PfJk= 2023-11-02T18:58:03.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t4Mmfrq8g0lZrG9AOlcKBI4PfJk=', 'Content-Length': '0'} 2023-11-02T18:58:03.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a679cedb7412d5e6-006543f13b-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:03.018 DEBUG:boto:path=/ibtljn-63/obj185 2023-11-02T18:58:03.018 DEBUG:boto:auth_path=/ibtljn-63/obj185 2023-11-02T18:58:03.019 DEBUG:boto:Method: PUT 2023-11-02T18:58:03.019 DEBUG:boto:Path: /ibtljn-63/obj185 2023-11-02T18:58:03.020 DEBUG:boto:Data: 2023-11-02T18:58:03.020 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T18:58:03.021 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T18:58:03.021 DEBUG:boto:Port: 8000 2023-11-02T18:58:03.022 DEBUG:boto:Params: {} 2023-11-02T18:58:03.022 DEBUG:boto:Token: None 2023-11-02T18:58:03.023 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 18:58:03 GMT /ibtljn-63/obj185 2023-11-02T18:58:03.023 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pYD7xIQ2IH7KUIgc1P3uzx13z3c= 2023-11-02T18:58:03.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pYD7xIQ2IH7KUIgc1P3uzx13z3c='} 2023-11-02T18:58:03.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed8fecf151ddd81f-006543f13b-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T18:58:03.034 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-63 2023-11-02T18:58:03.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:58:03.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:58:03.036 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:58:03.088 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:58:03.089 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:58:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:03.153+0000 7fb2b948cf40 1 waiting to reach incremental sync.. 2023-11-02T18:58:08.161 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:08.153+0000 7fb2b948cf40 1 waiting to reach incremental sync.. 2023-11-02T18:58:13.161 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:13.157+0000 7fb2b948cf40 1 waiting to reach incremental sync.. 2023-11-02T18:58:18.169 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:18.161+0000 7fb2b948cf40 1 bucket sync caught up with source: 2023-11-02T18:58:18.169 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.70.6, , , , , , 00000000002.130.6, , , , , 00000000002.114.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.97.6] 2023-11-02T18:58:18.169 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.70.6, , , , , , 00000000002.130.6, , , , , 00000000002.114.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.97.6] 2023-11-02T18:58:18.169 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:18.161+0000 7fb2b948cf40 0 bucket checkpoint complete 2023-11-02T18:58:18.199 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-63 2023-11-02T18:58:18.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:58:18.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T18:58:18.200 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T18:58:18.256 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T18:58:18.256 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T18:58:18.319 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:18.313+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:58:23.320 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:23.313+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:58:28.321 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:28.313+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:58:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:33.317+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:58:38.323 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:38.317+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:58:43.324 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:43.317+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:58:48.325 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:48.317+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:58:53.326 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:53.321+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:58:58.327 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:58:58.321+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:59:03.328 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:59:03.321+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:59:08.329 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:59:08.321+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:59:13.330 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:59:13.325+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:59:18.331 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:59:18.325+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:59:23.332 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:59:23.325+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:59:28.333 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:59:28.329+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:59:33.334 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:59:33.328+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:59:38.335 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:59:38.328+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:59:43.336 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:59:43.328+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:59:48.337 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:59:48.332+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:59:53.338 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:59:53.332+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T18:59:58.339 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T18:59:58.332+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:00:03.340 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:00:03.332+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:00:08.341 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:00:08.336+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:00:13.343 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:00:13.336+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:00:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:00:18.336+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:00:23.345 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:00:23.336+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:00:28.346 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:00:28.340+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:00:33.347 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:00:33.340+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:00:38.348 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:00:38.340+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:00:43.349 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:00:43.340+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:00:48.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:00:48.344+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:00:53.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:00:53.348+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:00:58.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:00:58.344+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:01:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:01:03.348+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:01:08.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:01:08.348+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:01:13.356 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:01:13.347+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:01:18.358 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:01:18.351+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:01:23.358 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:01:23.351+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:01:28.359 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:01:28.351+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:01:33.361 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:01:33.351+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:01:38.362 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:01:38.355+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:01:43.363 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:01:43.359+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:01:48.364 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:01:48.359+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:01:53.365 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:01:53.359+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:01:58.363+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:02:03.367 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:02:03.363+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:02:08.368 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:02:08.363+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:02:13.370 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:02:13.367+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:02:18.371 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:02:18.367+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:02:23.372 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:02:23.367+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:02:28.373 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:02:28.371+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:02:33.375 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:02:33.371+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:02:38.376 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:02:38.371+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:02:43.377 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:02:43.371+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:02:48.378 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:02:48.374+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:02:53.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:02:53.374+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:02:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:02:58.374+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:03:03.382 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:03.378+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:03:08.383 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:08.378+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:03:13.384 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:13.378+0000 7ff2bff08f40 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-11-02T19:03:13.385 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:13.378+0000 7ff2bff08f40 0 bucket sync checkpoint failed: (110) Connection timed out 2023-11-02T19:03:13.385 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:13.378+0000 7ff2bff08f40 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-11-02T19:03:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-11-02T19:03:13.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2023-11-02T19:03:13.422 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:13.422 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:13.423 DEBUG:boto:Method: PUT 2023-11-02T19:03:13.423 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:13.423 DEBUG:boto:Data: 2023-11-02T19:03:13.424 DEBUG:boto:Headers: {} 2023-11-02T19:03:13.424 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:13.425 DEBUG:boto:Port: 8000 2023-11-02T19:03:13.425 DEBUG:boto:Params: {} 2023-11-02T19:03:13.426 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T19:03:13.426 DEBUG:boto:Token: None 2023-11-02T19:03:13.427 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 19:03:13 GMT /ibtljn-64/ 2023-11-02T19:03:13.427 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O//SoENTdPNMorgeaXM8yAhJ9dg= 2023-11-02T19:03:13.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:13 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O//SoENTdPNMorgeaXM8yAhJ9dg=', 'Content-Length': '0'} 2023-11-02T19:03:13.464 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000861500cf017e7de9-006543f271-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:13 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:13.464 DEBUG:rgw_multi.tests:created bucket=ibtljn-64 2023-11-02T19:03:13.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:03:13.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:03:13.466 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:03:13.509 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:13.509 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:13.570 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:03:13.597 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:03:13.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:03:13.598 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:03:13.598 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:03:13.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:13.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:03:13.599 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:03:13.599 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:03:13.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:13.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:03:13.600 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951793.458718_622.1", 2023-11-02T19:03:13.600 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:03:13.458718Z" 2023-11-02T19:03:13.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:13.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:03:13.601 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:03:13.601 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:03:13.601 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:03:13.601 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:03:13.602 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951793.458718_622.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:03:13.603 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:03:13.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:03:13.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:03:13.604 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:03:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:13.747 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:13.742+0000 7f48f38cff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:03:13.749 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:13.742+0000 7f48f38cff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:03:13.776 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:03:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:03:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:03:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:03:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:03:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:03:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:03:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:03:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:03:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:03:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:03:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:03:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:03:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:03:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:03:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:03:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:03:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:03:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:03:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:03:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:03:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:03:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:03:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:03:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:03:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:03:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:03:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:03:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:03:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:03:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:03:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:03:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:03:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:03:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951793.447909_619.1", 2023-11-02T19:03:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:03:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:03:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:03:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:13.447909Z", 2023-11-02T19:03:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:03:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:03:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:03:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:03:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:03:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:03:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:03:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:03:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:03:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:03:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:03:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:03:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:03:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:03:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:03:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:03:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:03:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:03:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:03:13Z" 2023-11-02T19:03:13.791 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:03:13.792 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951793.447909_619.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:13.447909Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:03:13Z" } 2023-11-02T19:03:13.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951793.447909_619.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:13.447909Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:03:13.793 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951793.458718_622.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:03:13.794 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951793.447909_619.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:03:13.794 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951793.458718_622.1 target=1_1698951793.447909_619.1 2023-11-02T19:03:18.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:03:18.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:03:18.797 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:03:18.843 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:18.844 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:18.903 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:18.898+0000 7f97782ebf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:03:18.905 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:18.898+0000 7f97782ebf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:03:18.933 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:03:18.933 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:03:18.934 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:03:18.934 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:03:18.934 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:03:18.934 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:03:18.934 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:03:18.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:18.935 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:03:18.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:03:18.935 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:03:18.936 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:03:18.936 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:03:18.936 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:03:18.936 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:03:18.937 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:03:18.937 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:03:18.937 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:03:18.937 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:03:18.938 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:03:18.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:18.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:03:18.938 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:03:18.939 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:03:18.939 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:03:18.939 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:03:18.939 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:03:18.939 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:03:18.940 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:03:18.940 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:03:18.940 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:03:18.940 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:03:18.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:18.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:03:18.941 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:03:18.941 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:03:18.941 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:03:18.941 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951793.458718_622.1", 2023-11-02T19:03:18.942 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:03:18.942 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:03:18.942 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:03:18.942 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:13.458718Z", 2023-11-02T19:03:18.943 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:03:18.943 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:03:18.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:18.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:03:18.943 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:03:18.944 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:03:18.944 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:03:18.944 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:03:18.944 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:03:18.945 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:03:18.945 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:03:18.945 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:03:18.945 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:03:18.945 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:03:18.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:03:18.946 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:03:18.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:18.946 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:03:18.947 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:03:18.947 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:03:18.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:03:18.947 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:03:18Z" 2023-11-02T19:03:18.948 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:03:18.949 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951793.458718_622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:13.458718Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:03:18Z" } 2023-11-02T19:03:18.949 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951793.458718_622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:13.458718Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:03:18.950 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951793.458718_622.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:03:18.950 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951793.458718_622.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:03:18.951 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:03:18.951 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:18.952 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:18.952 DEBUG:boto:Method: HEAD 2023-11-02T19:03:18.953 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:18.953 DEBUG:boto:Data: 2023-11-02T19:03:18.954 DEBUG:boto:Headers: {} 2023-11-02T19:03:18.954 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:18.955 DEBUG:boto:Port: 8000 2023-11-02T19:03:18.955 DEBUG:boto:Params: {} 2023-11-02T19:03:18.955 DEBUG:boto:Token: None 2023-11-02T19:03:18.956 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:18 GMT /ibtljn-64/ 2023-11-02T19:03:18.956 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tVAD35AhX1tRx8Jsnr9En4BjckA= 2023-11-02T19:03:18.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tVAD35AhX1tRx8Jsnr9En4BjckA=', 'Content-Length': '0'} 2023-11-02T19:03:18.961 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc172f9bdc13f8ed-006543f276-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:18.962 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:03:18.963 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:03:18.964 DEBUG:boto:Method: PUT 2023-11-02T19:03:18.965 DEBUG:boto:Path: /ibtljn-64/obj11 2023-11-02T19:03:18.965 DEBUG:boto:Data: 2023-11-02T19:03:18.966 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:18.966 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:18.967 DEBUG:boto:Port: 8000 2023-11-02T19:03:18.967 DEBUG:boto:Params: {} 2023-11-02T19:03:18.968 DEBUG:boto:Token: None 2023-11-02T19:03:18.968 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:18 GMT /ibtljn-64/obj11 2023-11-02T19:03:18.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lX1Wi4KlTGb4KtGuG0ko8uElcdM= 2023-11-02T19:03:18.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lX1Wi4KlTGb4KtGuG0ko8uElcdM='} 2023-11-02T19:03:18.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cc16efa6bb4f51d-006543f276-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:18.986 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:18.986 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:18.987 DEBUG:boto:Method: HEAD 2023-11-02T19:03:18.987 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:18.988 DEBUG:boto:Data: 2023-11-02T19:03:18.988 DEBUG:boto:Headers: {} 2023-11-02T19:03:18.989 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:18.989 DEBUG:boto:Port: 8000 2023-11-02T19:03:18.990 DEBUG:boto:Params: {} 2023-11-02T19:03:18.990 DEBUG:boto:Token: None 2023-11-02T19:03:18.991 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:18 GMT /ibtljn-64/ 2023-11-02T19:03:18.991 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tVAD35AhX1tRx8Jsnr9En4BjckA= 2023-11-02T19:03:18.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:18 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tVAD35AhX1tRx8Jsnr9En4BjckA=', 'Content-Length': '0'} 2023-11-02T19:03:18.995 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7ba5df0f9f4444f-006543f276-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:18.996 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:03:18.996 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:03:18.998 DEBUG:boto:Method: PUT 2023-11-02T19:03:18.998 DEBUG:boto:Path: /ibtljn-64/obj22 2023-11-02T19:03:18.998 DEBUG:boto:Data: 2023-11-02T19:03:18.999 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:18.999 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:19.000 DEBUG:boto:Port: 8000 2023-11-02T19:03:19.000 DEBUG:boto:Params: {} 2023-11-02T19:03:19.001 DEBUG:boto:Token: None 2023-11-02T19:03:19.001 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:19 GMT /ibtljn-64/obj22 2023-11-02T19:03:19.002 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ppQH88J6xcnezLOnK3HTXbm6fos= 2023-11-02T19:03:19.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ppQH88J6xcnezLOnK3HTXbm6fos='} 2023-11-02T19:03:19.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007baa30bad6238e25-006543f276-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:19.048 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:19.048 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:19.049 DEBUG:boto:Method: HEAD 2023-11-02T19:03:19.049 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:19.049 DEBUG:boto:Data: 2023-11-02T19:03:19.050 DEBUG:boto:Headers: {} 2023-11-02T19:03:19.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:19.051 DEBUG:boto:Port: 8000 2023-11-02T19:03:19.051 DEBUG:boto:Params: {} 2023-11-02T19:03:19.052 DEBUG:boto:Token: None 2023-11-02T19:03:19.052 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:19 GMT /ibtljn-64/ 2023-11-02T19:03:19.053 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MZHrx3Xv3I8ta0ibxnpfsxL3ISE= 2023-11-02T19:03:19.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MZHrx3Xv3I8ta0ibxnpfsxL3ISE=', 'Content-Length': '0'} 2023-11-02T19:03:19.057 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8aafdc0214a49e1-006543f277-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:19.058 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:03:19.058 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:03:19.059 DEBUG:boto:Method: PUT 2023-11-02T19:03:19.059 DEBUG:boto:Path: /ibtljn-64/obj33 2023-11-02T19:03:19.060 DEBUG:boto:Data: 2023-11-02T19:03:19.060 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:19.061 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:19.061 DEBUG:boto:Port: 8000 2023-11-02T19:03:19.061 DEBUG:boto:Params: {} 2023-11-02T19:03:19.062 DEBUG:boto:Token: None 2023-11-02T19:03:19.062 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:19 GMT /ibtljn-64/obj33 2023-11-02T19:03:19.063 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ckAbKuLCx11a7lDlKMuMVIt9E0A= 2023-11-02T19:03:19.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ckAbKuLCx11a7lDlKMuMVIt9E0A='} 2023-11-02T19:03:19.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b257b361ee27c473-006543f277-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:19.076 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:19.077 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:19.077 DEBUG:boto:Method: HEAD 2023-11-02T19:03:19.077 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:19.078 DEBUG:boto:Data: 2023-11-02T19:03:19.078 DEBUG:boto:Headers: {} 2023-11-02T19:03:19.079 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:19.079 DEBUG:boto:Port: 8000 2023-11-02T19:03:19.080 DEBUG:boto:Params: {} 2023-11-02T19:03:19.080 DEBUG:boto:Token: None 2023-11-02T19:03:19.080 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:19 GMT /ibtljn-64/ 2023-11-02T19:03:19.081 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MZHrx3Xv3I8ta0ibxnpfsxL3ISE= 2023-11-02T19:03:19.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MZHrx3Xv3I8ta0ibxnpfsxL3ISE=', 'Content-Length': '0'} 2023-11-02T19:03:19.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051a3e2fd7688b01d-006543f277-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:19.086 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:03:19.086 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:03:19.090 DEBUG:boto:Method: PUT 2023-11-02T19:03:19.091 DEBUG:boto:Path: /ibtljn-64/obj44 2023-11-02T19:03:19.091 DEBUG:boto:Data: 2023-11-02T19:03:19.092 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:19.092 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:19.092 DEBUG:boto:Port: 8000 2023-11-02T19:03:19.093 DEBUG:boto:Params: {} 2023-11-02T19:03:19.093 DEBUG:boto:Token: None 2023-11-02T19:03:19.094 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:19 GMT /ibtljn-64/obj44 2023-11-02T19:03:19.094 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EDiRgllfrT5IovpFf8UtHc/hA3Y= 2023-11-02T19:03:19.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EDiRgllfrT5IovpFf8UtHc/hA3Y='} 2023-11-02T19:03:19.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000444b1b6d2cc00127-006543f277-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:19.139 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:19.140 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:19.140 DEBUG:boto:Method: HEAD 2023-11-02T19:03:19.141 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:19.141 DEBUG:boto:Data: 2023-11-02T19:03:19.141 DEBUG:boto:Headers: {} 2023-11-02T19:03:19.142 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:19.142 DEBUG:boto:Port: 8000 2023-11-02T19:03:19.143 DEBUG:boto:Params: {} 2023-11-02T19:03:19.143 DEBUG:boto:Token: None 2023-11-02T19:03:19.143 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:19 GMT /ibtljn-64/ 2023-11-02T19:03:19.144 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MZHrx3Xv3I8ta0ibxnpfsxL3ISE= 2023-11-02T19:03:19.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MZHrx3Xv3I8ta0ibxnpfsxL3ISE=', 'Content-Length': '0'} 2023-11-02T19:03:19.149 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2fff499982afbd9-006543f277-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:19.150 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:03:19.150 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:03:19.151 DEBUG:boto:Method: PUT 2023-11-02T19:03:19.151 DEBUG:boto:Path: /ibtljn-64/obj55 2023-11-02T19:03:19.152 DEBUG:boto:Data: 2023-11-02T19:03:19.152 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:19.153 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:19.153 DEBUG:boto:Port: 8000 2023-11-02T19:03:19.153 DEBUG:boto:Params: {} 2023-11-02T19:03:19.154 DEBUG:boto:Token: None 2023-11-02T19:03:19.154 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:19 GMT /ibtljn-64/obj55 2023-11-02T19:03:19.155 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ggJo0E3Mw9paiu6hBqpeAbXxQM4= 2023-11-02T19:03:19.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ggJo0E3Mw9paiu6hBqpeAbXxQM4='} 2023-11-02T19:03:19.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2d0253ad33b0ab9-006543f277-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:19.164 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-64 2023-11-02T19:03:19.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:03:19.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:03:19.166 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T19:03:19.219 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:19.219 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:19.288 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:19.282+0000 7fce365baf40 1 waiting to reach incremental sync.. 2023-11-02T19:03:24.289 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:24.282+0000 7fce365baf40 1 waiting to reach incremental sync.. 2023-11-02T19:03:29.292 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:29.286+0000 7fce365baf40 1 bucket sync caught up with source: 2023-11-02T19:03:29.292 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.83.6, 00000000002.98.6, 00000000002.123.6] 2023-11-02T19:03:29.293 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000001.83.6, 00000000002.98.6, 00000000002.123.6] 2023-11-02T19:03:29.293 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:29.286+0000 7fce365baf40 0 bucket checkpoint complete 2023-11-02T19:03:29.324 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-64 2023-11-02T19:03:29.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:03:29.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:03:29.325 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T19:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:29.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:29.442+0000 7f95ebbaaf40 1 waiting to reach incremental sync.. 2023-11-02T19:03:34.450 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:34.446+0000 7f95ebbaaf40 1 bucket sync caught up with source: 2023-11-02T19:03:34.451 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.331.6, 00000000002.362.6, 00000000002.337.6] 2023-11-02T19:03:34.451 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , , , 00000000001.331.6, 00000000002.362.6, 00000000002.337.6] 2023-11-02T19:03:34.451 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:34.446+0000 7f95ebbaaf40 0 bucket checkpoint complete 2023-11-02T19:03:34.484 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-64 zones={a2, a1} 2023-11-02T19:03:34.485 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:34.486 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:34.486 DEBUG:boto:Method: HEAD 2023-11-02T19:03:34.487 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:34.487 DEBUG:boto:Data: 2023-11-02T19:03:34.487 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.488 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.488 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.489 DEBUG:boto:Params: {} 2023-11-02T19:03:34.489 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:03:34.490 DEBUG:boto:Token: None 2023-11-02T19:03:34.491 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/ 2023-11-02T19:03:34.491 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m/BteY0vC3+roTFZobDibSVpLrg= 2023-11-02T19:03:34.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m/BteY0vC3+roTFZobDibSVpLrg=', 'Content-Length': '0'} 2023-11-02T19:03:34.497 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031303636e3ae86a6-006543f286-1fca-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.498 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:34.498 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:34.499 DEBUG:boto:Method: HEAD 2023-11-02T19:03:34.499 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:34.500 DEBUG:boto:Data: 2023-11-02T19:03:34.500 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.501 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.501 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.502 DEBUG:boto:Params: {} 2023-11-02T19:03:34.502 DEBUG:boto:Token: None 2023-11-02T19:03:34.503 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/ 2023-11-02T19:03:34.503 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m/BteY0vC3+roTFZobDibSVpLrg= 2023-11-02T19:03:34.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m/BteY0vC3+roTFZobDibSVpLrg=', 'Content-Length': '0'} 2023-11-02T19:03:34.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7d2060a9bcfe846-006543f286-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.509 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T19:03:34.509 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:34.509 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:34.510 DEBUG:boto:path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:03:34.510 DEBUG:boto:auth_path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:03:34.511 DEBUG:boto:Method: GET 2023-11-02T19:03:34.511 DEBUG:boto:Path: /ibtljn-64/?versions&max-keys=999 2023-11-02T19:03:34.512 DEBUG:boto:Data: 2023-11-02T19:03:34.512 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.512 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.513 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.513 DEBUG:boto:Params: {} 2023-11-02T19:03:34.514 DEBUG:boto:Token: None 2023-11-02T19:03:34.514 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/?versions 2023-11-02T19:03:34.515 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oKpCTu38mytDI4BoWl19rMorIFA= 2023-11-02T19:03:34.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oKpCTu38mytDI4BoWl19rMorIFA=', 'Content-Length': '0'} 2023-11-02T19:03:34.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000788c858b40f6c0b8-006543f286-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.521 DEBUG:boto:b'ibtljn-64999falseobj11nulltrue2023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:34.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T19:03:34.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T19:03:34.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T19:03:34.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T19:03:34.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T19:03:34.524 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T19:03:34.525 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:34.525 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:34.525 DEBUG:boto:path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:03:34.526 DEBUG:boto:auth_path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:03:34.526 DEBUG:boto:Method: GET 2023-11-02T19:03:34.527 DEBUG:boto:Path: /ibtljn-64/?versions&max-keys=999 2023-11-02T19:03:34.527 DEBUG:boto:Data: 2023-11-02T19:03:34.528 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.528 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.528 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.529 DEBUG:boto:Params: {} 2023-11-02T19:03:34.529 DEBUG:boto:Token: None 2023-11-02T19:03:34.530 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/?versions 2023-11-02T19:03:34.530 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oKpCTu38mytDI4BoWl19rMorIFA= 2023-11-02T19:03:34.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oKpCTu38mytDI4BoWl19rMorIFA=', 'Content-Length': '0'} 2023-11-02T19:03:34.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072fe70927129400b-006543f286-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.536 DEBUG:boto:b'ibtljn-64999falseobj11nulltrue2023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:34.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T19:03:34.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T19:03:34.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T19:03:34.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T19:03:34.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T19:03:34.539 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:34.540 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:34.540 DEBUG:boto:path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:03:34.541 DEBUG:boto:auth_path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:03:34.541 DEBUG:boto:Method: GET 2023-11-02T19:03:34.541 DEBUG:boto:Path: /ibtljn-64/?versions&max-keys=999 2023-11-02T19:03:34.542 DEBUG:boto:Data: 2023-11-02T19:03:34.542 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.543 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.543 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.544 DEBUG:boto:Params: {} 2023-11-02T19:03:34.544 DEBUG:boto:Token: None 2023-11-02T19:03:34.545 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/?versions 2023-11-02T19:03:34.545 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oKpCTu38mytDI4BoWl19rMorIFA= 2023-11-02T19:03:34.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oKpCTu38mytDI4BoWl19rMorIFA=', 'Content-Length': '0'} 2023-11-02T19:03:34.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000434806014de58ae5-006543f286-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.551 DEBUG:boto:b'ibtljn-64999falseobj11nulltrue2023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:34.552 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:34.552 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:34.553 DEBUG:boto:path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:03:34.553 DEBUG:boto:auth_path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:03:34.553 DEBUG:boto:Method: GET 2023-11-02T19:03:34.554 DEBUG:boto:Path: /ibtljn-64/?versions&max-keys=999 2023-11-02T19:03:34.554 DEBUG:boto:Data: 2023-11-02T19:03:34.555 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.555 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.556 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.556 DEBUG:boto:Params: {} 2023-11-02T19:03:34.557 DEBUG:boto:Token: None 2023-11-02T19:03:34.557 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/?versions 2023-11-02T19:03:34.558 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oKpCTu38mytDI4BoWl19rMorIFA= 2023-11-02T19:03:34.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oKpCTu38mytDI4BoWl19rMorIFA=', 'Content-Length': '0'} 2023-11-02T19:03:34.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb6b5da09df70c8-006543f286-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.563 DEBUG:boto:b'ibtljn-64999falseobj11nulltrue2023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:34.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T19:03:34.565 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:03:34.565 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:03:34.566 DEBUG:boto:path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.566 DEBUG:boto:auth_path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.567 DEBUG:boto:Method: GET 2023-11-02T19:03:34.567 DEBUG:boto:Path: /ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.567 DEBUG:boto:Data: 2023-11-02T19:03:34.568 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.568 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.569 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.569 DEBUG:boto:Params: {} 2023-11-02T19:03:34.570 DEBUG:boto:Token: None 2023-11-02T19:03:34.570 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.571 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:puA4aZ+DNaW0PSV+iUyBHVeduNo= 2023-11-02T19:03:34.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:puA4aZ+DNaW0PSV+iUyBHVeduNo=', 'Content-Length': '0'} 2023-11-02T19:03:34.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008968d301dc495d11-006543f286-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.575 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:03:34.576 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:03:34.576 DEBUG:boto:path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.576 DEBUG:boto:auth_path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.577 DEBUG:boto:Method: GET 2023-11-02T19:03:34.577 DEBUG:boto:Path: /ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.578 DEBUG:boto:Data: 2023-11-02T19:03:34.578 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.579 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.579 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.579 DEBUG:boto:Params: {} 2023-11-02T19:03:34.580 DEBUG:boto:Token: None 2023-11-02T19:03:34.580 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.581 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:puA4aZ+DNaW0PSV+iUyBHVeduNo= 2023-11-02T19:03:34.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:puA4aZ+DNaW0PSV+iUyBHVeduNo=', 'Content-Length': '0'} 2023-11-02T19:03:34.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8fd7be34fae5744-006543f286-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.626 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:03:34.627 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:03:34.627 DEBUG:boto:path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.628 DEBUG:boto:auth_path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.628 DEBUG:boto:Method: HEAD 2023-11-02T19:03:34.629 DEBUG:boto:Path: /ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.629 DEBUG:boto:Data: 2023-11-02T19:03:34.630 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.630 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.630 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.631 DEBUG:boto:Params: {} 2023-11-02T19:03:34.631 DEBUG:boto:Token: None 2023-11-02T19:03:34.632 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.632 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:onYSVzn964S9m8EXZ4QRZZ6QpXw= 2023-11-02T19:03:34.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:onYSVzn964S9m8EXZ4QRZZ6QpXw=', 'Content-Length': '0'} 2023-11-02T19:03:34.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae417e89e2b1f1d2-006543f286-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.636 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:03:34.637 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:03:34.637 DEBUG:boto:path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.638 DEBUG:boto:auth_path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.638 DEBUG:boto:Method: HEAD 2023-11-02T19:03:34.639 DEBUG:boto:Path: /ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.639 DEBUG:boto:Data: 2023-11-02T19:03:34.639 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.641 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.641 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.642 DEBUG:boto:Params: {} 2023-11-02T19:03:34.642 DEBUG:boto:Token: None 2023-11-02T19:03:34.643 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.643 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:onYSVzn964S9m8EXZ4QRZZ6QpXw= 2023-11-02T19:03:34.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:onYSVzn964S9m8EXZ4QRZZ6QpXw=', 'Content-Length': '0'} 2023-11-02T19:03:34.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000610f87ab80eba5e0-006543f286-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T19:03:34.648 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:03:34.648 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:03:34.649 DEBUG:boto:path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.649 DEBUG:boto:auth_path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.650 DEBUG:boto:Method: GET 2023-11-02T19:03:34.650 DEBUG:boto:Path: /ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.651 DEBUG:boto:Data: 2023-11-02T19:03:34.651 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.652 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.652 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.652 DEBUG:boto:Params: {} 2023-11-02T19:03:34.653 DEBUG:boto:Token: None 2023-11-02T19:03:34.654 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.654 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:puA4aZ+DNaW0PSV+iUyBHVeduNo= 2023-11-02T19:03:34.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:puA4aZ+DNaW0PSV+iUyBHVeduNo=', 'Content-Length': '0'} 2023-11-02T19:03:34.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bccd4b3320d5e63d-006543f286-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.698 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:03:34.699 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:03:34.699 DEBUG:boto:path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.700 DEBUG:boto:auth_path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.700 DEBUG:boto:Method: GET 2023-11-02T19:03:34.701 DEBUG:boto:Path: /ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.701 DEBUG:boto:Data: 2023-11-02T19:03:34.702 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.702 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.703 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.703 DEBUG:boto:Params: {} 2023-11-02T19:03:34.704 DEBUG:boto:Token: None 2023-11-02T19:03:34.704 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj11?versionId=null 2023-11-02T19:03:34.705 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:puA4aZ+DNaW0PSV+iUyBHVeduNo= 2023-11-02T19:03:34.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:puA4aZ+DNaW0PSV+iUyBHVeduNo=', 'Content-Length': '0'} 2023-11-02T19:03:34.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d77cf7553cc7eef-006543f286-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.709 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:34.709 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:34.710 DEBUG:boto:path=/ibtljn-64/?prefix=obj11 2023-11-02T19:03:34.710 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj11 2023-11-02T19:03:34.711 DEBUG:boto:Method: GET 2023-11-02T19:03:34.711 DEBUG:boto:Path: /ibtljn-64/?prefix=obj11 2023-11-02T19:03:34.712 DEBUG:boto:Data: 2023-11-02T19:03:34.712 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.713 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.713 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.713 DEBUG:boto:Params: {} 2023-11-02T19:03:34.714 DEBUG:boto:Token: None 2023-11-02T19:03:34.714 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/ 2023-11-02T19:03:34.715 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IbVVCsXlfGpLKQpMrnqQSluxsqM= 2023-11-02T19:03:34.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IbVVCsXlfGpLKQpMrnqQSluxsqM=', 'Content-Length': '0'} 2023-11-02T19:03:34.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099a004bdcab1020b-006543f286-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.721 DEBUG:boto:b'ibtljn-64obj111000falseobj112023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:34.721 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:34.722 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:34.722 DEBUG:boto:path=/ibtljn-64/?prefix=obj11 2023-11-02T19:03:34.723 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj11 2023-11-02T19:03:34.723 DEBUG:boto:Method: GET 2023-11-02T19:03:34.724 DEBUG:boto:Path: /ibtljn-64/?prefix=obj11 2023-11-02T19:03:34.724 DEBUG:boto:Data: 2023-11-02T19:03:34.725 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.725 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.726 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.726 DEBUG:boto:Params: {} 2023-11-02T19:03:34.726 DEBUG:boto:Token: None 2023-11-02T19:03:34.727 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/ 2023-11-02T19:03:34.728 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IbVVCsXlfGpLKQpMrnqQSluxsqM= 2023-11-02T19:03:34.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IbVVCsXlfGpLKQpMrnqQSluxsqM=', 'Content-Length': '0'} 2023-11-02T19:03:34.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012aa21b9747000ea-006543f286-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.733 DEBUG:boto:b'ibtljn-64obj111000falseobj112023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:34.733 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:03:34.734 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:03:34.734 DEBUG:boto:Method: HEAD 2023-11-02T19:03:34.735 DEBUG:boto:Path: /ibtljn-64/obj11 2023-11-02T19:03:34.735 DEBUG:boto:Data: 2023-11-02T19:03:34.735 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.736 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.736 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.737 DEBUG:boto:Params: {} 2023-11-02T19:03:34.737 DEBUG:boto:Token: None 2023-11-02T19:03:34.738 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj11 2023-11-02T19:03:34.738 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AZzkQM2Eg8uPbwVPkMzGkI+9odo= 2023-11-02T19:03:34.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AZzkQM2Eg8uPbwVPkMzGkI+9odo=', 'Content-Length': '0'} 2023-11-02T19:03:34.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c164cc987beb3cc-006543f286-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.742 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:03:34.743 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:03:34.743 DEBUG:boto:Method: HEAD 2023-11-02T19:03:34.744 DEBUG:boto:Path: /ibtljn-64/obj11 2023-11-02T19:03:34.744 DEBUG:boto:Data: 2023-11-02T19:03:34.744 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.745 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.745 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.746 DEBUG:boto:Params: {} 2023-11-02T19:03:34.746 DEBUG:boto:Token: None 2023-11-02T19:03:34.747 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj11 2023-11-02T19:03:34.747 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AZzkQM2Eg8uPbwVPkMzGkI+9odo= 2023-11-02T19:03:34.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AZzkQM2Eg8uPbwVPkMzGkI+9odo=', 'Content-Length': '0'} 2023-11-02T19:03:34.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080c50f990ff41cda-006543f286-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T19:03:34.752 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:03:34.752 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:03:34.752 DEBUG:boto:Method: GET 2023-11-02T19:03:34.753 DEBUG:boto:Path: /ibtljn-64/obj11 2023-11-02T19:03:34.753 DEBUG:boto:Data: 2023-11-02T19:03:34.754 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.755 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.755 DEBUG:boto:Params: {} 2023-11-02T19:03:34.756 DEBUG:boto:Token: None 2023-11-02T19:03:34.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj11 2023-11-02T19:03:34.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rj1jae2h/yWzRlGATvpSK4O7aA4= 2023-11-02T19:03:34.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rj1jae2h/yWzRlGATvpSK4O7aA4=', 'Content-Length': '0'} 2023-11-02T19:03:34.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f27836d81ad42d48-006543f286-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.761 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:03:34.761 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:03:34.762 DEBUG:boto:Method: GET 2023-11-02T19:03:34.762 DEBUG:boto:Path: /ibtljn-64/obj11 2023-11-02T19:03:34.763 DEBUG:boto:Data: 2023-11-02T19:03:34.763 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.764 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.764 DEBUG:boto:Params: {} 2023-11-02T19:03:34.765 DEBUG:boto:Token: None 2023-11-02T19:03:34.765 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj11 2023-11-02T19:03:34.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Rj1jae2h/yWzRlGATvpSK4O7aA4= 2023-11-02T19:03:34.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Rj1jae2h/yWzRlGATvpSK4O7aA4=', 'Content-Length': '0'} 2023-11-02T19:03:34.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7e77b14bce86ef6-006543f286-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.810 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:34.811 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:34.811 DEBUG:boto:path=/ibtljn-64/?prefix=obj11 2023-11-02T19:03:34.812 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj11 2023-11-02T19:03:34.812 DEBUG:boto:Method: GET 2023-11-02T19:03:34.813 DEBUG:boto:Path: /ibtljn-64/?prefix=obj11 2023-11-02T19:03:34.813 DEBUG:boto:Data: 2023-11-02T19:03:34.813 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.814 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.814 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.815 DEBUG:boto:Params: {} 2023-11-02T19:03:34.815 DEBUG:boto:Token: None 2023-11-02T19:03:34.816 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/ 2023-11-02T19:03:34.816 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IbVVCsXlfGpLKQpMrnqQSluxsqM= 2023-11-02T19:03:34.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IbVVCsXlfGpLKQpMrnqQSluxsqM=', 'Content-Length': '0'} 2023-11-02T19:03:34.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce89f3c24babec3-006543f286-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.822 DEBUG:boto:b'ibtljn-64obj111000falseobj112023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:34.823 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:34.823 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:34.824 DEBUG:boto:path=/ibtljn-64/?prefix=obj11 2023-11-02T19:03:34.824 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj11 2023-11-02T19:03:34.825 DEBUG:boto:Method: GET 2023-11-02T19:03:34.825 DEBUG:boto:Path: /ibtljn-64/?prefix=obj11 2023-11-02T19:03:34.826 DEBUG:boto:Data: 2023-11-02T19:03:34.826 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.827 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.827 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.828 DEBUG:boto:Params: {} 2023-11-02T19:03:34.828 DEBUG:boto:Token: None 2023-11-02T19:03:34.828 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/ 2023-11-02T19:03:34.829 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IbVVCsXlfGpLKQpMrnqQSluxsqM= 2023-11-02T19:03:34.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IbVVCsXlfGpLKQpMrnqQSluxsqM=', 'Content-Length': '0'} 2023-11-02T19:03:34.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8731b89f8610266-006543f286-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.835 DEBUG:boto:b'ibtljn-64obj111000falseobj112023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:34.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T19:03:34.836 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:03:34.836 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:03:34.837 DEBUG:boto:path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.837 DEBUG:boto:auth_path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.838 DEBUG:boto:Method: GET 2023-11-02T19:03:34.838 DEBUG:boto:Path: /ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.839 DEBUG:boto:Data: 2023-11-02T19:03:34.839 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.840 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.840 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.840 DEBUG:boto:Params: {} 2023-11-02T19:03:34.841 DEBUG:boto:Token: None 2023-11-02T19:03:34.841 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.842 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8hvY3Zplgku5N0KbS0gsrJiR2+o= 2023-11-02T19:03:34.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8hvY3Zplgku5N0KbS0gsrJiR2+o=', 'Content-Length': '0'} 2023-11-02T19:03:34.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcab98b198c0ac4b-006543f286-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.886 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:03:34.887 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:03:34.887 DEBUG:boto:path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.888 DEBUG:boto:auth_path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.888 DEBUG:boto:Method: GET 2023-11-02T19:03:34.889 DEBUG:boto:Path: /ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.889 DEBUG:boto:Data: 2023-11-02T19:03:34.889 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.890 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.890 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.891 DEBUG:boto:Params: {} 2023-11-02T19:03:34.891 DEBUG:boto:Token: None 2023-11-02T19:03:34.892 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.892 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8hvY3Zplgku5N0KbS0gsrJiR2+o= 2023-11-02T19:03:34.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8hvY3Zplgku5N0KbS0gsrJiR2+o=', 'Content-Length': '0'} 2023-11-02T19:03:34.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a6ca429fb69b15d-006543f286-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.897 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:03:34.897 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:03:34.898 DEBUG:boto:path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.898 DEBUG:boto:auth_path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.899 DEBUG:boto:Method: HEAD 2023-11-02T19:03:34.899 DEBUG:boto:Path: /ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.899 DEBUG:boto:Data: 2023-11-02T19:03:34.900 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.900 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.901 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.901 DEBUG:boto:Params: {} 2023-11-02T19:03:34.902 DEBUG:boto:Token: None 2023-11-02T19:03:34.902 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.903 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fcbt35C/C4gkv2TJlT+JxSspChU= 2023-11-02T19:03:34.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fcbt35C/C4gkv2TJlT+JxSspChU=', 'Content-Length': '0'} 2023-11-02T19:03:34.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004c2eb891edf7722-006543f286-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.907 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:03:34.907 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:03:34.907 DEBUG:boto:path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.908 DEBUG:boto:auth_path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.908 DEBUG:boto:Method: HEAD 2023-11-02T19:03:34.909 DEBUG:boto:Path: /ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.909 DEBUG:boto:Data: 2023-11-02T19:03:34.909 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.910 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.911 DEBUG:boto:Params: {} 2023-11-02T19:03:34.911 DEBUG:boto:Token: None 2023-11-02T19:03:34.912 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.912 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fcbt35C/C4gkv2TJlT+JxSspChU= 2023-11-02T19:03:34.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fcbt35C/C4gkv2TJlT+JxSspChU=', 'Content-Length': '0'} 2023-11-02T19:03:34.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e48bc95c18760528-006543f286-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T19:03:34.916 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:03:34.917 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:03:34.917 DEBUG:boto:path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.918 DEBUG:boto:auth_path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.918 DEBUG:boto:Method: GET 2023-11-02T19:03:34.919 DEBUG:boto:Path: /ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.919 DEBUG:boto:Data: 2023-11-02T19:03:34.920 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.920 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.920 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.921 DEBUG:boto:Params: {} 2023-11-02T19:03:34.921 DEBUG:boto:Token: None 2023-11-02T19:03:34.922 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8hvY3Zplgku5N0KbS0gsrJiR2+o= 2023-11-02T19:03:34.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8hvY3Zplgku5N0KbS0gsrJiR2+o=', 'Content-Length': '0'} 2023-11-02T19:03:34.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f566d4aa8e9086a-006543f286-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.927 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:03:34.928 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:03:34.928 DEBUG:boto:path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.929 DEBUG:boto:auth_path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.929 DEBUG:boto:Method: GET 2023-11-02T19:03:34.930 DEBUG:boto:Path: /ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.930 DEBUG:boto:Data: 2023-11-02T19:03:34.931 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.931 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.931 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.932 DEBUG:boto:Params: {} 2023-11-02T19:03:34.932 DEBUG:boto:Token: None 2023-11-02T19:03:34.933 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/obj22?versionId=null 2023-11-02T19:03:34.933 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8hvY3Zplgku5N0KbS0gsrJiR2+o= 2023-11-02T19:03:34.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8hvY3Zplgku5N0KbS0gsrJiR2+o=', 'Content-Length': '0'} 2023-11-02T19:03:34.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf3840d50f72cf8c-006543f286-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.978 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:34.979 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:34.979 DEBUG:boto:path=/ibtljn-64/?prefix=obj22 2023-11-02T19:03:34.980 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj22 2023-11-02T19:03:34.980 DEBUG:boto:Method: GET 2023-11-02T19:03:34.981 DEBUG:boto:Path: /ibtljn-64/?prefix=obj22 2023-11-02T19:03:34.981 DEBUG:boto:Data: 2023-11-02T19:03:34.982 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.982 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:34.983 DEBUG:boto:Port: 8001 2023-11-02T19:03:34.983 DEBUG:boto:Params: {} 2023-11-02T19:03:34.984 DEBUG:boto:Token: None 2023-11-02T19:03:34.984 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/ 2023-11-02T19:03:34.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IbVVCsXlfGpLKQpMrnqQSluxsqM= 2023-11-02T19:03:34.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IbVVCsXlfGpLKQpMrnqQSluxsqM=', 'Content-Length': '0'} 2023-11-02T19:03:34.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064bf0c62e291ab8d-006543f286-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:34.990 DEBUG:boto:b'ibtljn-64obj221000falseobj222023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:34.991 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:34.992 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:34.992 DEBUG:boto:path=/ibtljn-64/?prefix=obj22 2023-11-02T19:03:34.992 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj22 2023-11-02T19:03:34.993 DEBUG:boto:Method: GET 2023-11-02T19:03:34.993 DEBUG:boto:Path: /ibtljn-64/?prefix=obj22 2023-11-02T19:03:34.994 DEBUG:boto:Data: 2023-11-02T19:03:34.994 DEBUG:boto:Headers: {} 2023-11-02T19:03:34.995 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:34.995 DEBUG:boto:Port: 8000 2023-11-02T19:03:34.995 DEBUG:boto:Params: {} 2023-11-02T19:03:34.996 DEBUG:boto:Token: None 2023-11-02T19:03:34.996 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:34 GMT /ibtljn-64/ 2023-11-02T19:03:34.997 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IbVVCsXlfGpLKQpMrnqQSluxsqM= 2023-11-02T19:03:34.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:34 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IbVVCsXlfGpLKQpMrnqQSluxsqM=', 'Content-Length': '0'} 2023-11-02T19:03:35.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094c0d5842d02e9ed-006543f286-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:34 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.003 DEBUG:boto:b'ibtljn-64obj221000falseobj222023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.003 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:03:35.004 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:03:35.004 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.005 DEBUG:boto:Path: /ibtljn-64/obj22 2023-11-02T19:03:35.005 DEBUG:boto:Data: 2023-11-02T19:03:35.006 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.006 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.006 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.007 DEBUG:boto:Params: {} 2023-11-02T19:03:35.007 DEBUG:boto:Token: None 2023-11-02T19:03:35.008 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj22 2023-11-02T19:03:35.008 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VCCAIgelIkebiT10qtX0b9+TMhE= 2023-11-02T19:03:35.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VCCAIgelIkebiT10qtX0b9+TMhE=', 'Content-Length': '0'} 2023-11-02T19:03:35.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e748abed0b028b0-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.012 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:03:35.013 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:03:35.013 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.014 DEBUG:boto:Path: /ibtljn-64/obj22 2023-11-02T19:03:35.014 DEBUG:boto:Data: 2023-11-02T19:03:35.014 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.015 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.015 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.016 DEBUG:boto:Params: {} 2023-11-02T19:03:35.016 DEBUG:boto:Token: None 2023-11-02T19:03:35.016 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj22 2023-11-02T19:03:35.017 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VCCAIgelIkebiT10qtX0b9+TMhE= 2023-11-02T19:03:35.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VCCAIgelIkebiT10qtX0b9+TMhE=', 'Content-Length': '0'} 2023-11-02T19:03:35.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e55cd9084fe991b6-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T19:03:35.021 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:03:35.022 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:03:35.022 DEBUG:boto:Method: GET 2023-11-02T19:03:35.023 DEBUG:boto:Path: /ibtljn-64/obj22 2023-11-02T19:03:35.023 DEBUG:boto:Data: 2023-11-02T19:03:35.024 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.024 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.024 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.025 DEBUG:boto:Params: {} 2023-11-02T19:03:35.025 DEBUG:boto:Token: None 2023-11-02T19:03:35.026 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj22 2023-11-02T19:03:35.026 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xOBWuaNgdLA69cnXTR1Z/B0TBRs= 2023-11-02T19:03:35.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xOBWuaNgdLA69cnXTR1Z/B0TBRs=', 'Content-Length': '0'} 2023-11-02T19:03:35.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4806241df67fa02-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.070 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:03:35.071 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:03:35.071 DEBUG:boto:Method: GET 2023-11-02T19:03:35.072 DEBUG:boto:Path: /ibtljn-64/obj22 2023-11-02T19:03:35.072 DEBUG:boto:Data: 2023-11-02T19:03:35.073 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.073 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.073 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.074 DEBUG:boto:Params: {} 2023-11-02T19:03:35.074 DEBUG:boto:Token: None 2023-11-02T19:03:35.075 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj22 2023-11-02T19:03:35.075 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xOBWuaNgdLA69cnXTR1Z/B0TBRs= 2023-11-02T19:03:35.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xOBWuaNgdLA69cnXTR1Z/B0TBRs=', 'Content-Length': '0'} 2023-11-02T19:03:35.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006336238661d584bd-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.080 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.080 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.080 DEBUG:boto:path=/ibtljn-64/?prefix=obj22 2023-11-02T19:03:35.081 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj22 2023-11-02T19:03:35.081 DEBUG:boto:Method: GET 2023-11-02T19:03:35.082 DEBUG:boto:Path: /ibtljn-64/?prefix=obj22 2023-11-02T19:03:35.082 DEBUG:boto:Data: 2023-11-02T19:03:35.082 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.083 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.083 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.084 DEBUG:boto:Params: {} 2023-11-02T19:03:35.084 DEBUG:boto:Token: None 2023-11-02T19:03:35.085 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.085 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7be504380198e44-006543f287-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.091 DEBUG:boto:b'ibtljn-64obj221000falseobj222023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.092 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.092 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.092 DEBUG:boto:path=/ibtljn-64/?prefix=obj22 2023-11-02T19:03:35.093 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj22 2023-11-02T19:03:35.093 DEBUG:boto:Method: GET 2023-11-02T19:03:35.094 DEBUG:boto:Path: /ibtljn-64/?prefix=obj22 2023-11-02T19:03:35.094 DEBUG:boto:Data: 2023-11-02T19:03:35.095 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.095 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.095 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.096 DEBUG:boto:Params: {} 2023-11-02T19:03:35.096 DEBUG:boto:Token: None 2023-11-02T19:03:35.097 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.097 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e5ce72eb127ef52-006543f287-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.150 DEBUG:boto:b'ibtljn-64obj221000falseobj222023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T19:03:35.152 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:03:35.152 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:03:35.152 DEBUG:boto:path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.153 DEBUG:boto:auth_path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.153 DEBUG:boto:Method: GET 2023-11-02T19:03:35.154 DEBUG:boto:Path: /ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.154 DEBUG:boto:Data: 2023-11-02T19:03:35.155 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.155 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.156 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.156 DEBUG:boto:Params: {} 2023-11-02T19:03:35.156 DEBUG:boto:Token: None 2023-11-02T19:03:35.157 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.157 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:obJM/zplO+09jkzIxijhbVfayck= 2023-11-02T19:03:35.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:obJM/zplO+09jkzIxijhbVfayck=', 'Content-Length': '0'} 2023-11-02T19:03:35.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014b9f2e3a50af72b-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.162 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:03:35.162 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:03:35.163 DEBUG:boto:path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.163 DEBUG:boto:auth_path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.164 DEBUG:boto:Method: GET 2023-11-02T19:03:35.164 DEBUG:boto:Path: /ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.164 DEBUG:boto:Data: 2023-11-02T19:03:35.165 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.165 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.166 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.166 DEBUG:boto:Params: {} 2023-11-02T19:03:35.167 DEBUG:boto:Token: None 2023-11-02T19:03:35.167 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.168 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:obJM/zplO+09jkzIxijhbVfayck= 2023-11-02T19:03:35.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:obJM/zplO+09jkzIxijhbVfayck=', 'Content-Length': '0'} 2023-11-02T19:03:35.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038789548873ffdad-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.172 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:03:35.172 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:03:35.173 DEBUG:boto:path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.173 DEBUG:boto:auth_path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.173 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.174 DEBUG:boto:Path: /ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.174 DEBUG:boto:Data: 2023-11-02T19:03:35.175 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.175 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.176 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.176 DEBUG:boto:Params: {} 2023-11-02T19:03:35.176 DEBUG:boto:Token: None 2023-11-02T19:03:35.177 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.177 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n9PE8iB0jGKWVuoTaoRMNv7lMjc= 2023-11-02T19:03:35.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n9PE8iB0jGKWVuoTaoRMNv7lMjc=', 'Content-Length': '0'} 2023-11-02T19:03:35.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f1a680820dfe484-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.181 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:03:35.182 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:03:35.182 DEBUG:boto:path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.183 DEBUG:boto:auth_path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.183 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.184 DEBUG:boto:Path: /ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.184 DEBUG:boto:Data: 2023-11-02T19:03:35.184 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.185 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.185 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.186 DEBUG:boto:Params: {} 2023-11-02T19:03:35.186 DEBUG:boto:Token: None 2023-11-02T19:03:35.187 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.187 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n9PE8iB0jGKWVuoTaoRMNv7lMjc= 2023-11-02T19:03:35.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n9PE8iB0jGKWVuoTaoRMNv7lMjc=', 'Content-Length': '0'} 2023-11-02T19:03:35.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e0e900943eaf023-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T19:03:35.192 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:03:35.192 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:03:35.192 DEBUG:boto:path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.193 DEBUG:boto:auth_path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.193 DEBUG:boto:Method: GET 2023-11-02T19:03:35.194 DEBUG:boto:Path: /ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.194 DEBUG:boto:Data: 2023-11-02T19:03:35.195 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.195 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.195 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.196 DEBUG:boto:Params: {} 2023-11-02T19:03:35.196 DEBUG:boto:Token: None 2023-11-02T19:03:35.197 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.197 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:obJM/zplO+09jkzIxijhbVfayck= 2023-11-02T19:03:35.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:obJM/zplO+09jkzIxijhbVfayck=', 'Content-Length': '0'} 2023-11-02T19:03:35.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42661e4b733412c-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.201 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:03:35.202 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:03:35.202 DEBUG:boto:path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.203 DEBUG:boto:auth_path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.203 DEBUG:boto:Method: GET 2023-11-02T19:03:35.204 DEBUG:boto:Path: /ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.204 DEBUG:boto:Data: 2023-11-02T19:03:35.205 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.205 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.205 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.206 DEBUG:boto:Params: {} 2023-11-02T19:03:35.207 DEBUG:boto:Token: None 2023-11-02T19:03:35.207 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj33?versionId=null 2023-11-02T19:03:35.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:obJM/zplO+09jkzIxijhbVfayck= 2023-11-02T19:03:35.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:obJM/zplO+09jkzIxijhbVfayck=', 'Content-Length': '0'} 2023-11-02T19:03:35.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dd8db98b90f35e7-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.254 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.255 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.255 DEBUG:boto:path=/ibtljn-64/?prefix=obj33 2023-11-02T19:03:35.257 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj33 2023-11-02T19:03:35.257 DEBUG:boto:Method: GET 2023-11-02T19:03:35.258 DEBUG:boto:Path: /ibtljn-64/?prefix=obj33 2023-11-02T19:03:35.258 DEBUG:boto:Data: 2023-11-02T19:03:35.259 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.260 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.260 DEBUG:boto:Params: {} 2023-11-02T19:03:35.262 DEBUG:boto:Token: None 2023-11-02T19:03:35.262 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.263 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035baa11588614b1f-006543f287-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.269 DEBUG:boto:b'ibtljn-64obj331000falseobj332023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.269 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.270 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.270 DEBUG:boto:path=/ibtljn-64/?prefix=obj33 2023-11-02T19:03:35.271 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj33 2023-11-02T19:03:35.271 DEBUG:boto:Method: GET 2023-11-02T19:03:35.271 DEBUG:boto:Path: /ibtljn-64/?prefix=obj33 2023-11-02T19:03:35.272 DEBUG:boto:Data: 2023-11-02T19:03:35.272 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.273 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.273 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.273 DEBUG:boto:Params: {} 2023-11-02T19:03:35.274 DEBUG:boto:Token: None 2023-11-02T19:03:35.274 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.275 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073f2555d05bd030e-006543f287-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.280 DEBUG:boto:b'ibtljn-64obj331000falseobj332023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.281 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:03:35.282 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:03:35.282 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.282 DEBUG:boto:Path: /ibtljn-64/obj33 2023-11-02T19:03:35.283 DEBUG:boto:Data: 2023-11-02T19:03:35.283 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.284 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.284 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.285 DEBUG:boto:Params: {} 2023-11-02T19:03:35.285 DEBUG:boto:Token: None 2023-11-02T19:03:35.285 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj33 2023-11-02T19:03:35.286 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cNWyy4Komp9CQTXV1WZbYk/oJzs= 2023-11-02T19:03:35.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cNWyy4Komp9CQTXV1WZbYk/oJzs=', 'Content-Length': '0'} 2023-11-02T19:03:35.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033f48db22933c40a-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.290 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:03:35.290 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:03:35.291 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.291 DEBUG:boto:Path: /ibtljn-64/obj33 2023-11-02T19:03:35.292 DEBUG:boto:Data: 2023-11-02T19:03:35.292 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.292 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.293 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.293 DEBUG:boto:Params: {} 2023-11-02T19:03:35.294 DEBUG:boto:Token: None 2023-11-02T19:03:35.294 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj33 2023-11-02T19:03:35.295 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cNWyy4Komp9CQTXV1WZbYk/oJzs= 2023-11-02T19:03:35.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cNWyy4Komp9CQTXV1WZbYk/oJzs=', 'Content-Length': '0'} 2023-11-02T19:03:35.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c3119e266711399-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T19:03:35.299 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:03:35.300 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:03:35.300 DEBUG:boto:Method: GET 2023-11-02T19:03:35.300 DEBUG:boto:Path: /ibtljn-64/obj33 2023-11-02T19:03:35.301 DEBUG:boto:Data: 2023-11-02T19:03:35.301 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.302 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.302 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.303 DEBUG:boto:Params: {} 2023-11-02T19:03:35.303 DEBUG:boto:Token: None 2023-11-02T19:03:35.303 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj33 2023-11-02T19:03:35.304 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XlyIMKulLQkv0W3z/yDuf8pyQgA= 2023-11-02T19:03:35.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XlyIMKulLQkv0W3z/yDuf8pyQgA=', 'Content-Length': '0'} 2023-11-02T19:03:35.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002178b7117293438a-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.308 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:03:35.309 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:03:35.309 DEBUG:boto:Method: GET 2023-11-02T19:03:35.309 DEBUG:boto:Path: /ibtljn-64/obj33 2023-11-02T19:03:35.310 DEBUG:boto:Data: 2023-11-02T19:03:35.310 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.311 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.311 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.312 DEBUG:boto:Params: {} 2023-11-02T19:03:35.312 DEBUG:boto:Token: None 2023-11-02T19:03:35.313 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj33 2023-11-02T19:03:35.313 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XlyIMKulLQkv0W3z/yDuf8pyQgA= 2023-11-02T19:03:35.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XlyIMKulLQkv0W3z/yDuf8pyQgA=', 'Content-Length': '0'} 2023-11-02T19:03:35.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000745a5b3eded15ccd-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.317 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.318 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.318 DEBUG:boto:path=/ibtljn-64/?prefix=obj33 2023-11-02T19:03:35.318 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj33 2023-11-02T19:03:35.319 DEBUG:boto:Method: GET 2023-11-02T19:03:35.319 DEBUG:boto:Path: /ibtljn-64/?prefix=obj33 2023-11-02T19:03:35.320 DEBUG:boto:Data: 2023-11-02T19:03:35.320 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.321 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.321 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.321 DEBUG:boto:Params: {} 2023-11-02T19:03:35.322 DEBUG:boto:Token: None 2023-11-02T19:03:35.322 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.323 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eea59f73bc7b758-006543f287-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.370 DEBUG:boto:b'ibtljn-64obj331000falseobj332023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.371 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.372 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.372 DEBUG:boto:path=/ibtljn-64/?prefix=obj33 2023-11-02T19:03:35.373 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj33 2023-11-02T19:03:35.373 DEBUG:boto:Method: GET 2023-11-02T19:03:35.374 DEBUG:boto:Path: /ibtljn-64/?prefix=obj33 2023-11-02T19:03:35.374 DEBUG:boto:Data: 2023-11-02T19:03:35.374 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.375 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.375 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.376 DEBUG:boto:Params: {} 2023-11-02T19:03:35.376 DEBUG:boto:Token: None 2023-11-02T19:03:35.377 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.377 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057f7d0003d13c279-006543f287-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.383 DEBUG:boto:b'ibtljn-64obj331000falseobj332023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T19:03:35.384 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:03:35.385 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:03:35.385 DEBUG:boto:path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.386 DEBUG:boto:auth_path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.386 DEBUG:boto:Method: GET 2023-11-02T19:03:35.387 DEBUG:boto:Path: /ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.387 DEBUG:boto:Data: 2023-11-02T19:03:35.387 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.388 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.388 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.389 DEBUG:boto:Params: {} 2023-11-02T19:03:35.389 DEBUG:boto:Token: None 2023-11-02T19:03:35.390 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.390 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2lql+X7ybXWi08d/CjXlk+RlO8Y= 2023-11-02T19:03:35.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2lql+X7ybXWi08d/CjXlk+RlO8Y=', 'Content-Length': '0'} 2023-11-02T19:03:35.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c99ce803922341a-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.394 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:03:35.395 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:03:35.395 DEBUG:boto:path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.396 DEBUG:boto:auth_path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.396 DEBUG:boto:Method: GET 2023-11-02T19:03:35.397 DEBUG:boto:Path: /ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.397 DEBUG:boto:Data: 2023-11-02T19:03:35.397 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.398 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.399 DEBUG:boto:Params: {} 2023-11-02T19:03:35.399 DEBUG:boto:Token: None 2023-11-02T19:03:35.400 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2lql+X7ybXWi08d/CjXlk+RlO8Y= 2023-11-02T19:03:35.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2lql+X7ybXWi08d/CjXlk+RlO8Y=', 'Content-Length': '0'} 2023-11-02T19:03:35.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5a962c5108519a7-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.450 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:03:35.451 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:03:35.451 DEBUG:boto:path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.452 DEBUG:boto:auth_path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.452 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.453 DEBUG:boto:Path: /ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.453 DEBUG:boto:Data: 2023-11-02T19:03:35.454 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.455 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.455 DEBUG:boto:Params: {} 2023-11-02T19:03:35.456 DEBUG:boto:Token: None 2023-11-02T19:03:35.456 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.457 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3UxMtdPtnM6XmhZTPjvw03jpty8= 2023-11-02T19:03:35.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3UxMtdPtnM6XmhZTPjvw03jpty8=', 'Content-Length': '0'} 2023-11-02T19:03:35.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089aaf4bf46aad4f6-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.461 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:03:35.461 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:03:35.462 DEBUG:boto:path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.462 DEBUG:boto:auth_path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.463 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.463 DEBUG:boto:Path: /ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.463 DEBUG:boto:Data: 2023-11-02T19:03:35.466 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.466 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.467 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.467 DEBUG:boto:Params: {} 2023-11-02T19:03:35.468 DEBUG:boto:Token: None 2023-11-02T19:03:35.468 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.469 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3UxMtdPtnM6XmhZTPjvw03jpty8= 2023-11-02T19:03:35.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3UxMtdPtnM6XmhZTPjvw03jpty8=', 'Content-Length': '0'} 2023-11-02T19:03:35.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037e1f7994905db7d-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T19:03:35.474 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:03:35.474 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:03:35.475 DEBUG:boto:path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.475 DEBUG:boto:auth_path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.475 DEBUG:boto:Method: GET 2023-11-02T19:03:35.476 DEBUG:boto:Path: /ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.476 DEBUG:boto:Data: 2023-11-02T19:03:35.477 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.477 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.478 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.478 DEBUG:boto:Params: {} 2023-11-02T19:03:35.479 DEBUG:boto:Token: None 2023-11-02T19:03:35.479 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.480 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2lql+X7ybXWi08d/CjXlk+RlO8Y= 2023-11-02T19:03:35.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2lql+X7ybXWi08d/CjXlk+RlO8Y=', 'Content-Length': '0'} 2023-11-02T19:03:35.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf0ec279c7cb766e-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.484 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:03:35.485 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:03:35.485 DEBUG:boto:path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.485 DEBUG:boto:auth_path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.486 DEBUG:boto:Method: GET 2023-11-02T19:03:35.486 DEBUG:boto:Path: /ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.487 DEBUG:boto:Data: 2023-11-02T19:03:35.487 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.488 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.488 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.489 DEBUG:boto:Params: {} 2023-11-02T19:03:35.489 DEBUG:boto:Token: None 2023-11-02T19:03:35.489 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj44?versionId=null 2023-11-02T19:03:35.490 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2lql+X7ybXWi08d/CjXlk+RlO8Y= 2023-11-02T19:03:35.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2lql+X7ybXWi08d/CjXlk+RlO8Y=', 'Content-Length': '0'} 2023-11-02T19:03:35.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d82ece3671340f5-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.494 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.495 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.495 DEBUG:boto:path=/ibtljn-64/?prefix=obj44 2023-11-02T19:03:35.496 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj44 2023-11-02T19:03:35.496 DEBUG:boto:Method: GET 2023-11-02T19:03:35.497 DEBUG:boto:Path: /ibtljn-64/?prefix=obj44 2023-11-02T19:03:35.497 DEBUG:boto:Data: 2023-11-02T19:03:35.498 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.498 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.499 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.499 DEBUG:boto:Params: {} 2023-11-02T19:03:35.500 DEBUG:boto:Token: None 2023-11-02T19:03:35.500 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.500 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bedf67cb7eb3f44c-006543f287-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.550 DEBUG:boto:b'ibtljn-64obj441000falseobj442023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.551 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.552 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.552 DEBUG:boto:path=/ibtljn-64/?prefix=obj44 2023-11-02T19:03:35.553 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj44 2023-11-02T19:03:35.553 DEBUG:boto:Method: GET 2023-11-02T19:03:35.554 DEBUG:boto:Path: /ibtljn-64/?prefix=obj44 2023-11-02T19:03:35.554 DEBUG:boto:Data: 2023-11-02T19:03:35.554 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.555 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.555 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.556 DEBUG:boto:Params: {} 2023-11-02T19:03:35.556 DEBUG:boto:Token: None 2023-11-02T19:03:35.557 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.557 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc25b6872f50e7d9-006543f287-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.564 DEBUG:boto:b'ibtljn-64obj441000falseobj442023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.565 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:03:35.565 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:03:35.565 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.566 DEBUG:boto:Path: /ibtljn-64/obj44 2023-11-02T19:03:35.566 DEBUG:boto:Data: 2023-11-02T19:03:35.567 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.567 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.568 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.568 DEBUG:boto:Params: {} 2023-11-02T19:03:35.568 DEBUG:boto:Token: None 2023-11-02T19:03:35.569 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj44 2023-11-02T19:03:35.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v9n2vUPiNyAuV8436ih91CkSUxo= 2023-11-02T19:03:35.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v9n2vUPiNyAuV8436ih91CkSUxo=', 'Content-Length': '0'} 2023-11-02T19:03:35.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cab36a17672a11b3-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.573 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:03:35.574 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:03:35.574 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.575 DEBUG:boto:Path: /ibtljn-64/obj44 2023-11-02T19:03:35.575 DEBUG:boto:Data: 2023-11-02T19:03:35.575 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.576 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.576 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.577 DEBUG:boto:Params: {} 2023-11-02T19:03:35.577 DEBUG:boto:Token: None 2023-11-02T19:03:35.578 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj44 2023-11-02T19:03:35.578 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v9n2vUPiNyAuV8436ih91CkSUxo= 2023-11-02T19:03:35.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v9n2vUPiNyAuV8436ih91CkSUxo=', 'Content-Length': '0'} 2023-11-02T19:03:35.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef47ecc278f5bf4c-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T19:03:35.583 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:03:35.583 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:03:35.583 DEBUG:boto:Method: GET 2023-11-02T19:03:35.584 DEBUG:boto:Path: /ibtljn-64/obj44 2023-11-02T19:03:35.584 DEBUG:boto:Data: 2023-11-02T19:03:35.585 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.585 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.586 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.586 DEBUG:boto:Params: {} 2023-11-02T19:03:35.587 DEBUG:boto:Token: None 2023-11-02T19:03:35.587 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj44 2023-11-02T19:03:35.588 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5jZzeW0jMseVgtIN5IH52+inqS8= 2023-11-02T19:03:35.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5jZzeW0jMseVgtIN5IH52+inqS8=', 'Content-Length': '0'} 2023-11-02T19:03:35.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c19245264dadc606-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.593 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:03:35.593 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:03:35.593 DEBUG:boto:Method: GET 2023-11-02T19:03:35.594 DEBUG:boto:Path: /ibtljn-64/obj44 2023-11-02T19:03:35.594 DEBUG:boto:Data: 2023-11-02T19:03:35.595 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.595 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.596 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.596 DEBUG:boto:Params: {} 2023-11-02T19:03:35.596 DEBUG:boto:Token: None 2023-11-02T19:03:35.597 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj44 2023-11-02T19:03:35.597 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5jZzeW0jMseVgtIN5IH52+inqS8= 2023-11-02T19:03:35.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5jZzeW0jMseVgtIN5IH52+inqS8=', 'Content-Length': '0'} 2023-11-02T19:03:35.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d2a1129b4f2362d-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.642 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.643 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.643 DEBUG:boto:path=/ibtljn-64/?prefix=obj44 2023-11-02T19:03:35.644 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj44 2023-11-02T19:03:35.644 DEBUG:boto:Method: GET 2023-11-02T19:03:35.645 DEBUG:boto:Path: /ibtljn-64/?prefix=obj44 2023-11-02T19:03:35.645 DEBUG:boto:Data: 2023-11-02T19:03:35.646 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.646 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.647 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.647 DEBUG:boto:Params: {} 2023-11-02T19:03:35.648 DEBUG:boto:Token: None 2023-11-02T19:03:35.648 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.649 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000477f361bc3770661-006543f287-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.659 DEBUG:boto:b'ibtljn-64obj441000falseobj442023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.660 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.661 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.661 DEBUG:boto:path=/ibtljn-64/?prefix=obj44 2023-11-02T19:03:35.662 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj44 2023-11-02T19:03:35.662 DEBUG:boto:Method: GET 2023-11-02T19:03:35.663 DEBUG:boto:Path: /ibtljn-64/?prefix=obj44 2023-11-02T19:03:35.663 DEBUG:boto:Data: 2023-11-02T19:03:35.664 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.664 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.664 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.665 DEBUG:boto:Params: {} 2023-11-02T19:03:35.666 DEBUG:boto:Token: None 2023-11-02T19:03:35.669 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.670 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f1287a296c4100d-006543f287-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.675 DEBUG:boto:b'ibtljn-64obj441000falseobj442023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T19:03:35.676 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:03:35.676 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:03:35.677 DEBUG:boto:path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.677 DEBUG:boto:auth_path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.678 DEBUG:boto:Method: GET 2023-11-02T19:03:35.678 DEBUG:boto:Path: /ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.679 DEBUG:boto:Data: 2023-11-02T19:03:35.679 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.680 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.681 DEBUG:boto:Params: {} 2023-11-02T19:03:35.681 DEBUG:boto:Token: None 2023-11-02T19:03:35.682 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.682 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKJWspnJA6M3K9S4XJYrf7UjcZk= 2023-11-02T19:03:35.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKJWspnJA6M3K9S4XJYrf7UjcZk=', 'Content-Length': '0'} 2023-11-02T19:03:35.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e610aab1082da849-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.686 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:03:35.687 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:03:35.687 DEBUG:boto:path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.688 DEBUG:boto:auth_path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.688 DEBUG:boto:Method: GET 2023-11-02T19:03:35.688 DEBUG:boto:Path: /ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.689 DEBUG:boto:Data: 2023-11-02T19:03:35.689 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.690 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.690 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.691 DEBUG:boto:Params: {} 2023-11-02T19:03:35.691 DEBUG:boto:Token: None 2023-11-02T19:03:35.692 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.692 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKJWspnJA6M3K9S4XJYrf7UjcZk= 2023-11-02T19:03:35.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKJWspnJA6M3K9S4XJYrf7UjcZk=', 'Content-Length': '0'} 2023-11-02T19:03:35.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b0e895f227f9f73-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.697 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:03:35.697 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:03:35.698 DEBUG:boto:path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.698 DEBUG:boto:auth_path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.698 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.699 DEBUG:boto:Path: /ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.699 DEBUG:boto:Data: 2023-11-02T19:03:35.700 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.701 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.701 DEBUG:boto:Params: {} 2023-11-02T19:03:35.702 DEBUG:boto:Token: None 2023-11-02T19:03:35.702 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.703 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AYGZXgJQEypEzb60gU9oxOV9YQc= 2023-11-02T19:03:35.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AYGZXgJQEypEzb60gU9oxOV9YQc=', 'Content-Length': '0'} 2023-11-02T19:03:35.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004498d1c6878683d6-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.706 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:03:35.706 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:03:35.707 DEBUG:boto:path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.707 DEBUG:boto:auth_path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.708 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.708 DEBUG:boto:Path: /ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.708 DEBUG:boto:Data: 2023-11-02T19:03:35.709 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.709 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.710 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.710 DEBUG:boto:Params: {} 2023-11-02T19:03:35.711 DEBUG:boto:Token: None 2023-11-02T19:03:35.711 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.712 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AYGZXgJQEypEzb60gU9oxOV9YQc= 2023-11-02T19:03:35.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AYGZXgJQEypEzb60gU9oxOV9YQc=', 'Content-Length': '0'} 2023-11-02T19:03:35.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6d432a0cf417c34-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T19:03:35.715 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:03:35.716 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:03:35.716 DEBUG:boto:path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.717 DEBUG:boto:auth_path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.717 DEBUG:boto:Method: GET 2023-11-02T19:03:35.718 DEBUG:boto:Path: /ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.718 DEBUG:boto:Data: 2023-11-02T19:03:35.719 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.719 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.719 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.720 DEBUG:boto:Params: {} 2023-11-02T19:03:35.720 DEBUG:boto:Token: None 2023-11-02T19:03:35.721 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKJWspnJA6M3K9S4XJYrf7UjcZk= 2023-11-02T19:03:35.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKJWspnJA6M3K9S4XJYrf7UjcZk=', 'Content-Length': '0'} 2023-11-02T19:03:35.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c41e78c648a32bf-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.725 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:03:35.726 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:03:35.726 DEBUG:boto:path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.728 DEBUG:boto:auth_path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.728 DEBUG:boto:Method: GET 2023-11-02T19:03:35.729 DEBUG:boto:Path: /ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.729 DEBUG:boto:Data: 2023-11-02T19:03:35.730 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.730 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.731 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.731 DEBUG:boto:Params: {} 2023-11-02T19:03:35.732 DEBUG:boto:Token: None 2023-11-02T19:03:35.732 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj55?versionId=null 2023-11-02T19:03:35.733 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKJWspnJA6M3K9S4XJYrf7UjcZk= 2023-11-02T19:03:35.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKJWspnJA6M3K9S4XJYrf7UjcZk=', 'Content-Length': '0'} 2023-11-02T19:03:35.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2b9865319611819-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.779 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.779 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.780 DEBUG:boto:path=/ibtljn-64/?prefix=obj55 2023-11-02T19:03:35.780 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj55 2023-11-02T19:03:35.781 DEBUG:boto:Method: GET 2023-11-02T19:03:35.781 DEBUG:boto:Path: /ibtljn-64/?prefix=obj55 2023-11-02T19:03:35.782 DEBUG:boto:Data: 2023-11-02T19:03:35.782 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.782 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.783 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.783 DEBUG:boto:Params: {} 2023-11-02T19:03:35.784 DEBUG:boto:Token: None 2023-11-02T19:03:35.784 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078ac71751fe9b07d-006543f287-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.792 DEBUG:boto:b'ibtljn-64obj551000falseobj552023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.793 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.794 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.794 DEBUG:boto:path=/ibtljn-64/?prefix=obj55 2023-11-02T19:03:35.795 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj55 2023-11-02T19:03:35.795 DEBUG:boto:Method: GET 2023-11-02T19:03:35.796 DEBUG:boto:Path: /ibtljn-64/?prefix=obj55 2023-11-02T19:03:35.796 DEBUG:boto:Data: 2023-11-02T19:03:35.797 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.797 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.798 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.798 DEBUG:boto:Params: {} 2023-11-02T19:03:35.799 DEBUG:boto:Token: None 2023-11-02T19:03:35.799 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.800 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2aec3c894b7fc85-006543f287-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.806 DEBUG:boto:b'ibtljn-64obj551000falseobj552023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.806 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:03:35.807 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:03:35.807 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.808 DEBUG:boto:Path: /ibtljn-64/obj55 2023-11-02T19:03:35.808 DEBUG:boto:Data: 2023-11-02T19:03:35.808 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.809 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.809 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.810 DEBUG:boto:Params: {} 2023-11-02T19:03:35.810 DEBUG:boto:Token: None 2023-11-02T19:03:35.811 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj55 2023-11-02T19:03:35.811 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sjSSfYFbXXCF/WG3b/pF8096cPc= 2023-11-02T19:03:35.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sjSSfYFbXXCF/WG3b/pF8096cPc=', 'Content-Length': '0'} 2023-11-02T19:03:35.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000854108d8cc55ac4e-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.815 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:03:35.816 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:03:35.816 DEBUG:boto:Method: HEAD 2023-11-02T19:03:35.817 DEBUG:boto:Path: /ibtljn-64/obj55 2023-11-02T19:03:35.817 DEBUG:boto:Data: 2023-11-02T19:03:35.818 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.818 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.818 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.819 DEBUG:boto:Params: {} 2023-11-02T19:03:35.819 DEBUG:boto:Token: None 2023-11-02T19:03:35.820 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj55 2023-11-02T19:03:35.820 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sjSSfYFbXXCF/WG3b/pF8096cPc= 2023-11-02T19:03:35.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sjSSfYFbXXCF/WG3b/pF8096cPc=', 'Content-Length': '0'} 2023-11-02T19:03:35.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edb4456375a1bf38-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T19:03:35.825 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:03:35.825 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:03:35.826 DEBUG:boto:Method: GET 2023-11-02T19:03:35.826 DEBUG:boto:Path: /ibtljn-64/obj55 2023-11-02T19:03:35.827 DEBUG:boto:Data: 2023-11-02T19:03:35.827 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.828 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.828 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.829 DEBUG:boto:Params: {} 2023-11-02T19:03:35.829 DEBUG:boto:Token: None 2023-11-02T19:03:35.830 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj55 2023-11-02T19:03:35.830 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bh2QdJdPy66r8OCGLC/iMlla/pU= 2023-11-02T19:03:35.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bh2QdJdPy66r8OCGLC/iMlla/pU=', 'Content-Length': '0'} 2023-11-02T19:03:35.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f6774bc64a692a0-006543f287-1fca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.874 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:03:35.875 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:03:35.875 DEBUG:boto:Method: GET 2023-11-02T19:03:35.876 DEBUG:boto:Path: /ibtljn-64/obj55 2023-11-02T19:03:35.876 DEBUG:boto:Data: 2023-11-02T19:03:35.876 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.877 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.877 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.878 DEBUG:boto:Params: {} 2023-11-02T19:03:35.878 DEBUG:boto:Token: None 2023-11-02T19:03:35.879 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/obj55 2023-11-02T19:03:35.879 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bh2QdJdPy66r8OCGLC/iMlla/pU= 2023-11-02T19:03:35.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bh2QdJdPy66r8OCGLC/iMlla/pU=', 'Content-Length': '0'} 2023-11-02T19:03:35.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8cd573e5f02ede7-006543f287-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.884 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.884 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.885 DEBUG:boto:path=/ibtljn-64/?prefix=obj55 2023-11-02T19:03:35.885 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj55 2023-11-02T19:03:35.887 DEBUG:boto:Method: GET 2023-11-02T19:03:35.887 DEBUG:boto:Path: /ibtljn-64/?prefix=obj55 2023-11-02T19:03:35.888 DEBUG:boto:Data: 2023-11-02T19:03:35.888 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.889 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:35.889 DEBUG:boto:Port: 8001 2023-11-02T19:03:35.890 DEBUG:boto:Params: {} 2023-11-02T19:03:35.890 DEBUG:boto:Token: None 2023-11-02T19:03:35.891 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.891 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f6003e513bfa09d-006543f287-1fca-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.897 DEBUG:boto:b'ibtljn-64obj551000falseobj552023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.898 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:35.899 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:35.899 DEBUG:boto:path=/ibtljn-64/?prefix=obj55 2023-11-02T19:03:35.900 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj55 2023-11-02T19:03:35.900 DEBUG:boto:Method: GET 2023-11-02T19:03:35.900 DEBUG:boto:Path: /ibtljn-64/?prefix=obj55 2023-11-02T19:03:35.901 DEBUG:boto:Data: 2023-11-02T19:03:35.901 DEBUG:boto:Headers: {} 2023-11-02T19:03:35.902 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:35.902 DEBUG:boto:Port: 8000 2023-11-02T19:03:35.903 DEBUG:boto:Params: {} 2023-11-02T19:03:35.903 DEBUG:boto:Token: None 2023-11-02T19:03:35.904 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:03:35 GMT /ibtljn-64/ 2023-11-02T19:03:35.904 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8= 2023-11-02T19:03:35.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:F+KRjY2Ad+lZIb5H5H5irNFO6M8=', 'Content-Length': '0'} 2023-11-02T19:03:35.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccb32ab0daa64c9f-006543f287-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:03:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:35.950 DEBUG:boto:b'ibtljn-64obj551000falseobj552023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:03:35.951 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-64 zones={a2, a1} 2023-11-02T19:03:35.952 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-11-02T19:03:35.952 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:03:35.953 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T19:03:35.950+0000 7fb1c907a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 80049) UID: 0 2023-11-02T19:03:35.954 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T19:03:35.950+0000 7fb1cc8d9f80 -1 shutting down 2023-11-02T19:03:40.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-11-02T19:03:42.051 INFO:tasks.rgw.c1.client.1:Stopped 2023-11-02T19:03:42.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:42.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:42.053 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:03:42.103 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:42.103 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:42.183 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T19:03:42.183 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-64 2023-11-02T19:03:42.191 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 5 2023-11-02T19:03:42.275 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:42.276 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:42.277 DEBUG:boto:Method: HEAD 2023-11-02T19:03:42.277 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:42.278 DEBUG:boto:Data: 2023-11-02T19:03:42.278 DEBUG:boto:Headers: {} 2023-11-02T19:03:42.279 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.279 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.279 DEBUG:boto:Params: {} 2023-11-02T19:03:42.280 DEBUG:boto:Token: None 2023-11-02T19:03:42.281 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/ 2023-11-02T19:03:42.281 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA= 2023-11-02T19:03:42.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA=', 'Content-Length': '0'} 2023-11-02T19:03:42.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000343e949ea024ca33-006543f28e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.287 DEBUG:boto:path=/ibtljn-64/obj17 2023-11-02T19:03:42.287 DEBUG:boto:auth_path=/ibtljn-64/obj17 2023-11-02T19:03:42.288 DEBUG:boto:Method: PUT 2023-11-02T19:03:42.288 DEBUG:boto:Path: /ibtljn-64/obj17 2023-11-02T19:03:42.289 DEBUG:boto:Data: 2023-11-02T19:03:42.289 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:42.289 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.290 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.290 DEBUG:boto:Params: {} 2023-11-02T19:03:42.291 DEBUG:boto:Token: None 2023-11-02T19:03:42.291 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/obj17 2023-11-02T19:03:42.292 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vs/Ll4/96Ny+PrMacx3v+FxQg1g= 2023-11-02T19:03:42.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vs/Ll4/96Ny+PrMacx3v+FxQg1g='} 2023-11-02T19:03:42.305 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e9628e22e5419ff-006543f28e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.306 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:42.306 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:42.307 DEBUG:boto:Method: HEAD 2023-11-02T19:03:42.307 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:42.308 DEBUG:boto:Data: 2023-11-02T19:03:42.309 DEBUG:boto:Headers: {} 2023-11-02T19:03:42.309 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.309 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.310 DEBUG:boto:Params: {} 2023-11-02T19:03:42.311 DEBUG:boto:Token: None 2023-11-02T19:03:42.311 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/ 2023-11-02T19:03:42.312 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA= 2023-11-02T19:03:42.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA=', 'Content-Length': '0'} 2023-11-02T19:03:42.317 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a31d47e32242a51e-006543f28e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.318 DEBUG:boto:path=/ibtljn-64/obj34 2023-11-02T19:03:42.318 DEBUG:boto:auth_path=/ibtljn-64/obj34 2023-11-02T19:03:42.319 DEBUG:boto:Method: PUT 2023-11-02T19:03:42.319 DEBUG:boto:Path: /ibtljn-64/obj34 2023-11-02T19:03:42.320 DEBUG:boto:Data: 2023-11-02T19:03:42.320 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:42.320 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.321 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.321 DEBUG:boto:Params: {} 2023-11-02T19:03:42.322 DEBUG:boto:Token: None 2023-11-02T19:03:42.322 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/obj34 2023-11-02T19:03:42.323 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6z53l78H/Fo4o4CBWbdL/SOZ5Ao= 2023-11-02T19:03:42.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6z53l78H/Fo4o4CBWbdL/SOZ5Ao='} 2023-11-02T19:03:42.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002940390649bf3b57-006543f28e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.335 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:42.336 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:42.336 DEBUG:boto:Method: HEAD 2023-11-02T19:03:42.337 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:42.337 DEBUG:boto:Data: 2023-11-02T19:03:42.338 DEBUG:boto:Headers: {} 2023-11-02T19:03:42.338 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.339 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.339 DEBUG:boto:Params: {} 2023-11-02T19:03:42.340 DEBUG:boto:Token: None 2023-11-02T19:03:42.340 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/ 2023-11-02T19:03:42.341 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA= 2023-11-02T19:03:42.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA=', 'Content-Length': '0'} 2023-11-02T19:03:42.346 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060c35965ef387355-006543f28e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.347 DEBUG:boto:path=/ibtljn-64/obj51 2023-11-02T19:03:42.347 DEBUG:boto:auth_path=/ibtljn-64/obj51 2023-11-02T19:03:42.348 DEBUG:boto:Method: PUT 2023-11-02T19:03:42.348 DEBUG:boto:Path: /ibtljn-64/obj51 2023-11-02T19:03:42.348 DEBUG:boto:Data: 2023-11-02T19:03:42.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:42.349 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.350 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.350 DEBUG:boto:Params: {} 2023-11-02T19:03:42.351 DEBUG:boto:Token: None 2023-11-02T19:03:42.351 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/obj51 2023-11-02T19:03:42.352 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M9QSYrUQ+UY/1PuPFzy9ztMjOxw= 2023-11-02T19:03:42.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M9QSYrUQ+UY/1PuPFzy9ztMjOxw='} 2023-11-02T19:03:42.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0379362407f7873-006543f28e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.395 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:42.396 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:42.396 DEBUG:boto:Method: HEAD 2023-11-02T19:03:42.397 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:42.397 DEBUG:boto:Data: 2023-11-02T19:03:42.397 DEBUG:boto:Headers: {} 2023-11-02T19:03:42.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.398 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.399 DEBUG:boto:Params: {} 2023-11-02T19:03:42.399 DEBUG:boto:Token: None 2023-11-02T19:03:42.400 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/ 2023-11-02T19:03:42.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA= 2023-11-02T19:03:42.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA=', 'Content-Length': '0'} 2023-11-02T19:03:42.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000720afd5616d1fa14-006543f28e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.406 DEBUG:boto:path=/ibtljn-64/obj68 2023-11-02T19:03:42.407 DEBUG:boto:auth_path=/ibtljn-64/obj68 2023-11-02T19:03:42.407 DEBUG:boto:Method: PUT 2023-11-02T19:03:42.408 DEBUG:boto:Path: /ibtljn-64/obj68 2023-11-02T19:03:42.408 DEBUG:boto:Data: 2023-11-02T19:03:42.409 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:42.409 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.410 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.410 DEBUG:boto:Params: {} 2023-11-02T19:03:42.411 DEBUG:boto:Token: None 2023-11-02T19:03:42.411 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/obj68 2023-11-02T19:03:42.412 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XSRnwBqp91Wc8euoJMyP9HGyH/k= 2023-11-02T19:03:42.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XSRnwBqp91Wc8euoJMyP9HGyH/k='} 2023-11-02T19:03:42.425 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab929edc9faf7e6a-006543f28e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.426 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:42.426 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:42.426 DEBUG:boto:Method: HEAD 2023-11-02T19:03:42.427 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:42.427 DEBUG:boto:Data: 2023-11-02T19:03:42.428 DEBUG:boto:Headers: {} 2023-11-02T19:03:42.428 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.429 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.429 DEBUG:boto:Params: {} 2023-11-02T19:03:42.430 DEBUG:boto:Token: None 2023-11-02T19:03:42.430 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/ 2023-11-02T19:03:42.430 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA= 2023-11-02T19:03:42.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA=', 'Content-Length': '0'} 2023-11-02T19:03:42.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001818ef79b9c101a2-006543f28e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.436 DEBUG:boto:path=/ibtljn-64/obj85 2023-11-02T19:03:42.437 DEBUG:boto:auth_path=/ibtljn-64/obj85 2023-11-02T19:03:42.437 DEBUG:boto:Method: PUT 2023-11-02T19:03:42.438 DEBUG:boto:Path: /ibtljn-64/obj85 2023-11-02T19:03:42.438 DEBUG:boto:Data: 2023-11-02T19:03:42.439 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:42.439 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.439 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.440 DEBUG:boto:Params: {} 2023-11-02T19:03:42.440 DEBUG:boto:Token: None 2023-11-02T19:03:42.441 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/obj85 2023-11-02T19:03:42.441 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dKQDFYkGv/WG8jaHbmRpfUc9RjU= 2023-11-02T19:03:42.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dKQDFYkGv/WG8jaHbmRpfUc9RjU='} 2023-11-02T19:03:42.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f1c93cc9de70f02-006543f28e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:42.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:42.488 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:03:42.539 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:42.539 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:42.621 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T19:03:42.621 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-64 2023-11-02T19:03:42.634 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 10 2023-11-02T19:03:42.729 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:42.730 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:42.730 DEBUG:boto:Method: HEAD 2023-11-02T19:03:42.731 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:42.731 DEBUG:boto:Data: 2023-11-02T19:03:42.732 DEBUG:boto:Headers: {} 2023-11-02T19:03:42.732 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.732 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.733 DEBUG:boto:Params: {} 2023-11-02T19:03:42.733 DEBUG:boto:Token: None 2023-11-02T19:03:42.734 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/ 2023-11-02T19:03:42.734 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA= 2023-11-02T19:03:42.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA=', 'Content-Length': '0'} 2023-11-02T19:03:42.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caeff05f715fbac1-006543f28e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.740 DEBUG:boto:path=/ibtljn-64/obj19 2023-11-02T19:03:42.740 DEBUG:boto:auth_path=/ibtljn-64/obj19 2023-11-02T19:03:42.741 DEBUG:boto:Method: PUT 2023-11-02T19:03:42.741 DEBUG:boto:Path: /ibtljn-64/obj19 2023-11-02T19:03:42.742 DEBUG:boto:Data: 2023-11-02T19:03:42.742 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:42.743 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.743 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.744 DEBUG:boto:Params: {} 2023-11-02T19:03:42.744 DEBUG:boto:Token: None 2023-11-02T19:03:42.745 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/obj19 2023-11-02T19:03:42.745 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KGeRw5NhPQmpqf08+uM75j2WXz8= 2023-11-02T19:03:42.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KGeRw5NhPQmpqf08+uM75j2WXz8='} 2023-11-02T19:03:42.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008dd5f67b7cfccf3a-006543f28e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.757 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:42.758 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:42.758 DEBUG:boto:Method: HEAD 2023-11-02T19:03:42.758 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:42.759 DEBUG:boto:Data: 2023-11-02T19:03:42.759 DEBUG:boto:Headers: {} 2023-11-02T19:03:42.760 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.760 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.761 DEBUG:boto:Params: {} 2023-11-02T19:03:42.761 DEBUG:boto:Token: None 2023-11-02T19:03:42.762 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/ 2023-11-02T19:03:42.762 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA= 2023-11-02T19:03:42.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA=', 'Content-Length': '0'} 2023-11-02T19:03:42.768 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d2df38152e6044a-006543f28e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.769 DEBUG:boto:path=/ibtljn-64/obj38 2023-11-02T19:03:42.769 DEBUG:boto:auth_path=/ibtljn-64/obj38 2023-11-02T19:03:42.770 DEBUG:boto:Method: PUT 2023-11-02T19:03:42.770 DEBUG:boto:Path: /ibtljn-64/obj38 2023-11-02T19:03:42.771 DEBUG:boto:Data: 2023-11-02T19:03:42.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:42.772 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.772 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.772 DEBUG:boto:Params: {} 2023-11-02T19:03:42.773 DEBUG:boto:Token: None 2023-11-02T19:03:42.773 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/obj38 2023-11-02T19:03:42.774 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KbkG8DzuF0chjKs6G2nYEgjPkr4= 2023-11-02T19:03:42.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KbkG8DzuF0chjKs6G2nYEgjPkr4='} 2023-11-02T19:03:42.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096edf5bee58c2c91-006543f28e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.819 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:42.820 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:42.820 DEBUG:boto:Method: HEAD 2023-11-02T19:03:42.821 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:42.821 DEBUG:boto:Data: 2023-11-02T19:03:42.821 DEBUG:boto:Headers: {} 2023-11-02T19:03:42.822 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.822 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.823 DEBUG:boto:Params: {} 2023-11-02T19:03:42.823 DEBUG:boto:Token: None 2023-11-02T19:03:42.824 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/ 2023-11-02T19:03:42.824 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA= 2023-11-02T19:03:42.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA=', 'Content-Length': '0'} 2023-11-02T19:03:42.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000733712f37c9fd34f-006543f28e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.831 DEBUG:boto:path=/ibtljn-64/obj57 2023-11-02T19:03:42.832 DEBUG:boto:auth_path=/ibtljn-64/obj57 2023-11-02T19:03:42.832 DEBUG:boto:Method: PUT 2023-11-02T19:03:42.833 DEBUG:boto:Path: /ibtljn-64/obj57 2023-11-02T19:03:42.833 DEBUG:boto:Data: 2023-11-02T19:03:42.833 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:42.834 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.834 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.835 DEBUG:boto:Params: {} 2023-11-02T19:03:42.835 DEBUG:boto:Token: None 2023-11-02T19:03:42.836 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/obj57 2023-11-02T19:03:42.836 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:udgqHIQKKHwgq5Dnan38bhBsHH0= 2023-11-02T19:03:42.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:udgqHIQKKHwgq5Dnan38bhBsHH0='} 2023-11-02T19:03:42.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6306c52516ffd5b-006543f28e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.846 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:42.849 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:42.850 DEBUG:boto:Method: HEAD 2023-11-02T19:03:42.850 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:42.851 DEBUG:boto:Data: 2023-11-02T19:03:42.851 DEBUG:boto:Headers: {} 2023-11-02T19:03:42.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.852 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.852 DEBUG:boto:Params: {} 2023-11-02T19:03:42.853 DEBUG:boto:Token: None 2023-11-02T19:03:42.854 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/ 2023-11-02T19:03:42.854 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA= 2023-11-02T19:03:42.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA=', 'Content-Length': '0'} 2023-11-02T19:03:42.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a855208fb84791d5-006543f28e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.861 DEBUG:boto:path=/ibtljn-64/obj76 2023-11-02T19:03:42.862 DEBUG:boto:auth_path=/ibtljn-64/obj76 2023-11-02T19:03:42.862 DEBUG:boto:Method: PUT 2023-11-02T19:03:42.863 DEBUG:boto:Path: /ibtljn-64/obj76 2023-11-02T19:03:42.863 DEBUG:boto:Data: 2023-11-02T19:03:42.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:42.864 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.865 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.865 DEBUG:boto:Params: {} 2023-11-02T19:03:42.866 DEBUG:boto:Token: None 2023-11-02T19:03:42.866 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/obj76 2023-11-02T19:03:42.867 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:j/hdnXG6B0F5SVpLPvInABS5JsA= 2023-11-02T19:03:42.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j/hdnXG6B0F5SVpLPvInABS5JsA='} 2023-11-02T19:03:42.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b465dc804562a17-006543f28e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.911 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:42.912 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:42.912 DEBUG:boto:Method: HEAD 2023-11-02T19:03:42.912 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:42.913 DEBUG:boto:Data: 2023-11-02T19:03:42.913 DEBUG:boto:Headers: {} 2023-11-02T19:03:42.914 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.914 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.914 DEBUG:boto:Params: {} 2023-11-02T19:03:42.915 DEBUG:boto:Token: None 2023-11-02T19:03:42.916 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/ 2023-11-02T19:03:42.917 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA= 2023-11-02T19:03:42.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AUHaNBnFXicxjVCNUDfaGKrNeHA=', 'Content-Length': '0'} 2023-11-02T19:03:42.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000744f30aea4cf1d7c-006543f28e-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.923 DEBUG:boto:path=/ibtljn-64/obj95 2023-11-02T19:03:42.924 DEBUG:boto:auth_path=/ibtljn-64/obj95 2023-11-02T19:03:42.924 DEBUG:boto:Method: PUT 2023-11-02T19:03:42.924 DEBUG:boto:Path: /ibtljn-64/obj95 2023-11-02T19:03:42.925 DEBUG:boto:Data: 2023-11-02T19:03:42.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:42.925 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:42.926 DEBUG:boto:Port: 8000 2023-11-02T19:03:42.926 DEBUG:boto:Params: {} 2023-11-02T19:03:42.927 DEBUG:boto:Token: None 2023-11-02T19:03:42.927 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:42 GMT /ibtljn-64/obj95 2023-11-02T19:03:42.928 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yql5PMHDCrHGmxtQ39ThUlhT8Qw= 2023-11-02T19:03:42.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:42 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yql5PMHDCrHGmxtQ39ThUlhT8Qw='} 2023-11-02T19:03:42.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de382f9f7c9fd708-006543f28e-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:42.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:42.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:42.939 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:03:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:43.034 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:43.075 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T19:03:43.075 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-64 2023-11-02T19:03:43.090 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 15 2023-11-02T19:03:43.194 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:43.195 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:43.195 DEBUG:boto:Method: HEAD 2023-11-02T19:03:43.196 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:43.196 DEBUG:boto:Data: 2023-11-02T19:03:43.197 DEBUG:boto:Headers: {} 2023-11-02T19:03:43.197 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.197 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.198 DEBUG:boto:Params: {} 2023-11-02T19:03:43.198 DEBUG:boto:Token: None 2023-11-02T19:03:43.199 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/ 2023-11-02T19:03:43.199 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU= 2023-11-02T19:03:43.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU=', 'Content-Length': '0'} 2023-11-02T19:03:43.205 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbcbf57ea797c188-006543f28f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.205 DEBUG:boto:path=/ibtljn-64/obj23 2023-11-02T19:03:43.206 DEBUG:boto:auth_path=/ibtljn-64/obj23 2023-11-02T19:03:43.206 DEBUG:boto:Method: PUT 2023-11-02T19:03:43.207 DEBUG:boto:Path: /ibtljn-64/obj23 2023-11-02T19:03:43.207 DEBUG:boto:Data: 2023-11-02T19:03:43.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:43.208 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.208 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.209 DEBUG:boto:Params: {} 2023-11-02T19:03:43.209 DEBUG:boto:Token: None 2023-11-02T19:03:43.210 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/obj23 2023-11-02T19:03:43.210 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:a4glz3fOhj5WXGSwGBEtrxn/VHI= 2023-11-02T19:03:43.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:a4glz3fOhj5WXGSwGBEtrxn/VHI='} 2023-11-02T19:03:43.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ce3ccebcbb60d71-006543f28f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.259 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:43.260 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:43.260 DEBUG:boto:Method: HEAD 2023-11-02T19:03:43.260 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:43.261 DEBUG:boto:Data: 2023-11-02T19:03:43.261 DEBUG:boto:Headers: {} 2023-11-02T19:03:43.262 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.262 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.262 DEBUG:boto:Params: {} 2023-11-02T19:03:43.263 DEBUG:boto:Token: None 2023-11-02T19:03:43.263 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/ 2023-11-02T19:03:43.264 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU= 2023-11-02T19:03:43.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU=', 'Content-Length': '0'} 2023-11-02T19:03:43.270 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e279beedc3eb041-006543f28f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.270 DEBUG:boto:path=/ibtljn-64/obj46 2023-11-02T19:03:43.271 DEBUG:boto:auth_path=/ibtljn-64/obj46 2023-11-02T19:03:43.271 DEBUG:boto:Method: PUT 2023-11-02T19:03:43.272 DEBUG:boto:Path: /ibtljn-64/obj46 2023-11-02T19:03:43.272 DEBUG:boto:Data: 2023-11-02T19:03:43.272 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:43.273 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.273 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.274 DEBUG:boto:Params: {} 2023-11-02T19:03:43.274 DEBUG:boto:Token: None 2023-11-02T19:03:43.275 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/obj46 2023-11-02T19:03:43.275 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IYz8L0yWmHkL3yVN2EMAbrjOnrc= 2023-11-02T19:03:43.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IYz8L0yWmHkL3yVN2EMAbrjOnrc='} 2023-11-02T19:03:43.287 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eef8340f9666c23c-006543f28f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.288 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:43.288 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:43.289 DEBUG:boto:Method: HEAD 2023-11-02T19:03:43.290 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:43.291 DEBUG:boto:Data: 2023-11-02T19:03:43.291 DEBUG:boto:Headers: {} 2023-11-02T19:03:43.292 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.292 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.293 DEBUG:boto:Params: {} 2023-11-02T19:03:43.293 DEBUG:boto:Token: None 2023-11-02T19:03:43.293 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/ 2023-11-02T19:03:43.294 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU= 2023-11-02T19:03:43.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU=', 'Content-Length': '0'} 2023-11-02T19:03:43.299 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000038f7702f15ef385-006543f28f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.300 DEBUG:boto:path=/ibtljn-64/obj69 2023-11-02T19:03:43.300 DEBUG:boto:auth_path=/ibtljn-64/obj69 2023-11-02T19:03:43.301 DEBUG:boto:Method: PUT 2023-11-02T19:03:43.301 DEBUG:boto:Path: /ibtljn-64/obj69 2023-11-02T19:03:43.302 DEBUG:boto:Data: 2023-11-02T19:03:43.302 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:43.303 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.303 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.304 DEBUG:boto:Params: {} 2023-11-02T19:03:43.304 DEBUG:boto:Token: None 2023-11-02T19:03:43.305 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/obj69 2023-11-02T19:03:43.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f1DsKptCT3a7oiC1323mqofOYqw= 2023-11-02T19:03:43.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f1DsKptCT3a7oiC1323mqofOYqw='} 2023-11-02T19:03:43.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d63d12d62e4ea1bf-006543f28f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.351 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:43.352 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:43.352 DEBUG:boto:Method: HEAD 2023-11-02T19:03:43.353 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:43.353 DEBUG:boto:Data: 2023-11-02T19:03:43.353 DEBUG:boto:Headers: {} 2023-11-02T19:03:43.354 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.354 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.355 DEBUG:boto:Params: {} 2023-11-02T19:03:43.355 DEBUG:boto:Token: None 2023-11-02T19:03:43.356 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/ 2023-11-02T19:03:43.356 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU= 2023-11-02T19:03:43.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU=', 'Content-Length': '0'} 2023-11-02T19:03:43.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c9eff72c2af2641-006543f28f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.364 DEBUG:boto:path=/ibtljn-64/obj92 2023-11-02T19:03:43.364 DEBUG:boto:auth_path=/ibtljn-64/obj92 2023-11-02T19:03:43.364 DEBUG:boto:Method: PUT 2023-11-02T19:03:43.365 DEBUG:boto:Path: /ibtljn-64/obj92 2023-11-02T19:03:43.365 DEBUG:boto:Data: 2023-11-02T19:03:43.366 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:43.366 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.367 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.367 DEBUG:boto:Params: {} 2023-11-02T19:03:43.368 DEBUG:boto:Token: None 2023-11-02T19:03:43.369 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/obj92 2023-11-02T19:03:43.369 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+nc8F3Gd+t8EoYsWESUo0SQ7ap0= 2023-11-02T19:03:43.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+nc8F3Gd+t8EoYsWESUo0SQ7ap0='} 2023-11-02T19:03:43.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c0079fe5691e408-006543f28f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.383 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:43.384 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:43.384 DEBUG:boto:Method: HEAD 2023-11-02T19:03:43.384 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:43.385 DEBUG:boto:Data: 2023-11-02T19:03:43.385 DEBUG:boto:Headers: {} 2023-11-02T19:03:43.386 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.386 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.387 DEBUG:boto:Params: {} 2023-11-02T19:03:43.387 DEBUG:boto:Token: None 2023-11-02T19:03:43.388 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/ 2023-11-02T19:03:43.388 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU= 2023-11-02T19:03:43.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU=', 'Content-Length': '0'} 2023-11-02T19:03:43.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073e8c1d81c7b4f92-006543f28f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.395 DEBUG:boto:path=/ibtljn-64/obj115 2023-11-02T19:03:43.396 DEBUG:boto:auth_path=/ibtljn-64/obj115 2023-11-02T19:03:43.396 DEBUG:boto:Method: PUT 2023-11-02T19:03:43.397 DEBUG:boto:Path: /ibtljn-64/obj115 2023-11-02T19:03:43.397 DEBUG:boto:Data: 2023-11-02T19:03:43.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:43.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.398 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.399 DEBUG:boto:Params: {} 2023-11-02T19:03:43.399 DEBUG:boto:Token: None 2023-11-02T19:03:43.400 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/obj115 2023-11-02T19:03:43.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6qaN2v7hy/yZtm6MtRqqoFlUIPQ= 2023-11-02T19:03:43.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6qaN2v7hy/yZtm6MtRqqoFlUIPQ='} 2023-11-02T19:03:43.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065f13565660dcf00-006543f28f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:43.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:43.448 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:03:43.498 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:43.499 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:43.589 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T19:03:43.589 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-64 2023-11-02T19:03:43.608 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 20 2023-11-02T19:03:43.712 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:43.713 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:43.713 DEBUG:boto:Method: HEAD 2023-11-02T19:03:43.714 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:43.714 DEBUG:boto:Data: 2023-11-02T19:03:43.715 DEBUG:boto:Headers: {} 2023-11-02T19:03:43.715 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.715 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.716 DEBUG:boto:Params: {} 2023-11-02T19:03:43.716 DEBUG:boto:Token: None 2023-11-02T19:03:43.717 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/ 2023-11-02T19:03:43.717 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU= 2023-11-02T19:03:43.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU=', 'Content-Length': '0'} 2023-11-02T19:03:43.723 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000526339cd1743f75c-006543f28f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.724 DEBUG:boto:path=/ibtljn-64/obj29 2023-11-02T19:03:43.724 DEBUG:boto:auth_path=/ibtljn-64/obj29 2023-11-02T19:03:43.724 DEBUG:boto:Method: PUT 2023-11-02T19:03:43.725 DEBUG:boto:Path: /ibtljn-64/obj29 2023-11-02T19:03:43.725 DEBUG:boto:Data: 2023-11-02T19:03:43.726 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:43.726 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.727 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.727 DEBUG:boto:Params: {} 2023-11-02T19:03:43.728 DEBUG:boto:Token: None 2023-11-02T19:03:43.728 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/obj29 2023-11-02T19:03:43.729 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ImzHVi2IMBpC9wm+2KmoTitZ7M0= 2023-11-02T19:03:43.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ImzHVi2IMBpC9wm+2KmoTitZ7M0='} 2023-11-02T19:03:43.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5dd2b4ebf1ab992-006543f28f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.739 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:43.740 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:43.740 DEBUG:boto:Method: HEAD 2023-11-02T19:03:43.741 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:43.741 DEBUG:boto:Data: 2023-11-02T19:03:43.742 DEBUG:boto:Headers: {} 2023-11-02T19:03:43.742 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.743 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.743 DEBUG:boto:Params: {} 2023-11-02T19:03:43.744 DEBUG:boto:Token: None 2023-11-02T19:03:43.744 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/ 2023-11-02T19:03:43.745 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU= 2023-11-02T19:03:43.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU=', 'Content-Length': '0'} 2023-11-02T19:03:43.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002446b4a2aae006fa-006543f28f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.752 DEBUG:boto:path=/ibtljn-64/obj58 2023-11-02T19:03:43.752 DEBUG:boto:auth_path=/ibtljn-64/obj58 2023-11-02T19:03:43.753 DEBUG:boto:Method: PUT 2023-11-02T19:03:43.753 DEBUG:boto:Path: /ibtljn-64/obj58 2023-11-02T19:03:43.754 DEBUG:boto:Data: 2023-11-02T19:03:43.754 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:43.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.755 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.755 DEBUG:boto:Params: {} 2023-11-02T19:03:43.756 DEBUG:boto:Token: None 2023-11-02T19:03:43.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/obj58 2023-11-02T19:03:43.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:SrNfmiNcO0644DbXHGM1DF/BPDA= 2023-11-02T19:03:43.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SrNfmiNcO0644DbXHGM1DF/BPDA='} 2023-11-02T19:03:43.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000fcd7c7038139a3b-006543f28f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.803 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:43.804 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:43.804 DEBUG:boto:Method: HEAD 2023-11-02T19:03:43.805 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:43.805 DEBUG:boto:Data: 2023-11-02T19:03:43.805 DEBUG:boto:Headers: {} 2023-11-02T19:03:43.806 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.806 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.807 DEBUG:boto:Params: {} 2023-11-02T19:03:43.807 DEBUG:boto:Token: None 2023-11-02T19:03:43.808 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/ 2023-11-02T19:03:43.808 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU= 2023-11-02T19:03:43.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU=', 'Content-Length': '0'} 2023-11-02T19:03:43.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cca94af3ca4da74d-006543f28f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.816 DEBUG:boto:path=/ibtljn-64/obj87 2023-11-02T19:03:43.816 DEBUG:boto:auth_path=/ibtljn-64/obj87 2023-11-02T19:03:43.817 DEBUG:boto:Method: PUT 2023-11-02T19:03:43.817 DEBUG:boto:Path: /ibtljn-64/obj87 2023-11-02T19:03:43.818 DEBUG:boto:Data: 2023-11-02T19:03:43.818 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:43.819 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.819 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.819 DEBUG:boto:Params: {} 2023-11-02T19:03:43.820 DEBUG:boto:Token: None 2023-11-02T19:03:43.820 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/obj87 2023-11-02T19:03:43.821 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vd/jlA+gi/hwOSA80MkQ3aa5/1A= 2023-11-02T19:03:43.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vd/jlA+gi/hwOSA80MkQ3aa5/1A='} 2023-11-02T19:03:43.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003296fd3f633d6a0b-006543f28f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.833 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:43.833 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:43.833 DEBUG:boto:Method: HEAD 2023-11-02T19:03:43.834 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:43.834 DEBUG:boto:Data: 2023-11-02T19:03:43.835 DEBUG:boto:Headers: {} 2023-11-02T19:03:43.835 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.836 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.836 DEBUG:boto:Params: {} 2023-11-02T19:03:43.836 DEBUG:boto:Token: None 2023-11-02T19:03:43.837 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/ 2023-11-02T19:03:43.837 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU= 2023-11-02T19:03:43.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU=', 'Content-Length': '0'} 2023-11-02T19:03:43.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000531bea0cced7aa9a-006543f28f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.845 DEBUG:boto:path=/ibtljn-64/obj116 2023-11-02T19:03:43.845 DEBUG:boto:auth_path=/ibtljn-64/obj116 2023-11-02T19:03:43.846 DEBUG:boto:Method: PUT 2023-11-02T19:03:43.846 DEBUG:boto:Path: /ibtljn-64/obj116 2023-11-02T19:03:43.847 DEBUG:boto:Data: 2023-11-02T19:03:43.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:43.847 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.848 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.848 DEBUG:boto:Params: {} 2023-11-02T19:03:43.849 DEBUG:boto:Token: None 2023-11-02T19:03:43.849 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/obj116 2023-11-02T19:03:43.850 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lVU2frBexERd1i2vJ/IzB22InUg= 2023-11-02T19:03:43.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lVU2frBexERd1i2vJ/IzB22InUg='} 2023-11-02T19:03:43.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b19445b7b48bd0e1-006543f28f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.895 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:43.896 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:43.896 DEBUG:boto:Method: HEAD 2023-11-02T19:03:43.897 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:43.897 DEBUG:boto:Data: 2023-11-02T19:03:43.897 DEBUG:boto:Headers: {} 2023-11-02T19:03:43.898 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.898 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.899 DEBUG:boto:Params: {} 2023-11-02T19:03:43.899 DEBUG:boto:Token: None 2023-11-02T19:03:43.900 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/ 2023-11-02T19:03:43.900 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU= 2023-11-02T19:03:43.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kgy4EDwgVM1g2UgKc3TiglzlakU=', 'Content-Length': '0'} 2023-11-02T19:03:43.908 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e510bfb1c8f37e3-006543f28f-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.909 DEBUG:boto:path=/ibtljn-64/obj145 2023-11-02T19:03:43.909 DEBUG:boto:auth_path=/ibtljn-64/obj145 2023-11-02T19:03:43.910 DEBUG:boto:Method: PUT 2023-11-02T19:03:43.910 DEBUG:boto:Path: /ibtljn-64/obj145 2023-11-02T19:03:43.911 DEBUG:boto:Data: 2023-11-02T19:03:43.911 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:43.912 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:43.912 DEBUG:boto:Port: 8000 2023-11-02T19:03:43.912 DEBUG:boto:Params: {} 2023-11-02T19:03:43.913 DEBUG:boto:Token: None 2023-11-02T19:03:43.913 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:43 GMT /ibtljn-64/obj145 2023-11-02T19:03:43.914 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fjoa5i68PbQF26TRk97RPWsHQlk= 2023-11-02T19:03:43.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:43 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fjoa5i68PbQF26TRk97RPWsHQlk='} 2023-11-02T19:03:43.923 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5e2c1b99cb0b3e0-006543f28f-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:43.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:43.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:43.925 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:03:43.972 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:43.972 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:44.067 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T19:03:44.068 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-64 2023-11-02T19:03:44.090 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 25 2023-11-02T19:03:44.219 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:44.220 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:44.221 DEBUG:boto:Method: HEAD 2023-11-02T19:03:44.221 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:44.221 DEBUG:boto:Data: 2023-11-02T19:03:44.222 DEBUG:boto:Headers: {} 2023-11-02T19:03:44.222 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.223 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.223 DEBUG:boto:Params: {} 2023-11-02T19:03:44.224 DEBUG:boto:Token: None 2023-11-02T19:03:44.224 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/ 2023-11-02T19:03:44.224 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8= 2023-11-02T19:03:44.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8=', 'Content-Length': '0'} 2023-11-02T19:03:44.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e46c32f1655bc919-006543f290-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.233 DEBUG:boto:path=/ibtljn-64/obj31 2023-11-02T19:03:44.233 DEBUG:boto:auth_path=/ibtljn-64/obj31 2023-11-02T19:03:44.233 DEBUG:boto:Method: PUT 2023-11-02T19:03:44.234 DEBUG:boto:Path: /ibtljn-64/obj31 2023-11-02T19:03:44.234 DEBUG:boto:Data: 2023-11-02T19:03:44.235 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:44.235 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.236 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.236 DEBUG:boto:Params: {} 2023-11-02T19:03:44.236 DEBUG:boto:Token: None 2023-11-02T19:03:44.237 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/obj31 2023-11-02T19:03:44.237 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jK6qk0LnIuo5LHRmxkwoV3hmHiY= 2023-11-02T19:03:44.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jK6qk0LnIuo5LHRmxkwoV3hmHiY='} 2023-11-02T19:03:44.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd529f51416c3591-006543f290-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.283 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:44.284 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:44.284 DEBUG:boto:Method: HEAD 2023-11-02T19:03:44.285 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:44.285 DEBUG:boto:Data: 2023-11-02T19:03:44.285 DEBUG:boto:Headers: {} 2023-11-02T19:03:44.286 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.286 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.287 DEBUG:boto:Params: {} 2023-11-02T19:03:44.287 DEBUG:boto:Token: None 2023-11-02T19:03:44.288 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/ 2023-11-02T19:03:44.288 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8= 2023-11-02T19:03:44.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8=', 'Content-Length': '0'} 2023-11-02T19:03:44.296 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000406ad94a4c14f60f-006543f290-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.297 DEBUG:boto:path=/ibtljn-64/obj62 2023-11-02T19:03:44.297 DEBUG:boto:auth_path=/ibtljn-64/obj62 2023-11-02T19:03:44.297 DEBUG:boto:Method: PUT 2023-11-02T19:03:44.298 DEBUG:boto:Path: /ibtljn-64/obj62 2023-11-02T19:03:44.298 DEBUG:boto:Data: 2023-11-02T19:03:44.299 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:44.299 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.300 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.300 DEBUG:boto:Params: {} 2023-11-02T19:03:44.300 DEBUG:boto:Token: None 2023-11-02T19:03:44.301 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/obj62 2023-11-02T19:03:44.301 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qUOQBVRdmBPuq3vlPfaTnWFGH5k= 2023-11-02T19:03:44.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qUOQBVRdmBPuq3vlPfaTnWFGH5k='} 2023-11-02T19:03:44.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d886cfce1351ea22-006543f290-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.313 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:44.313 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:44.314 DEBUG:boto:Method: HEAD 2023-11-02T19:03:44.314 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:44.314 DEBUG:boto:Data: 2023-11-02T19:03:44.315 DEBUG:boto:Headers: {} 2023-11-02T19:03:44.315 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.316 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.316 DEBUG:boto:Params: {} 2023-11-02T19:03:44.316 DEBUG:boto:Token: None 2023-11-02T19:03:44.317 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/ 2023-11-02T19:03:44.317 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8= 2023-11-02T19:03:44.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8=', 'Content-Length': '0'} 2023-11-02T19:03:44.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f3170a8b0f11b70-006543f290-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.325 DEBUG:boto:path=/ibtljn-64/obj93 2023-11-02T19:03:44.326 DEBUG:boto:auth_path=/ibtljn-64/obj93 2023-11-02T19:03:44.326 DEBUG:boto:Method: PUT 2023-11-02T19:03:44.327 DEBUG:boto:Path: /ibtljn-64/obj93 2023-11-02T19:03:44.327 DEBUG:boto:Data: 2023-11-02T19:03:44.328 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:44.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.328 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.329 DEBUG:boto:Params: {} 2023-11-02T19:03:44.329 DEBUG:boto:Token: None 2023-11-02T19:03:44.330 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/obj93 2023-11-02T19:03:44.330 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:c4TfTkYPHGGz8DFKRl8Sa1rUZkw= 2023-11-02T19:03:44.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c4TfTkYPHGGz8DFKRl8Sa1rUZkw='} 2023-11-02T19:03:44.375 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a02e1809e3ba5a0-006543f290-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.375 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:44.376 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:44.376 DEBUG:boto:Method: HEAD 2023-11-02T19:03:44.377 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:44.377 DEBUG:boto:Data: 2023-11-02T19:03:44.377 DEBUG:boto:Headers: {} 2023-11-02T19:03:44.378 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.378 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.379 DEBUG:boto:Params: {} 2023-11-02T19:03:44.379 DEBUG:boto:Token: None 2023-11-02T19:03:44.380 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/ 2023-11-02T19:03:44.380 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8= 2023-11-02T19:03:44.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8=', 'Content-Length': '0'} 2023-11-02T19:03:44.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034bfef54f6f466f9-006543f290-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.388 DEBUG:boto:path=/ibtljn-64/obj124 2023-11-02T19:03:44.388 DEBUG:boto:auth_path=/ibtljn-64/obj124 2023-11-02T19:03:44.389 DEBUG:boto:Method: PUT 2023-11-02T19:03:44.389 DEBUG:boto:Path: /ibtljn-64/obj124 2023-11-02T19:03:44.390 DEBUG:boto:Data: 2023-11-02T19:03:44.390 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:44.390 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.391 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.391 DEBUG:boto:Params: {} 2023-11-02T19:03:44.392 DEBUG:boto:Token: None 2023-11-02T19:03:44.392 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/obj124 2023-11-02T19:03:44.393 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jzd9S97rmxGPHLnrhY4HJRvmJ8g= 2023-11-02T19:03:44.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jzd9S97rmxGPHLnrhY4HJRvmJ8g='} 2023-11-02T19:03:44.401 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000839c93842ac182b2-006543f290-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.402 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:44.402 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:44.403 DEBUG:boto:Method: HEAD 2023-11-02T19:03:44.403 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:44.403 DEBUG:boto:Data: 2023-11-02T19:03:44.404 DEBUG:boto:Headers: {} 2023-11-02T19:03:44.404 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.405 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.405 DEBUG:boto:Params: {} 2023-11-02T19:03:44.406 DEBUG:boto:Token: None 2023-11-02T19:03:44.406 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/ 2023-11-02T19:03:44.407 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8= 2023-11-02T19:03:44.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8=', 'Content-Length': '0'} 2023-11-02T19:03:44.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004afb02429eb8023a-006543f290-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.413 DEBUG:boto:path=/ibtljn-64/obj155 2023-11-02T19:03:44.414 DEBUG:boto:auth_path=/ibtljn-64/obj155 2023-11-02T19:03:44.414 DEBUG:boto:Method: PUT 2023-11-02T19:03:44.415 DEBUG:boto:Path: /ibtljn-64/obj155 2023-11-02T19:03:44.415 DEBUG:boto:Data: 2023-11-02T19:03:44.416 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:44.416 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.416 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.417 DEBUG:boto:Params: {} 2023-11-02T19:03:44.417 DEBUG:boto:Token: None 2023-11-02T19:03:44.418 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/obj155 2023-11-02T19:03:44.418 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xQ9LCCYEfcp3/dmuEhOfuYBZZng= 2023-11-02T19:03:44.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xQ9LCCYEfcp3/dmuEhOfuYBZZng='} 2023-11-02T19:03:44.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c58592be72ba971-006543f290-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:44.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:44.464 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:03:44.513 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:44.513 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:44.607 INFO:teuthology.orchestra.run.smithi017.stdout:tenant: 2023-11-02T19:03:44.608 INFO:teuthology.orchestra.run.smithi017.stdout:bucket name: ibtljn-64 2023-11-02T19:03:44.632 INFO:teuthology.orchestra.run.smithi017.stdout:total entries: 30 2023-11-02T19:03:44.767 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:44.768 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:44.768 DEBUG:boto:Method: HEAD 2023-11-02T19:03:44.769 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:44.769 DEBUG:boto:Data: 2023-11-02T19:03:44.770 DEBUG:boto:Headers: {} 2023-11-02T19:03:44.770 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.770 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.771 DEBUG:boto:Params: {} 2023-11-02T19:03:44.771 DEBUG:boto:Token: None 2023-11-02T19:03:44.772 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/ 2023-11-02T19:03:44.772 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8= 2023-11-02T19:03:44.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8=', 'Content-Length': '0'} 2023-11-02T19:03:44.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e3a4e3892e22822-006543f290-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.780 DEBUG:boto:path=/ibtljn-64/obj37 2023-11-02T19:03:44.781 DEBUG:boto:auth_path=/ibtljn-64/obj37 2023-11-02T19:03:44.781 DEBUG:boto:Method: PUT 2023-11-02T19:03:44.782 DEBUG:boto:Path: /ibtljn-64/obj37 2023-11-02T19:03:44.782 DEBUG:boto:Data: 2023-11-02T19:03:44.783 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:44.783 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.783 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.784 DEBUG:boto:Params: {} 2023-11-02T19:03:44.785 DEBUG:boto:Token: None 2023-11-02T19:03:44.785 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/obj37 2023-11-02T19:03:44.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WxXp/A/+yAnmcU7pVAYJepjRdG4= 2023-11-02T19:03:44.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WxXp/A/+yAnmcU7pVAYJepjRdG4='} 2023-11-02T19:03:44.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6b71e780588ff4b-006543f290-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.797 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:44.797 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:44.797 DEBUG:boto:Method: HEAD 2023-11-02T19:03:44.798 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:44.798 DEBUG:boto:Data: 2023-11-02T19:03:44.799 DEBUG:boto:Headers: {} 2023-11-02T19:03:44.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.800 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.800 DEBUG:boto:Params: {} 2023-11-02T19:03:44.800 DEBUG:boto:Token: None 2023-11-02T19:03:44.801 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/ 2023-11-02T19:03:44.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8= 2023-11-02T19:03:44.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8=', 'Content-Length': '0'} 2023-11-02T19:03:44.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8da1df08c6183d1-006543f290-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.809 DEBUG:boto:path=/ibtljn-64/obj74 2023-11-02T19:03:44.809 DEBUG:boto:auth_path=/ibtljn-64/obj74 2023-11-02T19:03:44.810 DEBUG:boto:Method: PUT 2023-11-02T19:03:44.810 DEBUG:boto:Path: /ibtljn-64/obj74 2023-11-02T19:03:44.811 DEBUG:boto:Data: 2023-11-02T19:03:44.811 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:44.811 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.812 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.812 DEBUG:boto:Params: {} 2023-11-02T19:03:44.813 DEBUG:boto:Token: None 2023-11-02T19:03:44.813 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/obj74 2023-11-02T19:03:44.814 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7aPqfN7rl8rHL3y9srg35dU3xwg= 2023-11-02T19:03:44.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7aPqfN7rl8rHL3y9srg35dU3xwg='} 2023-11-02T19:03:44.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a40fa36d6d28b9fa-006543f290-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.863 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:44.864 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:44.864 DEBUG:boto:Method: HEAD 2023-11-02T19:03:44.864 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:44.865 DEBUG:boto:Data: 2023-11-02T19:03:44.865 DEBUG:boto:Headers: {} 2023-11-02T19:03:44.866 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.866 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.867 DEBUG:boto:Params: {} 2023-11-02T19:03:44.867 DEBUG:boto:Token: None 2023-11-02T19:03:44.867 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/ 2023-11-02T19:03:44.868 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8= 2023-11-02T19:03:44.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8=', 'Content-Length': '0'} 2023-11-02T19:03:44.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e2605cacbf646e8-006543f290-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.876 DEBUG:boto:path=/ibtljn-64/obj111 2023-11-02T19:03:44.877 DEBUG:boto:auth_path=/ibtljn-64/obj111 2023-11-02T19:03:44.877 DEBUG:boto:Method: PUT 2023-11-02T19:03:44.878 DEBUG:boto:Path: /ibtljn-64/obj111 2023-11-02T19:03:44.878 DEBUG:boto:Data: 2023-11-02T19:03:44.879 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:44.879 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.879 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.880 DEBUG:boto:Params: {} 2023-11-02T19:03:44.880 DEBUG:boto:Token: None 2023-11-02T19:03:44.881 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/obj111 2023-11-02T19:03:44.881 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+9W+5IByFg3Z/ui+SKCF+ZqyO5g= 2023-11-02T19:03:44.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+9W+5IByFg3Z/ui+SKCF+ZqyO5g='} 2023-11-02T19:03:44.892 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d36d2107f5789e2-006543f290-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.893 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:44.893 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:44.894 DEBUG:boto:Method: HEAD 2023-11-02T19:03:44.894 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:44.894 DEBUG:boto:Data: 2023-11-02T19:03:44.895 DEBUG:boto:Headers: {} 2023-11-02T19:03:44.895 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.896 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.896 DEBUG:boto:Params: {} 2023-11-02T19:03:44.896 DEBUG:boto:Token: None 2023-11-02T19:03:44.897 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/ 2023-11-02T19:03:44.897 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8= 2023-11-02T19:03:44.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8=', 'Content-Length': '0'} 2023-11-02T19:03:44.904 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089b88f7940c6bcc8-006543f290-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.905 DEBUG:boto:path=/ibtljn-64/obj148 2023-11-02T19:03:44.905 DEBUG:boto:auth_path=/ibtljn-64/obj148 2023-11-02T19:03:44.905 DEBUG:boto:Method: PUT 2023-11-02T19:03:44.906 DEBUG:boto:Path: /ibtljn-64/obj148 2023-11-02T19:03:44.906 DEBUG:boto:Data: 2023-11-02T19:03:44.907 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:44.907 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.908 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.908 DEBUG:boto:Params: {} 2023-11-02T19:03:44.909 DEBUG:boto:Token: None 2023-11-02T19:03:44.909 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/obj148 2023-11-02T19:03:44.910 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+B4gCd44/0h1qHMG9TpNYAzx584= 2023-11-02T19:03:44.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+B4gCd44/0h1qHMG9TpNYAzx584='} 2023-11-02T19:03:44.955 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ce31ddc11efdd1d-006543f290-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.955 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:03:44.956 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:03:44.956 DEBUG:boto:Method: HEAD 2023-11-02T19:03:44.957 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:03:44.957 DEBUG:boto:Data: 2023-11-02T19:03:44.958 DEBUG:boto:Headers: {} 2023-11-02T19:03:44.958 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.959 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.959 DEBUG:boto:Params: {} 2023-11-02T19:03:44.960 DEBUG:boto:Token: None 2023-11-02T19:03:44.960 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/ 2023-11-02T19:03:44.960 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8= 2023-11-02T19:03:44.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UowOaWHCJQuCrT0Py+5wWbA0CQ8=', 'Content-Length': '0'} 2023-11-02T19:03:44.968 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b367d973804dbb19-006543f290-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.969 DEBUG:boto:path=/ibtljn-64/obj185 2023-11-02T19:03:44.970 DEBUG:boto:auth_path=/ibtljn-64/obj185 2023-11-02T19:03:44.970 DEBUG:boto:Method: PUT 2023-11-02T19:03:44.971 DEBUG:boto:Path: /ibtljn-64/obj185 2023-11-02T19:03:44.971 DEBUG:boto:Data: 2023-11-02T19:03:44.971 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:03:44.972 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:03:44.972 DEBUG:boto:Port: 8000 2023-11-02T19:03:44.973 DEBUG:boto:Params: {} 2023-11-02T19:03:44.973 DEBUG:boto:Token: None 2023-11-02T19:03:44.974 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 02 Nov 2023 19:03:44 GMT /ibtljn-64/obj185 2023-11-02T19:03:44.974 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wzhgfyuH3Kvgv67qp7Ny+OoTeuo= 2023-11-02T19:03:44.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:03:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wzhgfyuH3Kvgv67qp7Ny+OoTeuo='} 2023-11-02T19:03:44.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eac54b8ed2b585a3-006543f290-1669-a1'), ('Date', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:03:44.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:44.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:03:44.986 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:03:45.032 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:45.032 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:45.093 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:45.090+0000 7f0638c60f40 5 RGW-SYNC:data: waiting on data sync status lease 2023-11-02T19:03:45.096 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:45.090+0000 7f0638c60f40 5 RGW-SYNC:data: acquired data sync status lease 2023-11-02T19:03:45.136 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-11-02T19:03:45.136 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-11-02T19:03:45.139 INFO:tasks.rgw.c1.client.1:Started 2023-11-02T19:03:45.139 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:45.154 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T19:03:45.154 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T19:03:45.154 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-11-02T19:03:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-11-02T19:03:45.156 INFO:teuthology.orchestra.run.smithi017.stderr:curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8001: Connection refused 2023-11-02T19:03:46.158 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2023-11-02T19:03:46.176 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-11-02T19:03:46.176 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2023-11-02T19:03:46.179 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 104k 2023-11-02T19:03:46.180 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2023-11-02T19:03:46.181 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=ibtljn-64 2023-11-02T19:03:46.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:03:46.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:03:46.182 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T19:03:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:03:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:03:46.347 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:46.342+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:03:46.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-11-02T19:03:51.348 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:51.342+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:03:56.349 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:03:56.342+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:04:01.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:04:01.346+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:04:06.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:04:06.346+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:04:11.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:04:11.346+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:04:16.353 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:04:16.346+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:04:21.354 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:04:21.350+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:04:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:04:26.349+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:04:31.356 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:04:31.349+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:04:36.357 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:04:36.353+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:04:41.358 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:04:41.353+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:04:46.359 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:04:46.353+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:04:51.360 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:04:51.353+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:04:56.361 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:04:56.357+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:05:01.362 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:01.357+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:05:06.363 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:06.357+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:05:11.364 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:11.357+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:05:16.366 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:16.361+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:05:21.367 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:21.361+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:05:26.368 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:26.361+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:05:31.370 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:31.365+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:05:36.371 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:36.365+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:05:41.372 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:41.369+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:05:46.373 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:46.369+0000 7f514ff4ff40 1 waiting to reach latest gen 6, on 0.. 2023-11-02T19:05:51.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:51.373+0000 7f514ff4ff40 1 bucket sync caught up with source: 2023-11-02T19:05:51.380 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , 00000000002.106.6, , , , , , 00000000002.120.6, , , , , 00000000002.93.6, , , , , , , , , , , , , , , 00000000001.105.6, , , , 00000000001.120.6] 2023-11-02T19:05:51.380 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.120.6, , , , , 00000000002.93.6, , , , , , , , , , , , , , , 00000000001.105.6, , , , 00000000001.120.6] 2023-11-02T19:05:51.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:51.373+0000 7f514ff4ff40 0 bucket checkpoint complete 2023-11-02T19:05:51.406 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=ibtljn-64 2023-11-02T19:05:51.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:05:51.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:05:51.408 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T19:05:51.459 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:05:51.459 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:05:51.527 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:51.521+0000 7f08ad8a6f40 1 waiting for incremental sync to catch up: 2023-11-02T19:05:51.527 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, , , , , , , , 00000000001.331.6, 00000000002.362.6, 00000000002.337.6] 2023-11-02T19:05:51.527 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000003.348.6, 00000000002.122.6, 00000000002.190.6, 00000000005.350.6, 00000000004.345.6, 00000000001.186.6, 00000000001.118.6, 00000000004.343.6, 00000000007.378.6, 00000000006.351.6] 2023-11-02T19:05:56.530 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:56.525+0000 7f08ad8a6f40 1 bucket sync caught up with source: 2023-11-02T19:05:56.530 INFO:teuthology.orchestra.run.smithi017.stderr: local status: [, 00000000003.348.6, 00000000002.122.6, 00000000002.190.6, 00000000005.350.6, 00000000004.345.6, 00000000001.186.6, 00000000001.118.6, 00000000004.343.6, 00000000007.378.6, 00000000006.351.6] 2023-11-02T19:05:56.530 INFO:teuthology.orchestra.run.smithi017.stderr: remote markers: [, 00000000003.348.6, 00000000002.122.6, 00000000002.190.6, 00000000005.350.6, 00000000004.345.6, 00000000001.186.6, 00000000001.118.6, 00000000004.343.6, 00000000007.378.6, 00000000006.351.6] 2023-11-02T19:05:56.531 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:05:56.525+0000 7f08ad8a6f40 0 bucket checkpoint complete 2023-11-02T19:05:56.568 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ibtljn-64 zones={a2, a1} 2023-11-02T19:05:56.570 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:56.571 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:56.571 DEBUG:boto:Method: HEAD 2023-11-02T19:05:56.572 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:05:56.572 DEBUG:boto:Data: 2023-11-02T19:05:56.573 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:56.574 DEBUG:boto:Port: 8001 2023-11-02T19:05:56.575 DEBUG:boto:Params: {} 2023-11-02T19:05:56.575 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:05:56.576 DEBUG:boto:Token: None 2023-11-02T19:05:56.576 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/ 2023-11-02T19:05:56.577 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:md5821NaJS9IYwhCw0R9jvxDC1w= 2023-11-02T19:05:56.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:md5821NaJS9IYwhCw0R9jvxDC1w=', 'Content-Length': '0'} 2023-11-02T19:05:56.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024535212a315d967-006543f314-20d2-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.587 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:56.588 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:56.588 DEBUG:boto:Method: HEAD 2023-11-02T19:05:56.589 DEBUG:boto:Path: /ibtljn-64/ 2023-11-02T19:05:56.589 DEBUG:boto:Data: 2023-11-02T19:05:56.590 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:56.591 DEBUG:boto:Port: 8000 2023-11-02T19:05:56.591 DEBUG:boto:Params: {} 2023-11-02T19:05:56.592 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T19:05:56.592 DEBUG:boto:Token: None 2023-11-02T19:05:56.593 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/ 2023-11-02T19:05:56.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:md5821NaJS9IYwhCw0R9jvxDC1w= 2023-11-02T19:05:56.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:md5821NaJS9IYwhCw0R9jvxDC1w=', 'Content-Length': '0'} 2023-11-02T19:05:56.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de1e37842b284e20-006543f314-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.603 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-11-02T19:05:56.604 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:56.604 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:56.605 DEBUG:boto:path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:05:56.605 DEBUG:boto:auth_path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:05:56.606 DEBUG:boto:Method: GET 2023-11-02T19:05:56.606 DEBUG:boto:Path: /ibtljn-64/?versions&max-keys=999 2023-11-02T19:05:56.607 DEBUG:boto:Data: 2023-11-02T19:05:56.607 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.607 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:56.608 DEBUG:boto:Port: 8001 2023-11-02T19:05:56.608 DEBUG:boto:Params: {} 2023-11-02T19:05:56.609 DEBUG:boto:Token: None 2023-11-02T19:05:56.609 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/?versions 2023-11-02T19:05:56.610 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p3K6b83/AWpViZEPdtobG+X0rKo= 2023-11-02T19:05:56.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p3K6b83/AWpViZEPdtobG+X0rKo=', 'Content-Length': '0'} 2023-11-02T19:05:56.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083fd551bc9c9ff60-006543f314-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.616 DEBUG:boto:b'ibtljn-64999falseobj11nulltrue2023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T19:03:44.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T19:03:43.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T19:03:43.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T19:03:44.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T19:03:43.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T19:03:44.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T19:03:44.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T19:03:42.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T19:03:44.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T19:03:42.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T19:03:43.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T19:03:43.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T19:03:44.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T19:03:42.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T19:03:44.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T19:03:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T19:03:43.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T19:03:42.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T19:03:42.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T19:03:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T19:03:44.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T19:03:42.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T19:03:43.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T19:03:44.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T19:03:42.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T19:03:42.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T19:03:43.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T19:03:43.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T19:03:44.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T19:03:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:56.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T19:05:56.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-11-02T19:05:56.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T19:05:56.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-11-02T19:05:56.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-11-02T19:05:56.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-11-02T19:05:56.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-11-02T19:05:56.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-11-02T19:05:56.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T19:05:56.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-11-02T19:05:56.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T19:05:56.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T19:05:56.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T19:05:56.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-11-02T19:05:56.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-11-02T19:05:56.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T19:05:56.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T19:05:56.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-11-02T19:05:56.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T19:05:56.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T19:05:56.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T19:05:56.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T19:05:56.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T19:05:56.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T19:05:56.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-11-02T19:05:56.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-11-02T19:05:56.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T19:05:56.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T19:05:56.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-11-02T19:05:56.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T19:05:56.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T19:05:56.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-11-02T19:05:56.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T19:05:56.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-11-02T19:05:56.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T19:05:56.631 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-11-02T19:05:56.632 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:56.632 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:56.632 DEBUG:boto:path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:05:56.633 DEBUG:boto:auth_path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:05:56.633 DEBUG:boto:Method: GET 2023-11-02T19:05:56.634 DEBUG:boto:Path: /ibtljn-64/?versions&max-keys=999 2023-11-02T19:05:56.634 DEBUG:boto:Data: 2023-11-02T19:05:56.635 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.635 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:56.635 DEBUG:boto:Port: 8000 2023-11-02T19:05:56.636 DEBUG:boto:Params: {} 2023-11-02T19:05:56.636 DEBUG:boto:Token: None 2023-11-02T19:05:56.637 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/?versions 2023-11-02T19:05:56.637 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p3K6b83/AWpViZEPdtobG+X0rKo= 2023-11-02T19:05:56.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p3K6b83/AWpViZEPdtobG+X0rKo=', 'Content-Length': '0'} 2023-11-02T19:05:56.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c07100faf42dcdca-006543f314-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.647 DEBUG:boto:b'ibtljn-64999falseobj11nulltrue2023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T19:03:44.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T19:03:43.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T19:03:43.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T19:03:44.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T19:03:43.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T19:03:44.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T19:03:44.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T19:03:42.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T19:03:44.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T19:03:42.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T19:03:43.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T19:03:43.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T19:03:44.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T19:03:42.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T19:03:44.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T19:03:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T19:03:43.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T19:03:42.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T19:03:42.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T19:03:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T19:03:44.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T19:03:42.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T19:03:43.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T19:03:44.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T19:03:42.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T19:03:42.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T19:03:43.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T19:03:43.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T19:03:44.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T19:03:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:56.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-11-02T19:05:56.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-11-02T19:05:56.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-11-02T19:05:56.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-11-02T19:05:56.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-11-02T19:05:56.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-11-02T19:05:56.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-11-02T19:05:56.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-11-02T19:05:56.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-11-02T19:05:56.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-11-02T19:05:56.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-11-02T19:05:56.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-11-02T19:05:56.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-11-02T19:05:56.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-11-02T19:05:56.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-11-02T19:05:56.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-11-02T19:05:56.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-11-02T19:05:56.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-11-02T19:05:56.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-11-02T19:05:56.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-11-02T19:05:56.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-11-02T19:05:56.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-11-02T19:05:56.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-11-02T19:05:56.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-11-02T19:05:56.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-11-02T19:05:56.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-11-02T19:05:56.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-11-02T19:05:56.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-11-02T19:05:56.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-11-02T19:05:56.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-11-02T19:05:56.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-11-02T19:05:56.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-11-02T19:05:56.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-11-02T19:05:56.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-11-02T19:05:56.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-11-02T19:05:56.662 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:56.663 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:56.663 DEBUG:boto:path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:05:56.664 DEBUG:boto:auth_path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:05:56.664 DEBUG:boto:Method: GET 2023-11-02T19:05:56.664 DEBUG:boto:Path: /ibtljn-64/?versions&max-keys=999 2023-11-02T19:05:56.665 DEBUG:boto:Data: 2023-11-02T19:05:56.665 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.666 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:56.666 DEBUG:boto:Port: 8001 2023-11-02T19:05:56.667 DEBUG:boto:Params: {} 2023-11-02T19:05:56.667 DEBUG:boto:Token: None 2023-11-02T19:05:56.668 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/?versions 2023-11-02T19:05:56.668 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p3K6b83/AWpViZEPdtobG+X0rKo= 2023-11-02T19:05:56.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p3K6b83/AWpViZEPdtobG+X0rKo=', 'Content-Length': '0'} 2023-11-02T19:05:56.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f542c0a20df82ea-006543f314-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.675 DEBUG:boto:b'ibtljn-64999falseobj11nulltrue2023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T19:03:44.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T19:03:43.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T19:03:43.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T19:03:44.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T19:03:43.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T19:03:44.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T19:03:44.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T19:03:42.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T19:03:44.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T19:03:42.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T19:03:43.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T19:03:43.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T19:03:44.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T19:03:42.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T19:03:44.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T19:03:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T19:03:43.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T19:03:42.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T19:03:42.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T19:03:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T19:03:44.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T19:03:42.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T19:03:43.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T19:03:44.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T19:03:42.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T19:03:42.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T19:03:43.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T19:03:43.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T19:03:44.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T19:03:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:56.677 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:56.677 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:56.678 DEBUG:boto:path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:05:56.678 DEBUG:boto:auth_path=/ibtljn-64/?versions&max-keys=999 2023-11-02T19:05:56.679 DEBUG:boto:Method: GET 2023-11-02T19:05:56.679 DEBUG:boto:Path: /ibtljn-64/?versions&max-keys=999 2023-11-02T19:05:56.679 DEBUG:boto:Data: 2023-11-02T19:05:56.680 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.680 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:56.681 DEBUG:boto:Port: 8000 2023-11-02T19:05:56.681 DEBUG:boto:Params: {} 2023-11-02T19:05:56.682 DEBUG:boto:Token: None 2023-11-02T19:05:56.682 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/?versions 2023-11-02T19:05:56.683 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:p3K6b83/AWpViZEPdtobG+X0rKo= 2023-11-02T19:05:56.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:p3K6b83/AWpViZEPdtobG+X0rKo=', 'Content-Length': '0'} 2023-11-02T19:05:56.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000636a4afc211b140-006543f314-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.690 DEBUG:boto:b'ibtljn-64999falseobj11nulltrue2023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-11-02T19:03:44.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-11-02T19:03:43.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-11-02T19:03:43.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-11-02T19:03:44.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-11-02T19:03:43.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-11-02T19:03:44.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-11-02T19:03:44.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-11-02T19:03:42.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-11-02T19:03:44.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-11-02T19:03:42.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-11-02T19:03:43.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-11-02T19:03:43.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-11-02T19:03:44.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-11-02T19:03:42.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-11-02T19:03:44.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-11-02T19:03:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-11-02T19:03:43.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-11-02T19:03:42.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-11-02T19:03:42.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-11-02T19:03:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-11-02T19:03:44.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-11-02T19:03:42.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-11-02T19:03:43.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-11-02T19:03:44.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-11-02T19:03:42.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-11-02T19:03:42.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-11-02T19:03:43.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-11-02T19:03:43.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-11-02T19:03:44.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-11-02T19:03:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:56.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T19:05:56.692 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:05:56.693 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:05:56.693 DEBUG:boto:path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.694 DEBUG:boto:auth_path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.694 DEBUG:boto:Method: GET 2023-11-02T19:05:56.695 DEBUG:boto:Path: /ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.695 DEBUG:boto:Data: 2023-11-02T19:05:56.695 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.696 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:56.696 DEBUG:boto:Port: 8001 2023-11-02T19:05:56.697 DEBUG:boto:Params: {} 2023-11-02T19:05:56.697 DEBUG:boto:Token: None 2023-11-02T19:05:56.698 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.698 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sXxlEcwalnB9t++dhJ9HAwF7/n8= 2023-11-02T19:05:56.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sXxlEcwalnB9t++dhJ9HAwF7/n8=', 'Content-Length': '0'} 2023-11-02T19:05:56.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000641dab881f5bdc3f-006543f314-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.702 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:05:56.703 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:05:56.703 DEBUG:boto:path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.704 DEBUG:boto:auth_path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.704 DEBUG:boto:Method: GET 2023-11-02T19:05:56.705 DEBUG:boto:Path: /ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.705 DEBUG:boto:Data: 2023-11-02T19:05:56.706 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.706 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:56.706 DEBUG:boto:Port: 8000 2023-11-02T19:05:56.707 DEBUG:boto:Params: {} 2023-11-02T19:05:56.707 DEBUG:boto:Token: None 2023-11-02T19:05:56.708 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.708 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sXxlEcwalnB9t++dhJ9HAwF7/n8= 2023-11-02T19:05:56.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sXxlEcwalnB9t++dhJ9HAwF7/n8=', 'Content-Length': '0'} 2023-11-02T19:05:56.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3a8e6f25d04a5aa-006543f314-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.758 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:05:56.759 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:05:56.759 DEBUG:boto:path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.760 DEBUG:boto:auth_path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.760 DEBUG:boto:Method: HEAD 2023-11-02T19:05:56.760 DEBUG:boto:Path: /ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.761 DEBUG:boto:Data: 2023-11-02T19:05:56.761 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.762 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:56.762 DEBUG:boto:Port: 8001 2023-11-02T19:05:56.762 DEBUG:boto:Params: {} 2023-11-02T19:05:56.763 DEBUG:boto:Token: None 2023-11-02T19:05:56.763 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.764 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:r2L7PbTYHbwIsTAOJZ3XhEST+iA= 2023-11-02T19:05:56.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:r2L7PbTYHbwIsTAOJZ3XhEST+iA=', 'Content-Length': '0'} 2023-11-02T19:05:56.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096cd7cc93e4394f9-006543f314-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.767 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:05:56.768 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:05:56.768 DEBUG:boto:path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.769 DEBUG:boto:auth_path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.769 DEBUG:boto:Method: HEAD 2023-11-02T19:05:56.770 DEBUG:boto:Path: /ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.770 DEBUG:boto:Data: 2023-11-02T19:05:56.771 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.771 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:56.771 DEBUG:boto:Port: 8000 2023-11-02T19:05:56.772 DEBUG:boto:Params: {} 2023-11-02T19:05:56.772 DEBUG:boto:Token: None 2023-11-02T19:05:56.773 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.773 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:r2L7PbTYHbwIsTAOJZ3XhEST+iA= 2023-11-02T19:05:56.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:r2L7PbTYHbwIsTAOJZ3XhEST+iA=', 'Content-Length': '0'} 2023-11-02T19:05:56.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003f44b8127411bfe-006543f314-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T19:05:56.777 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:05:56.777 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:05:56.777 DEBUG:boto:path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.778 DEBUG:boto:auth_path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.778 DEBUG:boto:Method: GET 2023-11-02T19:05:56.779 DEBUG:boto:Path: /ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.779 DEBUG:boto:Data: 2023-11-02T19:05:56.780 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.780 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:56.780 DEBUG:boto:Port: 8001 2023-11-02T19:05:56.781 DEBUG:boto:Params: {} 2023-11-02T19:05:56.781 DEBUG:boto:Token: None 2023-11-02T19:05:56.782 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.782 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sXxlEcwalnB9t++dhJ9HAwF7/n8= 2023-11-02T19:05:56.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sXxlEcwalnB9t++dhJ9HAwF7/n8=', 'Content-Length': '0'} 2023-11-02T19:05:56.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef115c60fd4e1e65-006543f314-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.830 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:05:56.831 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:05:56.831 DEBUG:boto:path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.832 DEBUG:boto:auth_path=/ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.832 DEBUG:boto:Method: GET 2023-11-02T19:05:56.833 DEBUG:boto:Path: /ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.833 DEBUG:boto:Data: 2023-11-02T19:05:56.834 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.834 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:56.834 DEBUG:boto:Port: 8000 2023-11-02T19:05:56.835 DEBUG:boto:Params: {} 2023-11-02T19:05:56.835 DEBUG:boto:Token: None 2023-11-02T19:05:56.836 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/obj11?versionId=null 2023-11-02T19:05:56.836 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sXxlEcwalnB9t++dhJ9HAwF7/n8= 2023-11-02T19:05:56.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sXxlEcwalnB9t++dhJ9HAwF7/n8=', 'Content-Length': '0'} 2023-11-02T19:05:56.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2ca5b5d9b41f5b2-006543f314-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.841 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:56.841 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:56.842 DEBUG:boto:path=/ibtljn-64/?prefix=obj11 2023-11-02T19:05:56.842 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj11 2023-11-02T19:05:56.843 DEBUG:boto:Method: GET 2023-11-02T19:05:56.843 DEBUG:boto:Path: /ibtljn-64/?prefix=obj11 2023-11-02T19:05:56.844 DEBUG:boto:Data: 2023-11-02T19:05:56.844 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.844 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:56.845 DEBUG:boto:Port: 8001 2023-11-02T19:05:56.845 DEBUG:boto:Params: {} 2023-11-02T19:05:56.846 DEBUG:boto:Token: None 2023-11-02T19:05:56.846 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/ 2023-11-02T19:05:56.847 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWVsTxPi1CRV4ToHYpuAISFFGoI= 2023-11-02T19:05:56.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWVsTxPi1CRV4ToHYpuAISFFGoI=', 'Content-Length': '0'} 2023-11-02T19:05:56.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d182d56d830c218b-006543f314-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.853 DEBUG:boto:b'ibtljn-64obj111000falseobj112023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T19:03:44.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T19:03:43.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T19:03:43.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:56.854 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:56.854 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:56.855 DEBUG:boto:path=/ibtljn-64/?prefix=obj11 2023-11-02T19:05:56.855 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj11 2023-11-02T19:05:56.856 DEBUG:boto:Method: GET 2023-11-02T19:05:56.856 DEBUG:boto:Path: /ibtljn-64/?prefix=obj11 2023-11-02T19:05:56.856 DEBUG:boto:Data: 2023-11-02T19:05:56.857 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:56.858 DEBUG:boto:Port: 8000 2023-11-02T19:05:56.858 DEBUG:boto:Params: {} 2023-11-02T19:05:56.859 DEBUG:boto:Token: None 2023-11-02T19:05:56.859 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/ 2023-11-02T19:05:56.860 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWVsTxPi1CRV4ToHYpuAISFFGoI= 2023-11-02T19:05:56.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWVsTxPi1CRV4ToHYpuAISFFGoI=', 'Content-Length': '0'} 2023-11-02T19:05:56.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ed8fcf4542a8d0e-006543f314-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.868 DEBUG:boto:b'ibtljn-64obj111000falseobj112023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T19:03:44.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T19:03:43.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T19:03:43.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:56.869 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:05:56.869 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:05:56.870 DEBUG:boto:Method: HEAD 2023-11-02T19:05:56.870 DEBUG:boto:Path: /ibtljn-64/obj11 2023-11-02T19:05:56.870 DEBUG:boto:Data: 2023-11-02T19:05:56.871 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.871 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:56.872 DEBUG:boto:Port: 8001 2023-11-02T19:05:56.872 DEBUG:boto:Params: {} 2023-11-02T19:05:56.873 DEBUG:boto:Token: None 2023-11-02T19:05:56.873 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/obj11 2023-11-02T19:05:56.874 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZZH8qf8jSexjIDlsWFkicK5Bgdo= 2023-11-02T19:05:56.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZZH8qf8jSexjIDlsWFkicK5Bgdo=', 'Content-Length': '0'} 2023-11-02T19:05:56.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015e61e6996151f1b-006543f314-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.877 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:05:56.877 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:05:56.878 DEBUG:boto:Method: HEAD 2023-11-02T19:05:56.878 DEBUG:boto:Path: /ibtljn-64/obj11 2023-11-02T19:05:56.879 DEBUG:boto:Data: 2023-11-02T19:05:56.879 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.879 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:56.880 DEBUG:boto:Port: 8000 2023-11-02T19:05:56.880 DEBUG:boto:Params: {} 2023-11-02T19:05:56.881 DEBUG:boto:Token: None 2023-11-02T19:05:56.881 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/obj11 2023-11-02T19:05:56.882 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZZH8qf8jSexjIDlsWFkicK5Bgdo= 2023-11-02T19:05:56.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZZH8qf8jSexjIDlsWFkicK5Bgdo=', 'Content-Length': '0'} 2023-11-02T19:05:56.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ed317ccce35264d-006543f314-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-11-02T19:05:56.886 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:05:56.886 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:05:56.887 DEBUG:boto:Method: GET 2023-11-02T19:05:56.887 DEBUG:boto:Path: /ibtljn-64/obj11 2023-11-02T19:05:56.888 DEBUG:boto:Data: 2023-11-02T19:05:56.888 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.888 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:56.889 DEBUG:boto:Port: 8001 2023-11-02T19:05:56.889 DEBUG:boto:Params: {} 2023-11-02T19:05:56.890 DEBUG:boto:Token: None 2023-11-02T19:05:56.890 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/obj11 2023-11-02T19:05:56.891 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Mff3Zd4lIufIO9NEI+NstfXdTxU= 2023-11-02T19:05:56.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Mff3Zd4lIufIO9NEI+NstfXdTxU=', 'Content-Length': '0'} 2023-11-02T19:05:56.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000236e49992d8b38dd-006543f314-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.897 DEBUG:boto:path=/ibtljn-64/obj11 2023-11-02T19:05:56.897 DEBUG:boto:auth_path=/ibtljn-64/obj11 2023-11-02T19:05:56.898 DEBUG:boto:Method: GET 2023-11-02T19:05:56.898 DEBUG:boto:Path: /ibtljn-64/obj11 2023-11-02T19:05:56.899 DEBUG:boto:Data: 2023-11-02T19:05:56.899 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.899 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:56.900 DEBUG:boto:Port: 8000 2023-11-02T19:05:56.900 DEBUG:boto:Params: {} 2023-11-02T19:05:56.901 DEBUG:boto:Token: None 2023-11-02T19:05:56.901 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/obj11 2023-11-02T19:05:56.902 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Mff3Zd4lIufIO9NEI+NstfXdTxU= 2023-11-02T19:05:56.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Mff3Zd4lIufIO9NEI+NstfXdTxU=', 'Content-Length': '0'} 2023-11-02T19:05:56.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e61ea0a8312c000-006543f314-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.946 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:56.947 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:56.948 DEBUG:boto:path=/ibtljn-64/?prefix=obj11 2023-11-02T19:05:56.948 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj11 2023-11-02T19:05:56.949 DEBUG:boto:Method: GET 2023-11-02T19:05:56.950 DEBUG:boto:Path: /ibtljn-64/?prefix=obj11 2023-11-02T19:05:56.950 DEBUG:boto:Data: 2023-11-02T19:05:56.951 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.952 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:56.952 DEBUG:boto:Port: 8001 2023-11-02T19:05:56.953 DEBUG:boto:Params: {} 2023-11-02T19:05:56.954 DEBUG:boto:Token: None 2023-11-02T19:05:56.955 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/ 2023-11-02T19:05:56.956 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWVsTxPi1CRV4ToHYpuAISFFGoI= 2023-11-02T19:05:56.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWVsTxPi1CRV4ToHYpuAISFFGoI=', 'Content-Length': '0'} 2023-11-02T19:05:56.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c955fd89b39c0540-006543f314-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.964 DEBUG:boto:b'ibtljn-64obj111000falseobj112023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T19:03:44.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T19:03:43.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T19:03:43.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:56.965 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:56.965 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:56.966 DEBUG:boto:path=/ibtljn-64/?prefix=obj11 2023-11-02T19:05:56.966 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj11 2023-11-02T19:05:56.967 DEBUG:boto:Method: GET 2023-11-02T19:05:56.967 DEBUG:boto:Path: /ibtljn-64/?prefix=obj11 2023-11-02T19:05:56.967 DEBUG:boto:Data: 2023-11-02T19:05:56.968 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.968 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:56.969 DEBUG:boto:Port: 8000 2023-11-02T19:05:56.969 DEBUG:boto:Params: {} 2023-11-02T19:05:56.970 DEBUG:boto:Token: None 2023-11-02T19:05:56.970 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/ 2023-11-02T19:05:56.971 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vWVsTxPi1CRV4ToHYpuAISFFGoI= 2023-11-02T19:05:56.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vWVsTxPi1CRV4ToHYpuAISFFGoI=', 'Content-Length': '0'} 2023-11-02T19:05:56.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd526865a541d2fe-006543f314-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:56.978 DEBUG:boto:b'ibtljn-64obj111000falseobj112023-11-02T19:03:18.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-11-02T19:03:44.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-11-02T19:03:43.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-11-02T19:03:43.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:56.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-11-02T19:05:56.979 DEBUG:boto:path=/ibtljn-64/obj111 2023-11-02T19:05:56.980 DEBUG:boto:auth_path=/ibtljn-64/obj111 2023-11-02T19:05:56.980 DEBUG:boto:path=/ibtljn-64/obj111?versionId=null 2023-11-02T19:05:56.981 DEBUG:boto:auth_path=/ibtljn-64/obj111?versionId=null 2023-11-02T19:05:56.981 DEBUG:boto:Method: GET 2023-11-02T19:05:56.982 DEBUG:boto:Path: /ibtljn-64/obj111?versionId=null 2023-11-02T19:05:56.982 DEBUG:boto:Data: 2023-11-02T19:05:56.982 DEBUG:boto:Headers: {} 2023-11-02T19:05:56.983 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:56.983 DEBUG:boto:Port: 8001 2023-11-02T19:05:56.984 DEBUG:boto:Params: {} 2023-11-02T19:05:56.984 DEBUG:boto:Token: None 2023-11-02T19:05:56.985 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:56 GMT /ibtljn-64/obj111?versionId=null 2023-11-02T19:05:56.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:q/92z+GROhsFIQ0Wt4jgIiFFaI0= 2023-11-02T19:05:56.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:56 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:q/92z+GROhsFIQ0Wt4jgIiFFaI0=', 'Content-Length': '0'} 2023-11-02T19:05:56.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068ee3c21bda43a0e-006543f314-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.034 DEBUG:boto:path=/ibtljn-64/obj111 2023-11-02T19:05:57.035 DEBUG:boto:auth_path=/ibtljn-64/obj111 2023-11-02T19:05:57.035 DEBUG:boto:path=/ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.036 DEBUG:boto:auth_path=/ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.036 DEBUG:boto:Method: GET 2023-11-02T19:05:57.037 DEBUG:boto:Path: /ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.037 DEBUG:boto:Data: 2023-11-02T19:05:57.038 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.038 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.038 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.039 DEBUG:boto:Params: {} 2023-11-02T19:05:57.039 DEBUG:boto:Token: None 2023-11-02T19:05:57.040 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.040 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gwckgyNiE7CudFW7ShkpPiDU4v4= 2023-11-02T19:05:57.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gwckgyNiE7CudFW7ShkpPiDU4v4=', 'Content-Length': '0'} 2023-11-02T19:05:57.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a21736bcbbd14ad-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.045 DEBUG:boto:path=/ibtljn-64/obj111 2023-11-02T19:05:57.045 DEBUG:boto:auth_path=/ibtljn-64/obj111 2023-11-02T19:05:57.046 DEBUG:boto:path=/ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.046 DEBUG:boto:auth_path=/ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.047 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.047 DEBUG:boto:Path: /ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.047 DEBUG:boto:Data: 2023-11-02T19:05:57.048 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.048 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.049 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.049 DEBUG:boto:Params: {} 2023-11-02T19:05:57.049 DEBUG:boto:Token: None 2023-11-02T19:05:57.050 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.050 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l49yymLxHzJe3LzE8t/8l9mtgl8= 2023-11-02T19:05:57.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l49yymLxHzJe3LzE8t/8l9mtgl8=', 'Content-Length': '0'} 2023-11-02T19:05:57.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078f1deb8bae4d0c7-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.055 DEBUG:boto:path=/ibtljn-64/obj111 2023-11-02T19:05:57.055 DEBUG:boto:auth_path=/ibtljn-64/obj111 2023-11-02T19:05:57.055 DEBUG:boto:path=/ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.056 DEBUG:boto:auth_path=/ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.056 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.057 DEBUG:boto:Path: /ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.057 DEBUG:boto:Data: 2023-11-02T19:05:57.057 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.058 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.058 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.059 DEBUG:boto:Params: {} 2023-11-02T19:05:57.059 DEBUG:boto:Token: None 2023-11-02T19:05:57.060 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.060 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l49yymLxHzJe3LzE8t/8l9mtgl8= 2023-11-02T19:05:57.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l49yymLxHzJe3LzE8t/8l9mtgl8=', 'Content-Length': '0'} 2023-11-02T19:05:57.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061fafa2e4093880a-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-11-02T19:05:57.064 DEBUG:boto:path=/ibtljn-64/obj111 2023-11-02T19:05:57.065 DEBUG:boto:auth_path=/ibtljn-64/obj111 2023-11-02T19:05:57.065 DEBUG:boto:path=/ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.066 DEBUG:boto:auth_path=/ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.066 DEBUG:boto:Method: GET 2023-11-02T19:05:57.067 DEBUG:boto:Path: /ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.067 DEBUG:boto:Data: 2023-11-02T19:05:57.068 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.068 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.068 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.069 DEBUG:boto:Params: {} 2023-11-02T19:05:57.069 DEBUG:boto:Token: None 2023-11-02T19:05:57.070 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.070 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gwckgyNiE7CudFW7ShkpPiDU4v4= 2023-11-02T19:05:57.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gwckgyNiE7CudFW7ShkpPiDU4v4=', 'Content-Length': '0'} 2023-11-02T19:05:57.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000634a7ad550642ea8-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.075 DEBUG:boto:path=/ibtljn-64/obj111 2023-11-02T19:05:57.075 DEBUG:boto:auth_path=/ibtljn-64/obj111 2023-11-02T19:05:57.076 DEBUG:boto:path=/ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.076 DEBUG:boto:auth_path=/ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.076 DEBUG:boto:Method: GET 2023-11-02T19:05:57.077 DEBUG:boto:Path: /ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.077 DEBUG:boto:Data: 2023-11-02T19:05:57.078 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.078 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.079 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.079 DEBUG:boto:Params: {} 2023-11-02T19:05:57.080 DEBUG:boto:Token: None 2023-11-02T19:05:57.080 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj111?versionId=null 2023-11-02T19:05:57.081 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gwckgyNiE7CudFW7ShkpPiDU4v4= 2023-11-02T19:05:57.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gwckgyNiE7CudFW7ShkpPiDU4v4=', 'Content-Length': '0'} 2023-11-02T19:05:57.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a066c50cf4a255a-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.126 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.127 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.127 DEBUG:boto:path=/ibtljn-64/?prefix=obj111 2023-11-02T19:05:57.128 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj111 2023-11-02T19:05:57.128 DEBUG:boto:Method: GET 2023-11-02T19:05:57.129 DEBUG:boto:Path: /ibtljn-64/?prefix=obj111 2023-11-02T19:05:57.129 DEBUG:boto:Data: 2023-11-02T19:05:57.130 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.130 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.131 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.131 DEBUG:boto:Params: {} 2023-11-02T19:05:57.132 DEBUG:boto:Token: None 2023-11-02T19:05:57.132 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.132 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4136fa1dfc09eab-006543f315-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.138 DEBUG:boto:b'ibtljn-64obj1111000falseobj1112023-11-02T19:03:44.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.139 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.139 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.140 DEBUG:boto:path=/ibtljn-64/?prefix=obj111 2023-11-02T19:05:57.140 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj111 2023-11-02T19:05:57.141 DEBUG:boto:Method: GET 2023-11-02T19:05:57.141 DEBUG:boto:Path: /ibtljn-64/?prefix=obj111 2023-11-02T19:05:57.141 DEBUG:boto:Data: 2023-11-02T19:05:57.142 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.142 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.143 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.143 DEBUG:boto:Params: {} 2023-11-02T19:05:57.144 DEBUG:boto:Token: None 2023-11-02T19:05:57.144 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.145 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbc3867114546338-006543f315-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.155 DEBUG:boto:b'ibtljn-64obj1111000falseobj1112023-11-02T19:03:44.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.155 DEBUG:boto:path=/ibtljn-64/obj111 2023-11-02T19:05:57.156 DEBUG:boto:auth_path=/ibtljn-64/obj111 2023-11-02T19:05:57.156 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.157 DEBUG:boto:Path: /ibtljn-64/obj111 2023-11-02T19:05:57.157 DEBUG:boto:Data: 2023-11-02T19:05:57.157 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.158 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.158 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.159 DEBUG:boto:Params: {} 2023-11-02T19:05:57.159 DEBUG:boto:Token: None 2023-11-02T19:05:57.160 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj111 2023-11-02T19:05:57.160 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GvDl0uKCFRoLtmJAlJTzoEr+keA= 2023-11-02T19:05:57.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GvDl0uKCFRoLtmJAlJTzoEr+keA=', 'Content-Length': '0'} 2023-11-02T19:05:57.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000073fbcb7b623aa-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.163 DEBUG:boto:path=/ibtljn-64/obj111 2023-11-02T19:05:57.164 DEBUG:boto:auth_path=/ibtljn-64/obj111 2023-11-02T19:05:57.164 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.165 DEBUG:boto:Path: /ibtljn-64/obj111 2023-11-02T19:05:57.165 DEBUG:boto:Data: 2023-11-02T19:05:57.165 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.166 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.166 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.167 DEBUG:boto:Params: {} 2023-11-02T19:05:57.167 DEBUG:boto:Token: None 2023-11-02T19:05:57.168 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj111 2023-11-02T19:05:57.168 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GvDl0uKCFRoLtmJAlJTzoEr+keA= 2023-11-02T19:05:57.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GvDl0uKCFRoLtmJAlJTzoEr+keA=', 'Content-Length': '0'} 2023-11-02T19:05:57.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b374fe9ebe1fbc3a-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-11-02T19:05:57.171 DEBUG:boto:path=/ibtljn-64/obj111 2023-11-02T19:05:57.172 DEBUG:boto:auth_path=/ibtljn-64/obj111 2023-11-02T19:05:57.172 DEBUG:boto:Method: GET 2023-11-02T19:05:57.173 DEBUG:boto:Path: /ibtljn-64/obj111 2023-11-02T19:05:57.173 DEBUG:boto:Data: 2023-11-02T19:05:57.173 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.174 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.174 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.175 DEBUG:boto:Params: {} 2023-11-02T19:05:57.175 DEBUG:boto:Token: None 2023-11-02T19:05:57.176 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj111 2023-11-02T19:05:57.176 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iPcDHYr86+7rbgU6q9pxboQiiPg= 2023-11-02T19:05:57.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iPcDHYr86+7rbgU6q9pxboQiiPg=', 'Content-Length': '0'} 2023-11-02T19:05:57.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bcf0300b86121ef-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.222 DEBUG:boto:path=/ibtljn-64/obj111 2023-11-02T19:05:57.223 DEBUG:boto:auth_path=/ibtljn-64/obj111 2023-11-02T19:05:57.224 DEBUG:boto:Method: GET 2023-11-02T19:05:57.224 DEBUG:boto:Path: /ibtljn-64/obj111 2023-11-02T19:05:57.224 DEBUG:boto:Data: 2023-11-02T19:05:57.225 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.225 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.226 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.226 DEBUG:boto:Params: {} 2023-11-02T19:05:57.227 DEBUG:boto:Token: None 2023-11-02T19:05:57.227 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj111 2023-11-02T19:05:57.228 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iPcDHYr86+7rbgU6q9pxboQiiPg= 2023-11-02T19:05:57.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iPcDHYr86+7rbgU6q9pxboQiiPg=', 'Content-Length': '0'} 2023-11-02T19:05:57.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d3ae771f79072a-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.232 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.233 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.233 DEBUG:boto:path=/ibtljn-64/?prefix=obj111 2023-11-02T19:05:57.233 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj111 2023-11-02T19:05:57.234 DEBUG:boto:Method: GET 2023-11-02T19:05:57.234 DEBUG:boto:Path: /ibtljn-64/?prefix=obj111 2023-11-02T19:05:57.235 DEBUG:boto:Data: 2023-11-02T19:05:57.235 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.236 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.236 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.237 DEBUG:boto:Params: {} 2023-11-02T19:05:57.237 DEBUG:boto:Token: None 2023-11-02T19:05:57.238 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.238 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039511a93920be739-006543f315-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.244 DEBUG:boto:b'ibtljn-64obj1111000falseobj1112023-11-02T19:03:44.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.245 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.245 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.246 DEBUG:boto:path=/ibtljn-64/?prefix=obj111 2023-11-02T19:05:57.246 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj111 2023-11-02T19:05:57.246 DEBUG:boto:Method: GET 2023-11-02T19:05:57.247 DEBUG:boto:Path: /ibtljn-64/?prefix=obj111 2023-11-02T19:05:57.247 DEBUG:boto:Data: 2023-11-02T19:05:57.248 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.248 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.249 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.249 DEBUG:boto:Params: {} 2023-11-02T19:05:57.250 DEBUG:boto:Token: None 2023-11-02T19:05:57.250 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.251 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000075d92a08314cd19-006543f315-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.302 DEBUG:boto:b'ibtljn-64obj1111000falseobj1112023-11-02T19:03:44.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T19:05:57.304 DEBUG:boto:path=/ibtljn-64/obj115 2023-11-02T19:05:57.304 DEBUG:boto:auth_path=/ibtljn-64/obj115 2023-11-02T19:05:57.304 DEBUG:boto:path=/ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.305 DEBUG:boto:auth_path=/ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.305 DEBUG:boto:Method: GET 2023-11-02T19:05:57.306 DEBUG:boto:Path: /ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.306 DEBUG:boto:Data: 2023-11-02T19:05:57.307 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.307 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.307 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.308 DEBUG:boto:Params: {} 2023-11-02T19:05:57.308 DEBUG:boto:Token: None 2023-11-02T19:05:57.309 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.309 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M4akN5+seWipsYPottk+2r1qJ/Q= 2023-11-02T19:05:57.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M4akN5+seWipsYPottk+2r1qJ/Q=', 'Content-Length': '0'} 2023-11-02T19:05:57.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3b46cee85e8f215-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.315 DEBUG:boto:path=/ibtljn-64/obj115 2023-11-02T19:05:57.315 DEBUG:boto:auth_path=/ibtljn-64/obj115 2023-11-02T19:05:57.315 DEBUG:boto:path=/ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.316 DEBUG:boto:auth_path=/ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.316 DEBUG:boto:Method: GET 2023-11-02T19:05:57.317 DEBUG:boto:Path: /ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.317 DEBUG:boto:Data: 2023-11-02T19:05:57.318 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.318 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.319 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.319 DEBUG:boto:Params: {} 2023-11-02T19:05:57.320 DEBUG:boto:Token: None 2023-11-02T19:05:57.320 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.321 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M4akN5+seWipsYPottk+2r1qJ/Q= 2023-11-02T19:05:57.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M4akN5+seWipsYPottk+2r1qJ/Q=', 'Content-Length': '0'} 2023-11-02T19:05:57.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16a451e6a19fd9c-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.324 DEBUG:boto:path=/ibtljn-64/obj115 2023-11-02T19:05:57.325 DEBUG:boto:auth_path=/ibtljn-64/obj115 2023-11-02T19:05:57.325 DEBUG:boto:path=/ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.326 DEBUG:boto:auth_path=/ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.326 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.327 DEBUG:boto:Path: /ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.327 DEBUG:boto:Data: 2023-11-02T19:05:57.327 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.328 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.328 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.329 DEBUG:boto:Params: {} 2023-11-02T19:05:57.329 DEBUG:boto:Token: None 2023-11-02T19:05:57.330 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.330 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x+Shp/lWEDtqFXx5hNrvMEMhDaI= 2023-11-02T19:05:57.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x+Shp/lWEDtqFXx5hNrvMEMhDaI=', 'Content-Length': '0'} 2023-11-02T19:05:57.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a05b5bdf9a6e9ec7-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.334 DEBUG:boto:path=/ibtljn-64/obj115 2023-11-02T19:05:57.335 DEBUG:boto:auth_path=/ibtljn-64/obj115 2023-11-02T19:05:57.335 DEBUG:boto:path=/ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.336 DEBUG:boto:auth_path=/ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.336 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.336 DEBUG:boto:Path: /ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.337 DEBUG:boto:Data: 2023-11-02T19:05:57.337 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.338 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.338 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.339 DEBUG:boto:Params: {} 2023-11-02T19:05:57.339 DEBUG:boto:Token: None 2023-11-02T19:05:57.340 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.340 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x+Shp/lWEDtqFXx5hNrvMEMhDaI= 2023-11-02T19:05:57.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x+Shp/lWEDtqFXx5hNrvMEMhDaI=', 'Content-Length': '0'} 2023-11-02T19:05:57.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001401c1efaebff3e6-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T19:05:57.344 DEBUG:boto:path=/ibtljn-64/obj115 2023-11-02T19:05:57.345 DEBUG:boto:auth_path=/ibtljn-64/obj115 2023-11-02T19:05:57.345 DEBUG:boto:path=/ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.346 DEBUG:boto:auth_path=/ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.346 DEBUG:boto:Method: GET 2023-11-02T19:05:57.346 DEBUG:boto:Path: /ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.347 DEBUG:boto:Data: 2023-11-02T19:05:57.347 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.348 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.348 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.349 DEBUG:boto:Params: {} 2023-11-02T19:05:57.349 DEBUG:boto:Token: None 2023-11-02T19:05:57.350 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.350 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M4akN5+seWipsYPottk+2r1qJ/Q= 2023-11-02T19:05:57.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M4akN5+seWipsYPottk+2r1qJ/Q=', 'Content-Length': '0'} 2023-11-02T19:05:57.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dec04f24cc3e2b7-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.355 DEBUG:boto:path=/ibtljn-64/obj115 2023-11-02T19:05:57.355 DEBUG:boto:auth_path=/ibtljn-64/obj115 2023-11-02T19:05:57.356 DEBUG:boto:path=/ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.356 DEBUG:boto:auth_path=/ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.357 DEBUG:boto:Method: GET 2023-11-02T19:05:57.357 DEBUG:boto:Path: /ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.357 DEBUG:boto:Data: 2023-11-02T19:05:57.358 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.359 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.359 DEBUG:boto:Params: {} 2023-11-02T19:05:57.360 DEBUG:boto:Token: None 2023-11-02T19:05:57.360 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj115?versionId=null 2023-11-02T19:05:57.361 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M4akN5+seWipsYPottk+2r1qJ/Q= 2023-11-02T19:05:57.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M4akN5+seWipsYPottk+2r1qJ/Q=', 'Content-Length': '0'} 2023-11-02T19:05:57.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004371ca2cb95a8223-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.407 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.407 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.408 DEBUG:boto:path=/ibtljn-64/?prefix=obj115 2023-11-02T19:05:57.408 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj115 2023-11-02T19:05:57.409 DEBUG:boto:Method: GET 2023-11-02T19:05:57.409 DEBUG:boto:Path: /ibtljn-64/?prefix=obj115 2023-11-02T19:05:57.410 DEBUG:boto:Data: 2023-11-02T19:05:57.410 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.410 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.411 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.411 DEBUG:boto:Params: {} 2023-11-02T19:05:57.412 DEBUG:boto:Token: None 2023-11-02T19:05:57.412 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.413 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3faf78e533cf507-006543f315-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.421 DEBUG:boto:b'ibtljn-64obj1151000falseobj1152023-11-02T19:03:43.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.422 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.423 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.423 DEBUG:boto:path=/ibtljn-64/?prefix=obj115 2023-11-02T19:05:57.423 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj115 2023-11-02T19:05:57.424 DEBUG:boto:Method: GET 2023-11-02T19:05:57.424 DEBUG:boto:Path: /ibtljn-64/?prefix=obj115 2023-11-02T19:05:57.425 DEBUG:boto:Data: 2023-11-02T19:05:57.425 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.426 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.427 DEBUG:boto:Params: {} 2023-11-02T19:05:57.428 DEBUG:boto:Token: None 2023-11-02T19:05:57.428 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.429 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b39613abb59b6d1-006543f315-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.440 DEBUG:boto:b'ibtljn-64obj1151000falseobj1152023-11-02T19:03:43.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.441 DEBUG:boto:path=/ibtljn-64/obj115 2023-11-02T19:05:57.441 DEBUG:boto:auth_path=/ibtljn-64/obj115 2023-11-02T19:05:57.441 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.442 DEBUG:boto:Path: /ibtljn-64/obj115 2023-11-02T19:05:57.442 DEBUG:boto:Data: 2023-11-02T19:05:57.443 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.443 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.444 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.444 DEBUG:boto:Params: {} 2023-11-02T19:05:57.444 DEBUG:boto:Token: None 2023-11-02T19:05:57.445 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj115 2023-11-02T19:05:57.445 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RrF4fo26KAml/Q9O/pt+dKJunnw= 2023-11-02T19:05:57.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RrF4fo26KAml/Q9O/pt+dKJunnw=', 'Content-Length': '0'} 2023-11-02T19:05:57.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b48ae36942aebb9-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.449 DEBUG:boto:path=/ibtljn-64/obj115 2023-11-02T19:05:57.450 DEBUG:boto:auth_path=/ibtljn-64/obj115 2023-11-02T19:05:57.450 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.451 DEBUG:boto:Path: /ibtljn-64/obj115 2023-11-02T19:05:57.451 DEBUG:boto:Data: 2023-11-02T19:05:57.451 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.452 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.452 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.453 DEBUG:boto:Params: {} 2023-11-02T19:05:57.453 DEBUG:boto:Token: None 2023-11-02T19:05:57.453 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj115 2023-11-02T19:05:57.454 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RrF4fo26KAml/Q9O/pt+dKJunnw= 2023-11-02T19:05:57.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RrF4fo26KAml/Q9O/pt+dKJunnw=', 'Content-Length': '0'} 2023-11-02T19:05:57.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f666a75bc15ec70-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-11-02T19:05:57.458 DEBUG:boto:path=/ibtljn-64/obj115 2023-11-02T19:05:57.458 DEBUG:boto:auth_path=/ibtljn-64/obj115 2023-11-02T19:05:57.459 DEBUG:boto:Method: GET 2023-11-02T19:05:57.459 DEBUG:boto:Path: /ibtljn-64/obj115 2023-11-02T19:05:57.460 DEBUG:boto:Data: 2023-11-02T19:05:57.460 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.460 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.461 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.461 DEBUG:boto:Params: {} 2023-11-02T19:05:57.462 DEBUG:boto:Token: None 2023-11-02T19:05:57.462 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj115 2023-11-02T19:05:57.463 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+rXHa+PH/GBqWZqFmX6QbJXbX+8= 2023-11-02T19:05:57.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+rXHa+PH/GBqWZqFmX6QbJXbX+8=', 'Content-Length': '0'} 2023-11-02T19:05:57.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000772f43bfa35697ff-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.467 DEBUG:boto:path=/ibtljn-64/obj115 2023-11-02T19:05:57.467 DEBUG:boto:auth_path=/ibtljn-64/obj115 2023-11-02T19:05:57.468 DEBUG:boto:Method: GET 2023-11-02T19:05:57.468 DEBUG:boto:Path: /ibtljn-64/obj115 2023-11-02T19:05:57.469 DEBUG:boto:Data: 2023-11-02T19:05:57.469 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.469 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.470 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.470 DEBUG:boto:Params: {} 2023-11-02T19:05:57.471 DEBUG:boto:Token: None 2023-11-02T19:05:57.471 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj115 2023-11-02T19:05:57.472 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+rXHa+PH/GBqWZqFmX6QbJXbX+8= 2023-11-02T19:05:57.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+rXHa+PH/GBqWZqFmX6QbJXbX+8=', 'Content-Length': '0'} 2023-11-02T19:05:57.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de19c74fe475016e-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.476 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.476 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.477 DEBUG:boto:path=/ibtljn-64/?prefix=obj115 2023-11-02T19:05:57.477 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj115 2023-11-02T19:05:57.478 DEBUG:boto:Method: GET 2023-11-02T19:05:57.478 DEBUG:boto:Path: /ibtljn-64/?prefix=obj115 2023-11-02T19:05:57.479 DEBUG:boto:Data: 2023-11-02T19:05:57.479 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.479 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.480 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.480 DEBUG:boto:Params: {} 2023-11-02T19:05:57.481 DEBUG:boto:Token: None 2023-11-02T19:05:57.481 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.482 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7d72c24bfcc7b75-006543f315-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.488 DEBUG:boto:b'ibtljn-64obj1151000falseobj1152023-11-02T19:03:43.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.489 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.489 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.489 DEBUG:boto:path=/ibtljn-64/?prefix=obj115 2023-11-02T19:05:57.490 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj115 2023-11-02T19:05:57.490 DEBUG:boto:Method: GET 2023-11-02T19:05:57.491 DEBUG:boto:Path: /ibtljn-64/?prefix=obj115 2023-11-02T19:05:57.491 DEBUG:boto:Data: 2023-11-02T19:05:57.492 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.492 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.493 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.493 DEBUG:boto:Params: {} 2023-11-02T19:05:57.494 DEBUG:boto:Token: None 2023-11-02T19:05:57.494 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.495 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076dde5d368c16bce-006543f315-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.546 DEBUG:boto:b'ibtljn-64obj1151000falseobj1152023-11-02T19:03:43.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T19:05:57.548 DEBUG:boto:path=/ibtljn-64/obj116 2023-11-02T19:05:57.548 DEBUG:boto:auth_path=/ibtljn-64/obj116 2023-11-02T19:05:57.549 DEBUG:boto:path=/ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.549 DEBUG:boto:auth_path=/ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.550 DEBUG:boto:Method: GET 2023-11-02T19:05:57.550 DEBUG:boto:Path: /ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.551 DEBUG:boto:Data: 2023-11-02T19:05:57.551 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.551 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.552 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.552 DEBUG:boto:Params: {} 2023-11-02T19:05:57.553 DEBUG:boto:Token: None 2023-11-02T19:05:57.553 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.554 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MFjw5+Dg27vZ8ARpQBLN0tneZsc= 2023-11-02T19:05:57.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MFjw5+Dg27vZ8ARpQBLN0tneZsc=', 'Content-Length': '0'} 2023-11-02T19:05:57.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011cb8e33ce9c0aef-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.558 DEBUG:boto:path=/ibtljn-64/obj116 2023-11-02T19:05:57.559 DEBUG:boto:auth_path=/ibtljn-64/obj116 2023-11-02T19:05:57.559 DEBUG:boto:path=/ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.560 DEBUG:boto:auth_path=/ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.560 DEBUG:boto:Method: GET 2023-11-02T19:05:57.561 DEBUG:boto:Path: /ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.561 DEBUG:boto:Data: 2023-11-02T19:05:57.562 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.562 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.563 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.563 DEBUG:boto:Params: {} 2023-11-02T19:05:57.564 DEBUG:boto:Token: None 2023-11-02T19:05:57.564 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.565 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MFjw5+Dg27vZ8ARpQBLN0tneZsc= 2023-11-02T19:05:57.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MFjw5+Dg27vZ8ARpQBLN0tneZsc=', 'Content-Length': '0'} 2023-11-02T19:05:57.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000914af9005d199c38-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.569 DEBUG:boto:path=/ibtljn-64/obj116 2023-11-02T19:05:57.569 DEBUG:boto:auth_path=/ibtljn-64/obj116 2023-11-02T19:05:57.570 DEBUG:boto:path=/ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.570 DEBUG:boto:auth_path=/ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.571 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.571 DEBUG:boto:Path: /ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.572 DEBUG:boto:Data: 2023-11-02T19:05:57.572 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.573 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.573 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.574 DEBUG:boto:Params: {} 2023-11-02T19:05:57.574 DEBUG:boto:Token: None 2023-11-02T19:05:57.575 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.575 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BtX2bQ5bC8fDbvdIdQNvPZasjZM= 2023-11-02T19:05:57.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BtX2bQ5bC8fDbvdIdQNvPZasjZM=', 'Content-Length': '0'} 2023-11-02T19:05:57.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6ee746426807f81-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.579 DEBUG:boto:path=/ibtljn-64/obj116 2023-11-02T19:05:57.580 DEBUG:boto:auth_path=/ibtljn-64/obj116 2023-11-02T19:05:57.580 DEBUG:boto:path=/ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.580 DEBUG:boto:auth_path=/ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.581 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.581 DEBUG:boto:Path: /ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.582 DEBUG:boto:Data: 2023-11-02T19:05:57.582 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.583 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.583 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.583 DEBUG:boto:Params: {} 2023-11-02T19:05:57.584 DEBUG:boto:Token: None 2023-11-02T19:05:57.584 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.585 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BtX2bQ5bC8fDbvdIdQNvPZasjZM= 2023-11-02T19:05:57.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BtX2bQ5bC8fDbvdIdQNvPZasjZM=', 'Content-Length': '0'} 2023-11-02T19:05:57.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004df508616c1fa73e-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T19:05:57.589 DEBUG:boto:path=/ibtljn-64/obj116 2023-11-02T19:05:57.590 DEBUG:boto:auth_path=/ibtljn-64/obj116 2023-11-02T19:05:57.590 DEBUG:boto:path=/ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.591 DEBUG:boto:auth_path=/ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.591 DEBUG:boto:Method: GET 2023-11-02T19:05:57.592 DEBUG:boto:Path: /ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.592 DEBUG:boto:Data: 2023-11-02T19:05:57.593 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.593 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.594 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.594 DEBUG:boto:Params: {} 2023-11-02T19:05:57.595 DEBUG:boto:Token: None 2023-11-02T19:05:57.595 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.596 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MFjw5+Dg27vZ8ARpQBLN0tneZsc= 2023-11-02T19:05:57.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MFjw5+Dg27vZ8ARpQBLN0tneZsc=', 'Content-Length': '0'} 2023-11-02T19:05:57.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa64358488931d9-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.646 DEBUG:boto:path=/ibtljn-64/obj116 2023-11-02T19:05:57.647 DEBUG:boto:auth_path=/ibtljn-64/obj116 2023-11-02T19:05:57.647 DEBUG:boto:path=/ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.648 DEBUG:boto:auth_path=/ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.648 DEBUG:boto:Method: GET 2023-11-02T19:05:57.649 DEBUG:boto:Path: /ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.649 DEBUG:boto:Data: 2023-11-02T19:05:57.650 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.650 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.651 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.651 DEBUG:boto:Params: {} 2023-11-02T19:05:57.652 DEBUG:boto:Token: None 2023-11-02T19:05:57.652 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj116?versionId=null 2023-11-02T19:05:57.653 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MFjw5+Dg27vZ8ARpQBLN0tneZsc= 2023-11-02T19:05:57.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MFjw5+Dg27vZ8ARpQBLN0tneZsc=', 'Content-Length': '0'} 2023-11-02T19:05:57.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da834fd25bd13af9-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.657 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.657 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.658 DEBUG:boto:path=/ibtljn-64/?prefix=obj116 2023-11-02T19:05:57.658 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj116 2023-11-02T19:05:57.659 DEBUG:boto:Method: GET 2023-11-02T19:05:57.659 DEBUG:boto:Path: /ibtljn-64/?prefix=obj116 2023-11-02T19:05:57.660 DEBUG:boto:Data: 2023-11-02T19:05:57.660 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.661 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.661 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.661 DEBUG:boto:Params: {} 2023-11-02T19:05:57.662 DEBUG:boto:Token: None 2023-11-02T19:05:57.662 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.663 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c332d4ca6f983919-006543f315-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.669 DEBUG:boto:b'ibtljn-64obj1161000falseobj1162023-11-02T19:03:43.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.669 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.670 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.670 DEBUG:boto:path=/ibtljn-64/?prefix=obj116 2023-11-02T19:05:57.670 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj116 2023-11-02T19:05:57.671 DEBUG:boto:Method: GET 2023-11-02T19:05:57.671 DEBUG:boto:Path: /ibtljn-64/?prefix=obj116 2023-11-02T19:05:57.672 DEBUG:boto:Data: 2023-11-02T19:05:57.672 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.673 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.673 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.674 DEBUG:boto:Params: {} 2023-11-02T19:05:57.674 DEBUG:boto:Token: None 2023-11-02T19:05:57.675 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.675 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000046228c7ff3e67f5-006543f315-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.734 DEBUG:boto:b'ibtljn-64obj1161000falseobj1162023-11-02T19:03:43.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.735 DEBUG:boto:path=/ibtljn-64/obj116 2023-11-02T19:05:57.736 DEBUG:boto:auth_path=/ibtljn-64/obj116 2023-11-02T19:05:57.736 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.737 DEBUG:boto:Path: /ibtljn-64/obj116 2023-11-02T19:05:57.737 DEBUG:boto:Data: 2023-11-02T19:05:57.737 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.738 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.738 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.739 DEBUG:boto:Params: {} 2023-11-02T19:05:57.739 DEBUG:boto:Token: None 2023-11-02T19:05:57.740 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj116 2023-11-02T19:05:57.740 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ws2jWR583/1Fsfrs9JInnwD+snk= 2023-11-02T19:05:57.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ws2jWR583/1Fsfrs9JInnwD+snk=', 'Content-Length': '0'} 2023-11-02T19:05:57.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c833da00678ea37d-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.744 DEBUG:boto:path=/ibtljn-64/obj116 2023-11-02T19:05:57.745 DEBUG:boto:auth_path=/ibtljn-64/obj116 2023-11-02T19:05:57.745 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.745 DEBUG:boto:Path: /ibtljn-64/obj116 2023-11-02T19:05:57.746 DEBUG:boto:Data: 2023-11-02T19:05:57.746 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.747 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.747 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.748 DEBUG:boto:Params: {} 2023-11-02T19:05:57.748 DEBUG:boto:Token: None 2023-11-02T19:05:57.749 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj116 2023-11-02T19:05:57.749 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ws2jWR583/1Fsfrs9JInnwD+snk= 2023-11-02T19:05:57.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ws2jWR583/1Fsfrs9JInnwD+snk=', 'Content-Length': '0'} 2023-11-02T19:05:57.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ae73f139c20db9f-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-11-02T19:05:57.754 DEBUG:boto:path=/ibtljn-64/obj116 2023-11-02T19:05:57.754 DEBUG:boto:auth_path=/ibtljn-64/obj116 2023-11-02T19:05:57.755 DEBUG:boto:Method: GET 2023-11-02T19:05:57.755 DEBUG:boto:Path: /ibtljn-64/obj116 2023-11-02T19:05:57.756 DEBUG:boto:Data: 2023-11-02T19:05:57.756 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.757 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.757 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.758 DEBUG:boto:Params: {} 2023-11-02T19:05:57.758 DEBUG:boto:Token: None 2023-11-02T19:05:57.759 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj116 2023-11-02T19:05:57.759 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lW6hQhQZlvfP+x4BLZ2U8gpWIS8= 2023-11-02T19:05:57.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lW6hQhQZlvfP+x4BLZ2U8gpWIS8=', 'Content-Length': '0'} 2023-11-02T19:05:57.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001773005ea06ed084-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.763 DEBUG:boto:path=/ibtljn-64/obj116 2023-11-02T19:05:57.764 DEBUG:boto:auth_path=/ibtljn-64/obj116 2023-11-02T19:05:57.764 DEBUG:boto:Method: GET 2023-11-02T19:05:57.764 DEBUG:boto:Path: /ibtljn-64/obj116 2023-11-02T19:05:57.765 DEBUG:boto:Data: 2023-11-02T19:05:57.765 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.766 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.766 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.767 DEBUG:boto:Params: {} 2023-11-02T19:05:57.767 DEBUG:boto:Token: None 2023-11-02T19:05:57.768 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj116 2023-11-02T19:05:57.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lW6hQhQZlvfP+x4BLZ2U8gpWIS8= 2023-11-02T19:05:57.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lW6hQhQZlvfP+x4BLZ2U8gpWIS8=', 'Content-Length': '0'} 2023-11-02T19:05:57.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d7094bb9ea99f61-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.772 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.773 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.773 DEBUG:boto:path=/ibtljn-64/?prefix=obj116 2023-11-02T19:05:57.773 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj116 2023-11-02T19:05:57.774 DEBUG:boto:Method: GET 2023-11-02T19:05:57.774 DEBUG:boto:Path: /ibtljn-64/?prefix=obj116 2023-11-02T19:05:57.775 DEBUG:boto:Data: 2023-11-02T19:05:57.775 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.776 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.776 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.777 DEBUG:boto:Params: {} 2023-11-02T19:05:57.777 DEBUG:boto:Token: None 2023-11-02T19:05:57.778 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.778 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016951591d1224b04-006543f315-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.826 DEBUG:boto:b'ibtljn-64obj1161000falseobj1162023-11-02T19:03:43.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.827 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.829 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.829 DEBUG:boto:path=/ibtljn-64/?prefix=obj116 2023-11-02T19:05:57.830 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj116 2023-11-02T19:05:57.830 DEBUG:boto:Method: GET 2023-11-02T19:05:57.830 DEBUG:boto:Path: /ibtljn-64/?prefix=obj116 2023-11-02T19:05:57.831 DEBUG:boto:Data: 2023-11-02T19:05:57.831 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.832 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.832 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.833 DEBUG:boto:Params: {} 2023-11-02T19:05:57.834 DEBUG:boto:Token: None 2023-11-02T19:05:57.834 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.835 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070bdea8245e84d1c-006543f315-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.846 DEBUG:boto:b'ibtljn-64obj1161000falseobj1162023-11-02T19:03:43.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T19:05:57.847 DEBUG:boto:path=/ibtljn-64/obj124 2023-11-02T19:05:57.847 DEBUG:boto:auth_path=/ibtljn-64/obj124 2023-11-02T19:05:57.848 DEBUG:boto:path=/ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.848 DEBUG:boto:auth_path=/ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.849 DEBUG:boto:Method: GET 2023-11-02T19:05:57.849 DEBUG:boto:Path: /ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.850 DEBUG:boto:Data: 2023-11-02T19:05:57.850 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.851 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.851 DEBUG:boto:Params: {} 2023-11-02T19:05:57.852 DEBUG:boto:Token: None 2023-11-02T19:05:57.852 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.853 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+nJlYfjlTJ3IRvzP1FMcJ29kSUw= 2023-11-02T19:05:57.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+nJlYfjlTJ3IRvzP1FMcJ29kSUw=', 'Content-Length': '0'} 2023-11-02T19:05:57.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005209910a1bfd65a2-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.857 DEBUG:boto:path=/ibtljn-64/obj124 2023-11-02T19:05:57.857 DEBUG:boto:auth_path=/ibtljn-64/obj124 2023-11-02T19:05:57.858 DEBUG:boto:path=/ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.858 DEBUG:boto:auth_path=/ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.859 DEBUG:boto:Method: GET 2023-11-02T19:05:57.859 DEBUG:boto:Path: /ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.860 DEBUG:boto:Data: 2023-11-02T19:05:57.860 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.861 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.861 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.861 DEBUG:boto:Params: {} 2023-11-02T19:05:57.862 DEBUG:boto:Token: None 2023-11-02T19:05:57.863 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.863 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+nJlYfjlTJ3IRvzP1FMcJ29kSUw= 2023-11-02T19:05:57.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+nJlYfjlTJ3IRvzP1FMcJ29kSUw=', 'Content-Length': '0'} 2023-11-02T19:05:57.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c192a221fea869-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.867 DEBUG:boto:path=/ibtljn-64/obj124 2023-11-02T19:05:57.867 DEBUG:boto:auth_path=/ibtljn-64/obj124 2023-11-02T19:05:57.868 DEBUG:boto:path=/ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.868 DEBUG:boto:auth_path=/ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.868 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.869 DEBUG:boto:Path: /ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.869 DEBUG:boto:Data: 2023-11-02T19:05:57.870 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.871 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.871 DEBUG:boto:Params: {} 2023-11-02T19:05:57.871 DEBUG:boto:Token: None 2023-11-02T19:05:57.872 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.872 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f5XtEMNzMmAxPO/z2P/I2qwktgQ= 2023-11-02T19:05:57.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f5XtEMNzMmAxPO/z2P/I2qwktgQ=', 'Content-Length': '0'} 2023-11-02T19:05:57.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099b178f1be333525-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.876 DEBUG:boto:path=/ibtljn-64/obj124 2023-11-02T19:05:57.877 DEBUG:boto:auth_path=/ibtljn-64/obj124 2023-11-02T19:05:57.877 DEBUG:boto:path=/ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.878 DEBUG:boto:auth_path=/ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.878 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.878 DEBUG:boto:Path: /ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.879 DEBUG:boto:Data: 2023-11-02T19:05:57.879 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.880 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.880 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.880 DEBUG:boto:Params: {} 2023-11-02T19:05:57.881 DEBUG:boto:Token: None 2023-11-02T19:05:57.881 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.882 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:f5XtEMNzMmAxPO/z2P/I2qwktgQ= 2023-11-02T19:05:57.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:f5XtEMNzMmAxPO/z2P/I2qwktgQ=', 'Content-Length': '0'} 2023-11-02T19:05:57.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e32807e1259e252d-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T19:05:57.886 DEBUG:boto:path=/ibtljn-64/obj124 2023-11-02T19:05:57.886 DEBUG:boto:auth_path=/ibtljn-64/obj124 2023-11-02T19:05:57.887 DEBUG:boto:path=/ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.887 DEBUG:boto:auth_path=/ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.888 DEBUG:boto:Method: GET 2023-11-02T19:05:57.888 DEBUG:boto:Path: /ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.889 DEBUG:boto:Data: 2023-11-02T19:05:57.889 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.889 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.890 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.890 DEBUG:boto:Params: {} 2023-11-02T19:05:57.891 DEBUG:boto:Token: None 2023-11-02T19:05:57.891 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.892 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+nJlYfjlTJ3IRvzP1FMcJ29kSUw= 2023-11-02T19:05:57.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+nJlYfjlTJ3IRvzP1FMcJ29kSUw=', 'Content-Length': '0'} 2023-11-02T19:05:57.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000661ce670ce47df00-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.896 DEBUG:boto:path=/ibtljn-64/obj124 2023-11-02T19:05:57.896 DEBUG:boto:auth_path=/ibtljn-64/obj124 2023-11-02T19:05:57.897 DEBUG:boto:path=/ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.897 DEBUG:boto:auth_path=/ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.898 DEBUG:boto:Method: GET 2023-11-02T19:05:57.898 DEBUG:boto:Path: /ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.898 DEBUG:boto:Data: 2023-11-02T19:05:57.899 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.899 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.900 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.900 DEBUG:boto:Params: {} 2023-11-02T19:05:57.901 DEBUG:boto:Token: None 2023-11-02T19:05:57.901 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj124?versionId=null 2023-11-02T19:05:57.902 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+nJlYfjlTJ3IRvzP1FMcJ29kSUw= 2023-11-02T19:05:57.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+nJlYfjlTJ3IRvzP1FMcJ29kSUw=', 'Content-Length': '0'} 2023-11-02T19:05:57.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c30ccdc46f021f36-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.950 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.951 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.951 DEBUG:boto:path=/ibtljn-64/?prefix=obj124 2023-11-02T19:05:57.952 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj124 2023-11-02T19:05:57.952 DEBUG:boto:Method: GET 2023-11-02T19:05:57.953 DEBUG:boto:Path: /ibtljn-64/?prefix=obj124 2023-11-02T19:05:57.953 DEBUG:boto:Data: 2023-11-02T19:05:57.954 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.954 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.954 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.955 DEBUG:boto:Params: {} 2023-11-02T19:05:57.956 DEBUG:boto:Token: None 2023-11-02T19:05:57.956 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.957 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4aedbec95195491-006543f315-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.963 DEBUG:boto:b'ibtljn-64obj1241000falseobj1242023-11-02T19:03:44.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.963 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:57.964 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:57.964 DEBUG:boto:path=/ibtljn-64/?prefix=obj124 2023-11-02T19:05:57.965 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj124 2023-11-02T19:05:57.965 DEBUG:boto:Method: GET 2023-11-02T19:05:57.966 DEBUG:boto:Path: /ibtljn-64/?prefix=obj124 2023-11-02T19:05:57.966 DEBUG:boto:Data: 2023-11-02T19:05:57.967 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.967 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.967 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.968 DEBUG:boto:Params: {} 2023-11-02T19:05:57.968 DEBUG:boto:Token: None 2023-11-02T19:05:57.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/ 2023-11-02T19:05:57.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U= 2023-11-02T19:05:57.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wh/fbNvvNa9PLUUOzxcoD3/zk/U=', 'Content-Length': '0'} 2023-11-02T19:05:57.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dad6cddd2d77243-006543f315-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.980 DEBUG:boto:b'ibtljn-64obj1241000falseobj1242023-11-02T19:03:44.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:57.981 DEBUG:boto:path=/ibtljn-64/obj124 2023-11-02T19:05:57.982 DEBUG:boto:auth_path=/ibtljn-64/obj124 2023-11-02T19:05:57.982 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.983 DEBUG:boto:Path: /ibtljn-64/obj124 2023-11-02T19:05:57.983 DEBUG:boto:Data: 2023-11-02T19:05:57.983 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.984 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:57.984 DEBUG:boto:Port: 8001 2023-11-02T19:05:57.985 DEBUG:boto:Params: {} 2023-11-02T19:05:57.985 DEBUG:boto:Token: None 2023-11-02T19:05:57.986 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj124 2023-11-02T19:05:57.986 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xO3qcPI+FI98/mbwODkRZgdbq8w= 2023-11-02T19:05:57.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xO3qcPI+FI98/mbwODkRZgdbq8w=', 'Content-Length': '0'} 2023-11-02T19:05:57.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d0fd08efe4bd5cb-006543f315-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.990 DEBUG:boto:path=/ibtljn-64/obj124 2023-11-02T19:05:57.990 DEBUG:boto:auth_path=/ibtljn-64/obj124 2023-11-02T19:05:57.991 DEBUG:boto:Method: HEAD 2023-11-02T19:05:57.991 DEBUG:boto:Path: /ibtljn-64/obj124 2023-11-02T19:05:57.992 DEBUG:boto:Data: 2023-11-02T19:05:57.992 DEBUG:boto:Headers: {} 2023-11-02T19:05:57.992 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:57.993 DEBUG:boto:Port: 8000 2023-11-02T19:05:57.993 DEBUG:boto:Params: {} 2023-11-02T19:05:57.994 DEBUG:boto:Token: None 2023-11-02T19:05:57.994 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:57 GMT /ibtljn-64/obj124 2023-11-02T19:05:57.995 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xO3qcPI+FI98/mbwODkRZgdbq8w= 2023-11-02T19:05:57.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xO3qcPI+FI98/mbwODkRZgdbq8w=', 'Content-Length': '0'} 2023-11-02T19:05:57.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ae82c26d5b67e96-006543f315-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:57.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-11-02T19:05:57.999 DEBUG:boto:path=/ibtljn-64/obj124 2023-11-02T19:05:58.000 DEBUG:boto:auth_path=/ibtljn-64/obj124 2023-11-02T19:05:58.000 DEBUG:boto:Method: GET 2023-11-02T19:05:58.001 DEBUG:boto:Path: /ibtljn-64/obj124 2023-11-02T19:05:58.001 DEBUG:boto:Data: 2023-11-02T19:05:58.002 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.002 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.003 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.003 DEBUG:boto:Params: {} 2023-11-02T19:05:58.004 DEBUG:boto:Token: None 2023-11-02T19:05:58.004 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj124 2023-11-02T19:05:58.005 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xwviJ4I2wynak44XOeq13ItSTcA= 2023-11-02T19:05:58.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xwviJ4I2wynak44XOeq13ItSTcA=', 'Content-Length': '0'} 2023-11-02T19:05:58.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c3fc7150d500c2a-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.050 DEBUG:boto:path=/ibtljn-64/obj124 2023-11-02T19:05:58.051 DEBUG:boto:auth_path=/ibtljn-64/obj124 2023-11-02T19:05:58.052 DEBUG:boto:Method: GET 2023-11-02T19:05:58.052 DEBUG:boto:Path: /ibtljn-64/obj124 2023-11-02T19:05:58.052 DEBUG:boto:Data: 2023-11-02T19:05:58.053 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.053 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.054 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.054 DEBUG:boto:Params: {} 2023-11-02T19:05:58.055 DEBUG:boto:Token: None 2023-11-02T19:05:58.055 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj124 2023-11-02T19:05:58.056 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xwviJ4I2wynak44XOeq13ItSTcA= 2023-11-02T19:05:58.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xwviJ4I2wynak44XOeq13ItSTcA=', 'Content-Length': '0'} 2023-11-02T19:05:58.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000974667d0c17c5c20-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.060 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.061 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.061 DEBUG:boto:path=/ibtljn-64/?prefix=obj124 2023-11-02T19:05:58.061 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj124 2023-11-02T19:05:58.062 DEBUG:boto:Method: GET 2023-11-02T19:05:58.062 DEBUG:boto:Path: /ibtljn-64/?prefix=obj124 2023-11-02T19:05:58.063 DEBUG:boto:Data: 2023-11-02T19:05:58.063 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.064 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.064 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.065 DEBUG:boto:Params: {} 2023-11-02T19:05:58.065 DEBUG:boto:Token: None 2023-11-02T19:05:58.066 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.066 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c348ec4343fded2a-006543f316-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.072 DEBUG:boto:b'ibtljn-64obj1241000falseobj1242023-11-02T19:03:44.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.073 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.073 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.074 DEBUG:boto:path=/ibtljn-64/?prefix=obj124 2023-11-02T19:05:58.074 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj124 2023-11-02T19:05:58.075 DEBUG:boto:Method: GET 2023-11-02T19:05:58.075 DEBUG:boto:Path: /ibtljn-64/?prefix=obj124 2023-11-02T19:05:58.076 DEBUG:boto:Data: 2023-11-02T19:05:58.076 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.076 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.077 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.077 DEBUG:boto:Params: {} 2023-11-02T19:05:58.078 DEBUG:boto:Token: None 2023-11-02T19:05:58.078 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.079 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e423d46c4812675d-006543f316-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.130 DEBUG:boto:b'ibtljn-64obj1241000falseobj1242023-11-02T19:03:44.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T19:05:58.132 DEBUG:boto:path=/ibtljn-64/obj145 2023-11-02T19:05:58.132 DEBUG:boto:auth_path=/ibtljn-64/obj145 2023-11-02T19:05:58.133 DEBUG:boto:path=/ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.133 DEBUG:boto:auth_path=/ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.134 DEBUG:boto:Method: GET 2023-11-02T19:05:58.135 DEBUG:boto:Path: /ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.135 DEBUG:boto:Data: 2023-11-02T19:05:58.136 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.136 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.137 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.137 DEBUG:boto:Params: {} 2023-11-02T19:05:58.137 DEBUG:boto:Token: None 2023-11-02T19:05:58.138 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.139 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IwOuuiDZ4pPxWu5Q5hbHCKQZXYk= 2023-11-02T19:05:58.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IwOuuiDZ4pPxWu5Q5hbHCKQZXYk=', 'Content-Length': '0'} 2023-11-02T19:05:58.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006875e68c87103d81-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.143 DEBUG:boto:path=/ibtljn-64/obj145 2023-11-02T19:05:58.144 DEBUG:boto:auth_path=/ibtljn-64/obj145 2023-11-02T19:05:58.144 DEBUG:boto:path=/ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.145 DEBUG:boto:auth_path=/ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.145 DEBUG:boto:Method: GET 2023-11-02T19:05:58.145 DEBUG:boto:Path: /ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.146 DEBUG:boto:Data: 2023-11-02T19:05:58.146 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.147 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.147 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.148 DEBUG:boto:Params: {} 2023-11-02T19:05:58.148 DEBUG:boto:Token: None 2023-11-02T19:05:58.149 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.149 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IwOuuiDZ4pPxWu5Q5hbHCKQZXYk= 2023-11-02T19:05:58.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IwOuuiDZ4pPxWu5Q5hbHCKQZXYk=', 'Content-Length': '0'} 2023-11-02T19:05:58.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c4aaca503f00e04-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.153 DEBUG:boto:path=/ibtljn-64/obj145 2023-11-02T19:05:58.154 DEBUG:boto:auth_path=/ibtljn-64/obj145 2023-11-02T19:05:58.154 DEBUG:boto:path=/ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.155 DEBUG:boto:auth_path=/ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.155 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.155 DEBUG:boto:Path: /ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.156 DEBUG:boto:Data: 2023-11-02T19:05:58.156 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.157 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.157 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.157 DEBUG:boto:Params: {} 2023-11-02T19:05:58.158 DEBUG:boto:Token: None 2023-11-02T19:05:58.158 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.159 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TPkOle2zsFT5ImQXGGWzWUEDMQ8= 2023-11-02T19:05:58.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TPkOle2zsFT5ImQXGGWzWUEDMQ8=', 'Content-Length': '0'} 2023-11-02T19:05:58.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000850c649bcd12fb76-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.163 DEBUG:boto:path=/ibtljn-64/obj145 2023-11-02T19:05:58.163 DEBUG:boto:auth_path=/ibtljn-64/obj145 2023-11-02T19:05:58.164 DEBUG:boto:path=/ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.164 DEBUG:boto:auth_path=/ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.165 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.165 DEBUG:boto:Path: /ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.165 DEBUG:boto:Data: 2023-11-02T19:05:58.166 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.166 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.167 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.167 DEBUG:boto:Params: {} 2023-11-02T19:05:58.168 DEBUG:boto:Token: None 2023-11-02T19:05:58.168 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.169 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:TPkOle2zsFT5ImQXGGWzWUEDMQ8= 2023-11-02T19:05:58.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TPkOle2zsFT5ImQXGGWzWUEDMQ8=', 'Content-Length': '0'} 2023-11-02T19:05:58.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f437b8f59bc4e41a-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T19:05:58.173 DEBUG:boto:path=/ibtljn-64/obj145 2023-11-02T19:05:58.173 DEBUG:boto:auth_path=/ibtljn-64/obj145 2023-11-02T19:05:58.174 DEBUG:boto:path=/ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.174 DEBUG:boto:auth_path=/ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.175 DEBUG:boto:Method: GET 2023-11-02T19:05:58.175 DEBUG:boto:Path: /ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.176 DEBUG:boto:Data: 2023-11-02T19:05:58.176 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.177 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.177 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.177 DEBUG:boto:Params: {} 2023-11-02T19:05:58.178 DEBUG:boto:Token: None 2023-11-02T19:05:58.179 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.179 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IwOuuiDZ4pPxWu5Q5hbHCKQZXYk= 2023-11-02T19:05:58.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IwOuuiDZ4pPxWu5Q5hbHCKQZXYk=', 'Content-Length': '0'} 2023-11-02T19:05:58.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccc5a161d2b92edc-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.226 DEBUG:boto:path=/ibtljn-64/obj145 2023-11-02T19:05:58.227 DEBUG:boto:auth_path=/ibtljn-64/obj145 2023-11-02T19:05:58.227 DEBUG:boto:path=/ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.228 DEBUG:boto:auth_path=/ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.228 DEBUG:boto:Method: GET 2023-11-02T19:05:58.229 DEBUG:boto:Path: /ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.229 DEBUG:boto:Data: 2023-11-02T19:05:58.230 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.230 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.231 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.231 DEBUG:boto:Params: {} 2023-11-02T19:05:58.232 DEBUG:boto:Token: None 2023-11-02T19:05:58.232 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj145?versionId=null 2023-11-02T19:05:58.233 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IwOuuiDZ4pPxWu5Q5hbHCKQZXYk= 2023-11-02T19:05:58.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IwOuuiDZ4pPxWu5Q5hbHCKQZXYk=', 'Content-Length': '0'} 2023-11-02T19:05:58.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d73a5ab5934c8638-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.236 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.237 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.237 DEBUG:boto:path=/ibtljn-64/?prefix=obj145 2023-11-02T19:05:58.238 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj145 2023-11-02T19:05:58.238 DEBUG:boto:Method: GET 2023-11-02T19:05:58.239 DEBUG:boto:Path: /ibtljn-64/?prefix=obj145 2023-11-02T19:05:58.239 DEBUG:boto:Data: 2023-11-02T19:05:58.239 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.240 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.240 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.241 DEBUG:boto:Params: {} 2023-11-02T19:05:58.241 DEBUG:boto:Token: None 2023-11-02T19:05:58.242 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038c732a3d87f4cf1-006543f316-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.247 DEBUG:boto:b'ibtljn-64obj1451000falseobj1452023-11-02T19:03:43.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.248 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.248 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.249 DEBUG:boto:path=/ibtljn-64/?prefix=obj145 2023-11-02T19:05:58.249 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj145 2023-11-02T19:05:58.250 DEBUG:boto:Method: GET 2023-11-02T19:05:58.250 DEBUG:boto:Path: /ibtljn-64/?prefix=obj145 2023-11-02T19:05:58.251 DEBUG:boto:Data: 2023-11-02T19:05:58.251 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.251 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.252 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.252 DEBUG:boto:Params: {} 2023-11-02T19:05:58.253 DEBUG:boto:Token: None 2023-11-02T19:05:58.253 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.254 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e515aedbd0c29b10-006543f316-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.263 DEBUG:boto:b'ibtljn-64obj1451000falseobj1452023-11-02T19:03:43.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.263 DEBUG:boto:path=/ibtljn-64/obj145 2023-11-02T19:05:58.264 DEBUG:boto:auth_path=/ibtljn-64/obj145 2023-11-02T19:05:58.264 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.265 DEBUG:boto:Path: /ibtljn-64/obj145 2023-11-02T19:05:58.265 DEBUG:boto:Data: 2023-11-02T19:05:58.266 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.266 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.266 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.267 DEBUG:boto:Params: {} 2023-11-02T19:05:58.267 DEBUG:boto:Token: None 2023-11-02T19:05:58.268 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj145 2023-11-02T19:05:58.268 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:617lk7VLdoRrtFF/1dt9pr7BaZM= 2023-11-02T19:05:58.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:617lk7VLdoRrtFF/1dt9pr7BaZM=', 'Content-Length': '0'} 2023-11-02T19:05:58.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff174d37a943d1fb-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.272 DEBUG:boto:path=/ibtljn-64/obj145 2023-11-02T19:05:58.272 DEBUG:boto:auth_path=/ibtljn-64/obj145 2023-11-02T19:05:58.273 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.273 DEBUG:boto:Path: /ibtljn-64/obj145 2023-11-02T19:05:58.274 DEBUG:boto:Data: 2023-11-02T19:05:58.274 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.275 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.275 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.276 DEBUG:boto:Params: {} 2023-11-02T19:05:58.276 DEBUG:boto:Token: None 2023-11-02T19:05:58.277 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj145 2023-11-02T19:05:58.277 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:617lk7VLdoRrtFF/1dt9pr7BaZM= 2023-11-02T19:05:58.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:617lk7VLdoRrtFF/1dt9pr7BaZM=', 'Content-Length': '0'} 2023-11-02T19:05:58.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045192392d6798d51-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-11-02T19:05:58.281 DEBUG:boto:path=/ibtljn-64/obj145 2023-11-02T19:05:58.282 DEBUG:boto:auth_path=/ibtljn-64/obj145 2023-11-02T19:05:58.282 DEBUG:boto:Method: GET 2023-11-02T19:05:58.283 DEBUG:boto:Path: /ibtljn-64/obj145 2023-11-02T19:05:58.283 DEBUG:boto:Data: 2023-11-02T19:05:58.284 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.284 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.285 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.285 DEBUG:boto:Params: {} 2023-11-02T19:05:58.286 DEBUG:boto:Token: None 2023-11-02T19:05:58.286 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj145 2023-11-02T19:05:58.287 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VK6pdC267bhSxkbG6t8hy/WHx90= 2023-11-02T19:05:58.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VK6pdC267bhSxkbG6t8hy/WHx90=', 'Content-Length': '0'} 2023-11-02T19:05:58.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c888b127ea2f95a-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.330 DEBUG:boto:path=/ibtljn-64/obj145 2023-11-02T19:05:58.331 DEBUG:boto:auth_path=/ibtljn-64/obj145 2023-11-02T19:05:58.331 DEBUG:boto:Method: GET 2023-11-02T19:05:58.332 DEBUG:boto:Path: /ibtljn-64/obj145 2023-11-02T19:05:58.332 DEBUG:boto:Data: 2023-11-02T19:05:58.333 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.333 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.334 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.334 DEBUG:boto:Params: {} 2023-11-02T19:05:58.335 DEBUG:boto:Token: None 2023-11-02T19:05:58.335 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj145 2023-11-02T19:05:58.336 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VK6pdC267bhSxkbG6t8hy/WHx90= 2023-11-02T19:05:58.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VK6pdC267bhSxkbG6t8hy/WHx90=', 'Content-Length': '0'} 2023-11-02T19:05:58.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a1f9b5c99a59775-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.340 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.340 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.341 DEBUG:boto:path=/ibtljn-64/?prefix=obj145 2023-11-02T19:05:58.341 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj145 2023-11-02T19:05:58.342 DEBUG:boto:Method: GET 2023-11-02T19:05:58.342 DEBUG:boto:Path: /ibtljn-64/?prefix=obj145 2023-11-02T19:05:58.343 DEBUG:boto:Data: 2023-11-02T19:05:58.343 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.343 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.344 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.344 DEBUG:boto:Params: {} 2023-11-02T19:05:58.345 DEBUG:boto:Token: None 2023-11-02T19:05:58.345 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.346 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000695479612c896d52-006543f316-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.352 DEBUG:boto:b'ibtljn-64obj1451000falseobj1452023-11-02T19:03:43.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.353 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.353 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.354 DEBUG:boto:path=/ibtljn-64/?prefix=obj145 2023-11-02T19:05:58.354 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj145 2023-11-02T19:05:58.355 DEBUG:boto:Method: GET 2023-11-02T19:05:58.355 DEBUG:boto:Path: /ibtljn-64/?prefix=obj145 2023-11-02T19:05:58.356 DEBUG:boto:Data: 2023-11-02T19:05:58.356 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.357 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.357 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.357 DEBUG:boto:Params: {} 2023-11-02T19:05:58.358 DEBUG:boto:Token: None 2023-11-02T19:05:58.358 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.359 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5aabab9a0ea51c6-006543f316-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.369 DEBUG:boto:b'ibtljn-64obj1451000falseobj1452023-11-02T19:03:43.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-11-02T19:05:58.370 DEBUG:boto:path=/ibtljn-64/obj148 2023-11-02T19:05:58.370 DEBUG:boto:auth_path=/ibtljn-64/obj148 2023-11-02T19:05:58.371 DEBUG:boto:path=/ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.371 DEBUG:boto:auth_path=/ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.372 DEBUG:boto:Method: GET 2023-11-02T19:05:58.372 DEBUG:boto:Path: /ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.373 DEBUG:boto:Data: 2023-11-02T19:05:58.373 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.374 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.374 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.375 DEBUG:boto:Params: {} 2023-11-02T19:05:58.375 DEBUG:boto:Token: None 2023-11-02T19:05:58.376 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.376 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LCQARt0zf2f7Y8RSnBUGIeS/OJc= 2023-11-02T19:05:58.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LCQARt0zf2f7Y8RSnBUGIeS/OJc=', 'Content-Length': '0'} 2023-11-02T19:05:58.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096926b417337eb6c-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.380 DEBUG:boto:path=/ibtljn-64/obj148 2023-11-02T19:05:58.380 DEBUG:boto:auth_path=/ibtljn-64/obj148 2023-11-02T19:05:58.381 DEBUG:boto:path=/ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.381 DEBUG:boto:auth_path=/ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.382 DEBUG:boto:Method: GET 2023-11-02T19:05:58.382 DEBUG:boto:Path: /ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.383 DEBUG:boto:Data: 2023-11-02T19:05:58.383 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.383 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.384 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.384 DEBUG:boto:Params: {} 2023-11-02T19:05:58.385 DEBUG:boto:Token: None 2023-11-02T19:05:58.385 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.386 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LCQARt0zf2f7Y8RSnBUGIeS/OJc= 2023-11-02T19:05:58.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LCQARt0zf2f7Y8RSnBUGIeS/OJc=', 'Content-Length': '0'} 2023-11-02T19:05:58.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fd8568e88d4a510-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.390 DEBUG:boto:path=/ibtljn-64/obj148 2023-11-02T19:05:58.390 DEBUG:boto:auth_path=/ibtljn-64/obj148 2023-11-02T19:05:58.391 DEBUG:boto:path=/ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.391 DEBUG:boto:auth_path=/ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.391 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.392 DEBUG:boto:Path: /ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.392 DEBUG:boto:Data: 2023-11-02T19:05:58.393 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.393 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.394 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.394 DEBUG:boto:Params: {} 2023-11-02T19:05:58.394 DEBUG:boto:Token: None 2023-11-02T19:05:58.395 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.395 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:leepQ4cQCfxeaOXEFv/GXKA6Gno= 2023-11-02T19:05:58.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:leepQ4cQCfxeaOXEFv/GXKA6Gno=', 'Content-Length': '0'} 2023-11-02T19:05:58.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a952fa2a6809b72-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.399 DEBUG:boto:path=/ibtljn-64/obj148 2023-11-02T19:05:58.399 DEBUG:boto:auth_path=/ibtljn-64/obj148 2023-11-02T19:05:58.400 DEBUG:boto:path=/ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.400 DEBUG:boto:auth_path=/ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.401 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.401 DEBUG:boto:Path: /ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.402 DEBUG:boto:Data: 2023-11-02T19:05:58.402 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.403 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.403 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.403 DEBUG:boto:Params: {} 2023-11-02T19:05:58.404 DEBUG:boto:Token: None 2023-11-02T19:05:58.404 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.405 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:leepQ4cQCfxeaOXEFv/GXKA6Gno= 2023-11-02T19:05:58.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:leepQ4cQCfxeaOXEFv/GXKA6Gno=', 'Content-Length': '0'} 2023-11-02T19:05:58.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f740e293acc18e1-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-11-02T19:05:58.409 DEBUG:boto:path=/ibtljn-64/obj148 2023-11-02T19:05:58.410 DEBUG:boto:auth_path=/ibtljn-64/obj148 2023-11-02T19:05:58.410 DEBUG:boto:path=/ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.411 DEBUG:boto:auth_path=/ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.411 DEBUG:boto:Method: GET 2023-11-02T19:05:58.412 DEBUG:boto:Path: /ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.412 DEBUG:boto:Data: 2023-11-02T19:05:58.412 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.413 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.413 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.414 DEBUG:boto:Params: {} 2023-11-02T19:05:58.414 DEBUG:boto:Token: None 2023-11-02T19:05:58.415 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.415 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LCQARt0zf2f7Y8RSnBUGIeS/OJc= 2023-11-02T19:05:58.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LCQARt0zf2f7Y8RSnBUGIeS/OJc=', 'Content-Length': '0'} 2023-11-02T19:05:58.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6931dd84ecfd6ab-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.462 DEBUG:boto:path=/ibtljn-64/obj148 2023-11-02T19:05:58.463 DEBUG:boto:auth_path=/ibtljn-64/obj148 2023-11-02T19:05:58.464 DEBUG:boto:path=/ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.464 DEBUG:boto:auth_path=/ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.464 DEBUG:boto:Method: GET 2023-11-02T19:05:58.465 DEBUG:boto:Path: /ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.465 DEBUG:boto:Data: 2023-11-02T19:05:58.466 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.466 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.467 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.467 DEBUG:boto:Params: {} 2023-11-02T19:05:58.468 DEBUG:boto:Token: None 2023-11-02T19:05:58.468 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj148?versionId=null 2023-11-02T19:05:58.469 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LCQARt0zf2f7Y8RSnBUGIeS/OJc= 2023-11-02T19:05:58.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LCQARt0zf2f7Y8RSnBUGIeS/OJc=', 'Content-Length': '0'} 2023-11-02T19:05:58.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbb3be0f3c39acb4-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.473 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.473 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.474 DEBUG:boto:path=/ibtljn-64/?prefix=obj148 2023-11-02T19:05:58.474 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj148 2023-11-02T19:05:58.475 DEBUG:boto:Method: GET 2023-11-02T19:05:58.475 DEBUG:boto:Path: /ibtljn-64/?prefix=obj148 2023-11-02T19:05:58.475 DEBUG:boto:Data: 2023-11-02T19:05:58.476 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.476 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.477 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.477 DEBUG:boto:Params: {} 2023-11-02T19:05:58.478 DEBUG:boto:Token: None 2023-11-02T19:05:58.478 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.479 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb56972856652ebd-006543f316-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.485 DEBUG:boto:b'ibtljn-64obj1481000falseobj1482023-11-02T19:03:44.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.486 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.487 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.487 DEBUG:boto:path=/ibtljn-64/?prefix=obj148 2023-11-02T19:05:58.487 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj148 2023-11-02T19:05:58.488 DEBUG:boto:Method: GET 2023-11-02T19:05:58.488 DEBUG:boto:Path: /ibtljn-64/?prefix=obj148 2023-11-02T19:05:58.489 DEBUG:boto:Data: 2023-11-02T19:05:58.489 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.490 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.490 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.491 DEBUG:boto:Params: {} 2023-11-02T19:05:58.491 DEBUG:boto:Token: None 2023-11-02T19:05:58.492 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.492 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003cb68bc2f64bfe2-006543f316-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.550 DEBUG:boto:b'ibtljn-64obj1481000falseobj1482023-11-02T19:03:44.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.551 DEBUG:boto:path=/ibtljn-64/obj148 2023-11-02T19:05:58.552 DEBUG:boto:auth_path=/ibtljn-64/obj148 2023-11-02T19:05:58.552 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.553 DEBUG:boto:Path: /ibtljn-64/obj148 2023-11-02T19:05:58.553 DEBUG:boto:Data: 2023-11-02T19:05:58.553 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.554 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.554 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.555 DEBUG:boto:Params: {} 2023-11-02T19:05:58.555 DEBUG:boto:Token: None 2023-11-02T19:05:58.556 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj148 2023-11-02T19:05:58.556 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JIp18MTU5kX4sPNwc8PVjZHO0BI= 2023-11-02T19:05:58.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JIp18MTU5kX4sPNwc8PVjZHO0BI=', 'Content-Length': '0'} 2023-11-02T19:05:58.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd7283f05ca10488-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.560 DEBUG:boto:path=/ibtljn-64/obj148 2023-11-02T19:05:58.561 DEBUG:boto:auth_path=/ibtljn-64/obj148 2023-11-02T19:05:58.561 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.562 DEBUG:boto:Path: /ibtljn-64/obj148 2023-11-02T19:05:58.562 DEBUG:boto:Data: 2023-11-02T19:05:58.563 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.563 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.563 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.564 DEBUG:boto:Params: {} 2023-11-02T19:05:58.564 DEBUG:boto:Token: None 2023-11-02T19:05:58.565 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj148 2023-11-02T19:05:58.565 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JIp18MTU5kX4sPNwc8PVjZHO0BI= 2023-11-02T19:05:58.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JIp18MTU5kX4sPNwc8PVjZHO0BI=', 'Content-Length': '0'} 2023-11-02T19:05:58.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff7d73ed0ca3a1f-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-11-02T19:05:58.570 DEBUG:boto:path=/ibtljn-64/obj148 2023-11-02T19:05:58.570 DEBUG:boto:auth_path=/ibtljn-64/obj148 2023-11-02T19:05:58.571 DEBUG:boto:Method: GET 2023-11-02T19:05:58.571 DEBUG:boto:Path: /ibtljn-64/obj148 2023-11-02T19:05:58.572 DEBUG:boto:Data: 2023-11-02T19:05:58.572 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.573 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.573 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.573 DEBUG:boto:Params: {} 2023-11-02T19:05:58.574 DEBUG:boto:Token: None 2023-11-02T19:05:58.575 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj148 2023-11-02T19:05:58.575 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RQ9hcOLuRLzWqHQW6VuOfpb/bug= 2023-11-02T19:05:58.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RQ9hcOLuRLzWqHQW6VuOfpb/bug=', 'Content-Length': '0'} 2023-11-02T19:05:58.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001a5ce4ad27e045f-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.579 DEBUG:boto:path=/ibtljn-64/obj148 2023-11-02T19:05:58.580 DEBUG:boto:auth_path=/ibtljn-64/obj148 2023-11-02T19:05:58.581 DEBUG:boto:Method: GET 2023-11-02T19:05:58.581 DEBUG:boto:Path: /ibtljn-64/obj148 2023-11-02T19:05:58.581 DEBUG:boto:Data: 2023-11-02T19:05:58.582 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.582 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.583 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.583 DEBUG:boto:Params: {} 2023-11-02T19:05:58.584 DEBUG:boto:Token: None 2023-11-02T19:05:58.584 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj148 2023-11-02T19:05:58.585 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RQ9hcOLuRLzWqHQW6VuOfpb/bug= 2023-11-02T19:05:58.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RQ9hcOLuRLzWqHQW6VuOfpb/bug=', 'Content-Length': '0'} 2023-11-02T19:05:58.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cdb00a548a3f41f-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.589 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.589 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.590 DEBUG:boto:path=/ibtljn-64/?prefix=obj148 2023-11-02T19:05:58.590 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj148 2023-11-02T19:05:58.590 DEBUG:boto:Method: GET 2023-11-02T19:05:58.591 DEBUG:boto:Path: /ibtljn-64/?prefix=obj148 2023-11-02T19:05:58.591 DEBUG:boto:Data: 2023-11-02T19:05:58.592 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.592 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.592 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.593 DEBUG:boto:Params: {} 2023-11-02T19:05:58.593 DEBUG:boto:Token: None 2023-11-02T19:05:58.594 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.594 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009044d0e9eace812a-006543f316-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.642 DEBUG:boto:b'ibtljn-64obj1481000falseobj1482023-11-02T19:03:44.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.643 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.644 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.644 DEBUG:boto:path=/ibtljn-64/?prefix=obj148 2023-11-02T19:05:58.645 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj148 2023-11-02T19:05:58.645 DEBUG:boto:Method: GET 2023-11-02T19:05:58.645 DEBUG:boto:Path: /ibtljn-64/?prefix=obj148 2023-11-02T19:05:58.646 DEBUG:boto:Data: 2023-11-02T19:05:58.646 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.647 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.647 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.648 DEBUG:boto:Params: {} 2023-11-02T19:05:58.648 DEBUG:boto:Token: None 2023-11-02T19:05:58.649 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.649 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0aeb364584fbf81-006543f316-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.660 DEBUG:boto:b'ibtljn-64obj1481000falseobj1482023-11-02T19:03:44.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T19:05:58.661 DEBUG:boto:path=/ibtljn-64/obj155 2023-11-02T19:05:58.662 DEBUG:boto:auth_path=/ibtljn-64/obj155 2023-11-02T19:05:58.662 DEBUG:boto:path=/ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.663 DEBUG:boto:auth_path=/ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.663 DEBUG:boto:Method: GET 2023-11-02T19:05:58.664 DEBUG:boto:Path: /ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.664 DEBUG:boto:Data: 2023-11-02T19:05:58.665 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.665 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.665 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.666 DEBUG:boto:Params: {} 2023-11-02T19:05:58.666 DEBUG:boto:Token: None 2023-11-02T19:05:58.667 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.667 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RBABMjle4L7d8izn6vVFtGU6G6E= 2023-11-02T19:05:58.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RBABMjle4L7d8izn6vVFtGU6G6E=', 'Content-Length': '0'} 2023-11-02T19:05:58.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007debd1c59284bea4-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.671 DEBUG:boto:path=/ibtljn-64/obj155 2023-11-02T19:05:58.671 DEBUG:boto:auth_path=/ibtljn-64/obj155 2023-11-02T19:05:58.672 DEBUG:boto:path=/ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.672 DEBUG:boto:auth_path=/ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.673 DEBUG:boto:Method: GET 2023-11-02T19:05:58.673 DEBUG:boto:Path: /ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.674 DEBUG:boto:Data: 2023-11-02T19:05:58.674 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.675 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.675 DEBUG:boto:Params: {} 2023-11-02T19:05:58.676 DEBUG:boto:Token: None 2023-11-02T19:05:58.676 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.677 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RBABMjle4L7d8izn6vVFtGU6G6E= 2023-11-02T19:05:58.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RBABMjle4L7d8izn6vVFtGU6G6E=', 'Content-Length': '0'} 2023-11-02T19:05:58.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000868bb83463a83094-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.683 DEBUG:boto:path=/ibtljn-64/obj155 2023-11-02T19:05:58.683 DEBUG:boto:auth_path=/ibtljn-64/obj155 2023-11-02T19:05:58.683 DEBUG:boto:path=/ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.684 DEBUG:boto:auth_path=/ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.684 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.685 DEBUG:boto:Path: /ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.685 DEBUG:boto:Data: 2023-11-02T19:05:58.686 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.686 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.687 DEBUG:boto:Params: {} 2023-11-02T19:05:58.687 DEBUG:boto:Token: None 2023-11-02T19:05:58.688 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.688 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PUqF16qgg6YzM1qUprtj+brcIOo= 2023-11-02T19:05:58.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PUqF16qgg6YzM1qUprtj+brcIOo=', 'Content-Length': '0'} 2023-11-02T19:05:58.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd338bad51639c09-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.691 DEBUG:boto:path=/ibtljn-64/obj155 2023-11-02T19:05:58.692 DEBUG:boto:auth_path=/ibtljn-64/obj155 2023-11-02T19:05:58.692 DEBUG:boto:path=/ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.693 DEBUG:boto:auth_path=/ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.693 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.693 DEBUG:boto:Path: /ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.694 DEBUG:boto:Data: 2023-11-02T19:05:58.694 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.695 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.696 DEBUG:boto:Params: {} 2023-11-02T19:05:58.696 DEBUG:boto:Token: None 2023-11-02T19:05:58.697 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.697 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PUqF16qgg6YzM1qUprtj+brcIOo= 2023-11-02T19:05:58.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PUqF16qgg6YzM1qUprtj+brcIOo=', 'Content-Length': '0'} 2023-11-02T19:05:58.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bfeb07a8bef69ed-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T19:05:58.701 DEBUG:boto:path=/ibtljn-64/obj155 2023-11-02T19:05:58.701 DEBUG:boto:auth_path=/ibtljn-64/obj155 2023-11-02T19:05:58.701 DEBUG:boto:path=/ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.702 DEBUG:boto:auth_path=/ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.702 DEBUG:boto:Method: GET 2023-11-02T19:05:58.703 DEBUG:boto:Path: /ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.703 DEBUG:boto:Data: 2023-11-02T19:05:58.704 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.704 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.704 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.705 DEBUG:boto:Params: {} 2023-11-02T19:05:58.705 DEBUG:boto:Token: None 2023-11-02T19:05:58.706 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.706 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RBABMjle4L7d8izn6vVFtGU6G6E= 2023-11-02T19:05:58.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RBABMjle4L7d8izn6vVFtGU6G6E=', 'Content-Length': '0'} 2023-11-02T19:05:58.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c768ec85d5600c1e-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.750 DEBUG:boto:path=/ibtljn-64/obj155 2023-11-02T19:05:58.751 DEBUG:boto:auth_path=/ibtljn-64/obj155 2023-11-02T19:05:58.752 DEBUG:boto:path=/ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.752 DEBUG:boto:auth_path=/ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.752 DEBUG:boto:Method: GET 2023-11-02T19:05:58.753 DEBUG:boto:Path: /ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.753 DEBUG:boto:Data: 2023-11-02T19:05:58.754 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.755 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.755 DEBUG:boto:Params: {} 2023-11-02T19:05:58.756 DEBUG:boto:Token: None 2023-11-02T19:05:58.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj155?versionId=null 2023-11-02T19:05:58.757 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RBABMjle4L7d8izn6vVFtGU6G6E= 2023-11-02T19:05:58.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RBABMjle4L7d8izn6vVFtGU6G6E=', 'Content-Length': '0'} 2023-11-02T19:05:58.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bae31a4c3c33dbe-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.762 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.762 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.763 DEBUG:boto:path=/ibtljn-64/?prefix=obj155 2023-11-02T19:05:58.763 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj155 2023-11-02T19:05:58.764 DEBUG:boto:Method: GET 2023-11-02T19:05:58.764 DEBUG:boto:Path: /ibtljn-64/?prefix=obj155 2023-11-02T19:05:58.765 DEBUG:boto:Data: 2023-11-02T19:05:58.765 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.766 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.766 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.766 DEBUG:boto:Params: {} 2023-11-02T19:05:58.767 DEBUG:boto:Token: None 2023-11-02T19:05:58.767 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.768 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea0b1b355625be85-006543f316-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.774 DEBUG:boto:b'ibtljn-64obj1551000falseobj1552023-11-02T19:03:44.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.775 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.775 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.776 DEBUG:boto:path=/ibtljn-64/?prefix=obj155 2023-11-02T19:05:58.777 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj155 2023-11-02T19:05:58.777 DEBUG:boto:Method: GET 2023-11-02T19:05:58.777 DEBUG:boto:Path: /ibtljn-64/?prefix=obj155 2023-11-02T19:05:58.778 DEBUG:boto:Data: 2023-11-02T19:05:58.778 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.779 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.780 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.780 DEBUG:boto:Params: {} 2023-11-02T19:05:58.781 DEBUG:boto:Token: None 2023-11-02T19:05:58.781 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.782 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c84d06b950b9010-006543f316-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.792 DEBUG:boto:b'ibtljn-64obj1551000falseobj1552023-11-02T19:03:44.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.793 DEBUG:boto:path=/ibtljn-64/obj155 2023-11-02T19:05:58.793 DEBUG:boto:auth_path=/ibtljn-64/obj155 2023-11-02T19:05:58.793 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.794 DEBUG:boto:Path: /ibtljn-64/obj155 2023-11-02T19:05:58.794 DEBUG:boto:Data: 2023-11-02T19:05:58.795 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.795 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.795 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.796 DEBUG:boto:Params: {} 2023-11-02T19:05:58.796 DEBUG:boto:Token: None 2023-11-02T19:05:58.797 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj155 2023-11-02T19:05:58.797 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BspNmnGaLvaInjbf5myeyKu0tag= 2023-11-02T19:05:58.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BspNmnGaLvaInjbf5myeyKu0tag=', 'Content-Length': '0'} 2023-11-02T19:05:58.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa173b160df61204-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.801 DEBUG:boto:path=/ibtljn-64/obj155 2023-11-02T19:05:58.801 DEBUG:boto:auth_path=/ibtljn-64/obj155 2023-11-02T19:05:58.802 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.802 DEBUG:boto:Path: /ibtljn-64/obj155 2023-11-02T19:05:58.803 DEBUG:boto:Data: 2023-11-02T19:05:58.803 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.804 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.804 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.804 DEBUG:boto:Params: {} 2023-11-02T19:05:58.805 DEBUG:boto:Token: None 2023-11-02T19:05:58.805 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj155 2023-11-02T19:05:58.806 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:BspNmnGaLvaInjbf5myeyKu0tag= 2023-11-02T19:05:58.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BspNmnGaLvaInjbf5myeyKu0tag=', 'Content-Length': '0'} 2023-11-02T19:05:58.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8c0675c1c856369-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-11-02T19:05:58.810 DEBUG:boto:path=/ibtljn-64/obj155 2023-11-02T19:05:58.810 DEBUG:boto:auth_path=/ibtljn-64/obj155 2023-11-02T19:05:58.811 DEBUG:boto:Method: GET 2023-11-02T19:05:58.811 DEBUG:boto:Path: /ibtljn-64/obj155 2023-11-02T19:05:58.811 DEBUG:boto:Data: 2023-11-02T19:05:58.812 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.812 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.813 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.813 DEBUG:boto:Params: {} 2023-11-02T19:05:58.814 DEBUG:boto:Token: None 2023-11-02T19:05:58.814 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj155 2023-11-02T19:05:58.815 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JTBBjmlbu7sr25MMripMtR15B4w= 2023-11-02T19:05:58.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JTBBjmlbu7sr25MMripMtR15B4w=', 'Content-Length': '0'} 2023-11-02T19:05:58.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecf039d9eb0f0c30-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.819 DEBUG:boto:path=/ibtljn-64/obj155 2023-11-02T19:05:58.819 DEBUG:boto:auth_path=/ibtljn-64/obj155 2023-11-02T19:05:58.820 DEBUG:boto:Method: GET 2023-11-02T19:05:58.820 DEBUG:boto:Path: /ibtljn-64/obj155 2023-11-02T19:05:58.820 DEBUG:boto:Data: 2023-11-02T19:05:58.821 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.821 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.822 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.822 DEBUG:boto:Params: {} 2023-11-02T19:05:58.823 DEBUG:boto:Token: None 2023-11-02T19:05:58.823 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj155 2023-11-02T19:05:58.824 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JTBBjmlbu7sr25MMripMtR15B4w= 2023-11-02T19:05:58.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JTBBjmlbu7sr25MMripMtR15B4w=', 'Content-Length': '0'} 2023-11-02T19:05:58.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b22694c93d4ce2ac-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.828 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.828 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.829 DEBUG:boto:path=/ibtljn-64/?prefix=obj155 2023-11-02T19:05:58.829 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj155 2023-11-02T19:05:58.829 DEBUG:boto:Method: GET 2023-11-02T19:05:58.830 DEBUG:boto:Path: /ibtljn-64/?prefix=obj155 2023-11-02T19:05:58.830 DEBUG:boto:Data: 2023-11-02T19:05:58.831 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.831 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.831 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.832 DEBUG:boto:Params: {} 2023-11-02T19:05:58.832 DEBUG:boto:Token: None 2023-11-02T19:05:58.833 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.833 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c17ca57bb6b95f27-006543f316-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.886 DEBUG:boto:b'ibtljn-64obj1551000falseobj1552023-11-02T19:03:44.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.887 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:58.888 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:58.888 DEBUG:boto:path=/ibtljn-64/?prefix=obj155 2023-11-02T19:05:58.889 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj155 2023-11-02T19:05:58.889 DEBUG:boto:Method: GET 2023-11-02T19:05:58.889 DEBUG:boto:Path: /ibtljn-64/?prefix=obj155 2023-11-02T19:05:58.890 DEBUG:boto:Data: 2023-11-02T19:05:58.890 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.891 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.891 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.892 DEBUG:boto:Params: {} 2023-11-02T19:05:58.892 DEBUG:boto:Token: None 2023-11-02T19:05:58.892 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/ 2023-11-02T19:05:58.893 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI= 2023-11-02T19:05:58.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jKC5+YAg4TqqsiOLynZY/9Z+GPI=', 'Content-Length': '0'} 2023-11-02T19:05:58.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a48ddc9328e3b178-006543f316-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.903 DEBUG:boto:b'ibtljn-64obj1551000falseobj1552023-11-02T19:03:44.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:58.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T19:05:58.904 DEBUG:boto:path=/ibtljn-64/obj17 2023-11-02T19:05:58.904 DEBUG:boto:auth_path=/ibtljn-64/obj17 2023-11-02T19:05:58.905 DEBUG:boto:path=/ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.905 DEBUG:boto:auth_path=/ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.905 DEBUG:boto:Method: GET 2023-11-02T19:05:58.906 DEBUG:boto:Path: /ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.906 DEBUG:boto:Data: 2023-11-02T19:05:58.907 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.907 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.908 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.908 DEBUG:boto:Params: {} 2023-11-02T19:05:58.908 DEBUG:boto:Token: None 2023-11-02T19:05:58.909 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.909 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:leurUcA5bSL/ihUz8mhh4hlhpWY= 2023-11-02T19:05:58.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:leurUcA5bSL/ihUz8mhh4hlhpWY=', 'Content-Length': '0'} 2023-11-02T19:05:58.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000949b2d3c5d036cf6-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.913 DEBUG:boto:path=/ibtljn-64/obj17 2023-11-02T19:05:58.914 DEBUG:boto:auth_path=/ibtljn-64/obj17 2023-11-02T19:05:58.914 DEBUG:boto:path=/ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.915 DEBUG:boto:auth_path=/ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.915 DEBUG:boto:Method: GET 2023-11-02T19:05:58.916 DEBUG:boto:Path: /ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.916 DEBUG:boto:Data: 2023-11-02T19:05:58.917 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.917 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.917 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.918 DEBUG:boto:Params: {} 2023-11-02T19:05:58.918 DEBUG:boto:Token: None 2023-11-02T19:05:58.919 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.919 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:leurUcA5bSL/ihUz8mhh4hlhpWY= 2023-11-02T19:05:58.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:leurUcA5bSL/ihUz8mhh4hlhpWY=', 'Content-Length': '0'} 2023-11-02T19:05:58.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000406244ae658e2cee-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.923 DEBUG:boto:path=/ibtljn-64/obj17 2023-11-02T19:05:58.924 DEBUG:boto:auth_path=/ibtljn-64/obj17 2023-11-02T19:05:58.924 DEBUG:boto:path=/ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.925 DEBUG:boto:auth_path=/ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.925 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.926 DEBUG:boto:Path: /ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.926 DEBUG:boto:Data: 2023-11-02T19:05:58.926 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.927 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.927 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.928 DEBUG:boto:Params: {} 2023-11-02T19:05:58.928 DEBUG:boto:Token: None 2023-11-02T19:05:58.928 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.929 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fvC/+nH4czkTP7MSyt4/3J44eWE= 2023-11-02T19:05:58.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fvC/+nH4czkTP7MSyt4/3J44eWE=', 'Content-Length': '0'} 2023-11-02T19:05:58.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060bcb0f105182422-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.933 DEBUG:boto:path=/ibtljn-64/obj17 2023-11-02T19:05:58.933 DEBUG:boto:auth_path=/ibtljn-64/obj17 2023-11-02T19:05:58.934 DEBUG:boto:path=/ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.934 DEBUG:boto:auth_path=/ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.934 DEBUG:boto:Method: HEAD 2023-11-02T19:05:58.935 DEBUG:boto:Path: /ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.935 DEBUG:boto:Data: 2023-11-02T19:05:58.936 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.936 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.936 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.937 DEBUG:boto:Params: {} 2023-11-02T19:05:58.937 DEBUG:boto:Token: None 2023-11-02T19:05:58.938 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.939 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fvC/+nH4czkTP7MSyt4/3J44eWE= 2023-11-02T19:05:58.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fvC/+nH4czkTP7MSyt4/3J44eWE=', 'Content-Length': '0'} 2023-11-02T19:05:58.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000465c431f0d0592ba-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T19:05:58.946 DEBUG:boto:path=/ibtljn-64/obj17 2023-11-02T19:05:58.947 DEBUG:boto:auth_path=/ibtljn-64/obj17 2023-11-02T19:05:58.947 DEBUG:boto:path=/ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.948 DEBUG:boto:auth_path=/ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.948 DEBUG:boto:Method: GET 2023-11-02T19:05:58.948 DEBUG:boto:Path: /ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.949 DEBUG:boto:Data: 2023-11-02T19:05:58.949 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.950 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:58.950 DEBUG:boto:Port: 8001 2023-11-02T19:05:58.951 DEBUG:boto:Params: {} 2023-11-02T19:05:58.951 DEBUG:boto:Token: None 2023-11-02T19:05:58.952 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.952 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:leurUcA5bSL/ihUz8mhh4hlhpWY= 2023-11-02T19:05:58.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:leurUcA5bSL/ihUz8mhh4hlhpWY=', 'Content-Length': '0'} 2023-11-02T19:05:58.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c84a27e3d7e2361-006543f316-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:58.957 DEBUG:boto:path=/ibtljn-64/obj17 2023-11-02T19:05:58.957 DEBUG:boto:auth_path=/ibtljn-64/obj17 2023-11-02T19:05:58.958 DEBUG:boto:path=/ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.958 DEBUG:boto:auth_path=/ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.959 DEBUG:boto:Method: GET 2023-11-02T19:05:58.959 DEBUG:boto:Path: /ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.960 DEBUG:boto:Data: 2023-11-02T19:05:58.960 DEBUG:boto:Headers: {} 2023-11-02T19:05:58.960 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:58.961 DEBUG:boto:Port: 8000 2023-11-02T19:05:58.961 DEBUG:boto:Params: {} 2023-11-02T19:05:58.962 DEBUG:boto:Token: None 2023-11-02T19:05:58.962 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:58 GMT /ibtljn-64/obj17?versionId=null 2023-11-02T19:05:58.963 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:leurUcA5bSL/ihUz8mhh4hlhpWY= 2023-11-02T19:05:58.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:leurUcA5bSL/ihUz8mhh4hlhpWY=', 'Content-Length': '0'} 2023-11-02T19:05:58.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f3408d985933335-006543f316-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.010 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.011 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.011 DEBUG:boto:path=/ibtljn-64/?prefix=obj17 2023-11-02T19:05:59.012 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj17 2023-11-02T19:05:59.012 DEBUG:boto:Method: GET 2023-11-02T19:05:59.013 DEBUG:boto:Path: /ibtljn-64/?prefix=obj17 2023-11-02T19:05:59.013 DEBUG:boto:Data: 2023-11-02T19:05:59.014 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.014 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.015 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.015 DEBUG:boto:Params: {} 2023-11-02T19:05:59.016 DEBUG:boto:Token: None 2023-11-02T19:05:59.016 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.017 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dd9c1300ecf1062-006543f317-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.023 DEBUG:boto:b'ibtljn-64obj171000falseobj172023-11-02T19:03:42.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.023 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.024 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.024 DEBUG:boto:path=/ibtljn-64/?prefix=obj17 2023-11-02T19:05:59.025 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj17 2023-11-02T19:05:59.025 DEBUG:boto:Method: GET 2023-11-02T19:05:59.026 DEBUG:boto:Path: /ibtljn-64/?prefix=obj17 2023-11-02T19:05:59.026 DEBUG:boto:Data: 2023-11-02T19:05:59.027 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.027 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.028 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.028 DEBUG:boto:Params: {} 2023-11-02T19:05:59.029 DEBUG:boto:Token: None 2023-11-02T19:05:59.029 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.030 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000033dd85b9597a1ec-006543f317-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.042 DEBUG:boto:b'ibtljn-64obj171000falseobj172023-11-02T19:03:42.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.043 DEBUG:boto:path=/ibtljn-64/obj17 2023-11-02T19:05:59.044 DEBUG:boto:auth_path=/ibtljn-64/obj17 2023-11-02T19:05:59.044 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.044 DEBUG:boto:Path: /ibtljn-64/obj17 2023-11-02T19:05:59.045 DEBUG:boto:Data: 2023-11-02T19:05:59.045 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.047 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.048 DEBUG:boto:Params: {} 2023-11-02T19:05:59.049 DEBUG:boto:Token: None 2023-11-02T19:05:59.049 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj17 2023-11-02T19:05:59.050 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xxNlraU70Sw18JkuZK4kq85OGkc= 2023-11-02T19:05:59.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xxNlraU70Sw18JkuZK4kq85OGkc=', 'Content-Length': '0'} 2023-11-02T19:05:59.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e77a49cbfd20788a-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.054 DEBUG:boto:path=/ibtljn-64/obj17 2023-11-02T19:05:59.054 DEBUG:boto:auth_path=/ibtljn-64/obj17 2023-11-02T19:05:59.055 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.055 DEBUG:boto:Path: /ibtljn-64/obj17 2023-11-02T19:05:59.056 DEBUG:boto:Data: 2023-11-02T19:05:59.056 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.056 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.057 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.057 DEBUG:boto:Params: {} 2023-11-02T19:05:59.058 DEBUG:boto:Token: None 2023-11-02T19:05:59.058 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj17 2023-11-02T19:05:59.059 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xxNlraU70Sw18JkuZK4kq85OGkc= 2023-11-02T19:05:59.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xxNlraU70Sw18JkuZK4kq85OGkc=', 'Content-Length': '0'} 2023-11-02T19:05:59.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbad8dab0fac4220-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-11-02T19:05:59.063 DEBUG:boto:path=/ibtljn-64/obj17 2023-11-02T19:05:59.064 DEBUG:boto:auth_path=/ibtljn-64/obj17 2023-11-02T19:05:59.064 DEBUG:boto:Method: GET 2023-11-02T19:05:59.065 DEBUG:boto:Path: /ibtljn-64/obj17 2023-11-02T19:05:59.065 DEBUG:boto:Data: 2023-11-02T19:05:59.066 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.066 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.067 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.067 DEBUG:boto:Params: {} 2023-11-02T19:05:59.067 DEBUG:boto:Token: None 2023-11-02T19:05:59.068 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj17 2023-11-02T19:05:59.069 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KVjivf8/FojttogPbWxE83R5QVI= 2023-11-02T19:05:59.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KVjivf8/FojttogPbWxE83R5QVI=', 'Content-Length': '0'} 2023-11-02T19:05:59.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002ae9f882220d27-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.114 DEBUG:boto:path=/ibtljn-64/obj17 2023-11-02T19:05:59.115 DEBUG:boto:auth_path=/ibtljn-64/obj17 2023-11-02T19:05:59.116 DEBUG:boto:Method: GET 2023-11-02T19:05:59.116 DEBUG:boto:Path: /ibtljn-64/obj17 2023-11-02T19:05:59.116 DEBUG:boto:Data: 2023-11-02T19:05:59.117 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.117 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.118 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.118 DEBUG:boto:Params: {} 2023-11-02T19:05:59.119 DEBUG:boto:Token: None 2023-11-02T19:05:59.119 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj17 2023-11-02T19:05:59.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KVjivf8/FojttogPbWxE83R5QVI= 2023-11-02T19:05:59.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KVjivf8/FojttogPbWxE83R5QVI=', 'Content-Length': '0'} 2023-11-02T19:05:59.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0911bf32b4325b9-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.124 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.125 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.125 DEBUG:boto:path=/ibtljn-64/?prefix=obj17 2023-11-02T19:05:59.125 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj17 2023-11-02T19:05:59.126 DEBUG:boto:Method: GET 2023-11-02T19:05:59.126 DEBUG:boto:Path: /ibtljn-64/?prefix=obj17 2023-11-02T19:05:59.127 DEBUG:boto:Data: 2023-11-02T19:05:59.127 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.128 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.128 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.129 DEBUG:boto:Params: {} 2023-11-02T19:05:59.129 DEBUG:boto:Token: None 2023-11-02T19:05:59.130 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071239b4becc48443-006543f317-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.136 DEBUG:boto:b'ibtljn-64obj171000falseobj172023-11-02T19:03:42.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.137 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.138 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.138 DEBUG:boto:path=/ibtljn-64/?prefix=obj17 2023-11-02T19:05:59.139 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj17 2023-11-02T19:05:59.139 DEBUG:boto:Method: GET 2023-11-02T19:05:59.140 DEBUG:boto:Path: /ibtljn-64/?prefix=obj17 2023-11-02T19:05:59.140 DEBUG:boto:Data: 2023-11-02T19:05:59.141 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.141 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.142 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.142 DEBUG:boto:Params: {} 2023-11-02T19:05:59.143 DEBUG:boto:Token: None 2023-11-02T19:05:59.143 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.144 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2316ec4edfbe9fa-006543f317-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.198 DEBUG:boto:b'ibtljn-64obj171000falseobj172023-11-02T19:03:42.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-11-02T19:05:59.200 DEBUG:boto:path=/ibtljn-64/obj185 2023-11-02T19:05:59.200 DEBUG:boto:auth_path=/ibtljn-64/obj185 2023-11-02T19:05:59.200 DEBUG:boto:path=/ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.201 DEBUG:boto:auth_path=/ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.201 DEBUG:boto:Method: GET 2023-11-02T19:05:59.202 DEBUG:boto:Path: /ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.202 DEBUG:boto:Data: 2023-11-02T19:05:59.203 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.203 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.204 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.204 DEBUG:boto:Params: {} 2023-11-02T19:05:59.204 DEBUG:boto:Token: None 2023-11-02T19:05:59.205 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.206 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6NKUMrpyPQtfWYNiUWPRexF8ALM= 2023-11-02T19:05:59.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6NKUMrpyPQtfWYNiUWPRexF8ALM=', 'Content-Length': '0'} 2023-11-02T19:05:59.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059454bf1c3e62c1e-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.210 DEBUG:boto:path=/ibtljn-64/obj185 2023-11-02T19:05:59.210 DEBUG:boto:auth_path=/ibtljn-64/obj185 2023-11-02T19:05:59.211 DEBUG:boto:path=/ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.211 DEBUG:boto:auth_path=/ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.212 DEBUG:boto:Method: GET 2023-11-02T19:05:59.212 DEBUG:boto:Path: /ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.213 DEBUG:boto:Data: 2023-11-02T19:05:59.213 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.213 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.214 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.214 DEBUG:boto:Params: {} 2023-11-02T19:05:59.215 DEBUG:boto:Token: None 2023-11-02T19:05:59.215 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.216 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6NKUMrpyPQtfWYNiUWPRexF8ALM= 2023-11-02T19:05:59.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6NKUMrpyPQtfWYNiUWPRexF8ALM=', 'Content-Length': '0'} 2023-11-02T19:05:59.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a9f5d493ef2de1-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.220 DEBUG:boto:path=/ibtljn-64/obj185 2023-11-02T19:05:59.220 DEBUG:boto:auth_path=/ibtljn-64/obj185 2023-11-02T19:05:59.221 DEBUG:boto:path=/ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.221 DEBUG:boto:auth_path=/ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.222 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.222 DEBUG:boto:Path: /ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.223 DEBUG:boto:Data: 2023-11-02T19:05:59.223 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.224 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.224 DEBUG:boto:Params: {} 2023-11-02T19:05:59.225 DEBUG:boto:Token: None 2023-11-02T19:05:59.225 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o4+CbEV8QnK00OB9Sdr9KSvyz2w= 2023-11-02T19:05:59.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o4+CbEV8QnK00OB9Sdr9KSvyz2w=', 'Content-Length': '0'} 2023-11-02T19:05:59.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab21fbf45bc3aee7-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.230 DEBUG:boto:path=/ibtljn-64/obj185 2023-11-02T19:05:59.230 DEBUG:boto:auth_path=/ibtljn-64/obj185 2023-11-02T19:05:59.231 DEBUG:boto:path=/ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.231 DEBUG:boto:auth_path=/ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.232 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.232 DEBUG:boto:Path: /ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.232 DEBUG:boto:Data: 2023-11-02T19:05:59.233 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.233 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.234 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.234 DEBUG:boto:Params: {} 2023-11-02T19:05:59.234 DEBUG:boto:Token: None 2023-11-02T19:05:59.235 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.235 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o4+CbEV8QnK00OB9Sdr9KSvyz2w= 2023-11-02T19:05:59.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o4+CbEV8QnK00OB9Sdr9KSvyz2w=', 'Content-Length': '0'} 2023-11-02T19:05:59.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087ff47037123a88d-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-11-02T19:05:59.240 DEBUG:boto:path=/ibtljn-64/obj185 2023-11-02T19:05:59.241 DEBUG:boto:auth_path=/ibtljn-64/obj185 2023-11-02T19:05:59.241 DEBUG:boto:path=/ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.242 DEBUG:boto:auth_path=/ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.242 DEBUG:boto:Method: GET 2023-11-02T19:05:59.243 DEBUG:boto:Path: /ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.243 DEBUG:boto:Data: 2023-11-02T19:05:59.243 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.244 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.244 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.245 DEBUG:boto:Params: {} 2023-11-02T19:05:59.246 DEBUG:boto:Token: None 2023-11-02T19:05:59.247 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.247 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6NKUMrpyPQtfWYNiUWPRexF8ALM= 2023-11-02T19:05:59.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6NKUMrpyPQtfWYNiUWPRexF8ALM=', 'Content-Length': '0'} 2023-11-02T19:05:59.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000288f2a43fc03b08e-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.253 DEBUG:boto:path=/ibtljn-64/obj185 2023-11-02T19:05:59.254 DEBUG:boto:auth_path=/ibtljn-64/obj185 2023-11-02T19:05:59.254 DEBUG:boto:path=/ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.255 DEBUG:boto:auth_path=/ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.255 DEBUG:boto:Method: GET 2023-11-02T19:05:59.255 DEBUG:boto:Path: /ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.256 DEBUG:boto:Data: 2023-11-02T19:05:59.256 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.257 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.258 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.258 DEBUG:boto:Params: {} 2023-11-02T19:05:59.259 DEBUG:boto:Token: None 2023-11-02T19:05:59.259 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj185?versionId=null 2023-11-02T19:05:59.260 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6NKUMrpyPQtfWYNiUWPRexF8ALM= 2023-11-02T19:05:59.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6NKUMrpyPQtfWYNiUWPRexF8ALM=', 'Content-Length': '0'} 2023-11-02T19:05:59.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdaf7d6121e6f909-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.310 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.311 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.311 DEBUG:boto:path=/ibtljn-64/?prefix=obj185 2023-11-02T19:05:59.312 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj185 2023-11-02T19:05:59.312 DEBUG:boto:Method: GET 2023-11-02T19:05:59.313 DEBUG:boto:Path: /ibtljn-64/?prefix=obj185 2023-11-02T19:05:59.314 DEBUG:boto:Data: 2023-11-02T19:05:59.315 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.315 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.316 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.317 DEBUG:boto:Params: {} 2023-11-02T19:05:59.317 DEBUG:boto:Token: None 2023-11-02T19:05:59.318 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.319 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a476b82bca9234f5-006543f317-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.324 DEBUG:boto:b'ibtljn-64obj1851000falseobj1852023-11-02T19:03:44.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.325 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.326 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.326 DEBUG:boto:path=/ibtljn-64/?prefix=obj185 2023-11-02T19:05:59.327 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj185 2023-11-02T19:05:59.327 DEBUG:boto:Method: GET 2023-11-02T19:05:59.327 DEBUG:boto:Path: /ibtljn-64/?prefix=obj185 2023-11-02T19:05:59.328 DEBUG:boto:Data: 2023-11-02T19:05:59.328 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.329 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.329 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.330 DEBUG:boto:Params: {} 2023-11-02T19:05:59.330 DEBUG:boto:Token: None 2023-11-02T19:05:59.331 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.331 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e58b989ac974e9bf-006543f317-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.342 DEBUG:boto:b'ibtljn-64obj1851000falseobj1852023-11-02T19:03:44.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.343 DEBUG:boto:path=/ibtljn-64/obj185 2023-11-02T19:05:59.344 DEBUG:boto:auth_path=/ibtljn-64/obj185 2023-11-02T19:05:59.344 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.344 DEBUG:boto:Path: /ibtljn-64/obj185 2023-11-02T19:05:59.345 DEBUG:boto:Data: 2023-11-02T19:05:59.345 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.346 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.347 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.347 DEBUG:boto:Params: {} 2023-11-02T19:05:59.348 DEBUG:boto:Token: None 2023-11-02T19:05:59.348 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj185 2023-11-02T19:05:59.349 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/UemqbDv6BIaHhD6OCnR2f/x7Ls= 2023-11-02T19:05:59.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/UemqbDv6BIaHhD6OCnR2f/x7Ls=', 'Content-Length': '0'} 2023-11-02T19:05:59.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be47160860d58852-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.352 DEBUG:boto:path=/ibtljn-64/obj185 2023-11-02T19:05:59.353 DEBUG:boto:auth_path=/ibtljn-64/obj185 2023-11-02T19:05:59.353 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.354 DEBUG:boto:Path: /ibtljn-64/obj185 2023-11-02T19:05:59.354 DEBUG:boto:Data: 2023-11-02T19:05:59.355 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.356 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.357 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.357 DEBUG:boto:Params: {} 2023-11-02T19:05:59.358 DEBUG:boto:Token: None 2023-11-02T19:05:59.359 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj185 2023-11-02T19:05:59.360 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/UemqbDv6BIaHhD6OCnR2f/x7Ls= 2023-11-02T19:05:59.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/UemqbDv6BIaHhD6OCnR2f/x7Ls=', 'Content-Length': '0'} 2023-11-02T19:05:59.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006039a5e7d091165c-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-11-02T19:05:59.365 DEBUG:boto:path=/ibtljn-64/obj185 2023-11-02T19:05:59.366 DEBUG:boto:auth_path=/ibtljn-64/obj185 2023-11-02T19:05:59.367 DEBUG:boto:Method: GET 2023-11-02T19:05:59.368 DEBUG:boto:Path: /ibtljn-64/obj185 2023-11-02T19:05:59.368 DEBUG:boto:Data: 2023-11-02T19:05:59.369 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.371 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.372 DEBUG:boto:Params: {} 2023-11-02T19:05:59.373 DEBUG:boto:Token: None 2023-11-02T19:05:59.374 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj185 2023-11-02T19:05:59.375 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+lfgOP3WVetWD2gVQ18AuCkRdc8= 2023-11-02T19:05:59.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+lfgOP3WVetWD2gVQ18AuCkRdc8=', 'Content-Length': '0'} 2023-11-02T19:05:59.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdda22aca0cfaf00-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.422 DEBUG:boto:path=/ibtljn-64/obj185 2023-11-02T19:05:59.423 DEBUG:boto:auth_path=/ibtljn-64/obj185 2023-11-02T19:05:59.424 DEBUG:boto:Method: GET 2023-11-02T19:05:59.424 DEBUG:boto:Path: /ibtljn-64/obj185 2023-11-02T19:05:59.425 DEBUG:boto:Data: 2023-11-02T19:05:59.425 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.426 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.426 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.427 DEBUG:boto:Params: {} 2023-11-02T19:05:59.427 DEBUG:boto:Token: None 2023-11-02T19:05:59.428 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj185 2023-11-02T19:05:59.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+lfgOP3WVetWD2gVQ18AuCkRdc8= 2023-11-02T19:05:59.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+lfgOP3WVetWD2gVQ18AuCkRdc8=', 'Content-Length': '0'} 2023-11-02T19:05:59.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b703c033d272f3f-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.432 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.433 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.433 DEBUG:boto:path=/ibtljn-64/?prefix=obj185 2023-11-02T19:05:59.434 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj185 2023-11-02T19:05:59.434 DEBUG:boto:Method: GET 2023-11-02T19:05:59.435 DEBUG:boto:Path: /ibtljn-64/?prefix=obj185 2023-11-02T19:05:59.435 DEBUG:boto:Data: 2023-11-02T19:05:59.436 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.436 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.436 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.437 DEBUG:boto:Params: {} 2023-11-02T19:05:59.437 DEBUG:boto:Token: None 2023-11-02T19:05:59.438 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.438 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eb41899e8a741d7-006543f317-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.444 DEBUG:boto:b'ibtljn-64obj1851000falseobj1852023-11-02T19:03:44.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.445 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.445 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.446 DEBUG:boto:path=/ibtljn-64/?prefix=obj185 2023-11-02T19:05:59.446 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj185 2023-11-02T19:05:59.447 DEBUG:boto:Method: GET 2023-11-02T19:05:59.447 DEBUG:boto:Path: /ibtljn-64/?prefix=obj185 2023-11-02T19:05:59.447 DEBUG:boto:Data: 2023-11-02T19:05:59.448 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.448 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.449 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.449 DEBUG:boto:Params: {} 2023-11-02T19:05:59.450 DEBUG:boto:Token: None 2023-11-02T19:05:59.450 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.451 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008768fa37dc6510fa-006543f317-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.502 DEBUG:boto:b'ibtljn-64obj1851000falseobj1852023-11-02T19:03:44.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T19:05:59.504 DEBUG:boto:path=/ibtljn-64/obj19 2023-11-02T19:05:59.504 DEBUG:boto:auth_path=/ibtljn-64/obj19 2023-11-02T19:05:59.505 DEBUG:boto:path=/ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.505 DEBUG:boto:auth_path=/ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.505 DEBUG:boto:Method: GET 2023-11-02T19:05:59.506 DEBUG:boto:Path: /ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.506 DEBUG:boto:Data: 2023-11-02T19:05:59.507 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.507 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.508 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.509 DEBUG:boto:Params: {} 2023-11-02T19:05:59.510 DEBUG:boto:Token: None 2023-11-02T19:05:59.510 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.511 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iRltbFwpGWXryUeKyykBTM6pzPU= 2023-11-02T19:05:59.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iRltbFwpGWXryUeKyykBTM6pzPU=', 'Content-Length': '0'} 2023-11-02T19:05:59.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be8a2090429a2be7-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.516 DEBUG:boto:path=/ibtljn-64/obj19 2023-11-02T19:05:59.517 DEBUG:boto:auth_path=/ibtljn-64/obj19 2023-11-02T19:05:59.517 DEBUG:boto:path=/ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.518 DEBUG:boto:auth_path=/ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.518 DEBUG:boto:Method: GET 2023-11-02T19:05:59.519 DEBUG:boto:Path: /ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.519 DEBUG:boto:Data: 2023-11-02T19:05:59.520 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.520 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.520 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.521 DEBUG:boto:Params: {} 2023-11-02T19:05:59.521 DEBUG:boto:Token: None 2023-11-02T19:05:59.522 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.523 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iRltbFwpGWXryUeKyykBTM6pzPU= 2023-11-02T19:05:59.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iRltbFwpGWXryUeKyykBTM6pzPU=', 'Content-Length': '0'} 2023-11-02T19:05:59.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec160919050a3086-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.527 DEBUG:boto:path=/ibtljn-64/obj19 2023-11-02T19:05:59.528 DEBUG:boto:auth_path=/ibtljn-64/obj19 2023-11-02T19:05:59.528 DEBUG:boto:path=/ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.529 DEBUG:boto:auth_path=/ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.529 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.530 DEBUG:boto:Path: /ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.530 DEBUG:boto:Data: 2023-11-02T19:05:59.531 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.531 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.531 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.532 DEBUG:boto:Params: {} 2023-11-02T19:05:59.533 DEBUG:boto:Token: None 2023-11-02T19:05:59.533 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.533 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQYPZUkH4rtni2Hi3cI5UlnnTIA= 2023-11-02T19:05:59.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQYPZUkH4rtni2Hi3cI5UlnnTIA=', 'Content-Length': '0'} 2023-11-02T19:05:59.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e689905708bc595-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.538 DEBUG:boto:path=/ibtljn-64/obj19 2023-11-02T19:05:59.538 DEBUG:boto:auth_path=/ibtljn-64/obj19 2023-11-02T19:05:59.539 DEBUG:boto:path=/ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.539 DEBUG:boto:auth_path=/ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.540 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.540 DEBUG:boto:Path: /ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.541 DEBUG:boto:Data: 2023-11-02T19:05:59.541 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.542 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.542 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.543 DEBUG:boto:Params: {} 2023-11-02T19:05:59.544 DEBUG:boto:Token: None 2023-11-02T19:05:59.545 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.546 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FQYPZUkH4rtni2Hi3cI5UlnnTIA= 2023-11-02T19:05:59.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FQYPZUkH4rtni2Hi3cI5UlnnTIA=', 'Content-Length': '0'} 2023-11-02T19:05:59.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065b2a94cd248c62b-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T19:05:59.552 DEBUG:boto:path=/ibtljn-64/obj19 2023-11-02T19:05:59.552 DEBUG:boto:auth_path=/ibtljn-64/obj19 2023-11-02T19:05:59.553 DEBUG:boto:path=/ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.554 DEBUG:boto:auth_path=/ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.555 DEBUG:boto:Method: GET 2023-11-02T19:05:59.555 DEBUG:boto:Path: /ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.555 DEBUG:boto:Data: 2023-11-02T19:05:59.556 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.556 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.557 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.557 DEBUG:boto:Params: {} 2023-11-02T19:05:59.558 DEBUG:boto:Token: None 2023-11-02T19:05:59.558 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.560 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iRltbFwpGWXryUeKyykBTM6pzPU= 2023-11-02T19:05:59.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iRltbFwpGWXryUeKyykBTM6pzPU=', 'Content-Length': '0'} 2023-11-02T19:05:59.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a5c0ade382d69b1-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.564 DEBUG:boto:path=/ibtljn-64/obj19 2023-11-02T19:05:59.564 DEBUG:boto:auth_path=/ibtljn-64/obj19 2023-11-02T19:05:59.565 DEBUG:boto:path=/ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.566 DEBUG:boto:auth_path=/ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.566 DEBUG:boto:Method: GET 2023-11-02T19:05:59.567 DEBUG:boto:Path: /ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.567 DEBUG:boto:Data: 2023-11-02T19:05:59.568 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.569 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.570 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.571 DEBUG:boto:Params: {} 2023-11-02T19:05:59.571 DEBUG:boto:Token: None 2023-11-02T19:05:59.572 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj19?versionId=null 2023-11-02T19:05:59.573 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:iRltbFwpGWXryUeKyykBTM6pzPU= 2023-11-02T19:05:59.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iRltbFwpGWXryUeKyykBTM6pzPU=', 'Content-Length': '0'} 2023-11-02T19:05:59.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ca09a9b158ab2e0-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.579 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.580 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.580 DEBUG:boto:path=/ibtljn-64/?prefix=obj19 2023-11-02T19:05:59.581 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj19 2023-11-02T19:05:59.581 DEBUG:boto:Method: GET 2023-11-02T19:05:59.582 DEBUG:boto:Path: /ibtljn-64/?prefix=obj19 2023-11-02T19:05:59.582 DEBUG:boto:Data: 2023-11-02T19:05:59.583 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.583 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.584 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.584 DEBUG:boto:Params: {} 2023-11-02T19:05:59.585 DEBUG:boto:Token: None 2023-11-02T19:05:59.586 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.587 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d6805343abdcbad-006543f317-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.634 DEBUG:boto:b'ibtljn-64obj191000falseobj192023-11-02T19:03:42.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.635 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.636 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.636 DEBUG:boto:path=/ibtljn-64/?prefix=obj19 2023-11-02T19:05:59.637 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj19 2023-11-02T19:05:59.637 DEBUG:boto:Method: GET 2023-11-02T19:05:59.638 DEBUG:boto:Path: /ibtljn-64/?prefix=obj19 2023-11-02T19:05:59.638 DEBUG:boto:Data: 2023-11-02T19:05:59.639 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.639 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.640 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.640 DEBUG:boto:Params: {} 2023-11-02T19:05:59.641 DEBUG:boto:Token: None 2023-11-02T19:05:59.642 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.643 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd2dd5de63b4212d-006543f317-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.654 DEBUG:boto:b'ibtljn-64obj191000falseobj192023-11-02T19:03:42.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.654 DEBUG:boto:path=/ibtljn-64/obj19 2023-11-02T19:05:59.655 DEBUG:boto:auth_path=/ibtljn-64/obj19 2023-11-02T19:05:59.656 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.656 DEBUG:boto:Path: /ibtljn-64/obj19 2023-11-02T19:05:59.657 DEBUG:boto:Data: 2023-11-02T19:05:59.657 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.658 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.658 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.659 DEBUG:boto:Params: {} 2023-11-02T19:05:59.659 DEBUG:boto:Token: None 2023-11-02T19:05:59.659 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj19 2023-11-02T19:05:59.660 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3nv6F1zBQdHBOiTr3OnG8bul40E= 2023-11-02T19:05:59.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3nv6F1zBQdHBOiTr3OnG8bul40E=', 'Content-Length': '0'} 2023-11-02T19:05:59.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f619896234b2ec8b-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.664 DEBUG:boto:path=/ibtljn-64/obj19 2023-11-02T19:05:59.664 DEBUG:boto:auth_path=/ibtljn-64/obj19 2023-11-02T19:05:59.665 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.665 DEBUG:boto:Path: /ibtljn-64/obj19 2023-11-02T19:05:59.666 DEBUG:boto:Data: 2023-11-02T19:05:59.666 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.667 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.667 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.668 DEBUG:boto:Params: {} 2023-11-02T19:05:59.669 DEBUG:boto:Token: None 2023-11-02T19:05:59.669 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj19 2023-11-02T19:05:59.670 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3nv6F1zBQdHBOiTr3OnG8bul40E= 2023-11-02T19:05:59.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3nv6F1zBQdHBOiTr3OnG8bul40E=', 'Content-Length': '0'} 2023-11-02T19:05:59.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b746a6e9a1d8852d-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-11-02T19:05:59.674 DEBUG:boto:path=/ibtljn-64/obj19 2023-11-02T19:05:59.675 DEBUG:boto:auth_path=/ibtljn-64/obj19 2023-11-02T19:05:59.676 DEBUG:boto:Method: GET 2023-11-02T19:05:59.676 DEBUG:boto:Path: /ibtljn-64/obj19 2023-11-02T19:05:59.677 DEBUG:boto:Data: 2023-11-02T19:05:59.677 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.678 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.678 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.679 DEBUG:boto:Params: {} 2023-11-02T19:05:59.679 DEBUG:boto:Token: None 2023-11-02T19:05:59.680 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj19 2023-11-02T19:05:59.680 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P8k84vLev8CK2CCsly262JUxlRY= 2023-11-02T19:05:59.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P8k84vLev8CK2CCsly262JUxlRY=', 'Content-Length': '0'} 2023-11-02T19:05:59.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e836a91398559a76-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.684 DEBUG:boto:path=/ibtljn-64/obj19 2023-11-02T19:05:59.685 DEBUG:boto:auth_path=/ibtljn-64/obj19 2023-11-02T19:05:59.685 DEBUG:boto:Method: GET 2023-11-02T19:05:59.686 DEBUG:boto:Path: /ibtljn-64/obj19 2023-11-02T19:05:59.686 DEBUG:boto:Data: 2023-11-02T19:05:59.687 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.687 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.688 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.688 DEBUG:boto:Params: {} 2023-11-02T19:05:59.689 DEBUG:boto:Token: None 2023-11-02T19:05:59.689 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj19 2023-11-02T19:05:59.690 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:P8k84vLev8CK2CCsly262JUxlRY= 2023-11-02T19:05:59.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P8k84vLev8CK2CCsly262JUxlRY=', 'Content-Length': '0'} 2023-11-02T19:05:59.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055216e0ab6face19-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.734 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.736 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.736 DEBUG:boto:path=/ibtljn-64/?prefix=obj19 2023-11-02T19:05:59.737 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj19 2023-11-02T19:05:59.737 DEBUG:boto:Method: GET 2023-11-02T19:05:59.738 DEBUG:boto:Path: /ibtljn-64/?prefix=obj19 2023-11-02T19:05:59.738 DEBUG:boto:Data: 2023-11-02T19:05:59.739 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.739 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.739 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.740 DEBUG:boto:Params: {} 2023-11-02T19:05:59.741 DEBUG:boto:Token: None 2023-11-02T19:05:59.741 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.742 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065cd4ad2faa34aa8-006543f317-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.748 DEBUG:boto:b'ibtljn-64obj191000falseobj192023-11-02T19:03:42.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.749 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.749 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.750 DEBUG:boto:path=/ibtljn-64/?prefix=obj19 2023-11-02T19:05:59.750 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj19 2023-11-02T19:05:59.750 DEBUG:boto:Method: GET 2023-11-02T19:05:59.751 DEBUG:boto:Path: /ibtljn-64/?prefix=obj19 2023-11-02T19:05:59.751 DEBUG:boto:Data: 2023-11-02T19:05:59.752 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.752 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.753 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.753 DEBUG:boto:Params: {} 2023-11-02T19:05:59.754 DEBUG:boto:Token: None 2023-11-02T19:05:59.754 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.755 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e49087e807686f6-006543f317-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.765 DEBUG:boto:b'ibtljn-64obj191000falseobj192023-11-02T19:03:42.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T19:05:59.766 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:05:59.767 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:05:59.767 DEBUG:boto:path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.768 DEBUG:boto:auth_path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.768 DEBUG:boto:Method: GET 2023-11-02T19:05:59.769 DEBUG:boto:Path: /ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.769 DEBUG:boto:Data: 2023-11-02T19:05:59.770 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.770 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.771 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.771 DEBUG:boto:Params: {} 2023-11-02T19:05:59.772 DEBUG:boto:Token: None 2023-11-02T19:05:59.772 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.773 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4tXDPpU+/GT0eZyanlWfbNRs+Ic= 2023-11-02T19:05:59.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4tXDPpU+/GT0eZyanlWfbNRs+Ic=', 'Content-Length': '0'} 2023-11-02T19:05:59.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d907689f86b14bb5-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.818 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:05:59.819 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:05:59.819 DEBUG:boto:path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.820 DEBUG:boto:auth_path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.820 DEBUG:boto:Method: GET 2023-11-02T19:05:59.821 DEBUG:boto:Path: /ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.821 DEBUG:boto:Data: 2023-11-02T19:05:59.822 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.822 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.822 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.823 DEBUG:boto:Params: {} 2023-11-02T19:05:59.823 DEBUG:boto:Token: None 2023-11-02T19:05:59.824 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.824 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4tXDPpU+/GT0eZyanlWfbNRs+Ic= 2023-11-02T19:05:59.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4tXDPpU+/GT0eZyanlWfbNRs+Ic=', 'Content-Length': '0'} 2023-11-02T19:05:59.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af663197feeddf0a-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.829 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:05:59.829 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:05:59.829 DEBUG:boto:path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.830 DEBUG:boto:auth_path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.830 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.831 DEBUG:boto:Path: /ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.831 DEBUG:boto:Data: 2023-11-02T19:05:59.832 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.832 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.833 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.833 DEBUG:boto:Params: {} 2023-11-02T19:05:59.833 DEBUG:boto:Token: None 2023-11-02T19:05:59.834 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.834 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/LHItOY3wMdI4iyYwt/Gs11T5sY= 2023-11-02T19:05:59.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/LHItOY3wMdI4iyYwt/Gs11T5sY=', 'Content-Length': '0'} 2023-11-02T19:05:59.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006718a111678c358d-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.838 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:05:59.839 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:05:59.839 DEBUG:boto:path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.839 DEBUG:boto:auth_path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.840 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.840 DEBUG:boto:Path: /ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.841 DEBUG:boto:Data: 2023-11-02T19:05:59.841 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.842 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.842 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.842 DEBUG:boto:Params: {} 2023-11-02T19:05:59.843 DEBUG:boto:Token: None 2023-11-02T19:05:59.843 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.844 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/LHItOY3wMdI4iyYwt/Gs11T5sY= 2023-11-02T19:05:59.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/LHItOY3wMdI4iyYwt/Gs11T5sY=', 'Content-Length': '0'} 2023-11-02T19:05:59.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006dd76854faa9a22-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T19:05:59.848 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:05:59.849 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:05:59.849 DEBUG:boto:path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.850 DEBUG:boto:auth_path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.850 DEBUG:boto:Method: GET 2023-11-02T19:05:59.851 DEBUG:boto:Path: /ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.851 DEBUG:boto:Data: 2023-11-02T19:05:59.852 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.852 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.852 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.853 DEBUG:boto:Params: {} 2023-11-02T19:05:59.853 DEBUG:boto:Token: None 2023-11-02T19:05:59.854 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.854 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4tXDPpU+/GT0eZyanlWfbNRs+Ic= 2023-11-02T19:05:59.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4tXDPpU+/GT0eZyanlWfbNRs+Ic=', 'Content-Length': '0'} 2023-11-02T19:05:59.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000283268bd6a2cb0c-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.859 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:05:59.860 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:05:59.860 DEBUG:boto:path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.860 DEBUG:boto:auth_path=/ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.861 DEBUG:boto:Method: GET 2023-11-02T19:05:59.861 DEBUG:boto:Path: /ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.862 DEBUG:boto:Data: 2023-11-02T19:05:59.862 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.863 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.863 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.864 DEBUG:boto:Params: {} 2023-11-02T19:05:59.864 DEBUG:boto:Token: None 2023-11-02T19:05:59.865 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj22?versionId=null 2023-11-02T19:05:59.865 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4tXDPpU+/GT0eZyanlWfbNRs+Ic= 2023-11-02T19:05:59.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4tXDPpU+/GT0eZyanlWfbNRs+Ic=', 'Content-Length': '0'} 2023-11-02T19:05:59.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000967d331d115b0349-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.914 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.915 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.915 DEBUG:boto:path=/ibtljn-64/?prefix=obj22 2023-11-02T19:05:59.916 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj22 2023-11-02T19:05:59.916 DEBUG:boto:Method: GET 2023-11-02T19:05:59.917 DEBUG:boto:Path: /ibtljn-64/?prefix=obj22 2023-11-02T19:05:59.917 DEBUG:boto:Data: 2023-11-02T19:05:59.918 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.918 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.918 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.919 DEBUG:boto:Params: {} 2023-11-02T19:05:59.919 DEBUG:boto:Token: None 2023-11-02T19:05:59.920 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.920 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000565ac974eb5679fe-006543f317-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.927 DEBUG:boto:b'ibtljn-64obj221000falseobj222023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.928 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:05:59.928 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:05:59.929 DEBUG:boto:path=/ibtljn-64/?prefix=obj22 2023-11-02T19:05:59.929 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj22 2023-11-02T19:05:59.930 DEBUG:boto:Method: GET 2023-11-02T19:05:59.930 DEBUG:boto:Path: /ibtljn-64/?prefix=obj22 2023-11-02T19:05:59.931 DEBUG:boto:Data: 2023-11-02T19:05:59.931 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.932 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.932 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.932 DEBUG:boto:Params: {} 2023-11-02T19:05:59.933 DEBUG:boto:Token: None 2023-11-02T19:05:59.933 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/ 2023-11-02T19:05:59.934 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY= 2023-11-02T19:05:59.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n4XBF+VeoAdpVxcYqb7kFCFjxfY=', 'Content-Length': '0'} 2023-11-02T19:05:59.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a1b3179162de676-006543f317-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.943 DEBUG:boto:b'ibtljn-64obj221000falseobj222023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:05:59.944 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:05:59.944 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:05:59.944 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.945 DEBUG:boto:Path: /ibtljn-64/obj22 2023-11-02T19:05:59.945 DEBUG:boto:Data: 2023-11-02T19:05:59.946 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.946 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.947 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.947 DEBUG:boto:Params: {} 2023-11-02T19:05:59.947 DEBUG:boto:Token: None 2023-11-02T19:05:59.948 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj22 2023-11-02T19:05:59.948 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GIf5V/wjueQDf63YOpCmYrfZC7k= 2023-11-02T19:05:59.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GIf5V/wjueQDf63YOpCmYrfZC7k=', 'Content-Length': '0'} 2023-11-02T19:05:59.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000857eb115eb3fc6d5-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.952 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:05:59.952 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:05:59.953 DEBUG:boto:Method: HEAD 2023-11-02T19:05:59.953 DEBUG:boto:Path: /ibtljn-64/obj22 2023-11-02T19:05:59.954 DEBUG:boto:Data: 2023-11-02T19:05:59.954 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.955 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:05:59.955 DEBUG:boto:Port: 8000 2023-11-02T19:05:59.955 DEBUG:boto:Params: {} 2023-11-02T19:05:59.956 DEBUG:boto:Token: None 2023-11-02T19:05:59.956 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj22 2023-11-02T19:05:59.957 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GIf5V/wjueQDf63YOpCmYrfZC7k= 2023-11-02T19:05:59.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GIf5V/wjueQDf63YOpCmYrfZC7k=', 'Content-Length': '0'} 2023-11-02T19:05:59.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c362a581f554ebc-006543f317-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:05:59.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-11-02T19:05:59.961 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:05:59.961 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:05:59.962 DEBUG:boto:Method: GET 2023-11-02T19:05:59.962 DEBUG:boto:Path: /ibtljn-64/obj22 2023-11-02T19:05:59.963 DEBUG:boto:Data: 2023-11-02T19:05:59.963 DEBUG:boto:Headers: {} 2023-11-02T19:05:59.964 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:05:59.964 DEBUG:boto:Port: 8001 2023-11-02T19:05:59.965 DEBUG:boto:Params: {} 2023-11-02T19:05:59.965 DEBUG:boto:Token: None 2023-11-02T19:05:59.965 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:05:59 GMT /ibtljn-64/obj22 2023-11-02T19:05:59.966 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KQ74Ba+yPqWHrJixx7vaPGTq5Lw= 2023-11-02T19:05:59.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:05:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KQ74Ba+yPqWHrJixx7vaPGTq5Lw=', 'Content-Length': '0'} 2023-11-02T19:05:59.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006824588533cf5882-006543f317-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:05:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.010 DEBUG:boto:path=/ibtljn-64/obj22 2023-11-02T19:06:00.011 DEBUG:boto:auth_path=/ibtljn-64/obj22 2023-11-02T19:06:00.012 DEBUG:boto:Method: GET 2023-11-02T19:06:00.012 DEBUG:boto:Path: /ibtljn-64/obj22 2023-11-02T19:06:00.012 DEBUG:boto:Data: 2023-11-02T19:06:00.013 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.013 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.014 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.014 DEBUG:boto:Params: {} 2023-11-02T19:06:00.015 DEBUG:boto:Token: None 2023-11-02T19:06:00.015 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj22 2023-11-02T19:06:00.016 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mnG4gKDW/bLKAyzcm+OLdIJAzr8= 2023-11-02T19:06:00.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mnG4gKDW/bLKAyzcm+OLdIJAzr8=', 'Content-Length': '0'} 2023-11-02T19:06:00.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb95caf6c02c9289-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.020 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.020 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.020 DEBUG:boto:path=/ibtljn-64/?prefix=obj22 2023-11-02T19:06:00.021 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj22 2023-11-02T19:06:00.021 DEBUG:boto:Method: GET 2023-11-02T19:06:00.022 DEBUG:boto:Path: /ibtljn-64/?prefix=obj22 2023-11-02T19:06:00.022 DEBUG:boto:Data: 2023-11-02T19:06:00.023 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.023 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.023 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.024 DEBUG:boto:Params: {} 2023-11-02T19:06:00.024 DEBUG:boto:Token: None 2023-11-02T19:06:00.025 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.025 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b79eb87b4c5bfae-006543f318-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.031 DEBUG:boto:b'ibtljn-64obj221000falseobj222023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.032 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.033 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.033 DEBUG:boto:path=/ibtljn-64/?prefix=obj22 2023-11-02T19:06:00.034 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj22 2023-11-02T19:06:00.034 DEBUG:boto:Method: GET 2023-11-02T19:06:00.034 DEBUG:boto:Path: /ibtljn-64/?prefix=obj22 2023-11-02T19:06:00.035 DEBUG:boto:Data: 2023-11-02T19:06:00.035 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.036 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.036 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.037 DEBUG:boto:Params: {} 2023-11-02T19:06:00.037 DEBUG:boto:Token: None 2023-11-02T19:06:00.038 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.038 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f96bac4435e5d9-006543f318-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.094 DEBUG:boto:b'ibtljn-64obj221000falseobj222023-11-02T19:03:19.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T19:06:00.096 DEBUG:boto:path=/ibtljn-64/obj23 2023-11-02T19:06:00.096 DEBUG:boto:auth_path=/ibtljn-64/obj23 2023-11-02T19:06:00.097 DEBUG:boto:path=/ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.097 DEBUG:boto:auth_path=/ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.097 DEBUG:boto:Method: GET 2023-11-02T19:06:00.098 DEBUG:boto:Path: /ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.098 DEBUG:boto:Data: 2023-11-02T19:06:00.099 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.099 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.100 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.100 DEBUG:boto:Params: {} 2023-11-02T19:06:00.100 DEBUG:boto:Token: None 2023-11-02T19:06:00.101 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.101 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZxfSf7hxJxwiyX4vrDu8RntFjyk= 2023-11-02T19:06:00.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZxfSf7hxJxwiyX4vrDu8RntFjyk=', 'Content-Length': '0'} 2023-11-02T19:06:00.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e26d4f4b6346567-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.106 DEBUG:boto:path=/ibtljn-64/obj23 2023-11-02T19:06:00.107 DEBUG:boto:auth_path=/ibtljn-64/obj23 2023-11-02T19:06:00.107 DEBUG:boto:path=/ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.108 DEBUG:boto:auth_path=/ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.108 DEBUG:boto:Method: GET 2023-11-02T19:06:00.108 DEBUG:boto:Path: /ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.109 DEBUG:boto:Data: 2023-11-02T19:06:00.109 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.110 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.110 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.111 DEBUG:boto:Params: {} 2023-11-02T19:06:00.111 DEBUG:boto:Token: None 2023-11-02T19:06:00.112 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.112 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZxfSf7hxJxwiyX4vrDu8RntFjyk= 2023-11-02T19:06:00.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZxfSf7hxJxwiyX4vrDu8RntFjyk=', 'Content-Length': '0'} 2023-11-02T19:06:00.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d9aba5c9c199107-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.116 DEBUG:boto:path=/ibtljn-64/obj23 2023-11-02T19:06:00.116 DEBUG:boto:auth_path=/ibtljn-64/obj23 2023-11-02T19:06:00.117 DEBUG:boto:path=/ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.117 DEBUG:boto:auth_path=/ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.118 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.118 DEBUG:boto:Path: /ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.118 DEBUG:boto:Data: 2023-11-02T19:06:00.119 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.119 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.120 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.120 DEBUG:boto:Params: {} 2023-11-02T19:06:00.121 DEBUG:boto:Token: None 2023-11-02T19:06:00.121 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.122 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W+Wgy1pVYPdcunyxU+kj571Dl5Y= 2023-11-02T19:06:00.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W+Wgy1pVYPdcunyxU+kj571Dl5Y=', 'Content-Length': '0'} 2023-11-02T19:06:00.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000114f523517ff35b9-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.125 DEBUG:boto:path=/ibtljn-64/obj23 2023-11-02T19:06:00.126 DEBUG:boto:auth_path=/ibtljn-64/obj23 2023-11-02T19:06:00.126 DEBUG:boto:path=/ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.127 DEBUG:boto:auth_path=/ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.127 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.127 DEBUG:boto:Path: /ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.128 DEBUG:boto:Data: 2023-11-02T19:06:00.128 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.129 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.129 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.130 DEBUG:boto:Params: {} 2023-11-02T19:06:00.130 DEBUG:boto:Token: None 2023-11-02T19:06:00.130 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.131 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:W+Wgy1pVYPdcunyxU+kj571Dl5Y= 2023-11-02T19:06:00.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W+Wgy1pVYPdcunyxU+kj571Dl5Y=', 'Content-Length': '0'} 2023-11-02T19:06:00.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db72ddacb6a70c98-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T19:06:00.136 DEBUG:boto:path=/ibtljn-64/obj23 2023-11-02T19:06:00.136 DEBUG:boto:auth_path=/ibtljn-64/obj23 2023-11-02T19:06:00.136 DEBUG:boto:path=/ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.137 DEBUG:boto:auth_path=/ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.137 DEBUG:boto:Method: GET 2023-11-02T19:06:00.138 DEBUG:boto:Path: /ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.138 DEBUG:boto:Data: 2023-11-02T19:06:00.139 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.139 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.139 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.140 DEBUG:boto:Params: {} 2023-11-02T19:06:00.140 DEBUG:boto:Token: None 2023-11-02T19:06:00.141 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZxfSf7hxJxwiyX4vrDu8RntFjyk= 2023-11-02T19:06:00.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZxfSf7hxJxwiyX4vrDu8RntFjyk=', 'Content-Length': '0'} 2023-11-02T19:06:00.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f13c0b8575446eab-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.145 DEBUG:boto:path=/ibtljn-64/obj23 2023-11-02T19:06:00.146 DEBUG:boto:auth_path=/ibtljn-64/obj23 2023-11-02T19:06:00.146 DEBUG:boto:path=/ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.147 DEBUG:boto:auth_path=/ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.147 DEBUG:boto:Method: GET 2023-11-02T19:06:00.148 DEBUG:boto:Path: /ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.148 DEBUG:boto:Data: 2023-11-02T19:06:00.148 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.149 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.149 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.150 DEBUG:boto:Params: {} 2023-11-02T19:06:00.150 DEBUG:boto:Token: None 2023-11-02T19:06:00.151 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj23?versionId=null 2023-11-02T19:06:00.151 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZxfSf7hxJxwiyX4vrDu8RntFjyk= 2023-11-02T19:06:00.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZxfSf7hxJxwiyX4vrDu8RntFjyk=', 'Content-Length': '0'} 2023-11-02T19:06:00.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004988fab6d9e56fec-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.202 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.203 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.203 DEBUG:boto:path=/ibtljn-64/?prefix=obj23 2023-11-02T19:06:00.204 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj23 2023-11-02T19:06:00.204 DEBUG:boto:Method: GET 2023-11-02T19:06:00.205 DEBUG:boto:Path: /ibtljn-64/?prefix=obj23 2023-11-02T19:06:00.205 DEBUG:boto:Data: 2023-11-02T19:06:00.206 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.206 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.206 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.207 DEBUG:boto:Params: {} 2023-11-02T19:06:00.207 DEBUG:boto:Token: None 2023-11-02T19:06:00.208 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.208 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c66775a24b1d42d3-006543f318-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.214 DEBUG:boto:b'ibtljn-64obj231000falseobj232023-11-02T19:03:43.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.215 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.215 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.216 DEBUG:boto:path=/ibtljn-64/?prefix=obj23 2023-11-02T19:06:00.216 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj23 2023-11-02T19:06:00.216 DEBUG:boto:Method: GET 2023-11-02T19:06:00.217 DEBUG:boto:Path: /ibtljn-64/?prefix=obj23 2023-11-02T19:06:00.217 DEBUG:boto:Data: 2023-11-02T19:06:00.218 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.218 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.219 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.219 DEBUG:boto:Params: {} 2023-11-02T19:06:00.220 DEBUG:boto:Token: None 2023-11-02T19:06:00.220 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.220 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e86aa001bec7b24-006543f318-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.231 DEBUG:boto:b'ibtljn-64obj231000falseobj232023-11-02T19:03:43.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.232 DEBUG:boto:path=/ibtljn-64/obj23 2023-11-02T19:06:00.232 DEBUG:boto:auth_path=/ibtljn-64/obj23 2023-11-02T19:06:00.233 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.233 DEBUG:boto:Path: /ibtljn-64/obj23 2023-11-02T19:06:00.233 DEBUG:boto:Data: 2023-11-02T19:06:00.234 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.234 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.235 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.235 DEBUG:boto:Params: {} 2023-11-02T19:06:00.236 DEBUG:boto:Token: None 2023-11-02T19:06:00.236 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj23 2023-11-02T19:06:00.236 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vVZ6EWdK+UE7+7WfEH7TWEYJCN8= 2023-11-02T19:06:00.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vVZ6EWdK+UE7+7WfEH7TWEYJCN8=', 'Content-Length': '0'} 2023-11-02T19:06:00.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e38f28f8895cbce2-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.240 DEBUG:boto:path=/ibtljn-64/obj23 2023-11-02T19:06:00.241 DEBUG:boto:auth_path=/ibtljn-64/obj23 2023-11-02T19:06:00.241 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.241 DEBUG:boto:Path: /ibtljn-64/obj23 2023-11-02T19:06:00.242 DEBUG:boto:Data: 2023-11-02T19:06:00.242 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.243 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.243 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.244 DEBUG:boto:Params: {} 2023-11-02T19:06:00.244 DEBUG:boto:Token: None 2023-11-02T19:06:00.244 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj23 2023-11-02T19:06:00.245 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vVZ6EWdK+UE7+7WfEH7TWEYJCN8= 2023-11-02T19:06:00.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vVZ6EWdK+UE7+7WfEH7TWEYJCN8=', 'Content-Length': '0'} 2023-11-02T19:06:00.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059c89a431aee84b4-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-11-02T19:06:00.249 DEBUG:boto:path=/ibtljn-64/obj23 2023-11-02T19:06:00.249 DEBUG:boto:auth_path=/ibtljn-64/obj23 2023-11-02T19:06:00.250 DEBUG:boto:Method: GET 2023-11-02T19:06:00.250 DEBUG:boto:Path: /ibtljn-64/obj23 2023-11-02T19:06:00.251 DEBUG:boto:Data: 2023-11-02T19:06:00.251 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.251 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.252 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.252 DEBUG:boto:Params: {} 2023-11-02T19:06:00.253 DEBUG:boto:Token: None 2023-11-02T19:06:00.253 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj23 2023-11-02T19:06:00.254 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Hd2hfS1u1DZU4I+ocQf23Nazi58= 2023-11-02T19:06:00.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Hd2hfS1u1DZU4I+ocQf23Nazi58=', 'Content-Length': '0'} 2023-11-02T19:06:00.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebf0ef1a7e03714f-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.298 DEBUG:boto:path=/ibtljn-64/obj23 2023-11-02T19:06:00.299 DEBUG:boto:auth_path=/ibtljn-64/obj23 2023-11-02T19:06:00.299 DEBUG:boto:Method: GET 2023-11-02T19:06:00.300 DEBUG:boto:Path: /ibtljn-64/obj23 2023-11-02T19:06:00.300 DEBUG:boto:Data: 2023-11-02T19:06:00.301 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.301 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.302 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.302 DEBUG:boto:Params: {} 2023-11-02T19:06:00.303 DEBUG:boto:Token: None 2023-11-02T19:06:00.303 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj23 2023-11-02T19:06:00.304 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Hd2hfS1u1DZU4I+ocQf23Nazi58= 2023-11-02T19:06:00.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Hd2hfS1u1DZU4I+ocQf23Nazi58=', 'Content-Length': '0'} 2023-11-02T19:06:00.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f274bb1ad180b1a3-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.308 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.308 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.309 DEBUG:boto:path=/ibtljn-64/?prefix=obj23 2023-11-02T19:06:00.309 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj23 2023-11-02T19:06:00.310 DEBUG:boto:Method: GET 2023-11-02T19:06:00.310 DEBUG:boto:Path: /ibtljn-64/?prefix=obj23 2023-11-02T19:06:00.310 DEBUG:boto:Data: 2023-11-02T19:06:00.311 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.311 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.312 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.312 DEBUG:boto:Params: {} 2023-11-02T19:06:00.312 DEBUG:boto:Token: None 2023-11-02T19:06:00.313 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.313 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081764f35f6bd964c-006543f318-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.319 DEBUG:boto:b'ibtljn-64obj231000falseobj232023-11-02T19:03:43.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.320 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.320 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.321 DEBUG:boto:path=/ibtljn-64/?prefix=obj23 2023-11-02T19:06:00.321 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj23 2023-11-02T19:06:00.322 DEBUG:boto:Method: GET 2023-11-02T19:06:00.322 DEBUG:boto:Path: /ibtljn-64/?prefix=obj23 2023-11-02T19:06:00.323 DEBUG:boto:Data: 2023-11-02T19:06:00.323 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.323 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.324 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.324 DEBUG:boto:Params: {} 2023-11-02T19:06:00.325 DEBUG:boto:Token: None 2023-11-02T19:06:00.325 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.326 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c64045d63257a4c-006543f318-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.378 DEBUG:boto:b'ibtljn-64obj231000falseobj232023-11-02T19:03:43.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T19:06:00.382 DEBUG:boto:path=/ibtljn-64/obj29 2023-11-02T19:06:00.382 DEBUG:boto:auth_path=/ibtljn-64/obj29 2023-11-02T19:06:00.383 DEBUG:boto:path=/ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.383 DEBUG:boto:auth_path=/ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.384 DEBUG:boto:Method: GET 2023-11-02T19:06:00.384 DEBUG:boto:Path: /ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.385 DEBUG:boto:Data: 2023-11-02T19:06:00.385 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.386 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.386 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.387 DEBUG:boto:Params: {} 2023-11-02T19:06:00.387 DEBUG:boto:Token: None 2023-11-02T19:06:00.388 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.388 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QPErkYAxkhBBSjTXUKUGdvnWVZQ= 2023-11-02T19:06:00.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QPErkYAxkhBBSjTXUKUGdvnWVZQ=', 'Content-Length': '0'} 2023-11-02T19:06:00.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088c4753c90ca30d1-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.392 DEBUG:boto:path=/ibtljn-64/obj29 2023-11-02T19:06:00.393 DEBUG:boto:auth_path=/ibtljn-64/obj29 2023-11-02T19:06:00.393 DEBUG:boto:path=/ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.394 DEBUG:boto:auth_path=/ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.394 DEBUG:boto:Method: GET 2023-11-02T19:06:00.395 DEBUG:boto:Path: /ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.395 DEBUG:boto:Data: 2023-11-02T19:06:00.395 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.396 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.396 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.397 DEBUG:boto:Params: {} 2023-11-02T19:06:00.397 DEBUG:boto:Token: None 2023-11-02T19:06:00.398 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.398 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QPErkYAxkhBBSjTXUKUGdvnWVZQ= 2023-11-02T19:06:00.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QPErkYAxkhBBSjTXUKUGdvnWVZQ=', 'Content-Length': '0'} 2023-11-02T19:06:00.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095fae688be8123fc-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.402 DEBUG:boto:path=/ibtljn-64/obj29 2023-11-02T19:06:00.403 DEBUG:boto:auth_path=/ibtljn-64/obj29 2023-11-02T19:06:00.403 DEBUG:boto:path=/ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.404 DEBUG:boto:auth_path=/ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.404 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.404 DEBUG:boto:Path: /ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.405 DEBUG:boto:Data: 2023-11-02T19:06:00.405 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.406 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.406 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.406 DEBUG:boto:Params: {} 2023-11-02T19:06:00.407 DEBUG:boto:Token: None 2023-11-02T19:06:00.407 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.408 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yLt7ZwYjGg2zwCPd+yZ7Kj0lF/Y= 2023-11-02T19:06:00.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yLt7ZwYjGg2zwCPd+yZ7Kj0lF/Y=', 'Content-Length': '0'} 2023-11-02T19:06:00.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca90746f795a16ff-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.412 DEBUG:boto:path=/ibtljn-64/obj29 2023-11-02T19:06:00.412 DEBUG:boto:auth_path=/ibtljn-64/obj29 2023-11-02T19:06:00.413 DEBUG:boto:path=/ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.413 DEBUG:boto:auth_path=/ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.414 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.414 DEBUG:boto:Path: /ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.414 DEBUG:boto:Data: 2023-11-02T19:06:00.415 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.415 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.416 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.416 DEBUG:boto:Params: {} 2023-11-02T19:06:00.417 DEBUG:boto:Token: None 2023-11-02T19:06:00.417 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.417 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yLt7ZwYjGg2zwCPd+yZ7Kj0lF/Y= 2023-11-02T19:06:00.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yLt7ZwYjGg2zwCPd+yZ7Kj0lF/Y=', 'Content-Length': '0'} 2023-11-02T19:06:00.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000979b894f6a3bc688-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T19:06:00.422 DEBUG:boto:path=/ibtljn-64/obj29 2023-11-02T19:06:00.422 DEBUG:boto:auth_path=/ibtljn-64/obj29 2023-11-02T19:06:00.423 DEBUG:boto:path=/ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.423 DEBUG:boto:auth_path=/ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.423 DEBUG:boto:Method: GET 2023-11-02T19:06:00.424 DEBUG:boto:Path: /ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.424 DEBUG:boto:Data: 2023-11-02T19:06:00.425 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.425 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.426 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.426 DEBUG:boto:Params: {} 2023-11-02T19:06:00.427 DEBUG:boto:Token: None 2023-11-02T19:06:00.427 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.428 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QPErkYAxkhBBSjTXUKUGdvnWVZQ= 2023-11-02T19:06:00.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QPErkYAxkhBBSjTXUKUGdvnWVZQ=', 'Content-Length': '0'} 2023-11-02T19:06:00.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a66f56371568ea5-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.474 DEBUG:boto:path=/ibtljn-64/obj29 2023-11-02T19:06:00.475 DEBUG:boto:auth_path=/ibtljn-64/obj29 2023-11-02T19:06:00.475 DEBUG:boto:path=/ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.476 DEBUG:boto:auth_path=/ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.476 DEBUG:boto:Method: GET 2023-11-02T19:06:00.477 DEBUG:boto:Path: /ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.477 DEBUG:boto:Data: 2023-11-02T19:06:00.478 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.478 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.479 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.479 DEBUG:boto:Params: {} 2023-11-02T19:06:00.480 DEBUG:boto:Token: None 2023-11-02T19:06:00.480 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj29?versionId=null 2023-11-02T19:06:00.481 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QPErkYAxkhBBSjTXUKUGdvnWVZQ= 2023-11-02T19:06:00.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QPErkYAxkhBBSjTXUKUGdvnWVZQ=', 'Content-Length': '0'} 2023-11-02T19:06:00.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000442529a9b71113fd-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.485 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.486 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.486 DEBUG:boto:path=/ibtljn-64/?prefix=obj29 2023-11-02T19:06:00.486 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj29 2023-11-02T19:06:00.487 DEBUG:boto:Method: GET 2023-11-02T19:06:00.487 DEBUG:boto:Path: /ibtljn-64/?prefix=obj29 2023-11-02T19:06:00.488 DEBUG:boto:Data: 2023-11-02T19:06:00.488 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.489 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.489 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.489 DEBUG:boto:Params: {} 2023-11-02T19:06:00.490 DEBUG:boto:Token: None 2023-11-02T19:06:00.490 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.491 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c429e79e25a6275f-006543f318-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.497 DEBUG:boto:b'ibtljn-64obj291000falseobj292023-11-02T19:03:43.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.497 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.498 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.498 DEBUG:boto:path=/ibtljn-64/?prefix=obj29 2023-11-02T19:06:00.499 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj29 2023-11-02T19:06:00.499 DEBUG:boto:Method: GET 2023-11-02T19:06:00.500 DEBUG:boto:Path: /ibtljn-64/?prefix=obj29 2023-11-02T19:06:00.500 DEBUG:boto:Data: 2023-11-02T19:06:00.500 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.501 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.501 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.502 DEBUG:boto:Params: {} 2023-11-02T19:06:00.502 DEBUG:boto:Token: None 2023-11-02T19:06:00.503 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.503 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e124a30e540b051e-006543f318-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.514 DEBUG:boto:b'ibtljn-64obj291000falseobj292023-11-02T19:03:43.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.515 DEBUG:boto:path=/ibtljn-64/obj29 2023-11-02T19:06:00.515 DEBUG:boto:auth_path=/ibtljn-64/obj29 2023-11-02T19:06:00.515 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.516 DEBUG:boto:Path: /ibtljn-64/obj29 2023-11-02T19:06:00.516 DEBUG:boto:Data: 2023-11-02T19:06:00.517 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.517 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.518 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.518 DEBUG:boto:Params: {} 2023-11-02T19:06:00.519 DEBUG:boto:Token: None 2023-11-02T19:06:00.519 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj29 2023-11-02T19:06:00.520 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DGkm0pQKN3ZW78twU5/yKqWXBkc= 2023-11-02T19:06:00.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DGkm0pQKN3ZW78twU5/yKqWXBkc=', 'Content-Length': '0'} 2023-11-02T19:06:00.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe253b58844c427-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.523 DEBUG:boto:path=/ibtljn-64/obj29 2023-11-02T19:06:00.523 DEBUG:boto:auth_path=/ibtljn-64/obj29 2023-11-02T19:06:00.524 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.524 DEBUG:boto:Path: /ibtljn-64/obj29 2023-11-02T19:06:00.525 DEBUG:boto:Data: 2023-11-02T19:06:00.525 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.525 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.526 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.526 DEBUG:boto:Params: {} 2023-11-02T19:06:00.527 DEBUG:boto:Token: None 2023-11-02T19:06:00.527 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj29 2023-11-02T19:06:00.528 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DGkm0pQKN3ZW78twU5/yKqWXBkc= 2023-11-02T19:06:00.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DGkm0pQKN3ZW78twU5/yKqWXBkc=', 'Content-Length': '0'} 2023-11-02T19:06:00.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c77897465e4b806-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-11-02T19:06:00.532 DEBUG:boto:path=/ibtljn-64/obj29 2023-11-02T19:06:00.532 DEBUG:boto:auth_path=/ibtljn-64/obj29 2023-11-02T19:06:00.533 DEBUG:boto:Method: GET 2023-11-02T19:06:00.533 DEBUG:boto:Path: /ibtljn-64/obj29 2023-11-02T19:06:00.534 DEBUG:boto:Data: 2023-11-02T19:06:00.534 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.535 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.535 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.535 DEBUG:boto:Params: {} 2023-11-02T19:06:00.536 DEBUG:boto:Token: None 2023-11-02T19:06:00.536 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj29 2023-11-02T19:06:00.538 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lGSIk9AkYU6+Mug2/mw3+B9za/c= 2023-11-02T19:06:00.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lGSIk9AkYU6+Mug2/mw3+B9za/c=', 'Content-Length': '0'} 2023-11-02T19:06:00.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065e384b4e0ae6a6d-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.586 DEBUG:boto:path=/ibtljn-64/obj29 2023-11-02T19:06:00.587 DEBUG:boto:auth_path=/ibtljn-64/obj29 2023-11-02T19:06:00.587 DEBUG:boto:Method: GET 2023-11-02T19:06:00.588 DEBUG:boto:Path: /ibtljn-64/obj29 2023-11-02T19:06:00.589 DEBUG:boto:Data: 2023-11-02T19:06:00.589 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.590 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.591 DEBUG:boto:Params: {} 2023-11-02T19:06:00.591 DEBUG:boto:Token: None 2023-11-02T19:06:00.592 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj29 2023-11-02T19:06:00.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lGSIk9AkYU6+Mug2/mw3+B9za/c= 2023-11-02T19:06:00.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lGSIk9AkYU6+Mug2/mw3+B9za/c=', 'Content-Length': '0'} 2023-11-02T19:06:00.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b2049ef9d2bef52-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.597 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.598 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.599 DEBUG:boto:path=/ibtljn-64/?prefix=obj29 2023-11-02T19:06:00.599 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj29 2023-11-02T19:06:00.599 DEBUG:boto:Method: GET 2023-11-02T19:06:00.600 DEBUG:boto:Path: /ibtljn-64/?prefix=obj29 2023-11-02T19:06:00.600 DEBUG:boto:Data: 2023-11-02T19:06:00.601 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.602 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.602 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.603 DEBUG:boto:Params: {} 2023-11-02T19:06:00.603 DEBUG:boto:Token: None 2023-11-02T19:06:00.604 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.605 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da61808d84fb01e4-006543f318-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.612 DEBUG:boto:b'ibtljn-64obj291000falseobj292023-11-02T19:03:43.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.613 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.614 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.614 DEBUG:boto:path=/ibtljn-64/?prefix=obj29 2023-11-02T19:06:00.614 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj29 2023-11-02T19:06:00.615 DEBUG:boto:Method: GET 2023-11-02T19:06:00.615 DEBUG:boto:Path: /ibtljn-64/?prefix=obj29 2023-11-02T19:06:00.616 DEBUG:boto:Data: 2023-11-02T19:06:00.617 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.617 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.618 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.619 DEBUG:boto:Params: {} 2023-11-02T19:06:00.619 DEBUG:boto:Token: None 2023-11-02T19:06:00.620 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fb8e71c09b72866-006543f318-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.631 DEBUG:boto:b'ibtljn-64obj291000falseobj292023-11-02T19:03:43.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T19:06:00.633 DEBUG:boto:path=/ibtljn-64/obj31 2023-11-02T19:06:00.633 DEBUG:boto:auth_path=/ibtljn-64/obj31 2023-11-02T19:06:00.634 DEBUG:boto:path=/ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.634 DEBUG:boto:auth_path=/ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.635 DEBUG:boto:Method: GET 2023-11-02T19:06:00.635 DEBUG:boto:Path: /ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.636 DEBUG:boto:Data: 2023-11-02T19:06:00.637 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.637 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.638 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.638 DEBUG:boto:Params: {} 2023-11-02T19:06:00.639 DEBUG:boto:Token: None 2023-11-02T19:06:00.639 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.640 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1xw6RTmSZDL3/4HN5Z6yAhNxOBo= 2023-11-02T19:06:00.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1xw6RTmSZDL3/4HN5Z6yAhNxOBo=', 'Content-Length': '0'} 2023-11-02T19:06:00.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085284b4e1c3102ac-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.644 DEBUG:boto:path=/ibtljn-64/obj31 2023-11-02T19:06:00.644 DEBUG:boto:auth_path=/ibtljn-64/obj31 2023-11-02T19:06:00.645 DEBUG:boto:path=/ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.645 DEBUG:boto:auth_path=/ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.646 DEBUG:boto:Method: GET 2023-11-02T19:06:00.646 DEBUG:boto:Path: /ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.647 DEBUG:boto:Data: 2023-11-02T19:06:00.648 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.648 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.649 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.650 DEBUG:boto:Params: {} 2023-11-02T19:06:00.650 DEBUG:boto:Token: None 2023-11-02T19:06:00.651 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.652 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1xw6RTmSZDL3/4HN5Z6yAhNxOBo= 2023-11-02T19:06:00.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1xw6RTmSZDL3/4HN5Z6yAhNxOBo=', 'Content-Length': '0'} 2023-11-02T19:06:00.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005effa164c57cbe3c-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.657 DEBUG:boto:path=/ibtljn-64/obj31 2023-11-02T19:06:00.658 DEBUG:boto:auth_path=/ibtljn-64/obj31 2023-11-02T19:06:00.658 DEBUG:boto:path=/ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.659 DEBUG:boto:auth_path=/ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.659 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.660 DEBUG:boto:Path: /ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.660 DEBUG:boto:Data: 2023-11-02T19:06:00.661 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.661 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.662 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.662 DEBUG:boto:Params: {} 2023-11-02T19:06:00.663 DEBUG:boto:Token: None 2023-11-02T19:06:00.663 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.664 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:umeQ7p4vLHNp54sQqqUIGqmLdhE= 2023-11-02T19:06:00.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:umeQ7p4vLHNp54sQqqUIGqmLdhE=', 'Content-Length': '0'} 2023-11-02T19:06:00.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007352f8054ae840a0-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.669 DEBUG:boto:path=/ibtljn-64/obj31 2023-11-02T19:06:00.670 DEBUG:boto:auth_path=/ibtljn-64/obj31 2023-11-02T19:06:00.671 DEBUG:boto:path=/ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.672 DEBUG:boto:auth_path=/ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.672 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.673 DEBUG:boto:Path: /ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.673 DEBUG:boto:Data: 2023-11-02T19:06:00.674 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.675 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.675 DEBUG:boto:Params: {} 2023-11-02T19:06:00.676 DEBUG:boto:Token: None 2023-11-02T19:06:00.676 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.677 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:umeQ7p4vLHNp54sQqqUIGqmLdhE= 2023-11-02T19:06:00.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:umeQ7p4vLHNp54sQqqUIGqmLdhE=', 'Content-Length': '0'} 2023-11-02T19:06:00.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000671c86ca673a480b-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T19:06:00.681 DEBUG:boto:path=/ibtljn-64/obj31 2023-11-02T19:06:00.682 DEBUG:boto:auth_path=/ibtljn-64/obj31 2023-11-02T19:06:00.682 DEBUG:boto:path=/ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.683 DEBUG:boto:auth_path=/ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.683 DEBUG:boto:Method: GET 2023-11-02T19:06:00.684 DEBUG:boto:Path: /ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.685 DEBUG:boto:Data: 2023-11-02T19:06:00.685 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.686 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.686 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.686 DEBUG:boto:Params: {} 2023-11-02T19:06:00.687 DEBUG:boto:Token: None 2023-11-02T19:06:00.688 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.688 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1xw6RTmSZDL3/4HN5Z6yAhNxOBo= 2023-11-02T19:06:00.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1xw6RTmSZDL3/4HN5Z6yAhNxOBo=', 'Content-Length': '0'} 2023-11-02T19:06:00.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abc6bc1bd9b7717b-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.734 DEBUG:boto:path=/ibtljn-64/obj31 2023-11-02T19:06:00.735 DEBUG:boto:auth_path=/ibtljn-64/obj31 2023-11-02T19:06:00.736 DEBUG:boto:path=/ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.736 DEBUG:boto:auth_path=/ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.737 DEBUG:boto:Method: GET 2023-11-02T19:06:00.738 DEBUG:boto:Path: /ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.739 DEBUG:boto:Data: 2023-11-02T19:06:00.739 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.740 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.741 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.742 DEBUG:boto:Params: {} 2023-11-02T19:06:00.743 DEBUG:boto:Token: None 2023-11-02T19:06:00.743 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj31?versionId=null 2023-11-02T19:06:00.744 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1xw6RTmSZDL3/4HN5Z6yAhNxOBo= 2023-11-02T19:06:00.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1xw6RTmSZDL3/4HN5Z6yAhNxOBo=', 'Content-Length': '0'} 2023-11-02T19:06:00.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adc15476a81dcd2b-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.748 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.749 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.751 DEBUG:boto:path=/ibtljn-64/?prefix=obj31 2023-11-02T19:06:00.752 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj31 2023-11-02T19:06:00.752 DEBUG:boto:Method: GET 2023-11-02T19:06:00.753 DEBUG:boto:Path: /ibtljn-64/?prefix=obj31 2023-11-02T19:06:00.753 DEBUG:boto:Data: 2023-11-02T19:06:00.754 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.755 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.755 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.756 DEBUG:boto:Params: {} 2023-11-02T19:06:00.757 DEBUG:boto:Token: None 2023-11-02T19:06:00.757 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.758 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092e8ee8b29687769-006543f318-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.764 DEBUG:boto:b'ibtljn-64obj311000falseobj312023-11-02T19:03:44.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.765 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.766 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.766 DEBUG:boto:path=/ibtljn-64/?prefix=obj31 2023-11-02T19:06:00.766 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj31 2023-11-02T19:06:00.767 DEBUG:boto:Method: GET 2023-11-02T19:06:00.767 DEBUG:boto:Path: /ibtljn-64/?prefix=obj31 2023-11-02T19:06:00.768 DEBUG:boto:Data: 2023-11-02T19:06:00.768 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.769 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.769 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.770 DEBUG:boto:Params: {} 2023-11-02T19:06:00.770 DEBUG:boto:Token: None 2023-11-02T19:06:00.771 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.771 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ce684281044b591-006543f318-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.822 DEBUG:boto:b'ibtljn-64obj311000falseobj312023-11-02T19:03:44.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.823 DEBUG:boto:path=/ibtljn-64/obj31 2023-11-02T19:06:00.824 DEBUG:boto:auth_path=/ibtljn-64/obj31 2023-11-02T19:06:00.824 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.825 DEBUG:boto:Path: /ibtljn-64/obj31 2023-11-02T19:06:00.825 DEBUG:boto:Data: 2023-11-02T19:06:00.826 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.826 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.826 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.827 DEBUG:boto:Params: {} 2023-11-02T19:06:00.827 DEBUG:boto:Token: None 2023-11-02T19:06:00.828 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj31 2023-11-02T19:06:00.828 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G9F9MueyQRrseJvjEKCjRkO4Cvk= 2023-11-02T19:06:00.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G9F9MueyQRrseJvjEKCjRkO4Cvk=', 'Content-Length': '0'} 2023-11-02T19:06:00.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071c601feb6438c02-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.833 DEBUG:boto:path=/ibtljn-64/obj31 2023-11-02T19:06:00.833 DEBUG:boto:auth_path=/ibtljn-64/obj31 2023-11-02T19:06:00.833 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.834 DEBUG:boto:Path: /ibtljn-64/obj31 2023-11-02T19:06:00.834 DEBUG:boto:Data: 2023-11-02T19:06:00.835 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.835 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.836 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.836 DEBUG:boto:Params: {} 2023-11-02T19:06:00.837 DEBUG:boto:Token: None 2023-11-02T19:06:00.837 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj31 2023-11-02T19:06:00.837 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G9F9MueyQRrseJvjEKCjRkO4Cvk= 2023-11-02T19:06:00.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G9F9MueyQRrseJvjEKCjRkO4Cvk=', 'Content-Length': '0'} 2023-11-02T19:06:00.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089d1224f5b04f263-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-11-02T19:06:00.842 DEBUG:boto:path=/ibtljn-64/obj31 2023-11-02T19:06:00.842 DEBUG:boto:auth_path=/ibtljn-64/obj31 2023-11-02T19:06:00.843 DEBUG:boto:Method: GET 2023-11-02T19:06:00.843 DEBUG:boto:Path: /ibtljn-64/obj31 2023-11-02T19:06:00.844 DEBUG:boto:Data: 2023-11-02T19:06:00.844 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.845 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.845 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.846 DEBUG:boto:Params: {} 2023-11-02T19:06:00.846 DEBUG:boto:Token: None 2023-11-02T19:06:00.847 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj31 2023-11-02T19:06:00.847 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tIIdV4hthE1U0X6Oq0nLaIh+F1s= 2023-11-02T19:06:00.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tIIdV4hthE1U0X6Oq0nLaIh+F1s=', 'Content-Length': '0'} 2023-11-02T19:06:00.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a61432c3d7d36e8d-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.852 DEBUG:boto:path=/ibtljn-64/obj31 2023-11-02T19:06:00.852 DEBUG:boto:auth_path=/ibtljn-64/obj31 2023-11-02T19:06:00.853 DEBUG:boto:Method: GET 2023-11-02T19:06:00.853 DEBUG:boto:Path: /ibtljn-64/obj31 2023-11-02T19:06:00.853 DEBUG:boto:Data: 2023-11-02T19:06:00.854 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.854 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.855 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.855 DEBUG:boto:Params: {} 2023-11-02T19:06:00.856 DEBUG:boto:Token: None 2023-11-02T19:06:00.856 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj31 2023-11-02T19:06:00.857 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tIIdV4hthE1U0X6Oq0nLaIh+F1s= 2023-11-02T19:06:00.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tIIdV4hthE1U0X6Oq0nLaIh+F1s=', 'Content-Length': '0'} 2023-11-02T19:06:00.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de9b67b6df0214b9-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.861 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.861 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.862 DEBUG:boto:path=/ibtljn-64/?prefix=obj31 2023-11-02T19:06:00.862 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj31 2023-11-02T19:06:00.863 DEBUG:boto:Method: GET 2023-11-02T19:06:00.863 DEBUG:boto:Path: /ibtljn-64/?prefix=obj31 2023-11-02T19:06:00.864 DEBUG:boto:Data: 2023-11-02T19:06:00.864 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.864 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.865 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.865 DEBUG:boto:Params: {} 2023-11-02T19:06:00.866 DEBUG:boto:Token: None 2023-11-02T19:06:00.866 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.867 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e819df1054b51c9-006543f318-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.914 DEBUG:boto:b'ibtljn-64obj311000falseobj312023-11-02T19:03:44.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.915 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:00.916 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:00.916 DEBUG:boto:path=/ibtljn-64/?prefix=obj31 2023-11-02T19:06:00.916 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj31 2023-11-02T19:06:00.917 DEBUG:boto:Method: GET 2023-11-02T19:06:00.917 DEBUG:boto:Path: /ibtljn-64/?prefix=obj31 2023-11-02T19:06:00.918 DEBUG:boto:Data: 2023-11-02T19:06:00.918 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.919 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.919 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.920 DEBUG:boto:Params: {} 2023-11-02T19:06:00.920 DEBUG:boto:Token: None 2023-11-02T19:06:00.921 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/ 2023-11-02T19:06:00.921 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s= 2023-11-02T19:06:00.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/wvE0TaWDX27w3NZLBUhv+GP/s=', 'Content-Length': '0'} 2023-11-02T19:06:00.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bef3f9b80b65a58-006543f318-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.932 DEBUG:boto:b'ibtljn-64obj311000falseobj312023-11-02T19:03:44.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:00.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T19:06:00.933 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:06:00.934 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:06:00.934 DEBUG:boto:path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.935 DEBUG:boto:auth_path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.935 DEBUG:boto:Method: GET 2023-11-02T19:06:00.936 DEBUG:boto:Path: /ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.936 DEBUG:boto:Data: 2023-11-02T19:06:00.937 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.937 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.937 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.938 DEBUG:boto:Params: {} 2023-11-02T19:06:00.938 DEBUG:boto:Token: None 2023-11-02T19:06:00.939 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.940 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:07W2MnpNoUY4kPFOEYrhtnBfOtY= 2023-11-02T19:06:00.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:07W2MnpNoUY4kPFOEYrhtnBfOtY=', 'Content-Length': '0'} 2023-11-02T19:06:00.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007edf317762d63d6c-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.945 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:06:00.946 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:06:00.946 DEBUG:boto:path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.946 DEBUG:boto:auth_path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.947 DEBUG:boto:Method: GET 2023-11-02T19:06:00.947 DEBUG:boto:Path: /ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.948 DEBUG:boto:Data: 2023-11-02T19:06:00.948 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.949 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.949 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.950 DEBUG:boto:Params: {} 2023-11-02T19:06:00.950 DEBUG:boto:Token: None 2023-11-02T19:06:00.951 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.951 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:07W2MnpNoUY4kPFOEYrhtnBfOtY= 2023-11-02T19:06:00.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:07W2MnpNoUY4kPFOEYrhtnBfOtY=', 'Content-Length': '0'} 2023-11-02T19:06:00.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c89c771e117bc9-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.955 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:06:00.956 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:06:00.956 DEBUG:boto:path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.957 DEBUG:boto:auth_path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.957 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.957 DEBUG:boto:Path: /ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.958 DEBUG:boto:Data: 2023-11-02T19:06:00.958 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.959 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.959 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.959 DEBUG:boto:Params: {} 2023-11-02T19:06:00.960 DEBUG:boto:Token: None 2023-11-02T19:06:00.960 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.961 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6OImG/iSUFYesYVRoO0jPzn61Mc= 2023-11-02T19:06:00.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6OImG/iSUFYesYVRoO0jPzn61Mc=', 'Content-Length': '0'} 2023-11-02T19:06:00.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009142e7b2b9e9b1a3-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.965 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:06:00.965 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:06:00.966 DEBUG:boto:path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.966 DEBUG:boto:auth_path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.967 DEBUG:boto:Method: HEAD 2023-11-02T19:06:00.967 DEBUG:boto:Path: /ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.967 DEBUG:boto:Data: 2023-11-02T19:06:00.968 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.968 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.969 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.969 DEBUG:boto:Params: {} 2023-11-02T19:06:00.970 DEBUG:boto:Token: None 2023-11-02T19:06:00.970 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.971 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6OImG/iSUFYesYVRoO0jPzn61Mc= 2023-11-02T19:06:00.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6OImG/iSUFYesYVRoO0jPzn61Mc=', 'Content-Length': '0'} 2023-11-02T19:06:00.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c4b6d0f66a0fe7-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T19:06:00.975 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:06:00.976 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:06:00.976 DEBUG:boto:path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.977 DEBUG:boto:auth_path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.977 DEBUG:boto:Method: GET 2023-11-02T19:06:00.978 DEBUG:boto:Path: /ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.978 DEBUG:boto:Data: 2023-11-02T19:06:00.978 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.979 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:00.979 DEBUG:boto:Port: 8001 2023-11-02T19:06:00.980 DEBUG:boto:Params: {} 2023-11-02T19:06:00.980 DEBUG:boto:Token: None 2023-11-02T19:06:00.981 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.981 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:07W2MnpNoUY4kPFOEYrhtnBfOtY= 2023-11-02T19:06:00.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:07W2MnpNoUY4kPFOEYrhtnBfOtY=', 'Content-Length': '0'} 2023-11-02T19:06:00.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f62167db3dd1268d-006543f318-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:00.986 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:06:00.986 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:06:00.987 DEBUG:boto:path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.987 DEBUG:boto:auth_path=/ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.988 DEBUG:boto:Method: GET 2023-11-02T19:06:00.988 DEBUG:boto:Path: /ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.988 DEBUG:boto:Data: 2023-11-02T19:06:00.989 DEBUG:boto:Headers: {} 2023-11-02T19:06:00.989 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:00.990 DEBUG:boto:Port: 8000 2023-11-02T19:06:00.990 DEBUG:boto:Params: {} 2023-11-02T19:06:00.991 DEBUG:boto:Token: None 2023-11-02T19:06:00.991 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:00 GMT /ibtljn-64/obj33?versionId=null 2023-11-02T19:06:00.992 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:07W2MnpNoUY4kPFOEYrhtnBfOtY= 2023-11-02T19:06:00.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:07W2MnpNoUY4kPFOEYrhtnBfOtY=', 'Content-Length': '0'} 2023-11-02T19:06:00.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca724c1d020451e-006543f318-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.042 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.043 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.043 DEBUG:boto:path=/ibtljn-64/?prefix=obj33 2023-11-02T19:06:01.044 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj33 2023-11-02T19:06:01.044 DEBUG:boto:Method: GET 2023-11-02T19:06:01.045 DEBUG:boto:Path: /ibtljn-64/?prefix=obj33 2023-11-02T19:06:01.045 DEBUG:boto:Data: 2023-11-02T19:06:01.046 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.046 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.047 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.047 DEBUG:boto:Params: {} 2023-11-02T19:06:01.048 DEBUG:boto:Token: None 2023-11-02T19:06:01.048 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.049 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000616f4e6a9f4ed678-006543f319-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.055 DEBUG:boto:b'ibtljn-64obj331000falseobj332023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.055 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.056 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.056 DEBUG:boto:path=/ibtljn-64/?prefix=obj33 2023-11-02T19:06:01.057 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj33 2023-11-02T19:06:01.057 DEBUG:boto:Method: GET 2023-11-02T19:06:01.058 DEBUG:boto:Path: /ibtljn-64/?prefix=obj33 2023-11-02T19:06:01.058 DEBUG:boto:Data: 2023-11-02T19:06:01.058 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.059 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.059 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.060 DEBUG:boto:Params: {} 2023-11-02T19:06:01.060 DEBUG:boto:Token: None 2023-11-02T19:06:01.061 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.061 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5f8b785dfd26d2c-006543f319-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.072 DEBUG:boto:b'ibtljn-64obj331000falseobj332023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.072 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:06:01.073 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:06:01.074 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.074 DEBUG:boto:Path: /ibtljn-64/obj33 2023-11-02T19:06:01.075 DEBUG:boto:Data: 2023-11-02T19:06:01.075 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.075 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.076 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.076 DEBUG:boto:Params: {} 2023-11-02T19:06:01.077 DEBUG:boto:Token: None 2023-11-02T19:06:01.077 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj33 2023-11-02T19:06:01.077 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WQ1mqkfhFumSDcJzuHeSykQRMYw= 2023-11-02T19:06:01.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WQ1mqkfhFumSDcJzuHeSykQRMYw=', 'Content-Length': '0'} 2023-11-02T19:06:01.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009652c83e5068434d-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.081 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:06:01.081 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:06:01.082 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.082 DEBUG:boto:Path: /ibtljn-64/obj33 2023-11-02T19:06:01.083 DEBUG:boto:Data: 2023-11-02T19:06:01.083 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.083 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.084 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.084 DEBUG:boto:Params: {} 2023-11-02T19:06:01.085 DEBUG:boto:Token: None 2023-11-02T19:06:01.085 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj33 2023-11-02T19:06:01.086 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WQ1mqkfhFumSDcJzuHeSykQRMYw= 2023-11-02T19:06:01.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WQ1mqkfhFumSDcJzuHeSykQRMYw=', 'Content-Length': '0'} 2023-11-02T19:06:01.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001284ddedd7f94571-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-11-02T19:06:01.090 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:06:01.090 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:06:01.091 DEBUG:boto:Method: GET 2023-11-02T19:06:01.091 DEBUG:boto:Path: /ibtljn-64/obj33 2023-11-02T19:06:01.092 DEBUG:boto:Data: 2023-11-02T19:06:01.092 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.093 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.093 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.093 DEBUG:boto:Params: {} 2023-11-02T19:06:01.094 DEBUG:boto:Token: None 2023-11-02T19:06:01.094 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj33 2023-11-02T19:06:01.095 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CTCJFgLVSfcXXA277D0q0NVedDI= 2023-11-02T19:06:01.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CTCJFgLVSfcXXA277D0q0NVedDI=', 'Content-Length': '0'} 2023-11-02T19:06:01.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d21af72d084ae202-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.142 DEBUG:boto:path=/ibtljn-64/obj33 2023-11-02T19:06:01.143 DEBUG:boto:auth_path=/ibtljn-64/obj33 2023-11-02T19:06:01.143 DEBUG:boto:Method: GET 2023-11-02T19:06:01.144 DEBUG:boto:Path: /ibtljn-64/obj33 2023-11-02T19:06:01.144 DEBUG:boto:Data: 2023-11-02T19:06:01.145 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.145 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.146 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.146 DEBUG:boto:Params: {} 2023-11-02T19:06:01.147 DEBUG:boto:Token: None 2023-11-02T19:06:01.147 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj33 2023-11-02T19:06:01.148 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CTCJFgLVSfcXXA277D0q0NVedDI= 2023-11-02T19:06:01.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CTCJFgLVSfcXXA277D0q0NVedDI=', 'Content-Length': '0'} 2023-11-02T19:06:01.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4ab4c6bc1a1cfab-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.152 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.153 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.154 DEBUG:boto:path=/ibtljn-64/?prefix=obj33 2023-11-02T19:06:01.154 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj33 2023-11-02T19:06:01.155 DEBUG:boto:Method: GET 2023-11-02T19:06:01.155 DEBUG:boto:Path: /ibtljn-64/?prefix=obj33 2023-11-02T19:06:01.156 DEBUG:boto:Data: 2023-11-02T19:06:01.156 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.157 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.157 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.158 DEBUG:boto:Params: {} 2023-11-02T19:06:01.158 DEBUG:boto:Token: None 2023-11-02T19:06:01.159 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.159 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a34829bda03abdf3-006543f319-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.166 DEBUG:boto:b'ibtljn-64obj331000falseobj332023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.168 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.168 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.169 DEBUG:boto:path=/ibtljn-64/?prefix=obj33 2023-11-02T19:06:01.169 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj33 2023-11-02T19:06:01.170 DEBUG:boto:Method: GET 2023-11-02T19:06:01.171 DEBUG:boto:Path: /ibtljn-64/?prefix=obj33 2023-11-02T19:06:01.171 DEBUG:boto:Data: 2023-11-02T19:06:01.171 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.172 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.172 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.173 DEBUG:boto:Params: {} 2023-11-02T19:06:01.173 DEBUG:boto:Token: None 2023-11-02T19:06:01.174 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.174 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050a30d343ee507fc-006543f319-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.226 DEBUG:boto:b'ibtljn-64obj331000falseobj332023-11-02T19:03:19.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T19:06:01.229 DEBUG:boto:path=/ibtljn-64/obj34 2023-11-02T19:06:01.229 DEBUG:boto:auth_path=/ibtljn-64/obj34 2023-11-02T19:06:01.231 DEBUG:boto:path=/ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.231 DEBUG:boto:auth_path=/ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.232 DEBUG:boto:Method: GET 2023-11-02T19:06:01.233 DEBUG:boto:Path: /ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.233 DEBUG:boto:Data: 2023-11-02T19:06:01.234 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.235 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.235 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.236 DEBUG:boto:Params: {} 2023-11-02T19:06:01.237 DEBUG:boto:Token: None 2023-11-02T19:06:01.237 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.238 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pRyiPkbkvAcilvSpOW39+tjKErA= 2023-11-02T19:06:01.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pRyiPkbkvAcilvSpOW39+tjKErA=', 'Content-Length': '0'} 2023-11-02T19:06:01.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e81be3eb5dca2bf-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.242 DEBUG:boto:path=/ibtljn-64/obj34 2023-11-02T19:06:01.243 DEBUG:boto:auth_path=/ibtljn-64/obj34 2023-11-02T19:06:01.243 DEBUG:boto:path=/ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.244 DEBUG:boto:auth_path=/ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.244 DEBUG:boto:Method: GET 2023-11-02T19:06:01.245 DEBUG:boto:Path: /ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.245 DEBUG:boto:Data: 2023-11-02T19:06:01.246 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.247 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.247 DEBUG:boto:Params: {} 2023-11-02T19:06:01.248 DEBUG:boto:Token: None 2023-11-02T19:06:01.249 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.249 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pRyiPkbkvAcilvSpOW39+tjKErA= 2023-11-02T19:06:01.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pRyiPkbkvAcilvSpOW39+tjKErA=', 'Content-Length': '0'} 2023-11-02T19:06:01.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c6b85173c31eb48-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.254 DEBUG:boto:path=/ibtljn-64/obj34 2023-11-02T19:06:01.254 DEBUG:boto:auth_path=/ibtljn-64/obj34 2023-11-02T19:06:01.255 DEBUG:boto:path=/ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.255 DEBUG:boto:auth_path=/ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.255 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.256 DEBUG:boto:Path: /ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.257 DEBUG:boto:Data: 2023-11-02T19:06:01.257 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.257 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.258 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.258 DEBUG:boto:Params: {} 2023-11-02T19:06:01.259 DEBUG:boto:Token: None 2023-11-02T19:06:01.259 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.260 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xzJPKAiJFlFc+zPoudp5uKM8dBQ= 2023-11-02T19:06:01.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xzJPKAiJFlFc+zPoudp5uKM8dBQ=', 'Content-Length': '0'} 2023-11-02T19:06:01.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000293edfbe18ab9486-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.264 DEBUG:boto:path=/ibtljn-64/obj34 2023-11-02T19:06:01.265 DEBUG:boto:auth_path=/ibtljn-64/obj34 2023-11-02T19:06:01.265 DEBUG:boto:path=/ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.266 DEBUG:boto:auth_path=/ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.266 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.266 DEBUG:boto:Path: /ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.267 DEBUG:boto:Data: 2023-11-02T19:06:01.267 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.268 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.268 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.269 DEBUG:boto:Params: {} 2023-11-02T19:06:01.269 DEBUG:boto:Token: None 2023-11-02T19:06:01.270 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.270 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xzJPKAiJFlFc+zPoudp5uKM8dBQ= 2023-11-02T19:06:01.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xzJPKAiJFlFc+zPoudp5uKM8dBQ=', 'Content-Length': '0'} 2023-11-02T19:06:01.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de3e13a482024f80-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T19:06:01.276 DEBUG:boto:path=/ibtljn-64/obj34 2023-11-02T19:06:01.276 DEBUG:boto:auth_path=/ibtljn-64/obj34 2023-11-02T19:06:01.277 DEBUG:boto:path=/ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.278 DEBUG:boto:auth_path=/ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.279 DEBUG:boto:Method: GET 2023-11-02T19:06:01.279 DEBUG:boto:Path: /ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.280 DEBUG:boto:Data: 2023-11-02T19:06:01.281 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.281 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.282 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.282 DEBUG:boto:Params: {} 2023-11-02T19:06:01.283 DEBUG:boto:Token: None 2023-11-02T19:06:01.284 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.285 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pRyiPkbkvAcilvSpOW39+tjKErA= 2023-11-02T19:06:01.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pRyiPkbkvAcilvSpOW39+tjKErA=', 'Content-Length': '0'} 2023-11-02T19:06:01.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7e881fee3be7eaa-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.290 DEBUG:boto:path=/ibtljn-64/obj34 2023-11-02T19:06:01.291 DEBUG:boto:auth_path=/ibtljn-64/obj34 2023-11-02T19:06:01.291 DEBUG:boto:path=/ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.291 DEBUG:boto:auth_path=/ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.292 DEBUG:boto:Method: GET 2023-11-02T19:06:01.293 DEBUG:boto:Path: /ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.293 DEBUG:boto:Data: 2023-11-02T19:06:01.293 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.294 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.295 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.295 DEBUG:boto:Params: {} 2023-11-02T19:06:01.296 DEBUG:boto:Token: None 2023-11-02T19:06:01.296 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj34?versionId=null 2023-11-02T19:06:01.297 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pRyiPkbkvAcilvSpOW39+tjKErA= 2023-11-02T19:06:01.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pRyiPkbkvAcilvSpOW39+tjKErA=', 'Content-Length': '0'} 2023-11-02T19:06:01.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f543f8242da435c4-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.302 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.302 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.303 DEBUG:boto:path=/ibtljn-64/?prefix=obj34 2023-11-02T19:06:01.303 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj34 2023-11-02T19:06:01.303 DEBUG:boto:Method: GET 2023-11-02T19:06:01.304 DEBUG:boto:Path: /ibtljn-64/?prefix=obj34 2023-11-02T19:06:01.304 DEBUG:boto:Data: 2023-11-02T19:06:01.305 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.305 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.306 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.306 DEBUG:boto:Params: {} 2023-11-02T19:06:01.307 DEBUG:boto:Token: None 2023-11-02T19:06:01.307 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.308 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a2dae73d179d3f9-006543f319-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.313 DEBUG:boto:b'ibtljn-64obj341000falseobj342023-11-02T19:03:42.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.314 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.314 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.315 DEBUG:boto:path=/ibtljn-64/?prefix=obj34 2023-11-02T19:06:01.315 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj34 2023-11-02T19:06:01.315 DEBUG:boto:Method: GET 2023-11-02T19:06:01.316 DEBUG:boto:Path: /ibtljn-64/?prefix=obj34 2023-11-02T19:06:01.316 DEBUG:boto:Data: 2023-11-02T19:06:01.317 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.317 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.318 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.318 DEBUG:boto:Params: {} 2023-11-02T19:06:01.318 DEBUG:boto:Token: None 2023-11-02T19:06:01.319 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.320 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002280c6225e9d5283-006543f319-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.329 DEBUG:boto:b'ibtljn-64obj341000falseobj342023-11-02T19:03:42.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.329 DEBUG:boto:path=/ibtljn-64/obj34 2023-11-02T19:06:01.330 DEBUG:boto:auth_path=/ibtljn-64/obj34 2023-11-02T19:06:01.330 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.331 DEBUG:boto:Path: /ibtljn-64/obj34 2023-11-02T19:06:01.331 DEBUG:boto:Data: 2023-11-02T19:06:01.331 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.332 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.332 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.333 DEBUG:boto:Params: {} 2023-11-02T19:06:01.333 DEBUG:boto:Token: None 2023-11-02T19:06:01.334 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj34 2023-11-02T19:06:01.334 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7jj03sa3srkJnKitrfUqBexr6cs= 2023-11-02T19:06:01.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7jj03sa3srkJnKitrfUqBexr6cs=', 'Content-Length': '0'} 2023-11-02T19:06:01.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba42ca2e04fb17d-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.337 DEBUG:boto:path=/ibtljn-64/obj34 2023-11-02T19:06:01.338 DEBUG:boto:auth_path=/ibtljn-64/obj34 2023-11-02T19:06:01.338 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.339 DEBUG:boto:Path: /ibtljn-64/obj34 2023-11-02T19:06:01.339 DEBUG:boto:Data: 2023-11-02T19:06:01.340 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.340 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.340 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.341 DEBUG:boto:Params: {} 2023-11-02T19:06:01.341 DEBUG:boto:Token: None 2023-11-02T19:06:01.342 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj34 2023-11-02T19:06:01.342 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7jj03sa3srkJnKitrfUqBexr6cs= 2023-11-02T19:06:01.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7jj03sa3srkJnKitrfUqBexr6cs=', 'Content-Length': '0'} 2023-11-02T19:06:01.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cca21652185ab8e-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-11-02T19:06:01.346 DEBUG:boto:path=/ibtljn-64/obj34 2023-11-02T19:06:01.346 DEBUG:boto:auth_path=/ibtljn-64/obj34 2023-11-02T19:06:01.347 DEBUG:boto:Method: GET 2023-11-02T19:06:01.347 DEBUG:boto:Path: /ibtljn-64/obj34 2023-11-02T19:06:01.348 DEBUG:boto:Data: 2023-11-02T19:06:01.348 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.349 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.349 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.349 DEBUG:boto:Params: {} 2023-11-02T19:06:01.350 DEBUG:boto:Token: None 2023-11-02T19:06:01.350 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj34 2023-11-02T19:06:01.351 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IC0mTRmBwp+Hc2XRCu8Vm2ZWlm4= 2023-11-02T19:06:01.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IC0mTRmBwp+Hc2XRCu8Vm2ZWlm4=', 'Content-Length': '0'} 2023-11-02T19:06:01.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e03ef12307af9e1-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.394 DEBUG:boto:path=/ibtljn-64/obj34 2023-11-02T19:06:01.395 DEBUG:boto:auth_path=/ibtljn-64/obj34 2023-11-02T19:06:01.396 DEBUG:boto:Method: GET 2023-11-02T19:06:01.396 DEBUG:boto:Path: /ibtljn-64/obj34 2023-11-02T19:06:01.397 DEBUG:boto:Data: 2023-11-02T19:06:01.397 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.398 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.399 DEBUG:boto:Params: {} 2023-11-02T19:06:01.399 DEBUG:boto:Token: None 2023-11-02T19:06:01.400 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj34 2023-11-02T19:06:01.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IC0mTRmBwp+Hc2XRCu8Vm2ZWlm4= 2023-11-02T19:06:01.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IC0mTRmBwp+Hc2XRCu8Vm2ZWlm4=', 'Content-Length': '0'} 2023-11-02T19:06:01.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b9e78005337b218-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.405 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.405 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.406 DEBUG:boto:path=/ibtljn-64/?prefix=obj34 2023-11-02T19:06:01.406 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj34 2023-11-02T19:06:01.409 DEBUG:boto:Method: GET 2023-11-02T19:06:01.409 DEBUG:boto:Path: /ibtljn-64/?prefix=obj34 2023-11-02T19:06:01.410 DEBUG:boto:Data: 2023-11-02T19:06:01.410 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.411 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.411 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.412 DEBUG:boto:Params: {} 2023-11-02T19:06:01.412 DEBUG:boto:Token: None 2023-11-02T19:06:01.413 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000534c50e7a1773574-006543f319-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.419 DEBUG:boto:b'ibtljn-64obj341000falseobj342023-11-02T19:03:42.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.420 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.420 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.421 DEBUG:boto:path=/ibtljn-64/?prefix=obj34 2023-11-02T19:06:01.421 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj34 2023-11-02T19:06:01.422 DEBUG:boto:Method: GET 2023-11-02T19:06:01.422 DEBUG:boto:Path: /ibtljn-64/?prefix=obj34 2023-11-02T19:06:01.423 DEBUG:boto:Data: 2023-11-02T19:06:01.423 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.423 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.424 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.424 DEBUG:boto:Params: {} 2023-11-02T19:06:01.425 DEBUG:boto:Token: None 2023-11-02T19:06:01.425 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.426 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e6f7dad398b320e-006543f319-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.478 DEBUG:boto:b'ibtljn-64obj341000falseobj342023-11-02T19:03:42.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-11-02T19:06:01.480 DEBUG:boto:path=/ibtljn-64/obj37 2023-11-02T19:06:01.480 DEBUG:boto:auth_path=/ibtljn-64/obj37 2023-11-02T19:06:01.481 DEBUG:boto:path=/ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.481 DEBUG:boto:auth_path=/ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.482 DEBUG:boto:Method: GET 2023-11-02T19:06:01.482 DEBUG:boto:Path: /ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.483 DEBUG:boto:Data: 2023-11-02T19:06:01.483 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.484 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.484 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.485 DEBUG:boto:Params: {} 2023-11-02T19:06:01.485 DEBUG:boto:Token: None 2023-11-02T19:06:01.486 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.487 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H/bOwdsKu+UmdzFJTtydrCbKPMI= 2023-11-02T19:06:01.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H/bOwdsKu+UmdzFJTtydrCbKPMI=', 'Content-Length': '0'} 2023-11-02T19:06:01.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000511bd62f30a75060-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.492 DEBUG:boto:path=/ibtljn-64/obj37 2023-11-02T19:06:01.493 DEBUG:boto:auth_path=/ibtljn-64/obj37 2023-11-02T19:06:01.493 DEBUG:boto:path=/ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.494 DEBUG:boto:auth_path=/ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.494 DEBUG:boto:Method: GET 2023-11-02T19:06:01.495 DEBUG:boto:Path: /ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.496 DEBUG:boto:Data: 2023-11-02T19:06:01.496 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.497 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.497 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.498 DEBUG:boto:Params: {} 2023-11-02T19:06:01.499 DEBUG:boto:Token: None 2023-11-02T19:06:01.500 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.501 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H/bOwdsKu+UmdzFJTtydrCbKPMI= 2023-11-02T19:06:01.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H/bOwdsKu+UmdzFJTtydrCbKPMI=', 'Content-Length': '0'} 2023-11-02T19:06:01.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef6013f7dc01755c-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.507 DEBUG:boto:path=/ibtljn-64/obj37 2023-11-02T19:06:01.507 DEBUG:boto:auth_path=/ibtljn-64/obj37 2023-11-02T19:06:01.508 DEBUG:boto:path=/ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.509 DEBUG:boto:auth_path=/ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.510 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.510 DEBUG:boto:Path: /ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.511 DEBUG:boto:Data: 2023-11-02T19:06:01.512 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.513 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.513 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.514 DEBUG:boto:Params: {} 2023-11-02T19:06:01.515 DEBUG:boto:Token: None 2023-11-02T19:06:01.516 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.517 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WW6Mj4dIYFVP1Mr80w3iugguqyc= 2023-11-02T19:06:01.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WW6Mj4dIYFVP1Mr80w3iugguqyc=', 'Content-Length': '0'} 2023-11-02T19:06:01.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e651e5c8fc46057c-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.522 DEBUG:boto:path=/ibtljn-64/obj37 2023-11-02T19:06:01.523 DEBUG:boto:auth_path=/ibtljn-64/obj37 2023-11-02T19:06:01.523 DEBUG:boto:path=/ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.524 DEBUG:boto:auth_path=/ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.524 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.524 DEBUG:boto:Path: /ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.525 DEBUG:boto:Data: 2023-11-02T19:06:01.525 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.526 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.526 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.526 DEBUG:boto:Params: {} 2023-11-02T19:06:01.527 DEBUG:boto:Token: None 2023-11-02T19:06:01.527 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.528 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WW6Mj4dIYFVP1Mr80w3iugguqyc= 2023-11-02T19:06:01.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WW6Mj4dIYFVP1Mr80w3iugguqyc=', 'Content-Length': '0'} 2023-11-02T19:06:01.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e3740a4c20ee867-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-11-02T19:06:01.533 DEBUG:boto:path=/ibtljn-64/obj37 2023-11-02T19:06:01.533 DEBUG:boto:auth_path=/ibtljn-64/obj37 2023-11-02T19:06:01.534 DEBUG:boto:path=/ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.534 DEBUG:boto:auth_path=/ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.535 DEBUG:boto:Method: GET 2023-11-02T19:06:01.535 DEBUG:boto:Path: /ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.536 DEBUG:boto:Data: 2023-11-02T19:06:01.536 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.537 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.537 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.537 DEBUG:boto:Params: {} 2023-11-02T19:06:01.538 DEBUG:boto:Token: None 2023-11-02T19:06:01.539 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.539 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H/bOwdsKu+UmdzFJTtydrCbKPMI= 2023-11-02T19:06:01.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H/bOwdsKu+UmdzFJTtydrCbKPMI=', 'Content-Length': '0'} 2023-11-02T19:06:01.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb41d3eff07e745a-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.544 DEBUG:boto:path=/ibtljn-64/obj37 2023-11-02T19:06:01.544 DEBUG:boto:auth_path=/ibtljn-64/obj37 2023-11-02T19:06:01.544 DEBUG:boto:path=/ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.545 DEBUG:boto:auth_path=/ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.545 DEBUG:boto:Method: GET 2023-11-02T19:06:01.546 DEBUG:boto:Path: /ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.546 DEBUG:boto:Data: 2023-11-02T19:06:01.547 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.547 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.548 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.548 DEBUG:boto:Params: {} 2023-11-02T19:06:01.549 DEBUG:boto:Token: None 2023-11-02T19:06:01.549 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj37?versionId=null 2023-11-02T19:06:01.550 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:H/bOwdsKu+UmdzFJTtydrCbKPMI= 2023-11-02T19:06:01.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H/bOwdsKu+UmdzFJTtydrCbKPMI=', 'Content-Length': '0'} 2023-11-02T19:06:01.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092c4f44135ee8186-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.555 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.555 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.556 DEBUG:boto:path=/ibtljn-64/?prefix=obj37 2023-11-02T19:06:01.556 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj37 2023-11-02T19:06:01.557 DEBUG:boto:Method: GET 2023-11-02T19:06:01.557 DEBUG:boto:Path: /ibtljn-64/?prefix=obj37 2023-11-02T19:06:01.558 DEBUG:boto:Data: 2023-11-02T19:06:01.558 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.559 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.559 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.559 DEBUG:boto:Params: {} 2023-11-02T19:06:01.560 DEBUG:boto:Token: None 2023-11-02T19:06:01.560 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.561 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054d66e10c04fe188-006543f319-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.610 DEBUG:boto:b'ibtljn-64obj371000falseobj372023-11-02T19:03:44.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.611 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.612 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.612 DEBUG:boto:path=/ibtljn-64/?prefix=obj37 2023-11-02T19:06:01.613 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj37 2023-11-02T19:06:01.613 DEBUG:boto:Method: GET 2023-11-02T19:06:01.614 DEBUG:boto:Path: /ibtljn-64/?prefix=obj37 2023-11-02T19:06:01.614 DEBUG:boto:Data: 2023-11-02T19:06:01.615 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.618 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.619 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.619 DEBUG:boto:Params: {} 2023-11-02T19:06:01.620 DEBUG:boto:Token: None 2023-11-02T19:06:01.620 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e1b6b30bbc76fb2-006543f319-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.633 DEBUG:boto:b'ibtljn-64obj371000falseobj372023-11-02T19:03:44.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.633 DEBUG:boto:path=/ibtljn-64/obj37 2023-11-02T19:06:01.634 DEBUG:boto:auth_path=/ibtljn-64/obj37 2023-11-02T19:06:01.634 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.635 DEBUG:boto:Path: /ibtljn-64/obj37 2023-11-02T19:06:01.635 DEBUG:boto:Data: 2023-11-02T19:06:01.636 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.636 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.636 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.637 DEBUG:boto:Params: {} 2023-11-02T19:06:01.637 DEBUG:boto:Token: None 2023-11-02T19:06:01.638 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj37 2023-11-02T19:06:01.638 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3tNZL7UNj7MDxAwsWyoQHCp2jeM= 2023-11-02T19:06:01.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3tNZL7UNj7MDxAwsWyoQHCp2jeM=', 'Content-Length': '0'} 2023-11-02T19:06:01.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1243e814085eb3-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.642 DEBUG:boto:path=/ibtljn-64/obj37 2023-11-02T19:06:01.642 DEBUG:boto:auth_path=/ibtljn-64/obj37 2023-11-02T19:06:01.643 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.643 DEBUG:boto:Path: /ibtljn-64/obj37 2023-11-02T19:06:01.644 DEBUG:boto:Data: 2023-11-02T19:06:01.644 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.644 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.645 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.645 DEBUG:boto:Params: {} 2023-11-02T19:06:01.646 DEBUG:boto:Token: None 2023-11-02T19:06:01.646 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj37 2023-11-02T19:06:01.647 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3tNZL7UNj7MDxAwsWyoQHCp2jeM= 2023-11-02T19:06:01.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3tNZL7UNj7MDxAwsWyoQHCp2jeM=', 'Content-Length': '0'} 2023-11-02T19:06:01.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000137d362ab4e9ff12-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-11-02T19:06:01.651 DEBUG:boto:path=/ibtljn-64/obj37 2023-11-02T19:06:01.651 DEBUG:boto:auth_path=/ibtljn-64/obj37 2023-11-02T19:06:01.652 DEBUG:boto:Method: GET 2023-11-02T19:06:01.652 DEBUG:boto:Path: /ibtljn-64/obj37 2023-11-02T19:06:01.653 DEBUG:boto:Data: 2023-11-02T19:06:01.653 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.653 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.654 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.654 DEBUG:boto:Params: {} 2023-11-02T19:06:01.655 DEBUG:boto:Token: None 2023-11-02T19:06:01.655 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj37 2023-11-02T19:06:01.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EbWYPRdzd8VX2JU/SqD2k9wi0NU= 2023-11-02T19:06:01.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EbWYPRdzd8VX2JU/SqD2k9wi0NU=', 'Content-Length': '0'} 2023-11-02T19:06:01.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011337c48d17b9b65-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.660 DEBUG:boto:path=/ibtljn-64/obj37 2023-11-02T19:06:01.661 DEBUG:boto:auth_path=/ibtljn-64/obj37 2023-11-02T19:06:01.661 DEBUG:boto:Method: GET 2023-11-02T19:06:01.662 DEBUG:boto:Path: /ibtljn-64/obj37 2023-11-02T19:06:01.662 DEBUG:boto:Data: 2023-11-02T19:06:01.662 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.663 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.664 DEBUG:boto:Params: {} 2023-11-02T19:06:01.664 DEBUG:boto:Token: None 2023-11-02T19:06:01.665 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj37 2023-11-02T19:06:01.665 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EbWYPRdzd8VX2JU/SqD2k9wi0NU= 2023-11-02T19:06:01.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EbWYPRdzd8VX2JU/SqD2k9wi0NU=', 'Content-Length': '0'} 2023-11-02T19:06:01.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a456b73ae0d69c2a-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.714 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.715 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.715 DEBUG:boto:path=/ibtljn-64/?prefix=obj37 2023-11-02T19:06:01.716 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj37 2023-11-02T19:06:01.716 DEBUG:boto:Method: GET 2023-11-02T19:06:01.717 DEBUG:boto:Path: /ibtljn-64/?prefix=obj37 2023-11-02T19:06:01.717 DEBUG:boto:Data: 2023-11-02T19:06:01.718 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.718 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.718 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.719 DEBUG:boto:Params: {} 2023-11-02T19:06:01.719 DEBUG:boto:Token: None 2023-11-02T19:06:01.720 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.720 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc20bf3c638726d-006543f319-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.724 DEBUG:boto:b'ibtljn-64obj371000falseobj372023-11-02T19:03:44.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.725 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.726 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.726 DEBUG:boto:path=/ibtljn-64/?prefix=obj37 2023-11-02T19:06:01.727 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj37 2023-11-02T19:06:01.727 DEBUG:boto:Method: GET 2023-11-02T19:06:01.727 DEBUG:boto:Path: /ibtljn-64/?prefix=obj37 2023-11-02T19:06:01.728 DEBUG:boto:Data: 2023-11-02T19:06:01.728 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.729 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.729 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.730 DEBUG:boto:Params: {} 2023-11-02T19:06:01.730 DEBUG:boto:Token: None 2023-11-02T19:06:01.731 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.731 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a54c3d124b4c93f1-006543f319-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.739 DEBUG:boto:b'ibtljn-64obj371000falseobj372023-11-02T19:03:44.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T19:06:01.740 DEBUG:boto:path=/ibtljn-64/obj38 2023-11-02T19:06:01.740 DEBUG:boto:auth_path=/ibtljn-64/obj38 2023-11-02T19:06:01.741 DEBUG:boto:path=/ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.741 DEBUG:boto:auth_path=/ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.742 DEBUG:boto:Method: GET 2023-11-02T19:06:01.742 DEBUG:boto:Path: /ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.743 DEBUG:boto:Data: 2023-11-02T19:06:01.743 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.743 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.744 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.744 DEBUG:boto:Params: {} 2023-11-02T19:06:01.745 DEBUG:boto:Token: None 2023-11-02T19:06:01.745 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.746 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:He9Mi51MmvnfDjDJpP08CMNvsLE= 2023-11-02T19:06:01.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:He9Mi51MmvnfDjDJpP08CMNvsLE=', 'Content-Length': '0'} 2023-11-02T19:06:01.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001ffd5c17e561222-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.750 DEBUG:boto:path=/ibtljn-64/obj38 2023-11-02T19:06:01.751 DEBUG:boto:auth_path=/ibtljn-64/obj38 2023-11-02T19:06:01.751 DEBUG:boto:path=/ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.752 DEBUG:boto:auth_path=/ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.752 DEBUG:boto:Method: GET 2023-11-02T19:06:01.753 DEBUG:boto:Path: /ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.753 DEBUG:boto:Data: 2023-11-02T19:06:01.753 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.754 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.755 DEBUG:boto:Params: {} 2023-11-02T19:06:01.755 DEBUG:boto:Token: None 2023-11-02T19:06:01.756 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.756 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:He9Mi51MmvnfDjDJpP08CMNvsLE= 2023-11-02T19:06:01.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:He9Mi51MmvnfDjDJpP08CMNvsLE=', 'Content-Length': '0'} 2023-11-02T19:06:01.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085299bf248d63e15-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.760 DEBUG:boto:path=/ibtljn-64/obj38 2023-11-02T19:06:01.761 DEBUG:boto:auth_path=/ibtljn-64/obj38 2023-11-02T19:06:01.761 DEBUG:boto:path=/ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.762 DEBUG:boto:auth_path=/ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.762 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.762 DEBUG:boto:Path: /ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.763 DEBUG:boto:Data: 2023-11-02T19:06:01.763 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.764 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.764 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.765 DEBUG:boto:Params: {} 2023-11-02T19:06:01.765 DEBUG:boto:Token: None 2023-11-02T19:06:01.765 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.766 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tkoP+8gPvQryNcAubSe8cxuL5q4= 2023-11-02T19:06:01.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tkoP+8gPvQryNcAubSe8cxuL5q4=', 'Content-Length': '0'} 2023-11-02T19:06:01.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd4d559a6b2c313c-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.770 DEBUG:boto:path=/ibtljn-64/obj38 2023-11-02T19:06:01.770 DEBUG:boto:auth_path=/ibtljn-64/obj38 2023-11-02T19:06:01.771 DEBUG:boto:path=/ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.771 DEBUG:boto:auth_path=/ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.771 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.772 DEBUG:boto:Path: /ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.772 DEBUG:boto:Data: 2023-11-02T19:06:01.773 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.773 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.773 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.774 DEBUG:boto:Params: {} 2023-11-02T19:06:01.774 DEBUG:boto:Token: None 2023-11-02T19:06:01.775 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.775 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tkoP+8gPvQryNcAubSe8cxuL5q4= 2023-11-02T19:06:01.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tkoP+8gPvQryNcAubSe8cxuL5q4=', 'Content-Length': '0'} 2023-11-02T19:06:01.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9f0b2890af3163a-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T19:06:01.780 DEBUG:boto:path=/ibtljn-64/obj38 2023-11-02T19:06:01.780 DEBUG:boto:auth_path=/ibtljn-64/obj38 2023-11-02T19:06:01.781 DEBUG:boto:path=/ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.781 DEBUG:boto:auth_path=/ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.782 DEBUG:boto:Method: GET 2023-11-02T19:06:01.782 DEBUG:boto:Path: /ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.782 DEBUG:boto:Data: 2023-11-02T19:06:01.783 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.783 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.784 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.784 DEBUG:boto:Params: {} 2023-11-02T19:06:01.785 DEBUG:boto:Token: None 2023-11-02T19:06:01.785 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.786 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:He9Mi51MmvnfDjDJpP08CMNvsLE= 2023-11-02T19:06:01.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:He9Mi51MmvnfDjDJpP08CMNvsLE=', 'Content-Length': '0'} 2023-11-02T19:06:01.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034a187f7df45eef8-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.790 DEBUG:boto:path=/ibtljn-64/obj38 2023-11-02T19:06:01.790 DEBUG:boto:auth_path=/ibtljn-64/obj38 2023-11-02T19:06:01.791 DEBUG:boto:path=/ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.791 DEBUG:boto:auth_path=/ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.791 DEBUG:boto:Method: GET 2023-11-02T19:06:01.792 DEBUG:boto:Path: /ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.792 DEBUG:boto:Data: 2023-11-02T19:06:01.793 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.793 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.794 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.794 DEBUG:boto:Params: {} 2023-11-02T19:06:01.794 DEBUG:boto:Token: None 2023-11-02T19:06:01.795 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj38?versionId=null 2023-11-02T19:06:01.795 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:He9Mi51MmvnfDjDJpP08CMNvsLE= 2023-11-02T19:06:01.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:He9Mi51MmvnfDjDJpP08CMNvsLE=', 'Content-Length': '0'} 2023-11-02T19:06:01.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ba2341afd8d3ecc-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.842 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.843 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.844 DEBUG:boto:path=/ibtljn-64/?prefix=obj38 2023-11-02T19:06:01.844 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj38 2023-11-02T19:06:01.844 DEBUG:boto:Method: GET 2023-11-02T19:06:01.845 DEBUG:boto:Path: /ibtljn-64/?prefix=obj38 2023-11-02T19:06:01.845 DEBUG:boto:Data: 2023-11-02T19:06:01.846 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.846 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.847 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.847 DEBUG:boto:Params: {} 2023-11-02T19:06:01.847 DEBUG:boto:Token: None 2023-11-02T19:06:01.848 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.848 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aa88566f0faa1c4-006543f319-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.854 DEBUG:boto:b'ibtljn-64obj381000falseobj382023-11-02T19:03:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.855 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.856 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.856 DEBUG:boto:path=/ibtljn-64/?prefix=obj38 2023-11-02T19:06:01.856 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj38 2023-11-02T19:06:01.857 DEBUG:boto:Method: GET 2023-11-02T19:06:01.857 DEBUG:boto:Path: /ibtljn-64/?prefix=obj38 2023-11-02T19:06:01.858 DEBUG:boto:Data: 2023-11-02T19:06:01.858 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.859 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.859 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.860 DEBUG:boto:Params: {} 2023-11-02T19:06:01.860 DEBUG:boto:Token: None 2023-11-02T19:06:01.861 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.861 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2c14cb696255b64-006543f319-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.874 DEBUG:boto:b'ibtljn-64obj381000falseobj382023-11-02T19:03:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.874 DEBUG:boto:path=/ibtljn-64/obj38 2023-11-02T19:06:01.875 DEBUG:boto:auth_path=/ibtljn-64/obj38 2023-11-02T19:06:01.875 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.876 DEBUG:boto:Path: /ibtljn-64/obj38 2023-11-02T19:06:01.876 DEBUG:boto:Data: 2023-11-02T19:06:01.876 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.877 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.877 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.878 DEBUG:boto:Params: {} 2023-11-02T19:06:01.878 DEBUG:boto:Token: None 2023-11-02T19:06:01.879 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj38 2023-11-02T19:06:01.879 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s3cC7QlojgYuT9p16EkYf+H3zBs= 2023-11-02T19:06:01.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s3cC7QlojgYuT9p16EkYf+H3zBs=', 'Content-Length': '0'} 2023-11-02T19:06:01.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025cca95b27d32f5f-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.883 DEBUG:boto:path=/ibtljn-64/obj38 2023-11-02T19:06:01.883 DEBUG:boto:auth_path=/ibtljn-64/obj38 2023-11-02T19:06:01.883 DEBUG:boto:Method: HEAD 2023-11-02T19:06:01.884 DEBUG:boto:Path: /ibtljn-64/obj38 2023-11-02T19:06:01.884 DEBUG:boto:Data: 2023-11-02T19:06:01.885 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.885 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.886 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.886 DEBUG:boto:Params: {} 2023-11-02T19:06:01.887 DEBUG:boto:Token: None 2023-11-02T19:06:01.887 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj38 2023-11-02T19:06:01.887 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:s3cC7QlojgYuT9p16EkYf+H3zBs= 2023-11-02T19:06:01.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s3cC7QlojgYuT9p16EkYf+H3zBs=', 'Content-Length': '0'} 2023-11-02T19:06:01.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ce624a19f46e2b7-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-11-02T19:06:01.891 DEBUG:boto:path=/ibtljn-64/obj38 2023-11-02T19:06:01.892 DEBUG:boto:auth_path=/ibtljn-64/obj38 2023-11-02T19:06:01.892 DEBUG:boto:Method: GET 2023-11-02T19:06:01.893 DEBUG:boto:Path: /ibtljn-64/obj38 2023-11-02T19:06:01.893 DEBUG:boto:Data: 2023-11-02T19:06:01.894 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.894 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.895 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.895 DEBUG:boto:Params: {} 2023-11-02T19:06:01.896 DEBUG:boto:Token: None 2023-11-02T19:06:01.896 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj38 2023-11-02T19:06:01.897 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7hVTeafsp4ab7EqzWrJwvSh24pw= 2023-11-02T19:06:01.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7hVTeafsp4ab7EqzWrJwvSh24pw=', 'Content-Length': '0'} 2023-11-02T19:06:01.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fe0bf62d8e7b006-006543f319-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.942 DEBUG:boto:path=/ibtljn-64/obj38 2023-11-02T19:06:01.943 DEBUG:boto:auth_path=/ibtljn-64/obj38 2023-11-02T19:06:01.943 DEBUG:boto:Method: GET 2023-11-02T19:06:01.944 DEBUG:boto:Path: /ibtljn-64/obj38 2023-11-02T19:06:01.944 DEBUG:boto:Data: 2023-11-02T19:06:01.945 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.945 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.946 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.946 DEBUG:boto:Params: {} 2023-11-02T19:06:01.947 DEBUG:boto:Token: None 2023-11-02T19:06:01.947 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/obj38 2023-11-02T19:06:01.947 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7hVTeafsp4ab7EqzWrJwvSh24pw= 2023-11-02T19:06:01.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7hVTeafsp4ab7EqzWrJwvSh24pw=', 'Content-Length': '0'} 2023-11-02T19:06:01.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb599050fcbff7c6-006543f319-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.951 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.952 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.952 DEBUG:boto:path=/ibtljn-64/?prefix=obj38 2023-11-02T19:06:01.953 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj38 2023-11-02T19:06:01.953 DEBUG:boto:Method: GET 2023-11-02T19:06:01.954 DEBUG:boto:Path: /ibtljn-64/?prefix=obj38 2023-11-02T19:06:01.954 DEBUG:boto:Data: 2023-11-02T19:06:01.955 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.955 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:01.955 DEBUG:boto:Port: 8001 2023-11-02T19:06:01.956 DEBUG:boto:Params: {} 2023-11-02T19:06:01.956 DEBUG:boto:Token: None 2023-11-02T19:06:01.957 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.957 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f7d99f2c015e922-006543f319-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:01.963 DEBUG:boto:b'ibtljn-64obj381000falseobj382023-11-02T19:03:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:01.963 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:01.964 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:01.964 DEBUG:boto:path=/ibtljn-64/?prefix=obj38 2023-11-02T19:06:01.965 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj38 2023-11-02T19:06:01.965 DEBUG:boto:Method: GET 2023-11-02T19:06:01.966 DEBUG:boto:Path: /ibtljn-64/?prefix=obj38 2023-11-02T19:06:01.966 DEBUG:boto:Data: 2023-11-02T19:06:01.966 DEBUG:boto:Headers: {} 2023-11-02T19:06:01.967 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:01.967 DEBUG:boto:Port: 8000 2023-11-02T19:06:01.968 DEBUG:boto:Params: {} 2023-11-02T19:06:01.968 DEBUG:boto:Token: None 2023-11-02T19:06:01.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:01 GMT /ibtljn-64/ 2023-11-02T19:06:01.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc= 2023-11-02T19:06:01.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B1/kKdeNp+Lhm6o2jE9+5gTuUwc=', 'Content-Length': '0'} 2023-11-02T19:06:01.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099f6c90e4a3af329-006543f319-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.022 DEBUG:boto:b'ibtljn-64obj381000falseobj382023-11-02T19:03:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T19:06:02.026 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:06:02.027 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:06:02.027 DEBUG:boto:path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.027 DEBUG:boto:auth_path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.028 DEBUG:boto:Method: GET 2023-11-02T19:06:02.028 DEBUG:boto:Path: /ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.029 DEBUG:boto:Data: 2023-11-02T19:06:02.029 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.030 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.030 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.030 DEBUG:boto:Params: {} 2023-11-02T19:06:02.031 DEBUG:boto:Token: None 2023-11-02T19:06:02.031 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.032 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EgwokdMghpFLBxlwI/ilPsLoo7M= 2023-11-02T19:06:02.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EgwokdMghpFLBxlwI/ilPsLoo7M=', 'Content-Length': '0'} 2023-11-02T19:06:02.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000476363227e769f4b-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.036 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:06:02.037 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:06:02.037 DEBUG:boto:path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.038 DEBUG:boto:auth_path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.038 DEBUG:boto:Method: GET 2023-11-02T19:06:02.039 DEBUG:boto:Path: /ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.039 DEBUG:boto:Data: 2023-11-02T19:06:02.039 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.040 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.040 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.041 DEBUG:boto:Params: {} 2023-11-02T19:06:02.041 DEBUG:boto:Token: None 2023-11-02T19:06:02.042 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.042 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EgwokdMghpFLBxlwI/ilPsLoo7M= 2023-11-02T19:06:02.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EgwokdMghpFLBxlwI/ilPsLoo7M=', 'Content-Length': '0'} 2023-11-02T19:06:02.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c345ae271a59225-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.047 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:06:02.047 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:06:02.048 DEBUG:boto:path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.048 DEBUG:boto:auth_path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.048 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.049 DEBUG:boto:Path: /ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.049 DEBUG:boto:Data: 2023-11-02T19:06:02.050 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.050 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.051 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.051 DEBUG:boto:Params: {} 2023-11-02T19:06:02.051 DEBUG:boto:Token: None 2023-11-02T19:06:02.052 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.052 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QQSlHfYmpy5Ll6w4/orw7/+d0jE= 2023-11-02T19:06:02.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QQSlHfYmpy5Ll6w4/orw7/+d0jE=', 'Content-Length': '0'} 2023-11-02T19:06:02.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3e8f272356a78ec-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.056 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:06:02.057 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:06:02.057 DEBUG:boto:path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.057 DEBUG:boto:auth_path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.058 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.058 DEBUG:boto:Path: /ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.059 DEBUG:boto:Data: 2023-11-02T19:06:02.059 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.060 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.060 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.060 DEBUG:boto:Params: {} 2023-11-02T19:06:02.061 DEBUG:boto:Token: None 2023-11-02T19:06:02.061 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.062 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QQSlHfYmpy5Ll6w4/orw7/+d0jE= 2023-11-02T19:06:02.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QQSlHfYmpy5Ll6w4/orw7/+d0jE=', 'Content-Length': '0'} 2023-11-02T19:06:02.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b662d46f4d11f6-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T19:06:02.066 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:06:02.067 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:06:02.067 DEBUG:boto:path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.068 DEBUG:boto:auth_path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.068 DEBUG:boto:Method: GET 2023-11-02T19:06:02.069 DEBUG:boto:Path: /ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.069 DEBUG:boto:Data: 2023-11-02T19:06:02.069 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.070 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.070 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.071 DEBUG:boto:Params: {} 2023-11-02T19:06:02.071 DEBUG:boto:Token: None 2023-11-02T19:06:02.072 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.072 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EgwokdMghpFLBxlwI/ilPsLoo7M= 2023-11-02T19:06:02.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EgwokdMghpFLBxlwI/ilPsLoo7M=', 'Content-Length': '0'} 2023-11-02T19:06:02.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dadb2f816cb55d6-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.118 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:06:02.119 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:06:02.119 DEBUG:boto:path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.120 DEBUG:boto:auth_path=/ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.120 DEBUG:boto:Method: GET 2023-11-02T19:06:02.121 DEBUG:boto:Path: /ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.121 DEBUG:boto:Data: 2023-11-02T19:06:02.121 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.122 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.122 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.123 DEBUG:boto:Params: {} 2023-11-02T19:06:02.123 DEBUG:boto:Token: None 2023-11-02T19:06:02.123 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj44?versionId=null 2023-11-02T19:06:02.124 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:EgwokdMghpFLBxlwI/ilPsLoo7M= 2023-11-02T19:06:02.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EgwokdMghpFLBxlwI/ilPsLoo7M=', 'Content-Length': '0'} 2023-11-02T19:06:02.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6a7ca6a8918c103-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.128 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.129 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.129 DEBUG:boto:path=/ibtljn-64/?prefix=obj44 2023-11-02T19:06:02.130 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj44 2023-11-02T19:06:02.130 DEBUG:boto:Method: GET 2023-11-02T19:06:02.130 DEBUG:boto:Path: /ibtljn-64/?prefix=obj44 2023-11-02T19:06:02.131 DEBUG:boto:Data: 2023-11-02T19:06:02.131 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.132 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.132 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.133 DEBUG:boto:Params: {} 2023-11-02T19:06:02.133 DEBUG:boto:Token: None 2023-11-02T19:06:02.133 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.134 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc10e19f405948d3-006543f31a-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.140 DEBUG:boto:b'ibtljn-64obj441000falseobj442023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.141 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.141 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.142 DEBUG:boto:path=/ibtljn-64/?prefix=obj44 2023-11-02T19:06:02.142 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj44 2023-11-02T19:06:02.143 DEBUG:boto:Method: GET 2023-11-02T19:06:02.143 DEBUG:boto:Path: /ibtljn-64/?prefix=obj44 2023-11-02T19:06:02.143 DEBUG:boto:Data: 2023-11-02T19:06:02.144 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.144 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.145 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.145 DEBUG:boto:Params: {} 2023-11-02T19:06:02.146 DEBUG:boto:Token: None 2023-11-02T19:06:02.146 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.147 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4399b6d9332bc34-006543f31a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.157 DEBUG:boto:b'ibtljn-64obj441000falseobj442023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.158 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:06:02.158 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:06:02.159 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.159 DEBUG:boto:Path: /ibtljn-64/obj44 2023-11-02T19:06:02.160 DEBUG:boto:Data: 2023-11-02T19:06:02.160 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.160 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.161 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.161 DEBUG:boto:Params: {} 2023-11-02T19:06:02.162 DEBUG:boto:Token: None 2023-11-02T19:06:02.162 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj44 2023-11-02T19:06:02.162 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0iizLfKYnU3rdz2ro4wYB2H/imk= 2023-11-02T19:06:02.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0iizLfKYnU3rdz2ro4wYB2H/imk=', 'Content-Length': '0'} 2023-11-02T19:06:02.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006565fcbcffd175e2-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.166 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:06:02.166 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:06:02.166 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.167 DEBUG:boto:Path: /ibtljn-64/obj44 2023-11-02T19:06:02.167 DEBUG:boto:Data: 2023-11-02T19:06:02.168 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.168 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.168 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.169 DEBUG:boto:Params: {} 2023-11-02T19:06:02.169 DEBUG:boto:Token: None 2023-11-02T19:06:02.170 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj44 2023-11-02T19:06:02.170 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0iizLfKYnU3rdz2ro4wYB2H/imk= 2023-11-02T19:06:02.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0iizLfKYnU3rdz2ro4wYB2H/imk=', 'Content-Length': '0'} 2023-11-02T19:06:02.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c292dc10e5136656-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-11-02T19:06:02.174 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:06:02.174 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:06:02.175 DEBUG:boto:Method: GET 2023-11-02T19:06:02.175 DEBUG:boto:Path: /ibtljn-64/obj44 2023-11-02T19:06:02.176 DEBUG:boto:Data: 2023-11-02T19:06:02.176 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.177 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.177 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.177 DEBUG:boto:Params: {} 2023-11-02T19:06:02.178 DEBUG:boto:Token: None 2023-11-02T19:06:02.178 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj44 2023-11-02T19:06:02.179 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y3uoNxX2E/UbZiQW3ZF9oKZDcTg= 2023-11-02T19:06:02.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y3uoNxX2E/UbZiQW3ZF9oKZDcTg=', 'Content-Length': '0'} 2023-11-02T19:06:02.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006278201aef288467-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.183 DEBUG:boto:path=/ibtljn-64/obj44 2023-11-02T19:06:02.183 DEBUG:boto:auth_path=/ibtljn-64/obj44 2023-11-02T19:06:02.183 DEBUG:boto:Method: GET 2023-11-02T19:06:02.184 DEBUG:boto:Path: /ibtljn-64/obj44 2023-11-02T19:06:02.184 DEBUG:boto:Data: 2023-11-02T19:06:02.185 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.185 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.186 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.186 DEBUG:boto:Params: {} 2023-11-02T19:06:02.186 DEBUG:boto:Token: None 2023-11-02T19:06:02.187 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj44 2023-11-02T19:06:02.187 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Y3uoNxX2E/UbZiQW3ZF9oKZDcTg= 2023-11-02T19:06:02.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y3uoNxX2E/UbZiQW3ZF9oKZDcTg=', 'Content-Length': '0'} 2023-11-02T19:06:02.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed6528dee32bb31a-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.234 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.235 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.235 DEBUG:boto:path=/ibtljn-64/?prefix=obj44 2023-11-02T19:06:02.236 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj44 2023-11-02T19:06:02.236 DEBUG:boto:Method: GET 2023-11-02T19:06:02.236 DEBUG:boto:Path: /ibtljn-64/?prefix=obj44 2023-11-02T19:06:02.237 DEBUG:boto:Data: 2023-11-02T19:06:02.237 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.238 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.238 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.239 DEBUG:boto:Params: {} 2023-11-02T19:06:02.239 DEBUG:boto:Token: None 2023-11-02T19:06:02.240 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.240 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f350977492d030e3-006543f31a-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.246 DEBUG:boto:b'ibtljn-64obj441000falseobj442023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.247 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.247 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.248 DEBUG:boto:path=/ibtljn-64/?prefix=obj44 2023-11-02T19:06:02.248 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj44 2023-11-02T19:06:02.249 DEBUG:boto:Method: GET 2023-11-02T19:06:02.249 DEBUG:boto:Path: /ibtljn-64/?prefix=obj44 2023-11-02T19:06:02.250 DEBUG:boto:Data: 2023-11-02T19:06:02.250 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.251 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.251 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.251 DEBUG:boto:Params: {} 2023-11-02T19:06:02.252 DEBUG:boto:Token: None 2023-11-02T19:06:02.252 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.253 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c07a9b6c61074c5-006543f31a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.264 DEBUG:boto:b'ibtljn-64obj441000falseobj442023-11-02T19:03:19.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T19:06:02.265 DEBUG:boto:path=/ibtljn-64/obj46 2023-11-02T19:06:02.265 DEBUG:boto:auth_path=/ibtljn-64/obj46 2023-11-02T19:06:02.266 DEBUG:boto:path=/ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.266 DEBUG:boto:auth_path=/ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.267 DEBUG:boto:Method: GET 2023-11-02T19:06:02.267 DEBUG:boto:Path: /ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.267 DEBUG:boto:Data: 2023-11-02T19:06:02.268 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.268 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.269 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.269 DEBUG:boto:Params: {} 2023-11-02T19:06:02.270 DEBUG:boto:Token: None 2023-11-02T19:06:02.270 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.271 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1/vVmuJ6q851u4BZtVeTspDz9hQ= 2023-11-02T19:06:02.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1/vVmuJ6q851u4BZtVeTspDz9hQ=', 'Content-Length': '0'} 2023-11-02T19:06:02.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbcad95515dd67a0-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.318 DEBUG:boto:path=/ibtljn-64/obj46 2023-11-02T19:06:02.319 DEBUG:boto:auth_path=/ibtljn-64/obj46 2023-11-02T19:06:02.319 DEBUG:boto:path=/ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.320 DEBUG:boto:auth_path=/ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.320 DEBUG:boto:Method: GET 2023-11-02T19:06:02.321 DEBUG:boto:Path: /ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.321 DEBUG:boto:Data: 2023-11-02T19:06:02.321 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.322 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.322 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.323 DEBUG:boto:Params: {} 2023-11-02T19:06:02.323 DEBUG:boto:Token: None 2023-11-02T19:06:02.324 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.324 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1/vVmuJ6q851u4BZtVeTspDz9hQ= 2023-11-02T19:06:02.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1/vVmuJ6q851u4BZtVeTspDz9hQ=', 'Content-Length': '0'} 2023-11-02T19:06:02.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000456dbbaab8c1cc98-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.329 DEBUG:boto:path=/ibtljn-64/obj46 2023-11-02T19:06:02.329 DEBUG:boto:auth_path=/ibtljn-64/obj46 2023-11-02T19:06:02.330 DEBUG:boto:path=/ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.330 DEBUG:boto:auth_path=/ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.331 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.331 DEBUG:boto:Path: /ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.331 DEBUG:boto:Data: 2023-11-02T19:06:02.332 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.332 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.333 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.333 DEBUG:boto:Params: {} 2023-11-02T19:06:02.333 DEBUG:boto:Token: None 2023-11-02T19:06:02.334 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.334 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:191TysUVOxp+O9cB6pDdgtcn2U0= 2023-11-02T19:06:02.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:191TysUVOxp+O9cB6pDdgtcn2U0=', 'Content-Length': '0'} 2023-11-02T19:06:02.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6624e2b666e57e0-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.338 DEBUG:boto:path=/ibtljn-64/obj46 2023-11-02T19:06:02.339 DEBUG:boto:auth_path=/ibtljn-64/obj46 2023-11-02T19:06:02.339 DEBUG:boto:path=/ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.340 DEBUG:boto:auth_path=/ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.340 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.340 DEBUG:boto:Path: /ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.341 DEBUG:boto:Data: 2023-11-02T19:06:02.341 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.342 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.342 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.343 DEBUG:boto:Params: {} 2023-11-02T19:06:02.343 DEBUG:boto:Token: None 2023-11-02T19:06:02.344 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.344 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:191TysUVOxp+O9cB6pDdgtcn2U0= 2023-11-02T19:06:02.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:191TysUVOxp+O9cB6pDdgtcn2U0=', 'Content-Length': '0'} 2023-11-02T19:06:02.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049a5f298b51d7bb8-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T19:06:02.349 DEBUG:boto:path=/ibtljn-64/obj46 2023-11-02T19:06:02.349 DEBUG:boto:auth_path=/ibtljn-64/obj46 2023-11-02T19:06:02.350 DEBUG:boto:path=/ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.350 DEBUG:boto:auth_path=/ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.351 DEBUG:boto:Method: GET 2023-11-02T19:06:02.351 DEBUG:boto:Path: /ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.351 DEBUG:boto:Data: 2023-11-02T19:06:02.352 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.352 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.353 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.353 DEBUG:boto:Params: {} 2023-11-02T19:06:02.354 DEBUG:boto:Token: None 2023-11-02T19:06:02.354 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.355 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1/vVmuJ6q851u4BZtVeTspDz9hQ= 2023-11-02T19:06:02.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1/vVmuJ6q851u4BZtVeTspDz9hQ=', 'Content-Length': '0'} 2023-11-02T19:06:02.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027bf47835ee360ee-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.359 DEBUG:boto:path=/ibtljn-64/obj46 2023-11-02T19:06:02.360 DEBUG:boto:auth_path=/ibtljn-64/obj46 2023-11-02T19:06:02.360 DEBUG:boto:path=/ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.360 DEBUG:boto:auth_path=/ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.361 DEBUG:boto:Method: GET 2023-11-02T19:06:02.361 DEBUG:boto:Path: /ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.362 DEBUG:boto:Data: 2023-11-02T19:06:02.362 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.363 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.363 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.363 DEBUG:boto:Params: {} 2023-11-02T19:06:02.364 DEBUG:boto:Token: None 2023-11-02T19:06:02.364 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj46?versionId=null 2023-11-02T19:06:02.365 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1/vVmuJ6q851u4BZtVeTspDz9hQ= 2023-11-02T19:06:02.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1/vVmuJ6q851u4BZtVeTspDz9hQ=', 'Content-Length': '0'} 2023-11-02T19:06:02.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab02502f9db638d4-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.410 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.411 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.411 DEBUG:boto:path=/ibtljn-64/?prefix=obj46 2023-11-02T19:06:02.412 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj46 2023-11-02T19:06:02.412 DEBUG:boto:Method: GET 2023-11-02T19:06:02.413 DEBUG:boto:Path: /ibtljn-64/?prefix=obj46 2023-11-02T19:06:02.414 DEBUG:boto:Data: 2023-11-02T19:06:02.415 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.415 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.415 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.416 DEBUG:boto:Params: {} 2023-11-02T19:06:02.417 DEBUG:boto:Token: None 2023-11-02T19:06:02.417 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.418 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc468e1d2756a44-006543f31a-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.424 DEBUG:boto:b'ibtljn-64obj461000falseobj462023-11-02T19:03:43.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.424 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.425 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.425 DEBUG:boto:path=/ibtljn-64/?prefix=obj46 2023-11-02T19:06:02.426 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj46 2023-11-02T19:06:02.426 DEBUG:boto:Method: GET 2023-11-02T19:06:02.427 DEBUG:boto:Path: /ibtljn-64/?prefix=obj46 2023-11-02T19:06:02.427 DEBUG:boto:Data: 2023-11-02T19:06:02.428 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.428 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.428 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.429 DEBUG:boto:Params: {} 2023-11-02T19:06:02.429 DEBUG:boto:Token: None 2023-11-02T19:06:02.430 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.430 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008e3622becf72f7c-006543f31a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.441 DEBUG:boto:b'ibtljn-64obj461000falseobj462023-11-02T19:03:43.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.442 DEBUG:boto:path=/ibtljn-64/obj46 2023-11-02T19:06:02.443 DEBUG:boto:auth_path=/ibtljn-64/obj46 2023-11-02T19:06:02.443 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.444 DEBUG:boto:Path: /ibtljn-64/obj46 2023-11-02T19:06:02.444 DEBUG:boto:Data: 2023-11-02T19:06:02.444 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.445 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.445 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.446 DEBUG:boto:Params: {} 2023-11-02T19:06:02.446 DEBUG:boto:Token: None 2023-11-02T19:06:02.447 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj46 2023-11-02T19:06:02.447 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:67lPP0pkRhfzcELCTGxFJ46Lbyw= 2023-11-02T19:06:02.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:67lPP0pkRhfzcELCTGxFJ46Lbyw=', 'Content-Length': '0'} 2023-11-02T19:06:02.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fbdc15ac6ccde1b-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.450 DEBUG:boto:path=/ibtljn-64/obj46 2023-11-02T19:06:02.451 DEBUG:boto:auth_path=/ibtljn-64/obj46 2023-11-02T19:06:02.451 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.452 DEBUG:boto:Path: /ibtljn-64/obj46 2023-11-02T19:06:02.452 DEBUG:boto:Data: 2023-11-02T19:06:02.452 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.453 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.453 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.454 DEBUG:boto:Params: {} 2023-11-02T19:06:02.454 DEBUG:boto:Token: None 2023-11-02T19:06:02.455 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj46 2023-11-02T19:06:02.455 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:67lPP0pkRhfzcELCTGxFJ46Lbyw= 2023-11-02T19:06:02.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:67lPP0pkRhfzcELCTGxFJ46Lbyw=', 'Content-Length': '0'} 2023-11-02T19:06:02.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001caff91bd757b52b-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-11-02T19:06:02.459 DEBUG:boto:path=/ibtljn-64/obj46 2023-11-02T19:06:02.459 DEBUG:boto:auth_path=/ibtljn-64/obj46 2023-11-02T19:06:02.460 DEBUG:boto:Method: GET 2023-11-02T19:06:02.460 DEBUG:boto:Path: /ibtljn-64/obj46 2023-11-02T19:06:02.461 DEBUG:boto:Data: 2023-11-02T19:06:02.461 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.462 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.462 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.463 DEBUG:boto:Params: {} 2023-11-02T19:06:02.463 DEBUG:boto:Token: None 2023-11-02T19:06:02.463 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj46 2023-11-02T19:06:02.464 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V3s51OcxBmq8cp2mriC2yNHgcjo= 2023-11-02T19:06:02.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V3s51OcxBmq8cp2mriC2yNHgcjo=', 'Content-Length': '0'} 2023-11-02T19:06:02.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe88eb51a434511-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.468 DEBUG:boto:path=/ibtljn-64/obj46 2023-11-02T19:06:02.468 DEBUG:boto:auth_path=/ibtljn-64/obj46 2023-11-02T19:06:02.468 DEBUG:boto:Method: GET 2023-11-02T19:06:02.469 DEBUG:boto:Path: /ibtljn-64/obj46 2023-11-02T19:06:02.469 DEBUG:boto:Data: 2023-11-02T19:06:02.470 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.470 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.471 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.471 DEBUG:boto:Params: {} 2023-11-02T19:06:02.471 DEBUG:boto:Token: None 2023-11-02T19:06:02.472 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj46 2023-11-02T19:06:02.472 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V3s51OcxBmq8cp2mriC2yNHgcjo= 2023-11-02T19:06:02.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V3s51OcxBmq8cp2mriC2yNHgcjo=', 'Content-Length': '0'} 2023-11-02T19:06:02.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ce75c6c5f2df4d0-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.476 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.477 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.477 DEBUG:boto:path=/ibtljn-64/?prefix=obj46 2023-11-02T19:06:02.477 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj46 2023-11-02T19:06:02.478 DEBUG:boto:Method: GET 2023-11-02T19:06:02.478 DEBUG:boto:Path: /ibtljn-64/?prefix=obj46 2023-11-02T19:06:02.479 DEBUG:boto:Data: 2023-11-02T19:06:02.479 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.480 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.480 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.481 DEBUG:boto:Params: {} 2023-11-02T19:06:02.481 DEBUG:boto:Token: None 2023-11-02T19:06:02.482 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.482 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044354847fb65095f-006543f31a-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.534 DEBUG:boto:b'ibtljn-64obj461000falseobj462023-11-02T19:03:43.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.535 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.536 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.536 DEBUG:boto:path=/ibtljn-64/?prefix=obj46 2023-11-02T19:06:02.537 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj46 2023-11-02T19:06:02.537 DEBUG:boto:Method: GET 2023-11-02T19:06:02.538 DEBUG:boto:Path: /ibtljn-64/?prefix=obj46 2023-11-02T19:06:02.538 DEBUG:boto:Data: 2023-11-02T19:06:02.539 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.539 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.540 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.540 DEBUG:boto:Params: {} 2023-11-02T19:06:02.540 DEBUG:boto:Token: None 2023-11-02T19:06:02.541 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.542 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0690d7360431217-006543f31a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.552 DEBUG:boto:b'ibtljn-64obj461000falseobj462023-11-02T19:03:43.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T19:06:02.553 DEBUG:boto:path=/ibtljn-64/obj51 2023-11-02T19:06:02.553 DEBUG:boto:auth_path=/ibtljn-64/obj51 2023-11-02T19:06:02.554 DEBUG:boto:path=/ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.554 DEBUG:boto:auth_path=/ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.555 DEBUG:boto:Method: GET 2023-11-02T19:06:02.555 DEBUG:boto:Path: /ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.556 DEBUG:boto:Data: 2023-11-02T19:06:02.556 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.556 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.557 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.557 DEBUG:boto:Params: {} 2023-11-02T19:06:02.558 DEBUG:boto:Token: None 2023-11-02T19:06:02.558 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.559 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:alyfduwg9vsMIY0p3zfE6iAaBzA= 2023-11-02T19:06:02.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:alyfduwg9vsMIY0p3zfE6iAaBzA=', 'Content-Length': '0'} 2023-11-02T19:06:02.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef8e1f6ef92eaaf1-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.563 DEBUG:boto:path=/ibtljn-64/obj51 2023-11-02T19:06:02.564 DEBUG:boto:auth_path=/ibtljn-64/obj51 2023-11-02T19:06:02.564 DEBUG:boto:path=/ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.565 DEBUG:boto:auth_path=/ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.565 DEBUG:boto:Method: GET 2023-11-02T19:06:02.566 DEBUG:boto:Path: /ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.566 DEBUG:boto:Data: 2023-11-02T19:06:02.567 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.567 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.568 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.568 DEBUG:boto:Params: {} 2023-11-02T19:06:02.569 DEBUG:boto:Token: None 2023-11-02T19:06:02.570 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.570 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:alyfduwg9vsMIY0p3zfE6iAaBzA= 2023-11-02T19:06:02.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:alyfduwg9vsMIY0p3zfE6iAaBzA=', 'Content-Length': '0'} 2023-11-02T19:06:02.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c811a619dd1c6d93-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.618 DEBUG:boto:path=/ibtljn-64/obj51 2023-11-02T19:06:02.619 DEBUG:boto:auth_path=/ibtljn-64/obj51 2023-11-02T19:06:02.619 DEBUG:boto:path=/ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.620 DEBUG:boto:auth_path=/ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.620 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.621 DEBUG:boto:Path: /ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.621 DEBUG:boto:Data: 2023-11-02T19:06:02.622 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.622 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.622 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.623 DEBUG:boto:Params: {} 2023-11-02T19:06:02.623 DEBUG:boto:Token: None 2023-11-02T19:06:02.624 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.624 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vOKIQT8/uLOesbMQGpLI6ub38FA= 2023-11-02T19:06:02.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vOKIQT8/uLOesbMQGpLI6ub38FA=', 'Content-Length': '0'} 2023-11-02T19:06:02.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c036fa93b23166a-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.628 DEBUG:boto:path=/ibtljn-64/obj51 2023-11-02T19:06:02.629 DEBUG:boto:auth_path=/ibtljn-64/obj51 2023-11-02T19:06:02.629 DEBUG:boto:path=/ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.630 DEBUG:boto:auth_path=/ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.630 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.630 DEBUG:boto:Path: /ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.631 DEBUG:boto:Data: 2023-11-02T19:06:02.631 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.632 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.632 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.633 DEBUG:boto:Params: {} 2023-11-02T19:06:02.633 DEBUG:boto:Token: None 2023-11-02T19:06:02.634 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.634 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vOKIQT8/uLOesbMQGpLI6ub38FA= 2023-11-02T19:06:02.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vOKIQT8/uLOesbMQGpLI6ub38FA=', 'Content-Length': '0'} 2023-11-02T19:06:02.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a7d0aad249c3b79-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T19:06:02.639 DEBUG:boto:path=/ibtljn-64/obj51 2023-11-02T19:06:02.639 DEBUG:boto:auth_path=/ibtljn-64/obj51 2023-11-02T19:06:02.640 DEBUG:boto:path=/ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.640 DEBUG:boto:auth_path=/ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.641 DEBUG:boto:Method: GET 2023-11-02T19:06:02.641 DEBUG:boto:Path: /ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.641 DEBUG:boto:Data: 2023-11-02T19:06:02.642 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.642 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.643 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.643 DEBUG:boto:Params: {} 2023-11-02T19:06:02.644 DEBUG:boto:Token: None 2023-11-02T19:06:02.644 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:alyfduwg9vsMIY0p3zfE6iAaBzA= 2023-11-02T19:06:02.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:alyfduwg9vsMIY0p3zfE6iAaBzA=', 'Content-Length': '0'} 2023-11-02T19:06:02.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c5eecb5e6923f6b-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.649 DEBUG:boto:path=/ibtljn-64/obj51 2023-11-02T19:06:02.650 DEBUG:boto:auth_path=/ibtljn-64/obj51 2023-11-02T19:06:02.650 DEBUG:boto:path=/ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.651 DEBUG:boto:auth_path=/ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.651 DEBUG:boto:Method: GET 2023-11-02T19:06:02.651 DEBUG:boto:Path: /ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.652 DEBUG:boto:Data: 2023-11-02T19:06:02.652 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.653 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.653 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.653 DEBUG:boto:Params: {} 2023-11-02T19:06:02.654 DEBUG:boto:Token: None 2023-11-02T19:06:02.655 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj51?versionId=null 2023-11-02T19:06:02.655 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:alyfduwg9vsMIY0p3zfE6iAaBzA= 2023-11-02T19:06:02.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:alyfduwg9vsMIY0p3zfE6iAaBzA=', 'Content-Length': '0'} 2023-11-02T19:06:02.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048f292e5b74b3842-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.659 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.660 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.660 DEBUG:boto:path=/ibtljn-64/?prefix=obj51 2023-11-02T19:06:02.660 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj51 2023-11-02T19:06:02.661 DEBUG:boto:Method: GET 2023-11-02T19:06:02.661 DEBUG:boto:Path: /ibtljn-64/?prefix=obj51 2023-11-02T19:06:02.662 DEBUG:boto:Data: 2023-11-02T19:06:02.662 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.663 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.663 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.664 DEBUG:boto:Params: {} 2023-11-02T19:06:02.664 DEBUG:boto:Token: None 2023-11-02T19:06:02.665 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.665 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007400b35b63fd6a2e-006543f31a-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.671 DEBUG:boto:b'ibtljn-64obj511000falseobj512023-11-02T19:03:42.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.672 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.672 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.673 DEBUG:boto:path=/ibtljn-64/?prefix=obj51 2023-11-02T19:06:02.673 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj51 2023-11-02T19:06:02.674 DEBUG:boto:Method: GET 2023-11-02T19:06:02.674 DEBUG:boto:Path: /ibtljn-64/?prefix=obj51 2023-11-02T19:06:02.675 DEBUG:boto:Data: 2023-11-02T19:06:02.675 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.676 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.676 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.676 DEBUG:boto:Params: {} 2023-11-02T19:06:02.677 DEBUG:boto:Token: None 2023-11-02T19:06:02.677 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.678 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005db1ab3bfffc9a0b-006543f31a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.688 DEBUG:boto:b'ibtljn-64obj511000falseobj512023-11-02T19:03:42.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.689 DEBUG:boto:path=/ibtljn-64/obj51 2023-11-02T19:06:02.690 DEBUG:boto:auth_path=/ibtljn-64/obj51 2023-11-02T19:06:02.690 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.691 DEBUG:boto:Path: /ibtljn-64/obj51 2023-11-02T19:06:02.691 DEBUG:boto:Data: 2023-11-02T19:06:02.691 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.692 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.692 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.693 DEBUG:boto:Params: {} 2023-11-02T19:06:02.693 DEBUG:boto:Token: None 2023-11-02T19:06:02.694 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj51 2023-11-02T19:06:02.694 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DPWaM7bzHFV2FkpkU3hVh6TCJiM= 2023-11-02T19:06:02.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DPWaM7bzHFV2FkpkU3hVh6TCJiM=', 'Content-Length': '0'} 2023-11-02T19:06:02.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e825bee46bb6873-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.697 DEBUG:boto:path=/ibtljn-64/obj51 2023-11-02T19:06:02.698 DEBUG:boto:auth_path=/ibtljn-64/obj51 2023-11-02T19:06:02.698 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.699 DEBUG:boto:Path: /ibtljn-64/obj51 2023-11-02T19:06:02.699 DEBUG:boto:Data: 2023-11-02T19:06:02.700 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.700 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.700 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.701 DEBUG:boto:Params: {} 2023-11-02T19:06:02.701 DEBUG:boto:Token: None 2023-11-02T19:06:02.702 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj51 2023-11-02T19:06:02.702 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DPWaM7bzHFV2FkpkU3hVh6TCJiM= 2023-11-02T19:06:02.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DPWaM7bzHFV2FkpkU3hVh6TCJiM=', 'Content-Length': '0'} 2023-11-02T19:06:02.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a98e0047ad336646-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-11-02T19:06:02.706 DEBUG:boto:path=/ibtljn-64/obj51 2023-11-02T19:06:02.706 DEBUG:boto:auth_path=/ibtljn-64/obj51 2023-11-02T19:06:02.707 DEBUG:boto:Method: GET 2023-11-02T19:06:02.707 DEBUG:boto:Path: /ibtljn-64/obj51 2023-11-02T19:06:02.707 DEBUG:boto:Data: 2023-11-02T19:06:02.708 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.708 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.709 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.709 DEBUG:boto:Params: {} 2023-11-02T19:06:02.710 DEBUG:boto:Token: None 2023-11-02T19:06:02.710 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj51 2023-11-02T19:06:02.711 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MTbqnKHTBy0nZDmY5ZwuyG12m6w= 2023-11-02T19:06:02.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MTbqnKHTBy0nZDmY5ZwuyG12m6w=', 'Content-Length': '0'} 2023-11-02T19:06:02.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013c76acac34b159e-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.754 DEBUG:boto:path=/ibtljn-64/obj51 2023-11-02T19:06:02.755 DEBUG:boto:auth_path=/ibtljn-64/obj51 2023-11-02T19:06:02.755 DEBUG:boto:Method: GET 2023-11-02T19:06:02.756 DEBUG:boto:Path: /ibtljn-64/obj51 2023-11-02T19:06:02.756 DEBUG:boto:Data: 2023-11-02T19:06:02.757 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.757 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.758 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.758 DEBUG:boto:Params: {} 2023-11-02T19:06:02.759 DEBUG:boto:Token: None 2023-11-02T19:06:02.759 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj51 2023-11-02T19:06:02.760 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MTbqnKHTBy0nZDmY5ZwuyG12m6w= 2023-11-02T19:06:02.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MTbqnKHTBy0nZDmY5ZwuyG12m6w=', 'Content-Length': '0'} 2023-11-02T19:06:02.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e53da1a4cdb9cd-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.764 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.765 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.765 DEBUG:boto:path=/ibtljn-64/?prefix=obj51 2023-11-02T19:06:02.766 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj51 2023-11-02T19:06:02.766 DEBUG:boto:Method: GET 2023-11-02T19:06:02.767 DEBUG:boto:Path: /ibtljn-64/?prefix=obj51 2023-11-02T19:06:02.767 DEBUG:boto:Data: 2023-11-02T19:06:02.767 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.768 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.768 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.769 DEBUG:boto:Params: {} 2023-11-02T19:06:02.769 DEBUG:boto:Token: None 2023-11-02T19:06:02.770 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.770 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb2b41b71d256741-006543f31a-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.776 DEBUG:boto:b'ibtljn-64obj511000falseobj512023-11-02T19:03:42.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.777 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.777 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.778 DEBUG:boto:path=/ibtljn-64/?prefix=obj51 2023-11-02T19:06:02.778 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj51 2023-11-02T19:06:02.779 DEBUG:boto:Method: GET 2023-11-02T19:06:02.779 DEBUG:boto:Path: /ibtljn-64/?prefix=obj51 2023-11-02T19:06:02.780 DEBUG:boto:Data: 2023-11-02T19:06:02.780 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.781 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.781 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.781 DEBUG:boto:Params: {} 2023-11-02T19:06:02.782 DEBUG:boto:Token: None 2023-11-02T19:06:02.782 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.783 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e5a9ad3236c6304-006543f31a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.794 DEBUG:boto:b'ibtljn-64obj511000falseobj512023-11-02T19:03:42.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T19:06:02.796 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:06:02.796 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:06:02.796 DEBUG:boto:path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.797 DEBUG:boto:auth_path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.797 DEBUG:boto:Method: GET 2023-11-02T19:06:02.798 DEBUG:boto:Path: /ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.798 DEBUG:boto:Data: 2023-11-02T19:06:02.799 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.799 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.800 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.800 DEBUG:boto:Params: {} 2023-11-02T19:06:02.801 DEBUG:boto:Token: None 2023-11-02T19:06:02.801 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.802 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mLZfB++8a83d3St+EaCv+z0HPRE= 2023-11-02T19:06:02.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mLZfB++8a83d3St+EaCv+z0HPRE=', 'Content-Length': '0'} 2023-11-02T19:06:02.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999501e7250e6c7f-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.805 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:06:02.806 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:06:02.806 DEBUG:boto:path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.807 DEBUG:boto:auth_path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.807 DEBUG:boto:Method: GET 2023-11-02T19:06:02.807 DEBUG:boto:Path: /ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.808 DEBUG:boto:Data: 2023-11-02T19:06:02.808 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.809 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.809 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.810 DEBUG:boto:Params: {} 2023-11-02T19:06:02.810 DEBUG:boto:Token: None 2023-11-02T19:06:02.811 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.811 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mLZfB++8a83d3St+EaCv+z0HPRE= 2023-11-02T19:06:02.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mLZfB++8a83d3St+EaCv+z0HPRE=', 'Content-Length': '0'} 2023-11-02T19:06:02.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bf9308339dbbf44-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.816 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:06:02.816 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:06:02.816 DEBUG:boto:path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.817 DEBUG:boto:auth_path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.817 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.818 DEBUG:boto:Path: /ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.818 DEBUG:boto:Data: 2023-11-02T19:06:02.819 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.819 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.819 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.820 DEBUG:boto:Params: {} 2023-11-02T19:06:02.820 DEBUG:boto:Token: None 2023-11-02T19:06:02.821 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.821 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GRQQVYSPsavXxV+IAVrO4f5qdsw= 2023-11-02T19:06:02.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GRQQVYSPsavXxV+IAVrO4f5qdsw=', 'Content-Length': '0'} 2023-11-02T19:06:02.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000501bf8f8787e7a19-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.825 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:06:02.826 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:06:02.826 DEBUG:boto:path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.827 DEBUG:boto:auth_path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.827 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.827 DEBUG:boto:Path: /ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.828 DEBUG:boto:Data: 2023-11-02T19:06:02.828 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.829 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.829 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.829 DEBUG:boto:Params: {} 2023-11-02T19:06:02.830 DEBUG:boto:Token: None 2023-11-02T19:06:02.830 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.831 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GRQQVYSPsavXxV+IAVrO4f5qdsw= 2023-11-02T19:06:02.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GRQQVYSPsavXxV+IAVrO4f5qdsw=', 'Content-Length': '0'} 2023-11-02T19:06:02.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045fa511a51dd0de3-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T19:06:02.835 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:06:02.835 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:06:02.836 DEBUG:boto:path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.836 DEBUG:boto:auth_path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.837 DEBUG:boto:Method: GET 2023-11-02T19:06:02.837 DEBUG:boto:Path: /ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.838 DEBUG:boto:Data: 2023-11-02T19:06:02.838 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.839 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.839 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.841 DEBUG:boto:Params: {} 2023-11-02T19:06:02.841 DEBUG:boto:Token: None 2023-11-02T19:06:02.842 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.842 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mLZfB++8a83d3St+EaCv+z0HPRE= 2023-11-02T19:06:02.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mLZfB++8a83d3St+EaCv+z0HPRE=', 'Content-Length': '0'} 2023-11-02T19:06:02.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c17dd00dcfd8635c-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.847 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:06:02.847 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:06:02.848 DEBUG:boto:path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.848 DEBUG:boto:auth_path=/ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.849 DEBUG:boto:Method: GET 2023-11-02T19:06:02.849 DEBUG:boto:Path: /ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.849 DEBUG:boto:Data: 2023-11-02T19:06:02.850 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.851 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.851 DEBUG:boto:Params: {} 2023-11-02T19:06:02.852 DEBUG:boto:Token: None 2023-11-02T19:06:02.852 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj55?versionId=null 2023-11-02T19:06:02.853 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:mLZfB++8a83d3St+EaCv+z0HPRE= 2023-11-02T19:06:02.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mLZfB++8a83d3St+EaCv+z0HPRE=', 'Content-Length': '0'} 2023-11-02T19:06:02.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b49ffabc13b568c-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.898 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.899 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.900 DEBUG:boto:path=/ibtljn-64/?prefix=obj55 2023-11-02T19:06:02.900 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj55 2023-11-02T19:06:02.900 DEBUG:boto:Method: GET 2023-11-02T19:06:02.901 DEBUG:boto:Path: /ibtljn-64/?prefix=obj55 2023-11-02T19:06:02.901 DEBUG:boto:Data: 2023-11-02T19:06:02.902 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.902 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.903 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.903 DEBUG:boto:Params: {} 2023-11-02T19:06:02.904 DEBUG:boto:Token: None 2023-11-02T19:06:02.904 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.905 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fd6f4d993707274-006543f31a-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.911 DEBUG:boto:b'ibtljn-64obj551000falseobj552023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.912 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:02.912 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:02.913 DEBUG:boto:path=/ibtljn-64/?prefix=obj55 2023-11-02T19:06:02.913 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj55 2023-11-02T19:06:02.914 DEBUG:boto:Method: GET 2023-11-02T19:06:02.914 DEBUG:boto:Path: /ibtljn-64/?prefix=obj55 2023-11-02T19:06:02.915 DEBUG:boto:Data: 2023-11-02T19:06:02.915 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.916 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.916 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.917 DEBUG:boto:Params: {} 2023-11-02T19:06:02.917 DEBUG:boto:Token: None 2023-11-02T19:06:02.918 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/ 2023-11-02T19:06:02.918 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0= 2023-11-02T19:06:02.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2qFHSe6mOup1crsf9/uS9uHvHd0=', 'Content-Length': '0'} 2023-11-02T19:06:02.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004be6e8c599ddae0a-006543f31a-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.929 DEBUG:boto:b'ibtljn-64obj551000falseobj552023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:02.929 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:06:02.930 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:06:02.930 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.931 DEBUG:boto:Path: /ibtljn-64/obj55 2023-11-02T19:06:02.931 DEBUG:boto:Data: 2023-11-02T19:06:02.932 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.932 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.933 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.933 DEBUG:boto:Params: {} 2023-11-02T19:06:02.934 DEBUG:boto:Token: None 2023-11-02T19:06:02.934 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj55 2023-11-02T19:06:02.934 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wd912hoKF0oIwk5jgUqtycCtFRI= 2023-11-02T19:06:02.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wd912hoKF0oIwk5jgUqtycCtFRI=', 'Content-Length': '0'} 2023-11-02T19:06:02.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002faddce5c04bff0a-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.938 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:06:02.939 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:06:02.939 DEBUG:boto:Method: HEAD 2023-11-02T19:06:02.939 DEBUG:boto:Path: /ibtljn-64/obj55 2023-11-02T19:06:02.940 DEBUG:boto:Data: 2023-11-02T19:06:02.940 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.941 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:02.941 DEBUG:boto:Port: 8000 2023-11-02T19:06:02.942 DEBUG:boto:Params: {} 2023-11-02T19:06:02.942 DEBUG:boto:Token: None 2023-11-02T19:06:02.943 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj55 2023-11-02T19:06:02.943 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wd912hoKF0oIwk5jgUqtycCtFRI= 2023-11-02T19:06:02.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wd912hoKF0oIwk5jgUqtycCtFRI=', 'Content-Length': '0'} 2023-11-02T19:06:02.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000433828402ad5ae0b-006543f31a-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-11-02T19:06:02.948 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:06:02.948 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:06:02.949 DEBUG:boto:Method: GET 2023-11-02T19:06:02.949 DEBUG:boto:Path: /ibtljn-64/obj55 2023-11-02T19:06:02.950 DEBUG:boto:Data: 2023-11-02T19:06:02.950 DEBUG:boto:Headers: {} 2023-11-02T19:06:02.951 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:02.951 DEBUG:boto:Port: 8001 2023-11-02T19:06:02.952 DEBUG:boto:Params: {} 2023-11-02T19:06:02.952 DEBUG:boto:Token: None 2023-11-02T19:06:02.953 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:02 GMT /ibtljn-64/obj55 2023-11-02T19:06:02.953 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oJRFCHe6vV4G0Q2/ZwEWiQitugo= 2023-11-02T19:06:02.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oJRFCHe6vV4G0Q2/ZwEWiQitugo=', 'Content-Length': '0'} 2023-11-02T19:06:02.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007498aea5d5832dd3-006543f31a-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:02.998 DEBUG:boto:path=/ibtljn-64/obj55 2023-11-02T19:06:02.999 DEBUG:boto:auth_path=/ibtljn-64/obj55 2023-11-02T19:06:03.000 DEBUG:boto:Method: GET 2023-11-02T19:06:03.000 DEBUG:boto:Path: /ibtljn-64/obj55 2023-11-02T19:06:03.001 DEBUG:boto:Data: 2023-11-02T19:06:03.001 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.002 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.002 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.003 DEBUG:boto:Params: {} 2023-11-02T19:06:03.003 DEBUG:boto:Token: None 2023-11-02T19:06:03.004 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj55 2023-11-02T19:06:03.004 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v7nUDvSL3/X7tIlKQ2LBseQj4XU= 2023-11-02T19:06:03.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v7nUDvSL3/X7tIlKQ2LBseQj4XU=', 'Content-Length': '0'} 2023-11-02T19:06:03.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000effc65c8870ecdc1-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.009 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.009 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.010 DEBUG:boto:path=/ibtljn-64/?prefix=obj55 2023-11-02T19:06:03.010 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj55 2023-11-02T19:06:03.011 DEBUG:boto:Method: GET 2023-11-02T19:06:03.011 DEBUG:boto:Path: /ibtljn-64/?prefix=obj55 2023-11-02T19:06:03.012 DEBUG:boto:Data: 2023-11-02T19:06:03.012 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.013 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.013 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.014 DEBUG:boto:Params: {} 2023-11-02T19:06:03.014 DEBUG:boto:Token: None 2023-11-02T19:06:03.015 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.015 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bca70f45051d87a4-006543f31b-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.021 DEBUG:boto:b'ibtljn-64obj551000falseobj552023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.022 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.023 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.023 DEBUG:boto:path=/ibtljn-64/?prefix=obj55 2023-11-02T19:06:03.024 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj55 2023-11-02T19:06:03.024 DEBUG:boto:Method: GET 2023-11-02T19:06:03.025 DEBUG:boto:Path: /ibtljn-64/?prefix=obj55 2023-11-02T19:06:03.025 DEBUG:boto:Data: 2023-11-02T19:06:03.026 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.026 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.027 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.027 DEBUG:boto:Params: {} 2023-11-02T19:06:03.028 DEBUG:boto:Token: None 2023-11-02T19:06:03.028 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.029 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002996e356bbf3a875-006543f31b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.082 DEBUG:boto:b'ibtljn-64obj551000falseobj552023-11-02T19:03:19.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T19:06:03.084 DEBUG:boto:path=/ibtljn-64/obj57 2023-11-02T19:06:03.084 DEBUG:boto:auth_path=/ibtljn-64/obj57 2023-11-02T19:06:03.085 DEBUG:boto:path=/ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.085 DEBUG:boto:auth_path=/ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.086 DEBUG:boto:Method: GET 2023-11-02T19:06:03.086 DEBUG:boto:Path: /ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.087 DEBUG:boto:Data: 2023-11-02T19:06:03.087 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.088 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.088 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.089 DEBUG:boto:Params: {} 2023-11-02T19:06:03.089 DEBUG:boto:Token: None 2023-11-02T19:06:03.090 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.091 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G3yYrio7SXaseOVISdCHbMBqd2k= 2023-11-02T19:06:03.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G3yYrio7SXaseOVISdCHbMBqd2k=', 'Content-Length': '0'} 2023-11-02T19:06:03.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000959ff92c9cc23161-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.095 DEBUG:boto:path=/ibtljn-64/obj57 2023-11-02T19:06:03.096 DEBUG:boto:auth_path=/ibtljn-64/obj57 2023-11-02T19:06:03.096 DEBUG:boto:path=/ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.097 DEBUG:boto:auth_path=/ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.097 DEBUG:boto:Method: GET 2023-11-02T19:06:03.098 DEBUG:boto:Path: /ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.098 DEBUG:boto:Data: 2023-11-02T19:06:03.099 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.099 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.099 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.100 DEBUG:boto:Params: {} 2023-11-02T19:06:03.100 DEBUG:boto:Token: None 2023-11-02T19:06:03.101 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.101 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G3yYrio7SXaseOVISdCHbMBqd2k= 2023-11-02T19:06:03.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G3yYrio7SXaseOVISdCHbMBqd2k=', 'Content-Length': '0'} 2023-11-02T19:06:03.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c52474bca46ebdb-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.106 DEBUG:boto:path=/ibtljn-64/obj57 2023-11-02T19:06:03.106 DEBUG:boto:auth_path=/ibtljn-64/obj57 2023-11-02T19:06:03.107 DEBUG:boto:path=/ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.107 DEBUG:boto:auth_path=/ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.107 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.108 DEBUG:boto:Path: /ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.108 DEBUG:boto:Data: 2023-11-02T19:06:03.109 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.109 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.109 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.110 DEBUG:boto:Params: {} 2023-11-02T19:06:03.110 DEBUG:boto:Token: None 2023-11-02T19:06:03.111 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.111 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JJHw5GB95tE1x6BMFecQDaKdbFw= 2023-11-02T19:06:03.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JJHw5GB95tE1x6BMFecQDaKdbFw=', 'Content-Length': '0'} 2023-11-02T19:06:03.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086d3879037863576-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.115 DEBUG:boto:path=/ibtljn-64/obj57 2023-11-02T19:06:03.116 DEBUG:boto:auth_path=/ibtljn-64/obj57 2023-11-02T19:06:03.116 DEBUG:boto:path=/ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.117 DEBUG:boto:auth_path=/ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.117 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.117 DEBUG:boto:Path: /ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.118 DEBUG:boto:Data: 2023-11-02T19:06:03.118 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.119 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.119 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.119 DEBUG:boto:Params: {} 2023-11-02T19:06:03.120 DEBUG:boto:Token: None 2023-11-02T19:06:03.121 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.121 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JJHw5GB95tE1x6BMFecQDaKdbFw= 2023-11-02T19:06:03.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JJHw5GB95tE1x6BMFecQDaKdbFw=', 'Content-Length': '0'} 2023-11-02T19:06:03.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e48dd84326ae88f0-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T19:06:03.125 DEBUG:boto:path=/ibtljn-64/obj57 2023-11-02T19:06:03.126 DEBUG:boto:auth_path=/ibtljn-64/obj57 2023-11-02T19:06:03.126 DEBUG:boto:path=/ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.127 DEBUG:boto:auth_path=/ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.127 DEBUG:boto:Method: GET 2023-11-02T19:06:03.128 DEBUG:boto:Path: /ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.128 DEBUG:boto:Data: 2023-11-02T19:06:03.129 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.129 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.130 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.130 DEBUG:boto:Params: {} 2023-11-02T19:06:03.130 DEBUG:boto:Token: None 2023-11-02T19:06:03.131 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.132 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G3yYrio7SXaseOVISdCHbMBqd2k= 2023-11-02T19:06:03.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G3yYrio7SXaseOVISdCHbMBqd2k=', 'Content-Length': '0'} 2023-11-02T19:06:03.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037a25565a6a09dd9-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.178 DEBUG:boto:path=/ibtljn-64/obj57 2023-11-02T19:06:03.179 DEBUG:boto:auth_path=/ibtljn-64/obj57 2023-11-02T19:06:03.180 DEBUG:boto:path=/ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.180 DEBUG:boto:auth_path=/ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.181 DEBUG:boto:Method: GET 2023-11-02T19:06:03.181 DEBUG:boto:Path: /ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.182 DEBUG:boto:Data: 2023-11-02T19:06:03.182 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.183 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.183 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.184 DEBUG:boto:Params: {} 2023-11-02T19:06:03.185 DEBUG:boto:Token: None 2023-11-02T19:06:03.185 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj57?versionId=null 2023-11-02T19:06:03.186 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:G3yYrio7SXaseOVISdCHbMBqd2k= 2023-11-02T19:06:03.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:G3yYrio7SXaseOVISdCHbMBqd2k=', 'Content-Length': '0'} 2023-11-02T19:06:03.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8333ee3d80151fa-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.191 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.191 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.192 DEBUG:boto:path=/ibtljn-64/?prefix=obj57 2023-11-02T19:06:03.192 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj57 2023-11-02T19:06:03.193 DEBUG:boto:Method: GET 2023-11-02T19:06:03.193 DEBUG:boto:Path: /ibtljn-64/?prefix=obj57 2023-11-02T19:06:03.194 DEBUG:boto:Data: 2023-11-02T19:06:03.194 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.195 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.195 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.196 DEBUG:boto:Params: {} 2023-11-02T19:06:03.196 DEBUG:boto:Token: None 2023-11-02T19:06:03.197 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.197 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d28753ba9afa0bd8-006543f31b-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.203 DEBUG:boto:b'ibtljn-64obj571000falseobj572023-11-02T19:03:42.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.204 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.205 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.205 DEBUG:boto:path=/ibtljn-64/?prefix=obj57 2023-11-02T19:06:03.205 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj57 2023-11-02T19:06:03.206 DEBUG:boto:Method: GET 2023-11-02T19:06:03.206 DEBUG:boto:Path: /ibtljn-64/?prefix=obj57 2023-11-02T19:06:03.207 DEBUG:boto:Data: 2023-11-02T19:06:03.207 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.208 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.208 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.209 DEBUG:boto:Params: {} 2023-11-02T19:06:03.209 DEBUG:boto:Token: None 2023-11-02T19:06:03.210 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.211 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000985048e69c856de7-006543f31b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.222 DEBUG:boto:b'ibtljn-64obj571000falseobj572023-11-02T19:03:42.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.223 DEBUG:boto:path=/ibtljn-64/obj57 2023-11-02T19:06:03.223 DEBUG:boto:auth_path=/ibtljn-64/obj57 2023-11-02T19:06:03.224 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.224 DEBUG:boto:Path: /ibtljn-64/obj57 2023-11-02T19:06:03.225 DEBUG:boto:Data: 2023-11-02T19:06:03.225 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.226 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.226 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.227 DEBUG:boto:Params: {} 2023-11-02T19:06:03.227 DEBUG:boto:Token: None 2023-11-02T19:06:03.228 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj57 2023-11-02T19:06:03.228 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IuRbhiSeafR/wZBR9fKAdfVUBIM= 2023-11-02T19:06:03.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IuRbhiSeafR/wZBR9fKAdfVUBIM=', 'Content-Length': '0'} 2023-11-02T19:06:03.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0c4065aa813c1e-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.232 DEBUG:boto:path=/ibtljn-64/obj57 2023-11-02T19:06:03.233 DEBUG:boto:auth_path=/ibtljn-64/obj57 2023-11-02T19:06:03.233 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.234 DEBUG:boto:Path: /ibtljn-64/obj57 2023-11-02T19:06:03.234 DEBUG:boto:Data: 2023-11-02T19:06:03.235 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.235 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.235 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.236 DEBUG:boto:Params: {} 2023-11-02T19:06:03.236 DEBUG:boto:Token: None 2023-11-02T19:06:03.237 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj57 2023-11-02T19:06:03.237 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IuRbhiSeafR/wZBR9fKAdfVUBIM= 2023-11-02T19:06:03.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IuRbhiSeafR/wZBR9fKAdfVUBIM=', 'Content-Length': '0'} 2023-11-02T19:06:03.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008fdf28d8c86d046-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-11-02T19:06:03.242 DEBUG:boto:path=/ibtljn-64/obj57 2023-11-02T19:06:03.242 DEBUG:boto:auth_path=/ibtljn-64/obj57 2023-11-02T19:06:03.242 DEBUG:boto:Method: GET 2023-11-02T19:06:03.243 DEBUG:boto:Path: /ibtljn-64/obj57 2023-11-02T19:06:03.243 DEBUG:boto:Data: 2023-11-02T19:06:03.244 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.244 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.245 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.245 DEBUG:boto:Params: {} 2023-11-02T19:06:03.246 DEBUG:boto:Token: None 2023-11-02T19:06:03.246 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj57 2023-11-02T19:06:03.247 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KEE+E6ZsQEdxvI4JNKw03IiLDIQ= 2023-11-02T19:06:03.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KEE+E6ZsQEdxvI4JNKw03IiLDIQ=', 'Content-Length': '0'} 2023-11-02T19:06:03.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eea6f11969a584cd-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.294 DEBUG:boto:path=/ibtljn-64/obj57 2023-11-02T19:06:03.295 DEBUG:boto:auth_path=/ibtljn-64/obj57 2023-11-02T19:06:03.295 DEBUG:boto:Method: GET 2023-11-02T19:06:03.296 DEBUG:boto:Path: /ibtljn-64/obj57 2023-11-02T19:06:03.296 DEBUG:boto:Data: 2023-11-02T19:06:03.297 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.297 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.298 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.298 DEBUG:boto:Params: {} 2023-11-02T19:06:03.299 DEBUG:boto:Token: None 2023-11-02T19:06:03.299 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj57 2023-11-02T19:06:03.300 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KEE+E6ZsQEdxvI4JNKw03IiLDIQ= 2023-11-02T19:06:03.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KEE+E6ZsQEdxvI4JNKw03IiLDIQ=', 'Content-Length': '0'} 2023-11-02T19:06:03.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004f5517bf79c4894-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.305 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.305 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.306 DEBUG:boto:path=/ibtljn-64/?prefix=obj57 2023-11-02T19:06:03.306 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj57 2023-11-02T19:06:03.306 DEBUG:boto:Method: GET 2023-11-02T19:06:03.307 DEBUG:boto:Path: /ibtljn-64/?prefix=obj57 2023-11-02T19:06:03.307 DEBUG:boto:Data: 2023-11-02T19:06:03.308 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.308 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.309 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.309 DEBUG:boto:Params: {} 2023-11-02T19:06:03.310 DEBUG:boto:Token: None 2023-11-02T19:06:03.310 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.311 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bf955c28cf59fa7-006543f31b-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.317 DEBUG:boto:b'ibtljn-64obj571000falseobj572023-11-02T19:03:42.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.317 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.318 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.318 DEBUG:boto:path=/ibtljn-64/?prefix=obj57 2023-11-02T19:06:03.319 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj57 2023-11-02T19:06:03.319 DEBUG:boto:Method: GET 2023-11-02T19:06:03.320 DEBUG:boto:Path: /ibtljn-64/?prefix=obj57 2023-11-02T19:06:03.320 DEBUG:boto:Data: 2023-11-02T19:06:03.321 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.321 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.322 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.322 DEBUG:boto:Params: {} 2023-11-02T19:06:03.323 DEBUG:boto:Token: None 2023-11-02T19:06:03.323 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.324 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003596d7152daae6fb-006543f31b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.374 DEBUG:boto:b'ibtljn-64obj571000falseobj572023-11-02T19:03:42.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T19:06:03.376 DEBUG:boto:path=/ibtljn-64/obj58 2023-11-02T19:06:03.376 DEBUG:boto:auth_path=/ibtljn-64/obj58 2023-11-02T19:06:03.377 DEBUG:boto:path=/ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.377 DEBUG:boto:auth_path=/ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.378 DEBUG:boto:Method: GET 2023-11-02T19:06:03.378 DEBUG:boto:Path: /ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.379 DEBUG:boto:Data: 2023-11-02T19:06:03.379 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.379 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.380 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.380 DEBUG:boto:Params: {} 2023-11-02T19:06:03.381 DEBUG:boto:Token: None 2023-11-02T19:06:03.381 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.382 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UOtoFEtgxE+TX81YLt38BQubqZk= 2023-11-02T19:06:03.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UOtoFEtgxE+TX81YLt38BQubqZk=', 'Content-Length': '0'} 2023-11-02T19:06:03.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a86e53b525457b-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.387 DEBUG:boto:path=/ibtljn-64/obj58 2023-11-02T19:06:03.387 DEBUG:boto:auth_path=/ibtljn-64/obj58 2023-11-02T19:06:03.388 DEBUG:boto:path=/ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.388 DEBUG:boto:auth_path=/ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.388 DEBUG:boto:Method: GET 2023-11-02T19:06:03.389 DEBUG:boto:Path: /ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.389 DEBUG:boto:Data: 2023-11-02T19:06:03.390 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.390 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.391 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.391 DEBUG:boto:Params: {} 2023-11-02T19:06:03.392 DEBUG:boto:Token: None 2023-11-02T19:06:03.392 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.393 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UOtoFEtgxE+TX81YLt38BQubqZk= 2023-11-02T19:06:03.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UOtoFEtgxE+TX81YLt38BQubqZk=', 'Content-Length': '0'} 2023-11-02T19:06:03.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000243147862dd8e727-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.397 DEBUG:boto:path=/ibtljn-64/obj58 2023-11-02T19:06:03.399 DEBUG:boto:auth_path=/ibtljn-64/obj58 2023-11-02T19:06:03.399 DEBUG:boto:path=/ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.399 DEBUG:boto:auth_path=/ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.400 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.400 DEBUG:boto:Path: /ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.401 DEBUG:boto:Data: 2023-11-02T19:06:03.401 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.401 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.402 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.402 DEBUG:boto:Params: {} 2023-11-02T19:06:03.403 DEBUG:boto:Token: None 2023-11-02T19:06:03.403 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.404 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rNxNHh4AhWyRItpIk6fkUQqAgCI= 2023-11-02T19:06:03.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rNxNHh4AhWyRItpIk6fkUQqAgCI=', 'Content-Length': '0'} 2023-11-02T19:06:03.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057e1a79f60027358-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.408 DEBUG:boto:path=/ibtljn-64/obj58 2023-11-02T19:06:03.409 DEBUG:boto:auth_path=/ibtljn-64/obj58 2023-11-02T19:06:03.409 DEBUG:boto:path=/ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.410 DEBUG:boto:auth_path=/ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.410 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.410 DEBUG:boto:Path: /ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.411 DEBUG:boto:Data: 2023-11-02T19:06:03.411 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.412 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.412 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.412 DEBUG:boto:Params: {} 2023-11-02T19:06:03.413 DEBUG:boto:Token: None 2023-11-02T19:06:03.413 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.414 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:rNxNHh4AhWyRItpIk6fkUQqAgCI= 2023-11-02T19:06:03.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rNxNHh4AhWyRItpIk6fkUQqAgCI=', 'Content-Length': '0'} 2023-11-02T19:06:03.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cbe681d9e538cef-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T19:06:03.419 DEBUG:boto:path=/ibtljn-64/obj58 2023-11-02T19:06:03.419 DEBUG:boto:auth_path=/ibtljn-64/obj58 2023-11-02T19:06:03.420 DEBUG:boto:path=/ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.420 DEBUG:boto:auth_path=/ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.420 DEBUG:boto:Method: GET 2023-11-02T19:06:03.421 DEBUG:boto:Path: /ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.421 DEBUG:boto:Data: 2023-11-02T19:06:03.422 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.422 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.423 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.423 DEBUG:boto:Params: {} 2023-11-02T19:06:03.424 DEBUG:boto:Token: None 2023-11-02T19:06:03.424 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.425 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UOtoFEtgxE+TX81YLt38BQubqZk= 2023-11-02T19:06:03.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UOtoFEtgxE+TX81YLt38BQubqZk=', 'Content-Length': '0'} 2023-11-02T19:06:03.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2f0203819e9c383-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.429 DEBUG:boto:path=/ibtljn-64/obj58 2023-11-02T19:06:03.429 DEBUG:boto:auth_path=/ibtljn-64/obj58 2023-11-02T19:06:03.430 DEBUG:boto:path=/ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.430 DEBUG:boto:auth_path=/ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.431 DEBUG:boto:Method: GET 2023-11-02T19:06:03.431 DEBUG:boto:Path: /ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.432 DEBUG:boto:Data: 2023-11-02T19:06:03.432 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.432 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.433 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.433 DEBUG:boto:Params: {} 2023-11-02T19:06:03.434 DEBUG:boto:Token: None 2023-11-02T19:06:03.434 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj58?versionId=null 2023-11-02T19:06:03.435 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UOtoFEtgxE+TX81YLt38BQubqZk= 2023-11-02T19:06:03.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UOtoFEtgxE+TX81YLt38BQubqZk=', 'Content-Length': '0'} 2023-11-02T19:06:03.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe4aefa5da4f0ef5-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.439 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.440 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.440 DEBUG:boto:path=/ibtljn-64/?prefix=obj58 2023-11-02T19:06:03.440 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj58 2023-11-02T19:06:03.441 DEBUG:boto:Method: GET 2023-11-02T19:06:03.441 DEBUG:boto:Path: /ibtljn-64/?prefix=obj58 2023-11-02T19:06:03.442 DEBUG:boto:Data: 2023-11-02T19:06:03.442 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.443 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.443 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.444 DEBUG:boto:Params: {} 2023-11-02T19:06:03.444 DEBUG:boto:Token: None 2023-11-02T19:06:03.445 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.445 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063a365e3ea608cb4-006543f31b-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.494 DEBUG:boto:b'ibtljn-64obj581000falseobj582023-11-02T19:03:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.495 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.496 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.496 DEBUG:boto:path=/ibtljn-64/?prefix=obj58 2023-11-02T19:06:03.497 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj58 2023-11-02T19:06:03.497 DEBUG:boto:Method: GET 2023-11-02T19:06:03.498 DEBUG:boto:Path: /ibtljn-64/?prefix=obj58 2023-11-02T19:06:03.498 DEBUG:boto:Data: 2023-11-02T19:06:03.499 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.499 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.499 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.500 DEBUG:boto:Params: {} 2023-11-02T19:06:03.500 DEBUG:boto:Token: None 2023-11-02T19:06:03.501 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.501 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f452e47f0ae6d675-006543f31b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.512 DEBUG:boto:b'ibtljn-64obj581000falseobj582023-11-02T19:03:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.513 DEBUG:boto:path=/ibtljn-64/obj58 2023-11-02T19:06:03.513 DEBUG:boto:auth_path=/ibtljn-64/obj58 2023-11-02T19:06:03.514 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.514 DEBUG:boto:Path: /ibtljn-64/obj58 2023-11-02T19:06:03.515 DEBUG:boto:Data: 2023-11-02T19:06:03.515 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.515 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.516 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.516 DEBUG:boto:Params: {} 2023-11-02T19:06:03.517 DEBUG:boto:Token: None 2023-11-02T19:06:03.517 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj58 2023-11-02T19:06:03.518 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GM+OZGndItK4ca1W2SG7NgIpNEE= 2023-11-02T19:06:03.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GM+OZGndItK4ca1W2SG7NgIpNEE=', 'Content-Length': '0'} 2023-11-02T19:06:03.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003752146d7e2eac31-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.521 DEBUG:boto:path=/ibtljn-64/obj58 2023-11-02T19:06:03.522 DEBUG:boto:auth_path=/ibtljn-64/obj58 2023-11-02T19:06:03.522 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.523 DEBUG:boto:Path: /ibtljn-64/obj58 2023-11-02T19:06:03.523 DEBUG:boto:Data: 2023-11-02T19:06:03.523 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.524 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.524 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.525 DEBUG:boto:Params: {} 2023-11-02T19:06:03.525 DEBUG:boto:Token: None 2023-11-02T19:06:03.526 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj58 2023-11-02T19:06:03.526 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GM+OZGndItK4ca1W2SG7NgIpNEE= 2023-11-02T19:06:03.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GM+OZGndItK4ca1W2SG7NgIpNEE=', 'Content-Length': '0'} 2023-11-02T19:06:03.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039d02a0bbbb10a11-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-11-02T19:06:03.530 DEBUG:boto:path=/ibtljn-64/obj58 2023-11-02T19:06:03.531 DEBUG:boto:auth_path=/ibtljn-64/obj58 2023-11-02T19:06:03.531 DEBUG:boto:Method: GET 2023-11-02T19:06:03.532 DEBUG:boto:Path: /ibtljn-64/obj58 2023-11-02T19:06:03.532 DEBUG:boto:Data: 2023-11-02T19:06:03.532 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.533 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.533 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.534 DEBUG:boto:Params: {} 2023-11-02T19:06:03.534 DEBUG:boto:Token: None 2023-11-02T19:06:03.535 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj58 2023-11-02T19:06:03.535 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+f4YYQyQxL5r+GRIc/owPCS6NRc= 2023-11-02T19:06:03.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+f4YYQyQxL5r+GRIc/owPCS6NRc=', 'Content-Length': '0'} 2023-11-02T19:06:03.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc5644a3d61335e7-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.539 DEBUG:boto:path=/ibtljn-64/obj58 2023-11-02T19:06:03.539 DEBUG:boto:auth_path=/ibtljn-64/obj58 2023-11-02T19:06:03.540 DEBUG:boto:Method: GET 2023-11-02T19:06:03.540 DEBUG:boto:Path: /ibtljn-64/obj58 2023-11-02T19:06:03.541 DEBUG:boto:Data: 2023-11-02T19:06:03.541 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.542 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.542 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.543 DEBUG:boto:Params: {} 2023-11-02T19:06:03.543 DEBUG:boto:Token: None 2023-11-02T19:06:03.543 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj58 2023-11-02T19:06:03.544 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:+f4YYQyQxL5r+GRIc/owPCS6NRc= 2023-11-02T19:06:03.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+f4YYQyQxL5r+GRIc/owPCS6NRc=', 'Content-Length': '0'} 2023-11-02T19:06:03.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000327b731643947f15-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.590 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.591 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.591 DEBUG:boto:path=/ibtljn-64/?prefix=obj58 2023-11-02T19:06:03.592 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj58 2023-11-02T19:06:03.592 DEBUG:boto:Method: GET 2023-11-02T19:06:03.593 DEBUG:boto:Path: /ibtljn-64/?prefix=obj58 2023-11-02T19:06:03.593 DEBUG:boto:Data: 2023-11-02T19:06:03.593 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.594 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.594 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.595 DEBUG:boto:Params: {} 2023-11-02T19:06:03.595 DEBUG:boto:Token: None 2023-11-02T19:06:03.596 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.596 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c839373d27dbce6-006543f31b-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.602 DEBUG:boto:b'ibtljn-64obj581000falseobj582023-11-02T19:03:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.603 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.603 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.604 DEBUG:boto:path=/ibtljn-64/?prefix=obj58 2023-11-02T19:06:03.604 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj58 2023-11-02T19:06:03.604 DEBUG:boto:Method: GET 2023-11-02T19:06:03.605 DEBUG:boto:Path: /ibtljn-64/?prefix=obj58 2023-11-02T19:06:03.605 DEBUG:boto:Data: 2023-11-02T19:06:03.606 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.606 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.607 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.607 DEBUG:boto:Params: {} 2023-11-02T19:06:03.608 DEBUG:boto:Token: None 2023-11-02T19:06:03.608 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.609 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007db3ee62e9015acc-006543f31b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.620 DEBUG:boto:b'ibtljn-64obj581000falseobj582023-11-02T19:03:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T19:06:03.621 DEBUG:boto:path=/ibtljn-64/obj62 2023-11-02T19:06:03.622 DEBUG:boto:auth_path=/ibtljn-64/obj62 2023-11-02T19:06:03.622 DEBUG:boto:path=/ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.623 DEBUG:boto:auth_path=/ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.623 DEBUG:boto:Method: GET 2023-11-02T19:06:03.623 DEBUG:boto:Path: /ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.624 DEBUG:boto:Data: 2023-11-02T19:06:03.624 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.625 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.625 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.625 DEBUG:boto:Params: {} 2023-11-02T19:06:03.626 DEBUG:boto:Token: None 2023-11-02T19:06:03.626 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.627 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0x88cl01CXdj6D9B9D1mY/TT4xA= 2023-11-02T19:06:03.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0x88cl01CXdj6D9B9D1mY/TT4xA=', 'Content-Length': '0'} 2023-11-02T19:06:03.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000083e73ed5760139c-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.631 DEBUG:boto:path=/ibtljn-64/obj62 2023-11-02T19:06:03.631 DEBUG:boto:auth_path=/ibtljn-64/obj62 2023-11-02T19:06:03.631 DEBUG:boto:path=/ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.632 DEBUG:boto:auth_path=/ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.632 DEBUG:boto:Method: GET 2023-11-02T19:06:03.633 DEBUG:boto:Path: /ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.633 DEBUG:boto:Data: 2023-11-02T19:06:03.634 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.634 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.635 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.635 DEBUG:boto:Params: {} 2023-11-02T19:06:03.635 DEBUG:boto:Token: None 2023-11-02T19:06:03.636 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.636 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0x88cl01CXdj6D9B9D1mY/TT4xA= 2023-11-02T19:06:03.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0x88cl01CXdj6D9B9D1mY/TT4xA=', 'Content-Length': '0'} 2023-11-02T19:06:03.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f92a570530c8c9ce-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.640 DEBUG:boto:path=/ibtljn-64/obj62 2023-11-02T19:06:03.641 DEBUG:boto:auth_path=/ibtljn-64/obj62 2023-11-02T19:06:03.641 DEBUG:boto:path=/ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.641 DEBUG:boto:auth_path=/ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.642 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.642 DEBUG:boto:Path: /ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.643 DEBUG:boto:Data: 2023-11-02T19:06:03.643 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.644 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.645 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.645 DEBUG:boto:Params: {} 2023-11-02T19:06:03.646 DEBUG:boto:Token: None 2023-11-02T19:06:03.646 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.647 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VQeDGjBKsrmBGkR47GiVT1WGsrs= 2023-11-02T19:06:03.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VQeDGjBKsrmBGkR47GiVT1WGsrs=', 'Content-Length': '0'} 2023-11-02T19:06:03.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6e285e568028155-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.651 DEBUG:boto:path=/ibtljn-64/obj62 2023-11-02T19:06:03.651 DEBUG:boto:auth_path=/ibtljn-64/obj62 2023-11-02T19:06:03.652 DEBUG:boto:path=/ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.652 DEBUG:boto:auth_path=/ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.652 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.653 DEBUG:boto:Path: /ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.653 DEBUG:boto:Data: 2023-11-02T19:06:03.654 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.655 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.655 DEBUG:boto:Params: {} 2023-11-02T19:06:03.655 DEBUG:boto:Token: None 2023-11-02T19:06:03.656 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.656 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VQeDGjBKsrmBGkR47GiVT1WGsrs= 2023-11-02T19:06:03.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VQeDGjBKsrmBGkR47GiVT1WGsrs=', 'Content-Length': '0'} 2023-11-02T19:06:03.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066ab35c6ca03d06e-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T19:06:03.661 DEBUG:boto:path=/ibtljn-64/obj62 2023-11-02T19:06:03.661 DEBUG:boto:auth_path=/ibtljn-64/obj62 2023-11-02T19:06:03.661 DEBUG:boto:path=/ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.662 DEBUG:boto:auth_path=/ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.662 DEBUG:boto:Method: GET 2023-11-02T19:06:03.663 DEBUG:boto:Path: /ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.663 DEBUG:boto:Data: 2023-11-02T19:06:03.664 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.664 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.665 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.665 DEBUG:boto:Params: {} 2023-11-02T19:06:03.666 DEBUG:boto:Token: None 2023-11-02T19:06:03.666 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.667 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0x88cl01CXdj6D9B9D1mY/TT4xA= 2023-11-02T19:06:03.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0x88cl01CXdj6D9B9D1mY/TT4xA=', 'Content-Length': '0'} 2023-11-02T19:06:03.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009100c8c3d0f8b4ef-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.671 DEBUG:boto:path=/ibtljn-64/obj62 2023-11-02T19:06:03.671 DEBUG:boto:auth_path=/ibtljn-64/obj62 2023-11-02T19:06:03.672 DEBUG:boto:path=/ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.672 DEBUG:boto:auth_path=/ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.673 DEBUG:boto:Method: GET 2023-11-02T19:06:03.673 DEBUG:boto:Path: /ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.673 DEBUG:boto:Data: 2023-11-02T19:06:03.674 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.674 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.675 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.675 DEBUG:boto:Params: {} 2023-11-02T19:06:03.676 DEBUG:boto:Token: None 2023-11-02T19:06:03.676 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj62?versionId=null 2023-11-02T19:06:03.677 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0x88cl01CXdj6D9B9D1mY/TT4xA= 2023-11-02T19:06:03.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0x88cl01CXdj6D9B9D1mY/TT4xA=', 'Content-Length': '0'} 2023-11-02T19:06:03.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000817c8dc82a90a42c-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.722 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.723 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.724 DEBUG:boto:path=/ibtljn-64/?prefix=obj62 2023-11-02T19:06:03.724 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj62 2023-11-02T19:06:03.724 DEBUG:boto:Method: GET 2023-11-02T19:06:03.725 DEBUG:boto:Path: /ibtljn-64/?prefix=obj62 2023-11-02T19:06:03.725 DEBUG:boto:Data: 2023-11-02T19:06:03.726 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.726 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.727 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.727 DEBUG:boto:Params: {} 2023-11-02T19:06:03.728 DEBUG:boto:Token: None 2023-11-02T19:06:03.728 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.729 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009af19a9cebf772db-006543f31b-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.734 DEBUG:boto:b'ibtljn-64obj621000falseobj622023-11-02T19:03:44.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.735 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.736 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.736 DEBUG:boto:path=/ibtljn-64/?prefix=obj62 2023-11-02T19:06:03.736 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj62 2023-11-02T19:06:03.737 DEBUG:boto:Method: GET 2023-11-02T19:06:03.737 DEBUG:boto:Path: /ibtljn-64/?prefix=obj62 2023-11-02T19:06:03.738 DEBUG:boto:Data: 2023-11-02T19:06:03.738 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.739 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.739 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.740 DEBUG:boto:Params: {} 2023-11-02T19:06:03.740 DEBUG:boto:Token: None 2023-11-02T19:06:03.740 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.741 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df3f1011cd7750af-006543f31b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.750 DEBUG:boto:b'ibtljn-64obj621000falseobj622023-11-02T19:03:44.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.751 DEBUG:boto:path=/ibtljn-64/obj62 2023-11-02T19:06:03.752 DEBUG:boto:auth_path=/ibtljn-64/obj62 2023-11-02T19:06:03.752 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.752 DEBUG:boto:Path: /ibtljn-64/obj62 2023-11-02T19:06:03.753 DEBUG:boto:Data: 2023-11-02T19:06:03.753 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.754 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.754 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.755 DEBUG:boto:Params: {} 2023-11-02T19:06:03.755 DEBUG:boto:Token: None 2023-11-02T19:06:03.755 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj62 2023-11-02T19:06:03.756 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DaV0+aBW2FiHBgHJfQ3gfZ97sg8= 2023-11-02T19:06:03.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DaV0+aBW2FiHBgHJfQ3gfZ97sg8=', 'Content-Length': '0'} 2023-11-02T19:06:03.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000675578ef1b98f452-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.760 DEBUG:boto:path=/ibtljn-64/obj62 2023-11-02T19:06:03.760 DEBUG:boto:auth_path=/ibtljn-64/obj62 2023-11-02T19:06:03.761 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.761 DEBUG:boto:Path: /ibtljn-64/obj62 2023-11-02T19:06:03.761 DEBUG:boto:Data: 2023-11-02T19:06:03.762 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.762 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.763 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.763 DEBUG:boto:Params: {} 2023-11-02T19:06:03.764 DEBUG:boto:Token: None 2023-11-02T19:06:03.764 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj62 2023-11-02T19:06:03.765 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:DaV0+aBW2FiHBgHJfQ3gfZ97sg8= 2023-11-02T19:06:03.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DaV0+aBW2FiHBgHJfQ3gfZ97sg8=', 'Content-Length': '0'} 2023-11-02T19:06:03.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bdf8aa61291b648-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-11-02T19:06:03.769 DEBUG:boto:path=/ibtljn-64/obj62 2023-11-02T19:06:03.770 DEBUG:boto:auth_path=/ibtljn-64/obj62 2023-11-02T19:06:03.770 DEBUG:boto:Method: GET 2023-11-02T19:06:03.771 DEBUG:boto:Path: /ibtljn-64/obj62 2023-11-02T19:06:03.771 DEBUG:boto:Data: 2023-11-02T19:06:03.772 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.772 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.772 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.773 DEBUG:boto:Params: {} 2023-11-02T19:06:03.773 DEBUG:boto:Token: None 2023-11-02T19:06:03.774 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj62 2023-11-02T19:06:03.775 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fk+D168Svr+xVAvDvnn6Robigmk= 2023-11-02T19:06:03.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fk+D168Svr+xVAvDvnn6Robigmk=', 'Content-Length': '0'} 2023-11-02T19:06:03.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e14f4b36b46f6e4-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.822 DEBUG:boto:path=/ibtljn-64/obj62 2023-11-02T19:06:03.823 DEBUG:boto:auth_path=/ibtljn-64/obj62 2023-11-02T19:06:03.824 DEBUG:boto:Method: GET 2023-11-02T19:06:03.824 DEBUG:boto:Path: /ibtljn-64/obj62 2023-11-02T19:06:03.824 DEBUG:boto:Data: 2023-11-02T19:06:03.825 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.825 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.826 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.826 DEBUG:boto:Params: {} 2023-11-02T19:06:03.827 DEBUG:boto:Token: None 2023-11-02T19:06:03.827 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj62 2023-11-02T19:06:03.828 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Fk+D168Svr+xVAvDvnn6Robigmk= 2023-11-02T19:06:03.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Fk+D168Svr+xVAvDvnn6Robigmk=', 'Content-Length': '0'} 2023-11-02T19:06:03.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034eee8490ded39b1-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.833 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.833 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.833 DEBUG:boto:path=/ibtljn-64/?prefix=obj62 2023-11-02T19:06:03.834 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj62 2023-11-02T19:06:03.834 DEBUG:boto:Method: GET 2023-11-02T19:06:03.835 DEBUG:boto:Path: /ibtljn-64/?prefix=obj62 2023-11-02T19:06:03.835 DEBUG:boto:Data: 2023-11-02T19:06:03.836 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.836 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.837 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.837 DEBUG:boto:Params: {} 2023-11-02T19:06:03.838 DEBUG:boto:Token: None 2023-11-02T19:06:03.838 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.839 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054ba7497679097cc-006543f31b-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.845 DEBUG:boto:b'ibtljn-64obj621000falseobj622023-11-02T19:03:44.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.846 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:03.847 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:03.847 DEBUG:boto:path=/ibtljn-64/?prefix=obj62 2023-11-02T19:06:03.847 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj62 2023-11-02T19:06:03.848 DEBUG:boto:Method: GET 2023-11-02T19:06:03.848 DEBUG:boto:Path: /ibtljn-64/?prefix=obj62 2023-11-02T19:06:03.849 DEBUG:boto:Data: 2023-11-02T19:06:03.849 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.850 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.851 DEBUG:boto:Params: {} 2023-11-02T19:06:03.851 DEBUG:boto:Token: None 2023-11-02T19:06:03.852 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/ 2023-11-02T19:06:03.852 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4= 2023-11-02T19:06:03.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3ELVYrF+8rKk+0/SrbXT19WPiN4=', 'Content-Length': '0'} 2023-11-02T19:06:03.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044d05e628488fd27-006543f31b-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.910 DEBUG:boto:b'ibtljn-64obj621000falseobj622023-11-02T19:03:44.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:03.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T19:06:03.912 DEBUG:boto:path=/ibtljn-64/obj68 2023-11-02T19:06:03.912 DEBUG:boto:auth_path=/ibtljn-64/obj68 2023-11-02T19:06:03.913 DEBUG:boto:path=/ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.913 DEBUG:boto:auth_path=/ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.913 DEBUG:boto:Method: GET 2023-11-02T19:06:03.914 DEBUG:boto:Path: /ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.914 DEBUG:boto:Data: 2023-11-02T19:06:03.915 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.915 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.916 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.916 DEBUG:boto:Params: {} 2023-11-02T19:06:03.917 DEBUG:boto:Token: None 2023-11-02T19:06:03.917 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.918 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RYsfVmVLRFV1A8tNHSnsOHwqTlk= 2023-11-02T19:06:03.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RYsfVmVLRFV1A8tNHSnsOHwqTlk=', 'Content-Length': '0'} 2023-11-02T19:06:03.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035810d5e0125b159-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.923 DEBUG:boto:path=/ibtljn-64/obj68 2023-11-02T19:06:03.923 DEBUG:boto:auth_path=/ibtljn-64/obj68 2023-11-02T19:06:03.924 DEBUG:boto:path=/ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.924 DEBUG:boto:auth_path=/ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.924 DEBUG:boto:Method: GET 2023-11-02T19:06:03.925 DEBUG:boto:Path: /ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.925 DEBUG:boto:Data: 2023-11-02T19:06:03.926 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.926 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.927 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.927 DEBUG:boto:Params: {} 2023-11-02T19:06:03.928 DEBUG:boto:Token: None 2023-11-02T19:06:03.928 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.929 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RYsfVmVLRFV1A8tNHSnsOHwqTlk= 2023-11-02T19:06:03.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RYsfVmVLRFV1A8tNHSnsOHwqTlk=', 'Content-Length': '0'} 2023-11-02T19:06:03.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e36e76e9489364-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.933 DEBUG:boto:path=/ibtljn-64/obj68 2023-11-02T19:06:03.933 DEBUG:boto:auth_path=/ibtljn-64/obj68 2023-11-02T19:06:03.934 DEBUG:boto:path=/ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.934 DEBUG:boto:auth_path=/ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.935 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.935 DEBUG:boto:Path: /ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.936 DEBUG:boto:Data: 2023-11-02T19:06:03.936 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.936 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.937 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.937 DEBUG:boto:Params: {} 2023-11-02T19:06:03.938 DEBUG:boto:Token: None 2023-11-02T19:06:03.938 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.939 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aLegiue6+V0OVeR7qxjNx54DkVs= 2023-11-02T19:06:03.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aLegiue6+V0OVeR7qxjNx54DkVs=', 'Content-Length': '0'} 2023-11-02T19:06:03.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029923fa15ad6b71f-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.943 DEBUG:boto:path=/ibtljn-64/obj68 2023-11-02T19:06:03.943 DEBUG:boto:auth_path=/ibtljn-64/obj68 2023-11-02T19:06:03.944 DEBUG:boto:path=/ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.944 DEBUG:boto:auth_path=/ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.944 DEBUG:boto:Method: HEAD 2023-11-02T19:06:03.945 DEBUG:boto:Path: /ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.945 DEBUG:boto:Data: 2023-11-02T19:06:03.946 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.946 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.947 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.947 DEBUG:boto:Params: {} 2023-11-02T19:06:03.948 DEBUG:boto:Token: None 2023-11-02T19:06:03.948 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.948 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aLegiue6+V0OVeR7qxjNx54DkVs= 2023-11-02T19:06:03.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aLegiue6+V0OVeR7qxjNx54DkVs=', 'Content-Length': '0'} 2023-11-02T19:06:03.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003536c3fa6820dea-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T19:06:03.953 DEBUG:boto:path=/ibtljn-64/obj68 2023-11-02T19:06:03.953 DEBUG:boto:auth_path=/ibtljn-64/obj68 2023-11-02T19:06:03.954 DEBUG:boto:path=/ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.954 DEBUG:boto:auth_path=/ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.955 DEBUG:boto:Method: GET 2023-11-02T19:06:03.955 DEBUG:boto:Path: /ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.956 DEBUG:boto:Data: 2023-11-02T19:06:03.956 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.957 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:03.957 DEBUG:boto:Port: 8001 2023-11-02T19:06:03.957 DEBUG:boto:Params: {} 2023-11-02T19:06:03.958 DEBUG:boto:Token: None 2023-11-02T19:06:03.958 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.959 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RYsfVmVLRFV1A8tNHSnsOHwqTlk= 2023-11-02T19:06:03.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RYsfVmVLRFV1A8tNHSnsOHwqTlk=', 'Content-Length': '0'} 2023-11-02T19:06:03.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dea543d96d815e3c-006543f31b-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:03.963 DEBUG:boto:path=/ibtljn-64/obj68 2023-11-02T19:06:03.964 DEBUG:boto:auth_path=/ibtljn-64/obj68 2023-11-02T19:06:03.964 DEBUG:boto:path=/ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.965 DEBUG:boto:auth_path=/ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.965 DEBUG:boto:Method: GET 2023-11-02T19:06:03.966 DEBUG:boto:Path: /ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.966 DEBUG:boto:Data: 2023-11-02T19:06:03.966 DEBUG:boto:Headers: {} 2023-11-02T19:06:03.967 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:03.967 DEBUG:boto:Port: 8000 2023-11-02T19:06:03.968 DEBUG:boto:Params: {} 2023-11-02T19:06:03.968 DEBUG:boto:Token: None 2023-11-02T19:06:03.969 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:03 GMT /ibtljn-64/obj68?versionId=null 2023-11-02T19:06:03.969 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RYsfVmVLRFV1A8tNHSnsOHwqTlk= 2023-11-02T19:06:03.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RYsfVmVLRFV1A8tNHSnsOHwqTlk=', 'Content-Length': '0'} 2023-11-02T19:06:03.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e33cfdc0657e54b8-006543f31b-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.014 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.015 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.015 DEBUG:boto:path=/ibtljn-64/?prefix=obj68 2023-11-02T19:06:04.016 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj68 2023-11-02T19:06:04.016 DEBUG:boto:Method: GET 2023-11-02T19:06:04.017 DEBUG:boto:Path: /ibtljn-64/?prefix=obj68 2023-11-02T19:06:04.017 DEBUG:boto:Data: 2023-11-02T19:06:04.018 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.018 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.019 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.019 DEBUG:boto:Params: {} 2023-11-02T19:06:04.019 DEBUG:boto:Token: None 2023-11-02T19:06:04.020 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.020 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e525f5c929c2a448-006543f31c-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.027 DEBUG:boto:b'ibtljn-64obj681000falseobj682023-11-02T19:03:42.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.028 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.028 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.028 DEBUG:boto:path=/ibtljn-64/?prefix=obj68 2023-11-02T19:06:04.029 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj68 2023-11-02T19:06:04.029 DEBUG:boto:Method: GET 2023-11-02T19:06:04.030 DEBUG:boto:Path: /ibtljn-64/?prefix=obj68 2023-11-02T19:06:04.030 DEBUG:boto:Data: 2023-11-02T19:06:04.031 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.031 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.032 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.032 DEBUG:boto:Params: {} 2023-11-02T19:06:04.033 DEBUG:boto:Token: None 2023-11-02T19:06:04.033 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.034 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5eb288f8bbe837b-006543f31c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.045 DEBUG:boto:b'ibtljn-64obj681000falseobj682023-11-02T19:03:42.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.045 DEBUG:boto:path=/ibtljn-64/obj68 2023-11-02T19:06:04.046 DEBUG:boto:auth_path=/ibtljn-64/obj68 2023-11-02T19:06:04.046 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.047 DEBUG:boto:Path: /ibtljn-64/obj68 2023-11-02T19:06:04.047 DEBUG:boto:Data: 2023-11-02T19:06:04.047 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.048 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.048 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.049 DEBUG:boto:Params: {} 2023-11-02T19:06:04.049 DEBUG:boto:Token: None 2023-11-02T19:06:04.050 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj68 2023-11-02T19:06:04.050 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/mUiJCrszdObUeATM7OvCW265IY= 2023-11-02T19:06:04.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/mUiJCrszdObUeATM7OvCW265IY=', 'Content-Length': '0'} 2023-11-02T19:06:04.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afd0b6d11fe48cf1-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.053 DEBUG:boto:path=/ibtljn-64/obj68 2023-11-02T19:06:04.054 DEBUG:boto:auth_path=/ibtljn-64/obj68 2023-11-02T19:06:04.054 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.055 DEBUG:boto:Path: /ibtljn-64/obj68 2023-11-02T19:06:04.055 DEBUG:boto:Data: 2023-11-02T19:06:04.055 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.056 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.056 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.057 DEBUG:boto:Params: {} 2023-11-02T19:06:04.057 DEBUG:boto:Token: None 2023-11-02T19:06:04.058 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj68 2023-11-02T19:06:04.058 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:/mUiJCrszdObUeATM7OvCW265IY= 2023-11-02T19:06:04.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/mUiJCrszdObUeATM7OvCW265IY=', 'Content-Length': '0'} 2023-11-02T19:06:04.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d07becb72df7783c-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-11-02T19:06:04.062 DEBUG:boto:path=/ibtljn-64/obj68 2023-11-02T19:06:04.062 DEBUG:boto:auth_path=/ibtljn-64/obj68 2023-11-02T19:06:04.063 DEBUG:boto:Method: GET 2023-11-02T19:06:04.063 DEBUG:boto:Path: /ibtljn-64/obj68 2023-11-02T19:06:04.064 DEBUG:boto:Data: 2023-11-02T19:06:04.064 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.065 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.065 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.065 DEBUG:boto:Params: {} 2023-11-02T19:06:04.066 DEBUG:boto:Token: None 2023-11-02T19:06:04.066 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj68 2023-11-02T19:06:04.067 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eeHpJg2iYcltBCHdEwSt/ZlDAzI= 2023-11-02T19:06:04.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eeHpJg2iYcltBCHdEwSt/ZlDAzI=', 'Content-Length': '0'} 2023-11-02T19:06:04.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef9b774b50ffe63c-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.110 DEBUG:boto:path=/ibtljn-64/obj68 2023-11-02T19:06:04.111 DEBUG:boto:auth_path=/ibtljn-64/obj68 2023-11-02T19:06:04.111 DEBUG:boto:Method: GET 2023-11-02T19:06:04.112 DEBUG:boto:Path: /ibtljn-64/obj68 2023-11-02T19:06:04.112 DEBUG:boto:Data: 2023-11-02T19:06:04.113 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.113 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.114 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.114 DEBUG:boto:Params: {} 2023-11-02T19:06:04.115 DEBUG:boto:Token: None 2023-11-02T19:06:04.115 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj68 2023-11-02T19:06:04.116 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eeHpJg2iYcltBCHdEwSt/ZlDAzI= 2023-11-02T19:06:04.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eeHpJg2iYcltBCHdEwSt/ZlDAzI=', 'Content-Length': '0'} 2023-11-02T19:06:04.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adb0d80833ad49a9-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.120 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.120 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.121 DEBUG:boto:path=/ibtljn-64/?prefix=obj68 2023-11-02T19:06:04.121 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj68 2023-11-02T19:06:04.122 DEBUG:boto:Method: GET 2023-11-02T19:06:04.122 DEBUG:boto:Path: /ibtljn-64/?prefix=obj68 2023-11-02T19:06:04.123 DEBUG:boto:Data: 2023-11-02T19:06:04.123 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.123 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.124 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.124 DEBUG:boto:Params: {} 2023-11-02T19:06:04.125 DEBUG:boto:Token: None 2023-11-02T19:06:04.125 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.126 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab0aae59e85130a6-006543f31c-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.132 DEBUG:boto:b'ibtljn-64obj681000falseobj682023-11-02T19:03:42.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.133 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.133 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.133 DEBUG:boto:path=/ibtljn-64/?prefix=obj68 2023-11-02T19:06:04.134 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj68 2023-11-02T19:06:04.134 DEBUG:boto:Method: GET 2023-11-02T19:06:04.135 DEBUG:boto:Path: /ibtljn-64/?prefix=obj68 2023-11-02T19:06:04.135 DEBUG:boto:Data: 2023-11-02T19:06:04.136 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.136 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.137 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.137 DEBUG:boto:Params: {} 2023-11-02T19:06:04.137 DEBUG:boto:Token: None 2023-11-02T19:06:04.138 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.139 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a48de67e315773-006543f31c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.190 DEBUG:boto:b'ibtljn-64obj681000falseobj682023-11-02T19:03:42.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T19:06:04.192 DEBUG:boto:path=/ibtljn-64/obj69 2023-11-02T19:06:04.192 DEBUG:boto:auth_path=/ibtljn-64/obj69 2023-11-02T19:06:04.193 DEBUG:boto:path=/ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.193 DEBUG:boto:auth_path=/ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.194 DEBUG:boto:Method: GET 2023-11-02T19:06:04.195 DEBUG:boto:Path: /ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.196 DEBUG:boto:Data: 2023-11-02T19:06:04.196 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.197 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.198 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.198 DEBUG:boto:Params: {} 2023-11-02T19:06:04.199 DEBUG:boto:Token: None 2023-11-02T19:06:04.199 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.200 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2jWAE3XgaV56+aZpfDEYCMRkRRY= 2023-11-02T19:06:04.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2jWAE3XgaV56+aZpfDEYCMRkRRY=', 'Content-Length': '0'} 2023-11-02T19:06:04.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e86824d2790ba75-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.204 DEBUG:boto:path=/ibtljn-64/obj69 2023-11-02T19:06:04.205 DEBUG:boto:auth_path=/ibtljn-64/obj69 2023-11-02T19:06:04.205 DEBUG:boto:path=/ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.206 DEBUG:boto:auth_path=/ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.206 DEBUG:boto:Method: GET 2023-11-02T19:06:04.207 DEBUG:boto:Path: /ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.207 DEBUG:boto:Data: 2023-11-02T19:06:04.207 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.208 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.208 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.209 DEBUG:boto:Params: {} 2023-11-02T19:06:04.209 DEBUG:boto:Token: None 2023-11-02T19:06:04.210 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.210 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2jWAE3XgaV56+aZpfDEYCMRkRRY= 2023-11-02T19:06:04.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2jWAE3XgaV56+aZpfDEYCMRkRRY=', 'Content-Length': '0'} 2023-11-02T19:06:04.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043d07ff2cbbf5fc0-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.215 DEBUG:boto:path=/ibtljn-64/obj69 2023-11-02T19:06:04.215 DEBUG:boto:auth_path=/ibtljn-64/obj69 2023-11-02T19:06:04.216 DEBUG:boto:path=/ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.216 DEBUG:boto:auth_path=/ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.216 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.217 DEBUG:boto:Path: /ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.217 DEBUG:boto:Data: 2023-11-02T19:06:04.218 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.218 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.218 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.219 DEBUG:boto:Params: {} 2023-11-02T19:06:04.219 DEBUG:boto:Token: None 2023-11-02T19:06:04.220 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.220 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vjIMGF9s/0+wvrEv0n2tYLxHjUM= 2023-11-02T19:06:04.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vjIMGF9s/0+wvrEv0n2tYLxHjUM=', 'Content-Length': '0'} 2023-11-02T19:06:04.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cc3819dfbd802e2-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.225 DEBUG:boto:path=/ibtljn-64/obj69 2023-11-02T19:06:04.225 DEBUG:boto:auth_path=/ibtljn-64/obj69 2023-11-02T19:06:04.226 DEBUG:boto:path=/ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.226 DEBUG:boto:auth_path=/ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.227 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.227 DEBUG:boto:Path: /ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.228 DEBUG:boto:Data: 2023-11-02T19:06:04.228 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.228 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.229 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.229 DEBUG:boto:Params: {} 2023-11-02T19:06:04.230 DEBUG:boto:Token: None 2023-11-02T19:06:04.230 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.231 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vjIMGF9s/0+wvrEv0n2tYLxHjUM= 2023-11-02T19:06:04.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vjIMGF9s/0+wvrEv0n2tYLxHjUM=', 'Content-Length': '0'} 2023-11-02T19:06:04.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002faed22fbe17c46f-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T19:06:04.236 DEBUG:boto:path=/ibtljn-64/obj69 2023-11-02T19:06:04.236 DEBUG:boto:auth_path=/ibtljn-64/obj69 2023-11-02T19:06:04.237 DEBUG:boto:path=/ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.237 DEBUG:boto:auth_path=/ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.237 DEBUG:boto:Method: GET 2023-11-02T19:06:04.238 DEBUG:boto:Path: /ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.238 DEBUG:boto:Data: 2023-11-02T19:06:04.239 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.239 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.240 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.240 DEBUG:boto:Params: {} 2023-11-02T19:06:04.241 DEBUG:boto:Token: None 2023-11-02T19:06:04.241 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.242 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2jWAE3XgaV56+aZpfDEYCMRkRRY= 2023-11-02T19:06:04.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2jWAE3XgaV56+aZpfDEYCMRkRRY=', 'Content-Length': '0'} 2023-11-02T19:06:04.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dfe6310183bddc0-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.286 DEBUG:boto:path=/ibtljn-64/obj69 2023-11-02T19:06:04.287 DEBUG:boto:auth_path=/ibtljn-64/obj69 2023-11-02T19:06:04.287 DEBUG:boto:path=/ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.288 DEBUG:boto:auth_path=/ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.289 DEBUG:boto:Method: GET 2023-11-02T19:06:04.290 DEBUG:boto:Path: /ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.291 DEBUG:boto:Data: 2023-11-02T19:06:04.291 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.292 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.292 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.292 DEBUG:boto:Params: {} 2023-11-02T19:06:04.293 DEBUG:boto:Token: None 2023-11-02T19:06:04.293 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj69?versionId=null 2023-11-02T19:06:04.294 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2jWAE3XgaV56+aZpfDEYCMRkRRY= 2023-11-02T19:06:04.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2jWAE3XgaV56+aZpfDEYCMRkRRY=', 'Content-Length': '0'} 2023-11-02T19:06:04.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1b14e74f884b03-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.299 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.299 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.299 DEBUG:boto:path=/ibtljn-64/?prefix=obj69 2023-11-02T19:06:04.300 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj69 2023-11-02T19:06:04.300 DEBUG:boto:Method: GET 2023-11-02T19:06:04.301 DEBUG:boto:Path: /ibtljn-64/?prefix=obj69 2023-11-02T19:06:04.301 DEBUG:boto:Data: 2023-11-02T19:06:04.302 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.302 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.303 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.303 DEBUG:boto:Params: {} 2023-11-02T19:06:04.304 DEBUG:boto:Token: None 2023-11-02T19:06:04.304 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.305 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec495ef1d8cf0c75-006543f31c-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.310 DEBUG:boto:b'ibtljn-64obj691000falseobj692023-11-02T19:03:43.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.311 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.312 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.312 DEBUG:boto:path=/ibtljn-64/?prefix=obj69 2023-11-02T19:06:04.313 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj69 2023-11-02T19:06:04.313 DEBUG:boto:Method: GET 2023-11-02T19:06:04.314 DEBUG:boto:Path: /ibtljn-64/?prefix=obj69 2023-11-02T19:06:04.314 DEBUG:boto:Data: 2023-11-02T19:06:04.315 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.315 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.315 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.316 DEBUG:boto:Params: {} 2023-11-02T19:06:04.316 DEBUG:boto:Token: None 2023-11-02T19:06:04.317 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.317 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068df833bb0c51583-006543f31c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.328 DEBUG:boto:b'ibtljn-64obj691000falseobj692023-11-02T19:03:43.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.329 DEBUG:boto:path=/ibtljn-64/obj69 2023-11-02T19:06:04.329 DEBUG:boto:auth_path=/ibtljn-64/obj69 2023-11-02T19:06:04.330 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.330 DEBUG:boto:Path: /ibtljn-64/obj69 2023-11-02T19:06:04.331 DEBUG:boto:Data: 2023-11-02T19:06:04.331 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.332 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.332 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.332 DEBUG:boto:Params: {} 2023-11-02T19:06:04.333 DEBUG:boto:Token: None 2023-11-02T19:06:04.333 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj69 2023-11-02T19:06:04.334 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:57A3aSVTSNFuuhFIwsobm3h2mWc= 2023-11-02T19:06:04.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:57A3aSVTSNFuuhFIwsobm3h2mWc=', 'Content-Length': '0'} 2023-11-02T19:06:04.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae9e908b9dcfeb20-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.337 DEBUG:boto:path=/ibtljn-64/obj69 2023-11-02T19:06:04.338 DEBUG:boto:auth_path=/ibtljn-64/obj69 2023-11-02T19:06:04.338 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.339 DEBUG:boto:Path: /ibtljn-64/obj69 2023-11-02T19:06:04.339 DEBUG:boto:Data: 2023-11-02T19:06:04.340 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.340 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.341 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.341 DEBUG:boto:Params: {} 2023-11-02T19:06:04.342 DEBUG:boto:Token: None 2023-11-02T19:06:04.342 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj69 2023-11-02T19:06:04.343 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:57A3aSVTSNFuuhFIwsobm3h2mWc= 2023-11-02T19:06:04.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:57A3aSVTSNFuuhFIwsobm3h2mWc=', 'Content-Length': '0'} 2023-11-02T19:06:04.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e5d45e761bd0265-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-11-02T19:06:04.348 DEBUG:boto:path=/ibtljn-64/obj69 2023-11-02T19:06:04.348 DEBUG:boto:auth_path=/ibtljn-64/obj69 2023-11-02T19:06:04.349 DEBUG:boto:Method: GET 2023-11-02T19:06:04.349 DEBUG:boto:Path: /ibtljn-64/obj69 2023-11-02T19:06:04.350 DEBUG:boto:Data: 2023-11-02T19:06:04.350 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.351 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.351 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.351 DEBUG:boto:Params: {} 2023-11-02T19:06:04.352 DEBUG:boto:Token: None 2023-11-02T19:06:04.353 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj69 2023-11-02T19:06:04.353 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3t1UtGj35i5uroIONP2d6Si67SY= 2023-11-02T19:06:04.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3t1UtGj35i5uroIONP2d6Si67SY=', 'Content-Length': '0'} 2023-11-02T19:06:04.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bd97844d56a7e59-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.357 DEBUG:boto:path=/ibtljn-64/obj69 2023-11-02T19:06:04.358 DEBUG:boto:auth_path=/ibtljn-64/obj69 2023-11-02T19:06:04.358 DEBUG:boto:Method: GET 2023-11-02T19:06:04.359 DEBUG:boto:Path: /ibtljn-64/obj69 2023-11-02T19:06:04.359 DEBUG:boto:Data: 2023-11-02T19:06:04.360 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.360 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.360 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.361 DEBUG:boto:Params: {} 2023-11-02T19:06:04.361 DEBUG:boto:Token: None 2023-11-02T19:06:04.362 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj69 2023-11-02T19:06:04.362 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3t1UtGj35i5uroIONP2d6Si67SY= 2023-11-02T19:06:04.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3t1UtGj35i5uroIONP2d6Si67SY=', 'Content-Length': '0'} 2023-11-02T19:06:04.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daed2bb225ee6a26-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.410 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.411 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.411 DEBUG:boto:path=/ibtljn-64/?prefix=obj69 2023-11-02T19:06:04.412 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj69 2023-11-02T19:06:04.412 DEBUG:boto:Method: GET 2023-11-02T19:06:04.413 DEBUG:boto:Path: /ibtljn-64/?prefix=obj69 2023-11-02T19:06:04.413 DEBUG:boto:Data: 2023-11-02T19:06:04.414 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.414 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.415 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.415 DEBUG:boto:Params: {} 2023-11-02T19:06:04.416 DEBUG:boto:Token: None 2023-11-02T19:06:04.416 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.417 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023ebb55cfff1ac06-006543f31c-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.423 DEBUG:boto:b'ibtljn-64obj691000falseobj692023-11-02T19:03:43.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.424 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.424 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.424 DEBUG:boto:path=/ibtljn-64/?prefix=obj69 2023-11-02T19:06:04.425 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj69 2023-11-02T19:06:04.425 DEBUG:boto:Method: GET 2023-11-02T19:06:04.426 DEBUG:boto:Path: /ibtljn-64/?prefix=obj69 2023-11-02T19:06:04.426 DEBUG:boto:Data: 2023-11-02T19:06:04.427 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.427 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.428 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.429 DEBUG:boto:Params: {} 2023-11-02T19:06:04.429 DEBUG:boto:Token: None 2023-11-02T19:06:04.430 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.430 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b59f3965fa6890f8-006543f31c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.440 DEBUG:boto:b'ibtljn-64obj691000falseobj692023-11-02T19:03:43.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-11-02T19:06:04.441 DEBUG:boto:path=/ibtljn-64/obj74 2023-11-02T19:06:04.442 DEBUG:boto:auth_path=/ibtljn-64/obj74 2023-11-02T19:06:04.442 DEBUG:boto:path=/ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.443 DEBUG:boto:auth_path=/ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.443 DEBUG:boto:Method: GET 2023-11-02T19:06:04.444 DEBUG:boto:Path: /ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.444 DEBUG:boto:Data: 2023-11-02T19:06:04.444 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.445 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.445 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.446 DEBUG:boto:Params: {} 2023-11-02T19:06:04.447 DEBUG:boto:Token: None 2023-11-02T19:06:04.447 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.448 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x5lZIgr6ZBUzZbOQe3Ibclu6RCE= 2023-11-02T19:06:04.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x5lZIgr6ZBUzZbOQe3Ibclu6RCE=', 'Content-Length': '0'} 2023-11-02T19:06:04.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000664150db7b436314-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.452 DEBUG:boto:path=/ibtljn-64/obj74 2023-11-02T19:06:04.453 DEBUG:boto:auth_path=/ibtljn-64/obj74 2023-11-02T19:06:04.454 DEBUG:boto:path=/ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.455 DEBUG:boto:auth_path=/ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.455 DEBUG:boto:Method: GET 2023-11-02T19:06:04.456 DEBUG:boto:Path: /ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.457 DEBUG:boto:Data: 2023-11-02T19:06:04.457 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.458 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.459 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.459 DEBUG:boto:Params: {} 2023-11-02T19:06:04.460 DEBUG:boto:Token: None 2023-11-02T19:06:04.461 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.461 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x5lZIgr6ZBUzZbOQe3Ibclu6RCE= 2023-11-02T19:06:04.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x5lZIgr6ZBUzZbOQe3Ibclu6RCE=', 'Content-Length': '0'} 2023-11-02T19:06:04.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4afac499ff43045-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.465 DEBUG:boto:path=/ibtljn-64/obj74 2023-11-02T19:06:04.466 DEBUG:boto:auth_path=/ibtljn-64/obj74 2023-11-02T19:06:04.466 DEBUG:boto:path=/ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.467 DEBUG:boto:auth_path=/ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.467 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.467 DEBUG:boto:Path: /ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.468 DEBUG:boto:Data: 2023-11-02T19:06:04.468 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.469 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.469 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.469 DEBUG:boto:Params: {} 2023-11-02T19:06:04.470 DEBUG:boto:Token: None 2023-11-02T19:06:04.470 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.471 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3qOW7flvopy3iyBGPe6P8nvs1JQ= 2023-11-02T19:06:04.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3qOW7flvopy3iyBGPe6P8nvs1JQ=', 'Content-Length': '0'} 2023-11-02T19:06:04.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000991cbb5cf631287a-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.475 DEBUG:boto:path=/ibtljn-64/obj74 2023-11-02T19:06:04.475 DEBUG:boto:auth_path=/ibtljn-64/obj74 2023-11-02T19:06:04.475 DEBUG:boto:path=/ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.476 DEBUG:boto:auth_path=/ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.476 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.477 DEBUG:boto:Path: /ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.477 DEBUG:boto:Data: 2023-11-02T19:06:04.477 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.478 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.478 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.479 DEBUG:boto:Params: {} 2023-11-02T19:06:04.479 DEBUG:boto:Token: None 2023-11-02T19:06:04.480 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.480 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3qOW7flvopy3iyBGPe6P8nvs1JQ= 2023-11-02T19:06:04.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3qOW7flvopy3iyBGPe6P8nvs1JQ=', 'Content-Length': '0'} 2023-11-02T19:06:04.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007361590faab86b92-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-11-02T19:06:04.487 DEBUG:boto:path=/ibtljn-64/obj74 2023-11-02T19:06:04.488 DEBUG:boto:auth_path=/ibtljn-64/obj74 2023-11-02T19:06:04.489 DEBUG:boto:path=/ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.490 DEBUG:boto:auth_path=/ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.490 DEBUG:boto:Method: GET 2023-11-02T19:06:04.491 DEBUG:boto:Path: /ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.492 DEBUG:boto:Data: 2023-11-02T19:06:04.493 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.494 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.494 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.495 DEBUG:boto:Params: {} 2023-11-02T19:06:04.497 DEBUG:boto:Token: None 2023-11-02T19:06:04.498 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.499 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x5lZIgr6ZBUzZbOQe3Ibclu6RCE= 2023-11-02T19:06:04.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x5lZIgr6ZBUzZbOQe3Ibclu6RCE=', 'Content-Length': '0'} 2023-11-02T19:06:04.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090e9d3938ade0352-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.546 DEBUG:boto:path=/ibtljn-64/obj74 2023-11-02T19:06:04.547 DEBUG:boto:auth_path=/ibtljn-64/obj74 2023-11-02T19:06:04.548 DEBUG:boto:path=/ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.548 DEBUG:boto:auth_path=/ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.549 DEBUG:boto:Method: GET 2023-11-02T19:06:04.549 DEBUG:boto:Path: /ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.550 DEBUG:boto:Data: 2023-11-02T19:06:04.550 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.550 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.551 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.551 DEBUG:boto:Params: {} 2023-11-02T19:06:04.552 DEBUG:boto:Token: None 2023-11-02T19:06:04.552 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj74?versionId=null 2023-11-02T19:06:04.553 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x5lZIgr6ZBUzZbOQe3Ibclu6RCE= 2023-11-02T19:06:04.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x5lZIgr6ZBUzZbOQe3Ibclu6RCE=', 'Content-Length': '0'} 2023-11-02T19:06:04.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028eb29f92a753a3e-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.557 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.557 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.558 DEBUG:boto:path=/ibtljn-64/?prefix=obj74 2023-11-02T19:06:04.558 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj74 2023-11-02T19:06:04.559 DEBUG:boto:Method: GET 2023-11-02T19:06:04.559 DEBUG:boto:Path: /ibtljn-64/?prefix=obj74 2023-11-02T19:06:04.559 DEBUG:boto:Data: 2023-11-02T19:06:04.560 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.560 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.561 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.561 DEBUG:boto:Params: {} 2023-11-02T19:06:04.562 DEBUG:boto:Token: None 2023-11-02T19:06:04.562 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.563 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a37be377a7389bec-006543f31c-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.569 DEBUG:boto:b'ibtljn-64obj741000falseobj742023-11-02T19:03:44.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.570 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.570 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.571 DEBUG:boto:path=/ibtljn-64/?prefix=obj74 2023-11-02T19:06:04.571 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj74 2023-11-02T19:06:04.572 DEBUG:boto:Method: GET 2023-11-02T19:06:04.572 DEBUG:boto:Path: /ibtljn-64/?prefix=obj74 2023-11-02T19:06:04.573 DEBUG:boto:Data: 2023-11-02T19:06:04.573 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.574 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.574 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.575 DEBUG:boto:Params: {} 2023-11-02T19:06:04.575 DEBUG:boto:Token: None 2023-11-02T19:06:04.575 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.576 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f02cd8085f8dc29-006543f31c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.626 DEBUG:boto:b'ibtljn-64obj741000falseobj742023-11-02T19:03:44.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.627 DEBUG:boto:path=/ibtljn-64/obj74 2023-11-02T19:06:04.628 DEBUG:boto:auth_path=/ibtljn-64/obj74 2023-11-02T19:06:04.628 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.629 DEBUG:boto:Path: /ibtljn-64/obj74 2023-11-02T19:06:04.629 DEBUG:boto:Data: 2023-11-02T19:06:04.630 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.630 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.631 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.632 DEBUG:boto:Params: {} 2023-11-02T19:06:04.632 DEBUG:boto:Token: None 2023-11-02T19:06:04.633 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj74 2023-11-02T19:06:04.634 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:r65H8zfxNJl4cv3kMIicdRt+WKI= 2023-11-02T19:06:04.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:r65H8zfxNJl4cv3kMIicdRt+WKI=', 'Content-Length': '0'} 2023-11-02T19:06:04.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df189961139cb003-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.639 DEBUG:boto:path=/ibtljn-64/obj74 2023-11-02T19:06:04.640 DEBUG:boto:auth_path=/ibtljn-64/obj74 2023-11-02T19:06:04.641 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.641 DEBUG:boto:Path: /ibtljn-64/obj74 2023-11-02T19:06:04.642 DEBUG:boto:Data: 2023-11-02T19:06:04.643 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.643 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.643 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.644 DEBUG:boto:Params: {} 2023-11-02T19:06:04.644 DEBUG:boto:Token: None 2023-11-02T19:06:04.645 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj74 2023-11-02T19:06:04.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:r65H8zfxNJl4cv3kMIicdRt+WKI= 2023-11-02T19:06:04.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:r65H8zfxNJl4cv3kMIicdRt+WKI=', 'Content-Length': '0'} 2023-11-02T19:06:04.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cb854b0809f1b37-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-11-02T19:06:04.650 DEBUG:boto:path=/ibtljn-64/obj74 2023-11-02T19:06:04.651 DEBUG:boto:auth_path=/ibtljn-64/obj74 2023-11-02T19:06:04.652 DEBUG:boto:Method: GET 2023-11-02T19:06:04.652 DEBUG:boto:Path: /ibtljn-64/obj74 2023-11-02T19:06:04.653 DEBUG:boto:Data: 2023-11-02T19:06:04.654 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.654 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.655 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.656 DEBUG:boto:Params: {} 2023-11-02T19:06:04.656 DEBUG:boto:Token: None 2023-11-02T19:06:04.657 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj74 2023-11-02T19:06:04.658 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D5/3085cCc+ygAb6FG9LPjSDqu0= 2023-11-02T19:06:04.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D5/3085cCc+ygAb6FG9LPjSDqu0=', 'Content-Length': '0'} 2023-11-02T19:06:04.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc21b33260741662-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.663 DEBUG:boto:path=/ibtljn-64/obj74 2023-11-02T19:06:04.664 DEBUG:boto:auth_path=/ibtljn-64/obj74 2023-11-02T19:06:04.664 DEBUG:boto:Method: GET 2023-11-02T19:06:04.664 DEBUG:boto:Path: /ibtljn-64/obj74 2023-11-02T19:06:04.665 DEBUG:boto:Data: 2023-11-02T19:06:04.665 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.666 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.666 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.667 DEBUG:boto:Params: {} 2023-11-02T19:06:04.667 DEBUG:boto:Token: None 2023-11-02T19:06:04.668 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj74 2023-11-02T19:06:04.668 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:D5/3085cCc+ygAb6FG9LPjSDqu0= 2023-11-02T19:06:04.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D5/3085cCc+ygAb6FG9LPjSDqu0=', 'Content-Length': '0'} 2023-11-02T19:06:04.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003087b73979fe1832-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.673 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.673 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.674 DEBUG:boto:path=/ibtljn-64/?prefix=obj74 2023-11-02T19:06:04.674 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj74 2023-11-02T19:06:04.674 DEBUG:boto:Method: GET 2023-11-02T19:06:04.675 DEBUG:boto:Path: /ibtljn-64/?prefix=obj74 2023-11-02T19:06:04.675 DEBUG:boto:Data: 2023-11-02T19:06:04.676 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.676 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.677 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.677 DEBUG:boto:Params: {} 2023-11-02T19:06:04.677 DEBUG:boto:Token: None 2023-11-02T19:06:04.678 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.679 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb9ce21d83124940-006543f31c-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.687 DEBUG:boto:b'ibtljn-64obj741000falseobj742023-11-02T19:03:44.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.687 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.689 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.690 DEBUG:boto:path=/ibtljn-64/?prefix=obj74 2023-11-02T19:06:04.690 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj74 2023-11-02T19:06:04.691 DEBUG:boto:Method: GET 2023-11-02T19:06:04.691 DEBUG:boto:Path: /ibtljn-64/?prefix=obj74 2023-11-02T19:06:04.691 DEBUG:boto:Data: 2023-11-02T19:06:04.692 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.692 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.693 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.693 DEBUG:boto:Params: {} 2023-11-02T19:06:04.694 DEBUG:boto:Token: None 2023-11-02T19:06:04.695 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.695 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006e60a4335bf8917-006543f31c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.703 DEBUG:boto:b'ibtljn-64obj741000falseobj742023-11-02T19:03:44.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T19:06:04.704 DEBUG:boto:path=/ibtljn-64/obj76 2023-11-02T19:06:04.704 DEBUG:boto:auth_path=/ibtljn-64/obj76 2023-11-02T19:06:04.705 DEBUG:boto:path=/ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.705 DEBUG:boto:auth_path=/ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.706 DEBUG:boto:Method: GET 2023-11-02T19:06:04.706 DEBUG:boto:Path: /ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.707 DEBUG:boto:Data: 2023-11-02T19:06:04.707 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.708 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.708 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.708 DEBUG:boto:Params: {} 2023-11-02T19:06:04.709 DEBUG:boto:Token: None 2023-11-02T19:06:04.709 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.710 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FlbVDH7n0tpqJf8nw96B8zZd2Zc= 2023-11-02T19:06:04.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FlbVDH7n0tpqJf8nw96B8zZd2Zc=', 'Content-Length': '0'} 2023-11-02T19:06:04.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000655354496e39a9ec-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.714 DEBUG:boto:path=/ibtljn-64/obj76 2023-11-02T19:06:04.714 DEBUG:boto:auth_path=/ibtljn-64/obj76 2023-11-02T19:06:04.715 DEBUG:boto:path=/ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.715 DEBUG:boto:auth_path=/ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.716 DEBUG:boto:Method: GET 2023-11-02T19:06:04.716 DEBUG:boto:Path: /ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.717 DEBUG:boto:Data: 2023-11-02T19:06:04.717 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.718 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.718 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.719 DEBUG:boto:Params: {} 2023-11-02T19:06:04.719 DEBUG:boto:Token: None 2023-11-02T19:06:04.720 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.720 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FlbVDH7n0tpqJf8nw96B8zZd2Zc= 2023-11-02T19:06:04.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FlbVDH7n0tpqJf8nw96B8zZd2Zc=', 'Content-Length': '0'} 2023-11-02T19:06:04.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000081b2da54036eaa8-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.770 DEBUG:boto:path=/ibtljn-64/obj76 2023-11-02T19:06:04.771 DEBUG:boto:auth_path=/ibtljn-64/obj76 2023-11-02T19:06:04.771 DEBUG:boto:path=/ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.772 DEBUG:boto:auth_path=/ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.772 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.772 DEBUG:boto:Path: /ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.773 DEBUG:boto:Data: 2023-11-02T19:06:04.773 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.774 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.774 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.775 DEBUG:boto:Params: {} 2023-11-02T19:06:04.775 DEBUG:boto:Token: None 2023-11-02T19:06:04.776 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.776 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vuDm3NtZvITlopBfUigjLNQHsCQ= 2023-11-02T19:06:04.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vuDm3NtZvITlopBfUigjLNQHsCQ=', 'Content-Length': '0'} 2023-11-02T19:06:04.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cfc0f655cbec9d0-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.780 DEBUG:boto:path=/ibtljn-64/obj76 2023-11-02T19:06:04.780 DEBUG:boto:auth_path=/ibtljn-64/obj76 2023-11-02T19:06:04.781 DEBUG:boto:path=/ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.781 DEBUG:boto:auth_path=/ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.782 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.782 DEBUG:boto:Path: /ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.782 DEBUG:boto:Data: 2023-11-02T19:06:04.783 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.783 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.784 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.784 DEBUG:boto:Params: {} 2023-11-02T19:06:04.784 DEBUG:boto:Token: None 2023-11-02T19:06:04.785 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.785 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vuDm3NtZvITlopBfUigjLNQHsCQ= 2023-11-02T19:06:04.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vuDm3NtZvITlopBfUigjLNQHsCQ=', 'Content-Length': '0'} 2023-11-02T19:06:04.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e35f9187a1343df-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T19:06:04.789 DEBUG:boto:path=/ibtljn-64/obj76 2023-11-02T19:06:04.790 DEBUG:boto:auth_path=/ibtljn-64/obj76 2023-11-02T19:06:04.790 DEBUG:boto:path=/ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.791 DEBUG:boto:auth_path=/ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.791 DEBUG:boto:Method: GET 2023-11-02T19:06:04.792 DEBUG:boto:Path: /ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.792 DEBUG:boto:Data: 2023-11-02T19:06:04.793 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.793 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.794 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.794 DEBUG:boto:Params: {} 2023-11-02T19:06:04.795 DEBUG:boto:Token: None 2023-11-02T19:06:04.795 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.796 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FlbVDH7n0tpqJf8nw96B8zZd2Zc= 2023-11-02T19:06:04.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FlbVDH7n0tpqJf8nw96B8zZd2Zc=', 'Content-Length': '0'} 2023-11-02T19:06:04.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000004645e58a14e6c5-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.846 DEBUG:boto:path=/ibtljn-64/obj76 2023-11-02T19:06:04.847 DEBUG:boto:auth_path=/ibtljn-64/obj76 2023-11-02T19:06:04.847 DEBUG:boto:path=/ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.848 DEBUG:boto:auth_path=/ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.848 DEBUG:boto:Method: GET 2023-11-02T19:06:04.849 DEBUG:boto:Path: /ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.849 DEBUG:boto:Data: 2023-11-02T19:06:04.850 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.850 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.851 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.851 DEBUG:boto:Params: {} 2023-11-02T19:06:04.851 DEBUG:boto:Token: None 2023-11-02T19:06:04.852 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj76?versionId=null 2023-11-02T19:06:04.852 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:FlbVDH7n0tpqJf8nw96B8zZd2Zc= 2023-11-02T19:06:04.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FlbVDH7n0tpqJf8nw96B8zZd2Zc=', 'Content-Length': '0'} 2023-11-02T19:06:04.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3bc21eed9c233b3-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.857 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.857 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.858 DEBUG:boto:path=/ibtljn-64/?prefix=obj76 2023-11-02T19:06:04.858 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj76 2023-11-02T19:06:04.859 DEBUG:boto:Method: GET 2023-11-02T19:06:04.859 DEBUG:boto:Path: /ibtljn-64/?prefix=obj76 2023-11-02T19:06:04.859 DEBUG:boto:Data: 2023-11-02T19:06:04.860 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.860 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.861 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.861 DEBUG:boto:Params: {} 2023-11-02T19:06:04.862 DEBUG:boto:Token: None 2023-11-02T19:06:04.862 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.863 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e520e8639271d46c-006543f31c-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.869 DEBUG:boto:b'ibtljn-64obj761000falseobj762023-11-02T19:03:42.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.870 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.871 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.871 DEBUG:boto:path=/ibtljn-64/?prefix=obj76 2023-11-02T19:06:04.871 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj76 2023-11-02T19:06:04.872 DEBUG:boto:Method: GET 2023-11-02T19:06:04.872 DEBUG:boto:Path: /ibtljn-64/?prefix=obj76 2023-11-02T19:06:04.873 DEBUG:boto:Data: 2023-11-02T19:06:04.873 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.874 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.874 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.875 DEBUG:boto:Params: {} 2023-11-02T19:06:04.875 DEBUG:boto:Token: None 2023-11-02T19:06:04.875 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.876 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a528201c460a52b0-006543f31c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.887 DEBUG:boto:b'ibtljn-64obj761000falseobj762023-11-02T19:03:42.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.887 DEBUG:boto:path=/ibtljn-64/obj76 2023-11-02T19:06:04.888 DEBUG:boto:auth_path=/ibtljn-64/obj76 2023-11-02T19:06:04.888 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.889 DEBUG:boto:Path: /ibtljn-64/obj76 2023-11-02T19:06:04.889 DEBUG:boto:Data: 2023-11-02T19:06:04.890 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.890 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.890 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.893 DEBUG:boto:Params: {} 2023-11-02T19:06:04.894 DEBUG:boto:Token: None 2023-11-02T19:06:04.895 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj76 2023-11-02T19:06:04.895 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v9RlEX3+KNdtyyz4qtPim3a53rs= 2023-11-02T19:06:04.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v9RlEX3+KNdtyyz4qtPim3a53rs=', 'Content-Length': '0'} 2023-11-02T19:06:04.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000669f5fe8b8276436-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.899 DEBUG:boto:path=/ibtljn-64/obj76 2023-11-02T19:06:04.899 DEBUG:boto:auth_path=/ibtljn-64/obj76 2023-11-02T19:06:04.900 DEBUG:boto:Method: HEAD 2023-11-02T19:06:04.900 DEBUG:boto:Path: /ibtljn-64/obj76 2023-11-02T19:06:04.900 DEBUG:boto:Data: 2023-11-02T19:06:04.901 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.901 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.902 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.902 DEBUG:boto:Params: {} 2023-11-02T19:06:04.903 DEBUG:boto:Token: None 2023-11-02T19:06:04.903 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj76 2023-11-02T19:06:04.904 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v9RlEX3+KNdtyyz4qtPim3a53rs= 2023-11-02T19:06:04.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v9RlEX3+KNdtyyz4qtPim3a53rs=', 'Content-Length': '0'} 2023-11-02T19:06:04.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc989e249d0e7aee-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-11-02T19:06:04.908 DEBUG:boto:path=/ibtljn-64/obj76 2023-11-02T19:06:04.908 DEBUG:boto:auth_path=/ibtljn-64/obj76 2023-11-02T19:06:04.909 DEBUG:boto:Method: GET 2023-11-02T19:06:04.909 DEBUG:boto:Path: /ibtljn-64/obj76 2023-11-02T19:06:04.909 DEBUG:boto:Data: 2023-11-02T19:06:04.910 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.910 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.911 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.911 DEBUG:boto:Params: {} 2023-11-02T19:06:04.912 DEBUG:boto:Token: None 2023-11-02T19:06:04.912 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj76 2023-11-02T19:06:04.913 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CflPEblsFZZ8JpB4M2hKF7wvjV4= 2023-11-02T19:06:04.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CflPEblsFZZ8JpB4M2hKF7wvjV4=', 'Content-Length': '0'} 2023-11-02T19:06:04.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010a62d92595b7693-006543f31c-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.917 DEBUG:boto:path=/ibtljn-64/obj76 2023-11-02T19:06:04.917 DEBUG:boto:auth_path=/ibtljn-64/obj76 2023-11-02T19:06:04.918 DEBUG:boto:Method: GET 2023-11-02T19:06:04.919 DEBUG:boto:Path: /ibtljn-64/obj76 2023-11-02T19:06:04.919 DEBUG:boto:Data: 2023-11-02T19:06:04.919 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.920 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.920 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.921 DEBUG:boto:Params: {} 2023-11-02T19:06:04.921 DEBUG:boto:Token: None 2023-11-02T19:06:04.922 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/obj76 2023-11-02T19:06:04.922 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CflPEblsFZZ8JpB4M2hKF7wvjV4= 2023-11-02T19:06:04.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CflPEblsFZZ8JpB4M2hKF7wvjV4=', 'Content-Length': '0'} 2023-11-02T19:06:04.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ae9cca70b50532e-006543f31c-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.966 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.967 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.968 DEBUG:boto:path=/ibtljn-64/?prefix=obj76 2023-11-02T19:06:04.968 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj76 2023-11-02T19:06:04.969 DEBUG:boto:Method: GET 2023-11-02T19:06:04.969 DEBUG:boto:Path: /ibtljn-64/?prefix=obj76 2023-11-02T19:06:04.970 DEBUG:boto:Data: 2023-11-02T19:06:04.970 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.971 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:04.972 DEBUG:boto:Port: 8001 2023-11-02T19:06:04.972 DEBUG:boto:Params: {} 2023-11-02T19:06:04.973 DEBUG:boto:Token: None 2023-11-02T19:06:04.973 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.974 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000725bc3c883b7ef00-006543f31c-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.980 DEBUG:boto:b'ibtljn-64obj761000falseobj762023-11-02T19:03:42.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.981 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:04.981 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:04.982 DEBUG:boto:path=/ibtljn-64/?prefix=obj76 2023-11-02T19:06:04.982 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj76 2023-11-02T19:06:04.983 DEBUG:boto:Method: GET 2023-11-02T19:06:04.983 DEBUG:boto:Path: /ibtljn-64/?prefix=obj76 2023-11-02T19:06:04.984 DEBUG:boto:Data: 2023-11-02T19:06:04.984 DEBUG:boto:Headers: {} 2023-11-02T19:06:04.984 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:04.985 DEBUG:boto:Port: 8000 2023-11-02T19:06:04.986 DEBUG:boto:Params: {} 2023-11-02T19:06:04.986 DEBUG:boto:Token: None 2023-11-02T19:06:04.987 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:04 GMT /ibtljn-64/ 2023-11-02T19:06:04.987 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8= 2023-11-02T19:06:04.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:04 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sJD+6YOD0sV4Xumtw4FiYaE7Mi8=', 'Content-Length': '0'} 2023-11-02T19:06:04.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a40f3bd5381d073-006543f31c-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:04 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:04.998 DEBUG:boto:b'ibtljn-64obj761000falseobj762023-11-02T19:03:42.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:04.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T19:06:05.000 DEBUG:boto:path=/ibtljn-64/obj85 2023-11-02T19:06:05.000 DEBUG:boto:auth_path=/ibtljn-64/obj85 2023-11-02T19:06:05.001 DEBUG:boto:path=/ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.001 DEBUG:boto:auth_path=/ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.002 DEBUG:boto:Method: GET 2023-11-02T19:06:05.002 DEBUG:boto:Path: /ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.003 DEBUG:boto:Data: 2023-11-02T19:06:05.004 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.004 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.005 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.005 DEBUG:boto:Params: {} 2023-11-02T19:06:05.006 DEBUG:boto:Token: None 2023-11-02T19:06:05.006 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.007 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6dFD7LRFRy9yLg1VczjrOP0o6Fk= 2023-11-02T19:06:05.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6dFD7LRFRy9yLg1VczjrOP0o6Fk=', 'Content-Length': '0'} 2023-11-02T19:06:05.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040c2af17d7e0ec50-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.050 DEBUG:boto:path=/ibtljn-64/obj85 2023-11-02T19:06:05.051 DEBUG:boto:auth_path=/ibtljn-64/obj85 2023-11-02T19:06:05.052 DEBUG:boto:path=/ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.053 DEBUG:boto:auth_path=/ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.054 DEBUG:boto:Method: GET 2023-11-02T19:06:05.054 DEBUG:boto:Path: /ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.055 DEBUG:boto:Data: 2023-11-02T19:06:05.055 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.056 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.056 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.057 DEBUG:boto:Params: {} 2023-11-02T19:06:05.057 DEBUG:boto:Token: None 2023-11-02T19:06:05.058 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.058 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6dFD7LRFRy9yLg1VczjrOP0o6Fk= 2023-11-02T19:06:05.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6dFD7LRFRy9yLg1VczjrOP0o6Fk=', 'Content-Length': '0'} 2023-11-02T19:06:05.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd3de5950062ff15-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.063 DEBUG:boto:path=/ibtljn-64/obj85 2023-11-02T19:06:05.063 DEBUG:boto:auth_path=/ibtljn-64/obj85 2023-11-02T19:06:05.063 DEBUG:boto:path=/ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.064 DEBUG:boto:auth_path=/ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.064 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.065 DEBUG:boto:Path: /ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.065 DEBUG:boto:Data: 2023-11-02T19:06:05.066 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.066 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.067 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.067 DEBUG:boto:Params: {} 2023-11-02T19:06:05.068 DEBUG:boto:Token: None 2023-11-02T19:06:05.068 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.069 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n5Pe7U3qmGQ9NrDiwU4tJWcT/Fw= 2023-11-02T19:06:05.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n5Pe7U3qmGQ9NrDiwU4tJWcT/Fw=', 'Content-Length': '0'} 2023-11-02T19:06:05.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cae91ed0722afa64-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.073 DEBUG:boto:path=/ibtljn-64/obj85 2023-11-02T19:06:05.074 DEBUG:boto:auth_path=/ibtljn-64/obj85 2023-11-02T19:06:05.074 DEBUG:boto:path=/ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.075 DEBUG:boto:auth_path=/ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.075 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.075 DEBUG:boto:Path: /ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.076 DEBUG:boto:Data: 2023-11-02T19:06:05.076 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.077 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.077 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.078 DEBUG:boto:Params: {} 2023-11-02T19:06:05.079 DEBUG:boto:Token: None 2023-11-02T19:06:05.079 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.080 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:n5Pe7U3qmGQ9NrDiwU4tJWcT/Fw= 2023-11-02T19:06:05.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:n5Pe7U3qmGQ9NrDiwU4tJWcT/Fw=', 'Content-Length': '0'} 2023-11-02T19:06:05.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d3678b5b1851ac2-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T19:06:05.085 DEBUG:boto:path=/ibtljn-64/obj85 2023-11-02T19:06:05.085 DEBUG:boto:auth_path=/ibtljn-64/obj85 2023-11-02T19:06:05.086 DEBUG:boto:path=/ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.086 DEBUG:boto:auth_path=/ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.087 DEBUG:boto:Method: GET 2023-11-02T19:06:05.088 DEBUG:boto:Path: /ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.088 DEBUG:boto:Data: 2023-11-02T19:06:05.089 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.089 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.089 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.090 DEBUG:boto:Params: {} 2023-11-02T19:06:05.090 DEBUG:boto:Token: None 2023-11-02T19:06:05.091 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.091 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6dFD7LRFRy9yLg1VczjrOP0o6Fk= 2023-11-02T19:06:05.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6dFD7LRFRy9yLg1VczjrOP0o6Fk=', 'Content-Length': '0'} 2023-11-02T19:06:05.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000634c632ecad72509-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.095 DEBUG:boto:path=/ibtljn-64/obj85 2023-11-02T19:06:05.097 DEBUG:boto:auth_path=/ibtljn-64/obj85 2023-11-02T19:06:05.098 DEBUG:boto:path=/ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.098 DEBUG:boto:auth_path=/ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.099 DEBUG:boto:Method: GET 2023-11-02T19:06:05.099 DEBUG:boto:Path: /ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.100 DEBUG:boto:Data: 2023-11-02T19:06:05.100 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.101 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.101 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.102 DEBUG:boto:Params: {} 2023-11-02T19:06:05.103 DEBUG:boto:Token: None 2023-11-02T19:06:05.104 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj85?versionId=null 2023-11-02T19:06:05.104 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6dFD7LRFRy9yLg1VczjrOP0o6Fk= 2023-11-02T19:06:05.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6dFD7LRFRy9yLg1VczjrOP0o6Fk=', 'Content-Length': '0'} 2023-11-02T19:06:05.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e04313cd268d312-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.108 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.109 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.109 DEBUG:boto:path=/ibtljn-64/?prefix=obj85 2023-11-02T19:06:05.110 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj85 2023-11-02T19:06:05.110 DEBUG:boto:Method: GET 2023-11-02T19:06:05.110 DEBUG:boto:Path: /ibtljn-64/?prefix=obj85 2023-11-02T19:06:05.111 DEBUG:boto:Data: 2023-11-02T19:06:05.111 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.112 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.112 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.113 DEBUG:boto:Params: {} 2023-11-02T19:06:05.113 DEBUG:boto:Token: None 2023-11-02T19:06:05.114 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.114 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f0bb477d6bcea5d-006543f31d-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.162 DEBUG:boto:b'ibtljn-64obj851000falseobj852023-11-02T19:03:42.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.163 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.164 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.164 DEBUG:boto:path=/ibtljn-64/?prefix=obj85 2023-11-02T19:06:05.164 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj85 2023-11-02T19:06:05.165 DEBUG:boto:Method: GET 2023-11-02T19:06:05.165 DEBUG:boto:Path: /ibtljn-64/?prefix=obj85 2023-11-02T19:06:05.166 DEBUG:boto:Data: 2023-11-02T19:06:05.166 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.166 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.167 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.167 DEBUG:boto:Params: {} 2023-11-02T19:06:05.168 DEBUG:boto:Token: None 2023-11-02T19:06:05.168 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.169 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006edfebb9a25bca17-006543f31d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.179 DEBUG:boto:b'ibtljn-64obj851000falseobj852023-11-02T19:03:42.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.180 DEBUG:boto:path=/ibtljn-64/obj85 2023-11-02T19:06:05.181 DEBUG:boto:auth_path=/ibtljn-64/obj85 2023-11-02T19:06:05.182 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.182 DEBUG:boto:Path: /ibtljn-64/obj85 2023-11-02T19:06:05.183 DEBUG:boto:Data: 2023-11-02T19:06:05.184 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.184 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.184 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.185 DEBUG:boto:Params: {} 2023-11-02T19:06:05.185 DEBUG:boto:Token: None 2023-11-02T19:06:05.186 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj85 2023-11-02T19:06:05.186 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2RkoJHBjMUdMeJhVUMPHtdNXDyw= 2023-11-02T19:06:05.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2RkoJHBjMUdMeJhVUMPHtdNXDyw=', 'Content-Length': '0'} 2023-11-02T19:06:05.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000643373e64b2fe181-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.190 DEBUG:boto:path=/ibtljn-64/obj85 2023-11-02T19:06:05.191 DEBUG:boto:auth_path=/ibtljn-64/obj85 2023-11-02T19:06:05.191 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.191 DEBUG:boto:Path: /ibtljn-64/obj85 2023-11-02T19:06:05.192 DEBUG:boto:Data: 2023-11-02T19:06:05.192 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.193 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.193 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.193 DEBUG:boto:Params: {} 2023-11-02T19:06:05.194 DEBUG:boto:Token: None 2023-11-02T19:06:05.194 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj85 2023-11-02T19:06:05.195 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2RkoJHBjMUdMeJhVUMPHtdNXDyw= 2023-11-02T19:06:05.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2RkoJHBjMUdMeJhVUMPHtdNXDyw=', 'Content-Length': '0'} 2023-11-02T19:06:05.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cc2604bc7781fb6-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-11-02T19:06:05.199 DEBUG:boto:path=/ibtljn-64/obj85 2023-11-02T19:06:05.199 DEBUG:boto:auth_path=/ibtljn-64/obj85 2023-11-02T19:06:05.200 DEBUG:boto:Method: GET 2023-11-02T19:06:05.200 DEBUG:boto:Path: /ibtljn-64/obj85 2023-11-02T19:06:05.201 DEBUG:boto:Data: 2023-11-02T19:06:05.201 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.202 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.202 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.203 DEBUG:boto:Params: {} 2023-11-02T19:06:05.203 DEBUG:boto:Token: None 2023-11-02T19:06:05.204 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj85 2023-11-02T19:06:05.204 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JeBVegvmTcAI/lB7QcJtYjljuuk= 2023-11-02T19:06:05.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JeBVegvmTcAI/lB7QcJtYjljuuk=', 'Content-Length': '0'} 2023-11-02T19:06:05.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd8b655a9f2baceb-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.208 DEBUG:boto:path=/ibtljn-64/obj85 2023-11-02T19:06:05.208 DEBUG:boto:auth_path=/ibtljn-64/obj85 2023-11-02T19:06:05.209 DEBUG:boto:Method: GET 2023-11-02T19:06:05.209 DEBUG:boto:Path: /ibtljn-64/obj85 2023-11-02T19:06:05.210 DEBUG:boto:Data: 2023-11-02T19:06:05.210 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.210 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.211 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.211 DEBUG:boto:Params: {} 2023-11-02T19:06:05.212 DEBUG:boto:Token: None 2023-11-02T19:06:05.212 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj85 2023-11-02T19:06:05.212 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JeBVegvmTcAI/lB7QcJtYjljuuk= 2023-11-02T19:06:05.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JeBVegvmTcAI/lB7QcJtYjljuuk=', 'Content-Length': '0'} 2023-11-02T19:06:05.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000524caa8e0571520f-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.258 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.259 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.259 DEBUG:boto:path=/ibtljn-64/?prefix=obj85 2023-11-02T19:06:05.260 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj85 2023-11-02T19:06:05.260 DEBUG:boto:Method: GET 2023-11-02T19:06:05.261 DEBUG:boto:Path: /ibtljn-64/?prefix=obj85 2023-11-02T19:06:05.261 DEBUG:boto:Data: 2023-11-02T19:06:05.262 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.262 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.263 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.263 DEBUG:boto:Params: {} 2023-11-02T19:06:05.264 DEBUG:boto:Token: None 2023-11-02T19:06:05.264 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.265 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000188e4c28ea1b59b0-006543f31d-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.270 DEBUG:boto:b'ibtljn-64obj851000falseobj852023-11-02T19:03:42.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.271 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.272 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.272 DEBUG:boto:path=/ibtljn-64/?prefix=obj85 2023-11-02T19:06:05.272 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj85 2023-11-02T19:06:05.273 DEBUG:boto:Method: GET 2023-11-02T19:06:05.273 DEBUG:boto:Path: /ibtljn-64/?prefix=obj85 2023-11-02T19:06:05.274 DEBUG:boto:Data: 2023-11-02T19:06:05.274 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.275 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.275 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.276 DEBUG:boto:Params: {} 2023-11-02T19:06:05.276 DEBUG:boto:Token: None 2023-11-02T19:06:05.277 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.277 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3103c381184d50c-006543f31d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.288 DEBUG:boto:b'ibtljn-64obj851000falseobj852023-11-02T19:03:42.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T19:06:05.289 DEBUG:boto:path=/ibtljn-64/obj87 2023-11-02T19:06:05.289 DEBUG:boto:auth_path=/ibtljn-64/obj87 2023-11-02T19:06:05.290 DEBUG:boto:path=/ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.290 DEBUG:boto:auth_path=/ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.291 DEBUG:boto:Method: GET 2023-11-02T19:06:05.291 DEBUG:boto:Path: /ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.292 DEBUG:boto:Data: 2023-11-02T19:06:05.292 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.292 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.293 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.293 DEBUG:boto:Params: {} 2023-11-02T19:06:05.294 DEBUG:boto:Token: None 2023-11-02T19:06:05.294 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.295 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e95P1CRfSKq8PjgNQyrFZdBa9LM= 2023-11-02T19:06:05.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e95P1CRfSKq8PjgNQyrFZdBa9LM=', 'Content-Length': '0'} 2023-11-02T19:06:05.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a5a0f1ad3c4a27d-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.299 DEBUG:boto:path=/ibtljn-64/obj87 2023-11-02T19:06:05.301 DEBUG:boto:auth_path=/ibtljn-64/obj87 2023-11-02T19:06:05.302 DEBUG:boto:path=/ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.302 DEBUG:boto:auth_path=/ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.302 DEBUG:boto:Method: GET 2023-11-02T19:06:05.303 DEBUG:boto:Path: /ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.303 DEBUG:boto:Data: 2023-11-02T19:06:05.304 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.304 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.305 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.305 DEBUG:boto:Params: {} 2023-11-02T19:06:05.306 DEBUG:boto:Token: None 2023-11-02T19:06:05.306 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.307 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e95P1CRfSKq8PjgNQyrFZdBa9LM= 2023-11-02T19:06:05.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e95P1CRfSKq8PjgNQyrFZdBa9LM=', 'Content-Length': '0'} 2023-11-02T19:06:05.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1170f15dd650f23-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.311 DEBUG:boto:path=/ibtljn-64/obj87 2023-11-02T19:06:05.311 DEBUG:boto:auth_path=/ibtljn-64/obj87 2023-11-02T19:06:05.311 DEBUG:boto:path=/ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.312 DEBUG:boto:auth_path=/ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.312 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.313 DEBUG:boto:Path: /ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.313 DEBUG:boto:Data: 2023-11-02T19:06:05.314 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.314 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.315 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.315 DEBUG:boto:Params: {} 2023-11-02T19:06:05.315 DEBUG:boto:Token: None 2023-11-02T19:06:05.316 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.316 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9k1kMgJ8U2sCuVNRNqEno//tE4k= 2023-11-02T19:06:05.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9k1kMgJ8U2sCuVNRNqEno//tE4k=', 'Content-Length': '0'} 2023-11-02T19:06:05.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029c77e86a96a1487-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.320 DEBUG:boto:path=/ibtljn-64/obj87 2023-11-02T19:06:05.321 DEBUG:boto:auth_path=/ibtljn-64/obj87 2023-11-02T19:06:05.321 DEBUG:boto:path=/ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.321 DEBUG:boto:auth_path=/ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.322 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.322 DEBUG:boto:Path: /ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.323 DEBUG:boto:Data: 2023-11-02T19:06:05.323 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.324 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.324 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.324 DEBUG:boto:Params: {} 2023-11-02T19:06:05.325 DEBUG:boto:Token: None 2023-11-02T19:06:05.325 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.326 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9k1kMgJ8U2sCuVNRNqEno//tE4k= 2023-11-02T19:06:05.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9k1kMgJ8U2sCuVNRNqEno//tE4k=', 'Content-Length': '0'} 2023-11-02T19:06:05.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078ae763928ef5d27-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T19:06:05.330 DEBUG:boto:path=/ibtljn-64/obj87 2023-11-02T19:06:05.330 DEBUG:boto:auth_path=/ibtljn-64/obj87 2023-11-02T19:06:05.331 DEBUG:boto:path=/ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.331 DEBUG:boto:auth_path=/ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.332 DEBUG:boto:Method: GET 2023-11-02T19:06:05.332 DEBUG:boto:Path: /ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.332 DEBUG:boto:Data: 2023-11-02T19:06:05.333 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.333 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.334 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.334 DEBUG:boto:Params: {} 2023-11-02T19:06:05.335 DEBUG:boto:Token: None 2023-11-02T19:06:05.335 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.336 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e95P1CRfSKq8PjgNQyrFZdBa9LM= 2023-11-02T19:06:05.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e95P1CRfSKq8PjgNQyrFZdBa9LM=', 'Content-Length': '0'} 2023-11-02T19:06:05.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8f71e944c174d48-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.386 DEBUG:boto:path=/ibtljn-64/obj87 2023-11-02T19:06:05.387 DEBUG:boto:auth_path=/ibtljn-64/obj87 2023-11-02T19:06:05.387 DEBUG:boto:path=/ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.388 DEBUG:boto:auth_path=/ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.388 DEBUG:boto:Method: GET 2023-11-02T19:06:05.389 DEBUG:boto:Path: /ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.389 DEBUG:boto:Data: 2023-11-02T19:06:05.390 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.390 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.390 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.391 DEBUG:boto:Params: {} 2023-11-02T19:06:05.391 DEBUG:boto:Token: None 2023-11-02T19:06:05.392 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj87?versionId=null 2023-11-02T19:06:05.392 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:e95P1CRfSKq8PjgNQyrFZdBa9LM= 2023-11-02T19:06:05.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e95P1CRfSKq8PjgNQyrFZdBa9LM=', 'Content-Length': '0'} 2023-11-02T19:06:05.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f474251f189564d7-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.397 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.397 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.398 DEBUG:boto:path=/ibtljn-64/?prefix=obj87 2023-11-02T19:06:05.398 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj87 2023-11-02T19:06:05.399 DEBUG:boto:Method: GET 2023-11-02T19:06:05.399 DEBUG:boto:Path: /ibtljn-64/?prefix=obj87 2023-11-02T19:06:05.399 DEBUG:boto:Data: 2023-11-02T19:06:05.400 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.400 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.401 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.401 DEBUG:boto:Params: {} 2023-11-02T19:06:05.402 DEBUG:boto:Token: None 2023-11-02T19:06:05.402 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.403 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006487076727d38821-006543f31d-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.409 DEBUG:boto:b'ibtljn-64obj871000falseobj872023-11-02T19:03:43.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.409 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.410 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.410 DEBUG:boto:path=/ibtljn-64/?prefix=obj87 2023-11-02T19:06:05.411 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj87 2023-11-02T19:06:05.411 DEBUG:boto:Method: GET 2023-11-02T19:06:05.412 DEBUG:boto:Path: /ibtljn-64/?prefix=obj87 2023-11-02T19:06:05.412 DEBUG:boto:Data: 2023-11-02T19:06:05.413 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.413 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.413 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.414 DEBUG:boto:Params: {} 2023-11-02T19:06:05.414 DEBUG:boto:Token: None 2023-11-02T19:06:05.415 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.415 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000172128f9a78e1fa4-006543f31d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.466 DEBUG:boto:b'ibtljn-64obj871000falseobj872023-11-02T19:03:43.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.467 DEBUG:boto:path=/ibtljn-64/obj87 2023-11-02T19:06:05.469 DEBUG:boto:auth_path=/ibtljn-64/obj87 2023-11-02T19:06:05.470 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.470 DEBUG:boto:Path: /ibtljn-64/obj87 2023-11-02T19:06:05.471 DEBUG:boto:Data: 2023-11-02T19:06:05.471 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.472 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.472 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.473 DEBUG:boto:Params: {} 2023-11-02T19:06:05.473 DEBUG:boto:Token: None 2023-11-02T19:06:05.474 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj87 2023-11-02T19:06:05.474 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8VSQVZOR8XGOyP2wa1qtKLmCZw= 2023-11-02T19:06:05.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8VSQVZOR8XGOyP2wa1qtKLmCZw=', 'Content-Length': '0'} 2023-11-02T19:06:05.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcb0f65d593c8263-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.478 DEBUG:boto:path=/ibtljn-64/obj87 2023-11-02T19:06:05.479 DEBUG:boto:auth_path=/ibtljn-64/obj87 2023-11-02T19:06:05.479 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.480 DEBUG:boto:Path: /ibtljn-64/obj87 2023-11-02T19:06:05.480 DEBUG:boto:Data: 2023-11-02T19:06:05.480 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.481 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.481 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.482 DEBUG:boto:Params: {} 2023-11-02T19:06:05.482 DEBUG:boto:Token: None 2023-11-02T19:06:05.483 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj87 2023-11-02T19:06:05.483 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:O8VSQVZOR8XGOyP2wa1qtKLmCZw= 2023-11-02T19:06:05.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:O8VSQVZOR8XGOyP2wa1qtKLmCZw=', 'Content-Length': '0'} 2023-11-02T19:06:05.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000983683d9653cd10b-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-11-02T19:06:05.488 DEBUG:boto:path=/ibtljn-64/obj87 2023-11-02T19:06:05.488 DEBUG:boto:auth_path=/ibtljn-64/obj87 2023-11-02T19:06:05.488 DEBUG:boto:Method: GET 2023-11-02T19:06:05.489 DEBUG:boto:Path: /ibtljn-64/obj87 2023-11-02T19:06:05.489 DEBUG:boto:Data: 2023-11-02T19:06:05.490 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.490 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.491 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.491 DEBUG:boto:Params: {} 2023-11-02T19:06:05.492 DEBUG:boto:Token: None 2023-11-02T19:06:05.492 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj87 2023-11-02T19:06:05.492 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kB+UJVZnhT4XSaGVDBoOqZvJ8Tw= 2023-11-02T19:06:05.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kB+UJVZnhT4XSaGVDBoOqZvJ8Tw=', 'Content-Length': '0'} 2023-11-02T19:06:05.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b2eaf441f30cff6-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.497 DEBUG:boto:path=/ibtljn-64/obj87 2023-11-02T19:06:05.497 DEBUG:boto:auth_path=/ibtljn-64/obj87 2023-11-02T19:06:05.497 DEBUG:boto:Method: GET 2023-11-02T19:06:05.498 DEBUG:boto:Path: /ibtljn-64/obj87 2023-11-02T19:06:05.498 DEBUG:boto:Data: 2023-11-02T19:06:05.499 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.499 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.500 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.500 DEBUG:boto:Params: {} 2023-11-02T19:06:05.501 DEBUG:boto:Token: None 2023-11-02T19:06:05.501 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj87 2023-11-02T19:06:05.502 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kB+UJVZnhT4XSaGVDBoOqZvJ8Tw= 2023-11-02T19:06:05.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kB+UJVZnhT4XSaGVDBoOqZvJ8Tw=', 'Content-Length': '0'} 2023-11-02T19:06:05.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cbd80aa079ff755-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.506 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.506 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.507 DEBUG:boto:path=/ibtljn-64/?prefix=obj87 2023-11-02T19:06:05.507 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj87 2023-11-02T19:06:05.508 DEBUG:boto:Method: GET 2023-11-02T19:06:05.508 DEBUG:boto:Path: /ibtljn-64/?prefix=obj87 2023-11-02T19:06:05.509 DEBUG:boto:Data: 2023-11-02T19:06:05.509 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.509 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.510 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.510 DEBUG:boto:Params: {} 2023-11-02T19:06:05.511 DEBUG:boto:Token: None 2023-11-02T19:06:05.511 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.512 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e5e4227431a1930-006543f31d-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.518 DEBUG:boto:b'ibtljn-64obj871000falseobj872023-11-02T19:03:43.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.519 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.519 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.520 DEBUG:boto:path=/ibtljn-64/?prefix=obj87 2023-11-02T19:06:05.520 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj87 2023-11-02T19:06:05.520 DEBUG:boto:Method: GET 2023-11-02T19:06:05.521 DEBUG:boto:Path: /ibtljn-64/?prefix=obj87 2023-11-02T19:06:05.521 DEBUG:boto:Data: 2023-11-02T19:06:05.522 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.522 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.523 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.523 DEBUG:boto:Params: {} 2023-11-02T19:06:05.524 DEBUG:boto:Token: None 2023-11-02T19:06:05.524 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.525 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011c0e347d60fdeac-006543f31d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.535 DEBUG:boto:b'ibtljn-64obj871000falseobj872023-11-02T19:03:43.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T19:06:05.536 DEBUG:boto:path=/ibtljn-64/obj92 2023-11-02T19:06:05.536 DEBUG:boto:auth_path=/ibtljn-64/obj92 2023-11-02T19:06:05.537 DEBUG:boto:path=/ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.537 DEBUG:boto:auth_path=/ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.538 DEBUG:boto:Method: GET 2023-11-02T19:06:05.538 DEBUG:boto:Path: /ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.539 DEBUG:boto:Data: 2023-11-02T19:06:05.539 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.540 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.540 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.540 DEBUG:boto:Params: {} 2023-11-02T19:06:05.541 DEBUG:boto:Token: None 2023-11-02T19:06:05.541 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.542 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M/yGz4RxGvlr2l63wQWhpQx97t8= 2023-11-02T19:06:05.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M/yGz4RxGvlr2l63wQWhpQx97t8=', 'Content-Length': '0'} 2023-11-02T19:06:05.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f0c0c0083a217ea-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.586 DEBUG:boto:path=/ibtljn-64/obj92 2023-11-02T19:06:05.587 DEBUG:boto:auth_path=/ibtljn-64/obj92 2023-11-02T19:06:05.587 DEBUG:boto:path=/ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.588 DEBUG:boto:auth_path=/ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.588 DEBUG:boto:Method: GET 2023-11-02T19:06:05.589 DEBUG:boto:Path: /ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.589 DEBUG:boto:Data: 2023-11-02T19:06:05.590 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.590 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.591 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.591 DEBUG:boto:Params: {} 2023-11-02T19:06:05.592 DEBUG:boto:Token: None 2023-11-02T19:06:05.592 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.593 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M/yGz4RxGvlr2l63wQWhpQx97t8= 2023-11-02T19:06:05.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M/yGz4RxGvlr2l63wQWhpQx97t8=', 'Content-Length': '0'} 2023-11-02T19:06:05.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e52977f676659b2-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.597 DEBUG:boto:path=/ibtljn-64/obj92 2023-11-02T19:06:05.598 DEBUG:boto:auth_path=/ibtljn-64/obj92 2023-11-02T19:06:05.598 DEBUG:boto:path=/ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.599 DEBUG:boto:auth_path=/ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.599 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.600 DEBUG:boto:Path: /ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.600 DEBUG:boto:Data: 2023-11-02T19:06:05.600 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.601 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.601 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.602 DEBUG:boto:Params: {} 2023-11-02T19:06:05.602 DEBUG:boto:Token: None 2023-11-02T19:06:05.603 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.603 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xRcX6+XMEqu9kTWpzFhZkwiRIsI= 2023-11-02T19:06:05.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xRcX6+XMEqu9kTWpzFhZkwiRIsI=', 'Content-Length': '0'} 2023-11-02T19:06:05.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5580e719adb9c38-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.607 DEBUG:boto:path=/ibtljn-64/obj92 2023-11-02T19:06:05.607 DEBUG:boto:auth_path=/ibtljn-64/obj92 2023-11-02T19:06:05.608 DEBUG:boto:path=/ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.608 DEBUG:boto:auth_path=/ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.609 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.609 DEBUG:boto:Path: /ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.610 DEBUG:boto:Data: 2023-11-02T19:06:05.610 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.610 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.611 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.611 DEBUG:boto:Params: {} 2023-11-02T19:06:05.612 DEBUG:boto:Token: None 2023-11-02T19:06:05.612 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.613 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xRcX6+XMEqu9kTWpzFhZkwiRIsI= 2023-11-02T19:06:05.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xRcX6+XMEqu9kTWpzFhZkwiRIsI=', 'Content-Length': '0'} 2023-11-02T19:06:05.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000599a4fdf5d7bd79c-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T19:06:05.617 DEBUG:boto:path=/ibtljn-64/obj92 2023-11-02T19:06:05.618 DEBUG:boto:auth_path=/ibtljn-64/obj92 2023-11-02T19:06:05.618 DEBUG:boto:path=/ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.619 DEBUG:boto:auth_path=/ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.619 DEBUG:boto:Method: GET 2023-11-02T19:06:05.619 DEBUG:boto:Path: /ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.620 DEBUG:boto:Data: 2023-11-02T19:06:05.620 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.621 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.621 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.622 DEBUG:boto:Params: {} 2023-11-02T19:06:05.622 DEBUG:boto:Token: None 2023-11-02T19:06:05.623 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.623 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M/yGz4RxGvlr2l63wQWhpQx97t8= 2023-11-02T19:06:05.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M/yGz4RxGvlr2l63wQWhpQx97t8=', 'Content-Length': '0'} 2023-11-02T19:06:05.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a879497dc3ad30c6-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.628 DEBUG:boto:path=/ibtljn-64/obj92 2023-11-02T19:06:05.628 DEBUG:boto:auth_path=/ibtljn-64/obj92 2023-11-02T19:06:05.628 DEBUG:boto:path=/ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.629 DEBUG:boto:auth_path=/ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.629 DEBUG:boto:Method: GET 2023-11-02T19:06:05.630 DEBUG:boto:Path: /ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.630 DEBUG:boto:Data: 2023-11-02T19:06:05.631 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.631 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.631 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.632 DEBUG:boto:Params: {} 2023-11-02T19:06:05.632 DEBUG:boto:Token: None 2023-11-02T19:06:05.633 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj92?versionId=null 2023-11-02T19:06:05.633 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:M/yGz4RxGvlr2l63wQWhpQx97t8= 2023-11-02T19:06:05.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:M/yGz4RxGvlr2l63wQWhpQx97t8=', 'Content-Length': '0'} 2023-11-02T19:06:05.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80d51d1e4bad645-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.678 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.679 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.679 DEBUG:boto:path=/ibtljn-64/?prefix=obj92 2023-11-02T19:06:05.680 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj92 2023-11-02T19:06:05.680 DEBUG:boto:Method: GET 2023-11-02T19:06:05.681 DEBUG:boto:Path: /ibtljn-64/?prefix=obj92 2023-11-02T19:06:05.681 DEBUG:boto:Data: 2023-11-02T19:06:05.682 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.682 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.683 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.683 DEBUG:boto:Params: {} 2023-11-02T19:06:05.684 DEBUG:boto:Token: None 2023-11-02T19:06:05.684 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.685 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000594ce42aa8fd57a0-006543f31d-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.690 DEBUG:boto:b'ibtljn-64obj921000falseobj922023-11-02T19:03:43.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.691 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.692 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.692 DEBUG:boto:path=/ibtljn-64/?prefix=obj92 2023-11-02T19:06:05.693 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj92 2023-11-02T19:06:05.693 DEBUG:boto:Method: GET 2023-11-02T19:06:05.694 DEBUG:boto:Path: /ibtljn-64/?prefix=obj92 2023-11-02T19:06:05.694 DEBUG:boto:Data: 2023-11-02T19:06:05.694 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.695 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.695 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.696 DEBUG:boto:Params: {} 2023-11-02T19:06:05.696 DEBUG:boto:Token: None 2023-11-02T19:06:05.697 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.697 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095b343f2819920b1-006543f31d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.704 DEBUG:boto:b'ibtljn-64obj921000falseobj922023-11-02T19:03:43.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.705 DEBUG:boto:path=/ibtljn-64/obj92 2023-11-02T19:06:05.705 DEBUG:boto:auth_path=/ibtljn-64/obj92 2023-11-02T19:06:05.706 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.706 DEBUG:boto:Path: /ibtljn-64/obj92 2023-11-02T19:06:05.707 DEBUG:boto:Data: 2023-11-02T19:06:05.707 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.707 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.709 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.710 DEBUG:boto:Params: {} 2023-11-02T19:06:05.710 DEBUG:boto:Token: None 2023-11-02T19:06:05.711 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj92 2023-11-02T19:06:05.711 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b8V9H95C/N6Z4FiK6qKctartB+w= 2023-11-02T19:06:05.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b8V9H95C/N6Z4FiK6qKctartB+w=', 'Content-Length': '0'} 2023-11-02T19:06:05.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000290fbdc26ca80f23-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.716 DEBUG:boto:path=/ibtljn-64/obj92 2023-11-02T19:06:05.716 DEBUG:boto:auth_path=/ibtljn-64/obj92 2023-11-02T19:06:05.717 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.717 DEBUG:boto:Path: /ibtljn-64/obj92 2023-11-02T19:06:05.718 DEBUG:boto:Data: 2023-11-02T19:06:05.718 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.719 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.719 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.720 DEBUG:boto:Params: {} 2023-11-02T19:06:05.720 DEBUG:boto:Token: None 2023-11-02T19:06:05.721 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj92 2023-11-02T19:06:05.721 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:b8V9H95C/N6Z4FiK6qKctartB+w= 2023-11-02T19:06:05.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b8V9H95C/N6Z4FiK6qKctartB+w=', 'Content-Length': '0'} 2023-11-02T19:06:05.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d46dfbf3afc0b89-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-11-02T19:06:05.726 DEBUG:boto:path=/ibtljn-64/obj92 2023-11-02T19:06:05.726 DEBUG:boto:auth_path=/ibtljn-64/obj92 2023-11-02T19:06:05.727 DEBUG:boto:Method: GET 2023-11-02T19:06:05.727 DEBUG:boto:Path: /ibtljn-64/obj92 2023-11-02T19:06:05.728 DEBUG:boto:Data: 2023-11-02T19:06:05.728 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.729 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.729 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.729 DEBUG:boto:Params: {} 2023-11-02T19:06:05.730 DEBUG:boto:Token: None 2023-11-02T19:06:05.731 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj92 2023-11-02T19:06:05.731 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VxjF3I33NtQ0hdMl7qpwoEJ5KhU= 2023-11-02T19:06:05.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VxjF3I33NtQ0hdMl7qpwoEJ5KhU=', 'Content-Length': '0'} 2023-11-02T19:06:05.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009776f034254eb395-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.782 DEBUG:boto:path=/ibtljn-64/obj92 2023-11-02T19:06:05.783 DEBUG:boto:auth_path=/ibtljn-64/obj92 2023-11-02T19:06:05.783 DEBUG:boto:Method: GET 2023-11-02T19:06:05.784 DEBUG:boto:Path: /ibtljn-64/obj92 2023-11-02T19:06:05.784 DEBUG:boto:Data: 2023-11-02T19:06:05.785 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.785 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.786 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.786 DEBUG:boto:Params: {} 2023-11-02T19:06:05.787 DEBUG:boto:Token: None 2023-11-02T19:06:05.787 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj92 2023-11-02T19:06:05.788 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:VxjF3I33NtQ0hdMl7qpwoEJ5KhU= 2023-11-02T19:06:05.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VxjF3I33NtQ0hdMl7qpwoEJ5KhU=', 'Content-Length': '0'} 2023-11-02T19:06:05.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007101f209296c4dc2-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.792 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.793 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.793 DEBUG:boto:path=/ibtljn-64/?prefix=obj92 2023-11-02T19:06:05.794 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj92 2023-11-02T19:06:05.794 DEBUG:boto:Method: GET 2023-11-02T19:06:05.795 DEBUG:boto:Path: /ibtljn-64/?prefix=obj92 2023-11-02T19:06:05.795 DEBUG:boto:Data: 2023-11-02T19:06:05.796 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.796 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.797 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.797 DEBUG:boto:Params: {} 2023-11-02T19:06:05.798 DEBUG:boto:Token: None 2023-11-02T19:06:05.798 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.799 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011164fbddb419a19-006543f31d-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.805 DEBUG:boto:b'ibtljn-64obj921000falseobj922023-11-02T19:03:43.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.806 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.807 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.807 DEBUG:boto:path=/ibtljn-64/?prefix=obj92 2023-11-02T19:06:05.808 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj92 2023-11-02T19:06:05.808 DEBUG:boto:Method: GET 2023-11-02T19:06:05.809 DEBUG:boto:Path: /ibtljn-64/?prefix=obj92 2023-11-02T19:06:05.809 DEBUG:boto:Data: 2023-11-02T19:06:05.809 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.810 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.810 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.811 DEBUG:boto:Params: {} 2023-11-02T19:06:05.812 DEBUG:boto:Token: None 2023-11-02T19:06:05.812 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.813 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000635d0c9fd65cf2c4-006543f31d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.866 DEBUG:boto:b'ibtljn-64obj921000falseobj922023-11-02T19:03:43.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T19:06:05.868 DEBUG:boto:path=/ibtljn-64/obj93 2023-11-02T19:06:05.868 DEBUG:boto:auth_path=/ibtljn-64/obj93 2023-11-02T19:06:05.869 DEBUG:boto:path=/ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.869 DEBUG:boto:auth_path=/ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.870 DEBUG:boto:Method: GET 2023-11-02T19:06:05.870 DEBUG:boto:Path: /ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.871 DEBUG:boto:Data: 2023-11-02T19:06:05.871 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.872 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.872 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.873 DEBUG:boto:Params: {} 2023-11-02T19:06:05.873 DEBUG:boto:Token: None 2023-11-02T19:06:05.874 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.874 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MrlkeyvD3MFJLSup4Wto7KrHEg8= 2023-11-02T19:06:05.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MrlkeyvD3MFJLSup4Wto7KrHEg8=', 'Content-Length': '0'} 2023-11-02T19:06:05.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000add1c94a98a80e84-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.879 DEBUG:boto:path=/ibtljn-64/obj93 2023-11-02T19:06:05.880 DEBUG:boto:auth_path=/ibtljn-64/obj93 2023-11-02T19:06:05.880 DEBUG:boto:path=/ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.880 DEBUG:boto:auth_path=/ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.881 DEBUG:boto:Method: GET 2023-11-02T19:06:05.881 DEBUG:boto:Path: /ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.882 DEBUG:boto:Data: 2023-11-02T19:06:05.882 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.883 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.883 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.883 DEBUG:boto:Params: {} 2023-11-02T19:06:05.884 DEBUG:boto:Token: None 2023-11-02T19:06:05.884 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.885 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MrlkeyvD3MFJLSup4Wto7KrHEg8= 2023-11-02T19:06:05.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MrlkeyvD3MFJLSup4Wto7KrHEg8=', 'Content-Length': '0'} 2023-11-02T19:06:05.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d6e2a28a16cf3c1-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.889 DEBUG:boto:path=/ibtljn-64/obj93 2023-11-02T19:06:05.890 DEBUG:boto:auth_path=/ibtljn-64/obj93 2023-11-02T19:06:05.890 DEBUG:boto:path=/ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.891 DEBUG:boto:auth_path=/ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.891 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.892 DEBUG:boto:Path: /ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.892 DEBUG:boto:Data: 2023-11-02T19:06:05.893 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.893 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.894 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.894 DEBUG:boto:Params: {} 2023-11-02T19:06:05.895 DEBUG:boto:Token: None 2023-11-02T19:06:05.895 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.896 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ls/wkpXXfnNbKToJPTfRvwL8lKg= 2023-11-02T19:06:05.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ls/wkpXXfnNbKToJPTfRvwL8lKg=', 'Content-Length': '0'} 2023-11-02T19:06:05.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed364b5fda47af34-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.900 DEBUG:boto:path=/ibtljn-64/obj93 2023-11-02T19:06:05.900 DEBUG:boto:auth_path=/ibtljn-64/obj93 2023-11-02T19:06:05.901 DEBUG:boto:path=/ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.901 DEBUG:boto:auth_path=/ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.901 DEBUG:boto:Method: HEAD 2023-11-02T19:06:05.902 DEBUG:boto:Path: /ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.902 DEBUG:boto:Data: 2023-11-02T19:06:05.903 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.903 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.904 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.904 DEBUG:boto:Params: {} 2023-11-02T19:06:05.905 DEBUG:boto:Token: None 2023-11-02T19:06:05.905 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.906 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ls/wkpXXfnNbKToJPTfRvwL8lKg= 2023-11-02T19:06:05.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ls/wkpXXfnNbKToJPTfRvwL8lKg=', 'Content-Length': '0'} 2023-11-02T19:06:05.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f2717238226498d-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T19:06:05.911 DEBUG:boto:path=/ibtljn-64/obj93 2023-11-02T19:06:05.911 DEBUG:boto:auth_path=/ibtljn-64/obj93 2023-11-02T19:06:05.911 DEBUG:boto:path=/ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.913 DEBUG:boto:auth_path=/ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.914 DEBUG:boto:Method: GET 2023-11-02T19:06:05.914 DEBUG:boto:Path: /ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.915 DEBUG:boto:Data: 2023-11-02T19:06:05.915 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.916 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.916 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.917 DEBUG:boto:Params: {} 2023-11-02T19:06:05.917 DEBUG:boto:Token: None 2023-11-02T19:06:05.918 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.918 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MrlkeyvD3MFJLSup4Wto7KrHEg8= 2023-11-02T19:06:05.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MrlkeyvD3MFJLSup4Wto7KrHEg8=', 'Content-Length': '0'} 2023-11-02T19:06:05.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc3606c94d1a851d-006543f31d-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.966 DEBUG:boto:path=/ibtljn-64/obj93 2023-11-02T19:06:05.967 DEBUG:boto:auth_path=/ibtljn-64/obj93 2023-11-02T19:06:05.968 DEBUG:boto:path=/ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.970 DEBUG:boto:auth_path=/ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.970 DEBUG:boto:Method: GET 2023-11-02T19:06:05.971 DEBUG:boto:Path: /ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.971 DEBUG:boto:Data: 2023-11-02T19:06:05.972 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.972 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.973 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.973 DEBUG:boto:Params: {} 2023-11-02T19:06:05.974 DEBUG:boto:Token: None 2023-11-02T19:06:05.974 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/obj93?versionId=null 2023-11-02T19:06:05.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MrlkeyvD3MFJLSup4Wto7KrHEg8= 2023-11-02T19:06:05.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MrlkeyvD3MFJLSup4Wto7KrHEg8=', 'Content-Length': '0'} 2023-11-02T19:06:05.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a615daeb31660c4-006543f31d-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.979 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.980 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.980 DEBUG:boto:path=/ibtljn-64/?prefix=obj93 2023-11-02T19:06:05.981 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj93 2023-11-02T19:06:05.981 DEBUG:boto:Method: GET 2023-11-02T19:06:05.982 DEBUG:boto:Path: /ibtljn-64/?prefix=obj93 2023-11-02T19:06:05.982 DEBUG:boto:Data: 2023-11-02T19:06:05.983 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.983 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:05.983 DEBUG:boto:Port: 8001 2023-11-02T19:06:05.984 DEBUG:boto:Params: {} 2023-11-02T19:06:05.984 DEBUG:boto:Token: None 2023-11-02T19:06:05.985 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.985 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:05.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008796176a7f609ae7-006543f31d-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:05.991 DEBUG:boto:b'ibtljn-64obj931000falseobj932023-11-02T19:03:44.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:05.992 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:05.993 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:05.993 DEBUG:boto:path=/ibtljn-64/?prefix=obj93 2023-11-02T19:06:05.994 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj93 2023-11-02T19:06:05.994 DEBUG:boto:Method: GET 2023-11-02T19:06:05.994 DEBUG:boto:Path: /ibtljn-64/?prefix=obj93 2023-11-02T19:06:05.995 DEBUG:boto:Data: 2023-11-02T19:06:05.995 DEBUG:boto:Headers: {} 2023-11-02T19:06:05.996 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:05.996 DEBUG:boto:Port: 8000 2023-11-02T19:06:05.997 DEBUG:boto:Params: {} 2023-11-02T19:06:05.997 DEBUG:boto:Token: None 2023-11-02T19:06:05.998 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:05 GMT /ibtljn-64/ 2023-11-02T19:06:05.998 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE= 2023-11-02T19:06:05.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:05 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IfaMpL5YDoQuQUu+OCSCdJygloE=', 'Content-Length': '0'} 2023-11-02T19:06:06.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef9d738cd5a855ad-006543f31d-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.009 DEBUG:boto:b'ibtljn-64obj931000falseobj932023-11-02T19:03:44.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:06.009 DEBUG:boto:path=/ibtljn-64/obj93 2023-11-02T19:06:06.010 DEBUG:boto:auth_path=/ibtljn-64/obj93 2023-11-02T19:06:06.010 DEBUG:boto:Method: HEAD 2023-11-02T19:06:06.011 DEBUG:boto:Path: /ibtljn-64/obj93 2023-11-02T19:06:06.011 DEBUG:boto:Data: 2023-11-02T19:06:06.012 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.012 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:06.012 DEBUG:boto:Port: 8001 2023-11-02T19:06:06.013 DEBUG:boto:Params: {} 2023-11-02T19:06:06.013 DEBUG:boto:Token: None 2023-11-02T19:06:06.014 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj93 2023-11-02T19:06:06.015 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V3G3sdcUGiy2PljELHuKFxPsza8= 2023-11-02T19:06:06.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V3G3sdcUGiy2PljELHuKFxPsza8=', 'Content-Length': '0'} 2023-11-02T19:06:06.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2c99aaab86ea8f3-006543f31e-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.018 DEBUG:boto:path=/ibtljn-64/obj93 2023-11-02T19:06:06.018 DEBUG:boto:auth_path=/ibtljn-64/obj93 2023-11-02T19:06:06.019 DEBUG:boto:Method: HEAD 2023-11-02T19:06:06.019 DEBUG:boto:Path: /ibtljn-64/obj93 2023-11-02T19:06:06.020 DEBUG:boto:Data: 2023-11-02T19:06:06.020 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.021 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:06.021 DEBUG:boto:Port: 8000 2023-11-02T19:06:06.021 DEBUG:boto:Params: {} 2023-11-02T19:06:06.022 DEBUG:boto:Token: None 2023-11-02T19:06:06.022 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj93 2023-11-02T19:06:06.023 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:V3G3sdcUGiy2PljELHuKFxPsza8= 2023-11-02T19:06:06.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:V3G3sdcUGiy2PljELHuKFxPsza8=', 'Content-Length': '0'} 2023-11-02T19:06:06.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002043eed696db71e5-006543f31e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-11-02T19:06:06.028 DEBUG:boto:path=/ibtljn-64/obj93 2023-11-02T19:06:06.029 DEBUG:boto:auth_path=/ibtljn-64/obj93 2023-11-02T19:06:06.029 DEBUG:boto:Method: GET 2023-11-02T19:06:06.030 DEBUG:boto:Path: /ibtljn-64/obj93 2023-11-02T19:06:06.031 DEBUG:boto:Data: 2023-11-02T19:06:06.031 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.032 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:06.032 DEBUG:boto:Port: 8001 2023-11-02T19:06:06.033 DEBUG:boto:Params: {} 2023-11-02T19:06:06.033 DEBUG:boto:Token: None 2023-11-02T19:06:06.034 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj93 2023-11-02T19:06:06.034 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5vhkauG07lX8NiYRYo/u64H+7E0= 2023-11-02T19:06:06.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5vhkauG07lX8NiYRYo/u64H+7E0=', 'Content-Length': '0'} 2023-11-02T19:06:06.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a48d7acdf6bf9f86-006543f31e-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.039 DEBUG:boto:path=/ibtljn-64/obj93 2023-11-02T19:06:06.040 DEBUG:boto:auth_path=/ibtljn-64/obj93 2023-11-02T19:06:06.040 DEBUG:boto:Method: GET 2023-11-02T19:06:06.041 DEBUG:boto:Path: /ibtljn-64/obj93 2023-11-02T19:06:06.041 DEBUG:boto:Data: 2023-11-02T19:06:06.041 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.042 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:06.043 DEBUG:boto:Port: 8000 2023-11-02T19:06:06.043 DEBUG:boto:Params: {} 2023-11-02T19:06:06.044 DEBUG:boto:Token: None 2023-11-02T19:06:06.044 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj93 2023-11-02T19:06:06.045 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5vhkauG07lX8NiYRYo/u64H+7E0= 2023-11-02T19:06:06.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5vhkauG07lX8NiYRYo/u64H+7E0=', 'Content-Length': '0'} 2023-11-02T19:06:06.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f88cb495d922436-006543f31e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.090 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:06.091 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:06.091 DEBUG:boto:path=/ibtljn-64/?prefix=obj93 2023-11-02T19:06:06.092 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj93 2023-11-02T19:06:06.092 DEBUG:boto:Method: GET 2023-11-02T19:06:06.093 DEBUG:boto:Path: /ibtljn-64/?prefix=obj93 2023-11-02T19:06:06.093 DEBUG:boto:Data: 2023-11-02T19:06:06.094 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.094 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:06.094 DEBUG:boto:Port: 8001 2023-11-02T19:06:06.095 DEBUG:boto:Params: {} 2023-11-02T19:06:06.095 DEBUG:boto:Token: None 2023-11-02T19:06:06.096 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/ 2023-11-02T19:06:06.097 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gZVcryDIXBoYK6bygXqtnA1bCGI= 2023-11-02T19:06:06.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gZVcryDIXBoYK6bygXqtnA1bCGI=', 'Content-Length': '0'} 2023-11-02T19:06:06.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f63d5c4a93fa9bbf-006543f31e-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.103 DEBUG:boto:b'ibtljn-64obj931000falseobj932023-11-02T19:03:44.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:06.104 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:06.105 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:06.105 DEBUG:boto:path=/ibtljn-64/?prefix=obj93 2023-11-02T19:06:06.106 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj93 2023-11-02T19:06:06.106 DEBUG:boto:Method: GET 2023-11-02T19:06:06.107 DEBUG:boto:Path: /ibtljn-64/?prefix=obj93 2023-11-02T19:06:06.107 DEBUG:boto:Data: 2023-11-02T19:06:06.108 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.108 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:06.109 DEBUG:boto:Port: 8000 2023-11-02T19:06:06.109 DEBUG:boto:Params: {} 2023-11-02T19:06:06.110 DEBUG:boto:Token: None 2023-11-02T19:06:06.110 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/ 2023-11-02T19:06:06.111 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gZVcryDIXBoYK6bygXqtnA1bCGI= 2023-11-02T19:06:06.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gZVcryDIXBoYK6bygXqtnA1bCGI=', 'Content-Length': '0'} 2023-11-02T19:06:06.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe13fe0baccb6a39-006543f31e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.122 DEBUG:boto:b'ibtljn-64obj931000falseobj932023-11-02T19:03:44.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:06.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T19:06:06.123 DEBUG:boto:path=/ibtljn-64/obj95 2023-11-02T19:06:06.124 DEBUG:boto:auth_path=/ibtljn-64/obj95 2023-11-02T19:06:06.124 DEBUG:boto:path=/ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.125 DEBUG:boto:auth_path=/ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.125 DEBUG:boto:Method: GET 2023-11-02T19:06:06.126 DEBUG:boto:Path: /ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.126 DEBUG:boto:Data: 2023-11-02T19:06:06.127 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.127 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:06.128 DEBUG:boto:Port: 8001 2023-11-02T19:06:06.128 DEBUG:boto:Params: {} 2023-11-02T19:06:06.129 DEBUG:boto:Token: None 2023-11-02T19:06:06.129 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.130 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eS48FWEPkdmXWk3rsAh5zc1GQMU= 2023-11-02T19:06:06.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eS48FWEPkdmXWk3rsAh5zc1GQMU=', 'Content-Length': '0'} 2023-11-02T19:06:06.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f2b8ff08d41b95a-006543f31e-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.174 DEBUG:boto:path=/ibtljn-64/obj95 2023-11-02T19:06:06.175 DEBUG:boto:auth_path=/ibtljn-64/obj95 2023-11-02T19:06:06.176 DEBUG:boto:path=/ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.176 DEBUG:boto:auth_path=/ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.177 DEBUG:boto:Method: GET 2023-11-02T19:06:06.177 DEBUG:boto:Path: /ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.178 DEBUG:boto:Data: 2023-11-02T19:06:06.178 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.179 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:06.179 DEBUG:boto:Port: 8000 2023-11-02T19:06:06.180 DEBUG:boto:Params: {} 2023-11-02T19:06:06.180 DEBUG:boto:Token: None 2023-11-02T19:06:06.181 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.181 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eS48FWEPkdmXWk3rsAh5zc1GQMU= 2023-11-02T19:06:06.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eS48FWEPkdmXWk3rsAh5zc1GQMU=', 'Content-Length': '0'} 2023-11-02T19:06:06.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db026e140d0b7eea-006543f31e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.186 DEBUG:boto:path=/ibtljn-64/obj95 2023-11-02T19:06:06.186 DEBUG:boto:auth_path=/ibtljn-64/obj95 2023-11-02T19:06:06.187 DEBUG:boto:path=/ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.187 DEBUG:boto:auth_path=/ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.188 DEBUG:boto:Method: HEAD 2023-11-02T19:06:06.188 DEBUG:boto:Path: /ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.189 DEBUG:boto:Data: 2023-11-02T19:06:06.190 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.190 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:06.190 DEBUG:boto:Port: 8001 2023-11-02T19:06:06.191 DEBUG:boto:Params: {} 2023-11-02T19:06:06.192 DEBUG:boto:Token: None 2023-11-02T19:06:06.192 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.193 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CHUZKd5as7jKRj2+y/ZvGeMhNHo= 2023-11-02T19:06:06.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CHUZKd5as7jKRj2+y/ZvGeMhNHo=', 'Content-Length': '0'} 2023-11-02T19:06:06.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000784fa7d709eb4ccc-006543f31e-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.197 DEBUG:boto:path=/ibtljn-64/obj95 2023-11-02T19:06:06.197 DEBUG:boto:auth_path=/ibtljn-64/obj95 2023-11-02T19:06:06.198 DEBUG:boto:path=/ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.198 DEBUG:boto:auth_path=/ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.198 DEBUG:boto:Method: HEAD 2023-11-02T19:06:06.199 DEBUG:boto:Path: /ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.199 DEBUG:boto:Data: 2023-11-02T19:06:06.200 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.200 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:06.200 DEBUG:boto:Port: 8000 2023-11-02T19:06:06.201 DEBUG:boto:Params: {} 2023-11-02T19:06:06.202 DEBUG:boto:Token: None 2023-11-02T19:06:06.202 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.203 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CHUZKd5as7jKRj2+y/ZvGeMhNHo= 2023-11-02T19:06:06.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CHUZKd5as7jKRj2+y/ZvGeMhNHo=', 'Content-Length': '0'} 2023-11-02T19:06:06.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016f4ba46de1086b8-006543f31e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T19:06:06.208 DEBUG:boto:path=/ibtljn-64/obj95 2023-11-02T19:06:06.208 DEBUG:boto:auth_path=/ibtljn-64/obj95 2023-11-02T19:06:06.209 DEBUG:boto:path=/ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.209 DEBUG:boto:auth_path=/ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.209 DEBUG:boto:Method: GET 2023-11-02T19:06:06.210 DEBUG:boto:Path: /ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.210 DEBUG:boto:Data: 2023-11-02T19:06:06.211 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.211 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:06.212 DEBUG:boto:Port: 8001 2023-11-02T19:06:06.213 DEBUG:boto:Params: {} 2023-11-02T19:06:06.213 DEBUG:boto:Token: None 2023-11-02T19:06:06.214 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.214 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eS48FWEPkdmXWk3rsAh5zc1GQMU= 2023-11-02T19:06:06.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eS48FWEPkdmXWk3rsAh5zc1GQMU=', 'Content-Length': '0'} 2023-11-02T19:06:06.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c639e4856cafbae8-006543f31e-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.219 DEBUG:boto:path=/ibtljn-64/obj95 2023-11-02T19:06:06.219 DEBUG:boto:auth_path=/ibtljn-64/obj95 2023-11-02T19:06:06.220 DEBUG:boto:path=/ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.220 DEBUG:boto:auth_path=/ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.220 DEBUG:boto:Method: GET 2023-11-02T19:06:06.221 DEBUG:boto:Path: /ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.222 DEBUG:boto:Data: 2023-11-02T19:06:06.222 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:06.223 DEBUG:boto:Port: 8000 2023-11-02T19:06:06.224 DEBUG:boto:Params: {} 2023-11-02T19:06:06.224 DEBUG:boto:Token: None 2023-11-02T19:06:06.225 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj95?versionId=null 2023-11-02T19:06:06.225 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:eS48FWEPkdmXWk3rsAh5zc1GQMU= 2023-11-02T19:06:06.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:eS48FWEPkdmXWk3rsAh5zc1GQMU=', 'Content-Length': '0'} 2023-11-02T19:06:06.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b2d1c9950aee986-006543f31e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.230 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:06.230 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:06.231 DEBUG:boto:path=/ibtljn-64/?prefix=obj95 2023-11-02T19:06:06.231 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj95 2023-11-02T19:06:06.231 DEBUG:boto:Method: GET 2023-11-02T19:06:06.232 DEBUG:boto:Path: /ibtljn-64/?prefix=obj95 2023-11-02T19:06:06.232 DEBUG:boto:Data: 2023-11-02T19:06:06.233 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.233 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:06.234 DEBUG:boto:Port: 8001 2023-11-02T19:06:06.234 DEBUG:boto:Params: {} 2023-11-02T19:06:06.235 DEBUG:boto:Token: None 2023-11-02T19:06:06.236 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/ 2023-11-02T19:06:06.236 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gZVcryDIXBoYK6bygXqtnA1bCGI= 2023-11-02T19:06:06.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gZVcryDIXBoYK6bygXqtnA1bCGI=', 'Content-Length': '0'} 2023-11-02T19:06:06.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ba86c48ce1a1d20-006543f31e-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.242 DEBUG:boto:b'ibtljn-64obj951000falseobj952023-11-02T19:03:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:06.243 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:06.243 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:06.244 DEBUG:boto:path=/ibtljn-64/?prefix=obj95 2023-11-02T19:06:06.244 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj95 2023-11-02T19:06:06.245 DEBUG:boto:Method: GET 2023-11-02T19:06:06.245 DEBUG:boto:Path: /ibtljn-64/?prefix=obj95 2023-11-02T19:06:06.246 DEBUG:boto:Data: 2023-11-02T19:06:06.247 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.247 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:06.248 DEBUG:boto:Port: 8000 2023-11-02T19:06:06.248 DEBUG:boto:Params: {} 2023-11-02T19:06:06.249 DEBUG:boto:Token: None 2023-11-02T19:06:06.249 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/ 2023-11-02T19:06:06.250 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gZVcryDIXBoYK6bygXqtnA1bCGI= 2023-11-02T19:06:06.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gZVcryDIXBoYK6bygXqtnA1bCGI=', 'Content-Length': '0'} 2023-11-02T19:06:06.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020998a127e0a0595-006543f31e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.261 DEBUG:boto:b'ibtljn-64obj951000falseobj952023-11-02T19:03:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:06.262 DEBUG:boto:path=/ibtljn-64/obj95 2023-11-02T19:06:06.262 DEBUG:boto:auth_path=/ibtljn-64/obj95 2023-11-02T19:06:06.262 DEBUG:boto:Method: HEAD 2023-11-02T19:06:06.263 DEBUG:boto:Path: /ibtljn-64/obj95 2023-11-02T19:06:06.263 DEBUG:boto:Data: 2023-11-02T19:06:06.264 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.265 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:06.265 DEBUG:boto:Port: 8001 2023-11-02T19:06:06.265 DEBUG:boto:Params: {} 2023-11-02T19:06:06.266 DEBUG:boto:Token: None 2023-11-02T19:06:06.266 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj95 2023-11-02T19:06:06.267 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YAYU0HC6cxPFYZzS5gnrkRGm6Y4= 2023-11-02T19:06:06.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YAYU0HC6cxPFYZzS5gnrkRGm6Y4=', 'Content-Length': '0'} 2023-11-02T19:06:06.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ddc7217501d5e59-006543f31e-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.271 DEBUG:boto:path=/ibtljn-64/obj95 2023-11-02T19:06:06.271 DEBUG:boto:auth_path=/ibtljn-64/obj95 2023-11-02T19:06:06.272 DEBUG:boto:Method: HEAD 2023-11-02T19:06:06.272 DEBUG:boto:Path: /ibtljn-64/obj95 2023-11-02T19:06:06.273 DEBUG:boto:Data: 2023-11-02T19:06:06.273 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.273 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:06.274 DEBUG:boto:Port: 8000 2023-11-02T19:06:06.274 DEBUG:boto:Params: {} 2023-11-02T19:06:06.275 DEBUG:boto:Token: None 2023-11-02T19:06:06.275 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj95 2023-11-02T19:06:06.276 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YAYU0HC6cxPFYZzS5gnrkRGm6Y4= 2023-11-02T19:06:06.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YAYU0HC6cxPFYZzS5gnrkRGm6Y4=', 'Content-Length': '0'} 2023-11-02T19:06:06.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d9fcec5b7033266-006543f31e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-11-02T19:06:06.280 DEBUG:boto:path=/ibtljn-64/obj95 2023-11-02T19:06:06.280 DEBUG:boto:auth_path=/ibtljn-64/obj95 2023-11-02T19:06:06.281 DEBUG:boto:Method: GET 2023-11-02T19:06:06.281 DEBUG:boto:Path: /ibtljn-64/obj95 2023-11-02T19:06:06.282 DEBUG:boto:Data: 2023-11-02T19:06:06.282 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.283 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:06.283 DEBUG:boto:Port: 8001 2023-11-02T19:06:06.284 DEBUG:boto:Params: {} 2023-11-02T19:06:06.284 DEBUG:boto:Token: None 2023-11-02T19:06:06.285 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj95 2023-11-02T19:06:06.285 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XtB/ri7jWe5rplXY1ny5t+Tf8oE= 2023-11-02T19:06:06.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XtB/ri7jWe5rplXY1ny5t+Tf8oE=', 'Content-Length': '0'} 2023-11-02T19:06:06.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-64:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002af17355f7d18b8-006543f31e-20d2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.334 DEBUG:boto:path=/ibtljn-64/obj95 2023-11-02T19:06:06.342 DEBUG:boto:auth_path=/ibtljn-64/obj95 2023-11-02T19:06:06.343 DEBUG:boto:Method: GET 2023-11-02T19:06:06.343 DEBUG:boto:Path: /ibtljn-64/obj95 2023-11-02T19:06:06.344 DEBUG:boto:Data: 2023-11-02T19:06:06.344 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.345 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:06.345 DEBUG:boto:Port: 8000 2023-11-02T19:06:06.346 DEBUG:boto:Params: {} 2023-11-02T19:06:06.346 DEBUG:boto:Token: None 2023-11-02T19:06:06.347 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/obj95 2023-11-02T19:06:06.348 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XtB/ri7jWe5rplXY1ny5t+Tf8oE= 2023-11-02T19:06:06.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XtB/ri7jWe5rplXY1ny5t+Tf8oE=', 'Content-Length': '0'} 2023-11-02T19:06:06.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:03:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003385263a070d72e8-006543f31e-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.353 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:06.353 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:06.354 DEBUG:boto:path=/ibtljn-64/?prefix=obj95 2023-11-02T19:06:06.354 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj95 2023-11-02T19:06:06.355 DEBUG:boto:Method: GET 2023-11-02T19:06:06.355 DEBUG:boto:Path: /ibtljn-64/?prefix=obj95 2023-11-02T19:06:06.356 DEBUG:boto:Data: 2023-11-02T19:06:06.356 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.357 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:06:06.357 DEBUG:boto:Port: 8001 2023-11-02T19:06:06.357 DEBUG:boto:Params: {} 2023-11-02T19:06:06.358 DEBUG:boto:Token: None 2023-11-02T19:06:06.358 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/ 2023-11-02T19:06:06.359 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gZVcryDIXBoYK6bygXqtnA1bCGI= 2023-11-02T19:06:06.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gZVcryDIXBoYK6bygXqtnA1bCGI=', 'Content-Length': '0'} 2023-11-02T19:06:06.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071ecb5290f2215f3-006543f31e-20d2-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.365 DEBUG:boto:b'ibtljn-64obj951000falseobj952023-11-02T19:03:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:06.366 DEBUG:boto:path=/ibtljn-64/ 2023-11-02T19:06:06.367 DEBUG:boto:auth_path=/ibtljn-64/ 2023-11-02T19:06:06.367 DEBUG:boto:path=/ibtljn-64/?prefix=obj95 2023-11-02T19:06:06.367 DEBUG:boto:auth_path=/ibtljn-64/?prefix=obj95 2023-11-02T19:06:06.368 DEBUG:boto:Method: GET 2023-11-02T19:06:06.368 DEBUG:boto:Path: /ibtljn-64/?prefix=obj95 2023-11-02T19:06:06.369 DEBUG:boto:Data: 2023-11-02T19:06:06.369 DEBUG:boto:Headers: {} 2023-11-02T19:06:06.370 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:06.370 DEBUG:boto:Port: 8000 2023-11-02T19:06:06.371 DEBUG:boto:Params: {} 2023-11-02T19:06:06.371 DEBUG:boto:Token: None 2023-11-02T19:06:06.372 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:06:06 GMT /ibtljn-64/ 2023-11-02T19:06:06.372 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:gZVcryDIXBoYK6bygXqtnA1bCGI= 2023-11-02T19:06:06.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:06 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gZVcryDIXBoYK6bygXqtnA1bCGI=', 'Content-Length': '0'} 2023-11-02T19:06:06.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000168b446348410809-006543f31e-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:06:06 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:06.383 DEBUG:boto:b'ibtljn-64obj951000falseobj952023-11-02T19:03:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:06:06.384 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ibtljn-64 zones={a2, a1} 2023-11-02T19:06:06.384 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-11-02T19:06:06.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:06.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:06.386 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:06.428 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:06.429 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:06.524 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:06:06.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:06.525 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:06:06.525 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:06:06.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:06.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:06.526 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050545", 2023-11-02T19:06:06.526 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:03:44.981128Z" 2023-11-02T19:06:06.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:06.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:06.527 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:06:06.527 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:06:06.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:06.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:06.529 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-11-02T19:06:06.530 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:03:44.888935Z" 2023-11-02T19:06:06.530 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:06.530 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:06:06.531 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050545', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045375'} 2023-11-02T19:06:06.532 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:06:06.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:06.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:06.533 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:06.621 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:06.621 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:06.708 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:06.708 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:06.709 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:06.709 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:06.709 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:06.709 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:06:06.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:06.710 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:06.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:06.711 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:06.711 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:06.712 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:06.712 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:06:06.713 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:06.713 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:06:06.713 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:06.714 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:06:06.714 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:06.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:06.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:06.715 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:06.716 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:06.716 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:06.716 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050545", 2023-11-02T19:06:06.717 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:06.717 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:06:06.718 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:06.718 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.981128Z" 2023-11-02T19:06:06.718 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:06.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:06.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:06.720 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:06.720 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:06.720 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:06.721 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:06:06.721 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:06.721 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:06.722 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:06.722 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:06:06.723 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:06.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:06.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:06.724 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:06.724 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:06.725 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:06.725 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-11-02T19:06:06.726 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:06.726 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:06.726 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:06.727 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.888935Z" 2023-11-02T19:06:06.727 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:06.728 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:06.728 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:06.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:06.729 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:06.730 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:06:06.730 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:06:06.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:06.731 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:06Z" 2023-11-02T19:06:06.731 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:06.735 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050545", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:03:44.981128Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.888935Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:06:06Z" } 2023-11-02T19:06:06.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050545', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.981128Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.888935Z'}}] 2023-11-02T19:06:06.737 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050545', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045375'} 2023-11-02T19:06:06.737 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050545', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045375'} 2023-11-02T19:06:06.738 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:06:06.739 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-11-02T19:06:06.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:06.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:06.741 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:06.788 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:06.788 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:06.894 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:06:06.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:06.895 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000006569", 2023-11-02T19:06:06.895 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:05:57.264455Z" 2023-11-02T19:06:06.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:06.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:06.896 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000016293", 2023-11-02T19:06:06.896 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:05:57.266568Z" 2023-11-02T19:06:06.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:06.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:06.896 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000016159", 2023-11-02T19:06:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:05:57.268776Z" 2023-11-02T19:06:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000013235", 2023-11-02T19:06:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:05:57.270888Z" 2023-11-02T19:06:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:06.898 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:06:06.899 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000006569', 1: '00000000000000000000:00000000000000016293', 2: '00000000000000000000:00000000000000016159', 3: '00000000000000000000:00000000000000013235'} 2023-11-02T19:06:06.900 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T19:06:06.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:06.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:06.901 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:06.948 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:06.949 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:07.040 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:07.040 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:07.041 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:07.041 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:07.041 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:07.041 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T19:06:07.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:07.042 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:07.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:07.042 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:07.042 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:07.043 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:07.043 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000006428", 2023-11-02T19:06:07.043 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:07.043 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:07.044 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:07.044 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:05:48.124676Z" 2023-11-02T19:06:07.044 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:07.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:07.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:07.045 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:07.045 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:07.045 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:07.045 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000016011", 2023-11-02T19:06:07.046 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:07.046 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:07.046 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:07.046 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:05:47.845306Z" 2023-11-02T19:06:07.047 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:07.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:07.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:07.047 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:07.048 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:07.048 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:07.048 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000015736", 2023-11-02T19:06:07.048 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:07.048 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:07.049 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:07.049 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:05:47.649051Z" 2023-11-02T19:06:07.049 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:07.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:07.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:07.050 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:07.050 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:07.050 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:07.051 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000012953", 2023-11-02T19:06:07.051 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:07.051 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:07.051 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:07.051 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:05:48.093420Z" 2023-11-02T19:06:07.052 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:07.052 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:07.052 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:07.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:07.053 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:07.053 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T19:06:07.053 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T19:06:07.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:07.053 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:07Z" 2023-11-02T19:06:07.054 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:07.055 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006428", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:05:48.124676Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016011", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:05:47.845306Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015736", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:05:47.649051Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012953", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:05:48.093420Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T19:06:07Z" } 2023-11-02T19:06:07.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006428', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:05:48.124676Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016011', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:05:47.845306Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015736', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:05:47.649051Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012953', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:05:48.093420Z'}}] 2023-11-02T19:06:07.056 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006569', 1: '00000000000000000000:00000000000000016293', 2: '00000000000000000000:00000000000000016159', 3: '00000000000000000000:00000000000000013235'} 2023-11-02T19:06:07.056 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006428', 1: '00000000000000000000:00000000000000016011', 2: '00000000000000000000:00000000000000015736', 3: '00000000000000000000:00000000000000012953'} 2023-11-02T19:06:07.057 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000006569 target=00000000000000000000:00000000000000006428, shard=1 master=00000000000000000000:00000000000000016293 target=00000000000000000000:00000000000000016011, shard=2 master=00000000000000000000:00000000000000016159 target=00000000000000000000:00000000000000015736, shard=3 master=00000000000000000000:00000000000000013235 target=00000000000000000000:00000000000000012953 2023-11-02T19:06:12.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:12.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:12.060 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:12.108 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:12.200 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T19:06:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000006428", 2023-11-02T19:06:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:05:48.124676Z" 2023-11-02T19:06:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000016011", 2023-11-02T19:06:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:05:47.845306Z" 2023-11-02T19:06:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000015736", 2023-11-02T19:06:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:05:47.649051Z" 2023-11-02T19:06:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000012953", 2023-11-02T19:06:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:05:48.093420Z" 2023-11-02T19:06:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T19:06:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T19:06:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:12Z" 2023-11-02T19:06:12.213 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:12.214 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006428", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:05:48.124676Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016011", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:05:47.845306Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015736", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:05:47.649051Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012953", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:05:48.093420Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T19:06:12Z" } 2023-11-02T19:06:12.215 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006428', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:05:48.124676Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016011', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:05:47.845306Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015736', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:05:47.649051Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012953', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:05:48.093420Z'}}] 2023-11-02T19:06:12.215 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006569', 1: '00000000000000000000:00000000000000016293', 2: '00000000000000000000:00000000000000016159', 3: '00000000000000000000:00000000000000013235'} 2023-11-02T19:06:12.215 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006428', 1: '00000000000000000000:00000000000000016011', 2: '00000000000000000000:00000000000000015736', 3: '00000000000000000000:00000000000000012953'} 2023-11-02T19:06:12.216 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000006569 target=00000000000000000000:00000000000000006428, shard=1 master=00000000000000000000:00000000000000016293 target=00000000000000000000:00000000000000016011, shard=2 master=00000000000000000000:00000000000000016159 target=00000000000000000000:00000000000000015736, shard=3 master=00000000000000000000:00000000000000013235 target=00000000000000000000:00000000000000012953 2023-11-02T19:06:17.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:17.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:17.219 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:17.266 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:17.267 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:17.358 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:17.359 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:17.359 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:17.359 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:17.359 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:17.360 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T19:06:17.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.360 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:17.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:17.360 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:17.361 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:17.361 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:17.361 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000006569", 2023-11-02T19:06:17.361 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:17.362 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:17.362 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:17.362 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:05:57.264455Z" 2023-11-02T19:06:17.362 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:17.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:17.363 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:17.363 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:17.363 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:17.363 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000016293", 2023-11-02T19:06:17.364 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:17.364 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:17.364 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:17.364 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:05:57.266568Z" 2023-11-02T19:06:17.364 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:17.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:17.365 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:17.365 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:17.365 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:17.366 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000016159", 2023-11-02T19:06:17.366 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:17.366 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:17.366 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:17.367 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:05:57.268776Z" 2023-11-02T19:06:17.367 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:17.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:17.367 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:17.368 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:17.368 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:06:17.368 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000013235", 2023-11-02T19:06:17.368 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:17.368 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:17.369 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:17.369 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:05:57.270888Z" 2023-11-02T19:06:17.369 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:17.369 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:17.370 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:17.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.370 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:17.370 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T19:06:17.370 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T19:06:17.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.371 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:17Z" 2023-11-02T19:06:17.371 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:17.372 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006569", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:05:57.264455Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016293", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:05:57.266568Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016159", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:05:57.268776Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:05:57.270888Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T19:06:17Z" } 2023-11-02T19:06:17.373 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006569', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:05:57.264455Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016293', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:05:57.266568Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016159', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:05:57.268776Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:05:57.270888Z'}}] 2023-11-02T19:06:17.373 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006569', 1: '00000000000000000000:00000000000000016293', 2: '00000000000000000000:00000000000000016159', 3: '00000000000000000000:00000000000000013235'} 2023-11-02T19:06:17.373 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006569', 1: '00000000000000000000:00000000000000016293', 2: '00000000000000000000:00000000000000016159', 3: '00000000000000000000:00000000000000013235'} 2023-11-02T19:06:17.374 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T19:06:17.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... ok 2023-11-02T19:06:17.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:17.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:17.377 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:17.422 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:17.422 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:17.485 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:06:17.514 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:06:17.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:17.514 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:06:17.515 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:06:17.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:17.515 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:06:17.516 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:06:17.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:17.516 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:06:17.516 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:03:44.645395Z" 2023-11-02T19:06:17.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:17.517 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:06:17.517 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:06:17.518 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:17.518 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:06:17.519 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:17.519 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:06:17.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:17.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:17.521 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:17.566 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:17.566 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:17.624 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:17.616+0000 7f9fb731af40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:17.626 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:17.620+0000 7f9fb731af40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:17.656 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:17.656 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:17.656 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:17.656 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:17.657 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:17.657 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:06:17.657 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:17.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.658 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:17.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:17.658 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:17.658 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:17.659 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:17.659 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:06:17.659 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:17.659 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:06:17.659 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:17.660 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:06:17.660 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:17.660 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:17.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:17.661 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:17.661 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:17.661 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:17.661 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:06:17.662 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:17.662 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:17.662 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:17.662 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:06:17.663 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:17.663 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:17.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:17.663 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:17.664 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:17.664 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:17.664 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:06:17.664 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:17.664 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:17.665 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:17.665 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:06:17.665 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:17.665 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:17.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:17.666 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:17.666 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:17.666 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:17.666 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:06:17.667 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:17.667 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:17.667 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:17.667 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:06:17.667 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:17.668 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:17.668 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:17.668 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:17.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.668 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:17.669 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:06:17.669 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:06:17.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:17.669 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:17Z" 2023-11-02T19:06:17.670 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:17.671 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:06:17Z" } 2023-11-02T19:06:17.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:06:17.672 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:17.672 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:17.672 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:06:17.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:17.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:17.674 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:17.720 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:17.721 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:17.804 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[]} 2023-11-02T19:06:17.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:17.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:17.806 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:17.894 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:17.895 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:17.987 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-11-02T19:06:17.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:17.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:17.989 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:18.034 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:18.035 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:18.134 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-11-02T19:06:18.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:18.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:18.135 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:18.178 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:18.179 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:18.272 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2023-11-02T19:06:18.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:18.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:18.273 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:18.443 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-11-02T19:06:18.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:18.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:18.445 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:18.493 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:18.494 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:18.595 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-11-02T19:06:18.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:18.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:18.597 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:18.647 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:18.647 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:18.741 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-11-02T19:06:18.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:18.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:18.742 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:18.790 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:18.791 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:18.887 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:06:18.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:18.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:18.888 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:18.934 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:18.934 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:19.029 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2023-11-02T19:06:19.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:19.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:19.031 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:19.075 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:19.075 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:19.188 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:06:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 13, 2023-11-02T19:06:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T19:06:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T19:06:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T19:06:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:06:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T19:06:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T19:06:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T19:06:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:06:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T19:06:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:06:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:06:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:06:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:06:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:06:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:06:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:06:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T19:06:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:06:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T19:06:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:06:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:06:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:06:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:06:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:06:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:06:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:06:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:06:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:06:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:06:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:06:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:06:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T19:06:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:06:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:06:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:06:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:06:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:06:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:06:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:06:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:06:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:06:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:06:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:06:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:06:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:06:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:06:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:06:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:06:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:06:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:06:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:06:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:06:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:06:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [ 2023-11-02T19:06:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-group", 2023-11-02T19:06:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "data_flow": { 2023-11-02T19:06:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "symmetrical": [ 2023-11-02T19:06:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-flow1", 2023-11-02T19:06:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:06:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:06:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:06:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "directional": [ 2023-11-02T19:06:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "source_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:06:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "dest_zone": "39a01552-6d59-4ce7-b7e5-915ca69b979a" 2023-11-02T19:06:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "pipes": [ 2023-11-02T19:06:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-pipe", 2023-11-02T19:06:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:06:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "*", 2023-11-02T19:06:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:06:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:06:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:06:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": { 2023-11-02T19:06:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "*", 2023-11-02T19:06:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:06:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:06:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:06:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "params": { 2023-11-02T19:06:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:06:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "filter": { 2023-11-02T19:06:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [] 2023-11-02T19:06:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": {}, 2023-11-02T19:06:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2023-11-02T19:06:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "mode": "system", 2023-11-02T19:06:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "" 2023-11-02T19:06:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "enabled" 2023-11-02T19:06:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:06:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:06:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T19:06:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T19:06:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:06:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T19:06:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:06:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:06:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:06:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:06:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:06:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:06:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:06:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T19:06:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:06:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:06:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:06:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:06:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:06:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:06:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:06:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:06:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:06:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:06:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:06:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:06:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:06:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:06:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T19:06:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:06:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T19:06:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:06:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:06:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:06:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:06:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:06:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:06:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:06:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:06:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:06:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:06:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:06:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:06:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:06:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:06:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:06:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:06:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:06:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:06:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:06:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T19:06:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:06:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:06:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:06:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T19:06:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:06:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T19:06:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:06:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T19:06:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:06:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T19:06:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:06:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T19:06:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T19:06:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:06:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T19:06:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T19:06:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:06:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:06:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:06:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:06:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:06:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T19:06:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:06:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:06:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:06:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:06:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:06:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T19:06:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:06:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:06:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:06:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:06:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:06:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T19:06:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:06:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:06:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:06:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:06:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:06:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T19:06:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:06:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:06:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:06:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:06:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:06:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:06:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T19:06:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:19.247 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:19.248 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-65 2023-11-02T19:06:19.248 DEBUG:boto:path=/ibtljn-65/ 2023-11-02T19:06:19.249 DEBUG:boto:auth_path=/ibtljn-65/ 2023-11-02T19:06:19.249 DEBUG:boto:Method: PUT 2023-11-02T19:06:19.250 DEBUG:boto:Path: /ibtljn-65/ 2023-11-02T19:06:19.250 DEBUG:boto:Data: 2023-11-02T19:06:19.250 DEBUG:boto:Headers: {} 2023-11-02T19:06:19.251 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:06:19.251 DEBUG:boto:Port: 8000 2023-11-02T19:06:19.252 DEBUG:boto:Params: {} 2023-11-02T19:06:19.252 DEBUG:boto:Token: None 2023-11-02T19:06:19.253 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 19:06:19 GMT /ibtljn-65/ 2023-11-02T19:06:19.253 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:C43ln/Hf7OVHI+q5NgQiIbk/1ts= 2023-11-02T19:06:19.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:06:19 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C43ln/Hf7OVHI+q5NgQiIbk/1ts=', 'Content-Length': '0'} 2023-11-02T19:06:19.471 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009f6d2fe8751641c7-006543f32b-217a-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:06:19.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:19.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:19.473 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:19.534 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:19.534 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:19.667 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-11-02T19:06:19.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:19.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:19.669 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:19.726 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:19.727 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:19.825 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-11-02T19:06:19.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:19.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:19.827 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:19.877 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:19.878 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:19.953 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2023-11-02T19:06:19.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:19.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:19.954 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:19.997 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:19.998 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:20.077 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-11-02T19:06:20.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:20.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:20.079 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:20.173 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:20.174 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:20.278 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-11-02T19:06:20.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:20.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:20.279 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:20.331 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:20.331 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:20.436 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-11-02T19:06:20.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:20.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:20.437 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:20.485 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:20.485 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:20.591 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-11-02T19:06:20.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:20.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:06:20.592 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:06:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:20.729 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2023-11-02T19:06:20.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:20.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:20.731 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:20.773 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:20.774 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:20.833 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:06:20.864 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:06:20.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:20.864 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:06:20.865 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:06:20.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:20.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:20.865 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:06:20.865 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:06:20.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:20.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:20.866 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951980.556340_671.1", 2023-11-02T19:06:20.866 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:06:20.556340Z" 2023-11-02T19:06:20.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:20.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:20.867 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:06:20.867 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:06:20.867 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:20.868 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:06:20.869 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:20.869 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:06:20.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:20.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:20.870 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:20.915 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:20.915 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:20.973 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:20.968+0000 7fb9df744f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:20.975 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:20.968+0000 7fb9df744f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:21.003 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:21.004 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:21.004 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:21.004 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:21.004 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:21.005 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:06:21.005 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:21.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:21.005 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:21.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:21.006 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:21.006 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:21.006 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:21.006 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:06:21.007 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:21.007 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:06:21.007 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:21.007 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:06:21.007 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:21.008 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:21.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:21.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:21.008 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:21.009 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:21.009 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:21.009 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:06:21.009 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:21.009 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:21.010 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:21.010 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:06:21.010 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:21.010 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:21.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:21.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:21.011 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:21.011 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:21.011 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:21.012 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:06:21.012 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:21.012 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:21.012 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:21.012 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:06:21.013 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:21.013 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:21.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:21.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:21.013 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:21.014 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:21.014 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:21.014 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:06:21.014 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:21.015 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:21.015 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:21.015 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:06:21.015 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:21.015 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:21.016 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:21.016 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:21.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:21.016 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:21.017 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:06:21.017 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:06:21.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:21.017 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:20Z" 2023-11-02T19:06:21.017 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:21.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:06:20Z" } 2023-11-02T19:06:21.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:06:21.019 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:21.020 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:21.020 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:06:26.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:26.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:26.023 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:26.072 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:26.073 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:26.134 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:26.128+0000 7fe57810af40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:26.135 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:26.128+0000 7fe57810af40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:26.164 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:26.164 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:26.164 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:26.164 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:26.165 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:26.165 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:06:26.165 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:26.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:26.166 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:26.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:26.166 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:26.166 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:26.167 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:26.167 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:06:26.167 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:26.167 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:06:26.167 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:26.168 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:06:26.168 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:26.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:26.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:26.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:26.169 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:26.169 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:26.169 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:26.169 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:06:26.170 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:26.170 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:26.170 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:26.170 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:06:26.171 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:26.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:26.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:26.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:26.171 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:26.172 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:26.172 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:26.172 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:06:26.173 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:26.173 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:26.173 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:26.173 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:06:26.173 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:26.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:26.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:26.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:26.174 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:26.175 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:26.175 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:26.175 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:06:26.175 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:26.175 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:26.176 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:26.176 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:06:26.176 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:26.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:26.177 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:26.177 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:26.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:26.177 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:26.178 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:06:26.178 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:06:26.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:26.178 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:26Z" 2023-11-02T19:06:26.179 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:26.180 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:06:26Z" } 2023-11-02T19:06:26.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:06:26.181 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:26.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:26.182 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:06:31.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:31.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:31.184 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:31.229 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:31.230 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:31.284 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:31.276+0000 7f133cb6bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:31.285 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:31.280+0000 7f133cb6bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:31.313 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:31.314 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:31.314 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:31.314 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:31.314 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:31.314 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:06:31.315 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:31.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:31.315 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:31.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:31.316 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:31.316 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:31.316 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:31.316 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:06:31.316 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:31.317 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:06:31.317 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:31.317 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:06:31.317 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:31.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:31.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:31.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:31.318 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:31.318 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:31.319 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:31.319 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:06:31.319 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:31.319 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:31.319 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:31.320 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:06:31.320 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:31.320 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:31.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:31.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:31.321 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:31.321 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:31.321 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:31.321 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:06:31.321 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:31.322 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:31.322 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:31.322 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:06:31.322 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:31.323 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:31.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:31.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:31.323 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:31.323 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:31.324 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:31.325 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:06:31.325 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:31.326 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:31.326 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:31.326 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:06:31.326 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:31.326 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:31.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:31.327 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:31.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:31.327 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:31.328 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:06:31.328 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:06:31.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:31.328 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:31Z" 2023-11-02T19:06:31.328 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:31.329 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:06:31Z" } 2023-11-02T19:06:31.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:06:31.330 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:31.331 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:31.331 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:06:36.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:36.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:36.333 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:36.379 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:36.380 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:36.428+0000 7ff35dbcff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:36.432+0000 7ff35dbcff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:36.466 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:06:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:06:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:06:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:06:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:36.471 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:36.471 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:36.471 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:36.471 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:06:36.472 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:36.472 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:36.472 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:36.472 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:06:36.472 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:36.473 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:36.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:36.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:36.473 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:36.474 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:36.474 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:36.474 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:06:36.474 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:36.474 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:36.475 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:36.475 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:06:36.475 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:36.475 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:36.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:36.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:36.476 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:36.476 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:36.476 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:36.477 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:06:36.477 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:36.477 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:36.477 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:36.477 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:06:36.478 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:36.478 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:36.478 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:36.478 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:36.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:36.479 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:36.479 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:06:36.479 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:06:36.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:36.480 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:36Z" 2023-11-02T19:06:36.480 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:36.481 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:06:36Z" } 2023-11-02T19:06:36.481 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:06:36.482 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:36.482 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:36.483 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:06:41.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:41.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:41.485 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:41.533 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:41.534 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:41.596 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:41.588+0000 7f96ec689f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:41.597 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:41.592+0000 7f96ec689f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:41.626 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:06:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:06:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:06:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:06:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:06:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:06:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:06:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:06:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:06:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:06:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:06:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:06:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:41Z" 2023-11-02T19:06:41.641 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:41.642 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:06:41Z" } 2023-11-02T19:06:41.643 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:06:41.644 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:41.644 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:41.644 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:06:46.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:46.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:46.647 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:46.696 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:46.696 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:46.745 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:46.740+0000 7fd4a48fff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:46.746 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:46.740+0000 7fd4a48fff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:46.771 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:46.772 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:46.772 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:46.772 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:46.772 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:46.773 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:06:46.773 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:46.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:46.773 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:46.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:46.774 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:46.774 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:46.774 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:46.774 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:06:46.775 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:46.775 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:06:46.775 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:46.775 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:06:46.775 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:46.776 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:46.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:46.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:46.776 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:46.776 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:46.777 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:46.777 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:06:46.777 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:46.777 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:46.778 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:46.778 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:06:46.778 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:46.778 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:46.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:46.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:46.779 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:46.779 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:46.779 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:46.780 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:06:46.780 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:46.780 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:46.780 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:46.780 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:06:46.781 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:46.781 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:46.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:46.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:46.782 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:46.782 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:46.782 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:46.782 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:06:46.782 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:46.783 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:46.783 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:46.783 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:06:46.783 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:46.784 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:46.784 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:46.784 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:46.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:46.784 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:46.785 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:06:46.785 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:06:46.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:46.785 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:46Z" 2023-11-02T19:06:46.785 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:46.786 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:06:46Z" } 2023-11-02T19:06:46.787 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:06:46.787 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:46.788 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:46.788 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:06:51.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:51.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:51.791 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:51.836 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:51.836 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:51.899 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:51.892+0000 7f96008ebf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:51.901 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:51.896+0000 7f96008ebf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:51.929 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:51.930 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:51.930 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:51.930 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:51.931 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:51.931 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:06:51.931 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:51.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:51.931 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:51.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:51.932 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:51.932 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:51.932 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:51.933 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:06:51.933 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:51.933 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:06:51.933 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:51.933 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:06:51.934 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:51.934 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:51.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:51.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:51.935 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:51.935 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:51.935 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:51.935 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:06:51.935 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:51.936 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:51.936 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:51.936 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:06:51.936 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:51.936 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:51.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:51.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:51.937 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:51.937 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:51.938 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:51.938 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:06:51.938 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:51.938 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:51.938 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:51.939 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:06:51.939 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:51.939 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:51.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:51.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:51.940 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:51.940 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:51.940 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:51.940 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:06:51.941 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:51.941 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:51.941 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:51.941 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:06:51.941 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:51.942 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:51.942 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:51.942 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:51.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:51.942 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:51.943 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:06:51.943 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:06:51.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:51.943 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:51Z" 2023-11-02T19:06:51.943 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:51.945 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:06:51Z" } 2023-11-02T19:06:51.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:06:51.946 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:51.946 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:51.947 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:06:56.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:56.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:06:56.949 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:06:56.996 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:06:56.996 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:06:57.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:57.048+0000 7f10267ddf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:57.058 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:06:57.052+0000 7f10267ddf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:06:57.086 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:06:57.086 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:06:57.087 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:06:57.087 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:06:57.087 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:06:57.087 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:06:57.087 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:57.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:57.088 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:06:57.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:57.088 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:06:57.088 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:57.089 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:57.089 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:06:57.089 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:57.089 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:06:57.090 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:57.090 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:06:57.090 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:57.090 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:57.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:57.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:57.091 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:06:57.091 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:57.091 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:57.092 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:06:57.092 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:57.092 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:57.092 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:57.092 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:06:57.093 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:57.093 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:57.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:57.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:57.093 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:06:57.094 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:57.094 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:57.094 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:06:57.094 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:57.094 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:57.095 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:57.095 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:06:57.095 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:57.095 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:57.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:57.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:06:57.096 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:06:57.096 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:06:57.096 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:06:57.096 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:06:57.097 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:06:57.097 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:06:57.097 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:06:57.097 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:06:57.097 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:06:57.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:57.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:06:57.098 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:06:57.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:57.099 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:06:57.099 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:06:57.099 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:06:57.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:06:57.100 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:06:57Z" 2023-11-02T19:06:57.100 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:06:57.101 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:06:57Z" } 2023-11-02T19:06:57.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:06:57.102 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:57.102 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:06:57.103 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:07:02.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:02.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:02.105 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:07:02.152 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:07:02.152 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:07:02.210 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:02.204+0000 7f977288ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:02.212 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:02.204+0000 7f977288ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:02.239 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:07:02.240 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:07:02.240 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:07:02.240 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:07:02.240 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:07:02.241 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:07:02.241 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:02.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:02.241 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:07:02.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:02.242 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:07:02.242 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:02.242 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:02.243 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:07:02.243 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:02.243 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:07:02.243 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:02.243 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:07:02.244 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:02.244 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:02.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:02.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:02.244 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:07:02.245 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:02.245 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:02.245 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:07:02.245 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:02.245 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:02.246 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:02.246 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:07:02.246 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:02.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:02.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:02.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:02.247 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:07:02.247 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:02.247 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:02.248 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:07:02.248 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:02.248 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:02.248 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:02.248 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:07:02.249 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:02.249 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:02.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:02.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:02.249 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:07:02.250 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:02.250 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:02.250 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:07:02.250 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:02.251 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:02.251 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:02.251 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:07:02.251 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:02.251 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:02.252 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:02.252 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:07:02.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:02.252 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:07:02.252 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:07:02.253 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:07:02.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:02.253 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:07:02Z" 2023-11-02T19:07:02.253 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:07:02.254 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:07:02Z" } 2023-11-02T19:07:02.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:07:02.255 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:02.256 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:02.256 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:07:07.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:07.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:07.259 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:07:07.306 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:07:07.306 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:07:07.366 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:07.360+0000 7f3ae5ef4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:07.368 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:07.360+0000 7f3ae5ef4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:07.398 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:07:07.399 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:07:07.399 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:07:07.399 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:07:07.400 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:07:07.400 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:07:07.400 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:07.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:07.400 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:07:07.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:07.401 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:07:07.401 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:07.401 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:07.401 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:07:07.402 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:07.402 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:07:07.402 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:07.402 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:07:07.403 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:07.403 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:07.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:07.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:07.404 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:07:07.404 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:07.404 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:07.404 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:07:07.404 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:07.405 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:07.405 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:07.405 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:07:07.405 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:07.405 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:07.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:07.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:07.406 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:07:07.406 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:07.407 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:07.407 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:07:07.407 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:07.407 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:07.407 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:07.408 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:07:07.408 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:07.408 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:07.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:07.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:07.409 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:07:07.409 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:07.409 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:07.409 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:07:07.409 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:07.410 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:07.410 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:07.410 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:07:07.410 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:07.411 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:07.411 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:07.411 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:07:07.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:07.411 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:07:07.412 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:07:07.412 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:07:07.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:07.412 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:07:07Z" 2023-11-02T19:07:07.412 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:07:07.413 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:07:07Z" } 2023-11-02T19:07:07.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:07:07.415 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:07.415 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:07.415 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:07:12.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:12.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:12.418 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:07:12.464 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:07:12.465 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:07:12.525 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:12.516+0000 7f3fd7279f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:12.527 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:12.520+0000 7f3fd7279f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:12.556 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:07:12.557 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:07:12.557 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:07:12.557 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:07:12.557 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:07:12.558 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:07:12.558 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:12.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:12.558 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:07:12.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:12.559 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:07:12.559 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:12.559 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:12.559 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:07:12.560 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:12.560 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:07:12.560 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:12.560 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:07:12.561 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:12.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:12.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:12.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:12.561 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:07:12.562 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:12.562 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:12.562 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:07:12.562 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:12.563 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:12.563 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:12.563 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:07:12.563 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:12.563 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:12.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:12.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:12.564 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:07:12.564 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:12.564 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:12.565 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:07:12.565 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:12.565 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:12.565 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:12.566 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:07:12.566 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:12.566 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:12.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:12.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:12.567 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:07:12.567 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:12.567 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:12.567 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:07:12.568 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:12.568 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:12.568 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:12.568 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:07:12.568 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:12.569 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:12.569 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:12.569 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:07:12.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:12.569 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:07:12.570 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:07:12.570 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:07:12.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:12.570 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:07:12Z" 2023-11-02T19:07:12.571 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:07:12.572 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:07:12Z" } 2023-11-02T19:07:12.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:07:12.573 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:12.573 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:12.573 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:07:17.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:17.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:17.576 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:07:17.625 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:07:17.625 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:07:17.692 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:17.684+0000 7f8a1a6b7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:17.694 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:17.688+0000 7f8a1a6b7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:17.713 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:07:17.714 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:07:17.714 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:07:17.714 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:07:17.714 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:07:17.715 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:07:17.715 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:17.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:17.715 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:07:17.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:17.716 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:07:17.716 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:17.716 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:07:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:07:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:17.718 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:07:17.718 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:17.718 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:17.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:17.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:17.719 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:07:17.719 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:17.720 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:17.720 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:07:17.720 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:17.720 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:17.720 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:17.721 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:07:17.721 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:17.722 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:17.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:17.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:17.723 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:07:17.723 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:17.724 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:17.724 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:07:17.724 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:17.725 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:17.725 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:17.725 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:07:17.725 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:17.726 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:17.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:17.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:17.727 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:07:17.727 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:17.727 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:17.727 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:07:17.727 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:17.728 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:17.728 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:17.728 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:07:17.728 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:17.728 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:17.729 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:17.729 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:07:17.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:17.729 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:07:17.730 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:07:17.730 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:07:17.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:17.730 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:07:17Z" 2023-11-02T19:07:17.731 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:07:17.732 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:07:17Z" } 2023-11-02T19:07:17.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:07:17.733 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:17.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:17.734 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:07:22.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:22.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:22.737 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:07:22.783 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:07:22.784 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:07:22.842 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:22.836+0000 7f9aa82aaf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:22.843 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:22.836+0000 7f9aa82aaf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:22.874 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:07:22.874 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:07:22.875 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:07:22.875 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:07:22.875 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:07:22.876 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:07:22.877 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:22.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:22.877 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:07:22.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:22.878 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:07:22.878 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:22.878 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:22.878 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:07:22.879 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:22.879 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:07:22.879 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:22.879 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:07:22.879 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:22.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:22.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:22.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:22.880 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:07:22.881 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:22.881 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:22.881 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:07:22.881 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:22.881 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:07:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:22.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:22.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:22.883 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:07:22.883 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:22.884 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:22.884 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:07:22.884 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:22.884 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:22.885 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:22.885 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:07:22.885 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:22.885 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:22.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:22.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:22.886 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:07:22.886 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:22.887 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:22.887 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:07:22.887 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:22.887 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:22.887 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:22.888 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:07:22.888 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:22.888 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:22.888 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:22.889 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:07:22.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:22.889 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:07:22.889 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:07:22.890 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:07:22.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:22.890 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:07:22Z" 2023-11-02T19:07:22.890 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:07:22.892 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:07:22Z" } 2023-11-02T19:07:22.892 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:07:22.893 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:22.893 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:22.894 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:07:27.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:27.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:27.896 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:07:27.945 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:07:27.945 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:07:28.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:28.000+0000 7ff9ad36ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:28.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:28.000+0000 7ff9ad36ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:28.037 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:07:28.038 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:07:28.038 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:07:28.038 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:07:28.038 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:07:28.039 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:07:28.039 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:28.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:28.039 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:07:28.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:28.040 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:07:28.040 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:28.040 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:28.041 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:07:28.041 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:28.041 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:07:28.041 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:28.042 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:07:28.042 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:28.042 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:28.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:28.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:28.043 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:07:28.043 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:28.043 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:28.044 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:07:28.044 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:28.044 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:28.044 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:28.044 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:07:28.045 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:28.045 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:28.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:28.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:28.045 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:07:28.046 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:28.046 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:28.046 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:07:28.046 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:28.046 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:28.047 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:28.047 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:07:28.047 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:28.047 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:28.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:28.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:28.048 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:07:28.048 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:28.048 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:28.049 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:07:28.049 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:28.049 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:28.049 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:28.050 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:07:28.050 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:28.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:28.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:28.050 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:07:28.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:28.051 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:07:28.051 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:07:28.051 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:07:28.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:28.052 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:07:28Z" 2023-11-02T19:07:28.052 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:07:28.053 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:07:28Z" } 2023-11-02T19:07:28.054 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:07:28.054 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:28.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:28.055 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:07:33.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:33.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:33.058 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:07:33.104 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:07:33.104 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:07:33.163 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:33.156+0000 7f74ef738f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:33.164 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:33.156+0000 7f74ef738f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:33.192 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:07:33.193 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:07:33.193 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:07:33.193 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:07:33.194 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:07:33.194 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:07:33.194 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:33.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:33.194 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:07:33.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:33.195 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:07:33.195 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:33.195 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:33.196 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:07:33.196 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:33.196 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:07:33.196 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:33.196 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:07:33.197 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:33.197 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:33.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:33.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:33.197 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:07:33.198 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:33.198 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:33.198 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:07:33.198 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:33.199 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:33.199 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:33.199 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:07:33.199 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:33.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:33.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:33.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:33.200 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:07:33.200 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:33.201 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:33.201 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:07:33.201 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:33.201 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:33.201 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:33.202 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:07:33.202 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:33.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:33.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:33.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:33.203 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:07:33.203 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:33.203 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:33.203 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:07:33.204 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:33.204 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:33.204 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:33.204 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:07:33.204 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:33.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:33.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:33.205 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:07:33.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:33.205 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:07:33.206 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:07:33.206 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:07:33.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:33.206 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:07:33Z" 2023-11-02T19:07:33.207 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:07:33.208 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:07:33Z" } 2023-11-02T19:07:33.208 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:07:33.209 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:33.209 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:33.210 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:07:38.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:38.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:38.213 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:07:38.262 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:07:38.262 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:07:38.320 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:38.312+0000 7fcd8cf76f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:38.321 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:38.316+0000 7fcd8cf76f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:38.349 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:07:38.350 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:07:38.350 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:07:38.350 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:07:38.350 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:07:38.351 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:07:38.351 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:38.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:38.351 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:07:38.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:38.352 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:07:38.352 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:38.352 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:38.352 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:07:38.352 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:38.353 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:07:38.353 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:38.353 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:07:38.353 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:38.353 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:38.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:38.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:38.354 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:07:38.354 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:38.354 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:38.355 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:07:38.355 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:38.355 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:38.355 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:38.356 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:07:38.356 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:38.356 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:38.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:38.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:38.357 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:07:38.357 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:38.357 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:38.357 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:07:38.358 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:38.358 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:38.358 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:38.358 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:07:38.358 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:38.359 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:38.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:38.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:38.359 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:07:38.360 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:38.360 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:38.360 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:07:38.360 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:38.360 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:38.361 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:38.361 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:07:38.361 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:38.361 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:38.362 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:38.362 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:07:38.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:38.362 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:07:38.362 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:07:38.363 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:07:38.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:38.363 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:07:38Z" 2023-11-02T19:07:38.363 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:07:38.364 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:07:38Z" } 2023-11-02T19:07:38.365 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:07:38.366 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:38.366 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:38.366 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:07:43.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:43.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:43.369 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:07:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:07:43.416 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:07:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:43.463+0000 7f26682b4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:43.472 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:43.463+0000 7f26682b4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:43.499 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:07:43.500 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:07:43.500 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:07:43.500 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:07:43.500 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:07:43.501 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:07:43.501 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:43.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:43.501 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:07:43.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:43.502 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:07:43.502 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:43.502 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:43.503 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:07:43.503 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:43.503 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:07:43.503 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:43.503 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:07:43.504 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:43.504 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:43.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:43.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:43.505 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:07:43.505 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:43.505 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:43.505 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:07:43.505 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:43.506 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:43.506 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:43.506 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:07:43.506 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:43.507 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:43.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:43.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:43.507 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:07:43.507 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:43.508 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:43.508 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:07:43.508 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:43.508 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:43.508 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:43.509 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:07:43.509 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:43.509 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:43.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:43.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:43.510 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:07:43.510 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:43.510 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:43.510 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:07:43.511 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:43.511 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:43.511 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:43.511 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:07:43.512 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:43.512 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:43.512 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:43.512 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:07:43.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:43.513 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:07:43.513 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:07:43.513 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:07:43.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:43.513 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:07:43Z" 2023-11-02T19:07:43.514 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:07:43.515 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:07:43Z" } 2023-11-02T19:07:43.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:07:43.516 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:43.516 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:43.517 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:07:48.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:48.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:48.519 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:07:48.569 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:07:48.569 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:07:48.630 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:48.623+0000 7f793974af40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:48.632 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:48.623+0000 7f793974af40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:48.662 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:07:48.663 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:07:48.663 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:07:48.663 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:07:48.663 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:07:48.663 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:07:48.664 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:48.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:48.664 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:07:48.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:48.665 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:07:48.665 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:48.665 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:48.665 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:07:48.665 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:48.666 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:07:48.666 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:48.666 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:07:48.667 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:48.667 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:48.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:48.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:48.667 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:07:48.668 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:48.668 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:48.668 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:07:48.668 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:48.669 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:48.669 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:48.669 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:07:48.669 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:48.670 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:48.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:48.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:48.670 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:07:48.670 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:48.671 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:48.671 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:07:48.671 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:48.671 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:48.672 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:48.672 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:07:48.672 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:48.672 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:48.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:48.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:48.673 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:07:48.673 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:48.673 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:48.673 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:07:48.674 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:48.674 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:48.674 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:48.674 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:07:48.675 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:48.675 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:48.675 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:48.675 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:07:48.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:48.676 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:07:48.676 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:07:48.676 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:07:48.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:48.677 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:07:48Z" 2023-11-02T19:07:48.677 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:07:48.678 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:07:48Z" } 2023-11-02T19:07:48.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:07:48.679 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:48.680 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:48.680 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:07:53.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:53.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:53.683 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:07:53.735 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:07:53.736 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:07:53.793 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:53.787+0000 7fa5e8ff0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:53.794 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:53.787+0000 7fa5e8ff0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:53.821 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:07:53.821 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:07:53.821 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:07:53.822 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:07:53.822 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:07:53.822 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:07:53.823 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:53.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:53.823 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:07:53.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:53.824 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:07:53.824 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:53.824 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:53.824 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:07:53.825 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:53.825 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:07:53.825 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:53.825 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:07:53.825 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:07:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:07:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:07:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:07:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:07:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:07:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:07:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:07:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:07:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:07:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:07:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:07:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:07:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:53.836 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:07:53Z" 2023-11-02T19:07:53.836 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:07:53.837 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:07:53Z" } 2023-11-02T19:07:53.837 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:07:53.838 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:53.838 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:53.839 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:07:58.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:58.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:07:58.841 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:07:58.887 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:07:58.888 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:07:58.948 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:58.939+0000 7f23485b9f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:58.949 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:07:58.943+0000 7f23485b9f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:07:58.978 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:07:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:07:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:07:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:07:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:07:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:07:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:07:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:07:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:07:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:58.982 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:07:58.982 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:58.982 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:07:58.982 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:58.982 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:07:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:07:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:07:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:07:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:07:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:07:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:07:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:07:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:07:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:07:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:07:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:07:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:07:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:07:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:07:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:07:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:07:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:07:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:07:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:07:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:07:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:07:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:07:58Z" 2023-11-02T19:07:58.992 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:07:58.993 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:07:58Z" } 2023-11-02T19:07:58.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:07:58.994 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:58.995 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:07:58.995 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:08:03.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:03.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:03.998 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:04.044 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:04.045 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:04.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:04.095+0000 7f6b18965f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:04.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:04.095+0000 7f6b18965f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:04.132 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:04.133 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:04.133 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:04.133 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:04.133 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:08:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:08:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:04Z" 2023-11-02T19:08:04.146 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:04.147 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:04Z" } 2023-11-02T19:08:04.148 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:04.148 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:04.149 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:04.149 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:08:09.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:09.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:09.151 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:09.197 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:09.197 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:09.257 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:09.251+0000 7f532e9bbf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:09.258 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:09.255+0000 7f532e9bbf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:09.289 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:09.290 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:09.290 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:09.290 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:09.290 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:09.291 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:09.291 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:09.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:09.291 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:09.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:09.292 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:09.292 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:09.292 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:09.292 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:09.292 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:09.293 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:09.293 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:09.293 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:09.293 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:09.293 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:09.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:09.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:09.294 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:09.294 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:09.294 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:09.295 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:09.295 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:09.295 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:09.295 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:09.295 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:09.296 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:09.296 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:09.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:09.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:09.296 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:09.297 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:09.297 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:09.297 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951824.645395_646.1", 2023-11-02T19:08:09.297 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:09.297 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:09.298 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:09.298 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.645395Z", 2023-11-02T19:08:09.298 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:09.298 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:09.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:09.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:09.299 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:09.299 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:09.299 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:09.300 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:09.300 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:09.300 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:09.300 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:09.300 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:09.301 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:09.301 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:09.301 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:09.301 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:09.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:09.302 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:09.302 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:09.302 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:09.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:09.303 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:09Z" 2023-11-02T19:08:09.303 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:09.304 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951824.645395_646.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.645395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:09Z" } 2023-11-02T19:08:09.304 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951824.645395_646.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.645395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:09.305 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:09.305 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951824.645395_646.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:09.306 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698951980.556340_671.1 target=1_1698951824.645395_646.1 2023-11-02T19:08:14.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:14.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:14.308 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:14.355 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:14.407 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:14.418 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:14.411+0000 7f13e2c81f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:14.418 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:14.415+0000 7f13e2c81f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:14.446 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:14.447 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:14.447 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:14.447 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:14.447 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:14.448 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:14.448 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:14.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:14.448 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:14.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:14.449 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:14.449 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:14.449 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:14.449 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:14.450 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:14.450 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:14.450 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:14.450 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:14.451 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:14.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:14.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:14.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:14.451 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:14.452 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:14.452 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:14.452 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:14.452 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:14.452 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:14.453 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:14.453 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:14.453 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:14.453 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:14.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:14.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:14.454 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:14.454 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:14.454 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:14.455 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951980.556340_671.1", 2023-11-02T19:08:14.455 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:14.455 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:14.455 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:14.455 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:06:20.556340Z", 2023-11-02T19:08:14.456 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:14.456 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:14.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:14.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:14.456 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:14.457 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:14.457 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:14.457 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:14.457 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:14.457 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:14.458 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:14.458 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:14.458 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:14.458 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:14.459 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:14.459 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:14.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:14.459 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:14.459 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:14.460 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:14.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:14.460 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:14Z" 2023-11-02T19:08:14.460 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:14.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951980.556340_671.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:06:20.556340Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:14Z" } 2023-11-02T19:08:14.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951980.556340_671.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:06:20.556340Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:14.462 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:14.463 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:14.463 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:08:14.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:14.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:14.464 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:14.513 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:14.514 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:14.620 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-11-02T19:08:14.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:14.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:14.622 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:14.673 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:14.674 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:14.768 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-11-02T19:08:14.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:14.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:14.770 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:14.821 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:14.821 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:14.926 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-11-02T19:08:14.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:14.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:14.927 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:14.972 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:14.973 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:15.078 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_policy":{"groups":[]}} 2023-11-02T19:08:15.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:15.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'ibtljn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:15.079 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket ibtljn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:15.122 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:15.122 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:15.222 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[]} 2023-11-02T19:08:15.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:15.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:15.223 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:15.268 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:15.268 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:15.327 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:08:15.356 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:08:15.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:15.357 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:15.357 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:08:15.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:15.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:15.358 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:15.358 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:08:15.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:15.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:15.359 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:08:15.359 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:08:15.043328Z" 2023-11-02T19:08:15.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:15.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:15.360 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:15.360 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:08:15.360 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:15.361 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:08:15.362 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:15.362 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:08:15.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:15.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:15.364 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:15.410 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:15.410 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:15.472 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:15.467+0000 7fb38eabff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:15.474 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:15.471+0000 7fb38eabff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:15.502 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:15.503 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:15.503 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:15.503 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:15.504 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:15.504 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:15.504 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:15.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:15.504 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:15.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:15.505 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:15.505 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:15.505 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:15.506 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:15.506 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:15.506 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:15.506 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:15.506 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:15.507 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:15.507 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:15.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:15.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:15.508 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:15.508 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:15.508 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:15.508 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:15.509 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:15.509 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:15.509 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:15.509 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:15.510 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:15.510 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:15.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:15.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:15.510 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:15.511 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:15.511 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:15.511 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951980.556340_671.1", 2023-11-02T19:08:15.511 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:15.512 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:15.512 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:15.512 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:06:20.556340Z", 2023-11-02T19:08:15.512 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:15.512 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:15.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:15.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:15.513 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:15.513 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:15.514 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:15.514 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:15.514 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:15.514 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:15.515 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:15.515 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:15.515 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:15.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:15.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:15.516 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:15.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:15.516 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:15.516 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:15.516 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:15.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:15.517 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:15Z" 2023-11-02T19:08:15.517 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:15.518 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951980.556340_671.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:06:20.556340Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:15Z" } 2023-11-02T19:08:15.519 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951980.556340_671.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:06:20.556340Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:15.519 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:15.520 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:15.520 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952095.043328_679.1 target=1_1698951980.556340_671.1 2023-11-02T19:08:20.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:20.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:20.523 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:20.566 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:20.566 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:20.576 INFO:tasks.rgw.c2.client.1.smithi069.stdout:2023-11-02T19:08:20.570+0000 7f1bb5aa4700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-11-02T19:08:20.625 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:20.623+0000 7f28f05a5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:20.626 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:20.623+0000 7f28f05a5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:20.650 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:20.654 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:20.654 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:20.655 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:20.655 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:20.656 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:20.656 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:20.656 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951980.556340_671.1", 2023-11-02T19:08:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:06:20.556340Z", 2023-11-02T19:08:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:20.664 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:20.664 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:20.664 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:20.664 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:20.665 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:20.665 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:20.665 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:20.665 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:20.665 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:20.666 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:20.666 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:20.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:20.666 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:20.667 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:20.667 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:20.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:20.667 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:20Z" 2023-11-02T19:08:20.668 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:20.669 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951980.556340_671.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:06:20.556340Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:20Z" } 2023-11-02T19:08:20.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951980.556340_671.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:06:20.556340Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:20.670 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:20.671 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:20.671 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952095.043328_679.1 target=1_1698951980.556340_671.1 2023-11-02T19:08:25.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:25.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:25.674 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:25.727 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:25.727 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:25.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:25.775+0000 7faa5bad8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:25.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:25.775+0000 7faa5bad8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:25.808 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698951980.556340_671.1", 2023-11-02T19:08:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:06:20.556340Z", 2023-11-02T19:08:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:25Z" 2023-11-02T19:08:25.822 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:25.823 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698951980.556340_671.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:06:20.556340Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:25Z" } 2023-11-02T19:08:25.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951980.556340_671.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:06:20.556340Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:25.824 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:25.824 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951980.556340_671.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:25.825 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952095.043328_679.1 target=1_1698951980.556340_671.1 2023-11-02T19:08:30.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:30.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:30.828 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:30.873 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:30.874 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:30.932 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:30.927+0000 7efd38ab4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:30.933 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:30.931+0000 7efd38ab4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:30.959 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:30.960 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:30.960 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:30.960 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:30.960 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952094.580357_672.1", 2023-11-02T19:08:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:14.580357Z", 2023-11-02T19:08:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:30.971 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:30.971 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:30.971 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:30.971 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:30Z" 2023-11-02T19:08:30.975 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:30.976 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952094.580357_672.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:14.580357Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:30Z" } 2023-11-02T19:08:30.977 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952094.580357_672.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:14.580357Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:30.977 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:30.978 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952094.580357_672.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:30.978 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952095.043328_679.1 target=1_1698952094.580357_672.1 2023-11-02T19:08:35.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:35.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:35.981 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:36.089 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:36.083+0000 7f5a81595f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:36.090 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:36.087+0000 7f5a81595f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:36.118 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:08:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:08:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:36Z" 2023-11-02T19:08:36.135 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:36.136 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:36Z" } 2023-11-02T19:08:36.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:36.137 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:36.137 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:36.138 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:08:36.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:36.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:36.139 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:36.185 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:36.185 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:36.283 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-11-02T19:08:36.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:36.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:36.284 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:36.335 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:36.335 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:36.433 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-11-02T19:08:36.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:36.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:36.435 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:36.480 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:36.481 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:36.586 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-11-02T19:08:36.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:36.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:36.587 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:36.630 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:36.630 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:36.717 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_policy":{"groups":[]}} 2023-11-02T19:08:36.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:36.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:36.719 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:36.851 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[]} 2023-11-02T19:08:36.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:36.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:36.853 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:36.898 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:37.020 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:37.020 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:37.021 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 14, 2023-11-02T19:08:37.021 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T19:08:37.021 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T19:08:37.021 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T19:08:37.022 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:37.022 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T19:08:37.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.022 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T19:08:37.022 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T19:08:37.023 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:08:37.023 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T19:08:37.023 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:08:37.023 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:08:37.024 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:37.024 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:08:37.024 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:08:37.024 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:08:37.025 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:08:37.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.025 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:08:37.025 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T19:08:37.025 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:08:37.026 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T19:08:37.026 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:37.026 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:08:37.026 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:08:37.027 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:08:37.027 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:08:37.027 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:08:37.027 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:08:37.027 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:08:37.028 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:08:37.028 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:08:37.028 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:08:37.028 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:08:37.029 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:37.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.029 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:08:37.029 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T19:08:37.030 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:08:37.030 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:08:37.030 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:37.030 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:08:37.031 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:08:37.031 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:08:37.031 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:08:37.031 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:08:37.031 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:08:37.032 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:08:37.032 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:08:37.032 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:08:37.032 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:08:37.032 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:08:37.033 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:37.033 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.033 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:37.033 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:08:37.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.034 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:08:37.034 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:08:37.034 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:08:37.034 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:08:37.035 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:37.035 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.035 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:37.035 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:08:37.035 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:08:37.036 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:08:37.036 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T19:08:37.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.036 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:08:37.036 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:08:37.037 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:37.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.037 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T19:08:37.038 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T19:08:37.038 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:08:37.038 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T19:08:37.038 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:08:37.038 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:08:37.039 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:37.039 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:08:37.039 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:08:37.039 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:08:37.040 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:08:37.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.040 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:08:37.040 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T19:08:37.040 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:08:37.041 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:08:37.041 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:37.041 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:08:37.041 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:08:37.041 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:08:37.042 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:08:37.042 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:08:37.042 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:08:37.042 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:08:37.043 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:08:37.043 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:08:37.043 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:08:37.044 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:08:37.044 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:37.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.045 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:08:37.045 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T19:08:37.045 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:08:37.045 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T19:08:37.046 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:37.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:08:37.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:08:37.046 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:08:37.046 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:08:37.047 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:08:37.047 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:08:37.047 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:08:37.047 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:08:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:08:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:08:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:08:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:08:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:08:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:08:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:08:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:08:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:08:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:08:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:08:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T19:08:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:08:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:08:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T19:08:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:08:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T19:08:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:08:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T19:08:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:08:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T19:08:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:08:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T19:08:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T19:08:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:08:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T19:08:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T19:08:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:08:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:08:37.060 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:08:37.060 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:08:37.060 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:08:37.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.061 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T19:08:37.061 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:08:37.061 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:08:37.061 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:08:37.061 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:08:37.062 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:08:37.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.062 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T19:08:37.062 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:08:37.063 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:08:37.063 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:08:37.063 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:08:37.063 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:08:37.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T19:08:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:08:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:08:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:08:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:08:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:08:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T19:08:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:08:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:08:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:08:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:08:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:08:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:08:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T19:08:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:37.067 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:37.069 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2023-11-02T19:08:37.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:37.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:37.071 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:08:37.216 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:08:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:08:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:08:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:08:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:08:15.043328Z" 2023-11-02T19:08:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:08:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.221 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:08:37.222 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:37.223 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:08:37.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:37.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:37.224 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:37.290 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:37.290 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:37.367+0000 7f2641de2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:37.367+0000 7f2641de2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:37.405 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:08:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:08:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:37Z" 2023-11-02T19:08:37.421 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:37.423 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:37Z" } 2023-11-02T19:08:37.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:37.424 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:37.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:37.425 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:08:37.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:37.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:37.426 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:37.474 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:37.475 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:37.575 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[]} 2023-11-02T19:08:37.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:37.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:37.577 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:37.626 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:37.626 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:37.707 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-11-02T19:08:37.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:37.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:37.709 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:37.760 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:37.761 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:37.836 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-11-02T19:08:37.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:37.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:37.838 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:37.880 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:37.880 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:37.971 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:08:37.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:37.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:37.972 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:38.017 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:38.017 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:38.115 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:08:38.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:38.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:38.116 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:38.159 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:38.160 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:38.275 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:38.275 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:38.275 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 15, 2023-11-02T19:08:38.276 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T19:08:38.276 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T19:08:38.276 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T19:08:38.276 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:38.277 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T19:08:38.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.277 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T19:08:38.277 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T19:08:38.277 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:08:38.278 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T19:08:38.278 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:08:38.278 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:08:38.278 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:38.279 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:08:38.279 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:08:38.279 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:08:38.279 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:08:38.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.280 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:08:38.280 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T19:08:38.280 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:08:38.280 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T19:08:38.281 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:38.281 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:08:38.281 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:08:38.281 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:08:38.282 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:08:38.282 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:08:38.282 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:08:38.282 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:08:38.283 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:08:38.283 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:08:38.283 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:08:38.283 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:08:38.284 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.284 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:08:38.285 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T19:08:38.285 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:08:38.285 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:08:38.285 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:38.286 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:08:38.286 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:08:38.286 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:08:38.286 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:08:38.287 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:08:38.287 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:08:38.287 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:08:38.287 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:08:38.287 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:08:38.288 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:08:38.288 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:08:38.288 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.288 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.288 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:38.289 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:08:38.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.289 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:08:38.289 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:08:38.290 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:08:38.290 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:08:38.290 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.290 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.290 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:38.291 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:08:38.291 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:08:38.291 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:08:38.291 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [ 2023-11-02T19:08:38.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.292 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-group", 2023-11-02T19:08:38.292 INFO:teuthology.orchestra.run.smithi017.stdout: "data_flow": { 2023-11-02T19:08:38.292 INFO:teuthology.orchestra.run.smithi017.stdout: "symmetrical": [ 2023-11-02T19:08:38.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.293 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-flow1", 2023-11-02T19:08:38.293 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:08:38.293 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:08:38.293 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:08:38.294 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.294 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.294 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.294 INFO:teuthology.orchestra.run.smithi017.stdout: "pipes": [ 2023-11-02T19:08:38.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.295 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-pipe", 2023-11-02T19:08:38.295 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:08:38.295 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "*", 2023-11-02T19:08:38.295 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:08:38.296 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:08:38.296 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:08:38.296 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.297 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": { 2023-11-02T19:08:38.297 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "*", 2023-11-02T19:08:38.297 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:08:38.297 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:08:38.297 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:08:38.298 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.298 INFO:teuthology.orchestra.run.smithi017.stdout: "params": { 2023-11-02T19:08:38.298 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:08:38.299 INFO:teuthology.orchestra.run.smithi017.stdout: "filter": { 2023-11-02T19:08:38.299 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [] 2023-11-02T19:08:38.299 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.299 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": {}, 2023-11-02T19:08:38.300 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2023-11-02T19:08:38.300 INFO:teuthology.orchestra.run.smithi017.stdout: "mode": "system", 2023-11-02T19:08:38.300 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "" 2023-11-02T19:08:38.300 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.301 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.301 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:38.301 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "enabled" 2023-11-02T19:08:38.301 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.301 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.302 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:08:38.302 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:08:38.302 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.303 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T19:08:38.303 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T19:08:38.304 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:08:38.304 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T19:08:38.304 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:08:38.304 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:08:38.305 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:38.305 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:08:38.305 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:08:38.305 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:08:38.305 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:08:38.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.306 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:08:38.306 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T19:08:38.306 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:08:38.307 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:08:38.307 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:38.307 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:08:38.308 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:08:38.308 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:08:38.308 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:08:38.308 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:08:38.308 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:08:38.309 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:08:38.309 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:08:38.309 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:08:38.309 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:08:38.310 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:08:38.310 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.311 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:08:38.311 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T19:08:38.311 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:08:38.311 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T19:08:38.311 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:38.312 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:08:38.312 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:08:38.312 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:08:38.312 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:08:38.313 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:08:38.313 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:08:38.313 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:08:38.313 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:08:38.314 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:08:38.314 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:08:38.314 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:08:38.314 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.315 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.315 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:38.315 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:08:38.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.316 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:08:38.316 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:08:38.316 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:08:38.316 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:08:38.316 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.317 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:38.317 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:08:38.317 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:08:38.318 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:08:38.318 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T19:08:38.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.318 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:08:38.319 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:08:38.319 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.319 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.319 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:08:38.319 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T19:08:38.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.320 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:08:38.320 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T19:08:38.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.321 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:08:38.321 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T19:08:38.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.323 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:08:38.323 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T19:08:38.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.324 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:08:38.324 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T19:08:38.324 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.324 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:38.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.325 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T19:08:38.325 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:08:38.325 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T19:08:38.325 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T19:08:38.326 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:08:38.326 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:08:38.326 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:08:38.326 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:08:38.327 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:08:38.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.327 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T19:08:38.327 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:08:38.328 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:08:38.328 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:08:38.328 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:08:38.328 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:08:38.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.329 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T19:08:38.329 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:08:38.329 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:08:38.329 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:08:38.330 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:08:38.330 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:08:38.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.331 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T19:08:38.331 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:08:38.331 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:08:38.331 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:08:38.332 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:08:38.332 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:08:38.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.332 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T19:08:38.332 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:08:38.333 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:08:38.333 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:08:38.333 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:08:38.333 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:08:38.334 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.334 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:08:38.334 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T19:08:38.334 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:38.335 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:38.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:38.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:08:38.336 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:08:38.379 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:38.380 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:38.480 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:08:38.480 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-66 2023-11-02T19:08:38.481 DEBUG:boto:path=/ibtljn-66/ 2023-11-02T19:08:38.482 DEBUG:boto:auth_path=/ibtljn-66/ 2023-11-02T19:08:38.482 DEBUG:boto:Method: PUT 2023-11-02T19:08:38.483 DEBUG:boto:Path: /ibtljn-66/ 2023-11-02T19:08:38.483 DEBUG:boto:Data: 2023-11-02T19:08:38.483 DEBUG:boto:Headers: {} 2023-11-02T19:08:38.484 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:08:38.484 DEBUG:boto:Port: 8000 2023-11-02T19:08:38.485 DEBUG:boto:Params: {} 2023-11-02T19:08:38.485 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T19:08:38.486 DEBUG:boto:Token: None 2023-11-02T19:08:38.486 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 19:08:38 GMT /ibtljn-66/ 2023-11-02T19:08:38.487 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XzEbnvQyGzyV6g2gdfDGkqAqIDs= 2023-11-02T19:08:38.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:08:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XzEbnvQyGzyV6g2gdfDGkqAqIDs=', 'Content-Length': '0'} 2023-11-02T19:08:38.620 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b2323a65b66995cf-006543f3b6-2354-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:08:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:08:38.621 DEBUG:boto:path=/ibtljn-66/ 2023-11-02T19:08:38.621 DEBUG:boto:auth_path=/ibtljn-66/ 2023-11-02T19:08:38.622 DEBUG:boto:Method: HEAD 2023-11-02T19:08:38.622 DEBUG:boto:Path: /ibtljn-66/ 2023-11-02T19:08:38.622 DEBUG:boto:Data: 2023-11-02T19:08:38.623 DEBUG:boto:Headers: {} 2023-11-02T19:08:38.623 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:08:38.624 DEBUG:boto:Port: 8000 2023-11-02T19:08:38.624 DEBUG:boto:Params: {} 2023-11-02T19:08:38.625 DEBUG:boto:Token: None 2023-11-02T19:08:38.625 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:08:38 GMT /ibtljn-66/ 2023-11-02T19:08:38.626 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ga9c5x12Yjf/6PtSiEgOgzr5A0s= 2023-11-02T19:08:38.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:08:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ga9c5x12Yjf/6PtSiEgOgzr5A0s=', 'Content-Length': '0'} 2023-11-02T19:08:38.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000535223481dcd66df-006543f3b6-2354-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:08:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:08:38.631 DEBUG:boto:path=/ibtljn-66/obj1 2023-11-02T19:08:38.632 DEBUG:boto:auth_path=/ibtljn-66/obj1 2023-11-02T19:08:38.632 DEBUG:boto:Method: PUT 2023-11-02T19:08:38.633 DEBUG:boto:Path: /ibtljn-66/obj1 2023-11-02T19:08:38.633 DEBUG:boto:Data: 2023-11-02T19:08:38.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:08:38.634 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:08:38.634 DEBUG:boto:Port: 8000 2023-11-02T19:08:38.635 DEBUG:boto:Params: {} 2023-11-02T19:08:38.635 DEBUG:boto:Token: None 2023-11-02T19:08:38.636 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:08:38 GMT /ibtljn-66/obj1 2023-11-02T19:08:38.636 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:car9mIq5vnq98sSmpoPH4Phx/BM= 2023-11-02T19:08:38.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:08:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:car9mIq5vnq98sSmpoPH4Phx/BM='} 2023-11-02T19:08:38.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f435f204328ebad-006543f3b6-2354-a1'), ('Date', 'Thu, 02 Nov 2023 19:08:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:08:38.655 INFO:rgw_multi.tests:create bucket zone=a2 name=ibtljn-67 2023-11-02T19:08:38.655 DEBUG:boto:path=/ibtljn-67/ 2023-11-02T19:08:38.656 DEBUG:boto:auth_path=/ibtljn-67/ 2023-11-02T19:08:38.656 DEBUG:boto:Method: PUT 2023-11-02T19:08:38.657 DEBUG:boto:Path: /ibtljn-67/ 2023-11-02T19:08:38.657 DEBUG:boto:Data: 2023-11-02T19:08:38.657 DEBUG:boto:Headers: {} 2023-11-02T19:08:38.658 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:08:38.658 DEBUG:boto:Port: 8001 2023-11-02T19:08:38.659 DEBUG:boto:Params: {} 2023-11-02T19:08:38.659 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:08:38.660 DEBUG:boto:Token: None 2023-11-02T19:08:38.660 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 19:08:38 GMT /ibtljn-67/ 2023-11-02T19:08:38.661 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m3cBf87tkfyjNeYZR8CGmwEIngM= 2023-11-02T19:08:38.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:08:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m3cBf87tkfyjNeYZR8CGmwEIngM=', 'Content-Length': '0'} 2023-11-02T19:08:38.715 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a1cf10c43b71d7d1-006543f3b6-2356-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:08:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:08:38.716 DEBUG:boto:path=/ibtljn-67/ 2023-11-02T19:08:38.717 DEBUG:boto:auth_path=/ibtljn-67/ 2023-11-02T19:08:38.718 DEBUG:boto:Method: HEAD 2023-11-02T19:08:38.719 DEBUG:boto:Path: /ibtljn-67/ 2023-11-02T19:08:38.719 DEBUG:boto:Data: 2023-11-02T19:08:38.720 DEBUG:boto:Headers: {} 2023-11-02T19:08:38.720 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:08:38.720 DEBUG:boto:Port: 8001 2023-11-02T19:08:38.721 DEBUG:boto:Params: {} 2023-11-02T19:08:38.721 DEBUG:boto:Token: None 2023-11-02T19:08:38.722 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:08:38 GMT /ibtljn-67/ 2023-11-02T19:08:38.722 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:7VEfDJoAShjEzA2fjfntttjYhjc= 2023-11-02T19:08:38.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:08:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:7VEfDJoAShjEzA2fjfntttjYhjc=', 'Content-Length': '0'} 2023-11-02T19:08:38.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f824785764236075-006543f3b6-2356-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:08:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:08:38.726 DEBUG:boto:path=/ibtljn-67/obj2 2023-11-02T19:08:38.726 DEBUG:boto:auth_path=/ibtljn-67/obj2 2023-11-02T19:08:38.727 DEBUG:boto:Method: PUT 2023-11-02T19:08:38.727 DEBUG:boto:Path: /ibtljn-67/obj2 2023-11-02T19:08:38.728 DEBUG:boto:Data: 2023-11-02T19:08:38.728 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:08:38.728 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:08:38.729 DEBUG:boto:Port: 8001 2023-11-02T19:08:38.729 DEBUG:boto:Params: {} 2023-11-02T19:08:38.730 DEBUG:boto:Token: None 2023-11-02T19:08:38.730 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:08:38 GMT /ibtljn-67/obj2 2023-11-02T19:08:38.731 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xvofZzh2bdBezAyqXcDVDluDrOU= 2023-11-02T19:08:38.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:08:38 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xvofZzh2bdBezAyqXcDVDluDrOU='} 2023-11-02T19:08:38.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005d1e283cd9503ebe-006543f3b6-2356-a2'), ('Date', 'Thu, 02 Nov 2023 19:08:38 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:08:38.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:38.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:38.748 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:38.848 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:08:38.872 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:08:38.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:08:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:08:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952118.696539_697.1", 2023-11-02T19:08:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:08:38.696539Z" 2023-11-02T19:08:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:08:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:38.876 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:08:38.878 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:38.878 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:08:38.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:38.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:38.880 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:38.926 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:38.926 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:38.978 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:38.971+0000 7efdb8012f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:38.978 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:38.975+0000 7efdb8012f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:39.002 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:39.002 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:39.002 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:39.003 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:39.003 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:39.003 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:39.003 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:39.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:39.004 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:39.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:39.004 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:39.004 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:39.005 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:39.005 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:39.005 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:39.005 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:39.006 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:39.006 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:39.006 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:39.006 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:39.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:39.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:39.007 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:39.007 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:39.007 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:39.008 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:39.008 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:39.008 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:39.008 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:39.009 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:39.009 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:39.009 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:39.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:39.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:39.010 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:39.010 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:39.010 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:39.011 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:08:39.011 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:39.011 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:39.011 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:39.011 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:08:39.012 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:39.012 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:39.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:39.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:39.012 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:39.013 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:39.013 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:39.013 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:39.013 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:39.013 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:39.014 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:39.014 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:39.014 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:39.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:39.015 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:39.015 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:39.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:39.015 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:39.015 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:39.016 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:39.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:39.016 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:38Z" 2023-11-02T19:08:39.016 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:39.017 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:38Z" } 2023-11-02T19:08:39.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:39.019 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:39.019 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:39.020 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:08:44.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:44.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:44.022 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:44.070 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:44.071 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:44.133 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:44.131+0000 7fcbfcbe9f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:44.134 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:44.131+0000 7fcbfcbe9f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:44.163 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:44.163 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:44.164 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:44.164 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:44.164 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:44.164 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:44.165 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:44.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:44.165 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:44.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:44.165 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:44.166 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:44.166 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:44.166 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:44.166 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:44.167 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:44.167 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:44.167 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:44.167 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:44.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:44.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:44.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:44.168 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:44.168 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:44.169 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:44.169 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:44.169 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:44.169 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:44.169 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:44.170 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:44.170 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:44.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:44.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:44.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:44.171 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:44.171 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:44.171 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:44.171 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:08:44.172 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:44.172 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:44.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:44.172 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:08:44.172 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:44.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:44.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:44.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:44.173 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:44.174 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:44.174 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:44.174 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:44.174 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:44.175 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:44.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:44.175 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:44.175 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:44.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:44.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:44.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:44.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:44.176 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:44.176 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:44.177 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:44.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:44.177 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:44Z" 2023-11-02T19:08:44.177 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:44.178 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:44Z" } 2023-11-02T19:08:44.179 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:44.180 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:44.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:44.180 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:08:49.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:49.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:49.183 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:49.228 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:49.229 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:49.289 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:49.287+0000 7fdac3bc4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:49.291 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:49.287+0000 7fdac3bc4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:49.320 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:49.320 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:49.321 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:49.321 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:49.321 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:49.321 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:49.322 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:49.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:49.322 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:49.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:49.323 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:49.323 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:49.323 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:49.323 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:49.323 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:49.324 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:49.324 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:49.324 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:49.324 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:49.325 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:49.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:49.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:49.325 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:49.325 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:49.326 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:49.326 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:49.326 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:49.326 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:49.327 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:49.327 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:49.327 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:49.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:49.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:49.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:49.328 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:49.328 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:49.328 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:49.329 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:08:49.329 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:49.329 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:49.329 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:49.329 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:08:49.330 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:49.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:49.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:49.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:49.330 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:49.331 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:49.331 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:49.331 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:49.331 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:49.332 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:49.332 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:49.332 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:49.332 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:49.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:49.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:49.333 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:49.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:49.334 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:49.334 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:49.334 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:49.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:49.334 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:49Z" 2023-11-02T19:08:49.335 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:49.336 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:49Z" } 2023-11-02T19:08:49.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:49.337 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:49.338 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:49.338 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:08:54.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:54.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:54.341 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:54.388 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:54.388 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:54.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:54.447+0000 7fda94889f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:54.454 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:54.451+0000 7fda94889f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:54.485 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:54.485 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:54.485 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:54.485 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:54.486 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:54.486 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:54.486 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:54.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:54.487 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:54.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:54.487 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:54.487 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:54.487 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:54.488 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:54.488 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:54.488 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:54.488 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:54.489 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:54.489 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:54.489 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:54.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:54.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:54.490 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:54.490 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:54.490 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:54.491 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:54.491 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:54.491 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:54.491 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:54.492 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:54.492 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:54.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:54.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:54.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:54.493 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:54.493 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:54.493 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:54.493 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:08:54.494 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:54.494 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:54.494 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:54.494 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:08:54.495 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:54.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:54.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:54.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:54.495 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:54.496 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:54.496 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:54.496 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:54.496 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:54.496 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:54.497 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:54.497 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:54.497 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:54.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:54.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:54.498 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:54.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:54.498 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:54.498 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:54.499 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:54.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:54.499 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:54Z" 2023-11-02T19:08:54.499 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:54.500 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:54Z" } 2023-11-02T19:08:54.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:54.501 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:54.502 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:54.502 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:08:59.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:59.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:08:59.505 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:08:59.554 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:08:59.554 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:08:59.613 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:59.607+0000 7fa1e49def40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:59.614 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:08:59.611+0000 7fa1e49def40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:08:59.640 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:08:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:08:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:08:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:08:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:08:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:08:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:08:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:08:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:08:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:08:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:08:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:08:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:08:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:08:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:08:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:08:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:08:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:08:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:08:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:08:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:08:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:08:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:08:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:08:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:08:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:08:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:08:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:08:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:08:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:08:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:08:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:08:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:08:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:08:59Z" 2023-11-02T19:08:59.655 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:08:59.657 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:08:59Z" } 2023-11-02T19:08:59.657 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:08:59.658 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:59.658 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:08:59.659 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:09:04.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:04.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:04.662 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:09:04.710 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:09:04.711 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:09:04.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:04.767+0000 7fc0dd637f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:04.771 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:04.767+0000 7fc0dd637f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:04.800 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:09:04.800 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:09:04.801 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:09:04.801 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:09:04.801 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:09:04.801 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:09:04.801 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:04.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:04.802 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:09:04.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:04.802 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:09:04.803 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:04.803 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:04.803 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:09:04.803 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:04.804 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:09:04.804 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:04.804 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:09:04.804 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:04.804 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:04.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:04.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:04.805 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:09:04.805 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:04.805 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:04.806 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:09:04.806 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:04.806 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:04.807 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:04.807 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:09:04.807 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:04.807 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:04.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:04.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:04.808 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:09:04.808 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:04.808 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:04.808 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:09:04.809 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:04.809 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:04.809 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:04.809 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:09:04.809 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:04.810 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:04.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:04.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:04.810 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:09:04.811 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:04.811 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:04.811 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:09:04.811 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:04.812 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:04.812 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:04.812 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:09:04.812 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:04.812 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:04.813 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:04.813 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:09:04.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:04.813 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:09:04.813 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:09:04.814 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:09:04.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:04.814 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:09:04Z" 2023-11-02T19:09:04.814 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:09:04.815 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:09:04Z" } 2023-11-02T19:09:04.816 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:09:04.816 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:04.817 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:04.817 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:09:09.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:09.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:09.819 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:09:09.865 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:09:09.865 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:09:09.921 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:09.915+0000 7fda57447f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:09.922 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:09.919+0000 7fda57447f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:09.949 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:09:09.950 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:09:09.950 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:09:09.950 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:09:09.951 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:09:09.951 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:09:09.951 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:09.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:09.951 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:09:09.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:09.952 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:09:09.952 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:09.953 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:09.953 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:09:09.953 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:09.953 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:09:09.953 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:09.954 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:09:09.954 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:09.954 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:09.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:09.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:09.955 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:09:09.955 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:09.955 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:09.955 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:09:09.956 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:09.956 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:09.956 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:09.956 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:09:09.956 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:09.957 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:09.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:09.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:09.958 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:09:09.958 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:09.958 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:09.958 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:09:09.958 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:09.959 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:09.959 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:09.959 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:09:09.959 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:09.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:09.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:09.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:09.960 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:09:09.960 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:09.961 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:09.961 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:09:09.961 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:09.961 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:09.961 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:09.962 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:09:09.962 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:09.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:09.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:09.963 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:09:09.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:09.963 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:09:09.963 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:09:09.963 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:09:09.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:09.964 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:09:09Z" 2023-11-02T19:09:09.964 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:09:09.965 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:09:09Z" } 2023-11-02T19:09:09.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:09:09.966 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:09.967 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:09.967 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:09:14.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:14.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:14.969 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:09:15.018 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:09:15.019 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:09:15.080 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:15.074+0000 7febfbebff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:15.082 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:15.078+0000 7febfbebff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:15.111 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:09:15.111 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:09:15.111 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:09:15.112 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:09:15.112 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:09:15.112 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:09:15.112 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:15.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:15.113 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:09:15.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:15.113 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:09:15.113 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:15.113 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:15.114 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:09:15.114 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:15.114 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:09:15.114 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:15.115 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:09:15.115 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:15.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:15.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:15.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:15.116 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:09:15.116 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:15.116 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:15.116 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:09:15.117 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:15.117 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:15.117 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:15.117 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:09:15.117 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:15.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:15.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:15.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:15.118 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:09:15.119 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:15.119 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:15.119 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:09:15.119 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:15.119 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:15.120 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:15.120 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:09:15.120 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:15.120 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:15.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:15.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:15.121 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:09:15.121 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:15.121 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:15.122 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:09:15.122 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:15.122 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:15.122 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:15.122 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:09:15.123 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:15.123 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:15.123 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:15.123 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:09:15.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:15.124 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:09:15.124 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:09:15.124 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:09:15.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:15.125 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:09:15Z" 2023-11-02T19:09:15.125 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:09:15.126 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:09:15Z" } 2023-11-02T19:09:15.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:09:15.127 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:15.127 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:15.128 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:09:20.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:20.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:20.130 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:09:20.182 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:09:20.182 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:09:20.241 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:20.238+0000 7f343cc33f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:20.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:20.238+0000 7f343cc33f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:20.270 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:09:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:09:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:09:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:09:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:09:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:09:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:09:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:09:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:09:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:09:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:09:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:09:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:20.276 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:09:20.276 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:20.276 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:20.276 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:20.276 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:09:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:09:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:09:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:09:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:09:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:09:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:09:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:09:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:09:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:09:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:09:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:09:20Z" 2023-11-02T19:09:20.284 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:09:20.285 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:09:20Z" } 2023-11-02T19:09:20.285 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:09:20.286 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:20.286 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:20.287 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:09:25.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:25.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:25.290 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:09:25.337 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:09:25.338 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:09:25.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:25.390+0000 7ff7dc59bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:25.396 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:25.390+0000 7ff7dc59bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:25.424 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:09:25.425 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:09:25.425 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:09:25.425 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:09:25.425 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:09:25.426 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:09:25.426 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:25.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:25.427 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:09:25.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:25.427 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:09:25.427 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:25.428 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:25.428 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:09:25.428 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:25.428 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:09:25.428 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:25.429 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:09:25.429 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:25.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:25.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:25.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:25.430 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:09:25.430 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:25.430 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:25.430 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:09:25.431 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:25.431 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:25.431 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:25.431 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:09:25.432 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:25.432 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:25.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:25.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:25.433 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:09:25.433 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:25.433 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:25.433 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:09:25.433 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:25.434 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:25.434 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:25.434 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:09:25.434 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:25.435 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:25.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:25.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:25.435 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:09:25.436 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:25.436 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:25.436 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:09:25.436 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:25.436 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:25.437 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:25.437 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:09:25.437 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:25.437 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:25.437 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:25.438 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:09:25.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:25.438 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:09:25.438 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:09:25.439 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:09:25.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:25.439 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:09:25Z" 2023-11-02T19:09:25.439 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:09:25.440 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:09:25Z" } 2023-11-02T19:09:25.441 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:09:25.441 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:25.442 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:25.442 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:09:30.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:30.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:30.445 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:09:30.492 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:09:30.492 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:09:30.553 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:30.550+0000 7f3362705f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:30.554 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:30.550+0000 7f3362705f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:30.584 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:09:30.584 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:09:30.584 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:09:30.585 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:09:30.585 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:09:30.585 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:09:30.586 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:30.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:30.586 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:09:30.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:30.587 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:09:30.587 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:30.587 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:30.588 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:09:30.588 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:30.588 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:09:30.589 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:30.589 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:09:30.589 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:30.589 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:30.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:30.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:30.590 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:09:30.590 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:30.591 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:30.591 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:09:30.591 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:30.591 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:30.592 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:30.592 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:09:30.592 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:30.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:30.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:30.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:30.593 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:09:30.593 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:30.593 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:30.594 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:09:30.594 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:30.594 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:30.594 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:30.595 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:09:30.595 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:30.595 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:30.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:30.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:30.596 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:09:30.596 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:30.597 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:30.597 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:09:30.597 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:30.597 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:30.597 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:30.598 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:09:30.598 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:30.598 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:30.598 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:30.599 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:09:30.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:30.599 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:09:30.599 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:09:30.599 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:09:30.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:30.600 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:09:30Z" 2023-11-02T19:09:30.600 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:09:30.601 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:09:30Z" } 2023-11-02T19:09:30.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:09:30.602 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:30.602 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:30.603 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:09:35.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:35.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:35.606 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:09:35.652 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:09:35.652 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:09:35.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:35.710+0000 7ffb94372f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:35.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:35.710+0000 7ffb94372f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:35.737 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:09:35.737 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:09:35.737 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:09:35.738 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:09:35.738 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:09:35.738 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:09:35.738 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:35.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:35.739 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:09:35.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:35.739 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:09:35.739 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:35.740 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:35.740 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:09:35.740 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:35.741 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:09:35.741 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:35.741 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:09:35.741 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:35.742 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:35.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:35.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:35.742 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:09:35.742 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:35.743 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:35.743 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:09:35.743 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:35.743 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:35.743 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:35.744 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:09:35.744 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:35.744 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:35.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:35.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:35.745 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:09:35.745 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:35.745 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:35.745 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:09:35.746 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:35.746 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:35.746 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:35.746 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:09:35.746 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:35.747 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:35.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:35.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:35.747 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:09:35.748 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:35.748 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:35.748 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:09:35.748 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:35.748 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:35.749 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:35.749 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:09:35.750 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:35.750 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:35.750 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:35.751 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:09:35.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:35.751 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:09:35.751 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:09:35.751 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:09:35.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:35.752 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:09:35Z" 2023-11-02T19:09:35.752 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:09:35.753 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:09:35Z" } 2023-11-02T19:09:35.754 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:09:35.754 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:35.754 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:35.755 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:09:40.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:40.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:40.757 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:09:40.803 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:09:40.804 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:09:40.864 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:40.858+0000 7fb9c6a16f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:40.862+0000 7fb9c6a16f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:40.893 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:09:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:09:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:09:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:09:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:09:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:09:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:09:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:09:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:09:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:09:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:09:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:09:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:09:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:09:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:09:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:09:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:09:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:09:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:09:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:09:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:09:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:09:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:09:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:09:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:09:40Z" 2023-11-02T19:09:40.907 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:09:40.908 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:09:40Z" } 2023-11-02T19:09:40.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:09:40.909 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:40.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:40.910 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:09:45.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:45.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:45.913 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:09:45.961 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:09:45.963 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:09:46.020 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:46.014+0000 7f18e8f37f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:46.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:46.018+0000 7f18e8f37f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:46.050 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:09:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:09:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:09:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:09:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:09:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:09:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:09:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:09:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:09:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:09:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:09:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:09:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:09:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:09:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:09:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:09:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:09:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:09:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:09:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:09:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:09:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:09:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:09:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:09:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:09:46Z" 2023-11-02T19:09:46.067 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:09:46.068 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:09:46Z" } 2023-11-02T19:09:46.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:09:46.070 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:46.070 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:46.070 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:09:51.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:51.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:51.073 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:09:51.119 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:09:51.120 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:09:51.181 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:51.174+0000 7f2253971f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:51.182 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:51.178+0000 7f2253971f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:51.210 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:09:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:09:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:09:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:09:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:09:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:09:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:09:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:09:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:09:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:09:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:09:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:09:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:09:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:09:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:09:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:09:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:09:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:09:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:09:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:09:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:09:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:09:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:09:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:09:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:09:51Z" 2023-11-02T19:09:51.230 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:09:51.231 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:09:51Z" } 2023-11-02T19:09:51.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:09:51.232 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:51.232 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:51.233 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:09:56.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:56.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:09:56.235 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:09:56.281 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:09:56.282 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:09:56.345 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:56.338+0000 7f2c269c3f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:56.346 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:09:56.342+0000 7f2c269c3f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:09:56.375 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:09:56.375 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:09:56.375 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:09:56.376 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:09:56.376 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:09:56.376 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:09:56.376 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:56.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:56.377 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:09:56.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:56.377 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:09:56.377 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:56.378 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:56.378 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:09:56.378 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:56.378 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:09:56.378 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:56.379 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:09:56.379 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:56.379 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:56.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:56.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:56.380 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:09:56.380 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:56.380 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:56.380 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:09:56.381 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:56.381 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:56.381 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:56.381 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:09:56.382 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:56.382 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:56.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:56.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:56.382 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:09:56.383 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:56.383 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:56.383 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:09:56.383 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:56.384 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:56.384 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:56.384 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:09:56.384 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:56.384 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:56.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:56.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:09:56.385 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:09:56.385 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:09:56.385 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:09:56.386 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:09:56.386 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:09:56.386 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:09:56.386 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:09:56.387 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:09:56.387 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:09:56.387 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:56.387 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:09:56.387 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:09:56.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:56.388 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:09:56.388 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:09:56.388 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:09:56.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:09:56.389 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:09:56Z" 2023-11-02T19:09:56.389 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:09:56.390 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:09:56Z" } 2023-11-02T19:09:56.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:09:56.391 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:56.391 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:09:56.392 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:10:01.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:01.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:01.394 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:01.440 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:01.502 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:01.498+0000 7fd6b1b0bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:01.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:01.498+0000 7fd6b1b0bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:01.533 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:01.533 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:01.533 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:01.534 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:01.534 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:01.534 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:01.534 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:01.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:01.535 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:01.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:01.535 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:01.536 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:01.536 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:01.536 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:01.536 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:01.537 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:01.537 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:01.537 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:01.537 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:01.537 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:01.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:01.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:01.538 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:01.538 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:01.539 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:01.539 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:01.539 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:01.539 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:01.539 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:01.540 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:01.540 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:01.540 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:01.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:01.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:01.541 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:01.541 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:01.541 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:01.541 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:10:01.542 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:01.542 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:01.542 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:01.542 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:10:01.543 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:01.543 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:01.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:01.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:01.543 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:01.544 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:01.544 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:01.544 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:01.544 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:01.544 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:01.545 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:01.545 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:01.545 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:01.545 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:01.545 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:01.546 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:01.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:01.546 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:01.546 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:01.547 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:01.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:01.547 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:01Z" 2023-11-02T19:10:01.547 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:01.548 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:01Z" } 2023-11-02T19:10:01.549 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:01.549 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:01.550 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:01.550 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:10:06.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:06.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:06.552 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:06.599 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:06.599 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:06.655 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:06.650+0000 7f1476b08f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:06.657 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:06.650+0000 7f1476b08f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:06.683 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:06.684 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:06.684 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:06.684 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:06.685 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:06.685 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:06.685 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:06.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:06.685 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:06.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:06.686 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:06.686 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:06.686 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:06.687 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:06.687 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:06.687 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:06.687 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:06.687 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:06.688 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:06.688 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:06.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:06.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:06.689 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:06.689 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:06.689 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:06.689 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:06.689 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:06.690 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:06.690 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:06.690 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:06.691 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:06.691 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:06.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:06.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:06.691 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:06.692 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:06.692 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:06.692 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:10:06.692 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:06.693 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:06.693 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:06.693 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:10:06.693 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:06.694 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:06.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:06.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:06.694 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:06.694 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:06.695 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:06.695 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:06.695 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:06.695 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:06.696 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:06.696 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:06.696 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:06.696 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:06.696 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:06.697 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:06.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:06.697 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:06.697 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:06.698 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:06.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:06.698 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:06Z" 2023-11-02T19:10:06.698 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:06.699 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:06Z" } 2023-11-02T19:10:06.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:06.701 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:06.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:06.701 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:10:09.660 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T19:10:09.654+0000 7f25f2cc2700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-11-02T19:10:09.700 INFO:tasks.rgw.c2.client.0.smithi069.stdout:2023-11-02T19:10:09.695+0000 7fdaff3f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-11-02T19:10:11.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:11.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:11.703 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:11.743 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:11.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:11.794+0000 7f0e7f654f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:11.801 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:11.798+0000 7f0e7f654f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:11.830 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:10:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:10:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:11Z" 2023-11-02T19:10:11.844 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:11.845 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:11Z" } 2023-11-02T19:10:11.846 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:11.846 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:11.847 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:11.847 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:10:16.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:16.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:16.850 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:16.898 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:16.899 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:16.959 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:16.954+0000 7fea31803f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:16.960 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:16.954+0000 7fea31803f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:16.988 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:16.988 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:16.989 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:16.989 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:16.989 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:16.989 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:16.990 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:16.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:16.990 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:16.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:16.991 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:16.991 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:16.991 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:16.991 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:16.992 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:16.992 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:16.992 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:16.992 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:16.992 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:16.993 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:16.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:16.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:16.993 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:16.994 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:16.994 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:16.994 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:16.994 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:16.995 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:16.995 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:16.995 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:16.995 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:16.995 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:16.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:16.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:16.996 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:16.996 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:16.997 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:16.997 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:10:16.997 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:16.997 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:16.997 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:16.998 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:10:16.998 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:16.998 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:16.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:16.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:16.999 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:16.999 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:16.999 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:17.000 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:17.000 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:17.000 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:17.000 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:17.000 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:17.001 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:17.001 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:17.001 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:17.001 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:17.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:17.002 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:17.002 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:17.002 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:17.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:17.003 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:16Z" 2023-11-02T19:10:17.003 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:17.004 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:16Z" } 2023-11-02T19:10:17.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:17.005 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:17.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:17.006 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:10:22.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:22.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:22.009 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:22.055 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:22.056 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:22.116 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:22.110+0000 7efdda580f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:22.118 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:22.114+0000 7efdda580f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:22.164 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:22.164 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:22.164 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:22.165 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:22.165 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:22.165 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:22.165 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:22.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:22.166 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:22.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:22.166 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:22.167 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:22.167 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:22.167 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:22.167 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:22.167 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:22.168 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:22.168 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:22.168 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:22.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:22.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:22.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:22.169 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:22.169 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:22.169 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:22.170 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:22.170 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:22.170 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:22.170 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:22.170 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:22.171 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:22.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:22.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:22.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:22.172 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:22.172 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:22.172 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:22.172 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:10:22.172 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:22.173 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:22.173 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:22.173 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:10:22.173 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:22.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:22.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:22.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:22.174 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:22.174 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:22.175 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:22.175 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:22.175 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:22.175 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:22.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:22.176 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:22.176 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:22.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:22.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:22.177 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:22.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:22.177 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:22.177 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:22.177 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:22.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:22.178 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:22Z" 2023-11-02T19:10:22.178 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:22.179 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:22Z" } 2023-11-02T19:10:22.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:22.180 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:22.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:22.181 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:10:27.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:27.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:27.184 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:27.231 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:27.231 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:27.289 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:27.286+0000 7fc93e5d3f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:27.291 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:27.286+0000 7fc93e5d3f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:27.320 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:27.321 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:27.321 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:27.321 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:27.322 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:27.322 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:27.322 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:27.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:27.322 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:27.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:27.323 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:27.323 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:27.323 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:27.324 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:27.324 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:27.324 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:27.324 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:27.324 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:27.325 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:27.325 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:27.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:27.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:27.325 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:27.326 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:27.326 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:27.326 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:27.326 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:27.327 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:27.327 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:27.327 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:27.327 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:27.328 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:27.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:27.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:27.328 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:27.328 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:27.329 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:27.329 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:10:27.329 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:27.329 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:27.330 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:27.330 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:10:27.330 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:27.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:27.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:27.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:27.331 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:27.331 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:27.332 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:27.332 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:27.332 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:27.332 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:27.332 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:27.333 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:27.333 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:27.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:27.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:27.334 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:27.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:27.334 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:27.334 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:27.334 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:27.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:27.335 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:27Z" 2023-11-02T19:10:27.335 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:27.336 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:27Z" } 2023-11-02T19:10:27.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:27.337 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:27.338 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:27.338 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:10:32.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:32.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:32.341 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:32.390 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:32.391 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:32.445 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:32.442+0000 7f8668532f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:32.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:32.442+0000 7f8668532f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:32.475 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:32.475 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:32.475 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:32.475 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:32.476 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:32.476 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:32.476 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:32.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:32.476 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:32.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:32.477 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:32.477 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:32.477 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:32.477 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:32.478 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:32.478 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:32.478 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:32.478 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:32.478 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:32.479 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:32.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:32.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:32.479 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:32.480 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:32.480 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:32.480 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:32.480 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:32.480 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:32.481 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:32.481 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:32.481 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:32.481 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:32.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:32.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:32.482 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:32.482 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:32.488 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:32.488 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:10:32.488 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:32.488 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:32.489 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:32.489 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:10:32.489 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:32.489 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:32.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:32.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:32.490 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:32.490 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:32.490 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:32.491 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:32.491 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:32.491 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:32.491 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:32.491 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:32.492 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:32.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:32.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:32.492 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:32.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:32.493 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:32.493 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:32.493 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:32.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:32.493 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:32Z" 2023-11-02T19:10:32.494 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:32.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:32Z" } 2023-11-02T19:10:32.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:32.496 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:32.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:32.496 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:10:37.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:37.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:37.500 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:37.545 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:37.546 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:37.605 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:37.598+0000 7fdd7ccc5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:37.606 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:37.602+0000 7fdd7ccc5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:37.635 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:37.635 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:37.636 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:37.636 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:37.636 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:37.636 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:37.636 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:37.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:37.637 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:37.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:37.637 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:37.638 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:37.638 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:37.638 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:37.638 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:37.639 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:37.639 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:37.639 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:37.639 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:37.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:37.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:37.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:37.640 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:37.640 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:37.640 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:37.641 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:37.641 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:37.641 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:37.641 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:37.642 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:37.642 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:37.642 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:37.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:37.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:37.643 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:37.643 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:37.643 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:37.643 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:10:37.643 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:37.644 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:37.644 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:37.644 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:10:37.644 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:37.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:37.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:37.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:37.645 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:37.645 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:37.646 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:37.646 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:37.646 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:37.646 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:37.646 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:37.647 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:37.647 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:37.647 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:37.647 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:37.647 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:37.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:37.648 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:37.648 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:37.648 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:37.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:37.649 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:37Z" 2023-11-02T19:10:37.649 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:37.650 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:37Z" } 2023-11-02T19:10:37.651 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:37.651 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:37.651 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:37.652 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:10:42.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:42.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:42.654 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:42.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:42.753+0000 7f7910b70f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:42.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:42.753+0000 7f7910b70f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:42.788 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:42.788 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:42.788 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:42.789 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:42.789 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:42.789 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:42.789 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:42.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:42.790 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:42.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:42.790 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:42.790 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:42.791 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:42.791 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:42.791 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:42.791 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:42.791 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:42.792 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:42.792 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:42.792 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:42.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:42.794 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:42.794 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:42.794 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:42.794 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:42.794 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:42.795 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:42.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:42.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:42.795 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:42.795 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:42.796 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:42.796 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:10:42.796 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:42.796 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:42.796 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:42.797 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:10:42.797 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:42.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:42.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:42.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:42.798 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:42.798 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:42.798 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:42.798 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:42.799 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:42.799 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:42.799 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:42.799 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:42.799 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:42.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:42.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:42.800 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:42.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:42.800 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:42.801 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:42.801 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:42.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:42.801 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:42Z" 2023-11-02T19:10:42.801 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:42.803 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:42Z" } 2023-11-02T19:10:42.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:42.804 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:42.804 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:42.805 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:10:47.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:47.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:47.807 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:47.852 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:47.853 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:47.909 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:47.901+0000 7f0e1aca7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:47.905+0000 7f0e1aca7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:47.937 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:47.938 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:47.938 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:47.938 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:47.939 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:47.939 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:47.939 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:47.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:47.939 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:47.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:47.940 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:47.940 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:47.940 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:47.941 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:47.941 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:47.941 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:47.941 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:47.941 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:47.942 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:47.942 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:47.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:47.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:47.943 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:47.943 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:47.943 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:47.943 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:47.943 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:47.944 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:47.944 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:47.944 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:47.944 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:47.944 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:47.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:47.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:47.945 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:47.945 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:47.945 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:47.946 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952095.043328_679.1", 2023-11-02T19:10:47.946 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:47.946 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:47.946 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:47.947 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:15.043328Z", 2023-11-02T19:10:47.947 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:47.947 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:47.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:47.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:47.948 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:47.948 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:47.948 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:47.948 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:47.948 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:47.949 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:47.949 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:47.949 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:47.949 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:47.950 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:47.950 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:47.950 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:47.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:47.951 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:47.951 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:47.951 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:47.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:47.951 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:47Z" 2023-11-02T19:10:47.952 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:47.953 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952095.043328_679.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:15.043328Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:47Z" } 2023-11-02T19:10:47.953 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952095.043328_679.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:15.043328Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:47.954 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:47.954 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952095.043328_679.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:47.954 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952118.696539_697.1 target=1_1698952095.043328_679.1 2023-11-02T19:10:52.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:52.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:52.957 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:53.005 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:53.005 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:53.066 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:53.061+0000 7f7837626f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:53.068 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:53.061+0000 7f7837626f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:53.097 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:53.097 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:53.097 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:53.098 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:53.098 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:53.098 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:53.098 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:53.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.099 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:53.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.099 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:53.099 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:53.100 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:53.100 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:53.100 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:53.100 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:53.100 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:53.101 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:53.101 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:53.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.102 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:53.102 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:53.102 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:53.102 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:53.103 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:53.103 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:53.103 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:53.103 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:53.104 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:53.104 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.104 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:53.105 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:53.105 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:53.105 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952118.696539_697.1", 2023-11-02T19:10:53.105 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:53.105 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:53.106 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:53.106 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.696539Z", 2023-11-02T19:10:53.106 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:53.106 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.107 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:53.107 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:53.107 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:53.107 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:53.108 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:53.108 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:53.108 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:53.108 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:53.109 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:53.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.109 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:53.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.110 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:53.110 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:53.110 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:53.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.110 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:53Z" 2023-11-02T19:10:53.111 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:53.112 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952118.696539_697.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:38.696539Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:53Z" } 2023-11-02T19:10:53.112 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952118.696539_697.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.696539Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:53.113 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:53.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:53.113 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:10:53.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:53.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:53.115 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:53.264 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:10:53.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.264 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:10:53.265 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:10:53.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.265 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:10:53.266 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:10:53.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.266 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:10:53.266 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:10:53.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.267 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:10:53.267 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:10:53.268 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.268 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:10:53.269 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:10:53.269 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:10:53.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:53.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:53.271 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:53.318 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:53.319 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:53.406 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:53.406 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:53.407 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:53.407 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:53.407 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:53.407 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:10:53.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.408 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:53.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.408 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:53.409 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:53.409 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:10:53.409 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:10:53.410 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:53.410 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:10:53.410 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:53.410 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:10:53.411 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.411 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:53.412 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:53.412 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:10:53.412 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:10:53.412 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:53.413 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:10:53.413 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:53.413 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:10:53.413 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.414 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:53.414 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:53.415 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:10:53.415 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:10:53.415 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:53.415 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:53.415 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:53.416 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:10:53.416 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.417 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:53.417 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:53.417 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:10:53.417 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:10:53.417 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:53.418 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:53.418 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:53.418 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:10:53.418 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.419 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:53.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.419 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:53.420 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:10:53.420 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:10:53.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.420 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:53Z" 2023-11-02T19:10:53.421 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:53.422 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:10:53Z" } 2023-11-02T19:10:53.422 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:10:53.423 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:10:53.423 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:10:53.424 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:10:53.424 DEBUG:boto:path=/ibtljn-66/ 2023-11-02T19:10:53.425 DEBUG:boto:auth_path=/ibtljn-66/ 2023-11-02T19:10:53.425 DEBUG:boto:Method: HEAD 2023-11-02T19:10:53.426 DEBUG:boto:Path: /ibtljn-66/ 2023-11-02T19:10:53.426 DEBUG:boto:Data: 2023-11-02T19:10:53.427 DEBUG:boto:Headers: {} 2023-11-02T19:10:53.427 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:10:53.428 DEBUG:boto:Port: 8001 2023-11-02T19:10:53.428 DEBUG:boto:Params: {} 2023-11-02T19:10:53.429 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:10:53.430 DEBUG:boto:Token: None 2023-11-02T19:10:53.430 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:10:53 GMT /ibtljn-66/ 2023-11-02T19:10:53.431 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GsidEuvi/5inexm+ON1Jcc1DZ+I= 2023-11-02T19:10:53.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:10:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GsidEuvi/5inexm+ON1Jcc1DZ+I=', 'Content-Length': '0'} 2023-11-02T19:10:53.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a2904dc2d58aa22-006543f43d-2356-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:10:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:53.438 DEBUG:boto:path=/ibtljn-66/obj1 2023-11-02T19:10:53.438 DEBUG:boto:auth_path=/ibtljn-66/obj1 2023-11-02T19:10:53.439 DEBUG:boto:Method: HEAD 2023-11-02T19:10:53.439 DEBUG:boto:Path: /ibtljn-66/obj1 2023-11-02T19:10:53.440 DEBUG:boto:Data: 2023-11-02T19:10:53.440 DEBUG:boto:Headers: {} 2023-11-02T19:10:53.440 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:10:53.441 DEBUG:boto:Port: 8001 2023-11-02T19:10:53.441 DEBUG:boto:Params: {} 2023-11-02T19:10:53.442 DEBUG:boto:Token: None 2023-11-02T19:10:53.442 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:10:53 GMT /ibtljn-66/obj1 2023-11-02T19:10:53.443 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZRPYLP1HwIy19rfFwu9s6ezz+S0= 2023-11-02T19:10:53.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:10:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZRPYLP1HwIy19rfFwu9s6ezz+S0=', 'Content-Length': '0'} 2023-11-02T19:10:53.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-66:d61b2bea-59a8-4454-8491-ac3afbd04937.9048.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac6fb86503a9ba42-006543f43d-2356-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:10:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:53.447 DEBUG:boto:path=/ibtljn-66/obj1 2023-11-02T19:10:53.447 DEBUG:boto:auth_path=/ibtljn-66/obj1 2023-11-02T19:10:53.448 DEBUG:boto:Method: GET 2023-11-02T19:10:53.448 DEBUG:boto:Path: /ibtljn-66/obj1 2023-11-02T19:10:53.449 DEBUG:boto:Data: 2023-11-02T19:10:53.449 DEBUG:boto:Headers: {} 2023-11-02T19:10:53.450 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:10:53.450 DEBUG:boto:Port: 8001 2023-11-02T19:10:53.450 DEBUG:boto:Params: {} 2023-11-02T19:10:53.451 DEBUG:boto:Token: None 2023-11-02T19:10:53.451 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:10:53 GMT /ibtljn-66/obj1 2023-11-02T19:10:53.452 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6RrZ4VPaBel0Qfo8Ryw09kjITLE= 2023-11-02T19:10:53.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:10:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6RrZ4VPaBel0Qfo8Ryw09kjITLE=', 'Content-Length': '0'} 2023-11-02T19:10:53.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-66:d61b2bea-59a8-4454-8491-ac3afbd04937.9048.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083db4786d2fed32b-006543f43d-2356-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:10:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:53.457 DEBUG:boto:path=/ibtljn-67/ 2023-11-02T19:10:53.457 DEBUG:boto:auth_path=/ibtljn-67/ 2023-11-02T19:10:53.458 DEBUG:boto:Method: HEAD 2023-11-02T19:10:53.458 DEBUG:boto:Path: /ibtljn-67/ 2023-11-02T19:10:53.458 DEBUG:boto:Data: 2023-11-02T19:10:53.459 DEBUG:boto:Headers: {} 2023-11-02T19:10:53.459 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:53.460 DEBUG:boto:Port: 8000 2023-11-02T19:10:53.460 DEBUG:boto:Params: {} 2023-11-02T19:10:53.461 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T19:10:53.461 DEBUG:boto:Token: None 2023-11-02T19:10:53.462 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:10:53 GMT /ibtljn-67/ 2023-11-02T19:10:53.462 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:KMPOBSHN0ZNVcsRssEU9zz1H90E= 2023-11-02T19:10:53.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:10:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KMPOBSHN0ZNVcsRssEU9zz1H90E=', 'Content-Length': '0'} 2023-11-02T19:10:53.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000db5a3c2694e2d6f-006543f43d-2354-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:10:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:53.469 DEBUG:boto:path=/ibtljn-67/obj2 2023-11-02T19:10:53.469 DEBUG:boto:auth_path=/ibtljn-67/obj2 2023-11-02T19:10:53.470 DEBUG:boto:Method: HEAD 2023-11-02T19:10:53.470 DEBUG:boto:Path: /ibtljn-67/obj2 2023-11-02T19:10:53.471 DEBUG:boto:Data: 2023-11-02T19:10:53.471 DEBUG:boto:Headers: {} 2023-11-02T19:10:53.472 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:53.472 DEBUG:boto:Port: 8000 2023-11-02T19:10:53.472 DEBUG:boto:Params: {} 2023-11-02T19:10:53.473 DEBUG:boto:Token: None 2023-11-02T19:10:53.473 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:10:53 GMT /ibtljn-67/obj2 2023-11-02T19:10:53.474 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:pW4NXHE4ETilnX6pxY5ztXADPcA= 2023-11-02T19:10:53.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:10:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pW4NXHE4ETilnX6pxY5ztXADPcA=', 'Content-Length': '0'} 2023-11-02T19:10:53.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-67:d61b2bea-59a8-4454-8491-ac3afbd04937.9048.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b134ff0257d2c12e-006543f43d-2354-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:10:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:53.478 DEBUG:boto:path=/ibtljn-67/obj2 2023-11-02T19:10:53.478 DEBUG:boto:auth_path=/ibtljn-67/obj2 2023-11-02T19:10:53.479 DEBUG:boto:Method: GET 2023-11-02T19:10:53.479 DEBUG:boto:Path: /ibtljn-67/obj2 2023-11-02T19:10:53.480 DEBUG:boto:Data: 2023-11-02T19:10:53.480 DEBUG:boto:Headers: {} 2023-11-02T19:10:53.481 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:53.481 DEBUG:boto:Port: 8000 2023-11-02T19:10:53.482 DEBUG:boto:Params: {} 2023-11-02T19:10:53.482 DEBUG:boto:Token: None 2023-11-02T19:10:53.483 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:10:53 GMT /ibtljn-67/obj2 2023-11-02T19:10:53.483 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:IOGwbyMcrwOyAabZUxipMVD+abo= 2023-11-02T19:10:53.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:10:53 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:IOGwbyMcrwOyAabZUxipMVD+abo=', 'Content-Length': '0'} 2023-11-02T19:10:53.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-67:d61b2bea-59a8-4454-8491-ac3afbd04937.9048.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb1beb13bf60b16e-006543f43d-2354-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:10:53 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:53.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:53.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:53.489 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:10:53.532 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:53.532 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:53.632 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_policy":{"groups":[]}} 2023-11-02T19:10:53.633 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... ok 2023-11-02T19:10:53.634 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-11-02T19:10:53.635 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-11-02T19:10:53.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:53.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:53.637 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:53.683 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:53.683 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:53.740 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:10:53.769 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:10:53.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.770 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:53.770 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:10:53.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.771 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:53.771 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:10:53.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.771 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952118.696539_697.1", 2023-11-02T19:10:53.772 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:08:38.696539Z" 2023-11-02T19:10:53.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.772 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:53.773 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:10:53.773 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.773 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:10:53.774 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:53.775 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:10:53.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:53.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:53.776 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:53.821 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:53.822 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:53.879 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:53.873+0000 7fe969c1af40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:53.881 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:53.877+0000 7fe969c1af40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:53.910 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952118.696539_697.1", 2023-11-02T19:10:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.696539Z", 2023-11-02T19:10:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:53Z" 2023-11-02T19:10:53.926 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:53.927 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952118.696539_697.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:08:38.696539Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:53Z" } 2023-11-02T19:10:53.928 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952118.696539_697.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.696539Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:53.928 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:53.929 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952118.696539_697.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:53.929 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:10:53.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:53.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:53.931 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:10:53.972 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:53.973 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:54.070 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[]} 2023-11-02T19:10:54.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:54.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:54.071 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:10:54.114 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:54.115 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:54.210 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[]} 2023-11-02T19:10:54.211 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-68 2023-11-02T19:10:54.211 DEBUG:boto:path=/ibtljn-68/ 2023-11-02T19:10:54.212 DEBUG:boto:auth_path=/ibtljn-68/ 2023-11-02T19:10:54.212 DEBUG:boto:Method: PUT 2023-11-02T19:10:54.213 DEBUG:boto:Path: /ibtljn-68/ 2023-11-02T19:10:54.213 DEBUG:boto:Data: 2023-11-02T19:10:54.214 DEBUG:boto:Headers: {} 2023-11-02T19:10:54.214 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:54.214 DEBUG:boto:Port: 8000 2023-11-02T19:10:54.215 DEBUG:boto:Params: {} 2023-11-02T19:10:54.215 DEBUG:boto:Token: None 2023-11-02T19:10:54.216 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 19:10:54 GMT /ibtljn-68/ 2023-11-02T19:10:54.216 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qlmxOEEQIuXCA4XAthYfDHXfW+Q= 2023-11-02T19:10:54.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:10:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qlmxOEEQIuXCA4XAthYfDHXfW+Q=', 'Content-Length': '0'} 2023-11-02T19:10:54.248 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000351bcb1412f4a98d-006543f43e-2354-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:54.248 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-69 2023-11-02T19:10:54.249 DEBUG:boto:path=/ibtljn-69/ 2023-11-02T19:10:54.249 DEBUG:boto:auth_path=/ibtljn-69/ 2023-11-02T19:10:54.250 DEBUG:boto:Method: PUT 2023-11-02T19:10:54.250 DEBUG:boto:Path: /ibtljn-69/ 2023-11-02T19:10:54.251 DEBUG:boto:Data: 2023-11-02T19:10:54.251 DEBUG:boto:Headers: {} 2023-11-02T19:10:54.252 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:54.252 DEBUG:boto:Port: 8000 2023-11-02T19:10:54.253 DEBUG:boto:Params: {} 2023-11-02T19:10:54.253 DEBUG:boto:Token: None 2023-11-02T19:10:54.254 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 19:10:54 GMT /ibtljn-69/ 2023-11-02T19:10:54.254 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:YIxr0xdgKUxYBOC6/tYFKRB5okU= 2023-11-02T19:10:54.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:10:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YIxr0xdgKUxYBOC6/tYFKRB5okU=', 'Content-Length': '0'} 2023-11-02T19:10:54.286 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000174b3c9bbbc3b2a3-006543f43e-2354-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:54.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:54.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:54.288 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:54.333 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:54.333 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:54.397 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:10:54.426 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:10:54.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:54.426 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:54.427 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:10:54.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:54.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:54.427 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:54.428 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:10:54.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:54.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:54.428 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952254.281607_720.1", 2023-11-02T19:10:54.428 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:10:54.281607Z" 2023-11-02T19:10:54.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:54.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:54.429 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:54.429 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:10:54.430 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:54.430 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:10:54.431 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952254.281607_720.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:54.431 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:10:54.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:54.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:54.433 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:54.487 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:54.487 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:54.540 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:54.533+0000 7f41d1781f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:54.541 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:54.537+0000 7f41d1781f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:54.569 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:54.569 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:54.570 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:54.570 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:54.570 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:54.570 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:54.571 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:54.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:54.571 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:54.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:54.572 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:54.572 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:54.572 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:54.572 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:54.573 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:54.573 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:54.573 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:54.573 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:54.573 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:54.574 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:54.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:54.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:54.574 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:54.575 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:54.575 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:54.575 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:54.575 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:54.576 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:54.576 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:54.576 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:54.576 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:54.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:54.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:54.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:54.577 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:54.578 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:54.578 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:54.578 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952254.281607_720.1", 2023-11-02T19:10:54.579 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:54.579 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:54.579 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:54.579 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:54.281607Z", 2023-11-02T19:10:54.580 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:54.580 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:54.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:54.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:54.580 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:54.581 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:54.581 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:54.581 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:54.581 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:54.582 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:54.582 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:54.582 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:54.582 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:54.583 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:54.583 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:54.583 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:54.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:54.583 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:54Z" 2023-11-02T19:10:54.585 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:54.586 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952254.281607_720.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:54.281607Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:54Z" } 2023-11-02T19:10:54.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952254.281607_720.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:54.281607Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:54.587 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952254.281607_720.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:54.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952254.281607_720.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:54.588 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:10:54.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:54.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:54.589 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:10:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:54.636 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:54.728 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-11-02T19:10:54.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:54.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:54.729 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:10:54.779 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:54.780 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:54.880 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-11-02T19:10:54.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=ibtljn-68', '--dest-bucket=ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:54.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=ibtljn-68', '--dest-bucket=ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:54.882 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=ibtljn-68 --dest-bucket=ibtljn-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:10:54.929 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:54.930 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:55.026 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"ibtljn-68","zones":["a2","a1"]},"dest":{"bucket":"ibtljn-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:10:55.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:55.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:55.028 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:10:55.073 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:55.073 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:55.172 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"ibtljn-68","zones":["a2","a1"]},"dest":{"bucket":"ibtljn-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:10:55.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:55.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:55.174 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:10:55.213 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:55.214 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:55.306 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"ibtljn-68","zones":["a2","a1"]},"dest":{"bucket":"ibtljn-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:10:55.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:55.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:55.308 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:10:55.392 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:55.392 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:55.513 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:55.513 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:55.513 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 16, 2023-11-02T19:10:55.514 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T19:10:55.514 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T19:10:55.514 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T19:10:55.514 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:55.514 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T19:10:55.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.515 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T19:10:55.515 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T19:10:55.515 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:10:55.516 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T19:10:55.516 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:10:55.516 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:10:55.516 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:10:55.517 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:10:55.517 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:10:55.517 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:10:55.518 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:10:55.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.518 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:10:55.519 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T19:10:55.519 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:10:55.519 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T19:10:55.519 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:10:55.520 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:10:55.520 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:10:55.520 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:10:55.520 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:10:55.521 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:10:55.521 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:10:55.521 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:10:55.521 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:10:55.521 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:10:55.522 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:10:55.522 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:10:55.522 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.523 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:10:55.523 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T19:10:55.523 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:10:55.524 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:10:55.524 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:10:55.524 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:10:55.524 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:10:55.524 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:10:55.525 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:10:55.525 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:10:55.525 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:10:55.525 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:10:55.525 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:10:55.526 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:10:55.526 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:10:55.526 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:10:55.526 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.526 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:55.527 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:10:55.527 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:10:55.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.527 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:10:55.528 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:10:55.528 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:10:55.528 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:10:55.528 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.529 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:55.529 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:10:55.529 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:10:55.529 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:10:55.530 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:10:55.530 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [ 2023-11-02T19:10:55.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.530 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-group", 2023-11-02T19:10:55.531 INFO:teuthology.orchestra.run.smithi017.stdout: "data_flow": { 2023-11-02T19:10:55.531 INFO:teuthology.orchestra.run.smithi017.stdout: "symmetrical": [ 2023-11-02T19:10:55.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.531 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-flow1", 2023-11-02T19:10:55.531 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:10:55.532 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:10:55.532 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:10:55.532 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.532 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:55.533 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.533 INFO:teuthology.orchestra.run.smithi017.stdout: "pipes": [ 2023-11-02T19:10:55.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.533 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-pipe", 2023-11-02T19:10:55.534 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:10:55.534 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-68", 2023-11-02T19:10:55.534 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:10:55.534 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:10:55.535 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:10:55.535 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.535 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": { 2023-11-02T19:10:55.536 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-68", 2023-11-02T19:10:55.536 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:10:55.536 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:10:55.536 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:10:55.536 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.537 INFO:teuthology.orchestra.run.smithi017.stdout: "params": { 2023-11-02T19:10:55.537 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:10:55.537 INFO:teuthology.orchestra.run.smithi017.stdout: "filter": { 2023-11-02T19:10:55.538 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [] 2023-11-02T19:10:55.538 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:55.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.538 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": {}, 2023-11-02T19:10:55.539 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2023-11-02T19:10:55.539 INFO:teuthology.orchestra.run.smithi017.stdout: "mode": "system", 2023-11-02T19:10:55.539 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "" 2023-11-02T19:10:55.539 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:55.539 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:55.540 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:10:55.540 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "enabled" 2023-11-02T19:10:55.540 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:55.540 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.541 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:10:55.541 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:10:55.541 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.542 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T19:10:55.542 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T19:10:55.543 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:10:55.543 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T19:10:55.543 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:10:55.543 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:10:55.544 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:10:55.544 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:10:55.544 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:10:55.544 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:10:55.545 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:10:55.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.545 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:10:55.545 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T19:10:55.545 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:10:55.546 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:10:55.546 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:10:55.546 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:10:55.546 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:10:55.547 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:10:55.547 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:10:55.547 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:10:55.547 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:10:55.548 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:10:55.548 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:10:55.548 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:10:55.548 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:10:55.549 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:10:55.549 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.550 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:10:55.550 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T19:10:55.550 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:10:55.550 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T19:10:55.550 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:10:55.551 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:10:55.551 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:10:55.551 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:10:55.551 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:10:55.552 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:10:55.552 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:10:55.552 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:10:55.552 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:10:55.553 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:10:55.553 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:10:55.553 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:10:55.553 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.554 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:55.554 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:10:55.554 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:10:55.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.554 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:10:55.555 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:10:55.555 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:10:55.555 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:10:55.556 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:55.556 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:10:55.556 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:10:55.557 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:10:55.557 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:10:55.557 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T19:10:55.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.558 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:10:55.558 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:10:55.558 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.559 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:55.559 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:10:55.559 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T19:10:55.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.560 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:10:55.560 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T19:10:55.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.561 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:10:55.562 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T19:10:55.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.563 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:10:55.563 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T19:10:55.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:55.564 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:10:55.564 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T19:10:55.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:55.564 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:55.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.565 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T19:10:55.565 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:10:55.565 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T19:10:55.565 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T19:10:55.566 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:10:55.566 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:10:55.566 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:10:55.566 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:10:55.567 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:10:55.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.567 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T19:10:55.567 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:10:55.568 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:10:55.568 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:10:55.568 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:10:55.568 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:10:55.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.569 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T19:10:55.569 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:10:55.569 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:10:55.569 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:10:55.570 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:10:55.570 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:10:55.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.570 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T19:10:55.571 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:10:55.571 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:10:55.571 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:10:55.571 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:10:55.571 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:10:55.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.572 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T19:10:55.572 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:10:55.572 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:10:55.573 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:10:55.573 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:10:55.573 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:10:55.573 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:55.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:55.574 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:10:55.574 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T19:10:55.574 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:55.575 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:55.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:55.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:10:55.576 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:10:55.617 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:55.617 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:55.705 INFO:teuthology.orchestra.run.smithi017.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"ibtljn-68"},"dest":{"zone":"a1","bucket":"ibtljn-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"ibtljn-68"},"dest":{"zone":"a2","bucket":"ibtljn-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2023-11-02T19:10:55.705 DEBUG:boto:path=/ibtljn-68/ 2023-11-02T19:10:55.706 DEBUG:boto:auth_path=/ibtljn-68/ 2023-11-02T19:10:55.706 DEBUG:boto:Method: HEAD 2023-11-02T19:10:55.707 DEBUG:boto:Path: /ibtljn-68/ 2023-11-02T19:10:55.707 DEBUG:boto:Data: 2023-11-02T19:10:55.708 DEBUG:boto:Headers: {} 2023-11-02T19:10:55.708 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:55.709 DEBUG:boto:Port: 8000 2023-11-02T19:10:55.709 DEBUG:boto:Params: {} 2023-11-02T19:10:55.710 DEBUG:boto:Token: None 2023-11-02T19:10:55.710 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:10:55 GMT /ibtljn-68/ 2023-11-02T19:10:55.711 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PKT6xRivxW9D94cccm4d0xVEzmU= 2023-11-02T19:10:55.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:10:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PKT6xRivxW9D94cccm4d0xVEzmU=', 'Content-Length': '0'} 2023-11-02T19:10:55.792 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ec75949c85204cb-006543f43f-24ba-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:55.793 DEBUG:boto:path=/ibtljn-68/obj1 2023-11-02T19:10:55.793 DEBUG:boto:auth_path=/ibtljn-68/obj1 2023-11-02T19:10:55.794 DEBUG:boto:Method: PUT 2023-11-02T19:10:55.794 DEBUG:boto:Path: /ibtljn-68/obj1 2023-11-02T19:10:55.795 DEBUG:boto:Data: 2023-11-02T19:10:55.795 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:10:55.796 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:55.796 DEBUG:boto:Port: 8000 2023-11-02T19:10:55.797 DEBUG:boto:Params: {} 2023-11-02T19:10:55.797 DEBUG:boto:Token: None 2023-11-02T19:10:55.798 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:10:55 GMT /ibtljn-68/obj1 2023-11-02T19:10:55.798 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nLYRod+dyn51l00TeMQriHt4PAo= 2023-11-02T19:10:55.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:10:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nLYRod+dyn51l00TeMQriHt4PAo='} 2023-11-02T19:10:55.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c4edb2668f05ae7-006543f43f-24ba-a1'), ('Date', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:55.843 DEBUG:boto:path=/ibtljn-68/ 2023-11-02T19:10:55.843 DEBUG:boto:auth_path=/ibtljn-68/ 2023-11-02T19:10:55.844 DEBUG:boto:Method: HEAD 2023-11-02T19:10:55.844 DEBUG:boto:Path: /ibtljn-68/ 2023-11-02T19:10:55.845 DEBUG:boto:Data: 2023-11-02T19:10:55.845 DEBUG:boto:Headers: {} 2023-11-02T19:10:55.846 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:55.846 DEBUG:boto:Port: 8000 2023-11-02T19:10:55.846 DEBUG:boto:Params: {} 2023-11-02T19:10:55.847 DEBUG:boto:Token: None 2023-11-02T19:10:55.847 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:10:55 GMT /ibtljn-68/ 2023-11-02T19:10:55.848 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PKT6xRivxW9D94cccm4d0xVEzmU= 2023-11-02T19:10:55.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:10:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PKT6xRivxW9D94cccm4d0xVEzmU=', 'Content-Length': '0'} 2023-11-02T19:10:55.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f456b6c18099a532-006543f43f-24ba-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:55.853 DEBUG:boto:path=/ibtljn-68/obj2 2023-11-02T19:10:55.853 DEBUG:boto:auth_path=/ibtljn-68/obj2 2023-11-02T19:10:55.854 DEBUG:boto:Method: PUT 2023-11-02T19:10:55.854 DEBUG:boto:Path: /ibtljn-68/obj2 2023-11-02T19:10:55.855 DEBUG:boto:Data: 2023-11-02T19:10:55.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:10:55.856 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:55.856 DEBUG:boto:Port: 8000 2023-11-02T19:10:55.856 DEBUG:boto:Params: {} 2023-11-02T19:10:55.857 DEBUG:boto:Token: None 2023-11-02T19:10:55.858 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:10:55 GMT /ibtljn-68/obj2 2023-11-02T19:10:55.858 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:oHWlSGoWmGX7XxLj5UiR3rIt1LQ= 2023-11-02T19:10:55.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:10:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oHWlSGoWmGX7XxLj5UiR3rIt1LQ='} 2023-11-02T19:10:55.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a66fb6e746feed3-006543f43f-24ba-a1'), ('Date', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:55.877 DEBUG:boto:path=/ibtljn-68/ 2023-11-02T19:10:55.877 DEBUG:boto:auth_path=/ibtljn-68/ 2023-11-02T19:10:55.877 DEBUG:boto:Method: HEAD 2023-11-02T19:10:55.878 DEBUG:boto:Path: /ibtljn-68/ 2023-11-02T19:10:55.878 DEBUG:boto:Data: 2023-11-02T19:10:55.879 DEBUG:boto:Headers: {} 2023-11-02T19:10:55.879 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:55.880 DEBUG:boto:Port: 8000 2023-11-02T19:10:55.880 DEBUG:boto:Params: {} 2023-11-02T19:10:55.881 DEBUG:boto:Token: None 2023-11-02T19:10:55.881 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:10:55 GMT /ibtljn-68/ 2023-11-02T19:10:55.882 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PKT6xRivxW9D94cccm4d0xVEzmU= 2023-11-02T19:10:55.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:10:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PKT6xRivxW9D94cccm4d0xVEzmU=', 'Content-Length': '0'} 2023-11-02T19:10:55.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6d53f50d90d3715-006543f43f-24ba-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:55.886 DEBUG:boto:path=/ibtljn-68/obj3 2023-11-02T19:10:55.887 DEBUG:boto:auth_path=/ibtljn-68/obj3 2023-11-02T19:10:55.887 DEBUG:boto:Method: PUT 2023-11-02T19:10:55.887 DEBUG:boto:Path: /ibtljn-68/obj3 2023-11-02T19:10:55.888 DEBUG:boto:Data: 2023-11-02T19:10:55.888 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:10:55.889 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:55.889 DEBUG:boto:Port: 8000 2023-11-02T19:10:55.890 DEBUG:boto:Params: {} 2023-11-02T19:10:55.890 DEBUG:boto:Token: None 2023-11-02T19:10:55.891 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:10:55 GMT /ibtljn-68/obj3 2023-11-02T19:10:55.891 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WU/vmesEzgO0xoY4l+gkQlkxn2U= 2023-11-02T19:10:55.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:10:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WU/vmesEzgO0xoY4l+gkQlkxn2U='} 2023-11-02T19:10:55.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d30dafad82ddf9a0-006543f43f-24ba-a1'), ('Date', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:55.935 DEBUG:boto:path=/ibtljn-69/ 2023-11-02T19:10:55.935 DEBUG:boto:auth_path=/ibtljn-69/ 2023-11-02T19:10:55.936 DEBUG:boto:Method: HEAD 2023-11-02T19:10:55.936 DEBUG:boto:Path: /ibtljn-69/ 2023-11-02T19:10:55.936 DEBUG:boto:Data: 2023-11-02T19:10:55.937 DEBUG:boto:Headers: {} 2023-11-02T19:10:55.937 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:55.938 DEBUG:boto:Port: 8000 2023-11-02T19:10:55.938 DEBUG:boto:Params: {} 2023-11-02T19:10:55.938 DEBUG:boto:Token: None 2023-11-02T19:10:55.939 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:10:55 GMT /ibtljn-69/ 2023-11-02T19:10:55.939 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:h4M5AgsPoResty3vrMNgbI/64I4= 2023-11-02T19:10:55.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:10:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h4M5AgsPoResty3vrMNgbI/64I4=', 'Content-Length': '0'} 2023-11-02T19:10:55.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ffe5f70aa5f2e9a-006543f43f-24ba-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:55.945 DEBUG:boto:path=/ibtljn-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-11-02T19:10:55.945 DEBUG:boto:auth_path=/ibtljn-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-11-02T19:10:55.946 DEBUG:boto:Method: PUT 2023-11-02T19:10:55.946 DEBUG:boto:Path: /ibtljn-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-11-02T19:10:55.947 DEBUG:boto:Data: 2023-11-02T19:10:55.947 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:10:55.948 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:10:55.948 DEBUG:boto:Port: 8000 2023-11-02T19:10:55.948 DEBUG:boto:Params: {} 2023-11-02T19:10:55.949 DEBUG:boto:Token: None 2023-11-02T19:10:55.949 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:10:55 GMT /ibtljn-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-11-02T19:10:55.950 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:A7cxJeZtWNi//GkGbLKh9ZUZsGA= 2023-11-02T19:10:55.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:10:55 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:A7cxJeZtWNi//GkGbLKh9ZUZsGA='} 2023-11-02T19:10:55.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b850fcbc38c04e3d-006543f43f-24ba-a1'), ('Date', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:10:55.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:55.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:55.972 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:56.121 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:10:56.161 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:10:56.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.161 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:56.162 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:10:56.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.163 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:56.163 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:10:56.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.163 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952254.281607_720.1", 2023-11-02T19:10:56.164 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:10:54.281607Z" 2023-11-02T19:10:56.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.164 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:56.165 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:10:56.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:56.165 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:10:56.166 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952254.281607_720.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:56.166 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:10:56.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:56.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:56.168 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:56.218 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:56.218 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:56.280 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:56.273+0000 7f34c0037f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:56.282 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:10:56.277+0000 7f34c0037f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:10:56.307 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:10:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:10:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:10:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:10:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:10:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:10:56.314 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:56.314 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:56.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.314 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952254.281607_720.1", 2023-11-02T19:10:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:54.281607Z", 2023-11-02T19:10:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:10:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:10:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:10:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:10:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:10:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:10:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:56Z" 2023-11-02T19:10:56.321 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:56.322 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952254.281607_720.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:54.281607Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:10:56Z" } 2023-11-02T19:10:56.323 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952254.281607_720.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:54.281607Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:10:56.323 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952254.281607_720.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:56.324 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952254.281607_720.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:10:56.324 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:10:56.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:56.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:56.326 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:56.373 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:56.373 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:56.456 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:10:56.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.457 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:10:56.457 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:10:56.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.458 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:10:56.458 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:10:56.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.459 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:10:56.459 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:10:56.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.460 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:10:56.460 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:10:56.460 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:56.461 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:10:56.462 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:10:56.462 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:10:56.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:56.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:10:56.464 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:10:56.507 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:10:56.508 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:10:56.596 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:10:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:10:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:10:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:10:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:10:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:10:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:10:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:10:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:10:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:10:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:10:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:10:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:10:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:10:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:10:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:10:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:10:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:10:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:10:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:10:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:10:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:10:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:10:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:10:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:10:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:10:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:10:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:10:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:10:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:10:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:10:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:10:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:10:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:10:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:10:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:10:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:10:56Z" 2023-11-02T19:10:56.611 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:10:56.612 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:10:56Z" } 2023-11-02T19:10:56.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:10:56.613 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:10:56.613 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:10:56.614 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:11:01.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:01.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:01.617 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:11:01.666 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:11:01.667 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:11:01.747 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:11:01.747 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:11:01.747 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:11:01.748 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:11:01.748 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:11:01.748 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:11:01.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:01.749 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:11:01.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:01.749 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:11:01.749 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:01.750 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:01.750 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:11:01.750 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:01.750 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:11:01.750 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:01.751 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:11:01.751 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:01.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:01.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:01.752 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:11:01.752 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:01.752 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:01.752 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:11:01.752 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:01.753 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:11:01.753 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:01.753 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:11:01.753 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:01.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:01.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:01.754 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:11:01.754 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:01.754 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:01.755 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:11:01.755 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:01.755 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:01.755 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:01.755 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:11:01.756 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:01.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:01.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:01.756 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:11:01.757 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:01.757 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:01.757 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:11:01.757 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:01.757 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:01.758 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:01.758 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:11:01.758 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:01.758 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:01.759 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:11:01.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:01.759 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:11:01.759 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:11:01.760 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:11:01.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:01.760 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:11:01Z" 2023-11-02T19:11:01.760 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:11:01.761 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:11:01Z" } 2023-11-02T19:11:01.762 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:11:01.762 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:11:01.763 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:11:01.763 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:11:06.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:06.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:06.766 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:11:06.814 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:11:06.815 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:11:06.902 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:11:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:11:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:11:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:11:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:11:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:11:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:11:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:11:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:11:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:11:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:11:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:11:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:11:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:11:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:11:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:11:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:06.910 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:06.910 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:11:06.910 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:06.911 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:06.911 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:06.911 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:11:06.911 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:06.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:06.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:06.912 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:11:06.912 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:06.913 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:06.913 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:11:06.913 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:06.913 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:06.914 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:06.914 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:11:06.914 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:06.915 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:06.915 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:11:06.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:06.915 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:11:06.916 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:11:06.916 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:11:06.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:06.916 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:11:06Z" 2023-11-02T19:11:06.916 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:11:06.917 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:11:06Z" } 2023-11-02T19:11:06.918 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:11:06.919 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:11:06.919 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:11:06.919 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:11:11.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:11.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:11.921 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:11:11.970 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:11:11.970 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:11:12.063 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:11:12.064 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:11:12.064 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:11:12.064 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:11:12.065 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:11:12.065 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:11:12.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:11:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:11:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:11:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:11:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:11:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:11:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:11:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:11:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:11:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:11:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:11:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:11:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:11:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:11:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:11:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:11:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:11:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:11:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:11:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:11:12Z" 2023-11-02T19:11:12.080 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:11:12.081 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:11:12Z" } 2023-11-02T19:11:12.082 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:11:12.083 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:11:12.083 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:11:12.083 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:11:17.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:17.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:17.086 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:11:17.136 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:11:17.136 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:11:17.226 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:11:17.226 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:11:17.226 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:11:17.227 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:11:17.227 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:11:17.227 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:11:17.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:17.227 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:11:17.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:17.228 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:11:17.228 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:17.228 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:17.229 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:11:17.229 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:17.229 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:11:17.229 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:17.229 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:11:17.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:17.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:17.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:17.230 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:11:17.231 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:17.231 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:17.231 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:11:17.231 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:17.232 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:11:17.232 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:17.232 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:11:17.232 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:17.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:17.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:17.233 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:11:17.233 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:17.233 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:17.233 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:11:17.234 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:17.234 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:17.234 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:17.234 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:11:17.234 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:17.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:17.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:17.235 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:11:17.235 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:17.236 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:17.236 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:11:17.236 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:17.236 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:17.236 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:17.237 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:11:17.237 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:17.237 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:17.237 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:11:17.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:17.238 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:11:17.238 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:11:17.238 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:11:17.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:17.239 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:11:17Z" 2023-11-02T19:11:17.239 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:11:17.240 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:11:17Z" } 2023-11-02T19:11:17.240 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:11:17.241 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:11:17.241 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:11:17.242 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:11:22.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:22.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:22.245 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:11:22.295 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:11:22.296 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:11:22.392 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:11:22.392 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:11:22.392 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:11:22.392 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:11:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:11:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:11:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:11:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:11:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:11:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:11:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:11:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:11:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:11:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:11:22.398 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:22.398 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:11:22.398 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:22.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:11:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:11:22.400 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:22.400 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:22.400 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:22.400 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:11:22.400 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:22.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:22.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:22.401 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:11:22.401 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:22.401 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:11:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:11:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:11:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:11:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:11:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:11:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:11:22Z" 2023-11-02T19:11:22.405 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:11:22.406 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:11:22Z" } 2023-11-02T19:11:22.406 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:11:22.407 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:11:22.407 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:11:22.407 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:11:27.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:27.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:27.410 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:11:27.460 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:11:27.461 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:11:27.547 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:11:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:11:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:11:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:11:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:11:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:11:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:11:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:11:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:11:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:11:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:11:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:11:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:11:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:11:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:11:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:11:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:11:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:11:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:11:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:11:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:11:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:11:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:11:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:11:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:11:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:11:27Z" 2023-11-02T19:11:27.560 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:11:27.561 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:11:27Z" } 2023-11-02T19:11:27.562 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:11:27.562 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:11:27.563 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:11:27.564 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:11:32.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:32.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:32.567 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:11:32.617 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:11:32.617 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:11:32.711 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:11:32.711 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:11:32.712 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:11:32.712 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:11:32.712 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:11:32.712 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:11:32.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:32.713 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:11:32.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:32.713 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:11:32.713 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:32.714 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:32.714 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:11:32.714 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:32.714 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:11:32.714 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:32.715 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:11:32.715 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:32.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:32.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:32.715 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:11:32.716 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:32.716 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:32.716 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:11:32.716 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:32.716 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:11:32.717 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:32.717 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:11:32.717 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:32.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:32.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:32.718 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:11:32.718 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:32.718 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:32.719 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:11:32.719 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:32.719 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:32.719 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:32.719 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:11:32.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:32.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:32.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:32.720 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:11:32.721 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:32.721 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:32.721 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:11:32.721 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:32.721 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:32.722 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:32.722 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:11:32.722 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:32.722 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:32.722 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:11:32.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:32.723 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:11:32.723 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:11:32.723 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:11:32.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:32.724 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:11:32Z" 2023-11-02T19:11:32.724 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:11:32.725 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:11:32Z" } 2023-11-02T19:11:32.725 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:11:32.726 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:11:32.726 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:11:32.727 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:11:37.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:37.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:37.729 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:11:37.773 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:11:37.774 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:11:37.862 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:11:37.863 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:11:37.863 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:11:37.863 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:11:37.863 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:11:37.864 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:11:37.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:37.864 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:11:37.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:37.864 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:11:37.865 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:37.865 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:37.865 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:11:37.865 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:37.866 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:11:37.866 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:37.866 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:11:37.866 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:37.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:37.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:37.867 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:11:37.867 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:37.867 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:37.868 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:11:37.868 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:37.868 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:11:37.868 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:37.868 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:11:37.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:37.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:37.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:37.869 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:11:37.869 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:37.870 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:37.870 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:11:37.870 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:37.870 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:37.871 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:37.871 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:11:37.871 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:37.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:37.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:37.872 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:11:37.872 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:37.872 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:37.872 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:11:37.873 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:37.873 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:37.873 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:37.873 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:11:37.873 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:37.874 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:37.874 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:11:37.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:37.874 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:11:37.875 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:11:37.875 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:11:37.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:37.875 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:11:37Z" 2023-11-02T19:11:37.876 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:11:37.877 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:11:37Z" } 2023-11-02T19:11:37.877 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:11:37.878 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:11:37.878 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:11:37.878 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:11:42.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:42.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:42.881 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:11:42.931 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:11:42.931 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:11:43.023 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:11:43.023 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:11:43.024 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:11:43.024 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:11:43.024 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:11:43.024 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:11:43.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:43.025 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:11:43.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:43.025 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:11:43.025 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:43.026 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:43.026 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:11:43.026 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:43.026 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:11:43.026 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:43.027 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:11:43.027 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:43.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:43.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:43.028 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:11:43.028 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:43.028 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:43.028 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:11:43.028 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:43.029 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:11:43.029 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:43.029 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:11:43.029 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:43.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:43.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:43.030 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:11:43.030 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:43.030 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:43.031 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:11:43.031 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:43.031 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:43.031 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:43.031 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:11:43.032 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:43.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:43.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:43.032 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:11:43.032 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:43.033 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:43.033 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:11:43.033 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:43.033 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:43.033 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:43.034 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:11:43.034 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:43.034 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:43.034 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:11:43.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:43.035 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:11:43.035 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:11:43.035 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:11:43.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:43.035 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:11:42Z" 2023-11-02T19:11:43.036 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:11:43.037 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:11:42Z" } 2023-11-02T19:11:43.037 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:11:43.038 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:11:43.038 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:11:43.038 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:11:48.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:48.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:48.041 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:11:48.089 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:11:48.090 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:11:48.179 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:11:48.179 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:11:48.179 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:11:48.180 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:11:48.180 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:11:48.180 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:11:48.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:48.181 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:11:48.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:48.181 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:11:48.181 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:48.181 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:48.182 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:11:48.182 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:48.182 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:11:48.182 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:48.183 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:11:48.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:48.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:48.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:48.183 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:11:48.184 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:48.184 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:48.184 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:11:48.184 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:48.185 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:11:48.185 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:48.185 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:11:48.185 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:48.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:48.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:48.186 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:11:48.186 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:48.187 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:48.187 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:11:48.187 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:48.187 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:48.188 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:48.188 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:11:48.188 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:48.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:48.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:48.189 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:11:48.189 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:48.189 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:48.190 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:11:48.190 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:48.190 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:48.190 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:48.191 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:11:48.191 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:48.191 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:48.191 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:11:48.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:48.192 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:11:48.192 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:11:48.192 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:11:48.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:48.193 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:11:48Z" 2023-11-02T19:11:48.193 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:11:48.194 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:11:48Z" } 2023-11-02T19:11:48.195 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:11:48.195 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:11:48.195 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:11:48.196 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:11:53.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:53.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:53.199 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:11:53.246 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:11:53.246 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:11:53.332 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:11:53.333 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:11:53.333 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:11:53.333 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:11:53.333 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:11:53.334 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:11:53.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:53.334 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:11:53.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:53.335 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:11:53.335 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:53.335 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:53.335 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:11:53.335 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:53.336 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:11:53.336 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:53.336 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:11:53.336 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:53.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:53.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:53.337 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:11:53.337 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:53.338 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:53.338 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:11:53.338 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:53.338 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:11:53.339 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:53.339 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:11:53.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:53.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:53.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:53.340 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:11:53.340 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:53.340 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:53.340 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:11:53.341 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:53.341 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:53.341 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:53.341 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:11:53.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:53.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:53.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:53.342 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:11:53.343 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:53.343 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:53.343 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:11:53.343 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:53.344 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:53.344 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:53.344 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:11:53.344 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:53.345 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:53.345 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:11:53.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:53.345 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:11:53.345 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:11:53.346 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:11:53.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:53.346 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:11:53Z" 2023-11-02T19:11:53.346 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:11:53.348 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:11:53Z" } 2023-11-02T19:11:53.348 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:11:53.349 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:11:53.349 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:11:53.350 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:11:58.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:58.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:11:58.352 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:11:58.401 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:11:58.401 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:11:58.491 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:11:58.492 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:11:58.492 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:11:58.492 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:11:58.492 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:11:58.493 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:11:58.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:58.493 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:11:58.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:58.494 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:11:58.494 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:58.494 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:58.494 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:11:58.495 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:58.495 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:11:58.495 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:58.495 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:11:58.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:58.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:58.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:58.496 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:11:58.496 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:58.497 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:58.497 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:11:58.497 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:58.497 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:11:58.498 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:58.498 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:11:58.498 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:58.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:58.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:58.499 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:11:58.499 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:58.499 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:58.499 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:11:58.500 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:58.500 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:58.500 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:58.500 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:11:58.500 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:58.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:58.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:11:58.501 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:11:58.501 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:11:58.502 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:11:58.502 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:11:58.502 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:11:58.502 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:11:58.503 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:11:58.503 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:11:58.503 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:58.503 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:11:58.503 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:11:58.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:58.504 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:11:58.504 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:11:58.504 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:11:58.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:11:58.505 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:11:58Z" 2023-11-02T19:11:58.505 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:11:58.506 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:11:58Z" } 2023-11-02T19:11:58.507 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:11:58.507 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:11:58.507 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:11:58.508 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:12:03.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:03.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:03.510 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:03.558 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:03.558 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:03.650 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:03.651 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:12:03.651 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:12:03.652 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:12:03.652 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:12:03.652 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:12:03.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:03.652 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:12:03.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:03.653 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:12:03.653 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:03.653 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:03.654 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:12:03.654 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:03.654 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:12:03.654 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:03.655 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:12:03.655 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:03.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:03.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:03.655 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:12:03.656 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:03.656 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:03.656 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:12:03.656 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:03.657 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:12:03.657 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:03.657 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:12:03.657 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:03.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:03.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:03.658 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:12:03.658 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:03.658 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:03.659 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:12:03.659 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:03.659 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:03.659 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:03.659 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:12:03.660 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:03.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:03.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:03.660 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:12:03.661 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:03.661 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:03.661 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:12:03.661 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:03.661 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:03.662 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:03.662 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:12:03.662 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:03.662 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:03.662 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:03.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:03.663 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:12:03.663 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:12:03.663 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:12:03.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:03.664 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:12:03Z" 2023-11-02T19:12:03.664 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:03.665 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:12:03Z" } 2023-11-02T19:12:03.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:12:03.666 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:12:03.667 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:12:03.667 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:12:08.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:08.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:08.669 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:08.719 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:08.719 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:08.816 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:08.817 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:12:08.817 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:12:08.817 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:12:08.817 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:12:08.818 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:12:08.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:08.818 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:12:08.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:08.819 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:12:08.819 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:08.819 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:08.819 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:12:08.820 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:08.820 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:12:08.820 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:08.820 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:12:08.821 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:08.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:08.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:08.822 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:12:08.822 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:08.823 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:08.823 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:12:08.823 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:08.823 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:12:08.824 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:08.824 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:12:08.824 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:08.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:08.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:08.825 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:12:08.826 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:08.826 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:08.826 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:12:08.826 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:08.827 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:08.827 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:08.827 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:12:08.827 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:08.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:08.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:08.828 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:12:08.829 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:08.829 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:08.830 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:12:08.830 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:08.831 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:08.831 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:08.832 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:12:08.832 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:08.833 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:08.833 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:08.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:08.834 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:12:08.834 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:12:08.835 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:12:08.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:08.835 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:12:08Z" 2023-11-02T19:12:08.836 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:08.838 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:12:08Z" } 2023-11-02T19:12:08.839 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:12:08.840 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:12:08.840 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:12:08.841 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:12:13.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:13.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:13.844 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:13.891 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:13.892 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:13.982 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:13.982 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:12:13.982 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:12:13.983 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:12:13.983 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:12:13.983 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:12:13.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:13.984 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:12:13.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:13.984 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:12:13.984 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:13.985 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:13.985 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:12:13.985 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:13.985 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:12:13.986 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:13.986 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:12:13.986 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:13.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:13.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:13.987 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:12:13.987 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:13.987 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:13.988 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:12:13.988 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:13.988 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:12:13.988 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:13.989 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:12:13.989 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:13.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:13.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:13.990 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:12:13.990 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:13.990 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:13.990 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:12:13.991 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:13.991 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:13.991 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:13.991 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:12:13.991 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:13.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:13.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:13.992 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:12:13.992 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:13.993 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:13.993 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:12:13.993 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:13.993 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:13.994 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:13.994 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:12:13.994 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:13.994 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:13.995 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:13.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:13.995 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:12:13.995 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:12:13.995 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:12:13.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:13.996 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:12:13Z" 2023-11-02T19:12:13.996 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:13.997 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:12:13Z" } 2023-11-02T19:12:13.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:12:13.998 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:12:13.999 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:12:13.999 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:12:19.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:19.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:19.002 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:19.052 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:19.052 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:19.144 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:19.144 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:12:19.145 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:12:19.145 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:12:19.145 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:12:19.145 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:12:19.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:19.146 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:12:19.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:19.146 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:12:19.147 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:19.147 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:19.147 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:12:19.147 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:19.147 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:12:19.148 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:19.148 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:12:19.148 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:19.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:19.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:19.149 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:12:19.149 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:19.149 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:19.149 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:12:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:12:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:12:19.151 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:19.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:19.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:19.151 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:12:19.152 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:19.152 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:19.152 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:12:19.152 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:12:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:12:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:12:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:12:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:12:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:12:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:12:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:12:19Z" 2023-11-02T19:12:19.158 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:19.159 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:12:19Z" } 2023-11-02T19:12:19.160 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:12:19.161 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:12:19.161 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:12:19.162 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:12:24.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:24.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:24.164 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:24.211 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:24.211 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:24.303 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:24.304 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:12:24.304 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:12:24.304 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:12:24.305 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:12:24.305 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:12:24.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:24.305 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:12:24.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:24.306 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:12:24.306 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:24.306 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:24.306 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:12:24.307 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:24.307 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:12:24.307 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:24.307 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:12:24.308 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:24.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:24.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:24.308 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:12:24.308 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:24.309 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:24.309 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050687", 2023-11-02T19:12:24.309 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:24.309 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:12:24.309 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:24.310 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:08:38.651238Z" 2023-11-02T19:12:24.310 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:24.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:24.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:24.311 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:12:24.311 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:24.311 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:24.311 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-11-02T19:12:24.311 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:24.312 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:24.312 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:24.312 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:44.792797Z" 2023-11-02T19:12:24.312 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:24.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:24.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:24.313 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:12:24.313 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:24.313 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:24.313 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045516", 2023-11-02T19:12:24.314 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:24.314 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:24.314 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:24.314 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:09.716986Z" 2023-11-02T19:12:24.315 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:24.315 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:24.315 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:24.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:24.315 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:12:24.316 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:12:24.316 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:12:24.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:24.316 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:12:24Z" 2023-11-02T19:12:24.316 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:24.318 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050687", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:08:38.651238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:03:44.792797Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045516", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:09.716986Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:12:24Z" } 2023-11-02T19:12:24.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050687', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:08:38.651238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:03:44.792797Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045516', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:09.716986Z'}}] 2023-11-02T19:12:24.319 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:12:24.319 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050687', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000045516'} 2023-11-02T19:12:24.319 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050967 target=00000000000000000000:00000000000000050687, shard=2 master=00000000000000000000:00000000000000044747 target=00000000000000000000:00000000000000044605, shard=3 master=00000000000000000000:00000000000000045656 target=00000000000000000000:00000000000000045516 2023-11-02T19:12:29.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:29.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:29.322 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:29.456 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:29.456 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:12:29.456 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:12:29.456 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:12:29.457 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:12:29.457 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:12:29.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:29.457 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:12:29.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:29.458 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:12:29.458 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:29.458 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:29.458 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:12:29.459 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:29.459 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:12:29.459 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:29.459 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:12:29.460 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:29.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:29.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:29.460 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:12:29.460 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:29.461 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:29.461 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:12:29.461 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:29.461 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:12:29.461 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:29.462 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:12:29.462 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:29.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:29.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:29.462 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:12:29.463 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:29.463 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:29.463 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:12:29.463 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:29.463 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:12:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:12:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:12:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:12:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:12:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:12:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:12:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:12:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:12:29Z" 2023-11-02T19:12:29.468 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:29.469 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:12:29Z" } 2023-11-02T19:12:29.470 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:12:29.470 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:12:29.471 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:12:29.471 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:12:29.472 DEBUG:boto:path=/ibtljn-68/ 2023-11-02T19:12:29.472 DEBUG:boto:auth_path=/ibtljn-68/ 2023-11-02T19:12:29.473 DEBUG:boto:Method: HEAD 2023-11-02T19:12:29.473 DEBUG:boto:Path: /ibtljn-68/ 2023-11-02T19:12:29.473 DEBUG:boto:Data: 2023-11-02T19:12:29.474 DEBUG:boto:Headers: {} 2023-11-02T19:12:29.474 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:12:29.475 DEBUG:boto:Port: 8001 2023-11-02T19:12:29.475 DEBUG:boto:Params: {} 2023-11-02T19:12:29.476 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:12:29.476 DEBUG:boto:Token: None 2023-11-02T19:12:29.477 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:12:29 GMT /ibtljn-68/ 2023-11-02T19:12:29.477 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S9/ryGZoidHTBGmt9qDAX+kjJd0= 2023-11-02T19:12:29.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S9/ryGZoidHTBGmt9qDAX+kjJd0=', 'Content-Length': '0'} 2023-11-02T19:12:29.488 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ed8ceb9dc775e3e-006543f49d-24be-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:29.489 DEBUG:boto:path=/ibtljn-68/obj1 2023-11-02T19:12:29.489 DEBUG:boto:auth_path=/ibtljn-68/obj1 2023-11-02T19:12:29.490 DEBUG:boto:Method: HEAD 2023-11-02T19:12:29.490 DEBUG:boto:Path: /ibtljn-68/obj1 2023-11-02T19:12:29.491 DEBUG:boto:Data: 2023-11-02T19:12:29.491 DEBUG:boto:Headers: {} 2023-11-02T19:12:29.491 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:12:29.492 DEBUG:boto:Port: 8001 2023-11-02T19:12:29.492 DEBUG:boto:Params: {} 2023-11-02T19:12:29.493 DEBUG:boto:Token: None 2023-11-02T19:12:29.493 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:12:29 GMT /ibtljn-68/obj1 2023-11-02T19:12:29.494 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E2BfuICogo+j2Jrc8VXZjpL/5ys= 2023-11-02T19:12:29.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E2BfuICogo+j2Jrc8VXZjpL/5ys=', 'Content-Length': '0'} 2023-11-02T19:12:29.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-68:d61b2bea-59a8-4454-8491-ac3afbd04937.9048.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097933eab29a44718-006543f49d-24be-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:29.498 DEBUG:boto:path=/ibtljn-68/obj1 2023-11-02T19:12:29.499 DEBUG:boto:auth_path=/ibtljn-68/obj1 2023-11-02T19:12:29.499 DEBUG:boto:Method: GET 2023-11-02T19:12:29.500 DEBUG:boto:Path: /ibtljn-68/obj1 2023-11-02T19:12:29.500 DEBUG:boto:Data: 2023-11-02T19:12:29.500 DEBUG:boto:Headers: {} 2023-11-02T19:12:29.501 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:12:29.501 DEBUG:boto:Port: 8001 2023-11-02T19:12:29.502 DEBUG:boto:Params: {} 2023-11-02T19:12:29.502 DEBUG:boto:Token: None 2023-11-02T19:12:29.503 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:12:29 GMT /ibtljn-68/obj1 2023-11-02T19:12:29.503 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:txm+I5Thu7QF0GiX5aHpW+rwXqA= 2023-11-02T19:12:29.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:txm+I5Thu7QF0GiX5aHpW+rwXqA=', 'Content-Length': '0'} 2023-11-02T19:12:29.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-68:d61b2bea-59a8-4454-8491-ac3afbd04937.9048.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2eb6d96f1255b7c-006543f49d-24be-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:29.508 DEBUG:boto:path=/ibtljn-68/obj2 2023-11-02T19:12:29.508 DEBUG:boto:auth_path=/ibtljn-68/obj2 2023-11-02T19:12:29.509 DEBUG:boto:Method: HEAD 2023-11-02T19:12:29.509 DEBUG:boto:Path: /ibtljn-68/obj2 2023-11-02T19:12:29.510 DEBUG:boto:Data: 2023-11-02T19:12:29.510 DEBUG:boto:Headers: {} 2023-11-02T19:12:29.511 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:12:29.511 DEBUG:boto:Port: 8001 2023-11-02T19:12:29.512 DEBUG:boto:Params: {} 2023-11-02T19:12:29.512 DEBUG:boto:Token: None 2023-11-02T19:12:29.513 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:12:29 GMT /ibtljn-68/obj2 2023-11-02T19:12:29.513 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8G3WIH0CgaZjWdExo8ycLd4gQPM= 2023-11-02T19:12:29.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8G3WIH0CgaZjWdExo8ycLd4gQPM=', 'Content-Length': '0'} 2023-11-02T19:12:29.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-68:d61b2bea-59a8-4454-8491-ac3afbd04937.9048.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c77f5071d11101d-006543f49d-24be-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:29.517 DEBUG:boto:path=/ibtljn-68/obj2 2023-11-02T19:12:29.517 DEBUG:boto:auth_path=/ibtljn-68/obj2 2023-11-02T19:12:29.518 DEBUG:boto:Method: GET 2023-11-02T19:12:29.518 DEBUG:boto:Path: /ibtljn-68/obj2 2023-11-02T19:12:29.519 DEBUG:boto:Data: 2023-11-02T19:12:29.519 DEBUG:boto:Headers: {} 2023-11-02T19:12:29.520 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:12:29.520 DEBUG:boto:Port: 8001 2023-11-02T19:12:29.520 DEBUG:boto:Params: {} 2023-11-02T19:12:29.521 DEBUG:boto:Token: None 2023-11-02T19:12:29.521 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:12:29 GMT /ibtljn-68/obj2 2023-11-02T19:12:29.522 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:m4FqvRStkN88s3Xwvzle2+dhi4Q= 2023-11-02T19:12:29.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m4FqvRStkN88s3Xwvzle2+dhi4Q=', 'Content-Length': '0'} 2023-11-02T19:12:29.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-68:d61b2bea-59a8-4454-8491-ac3afbd04937.9048.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b2075c973a98cd0-006543f49d-24be-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:29.526 DEBUG:boto:path=/ibtljn-68/obj3 2023-11-02T19:12:29.526 DEBUG:boto:auth_path=/ibtljn-68/obj3 2023-11-02T19:12:29.527 DEBUG:boto:Method: HEAD 2023-11-02T19:12:29.527 DEBUG:boto:Path: /ibtljn-68/obj3 2023-11-02T19:12:29.528 DEBUG:boto:Data: 2023-11-02T19:12:29.528 DEBUG:boto:Headers: {} 2023-11-02T19:12:29.528 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:12:29.529 DEBUG:boto:Port: 8001 2023-11-02T19:12:29.529 DEBUG:boto:Params: {} 2023-11-02T19:12:29.530 DEBUG:boto:Token: None 2023-11-02T19:12:29.530 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:12:29 GMT /ibtljn-68/obj3 2023-11-02T19:12:29.531 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:dCTycmU4JEpJXMILkAFx+ez6Gwo= 2023-11-02T19:12:29.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dCTycmU4JEpJXMILkAFx+ez6Gwo=', 'Content-Length': '0'} 2023-11-02T19:12:29.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-68:d61b2bea-59a8-4454-8491-ac3afbd04937.9048.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e255948537d4f9f-006543f49d-24be-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:29.535 DEBUG:boto:path=/ibtljn-68/obj3 2023-11-02T19:12:29.535 DEBUG:boto:auth_path=/ibtljn-68/obj3 2023-11-02T19:12:29.535 DEBUG:boto:Method: GET 2023-11-02T19:12:29.536 DEBUG:boto:Path: /ibtljn-68/obj3 2023-11-02T19:12:29.536 DEBUG:boto:Data: 2023-11-02T19:12:29.537 DEBUG:boto:Headers: {} 2023-11-02T19:12:29.537 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:12:29.538 DEBUG:boto:Port: 8001 2023-11-02T19:12:29.538 DEBUG:boto:Params: {} 2023-11-02T19:12:29.539 DEBUG:boto:Token: None 2023-11-02T19:12:29.539 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:12:29 GMT /ibtljn-68/obj3 2023-11-02T19:12:29.540 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ARGL4Z1vkHyz5+2NLrPQKSNq2LM= 2023-11-02T19:12:29.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ARGL4Z1vkHyz5+2NLrPQKSNq2LM=', 'Content-Length': '0'} 2023-11-02T19:12:29.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:10:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-68:d61b2bea-59a8-4454-8491-ac3afbd04937.9048.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a0d57363b7d2a29-006543f49d-24be-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:29.544 DEBUG:boto:path=/ibtljn-69/ 2023-11-02T19:12:29.545 DEBUG:boto:auth_path=/ibtljn-69/ 2023-11-02T19:12:29.545 DEBUG:boto:Method: HEAD 2023-11-02T19:12:29.545 DEBUG:boto:Path: /ibtljn-69/ 2023-11-02T19:12:29.546 DEBUG:boto:Data: 2023-11-02T19:12:29.546 DEBUG:boto:Headers: {} 2023-11-02T19:12:29.547 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:12:29.547 DEBUG:boto:Port: 8001 2023-11-02T19:12:29.548 DEBUG:boto:Params: {} 2023-11-02T19:12:29.548 DEBUG:boto:Token: None 2023-11-02T19:12:29.549 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:12:29 GMT /ibtljn-69/ 2023-11-02T19:12:29.549 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RTX+MtdYH0DdbAiBEn8ZJUuukJ4= 2023-11-02T19:12:29.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RTX+MtdYH0DdbAiBEn8ZJUuukJ4=', 'Content-Length': '0'} 2023-11-02T19:12:29.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037f5602bbd250d6a-006543f49d-24be-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:29.555 DEBUG:boto:path=/ibtljn-69/obj1 2023-11-02T19:12:29.555 DEBUG:boto:auth_path=/ibtljn-69/obj1 2023-11-02T19:12:29.556 DEBUG:boto:Method: HEAD 2023-11-02T19:12:29.556 DEBUG:boto:Path: /ibtljn-69/obj1 2023-11-02T19:12:29.557 DEBUG:boto:Data: 2023-11-02T19:12:29.557 DEBUG:boto:Headers: {} 2023-11-02T19:12:29.558 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:12:29.558 DEBUG:boto:Port: 8001 2023-11-02T19:12:29.558 DEBUG:boto:Params: {} 2023-11-02T19:12:29.559 DEBUG:boto:Token: None 2023-11-02T19:12:29.559 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:12:29 GMT /ibtljn-69/obj1 2023-11-02T19:12:29.560 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Uo9S9KR2cKcHMzFoGec8Kb8CWa8= 2023-11-02T19:12:29.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Uo9S9KR2cKcHMzFoGec8Kb8CWa8=', 'Content-Length': '0'} 2023-11-02T19:12:29.563 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c4a55787f3993f82-006543f49d-24be-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:29.564 DEBUG:boto:path=/ibtljn-69/obj2 2023-11-02T19:12:29.564 DEBUG:boto:auth_path=/ibtljn-69/obj2 2023-11-02T19:12:29.565 DEBUG:boto:Method: HEAD 2023-11-02T19:12:29.565 DEBUG:boto:Path: /ibtljn-69/obj2 2023-11-02T19:12:29.566 DEBUG:boto:Data: 2023-11-02T19:12:29.566 DEBUG:boto:Headers: {} 2023-11-02T19:12:29.566 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:12:29.567 DEBUG:boto:Port: 8001 2023-11-02T19:12:29.567 DEBUG:boto:Params: {} 2023-11-02T19:12:29.568 DEBUG:boto:Token: None 2023-11-02T19:12:29.568 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:12:29 GMT /ibtljn-69/obj2 2023-11-02T19:12:29.569 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:3REpB1Rg0o7UqsEQZnHHh0nRuVY= 2023-11-02T19:12:29.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3REpB1Rg0o7UqsEQZnHHh0nRuVY=', 'Content-Length': '0'} 2023-11-02T19:12:29.572 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000025a7c0e53a339b1d-006543f49d-24be-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:29.572 DEBUG:boto:path=/ibtljn-69/obj3 2023-11-02T19:12:29.573 DEBUG:boto:auth_path=/ibtljn-69/obj3 2023-11-02T19:12:29.573 DEBUG:boto:Method: HEAD 2023-11-02T19:12:29.573 DEBUG:boto:Path: /ibtljn-69/obj3 2023-11-02T19:12:29.574 DEBUG:boto:Data: 2023-11-02T19:12:29.574 DEBUG:boto:Headers: {} 2023-11-02T19:12:29.575 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:12:29.575 DEBUG:boto:Port: 8001 2023-11-02T19:12:29.575 DEBUG:boto:Params: {} 2023-11-02T19:12:29.576 DEBUG:boto:Token: None 2023-11-02T19:12:29.576 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:12:29 GMT /ibtljn-69/obj3 2023-11-02T19:12:29.577 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ifPjvDJTbfu9UTR4LNPus1y8E3E= 2023-11-02T19:12:29.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:29 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ifPjvDJTbfu9UTR4LNPus1y8E3E=', 'Content-Length': '0'} 2023-11-02T19:12:29.580 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000099bde9051f259555-006543f49d-24be-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:29.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:29.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:29.581 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:12:29.629 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:29.629 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:29.727 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-11-02T19:12:29.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:29.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:29.729 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:12:29.776 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:29.776 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:29.875 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:12:29.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:29.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:29.876 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:12:29.920 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:29.921 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:30.016 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:12:30.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:30.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:30.018 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:12:30.062 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:30.062 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:30.181 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:30.181 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:12:30.181 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 17, 2023-11-02T19:12:30.182 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T19:12:30.182 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T19:12:30.182 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T19:12:30.182 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:12:30.183 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T19:12:30.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.183 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T19:12:30.183 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T19:12:30.183 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:12:30.184 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T19:12:30.184 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:12:30.184 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:12:30.184 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:12:30.184 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:12:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:12:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:12:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:12:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:12:30.186 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T19:12:30.186 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:12:30.186 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T19:12:30.186 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:12:30.187 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:12:30.187 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:12:30.187 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:12:30.187 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:12:30.188 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:12:30.188 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:12:30.188 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:12:30.188 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:12:30.188 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:12:30.189 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:12:30.189 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:12:30.189 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.190 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:12:30.190 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T19:12:30.190 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:12:30.190 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:12:30.191 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:12:30.191 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:12:30.191 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:12:30.191 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:12:30.192 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:12:30.192 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:12:30.192 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:12:30.192 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:12:30.192 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:12:30.193 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:12:30.193 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:12:30.193 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:12:30.193 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:30.194 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:12:30.194 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:12:30.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.195 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:12:30.195 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:12:30.195 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:12:30.195 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:12:30.196 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.196 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:30.196 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:12:30.196 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:12:30.196 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:12:30.197 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:12:30.197 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [ 2023-11-02T19:12:30.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.197 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-group", 2023-11-02T19:12:30.198 INFO:teuthology.orchestra.run.smithi017.stdout: "data_flow": { 2023-11-02T19:12:30.198 INFO:teuthology.orchestra.run.smithi017.stdout: "symmetrical": [ 2023-11-02T19:12:30.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.198 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-flow1", 2023-11-02T19:12:30.199 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:12:30.199 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:12:30.199 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:12:30.199 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:30.200 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.200 INFO:teuthology.orchestra.run.smithi017.stdout: "pipes": [ 2023-11-02T19:12:30.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.201 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-pipe", 2023-11-02T19:12:30.201 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:12:30.201 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "*", 2023-11-02T19:12:30.201 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:12:30.201 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:12:30.202 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:12:30.202 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.202 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": { 2023-11-02T19:12:30.203 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "*", 2023-11-02T19:12:30.203 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:12:30.203 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:12:30.203 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:12:30.203 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.204 INFO:teuthology.orchestra.run.smithi017.stdout: "params": { 2023-11-02T19:12:30.204 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:12:30.204 INFO:teuthology.orchestra.run.smithi017.stdout: "filter": { 2023-11-02T19:12:30.204 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [] 2023-11-02T19:12:30.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:30.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.205 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": {}, 2023-11-02T19:12:30.205 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2023-11-02T19:12:30.206 INFO:teuthology.orchestra.run.smithi017.stdout: "mode": "system", 2023-11-02T19:12:30.206 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "" 2023-11-02T19:12:30.206 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:30.206 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:30.206 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:12:30.207 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "allowed" 2023-11-02T19:12:30.207 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:30.207 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.207 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:12:30.208 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:12:30.208 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.209 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T19:12:30.209 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T19:12:30.209 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:12:30.209 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T19:12:30.209 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:12:30.210 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:12:30.210 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:12:30.210 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:12:30.210 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:12:30.211 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:12:30.211 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:12:30.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.211 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:12:30.211 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T19:12:30.212 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:12:30.212 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:12:30.212 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:12:30.212 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:12:30.213 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:12:30.213 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:12:30.213 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:12:30.213 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:12:30.214 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:12:30.214 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:12:30.214 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:12:30.214 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:12:30.215 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:12:30.215 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:12:30.215 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.216 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:12:30.216 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T19:12:30.216 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:12:30.216 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T19:12:30.217 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:12:30.217 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:12:30.217 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:12:30.217 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:12:30.217 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:12:30.218 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:12:30.218 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:12:30.218 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:12:30.218 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:12:30.218 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:12:30.219 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:12:30.219 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:12:30.219 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:30.220 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:12:30.220 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:12:30.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.220 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:12:30.220 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:12:30.221 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:12:30.221 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:12:30.221 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.221 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:30.222 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:12:30.222 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:12:30.222 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:12:30.222 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:12:30.222 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T19:12:30.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.223 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:12:30.223 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:12:30.223 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:30.224 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:12:30.224 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T19:12:30.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.224 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:12:30.225 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T19:12:30.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.225 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:12:30.225 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T19:12:30.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.227 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:12:30.227 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T19:12:30.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:30.228 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:12:30.228 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T19:12:30.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:30.229 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:30.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.229 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T19:12:30.229 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:12:30.230 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T19:12:30.230 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T19:12:30.230 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:12:30.230 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:12:30.230 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:12:30.231 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:12:30.231 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:12:30.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.231 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T19:12:30.231 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:12:30.232 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:12:30.232 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:12:30.232 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:12:30.232 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:12:30.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.233 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T19:12:30.233 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:12:30.233 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:12:30.233 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:12:30.234 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:12:30.234 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:12:30.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.234 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T19:12:30.234 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:12:30.235 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:12:30.235 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:12:30.235 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:12:30.235 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:12:30.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.236 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T19:12:30.236 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:12:30.236 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:12:30.236 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:12:30.237 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:12:30.237 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:12:30.237 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:30.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:30.237 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:12:30.238 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T19:12:30.238 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:30.238 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:30.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:30.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:30.240 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:12:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:30.373 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:12:30.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:30.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:30.374 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket ibtljn-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:12:30.484 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:30.484 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:30.592 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-11-02T19:12:30.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:30.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:30.593 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket ibtljn-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:12:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:30.734 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-11-02T19:12:30.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:30.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:30.736 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=ibtljn-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:12:30.789 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:30.789 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:30.884 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-11-02T19:12:30.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:30.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:30.886 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket ibtljn-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:12:30.931 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:30.931 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:31.024 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-11-02T19:12:31.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:31.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:12:31.026 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket ibtljn-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:12:31.069 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:31.069 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:31.166 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-11-02T19:12:31.167 DEBUG:boto:path=/ibtljn-68/ 2023-11-02T19:12:31.167 DEBUG:boto:auth_path=/ibtljn-68/ 2023-11-02T19:12:31.168 DEBUG:boto:Method: HEAD 2023-11-02T19:12:31.168 DEBUG:boto:Path: /ibtljn-68/ 2023-11-02T19:12:31.169 DEBUG:boto:Data: 2023-11-02T19:12:31.169 DEBUG:boto:Headers: {} 2023-11-02T19:12:31.170 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:12:31.170 DEBUG:boto:Port: 8000 2023-11-02T19:12:31.170 DEBUG:boto:Params: {} 2023-11-02T19:12:31.171 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T19:12:31.172 DEBUG:boto:Token: None 2023-11-02T19:12:31.172 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:12:31 GMT /ibtljn-68/ 2023-11-02T19:12:31.173 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:q78qPr/i5qhwFTqbcnrCFqRyn6A= 2023-11-02T19:12:31.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:q78qPr/i5qhwFTqbcnrCFqRyn6A=', 'Content-Length': '0'} 2023-11-02T19:12:31.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077ca16b48a0ab0ca-006543f49f-2593-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:12:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:31.184 DEBUG:boto:path=/ibtljn-68/obj3 2023-11-02T19:12:31.184 DEBUG:boto:auth_path=/ibtljn-68/obj3 2023-11-02T19:12:31.185 DEBUG:boto:Method: PUT 2023-11-02T19:12:31.185 DEBUG:boto:Path: /ibtljn-68/obj3 2023-11-02T19:12:31.186 DEBUG:boto:Data: 2023-11-02T19:12:31.186 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:12:31.187 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:12:31.187 DEBUG:boto:Port: 8000 2023-11-02T19:12:31.187 DEBUG:boto:Params: {} 2023-11-02T19:12:31.188 DEBUG:boto:Token: None 2023-11-02T19:12:31.188 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:12:31 GMT /ibtljn-68/obj3 2023-11-02T19:12:31.189 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NcA7WC4QE/XvV/HLLvgkLr9XXok= 2023-11-02T19:12:31.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:12:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NcA7WC4QE/XvV/HLLvgkLr9XXok='} 2023-11-02T19:12:31.211 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000268b5f1e75f6c526-006543f49f-2593-a1'), ('Date', 'Thu, 02 Nov 2023 19:12:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:31.212 DEBUG:boto:path=/ibtljn-69/ 2023-11-02T19:12:31.213 DEBUG:boto:auth_path=/ibtljn-69/ 2023-11-02T19:12:31.213 DEBUG:boto:Method: HEAD 2023-11-02T19:12:31.214 DEBUG:boto:Path: /ibtljn-69/ 2023-11-02T19:12:31.214 DEBUG:boto:Data: 2023-11-02T19:12:31.214 DEBUG:boto:Headers: {} 2023-11-02T19:12:31.215 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:12:31.215 DEBUG:boto:Port: 8000 2023-11-02T19:12:31.216 DEBUG:boto:Params: {} 2023-11-02T19:12:31.216 DEBUG:boto:Token: None 2023-11-02T19:12:31.217 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:12:31 GMT /ibtljn-69/ 2023-11-02T19:12:31.217 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qMRUY7ln6J0uoT56Z3OGcVStblk= 2023-11-02T19:12:31.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:12:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qMRUY7ln6J0uoT56Z3OGcVStblk=', 'Content-Length': '0'} 2023-11-02T19:12:31.224 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a8a96baf7318f84-006543f49f-2593-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:12:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:31.225 DEBUG:boto:path=/ibtljn-69/obj3 2023-11-02T19:12:31.225 DEBUG:boto:auth_path=/ibtljn-69/obj3 2023-11-02T19:12:31.225 DEBUG:boto:Method: PUT 2023-11-02T19:12:31.226 DEBUG:boto:Path: /ibtljn-69/obj3 2023-11-02T19:12:31.226 DEBUG:boto:Data: 2023-11-02T19:12:31.227 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:12:31.227 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:12:31.228 DEBUG:boto:Port: 8000 2023-11-02T19:12:31.228 DEBUG:boto:Params: {} 2023-11-02T19:12:31.229 DEBUG:boto:Token: None 2023-11-02T19:12:31.229 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:12:31 GMT /ibtljn-69/obj3 2023-11-02T19:12:31.230 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:PhCcgI0I8AVcmZqadnARGRuLZPY= 2023-11-02T19:12:31.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:12:31 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PhCcgI0I8AVcmZqadnARGRuLZPY='} 2023-11-02T19:12:31.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035903f974c3da288-006543f49f-2593-a1'), ('Date', 'Thu, 02 Nov 2023 19:12:31 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:12:31.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:31.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:31.276 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:31.322 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:31.322 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:31.385 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:12:31.412 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:12:31.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:31.413 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:12:31.413 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:12:31.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:31.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:31.414 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:12:31.414 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:12:31.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:31.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:31.415 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952350.991131_748.1", 2023-11-02T19:12:31.415 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:12:30.991131Z" 2023-11-02T19:12:31.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:31.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:31.416 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:12:31.416 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:12:31.416 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:31.416 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:12:31.417 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952350.991131_748.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:31.418 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:12:31.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:31.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:31.419 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:31.465 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:31.465 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:31.521 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:12:31.516+0000 7f60a6a82f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:12:31.523 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:12:31.516+0000 7f60a6a82f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:12:31.552 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:31.552 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:12:31.553 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:12:31.553 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:12:31.553 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:12:31.553 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:12:31.554 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:31.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:31.554 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:12:31.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:31.555 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:12:31.555 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:31.555 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:31.555 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:12:31.556 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:31.556 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:12:31.556 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:31.556 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:12:31.556 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:31.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:31.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:31.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:31.557 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:12:31.558 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:31.558 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:31.558 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:12:31.558 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:31.559 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:31.559 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:31.559 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:12:31.559 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:31.560 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:31.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:31.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:31.560 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:12:31.560 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:31.561 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:31.561 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952254.281607_720.1", 2023-11-02T19:12:31.561 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:31.561 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:31.562 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:31.562 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:54.281607Z", 2023-11-02T19:12:31.562 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:31.562 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:31.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:31.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:31.563 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:12:31.563 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:31.564 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:31.564 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:12:31.564 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:31.565 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:31.565 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:31.565 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:12:31.565 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:31.566 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:31.566 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:31.566 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:31.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:31.567 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:12:31.567 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:12:31.567 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:12:31.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:31.568 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:12:31Z" 2023-11-02T19:12:31.568 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:31.569 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952254.281607_720.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:54.281607Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:12:31Z" } 2023-11-02T19:12:31.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952254.281607_720.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:54.281607Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:12:31.570 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952350.991131_748.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:31.571 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952254.281607_720.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:31.571 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952350.991131_748.1 target=1_1698952254.281607_720.1 2023-11-02T19:12:36.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:36.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:36.573 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:36.682 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:12:36.676+0000 7f978f057f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:12:36.684 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:12:36.676+0000 7f978f057f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:12:36.711 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:36.712 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:12:36.712 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:12:36.712 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:12:36.712 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:12:36.712 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:12:36.713 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:36.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:36.713 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:12:36.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:36.714 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:12:36.714 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:36.714 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:36.714 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:12:36.715 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:36.715 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:12:36.715 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:36.715 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:12:36.715 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:36.716 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:36.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:36.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:36.716 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:12:36.717 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:36.717 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:36.717 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:12:36.717 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:36.718 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:36.718 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:36.718 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:12:36.718 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:36.719 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:36.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:36.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:36.719 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:12:36.720 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:36.720 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:36.720 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952254.281607_720.1", 2023-11-02T19:12:36.720 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:36.720 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:36.721 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:36.721 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:54.281607Z", 2023-11-02T19:12:36.721 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:36.721 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:36.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:36.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:36.722 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:12:36.722 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:36.723 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:36.723 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:12:36.723 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:36.723 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:36.723 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:36.723 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:12:36.724 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:36.724 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:36.724 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:36.724 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:36.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:36.725 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:12:36.725 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:12:36.725 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:12:36.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:36.725 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:12:36Z" 2023-11-02T19:12:36.726 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:36.727 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952254.281607_720.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:54.281607Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:12:36Z" } 2023-11-02T19:12:36.727 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952254.281607_720.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:54.281607Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:12:36.728 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952350.991131_748.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:36.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952254.281607_720.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:36.728 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952350.991131_748.1 target=1_1698952254.281607_720.1 2023-11-02T19:12:41.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:41.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:41.731 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:41.773 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:41.774 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:12:41.840+0000 7ff8f2b5df40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:12:41.840+0000 7ff8f2b5df40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:12:41.876 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:41.876 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:12:41.876 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:12:41.876 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:12:41.877 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:12:41.877 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:12:41.877 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:41.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:41.877 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:12:41.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:41.878 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:12:41.878 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:41.878 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:41.879 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:12:41.879 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:41.879 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:12:41.879 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:41.879 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:12:41.880 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:41.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:41.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:41.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:41.880 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:12:41.881 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:41.881 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:41.881 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:12:41.881 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:41.881 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:41.882 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:41.882 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:12:41.882 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:41.882 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:41.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:41.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:41.883 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:12:41.883 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:41.883 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:41.884 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952254.281607_720.1", 2023-11-02T19:12:41.884 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:41.884 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:41.884 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:41.884 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:54.281607Z", 2023-11-02T19:12:41.885 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:41.885 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:41.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:41.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:41.886 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:12:41.886 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:41.886 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:41.886 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:12:41.886 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:41.887 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:41.887 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:41.887 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:12:41.887 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:41.887 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:41.888 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:41.888 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:41.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:41.888 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:12:41.888 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:12:41.889 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:12:41.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:41.889 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:12:41Z" 2023-11-02T19:12:41.889 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:41.890 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952254.281607_720.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:54.281607Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:12:41Z" } 2023-11-02T19:12:41.891 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952254.281607_720.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:54.281607Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:12:41.891 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952350.991131_748.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:41.892 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952254.281607_720.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:41.892 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952350.991131_748.1 target=1_1698952254.281607_720.1 2023-11-02T19:12:46.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:46.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:46.894 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:46.943 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:46.943 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:46.998 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:12:46.992+0000 7f490d0bef40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:12:47.000 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:12:46.992+0000 7f490d0bef40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:12:47.027 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:47.028 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:12:47.028 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:12:47.028 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:12:47.029 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:12:47.029 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:12:47.029 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:47.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:47.030 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:12:47.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:47.030 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:12:47.030 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:47.031 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:47.031 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:12:47.031 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:47.031 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:12:47.031 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:47.032 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:12:47.032 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:47.032 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:47.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:47.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:47.033 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:12:47.033 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:47.033 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:47.033 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:12:47.034 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:47.034 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:47.034 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:47.034 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:12:47.035 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:47.035 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:47.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:47.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:47.035 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:12:47.036 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:47.036 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:47.036 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952254.281607_720.1", 2023-11-02T19:12:47.036 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:47.036 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:47.037 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:47.037 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:54.281607Z", 2023-11-02T19:12:47.037 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:47.037 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:47.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:47.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:47.038 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:12:47.038 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:47.039 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:47.039 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:12:47.039 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:47.039 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:47.040 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:47.040 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:12:47.040 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:47.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:47.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:47.041 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:47.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:47.041 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:12:47.041 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:12:47.041 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:12:47.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:47.042 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:12:47Z" 2023-11-02T19:12:47.042 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:47.043 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952254.281607_720.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:54.281607Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:12:47Z" } 2023-11-02T19:12:47.044 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952254.281607_720.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:54.281607Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:12:47.044 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952350.991131_748.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:47.045 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952254.281607_720.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:47.045 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952350.991131_748.1 target=1_1698952254.281607_720.1 2023-11-02T19:12:52.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:52.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:52.048 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:52.095 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:52.096 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:52.153 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:12:52.148+0000 7f1885601f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:12:52.154 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:12:52.148+0000 7f1885601f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:12:52.183 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:12:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:12:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:12:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:12:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:12:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:12:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:12:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:12:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:12:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:12:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:12:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:12:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:12:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:12:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952254.281607_720.1", 2023-11-02T19:12:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:54.281607Z", 2023-11-02T19:12:52.193 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:52.193 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:52.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:52.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:52.194 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:12:52.194 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:52.194 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:52.194 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:12:52.194 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:52.195 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:52.195 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:52.195 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:12:52.195 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:52.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:52.196 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:52.196 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:52.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:52.196 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:12:52.196 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:12:52.197 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:12:52.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:52.197 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:12:52Z" 2023-11-02T19:12:52.197 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:52.198 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952254.281607_720.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:54.281607Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:12:52Z" } 2023-11-02T19:12:52.199 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952254.281607_720.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:54.281607Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:12:52.199 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952350.991131_748.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:52.200 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952254.281607_720.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:52.200 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952350.991131_748.1 target=1_1698952254.281607_720.1 2023-11-02T19:12:57.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:57.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:12:57.202 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:12:57.248 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:12:57.248 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:12:57.304 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:12:57.296+0000 7f369aeecf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:12:57.306 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:12:57.300+0000 7f369aeecf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:12:57.334 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:12:57.334 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:12:57.334 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:12:57.335 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:12:57.335 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:12:57.335 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:12:57.336 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:57.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:57.336 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:12:57.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:57.337 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:12:57.337 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:57.337 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:57.337 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:12:57.338 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:57.338 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:12:57.338 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:57.338 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:12:57.339 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:57.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:57.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:57.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:57.340 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:12:57.340 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:57.340 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:57.340 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:12:57.341 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:57.341 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:57.341 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:57.341 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:12:57.342 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:57.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:57.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:57.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:57.343 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:12:57.343 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:57.343 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:57.343 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952254.281607_720.1", 2023-11-02T19:12:57.345 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:57.345 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:57.345 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:57.346 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:54.281607Z", 2023-11-02T19:12:57.346 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:57.346 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:57.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:57.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:12:57.347 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:12:57.347 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:12:57.347 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:12:57.348 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:12:57.348 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:12:57.348 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:12:57.348 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:12:57.349 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:12:57.349 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:12:57.349 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:57.349 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:12:57.350 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:12:57.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:57.350 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:12:57.350 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:12:57.350 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:12:57.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:12:57.351 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:12:57Z" 2023-11-02T19:12:57.351 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:12:57.352 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952254.281607_720.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:54.281607Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:12:57Z" } 2023-11-02T19:12:57.353 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952254.281607_720.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:54.281607Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:12:57.353 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952350.991131_748.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:57.354 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952254.281607_720.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:12:57.354 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952350.991131_748.1 target=1_1698952254.281607_720.1 2023-11-02T19:13:02.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:02.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:02.357 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:02.404 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:02.404 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:02.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:13:02.456+0000 7f908cf5bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:13:02.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:13:02.460+0000 7f908cf5bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:13:02.499 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:02.499 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:02.500 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:02.500 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:02.500 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:02.500 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:13:02.501 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:13:02.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.501 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:02.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:02.501 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:02.502 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:02.502 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:13:02.502 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:13:02.502 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:02.503 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:13:02.503 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:02.503 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:13:02.503 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:13:02.503 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:02.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:02.504 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:02.504 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:02.504 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:13:02.505 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:13:02.505 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:02.505 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:02.505 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:02.506 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:13:02.506 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:13:02.506 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:02.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:02.507 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:02.507 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:02.507 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:13:02.507 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952350.991131_748.1", 2023-11-02T19:13:02.508 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:02.508 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:02.508 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:02.508 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:30.991131Z", 2023-11-02T19:13:02.509 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:13:02.509 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:02.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:02.510 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:02.510 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:02.510 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:13:02.510 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:13:02.510 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:02.511 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:02.511 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:02.511 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:13:02.512 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:13:02.512 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:02.512 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:02.513 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:02.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.514 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:02.514 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:13:02.514 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:13:02.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.515 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:02Z" 2023-11-02T19:13:02.515 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:02.516 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952350.991131_748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:12:30.991131Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:13:02Z" } 2023-11-02T19:13:02.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952350.991131_748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:12:30.991131Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:13:02.518 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952350.991131_748.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:13:02.518 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952350.991131_748.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:13:02.518 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:13:02.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:02.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:02.521 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:02.567 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:02.567 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:02.672 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:13:02.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:02.673 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:02.673 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:02.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:02.675 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:13:02.675 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:13:02.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:02.676 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:02.676 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:02.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:02.678 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:02.678 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:02.678 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:02.678 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:13:02.680 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:02.680 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:13:02.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:02.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:02.681 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:02.735 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:02.735 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:02.823 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:02.824 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:02.824 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:02.824 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:02.824 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:02.824 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:13:02.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.825 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:02.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:02.825 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:02.826 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:02.826 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:02.826 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:02.826 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:02.827 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:13:02.827 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:02.827 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:02.827 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:02.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:02.828 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:02.828 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:02.829 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:02.829 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:13:02.829 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:02.829 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:13:02.829 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:02.830 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:13:02.830 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:02.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:02.831 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:02.831 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:02.831 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:02.831 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:02.831 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:02.832 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:02.832 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:02.832 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:02.832 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:02.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:02.833 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:02.833 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:02.833 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:02.834 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:02.834 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:02.834 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:02.834 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:02.835 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:02.835 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:02.835 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:02.835 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:02.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.836 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:02.836 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:13:02.836 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:13:02.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:02.837 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:02Z" 2023-11-02T19:13:02.837 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:02.838 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:13:02Z" } 2023-11-02T19:13:02.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:13:02.839 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:02.839 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:02.840 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:13:07.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:07.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:07.842 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:07.890 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:07.890 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:07.977 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:07.977 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:07.977 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:07.978 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:07.978 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:07.978 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:13:07.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:07.978 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:07.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:07.979 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:07.979 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:07.979 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:07.980 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:07.980 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:07.980 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:13:07.980 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:07.980 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:07.981 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:07.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:07.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:07.981 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:07.981 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:07.982 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:07.982 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:13:07.982 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:07.982 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:13:07.983 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:07.983 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:13:07.983 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:07.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:07.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:07.984 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:07.984 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:07.984 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:07.984 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:07.984 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:07.985 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:07.985 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:07.985 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:07.985 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:07.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:07.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:07.986 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:07.986 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:07.986 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:07.987 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:07.987 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:07.987 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:07.987 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:07.987 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:07.988 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:07.988 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:07.988 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:07.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:07.988 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:07.989 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:13:07.989 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:13:07.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:07.989 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:07Z" 2023-11-02T19:13:07.990 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:07.991 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:13:07Z" } 2023-11-02T19:13:07.991 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:13:07.992 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:07.992 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:07.992 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:13:12.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:12.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:12.995 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:13.045 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:13.045 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:13.133 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:13.134 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:13.134 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:13.134 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:13.134 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:13:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:13:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:13:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:13:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:13:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:13:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:13:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:13Z" 2023-11-02T19:13:13.147 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:13.148 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:13:13Z" } 2023-11-02T19:13:13.149 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:13:13.150 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:13.150 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:13.150 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:13:18.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:18.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:18.153 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:18.203 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:18.204 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:18.294 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:13:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:13:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:18.299 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:18.299 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:18.299 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:18.299 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:13:18.299 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:13:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:13:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:13:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:13:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:18Z" 2023-11-02T19:13:18.307 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:18.308 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:13:18Z" } 2023-11-02T19:13:18.309 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:13:18.309 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:18.310 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:18.310 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:13:23.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:23.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:23.312 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:23.361 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:23.361 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:23.456 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:23.456 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:23.457 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:23.457 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:23.457 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:23.457 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:13:23.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:23.458 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:23.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:23.458 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:23.458 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:23.458 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:23.459 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:23.459 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:23.459 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:13:23.459 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:23.460 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:23.460 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:23.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:23.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:23.460 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:23.461 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:23.461 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:23.461 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:13:23.461 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:23.462 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:13:23.462 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:23.462 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:13:23.462 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:23.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:23.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:23.463 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:23.463 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:23.463 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:23.464 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:23.464 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:23.464 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:23.464 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:23.464 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:23.465 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:23.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:23.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:23.465 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:23.465 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:23.466 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:23.466 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:23.466 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:23.466 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:23.467 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:23.467 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:23.467 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:23.467 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:23.467 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:23.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:23.468 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:23.468 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:13:23.468 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:13:23.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:23.469 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:23Z" 2023-11-02T19:13:23.469 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:23.470 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:13:23Z" } 2023-11-02T19:13:23.470 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:13:23.471 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:23.471 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:23.472 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:13:28.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:28.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:28.474 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:28.522 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:28.523 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:28.613 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:13:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:13:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:13:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:13:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:13:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:28.621 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:28.621 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:28.621 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:28.621 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:28.622 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:28.622 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:28.622 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:28.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:28.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:28.623 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:28.623 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:28.623 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:28.623 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:28.624 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:28.624 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:28.624 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:28.624 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:28.625 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:28.625 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:28.625 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:28.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:28.625 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:28.626 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:13:28.626 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:13:28.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:28.626 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:28Z" 2023-11-02T19:13:28.627 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:28.628 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:13:28Z" } 2023-11-02T19:13:28.628 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:13:28.629 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:28.629 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:28.630 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:13:33.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:33.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:33.632 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:33.681 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:33.681 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:33.766 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:33.767 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:33.767 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:33.767 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:33.768 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:33.768 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:13:33.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:33.768 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:33.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:33.770 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:33.770 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:33.770 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:33.770 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:33.771 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:33.771 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:13:33.771 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:33.771 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:33.772 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:33.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:33.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:33.772 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:33.772 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:33.773 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:33.773 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:13:33.773 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:33.773 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:13:33.773 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:33.774 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:13:33.774 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:33.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:33.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:33.775 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:33.775 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:33.775 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:33.775 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:33.775 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:33.776 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:33.776 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:33.776 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:33.776 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:33.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:33.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:33.777 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:33.777 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:33.777 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:33.778 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:33.778 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:33.778 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:33.778 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:33.779 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:33.779 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:33.779 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:33.779 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:33.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:33.780 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:33.780 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:13:33.780 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:13:33.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:33.780 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:33Z" 2023-11-02T19:13:33.781 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:33.782 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:13:33Z" } 2023-11-02T19:13:33.782 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:13:33.783 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:33.783 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:33.784 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:13:38.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:38.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:38.786 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:38.834 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:38.835 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:38.927 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:13:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:13:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:13:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:13:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:13:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:13:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:13:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:38Z" 2023-11-02T19:13:38.941 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:38.942 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:13:38Z" } 2023-11-02T19:13:38.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:13:38.943 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:38.943 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:38.944 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:13:43.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:43.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:43.946 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:43.994 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:43.994 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:44.084 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:44.085 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:44.085 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:44.085 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:44.085 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:44.086 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:13:44.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:44.086 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:44.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:44.086 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:44.087 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:44.087 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:44.087 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:44.087 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:44.088 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:13:44.088 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:44.088 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:44.088 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:44.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:44.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:44.089 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:44.089 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:44.090 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:44.090 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:13:44.090 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:44.090 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:13:44.091 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:44.091 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:13:44.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:44.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:44.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:44.092 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:44.092 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:44.092 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:44.093 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:44.093 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:44.093 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:44.093 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:44.094 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:44.094 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:44.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:44.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:44.094 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:44.095 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:44.095 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:44.095 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:44.095 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:44.096 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:44.096 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:44.096 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:44.096 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:44.097 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:44.097 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:44.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:44.097 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:44.097 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:13:44.098 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:13:44.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:44.098 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:44Z" 2023-11-02T19:13:44.098 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:44.099 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:13:44Z" } 2023-11-02T19:13:44.100 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:13:44.101 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:44.101 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:44.101 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:13:49.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:49.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:49.104 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:49.153 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:49.154 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:49.245 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:49.246 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:49.246 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:49.246 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:49.247 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:49.247 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:13:49.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:49.247 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:49.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:49.248 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:49.248 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:49.248 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:49.248 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:49.249 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:49.249 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:13:49.249 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:49.249 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:49.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:49.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:49.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:49.250 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:49.250 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:49.251 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:49.251 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:13:49.251 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:49.251 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:13:49.252 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:49.252 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:13:49.252 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:49.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:49.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:49.253 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:49.253 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:49.253 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:49.253 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:49.253 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:49.254 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:49.254 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:49.254 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:49.254 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:49.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:49.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:49.255 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:49.255 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:49.255 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:49.256 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:49.256 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:49.256 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:49.256 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:49.257 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:49.257 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:49.257 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:49.257 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:49.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:49.258 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:49.258 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:13:49.258 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:13:49.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:49.259 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:49Z" 2023-11-02T19:13:49.259 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:49.260 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:13:49Z" } 2023-11-02T19:13:49.261 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:13:49.261 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:49.262 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:49.262 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:13:54.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:54.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:54.265 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:54.314 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:54.315 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:54.405 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:54.405 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:54.406 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:54.406 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:54.406 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:54.407 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:13:54.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:54.407 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:54.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:54.408 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:54.408 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:54.409 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:54.409 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:54.410 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:54.410 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:13:54.410 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:54.411 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:54.411 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:54.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:54.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:54.413 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:54.413 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:54.413 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:54.414 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:13:54.414 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:54.415 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:13:54.415 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:54.415 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:13:54.415 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:54.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:54.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:54.416 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:54.417 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:54.417 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:54.417 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:54.417 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:54.418 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:54.418 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:54.419 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:54.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:54.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:54.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:54.420 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:54.421 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:54.421 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:54.421 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:54.422 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:54.422 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:54.422 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:54.423 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:54.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:54.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:54.424 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:54.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:54.424 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:54.424 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:13:54.424 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:13:54.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:54.425 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:54Z" 2023-11-02T19:13:54.425 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:54.427 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:13:54Z" } 2023-11-02T19:13:54.427 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:13:54.428 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:54.429 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:54.430 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:13:59.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:59.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:13:59.432 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:13:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:13:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:13:59.572 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:13:59.572 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:13:59.573 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:13:59.573 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:13:59.573 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:13:59.574 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:13:59.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:59.574 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:13:59.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:13:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:13:59.576 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:59.576 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:13:59.576 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:59.576 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:13:59.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:59.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:59.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:59.577 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:13:59.578 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:59.578 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:59.578 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:13:59.578 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:59.578 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:13:59.579 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:59.579 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:13:59.579 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:59.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:59.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:59.580 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:13:59.580 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:59.580 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:59.581 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:13:59.581 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:59.581 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:59.581 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:59.581 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:13:59.582 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:59.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:59.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:13:59.582 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:13:59.583 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:13:59.583 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:13:59.583 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:13:59.583 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:13:59.584 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:13:59.584 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:13:59.584 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:13:59.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:59.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:13:59.585 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:13:59.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:59.585 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:13:59.585 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:13:59.586 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:13:59.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:13:59.586 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:13:59Z" 2023-11-02T19:13:59.586 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:13:59.587 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:13:59Z" } 2023-11-02T19:13:59.588 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:13:59.588 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:59.589 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:13:59.589 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:14:04.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:04.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:04.595 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:04.643 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:04.644 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:04.727 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:04.727 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:04.728 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:04.728 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:04.728 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:04.728 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:14:04.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:04.729 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:04.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:04.729 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:04.729 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:04.730 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:04.730 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:14:04.730 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:04.730 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:14:04.731 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:04.731 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:14:04.731 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:04.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:04.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:04.733 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:04.733 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:04.733 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:04.734 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:14:04.734 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:04.734 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:14:04.734 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:04.735 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:14:04.735 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:04.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:04.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:04.736 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:04.736 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:04.736 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:04.736 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:14:04.737 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:04.737 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:04.737 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:04.737 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:14:04.738 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:04.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:04.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:04.738 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:04.739 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:04.739 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:04.739 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:14:04.739 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:04.739 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:04.740 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:04.740 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:14:04.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:04.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:04.740 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:04.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:04.741 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:04.741 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:14:04.741 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:14:04.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:04.742 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:04Z" 2023-11-02T19:14:04.742 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:04.743 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:14:04Z" } 2023-11-02T19:14:04.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:14:04.744 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:04.745 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:04.745 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:14:09.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:09.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:09.748 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:09.795 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:09.795 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:09.887 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:14:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:14:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:14:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:14:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:14:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:14:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:14:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:14:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:14:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:14:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:14:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:14:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:14:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:09Z" 2023-11-02T19:14:09.901 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:09.902 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:14:09Z" } 2023-11-02T19:14:09.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:14:09.903 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:09.903 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:09.904 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:14:14.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:14.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:14.907 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:14.954 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:14.954 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:15.043 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:15.043 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:15.044 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:15.044 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:15.044 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:15.044 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:14:15.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:15.045 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:15.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:15.045 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:15.045 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:15.046 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:15.046 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:14:15.046 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:15.046 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:14:15.047 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:15.047 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:14:15.047 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:15.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:15.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:15.049 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:15.049 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:15.049 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:15.049 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:14:15.050 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:15.050 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:14:15.050 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:15.050 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:14:15.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:15.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:15.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:15.051 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:15.052 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:15.052 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:15.052 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:14:15.052 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:15.052 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:15.053 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:15.053 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:14:15.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:15.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:15.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:15.054 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:15.054 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:15.054 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:15.057 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:14:15.057 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:15.057 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:15.058 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:15.058 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:14:15.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:15.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:15.059 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:15.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:15.059 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:15.059 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:14:15.060 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:14:15.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:15.060 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:15Z" 2023-11-02T19:14:15.060 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:15.062 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:14:15Z" } 2023-11-02T19:14:15.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:14:15.063 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:15.063 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:15.063 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:14:20.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:20.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:20.066 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:20.114 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:20.114 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:20.208 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:20.209 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:20.209 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:20.209 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:20.209 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:20.210 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:14:20.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:20.210 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:20.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:20.210 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:20.211 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:20.211 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:20.211 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:14:20.211 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:20.211 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:14:20.212 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:20.212 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:14:20.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:20.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:20.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:20.213 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:20.213 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:20.213 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:20.213 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:14:20.214 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:20.214 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:14:20.214 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:20.214 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:14:20.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:20.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:20.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:20.215 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:20.215 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:20.215 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:20.216 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:14:20.216 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:20.216 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:20.216 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:20.216 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:14:20.217 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:20.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:20.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:20.217 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:20.218 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:20.218 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:20.218 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:14:20.218 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:20.218 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:20.219 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:20.219 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:14:20.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:20.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:20.219 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:20.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:20.220 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:20.220 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:14:20.220 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:14:20.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:20.221 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:20Z" 2023-11-02T19:14:20.221 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:20.222 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:14:20Z" } 2023-11-02T19:14:20.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:14:20.223 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:20.223 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:20.224 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:14:25.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:25.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:25.227 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:25.274 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:25.274 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:25.364 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:25.365 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:25.365 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:25.365 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:25.366 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:25.366 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:14:25.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:25.366 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:25.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:25.367 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:25.367 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:25.367 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:25.367 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:14:25.368 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:25.368 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:14:25.368 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:25.368 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:14:25.369 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:25.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:25.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:25.369 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:25.369 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:25.370 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:25.370 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:14:25.370 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:25.370 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:14:25.371 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:25.371 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:14:25.371 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:25.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:25.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:25.372 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:25.372 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:25.372 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:25.373 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:14:25.373 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:25.373 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:25.373 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:25.374 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:14:25.374 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:25.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:25.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:25.375 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:25.375 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:25.375 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:25.375 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:14:25.376 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:25.376 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:25.376 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:25.376 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:14:25.376 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:25.377 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:25.377 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:25.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:25.377 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:25.378 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:14:25.378 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:14:25.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:25.378 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:25Z" 2023-11-02T19:14:25.379 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:25.380 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:14:25Z" } 2023-11-02T19:14:25.380 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:14:25.381 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:25.381 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:25.382 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:14:30.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:30.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:30.384 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:30.435 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:30.436 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:30.529 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:30.529 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:30.529 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:30.529 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:30.530 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:30.530 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:14:30.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:30.530 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:30.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:30.531 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:30.531 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:30.531 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:30.532 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:14:30.532 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:30.532 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:14:30.532 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:30.533 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:14:30.533 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:30.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:30.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:30.534 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:30.534 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:30.534 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:30.534 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000050967", 2023-11-02T19:14:30.535 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:30.535 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:14:30.535 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:30.535 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.900251Z" 2023-11-02T19:14:30.536 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:30.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:30.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:30.536 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:30.537 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:30.537 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:30.537 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:14:30.537 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:30.537 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:30.538 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:30.538 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:14:30.538 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:30.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:30.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:30.539 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:30.539 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:30.539 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:30.540 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:14:30.540 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:30.540 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:30.540 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:30.540 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:14:30.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:30.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:30.541 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:30.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:30.542 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:30.542 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:14:30.542 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:14:30.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:30.543 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:30Z" 2023-11-02T19:14:30.543 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:30.544 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050967", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:10:55.900251Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:14:30Z" } 2023-11-02T19:14:30.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050967', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.900251Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:14:30.545 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:30.546 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000050967', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:30.546 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051247 target=00000000000000000000:00000000000000050967 2023-11-02T19:14:35.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:35.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:35.549 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:35.597 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:35.598 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:35.689 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:35.690 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:35.690 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:35.690 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:35.691 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:35.691 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:14:35.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:35.691 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:35.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:35.692 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:35.692 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:35.692 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:35.693 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:14:35.693 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:35.693 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:14:35.693 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:35.694 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:14:35.694 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:35.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:35.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:35.695 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:35.695 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:35.695 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:35.695 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:14:35.695 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:35.696 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:14:35.696 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:35.696 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:14:35.697 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:35.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:35.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:35.698 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:35.698 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:35.698 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:35.698 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:14:35.699 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:35.699 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:35.699 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:35.699 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:14:35.700 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:35.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:35.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:35.700 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:35.700 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:35.701 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:35.701 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:14:35.701 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:35.701 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:35.701 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:35.702 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:14:35.702 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:35.702 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:35.702 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:35.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:35.703 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:35.703 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:14:35.703 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:14:35.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:35.704 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:35Z" 2023-11-02T19:14:35.704 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:35.705 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:14:35Z" } 2023-11-02T19:14:35.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:14:35.706 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:35.707 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:35.707 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:14:35.708 DEBUG:boto:path=/ibtljn-68/ 2023-11-02T19:14:35.708 DEBUG:boto:auth_path=/ibtljn-68/ 2023-11-02T19:14:35.709 DEBUG:boto:Method: HEAD 2023-11-02T19:14:35.709 DEBUG:boto:Path: /ibtljn-68/ 2023-11-02T19:14:35.710 DEBUG:boto:Data: 2023-11-02T19:14:35.710 DEBUG:boto:Headers: {} 2023-11-02T19:14:35.710 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:14:35.711 DEBUG:boto:Port: 8001 2023-11-02T19:14:35.711 DEBUG:boto:Params: {} 2023-11-02T19:14:35.712 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:14:35.713 DEBUG:boto:Token: None 2023-11-02T19:14:35.713 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:14:35 GMT /ibtljn-68/ 2023-11-02T19:14:35.714 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:hVqQ2QmPbRhAuEFoVajAmAlrQz4= 2023-11-02T19:14:35.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:14:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hVqQ2QmPbRhAuEFoVajAmAlrQz4=', 'Content-Length': '0'} 2023-11-02T19:14:35.721 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000566587b06dc9e9bc-006543f51b-259c-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:14:35.721 DEBUG:boto:path=/ibtljn-68/obj3 2023-11-02T19:14:35.722 DEBUG:boto:auth_path=/ibtljn-68/obj3 2023-11-02T19:14:35.722 DEBUG:boto:Method: HEAD 2023-11-02T19:14:35.723 DEBUG:boto:Path: /ibtljn-68/obj3 2023-11-02T19:14:35.723 DEBUG:boto:Data: 2023-11-02T19:14:35.723 DEBUG:boto:Headers: {} 2023-11-02T19:14:35.724 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:14:35.724 DEBUG:boto:Port: 8001 2023-11-02T19:14:35.725 DEBUG:boto:Params: {} 2023-11-02T19:14:35.725 DEBUG:boto:Token: None 2023-11-02T19:14:35.726 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:14:35 GMT /ibtljn-68/obj3 2023-11-02T19:14:35.726 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uXHP0dfPmymX27GMifA5VvvF+Fw= 2023-11-02T19:14:35.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:14:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uXHP0dfPmymX27GMifA5VvvF+Fw=', 'Content-Length': '0'} 2023-11-02T19:14:35.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:12:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-68:d61b2bea-59a8-4454-8491-ac3afbd04937.9048.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033ef091654bfdc92-006543f51b-259c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:14:35.730 DEBUG:boto:path=/ibtljn-68/obj3 2023-11-02T19:14:35.730 DEBUG:boto:auth_path=/ibtljn-68/obj3 2023-11-02T19:14:35.730 DEBUG:boto:Method: GET 2023-11-02T19:14:35.731 DEBUG:boto:Path: /ibtljn-68/obj3 2023-11-02T19:14:35.731 DEBUG:boto:Data: 2023-11-02T19:14:35.732 DEBUG:boto:Headers: {} 2023-11-02T19:14:35.732 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:14:35.733 DEBUG:boto:Port: 8001 2023-11-02T19:14:35.733 DEBUG:boto:Params: {} 2023-11-02T19:14:35.733 DEBUG:boto:Token: None 2023-11-02T19:14:35.734 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:14:35 GMT /ibtljn-68/obj3 2023-11-02T19:14:35.735 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:g7rZy+7K0ymVtHlFsrhGxiDgeDk= 2023-11-02T19:14:35.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:14:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:g7rZy+7K0ymVtHlFsrhGxiDgeDk=', 'Content-Length': '0'} 2023-11-02T19:14:35.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:12:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-68:d61b2bea-59a8-4454-8491-ac3afbd04937.9048.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005501ab2ced14dfcc-006543f51b-259c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:14:35.738 DEBUG:boto:path=/ibtljn-69/ 2023-11-02T19:14:35.739 DEBUG:boto:auth_path=/ibtljn-69/ 2023-11-02T19:14:35.739 DEBUG:boto:Method: HEAD 2023-11-02T19:14:35.740 DEBUG:boto:Path: /ibtljn-69/ 2023-11-02T19:14:35.740 DEBUG:boto:Data: 2023-11-02T19:14:35.740 DEBUG:boto:Headers: {} 2023-11-02T19:14:35.741 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:14:35.741 DEBUG:boto:Port: 8001 2023-11-02T19:14:35.742 DEBUG:boto:Params: {} 2023-11-02T19:14:35.742 DEBUG:boto:Token: None 2023-11-02T19:14:35.743 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:14:35 GMT /ibtljn-69/ 2023-11-02T19:14:35.743 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UGrP7CE4Jq0NjLC1TsL+5DmN7Uw= 2023-11-02T19:14:35.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:14:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UGrP7CE4Jq0NjLC1TsL+5DmN7Uw=', 'Content-Length': '0'} 2023-11-02T19:14:35.748 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d607190d7ff6d4a-006543f51b-259c-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:14:35.748 DEBUG:boto:path=/ibtljn-69/obj3 2023-11-02T19:14:35.749 DEBUG:boto:auth_path=/ibtljn-69/obj3 2023-11-02T19:14:35.749 DEBUG:boto:Method: HEAD 2023-11-02T19:14:35.750 DEBUG:boto:Path: /ibtljn-69/obj3 2023-11-02T19:14:35.750 DEBUG:boto:Data: 2023-11-02T19:14:35.750 DEBUG:boto:Headers: {} 2023-11-02T19:14:35.751 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:14:35.751 DEBUG:boto:Port: 8001 2023-11-02T19:14:35.752 DEBUG:boto:Params: {} 2023-11-02T19:14:35.752 DEBUG:boto:Token: None 2023-11-02T19:14:35.753 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:14:35 GMT /ibtljn-69/obj3 2023-11-02T19:14:35.753 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2SdLsLPI7Y4d6DMosQQQZE1qebQ= 2023-11-02T19:14:35.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:14:35 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2SdLsLPI7Y4d6DMosQQQZE1qebQ=', 'Content-Length': '0'} 2023-11-02T19:14:35.756 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000000f2417d2589feb87-006543f51b-259c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:14:35.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:35.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:35.757 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket ibtljn-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:14:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:35.912 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_policy":{"groups":[]}} 2023-11-02T19:14:35.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:35.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:35.913 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:14:35.996 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:35.997 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:36.096 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_policy":{"groups":[]}} 2023-11-02T19:14:36.097 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ok 2023-11-02T19:14:36.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:36.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:36.099 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:36.143 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:36.143 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:14:36.232 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:14:36.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:36.233 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:14:36.233 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:14:36.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:36.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:36.234 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:14:36.234 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:14:36.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:36.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:36.234 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952475.877925_751.1", 2023-11-02T19:14:36.235 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:14:35.877925Z" 2023-11-02T19:14:36.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:36.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:36.235 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:14:36.235 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:14:36.236 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:36.236 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:14:36.237 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952475.877925_751.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:36.237 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:14:36.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:36.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:36.239 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:36.325 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:36.326 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:36.384 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:14:36.379+0000 7f65a43b2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:14:36.386 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:14:36.383+0000 7f65a43b2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:14:36.415 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:14:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:14:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:14:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:14:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:14:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:14:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952350.991131_748.1", 2023-11-02T19:14:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:30.991131Z", 2023-11-02T19:14:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:14:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:14:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:36.429 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:14:36.429 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:14:36.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:36.429 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:36Z" 2023-11-02T19:14:36.430 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:36.431 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952350.991131_748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:12:30.991131Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:14:36Z" } 2023-11-02T19:14:36.431 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952350.991131_748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:12:30.991131Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:14:36.432 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952475.877925_751.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:36.432 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952350.991131_748.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:36.433 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952475.877925_751.1 target=1_1698952350.991131_748.1 2023-11-02T19:14:41.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:41.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:41.435 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:41.536 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:14:41.531+0000 7f98c90cdf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:14:41.537 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:14:41.535+0000 7f98c90cdf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:14:41.560 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:14:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:14:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:14:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:14:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:14:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:14:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952475.877925_751.1", 2023-11-02T19:14:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:35.877925Z", 2023-11-02T19:14:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:14:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:14:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:14:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:14:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:41Z" 2023-11-02T19:14:41.575 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:41.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952475.877925_751.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:35.877925Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:14:41Z" } 2023-11-02T19:14:41.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952475.877925_751.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:35.877925Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:14:41.577 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952475.877925_751.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:41.578 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952475.877925_751.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:41.578 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:14:41.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:41.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:41.579 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:14:41.621 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:41.621 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:41.720 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[]} 2023-11-02T19:14:41.721 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-70 2023-11-02T19:14:41.721 DEBUG:boto:path=/ibtljn-70/ 2023-11-02T19:14:41.722 DEBUG:boto:auth_path=/ibtljn-70/ 2023-11-02T19:14:41.722 DEBUG:boto:Method: PUT 2023-11-02T19:14:41.723 DEBUG:boto:Path: /ibtljn-70/ 2023-11-02T19:14:41.723 DEBUG:boto:Data: 2023-11-02T19:14:41.723 DEBUG:boto:Headers: {} 2023-11-02T19:14:41.724 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:14:41.724 DEBUG:boto:Port: 8000 2023-11-02T19:14:41.725 DEBUG:boto:Params: {} 2023-11-02T19:14:41.725 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T19:14:41.726 DEBUG:boto:Token: None 2023-11-02T19:14:41.726 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 19:14:41 GMT /ibtljn-70/ 2023-11-02T19:14:41.727 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CriKof1FDYHl/ipuN61lZprMtY4= 2023-11-02T19:14:41.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:14:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CriKof1FDYHl/ipuN61lZprMtY4=', 'Content-Length': '0'} 2023-11-02T19:14:41.759 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ab724e220d08b8e0-006543f521-2593-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:14:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:14:41.760 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-71 2023-11-02T19:14:41.760 DEBUG:boto:path=/ibtljn-71/ 2023-11-02T19:14:41.761 DEBUG:boto:auth_path=/ibtljn-71/ 2023-11-02T19:14:41.761 DEBUG:boto:Method: PUT 2023-11-02T19:14:41.762 DEBUG:boto:Path: /ibtljn-71/ 2023-11-02T19:14:41.762 DEBUG:boto:Data: 2023-11-02T19:14:41.763 DEBUG:boto:Headers: {} 2023-11-02T19:14:41.763 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:14:41.763 DEBUG:boto:Port: 8000 2023-11-02T19:14:41.764 DEBUG:boto:Params: {} 2023-11-02T19:14:41.764 DEBUG:boto:Token: None 2023-11-02T19:14:41.765 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 19:14:41 GMT /ibtljn-71/ 2023-11-02T19:14:41.765 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5f9j8MwTH7GAJG5xtaGkb9wMZFI= 2023-11-02T19:14:41.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:14:41 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5f9j8MwTH7GAJG5xtaGkb9wMZFI=', 'Content-Length': '0'} 2023-11-02T19:14:41.797 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000082dc6a2c9e0dc82-006543f521-2593-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:14:41 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:14:41.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:41.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:41.799 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:41.902 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:14:41.928 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:14:41.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:41.929 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:14:41.929 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:14:41.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:41.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:41.930 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:14:41.930 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:14:41.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:41.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:41.931 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:14:41.931 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:14:41.792845Z" 2023-11-02T19:14:41.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:41.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:41.932 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:14:41.932 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:14:41.932 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:41.932 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:14:41.933 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:41.934 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:14:41.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:41.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:41.935 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:41.984 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:41.984 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:14:42.039+0000 7ff73551bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:14:42.045 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:14:42.043+0000 7ff73551bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:14:42.073 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:14:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:14:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:14:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:14:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:14:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:14:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.746410_752.1", 2023-11-02T19:14:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.746410Z", 2023-11-02T19:14:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:14:42.085 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:42.085 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:42.085 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:42.085 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:14:42.085 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:14:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:14:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:42Z" 2023-11-02T19:14:42.088 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:42.089 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.746410_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.746410Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:14:42Z" } 2023-11-02T19:14:42.090 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.746410_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.746410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:14:42.091 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:42.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.746410_752.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:42.092 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952481.792845_759.1 target=1_1698952481.746410_752.1 2023-11-02T19:14:47.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:47.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:47.095 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:47.141 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:47.142 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:47.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:14:47.195+0000 7f02a6732f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:14:47.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:14:47.199+0000 7f02a6732f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:14:47.237 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:47.237 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:47.238 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:47.238 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:47.238 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:47.239 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:14:47.239 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:47.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:47.239 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:47.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:47.240 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:47.240 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:47.240 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:47.240 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:14:47.241 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:47.241 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:14:47.241 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:47.241 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:14:47.242 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:47.242 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:47.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:47.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:47.242 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:47.243 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:47.243 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:47.243 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:14:47.243 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:47.244 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:47.244 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:47.244 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:14:47.244 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:47.244 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:47.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:47.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:47.245 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:47.245 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:47.245 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:47.246 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:14:47.246 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:47.246 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:47.246 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:47.247 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:14:47.247 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:47.247 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:47.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:47.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:47.248 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:47.248 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:47.248 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:47.248 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:14:47.249 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:47.249 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:47.249 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:47.249 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:14:47.249 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:47.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:47.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:47.250 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:47.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:47.250 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:47.251 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:14:47.251 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:14:47.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:47.251 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:47Z" 2023-11-02T19:14:47.252 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:47.253 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:14:47Z" } 2023-11-02T19:14:47.253 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:14:47.254 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:47.254 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:47.255 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:14:47.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:47.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:47.257 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:14:47.302 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:47.302 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:47.399 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-11-02T19:14:47.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:47.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:47.400 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:14:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:47.546 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-11-02T19:14:47.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-70', '--dest-bucket=ibtljn-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:47.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-70', '--dest-bucket=ibtljn-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:47.548 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=ibtljn-70 --dest-bucket=ibtljn-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:14:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:47.696 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"ibtljn-70","zones":["a1"]},"dest":{"bucket":"ibtljn-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:14:47.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:47.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:47.698 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:14:47.741 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:47.741 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:47.831 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"ibtljn-70","zones":["a1"]},"dest":{"bucket":"ibtljn-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:14:47.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:47.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:47.833 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:14:47.877 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:47.878 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:48.004 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:48.004 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:14:48.005 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 18, 2023-11-02T19:14:48.005 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T19:14:48.005 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T19:14:48.005 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T19:14:48.005 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:14:48.006 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T19:14:48.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.006 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T19:14:48.006 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T19:14:48.007 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:14:48.007 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T19:14:48.007 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:14:48.007 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:14:48.008 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:14:48.008 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:14:48.008 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:14:48.008 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:14:48.008 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:14:48.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.009 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:14:48.009 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T19:14:48.009 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:14:48.010 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T19:14:48.010 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:14:48.010 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:14:48.010 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:14:48.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:14:48.011 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:14:48.011 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:14:48.011 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:14:48.011 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:14:48.012 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:14:48.012 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:14:48.012 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:14:48.012 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:14:48.012 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.014 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:14:48.014 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T19:14:48.014 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:14:48.014 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:14:48.015 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:14:48.015 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:14:48.015 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:14:48.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:14:48.016 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:14:48.016 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:14:48.016 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:14:48.016 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:14:48.016 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:14:48.017 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:14:48.017 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:14:48.017 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:14:48.017 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.017 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.018 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:14:48.018 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:14:48.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.018 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:14:48.019 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:14:48.019 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:14:48.019 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:14:48.019 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.020 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.020 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:14:48.020 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:14:48.020 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:14:48.020 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:14:48.021 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [ 2023-11-02T19:14:48.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.021 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-group", 2023-11-02T19:14:48.021 INFO:teuthology.orchestra.run.smithi017.stdout: "data_flow": { 2023-11-02T19:14:48.022 INFO:teuthology.orchestra.run.smithi017.stdout: "symmetrical": [ 2023-11-02T19:14:48.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.022 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-flow1", 2023-11-02T19:14:48.022 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:14:48.022 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:14:48.023 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:14:48.023 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.023 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.023 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.024 INFO:teuthology.orchestra.run.smithi017.stdout: "pipes": [ 2023-11-02T19:14:48.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.024 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-pipe", 2023-11-02T19:14:48.025 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:14:48.025 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-70", 2023-11-02T19:14:48.025 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:14:48.025 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:14:48.025 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.026 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": { 2023-11-02T19:14:48.026 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-71", 2023-11-02T19:14:48.026 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:14:48.027 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a" 2023-11-02T19:14:48.027 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.027 INFO:teuthology.orchestra.run.smithi017.stdout: "params": { 2023-11-02T19:14:48.027 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:14:48.028 INFO:teuthology.orchestra.run.smithi017.stdout: "filter": { 2023-11-02T19:14:48.028 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [] 2023-11-02T19:14:48.028 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.029 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": {}, 2023-11-02T19:14:48.029 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2023-11-02T19:14:48.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mode": "system", 2023-11-02T19:14:48.029 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "" 2023-11-02T19:14:48.029 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.030 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.030 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:14:48.030 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "enabled" 2023-11-02T19:14:48.030 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.031 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.031 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:14:48.031 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:14:48.032 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.032 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T19:14:48.033 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T19:14:48.033 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:14:48.033 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T19:14:48.033 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:14:48.033 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:14:48.034 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:14:48.034 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:14:48.034 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:14:48.034 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:14:48.035 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:14:48.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.035 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:14:48.035 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T19:14:48.036 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:14:48.036 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:14:48.036 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:14:48.036 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:14:48.036 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:14:48.037 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:14:48.037 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:14:48.037 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:14:48.037 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:14:48.038 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:14:48.038 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:14:48.038 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:14:48.038 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:14:48.039 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:14:48.039 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.040 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:14:48.040 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T19:14:48.040 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:14:48.040 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T19:14:48.041 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:14:48.041 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:14:48.041 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:14:48.041 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:14:48.042 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:14:48.042 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:14:48.042 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:14:48.042 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:14:48.043 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:14:48.043 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:14:48.043 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:14:48.043 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:14:48.043 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.044 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.044 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:14:48.044 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:14:48.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.045 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:14:48.045 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:14:48.045 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:14:48.045 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:14:48.046 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.046 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:14:48.046 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:14:48.047 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:14:48.047 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:14:48.047 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T19:14:48.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.049 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:14:48.049 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:14:48.049 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.050 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:14:48.050 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T19:14:48.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.050 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:14:48.051 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T19:14:48.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.051 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:14:48.052 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T19:14:48.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.052 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:14:48.052 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T19:14:48.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.054 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:14:48.055 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T19:14:48.055 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.055 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.056 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T19:14:48.056 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:14:48.056 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T19:14:48.056 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T19:14:48.056 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:14:48.057 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:14:48.057 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:14:48.057 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:14:48.057 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:14:48.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.058 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T19:14:48.058 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:14:48.058 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:14:48.058 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:14:48.059 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:14:48.059 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:14:48.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.059 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T19:14:48.060 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:14:48.060 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:14:48.060 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:14:48.060 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:14:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:14:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T19:14:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:14:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:14:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:14:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:14:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:14:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.063 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T19:14:48.063 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:14:48.063 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:14:48.063 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:14:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:14:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:14:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:14:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T19:14:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:48.065 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:48.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:48.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:14:48.067 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:14:48.119 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:48.119 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:48.206 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"ibtljn-70","zones":["a1"]},"dest":{"bucket":"ibtljn-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:14:48.207 DEBUG:boto:path=/ibtljn-70/ 2023-11-02T19:14:48.207 DEBUG:boto:auth_path=/ibtljn-70/ 2023-11-02T19:14:48.208 DEBUG:boto:Method: HEAD 2023-11-02T19:14:48.208 DEBUG:boto:Path: /ibtljn-70/ 2023-11-02T19:14:48.209 DEBUG:boto:Data: 2023-11-02T19:14:48.209 DEBUG:boto:Headers: {} 2023-11-02T19:14:48.209 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:14:48.210 DEBUG:boto:Port: 8000 2023-11-02T19:14:48.210 DEBUG:boto:Params: {} 2023-11-02T19:14:48.211 DEBUG:boto:Token: None 2023-11-02T19:14:48.211 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:14:48 GMT /ibtljn-70/ 2023-11-02T19:14:48.212 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R3mrKoqRJQ/VIk0t8bvxAThtD+Y= 2023-11-02T19:14:48.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:14:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R3mrKoqRJQ/VIk0t8bvxAThtD+Y=', 'Content-Length': '0'} 2023-11-02T19:14:48.220 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2cea20eacabac69-006543f528-271b-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:14:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:14:48.220 DEBUG:boto:path=/ibtljn-70/obj1 2023-11-02T19:14:48.221 DEBUG:boto:auth_path=/ibtljn-70/obj1 2023-11-02T19:14:48.221 DEBUG:boto:Method: PUT 2023-11-02T19:14:48.222 DEBUG:boto:Path: /ibtljn-70/obj1 2023-11-02T19:14:48.222 DEBUG:boto:Data: 2023-11-02T19:14:48.223 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:14:48.223 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:14:48.224 DEBUG:boto:Port: 8000 2023-11-02T19:14:48.224 DEBUG:boto:Params: {} 2023-11-02T19:14:48.225 DEBUG:boto:Token: None 2023-11-02T19:14:48.225 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:14:48 GMT /ibtljn-70/obj1 2023-11-02T19:14:48.226 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xFqPRCLNSthIHLzXo9baHoNGXD4= 2023-11-02T19:14:48.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:14:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xFqPRCLNSthIHLzXo9baHoNGXD4='} 2023-11-02T19:14:48.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3f82bd0442bc5af-006543f528-271b-a1'), ('Date', 'Thu, 02 Nov 2023 19:14:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:14:48.243 DEBUG:boto:path=/ibtljn-70/ 2023-11-02T19:14:48.243 DEBUG:boto:auth_path=/ibtljn-70/ 2023-11-02T19:14:48.244 DEBUG:boto:Method: HEAD 2023-11-02T19:14:48.244 DEBUG:boto:Path: /ibtljn-70/ 2023-11-02T19:14:48.245 DEBUG:boto:Data: 2023-11-02T19:14:48.245 DEBUG:boto:Headers: {} 2023-11-02T19:14:48.246 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:14:48.246 DEBUG:boto:Port: 8000 2023-11-02T19:14:48.247 DEBUG:boto:Params: {} 2023-11-02T19:14:48.247 DEBUG:boto:Token: None 2023-11-02T19:14:48.247 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:14:48 GMT /ibtljn-70/ 2023-11-02T19:14:48.248 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:R3mrKoqRJQ/VIk0t8bvxAThtD+Y= 2023-11-02T19:14:48.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:14:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:R3mrKoqRJQ/VIk0t8bvxAThtD+Y=', 'Content-Length': '0'} 2023-11-02T19:14:48.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072baa3b42bba2bf4-006543f528-271b-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:14:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:14:48.256 DEBUG:boto:path=/ibtljn-70/obj2 2023-11-02T19:14:48.256 DEBUG:boto:auth_path=/ibtljn-70/obj2 2023-11-02T19:14:48.257 DEBUG:boto:Method: PUT 2023-11-02T19:14:48.257 DEBUG:boto:Path: /ibtljn-70/obj2 2023-11-02T19:14:48.258 DEBUG:boto:Data: 2023-11-02T19:14:48.258 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:14:48.259 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:14:48.259 DEBUG:boto:Port: 8000 2023-11-02T19:14:48.259 DEBUG:boto:Params: {} 2023-11-02T19:14:48.260 DEBUG:boto:Token: None 2023-11-02T19:14:48.260 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:14:48 GMT /ibtljn-70/obj2 2023-11-02T19:14:48.261 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fm59/Rj4hTth7WiPznp9dZ6TcUM= 2023-11-02T19:14:48.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:14:48 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fm59/Rj4hTth7WiPznp9dZ6TcUM='} 2023-11-02T19:14:48.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003fe11195230f5a89-006543f528-271b-a1'), ('Date', 'Thu, 02 Nov 2023 19:14:48 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:14:48.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:48.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:48.308 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:48.360 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:48.360 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:48.460 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:14:48.497 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:14:48.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.497 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:14:48.498 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:14:48.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.498 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:14:48.499 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:14:48.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.499 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:14:48.500 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:14:41.792845Z" 2023-11-02T19:14:48.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.500 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:14:48.500 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:14:48.501 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.501 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:14:48.501 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:48.502 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:14:48.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:48.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:48.503 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:48.563 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:48.564 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:48.635 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:14:48.631+0000 7f52be0ccf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:14:48.637 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:14:48.635+0000 7f52be0ccf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:14:48.668 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:48.668 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:48.668 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:48.669 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:48.669 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:48.669 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:14:48.669 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:48.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.670 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:48.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.670 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:48.671 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:48.671 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:48.671 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:14:48.671 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:48.671 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:14:48.672 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:48.672 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:14:48.672 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:48.672 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.673 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:48.673 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:48.673 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:48.674 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:14:48.674 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:48.674 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:48.674 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:48.674 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:14:48.675 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:48.675 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.676 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:48.676 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:48.676 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:48.676 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:14:48.676 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:48.677 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:48.677 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:48.677 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:14:48.677 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:48.678 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.678 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:48.679 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:48.679 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:14:48.679 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:14:48.679 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:48.679 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:48.680 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:48.680 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:14:48.680 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:14:48.680 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.681 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.681 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:48.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.681 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:48.682 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:14:48.682 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:14:48.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.682 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:48Z" 2023-11-02T19:14:48.683 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:48.683 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:14:48Z" } 2023-11-02T19:14:48.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:14:48.684 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:48.684 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:14:48.685 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:14:48.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:48.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:48.686 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:48.731 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:48.731 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:48.815 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:14:48.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.816 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:14:48.816 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:14:48.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.817 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051387", 2023-11-02T19:14:48.817 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:14:48.238962Z" 2023-11-02T19:14:48.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.818 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:14:48.818 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:14:48.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.819 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045796", 2023-11-02T19:14:48.820 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:14:48.276692Z" 2023-11-02T19:14:48.820 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.820 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:14:48.822 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:14:48.823 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:14:48.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:48.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:48.824 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:48.910 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:48.994 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:48.994 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:48.995 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:48.995 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:48.995 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:48.995 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:14:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:48.997 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:48.997 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:14:48.997 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:48.997 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:14:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:14:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:48.999 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:48.999 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:48.999 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:48.999 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:14:49.000 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:49.000 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:14:49.000 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:49.000 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:14:49.000 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:49.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:49.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:49.001 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:49.001 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:49.001 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:14:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:14:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:14:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:14:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:14:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:14:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:48Z" 2023-11-02T19:14:49.007 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:49.008 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:14:48Z" } 2023-11-02T19:14:49.009 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:14:49.009 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:14:49.010 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:49.010 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:14:54.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:54.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:54.013 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:54.062 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:54.062 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:54.152 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:54.152 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:54.152 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:54.153 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:54.153 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:54.153 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:14:54.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:54.154 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:54.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:54.154 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:54.154 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:54.155 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:54.155 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:14:54.155 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:54.156 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:14:54.156 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:54.156 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:14:54.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:54.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:54.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:54.157 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:54.158 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:54.158 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:54.158 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:14:54.159 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:54.159 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:14:54.160 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:54.160 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:14:54.161 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:54.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:54.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:54.162 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:54.162 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:54.163 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:54.163 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:14:54.163 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:54.163 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:54.164 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:54.164 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:14:54.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:54.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:54.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:54.165 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:54.165 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:54.165 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:14:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:14:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:14:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:14:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:54Z" 2023-11-02T19:14:54.169 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:54.170 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:14:54Z" } 2023-11-02T19:14:54.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:14:54.171 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:14:54.171 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:54.172 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:14:59.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:59.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:14:59.175 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:14:59.219 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:14:59.219 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:14:59.310 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:14:59.310 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:14:59.310 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:14:59.311 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:14:59.311 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:14:59.311 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:14:59.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:59.312 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:14:59.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:59.312 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:14:59.312 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:59.312 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:59.313 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:14:59.313 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:59.313 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:14:59.313 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:59.313 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:14:59.314 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:59.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:59.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:59.314 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:14:59.315 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:59.315 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:59.315 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:14:59.315 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:59.315 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:14:59.316 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:59.316 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:14:59.316 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:59.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:59.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:59.317 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:14:59.317 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:59.317 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:59.317 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:14:59.318 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:59.318 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:59.318 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:59.318 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:14:59.318 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:59.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:59.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:14:59.319 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:14:59.319 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:14:59.320 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:14:59.320 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:14:59.320 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:14:59.320 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:14:59.321 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:14:59.321 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:14:59.321 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:59.321 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:14:59.321 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:14:59.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:59.322 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:14:59.322 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:14:59.322 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:14:59.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:14:59.323 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:14:59Z" 2023-11-02T19:14:59.323 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:14:59.324 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:14:59Z" } 2023-11-02T19:14:59.325 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:14:59.325 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:14:59.326 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:14:59.326 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:15:04.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:04.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:04.328 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:15:04.377 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:15:04.378 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:15:04.477 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:15:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:15:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:15:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:15:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:15:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:15:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:15:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:15:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:15:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:15:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:15:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:15:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:15:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:15:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:15:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:15:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:15:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:15:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:15:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:15:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:15:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:15:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:15:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:15:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:15:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:15:04Z" 2023-11-02T19:15:04.493 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:15:04.494 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:15:04Z" } 2023-11-02T19:15:04.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:15:04.496 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:15:04.496 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:15:04.496 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:15:09.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:09.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:09.499 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:15:09.547 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:15:09.547 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:15:09.633 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:15:09.634 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:15:09.634 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:15:09.634 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:15:09.634 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:15:09.635 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:15:09.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:09.635 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:15:09.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:09.636 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:15:09.636 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:09.636 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:09.636 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:15:09.637 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:09.637 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:15:09.637 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:09.637 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:15:09.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:09.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:09.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:09.638 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:15:09.638 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:09.639 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:09.639 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:15:09.639 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:09.639 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:15:09.639 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:09.640 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:15:09.640 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:09.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:09.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:09.641 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:15:09.641 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:09.641 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:09.641 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:15:09.642 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:09.642 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:09.642 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:09.642 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:15:09.642 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:09.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:09.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:09.643 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:15:09.643 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:09.643 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:09.644 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:15:09.644 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:09.644 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:09.644 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:09.645 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:15:09.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:09.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:09.645 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:15:09.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:09.646 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:15:09.646 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:15:09.646 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:15:09.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:09.647 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:15:09Z" 2023-11-02T19:15:09.647 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:15:09.648 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:15:09Z" } 2023-11-02T19:15:09.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:15:09.649 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:15:09.649 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:15:09.650 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:15:14.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:14.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:14.652 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:15:14.700 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:15:14.700 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:15:14.786 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:15:14.787 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:15:14.787 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:15:14.787 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:15:14.788 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:15:14.788 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:15:14.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:14.788 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:15:14.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:14.789 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:15:14.789 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:14.789 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:14.789 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:15:14.790 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:14.790 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:15:14.790 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:14.790 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:15:14.790 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:14.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:14.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:14.791 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:15:14.791 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:14.791 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:14.792 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:15:14.792 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:14.792 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:15:14.792 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:14.793 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:15:14.793 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:14.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:14.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:14.793 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:15:14.794 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:14.794 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:14.794 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:15:14.794 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:14.795 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:14.795 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:14.795 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:15:14.795 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:14.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:14.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:14.796 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:15:14.796 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:14.796 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:14.796 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:15:14.797 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:14.797 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:14.797 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:14.797 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:15:14.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:14.798 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:14.798 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:15:14.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:14.798 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:15:14.799 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:15:14.799 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:15:14.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:14.799 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:15:14Z" 2023-11-02T19:15:14.799 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:15:14.801 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:15:14Z" } 2023-11-02T19:15:14.802 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:15:14.802 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:15:14.803 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:15:14.803 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:15:19.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:19.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:19.806 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:15:19.854 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:15:19.855 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:15:19.941 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:15:19.941 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:15:19.942 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:15:19.942 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:15:19.942 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:15:19.943 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:15:19.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:19.943 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:15:19.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:19.944 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:15:19.944 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:19.945 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:19.945 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:15:19.945 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:19.945 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:15:19.946 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:19.946 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:15:19.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:19.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:19.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:19.947 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:15:19.947 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:19.947 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:19.947 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:15:19.948 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:19.948 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:15:19.948 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:19.948 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:15:19.949 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:19.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:19.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:19.949 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:15:19.950 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:19.950 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:19.950 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:15:19.950 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:19.950 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:19.951 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:19.951 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:15:19.951 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:19.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:19.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:19.952 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:15:19.952 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:19.952 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:19.953 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:15:19.953 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:19.953 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:19.953 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:19.954 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:15:19.954 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:19.955 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:19.955 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:15:19.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:19.955 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:15:19.956 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:15:19.956 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:15:19.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:19.957 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:15:19Z" 2023-11-02T19:15:19.957 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:15:19.958 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:15:19Z" } 2023-11-02T19:15:19.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:15:19.959 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:15:19.960 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:15:19.960 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:15:24.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:24.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:24.963 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:15:25.010 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:15:25.011 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:15:25.098 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:15:25.099 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:15:25.099 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:15:25.099 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:15:25.100 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:15:25.100 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:15:25.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:25.100 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:15:25.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:25.101 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:15:25.101 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:25.101 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:25.101 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:15:25.102 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:25.102 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:15:25.102 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:25.102 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:15:25.103 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:25.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:25.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:25.103 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:15:25.103 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:25.104 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:25.104 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:15:25.104 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:25.104 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:15:25.104 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:25.105 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:15:25.105 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:25.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:25.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:25.106 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:15:25.106 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:25.106 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:25.106 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:15:25.106 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:25.107 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:25.107 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:25.107 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:15:25.107 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:25.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:25.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:25.108 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:15:25.108 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:25.108 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:25.109 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:15:25.109 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:25.109 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:25.109 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:25.109 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:15:25.110 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:25.110 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:25.110 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:15:25.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:25.111 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:15:25.111 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:15:25.111 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:15:25.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:25.111 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:15:25Z" 2023-11-02T19:15:25.112 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:15:25.113 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:15:25Z" } 2023-11-02T19:15:25.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:15:25.114 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:15:25.114 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:15:25.115 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:15:30.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:30.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:30.117 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:15:30.165 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:15:30.166 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:15:30.261 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:15:30.261 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:15:30.262 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:15:30.262 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:15:30.262 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:15:30.262 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:15:30.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:30.263 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:15:30.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:30.263 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:15:30.265 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:30.265 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:30.266 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:15:30.266 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:30.266 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:15:30.266 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:30.267 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:15:30.267 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:30.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:30.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:30.267 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:15:30.268 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:30.268 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:30.268 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:15:30.269 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:30.269 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:15:30.269 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:30.269 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:15:30.270 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:30.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:30.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:30.270 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:15:30.270 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:30.271 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:30.271 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:15:30.271 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:30.271 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:30.272 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:30.272 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:15:30.272 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:30.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:30.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:30.273 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:15:30.273 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:30.273 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:30.273 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:15:30.274 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:30.274 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:30.274 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:30.274 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:15:30.275 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:30.275 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:30.275 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:15:30.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:30.276 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:15:30.276 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:15:30.276 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:15:30.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:30.276 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:15:30Z" 2023-11-02T19:15:30.277 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:15:30.278 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:15:30Z" } 2023-11-02T19:15:30.278 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:15:30.279 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:15:30.280 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:15:30.280 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:15:35.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:35.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:35.283 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:15:35.334 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:15:35.335 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:15:35.427 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:15:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:15:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:15:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:15:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:15:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:15:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:15:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:15:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:15:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:15:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:15:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:15:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:35.433 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:15:35.433 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:35.433 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:15:35.433 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:35.434 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:15:35.434 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:35.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:35.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:35.434 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:15:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:15:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:15:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:15:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:15:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:15:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:15:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:15:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:15:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:15:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:15:35Z" 2023-11-02T19:15:35.441 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:15:35.442 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:15:35Z" } 2023-11-02T19:15:35.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:15:35.443 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:15:35.444 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:15:35.444 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:15:40.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:40.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:40.447 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:15:40.494 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:15:40.494 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:15:40.585 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:15:40.585 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:15:40.586 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:15:40.586 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:15:40.586 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:15:40.586 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:15:40.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:40.587 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:15:40.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:40.587 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:15:40.587 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:40.588 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:40.588 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:15:40.588 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:40.588 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:15:40.589 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:40.589 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:15:40.589 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:40.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:40.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:40.590 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:15:40.590 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:40.590 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:40.590 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:15:40.590 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:40.591 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:15:40.591 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:40.591 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:15:40.591 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:40.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:40.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:40.592 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:15:40.592 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:40.592 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:40.593 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:15:40.593 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:40.593 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:40.593 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:40.594 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:15:40.594 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:40.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:40.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:40.594 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:15:40.595 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:40.595 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:40.595 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:15:40.595 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:40.595 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:40.596 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:40.596 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:15:40.596 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:40.596 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:40.596 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:15:40.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:40.597 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:15:40.597 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:15:40.597 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:15:40.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:40.598 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:15:40Z" 2023-11-02T19:15:40.598 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:15:40.599 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:15:40Z" } 2023-11-02T19:15:40.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:15:40.600 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:15:40.600 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:15:40.601 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:15:45.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:45.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:45.604 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:15:45.653 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:15:45.654 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:15:45.743 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:15:45.743 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:15:45.743 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:15:45.743 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:15:45.744 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:15:45.744 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:15:45.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:45.744 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:15:45.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:45.745 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:15:45.745 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:45.745 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:45.745 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:15:45.746 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:45.746 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:15:45.746 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:45.746 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:15:45.747 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:45.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:45.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:45.747 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:15:45.747 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:45.748 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:45.748 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:15:45.748 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:45.748 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:15:45.749 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:45.749 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:15:45.749 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:45.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:45.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:45.750 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:15:45.750 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:45.750 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:45.750 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:15:45.751 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:45.751 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:45.751 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:45.751 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:15:45.751 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:45.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:45.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:45.752 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:15:45.752 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:45.753 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:45.753 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:15:45.753 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:45.753 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:45.753 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:45.754 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:15:45.754 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:45.754 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:45.754 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:15:45.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:45.755 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:15:45.755 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:15:45.755 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:15:45.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:45.756 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:15:45Z" 2023-11-02T19:15:45.756 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:15:45.757 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:15:45Z" } 2023-11-02T19:15:45.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:15:45.758 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:15:45.758 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:15:45.759 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:15:50.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:50.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:50.762 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:15:50.813 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:15:50.813 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:15:50.905 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:15:50.905 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:15:50.905 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:15:50.905 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:15:50.906 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:15:50.906 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:15:50.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:50.906 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:15:50.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:50.907 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:15:50.907 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:50.907 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:50.908 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:15:50.908 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:50.908 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:15:50.908 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:50.908 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:15:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:15:50.910 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:50.910 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:50.910 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:15:50.910 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:50.911 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:15:50.911 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:50.911 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:15:50.911 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:50.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:50.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:50.912 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:15:50.912 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:50.912 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:50.913 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:15:50.913 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:50.913 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:50.913 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:50.913 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:15:50.914 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:50.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:50.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:50.914 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:15:50.915 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:50.915 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:50.915 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:15:50.915 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:50.916 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:50.916 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:50.916 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:15:50.916 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:50.916 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:50.917 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:15:50.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:50.917 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:15:50.917 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:15:50.917 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:15:50.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:50.918 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:15:50Z" 2023-11-02T19:15:50.918 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:15:50.919 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:15:50Z" } 2023-11-02T19:15:50.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:15:50.920 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:15:50.921 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:15:50.921 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:15:55.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:55.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:15:55.924 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:15:55.973 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:15:55.974 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:15:56.063 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:15:56.063 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:15:56.064 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:15:56.064 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:15:56.064 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:15:56.064 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:15:56.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:56.065 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:15:56.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:56.065 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:15:56.065 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:56.066 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:56.066 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:15:56.066 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:56.066 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:15:56.067 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:56.067 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:15:56.067 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:56.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:56.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:56.068 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:15:56.068 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:56.068 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:56.068 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:15:56.069 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:56.069 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:15:56.069 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:56.069 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:15:56.070 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:56.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:56.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:56.070 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:15:56.071 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:56.071 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:56.071 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:15:56.071 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:56.071 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:56.072 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:56.072 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:15:56.072 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:56.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:56.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:15:56.073 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:15:56.073 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:15:56.073 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:15:56.073 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:15:56.074 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:15:56.074 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:15:56.074 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:15:56.074 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:15:56.075 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:56.075 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:15:56.075 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:15:56.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:56.076 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:15:56.076 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:15:56.076 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:15:56.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:15:56.076 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:15:56Z" 2023-11-02T19:15:56.077 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:15:56.078 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:15:56Z" } 2023-11-02T19:15:56.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:15:56.079 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:15:56.080 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:15:56.080 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:16:01.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:01.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:01.083 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:01.131 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:01.131 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:01.222 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:01.223 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:01.223 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:01.223 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:01.223 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:16:01.224 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:16:01.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:01.224 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:16:01.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:01.225 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:16:01.225 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:01.225 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:01.225 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:16:01.225 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:01.226 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:16:01.226 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:01.226 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:16:01.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:01.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:01.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:01.227 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:16:01.228 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:01.228 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:01.229 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:16:01.229 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:01.229 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:16:01.229 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:01.229 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:16:01.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:01.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:01.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:01.230 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:16:01.231 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:01.231 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:01.231 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:16:01.231 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:01.231 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:01.232 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:01.232 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:16:01.232 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:01.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:01.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:01.233 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:16:01.233 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:01.233 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:01.233 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:16:01.234 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:01.234 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:01.234 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:01.234 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:16:01.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:01.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:01.235 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:01.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:01.235 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:16:01.236 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:16:01.236 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:16:01.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:01.236 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:01Z" 2023-11-02T19:16:01.236 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:01.237 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:16:01Z" } 2023-11-02T19:16:01.238 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:16:01.239 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:16:01.239 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:16:01.239 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:16:06.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:06.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:06.242 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:06.292 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:06.292 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:06.384 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:06.384 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:06.385 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:06.385 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:06.385 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:16:06.386 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:16:06.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:06.386 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:16:06.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:06.387 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:16:06.387 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:06.387 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:06.388 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:16:06.388 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:06.389 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:16:06.390 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:06.390 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:16:06.390 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:06.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:06.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:06.391 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:16:06.391 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:06.391 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:06.392 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:16:06.392 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:06.393 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:16:06.393 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:06.393 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:16:06.394 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:06.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:06.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:06.395 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:16:06.395 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:06.395 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:06.395 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:16:06.396 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:06.396 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:06.397 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:06.397 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:16:06.397 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:06.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:06.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:06.398 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:16:06.399 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:06.399 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:06.399 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:16:06.400 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:06.400 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:06.401 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:06.401 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:16:06.401 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:06.402 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:06.402 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:06.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:06.403 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:16:06.403 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:16:06.404 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:16:06.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:06.405 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:06Z" 2023-11-02T19:16:06.405 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:06.407 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:16:06Z" } 2023-11-02T19:16:06.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:16:06.408 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:16:06.409 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:16:06.409 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:16:11.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:11.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:11.411 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:11.458 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:11.459 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:11.543 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:11.544 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:11.544 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:11.544 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:11.545 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:16:11.545 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:16:11.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:11.545 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:16:11.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:11.546 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:16:11.546 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:11.546 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:11.546 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:16:11.547 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:11.547 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:16:11.547 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:11.547 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:16:11.548 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:11.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:11.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:11.548 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:16:11.549 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:11.549 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:11.549 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:16:11.549 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:11.549 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:16:11.550 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:11.550 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:16:11.550 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:11.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:11.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:11.551 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:16:11.551 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:11.551 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:11.552 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:16:11.552 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:11.552 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:11.552 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:11.553 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:16:11.553 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:11.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:11.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:11.554 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:16:11.555 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:11.555 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:11.555 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:16:11.555 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:11.555 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:11.556 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:11.556 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:16:11.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:11.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:11.557 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:11.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:11.557 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:16:11.557 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:16:11.557 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:16:11.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:11.558 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:11Z" 2023-11-02T19:16:11.558 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:11.559 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:16:11Z" } 2023-11-02T19:16:11.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:16:11.560 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:16:11.561 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:16:11.561 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:16:16.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:16.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:16.564 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:16.612 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:16.613 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:16.701 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:16.702 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:16.702 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:16:16.702 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:16:16.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:16.703 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:16:16.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:16.703 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:16:16.703 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:16.703 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:16.704 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:16:16.704 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:16.704 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:16:16.704 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:16:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:16:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:16:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:16:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:16:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:16:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:16:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:16:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:16:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:16:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:16:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:16:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:16:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:16:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:16Z" 2023-11-02T19:16:16.714 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:16.715 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:16:16Z" } 2023-11-02T19:16:16.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:16:16.716 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:16:16.716 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:16:16.717 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:16:21.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:21.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:21.719 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:21.761 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:21.761 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:21.848 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:16:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:16:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:16:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:16:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:16:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:16:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:16:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:16:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:16:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:16:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:16:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:16:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:16:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:16:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:16:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:16:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:16:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:16:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:16:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:16:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:21Z" 2023-11-02T19:16:21.861 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:21.862 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:16:21Z" } 2023-11-02T19:16:21.863 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:16:21.863 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:16:21.863 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:16:21.864 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:16:26.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:26.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:26.866 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:26.914 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:26.914 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:27.002 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:16:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:16:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:16:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:16:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:16:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:16:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:16:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:16:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:16:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:16:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:16:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:16:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:16:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:16:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:16:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:16:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:16:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:16:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:16:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:16:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:26Z" 2023-11-02T19:16:27.016 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:27.017 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:16:26Z" } 2023-11-02T19:16:27.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:16:27.018 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:16:27.018 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:16:27.019 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:16:32.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:32.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:32.021 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:32.069 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:32.070 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:32.162 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:32.162 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:32.162 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:32.163 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:32.163 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:16:32.163 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:16:32.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:32.164 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:16:32.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:32.164 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:16:32.164 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:32.164 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:32.165 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:16:32.165 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:32.165 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:16:32.165 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:32.165 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:16:32.166 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:32.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:32.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:32.166 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:16:32.167 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:32.167 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:32.167 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:16:32.167 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:32.168 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:16:32.168 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:32.168 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:16:32.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:32.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:32.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:32.169 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:16:32.169 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:32.169 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:32.169 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:16:32.170 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:32.170 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:32.170 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:32.170 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:16:32.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:32.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:32.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:32.171 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:16:32.171 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:32.172 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:32.172 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:16:32.172 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:32.172 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:32.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:32.173 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:16:32.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:32.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:32.173 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:32.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:32.174 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:16:32.174 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:16:32.174 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:16:32.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:32.175 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:32Z" 2023-11-02T19:16:32.175 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:32.176 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:16:32Z" } 2023-11-02T19:16:32.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:16:32.177 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:16:32.177 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:16:32.178 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:16:37.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:37.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:37.180 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:37.324 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:37.324 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:37.324 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:37.325 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:37.325 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:16:37.325 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:16:37.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:37.325 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:16:37.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:37.326 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:16:37.326 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:37.326 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:37.327 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:16:37.327 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:37.327 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:16:37.327 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:37.327 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:16:37.328 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:37.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:37.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:37.328 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:16:37.329 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:37.329 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:37.329 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:16:37.329 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:37.329 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:16:37.330 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:37.330 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:16:37.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:37.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:37.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:37.331 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:16:37.331 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:37.331 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:37.331 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:16:37.332 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:37.332 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:37.332 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:37.332 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:16:37.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:37.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:37.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:37.333 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:16:37.333 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:37.334 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:37.334 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:16:37.334 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:37.334 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:37.335 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:37.335 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:16:37.335 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:37.335 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:37.335 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:37.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:37.336 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:16:37.336 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:16:37.336 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:16:37.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:37.337 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:37Z" 2023-11-02T19:16:37.337 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:37.338 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:16:37Z" } 2023-11-02T19:16:37.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:16:37.339 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:16:37.339 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:16:37.340 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:16:42.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:42.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:42.343 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:42.391 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:42.391 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:42.478 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:42.478 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:42.479 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:42.479 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:42.479 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:16:42.479 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:16:42.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:42.480 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:16:42.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:42.481 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:16:42.481 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:42.481 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:42.481 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:16:42.482 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:42.482 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:16:42.482 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:42.482 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:16:42.482 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:42.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:42.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:42.483 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:16:42.483 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:42.484 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:42.484 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:16:42.484 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:42.484 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:16:42.485 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:42.485 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:16:42.485 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:42.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:42.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:42.486 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:16:42.486 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:42.486 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:42.486 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:16:42.487 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:42.487 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:42.487 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:42.487 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:16:42.488 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:42.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:42.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:42.488 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:16:42.489 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:42.489 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:42.489 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:16:42.489 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:42.490 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:42.490 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:42.490 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:16:42.490 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:42.490 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:42.491 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:42.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:42.491 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:16:42.491 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:16:42.492 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:16:42.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:42.492 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:42Z" 2023-11-02T19:16:42.492 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:42.493 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:16:42Z" } 2023-11-02T19:16:42.494 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:16:42.495 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:16:42.495 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:16:42.495 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:16:47.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:47.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:47.498 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:47.545 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:47.546 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:47.639 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:16:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:16:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:16:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:16:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:16:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:16:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:16:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:16:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051247", 2023-11-02T19:16:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:16:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:12:31.241583Z" 2023-11-02T19:16:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:16:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:16:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:16:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:16:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045656", 2023-11-02T19:16:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.872114Z" 2023-11-02T19:16:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:16:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:16:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:16:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:47Z" 2023-11-02T19:16:47.653 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:47.655 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051247", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:12:31.241583Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045656", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.872114Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:16:47Z" } 2023-11-02T19:16:47.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051247', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:12:31.241583Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045656', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.872114Z'}}] 2023-11-02T19:16:47.656 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:16:47.656 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051247', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045656'} 2023-11-02T19:16:47.657 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051387 target=00000000000000000000:00000000000000051247, shard=3 master=00000000000000000000:00000000000000045796 target=00000000000000000000:00000000000000045656 2023-11-02T19:16:49.321 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T19:16:49.318+0000 7f25de499700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-11-02T19:16:49.426 INFO:tasks.rgw.c2.client.1.smithi069.stdout:2023-11-02T19:16:49.422+0000 7f1bb5aa4700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-11-02T19:16:52.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:52.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:52.660 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:52.711 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:52.711 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:52.794 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:52.794 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:52.795 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:52.795 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:52.795 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:16:52.795 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:16:52.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:52.796 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:16:52.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:52.796 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:16:52.797 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:52.797 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:52.797 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:16:52.797 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:52.797 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:16:52.798 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:52.798 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:16:52.798 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:52.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:52.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:52.799 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:16:52.799 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:52.799 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:52.799 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051387", 2023-11-02T19:16:52.800 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:52.800 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:16:52.800 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:52.800 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:48.238962Z" 2023-11-02T19:16:52.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:52.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:52.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:52.801 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:16:52.801 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:52.802 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:52.802 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:16:52.802 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:52.802 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:52.802 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:52.803 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:16:52.803 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:52.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:52.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:52.803 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:16:52.804 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:52.804 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:16:52.804 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045796", 2023-11-02T19:16:52.804 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:52.805 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:52.805 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:52.805 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:48.276692Z" 2023-11-02T19:16:52.805 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:52.805 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:52.806 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:52.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:52.806 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:16:52.806 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:16:52.806 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:16:52.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:52.807 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:52Z" 2023-11-02T19:16:52.807 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:52.808 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051387", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:14:48.238962Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:48.276692Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:16:52Z" } 2023-11-02T19:16:52.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051387', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:14:48.238962Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:48.276692Z'}}] 2023-11-02T19:16:52.809 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:16:52.810 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:16:52.810 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:16:52.811 DEBUG:boto:path=/ibtljn-70/ 2023-11-02T19:16:52.811 DEBUG:boto:auth_path=/ibtljn-70/ 2023-11-02T19:16:52.811 DEBUG:boto:Method: HEAD 2023-11-02T19:16:52.812 DEBUG:boto:Path: /ibtljn-70/ 2023-11-02T19:16:52.812 DEBUG:boto:Data: 2023-11-02T19:16:52.813 DEBUG:boto:Headers: {} 2023-11-02T19:16:52.813 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:16:52.813 DEBUG:boto:Port: 8001 2023-11-02T19:16:52.814 DEBUG:boto:Params: {} 2023-11-02T19:16:52.815 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:16:52.815 DEBUG:boto:Token: None 2023-11-02T19:16:52.816 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:16:52 GMT /ibtljn-70/ 2023-11-02T19:16:52.817 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Vnufi8ba8bV5w/8YREv94sXJAdA= 2023-11-02T19:16:52.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:16:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vnufi8ba8bV5w/8YREv94sXJAdA=', 'Content-Length': '0'} 2023-11-02T19:16:52.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd43b7055c3d876a-006543f5a4-2721-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:16:52.829 DEBUG:boto:path=/ibtljn-70/obj1 2023-11-02T19:16:52.830 DEBUG:boto:auth_path=/ibtljn-70/obj1 2023-11-02T19:16:52.830 DEBUG:boto:Method: HEAD 2023-11-02T19:16:52.831 DEBUG:boto:Path: /ibtljn-70/obj1 2023-11-02T19:16:52.831 DEBUG:boto:Data: 2023-11-02T19:16:52.832 DEBUG:boto:Headers: {} 2023-11-02T19:16:52.832 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:16:52.832 DEBUG:boto:Port: 8001 2023-11-02T19:16:52.833 DEBUG:boto:Params: {} 2023-11-02T19:16:52.833 DEBUG:boto:Token: None 2023-11-02T19:16:52.834 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:16:52 GMT /ibtljn-70/obj1 2023-11-02T19:16:52.834 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NzER9X+oj6reTdL03A0oPO9Nsdg= 2023-11-02T19:16:52.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:16:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NzER9X+oj6reTdL03A0oPO9Nsdg=', 'Content-Length': '0'} 2023-11-02T19:16:52.837 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000027cb5cfd915f45e4-006543f5a4-2721-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:16:52.838 DEBUG:boto:path=/ibtljn-70/obj2 2023-11-02T19:16:52.838 DEBUG:boto:auth_path=/ibtljn-70/obj2 2023-11-02T19:16:52.839 DEBUG:boto:Method: HEAD 2023-11-02T19:16:52.839 DEBUG:boto:Path: /ibtljn-70/obj2 2023-11-02T19:16:52.839 DEBUG:boto:Data: 2023-11-02T19:16:52.840 DEBUG:boto:Headers: {} 2023-11-02T19:16:52.840 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:16:52.841 DEBUG:boto:Port: 8001 2023-11-02T19:16:52.841 DEBUG:boto:Params: {} 2023-11-02T19:16:52.842 DEBUG:boto:Token: None 2023-11-02T19:16:52.842 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:16:52 GMT /ibtljn-70/obj2 2023-11-02T19:16:52.843 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:T3YBSFH1jyh3uY6A0hvwNWxhQJQ= 2023-11-02T19:16:52.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:16:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:T3YBSFH1jyh3uY6A0hvwNWxhQJQ=', 'Content-Length': '0'} 2023-11-02T19:16:52.845 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000063de13e78265f9e-006543f5a4-2721-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:16:52.846 DEBUG:boto:path=/ibtljn-71/ 2023-11-02T19:16:52.847 DEBUG:boto:auth_path=/ibtljn-71/ 2023-11-02T19:16:52.847 DEBUG:boto:Method: HEAD 2023-11-02T19:16:52.847 DEBUG:boto:Path: /ibtljn-71/ 2023-11-02T19:16:52.848 DEBUG:boto:Data: 2023-11-02T19:16:52.848 DEBUG:boto:Headers: {} 2023-11-02T19:16:52.849 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:16:52.849 DEBUG:boto:Port: 8001 2023-11-02T19:16:52.850 DEBUG:boto:Params: {} 2023-11-02T19:16:52.850 DEBUG:boto:Token: None 2023-11-02T19:16:52.851 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:16:52 GMT /ibtljn-71/ 2023-11-02T19:16:52.851 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:uTKyCGtxqCWLKWbJfKfmOIG1N5c= 2023-11-02T19:16:52.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:16:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uTKyCGtxqCWLKWbJfKfmOIG1N5c=', 'Content-Length': '0'} 2023-11-02T19:16:52.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000121cbc8af0c9f568-006543f5a4-2721-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:16:52.857 DEBUG:boto:path=/ibtljn-71/obj1 2023-11-02T19:16:52.857 DEBUG:boto:auth_path=/ibtljn-71/obj1 2023-11-02T19:16:52.858 DEBUG:boto:Method: HEAD 2023-11-02T19:16:52.858 DEBUG:boto:Path: /ibtljn-71/obj1 2023-11-02T19:16:52.859 DEBUG:boto:Data: 2023-11-02T19:16:52.859 DEBUG:boto:Headers: {} 2023-11-02T19:16:52.860 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:16:52.860 DEBUG:boto:Port: 8001 2023-11-02T19:16:52.861 DEBUG:boto:Params: {} 2023-11-02T19:16:52.861 DEBUG:boto:Token: None 2023-11-02T19:16:52.862 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:16:52 GMT /ibtljn-71/obj1 2023-11-02T19:16:52.862 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:aF52TFTSnlQkX2UlFUnkXWg1+Zg= 2023-11-02T19:16:52.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:16:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aF52TFTSnlQkX2UlFUnkXWg1+Zg=', 'Content-Length': '0'} 2023-11-02T19:16:52.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:14:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-70:d61b2bea-59a8-4454-8491-ac3afbd04937.9624.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000741a2b9d73e3f35c-006543f5a4-2721-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:16:52.867 DEBUG:boto:path=/ibtljn-71/obj1 2023-11-02T19:16:52.867 DEBUG:boto:auth_path=/ibtljn-71/obj1 2023-11-02T19:16:52.868 DEBUG:boto:Method: GET 2023-11-02T19:16:52.868 DEBUG:boto:Path: /ibtljn-71/obj1 2023-11-02T19:16:52.869 DEBUG:boto:Data: 2023-11-02T19:16:52.869 DEBUG:boto:Headers: {} 2023-11-02T19:16:52.870 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:16:52.870 DEBUG:boto:Port: 8001 2023-11-02T19:16:52.870 DEBUG:boto:Params: {} 2023-11-02T19:16:52.871 DEBUG:boto:Token: None 2023-11-02T19:16:52.871 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:16:52 GMT /ibtljn-71/obj1 2023-11-02T19:16:52.872 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ORcdvv3HwJVhL4spFofnX/+o3Ro= 2023-11-02T19:16:52.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:16:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ORcdvv3HwJVhL4spFofnX/+o3Ro=', 'Content-Length': '0'} 2023-11-02T19:16:52.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:14:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-70:d61b2bea-59a8-4454-8491-ac3afbd04937.9624.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b234aa1ef649cdf1-006543f5a4-2721-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:16:52.922 DEBUG:boto:path=/ibtljn-71/obj2 2023-11-02T19:16:52.923 DEBUG:boto:auth_path=/ibtljn-71/obj2 2023-11-02T19:16:52.924 DEBUG:boto:Method: HEAD 2023-11-02T19:16:52.924 DEBUG:boto:Path: /ibtljn-71/obj2 2023-11-02T19:16:52.924 DEBUG:boto:Data: 2023-11-02T19:16:52.925 DEBUG:boto:Headers: {} 2023-11-02T19:16:52.925 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:16:52.926 DEBUG:boto:Port: 8001 2023-11-02T19:16:52.926 DEBUG:boto:Params: {} 2023-11-02T19:16:52.927 DEBUG:boto:Token: None 2023-11-02T19:16:52.927 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:16:52 GMT /ibtljn-71/obj2 2023-11-02T19:16:52.928 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:I3UlVDa/s8ooDuFQcfq41gVpWmM= 2023-11-02T19:16:52.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:16:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:I3UlVDa/s8ooDuFQcfq41gVpWmM=', 'Content-Length': '0'} 2023-11-02T19:16:52.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:14:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-70:d61b2bea-59a8-4454-8491-ac3afbd04937.9624.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006bae4df9524cfa8-006543f5a4-2721-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:16:52.932 DEBUG:boto:path=/ibtljn-71/obj2 2023-11-02T19:16:52.932 DEBUG:boto:auth_path=/ibtljn-71/obj2 2023-11-02T19:16:52.933 DEBUG:boto:Method: GET 2023-11-02T19:16:52.933 DEBUG:boto:Path: /ibtljn-71/obj2 2023-11-02T19:16:52.934 DEBUG:boto:Data: 2023-11-02T19:16:52.934 DEBUG:boto:Headers: {} 2023-11-02T19:16:52.934 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:16:52.935 DEBUG:boto:Port: 8001 2023-11-02T19:16:52.935 DEBUG:boto:Params: {} 2023-11-02T19:16:52.936 DEBUG:boto:Token: None 2023-11-02T19:16:52.936 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:16:52 GMT /ibtljn-71/obj2 2023-11-02T19:16:52.937 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:xAyfF0ciikN2zpOCsfusrC9ar2M= 2023-11-02T19:16:52.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:16:52 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xAyfF0ciikN2zpOCsfusrC9ar2M=', 'Content-Length': '0'} 2023-11-02T19:16:52.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:14:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-70:d61b2bea-59a8-4454-8491-ac3afbd04937.9624.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007294cda8444269fb-006543f5a4-2721-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:16:52.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:52.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:52.942 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:16:52.991 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:53.092 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-11-02T19:16:53.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:53.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:53.093 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:16:53.141 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:53.142 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:53.237 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:16:53.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:53.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:53.239 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:16:53.284 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:53.284 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:53.385 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:16:53.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:53.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:53.386 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:16:53.430 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:53.430 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:53.551 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:16:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 19, 2023-11-02T19:16:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T19:16:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T19:16:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T19:16:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:16:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T19:16:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T19:16:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T19:16:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:16:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T19:16:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:16:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:16:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:16:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:16:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:16:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:16:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:16:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:16:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T19:16:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:16:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T19:16:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:16:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:16:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:16:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:16:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:16:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:16:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:16:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:16:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:16:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:16:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:16:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:16:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:16:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T19:16:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:16:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:16:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:16:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:16:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:16:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:16:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:16:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:16:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:16:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:16:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:16:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:16:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:16:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:16:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:16:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:16:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:16:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:16:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:16:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:16:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:16:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:16:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:16:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:16:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [ 2023-11-02T19:16:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-group", 2023-11-02T19:16:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: "data_flow": { 2023-11-02T19:16:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: "symmetrical": [ 2023-11-02T19:16:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-flow1", 2023-11-02T19:16:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:16:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:16:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:16:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: "pipes": [ 2023-11-02T19:16:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-pipe", 2023-11-02T19:16:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:16:53.572 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "*", 2023-11-02T19:16:53.572 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:16:53.572 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:16:53.572 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:16:53.572 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": { 2023-11-02T19:16:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "*", 2023-11-02T19:16:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:16:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:16:53.574 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:16:53.574 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: "params": { 2023-11-02T19:16:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:16:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: "filter": { 2023-11-02T19:16:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [] 2023-11-02T19:16:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": {}, 2023-11-02T19:16:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2023-11-02T19:16:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: "mode": "system", 2023-11-02T19:16:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "" 2023-11-02T19:16:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:16:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "allowed" 2023-11-02T19:16:53.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:53.578 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.578 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:16:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:16:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T19:16:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T19:16:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:16:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T19:16:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:16:53.581 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:16:53.581 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:16:53.581 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:16:53.581 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:16:53.582 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:16:53.582 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:16:53.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.583 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:16:53.583 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T19:16:53.583 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:16:53.585 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:16:53.585 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:16:53.585 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:16:53.586 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:16:53.586 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:16:53.586 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:16:53.586 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:16:53.586 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:16:53.587 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:16:53.587 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:16:53.587 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:16:53.587 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:16:53.588 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:16:53.588 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.589 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:16:53.589 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T19:16:53.589 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:16:53.589 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T19:16:53.589 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:16:53.590 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:16:53.590 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:16:53.590 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:16:53.590 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:16:53.591 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:16:53.591 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:16:53.591 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:16:53.591 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:16:53.592 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:16:53.592 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:16:53.592 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:16:53.592 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:53.593 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:16:53.593 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:16:53.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.594 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:16:53.594 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:16:53.594 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:16:53.594 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:16:53.595 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.595 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:53.595 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:16:53.595 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:16:53.596 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:16:53.596 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:16:53.596 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T19:16:53.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.596 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:16:53.597 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:16:53.597 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.597 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:53.597 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:16:53.598 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T19:16:53.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.598 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:16:53.598 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T19:16:53.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.599 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:16:53.599 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T19:16:53.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.601 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:16:53.601 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T19:16:53.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:53.602 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:16:53.602 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T19:16:53.603 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:53.603 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:53.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.603 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T19:16:53.604 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:16:53.604 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T19:16:53.604 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T19:16:53.604 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:16:53.604 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:16:53.605 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:16:53.605 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:16:53.605 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:16:53.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.606 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T19:16:53.606 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:16:53.606 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:16:53.607 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:16:53.607 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:16:53.607 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:16:53.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.608 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T19:16:53.608 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:16:53.608 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:16:53.608 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:16:53.608 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:16:53.609 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:16:53.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.609 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T19:16:53.609 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:16:53.610 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:16:53.610 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:16:53.610 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:16:53.610 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:16:53.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.611 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T19:16:53.611 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:16:53.611 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:16:53.612 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:16:53.612 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:16:53.612 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:16:53.612 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:53.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:53.613 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:16:53.613 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T19:16:53.613 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:16:53.613 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:53.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:53.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:53.615 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:16:53.689 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:53.689 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:53.786 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:16:53.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:53.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:53.788 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket ibtljn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:16:53.844 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:53.844 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:53.980 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-11-02T19:16:53.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:53.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:53.982 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket ibtljn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:16:54.045 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:54.045 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:54.134 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-11-02T19:16:54.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=ibtljn-70', '--source-bucket=*', '--dest-bucket=ibtljn-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:54.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=ibtljn-70', '--source-bucket=*', '--dest-bucket=ibtljn-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:54.135 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=ibtljn-70 '--source-bucket=*' --dest-bucket=ibtljn-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:16:54.191 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:54.192 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:54.290 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"ibtljn-71","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-11-02T19:16:54.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:54.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:54.291 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket ibtljn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:16:54.344 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:54.345 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:54.429 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"ibtljn-71","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-11-02T19:16:54.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:54.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:16:54.430 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket ibtljn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:16:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:54.566 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"ibtljn-71","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-11-02T19:16:54.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:54.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:54.568 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:54.672 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:16:54.701 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:16:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:16:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:16:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:16:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:16:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952614.402762_787.1", 2023-11-02T19:16:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:16:54.402762Z" 2023-11-02T19:16:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:16:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:16:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:54.705 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:16:54.706 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:16:54.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:16:54.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:54.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:54.708 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:54.809 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:16:54.806+0000 7f7340f55f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:16:54.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:16:54.806+0000 7f7340f55f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:16:54.839 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:16:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:16:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:16:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:16:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:16:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:16:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:16:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:54.843 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:16:54.843 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:54.843 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:16:54.843 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:16:54.843 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:16:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:16:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:16:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:16:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:16:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:16:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:16:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:16:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:16:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:16:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:16:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:16:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:16:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:16:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:16:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:16:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:16:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:16:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:16:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:16:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:16:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:16:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:16:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:16:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:16:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:16:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:54Z" 2023-11-02T19:16:54.855 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:16:54.857 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:16:54Z" } 2023-11-02T19:16:54.857 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:16:54.858 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:16:54.858 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:16:54.859 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:16:59.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:59.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:16:59.861 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:16:59.910 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:16:59.910 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:16:59.967 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:16:59.962+0000 7fc645d6ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:16:59.968 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:16:59.966+0000 7fc645d6ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:16:59.998 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:16:59.999 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:16:59.999 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:16:59.999 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:16:59.999 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:17:00.000 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:17:00.000 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:00.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:00.000 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:17:00.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:00.001 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:17:00.001 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:00.001 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:00.002 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:17:00.002 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:00.002 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:17:00.002 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:00.003 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:17:00.003 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:00.003 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:00.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:00.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:00.004 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:17:00.004 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:00.004 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:00.004 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:17:00.004 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:00.005 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:00.005 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:00.005 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:17:00.005 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:00.006 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:00.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:00.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:00.006 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:17:00.006 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:00.007 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:00.007 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:17:00.007 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:00.007 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:00.008 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:00.008 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:17:00.008 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:00.008 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:00.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:00.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:00.009 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:17:00.009 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:00.009 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:00.010 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:17:00.010 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:00.010 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:00.010 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:00.010 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:17:00.011 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:00.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:00.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:00.011 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:17:00.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:00.012 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:17:00.012 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:17:00.012 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:17:00.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:00.013 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:16:59Z" 2023-11-02T19:17:00.013 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:17:00.014 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:16:59Z" } 2023-11-02T19:17:00.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:17:00.015 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:00.015 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:00.016 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:17:05.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:05.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:05.018 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:17:05.065 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:17:05.065 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:17:05.126 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:05.122+0000 7efe8d1d6f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:05.127 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:05.122+0000 7efe8d1d6f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:05.157 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:17:05.157 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:17:05.157 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:17:05.157 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:17:05.158 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:17:05.158 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:17:05.158 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:05.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:05.159 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:17:05.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:05.159 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:17:05.159 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:17:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:17:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:05.161 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:17:05.161 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:05.161 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:05.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:05.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:05.162 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:17:05.162 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:05.162 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:05.163 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:17:05.163 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:05.163 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:05.163 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:05.163 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:17:05.164 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:05.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:05.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:05.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:17:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:17:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:17:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:17:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:17:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:17:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:17:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:17:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:17:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:17:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:17:05Z" 2023-11-02T19:17:05.171 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:17:05.172 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:17:05Z" } 2023-11-02T19:17:05.173 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:17:05.173 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:05.174 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:05.174 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:17:10.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:10.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:10.177 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:17:10.223 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:17:10.224 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:17:10.285 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:10.281+0000 7fc53d888f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:10.286 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:10.281+0000 7fc53d888f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:10.314 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:17:10.315 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:17:10.315 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:17:10.315 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:17:10.316 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:17:10.316 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:17:10.316 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:10.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:10.317 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:17:10.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:10.317 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:17:10.317 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:10.317 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:10.318 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:17:10.318 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:10.318 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:17:10.318 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:10.319 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:17:10.319 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:10.319 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:10.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:10.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:10.320 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:17:10.320 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:10.320 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:10.320 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:17:10.320 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:10.321 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:10.321 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:10.321 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:17:10.321 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:10.322 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:10.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:10.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:10.322 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:17:10.322 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:10.323 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:10.323 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:17:10.323 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:10.323 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:10.323 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:10.324 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:17:10.324 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:10.324 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:10.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:10.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:10.325 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:17:10.325 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:10.325 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:10.325 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:17:10.326 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:10.326 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:10.326 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:10.326 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:17:10.326 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:10.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:10.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:10.327 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:17:10.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:10.328 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:17:10.328 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:17:10.328 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:17:10.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:10.328 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:17:10Z" 2023-11-02T19:17:10.329 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:17:10.330 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:17:10Z" } 2023-11-02T19:17:10.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:17:10.331 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:10.331 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:10.332 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:17:15.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:15.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:15.334 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:17:15.382 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:17:15.382 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:17:15.440 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:15.433+0000 7fd46314bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:15.441 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:15.437+0000 7fd46314bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:15.472 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:17:15.472 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:17:15.472 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:17:15.473 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:17:15.473 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:17:15.473 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:17:15.473 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:15.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:15.474 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:17:15.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:15.474 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:17:15.475 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:15.475 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:15.475 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:17:15.475 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:15.475 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:17:15.476 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:15.476 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:17:15.476 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:15.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:15.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:15.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:15.477 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:17:15.477 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:15.477 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:15.478 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:17:15.478 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:15.478 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:15.478 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:15.478 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:17:15.479 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:15.479 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:15.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:15.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:15.479 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:17:15.480 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:15.480 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:15.480 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:17:15.480 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:15.481 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:15.481 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:15.481 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:17:15.481 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:15.481 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:15.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:15.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:15.482 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:17:15.482 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:15.482 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:15.483 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:17:15.483 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:15.483 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:15.483 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:15.484 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:17:15.484 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:15.484 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:15.484 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:15.484 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:17:15.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:15.485 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:17:15.485 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:17:15.485 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:17:15.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:15.486 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:17:15Z" 2023-11-02T19:17:15.486 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:17:15.487 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:17:15Z" } 2023-11-02T19:17:15.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:17:15.488 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:15.489 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:15.489 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:17:20.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:20.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:20.491 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:17:20.536 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:17:20.537 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:17:20.601 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:20.597+0000 7fda5a667f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:20.603 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:20.597+0000 7fda5a667f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:20.633 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:17:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:17:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:17:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:17:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:17:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:17:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:17:20.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:20.635 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:17:20.635 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:20.635 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:17:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:17:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:17:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:17:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:17:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:17:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:17:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:17:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:17:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:17:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:17:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:17:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:17:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:17:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:17:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:17:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:17:20Z" 2023-11-02T19:17:20.646 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:17:20.647 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:17:20Z" } 2023-11-02T19:17:20.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:17:20.649 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:20.649 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:20.649 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:17:25.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:25.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:25.652 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:17:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:17:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:17:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:25.749+0000 7fdfc3f78f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:25.753+0000 7fdfc3f78f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:25.787 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:17:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:17:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:17:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:17:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:17:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:17:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:17:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:17:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:17:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:17:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:17:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:17:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:17:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:17:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:17:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:17:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:17:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:17:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:17:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:17:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:17:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:17:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:17:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:17:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:17:25Z" 2023-11-02T19:17:25.803 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:17:25.804 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:17:25Z" } 2023-11-02T19:17:25.805 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:17:25.805 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:25.806 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:25.807 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:17:30.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:30.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:30.809 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:17:30.855 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:17:30.856 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:17:30.914 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:30.909+0000 7f46735d3f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:30.915 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:30.909+0000 7f46735d3f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:30.943 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:17:30.943 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:17:30.943 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:17:30.943 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:17:30.944 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:17:30.944 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:17:30.944 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:30.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:30.945 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:17:30.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:30.945 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:17:30.945 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:30.946 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:30.946 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:17:30.946 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:30.946 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:17:30.946 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:30.947 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:17:30.947 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:30.947 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:30.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:30.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:30.948 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:17:30.948 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:30.948 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:30.948 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:17:30.949 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:30.949 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:30.949 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:30.949 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:17:30.949 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:17:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:17:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:17:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:17:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:17:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:17:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:17:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:17:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:17:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:17:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:30.957 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:17:30Z" 2023-11-02T19:17:30.957 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:17:30.958 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:17:30Z" } 2023-11-02T19:17:30.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:17:30.959 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:30.960 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:30.960 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:17:35.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:35.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:35.963 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:17:36.010 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:17:36.011 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:17:36.072 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:36.065+0000 7f4b9867bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:36.074 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:36.069+0000 7f4b9867bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:36.101 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:17:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:17:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:17:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:17:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:17:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:17:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:17:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:17:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:17:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:17:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:17:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:17:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:17:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:17:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:17:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:17:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:17:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:17:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:17:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:17:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:17:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:17:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:17:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:17:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:17:36Z" 2023-11-02T19:17:36.116 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:17:36.118 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:17:36Z" } 2023-11-02T19:17:36.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:17:36.119 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:36.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:36.120 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:17:41.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:41.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:41.121 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:17:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:17:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:17:41.228 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:41.221+0000 7f307bfccf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:41.229 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:41.225+0000 7f307bfccf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:41.258 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:17:41.259 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:17:41.259 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:17:41.259 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:17:41.260 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:17:41.260 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:17:41.260 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:41.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:41.260 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:17:41.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:41.261 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:17:41.261 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:41.261 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:41.262 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:17:41.262 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:41.262 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:17:41.262 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:41.263 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:17:41.263 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:41.263 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:41.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:41.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:41.264 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:17:41.264 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:41.264 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:41.264 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:17:41.265 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:41.265 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:41.265 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:41.265 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:17:41.265 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:41.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:41.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:41.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:41.266 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:17:41.267 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:41.267 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:41.267 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:17:41.267 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:41.267 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:41.268 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:41.268 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:17:41.268 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:41.268 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:41.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:41.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:41.269 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:17:41.269 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:41.269 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:41.270 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:17:41.270 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:41.270 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:41.270 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:41.270 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:17:41.271 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:41.271 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:41.271 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:41.271 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:17:41.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:41.272 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:17:41.272 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:17:41.272 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:17:41.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:41.273 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:17:41Z" 2023-11-02T19:17:41.273 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:17:41.274 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:17:41Z" } 2023-11-02T19:17:41.275 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:17:41.275 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:41.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:41.276 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:17:46.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:46.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:46.278 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:17:46.324 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:17:46.324 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:17:46.387 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:46.381+0000 7f11d2f45f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:46.389 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:46.385+0000 7f11d2f45f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:46.431 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:17:46.432 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:17:46.432 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:17:46.432 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:17:46.432 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:17:46.433 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:17:46.433 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:46.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:46.433 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:17:46.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:46.434 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:17:46.434 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:46.434 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:46.435 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:17:46.435 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:46.435 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:17:46.435 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:46.435 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:17:46.437 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:46.437 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:46.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:46.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:46.438 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:17:46.438 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:46.438 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:46.439 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:17:46.439 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:46.439 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:46.439 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:46.439 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:17:46.440 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:46.440 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:46.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:46.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:46.441 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:17:46.441 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:46.441 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:46.441 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:17:46.442 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:46.442 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:46.442 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:46.442 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:17:46.442 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:46.443 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:46.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:46.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:46.443 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:17:46.444 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:46.444 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:46.444 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:17:46.444 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:46.444 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:46.445 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:46.445 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:17:46.445 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:46.445 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:46.445 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:46.446 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:17:46.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:46.446 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:17:46.446 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:17:46.447 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:17:46.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:46.447 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:17:46Z" 2023-11-02T19:17:46.447 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:17:46.448 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:17:46Z" } 2023-11-02T19:17:46.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:17:46.449 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:46.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:46.450 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:17:51.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:51.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:51.453 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:17:51.498 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:17:51.499 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:17:51.562 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:51.557+0000 7ff94bcfef40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:51.566 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:51.557+0000 7ff94bcfef40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:51.591 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:17:51.591 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:17:51.592 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:17:51.592 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:17:51.592 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:17:51.593 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:17:51.593 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:51.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:51.594 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:17:51.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:51.594 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:17:51.594 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:51.595 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:51.595 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:17:51.596 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:51.596 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:17:51.597 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:51.597 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:17:51.597 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:51.598 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:51.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:51.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:51.599 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:17:51.599 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:51.599 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:51.600 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:17:51.600 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:51.600 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:51.601 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:51.601 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:17:51.601 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:51.601 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:51.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:51.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:51.602 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:17:51.602 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:51.603 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:51.603 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:17:51.603 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:51.603 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:51.604 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:51.604 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:17:51.604 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:51.604 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:51.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:51.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:51.605 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:17:51.605 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:51.606 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:51.606 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:17:51.606 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:51.606 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:51.607 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:51.607 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:17:51.607 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:51.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:51.608 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:51.608 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:17:51.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:51.608 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:17:51.608 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:17:51.609 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:17:51.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:51.609 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:17:51Z" 2023-11-02T19:17:51.610 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:17:51.611 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:17:51Z" } 2023-11-02T19:17:51.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:17:51.612 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:51.613 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:51.613 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:17:56.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:56.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:17:56.616 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:17:56.664 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:17:56.665 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:17:56.722 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:56.717+0000 7f249a352f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:56.723 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:17:56.717+0000 7f249a352f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:17:56.743 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:17:56.744 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:17:56.744 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:17:56.744 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:17:56.744 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:17:56.745 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:17:56.745 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:56.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:56.745 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:17:56.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:56.746 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:17:56.746 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:56.746 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:56.747 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:17:56.747 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:56.747 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:17:56.747 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:56.747 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:17:56.748 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:56.748 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:56.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:56.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:56.749 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:17:56.749 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:56.749 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:56.749 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:17:56.750 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:56.750 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:56.750 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:56.750 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:17:56.750 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:56.751 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:56.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:56.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:56.751 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:17:56.752 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:56.752 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:56.752 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:17:56.752 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:56.753 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:56.753 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:56.753 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:17:56.753 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:56.753 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:56.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:56.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:17:56.754 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:17:56.754 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:17:56.755 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:17:56.755 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:17:56.755 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:17:56.755 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:17:56.755 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:17:56.756 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:17:56.756 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:17:56.756 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:56.756 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:17:56.757 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:17:56.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:56.757 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:17:56.757 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:17:56.757 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:17:56.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:17:56.758 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:17:56Z" 2023-11-02T19:17:56.758 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:17:56.759 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:17:56Z" } 2023-11-02T19:17:56.760 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:17:56.760 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:56.761 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:17:56.761 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:18:01.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:01.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:01.764 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:01.810 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:01.811 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:01.868 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:01.861+0000 7f1645ff1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:01.869 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:01.865+0000 7f1645ff1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:01.893 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:01.894 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:01.894 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:01.894 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:01.894 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:01.894 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:01.895 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:01.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:01.895 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:01.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:01.896 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:01.896 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:01.896 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:01.896 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:01.896 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:01.897 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:01.897 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:01.897 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:01.897 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:01.898 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:01.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:01.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:01.898 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:01.898 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:01.899 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:01.899 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:01.899 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:01.899 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:01.900 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:01.900 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:01.900 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:01.900 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:01.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:01.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:01.901 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:01.901 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:01.901 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:01.902 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:18:01.902 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:01.902 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:01.902 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:01.903 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:18:01.903 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:01.903 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:01.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:01.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:01.904 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:01.904 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:01.904 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:01.904 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:01.904 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:01.905 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:01.905 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:01.905 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:01.905 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:01.906 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:01.906 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:01.906 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:01.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:01.907 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:01.907 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:01.907 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:01.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:01.907 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:01Z" 2023-11-02T19:18:01.908 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:01.909 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:01Z" } 2023-11-02T19:18:01.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:01.910 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:01.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:01.911 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:18:06.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:06.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:06.913 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:06.959 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:06.959 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:07.009+0000 7f51f5b50f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:07.013+0000 7f51f5b50f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:07.046 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:07.046 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:07.046 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:07.047 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:07.047 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:07.047 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:07.047 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:07.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:07.048 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:07.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:07.048 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:07.048 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:07.049 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:07.049 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:07.049 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:07.049 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:07.050 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:07.050 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:07.050 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:07.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:07.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:07.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:07.051 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:07.051 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:07.051 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:07.051 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:07.052 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:07.052 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:07.052 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:07.052 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:07.052 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:07.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:07.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:07.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:07.053 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:07.053 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:07.054 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:07.054 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:18:07.054 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:07.054 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:07.054 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:07.055 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:18:07.055 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:07.055 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:07.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:07.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:07.056 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:07.056 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:07.056 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:07.056 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:07.057 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:07.057 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:07.057 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:07.057 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:07.057 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:07.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:07.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:07.058 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:07.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:07.058 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:07.059 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:07.059 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:07.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:07.059 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:07Z" 2023-11-02T19:18:07.060 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:07.061 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:07Z" } 2023-11-02T19:18:07.061 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:07.062 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:07.062 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:07.062 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:18:12.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:12.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:12.065 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:12.113 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:12.113 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:12.175 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:12.169+0000 7ff43e1b1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:12.177 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:12.173+0000 7ff43e1b1f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:12.205 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:18:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:18:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:12.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:12.216 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:12.216 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:12.216 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:12.218 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:12.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:12.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:12.218 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:12.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:12.219 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:12.219 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:12.219 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:12.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:12.220 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:12Z" 2023-11-02T19:18:12.220 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:12.221 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:12Z" } 2023-11-02T19:18:12.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:12.222 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:12.223 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:12.223 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:18:17.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:17.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:17.226 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:17.273 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:17.274 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:17.333 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:17.329+0000 7f7d98e0af40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:17.335 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:17.329+0000 7f7d98e0af40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:17.362 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:17.363 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:17.363 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:17.363 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:17.363 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:17.363 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:17.364 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:17.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:17.364 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:17.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:17.365 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:17.365 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:17.365 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:17.365 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:17.366 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:17.366 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:17.366 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:17.366 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:17.367 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:17.367 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:17.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:17.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:17.368 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:17.368 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:17.368 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:17.368 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:17.368 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:17.369 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:17.369 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:17.369 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:17.369 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:17.370 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:17.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:17.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:17.370 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:17.371 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:17.371 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:17.371 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:18:17.371 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:17.371 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:17.372 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:17.372 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:18:17.372 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:17.372 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:17.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:17.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:17.373 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:17.373 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:17.374 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:17.374 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:17.374 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:17.374 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:17.374 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:17.375 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:17.375 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:17.375 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:17.375 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:17.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:17.377 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:17Z" 2023-11-02T19:18:17.377 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:17.378 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:17Z" } 2023-11-02T19:18:17.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:17.379 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:17.380 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:17.380 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:18:22.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:22.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:22.383 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:22.430 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:22.430 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:22.492 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:22.485+0000 7f9f50c34f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:22.493 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:22.489+0000 7f9f50c34f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:22.522 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:22.523 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:22.523 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:22.523 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:22.523 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:22.524 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:22.524 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:22.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:22.524 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:22.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:22.525 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:22.525 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:22.525 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:22.525 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:22.526 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:22.526 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:22.526 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:22.526 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:22.526 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:22.527 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:22.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:22.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:22.529 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:22.529 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:22.530 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:22.530 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:22.530 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:22.530 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:22.531 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:22.531 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:22.531 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:22.531 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:22.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:22.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:22.532 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:22.532 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:22.533 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:22.533 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:18:22.533 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:22.533 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:22.533 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:22.534 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:18:22.534 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:22.534 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:22.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:22.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:22.535 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:22.535 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:22.535 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:22.536 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:22.536 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:22.536 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:22.536 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:22.537 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:22.537 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:22.537 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:22.537 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:22.537 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:22.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:22.538 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:22.538 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:22.538 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:22.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:22.539 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:22Z" 2023-11-02T19:18:22.539 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:22.540 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:22Z" } 2023-11-02T19:18:22.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:22.541 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:22.541 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:22.542 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:18:27.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:27.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:27.544 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:27.593 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:27.593 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:27.653 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:27.649+0000 7f9f50ee4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:27.655 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:27.649+0000 7f9f50ee4f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:27.681 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:27.682 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:27.682 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:27.682 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:27.683 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:27.683 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:27.683 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:27.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:27.683 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:27.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:27.684 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:27.684 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:27.684 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:27.684 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:27.685 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:27.685 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:27.685 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:27.685 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:27.686 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:27.686 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:27.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:27.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:27.686 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:27.687 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:27.687 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:27.687 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:27.687 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:27.688 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:27.688 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:27.688 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:27.688 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:27.688 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:27.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:27.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:27.689 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:27.689 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:27.689 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:27.690 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:18:27.690 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:27.690 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:27.690 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:27.691 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:18:27.691 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:27.691 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:27.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:27.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:27.692 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:27.692 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:27.692 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:27.692 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:27.693 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:27.693 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:27.693 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:27.693 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:27.693 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:27.694 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:27.694 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:27.694 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:27.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:27.694 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:27.695 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:27.695 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:27.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:27.695 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:27Z" 2023-11-02T19:18:27.696 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:27.697 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:27Z" } 2023-11-02T19:18:27.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:27.698 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:27.698 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:27.699 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:18:32.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:32.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:32.702 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:32.749 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:32.750 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:32.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:32.805+0000 7f28f68e8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:32.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:32.805+0000 7f28f68e8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:32.841 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:32.841 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:32.841 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:32.842 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:32.842 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:32.843 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:32.843 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:32.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:32.843 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:32.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:32.844 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:32.844 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:32.844 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:32.845 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:32.845 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:32.846 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:32.846 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:32.847 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:32.847 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:32.847 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:32.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:32.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:32.848 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:32.849 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:32.849 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:32.849 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:32.849 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:32.850 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:32.850 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:32.850 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:32.850 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:32.851 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:32.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:32.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:32.851 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:32.851 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:32.852 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:32.852 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:18:32.852 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:32.852 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:32.853 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:32.853 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:18:32.853 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:32.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:32.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:32.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:32.854 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:32.854 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:32.854 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:32.854 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:32.855 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:32.855 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:32.855 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:32.855 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:32.856 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:32.856 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:32.856 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:32.856 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:32.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:32.857 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:32.857 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:32.857 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:32.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:32.858 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:32Z" 2023-11-02T19:18:32.858 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:32.859 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:32Z" } 2023-11-02T19:18:32.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:32.861 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:32.861 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:32.862 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:18:37.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:37.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:37.865 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:37.912 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:37.913 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:37.973 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:37.969+0000 7fefc750bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:37.975 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:37.969+0000 7fefc750bf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:38.003 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:18:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:18:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:38.017 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:37Z" 2023-11-02T19:18:38.017 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:38.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:37Z" } 2023-11-02T19:18:38.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:38.019 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:38.019 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:38.020 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:18:43.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:43.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:43.025 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:43.074 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:43.074 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:43.128+0000 7f32acf38f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:43.132+0000 7f32acf38f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:43.165 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:43.169 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:43.169 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:43.169 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:43.169 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:43.169 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:43.170 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:43.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:43.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:43.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:43.171 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:43.171 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:43.171 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:43.171 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:18:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:18:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:43Z" 2023-11-02T19:18:43.180 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:43.181 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:43Z" } 2023-11-02T19:18:43.181 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:43.182 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:43.182 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:43.183 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:18:48.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:48.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:48.186 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:48.233 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:48.233 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:48.291 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:48.284+0000 7f59b4d64f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:48.293 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:48.288+0000 7f59b4d64f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:48.321 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:48.322 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:48.322 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:48.322 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:48.322 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:48.323 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:48.323 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:48.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:48.323 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:48.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:48.324 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:48.324 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:48.324 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:48.324 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:48.325 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:48.325 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:48.325 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:48.325 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:48.326 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:48.326 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:48.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:48.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:48.327 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:48.327 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:48.327 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:48.327 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:48.328 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:48.328 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:48.328 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:48.328 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:48.329 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:48.329 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:48.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:48.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:48.330 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:48.330 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:48.330 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:48.330 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:18:48.331 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:48.331 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:48.331 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:48.331 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:18:48.332 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:48.332 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:48.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:48.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:48.333 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:48.333 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:48.333 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:48.333 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:48.334 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:48.334 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:48.334 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:48.334 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:48.335 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:48.335 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:48.335 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:48.335 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:48.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:48.336 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:48.336 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:48.336 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:48.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:48.337 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:48Z" 2023-11-02T19:18:48.337 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:48.338 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:48Z" } 2023-11-02T19:18:48.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:48.339 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:48.340 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:48.340 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:18:53.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:53.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:53.343 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:53.388 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:53.388 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:53.450 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:53.444+0000 7fd72ba1ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:53.452 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:53.444+0000 7fd72ba1ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:53.481 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:53.481 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952481.792845_759.1", 2023-11-02T19:18:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:53.491 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:53.491 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:53.491 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:41.792845Z", 2023-11-02T19:18:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:53.495 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:53.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:53.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:53.495 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:53.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:53.496 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:53.496 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:53Z" 2023-11-02T19:18:53.498 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:53.499 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952481.792845_759.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:41.792845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:53Z" } 2023-11-02T19:18:53.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952481.792845_759.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:41.792845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:53.501 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:53.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952481.792845_759.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:53.502 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952614.402762_787.1 target=1_1698952481.792845_759.1 2023-11-02T19:18:58.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:58.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:58.505 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:58.552 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:58.552 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:58.614 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:58.608+0000 7f9ec5b03f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:58.617 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:58.608+0000 7f9ec5b03f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:58.645 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:58.645 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:58.645 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:58.646 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:58.646 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:58.646 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:58.647 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:58.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:58.647 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:58.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:58.648 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:58.648 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:58.648 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:58.648 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:58.649 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:58.649 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:58.649 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:58.649 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:58.650 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:58.650 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:58.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:58.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:58.650 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:58.651 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:58.651 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:58.651 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:58.651 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:58.651 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:58.652 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:58.652 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:58.652 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:58.652 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:58.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:58.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:58.653 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:58.653 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:58.653 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:58.654 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952614.402762_787.1", 2023-11-02T19:18:58.654 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:58.654 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:58.654 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:58.655 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:16:54.402762Z", 2023-11-02T19:18:58.655 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:58.655 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:58.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:58.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:58.656 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:58.656 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:58.656 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:58.656 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:58.657 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:58.657 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:58.657 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:58.657 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:58.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:58.659 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:58.659 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:58.660 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:58.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:58.660 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:58Z" 2023-11-02T19:18:58.660 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:58.661 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952614.402762_787.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:16:54.402762Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:58Z" } 2023-11-02T19:18:58.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952614.402762_787.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:16:54.402762Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:58.662 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:58.663 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:58.663 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:18:58.664 DEBUG:boto:path=/ibtljn-70/ 2023-11-02T19:18:58.665 DEBUG:boto:auth_path=/ibtljn-70/ 2023-11-02T19:18:58.665 DEBUG:boto:Method: HEAD 2023-11-02T19:18:58.666 DEBUG:boto:Path: /ibtljn-70/ 2023-11-02T19:18:58.667 DEBUG:boto:Data: 2023-11-02T19:18:58.667 DEBUG:boto:Headers: {} 2023-11-02T19:18:58.668 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:18:58.669 DEBUG:boto:Port: 8000 2023-11-02T19:18:58.669 DEBUG:boto:Params: {} 2023-11-02T19:18:58.670 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T19:18:58.671 DEBUG:boto:Token: None 2023-11-02T19:18:58.671 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:18:58 GMT /ibtljn-70/ 2023-11-02T19:18:58.672 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0JeXoA/63zlX8vQ4kqDVEVT1uM0= 2023-11-02T19:18:58.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:18:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0JeXoA/63zlX8vQ4kqDVEVT1uM0=', 'Content-Length': '0'} 2023-11-02T19:18:58.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069d6b74ff7e0a1ee-006543f622-282b-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:18:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:18:58.684 DEBUG:boto:path=/ibtljn-70/obj5 2023-11-02T19:18:58.684 DEBUG:boto:auth_path=/ibtljn-70/obj5 2023-11-02T19:18:58.685 DEBUG:boto:Method: PUT 2023-11-02T19:18:58.685 DEBUG:boto:Path: /ibtljn-70/obj5 2023-11-02T19:18:58.686 DEBUG:boto:Data: 2023-11-02T19:18:58.686 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:18:58.687 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:18:58.687 DEBUG:boto:Port: 8000 2023-11-02T19:18:58.687 DEBUG:boto:Params: {} 2023-11-02T19:18:58.688 DEBUG:boto:Token: None 2023-11-02T19:18:58.689 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:18:58 GMT /ibtljn-70/obj5 2023-11-02T19:18:58.689 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:CgEJ/kjWcdUbeQLoCdpNniU5vSo= 2023-11-02T19:18:58.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:18:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CgEJ/kjWcdUbeQLoCdpNniU5vSo='} 2023-11-02T19:18:58.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7ba5119b35f2d49-006543f622-282b-a1'), ('Date', 'Thu, 02 Nov 2023 19:18:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:18:58.708 DEBUG:boto:path=/ibtljn-70/ 2023-11-02T19:18:58.708 DEBUG:boto:auth_path=/ibtljn-70/ 2023-11-02T19:18:58.709 DEBUG:boto:Method: HEAD 2023-11-02T19:18:58.710 DEBUG:boto:Path: /ibtljn-70/ 2023-11-02T19:18:58.710 DEBUG:boto:Data: 2023-11-02T19:18:58.710 DEBUG:boto:Headers: {} 2023-11-02T19:18:58.711 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:18:58.711 DEBUG:boto:Port: 8000 2023-11-02T19:18:58.712 DEBUG:boto:Params: {} 2023-11-02T19:18:58.712 DEBUG:boto:Token: None 2023-11-02T19:18:58.713 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:18:58 GMT /ibtljn-70/ 2023-11-02T19:18:58.713 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:0JeXoA/63zlX8vQ4kqDVEVT1uM0= 2023-11-02T19:18:58.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:18:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0JeXoA/63zlX8vQ4kqDVEVT1uM0=', 'Content-Length': '0'} 2023-11-02T19:18:58.718 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ab82c2fd7daa91d-006543f622-282b-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:18:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:18:58.719 DEBUG:boto:path=/ibtljn-70/obj6 2023-11-02T19:18:58.719 DEBUG:boto:auth_path=/ibtljn-70/obj6 2023-11-02T19:18:58.720 DEBUG:boto:Method: PUT 2023-11-02T19:18:58.720 DEBUG:boto:Path: /ibtljn-70/obj6 2023-11-02T19:18:58.721 DEBUG:boto:Data: 2023-11-02T19:18:58.721 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:18:58.721 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:18:58.722 DEBUG:boto:Port: 8000 2023-11-02T19:18:58.722 DEBUG:boto:Params: {} 2023-11-02T19:18:58.723 DEBUG:boto:Token: None 2023-11-02T19:18:58.723 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:18:58 GMT /ibtljn-70/obj6 2023-11-02T19:18:58.724 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:fVukrBDgh/XbLugTIVkdFsH9fa0= 2023-11-02T19:18:58.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:18:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:fVukrBDgh/XbLugTIVkdFsH9fa0='} 2023-11-02T19:18:58.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df2fcd82f9fee649-006543f622-282b-a1'), ('Date', 'Thu, 02 Nov 2023 19:18:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:18:58.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:58.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:58.769 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:58.815 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:58.816 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:58.881 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:18:58.910 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:18:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:18:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:18:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952614.402762_787.1", 2023-11-02T19:18:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:16:54.402762Z" 2023-11-02T19:18:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:18:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:58.914 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:18:58.916 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:58.916 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:18:58.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:58.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:58.918 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:58.962 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:58.963 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:59.021 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:59.016+0000 7fb033755f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:59.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:18:59.016+0000 7fb033755f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:18:59.050 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:59.051 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:59.051 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:59.051 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:59.052 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:59.052 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:18:59.052 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:59.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.053 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:59.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:59.053 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:59.053 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:59.054 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:59.054 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:18:59.054 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:59.054 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:18:59.055 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:59.055 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:18:59.055 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:59.055 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:59.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:59.056 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:59.056 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:59.056 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:59.057 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:18:59.057 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:59.057 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:59.057 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:59.057 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:18:59.058 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:59.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:59.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:59.058 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:59.059 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:59.059 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:59.059 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952614.402762_787.1", 2023-11-02T19:18:59.059 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:59.060 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:59.060 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:59.060 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:16:54.402762Z", 2023-11-02T19:18:59.060 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:59.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:59.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:59.061 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:59.062 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:59.062 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:18:59.062 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:18:59.062 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:59.062 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:59.063 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:59.063 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:18:59.063 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:18:59.063 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:59.063 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:59.064 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:59.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.064 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:59.064 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:18:59.064 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:18:59.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.065 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:59Z" 2023-11-02T19:18:59.065 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:59.066 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952614.402762_787.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:16:54.402762Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:18:59Z" } 2023-11-02T19:18:59.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952614.402762_787.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:16:54.402762Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:18:59.067 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:59.068 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:18:59.068 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:18:59.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:59.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:59.070 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:59.262 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:18:59.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:59.263 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:18:59.263 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:18:59.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:59.263 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:18:59.264 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:18:59.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:59.264 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:18:59.265 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:18:59.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:59.265 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:18:59.265 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:18:59.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:59.266 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:18:59.267 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:18:59.267 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:18:59.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:59.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:18:59.269 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:18:59.320 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:18:59.321 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:18:59.412 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:18:59.413 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:18:59.413 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:18:59.413 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:18:59.413 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:18:59.414 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:18:59.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.414 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:18:59.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:59.415 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:18:59.415 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:59.416 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:18:59.416 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:18:59.416 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:59.416 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:18:59.417 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:59.417 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:18:59.417 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:59.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:59.418 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:18:59.419 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:59.419 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:18:59.420 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051387", 2023-11-02T19:18:59.420 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:59.420 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:18:59.421 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:59.421 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:48.238962Z" 2023-11-02T19:18:59.421 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:59.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:59.422 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:18:59.422 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:59.423 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:18:59.423 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:18:59.423 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:59.423 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:59.424 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:59.424 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:18:59.424 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:59.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:18:59.425 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:18:59.425 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:18:59.425 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:18:59.425 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045796", 2023-11-02T19:18:59.426 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:18:59.426 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:18:59.426 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:18:59.426 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:48.276692Z" 2023-11-02T19:18:59.427 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:59.427 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:18:59.427 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:18:59.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.427 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:18:59.428 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:18:59.428 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:18:59.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:18:59.428 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:18:59Z" 2023-11-02T19:18:59.429 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:18:59.430 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051387", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:14:48.238962Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:48.276692Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:18:59Z" } 2023-11-02T19:18:59.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051387', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:14:48.238962Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:48.276692Z'}}] 2023-11-02T19:18:59.431 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:18:59.431 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:18:59.432 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051527 target=00000000000000000000:00000000000000051387, shard=3 master=00000000000000000000:00000000000000045936 target=00000000000000000000:00000000000000045796 2023-11-02T19:19:04.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:04.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:04.435 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:04.483 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:04.484 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:04.575 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:04.576 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:04.576 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:04.576 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:04.576 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:04.577 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:19:04.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:04.577 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:04.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:04.577 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:04.578 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:04.578 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:04.578 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:19:04.578 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:04.579 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:19:04.579 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:04.579 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:19:04.579 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:04.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:04.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:04.580 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:04.580 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:04.580 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:04.581 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051387", 2023-11-02T19:19:04.581 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:04.581 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:19:04.581 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:04.582 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:48.238962Z" 2023-11-02T19:19:04.582 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:04.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:04.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:04.582 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:04.583 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:04.583 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:04.583 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:19:04.583 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:04.584 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:04.584 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:04.584 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:19:04.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:04.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:04.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:04.585 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:04.585 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:04.585 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:04.586 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045796", 2023-11-02T19:19:04.586 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:04.586 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:04.586 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:04.586 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:48.276692Z" 2023-11-02T19:19:04.587 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:04.587 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:04.587 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:04.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:04.588 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:04.588 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:19:04.588 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:19:04.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:04.588 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:04Z" 2023-11-02T19:19:04.589 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:04.590 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051387", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:14:48.238962Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:48.276692Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:19:04Z" } 2023-11-02T19:19:04.590 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051387', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:14:48.238962Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:48.276692Z'}}] 2023-11-02T19:19:04.591 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:19:04.591 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:19:04.591 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051527 target=00000000000000000000:00000000000000051387, shard=3 master=00000000000000000000:00000000000000045936 target=00000000000000000000:00000000000000045796 2023-11-02T19:19:09.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:09.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:09.593 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:09.641 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:09.642 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:09.736 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:09.736 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:09.736 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:09.736 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:09.737 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:09.737 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:19:09.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:09.737 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:09.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:09.738 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:09.738 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:09.738 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:09.738 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:19:09.739 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:09.739 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:19:09.739 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:09.739 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:19:09.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:09.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:09.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:09.740 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:09.740 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:09.741 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:09.741 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051387", 2023-11-02T19:19:09.741 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:09.741 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:19:09.741 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:09.742 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:48.238962Z" 2023-11-02T19:19:09.742 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:09.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:09.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:09.743 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:09.743 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:09.743 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:09.743 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:19:09.743 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:09.744 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:09.744 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:09.744 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:19:09.744 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:09.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:09.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:09.745 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:09.745 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:09.745 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:09.746 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045796", 2023-11-02T19:19:09.746 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:09.746 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:09.746 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:09.746 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:48.276692Z" 2023-11-02T19:19:09.747 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:09.747 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:09.747 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:09.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:09.748 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:09.748 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:19:09.748 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:19:09.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:09.748 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:09Z" 2023-11-02T19:19:09.749 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:09.750 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051387", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:14:48.238962Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:48.276692Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:19:09Z" } 2023-11-02T19:19:09.750 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051387', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:14:48.238962Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:48.276692Z'}}] 2023-11-02T19:19:09.751 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:19:09.751 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:19:09.751 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051527 target=00000000000000000000:00000000000000051387, shard=3 master=00000000000000000000:00000000000000045936 target=00000000000000000000:00000000000000045796 2023-11-02T19:19:14.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:14.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:14.754 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:14.802 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:14.802 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:14.893 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:14.893 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:14.893 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:14.893 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:14.894 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:14.894 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:19:14.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:14.894 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:14.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:14.895 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:14.895 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:14.895 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:14.895 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:19:14.896 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:14.896 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:19:14.896 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:14.896 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:19:14.897 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:14.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:14.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:14.897 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:14.898 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:14.898 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:14.898 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051387", 2023-11-02T19:19:14.898 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:14.899 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:19:14.899 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:14.899 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:48.238962Z" 2023-11-02T19:19:14.899 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:14.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:14.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:14.900 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:14.900 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:14.900 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:14.901 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:19:14.901 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:14.901 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:14.901 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:14.902 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:19:14.902 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:14.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:14.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:14.903 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:14.903 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:14.903 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:14.903 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045796", 2023-11-02T19:19:14.904 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:14.904 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:14.904 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:14.904 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:14:48.276692Z" 2023-11-02T19:19:14.904 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:14.905 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:14.905 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:14.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:14.905 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:14.906 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:19:14.906 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:19:14.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:14.906 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:14Z" 2023-11-02T19:19:14.907 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:14.908 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051387", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:14:48.238962Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:14:48.276692Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:19:14Z" } 2023-11-02T19:19:14.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051387', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:14:48.238962Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:14:48.276692Z'}}] 2023-11-02T19:19:14.909 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:19:14.909 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051387', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045796'} 2023-11-02T19:19:14.910 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051527 target=00000000000000000000:00000000000000051387, shard=3 master=00000000000000000000:00000000000000045936 target=00000000000000000000:00000000000000045796 2023-11-02T19:19:19.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:19.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:19.912 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:19.963 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:19.964 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:20.059 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:20.059 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:20.060 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:20.060 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:20.060 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:20.060 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:19:20.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.061 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:20.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:20.061 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:20.061 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:20.062 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:20.062 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032058", 2023-11-02T19:19:20.062 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:20.062 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:19:20.063 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:20.063 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:03:49.430491Z" 2023-11-02T19:19:20.063 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:20.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:20.064 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:20.064 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:20.064 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:20.064 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:19:20.065 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:20.065 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:19:20.065 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:20.065 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:19:20.066 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:20.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:20.066 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:20.067 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:20.067 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:20.067 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044747", 2023-11-02T19:19:20.067 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:20.067 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:20.068 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:20.068 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:10:55.965459Z" 2023-11-02T19:19:20.068 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:20.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:20.069 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:20.069 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:20.069 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:20.069 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:19:20.070 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:20.070 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:20.070 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:20.070 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:19:20.071 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:20.071 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:20.071 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:20.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.071 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:20.072 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:19:20.072 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:19:20.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.072 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:20Z" 2023-11-02T19:19:20.072 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:20.074 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032058", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:03:49.430491Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:10:55.965459Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:19:20Z" } 2023-11-02T19:19:20.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032058', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:03:49.430491Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:10:55.965459Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:19:20.075 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:19:20.075 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032058', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044747', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:19:20.076 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:19:20.076 DEBUG:boto:path=/ibtljn-70/ 2023-11-02T19:19:20.076 DEBUG:boto:auth_path=/ibtljn-70/ 2023-11-02T19:19:20.077 DEBUG:boto:Method: HEAD 2023-11-02T19:19:20.078 DEBUG:boto:Path: /ibtljn-70/ 2023-11-02T19:19:20.078 DEBUG:boto:Data: 2023-11-02T19:19:20.079 DEBUG:boto:Headers: {} 2023-11-02T19:19:20.079 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:20.080 DEBUG:boto:Port: 8001 2023-11-02T19:19:20.080 DEBUG:boto:Params: {} 2023-11-02T19:19:20.081 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:19:20.081 DEBUG:boto:Token: None 2023-11-02T19:19:20.082 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:20 GMT /ibtljn-70/ 2023-11-02T19:19:20.082 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:86PsXSmuED3eG9m/pn0bmvVh4uU= 2023-11-02T19:19:20.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:86PsXSmuED3eG9m/pn0bmvVh4uU=', 'Content-Length': '0'} 2023-11-02T19:19:20.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000212a46ae38641d9c-006543f638-2835-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:20.093 DEBUG:boto:path=/ibtljn-70/obj5 2023-11-02T19:19:20.094 DEBUG:boto:auth_path=/ibtljn-70/obj5 2023-11-02T19:19:20.094 DEBUG:boto:Method: HEAD 2023-11-02T19:19:20.094 DEBUG:boto:Path: /ibtljn-70/obj5 2023-11-02T19:19:20.095 DEBUG:boto:Data: 2023-11-02T19:19:20.095 DEBUG:boto:Headers: {} 2023-11-02T19:19:20.096 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:20.096 DEBUG:boto:Port: 8001 2023-11-02T19:19:20.096 DEBUG:boto:Params: {} 2023-11-02T19:19:20.097 DEBUG:boto:Token: None 2023-11-02T19:19:20.097 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:20 GMT /ibtljn-70/obj5 2023-11-02T19:19:20.099 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:y7kKnPkR1vA16FVSAf6wyRxdseA= 2023-11-02T19:19:20.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:y7kKnPkR1vA16FVSAf6wyRxdseA=', 'Content-Length': '0'} 2023-11-02T19:19:20.102 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000010defe1f7c7fe68f-006543f638-2835-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:20.102 DEBUG:boto:path=/ibtljn-70/obj6 2023-11-02T19:19:20.103 DEBUG:boto:auth_path=/ibtljn-70/obj6 2023-11-02T19:19:20.103 DEBUG:boto:Method: HEAD 2023-11-02T19:19:20.104 DEBUG:boto:Path: /ibtljn-70/obj6 2023-11-02T19:19:20.104 DEBUG:boto:Data: 2023-11-02T19:19:20.104 DEBUG:boto:Headers: {} 2023-11-02T19:19:20.105 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:20.105 DEBUG:boto:Port: 8001 2023-11-02T19:19:20.106 DEBUG:boto:Params: {} 2023-11-02T19:19:20.106 DEBUG:boto:Token: None 2023-11-02T19:19:20.107 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:20 GMT /ibtljn-70/obj6 2023-11-02T19:19:20.107 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:MgoImVJIe5t/BoCMNGV4J/7jo1Y= 2023-11-02T19:19:20.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:MgoImVJIe5t/BoCMNGV4J/7jo1Y=', 'Content-Length': '0'} 2023-11-02T19:19:20.111 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e4ab48333f78b46e-006543f638-2835-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:20.112 DEBUG:boto:path=/ibtljn-71/ 2023-11-02T19:19:20.112 DEBUG:boto:auth_path=/ibtljn-71/ 2023-11-02T19:19:20.113 DEBUG:boto:Method: HEAD 2023-11-02T19:19:20.114 DEBUG:boto:Path: /ibtljn-71/ 2023-11-02T19:19:20.114 DEBUG:boto:Data: 2023-11-02T19:19:20.116 DEBUG:boto:Headers: {} 2023-11-02T19:19:20.117 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:20.118 DEBUG:boto:Port: 8001 2023-11-02T19:19:20.118 DEBUG:boto:Params: {} 2023-11-02T19:19:20.119 DEBUG:boto:Token: None 2023-11-02T19:19:20.119 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:20 GMT /ibtljn-71/ 2023-11-02T19:19:20.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JDnrejuNOQAo9O37m4KAJKAFB9M= 2023-11-02T19:19:20.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JDnrejuNOQAo9O37m4KAJKAFB9M=', 'Content-Length': '0'} 2023-11-02T19:19:20.125 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074bd649ca93ceab9-006543f638-2835-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:20.126 DEBUG:boto:path=/ibtljn-71/obj5 2023-11-02T19:19:20.126 DEBUG:boto:auth_path=/ibtljn-71/obj5 2023-11-02T19:19:20.127 DEBUG:boto:Method: HEAD 2023-11-02T19:19:20.127 DEBUG:boto:Path: /ibtljn-71/obj5 2023-11-02T19:19:20.128 DEBUG:boto:Data: 2023-11-02T19:19:20.128 DEBUG:boto:Headers: {} 2023-11-02T19:19:20.128 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:20.129 DEBUG:boto:Port: 8001 2023-11-02T19:19:20.129 DEBUG:boto:Params: {} 2023-11-02T19:19:20.130 DEBUG:boto:Token: None 2023-11-02T19:19:20.130 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:20 GMT /ibtljn-71/obj5 2023-11-02T19:19:20.131 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LZqe0dna+Dr+HrAEeCZWj5H+bCI= 2023-11-02T19:19:20.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LZqe0dna+Dr+HrAEeCZWj5H+bCI=', 'Content-Length': '0'} 2023-11-02T19:19:20.135 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:18:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-70:d61b2bea-59a8-4454-8491-ac3afbd04937.9624.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff5700305442b791-006543f638-2835-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:20.136 DEBUG:boto:path=/ibtljn-71/obj5 2023-11-02T19:19:20.136 DEBUG:boto:auth_path=/ibtljn-71/obj5 2023-11-02T19:19:20.137 DEBUG:boto:Method: GET 2023-11-02T19:19:20.137 DEBUG:boto:Path: /ibtljn-71/obj5 2023-11-02T19:19:20.137 DEBUG:boto:Data: 2023-11-02T19:19:20.138 DEBUG:boto:Headers: {} 2023-11-02T19:19:20.138 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:20.139 DEBUG:boto:Port: 8001 2023-11-02T19:19:20.139 DEBUG:boto:Params: {} 2023-11-02T19:19:20.140 DEBUG:boto:Token: None 2023-11-02T19:19:20.140 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:19:20 GMT /ibtljn-71/obj5 2023-11-02T19:19:20.141 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8fH6jS4zoQn+goLQFeWf4ybVJEw= 2023-11-02T19:19:20.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8fH6jS4zoQn+goLQFeWf4ybVJEw=', 'Content-Length': '0'} 2023-11-02T19:19:20.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:18:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-70:d61b2bea-59a8-4454-8491-ac3afbd04937.9624.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000810a3a9e88f65d38-006543f638-2835-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:20.186 DEBUG:boto:path=/ibtljn-71/obj6 2023-11-02T19:19:20.187 DEBUG:boto:auth_path=/ibtljn-71/obj6 2023-11-02T19:19:20.187 DEBUG:boto:Method: HEAD 2023-11-02T19:19:20.188 DEBUG:boto:Path: /ibtljn-71/obj6 2023-11-02T19:19:20.188 DEBUG:boto:Data: 2023-11-02T19:19:20.189 DEBUG:boto:Headers: {} 2023-11-02T19:19:20.189 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:20.189 DEBUG:boto:Port: 8001 2023-11-02T19:19:20.190 DEBUG:boto:Params: {} 2023-11-02T19:19:20.190 DEBUG:boto:Token: None 2023-11-02T19:19:20.191 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:20 GMT /ibtljn-71/obj6 2023-11-02T19:19:20.191 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:sSavYQUVsKo0iN7sKB+iv9l3NWw= 2023-11-02T19:19:20.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sSavYQUVsKo0iN7sKB+iv9l3NWw=', 'Content-Length': '0'} 2023-11-02T19:19:20.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:18:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-70:d61b2bea-59a8-4454-8491-ac3afbd04937.9624.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075f012a8f0f3a111-006543f638-2835-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:20.195 DEBUG:boto:path=/ibtljn-71/obj6 2023-11-02T19:19:20.196 DEBUG:boto:auth_path=/ibtljn-71/obj6 2023-11-02T19:19:20.196 DEBUG:boto:Method: GET 2023-11-02T19:19:20.197 DEBUG:boto:Path: /ibtljn-71/obj6 2023-11-02T19:19:20.197 DEBUG:boto:Data: 2023-11-02T19:19:20.198 DEBUG:boto:Headers: {} 2023-11-02T19:19:20.198 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:20.198 DEBUG:boto:Port: 8001 2023-11-02T19:19:20.199 DEBUG:boto:Params: {} 2023-11-02T19:19:20.199 DEBUG:boto:Token: None 2023-11-02T19:19:20.200 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:19:20 GMT /ibtljn-71/obj6 2023-11-02T19:19:20.200 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:67sJsLji/m7FG49S/4Ub5SThr/A= 2023-11-02T19:19:20.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:67sJsLji/m7FG49S/4Ub5SThr/A=', 'Content-Length': '0'} 2023-11-02T19:19:20.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:18:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-70:d61b2bea-59a8-4454-8491-ac3afbd04937.9624.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a9adc9b6fa8bafb-006543f638-2835-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:20.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:20.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:20.206 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket ibtljn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:19:20.254 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:20.254 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:20.368 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_policy":{"groups":[]}} 2023-11-02T19:19:20.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:20.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:20.370 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:20.414 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:20.414 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:20.472 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:19:20.503 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:19:20.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:20.504 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:20.504 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:19:20.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:20.505 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:20.505 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:19:20.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:20.506 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952760.325437_789.1", 2023-11-02T19:19:20.506 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:19:20.325437Z" 2023-11-02T19:19:20.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:20.506 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:20.507 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:19:20.507 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:20.507 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:19:20.508 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952760.325437_789.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:20.509 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:19:20.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:20.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:20.510 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:20.556 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:20.556 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:20.616 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:20.608+0000 7f0a29948f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:20.617 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:20.612+0000 7f0a29948f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:20.645 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:19:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:19:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:19:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:19:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:20.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:20.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:20.654 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:20.654 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:20.655 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:20.655 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952614.402762_787.1", 2023-11-02T19:19:20.655 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:20.656 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:16:54.402762Z", 2023-11-02T19:19:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:19:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:19:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:19:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:20Z" 2023-11-02T19:19:20.663 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:20.664 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952614.402762_787.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:16:54.402762Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:19:20Z" } 2023-11-02T19:19:20.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952614.402762_787.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:16:54.402762Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:19:20.665 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952760.325437_789.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:20.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:20.666 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952760.325437_789.1 target=1_1698952614.402762_787.1 2023-11-02T19:19:25.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:25.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:25.668 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:25.715 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:25.715 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:25.766 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:25.760+0000 7f1a87448f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:25.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:25.760+0000 7f1a87448f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:25.794 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:19:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:19:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:19:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:19:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952614.402762_787.1", 2023-11-02T19:19:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:16:54.402762Z", 2023-11-02T19:19:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:19:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:19:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:19:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:25Z" 2023-11-02T19:19:25.808 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:25.809 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952614.402762_787.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:16:54.402762Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:19:25Z" } 2023-11-02T19:19:25.810 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952614.402762_787.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:16:54.402762Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:19:25.810 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952760.325437_789.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:25.811 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:25.811 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952760.325437_789.1 target=1_1698952614.402762_787.1 2023-11-02T19:19:30.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:30.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:30.813 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:30.858 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:30.859 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:30.918 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:30.912+0000 7fb50c3d5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:30.920 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:30.912+0000 7fb50c3d5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:30.949 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:30.949 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:19:30.950 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:30.951 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:30.952 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:19:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:19:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:30.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:30.954 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:30.955 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:19:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:30.956 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:30.957 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:30.957 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:30.957 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952614.402762_787.1", 2023-11-02T19:19:30.957 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:30.958 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:30.958 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:30.958 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:16:54.402762Z", 2023-11-02T19:19:30.958 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:30.958 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:30.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:30.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:30.959 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:30.959 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:30.960 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:30.960 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:30.960 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:30.960 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:30.960 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:19:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:19:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:19:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:30Z" 2023-11-02T19:19:30.963 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:30.964 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952614.402762_787.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:16:54.402762Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:19:30Z" } 2023-11-02T19:19:30.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952614.402762_787.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:16:54.402762Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:19:30.965 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952760.325437_789.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:30.965 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952614.402762_787.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:30.966 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952760.325437_789.1 target=1_1698952614.402762_787.1 2023-11-02T19:19:35.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:35.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:35.969 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:36.074 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:36.068+0000 7f6b65ad0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:36.075 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:36.068+0000 7f6b65ad0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:36.104 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:19:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:19:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:19:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:19:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952760.325437_789.1", 2023-11-02T19:19:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:20.325437Z", 2023-11-02T19:19:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:19:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:19:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:19:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:36Z" 2023-11-02T19:19:36.118 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:36.119 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952760.325437_789.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:20.325437Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:19:36Z" } 2023-11-02T19:19:36.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952760.325437_789.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:20.325437Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:19:36.120 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952760.325437_789.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:36.121 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952760.325437_789.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:36.121 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:19:36.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:36.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:36.122 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket ibtljn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:19:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:36.295 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[]} 2023-11-02T19:19:36.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:36.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:36.296 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket ibtljn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:19:36.347 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:36.671 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:36.673 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-11-02T19:19:36.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:36.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:36.674 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket ibtljn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:19:36.726 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:36.727 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:36.829 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-11-02T19:19:36.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=ibtljn-70', '--source-bucket=ibtljn-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:36.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=ibtljn-70', '--source-bucket=ibtljn-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:36.830 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=ibtljn-70 --source-bucket=ibtljn-71 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:19:36.881 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:36.881 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:36.998 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"ibtljn-71","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-11-02T19:19:36.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:36.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:36.999 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket ibtljn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:19:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:37.151 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"ibtljn-71","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-11-02T19:19:37.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:37.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:37.152 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket ibtljn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:19:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:37.293 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"ibtljn-71","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-11-02T19:19:37.293 DEBUG:boto:path=/ibtljn-70/ 2023-11-02T19:19:37.294 DEBUG:boto:auth_path=/ibtljn-70/ 2023-11-02T19:19:37.294 DEBUG:boto:Method: HEAD 2023-11-02T19:19:37.295 DEBUG:boto:Path: /ibtljn-70/ 2023-11-02T19:19:37.295 DEBUG:boto:Data: 2023-11-02T19:19:37.296 DEBUG:boto:Headers: {} 2023-11-02T19:19:37.296 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:37.296 DEBUG:boto:Port: 8001 2023-11-02T19:19:37.297 DEBUG:boto:Params: {} 2023-11-02T19:19:37.298 DEBUG:boto:Token: None 2023-11-02T19:19:37.298 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:37 GMT /ibtljn-70/ 2023-11-02T19:19:37.299 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTaxVH2IxkJ+zIkYNB+vwgpZcYQ= 2023-11-02T19:19:37.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTaxVH2IxkJ+zIkYNB+vwgpZcYQ=', 'Content-Length': '0'} 2023-11-02T19:19:37.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096fc2f4a311b3173-006543f649-2835-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:19:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:37.304 DEBUG:boto:path=/ibtljn-70/obj7 2023-11-02T19:19:37.304 DEBUG:boto:auth_path=/ibtljn-70/obj7 2023-11-02T19:19:37.305 DEBUG:boto:Method: PUT 2023-11-02T19:19:37.305 DEBUG:boto:Path: /ibtljn-70/obj7 2023-11-02T19:19:37.306 DEBUG:boto:Data: 2023-11-02T19:19:37.306 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:19:37.307 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:37.307 DEBUG:boto:Port: 8001 2023-11-02T19:19:37.307 DEBUG:boto:Params: {} 2023-11-02T19:19:37.308 DEBUG:boto:Token: None 2023-11-02T19:19:37.308 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:19:37 GMT /ibtljn-70/obj7 2023-11-02T19:19:37.309 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:NJlPIYnF1Oa9fi3mImd+UBh5jkg= 2023-11-02T19:19:37.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:19:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NJlPIYnF1Oa9fi3mImd+UBh5jkg='} 2023-11-02T19:19:37.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c2052374b6a9974-006543f649-2835-a2'), ('Date', 'Thu, 02 Nov 2023 19:19:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:37.355 DEBUG:boto:path=/ibtljn-70/ 2023-11-02T19:19:37.356 DEBUG:boto:auth_path=/ibtljn-70/ 2023-11-02T19:19:37.356 DEBUG:boto:Method: HEAD 2023-11-02T19:19:37.357 DEBUG:boto:Path: /ibtljn-70/ 2023-11-02T19:19:37.357 DEBUG:boto:Data: 2023-11-02T19:19:37.357 DEBUG:boto:Headers: {} 2023-11-02T19:19:37.358 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:37.358 DEBUG:boto:Port: 8001 2023-11-02T19:19:37.359 DEBUG:boto:Params: {} 2023-11-02T19:19:37.359 DEBUG:boto:Token: None 2023-11-02T19:19:37.360 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:37 GMT /ibtljn-70/ 2023-11-02T19:19:37.360 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tTaxVH2IxkJ+zIkYNB+vwgpZcYQ= 2023-11-02T19:19:37.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tTaxVH2IxkJ+zIkYNB+vwgpZcYQ=', 'Content-Length': '0'} 2023-11-02T19:19:37.365 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b02a9e18bb3afe78-006543f649-2835-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:19:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:37.365 DEBUG:boto:path=/ibtljn-70/obj8 2023-11-02T19:19:37.366 DEBUG:boto:auth_path=/ibtljn-70/obj8 2023-11-02T19:19:37.366 DEBUG:boto:Method: PUT 2023-11-02T19:19:37.367 DEBUG:boto:Path: /ibtljn-70/obj8 2023-11-02T19:19:37.367 DEBUG:boto:Data: 2023-11-02T19:19:37.368 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:19:37.368 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:37.369 DEBUG:boto:Port: 8001 2023-11-02T19:19:37.369 DEBUG:boto:Params: {} 2023-11-02T19:19:37.370 DEBUG:boto:Token: None 2023-11-02T19:19:37.370 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:19:37 GMT /ibtljn-70/obj8 2023-11-02T19:19:37.371 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:RgfiqI4Y7ztjpdC4CjIFSuw/Dzc= 2023-11-02T19:19:37.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:19:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RgfiqI4Y7ztjpdC4CjIFSuw/Dzc='} 2023-11-02T19:19:37.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089f7d3d0b30944d4-006543f649-2835-a2'), ('Date', 'Thu, 02 Nov 2023 19:19:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:37.386 DEBUG:boto:path=/ibtljn-71/ 2023-11-02T19:19:37.387 DEBUG:boto:auth_path=/ibtljn-71/ 2023-11-02T19:19:37.387 DEBUG:boto:Method: HEAD 2023-11-02T19:19:37.387 DEBUG:boto:Path: /ibtljn-71/ 2023-11-02T19:19:37.388 DEBUG:boto:Data: 2023-11-02T19:19:37.388 DEBUG:boto:Headers: {} 2023-11-02T19:19:37.388 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:37.389 DEBUG:boto:Port: 8001 2023-11-02T19:19:37.389 DEBUG:boto:Params: {} 2023-11-02T19:19:37.390 DEBUG:boto:Token: None 2023-11-02T19:19:37.390 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:37 GMT /ibtljn-71/ 2023-11-02T19:19:37.391 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S6ynTKM8n8+Iry62zOE3N9SqJeM= 2023-11-02T19:19:37.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S6ynTKM8n8+Iry62zOE3N9SqJeM=', 'Content-Length': '0'} 2023-11-02T19:19:37.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024d55527503e3b59-006543f649-2835-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:19:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:37.396 DEBUG:boto:path=/ibtljn-71/obj9 2023-11-02T19:19:37.396 DEBUG:boto:auth_path=/ibtljn-71/obj9 2023-11-02T19:19:37.396 DEBUG:boto:Method: PUT 2023-11-02T19:19:37.397 DEBUG:boto:Path: /ibtljn-71/obj9 2023-11-02T19:19:37.397 DEBUG:boto:Data: 2023-11-02T19:19:37.398 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:19:37.398 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:37.399 DEBUG:boto:Port: 8001 2023-11-02T19:19:37.399 DEBUG:boto:Params: {} 2023-11-02T19:19:37.399 DEBUG:boto:Token: None 2023-11-02T19:19:37.400 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:19:37 GMT /ibtljn-71/obj9 2023-11-02T19:19:37.400 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:E1U5WFIyxolZmOgXEoJ0pVUrqGY= 2023-11-02T19:19:37.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:19:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:E1U5WFIyxolZmOgXEoJ0pVUrqGY='} 2023-11-02T19:19:37.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a27242cfed3a324b-006543f649-2835-a2'), ('Date', 'Thu, 02 Nov 2023 19:19:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:37.451 DEBUG:boto:path=/ibtljn-71/ 2023-11-02T19:19:37.452 DEBUG:boto:auth_path=/ibtljn-71/ 2023-11-02T19:19:37.452 DEBUG:boto:Method: HEAD 2023-11-02T19:19:37.452 DEBUG:boto:Path: /ibtljn-71/ 2023-11-02T19:19:37.453 DEBUG:boto:Data: 2023-11-02T19:19:37.453 DEBUG:boto:Headers: {} 2023-11-02T19:19:37.454 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:37.454 DEBUG:boto:Port: 8001 2023-11-02T19:19:37.454 DEBUG:boto:Params: {} 2023-11-02T19:19:37.455 DEBUG:boto:Token: None 2023-11-02T19:19:37.455 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:37 GMT /ibtljn-71/ 2023-11-02T19:19:37.456 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:S6ynTKM8n8+Iry62zOE3N9SqJeM= 2023-11-02T19:19:37.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:S6ynTKM8n8+Iry62zOE3N9SqJeM=', 'Content-Length': '0'} 2023-11-02T19:19:37.460 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9e8317ace73ae8f-006543f649-2835-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:19:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:37.461 DEBUG:boto:path=/ibtljn-71/obj10 2023-11-02T19:19:37.461 DEBUG:boto:auth_path=/ibtljn-71/obj10 2023-11-02T19:19:37.462 DEBUG:boto:Method: PUT 2023-11-02T19:19:37.462 DEBUG:boto:Path: /ibtljn-71/obj10 2023-11-02T19:19:37.463 DEBUG:boto:Data: 2023-11-02T19:19:37.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:19:37.463 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:19:37.464 DEBUG:boto:Port: 8001 2023-11-02T19:19:37.464 DEBUG:boto:Params: {} 2023-11-02T19:19:37.465 DEBUG:boto:Token: None 2023-11-02T19:19:37.465 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:19:37 GMT /ibtljn-71/obj10 2023-11-02T19:19:37.466 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i8WGTPwcn1KPbN1jY5hridAZeNg= 2023-11-02T19:19:37.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:19:37 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i8WGTPwcn1KPbN1jY5hridAZeNg='} 2023-11-02T19:19:37.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf1ed18684237741-006543f649-2835-a2'), ('Date', 'Thu, 02 Nov 2023 19:19:37 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:37.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:37.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:37.481 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:37.526 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:37.526 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:37.589 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:19:37.618 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:19:37.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:37.619 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:37.619 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:19:37.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:37.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:37.620 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:37.620 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:19:37.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:37.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:37.621 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952777.117342_797.1", 2023-11-02T19:19:37.621 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:19:37.117342Z" 2023-11-02T19:19:37.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:37.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:37.622 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:37.622 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:19:37.622 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:37.623 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:19:37.624 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952777.117342_797.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:37.624 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:19:37.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:37.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:37.626 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:37.710 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:37.710 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:37.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:37.760+0000 7f52954cdf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:37.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:37.764+0000 7f52954cdf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:37.793 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:37.794 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:37.794 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:37.794 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:37.794 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:37.795 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:19:37.795 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:37.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:37.795 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:37.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:37.796 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:37.796 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:37.796 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:37.796 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:37.796 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:37.797 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:19:37.797 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:37.797 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:19:37.797 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:37.799 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:37.799 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:37.799 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:37.799 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:37.799 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:37.800 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:37.800 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:19:37.800 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:37.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:37.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:37.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:37.801 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:37.801 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:37.801 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:37.802 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952760.325437_789.1", 2023-11-02T19:19:37.802 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:37.802 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:37.802 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:37.802 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:20.325437Z", 2023-11-02T19:19:37.803 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:37.803 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:37.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:37.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:37.803 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:37.804 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:37.804 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:37.804 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:37.804 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:37.804 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:37.805 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:37.805 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:19:37.805 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:37.805 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:37.806 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:37.806 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:37.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:37.806 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:37.806 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:19:37.807 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:19:37.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:37.807 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:37Z" 2023-11-02T19:19:37.807 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:37.808 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952760.325437_789.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:20.325437Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:19:37Z" } 2023-11-02T19:19:37.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952760.325437_789.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:20.325437Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:19:37.809 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952777.117342_797.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:37.810 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952760.325437_789.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:37.810 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952777.117342_797.1 target=1_1698952760.325437_789.1 2023-11-02T19:19:42.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:42.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:42.812 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:42.857 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:42.858 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:42.919 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:42.912+0000 7f3e6ebf2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:42.921 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:42.916+0000 7f3e6ebf2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:42.949 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:42.949 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:42.950 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:42.950 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:42.950 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:42.950 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:19:42.950 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:42.953 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:19:42.953 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:42.953 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:19:42.953 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:42.954 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:42.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:42.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:42.954 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:42.955 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:42.955 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:42.955 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:42.955 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:42.955 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:42.956 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:42.956 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:19:42.956 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:42.956 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:42.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:42.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:42.957 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:42.957 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:42.958 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:42.958 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952760.325437_789.1", 2023-11-02T19:19:42.958 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:42.958 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:42.958 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:42.959 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:20.325437Z", 2023-11-02T19:19:42.959 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:42.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:42.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:42.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:42.960 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:42.960 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:42.960 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:42.960 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:42.960 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:42.961 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:42.961 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:42.961 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:19:42.961 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:42.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:42.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:42.962 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:42.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:42.962 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:42.962 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:19:42.963 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:19:42.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:42.963 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:42Z" 2023-11-02T19:19:42.963 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:42.964 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952760.325437_789.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:20.325437Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:19:42Z" } 2023-11-02T19:19:42.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952760.325437_789.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:20.325437Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:19:42.965 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952777.117342_797.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:42.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952760.325437_789.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:42.966 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952777.117342_797.1 target=1_1698952760.325437_789.1 2023-11-02T19:19:47.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:47.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:47.969 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:48.018 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:48.019 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:48.077 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:48.072+0000 7f0aecadff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:48.079 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:48.072+0000 7f0aecadff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:48.107 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:48.108 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:48.108 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:48.108 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:48.108 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:48.109 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:19:48.109 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:48.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:48.109 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:48.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:48.110 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:48.110 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:48.111 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:48.111 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:48.111 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:48.111 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:19:48.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:48.112 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:19:48.112 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:48.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:48.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:48.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:48.113 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:48.113 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:48.113 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:48.114 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:48.114 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:48.114 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:48.114 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:48.115 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:19:48.115 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:48.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:48.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:48.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:48.116 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:48.116 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:48.116 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:48.116 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952760.325437_789.1", 2023-11-02T19:19:48.117 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:48.117 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:48.117 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:48.117 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:20.325437Z", 2023-11-02T19:19:48.117 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:48.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:48.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:48.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:48.118 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:48.119 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:48.119 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:48.119 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:48.119 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:48.120 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:48.120 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:48.120 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:19:48.120 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:48.121 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:48.121 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:48.121 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:48.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:48.121 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:48.122 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:19:48.122 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:19:48.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:48.122 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:48Z" 2023-11-02T19:19:48.122 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:48.124 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952760.325437_789.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:20.325437Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:19:48Z" } 2023-11-02T19:19:48.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952760.325437_789.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:20.325437Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:19:48.125 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952777.117342_797.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:48.125 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952760.325437_789.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:48.125 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952777.117342_797.1 target=1_1698952760.325437_789.1 2023-11-02T19:19:53.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:53.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:53.130 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:53.177 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:53.177 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:53.240 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:53.232+0000 7f5e7329ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:53.241 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:53.236+0000 7f5e7329ff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:53.270 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:53.270 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:53.270 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:53.270 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:53.271 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:53.271 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:19:53.271 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:53.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:53.272 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:53.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:53.272 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:53.272 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:53.272 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:53.273 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:53.273 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:53.273 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:19:53.273 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:53.274 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:19:53.274 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:53.274 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:53.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:53.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:53.275 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:53.275 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:53.275 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:53.275 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:53.276 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:53.276 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:53.276 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:53.276 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:19:53.276 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:53.277 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:53.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:53.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:53.277 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:53.277 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:53.278 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:53.278 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952760.325437_789.1", 2023-11-02T19:19:53.278 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:53.278 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:53.279 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:53.279 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:20.325437Z", 2023-11-02T19:19:53.279 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:53.279 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:53.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:53.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:53.280 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:53.280 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:53.280 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:53.281 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:53.281 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:53.281 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:53.281 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:53.281 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:19:53.282 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:53.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:53.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:53.282 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:53.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:53.283 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:53.284 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:19:53.284 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:19:53.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:53.284 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:53Z" 2023-11-02T19:19:53.285 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:53.286 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952760.325437_789.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:20.325437Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:19:53Z" } 2023-11-02T19:19:53.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952760.325437_789.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:20.325437Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:19:53.287 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952777.117342_797.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:53.287 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952760.325437_789.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:53.288 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952777.117342_797.1 target=1_1698952760.325437_789.1 2023-11-02T19:19:58.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:58.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:58.290 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:58.336 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:58.337 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:58.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:58.388+0000 7fe113501f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:58.395 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:58.388+0000 7fe113501f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:58.423 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:58.424 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:58.424 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:58.424 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:58.424 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:58.425 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:19:58.425 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:58.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.425 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:58.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:58.426 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:58.426 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:58.426 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:58.427 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:58.427 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:58.427 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:19:58.427 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:58.427 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:19:58.428 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:58.428 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:58.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:58.429 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:58.429 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:58.429 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:58.429 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:58.429 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:58.430 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:58.430 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:58.430 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:19:58.430 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:58.431 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:58.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.431 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:58.431 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:58.431 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:58.432 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:58.432 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952777.117342_797.1", 2023-11-02T19:19:58.432 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:58.432 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:58.432 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:58.433 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:37.117342Z", 2023-11-02T19:19:58.433 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:58.433 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:58.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:58.434 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:58.434 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:58.434 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:58.434 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:58.435 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:58.435 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:58.435 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:58.435 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:19:58.436 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:58.436 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:58.436 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:58.437 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:58.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.437 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:58.437 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:19:58.438 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:19:58.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.438 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:58Z" 2023-11-02T19:19:58.438 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:58.440 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952777.117342_797.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:37.117342Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:19:58Z" } 2023-11-02T19:19:58.441 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952777.117342_797.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:37.117342Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:19:58.442 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952777.117342_797.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:58.442 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952777.117342_797.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:58.443 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:19:58.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:58.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:58.445 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:58.633 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:19:58.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:58.634 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000006850", 2023-11-02T19:19:58.634 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:19:37.476185Z" 2023-11-02T19:19:58.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:58.635 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000017274", 2023-11-02T19:19:58.635 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:19:37.320371Z" 2023-11-02T19:19:58.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:58.636 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000016300", 2023-11-02T19:19:58.636 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:19:37.409832Z" 2023-11-02T19:19:58.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:58.637 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000013797", 2023-11-02T19:19:58.637 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:19:14.965646Z" 2023-11-02T19:19:58.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:58.637 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:19:58.639 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000006850', 1: '00000000000000000000:00000000000000017274', 2: '00000000000000000000:00000000000000016300', 3: '00000000000000000000:00000000000000013797'} 2023-11-02T19:19:58.639 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T19:19:58.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:58.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:58.641 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:58.691 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:58.691 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:58.778 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:58.779 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:58.779 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:58.779 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:58.779 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:58.780 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 3858412311781056324 2023-11-02T19:19:58.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.780 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:58.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:58.781 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:58.781 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:58.781 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:58.781 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000006850", 2023-11-02T19:19:58.782 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:58.782 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:58.782 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:58.782 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:37.476185Z" 2023-11-02T19:19:58.783 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:58.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:58.783 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:58.783 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:58.784 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:58.784 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000017274", 2023-11-02T19:19:58.784 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:58.784 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:58.785 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:58.785 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:37.320371Z" 2023-11-02T19:19:58.785 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:58.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:58.786 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:58.786 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:58.786 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:58.786 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000016300", 2023-11-02T19:19:58.786 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:58.787 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:58.787 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:58.787 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:37.409832Z" 2023-11-02T19:19:58.787 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:58.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:58.788 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:58.788 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:58.789 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:19:58.789 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000013797", 2023-11-02T19:19:58.789 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:58.789 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:58.789 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:58.790 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:14.965646Z" 2023-11-02T19:19:58.790 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:58.790 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:58.790 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 0, 2023-11-02T19:19:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 0 2023-11-02T19:19:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:58.792 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:58Z" 2023-11-02T19:19:58.792 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:58.793 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3858412311781056324 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006850", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:37.476185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017274", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:37.320371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016300", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:37.409832Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:14.965646Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-11-02T19:19:58Z" } 2023-11-02T19:19:58.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006850', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:37.476185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:37.320371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016300', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:37.409832Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:14.965646Z'}}] 2023-11-02T19:19:58.794 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006850', 1: '00000000000000000000:00000000000000017274', 2: '00000000000000000000:00000000000000016300', 3: '00000000000000000000:00000000000000013797'} 2023-11-02T19:19:58.794 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006850', 1: '00000000000000000000:00000000000000017274', 2: '00000000000000000000:00000000000000016300', 3: '00000000000000000000:00000000000000013797'} 2023-11-02T19:19:58.795 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-11-02T19:19:58.795 DEBUG:boto:path=/ibtljn-70/ 2023-11-02T19:19:58.795 DEBUG:boto:auth_path=/ibtljn-70/ 2023-11-02T19:19:58.796 DEBUG:boto:Method: HEAD 2023-11-02T19:19:58.796 DEBUG:boto:Path: /ibtljn-70/ 2023-11-02T19:19:58.797 DEBUG:boto:Data: 2023-11-02T19:19:58.797 DEBUG:boto:Headers: {} 2023-11-02T19:19:58.798 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:19:58.798 DEBUG:boto:Port: 8000 2023-11-02T19:19:58.798 DEBUG:boto:Params: {} 2023-11-02T19:19:58.799 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T19:19:58.800 DEBUG:boto:Token: None 2023-11-02T19:19:58.800 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:58 GMT /ibtljn-70/ 2023-11-02T19:19:58.801 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:UJKE2Q0by3TZv9LGszZcsCOBcUc= 2023-11-02T19:19:58.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UJKE2Q0by3TZv9LGszZcsCOBcUc=', 'Content-Length': '0'} 2023-11-02T19:19:58.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1c79647e59aba4c-006543f65e-282b-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:58.808 DEBUG:boto:path=/ibtljn-70/obj7 2023-11-02T19:19:58.809 DEBUG:boto:auth_path=/ibtljn-70/obj7 2023-11-02T19:19:58.809 DEBUG:boto:Method: HEAD 2023-11-02T19:19:58.810 DEBUG:boto:Path: /ibtljn-70/obj7 2023-11-02T19:19:58.810 DEBUG:boto:Data: 2023-11-02T19:19:58.811 DEBUG:boto:Headers: {} 2023-11-02T19:19:58.811 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:19:58.811 DEBUG:boto:Port: 8000 2023-11-02T19:19:58.812 DEBUG:boto:Params: {} 2023-11-02T19:19:58.812 DEBUG:boto:Token: None 2023-11-02T19:19:58.813 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:58 GMT /ibtljn-70/obj7 2023-11-02T19:19:58.813 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:lpvtjqmazM7f0darWBk7GA8aAW4= 2023-11-02T19:19:58.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lpvtjqmazM7f0darWBk7GA8aAW4=', 'Content-Length': '0'} 2023-11-02T19:19:58.816 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002d9937b315cb64cf-006543f65e-282b-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:58.817 DEBUG:boto:path=/ibtljn-70/obj8 2023-11-02T19:19:58.817 DEBUG:boto:auth_path=/ibtljn-70/obj8 2023-11-02T19:19:58.818 DEBUG:boto:Method: HEAD 2023-11-02T19:19:58.818 DEBUG:boto:Path: /ibtljn-70/obj8 2023-11-02T19:19:58.819 DEBUG:boto:Data: 2023-11-02T19:19:58.819 DEBUG:boto:Headers: {} 2023-11-02T19:19:58.819 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:19:58.820 DEBUG:boto:Port: 8000 2023-11-02T19:19:58.820 DEBUG:boto:Params: {} 2023-11-02T19:19:58.821 DEBUG:boto:Token: None 2023-11-02T19:19:58.821 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:58 GMT /ibtljn-70/obj8 2023-11-02T19:19:58.822 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:cdkUMwAMSo3LA53Hb7vcZ+n+C6c= 2023-11-02T19:19:58.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:cdkUMwAMSo3LA53Hb7vcZ+n+C6c=', 'Content-Length': '0'} 2023-11-02T19:19:58.824 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000bcef66439e8b2aa5-006543f65e-282b-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:58.825 DEBUG:boto:path=/ibtljn-70/obj9 2023-11-02T19:19:58.825 DEBUG:boto:auth_path=/ibtljn-70/obj9 2023-11-02T19:19:58.826 DEBUG:boto:Method: HEAD 2023-11-02T19:19:58.826 DEBUG:boto:Path: /ibtljn-70/obj9 2023-11-02T19:19:58.827 DEBUG:boto:Data: 2023-11-02T19:19:58.827 DEBUG:boto:Headers: {} 2023-11-02T19:19:58.828 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:19:58.828 DEBUG:boto:Port: 8000 2023-11-02T19:19:58.828 DEBUG:boto:Params: {} 2023-11-02T19:19:58.829 DEBUG:boto:Token: None 2023-11-02T19:19:58.829 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:58 GMT /ibtljn-70/obj9 2023-11-02T19:19:58.830 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:kL27rXI0uWp4RCJa+yt0MSJYByY= 2023-11-02T19:19:58.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:kL27rXI0uWp4RCJa+yt0MSJYByY=', 'Content-Length': '0'} 2023-11-02T19:19:58.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:19:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-71:d61b2bea-59a8-4454-8491-ac3afbd04937.9624.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a14cc2d6e7e6c824-006543f65e-282b-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:58.834 DEBUG:boto:path=/ibtljn-70/obj9 2023-11-02T19:19:58.834 DEBUG:boto:auth_path=/ibtljn-70/obj9 2023-11-02T19:19:58.835 DEBUG:boto:Method: GET 2023-11-02T19:19:58.835 DEBUG:boto:Path: /ibtljn-70/obj9 2023-11-02T19:19:58.836 DEBUG:boto:Data: 2023-11-02T19:19:58.836 DEBUG:boto:Headers: {} 2023-11-02T19:19:58.837 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:19:58.837 DEBUG:boto:Port: 8000 2023-11-02T19:19:58.837 DEBUG:boto:Params: {} 2023-11-02T19:19:58.838 DEBUG:boto:Token: None 2023-11-02T19:19:58.838 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:19:58 GMT /ibtljn-70/obj9 2023-11-02T19:19:58.839 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:1zMSzpl97av+N8ugMMSry+JXMAM= 2023-11-02T19:19:58.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1zMSzpl97av+N8ugMMSry+JXMAM=', 'Content-Length': '0'} 2023-11-02T19:19:58.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:19:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-71:d61b2bea-59a8-4454-8491-ac3afbd04937.9624.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d758170e5350000-006543f65e-282b-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:58.843 DEBUG:boto:path=/ibtljn-70/obj10 2023-11-02T19:19:58.843 DEBUG:boto:auth_path=/ibtljn-70/obj10 2023-11-02T19:19:58.844 DEBUG:boto:Method: HEAD 2023-11-02T19:19:58.844 DEBUG:boto:Path: /ibtljn-70/obj10 2023-11-02T19:19:58.845 DEBUG:boto:Data: 2023-11-02T19:19:58.845 DEBUG:boto:Headers: {} 2023-11-02T19:19:58.845 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:19:58.846 DEBUG:boto:Port: 8000 2023-11-02T19:19:58.846 DEBUG:boto:Params: {} 2023-11-02T19:19:58.847 DEBUG:boto:Token: None 2023-11-02T19:19:58.847 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:19:58 GMT /ibtljn-70/obj10 2023-11-02T19:19:58.848 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Hdf6GJPPrsAdzn35LsSZTnhpILk= 2023-11-02T19:19:58.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Hdf6GJPPrsAdzn35LsSZTnhpILk=', 'Content-Length': '0'} 2023-11-02T19:19:58.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:19:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-71:d61b2bea-59a8-4454-8491-ac3afbd04937.9624.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007e90d7f0b462d79-006543f65e-282b-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:58.852 DEBUG:boto:path=/ibtljn-70/obj10 2023-11-02T19:19:58.852 DEBUG:boto:auth_path=/ibtljn-70/obj10 2023-11-02T19:19:58.853 DEBUG:boto:Method: GET 2023-11-02T19:19:58.853 DEBUG:boto:Path: /ibtljn-70/obj10 2023-11-02T19:19:58.853 DEBUG:boto:Data: 2023-11-02T19:19:58.854 DEBUG:boto:Headers: {} 2023-11-02T19:19:58.854 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:19:58.855 DEBUG:boto:Port: 8000 2023-11-02T19:19:58.855 DEBUG:boto:Params: {} 2023-11-02T19:19:58.857 DEBUG:boto:Token: None 2023-11-02T19:19:58.858 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:19:58 GMT /ibtljn-70/obj10 2023-11-02T19:19:58.858 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:i6iKLpvpFWU8U+Wb5i+Zui3PMMw= 2023-11-02T19:19:58.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:19:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:i6iKLpvpFWU8U+Wb5i+Zui3PMMw=', 'Content-Length': '0'} 2023-11-02T19:19:58.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:19:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '39a01552-6d59-4ce7-b7e5-915ca69b979a:ibtljn-71:d61b2bea-59a8-4454-8491-ac3afbd04937.9624.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e87efa242ed8891-006543f65e-282b-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:19:58.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:58.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:58.903 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket ibtljn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:19:58.951 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:58.951 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:59.065 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_policy":{"groups":[]}} 2023-11-02T19:19:59.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:59.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:19:59.067 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:19:59.107 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:59.107 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:59.207 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_policy":{"groups":[]}} 2023-11-02T19:19:59.208 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ok 2023-11-02T19:19:59.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:59.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:59.211 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:59.256 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:59.257 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:59.320 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:19:59.350 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:19:59.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:59.351 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:59.351 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:19:59.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:59.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:59.352 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:59.352 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:19:59.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:59.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:59.352 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952799.024424_800.1", 2023-11-02T19:19:59.353 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:19:59.024424Z" 2023-11-02T19:19:59.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:59.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:59.353 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:59.354 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:19:59.354 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:59.354 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:19:59.355 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952799.024424_800.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:59.356 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:19:59.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:59.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:19:59.357 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:19:59.402 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:19:59.402 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:19:59.462 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:59.456+0000 7f20a419ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:59.463 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:19:59.456+0000 7f20a419ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:19:59.492 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:19:59.492 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:19:59.492 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:19:59.492 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:19:59.493 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:19:59.493 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:19:59.493 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:59.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:59.494 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:19:59.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:59.494 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:19:59.494 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:59.494 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:59.495 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:19:59.495 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:59.495 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:19:59.495 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:59.496 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:19:59.496 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:59.496 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:59.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:59.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:59.497 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:19:59.497 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:59.497 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:59.497 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:19:59.498 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:59.498 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:59.498 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:59.498 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:19:59.499 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:59.499 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:59.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:59.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:59.500 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:19:59.500 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:59.500 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:59.500 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952777.117342_797.1", 2023-11-02T19:19:59.500 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:59.501 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:59.501 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:59.501 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:37.117342Z", 2023-11-02T19:19:59.502 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:59.502 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:59.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:59.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:19:59.502 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:19:59.503 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:19:59.503 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:19:59.503 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:19:59.503 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:19:59.504 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:19:59.504 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:19:59.504 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:19:59.504 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:19:59.504 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:59.505 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:19:59.505 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:19:59.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:59.505 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:19:59.505 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:19:59.506 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:19:59.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:19:59.506 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:19:59Z" 2023-11-02T19:19:59.506 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:19:59.507 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952777.117342_797.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:37.117342Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:19:59Z" } 2023-11-02T19:19:59.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952777.117342_797.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:37.117342Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:19:59.508 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952799.024424_800.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:59.509 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952777.117342_797.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:19:59.509 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952799.024424_800.1 target=1_1698952777.117342_797.1 2023-11-02T19:20:04.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:04.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:04.511 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:04.557 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:04.558 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:04.620 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:20:04.612+0000 7fec5771cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:20:04.622 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:20:04.616+0000 7fec5771cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:20:04.650 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:04.650 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:04.651 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:04.651 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:04.651 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:04.651 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:20:04.651 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:04.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:04.652 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:04.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:04.652 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:04.653 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:04.653 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:04.653 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:20:04.653 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:04.653 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:20:04.654 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:04.654 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:20:04.654 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:04.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:04.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:04.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:04.655 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:04.655 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:04.655 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:04.656 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:20:04.656 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:04.656 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:04.656 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:04.656 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:20:04.657 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:04.657 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:04.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:04.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:04.658 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:04.658 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:04.658 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:04.658 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952777.117342_797.1", 2023-11-02T19:20:04.658 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:04.659 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:04.659 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:04.659 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:37.117342Z", 2023-11-02T19:20:04.659 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:04.660 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:04.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:04.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:04.660 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:04.660 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:04.661 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:04.661 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:20:04.661 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:04.661 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:04.661 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:04.662 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:20:04.662 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:04.662 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:04.662 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:04.663 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:04.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:04.663 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:04.663 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:20:04.663 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:20:04.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:04.664 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:04Z" 2023-11-02T19:20:04.664 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:04.665 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952777.117342_797.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:37.117342Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:20:04Z" } 2023-11-02T19:20:04.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952777.117342_797.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:37.117342Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:20:04.666 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952799.024424_800.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:04.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952777.117342_797.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:04.667 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952799.024424_800.1 target=1_1698952777.117342_797.1 2023-11-02T19:20:09.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:09.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:09.669 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:09.717 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:09.717 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:09.774 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:20:09.768+0000 7fa784c42f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:20:09.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:20:09.768+0000 7fa784c42f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:20:09.801 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:09.801 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:09.801 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:20:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:09.804 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:09.804 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:09.804 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:20:09.804 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:09.805 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:20:09.805 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:09.805 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:20:09.805 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:20:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:20:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952777.117342_797.1", 2023-11-02T19:20:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:37.117342Z", 2023-11-02T19:20:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:09.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:09.812 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:09.812 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:09.812 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:20:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:20:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:20:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:20:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:09Z" 2023-11-02T19:20:09.816 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:09.817 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952777.117342_797.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:37.117342Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:20:09Z" } 2023-11-02T19:20:09.817 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952777.117342_797.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:37.117342Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:20:09.818 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952799.024424_800.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:09.818 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952777.117342_797.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:09.819 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952799.024424_800.1 target=1_1698952777.117342_797.1 2023-11-02T19:20:14.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:14.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:14.822 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:14.868 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:14.868 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:14.945 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:20:14.940+0000 7ff2581d2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:20:14.947 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:20:14.940+0000 7ff2581d2f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:20:14.971 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:14.972 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:14.972 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:14.972 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:14.972 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:14.973 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:20:14.973 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:14.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:14.973 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:14.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:14.974 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:14.974 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:14.974 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:14.975 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:20:14.975 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:14.975 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:20:14.975 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:14.976 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:20:14.976 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:14.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:14.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:14.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:14.977 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:14.977 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:14.977 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:14.978 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:20:14.978 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:14.978 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:14.978 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:14.978 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:20:14.979 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:14.979 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:14.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:14.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:14.980 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:14.980 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:14.980 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:14.980 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952777.117342_797.1", 2023-11-02T19:20:14.981 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:14.981 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:14.981 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:14.981 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:37.117342Z", 2023-11-02T19:20:14.981 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:14.982 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:14.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:14.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:14.982 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:14.983 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:14.983 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:14.983 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:20:14.983 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:14.984 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:14.984 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:14.984 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:20:14.984 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:14.985 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:14.985 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:14.985 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:14.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:14.985 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:14.986 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:20:14.986 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:20:14.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:14.987 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:14Z" 2023-11-02T19:20:14.988 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:14.989 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952777.117342_797.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:37.117342Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:20:14Z" } 2023-11-02T19:20:14.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952777.117342_797.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:37.117342Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:20:14.990 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952799.024424_800.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:14.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952777.117342_797.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:14.991 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952799.024424_800.1 target=1_1698952777.117342_797.1 2023-11-02T19:20:19.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:19.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:19.993 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:20.040 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:20.041 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:20.097 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:20:20.091+0000 7f7251199f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:20:20.099 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:20:20.091+0000 7f7251199f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:20:20.128 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:20.128 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:20.128 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:20.129 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:20.129 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:20.129 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:20:20.129 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:20.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.130 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:20.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:20.130 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:20.130 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:20.131 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:20.131 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:20:20.131 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:20.131 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:20:20.132 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:20.132 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:20:20.132 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:20.132 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:20.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:20.133 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:20.133 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:20.133 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:20.133 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:20:20.134 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:20.134 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:20.134 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:20.134 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:20:20.134 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:20.135 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:20.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:20.135 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:20.136 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:20.136 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:20.136 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952799.024424_800.1", 2023-11-02T19:20:20.136 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:20.136 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:20.137 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:20.137 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:59.024424Z", 2023-11-02T19:20:20.137 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:20.137 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:20.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:20.138 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:20.138 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:20.138 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:20.138 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:20:20.139 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:20.139 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:20.139 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:20.139 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:20:20.139 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:20.140 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:20.141 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:20.141 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:20.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.141 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:20.142 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:20:20.142 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:20:20.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.142 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:20Z" 2023-11-02T19:20:20.142 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:20.144 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952799.024424_800.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:59.024424Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:20:20Z" } 2023-11-02T19:20:20.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952799.024424_800.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:59.024424Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:20:20.145 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952799.024424_800.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:20.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952799.024424_800.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:20.145 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:20:20.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:20.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:20.149 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:20:20.192 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:20.193 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:20.288 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[]} 2023-11-02T19:20:20.289 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-72 2023-11-02T19:20:20.289 DEBUG:boto:path=/ibtljn-72/ 2023-11-02T19:20:20.290 DEBUG:boto:auth_path=/ibtljn-72/ 2023-11-02T19:20:20.290 DEBUG:boto:Method: PUT 2023-11-02T19:20:20.291 DEBUG:boto:Path: /ibtljn-72/ 2023-11-02T19:20:20.291 DEBUG:boto:Data: 2023-11-02T19:20:20.292 DEBUG:boto:Headers: {} 2023-11-02T19:20:20.292 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:20:20.292 DEBUG:boto:Port: 8000 2023-11-02T19:20:20.293 DEBUG:boto:Params: {} 2023-11-02T19:20:20.293 DEBUG:boto:Token: None 2023-11-02T19:20:20.294 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 19:20:20 GMT /ibtljn-72/ 2023-11-02T19:20:20.294 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:q9aQPaAMj5L9n1mUJFi3VJr8QoU= 2023-11-02T19:20:20.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:20:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:q9aQPaAMj5L9n1mUJFi3VJr8QoU=', 'Content-Length': '0'} 2023-11-02T19:20:20.327 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000130eb9048aef1dff-006543f674-282b-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:20:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:20:20.328 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-73 2023-11-02T19:20:20.329 DEBUG:boto:path=/ibtljn-73/ 2023-11-02T19:20:20.329 DEBUG:boto:auth_path=/ibtljn-73/ 2023-11-02T19:20:20.329 DEBUG:boto:Method: PUT 2023-11-02T19:20:20.330 DEBUG:boto:Path: /ibtljn-73/ 2023-11-02T19:20:20.330 DEBUG:boto:Data: 2023-11-02T19:20:20.331 DEBUG:boto:Headers: {} 2023-11-02T19:20:20.331 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:20:20.331 DEBUG:boto:Port: 8000 2023-11-02T19:20:20.332 DEBUG:boto:Params: {} 2023-11-02T19:20:20.332 DEBUG:boto:Token: None 2023-11-02T19:20:20.333 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 19:20:20 GMT /ibtljn-73/ 2023-11-02T19:20:20.333 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8aUcdfZUnEuTCPufHECwRRdplFo= 2023-11-02T19:20:20.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:20:20 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8aUcdfZUnEuTCPufHECwRRdplFo=', 'Content-Length': '0'} 2023-11-02T19:20:20.364 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000da4872e52bc0173b-006543f674-282b-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:20:20 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:20:20.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:20.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:20.365 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:20.405 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:20.406 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:20.465 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:20:20.493 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:20:20.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:20.494 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:20:20.494 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:20:20.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:20.495 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:20:20.495 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:20:20.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:20.496 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:20:20.496 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:20:20.360316Z" 2023-11-02T19:20:20.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:20.496 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:20:20.497 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:20:20.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:20.497 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:20:20.498 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:20.499 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:20:20.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:20.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:20.500 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:20.553 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:20.553 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:20.605 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:20:20.599+0000 7f78d50b5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:20:20.607 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:20:20.599+0000 7f78d50b5f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:20:20.635 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:20:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:20:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:20:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:20:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:20:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:20:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:20:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:20:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:20:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:20:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:20:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:20:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:20Z" 2023-11-02T19:20:20.651 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:20.652 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:20:20Z" } 2023-11-02T19:20:20.653 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:20:20.653 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:20.654 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:20.654 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:20:20.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:20.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:20.655 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:20:20.701 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:20.702 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:20.794 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-11-02T19:20:20.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:20.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:20.795 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:20:20.848 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:20.848 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:20.944 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2023-11-02T19:20:20.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-72', '--dest-bucket=ibtljn-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:20.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-72', '--dest-bucket=ibtljn-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:20.945 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-ibtljn-72 --source-zones=a1 --dest-zones=a2 --source-bucket=ibtljn-72 --dest-bucket=ibtljn-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:20:20.996 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:20.997 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:21.099 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-ibtljn-72","source":{"bucket":"ibtljn-72","zones":["a1"]},"dest":{"bucket":"ibtljn-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:20:21.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-73', '--dest-bucket=ibtljn-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:21.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-73', '--dest-bucket=ibtljn-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:21.101 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-ibtljn-73 --source-zones=a1 --dest-zones=a2 --source-bucket=ibtljn-73 --dest-bucket=ibtljn-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:20:21.194 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:21.194 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:21.291 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-ibtljn-72","source":{"bucket":"ibtljn-72","zones":["a1"]},"dest":{"bucket":"ibtljn-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-ibtljn-73","source":{"bucket":"ibtljn-73","zones":["a1"]},"dest":{"bucket":"ibtljn-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:20:21.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:21.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:21.293 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:20:21.337 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:21.338 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:21.437 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-ibtljn-72","source":{"bucket":"ibtljn-72","zones":["a1"]},"dest":{"bucket":"ibtljn-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-ibtljn-73","source":{"bucket":"ibtljn-73","zones":["a1"]},"dest":{"bucket":"ibtljn-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:20:21.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:21.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:21.438 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:20:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:21.603 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:21.604 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:20:21.604 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 20, 2023-11-02T19:20:21.604 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T19:20:21.604 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T19:20:21.605 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T19:20:21.605 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:20:21.605 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T19:20:21.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.606 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T19:20:21.606 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T19:20:21.606 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:20:21.607 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T19:20:21.607 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:20:21.607 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:20:21.607 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:20:21.608 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:20:21.608 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:20:21.608 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:20:21.608 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:20:21.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.609 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:20:21.609 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T19:20:21.609 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:20:21.610 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T19:20:21.610 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:20:21.610 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:20:21.610 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:20:21.611 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:20:21.611 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:20:21.611 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:20:21.611 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:20:21.612 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:20:21.612 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:20:21.612 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:20:21.612 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:20:21.613 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:20:21.613 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.614 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:20:21.614 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T19:20:21.614 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:20:21.615 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:20:21.615 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:20:21.615 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:20:21.615 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:20:21.616 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:20:21.616 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:20:21.616 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:20:21.616 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:20:21.617 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:20:21.617 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:20:21.617 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:20:21.617 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:20:21.617 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:20:21.618 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.618 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.618 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:20:21.618 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:20:21.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.619 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:20:21.619 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:20:21.619 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:20:21.620 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:20:21.620 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.620 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.620 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:20:21.621 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:20:21.621 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:20:21.621 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:20:21.621 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [ 2023-11-02T19:20:21.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.622 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-group", 2023-11-02T19:20:21.622 INFO:teuthology.orchestra.run.smithi017.stdout: "data_flow": { 2023-11-02T19:20:21.623 INFO:teuthology.orchestra.run.smithi017.stdout: "directional": [ 2023-11-02T19:20:21.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.623 INFO:teuthology.orchestra.run.smithi017.stdout: "source_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:20:21.623 INFO:teuthology.orchestra.run.smithi017.stdout: "dest_zone": "39a01552-6d59-4ce7-b7e5-915ca69b979a" 2023-11-02T19:20:21.624 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.624 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.624 INFO:teuthology.orchestra.run.smithi017.stdout: "pipes": [ 2023-11-02T19:20:21.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.625 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-pipe-ibtljn-72", 2023-11-02T19:20:21.625 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:20:21.625 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-72", 2023-11-02T19:20:21.626 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:20:21.626 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:20:21.626 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.627 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": { 2023-11-02T19:20:21.627 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-73", 2023-11-02T19:20:21.627 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:20:21.627 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a" 2023-11-02T19:20:21.628 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.628 INFO:teuthology.orchestra.run.smithi017.stdout: "params": { 2023-11-02T19:20:21.628 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:20:21.629 INFO:teuthology.orchestra.run.smithi017.stdout: "filter": { 2023-11-02T19:20:21.629 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [] 2023-11-02T19:20:21.629 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.630 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": {}, 2023-11-02T19:20:21.630 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2023-11-02T19:20:21.630 INFO:teuthology.orchestra.run.smithi017.stdout: "mode": "system", 2023-11-02T19:20:21.631 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "" 2023-11-02T19:20:21.631 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.632 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-pipe-ibtljn-73", 2023-11-02T19:20:21.632 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:20:21.632 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-73", 2023-11-02T19:20:21.632 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:20:21.633 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:20:21.633 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.633 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": { 2023-11-02T19:20:21.634 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-73", 2023-11-02T19:20:21.634 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:20:21.634 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a" 2023-11-02T19:20:21.634 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.635 INFO:teuthology.orchestra.run.smithi017.stdout: "params": { 2023-11-02T19:20:21.635 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:20:21.635 INFO:teuthology.orchestra.run.smithi017.stdout: "filter": { 2023-11-02T19:20:21.636 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [] 2023-11-02T19:20:21.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.636 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": {}, 2023-11-02T19:20:21.637 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2023-11-02T19:20:21.637 INFO:teuthology.orchestra.run.smithi017.stdout: "mode": "system", 2023-11-02T19:20:21.637 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "" 2023-11-02T19:20:21.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.638 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.638 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:20:21.638 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "enabled" 2023-11-02T19:20:21.638 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.639 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.639 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:20:21.639 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:20:21.640 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.640 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T19:20:21.640 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T19:20:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:20:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T19:20:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:20:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:20:21.642 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:20:21.642 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:20:21.642 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:20:21.642 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:20:21.643 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:20:21.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.643 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:20:21.644 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T19:20:21.644 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:20:21.644 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:20:21.644 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:20:21.645 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:20:21.645 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:20:21.645 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:20:21.645 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:20:21.646 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:20:21.646 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:20:21.646 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:20:21.646 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:20:21.646 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:20:21.647 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:20:21.647 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:20:21.647 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.648 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:20:21.648 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T19:20:21.648 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:20:21.649 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T19:20:21.649 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:20:21.649 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:20:21.650 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:20:21.650 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:20:21.650 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:20:21.650 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:20:21.651 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:20:21.651 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:20:21.651 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:20:21.652 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:20:21.653 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:20:21.653 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:20:21.653 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.654 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:20:21.654 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:20:21.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.655 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:20:21.655 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:20:21.655 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:20:21.655 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:20:21.656 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.656 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.656 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:20:21.656 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:20:21.657 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:20:21.657 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:20:21.657 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T19:20:21.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.658 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:20:21.658 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:20:21.658 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.658 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.659 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:20:21.659 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T19:20:21.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.660 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:20:21.660 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T19:20:21.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.660 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:20:21.661 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T19:20:21.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.662 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:20:21.662 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T19:20:21.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:21.663 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:20:21.663 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T19:20:21.663 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.663 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:21.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.664 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T19:20:21.664 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:20:21.664 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T19:20:21.665 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T19:20:21.665 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:20:21.665 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:20:21.665 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:20:21.666 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:20:21.666 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:20:21.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.667 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T19:20:21.667 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:20:21.667 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:20:21.667 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:20:21.668 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:20:21.668 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:20:21.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.668 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T19:20:21.668 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:20:21.669 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:20:21.669 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:20:21.669 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:20:21.669 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:20:21.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.670 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T19:20:21.670 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:20:21.670 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:20:21.671 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:20:21.671 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:20:21.671 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:20:21.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.672 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T19:20:21.672 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:20:21.672 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:20:21.672 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:20:21.673 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:20:21.673 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:20:21.673 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:21.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:21.674 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:20:21.674 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T19:20:21.674 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:21.674 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:21.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:21.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:20:21.676 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:20:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:21.796 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-ibtljn-72","source":{"bucket":"ibtljn-72","zones":["a1"]},"dest":{"bucket":"ibtljn-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-ibtljn-73","source":{"bucket":"ibtljn-73","zones":["a1"]},"dest":{"bucket":"ibtljn-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:20:21.797 DEBUG:boto:path=/ibtljn-72/ 2023-11-02T19:20:21.798 DEBUG:boto:auth_path=/ibtljn-72/ 2023-11-02T19:20:21.798 DEBUG:boto:Method: HEAD 2023-11-02T19:20:21.799 DEBUG:boto:Path: /ibtljn-72/ 2023-11-02T19:20:21.799 DEBUG:boto:Data: 2023-11-02T19:20:21.800 DEBUG:boto:Headers: {} 2023-11-02T19:20:21.800 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:20:21.800 DEBUG:boto:Port: 8000 2023-11-02T19:20:21.801 DEBUG:boto:Params: {} 2023-11-02T19:20:21.801 DEBUG:boto:Token: None 2023-11-02T19:20:21.802 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:20:21 GMT /ibtljn-72/ 2023-11-02T19:20:21.802 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x/0QNmU5GUoPX+lZeIwkbClYEK4= 2023-11-02T19:20:21.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:20:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x/0QNmU5GUoPX+lZeIwkbClYEK4=', 'Content-Length': '0'} 2023-11-02T19:20:21.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099252f8dcc17d1ad-006543f675-2a9d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:20:21.875 DEBUG:boto:path=/ibtljn-72/obj1 2023-11-02T19:20:21.875 DEBUG:boto:auth_path=/ibtljn-72/obj1 2023-11-02T19:20:21.876 DEBUG:boto:Method: PUT 2023-11-02T19:20:21.876 DEBUG:boto:Path: /ibtljn-72/obj1 2023-11-02T19:20:21.877 DEBUG:boto:Data: 2023-11-02T19:20:21.877 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:20:21.877 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:20:21.878 DEBUG:boto:Port: 8000 2023-11-02T19:20:21.878 DEBUG:boto:Params: {} 2023-11-02T19:20:21.879 DEBUG:boto:Token: None 2023-11-02T19:20:21.879 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:20:21 GMT /ibtljn-72/obj1 2023-11-02T19:20:21.880 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:AxDOPEWEVnL7k87hwB75EbZAxfg= 2023-11-02T19:20:21.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:20:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AxDOPEWEVnL7k87hwB75EbZAxfg='} 2023-11-02T19:20:21.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082fff1f7d6cfe9f4-006543f675-2a9d-a1'), ('Date', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:20:21.899 DEBUG:boto:path=/ibtljn-72/ 2023-11-02T19:20:21.900 DEBUG:boto:auth_path=/ibtljn-72/ 2023-11-02T19:20:21.900 DEBUG:boto:Method: HEAD 2023-11-02T19:20:21.901 DEBUG:boto:Path: /ibtljn-72/ 2023-11-02T19:20:21.901 DEBUG:boto:Data: 2023-11-02T19:20:21.901 DEBUG:boto:Headers: {} 2023-11-02T19:20:21.902 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:20:21.902 DEBUG:boto:Port: 8000 2023-11-02T19:20:21.903 DEBUG:boto:Params: {} 2023-11-02T19:20:21.903 DEBUG:boto:Token: None 2023-11-02T19:20:21.904 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:20:21 GMT /ibtljn-72/ 2023-11-02T19:20:21.904 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:x/0QNmU5GUoPX+lZeIwkbClYEK4= 2023-11-02T19:20:21.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:20:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:x/0QNmU5GUoPX+lZeIwkbClYEK4=', 'Content-Length': '0'} 2023-11-02T19:20:21.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000198572562a15ac55-006543f675-2a9d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:20:21.910 DEBUG:boto:path=/ibtljn-72/obj2 2023-11-02T19:20:21.911 DEBUG:boto:auth_path=/ibtljn-72/obj2 2023-11-02T19:20:21.911 DEBUG:boto:Method: PUT 2023-11-02T19:20:21.911 DEBUG:boto:Path: /ibtljn-72/obj2 2023-11-02T19:20:21.912 DEBUG:boto:Data: 2023-11-02T19:20:21.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:20:21.913 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:20:21.913 DEBUG:boto:Port: 8000 2023-11-02T19:20:21.914 DEBUG:boto:Params: {} 2023-11-02T19:20:21.914 DEBUG:boto:Token: None 2023-11-02T19:20:21.915 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:20:21 GMT /ibtljn-72/obj2 2023-11-02T19:20:21.915 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Nptca+FQDniV+EWXbpx5UErE9sM= 2023-11-02T19:20:21.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:20:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Nptca+FQDniV+EWXbpx5UErE9sM='} 2023-11-02T19:20:21.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018a69819dffda073-006543f675-2a9d-a1'), ('Date', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:20:21.959 DEBUG:boto:path=/ibtljn-73/ 2023-11-02T19:20:21.960 DEBUG:boto:auth_path=/ibtljn-73/ 2023-11-02T19:20:21.960 DEBUG:boto:Method: HEAD 2023-11-02T19:20:21.961 DEBUG:boto:Path: /ibtljn-73/ 2023-11-02T19:20:21.961 DEBUG:boto:Data: 2023-11-02T19:20:21.961 DEBUG:boto:Headers: {} 2023-11-02T19:20:21.962 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:20:21.962 DEBUG:boto:Port: 8000 2023-11-02T19:20:21.963 DEBUG:boto:Params: {} 2023-11-02T19:20:21.963 DEBUG:boto:Token: None 2023-11-02T19:20:21.964 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:20:21 GMT /ibtljn-73/ 2023-11-02T19:20:21.964 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bwqp7DawjnrwKlhj3TecC+ufXrA= 2023-11-02T19:20:21.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:20:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bwqp7DawjnrwKlhj3TecC+ufXrA=', 'Content-Length': '0'} 2023-11-02T19:20:21.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aee50d51c4094ea4-006543f675-2a9d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:20:21.970 DEBUG:boto:path=/ibtljn-73/obj3 2023-11-02T19:20:21.971 DEBUG:boto:auth_path=/ibtljn-73/obj3 2023-11-02T19:20:21.971 DEBUG:boto:Method: PUT 2023-11-02T19:20:21.972 DEBUG:boto:Path: /ibtljn-73/obj3 2023-11-02T19:20:21.972 DEBUG:boto:Data: 2023-11-02T19:20:21.973 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:20:21.973 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:20:21.973 DEBUG:boto:Port: 8000 2023-11-02T19:20:21.974 DEBUG:boto:Params: {} 2023-11-02T19:20:21.974 DEBUG:boto:Token: None 2023-11-02T19:20:21.975 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:20:21 GMT /ibtljn-73/obj3 2023-11-02T19:20:21.975 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:ZWiPNR5E9D101ZYIQW2bWsLCkwo= 2023-11-02T19:20:21.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:20:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ZWiPNR5E9D101ZYIQW2bWsLCkwo='} 2023-11-02T19:20:21.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002153d95b0a770ac9-006543f675-2a9d-a1'), ('Date', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:20:21.992 DEBUG:boto:path=/ibtljn-73/ 2023-11-02T19:20:21.992 DEBUG:boto:auth_path=/ibtljn-73/ 2023-11-02T19:20:21.993 DEBUG:boto:Method: HEAD 2023-11-02T19:20:21.993 DEBUG:boto:Path: /ibtljn-73/ 2023-11-02T19:20:21.994 DEBUG:boto:Data: 2023-11-02T19:20:21.994 DEBUG:boto:Headers: {} 2023-11-02T19:20:21.995 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:20:21.995 DEBUG:boto:Port: 8000 2023-11-02T19:20:21.995 DEBUG:boto:Params: {} 2023-11-02T19:20:21.996 DEBUG:boto:Token: None 2023-11-02T19:20:21.996 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:20:21 GMT /ibtljn-73/ 2023-11-02T19:20:21.997 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bwqp7DawjnrwKlhj3TecC+ufXrA= 2023-11-02T19:20:21.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:20:21 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bwqp7DawjnrwKlhj3TecC+ufXrA=', 'Content-Length': '0'} 2023-11-02T19:20:22.001 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f522c24ba5823e2-006543f675-2a9d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:20:22.002 DEBUG:boto:path=/ibtljn-73/obj4 2023-11-02T19:20:22.002 DEBUG:boto:auth_path=/ibtljn-73/obj4 2023-11-02T19:20:22.003 DEBUG:boto:Method: PUT 2023-11-02T19:20:22.003 DEBUG:boto:Path: /ibtljn-73/obj4 2023-11-02T19:20:22.004 DEBUG:boto:Data: 2023-11-02T19:20:22.004 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:20:22.004 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:20:22.005 DEBUG:boto:Port: 8000 2023-11-02T19:20:22.005 DEBUG:boto:Params: {} 2023-11-02T19:20:22.006 DEBUG:boto:Token: None 2023-11-02T19:20:22.006 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:20:22 GMT /ibtljn-73/obj4 2023-11-02T19:20:22.007 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:K1p/VIP04afAnBj8Bt+wsizUm/s= 2023-11-02T19:20:22.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:20:22 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:K1p/VIP04afAnBj8Bt+wsizUm/s='} 2023-11-02T19:20:22.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a9e5e201bf1e1ca-006543f676-2a9d-a1'), ('Date', 'Thu, 02 Nov 2023 19:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:20:22.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:22.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:22.052 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:22.108 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:22.109 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:22.199 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:20:22.233 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:20:22.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.234 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:20:22.234 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:20:22.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.235 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:20:22.235 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:20:22.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.236 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:20:22.236 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:20:20.360316Z" 2023-11-02T19:20:22.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.237 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:20:22.237 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:20:22.237 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:22.237 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:20:22.238 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:22.239 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:20:22.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:22.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:22.240 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:22.297 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:22.297 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:22.361 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:20:22.355+0000 7f0f6ebecf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:20:22.362 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:20:22.355+0000 7f0f6ebecf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:20:22.389 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:22.389 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:22.389 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:22.390 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:22.390 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:22.390 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:20:22.390 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:22.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.391 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:22.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.391 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:22.392 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:22.392 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:22.392 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:20:22.392 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:22.392 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:20:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:20:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:22.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:22.394 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:20:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:22.395 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:20:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:22.397 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:20:22.398 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:22.398 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:22.398 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:22.398 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:20:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.400 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:22.401 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:22.401 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:20:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:20:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:22.402 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:20:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:20:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:22.403 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.404 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:22.405 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:20:22.405 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:20:22.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.405 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:22Z" 2023-11-02T19:20:22.406 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:22.407 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:20:22Z" } 2023-11-02T19:20:22.407 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:20:22.408 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:22.408 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:20:22.409 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:20:22.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:22.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:22.410 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:22.457 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:22.457 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:22.541 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:20:22.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.542 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-11-02T19:20:22.542 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:20:22.019111Z" 2023-11-02T19:20:22.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.543 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051808", 2023-11-02T19:20:22.543 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:20:21.987436Z" 2023-11-02T19:20:22.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.544 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:20:22.544 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:20:22.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.544 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046076", 2023-11-02T19:20:22.545 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:20:21.926078Z" 2023-11-02T19:20:22.545 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:22.545 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:20:22.546 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:20:22.547 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:20:22.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:22.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:22.548 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:22.636 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:22.637 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:22.730 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:22.730 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:22.731 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:22.731 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:22.731 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:22.731 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:20:22.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.732 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:22.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.732 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:22.732 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:22.732 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:22.733 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:20:22.733 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:22.733 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:20:22.733 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:22.733 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:20:22.734 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:22.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.734 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:22.734 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:22.735 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:22.735 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:20:22.735 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:22.735 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:20:22.735 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:22.736 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:20:22.736 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:22.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.737 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:22.737 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:22.737 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:22.737 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:20:22.737 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:22.738 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:22.738 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:22.738 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:20:22.738 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:22.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:22.739 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:22.739 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:22.739 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:22.740 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:20:22.740 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:22.740 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:22.740 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:22.740 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:20:22.741 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:22.741 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:22.741 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:22.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.741 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:22.742 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:20:22.742 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:20:22.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:22.742 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:22Z" 2023-11-02T19:20:22.743 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:22.743 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:20:22Z" } 2023-11-02T19:20:22.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:20:22.744 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:20:22.745 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:20:22.745 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:20:27.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:27.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:27.747 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:27.793 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:27.793 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:27.884 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:27.884 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:27.884 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:27.885 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:27.885 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:27.885 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:20:27.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:27.885 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:27.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:27.886 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:27.886 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:27.886 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:27.887 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:20:27.887 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:27.887 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:20:27.887 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:27.887 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:20:27.888 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:27.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:27.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:27.888 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:27.888 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:27.889 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:27.889 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:20:27.889 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:27.889 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:20:27.889 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:27.890 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:20:27.890 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:27.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:27.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:27.891 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:27.891 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:27.891 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:27.891 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:20:27.891 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:27.891 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:27.892 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:27.892 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:20:27.892 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:27.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:27.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:27.893 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:27.893 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:27.893 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:27.893 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:20:27.893 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:27.894 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:27.894 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:27.894 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:20:27.894 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:27.894 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:27.895 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:27.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:27.895 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:27.895 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:20:27.896 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:20:27.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:27.896 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:27Z" 2023-11-02T19:20:27.896 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:27.897 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:20:27Z" } 2023-11-02T19:20:27.898 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:20:27.898 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:20:27.899 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:20:27.899 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:20:32.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:32.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:32.902 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:33.044 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:20:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:20:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:20:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:20:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:20:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:20:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:20:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:20:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:20:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:20:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:20:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:20:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:20:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:33Z" 2023-11-02T19:20:33.058 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:33.059 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:20:33Z" } 2023-11-02T19:20:33.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:20:33.060 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:20:33.061 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:20:33.061 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:20:38.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:38.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:38.063 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:38.114 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:38.114 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:38.201 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:38.202 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:38.202 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:38.202 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:38.202 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:38.203 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:20:38.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:38.203 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:38.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:38.204 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:38.204 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:38.204 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:38.204 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:20:38.205 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:38.205 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:20:38.205 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:38.205 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:20:38.206 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:38.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:38.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:38.206 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:38.207 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:38.207 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:38.207 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:20:38.208 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:38.208 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:20:38.208 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:38.208 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:20:38.208 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:38.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:38.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:38.209 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:38.209 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:38.210 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:38.210 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:20:38.211 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:38.211 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:38.211 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:38.211 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:20:38.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:38.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:38.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:38.212 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:38.213 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:38.213 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:38.213 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:20:38.213 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:38.213 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:38.214 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:38.214 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:20:38.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:38.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:38.215 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:38.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:38.215 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:38.215 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:20:38.215 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:20:38.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:38.216 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:38Z" 2023-11-02T19:20:38.216 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:38.217 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:20:38Z" } 2023-11-02T19:20:38.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:20:38.218 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:20:38.219 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:20:38.219 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:20:43.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:43.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:43.222 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:43.269 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:43.270 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:43.359 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:43.359 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:43.359 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:43.360 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:43.360 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:43.360 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:20:43.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:43.361 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:43.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:43.361 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:43.361 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:43.361 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:43.362 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:20:43.362 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:43.362 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:20:43.363 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:43.363 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:20:43.363 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:43.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:43.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:43.364 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:43.364 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:43.364 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:43.364 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:20:43.364 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:43.365 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:20:43.365 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:43.365 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:20:43.365 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:43.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:43.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:43.366 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:43.366 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:43.367 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:43.367 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:20:43.367 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:43.367 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:43.368 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:43.368 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:20:43.368 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:43.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:43.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:43.369 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:43.369 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:43.369 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:43.369 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:20:43.370 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:43.370 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:43.370 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:43.370 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:20:43.371 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:43.371 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:43.371 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:43.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:43.371 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:43.372 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:20:43.372 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:20:43.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:43.372 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:43Z" 2023-11-02T19:20:43.373 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:43.374 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:20:43Z" } 2023-11-02T19:20:43.375 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:20:43.375 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:20:43.375 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:20:43.376 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:20:48.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:48.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:48.379 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:48.427 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:48.427 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:48.512 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:48.512 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:48.512 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:48.513 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:48.513 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:48.513 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:20:48.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:48.514 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:48.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:48.514 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:48.514 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:48.515 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:48.515 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:20:48.515 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:48.515 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:20:48.515 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:48.516 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:20:48.516 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:48.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:48.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:48.517 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:48.517 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:48.517 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:48.517 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:20:48.518 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:48.518 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:20:48.518 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:48.518 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:20:48.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:48.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:48.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:48.519 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:48.520 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:48.520 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:48.520 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:20:48.520 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:48.520 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:48.521 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:48.521 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:20:48.521 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:48.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:48.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:48.522 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:48.522 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:48.522 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:48.523 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:20:48.523 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:48.523 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:48.523 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:48.523 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:20:48.524 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:48.524 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:48.524 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:48.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:48.525 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:48.525 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:20:48.525 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:20:48.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:48.526 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:48Z" 2023-11-02T19:20:48.526 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:48.527 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:20:48Z" } 2023-11-02T19:20:48.528 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:20:48.528 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:20:48.529 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:20:48.529 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:20:53.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:53.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:53.531 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:53.578 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:53.578 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:53.672 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:53.673 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:53.673 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:53.673 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:53.673 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:53.674 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:20:53.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:53.674 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:53.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:53.675 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:53.675 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:53.675 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:53.675 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:20:53.675 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:53.676 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:20:53.676 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:53.676 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:20:53.676 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:53.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:53.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:53.677 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:53.677 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:53.677 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:53.678 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:20:53.678 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:53.678 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:20:53.678 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:53.678 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:20:53.679 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:53.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:53.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:53.679 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:53.679 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:53.680 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:53.680 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:20:53.680 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:53.680 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:53.681 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:53.681 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:20:53.681 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:53.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:53.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:53.682 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:53.682 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:53.682 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:53.682 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:20:53.683 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:53.683 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:53.683 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:53.683 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:20:53.684 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:53.684 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:53.684 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:53.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:53.684 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:53.685 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:20:53.685 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:20:53.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:53.685 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:53Z" 2023-11-02T19:20:53.685 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:53.686 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:20:53Z" } 2023-11-02T19:20:53.687 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:20:53.688 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:20:53.688 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:20:53.689 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:20:58.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:58.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:20:58.692 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:20:58.740 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:20:58.740 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:20:58.833 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:20:58.833 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:20:58.834 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:20:58.834 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:20:58.835 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:20:58.835 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:20:58.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:58.836 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:20:58.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:58.837 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:20:58.837 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:58.838 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:58.838 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:20:58.839 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:58.839 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:20:58.840 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:58.840 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:20:58.841 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:58.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:58.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:58.842 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:20:58.843 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:58.843 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:58.843 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:20:58.844 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:58.844 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:20:58.845 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:58.845 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:20:58.846 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:58.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:58.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:58.847 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:20:58.847 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:58.848 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:58.848 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:20:58.849 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:58.849 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:58.849 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:58.850 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:20:58.850 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:58.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:58.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:20:58.852 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:20:58.852 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:20:58.853 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:20:58.854 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:20:58.854 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:20:58.854 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:20:58.854 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:20:58.855 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:20:58.855 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:58.855 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:20:58.855 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:20:58.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:58.856 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:20:58.856 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:20:58.856 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:20:58.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:20:58.857 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:20:58Z" 2023-11-02T19:20:58.857 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:20:58.858 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:20:58Z" } 2023-11-02T19:20:58.858 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:20:58.859 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:20:58.859 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:20:58.860 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:21:03.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:03.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:03.862 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:21:03.912 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:21:03.913 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:21:04.003 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:21:04.003 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:21:04.003 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:21:04.003 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:21:04.004 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:21:04.004 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:21:04.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:04.004 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:21:04.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:04.005 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:21:04.005 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:04.005 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:04.005 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:21:04.006 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:04.006 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:21:04.006 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:04.006 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:21:04.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:04.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:04.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:04.007 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:21:04.007 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:04.008 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:04.008 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:21:04.008 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:04.008 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:21:04.009 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:04.009 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:21:04.009 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:04.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:04.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:04.010 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:21:04.010 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:04.010 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:04.010 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:21:04.011 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:04.011 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:04.011 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:04.011 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:21:04.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:04.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:04.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:04.012 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:21:04.012 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:04.012 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:04.013 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:21:04.013 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:04.013 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:04.013 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:04.014 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:21:04.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:04.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:04.014 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:21:04.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:04.015 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:21:04.015 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:21:04.015 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:21:04.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:04.016 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:21:03Z" 2023-11-02T19:21:04.017 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:21:04.018 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:21:03Z" } 2023-11-02T19:21:04.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:21:04.019 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:21:04.019 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:21:04.020 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:21:09.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:09.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:09.022 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:21:09.072 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:21:09.072 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:21:09.166 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:21:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:21:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:21:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:21:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:21:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:21:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:21:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:21:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:21:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:21:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:21:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:21:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:21:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:21:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:21:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:21:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:09.174 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:21:09.174 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:09.174 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:09.174 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:21:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:21:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:21:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:21:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:21:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:21:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:21:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:21:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:21:09Z" 2023-11-02T19:21:09.179 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:21:09.179 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:21:09Z" } 2023-11-02T19:21:09.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:21:09.180 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:21:09.181 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:21:09.181 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:21:14.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:14.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:14.183 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:21:14.232 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:21:14.233 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:21:14.326 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:21:14.326 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:21:14.326 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:21:14.327 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:21:14.327 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:21:14.327 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:21:14.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:14.328 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:21:14.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:14.328 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:21:14.328 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:14.328 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:14.329 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:21:14.329 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:14.329 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:21:14.329 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:14.330 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:21:14.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:14.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:14.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:14.331 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:21:14.331 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:14.332 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:14.332 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:21:14.332 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:14.333 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:21:14.333 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:14.333 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:21:14.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:14.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:14.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:14.335 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:21:14.335 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:14.335 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:14.335 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:21:14.336 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:14.336 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:14.336 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:14.336 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:21:14.337 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:14.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:14.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:14.337 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:21:14.337 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:14.338 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:14.338 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:21:14.338 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:14.338 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:14.339 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:14.339 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:21:14.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:14.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:14.340 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:21:14.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:14.340 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:21:14.341 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:21:14.341 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:21:14.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:14.341 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:21:14Z" 2023-11-02T19:21:14.341 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:21:14.343 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:21:14Z" } 2023-11-02T19:21:14.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:21:14.344 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:21:14.344 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:21:14.345 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:21:19.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:19.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:19.347 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:21:19.395 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:21:19.396 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:21:19.486 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:21:19.486 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:21:19.486 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:21:19.487 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:21:19.487 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:21:19.487 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:21:19.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:19.487 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:21:19.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:19.488 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:21:19.488 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:19.488 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:19.489 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:21:19.489 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:19.489 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:21:19.489 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:19.489 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:21:19.490 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:19.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:19.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:19.490 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:21:19.491 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:19.491 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:19.491 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:21:19.491 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:19.492 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:21:19.492 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:19.492 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:21:19.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:19.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:19.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:19.493 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:21:19.493 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:19.493 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:19.494 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:21:19.494 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:19.494 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:19.495 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:19.495 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:21:19.496 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:19.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:19.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:19.496 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:21:19.496 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:19.497 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:19.497 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:21:19.497 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:19.497 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:19.498 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:19.498 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:21:19.498 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:19.498 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:19.498 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:21:19.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:19.499 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:21:19.499 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:21:19.499 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:21:19.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:19.500 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:21:19Z" 2023-11-02T19:21:19.500 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:21:19.501 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:21:19Z" } 2023-11-02T19:21:19.502 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:21:19.502 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:21:19.503 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:21:19.503 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:21:24.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:24.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:24.505 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:21:24.555 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:21:24.556 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:21:24.641 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:21:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:21:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:21:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:21:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:21:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:21:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:21:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:21:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:21:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:21:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:21:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:21:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:21:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:21:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:21:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:21:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:21:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:21:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:21:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:21:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:21:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:21:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:21:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:21:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:21:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:21:24Z" 2023-11-02T19:21:24.655 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:21:24.656 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:21:24Z" } 2023-11-02T19:21:24.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:21:24.657 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:21:24.657 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:21:24.658 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:21:29.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:29.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:29.660 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:21:29.709 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:21:29.709 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:21:29.791 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:21:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:21:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:21:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:21:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:21:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:21:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:21:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:21:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:21:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:21:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:21:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:21:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:21:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:21:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:21:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:21:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:21:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:21:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:21:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:21:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:21:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:21:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:21:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:21:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:21:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:21:29Z" 2023-11-02T19:21:29.805 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:21:29.806 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:21:29Z" } 2023-11-02T19:21:29.807 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:21:29.807 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:21:29.808 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:21:29.808 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:21:34.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:34.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:34.810 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:21:34.858 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:21:34.858 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:21:34.949 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:21:34.950 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:21:34.950 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:21:34.950 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:21:34.951 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:21:34.951 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:21:34.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:34.951 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:21:34.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:34.952 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:21:34.952 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:34.952 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:34.952 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:21:34.953 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:34.953 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:21:34.953 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:34.953 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:21:34.954 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:34.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:34.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:34.954 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:21:34.955 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:34.955 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:34.955 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:21:34.955 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:34.955 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:21:34.956 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:34.956 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:21:34.956 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:34.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:34.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:34.957 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:21:34.957 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:34.957 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:34.957 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:21:34.958 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:34.958 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:34.958 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:34.958 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:21:34.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:34.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:34.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:34.959 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:21:34.960 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:34.960 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:34.960 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:21:34.960 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:34.961 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:34.961 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:34.961 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:21:34.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:34.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:34.962 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:21:34.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:34.963 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:21:34.963 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:21:34.963 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:21:34.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:34.964 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:21:34Z" 2023-11-02T19:21:34.964 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:21:34.965 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:21:34Z" } 2023-11-02T19:21:34.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:21:34.966 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:21:34.966 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:21:34.967 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:21:39.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:39.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:39.969 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:21:40.021 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:21:40.021 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:21:40.112 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:21:40.113 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:21:40.113 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:21:40.113 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:21:40.113 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:21:40.114 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:21:40.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:40.114 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:21:40.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:40.115 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:21:40.115 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:40.115 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:40.115 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:21:40.116 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:40.116 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:21:40.116 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:40.116 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:21:40.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:40.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:40.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:40.117 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:21:40.117 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:40.117 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:40.118 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:21:40.118 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:40.118 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:21:40.118 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:40.119 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:21:40.119 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:40.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:40.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:40.119 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:21:40.120 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:40.120 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:40.120 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:21:40.120 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:40.121 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:40.121 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:40.121 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:21:40.121 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:40.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:40.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:40.122 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:21:40.122 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:40.122 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:40.122 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:21:40.123 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:40.123 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:40.123 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:40.123 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:21:40.124 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:40.124 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:40.124 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:21:40.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:40.124 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:21:40.125 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:21:40.125 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:21:40.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:40.125 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:21:40Z" 2023-11-02T19:21:40.125 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:21:40.127 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:21:40Z" } 2023-11-02T19:21:40.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:21:40.128 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:21:40.128 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:21:40.128 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:21:45.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:45.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:45.131 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:21:45.180 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:21:45.181 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:21:45.273 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:21:45.274 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:21:45.274 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:21:45.274 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:21:45.275 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:21:45.275 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:21:45.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:45.275 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:21:45.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:45.276 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:21:45.276 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:45.276 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:45.276 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:21:45.277 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:45.277 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:21:45.277 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:45.277 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:21:45.278 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:45.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:45.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:45.278 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:21:45.278 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:45.279 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:45.279 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:21:45.279 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:45.280 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:21:45.280 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:45.280 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:21:45.280 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:45.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:45.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:45.281 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:21:45.281 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:45.281 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:45.281 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:21:45.282 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:45.282 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:45.282 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:45.282 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:21:45.283 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:45.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:45.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:45.283 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:21:45.283 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:45.284 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:45.284 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:21:45.284 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:45.284 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:45.284 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:45.285 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:21:45.285 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:45.285 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:45.285 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:21:45.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:45.286 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:21:45.286 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:21:45.286 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:21:45.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:45.286 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:21:45Z" 2023-11-02T19:21:45.287 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:21:45.288 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:21:45Z" } 2023-11-02T19:21:45.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:21:45.289 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:21:45.289 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:21:45.290 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:21:50.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:50.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:50.292 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:21:50.341 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:21:50.341 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:21:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:21:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:21:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:21:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:21:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:21:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:21:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:21:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:21:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:21:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:21:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:21:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:21:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:21:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:21:50Z" 2023-11-02T19:21:50.451 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:21:50.452 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:21:50Z" } 2023-11-02T19:21:50.453 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:21:50.454 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:21:50.454 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:21:50.455 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:21:55.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:55.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:21:55.457 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:21:55.508 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:21:55.508 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:21:55.597 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:21:55.598 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:21:55.598 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:21:55.598 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:21:55.598 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:21:55.599 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:21:55.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:55.599 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:21:55.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:55.600 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:21:55.600 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:55.600 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:55.600 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:21:55.601 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:55.601 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:21:55.601 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:55.601 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:21:55.602 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:55.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:55.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:55.602 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:21:55.603 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:55.603 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:55.603 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:21:55.603 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:55.604 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:21:55.604 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:55.604 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:21:55.604 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:55.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:55.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:55.605 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:21:55.605 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:55.605 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:55.605 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:21:55.606 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:55.606 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:55.606 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:55.607 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:21:55.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:55.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:55.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:21:55.608 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:21:55.608 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:21:55.608 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:21:55.609 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:21:55.609 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:21:55.609 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:21:55.609 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:21:55.610 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:21:55.610 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:55.611 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:21:55.611 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:21:55.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:55.612 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:21:55.612 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:21:55.613 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:21:55.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:21:55.613 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:21:55Z" 2023-11-02T19:21:55.614 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:21:55.615 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:21:55Z" } 2023-11-02T19:21:55.616 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:21:55.616 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:21:55.617 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:21:55.617 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:22:00.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:00.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:00.620 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:00.668 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:00.668 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:00.757 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:22:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:22:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:22:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:22:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:22:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:22:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:22:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:22:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:22:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:22:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:22:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:22:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:22:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:22:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:22:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:22:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:22:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:22:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:22:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:22:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:22:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:22:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:22:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:22:00Z" 2023-11-02T19:22:00.771 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:00.772 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:22:00Z" } 2023-11-02T19:22:00.772 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:22:00.773 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:22:00.773 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:22:00.774 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:22:05.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:05.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:05.776 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:05.825 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:05.825 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:05.918 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:05.919 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:22:05.919 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:22:05.920 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:22:05.920 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:22:05.920 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:22:05.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:05.921 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:22:05.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:05.921 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:22:05.921 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:05.922 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:05.922 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:22:05.922 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:05.922 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:22:05.923 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:05.923 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:22:05.923 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:05.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:05.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:05.924 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:22:05.924 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:05.925 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:05.925 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:22:05.925 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:05.925 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:22:05.926 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:05.927 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:22:05.927 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:05.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:05.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:05.928 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:22:05.928 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:05.928 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:05.928 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:22:05.929 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:05.929 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:05.929 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:05.929 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:22:05.930 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:05.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:05.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:05.930 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:22:05.930 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:05.931 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:05.931 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:22:05.931 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:05.931 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:05.932 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:05.932 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:22:05.932 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:05.932 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:05.932 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:05.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:05.933 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:22:05.933 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:22:05.933 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:22:05.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:05.934 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:22:05Z" 2023-11-02T19:22:05.934 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:05.935 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:22:05Z" } 2023-11-02T19:22:05.936 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:22:05.936 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:22:05.937 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:22:05.937 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:22:10.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:10.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:10.940 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:10.989 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:10.989 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:11.080 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:22:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:22:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:22:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:22:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:22:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:22:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:22:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:22:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:22:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:22:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:22:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:22:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:22:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:22:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:22:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:22:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:22:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:22:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:22:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:22:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:11.093 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:22:11.093 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:22:11.093 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:22:11.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:11.093 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:22:11Z" 2023-11-02T19:22:11.094 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:11.095 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:22:11Z" } 2023-11-02T19:22:11.095 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:22:11.096 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:22:11.096 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:22:11.097 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:22:16.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:16.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:16.099 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:16.147 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:16.147 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:16.242 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:16.242 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:22:16.243 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:22:16.243 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:22:16.243 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:22:16.243 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:22:16.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:16.244 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:22:16.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:16.244 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:22:16.244 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:16.245 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:16.245 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:22:16.245 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:16.245 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:22:16.245 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:16.246 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:22:16.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:16.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:16.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:16.247 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:22:16.247 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:16.247 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:16.247 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:22:16.247 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:16.248 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:22:16.248 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:16.248 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:22:16.248 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:16.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:16.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:16.249 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:22:16.249 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:16.250 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:16.250 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:22:16.250 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:16.250 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:16.251 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:16.251 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:22:16.251 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:16.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:16.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:16.252 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:22:16.252 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:16.252 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:16.252 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:22:16.252 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:16.253 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:16.253 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:16.253 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:22:16.253 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:16.253 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:16.254 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:16.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:16.254 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:22:16.254 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:22:16.255 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:22:16.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:16.255 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:22:16Z" 2023-11-02T19:22:16.255 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:16.256 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:22:16Z" } 2023-11-02T19:22:16.257 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:22:16.257 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:22:16.258 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:22:16.258 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:22:21.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:21.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:21.260 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:21.309 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:21.309 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:21.401 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:21.402 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:22:21.402 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:22:21.403 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:22:21.403 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:22:21.404 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:22:21.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:21.404 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:22:21.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:21.405 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:22:21.405 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:21.406 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:21.406 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032199", 2023-11-02T19:22:21.406 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:21.406 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:22:21.407 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:21.407 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.937465Z" 2023-11-02T19:22:21.407 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:21.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:21.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:21.408 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:22:21.408 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:21.408 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:21.408 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051527", 2023-11-02T19:22:21.408 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:21.409 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:22:21.409 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:21.409 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.704188Z" 2023-11-02T19:22:21.409 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:21.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:21.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:21.410 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:22:21.410 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:21.410 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:21.411 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:22:21.411 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:21.411 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:21.411 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:21.411 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:22:21.411 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:21.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:21.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:21.412 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:22:21.412 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:21.412 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:21.413 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000045936", 2023-11-02T19:22:21.413 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:21.413 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:21.413 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:21.413 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:18:58.733917Z" 2023-11-02T19:22:21.414 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:21.414 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:21.414 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:21.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:21.415 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:22:21.415 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:22:21.415 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:22:21.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:21.415 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:22:21Z" 2023-11-02T19:22:21.416 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:21.416 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032199", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:19:53.937465Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051527", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:18:58.704188Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045936", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:18:58.733917Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:22:21Z" } 2023-11-02T19:22:21.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032199', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.937465Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051527', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.704188Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045936', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:18:58.733917Z'}}] 2023-11-02T19:22:21.417 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:22:21.418 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032199', 1: '00000000000000000000:00000000000000051527', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000045936'} 2023-11-02T19:22:21.418 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032339 target=00000000000000000000:00000000000000032199, shard=1 master=00000000000000000000:00000000000000051808 target=00000000000000000000:00000000000000051527, shard=3 master=00000000000000000000:00000000000000046076 target=00000000000000000000:00000000000000045936 2023-11-02T19:22:23.043 INFO:tasks.rgw.c2.client.0.smithi069.stdout:2023-11-02T19:22:23.035+0000 7fdaff3f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-11-02T19:22:26.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:26.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:26.421 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:26.566 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:22:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:22:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:22:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:22:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:22:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:22:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:22:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-11-02T19:22:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:22:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:22.019111Z" 2023-11-02T19:22:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:22:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051808", 2023-11-02T19:22:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:22:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:21.987436Z" 2023-11-02T19:22:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:22:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:22:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:22:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:22:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:22:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046076", 2023-11-02T19:22:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:21.926078Z" 2023-11-02T19:22:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:22:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:22:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:22:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:22:26Z" 2023-11-02T19:22:26.579 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:26.581 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:20:22.019111Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051808", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:20:21.987436Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046076", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:21.926078Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:22:26Z" } 2023-11-02T19:22:26.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:20:22.019111Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051808', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:20:21.987436Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046076', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:21.926078Z'}}] 2023-11-02T19:22:26.582 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:22:26.582 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:22:26.582 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:22:26.583 DEBUG:boto:path=/ibtljn-72/ 2023-11-02T19:22:26.583 DEBUG:boto:auth_path=/ibtljn-72/ 2023-11-02T19:22:26.584 DEBUG:boto:Method: HEAD 2023-11-02T19:22:26.584 DEBUG:boto:Path: /ibtljn-72/ 2023-11-02T19:22:26.585 DEBUG:boto:Data: 2023-11-02T19:22:26.585 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.585 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.586 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.586 DEBUG:boto:Params: {} 2023-11-02T19:22:26.587 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:22:26.587 DEBUG:boto:Token: None 2023-11-02T19:22:26.588 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-72/ 2023-11-02T19:22:26.588 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Tnmj/AKhaOFmXWwTLxi3CvtM8Tc= 2023-11-02T19:22:26.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Tnmj/AKhaOFmXWwTLxi3CvtM8Tc=', 'Content-Length': '0'} 2023-11-02T19:22:26.599 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cffae6db82d2261-006543f6f2-2aa3-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.600 DEBUG:boto:path=/ibtljn-72/obj1 2023-11-02T19:22:26.600 DEBUG:boto:auth_path=/ibtljn-72/obj1 2023-11-02T19:22:26.600 DEBUG:boto:Method: HEAD 2023-11-02T19:22:26.601 DEBUG:boto:Path: /ibtljn-72/obj1 2023-11-02T19:22:26.601 DEBUG:boto:Data: 2023-11-02T19:22:26.602 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.602 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.602 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.603 DEBUG:boto:Params: {} 2023-11-02T19:22:26.603 DEBUG:boto:Token: None 2023-11-02T19:22:26.604 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-72/obj1 2023-11-02T19:22:26.604 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:w0OoldsU6V/7tXa4mDPiGnsmvq0= 2023-11-02T19:22:26.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:w0OoldsU6V/7tXa4mDPiGnsmvq0=', 'Content-Length': '0'} 2023-11-02T19:22:26.607 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000ba01dc75b85ce818-006543f6f2-2aa3-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.608 DEBUG:boto:path=/ibtljn-72/obj2 2023-11-02T19:22:26.608 DEBUG:boto:auth_path=/ibtljn-72/obj2 2023-11-02T19:22:26.609 DEBUG:boto:Method: HEAD 2023-11-02T19:22:26.609 DEBUG:boto:Path: /ibtljn-72/obj2 2023-11-02T19:22:26.610 DEBUG:boto:Data: 2023-11-02T19:22:26.610 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.610 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.611 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.611 DEBUG:boto:Params: {} 2023-11-02T19:22:26.612 DEBUG:boto:Token: None 2023-11-02T19:22:26.612 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-72/obj2 2023-11-02T19:22:26.613 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bneP1n2it762uyBBmnD/hmhRA6I= 2023-11-02T19:22:26.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bneP1n2it762uyBBmnD/hmhRA6I=', 'Content-Length': '0'} 2023-11-02T19:22:26.615 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000088bbe1fe2af6a537-006543f6f2-2aa3-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.616 DEBUG:boto:path=/ibtljn-72/obj3 2023-11-02T19:22:26.616 DEBUG:boto:auth_path=/ibtljn-72/obj3 2023-11-02T19:22:26.617 DEBUG:boto:Method: HEAD 2023-11-02T19:22:26.617 DEBUG:boto:Path: /ibtljn-72/obj3 2023-11-02T19:22:26.618 DEBUG:boto:Data: 2023-11-02T19:22:26.618 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.619 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.619 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.620 DEBUG:boto:Params: {} 2023-11-02T19:22:26.620 DEBUG:boto:Token: None 2023-11-02T19:22:26.620 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-72/obj3 2023-11-02T19:22:26.621 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:qwiS94LpsPe4XrONRCv4DirRjmk= 2023-11-02T19:22:26.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qwiS94LpsPe4XrONRCv4DirRjmk=', 'Content-Length': '0'} 2023-11-02T19:22:26.624 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007ab26112bf3e8e97-006543f6f2-2aa3-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.625 DEBUG:boto:path=/ibtljn-72/obj4 2023-11-02T19:22:26.625 DEBUG:boto:auth_path=/ibtljn-72/obj4 2023-11-02T19:22:26.625 DEBUG:boto:Method: HEAD 2023-11-02T19:22:26.626 DEBUG:boto:Path: /ibtljn-72/obj4 2023-11-02T19:22:26.626 DEBUG:boto:Data: 2023-11-02T19:22:26.627 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.627 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.627 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.628 DEBUG:boto:Params: {} 2023-11-02T19:22:26.628 DEBUG:boto:Token: None 2023-11-02T19:22:26.629 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-72/obj4 2023-11-02T19:22:26.629 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5yA8WQ25kyjwtQSW4Lf55Ap5pMM= 2023-11-02T19:22:26.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5yA8WQ25kyjwtQSW4Lf55Ap5pMM=', 'Content-Length': '0'} 2023-11-02T19:22:26.632 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004ab7f6f23c8c8b30-006543f6f2-2aa3-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.633 DEBUG:boto:path=/ibtljn-73/ 2023-11-02T19:22:26.633 DEBUG:boto:auth_path=/ibtljn-73/ 2023-11-02T19:22:26.634 DEBUG:boto:Method: HEAD 2023-11-02T19:22:26.634 DEBUG:boto:Path: /ibtljn-73/ 2023-11-02T19:22:26.634 DEBUG:boto:Data: 2023-11-02T19:22:26.635 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.635 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.636 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.636 DEBUG:boto:Params: {} 2023-11-02T19:22:26.637 DEBUG:boto:Token: None 2023-11-02T19:22:26.637 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-73/ 2023-11-02T19:22:26.638 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:XXrsYNUrbvsNoA2uAUTeaxvLaCI= 2023-11-02T19:22:26.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XXrsYNUrbvsNoA2uAUTeaxvLaCI=', 'Content-Length': '0'} 2023-11-02T19:22:26.643 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000139bc3b6e02d8da9-006543f6f2-2aa3-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.643 DEBUG:boto:path=/ibtljn-73/obj1 2023-11-02T19:22:26.644 DEBUG:boto:auth_path=/ibtljn-73/obj1 2023-11-02T19:22:26.644 DEBUG:boto:Method: HEAD 2023-11-02T19:22:26.645 DEBUG:boto:Path: /ibtljn-73/obj1 2023-11-02T19:22:26.645 DEBUG:boto:Data: 2023-11-02T19:22:26.645 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.646 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.646 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.647 DEBUG:boto:Params: {} 2023-11-02T19:22:26.647 DEBUG:boto:Token: None 2023-11-02T19:22:26.647 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-73/obj1 2023-11-02T19:22:26.648 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:N30lMPcsCRYBCjGAWcHPD/lQhH8= 2023-11-02T19:22:26.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:N30lMPcsCRYBCjGAWcHPD/lQhH8=', 'Content-Length': '0'} 2023-11-02T19:22:26.652 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-72:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0a14992f747edcc-006543f6f2-2aa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.653 DEBUG:boto:path=/ibtljn-73/obj1 2023-11-02T19:22:26.653 DEBUG:boto:auth_path=/ibtljn-73/obj1 2023-11-02T19:22:26.654 DEBUG:boto:Method: GET 2023-11-02T19:22:26.654 DEBUG:boto:Path: /ibtljn-73/obj1 2023-11-02T19:22:26.655 DEBUG:boto:Data: 2023-11-02T19:22:26.655 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.655 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.656 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.656 DEBUG:boto:Params: {} 2023-11-02T19:22:26.657 DEBUG:boto:Token: None 2023-11-02T19:22:26.657 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-73/obj1 2023-11-02T19:22:26.658 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:GIR3+u7NoKmGuVd3DBOkYjdUVSo= 2023-11-02T19:22:26.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GIR3+u7NoKmGuVd3DBOkYjdUVSo=', 'Content-Length': '0'} 2023-11-02T19:22:26.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-72:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff1801030578ed4f-006543f6f2-2aa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.702 DEBUG:boto:path=/ibtljn-73/obj2 2023-11-02T19:22:26.703 DEBUG:boto:auth_path=/ibtljn-73/obj2 2023-11-02T19:22:26.703 DEBUG:boto:Method: HEAD 2023-11-02T19:22:26.704 DEBUG:boto:Path: /ibtljn-73/obj2 2023-11-02T19:22:26.704 DEBUG:boto:Data: 2023-11-02T19:22:26.705 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.705 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.705 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.706 DEBUG:boto:Params: {} 2023-11-02T19:22:26.706 DEBUG:boto:Token: None 2023-11-02T19:22:26.707 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-73/obj2 2023-11-02T19:22:26.707 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:tN+vcIzjbv8U8LYpTpMOEHRP3Os= 2023-11-02T19:22:26.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tN+vcIzjbv8U8LYpTpMOEHRP3Os=', 'Content-Length': '0'} 2023-11-02T19:22:26.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-72:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d766313c0ebf5cc6-006543f6f2-2aa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.712 DEBUG:boto:path=/ibtljn-73/obj2 2023-11-02T19:22:26.712 DEBUG:boto:auth_path=/ibtljn-73/obj2 2023-11-02T19:22:26.712 DEBUG:boto:Method: GET 2023-11-02T19:22:26.713 DEBUG:boto:Path: /ibtljn-73/obj2 2023-11-02T19:22:26.713 DEBUG:boto:Data: 2023-11-02T19:22:26.714 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.714 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.715 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.715 DEBUG:boto:Params: {} 2023-11-02T19:22:26.716 DEBUG:boto:Token: None 2023-11-02T19:22:26.716 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-73/obj2 2023-11-02T19:22:26.717 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:v/c9Wj4KcLvx++YNeuA+aPa5HXA= 2023-11-02T19:22:26.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:v/c9Wj4KcLvx++YNeuA+aPa5HXA=', 'Content-Length': '0'} 2023-11-02T19:22:26.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-72:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7e1b9ce29572aff-006543f6f2-2aa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.721 DEBUG:boto:path=/ibtljn-73/obj3 2023-11-02T19:22:26.722 DEBUG:boto:auth_path=/ibtljn-73/obj3 2023-11-02T19:22:26.722 DEBUG:boto:Method: HEAD 2023-11-02T19:22:26.722 DEBUG:boto:Path: /ibtljn-73/obj3 2023-11-02T19:22:26.723 DEBUG:boto:Data: 2023-11-02T19:22:26.723 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.724 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.724 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.724 DEBUG:boto:Params: {} 2023-11-02T19:22:26.725 DEBUG:boto:Token: None 2023-11-02T19:22:26.725 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-73/obj3 2023-11-02T19:22:26.726 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OeDPqf63Y7NHSsMmifv5BfC9YVw= 2023-11-02T19:22:26.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OeDPqf63Y7NHSsMmifv5BfC9YVw=', 'Content-Length': '0'} 2023-11-02T19:22:26.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-73:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dadca7e22db4dff5-006543f6f2-2aa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.730 DEBUG:boto:path=/ibtljn-73/obj3 2023-11-02T19:22:26.730 DEBUG:boto:auth_path=/ibtljn-73/obj3 2023-11-02T19:22:26.733 DEBUG:boto:Method: GET 2023-11-02T19:22:26.734 DEBUG:boto:Path: /ibtljn-73/obj3 2023-11-02T19:22:26.734 DEBUG:boto:Data: 2023-11-02T19:22:26.734 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.735 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.735 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.736 DEBUG:boto:Params: {} 2023-11-02T19:22:26.736 DEBUG:boto:Token: None 2023-11-02T19:22:26.737 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-73/obj3 2023-11-02T19:22:26.737 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:4+MfDkR9l6OgVO/Fk+u6E+hlTlY= 2023-11-02T19:22:26.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:4+MfDkR9l6OgVO/Fk+u6E+hlTlY=', 'Content-Length': '0'} 2023-11-02T19:22:26.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:20:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-73:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008e5daa431536a2e-006543f6f2-2aa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.782 DEBUG:boto:path=/ibtljn-73/obj4 2023-11-02T19:22:26.783 DEBUG:boto:auth_path=/ibtljn-73/obj4 2023-11-02T19:22:26.783 DEBUG:boto:Method: HEAD 2023-11-02T19:22:26.784 DEBUG:boto:Path: /ibtljn-73/obj4 2023-11-02T19:22:26.784 DEBUG:boto:Data: 2023-11-02T19:22:26.784 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.785 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.785 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.786 DEBUG:boto:Params: {} 2023-11-02T19:22:26.786 DEBUG:boto:Token: None 2023-11-02T19:22:26.787 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-73/obj4 2023-11-02T19:22:26.787 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8juFsDcWlbVHpjaU5z7ptTI2z24= 2023-11-02T19:22:26.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8juFsDcWlbVHpjaU5z7ptTI2z24=', 'Content-Length': '0'} 2023-11-02T19:22:26.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:20:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-73:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f14f621f604566c-006543f6f2-2aa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.792 DEBUG:boto:path=/ibtljn-73/obj4 2023-11-02T19:22:26.793 DEBUG:boto:auth_path=/ibtljn-73/obj4 2023-11-02T19:22:26.793 DEBUG:boto:Method: GET 2023-11-02T19:22:26.794 DEBUG:boto:Path: /ibtljn-73/obj4 2023-11-02T19:22:26.794 DEBUG:boto:Data: 2023-11-02T19:22:26.794 DEBUG:boto:Headers: {} 2023-11-02T19:22:26.795 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:22:26.795 DEBUG:boto:Port: 8001 2023-11-02T19:22:26.796 DEBUG:boto:Params: {} 2023-11-02T19:22:26.796 DEBUG:boto:Token: None 2023-11-02T19:22:26.797 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:22:26 GMT /ibtljn-73/obj4 2023-11-02T19:22:26.797 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:bymrlNhICgfHrRJ34vc9ki6oMD0= 2023-11-02T19:22:26.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:22:26 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bymrlNhICgfHrRJ34vc9ki6oMD0=', 'Content-Length': '0'} 2023-11-02T19:22:26.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:20:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-73:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008223e4897c54bd47-006543f6f2-2aa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:22:26 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:22:26.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:26.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:26.803 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-ibtljn-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:26.849 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:26.849 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:26.945 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-ibtljn-73","source":{"bucket":"ibtljn-73","zones":["a1"]},"dest":{"bucket":"ibtljn-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:22:26.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:26.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:26.947 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-ibtljn-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:26.990 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:26.991 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:27.087 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-11-02T19:22:27.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:27.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:27.089 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:27.139 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:27.139 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:27.235 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-11-02T19:22:27.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:27.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:27.236 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:27.287 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:27.287 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:27.390 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-11-02T19:22:27.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:27.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:27.392 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:27.440 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:27.441 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:27.534 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:22:27.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:27.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:27.535 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:27.581 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:27.582 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:27.677 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:22:27.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:27.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:27.679 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:27.724 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:27.724 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:27.836 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:22:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 21, 2023-11-02T19:22:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T19:22:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T19:22:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T19:22:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:22:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T19:22:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T19:22:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T19:22:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:22:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T19:22:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:22:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:22:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:22:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:22:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:22:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:22:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:22:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:22:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T19:22:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:22:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T19:22:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:22:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:22:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:22:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:22:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:22:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:22:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:22:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:22:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:22:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:22:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:22:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:22:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:22:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T19:22:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:22:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:22:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:22:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:22:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:22:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:22:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:22:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:22:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:22:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:22:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:22:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:22:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:22:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:22:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:22:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:22:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:22:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:22:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:22:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:22:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:22:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:22:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:22:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:22:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [ 2023-11-02T19:22:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-group", 2023-11-02T19:22:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "data_flow": { 2023-11-02T19:22:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "symmetrical": [ 2023-11-02T19:22:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-flow1", 2023-11-02T19:22:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:22:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:22:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:22:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: "pipes": [ 2023-11-02T19:22:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-pipe", 2023-11-02T19:22:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:22:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "*", 2023-11-02T19:22:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:22:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:22:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:22:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": { 2023-11-02T19:22:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "*", 2023-11-02T19:22:27.858 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:22:27.858 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:22:27.858 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:22:27.858 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.859 INFO:teuthology.orchestra.run.smithi017.stdout: "params": { 2023-11-02T19:22:27.859 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:22:27.859 INFO:teuthology.orchestra.run.smithi017.stdout: "filter": { 2023-11-02T19:22:27.860 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [] 2023-11-02T19:22:27.860 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:27.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.860 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": {}, 2023-11-02T19:22:27.860 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2023-11-02T19:22:27.861 INFO:teuthology.orchestra.run.smithi017.stdout: "mode": "system", 2023-11-02T19:22:27.861 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "" 2023-11-02T19:22:27.861 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:27.861 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:27.861 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:22:27.862 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "allowed" 2023-11-02T19:22:27.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:27.862 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.862 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:22:27.863 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:22:27.863 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.863 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T19:22:27.864 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T19:22:27.864 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:22:27.864 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T19:22:27.864 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:22:27.864 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:22:27.865 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:22:27.865 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:22:27.865 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:22:27.865 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:22:27.865 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:22:27.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.866 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:22:27.866 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T19:22:27.866 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:22:27.867 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:22:27.867 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:22:27.867 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:22:27.867 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:22:27.867 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:22:27.868 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:22:27.868 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:22:27.868 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:22:27.868 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:22:27.868 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:22:27.869 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:22:27.869 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:22:27.869 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:22:27.869 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.870 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:22:27.870 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T19:22:27.870 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:22:27.871 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T19:22:27.871 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:22:27.871 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:22:27.871 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:22:27.871 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:22:27.872 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:22:27.872 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:22:27.872 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:22:27.872 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:22:27.872 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:22:27.873 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:22:27.873 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:22:27.873 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:22:27.873 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.873 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:27.874 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:22:27.874 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:22:27.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.874 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:22:27.875 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:22:27.875 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:22:27.875 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:22:27.875 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.875 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:27.876 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:22:27.876 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:22:27.876 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:22:27.876 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:22:27.876 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T19:22:27.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.877 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:22:27.877 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:22:27.877 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.877 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:27.878 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:22:27.878 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T19:22:27.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.878 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:22:27.878 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T19:22:27.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.879 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:22:27.879 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T19:22:27.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.881 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:22:27.881 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T19:22:27.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:27.882 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:22:27.882 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T19:22:27.882 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:27.882 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:27.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.883 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T19:22:27.883 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:22:27.883 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T19:22:27.883 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T19:22:27.884 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:22:27.884 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:22:27.884 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:22:27.884 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:22:27.885 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:22:27.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.885 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T19:22:27.885 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:22:27.885 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:22:27.886 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:22:27.886 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:22:27.886 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:22:27.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.886 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T19:22:27.887 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:22:27.887 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:22:27.887 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:22:27.887 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:22:27.887 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:22:27.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.888 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T19:22:27.888 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:22:27.888 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:22:27.889 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:22:27.889 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:22:27.889 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:22:27.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.889 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T19:22:27.890 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:22:27.890 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:22:27.890 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:22:27.890 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:22:27.890 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:22:27.891 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:27.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:27.891 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:22:27.891 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T19:22:27.892 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:27.892 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:27.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:27.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:27.893 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:27.968 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:27.968 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:28.064 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:22:28.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ibtljn-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:28.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'ibtljn-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:28.065 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket ibtljn-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:28.121 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:28.121 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:28.248 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-11-02T19:22:28.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ibtljn-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:28.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'ibtljn-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:28.249 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket ibtljn-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:28.311 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:28.312 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:28.403 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-11-02T19:22:28.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-ibtljn-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=ibtljn-72', '--source-bucket=ibtljn-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:28.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-ibtljn-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=ibtljn-72', '--source-bucket=ibtljn-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:28.404 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-ibtljn-72 --source-zones=a1 --dest-zones=a2 --bucket=ibtljn-72 --source-bucket=ibtljn-72 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:28.461 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:28.461 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:28.551 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-72","source":{"bucket":"ibtljn-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-11-02T19:22:28.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-ibtljn-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=ibtljn-72', '--source-bucket=ibtljn-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:28.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-ibtljn-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=ibtljn-72', '--source-bucket=ibtljn-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:28.553 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-ibtljn-73 --source-zones=a1 --dest-zones=a2 --bucket=ibtljn-72 --source-bucket=ibtljn-73 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:28.610 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:28.610 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:28.702 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-72","source":{"bucket":"ibtljn-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-ibtljn-73","source":{"bucket":"ibtljn-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-11-02T19:22:28.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ibtljn-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:28.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'ibtljn-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:28.704 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket ibtljn-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:28.753 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:28.753 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:28.858 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-72","source":{"bucket":"ibtljn-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-ibtljn-73","source":{"bucket":"ibtljn-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-11-02T19:22:28.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:28.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:22:28.860 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:22:28.904 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:28.904 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:28.996 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:22:28.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:28.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:28.998 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:29.041 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:29.041 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:29.103 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:22:29.133 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:22:29.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:29.133 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:22:29.134 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:22:29.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:29.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:29.134 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:22:29.134 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:22:29.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:29.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:29.135 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:22:29.135 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:22:28.822959Z" 2023-11-02T19:22:29.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:29.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:29.136 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:22:29.136 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:22:29.137 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:29.137 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:22:29.138 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:29.138 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:22:29.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:29.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:29.140 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:29.190 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:29.190 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:29.246 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:22:29.242+0000 7f37f8103f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:22:29.248 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:22:29.242+0000 7f37f8103f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:22:29.277 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:29.277 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:22:29.277 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:22:29.277 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:22:29.278 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:22:29.278 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:22:29.278 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:29.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:29.279 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:22:29.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:29.279 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:22:29.280 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:29.280 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:29.280 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:22:29.280 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:29.281 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:22:29.281 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:29.281 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:22:29.281 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:29.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:29.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:29.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:29.282 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:22:29.283 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:29.283 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:29.283 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:22:29.283 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:29.284 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:29.284 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:29.284 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:22:29.284 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:29.284 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:29.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:29.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:29.285 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:22:29.285 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:29.286 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:29.286 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:22:29.287 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:29.287 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:29.287 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:29.287 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:22:29.288 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:29.288 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:29.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:29.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:29.288 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:22:29.289 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:29.289 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:29.289 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:22:29.289 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:29.289 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:29.290 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:29.290 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:22:29.290 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:29.291 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:29.291 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:29.291 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:29.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:29.291 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:22:29.292 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:22:29.292 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:22:29.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:29.292 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:22:29Z" 2023-11-02T19:22:29.293 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:29.295 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:22:29Z" } 2023-11-02T19:22:29.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:22:29.296 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:29.296 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:29.297 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:22:34.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:34.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:34.302 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:34.352 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:34.352 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:34.414 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:22:34.410+0000 7fb303022f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:22:34.416 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:22:34.410+0000 7fb303022f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:22:34.445 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:34.445 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:22:34.446 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:22:34.446 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:22:34.446 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:22:34.446 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:22:34.447 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:34.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:34.447 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:22:34.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:34.448 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:22:34.448 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:34.448 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:34.448 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:22:34.449 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:34.449 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:22:34.449 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:34.449 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:22:34.449 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:34.450 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:34.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:34.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:34.450 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:22:34.450 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:34.451 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:34.451 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:22:34.451 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:34.451 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:34.452 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:34.452 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:22:34.452 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:34.452 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:34.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:34.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:34.453 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:22:34.453 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:34.453 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:34.454 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:22:34.454 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:34.454 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:34.454 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:34.454 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:22:34.455 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:34.455 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:34.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:34.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:34.456 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:22:34.456 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:34.456 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:34.456 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:22:34.456 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:34.457 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:34.457 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:34.457 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:22:34.457 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:34.457 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:34.458 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:34.458 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:34.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:34.458 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:22:34.459 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:22:34.459 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:22:34.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:34.459 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:22:34Z" 2023-11-02T19:22:34.459 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:34.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:22:34Z" } 2023-11-02T19:22:34.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:22:34.462 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:34.462 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:34.462 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:22:39.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:39.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:39.465 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:39.575 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:22:39.570+0000 7f3e47e08f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:22:39.577 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:22:39.574+0000 7f3e47e08f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:22:39.603 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:39.603 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:22:39.604 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:22:39.604 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:22:39.604 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:22:39.604 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:22:39.604 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:39.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:39.605 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:22:39.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:39.605 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:22:39.606 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:39.606 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:39.606 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:22:39.606 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:39.607 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:22:39.607 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:39.607 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:22:39.607 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:39.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:39.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:39.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:39.608 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:22:39.608 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:39.608 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:39.609 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:22:39.609 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:39.609 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:39.609 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:39.610 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:22:39.610 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:39.610 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:39.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:39.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:39.611 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:22:39.611 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:39.611 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:39.611 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:22:39.612 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:39.612 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:39.612 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:39.612 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:22:39.612 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:39.613 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:39.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:39.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:39.613 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:22:39.614 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:39.614 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:39.614 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:22:39.614 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:39.614 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:39.615 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:39.615 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:22:39.615 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:39.615 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:39.615 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:39.616 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:39.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:39.616 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:22:39.616 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:22:39.616 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:22:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:22:39Z" 2023-11-02T19:22:39.617 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:39.618 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:22:39Z" } 2023-11-02T19:22:39.619 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:22:39.619 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:39.620 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:39.620 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:22:44.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:44.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:44.623 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:44.670 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:44.670 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:44.733 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:22:44.730+0000 7f56a3f12f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:22:44.734 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:22:44.730+0000 7f56a3f12f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:22:44.755 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:44.756 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:22:44.756 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:22:44.756 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:22:44.756 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:22:44.756 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:22:44.757 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:44.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:44.757 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:22:44.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:44.758 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:22:44.758 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:44.758 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:44.758 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:22:44.758 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:44.759 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:22:44.759 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:44.759 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:22:44.759 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:44.760 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:44.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:44.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:44.760 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:22:44.760 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:44.761 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:44.761 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:22:44.761 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:44.761 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:44.761 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:44.762 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:22:44.762 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:44.762 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:44.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:44.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:44.763 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:22:44.763 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:44.763 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:44.763 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:22:44.764 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:44.764 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:44.764 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:44.764 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:22:44.764 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:44.765 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:44.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:44.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:44.765 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:22:44.765 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:44.766 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:44.766 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:22:44.766 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:44.766 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:44.767 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:44.767 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:22:44.767 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:44.767 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:44.767 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:44.768 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:44.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:44.768 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:22:44.768 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:22:44.768 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:22:44.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:44.769 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:22:44Z" 2023-11-02T19:22:44.769 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:44.770 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:22:44Z" } 2023-11-02T19:22:44.770 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:22:44.771 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:44.771 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:44.772 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:22:49.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:49.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:49.774 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:49.819 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:49.819 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:49.875 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:22:49.870+0000 7ff490fa8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:22:49.876 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:22:49.874+0000 7ff490fa8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:22:49.903 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:49.904 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:22:49.904 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:22:49.904 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:22:49.904 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:22:49.905 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:22:49.905 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:49.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:49.905 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:22:49.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:49.905 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:22:49.906 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:49.906 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:49.906 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:22:49.906 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:49.907 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:22:49.907 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:49.907 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:22:49.907 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:49.907 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:49.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:49.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:49.908 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:22:49.908 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:49.908 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:49.909 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:22:49.909 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:49.909 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:49.909 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:49.909 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:22:49.910 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:49.910 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:49.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:49.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:49.911 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:22:49.911 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:49.911 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:49.911 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:22:49.911 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:49.912 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:49.912 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:49.912 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:22:49.912 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:49.912 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:49.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:49.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:49.913 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:22:49.913 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:49.913 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:49.914 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:22:49.914 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:49.914 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:49.914 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:49.915 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:22:49.915 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:49.915 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:49.915 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:49.915 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:49.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:49.916 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:22:49.916 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:22:49.916 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:22:49.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:49.917 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:22:49Z" 2023-11-02T19:22:49.917 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:49.918 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:22:49Z" } 2023-11-02T19:22:49.918 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:22:49.919 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:49.919 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:49.920 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:22:54.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:54.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:22:54.922 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:22:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:22:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:22:55.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:22:55.018+0000 7fc924f8af40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:22:55.025 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:22:55.022+0000 7fc924f8af40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:22:55.053 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:22:55.053 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:22:55.053 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:22:55.053 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:22:55.054 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:22:55.054 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:22:55.054 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:55.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:55.055 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:22:55.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:55.055 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:22:55.055 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:55.055 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:55.056 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:22:55.056 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:55.056 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:22:55.056 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:55.056 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:22:55.056 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:55.057 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:55.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:55.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:55.057 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:22:55.057 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:55.058 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:55.058 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:22:55.058 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:55.058 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:55.059 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:55.059 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:22:55.059 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:55.059 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:55.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:55.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:55.060 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:22:55.060 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:55.060 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:55.060 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:22:55.060 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:55.061 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:55.061 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:55.061 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:22:55.061 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:55.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:55.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:55.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:22:55.062 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:22:55.062 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:22:55.063 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:22:55.063 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:22:55.063 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:22:55.063 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:22:55.063 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:22:55.063 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:22:55.064 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:22:55.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:55.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:22:55.064 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:22:55.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:55.065 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:22:55.065 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:22:55.065 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:22:55.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:22:55.065 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:22:55Z" 2023-11-02T19:22:55.066 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:22:55.066 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:22:55Z" } 2023-11-02T19:22:55.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:22:55.067 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:55.068 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:22:55.068 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:23:00.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:00.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:00.071 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:23:00.118 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:23:00.119 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:23:00.176 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:00.170+0000 7f38c9e67f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:00.177 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:00.174+0000 7f38c9e67f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:00.206 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:23:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:23:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:23:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:23:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:23:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:23:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:23:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:23:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:23:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:23:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:23:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:23:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:23:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:23:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:23:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:23:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:23:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:23:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:23:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:23:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:23:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:23:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:23:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:23:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:23:00Z" 2023-11-02T19:23:00.221 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:23:00.222 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:23:00Z" } 2023-11-02T19:23:00.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:23:00.223 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:00.223 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:00.223 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:23:05.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:05.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:05.226 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:23:05.272 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:23:05.272 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:23:05.334 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:05.330+0000 7f3ed04aaf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:05.335 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:05.330+0000 7f3ed04aaf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:05.365 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:23:05.365 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:23:05.365 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:23:05.366 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:23:05.366 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:23:05.366 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:23:05.367 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:05.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:05.367 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:23:05.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:05.367 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:23:05.368 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:05.368 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:05.368 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:23:05.368 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:05.369 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:23:05.369 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:05.369 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:23:05.369 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:05.370 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:05.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:05.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:05.370 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:23:05.370 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:05.371 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:05.371 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:23:05.371 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:05.371 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:05.372 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:05.372 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:23:05.372 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:05.372 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:05.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:05.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:05.373 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:23:05.373 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:05.373 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:05.373 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:23:05.374 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:05.374 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:05.374 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:05.374 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:23:05.375 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:05.375 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:05.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:05.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:05.376 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:23:05.376 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:05.376 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:05.377 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:23:05.377 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:05.377 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:05.377 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:05.378 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:23:05.378 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:05.378 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:05.378 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:05.379 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:23:05.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:05.379 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:23:05.379 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:23:05.380 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:23:05.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:05.380 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:23:05Z" 2023-11-02T19:23:05.380 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:23:05.381 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:23:05Z" } 2023-11-02T19:23:05.382 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:23:05.382 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:05.383 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:05.383 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:23:10.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:10.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:10.386 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:23:10.432 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:23:10.432 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:23:10.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:10.486+0000 7fd64679ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:10.486+0000 7fd64679ef40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:10.519 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:23:10.519 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:23:10.519 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:23:10.520 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:23:10.520 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:23:10.520 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:23:10.520 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:10.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:10.521 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:23:10.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:10.521 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:23:10.521 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:10.522 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:10.522 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:23:10.522 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:10.522 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:23:10.523 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:10.523 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:23:10.523 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:10.523 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:10.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:10.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:10.524 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:23:10.524 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:10.524 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:10.525 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:23:10.525 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:10.525 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:10.525 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:10.525 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:23:10.526 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:10.526 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:10.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:10.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:10.526 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:23:10.527 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:10.527 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:10.527 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:23:10.527 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:10.527 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:10.528 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:10.528 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:23:10.528 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:10.528 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:10.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:10.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:10.529 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:23:10.529 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:10.529 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:10.530 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:23:10.530 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:10.530 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:10.530 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:10.530 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:23:10.531 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:10.531 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:10.531 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:10.531 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:23:10.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:10.532 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:23:10.532 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:23:10.532 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:23:10.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:10.533 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:23:10Z" 2023-11-02T19:23:10.533 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:23:10.534 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:23:10Z" } 2023-11-02T19:23:10.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:23:10.535 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:10.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:10.536 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:23:15.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:15.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:15.538 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:23:15.584 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:23:15.584 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:23:15.642 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:15.638+0000 7f5d22792f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:15.645 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:15.638+0000 7f5d22792f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:15.672 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:23:15.672 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:23:15.673 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:23:15.673 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:23:15.673 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:23:15.673 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:23:15.673 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:15.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:15.674 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:23:15.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:15.674 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:23:15.675 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:15.675 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:15.675 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:23:15.675 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:15.675 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:23:15.676 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:15.676 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:23:15.676 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:15.676 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:15.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:15.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:15.677 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:23:15.677 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:15.678 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:15.678 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:23:15.678 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:15.678 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:15.678 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:15.679 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:23:15.679 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:15.679 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:15.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:15.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:15.680 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:23:15.680 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:15.680 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:15.681 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:23:15.681 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:15.681 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:15.681 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:15.681 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:23:15.682 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:15.682 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:15.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:15.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:15.683 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:23:15.683 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:15.683 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:15.683 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:23:15.683 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:15.684 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:15.684 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:15.684 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:23:15.684 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:15.685 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:15.685 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:15.685 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:23:15.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:15.685 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:23:15.686 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:23:15.686 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:23:15.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:15.686 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:23:15Z" 2023-11-02T19:23:15.686 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:23:15.687 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:23:15Z" } 2023-11-02T19:23:15.688 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:23:15.689 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:15.689 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:15.689 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:23:20.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:20.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:20.692 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:23:20.739 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:23:20.739 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:23:20.798 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:20.794+0000 7fe7f7269f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:20.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:20.794+0000 7fe7f7269f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:20.829 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:23:20.829 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:23:20.829 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:23:20.830 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:23:20.830 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:23:20.830 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:23:20.830 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:20.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:20.831 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:23:20.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:20.831 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:23:20.831 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:20.832 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:20.832 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:23:20.832 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:20.832 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:23:20.832 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:20.833 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:23:20.833 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:20.833 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:20.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:20.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:20.834 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:23:20.835 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:20.835 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:20.835 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:23:20.835 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:20.836 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:20.836 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:20.836 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:23:20.836 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:20.836 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:20.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:20.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:20.837 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:23:20.837 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:20.838 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:20.838 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:23:20.838 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:20.838 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:20.839 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:20.839 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:23:20.839 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:20.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:20.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:20.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:20.840 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:23:20.840 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:20.840 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:20.841 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:23:20.841 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:20.841 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:20.841 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:20.842 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:23:20.842 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:20.842 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:20.842 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:20.843 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:23:20.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:20.843 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:23:20.843 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:23:20.843 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:23:20.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:20.844 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:23:20Z" 2023-11-02T19:23:20.844 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:23:20.845 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:23:20Z" } 2023-11-02T19:23:20.846 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:23:20.846 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:20.847 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:20.847 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:23:25.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:25.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:25.850 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:23:25.896 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:23:25.896 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:23:25.956 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:25.954+0000 7f62a4745f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:25.958 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:25.954+0000 7f62a4745f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:25.986 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:23:25.986 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:23:25.986 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:23:25.987 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:23:25.987 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:23:25.987 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:23:25.987 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:25.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:25.988 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:23:25.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:25.988 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:23:25.989 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:25.989 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:25.989 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:23:25.989 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:25.990 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:23:25.990 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:25.990 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:23:25.990 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:25.991 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:25.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:25.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:25.991 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:23:25.992 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:25.992 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:25.992 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:23:25.992 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:25.993 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:25.993 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:25.993 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:23:25.993 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:25.994 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:25.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:25.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:25.994 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:23:25.995 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:25.995 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:25.995 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:23:25.995 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:25.996 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:25.996 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:25.996 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:23:25.997 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:25.997 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:25.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:25.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:25.997 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:23:25.998 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:25.998 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:25.998 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:23:25.998 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:25.999 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:25.999 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:25.999 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:23:25.999 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:26.000 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:26.000 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:26.000 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:23:26.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:26.001 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:23:26.001 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:23:26.001 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:23:26.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:26.002 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:23:25Z" 2023-11-02T19:23:26.002 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:23:26.003 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:23:25Z" } 2023-11-02T19:23:26.004 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:23:26.004 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:26.004 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:26.005 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:23:31.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:31.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:31.007 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:23:31.057 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:23:31.057 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:23:31.119 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:31.113+0000 7fce74028f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:31.121 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:31.117+0000 7fce74028f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:31.149 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:23:31.150 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:23:31.150 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:23:31.150 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:23:31.151 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:23:31.151 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:23:31.151 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:31.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:31.152 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:23:31.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:31.152 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:23:31.152 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:31.153 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:31.153 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:23:31.153 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:31.153 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:23:31.154 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:31.154 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:23:31.154 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:31.154 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:31.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:31.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:31.155 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:23:31.155 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:31.155 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:31.156 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:23:31.156 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:31.156 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:31.156 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:31.157 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:23:31.157 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:31.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:31.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:31.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:31.158 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:23:31.158 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:31.158 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:31.159 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:23:31.159 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:31.159 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:31.159 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:31.160 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:23:31.160 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:31.160 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:31.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:31.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:31.162 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:23:31.162 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:31.162 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:31.163 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:23:31.163 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:31.163 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:31.164 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:31.164 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:23:31.164 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:31.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:31.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:31.165 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:23:31.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:31.165 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:23:31.165 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:23:31.166 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:23:31.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:31.166 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:23:31Z" 2023-11-02T19:23:31.167 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:23:31.168 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:23:31Z" } 2023-11-02T19:23:31.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:23:31.169 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:31.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:31.170 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:23:36.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:36.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:36.172 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:23:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:23:36.219 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:23:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:36.273+0000 7f3f8206df40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:36.273+0000 7f3f8206df40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:36.308 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:23:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:23:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:23:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:23:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:23:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:23:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:23:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:23:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:23:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:23:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:23:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:23:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:23:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:23:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:36.316 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:23:36.316 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:36.316 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:36.316 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:23:36.317 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:36.317 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:36.317 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:36.318 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:23:36.318 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:36.318 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:36.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:36.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:36.319 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:23:36.319 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:36.319 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:36.319 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:23:36.320 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:36.320 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:36.320 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:36.320 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:23:36.320 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:23:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:23:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:23:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:23:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:23:36Z" 2023-11-02T19:23:36.324 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:23:36.326 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:23:36Z" } 2023-11-02T19:23:36.326 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:23:36.327 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:36.328 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:36.328 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:23:41.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:41.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:41.331 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:23:41.378 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:23:41.378 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:23:41.436 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:41.429+0000 7f36f234df40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:41.433+0000 7f36f234df40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:41.467 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:23:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:23:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:23:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:23:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:23:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:23:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:23:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:23:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:23:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:23:41.471 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:41.471 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:23:41.471 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:41.471 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:41.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:41.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:41.472 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:23:41.472 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:41.473 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:41.473 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:23:41.473 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:41.473 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:41.473 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:41.474 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:23:41.474 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:41.474 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:41.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:41.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:41.475 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:23:41.475 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:41.475 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:23:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:23:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:23:41.478 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:41.478 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:41.478 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:23:41.478 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:23:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:23:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:23:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:23:41.481 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:23:41.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:41.481 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:23:41Z" 2023-11-02T19:23:41.481 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:23:41.482 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:23:41Z" } 2023-11-02T19:23:41.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:23:41.484 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:41.484 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:41.484 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:23:46.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:46.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:46.487 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:23:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:23:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:23:46.591 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:46.585+0000 7f4903755f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:46.593 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:46.589+0000 7f4903755f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:46.621 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:23:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:23:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:23:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:23:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:23:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:23:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:23:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:23:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:23:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:23:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:23:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:23:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:23:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:23:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:46.629 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:23:46.630 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:46.630 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:46.630 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:23:46.631 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:46.631 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:46.631 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:46.631 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:23:46.632 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:46.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:46.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:46.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:46.633 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:23:46.633 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:46.634 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:46.634 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:23:46.635 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:46.635 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:46.636 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:46.636 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:23:46.636 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:46.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:46.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:46.637 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:23:46.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:46.638 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:23:46.638 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:23:46.639 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:23:46.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:46.639 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:23:46Z" 2023-11-02T19:23:46.640 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:23:46.641 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:23:46Z" } 2023-11-02T19:23:46.641 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:23:46.642 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:46.642 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:46.643 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:23:51.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:51.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:51.646 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:23:51.691 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:23:51.692 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:23:51.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:51.749+0000 7f030cdd0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:51.753 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:51.749+0000 7f030cdd0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:51.776 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:23:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:23:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:23:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:23:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:23:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:23:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:23:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:23:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:23:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:23:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:23:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:23:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:23:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:23:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:23:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:23:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:23:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:51.786 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:51.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:51.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:51.786 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:23:51.787 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:51.787 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:51.787 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:23:51.787 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:51.787 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:51.788 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:51.788 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:23:51.788 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:51.788 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:51.789 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:51.789 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:23:51.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:51.789 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:23:51.789 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:23:51.790 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:23:51.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:51.790 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:23:51Z" 2023-11-02T19:23:51.790 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:23:51.791 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:23:51Z" } 2023-11-02T19:23:51.792 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:23:51.793 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:51.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:51.793 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:23:56.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:56.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:23:56.795 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:23:56.842 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:23:56.842 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:23:56.902 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:56.897+0000 7f6132fa3f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:56.904 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:23:56.897+0000 7f6132fa3f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:23:56.932 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:23:56.932 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:23:56Z" 2023-11-02T19:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:23:56.947 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:23:56Z" } 2023-11-02T19:23:56.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:23:56.948 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:56.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:23:56.949 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:24:01.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:01.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:01.952 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:02.000 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:02.000 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:02.058 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:02.053+0000 7f7a7441df40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:02.059 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:02.053+0000 7f7a7441df40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:02.088 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:02.088 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:02.089 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:02.089 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:02.089 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:02.089 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:24:02.090 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:02.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:02.090 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:02.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:02.091 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:02.091 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:02.091 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:02.091 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:24:02.091 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:02.092 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:24:02.092 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:02.092 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:24:02.092 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:02.093 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:02.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:02.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:02.093 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:02.093 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:02.094 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:02.094 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:24:02.094 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:02.094 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:02.094 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:02.095 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:24:02.095 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:02.095 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:02.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:02.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:02.096 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:02.096 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:02.096 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:02.096 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:24:02.097 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:02.097 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:02.097 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:02.097 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:24:02.098 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:02.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:02.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:02.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:02.099 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:02.099 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:02.099 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:02.099 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:24:02.099 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:02.100 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:02.100 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:02.100 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:24:02.100 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:02.100 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:02.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:02.101 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:02.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:02.101 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:02.102 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:24:02.102 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:24:02.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:02.102 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:02Z" 2023-11-02T19:24:02.102 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:02.104 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:24:02Z" } 2023-11-02T19:24:02.104 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:24:02.105 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:02.105 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:02.105 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:24:07.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:07.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:07.108 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:07.157 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:07.157 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:07.215 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:07.209+0000 7fe128889f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:07.217 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:07.213+0000 7fe128889f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:07.246 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:07.247 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:07.247 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:07.247 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:07.248 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:07.248 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:24:07.248 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:07.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:07.248 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:07.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:07.249 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:07.249 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:07.249 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:07.250 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:24:07.250 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:07.250 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:24:07.250 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:07.250 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:24:07.251 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:07.251 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:07.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:07.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:07.252 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:07.252 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:07.252 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:07.252 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:24:07.252 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:07.253 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:07.253 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:07.253 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:24:07.253 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:07.254 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:07.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:07.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:07.254 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:07.254 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:07.255 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:07.255 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:24:07.255 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:07.255 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:07.255 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:07.256 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:24:07.256 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:07.256 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:07.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:07.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:07.257 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:07.257 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:07.257 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:07.257 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:24:07.258 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:07.258 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:07.258 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:07.258 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:24:07.258 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:07.259 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:07.259 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:07.259 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:07.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:07.260 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:07.260 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:24:07.260 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:24:07.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:07.260 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:07Z" 2023-11-02T19:24:07.261 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:07.262 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:24:07Z" } 2023-11-02T19:24:07.262 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:24:07.263 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:07.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:07.264 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:24:12.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:12.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:12.266 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:12.312 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:12.313 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:12.372 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:12.365+0000 7f3c273a0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:12.374 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:12.369+0000 7f3c273a0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:12.403 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:12.403 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:12.403 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:12.404 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:12.404 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:12.404 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:24:12.404 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:12.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:12.405 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:12.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:12.405 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:12.406 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:12.406 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:12.406 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:24:12.406 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:12.409 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:24:12.409 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:12.409 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:24:12.410 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:12.410 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:12.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:12.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:12.410 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:12.411 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:12.411 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:12.411 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:24:12.411 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:12.412 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:12.412 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:12.412 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:24:12.412 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:12.412 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:12.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:12.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:12.413 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:12.413 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:12.414 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:12.414 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:24:12.414 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:12.414 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:12.414 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:12.415 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:24:12.415 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:12.415 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:12.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:12.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:12.416 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:12.416 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:12.416 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:12.416 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:24:12.417 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:12.417 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:12.417 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:12.417 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:24:12.418 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:12.418 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:12.418 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:12.418 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:12.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:12.419 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:12.419 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:24:12.419 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:24:12.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:12.420 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:12Z" 2023-11-02T19:24:12.420 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:12.421 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:24:12Z" } 2023-11-02T19:24:12.422 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:24:12.422 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:12.423 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:12.423 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:24:17.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:17.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:17.425 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:17.471 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:17.471 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:17.531 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:17.525+0000 7faf8cdebf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:17.533 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:17.529+0000 7faf8cdebf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:17.561 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:17.562 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:17.562 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:17.562 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:17.562 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:17.563 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:24:17.563 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:17.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:17.563 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:17.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:17.564 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:17.564 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:17.564 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:17.564 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:24:17.564 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:17.565 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:24:17.565 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:17.565 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:24:17.565 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:17.565 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:17.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:17.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:17.566 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:17.566 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:17.566 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:17.567 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:24:17.568 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:17.568 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:17.568 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:17.569 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:24:17.569 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:17.569 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:17.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:17.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:17.570 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:17.570 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:17.570 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:17.570 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:24:17.570 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:17.571 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:17.571 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:17.571 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:24:17.571 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:17.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:17.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:17.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:17.572 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:17.572 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:17.573 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:17.573 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:24:17.573 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:17.573 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:17.573 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:17.574 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:24:17.574 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:17.574 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:17.574 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:17.575 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:17.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:17.575 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:17.575 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:24:17.575 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:24:17.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:17.576 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:17Z" 2023-11-02T19:24:17.576 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:17.577 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:24:17Z" } 2023-11-02T19:24:17.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:24:17.578 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:17.578 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:17.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:24:22.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:22.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:22.581 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:22.694 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:22.689+0000 7f57ef356f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:22.695 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:22.689+0000 7f57ef356f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:22.723 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:22.723 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:22.723 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:22.724 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:22.724 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:22.724 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:24:22.724 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:22.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:22.725 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:22.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:22.725 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:22.725 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:22.726 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:22.726 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:24:22.726 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:22.726 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:24:22.727 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:22.727 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:24:22.727 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:22.727 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:22.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:22.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:22.729 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:22.729 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:22.729 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:22.729 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:24:22.729 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:22.730 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:22.730 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:22.730 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:24:22.730 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:22.731 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:22.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:22.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:22.732 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:22.732 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:22.732 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:22.732 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:24:22.733 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:22.733 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:22.733 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:22.733 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:24:22.733 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:22.734 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:22.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:22.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:22.734 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:22.735 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:22.735 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:22.735 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:24:22.735 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:22.736 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:22.736 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:22.736 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:24:22.736 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:22.736 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:22.737 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:22.737 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:22.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:22.737 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:22.738 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:24:22.738 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:24:22.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:22.738 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:22Z" 2023-11-02T19:24:22.739 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:22.740 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:24:22Z" } 2023-11-02T19:24:22.741 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:24:22.741 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:22.742 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:22.742 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:24:27.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:27.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:27.744 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:27.783 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:27.783 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:27.840 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:27.833+0000 7f3eac704f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:27.842 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:27.837+0000 7f3eac704f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:27.870 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:27.871 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:27.871 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:27.871 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:27.871 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:27.872 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:24:27.872 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:27.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:27.872 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:27.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:27.873 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:27.873 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:27.873 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:27.873 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:24:27.873 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:27.874 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:24:27.874 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:27.874 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:24:27.874 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:27.875 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:27.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:27.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:27.875 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:27.875 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:27.876 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:27.876 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:24:27.876 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:27.876 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:27.876 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:27.877 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:24:27.877 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:27.877 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:27.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:27.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:27.878 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:27.878 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:27.878 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:27.878 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952820.360316_808.1", 2023-11-02T19:24:27.879 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:27.879 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:27.879 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:27.879 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:20.360316Z", 2023-11-02T19:24:27.880 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:27.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:27.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:27.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:27.881 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:27.881 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:27.881 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:27.881 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:24:27.881 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:27.882 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:27.882 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:27.882 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:24:27.882 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:27.883 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:27.883 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:27.883 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:27.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:27.883 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:27.884 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:24:27.884 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:24:27.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:27.884 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:27Z" 2023-11-02T19:24:27.885 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:27.885 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952820.360316_808.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:20.360316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:24:27Z" } 2023-11-02T19:24:27.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952820.360316_808.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:20.360316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:24:27.886 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:27.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952820.360316_808.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:27.887 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698952948.822959_838.1 target=1_1698952820.360316_808.1 2023-11-02T19:24:32.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:32.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:32.889 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:32.938 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:32.938 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:32.997 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:32.993+0000 7f82f594cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:32.998 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:32.993+0000 7f82f594cf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:33.027 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:24:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:24:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:24:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:33.032 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:24:33.032 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:33.032 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.033 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:33.033 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:33.033 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:33.033 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:24:33.033 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:33.034 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:33.034 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:33.034 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:24:33.034 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:33.035 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.035 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:33.035 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:24:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:22:28.822959Z", 2023-11-02T19:24:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.038 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:33.038 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:33.038 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:33.038 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:24:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:24:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:24:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:24:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:32Z" 2023-11-02T19:24:33.042 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:33.043 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952948.822959_838.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:22:28.822959Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:24:32Z" } 2023-11-02T19:24:33.043 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952948.822959_838.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:22:28.822959Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:24:33.044 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:33.044 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:33.045 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:24:33.045 DEBUG:boto:path=/ibtljn-72/ 2023-11-02T19:24:33.046 DEBUG:boto:auth_path=/ibtljn-72/ 2023-11-02T19:24:33.046 DEBUG:boto:Method: HEAD 2023-11-02T19:24:33.046 DEBUG:boto:Path: /ibtljn-72/ 2023-11-02T19:24:33.047 DEBUG:boto:Data: 2023-11-02T19:24:33.047 DEBUG:boto:Headers: {} 2023-11-02T19:24:33.048 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:24:33.048 DEBUG:boto:Port: 8000 2023-11-02T19:24:33.049 DEBUG:boto:Params: {} 2023-11-02T19:24:33.049 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T19:24:33.050 DEBUG:boto:Token: None 2023-11-02T19:24:33.050 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:33 GMT /ibtljn-72/ 2023-11-02T19:24:33.051 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8N5FSgVAIMckajbOhk2s1SppEKk= 2023-11-02T19:24:33.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8N5FSgVAIMckajbOhk2s1SppEKk=', 'Content-Length': '0'} 2023-11-02T19:24:33.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f979bf2fea0aa5e8-006543f771-2bbf-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:33.063 DEBUG:boto:path=/ibtljn-72/obj5 2023-11-02T19:24:33.064 DEBUG:boto:auth_path=/ibtljn-72/obj5 2023-11-02T19:24:33.064 DEBUG:boto:Method: PUT 2023-11-02T19:24:33.065 DEBUG:boto:Path: /ibtljn-72/obj5 2023-11-02T19:24:33.065 DEBUG:boto:Data: 2023-11-02T19:24:33.066 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:24:33.066 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:24:33.066 DEBUG:boto:Port: 8000 2023-11-02T19:24:33.067 DEBUG:boto:Params: {} 2023-11-02T19:24:33.067 DEBUG:boto:Token: None 2023-11-02T19:24:33.068 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:24:33 GMT /ibtljn-72/obj5 2023-11-02T19:24:33.068 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:2FCuo5DOjO0DCWdKnTy0O0bKgZ4= 2023-11-02T19:24:33.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:24:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2FCuo5DOjO0DCWdKnTy0O0bKgZ4='} 2023-11-02T19:24:33.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7de797c9e7d30fd-006543f771-2bbf-a1'), ('Date', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:33.085 DEBUG:boto:path=/ibtljn-72/ 2023-11-02T19:24:33.086 DEBUG:boto:auth_path=/ibtljn-72/ 2023-11-02T19:24:33.086 DEBUG:boto:Method: HEAD 2023-11-02T19:24:33.086 DEBUG:boto:Path: /ibtljn-72/ 2023-11-02T19:24:33.087 DEBUG:boto:Data: 2023-11-02T19:24:33.087 DEBUG:boto:Headers: {} 2023-11-02T19:24:33.088 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:24:33.088 DEBUG:boto:Port: 8000 2023-11-02T19:24:33.089 DEBUG:boto:Params: {} 2023-11-02T19:24:33.089 DEBUG:boto:Token: None 2023-11-02T19:24:33.090 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:33 GMT /ibtljn-72/ 2023-11-02T19:24:33.090 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8N5FSgVAIMckajbOhk2s1SppEKk= 2023-11-02T19:24:33.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8N5FSgVAIMckajbOhk2s1SppEKk=', 'Content-Length': '0'} 2023-11-02T19:24:33.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023c40cf0294b355b-006543f771-2bbf-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:33.095 DEBUG:boto:path=/ibtljn-72/obj6 2023-11-02T19:24:33.095 DEBUG:boto:auth_path=/ibtljn-72/obj6 2023-11-02T19:24:33.095 DEBUG:boto:Method: PUT 2023-11-02T19:24:33.096 DEBUG:boto:Path: /ibtljn-72/obj6 2023-11-02T19:24:33.096 DEBUG:boto:Data: 2023-11-02T19:24:33.097 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:24:33.097 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:24:33.098 DEBUG:boto:Port: 8000 2023-11-02T19:24:33.098 DEBUG:boto:Params: {} 2023-11-02T19:24:33.099 DEBUG:boto:Token: None 2023-11-02T19:24:33.099 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:24:33 GMT /ibtljn-72/obj6 2023-11-02T19:24:33.100 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:8jHq7I5QFLMxedZQCtWsB8SdrYM= 2023-11-02T19:24:33.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:24:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8jHq7I5QFLMxedZQCtWsB8SdrYM='} 2023-11-02T19:24:33.115 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef07b210085b6ff1-006543f771-2bbf-a1'), ('Date', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:33.116 DEBUG:boto:path=/ibtljn-73/ 2023-11-02T19:24:33.116 DEBUG:boto:auth_path=/ibtljn-73/ 2023-11-02T19:24:33.116 DEBUG:boto:Method: HEAD 2023-11-02T19:24:33.117 DEBUG:boto:Path: /ibtljn-73/ 2023-11-02T19:24:33.117 DEBUG:boto:Data: 2023-11-02T19:24:33.118 DEBUG:boto:Headers: {} 2023-11-02T19:24:33.118 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:24:33.118 DEBUG:boto:Port: 8000 2023-11-02T19:24:33.119 DEBUG:boto:Params: {} 2023-11-02T19:24:33.119 DEBUG:boto:Token: None 2023-11-02T19:24:33.120 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:33 GMT /ibtljn-73/ 2023-11-02T19:24:33.120 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6M6mvuaHqKf/IwLVhCyl6GgJzpQ= 2023-11-02T19:24:33.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6M6mvuaHqKf/IwLVhCyl6GgJzpQ=', 'Content-Length': '0'} 2023-11-02T19:24:33.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7a7ed44e4f79e1b-006543f771-2bbf-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:33.124 DEBUG:boto:path=/ibtljn-73/obj7 2023-11-02T19:24:33.125 DEBUG:boto:auth_path=/ibtljn-73/obj7 2023-11-02T19:24:33.125 DEBUG:boto:Method: PUT 2023-11-02T19:24:33.126 DEBUG:boto:Path: /ibtljn-73/obj7 2023-11-02T19:24:33.126 DEBUG:boto:Data: 2023-11-02T19:24:33.126 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:24:33.127 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:24:33.127 DEBUG:boto:Port: 8000 2023-11-02T19:24:33.127 DEBUG:boto:Params: {} 2023-11-02T19:24:33.128 DEBUG:boto:Token: None 2023-11-02T19:24:33.128 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:24:33 GMT /ibtljn-73/obj7 2023-11-02T19:24:33.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:07zwsEUnOxdMdXKQcvoe5nx/g9Q= 2023-11-02T19:24:33.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:24:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:07zwsEUnOxdMdXKQcvoe5nx/g9Q='} 2023-11-02T19:24:33.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000133e9efd1b75c185-006543f771-2bbf-a1'), ('Date', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:33.175 DEBUG:boto:path=/ibtljn-73/ 2023-11-02T19:24:33.175 DEBUG:boto:auth_path=/ibtljn-73/ 2023-11-02T19:24:33.176 DEBUG:boto:Method: HEAD 2023-11-02T19:24:33.176 DEBUG:boto:Path: /ibtljn-73/ 2023-11-02T19:24:33.176 DEBUG:boto:Data: 2023-11-02T19:24:33.177 DEBUG:boto:Headers: {} 2023-11-02T19:24:33.177 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:24:33.178 DEBUG:boto:Port: 8000 2023-11-02T19:24:33.178 DEBUG:boto:Params: {} 2023-11-02T19:24:33.179 DEBUG:boto:Token: None 2023-11-02T19:24:33.179 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:33 GMT /ibtljn-73/ 2023-11-02T19:24:33.179 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:6M6mvuaHqKf/IwLVhCyl6GgJzpQ= 2023-11-02T19:24:33.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6M6mvuaHqKf/IwLVhCyl6GgJzpQ=', 'Content-Length': '0'} 2023-11-02T19:24:33.184 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb18bf855f2c0de7-006543f771-2bbf-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:33.185 DEBUG:boto:path=/ibtljn-73/obj8 2023-11-02T19:24:33.185 DEBUG:boto:auth_path=/ibtljn-73/obj8 2023-11-02T19:24:33.186 DEBUG:boto:Method: PUT 2023-11-02T19:24:33.186 DEBUG:boto:Path: /ibtljn-73/obj8 2023-11-02T19:24:33.187 DEBUG:boto:Data: 2023-11-02T19:24:33.188 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:24:33.188 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:24:33.189 DEBUG:boto:Port: 8000 2023-11-02T19:24:33.190 DEBUG:boto:Params: {} 2023-11-02T19:24:33.191 DEBUG:boto:Token: None 2023-11-02T19:24:33.191 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:24:33 GMT /ibtljn-73/obj8 2023-11-02T19:24:33.192 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:wjcWZOM4rI0ew3NSJgyRrP41NOk= 2023-11-02T19:24:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:24:33 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:wjcWZOM4rI0ew3NSJgyRrP41NOk='} 2023-11-02T19:24:33.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e54081ea472408ed-006543f771-2bbf-a1'), ('Date', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:33.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:33.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:33.208 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:33.253 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:33.254 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:33.316 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:24:33.345 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:24:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:24:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:24:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:24:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:24:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:24:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:22:28.822959Z" 2023-11-02T19:24:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:24:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:24:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.349 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:24:33.350 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:33.351 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:24:33.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:33.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:33.352 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:33.397 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:33.398 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:33.453+0000 7fdbe59dbf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:33.453+0000 7fdbe59dbf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:33.487 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:33.488 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:33.488 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:33.488 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:33.488 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:33.488 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:24:33.489 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:33.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.489 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:33.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.490 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:33.490 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:33.490 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:33.490 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:24:33.490 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:33.491 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:24:33.491 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:33.491 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:24:33.491 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:33.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.492 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:33.492 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:33.493 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:33.493 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:24:33.493 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:33.493 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:33.493 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:33.494 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:24:33.494 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:33.494 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.495 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:33.495 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:33.495 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:33.496 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:24:33.496 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:33.496 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:33.496 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:33.496 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:22:28.822959Z", 2023-11-02T19:24:33.497 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:33.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.498 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:33.498 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:33.499 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:33.499 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:24:33.499 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:33.499 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:33.500 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:33.500 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:24:33.500 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:33.500 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.501 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.501 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:33.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.501 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:33.501 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:24:33.502 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:24:33.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.502 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:33Z" 2023-11-02T19:24:33.502 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:33.503 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952948.822959_838.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:22:28.822959Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:24:33Z" } 2023-11-02T19:24:33.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952948.822959_838.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:22:28.822959Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:24:33.505 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:33.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:33.505 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:24:33.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:33.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:33.507 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:33.551 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:33.551 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:33.656 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:24:33.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.656 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:24:33.656 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:24:33.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.657 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:24:33.657 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:24:33.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.658 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:24:33.658 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:24:33.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.659 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:24:33.659 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:24:33.659 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.659 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:24:33.660 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:24:33.661 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:24:33.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:33.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:33.662 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:33.754 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:33.755 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:33.838 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:33.838 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:33.838 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:33.839 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:33.839 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:33.839 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:24:33.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.839 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:33.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.840 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:33.840 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:33.840 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:33.841 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-11-02T19:24:33.841 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:33.841 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:24:33.841 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:33.841 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:22.019111Z" 2023-11-02T19:24:33.842 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.842 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:33.843 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:33.843 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:33.843 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051808", 2023-11-02T19:24:33.843 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:33.843 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:24:33.844 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:33.844 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:21.987436Z" 2023-11-02T19:24:33.844 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.845 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:33.845 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:33.845 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:33.845 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:24:33.845 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:33.846 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:33.846 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:33.846 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:24:33.846 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:33.847 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:33.847 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:33.847 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:33.847 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046076", 2023-11-02T19:24:33.848 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:33.848 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:33.848 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:33.848 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:21.926078Z" 2023-11-02T19:24:33.849 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.849 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:33.849 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:33.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.849 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:33.850 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:24:33.850 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:24:33.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:33.850 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:33Z" 2023-11-02T19:24:33.850 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:33.851 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:20:22.019111Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051808", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:20:21.987436Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046076", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:21.926078Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:24:33Z" } 2023-11-02T19:24:33.852 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:20:22.019111Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051808', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:20:21.987436Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046076', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:21.926078Z'}}] 2023-11-02T19:24:33.852 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:24:33.853 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:24:33.853 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032480 target=00000000000000000000:00000000000000032339, shard=1 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051808, shard=3 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046076 2023-11-02T19:24:38.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:38.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:38.855 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:38.906 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:38.907 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:38.998 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:38.999 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:38.999 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:38.999 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:38.999 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:39.000 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:24:39.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:39.000 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:39.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:39.001 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:39.001 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:39.001 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:39.001 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-11-02T19:24:39.001 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:39.002 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:24:39.002 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:39.002 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:22.019111Z" 2023-11-02T19:24:39.002 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:39.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:39.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:39.003 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:39.003 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:39.004 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:39.004 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051808", 2023-11-02T19:24:39.004 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:39.004 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:24:39.005 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:39.005 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:21.987436Z" 2023-11-02T19:24:39.005 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:39.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:39.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:39.006 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:39.006 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:39.006 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:39.006 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:24:39.007 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:39.007 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:39.007 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:39.007 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:24:39.008 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:39.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:39.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:39.008 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:39.009 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:39.009 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:39.009 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046076", 2023-11-02T19:24:39.009 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:39.010 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:39.010 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:39.010 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:21.926078Z" 2023-11-02T19:24:39.010 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:39.010 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:39.011 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:39.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:39.011 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:39.011 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:24:39.012 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:24:39.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:39.012 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:38Z" 2023-11-02T19:24:39.012 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:39.013 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:20:22.019111Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051808", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:20:21.987436Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046076", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:21.926078Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:24:38Z" } 2023-11-02T19:24:39.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:20:22.019111Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051808', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:20:21.987436Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046076', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:21.926078Z'}}] 2023-11-02T19:24:39.015 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:24:39.015 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:24:39.015 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032480 target=00000000000000000000:00000000000000032339, shard=1 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051808, shard=3 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046076 2023-11-02T19:24:44.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:44.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:44.017 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:44.067 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:44.067 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:44.163 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:44.164 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:44.164 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:44.164 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:44.164 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:44.164 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:24:44.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:44.165 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:44.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:44.165 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:44.166 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:44.166 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:44.166 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-11-02T19:24:44.166 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:44.166 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:24:44.167 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:44.167 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:22.019111Z" 2023-11-02T19:24:44.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:44.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:44.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:44.168 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:44.168 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:44.168 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:44.168 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051808", 2023-11-02T19:24:44.169 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:44.169 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:24:44.169 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:44.169 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:21.987436Z" 2023-11-02T19:24:44.169 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:44.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:44.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:44.170 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:44.170 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:44.171 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:44.171 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:24:44.171 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:44.171 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:44.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:44.172 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:24:44.172 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:44.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:44.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:44.173 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:44.173 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:44.174 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:44.174 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046076", 2023-11-02T19:24:44.175 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:44.175 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:44.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:44.177 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:20:21.926078Z" 2023-11-02T19:24:44.177 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:44.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:44.178 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:44.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:44.178 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:44.179 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:24:44.179 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:24:44.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:44.179 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:44Z" 2023-11-02T19:24:44.180 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:44.181 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:20:22.019111Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051808", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:20:21.987436Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046076", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:20:21.926078Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:24:44Z" } 2023-11-02T19:24:44.181 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:20:22.019111Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051808', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:20:21.987436Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046076', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:20:21.926078Z'}}] 2023-11-02T19:24:44.182 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:24:44.182 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051808', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046076'} 2023-11-02T19:24:44.183 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032480 target=00000000000000000000:00000000000000032339, shard=1 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051808, shard=3 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046076 2023-11-02T19:24:49.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:49.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:49.185 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:49.233 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:49.234 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:49.326 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:49.326 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:49.326 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:49.327 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:49.327 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:49.327 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:24:49.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:49.328 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:49.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:49.328 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:49.328 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:49.329 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:49.329 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:24:49.329 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:49.329 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:24:49.330 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:49.330 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:24:49.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:49.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:49.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:49.331 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:49.331 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:49.331 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:49.331 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000051949", 2023-11-02T19:24:49.332 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:49.332 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:24:49.332 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:49.332 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.081651Z" 2023-11-02T19:24:49.332 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:49.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:49.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:49.333 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:49.333 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:49.334 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:49.334 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:24:49.334 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:49.334 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:49.335 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:49.335 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:24:49.335 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:49.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:49.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:49.336 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:49.336 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:49.336 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:49.336 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:24:49.337 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:49.337 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:49.337 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:49.337 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:24:49.337 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:49.338 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:49.338 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:49.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:49.338 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:49.338 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:24:49.339 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:24:49.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:49.339 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:49Z" 2023-11-02T19:24:49.339 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:49.340 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051949", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.081651Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:24:49Z" } 2023-11-02T19:24:49.341 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051949', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.081651Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:24:49.341 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:24:49.342 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051949', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:24:49.342 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051949 2023-11-02T19:24:54.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:54.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:54.344 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:54.394 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:54.394 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:54.489 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:54.489 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:54.489 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:54.490 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:54.490 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:54.490 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:24:54.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:54.491 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:54.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:54.491 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:54.491 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:54.491 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:54.492 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:24:54.492 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:54.492 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:24:54.492 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:54.493 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:24:54.493 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:54.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:54.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:54.493 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:54.494 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:54.494 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:54.494 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:24:54.494 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:54.495 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:24:54.495 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:54.495 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:24:54.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:54.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:54.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:54.496 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:54.496 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:54.496 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:54.496 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:24:54.497 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:54.497 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:54.497 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:54.497 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:24:54.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:54.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:54.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:54.498 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:54.498 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:54.498 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:24:54.499 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:24:54.499 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:54.499 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:54.499 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:54.499 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:24:54.500 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:54.500 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:54.500 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:54.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:54.500 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:54.501 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:24:54.501 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:24:54.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:54.501 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:54Z" 2023-11-02T19:24:54.502 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:54.503 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:24:54Z" } 2023-11-02T19:24:54.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:24:54.504 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:24:54.504 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:24:54.504 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:24:54.505 DEBUG:boto:path=/ibtljn-73/ 2023-11-02T19:24:54.505 DEBUG:boto:auth_path=/ibtljn-73/ 2023-11-02T19:24:54.506 DEBUG:boto:Method: HEAD 2023-11-02T19:24:54.506 DEBUG:boto:Path: /ibtljn-73/ 2023-11-02T19:24:54.506 DEBUG:boto:Data: 2023-11-02T19:24:54.507 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.507 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.508 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.508 DEBUG:boto:Params: {} 2023-11-02T19:24:54.509 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:24:54.509 DEBUG:boto:Token: None 2023-11-02T19:24:54.510 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-73/ 2023-11-02T19:24:54.510 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:vkvi+WvpMgMLuC96I1LGSqMc6Sc= 2023-11-02T19:24:54.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vkvi+WvpMgMLuC96I1LGSqMc6Sc=', 'Content-Length': '0'} 2023-11-02T19:24:54.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c054a0126662d0f8-006543f786-2bca-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.520 DEBUG:boto:path=/ibtljn-73/obj5 2023-11-02T19:24:54.521 DEBUG:boto:auth_path=/ibtljn-73/obj5 2023-11-02T19:24:54.521 DEBUG:boto:Method: HEAD 2023-11-02T19:24:54.521 DEBUG:boto:Path: /ibtljn-73/obj5 2023-11-02T19:24:54.522 DEBUG:boto:Data: 2023-11-02T19:24:54.522 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.523 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.523 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.524 DEBUG:boto:Params: {} 2023-11-02T19:24:54.524 DEBUG:boto:Token: None 2023-11-02T19:24:54.524 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-73/obj5 2023-11-02T19:24:54.525 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:OuBzSyXwNOicXNy9wGrK8E+61Dg= 2023-11-02T19:24:54.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OuBzSyXwNOicXNy9wGrK8E+61Dg=', 'Content-Length': '0'} 2023-11-02T19:24:54.528 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000891642c2049aea3c-006543f786-2bca-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.530 DEBUG:boto:path=/ibtljn-73/obj6 2023-11-02T19:24:54.530 DEBUG:boto:auth_path=/ibtljn-73/obj6 2023-11-02T19:24:54.531 DEBUG:boto:Method: HEAD 2023-11-02T19:24:54.531 DEBUG:boto:Path: /ibtljn-73/obj6 2023-11-02T19:24:54.532 DEBUG:boto:Data: 2023-11-02T19:24:54.532 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.532 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.533 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.533 DEBUG:boto:Params: {} 2023-11-02T19:24:54.534 DEBUG:boto:Token: None 2023-11-02T19:24:54.534 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-73/obj6 2023-11-02T19:24:54.535 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:WjphAucY3F/Ibk8VUpKa1TKbmx0= 2023-11-02T19:24:54.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WjphAucY3F/Ibk8VUpKa1TKbmx0=', 'Content-Length': '0'} 2023-11-02T19:24:54.538 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007258fa1e15b76434-006543f786-2bca-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.538 DEBUG:boto:path=/ibtljn-73/obj7 2023-11-02T19:24:54.539 DEBUG:boto:auth_path=/ibtljn-73/obj7 2023-11-02T19:24:54.539 DEBUG:boto:Method: HEAD 2023-11-02T19:24:54.540 DEBUG:boto:Path: /ibtljn-73/obj7 2023-11-02T19:24:54.540 DEBUG:boto:Data: 2023-11-02T19:24:54.540 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.541 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.541 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.542 DEBUG:boto:Params: {} 2023-11-02T19:24:54.542 DEBUG:boto:Token: None 2023-11-02T19:24:54.543 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-73/obj7 2023-11-02T19:24:54.543 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:yoqauodUPsxQh7ujmqABujglIgE= 2023-11-02T19:24:54.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:yoqauodUPsxQh7ujmqABujglIgE=', 'Content-Length': '0'} 2023-11-02T19:24:54.546 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000009944ddfe8b83cfd2-006543f786-2bca-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.546 DEBUG:boto:path=/ibtljn-73/obj8 2023-11-02T19:24:54.547 DEBUG:boto:auth_path=/ibtljn-73/obj8 2023-11-02T19:24:54.547 DEBUG:boto:Method: HEAD 2023-11-02T19:24:54.548 DEBUG:boto:Path: /ibtljn-73/obj8 2023-11-02T19:24:54.548 DEBUG:boto:Data: 2023-11-02T19:24:54.548 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.549 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.549 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.550 DEBUG:boto:Params: {} 2023-11-02T19:24:54.550 DEBUG:boto:Token: None 2023-11-02T19:24:54.551 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-73/obj8 2023-11-02T19:24:54.551 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:QHjLJPpKltQMUWV/L2hmWX4srKI= 2023-11-02T19:24:54.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:QHjLJPpKltQMUWV/L2hmWX4srKI=', 'Content-Length': '0'} 2023-11-02T19:24:54.555 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000158b5d628b12bdb8-006543f786-2bca-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.555 DEBUG:boto:path=/ibtljn-72/ 2023-11-02T19:24:54.556 DEBUG:boto:auth_path=/ibtljn-72/ 2023-11-02T19:24:54.556 DEBUG:boto:Method: HEAD 2023-11-02T19:24:54.557 DEBUG:boto:Path: /ibtljn-72/ 2023-11-02T19:24:54.557 DEBUG:boto:Data: 2023-11-02T19:24:54.557 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.558 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.558 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.559 DEBUG:boto:Params: {} 2023-11-02T19:24:54.559 DEBUG:boto:Token: None 2023-11-02T19:24:54.560 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-72/ 2023-11-02T19:24:54.560 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:5f8ZD9FBvwiLneua+3w1chfsWAI= 2023-11-02T19:24:54.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5f8ZD9FBvwiLneua+3w1chfsWAI=', 'Content-Length': '0'} 2023-11-02T19:24:54.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d85a67eccd673d3-006543f786-2bca-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.565 DEBUG:boto:path=/ibtljn-72/obj7 2023-11-02T19:24:54.565 DEBUG:boto:auth_path=/ibtljn-72/obj7 2023-11-02T19:24:54.566 DEBUG:boto:Method: HEAD 2023-11-02T19:24:54.566 DEBUG:boto:Path: /ibtljn-72/obj7 2023-11-02T19:24:54.567 DEBUG:boto:Data: 2023-11-02T19:24:54.567 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.568 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.568 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.568 DEBUG:boto:Params: {} 2023-11-02T19:24:54.569 DEBUG:boto:Token: None 2023-11-02T19:24:54.569 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-72/obj7 2023-11-02T19:24:54.570 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d/CpltFPjn3qbRBqzX83gKWBwQg= 2023-11-02T19:24:54.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d/CpltFPjn3qbRBqzX83gKWBwQg=', 'Content-Length': '0'} 2023-11-02T19:24:54.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-73:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f92a89d04ad5597c-006543f786-2bca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.575 DEBUG:boto:path=/ibtljn-72/obj7 2023-11-02T19:24:54.575 DEBUG:boto:auth_path=/ibtljn-72/obj7 2023-11-02T19:24:54.576 DEBUG:boto:Method: GET 2023-11-02T19:24:54.576 DEBUG:boto:Path: /ibtljn-72/obj7 2023-11-02T19:24:54.577 DEBUG:boto:Data: 2023-11-02T19:24:54.577 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.577 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.578 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.578 DEBUG:boto:Params: {} 2023-11-02T19:24:54.579 DEBUG:boto:Token: None 2023-11-02T19:24:54.579 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-72/obj7 2023-11-02T19:24:54.580 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nuV1DR+ylKnPowc2h8a5CGSLDPE= 2023-11-02T19:24:54.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nuV1DR+ylKnPowc2h8a5CGSLDPE=', 'Content-Length': '0'} 2023-11-02T19:24:54.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-73:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a1d7543ff85fdae-006543f786-2bca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.630 DEBUG:boto:path=/ibtljn-72/obj8 2023-11-02T19:24:54.631 DEBUG:boto:auth_path=/ibtljn-72/obj8 2023-11-02T19:24:54.631 DEBUG:boto:Method: HEAD 2023-11-02T19:24:54.632 DEBUG:boto:Path: /ibtljn-72/obj8 2023-11-02T19:24:54.632 DEBUG:boto:Data: 2023-11-02T19:24:54.633 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.633 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.633 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.634 DEBUG:boto:Params: {} 2023-11-02T19:24:54.634 DEBUG:boto:Token: None 2023-11-02T19:24:54.635 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-72/obj8 2023-11-02T19:24:54.635 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LVj+UwL6iomedFDGpTstNAU2x/8= 2023-11-02T19:24:54.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LVj+UwL6iomedFDGpTstNAU2x/8=', 'Content-Length': '0'} 2023-11-02T19:24:54.639 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-73:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba39fa2dffcaa781-006543f786-2bca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.640 DEBUG:boto:path=/ibtljn-72/obj8 2023-11-02T19:24:54.640 DEBUG:boto:auth_path=/ibtljn-72/obj8 2023-11-02T19:24:54.640 DEBUG:boto:Method: GET 2023-11-02T19:24:54.641 DEBUG:boto:Path: /ibtljn-72/obj8 2023-11-02T19:24:54.641 DEBUG:boto:Data: 2023-11-02T19:24:54.642 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.642 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.643 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.643 DEBUG:boto:Params: {} 2023-11-02T19:24:54.644 DEBUG:boto:Token: None 2023-11-02T19:24:54.644 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-72/obj8 2023-11-02T19:24:54.645 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l/nDAVptSiowD/yBzoDZIfjJj8M= 2023-11-02T19:24:54.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l/nDAVptSiowD/yBzoDZIfjJj8M=', 'Content-Length': '0'} 2023-11-02T19:24:54.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-73:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4e11ef07bebd056-006543f786-2bca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.649 DEBUG:boto:path=/ibtljn-72/obj7 2023-11-02T19:24:54.650 DEBUG:boto:auth_path=/ibtljn-72/obj7 2023-11-02T19:24:54.650 DEBUG:boto:Method: HEAD 2023-11-02T19:24:54.651 DEBUG:boto:Path: /ibtljn-72/obj7 2023-11-02T19:24:54.651 DEBUG:boto:Data: 2023-11-02T19:24:54.651 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.652 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.652 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.653 DEBUG:boto:Params: {} 2023-11-02T19:24:54.653 DEBUG:boto:Token: None 2023-11-02T19:24:54.654 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-72/obj7 2023-11-02T19:24:54.654 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:d/CpltFPjn3qbRBqzX83gKWBwQg= 2023-11-02T19:24:54.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d/CpltFPjn3qbRBqzX83gKWBwQg=', 'Content-Length': '0'} 2023-11-02T19:24:54.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-73:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7766dd4a2738e8a-006543f786-2bca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.659 DEBUG:boto:path=/ibtljn-72/obj7 2023-11-02T19:24:54.659 DEBUG:boto:auth_path=/ibtljn-72/obj7 2023-11-02T19:24:54.660 DEBUG:boto:Method: GET 2023-11-02T19:24:54.660 DEBUG:boto:Path: /ibtljn-72/obj7 2023-11-02T19:24:54.660 DEBUG:boto:Data: 2023-11-02T19:24:54.661 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.661 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.662 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.662 DEBUG:boto:Params: {} 2023-11-02T19:24:54.663 DEBUG:boto:Token: None 2023-11-02T19:24:54.663 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-72/obj7 2023-11-02T19:24:54.664 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nuV1DR+ylKnPowc2h8a5CGSLDPE= 2023-11-02T19:24:54.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nuV1DR+ylKnPowc2h8a5CGSLDPE=', 'Content-Length': '0'} 2023-11-02T19:24:54.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-73:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068445e692014b88b-006543f786-2bca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.714 DEBUG:boto:path=/ibtljn-72/obj8 2023-11-02T19:24:54.715 DEBUG:boto:auth_path=/ibtljn-72/obj8 2023-11-02T19:24:54.715 DEBUG:boto:Method: HEAD 2023-11-02T19:24:54.716 DEBUG:boto:Path: /ibtljn-72/obj8 2023-11-02T19:24:54.716 DEBUG:boto:Data: 2023-11-02T19:24:54.717 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.717 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.718 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.718 DEBUG:boto:Params: {} 2023-11-02T19:24:54.718 DEBUG:boto:Token: None 2023-11-02T19:24:54.719 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-72/obj8 2023-11-02T19:24:54.719 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LVj+UwL6iomedFDGpTstNAU2x/8= 2023-11-02T19:24:54.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LVj+UwL6iomedFDGpTstNAU2x/8=', 'Content-Length': '0'} 2023-11-02T19:24:54.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-73:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000683cbf8e3796bc35-006543f786-2bca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.723 DEBUG:boto:path=/ibtljn-72/obj8 2023-11-02T19:24:54.724 DEBUG:boto:auth_path=/ibtljn-72/obj8 2023-11-02T19:24:54.724 DEBUG:boto:Method: GET 2023-11-02T19:24:54.725 DEBUG:boto:Path: /ibtljn-72/obj8 2023-11-02T19:24:54.725 DEBUG:boto:Data: 2023-11-02T19:24:54.725 DEBUG:boto:Headers: {} 2023-11-02T19:24:54.726 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:24:54.726 DEBUG:boto:Port: 8001 2023-11-02T19:24:54.727 DEBUG:boto:Params: {} 2023-11-02T19:24:54.727 DEBUG:boto:Token: None 2023-11-02T19:24:54.728 DEBUG:boto:StringToSign: GET Thu, 02 Nov 2023 19:24:54 GMT /ibtljn-72/obj8 2023-11-02T19:24:54.728 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:l/nDAVptSiowD/yBzoDZIfjJj8M= 2023-11-02T19:24:54.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:24:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:l/nDAVptSiowD/yBzoDZIfjJj8M=', 'Content-Length': '0'} 2023-11-02T19:24:54.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 19:24:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-73:d61b2bea-59a8-4454-8491-ac3afbd04937.10291.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000151301add15add90-006543f786-2bca-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 19:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:24:54.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:24:54.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'ibtljn-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:24:54.735 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket ibtljn-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:24:54.779 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:54.780 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:54.902 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_policy":{"groups":[]}} 2023-11-02T19:24:54.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:24:54.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:24:54.904 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:24:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:55.083 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_policy":{"groups":[]}} 2023-11-02T19:24:55.084 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... ok 2023-11-02T19:24:55.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:55.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:55.086 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:55.130 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:55.131 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:55.194 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:24:55.225 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:24:55.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:55.226 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:24:55.226 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:24:55.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:55.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:55.227 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:24:55.227 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:24:55.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:55.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:55.228 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698953094.851156_840.1", 2023-11-02T19:24:55.228 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:24:54.851156Z" 2023-11-02T19:24:55.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:55.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:55.228 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:24:55.229 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:24:55.229 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:55.229 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:24:55.230 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953094.851156_840.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:55.231 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:24:55.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:55.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:24:55.232 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:24:55.276 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:24:55.276 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:24:55.335 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:55.329+0000 7f6d641e7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:55.337 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:24:55.333+0000 7f6d641e7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:24:55.366 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:24:55.366 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:24:55.366 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:24:55.367 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:24:55.367 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:24:55.367 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:24:55.367 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:55.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:55.368 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:24:55.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:55.368 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:24:55.369 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:55.369 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:55.369 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:24:55.369 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:55.369 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:24:55.370 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:55.370 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:24:55.370 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:55.370 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:55.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:55.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:55.371 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:24:55.371 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:55.372 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:55.372 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:24:55.372 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:55.372 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:55.372 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:55.373 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:24:55.373 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:55.373 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:55.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:55.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:55.374 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:24:55.374 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:55.374 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:55.375 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:24:55.375 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:55.375 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:55.375 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:55.376 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:22:28.822959Z", 2023-11-02T19:24:55.376 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:55.376 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:55.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:55.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:24:55.377 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:24:55.377 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:24:55.377 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:24:55.377 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:24:55.378 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:24:55.378 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:24:55.378 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:24:55.378 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:24:55.379 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:24:55.379 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:55.379 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:24:55.379 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:24:55.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:55.380 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:24:55.380 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:24:55.380 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:24:55.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:24:55.381 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:24:55Z" 2023-11-02T19:24:55.381 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:24:55.382 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952948.822959_838.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:22:28.822959Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:24:55Z" } 2023-11-02T19:24:55.383 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952948.822959_838.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:22:28.822959Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:24:55.383 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953094.851156_840.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:55.383 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:24:55.384 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698953094.851156_840.1 target=1_1698952948.822959_838.1 2023-11-02T19:25:00.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:00.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:00.386 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:00.432 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:00.432 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:25:00.480+0000 7f978c818f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:25:00.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:25:00.484+0000 7f978c818f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:25:00.563 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:00.563 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:00.563 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:00.564 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:00.564 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:00.564 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:25:00.564 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:00.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:00.565 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:00.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:00.565 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:00.566 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:00.566 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:00.566 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:25:00.566 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:00.567 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:25:00.567 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:00.567 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:25:00.567 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:00.567 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:00.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:00.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:00.568 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:00.568 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:00.569 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:00.569 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:25:00.569 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:00.569 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:00.570 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:00.570 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:25:00.570 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:00.570 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:00.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:00.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:00.571 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:00.571 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:00.572 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:00.572 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:25:00.572 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:00.572 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:00.573 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:00.573 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:22:28.822959Z", 2023-11-02T19:25:00.573 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:00.573 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:00.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:00.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:00.574 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:00.574 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:00.575 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:00.575 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:25:00.576 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:00.576 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:00.576 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:00.576 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:25:00.576 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:00.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:00.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:00.577 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:00.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:00.578 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:00.578 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:25:00.578 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:25:00.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:00.578 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:00Z" 2023-11-02T19:25:00.579 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:00.580 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952948.822959_838.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:22:28.822959Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:25:00Z" } 2023-11-02T19:25:00.580 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952948.822959_838.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:22:28.822959Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:25:00.581 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953094.851156_840.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:00.582 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:00.582 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698953094.851156_840.1 target=1_1698952948.822959_838.1 2023-11-02T19:25:05.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:05.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:05.585 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:05.631 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:05.631 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:05.700 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:25:05.692+0000 7f94aa895f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:25:05.702 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:25:05.696+0000 7f94aa895f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:25:05.731 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:05.732 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:05.732 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:05.732 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:05.732 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:05.732 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:25:05.733 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:05.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:05.733 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:05.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:05.734 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:05.734 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:05.734 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:05.734 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:25:05.734 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:05.735 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:25:05.735 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:05.735 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:25:05.735 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:05.736 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:05.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:05.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:05.736 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:05.737 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:05.737 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:05.737 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:25:05.737 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:05.737 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:05.738 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:05.738 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:25:05.738 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:05.738 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:05.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:05.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:05.739 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:05.739 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:05.739 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:05.740 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:25:05.740 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:05.740 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:05.740 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:05.740 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:22:28.822959Z", 2023-11-02T19:25:05.741 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:05.741 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:05.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:05.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:05.741 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:05.742 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:05.742 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:05.742 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:25:05.742 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:05.743 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:05.743 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:05.743 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:25:05.743 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:05.744 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:05.744 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:05.744 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:05.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:05.744 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:05.745 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:25:05.745 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:25:05.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:05.745 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:05Z" 2023-11-02T19:25:05.746 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:05.747 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698952948.822959_838.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:22:28.822959Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:25:05Z" } 2023-11-02T19:25:05.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952948.822959_838.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:22:28.822959Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:25:05.748 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953094.851156_840.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:05.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:05.748 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698953094.851156_840.1 target=1_1698952948.822959_838.1 2023-11-02T19:25:10.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:10.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:10.751 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:10.795 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:10.796 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:10.856 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:25:10.848+0000 7f4067bcaf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:25:10.857 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:25:10.852+0000 7f4067bcaf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:25:10.887 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:10.888 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:10.888 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:10.888 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:10.888 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:10.889 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:25:10.889 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:10.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:10.889 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:10.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:10.889 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:10.890 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:10.890 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:10.890 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:25:10.890 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:10.891 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:25:10.891 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:10.891 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:25:10.891 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:10.892 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:10.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:10.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:10.892 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:10.892 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:10.893 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:10.893 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:25:10.893 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:10.893 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:10.894 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:10.894 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:25:10.894 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:10.894 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:10.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:10.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:10.895 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:10.895 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:10.895 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:10.896 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698953094.851156_840.1", 2023-11-02T19:25:10.896 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:10.896 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:10.896 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:10.896 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:54.851156Z", 2023-11-02T19:25:10.897 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:10.897 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:10.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:10.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:10.898 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:10.898 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:10.898 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:10.898 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:25:10.898 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:10.899 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:10.899 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:10.899 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:25:10.899 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:10.899 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:10.900 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:10.900 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:10.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:10.900 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:10.901 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:25:10.901 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:25:10.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:10.901 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:10Z" 2023-11-02T19:25:10.901 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:10.902 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698953094.851156_840.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:54.851156Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:25:10Z" } 2023-11-02T19:25:10.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698953094.851156_840.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:54.851156Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:25:10.903 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953094.851156_840.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:10.904 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953094.851156_840.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:10.904 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:25:10.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:10.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:10.905 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:25:10.986 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:10.987 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:11.083 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[]} 2023-11-02T19:25:11.084 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-74 2023-11-02T19:25:11.084 DEBUG:boto:path=/ibtljn-74/ 2023-11-02T19:25:11.084 DEBUG:boto:auth_path=/ibtljn-74/ 2023-11-02T19:25:11.085 DEBUG:boto:Method: PUT 2023-11-02T19:25:11.085 DEBUG:boto:Path: /ibtljn-74/ 2023-11-02T19:25:11.086 DEBUG:boto:Data: 2023-11-02T19:25:11.086 DEBUG:boto:Headers: {} 2023-11-02T19:25:11.086 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:25:11.087 DEBUG:boto:Port: 8000 2023-11-02T19:25:11.087 DEBUG:boto:Params: {} 2023-11-02T19:25:11.088 DEBUG:boto:Token: None 2023-11-02T19:25:11.088 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 19:25:11 GMT /ibtljn-74/ 2023-11-02T19:25:11.089 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:o4bRco7x9/uzT/9dpHocmbebtaQ= 2023-11-02T19:25:11.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:25:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o4bRco7x9/uzT/9dpHocmbebtaQ=', 'Content-Length': '0'} 2023-11-02T19:25:11.124 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f8cbaf2325839c19-006543f797-2bbf-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:25:11.124 INFO:rgw_multi.tests:create bucket zone=a1 name=ibtljn-75 2023-11-02T19:25:11.125 DEBUG:boto:path=/ibtljn-75/ 2023-11-02T19:25:11.125 DEBUG:boto:auth_path=/ibtljn-75/ 2023-11-02T19:25:11.126 DEBUG:boto:Method: PUT 2023-11-02T19:25:11.126 DEBUG:boto:Path: /ibtljn-75/ 2023-11-02T19:25:11.126 DEBUG:boto:Data: 2023-11-02T19:25:11.127 DEBUG:boto:Headers: {} 2023-11-02T19:25:11.127 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:25:11.128 DEBUG:boto:Port: 8000 2023-11-02T19:25:11.128 DEBUG:boto:Params: {} 2023-11-02T19:25:11.129 DEBUG:boto:Token: None 2023-11-02T19:25:11.129 DEBUG:boto:StringToSign: PUT Thu, 02 Nov 2023 19:25:11 GMT /ibtljn-75/ 2023-11-02T19:25:11.129 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nBgnqmX7jX4MCk1O4b/LL/fc2X8= 2023-11-02T19:25:11.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:25:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nBgnqmX7jX4MCk1O4b/LL/fc2X8=', 'Content-Length': '0'} 2023-11-02T19:25:11.160 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000ff2e7d868db2017-006543f797-2bbf-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:25:11.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:11.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:11.162 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:11.213 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:11.214 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:11.266 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:25:11.296 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:25:11.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:11.296 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:25:11.297 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:25:11.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:11.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:11.297 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:25:11.298 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:25:11.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:11.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:11.298 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698953111.156002_848.1", 2023-11-02T19:25:11.298 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:25:11.156002Z" 2023-11-02T19:25:11.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:11.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:11.299 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:25:11.299 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:25:11.300 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:11.300 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:25:11.301 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:11.301 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:25:11.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:11.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:11.303 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:11.348 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:11.349 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:11.406 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:25:11.400+0000 7fab723c8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:25:11.407 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:25:11.400+0000 7fab723c8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:25:11.447 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:11.447 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:11.447 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:11.448 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:11.448 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:11.448 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:25:11.448 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:11.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:11.449 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:11.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:11.449 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:11.450 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:11.450 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:11.450 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:25:11.450 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:11.451 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:25:11.451 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:11.451 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:25:11.451 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:11.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:11.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:11.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:11.452 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:11.452 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:11.453 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:11.453 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:25:11.453 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:11.453 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:11.454 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:11.454 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:25:11.454 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:11.454 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:11.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:11.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:11.455 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:11.455 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:11.455 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:11.456 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698953111.108996_841.1", 2023-11-02T19:25:11.456 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:11.456 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:11.456 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:11.456 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:25:11.108996Z", 2023-11-02T19:25:11.457 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:11.457 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:11.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:11.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:11.458 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:11.458 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:11.458 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:11.458 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:25:11.458 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:11.459 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:11.459 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:11.459 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:25:11.459 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:11.460 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:11.460 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:11.460 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:11.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:11.460 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:11.461 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:25:11.461 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:25:11.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:11.461 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:11Z" 2023-11-02T19:25:11.462 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:11.463 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698953111.108996_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:25:11.108996Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:25:11Z" } 2023-11-02T19:25:11.463 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698953111.108996_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:25:11.108996Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:25:11.464 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:11.464 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.108996_841.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:11.464 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1698953111.156002_848.1 target=1_1698953111.108996_841.1 2023-11-02T19:25:16.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:16.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:16.467 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:16.510 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:16.510 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:25:16.564+0000 7f2561ac0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:25:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:25:16.568+0000 7f2561ac0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:25:16.601 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:16.601 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:16.602 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:16.602 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:16.602 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:16.602 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:25:16.603 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:16.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:16.603 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:16.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:16.603 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:16.604 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:16.604 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:16.604 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:25:16.604 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:16.604 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:25:16.605 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:16.605 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:25:16.605 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:16.605 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:16.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:16.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:16.606 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:16.606 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:16.606 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:16.607 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:25:16.607 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:16.607 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:16.607 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:16.607 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:25:16.608 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:16.608 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:16.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:16.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:16.608 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:16.609 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:16.609 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:16.609 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698953111.156002_848.1", 2023-11-02T19:25:16.609 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:16.610 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:16.610 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:16.610 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:25:11.156002Z", 2023-11-02T19:25:16.610 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:16.610 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:16.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:16.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:16.611 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:16.611 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:16.611 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:16.612 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:25:16.612 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:16.612 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:16.612 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:25:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:25:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:25:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:16.615 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:16Z" 2023-11-02T19:25:16.615 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:16.616 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698953111.156002_848.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:25:11.156002Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:25:16Z" } 2023-11-02T19:25:16.617 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698953111.156002_848.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:25:11.156002Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:25:16.617 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:16.618 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:16.618 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:25:16.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:16.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:16.619 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:25:16.664 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:16.665 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:16.759 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-11-02T19:25:16.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:16.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:16.761 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:25:16.809 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:16.809 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:16.907 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-11-02T19:25:16.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-74', '--dest-bucket=ibtljn-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:16.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-74', '--dest-bucket=ibtljn-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:16.908 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-ibtljn-74 --source-zones=a1 --dest-zones=a2 --source-bucket=ibtljn-74 --dest-bucket=ibtljn-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:25:16.960 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:16.961 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:17.053 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-74","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:25:17.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-74', '--dest-bucket=ibtljn-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:17.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-74', '--dest-bucket=ibtljn-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:17.054 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-ibtljn-75 --source-zones=a1 --dest-zones=a2 --source-bucket=ibtljn-74 --dest-bucket=ibtljn-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:25:17.106 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:17.107 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:17.206 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-74","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-ibtljn-75","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:25:17.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-74', '--dest-bucket=ibtljn-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:17.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-74', '--dest-bucket=ibtljn-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:17.207 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=ibtljn-74 --dest-bucket=ibtljn-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:25:17.258 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:17.259 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:17.353 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-74","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-ibtljn-75","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:25:17.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:17.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:17.355 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:25:17.400 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:17.401 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:17.496 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-74","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-ibtljn-75","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:25:17.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:17.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:17.498 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:25:17.542 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:17.542 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:17.659 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:17.659 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:25:17.659 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 22, 2023-11-02T19:25:17.660 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T19:25:17.660 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2023-11-02T19:25:17.660 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2023-11-02T19:25:17.660 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:25:17.660 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2023-11-02T19:25:17.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.661 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "a", 2023-11-02T19:25:17.661 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2023-11-02T19:25:17.661 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:25:17.662 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": true, 2023-11-02T19:25:17.662 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:25:17.662 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:25:17.662 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:25:17.662 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:25:17.663 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:25:17.663 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:25:17.663 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:25:17.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.664 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:25:17.664 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a2", 2023-11-02T19:25:17.664 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:25:17.664 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T19:25:17.664 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:25:17.665 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:25:17.665 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:25:17.665 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:25:17.665 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:25:17.666 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:25:17.666 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:25:17.666 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:25:17.667 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:25:17.667 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:25:17.667 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:25:17.668 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:25:17.668 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.668 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:25:17.669 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a1", 2023-11-02T19:25:17.669 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:25:17.669 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:25:17.669 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:25:17.670 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:25:17.670 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:25:17.670 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:25:17.670 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:25:17.670 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:25:17.671 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:25:17.671 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:25:17.671 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:25:17.671 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:25:17.672 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:25:17.672 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:25:17.672 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.672 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.672 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:25:17.673 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:25:17.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.673 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:25:17.673 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:25:17.673 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:25:17.674 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:25:17.674 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.674 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.674 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:25:17.675 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:25:17.675 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:25:17.675 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:25:17.675 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [ 2023-11-02T19:25:17.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.676 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-group", 2023-11-02T19:25:17.676 INFO:teuthology.orchestra.run.smithi017.stdout: "data_flow": { 2023-11-02T19:25:17.676 INFO:teuthology.orchestra.run.smithi017.stdout: "symmetrical": [ 2023-11-02T19:25:17.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.676 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-flow1", 2023-11-02T19:25:17.677 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:25:17.677 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:25:17.677 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:25:17.677 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.678 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.678 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.678 INFO:teuthology.orchestra.run.smithi017.stdout: "pipes": [ 2023-11-02T19:25:17.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.679 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-pipe-ibtljn-74", 2023-11-02T19:25:17.679 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:25:17.679 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-74", 2023-11-02T19:25:17.679 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:25:17.679 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:25:17.680 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.680 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": { 2023-11-02T19:25:17.680 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-74", 2023-11-02T19:25:17.681 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:25:17.681 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a" 2023-11-02T19:25:17.681 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.681 INFO:teuthology.orchestra.run.smithi017.stdout: "params": { 2023-11-02T19:25:17.682 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:25:17.682 INFO:teuthology.orchestra.run.smithi017.stdout: "filter": { 2023-11-02T19:25:17.682 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [] 2023-11-02T19:25:17.682 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.683 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": {}, 2023-11-02T19:25:17.683 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2023-11-02T19:25:17.683 INFO:teuthology.orchestra.run.smithi017.stdout: "mode": "system", 2023-11-02T19:25:17.683 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "" 2023-11-02T19:25:17.684 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.684 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-pipe-ibtljn-75", 2023-11-02T19:25:17.685 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:25:17.685 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-74", 2023-11-02T19:25:17.685 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:25:17.685 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:25:17.685 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.686 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": { 2023-11-02T19:25:17.686 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-75", 2023-11-02T19:25:17.686 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:25:17.687 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a" 2023-11-02T19:25:17.687 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.687 INFO:teuthology.orchestra.run.smithi017.stdout: "params": { 2023-11-02T19:25:17.687 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:25:17.688 INFO:teuthology.orchestra.run.smithi017.stdout: "filter": { 2023-11-02T19:25:17.688 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [] 2023-11-02T19:25:17.688 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.689 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": {}, 2023-11-02T19:25:17.689 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2023-11-02T19:25:17.689 INFO:teuthology.orchestra.run.smithi017.stdout: "mode": "system", 2023-11-02T19:25:17.689 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "" 2023-11-02T19:25:17.689 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.690 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "sync-pipe", 2023-11-02T19:25:17.690 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:25:17.691 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-74", 2023-11-02T19:25:17.691 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:25:17.691 INFO:teuthology.orchestra.run.smithi017.stdout: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:25:17.691 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.692 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": { 2023-11-02T19:25:17.692 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket": "ibtljn-75", 2023-11-02T19:25:17.692 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:25:17.692 INFO:teuthology.orchestra.run.smithi017.stdout: "39a01552-6d59-4ce7-b7e5-915ca69b979a" 2023-11-02T19:25:17.693 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.693 INFO:teuthology.orchestra.run.smithi017.stdout: "params": { 2023-11-02T19:25:17.693 INFO:teuthology.orchestra.run.smithi017.stdout: "source": { 2023-11-02T19:25:17.693 INFO:teuthology.orchestra.run.smithi017.stdout: "filter": { 2023-11-02T19:25:17.694 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [] 2023-11-02T19:25:17.694 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.694 INFO:teuthology.orchestra.run.smithi017.stdout: "dest": {}, 2023-11-02T19:25:17.694 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2023-11-02T19:25:17.695 INFO:teuthology.orchestra.run.smithi017.stdout: "mode": "system", 2023-11-02T19:25:17.695 INFO:teuthology.orchestra.run.smithi017.stdout: "user": "" 2023-11-02T19:25:17.695 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.695 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.696 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:25:17.696 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "enabled" 2023-11-02T19:25:17.696 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.696 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.697 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:25:17.697 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:25:17.697 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.698 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "b", 2023-11-02T19:25:17.698 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2023-11-02T19:25:17.698 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "", 2023-11-02T19:25:17.698 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": false, 2023-11-02T19:25:17.698 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:25:17.699 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:25:17.699 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:25:17.699 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2023-11-02T19:25:17.699 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2023-11-02T19:25:17.700 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:25:17.700 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2023-11-02T19:25:17.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.701 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:25:17.702 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b1", 2023-11-02T19:25:17.702 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:25:17.702 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:25:17.702 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:25:17.702 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:25:17.703 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:25:17.703 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:25:17.703 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:25:17.703 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:25:17.704 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:25:17.704 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:25:17.704 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:25:17.704 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:25:17.704 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:25:17.705 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:25:17.705 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.705 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:25:17.706 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b2", 2023-11-02T19:25:17.706 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [ 2023-11-02T19:25:17.706 INFO:teuthology.orchestra.run.smithi017.stdout: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T19:25:17.706 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:25:17.707 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": false, 2023-11-02T19:25:17.707 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": true, 2023-11-02T19:25:17.707 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2023-11-02T19:25:17.707 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": false, 2023-11-02T19:25:17.707 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2023-11-02T19:25:17.708 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": true, 2023-11-02T19:25:17.708 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2023-11-02T19:25:17.708 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "", 2023-11-02T19:25:17.708 INFO:teuthology.orchestra.run.smithi017.stdout: "supported_features": [ 2023-11-02T19:25:17.709 INFO:teuthology.orchestra.run.smithi017.stdout: "compress-encrypted", 2023-11-02T19:25:17.709 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:25:17.709 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.709 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.709 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:25:17.710 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2023-11-02T19:25:17.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.710 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2023-11-02T19:25:17.710 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2023-11-02T19:25:17.711 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2023-11-02T19:25:17.711 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2023-11-02T19:25:17.711 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.712 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:25:17.712 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2023-11-02T19:25:17.712 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:25:17.712 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2023-11-02T19:25:17.712 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2023-11-02T19:25:17.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.713 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled_features": [ 2023-11-02T19:25:17.713 INFO:teuthology.orchestra.run.smithi017.stdout: "resharding" 2023-11-02T19:25:17.713 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.714 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.714 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2023-11-02T19:25:17.714 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2023-11-02T19:25:17.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.714 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:25:17.715 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3699239738 2023-11-02T19:25:17.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.715 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:25:17.716 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2602129372 2023-11-02T19:25:17.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.716 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:25:17.716 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 2073569678 2023-11-02T19:25:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:25:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3449465395 2023-11-02T19:25:17.718 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.718 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:17.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.718 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "a", 2023-11-02T19:25:17.719 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:25:17.719 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2023-11-02T19:25:17.719 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2023-11-02T19:25:17.719 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:25:17.719 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:25:17.720 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:25:17.720 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:25:17.720 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:25:17.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.720 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2023-11-02T19:25:17.721 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2023-11-02T19:25:17.721 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2023-11-02T19:25:17.721 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2023-11-02T19:25:17.721 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2023-11-02T19:25:17.722 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2023-11-02T19:25:17.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.722 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2023-11-02T19:25:17.722 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:25:17.722 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:25:17.723 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:25:17.723 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:25:17.723 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:25:17.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.724 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2023-11-02T19:25:17.724 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:25:17.724 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:25:17.724 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:25:17.724 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:25:17.725 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:25:17.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.725 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2023-11-02T19:25:17.725 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2023-11-02T19:25:17.725 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2023-11-02T19:25:17.726 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2023-11-02T19:25:17.726 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2023-11-02T19:25:17.726 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2023-11-02T19:25:17.726 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:17.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:17.727 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:25:17.727 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "test-realm", 2023-11-02T19:25:17.727 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:17.727 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:17.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:17.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:25:17.729 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:25:17.772 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:17.772 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:17.854 INFO:teuthology.orchestra.run.smithi017.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-74","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-ibtljn-75","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:25:17.854 DEBUG:boto:path=/ibtljn-74/ 2023-11-02T19:25:17.855 DEBUG:boto:auth_path=/ibtljn-74/ 2023-11-02T19:25:17.855 DEBUG:boto:Method: HEAD 2023-11-02T19:25:17.856 DEBUG:boto:Path: /ibtljn-74/ 2023-11-02T19:25:17.856 DEBUG:boto:Data: 2023-11-02T19:25:17.857 DEBUG:boto:Headers: {} 2023-11-02T19:25:17.857 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:25:17.858 DEBUG:boto:Port: 8000 2023-11-02T19:25:17.858 DEBUG:boto:Params: {} 2023-11-02T19:25:17.859 DEBUG:boto:Token: None 2023-11-02T19:25:17.859 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:25:17 GMT /ibtljn-74/ 2023-11-02T19:25:17.860 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JYwK9a1HuWnXAfBs286Z72atqr4= 2023-11-02T19:25:17.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:25:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JYwK9a1HuWnXAfBs286Z72atqr4=', 'Content-Length': '0'} 2023-11-02T19:25:17.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d7e133c4a4b4df4-006543f79d-2da3-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:25:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:25:17.936 DEBUG:boto:path=/ibtljn-74/obj1 2023-11-02T19:25:17.937 DEBUG:boto:auth_path=/ibtljn-74/obj1 2023-11-02T19:25:17.937 DEBUG:boto:Method: PUT 2023-11-02T19:25:17.938 DEBUG:boto:Path: /ibtljn-74/obj1 2023-11-02T19:25:17.938 DEBUG:boto:Data: 2023-11-02T19:25:17.939 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:25:17.939 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:25:17.940 DEBUG:boto:Port: 8000 2023-11-02T19:25:17.940 DEBUG:boto:Params: {} 2023-11-02T19:25:17.940 DEBUG:boto:Token: None 2023-11-02T19:25:17.941 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:25:17 GMT /ibtljn-74/obj1 2023-11-02T19:25:17.941 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:Ndfr1Hm0X33RWd087dE1w+yRM4E= 2023-11-02T19:25:17.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:25:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ndfr1Hm0X33RWd087dE1w+yRM4E='} 2023-11-02T19:25:17.959 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000720827c3a2b64638-006543f79d-2da3-a1'), ('Date', 'Thu, 02 Nov 2023 19:25:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:25:17.960 DEBUG:boto:path=/ibtljn-74/ 2023-11-02T19:25:17.960 DEBUG:boto:auth_path=/ibtljn-74/ 2023-11-02T19:25:17.961 DEBUG:boto:Method: HEAD 2023-11-02T19:25:17.961 DEBUG:boto:Path: /ibtljn-74/ 2023-11-02T19:25:17.962 DEBUG:boto:Data: 2023-11-02T19:25:17.962 DEBUG:boto:Headers: {} 2023-11-02T19:25:17.963 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:25:17.963 DEBUG:boto:Port: 8000 2023-11-02T19:25:17.963 DEBUG:boto:Params: {} 2023-11-02T19:25:17.965 DEBUG:boto:Token: None 2023-11-02T19:25:17.965 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:25:17 GMT /ibtljn-74/ 2023-11-02T19:25:17.966 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:JYwK9a1HuWnXAfBs286Z72atqr4= 2023-11-02T19:25:17.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:25:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JYwK9a1HuWnXAfBs286Z72atqr4=', 'Content-Length': '0'} 2023-11-02T19:25:17.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062fa800b4bf051f2-006543f79d-2da3-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:25:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:25:17.973 DEBUG:boto:path=/ibtljn-74/obj2 2023-11-02T19:25:17.974 DEBUG:boto:auth_path=/ibtljn-74/obj2 2023-11-02T19:25:17.974 DEBUG:boto:Method: PUT 2023-11-02T19:25:17.974 DEBUG:boto:Path: /ibtljn-74/obj2 2023-11-02T19:25:17.975 DEBUG:boto:Data: 2023-11-02T19:25:17.975 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:25:17.976 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:25:17.976 DEBUG:boto:Port: 8000 2023-11-02T19:25:17.977 DEBUG:boto:Params: {} 2023-11-02T19:25:17.977 DEBUG:boto:Token: None 2023-11-02T19:25:17.978 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 02 Nov 2023 19:25:17 GMT /ibtljn-74/obj2 2023-11-02T19:25:17.978 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:9t4s4drzRFYmiscgsOyKcaYJ4Kw= 2023-11-02T19:25:17.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:25:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9t4s4drzRFYmiscgsOyKcaYJ4Kw='} 2023-11-02T19:25:17.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d877418519036df8-006543f79d-2da3-a1'), ('Date', 'Thu, 02 Nov 2023 19:25:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:25:17.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:17.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:17.998 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:18.052 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:18.052 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:18.148 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:25:18.191 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:25:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:25:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:25:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:25:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:25:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698953111.156002_848.1", 2023-11-02T19:25:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:25:11.156002Z" 2023-11-02T19:25:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:25:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:25:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:18.195 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:25:18.196 DEBUG:rgw_multi.tests:master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:18.197 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-11-02T19:25:18.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:18.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:18.198 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:18.254 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:18.255 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:18.336 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:25:18.328+0000 7f0842c96f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:25:18.338 INFO:teuthology.orchestra.run.smithi017.stderr:2023-11-02T19:25:18.332+0000 7f0842c96f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:25:18.369 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:18.369 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:18.370 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:18.370 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:18.370 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:18.370 INFO:teuthology.orchestra.run.smithi017.stdout: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:25:18.371 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:18.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.371 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:18.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.372 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:18.372 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:18.372 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:18.372 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:25:18.372 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:18.373 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 1, 2023-11-02T19:25:18.373 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:18.373 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:25:18.373 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:18.374 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:18.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.374 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:18.374 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:18.375 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:18.375 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:25:18.375 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:18.375 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:18.376 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:18.376 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:25:18.376 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:18.376 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:18.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.377 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:18.377 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:18.377 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:18.377 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698953111.156002_848.1", 2023-11-02T19:25:18.378 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:18.378 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:18.378 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:18.378 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:25:11.156002Z", 2023-11-02T19:25:18.378 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:18.379 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:18.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.379 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:18.380 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:18.380 INFO:teuthology.orchestra.run.smithi017.stdout: "state": 1, 2023-11-02T19:25:18.380 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:25:18.380 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:18.380 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:18.381 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:18.381 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:25:18.381 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2023-11-02T19:25:18.381 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:18.382 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:18.382 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:18.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.382 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:18.383 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 1, 2023-11-02T19:25:18.383 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 1 2023-11-02T19:25:18.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.383 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:18Z" 2023-11-02T19:25:18.383 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:18.384 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b3a94447-46df-4768-8340-cf63e4392a5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1698947386.719075_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-11-02T17:49:46.719075Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1698950762.156551_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.156551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1698953111.156002_848.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:25:11.156002Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1698950762.124040_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T18:46:02.124040Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-11-02T19:25:18Z" } 2023-11-02T19:25:18.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698953111.156002_848.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:25:11.156002Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:25:18.386 DEBUG:rgw_multi.tests:log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:18.386 DEBUG:rgw_multi.tests:sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:25:18.386 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-11-02T19:25:18.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:18.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:18.391 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:18.442 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:18.442 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:18.528 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-11-02T19:25:18.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.529 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:25:18.529 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:25:18.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.530 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052231", 2023-11-02T19:25:18.530 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:25:17.957254Z" 2023-11-02T19:25:18.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.531 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:25:18.531 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:25:18.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.532 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046358", 2023-11-02T19:25:18.532 INFO:teuthology.orchestra.run.smithi017.stdout: "last_update": "2023-11-02T19:25:17.991605Z" 2023-11-02T19:25:18.532 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:18.532 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-11-02T19:25:18.534 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:25:18.534 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:25:18.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:18.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:18.536 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:18.583 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:18.583 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:18.655 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:18.657 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:18.657 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:18.658 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:18.658 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:18.658 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:25:18.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.659 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:18.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.659 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:18.659 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:18.659 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:18.660 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:25:18.660 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:18.660 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:25:18.660 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:18.661 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:25:18.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:18.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.662 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:18.662 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:18.662 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:18.662 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:25:18.663 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:18.663 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:25:18.663 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:18.663 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:25:18.664 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:18.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.664 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:18.665 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:18.665 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:18.665 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:25:18.666 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:18.666 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:18.666 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:18.666 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:25:18.667 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:18.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:18.667 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:18.668 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:18.668 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:18.668 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:25:18.668 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:18.668 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:18.669 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:18.669 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:25:18.669 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:18.669 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:18.669 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:18.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.670 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:18.670 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:25:18.670 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:25:18.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:18.671 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:18Z" 2023-11-02T19:25:18.671 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:18.672 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:25:18Z" } 2023-11-02T19:25:18.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:25:18.673 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:25:18.674 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:25:18.674 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:25:23.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:23.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:23.677 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:23.726 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:23.727 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:23.810 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:25:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:25:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:25:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:25:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:25:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:25:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:25:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:25:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:25:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:25:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:25:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:25:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:25:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:23Z" 2023-11-02T19:25:23.824 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:23.825 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:25:23Z" } 2023-11-02T19:25:23.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:25:23.826 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:25:23.827 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:25:23.827 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:25:28.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:28.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:28.830 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:28.879 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:28.879 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:28.969 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:28.969 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:28.970 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:28.970 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:28.970 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:28.970 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:25:28.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:28.971 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:28.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:28.971 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:28.971 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:28.972 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:28.972 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:25:28.972 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:28.972 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:25:28.972 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:28.973 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:25:28.973 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:28.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:28.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:28.974 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:28.974 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:28.974 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:28.974 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:25:28.974 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:28.975 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:25:28.975 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:28.975 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:25:28.975 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:28.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:28.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:28.976 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:28.976 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:28.976 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:28.977 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:25:28.977 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:28.977 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:28.977 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:28.977 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:25:28.978 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:28.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:28.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:28.978 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:28.979 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:28.979 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:28.979 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:25:28.979 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:28.980 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:28.980 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:28.980 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:25:28.980 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:28.980 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:28.981 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:28.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:28.981 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:28.981 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:25:28.981 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:25:28.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:28.982 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:28Z" 2023-11-02T19:25:28.982 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:28.983 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:25:28Z" } 2023-11-02T19:25:28.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:25:28.984 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:25:28.985 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:25:28.985 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:25:33.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:33.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:33.988 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:34.039 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:34.039 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:34.129 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:34.129 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:34.130 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:34.130 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:34.130 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:34.130 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:25:34.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:34.131 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:34.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:34.131 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:34.132 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:34.132 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:34.132 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:25:34.132 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:34.133 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:25:34.133 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:34.133 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:25:34.133 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:34.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:34.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:34.134 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:34.134 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:34.134 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:34.135 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:25:34.135 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:34.135 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:25:34.135 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:34.136 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:25:34.136 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:34.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:34.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:34.137 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:34.137 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:34.137 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:34.137 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:25:34.138 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:34.138 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:34.138 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:34.138 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:25:34.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:34.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:34.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:34.139 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:34.140 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:34.140 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:34.140 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:25:34.140 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:34.140 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:34.141 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:34.141 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:25:34.141 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:34.141 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:34.141 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:34.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:34.142 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:34.142 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:25:34.142 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:25:34.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:34.143 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:34Z" 2023-11-02T19:25:34.143 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:34.144 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:25:34Z" } 2023-11-02T19:25:34.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:25:34.145 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:25:34.145 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:25:34.146 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:25:39.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:39.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:39.148 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:39.197 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:39.197 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:39.285 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:39.285 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:39.285 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:39.286 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:39.286 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:39.286 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:25:39.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:39.287 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:39.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:39.287 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:39.287 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:39.288 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:39.288 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:25:39.288 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:39.288 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:25:39.289 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:39.289 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:25:39.289 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:39.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:39.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:39.290 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:39.290 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:39.290 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:39.291 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:25:39.291 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:39.291 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:25:39.291 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:39.291 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:25:39.292 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:39.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:39.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:39.292 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:39.293 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:39.293 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:39.293 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:25:39.293 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:39.293 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:39.294 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:39.294 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:25:39.294 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:39.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:39.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:39.295 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:39.295 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:39.295 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:39.297 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:25:39.298 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:39.298 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:39.298 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:39.298 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:25:39.299 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:39.299 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:39.299 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:39.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:39.299 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:39.300 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:25:39.300 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:25:39.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:39.300 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:39Z" 2023-11-02T19:25:39.300 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:39.302 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:25:39Z" } 2023-11-02T19:25:39.302 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:25:39.303 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:25:39.303 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:25:39.303 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:25:44.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:44.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:44.306 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:44.439 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:44.439 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:44.440 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:44.440 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:44.440 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:44.440 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:25:44.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:44.441 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:44.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:44.441 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:44.442 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:44.442 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:44.442 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:25:44.442 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:44.443 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:25:44.443 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:44.443 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:25:44.443 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:44.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:44.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:44.444 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:44.444 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:44.444 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:44.444 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:25:44.445 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:44.445 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:25:44.445 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:44.445 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:25:44.446 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:44.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:44.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:44.446 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:44.446 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:44.447 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:44.447 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:25:44.447 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:44.447 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:44.448 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:44.448 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:25:44.448 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:44.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:44.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:44.449 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:44.449 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:44.449 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:44.449 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:25:44.449 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:44.450 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:44.450 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:44.450 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:25:44.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:44.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:44.451 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:44.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:44.451 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:44.452 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:25:44.452 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:25:44.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:44.452 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:44Z" 2023-11-02T19:25:44.453 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:44.454 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:25:44Z" } 2023-11-02T19:25:44.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:25:44.455 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:25:44.455 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:25:44.456 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:25:49.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:49.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:49.459 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:49.506 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:49.507 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:49.593 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:49.593 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:49.594 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:49.594 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:49.594 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:49.595 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:25:49.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:49.595 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:49.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:49.596 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:49.596 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:49.596 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:49.597 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:25:49.597 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:49.597 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:25:49.598 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:49.598 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:25:49.598 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:49.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:49.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:49.599 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:49.599 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:49.599 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:49.600 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:25:49.600 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:49.600 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:25:49.601 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:49.601 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:25:49.601 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:49.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:49.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:49.602 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:49.602 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:49.602 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:49.603 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:25:49.603 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:49.603 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:49.604 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:49.604 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:25:49.604 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:49.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:49.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:49.605 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:49.605 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:49.605 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:49.606 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:25:49.606 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:49.606 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:49.607 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:49.607 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:25:49.608 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:49.608 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:49.608 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:49.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:49.609 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:49.610 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:25:49.610 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:25:49.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:49.611 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:49Z" 2023-11-02T19:25:49.611 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:49.612 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:25:49Z" } 2023-11-02T19:25:49.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:25:49.613 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:25:49.614 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:25:49.614 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:25:54.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:54.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:54.617 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:54.666 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:54.756 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:54.757 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:54.757 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:54.757 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:54.758 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:54.758 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:25:54.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:54.758 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:54.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:54.759 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:54.759 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:54.759 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:54.759 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:25:54.760 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:54.760 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:25:54.760 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:54.760 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:25:54.761 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:54.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:54.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:54.761 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:54.761 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:54.762 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:54.762 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:25:54.762 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:54.762 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:25:54.763 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:54.763 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:25:54.763 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:54.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:54.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:54.764 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:54.764 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:54.764 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:54.764 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:25:54.764 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:54.765 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:54.765 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:54.765 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:25:54.765 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:54.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:54.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:54.766 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:54.766 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:54.767 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:54.767 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:25:54.767 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:54.767 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:54.767 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:54.767 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:25:54.768 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:54.768 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:54.768 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:54.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:54.769 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:54.769 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:25:54.769 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:25:54.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:54.769 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:54Z" 2023-11-02T19:25:54.770 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:54.771 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:25:54Z" } 2023-11-02T19:25:54.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:25:54.772 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:25:54.772 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:25:54.772 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:25:59.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:59.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:25:59.775 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:25:59.821 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:25:59.822 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:25:59.914 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:25:59.914 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:25:59.914 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:25:59.915 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:25:59.915 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:25:59.915 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:25:59.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:59.916 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:25:59.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:59.916 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:25:59.916 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:59.917 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:59.917 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:25:59.917 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:59.917 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:25:59.917 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:59.918 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:25:59.918 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:59.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:59.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:59.919 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:25:59.919 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:59.919 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:59.919 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:25:59.920 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:59.920 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:25:59.920 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:59.920 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:25:59.921 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:59.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:59.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:59.921 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:25:59.922 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:59.922 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:59.922 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:25:59.922 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:59.923 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:59.923 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:59.923 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:25:59.923 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:59.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:59.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:25:59.924 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:25:59.924 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:25:59.925 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:25:59.925 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:25:59.925 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:25:59.925 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:25:59.926 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:25:59.926 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:25:59.926 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:59.926 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:25:59.927 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:25:59.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:59.927 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:25:59.927 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:25:59.928 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:25:59.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:25:59.928 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:25:59Z" 2023-11-02T19:25:59.928 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:25:59.930 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:25:59Z" } 2023-11-02T19:25:59.930 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:25:59.931 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:25:59.931 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:25:59.932 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:04.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:04.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:04.934 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:04.982 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:26:04.983 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:26:05.070 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:26:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:26:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:26:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:26:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:26:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:26:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:26:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:26:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:26:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:05.074 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:05.074 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:05.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:05.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:05.075 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:26:05.075 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:05.075 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:05.075 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:05.075 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:05.076 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:26:05.076 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:05.076 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:05.076 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:05.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:05.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:05.077 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:26:05.078 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:05.078 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:05.078 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:05.078 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:05.079 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:05.079 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:05.079 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:05.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:05.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:05.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:05.080 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:26:05.080 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:05.081 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:05.081 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:05.081 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:05.081 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:05.081 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:05.082 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:05.082 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:05.082 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:05.082 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:26:05.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:05.083 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:26:05.083 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:26:05.084 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:26:05.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:05.085 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:26:05Z" 2023-11-02T19:26:05.085 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:26:05.086 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:26:05Z" } 2023-11-02T19:26:05.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:05.088 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:05.088 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:05.088 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:10.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:10.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:10.090 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:10.139 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:26:10.140 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:26:10.228 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:26:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:26:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:26:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:26:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:26:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:26:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:26:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:26:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:26:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:10.233 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:26:10.233 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:10.233 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:10.233 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:26:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:26:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:26:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:26:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:26:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:26:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:26:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:26:10Z" 2023-11-02T19:26:10.241 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:26:10.242 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:26:10Z" } 2023-11-02T19:26:10.242 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:10.243 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:10.243 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:10.244 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:15.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:15.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:15.247 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:15.294 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:26:15.295 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:26:15.389 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:26:15.389 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:26:15.390 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:26:15.390 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:26:15.390 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:26:15.390 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:26:15.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:15.391 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:26:15.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:15.391 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:26:15.392 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:15.392 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:15.392 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:15.392 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:15.392 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:26:15.393 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:15.393 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:15.393 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:15.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:15.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:15.394 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:26:15.394 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:15.394 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:15.394 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:15.395 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:15.395 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:26:15.395 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:15.395 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:15.396 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:15.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:15.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:15.396 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:26:15.396 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:15.397 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:15.397 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:15.397 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:15.397 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:15.398 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:15.398 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:15.398 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:15.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:15.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:15.399 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:26:15.399 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:15.399 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:15.399 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:15.399 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:15.400 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:15.400 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:15.400 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:15.400 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:15.401 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:15.401 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:26:15.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:15.401 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:26:15.401 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:26:15.402 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:26:15.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:15.402 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:26:15Z" 2023-11-02T19:26:15.402 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:26:15.403 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:26:15Z" } 2023-11-02T19:26:15.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:15.404 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:15.405 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:15.405 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:20.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:20.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:20.407 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:20.454 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:26:20.455 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:26:20.539 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:26:20.539 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:26:20.540 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:26:20.540 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:26:20.541 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:26:20.541 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:26:20.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:20.542 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:26:20.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:20.543 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:26:20.543 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:20.544 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:20.544 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:20.545 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:20.545 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:26:20.545 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:26:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:26:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:26:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:26:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:26:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:26:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:26:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:26:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:26:20Z" 2023-11-02T19:26:20.557 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:26:20.558 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:26:20Z" } 2023-11-02T19:26:20.559 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:20.560 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:20.561 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:20.561 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:25.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:25.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:25.564 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:25.613 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:26:25.614 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:26:25.703 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:26:25.704 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:26:25.704 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:26:25.704 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:26:25.704 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:26:25.705 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:26:25.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:25.705 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:26:25.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:25.706 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:26:25.706 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:25.706 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:25.706 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:25.706 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:25.707 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:26:25.707 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:25.707 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:25.707 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:25.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:25.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:25.708 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:26:25.708 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:25.708 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:25.709 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:25.709 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:25.709 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:26:25.709 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:25.709 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:25.710 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:25.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:25.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:25.710 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:26:25.710 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:25.711 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:25.711 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:25.711 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:25.711 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:25.712 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:25.712 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:25.712 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:25.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:25.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:25.713 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:26:25.713 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:25.713 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:25.713 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:25.714 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:25.714 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:25.714 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:25.714 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:25.714 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:25.715 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:25.715 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:26:25.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:25.715 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:26:25.715 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:26:25.716 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:26:25.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:25.716 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:26:25Z" 2023-11-02T19:26:25.716 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:26:25.717 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:26:25Z" } 2023-11-02T19:26:25.718 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:25.718 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:25.719 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:25.719 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:30.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:30.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:30.721 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:30.772 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:26:30.772 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:26:30.856 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:26:30.856 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:26:30.856 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:26:30.856 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:26:30.856 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:26:30.857 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:26:30.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:30.857 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:26:30.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:30.857 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:26:30.858 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:30.858 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:30.858 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:30.858 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:30.859 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:26:30.859 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:30.859 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:30.859 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:30.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:30.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:30.860 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:26:30.860 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:30.860 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:30.861 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:30.861 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:30.861 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:26:30.861 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:30.861 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:30.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:30.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:30.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:30.862 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:26:30.862 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:30.863 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:30.863 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:30.863 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:30.863 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:30.863 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:30.864 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:30.864 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:30.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:30.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:30.864 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:26:30.865 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:30.865 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:30.865 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:30.865 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:30.865 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:30.866 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:30.866 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:30.866 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:30.866 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:30.866 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:26:30.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:30.867 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:26:30.867 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:26:30.867 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:26:30.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:30.868 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:26:30Z" 2023-11-02T19:26:30.868 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:26:30.869 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:26:30Z" } 2023-11-02T19:26:30.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:30.870 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:30.870 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:30.871 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:35.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:35.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:35.872 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:26:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:26:36.005 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:26:36.006 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:26:36.006 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:26:36.006 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:26:36.007 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:26:36.007 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:26:36.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:36.007 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:26:36.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:36.008 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:26:36.008 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:36.008 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:36.008 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:36.009 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:36.009 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:26:36.009 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:36.009 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:36.010 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:36.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:36.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:36.010 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:26:36.010 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:36.011 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:36.011 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:36.011 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:36.011 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:26:36.012 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:36.012 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:36.012 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:36.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:36.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:36.013 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:26:36.013 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:36.013 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:36.013 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:36.013 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:36.014 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:36.014 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:36.014 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:36.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:36.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:36.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:36.015 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:26:36.015 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:36.015 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:36.016 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:36.016 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:36.016 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:36.016 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:36.016 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:36.017 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:36.017 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:36.017 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:26:36.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:36.018 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:26:36.018 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:26:36.018 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:26:36.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:36.019 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:26:35Z" 2023-11-02T19:26:36.019 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:26:36.020 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:26:35Z" } 2023-11-02T19:26:36.021 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:36.021 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:36.022 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:36.022 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:41.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:41.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:41.024 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:41.073 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:26:41.073 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:26:41.163 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:26:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:26:41.164 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:26:41.164 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:26:41.164 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:26:41.164 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:26:41.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:41.165 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:26:41.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:41.165 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:26:41.166 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:41.166 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:41.166 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:41.166 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:41.167 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:26:41.167 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:41.167 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:41.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:41.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:41.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:41.168 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:26:41.168 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:41.169 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:41.169 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:41.169 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:41.169 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:26:41.170 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:41.170 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:41.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:41.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:41.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:41.171 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:26:41.171 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:41.171 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:26:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:26:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:26:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:26:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:26:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:26:41Z" 2023-11-02T19:26:41.177 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:26:41.179 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:26:41Z" } 2023-11-02T19:26:41.179 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:41.180 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:41.181 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:41.181 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:46.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:46.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:46.184 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:46.234 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:26:46.235 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:26:46.326 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:26:46.326 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:26:46.327 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:26:46.327 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:26:46.327 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:26:46.327 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:26:46.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:46.328 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:26:46.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:46.328 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:26:46.329 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:46.329 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:46.329 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:46.329 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:46.330 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:26:46.330 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:46.330 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:46.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:46.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:46.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:46.331 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:26:46.331 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:46.331 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:46.332 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:46.332 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:46.332 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:26:46.333 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:46.333 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:46.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:46.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:46.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:46.334 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:26:46.334 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:46.334 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:46.335 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:46.335 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:46.335 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:46.335 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:46.336 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:46.336 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:46.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:46.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:46.337 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:26:46.337 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:46.337 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:46.337 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:46.338 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:46.338 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:46.338 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:46.338 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:46.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:46.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:46.339 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:26:46.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:46.340 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:26:46.340 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:26:46.340 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:26:46.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:46.341 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:26:46Z" 2023-11-02T19:26:46.341 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:26:46.342 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:24:33.140415Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:24:33.112241Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:26:46Z" } 2023-11-02T19:26:46.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:46.343 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:46.344 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:46.344 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:51.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:51.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:51.347 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:51.395 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2023-11-02T19:26:51.396 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2023-11-02T19:26:51.491 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-11-02T19:26:51.491 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": { 2023-11-02T19:26:51.491 INFO:teuthology.orchestra.run.smithi017.stdout: "info": { 2023-11-02T19:26:51.491 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "sync", 2023-11-02T19:26:51.492 INFO:teuthology.orchestra.run.smithi017.stdout: "num_shards": 4, 2023-11-02T19:26:51.492 INFO:teuthology.orchestra.run.smithi017.stdout: "instance_id": 1320917798396668347 2023-11-02T19:26:51.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:51.492 INFO:teuthology.orchestra.run.smithi017.stdout: "markers": [ 2023-11-02T19:26:51.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:51.493 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 0, 2023-11-02T19:26:51.493 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:51.493 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:51.494 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:51.494 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:51.494 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 9, 2023-11-02T19:26:51.494 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:51.495 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:51.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:51.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:51.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:51.495 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 1, 2023-11-02T19:26:51.496 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:51.496 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:51.496 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000052231", 2023-11-02T19:26:51.496 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:51.497 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 55, 2023-11-02T19:26:51.497 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:51.497 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:25:17.957254Z" 2023-11-02T19:26:51.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:51.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:51.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:51.499 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 2, 2023-11-02T19:26:51.499 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:51.499 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:51.500 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:51.500 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:51.500 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:51.501 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:51.501 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:51.501 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:51.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:51.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-11-02T19:26:51.502 INFO:teuthology.orchestra.run.smithi017.stdout: "key": 3, 2023-11-02T19:26:51.502 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2023-11-02T19:26:51.502 INFO:teuthology.orchestra.run.smithi017.stdout: "status": "incremental-sync", 2023-11-02T19:26:51.503 INFO:teuthology.orchestra.run.smithi017.stdout: "marker": "00000000000000000000:00000000000000046358", 2023-11-02T19:26:51.503 INFO:teuthology.orchestra.run.smithi017.stdout: "next_step_marker": "", 2023-11-02T19:26:51.503 INFO:teuthology.orchestra.run.smithi017.stdout: "total_entries": 0, 2023-11-02T19:26:51.503 INFO:teuthology.orchestra.run.smithi017.stdout: "pos": 0, 2023-11-02T19:26:51.504 INFO:teuthology.orchestra.run.smithi017.stdout: "timestamp": "2023-11-02T19:25:17.991605Z" 2023-11-02T19:26:51.504 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:51.504 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-11-02T19:26:51.504 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-11-02T19:26:51.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:51.505 INFO:teuthology.orchestra.run.smithi017.stdout: "full_sync": { 2023-11-02T19:26:51.505 INFO:teuthology.orchestra.run.smithi017.stdout: "total": 64, 2023-11-02T19:26:51.505 INFO:teuthology.orchestra.run.smithi017.stdout: "complete": 64 2023-11-02T19:26:51.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2023-11-02T19:26:51.505 INFO:teuthology.orchestra.run.smithi017.stdout: "current_time": "2023-11-02T19:26:51Z" 2023-11-02T19:26:51.506 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-11-02T19:26:51.507 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1320917798396668347 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-11-02T19:24:33.203114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052231", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-11-02T19:25:17.957254Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044887", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:19:53.938682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046358", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-11-02T19:25:17.991605Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-11-02T19:26:51Z" } 2023-11-02T19:26:51.507 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052231', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:25:17.957254Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:25:17.991605Z'}}] 2023-11-02T19:26:51.508 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:51.508 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:51.508 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:26:51.509 DEBUG:boto:path=/ibtljn-74/ 2023-11-02T19:26:51.509 DEBUG:boto:auth_path=/ibtljn-74/ 2023-11-02T19:26:51.510 DEBUG:boto:Method: HEAD 2023-11-02T19:26:51.510 DEBUG:boto:Path: /ibtljn-74/ 2023-11-02T19:26:51.511 DEBUG:boto:Data: 2023-11-02T19:26:51.511 DEBUG:boto:Headers: {} 2023-11-02T19:26:51.512 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.512 DEBUG:boto:Port: 8001 2023-11-02T19:26:51.512 DEBUG:boto:Params: {} 2023-11-02T19:26:51.513 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:26:51.514 DEBUG:boto:Token: None 2023-11-02T19:26:51.514 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:26:51 GMT /ibtljn-74/ 2023-11-02T19:26:51.515 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:nejfX5PcSy7FKAZOMEtLJ4M39Fk= 2023-11-02T19:26:51.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:26:51 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nejfX5PcSy7FKAZOMEtLJ4M39Fk=', 'Content-Length': '0'} 2023-11-02T19:26:51.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c3f9c53f762ed60-006543f7fb-2da9-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.526 DEBUG:boto:path=/ibtljn-74/obj1 2023-11-02T19:26:51.526 DEBUG:boto:auth_path=/ibtljn-74/obj1 2023-11-02T19:26:51.526 DEBUG:boto:Method: HEAD 2023-11-02T19:26:51.527 DEBUG:boto:Path: /ibtljn-74/obj1 2023-11-02T19:26:51.527 DEBUG:boto:Data: 2023-11-02T19:26:51.528 DEBUG:boto:Headers: {} 2023-11-02T19:26:51.528 DEBUG:boto:Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.529 DEBUG:boto:Port: 8001 2023-11-02T19:26:51.529 DEBUG:boto:Params: {} 2023-11-02T19:26:51.529 DEBUG:boto:Token: None 2023-11-02T19:26:51.530 DEBUG:boto:StringToSign: HEAD Thu, 02 Nov 2023 19:26:51 GMT /ibtljn-74/obj1 2023-11-02T19:26:51.530 DEBUG:boto:Signature: AWS 7LD6AK3P9XRGER738BKH:LNW8tdA1ZAJsEBnnECNZNLDAo+g= 2023-11-02T19:26:51.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:26:51 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LNW8tdA1ZAJsEBnnECNZNLDAo+g=', 'Content-Length': '0'} 2023-11-02T19:26:51.533 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006425f18edd956da8-006543f7fb-2da9-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.536 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... FAIL 2023-11-02T19:26:51.536 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.537 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-11-02T19:26:51.537 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2023-11-02T19:26:51.537 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-11-02T19:26:51.540 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-11-02T19:26:51.540 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-11-02T19:26:51.540 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-11-02T19:26:51.541 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/../src/test/rgw/rgw_multi/tests.py", line 1418, in test_bucket_reshard_index_log_trim 2023-11-02T19:26:51.541 INFO:tasks.rgw_multisite_tests: assert(len(json_obj_5['layout']['logs']) == 1) 2023-11-02T19:26:51.541 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-11-02T19:26:51.542 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-11-02T19:26:51.542 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=ibtljn-55 2023-11-02T19:26:51.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.546 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:51.546 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.547 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.547 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:36 GMT 2023-11-02T19:26:51.547 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.548 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KWQJTIzst26NURitiqXZOpgkHyY= 2023-11-02T19:26:51.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KWQJTIzst26NURitiqXZOpgkHyY=', 'Content-Length': '0'} 2023-11-02T19:26:51.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000010dfd14f68ce8270-006543e8b0-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.552 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.553 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.553 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.553 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:36 GMT 2023-11-02T19:26:51.554 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.554 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ= 2023-11-02T19:26:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ=', 'Content-Length': '0'} 2023-11-02T19:26:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1bafb3ccea03e40-006543e8b0-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:51.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:51.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:51.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.561 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:51.561 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:51.561 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:51.562 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:36 GMT 2023-11-02T19:26:51.562 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:51.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.563 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:1QItATWRWZvkb1hG0ibo43SHNIg= 2023-11-02T19:26:51.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:1QItATWRWZvkb1hG0ibo43SHNIg='} 2023-11-02T19:26:51.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b1f432976685606-006543e8b0-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:51.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.568 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.569 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.569 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.570 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:36 GMT 2023-11-02T19:26:51.570 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.571 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ= 2023-11-02T19:26:51.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ=', 'Content-Length': '0'} 2023-11-02T19:26:51.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093a04cba339c0061-006543e8b0-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:51.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:51.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:51.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:51.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:51.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.575 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:51.576 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:51.576 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:51.576 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:36 GMT 2023-11-02T19:26:51.577 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:51.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.577 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:bl/F+FQVilDek52c8cL/fv2gxf0= 2023-11-02T19:26:51.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bl/F+FQVilDek52c8cL/fv2gxf0='} 2023-11-02T19:26:51.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc15342477d33249-006543e8b0-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:51.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.582 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.582 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.583 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.583 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:36 GMT 2023-11-02T19:26:51.583 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.584 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ= 2023-11-02T19:26:51.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ=', 'Content-Length': '0'} 2023-11-02T19:26:51.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000043a39e20473a750-006543e8b0-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:51.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:51.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:51.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:51.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:51.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.590 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:51.590 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:51.591 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:51.591 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:36 GMT 2023-11-02T19:26:51.591 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:51.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.592 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:L97vg1G1C+q1aMO+gDPeoDs5M9Q= 2023-11-02T19:26:51.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:L97vg1G1C+q1aMO+gDPeoDs5M9Q='} 2023-11-02T19:26:51.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7049e1fc442de2b-006543e8b0-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:51.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.596 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.597 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.597 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.597 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:36 GMT 2023-11-02T19:26:51.598 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.598 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ= 2023-11-02T19:26:51.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CLb5GMY4dtJny52LX1EqhgQswnQ=', 'Content-Length': '0'} 2023-11-02T19:26:51.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d22d035ec9bffe89-006543e8b0-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:51.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:51.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:51.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:51.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:51.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.603 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:51.603 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:51.604 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:51.604 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:36 GMT 2023-11-02T19:26:51.605 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:51.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.606 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:dIWtvkavS8dtkiNvrYJyEW4D/ws= 2023-11-02T19:26:51.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:36 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dIWtvkavS8dtkiNvrYJyEW4D/ws='} 2023-11-02T19:26:51.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b126af4600eb7a7-006543e8b0-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:51.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:26:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: [ 2023-11-02T19:26:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:26:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "", 2023-11-02T19:26:51.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "0.000000" 2023-11-02T19:26:51.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:51.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:51.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698949296.521130_419.1", 2023-11-02T19:26:51.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T18:21:36.521130Z" 2023-11-02T19:26:51.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:51.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:51.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947722.231870_98.1", 2023-11-02T19:26:51.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T17:55:22.231870Z" 2023-11-02T19:26:51.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:51.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:51.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949296.521130_419.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T19:26:51.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-11-02T19:26:51.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:51.616 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:51.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:51.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:51.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:51.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:21:36.946+0000 7f79c7c90f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:51.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:21:36.950+0000 7f79c7c90f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:51.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:51.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:51.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:51.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:51.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:51.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:51.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:51.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 1, 2023-11-02T19:26:51.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:51.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:51.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:51.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:51.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:51.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:51.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:51.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:51.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:51.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "", 2023-11-02T19:26:51.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:51.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:51.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:51.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "0.000000", 2023-11-02T19:26:51.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:51.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:51.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698949296.521130_419.1", 2023-11-02T19:26:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:51.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:21:36.521130Z", 2023-11-02T19:26:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:51.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:51.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:51.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:51.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:51.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:51.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:51.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947722.231870_98.1", 2023-11-02T19:26:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:51.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T19:26:51.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:51.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:51.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:51.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:51.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 1, 2023-11-02T19:26:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 1 2023-11-02T19:26:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T18:21:36Z" 2023-11-02T19:26:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:51.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-11-02T19:26:51.639 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:51.639 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:51.640 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:51.640 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:51.641 INFO:tasks.rgw_multisite_tests: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:51.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:51.642 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:51.642 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:51.643 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:51.643 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:51.643 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:51.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:51.644 INFO:tasks.rgw_multisite_tests: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:51.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:51.645 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-11-02T19:26:51.645 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:51.645 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:51.645 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:51.646 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:51.646 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:51.646 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:51.647 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:51.647 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:51.647 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:51.648 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-11-02T19:26:51.648 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:51.648 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:51.649 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:51.649 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-11-02T19:26:51.649 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:51.650 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:51.650 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:51.650 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:51.651 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:51.651 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:51.651 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:51.652 INFO:tasks.rgw_multisite_tests: "marker": "1_1698949296.521130_419.1", 2023-11-02T19:26:51.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:51.652 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:51.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:51.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:21:36.521130Z", 2023-11-02T19:26:51.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:51.653 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:51.654 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:51.654 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:51.654 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:51.655 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:51.655 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:51.655 INFO:tasks.rgw_multisite_tests: "marker": "1_1698947722.231870_98.1", 2023-11-02T19:26:51.656 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:51.656 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:51.656 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:51.657 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T17:55:22.231870Z", 2023-11-02T19:26:51.657 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:51.657 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:51.658 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:51.658 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:51.659 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:51.659 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:51.660 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-11-02T19:26:51.660 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-11-02T19:26:51.660 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:51.661 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T18:21:36Z" 2023-11-02T19:26:51.662 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:51.662 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698949296.521130_419.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:21:36.521130Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698947722.231870_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T17:55:22.231870Z', 'realm_epoch': 2}}] 2023-11-02T19:26:51.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949296.521130_419.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T19:26:51.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1698947386.719075_41.1', 1: '', 2: '1_1698949296.521130_419.1', 3: '1_1698947722.231870_98.1'} 2023-11-02T19:26:51.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-11-02T19:26:51.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=ibtljn-55 2023-11-02T19:26:51.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:51.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:51.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T19:26:51.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:51.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:51.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:21:37.114+0000 7f8931096f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:51.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:21:42.113+0000 7f8931096f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:51.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:21:47.117+0000 7f8931096f40 1 bucket sync caught up with source: 2023-11-02T19:26:51.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, 00000000001.24.6, , , , 00000000001.66.6, , , 00000000001.64.6, , 00000000001.29.6] 2023-11-02T19:26:51.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [, 00000000001.24.6, , , , 00000000001.66.6, , , 00000000001.64.6, , 00000000001.29.6] 2023-11-02T19:26:51.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:21:47.117+0000 7f8931096f40 0 bucket checkpoint complete 2023-11-02T19:26:51.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=ibtljn-55 2023-11-02T19:26:51.668 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:51.668 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T19:26:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:21:47.273+0000 7fdf2619cf40 1 waiting to reach incremental sync.. 2023-11-02T19:26:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:21:52.273+0000 7fdf2619cf40 1 waiting to reach incremental sync.. 2023-11-02T19:26:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:21:57.277+0000 7fdf2619cf40 1 bucket sync caught up with source: 2023-11-02T19:26:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, 00000000001.174.6, , , , 00000000001.191.6, , , 00000000001.192.6, , 00000000001.177.6] 2023-11-02T19:26:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [, 00000000001.174.6, , , , 00000000001.191.6, , , 00000000001.192.6, , 00000000001.177.6] 2023-11-02T19:26:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:21:57.277+0000 7fdf2619cf40 0 bucket checkpoint complete 2023-11-02T19:26:51.671 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=ibtljn-55 zones={a2, a1} 2023-11-02T19:26:51.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:51.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.675 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.676 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.676 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.676 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.677 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.677 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:W/QoP5vu5m/Qh/K7By+5WmKLIeQ= 2023-11-02T19:26:51.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W/QoP5vu5m/Qh/K7By+5WmKLIeQ=', 'Content-Length': '0'} 2023-11-02T19:26:51.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b283ad7bfbd95e24-006543e8c5-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:51.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.682 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.682 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.682 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.683 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.683 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.684 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:W/QoP5vu5m/Qh/K7By+5WmKLIeQ= 2023-11-02T19:26:51.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:W/QoP5vu5m/Qh/K7By+5WmKLIeQ=', 'Content-Length': '0'} 2023-11-02T19:26:51.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6040f3ff0fd9483-006543e8c5-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.684 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-11-02T19:26:51.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:51.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:51.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:51.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.689 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.689 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.690 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.690 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.690 INFO:tasks.rgw_multisite_tests:/ibtljn-55/?versions 2023-11-02T19:26:51.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.691 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ= 2023-11-02T19:26:51.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ=', 'Content-Length': '0'} 2023-11-02T19:26:51.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009da9aca866688f78-006543e8c5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.692 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-11-02T19:26:51.692 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-11-02T19:26:51.693 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-11-02T19:26:51.693 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-11-02T19:26:51.693 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-11-02T19:26:51.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:51.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.698 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.698 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.698 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.699 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.699 INFO:tasks.rgw_multisite_tests:/ibtljn-55/?versions 2023-11-02T19:26:51.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.700 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ= 2023-11-02T19:26:51.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ=', 'Content-Length': '0'} 2023-11-02T19:26:51.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1cec51f94a6ccc4-006543e8c5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.701 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-11-02T19:26:51.701 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-11-02T19:26:51.701 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-11-02T19:26:51.702 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-11-02T19:26:51.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:51.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:51.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:51.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.706 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.706 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.707 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.707 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.707 INFO:tasks.rgw_multisite_tests:/ibtljn-55/?versions 2023-11-02T19:26:51.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.708 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ= 2023-11-02T19:26:51.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ=', 'Content-Length': '0'} 2023-11-02T19:26:51.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003802765ff34a147f-006543e8c5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:51.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:51.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:51.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.715 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.715 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.715 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.716 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.716 INFO:tasks.rgw_multisite_tests:/ibtljn-55/?versions 2023-11-02T19:26:51.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.717 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ= 2023-11-02T19:26:51.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SzUgV0NSC/KJirLUzK4ulfJfhLQ=', 'Content-Length': '0'} 2023-11-02T19:26:51.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003685fddc53a3f276-006543e8c5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.718 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-11-02T19:26:51.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:51.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:51.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:51.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.722 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.723 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.723 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.723 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.724 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.724 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c= 2023-11-02T19:26:51.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c=', 'Content-Length': '0'} 2023-11-02T19:26:51.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6fcc98f203a2f67-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:51.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:51.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:51.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.731 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.731 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.731 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.732 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.732 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.733 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c= 2023-11-02T19:26:51.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c=', 'Content-Length': '0'} 2023-11-02T19:26:51.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad2439f91625ebc3-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:51.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:51.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:51.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.738 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.738 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.739 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.739 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.739 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.740 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Wg99zl2nwWpDdhaQWcBGN4dXL3o= 2023-11-02T19:26:51.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Wg99zl2nwWpDdhaQWcBGN4dXL3o=', 'Content-Length': '0'} 2023-11-02T19:26:51.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f27d75bd4b87cc25-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:51.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:51.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:51.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.746 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.746 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.747 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.747 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.747 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.748 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Wg99zl2nwWpDdhaQWcBGN4dXL3o= 2023-11-02T19:26:51.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Wg99zl2nwWpDdhaQWcBGN4dXL3o=', 'Content-Length': '0'} 2023-11-02T19:26:51.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011de2edbdc347392-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.749 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-11-02T19:26:51.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:51.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:51.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:51.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.753 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.754 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.754 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.754 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.755 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.755 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c= 2023-11-02T19:26:51.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c=', 'Content-Length': '0'} 2023-11-02T19:26:51.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007467a1312d726ace-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.760 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.761 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.761 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.762 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.762 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:51.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.765 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c= 2023-11-02T19:26:51.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RhE3m09epoG7nSqlpaU8Bhqqj9c=', 'Content-Length': '0'} 2023-11-02T19:26:51.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ce636c2e8925f21-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=a 2023-11-02T19:26:51.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=a 2023-11-02T19:26:51.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=a 2023-11-02T19:26:51.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.770 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.771 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.771 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.771 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.772 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.772 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T19:26:51.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T19:26:51.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cdd8172d145f58f-006543e8c5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=a 2023-11-02T19:26:51.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=a 2023-11-02T19:26:51.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=a 2023-11-02T19:26:51.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.778 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.779 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.779 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.780 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.780 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.781 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T19:26:51.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T19:26:51.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003a80e7f0dff731c-006543e8c5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:51.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:51.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:51.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.786 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.786 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.786 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.787 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.787 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:51.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.788 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:9y0zeuFgq1MDsb1o4x5CNo1im60= 2023-11-02T19:26:51.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9y0zeuFgq1MDsb1o4x5CNo1im60=', 'Content-Length': '0'} 2023-11-02T19:26:51.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f81135958368ce4d-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:51.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:51.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:51.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.792 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.792 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.793 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.793 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.793 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:51.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.794 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:9y0zeuFgq1MDsb1o4x5CNo1im60= 2023-11-02T19:26:51.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9y0zeuFgq1MDsb1o4x5CNo1im60=', 'Content-Length': '0'} 2023-11-02T19:26:51.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d66b97c909dfb89-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-11-02T19:26:51.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:51.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:51.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:51.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.799 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.799 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.799 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.800 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.800 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:51.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.801 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:h5x1IbW/feyb8fxMOP1cDOk6pU0= 2023-11-02T19:26:51.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h5x1IbW/feyb8fxMOP1cDOk6pU0=', 'Content-Length': '0'} 2023-11-02T19:26:51.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029ad2ce82a0a611b-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:51.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:51.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:51.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.805 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.805 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.806 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.806 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.806 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:51.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.807 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:h5x1IbW/feyb8fxMOP1cDOk6pU0= 2023-11-02T19:26:51.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h5x1IbW/feyb8fxMOP1cDOk6pU0=', 'Content-Length': '0'} 2023-11-02T19:26:51.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd73cc3645153ec-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=a 2023-11-02T19:26:51.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=a 2023-11-02T19:26:51.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=a 2023-11-02T19:26:51.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.812 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.812 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.813 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.813 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.813 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.814 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T19:26:51.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T19:26:51.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033bbbab684d0149d-006543e8c5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=a 2023-11-02T19:26:51.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=a 2023-11-02T19:26:51.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=a 2023-11-02T19:26:51.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.819 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.820 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.820 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.820 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.821 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.821 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T19:26:51.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T19:26:51.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e7217578f02e40-006543e8c5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.822 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-11-02T19:26:51.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:51.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:51.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.827 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.827 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.827 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.828 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.828 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.829 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o= 2023-11-02T19:26:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o=', 'Content-Length': '0'} 2023-11-02T19:26:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c03952e8e602465d-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:51.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.834 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.834 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.834 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.835 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.835 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.835 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o= 2023-11-02T19:26:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o=', 'Content-Length': '0'} 2023-11-02T19:26:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032260631f42a1730-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:51.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:51.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:51.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.841 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.841 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.841 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.841 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.842 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.842 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:ytciB+kcP/pVcP/BeTULzcqkakQ= 2023-11-02T19:26:51.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ytciB+kcP/pVcP/BeTULzcqkakQ=', 'Content-Length': '0'} 2023-11-02T19:26:51.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0ae4e70e4899425-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:51.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:51.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:51.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.848 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.849 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.849 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.850 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.850 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.851 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:ytciB+kcP/pVcP/BeTULzcqkakQ= 2023-11-02T19:26:51.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:ytciB+kcP/pVcP/BeTULzcqkakQ=', 'Content-Length': '0'} 2023-11-02T19:26:51.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e0224a3055ec304-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.852 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-11-02T19:26:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:51.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:51.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:51.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.856 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.857 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.857 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.857 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.858 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.858 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o= 2023-11-02T19:26:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o=', 'Content-Length': '0'} 2023-11-02T19:26:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003af9adce501da8fb-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:51.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:51.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.863 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.863 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.864 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.864 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.864 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:51.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.865 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o= 2023-11-02T19:26:51.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:B9OMLuN04bZ3Oo4aUYTISuZf+3o=', 'Content-Length': '0'} 2023-11-02T19:26:51.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cd7322874e1a5a0-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=b 2023-11-02T19:26:51.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=b 2023-11-02T19:26:51.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=b 2023-11-02T19:26:51.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.871 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.872 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.872 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.872 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.873 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.873 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T19:26:51.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T19:26:51.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000145314a7e396470f-006543e8c5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=b 2023-11-02T19:26:51.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=b 2023-11-02T19:26:51.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=b 2023-11-02T19:26:51.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.879 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.880 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.880 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.881 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.881 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.882 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T19:26:51.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T19:26:51.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000075002ef0ea16f3c-006543e8c5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:51.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:51.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:51.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.887 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.888 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.888 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.889 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.889 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:51.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.890 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:9XeMFcPloEhJFcWD0fvX393mT2w= 2023-11-02T19:26:51.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9XeMFcPloEhJFcWD0fvX393mT2w=', 'Content-Length': '0'} 2023-11-02T19:26:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7cfdeec812c3f98-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:51.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:51.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.895 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.895 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.895 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.896 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.896 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.897 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:9XeMFcPloEhJFcWD0fvX393mT2w= 2023-11-02T19:26:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9XeMFcPloEhJFcWD0fvX393mT2w=', 'Content-Length': '0'} 2023-11-02T19:26:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea0a5efc759e6cb0-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.897 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-11-02T19:26:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:51.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:51.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.901 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.902 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.902 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.902 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.903 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:51.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.903 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:uzfMXCHJklvTufQCZliYSSqD+Ao= 2023-11-02T19:26:51.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uzfMXCHJklvTufQCZliYSSqD+Ao=', 'Content-Length': '0'} 2023-11-02T19:26:51.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e160489ceb00cc0c-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:51.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:51.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:51.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.909 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.909 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.910 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.910 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.910 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.911 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:uzfMXCHJklvTufQCZliYSSqD+Ao= 2023-11-02T19:26:51.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:uzfMXCHJklvTufQCZliYSSqD+Ao=', 'Content-Length': '0'} 2023-11-02T19:26:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000400fc79297b2bbb-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=b 2023-11-02T19:26:51.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=b 2023-11-02T19:26:51.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=b 2023-11-02T19:26:51.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.916 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.916 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.917 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.917 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.917 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.918 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T19:26:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T19:26:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d4a6998ab8c4155-006543e8c5-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=b 2023-11-02T19:26:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=b 2023-11-02T19:26:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=b 2023-11-02T19:26:51.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.924 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.924 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.924 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.925 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.925 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.926 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk= 2023-11-02T19:26:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jdBaC7NTRKnYR6G59cYo1opT2Kk=', 'Content-Length': '0'} 2023-11-02T19:26:51.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd438608631e7afe-006543e8c5-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.927 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-11-02T19:26:51.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.931 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.932 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.932 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.932 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.933 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.933 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:gDEJYpwC9/itU80hbyQpG80smEA= 2023-11-02T19:26:51.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gDEJYpwC9/itU80hbyQpG80smEA=', 'Content-Length': '0'} 2023-11-02T19:26:51.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018366b08d880e616-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:51.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.938 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.939 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.939 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.939 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.940 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.940 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:gDEJYpwC9/itU80hbyQpG80smEA= 2023-11-02T19:26:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gDEJYpwC9/itU80hbyQpG80smEA=', 'Content-Length': '0'} 2023-11-02T19:26:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b57881d11f4640be-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:51.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.946 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.946 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.946 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.947 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:57 GMT 2023-11-02T19:26:51.947 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.948 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:myD3z9bFnmGf6OdgjKOAvqQL7sY= 2023-11-02T19:26:51.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:57 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:myD3z9bFnmGf6OdgjKOAvqQL7sY=', 'Content-Length': '0'} 2023-11-02T19:26:51.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000895508fbb62fa39b-006543e8c5-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:51.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:51.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.953 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.953 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.953 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.954 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:51.954 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.955 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Lrme/QddLhJqTjUGwd0F8VVNAo0= 2023-11-02T19:26:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Lrme/QddLhJqTjUGwd0F8VVNAo0=', 'Content-Length': '0'} 2023-11-02T19:26:51.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ac829582b05a0cc-006543e8c5-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.956 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-11-02T19:26:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:51.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:51.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:51.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.960 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.961 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.961 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.961 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:51.962 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.962 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:gb27Q640zqifRsOcMCkQJgObeMw= 2023-11-02T19:26:51.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gb27Q640zqifRsOcMCkQJgObeMw=', 'Content-Length': '0'} 2023-11-02T19:26:51.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1258ea0ccdb14f8-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:51.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.969 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.969 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.970 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.970 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:51.971 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:51.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.971 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:gb27Q640zqifRsOcMCkQJgObeMw= 2023-11-02T19:26:51.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:gb27Q640zqifRsOcMCkQJgObeMw=', 'Content-Length': '0'} 2023-11-02T19:26:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8f5caf81ded54c7-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=c 2023-11-02T19:26:51.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=c 2023-11-02T19:26:51.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=c 2023-11-02T19:26:51.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.976 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.977 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.977 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.977 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:51.977 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.978 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T19:26:51.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T19:26:51.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c9b4b4b9c6dc217-006543e8c6-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=c 2023-11-02T19:26:51.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=c 2023-11-02T19:26:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=c 2023-11-02T19:26:51.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.984 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:51.984 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.984 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.985 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:51.985 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:51.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.985 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T19:26:51.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T19:26:51.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c3ea9139aa32ef0-006543e8c6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:51.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:51.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:51.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:51.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:51.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:51.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.990 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.991 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.991 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.991 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:51.992 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:51.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.992 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Osypdm7cOvEv2stQd2/LWJsxs8U= 2023-11-02T19:26:51.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Osypdm7cOvEv2stQd2/LWJsxs8U=', 'Content-Length': '0'} 2023-11-02T19:26:51.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d6c5f91d63db2cf-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:51.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:51.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:51.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:51.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:51.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:51.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:51.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:51.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:51.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:51.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:51.997 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:51.997 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.997 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:51.998 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:51.998 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:51.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:51.998 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Osypdm7cOvEv2stQd2/LWJsxs8U= 2023-11-02T19:26:51.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Osypdm7cOvEv2stQd2/LWJsxs8U=', 'Content-Length': '0'} 2023-11-02T19:26:51.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a77fe98cdf7f211c-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:51.999 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-11-02T19:26:52.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:52.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:52.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.003 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.004 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.004 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.004 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.005 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:52.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.005 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:NzPIT1CsBJmI5RgUUiXVnrblaMA= 2023-11-02T19:26:52.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NzPIT1CsBJmI5RgUUiXVnrblaMA=', 'Content-Length': '0'} 2023-11-02T19:26:52.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f882c99cc3a14c-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:52.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:52.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.010 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.010 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.010 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.011 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.011 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:52.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.012 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:NzPIT1CsBJmI5RgUUiXVnrblaMA= 2023-11-02T19:26:52.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:NzPIT1CsBJmI5RgUUiXVnrblaMA=', 'Content-Length': '0'} 2023-11-02T19:26:52.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df6ae382c7554286-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=c 2023-11-02T19:26:52.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=c 2023-11-02T19:26:52.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=c 2023-11-02T19:26:52.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.017 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.017 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.017 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.018 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.018 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.019 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T19:26:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T19:26:52.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f46bebc460db8126-006543e8c6-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=c 2023-11-02T19:26:52.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=c 2023-11-02T19:26:52.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=c 2023-11-02T19:26:52.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.024 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.024 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.025 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.025 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.025 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.026 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T19:26:52.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T19:26:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f4306d2f5dfac5c-006543e8c6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-11-02T19:26:52.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:52.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.032 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.032 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.032 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.033 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.033 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.034 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA= 2023-11-02T19:26:52.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA=', 'Content-Length': '0'} 2023-11-02T19:26:52.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005881b767765a5bcc-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:52.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.039 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.039 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.039 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.040 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.040 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.040 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA= 2023-11-02T19:26:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA=', 'Content-Length': '0'} 2023-11-02T19:26:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aca91e08f34407e5-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:52.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.045 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.046 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.046 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.046 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.047 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.047 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:tgDkFm2y571d4U8TDoRhLRy4A0M= 2023-11-02T19:26:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tgDkFm2y571d4U8TDoRhLRy4A0M=', 'Content-Length': '0'} 2023-11-02T19:26:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16ad4233372a269-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:52.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.052 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.053 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.053 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.053 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.053 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.054 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:tgDkFm2y571d4U8TDoRhLRy4A0M= 2023-11-02T19:26:52.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tgDkFm2y571d4U8TDoRhLRy4A0M=', 'Content-Length': '0'} 2023-11-02T19:26:52.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000933bfc2941628310-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.055 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-11-02T19:26:52.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:52.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.059 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.060 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.060 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.060 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.061 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.061 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA= 2023-11-02T19:26:52.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA=', 'Content-Length': '0'} 2023-11-02T19:26:52.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6abfa630eda764d-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:52.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.066 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.067 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.067 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.067 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.068 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.068 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA= 2023-11-02T19:26:52.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:lW0YiuaYHWIHNFi+M/efcr7fNxA=', 'Content-Length': '0'} 2023-11-02T19:26:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2907d1a0dcb3bdd-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=d 2023-11-02T19:26:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.073 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.074 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.074 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.074 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.075 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.075 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T19:26:52.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T19:26:52.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5e18fa9ffba4ce4-006543e8c6-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=d 2023-11-02T19:26:52.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.081 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.081 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.081 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.082 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.082 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.082 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T19:26:52.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T19:26:52.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aa47b26be909931-006543e8c6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.087 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.087 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.088 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.088 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.088 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:52.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.089 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:tRJqQ3TnddLM261w4X08TzYu2OA= 2023-11-02T19:26:52.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tRJqQ3TnddLM261w4X08TzYu2OA=', 'Content-Length': '0'} 2023-11-02T19:26:52.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c7fb2257529d08-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:52.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.093 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.094 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.094 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.094 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.095 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:52.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.095 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:tRJqQ3TnddLM261w4X08TzYu2OA= 2023-11-02T19:26:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:tRJqQ3TnddLM261w4X08TzYu2OA=', 'Content-Length': '0'} 2023-11-02T19:26:52.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8912922220555fb-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.096 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-11-02T19:26:52.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.100 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.100 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.101 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.101 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.101 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:52.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.102 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:hbXE2TweZhV3wKXwNF/3crtBuNI= 2023-11-02T19:26:52.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hbXE2TweZhV3wKXwNF/3crtBuNI=', 'Content-Length': '0'} 2023-11-02T19:26:52.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7e48addbb57ef4-006543e8c6-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:52.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.106 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.107 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.107 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.107 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.107 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:52.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.108 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:hbXE2TweZhV3wKXwNF/3crtBuNI= 2023-11-02T19:26:52.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hbXE2TweZhV3wKXwNF/3crtBuNI=', 'Content-Length': '0'} 2023-11-02T19:26:52.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006506712b7fc7a8f1-006543e8c6-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=d 2023-11-02T19:26:52.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.113 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.113 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.114 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.114 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.114 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.115 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T19:26:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T19:26:52.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096fbc9067ca8d5ca-006543e8c6-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=d 2023-11-02T19:26:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.120 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.121 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.121 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.121 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:58 GMT 2023-11-02T19:26:52.122 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.122 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA= 2023-11-02T19:26:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:58 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aAei2dJEh02OMwsQ6RpUyRj+1RA=', 'Content-Length': '0'} 2023-11-02T19:26:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000650327a5d034d63-006543e8c6-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.123 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=ibtljn-55 zones={a2, a1} 2023-11-02T19:26:52.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:52.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:52.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ibtljn-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:52.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:52.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:52.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:52.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "resharding": "None", 2023-11-02T19:26:52.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_index": { 2023-11-02T19:26:52.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 0, 2023-11-02T19:26:52.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:52.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "Normal", 2023-11-02T19:26:52.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "normal": { 2023-11-02T19:26:52.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 11, 2023-11-02T19:26:52.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:52.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:52.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "logs": [ 2023-11-02T19:26:52.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 0, 2023-11-02T19:26:52.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:52.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "InIndex", 2023-11-02T19:26:52.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "in_index": { 2023-11-02T19:26:52.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 0, 2023-11-02T19:26:52.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:52.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 11, 2023-11-02T19:26:52.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:52.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:52.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.135 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ibtljn-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:52.135 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ibtljn-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:52.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ibtljn-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:52.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:52.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:52.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: [ 2023-11-02T19:26:52.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "op_id": "1#00000000001.24.6", 2023-11-02T19:26:52.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "op_tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.3107276113268200966", 2023-11-02T19:26:52.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "op": "write", 2023-11-02T19:26:52.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "object": "a", 2023-11-02T19:26:52.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance": "", 2023-11-02T19:26:52.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": "complete", 2023-11-02T19:26:52.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "index_ver": 1, 2023-11-02T19:26:52.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:21:36.542931161Z", 2023-11-02T19:26:52.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "ver": { 2023-11-02T19:26:52.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pool": 9, 2023-11-02T19:26:52.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "epoch": 8 2023-11-02T19:26:52.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:52.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bilog_flags": 0, 2023-11-02T19:26:52.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "versioned": false, 2023-11-02T19:26:52.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "owner": "", 2023-11-02T19:26:52.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "owner_display_name": "", 2023-11-02T19:26:52.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones_trace": [ 2023-11-02T19:26:52.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "entry": "d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20" 2023-11-02T19:26:52.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:52.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:52.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "op_id": "5#00000000001.66.6", 2023-11-02T19:26:52.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "op_tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.3103660691027834791", 2023-11-02T19:26:52.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "op": "write", 2023-11-02T19:26:52.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "object": "d", 2023-11-02T19:26:52.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance": "", 2023-11-02T19:26:52.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": "complete", 2023-11-02T19:26:52.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "index_ver": 1, 2023-11-02T19:26:52.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:21:36.658842131Z", 2023-11-02T19:26:52.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "ver": { 2023-11-02T19:26:52.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pool": 9, 2023-11-02T19:26:52.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "epoch": 3 2023-11-02T19:26:52.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:52.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bilog_flags": 0, 2023-11-02T19:26:52.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "versioned": false, 2023-11-02T19:26:52.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "owner": "", 2023-11-02T19:26:52.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "owner_display_name": "", 2023-11-02T19:26:52.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones_trace": [ 2023-11-02T19:26:52.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "entry": "d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20" 2023-11-02T19:26:52.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:52.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:52.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "op_id": "8#00000000001.64.6", 2023-11-02T19:26:52.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "op_tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.13187839468121546283", 2023-11-02T19:26:52.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "op": "write", 2023-11-02T19:26:52.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "object": "c", 2023-11-02T19:26:52.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance": "", 2023-11-02T19:26:52.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": "complete", 2023-11-02T19:26:52.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "index_ver": 1, 2023-11-02T19:26:52.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:21:36.631179381Z", 2023-11-02T19:26:52.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "ver": { 2023-11-02T19:26:52.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pool": 9, 2023-11-02T19:26:52.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "epoch": 3 2023-11-02T19:26:52.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:52.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bilog_flags": 0, 2023-11-02T19:26:52.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "versioned": false, 2023-11-02T19:26:52.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "owner": "", 2023-11-02T19:26:52.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "owner_display_name": "", 2023-11-02T19:26:52.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones_trace": [ 2023-11-02T19:26:52.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "entry": "d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20" 2023-11-02T19:26:52.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:52.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:52.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "op_id": "10#00000000001.29.6", 2023-11-02T19:26:52.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "op_tag": "d61b2bea-59a8-4454-8491-ac3afbd04937.5737.15858638994088866377", 2023-11-02T19:26:52.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "op": "write", 2023-11-02T19:26:52.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "object": "b", 2023-11-02T19:26:52.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance": "", 2023-11-02T19:26:52.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": "complete", 2023-11-02T19:26:52.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "index_ver": 1, 2023-11-02T19:26:52.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:21:36.570165453Z", 2023-11-02T19:26:52.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "ver": { 2023-11-02T19:26:52.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pool": 9, 2023-11-02T19:26:52.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "epoch": 10 2023-11-02T19:26:52.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:52.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bilog_flags": 0, 2023-11-02T19:26:52.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "versioned": false, 2023-11-02T19:26:52.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "owner": "", 2023-11-02T19:26:52.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "owner_display_name": "", 2023-11-02T19:26:52.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones_trace": [ 2023-11-02T19:26:52.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "entry": "d61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20" 2023-11-02T19:26:52.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:52.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:52.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'ibtljn-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:52.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'ibtljn-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:52.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket ibtljn-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:52.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:52.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:52.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: tenant: 2023-11-02T19:26:52.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: bucket name: ibtljn-55 2023-11-02T19:26:52.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: total entries: 4 2023-11-02T19:26:52.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:52.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:52.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ibtljn-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:52.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:52.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:52.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:52.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "resharding": "None", 2023-11-02T19:26:52.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_index": { 2023-11-02T19:26:52.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 1, 2023-11-02T19:26:52.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:52.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "Normal", 2023-11-02T19:26:52.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "normal": { 2023-11-02T19:26:52.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 3, 2023-11-02T19:26:52.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:52.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:52.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "logs": [ 2023-11-02T19:26:52.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 0, 2023-11-02T19:26:52.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:52.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "InIndex", 2023-11-02T19:26:52.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "in_index": { 2023-11-02T19:26:52.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 0, 2023-11-02T19:26:52.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:52.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 11, 2023-11-02T19:26:52.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:52.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:52.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:52.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 1, 2023-11-02T19:26:52.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:52.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "InIndex", 2023-11-02T19:26:52.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "in_index": { 2023-11-02T19:26:52.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 1, 2023-11-02T19:26:52.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:52.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 3, 2023-11-02T19:26:52.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:52.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:52.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:52.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'ibtljn-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:52.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'ibtljn-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:52.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket ibtljn-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:52.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:52.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:52.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: [] 2023-11-02T19:26:52.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:52.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.189 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.190 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.190 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.190 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:59 GMT 2023-11-02T19:26:52.191 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.191 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY= 2023-11-02T19:26:52.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY=', 'Content-Length': '0'} 2023-11-02T19:26:52.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d29cbcb815e41db5-006543e8c7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e 2023-11-02T19:26:52.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.196 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:52.196 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:52.196 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:52.197 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:59 GMT 2023-11-02T19:26:52.197 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e 2023-11-02T19:26:52.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.198 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:m2ORivXwqPxYi5Zt0PdacwHhQr4= 2023-11-02T19:26:52.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:m2ORivXwqPxYi5Zt0PdacwHhQr4='} 2023-11-02T19:26:52.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038fb63939ca04287-006543e8c7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:52.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.202 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.202 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.203 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.203 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:59 GMT 2023-11-02T19:26:52.203 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.204 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY= 2023-11-02T19:26:52.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY=', 'Content-Length': '0'} 2023-11-02T19:26:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de5683270a229478-006543e8c7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:52.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:52.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:52.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f 2023-11-02T19:26:52.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:52.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.208 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:52.209 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:52.209 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:52.209 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:59 GMT 2023-11-02T19:26:52.210 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f 2023-11-02T19:26:52.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.210 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Xbtgbsgb93iiMBFOKU1zgENNuJI= 2023-11-02T19:26:52.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Xbtgbsgb93iiMBFOKU1zgENNuJI='} 2023-11-02T19:26:52.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077a1edea718f883d-006543e8c7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:52.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.215 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.215 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.215 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.216 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:59 GMT 2023-11-02T19:26:52.216 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.216 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY= 2023-11-02T19:26:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY=', 'Content-Length': '0'} 2023-11-02T19:26:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cc56f0ad8e51957-006543e8c7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:52.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:52.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:52.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g 2023-11-02T19:26:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:52.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.221 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:52.221 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:52.221 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:52.222 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:59 GMT 2023-11-02T19:26:52.222 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g 2023-11-02T19:26:52.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.223 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:2Wk5nFJoKaRJ03dE2gQapgSAuuw= 2023-11-02T19:26:52.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2Wk5nFJoKaRJ03dE2gQapgSAuuw='} 2023-11-02T19:26:52.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7870befa8280cd1-006543e8c7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.227 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.227 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.228 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.228 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:59 GMT 2023-11-02T19:26:52.228 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.229 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY= 2023-11-02T19:26:52.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VGGXrcYlNAj4iaxuPv7Y/ZrQFkY=', 'Content-Length': '0'} 2023-11-02T19:26:52.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002363a9f6bac8ec59-006543e8c7-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:52.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:52.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:52.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h 2023-11-02T19:26:52.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.233 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:52.234 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:52.234 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:52.234 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:21:59 GMT 2023-11-02T19:26:52.234 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h 2023-11-02T19:26:52.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.235 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:zjuOqJ37PlqRYPu9nzZnqjiUY+0= 2023-11-02T19:26:52.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:21:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zjuOqJ37PlqRYPu9nzZnqjiUY+0='} 2023-11-02T19:26:52.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2963113fa8a3027-006543e8c7-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=ibtljn-55 2023-11-02T19:26:52.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:52.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:52.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T19:26:52.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:52.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:52.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:21:59.489+0000 7f8dc642ff40 1 waiting to reach latest gen 1, on 0.. 2023-11-02T19:26:52.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:22:04.489+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , ] 2023-11-02T19:26:52.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:22:09.493+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , ] 2023-11-02T19:26:52.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:22:14.493+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , ] 2023-11-02T19:26:52.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:22:19.493+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , ] 2023-11-02T19:26:52.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:22:24.497+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , ] 2023-11-02T19:26:52.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:22:29.497+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , ] 2023-11-02T19:26:52.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:22:34.497+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , ] 2023-11-02T19:26:52.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:22:39.501+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , ] 2023-11-02T19:26:52.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:22:44.501+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , ] 2023-11-02T19:26:52.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:22:49.501+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , ] 2023-11-02T19:26:52.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:22:54.505+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , ] 2023-11-02T19:26:52.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:22:59.505+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , ] 2023-11-02T19:26:52.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:04.505+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [00000000003.24.6, , ] 2023-11-02T19:26:52.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:09.509+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [00000000003.24.6, , ] 2023-11-02T19:26:52.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:14.509+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [00000000003.24.6, , ] 2023-11-02T19:26:52.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:19.509+0000 7f8dc642ff40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [00000000003.24.6, , ] 2023-11-02T19:26:52.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:24.513+0000 7f8dc642ff40 1 bucket sync caught up with source: 2023-11-02T19:26:52.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [00000000003.24.6, 00000000002.17.6, 00000000002.29.6] 2023-11-02T19:26:52.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:24.513+0000 7f8dc642ff40 0 bucket checkpoint complete 2023-11-02T19:26:52.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=ibtljn-55 2023-11-02T19:26:52.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:52.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:52.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T19:26:52.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:52.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:52.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:24.665+0000 7fba2bb45f40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, 00000000001.174.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000001.192.6, , 00000000002.181.6] 2023-11-02T19:26:52.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T19:26:52.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:29.665+0000 7fba2bb45f40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, 00000000001.174.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000001.192.6, , 00000000002.181.6] 2023-11-02T19:26:52.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T19:26:52.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:34.669+0000 7fba2bb45f40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000001.192.6, , 00000000002.181.6] 2023-11-02T19:26:52.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T19:26:52.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:39.673+0000 7fba2bb45f40 1 waiting for incremental sync to catch up: 2023-11-02T19:26:52.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000001.192.6, , 00000000002.181.6] 2023-11-02T19:26:52.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T19:26:52.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:44.673+0000 7fba2bb45f40 1 bucket sync caught up with source: 2023-11-02T19:26:52.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T19:26:52.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T19:26:52.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:44.673+0000 7fba2bb45f40 0 bucket checkpoint complete 2023-11-02T19:26:52.261 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=ibtljn-55 zones={a2, a1} 2023-11-02T19:26:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:52.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:26:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.265 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.266 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.266 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.266 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.267 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.267 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:+a0H42agG1AtvF3zm5w/tCNf+p4= 2023-11-02T19:26:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+a0H42agG1AtvF3zm5w/tCNf+p4=', 'Content-Length': '0'} 2023-11-02T19:26:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000514e3ee38acdf241-006543e930-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:52.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-11-02T19:26:52.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.272 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.272 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.273 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.273 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.273 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.274 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:+a0H42agG1AtvF3zm5w/tCNf+p4= 2023-11-02T19:26:52.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+a0H42agG1AtvF3zm5w/tCNf+p4=', 'Content-Length': '0'} 2023-11-02T19:26:52.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020b245db2d47e723-006543e930-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.275 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-11-02T19:26:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:52.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:52.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.279 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.279 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.280 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.280 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.280 INFO:tasks.rgw_multisite_tests:/ibtljn-55/?versions 2023-11-02T19:26:52.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.281 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE= 2023-11-02T19:26:52.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE=', 'Content-Length': '0'} 2023-11-02T19:26:52.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f37251e9df0ae2a-006543e930-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.282 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-11-02T19:26:52.283 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-11-02T19:26:52.283 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-11-02T19:26:52.283 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-11-02T19:26:52.284 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-11-02T19:26:52.284 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-11-02T19:26:52.284 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-11-02T19:26:52.284 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-11-02T19:26:52.285 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-11-02T19:26:52.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:52.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:52.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.289 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.290 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.290 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.290 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.290 INFO:tasks.rgw_multisite_tests:/ibtljn-55/?versions 2023-11-02T19:26:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.291 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE= 2023-11-02T19:26:52.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE=', 'Content-Length': '0'} 2023-11-02T19:26:52.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bf11f0af695f87d-006543e930-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.292 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-11-02T19:26:52.293 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-11-02T19:26:52.293 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-11-02T19:26:52.293 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-11-02T19:26:52.294 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-11-02T19:26:52.294 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-11-02T19:26:52.294 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-11-02T19:26:52.295 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-11-02T19:26:52.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:52.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:52.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.299 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.299 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.300 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.300 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.300 INFO:tasks.rgw_multisite_tests:/ibtljn-55/?versions 2023-11-02T19:26:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.301 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE= 2023-11-02T19:26:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE=', 'Content-Length': '0'} 2023-11-02T19:26:52.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000afbf7b61c980986-006543e930-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:52.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:52.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.306 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.307 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.307 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.307 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.307 INFO:tasks.rgw_multisite_tests:/ibtljn-55/?versions 2023-11-02T19:26:52.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.308 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE= 2023-11-02T19:26:52.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xVRCyJjN8UM+sFVvnj546/11NoE=', 'Content-Length': '0'} 2023-11-02T19:26:52.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000828a25bb6d3fea7e-006543e930-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.309 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-11-02T19:26:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:52.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.314 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.314 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.314 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.315 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.315 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.316 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak= 2023-11-02T19:26:52.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak=', 'Content-Length': '0'} 2023-11-02T19:26:52.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc34b0b67dabda34-006543e930-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:52.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:52.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:52.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.321 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.321 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.321 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.322 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.322 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.323 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak= 2023-11-02T19:26:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak=', 'Content-Length': '0'} 2023-11-02T19:26:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e71306008795bccd-006543e930-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:52.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:52.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:52.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.327 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.328 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.328 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.328 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.329 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.329 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:xCIMqLHd3gWHY39njcXj/dPFDgo= 2023-11-02T19:26:52.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xCIMqLHd3gWHY39njcXj/dPFDgo=', 'Content-Length': '0'} 2023-11-02T19:26:52.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af3c2767ef905cff-006543e930-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:52.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.334 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.335 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.335 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.335 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.336 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.336 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:xCIMqLHd3gWHY39njcXj/dPFDgo= 2023-11-02T19:26:52.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xCIMqLHd3gWHY39njcXj/dPFDgo=', 'Content-Length': '0'} 2023-11-02T19:26:52.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d16c5576fac4f9-006543e930-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.337 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-11-02T19:26:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:52.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:52.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.342 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.342 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.342 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.343 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.343 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.344 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak= 2023-11-02T19:26:52.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak=', 'Content-Length': '0'} 2023-11-02T19:26:52.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000294c65f0884036ad-006543e930-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:52.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.349 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.349 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.349 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.349 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.350 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:52.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.350 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak= 2023-11-02T19:26:52.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BmAglSbO1pmufxbFnCFLQ0Jiuak=', 'Content-Length': '0'} 2023-11-02T19:26:52.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007159c289b28054c1-006543e930-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=a 2023-11-02T19:26:52.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=a 2023-11-02T19:26:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=a 2023-11-02T19:26:52.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.355 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.356 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.356 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.356 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.357 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.357 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:/4eBOtjozJ7Mr+oTagkPsp2tLks= 2023-11-02T19:26:52.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/4eBOtjozJ7Mr+oTagkPsp2tLks=', 'Content-Length': '0'} 2023-11-02T19:26:52.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530f11c7b875d79b-006543e930-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=a 2023-11-02T19:26:52.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=a 2023-11-02T19:26:52.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=a 2023-11-02T19:26:52.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.363 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.363 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.363 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.363 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.364 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.364 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:/4eBOtjozJ7Mr+oTagkPsp2tLks= 2023-11-02T19:26:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/4eBOtjozJ7Mr+oTagkPsp2tLks=', 'Content-Length': '0'} 2023-11-02T19:26:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f49949c32527313-006543e930-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:52.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:52.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.369 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.369 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.370 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.370 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.370 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:52.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.371 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:2LnUXSqDDiQbR5kNcRgEQJHtIQQ= 2023-11-02T19:26:52.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2LnUXSqDDiQbR5kNcRgEQJHtIQQ=', 'Content-Length': '0'} 2023-11-02T19:26:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5f37cc431434c0-006543e930-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:52.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.375 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.376 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.376 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.376 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.377 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:52.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.377 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:2LnUXSqDDiQbR5kNcRgEQJHtIQQ= 2023-11-02T19:26:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2LnUXSqDDiQbR5kNcRgEQJHtIQQ=', 'Content-Length': '0'} 2023-11-02T19:26:52.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fbcd0036b069fc6-006543e930-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.378 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-11-02T19:26:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:52.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.382 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.382 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.383 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.383 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.383 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.384 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:5EEOW46llURoq4x6cUCrR/ukZc8= 2023-11-02T19:26:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5EEOW46llURoq4x6cUCrR/ukZc8=', 'Content-Length': '0'} 2023-11-02T19:26:52.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80a827ef3d36f44-006543e930-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:52.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:52.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.388 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.389 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.389 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.389 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:44 GMT 2023-11-02T19:26:52.389 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:52.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.390 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:5EEOW46llURoq4x6cUCrR/ukZc8= 2023-11-02T19:26:52.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:44 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5EEOW46llURoq4x6cUCrR/ukZc8=', 'Content-Length': '0'} 2023-11-02T19:26:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008770b7e5d4b1ef4e-006543e930-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:44 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=a 2023-11-02T19:26:52.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=a 2023-11-02T19:26:52.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=a 2023-11-02T19:26:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.395 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.395 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.396 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.396 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.396 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.397 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045a3b2d4e5d5bcf3-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=a 2023-11-02T19:26:52.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=a 2023-11-02T19:26:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=a 2023-11-02T19:26:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.402 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.403 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.403 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.403 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.404 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.404 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeb3d1f4046e5383-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.406 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-11-02T19:26:52.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:52.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:52.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:52.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.410 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.410 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.411 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.411 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.411 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.412 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E= 2023-11-02T19:26:52.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E=', 'Content-Length': '0'} 2023-11-02T19:26:52.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beff83737622eafd-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:52.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:52.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.417 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.417 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.417 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.418 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.418 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.419 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E= 2023-11-02T19:26:52.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E=', 'Content-Length': '0'} 2023-11-02T19:26:52.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000270350b879446a25-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:52.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:52.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:52.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.424 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.424 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.424 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.425 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.425 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.426 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:s8QtRkov+rRtxMymyHvibzZXuzA= 2023-11-02T19:26:52.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s8QtRkov+rRtxMymyHvibzZXuzA=', 'Content-Length': '0'} 2023-11-02T19:26:52.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095aff3164e432d16-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:52.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:52.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:52.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.431 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.431 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.431 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.431 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.432 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.432 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:s8QtRkov+rRtxMymyHvibzZXuzA= 2023-11-02T19:26:52.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:s8QtRkov+rRtxMymyHvibzZXuzA=', 'Content-Length': '0'} 2023-11-02T19:26:52.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c86cbbec939bb5ef-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.433 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-11-02T19:26:52.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:52.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:52.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:52.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.438 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.438 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.438 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.439 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.439 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.440 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E= 2023-11-02T19:26:52.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E=', 'Content-Length': '0'} 2023-11-02T19:26:52.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5d240e9c6019b65-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:52.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.445 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.445 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.445 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.445 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.446 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:52.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.446 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E= 2023-11-02T19:26:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:+iJLbYOvs933PR1Yd1PIbOk+01E=', 'Content-Length': '0'} 2023-11-02T19:26:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8e45923aba1a930-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=b 2023-11-02T19:26:52.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=b 2023-11-02T19:26:52.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=b 2023-11-02T19:26:52.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.451 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.452 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.452 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.452 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.453 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.453 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0c9853cb9a6926-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=b 2023-11-02T19:26:52.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=b 2023-11-02T19:26:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=b 2023-11-02T19:26:52.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.459 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.459 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.459 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.460 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.460 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.460 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000369df3f21d7e1545-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:52.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:52.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.465 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.466 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.466 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.466 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.466 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.467 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:sMVL5IGZSb8fW0iYYxsbD1T2E9o= 2023-11-02T19:26:52.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sMVL5IGZSb8fW0iYYxsbD1T2E9o=', 'Content-Length': '0'} 2023-11-02T19:26:52.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e74008c69d0d178-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:52.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:52.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.471 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.472 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.472 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.472 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.473 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:52.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.473 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:sMVL5IGZSb8fW0iYYxsbD1T2E9o= 2023-11-02T19:26:52.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sMVL5IGZSb8fW0iYYxsbD1T2E9o=', 'Content-Length': '0'} 2023-11-02T19:26:52.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c613a15f4d6784e-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.474 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-11-02T19:26:52.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:52.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:52.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.478 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.478 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.479 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.479 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.479 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:52.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.480 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:D+QpjtNJDrgekXq2in5hKFPhyXo= 2023-11-02T19:26:52.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+QpjtNJDrgekXq2in5hKFPhyXo=', 'Content-Length': '0'} 2023-11-02T19:26:52.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1351a0d45cc51a-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:52.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.484 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.485 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.485 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.485 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.486 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:52.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.486 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:D+QpjtNJDrgekXq2in5hKFPhyXo= 2023-11-02T19:26:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D+QpjtNJDrgekXq2in5hKFPhyXo=', 'Content-Length': '0'} 2023-11-02T19:26:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000310845c36817ea9a-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=b 2023-11-02T19:26:52.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=b 2023-11-02T19:26:52.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=b 2023-11-02T19:26:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.491 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.492 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.492 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.492 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.493 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.493 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000755ffc128bbe157c-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=b 2023-11-02T19:26:52.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=b 2023-11-02T19:26:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=b 2023-11-02T19:26:52.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.498 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.499 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.499 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.499 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.500 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.500 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002362434602d4ebd3-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.502 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-11-02T19:26:52.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:52.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:52.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.506 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.506 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.507 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.507 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.507 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.508 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s= 2023-11-02T19:26:52.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s=', 'Content-Length': '0'} 2023-11-02T19:26:52.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b33e7fedcb8a89d8-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:52.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:52.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.513 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.513 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.514 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.514 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.514 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.515 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s= 2023-11-02T19:26:52.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s=', 'Content-Length': '0'} 2023-11-02T19:26:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e96898977e5563d8-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:52.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:52.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.520 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.520 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.521 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.521 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.521 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.522 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:aDB6OxN1cjCMQutFXr4i5H/YmV0= 2023-11-02T19:26:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aDB6OxN1cjCMQutFXr4i5H/YmV0=', 'Content-Length': '0'} 2023-11-02T19:26:52.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000280b19ff4183ad14-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:52.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:52.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.527 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.527 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.527 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.528 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.528 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.529 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:aDB6OxN1cjCMQutFXr4i5H/YmV0= 2023-11-02T19:26:52.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:aDB6OxN1cjCMQutFXr4i5H/YmV0=', 'Content-Length': '0'} 2023-11-02T19:26:52.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004891944ff8fb2028-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.530 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-11-02T19:26:52.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:52.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:52.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.534 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.534 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.535 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.535 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.535 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.536 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s= 2023-11-02T19:26:52.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s=', 'Content-Length': '0'} 2023-11-02T19:26:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d20be8ec4fa5c1ce-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:52.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:52.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.541 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.541 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.541 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.542 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.542 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:52.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.543 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s= 2023-11-02T19:26:52.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5cfu96kMD9jwmWs2NEwrSiYwt0s=', 'Content-Length': '0'} 2023-11-02T19:26:52.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006921a05818c71584-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=c 2023-11-02T19:26:52.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=c 2023-11-02T19:26:52.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=c 2023-11-02T19:26:52.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.548 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.548 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.548 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.549 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.549 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.550 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ceb945f0907386d-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=c 2023-11-02T19:26:52.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=c 2023-11-02T19:26:52.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=c 2023-11-02T19:26:52.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.555 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.555 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.555 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.556 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.556 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.557 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81b0d132f85de3c-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:52.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:52.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:52.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.561 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.562 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.562 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.562 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.563 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:52.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.563 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:0dhauboEcquEqTNzJho1KKFS5AQ= 2023-11-02T19:26:52.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0dhauboEcquEqTNzJho1KKFS5AQ=', 'Content-Length': '0'} 2023-11-02T19:26:52.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d280a9a6b01864f-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:52.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:52.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:52.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.568 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.568 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.568 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.569 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.569 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:52.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.570 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:0dhauboEcquEqTNzJho1KKFS5AQ= 2023-11-02T19:26:52.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0dhauboEcquEqTNzJho1KKFS5AQ=', 'Content-Length': '0'} 2023-11-02T19:26:52.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d0bd2f5b25f9f06-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.571 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-11-02T19:26:52.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:52.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:52.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.574 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.575 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.575 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.575 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.576 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:52.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.576 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:9aOnocZQXER+3ZnWoOp3MuYRyyI= 2023-11-02T19:26:52.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9aOnocZQXER+3ZnWoOp3MuYRyyI=', 'Content-Length': '0'} 2023-11-02T19:26:52.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087d95613fbe174ab-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:52.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:52.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:52.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.582 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.582 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.582 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.583 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.583 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:52.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.583 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:9aOnocZQXER+3ZnWoOp3MuYRyyI= 2023-11-02T19:26:52.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9aOnocZQXER+3ZnWoOp3MuYRyyI=', 'Content-Length': '0'} 2023-11-02T19:26:52.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000865b15b77276136-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=c 2023-11-02T19:26:52.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=c 2023-11-02T19:26:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=c 2023-11-02T19:26:52.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.589 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.589 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.589 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.589 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.590 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.590 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1df7b3f46ade27f-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=c 2023-11-02T19:26:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=c 2023-11-02T19:26:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=c 2023-11-02T19:26:52.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.596 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.596 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.596 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.597 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.597 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.598 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083e145a5e5c42527-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.599 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-11-02T19:26:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:52.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.603 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.603 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.604 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.604 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.604 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.605 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q= 2023-11-02T19:26:52.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q=', 'Content-Length': '0'} 2023-11-02T19:26:52.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005655c90f5ab06c22-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.610 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.611 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.611 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.611 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.612 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.612 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q= 2023-11-02T19:26:52.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q=', 'Content-Length': '0'} 2023-11-02T19:26:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761ee09bf1de827f-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:52.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.617 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.618 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.618 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.618 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.619 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.619 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:WhisJ7xYy6a7hFxvSyh0Hbinnk8= 2023-11-02T19:26:52.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhisJ7xYy6a7hFxvSyh0Hbinnk8=', 'Content-Length': '0'} 2023-11-02T19:26:52.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d33942175b7ae70-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:52.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.624 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.625 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.625 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.625 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.625 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.626 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:WhisJ7xYy6a7hFxvSyh0Hbinnk8= 2023-11-02T19:26:52.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WhisJ7xYy6a7hFxvSyh0Hbinnk8=', 'Content-Length': '0'} 2023-11-02T19:26:52.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c8cdd8216139739-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.627 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-11-02T19:26:52.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:52.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.631 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.632 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.632 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.632 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.633 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.633 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q= 2023-11-02T19:26:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q=', 'Content-Length': '0'} 2023-11-02T19:26:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b006fddc760b695-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.638 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.639 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.639 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.639 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.640 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:52.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.640 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q= 2023-11-02T19:26:52.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XkT4SlJ+77nYufMFucZRdZzTj6Q=', 'Content-Length': '0'} 2023-11-02T19:26:52.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b28c3bd9df2c9f59-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=d 2023-11-02T19:26:52.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.645 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.646 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.646 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.646 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.647 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.647 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bc1b497e568bc87-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=d 2023-11-02T19:26:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.653 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.653 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.653 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.654 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.654 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.655 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009ec97546fd447c1-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.659 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.660 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.660 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.660 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.661 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:52.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.661 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:VufUPEqhWGiXT6BNWEutto5ODXg= 2023-11-02T19:26:52.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VufUPEqhWGiXT6BNWEutto5ODXg=', 'Content-Length': '0'} 2023-11-02T19:26:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1b72434567abc47-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:52.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.666 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.666 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.666 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.667 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.667 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:52.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.667 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:VufUPEqhWGiXT6BNWEutto5ODXg= 2023-11-02T19:26:52.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VufUPEqhWGiXT6BNWEutto5ODXg=', 'Content-Length': '0'} 2023-11-02T19:26:52.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009295ed00179fe3e6-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.668 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-11-02T19:26:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.672 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.672 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.673 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.673 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.673 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:52.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.674 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:qCydm+T4m2OeThS7FCpOjNyeGtA= 2023-11-02T19:26:52.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qCydm+T4m2OeThS7FCpOjNyeGtA=', 'Content-Length': '0'} 2023-11-02T19:26:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c59731097be6ba6-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:52.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:52.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:52.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.679 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.679 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.679 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.680 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.680 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:52.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.681 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:qCydm+T4m2OeThS7FCpOjNyeGtA= 2023-11-02T19:26:52.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qCydm+T4m2OeThS7FCpOjNyeGtA=', 'Content-Length': '0'} 2023-11-02T19:26:52.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000293a76c8d5af9d0a-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=d 2023-11-02T19:26:52.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.686 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.686 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.686 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.687 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.687 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.688 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddc5e7ade0c67644-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=d 2023-11-02T19:26:52.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=d 2023-11-02T19:26:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.693 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.693 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.694 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.694 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.694 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.695 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082cd7572f27fdc4e-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.696 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-11-02T19:26:52.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:52.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:52.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e?versionId=null 2023-11-02T19:26:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.701 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.701 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.701 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.702 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.702 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.703 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo= 2023-11-02T19:26:52.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo=', 'Content-Length': '0'} 2023-11-02T19:26:52.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbf89dc28e9bf650-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:52.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:52.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e?versionId=null 2023-11-02T19:26:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.708 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.708 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.708 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.709 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.709 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.710 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo= 2023-11-02T19:26:52.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo=', 'Content-Length': '0'} 2023-11-02T19:26:52.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca0ed20c7dc87cdc-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e?versionId=null 2023-11-02T19:26:52.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.715 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.715 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.715 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.716 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.716 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.716 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KMINFBuySUK3RWKayVmusWhMaSc= 2023-11-02T19:26:52.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KMINFBuySUK3RWKayVmusWhMaSc=', 'Content-Length': '0'} 2023-11-02T19:26:52.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b4f648105616df3-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e?versionId=null 2023-11-02T19:26:52.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.722 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.722 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.722 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.723 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.723 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.723 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KMINFBuySUK3RWKayVmusWhMaSc= 2023-11-02T19:26:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KMINFBuySUK3RWKayVmusWhMaSc=', 'Content-Length': '0'} 2023-11-02T19:26:52.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e5053107dbd7c8-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.724 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-11-02T19:26:52.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:52.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:52.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e?versionId=null 2023-11-02T19:26:52.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.729 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.729 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.729 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.730 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.730 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.731 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo= 2023-11-02T19:26:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo=', 'Content-Length': '0'} 2023-11-02T19:26:52.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5fc00b70f4bdd6-006543e931-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:52.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e?versionId=null 2023-11-02T19:26:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.736 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.736 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.736 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.737 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.737 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e?versionId=null 2023-11-02T19:26:52.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.738 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo= 2023-11-02T19:26:52.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:FuaxoZMV8NEyouSJSwBldhgDvvo=', 'Content-Length': '0'} 2023-11-02T19:26:52.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ba4c68b2f3d634-006543e931-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=e 2023-11-02T19:26:52.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=e 2023-11-02T19:26:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=e 2023-11-02T19:26:52.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.743 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.743 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.743 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.744 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.744 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.745 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fa88bc1663ccc8e-006543e931-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=e 2023-11-02T19:26:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=e 2023-11-02T19:26:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=e 2023-11-02T19:26:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.750 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.750 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.751 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.751 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:45 GMT 2023-11-02T19:26:52.751 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.752 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo= 2023-11-02T19:26:52.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:45 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:C9j5rx1s0mWFS3gqugPYoEOOYxo=', 'Content-Length': '0'} 2023-11-02T19:26:52.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a651c42beba3da-006543e931-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:52.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e 2023-11-02T19:26:52.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.757 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.757 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.757 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.757 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.758 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e 2023-11-02T19:26:52.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.758 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:h9PpD4i1qpfzW3GZuon6Mlfc14c= 2023-11-02T19:26:52.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h9PpD4i1qpfzW3GZuon6Mlfc14c=', 'Content-Length': '0'} 2023-11-02T19:26:52.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004153762d179a799c-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:52.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:52.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e 2023-11-02T19:26:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.763 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.763 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.763 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.764 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.764 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e 2023-11-02T19:26:52.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.765 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:h9PpD4i1qpfzW3GZuon6Mlfc14c= 2023-11-02T19:26:52.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:h9PpD4i1qpfzW3GZuon6Mlfc14c=', 'Content-Length': '0'} 2023-11-02T19:26:52.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fbbf846dc638251-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.766 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-11-02T19:26:52.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:52.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e 2023-11-02T19:26:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.769 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.770 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.770 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.770 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.771 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e 2023-11-02T19:26:52.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.771 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:P7NA20Id1QaZ96le+NgamiQEXLo= 2023-11-02T19:26:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P7NA20Id1QaZ96le+NgamiQEXLo=', 'Content-Length': '0'} 2023-11-02T19:26:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053658e0213076eb7-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:52.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:52.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e 2023-11-02T19:26:52.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.776 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.776 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.776 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.777 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.777 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e 2023-11-02T19:26:52.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.778 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:P7NA20Id1QaZ96le+NgamiQEXLo= 2023-11-02T19:26:52.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:P7NA20Id1QaZ96le+NgamiQEXLo=', 'Content-Length': '0'} 2023-11-02T19:26:52.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006692644d0fa616b6-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=e 2023-11-02T19:26:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=e 2023-11-02T19:26:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=e 2023-11-02T19:26:52.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.783 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.783 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.783 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.785 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.785 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.786 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:52.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:52.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045d478a16926d168-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=e 2023-11-02T19:26:52.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=e 2023-11-02T19:26:52.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=e 2023-11-02T19:26:52.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.791 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.792 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.792 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.792 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.793 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.793 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000210f1a7c48ebc10c-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-11-02T19:26:52.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:52.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f?versionId=null 2023-11-02T19:26:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.799 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.799 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.800 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.800 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.800 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.801 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM= 2023-11-02T19:26:52.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM=', 'Content-Length': '0'} 2023-11-02T19:26:52.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065633ff7186d92a3-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:52.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:52.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f?versionId=null 2023-11-02T19:26:52.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.806 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.806 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.807 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.807 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.807 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.808 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM= 2023-11-02T19:26:52.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM=', 'Content-Length': '0'} 2023-11-02T19:26:52.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b484262fcdc894f-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:52.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:52.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f?versionId=null 2023-11-02T19:26:52.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.813 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.813 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.814 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.814 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.814 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.815 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:iQdClzMjWM0C8nDKP3tG1BMS2Nk= 2023-11-02T19:26:52.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iQdClzMjWM0C8nDKP3tG1BMS2Nk=', 'Content-Length': '0'} 2023-11-02T19:26:52.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b3987236ec9f0fd-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:52.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:52.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f?versionId=null 2023-11-02T19:26:52.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.820 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.820 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.820 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.821 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.821 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.822 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:iQdClzMjWM0C8nDKP3tG1BMS2Nk= 2023-11-02T19:26:52.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:iQdClzMjWM0C8nDKP3tG1BMS2Nk=', 'Content-Length': '0'} 2023-11-02T19:26:52.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da90ffdd60dbbf35-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.823 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-11-02T19:26:52.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:52.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:52.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f?versionId=null 2023-11-02T19:26:52.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.827 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.827 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.828 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.828 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.828 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.829 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM= 2023-11-02T19:26:52.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM=', 'Content-Length': '0'} 2023-11-02T19:26:52.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090d3649409952ee0-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:52.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:52.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f?versionId=null 2023-11-02T19:26:52.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.834 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.834 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.834 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.835 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.835 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f?versionId=null 2023-11-02T19:26:52.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.836 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM= 2023-11-02T19:26:52.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6d5c85cDA7/cdD3p81tRxyVZPEM=', 'Content-Length': '0'} 2023-11-02T19:26:52.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce73a4d85acbd456-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=f 2023-11-02T19:26:52.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=f 2023-11-02T19:26:52.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=f 2023-11-02T19:26:52.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.841 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.841 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.841 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.842 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.842 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.843 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:52.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:52.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea5648ab64248694-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=f 2023-11-02T19:26:52.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=f 2023-11-02T19:26:52.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=f 2023-11-02T19:26:52.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.848 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.848 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.849 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.849 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.849 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.850 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:52.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ff01f2abb5c7640-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:52.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:52.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f 2023-11-02T19:26:52.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.855 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.855 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.855 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.856 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.856 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f 2023-11-02T19:26:52.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.857 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:rtcdYEBTOJPW+1jkvrOj5cXN3cw= 2023-11-02T19:26:52.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rtcdYEBTOJPW+1jkvrOj5cXN3cw=', 'Content-Length': '0'} 2023-11-02T19:26:52.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097bddae003e44bf0-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:52.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:52.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f 2023-11-02T19:26:52.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.861 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.861 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.862 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.862 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.862 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f 2023-11-02T19:26:52.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.863 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:rtcdYEBTOJPW+1jkvrOj5cXN3cw= 2023-11-02T19:26:52.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rtcdYEBTOJPW+1jkvrOj5cXN3cw=', 'Content-Length': '0'} 2023-11-02T19:26:52.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006fc6a7b2f74015e-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.864 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-11-02T19:26:52.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:52.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:52.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f 2023-11-02T19:26:52.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.868 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.868 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.868 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.869 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.869 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f 2023-11-02T19:26:52.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.870 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Z2lPSO8aoobcY5oFTRF8nCfWmCU= 2023-11-02T19:26:52.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z2lPSO8aoobcY5oFTRF8nCfWmCU=', 'Content-Length': '0'} 2023-11-02T19:26:52.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000783684df25c28aa8-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:52.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:52.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f 2023-11-02T19:26:52.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.874 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.874 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.874 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.875 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.875 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f 2023-11-02T19:26:52.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.876 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Z2lPSO8aoobcY5oFTRF8nCfWmCU= 2023-11-02T19:26:52.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Z2lPSO8aoobcY5oFTRF8nCfWmCU=', 'Content-Length': '0'} 2023-11-02T19:26:52.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f18afd07f1bdb7f-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=f 2023-11-02T19:26:52.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=f 2023-11-02T19:26:52.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=f 2023-11-02T19:26:52.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.881 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.881 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.881 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.882 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.882 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.883 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:52.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:52.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e7393c8f20b012c-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=f 2023-11-02T19:26:52.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=f 2023-11-02T19:26:52.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=f 2023-11-02T19:26:52.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.888 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.888 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.889 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.889 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.889 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.890 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:52.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:52.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8962867d21802a7-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.891 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-11-02T19:26:52.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:52.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:52.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g?versionId=null 2023-11-02T19:26:52.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.896 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.896 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.896 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.897 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.897 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.898 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE= 2023-11-02T19:26:52.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE=', 'Content-Length': '0'} 2023-11-02T19:26:52.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3c6e5f1bf2c138c-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:52.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:52.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g?versionId=null 2023-11-02T19:26:52.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.903 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.903 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.903 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.904 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.904 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.905 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE= 2023-11-02T19:26:52.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE=', 'Content-Length': '0'} 2023-11-02T19:26:52.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2d991ca08824da0-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:52.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:52.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g?versionId=null 2023-11-02T19:26:52.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.909 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.910 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.910 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.911 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.911 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.912 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:r/FX1ZNLpK2vbFJbue/M20RrlLY= 2023-11-02T19:26:52.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:r/FX1ZNLpK2vbFJbue/M20RrlLY=', 'Content-Length': '0'} 2023-11-02T19:26:52.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003af77e6c313cd0a5-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:52.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:52.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g?versionId=null 2023-11-02T19:26:52.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.917 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.917 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.917 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.918 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.918 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.918 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:r/FX1ZNLpK2vbFJbue/M20RrlLY= 2023-11-02T19:26:52.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:r/FX1ZNLpK2vbFJbue/M20RrlLY=', 'Content-Length': '0'} 2023-11-02T19:26:52.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dce58ec352820ae-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.919 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-11-02T19:26:52.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:52.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:52.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g?versionId=null 2023-11-02T19:26:52.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.924 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.924 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.924 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.925 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.925 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.926 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE= 2023-11-02T19:26:52.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE=', 'Content-Length': '0'} 2023-11-02T19:26:52.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a1810ce3bcb583-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:52.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:52.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g?versionId=null 2023-11-02T19:26:52.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.931 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.931 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.931 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.932 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.932 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g?versionId=null 2023-11-02T19:26:52.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.933 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE= 2023-11-02T19:26:52.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xbScoDJZL0gJVMVdFVbiYj3qxLE=', 'Content-Length': '0'} 2023-11-02T19:26:52.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e7ad1e1a8e8b8d6-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=g 2023-11-02T19:26:52.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=g 2023-11-02T19:26:52.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=g 2023-11-02T19:26:52.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.938 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.938 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.938 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.939 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.939 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.940 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:52.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:52.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef4184ea2ff7f130-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=g 2023-11-02T19:26:52.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=g 2023-11-02T19:26:52.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=g 2023-11-02T19:26:52.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.945 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.945 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.946 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.946 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.946 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.947 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:52.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:52.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a3d870d11e71a82-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:52.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:52.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g 2023-11-02T19:26:52.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.951 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.952 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.952 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.952 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.953 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g 2023-11-02T19:26:52.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.953 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:JCkhL1r3KNkAJucKYlMvOuz+3ys= 2023-11-02T19:26:52.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JCkhL1r3KNkAJucKYlMvOuz+3ys=', 'Content-Length': '0'} 2023-11-02T19:26:52.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027b102328d4b261b-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:52.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:52.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:52.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g 2023-11-02T19:26:52.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.958 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:52.958 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.958 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.959 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.959 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g 2023-11-02T19:26:52.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.960 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:JCkhL1r3KNkAJucKYlMvOuz+3ys= 2023-11-02T19:26:52.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JCkhL1r3KNkAJucKYlMvOuz+3ys=', 'Content-Length': '0'} 2023-11-02T19:26:52.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094f5e3cbe2a7a14e-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.961 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-11-02T19:26:52.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:52.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:52.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g 2023-11-02T19:26:52.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.965 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.965 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.965 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.966 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.966 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g 2023-11-02T19:26:52.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.967 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:AcqfVwEqCcMxmM+ykHd5XCuo00Q= 2023-11-02T19:26:52.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AcqfVwEqCcMxmM+ykHd5XCuo00Q=', 'Content-Length': '0'} 2023-11-02T19:26:52.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000193687b1b133aaf7-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:52.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:52.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g 2023-11-02T19:26:52.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.971 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.971 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.972 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.972 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.972 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g 2023-11-02T19:26:52.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.973 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:AcqfVwEqCcMxmM+ykHd5XCuo00Q= 2023-11-02T19:26:52.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:AcqfVwEqCcMxmM+ykHd5XCuo00Q=', 'Content-Length': '0'} 2023-11-02T19:26:52.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3cc0009bd009b0a-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=g 2023-11-02T19:26:52.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=g 2023-11-02T19:26:52.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=g 2023-11-02T19:26:52.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.978 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.978 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.979 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.979 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.979 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.980 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:52.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:52.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3379458befd31bf-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:52.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:52.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=g 2023-11-02T19:26:52.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=g 2023-11-02T19:26:52.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=g 2023-11-02T19:26:52.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:52.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:52.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.985 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.986 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.986 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.986 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.987 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:52.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.987 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:52.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:52.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045c34c5c9a88b664-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:52.990 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-11-02T19:26:52.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:52.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:52.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:52.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:52.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h?versionId=null 2023-11-02T19:26:52.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:52.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:52.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:52.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:52.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:52.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:52.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:52.994 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:52.995 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.995 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:52.995 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:52.996 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h?versionId=null 2023-11-02T19:26:52.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:52.996 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk= 2023-11-02T19:26:52.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk=', 'Content-Length': '0'} 2023-11-02T19:26:52.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceb5fbdbe07fc61d-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:52.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:52.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:52.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:52.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:52.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:52.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h?versionId=null 2023-11-02T19:26:52.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.001 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.002 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.002 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.002 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.003 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.003 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk= 2023-11-02T19:26:53.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk=', 'Content-Length': '0'} 2023-11-02T19:26:53.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd3d42c8ace54a77-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h?versionId=null 2023-11-02T19:26:53.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.008 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.009 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.009 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.009 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.010 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.010 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:oGZvOYLtMkqsI9xAhSVbMwXqEdA= 2023-11-02T19:26:53.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oGZvOYLtMkqsI9xAhSVbMwXqEdA=', 'Content-Length': '0'} 2023-11-02T19:26:53.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e44110f7468108a9-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h?versionId=null 2023-11-02T19:26:53.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.015 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.016 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.016 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.016 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.016 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.017 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:oGZvOYLtMkqsI9xAhSVbMwXqEdA= 2023-11-02T19:26:53.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:oGZvOYLtMkqsI9xAhSVbMwXqEdA=', 'Content-Length': '0'} 2023-11-02T19:26:53.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037bc8aee9798f0e8-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.018 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-11-02T19:26:53.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h?versionId=null 2023-11-02T19:26:53.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.022 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.023 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.023 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.023 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.024 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.024 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk= 2023-11-02T19:26:53.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk=', 'Content-Length': '0'} 2023-11-02T19:26:53.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005605f5faea99ea63-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h?versionId=null 2023-11-02T19:26:53.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.029 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.030 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.030 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.030 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.031 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.031 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk= 2023-11-02T19:26:53.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:e3dteMwa8YQ2b/RnDF7TutJf7Tk=', 'Content-Length': '0'} 2023-11-02T19:26:53.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000990a6ad111ae3bef-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=h 2023-11-02T19:26:53.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.036 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.037 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.037 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.037 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.037 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.038 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:53.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:53.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b0c727c30313812-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=h 2023-11-02T19:26:53.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.043 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.044 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.044 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.044 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.045 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.045 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:53.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:53.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026fbc4791a516142-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h 2023-11-02T19:26:53.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.050 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.050 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.051 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.051 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.051 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h 2023-11-02T19:26:53.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.052 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Kk73kx5+YCQDMSm12AEbrNxOB9s= 2023-11-02T19:26:53.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Kk73kx5+YCQDMSm12AEbrNxOB9s=', 'Content-Length': '0'} 2023-11-02T19:26:53.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6563a91d4e86c6c-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h 2023-11-02T19:26:53.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.056 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.057 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.057 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.057 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.058 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h 2023-11-02T19:26:53.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.058 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Kk73kx5+YCQDMSm12AEbrNxOB9s= 2023-11-02T19:26:53.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Kk73kx5+YCQDMSm12AEbrNxOB9s=', 'Content-Length': '0'} 2023-11-02T19:26:53.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b0bf2e3eebf3f4c-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.059 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-11-02T19:26:53.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h 2023-11-02T19:26:53.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.063 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.063 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.064 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.064 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.064 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h 2023-11-02T19:26:53.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.065 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:0YZMn7mt34U/cK43a8pohoZ8+0s= 2023-11-02T19:26:53.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0YZMn7mt34U/cK43a8pohoZ8+0s=', 'Content-Length': '0'} 2023-11-02T19:26:53.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080b25521751ece16-006543e932-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h 2023-11-02T19:26:53.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.069 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.069 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.070 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.070 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.070 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h 2023-11-02T19:26:53.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.071 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:0YZMn7mt34U/cK43a8pohoZ8+0s= 2023-11-02T19:26:53.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0YZMn7mt34U/cK43a8pohoZ8+0s=', 'Content-Length': '0'} 2023-11-02T19:26:53.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003857743809c0fd46-006543e932-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=h 2023-11-02T19:26:53.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.076 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.076 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.077 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.077 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.077 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.078 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:53.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:53.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045256fcba508c0d8-006543e932-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=h 2023-11-02T19:26:53.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.083 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.084 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.084 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.084 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:23:46 GMT 2023-11-02T19:26:53.085 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.085 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM= 2023-11-02T19:26:53.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:23:46 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J1kk7k75gazQ5HJn7K5ALKl4lAM=', 'Content-Length': '0'} 2023-11-02T19:26:53.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076a0ed7ca40d1d69-006543e932-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:23:46 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.086 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=ibtljn-55 zones={a2, a1} 2023-11-02T19:26:53.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'ibtljn-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:53.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'ibtljn-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:53.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket ibtljn-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:53.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:53.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:53.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: tenant: 2023-11-02T19:26:53.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: bucket name: ibtljn-55 2023-11-02T19:26:53.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: total entries: 8 2023-11-02T19:26:53.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:53.090 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:53.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ibtljn-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:53.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:53.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:53.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:53.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "resharding": "None", 2023-11-02T19:26:53.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_index": { 2023-11-02T19:26:53.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 2, 2023-11-02T19:26:53.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "Normal", 2023-11-02T19:26:53.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "normal": { 2023-11-02T19:26:53.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 3, 2023-11-02T19:26:53.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:53.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:53.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "logs": [ 2023-11-02T19:26:53.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:53.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 0, 2023-11-02T19:26:53.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "InIndex", 2023-11-02T19:26:53.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "in_index": { 2023-11-02T19:26:53.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 0, 2023-11-02T19:26:53.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 11, 2023-11-02T19:26:53.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:53.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:53.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:53.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 1, 2023-11-02T19:26:53.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "InIndex", 2023-11-02T19:26:53.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "in_index": { 2023-11-02T19:26:53.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 1, 2023-11-02T19:26:53.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 3, 2023-11-02T19:26:53.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:53.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:53.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:53.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 2, 2023-11-02T19:26:53.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "InIndex", 2023-11-02T19:26:53.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "in_index": { 2023-11-02T19:26:53.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 2, 2023-11-02T19:26:53.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 3, 2023-11-02T19:26:53.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:53.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:53.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=ibtljn-55 2023-11-02T19:26:53.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:53.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:53.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T19:26:53.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:53.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:53.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:47.381+0000 7fc40df17f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T19:26:53.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:52.380+0000 7fc40df17f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T19:26:53.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:23:57.380+0000 7fc40df17f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T19:26:53.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:24:02.380+0000 7fc40df17f40 1 waiting to reach latest gen 2, on 1.. 2023-11-02T19:26:53.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:24:07.384+0000 7fc40df17f40 1 bucket sync caught up with empty source 2023-11-02T19:26:53.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:24:07.384+0000 7fc40df17f40 0 bucket checkpoint complete 2023-11-02T19:26:53.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=ibtljn-55 2023-11-02T19:26:53.113 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:53.113 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:53.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T19:26:53.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:53.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:53.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:24:07.540+0000 7f29e4ef5f40 1 bucket sync caught up with source: 2023-11-02T19:26:53.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T19:26:53.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [, 00000000002.178.6, , , 00000000001.312.6, 00000000001.191.6, , , 00000000002.198.6, , 00000000002.181.6] 2023-11-02T19:26:53.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:24:07.540+0000 7f29e4ef5f40 0 bucket checkpoint complete 2023-11-02T19:26:53.115 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=ibtljn-55 zones={a2, a1} 2023-11-02T19:26:53.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:53.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.119 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.120 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.120 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.120 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.120 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.121 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:D1uof6ZKvDDUN9XXB1PgrLzCjN8= 2023-11-02T19:26:53.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D1uof6ZKvDDUN9XXB1PgrLzCjN8=', 'Content-Length': '0'} 2023-11-02T19:26:53.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029f444051f6ae6c6-006543e947-1665-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/ 2023-11-02T19:26:53.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.126 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.126 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.126 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.126 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.127 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.127 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:D1uof6ZKvDDUN9XXB1PgrLzCjN8= 2023-11-02T19:26:53.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D1uof6ZKvDDUN9XXB1PgrLzCjN8=', 'Content-Length': '0'} 2023-11-02T19:26:53.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000713a767da58a3529-006543e947-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.128 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-11-02T19:26:53.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:53.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:53.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:53.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.133 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.133 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.134 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.134 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.134 INFO:tasks.rgw_multisite_tests:/ibtljn-55/?versions 2023-11-02T19:26:53.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.135 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM= 2023-11-02T19:26:53.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM=', 'Content-Length': '0'} 2023-11-02T19:26:53.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ab5b148523ecc82-006543e947-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.136 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-11-02T19:26:53.137 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-11-02T19:26:53.137 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-11-02T19:26:53.137 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-11-02T19:26:53.138 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-11-02T19:26:53.138 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-11-02T19:26:53.138 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-11-02T19:26:53.139 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-11-02T19:26:53.139 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-11-02T19:26:53.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:53.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:53.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:53.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.143 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.144 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.144 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.144 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.145 INFO:tasks.rgw_multisite_tests:/ibtljn-55/?versions 2023-11-02T19:26:53.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.145 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM= 2023-11-02T19:26:53.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM=', 'Content-Length': '0'} 2023-11-02T19:26:53.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fbc73bd7bb4c5a9-006543e947-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.147 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-11-02T19:26:53.147 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-11-02T19:26:53.147 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-11-02T19:26:53.148 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-11-02T19:26:53.148 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-11-02T19:26:53.148 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-11-02T19:26:53.149 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-11-02T19:26:53.149 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-11-02T19:26:53.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:53.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:53.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:53.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.153 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.154 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.154 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.154 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.155 INFO:tasks.rgw_multisite_tests:/ibtljn-55/?versions 2023-11-02T19:26:53.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.155 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM= 2023-11-02T19:26:53.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM=', 'Content-Length': '0'} 2023-11-02T19:26:53.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ba524541f542c58-006543e947-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:53.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:53.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?versions&max-keys=999 2023-11-02T19:26:53.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.161 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.161 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.161 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.162 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.162 INFO:tasks.rgw_multisite_tests:/ibtljn-55/?versions 2023-11-02T19:26:53.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.162 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM= 2023-11-02T19:26:53.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mqa47naOxZhBKnB4RcNtDpCCyVM=', 'Content-Length': '0'} 2023-11-02T19:26:53.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000695ef0be0aa4853f-006543e947-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55999falseanulltrue2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.164 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-11-02T19:26:53.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:53.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:53.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:53.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.168 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.169 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.169 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.169 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.170 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.170 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc= 2023-11-02T19:26:53.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc=', 'Content-Length': '0'} 2023-11-02T19:26:53.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c7e652e2a51d2e2-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:53.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:53.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:53.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.175 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.176 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.176 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.176 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.177 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.177 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc= 2023-11-02T19:26:53.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc=', 'Content-Length': '0'} 2023-11-02T19:26:53.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1087e880536ee17-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:53.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:53.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:53.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.183 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.183 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.183 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.184 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.184 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.185 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OqWSAtn8UWdJHKBnRZkKT8Btuck= 2023-11-02T19:26:53.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OqWSAtn8UWdJHKBnRZkKT8Btuck=', 'Content-Length': '0'} 2023-11-02T19:26:53.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031d5c35a4ac27e13-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:53.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:53.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:53.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.190 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.190 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.190 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.191 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.191 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.193 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OqWSAtn8UWdJHKBnRZkKT8Btuck= 2023-11-02T19:26:53.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OqWSAtn8UWdJHKBnRZkKT8Btuck=', 'Content-Length': '0'} 2023-11-02T19:26:53.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e5966aac562f846-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.194 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-11-02T19:26:53.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:53.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:53.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:53.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.198 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.198 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.199 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.199 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.199 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.200 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc= 2023-11-02T19:26:53.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc=', 'Content-Length': '0'} 2023-11-02T19:26:53.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f002a7423c5ee766-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:53.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:53.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a?versionId=null 2023-11-02T19:26:53.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.205 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.206 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.206 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.206 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.207 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a?versionId=null 2023-11-02T19:26:53.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.207 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc= 2023-11-02T19:26:53.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mn6AGm6AmLoHQixfBtBNiK3woTc=', 'Content-Length': '0'} 2023-11-02T19:26:53.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006010289309709e45-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=a 2023-11-02T19:26:53.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=a 2023-11-02T19:26:53.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=a 2023-11-02T19:26:53.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.212 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.213 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.213 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.213 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.214 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.214 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw= 2023-11-02T19:26:53.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw=', 'Content-Length': '0'} 2023-11-02T19:26:53.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000104e148aab1521e8-006543e947-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=a 2023-11-02T19:26:53.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=a 2023-11-02T19:26:53.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=a 2023-11-02T19:26:53.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.220 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.220 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.220 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.221 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.221 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.222 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw= 2023-11-02T19:26:53.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw=', 'Content-Length': '0'} 2023-11-02T19:26:53.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d360306bd298ac86-006543e947-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:53.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:53.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:53.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.226 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.227 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.227 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.227 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.227 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:53.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.228 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:zLYVvIEAfo3mVe8DtH4MRmLyNa4= 2023-11-02T19:26:53.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zLYVvIEAfo3mVe8DtH4MRmLyNa4=', 'Content-Length': '0'} 2023-11-02T19:26:53.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e27b4d50903e6cc-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:53.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:53.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:53.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.233 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.233 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.233 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.234 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.234 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:53.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.234 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:zLYVvIEAfo3mVe8DtH4MRmLyNa4= 2023-11-02T19:26:53.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zLYVvIEAfo3mVe8DtH4MRmLyNa4=', 'Content-Length': '0'} 2023-11-02T19:26:53.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc1bd3249d50a69-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.235 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-11-02T19:26:53.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:53.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:53.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:53.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.239 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.240 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.240 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.240 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.240 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:53.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.241 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:9ZvaPMigGoNKNNEfCazQU1Op018= 2023-11-02T19:26:53.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9ZvaPMigGoNKNNEfCazQU1Op018=', 'Content-Length': '0'} 2023-11-02T19:26:53.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba8f92bd65be584-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/a 2023-11-02T19:26:53.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/a 2023-11-02T19:26:53.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/a 2023-11-02T19:26:53.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.245 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.245 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.246 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.246 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.246 INFO:tasks.rgw_multisite_tests:/ibtljn-55/a 2023-11-02T19:26:53.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.247 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:9ZvaPMigGoNKNNEfCazQU1Op018= 2023-11-02T19:26:53.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9ZvaPMigGoNKNNEfCazQU1Op018=', 'Content-Length': '0'} 2023-11-02T19:26:53.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005847dc4aca816fd4-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=a 2023-11-02T19:26:53.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=a 2023-11-02T19:26:53.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=a 2023-11-02T19:26:53.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.252 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.252 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.253 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.253 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.253 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.254 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw= 2023-11-02T19:26:53.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw=', 'Content-Length': '0'} 2023-11-02T19:26:53.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000431d7ee7fe78b280-006543e947-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=a 2023-11-02T19:26:53.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=a 2023-11-02T19:26:53.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=a 2023-11-02T19:26:53.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.259 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.260 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.260 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.260 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.261 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.261 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw= 2023-11-02T19:26:53.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3TOny8n/GvDssNu6ZdUy6dFTtKw=', 'Content-Length': '0'} 2023-11-02T19:26:53.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000861118878080c2ba-006543e947-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55a1000falsea2023-11-02T18:21:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.263 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-11-02T19:26:53.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:53.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:53.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:53.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.267 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.267 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.268 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.268 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.268 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.269 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:PtM0taec9K1xsfdA+le+ESOf4VE= 2023-11-02T19:26:53.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PtM0taec9K1xsfdA+le+ESOf4VE=', 'Content-Length': '0'} 2023-11-02T19:26:53.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022875dea40308a5a-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:53.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:53.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:53.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.274 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.274 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.274 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.275 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.275 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.276 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:PtM0taec9K1xsfdA+le+ESOf4VE= 2023-11-02T19:26:53.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PtM0taec9K1xsfdA+le+ESOf4VE=', 'Content-Length': '0'} 2023-11-02T19:26:53.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b5853339a1d0d53-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:53.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:53.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:53.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.281 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.281 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.281 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.282 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.282 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.283 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:2bNCv/m/y4a2JcvkQwGCIh0eSOA= 2023-11-02T19:26:53.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2bNCv/m/y4a2JcvkQwGCIh0eSOA=', 'Content-Length': '0'} 2023-11-02T19:26:53.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aec6cc9a402db99-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:53.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:53.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:53.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.287 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.288 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.288 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.288 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.288 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.289 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:2bNCv/m/y4a2JcvkQwGCIh0eSOA= 2023-11-02T19:26:53.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2bNCv/m/y4a2JcvkQwGCIh0eSOA=', 'Content-Length': '0'} 2023-11-02T19:26:53.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f48dba733764499-006543e947-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.290 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-11-02T19:26:53.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:53.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:53.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:53.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.294 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.295 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.295 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.295 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:07 GMT 2023-11-02T19:26:53.295 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.296 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:PtM0taec9K1xsfdA+le+ESOf4VE= 2023-11-02T19:26:53.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:07 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PtM0taec9K1xsfdA+le+ESOf4VE=', 'Content-Length': '0'} 2023-11-02T19:26:53.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce4776aeb266fe04-006543e947-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:07 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:53.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:53.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b?versionId=null 2023-11-02T19:26:53.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.301 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.301 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.302 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.302 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.302 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b?versionId=null 2023-11-02T19:26:53.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.303 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:YydcZheNKNJEivvCzNQAJNkzMFY= 2023-11-02T19:26:53.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:YydcZheNKNJEivvCzNQAJNkzMFY=', 'Content-Length': '0'} 2023-11-02T19:26:53.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd03752a67f87372-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=b 2023-11-02T19:26:53.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=b 2023-11-02T19:26:53.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=b 2023-11-02T19:26:53.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.308 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.308 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.309 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.309 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.309 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.310 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000575931eb645550c4-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=b 2023-11-02T19:26:53.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=b 2023-11-02T19:26:53.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=b 2023-11-02T19:26:53.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.315 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.316 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.316 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.316 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.317 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.317 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a00345266d57204d-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:53.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:53.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:53.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.322 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.322 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.323 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.323 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.323 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:53.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.324 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:svPZmLgFeM2S+QTVeBGSQY4xhLI= 2023-11-02T19:26:53.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:svPZmLgFeM2S+QTVeBGSQY4xhLI=', 'Content-Length': '0'} 2023-11-02T19:26:53.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f173d69912501d07-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:53.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:53.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:53.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.328 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.329 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.329 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.329 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.329 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:53.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.330 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:svPZmLgFeM2S+QTVeBGSQY4xhLI= 2023-11-02T19:26:53.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:svPZmLgFeM2S+QTVeBGSQY4xhLI=', 'Content-Length': '0'} 2023-11-02T19:26:53.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002313e8e9d3f17745-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.331 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-11-02T19:26:53.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:53.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:53.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:53.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.335 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.335 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.335 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.336 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.336 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:53.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.337 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:CgaudSY6A/l3oxYzXyeGELhrzqY= 2023-11-02T19:26:53.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CgaudSY6A/l3oxYzXyeGELhrzqY=', 'Content-Length': '0'} 2023-11-02T19:26:53.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008604b393a542b825-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/b 2023-11-02T19:26:53.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/b 2023-11-02T19:26:53.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/b 2023-11-02T19:26:53.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.341 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.341 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.342 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.342 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.342 INFO:tasks.rgw_multisite_tests:/ibtljn-55/b 2023-11-02T19:26:53.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.343 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:CgaudSY6A/l3oxYzXyeGELhrzqY= 2023-11-02T19:26:53.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:CgaudSY6A/l3oxYzXyeGELhrzqY=', 'Content-Length': '0'} 2023-11-02T19:26:53.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b618e95362bd094-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=b 2023-11-02T19:26:53.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=b 2023-11-02T19:26:53.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=b 2023-11-02T19:26:53.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.348 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.348 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.349 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.349 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.349 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.350 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5443adacfdae205-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=b 2023-11-02T19:26:53.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=b 2023-11-02T19:26:53.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=b 2023-11-02T19:26:53.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.355 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.355 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.356 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.356 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.356 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.357 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000542cccef41d328ab-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55b1000falseb2023-11-02T18:21:36.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.358 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-11-02T19:26:53.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:53.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:53.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:53.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.363 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.363 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.363 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.364 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.364 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.365 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko= 2023-11-02T19:26:53.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko=', 'Content-Length': '0'} 2023-11-02T19:26:53.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c01c17fe68693ef6-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:53.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:53.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:53.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.370 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.370 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.370 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.371 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.371 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.372 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko= 2023-11-02T19:26:53.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko=', 'Content-Length': '0'} 2023-11-02T19:26:53.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff05984b94a5873-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:53.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:53.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:53.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.378 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.378 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.378 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.378 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.379 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.379 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:dJooGhNZZPC2CNArUsW9wiaxcoc= 2023-11-02T19:26:53.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dJooGhNZZPC2CNArUsW9wiaxcoc=', 'Content-Length': '0'} 2023-11-02T19:26:53.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005513f1bb4c783090-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:53.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:53.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:53.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.385 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.385 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.385 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.385 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.386 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.387 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:dJooGhNZZPC2CNArUsW9wiaxcoc= 2023-11-02T19:26:53.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dJooGhNZZPC2CNArUsW9wiaxcoc=', 'Content-Length': '0'} 2023-11-02T19:26:53.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c943ba09d5f0bb1-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.387 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-11-02T19:26:53.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:53.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:53.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:53.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.392 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.392 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.393 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.393 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.393 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.394 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko= 2023-11-02T19:26:53.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko=', 'Content-Length': '0'} 2023-11-02T19:26:53.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bd817d340a992da-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:53.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:53.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c?versionId=null 2023-11-02T19:26:53.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.399 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.399 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.399 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.400 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.400 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c?versionId=null 2023-11-02T19:26:53.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.401 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko= 2023-11-02T19:26:53.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vs65B23mF9Qw+vxoWfLWFVAX7ko=', 'Content-Length': '0'} 2023-11-02T19:26:53.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587de4470035b2e3-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=c 2023-11-02T19:26:53.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=c 2023-11-02T19:26:53.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=c 2023-11-02T19:26:53.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.406 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.406 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.407 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.407 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.407 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.408 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bda2044ab9207c2-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=c 2023-11-02T19:26:53.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=c 2023-11-02T19:26:53.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=c 2023-11-02T19:26:53.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.413 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.414 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.414 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.414 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.415 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.415 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005af5147a3453fece-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:53.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:53.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:53.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.420 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.420 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.421 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.421 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.421 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:53.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.422 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:D/ZG6GT72iKVkybZM/co9KOrjXA= 2023-11-02T19:26:53.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/ZG6GT72iKVkybZM/co9KOrjXA=', 'Content-Length': '0'} 2023-11-02T19:26:53.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1c1c18c280fedb3-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:53.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:53.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:53.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.426 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.427 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.427 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.427 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.428 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:53.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.428 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:D/ZG6GT72iKVkybZM/co9KOrjXA= 2023-11-02T19:26:53.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:D/ZG6GT72iKVkybZM/co9KOrjXA=', 'Content-Length': '0'} 2023-11-02T19:26:53.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd0fa749492418ed-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.429 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-11-02T19:26:53.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:53.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:53.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:53.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.433 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.433 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.434 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.434 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.434 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:53.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.435 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:hIxFiRJtKRrybPW3yVkMJthCak0= 2023-11-02T19:26:53.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hIxFiRJtKRrybPW3yVkMJthCak0=', 'Content-Length': '0'} 2023-11-02T19:26:53.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d141061542ec29d8-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/c 2023-11-02T19:26:53.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/c 2023-11-02T19:26:53.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/c 2023-11-02T19:26:53.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.439 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.440 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.440 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.440 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.441 INFO:tasks.rgw_multisite_tests:/ibtljn-55/c 2023-11-02T19:26:53.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.441 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:hIxFiRJtKRrybPW3yVkMJthCak0= 2023-11-02T19:26:53.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hIxFiRJtKRrybPW3yVkMJthCak0=', 'Content-Length': '0'} 2023-11-02T19:26:53.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053bd522c95c7e2da-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=c 2023-11-02T19:26:53.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=c 2023-11-02T19:26:53.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=c 2023-11-02T19:26:53.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.446 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.447 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.447 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.447 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.448 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.448 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbf6c4e51c1928d5-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=c 2023-11-02T19:26:53.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=c 2023-11-02T19:26:53.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=c 2023-11-02T19:26:53.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.454 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.454 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.454 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.455 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.455 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.456 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd218f0357e4f0b3-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55c1000falsec2023-11-02T18:21:36.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.457 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-11-02T19:26:53.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:53.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:53.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:53.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.461 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.462 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.462 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.462 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.463 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.463 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0= 2023-11-02T19:26:53.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0=', 'Content-Length': '0'} 2023-11-02T19:26:53.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e3246f2d4f5af9-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:53.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:53.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:53.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.468 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.468 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.469 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.469 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.469 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.470 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0= 2023-11-02T19:26:53.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0=', 'Content-Length': '0'} 2023-11-02T19:26:53.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da2ade5d00314e27-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:53.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:53.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:53.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.475 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.475 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.476 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.476 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.476 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.477 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OcKhnlKozTwOKRS9lJw7yPFPSGo= 2023-11-02T19:26:53.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OcKhnlKozTwOKRS9lJw7yPFPSGo=', 'Content-Length': '0'} 2023-11-02T19:26:53.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d912410b1692dd5-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:53.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:53.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:53.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.482 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.482 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.483 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.483 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.483 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.484 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OcKhnlKozTwOKRS9lJw7yPFPSGo= 2023-11-02T19:26:53.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OcKhnlKozTwOKRS9lJw7yPFPSGo=', 'Content-Length': '0'} 2023-11-02T19:26:53.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008b88a84743a2265-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.485 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-11-02T19:26:53.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:53.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:53.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:53.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.489 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.490 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.490 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.490 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.491 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.491 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0= 2023-11-02T19:26:53.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0=', 'Content-Length': '0'} 2023-11-02T19:26:53.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1d81bed5ec6134c-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:53.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:53.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d?versionId=null 2023-11-02T19:26:53.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.496 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.496 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.497 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.497 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.497 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d?versionId=null 2023-11-02T19:26:53.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.498 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0= 2023-11-02T19:26:53.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:loJxBl/TixmGC2X93sqq9NtDkj0=', 'Content-Length': '0'} 2023-11-02T19:26:53.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9a73809fc06c0a2-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=d 2023-11-02T19:26:53.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=d 2023-11-02T19:26:53.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=d 2023-11-02T19:26:53.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.503 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.503 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.504 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.504 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.504 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.505 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cca119f396f1c61-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=d 2023-11-02T19:26:53.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=d 2023-11-02T19:26:53.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=d 2023-11-02T19:26:53.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.511 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.511 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.511 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.511 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.512 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.512 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8978c00c87573c3-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:53.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:53.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:53.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.517 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.518 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.518 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.518 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.519 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:53.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.519 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:qfTV4Sj6HH1z6ekJapTYjrD8yuk= 2023-11-02T19:26:53.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qfTV4Sj6HH1z6ekJapTYjrD8yuk=', 'Content-Length': '0'} 2023-11-02T19:26:53.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000412dcbeac6973abc-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:53.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:53.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:53.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.524 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.524 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.524 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.525 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.525 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:53.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.526 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:qfTV4Sj6HH1z6ekJapTYjrD8yuk= 2023-11-02T19:26:53.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qfTV4Sj6HH1z6ekJapTYjrD8yuk=', 'Content-Length': '0'} 2023-11-02T19:26:53.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000347c1232a92dd33a-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.527 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-11-02T19:26:53.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:53.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:53.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:53.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.530 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.531 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.531 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.531 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.532 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:53.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.532 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:bnDobB9W/g/BJKIf9FR7l6wPpmk= 2023-11-02T19:26:53.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bnDobB9W/g/BJKIf9FR7l6wPpmk=', 'Content-Length': '0'} 2023-11-02T19:26:53.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029f9f5250ab37a0c-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/d 2023-11-02T19:26:53.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/d 2023-11-02T19:26:53.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/d 2023-11-02T19:26:53.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.537 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.537 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.537 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.538 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.538 INFO:tasks.rgw_multisite_tests:/ibtljn-55/d 2023-11-02T19:26:53.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.539 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:bnDobB9W/g/BJKIf9FR7l6wPpmk= 2023-11-02T19:26:53.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:bnDobB9W/g/BJKIf9FR7l6wPpmk=', 'Content-Length': '0'} 2023-11-02T19:26:53.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca9df948683d88c7-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=d 2023-11-02T19:26:53.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=d 2023-11-02T19:26:53.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=d 2023-11-02T19:26:53.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.544 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.544 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.544 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.545 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.545 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.546 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7168cbb568b17a1-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=d 2023-11-02T19:26:53.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=d 2023-11-02T19:26:53.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=d 2023-11-02T19:26:53.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.551 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.551 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.552 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.552 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.552 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.553 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e45ee1a890f0582-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55d1000falsed2023-11-02T18:21:36.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.554 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-11-02T19:26:53.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:53.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:53.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e?versionId=null 2023-11-02T19:26:53.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.559 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.559 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.559 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.560 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.560 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.560 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4= 2023-11-02T19:26:53.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4=', 'Content-Length': '0'} 2023-11-02T19:26:53.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002517bb285da23ae7-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:53.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:53.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e?versionId=null 2023-11-02T19:26:53.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.565 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.566 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.566 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.566 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.567 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.567 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4= 2023-11-02T19:26:53.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4=', 'Content-Length': '0'} 2023-11-02T19:26:53.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007581ea57727a4491-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:53.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:53.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e?versionId=null 2023-11-02T19:26:53.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.572 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.573 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.573 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.573 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.574 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.574 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:su2UvNz7iRM8XFNJC1pI7jwcPqI= 2023-11-02T19:26:53.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:su2UvNz7iRM8XFNJC1pI7jwcPqI=', 'Content-Length': '0'} 2023-11-02T19:26:53.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff9df02d35928dfc-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:53.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:53.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e?versionId=null 2023-11-02T19:26:53.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.579 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.580 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.580 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.580 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.580 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.581 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:su2UvNz7iRM8XFNJC1pI7jwcPqI= 2023-11-02T19:26:53.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:su2UvNz7iRM8XFNJC1pI7jwcPqI=', 'Content-Length': '0'} 2023-11-02T19:26:53.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee90d4509286579c-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.582 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-11-02T19:26:53.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:53.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:53.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e?versionId=null 2023-11-02T19:26:53.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.587 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.587 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.587 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.587 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.588 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.588 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4= 2023-11-02T19:26:53.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4=', 'Content-Length': '0'} 2023-11-02T19:26:53.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a5173a349d112a-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:53.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:53.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e?versionId=null 2023-11-02T19:26:53.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.593 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.594 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.594 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.594 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.595 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e?versionId=null 2023-11-02T19:26:53.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.595 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4= 2023-11-02T19:26:53.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Jj3PrYEwzYkUVdAeLHY4D+QfyI4=', 'Content-Length': '0'} 2023-11-02T19:26:53.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdae0416606781c1-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=e 2023-11-02T19:26:53.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=e 2023-11-02T19:26:53.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=e 2023-11-02T19:26:53.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.602 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.602 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.602 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.603 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.603 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.604 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000272924bb5fb0fa84-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=e 2023-11-02T19:26:53.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=e 2023-11-02T19:26:53.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=e 2023-11-02T19:26:53.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.609 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.609 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.609 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.610 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.610 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.611 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c377279af9b91cca-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:53.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:53.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e 2023-11-02T19:26:53.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.615 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.616 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.616 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.616 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.617 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e 2023-11-02T19:26:53.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.617 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:nGuV+17ltnhpNZ+yVxSl78VY2Bk= 2023-11-02T19:26:53.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nGuV+17ltnhpNZ+yVxSl78VY2Bk=', 'Content-Length': '0'} 2023-11-02T19:26:53.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d4ed72270eb063-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:53.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:53.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e 2023-11-02T19:26:53.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.622 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.622 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.622 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.623 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.623 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e 2023-11-02T19:26:53.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.624 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:nGuV+17ltnhpNZ+yVxSl78VY2Bk= 2023-11-02T19:26:53.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nGuV+17ltnhpNZ+yVxSl78VY2Bk=', 'Content-Length': '0'} 2023-11-02T19:26:53.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095abdc1735500a5c-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.624 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-11-02T19:26:53.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:53.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:53.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e 2023-11-02T19:26:53.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.628 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.629 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.629 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.629 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.629 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e 2023-11-02T19:26:53.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.630 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:qN29qHJ4W3l4RsZRXMdEJtf/m3k= 2023-11-02T19:26:53.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qN29qHJ4W3l4RsZRXMdEJtf/m3k=', 'Content-Length': '0'} 2023-11-02T19:26:53.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a18ea6a7b28695d2-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/e 2023-11-02T19:26:53.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/e 2023-11-02T19:26:53.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/e 2023-11-02T19:26:53.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.635 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.635 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.635 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.636 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.636 INFO:tasks.rgw_multisite_tests:/ibtljn-55/e 2023-11-02T19:26:53.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.637 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:qN29qHJ4W3l4RsZRXMdEJtf/m3k= 2023-11-02T19:26:53.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:qN29qHJ4W3l4RsZRXMdEJtf/m3k=', 'Content-Length': '0'} 2023-11-02T19:26:53.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5f9db20155e12cf-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=e 2023-11-02T19:26:53.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=e 2023-11-02T19:26:53.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=e 2023-11-02T19:26:53.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.642 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.642 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.642 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.643 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.643 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.643 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b5e53c1ee2242dd-006543e948-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=e 2023-11-02T19:26:53.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=e 2023-11-02T19:26:53.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=e 2023-11-02T19:26:53.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.649 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.649 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.649 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.650 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.650 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.651 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo= 2023-11-02T19:26:53.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:OGAqKlUtRJrp1QFm0ZKRx9JqRCo=', 'Content-Length': '0'} 2023-11-02T19:26:53.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057b57321345eccbf-006543e948-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55e1000falsee2023-11-02T18:21:59.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.652 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-11-02T19:26:53.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:53.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:53.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f?versionId=null 2023-11-02T19:26:53.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.656 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.657 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.657 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.657 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.658 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.658 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:LG8cifl9QDf0NnndH6/Fv+SS4Ic= 2023-11-02T19:26:53.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LG8cifl9QDf0NnndH6/Fv+SS4Ic=', 'Content-Length': '0'} 2023-11-02T19:26:53.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db630df81acba4db-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:53.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:53.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f?versionId=null 2023-11-02T19:26:53.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.663 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.664 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.664 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.664 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.665 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.665 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:LG8cifl9QDf0NnndH6/Fv+SS4Ic= 2023-11-02T19:26:53.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LG8cifl9QDf0NnndH6/Fv+SS4Ic=', 'Content-Length': '0'} 2023-11-02T19:26:53.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bab75ef9ceaf9f1-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:53.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:53.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f?versionId=null 2023-11-02T19:26:53.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.670 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.670 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.671 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.671 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.671 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.672 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:xw3VTYZ8pw6z09do6F9vm8EW4G4= 2023-11-02T19:26:53.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xw3VTYZ8pw6z09do6F9vm8EW4G4=', 'Content-Length': '0'} 2023-11-02T19:26:53.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d17d9587d1f7959-006543e948-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:53.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:53.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f?versionId=null 2023-11-02T19:26:53.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.677 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.677 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.678 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.678 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:08 GMT 2023-11-02T19:26:53.678 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.679 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:xw3VTYZ8pw6z09do6F9vm8EW4G4= 2023-11-02T19:26:53.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:08 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:xw3VTYZ8pw6z09do6F9vm8EW4G4=', 'Content-Length': '0'} 2023-11-02T19:26:53.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c5961dd0eec1843-006543e948-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:08 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.680 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-11-02T19:26:53.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:53.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:53.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f?versionId=null 2023-11-02T19:26:53.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.684 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.685 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.685 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.685 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.686 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.686 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:H5p+5YTFGaFzyoMtg9LUPLVWbV4= 2023-11-02T19:26:53.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H5p+5YTFGaFzyoMtg9LUPLVWbV4=', 'Content-Length': '0'} 2023-11-02T19:26:53.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007228a80ecdc1fead-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:53.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:53.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f?versionId=null 2023-11-02T19:26:53.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.691 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.692 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.692 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.692 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.693 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f?versionId=null 2023-11-02T19:26:53.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.693 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:H5p+5YTFGaFzyoMtg9LUPLVWbV4= 2023-11-02T19:26:53.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:H5p+5YTFGaFzyoMtg9LUPLVWbV4=', 'Content-Length': '0'} 2023-11-02T19:26:53.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a34c45813bc3327-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=f 2023-11-02T19:26:53.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=f 2023-11-02T19:26:53.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=f 2023-11-02T19:26:53.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.698 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.699 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.699 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.699 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.700 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.700 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T19:26:53.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T19:26:53.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd812aef20526b6d-006543e949-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=f 2023-11-02T19:26:53.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=f 2023-11-02T19:26:53.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=f 2023-11-02T19:26:53.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.706 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.706 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.706 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.707 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.707 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.708 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T19:26:53.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T19:26:53.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cc67d09c35e5d2f-006543e949-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:53.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:53.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f 2023-11-02T19:26:53.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.712 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.713 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.713 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.713 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.713 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f 2023-11-02T19:26:53.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.714 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:6JXtSCcmhdNtp/N1WBXCRxSaoJM= 2023-11-02T19:26:53.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6JXtSCcmhdNtp/N1WBXCRxSaoJM=', 'Content-Length': '0'} 2023-11-02T19:26:53.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf618cab3d313d92-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:53.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:53.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f 2023-11-02T19:26:53.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.718 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.719 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.719 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.719 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.720 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f 2023-11-02T19:26:53.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.720 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:6JXtSCcmhdNtp/N1WBXCRxSaoJM= 2023-11-02T19:26:53.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6JXtSCcmhdNtp/N1WBXCRxSaoJM=', 'Content-Length': '0'} 2023-11-02T19:26:53.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f74edd0657ea0730-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.721 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-11-02T19:26:53.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:53.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:53.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f 2023-11-02T19:26:53.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.725 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.725 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.726 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.726 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.726 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f 2023-11-02T19:26:53.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.727 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:K4+LqtNTBjSicoEVxivsYDgo/og= 2023-11-02T19:26:53.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:K4+LqtNTBjSicoEVxivsYDgo/og=', 'Content-Length': '0'} 2023-11-02T19:26:53.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028c2a195e0f32477-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/f 2023-11-02T19:26:53.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/f 2023-11-02T19:26:53.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/f 2023-11-02T19:26:53.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.731 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.732 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.732 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.732 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.733 INFO:tasks.rgw_multisite_tests:/ibtljn-55/f 2023-11-02T19:26:53.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.733 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:K4+LqtNTBjSicoEVxivsYDgo/og= 2023-11-02T19:26:53.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:K4+LqtNTBjSicoEVxivsYDgo/og=', 'Content-Length': '0'} 2023-11-02T19:26:53.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004705a6e56c0afe49-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=f 2023-11-02T19:26:53.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=f 2023-11-02T19:26:53.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=f 2023-11-02T19:26:53.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.738 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.738 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.739 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.739 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.739 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.740 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T19:26:53.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T19:26:53.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bda91ddf5d2ed0d6-006543e949-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=f 2023-11-02T19:26:53.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=f 2023-11-02T19:26:53.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=f 2023-11-02T19:26:53.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.745 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.746 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.746 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.746 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.746 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.747 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T19:26:53.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T19:26:53.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c77a1f7d2770820-006543e949-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55f1000falsef2023-11-02T18:21:59.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.748 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-11-02T19:26:53.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:53.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:53.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g?versionId=null 2023-11-02T19:26:53.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.753 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.753 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.753 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.754 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.754 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.755 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk= 2023-11-02T19:26:53.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk=', 'Content-Length': '0'} 2023-11-02T19:26:53.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7441279cf248e70-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:53.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:53.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g?versionId=null 2023-11-02T19:26:53.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.760 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.760 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.760 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.761 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.761 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.762 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk= 2023-11-02T19:26:53.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk=', 'Content-Length': '0'} 2023-11-02T19:26:53.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b382b3a7c5dc8c1-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:53.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:53.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g?versionId=null 2023-11-02T19:26:53.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.767 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.767 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.767 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.768 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.768 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.769 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:6C3UgQ5IUl2YJl+QSiXs9U7msco= 2023-11-02T19:26:53.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6C3UgQ5IUl2YJl+QSiXs9U7msco=', 'Content-Length': '0'} 2023-11-02T19:26:53.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000ef480fb70b0596-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:53.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:53.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g?versionId=null 2023-11-02T19:26:53.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.774 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.774 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.774 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.775 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.775 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.776 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:6C3UgQ5IUl2YJl+QSiXs9U7msco= 2023-11-02T19:26:53.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:6C3UgQ5IUl2YJl+QSiXs9U7msco=', 'Content-Length': '0'} 2023-11-02T19:26:53.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5d8aa78500287d-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.777 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-11-02T19:26:53.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:53.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:53.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g?versionId=null 2023-11-02T19:26:53.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.781 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.781 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.782 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.782 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.782 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.783 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk= 2023-11-02T19:26:53.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk=', 'Content-Length': '0'} 2023-11-02T19:26:53.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6b4f9696c006e5-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:53.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:53.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g?versionId=null 2023-11-02T19:26:53.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.788 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.788 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.789 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.789 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.789 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g?versionId=null 2023-11-02T19:26:53.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.790 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk= 2023-11-02T19:26:53.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:EF5k4WwNFok55lgmwZRnbYWEvPk=', 'Content-Length': '0'} 2023-11-02T19:26:53.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000347551c232c0e2d4-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=g 2023-11-02T19:26:53.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=g 2023-11-02T19:26:53.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=g 2023-11-02T19:26:53.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.795 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.795 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.796 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.796 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.796 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.797 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T19:26:53.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T19:26:53.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039b13d595fa146d8-006543e949-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=g 2023-11-02T19:26:53.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=g 2023-11-02T19:26:53.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=g 2023-11-02T19:26:53.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.802 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.803 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.803 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.803 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.804 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.805 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T19:26:53.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T19:26:53.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f3472495ba71497-006543e949-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:53.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:53.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g 2023-11-02T19:26:53.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.810 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.811 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.811 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.811 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.812 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g 2023-11-02T19:26:53.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.812 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:PZH0cfwKnUA3UOpTJgUKxtlWyro= 2023-11-02T19:26:53.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PZH0cfwKnUA3UOpTJgUKxtlWyro=', 'Content-Length': '0'} 2023-11-02T19:26:53.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a00d83252e166f4-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:53.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:53.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g 2023-11-02T19:26:53.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.817 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.817 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.817 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.818 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.818 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g 2023-11-02T19:26:53.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.818 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:PZH0cfwKnUA3UOpTJgUKxtlWyro= 2023-11-02T19:26:53.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:PZH0cfwKnUA3UOpTJgUKxtlWyro=', 'Content-Length': '0'} 2023-11-02T19:26:53.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e80a71d46b486513-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.819 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-11-02T19:26:53.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:53.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:53.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g 2023-11-02T19:26:53.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.823 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.824 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.824 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.824 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.824 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g 2023-11-02T19:26:53.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.825 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:mW172XeXFFY74UKjjfyv/1BWQtk= 2023-11-02T19:26:53.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mW172XeXFFY74UKjjfyv/1BWQtk=', 'Content-Length': '0'} 2023-11-02T19:26:53.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ede57073ace1620a-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/g 2023-11-02T19:26:53.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/g 2023-11-02T19:26:53.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/g 2023-11-02T19:26:53.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.830 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.830 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.830 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.831 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.831 INFO:tasks.rgw_multisite_tests:/ibtljn-55/g 2023-11-02T19:26:53.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.831 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:mW172XeXFFY74UKjjfyv/1BWQtk= 2023-11-02T19:26:53.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:mW172XeXFFY74UKjjfyv/1BWQtk=', 'Content-Length': '0'} 2023-11-02T19:26:53.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003028c3b2c7eddf70-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=g 2023-11-02T19:26:53.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=g 2023-11-02T19:26:53.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=g 2023-11-02T19:26:53.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.836 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.837 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.837 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.837 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.838 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.838 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T19:26:53.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T19:26:53.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008add80fe7d0bdf0c-006543e949-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=g 2023-11-02T19:26:53.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=g 2023-11-02T19:26:53.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=g 2023-11-02T19:26:53.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.844 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.844 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.844 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.845 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.845 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.846 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T19:26:53.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T19:26:53.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c8cc6f92a844417-006543e949-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55g1000falseg2023-11-02T18:21:59.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.847 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-11-02T19:26:53.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h?versionId=null 2023-11-02T19:26:53.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.851 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.852 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.852 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.852 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.853 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.853 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY= 2023-11-02T19:26:53.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY=', 'Content-Length': '0'} 2023-11-02T19:26:53.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000832c9a393dd21191-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h?versionId=null 2023-11-02T19:26:53.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.858 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.858 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.859 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.859 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.859 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.860 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY= 2023-11-02T19:26:53.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY=', 'Content-Length': '0'} 2023-11-02T19:26:53.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7b5143c23b64d26-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h?versionId=null 2023-11-02T19:26:53.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.865 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.865 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.866 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.866 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.866 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.867 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:0DR3vCYFZr+I7l81PbYsu9Kw+QM= 2023-11-02T19:26:53.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0DR3vCYFZr+I7l81PbYsu9Kw+QM=', 'Content-Length': '0'} 2023-11-02T19:26:53.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f49e315e29012c38-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h?versionId=null 2023-11-02T19:26:53.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.872 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.873 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.873 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.873 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.874 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.874 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:0DR3vCYFZr+I7l81PbYsu9Kw+QM= 2023-11-02T19:26:53.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0DR3vCYFZr+I7l81PbYsu9Kw+QM=', 'Content-Length': '0'} 2023-11-02T19:26:53.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005688bf24fd588f0d-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.875 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-11-02T19:26:53.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h?versionId=null 2023-11-02T19:26:53.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.880 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.880 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.880 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.881 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.881 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.882 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY= 2023-11-02T19:26:53.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY=', 'Content-Length': '0'} 2023-11-02T19:26:53.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1423a5180ff42a8-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h?versionId=null 2023-11-02T19:26:53.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.887 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.887 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.887 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.888 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.888 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h?versionId=null 2023-11-02T19:26:53.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.888 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY= 2023-11-02T19:26:53.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2BP/dsavKt7J/i/eNq/Ho69LoGY=', 'Content-Length': '0'} 2023-11-02T19:26:53.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a30270155e11d976-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=h 2023-11-02T19:26:53.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.894 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.894 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.894 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.894 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.895 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.895 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T19:26:53.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T19:26:53.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004921f3e983a24f9b-006543e949-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=h 2023-11-02T19:26:53.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.901 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.901 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.901 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.902 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.902 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.903 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T19:26:53.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T19:26:53.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c1c992422a30512-006543e949-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h 2023-11-02T19:26:53.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.907 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.907 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.908 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.908 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.908 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h 2023-11-02T19:26:53.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.909 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:GdQAIQYhsstIIsb0AZ+xbmZuZb0= 2023-11-02T19:26:53.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GdQAIQYhsstIIsb0AZ+xbmZuZb0=', 'Content-Length': '0'} 2023-11-02T19:26:53.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b67315fba1ad0b2-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:53.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h 2023-11-02T19:26:53.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.913 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:53.914 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.914 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.914 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.915 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h 2023-11-02T19:26:53.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.915 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:GdQAIQYhsstIIsb0AZ+xbmZuZb0= 2023-11-02T19:26:53.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GdQAIQYhsstIIsb0AZ+xbmZuZb0=', 'Content-Length': '0'} 2023-11-02T19:26:53.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000804f63bdb6ff1c6e-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.916 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-11-02T19:26:53.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h 2023-11-02T19:26:53.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.920 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.920 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.921 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.921 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.921 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h 2023-11-02T19:26:53.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.922 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:8kH4kdXAxv4spFGgabodqcmIhcA= 2023-11-02T19:26:53.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8kH4kdXAxv4spFGgabodqcmIhcA=', 'Content-Length': '0'} 2023-11-02T19:26:53.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd61b2bea-59a8-4454-8491-ac3afbd04937:ibtljn-55:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5c983f7aacad558-006543e949-1665-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/h 2023-11-02T19:26:53.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/h 2023-11-02T19:26:53.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/h 2023-11-02T19:26:53.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.926 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.927 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.927 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.927 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.927 INFO:tasks.rgw_multisite_tests:/ibtljn-55/h 2023-11-02T19:26:53.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.928 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:8kH4kdXAxv4spFGgabodqcmIhcA= 2023-11-02T19:26:53.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8kH4kdXAxv4spFGgabodqcmIhcA=', 'Content-Length': '0'} 2023-11-02T19:26:53.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 02 Nov 2023 18:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000616dede86a86293d-006543e949-1669-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=h 2023-11-02T19:26:53.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:53.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:53.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.933 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.934 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.934 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.934 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.935 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.935 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T19:26:53.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T19:26:53.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d5e707230d82ce8-006543e949-1665-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/ 2023-11-02T19:26:53.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/ 2023-11-02T19:26:53.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-55/?prefix=h 2023-11-02T19:26:53.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-11-02T19:26:53.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-55/?prefix=h 2023-11-02T19:26:53.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.941 INFO:tasks.rgw_multisite_tests:GET 2023-11-02T19:26:53.941 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.941 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.942 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:24:09 GMT 2023-11-02T19:26:53.942 INFO:tasks.rgw_multisite_tests:/ibtljn-55/ 2023-11-02T19:26:53.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.943 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8= 2023-11-02T19:26:53.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:24:09 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jY858yfBR/PBvVUCR5B0AIj9TM8=', 'Content-Length': '0'} 2023-11-02T19:26:53.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7020fdcd4e438a-006543e949-1669-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 18:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ibtljn-55h1000falseh2023-11-02T18:21:59.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-11-02T19:26:53.944 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=ibtljn-55 zones={a2, a1} 2023-11-02T19:26:53.944 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:53.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:53.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:53.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:53.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:53.946 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:09.860+0000 7f6d28b57700 -1 req 6474304447377707186 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-19:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.19,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-19:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.19,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.946 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:09.864+0000 7f6d6cbdf700 -1 req 9826087319285350072 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-28:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.28,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-28:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.28,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.947 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:09.872+0000 7f6d67bd5700 -1 req 13240611044736563099 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-27:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.27,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-27:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.27,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.947 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:09.884+0000 7f6cfd300700 -1 req 11955232559906362647 0.003999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-43:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.8,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-43:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.8,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.947 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:09.884+0000 7f6d12b2b700 -1 req 14260232747133502469 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-29:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.29,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-29:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.29,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.947 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:09.884+0000 7f6de1cc9700 -1 req 976322668675659743 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-3:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.3,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-3:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.3,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.948 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:09.888+0000 7f6cf3aed700 -1 req 15793413793908589666 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-42:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.7,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-42:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.7,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.948 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:09.888+0000 7f6da6452700 -1 req 2843339270896427370 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-2:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.2,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-2:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.2,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.948 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:09.888+0000 7f6d1ab3b700 -1 req 3204166254874922139 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-36:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.1,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-36:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.1,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:53.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:53.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ibtljn-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:53.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:53.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:53.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:53.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "resharding": "None", 2023-11-02T19:26:53.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_index": { 2023-11-02T19:26:53.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 2, 2023-11-02T19:26:53.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "Normal", 2023-11-02T19:26:53.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "normal": { 2023-11-02T19:26:53.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 3, 2023-11-02T19:26:53.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:53.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:53.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "logs": [ 2023-11-02T19:26:53.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:53.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 1, 2023-11-02T19:26:53.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "InIndex", 2023-11-02T19:26:53.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "in_index": { 2023-11-02T19:26:53.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 1, 2023-11-02T19:26:53.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 3, 2023-11-02T19:26:53.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:53.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:53.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:53.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 2, 2023-11-02T19:26:53.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "InIndex", 2023-11-02T19:26:53.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "in_index": { 2023-11-02T19:26:53.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 2, 2023-11-02T19:26:53.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 3, 2023-11-02T19:26:53.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:53.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:53.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.964 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:53.964 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:53.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:53.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:53.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:53.965 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:10.268+0000 7f6dce4a2700 -1 req 6348831972394980737 0.003999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-20:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.20,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-20:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.20,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.966 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:10.268+0000 7f6dcac9b700 -1 req 10187714122627611485 0.003999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-41:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.6,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-41:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.6,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.966 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:10.272+0000 7f6dc9c99700 -1 req 16383045101260734179 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-21:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.21,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-21:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.21,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.966 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:10.276+0000 7f6de3ccd700 -1 req 13267280008896322252 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-1:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.1,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-1:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.1,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.967 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:10.276+0000 7f6dc0486700 -1 req 8184490768704125574 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-4:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.4,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-4:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.4,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.967 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:10.276+0000 7f6d8ec23700 -1 req 13835577623689498269 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-5:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.5,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-5:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.5,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.967 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:10.276+0000 7f6d8fc25700 -1 req 5231040344823808504 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-40:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.5,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-40:d61b2bea-59a8-4454-8491-ac3afbd04937.5739.5,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.968 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:10.280+0000 7f6dd3cad700 -1 req 17837592990121390711 0.003999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-25:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.25,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-25:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.25,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.968 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi017.stdout: INFO: 2023-11-02T18:24:10.280+0000 7f6d8a41a700 -1 req 13427702441716128958 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=ibtljn-22:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.22,z=d61b2bea-59a8-4454-8491-ac3afbd04937,az=0},d={b=ibtljn-22:d61b2bea-59a8-4454-8491-ac3afbd04937.4371.22,z=39a01552-6d59-4ce7-b7e5-915ca69b979a,az=0}} returned ret=-2 2023-11-02T19:26:53.968 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:53.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'ibtljn-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:53.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket ibtljn-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:53.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:53.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:53.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:53.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "resharding": "None", 2023-11-02T19:26:53.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_index": { 2023-11-02T19:26:53.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 2, 2023-11-02T19:26:53.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "Normal", 2023-11-02T19:26:53.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "normal": { 2023-11-02T19:26:53.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 3, 2023-11-02T19:26:53.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:53.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:53.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "logs": [ 2023-11-02T19:26:53.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:53.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 1, 2023-11-02T19:26:53.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "InIndex", 2023-11-02T19:26:53.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "in_index": { 2023-11-02T19:26:53.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 1, 2023-11-02T19:26:53.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 3, 2023-11-02T19:26:53.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:53.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:53.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:53.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 2, 2023-11-02T19:26:53.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "type": "InIndex", 2023-11-02T19:26:53.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "in_index": { 2023-11-02T19:26:53.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "gen": 2, 2023-11-02T19:26:53.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "layout": { 2023-11-02T19:26:53.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 3, 2023-11-02T19:26:53.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hash_type": "Mod" 2023-11-02T19:26:53.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:53.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:53.983 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-11-02T19:26:53.984 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.984 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-11-02T19:26:53.985 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-11-02T19:26:53.985 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-11-02T19:26:53.986 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-11-02T19:26:53.986 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-11-02T19:26:53.986 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-11-02T19:26:53.987 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/../src/test/rgw/rgw_multi/tests.py", line 1842, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-11-02T19:26:53.987 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-11-02T19:26:53.987 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-11-02T19:26:53.988 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-11-02T19:26:53.988 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-11-02T19:26:53.988 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-11-02T19:26:53.989 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/tasks/rgw_multisite.py", line 197, in admin 2023-11-02T19:26:53.989 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-11-02T19:26:53.989 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-11-02T19:26:53.990 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-11-02T19:26:53.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:53.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:53.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:53.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:53.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:53.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:53.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:53.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:53.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:53.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:53.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:53.993 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:53.994 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.994 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:53.994 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:57:54 GMT 2023-11-02T19:26:53.995 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:53.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:53.995 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:WmCbU+e/xPXfoJLNIbat4jj3Lxc= 2023-11-02T19:26:53.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:54 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:WmCbU+e/xPXfoJLNIbat4jj3Lxc=', 'Content-Length': '0'} 2023-11-02T19:26:53.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000141d599255523221-006543f132-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:57:54 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:53.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=ibtljn-63 2023-11-02T19:26:53.997 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:53.997 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:53.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:53.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:53.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:53.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:26:53.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: [ 2023-11-02T19:26:53.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:53.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:26:54.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:26:54.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698951474.369610_580.1", 2023-11-02T19:26:54.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T18:57:54.369610Z" 2023-11-02T19:26:54.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:26:54.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951474.369610_580.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-11-02T19:26:54.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:54.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:57:54.661+0000 7f06b38c7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:57:54.661+0000 7f06b38c7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:54.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:54.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:54.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:54.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:54.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:54.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 1, 2023-11-02T19:26:54.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:54.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:54.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698951474.358322_577.1", 2023-11-02T19:26:54.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:57:54.358322Z", 2023-11-02T19:26:54.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:54.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:54.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 1, 2023-11-02T19:26:54.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 1 2023-11-02T19:26:54.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T18:57:54Z" 2023-11-02T19:26:54.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-11-02T19:26:54.028 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:54.028 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:54.028 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:54.029 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:54.029 INFO:tasks.rgw_multisite_tests: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.029 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.030 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.030 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:54.030 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.031 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:54.031 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.031 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.031 INFO:tasks.rgw_multisite_tests: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.032 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.032 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-11-02T19:26:54.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.033 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.033 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.033 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.034 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.034 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.034 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:54.035 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.035 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.035 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.036 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.037 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.037 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.038 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.038 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:54.038 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1698951474.358322_577.1", 2023-11-02T19:26:54.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.040 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:57:54.358322Z", 2023-11-02T19:26:54.040 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.041 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.041 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.041 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.042 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:54.042 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.042 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.043 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.043 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.043 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.043 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.044 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.044 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.044 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.045 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.045 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:54.045 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.046 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:54.046 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-11-02T19:26:54.046 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-11-02T19:26:54.046 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.047 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T18:57:54Z" 2023-11-02T19:26:54.047 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:54.047 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951474.358322_577.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:54.358322Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:26:54.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951474.369610_580.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951474.358322_577.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1698951474.369610_580.1 target=1_1698951474.358322_577.1 2023-11-02T19:26:54.049 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.049 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:54.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:57:59.813+0000 7f851d3fff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:57:59.813+0000 7f851d3fff40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:54.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:54.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:54.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:54.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:54.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:54.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 1, 2023-11-02T19:26:54.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:54.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:54.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698951474.369610_580.1", 2023-11-02T19:26:54.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:57:54.369610Z", 2023-11-02T19:26:54.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:54.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:54.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 1, 2023-11-02T19:26:54.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 1 2023-11-02T19:26:54.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T18:57:59Z" 2023-11-02T19:26:54.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-11-02T19:26:54.072 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:54.072 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:54.073 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:54.073 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:54.073 INFO:tasks.rgw_multisite_tests: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.074 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.074 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:54.075 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.075 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:54.075 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.075 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.076 INFO:tasks.rgw_multisite_tests: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.076 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.076 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-11-02T19:26:54.077 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.077 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.077 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.078 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.078 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.078 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.079 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:54.079 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.079 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.080 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.080 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.080 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.080 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.081 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.081 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.081 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.082 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.082 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.082 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:54.083 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.083 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.083 INFO:tasks.rgw_multisite_tests: "marker": "1_1698951474.369610_580.1", 2023-11-02T19:26:54.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.084 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.084 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.084 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:57:54.369610Z", 2023-11-02T19:26:54.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.085 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.085 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.086 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.086 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:54.086 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.087 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.087 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.088 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.088 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.088 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.089 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.089 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.089 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.089 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:54.090 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.090 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:54.090 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-11-02T19:26:54.091 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-11-02T19:26:54.091 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.091 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T18:57:59Z" 2023-11-02T19:26:54.092 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:54.092 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698951474.369610_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:57:54.369610Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:26:54.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951474.369610_580.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698951474.369610_580.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-11-02T19:26:54.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.097 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.097 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.098 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.098 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:57:59 GMT 2023-11-02T19:26:54.098 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.099 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE= 2023-11-02T19:26:54.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE=', 'Content-Length': '0'} 2023-11-02T19:26:54.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf85ebf584783ad1-006543f137-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj11 2023-11-02T19:26:54.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj11 2023-11-02T19:26:54.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj11 2023-11-02T19:26:54.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.103 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.104 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.104 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.104 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:57:59 GMT 2023-11-02T19:26:54.105 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj11 2023-11-02T19:26:54.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.105 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:VY5LUhHg3nx363zhtOjm56h843w= 2023-11-02T19:26:54.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:VY5LUhHg3nx363zhtOjm56h843w='} 2023-11-02T19:26:54.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c8d91b6527e39d8-006543f137-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.110 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.110 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.110 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.110 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:57:59 GMT 2023-11-02T19:26:54.111 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.111 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE= 2023-11-02T19:26:54.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE=', 'Content-Length': '0'} 2023-11-02T19:26:54.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b6cb921502a686f-006543f137-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj22 2023-11-02T19:26:54.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj22 2023-11-02T19:26:54.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj22 2023-11-02T19:26:54.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.115 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.116 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.116 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.116 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:57:59 GMT 2023-11-02T19:26:54.117 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj22 2023-11-02T19:26:54.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.117 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Y7x3pQcT13BUTT2aL5hmt1YyiEQ= 2023-11-02T19:26:54.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Y7x3pQcT13BUTT2aL5hmt1YyiEQ='} 2023-11-02T19:26:54.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007432a0e8fefb7dc8-006543f137-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.122 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.122 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.122 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.123 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:57:59 GMT 2023-11-02T19:26:54.123 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.124 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE= 2023-11-02T19:26:54.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE=', 'Content-Length': '0'} 2023-11-02T19:26:54.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000474aa5c9ce6b5877-006543f137-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj33 2023-11-02T19:26:54.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj33 2023-11-02T19:26:54.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj33 2023-11-02T19:26:54.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.128 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.128 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.129 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.129 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:57:59 GMT 2023-11-02T19:26:54.130 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj33 2023-11-02T19:26:54.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.130 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:czlt7VanvPNrm34+ZKqUbnP/tj4= 2023-11-02T19:26:54.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:czlt7VanvPNrm34+ZKqUbnP/tj4='} 2023-11-02T19:26:54.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f2d6bece15d33ca-006543f137-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.135 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.135 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.135 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.136 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:57:59 GMT 2023-11-02T19:26:54.136 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.137 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE= 2023-11-02T19:26:54.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:57:59 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RGy4hiv4xhcWRdMJdsHqJQfA2NE=', 'Content-Length': '0'} 2023-11-02T19:26:54.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094ffb2f26a7e35f9-006543f137-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:57:59 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj44 2023-11-02T19:26:54.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj44 2023-11-02T19:26:54.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj44 2023-11-02T19:26:54.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.141 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.141 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.142 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.142 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.142 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj44 2023-11-02T19:26:54.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.143 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Vinwwq86xjYYeLuu8r2FPiJ0d5Y= 2023-11-02T19:26:54.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Vinwwq86xjYYeLuu8r2FPiJ0d5Y='} 2023-11-02T19:26:54.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2b9191d215e8204-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.147 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.148 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.148 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.148 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.148 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.149 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T19:26:54.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T19:26:54.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000660b26b6b40a452a-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj55 2023-11-02T19:26:54.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj55 2023-11-02T19:26:54.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj55 2023-11-02T19:26:54.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.153 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.154 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.154 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.154 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.155 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj55 2023-11-02T19:26:54.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.155 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:SPnClwpAaby1yEAsouobAvdM1rg= 2023-11-02T19:26:54.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:SPnClwpAaby1yEAsouobAvdM1rg='} 2023-11-02T19:26:54.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e73933bd1c093d1-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.156 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.157 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: tenant: 2023-11-02T19:26:54.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: bucket name: ibtljn-63 2023-11-02T19:26:54.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: total entries: 5 2023-11-02T19:26:54.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.162 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.163 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.163 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.163 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.163 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.164 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T19:26:54.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T19:26:54.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dec164db5a86eeed-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj17 2023-11-02T19:26:54.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj17 2023-11-02T19:26:54.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj17 2023-11-02T19:26:54.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.168 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.169 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.169 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.169 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.170 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj17 2023-11-02T19:26:54.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.170 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:HF5Ac4/n28U5nHDB+ir0G266mRk= 2023-11-02T19:26:54.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:HF5Ac4/n28U5nHDB+ir0G266mRk='} 2023-11-02T19:26:54.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014ce0e30f4d38427-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.175 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.175 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.175 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.176 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.176 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.176 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T19:26:54.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T19:26:54.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000aea9e71b6e62e27-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj34 2023-11-02T19:26:54.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj34 2023-11-02T19:26:54.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj34 2023-11-02T19:26:54.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.181 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.181 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.181 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.182 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.182 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj34 2023-11-02T19:26:54.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.183 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:vtmtirGUHwcyx2uqexTTJYT4vBQ= 2023-11-02T19:26:54.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vtmtirGUHwcyx2uqexTTJYT4vBQ='} 2023-11-02T19:26:54.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096a4c9acded83912-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.187 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.187 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.188 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.188 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.188 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.189 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T19:26:54.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T19:26:54.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa2980caece9d769-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj51 2023-11-02T19:26:54.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj51 2023-11-02T19:26:54.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj51 2023-11-02T19:26:54.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.193 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.193 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.194 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.194 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.194 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj51 2023-11-02T19:26:54.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.195 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:DBAzwEYRpiASOabBuOoRrHnGCfs= 2023-11-02T19:26:54.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:DBAzwEYRpiASOabBuOoRrHnGCfs='} 2023-11-02T19:26:54.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000918070cec8dfbbc-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.200 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.200 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.200 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.200 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.201 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.201 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T19:26:54.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T19:26:54.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007029686c1c692e82-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj68 2023-11-02T19:26:54.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj68 2023-11-02T19:26:54.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj68 2023-11-02T19:26:54.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.206 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.206 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.206 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.207 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.207 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj68 2023-11-02T19:26:54.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.208 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:UuNFSdLnvgEY1g0rCJHDu4C6/b8= 2023-11-02T19:26:54.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:UuNFSdLnvgEY1g0rCJHDu4C6/b8='} 2023-11-02T19:26:54.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004907d37e1d7deded-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.215 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.215 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.215 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.216 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.216 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.217 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T19:26:54.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T19:26:54.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022a4f4946cd7159c-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj85 2023-11-02T19:26:54.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj85 2023-11-02T19:26:54.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj85 2023-11-02T19:26:54.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.221 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.221 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.222 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.222 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.222 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj85 2023-11-02T19:26:54.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.223 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:j6tpHHsMvN2podymvT5y/Zbzi7A= 2023-11-02T19:26:54.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:j6tpHHsMvN2podymvT5y/Zbzi7A='} 2023-11-02T19:26:54.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000339ea9f31cd5317c-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.224 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.224 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: tenant: 2023-11-02T19:26:54.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: bucket name: ibtljn-63 2023-11-02T19:26:54.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: total entries: 10 2023-11-02T19:26:54.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.230 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.230 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.231 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.231 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.231 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.232 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T19:26:54.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T19:26:54.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000994fd4da2b01bf76-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj19 2023-11-02T19:26:54.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj19 2023-11-02T19:26:54.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj19 2023-11-02T19:26:54.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.236 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.236 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.237 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.237 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.237 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj19 2023-11-02T19:26:54.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.238 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:2bMbGTpeObSfnAPILFVWb8FqONg= 2023-11-02T19:26:54.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:2bMbGTpeObSfnAPILFVWb8FqONg='} 2023-11-02T19:26:54.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5b89cf6dfb239ed-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.243 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.243 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.243 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.244 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.244 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.245 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T19:26:54.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T19:26:54.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e78ec5ccfe89cd9-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj38 2023-11-02T19:26:54.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj38 2023-11-02T19:26:54.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj38 2023-11-02T19:26:54.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.249 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.249 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.250 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.250 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.250 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj38 2023-11-02T19:26:54.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.251 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:hsm2mlFB643Pcs4a8LKGRccW4Oo= 2023-11-02T19:26:54.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:hsm2mlFB643Pcs4a8LKGRccW4Oo='} 2023-11-02T19:26:54.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e86c23d3059b1488-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.256 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.256 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.256 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.257 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.257 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.258 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T19:26:54.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T19:26:54.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003065be418dc5385f-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj57 2023-11-02T19:26:54.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj57 2023-11-02T19:26:54.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj57 2023-11-02T19:26:54.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.263 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.263 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.263 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.264 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.264 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj57 2023-11-02T19:26:54.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.265 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:BpPIhsUn6yPe3RcNqnMsWEBYK5I= 2023-11-02T19:26:54.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:BpPIhsUn6yPe3RcNqnMsWEBYK5I='} 2023-11-02T19:26:54.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d545f036f0270026-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.269 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.270 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.270 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.270 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.271 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.271 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T19:26:54.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T19:26:54.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3590b4ce8897701-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj76 2023-11-02T19:26:54.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj76 2023-11-02T19:26:54.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj76 2023-11-02T19:26:54.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.276 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.276 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.276 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.277 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.277 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj76 2023-11-02T19:26:54.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.278 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:5fdzGslu5QGrFAgnd3wwqoCEaZM= 2023-11-02T19:26:54.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:5fdzGslu5QGrFAgnd3wwqoCEaZM='} 2023-11-02T19:26:54.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cd76f879459348c-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.282 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.282 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.283 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.283 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.283 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.284 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0= 2023-11-02T19:26:54.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:vfDrdT7uxrdw3MJsIp9/4+Haya0=', 'Content-Length': '0'} 2023-11-02T19:26:54.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a5e6a348fa43f6e-006543f138-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:00 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj95 2023-11-02T19:26:54.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj95 2023-11-02T19:26:54.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj95 2023-11-02T19:26:54.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.288 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.289 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.289 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.289 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:00 GMT 2023-11-02T19:26:54.290 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj95 2023-11-02T19:26:54.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.290 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:INSngDeiqoVHTu2xtecIMp9sHj0= 2023-11-02T19:26:54.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:00 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:INSngDeiqoVHTu2xtecIMp9sHj0='} 2023-11-02T19:26:54.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080037615466f2feb-006543f138-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: tenant: 2023-11-02T19:26:54.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: bucket name: ibtljn-63 2023-11-02T19:26:54.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: total entries: 15 2023-11-02T19:26:54.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.297 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.297 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.298 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.298 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.298 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.299 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T19:26:54.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T19:26:54.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d94a42fbcfc7647-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj23 2023-11-02T19:26:54.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj23 2023-11-02T19:26:54.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj23 2023-11-02T19:26:54.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.303 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.304 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.304 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.304 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.305 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj23 2023-11-02T19:26:54.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.305 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:b36FpwxY61j0nTbV9L2ZpPEhPuQ= 2023-11-02T19:26:54.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:b36FpwxY61j0nTbV9L2ZpPEhPuQ='} 2023-11-02T19:26:54.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b85bacc533ebcbe2-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.310 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.310 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.311 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.311 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.311 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.312 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T19:26:54.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T19:26:54.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a860776008fb5ba6-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj46 2023-11-02T19:26:54.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj46 2023-11-02T19:26:54.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj46 2023-11-02T19:26:54.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.317 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.317 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.317 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.318 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.318 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj46 2023-11-02T19:26:54.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.319 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:/xLyVjOQgsLBiZbxePaI0iPkXos= 2023-11-02T19:26:54.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:/xLyVjOQgsLBiZbxePaI0iPkXos='} 2023-11-02T19:26:54.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd1c37641a833fb8-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.323 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.323 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.324 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.324 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.324 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.325 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T19:26:54.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T19:26:54.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071844c61035573b8-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj69 2023-11-02T19:26:54.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj69 2023-11-02T19:26:54.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj69 2023-11-02T19:26:54.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.329 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.329 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.330 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.330 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.330 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj69 2023-11-02T19:26:54.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.331 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:d8ZmfTg6EOhSzh0h22SQW0Z6PAU= 2023-11-02T19:26:54.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:d8ZmfTg6EOhSzh0h22SQW0Z6PAU='} 2023-11-02T19:26:54.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a90dd9d7747cf963-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.335 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.336 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.336 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.336 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.337 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.337 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T19:26:54.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T19:26:54.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005ae94bb7a93382d-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj92 2023-11-02T19:26:54.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj92 2023-11-02T19:26:54.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj92 2023-11-02T19:26:54.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.342 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.342 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.343 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.343 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.343 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj92 2023-11-02T19:26:54.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.344 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:3f/j928m1BTYdMVhEAzvVxiocbo= 2023-11-02T19:26:54.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:3f/j928m1BTYdMVhEAzvVxiocbo='} 2023-11-02T19:26:54.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b88a4dca9dcbe683-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.348 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.349 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.349 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.349 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.350 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.350 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T19:26:54.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T19:26:54.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004242f9626cfa86e3-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj115 2023-11-02T19:26:54.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj115 2023-11-02T19:26:54.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj115 2023-11-02T19:26:54.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.355 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.355 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.355 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.356 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.356 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj115 2023-11-02T19:26:54.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.357 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:J3FabMctedHifIr8TYgNSKjSmxs= 2023-11-02T19:26:54.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:J3FabMctedHifIr8TYgNSKjSmxs='} 2023-11-02T19:26:54.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb494ffd4722f723-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: tenant: 2023-11-02T19:26:54.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: bucket name: ibtljn-63 2023-11-02T19:26:54.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: total entries: 20 2023-11-02T19:26:54.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.364 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.364 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.364 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.365 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.365 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.366 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T19:26:54.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T19:26:54.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001865950e88a3dc5d-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj29 2023-11-02T19:26:54.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj29 2023-11-02T19:26:54.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj29 2023-11-02T19:26:54.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.370 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.370 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.371 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.371 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.371 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj29 2023-11-02T19:26:54.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.372 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:dFGg5gjTfvj9n5rVIZx0H64JVkM= 2023-11-02T19:26:54.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:dFGg5gjTfvj9n5rVIZx0H64JVkM='} 2023-11-02T19:26:54.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f90dafb1ff01c5b-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.377 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.377 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.377 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.378 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.378 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.379 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T19:26:54.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T19:26:54.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063a7c9e4aad1f803-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj58 2023-11-02T19:26:54.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj58 2023-11-02T19:26:54.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj58 2023-11-02T19:26:54.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.383 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.383 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.383 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.384 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.384 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj58 2023-11-02T19:26:54.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.385 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:rYdGhcqgBakArbK4eQmOM5YlKII= 2023-11-02T19:26:54.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:rYdGhcqgBakArbK4eQmOM5YlKII='} 2023-11-02T19:26:54.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fed7e4dfa48aa04a-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.389 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.389 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.390 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.390 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.390 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.391 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T19:26:54.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T19:26:54.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd44eee09c2d516f-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj87 2023-11-02T19:26:54.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj87 2023-11-02T19:26:54.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj87 2023-11-02T19:26:54.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.396 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.396 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.396 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.397 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.397 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj87 2023-11-02T19:26:54.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.398 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:TSU1MKpZuejo8MHA0nDLnszk1r8= 2023-11-02T19:26:54.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:TSU1MKpZuejo8MHA0nDLnszk1r8='} 2023-11-02T19:26:54.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005bb560cf52f98f25-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.402 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.402 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.402 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.403 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.403 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.404 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T19:26:54.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T19:26:54.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053ad6928ecf1bd47-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj116 2023-11-02T19:26:54.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj116 2023-11-02T19:26:54.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj116 2023-11-02T19:26:54.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.408 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.409 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.409 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.410 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.410 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj116 2023-11-02T19:26:54.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.411 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:c/O10xEaST8VrT0SqYqFGtTlTXA= 2023-11-02T19:26:54.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:c/O10xEaST8VrT0SqYqFGtTlTXA='} 2023-11-02T19:26:54.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc875cdfc3ab4dea-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.415 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.415 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.416 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.416 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.416 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.417 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU= 2023-11-02T19:26:54.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:KDKi+WJcZ0LM3bdzAtlbcWMkCCU=', 'Content-Length': '0'} 2023-11-02T19:26:54.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaf8fa01d3eacc16-006543f139-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj145 2023-11-02T19:26:54.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj145 2023-11-02T19:26:54.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj145 2023-11-02T19:26:54.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.423 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.423 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.424 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.424 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:01 GMT 2023-11-02T19:26:54.424 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj145 2023-11-02T19:26:54.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.425 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:sATZYa3juQ4lsr6sXN+QwJd18LQ= 2023-11-02T19:26:54.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:01 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:sATZYa3juQ4lsr6sXN+QwJd18LQ='} 2023-11-02T19:26:54.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b85047405bd1098-006543f139-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:01 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: tenant: 2023-11-02T19:26:54.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: bucket name: ibtljn-63 2023-11-02T19:26:54.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: total entries: 25 2023-11-02T19:26:54.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.432 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.433 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.433 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.433 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.434 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.435 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T19:26:54.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T19:26:54.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000839a9073a29b2e6b-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj31 2023-11-02T19:26:54.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj31 2023-11-02T19:26:54.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj31 2023-11-02T19:26:54.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.439 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.440 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.440 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.440 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.440 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj31 2023-11-02T19:26:54.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.441 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Bk2Jg/pKnDCSbFHP59KJ5pyJQsE= 2023-11-02T19:26:54.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Bk2Jg/pKnDCSbFHP59KJ5pyJQsE='} 2023-11-02T19:26:54.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f74ff07f65438d2-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.445 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.446 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.446 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.446 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.447 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.447 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T19:26:54.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T19:26:54.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f5ff619acaee089-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj62 2023-11-02T19:26:54.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj62 2023-11-02T19:26:54.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj62 2023-11-02T19:26:54.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.452 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.452 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.453 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.453 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.453 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj62 2023-11-02T19:26:54.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.454 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:0OmLa8OcO6YRjVeUAuyOnXe4deY= 2023-11-02T19:26:54.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0OmLa8OcO6YRjVeUAuyOnXe4deY='} 2023-11-02T19:26:54.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ab0364662b620e7-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.458 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.459 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.459 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.459 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.459 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.460 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T19:26:54.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T19:26:54.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b974a0e74a0fc4d6-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj93 2023-11-02T19:26:54.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj93 2023-11-02T19:26:54.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj93 2023-11-02T19:26:54.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.465 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.465 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.465 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.466 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.466 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj93 2023-11-02T19:26:54.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.467 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:RpHE0ZxOC4JrLb2hOe+o92owwBY= 2023-11-02T19:26:54.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:RpHE0ZxOC4JrLb2hOe+o92owwBY='} 2023-11-02T19:26:54.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbc476bccf90de04-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.471 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.471 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.472 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.472 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.472 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.473 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T19:26:54.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T19:26:54.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6ea3a3b3b841083-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj124 2023-11-02T19:26:54.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj124 2023-11-02T19:26:54.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj124 2023-11-02T19:26:54.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.478 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.478 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.478 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.479 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.479 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj124 2023-11-02T19:26:54.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.480 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:0r8TiryND2XSk18xD5a2PxEgzp4= 2023-11-02T19:26:54.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:0r8TiryND2XSk18xD5a2PxEgzp4='} 2023-11-02T19:26:54.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e7c772a80f79fe4-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.484 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.485 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.485 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.485 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.485 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.486 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T19:26:54.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T19:26:54.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3a5832f1ffbbe0b-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj155 2023-11-02T19:26:54.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj155 2023-11-02T19:26:54.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj155 2023-11-02T19:26:54.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.491 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.491 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.491 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.492 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.492 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj155 2023-11-02T19:26:54.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.493 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:XcCyHSdrNqGKLAz0CV4yCfHZ3D8= 2023-11-02T19:26:54.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XcCyHSdrNqGKLAz0CV4yCfHZ3D8='} 2023-11-02T19:26:54.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4f0c77483ad0bf6-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'ibtljn-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket ibtljn-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: tenant: 2023-11-02T19:26:54.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: bucket name: ibtljn-63 2023-11-02T19:26:54.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: total entries: 30 2023-11-02T19:26:54.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.500 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.500 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.501 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.501 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.501 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.502 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T19:26:54.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T19:26:54.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c10ac17c4e37c61-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj37 2023-11-02T19:26:54.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj37 2023-11-02T19:26:54.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj37 2023-11-02T19:26:54.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.506 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.507 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.507 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.507 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.508 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj37 2023-11-02T19:26:54.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.508 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:jnDXNCgJvYoUhU1No/HgAPmCAck= 2023-11-02T19:26:54.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:jnDXNCgJvYoUhU1No/HgAPmCAck='} 2023-11-02T19:26:54.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000851fc34cbc20edb4-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.513 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.513 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.514 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.514 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.514 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.515 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T19:26:54.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T19:26:54.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b29bde3be3c9b5bd-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj74 2023-11-02T19:26:54.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj74 2023-11-02T19:26:54.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj74 2023-11-02T19:26:54.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.519 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.520 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.520 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.520 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.521 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj74 2023-11-02T19:26:54.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.521 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:8rrAn2ebV86nm5P8Y2DFrcXFc4E= 2023-11-02T19:26:54.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:8rrAn2ebV86nm5P8Y2DFrcXFc4E='} 2023-11-02T19:26:54.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009db2f2a5a7e78349-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.526 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.526 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.526 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.527 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.527 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.528 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T19:26:54.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T19:26:54.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f21b4b03f34a85a1-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj111 2023-11-02T19:26:54.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj111 2023-11-02T19:26:54.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj111 2023-11-02T19:26:54.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.532 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.533 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.533 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.533 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.534 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj111 2023-11-02T19:26:54.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.534 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:zkxXDRZ9NDBwwR0hsuMMkLNxwuU= 2023-11-02T19:26:54.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:zkxXDRZ9NDBwwR0hsuMMkLNxwuU='} 2023-11-02T19:26:54.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063a53687d8aac81e-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.539 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.539 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.539 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.540 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.540 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.541 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM= 2023-11-02T19:26:54.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:GXsYcMXBxCllLXQONKEEdR+KLXM=', 'Content-Length': '0'} 2023-11-02T19:26:54.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b08a3cadc50e91e-006543f13a-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj148 2023-11-02T19:26:54.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj148 2023-11-02T19:26:54.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj148 2023-11-02T19:26:54.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.545 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.545 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.546 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.546 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:02 GMT 2023-11-02T19:26:54.546 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj148 2023-11-02T19:26:54.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.547 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:XQxGr2z5Ov3dSGhv54i0saLdeKE= 2023-11-02T19:26:54.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:02 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:XQxGr2z5Ov3dSGhv54i0saLdeKE='} 2023-11-02T19:26:54.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a4de0d57491a170-006543f13a-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:02 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/ 2023-11-02T19:26:54.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/ 2023-11-02T19:26:54.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:54.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/ 2023-11-02T19:26:54.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.551 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:54.551 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.552 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.552 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:03 GMT 2023-11-02T19:26:54.552 INFO:tasks.rgw_multisite_tests:/ibtljn-63/ 2023-11-02T19:26:54.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.553 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:t4Mmfrq8g0lZrG9AOlcKBI4PfJk= 2023-11-02T19:26:54.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 18:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:t4Mmfrq8g0lZrG9AOlcKBI4PfJk=', 'Content-Length': '0'} 2023-11-02T19:26:54.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a679cedb7412d5e6-006543f13b-1669-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 18:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-63/obj185 2023-11-02T19:26:54.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-63/obj185 2023-11-02T19:26:54.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-63/obj185 2023-11-02T19:26:54.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-11-02T19:26:54.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.557 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.558 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-11-02T19:26:54.558 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:54.558 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 18:58:03 GMT 2023-11-02T19:26:54.559 INFO:tasks.rgw_multisite_tests:/ibtljn-63/obj185 2023-11-02T19:26:54.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.559 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:pYD7xIQ2IH7KUIgc1P3uzx13z3c= 2023-11-02T19:26:54.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 18:58:03 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:pYD7xIQ2IH7KUIgc1P3uzx13z3c='} 2023-11-02T19:26:54.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed8fecf151ddd81f-006543f13b-1669-a1'), ('Date', 'Thu, 02 Nov 2023 18:58:03 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=ibtljn-63 2023-11-02T19:26:54.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:54.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:54.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T19:26:54.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:03.153+0000 7fb2b948cf40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:08.153+0000 7fb2b948cf40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:13.157+0000 7fb2b948cf40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:18.161+0000 7fb2b948cf40 1 bucket sync caught up with source: 2023-11-02T19:26:54.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: local status: [, , , , , , 00000000002.70.6, , , , , , 00000000002.130.6, , , , , 00000000002.114.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.97.6] 2023-11-02T19:26:54.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: remote markers: [, , , , , , 00000000002.70.6, , , , , , 00000000002.130.6, , , , , 00000000002.114.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.97.6] 2023-11-02T19:26:54.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:18.161+0000 7fb2b948cf40 0 bucket checkpoint complete 2023-11-02T19:26:54.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=ibtljn-63 2023-11-02T19:26:54.565 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:54.565 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ibtljn-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-11-02T19:26:54.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ibtljn-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-11-02T19:26:54.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:18.313+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:23.313+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:28.313+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:33.317+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:38.317+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:43.317+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:48.317+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:53.321+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:58:58.321+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:59:03.321+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:59:08.321+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:59:13.325+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:59:18.325+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:59:23.325+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:59:28.329+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:59:33.328+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:59:38.328+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:59:43.328+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:59:48.332+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:59:53.332+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T18:59:58.332+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:00:03.332+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:00:08.336+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:00:13.336+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:00:18.336+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:00:23.336+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:00:28.340+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:00:33.340+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:00:38.340+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:00:43.340+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:00:48.344+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:00:53.348+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:00:58.344+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:01:03.348+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:01:08.348+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:01:13.347+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:01:18.351+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:01:23.351+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:01:28.351+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:01:33.351+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:01:38.355+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:01:43.359+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:01:48.359+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:01:53.359+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:01:58.363+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:02:03.363+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:02:08.363+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:02:13.367+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:02:18.367+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:02:23.367+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:02:28.371+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:02:33.371+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:02:38.371+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:02:43.371+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:02:48.374+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:02:53.374+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:02:58.374+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:03:03.378+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:03:08.378+0000 7ff2bff08f40 1 waiting to reach incremental sync.. 2023-11-02T19:26:54.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:03:13.378+0000 7ff2bff08f40 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-11-02T19:26:54.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:03:13.378+0000 7ff2bff08f40 0 bucket sync checkpoint failed: (110) Connection timed out 2023-11-02T19:26:54.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:03:13.378+0000 7ff2bff08f40 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-11-02T19:26:54.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-11-02T19:26:54.586 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-11-02T19:26:54.587 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.587 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-11-02T19:26:54.587 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket_to_multiple: 2023-11-02T19:26:54.588 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-11-02T19:26:54.588 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-11-02T19:26:54.589 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-11-02T19:26:54.589 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-11-02T19:26:54.590 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/../src/test/rgw/rgw_multi/tests.py", line 2813, in test_sync_single_bucket_to_multiple 2023-11-02T19:26:54.590 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesA, content) 2023-11-02T19:26:54.590 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/../src/test/rgw/rgw_multi/tests.py", line 2047, in check_objects_exist 2023-11-02T19:26:54.591 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2023-11-02T19:26:54.591 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-11-02T19:26:54.591 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-11-02T19:26:54.591 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-11-02T19:26:54.592 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-11-02T19:26:54.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:54.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:26:54.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: [ 2023-11-02T19:26:54.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:26:54.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:26:54.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698953094.851156_840.1", 2023-11-02T19:26:54.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T19:24:54.851156Z" 2023-11-02T19:26:54.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:26:54.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953094.851156_840.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-11-02T19:26:54.601 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.601 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:54.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:24:55.329+0000 7f6d641e7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:24:55.333+0000 7f6d641e7f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:54.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:54.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:54.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:54.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:54.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:54.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 1, 2023-11-02T19:26:54.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:54.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:54.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:26:54.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:22:28.822959Z", 2023-11-02T19:26:54.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:54.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:54.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 1, 2023-11-02T19:26:54.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 1 2023-11-02T19:26:54.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:24:55Z" 2023-11-02T19:26:54.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-11-02T19:26:54.625 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:54.625 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:54.626 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:54.626 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:54.626 INFO:tasks.rgw_multisite_tests: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.626 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.627 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.627 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:54.627 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.628 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:54.628 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.628 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.629 INFO:tasks.rgw_multisite_tests: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.629 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-11-02T19:26:54.630 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.630 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.631 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.631 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.631 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.631 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:54.632 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.632 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.634 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.634 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.635 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.635 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.635 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:54.636 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.636 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.636 INFO:tasks.rgw_multisite_tests: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:26:54.637 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.637 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.637 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.637 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:22:28.822959Z", 2023-11-02T19:26:54.638 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.638 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.638 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.639 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.639 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:54.639 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.640 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.640 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.640 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.641 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.641 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.641 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.642 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.642 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.643 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:54.643 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.643 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:54.643 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-11-02T19:26:54.644 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-11-02T19:26:54.644 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.644 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:24:55Z" 2023-11-02T19:26:54.645 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:54.645 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952948.822959_838.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:22:28.822959Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:26:54.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953094.851156_840.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1698953094.851156_840.1 target=1_1698952948.822959_838.1 2023-11-02T19:26:54.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:54.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:25:00.480+0000 7f978c818f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:25:00.484+0000 7f978c818f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:54.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:54.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:54.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:54.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:54.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:54.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 1, 2023-11-02T19:26:54.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:54.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:54.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:26:54.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:22:28.822959Z", 2023-11-02T19:26:54.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:54.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:54.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 1, 2023-11-02T19:26:54.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 1 2023-11-02T19:26:54.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:00Z" 2023-11-02T19:26:54.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-11-02T19:26:54.670 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:54.670 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:54.671 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:54.671 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:54.671 INFO:tasks.rgw_multisite_tests: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.672 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.672 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.672 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:54.672 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.673 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:54.673 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.673 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.674 INFO:tasks.rgw_multisite_tests: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.674 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-11-02T19:26:54.675 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.675 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.676 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.676 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.676 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.677 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:54.677 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.677 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.677 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.678 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.678 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.678 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.679 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.679 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.679 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.680 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.680 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.680 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:54.681 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.681 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.681 INFO:tasks.rgw_multisite_tests: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:26:54.682 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.682 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.682 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.683 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:22:28.822959Z", 2023-11-02T19:26:54.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.683 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.684 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.684 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.684 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:54.685 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.685 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.685 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.686 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.686 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.686 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.687 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.687 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.687 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.688 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:54.688 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.688 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:54.689 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-11-02T19:26:54.689 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-11-02T19:26:54.689 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.690 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:00Z" 2023-11-02T19:26:54.690 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:54.690 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952948.822959_838.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:22:28.822959Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:26:54.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953094.851156_840.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1698953094.851156_840.1 target=1_1698952948.822959_838.1 2023-11-02T19:26:54.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:54.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:25:05.692+0000 7f94aa895f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:25:05.696+0000 7f94aa895f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:54.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:54.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:54.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:54.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:54.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:54.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 1, 2023-11-02T19:26:54.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:54.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:54.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:26:54.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:22:28.822959Z", 2023-11-02T19:26:54.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:54.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:54.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 1, 2023-11-02T19:26:54.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 1 2023-11-02T19:26:54.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:05Z" 2023-11-02T19:26:54.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-11-02T19:26:54.716 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:54.716 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:54.717 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:54.717 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:54.717 INFO:tasks.rgw_multisite_tests: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.718 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.718 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.718 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:54.719 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.719 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:54.719 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.720 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.720 INFO:tasks.rgw_multisite_tests: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.721 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-11-02T19:26:54.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.721 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.722 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.722 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.722 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.723 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.723 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:54.723 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.724 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.724 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.724 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.725 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.725 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.725 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.725 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.726 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.726 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.727 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.727 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:54.727 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.727 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.728 INFO:tasks.rgw_multisite_tests: "marker": "1_1698952948.822959_838.1", 2023-11-02T19:26:54.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.729 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:22:28.822959Z", 2023-11-02T19:26:54.729 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.730 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.730 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.730 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.731 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:54.731 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.731 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.732 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.732 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.732 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.733 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.733 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.734 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.734 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.734 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:54.734 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.735 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:54.735 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-11-02T19:26:54.735 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-11-02T19:26:54.736 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.736 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:05Z" 2023-11-02T19:26:54.736 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:54.737 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698952948.822959_838.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:22:28.822959Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:26:54.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953094.851156_840.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698952948.822959_838.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1698953094.851156_840.1 target=1_1698952948.822959_838.1 2023-11-02T19:26:54.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.739 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:54.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:25:10.848+0000 7f4067bcaf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:25:10.852+0000 7f4067bcaf40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:54.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:54.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:54.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:54.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:54.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:54.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 1, 2023-11-02T19:26:54.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:54.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:54.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698953094.851156_840.1", 2023-11-02T19:26:54.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:54.851156Z", 2023-11-02T19:26:54.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:54.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:54.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 1, 2023-11-02T19:26:54.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 1 2023-11-02T19:26:54.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:10Z" 2023-11-02T19:26:54.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-11-02T19:26:54.761 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:54.762 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:54.762 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:54.762 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:54.763 INFO:tasks.rgw_multisite_tests: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.763 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.764 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:54.764 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.764 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:54.765 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.765 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.765 INFO:tasks.rgw_multisite_tests: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.766 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.766 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-11-02T19:26:54.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.767 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.767 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.767 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.768 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.768 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.768 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:54.769 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.769 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.769 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.770 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.771 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.771 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.772 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.772 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:54.772 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.773 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.773 INFO:tasks.rgw_multisite_tests: "marker": "1_1698953094.851156_840.1", 2023-11-02T19:26:54.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.774 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.774 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:54.851156Z", 2023-11-02T19:26:54.775 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.775 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.775 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.775 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.776 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:54.776 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.776 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.777 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.777 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.777 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.778 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.778 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.778 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.779 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.779 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.779 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:54.779 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.780 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:54.780 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-11-02T19:26:54.780 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-11-02T19:26:54.781 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.781 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:10Z" 2023-11-02T19:26:54.781 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:54.782 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698953094.851156_840.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:54.851156Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:26:54.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953094.851156_840.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953094.851156_840.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-11-02T19:26:54.783 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.784 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: {"groups":[]} 2023-11-02T19:26:54.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=ibtljn-74 2023-11-02T19:26:54.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-74/ 2023-11-02T19:26:54.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-74/ 2023-11-02T19:26:54.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-74/ 2023-11-02T19:26:54.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.789 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.789 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.790 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.790 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 19:25:11 GMT 2023-11-02T19:26:54.790 INFO:tasks.rgw_multisite_tests:/ibtljn-74/ 2023-11-02T19:26:54.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.791 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:o4bRco7x9/uzT/9dpHocmbebtaQ= 2023-11-02T19:26:54.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:25:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:o4bRco7x9/uzT/9dpHocmbebtaQ=', 'Content-Length': '0'} 2023-11-02T19:26:54.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f8cbaf2325839c19-006543f797-2bbf-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=ibtljn-75 2023-11-02T19:26:54.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-75/ 2023-11-02T19:26:54.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-75/ 2023-11-02T19:26:54.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:54.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-75/ 2023-11-02T19:26:54.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:54.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:54.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:54.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:54.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:54.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:54.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:54.796 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:54.796 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.797 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.797 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 19:25:11 GMT 2023-11-02T19:26:54.797 INFO:tasks.rgw_multisite_tests:/ibtljn-75/ 2023-11-02T19:26:54.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:54.798 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:nBgnqmX7jX4MCk1O4b/LL/fc2X8= 2023-11-02T19:26:54.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:25:11 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nBgnqmX7jX4MCk1O4b/LL/fc2X8=', 'Content-Length': '0'} 2023-11-02T19:26:54.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000000ff2e7d868db2017-006543f797-2bbf-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:54.799 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.799 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:54.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:26:54.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: [ 2023-11-02T19:26:54.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:26:54.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:26:54.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698953111.156002_848.1", 2023-11-02T19:26:54.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T19:25:11.156002Z" 2023-11-02T19:26:54.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:26:54.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-11-02T19:26:54.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.808 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:54.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:25:11.400+0000 7fab723c8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:25:11.400+0000 7fab723c8f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:54.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:54.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:54.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:54.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:54.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:54.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 1, 2023-11-02T19:26:54.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:54.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:54.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698953111.108996_841.1", 2023-11-02T19:26:54.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:25:11.108996Z", 2023-11-02T19:26:54.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:54.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:54.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 1, 2023-11-02T19:26:54.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 1 2023-11-02T19:26:54.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:11Z" 2023-11-02T19:26:54.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-11-02T19:26:54.830 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:54.831 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:54.831 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:54.831 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:54.833 INFO:tasks.rgw_multisite_tests: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.834 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.834 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:54.834 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.835 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:54.835 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.836 INFO:tasks.rgw_multisite_tests: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.836 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.836 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-11-02T19:26:54.837 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.837 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.837 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.837 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.838 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.838 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.838 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:54.839 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.839 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.839 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.840 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.840 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.840 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.841 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.841 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.841 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.842 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.842 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.842 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:54.843 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.843 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.843 INFO:tasks.rgw_multisite_tests: "marker": "1_1698953111.108996_841.1", 2023-11-02T19:26:54.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.844 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.844 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.844 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:25:11.108996Z", 2023-11-02T19:26:54.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.845 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.845 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.846 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.846 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:54.846 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.847 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.847 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.848 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.848 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.848 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.849 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.849 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.849 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.850 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:54.850 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.850 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:54.851 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-11-02T19:26:54.851 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-11-02T19:26:54.851 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.851 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:11Z" 2023-11-02T19:26:54.852 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:54.852 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698953111.108996_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:25:11.108996Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:26:54.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.108996_841.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1698953111.156002_848.1 target=1_1698953111.108996_841.1 2023-11-02T19:26:54.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:54.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:54.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:25:16.564+0000 7f2561ac0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:25:16.568+0000 7f2561ac0f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:54.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:54.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:54.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:54.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:54.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:54.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:54.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 1, 2023-11-02T19:26:54.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:54.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:54.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698953111.156002_848.1", 2023-11-02T19:26:54.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:25:11.156002Z", 2023-11-02T19:26:54.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:54.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:54.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:54.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:54.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:54.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:54.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:54.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:54.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 1, 2023-11-02T19:26:54.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 1 2023-11-02T19:26:54.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:16Z" 2023-11-02T19:26:54.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-11-02T19:26:54.876 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:54.877 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:54.877 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:54.877 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:54.878 INFO:tasks.rgw_multisite_tests: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.878 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.878 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.879 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:54.879 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.879 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:54.880 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.880 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.880 INFO:tasks.rgw_multisite_tests: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:54.880 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.881 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-11-02T19:26:54.881 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.881 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:54.882 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.882 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.882 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.883 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.883 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:54.883 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.883 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.884 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:54.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.884 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.885 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:54.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.886 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.886 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.886 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.887 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:54.887 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.887 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.888 INFO:tasks.rgw_multisite_tests: "marker": "1_1698953111.156002_848.1", 2023-11-02T19:26:54.888 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.888 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.889 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.889 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:25:11.156002Z", 2023-11-02T19:26:54.889 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.890 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.890 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.890 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:54.891 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:54.891 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:54.891 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:54.892 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:54.892 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:54.892 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:54.892 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:54.893 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:54.893 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:54.893 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.894 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:54.894 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:54.894 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.895 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:54.895 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-11-02T19:26:54.895 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-11-02T19:26:54.896 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:54.896 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:16Z" 2023-11-02T19:26:54.896 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:54.897 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:54.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698953111.156002_848.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:25:11.156002Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:26:54.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:54.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-11-02T19:26:54.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-11-02T19:26:54.900 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.900 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-11-02T19:26:54.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-74', '--dest-bucket=ibtljn-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-74', '--dest-bucket=ibtljn-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-ibtljn-74 --source-zones=a1 --dest-zones=a2 --source-bucket=ibtljn-74 --dest-bucket=ibtljn-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-74","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:26:54.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-74', '--dest-bucket=ibtljn-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-ibtljn-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-74', '--dest-bucket=ibtljn-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-ibtljn-75 --source-zones=a1 --dest-zones=a2 --source-bucket=ibtljn-74 --dest-bucket=ibtljn-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-74","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-ibtljn-75","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:26:54.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-74', '--dest-bucket=ibtljn-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=ibtljn-74', '--dest-bucket=ibtljn-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=ibtljn-74 --dest-bucket=ibtljn-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-74","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-ibtljn-75","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-11-02T19:26:54.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-74","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-ibtljn-75","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:26:54.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:54.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:54.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:54.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:54.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "epoch": 22, 2023-11-02T19:26:54.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "predecessor_uuid": "5f87da71-3b30-4008-9009-a0341777f9d7", 2023-11-02T19:26:54.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": [], 2023-11-02T19:26:54.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "period_map": { 2023-11-02T19:26:54.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:54.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zonegroups": [ 2023-11-02T19:26:54.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "a", 2023-11-02T19:26:54.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "name": "a", 2023-11-02T19:26:54.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "api_name": "", 2023-11-02T19:26:54.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "is_master": true, 2023-11-02T19:26:54.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "endpoints": [ 2023-11-02T19:26:54.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:26:54.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ], 2023-11-02T19:26:54.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hostnames": [], 2023-11-02T19:26:54.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hostnames_s3website": [], 2023-11-02T19:26:54.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:26:54.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones": [ 2023-11-02T19:26:54.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:26:54.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "name": "a2", 2023-11-02T19:26:54.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "endpoints": [ 2023-11-02T19:26:54.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "http://smithi017.front.sepia.ceph.com:8001" 2023-11-02T19:26:54.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ], 2023-11-02T19:26:54.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "log_meta": false, 2023-11-02T19:26:54.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "log_data": true, 2023-11-02T19:26:54.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bucket_index_max_shards": 11, 2023-11-02T19:26:54.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "read_only": false, 2023-11-02T19:26:54.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "tier_type": "", 2023-11-02T19:26:54.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_from_all": true, 2023-11-02T19:26:54.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_from": [], 2023-11-02T19:26:54.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "redirect_zone": "", 2023-11-02T19:26:54.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "supported_features": [ 2023-11-02T19:26:54.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "compress-encrypted", 2023-11-02T19:26:54.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "resharding" 2023-11-02T19:26:54.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:26:54.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "name": "a1", 2023-11-02T19:26:54.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "endpoints": [ 2023-11-02T19:26:54.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "http://smithi017.front.sepia.ceph.com:8000" 2023-11-02T19:26:54.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ], 2023-11-02T19:26:54.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "log_meta": false, 2023-11-02T19:26:54.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "log_data": true, 2023-11-02T19:26:54.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bucket_index_max_shards": 11, 2023-11-02T19:26:54.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "read_only": false, 2023-11-02T19:26:54.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "tier_type": "", 2023-11-02T19:26:54.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_from_all": true, 2023-11-02T19:26:54.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_from": [], 2023-11-02T19:26:54.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "redirect_zone": "", 2023-11-02T19:26:54.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "supported_features": [ 2023-11-02T19:26:54.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "compress-encrypted", 2023-11-02T19:26:54.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "resharding" 2023-11-02T19:26:54.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ], 2023-11-02T19:26:54.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "placement_targets": [ 2023-11-02T19:26:54.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "name": "default-placement", 2023-11-02T19:26:54.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "tags": [], 2023-11-02T19:26:54.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "storage_classes": [ 2023-11-02T19:26:54.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "STANDARD" 2023-11-02T19:26:54.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ], 2023-11-02T19:26:54.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "default_placement": "default-placement", 2023-11-02T19:26:54.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:26:54.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_policy": { 2023-11-02T19:26:54.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "groups": [ 2023-11-02T19:26:54.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "sync-group", 2023-11-02T19:26:54.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "data_flow": { 2023-11-02T19:26:54.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "symmetrical": [ 2023-11-02T19:26:54.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "sync-flow1", 2023-11-02T19:26:54.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones": [ 2023-11-02T19:26:54.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:26:54.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:26:54.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pipes": [ 2023-11-02T19:26:54.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "sync-pipe-ibtljn-74", 2023-11-02T19:26:54.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "source": { 2023-11-02T19:26:54.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bucket": "ibtljn-74", 2023-11-02T19:26:54.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones": [ 2023-11-02T19:26:54.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:26:54.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "dest": { 2023-11-02T19:26:54.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bucket": "ibtljn-74", 2023-11-02T19:26:54.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones": [ 2023-11-02T19:26:54.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "39a01552-6d59-4ce7-b7e5-915ca69b979a" 2023-11-02T19:26:54.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "params": { 2023-11-02T19:26:54.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "source": { 2023-11-02T19:26:54.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "filter": { 2023-11-02T19:26:54.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "tags": [] 2023-11-02T19:26:54.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "dest": {}, 2023-11-02T19:26:54.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "priority": 0, 2023-11-02T19:26:54.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "mode": "system", 2023-11-02T19:26:54.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "user": "" 2023-11-02T19:26:54.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "sync-pipe-ibtljn-75", 2023-11-02T19:26:54.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "source": { 2023-11-02T19:26:54.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bucket": "ibtljn-74", 2023-11-02T19:26:54.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones": [ 2023-11-02T19:26:54.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:26:54.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "dest": { 2023-11-02T19:26:54.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bucket": "ibtljn-75", 2023-11-02T19:26:54.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones": [ 2023-11-02T19:26:54.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "39a01552-6d59-4ce7-b7e5-915ca69b979a" 2023-11-02T19:26:54.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "params": { 2023-11-02T19:26:54.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "source": { 2023-11-02T19:26:54.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "filter": { 2023-11-02T19:26:54.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "tags": [] 2023-11-02T19:26:54.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "dest": {}, 2023-11-02T19:26:54.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "priority": 0, 2023-11-02T19:26:54.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "mode": "system", 2023-11-02T19:26:54.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "user": "" 2023-11-02T19:26:54.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "sync-pipe", 2023-11-02T19:26:54.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "source": { 2023-11-02T19:26:54.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bucket": "ibtljn-74", 2023-11-02T19:26:54.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones": [ 2023-11-02T19:26:54.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "d61b2bea-59a8-4454-8491-ac3afbd04937" 2023-11-02T19:26:54.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "dest": { 2023-11-02T19:26:54.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bucket": "ibtljn-75", 2023-11-02T19:26:54.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones": [ 2023-11-02T19:26:54.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "39a01552-6d59-4ce7-b7e5-915ca69b979a" 2023-11-02T19:26:54.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "params": { 2023-11-02T19:26:54.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "source": { 2023-11-02T19:26:54.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "filter": { 2023-11-02T19:26:54.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "tags": [] 2023-11-02T19:26:54.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "dest": {}, 2023-11-02T19:26:54.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "priority": 0, 2023-11-02T19:26:54.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "mode": "system", 2023-11-02T19:26:54.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "user": "" 2023-11-02T19:26:54.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ], 2023-11-02T19:26:54.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "enabled" 2023-11-02T19:26:54.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "enabled_features": [ 2023-11-02T19:26:54.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "resharding" 2023-11-02T19:26:54.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "b", 2023-11-02T19:26:54.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "name": "b", 2023-11-02T19:26:54.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "api_name": "", 2023-11-02T19:26:54.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "is_master": false, 2023-11-02T19:26:54.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "endpoints": [ 2023-11-02T19:26:54.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:26:54.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ], 2023-11-02T19:26:54.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hostnames": [], 2023-11-02T19:26:54.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "hostnames_s3website": [], 2023-11-02T19:26:54.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "master_zone": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:26:54.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "zones": [ 2023-11-02T19:26:54.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:26:54.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "name": "b1", 2023-11-02T19:26:54.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "endpoints": [ 2023-11-02T19:26:54.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "http://smithi069.front.sepia.ceph.com:8000" 2023-11-02T19:26:54.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ], 2023-11-02T19:26:54.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "log_meta": false, 2023-11-02T19:26:54.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "log_data": true, 2023-11-02T19:26:54.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bucket_index_max_shards": 11, 2023-11-02T19:26:54.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "read_only": false, 2023-11-02T19:26:54.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "tier_type": "", 2023-11-02T19:26:54.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_from_all": true, 2023-11-02T19:26:54.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_from": [], 2023-11-02T19:26:54.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "redirect_zone": "", 2023-11-02T19:26:54.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "supported_features": [ 2023-11-02T19:26:54.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "compress-encrypted", 2023-11-02T19:26:54.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "resharding" 2023-11-02T19:26:54.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "id": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:26:54.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "name": "b2", 2023-11-02T19:26:54.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "endpoints": [ 2023-11-02T19:26:54.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "http://smithi069.front.sepia.ceph.com:8001" 2023-11-02T19:26:54.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ], 2023-11-02T19:26:54.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "log_meta": false, 2023-11-02T19:26:54.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "log_data": true, 2023-11-02T19:26:54.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bucket_index_max_shards": 11, 2023-11-02T19:26:54.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "read_only": false, 2023-11-02T19:26:54.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "tier_type": "", 2023-11-02T19:26:54.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_from_all": true, 2023-11-02T19:26:54.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_from": [], 2023-11-02T19:26:54.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "redirect_zone": "", 2023-11-02T19:26:54.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "supported_features": [ 2023-11-02T19:26:54.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "compress-encrypted", 2023-11-02T19:26:54.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "resharding" 2023-11-02T19:26:54.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ], 2023-11-02T19:26:54.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "placement_targets": [ 2023-11-02T19:26:54.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "name": "default-placement", 2023-11-02T19:26:54.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "tags": [], 2023-11-02T19:26:54.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "storage_classes": [ 2023-11-02T19:26:54.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "STANDARD" 2023-11-02T19:26:54.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ], 2023-11-02T19:26:54.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "default_placement": "default-placement", 2023-11-02T19:26:54.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:26:54.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_policy": { 2023-11-02T19:26:54.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "groups": [] 2023-11-02T19:26:54.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "enabled_features": [ 2023-11-02T19:26:54.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "resharding" 2023-11-02T19:26:54.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ], 2023-11-02T19:26:54.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "short_zone_ids": [ 2023-11-02T19:26:54.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": "39a01552-6d59-4ce7-b7e5-915ca69b979a", 2023-11-02T19:26:54.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": 3699239738 2023-11-02T19:26:54.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": "bf0a8e78-6008-4d42-8f71-c8dec4a40cb0", 2023-11-02T19:26:54.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": 2602129372 2023-11-02T19:26:54.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:26:54.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": 2073569678 2023-11-02T19:26:54.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:54.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": "da3bc04c-8ffb-4e5a-bce3-cbd8c10bac7c", 2023-11-02T19:26:54.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": 3449465395 2023-11-02T19:26:54.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:54.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:54.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "master_zonegroup": "a", 2023-11-02T19:26:54.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "master_zone": "d61b2bea-59a8-4454-8491-ac3afbd04937", 2023-11-02T19:26:54.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "period_config": { 2023-11-02T19:26:54.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bucket_quota": { 2023-11-02T19:26:54.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "enabled": false, 2023-11-02T19:26:54.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "check_on_raw": false, 2023-11-02T19:26:54.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_size": -1, 2023-11-02T19:26:54.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_size_kb": 0, 2023-11-02T19:26:54.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_objects": -1 2023-11-02T19:26:54.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "user_quota": { 2023-11-02T19:26:54.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "enabled": false, 2023-11-02T19:26:54.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "check_on_raw": false, 2023-11-02T19:26:54.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_size": -1, 2023-11-02T19:26:54.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_size_kb": 0, 2023-11-02T19:26:54.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_objects": -1 2023-11-02T19:26:54.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:54.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "user_ratelimit": { 2023-11-02T19:26:54.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_read_ops": 0, 2023-11-02T19:26:54.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_write_ops": 0, 2023-11-02T19:26:54.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_read_bytes": 0, 2023-11-02T19:26:55.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_write_bytes": 0, 2023-11-02T19:26:55.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "enabled": false 2023-11-02T19:26:55.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "bucket_ratelimit": { 2023-11-02T19:26:55.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_read_ops": 0, 2023-11-02T19:26:55.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_write_ops": 0, 2023-11-02T19:26:55.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_read_bytes": 0, 2023-11-02T19:26:55.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_write_bytes": 0, 2023-11-02T19:26:55.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "enabled": false 2023-11-02T19:26:55.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "anonymous_ratelimit": { 2023-11-02T19:26:55.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_read_ops": 0, 2023-11-02T19:26:55.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_write_ops": 0, 2023-11-02T19:26:55.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_read_bytes": 0, 2023-11-02T19:26:55.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "max_write_bytes": 0, 2023-11-02T19:26:55.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "enabled": false 2023-11-02T19:26:55.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_id": "a2bfe17b-2975-4441-84f6-480a071e229b", 2023-11-02T19:26:55.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_name": "test-realm", 2023-11-02T19:26:55.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:55.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.007 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:55.007 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-11-02T19:26:55.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-11-02T19:26:55.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-ibtljn-74","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-ibtljn-75","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"ibtljn-74","zones":["a1"]},"dest":{"bucket":"ibtljn-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-11-02T19:26:55.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-74/ 2023-11-02T19:26:55.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-74/ 2023-11-02T19:26:55.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:55.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-74/ 2023-11-02T19:26:55.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:55.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:55.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:55.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:55.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:55.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:55.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:55.012 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:55.012 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.013 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.013 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 19:25:17 GMT 2023-11-02T19:26:55.013 INFO:tasks.rgw_multisite_tests:/ibtljn-74/ 2023-11-02T19:26:55.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:55.014 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:JYwK9a1HuWnXAfBs286Z72atqr4= 2023-11-02T19:26:55.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:25:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JYwK9a1HuWnXAfBs286Z72atqr4=', 'Content-Length': '0'} 2023-11-02T19:26:55.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d7e133c4a4b4df4-006543f79d-2da3-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:25:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:55.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-74/obj1 2023-11-02T19:26:55.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-74/obj1 2023-11-02T19:26:55.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:55.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-74/obj1 2023-11-02T19:26:55.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:55.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:26:55.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:55.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:55.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:55.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:55.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:55.018 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:55.019 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-11-02T19:26:55.019 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:55.019 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 19:25:17 GMT 2023-11-02T19:26:55.020 INFO:tasks.rgw_multisite_tests:/ibtljn-74/obj1 2023-11-02T19:26:55.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:55.020 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:Ndfr1Hm0X33RWd087dE1w+yRM4E= 2023-11-02T19:26:55.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:25:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:Ndfr1Hm0X33RWd087dE1w+yRM4E='} 2023-11-02T19:26:55.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000720827c3a2b64638-006543f79d-2da3-a1'), ('Date', 'Thu, 02 Nov 2023 19:25:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:55.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-74/ 2023-11-02T19:26:55.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-74/ 2023-11-02T19:26:55.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:55.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-74/ 2023-11-02T19:26:55.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:55.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:55.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:55.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:55.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:55.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:55.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:55.025 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:55.025 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.025 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.026 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 19:25:17 GMT 2023-11-02T19:26:55.026 INFO:tasks.rgw_multisite_tests:/ibtljn-74/ 2023-11-02T19:26:55.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:55.027 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:JYwK9a1HuWnXAfBs286Z72atqr4= 2023-11-02T19:26:55.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:25:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:JYwK9a1HuWnXAfBs286Z72atqr4=', 'Content-Length': '0'} 2023-11-02T19:26:55.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062fa800b4bf051f2-006543f79d-2da3-a1'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:25:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:55.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-74/obj2 2023-11-02T19:26:55.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-74/obj2 2023-11-02T19:26:55.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-11-02T19:26:55.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-74/obj2 2023-11-02T19:26:55.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:55.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-11-02T19:26:55.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8000 2023-11-02T19:26:55.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-11-02T19:26:55.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:55.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:55.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:55.031 INFO:tasks.rgw_multisite_tests:PUT 2023-11-02T19:26:55.031 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-11-02T19:26:55.032 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-11-02T19:26:55.032 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 19:25:17 GMT 2023-11-02T19:26:55.032 INFO:tasks.rgw_multisite_tests:/ibtljn-74/obj2 2023-11-02T19:26:55.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:55.033 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:9t4s4drzRFYmiscgsOyKcaYJ4Kw= 2023-11-02T19:26:55.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 02 Nov 2023 19:25:17 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:9t4s4drzRFYmiscgsOyKcaYJ4Kw='} 2023-11-02T19:26:55.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d877418519036df8-006543f79d-2da3-a1'), ('Date', 'Thu, 02 Nov 2023 19:25:17 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:55.034 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.034 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: No --period given, using current period=b3a94447-46df-4768-8340-cf63e4392a5c 2023-11-02T19:26:55.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: [ 2023-11-02T19:26:55.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:55.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T17:49:46.719075Z" 2023-11-02T19:26:55.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:55.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T18:46:02.156551Z" 2023-11-02T19:26:55.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698953111.156002_848.1", 2023-11-02T19:26:55.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T19:25:11.156002Z" 2023-11-02T19:26:55.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:55.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T18:46:02.124040Z" 2023-11-02T19:26:55.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:55.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-11-02T19:26:55.043 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.044 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:25:18.328+0000 7f0842c96f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:55.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: 2023-11-02T19:25:18.332+0000 7f0842c96f40 20 RGW-SYNC:meta: read sync status 2023-11-02T19:26:55.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:55.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:55.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:55.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:55.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 1, 2023-11-02T19:26:55.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:55.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:55.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:55.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:55.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:55.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:55.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:55.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698953111.156002_848.1", 2023-11-02T19:26:55.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:25:11.156002Z", 2023-11-02T19:26:55.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:55.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "state": 1, 2023-11-02T19:26:55.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:55.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:55.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "realm_epoch": 2 2023-11-02T19:26:55.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 1, 2023-11-02T19:26:55.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 1 2023-11-02T19:26:55.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:18Z" 2023-11-02T19:26:55.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-11-02T19:26:55.067 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.067 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.067 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.067 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.068 INFO:tasks.rgw_multisite_tests: "period": "b3a94447-46df-4768-8340-cf63e4392a5c", 2023-11-02T19:26:55.068 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:55.068 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.069 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.069 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.069 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.070 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.070 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:55.070 INFO:tasks.rgw_multisite_tests: "marker": "1_1698947386.719075_41.1", 2023-11-02T19:26:55.071 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.071 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-11-02T19:26:55.071 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.072 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T17:49:46.719075Z", 2023-11-02T19:26:55.072 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:55.072 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.072 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.073 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.073 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.074 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:55.075 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.156551_196.1", 2023-11-02T19:26:55.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.076 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.156551Z", 2023-11-02T19:26:55.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:55.076 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.077 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.077 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.077 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.078 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.078 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:55.078 INFO:tasks.rgw_multisite_tests: "marker": "1_1698953111.156002_848.1", 2023-11-02T19:26:55.079 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.079 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.079 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.079 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:25:11.156002Z", 2023-11-02T19:26:55.080 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:55.080 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.080 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.081 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.081 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.081 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.082 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-11-02T19:26:55.082 INFO:tasks.rgw_multisite_tests: "marker": "1_1698950762.124040_294.1", 2023-11-02T19:26:55.082 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.082 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.083 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.083 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T18:46:02.124040Z", 2023-11-02T19:26:55.083 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-11-02T19:26:55.084 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.084 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.084 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.085 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.085 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.085 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-11-02T19:26:55.085 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-11-02T19:26:55.086 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.086 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:18Z" 2023-11-02T19:26:55.086 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.087 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1698947386.719075_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-11-02T17:49:46.719075Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1698950762.156551_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.156551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1698953111.156002_848.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:25:11.156002Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1698950762.124040_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T18:46:02.124040Z', 'realm_epoch': 2}}] 2023-11-02T19:26:55.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:55.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1698947386.719075_41.1', 1: '1_1698950762.156551_196.1', 2: '1_1698953111.156002_848.1', 3: '1_1698950762.124040_294.1'} 2023-11-02T19:26:55.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-11-02T19:26:55.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: [ 2023-11-02T19:26:55.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052231", 2023-11-02T19:26:55.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T19:25:17.957254Z" 2023-11-02T19:26:55.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046358", 2023-11-02T19:26:55.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "last_update": "2023-11-02T19:25:17.991605Z" 2023-11-02T19:26:55.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:26:55.096 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.096 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:18Z" 2023-11-02T19:26:55.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.117 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.117 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.117 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.118 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.118 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.118 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.119 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.119 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.119 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.120 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.120 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.120 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.121 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.121 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.121 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.122 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.122 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.122 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.123 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.123 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.123 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.124 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.124 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.124 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.125 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.125 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.125 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.126 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.126 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.126 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.126 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.127 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.127 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.127 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.128 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.128 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.129 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.129 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.129 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.130 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.130 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.130 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.131 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.131 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.131 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.132 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.132 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.132 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.133 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.133 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.133 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.134 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.134 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.134 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.135 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:18Z" 2023-11-02T19:26:55.135 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.135 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.137 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.137 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:23Z" 2023-11-02T19:26:55.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.157 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.158 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.158 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.158 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.159 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.159 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.159 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.160 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.160 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.160 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.161 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.161 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.161 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.162 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.163 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.163 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.163 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.164 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.164 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.164 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.165 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.165 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.165 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.165 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.166 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.166 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.166 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.167 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.167 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.167 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.168 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.168 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.168 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.169 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.169 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.169 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.170 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.170 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.170 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.171 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.171 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.171 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.172 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.172 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.172 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.172 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.173 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.173 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.173 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.174 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.174 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.174 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.175 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.175 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.175 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:23Z" 2023-11-02T19:26:55.176 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.176 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:28Z" 2023-11-02T19:26:55.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.198 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.199 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.199 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.199 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.200 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.200 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.200 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.201 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.201 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.201 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.202 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.202 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.202 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.203 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.203 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.203 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.204 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.204 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.204 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.204 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.205 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.205 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.205 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.206 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.206 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.207 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.207 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.207 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.208 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.208 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.208 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.209 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.209 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.209 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.210 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.210 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.210 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.211 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.211 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.211 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.212 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.212 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.212 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.212 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.213 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.213 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.214 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.214 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.214 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.215 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.215 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.215 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.216 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.216 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.216 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.217 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:28Z" 2023-11-02T19:26:55.217 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.217 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:34Z" 2023-11-02T19:26:55.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.244 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.244 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.244 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.245 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.245 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.245 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.245 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.246 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.246 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.246 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.247 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.247 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.247 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.248 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.248 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.249 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.249 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.249 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.250 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.250 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.250 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.251 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.251 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.252 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.252 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.252 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.253 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.253 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.253 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.253 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.254 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.254 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.255 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.255 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.255 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.256 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.256 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.256 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.256 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.257 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.257 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.257 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.258 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.258 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.258 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.259 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.259 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.259 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.259 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.260 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.260 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.260 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.261 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.261 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.261 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.262 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:34Z" 2023-11-02T19:26:55.262 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.262 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:39Z" 2023-11-02T19:26:55.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.285 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.285 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.285 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.286 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.286 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.286 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.287 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.287 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.287 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.288 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.288 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.288 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.289 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.289 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.289 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.290 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.290 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.290 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.291 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.291 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.291 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.291 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.292 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.292 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.292 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.293 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.293 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.293 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.294 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.294 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.294 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.295 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.295 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.295 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.296 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.296 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.296 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.296 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.297 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.297 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.297 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.298 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.298 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.298 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.299 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.299 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.299 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.300 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.300 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.300 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.300 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.301 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.301 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.301 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.302 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.302 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.302 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.303 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:39Z" 2023-11-02T19:26:55.303 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.303 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:44Z" 2023-11-02T19:26:55.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.326 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.326 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.326 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.326 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.327 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.327 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.327 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.328 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.328 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.328 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.329 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.329 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.330 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.330 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.330 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.331 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.331 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.331 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.332 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.332 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.332 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.332 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.333 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.333 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.333 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.334 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.334 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.334 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.335 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.335 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.335 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.336 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.336 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.336 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.337 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.337 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.337 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.338 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.338 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.338 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.339 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.339 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.339 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.340 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.340 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.340 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.341 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.341 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.341 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.341 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.342 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.342 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.342 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.343 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.343 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.343 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:44Z" 2023-11-02T19:26:55.344 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.344 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:49Z" 2023-11-02T19:26:55.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.366 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.366 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.367 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.367 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.367 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.368 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.368 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.368 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.369 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.369 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.369 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.370 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.370 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.370 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.371 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.371 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.371 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.372 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.372 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.372 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.373 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.373 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.373 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.373 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.374 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.374 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.374 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.375 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.375 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.375 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.376 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.376 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.376 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.377 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.377 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.377 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.378 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.378 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.378 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.379 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.379 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.379 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.379 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.380 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.380 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.380 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.381 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.381 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.381 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.382 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.382 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.382 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.383 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.383 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.383 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.383 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.384 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.384 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:49Z" 2023-11-02T19:26:55.384 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.385 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:54Z" 2023-11-02T19:26:55.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.408 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.409 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.409 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.409 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.409 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.410 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.410 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.410 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.411 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.411 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.411 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.412 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.412 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.412 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.412 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.413 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.413 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.413 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.414 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.414 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.414 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.415 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.415 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.415 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.416 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.416 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.416 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.417 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.417 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.417 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.417 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.418 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.418 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.418 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.419 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.419 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.419 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.420 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.420 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.420 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.421 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.421 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.421 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.421 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.422 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.422 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.423 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.423 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.424 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.424 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.424 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.425 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.425 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.425 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.426 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.426 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:54Z" 2023-11-02T19:26:55.426 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.426 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.428 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.428 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:25:59Z" 2023-11-02T19:26:55.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.449 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.449 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.449 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.450 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.450 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.450 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.451 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.451 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.451 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.452 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.452 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.452 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.453 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.454 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.454 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.455 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.455 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.455 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.455 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.456 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.456 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.456 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.457 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.457 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.457 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.458 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.458 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.458 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.459 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.459 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.459 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.460 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.460 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.461 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.461 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.461 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.462 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.462 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.462 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.463 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.463 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.464 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.465 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.465 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.465 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.465 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.466 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.466 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.466 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.467 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:25:59Z" 2023-11-02T19:26:55.467 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.467 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:26:05Z" 2023-11-02T19:26:55.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.490 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.490 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.491 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.491 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.491 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.492 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.492 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.492 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.492 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.493 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.493 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.493 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.494 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.494 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.494 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.495 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.495 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.495 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.496 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.496 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.496 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.496 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.497 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.497 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.498 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.498 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.498 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.499 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.499 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.499 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.500 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.500 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.500 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.501 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.502 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.502 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.502 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.503 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.503 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.503 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.504 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.504 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.505 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.505 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.506 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.506 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.506 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.507 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.507 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.507 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.508 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.508 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:26:05Z" 2023-11-02T19:26:55.508 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.509 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.510 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.510 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:26:10Z" 2023-11-02T19:26:55.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.531 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.531 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.532 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.532 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.532 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.533 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.533 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.533 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.534 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.534 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.534 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.535 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.535 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.535 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.536 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.536 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.536 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.537 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.537 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.537 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.538 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.538 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.538 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.539 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.539 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.539 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.540 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.540 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.540 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.541 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.541 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.541 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.541 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.542 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.542 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.543 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.543 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.543 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.544 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.544 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.544 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.545 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.545 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.545 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.546 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.546 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.546 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.547 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.547 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.547 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.547 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.548 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.548 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.548 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.549 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.549 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:26:10Z" 2023-11-02T19:26:55.549 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.550 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.551 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.552 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:26:15Z" 2023-11-02T19:26:55.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.572 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.573 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.573 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.573 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.574 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.574 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.574 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.575 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.575 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.575 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.575 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.576 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.576 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.576 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.577 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.577 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.578 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.578 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.578 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.579 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.579 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.579 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.580 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.580 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.580 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.580 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.581 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.581 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.581 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.582 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.582 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.582 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.583 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.583 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.584 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.584 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.584 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.585 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.585 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.585 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.585 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.586 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.586 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.586 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.587 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.587 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.587 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.588 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.588 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.588 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.589 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.589 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.589 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.590 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.590 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.590 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:26:15Z" 2023-11-02T19:26:55.591 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.591 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:26:20Z" 2023-11-02T19:26:55.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.613 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.614 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.614 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.614 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.615 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.615 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.615 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.615 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.616 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.616 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.616 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.617 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.617 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.617 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.618 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.618 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.618 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.619 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.619 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.619 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.620 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.620 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.620 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.620 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.621 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.621 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.621 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.622 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.622 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.622 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.623 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.623 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.623 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.624 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.624 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.624 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.625 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.625 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.625 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.626 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.626 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.626 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.627 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.627 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.627 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.628 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.628 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.628 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.629 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.629 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.629 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.630 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.630 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.630 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.631 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.631 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.631 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:26:20Z" 2023-11-02T19:26:55.632 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.632 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.633 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:26:25Z" 2023-11-02T19:26:55.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.655 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.656 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.656 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.656 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.656 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.657 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.657 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.657 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.658 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.658 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.658 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.659 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.659 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.659 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.660 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.660 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.660 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.661 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.661 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.661 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.661 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.662 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.662 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.662 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.663 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.663 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.663 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.664 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.664 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.664 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.665 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.665 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.665 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.666 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.666 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.666 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.667 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.667 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.667 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.668 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.668 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.668 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.669 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.669 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.669 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.670 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.670 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.670 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.671 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.671 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.671 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.672 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.672 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.672 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.672 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.673 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.673 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:26:25Z" 2023-11-02T19:26:55.673 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.674 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.676 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:26:30Z" 2023-11-02T19:26:55.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.696 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.696 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.697 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.697 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.697 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.697 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.698 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.698 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.698 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.699 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.699 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.699 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.700 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.701 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.701 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.701 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.702 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.702 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.702 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.703 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.703 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.703 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.704 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.704 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.705 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.705 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.705 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.706 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.706 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.706 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.707 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.707 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.708 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.708 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.708 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.709 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.709 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.709 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.709 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.710 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.710 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.711 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.711 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.712 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.712 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.712 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.713 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.713 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.713 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.713 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.714 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:26:30Z" 2023-11-02T19:26:55.714 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.714 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:26:35Z" 2023-11-02T19:26:55.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.737 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.737 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.737 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.738 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.738 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.738 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.739 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.739 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.739 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.740 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.740 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.740 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.741 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.742 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.742 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.742 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.743 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.743 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.743 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.743 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.744 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.744 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.745 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.745 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.745 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.746 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.746 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.746 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.747 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.747 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.747 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.747 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.748 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.748 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.748 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.749 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.749 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.749 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.750 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.750 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.750 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.751 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.751 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.752 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.752 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.753 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.753 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.753 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.754 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.754 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.754 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.755 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:26:35Z" 2023-11-02T19:26:55.755 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.755 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:26:41Z" 2023-11-02T19:26:55.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.777 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.778 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.778 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.778 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.779 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.779 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.779 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.779 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.780 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.780 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.780 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.781 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.781 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.782 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.782 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.783 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.783 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.783 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.783 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.784 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.784 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.785 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.785 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.785 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.786 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.786 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.786 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.787 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.787 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.787 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.788 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.788 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.788 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.789 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.789 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.789 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.789 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.790 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.790 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.790 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.791 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.791 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.791 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.792 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.793 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.793 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.793 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.794 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.794 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.794 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.794 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.795 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.795 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:26:41Z" 2023-11-02T19:26:55.795 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.796 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.797 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.798 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:26:46Z" 2023-11-02T19:26:55.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.818 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.818 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.819 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.819 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.819 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.820 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.820 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.820 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.820 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.821 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.821 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.821 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.822 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.823 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.823 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.824 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.824 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.824 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.824 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.825 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052090", 2023-11-02T19:26:55.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.825 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.826 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.826 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.140415Z" 2023-11-02T19:26:55.826 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.827 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.827 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.827 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.828 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.828 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.828 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.829 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.829 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.829 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.830 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.830 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.830 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.831 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.831 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.831 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.832 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046217", 2023-11-02T19:26:55.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.832 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.112241Z" 2023-11-02T19:26:55.833 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.833 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.834 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.834 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.834 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.835 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.835 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.835 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.836 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:26:46Z" 2023-11-02T19:26:55.836 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.836 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.140415Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.112241Z'}}] 2023-11-02T19:26:55.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052090', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046217'} 2023-11-02T19:26:55.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052231 target=00000000000000000000:00000000000000052090, shard=3 master=00000000000000000000:00000000000000046358 target=00000000000000000000:00000000000000046217 2023-11-02T19:26:55.838 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.838 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-11-02T19:26:55.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-11-02T19:26:55.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setuser ceph since I am not root 2023-11-02T19:26:55.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-11-02T19:26:55.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "sync_status": { 2023-11-02T19:26:55.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "info": { 2023-11-02T19:26:55.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "sync", 2023-11-02T19:26:55.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "num_shards": 4, 2023-11-02T19:26:55.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "instance_id": 1320917798396668347 2023-11-02T19:26:55.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "markers": [ 2023-11-02T19:26:55.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 0, 2023-11-02T19:26:55.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 9, 2023-11-02T19:26:55.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 1, 2023-11-02T19:26:55.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000052231", 2023-11-02T19:26:55.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 55, 2023-11-02T19:26:55.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:25:17.957254Z" 2023-11-02T19:26:55.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 2, 2023-11-02T19:26:55.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: { 2023-11-02T19:26:55.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "key": 3, 2023-11-02T19:26:55.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "val": { 2023-11-02T19:26:55.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "status": "incremental-sync", 2023-11-02T19:26:55.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "marker": "00000000000000000000:00000000000000046358", 2023-11-02T19:26:55.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "next_step_marker": "", 2023-11-02T19:26:55.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total_entries": 0, 2023-11-02T19:26:55.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "pos": 0, 2023-11-02T19:26:55.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "timestamp": "2023-11-02T19:25:17.991605Z" 2023-11-02T19:26:55.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: ] 2023-11-02T19:26:55.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "full_sync": { 2023-11-02T19:26:55.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "total": 64, 2023-11-02T19:26:55.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "complete": 64 2023-11-02T19:26:55.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: }, 2023-11-02T19:26:55.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: "current_time": "2023-11-02T19:26:51Z" 2023-11-02T19:26:55.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi017.stdout: INFO: } 2023-11-02T19:26:55.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-11-02T19:26:55.860 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-11-02T19:26:55.860 INFO:tasks.rgw_multisite_tests: "info": { 2023-11-02T19:26:55.860 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-11-02T19:26:55.861 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-11-02T19:26:55.861 INFO:tasks.rgw_multisite_tests: "instance_id": 1320917798396668347 2023-11-02T19:26:55.861 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.862 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-11-02T19:26:55.862 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.862 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-11-02T19:26:55.863 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.863 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.863 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032480", 2023-11-02T19:26:55.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.864 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-11-02T19:26:55.864 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:24:33.203114Z" 2023-11-02T19:26:55.865 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.865 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.865 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.866 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-11-02T19:26:55.866 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.866 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.867 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052231", 2023-11-02T19:26:55.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.867 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-11-02T19:26:55.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.868 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:25:17.957254Z" 2023-11-02T19:26:55.868 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.869 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.869 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.869 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-11-02T19:26:55.869 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.870 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.870 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044887", 2023-11-02T19:26:55.870 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.871 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.871 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.871 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:19:53.938682Z" 2023-11-02T19:26:55.872 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.872 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.872 INFO:tasks.rgw_multisite_tests: { 2023-11-02T19:26:55.873 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-11-02T19:26:55.873 INFO:tasks.rgw_multisite_tests: "val": { 2023-11-02T19:26:55.873 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-11-02T19:26:55.873 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046358", 2023-11-02T19:26:55.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-11-02T19:26:55.874 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-11-02T19:26:55.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-11-02T19:26:55.875 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-11-02T19:25:17.991605Z" 2023-11-02T19:26:55.875 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.875 INFO:tasks.rgw_multisite_tests: } 2023-11-02T19:26:55.876 INFO:tasks.rgw_multisite_tests: ] 2023-11-02T19:26:55.876 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.876 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-11-02T19:26:55.877 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-11-02T19:26:55.877 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-11-02T19:26:55.877 INFO:tasks.rgw_multisite_tests: }, 2023-11-02T19:26:55.877 INFO:tasks.rgw_multisite_tests: "current_time": "2023-11-02T19:26:51Z" 2023-11-02T19:26:55.878 INFO:tasks.rgw_multisite_tests:} 2023-11-02T19:26:55.878 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-11-02T19:24:33.203114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052231', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-11-02T19:25:17.957254Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044887', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:19:53.938682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-11-02T19:25:17.991605Z'}}] 2023-11-02T19:26:55.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000052231', 2: '00000000000000000000:00000000000000044887', 3: '00000000000000000000:00000000000000046358'} 2023-11-02T19:26:55.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-11-02T19:26:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-74/ 2023-11-02T19:26:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-74/ 2023-11-02T19:26:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-74/ 2023-11-02T19:26:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-11-02T19:26:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:55.883 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:55.884 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.884 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.884 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 19:26:51 GMT 2023-11-02T19:26:55.885 INFO:tasks.rgw_multisite_tests:/ibtljn-74/ 2023-11-02T19:26:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:55.885 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:nejfX5PcSy7FKAZOMEtLJ4M39Fk= 2023-11-02T19:26:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:26:51 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:nejfX5PcSy7FKAZOMEtLJ4M39Fk=', 'Content-Length': '0'} 2023-11-02T19:26:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c3f9c53f762ed60-006543f7fb-2da9-a2'), ('Content-Length', '0'), ('Date', 'Thu, 02 Nov 2023 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ibtljn-74/obj1 2023-11-02T19:26:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ibtljn-74/obj1 2023-11-02T19:26:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-11-02T19:26:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ibtljn-74/obj1 2023-11-02T19:26:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-11-02T19:26:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-11-02T19:26:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi017.front.sepia.ceph.com:8001 2023-11-02T19:26:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-11-02T19:26:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-11-02T19:26:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-11-02T19:26:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-11-02T19:26:55.889 INFO:tasks.rgw_multisite_tests:HEAD 2023-11-02T19:26:55.890 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.890 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.890 INFO:tasks.rgw_multisite_tests:Thu, 02 Nov 2023 19:26:51 GMT 2023-11-02T19:26:55.891 INFO:tasks.rgw_multisite_tests:/ibtljn-74/obj1 2023-11-02T19:26:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-11-02T19:26:55.891 INFO:tasks.rgw_multisite_tests:AWS 7LD6AK3P9XRGER738BKH:LNW8tdA1ZAJsEBnnECNZNLDAo+g= 2023-11-02T19:26:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 02 Nov 2023 19:26:51 GMT', 'Authorization': 'AWS 7LD6AK3P9XRGER738BKH:LNW8tdA1ZAJsEBnnECNZNLDAo+g=', 'Content-Length': '0'} 2023-11-02T19:26:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006425f18edd956da8-006543f7fb-2da9-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 02 Nov 2023 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2023-11-02T19:26:55.892 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-11-02T19:26:55.893 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.893 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-11-02T19:26:55.893 INFO:tasks.rgw_multisite_tests:Ran 47 tests in 5705.295s 2023-11-02T19:26:55.894 INFO:tasks.rgw_multisite_tests: 2023-11-02T19:26:55.894 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, failures=3) 2023-11-02T19:26:55.895 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-11-02T19:26:55.963 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=de1e92c8409740d39211377b105a8cc0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_b8ceb241941938aa7529adcc4b99d42d196489ec/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-11-02T19:26:55.965 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2023-11-02T19:26:55.975 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2023-11-02T19:26:55.993 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-11-02T19:26:56.002 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-11-02T19:26:56.002 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:26:56.015 INFO:tasks.rgw.c1.client.0.smithi017.stdout:2023-11-02T19:26:55.999+0000 7f0dea985700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 33987) UID: 0 2023-11-02T19:26:56.016 INFO:tasks.rgw.c1.client.0.smithi017.stdout:2023-11-02T19:26:55.999+0000 7f0dee1e4f80 -1 shutting down 2023-11-02T19:26:56.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-11-02T19:27:02.104 INFO:tasks.rgw.c1.client.0:Stopped 2023-11-02T19:27:02.105 DEBUG:teuthology.orchestra.run.smithi017:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2023-11-02T19:27:02.112 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/vault-root-token 2023-11-02T19:27:02.172 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-11-02T19:27:02.172 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:02.173 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T19:27:02.171+0000 7f25fbcd4700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 83487) UID: 0 2023-11-02T19:27:02.173 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2023-11-02T19:27:02.171+0000 7f25ff533f80 -1 shutting down 2023-11-02T19:27:02.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2023-11-02T19:27:07.729 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-11-02T19:27:07.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-11-02T19:27:08.274 INFO:tasks.rgw.c1.client.1:Stopped 2023-11-02T19:27:08.275 DEBUG:teuthology.orchestra.run.smithi017:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2023-11-02T19:27:08.282 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/vault-root-token 2023-11-02T19:27:08.343 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-11-02T19:27:08.343 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:08.344 INFO:tasks.rgw.c2.client.0.smithi069.stdout:2023-11-02T19:27:08.340+0000 7fdb1d434700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 34939) UID: 0 2023-11-02T19:27:08.345 INFO:tasks.rgw.c2.client.0.smithi069.stdout:2023-11-02T19:27:08.340+0000 7fdb20c93f80 -1 shutting down 2023-11-02T19:27:13.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-11-02T19:27:13.737 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-11-02T19:27:14.446 INFO:tasks.rgw.c2.client.0:Stopped 2023-11-02T19:27:14.447 DEBUG:teuthology.orchestra.run.smithi069:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2023-11-02T19:27:14.454 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/vault-root-token 2023-11-02T19:27:14.466 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-11-02T19:27:14.467 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:14.510 INFO:tasks.rgw.c2.client.1.smithi069.stdout:2023-11-02T19:27:14.508+0000 7f1bd42e9700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 36041) UID: 0 2023-11-02T19:27:14.511 INFO:tasks.rgw.c2.client.1.smithi069.stdout:2023-11-02T19:27:14.508+0000 7f1bd7b48f80 -1 shutting down 2023-11-02T19:27:19.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-11-02T19:27:19.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-11-02T19:27:20.569 INFO:tasks.rgw.c2.client.1:Stopped 2023-11-02T19:27:20.569 DEBUG:teuthology.orchestra.run.smithi069:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2023-11-02T19:27:20.577 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/vault-root-token 2023-11-02T19:27:20.591 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-02T19:27:20.602 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2023-11-02T19:27:20.968 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-02T19:27:20.969 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-02T19:27:20.969 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2023-11-02T19:27:20.970 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:21.071 INFO:tasks.ceph.c2.osd.0:Stopped 2023-11-02T19:27:21.072 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2023-11-02T19:27:21.073 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:21.147 INFO:tasks.ceph.c2.osd.1:Stopped 2023-11-02T19:27:21.147 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2023-11-02T19:27:21.148 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:21.237 INFO:tasks.ceph.c2.osd.2:Stopped 2023-11-02T19:27:21.238 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-02T19:27:21.239 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2023-11-02T19:27:21.239 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:21.268 INFO:tasks.ceph.c2.mgr.x:Stopped 2023-11-02T19:27:21.268 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-02T19:27:21.269 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2023-11-02T19:27:21.269 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:21.301 INFO:tasks.ceph.c2.mon.a:Stopped 2023-11-02T19:27:21.303 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-02T19:27:21.303 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2023-11-02T19:27:21.322 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-02T19:27:21.322 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2023-11-02T19:27:21.973 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-02T19:27:21.974 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2023-11-02T19:27:22.197 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-02T19:27:22.198 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2023-11-02T19:27:22.389 INFO:tasks.ceph:Archiving mon data... 2023-11-02T19:27:22.391 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849/data/c2.mon.a.tgz 2023-11-02T19:27:22.391 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2023-11-02T19:27:22.397 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.JWtSXnPkIi 2023-11-02T19:27:22.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.JWtSXnPkIi 2023-11-02T19:27:22.518 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.JWtSXnPkIi 2023-11-02T19:27:22.540 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.JWtSXnPkIi is 386KB 2023-11-02T19:27:22.604 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.JWtSXnPkIi 2023-11-02T19:27:22.611 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-02T19:27:22.612 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-11-02T19:27:22.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-11-02T19:27:22.629 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-02T19:27:22.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2023-11-02T19:27:22.964 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-02T19:27:22.965 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-02T19:27:22.966 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2023-11-02T19:27:22.966 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:23.054 INFO:tasks.ceph.c1.osd.0:Stopped 2023-11-02T19:27:23.055 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2023-11-02T19:27:23.056 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:24.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~0s 2023-11-02T19:27:24.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-11-02T19:27:29.158 INFO:tasks.ceph.c1.osd.1:Stopped 2023-11-02T19:27:29.159 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2023-11-02T19:27:29.159 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:29.250 INFO:tasks.ceph.c1.osd.2:Stopped 2023-11-02T19:27:29.250 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-02T19:27:29.251 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2023-11-02T19:27:29.251 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:29.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~5s 2023-11-02T19:27:29.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~0s 2023-11-02T19:27:29.283 INFO:tasks.ceph.c1.mgr.x:Stopped 2023-11-02T19:27:29.284 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-02T19:27:29.284 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2023-11-02T19:27:29.285 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T19:27:29.312 INFO:tasks.ceph.c1.mon.a:Stopped 2023-11-02T19:27:29.313 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-02T19:27:29.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2023-11-02T19:27:29.331 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi017.front.sepia.ceph.com 2023-11-02T19:27:29.331 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2023-11-02T19:27:29.999 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi017.front.sepia.ceph.com 2023-11-02T19:27:29.999 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2023-11-02T19:27:30.289 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi017.front.sepia.ceph.com 2023-11-02T19:27:30.290 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2023-11-02T19:27:30.473 INFO:tasks.ceph:Archiving mon data... 2023-11-02T19:27:30.473 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849/data/c1.mon.a.tgz 2023-11-02T19:27:30.474 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2023-11-02T19:27:30.478 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.Puwg1IPIDj 2023-11-02T19:27:30.479 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.Puwg1IPIDj 2023-11-02T19:27:30.673 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.Puwg1IPIDj 2023-11-02T19:27:30.698 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.Puwg1IPIDj is 490KB 2023-11-02T19:27:30.759 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.Puwg1IPIDj 2023-11-02T19:27:30.766 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-02T19:27:30.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-11-02T19:27:30.769 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-11-02T19:27:30.785 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-02T19:27:30.786 DEBUG:teuthology.orchestra.run.smithi017:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-02T19:27:30.825 DEBUG:teuthology.orchestra.run.smithi069:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-02T19:27:30.846 INFO:teuthology.orchestra.run.smithi017.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-02T19:27:30.909 INFO:teuthology.orchestra.run.smithi069.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-02T19:27:30.916 INFO:tasks.ceph:Archiving crash dumps... 2023-11-02T19:27:30.918 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849/remote/smithi017/crash 2023-11-02T19:27:30.919 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-02T19:27:30.935 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849/remote/smithi069/crash 2023-11-02T19:27:30.937 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-02T19:27:30.954 INFO:tasks.ceph:Compressing logs... 2023-11-02T19:27:30.954 DEBUG:teuthology.orchestra.run.smithi017:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-02T19:27:30.957 DEBUG:teuthology.orchestra.run.smithi069:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-02T19:27:34.284 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 has been restored 2023-11-02T19:27:34.284 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x has been restored 2023-11-02T19:29:28.586 INFO:tasks.ceph:Archiving logs... 2023-11-02T19:29:28.588 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849/remote/smithi017/log 2023-11-02T19:29:28.589 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-02T19:29:42.518 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849/remote/smithi069/log 2023-11-02T19:29:42.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-02T19:29:52.339 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-02T19:29:52.350 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-02T19:29:52.351 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-02T19:29:52.354 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-02T19:29:52.432 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-02T19:29:52.442 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-02T19:29:52.442 DEBUG:teuthology.orchestra.run.smithi017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-02T19:29:52.445 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-02T19:29:52.468 INFO:teuthology.orchestra.run.smithi017.stdout: remote refid st t when poll reach delay offset jitter 2023-11-02T19:29:52.468 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================== 2023-11-02T19:29:52.469 INFO:teuthology.orchestra.run.smithi017.stdout:*hv01.front.sepi 44.4.53.2 2 u 93 128 377 0.097 0.045 0.029 2023-11-02T19:29:52.469 INFO:teuthology.orchestra.run.smithi017.stdout:+hv02.front.sepi 63.231.80.2 3 u 69 128 377 0.084 1.017 0.034 2023-11-02T19:29:52.469 INFO:teuthology.orchestra.run.smithi017.stdout:+hv03.front.sepi 74.6.168.72 3 u 90 128 377 0.090 -0.166 0.072 2023-11-02T19:29:52.470 INFO:teuthology.orchestra.run.smithi017.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-11-02T19:29:52.471 INFO:teuthology.orchestra.run.smithi069.stdout: remote refid st t when poll reach delay offset jitter 2023-11-02T19:29:52.472 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================== 2023-11-02T19:29:52.472 INFO:teuthology.orchestra.run.smithi069.stdout:*hv01.front.sepi 44.4.53.2 2 u 137 256 377 0.165 -0.173 0.018 2023-11-02T19:29:52.472 INFO:teuthology.orchestra.run.smithi069.stdout:+hv02.front.sepi 63.231.80.2 3 u 121 256 377 0.161 0.744 0.063 2023-11-02T19:29:52.473 INFO:teuthology.orchestra.run.smithi069.stdout:+hv03.front.sepi 74.6.168.72 3 u 141 256 377 0.175 -0.388 0.280 2023-11-02T19:29:52.473 INFO:teuthology.orchestra.run.smithi069.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-11-02T19:29:52.474 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-02T19:29:52.484 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-02T19:29:52.484 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-02T19:29:52.498 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-02T19:29:52.512 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-02T19:29:52.524 INFO:teuthology.task.internal:Duration was 6660.404866 seconds 2023-11-02T19:29:52.525 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-02T19:29:52.540 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-02T19:29:52.540 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-02T19:29:52.543 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-02T19:29:52.595 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-02T19:29:52.596 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi017.front.sepia.ceph.com 2023-11-02T19:29:52.596 DEBUG:teuthology.orchestra.run.smithi017:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-02T19:29:52.606 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2023-11-02T19:29:52.607 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-02T19:29:52.620 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-02T19:29:52.621 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-02T19:29:52.623 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-02T19:29:52.654 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-02T19:29:52.654 DEBUG:teuthology.orchestra.run.smithi017:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-02T19:29:52.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-02T19:29:52.774 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-02T19:29:52.785 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-02T19:29:52.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-02T19:29:52.802 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-02T19:29:52.819 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-02T19:29:52.830 DEBUG:teuthology.orchestra.run.smithi017:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-02T19:29:52.833 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-02T19:29:52.845 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2023-11-02T19:29:52.846 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = core 2023-11-02T19:29:52.947 DEBUG:teuthology.orchestra.run.smithi017:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-02T19:29:52.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T19:29:52.954 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-02T19:29:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T19:29:52.996 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-02T19:29:53.006 INFO:teuthology.task.internal:Transferring archived files... 2023-11-02T19:29:53.007 DEBUG:teuthology.misc:Transferring archived files from smithi017:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849/remote/smithi017 2023-11-02T19:29:53.008 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-02T19:29:53.030 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849/remote/smithi069 2023-11-02T19:29:53.031 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-02T19:29:53.053 INFO:teuthology.task.internal:Removing archive directory... 2023-11-02T19:29:53.053 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-02T19:29:53.056 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-02T19:29:53.096 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-02T19:29:53.108 INFO:teuthology.task.internal:Not uploading archives. 2023-11-02T19:29:53.108 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-02T19:29:53.120 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-02T19:29:53.120 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-02T19:29:53.122 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-02T19:29:53.128 INFO:teuthology.orchestra.run.smithi017.stdout: 924113 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 2 19:29 /home/ubuntu/cephtest 2023-11-02T19:29:53.130 INFO:teuthology.orchestra.run.smithi069.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 2 19:29 /home/ubuntu/cephtest 2023-11-02T19:29:53.133 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-02T19:29:53.145 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-02T19:29:53.169 INFO:teuthology.nuke:Checking targets against current locks 2023-11-02T19:29:53.197 DEBUG:teuthology.nuke:shortname: smithi017 2023-11-02T19:29:53.198 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-02T19:29:53.224 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-02 17:29:43.177004', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-11-02T19:29:53.228 DEBUG:teuthology.nuke:shortname: smithi069 2023-11-02T19:29:53.228 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-02T19:29:53.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-01_14:50:55-rgw-wip-yuri3-testing-2023-10-31-1116-reef-distro-default-smithi/7442849', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-02 17:29:43.175265', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-11-02T19:29:53.280 INFO:teuthology.orchestra.console.smithi017:Power off 2023-11-02T19:29:53.281 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-02T19:29:53.297 INFO:teuthology.orchestra.console.smithi069:Power off 2023-11-02T19:29:53.298 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-02T19:29:53.308 DEBUG:teuthology.orchestra.console.smithi017:power off output: Chassis Power Control: Down/Off 2023-11-02T19:29:53.309 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T19:29:53.322 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2023-11-02T19:29:53.322 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T19:29:53.332 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2023-11-02T19:29:53.345 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-11-02T19:29:57.333 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T19:29:57.346 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T19:29:57.446 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2023-11-02T19:29:57.457 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-11-02T19:30:01.449 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T19:30:01.459 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T19:30:01.561 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2023-11-02T19:30:01.570 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2023-11-02T19:30:01.671 INFO:teuthology.orchestra.console.smithi069:Power off completed 2023-11-02T19:30:05.564 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T19:30:05.676 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is off 2023-11-02T19:30:05.778 INFO:teuthology.orchestra.console.smithi017:Power off completed 2023-11-02T19:30:05.882 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi} duration: 6660.4048664569855 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=de1e92c8409740d39211377b105a8cc0 status: fail success: false 2023-11-02T19:30:05.883 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-02T19:30:05.950 INFO:teuthology.run:FAIL